Testing multiple hypotheses

[Agent Smith]

Say we're considering multiple hypotheses: $H_0, H_1, H_2, ... H_ n$
$P(H_0) \leq 0.05$ (our P-value).

Is it possible that $P(H_1) < P(H_0), P(H_2) < P(H_0), ... P(H_n) < P(H_0)$

and yet $P(H_0) + [P(H_1) + \dots + P(H_n)] = 1$?

We would reject the null hypothesis but all the alternative hypotheses are less probable than the null.

 

[Agent Smith]

Say we're considering multiple hypotheses: $H_0, H_1, H_2, ... H_ n$
$P(H_0) \leq 0.05$ (our P-value).

Is it possible that $P(H_1) < P(H_0), P(H_2) < P(H_0), ... P(H_n) < P(H_0)$

and yet $P(H_0) + [P(H_1) + \dots + P(H_n)] = 1$?

We would reject the null hypothesis but all the alternative hypotheses are less probable than the null.

As an example $\frac{1}{2} + \left[\frac{1}{4} + \frac{1}{8} + ...\right] = 1$

 

[FactChecker]

Say we're considering multiple hypotheses: $H_0, H_1, H_2, ... H_ n$
$P(H_0) \leq 0.05$ (our P-value).

Is it possible that $P(H_1) < P(H_0), P(H_2) < P(H_0), ... P(H_n) < P(H_0)$

and yet $P(H_0) + [P(H_1) + \dots + P(H_n)] = 1$?

We would reject the null hypothesis but all the alternative hypotheses are less probable than the null.

This is an unusual statistical test setup. Usually, the probability parameters of the null hypothesis are initially assumed. Suppose that with the null hypothesis probability distribution, the sample is an unlikely event with a probability less than 0.05. Then the alternative hypotheses may be considered and their alternative probability parameters (mean, variance, etc.) are assumed. These are not the same probabilities that they had under the null hypothesis. With maximum likelihood estimated parameters the sample results are much more likely.

For instance, suppose that the null hypothesis is that the sample is from a normal distribution with a mean $\mu_0=0$ and variance $\sigma_0 = 1$. Now suppose that the sample (N=101) has a sample results of $\mu_s=0.4$ and $\sigma_s=1$. Then the null hypothesis mean of 0 is far (4 $\sigma$) outside the 95% confidence interval for that sample (the 95% confidence interval is [0.2, 0.6]). So now we could consider the alternative hypothesis, $H_1$, that the true population distribution has a mean of 0.4. That hypothesized mean would be much more compatible with the sample.

 

[Vanadium 50]

First, there exist tests that distinguish between exactly two hypotheses. (The F test is an example) You are not limited to tests of rejecting the null hypothesis. And yes, both hypotheses can be wrong.

Next, I would say if you calculate the p-value first and then set your threshold, you are not doing statistics. Not sure what I would call it, but this isn't statistics.

Finally, statistics are there to help you make a decision. I remember a test that caused us to reject the null hypothesis at 90 but not 95%. What we did was to make small interventions - less than we would do at 95% - but wouldn't be harmful if the null hypotheses were correct. Probably not a perfect outcome, but one supported by what data we had.

 

[Dale]

Say we're considering multiple hypotheses: $H_0, H_1, H_2, ... H_ n$
$P(H_0) \leq 0.05$ (our P-value).

Is it possible that $P(H_1) < P(H_0), P(H_2) < P(H_0), ... P(H_n) < P(H_0)$

and yet $P(H_0) + [P(H_1) + \dots + P(H_n)] = 1$?

We would reject the null hypothesis but all the alternative hypotheses are less probable than the null.

In frequentist statistics you never assign a probability to a hypothesis. You assign the probability to the data. So it would be $$P( data | H_0)$$ etc.

Bayesian statistics does assign probabilities to hypothesis. So you could indeed see that $$P(H_1|data)< P(H_0|data)<0.05$$ and consider the evidence to refute both hypotheses

 

[Agent Smith]

I think I'm talking about Bayesian statistics. The gist of what I want to say seems to be that every possible hypothesis is improbable at a threshold probability of 0.05.

 

[Dale]

every possible hypothesis is improbable at a threshold probability of 0.05

Certainly not every possible hypothesis. But all of the hypotheses in some specified set of hypotheses.