Which one is correct about hypothesis testing?

[songoku]

Homework Statement

Which one is valid about hypothesis testing?
a) The distribution of test statistic under the null hypothesis is the same as its distribution across experiments using random samples of the same size from the same population

b) The test statistic has same distribution when the null hypothesis is true and when alternative hypothesis is true

c) The null hypothesis is always more likely to be correct than alternative hypothesis

d) We use the distribution of the test statistic when null hypothesis is true

e) The distribution of test statistic under the null hypothesis depends on the direction of the extreme which comes from the alternative hypothesis

Relevant Equations

None

My attempt:

a) I am not really sure I understand this option fully but my answer will be this one is only applicable if the sample size is large and central limit theorem can be applied so (a) is wrong

b) The test statistic has "same distribution" to what? My opinion is (b) is wrong because distribution of test statistic is independent of whether null and alternative hypothesis are correct or not

c) I think this is correct because the probability of alternative hypothesis to be correct is the same as significance level used which is always lower than probability to accept null hypothesis

d) This is wrong because of the same reasoning as (b), distribution of test statistic is independent of whether null hypothesis is correct or not

e) This is wrong because distribution of test statistic is independent of alternative hypothesis


Am I correct? Thanks

 

[WWGD]

It may differ depending on the statistic you're testing for: mean, proportion, F-test for equality of Variance, etc.

 

[songoku]

It may differ depending on the statistic you're testing for: mean, proportion, F-test for equality of Variance, etc.

Oh I don't know about that.

Let say the context is hypothesis testing for difference in mean

 

[FactChecker]

a) I am not really sure I understand this option fully but my answer will be this one is only applicable if the sample size is large and central limit theorem can be applied so (a) is wrong

I agree that it is false, but for a different reason. I interpret "under the null hypothesis" to mean that the null hypothesis is true and the sample is from that distribution. But if the null hypothesis is false, that is not true.

b) The test statistic has "same distribution" to what? My opinion is (b) is wrong because distribution of test statistic is independent of whether null and alternative hypothesis are correct or not

Again, I agree that it is false, but for a different reason. The null hypothesis and the alternative hypothesis usually hypothesize different distributions. So which hypothesis is true definitely effects the distribution and the associated test results.

c) I think this is correct because the probability of alternative hypothesis to be correct is the same as significance level used which is always lower than probability to accept null hypothesis

I disagree. IF the null hypothesis is true, the test result should end up agreeing with the null hypothesis a large percent of the time (95%, 99.5%, etc.). But the truth of the null hypothesis is not a "given". That is what you are testing. For example, suppose I am tossing a coin and the null hypotheses is that the coin is fair. Now suppose that the coin used is actually two-headed. Then the null hypothesis will not be accepted.

d) This is wrong because of the same reasoning as (b), distribution of test statistic is independent of whether null hypothesis is correct or not

We need to be careful here. The actual distribution of the test statistic is unknown. I interpret "we use" to mean "we hypothesize". We hypothesize the distribution of the test statistic to be the one derived if the null hypothesis is true.

e) This is wrong because distribution of test statistic is independent of alternative hypothesis

The hypothesized distribution of the test statistic assumes the null hypothesis is true. Then we test an alternative hypothesis, which will be accepted in one of three possible problem situations:
1) if the test result is too high (one-tail on the high side),
2) too low (one tail on the low side), or
3) too far from the hypothesized test statistic mean (two tails).

 

[songoku]

I think I don't really understand the term "distribution".

I thought something like this:
let say we want to check whether the average running time of an athlete decreases after practicing for a month and the average time is normally distributed. I thought the distribution of the average time is normal and null and alternative hypothesis also have normal distribution because we are testing the mean, which comes from the test statistics. In the case the distribution is not given, I also thought the distribution of test statistics would always be the same as null and alternative hypothesis.

So this is not the case?

 

[WWGD]

I think I don't really understand the term "distribution".

I thought something like this:
let say we want to check whether the average running time of an athlete decreases after practicing for a month and the average time is normally distributed. I thought the distribution of the average time is normal and null and alternative hypothesis also have normal distribution because we are testing the mean, which comes from the test statistics. In the case the distribution is not given, I also thought the distribution of test statistics would always be the same as null and alternative hypothesis.

So this is not the case?

In this case, you would be testing; using the test statistic for the _difference of means_ $T_1, T_2$, not the mean itself. By default, definition, the Null Hypothesis $H_0$ assumes " No effect", so that $T_1=T_2$ *, thus, in this case, that the training didnt improve running time. The alternative could be 1-sided in 2 ways : $T_1>T_2$, $T_1<T_2$, or the more general $T_1 \neq T_2$.
Please let me know if I misunderstood what you wrote.

*This has to see with the fact that you can only disproof through a counterexample, but you can't actually prove anything.

Regarding c), this is false, as Null Hypothesis states " No Effect" by default, design. If you say this is correct, you're saying that most of the time, differences are due to noise, rather than to actual effect(s). But actually something slightly different: That those who test hupotheses are more likely to detect actual effects, rather than noise.

 

[FactChecker]

I think I don't really understand the term "distribution".

It's important to be precise with terminology. It's a learned skill. I'll do my best and hope it doesn't confuse you more.
A distribution of a random variable is the probabilities of results given an infinite number of trials. A "distribution" of a specific, already determined, sample result is different. If there is only one run and he brakes his leg, the "distribution" is that one result (1 hour?)

I thought something like this:
let say we want to check whether the average running time of an athlete

I would say the "expected running time" if you are talking about an imaginary infinite number of trials.

decreases after practicing for a month and the average time is normally distributed.

The "expected value" of a distribution is a single (known or unknown) number. It has no distribution. It is the "average" you would get from an imaginary infinite number of trials. The "expected value" of the average of a set of N runs is also a single (known or unknown) number. It has no distribution. It is the "average" you would get from an imaginary infinite number of N-averaged-trials. A single sample "average" of a set of N results, yet to be determined in a test, is a random variable from a distribution. A single sample "average" of a set of N results, already determined in a test, is a single calculated number which came from a distribution, but is now a known experimental result.

I thought the distribution of the average time is normal and null and alternative hypothesis also have normal distribution

Not all normal distributions are the same. If they have different means or variances, they are not the same distribution.

because we are testing the mean, which comes from the test statistics. In the case the distribution is not given, I also thought the distribution of test statistics would always be the same as null and alternative hypothesis.

If the null hypothesis is that the random variable is from a $N(0,1)$ distribution and the alternative hypothesis is that the random variable is from a $N(100,1)$ distribution, than those two distributions are not the same.

 

[WWGD]

The questions seem overall ambiguous or poorly-written.
Re d) , I assume they mean the test statistic is used , designed if you will, to test whether the Null is true, and assuming it is. Under this interpretation, I say d) Is true.

 

[WWGD]

A distribution is a function that assigns a probability to an event, in the continuous case *, and to singletons and events in the discrete case.
*There may be exceptions to these, but that is the general case.

 

[Hornbein]

b, c, and e are false. That leaves a and d. It's difficult because I don't quite understand what the question asker means. "d) We use the distribution of the test statistic when null hypothesis is true" I think he is saying during a statistical test we assume that the distribution of the test statistic is the same as what the population distribution would be if the null hypothesis were true and we know what the population distribution is (presumed normal-Gaussian). If so, then it is true.

a) The distribution of test statistic under the null hypothesis is the same as its distribution across experiments using random samples of the same size from the same population.

What do they mean by the same population? It seems to me that it is the same as in d). But then you have two true answers, which is wrong. So I suppose it means the test samples are from the same population but this is not the same population you would get assuming the null hypothesis. That's shaky. I think I would answer d and cover myself by explaining my interpretation of the question.

 

[WWGD]

From what I gather, the test statistic is designed specifically to test the Null. I doubt the same statistic could test the alternative; $H_{A}$

Edit: Maybe @StatGuy2000 or @statdad can chime in.

 

[songoku]

I am sorry for late reply.

In this case, you would be testing; using the test statistic for the _difference of means_ $T_1, T_2$, not the mean itself. By default, definition, the Null Hypothesis $H_0$ assumes " No effect", so that $T_1=T_2$ *, thus, in this case, that the training didnt improve running time. The alternative could be 1-sided in 2 ways : $T_1>T_2$, $T_1<T_2$, or the more general $T_1 \neq T_2$.
Please let me know if I misunderstood what you wrote.

Yes, sorry, that's what I mean.

If the null hypothesis is that the random variable is from a $N(0,1)$ distribution and the alternative hypothesis is that the random variable is from a $N(100,1)$ distribution, than those two distributions are not the same.

Oh I see. When considering distribution, we also need to consider the parameters.

Thank you very much for all the help and explanation FactChecker, WWGD, Hornbein