How to compare 2 means?

[Agent Smith]

TL;DR Summary

Hypothesis testing by comparing 2 means.

We have a question where we're testing the hypothesis that a certain diet (call it diet A) causes weight loss. We get $n_1$ people who we put on diet A (treatment group) and another $n_2$ people who we keep on a normal diet (control group). We find that the mean weight loss in the treatment group is $m_T$ with a standard deviation $s_T$. The mean weight loss in the control group is $m_C$ and standard deviation is $s_C$.

$H_0$: Diet A is not effective i.e. $m_T = m_C$
$H_1$: Diet A is effective i.e. $m_T > m_C$

Assume all conditions for inference have been met

We "combine the distributions" (I don't know the appropriate word) and compute $m_T - m_C$. This is the difference in the means of weight loss for the treatment and control groups. Correct?

We compute the standard deviation for $m_T - m_C$ like so: $\sigma_{T - C} = \sqrt{\frac{s_T^2}{n_1} + \frac{s_C ^2}{n_2}}$. This is the standard deviaton of the sampling distribution of the difference in mean weight loss for the treatment and control groups. Correct?


We compute the z/t score $z = \frac{\left(m_T - m_C\right) - 0}{\sigma_{T - C}}$

We then look up the p-value from a z/t table.
If the p-value $\leq$ alpha then we reject $H_0$ and accept $H_1$ and if the p-value > alpha, we fail to reject $H_0$

 

[Dale]

In general, if $p<\alpha$ then you reject $H_0$. Full stop. Rejecting $H_0$ is not the same as accepting $H_1$. That is especially the case when $H_0$ and $H_1$ are not mutually exclusive and collectively exhaustive as is the case here.

In this case it is possible that the diet is anti-effective, meaning $m_T<m_C$ (let's call this $H_2$). Now, you may look at your data and see that your sample $\bar m_T>\bar m_C$, and you might incorrectly reason that your sample $\bar m_T>\bar m_C$ combined with $p<\alpha$ should entitle you to not just reject $H_0$ but also to accept $H_1$. But the problem is that you have not accounted for the possibility that $H_2$ is correct ($m_T<m_C$)) but a sample randomly has $\bar m_T>\bar m_C$.

To justify that you would actually have to calculate the probabilities of the data given $H_1$ and $H_2$

 

[Agent Smith]

@Dale If $m_T > m_C$, I don't think we have to worry about $m_T < m_C$, no? I'm not sure.

Also what about my questions (in bold)? Are my conclusions correct?

 

[Dale]

@Dale If $m_T > m_C$, I don't think we have to worry about $m_T < m_C$, no? I'm not sure.

Even if the population $m_T>m_C$ it is possible that the sample $\bar m_T<\bar m_C$. Since all you observe is the sample, you cannot rule the other possibility out.

TL;DR Summary: Hypothesis testing by comparing 2 means.

This is the standard deviaton of the sampling distribution of the difference in mean weight loss for the treatment and control groups. Correct?

I believe so, but would have to look it up to be sure. There may be a correction for $N-1$ somewhere.