Statistical Errors, Type I and Type II

[FactChecker]

Is there any difference in assuming something is or isn't safe? If you compare it to a safe population the process is the same. in one case you look for evidence that it can be a member of the safe population and there is assumed to be safe and in the other you look for evidence that it is not a member of the safe population and therefore is assumed to be unsafe.

Good question. There is a big difference because you are giving the null hypothesis every advantage. You start by picking one hypothesis as the null hypothesis, giving it all the benefit of the doubt by using its distribution and parameters and saying that you will only change that assumption if there is strong test indications (over 95%, 99%, etc.) that it might be wrong.
In the case of testing the safety and effectiveness of a drug, they should assume that it is not safe or not effective and run tests that would convince even a skeptical audience that it is safe and effective. The burden of proof must be on the drug company to prove its drug is probably (95%, 99%, etc.) safe and effective. Otherwise, many unsafe and/or ineffective drugs would pass a minimal test and be approved for public use.

 

[gleem]

We are talking about a procedure, I don't think, that is used by drug companies that the endpoint of toxicity studies is death or estimates of death but instead some noticeable change in a physiological characteristic that could be detrimental if excessive like anemias, constipation, vomiting, reduced liver or kidney function. Typically they will start at a dose believed to have no untoward effects and increase the dose until side effects occur. They decide on what they think are acceptable side effects at an effective dose. If they do a comparison between equivalent populations of those who take the drug and those who don't at a reasonable confidence level and see no difference then what? There are side effects some serious but there is a benefit from taking the drug, Sometimes the risk can result in death. The patient must make a choice under guidance from their physician.

 

[FactChecker]

Good point. The actual decision process is complicated. My point is that, in the simplest terms, the drug company has the burden of proof to convince a skeptical audience that their drug has a net benefit. They can only do that if the original assumption, the null hypotheses, is that the drug is not beneficial and then show that the data results are strong enough to convince the skeptics otherwise.

On the other hand, if they start by assuming that the drug is beneficial and then set a very high standard (95%, 99%, etc.) to statistically indicate otherwise, they will not convince anyone.

 

[Agent Smith]

Would I be correct to say that $H_0$ = The drug is not beneficial?

 

[FactChecker]

Would I be correct to say that $H_0$ = The drug is not beneficial?

Yes. The null hypothesis, $H_0$, is the option that you are willing to assume if there is none or inadequate data to prove the alternative hypothesis, $H_1$, because that is what you will statistically recommend in those cases. There is no level of confidence required for the null hypothesis.

In the case of a drug, if it is presented with no testing you would not recommend its use for the general public with no further testing. Instead, you would say that the drug company has the burden of proof and must conduct enough testing to indicate at a level of 95% (or higher) that it is safe and effective. That is, they must statistically support the alternative hypothesis, $H_1$= "the drug is safe", at some level of confidence.

 

[Agent Smith]

Good point. The actual decision process is complicated. My point is that, in the simplest terms, the drug company has the burden of proof to convince a skeptical audience that their drug has a net benefit. They can only do that if the original assumption, the null hypotheses, is that the drug is not beneficial and then show that the data results are strong enough to convince the skeptics otherwise.

On the other hand, if they start by assuming that the drug is beneficial and then set a very high standard (95%, 99%, etc.) to statistically indicate otherwise, they will not convince anyone.

The null hypothesis contains an equality statement for a given population parameter (high school level statistics)

 

[statdad]

So, a null hypothesis is neither true nor false.
We can either reject it or fail to reject it

What exactly is the terminology we use here.

We're proffering $2$ hypotheses: The Null $H_0$ and The Alternative $H_a$.

Then we compute the P-value. If $\text{P-value} \leq \alpha$, we reject $H_0$ and accept $H_a$. If $\text{P-value} > \alpha$ we fail to reject $H_0$

Perhaps I should've said
Type I Error: Rejecting $H_0$ when $H_0$ shouldn't be rejected. $\text{P-value} \geq \alpha$
Type II Error: Failing to reject $H_0$ when $H_0$ should be rejected. $\text{P-value} < \alpha$

Imagine you want to test H0: mu = 98.6F versus Ha: mu < 98.6F

where the quantity we're concerned with is body temp of a "normal healthy" adult.

Concerning your "neither true nor false" question, think about it this way: are we trying to say that the mean temperature is EXACTLY 98.6 degrees F? Certainly not -- we're saying that the true mean temp is close enough to that value to make it a very usable description, so is H0 true? Not with that interpretation. Is H0 false? Not in the sense that we want to know if 98.6 is a good usable reference value. The purpose of this hypothesis test is this: determining whether the true mean temp is close enough to 98.6 that we can continue to use it or whether it is enough smaller than 98.6 that we need to move on to a new value. Remember that hypothesis testing is ALWAYS about examining the evidence against H0, not the evidence it its favor.

The possible decisions of the test are:
- Reject H0: this means the data indicates to us that the true mean is noticeably smaller than 98.6. We might have a sample mean of 98.48, but after sample size and sample standard deviation are taken into account we decide that is not far enough from 98.6 to convince us Ha makes more sense than H0
- Do not reject H0 -- here the data indicates that the true mean is smaller than 98.6

Why not say "Accept H0" in the first case? Because the word "Accept" indicates we've proven H0 to be true.

A final comment: imagine how the US justice system is supposed to work -- the philosophy of it. Please don't comment on views of how people believe it actually works: I won't respond and it isn't appropriate.

To avoid language awkwardness I'll assume the verdict comes from a judge instead of a jury.

There are two possibilities about the defendant: The defendant is Guilty: DG, or the defendant is innocent: DI
One tenet of the US judicial system is to assume DNG. The goal of the prosecutor is to convince the judge, that DG is the correct assumption: in other words, the prosecutor has to convince the judge to reject the assumption of DNG

There are two possible verdicts: Guilty, G, or Not Guilty, NG. (Note that there is no such thing as a verdict of "Innocent".) Again, as noted above, to obtain a verdict of G the prosecutor has to convince the judge, beyond a reasonable doubt, to drop the assumption that the defendant is guilty. The prosector's job is not to present evidence showing the defendant is innocent, it's to present evidence of the defendant's guilt

There are four possibilities for what can happen at the end of the trial:

The verdict is G and the defendant is Guilty: thus G and DG is a correct outcome
The verdict is G and the defendant is Innocent: thus G and DI is an incorrect output. Since this corresponds to the prosecutor convincing the judge to reject a basic assumption when that assumption should not be rejected this can be considered a Type I Error
The verdict is NG and the defendant is innocent: here NG and DI is a correct outcome
The verdict is NG and the defendant is guilty: here NG and DG can be considered a Type II error: the judge should have rejected the assumption about the defendant but did not

In hypothesis testing:
- the null hypothesis corresponds to DI
-the alternative hypothesis corresponds to DG
- rejecting H0 corresponds to a verdict of guilty
- failing to reject H0 corresponds to a verdict of not guilty

 

[FactChecker]

There are two possibilities about the defendant: The defendant is Guilty: DG, or the defendant is innocent: DI
One tenet of the US judicial system is to assume DNG. The goal of the prosecutor is to convince the judge, that DG is the correct assumption: in other words, the prosecutor has to convince the judge to reject the assumption of DNG

That's a good analogy. The common hypothesis testing is set up to require that the alternative hypothesis will only be accepted if it is proven "beyond a reasonable doubt" to some level (95%, 99%, 5 sigma, etc.)
So the first thing to ask is: What hypothesis really requires proof?
That should be the alternative hypothesis.

 

[Agent Smith]

Good point. The actual decision process is complicated. My point is that, in the simplest terms, the drug company has the burden of proof to convince a skeptical audience that their drug has a net benefit. They can only do that if the original assumption, the null hypotheses, is that the drug is not beneficial and then show that the data results are strong enough to convince the skeptics otherwise.

On the other hand, if they start by assuming that the drug is beneficial and then set a very high standard (95%, 99%, etc.) to statistically indicate otherwise, they will not convince anyone.

The null hypothesis contains an equality statement for a given population parameter.