Regarding Simulations and Sample Size

[Agent Smith]

TL;DR Summary: Sims and sample size

A statistics question I have in my notes goes like this:

Our significance level $\alpha = 0.01$
The percentage of left-handed people in the general population is $10\%$. Liliana is curious if this is true for her arts class and so she takes a random sample of $8$ [please note this number] students from her arts class and finds that $1$ is left-handed. That is the proportion of lefties in her class is $0.125$.

The null hypothesis: $H_0$ is that the proportion of lefties in Liliana's class = $10\%$
The alternative hypothesis: $H_a$ is that the proportion of lefties in Liliana's class $> 10\%$

She then conducts a 100 simulations, each time taking a sample size of $8$ [please note this number] from a virtual population in which $10\%$ are lefties. It turns out that in $2$ of her simulations the proportion of lefties is $\geq 0.125$. This means, I'm told, that the probability of getting a proportion of lefties $\geq 0.125$ is $\frac{2}{100} = 0.02$.

Then, the back-of-the-book answer says, since the $\text{P-value} = 0.02$ and $\alpha = 0.01$ and $0.02 > 0.01$, we can't reject $H_0$.

I hope all the above is correct.

My question concerns the sample size $8$ (the number I asked be noted). This sample size is too small for the number of successes and the number of failures to be $\geq 10$ i.e. one condition for inference from the sample is unmet and yet we have made an inference. Am I supposed to conclude that with simulations like the one described above we need not bother about sample size? So for this particular question, if my sample size is $6$, I need only ensure that the simulation consists of samples of size $6$ and I'll still be able to make legitimate inferences from the sim???

N.B. Also if we reset $\alpha = 0.05$, since $0.02 < 0.05$, we can reject $H_0$ and conclude that Liliana's arts class has an "unusually high number" of lefties, right?

 

[FactChecker]

we have made an inference.

No, We have just made an assumption (the null hypothesis) and have not conducted enough testing to reject the assumption at that level of confidence.

Am I supposed to conclude that with simulations like the one described above we need not bother about sample size? So for this particular question, if my sample size is $6$, I need only ensure that the simulation consists of samples of size $6$ and I'll still be able to make legitimate inferences from the sim???

You can make assumptions (hypothesis) for the purpose of a statistical test with no data at all, but proving that assumption should be rejected at some confidence level requires enough sample data with contrary results.

N.B. Also if we reset $\alpha = 0.05$, since $0.02 < 0.05$, we can reject $H_0$ and conclude that Liliana's arts class has an "unusually high number" of lefties, right?

With much less confidence. The chance of a Type I error is increased at the 0.05 level.

 

[Dale]

It turns out that in 2 of her simulations the proportion of lefties is ≥0.125.

That is interesting. I just did this same thing, and I got 58 out of 100 with a proportion of at least 0.125. That seems to indicate a coding error in the simulation.

This sample size is too small for the number of successes and the number of failures to be ≥10

This may be a good recommendation for simulations as well. If they had only two successes with 100 simulations they probably should have tried 1000 or 10000 instead.

i.e. one condition for inference from the sample is unmet and yet we have made an inference

Yes, you have made an unreliable inference. That is exactly what those rules are intended to prevent.

 

[Agent Smith]

Why would they post a wrong question?

That's not the only question with a sample that fails the successes $\geq 10$ and failures $\geq 10$ test. Actually there's only $1$ other question, but this time we're checking for mean. @Dale , do you know if there's an interactive website with a Monte Carlo simulation?

I believe the point to note is this is a simulation of a sampling distribution of sample proportions. Could it be that the condition, the distribution of the sample is uniform (no outliers, etc.?). It feels right to me to conduct a simulation in this way, with the simulation sample size = whatever the sample size was initially.

 

[Agent Smith]

These are the sampling distribution of the sample proportions. Would you agree that the distribution is roughly normal and SO we don't have to concern ourselves with sample size being inadequate to satisfy the succeses $\geq 10$ and failures $\geq 10$ condition for inference.

 

[Dale]

View attachment 351854

These are the sampling distribution of the sample proportions. Would you agree that the distribution is roughly normal and SO we don't have to concern ourselves with sample size being inadequate to satisfy the succeses $\geq 10$ and failures $\geq 10$ condition for inference.

This is not the distribution of proportions for a sample size of 8. For a sample size of 8 the possible proportions are only integer multiples of 0.125, and between 0 and 1.

It feels right to me to conduct a simulation in this way, with the simulation sample size = whatever the sample size was initially

I agree. Unless the simulation were intended to explore “what if” for different experimental approaches.

 

[Agent Smith]

This is not the distribution of proportions for a sample size of 8. For a sample size of 8 the possible proportions are only integer multiples of 0.125, and between 0 and 1.

I agree. Unless the simulation were intended to explore “what if” for different experimental approaches.

Apologies. Got lost in the woods. However, these are from similar questions, where the sample size is too small for the conditions for inference to be satisfied.

Would you agree that if the distribution is normal, we can ignore the small sample size?

 

[Dale]

Would you agree that if the distribution is normal, we can ignore the small sample size?

No. The inferences will be unreliable. Even if it looks more or less normal the mean and standard deviation will not be reliable.

By the way, in my simulation I did not make the normal approximation. I used a binomial distribution. So if their simulation used the normal approximation that could be an issue.

 

[Dale]

@Agent Smith I know you probably get tired of my constant references to Bayesian methods since it probably is not super helpful for your class. But I think that the principles are more intuitive.

So, in a Bayesian analysis, we can explicitly deal with the probability of a hypothesis, not just probabilities of data given a hypothesis. In this type of experiment, where we are discovering the relative frequency of some trait in a population, the Bayesian analysis is particularly easy because we can use the Beta distribution for the probability distribution of the frequency.

The way we go is that we start with some prior probability, this reflects our beliefs about the subject before looking at the new data. Then we observe the data. And given the data we update our beliefs to obtain the posterior probability.

So in this case, if we start with complete ignorance about handedness, then that would be a $\beta(1,1)$ distribution. We observe 1 left-handed person and 7 right-handed persons. This updates our posterior to a $\beta(1+1,1+7)=\beta(2,8)$ distribution.

With this kind of information we can calculate probabilities about any hypothesis we would like to form about the data.

For example, suppose that we want to calculate the probability that the frequency of left-handers is between 5 % and 15 %. Under the prior $P(0.05<X<0.15)=0.1$ while under the posterior $P(0.05<X<0.15|data)=0.33$. So even with that small amount of data we have learned quite a bit.

Or we can calculate hypotheses that have a certain probability. For example we can determine $x$ such that $P(X<x)=0.95$ as a metric that "we would be surprised if ...". So under the prior $P(X<x)=0.95$ gives us $x=0.95$, meaning we would be surprised to learn that 95% of the population are left handed. Having seen the data we instead learn $P(X<x|data)=0.95$ gives us $x=0.43$. So with this data we would be surprised if there are more than 43 % of the population that is left handed.

We are not limited to ignorant priors. We could say, for example, that we would be surprised if there were more than 27 % of the population that is left handed, and construct an appropriate prior, like $\beta(3,19)$. In this case, observing our new data makes a much smaller change in our posterior.

Our posterior is a little more concentrated, but pretty close to the prior. Our small amount of data would teach a completely ignorant person a lot, but doesn't really teach someone much if they have even a small amount of pre-existing knowledge.

The $P(0.05<X<0.15|data)$ now only changes from 0.54 to 0.60, and the $P(X<x|data)=0.95$ surprise hypothesis only goes from 0.27 to 0.25.

With this approach you get a pretty good sense on how limited such a small data set is.

 

[Agent Smith]

@Dale I'm glad that you mentioned Bayesian statistics. It's basically this $\text{P(Hypothesis|Evidence})$ isn't it? Here Evidence = Data (we get from the sample taken from the population). I don't understand the distinction Frequentists vs Bayesians though. Isn't probably ultimately about frequency, explaining the close link between probability and statistics? Can you explain a bit more please or you could, if it's not too much to ask, post a link or something?

 

[Dale]

I'm glad that you mentioned Bayesian statistics. It's basically this P(Hypothesis|Evidence) isn't it? Here Evidence = Data (we get from the sample taken from the population).

Yes, that is correct.

Can you explain a bit more please or you could, if it's not too much to ask, post a link or something?

Sure. I posted a series of articles here about Bayesian statistics. Here is the second in the series which discusses the main differences between Bayesian statistics and the standard frequentist statistics.

https://www.physicsforums.com/insights/frequentist-probability-vs-bayesian-probability/

There really are a couple of main differences. The first is the meaning of probability. Frequentists use probability to mean the long-term frequency of something, while Bayesians use it to describe our uncertainty about something. So a frequentist might say that the probability of a person in a given population having height greater than 5'10" is 0.50, meaning that if you hypothetically sampled 2 billion or so then you would find 1 billion were greater than 5'10" and 1 billion were not. A Bayesian would say that the probability of that population having a height greater than 5'10" is 0.50, meaning that we are maximally uncertain about it and would not be surprised to find any given person over that threshold (or under).

For things like the height, that really isn't a difference, but what about things like the mean of the population height? For a frequentist there is no long-term frequency involved, there is only one population, so the population's mean height isn't a random variable. For a Bayesian, however, the fact that we are uncertain about the mean height implies that the mean height can be treated as a random variable. Probability statements about this random variable express our uncertainty about the mean height, not frequencies in a hypothetical population of populations.

Because Bayesians use probability to represent uncertainty, we can use probabilities to describe hypotheses, something that is not done in frequentist statistics. To me this is good because when I think about science I am explicitly interested in the probability of my hypothesis. So to me it is a much more natural way to think about science.

All it requires is recognizing that probability can describe uncertainty, not just long run frequencies.