How are critical values in statistical tests obtained?

[Mayhem]

In science, statistics are constantly used to give 'rigorous' interpretations of data sets. In this process, tests are often employed to verify a property that is being investigated. For example, normal distribution or randomness. Usually an algorithm is employed on the data set and a test statistic is obtained and compared to a critical value. This critical value is dependent on the degrees of freedom of the dat set and confidence level used, but are almost always "strange" value that must be looked up in a table. How our test statistic compares to this value can lead to very different conclusions.

Where do these critical values come from and how can we know that they can be trusted?

 

[FactChecker]

There is not a hard and fast statistical proof. There is only "confidence" at a certain level. You need to start with a confidence level. Many subjects traditionally use 95% or 97.2% confidence. But some subjects, like particle physics insist on "five sigma" level, which is about 99.9999% confidence! From that point on, the translation of that, using the statistically correct degrees of freedom, leads to a direct way to interpret your parameter of interest. The final result, as a limit on that parameter, often looks like a "strange" value. But it came from the initial confidence level.

 

[Mayhem]

There is not a hard and fast statistical proof. There is only "confidence" at a certain level. You need to start with a confidence level. Many subjects traditionally use 95% or 97.2% confidence. But some subjects, like particle physics insist on "five sigma" level, which is about 99.9999% confidence! From that point on, the translation of that, using the statistically correct degrees of freedom, leads to a direct way to interpret your parameter of interest. The final result, as a limit on that parameter, often looks like a "strange" value. But it came from the initial confidence level.

I understand that it comes from a confidence level, but how are these critical values actually produced?

 

[Dale]

I understand that it comes from a confidence level, but how are these critical values actually produced?

They are the inverse of the cumulative density function for the distribution in question.

 

[FactChecker]

They are the inverse of the cumulative density function for the distribution in question.

Furthermore, each CDF may require different numerical methods and some use special functions that were motivated by that CDF. Often, the user relies on tables are generated and published.

 

[Hornbein]

I understand that it comes from a confidence level, but how are these critical values actually produced?

You've got some data. You assume that these data are random and from a certain distribution. This is applied mathematics so close is good enough (or better than nothing, depending). Again assuming that distribution, one may look up in a table the critical value for the confidence level you have chosen. The values in the table for that distribution were usually computed numerically i.e. with a computer.

 

[Mayhem]

They are the inverse of the cumulative density function for the distribution in question.

This sounds interesting. Is it something that can be explained to a chemist who is more on the practical side of statistics?

 

[Dale]

This sounds interesting. Is it something that can be explained to a chemist who is more on the practical side of statistics?

Are you familiar with what a probability density function is?

The cumulative density function is just the integral of the probability density function. Once you calculate that integral, then you just use it to fill a table. Inverting the cumulative density function is then just a matter of looking up a value in the table backwards.