Observational Studies: What to Believe

[gleem]

The behavioral sciences use statistical methods to link behavior to specific outcomes. Medical science also tries to find relationships between various behaviors and their health effects. Such studies use statistical methods unfamiliar to most physical scientists.

Regularly, studies with the same intent produce conflicting results causing confusion and perhaps distrust in otherwise valid research. Several years ago a European group wondered about the analysis of these studies i.e., the analytic approach of the various investigators. These approaches depend on assumptions like what covariants to include and the statistical method to apply. In general, the approaches to analysis are usually defensible. So why the wide variation in results? To investigate this issue they used studies meant to determine if there was a bias toward dark-skinned soccer players for being given red cards and used a variety of methods to analyze the data and compare the results.

The result of the study: “We recommend use of a specification curve (Simonsohn, Simmons, & Nelson, 2016) or multiverse analysis* (Steegen et al., 2016). With these approaches, the analyst in effect tries to come up with every different defensible analysis he or she can, runs them all, and then computes the likelihood that the number of observed significant results would be seen if there really is no effect (Simonsohn et al., 2016).”

*specification curve analysis—a novel analytic method that involves defining and implementing all plausible and valid analytic approaches for addressing a research question—to nutritional epidemiology.


Conclusion:

“The observed results from analyzing a complex data set can be highly contingent on justifiable, but subjective, analytic decisions. Uncertainty in interpreting research results is therefore not just a function of statistical power or the use of questionable research practices; it is also a function of the many reasonable decisions that researchers must make in order to conduct the research. This does not mean that analyzing data and drawing research conclusions is a subjective enterprise with no connection to reality. It does mean that many subjective decisions are part of the research process and can affect the outcomes. The best defense against subjectivity in science is to expose it. Transparency in data, methods, and process gives the rest of the community opportunity to see the decisions, question them, offer alternatives, and test these alternatives in further research.”

Because the above study was not published in a medical journal a similar question was asked by a Canadian group on the effect of the consumption of red meat on one's lifespan using specification curve analysis. For decades we have been warned that overeating red meat is associated with a shorter lifespan.

They analyzed a particular data set in 1200 different ways to determine the Hazard Ratio ( HR, A measure of how often a particular event happens in one group compared to how often it happens in another group, over time.). They came up with similar conclusions that the choice of analysis gave widely varying results. They found, 36% had hazard ratios above 1.0 and 64% less than 1.0. 48 analyses were statistically significant to P ≤ 0.05. with 40 having an HR < 1.0 ( red meat will not decrease your life expectancy?). The data that they used was recognized as suboptimal and they made no claim about the conclusion that might be drawn from their results. They said: “Nevertheless, our primary objective is not to provide conclusive answers about the health effects of red meat but to demonstrate a proof-of-concept application of specification curve analysis to nutritional epidemiology.”

Their Conclusion:

In this study, we apply specification curve analysis—a novel analytic method that involves defining and implementing all plausible and valid analytic approaches for addressing a research question—to investigate the effect of red meat on all-cause mortality. We show variability in results across plausible analytic specifications. This research demonstrates how specification curve analysis can be effectively applied to nutritional epidemiology, providing a practical and innovative solution to analytic flexibility. This approach has the potential to improve the credibility of inferences from such epidemiologic studies.

Perhaps someday we can have confidence in the results of these types of studies.

 

[Dale]

I must admit that I don’t know anything about specification curve analysis. There are a lot of other statistical tools for addressing this issue, some with much more recognition. I think the issues are more challenging than just a novel statistical method can address

 

[gleem]

There are a lot of other statistical tools for addressing this issue, some with much more recognition.

As I noted above physical scientists are usually unfamiliar with methods of the behavioral sciences, particularly with observational studies. I being no exception. Do you have any thoughts as to why the authors used this approach when methods "with more recognition" are available? Also, I might have expected some reference to alternative methods to further clarify/justify their approach since it is labor-intensive.

 

[Dale]

Do you have any thoughts as to why the authors used this approach when methods "with more recognition" are available?

I don’t know specification curve analysis, so there may be something that it does uniquely that I don’t see in my ignorance. But generally these sorts of “look at what we can do with this new tool” studies are published for every statistical method.

I might have expected some reference to alternative methods to further clarify/justify their approach since it is labor-intensive

That is always a good thing to do. No method is developed in a vacuum, unrelated to other alternatives. It is always good to identify the alternatives and place the new method in context.