Evaluate forecasts with contingency table

[algkott]

Hi! This is my first post...


I’m trying to evaluate directional forecast results in a contingency table. I´m not quite sure exactly what kind of hypothesis I should set up to decide whether the forecaster is better than just a random guess or better than another forecaster, or how I should interpret my results.
In my contingency table I have three directions to consider, Up, Down or Neutral. The result can be right or wrong.
I know I can use the Pearson Chi-squared or the Fisher exact test. For instance I calculate my Chi-square to 2,837 which gives me a P-value of 0,242 with 2 degrees of freedom. This means I reject the hypothesis that the variables are independent?
Can someone please explain in words and/or equations how I should formulate this problem or how to interpret the results when I’m calculating X^2s and P-values?

 

[mmwave]

Hello there! Welcome to the forum and congratulations on your first post. It sounds like you are working on evaluating directional forecast results using a contingency table. This is a common method used in statistical analysis to compare the performance of different forecasters or to determine if a forecaster is better than a random guess.

To set up a hypothesis for your analysis, you could state: "The forecaster's directional predictions are better than a random guess." This is called the null hypothesis. The alternative hypothesis would be: "The forecaster's directional predictions are not better than a random guess."

To interpret your results, the Chi-squared test compares the observed frequencies in your contingency table to the expected frequencies under the assumption of independence between the variables. In this case, the variables are the forecasted direction and the actual direction. A low P-value (usually less than 0.05) indicates that the observed frequencies are significantly different from the expected frequencies, and thus, the null hypothesis can be rejected. In your case, with a P-value of 0.242, we cannot reject the null hypothesis and conclude that the forecaster's predictions are not significantly better than a random guess.

If you are comparing the performance of two forecasters, you could set up a different hypothesis stating that one forecaster's predictions are better than the other. In this case, the Chi-squared test would compare the observed frequencies for each forecaster and determine if there is a significant difference between them.

I hope this helps to clarify the use of the Chi-squared test in your analysis. Good luck with your research!