Statistical significance of a ML model...

[fog37]

TL;DR Summary

Determining if a ML model is statistically significant...

Hello,

How do we check if a ML model is statistically significant? For models like linear regression, logistic regression, etc. there are tests (t-tests, F-tests, etc.) that will tell us if the model, trained on some dataset, is statistically significant or not.

But in the case of ML models, like decision trees, SVM, or neural nets, how do we determine if the model is statistically significant? I have not seen any specific test to do that...

Thank you!

 

[Vanadium 50]

There is a whole subfield on this called UQ - uncertainty quantification. It is an area or active development.

 

[willem2]

TL;DR Summary: Determining if a ML model is statistically significant...

But in the case of ML models, like decision trees, SVM, or neural nets, how do we determine if the model is statistically significant? I have not seen any specific test to do that...

The t test will work with any predictive model. You're supposed to set aside a part of the input data, and not use it in your model and use it for testing later. (Because predicting your input data with a ML model is cheating). For a yes/no model, you can score a 1 for correct, and 0 for wrong, and you can compare it other ways to predict the outcomes (or random guessing),