[Feature Request] Equal number of samples for each target variable in Machine Learning Classification

[coopa33]

Hi,
I'm volunteering for the JASP team, and while looking at the Machine Learning modules, I noted two things that I think are worth discussing:

1. Currently, machine learning classification analyses do not include (as far as I can see) any functionality for balancing the dataset based on the ratio of target labels/classes. I think this could be a useful feature to add, as the performance of the model on the underrepresented class could be worse than the over-represented class (f.e. if a relatively simple model is used, and if the features are noisy). If the validation data is also biased (which I think it certainly is if it is sampled from a biased dataset randomly), then the average evaluation metrics will not show this bias, and could be misleading if one does not look at the class performance metrics. Adding a feature to balance the distribution of labels in both training and validation data could be educational: One could see how the class prediction accuracy changes while the average accuracy maybe does not change that much.

2. In the most extreme case of this bias (let's say out of label A and B, label B got 0 correct predictions), the table produced by Model Performance will give the same accuracy for both label A and B. The accuracy for label B in this case should be 0, as precision = 0 / (0 + FN).
   Please see the attached image

[koenderks]

1. Certainly an interesting feature to implement!

2. Isn't that the recall what you calculated as 0? Maybe we should remove accuracy per class from this table, aa it considers all predictions, not just those per class? Isn't accuracy per class the same as the recall?

[coopa33]

About 2)
Oof, sorry, indeed what I said was bogus. I agree that accuracy maybe should be at least edited to only display for Average/Total, and not for the individual classes. As you say, accuracy per class doesn't make sense

[koenderks]

I wonder what the best way to do this in the table is, having a row with only one value for (total) accuracy while the rest of the columns is empty also seems a bit ugly to me. Any idea's @coopa33 @vandenman?

[coopa33]

The model accuracy seems to already be displayed in the Model Summary table, which is always open when the analysis is run. This seems to be the case as well for other machine learning analyses. We could then maybe remove accuracy from the Performance metrics entirely?