Clarification for the "calib_class_thresh" documentation and its use

[jmburgos]

Hello,

First of all, thank you for the tidysdm package. I am having a lot of fun using it.

Here is the issue. The description of the calib_class_thres() function states that the function

"predicts for a new dataset by using a simple ensemble. Predictions from individual models are combined according to 'fun'"

But what the function does is to update the simple ensemble with the thresholds. The function does not predicts into a new dataset. It would be good to fix this.

The resulting ensemble has a new attribute, with four values for the threshold. For example,

> attr(updated_ensemble, "class_thresholds")
# A tibble: 4 × 4
  class_thresh metric_thresh fun             optim_value
  <list>       <list>        <chr>                 <dbl>
1 <chr [1]>    <chr [2]>     mean                  0.574
2 <chr [1]>    <chr [2]>     median                0.593
3 <chr [1]>    <chr [2]>     weighted_mean         0.574
4 <chr [1]>    <chr [2]>     weighted_median       0.555

It would be useful to clarify (either in the description of the function, or in the vignette)

a) How these values are calculated (I presume these are the mean/median values of the thresholds from the models in the ensemble right? and where the weights come from?).

b) How these thresholds are used when calling the predict.simple_ensemble() function? Does the class_thresh argument needs to match the thresholds calculated with calib_class_thres()? Also, according to the documentation the class_thresh argument can be only "tss_max" or sensitivity. What happens if I computed the Kappa max threshold with calib_class_thres()?

c) Which of the four calculated threshold are used? I presume that the one matching the "fun" attribute, right?

Also, it would be handy to have a collect_ function to extract the threshold values, something like this:

collect_thresholds.simple_ensemble <- function(x, ...) {
    attr(x,"class_thresholds") |>
    mutate(class_thresh = unlist(class_thresh),
           metric_thresh = unlist(metric_thresh))
}

Thanks for your reply!

[dramanica]

@jmburgos I am currently on leave, so it will probably be two weeks before I can respond to this properly. Apologies, bad timing.

[jmburgos]

No problem :)