ENH Added evaluation

[merveenoyan]

This PR is actually a fragment of #26 as #26 is a bit overwhelming. I changed the API and put evaluation a part of the Card class not to bother user with extra attributes.
@BenjaminBossan @adrinjalali

[BenjaminBossan]

@merveenoyan Did you ping us because we should already take a look or just to let us know?

[merveenoyan]

@BenjaminBossan I pinged you to mainly look at UX of the evaluation and let me know if the direction is right.

[BenjaminBossan]

Hmm, not so much I can say about this yet. It is very specific to HF, right? E.g. IIUC we assume that the metric used by sklearn is the same as defined on HF/metrics.

[merveenoyan]

@BenjaminBossan having metrics in metadata is essentially for HF Hub, it also passes them to paperswithcode to associated task on associated dataset. (let's say you have text-classification, you'd like to evaluate it on GLUE) this is not the case usually for tabular models imo as people split their data on their own. I don't expect heavy usage for this for when you put your model for outside world, if someone else wants to benchmark their own models and programmatically access the best metrics, it would be good IMHO.

[adrinjalali]

Overall I'd say it's in the right direction. I would change it a bit so that the method accepts the results of a scorer instead of trying to get the name and call it itself. It can be simplified by accepting a float value and a metric name. We probably also want to have it more like:

data description: ...
   metric 1: ...
   metric 2: ...

instead of a list of (data description, metric specs) as is right now.

But it seems like the eval results can't support that?

[merveenoyan]

@adrinjalali with #71 I think we can let the user add EvalResults instead, no?

[adrinjalali]

Yeah that would also make sense. I'm impartial on whether to construct EvalResult inside a method or accept one from the user. Note that #71 doesn't really make user ever create a CardData object themselves, the utility function does that.

[BenjaminBossan]

I would tend towards not exposing modelcards internals to the user.

[merveenoyan]

BTW evaluation metrics can be seen in this repository:
https://huggingface.co/merve/hf_hub_example-d4b188ba-be75-44ed-9c2b-d2cd0640c233

[adrinjalali]

should we add to a table or set the table? The current code doesn't allow the user to call this method multiple times.

[adrinjalali]

Thanks @merveenoyan

[BenjaminBossan]

This is almost good to go from my point of view. I have a suggestion for better formatting of one of the examples, which you may or may not agree with. But I think the test needs to be changed, as I think it doesn't work as intended.

[merveenoyan]

@adrinjalali I addressed your comments.

[adrinjalali]

Otherwise LGTM.

[merveenoyan]

@BenjaminBossan can you review?

[BenjaminBossan]

Perfect, thanks