ENH Add performance test + GH action to run it weekly #289
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #275
Ready for review @skops-dev/maintainers
Description
This PR adds a GitHub action that is run on a weekly basis on Sunday. For all
tested sklearn estimators, it persists them once with pickle and once with skops,
and then records the timing difference. If the absolute difference is too large,
an error is raised and we can take a look at what went wrong.
For convenience, that 10 slowest estimators are printed to stdout. That way, we
can find out problematic candidates by looking at the log.
The persistence of an estimator is considered to be too slow when it takes more
than 1 sec longer to persist with skops than with pickle.
Comments
If someone has a good idea how to test that the GH action is specified correctly
without actually merging it, let me know. For reference, this action is based on
https://github.com/skorch-dev/skorch/pull/904/files, which works.
On top of running the action, we could also add a badge to the README. This
would show at a glance if the last run failed, but it's not strictly necessary.
Results
Running the script locally, I found these results:
So the slowest absolute difference is 0.24 sec, which I think is more than
acceptable. Relative differences can be quite large (100x), which is maybe not
too surprising, given that pickle is highly optimized and skops performs some
extra checks.
According to the criterion described above, none of the estimators is
unacceptably slow, so the action would pass successfully.