Manage lifecycle of SLO

[victor-smg]

At my company we plan to use Sloth to generate SLOs with http_request_duration_seconds_count for all apps and handlers.

For this I made a cronjob that:

1. discover the handler in used, we querying prometheus:
   'query': 'count by (app, handler) (http_request_duration_seconds_count)'
2. generate the SLOs yaml according to specs
3. Use sloth tool to generate the rules
4. Use mimirtool to load the rules

At first glance this process works, but when a handler gets deleted we still keep it and it breaks dashboard.
Because we only add new rules, we keep rules where there is no metric anymore.

Would you recommend to add a logic in the cronjob to delete all SLOs rules where there is no metric (at risk of deleting SLOs if something goes bad in metric retrieval), or have manual clean-up process regularly?

Also does someone else generates its SLO in similar way?