List of current used/maintained Sloth fork features

[slok]

This is a centralized list of features that have been implemented (and continue being used/maintained) in various Sloth forks during the two years of inactivity in the upstream project.

The goal is to gather them all in one place, review which ones make sense to include in Sloth, and take action accordingly.

I started the list with the ones that I'm aware, but feel free to comment here others so I can add them to the list. Thanks! 🙇

• 🦥 https://github.com/linode-obs/sloth (@cxdy, @wbollock)
  • Custom info labels: feat: add custom labels to sloth_slo_info metric linode-obs/sloth#4
  • Rule group interval: feat: adjustable rule_group interval linode-obs/sloth#1 (feat: adjustable rule_group interval #515)
• 🦥 https://github.com/kolonialno/sloth (@gbbirkisson)
  • Denominator corrected: https://github.com/kolonialno/sloth/pull/7/files
  • Fix label value changes: Fix recording rule errors the label values change kolonialno/sloth#6
• 🦥 https://github.com/ostrovok-tech/sloth (@nixargh, @aka-switch)
  • Victoria metrics support

[nixargh]

Hi!
Glad to see you're back in business :)
It would be nice to have all these changes bundled together. Maybe my VictoriaMetrics validation is too straightforward to take it as is, but if I may help somehow, please feel free to ping me.

[slok]

Hey! 👋 Thanks!

Yes, I’ll make an announcement for contributors once Sloth is ready. Right now, I’m already working on a refactor to make Sloth more pluggable and easier to contribute to. The idea is to eventually delegate parts of the support and maintenance to others who want to get involved in the project 😄.

[slok]

@nixargh / @aka-switch Hi there! 👋 Could you share an example of a working victoria-metrics SLO spec that fails with Sloth upstream please? thanks! 🙇

[nixargh]

@slok hi!
The need came from the fact that some development teams use VictoriaMetrics go library instead of Prometheus one and by default the VM lib generates histograms with fluid buckets set. So we were looking for a way to use them and came to the solution below, which uses VM function histogram_share.

Here is an example of Sloth config:

version: prometheus/v1
service: foo-bar
labels:
  owner: foo
  repo: content/foo-bar
  generated: true
  type: latency
  application: slowpoke
slos:
- name: grpc-latency-percentile-90
  objective: 90
  description: '"grpc" 90 percentile Latency SLO for grade "A"'
  labels:
    actual_grade: A
    target_grade: A
    actual_le: 0.2
    target_le: 0.2
  sli:
    events:
      error_query: 1 - histogram_share(0.2, sum by (vmrange) (rate(requests_duration_seconds_bucket{container_name="foo-bar-grpc", agent_dc="dc_sl"}[{{.window}}])))
      total_query: vector(1)[{{.window}}]
  alerting:
    name: slo_foo-bar_grpc-latency-percentile-90_fail
    labels:
      objective: 90
      objective_reversed: 10
- name: grpc-latency-percentile-99
  objective: 99
  description: '"grpc" 99 percentile Latency SLO for grade "A"'
  labels:
    actual_grade: A
    target_grade: A
    actual_le: 0.4
    target_le: 0.4
  sli:
    events:
      error_query: 1 - histogram_share(0.4, sum by (vmrange) (rate(requests_duration_seconds_bucket{container_name="foo-bar-grpc", agent_dc="dc_sl"}[{{.window}}])))
      total_query: vector(1)[{{.window}}]
  alerting:
    name: slo_foo-bar_grpc-latency-percentile-99_fail
    labels:
      objective: 99
      objective_reversed: 1

[aka-switch]

Thanks. I actually don't have any ready-to-go interpretation of Victoria Metrics in SLOth on my side or use what @nixargh has done. I just planned to do it. So, unfortunately, I have no code to offer as a PR for now. But let me know if I can help anyway.

[slok]

Awesome! thanks @nixargh!

NP @aka-switch, With @nixargh's example should be more than enough for the tests :)

Thanks both 👌

[cxdy]

For Linode's usecase, both of our features are necessary for us to be able to switch back to upstream and personally I think they're nice to have even if they go unused.

Regarding dynamic rule intervals, I have a PR open to extend that to the Kubernetes spec but I just need to add tests. linode-obs#24

Happy to get these PRs in for upstream too. I think @wbollock might have a couple PRs in the upstream repo too.

[ishantanu]

Yes, I suppose API calls can be done using these packages. We want to minimise manual user inputs for fields like owner, team, etc. and also add more metadata at runtime from CMDB to have consistency in SLO reporting.

[foax]

@slok as you suggested in #602 I tried the info_labels_v1 SLO plugin and that seems to suit our use case well 👍

In #560 (reply in thread) you mention that the new SLO plugin functionality does not yet support CRDs. Are you planning on adding support for this, or is there some sort of limitation that makes it harder to implement SLO plugins for Sloth in kubernetes-controller mode?

[slok]

👋 Hey @foax, happy that works for you!

Oh, at that time sloth didn't have the k8s support because I was in the middle of adding and testing SLO plugins, but during this time, SLO plugin support for k8s CRs was added. Take a look to this example.

[foax]

Ah nice, I was using the ghcr.io/slok/sloth:3c0615e6df0bb55319cf4d9166cace93ee8b5057 container image you mentioned above but I'll try it out with a build off main.

[slok]

You have images available for all main latest commits. For example last one c8ddd5d7e29f70a8ad10a7739c528e556d67ac8a -> ghcr.io/slok/sloth:c8ddd5d7e29f70a8ad10a7739c528e556d67ac8a.