Support for multi-tenant environments and federated queries on backends like Mimir

[jnavarrof]

Hi all,

I’m exploring the use of Sloth to generate and manage SLOs in a multi-tenant observability stack. We’re currently using Grafana Mimir as our backend, which supports multi-tenancy and federated queries across tenants (via X-Scope-OrgID and tenancy-aware query endpoints).

I wanted to ask:

• Does Sloth currently support defining SLOs across tenants (e.g., generating recording rules or alerting rules that refer to metrics from different tenants)?
• Can Sloth work with federated queries in backends like Mimir?

Would love to hear from anyone who has done something similar—or if this is something on the roadmap.

Thanks!

[ishantanu]

We use Sloth with Mimir backend. As far as I am aware, Sloth does not have native Mimir integration which would allow defining SLOs across multiple tenants or run federated queries.

What we have done so far (a very simplified version)

• Sloth (or modified open-source Sloth) generates recording/alerting rules
• Custom code which has logic for injecting namespace: my-namespace to rules files. The actual value is generated based on multiple parameters like tenantID, application and other organisation specific details
• Using mimirtool CLI to prepare and load rules to specific tenants and mimir endpoints

There's quite a bit of structure and logic around how tenants and Mimir endpoints are chosen for deployment of rules. There's no use of query federation at the moment though. Users usually define SLOs based on metrics available to them in a specific tenant.

[jnavarrof]

Thanks for the comment, @ishantanu really helpful!

We’re in a similar situation. Managing the creation and deployment of rules per tenant is doable with automation, especially using tools like mimirtool and some custom logic. However, querying metrics across tenants to compute global or shared SLOs remains a challenge.

Is it possible for Sloth to support querying metrics from other tenants? For example, could we specify a source tenant or org scope for this SLI?

version: "prometheus/v1"
service: "myapp"
labels:
  owner: "myteam"
slos:
  - name: "http-availability"
    objective: 99.99
    description: "Common SLO based on availability for HTTP request responses."
    sli:
      events:
        # Specify the tenant(s) from which to query metrics for this SLI. Useful for federated setups.
        source: ["other_metrics_tenant"]
        error_query: |
          sum(
            rate(http_request_duration_seconds_count{job="myapp", code=~"(5..|429)"}[{{.window}}])
          )
        total_query: |
          sum(
            rate(http_request_duration_seconds_count{job="myapp"}[{{.window}}])
          )

This would enable federated or cross-tenant queries in backends like Mimir. Would be great to know if others are exploring similar ideas or if something like this is already being considered upstream in Sloth.

[victor-smg]

Like ishantanu, I set-up a dockerfile with a script that generate the sloth templating, and then use the CLI tools to generate rules (that are independent of tenant) and apply those in specific mimir tenant.

for file in http-duration/*; do
  filename=$(basename "$file")
  ./sloth-linux-amd64 generate -i "$file" -o "slo/generated/$filename" --default-slo-period=28d
done

for file in $(find slo/generated -type f); do
  ./mimirtool rules load --id=prod-company --address=http://mimir-ruler.mimir:8080 $file
done

Sloth is not aware of our multi tenant set-up, we don't have use-cases for cross tenant queries

[slok]

Hi there folks! 👋

In the next release (v0.15.0), Sloth will be available as a Go library too. This may help to some of you, as if SLO plugins are not enough for your use case, you can even customize it more by making a small Go app that uses Sloth as a framework with the new library.

Check this one #687, there you can see that you already can start testing it.

Hope this solves some of your use cases too 😄.