Build a Prometheus Cardinality Agent: Find Label Explosions, Prove the Metric Is Unused, and Ship metric_relabel_configs Drops as PRs

What This Agent Does

A Prometheus cardinality agent finds the metrics and labels that are eating your active-series budget, proves which of them nobody actually queries, and opens one pull request per scrape job with a metric_relabel_configs drop and a before/after series estimate. The finding and the proving are deterministic: the TSDB status API and mimirtool analyze do that work, not a model. The LLM enters at the one point where judgment lives, choosing between dropping a metric, dropping a label, shrinking a histogram, or sending the problem back to the team that instrumented it. Nothing is dropped until a human merges.

The reason this is worth automating is that cardinality is the one Prometheus cost that nobody owns. Every team adds labels; nobody removes them. If you pay for a hosted backend, active series are the bill. Grafana Cloud, Mimir, and most SaaS metrics vendors meter per thousand active series per month, so a single pod_template_hash label on a busy histogram is a line item. If you self-host, the same series cost you memory on the Prometheus head block and query latency on every dashboard that touches them. This is the metrics-side sibling of the alert tuning agent: same shape, same guardrails, different disease.

Where Cardinality Actually Shows Up

Three places tell you where series come from, and the miner reads all three.

The TSDB status endpoint is the cheapest and the first stop. It costs nothing to call and returns the top offenders by metric name, label name, and label value pair straight from the head block:

curl -s "http://prometheus:9090/api/v1/status/tsdb" | jq '{
  head: .data.headStats.numSeries,
  by_metric: .data.seriesCountByMetricName[:5],
  by_label:  .data.labelValueCountByLabelName[:5],
  by_pair:   .data.seriesCountByLabelValuePair[:5]
}'

The seriesCountByMetricName list tells you which metric is big. The labelValueCountByLabelName list tells you why it is big: a label with 40,000 distinct values is almost always a request ID, a user ID, a full URL path, or a pod hash that somebody attached to a counter. Newer Prometheus versions accept a limit query parameter if the default top ten is not enough.

Scrape-level metrics tell you which job is responsible. Prometheus exports these for every target, and they make the per-job blame trivial:

# Series each job contributes to the head block, approximated per scrape
topk(10, sum by (job) (scrape_samples_scraped))

# Churn: new series being created per second. Steady churn means a
# label that changes on every restart or every request.
sum(rate(prometheus_tsdb_head_series_created_total[5m]))

# Distinct values of one suspect label on one suspect metric
count(count by (pod_template_hash) (container_cpu_usage_seconds_total))

Churn is the number most teams never look at. Total series can be flat while series created runs at hundreds per second, which means the head block is full of short-lived series from rolling deployments. Those still count as active series for the billing window and still get indexed.

Usage data is the half that makes a drop safe, and it lives outside Prometheus. A metric with two million series that feeds the latency SLO is not waste; the same metric with zero references anywhere is. Grafana ships a tool that does this cross-reference for you:

# What do dashboards query?
mimirtool analyze grafana --address=https://grafana.internal \
  --key="$GRAFANA_SA_TOKEN" --output=metrics-in-grafana.json

# What do alerting and recording rules query?
mimirtool analyze rule-file rules/*.yaml --output=metrics-in-ruler.json

# Join against what Prometheus actually holds
mimirtool analyze prometheus --address=http://prometheus:9090 \
  --grafana-metrics-file=metrics-in-grafana.json \
  --ruler-metrics-file=metrics-in-ruler.json \
  --output=prometheus-metrics.json

The output splits every metric into in_use_metric_counts and additional_metric_counts, each with its series count. The second list is your candidate pool: metrics that exist, cost series, and are referenced by nothing. On a typical Kubernetes cluster the unused list is a majority of total series, because cadvisor and kube-state-metrics export far more than any dashboard reads.

The Miner: Ranking Stays in Code

Never ask a model which metric is expensive. That is arithmetic over the three sources above, and it belongs in Python, where it is testable and where its output is the same every week. The miner joins the TSDB top list with the usage analysis and the per-job attribution, and produces a short ranked list:

# cardinality_miner.py — deterministic, no model involved
import json, os, httpx

PROM = os.environ["PROM_URL"]
PROTECTED = {"up", "scrape_samples_scraped", "scrape_series_added",
             "ALERTS", "ALERTS_FOR_STATE", "kube_pod_info",
             "kube_deployment_status_replicas_available"}

def tsdb_status() -> dict:
    r = httpx.get(f"{PROM}/api/v1/status/tsdb", params={"limit": 50}, timeout=30)
    r.raise_for_status()
    return r.json()["data"]

def label_values(metric: str, label: str) -> int:
    q = f"count(count by ({label}) ({metric}))"
    r = httpx.get(f"{PROM}/api/v1/query", params={"query": q}, timeout=60)
    res = r.json()["data"]["result"]
    return int(float(res[0]["value"][1])) if res else 0

def mine() -> list[dict]:
    status = tsdb_status()
    head = status["headStats"]["numSeries"]
    usage = json.load(open("prometheus-metrics.json"))
    unused = {m["metric"]: m["count"] for m in usage["additional_metric_counts"]}
    in_use = {m["metric"]: m["count"] for m in usage["in_use_metric_counts"]}

    suspects = [p["name"] for p in status["labelValueCountByLabelName"]
                if p["value"] > 1000]          # a label this wide is a smell

    report = []
    for entry in status["seriesCountByMetricName"]:
        name, series = entry["name"], entry["value"]
        if name in PROTECTED or series / head < 0.01:
            continue                            # under 1% of head is not this week's problem
        wide = {lbl: label_values(name, lbl) for lbl in suspects}
        wide = {k: v for k, v in wide.items() if v > 100}
        report.append({
            "metric": name,
            "series": series,
            "share_of_head": round(series / head, 3),
            "referenced": name in in_use,
            "wide_labels": wide,                # {label: distinct_values}
            "score": series * (1.0 if name in unused else 0.25)
                     * (2.0 if wide else 1.0),
        })
    return sorted(report, key=lambda r: -r["score"])[:10]

Two choices carry the design. Unreferenced metrics score four times higher than referenced ones, because a drop there is nearly free and the LLM should spend its attention where the win is cheap. And the wide_labels dictionary is computed per metric rather than globally, because a label like pod is legitimately wide on kube_pod_info and a bug on http_requests_total. The model needs that per-metric detail to choose a fix.

The LLM's Job: One Fix Per Metric, Forced Schema

For each of the ten, the agent fetches the scrape config that produces the metric, a sample of raw series (twenty label sets, never the firehose), and the usage verdict, then makes a single call with a forced tool schema:

PROPOSAL_TOOL = {
    "name": "propose_cardinality_fix",
    "description": "Propose exactly one change for a high-cardinality metric.",
    "input_schema": {
        "type": "object",
        "properties": {
            "diagnosis": {"enum": [
                "drop_metric_unused", "drop_label", "histogram_too_many_buckets",
                "label_is_instrumentation_bug", "legitimate_leave_alone"]},
            "relabel_yaml": {"type": "string",
                "description": "A metric_relabel_configs block, or empty for "
                               "leave_alone and instrumentation_bug."},
            "expected_series_removed": {"type": "integer"},
            "evidence": {"type": "string",
                "description": "2-4 sentences citing series count, share of "
                               "head, usage verdict, and the wide label."},
            "what_breaks": {"type": "string",
                "description": "Which query or dashboard could lose data, "
                               "and why that is acceptable."},
        },
        "required": ["diagnosis", "relabel_yaml", "expected_series_removed",
                     "evidence", "what_breaks"],
    },
}

SYSTEM = (
    "You reduce Prometheus active-series cost for an SRE team.\n"
    "Prefer the narrowest change: drop one label before dropping a metric, "
    "drop a metric only when the usage analysis says it is unreferenced.\n"
    "If the wide label is a request ID, user ID, URL path, or trace ID, the "
    "fix belongs in the application, not in relabeling: answer "
    "label_is_instrumentation_bug and say which team owns the exporter.\n"
    "A referenced metric with a legitimately wide label is leave_alone. "
    "Silence is an acceptable output.\n"
    "Your proposal becomes a pull request a human reviews. Cite numbers."
)

The what_breaks field does the same job the risk field does in the alert tuning agent. A model forced to name the dashboard that loses a panel, before it is allowed to drop the series behind it, produces far fewer confident mistakes than a model asked only for YAML. The label_is_instrumentation_bug branch matters just as much. Relabeling a user_id label away at scrape time hides the symptom while the application keeps allocating a label set per user. The right output there is an issue for the owning team, not a relabel rule, and the schema makes that a first-class answer instead of a workaround.

Every Drop Is a PR With an Impact Estimate

Proposals ship as pull requests against the scrape config repo, one scrape job per PR, following the GitOps-for-agents pattern. A typical drop looks like this:

# prometheus/scrape-configs/kubelet-cadvisor.yaml
- job_name: kubernetes-cadvisor
  metric_relabel_configs:
    # Agent PR #412: unreferenced per-container network metrics,
    # 184,220 series (9.1% of head). Zero dashboard or rule references.
    - source_labels: [__name__]
      regex: 'container_network_(tcp|udp)_usage_total'
      action: drop
    # Agent PR #412: container_memory_failures_total is referenced by
    # one dashboard but the 'scope' label doubles every series.
    # Blank the label on this metric only; an empty value removes it.
    # (A plain labeldrop would strip 'scope' from every metric in the job.)
    - source_labels: [__name__, scope]
      regex: 'container_memory_failures_total;.*'
      target_label: scope
      replacement: ''
      action: replace

The wrapper adds two things to the PR body before a human sees it. First, the exact impact: it runs count({__name__=~"container_network_(tcp|udp)_usage_total"}) and count(sum without (scope) (container_memory_failures_total)) and quotes the results, so the reviewer sees the number the drop removes rather than the number the model guessed. Second, a duplicate-collision check for every label removal. If blanking a label collapses two series into one, Prometheus rejects the second sample as a duplicate and you lose data silently. The check is one query: if count(sum without (scope) (m)) is smaller than count(m), the label was load-bearing and the PR is downgraded to a comment asking for a human decision.

CI then runs promtool check config prometheus.yml so a bad regex never reaches a reload. Once merged, the canary is the metric you already have. The wrapper posts prometheus_tsdb_head_series an hour before and an hour after the reload to the PR as a follow-up comment, and the team sees the saving land. A git revert undoes the whole decision, which is more than you can say for a label somebody deleted in the exporter.

Guardrails: The Do-Not-Touch List Lives in Code

Four rules live in the wrapper, not the prompt, for the same reason the read-only Prometheus MCP server enforces read-only at the HTTP layer rather than asking nicely:

• Usage is a hard filter, not a hint. A metric that appears in in_use_metric_counts can only receive a drop_label or leave_alone proposal. The wrapper rejects any drop_metric_unused verdict on a referenced metric before it becomes a PR, no matter how the model argued it.
• Protected names never reach the model. Anything consumed by SLO recording rules, the up and scrape_* families, ALERTS, and the core kube_* state metrics that the capacity and event-triage agents depend on are filtered in the miner.
• A per-PR ceiling. No single PR may remove more than 15% of head series. A drop that large is a config bug or a landmark decision, and either way it deserves a meeting, not an auto-generated diff.
• Rate limiting. Three open cardinality PRs at a time. Twenty simultaneous PRs get rubber-stamped, and a rubber-stamped loss of observability is the worst thing this agent can produce.

Run the first month in shadow mode: proposals land in a channel as messages with their impact estimates, and you score them against what the team would have done. Dashboards are the place agents most often get usage wrong, so point the Grafana MCP server at the same instance and have the agent cite the panel it checked by dashboard UID. Agreement rate above 80% is the signal to wire up the PR path.

Honest Limits

The usage analysis is only as complete as what mimirtool can see. Dashboards with templated variables are resolved only if the variable query is a label_values() call on a real metric. Ad-hoc Explore queries, Jupyter notebooks hitting the API, and a colleague's shell alias are invisible, so "unreferenced" means unreferenced by dashboards and rules, not unloved. The protected list and the shadow-mode month exist precisely because that gap is real. Recording rules that reference other recording rules are followed, but only if you feed every rule file to analyze rule-file, so run it from the repo, not from a partial export.

The TSDB status endpoint reports the head block, which is roughly the last two hours. A metric that explodes only during a nightly batch job will not appear at a 10 a.m. run. Schedule the miner twice, once at peak and once at the quietest hour, and union the results.

The deeper limit is the same one every cost agent in this series shares, from the LLM inference cost post onward: the model can tell you a drop is safe, and it can be wrong in a way that only shows up during the next incident, when the panel you needed is empty. That is why usage is a hard filter, why label removal needs a collision check, why the ceiling is 15%, and why a human merges every change. The agent's real product is not the relabel YAML. It is turning an annual "why is our metrics bill up 40%" panic into a weekly, evidence-backed, individually reviewable decision, with the series count on the PR to prove it.