One collector, two pipelines, one memory

In Part II we added the otelcol.processor.memory_limiter to protect the collector from running out of memory. That post used the percentage settings (limit_percentage = 80). I need to correct that advice, and show a second, bigger problem.

You can run two pipelines in one Grafana Alloy collector. One pipeline takes OpenTelemetry (OTLP) data. The other scrapes Prometheus metrics. This looks nice: one deployment, one set of credentials, one thing to run.

But there is a hidden risk. The memory limiter protects only the OTLP pipeline. It does nothing for the Prometheus pipeline. And both pipelines use the same memory. So when the Prometheus side uses too much memory, it either starves the healthy OTLP pipeline or kills the whole collector.

This post shows both problems on a small kind cluster that sends data to Grafana Cloud. Then it shows how to fix the Prometheus side.

First, fix the percentage setting

Part II suggested limit_percentage = 80. There is a trap here. The percentage setting looks at the total memory of the host or node. It does not look at the container memory limit.

Inside a 256 MiB pod on a large node, "80 percent" means several gigabytes. So Kubernetes kills the pod long before the limiter starts. In a container, always use fixed sizes, set below the container limit:

otelcol.processor.memory_limiter "guard" {
  check_interval = "1s"
  limit          = "180MiB"   // hard limit, below the 256Mi container limit
  spike_limit    = "40MiB"    // soft limit = 180 - 40 = 140MiB
  output { ... }
}

What memory_limiter can and cannot reach

Every second, the limiter checks the memory the process uses. If memory is too high, it starts to refuse new data. "Refuse" means it returns an error to the OTLP receiver in front of it, which then tells the sender to slow down. So the limiter has only one tool: it pushes back on data that goes through the OpenTelemetry pipeline.

The Prometheus pipeline is separate: prometheus.scrape collects samples, prometheus.relabel changes them, and prometheus.remote_write sends them out. This data never goes through the OpenTelemetry pipeline. So the limiter cannot refuse, drop, or slow it. There is no memory limiter on the Prometheus side.

But both pipelines run in one process and share one memory (one Go heap). Every series the scraper loads, and every batch the remote-write queue holds, uses the same memory the limiter measures. So the limiter sees the pressure, but it cannot fix the cause. It can only slow the OTLP pipeline. The pipeline that caused the problem keeps running. The pipeline that did nothing wrong gets punished.

Reproducing it

The test is a kind cluster with one Alloy deployment that runs both pipelines. It uses two load generators: telemetrygen sends a steady, healthy OTLP stream, and avalanche is a Prometheus cardinality bomb that we can tune. Both pipelines send to Grafana Cloud. Alloy also scrapes its own metrics, so we can watch the failure on a dashboard.

There are two scenarios. They differ only in how hard we push the Prometheus path and how much memory the container has.

Problem one: the fast OOM (limiter is skipped)

Container limit: 256 MiB. avalanche serves about 400,000 series.

On each scrape, Alloy must load and parse all those series into memory at once. This pushes the heap to about 410 MiB in less than one second. That is faster than the limiter's one-second check. So Kubernetes sees the container go over its limit and kills it (exit code 137) before the limiter even runs. The pod keeps crashing and restarting.

lastTerminatedReason = OOMKilled   exitCode = 137   restarts keep rising
otelcol_receiver_refused_metric_points_total = 0   (the limiter never ran)

Here the limiter is not protecting the wrong pipeline. It is skipped completely. And it gets worse: when the collector dies, it also stops sending its own metrics. So the metrics you need to understand the problem disappear. A collector that runs out of memory goes blind while it falls.

Problem two: the slow cascade (limiter starves the healthy pipeline)

Now give the container more room (1 GiB), so the limiter, not Kubernetes, is the real limit. Use a steady load of about 40,000 series.

The heap now rises and stays between about 256 and 900 MiB. That is well above the 180 MiB hard limit. So the limiter does its job: it refuses OTLP data. On the dashboard, otelcol_receiver_refused_metric_points_total goes up, and otelcol_receiver_accepted_metric_points_total drops to zero. The healthy telemetrygen stream is now being dropped.

At the same time, the Prometheus pipeline that caused the problem is not slowed at all. prometheus.remote_write keeps sending 6,000 to 7,000 samples per second (about 250 kB/s) to the backend. So the pipeline at fault floods the backend, while the healthy pipeline starves. This run already had GOMEMLIMIT set, but the heap still spikes near the limit during each scrape, so the collector also restarts a few times. GOMEMLIMIT bounds the steady-state heap, not these sudden spikes.

How to protect the Prometheus path

The Prometheus pipeline has its own controls. Together they stop one noisy target from killing the collector.

• Set GOMEMLIMIT (the most convenient option). Unlike the memory limiter, which only sees the OTLP pipeline, GOMEMLIMIT is a Go runtime setting for the whole process. As the heap nears the limit, the garbage collector works harder and frees memory for both pipelines, including the Prometheus path. It is the single easiest knob: one environment variable, set a bit below the container limit (about 90 percent).

env:
  - name: GOMEMLIMIT
    value: "230MiB"   # about 90% of the 256Mi container limit

But it is necessary, not sufficient. It is a soft limit: it bounds the steady-state heap by making garbage collection work harder, but it cannot stop a sudden scrape spike. Both failures above still happened with GOMEMLIMIT set. And very aggressive garbage collection can raise CPU. Treat it as the easy baseline, then add the limits below.

• Limit the scrape. prometheus.scrape supports body_size_limit, sample_limit, and label_limit. These are not the same: body_size_limit caps how many bytes Alloy reads from a target, so it limits the memory used while reading a huge response. This is the one that helps against the fast OOM above. sample_limit is checked only after the body is parsed: it rejects a scrape with too many series, which protects the remote-write queue and the backend, but it cannot stop the short parse spike.

prometheus.scrape "targets" {
  body_size_limit = "10MiB"   // caps bytes read -> limits the parse spike
  sample_limit    = 50000     // rejects after parse -> protects queue + backend
  label_limit     = 30
}

• Limit the remote-write queue. queue_config (capacity, max shards) controls how much remote-write keeps in memory when the backend is slow.

prometheus.remote_write "cloud" {
  endpoint {
    url = "..."
    queue_config {
      capacity   = 2500
      max_shards = 50
    }
  }
}

• Drop series before remote-write. Use prometheus.relabel to keep only the series you need (with a keep or drop action) before they reach the remote-write queue. This lowers how much the queue and the WAL hold in collector memory. The scrape still loads everything for a short moment, so use this together with sample_limit.

prometheus.relabel "keep_needed" {
  rule {
    source_labels = ["__name__"]
    regex         = "go_.*|process_.*|up"
    action        = "keep"
  }
}

Note: Grafana Cloud Adaptive Metrics also reduces cardinality, but it runs in the backend after the collector sends the data. So it lowers storage and cost, not collector memory.

• Split the collectors. The safest option is to run the noisy Prometheus scraping in its own collector, with its own memory budget. Then a cardinality spike there cannot starve or kill the OTLP pipeline that shares its memory today.

Summary

otelcol.processor.memory_limiter protects one pipeline, not the whole process. In a mixed collector it can only slow the OpenTelemetry side, even when the Prometheus side is the real cause. And with a small container limit, it does not even get the chance to run.

If you run both pipelines in one Alloy collector, give the Prometheus path its own limits, or give it its own collector. Do not expect the memory limiter to cover the Prometheus side. It never did.