Migrating from Promtail to Grafana Alloy: Logs, Cardinality, and Production Failure Modes

Tested Configuration

• Grafana Alloy: v1.19.2
• Deployment: Linux systemd and Kubernetes
• Log backend: Loki
• Metrics backend: Prometheus or Mimir
• Validation: Verified using alloy fmt and alloy validate
• Last updated: 13 September 2026 — re-checked against Alloy v1.19.2

Running separate agents for logs, metrics, and traces increases operational overhead across fleet infrastructure. Whether you are managing thousands of bare-metal Linux servers or autoscaling Kubernetes clusters, each distinct collector adds duplicate service discovery, a unique configuration language, and its own CPU and memory footprint.

With Grafana announcing that Promtail reached End-of-Life on March 2, 2026, migrating to a unified collector should now be treated as a maintenance and support priority.

This guide explains how Grafana Alloy acts as a single, consolidated agent. It covers how to execute a safe migration, filter unnecessary logs, control Prometheus cardinality at the edge, and deploy the collector safely across both traditional infrastructure and Kubernetes.

In this guide

• Why consolidate telemetry with Grafana Alloy
• Migrating from Promtail
• Building a log processing pipeline
• Controlling Prometheus cardinality
• Deployment Topologies: Bare-Metal vs Kubernetes
• Resource sizing and security
• Failure modes to expect in production
• Validating and operating Alloy
• Frequently Asked Questions

The Architecture: Fragmented vs. Unified

Historically, a standard infrastructure observability stack required deploying three distinct agents to every host:

Before (Fragmented):

Promtail          → Loki (Logs)
Node Exporter     → Prometheus (Metrics)
OTel Collector    → Tempo (Traces)

Grafana Alloy resolves this by embedding the OpenTelemetry Collector stack alongside native Prometheus scraping and remote-write capabilities in a single binary.

After (Unified):

Grafana Alloy
├── Logs          → Loki
├── Metrics       → Prometheus or Mimir
└── Traces        → Tempo or an OTLP backend

Migrating from Promtail

Because Promtail is no longer maintained, migrating to Alloy is the official upgrade path. A safe migration workflow follows these steps:

1. Back up existing configurations: Preserve the running Promtail YAML configuration.
2. Convert the configuration: Use the built-in Alloy conversion command (alloy convert --source-format=promtail --output=config.alloy promtail.yaml). This command is public-preview, not GA — treat its output as a first draft, not a finished config.
3. Review pipeline stages: Inspect the generated Alloy components to ensure all labels and drop stages were translated correctly. Check the conversion report (-r report.txt) for any stages the converter flagged as unsupported.
4. Run in parallel (optional): Deploy Alloy alongside Promtail temporarily, but point Alloy to a null or testing backend to avoid double-shipping logs to production Loki.
5. Compare and cutover: Once ingestion matches expectations, stop the promtail service (or delete the DaemonSet) and redirect Alloy to the production endpoint.

Building a Log Processing Pipeline

Alloy abandons the static YAML configurations used by legacy agents in favour of the Alloy configuration syntax, an HCL-inspired, component-based DSL (this language was called “River” in Grafana Agent Flow; Grafana renamed it when Alloy launched at 1.0, and the name persists in some older blog posts and forum threads). Instead of managing disconnected snippets for ingestion, filtering, and exporting, the Alloy syntax allows for a cohesive end-to-end flow.

The following pipeline demonstrates reading application logs, dropping health-check noise, extracting low-cardinality metadata from JSON logs (e.g., {"level":"error", "service":"frontend", "message":"..."}), and pushing the result to Loki:

loki.source.file "app_logs" {
	targets = [
		{__path__ = "/var/log/applications/*.log", env = "production"},
	]
	file_match {
		enabled = true
	}
	forward_to = [loki.process.filter_and_extract.receiver]
}

loki.process "filter_and_extract" {
	stage.drop {
		expression = ".*GET /healthz.*"
	}

	stage.json {
		expressions = {log_level = "level", app_service = "service"}
	}

	stage.labels {
		values = {
			level   = "log_level",
			service = "app_service",
		}
	}

	forward_to = [loki.write.default.receiver]
}

loki.write "default" {
	endpoint {
		url = "http://loki-service:3100/loki/api/v1/push"
	}
}

The Glob Trap

By default, loki.source.file requires absolute file paths — it does not expand glob patterns on its own. Pass a __path__ glob straight into targets without enabling discovery first, and Alloy treats it as a literal path: the component still reports health: healthy, alloy validate passes clean, and it silently ships zero logs. The only trace is a single ERROR-level line logged once at startup (failed to create source, skipping... stat failed: ... no such file or directory) — easy to miss, since nothing about the component’s ongoing health or validate output flags it afterward.

Enable glob discovery with the built-in file_match block on loki.source.file itself, as shown in the pipeline above — this is the recommended approach, since it avoids a separate component and has less overhead. Reach for the standalone local.file_match component instead only when you need to share the same discovered targets across multiple consuming components; it’s still supported, just no longer the first choice for a single loki.source.file pipeline.

Edge Filtering Nuances

The stage.drop example above applies the regex .*GET /healthz.* before stage.json runs, so it correctly matches against the raw log line. If you reorder the pipeline to filter on a parsed field instead (for example, dropping by an extracted path value), move stage.drop after stage.json and match on the parsed field with source = "path" — matching a JSON key like {"path":"/healthz"} against a line-level regex won’t work until the field has actually been extracted.

Furthermore, stripping all /healthz traffic removes data that may be critical during load-balancer troubleshooting or availability analysis. Only drop noise that has zero investigatory value.

Similarly, care must be taken with stage.labels. The stage.json block extracts the level and service fields, while the original log line remains available as the log content. Promoting user IDs or email addresses into Loki labels creates high cardinality and privacy concerns. Always extract low-cardinality metadata, leaving highly unique identifiers in the JSON payload.

Controlling Metric Cardinality

High-cardinality labels (e.g., injecting client IPs directly into Prometheus labels) multiply the number of active time series. A metric exported with thousands of unique label values creates thousands of distinct series, rapidly exhausting backend memory.

Alloy handles cardinality pruning at the edge using the prometheus.relabel component. It sits between a scrape job and a remote write destination, allowing you to selectively drop high-cardinality labels:

prometheus.scrape "node_metrics" {
	targets = [
		{__address__ = "localhost:9100"},
	]
	forward_to = [prometheus.relabel.prune_labels.receiver]
}

prometheus.relabel "prune_labels" {
	forward_to = [prometheus.remote_write.main.receiver]

	rule {
		action = "labeldrop"
		regex  = "(user_id|session_id|client_ip)"
	}
}

prometheus.remote_write "main" {
	endpoint {
		url = "http://mimir.example.com/api/v1/write"
	}
}

Trade-off: It can remove selected labels while retaining the metric samples, although doing so may merge previously distinct series and change how the metric should be interpreted. If two requests differ only by client_ip, dropping that label merges those samples, making per-client troubleshooting impossible.

Deployment Topologies: Bare-Metal vs Kubernetes

Alloy is infrastructure-agnostic. While the Alloy configuration syntax remains identical across environments, the deployment topology depends entirely on the host architecture.

Bare-Metal and VMs

For legacy or traditional infrastructure, Alloy is deployed as a standard systemd service, typically provisioned via Ansible or Puppet. It runs as a single daemon per host, mounting local directories (/var/log) and scraping local endpoints.

Kubernetes

In Kubernetes environments, the workload controller type must be deliberately chosen based on the pipeline’s requirement:

For log collection via a DaemonSet, ensure the workload has host filesystem mounts, access to container log paths (/var/log/pods), and the appropriate RBAC permissions to watch the Kubernetes API for service discovery.

For clustered topologies designed to distribute heavy scrape targets across replicas, a StatefulSet is recommended. A StatefulSet provides the stable pod identities that make peer discovery and load balancing via Alloy’s clustering mode straightforward; a Deployment can technically cluster too, but you lose stable network identities and have to solve peer discovery yourself. For a first-hand account of why you might run multiple Alloy instances rather than one large cluster, see Grafana Alloy in My Homelab: Why I Run Three Separate Instances.

Resource Sizing and Security

CPU and Memory Allowances

The primary directive of any telemetry agent is to never starve the host node. Grafana’s resource estimation guidance gives a rule of thumb of approximately 0.4 CPU cores and 11 GiB of memory per 1 million active series under its documented default conditions.

Actual usage varies significantly with scrape interval, log volume, batching, queues, and enabled components.

• Linux environments: Enforce limits using systemd constraints (e.g., MemoryMax= and CPUQuota=). MemoryLimit= is the legacy cgroup v1 directive and is deprecated on any modern distro running cgroup v2.
• Kubernetes environments: Set strict resources.requests based on baseline usage, and apply resources.limits with a generous safety margin (typically 1.5× the request) to prevent OOM kills during usage spikes.

Deployment Security Guidance

Regardless of the host platform, implement strict security defaults:

• Credentials: Avoid putting credentials directly in Alloy configuration files. Source them from secure environment variables or Kubernetes Secrets.
• Transport: Enforce TLS for all remote endpoints (loki.write and prometheus.remote_write).
• Execution: Run the alloy process as a non-root system user, or configure a non-root securityContext in Kubernetes pods.
• Network: Restrict network access to the Alloy UI and metrics endpoints.

Failure Modes to Expect in Production

Log Duplication on Restart

loki.source.file tracks its read offset per file in a positions file so that a restart resumes tailing from the same point rather than re-reading everything. A missing positions file is harmless — Alloy just starts fresh from an empty map — and a fully corrupted positions file stops Alloy from starting at all. The on_positions_file_error argument governs a narrower case: what to do when one file’s individual stored offset entry fails to parse. The default is restart_from_beginning — which re-reads that file from the start and re-ships every line as a duplicate. If you’d rather lose a few log lines than duplicate an entire file’s worth, set on_positions_file_error = "restart_from_end" (this mirrors tail_from_end and isn’t supported alongside decompression).

Remote-Write Backpressure and the WAL

prometheus.remote_write buffers samples in a Write-Ahead Log before shipping them, and by default retries HTTP 429 responses, honouring any Retry-After header and otherwise falling back to an exponential backoff between 30ms and 5s. This means a slow or rate-limiting Mimir/Prometheus endpoint doesn’t immediately drop data — it backs up in the WAL. If the endpoint stays unreachable for longer than the WAL’s max_keepalive_time (8h by default), older samples get truncated and are gone for good. Watch prometheus_remote_storage_queue_highest_sent_timestamp_seconds against wall-clock time to catch a growing backlog before you hit that ceiling, and treat prometheus_remote_write_wal_out_of_order_samples_total as an early signal that clustered Alloy instances are scraping the same targets and racing each other.

Validating and Operating Alloy

Validating Configuration Locally

Before deploying configuration changes, always verify them locally using the Alloy binary:

1. alloy fmt -w config.alloy: Formats the file and can expose basic formatting or parsing issues.
2. alloy validate config.alloy: Checks the configuration for syntax errors, unknown or missing components, name conflicts, and invalid arguments — it does not guarantee the config will load cleanly at runtime, so a config can pass validate and still fail on run.

Operational Commands

To validate the deployment and troubleshoot the collector, use the standard tooling for your environment:

For Bare-Metal/VMs:

systemctl status alloy
journalctl -u alloy --since "15 minutes ago"

For Kubernetes:

kubectl get pods -n alloy
kubectl logs -n alloy -l app.kubernetes.io/name=alloy

You can also verify Alloy’s internal status by querying its health endpoints directly:

curl http://localhost:12345/-/ready
curl http://localhost:12345/metrics

Common Troubleshooting Scenarios:

• Alloy starts but cannot read logs: Verify the process has the correct filesystem permissions, SELinux/AppArmor profiles, or Kubernetes host mounts for /var/log.
• Metrics disappear after labeldrop: Ensure the relabeling regex did not inadvertently match and drop a required system label.
• Clustered peers cannot discover one another: If running Alloy in cluster mode, check firewall rules allowing peer-to-peer gossip traffic (bare-metal) or headless service DNS resolution (Kubernetes).

Conclusion

Grafana Alloy is a strong fit when an engineering team wants one supported collector for Prometheus, Loki, and OpenTelemetry pipelines across hybrid infrastructure. Start with a small Promtail migration on a subset of hosts or a single cluster, validate data quality and resource usage, and then expand the deployment. Consolidation is valuable, but only when filtering rules, cardinality changes, and failure behaviour are actively measured and monitored.

This kind of migration sits within Site Reliability Engineering — the same discipline covers SLOs, alerting, and the Prometheus/Grafana/Loki/Mimir/Alloy stack this article works through. Get in touch about a migration or observability audit.

Frequently asked questions