DevOps engineer reviewing Kubernetes logs, cluster status, and observability dashboards across three monitors.

Promtail Is EOL: Migrate Kubernetes Logging to Grafana Alloy Without Dropping Labels

Mon, Sep 7, 2026

The dangerous collector migration is not the one where Grafana Alloy crashes. It is the one where Alloy is green, CPU is normal, Loki ingestion volume is unchanged, and the on-call engineer gets zero results from the incident query that worked yesterday.

Imagine checkout-api is still producing 100 log events per second. Promtail used to attach cluster="prod-eu-1". Alloy now sends the same lines, timestamps, and bytes, but a migrated relabel rule emits cluster="prod_eu_1". Loki receives everything. The volume graph is flat. Yet {cluster="prod-eu-1", namespace="payments"} |= "payment timeout" returns nothing. Operationally, logging is broken.

That is the standard for this Promtail to Grafana Alloy migration: not “the replacement agent runs,” but “the replacement preserves useful evidence.”

Grafana’s Loki “Promtail agent” documentation states that Promtail reached end of life on March 2, 2026, commercial support ended, and no future support or updates will be provided; Lambda Promtail is explicitly separate from that deprecation.

So the migration is mandatory for teams that want a maintained client. But urgency does not justify an unproved cutover. Before deleting Promtail, prove event continuity, label and query equivalence, tenant correctness, restart behavior, and resilience under backpressure.

Treat Promtail End of Life as an Operational Migration

A Promtail EOL migration transfers operational responsibility, not merely a YAML file. After March 2, 2026, running standalone Promtail means accepting an unmaintained collector while Grafana directs future development toward Alloy. Grafana also notes that Promtail was subsequently removed from Loki as of Loki 3.7.3, while Lambda Promtail remains outside that removal.

Do not use agent health as the acceptance criterion. Current Alloy documentation says loki.source.file and loki.write report unhealthy primarily for invalid configuration. A collector can therefore be “healthy” while reading the wrong files, producing the wrong label values, writing to the wrong tenant, or failing the queries operators depend on. loki.source.file exposes target offsets and read metrics, while loki.write exposes retries and dropped-entry metrics; those are stronger evidence, but still need backend/query validation.

For broader ecosystem context, Refonte Learning already covers the Prometheus, Grafana, ELK, and Loki monitoring stack and observability in DevOps. This runbook deliberately starts later: at the point where a production logging path already exists and must be replaced safely.

Use this migration contract:

Proof category

What “equivalent” means

Promotion blocker

Event delivery

Expected source events arrive once, within an agreed delay

Missing sequence IDs, unexplained count gap, sustained drops

Timestamp semantics

Event time survives parsing identically

Material time shifts, parse fallback, rejected old/future entries

Indexed labels

Incident-routing labels preserve names and values

Missing/renamed/transformed required label

Parsed data

JSON, regex, logfmt and multiline output remain usable

Field extraction or multiline boundary changes

Tenant routing

Each event reaches the intended Loki tenant

Missing or incorrect tenant, cross-tenant leakage

Query behavior

Top incident queries, alerts and dashboards still return equivalent evidence

Empty or materially divergent acceptance query

Resilience

Restarts, rotations and Loki outages do not create unexplained gaps

Loss, uncontrolled duplicates, backlog that cannot recover

“No dropped labels” therefore means more than equal label counts. A cluster label renamed to cluster_name, moved into structured metadata, or normalized to a different value is a semantic break if existing LogQL, alerting, dashboards, or tenancy logic selects on it.

Decision rule: Promtail stays recoverable until all seven proof categories have owners and signed evidence.

Inventory Targets, Environments, Labels, Pipelines, and Owners

Do not begin with alloy convert. Begin with the estate. Collector migrations fail when the configuration repository represents only part of reality: a Helm override contains another label, a VM unit passes a command-line flag, an environment variable expands the tenant, or a node image changes the path mounted into the collector.

Create one inventory row per distinct collection contract, not merely per cluster.

Field

Record

Environment

Production, staging, development, regulated boundary

Cluster/host

Cluster ID, node pool, VM group, hostname class

Source

Kubernetes pod, host file, journal, syslog/network, application file

Physical location

File glob, symlink chain, journal selector, listener

Runtime

containerd/CRI-O/runtime version where relevant

Discovery

Kubernetes role/selectors, static targets, file matching

Pipeline

Ordered parsing/filtering/transformation stages

Indexed labels

Names, source, normalization and allowed values

Structured metadata

Fields deliberately not indexed

Tenant

Static tenant, derived tenant and source field/label

Destination

Loki URL/service and environment

Security

Authentication, TLS, proxy, certificate/Secret

Dependencies

LogQL, alerts, dashboards, investigations, exports

Position state

File location, volume/mount, persistence behavior

Owner

Platform owner plus consuming service/team

Rollback

Promtail manifest/package/config/state retained where safe

The owner column matters. Platform engineering can prove bytes arrived; only the service or incident-response owner can say whether the evidence remains operationally useful.

Map Kubernetes, VM, File, Journal, and Network Sources

For Kubernetes, inspect the real node filesystem and container runtime. Do not copy /var/log/pods or /var/log/containers assumptions from another cluster without checking the runtime, distribution, symlinks, host mounts, permissions, rotation pattern and exclusions. Grafana’s Kubernetes logging guidance for Alloy supports Kubernetes discovery plus file or Kubernetes-API collection; it specifically notes that node-file collection requires system privileges, while API-based pod collection can avoid them.

On virtual machines, enumerate static application files, glob expansion, compressed rotations, journald selectors, systemd units, Windows sources where applicable, and network listeners such as syslog. For journals, record group membership and journal retention as part of replay capability.

For every file source, capture a sample lifecycle: current filename → rotate → rename/compress → new file. File discovery that looks correct at startup can behave differently during rotation.

Capture Label, Parsing, Tenant, and Query Dependencies

Export the complete Promtail effective configuration and the process arguments. Promtail command-line options matter because Grafana’s migration guidance explicitly warns teams to account for additional arguments such as line-size settings and environment expansion rather than assuming YAML alone represents behavior. The guide also warns that Alloy meta-monitoring metric names differ from Promtail’s.

Record stages in execution order: container-runtime decoding, JSON or logfmt extraction, regular expressions, match, drop, replace, timestamp parsing, labels, output rewriting, multiline, tenant selection, and anything custom.

Then search the operational estate for each important label name: LogQL queries, Grafana dashboards, Loki alert rules, runbooks, scripts, support tooling and tenant routers. A label with only three values may be “low cardinality,” yet still be the most critical field in an outage.

Inventory gate: no conversion begins while a production source has an unknown owner, unknown position state, or unidentified incident-query dependency.

Freeze Log-Continuity and Incident-Query Baselines

A migration needs a before-image that can be replayed as tests. Save at least one normal operating window and one peak or failure-prone window. My recommended control is seven days of historical distributions plus controlled marker tests; that duration is an engineering policy, not a Grafana requirement.

For each collection contract, preserve:

Baseline

Measure

Volume

Lines and bytes by source, workload and tenant

Timing

Event-time distribution, ingestion delay, oldest backlog

Parsing

Success/failure counts for JSON, regex, logfmt and timestamp extraction

Labels

Exact label names, representative values and value counts

Cardinality

Active streams and distinct values per indexed label

Routing

Event counts per tenant/destination

Integrity

Missing IDs, duplicate IDs and malformed lines

Ordering

Monotonic sequence inversions per source

Operations

Top incident queries, alerts, dashboards and saved investigations

Do not compare only aggregate Loki bytes. Two collectors can produce identical byte volume while creating entirely different streams.

For applications you control, emit a migration-test record containing a globally unique event_id, source identifier and monotonic seq field. That gives you something stronger than heuristically hashing log text. Where application changes are impossible, calculate an offline fingerprint from normalized source identity, timestamp and line content; do not turn a per-event fingerprint into a Loki index label because that creates extreme cardinality.

Freeze representative LogQL acceptance tests, for example:

sum(count_over_time(
  {cluster="prod-eu-1", namespace="payments", app="checkout-api"}[5m]
))

and:

{cluster="prod-eu-1", namespace="payments"} |= "MIGRATION-MARKER-"

Store expected result characteristics, not merely query text: required labels, expected tenant, minimum event count, timestamp range and alert behavior.

Grafana’s Loki label guidance emphasizes that labels describe and select streams, while higher-cardinality search data is better suited to structured metadata. Loki also documents a default limit of 15 index labels. During parity migration, however, do not opportunistically redesign a label that existing incident queries require. First establish equivalence; optimize taxonomy as a separately reviewed change.

That distinction is central to Loki label migration: good future architecture does not excuse broken current operations.

Use the same discipline described in broader DevOps monitoring tools and practices: validation belongs in operating controls, not only installation steps.

Baseline gate: every tier-one workload must have at least one event-integrity test and one incident-query acceptance test.

Choose Conversion, Native Alloy Design, or a Staged Hybrid

Grafana provides more than one transition path, and they solve different problems. The official “Migrate from Promtail to Grafana Alloy” guide documents both generated conversion and temporarily running a Promtail-format configuration through Alloy.

Use this decision matrix:

Path

Initial fidelity

Long-term maintainability

Migration risk

Best use

Convert Promtail config

High starting fidelity when constructs convert cleanly

Medium until reviewed/refactored

Converter/default mismatch

First implementation for conventional configs

Native Alloy design

Must be proved manually

Highest

Semantic drift during redesign

Complex estates or intentional architecture change

Alloy running Promtail format

Fastest transition

Low as permanent state

Hidden runtime conversion/deferred cleanup

Temporary bridge only

Hybrid

Controlled

High after staged redesign

More temporary complexity

Large mixed Kubernetes/VM fleets

There is an important documentation boundary. The current alloy convert CLI reference labels the command Public Preview and specifically describes --source-format=promtail as converting Promtail v2.8.x configurations. It says nearly all Promtail features are supported, with unsupported constructs generating errors and potentially significant options generating warnings.

That is not a promise that every later Promtail deployment has production-equivalent semantics. Treat it as a conversion input specification and review boundary.

Use alloy convert as Reviewed Generated Configuration

Pin the tool first. The August 29, 2026 research baseline used Alloy 1.19.2, released August 26, 2026. Capture that version, the input checksum, command, standard error, diagnostic report and generated file:

alloy --version
sha256sum promtail-prod.yaml

alloy convert \
  --source-format=promtail \
  --report=promtail-convert-report.txt \
  --output=config.alloy \
  promtail-prod.yaml

Grafana documents --report, --output, --source-format, --extra-args, and --bypass-errors. The last option is not an approval mechanism: Grafana explicitly warns that bypassed errors can produce behavior that does not match the original configuration and requires full testing.

The temporary --config.format=promtail path likewise converts the Promtail configuration when Alloy loads it; Grafana explicitly says not to use the conversion-error bypass mode in production.

Decision rule: generated configuration receives the same code review and acceptance testing as hand-written configuration. A successful converter exit code proves syntax conversion, not log continuity.

Rebuild Discovery, File Targeting, and Relabeling

Promtail’s conceptual scrape job becomes a graph of Alloy components. Make that graph explicit because discovery errors are often volume-selective: one node pool, namespace or rotated filename disappears while the rest of the fleet hides the deficit.

A practical mapping is:

Source requirement

Typical Alloy components

Kubernetes pod discovery

discovery.kubernetes → discovery.relabel

Kubernetes API pod logs

discovery → loki.source.kubernetes

Kubernetes node files

discovery/relabel → loki.source.file

Static/globbed files

loki.source.file with file matching

Journald

loki.source.journal plus relabeling

Parsing/transforms

loki.process

Loki destination

loki.write

Grafana’s Kubernetes example creates namespace, pod, container, app and file-path labels from Kubernetes discovery metadata. For DaemonSet deployment it recommends restricting discovery to pods on the same node; its example uses spec.nodeName to reduce unnecessary CPU and memory.

For filesystem collection, current loki.source.file documentation recommends its built-in file_match capability over a separate local.file_match for simple glob discovery. The component requires an absolute path; double-underscore labels are internal and removed before forwarding.

Validate discovery in three directions: expected target found, excluded target absent, lifecycle target survives rotation. On Kubernetes, sample every node pool, including tainted pools if their workloads should be logged. Inspect the generated target path against the host filesystem rather than assuming containerd or distribution defaults.

Relabeling deserves its own diff. For each output label, record:

input metadata → regex/action → replacement → final label name → expected value domain.
That catches a change such as:
rule {
  action       = "replace"
  replacement  = "prod_eu_1" // incorrect: legacy value is prod-eu-1
  target_label = "cluster"
}

The rule is syntactically valid. Every line can still reach Loki. Only semantic validation finds the problem.

For journald and VM files, test permissions under the actual Alloy service account, not root in a shell. For file exclusions, create a known excluded file and prove it does not ingest.

Targeting gate: expected source count, source identities, paths and exclusions must match the frozen inventory before parsing correctness is assessed.

Preserve Parsing, Multiline, Timestamps, and Positions

A parser migration can damage evidence without reducing line volume. A Java stack trace can turn from one incident event into 40 independent lines. A JSON timestamp failure can silently switch investigation ordering from application time to collection time. A drop condition can invert because an extracted field changed type.

Alloy’s loki.process supports stages covering JSON, logfmt, regex, matching, dropping, labels, replacement, multiline, output, structured metadata, tenants and timestamps, among others. stage.tenant can derive tenancy from an extracted value, a label or a fixed value.

Build a golden corpus:

Fixture

Expected proof

Valid JSON

Same extracted fields and labels

Malformed JSON

Same fallback/drop decision

logfmt with escaping

Same field values

Regex match/non-match

Same branch behavior

Stack trace

Same first-line boundary and final event

Very long line

Explicit accept/truncate/drop behavior

Invalid UTF-8/alternate encoding

Defined handling

Old/future timestamp

Same failure policy

File truncate

No unexplained gap/duplication

Rename/create rotation

Continuous event IDs

Collector restart

Resume at known boundary

Multiline requires exact configuration, not default trust. Current Alloy documentation defines a first-line RE2 regular expression and defaults to 128 lines and a three-second maximum wait before flushing a block. If Promtail used different values, or an application regularly emits larger traces, set and test them explicitly.

Timestamp extraction is equally important. Without a timestamp stage, loki.process preserves the upstream timestamp; its documented failure and duplicate policies can also alter the resulting timestamp when explicit parsing is configured.

Test Pipeline Stages and Restart Semantics

Positions are where collector replacement turns into data recovery engineering. Grafana’s migration guide states that Alloy stores positions in a different location and tells users with persistent volumes to update their setup accordingly. loki.source.file stores offsets inside a component-specific data path under Alloy’s configured --storage.path, allowing Alloy to resume its own offsets after restart.

Do not assume legacy_positions_file imports Promtail positions. Current component documentation describes that option specifically for migration from Grafana Agent Static Mode to Alloy. For Promtail, prove whatever state-transfer strategy you choose in a lab.

A safe restart test is deterministic:

  1. Write marker IDs 0001–0100; wait until 0100 is queryable.

  2. Stop the collector and capture position state.

  3. Append 0101–0150; rotate the file; create the next file with 0151–0200.

  4. Restart Alloy with the intended storage volume.

  5. Reconcile IDs 0001–0200, duplicates, timestamps and order.

  6. Restart again and prove no replay appears unexpectedly.

Corrupt-position behavior must also be chosen knowingly. Alloy documents that restarting from the end reduces duplicates but can lose logs, while restarting from the beginning can resend the complete file. If a target disappears from loki.source.file, its position entry is removed; re-adding it later starts the file from the beginning.

Retain Promtail’s old position state until rollback and replay decisions are closed.

Positions gate: no production wave proceeds after an unexplained marker gap or duplicate burst in restart/rotation tests.

Migrate Authentication, TLS, Tenancy, and Secrets

Collector replacement changes an authenticated network client. Treat endpoint and identity parity as a separate workstream from parsing.

Create a credential contract:

Control

Promtail baseline

Alloy target

Proof

Loki endpoint

Exact scheme/host/path

Same approved endpoint

Test push/query

Authentication

Basic/bearer/OAuth mechanism

Equivalent supported method

Expected success/failure

Tenant

Header/static/derived

tenant_id or processing rule

Query correct tenant

Trust

CA bundle/system store

Explicit equivalent

Valid chain succeeds

mTLS

Client cert/key

Correct cert/key pair

Server accepts identity

Proxy

URL/env/no-proxy

Equivalent routing

Trace/connectivity test

Secret rotation

Existing mechanism

Kubernetes/VM secret delivery

Rotate without outage

RBAC

Discovery permissions

Least required permissions

Denied unneeded access

loki.write supports TLS configuration, proxy behavior and tenant IDs. The current documentation says that when no tenant_id is configured, it assumes single-tenant operation and does not send X-Scope-OrgID. Multiple endpoint blocks fan out sequentially, so a bottlenecked endpoint can affect the others.

That means “HTTP 204 from Loki” is not sufficient tenancy evidence. Inject a unique marker for every tenant and prove it is visible in the intended tenant and absent from tenants where it does not belong. Where loki.process derives tenancy dynamically, validate every source value and the behavior for a missing or malformed source field.

For Transport Layer Security (TLS), test valid trust, expired/untrusted certificate behavior, server-name verification and mutual TLS (mTLS) where used. Grafana warns that insecure_skip_verify disables certificate-chain and hostname verification and recommends proper certificate authority configuration for production.

Do not bake tokens or private keys into generated Alloy configuration simply because conversion makes it convenient. Preserve your normal Kubernetes Secret, secret-management or VM credential-delivery boundaries.

Security gate: security/platform owners sign off tenant isolation, authentication failure behavior, TLS verification and secret rotation before a production dual-ship cohort is opened.

Deploy Alloy Across Kubernetes and VM Fleets

Version-pin the migration. For the August 29, 2026 validation baseline used in the worked tests below, pin Promtail 3.5.1 with deprecated Promtail chart 6.17.1, Alloy 1.19.2, Alloy Helm chart 1.12.1, Loki 3.7.7, Kubernetes 1.36.4 and containerd 2.3.4. The Promtail chart currently identifies appVersion: 3.5.1 and marks itself deprecated; Alloy 1.19.2 was released August 26, Alloy chart 1.12.1 on August 27, and Loki 3.7.7 on August 27. Kubernetes 1.36.4 was released August 11, while containerd 2.3.4 was released August 12.

Those are reproducibility pins, not a recommendation to downgrade a fleet. Substitute your production versions and rerun the matrix.

For Kubernetes, deploy the tested chart explicitly:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm upgrade --install alloy grafana/alloy \
  --namespace alloy \
  --create-namespace \
  --version 1.12.1 \
  -f values-prod.yaml

Grafana’s Helm repository documents that repository setup, while the release history identifies chart 1.12.1.

Choose topology by the actual source:

Pattern

Good fit

Main migration concern

DaemonSet

Node-local Kubernetes files/system logs

Host mounts, every node pool, positions

Kubernetes API collection

Pod logs without direct host-file privilege

Discovery scale, API permissions

Sidecar

Exceptional per-workload isolation

Cost, duplicate configuration, lifecycle coupling

Central collector

Network/journal aggregation scenarios

Failure domain and bandwidth

VM systemd service

Host files/journal

Persistent storage path, service user, upgrades

Qualify DaemonSet Permissions, Paths, and Scheduling

A DaemonSet is not “qualified” because desired replicas equal ready replicas. Check service account and role-based access control (RBAC), security context, required hostPath mounts, read permissions, node selectors, tolerations, architectures and every node pool that should run the collector.

For API-based pod collection, Grafana’s example limits discovery to the local node when Alloy runs as a DaemonSet, reducing unnecessary resource use. For file collection, prove the host’s actual runtime paths and rotations against the mounted paths inside Alloy.

Set CPU and memory requests from measured canary usage, then test limits under peak parsing and recovery load. Logging is one of the last systems you want evicted during an incident.

On VMs, run Alloy under a dedicated service identity, persist its storage path on the intended filesystem, version-control configuration delivery and make upgrade/rollback atomic. Journal collection may require appropriate journal-reading groups; validate those permissions under the service account.

Deployment gate: one node from every scheduling/runtime class and one VM from every host class must pass source, position, credential and restart tests.

Dual Ship Without Hiding Duplicates or Loss

Grafana Alloy dual shipping should answer a comparison question, not simply send twice as much data.

The cleanest design is a bounded cohort with destination isolation: Promtail continues writing to the production tenant while Alloy writes the same workload to a shadow tenant or test Loki destination. That prevents backend deduplication or stream interactions from obscuring which collector produced which event.

Dual-ship mode

Comparison quality

Cost/risk

Recommendation

Shadow tenant/destination

Excellent

Extra ingestion/storage

Preferred for strict proof

Same Loki with temporary collector label

Good

Extra stream cardinality

Useful when shadow tenant unavailable

Exact same stream

Weak

Duplicates/order interactions

Avoid as primary parity evidence

A temporary collector="promtail" versus collector="alloy" label can separate evidence, but remove it deliberately after testing. Never attach event_id as an indexed label; it is deliberately high-cardinality.

Start with a cohort small enough that you can inspect it deeply: one controlled namespace or a few nodes, then include rotation, restart and load events. Compare explicit windows whose start and end are marker IDs rather than wall-clock guesses.

For every window calculate:

expected IDs – observed Alloy IDs = loss candidates

observed Alloy IDs occurring >1 time = duplicate candidates

Then compare Promtail and Alloy field-by-field: timestamp, original line, indexed labels, structured metadata, tenant and source identity.

If an application has no event ID, an offline fingerprint can approximate continuity, but distinguish true duplicate lines from legitimate repeated application messages.

Also isolate fan-out risk. Alloy documentation warns that multiple endpoints within one loki.write component are serviced in succession and a bottlenecked endpoint may impact the others. A “shadow” endpoint that is slow must not accidentally invalidate the production path; load-test that topology or use separate paths.

Dual-ship gate: promotion requires zero unexplained lost marker IDs, a documented duplicate rate, equivalent tenant routing and passing incident queries, not merely matching total bytes.

Validate Ordering, Cardinality, and Query Equivalence

Now run the failure that ingestion graphs cannot detect.

Use this synthetic acceptance workload (not a claimed production benchmark):

Component

Pinned test version

Kubernetes

1.36.4

container runtime

containerd 2.3.4

old collector

Promtail 3.5.1

old Helm chart

Promtail 6.17.1

new collector

Alloy 1.19.2

Alloy Helm chart

1.12.1

backend

Loki 3.7.7

workload

checkout-api, namespace payments, cluster prod-eu-1

Generate exactly 60,000 deterministic JSON events over ten minutes, each carrying event_id, monotonic seq, application timestamp and an incident marker. Promtail sends the cohort to one comparison tenant; Alloy sends it to another.

The intended labels are:

cluster="prod-eu-1"
namespace="payments"
app="checkout-api"

But the migrated Alloy relabel configuration contains:

rule {
  action       = "replace"
  replacement  = "prod_eu_1"
  target_label = "cluster"
}

The results can look deceptively excellent:

Test

Promtail

Alloy

Naive verdict

Events

60,000

60,000

Pass

Event IDs unique

60,000

60,000

Pass

Payload bytes

Equivalent by construction

Equivalent

Pass

Parser success

60,000

60,000

Pass

cluster cardinality

1

1

Pass

Expected cluster value

prod-eu-1

prod_eu_1

Fail

Legacy incident query

60,000 eligible events

0

Fail

The operational query is:

sum(count_over_time(
  {cluster="prod-eu-1", namespace="payments", app="checkout-api"}[10m]
))

Alloy’s data exists, but the query returns no Alloy events. Searching with the unintended label value proves ingestion occurred:

sum(count_over_time(
  {cluster="prod_eu_1", namespace="payments", app="checkout-api"}[10m]
))

That is why cardinality counts alone are insufficient: “one cluster value” remained “one cluster value,” yet its domain changed.

Fix the relabel rule, rerun the entire window, and require the old query to work unchanged unless the query change itself is an explicitly approved migration deliverable.

Grafana’s Loki documentation recommends low-cardinality indexed labels and structured metadata for high-cardinality values, and specifically notes that labels such as Kubernetes pod identity can create cardinality concerns. Apply that guidance, but do not combine taxonomy cleanup with parity validation unless you are prepared to update and test every consumer.

Ordering needs its own acceptance rule. Compare seq per source stream. Flag gaps, duplicates and inversions independently. Equal counts can conceal both one missing event and one duplicate event.

Finally, execute the frozen dashboards and alerts. A migration has query equivalence only when incident selectors, parsed-field filters, structured-metadata access, alert thresholds and tenant context all behave as intended.

Query gate: exact label-value domains and tier-one incident workflows must pass, even when volume, cardinality count and collector health already pass.

Tune Queues, Backpressure, CPU, Memory, and Storage

Log collectors are easy to benchmark while Loki is healthy and dangerously under-tested when Loki is slow.

Load-test at steady state, expected peak and recovery peak. Capture Alloy CPU, resident memory, source read rate, position lag, network throughput, retries, send latency and drop counters. loki.source.file exposes active-file, total-file-byte, read-byte and read-line metrics; loki.write exposes retries, sent bytes, dropped bytes and dropped entries.

Use a failure matrix:

Fault

Evidence required

Loki returns HTTP 429

Retries occur; backlog eventually drains

Loki 5xx/unreachable

No unexplained loss; recovery rate measured

DNS/proxy failure

Failure visible in monitoring; source backlog bounded

TLS rejection

Explicit error; no fallback to insecure connection

Network throttling

CPU/memory remain controlled

Collector restart during outage

Position/durability behavior understood

Log burst

No silent queue overflow or rotation loss

Be precise about current Alloy stability levels. loki.write has an optional experimental queue_config; its documented default when enabled blocks on overflow and uses a 10 MiB capacity. The optional write-ahead log (WAL) is also experimental and disabled by default, requiring Alloy’s experimental stability level. Do not quietly enable either feature in a migration and describe it as ordinary stable durability.

Model Outage Buffers and Ingestion Recovery

The useful capacity question is not “how much memory should Alloy get?” It is “how many minutes of unavailable Loki can the complete system survive before source data rotates away?”

For a hypothetical source generating 20 MiB/s with a 15-minute required outage tolerance and 25% safety margin:

required retained data
= 20 MiB/s × 900 s × 1.25
= 22,500 MiB
≈ 22 GiB

That does not mean provision a 22 GiB in-memory queue. In a file-tail architecture, source-file retention may provide much of the recoverable backlog while downstream backpressure slows reading. Whether that works in your exact topology is an engineering inference that must be fault-tested, especially across rotations.

Measure two rates: arrival during outage and drain after recovery. If production emits 100 MiB/s and the recovered path can only sustain 105 MiB/s, a large backlog may take an unacceptable time to clear. Capacity planning must therefore include recovery headroom, not merely normal throughput.

Set promotion thresholds such as zero increase in loki_write_dropped_entries_total, bounded position lag, no out-of-memory termination, and a tested backlog-clearance objective. The exact thresholds are yours; evidence must show they are achievable.

Resilience gate: a controlled Loki outage and restart must complete without unexplained marker loss and with backlog recovery inside the documented operational objective.

Cut Over with Evidence and Decommission Gates

Cutover should be a promotion of already-proved cohorts, not the first moment Alloy receives serious production traffic.

Use a sign-off record like this:

Gate

Evidence owner

Required state

Inventory

Platform

All sources/owners classified

Pipeline parity

Platform + service owner

Golden corpus passes

Position/restart

Platform

Rotation/restart reconciliation passes

Security/tenancy

Security/platform

Auth, TLS, tenant isolation pass

Event continuity

Observability

No unexplained marker loss

Queries

On-call/service owners

Tier-one incident queries pass

Alerts/dashboards

Monitoring owners

Expected firing/rendering behavior

Cardinality

Loki owner

Within approved stream budget

Resilience

Platform

Outage/recovery test passes

Rollback

Incident/change owner

Rehearsal completed

A reasonable operational sequence is canary → small production cohort → larger cohort → full fleet, with a defined soak at every boundary. The percentages and soak periods should come from your risk tolerance and traffic diversity rather than an invented universal rule.

At each wave, compare both rate-level signals and event-level markers. Watch label-value populations as carefully as label cardinality.

Once Alloy owns 100% of intended traffic, stop Promtail shipping in a reversible manner before deleting it. Keep manifests, package/image identifiers, configuration checksums and relevant position-state artifacts through the agreed rollback hold. Retain secret material only according to your security policy; preserving rollback does not mean preserving obsolete credentials indefinitely.

Do not destroy the old collector because “no one complained for 24 hours.” Silent query incompatibility often appears only when the next rare incident exercises a label, multiline parser or tenant path that normal dashboards do not.

Decommission rule: disabling Promtail and deleting Promtail are separate change records. Deletion requires the full evidence packet plus rollback/replay closure.

Rehearse Rollback, Position Recovery, and Replay

A Promtail rollback plan has four separate operations. Conflating them is how a rollback duplicates an entire retained file or skips the very interval you are trying to recover.

Rollback type

Action

Primary danger

Deployment rollback

Restore/start Promtail package, pod or DaemonSet

Old unsupported agent resumes indefinitely

Configuration rollback

Revert Alloy to last-known-good config

Current position state may not match old target graph

Position rollback

Restore selected offset state

Stale offset duplicates; advanced offset skips

Data replay

Re-read retained source evidence

Duplicates/order differences and expired source data

Rehearse the choice before cutover. Suppose Alloy is deployed successfully but a namespace label is wrong. If all lines reached a shadow tenant, configuration rollback plus query repair may be enough; replay might be unnecessary. If Alloy skipped files because discovery was wrong and the files still exist, replay may recover them. If the files rotated away and no other durable copy exists, the gap cannot be reconstructed by restarting Promtail.

Positions deserve a written decision record. Alloy stores its own component position data beneath its storage path and removes a file’s position entry if that target is removed; reintroducing the target can therefore cause rereading from the beginning. Promtail’s prior positions should not simply be copied into that scheme based on filename resemblance.

For every rollback exercise, record the time boundary:

last definitely confirmed Promtail event
first definitely confirmed Alloy event
last definitely confirmed Alloy event
first event after rollback

Reconcile those boundaries with marker IDs. Decide whether duplicate events are operationally acceptable and how analysts will recognize them.

Because Promtail is EOL, rollback is an emergency continuity mechanism, not a strategic destination. The rollback plan should therefore contain an explicit time-bounded route back to Alloy remediation.

Rollback gate: the team must demonstrate at least one rollback/restart exercise and document which gaps are replayable, which create duplicates, and which would be irreversible.

Execute the Thirty-Day Promtail-to-Alloy Migration Plan

The calendar is a coordination framework, not a substitute for gates. A failed test delays promotion even when the schedule says “production.”

Days

Work

Exit evidence

1–4

Inventory sources, owners, runtimes, configs, labels, positions, tenants

Complete inventory and risk register

5–7

Freeze volumes, label domains, query/alert/dashboard baselines

Signed baseline pack

8–10

Run conversion, capture report/warnings, choose conversion/native/hybrid

Reviewed Alloy configuration

11–14

Lab parser, multiline, timestamp, file rotation, position, auth/TLS tests

Golden corpus and restart suite pass

15–18

Dual ship non-production then bounded production canary

Event/query/tenant parity evidence

19–23

Expand production waves; compare markers, labels and resource pressure

Cohort gates pass

24–27

Full-fleet soak plus Loki outage/backpressure/restart exercise

Resilience and recovery evidence

28–30

Disable Promtail writes reversibly, archive evidence, begin decommission hold

Ownership transferred; rollback retained

During days 1–7, resist conversion work until query owners have frozen what “working logs” actually means. During days 8–14, use the generated configuration to accelerate implementation, but manually audit every relabel, position, parser, tenant and command-line dependency. The converter itself is currently Public Preview and its Promtail source-format documentation names v2.8.x, so its output belongs in the test pipeline rather than directly in production.

During days 15–23, expand by failure-domain diversity rather than only percentage. Include different Kubernetes versions, node pools, operating systems, runtimes, VM images, high-volume namespaces and unusual parsers.

Days 24–27 should include the failure modes that ordinary canaries avoid: Loki throttling, endpoint unavailability, network impairment, restart during backlog and file rotation while the receiver is unavailable.

On days 28–30, “closure” should mean Promtail no longer owns collection, not necessarily that every rollback artifact has been destroyed. Keep deletion behind the decommission hold defined by your change policy.

Teams building the surrounding knowledge separately can use Refonte Learning’s free Grafana learning roadmap, guide to learning Prometheus and Grafana, and top observability tools for DevOps engineers as adjacent context. They do not replace this migration’s production evidence.

Final closure artifact: retain a signed migration packet containing source inventory, old/new checksums and versions, conversion report, parser fixtures, query results, label-domain comparison, duplicate/loss reconciliation, cardinality review, resource/outage tests, cutover timeline and rollback decisions.

A successful Promtail to Grafana Alloy migration has three proofs.

First is event continuity: the expected log evidence arrives with controlled timestamps, order, parsing, tenant routing and duplicate behavior. Second is query and label equivalence: the labels operators select on still have the right names and values, and the incident queries, alerts and dashboards that depend on them still work. Third is collector resilience: rotations, restarts, throttling and Loki outages have been exercised without unexplained loss.

Start by freezing the Promtail configuration, command-line arguments, position state and version. Build the source-and-owner inventory. Capture tier-one LogQL queries and label-value baselines. Then convert or redesign one bounded cohort, inject deterministic marker events and prove the entire evidence path before expanding it.

That is the difference between “Alloy is running” and Grafana Alloy Kubernetes logging that is safe to depend on during an incident.

For adjacent skills validation, Refonte Learning’s DevOps monitoring interview questions can remain separate from the production runbook. For practitioners strengthening the Linux, Git/GitHub, continuous integration and delivery, Docker/Kubernetes, Terraform, cloud, monitoring/logging and capstone foundations used around this work, the Refonte Learning DevOps Engineer Program currently describes a three-month program at 12–14 hours per week, with admissions information referencing bachelor’s or postgraduate studies and career directions including DevOps Engineer, Cloud Engineer and Site Reliability Engineer.

The page also names Oskar Eriksson with more than 10 years of experience and lists a Training Certificate and Certificate of Internship. It does not explicitly name Grafana Alloy, Promtail, Loki, LogQL, label migration or collector-conversion tooling, so product-specific coverage of this migration is not confirmed.