Why Averaging Instance p95s Misrepresents Service Latency

Two dashboards can display a single service p95 and still represent different request populations. Consider a synthetic two-instance service: instance A handles 990 requests at 100 ms, while instance B handles 10 requests at 1,000 ms. The combined population contains 1,000 observations, 990 at 0.1 seconds and 10 at 1.0 second. Under the nearest-rank convention, the combined p95 is 0.1 seconds. Averaging the two instance p95 values gives 0.55 seconds; weighting those p95 values by request count gives 0.109 seconds. Neither operation computes the percentile of the combined request distribution.

Before anyone changes capacity, alert thresholds, or incident policy based on a service p95, the metric pipeline needs an evidence contract. This playbook follows one documented HTTP request-duration population from instrumentation through exporter schema, per-series counters, PromQL aggregation, recording rules, and the dashboard. The examples are synthetic and the commands are proposed acceptance checks. The final outcome can be accept, reject, or hold; a plausible number alone is not sufficient evidence.

Define what one latency observation represents

Start with a metric contract. In this fixture, one observation is the server-side duration, in seconds, of a completed GET request to the orders service route /items that ended with an HTTP 2xx response. The timer starts when the service accepts the request and stops when response handling completes. The Prometheus instrumentation guidance supports explicit, consistent definitions for request counts, errors, and latency, but the exact inclusion rules remain a local engineering decision.

Retries count as separate server attempts because the server handled each attempt. Rejected requests, aborted requests, and non-2xx responses are excluded from this percentile and must remain observable in separate counters or intentionally labeled populations. Client-observed latency is a different measurement point because it can include network, queueing, and caller-side effects; it must not be averaged into the server-side p95.

The orders-service team owns the instrumentation code and histogram configuration. The observability-platform team owns scrape configuration, relabeling, recording rules, and dashboards. The service-performance reviewer approves the population contract. If any of those owners cannot identify what one sample represents, the percentile remains on hold.

This contract makes the percentile auditable. Readers who need the surrounding platform context can review the broader monitoring and logging stack, but tool selection does not replace a request-population definition.

Inventory the exporters, query engine and rule versions

Record the exact lab and deployed versions before reviewing a query. The manifest below is a proposed synthetic fixture for this article, not evidence that a particular production environment already uses these versions. Retain the original exposition, scrape configuration, relabeling, rule files, and dashboard JSON beside the review record so later changes can be compared against the same inputs.

A successful syntax check proves only that a rule file is parseable. The Prometheus recording-rule documentation describes rule files and syntax checking, while semantic acceptance requires deliberate input series and expected outputs. Preserve the current dashboard query before changing it; identical metric names do not prove identical bucket layouts, units, labels, or inclusion rules.

The inventory also establishes ownership for every transformation. A service team can change buckets, a platform team can drop labels, and a dashboard can add an average after the recording rule. Each change can alter the population even when the panel title stays the same.

For a broader view of monitoring-tool roles and operational ownership, use that material as organizational context. The acceptance evidence here still depends on the pinned exporter, query, and consumer versions.

Build an unequal-traffic counterexample

Use invented teaching data before touching a production rule. Instance A handles 990 requests, each lasting 0.1 seconds. Instance B handles 10 requests, each lasting 1.0 second. The observations are server-side durations from the same route, method, status class, and unit. This narrow fixture keeps instrumentation boundaries and telemetry ownership separate from the arithmetic being tested.

Calculate the percentile from the combined raw observations

Define the raw-data oracle with the nearest-rank convention: sort N observations and select observation ceil(0.95 x N). With N = 1,000, the rank is 950. The first 990 ordered observations are 0.1 seconds, so the 950th observation is 0.1 seconds. The exact raw-event p95 is therefore 0.1 seconds. A library that uses a different interpolation convention must document it before its result is compared with this oracle.

• Total observations: 1,000.

• Sorted positions 1 through 990: 0.1 seconds.

• Sorted positions 991 through 1,000: 1.0 second.

• Nearest-rank p95 position: 950, so the exact value is 0.1 seconds.

Now calculate the shortcuts. The unweighted mean of the two instance p95 values is 0.55 seconds. A request-count weighted mean is (990 x 0.1 + 10 x 1.0) / 1,000 = 0.109 seconds. The weighted value is numerically closer, but it is still not a percentile. Percentiles are nonlinear order statistics; weighting already-computed percentiles does not reconstruct the combined distribution.

Synthetic arithmetic

unweighted mean = (0.1 + 1.0) / 2 = 0.55 seconds
weighted mean   = (990 x 0.1 + 10 x 1.0) / 1,000 = 0.109 seconds
raw combined p95 (nearest rank) = 0.1 seconds

Compare estimates only after defining bucket uncertainty

Represent the same observations with compatible classic buckets at 0.1, 0.5, 1.0, and +Inf seconds. Classic bucket counters are cumulative. After combining both instances, 990 observations are at or below 0.1, 990 are at or below 0.5, and all 1,000 are at or below 1.0 and +Inf.

The desired rank, 950, falls inside the first bucket. Under the classic-histogram interpolation used by histogram_quantile, the estimate is 0.1 x 950 / 990, or approximately 0.0959596 seconds. That is not an error in aggregation; it is the expected estimate from this bucket representation. The exact raw p95 remains 0.1 seconds, and the histogram estimate remains within the bucket interval that contains it.

The Prometheus histograms and summaries guidance distinguishes exact raw quantiles, classic-histogram estimates, and non-aggregatable summary quantiles. Current guidance favors native histograms where they are feasible, but this classic fixture remains useful because it exposes bucket and rollout failures in installed estates.

Aggregate distributions rather than instance quantiles

For classic histograms, calculate the per-series rate first, sum bucket rates across instances while retaining the le label, and only then apply histogram_quantile. The selectors below enforce the single population defined in the contract. The instance label is intentionally removed; service and route remain on the output.

Approved classic-histogram query shape

histogram_quantile(
  0.95,
  sum by (service, route, le) (
    rate(http_request_duration_seconds_bucket{
      job="orders",
      service="orders",
      route="/items",
      method="GET",
      status_class="2xx"
    }[5m])
  )
)

The rate function operates on each monotonically increasing bucket counter. The sum then combines counts from all selected instances at each compatible upper bound. Keeping le is mandatory because it identifies the classic bucket boundary. The final function estimates p95 from the combined cumulative distribution.

Rejected query: averages instance p95 estimates

avg by (service, route) (
  histogram_quantile(
    0.95,
    sum by (instance, service, route, le) (
      rate(http_request_duration_seconds_bucket{
        job="orders",
        service="orders",
        route="/items",
        method="GET",
        status_class="2xx"
      }[5m])
    )
  )
)

The rejected expression is syntactically plausible. It creates one histogram per instance, estimates each instance p95, and then averages those estimates. With the synthetic fixture, it trends toward the equal-weight 0.55-second result rather than the service distribution. A parser cannot tell that the request population was lost; the acceptance fixture must do that.

The Prometheus query-function documentation provides separate classic and native histogram forms. Validate the exact expression against the pinned server version; do not reuse a query across routes, units, or measurement points merely because the metric name matches.

Keep counter resets visible to the query

Histogram buckets and counts are counters, so a process restart can reset one instance to zero while another continues. The rate semantics documented by Prometheus require rate to be applied before aggregation so each source series can expose its own reset. This is a measurement requirement, not a query-optimization preference.

Correct order: detect resets per series, then aggregate

sum by (service, route) (
  rate(http_request_duration_seconds_count{
    job="orders",
    service="orders",
    route="/items",
    method="GET",
    status_class="2xx"
  }[5m])
)

In a controlled fixture, instance A should increase, reset to zero, and increase again while instance B remains monotonic. Evaluate the rule at a declared timestamp with enough samples in the five-minute range. The expected aggregate is the sum of the two reset-aware per-series rates. Record the input values and expected result in the test file rather than relying on a smooth dashboard line.

Rejected order: aggregate counters first, then apply rate to a subquery

rate(
  (sum by (service, route) (
    http_request_duration_seconds_count{
      job="orders",
      service="orders",
      route="/items",
      method="GET",
      status_class="2xx"
    }
  ))[5m:]
)

The rejected expression is valid PromQL syntax, but the aggregate can conceal an individual reset. If instance A drops while instance B increases, the combined series may not show the shape required for correct reset handling. Do not clamp the output or fill the gap merely to make the dashboard look continuous.

A reset test should reject the aggregate-first query and accept the rate-first query for the declared series. A separate coverage check must distinguish resets from missing scrapes and intentional scale-in. These conditions require different operational responses even when they all disturb a panel line.

Reject incompatible classic bucket populations

Classic histograms can produce a numerical result even when a rollout exposes different bucket boundaries under the same metric name. That number is not automatically a valid service percentile. The review must confirm that every boundary used by the combined rule has comparable population coverage across all intended instances and that the +Inf bucket reconciles with the corresponding count.

Consider a staged deployment in which instance A exposes 0.1, 0.5, 1.0, and +Inf, while instance B exposes 0.2, 1.0, and +Inf. At le="0.1", only A contributes. At le="0.2", only B contributes. Summing by le produces apparently valid series, but those series do not describe the same request population at every boundary.

At a shared boundary such as 1.0 or +Inf, both instances participate. At the lower boundaries, participation is partial. Interpolation can therefore look plausible while representing different subsets of traffic. Unless a reviewed migration design proves that the intended aggregate remains meaningful, the service p95 stays on hold.

Audit bucket boundaries as part of the metric schema

Treat the complete ordered bucket set, unit, label contract, and observation population as one versioned metric schema. Capture an instance-and-version matrix from raw exposition, not only from the aggregate query. Confirm that cumulative bucket values are nondecreasing, that +Inf is present, and that +Inf agrees with _count for the same selected population.

• Accept when every intended instance exposes the approved classic boundaries and units.

• Hold when boundaries differ, an instance is missing a required boundary, or population coverage cannot be reconciled.

• Reject a rule that silently combines incompatible schemas and presents the result as complete.

Rehearse a staged schema change

Do not mutate bucket boundaries in place under a single trusted output without a transition plan. A safer rollout uses a new metric name or a version label, parallel candidate recording rules, and explicit reconciliation before consumer dashboards move. The old and new outputs must remain distinguishable throughout the transition.

Parallel candidate rules for a schema transition

groups:
- name: orders_latency_v1
  rules:
  - record: service_route:http_request_duration_seconds:p95_5m_v1
    expr: |
      histogram_quantile(
        0.95,
        sum by (service, route, le) (
          rate(http_request_duration_seconds_bucket[5m])
        )
      )

- name: orders_latency_v2
  rules:
  - record: service_route:http_request_duration_seconds:p95_5m_v2
    expr: |
      histogram_quantile(
        0.95,
        sum by (service, route, le) (
          rate(http_request_duration_seconds_v2_bucket[5m])
        )
      )

The transition record should identify which application versions emit each schema, the coverage achieved by each rule, the comparison window, the approval owner, and the rollback trigger. Do not retire the old metric until every consumer has moved and the new population has passed the same reset, missing-data, and schema tests.

Qualify a native-histogram path separately

Current Prometheus histogram guidance favors native histograms where they are feasible. Native histograms use a different sample representation and do not require the classic le label in the aggregation expression. They still need pinned client, server, schema, and query support. The classic fixture remains an audit baseline, not a recommendation to avoid native histograms.

For a Go client, enable and document the selected native-histogram settings in the HistogramOpts used by the instrumentation, including the native bucket factor and any limits. Verify the actual scraped and ingested sample type on the pinned versions. Do not infer native support solely from a metric name or from the presence of count and sum series.

Documented native-histogram query shape

histogram_quantile(
  0.95,
  sum by (service, route) (
    rate(http_request_duration_seconds{
      job="orders",
      service="orders",
      route="/items",
      method="GET",
      status_class="2xx"
    }[5m])
  )
)

The native expression omits le because it aggregates histogram samples directly. Confirm the exact behavior in the pinned query-function documentation and test the deployed histogram schema. Standard exponential and custom-bucket native histograms do not have identical reconciliation rules; retain any query annotations and reject unsupported combinations.

If the native path has not been exercised with the pinned fixture, label it unqualified. Do not claim that a classic and native result must be numerically identical; their resolutions and interpolation behavior can differ. Compare each path with the same raw-event oracle and its own declared uncertainty.

Treat missing coverage as a measurement failure

A low p95 can be falsely reassuring when one instance or metric family is absent. PromQL aggregation normally uses the series that exist; it does not know which targets the service contract expected. The percentile must therefore be paired with an independently reviewed coverage signal based on discovery, up status, and observed request counters.

Compare the expected target set with the targets contributing usable bucket and count series. Also compare the aggregate +Inf bucket rate with the aggregate _count rate for the same labels. A quiet but incomplete result must not pass simply because its p95 is low.

Separate no requests from no usable observations

For true zero traffic, every expected target remains visible and its request counters are present, but the rate is zero. With no observations in the interval, a percentile is undefined; a dashboard should show no requests or unavailable, not 0 seconds. Operational evidence should include discovery state, up, and a zero aggregate request rate.

For missing observations, at least one expected target or selected metric series is absent, stale, or too sparse for the range calculation. The remaining instances may still return a numerical p95. That number represents a different population and stays on hold until coverage is restored or the population contract is intentionally revised.

Retain warnings and unexpected series shape

Capture query warnings and annotations returned by the Prometheus API, including information about histogram repairs or schema reconciliation where the pinned version emits them. The query-function documentation describes cases in which histogram functions can add annotations. Warnings are evidence, not a complete validator; some incompatible classic bucket populations can still return plausible values.

• Store the raw API response, including warnings and annotations, with the review record.

• Record unexpected output labels, duplicate series, absent series, and nonmonotonic cumulative buckets.

• Require an independent population-coverage and schema check even when the query returns no warning.

Distinguish a percentile from a latency-threshold objective

A p95 estimate answers a rank question: the estimated latency below which 95% of the selected observations fall. A threshold ratio answers a different question: what fraction of selected requests completed at or below a fixed boundary? Those measurements can move differently and must not be substituted for one another.

For a classic histogram with an approved 0.3-second bucket, compute the threshold ratio directly from that cumulative bucket and the matching count. Both numerator and denominator must use the same selectors, rate window, and aggregation labels.

Classic threshold ratio when 0.3 seconds is an approved bucket boundary

sum by (service, route) (
  rate(http_request_duration_seconds_bucket{
    job="orders",
    service="orders",
    route="/items",
    method="GET",
    status_class="2xx",
    le="0.3"
  }[5m])
)
/
sum by (service, route) (
  rate(http_request_duration_seconds_count{
    job="orders",
    service="orders",
    route="/items",
    method="GET",
    status_class="2xx"
  }[5m])
)

If the requested threshold is not an actual classic bucket boundary, an interpolated fraction has additional uncertainty. Keep the boundary and population review explicit. This section is a measurement clarification, not a complete SLO or error-budget design.

A visually lower p95 does not by itself prove that an application optimization succeeded. Perform API performance work after measurement validation and use logs, traces, load data, and threshold measures that answer the intended engineering question.

Turn the contract into promtool rule tests

Encode the approved population and aggregation order in a versioned recording rule. The file below is a proposed fixture; it is not a transcript of a successful production deployment. Substitute the repository paths and labels only after the local metric contract has been reviewed.

rules/latency.rules.yml

groups:
- name: orders_latency_v1
  interval: 1m
  rules:
  - record: service_route:http_request_duration_seconds:p95_5m
    expr: |
      histogram_quantile(
        0.95,
        sum by (service, route, le) (
          rate(http_request_duration_seconds_bucket{
            job="orders",
            service="orders",
            route="/items",
            method="GET",
            status_class="2xx"
          }[5m])
        )
      )
    labels:
      metric_definition: "orders_server_2xx_v1"

Use the syntax check described in the recording-rule documentation before running semantic tests. A successful parse remains only a syntax result. It cannot prove that the selectors cover the intended requests, that bucket schemas match, or that the expected numerical estimate is correct.

latency_test.yml: proposed unequal-traffic semantic fixture

rule_files:
- latency.rules.yml

evaluation_interval: 1m
fuzzy_compare: true

tests:
- name: unequal traffic preserves the combined population
  interval: 1m
  input_series:
  - series: 'http_request_duration_seconds_bucket{instance="A",job="orders",service="orders",route="/items",method="GET",status_class="2xx",le="0.1"}'
    values: '0+198x5'
  - series: 'http_request_duration_seconds_bucket{instance="A",job="orders",service="orders",route="/items",method="GET",status_class="2xx",le="0.5"}'
    values: '0+198x5'
  - series: 'http_request_duration_seconds_bucket{instance="A",job="orders",service="orders",route="/items",method="GET",status_class="2xx",le="1.0"}'
    values: '0+198x5'
  - series: 'http_request_duration_seconds_bucket{instance="A",job="orders",service="orders",route="/items",method="GET",status_class="2xx",le="+Inf"}'
    values: '0+198x5'
  - series: 'http_request_duration_seconds_bucket{instance="B",job="orders",service="orders",route="/items",method="GET",status_class="2xx",le="0.1"}'
    values: '0+0x5'
  - series: 'http_request_duration_seconds_bucket{instance="B",job="orders",service="orders",route="/items",method="GET",status_class="2xx",le="0.5"}'
    values: '0+0x5'
  - series: 'http_request_duration_seconds_bucket{instance="B",job="orders",service="orders",route="/items",method="GET",status_class="2xx",le="1.0"}'
    values: '0+2x5'
  - series: 'http_request_duration_seconds_bucket{instance="B",job="orders",service="orders",route="/items",method="GET",status_class="2xx",le="+Inf"}'
    values: '0+2x5'
  promql_expr_test:
  - expr: 'service_route:http_request_duration_seconds:p95_5m{metric_definition="orders_server_2xx_v1"}'
    eval_time: 5m
    exp_samples:
    - labels: 'service_route:http_request_duration_seconds:p95_5m{metric_definition="orders_server_2xx_v1",route="/items",service="orders"}'
      value: 0.09595959595959596

The series advance once per minute for five minutes. Instance A adds 198 observations per step, for 990 total; instance B adds 2 observations per step, for 10 total. The expected recording-rule value is the classic histogram estimate, approximately 0.0959596 seconds, not the exact raw-event value of 0.1 seconds. Until this file is executed with the pinned promtool binary, its status is proposed.

Deliberately failing semantic test

- name: wrong instance-average expression is rejected
  interval: 1m
  input_series:
  # Reuse the unequal-traffic fixture above.
  promql_expr_test:
  - expr: |
      avg by (service, route) (
        histogram_quantile(
          0.95,
          sum by (instance, service, route, le) (
            rate(http_request_duration_seconds_bucket[5m])
          )
        )
      )
    eval_time: 5m
    exp_samples:
    - labels: '{route="/items",service="orders"}'
      value: 0.09595959595959596

The wrong expression should not satisfy the expected service-level result. Keep this case as a regression guard rather than changing the expected value to make the test green. Missing-input and schema-mismatch cases also need an accompanying coverage or schema guard because a bare percentile expression may still return a plausible sample.

The Prometheus rule-unit-testing documentation defines the fixture structure and promql_expr_test expectations. Pin promtool, run syntax and semantic checks in CI, retain the output as evidence, and do not publish an invented passing transcript.

Compare raw queries, recording rules and dashboards

After the tests pass in the pinned environment, compare the raw PromQL expression, the recorded series, and the dashboard at aligned evaluation timestamps. A one-minute rule interval and a five-minute rate window can create apparent differences when an ad hoc query is evaluated between rule timestamps. Record the exact timestamp, lookback, step, and data-source version.

The dashboard must not average already-recorded p95 series. Inspect panel transformations, legend calculations, templating variables, and data-source expressions. A panel titled service p95 can still apply a client-side average or select only one instance.

Before consumer migration, create an evidence record containing the evaluation timestamp, Prometheus version, raw query value, recording-rule value, coverage state, dashboard query revision, owners, and execution status. This article does not claim that the comparison was executed.

If the raw and recorded values differ outside the expected evaluation alignment, hold consumer migration and trace the difference to a specific rule file, label set, time range, or server version. A measurement gap is not evidence that service latency actually changed.

Use complementary observability signals for diagnosing measurement gaps, but keep logs and traces as diagnostic evidence rather than substitutes for the histogram population contract.

Roll out the corrected rule with a clear hold path

Deploy the candidate rule beside the existing output. Version the new rule name or attach an explicit metric definition label so reviewers and consumers cannot confuse it with the old calculation. Preserve the prior measurement for comparison, but mark it as deprecated or invalid when it is known to average percentiles.

1. Publish the candidate rule under a distinct name or version label. Do not overwrite the current output in place.

2. Run the pinned syntax, semantic, coverage, reset, and schema checks. Store the command, tool version, fixture revision, and actual result.

3. Compare candidate and current outputs during a declared observation window. Investigate differences rather than assuming the lower value is better.

4. Obtain approval from the service instrumentation owner, observability-platform owner, and service-performance reviewer.

5. Migrate dashboards and alerts through a reviewed change. Confirm that no consumer re-aggregates the recorded p95.

6. Retire the old rule only after consumer inventory and rollback evidence are complete.

Rollback means stopping the candidate deployment or restoring consumer stability; it does not mean reclassifying a known misleading metric as trusted. Historical dashboard periods that cannot be recomputed from retained bucket data need a correction note rather than silent replacement.

The change owner records who approved each item, the tested version and time window, the evidence location, and the reason for any hold. Anyone responsible for the metric pipeline can stop the rollout when the population, schema, or consumer behavior is unresolved.

Build the operational foundations for metric reviews

Reliable metric reviews depend on disciplined Linux operations, source control, CI/CD, container platforms, infrastructure automation, cloud services, and monitoring practices. Teams need those foundations to version instrumentation, review rule changes, execute repeatable tests, and preserve deployment evidence.

The Refonte Learning DevOps Engineer Program page describes a three-month learning path at 12 to 14 hours per week covering Linux, Git and GitHub, CI/CD, Docker and Kubernetes, Terraform, cloud platforms, and monitoring and logging tools, including practical CI/CD and containerization work. This specialized histogram acceptance fixture is not presented here as a confirmed program module.

Use this compact skills-to-evidence checklist during a review:

• Version control: the metric contract, instrumentation, bucket schema, rule files, tests, and dashboard queries are revisioned together.

• Automation: syntax and semantic checks run in a controlled environment with pinned binaries.

• Platform operations: scrape discovery, target health, relabeling, and rule loading are observable and owned.

• Monitoring discipline: coverage and schema guards can block a misleading percentile from reaching consumers.

• Recovery: the rollout has a stop condition, a consumer rollback path, and a historical correction note when source data cannot be rebuilt.

Accept the measurement before acting on the number

A service p95 is eligible for acceptance only when the request-population contract, exporter schema, query identity, reset handling, semantic tests, coverage evidence, recording-rule output, and consumer migration record all agree for the exact version and time window under review. A green parser or a low dashboard value is not enough.

For the synthetic article fixture, reject the instance-p95 average. The classic sum-of-rates histogram query has a defined expected estimate and a proposed rule-test package, but production promotion remains on hold until the pinned tests and live coverage checks are executed and retained. The native-histogram path remains separately unqualified until its client, schema, ingestion, query, and consumer chain pass their own tests.

Record the final decision with the service, metric name, population version, histogram form, bucket or native schema, Prometheus and client versions, evaluation window, rule revision, dashboard revision, owners, and evidence links. Repeat the review whenever instrumentation, labels, buckets, software versions, or consumer queries change. Accept the measurement first; only then act on the number.