Canonical: https://solanashreds.xyz/blog/observability-for-decoders

# Observability for decoders: the metrics that catch silent failure

A decoder rarely crashes. It keeps running while it drops a stage's output, misreads a program after an upgrade, or falls a few slots behind. These are the metrics that notice.

For Solana pipeline monitoring, five decoder metrics catch most silent failures: slot lag between receiving and writing, FEC sets that could not be recovered, lookup tables that could not be resolved, instructions with unknown discriminators, and write lag. Each fails quietly in its own stage, so measure per stage and alert on rates, not single events.

## Why decoders fail quietly

Receivers and decoders are built to keep going. A missing packet becomes an unrecoverable set and the decoder moves on. A program upgrade changes a layout and the decoder reads nonsense numbers or skips instructions it no longer recognises. A database slows down and rows arrive a few slots late. Nothing crashes, the process stays green, and the data is wrong for hours.

The fix is to count what each stage gives up on, and to measure how far behind each stage runs. Those two kinds of numbers, losses and lag, cover almost every failure that does not end in a crash.

## Metrics per stage

| Stage | Metric | Type | What a rise means |
|---|---|---|---|
| Receive | datagrams received, socket drops | counters | Packets arriving; the kernel discarding them because the reader is slow |
| Recovery | FEC sets completed, FEC sets unrecoverable | counters | Loss beyond what coding shreds can repair |
| Entries | slots completed, slots abandoned | counters | Slots you gave up on, from loss or skips |
| Transactions | transactions parsed, parse errors | counters | Malformed input, or a format your parser does not know |
| Lookups | lookups resolved, lookups unresolved, table fetch time | counters and histogram | Cold cache, missing tables, or a slow RPC |
| Programs | instructions decoded, unknown discriminators per program | counters | A program upgrade or an IDL your decoder does not have |
| Writes | rows written, write time, write lag in slots | counters, histogram, gauge | The store falling behind |
| Whole pipeline | slot lag | gauge | The decoder falling behind the network |

## Slot lag, defined so it means something

"Lag" is only useful if both ends are well defined. Two versions are worth keeping:

- **Internal lag:** the highest slot your receiver has seen data for, minus the highest slot your writer has stored. This needs no outside calls and shows whether your own stages keep up.
- **External lag:** the cluster's current slot from the getSlot RPC method, minus the highest slot you stored. It catches the case where the receiver itself stopped getting packets, which internal lag cannot see.

Report both in slots rather than seconds. Slot times change with network upgrades, and a count of slots stays comparable across them.

## Alert rules that do not page you every night

- **Alert on rates over a window,** not on single events. One unrecoverable FEC set is normal on a lossy day; a sustained rise is not.
- **Alert on ratios,** such as unknown discriminators divided by instructions decoded for one program. A ratio jumping from near zero is the clearest sign of an upgrade; [IDL drift](https://solanashreds.xyz/blog/idl-drift) covers what to do next.
- **Alert when counters stop moving.** Datagrams received flat at zero for a minute means delivery stopped, whatever every other metric says. The [delivery check](https://solanashreds.xyz/docs/start/verify-delivery) helps you find out where.
- **Page on lag, ticket on losses.** Lag that keeps growing needs a human now. Rising losses usually need a fix this week.

## A Prometheus layout

Follow the Prometheus naming advice: a prefix for your pipeline, base units such as seconds in the name, `_total` on counters, and labels for things with few values, such as stage or program name. Never put signatures or account keys in labels; they create a new series per value.

```text file="metrics.txt"
decoder_datagrams_received_total
decoder_socket_drops_total
decoder_fec_sets_total{result="complete"|"unrecoverable"}
decoder_slots_total{result="complete"|"abandoned"}
decoder_transactions_total{result="parsed"|"error"}
decoder_lookups_total{result="resolved"|"unresolved"}
decoder_lookup_fetch_seconds            (histogram)
decoder_instructions_total{program="...", result="decoded"|"unknown_discriminator"}
decoder_write_seconds                   (histogram)
decoder_slot_lag{kind="internal"|"external"}   (gauge, in slots)
```

```yaml file="alerts.yml"
groups:
  - name: decoder
    rules:
      - alert: DecoderFallingBehind
        expr: decoder_slot_lag{kind="external"} > 50
        for: 2m
      - alert: DeliveryStopped
        expr: rate(decoder_datagrams_received_total[1m]) == 0
        for: 1m
      - alert: UnknownDiscriminatorsRising
        expr: |
          sum by (program) (rate(decoder_instructions_total{result="unknown_discriminator"}[10m]))
          / sum by (program) (rate(decoder_instructions_total[10m])) > 0.05
        for: 10m
```

The thresholds are placeholders to replace with values from your own baseline; start by recording a week without alerting and set them above the normal spread. [Write batching with 200 ms slots](https://solanashreds.xyz/blog/200ms-slots-and-your-database) explains why write lag behaves the way it does.

## What metrics cannot catch

A decoder that reads a field at the wrong offset can produce plausible numbers with every counter looking healthy. Metrics tell you a stage is losing or lagging; only tests tell you it is right. Keep [regression tests](https://solanashreds.xyz/blog/regression-tests-for-decoders) on recorded transactions next to the dashboards, and look at the [pipeline stages](https://solanashreds.xyz/docs/decode/pipeline) for where each counter belongs. [Anchor discriminators](https://solanashreds.xyz/learn/anchor-discriminators) explains why an unknown one appears after an upgrade.

## Related reading

[What decoding costs in CPU](https://solanashreds.xyz/blog/what-decoding-costs-in-cpu) reuses the same per stage timers for sizing. To instrument a pipeline from the first packet, start with the [quickstart](https://solanashreds.xyz/docs/quickstart).
