Operations · November 18, 2026 · 8 min read

Observability for decoders: the metrics that catch silent failure

A decoder rarely crashes. It keeps running while it drops a stage's output, misreads a program after an upgrade, or falls a few slots behind. These are the metrics that notice.

For Solana pipeline monitoring, five decoder metrics catch most silent failures: slot lag between receiving and writing, FEC sets that could not be recovered, lookup tables that could not be resolved, instructions with unknown discriminators, and write lag. Each fails quietly in its own stage, so measure per stage and alert on rates, not single events.

Why decoders fail quietly#

Receivers and decoders are built to keep going. A missing packet becomes an unrecoverable set and the decoder moves on. A program upgrade changes a layout and the decoder reads nonsense numbers or skips instructions it no longer recognises. A database slows down and rows arrive a few slots late. Nothing crashes, the process stays green, and the data is wrong for hours.

The fix is to count what each stage gives up on, and to measure how far behind each stage runs. Those two kinds of numbers, losses and lag, cover almost every failure that does not end in a crash.

Metrics per stage#

StageMetricTypeWhat a rise means
Receivedatagrams received, socket dropscountersPackets arriving; the kernel discarding them because the reader is slow
RecoveryFEC sets completed, FEC sets unrecoverablecountersLoss beyond what coding shreds can repair
Entriesslots completed, slots abandonedcountersSlots you gave up on, from loss or skips
Transactionstransactions parsed, parse errorscountersMalformed input, or a format your parser does not know
Lookupslookups resolved, lookups unresolved, table fetch timecounters and histogramCold cache, missing tables, or a slow RPC
Programsinstructions decoded, unknown discriminators per programcountersA program upgrade or an IDL your decoder does not have
Writesrows written, write time, write lag in slotscounters, histogram, gaugeThe store falling behind
Whole pipelineslot laggaugeThe decoder falling behind the network

Slot lag, defined so it means something#

"Lag" is only useful if both ends are well defined. Two versions are worth keeping:

  • Internal lag: the highest slot your receiver has seen data for, minus the highest slot your writer has stored. This needs no outside calls and shows whether your own stages keep up.
  • External lag: the cluster's current slot from the getSlot RPC method, minus the highest slot you stored. It catches the case where the receiver itself stopped getting packets, which internal lag cannot see.

Report both in slots rather than seconds. Slot times change with network upgrades, and a count of slots stays comparable across them.

Alert rules that do not page you every night#

  • Alert on rates over a window, not on single events. One unrecoverable FEC set is normal on a lossy day; a sustained rise is not.
  • Alert on ratios, such as unknown discriminators divided by instructions decoded for one program. A ratio jumping from near zero is the clearest sign of an upgrade; IDL drift covers what to do next.
  • Alert when counters stop moving. Datagrams received flat at zero for a minute means delivery stopped, whatever every other metric says. The delivery check helps you find out where.
  • Page on lag, ticket on losses. Lag that keeps growing needs a human now. Rising losses usually need a fix this week.

A Prometheus layout#

Follow the Prometheus naming advice: a prefix for your pipeline, base units such as seconds in the name, _total on counters, and labels for things with few values, such as stage or program name. Never put signatures or account keys in labels; they create a new series per value.

metrics.txt
decoder_datagrams_received_total
decoder_socket_drops_total
decoder_fec_sets_total{result="complete"|"unrecoverable"}
decoder_slots_total{result="complete"|"abandoned"}
decoder_transactions_total{result="parsed"|"error"}
decoder_lookups_total{result="resolved"|"unresolved"}
decoder_lookup_fetch_seconds (histogram)
decoder_instructions_total{program="...", result="decoded"|"unknown_discriminator"}
decoder_write_seconds (histogram)
decoder_slot_lag{kind="internal"|"external"} (gauge, in slots)
alerts.yml
groups:
- name: decoder
rules:
- alert: DecoderFallingBehind
expr: decoder_slot_lag{kind="external"} > 50
for: 2m
- alert: DeliveryStopped
expr: rate(decoder_datagrams_received_total[1m]) == 0
for: 1m
- alert: UnknownDiscriminatorsRising
expr: |
sum by (program) (rate(decoder_instructions_total{result="unknown_discriminator"}[10m]))
/ sum by (program) (rate(decoder_instructions_total[10m])) > 0.05
for: 10m

The thresholds are placeholders to replace with values from your own baseline; start by recording a week without alerting and set them above the normal spread. Write batching with 200 ms slots explains why write lag behaves the way it does.

What metrics cannot catch#

A decoder that reads a field at the wrong offset can produce plausible numbers with every counter looking healthy. Metrics tell you a stage is losing or lagging; only tests tell you it is right. Keep regression tests on recorded transactions next to the dashboards, and look at the pipeline stages for where each counter belongs. Anchor discriminators explains why an unknown one appears after an upgrade.

What decoding costs in CPU reuses the same per stage timers for sizing. To instrument a pipeline from the first packet, start with the quickstart.

Sources#

Raw shreds on your own server

Ultra Low Latency Shreds is $200 and Standard Shreds is $100. You pay in crypto, tell us which IPv4 address and UDP port should receive the stream, and we wire it up by hand.

Read nextOrder inside a slot: what entry order tells you