Decode

From UDP packet to transaction: the decoding pipeline

Every shred decoder, in any language, runs the same nine stages. This page maps them: what each stage takes in, what it hands on, what it has to remember, and how it fails.

The Solana shred decoding pipeline has nine stages: receive each datagram, parse its header, group shreds by slot and FEC set, recover lost data shreds from coding shreds, join data runs into block components, read the entries, parse each transaction as legacy, v0 or v1, resolve v0 lookup tables, then filter and emit.

Shred decoding pipelineNine stages from top to bottom: receive a UDP datagram, parse the shred header, group shreds by slot and FEC set, recover lost data shreds from coding shreds, assemble data runs that end at a DATA_COMPLETE flag, deserialize the block component, parse each transaction as legacy, v0 or v1, resolve lookup tables for v0 transactions, then filter and emit.Receiveone UDP datagram per shredParse headerslot, index, FEC set, flagsGroupper slot and FEC setRecoverrebuild lost data shredsAssembledata runs to DATA_COMPLETEBlock componentheader, entry batch, footerTransactionslegacy, v0 or v1Resolve lookupsv0 only, needs table stateFilter and emityour programs, your sink

Shred decoding pipeline

Each stage hands its output to the next. Only the first five deal with shreds; from the block component on, you are working with ordinary Solana data structures.

Stage by stage#

StageInputOutputState it keepsHow it fails
ReceiveUDP datagramsraw bytesa socket bufferthe kernel drops packets when you read too slowly
Parse headerone datagramslot, index, FEC set index, type, flagsnonenon-shred or truncated packets; drop them
Groupparsed shredsshreds keyed by slot, FEC set and indexevery incomplete slotmemory grows if old slots are never released
Recoveran FEC set with gapsthe missing data shredsthe same setstoo few shreds survived to rebuild the set
Assembleconsecutive data shredsone run of bytes per componentwhich runs are already emitteda gap that recovery could not fill holds the run back
Block componentone runa header marker, an entry batch or a footernonea corrupt run fails to deserialize
Transactionsan entry batchsigned transactions in block ordernonea parse bug misplaces every later transaction in the entry
Resolve lookupsv0 transactionsfull account listsa lookup table cachean unknown or freshly extended table
Filter and emittransactionsrecords for your applicationdedupe memoryduplicates, and records that later turn out to have failed

What deshredding means#

Deshredding is joining the data shreds of one run back into the bytes the leader cut up. A leader shreds each block component separately and flags the last data shred of each with DATA_COMPLETE_SHRED. Every data shred's payload holds 88 bytes of headers followed by data, and its size field says where the data ends. Concatenate the data from consecutive indexes up to and including a shred with that flag, and you have exactly one serialized component.

Two facts make this stage easier than it looks. A run's start is always known: index 0, or the index right after a shred flagged complete. So a missing shred costs you only the component it belongs to. And LAST_IN_SLOT marks the final data shred a leader sends for a slot, so seeing it tells you no further data is coming for that slot.

The loop in pseudocode#

Text
on datagram(bytes):
shred = parse_header(bytes) # variant byte at offset 64 says data or coding
if shred is None: return # not a shred
slot = slots[shred.slot] # create on first sight, drop when too old
slot.add(shred) # ignore an index already held
slot.try_recover(shred.fec_set_index) # once the set holds enough shreds
for run in slot.complete_runs(): # each run ends at DATA_COMPLETE
component = deserialize(run)
if component is entry batch:
for entry in component.entries:
for tx in entry.transactions: # parse to find each length
accounts = resolve_lookups(tx) # v0 only
if wanted(tx, accounts): emit(slot, tx)

The language pages turn this into running code. Each one is tested on fixtures made by Agave's own shredder from real mainnet transactions.

Where to cut: per component or per slot#

You can emit as soon as one entry batch is whole, or hold everything until the slot is complete.

Emitting per component is the natural default for a shred pipeline. A leader shreds and broadcasts a block in several components while it is still producing it, so the first batch of transactions is decodable long before the slot ends. Your consumer has to accept that a slot arrives in pieces.

Waiting for the slot gives you a whole block at once, in order, which is simpler for anything that writes blocks to storage. It also gives up much of the reason to read shreds. A common design does both: act on components, then write the slot when LAST_IN_SLOT and every earlier run have arrived. Ordering inside a slot looks at what order you can rely on.

What goes wrong between stages#

Duplicates. The same shred can arrive twice. Key data shreds by slot and index and ignore repeats, then dedupe transactions again by signature before you emit, because the same transaction can also appear in more than one place.

A missing FEC set. If too few shreds of a set survive, recovery cannot rebuild it, and the component it belongs to is lost. Later components in the slot still decode, because their start index is known.

A slot that never completes. Leaders can abandon a block, and a slot can be skipped. Never wait forever: drop slots that fall a fixed distance behind the newest one you have seen.

Results you do not have. Every record you emit describes a request, not an outcome. Mark it provisional and settle it later with reconciliation after confirmation.

Pick an implementation#

LanguagePageNotes
Rustthe Rust implementationrecovers lost data shreds from coding shreds
TypeScriptthe TypeScript versionNode 24, @solana/kit 8.6.0
Pythona Python decoderstandard library only
Goa Go decodercross-checked against solana-go

After the transactions stage, two recipes plug in: lookup-table resolution and filtering by program. If one transaction carries several programs, one transaction, many instructions explains how to split it.

What the pipeline cannot produce#

The pipeline ends with transactions as they were requested. It cannot tell you whether they succeeded, what they logged or which programs they called through CPIs. It also cannot fill a gap: a component you lost and could not recover stays lost. Why packets are not yet data covers what you still have to add, and what is inside a Solana transaction explains the format the pipeline ends with.

Next#

Build the first implementation with the Rust decoder, or get a feed to run it on. Resolving v0 accounts is covered in address lookup tables.

Sources#

Raw shreds on your own server

Ultra Low Latency Shreds is $200 and Standard Shreds is $100. You pay in crypto, tell us which IPv4 address and UDP port should receive the stream, and we wire it up by hand.