Canonical: https://solanashreds.xyz/docs/decode/pipeline

# From UDP packet to transaction: the decoding pipeline

Every shred decoder, in any language, runs the same nine stages. This page maps them: what each stage takes in, what it hands on, what it has to remember, and how it fails.

The Solana shred decoding pipeline has nine stages: receive each datagram, parse its header, group shreds by slot and FEC set, recover lost data shreds from coding shreds, join data runs into block components, read the entries, parse each transaction as legacy, v0 or v1, resolve v0 lookup tables, then filter and emit.

Each stage hands its output to the next. Only the first five deal with shreds; from the block component on, you are working with ordinary Solana data structures.

## Stage by stage

| Stage | Input | Output | State it keeps | How it fails |
| --- | --- | --- | --- | --- |
| Receive | UDP datagrams | raw bytes | a socket buffer | the kernel drops packets when you read too slowly |
| Parse header | one datagram | slot, index, FEC set index, type, flags | none | non-shred or truncated packets; drop them |
| Group | parsed shreds | shreds keyed by slot, FEC set and index | every incomplete slot | memory grows if old slots are never released |
| Recover | an FEC set with gaps | the missing data shreds | the same sets | too few shreds survived to rebuild the set |
| Assemble | consecutive data shreds | one run of bytes per component | which runs are already emitted | a gap that recovery could not fill holds the run back |
| Block component | one run | a header marker, an entry batch or a footer | none | a corrupt run fails to deserialize |
| Transactions | an entry batch | signed transactions in block order | none | a parse bug misplaces every later transaction in the entry |
| Resolve lookups | v0 transactions | full account lists | a lookup table cache | an unknown or freshly extended table |
| Filter and emit | transactions | records for your application | dedupe memory | duplicates, and records that later turn out to have failed |

## What deshredding means

Deshredding is joining the data shreds of one run back into the bytes the leader cut up. A leader shreds each block component separately and flags the last data shred of each with `DATA_COMPLETE_SHRED`. Every data shred's payload holds 88 bytes of headers followed by data, and its `size` field says where the data ends. Concatenate the data from consecutive indexes up to and including a shred with that flag, and you have exactly one serialized component.

Two facts make this stage easier than it looks. A run's start is always known: index 0, or the index right after a shred flagged complete. So a missing shred costs you only the component it belongs to. And `LAST_IN_SLOT` marks the final data shred a leader sends for a slot, so seeing it tells you no further data is coming for that slot.

## The loop in pseudocode

```text
on datagram(bytes):
    shred = parse_header(bytes)            # variant byte at offset 64 says data or coding
    if shred is None: return               # not a shred
    slot = slots[shred.slot]               # create on first sight, drop when too old
    slot.add(shred)                        # ignore an index already held
    slot.try_recover(shred.fec_set_index)  # once the set holds enough shreds
    for run in slot.complete_runs():       # each run ends at DATA_COMPLETE
        component = deserialize(run)
        if component is entry batch:
            for entry in component.entries:
                for tx in entry.transactions:          # parse to find each length
                    accounts = resolve_lookups(tx)     # v0 only
                    if wanted(tx, accounts): emit(slot, tx)
```

The language pages turn this into running code. Each one is tested on fixtures made by Agave's own shredder from real mainnet transactions.

## Where to cut: per component or per slot

You can emit as soon as one entry batch is whole, or hold everything until the slot is complete.

Emitting per component is the natural default for a shred pipeline. A leader shreds and broadcasts a block in several components while it is still producing it, so the first batch of transactions is decodable long before the slot ends. Your consumer has to accept that a slot arrives in pieces.

Waiting for the slot gives you a whole block at once, in order, which is simpler for anything that writes blocks to storage. It also gives up much of the reason to read shreds. A common design does both: act on components, then write the slot when `LAST_IN_SLOT` and every earlier run have arrived. [Ordering inside a slot](https://solanashreds.xyz/blog/order-inside-a-slot) looks at what order you can rely on.

## What goes wrong between stages

**Duplicates.** The same shred can arrive twice. Key data shreds by slot and index and ignore repeats, then dedupe transactions again by signature before you emit, because the same transaction can also appear in more than one place.

**A missing FEC set.** If too few shreds of a set survive, recovery cannot rebuild it, and the component it belongs to is lost. Later components in the slot still decode, because their start index is known.

**A slot that never completes.** Leaders can abandon a block, and a slot can be skipped. Never wait forever: drop slots that fall a fixed distance behind the newest one you have seen.

**Results you do not have.** Every record you emit describes a request, not an outcome. Mark it provisional and settle it later with [reconciliation after confirmation](https://solanashreds.xyz/docs/pipeline/reconcile).

## Pick an implementation

| Language | Page | Notes |
| --- | --- | --- |
| Rust | [the Rust implementation](https://solanashreds.xyz/docs/decode/rust) | recovers lost data shreds from coding shreds |
| TypeScript | [the TypeScript version](https://solanashreds.xyz/docs/decode/typescript) | Node 24, @solana/kit 8.6.0 |
| Python | [a Python decoder](https://solanashreds.xyz/docs/decode/python) | standard library only |
| Go | [a Go decoder](https://solanashreds.xyz/docs/decode/go) | cross-checked against solana-go |

After the transactions stage, two recipes plug in: [lookup-table resolution](https://solanashreds.xyz/docs/decode/resolve-lookup-tables) and [filtering by program](https://solanashreds.xyz/docs/decode/filter-by-program). If one transaction carries several programs, [one transaction, many instructions](https://solanashreds.xyz/blog/one-transaction-many-instructions) explains how to split it.

## What the pipeline cannot produce

The pipeline ends with transactions as they were requested. It cannot tell you whether they succeeded, what they logged or which programs they called through CPIs. It also cannot fill a gap: a component you lost and could not recover stays lost. [Why packets are not yet data](https://solanashreds.xyz/blog/packets-are-not-data) covers what you still have to add, and [what is inside a Solana transaction](https://solanashreds.xyz/learn/solana-transaction-structure) explains the format the pipeline ends with.

## Next

Build the first implementation with [the Rust decoder](https://solanashreds.xyz/docs/decode/rust), or [get a feed to run it on](https://solanashreds.xyz/pricing). Resolving v0 accounts is covered in [address lookup tables](https://solanashreds.xyz/learn/address-lookup-tables).
