Decode

A Rust decoder: shreds in, transactions out

A complete Rust receiver and decoder: it reads shreds from a UDP port, rebuilds lost data shreds from coding shreds, and prints every transaction as a JSON line. Every line of code below was tested against Agave's own shredder output.

A Solana transaction decoder in Rust can rest on three main crates: solana-entry, solana-transaction and a Reed-Solomon crate. It receives shreds over UDP, rebuilds lost data shreds, deserializes each block component and prints every legacy, v0 or v1 transaction as a JSON line with slot, signature, version, static keys and program IDs. No RocksDB, no validator state.

Crates#

CrateVersionRepositoryWhat it does here
solana-entry4.4.1, feature agave-unstable-apianza-xyz/agaveBlockComponent, Entry and their deserialization
solana-transaction5.1.0anza-xyz/solana-sdkVersionedTransaction for legacy, v0 and v1
wincode0.6.3crates.iothe serialization format block components use
reed-solomon-erasure6.0.0crates.iorebuilds lost data shreds, as Agave does
serde_json1.0.154crates.ioprints the JSON lines

Built with Rust 1.98.1, the toolchain Agave 4.4.1 pins. solana-entry compiles to an empty crate unless you enable agave-unstable-api.

Why not solana-ledger#

solana-ledger holds Agave's shred types and the Shredder, so it looks like the obvious dependency. Two problems make it a poor fit for a receiver. From crates.io it only compiles with the forks Agave lists under [patch.crates-io] in its own Cargo.toml (rocksdb, librocksdb-sys, crossbeam-epoch and blstrs at fixed revisions), and it brings RocksDB with it. And its public recovery entry point expects a Bank, which only a validator has.

The shred header is small and fixed, so this decoder reads it by hand, uses reed-solomon-erasure for recovery the same way Agave does, and leaves the block format to solana-entry. The shred parsing libraries comparison weighs the alternatives.

Steps#

  1. Receive each UDP datagram and read the variant byte at offset 64: 0x9_ and 0xB_ are data shreds, 0x6_ and 0x7_ coding shreds, anything else is dropped.
  2. Read the slot, index and FEC set index from the common header, and for data shreds the flags and size field.
  3. File each shred under its slot and its erasure set, and skip an index you already have.
  4. When a set holds at least as many shreds as it has data shreds, rebuild the missing data shreds with Reed-Solomon.
  5. Once a run of data shreds reaches one flagged DATA_COMPLETE, glue their payloads together and hand the bytes to solana-entry.
  6. For an entry batch, print one JSON line per transaction; markers (block header and footer) carry no transactions and are skipped.

Cargo.toml#

Cargo.toml
[package]
name = "shred-receiver"
version = "0.1.0"
edition = "2024"
publish = false
# A receiver that does NOT need solana-ledger (no RocksDB, no Bank): headers are read by hand,
# lost data shreds are rebuilt from coding shreds with reed-solomon-erasure (the crate Agave uses),
# and block components are deserialized with solana-entry.
[dependencies]
solana-entry = { version = "=4.4.1", features = ["agave-unstable-api"] }
solana-transaction = "=5.1.0"
wincode = "=0.6.3"
reed-solomon-erasure = "=6.0.0"
serde_json = "=1.0.154"

src/main.rs#

src/main.rs
//! shred-receiver: UDP shreds -> (recover) -> block components -> transactions, printed as JSON lines.
//! shred-receiver <udp-port> or shred-receiver --file <fixture.shreds>
//! Offsets and sizes: Agave v4.4.1 ledger/src/shred.rs and shred/merkle.rs (SOLANASHREDS-TECH-FACTS-E4 P-01 to P-08).
use {
reed_solomon_erasure::galois_8::ReedSolomon,
solana_entry::block_component::BlockComponent,
std::collections::{BTreeMap, HashMap, HashSet},
};
const SIG: usize = 64;
const DATA_HEADERS: usize = 88;
const CODE_HEADERS: usize = 89;
const DATA_PAYLOAD: usize = 1203;
const CODE_PAYLOAD: usize = 1228;
const DATA_COMPLETE: u8 = 0x40;
struct Header {
slot: u64,
index: u32,
fec: u32,
is_data: bool,
proof: usize,
resigned: bool,
}
fn header(p: &[u8]) -> Option<Header> {
let v = *p.get(64)?;
let (is_data, resigned) = match v & 0xf0 {
0x90 => (true, false),
0xb0 => (true, true),
0x60 => (false, false),
0x70 => (false, true),
_ => return None,
};
let u32_at = |o: usize| u32::from_le_bytes(p[o..o + 4].try_into().unwrap());
if p.len() < if is_data { DATA_PAYLOAD } else { CODE_PAYLOAD } {
return None;
}
Some(Header {
slot: u64::from_le_bytes(p[65..73].try_into().unwrap()),
index: u32_at(73),
fec: u32_at(79),
is_data,
proof: (v & 0x0f) as usize,
resigned,
})
}
/// Bytes of a shred that are erasure coded: after the signature for data shreds, after the headers for
/// coding shreds, up to the chained Merkle root (always present now), proof and optional retransmitter signature.
fn shard_range(h: &Header) -> std::ops::Range<usize> {
let tail = 32 + 20 * h.proof + if h.resigned { 64 } else { 0 };
if h.is_data {
SIG..DATA_PAYLOAD - tail
} else {
CODE_HEADERS..CODE_PAYLOAD - tail
}
}
#[derive(Default)]
struct FecSet {
data: BTreeMap<u32, Vec<u8>>,
code: HashMap<u16, Vec<u8>>,
num_data: u16,
num_code: u16,
sig: Vec<u8>,
data_hdr_tail: Option<(usize, bool)>,
}
#[derive(Default)]
struct Slot {
data: BTreeMap<u32, Vec<u8>>,
fec: HashMap<u32, FecSet>,
starts: HashSet<u32>,
emitted: HashSet<u32>,
}
#[derive(Default)]
struct Assembler {
slots: HashMap<u64, Slot>,
}
impl Assembler {
/// Returns completed block components (raw bytes) ready to deserialize.
fn push(&mut self, p: &[u8]) -> Vec<(u64, u32, Vec<u8>)> {
let Some(h) = header(p) else { return vec![] };
let st = self.slots.entry(h.slot).or_insert_with(|| Slot {
starts: HashSet::from([0]),
..Default::default()
});
let set = st.fec.entry(h.fec).or_default();
if set.sig.is_empty() {
set.sig = p[..SIG].to_vec();
}
if h.is_data {
if st.data.contains_key(&h.index) {
return vec![];
}
set.data.insert(h.index, p[..DATA_PAYLOAD].to_vec());
set.data_hdr_tail = Some((h.proof, h.resigned));
Self::add_data(st, h.index, p[..DATA_PAYLOAD].to_vec());
} else {
let pos = u16::from_le_bytes(p[87..89].try_into().unwrap());
set.num_data = u16::from_le_bytes(p[83..85].try_into().unwrap());
set.num_code = u16::from_le_bytes(p[85..87].try_into().unwrap());
set.code
.entry(pos)
.or_insert_with(|| p[..CODE_PAYLOAD].to_vec());
}
Self::try_recover(st, h.fec);
Self::emit(h.slot, st)
}
fn add_data(st: &mut Slot, index: u32, payload: Vec<u8>) {
if payload[85] & DATA_COMPLETE != 0 {
st.starts.insert(index + 1);
}
st.data.insert(index, payload);
}
/// Rebuild missing data shreds once a FEC set holds at least num_data shreds in total.
fn try_recover(st: &mut Slot, fec: u32) {
let Some(set) = st.fec.get(&fec) else { return };
let (nd, nc) = (set.num_data as usize, set.num_code as usize);
if nd == 0 || set.data.len() >= nd || set.data.len() + set.code.len() < nd {
return;
}
let Some((proof, resigned)) = set.data_hdr_tail.or_else(|| {
set.code
.values()
.next()
.map(|c| ((c[64] & 0x0f) as usize, c[64] & 0xf0 == 0x70))
}) else {
return;
};
let dh = Header {
slot: 0,
index: 0,
fec: 0,
is_data: true,
proof,
resigned,
};
let ch = Header {
is_data: false,
..dh
};
let mut shards: Vec<Option<Vec<u8>>> = (0..nd)
.map(|i| {
set.data
.get(&(fec + i as u32))
.map(|p| p[shard_range(&dh)].to_vec())
})
.collect();
shards.extend((0..nc).map(|i| {
set.code
.get(&(i as u16))
.map(|p| p[shard_range(&ch)].to_vec())
}));
let Ok(rs) = ReedSolomon::new(nd, nc) else {
return;
};
if rs.reconstruct_data(&mut shards).is_err() {
return;
}
let sig = set.sig.clone();
for i in 0..nd {
let index = fec + i as u32;
if st.data.contains_key(&index) {
continue;
}
// Signature + recovered shard = the data shred up to its Merkle proof; headers and data are inside.
let mut payload = sig.clone();
payload.extend_from_slice(shards[i].as_ref().unwrap());
payload.resize(DATA_PAYLOAD, 0);
Self::add_data(st, index, payload.clone());
st.fec.get_mut(&fec).unwrap().data.insert(index, payload);
}
}
fn emit(slot: u64, st: &mut Slot) -> Vec<(u64, u32, Vec<u8>)> {
let mut out = vec![];
let mut starts: Vec<u32> = st.starts.iter().copied().collect();
starts.sort_unstable(); // index order, so output order is stable
for start in starts {
if st.emitted.contains(&start) {
continue;
}
let mut bytes = vec![];
let mut i = start;
loop {
let Some(p) = st.data.get(&i) else { break };
let size = u16::from_le_bytes(p[86..88].try_into().unwrap()) as usize;
bytes.extend_from_slice(&p[DATA_HEADERS..size]);
if p[85] & DATA_COMPLETE != 0 {
out.push((slot, start, std::mem::take(&mut bytes)));
st.emitted.insert(start);
break;
}
i += 1;
}
}
out
}
}
/// Prints one JSON line per transaction. `component` is the index of the first data shred of the block component
/// the transaction came from and `position` its place inside that component: together they order transactions within
/// a slot even when components complete out of order.
fn print_component(slot: u64, component: u32, bytes: &[u8]) {
match wincode::deserialize::<BlockComponent>(bytes) {
Ok(BlockComponent::EntryBatch(entries)) => {
let mut position = 0u32;
for e in entries {
for tx in e.transactions {
let version = match tx.version() {
solana_transaction::versioned::TransactionVersion::Legacy(_) => {
"legacy".to_string()
}
solana_transaction::versioned::TransactionVersion::Number(n) => {
n.to_string()
}
};
let keys = tx.message.static_account_keys();
// Program ids are always static keys (a v0 message may not load them from lookup tables).
let mut programs: Vec<String> = tx
.message
.instructions()
.iter()
.map(|ix| keys[ix.program_id_index as usize].to_string())
.collect();
programs.dedup(); // collapses repeats that sit next to each other
println!(
"{}",
serde_json::json!({"slot": slot, "component": component, "position": position,
"signature": tx.signatures[0].to_string(), "version": version,
"static_keys": keys.iter().map(|k| k.to_string()).collect::<Vec<_>>(), "program_ids": programs})
);
position += 1;
}
}
}
Ok(BlockComponent::BlockMarker(_)) => {}
Err(e) => eprintln!("slot {slot}: undecodable component ({e})"),
}
}
fn main() {
let args: Vec<String> = std::env::args().collect();
let mut asm = Assembler::default();
if args.get(1).map(String::as_str) == Some("--file") {
let raw = std::fs::read(&args[2]).unwrap();
let drop_every: usize = args.get(3).map(|s| s.parse().unwrap()).unwrap_or(0); // test: drop every Nth data shred
let (mut o, mut k) = (0, 0);
while o < raw.len() {
let n = u16::from_le_bytes([raw[o], raw[o + 1]]) as usize;
o += 2;
let p = &raw[o..o + n];
o += n;
if drop_every > 0 && matches!(p[64] & 0xf0, 0x90 | 0xb0) {
k += 1;
if k % drop_every == 0 {
continue;
}
}
for (slot, start, c) in asm.push(p) {
print_component(slot, start, &c);
}
}
return;
}
let sock = std::net::UdpSocket::bind(("0.0.0.0", args[1].parse::<u16>().unwrap())).unwrap();
let mut buf = [0u8; 1280];
loop {
let Ok(n) = sock.recv(&mut buf) else { continue };
for (slot, start, c) in asm.push(&buf[..n]) {
print_component(slot, start, &c);
}
}
}

Running it and what it prints#

Build it with cargo build --release, allow inbound UDP to your port from any source port (the sending port is not fixed), then run shred-receiver 8001 with the port your feed is delivered to.

The receiver ignores every datagram that is not a shred, and that includes the verification packet sent while your destination is being activated. During activation, read that packet with a plain UDP listener, for example nc -ul 8001, reply with its token, and start the decoder once delivery is on. Checking that shreds reach your server walks through it. The --file mode reads a saved capture in the same length-prefixed format the tests use.

Each transaction becomes one line. This is the first line from the fixture for slot 470,901,815, with the key list shortened here:

JSON
{"component":32,"position":0,"program_ids":["Archer8kgiavM61GyusMzaaS2ft5sALtNsD1HxkUPMhy"],"signature":"36rFNmu3EShgKrPSipWyK6A8Yn58jWuHYrByCDDJCFVKzLCMiAGu4mFD4hw5zVKABJnenQSYmhbTWUhJfPsUKiuy","slot":470901815,"static_keys":["4M1ViRmsMu2vHBSsWK4b2PMKjQihwzXv78Luy3si5Moi","kmBG7GCnHbnhNkmb1aVPrsbebduhBaRqBqQJpGinmtE","..."],"version":"1"}

component names the block component a transaction came from by the shred index it starts at, and position counts transactions within that component. Sorting by the pair restores block order even when components finish out of order. version is legacy, 0 or 1; the v1 format is the one that keeps its signatures at the end, and solana-transaction handles that for you.

How this code was tested#

The fixtures are two consecutive mainnet blocks, slots 470,901,815 and 470,901,816, fetched with getBlock and shredded locally by Agave 4.4.1's own Shredder. The transactions and their order are real. Entry grouping, entry hashes and the signing key are synthetic, because RPC does not expose entries and the real leader key is not ours to use.

TestResult
Output order against the shredder's own record of the blocks269 of 269 and 512 of 512 transactions, in order
Versions decoded178 legacy, 331 v0 and 272 v1
Every 3rd data shred dropped before decodingall 781 transactions recovered from coding shreds
Over real UDP: reordered, duplicated, 1 in 4 data shreds dropped781 of 781, no duplicates, nothing foreign
static_keys and program_ids against an independent TypeScript decoder781 of 781 identical

Production notes#

  • Bound memory. The assembler keeps every slot it has seen; in a long-running process, drop slots that fall a fixed number of slots behind the newest one.
  • Read fast, decode elsewhere. If decoding blocks the socket, the kernel buffer fills and packets are dropped. Move parsing to another thread once you go beyond a test.
  • Expect duplicates. Your receiver can see the same shred twice, and the same transaction can appear in a block that is later skipped and again in a later block; deduplicate by signature before you store anything.
  • v0 accounts are incomplete here: the line lists static keys only. Resolving v0 lookup tables adds the rest.
  • Recovery rebuilds data but does not re-check the leader's signature over the Merkle root. Add that check if your source of shreds is not one you trust.

Questions#

Which Rust crate decodes Solana shreds?#

solana-entry 4.4.1 decodes the block components inside them, and solana-transaction the transactions. The shred header itself is simple enough to read by hand, which avoids solana-ledger and its RocksDB build.

Does it decode instruction data?#

No. It identifies programs and accounts. Instruction meaning comes next: decoding Anchor instruction data and the program recipes build on these lines.

Why are transactions sometimes printed before the slot is complete?#

By design. Each block component is decoded as soon as it is whole, so you see the first transactions of a slot early. The pipeline stages explain the trade-off.

What this decoder does not do#

It reports requested transactions, not outcomes: no success flag, logs, inner instructions or balance changes, because shreds do not carry them. It cannot recover a set that lost more shreds than the set has coding shreds, and it does not resolve lookup tables. Choosing a language on other grounds is covered in choosing a language, and a faster parser design in zero-copy parsing.

Sources#

Raw shreds on your own server

Ultra Low Latency Shreds is $200 and Standard Shreds is $100. You pay in crypto, tell us which IPv4 address and UDP port should receive the stream, and we wire it up by hand.