Pipeline

Reconciling what you decoded with what confirmed

Everything a shred decoder emits is provisional. This page settles it: ask RPC what happened to each signature, mark it confirmed, failed or dropped, and know exactly when "no answer" is allowed to mean "dropped".

To reconcile shred transactions with RPC, send their signatures to getSignatureStatuses in batches of up to 256 with history search on. A confirmed or finalized status settles each one as confirmed or failed. No status means dropped only once the blockhash has expired and the node still holds that slot. Durable nonce transactions never expire that way.

Three outcomes, and one waiting state#

StatusWhenWhat RPC returned
confirmedthe transaction is in a block that will stay, and it succeededa status at confirmed or finalized with err null
failedit is in a block that will stay, it paid its fee, and it faileda status at confirmed or finalized with an error
droppedit can no longer land anywhereno status, the blockhash expired on the finalized bank, and still no status when asked again after that
seen (waiting)anything elsea processed status, or no status while the blockhash is still valid, or no status for a durable nonce transaction

A failed transaction is not dropped: it is in the block and it changed the fee payer's balance. Keep it, with its outcome, as landed but failed explains.

Steps#

  1. Queue each decoded transaction with its slot, signature and raw bytes, status seen.
  2. Wait until the slot can have reached the commitment you care about. Do not hard-code a number of slots; poll and let the status tell you.
  3. Call getSignatureStatuses with up to 256 signatures and searchTransactionHistory: true. Without the flag a node keeps only very recent statuses.
  4. Settle only confirmed or finalized answers. A processed answer, success or error, can still be rolled back.
  5. For signatures with no status, read the blockhash from the raw transaction and ask isBlockhashValid with finalized commitment, once per distinct blockhash. Valid means wait.
  6. For the expired ones, call getSignatureStatuses again. The transaction may have landed between your first call and the expiry check; only a missing status on this second call means dropped.
  7. Skip steps 5 and 6 for slots older than the node's minimumLedgerSlot: below it, a missing status may only mean the node no longer keeps that history.
  8. Skip steps 5 and 6 for durable nonce transactions: their first instruction advances a nonce account, and the field that normally holds a blockhash holds a nonce, so it never expires.
  9. Write the new status, moving forward only: seen to anything, never back.

reconcile.ts#

reconcile.ts
// Reconcile: turn provisional "seen" records into confirmed, failed or dropped.
// Seeing a transaction in shreds proves it was in a leader's block data, not
// that it succeeded or that the block survived. Two ways to decide:
// offline: compare with a getBlock result for the record's slot
// live: ask an RPC node with getSignatureStatuses
import { readFileSync, existsSync } from "node:fs";
import { readJsonl, type Status, type TxRecord } from "./records.ts";
import { applyStatuses } from "./postgres.ts";
import { insertBatch } from "./clickhouse.ts";
import { decodeTx } from "../typescript/src/tx.ts";
export type Decision = { signature: string; status: Status };
const B58 = "123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz";
function base58(buf: Uint8Array): string {
let n = 0n;
for (const x of buf) n = n * 256n + BigInt(x);
let s = "";
while (n > 0n) { s = B58[Number(n % 58n)] + s; n /= 58n; }
for (const x of buf) { if (x) break; s = "1" + s; }
return s;
}
// The first signature is the transaction id. Legacy and v0: compact-u16
// signature count, then the 64 byte signatures. v1 (SIMD-0385, first byte
// 0x81): the signature count is byte 1 and the signatures come last, after the
// message, so read them from the end of the buffer.
export function firstSignature(wireBase64: string): string {
const b = Buffer.from(wireBase64, "base64");
if (b[0] === 0x81) {
const start = b.length - 64 * b[1];
return base58(b.subarray(start, start + 64));
}
let off = 0;
while (b[off++] & 0x80); // skip the compact-u16 length (1 to 3 bytes)
return base58(b.subarray(off, off + 64));
}
// signature -> meta.err (null = success) for one block, or null when there is
// no block for the slot. In this offline example a missing file stands for a
// skipped slot; against a real RPC, check for the "slot was skipped" error
// rather than treating "not fetched yet" as skipped.
export function loadBlock(blocksDir: string, slot: number): Map<string, unknown> | null {
const path = `${blocksDir}/${slot}.json`;
if (!existsSync(path)) return null;
const block = JSON.parse(readFileSync(path, "utf8"));
const out = new Map<string, unknown>();
for (const tx of block.transactions) out.set(firstSignature(tx.transaction[0]), tx.meta.err);
return out;
}
// Offline: a record whose signature is not in its slot's saved block is NOT dropped. Seeing a transaction in the
// shreds of slot S does not pin it to S: if that block was skipped or abandoned, the transaction can still land in a
// later slot until its blockhash expires. Without a live RPC to check expiry it stays "seen".
export function reconcileOffline(records: TxRecord[], blocksDir: string): Decision[] {
const blocks = new Map<number, Map<string, unknown> | null>();
return records.map((r) => {
if (!blocks.has(r.slot)) blocks.set(r.slot, loadBlock(blocksDir, r.slot));
const block = blocks.get(r.slot)!;
if (!block || !block.has(r.signature)) return { signature: r.signature, status: "seen" };
return { signature: r.signature, status: block.get(r.signature) === null ? "confirmed" : "failed" };
});
}
async function rpc(url: string, method: string, params: unknown[]): Promise<any> {
for (let attempt = 0; ; attempt++) {
const res = await fetch(url, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ jsonrpc: "2.0", id: 1, method, params }),
});
// Public endpoints rate limit; back off exponentially, honour Retry-After.
if (res.status === 429 && attempt < 6) {
const wait = Number(res.headers.get("retry-after")) * 1000 || 500 * 2 ** attempt;
console.warn(`${method}: HTTP 429, retrying in ${wait} ms`);
await new Promise((r) => setTimeout(r, wait));
continue;
}
if (!res.ok) throw new Error(`${method}: HTTP ${res.status}`);
const body = await res.json();
if (body.error) throw new Error(`${method}: ${JSON.stringify(body.error)}`);
return body.result;
}
}
// A transaction that never landed can still land later, in any slot, until its blockhash expires. So "no status"
// only becomes "dropped" once isBlockhashValid says the blockhash is gone. Durable nonce transactions (first
// instruction: System AdvanceNonceAccount, solana-nonce NONCED_TX_MARKER_IX_INDEX = 0) carry a nonce in that field
// and never expire that way: they stay "seen" until your own time limit.
export function lifetime(rawBase64: string): { blockhash: string; durableNonce: boolean } {
const tx = decodeTx(Buffer.from(rawBase64, "base64"));
const first = tx.instructions[0];
const durableNonce = !!first && tx.staticKeys[first.programIndex] === "11111111111111111111111111111111"
&& first.data.length >= 4 && Buffer.from(first.data).readUInt32LE(0) === 4;
return { blockhash: tx.blockhash, durableNonce };
}
const settledStatus = (s: any): Status | null =>
s && (s.confirmationStatus === "confirmed" || s.confirmationStatus === "finalized") ? (s.err === null ? "confirmed" : "failed") : null;
async function statuses(url: string, signatures: string[]): Promise<any[]> {
const out: any[] = [];
for (let i = 0; i < signatures.length; i += 256) { // getSignatureStatuses accepts at most 256 signatures per call
const result = await rpc(url, "getSignatureStatuses", [signatures.slice(i, i + 256), { searchTransactionHistory: true }]);
out.push(...result.value); // without searchTransactionHistory only very recent statuses are kept
}
return out;
}
// Use ONE RPC node for all of this (a sticky endpoint): behind a load balancer the calls can reach different nodes,
// one ahead and one behind, and their answers will not line up.
export async function reconcileLive(records: TxRecord[], url = "https://api.mainnet-beta.solana.com"): Promise<Decision[]> {
const first = await statuses(url, records.map((r) => r.signature));
// "processed" can still be rolled back: settle only confirmed or finalized, for success and failure alike
const out: Decision[] = records.map((r, i) => ({ signature: r.signature, status: settledStatus(first[i]) ?? "seen" }));
const unknown = records.filter((_, i) => first[i] === null);
if (!unknown.length) return out;
// No status: dropped only if the blockhash has expired on the FINALIZED bank (every fork that can still win
// descends from it, so expired there means expired everywhere), the node still holds the slot (below
// minimumLedgerSlot a missing status may be pruned history) and the transaction is not a durable nonce one.
const oldestHeld: number = await rpc(url, "minimumLedgerSlot", []);
const valid = new Map<string, boolean>();
const expired: TxRecord[] = [];
for (const r of unknown) {
const { blockhash, durableNonce } = lifetime(r.raw_base64);
if (durableNonce || r.slot < oldestHeld) continue;
if (!valid.has(blockhash)) valid.set(blockhash, (await rpc(url, "isBlockhashValid", [blockhash, { commitment: "finalized" }])).value);
if (!valid.get(blockhash)) expired.push(r);
}
// The transaction may have landed between the first status call and the expiry check, so ask again now that the
// blockhash is known to be expired. Only a missing status on this second call means it can no longer land.
const second = await statuses(url, expired.map((r) => r.signature));
expired.forEach((r, i) => {
out.find((d) => d.signature === r.signature)!.status = second[i] === null ? "dropped" : settledStatus(second[i]) ?? "seen";
});
return out;
}
// Apply decisions to both stores. Postgres updates in place, forward only.
// ClickHouse gets a new row per signature; its higher status_rank wins.
export async function applyToPostgres(decisions: Decision[], table = "solana_tx"): Promise<number> {
return applyStatuses(decisions.filter((d) => d.status !== "seen"), table);
}
export async function applyToClickHouse(records: TxRecord[], decisions: Decision[], table = "solana_tx"): Promise<void> {
const bySig = new Map(decisions.map((d) => [d.signature, d.status]));
const changed = records
.filter((r) => (bySig.get(r.signature) ?? "seen") !== "seen")
.map((r) => ({ ...r, status: bySig.get(r.signature)! }));
if (changed.length) await insertBatch(changed, table);
}
export function tally(decisions: Decision[]): Record<string, number> {
const t: Record<string, number> = {};
for (const d of decisions) t[d.status] = (t[d.status] ?? 0) + 1;
return t;
}
// node reconcile.ts --live runs the live mode once over the fixtures.
if (import.meta.main && process.argv.includes("--live")) {
const dir = new URL("../fixtures/decoded/", import.meta.url).pathname;
const records = [470901815, 470901816].flatMap((s) => readJsonl(`${dir}${s}.jsonl`));
const decisions = await reconcileLive(records);
console.log(JSON.stringify({ checked: records.length, ...tally(decisions) }));
}

records.ts, postgres.ts and clickhouse.ts are the sink modules from the Postgres and ClickHouse pages; tx.ts is the transaction parser from the TypeScript decoder. The offline mode compares against saved getBlock results and is what the tests run without a network. It never marks a row dropped: a transaction missing from one slot's block may still land in another, and only the live mode can check expiry.

How this code was tested#

TestResult
Live, against mainnet RPC, all 781 transactions of two blocks502 confirmed, 279 failed, exactly as the blocks' own metadata records
Offline, against the saved blocks, plus fabricated records for a slot with no block502 confirmed, 279 failed, and the 2 fabricated records stay seen (absence from one slot's block is not a drop); written to Postgres and ClickHouse
Fake RPC: batching and backoffbatches of 256, 256 and 92, then a re-check of the 2 expired signatures; one HTTP 429 retried
Fake RPC: no status, expired blockhash, still no status on the re-checkdropped
Fake RPC: no status at first, landed before expiryconfirmed on the re-check
Fake RPC: commitment used for the expiry checkfinalized, every time
Fake RPC: no status, valid blockhashstays seen
Fake RPC: no status, a real durable nonce transactionstays seen, and no blockhash call is made
Fake RPC: processed, with and without an errorboth stay seen
Fake RPC: no status, slot below the node's oldest held slotstays seen

In the same two blocks, 24 transactions used a durable nonce, so the nonce rule matters in ordinary traffic.

When no answer means dropped#

"RPC has no status" has three possible reasons: the transaction has not landed yet, it never will, or the node does not keep history that far back. Only the second is a drop, and the blockhash is how you tell. A transaction can land in any later slot until its blockhash expires, so a transaction you saw in a block that was then skipped may still land somewhere else. Once isBlockhashValid says the blockhash is gone, it cannot.

Two details make that rule sound. Check expiry on the finalized bank: every fork that can still win builds on it, so a blockhash expired there is expired everywhere, while the processed tip may be a fork that never survives. And ask for the status again after the expiry check, because the transaction may have landed in between; only a second missing status means it can no longer land.

History depth is the third reason, and the blockhash check cannot see it: an old blockhash is always expired, whether or not the transaction landed. So the code first asks the node for minimumLedgerSlot, the oldest slot it still holds, and never concludes a drop for a record older than that. How much history a node keeps depends on the provider; reconcile promptly and those records stay rare.

Some risk remains that no single node can rule out. Behind a load-balanced endpoint, consecutive calls can reach different nodes, one ahead and one behind, and minimumLedgerSlot only says where a node's ledger starts, not that it has no gaps. Use one node for the whole check, and if money depends on a drop, treat it as final only when a second source agrees. Provisional rows covers designing tables around that uncertainty.

Alpenglow note#

Under Alpenglow, live on mainnet since epoch 1064 (SIMD-0326), finality comes from certificates that validators aggregate from their votes, and votes left the blocks from slot 459,653,000. Nothing in a shred tells you a block was certified, which is exactly why this step exists. confirmed and finalized are equivalent under Alpenglow, so settling at confirmed costs no safety; processed is still earlier and can still be undone. Alpenglow for indexers explains the rest.

What reconciliation cannot fix#

Reconciliation settles outcomes; it does not add the data shreds lack. Logs, inner instructions and balance changes still need getTransaction or another executed source. It also cannot recover a transaction your receiver never decoded: a component lost to packet loss is not in your queue at all. For durable nonce transactions it cannot decide a drop, so give them your own time limit.

Where the statuses go#

The Postgres status column stores them with forward-only updates, the ClickHouse schema keeps the highest status per signature, and deduplication comes before both. Slot or block height explains the two numbers you will see in these rows. To feed this loop from your own shreds, see pricing.

Sources#

Raw shreds on your own server

Ultra Low Latency Shreds is $200 and Standard Shreds is $100. You pay in crypto, tell us which IPv4 address and UDP port should receive the stream, and we wire it up by hand.