Pipeline

Deduplicating transactions across slots, feeds and restarts

The same transaction can reach your pipeline more than once. Drop the repeats early with a small, bounded set keyed on the signature, and let the database key catch whatever slips past the window.

To deduplicate Solana transactions, key them on the first signature, the value explorers and RPC use as the transaction ID. Keep recently seen signatures in a bounded in-memory set so repeats are dropped before they reach a sink, and let a database key such as a Postgres primary key catch any repeat that arrives after the set has forgotten it.

Where repeats come from#

SourceWhat happensWhere it is caught
The shred stream itselfa transaction can show up again, for example in a later block after a skipthe in-memory set
Two feeds side by sideeach feed delivers its own copythe in-memory set, if both feeds go through one decoder process
A decoder restartthe decoder replays a stretch of captured data it already wrotethe database key, since the in-memory set starts empty
Reconciliationthe reconcile job writes a second record for the same signaturenot a duplicate: it is a status change, handled by forward-only updates
A forkthe transaction appears in a block that is skipped, then lands in a later blockdropped by the set like any repeat. The stored row keeps the first slot, which may be the skipped one. The reconcile job settles the outcome by signature; take the slot it landed in from RPC if you need it

Shred-level repeats, the same packet arriving twice, are handled earlier, while the decoder assembles entries. This page works one level up, on whole transactions.

Steps#

  1. Take the first signature of each decoded transaction as its key. In legacy and v0 transactions it comes right after the signature count; in v1 the signatures sit at the end of the transaction, as signatures as IDs shows.
  2. Check the key against a bounded set of recent signatures. Pass the transaction on only the first time.
  3. Keep the slot in the record, never in the key: the same transaction can legitimately be seen at two slots.
  4. Store with a key that settles repeats for good: the primary key in Postgres, which rejects them on insert, or the ReplacingMergeTree sort key in ClickHouse, which collapses them at merge time and when you read with FINAL.

dedupe.ts#

The tested in-process dedupe, unchanged. It uses two rotating sets, so memory stays bounded and the check is exact inside its window.

dedupe.ts
// In-process signature dedupe with bounded memory. The shred stream can carry
// the same transaction more than once (repeated entries, client retries), so
// drop repeats before they reach a sink.
//
// Two rotating Sets: new signatures go into `current`; when it is full it
// becomes `previous` and the old `previous` is discarded. Memory stays at most
// 2 * capacity signatures, and a signature is remembered at least until
// `capacity` newer distinct signatures have been added. Beyond that window a repeat is let
// through, so keep the database level dedupe (primary key, ReplacingMergeTree)
// as the source of truth. The check is exact inside the window: no false
// positives, unlike a Bloom filter, which can wrongly report a new signature
// as already seen and so silently drop a real transaction.
export class SignatureDedupe {
private current = new Set<string>();
private previous = new Set<string>();
private readonly capacity: number;
constructor(capacity: number) {
this.capacity = capacity;
}
// Returns true the first time a signature is offered, false for a repeat.
firstSeen(signature: string): boolean {
if (this.current.has(signature) || this.previous.has(signature)) return false;
this.current.add(signature);
if (this.current.size >= this.capacity) {
this.previous = this.current;
this.current = new Set();
}
return true;
}
get size(): number {
return this.current.size + this.previous.size;
}
}

Test results#

The test feeds the decoded records of two real mainnet blocks with every fifth record repeated, then probes the memory bound and the edge of the window.

CheckResult on 18 November 2026
Records in, records kept937 in, 781 kept, 156 repeats dropped
Kept records with distinct signatures781
Capacity 100: the largest the two sets ever grew199 signatures, within the bound of 200
A repeat still inside the windowrejected
A signature offered again after the window moved past itlet through, for the database key to catch

Same signature, two forks#

A transaction decoded from shreds was in a leader's block data, but that block can still be skipped. When it is, the transaction is not lost: if its blockhash is still valid it can land in a later block, and your decoder will see it again at a different slot. That second sighting carries the same signature and is the one that may end up confirmed.

So treat a repeat signature as "already known", not as an error. Keep the record you have; its slot is the first one you saw, which may be the skipped one. Let the reconcile job decide the outcome from what RPC reports for that signature. Once a transaction has landed and is confirmed, the network itself refuses a second copy as already processed, so on the confirmed chain each signature appears once.

Bloom filters: when a false positive is acceptable#

A Bloom filter stores far more signatures in the same memory, but it can report a signature it has never seen as already present. In a dedupe stage that means silently dropping a real transaction. That trade is acceptable only when losing a real transaction costs little, for example in an approximate count for a dashboard. No database key downstream can bring back a transaction the filter dropped. For anything that feeds alerts, balances or trading decisions, use an exact set like the one above and accept its bounded window.

What in-memory dedupe cannot promise#

The set forgets. After enough newer signatures, an old repeat passes through, and after a restart the set starts empty. That is why it sits in front of a database key and never replaces it. It also works per process: two decoder processes each keep their own set, so either share one stage for both feeds or rely on the database. And it says nothing about outcomes; a kept transaction may still fail or be dropped.

Idempotent writes for chain data explains the database half of the same idea. The Kafka page keys messages by the same signature, so repeats share a partition. The Rust decoder produces the records this stage reads. To run it on live data, choose a plan.

Sources#

Raw shreds on your own server

Ultra Low Latency Shreds is $200 and Standard Shreds is $100. You pay in crypto, tell us which IPv4 address and UDP port should receive the stream, and we wire it up by hand.