Canonical: https://solanashreds.xyz/docs/pipeline/deduplicate

# Deduplicating transactions across slots, feeds and restarts

The same transaction can reach your pipeline more than once. Drop the repeats early with a small, bounded set keyed on the signature, and let the database key catch whatever slips past the window.

To deduplicate Solana transactions, key them on the first signature, the value explorers and RPC use as the transaction ID. Keep recently seen signatures in a bounded in-memory set so repeats are dropped before they reach a sink, and let a database key such as a Postgres primary key catch any repeat that arrives after the set has forgotten it.

## Where repeats come from

| Source | What happens | Where it is caught |
| --- | --- | --- |
| The shred stream itself | a transaction can show up again, for example in a later block after a skip | the in-memory set |
| Two feeds side by side | each feed delivers its own copy | the in-memory set, if both feeds go through one decoder process |
| A decoder restart | the decoder replays a stretch of captured data it already wrote | the database key, since the in-memory set starts empty |
| Reconciliation | the reconcile job writes a second record for the same signature | not a duplicate: it is a status change, handled by forward-only updates |
| A fork | the transaction appears in a block that is skipped, then lands in a later block | dropped by the set like any repeat. The stored row keeps the first slot, which may be the skipped one. The reconcile job settles the outcome by signature; take the slot it landed in from RPC if you need it |

Shred-level repeats, the same packet arriving twice, are handled earlier, while the decoder assembles entries. This page works one level up, on whole transactions.

## Steps

1. Take the first signature of each decoded transaction as its key. In legacy and v0 transactions it comes right after the signature count; in v1 the signatures sit at the end of the transaction, as [signatures as IDs](https://solanashreds.xyz/learn/transaction-signatures-as-ids) shows.
2. Check the key against a bounded set of recent signatures. Pass the transaction on only the first time.
3. Keep the slot in the record, never in the key: the same transaction can legitimately be seen at two slots.
4. Store with a key that settles repeats for good: the primary key in [Postgres](https://solanashreds.xyz/docs/pipeline/postgres), which rejects them on insert, or the ReplacingMergeTree sort key in [ClickHouse](https://solanashreds.xyz/docs/pipeline/clickhouse), which collapses them at merge time and when you read with FINAL.

## dedupe.ts

The tested in-process dedupe, unchanged. It uses two rotating sets, so memory stays bounded and the check is exact inside its window.

```typescript file="dedupe.ts"
// In-process signature dedupe with bounded memory. The shred stream can carry
// the same transaction more than once (repeated entries, client retries), so
// drop repeats before they reach a sink.
//
// Two rotating Sets: new signatures go into `current`; when it is full it
// becomes `previous` and the old `previous` is discarded. Memory stays at most
// 2 * capacity signatures, and a signature is remembered at least until
// `capacity` newer distinct signatures have been added. Beyond that window a repeat is let
// through, so keep the database level dedupe (primary key, ReplacingMergeTree)
// as the source of truth. The check is exact inside the window: no false
// positives, unlike a Bloom filter, which can wrongly report a new signature
// as already seen and so silently drop a real transaction.
export class SignatureDedupe {
  private current = new Set<string>();
  private previous = new Set<string>();
  private readonly capacity: number;

  constructor(capacity: number) {
    this.capacity = capacity;
  }

  // Returns true the first time a signature is offered, false for a repeat.
  firstSeen(signature: string): boolean {
    if (this.current.has(signature) || this.previous.has(signature)) return false;
    this.current.add(signature);
    if (this.current.size >= this.capacity) {
      this.previous = this.current;
      this.current = new Set();
    }
    return true;
  }

  get size(): number {
    return this.current.size + this.previous.size;
  }
}
```

## Test results

The test feeds the decoded records of two real mainnet blocks with every fifth record repeated, then probes the memory bound and the edge of the window.

| Check | Result on 18 November 2026 |
| --- | --- |
| Records in, records kept | 937 in, 781 kept, 156 repeats dropped |
| Kept records with distinct signatures | 781 |
| Capacity 100: the largest the two sets ever grew | 199 signatures, within the bound of 200 |
| A repeat still inside the window | rejected |
| A signature offered again after the window moved past it | let through, for the database key to catch |

## Same signature, two forks

A transaction decoded from shreds was in a leader's block data, but that block can still be skipped. When it is, the transaction is not lost: if its blockhash is still valid it can land in a later block, and your decoder will see it again at a different slot. That second sighting carries the same signature and is the one that may end up confirmed.

So treat a repeat signature as "already known", not as an error. Keep the record you have; its slot is the first one you saw, which may be the skipped one. Let the [reconcile job](https://solanashreds.xyz/docs/pipeline/reconcile) decide the outcome from what RPC reports for that signature. Once a transaction has landed and is confirmed, the network itself refuses a second copy as already processed, so on the confirmed chain each signature appears once.

## Bloom filters: when a false positive is acceptable

A Bloom filter stores far more signatures in the same memory, but it can report a signature it has never seen as already present. In a dedupe stage that means silently dropping a real transaction. That trade is acceptable only when losing a real transaction costs little, for example in an approximate count for a dashboard. No database key downstream can bring back a transaction the filter dropped. For anything that feeds alerts, balances or trading decisions, use an exact set like the one above and accept its bounded window.

## What in-memory dedupe cannot promise

The set forgets. After enough newer signatures, an old repeat passes through, and after a restart the set starts empty. That is why it sits in front of a database key and never replaces it. It also works per process: two decoder processes each keep their own set, so either share one stage for both feeds or rely on the database. And it says nothing about outcomes; a kept transaction may still fail or be dropped.

## Related pages

[Idempotent writes for chain data](https://solanashreds.xyz/blog/idempotent-writes-for-chain-data) explains the database half of the same idea. The [Kafka page](https://solanashreds.xyz/docs/pipeline/kafka) keys messages by the same signature, so repeats share a partition. The [Rust decoder](https://solanashreds.xyz/docs/decode/rust) produces the records this stage reads. To run it on live data, [choose a plan](https://solanashreds.xyz/pricing).
