Publishing decoded transactions to Kafka
Your decoder turns shreds into transaction records long before anyone knows how they ended. A Kafka topic keyed by signature carries those early records to every consumer, and later status events for the same key settle them.
To send Solana transactions to Kafka from a shred decoder, publish one record per decoded transaction with the first signature as the message key. Kafka then keeps every copy of that transaction in one partition, in order. Status events from your reconcile job use the same key, so consumers can let a confirmed or failed event replace the earlier one.
Topic layout for data that is not final yet#
Everything a shred decoder emits starts out provisional. The transaction was in a leader's block data, but nobody yet knows whether it succeeded or whether its block will stay (E4 tech facts X-04). The topic design has to carry that uncertainty instead of hiding it.
Two layouts work:
- One topic for every event. The decoder publishes a record with status
seen; the reconcile job later publishes a second record with the same key and statusconfirmed,failedordropped. Consumers that need settled data wait for the second event; consumers that react early act on the first and correct themselves on the second. - Two topics. A
seentopic for the decoder's output and a separate status topic for reconcile results, joined by signature in the consumer. This keeps the high-volume stream apart from the much smaller stream of outcomes.
The tested example below covers the decoder side: the solana.tx.seen topic. Publishing status events is the same produce call with a different status in the record, made by whatever runs reconciliation.
Keys and partitions#
| Choice | Effect |
|---|---|
| Key = first signature | Every copy of one transaction (a repeat in the shred stream or a later status event) hashes to the same partition, so one consumer sees all of them in the order they were written |
| Key = slot | All transactions of a busy slot pile into one partition, and a status event for one transaction lands far from its other copies; avoid it |
| Slot as a header | Consumers can route or skip by slot without parsing the JSON value |
| Fixed partition count | Adding partitions later changes which partition a key maps to, so the "same signature, same partition" guarantee only holds while the count stays the same |
The signature is the transaction's ID, which is why it is the natural key here and in the databases downstream.
Steps#
- Create the topic once with a fixed number of partitions, and treat that number as part of your design.
- Turn each decoded transaction into a record: slot, entry and transaction position, signature, version, fee payer, top-level programs, status and the raw bytes.
- Publish with the signature as the key, the slot and version as headers (
slot,tx_version), and the idempotent producer switched on. - Publish the reconcile job's outcome for each signature as another record with the same key.
- In every consumer, drop repeats by key and keep status moving forward only, as the deduplication page and the Postgres page describe.
kafka.ts#
This is the producer half of the tested sink, unchanged. It runs on Node 24 with @confluentinc/kafka-javascript 1.11.0 against Apache Kafka 4.3.2.
// Kafka sink for decoded transactions, using @confluentinc/kafka-javascript// (librdkafka based, KafkaJS style API). kafkajs itself has had no release// since February 2023, so new code should not start on it.//// Topic design: key = signature. Kafka hashes the key to pick the partition,// so every copy of one transaction (shred stream repeats, retries) lands in// the same partition, in order, and one consumer sees all of them. The slot// rides in a header so consumers can route or filter without parsing JSON.import KafkaPkg from "@confluentinc/kafka-javascript";import type { TxRecord } from "./records.ts";const { Kafka, logLevel } = KafkaPkg.KafkaJS;export const TOPIC = "solana.tx.seen";const brokers = (process.env.KAFKA_BROKERS ?? "127.0.0.1:18392").split(",");const kafka = new Kafka({ kafkaJS: { brokers, clientId: "solana-tx-sink", logLevel: logLevel.WARN } });export async function createTopic(topic = TOPIC, partitions = 6): Promise<void> {const admin = kafka.admin();await admin.connect();try {// Fixed partition count: adding partitions later remaps keys, so the// "same signature, same partition" property only holds per partition count.await admin.createTopics({ topics: [{ topic, numPartitions: partitions, replicationFactor: 1 }] });} catch (e) {if (!String(e).includes("already exists")) throw e;} finally {await admin.disconnect();}}export async function produce(records: TxRecord[], topic = TOPIC): Promise<void> {const producer = kafka.producer({// Idempotence stops broker side duplicates caused by producer retries// (it implies acks=all). It does NOT remove duplicates that were already// duplicates in the input; consumers still dedupe by key.kafkaJS: { idempotent: true, acks: -1 },// Batching: wait briefly so many small records share one request."linger.ms": 20,"batch.num.messages": 10000,"compression.type": "lz4",});await producer.connect();try {await producer.send({topic,messages: records.map((r) => ({key: r.signature,value: JSON.stringify(r),headers: { slot: String(r.slot), tx_version: r.version },})),});await producer.flush({ timeout: 10000 });} finally {await producer.disconnect();}}
The record type it sends:
export type Status = "seen" | "confirmed" | "failed" | "dropped";export type TxRecord = {slot: number;entry_index: number;tx_index: number;signature: string;version: "legacy" | "0" | "1";fee_payer: string;programs: string[];status: Status;raw_base64: string;};
What the test run showed#
The test publishes the decoded records of two real mainnet blocks, repeating every fifth record to imitate a transaction that shows up twice, then reads the whole topic back.
| Check | Result on 18 November 2026 |
|---|---|
| Records published, records read back | 937 and 937 |
| Distinct keys among them | 781, one per transaction in the two blocks |
| Key equals the record's signature | every message |
| Slot header equals the record's slot | every message |
| Both copies of a repeated transaction in the same partition | every repeat |
The transactions are real. Their grouping into entries, and so the entry_index field, comes from a locally built shred fixture, because RPC does not expose entries.
Consumers, duplicates and compaction#
The idempotent producer only removes the duplicates its own retries would create on the broker. A transaction that was already in the input twice is published twice, and the test above shows exactly that: 937 messages for 781 keys. Every consumer therefore drops repeats itself, by key.
A compacted topic keeps, in time, only the last record written for each key. Compaction runs in the background and never touches the segment still being written, so a consumer that replays the topic can see both the seen record and the later status record for one signature. Do not compact a topic where seen and status records share a key: a late seen written after the outcome would be the record compaction keeps. If you need compaction, use the two-topic layout and compact only the status topic, where every record is an outcome. In every consumer, still move status forward only, so an old seen read after a confirmed changes nothing. Kafka, NATS or Redis Streams compares how each bus handles that pattern, and idempotent writes covers the database side.
What a topic does not settle#
A message in this topic says a transaction was seen in shreds, nothing more. It carries no execution result, and its block may still be skipped. Kafka adds ordering inside a partition and durable delivery to consumers; it adds no truth about the chain. If your consumers only ever need confirmed data, a stream fed from an RPC or Geyser source after confirmation may serve them with less work than raw shreds plus reconciliation.
From the topic to storage#
Consumers usually land in an analytical store or an operational database: ClickHouse for large scans, Postgres for application queries. If you are joining this stream with history, backfill and live shows where to cut over. Deciding whether to run this pipeline at all is the subject of build or buy an indexer. To feed it, a Standard or Ultra plan delivers the raw shreds.
Sources#
- Apache Kafka documentation: producer idempotence, partitioning by key, log compaction (checked 2026-11-18)
- Confluent's JavaScript client for Apache Kafka (librdkafka based), version 1.11.0 used here (checked 2026-11-18)
- Confluent JavaScript client overview (checked 2026-11-18)
- Solana RPC getSignatureStatuses: how a seen transaction is later confirmed (E4 tech facts X-04) (checked 2026-11-18)
- Fixtures: mainnet blocks 470,901,815 and 470,901,816 (getBlock), 781 transactions (checked 2026-11-18)
Raw shreds on your own server
Ultra Low Latency Shreds is $200 and Standard Shreds is $100. You pay in crypto, tell us which IPv4 address and UDP port should receive the stream, and we wire it up by hand.