Canonical: https://solanashreds.xyz/docs/pipeline/kafka

# Publishing decoded transactions to Kafka

Your decoder turns shreds into transaction records long before anyone knows how they ended. A Kafka topic keyed by signature carries those early records to every consumer, and later status events for the same key settle them.

To send Solana transactions to Kafka from a shred decoder, publish one record per decoded transaction with the first signature as the message key. Kafka then keeps every copy of that transaction in one partition, in order. Status events from your reconcile job use the same key, so consumers can let a confirmed or failed event replace the earlier one.

## Topic layout for data that is not final yet

Everything a shred decoder emits starts out provisional. The transaction was in a leader's block data, but nobody yet knows whether it succeeded or whether its block will stay (E4 tech facts X-04). The topic design has to carry that uncertainty instead of hiding it.

Two layouts work:

- **One topic for every event.** The decoder publishes a record with status `seen`; the [reconcile job](https://solanashreds.xyz/docs/pipeline/reconcile) later publishes a second record with the same key and status `confirmed`, `failed` or `dropped`. Consumers that need settled data wait for the second event; consumers that react early act on the first and correct themselves on the second.
- **Two topics.** A `seen` topic for the decoder's output and a separate status topic for reconcile results, joined by signature in the consumer. This keeps the high-volume stream apart from the much smaller stream of outcomes.

The tested example below covers the decoder side: the `solana.tx.seen` topic. Publishing status events is the same `produce` call with a different status in the record, made by whatever runs reconciliation.

## Keys and partitions

| Choice | Effect |
| --- | --- |
| Key = first signature | Every copy of one transaction (a repeat in the shred stream or a later status event) hashes to the same partition, so one consumer sees all of them in the order they were written |
| Key = slot | All transactions of a busy slot pile into one partition, and a status event for one transaction lands far from its other copies; avoid it |
| Slot as a header | Consumers can route or skip by slot without parsing the JSON value |
| Fixed partition count | Adding partitions later changes which partition a key maps to, so the "same signature, same partition" guarantee only holds while the count stays the same |

The [signature is the transaction's ID](https://solanashreds.xyz/learn/transaction-signatures-as-ids), which is why it is the natural key here and in the databases downstream.

## Steps

1. Create the topic once with a fixed number of partitions, and treat that number as part of your design.
2. Turn each decoded transaction into a record: slot, entry and transaction position, signature, version, fee payer, top-level programs, status and the raw bytes.
3. Publish with the signature as the key, the slot and version as headers (`slot`, `tx_version`), and the idempotent producer switched on.
4. Publish the reconcile job's outcome for each signature as another record with the same key.
5. In every consumer, drop repeats by key and keep status moving forward only, as [the deduplication page](https://solanashreds.xyz/docs/pipeline/deduplicate) and [the Postgres page](https://solanashreds.xyz/docs/pipeline/postgres) describe.

## kafka.ts

This is the producer half of the tested sink, unchanged. It runs on Node 24 with `@confluentinc/kafka-javascript` 1.11.0 against Apache Kafka 4.3.2.

```typescript file="kafka.ts (excerpt: topic and producer)"
// Kafka sink for decoded transactions, using @confluentinc/kafka-javascript
// (librdkafka based, KafkaJS style API). kafkajs itself has had no release
// since February 2023, so new code should not start on it.
//
// Topic design: key = signature. Kafka hashes the key to pick the partition,
// so every copy of one transaction (shred stream repeats, retries) lands in
// the same partition, in order, and one consumer sees all of them. The slot
// rides in a header so consumers can route or filter without parsing JSON.
import KafkaPkg from "@confluentinc/kafka-javascript";
import type { TxRecord } from "./records.ts";

const { Kafka, logLevel } = KafkaPkg.KafkaJS;

export const TOPIC = "solana.tx.seen";
const brokers = (process.env.KAFKA_BROKERS ?? "127.0.0.1:18392").split(",");
const kafka = new Kafka({ kafkaJS: { brokers, clientId: "solana-tx-sink", logLevel: logLevel.WARN } });

export async function createTopic(topic = TOPIC, partitions = 6): Promise<void> {
  const admin = kafka.admin();
  await admin.connect();
  try {
    // Fixed partition count: adding partitions later remaps keys, so the
    // "same signature, same partition" property only holds per partition count.
    await admin.createTopics({ topics: [{ topic, numPartitions: partitions, replicationFactor: 1 }] });
  } catch (e) {
    if (!String(e).includes("already exists")) throw e;
  } finally {
    await admin.disconnect();
  }
}

export async function produce(records: TxRecord[], topic = TOPIC): Promise<void> {
  const producer = kafka.producer({
    // Idempotence stops broker side duplicates caused by producer retries
    // (it implies acks=all). It does NOT remove duplicates that were already
    // duplicates in the input; consumers still dedupe by key.
    kafkaJS: { idempotent: true, acks: -1 },
    // Batching: wait briefly so many small records share one request.
    "linger.ms": 20,
    "batch.num.messages": 10000,
    "compression.type": "lz4",
  });
  await producer.connect();
  try {
    await producer.send({
      topic,
      messages: records.map((r) => ({
        key: r.signature,
        value: JSON.stringify(r),
        headers: { slot: String(r.slot), tx_version: r.version },
      })),
    });
    await producer.flush({ timeout: 10000 });
  } finally {
    await producer.disconnect();
  }
}
```

The record type it sends:

```typescript file="records.ts (excerpt)"
export type Status = "seen" | "confirmed" | "failed" | "dropped";

export type TxRecord = {
  slot: number;
  entry_index: number;
  tx_index: number;
  signature: string;
  version: "legacy" | "0" | "1";
  fee_payer: string;
  programs: string[];
  status: Status;
  raw_base64: string;
};
```

## What the test run showed

The test publishes the decoded records of two real mainnet blocks, repeating every fifth record to imitate a transaction that shows up twice, then reads the whole topic back.

| Check | Result on 18 November 2026 |
| --- | --- |
| Records published, records read back | 937 and 937 |
| Distinct keys among them | 781, one per transaction in the two blocks |
| Key equals the record's signature | every message |
| Slot header equals the record's slot | every message |
| Both copies of a repeated transaction in the same partition | every repeat |

The transactions are real. Their grouping into entries, and so the `entry_index` field, comes from a locally built shred fixture, because RPC does not expose entries.

## Consumers, duplicates and compaction

The idempotent producer only removes the duplicates its own retries would create on the broker. A transaction that was already in the input twice is published twice, and the test above shows exactly that: 937 messages for 781 keys. Every consumer therefore drops repeats itself, by key.

A compacted topic keeps, in time, only the last record written for each key. Compaction runs in the background and never touches the segment still being written, so a consumer that replays the topic can see both the `seen` record and the later status record for one signature. Do not compact a topic where `seen` and status records share a key: a late `seen` written after the outcome would be the record compaction keeps. If you need compaction, use the two-topic layout and compact only the status topic, where every record is an outcome. In every consumer, still move status forward only, so an old `seen` read after a `confirmed` changes nothing. [Kafka, NATS or Redis Streams](https://solanashreds.xyz/compare/kafka-vs-nats-vs-redis-streams) compares how each bus handles that pattern, and [idempotent writes](https://solanashreds.xyz/blog/idempotent-writes-for-chain-data) covers the database side.

## What a topic does not settle

A message in this topic says a transaction was seen in shreds, nothing more. It carries no execution result, and its block may still be skipped. Kafka adds ordering inside a partition and durable delivery to consumers; it adds no truth about the chain. If your consumers only ever need confirmed data, a stream fed from an RPC or Geyser source after confirmation may serve them with less work than raw shreds plus reconciliation.

## From the topic to storage

Consumers usually land in an analytical store or an operational database: [ClickHouse](https://solanashreds.xyz/docs/pipeline/clickhouse) for large scans, [Postgres](https://solanashreds.xyz/docs/pipeline/postgres) for application queries. If you are joining this stream with history, [backfill and live](https://solanashreds.xyz/blog/backfill-and-live) shows where to cut over. Deciding whether to run this pipeline at all is the subject of [build or buy an indexer](https://solanashreds.xyz/compare/indexer-build-vs-buy). To feed it, a [Standard or Ultra plan](https://solanashreds.xyz/pricing) delivers the raw shreds.
