Kafka fits best when many consumers need long replay and a compacted topic that keeps only the latest status per transaction. NATS JetStream is lighter to run and deduplicates on publish. Redis Streams suits a small team with modest retention and one or two consumers. All three can carry provisional rows if keys are chosen well.
What chain data asks of a bus#
Decoded shred data is not ordinary event traffic. Each transaction first appears without an outcome, then later turns out confirmed, failed or dropped, so the same record changes state after it has been published. Consumers join late and need to replay. Several services read the same events for different jobs. And the natural keys, the slot and the first signature, are known the moment a transaction is decoded (E4 tech facts X-01, X-04).
So the useful questions are: can a consumer replay from a point in the past, is order kept per key, can old versions of a record be dropped in favour of the newest, can many consumers read independently, how heavy is the bus to operate, and does it help you avoid duplicates.
Scorecard#
| Criterion | Kafka | NATS JetStream | Redis Streams |
|---|---|---|---|
| Replay from an earlier point | Fits: consumers seek to any retained offset | Fits: consumers start from a sequence or time | Fits: read from any retained entry ID |
| Order per key | Fits: per partition, and a key always lands in the same partition | Fits: per stream; split by subject when needed | Partly: per stream; one stream per key range is on you |
| Keep only the newest version of a record | Fits: log compaction keeps the last value per key | Partly: a per-subject message limit can keep the last message per subject | No: trimming is by length or ID, not by key |
| Many independent consumers | Fits: consumer groups with their own offsets | Fits: durable consumers per reader | Fits: consumer groups with acknowledgements |
| Operations load | Heaviest of the three | Light: a single server binary, clustering when needed | Light if you already run Redis; memory sets the retention ceiling |
| Help with duplicates | Partly: the idempotent producer removes retry duplicates only; dedupe by key in the consumer | Fits: deduplication by message ID within a time window | No: at-least-once; dedupe in the consumer |
"Fits" means the feature exists in the official documentation and maps directly to the chain data need. "Partly" means it works with extra design on your side.
Keys and subjects for slots and signatures#
Key each message by the transaction's first signature. That keeps every version of one transaction in order, lets Kafka compaction or a JetStream per-subject limit hold only the latest status, and makes duplicates easy to drop downstream (deduplicating transactions covers the window). Put the slot and the position inside the slot in the message body, not in the key, because ordering across different transactions is a consumer concern that a per-key bus does not solve.
If a consumer needs whole slots in order, publish a separate small stream of slot markers keyed by slot number, and let that consumer wait for the marker before it processes the slot.
Carrying a row from provisional to settled#
Publish twice. The decoder emits a provisional event the moment it sees a transaction, and the reconcile loop emits a second event with the same key once the outcome is known. Consumers that only want settled data skip the first status; latency-sensitive consumers act on the first and correct on the second. Because both events share a key, compaction eventually keeps only the last event written for it, but a consumer replaying later can still see both, so consumers must also keep status moving only forward. The Kafka producer guide shows the event shape, and idempotent writes explains the matching database side.
Message bus questions#
Which bus keeps order by slot?#
None of them orders across keys for you. Each keeps order within a partition or stream. Keep slot and position in the payload, and add a slot marker stream if a consumer needs whole slots.
Can NATS JetStream drop duplicate events?#
Yes, within its deduplication window, when the publisher sets a message ID. Use the transaction signature plus status as that ID.
Is Redis Streams enough for an indexer?#
For one or two consumers and short retention, often yes. It has no compaction by key, so long histories grow until trimmed.
Does throughput decide the choice?#
Rarely for a single shred feed. The provisional update pattern and the number of consumers matter more than peak rate for this kind of data.
When you do not need a bus at all#
If one process decodes and the same process acts, a bus only adds a hop and something to operate. Write straight to your database or act in memory until a second consumer appears; the decoder placement comparison describes the point where splitting starts to pay. A bus also cannot add data the feed does not carry, such as execution results.
Wire it up#
Start from one raw feed and one producer; the docs take you from packets to decoded transactions, and the event schema post helps you version what you publish.
Sources#
- Apache Kafka documentation: partitions, consumer groups, log compaction, idempotent and transactional producers (checked 2026-11-18)
- NATS documentation: JetStream streams, consumers and message deduplication (checked 2026-11-18)
- Redis documentation: Streams, consumer groups and trimming (checked 2026-11-18)
- Agave source: the Entry struct (no execution result in shred data) (checked 2026-11-18)
Raw shreds on your own server
Ultra Low Latency Shreds is $200 and Standard Shreds is $100. You pay in crypto, tell us which IPv4 address and UDP port should receive the stream, and we wire it up by hand.