A Solana historical data backfill meets a live raw feed cleanly when you fix one cutover slot, start the live receiver first, backfill up to that slot from RPC, and let a short overlap be absorbed by a dedupe key. Completeness is then proven by comparing block lists, not by trusting that both halves ran.
Pick a cutover slot before you start#
The join fails when each half decides on its own where it begins. Write one number down instead: the cutover slot. Everything strictly below it comes from the backfill. Everything at or above it comes from the live stream.
Choose it after the live receiver is already running and writing. Read the oldest slot your live pipeline has stored completely, then pick a cutover a few slots above that value. The live side now owns a short stretch of history it has already seen, and the backfill has a fixed target that will not move while it works.
Store the cutover slot next to your data, in the same database. Later, when someone asks why a row exists twice or not at all, the first question is which side of the cutover it sits on.
The handover, step by step#
- Start the live receiver and decoder, writing rows with their slot and a provisional status.
- Wait until it has written rows for several consecutive slots, then record a cutover slot a few slots above the oldest of them.
- List the slots that have blocks between your history start and the cutover, where the backfill stops at the cutover and the live rows already cover the slots just below it, with the getBlocks RPC method, in ranges of at most 500,000 slots per call.
- Fetch each listed block with the getBlock RPC method and write its transactions with the same row keys the live side uses.
- Run the completeness check described below for the whole range, then for the overlap.
- Mark the backfill finished only after the check passes, and keep the record of which slots it covered.
Backfill with the getBlock RPC method, carefully#
The getBlock RPC method returns a block that the cluster has confirmed, which is what a history needs. Three settings decide whether the result matches what your live decoder produces.
Transaction versions. Legacy, v0 and v1 transactions are all live on mainnet. Ask for maxSupportedTransactionVersion: 1, or the node refuses blocks that contain newer formats. A backfill that silently skips such blocks looks finished and is not.
Full transactions. Request full transaction details. You want the same signed bytes the live side decodes, plus the execution metadata that only RPC can give you, such as the error field.
Skipped slots. Not every slot has a block. The getBlocks list tells you which slots do, so a slot missing from that list is expected, while a listed slot that fails to fetch is an error to retry.
Request pacing depends on the RPC provider you use. Read their published limits and stay under them; a burst of rejected calls is the most common cause of holes in a backfill.
The overlap window and how dedupe closes it#
Because the cutover sits above where the live side started, the slots between the live start and the cutover are covered twice: once by live rows and once by backfill rows. That is deliberate. A small overlap is easy to deduplicate, while a gap between the two halves is invisible until a user finds it.
Use the first signature of each transaction as the row key, with the slot alongside it. When the backfill writes a transaction the live side already stored, the write becomes an update instead of a new row. Rows from the backfill carry confirmed data, so they may upgrade a provisional live row; they should never downgrade one. The idempotent writes post covers the upsert rules, and the deduplication guide shows the key handling in code.
Filling gaps after your receiver was down#
A raw UDP feed has no replay. Packets that arrive while your receiver is stopped are gone, so the history of that period has to come from RPC.
Treat every restart as a small backfill with its own cutover. Note the last slot the receiver wrote before it stopped and the first slot it wrote after it came back. List the blocks between them with getBlocks, fetch them with getBlock, and write them with the same keys. The reconcile guide runs the same loop on a schedule, so a gap you did not notice still gets filled.
Proving the joined data is complete#
"Both jobs finished" is not proof. Compare what you stored with what the cluster says exists:
- For every slot in your range, the set of slots with blocks from getBlocks should match the set of slots in your table.
- For each slot, the number of transactions you stored should match the number getBlock returns.
- For the overlap, no first signature should appear in more than one row.
- Any provisional live row older than the overlap should by now have a confirmed status. If not, the getSignatureStatuses RPC method can settle it.
Run the check per slot range and keep its output. A dataset that ships with its completeness report is one other people can trust, which is the point of the research dataset post.
Where this method stops working#
This approach leans on RPC for everything before the cutover. Public endpoints often keep only recent history, so a long backfill needs a provider that serves old blocks, or an archive you already hold. The comparison is not perfect either: the count check proves you have the same transactions, not that your decoder read them correctly. Decoder tests cover that separately. If you need history from before the 200 ms slot change or before votes left blocks, expect those periods to look different and document the boundary.
Related reading#
Polling versus streaming explains when RPC alone is enough. Slot versus block height covers which number to key rows on. For the live half, Standard Shreds delivers raw packets to the receiver you run.
Sources#
- Solana RPC docs: getBlocks (confirmed block slots in a range, at most 500,000 slots per call) (checked 2026-11-18)
- Solana RPC docs: getBlock (maxSupportedTransactionVersion, transactionDetails) (checked 2026-11-18)
- Solana RPC docs: getSignatureStatuses (up to 256 signatures, searchTransactionHistory) (checked 2026-11-18)
- Solana improvement document 0385, the v1 transaction layout (checked 2026-11-18)
- Agave commitment levels (confirmed and finalized under Alpenglow) (checked 2026-11-18)
Raw shreds on your own server
Ultra Low Latency Shreds is $200 and Standard Shreds is $100. You pay in crypto, tell us which IPv4 address and UDP port should receive the stream, and we wire it up by hand.