Keep pcap when you want to replay packets exactly as they arrived, a length-prefixed binary log when you want the same bytes in a smaller, simpler file, and Parquet when you want to query decoded transactions. Raw formats preserve everything for future decoders; Parquet is only as good as the decoder that produced it.
Decide what you will want later#
An archive is a promise to your future self, so start from the question you expect to ask. Replaying a day of traffic through a new decoder needs the original datagrams with their arrival times. Regression tests need a small, fixed set of real packets. Research and backtests usually need decoded transactions with slot, position and signature columns. Each need points to a different format, and the cheapest archive is the one that serves your real question without keeping what you will never read.
Three archive formats compared#
| pcap | Length-prefixed binary log | Parquet of decoded rows | |
|---|---|---|---|
| What is kept | Every datagram, with capture timestamps and link headers | Every datagram payload with your own timestamp | The transactions your decoder produced, as columns |
| Replay fidelity | Exact, including arrival order and timing | Exact payloads; timing as precise as you recorded it | None at the packet level |
| Queryable without code | No | No | Yes, with any Parquet-aware engine |
| Tooling | tcpdump, Wireshark, every pcap library | Your own reader and writer | Query engines and dataframe libraries |
| Survives a decoder bug | Yes, decode again | Yes, decode again | No, rebuild from a raw archive |
| Growth per day | Formula below, with per-packet header | Formula below, with your record header | Depends on decoded row count and compression |
Estimating storage from your own capture#
Measure, do not guess. Capture one hour, count the datagrams, and note the average payload size. Then:
bytes per day = datagrams per second × (average payload + record overhead) × 86,400
For pcap, the record overhead is the 16-octet per-packet header plus whatever link-layer, IP and UDP headers your capture keeps; the file adds one 24-octet header. For a binary log it is the length prefix plus the timestamp you choose to store. Payloads from our feed are at most 1228 bytes each, so the payload term is bounded, but your average comes from your capture, not from that maximum. Compression changes the result again, so compress one hour and use the measured ratio.
For Parquet, estimate from decoded rows instead: rows per second times the compressed bytes per row you see in a test file.
Keeping raw and decoded side by side#
The safe layout is a short raw window plus long decoded history. Keep pcap or a binary log for as many days as you need to re-run a decoder after a fix, and keep Parquet for as long as anyone queries it. When a program changes its layout and your decoder learns about it, decode the raw window again and replace the affected Parquet partitions. A small, labelled slice of the raw archive also makes the best regression fixtures, and the fixture guide shows how to store it.
Archive questions#
Is pcap or Parquet better for shreds?#
They answer different questions. pcap keeps the packets for replay; Parquet keeps decoded transactions for queries. Most teams keep a short pcap window and long Parquet history.
Can I rebuild Parquet from pcap later?#
Yes, as long as the pcap window still covers the period and your decoder can read it. You cannot go the other way.
Should I capture with link-layer headers?#
Only if you need them. A pcap file always keeps the link, IP and UDP headers; if you want payloads only, write a length-prefixed log instead.
How do I check a capture is complete?#
Compare the slots you decoded with the slots RPC reports as produced, the same check used when you verify delivery.
When an archive will not answer your question#
A raw archive holds what the feed carried and nothing else: no balances, logs or execution results, which shreds never include (E4 tech facts X-01, X-02). Research that needs outcomes must join confirmed data from RPC or another source. And no archive can replace packets your receiver dropped, since a live feed never resends them.
Start capturing#
A Standard feed is the usual choice for research capture, since a capture is judged by completeness and both plans carry the same shreds. Pair it with a database for the decoded side; ClickHouse or Postgres helps you pick, and the research dataset post covers documentation.
An archive is also what lets you join backfill with the live stream without a gap.
Sources#
- tcpdump and libpcap: the pcap savefile format (24-octet file header, 16-octet per-packet header) (checked 2026-11-18)
- tcpdump manual page (checked 2026-11-18)
- Apache Parquet documentation (checked 2026-11-18)
- Agave source: the Entry struct (checked 2026-11-18)
Raw shreds on your own server
Ultra Low Latency Shreds is $200 and Standard Shreds is $100. You pay in crypto, tell us which IPv4 address and UDP port should receive the stream, and we wire it up by hand.