Canonical: https://solanashreds.xyz/blog/research-datasets-from-shreds

# A research dataset from raw shreds: what to keep and how

A dataset is only reusable if someone else can tell what it contains, how it was captured, and where it has holes. Here is what to record from raw shreds, in which formats, and what to write down next to it.

A reusable Solana research dataset records what was captured, how and where, not only the data. Keep raw packets as pcap if anyone may need to re-decode them, store decoded rows as Parquet for analysis, and document the capture point, clock source, gaps and protocol boundaries such as the vote and slot time changes.

## Raw packets, decoded rows, or both

**Raw packets** are the ground truth. Keep them when your questions might change, when your decoder might have bugs, or when timing of arrival matters to the study. They are large and need a decoder before anyone can use them.

**Decoded rows** are what analysis actually runs on: one row per transaction or instruction, with slot, position, signature, program and decoded fields. They are compact and easy to share, but they freeze your decoder's interpretation, mistakes included.

**Both** is the usual answer for serious work. Keep the packets privately as the source, publish rows, and record the decoder version that produced them so a corrected decoder can regenerate the rows. [Archive formats](https://solanashreds.xyz/compare/shred-archive-formats) compares the storage cost of each choice.

## Formats: pcap for packets, Parquet for rows

| Need | Format | Why |
|---|---|---|
| Keep packets exactly as received, with arrival times | pcap | One record per packet with its timestamp; nanosecond resolution is available through the nanosecond variant of the file format; readable by standard tools |
| Analyse transactions and instructions | Parquet | Columnar, compressed, typed; most analysis tools read it directly |
| Hand a small sample to a reviewer | CSV or JSON lines | Readable without tools, fine for small extracts only |

For Parquet, partition by date or slot range, keep signatures and keys as their raw bytes or base58 consistently across all files, and write the schema down rather than relying on inference.

## The metadata a dataset cannot do without

- **Capture point:** which receiver, which network location, which delivery path. Arrival times mean nothing without it.
- **Clock source:** how the receiving machine kept time, and whether timestamps came from the kernel or from your program.
- **Coverage:** first and last slot, and every gap where the receiver was down or packets were lost beyond recovery.
- **Decoder version:** the exact commit and library versions that produced the rows.
- **Status of rows:** whether they are provisional, settled against RPC, or both, as described in [backfill and live](https://solanashreds.xyz/blog/backfill-and-live).
- **Ordering:** whether rows carry the in-block transaction index, which [order inside a slot](https://solanashreds.xyz/blog/order-inside-a-slot) explains how to derive.

## Known boundaries inside recent history

Several protocol changes in 2026 change what the data looks like. A study that crosses them must say so, and ideally split its analysis at each one.

| Change | Boundary | What changes in the data |
|---|---|---|
| Transaction v1 activated | slot 447,120,000, 2026-09-15 | A third transaction format appears next to legacy and v0 |
| 200 ms slots took effect | slot 454,896,000, 2026-10-09 | More slots per unit of time; any per slot statistic shifts |
| Votes left the blocks | from slot 459,653,000, about 13:53 UTC on 2026-10-21 | Vote transactions disappear from block data |

The Alpenglow feature itself activated a few thousand slots before votes left the blocks, so do not use the activation slot as the vote boundary. [Votes left the block](https://solanashreds.xyz/blog/votes-left-the-block) and [200 ms slots and your database](https://solanashreds.xyz/blog/200ms-slots-and-your-database) explain each change from a data point of view, and [versioned transactions](https://solanashreds.xyz/learn/versioned-transactions) covers the formats.

## Licensing and sharing

Put a licence file next to the data and state who may reuse it and how to cite it. Publish the decoder or at least its version, so others can check your rows.

If the packets came from a paid feed, read its terms before publishing raw captures. With SolanaShreds, giving the live stream to anyone outside your own machines requires a written agreement with us, so ask before sharing packet captures. Decoded, aggregated results are a different question from the stream itself; when unsure, ask.

## What a capture from one receiver cannot represent

A capture shows what reached one receiver along one path. It does not show what other nodes saw or when, and loss on your side looks the same as data that never existed until you check against settled blocks. For questions about the whole network, combine your capture with settled RPC data and say which conclusions rest on which. Test fixtures built from the same capture, as in the [decoder test guide](https://solanashreds.xyz/docs/decode/test-fixtures), help show the decoding was right.

## Start recording

Pick your formats and metadata before the first packet, not after. To record your own capture, see the [plans and prices](https://solanashreds.xyz/pricing).

Many datasets start from [newly created accounts](https://solanashreds.xyz/blog/spotting-new-accounts) as an index, and for capture work [the Standard plan is usually the right fit](https://solanashreds.xyz/compare/solanashreds-standard-vs-ultra).

Datasets that span the vote change need the boundary handled explicitly, as [the Alpenglow post for indexers](https://solanashreds.xyz/blog/alpenglow-for-indexers) explains.
