What is a flat-file blockchain dataset?

Updated August 2026

A flat-file blockchain dataset is a chain's history extracted once and written to files — typically protobuf-encoded blocks stored in object storage — rather than being queried repeatedly from a node. Consumers read the files directly, in parallel, as many times as they like.

It is a simple idea with a large consequence: it decouples how often you read blockchain data from how much load you put on a node.

Why does this matter?

Because the usual alternative scales badly. If your data pipeline reads history over JSON-RPC, every consumer, every re-run, and every schema change means asking a node for the same data again. The node becomes both a bottleneck and a single point of failure, and reading history in bulk means an enormous number of sequential round trips.

With flat files, extraction happens once. After that, reading is a storage problem rather than a node problem — and storage parallelises trivially.

	Node / RPC access	Flat files
Cost of reading history again	Full re-query	Read files again, free
Parallel consumers	Compete for node capacity	Independent
Re-running after a schema change	Re-extract everything	Re-read files
Node load	Grows with consumers	Constant

What's actually in the files?

Complete block data as captured during execution — not just the published results. That includes transactions and their receipts, but also the execution detail that logs never expose: call trees showing internal transactions, state changes, and balance changes.

That last point is the practical difference. If a contract moves value through an internal call without emitting an event, a log-based pipeline cannot see it. It is present in the flat-file record because the record was made while the block executed rather than reconstructed afterwards.

How does this relate to Firehose?

Firehose is the system that produces these files. An instrumented node emits data as it executes, a reader captures it, and a merger consolidates blocks into files in object storage. Firehose then serves that data as a stream, backed by the files for history and by live blocks at the chain head.

Substreams reads this same storage, which is why it can process historical segments in parallel — the segments are just files, and files can be read simultaneously.

Is this the same as an archive node?

No, and the distinction is worth being precise about. An archive node stores full historical state, so you can ask what a contract's storage looked like at a given block. Flat files store the full historical record of execution — what happened in each block, in order, with complete detail.

For most data pipelines the flat-file record is what you actually need, and it is far cheaper to read. See do I need an archive node? for the longer comparison.

Frequently asked questions

What format are the files in? Protobuf-encoded blocks, merged into files and stored in object storage such as S3 or GCS.

Do I have to store them myself? No. You can self-host the full stack, or consume hosted endpoints and never handle files directly.

Can I query the files with SQL? Not directly — they are a streaming source, not a warehouse. You transform them into your own tables using a sink.

Read the data without operating the storage — get an API key at thegraph.market, no personal information required.