Can I get full history and real-time data from one source?

Updated August 2026

Yes. Substreams processes a chain's complete history in parallel and then continues streaming as new blocks arrive, using the same module code for both. You define your transformation once and it serves backfill and live data.

This is unusual. Most blockchain data tooling does one of these well and the other poorly, which is why teams so often end up running two systems.

Why do most teams end up with two pipelines?

Because the two problems pull in opposite directions.

Real-time streaming optimises for latency. Systems built for it — Geyser plugins, Yellowstone gRPC, webhook services — hold a limited buffer of recent data because that is all a low-latency stream needs. Ask them for last year and they cannot help.

Historical access optimises for completeness. Archive nodes and batch extraction tools can reach any block, but they are not built to deliver the current block within a second.

So teams build both: a streaming consumer for live data, a batch job for history, and a reconciliation layer to stitch them together at the boundary. That boundary is where the bugs live — duplicated records, gaps during restarts, and two implementations of the same business logic that drift apart.

	Streaming-only tools	Historical-only tools	Substreams
Live data	Yes	No	Yes
Full history	No — limited replay window	Yes	Yes
Same code for both	—	—	Yes
Reconciliation layer needed	Yes	Yes	No

How does one engine do both?

By separating extraction from transformation. Firehose extracts blockchain data once and stores it as flat files, while also serving live blocks. Historical data and live data are therefore the same kind of thing from the consumer's perspective — a stream of blocks — differing only in whether they come from storage or from the chain head.

Substreams runs your modules over that stream. For history it splits the range into segments and processes them in parallel; when it catches up, it keeps going block by block. The transition is automatic and your module does not know the difference.

What does this change in practice?

One implementation of your logic. A change to how you decode a swap applies to history and live data simultaneously, because it is one piece of code.

Backfills stop being projects. Changing your schema means re-running the module over history, not planning a migration.

No boundary bugs. There is no seam between historical and live data, so there is nothing to reconcile.

Reorg handling is consistent. The same cursor mechanism applies throughout, rather than being handled in the streaming path and ignored in the batch path.

Frequently asked questions

How far back can I go? The chain's full history, from genesis, on supported networks.

Is historical processing slower than streaming? It is bounded by parallel throughput rather than block time, so large ranges are processed far faster than real time. Once caught up, you receive blocks as they are produced.

Do I need to configure the switch from history to live? No. You specify a start block; the engine handles the transition.

Get an API key at thegraph.market and stream both — no personal information required.