How do I backfill historical blockchain data?

Updated August 2026

Specify a start block and run your Substreams module through a sink. History is split into segments and processed in parallel rather than sequentially, and when the pipeline reaches the chain head it continues into live streaming automatically.

There is no separate backfill job to write, and no handover between the historical and live paths.

Why are backfills usually so painful?

Because most indexers process history the same way they process live data — one block after another. That is correct and it is slow: syncing years of a busy chain can take days or weeks, during which you have no usable data.

Worse, it recurs. Change your schema, add a field, fix a decoding bug, and you start again. Teams end up treating any change to historical logic as a multi-week project, which quietly discourages fixing things.

What makes parallel processing possible?

Two properties working together.

Data is already extracted. Firehose has written the chain's execution record to flat files. Reading block 5,000,000 does not require replaying everything before it — the file is just there.

Modules are structured for independence. Substreams modules are defined so historical segments can be computed separately and merged. That is why the programming model imposes map and store module types rather than letting you write arbitrary stateful code.

The result: throughput is bounded by parallel capacity, not by block time.

How do I plan a backfill?

Pick a genuine start block. Not genesis by reflex — the block your contract was deployed, or the earliest date your application actually needs. This is the single biggest lever on how long the backfill takes.

Test on a narrow range first. Run a few thousand blocks, inspect the output, verify it against a block explorer or an existing query. Finding a decoding error after processing five years is avoidable.

Tune batch sizes for the backfill. Streaming defaults are conservative. Larger batches materially improve ingest throughput, and can be reduced once you are tailing live.

Consider a staging table. For large loads into a table already serving queries, write to staging and swap.

How do I know it's correct?

Verify independently. Pick a set of blocks, compare your output against a block explorer or a known-good query, and confirm counts and values match. Do this before the full run, not after.

Watch for the failure that is not obvious: internal transactions and state changes never appear in event logs, so if your expectations were formed by a log-based tool, complete extraction may legitimately produce more records than you anticipated. That is usually correct, not a bug.

What happens at the end?

Nothing you have to do. The pipeline reaches the chain head and continues into live streaming with the same module and the same sink. The cursor carries across, so there is no seam and nothing to reconcile.

Frequently asked questions

Can I backfill into a database that's already serving live data? Yes, though a staging table and swap is safer for large loads.

What if the backfill fails halfway? The sink resumes from its last committed cursor rather than restarting.

Do I need an archive node? No — see do I need an archive node?

Get an API key at thegraph.market — no personal information required.