Start by raising the number of parallel workers, which you control with the X-Substreams-Parallel-Workers header. Then make sure the sink is not the bottleneck. Both matter: parallel workers speed up extraction, but the database write path often limits the total.
How many workers should I use?
More workers help until the work runs out. In the BuildersDAO Uniswap V4 benchmark, Ethereum (about 1.8M blocks) went from 8h 12m at 1 worker to 7m 36s at 75 workers and barely improved at 300. Base (about 11.4M blocks) kept improving: 2h 06m at 75 workers, 56m 38s at 300.
Rule of thumb: short ranges plateau early, long ranges keep gaining. Full numbers are in How much faster is Substreams than a subgraph?.
What should I check in the sink?
With the built-in substreams sink postgres and substreams sink clickhouse commands (CLI v1.21 and later, with the biggest changes in v1.22):
- Disk headroom. Data downloads to a local spool (8 GiB by default) and is committed to the database asynchronously. A slow database no longer stalls the stream, but the sink host needs disk space.
- Deferred constraints. Postgres constraints are applied when the sink reaches chain head, or on demand with
substreams sink postgres constraints apply. Don't create them yourself up front. - Materialised views. Heavy views refreshing during the backfill were the biggest bottleneck BuildersDAO found. Ingest the raw tables to chain head first, then build views.
- Database sizing. If the database is the limit, adding workers will not help.
What else affects speed?
The module. A module that decodes only what it needs is cheaper per block. Substreams 0.8.0 generates Rust bindings with buffa instead of prost, which decodes blocks 2.9x to 4.9x faster, with real backfills up to 2.9x faster end to end. See how to upgrade to Substreams 0.8.
Map modules over stores. Where you can, build the pipeline from map modules only. A store module has to be processed before any module downstream of it can run, so each store in the dependency chain adds a stage that must finish first and limits how much of the backfill runs in parallel. Reach for a store only when you need state that accumulates across blocks, and consider whether a Hosted Store or a database lookup fits instead.
The start block. Start at the contract's deployment block, not genesis.
Cached output. Module outputs are cached, so rerunning an unchanged module over a range already processed is much faster than the first run.
Frequently asked questions
Does the worker count affect live streaming? No. It applies to historical processing; once the sink reaches chain head it streams live.
Will changing the worker count change my results? No. It changes how the work is split, not what the module produces.
Get an API key at thegraph.market.
Related: How do I backfill historical blockchain data? · How do I get blockchain data into Postgres? · How do I get blockchain data into ClickHouse?