Substreams 0.8.0: A New Protobuf Codec Makes Backfills Up to 2.9x Faster

Every Substreams module pays the same computational tax before it does any work of its own. The runtime hands it a protobuf block, the module decodes it, runs its logic, and re-encodes the result. That decode is paid once per block per module, so a chain with ten active modules decodes every block ten times over. None of it is your logic. It is setup.

High-frequency chains produce blocks several times faster than the chains that were there when Substreams launched, and the blocks themselves have grown: a median block on the Base Network costs an order of magnitude more to decode than a median Ethereum block. Backfills are where you notice, because a reprocessing job replays years of history through that same per-block cost.

Protobuf decoding can take up to 90% of the total compute of processing a single block. Anthropic announced a new performant Rust protobuf implementation recently, so we went and looked at the decoder right away.

Substreams 0.8.0 now generates its Rust protobuf bindings with buffa instead of prost. Modules decode blocks 2.9x to 4.9x faster depending on the chain, and the real backfills we measured finished up to 2.9x faster end to end. The new implementation brings zero-copy views, lazy decoding and a few other high-performance knobs. The protobuf wire format is identical under both, so nothing downstream changes, and upgrading is opt-in.

Why the decoder was the bottleneck

prost decodes eagerly. It walks the whole block and builds owned Rust structs for all of it, every transaction and log and nested message, allocating as it goes. Your handler then reads the two or three fields it actually cares about.

buffa's lazy views do not do that. They borrow string and bytes fields directly out of the input buffer instead of copying them, and leave nested messages encoded until something reads them, so a handler only pays to decode the fields it touches. A module pulling swap events out of a Solana block stops paying for the hundred other things in that block.

That is the whole idea. The win is largest where blocks are biggest and modules are pickiest, which happens to describe where the pain already was.

What we measured

First the codec on its own: real mainnet blocks, native Rust, no WASM. Small, median and large are the p10, p50 and p95 blocks by encoded size from a 2,000-block scan per chain. Each figure is the median of nine runs, and every library was checked to round-trip each block to an identical result before anything was timed.

A full table of the benchmark results can be found in the Appendix at the end of this article.

We compared buffa lazy views against prost stock and against prost with its bytes feature enabled, which makes generated bytes fields bytes::Bytes rather than Vec<u8>. It is a real 1.4x to 1.9x at decode for one line of config, and worth knowing about if you are staying on prost for now. Reusing buffers across calls made no measurable difference for either prost or buffa.

One thing the numbers make obvious: there is no single number here. Solana gains the most at decode, 4.9x, and the least at encode, 1.4x. Quoting one figure for both would misrepresent both.

What it looks like end to end

Codec benchmarks are the easy part. What matters is whether a real module finishes sooner. We ran two production modules over real ranges against production data, on Apple Silicon (darwin arm64), median of three runs.

Module	Chain	Blocks	prost	buffa	Speedup
uniswap-v4 map_events	Ethereum (22.0M–23.5M)	1.5M	342 s	118 s	2.90x
solana-idl-history map_idl_accounts	Solana (450.0M–451.5M)	1.5M	846 s	384.5 s	2.20x

Two things are worth drawing out.

On Ethereum, the more recent the blocks, the bigger the gain. The newest 1.5M blocks ran 2.90x faster; the oldest 1.5M blocks we tested, 16.0M–17.5M, 1.74x. Bigger, busier blocks mean a bigger win, so this gets more useful as chains get heavier rather than less.

On Solana the end-to-end figure of 2.20x is lower than the 4.4x decode number, and that is expected. End to end includes your module's own logic and the encode on the way out, where buffa's margin is a narrower 1.4x. Decode stops being the bottleneck and whatever sits behind it becomes one.

Your handler code gets shorter

The ergonomics changed along with the speed. Nested messages that were all typed as Option<T> are now MessageField<T>, which dereferences to a default instance when unset, so most nested access no longer needs a guard. Enum fields are EnumValue<E> instead of a bare i32.

prost	buffa
blk.header.as_ref().unwrap().number	blk.header.number
msg.field.is_some()	msg.field.is_set()
msg.field = Some(value)	msg.field = value.into()
Status::try_from(msg.status)? == Status::Active	msg.status == Status::Active
MyMessage::decode(&*bytes)?	MyMessage::decode_from_slice(&bytes)?

The .as_ref().unwrap() chain is gone from most handlers.

Upgrading

Upgrading is optional. A module pinned to substreams 0.7 keeps building and streaming, cargo update will not move it to 0.8.0, and the wire format is the same either way: existing packages keep streaming, cached module outputs stay valid, sinks read the same bytes, and Go or JavaScript consumers need no change at all. A new project from substreams init already uses buffa.

If you are moving an existing module, every crate has to come from the same row. Mixing rows puts two incompatible copies of substreams in the dependency tree.

substreams	substreams-ethereum	substreams-solana	substreams-database-change	Protobuf
0.8.0	0.12.0	0.16.0	5.0.0	buffa
0.7.x	0.11.x	0.15.x	3.x, 4.x	prost
0.6.x	0.10.x	0.14.x	2.x	prost

Update your dependencies:

 [dependencies]
-prost = "0.13"
-prost-types = "0.13"
-substreams = "0.7"
-substreams-ethereum = "0.11"
+buffa = "0.9.2"
+buffa-types = "0.9.2"
+substreams = "0.8.0"
+substreams-ethereum = "0.12.0"

Then delete buf.gen.yaml and src/pb, and run substreams build to regenerate. This also needs Substreams CLI v1.25.0 or later, the release where substreams build and substreams protogen started generating buffa bindings for any project whose Cargo.toml depends on buffa.

The CLI reads your Cargo.toml to decide which protobuf plugin to use, so add buffa to [dependencies] before you build or you will get prost bindings that do not compile against 0.8.0. Upgrading the CLI on its own changes nothing for a prost project.

substreams-skills has been updated for buffa as well, so the AI coding Skills we shipped for Claude Code, Cursor and VS Code generate against the new APIs rather than the prost ones. If you build Substreams with an AI assistant, update the Skills along with your crates.

The full walkthrough, including troubleshooting for the two errors we hit most often during our own migration, is in Migrating from prost to buffa.

Getting started

• Starting fresh: substreams init already gives you buffa.
• Moving an existing module: follow Migrating from prost to buffa.
• Hitting an error: the troubleshooting section covers duplicate alloc symbols and unsatisfied Message trait bounds, which between them account for most of what we ran into.

If you hit something the guide does not cover, join the Discord and tell us what you found.

Appendix: Benchmarks

Median per-block time, native Rust, real mainnet blocks. Speedup over prost stock in parentheses. Lower is better.

Decode benchmarks

Ethereum decode

Block size	prost	prost + bytes	buffa
Small	0.8950 ms (1.0x)	0.4651 ms (1.9x)	0.2393 ms (3.7x)
Median	1.8205 ms (1.0x)	0.9611 ms (1.9x)	0.4961 ms (3.7x)
Large	2.4703 ms (1.0x)	1.3448 ms (1.8x)	0.6576 ms (3.8x)

Base decode

Block size	prost	prost + bytes	buffa
Small	6.0469 ms (1.0x)	3.7501 ms (1.6x)	2.0682 ms (2.9x)
Median	7.9017 ms (1.0x)	4.8873 ms (1.6x)	2.6242 ms (3.0x)
Large	14.4194 ms (1.0x)	9.1413 ms (1.6x)	5.0012 ms (2.9x)

Solana decode

Block size	prost	prost + bytes	buffa
Small	2.7674 ms (1.0x)	1.7508 ms (1.6x)	0.6961 ms (4.0x)
Median	4.1832 ms (1.0x)	2.8293 ms (1.5x)	0.9472 ms (4.4x)
Large	5.3339 ms (1.0x)	3.7439 ms (1.4x)	1.0791 ms (4.9x)

Encode benchmarks

Ethereum encode

Block size	prost	buffa
Small	0.2191 ms (1.0x)	0.1303 ms (1.7x)
Median	0.4578 ms (1.0x)	0.2788 ms (1.6x)
Large	0.6527 ms (1.0x)	0.3980 ms (1.6x)

Base encode

Block size	prost	buffa
Small	2.0857 ms (1.0x)	1.1424 ms (1.8x)
Median	2.6309 ms (1.0x)	1.4809 ms (1.8x)
Large	4.6473 ms (1.0x)	2.5890 ms (1.8x)

Solana encode

Block size	prost	buffa
Small	0.5999 ms (1.0x)	0.4221 ms (1.4x)
Median	0.9326 ms (1.0x)	0.6847 ms (1.4x)
Large	1.1897 ms (1.0x)	0.8725 ms (1.4x)