How reliable is AI-generated blockchain indexing code?

Q: Which model was used?

A single model, sonnet-4-6. Skill quality is largely model-orthogonal, but multi-model coverage was not part of this pass.

Updated August 2026

For Substreams specifically, it has been measured: in a published evaluation of 14 tasks built from plain-English prompts, 100% of the generated projects built and ran, and 12 of 14 produced output matching a golden reference byte for byte.

That is a narrow, mechanical claim, and it is deliberately narrow. "Does it build, does it run, does the output match" is checkable. "Is it good code" is not.

What was actually tested?

An AI agent was given a plain-English prompt for each task, working in an isolated environment with no access to the reference output or to other tasks. The resulting project was then scored mechanically: substreams build for the build axis, substreams run for the run axis, and a diff against a golden JSONL reference for correctness.

The 14 tasks spanned both EVM and Solana, and ranged from trivial to genuinely awkward:

Task type	Examples	Result
Basic extraction	Block stats, USDC transfers, NFT mints	100%
DEX decoding	Uniswap V2 swaps, Uniswap V3 with USD pricing, cross-DEX volume	96–100%
Solana programs	Slot stats, SPL transfers, Raydium CLMM swaps, Pump.fun launches	100%
No ABI / no IDL	Uniswap V2 from Solidity source, Marinade from Rust source	100%
Sink deployment	Substreams to Postgres	537/537 rows

The two no-ABI tasks are the most interesting result. The agent derived an event signature from Solidity source and an Anchor discriminator from Rust source — reconstructing the interface rather than being handed it.

Where does it fail?

The evaluation is explicit about this, and the failure modes matter more than the headline number if you are deciding whether to rely on it.

Vague prompts produce confident guesses, not questions. Two tasks were intentionally underspecified. The agent shipped working pipelines with hardcoded thresholds and invented token universes rather than asking what was meant. Skill text does not override model posture — the fix is on your side: be specific.

Non-standard tokens break field decoding. MKR returns bytes32 from symbol() rather than a string, which caused decode failures on roughly 4% of swaps in one task. The failure was visible rather than silent — the swap was still emitted, with an empty symbol.

Field naming drifts without a schema. When the prompt does not specify output field names, the agent picks reasonable but inconsistent ones — txCount versus transactionCount. If downstream consumers depend on the schema, state it in the prompt.

How should I use this?

Treat generated Substreams the way you would treat any generated code that touches production data. Be specific in the prompt, run it over a block range you can verify independently, and check the output against a block explorer or an existing query before you trust a backfill.

The evaluation used a single model and the harness itself is not public, so it is evidence rather than proof. But it is measured evidence, published with its own failure modes attached — which is more than most claims about AI code generation come with.

Frequently asked questions

Can I reproduce the evaluation? The results, per-task breakdown, and 16 worked examples are in the public repository. The harness and per-trial transcripts are not published.

Which model was used? A single model, sonnet-4-6. Skill quality is largely model-orthogonal, but multi-model coverage was not part of this pass.

Does this mean I don't need to review the code? No. It means the code will build, run, and very likely be correct — not that you should skip verifying it against data you can independently check.

Try it yourself: install the skills from github.com/streamingfast/substreams-skills and get an API key at thegraph.market, no personal information required.