How do I stream blockchain data to Kafka?

Updated August 2026

There is no managed Kafka sink today. Substreams ships a PubSub sink out of the box — if that fits your infrastructure, use it directly and skip writing a consumer at all.

For Kafka specifically, consume the module's output with the Substreams Sink SDK — client libraries for Go, JavaScript, Python, and Rust — and produce it into Kafka yourself with a standard producer. This is a thin service, not a pipeline built from scratch: Substreams still does the extraction, decoding, and reorg signalling; your code just reads the stream and writes to a topic.

When does Kafka make sense?

Multiple consumers of the same data. A trading service, a notification service, and a warehouse loader all need the same swap events. One topic, three consumer groups.

Existing event-driven architecture. If your organisation already runs Kafka, onchain data becomes just another topic rather than a special case requiring bespoke infrastructure.

Replay. Kafka's retention lets consumers reprocess from an earlier offset without going back to the chain.

Decoupling. Producers and consumers scale and deploy independently.

If only one service needs the data and it wants it in a database, a direct database sink is simpler — that path has a managed sink, so there is no consumer service to write. Kafka earns the extra step when there are several consumers.

How should I structure topics?

One topic per logical dataset — swaps, transfers, liquidity events — rather than one topic for everything. Consumers then subscribe to what they need instead of filtering.

Partition key matters. Keying by contract address or pool gives you ordering per contract and parallelism across contracts, which is usually what you want. Keying by transaction hash gives good distribution but loses per-entity ordering.

Include block number and transaction hash in every message your producer writes, regardless of key. Downstream consumers need them for deduplication and for reasoning about reorgs.

How do reorgs work with an append-only log?

This is the design question Kafka forces you to answer explicitly, and it is better to decide deliberately than discover it later.

Substreams delivers an explicit undo signal when a block is reorged out — your consumer service receives it alongside normal data. Kafka topics themselves are append-only, so a reorged block cannot be retracted from the topic; your consumer has to translate that signal into one of two patterns:

Emit reorg events. On receiving the undo signal, publish an explicit message indicating that blocks from a given height were undone, and let downstream consumers compensate. Most faithful to what happened, most work for consumers.

Delay by confirmation depth. Only produce blocks past a depth where reorgs are vanishingly unlikely, and drop the undo signal on the floor since it will never apply to something you haven't published yet. Simpler for every consumer, at the cost of latency.

Choose based on whether your downstream consumers can tolerate compensating logic. Most teams take the second option unless they need sub-finality latency.

What about ordering and delivery guarantees?

Kafka guarantees ordering within a partition. With a sensible partition key, per-contract ordering holds.

Persist the Substreams cursor in your own consumer, the same way the built-in sinks do, so a restart resumes at the right stream position instead of reprocessing or skipping blocks. Combined with block number and transaction hash in the payload, downstream consumers can deduplicate if a message is redelivered — worth building, since exactly-once end to end is harder than it looks.

Frequently asked questions

Does this work with managed Kafka? Yes — since your own service is producing to Kafka, MSK, Confluent Cloud, and self-hosted clusters all work identically. You need connection details and produce permissions, same as any other Kafka producer.

Can I use PubSub instead? Yes, and it is simpler: Substreams has a built-in PubSub sink, so you skip writing a consumer entirely. Kafka needs the extra step above because there is no managed Kafka sink yet.

How do I backfill an existing topic? Run your consumer from your chosen start block using the SDK's historical range support. Historical processing runs in parallel on the Substreams side, so large ranges arrive quickly — the constraint becomes your own producer's throughput into Kafka.

Get an API key at thegraph.market — no personal information required.