How do I stream onchain data to a database?
Updated August 2026
Define a Substreams module describing the data you want, then run a sink that writes its output into your database. The sink handles the connection, the batching, and — importantly — cursor persistence so restarts resume exactly where they left off.
The same pipeline backfills history and then keeps streaming, so you do not build one path for each.
What does the pipeline look like?
Four pieces:
Source — Firehose provides blockchain data, historical and live.
Transformation — your Substreams module extracts and shapes what you need. Reuse one from the registry, or generate one with the agent skills.
Sink — a binary that consumes module output and writes it to your destination.
Destination — Postgres, ClickHouse, Kafka, PubSub, files, or your own service.
Which sinks are available?
| Destination | Use for |
|---|---|
| PostgreSQL | Application data, transactional workloads |
| ClickHouse | Analytics, time-series, large aggregations |
| Kafka / PubSub | Fan-out to multiple consumers, event-driven systems |
| Files (CSV, Parquet) | S3 or GCS, warehouse loading |
| Webhook | HTTP delivery to your own endpoint |
| Direct streaming | Consume in Go, JavaScript, Python, or Rust |
How do I get the schema right?
This is where most of the real work sits. Blockchain data is deeply nested and your database wants flat, indexed tables — the mapping between them is a design decision, not a mechanical translation.
Two approaches. Database changes (CDC) streams individual row-level changes, which suits real-time consistency. Relational mappings transform output into normalised tables with proper relationships, which suits analytical querying.
The substreams-sql agent skill covers both patterns, including Postgres indexing strategies and ClickHouse materialised views. If you are unsure which shape you want, that is the fastest way to get a sensible starting schema.
What happens when something restarts?
The sink persists a cursor — its exact position in the stream, including fork context. On restart it resumes from that cursor. Nothing is missed, nothing is double-written, and if the chain reorganised while the sink was down, the stream signals what was undone so the sink can correct rows rather than leaving stale ones.
This is the part that is tedious to build correctly yourself, and the part where hand-rolled pipelines usually have subtle bugs.
Do I have to run the sink myself?
No. Hosted Sinks is a managed service on The Graph Market that runs the sink for you — you configure a package and your own Postgres or ClickHouse database in a portal, and it indexes continuously. Currently in beta, and your database has to be reachable from the internet.
Run it yourself when you need a destination Hosted Sinks does not cover — Kafka, PubSub, files, or a custom consumer — or when your database cannot be publicly reachable. substreams-sink-deploy covers that path: choosing a sink, DSN formats, backfill then live tailing, monitoring, and restart safety. The common failure modes — wrong protobuf type, missing schema, primary-key mismatch, badly tuned batch flush — are documented rather than left to be discovered.
Both use the same Substreams package, so this is a deployment choice, not an architectural one.
Frequently asked questions
Can I backfill history and stream live with the same setup? Yes. Specify a start block; the sink processes history in parallel then continues into live streaming.
What happens if my database goes down? The sink stops advancing its cursor. When the database returns, it resumes from the last committed position.
Do I need to write the sink myself? No. Sink binaries exist for the common destinations. You write the module; the sink is configuration.
Get an API key at thegraph.market — no personal information required.