Rebuilding how Helium stores and queries network data

We are moving the Helium network's data off gzipped protobuf files in S3 and onto Apache Iceberg tables, queried with Trino through an Iceberg REST catalog, with Cloudflare R2 underneath. It is underway rather than finished: some tables are fully across, others are still written both ways while we check the new path against the old. The switch-over is due to complete on Nov 1, 2026. One scoping note before anything else: this is the Mobile network only. IoT runs its own oracles and its own tables, and nothing below describes them.

The old design optimized for the write: files in a bucket are cheap, simple and immutable. But querying them was the problem.

The old pipeline

For most of the Helium Mobile network's life, the answer to "where does this data go?" was: into a gzipped protobuf file in S3.

An oracle service — ingest, the packet verifier, the mobile verifier — took a protobuf message, serialized it, and appended it to a length-delimited frame stream. That stream was gzipped as it was written, rolled over on a timer or at 50 MB, and uploaded to S3 under a key like data_transfer_session_ingest_report.1782427072597.gz. The millisecond timestamp in the filename was the only index that existed.

That design has real virtues. It is cheap: object storage is the least expensive durable thing you can buy, and gzipped protobuf is compact. It is simple: no schema registry, no coordinator, no cluster to keep alive — a writer that can PUT to a bucket is a complete implementation. It is immutable by construction, which matters when those files are the audit trail behind on-chain token emissions. And because the timestamps are fixed-width, keys sort chronologically, so "everything after this point" is one ListObjectsV2 call with StartAfter.

The problem

Four things made that data hard to live with, and together they meant nobody could get a full view of the network without writing a program first.

• You could not query it. Reading meant replaying: list keys after a watermark, download each object, gunzip it, decode frame by frame, skip what you had already seen. Every consumer carried its own copy of that state machine and its own durable cursor.

• A full view meant downloading everything. There is no predicate pushdown on a gzipped blob. Any aggregate, join or filter over a month of traffic had to pull and decode a month of files — which is why the mobile verifier kept its own Postgres copy of whatever the reward math needed: a partial, hand-picked shadow of the lake, useless for any question nobody had anticipated.

• It cost real money to read. S3 bills egress, and a replay is a full re-read. The more anyone asked of the data, the more it cost — a bad incentive to put in front of people trying to understand the network.

• The schema lived in the code. The protobuf definitions were the schema, so nothing about a .gz file in a bucket told you what was inside it. You needed the matching generated types, the right decoder, and the knowledge of which message a given prefix holds. An analyst with a SQL client had no way in at all.

So any new question — how much data a carrier moved last Tuesday, which hotspots heartbeat but pass no traffic, what rejections look like by region — started as a ticket, became a Rust program, and ended as a table someone had to maintain.

Meanwhile the data kept growing, and so did the number of people who needed to reason about it: reward engineering, BD, carrier partnerships, support. None of them wanted to write a file poller. They wanted SELECT.

flowchart LR

    ORC[Oracles<br/>ingest, verifiers]

    S3[(S3 bucket<br/>gzipped protobuf)]

    POLL[File poller<br/>list, download, decode]

    PG[(Postgres<br/>partial copy)]

    OUT[Rewarder]

    AN[Analyst]

    ORC -->|PUT .gz| S3

    S3 --> POLL

    POLL -->|INSERT| PG

    PG --> OUT

    AN -.->|no way in| S3

Everything downstream of the bucket is replay machinery, and Postgres holds only the columns some consumer already decided to keep. The dotted line is the whole problem: there is no path from a question to the data without writing a program first.

That gap is what this migration set out to close.

Solution

The new stack is three pieces, and the important part is that they stay separable.

• Apache Iceberg is the table format. Data still lands as immutable files in object storage — Parquet now rather than gzipped protobuf — but alongside them sits metadata describing the columns, the partitioning, and which files belong to which committed snapshot.

• An Iceberg REST catalog sits in front of those tables. It is the service that answers what tables exist in the hotspots namespace and where the current metadata for heartbeats lives. Writers and readers both talk to it, which is what keeps them agreeing.

• Trino is the query engine. It reads the catalog, plans across the Parquet files, and speaks SQL to anything that can open a JDBC connection.

None of them owns the data. The data is still files in a bucket, and that was never up for negotiation: it is what makes the reward pipeline auditable. What changes is that the files describe themselves.

flowchart LR

    ORC[Oracles<br/>ingest, verifiers]

    OBJ[(Object storage<br/>Parquet)]

    CAT[Iceberg REST catalog]

    TR[Trino]

    OUT[Services, analysts<br/>dashboards]

    ORC --> OBJ

    ORC -.->|commit snapshot| CAT

    OBJ --> TR

    CAT -.-> TR

    TR --> OUT

Solid lines carry data, dotted lines carry metadata commits. Where a table has made the move, the replay machinery goes with it: the mobile verifier arrives at its own inputs the same way an analyst does, through SQL, and everyone reads the same tables the writers committed.

Why Iceberg

What the data needed was to behave like a table — something a query engine can plan against, and something a person outside the team can point a query at without being handed a decoder first.

Iceberg supplies that. The files in the bucket stop being a pile and start being a table that declares what it holds: which files are part of it right now, what the columns are, how it is partitioned. Point an Iceberg-aware engine at the catalog and it works out what to scan and what to skip, without being told anything by us.

Iceberg is an open table format for very large analytic tables. Three of the properties it advertises are the ones that did the real work here.

The schema is a declared artifact, not a side effect. A table is defined once in the codebase — fields, nullability, partitioning — and that definition is what creates it in the catalog. So a query knows a column is never null and never needs a COALESCE, and a filter on the partition column prunes files instead of scanning the table. Both facts are readable by a tool now, not only by someone who knows the codebase. And because Iceberg tracks columns by id rather than by position, adding, renaming or reordering one is a metadata change: no table rewrite, and no zombie data resurfacing out of old files.

Atomic commits, and many writers at once. A commit is a metadata pointer swap, not a directory of files everyone agrees not to touch, so a reader never sees half a batch and several engines can write the same table concurrently. Under the old design a consumer could list an object that was still uploading, and every mitigation for that was a heuristic — wait a bit, rewind the watermark, retry.

Hidden partitioning. Iceberg derives partition values itself from the column it is told to partition on, and skips the files a query cannot need. Nobody writing SQL has to know the physical layout, or remember to add the extra predicate that makes pruning happen — and the layout can change later without rewriting history or breaking queries already written against it.

None of that is bespoke. It is what Apache Iceberg is for; the work on our side was writing into it and reading back out.

That is what makes the next part possible. The tables sit in R2, which does not bill egress, so reading them costs the reader nothing per query and costs us nothing either. Someone can stand up their own Trino, point it at the catalog, and query the network directly — what traffic a hotspot carried, what the rewarder paid, which records were rejected and why. No API to build, no export pipeline to maintain, and no bill arriving for someone else's curiosity. For a network whose payouts are meant to be verifiable, that is the difference between publishing a number and publishing the data behind it.

What Iceberg does not do is maintain itself.

Compaction

Streaming writers produce small files. Each commit lands whatever accumulated in the last batch window, so a table taking continuous writes ends the day with thousands of objects of a few megabytes each rather than a handful of large ones.

That is slow and expensive to query. The engine has to open every file it cannot rule out and read a footer from each, so query cost starts tracking file count rather than data size — and against object storage, each of those opens is a network round trip. Metadata grows the same way: every snapshot keeps its manifest, and old files stay where they are until something removes them.

Compaction is the fix, and Iceberg supports it directly. OPTIMIZE rewrites a partition's small files into fewer large ones, EXPIRE_SNAPSHOTS drops history past a retention window, and REMOVE_ORPHAN_FILES clears what the rewrites left behind. We run all three on a schedule per table: discover the tables from the catalog, pick the partitions holding enough undersized files to be worth rewriting, and leave partitions that are still taking writes alone until they settle.

What's in there

Three namespaces, each mirroring a stage of the pipeline that produces it.

Always filter on the partition column. That predicate is what lets Iceberg skip files; without one, a query reads the whole table, however narrow the result looks:

SELECT hotspot_pubkey, count(*)

FROM mobile.hotspots.heartbeats

WHERE received_timestamp >= TIMESTAMP '2026-09-21 00:00:00 UTC'

  AND received_timestamp <  TIMESTAMP '2026-09-22 00:00:00 UTC'

GROUP BY hotspot_pubkey

It has to be the partition column itself — filtering on some other timestamp in the same row prunes nothing, even when the two move together. And multiplier_ticket_history is the exception to all of this: it is bucketed by hotspot_pubkey, so a filter on the pubkey prunes it and a date range does not.

For the actual schema, ask the catalog rather than this post: SHOW TABLES IN mobile.hotspots, DESCRIBE mobile.hotspots.heartbeats, or query mobile.information_schema.columns.

Connecting to it

The catalog is a standard Iceberg REST endpoint, so anything that speaks Iceberg can read these tables. Every client needs the same two facts: the catalog is at https://catalog.mobile.helium.io/api/catalog with warehouse mobile, and the storage behind it is reached anonymously. Those are the live values, not placeholders — they work as written.

Trino is what we run, so that config is the one we can vouch for. The others are the same values in each project's own spelling.

Trino

Trino's own docs are the reference — Deploying Trino for standing the server up, and the Iceberg connector page for every property below and the many more we are not using.

# etc/catalog/mobile.properties

connector.name=iceberg

iceberg.catalog.type=rest

iceberg.rest-catalog.uri=https://catalog.mobile.helium.io/api/catalog

iceberg.rest-catalog.security=NONE

iceberg.rest-catalog.warehouse=mobile

iceberg.rest-catalog.vended-credentials-enabled=false

iceberg.security=READ_ONLY

fs.native-s3.enabled=true

s3.region=us-east-1

s3.endpoint=https://helium.io

s3.path-style-access=false

s3.aws-access-key=anonymous

s3.aws-secret-key=anonymous

PyIceberg

The lightest way in, and the one that needs no cluster at all. Define the catalog in ~/.pyiceberg.yaml:

catalog:

  mobile:

    type: rest

    uri: https://catalog.mobile.helium.io/api/catalog

    warehouse: mobile

    s3.endpoint: https://helium.io

    s3.region: us-east-1

    s3.path-style-access: false

    s3.access-key-id: anonymous

    s3.secret-access-key: anonymous

Or skip the file and pass the same properties inline:

from pyiceberg.catalog import load_catalog

catalog = load_catalog("mobile", **{

    "type": "rest",

    "uri": "https://catalog.mobile.helium.io/api/catalog",

    "warehouse": "mobile",

    "s3.endpoint": "https://helium.io",

    "s3.region": "us-east-1",

    "s3.path-style-access": "false",

    "s3.access-key-id": "anonymous",

    "s3.secret-access-key": "anonymous",

})

table = catalog.load_table("hotspots.heartbeats")

df = table.scan(limit=10).to_arrow()

Note the table naming: PyIceberg addresses hotspots.heartbeats within the loaded catalog, where Trino wants the catalog name in front — mobile.hotspots.heartbeats.

Spark

The same values as spark.sql.catalog.* properties, in spark-defaults.conf or as --conf flags:

spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

spark.sql.catalog.mobile=org.apache.iceberg.spark.SparkCatalog

spark.sql.catalog.mobile.type=rest

spark.sql.catalog.mobile.uri=https://catalog.mobile.helium.io/api/catalog

spark.sql.catalog.mobile.warehouse=mobile

spark.sql.catalog.mobile.io-impl=org.apache.iceberg.aws.s3.S3FileIO

spark.sql.catalog.mobile.s3.endpoint=https://helium.io

spark.sql.catalog.mobile.s3.region=us-east-1

spark.sql.catalog.mobile.s3.path-style-access=false

spark.sql.catalog.mobile.s3.access-key-id=anonymous

spark.sql.catalog.mobile.s3.secret-access-key=anonymous

Then spark.sql("SELECT * FROM mobile.hotspots.heartbeats LIMIT 10"). You need iceberg-spark-runtime and iceberg-aws-bundle on the classpath; without the AWS bundle the catalog resolves and every read fails at the storage layer.

A few things that will cost you an afternoon otherwise, whichever client you use:

• Nothing in there is a secret. iceberg.rest-catalog.security=NONE means the catalog is unauthenticated, and the storage credentials are the literal string anonymous. There is nothing to request, nothing to rotate, and nothing to keep out of version control.

• warehouse is not the bucket name. It names the warehouse as the REST catalog knows it. Getting this wrong produces a catalog that connects cleanly and lists nothing.

• Vended credentials are off. The catalog does not hand back per-table storage credentials, so Trino uses the s3.* settings above for every table it touches.

• Multiple catalogs, one engine. A second .properties file is a second catalog, and Trino will join across them — so you can put this one next to your own data and query both in a single statement.

• The catalog is rate-limited. Open is not the same as unmetered. Requests to the catalog are throttled, so an engine that hammers it will start getting turned away.

What this unlocks

The goal was never Iceberg. It was to make the network's data easy to query and cheap to query. Network data is queryable the ordinary way now — SQL over a JDBC connection, from whatever BI tool or notebook you already use. The only real cost of asking is running a Trino instance; the reads themselves are not billed. That is a low enough floor that building an app, a dashboard or a set of public stats on top of the network's data is a weekend rather than a project.

How much data each carrier moved last month, which hotspots heartbeat but pass no traffic, what a hotspot earned and why, which approved sites have never carried a byte — each of those is one query now, against tables anyone can point their own engine at.