What actually gets read: netCDF vs Zarr v3

1The dataset is an array — space × time

One Tier-2 product — say CAMS-GLOB-ANT black carbon, remapped onto your mesh for a year — is a single gridded array with a space axis (your mesh cells) and a time axis (the months). About 415 MB for that one species-year, the same figure the other docs use.

The schematic below stands in for that array: a flattened map (longitude across, latitude down) on one axis, the twelve months on the other. A chunk is a small rectangular block of it that is stored — and compressed, and fetched — as one unit. netCDF read whole is one big object; Zarr v3 is a grid of independently addressable chunks. Everything below follows from that one difference.

2Try it — pick a workload, see what's transferred

A real read is one choice on each axis: a spatial extent (how much of the map) and a time period (how much of the year). Pick one of each — or drag directly on the Your query map and timeline to make your own.

Reading it fairly. Zarr fetches whole chunks, so a query that straddles a chunk edge rounds up to every chunk it touches — drag a region across a boundary on the map and watch it pull the neighbouring chunk too. Chunk shape is a design choice you tune to how the data will be read (here: 3×3 cells spatially, one month per time chunk — which is exactly why sub-seasonal stays cheap). And flip the netCDF toggle to netCDF-4 hyperslab: a chunked netCDF-4/HDF5 file can read a sub-region too, and the byte counts converge — see the next section for what then stays different.

3The workloads: pick an extent, pick a period

These two axes aren't arbitrary — crossed, they are the access patterns this store has to serve well.

Spatial extent. Global is the whole mesh — and the UPTEMPO sample data behind these numbers is exactly that: CAMS-GLOB-ANT black carbon remapped onto the global MPAS x1.163842 mesh (≈415 MB on disk). Regional is a contiguous sub-domain — the kind of area you'd refine a variable-resolution MPAS mesh over, or overlay a regional inventory on (MIEM's own example overlays a US NEI inventory on CAMS). At this schematic resolution one chunk ≈ a continental domain.

Time period. These are distinct timescales, not three sizes of the same thing:

• Climatology. The full annual cycle — all twelve months, e.g. a multi-year-mean run. It touches every time chunk; the win has to come from the space axis.
• Seasonal. A single season, ~three months (here Jul–Sep — a fire / ozone season).
• Sub-seasonal. The S2S band: shorter than a season — roughly two to six weeks. At monthly storage that's a single month, and because this store chunks time by the month, you fetch just that month's chunks — the smallest footprint and the largest relative win of the three. (This rests on the chunk choice: had time been chunked in quarters, a one-month request would round up to the whole quarter and read exactly what a season does. Chunking time by the month is what keeps a sub-monthly read genuinely cheap — see §4.)

Crossing the two axes gives the six reads in the demo. Two are worth calling out. Global × climatology needs the whole array — the byte counts are basically equal across layouts, and Zarr's win is not fewer bytes but that each MPI rank reads its own cells as independent parallel ranged requests, streaming one time slice at a time instead of holding one monolithic file. Global × sub-seasonal is the instructive opposite: the thinnest slice in time, yet a global read still has to touch every spatial chunk — so the spatial extent, not the short time window, sets the floor on what moves. Sub-seasonal is the big win regionally; globally it is not.

4"What about different-resolution data?"

A real question from a colleague. It splits into two — and the store answers them in different places.

Inputs at different native resolutions. Inventories don't share a grid: CAMS-GLOB-ANT is 0.1°, fire and biogenic products are coarser, some arrive on entirely different projections. That's normal and fully handled — Tier 1 keeps each inventory at its own native resolution, with a chunk shape chosen to fit it (a fine 0.1° array uses spatial chunks that cover more degrees so each chunk stays a sensible size; a coarse array uses smaller ones). The conservative remap is computed per source grid → your mesh, and Tier 2 is where they all land on one common mesh. So "mixed resolutions in, one resolution out" is the normal path, not a special case.

Wanting the data at a different resolution than it's stored. This is the honest part. Zarr v3 has no automatic multi-resolution pyramids the way a Cloud-Optimized GeoTIFF does — asking for a coarse view of a fine store still reads the fine chunks and averages them; chunking gives you cheap subsetting, not free decimation. So the design's answer to "I need it at resolution X" is the same answer as everything else: conservatively remap to grid X, once, as another Tier-2 product. If a few target resolutions are used often, they're simply stored as separate Tier-2 stores (a multiscale group), each chunked and each carrying its own provenance — rather than resampled on the fly and risking a non-conservative result.

The same logic on the time axis. "Resolution" isn't only spatial, and neither is chunk shape. The time-chunk size sets how finely you can subset in time: this store chunks by the month, so a sub-seasonal (one-month) read fetches just that month — chunk time in quarters instead and the same request would round up to three. Going finer than monthly — a weekly cadence, or resolving the diurnal cycle — is itself a temporal-resolution choice and a separate Tier-2 product, not a free reslice of a monthly store.

5So when is plain netCDF still the right call?

Often. If the deliverable is a single file a colleague opens in ncview, or a slice you hand to a downstream tool, netCDF is exactly right — and the store exports to netCDF for precisely that. A chunked netCDF-4/HDF5 file on a local disk can also hyperslab a sub-region efficiently (flip the toggle and the byte counts match Zarr). What the chunk-addressed layout adds is specific to the build-once / serve-many job:

• Independent parallel writes. Many ranks write different chunks at once with no central file lock — so remapping a fine mesh, or many inventories, scales on HPC.
• Append without rewrite. A new year, sector, or inventory is new chunks beside the old ones — not a rewritten file.
• Object-store-native random access. Each chunk is its own key, fetched by an independent ranged GET; a plain netCDF file over the network is fetched as one object unless a server (THREDDS / OPeNDAP) or a kerchunk sidecar fronts it.
• Parallel reads at run time. Each rank pulls only its partition's chunks, concurrently — the count in the Zarr panel is the number of independent requests, not one serial file traversal.

In other words: with the hyperslab path the bytes can match, but the concurrency, append, and object-store behavior is what makes the chunked layout fit a store that's written by many ranks and read on every model step. That's the trade the demo is really illustrating.

Full design and trade-offs: the technical concept →  ·  the same story from the emissions side: for scientists →