Manifests and registries

Manifest schema

A manifest is a TOML file with one table per dataset. A table is recognized as a dataset by the presence of the subdir key; tables without subdir are grouping levels.

[interior.eos.wolf_bower_2018]
name = "Wolf & Bower (2018) MgSiO3 equation of state"    # optional display name
subdir = "interior/eos/wolf_bower_2018"                  # location below FWL_DATA
zenodo = "10.5281/zenodo.1234567"                        # required, version DOI
dataverse = "10.34894/ABCDEF"                            # optional mirror DOI
required_by = ["aragog", "zalmoxis", "spider"]           # models that need it

Validation at load time:

• subdir must be a relative path with no .. components and no backslashes; it can never escape the data root.
• zenodo is required and must have the form 10.5281/zenodo.<record-id>.
• dataverse, when present, must be a DOI.
• A dataset table must not contain sub-tables, and arrays of tables are rejected; ambiguous structures fail loudly instead of being silently dropped.

Registries

Each dataset has a registry file next to its manifest, named <dotted-key>.registry.txt, one <name> <hash> entry per line in the format pooch reads natively. Registries are generated by fwl-io sync and committed; they are never edited by hand. File names may contain forward slashes for files nested below the dataset directory, with the same escape rules as subdir.

Discovery

fwl-io aggregates manifests from every installed package that registers one under the fwl_io.manifests entry-point group. The entry point resolves to a zero-argument callable returning the manifest path. fwl-io list shows everything installed, flags datasets whose registry file is missing, and reports providers whose manifest failed to load; a broken provider is skipped, not fatal.

Fetch resolution order

For every requested file:

1. Present under the data root with a matching checksum: done.
2. Present in the read-only shared cache (FWL_DATA_CACHE) with a matching checksum: copied in atomically.
3. Offline mode active: error naming the file and resolved root.
4. Otherwise: downloaded via Zenodo, then the Dataverse mirror, verified, placed atomically.

Provenance

Fetcher.provenance() returns one record per registry entry with the file path, checksum, and the actual origin of files resolved in the current session (local, cache:<path>, or the mirror that served the download). Files not yet fetched are marked declared: since their true origin is unknown. These records are designed to feed per-run provenance manifests in PROTEUS.

The FWL_DATA layout

This section is the target layout specification: new datasets and migrating models use it; existing trees keep their legacy directory names until their consumers migrate, so both forms coexist during the transition. A flat copy left by a pre-versioning fetch is re-fetched rather than adopted; a command that relocates such trees in place is tracked in #13.

The target tree is organized by physical domain, mirroring the package structure of the PROTEUS source tree (src/proteus/), with one deliberate exception: the two interior packages (interior_struct, interior_energetics) share a single interior/ data domain, because the equation-of-state tables serve both.

Naming rules for dataset directories: all lowercase snake_case; for datasets identified by a publication, author tag first and year second, then any descriptor (baraffe_2015, zeng_2019, dk09_1tpa_elec_free); datasets without a citation use their plain source or product name (solar, phoenix, muscles).

Below its dataset directory, every dataset resolves into a version directory r<zenodo-record-id> derived from its manifest pin, so updated deposits land beside superseded ones instead of overwriting them. Fetching a dataset as a whole (the model-facing fetch_for) writes a .fwl-io.json stamp into its version directory, recording the schema version, the pinned DOI, the file checksums, and the fetch date, so a completed dataset directory or a shared read-only cache is self-describing.

FWL_DATA/
  atmos_clim/
    spectral_files/<set>/<bands>/r<recid>/
    surface_albedos/hammond_2024/r<recid>/
  atmos_chem/                        # chemistry networks and cross-sections
  interior/
    eos/<dataset>/r<recid>/
    melting_curves/<dataset>/r<recid>/
  star/
    tracks/<dataset>/r<recid>/
    spectra/<dataset>/r<recid>/
  observe/
    exoplanet_reference/r<recid>/
    mass_radius/zeng_2019/r<recid>/
  outgas/  escape/  orbit/           # created when their first dataset lands

The tree holds immutable fetched reference data only: anything generated at runtime (derived tables, interpolation caches, solver caches) belongs in run output or cache directories, never below FWL_DATA. This keeps a shared read-only cache trustworthy as a whole.

Models adopt this layout when they migrate to fwl-io; legacy directories from the previous layout remain readable by unmigrated code and age out when their last consumer migrates (a relocate command for cleaning local trees immediately is tracked in #13). The mapping from the legacy locations: