Skip to content

Design decisions

This page states the decisions behind fwl-io and the reasoning for each, so feedback can target specific choices. None of these are set in stone at version 0; if you disagree with one, open an issue and say which decision and why.

One shared package, per-model data ownership

fwl-io provides the machinery (fetching, verification, mirrors, offline mode, the sync tool) and the manifest for datasets shared by several models. Each model ships its own manifest for model-specific data and exposes it through the fwl_io.manifests entry-point group.

Why: the ecosystem previously carried around six independent download implementations. Centralizing the machinery removes that duplication and makes it testable in one place, while per-model manifests keep data ownership where it belongs: adding data to a model is a PR on that model, never an fwl-io release. A broken manifest in one package is skipped with a warning, so it cannot break data access for the other models.

Committed registries instead of live API lookups

The file list and checksums of every dataset live in registry files committed to the owning repository, regenerated by fwl-io sync from the Zenodo API.

Why: runtime API lookups need network exactly where it does not exist (compute nodes) and trust whatever the live record states at that moment. Committed registries make every fetch reproducible and offline-capable, put checksum changes through code review, and still involve no hand-maintained hashes.

Version DOIs only

Manifests must pin Zenodo version DOIs. fwl-io sync rejects concept DOIs, which it detects because the API resolves a concept id to the newest deposit.

Why: a concept DOI lets a dataset change underneath pinned code, producing silently different results between machines that fetched at different times. This failure mode has cost real debugging effort in the ecosystem. Data updates should be deliberate manifest changes, visible in review.

Zenodo required, Dataverse as mirror

Every dataset requires a Zenodo version DOI; Dataverse is an optional second download source tried when Zenodo fails.

Why: registries are generated from the Zenodo record, so Zenodo is the source of truth for names and checksums. The mirror must host byte-identical files (tabular ingest disabled). Checksums are whatever Zenodo publishes for the record, md5 today; a mirror is therefore protection against outages, not a second authority.

FWL_DATA is required

The data root comes from an explicit path or the FWL_DATA environment variable. There is no fallback default.

Why: the rest of the ecosystem already treats FWL_DATA as required. A silent per-user default would hide misconfiguration, worst on clusters, where an unset variable would quietly download gigabytes into a small home directory instead of the shared tree.

Offline-first

FWL_IO_OFFLINE=1 forbids all network access; FWL_DATA_CACHE provides a read-only, pre-populated group cache consulted before any download; every failure names the file and the resolved root.

Why: production runs happen on compute nodes without internet. Anything that only works online is not usable for the campaigns this ecosystem runs.

Atomic placement, distinct failure classes

Every write is staged on the destination filesystem and moved into place with an atomic rename. Network or checksum failures raise a download error listing every mirror attempt; local problems (read-only tree, full disk) raise their own OSError.

Why: interrupted jobs must never leave half-written files that later runs trust, and a permissions problem on the data root must not masquerade as a network problem, which sends the user debugging the wrong thing.

Known limitations at version 0

These are tracked as issues and open for input:

  • Archive extraction: archives (tar, zip) are downloaded and verified verbatim but not unpacked.
  • Dataverse mirror script: mirroring a Zenodo deposit to Dataverse is a manual step; a sync script is planned.
  • Checksum strength: registries carry the checksums Zenodo publishes (md5); recording sha256 at sync time is under consideration.
  • Progress reporting: downloads are silent by default; a progress option exists per fetcher but is not yet wired through the CLI.