CLI reference
The fwl-io command has fifteen subcommands. Failures are reported as concise messages on stderr (never a traceback) and exit with status 1; success exits 0. check-mirrors, latest, mirror-status and mirror-sync have further exit codes, which their sections give. With FWL_IO_OFFLINE set, every command that needs the network (sync, mirror, mirror-publish, mirror-sync, mirror-pin, check-mirrors, mirror-status, latest) exits 1 with the offline message before it connects. sync and fetch <model> aggregate per-dataset failures into a multi-line report, and a download failure lists every mirror attempt.
fwl-io sync
fwl-io sync path/to/manifest.toml
Queries the Zenodo record of every dataset in the manifest and rewrites the registry file next to the manifest. Rejects concept DOIs. All datasets are attempted; per-dataset failures are collected and reported together at the end, and successfully synced registries are still written.
fwl-io list
fwl-io list
Lists every dataset from all installed manifests with its key and consumers. When a manifest gives a dataset a human-readable name that differs from its key, that label is printed on the next line below the key. Datasets without a committed registry are flagged [NO REGISTRY]. Providers whose manifest fails to load are reported on stderr as FAILED TO LOAD, and providers left out because they conflict with another installed manifest as NOT USED; either sets the exit status to 1.
fwl-io fetch
fwl-io fetch <model> [--data-root PATH] [--progress | --no-progress]
fwl-io fetch --key <dotted.key> [--data-root PATH] [--progress | --no-progress]
Fetches every dataset that lists <model> in its required_by, or with --key the one dataset of that key, through the same mirrors and checksum checks. With a model, all datasets are attempted and failures are aggregated into one report. With --key, the first file that cannot be fetched stops the command with exit 1, as does a stamp that cannot be written after the fetch (then fwl-io path could not confirm the dataset); on success it prints the number of files. --data-root overrides the FWL_DATA tree. fwl-io fetch zalmoxis downloads the ten shared interior tables and interior.radial_profiles, about 2.2 GB (measured on 2026-10-07).
A download progress bar shows by default when stderr is a terminal; --progress forces it on and --no-progress off. The bar needs the optional tqdm dependency (pip install fwl-io[progress]); without it, or when there is no stderr to draw on, the fetch runs with no bar.
fwl-io path
fwl-io path <dotted.key> [--data-root PATH]
Prints the version directory of a dataset, for example $FWL_DATA/atmos_clim/spectral_files/dayspring/48/r15721749, when a completed fetch left it in place: a current stamp and every registry file present (every recorded member for an archive dataset). Contents are not hashed again; fwl-io check does that. Otherwise it names the directory and the fetch command on stderr and exits 1. A key that no installed manifest declares, or a data root that does not exist, exits 1 too. Nothing is downloaded or written.
fwl-io check
fwl-io check <model> [--data-root PATH]
Reports whether every dataset that lists <model> in its required_by is present and matches its registry, without downloading anything. Each file is reported in one of five states: ok (present, checksum matches), missing, mismatch (present, contents differ), unreadable (present, could not be read to be checked), or present. The last means the file is there and nothing was available to verify it against, which is the case for the members of an archive dataset: the registry pins the checksum of the archive, not of the files extracted from it, so such a dataset is reported presence only. That is not counted as a fault, since presence is all that is checkable there, but it is never reported as verification.
Two kinds of failure are reported apart from the datasets, because they call for different repairs. A manifest line means an installed package's manifest was left out, so nothing it declares was inspected: MANIFEST FAILED TO LOAD for a manifest that could not be read, which is always reported, since its datasets are unknown, and MANIFEST NOT USED for a manifest that conflicts with another installed manifest, which is reported only when the conflict drops a dataset <model> reads, the same condition under which fwl-io fetch <model> fails; fwl-io list reports every conflict. NOT CHECKED means the manifest was fine but one dataset could not be resolved, most often because its registry has not been generated yet; run fwl-io sync for it. Either is on its own enough to fail the check.
The report goes to stdout whatever the verdict, so a caller running this to find out what is wrong gets the detail and not only the exit status. Exit is 1 on any missing, corrupt or unreadable file, any manifest reported as above, any unresolvable dataset, or a model no manifest declares. The closing line says all data present and verified only when every file was compared against a digest; a sound tree holding a presence-only dataset closes with all data present, N dataset(s) by presence only instead, and still exits 0.
Checking reads and hashes every file a plain dataset declares, so for those the cost is one full pass over the data: on a multi-gigabyte tree, or a shared cluster filesystem, expect it to take as long as reading that data once. An archive dataset costs far less, since its members have no digests to check and are only tested for presence.
Nothing is downloaded and no dataset directory or file is written, which makes this safe to run against a tree another process is reading. Resolving the data root creates that root if it does not exist, as it does for every other subcommand. The equivalent Python entry point is fwl_io.check_for, whose CheckReport.ok is false when nothing was checked, so a model that matches no dataset can never read as a clean tree. CheckReport.verified is the stricter question, false whenever any part of the tree was checked by presence alone.
fwl-io cache
fwl-io cache key --prefix PREFIX [--provider NAME ...] [--key DOTTED.KEY ...]
fwl-io cache fetch [--provider NAME ...] [--key DOTTED.KEY ...] [--data-root PATH]
fwl-io cache check [--provider NAME ...] [--key DOTTED.KEY ...] [--data-root PATH]
Keys, fills and checks the data tree an actions/cache entry holds in a CI workflow. All three take the same selection: --provider adds every dataset of an installed manifest provider (the entry-point name fwl-io list prints in brackets, for example mors), sorted by key, and --key adds one dataset; both repeat, and their order does not change the key. A selection that is empty, names a dataset twice, or names a provider or key no usable manifest declares exits 1.
key writes key=<prefix><sha256> and restore-keys=<value> to the file GITHUB_OUTPUT names, or to stdout when it is unset, and prints the key on stderr. The digest covers, per dataset in folder order, its folder below the data root, its extract kind and the checksums its registry pins, plus the fwl-io release that lays out the tree (the local version part after + left out), so the key moves when the data or its layout may have moved and stays put otherwise. The prefix holds 1 to 100 letters, digits, ., _ and -, which keeps the key well below the 512 characters actions/cache takes; two selections cached in one repository need prefixes of which neither starts the other, or a restore can pick up the other selection's tree. restore-keys is the prefix: every manifest dataset has a Zenodo pin and so a version folder r<record-id>, a re-pinned dataset lands in a folder an older tree does not hold, and a fetch keeps a restored file only when its checksum matches, or an extracted archive tree only when its stamp records the same registry. Through the Python API (fwl_io.cache_key) a Dataset built without a Zenodo pin can be passed; its files would sit at the same path whatever the pin, so such a selection gets an empty restore-keys and a line on stderr naming the dataset. A workflow passes both outputs on:
- id: cachekey
run: fwl-io cache key --prefix mors-fwl-data- --provider mors
- uses: actions/cache@v4
with:
path: ${{ github.workspace }}/fwl_data
key: ${{ steps.cachekey.outputs.key }}
restore-keys: ${{ steps.cachekey.outputs.restore-keys }}
- run: fwl-io cache fetch --provider mors --data-root ${{ github.workspace }}/fwl_data
- run: fwl-io cache check --provider mors --data-root ${{ github.workspace }}/fwl_data
actions/cache saves an entry only on an exact-key miss and never saves over one it hit, so fetch runs every time: on a complete tree it downloads nothing, and after a restore from an older entry it completes the tree before the save. A restored tree keeps the version folders of older pins beside the new ones until fwl-io prune removes them. check then reports each dataset as <folder>: <found>/<expected> files ok (present for an archive dataset, whose members are checked for presence only, and faulty when a file is missing or differs) and exits 1 on any fault; a data root that does not exist exits 1 and is not created.
fwl-io relocate
fwl-io relocate [--data-root PATH] [--dry-run]
Moves data left by the previous layout into the place it belongs now, for a tree fetched before the current layout existed. Unmigrated code still reads the old directories, so they are otherwise left alone and age out as their consumers migrate; this is for cleaning a tree up straight away instead.
Every present file in the old location that matches its recorded digest moves; an absent file is reported for a later fetch to fill in at the new location. A present file whose contents differ, a legacy tree holding none of the registry's files, or a dataset whose registry has not been generated is reported and left exactly where it is. Verifying first is the point, since moving a stale copy would put it where the fetcher then trusts it. A file already at the new location is not overwritten: one that matches the registry is skipped and its old copy stays where it is, and one that does not blocks the dataset as a mismatch. A corrupt old copy of a file the new location already holds intact still blocks the dataset, and the report says the bad file is only a copy. On a second run after a partial move, a dataset whose old directory holds nothing more to move, and whose new location already holds verified files, is reported as absent with the count already there. It is not a fault, even when the new location also holds a file that differs from the registry; that file is named in the report with the repair fwl-io check <model>, then fwl-io fetch <model>. An old copy of a file that is already at the new location stays where it is. A file at the new location counts as verified only when it is a regular file with the right digest and every directory between the data root and it is a plain directory, so a symlink at the file or at any directory on the way is never treated as one. A file behind such a symlink is named in the report as not verified. An old copy is called redundant only when it is also reached through plain directories; one that is a symlink, or has a symlinked directory on the way, may lead back to the new location, so the dataset is reported and both are left alone. With no old directory nothing is moved and nothing is called redundant, so the new location counts as current wherever its directories point. Once a dataset's files have moved, the emptied directories are removed, and the walk upward stops at the data root.
Some datasets cannot be verified or moved and are refused rather than touched, each named with its reason. An archive dataset's registry pins the packed archive, while an old tree holds the files extracted from it, so there is nothing to hash the tree against; move such a tree by hand, or delete it and let the fetcher rebuild it at the current location. A dataset whose registry is empty offers no files to compare, so every check over it would pass for want of anything to fail; run fwl-io sync for it. Both are reported only when an old directory is actually there, so a machine that never had the previous layout is unaffected.
The move opens every directory without following a symlink, so a dataset is also refused when the old directory is a symlink, when a symlink lies anywhere between the data root and the old directory or the part of the new location that exists, when a registry file or a directory inside a registry name is a symlink, when a path resolves outside the data root, or when a directory on the way cannot be read. A platform that lacks O_DIRECTORY, O_NOFOLLOW, or a directory-relative open, stat, mkdir or rename, or lacks a stat that leaves a final symlink unfollowed, such as Windows, is refused the same way; unlike deletion, a move needs neither flock nor a symlink-safe rmtree. --dry-run reports every one of these refusals. It does not check write permission or free space, so a directory this user cannot write still fails the real run, which then puts back what it moved.
A run assumes that no other process renames or replaces directories under the data root while it moves files. Unlike a deletion, a move does not check the fetch lock, so a fetch running at the same time is not detected: do not relocate while a fetch could be running. A file created at a destination between the check that the name is free and the move is overwritten. A move that fails part way puts back the files it already moved, and the datasets after it are still attempted. A dataset whose files cannot be put back is reported as split between the two locations, which needs a person to look at both directories.
A dataset already at its current location is not a fault, and a copy still sitting at the old location beside it is named rather than deleted. Nothing here removes data: the only directories it removes are ones it has just emptied itself.
A manifest that could not be included is reported apart from the datasets, as either MANIFEST FAILED TO LOAD or MANIFEST NOT USED, the same split fwl-io check and fwl-io list report. The first means the manifest could not be read at all; the second means it read fine but was dropped for conflicting with another installed manifest. Either way, any dataset that manifest would have declared is left out of the plan and cannot be relocated.
Exit is 1 when a legacy tree was found and could not be moved, when an installed manifest failed to load or was dropped as a conflict, since that manifest may be the one declaring the dataset a tree still holds, or when the shipped table of old locations could not be read, since without it no dataset has an old location to look at and a run that reported nothing would read like a tidy tree. A tree that was already tidy exits 0. --dry-run reports the same plan without moving anything. The equivalent Python entry points are fwl_io.relocate_all and fwl_io.plan_relocations; Relocation.files names the files a move would move: in a plan it is empty unless the state is ready, and the results of relocate_all keep it on moved, failed and split entries.
fwl-io prune
fwl-io prune [--data-root PATH] [--delete] [--include-orphans] [--yes] \
[--allow-empty-reference-set]
Removes versioned dataset directories no installed manifest references. A dataset pinned to a Zenodo version lives at <data-root>/<subdir>/r<record-id>; when a pin advances, the fetcher writes the new version beside the old one and the old one stays on disk. This finds those left-behind version directories and, only when asked, deletes them.
Each version directory is classified against the manifests installed in this environment. One a current pin uses is referenced and always kept. One under a subdirectory a current dataset uses, but which no current pin names, is superseded: an older version of a known dataset, the default delete target. One under a subdirectory no installed manifest knows is orphaned; on a shared data tree it may be the pinned version for a manifest installed in another environment, which this process cannot see, so it is removed only under an explicit opt-in. One that matches the version-directory name shape but carries no stamp naming its own record id and the subdir it sits under is unrecognised: the name alone is not proof of what the directory holds, and a fetched version copied elsewhere keeps a stamp that names its original subdir, so it is reported but never a delete target. A directory that contains a further version directory, a stamp or a subdirectory on another filesystem, or that cannot be fully read, is unrecognised too.
Nothing is deleted unless the reference set is complete. If any installed manifest fails to load or is dropped for a conflict with another, or any dataset's version directory cannot be computed, the set of referenced directories is a subset of the truth, so no directory can be proven unreferenced: the run reports the failure and deletes nothing. The same holds when part of the data root, or of a referenced version, cannot be read; a referenced version is only read through when deletion starts, so a dry run can print a clean plan that --delete then refuses. Deletion also refuses outright while a fetch lock is held anywhere on the data root. Each lock file is probed read-only with a shared, non-blocking flock, so the check writes nothing. A lock entry that is not a regular file (a symlink, a directory, a FIFO, a socket, a device), a lock directory that is a symlink or not a directory, and a lock file that cannot be probed each block deletion under their own reason, since prune does not probe them and cannot tell whether a fetch holds them. A lock file that this user cannot write, a lock directory this user cannot write or search (which matters for a lock file that does not exist yet), or, with no lock directory yet, a data root this user cannot write does not block, so a tree shared with other users stays usable, but the plan and the delete run print a warning: a fetch by this user through it runs without a lock, which prune cannot see, so do not fetch while prune runs. On a filesystem without flock, a fetch can leave a lock file that no probe can test; remove the files in .fwl-io-locks by hand when no fetch is running. Locks are only seen from the host prune runs on: on a mount whose locks are local to each host (NFS with nolock or local_lock=flock, Lustre with localflock), run prune only when no host is fetching. Do not run prune in a process that is also fetching, since on Linux NFS its probe can release that process's own lock. A fetch holds a lock only while it downloads one file (an archive dataset holds it for the whole download and extraction), a file already on disk is served without one, and a fetch that waits too long for a lock, or runs on a filesystem that cannot lock, proceeds without one, so this sees a fetch in the middle of a download, not one between two files: run prune when no fetch is running.
The reference set is computed once when deletion starts. Just before each directory is removed, it is checked again: it must not be a symlink, a mount point, or on another filesystem, it must resolve inside the data root, it must not be or contain a referenced version or be passed through by a symlink inside one (directly or along a chain of links), it must still carry its matching stamp and hold no nested version or subdirectory on another filesystem (a bind mount of the same filesystem is not detected), and no fetch lock may be held on the data root; every lock file is probed again right before each removal, so the cost of a run grows with the number of removals times the number of lock files, and there is one lock file for each file ever fetched into the tree. A fetch that takes its lock after that probe is not seen. Every environment that fetches into a shared tree must run filelock 3.29.5 or newer: older releases either remove a lock file when they release it or do not notice that the file they locked has been removed, and a mix of the two can hold a lock on a file that no longer has a name, which no probe can see. A directory that passes is first moved into the .fwl-io-staging directory under the data root and then deleted there, so it never appears half deleted at its own path; right before the move, the directory and its parent are checked once more to be the entries the checks read, by device and inode, and the entry moved is compared again after the move and put back if it differs. A directory that changed in between is reported as changed during prune and is not deleted. If the deletion fails part way, the run reports the failure and the staging path of what is left, and every later plan lists what is still there; a later download also clears staging entries last modified more than a day ago.
Prune assumes that no other process renames or replaces directories under the data root while it runs. An fwl-io fetch is kept out only while it holds the fetch lock, within the limits described above, so run prune when no fetch is running. The checks guard against a wrong classification (a missing stamp, a nested version, a reference, a symlink, a mount), and the re-checks around the move narrow, but cannot close, the window in which another program that moves directories around the tree could make prune act on a directory other than the one it checked. Do not run prune while another tool reorganises the data tree.
The default is a dry run: the plan is printed, with reclaimable bytes per category, and nothing is touched, not even a probe file; when no directory name in the data root has cased letters to show whether the filesystem ignores case, a dry run matches subdirectories case-sensitively, which only keeps more, so on such a root its counts of superseded directories can be lower than what the delete run finds. --delete removes the superseded directories; --include-orphans adds the orphaned ones. Deletion is confirmed interactively, after a warning that a superseded version may still be referenced by another environment on a shared tree, unless --yes skips the prompt for non-interactive use.
An orphaned directory is refused by default whenever no installed manifest declares any dataset at all, since an empty reference set is as likely to mean "nothing to keep" as "the manifests failed to install". --allow-empty-reference-set overrides this and lets --include-orphans proceed anyway; it has no effect without --include-orphans and no effect when at least one manifest declares a dataset.
The plan and the delete happen in two separate calls, so the tree can change between them: a fetch can start, a pin can advance, or a candidate's classification can move between superseded and orphaned. The delete call removes a directory only when the plan listed it as a target and a fresh classification still does, so nothing is deleted that the confirmation did not list. A listed directory that is no longer on disk is reported as gone; a listed target that the fresh classification keeps (now referenced, orphaned or unrecognised) is reported as not removed, and the run exits 1. It refuses the whole run when the fresh reference set no longer permits deletion, or when the plan lists a directory outside this data root.
The data root must already exist; fwl-io prune never creates it, and a root that does not exist is an error.
Deletion needs directory handles opened without following symlinks, directory-relative renames and flock. A platform without them, such as Windows, refuses --delete before anything is touched; the dry run still prints the plan there, and exits 1 when a lock directory exists, since its locks cannot be checked.
Exit is 1 when the reference set is incomplete, when the data root cannot be fully read, when a fetch lock is held or cannot be checked, when an empty reference set blocks orphan deletion without the override, when deletion is refused at apply time because the tree changed since the plan or the platform cannot delete safely, when a confirmed target is kept or a removal fails, or when a confirmation is declined; a clean plan or a completed deletion exits 0. The command runs fwl_io.plan_prune and, after the confirmation, fwl_io.apply_prune on that plan; fwl_io.prune_versions plans and deletes in one call.
fwl-io mirror
DATAVERSE_TOKEN=... fwl-io mirror <zenodo-doi> --collection <alias> \
[--dataverse-url URL] [--contact-email EMAIL] [--subject SUBJECT] \
[--file NAME ...] [--licence NAME] [--into PERSISTENT_ID] [--no-publish] [--dry-run]
Mirrors a pinned Zenodo deposit to a Dataverse collection. The deposit must be an accepted record of a Zenodo community of the framework (the rule): any other record is refused before a Dataverse write, also with --dry-run and --into. For an accepted record it downloads and checksum-verifies the deposit's files, creates a matching Dataverse dataset with citation metadata taken from the Zenodo record, uploads the files byte-identically (tabular ingest disabled), and by default publishes the dataset, then prints the Dataverse DOI to add to the consuming manifest. The API token is read from the DATAVERSE_TOKEN environment variable, never a command-line argument. A contact email (--contact-email) is required to create a dataset; only --dry-run, which makes no Dataverse writes, is exempt. --subject is validated by the server when the dataset is created, so a value outside the target installation's citation vocabulary is rejected then. --dry-run performs the download and metadata mapping only, making no Dataverse changes; --no-publish leaves the created dataset as a private draft. --file NAME restricts the mirror to that file of the deposit; repeat it for several files. Without --file, every file of the deposit is mirrored, and a name the deposit does not hold is an error. Files go one at a time (download, upload, check, delete the local copy), 60 s apart. A failure once a file has reached the draft keeps it and names the missing files; --into PERSISTENT_ID finishes such a draft (a never-published draft of the same record): it keeps a file of the Zenodo name and size, and of the registry checksum where the draft lists one of that type, without a download (where the draft lists another checksum type, as DataverseNL does, only the size is compared, so check the contents before a publish), sends missing and differing files, removes files the Zenodo record does not hold, needs no contact email, never publishes (use mirror-publish), refuses --dry-run, and never deletes the draft. --licence NAME gives a new draft the Dataverse license of that name in place of the one matched from the Zenodo record, for a record whose author licenses it so; the draft's description then names it and the Zenodo licence field. It needs --no-publish, so the draft is reviewed first, needs a record with exactly one Zenodo license entry, and is refused when that license matches one or more licenses the server lists. Without it the license always comes from the Zenodo record. See Mirror a deposit to Dataverse.
fwl-io mirror-publish
DATAVERSE_TOKEN=... fwl-io mirror-publish <persistent-id> \
[--dataverse-url URL] [--version-type VERSION_TYPE]
Publishes an existing Dataverse draft by its persistent id. A dataset that is published already exits 1 with "already published". For a draft, the command first checks the Zenodo record that the source note of the description names, and exits 1 without a publish when that check fails (The check before a publish). Otherwise it publishes: it never creates a dataset, so it is the second step of a create-draft-then-publish workflow, run once a draft created by fwl-io mirror --no-publish has been reviewed. <persistent-id> must be of the form doi:<prefix>/<suffix>, for example doi:10.34894/EXAMPLE. The API token is read from the DATAVERSE_TOKEN environment variable, never a command-line argument. --version-type is major by default and accepts only major or minor. Fails clearly if the dataset is already published or the persistent id does not resolve to a draft. See Mirror a deposit to Dataverse.
fwl-io mirror-status
fwl-io mirror-status
Lists the datasets of the installed manifests whose Dataverse mirror needs work: one without a dataverse pin (UNPINNED, with its Zenodo DOI), and one whose Zenodo record has a newer version than the one pinned (STALE: the manifest needs a new pin, and that version a mirror). A dataset can be both. A dataset without a pin whose record is not an accepted record of a Zenodo community of the framework (the rule) is also listed as OUTSIDE: no mirror is created for it. The command reads only Zenodo: <record>/versions/latest once per record, and the record itself for a dataset without a pin; a record that cannot be read is listed as UNREADABLE. A manifest left out of discovery is a FAIL line, reported as MANIFEST FAILED TO LOAD or MANIFEST NOT USED as fwl-io check-mirrors reports it. Exit is 1 when a manifest is left out or offline mode is active, 5 when a dataset needs work, 3 when the only problem is a record that could not be read, and 0 otherwise.
fwl-io latest
fwl-io latest [--prefix PREFIX]
Checks that the Zenodo record each dataset pins is the newest version of that record, for every dataset of the installed manifests whose key starts with PREFIX (all datasets without it). It reads <record>/versions/latest once per pinned record (Zenodo answers it with a redirect to the newest version), with no retry, and nothing else. A dataset with a newer version is a STALE line (the manifest needs a new pin). A record that Zenodo does not serve (HTTP 4xx other than 401, 403, 408 and 429, or a reply without a record id) is a GONE line; any other read failure (a connection fault, a timeout, a cut body, a reply that is not JSON, HTTP 401, 403, 408, 429 or 5xx) is an UNREADABLE line. A key that more than one dataset declares is named <key> (Zenodo <record>) on each line, once per declaration. A last line counts the datasets whose pin is the newest version, those with a newer version and those not checked, gone ones included. A prefix that no dataset key starts with exits 1, so a mistyped prefix never passes. Exit is 1 when a manifest is left out of discovery or offline mode is active, 5 when a newer version exists, 4 when a record is gone or no record could be read, 3 when the only problem is a record that could not be read for now, and 0 otherwise. The workflow latest-paleos.yaml runs fwl-io latest --prefix interior.eos.paleos each day and fails on every exit status other than 0 and 3; on exit 3 it leaves a warning, and the unread datasets were not checked. A Zenodo outage that lasts a day and covers every PALEOS record therefore gives one failed run that day.
fwl-io mirror-sync
DATAVERSE_TOKEN=... fwl-io mirror-sync --collection <alias> \
[--dataverse-url URL] [--contact-name NAME] [--contact-email EMAIL] [--dry-run]
Creates a draft mirror for one Zenodo record that has none. When a manifest is left out of discovery it prints only the FAIL line of each one, as fwl-io mirror-status prints them, and stops. Otherwise it prints the fwl-io mirror-status report, lists the datasets of the collection (with the token, so that drafts are seen) and sorts each dataset without a pin: SKIPPED when its Zenodo record has a newer version, which needs a new pin in the manifest first, is in no community of the framework, or could not be read; PIN MISSING when the record already has a published mirror, either one another dataset pins or one in the collection, with the fwl-io mirror-pin command to run; WAITING when the collection holds a dataset of the record that is not published, listed with its state (a draft to review and publish, or one to look at by hand). A dataset of the collection whose description names no Zenodo record in a source note, or several, is filed under no record and gets a NOT FILED line. The first record left gets a draft, created as fwl-io mirror --no-publish creates it and holding the files every dataset of that record asks for (the whole record when one of them names no files). One draft is created per run, and such a run needs --contact-email. The draft is then checked: it must be a draft with a license and a source note that names this record and no other (the rule of the check before a publish), hold exactly those files, and each file, downloaded from Dataverse, must have the Zenodo checksum and the checksum Dataverse lists. --dry-run stops before the creation and says which record a run would take. The command never publishes and never edits a manifest; the only dataset it deletes is the draft of its own run when the creation fails before a file has reached it. A listing or a creation that fails is a FAIL line after the lines already gathered. A record that the mirror refuses before a draft exists (a license that the server does not list, for example) gets a FAIL draft line that ends with no draft was created, and the next record gets its turn; the exit is then 1, also when a later draft is verified. A creation that fails once a file has reached the draft keeps the draft, and the line names it; a draft without files that could not be deleted is named in the FAIL draft line when the error behind it is a ValueError ("was created and could not be deleted: delete it by hand") and in the log only for another error, so look in the collection after a FAIL draft line that does not end with no draft was created. A check that cannot run (a failed download, for example) counts as not verified, and the line still names the draft. The token is sent to the Dataverse server only: a file download that the server redirects to a storage server is read there without it, and no other request that carries the token follows a redirect. Exit is 0 when there was nothing to create, or the draft is verified; 1 when a manifest was left out, offline mode is active, the listing or the creation failed, or the draft is not verified; 3 when nothing failed but a Zenodo record could not be read. The listing comes from the Dataverse search index, which can lag a dataset created seconds ago; a listing whose size differs from the count the server gives is an error.
fwl-io mirror-pin
fwl-io mirror-pin <persistent-id> [--manifest FILE]
Writes the pin of a published mirror into the shared manifest. The dataset must be published and name one Zenodo record in the source note; a description that names none, or several, is refused. Every installed dataset of that record is checked against the mirror as fwl-io check-mirrors checks a pin; one that the mirror serves gets dataverse = "<doi>" written after the zenodo line of its table (an edit that would change anything but that pin is refused) when the manifest file declares it (PINNED, with the pin it replaces when the dataset had another one), and otherwise a line with the pin to add in the package that declares it. A dataset the mirror does not serve is a FAIL line and exit 1; so is a dataset that could not be checked because a server did not answer, and an installed manifest left out of discovery, since its datasets were not seen. Pins are written only when the exit is 0: after a FAIL, or when one pin cannot be written, the file stays as it was and each pin that waits is a NOT PINNED line. When a pin was written, a later one failed and the file could not be put back, a FAIL line says so and each pin is a CHECK line, since it can be in the file. It needs no token, makes no Dataverse changes and does not commit: the edit is yours to commit. --manifest FILE edits another manifest file in place of the shared one.
fwl-io check-mirrors
fwl-io check-mirrors [--dataverse-url URL]
Checks every dataset with a dataverse pin in the installed manifests. The pinned DOI must name a released Dataverse dataset whose description carries the note every mirror has, "Mirror of Zenodo deposit dataFile.embargo.dateAvailable; DataverseNL refuses such a download with HTTP 403; an embargo date that cannot be read counts as one that blocks): with the registry checksum where the server uses the same algorithm, otherwise with the file size of the Zenodo record (DataverseNL stores SHA-1, the registries MD5). A same-size change of content passes this check; the fetch still verifies each downloaded file against the registry. The command reads only published data and needs no token, and it reads each Zenodo record once, also when that read fails. Each pin is read from the Dataverse server of its DOI: the prefix 10.34894 is DataverseNL, and any other DOI is looked up through the doi.org handle API, whose landing page names the server (scheme, host and port); one client serves each server. A DOI that doi.org does not know, or an answer with no landing page, is a FAIL; a doi.org outage leaves the pin UNREACHABLE, and a certificate failure there is retried like the other reads. --dataverse-url reads every pin from the one server it names.
A pin is served when all of these hold: no FAIL and no UNREACHABLE line names its dataset key, no FAIL line names the entry point of its manifest (FAIL <name>: MANIFEST ...), and no line unpinned <dataset key> stands below the counts.
It prints one line per wrong pin (FAIL), one per pin that could not be read for a transient reason (UNREACHABLE), the counts, and the datasets without a pin, which a fallback never reaches; a manifest left out is a FAIL, reported as MANIFEST FAILED TO LOAD or MANIFEST NOT USED, the same split fwl-io check reports. A transient reason is, from either server, an HTTP 408, 429 or 5xx other than 501 and 505, a lost connection, a timeout, or a cut body; from DataverseNL also the bot-check page or another HTML page in place of the API answer, and from Zenodo a non-JSON body. A certificate failure on either read is tried again up to twice, 30 s apart: a pin served after such a retry is listed with a WARNING line, and one that fails on every attempt is a FAIL. Once a host has failed every attempt, later reads of it in the same run are tried once, without the waits, and their WARNING line says so. A pin may be written with or without a doi: prefix. Any other failure makes the pin a FAIL. Exit is 1 when a pin is wrong, a manifest is left out, no pin was checked or offline mode is active, 4 when no pin is wrong and none could be read, 3 when no pin is wrong, some could not be read and the rest are served, and 0 when every pin is served. The weekly nightly runs fwl-io check-mirrors as its own step: a wrong pin (exit 1) and a run in which no pin could be read (exit 4) fail the job, and pins that could not be read while the rest are served (exit 3) leave a warning on the run and in the step summary while the job passes.
fwl-io --version
Prints the installed version.