Update a dataset or its mirror

This page is for the person who owns a dataset: you uploaded the files to Zenodo, and a model reads them through fwl-io.

A dataset is pinned to one version of a Zenodo record and to a committed list of its files and checksums. Nothing follows Zenodo by itself: a new version on Zenodo changes neither the pin nor the DataverseNL mirror, and a fetch keeps downloading the pinned version. A new version reaches users only through a pull request that changes the pin and the registry. Data governance gives the reason for this rule.

Terms

Who does what

The shared manifest is src/fwl_io/data/shared_manifest.toml in the fwl-io repository; its package is fwl-io. A dataset that one model reads is declared in that model's own manifest, and its pull request goes to that model's repository, where the maintainers of the model review it.

A record has a new version

The example is the real Zenodo record "atmos_clim/thermo". Its version 10.5281/zenodo.21390786 holds 6 files. Its next version, 10.5281/zenodo.23278603, holds a changed gases.zip, a new plt.zip and the same 5 other files. The dataset doi:10.34894/EHV2DU on DataverseNL is a mirror of the first of the two. The manifest entry is hypothetical: no fwl-io manifest declares this record (see Data that a code also downloads with its own script). Assume that a manifest holds this table:

[atmos_clim.thermo]
zenodo = "10.5281/zenodo.21390786"
dataverse = "10.34894/EHV2DU"

After the steps it holds:

[atmos_clim.thermo]
zenodo = "10.5281/zenodo.23278603"

1. Publish the new version of the record on Zenodo, and make sure that it is an accepted record of a community of the framework: Add a dataset has the rule, the routes, and who can make a new version of a record. Open the page of that version. Its address ends in the record id, https://zenodo.org/records/<record-id>, and the page shows the same number in its DOI, 10.5281/zenodo.<record-id>: that is the version DOI. The concept DOI of the record has another number, and its address leads to the page of the newest version; a manifest cannot pin it.

2. Clone the repository that holds the manifest, and install its package in editable mode, in a Python environment with fwl-io (Installation). For the shared manifest the repository is fwl-io itself:

   pip install -e .
   

   Print the manifest file of every installed package:

   python -c "
   from importlib.metadata import entry_points
   for e in entry_points(group='fwl_io.manifests'):
       try:
           print(e.name, e.load()())
       except Exception as exc:
           print(e.name, 'FAILED:', exc)
   "
   

   Each line holds the name of an entry point and the path of its manifest, or FAILED: and the reason for a package that does not load; fwl-io-shared is the shared manifest. The path of your manifest must lie inside your clone; a path in site-packages means that the package is not installed in editable mode. fwl-io list prints the datasets of each manifest under the same names, so it shows which manifest declares your dataset key. 3. Open that manifest and replace the DOI in the zenodo line of your dataset with the new version DOI. When the table has a files line, it lists the files of the record that the dataset uses: add the name of a new file that the model needs, and remove the name of a file that the new version does not hold. 4. Remove the dataverse line of that dataset (What happens to the mirror gives the reason). When a mirror of the new version is published, put its DOI there in place of the old one. 5. Regenerate the registry. Run the command in the root of the clone, with the path of the manifest from step 2:

   fwl-io sync path/to/manifest.toml
   

   The command prints one line per registry file, wrote path/to/<dataset key>.registry.txt. It reads the Zenodo record of every dataset in the manifest, so for the shared manifest it takes some minutes.

   When it fails, it prints no wrote line and exits 1:

   • N dataset(s) failed to sync, with one line per dataset and its reason. The registries of the other datasets are written (git status shows them). When your dataset is in the list because Zenodo did not answer, run the command again. For a 404, check the record id. For a concept DOI, use the version DOI of step 1. For a name in files that is not in the record, correct the files line. A dataset of another owner in the list does not block you: its registry stays as it was.
   • One line that names your dataset and says that its zenodo value is not a Zenodo DOI of the form 10.5281/zenodo.<record-id>: correct the line. No registry is written.
   • Read the change with git diff. The registry of your dataset shows the files that changed and their new checksums. No other registry must change: when one does, its Zenodo record differs from what is committed, which is not part of your change. Restore that file with git checkout -- <file> and open an issue on the repository that holds the manifest.
   • Fetch the dataset. FWL_DATA must name the directory that fwl-io downloads into (export FWL_DATA=/path/to/data; Getting started):

   fwl-io fetch --key <dataset key>
   fwl-io path <dataset key>
   

   The fetch prints <dataset key>: N file(s). The path ends in r<record-id> with the id of the new version: each version has its own directory, and the directory of the previous version stays on disk until fwl-io prune --delete removes it (CLI reference).

   When Zenodo is slow, the fetch prints lines that start with mirror failed for and retrying; it tries each file up to 4 times. When retrying lines came first and the fetch then prints could not obtain '<file>' from any mirror, with one line per source and its error, run it again later. Without retrying lines the error is permanent (a 404, or a checksum that is not the one in the registry): run fwl-io sync again and compare the registry. 8. Check the files on disk against the registry, when the table of your dataset has a required_by line. Use one of the models that the line names:

   fwl-io check <model>
   

   Read the line of your dataset. It must say <dataset key>: ok, N file(s), or <dataset key>: ok, N file(s), presence only for a dataset with an extract line, whose unpacked files are tested for presence and not hashed.

   The command covers every dataset of the model. The other datasets are FAILED and the last line is data check FAILED unless you fetched them too, with fwl-io fetch <model>, which can download many gigabytes. The exit code 1 of such a run does not concern your dataset. A dataset without a required_by line has no check command; the fetch of step 7 has compared each file with the registry. 9. Commit the manifest and the registry file together and open a pull request. It must contain: - the new zenodo DOI, - the regenerated registry file of that dataset, and of no other dataset, - the change in the model code, when a file has a new name or a new layout. 10. A maintainer of that repository reviews and merges the pull request. Users get the version with the next release of that package, which its maintainers make.

What happens to the mirror

A mirror dataset on DataverseNL is a copy of one Zenodo version. It stays a correct mirror of that version, and it does not become a mirror of the next one. The new version needs a new mirror dataset and a new dataverse pin (next section). With the old dataverse line removed (step 4) and until the new pin is merged, the dataset has no fallback: a fetch downloads from Zenodo, and fails when Zenodo does not answer.

Do not keep the old dataverse pin beside the new zenodo DOI. fwl-io check-mirrors reports such a pin as wrong and exits 1, which fails the weekly job when the dataset is in the shared manifest:

FAIL <dataset key>: doi:10.34894/EHV2DU does not name Zenodo 10.5281/zenodo.23278603 as its source

A mirror for my dataset

A mirror is a second download source, used when Zenodo does not answer. The mirror workflow needs only the version DOI of an accepted record (the rule), so you can ask for a mirror as soon as a maintainer has accepted the version. The usual order is two pull requests: the first with the zenodo pin and no dataverse line, the second with the dataverse pin when the mirror is published. One pull request with both pins is as good when the mirror is published before the review.

The dataset owner asks

1. Write into the dataset request that brought the record into the community, or open a dataset request with the route "My record is an accepted record; I ask for a mirror". Give the version DOI, the dataset key, the repository that holds the manifest and, when the dataset sets files, the names of those files. A maintainer starts the mirror when the record is an accepted record of a community of the framework, and says so in the issue.

A maintainer creates the mirror

The mirror workflows are on the Actions tab of the fwl-io repository, and each run of a job that holds the DataverseNL token waits for the approval of a maintainer (Approval); the weekly report and the pin workflow hold no DataverseNL token and need none. Mirror a deposit to Dataverse has the details of each input and says what to do when a run fails.

1. Run the workflow Mirror a Zenodo deposit to Dataverse. The table gives the name of each input, the description that the workflow file gives it, and the value:

   Input	Description in the workflow	Value
   zenodo_doi	Zenodo version DOI to mirror (10.5281/zenodo.<record-id>)	the version DOI
   collection	Dataverse collection alias	Proteus_Fr (the default of the form)
   files	Only these files of the deposit, separated by spaces (empty: the whole deposit)	empty for the whole record; the names from the manifest when the dataset sets files (a name must not contain a space)
   licence	Dataverse license name for a new draft in place of the Zenodo one, only where the author licenses it so; never published (empty: from Zenodo)	empty (What the mirror does gives the one case for a value)
   into	Persistent id of an existing draft of this record to complete, never published (empty: create one)	empty
   dry_run	Download and map metadata only; make no Dataverse changes	unchecked
   publish	Publish the created dataset (unchecked: leave it a draft to review)	unchecked

   The workflow runs in the dataverse environment of the repository and takes the DataverseNL token and the contact email that every mirror dataset carries from its secrets; the form has no input for them. The run downloads each file from Zenodo, checks it, uploads it to a new draft dataset and checks it there. Its log ends with draft doi:10.34894/<id> created, not published; verify its files, then run fwl-io mirror-publish doi:10.34894/<id>; the workflow of step 3 runs that command. The draft is private. 2. Open the draft on DataverseNL (https://dataverse.nl/dataset.xhtml?persistentId=doi:10.34894/<id>, signed in with access to the collection). Compare the title, the authors, the description, the license and the names of the files with the Zenodo record. The maintainer decides: when all of them agree, publish; when one differs, do not publish, and say in the issue what differs. A published dataset is public and has a permanent DOI. 3. Run the workflow Publish an existing Dataverse draft with persistent_id doi:10.34894/<id> and version_type major (the default of the form). The run checks the source record of the draft first (The check before a publish). 4. Write the DOI of the mirror, 10.34894/<id>, into the issue.

The dataset owner pins the mirror

For a dataset of the shared manifest, a maintainer can do steps 1 and 2 with the workflow Pin a published Dataverse mirror: it checks the pin and pushes a branch with the manifest edit, and a person opens the pull request of step 3 from that branch (The pin). By hand:

1. Add the pin to the manifest entry, below its zenodo line:

   dataverse = "10.34894/<id>"
   

2. Check the pin, with the package that holds the manifest installed in editable mode:

   fwl-io check-mirrors
   

   The command checks the pins of every installed manifest: it reads each pin from its Dataverse server and the file sizes from Zenodo. The CLI reference lists its lines and exit codes and gives the test for a served pin. When your pin is UNREACHABLE, a server did not answer: run the command again later. A line that names another dataset is not caused by your change. 3. Open a pull request with the manifest change on the repository that holds the manifest. The registry does not change. On fwl-io, a pull request gets a review request to the team proteus-maintainer and needs the approval of a code owner.

Data that a code also downloads with its own script

A code can hold Zenodo record ids outside any manifest. AGNI's src/get_data.sh downloads its data with record ids written in the script; gases.zip of the example comes from record 21390786 there, and no fwl-io manifest declares that record. A new version of such a record needs the id changed in that script too, in a pull request on that code. fwl-io does not read these ids: fwl-io check, fwl-io check-mirrors and the weekly job do not see them.

Use fwl-io data from a non-Python code shows how such a code reads a dataset through the fwl-io command.

What the weekly job checks

Two things run every week, and neither changes anything: the workflow named Nightly, which this section describes, and the weekly report of datasets without a mirror (The weekly report, the draft and the pin). A third runs every day and changes nothing either: the PALEOS check latest-paleos.yaml, which fails when a PALEOS record has a newer version than its pin (fwl-io latest).

The Nightly workflow runs every Monday, in two steps.

• Run slow tier (live Zenodo checks) reads the Zenodo record of every dataset in the shared manifest and compares its file names and checksums with the committed registry. A difference means that the record changed after the registry was written, or that the registry was edited by hand. Run fwl-io sync on the manifest, read the diff, and open a pull request, or restore the registry. The same step fails with not in a Zenodo community of ('proteus_framework', 'paleos') and a list of DOIs when a record of the shared manifest is not an accepted record of one of them, and with not read after the retries when Zenodo did not answer for a record. For a record outside the communities: bring the record into the community (Add a dataset), or replace its pin.
• Check the DataverseNL mirror pins runs fwl-io check-mirrors. A FAIL line names a pin that does not serve its dataset: remove or replace the pin in a pull request. An UNREACHABLE line names a pin that could not be read: when the other pins are served, the job passes with a warning, and the next run reads the pin again.

The weekly report lists the records with a newer version, and a report changes nothing: a new version stays unused until its owner follows A record has a new version. The job reads the manifests installed in its run, which is the shared manifest alone. It does not read the manifest of a model; the repository of that model is the place for such a check.