Skip to content

Check

check

Report whether a data tree matches its manifest, without downloading.

This sits between the two behaviours the fetcher already has. Offline mode serves what is present and raises as soon as something is not; online mode downloads whatever is missing. Neither can answer "is this tree complete and intact", which is what a diagnostic such as proteus doctor needs: it wants the whole picture in one pass, and it must not repair the thing it is inspecting.

A check therefore never reaches the network, never downloads, and never creates or repairs a dataset. It reads the manifest, hashes what is on disk, and returns a report. Resolving a path creates the data root when it is absent, exactly as every other entry point does; no dataset directory and no file is written.

The report is careful about what it did not establish, because a diagnostic that overstates its own coverage is worse than none. A manifest that fails to load and a dataset whose registry cannot be read are both carried in the report and both count against the verdict, since their files were never looked at, and they are carried apart from each other because they call for different repairs. An archive dataset's members are reported as present rather than as verified: the archive-only checksum policy records member names, not per-file digests, so once the archive is gone there is nothing to hash them against.

CheckReport(datasets=dict(), manifest_errors=dict(), dataset_errors=dict()) dataclass

Every dataset checked, and everything that stopped one being checked.

The two error maps are kept apart because they call for different repairs. A manifest error means an installed package's manifest could not be read at all, so nothing it declares was inspected. A dataset error means the manifest was fine but that one dataset could not be resolved, most often because its registry has never been generated.

faults property

Datasets with something wrong, the worst affected named first.

Datasets only. A report failed by an unreadable manifest or a dataset that could not be resolved has nothing to put here, so this being empty is not the same as nothing being wrong; ok is the question that covers every reason.

ok property

True only when something was checked and all of it was sound.

An empty report is not ok. A caller asking about a model and being told nothing is wrong, when in truth nothing was looked at, is the failure this whole module exists to avoid; the two are indistinguishable to anyone reading a boolean.

presence_only property

Datasets whose files carry no digest to be checked against.

verified property

True when the tree is sound and every file in it was hashed.

Stricter than ok, which a presence-only dataset satisfies. Presence is all the archive-only checksum policy makes checkable, so such a dataset is not a fault; but a caller that needs to know the contents were compared against a digest must ask this and not ok.

summary()

A short human-readable report, one line per dataset plus a verdict.

Source code in src/fwl_io/check.py
187
188
189
190
191
192
193
194
195
196
197
def summary(self) -> str:
    """A short human-readable report, one line per dataset plus a verdict."""
    lines = [d.summary() for d in sorted(self.datasets.values(), key=lambda d: d.key)]
    for provider, error in sorted(self.manifest_errors.items()):
        lines.append(f'{provider}: MANIFEST UNREADABLE, {error}')
    for key, error in sorted(self.dataset_errors.items()):
        lines.append(f'{key}: NOT CHECKED, {error}')
    if not lines:
        return 'nothing was checked'
    lines.append(self._verdict())
    return '\n'.join(lines)

DatasetCheck(key, subdir, directory, files, verifiable) dataclass

The state of every file in one dataset.

verifiable says whether this dataset's registry carries a digest for each file it declares. It is false for an archive dataset, whose members are recorded by name only, and it is a property of the dataset rather than of what happens to be on disk, so an archive dataset with no members left cannot read as verifiable. It has no default: the wrong value is the one that lets a presence-only dataset read as verified, so every caller states it rather than inheriting it.

complete property

True when nothing is missing, corrupt, or unreadable.

faults property

Every file that is not in a usable state.

mismatched property

Files on disk whose contents differ from the registry.

missing property

Files the manifest declares that are not on disk.

unreadable property

Files on disk that could not be read to be checked.

summary()

One line naming the counts, for a report a person reads.

Source code in src/fwl_io/check.py
115
116
117
118
119
120
121
122
123
124
125
def summary(self) -> str:
    """One line naming the counts, for a report a person reads."""
    parts = [f'{len(self.files)} file(s)']
    for state, label in FAULT_LABELS.items():
        group = self._in_state(state)
        if group:
            parts.append(f'{len(group)} {label}')
    if not self.verifiable:
        parts.append('presence only')
    state = 'ok' if self.complete else 'FAILED'
    return f'{self.key}: {state}, ' + ', '.join(parts)

FileCheck(name, path, state) dataclass

The state of one file the manifest declares.

faulty property

True when this file is absent, corrupt, or could not be read.

check_dataset(fetcher, key='')

Report the state of one dataset's files without touching the network.

Parameters:

Name Type Description Default
fetcher Fetcher

Configured for the dataset to inspect. Nothing is fetched.

required
key str

Name for the dataset in the report; defaults to its subdirectory.

''

Returns:

Type Description
DatasetCheck

One entry per file the manifest declares. For an archive dataset the entries are the extracted members recorded in the provenance stamp, reported as present rather than verified, since the registry pins the archive's checksum and not the checksums of what came out of it.

Source code in src/fwl_io/check.py
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
def check_dataset(fetcher: Fetcher, key: str = '') -> DatasetCheck:
    """Report the state of one dataset's files without touching the network.

    Parameters
    ----------
    fetcher : Fetcher
        Configured for the dataset to inspect. Nothing is fetched.
    key : str, optional
        Name for the dataset in the report; defaults to its subdirectory.

    Returns
    -------
    DatasetCheck
        One entry per file the manifest declares. For an archive dataset the
        entries are the extracted members recorded in the provenance stamp,
        reported as present rather than verified, since the registry pins the
        archive's checksum and not the checksums of what came out of it.
    """
    key = key or fetcher.subdir

    if fetcher.extract is not None:
        members = fetcher.recorded_members()
        if members is None:
            # No usable stamp means no extracted tree to speak of. The dataset
            # is reported as one missing item under the archive's own name,
            # rather than as zero items, which would read as complete.
            archive_name = next(iter(fetcher.registry))
            files = (FileCheck(archive_name, fetcher.target_dir / archive_name, MISSING),)
            return DatasetCheck(key, fetcher.subdir, fetcher.target_dir, files, verifiable=False)
        checks = [
            FileCheck(name, fetcher.target_dir / name, _member_state(fetcher.target_dir / name))
            for name in sorted(members)
        ]
        return DatasetCheck(
            key, fetcher.subdir, fetcher.target_dir, tuple(checks), verifiable=False
        )

    checks = [
        FileCheck(name, fetcher.target_dir / name, _file_state(fetcher, name))
        for name in sorted(fetcher.registry)
    ]
    return DatasetCheck(key, fetcher.subdir, fetcher.target_dir, tuple(checks), verifiable=True)

check_for(model, data_root=None)

Report the state of every dataset a given model requires.

Nothing is downloaded and no dataset directory or file is written, so this is safe to run against a tree another process is reading.

Nothing here raises for the state of the data or of a manifest. A caller running a check wants the whole picture, including the parts that could not be established, so every failure is carried in the report instead.

Parameters:

Name Type Description Default
model str

Model name matched (case-insensitively) against required_by.

required
data_root str | Path | None

Override for the data root; defaults to the resolved FWL_DATA tree.

None

Returns:

Type Description
CheckReport

Keyed by dataset, alongside the manifests that could not be read and the datasets that could not be resolved.

Source code in src/fwl_io/check.py
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
def check_for(model: str, data_root: str | Path | None = None) -> CheckReport:
    """Report the state of every dataset a given model requires.

    Nothing is downloaded and no dataset directory or file is written, so this
    is safe to run against a tree another process is reading.

    Nothing here raises for the state of the data or of a manifest. A caller
    running a check wants the whole picture, including the parts that could not
    be established, so every failure is carried in the report instead.

    Parameters
    ----------
    model : str
        Model name matched (case-insensitively) against ``required_by``.
    data_root : str | Path | None
        Override for the data root; defaults to the resolved FWL_DATA tree.

    Returns
    -------
    CheckReport
        Keyed by dataset, alongside the manifests that could not be read and
        the datasets that could not be resolved.
    """
    from fwl_io.manifest import _discover

    model = model.lower()
    datasets: dict[str, DatasetCheck] = {}
    dataset_errors: dict[str, str] = {}
    providers, manifest_errors = _discover()
    for provider_datasets in providers.values():
        for ds in provider_datasets:
            if model not in tuple(r.lower() for r in ds.required_by):
                continue
            try:
                fetcher = create_fetcher(
                    subdir=ds.subdir,
                    zenodo=ds.zenodo,
                    dataverse=ds.dataverse,
                    registry=ds.registry(),
                    data_root=data_root,
                    extract=ds.extract,
                )
                datasets[ds.key] = check_dataset(fetcher, key=ds.key)
            except Exception as exc:  # noqa: BLE001 -- reported, never raised
                dataset_errors[ds.key] = str(exc)
                log.warning('cannot check dataset %r: %s', ds.key, exc)
    return CheckReport(
        datasets=datasets,
        manifest_errors=dict(manifest_errors),
        dataset_errors=dataset_errors,
    )