Check
check
Report whether a data tree matches its manifest, without downloading.
This sits between the two behaviours the fetcher already has. Offline mode
serves what is present and raises as soon as something is not; online mode
downloads whatever is missing. Neither can answer "is this tree complete and
intact", which is what a diagnostic such as proteus doctor needs: it wants
the whole picture in one pass, and it must not repair the thing it is
inspecting.
A check therefore never reaches the network, never downloads, and never creates or repairs a dataset. It reads the manifest, hashes what is on disk, and returns a report. Resolving a path creates the data root when it is absent, exactly as every other entry point does; no dataset directory and no file is written.
The report is careful about what it did not establish, because a diagnostic that overstates its own coverage is worse than none. A manifest that fails to load and a dataset whose registry cannot be read are both carried in the report and both count against the verdict, since their files were never looked at, and they are carried apart from each other because they call for different repairs. An archive dataset's members are reported as present rather than as verified: the archive-only checksum policy records member names, not per-file digests, so once the archive is gone there is nothing to hash them against.
CheckReport(datasets=dict(), manifest_errors=dict(), dataset_errors=dict())
dataclass
Every dataset checked, and everything that stopped one being checked.
The two error maps are kept apart because they call for different repairs. A manifest error means an installed package's manifest could not be read at all, so nothing it declares was inspected. A dataset error means the manifest was fine but that one dataset could not be resolved, most often because its registry has never been generated.
faults
property
Datasets with something wrong, the worst affected named first.
Datasets only. A report failed by an unreadable manifest or a dataset
that could not be resolved has nothing to put here, so this being
empty is not the same as nothing being wrong; ok is the question
that covers every reason.
ok
property
True only when something was checked and all of it was sound.
An empty report is not ok. A caller asking about a model and being told nothing is wrong, when in truth nothing was looked at, is the failure this whole module exists to avoid; the two are indistinguishable to anyone reading a boolean.
presence_only
property
Datasets whose files carry no digest to be checked against.
verified
property
True when the tree is sound and every file in it was hashed.
Stricter than ok, which a presence-only dataset satisfies. Presence
is all the archive-only checksum policy makes checkable, so such a
dataset is not a fault; but a caller that needs to know the contents
were compared against a digest must ask this and not ok.
summary()
A short human-readable report, one line per dataset plus a verdict.
Source code in src/fwl_io/check.py
187 188 189 190 191 192 193 194 195 196 197 | |
DatasetCheck(key, subdir, directory, files, verifiable)
dataclass
The state of every file in one dataset.
verifiable says whether this dataset's registry carries a digest for
each file it declares. It is false for an archive dataset, whose members
are recorded by name only, and it is a property of the dataset rather than
of what happens to be on disk, so an archive dataset with no members left
cannot read as verifiable. It has no default: the wrong value is the one
that lets a presence-only dataset read as verified, so every caller states
it rather than inheriting it.
complete
property
True when nothing is missing, corrupt, or unreadable.
faults
property
Every file that is not in a usable state.
mismatched
property
Files on disk whose contents differ from the registry.
missing
property
Files the manifest declares that are not on disk.
unreadable
property
Files on disk that could not be read to be checked.
summary()
One line naming the counts, for a report a person reads.
Source code in src/fwl_io/check.py
115 116 117 118 119 120 121 122 123 124 125 | |
FileCheck(name, path, state)
dataclass
The state of one file the manifest declares.
faulty
property
True when this file is absent, corrupt, or could not be read.
check_dataset(fetcher, key='')
Report the state of one dataset's files without touching the network.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
fetcher
|
Fetcher
|
Configured for the dataset to inspect. Nothing is fetched. |
required |
key
|
str
|
Name for the dataset in the report; defaults to its subdirectory. |
''
|
Returns:
| Type | Description |
|---|---|
DatasetCheck
|
One entry per file the manifest declares. For an archive dataset the entries are the extracted members recorded in the provenance stamp, reported as present rather than verified, since the registry pins the archive's checksum and not the checksums of what came out of it. |
Source code in src/fwl_io/check.py
240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 | |
check_for(model, data_root=None)
Report the state of every dataset a given model requires.
Nothing is downloaded and no dataset directory or file is written, so this is safe to run against a tree another process is reading.
Nothing here raises for the state of the data or of a manifest. A caller running a check wants the whole picture, including the parts that could not be established, so every failure is carried in the report instead.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
str
|
Model name matched (case-insensitively) against |
required |
data_root
|
str | Path | None
|
Override for the data root; defaults to the resolved FWL_DATA tree. |
None
|
Returns:
| Type | Description |
|---|---|
CheckReport
|
Keyed by dataset, alongside the manifests that could not be read and the datasets that could not be resolved. |
Source code in src/fwl_io/check.py
284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 | |