Prune
prune
Remove versioned dataset directories that no installed manifest references.
A manifest dataset lives at <data_root>/<subdir>/r<record-id>, where the
record id is the Zenodo version the manifest pins. When a pin advances the
fetcher writes a new r<record-id> beside the old one, and the old one stays
on disk for as long as the tree exists. This finds those left-behind version
directories and, only when asked, deletes them.
Nothing is deleted on trust. A version directory is removed only when it can be proven unreferenced: it must carry the fetcher's own stamp naming the record it holds and the subdir it sits under, it must sit under a subdirectory that a currently declared dataset uses (an older pin of a known dataset), and the reference set it is checked against must be complete. If any installed manifest is left out (it fails to load or conflicts with another), any dataset's version directory cannot be computed, part of the tree cannot be read, or a fetch lock is held anywhere on the tree, the run refuses to delete rather than act on a partial or contested view. An orphan-including run also refuses when the reference set is empty.
A directory that matches the version-directory name shape but carries no stamp naming its own record id and location is never a delete target: it is reported separately as unrecognised, since the name alone is not proof of what it holds.
A version directory under a subdirectory no installed manifest knows is reported as orphaned rather than removed. On a shared data tree it may be the pinned version for a manifest installed in another environment, which this process cannot see, so it is never deleted without an explicit opt-in.
The default is a dry run: the plan is printed and nothing is touched. Deletion needs an explicit request and, unless suppressed, an interactive confirmation.
A run assumes that no other process renames or replaces directories under the data root while it deletes. An fwl-io fetch is kept out only while it holds the fetch lock, which is not the whole of every fetch.
PruneCandidate(path, rel, state, size=0, detail='')
dataclass
One r<record-id> directory on disk, and what may be done with it.
summary()
One line naming the directory, its state, and its size on disk.
Source code in src/fwl_io/prune.py
117 118 119 120 121 122 | |
PruneReport(candidates=(), manifest_errors=dict(), resolve_error=None, scan_error=None, known_subdirs=frozenset(), lock_problem=None, apply_refusal=None, lock_warnings=(), staged_remnants=())
dataclass
Every version directory found, and the state of the reference set.
manifest_errors and resolve_error are carried because either one
means the reference set is incomplete. Without them a run that could prove
nothing unreferenced would read exactly like a tree with nothing to remove.
blocked
property
True when the reference set is incomplete, so nothing may be deleted.
empty_reference_set
property
True when no installed manifest declares a single dataset.
ok
property
True when the reference set was complete and no removal failed or was refused.
orphaned
property
Under a subdir no installed manifest knows; deleted only on opt-in.
problems
property
Delete targets that could not be removed or were refused at act time.
referenced
property
Version directories a current pin uses; always kept.
removed
property
Version directories this run deleted.
superseded
property
Older pins of known datasets; the default delete target.
unrecognised
property
Version-shaped directories with no matching stamp; never deleted.
deletion_refusal(*, include_orphans, allow_empty_reference_set=False)
Why deletion may not proceed, or None if it may.
Checked in order: an incomplete reference set, an unreadable tree, a held or untestable fetch lock, and, only when orphans are in scope, an empty reference set without the explicit override. Every one of these makes "not referenced" indistinguishable from "not yet proven referenced", so each blocks deletion on its own.
Source code in src/fwl_io/prune.py
202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 | |
reclaimable(*states)
Total bytes held by the version directories in the given states.
Source code in src/fwl_io/prune.py
187 188 189 | |
summary()
A short report: one line per version directory plus per-category totals.
Source code in src/fwl_io/prune.py
226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 | |
apply_prune(report, data_root=None, *, include_orphans=False, allow_empty_reference_set=False)
Delete exactly the version directories a prior plan reported.
The plan the caller showed and confirmed is the most that is deleted. A
directory is removed only when the plan listed it as a target (superseded,
or orphaned when include_orphans is set) and a fresh classification
made by this call still does. A directory the plan showed in any other
state is kept, even when the tree changed so that it now looks like a
target, and so is a directory a manifest started to use between plan and
apply. Every removal also re-checks its own guards at the moment it runs.
Deletion is refused outright on a platform that cannot delete safely
(see :func:_delete_unsupported), when the current reference-set state
no longer permits it, or when the plan lists a directory outside this
data root.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
report
|
PruneReport
|
The plan to act on, as returned by :func: |
required |
data_root
|
str | Path | None
|
Override for the data root; defaults to the resolved FWL_DATA tree. Must be the same root the plan was built against. |
None
|
include_orphans
|
bool
|
Also remove the orphaned version directories in the plan. |
False
|
allow_empty_reference_set
|
bool
|
Permit an orphan-including apply to proceed even when the current reference set declares no dataset. |
False
|
Returns:
| Type | Description |
|---|---|
PruneReport
|
The plan, with each removed directory's entry rewritten to say so, and
the reference-set error signals recomputed at apply time. |
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
When the data root does not exist. |
Source code in src/fwl_io/prune.py
872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 | |
human_bytes(count)
Render a byte count as a short human-readable size.
Source code in src/fwl_io/prune.py
266 267 268 269 270 271 272 273 | |
plan_prune(data_root=None)
Report which version directories would be removed, touching nothing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_root
|
str | Path | None
|
Override for the data root; defaults to the resolved FWL_DATA tree. |
None
|
Returns:
| Type | Description |
|---|---|
PruneReport
|
One entry per |
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
When the data root does not exist. A plan never creates it. |
Source code in src/fwl_io/prune.py
546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 | |
prune_versions(data_root=None, *, delete=False, include_orphans=False, allow_empty_reference_set=False)
Plan a prune and, when delete is set, remove the proven-unreferenced set.
A superseded version directory is removed by default. An orphaned one is
removed only when include_orphans is set, because it cannot be proven
unreferenced by manifests this environment cannot see. Deletion is refused
outright on a platform that cannot delete safely (see
:func:_delete_unsupported), when the reference set is incomplete, the
tree cannot be fully read, a fetch lock is held or cannot be checked, or
(for an orphan-including run) the reference set is empty and the override
was not given.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_root
|
str | Path | None
|
Override for the data root; defaults to the resolved FWL_DATA tree. |
None
|
delete
|
bool
|
Remove the target directories. When False this is a dry run. |
False
|
include_orphans
|
bool
|
Also remove orphaned version directories. Has no effect without
|
False
|
allow_empty_reference_set
|
bool
|
Permit an orphan-including run to proceed even when no installed
manifest declares any dataset. Has no effect without |
False
|
Returns:
| Type | Description |
|---|---|
PruneReport
|
The plan, with each removed directory's entry rewritten to say so. |
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
When the data root does not exist. |
Source code in src/fwl_io/prune.py
803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 | |