Skip to content

Prune

prune

Remove versioned dataset directories that no installed manifest references.

A manifest dataset lives at <data_root>/<subdir>/r<record-id>, where the record id is the Zenodo version the manifest pins. When a pin advances the fetcher writes a new r<record-id> beside the old one, and the old one stays on disk for as long as the tree exists. This finds those left-behind version directories and, only when asked, deletes them.

Nothing is deleted on trust. A version directory is removed only when it can be proven unreferenced: it must carry the fetcher's own stamp naming the record it holds and the subdir it sits under, it must sit under a subdirectory that a currently declared dataset uses (an older pin of a known dataset), and the reference set it is checked against must be complete. If any installed manifest is left out (it fails to load or conflicts with another), any dataset's version directory cannot be computed, part of the tree cannot be read, or a fetch lock is held anywhere on the tree, the run refuses to delete rather than act on a partial or contested view. An orphan-including run also refuses when the reference set is empty.

A directory that matches the version-directory name shape but carries no stamp naming its own record id and location is never a delete target: it is reported separately as unrecognised, since the name alone is not proof of what it holds.

A version directory under a subdirectory no installed manifest knows is reported as orphaned rather than removed. On a shared data tree it may be the pinned version for a manifest installed in another environment, which this process cannot see, so it is never deleted without an explicit opt-in.

The default is a dry run: the plan is printed and nothing is touched. Deletion needs an explicit request and, unless suppressed, an interactive confirmation.

A run assumes that no other process renames or replaces directories under the data root while it deletes. An fwl-io fetch is kept out only while it holds the fetch lock, which is not the whole of every fetch.

PruneCandidate(path, rel, state, size=0, detail='') dataclass

One r<record-id> directory on disk, and what may be done with it.

summary()

One line naming the directory, its state, and its size on disk.

Source code in src/fwl_io/prune.py
117
118
119
120
121
122
def summary(self) -> str:
    """One line naming the directory, its state, and its size on disk."""
    line = f'{self.rel}: {self.state}, {human_bytes(self.size)}'
    if self.detail:
        line += f', {self.detail}'
    return line

PruneReport(candidates=(), manifest_errors=dict(), resolve_error=None, scan_error=None, known_subdirs=frozenset(), lock_problem=None, apply_refusal=None, lock_warnings=(), staged_remnants=()) dataclass

Every version directory found, and the state of the reference set.

manifest_errors and resolve_error are carried because either one means the reference set is incomplete. Without them a run that could prove nothing unreferenced would read exactly like a tree with nothing to remove.

blocked property

True when the reference set is incomplete, so nothing may be deleted.

empty_reference_set property

True when no installed manifest declares a single dataset.

ok property

True when the reference set was complete and no removal failed or was refused.

orphaned property

Under a subdir no installed manifest knows; deleted only on opt-in.

problems property

Delete targets that could not be removed or were refused at act time.

referenced property

Version directories a current pin uses; always kept.

removed property

Version directories this run deleted.

superseded property

Older pins of known datasets; the default delete target.

unrecognised property

Version-shaped directories with no matching stamp; never deleted.

deletion_refusal(*, include_orphans, allow_empty_reference_set=False)

Why deletion may not proceed, or None if it may.

Checked in order: an incomplete reference set, an unreadable tree, a held or untestable fetch lock, and, only when orphans are in scope, an empty reference set without the explicit override. Every one of these makes "not referenced" indistinguishable from "not yet proven referenced", so each blocks deletion on its own.

Source code in src/fwl_io/prune.py
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
def deletion_refusal(
    self, *, include_orphans: bool, allow_empty_reference_set: bool = False
) -> str | None:
    """Why deletion may not proceed, or ``None`` if it may.

    Checked in order: an incomplete reference set, an unreadable tree, a
    held or untestable fetch lock, and, only when orphans are in scope, an empty
    reference set without the explicit override. Every one of these makes
    "not referenced" indistinguishable from "not yet proven referenced",
    so each blocks deletion on its own.
    """
    if self.blocked:
        return f'the reference set is incomplete ({_INCOMPLETE})'
    if self.scan_error is not None:
        return f'the data root cannot be fully read ({self.scan_error})'
    if self.lock_problem is not None:
        return self.lock_problem
    if include_orphans and self.empty_reference_set and not allow_empty_reference_set:
        return (
            'no installed manifest declares any dataset; '
            'pass the override to prune orphans anyway'
        )
    return None

reclaimable(*states)

Total bytes held by the version directories in the given states.

Source code in src/fwl_io/prune.py
187
188
189
def reclaimable(self, *states: str) -> int:
    """Total bytes held by the version directories in the given states."""
    return sum(c.size for c in self.candidates if c.state in states)

summary()

A short report: one line per version directory plus per-category totals.

Source code in src/fwl_io/prune.py
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
def summary(self) -> str:
    """A short report: one line per version directory plus per-category totals."""
    lines: list[str] = []
    if self.scan_error is not None:
        lines.append(f'DATA ROOT UNREADABLE, {self.scan_error}')
    for c in sorted(self.candidates, key=lambda c: c.rel):
        lines.append(c.summary())
    for provider, error in sorted(self.manifest_errors.items()):
        lines.append(f'{provider}: MANIFEST {error.verdict}, {error.message}')
    if self.resolve_error is not None:
        lines.append(f'VERSION DIR UNRESOLVABLE, {self.resolve_error}')
    kept = len(self.referenced)
    sup, orph, unrec = len(self.superseded), len(self.orphaned), len(self.unrecognised)
    gone, bad = len(self.removed), len(self.problems)
    closing = (
        f'{kept} referenced kept; '
        f'{sup} superseded ({human_bytes(self.reclaimable(SUPERSEDED))}); '
        f'{orph} orphaned ({human_bytes(self.reclaimable(ORPHANED))}); '
        f'{unrec} unrecognised ({human_bytes(self.reclaimable(UNRECOGNISED))})'
    )
    if gone or bad:
        closing += f'; {gone} removed, {bad} not removed'
    if self.blocked:
        # The reason deletion is refused: a version listed superseded above
        # could be the referenced one whose pin this run failed to read.
        closing += f'; reference set is INCOMPLETE ({_INCOMPLETE}), so nothing can be deleted'
    if self.lock_problem is not None:
        closing += f'; {self.lock_problem}, so nothing can be deleted'
    if self.apply_refusal is not None:
        closing += f'; deletion was refused at apply time ({self.apply_refusal})'
    lines.append(closing)
    lines.extend(f'WARNING: {warning}' for warning in self.lock_warnings)
    if self.staged_remnants:
        lines.append(
            f'{len(self.staged_remnants)} partly deleted version dir(s) left in '
            f'{self.staged_remnants[0].parent} by an earlier prune; delete them by hand'
        )
    return '\n'.join(lines)

apply_prune(report, data_root=None, *, include_orphans=False, allow_empty_reference_set=False)

Delete exactly the version directories a prior plan reported.

The plan the caller showed and confirmed is the most that is deleted. A directory is removed only when the plan listed it as a target (superseded, or orphaned when include_orphans is set) and a fresh classification made by this call still does. A directory the plan showed in any other state is kept, even when the tree changed so that it now looks like a target, and so is a directory a manifest started to use between plan and apply. Every removal also re-checks its own guards at the moment it runs.

Deletion is refused outright on a platform that cannot delete safely (see :func:_delete_unsupported), when the current reference-set state no longer permits it, or when the plan lists a directory outside this data root.

Parameters:

Name Type Description Default
report PruneReport

The plan to act on, as returned by :func:plan_prune.

required
data_root str | Path | None

Override for the data root; defaults to the resolved FWL_DATA tree. Must be the same root the plan was built against.

None
include_orphans bool

Also remove the orphaned version directories in the plan.

False
allow_empty_reference_set bool

Permit an orphan-including apply to proceed even when the current reference set declares no dataset.

False

Returns:

Type Description
PruneReport

The plan, with each removed directory's entry rewritten to say so, and the reference-set error signals recomputed at apply time. ok is false, and apply_refusal names why, when the tree changed under this call into a state the plan could no longer delete from.

Raises:

Type Description
FileNotFoundError

When the data root does not exist.

Source code in src/fwl_io/prune.py
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
def apply_prune(
    report: PruneReport,
    data_root: str | Path | None = None,
    *,
    include_orphans: bool = False,
    allow_empty_reference_set: bool = False,
) -> PruneReport:
    """Delete exactly the version directories a prior plan reported.

    The plan the caller showed and confirmed is the most that is deleted. A
    directory is removed only when the plan listed it as a target (superseded,
    or orphaned when ``include_orphans`` is set) and a fresh classification
    made by this call still does. A directory the plan showed in any other
    state is kept, even when the tree changed so that it now looks like a
    target, and so is a directory a manifest started to use between plan and
    apply. Every removal also re-checks its own guards at the moment it runs.

    Deletion is refused outright on a platform that cannot delete safely
    (see :func:`_delete_unsupported`), when the current reference-set state
    no longer permits it, or when the plan lists a directory outside this
    data root.

    Parameters
    ----------
    report : PruneReport
        The plan to act on, as returned by :func:`plan_prune`.
    data_root : str | Path | None
        Override for the data root; defaults to the resolved FWL_DATA tree. Must
        be the same root the plan was built against.
    include_orphans : bool
        Also remove the orphaned version directories in the plan.
    allow_empty_reference_set : bool
        Permit an orphan-including apply to proceed even when the current
        reference set declares no dataset.

    Returns
    -------
    PruneReport
        The plan, with each removed directory's entry rewritten to say so, and
        the reference-set error signals recomputed at apply time. ``ok`` is
        false, and ``apply_refusal`` names why, when the tree changed under
        this call into a state the plan could no longer delete from.

    Raises
    ------
    FileNotFoundError
        When the data root does not exist.
    """
    root = existing_data_root(data_root)
    unsupported = _delete_unsupported()
    if unsupported is not None:
        return _refused(report, unsupported)
    stray = next((c for c in report.candidates if not _inside(c.path, root)), None)
    if stray is not None:
        return _refused(
            report, f'the plan lists {stray.path}, outside {root}; it was built for another root'
        )
    build = _build(root, for_delete=True)
    current = build.report()
    refusal = current.deletion_refusal(
        include_orphans=include_orphans, allow_empty_reference_set=allow_empty_reference_set
    )
    if refusal:
        # The tree changed under us into a state the plan could not delete from.
        return _refused(current, refusal, candidates=report.candidates)
    targets = _target_states(include_orphans=include_orphans)
    # A candidate must be a target both in the plan the caller confirmed and
    # in this call's fresh build: the plan bounds what may go, and the fresh
    # state catches a directory that stopped being safe to remove since.
    fresh_by_dir = {fresh.path: fresh for fresh in build.candidates}
    results = []
    for c in report.candidates:
        fresh = fresh_by_dir.get(c.path)
        if fresh is None:
            fresh = next((f for f in build.candidates if _same_entry(f.path, c.path)), None)
        if fresh is None:
            if not os.path.lexists(c.path):
                results.append(replace(c, state=GONE, size=0, detail='no longer on disk'))
            elif c.state in targets:
                detail = 'no longer a version dir; not removed'
                results.append(replace(c, state=REFUSED, detail=detail))
            else:
                results.append(c)
            continue
        if c.state not in targets:
            results.append(fresh)
            continue
        if fresh.state not in targets:
            # Confirmed for deletion but no longer a target: kept, and not a clean run.
            detail = f'now {fresh.state}; not removed'
            results.append(replace(fresh, state=REFUSED, detail=detail))
            continue
        results.append(
            _remove_one(
                fresh,
                root,
                build.referenced,
                build.referenced_link_ids,
                case_insensitive=build.case_insensitive,
            )
        )
    return replace(current, candidates=tuple(results))

human_bytes(count)

Render a byte count as a short human-readable size.

Source code in src/fwl_io/prune.py
266
267
268
269
270
271
272
273
def human_bytes(count: int) -> str:
    """Render a byte count as a short human-readable size."""
    size = float(count)
    for unit in ('B', 'KiB', 'MiB', 'GiB', 'TiB'):
        if size < 1024 or unit == 'TiB':
            return f'{size:.0f} {unit}' if unit == 'B' else f'{size:.1f} {unit}'
        size /= 1024
    return f'{count} B'

plan_prune(data_root=None)

Report which version directories would be removed, touching nothing.

Parameters:

Name Type Description Default
data_root str | Path | None

Override for the data root; defaults to the resolved FWL_DATA tree.

None

Returns:

Type Description
PruneReport

One entry per r<record-id> directory found, classified against the installed manifests, with per-category reclaimable bytes.

Raises:

Type Description
FileNotFoundError

When the data root does not exist. A plan never creates it.

Source code in src/fwl_io/prune.py
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
def plan_prune(data_root: str | Path | None = None) -> PruneReport:
    """Report which version directories would be removed, touching nothing.

    Parameters
    ----------
    data_root : str | Path | None
        Override for the data root; defaults to the resolved FWL_DATA tree.

    Returns
    -------
    PruneReport
        One entry per ``r<record-id>`` directory found, classified against the
        installed manifests, with per-category reclaimable bytes.

    Raises
    ------
    FileNotFoundError
        When the data root does not exist. A plan never creates it.
    """
    return _build(existing_data_root(data_root)).report()

prune_versions(data_root=None, *, delete=False, include_orphans=False, allow_empty_reference_set=False)

Plan a prune and, when delete is set, remove the proven-unreferenced set.

A superseded version directory is removed by default. An orphaned one is removed only when include_orphans is set, because it cannot be proven unreferenced by manifests this environment cannot see. Deletion is refused outright on a platform that cannot delete safely (see :func:_delete_unsupported), when the reference set is incomplete, the tree cannot be fully read, a fetch lock is held or cannot be checked, or (for an orphan-including run) the reference set is empty and the override was not given.

Parameters:

Name Type Description Default
data_root str | Path | None

Override for the data root; defaults to the resolved FWL_DATA tree.

None
delete bool

Remove the target directories. When False this is a dry run.

False
include_orphans bool

Also remove orphaned version directories. Has no effect without delete.

False
allow_empty_reference_set bool

Permit an orphan-including run to proceed even when no installed manifest declares any dataset. Has no effect without delete and include_orphans.

False

Returns:

Type Description
PruneReport

The plan, with each removed directory's entry rewritten to say so.

Raises:

Type Description
FileNotFoundError

When the data root does not exist.

Source code in src/fwl_io/prune.py
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
def prune_versions(
    data_root: str | Path | None = None,
    *,
    delete: bool = False,
    include_orphans: bool = False,
    allow_empty_reference_set: bool = False,
) -> PruneReport:
    """Plan a prune and, when ``delete`` is set, remove the proven-unreferenced set.

    A superseded version directory is removed by default. An orphaned one is
    removed only when ``include_orphans`` is set, because it cannot be proven
    unreferenced by manifests this environment cannot see. Deletion is refused
    outright on a platform that cannot delete safely (see
    :func:`_delete_unsupported`), when the reference set is incomplete, the
    tree cannot be fully read, a fetch lock is held or cannot be checked, or
    (for an orphan-including run) the reference set is empty and the override
    was not given.

    Parameters
    ----------
    data_root : str | Path | None
        Override for the data root; defaults to the resolved FWL_DATA tree.
    delete : bool
        Remove the target directories. When False this is a dry run.
    include_orphans : bool
        Also remove orphaned version directories. Has no effect without
        ``delete``.
    allow_empty_reference_set : bool
        Permit an orphan-including run to proceed even when no installed
        manifest declares any dataset. Has no effect without ``delete`` and
        ``include_orphans``.

    Returns
    -------
    PruneReport
        The plan, with each removed directory's entry rewritten to say so.

    Raises
    ------
    FileNotFoundError
        When the data root does not exist.
    """
    root = existing_data_root(data_root)
    unsupported = _delete_unsupported() if delete else None
    build = _build(root, for_delete=delete and unsupported is None)
    report = build.report()
    if unsupported is not None:
        return _refused(report, unsupported)
    if not delete or report.deletion_refusal(
        include_orphans=include_orphans, allow_empty_reference_set=allow_empty_reference_set
    ):
        # A dry run, or a run the reference set will not allow: report, do nothing.
        return report
    targets = _target_states(include_orphans=include_orphans)
    results = [
        _remove_one(
            c,
            root,
            build.referenced,
            build.referenced_link_ids,
            case_insensitive=build.case_insensitive,
        )
        if c.state in targets
        else c
        for c in build.candidates
    ]
    return replace(report, candidates=tuple(results))