Skip to content

Relocate

relocate

Move a dataset left by the previous layout into the place it belongs now.

A tree fetched before the current layout existed holds directories such as stellar_evolution_tracks/Baraffe. Unmigrated code still reads them, so they are left alone and age out as their consumers migrate. That is the right default and the wrong one for anybody who wants the tree tidy today, which is what this does: it finds the datasets an installed manifest declares, works out where each one used to live, and moves the files across.

Nothing is moved on trust. Every present file is hashed against the registry the manifest ships before anything is touched, and only a file that matches moves; a present file whose contents do not match blocks the whole dataset, so an absent file can never mask a corrupt one. A file already at the new location is not replaced: an intact one is skipped and a differing one blocks the dataset. A legacy tree holding none of its registry's files is reported and left exactly where it is. The alternative, moving first and discovering afterwards, turns a stale copy into a stale copy in the place the fetcher will now believe.

Nothing is downloaded either. A dataset moved with some registry files still absent stays that way at its new location; the fetcher is what fills it in, once the move has happened.

A dataset packaged as an archive is reported rather than moved. Its registry pins the packed archive, and a legacy tree holds the extracted members, so there is nothing to hash the tree against.

A run assumes that no other process renames or replaces directories under the data root while it moves files. Unlike a deletion, a move does not check the fetch lock, so a fetch running at the same time is not detected at all; do not relocate while a fetch could be running.

Relocation(key, state, legacy_dir=None, target_dir=None, files=(), detail='', legacy_present=False) dataclass

One dataset's legacy tree, and what can be done with it.

faulty property

True when a legacy tree is present but was not usable.

summary()

One line a person reads, naming the dataset and its outcome.

Source code in src/fwl_io/relocate.py
 98
 99
100
101
102
103
104
105
def summary(self) -> str:
    """One line a person reads, naming the dataset and its outcome."""
    line = f'{self.key}: {self.state}'
    if self.state in (READY, MOVED):
        line += f', {len(self.files)} file(s) {self.legacy_dir} -> {self.target_dir}'
    if self.detail:
        line += f', {self.detail}'
    return line

RelocationReport(entries=(), manifest_errors=dict(), layout_error=None) dataclass

Every dataset considered, whether or not anything happened to it.

manifest_errors is carried beside them because a manifest that failed to load, or was dropped for a conflict, declares datasets nobody here got to look at. Without it a report covering nothing would read exactly like a tree with nothing left to move. Each is a ProviderError, whose kind tells a conflict from a load failure, so the summary can tell a caller which repair applies.

faults property

Legacy trees that are present and could not be moved.

moved property

Datasets whose files were moved into the current layout.

ok property

True when every legacy tree found was dealt with and none was skipped.

ready property

Datasets whose legacy tree checks out and is waiting to be moved.

redundant property

Datasets whose old copy is still on disk beside the current one.

Nothing here removes it, so it is worth counting: this is the disk the user can reclaim by hand, and it is the whole reason to run the command against a tree where every dataset has already been refetched.

summary()

A short report, one line per dataset plus a closing count.

Source code in src/fwl_io/relocate.py
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
def summary(self) -> str:
    """A short report, one line per dataset plus a closing count."""
    lines = [e.summary() for e in sorted(self.entries, key=lambda e: e.key)]
    for provider, error in sorted(self.manifest_errors.items()):
        lines.append(f'{provider}: MANIFEST {error.verdict}, {error.message}')
    if self.layout_error is not None:
        # Without this the run reports nothing to do, which is what a tidy
        # tree also reports, and the two are not the same answer.
        lines.append(f'LEGACY LAYOUT UNREADABLE, {self.layout_error}')
    if not lines:
        return 'no dataset declares a legacy location'
    done, waiting, bad = len(self.moved), len(self.ready), len(self.faults)
    closing = f'{done} moved, {waiting} ready to move, {bad} left in place'
    if self.redundant:
        # Named because it is the disk a user can reclaim by hand, and
        # because on a tree where everything has already been refetched it
        # is the only thing the run has to tell them.
        closing += (
            f'; {len(self.redundant)} dataset(s) still have an old copy on disk, '
            'which was left alone'
        )
    if self.manifest_errors:
        # An unloaded or conflict-dropped manifest may declare a dataset this
        # tree still holds, so the counts above are a floor.
        closing += f'; {len(self.manifest_errors)} manifest(s) left out, so this may be partial'
    lines.append(closing)
    return '\n'.join(lines)

plan_relocations(data_root=None)

Report what a relocation would do, touching nothing.

Parameters:

Name Type Description Default
data_root str | Path | None

Override for the data root; defaults to the resolved FWL_DATA tree.

None

Returns:

Type Description
RelocationReport

One entry per dataset that declares a legacy location, whether or not that location exists on this machine.

Source code in src/fwl_io/relocate.py
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
def plan_relocations(data_root: str | Path | None = None) -> RelocationReport:
    """Report what a relocation would do, touching nothing.

    Parameters
    ----------
    data_root : str | Path | None
        Override for the data root; defaults to the resolved FWL_DATA tree.

    Returns
    -------
    RelocationReport
        One entry per dataset that declares a legacy location, whether or not
        that location exists on this machine.
    """
    root = resolve_data_root(data_root)
    locations, layout_error = _legacy_locations()
    entries: list[Relocation] = []
    seen: set[str] = set()
    discovery = _discover_all()
    for provider_datasets in discovery.found.values():
        for ds in provider_datasets:
            legacy = locations.get(ds.key)
            if legacy is None or ds.key in seen:
                continue
            seen.add(ds.key)
            legacy_dir = root / legacy
            try:
                registry = ds.registry()
                target_dir = root / _version_dir(ds)
            except Exception as exc:  # noqa: BLE001 -- reported, never raised
                entries.append(
                    Relocation(ds.key, UNRESOLVABLE, legacy_dir=legacy_dir, detail=str(exc))
                )
                continue
            legacy_present = True  # a directory that cannot be read is still there
            try:
                legacy_present = _is_directory(legacy_dir)
                state, detail, files = _assess(
                    ds, registry, legacy_dir, target_dir, root, legacy_present
                )
            except OSError as exc:
                state, detail, files = (
                    UNRESOLVABLE,
                    f'cannot read {legacy_dir} or {target_dir}: {exc}',
                    (),
                )
            except ValueError as exc:
                state, detail, files = UNRESOLVABLE, f'{ds.key} registry cannot be used: {exc}', ()
            entries.append(
                Relocation(
                    ds.key,
                    state,
                    legacy_dir=legacy_dir,
                    target_dir=target_dir,
                    files=files,
                    detail=detail,
                    legacy_present=legacy_present,
                )
            )
    return RelocationReport(tuple(entries), discovery.errors, layout_error)

relocate_all(data_root=None, dry_run=False)

Move every legacy tree that checks out into the current layout.

Every present file in the legacy location that matches its recorded digest moves; an absent file is reported for the fetcher to fill in at the new location. A present file that does not match its digest, or a legacy tree holding none of its registry's files, is reported and left untouched, as is a dataset already at its current location, which is the ordinary state once a fetch has happened there.

Parameters:

Name Type Description Default
data_root str | Path | None

Override for the data root; defaults to the resolved FWL_DATA tree.

None
dry_run bool

Report what would move without moving it.

False

Returns:

Type Description
RelocationReport

The plan, with each moved dataset's entry rewritten to say so.

Source code in src/fwl_io/relocate.py
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
def relocate_all(data_root: str | Path | None = None, dry_run: bool = False) -> RelocationReport:
    """Move every legacy tree that checks out into the current layout.

    Every present file in the legacy location that matches its recorded
    digest moves; an absent file is reported for the fetcher to fill in at
    the new location. A present file that does not match its digest, or a
    legacy tree holding none of its registry's files, is reported and left
    untouched, as is a dataset already at its current location, which is the
    ordinary state once a fetch has happened there.

    Parameters
    ----------
    data_root : str | Path | None
        Override for the data root; defaults to the resolved FWL_DATA tree.
    dry_run : bool
        Report what would move without moving it.

    Returns
    -------
    RelocationReport
        The plan, with each moved dataset's entry rewritten to say so.
    """
    plan = plan_relocations(data_root)
    if dry_run:
        return plan
    root = resolve_data_root(data_root)
    done = []
    for entry in plan.entries:
        if entry.state != READY:
            done.append(entry)
            continue
        moved = _move_one(entry, root)
        done.append(moved)
        if moved.state in (FAILED, SPLIT):
            # Reported and left for a person to look at; a fault in one
            # dataset says nothing about the ones after it, so they still run.
            log.error('%s could not be relocated', moved.key)
    return RelocationReport(tuple(done), dict(plan.manifest_errors), plan.layout_error)