Skip to content

Relocate

relocate

Move a dataset left by the previous layout into the place it belongs now.

A tree fetched before the current layout existed holds directories such as stellar_evolution_tracks/Baraffe. Unmigrated code still reads them, so they are left alone and age out as their consumers migrate. That is the right default and the wrong one for anybody who wants the tree tidy today, which is what this does: it finds the datasets an installed manifest declares, works out where each one used to live, and moves the files across.

Nothing is moved on trust. Every file is hashed against the registry the manifest ships before anything is touched, and a dataset with a file missing or a file whose contents do not match is reported and left exactly where it is. The alternative, moving first and discovering afterwards, turns a stale copy into a stale copy in the place the fetcher will now believe.

Nothing is downloaded either. A dataset whose legacy tree is incomplete stays incomplete here; the fetcher is what fills it, and it will do so at the current location once the move has happened.

A dataset packaged as an archive is reported rather than moved. Its registry pins the packed archive, and a legacy tree holds the extracted members, so there is nothing to hash the tree against.

Relocation(key, state, legacy_dir=None, target_dir=None, files=(), detail='', legacy_present=False) dataclass

One dataset's legacy tree, and what can be done with it.

faulty property

True when a legacy tree is present but was not usable.

summary()

One line a person reads, naming the dataset and its outcome.

Source code in src/fwl_io/relocate.py
81
82
83
84
85
86
87
88
def summary(self) -> str:
    """One line a person reads, naming the dataset and its outcome."""
    line = f'{self.key}: {self.state}'
    if self.state in (READY, MOVED):
        line += f', {len(self.files)} file(s) {self.legacy_dir} -> {self.target_dir}'
    if self.detail:
        line += f', {self.detail}'
    return line

RelocationReport(entries=(), manifest_errors=dict(), layout_error=None) dataclass

Every dataset considered, whether or not anything happened to it.

manifest_errors is carried beside them because a manifest that failed to load declares datasets nobody here got to look at. Without it a report covering nothing would read exactly like a tree with nothing left to move.

faults property

Legacy trees that are present and could not be moved.

moved property

Datasets whose files were moved into the current layout.

ok property

True when every legacy tree found was dealt with and none was skipped.

ready property

Datasets whose legacy tree checks out and is waiting to be moved.

redundant property

Datasets whose old copy is still on disk beside the current one.

Nothing here removes it, so it is worth counting: this is the disk the user can reclaim by hand, and it is the whole reason to run the command against a tree where every dataset has already been refetched.

summary()

A short report, one line per dataset plus a closing count.

Source code in src/fwl_io/relocate.py
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
def summary(self) -> str:
    """A short report, one line per dataset plus a closing count."""
    lines = [e.summary() for e in sorted(self.entries, key=lambda e: e.key)]
    for provider, error in sorted(self.manifest_errors.items()):
        lines.append(f'{provider}: MANIFEST UNREADABLE, {error}')
    if self.layout_error is not None:
        # Without this the run reports nothing to do, which is what a tidy
        # tree also reports, and the two are not the same answer.
        lines.append(f'LEGACY LAYOUT UNREADABLE, {self.layout_error}')
    if not lines:
        return 'no dataset declares a legacy location'
    done, waiting, bad = len(self.moved), len(self.ready), len(self.faults)
    closing = f'{done} moved, {waiting} ready to move, {bad} left in place'
    if self.redundant:
        # Named because it is the disk a user can reclaim by hand, and
        # because on a tree where everything has already been refetched it
        # is the only thing the run has to tell them.
        closing += (
            f'; {len(self.redundant)} dataset(s) still have an old copy on disk, '
            'which was left alone'
        )
    if self.manifest_errors:
        # A manifest that did not load may be the one declaring the dataset
        # this tree still holds, so the counts above are a floor and saying
        # otherwise would be the overstatement the report exists to avoid.
        closing += f'; {len(self.manifest_errors)} manifest(s) not read, so this may be partial'
    lines.append(closing)
    return '\n'.join(lines)

plan_relocations(data_root=None)

Report what a relocation would do, touching nothing.

Parameters:

Name Type Description Default
data_root str | Path | None

Override for the data root; defaults to the resolved FWL_DATA tree.

None

Returns:

Type Description
RelocationReport

One entry per dataset that declares a legacy location, whether or not that location exists on this machine.

Source code in src/fwl_io/relocate.py
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
def plan_relocations(data_root: str | Path | None = None) -> RelocationReport:
    """Report what a relocation would do, touching nothing.

    Parameters
    ----------
    data_root : str | Path | None
        Override for the data root; defaults to the resolved FWL_DATA tree.

    Returns
    -------
    RelocationReport
        One entry per dataset that declares a legacy location, whether or not
        that location exists on this machine.
    """
    from fwl_io.manifest import _discover

    root = resolve_data_root(data_root)
    locations, layout_error = _legacy_locations()
    entries: list[Relocation] = []
    seen: set[str] = set()
    providers, manifest_errors = _discover()
    for provider_datasets in providers.values():
        for ds in provider_datasets:
            legacy = locations.get(ds.key)
            if legacy is None or ds.key in seen:
                continue
            seen.add(ds.key)
            legacy_dir = root / legacy
            try:
                registry = ds.registry()
                target_dir = root / _version_dir(ds)
            except Exception as exc:  # noqa: BLE001 -- reported, never raised
                entries.append(
                    Relocation(ds.key, UNRESOLVABLE, legacy_dir=legacy_dir, detail=str(exc))
                )
                continue
            unmovable = _unmovable(ds, registry) if legacy_dir.is_dir() else None
            if unmovable is not None:
                entries.append(
                    Relocation(
                        ds.key,
                        UNRESOLVABLE,
                        legacy_dir=legacy_dir,
                        target_dir=target_dir,
                        detail=unmovable,
                    )
                )
                continue
            outside = _escaping(legacy_dir, target_dir, tuple(registry), root)
            if outside is not None:
                # A symlink is how this happens in a real tree: every joined
                # path looks clean and only resolving one shows it leaves.
                entries.append(
                    Relocation(
                        ds.key,
                        UNRESOLVABLE,
                        legacy_dir=legacy_dir,
                        target_dir=target_dir,
                        detail=f'{outside} resolves outside the data root {root}',
                    )
                )
                continue
            state, detail = _classify(legacy_dir, target_dir, registry)
            entries.append(
                Relocation(
                    ds.key,
                    state,
                    legacy_dir=legacy_dir,
                    target_dir=target_dir,
                    files=tuple(sorted(registry)),
                    detail=detail,
                    legacy_present=legacy_dir.is_dir(),
                )
            )
    return RelocationReport(tuple(entries), dict(manifest_errors), layout_error)

relocate_all(data_root=None, dry_run=False)

Move every legacy tree that checks out into the current layout.

A dataset is moved only when every file its registry declares is present in the legacy location and matches its recorded digest. Anything else is reported and left untouched, including a dataset already at its current location, which is the ordinary state once a fetch has happened there.

Parameters:

Name Type Description Default
data_root str | Path | None

Override for the data root; defaults to the resolved FWL_DATA tree.

None
dry_run bool

Report what would move without moving it.

False

Returns:

Type Description
RelocationReport

The plan, with each moved dataset's entry rewritten to say so.

Source code in src/fwl_io/relocate.py
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
def relocate_all(data_root: str | Path | None = None, dry_run: bool = False) -> RelocationReport:
    """Move every legacy tree that checks out into the current layout.

    A dataset is moved only when every file its registry declares is present
    in the legacy location and matches its recorded digest. Anything else is
    reported and left untouched, including a dataset already at its current
    location, which is the ordinary state once a fetch has happened there.

    Parameters
    ----------
    data_root : str | Path | None
        Override for the data root; defaults to the resolved FWL_DATA tree.
    dry_run : bool
        Report what would move without moving it.

    Returns
    -------
    RelocationReport
        The plan, with each moved dataset's entry rewritten to say so.
    """
    plan = plan_relocations(data_root)
    if dry_run:
        return plan
    root = resolve_data_root(data_root)
    done, halted = [], False
    for entry in plan.entries:
        if entry.state != READY or halted:
            done.append(entry)
            continue
        moved = _move_one(entry, root)
        done.append(moved)
        if moved.state in (FAILED, SPLIT):
            # Stop rather than move more data past a tree that is already in a
            # state somebody has to look at.
            log.error('stopping after %s could not be relocated', moved.key)
            halted = True
    return RelocationReport(tuple(done), dict(plan.manifest_errors), plan.layout_error)