Testing suite

Tests verify that the code does what was written; physical correctness is judged by data, not by tests. The suite catches regressions in the energy-limited mass-loss formula, the tidal correction, the radius-scaling selection, and the MORS stellar-flux coupling, but it cannot certify that those formulae match nature. That judgement belongs to the validation runs and the published comparisons the model is built on.

This page describes the suite as a whole and how to run it.

Test quality contract

Five layers enforce test rigor across the suite:

1. A four-marker tier scheme (unit, smoke, integration, slow) selects what runs in the PR gate versus the nightly.
2. Two validation markers (physics_invariant, reference_pinned) tag tests that carry physical meaning beyond pure code coverage.
3. A source-to-test mirroring rule pairs the physics source with a same-named test file.
4. An AST linter (tools/check_test_quality.py) rejects weak-test patterns on every PR.
5. A coverage gate held at the 90 % ceiling, enforced on every PR; a helper (tools/update_coverage_threshold.py, run manually) raises fail_under one-way toward that ceiling and never lets it drop.

Layers 1, 4, and 5 are blocking on PRs. Layers 2 and 3 are advisory: the linter reports gaps but does not fail the build.

The four-marker tier scheme

Every test in the suite carries a tier marker, applied at module level (pytestmark = [pytest.mark.X, pytest.mark.timeout(...)]).

Tests without a tier marker are invisible to CI. The PR gate runs pytest -m "(unit or smoke) and not skip"; the nightly runs everything in (unit or smoke or integration or slow) and not skip.

Module-level marker and timeout

Every test file declares its tier and its wall-time ceiling at module top:

pytestmark = [pytest.mark.unit, pytest.mark.timeout(30)]

The timeout is a defensive ceiling, not a target. A unit test that takes 25 s of wall time has either picked the wrong tier or has a leak somewhere; the ceiling catches future regressions that introduce a hang. Per-function markers (for example @pytest.mark.parametrize) are additive and do not replace the module-level marker.

Physics-invariant tiering

A unit test on the physics source (escape.py) must assert at least one of four invariant families:

• Conservation: the mass-loss rate equals the deposited XUV power divided by the gravitational binding energy per unit mass, up to the geometric and efficiency factors of the formula; in a coupled step the removed mass never exceeds the available atmospheric mass.
• Positivity or boundedness: the escape rate is non-negative for valid inputs; the tidal factor K_tide lies in (0, 1) when the Hill radius exceeds the XUV radius (ksi = Rhill / Rxuv > 1), rising toward 1 as the orbit widens, and the tidal branch raises ValueError for ksi <= 1; radii, masses, and semi-major axis are strictly positive; the XUV flux and efficiency are non-negative.
• Monotonicity or symmetry: the rate is linear in the XUV flux at fixed geometry; it scales as 1 / Mp; it is larger with the tidal correction than without (K_tide < 1) for a close-in orbit; it is zero when the flux is zero.
• Pinned numeric value with a discrimination guard: a closed-form value pinned via pytest.approx, plus explicit assertions that a wrong exponent, a wrong scaling branch, a sign flip, or a unit slip would each differ from the correct value by more than the tolerance.

Tests that meet one or more of these are tagged @pytest.mark.physics_invariant. The marker is per-function, not module-level: an error-contract test (an unsupported scaling raising ValueError) does not carry it.

The physics source in ZEPHYRUS is escape.py. The utility sources (__init__.py, constants.py, planets_parameters.py) are exempt from the invariant requirement but remain subject to the anti-happy-path rules.

Reference-pinned validation

Tests that pin behaviour against an external anchor are tagged @pytest.mark.reference_pinned. The anchor is one of a published benchmark, an analytical limit, or a cross-implementation check. escape.py carries at least one such test, recorded on the escape validation page. The current anchors:

The marker is not the same thing as physical correctness: a reference-pinned test certifies that this implementation reproduces that anchor; it does not certify that the anchor is the right physics for every planetary regime.

Source-to-test mirroring

Each source module with executable content has a same-named companion in tests/:

Cross-cutting and companion tests are the documented exception, not the rule:

• tests/test_mors_coupling.py: the MORS-to-escape flux hand-off with the stellar lookup mocked, so the coupling recipe runs in the fast unit tier without a download.
• tests/test_earth.py: an Earth-analogue regression that spans the real MORS lookup and the escape formula end to end. It carries the integration tier because it downloads the stellar-evolution tracks.
• tests/test_escape_properties.py: the Hypothesis-driven property sweeps for escape.py, kept in their own module so the pytest.importorskip('hypothesis') skip applies only to these tests and the closed-form pins in tests/test_escape.py still run when the develop-extra dependency is absent.

__init__.py holds only the package version string and has no dedicated test file.

AST test-quality linter

tools/check_test_quality.py walks tests/test_*.py as an AST and enforces these rules:

The linter runs in two blocking modes:

• python tools/check_test_quality.py --baseline walks the suite and writes the per-rule violation counts to tools/test_quality_baseline.json. This is the floor. Regenerate it only after a deliberate reduction in violations.
• python tools/check_test_quality.py --check (CI mode) walks the suite, compares the current counts to the baseline, and exits non-zero if any rule's count exceeds the baseline.

The baseline ratchets one way. Override with ZEPHYRUS_TEST_QUALITY_ALLOW_REGRESS=1 only when a new rule was added that surfaces pre-existing violations.

Two advisory modes report gaps without failing CI:

• python tools/check_test_quality.py --reference-pinned-status: lists physics sources whose test file has no @pytest.mark.reference_pinned test.
• python tools/check_test_quality.py --physics-invariant-status: lists physics-source tests that assert no invariant and are not tagged @pytest.mark.physics_invariant.

Local commands

pytest -m unit                              # fast unit tests
pytest -m smoke                             # minimal-input dependency tests
pytest -m integration                       # real MORS-coupled escape
pytest -m slow                              # long-running validation (nightly tier)
pytest -m "(unit or smoke) and not skip"    # PR-gate selection
pytest -m "not skip"                        # everything that should ever run

The integration tier needs the stellar-evolution data downloaded once:

export FWL_DATA=/path/to/your/data
mors download all

Coverage:

pytest --cov=zephyrus --cov-report=term -m "not skip"
pytest --cov=zephyrus --cov-report=html -m "not skip"   # htmlcov/

Lint and structure:

bash tools/validate_test_structure.sh         # module-level marker validator
python tools/check_test_quality.py --check     # AST linter against baseline
python tools/check_test_quality.py --reference-pinned-status
python tools/check_test_quality.py --physics-invariant-status

Public-facing badges versus internal taxonomy

Public-facing badges (README, project website) collapse smoke + integration + slow into a single Integration Tests category, because a four-way taxonomy is confusing to non-developer readers. The four-marker internal scheme remains for CI granularity: the PR gate runs (unit or smoke), the nightly runs everything, and the test-count badges fetch the JSON files written into the documentation site during the docs deploy.

Badge system

The documentation deploy (.github/workflows/docs.yaml) regenerates three JSON files, tests-{total,unit,integration}.json, from pytest --collect-only and writes them into the published site under badges/. Shields.io fetches them live via the endpoint URL embedded in the test-count badge. The counts refresh on every documentation build, so they track the suite without running it.

Coverage gates

Two gates are declared in pyproject.toml:

Both gates sit at the 90 % ceiling. The tools/update_coverage_threshold.py helper (run manually) raises fail_under one-way toward the ceiling and never lets it drop; ECOSYSTEM_CEILING = 90.0 caps it, and neither gate may be manually decreased. ZEPHYRUS has no compiled dependencies, so the unit and smoke tier already covers the whole package, which is why both gates are already at the ceiling. The PR gate has a pre-flight step that fetches the base branch's pyproject.toml and rejects any PR that drops [tool.coverage.report].fail_under below min(base, 90.0).

PR validation pipeline

.github/workflows/tests.yaml runs on every push and pull request to main, and on manual workflow_dispatch. Draft PRs run only ubuntu-latest with Python 3.12; non-draft events run the full matrix (ubuntu-latest, macos-latest x Python 3.10, 3.11, 3.12). The step sequence:

1. Validate test structure (bash tools/validate_test_structure.sh): rejects any test file without a module-level pytestmark.
2. Run test-quality lint (python tools/check_test_quality.py --check): blocking; rejects regression against tools/test_quality_baseline.json.
3. Pre-flight fail_under ratchet check: rejects any PR that drops [tool.coverage.report].fail_under below min(base, 90.0).
4. Run unit + smoke tests: pytest -m "(unit or smoke) and not skip" --cov=zephyrus --cov-fail-under=${FAST_FAIL_UNDER}, where FAST_FAIL_UNDER is read live from [tool.zephyrus.coverage_fast].fail_under.

Steps 1 to 3 are gated to ubuntu-latest, python 3.12 to avoid redundant runs; step 4 runs across the full matrix.

Nightly (.github/workflows/nightly.yml) runs the full suite, uploads coverage to Codecov, and enforces --cov-fail-under=90.

Canonical specification

The repository-wide rules that every PROTEUS-ecosystem submodule follows are at proteus-framework.org/PROTEUS/Explanations/ecosystem_testing_standard/.