Testing suite

Tests verify that the code does what was written; physical correctness is judged by data, not by tests. The suite catches regressions in the saturation-vapour-pressure relations, the Planck emission, the hydrostatic height integration, the heat-capacity fits, the moist and dry adiabats, and the coupled radiative-convective solve, but it cannot certify that those formulae match nature. That judgement belongs to the validation runs and the published comparisons against measured atmospheric structure.

This page describes the suite as a whole. Contributors writing or modifying tests should read it together with the Run and build tests how-to.

Test quality contract

Five layers enforce test rigor across the suite:

1. A four-marker tier scheme (unit, smoke, integration, slow) selects what runs in the PR gate versus the nightly.
2. Two validation markers (physics_invariant, reference_pinned) tag tests that carry physical meaning beyond pure code coverage.
3. A per-source mirroring rule pairs every physics source with a same-named test file under tests/<subdir>/.
4. An AST linter (tools/check_test_quality.py) rejects weak-test patterns on every PR.
5. A coverage ratchet capped at 90 % keeps the gate moving upward over time.

Layers 1, 4, and 5 are blocking on PRs. Layers 2 and 3 are advisory: the linter reports gaps but does not fail the build.

The four-marker tier scheme

Every test in the suite carries exactly one tier marker, applied at module level (pytestmark = [pytest.mark.X, pytest.mark.timeout(N)]).

Live counts are shown by the tests, unit tests, and integration tests badges at the top of this page and by pytest -m <tier> --collect-only -q locally.

Tests without a tier marker are invisible to CI. The PR gate runs pytest -m "(unit or smoke) and not skip"; the nightly runs everything in (unit or smoke or integration or slow) and not skip.

Module-level marker and timeout

Every test file declares its tier and its wall-time ceiling at module top:

pytestmark = [pytest.mark.unit, pytest.mark.timeout(30)]

The timeout is a defensive ceiling, not a target. A unit test that takes 25 s of wall time has either picked the wrong tier or has a leak somewhere; the ceiling catches future regressions that introduce a hang. Per-function markers are additive and do not replace the module-level marker.

Physics-invariant tiering

A unit test on any physics source must assert at least one of four invariant families:

• Conservation or balance: hydrostatic balance along the column; band-integrated surface flux equal to the grey Stefan-Boltzmann limit (1 - albedo) sigma ts**4; column array-length and ordering integrity across an adjustment.
• Positivity or boundedness: T > 0, P > 0, saturation vapour pressures non-negative and finite, Planck emission non-negative, cpv strictly positive, mole fractions in [0, 1].
• Monotonicity or symmetry: saturation vapour pressure increasing with T; gravity decreasing with radius; dry-adiabat temperature increasing with pressure; integrated height increasing upward; Planck emission increasing with T at fixed frequency.
• Pinned numeric value with a discrimination guard: a closed-form value or published table entry pinned via pytest.approx, plus an explicit follow-up assertion showing that the most plausible wrong formula would differ from the correct one by more than the tolerance.

Tests that meet one or more of these are tagged @pytest.mark.physics_invariant. The marker is per-function, not module-level: structural tests in the same file (an ordering or pass-through-assignment check) should not carry it.

The six physics sources are:

Utility sources (__init__.py, set_socrates_env.py, plotting-only modules under utils/ and modules/) are exempt from the invariant requirement but remain subject to the anti-happy-path rules below.

Reference-pinned validation

Tests that pin behaviour against an external anchor are tagged @pytest.mark.reference_pinned. The anchor is one of:

• a published benchmark (cite paper, table, or figure),
• an analytical limit (for example, the Stefan-Boltzmann black-body limit or the Poisson potential-temperature identity),
• a cross-implementation check (a second, independent code path evaluating the same quantity).

Each of the six physics sources carries at least one reference-pinned test, recorded on the matching docs/Validation/<file>.md page. The current anchors:

The marker is not the same thing as physical correctness: a reference-pinned test certifies that this implementation reproduces that anchor; it does not certify that the anchor is the right physics for every atmospheric regime.

Per-source-to-test mirroring

New physics tests mirror source at tests/<subdir>/test_<file>.py:

Cross-cutting and pipeline tests are the documented exception, not the rule:

• tests/test_constants.py, tests/test_code.py: physical constants and utility / CLI code that span many sources.
• tests/test_runaway_greenhouse.py, tests/test_instellation.py: the full RadConvEqm / MCPA_CBL pipeline, which composes the adiabat, adjustment, and radiative steps together and runs the real SOCRATES binary (integration tier).

When a new physics source is added, its per-source test file is created at the same time; the matching docs/Validation/<file>.md page is added when the first reference-pinned test for that source lands.

AST test-quality linter

tools/check_test_quality.py walks the test files as an AST and enforces the anti-happy-path rules:

The linter runs in two modes:

• python tools/check_test_quality.py --baseline walks the suite and writes the per-rule violation counts to tools/test_quality_baseline.json. This is the floor. Regenerate the baseline only after a deliberate sweep that reduced violations.
• python tools/check_test_quality.py --check (CI mode) walks the suite, compares the current counts to the baseline, and exits non-zero if any rule's count exceeds the baseline. The PR workflow runs this and blocks on regression.

An advisory mode reports gaps without failing CI:

• python tools/check_test_quality.py --reference-pinned-status: lists physics sources whose matching test file has no @pytest.mark.reference_pinned test.

Local commands

pytest -m unit                              # fast unit tests
pytest -m smoke                             # minimal-config SOCRATES tests
pytest -m integration                       # full pipeline coupling
pytest -m slow                              # sweeps and full validation
pytest -m "(unit or smoke) and not skip"    # PR-gate selection
pytest -m "not skip"                        # everything that should ever run

Coverage:

pytest --cov=janus --cov-report=term -m "not skip"
pytest --cov=janus --cov-report=html -m "not skip"   # htmlcov/

Lint and structure:

bash tools/validate_test_structure.sh         # module-level marker validator
python tools/check_test_quality.py --check     # AST linter against baseline
python tools/check_test_quality.py --reference-pinned-status

Public-facing badges versus internal taxonomy

The three test-count badges collapse the four-marker scheme into the two categories a non-developer reader cares about: the mocked unit tests that run on every pull request, and the integration tests (smoke, integration, and slow counted together) that run the real SOCRATES pipeline nightly. The tests badge shows the combined total. The four-marker scheme remains underneath for CI granularity: the PR gate runs (unit or smoke) and the nightly runs everything.

Badge system

The three badge JSON files (tests-{total,unit,integration}.json) live at the root of the repository's badges branch in the shields.io endpoint schema. The Refresh test count badges workflow (.github/workflows/publish-test-badges.yml) regenerates them from pytest --collect-only whenever the test suite or source changes on main, then commits them to the badges branch; shields.io fetches them live by raw URL. Publishing to a dedicated branch keeps the generated files off main, where direct pushes need review. The same files back the module's row on the central PROTEUS validation page.

Coverage gates

Two gates are declared in pyproject.toml:

Both gates ratchet toward 90 %, capped at 90 % (tools/update_coverage_threshold.py enforces ECOSYSTEM_CEILING = 90.0); neither may be manually decreased. The PR gate has a pre-flight step that fetches the base branch's pyproject.toml and rejects any PR that drops [tool.coverage.report].fail_under below min(base, 90.0). A one-time ratchet down to the 90 % ceiling is allowed; any drop below 90 % is blocked.

PR validation pipeline

.github/workflows/tests.yaml runs on every PR over ubuntu-latest and Python 3.12. The step sequence:

1. Build SOCRATES: checkout and compile the Fortran radiative-transfer dependency (cached across runs).
2. Build JANUS: pip install -e .[develop].
3. Validate test markers (bash tools/validate_test_structure.sh): rejects any test file without a module-level pytestmark.
4. Run test-quality lint (python tools/check_test_quality.py --check): blocking; rejects regression against tools/test_quality_baseline.json.
5. Pre-flight fail_under ratchet check: rejects any PR that drops [tool.coverage.report].fail_under below min(base, 90.0).
6. Run unit + smoke tests: pytest -m "(unit or smoke) and not skip" --cov=janus --cov-fail-under=${FAST_FAIL_UNDER}, where FAST_FAIL_UNDER is read live from [tool.janus.coverage_fast].fail_under.

Nightly (.github/workflows/nightly.yml) runs the full suite, uploads coverage to Codecov, and enforces the full-gate fail_under.

Canonical specification

The repository-wide rules that every PROTEUS-ecosystem submodule follows are at proteus-framework.org/PROTEUS/Explanations/ecosystem_testing_standard/.