Every change to code within the PROTEUS ecosystem must pass test 'gates' before it can be merged into the main versions of our models. In addition, the modelled physics is cross-checked against independent codes and against peer-reviewed papers. The 'coverage' of these tests (i.e. the fraction of the code which is probed) is tracked publicly. Validating workflows ensure that the coverage is only allowed only to increase as development progresses. We now have thousands of tests across all of our open-source repositories.
How do we know the code is accurately modelling the real world? This can be approached as two questions. Firstly: does the code do what we wrote it to do, every time? Our test suite tracks the modelled behaviours for reproducibility, and this page describes how. Secondly: do codes and models we have written capture the real behaviour of planets? This is judged by comparison against empirical data, including JWST spectra, geochemical and geophysical measurements, as well as with independent codes from other research groups.
The test suite is a precondition for iterative model development, not the substitute for validation by humans. The tests guarantee that the code does what we intended reproducibly, so that any disagreement with data is a statement about the physics we chose, not a bug we failed to catch.
Continuous integration
Every change to PROTEUS or its modules runs through automated tests on GitHub Actions before it reaches the main branch. Two pipelines serve different purposes:| Pipeline | Trigger | What it does |
|---|---|---|
| Fast PR gate | Every pull request | Runs unit tests with mocked physics and smoke tests with real binaries on a small configuration. Roughly 5 minutes; gates merging. |
| Nightly full suite | Nightly (03:00 UTC) | Runs the complete suite including integration and slow tests, uploads coverage to Codecov, and ratchets the coverage threshold so it can only ever increase. |
Three categories of test, three things they catch
Unit tests
Isolate one function with mocked physics. Catch algebra errors, input-output regressions, and broken assumptions about types and units. They run in milliseconds, so the suite stays cheap to invoke on every commit.
Integration and smoke tests
Run the real models under a minimal-configuration scenario. Often, using a single simulation time step at spatial low resolution. These catch bugs that could arise due to model coupling, which specific unit tests cannot see. For example, when two correct modules disagree on a boundary condition, on a sign convention, or on which way time runs.
Physically-motivated validation
Cross-check against analytic limits, independent implementations (SPIDER versus Aragog for the magma-ocean energy budget), and published reference runs. This is the hardest category: pass and fail are not always binary, and a "passing" run that disagrees with a known limit is a stronger signal than a green check.
What counts as a test
Coverage measures how much of the code runs under test; it does not say whether a test would notice when the code is wrong. A test that passes for the wrong reason is worse than none, because it manufactures confidence. Every test in the suite is held to a standard a happy-path test cannot meet.
Required in every test
- A boundary or edge case: a temperature at the solidus, an empty inventory of volatiles, or an extreme physical parameter.
- The failure path. Where the code validates its input, the error must fire and leave no side effect behind; for closed-form physics, the limiting case must hold exactly.
- A value a plausible bug would break. A wrong sign, a wrong exponent, a missing factor, or a unit slip must make the test fail, so inputs are chosen where a wrong formula lands far from the right one.
- On physics modules, at least one physical invariant: conservation of mass, energy, or angular momentum; boundedness (temperatures positive, mass fractions within zero and one); monotonicity or symmetry; or a value pinned to a published benchmark or an independent code.
Rejected automatically
- Single-assertion tests, and weak checks such as "the result is not empty" or "the result is positive" left to stand alone.
- Floating-point equality against a literal value.
- Tests that used a dummy 'mocked' implementation of the very function they are meant to check.
- Tests which never explicitly state their targeted physical scenario or 'contract clause' (i.e. some obligation to reproduce a certain physical behaviour)
These rules are enforced mechanically: an automated check runs on every pull request and rejects new tests that break them. The strongest requirement, a value pinned to a refereed result or to an independent implementation, is documented under Tests and papers below.
Code coverage and test counts
The PROTEUS framework and its mature submodules expose passing-tests badges and, where wired, line-coverage badges from Codecov. Test counts marked with a live badge refresh automatically from each repository's CI. Each row links to the public repository behind it, and every live badge to the workflow or coverage report that generated it, so the figures can be checked at source.Mature suites
| Module | Tests on main | Breakdown | CI | Coverage |
|---|---|---|---|---|
| PROTEUS |  |   |  |  |
| Zalmoxis |  |   |  |  |
| AGNI |  |   |  |  |
| atmodeller |  |   |  |  |
| CALLIOPE |  |   |  |  |
| Aragog |  |   |  |  |
| MORS |  |   |  |  |
| ZEPHYRUS |  |   |  |  |
| SPIDER |  |   |  |  |
Suites in development
| Module | Tests on main | Breakdown | CI |
|---|---|---|---|
| JANUS |  | Total count; tests parametrized, not tier-marked |  |
| VULCAN |  |   |  |
| Obliqua | custom suite | Hand-written Julia validation runner, Test.jl migration planned |  |
| LovePy | none yet | Cross-validated against analytic Love number solutions in published applications | No CI yet |
Rows displaying a numeric badge are live. The count is fetched from each repository's automated workflows, and republished whenever the repository's tests change. Ecosystem-wide adoption of the PROTEUS marker convention (unit, smoke, integration, slow) is a point of active development.
Tests and papers
Testing extends far beyond software workflows. We also probe whether each code within the PROTEUS ecosystem matches real world observations, and whether independent codes (models) agree with one another in the expected regimes. These other testing modes are both on the public record. The papers highlighted below are part of the complete PROTEUS publications list.Tested against data: published applications
A simulation that reaches a refereed journal has been argued for in writing, exposed to independent review, and tied to physical predictions that observation can falsify. Two recent results close that loop.
Early Earth: the magma ocean lifetime
Coupled PROTEUS simulations of post-impact Earth resolve the interplay between tidal heating from the early Moon and greenhouse forcing from the outgassed atmosphere. The magma ocean lifetime ranges from roughly 30 to 500 Myr depending on mantle redox and volatile inventory, a prediction tied to specific geochemical observables.
van Dijk, Nicholls & Lichtenberg, Planetary Science Journal 7, 94 (2026). doi:10.3847/PSJ/ae5928
L 98-59 d: a JWST detection meets a model prediction
JWST detected sulphur dioxide in the atmosphere of L 98-59 d, the first sulphur detection on a planet of this size. PROTEUS explains it with a permanent, sulphur- and hydrogen-rich magma ocean whose outgassed SO2 is produced photochemically, ruling out both the standard "gas dwarf" and "water world" categories. The falsification loop closes: data constrains the model, the model returns a testable interior-composition claim, and the next observation can break it.
Nicholls, Lichtenberg, Chatterjee, Guimond, Postolec & Pierrehumbert, Nature Astronomy 10, 809 (2026). doi:10.1038/s41550-026-02815-8
Tested against independent codes: the CHILI intercomparison
Publication tests one code against data. A harder test is whether independent codes, built by different groups with different numerical choices, agree on the same problem. We initiated the Coupled atmospHere Interior modeL Intercomparison (CHILI), the first shared benchmark suite for coupled interior-atmosphere models, to make that test rigorous.
CHILI: a shared benchmark for coupled codes
CHILI defines a common protocol of standardized cases, early Earth, Venus, and rocky exoplanets around M-dwarfs, that any coupled interior-atmosphere code can run. Fixing the inputs turns model disagreement into a measured quantity: differences in energy transport, geochemistry, and volatile cycling become numbers to explain rather than buried assumptions. It is the community-scale counterpart of the cross-code checks PROTEUS runs internally, such as SPIDER against Aragog for the magma-ocean energy budget.
Lichtenberg et al., Planetary Science Journal 7, 108 (2026). doi:10.3847/PSJ/ae593b
CHILI I: primordial magma oceans of Earth and Venus
The model intercomparison results benchmark planetary evolution codes against the scenarios of Earth and Venus. Earth models agree on solidification timescale to within 4 Myr. Simulated Venus scenarios diverge more, between the participating models. The intercomparison highlights that divergence arise through variant treatments of volatile partitioning, vertical energy transport, mantle rheology, and melting. Locating those sensitivities is what separates a code that runs from one that agrees with independent implementations.
Nicholls et al. (2026). arXiv:2606.24757