Skip to content

Testing suite

Coverage Tests tests unit tests integration tests

The badge row above shows, left to right, the fraction of the code base exercised by the test suite, the status of the Tests workflow on main, and the live counts of total, unit, and integration tests. The coverage and status badges track the Tests GitHub Actions workflow; the three count badges refresh from the repository's badges branch whenever a push to main changes the tests, the src/vulcan package, pyproject.toml, or the badge tooling, and on manual dispatch, so these counts track the suite without manual updates.

Tests verify that the code keeps doing what was written. A passing suite confirms that a recent change has not perturbed locked behaviour; it is a regression net, not a proof of physical correctness, which is judged separately against data and published references.

Markers

VULCAN registers four pytest markers under [tool.pytest.ini_options] in pyproject.toml, with --strict-markers enforced:

  • unit: fast, isolated checks that finish in well under a second, such as logger file output and the FastChem install check.
  • smoke: quick validation with real binaries at one timestep and low resolution.
  • integration: a full default-config run that produces the expected outputs.
  • slow: long-running validation, launched manually.

Not every test carries a tier marker. The package-import checks and the log-level validation check run without one, so they count toward the total but toward neither sub-category; see Public categories.

Running tests by marker

pytest -m unit
pytest -m integration
pytest -m "not skip"   # full suite

pytest --collect-only -m <marker> reports the live count without running anything.

Public categories

The public badges collapse smoke + integration + slow into a single "integration tests" category, because a four-way split is confusing to non-developer readers. The "tests" badge counts every non-skipped test and is a strict superset of the two sub-categories: a test that carries no tier marker is counted in the total but in neither "unit tests" nor "integration tests", so the three numbers need not sum.

Badge system

The JSON the badges read lives at the root of the badges branch as tests-{total,unit,integration}.json, regenerated by the "Refresh test count badges" workflow when a push to main touches the tests, the package, pyproject.toml, or the badge tooling, or when the workflow is dispatched manually, and fetched live by shields.io. Publishing to a dedicated branch keeps the counts off main, which a branch-protection ruleset holds to reviewed pull requests.

Prerequisites

Install the development extra:

pip install -e ".[develop]"

Canonical specification

The central test-count dashboard for the ecosystem is at https://proteus-framework.org/testing. The full PROTEUS Ecosystem Testing Standard is documented at https://proteus-framework.org/PROTEUS/Explanations/ecosystem_testing_standard/.