Troubleshooting¶

This section includes troubleshooting advice for common errors, organized by platform. If you encounter errors that you cannot solve via the standard step-by-step guide or the advice below, contact the developers.

Quick error index¶

General (all platforms)¶

Numerically fragile coupled runs¶

Some coupled atmosphere-interior-outgassing configurations occasionally fail with messages such as RuntimeError: Aragog retry ladder exhausted or with warnings about T_core jumping >1500 K despite Aragog reporting status=0. The same config may succeed on one launch and fail on the next.

The cause is sub-1e-7 floating-point noise in JAX/XLA reduction order that compounds through Aragog's tight tolerances and lands the solver on a wrong P-S branch within ~15 iterations.

Workaround: launch PROTEUS with --deterministic:

nohup proteus start -c <cfg.toml> --offline --deterministic \
    > output/<run>/launch.log 2>&1 & disown

The flag intercepts itself in sys.argv before any heavy imports, sets JAX_ENABLE_X64=1 and XLA_FLAGS=--xla_cpu_enable_fast_math=false, and self-re-execs once. PROTEUS already pins BLAS thread counts at import time; --deterministic adds the JAX/XLA layer that BLAS pinning alone does not cover.

Do not enable by default; the flag has a small per-step cost. Use only when a config shows noise-floor divergence between launches.

For broader convergence problems,see stabilising simulations.

Aragog stops at setup: CVODE cannot be imported¶

proteus start with interior_energetics.module = "aragog" stops before any work with ImportError: Aragog needs the SUNDIALS CVODE solver (solver_method = "cvode"), but scikits_odes_sundials.cvode cannot be imported. The default Aragog integrator is CVODE from the SUNDIALS library; the Aragog package alone would switch to scipy Radau with only a log warning, so PROTEUS refuses to start instead.

Install CVODE into the active conda environment from the PROTEUS root:

bash tools/get_cvode.sh
python -c "from scikits_odes_sundials.cvode import CVODE, CV_RootFunction, StatusEnum"

proteus install-all, proteus update-all and install.sh run the same script and only warn when it fails, so the failure first shows at the start of an Aragog run. To use scipy on purpose, set solver_method = "radau" or "bdf" in [interior_energetics.aragog]; the run then needs no CVODE, and its results are those of a different integrator.

Resuming a run written by an older PROTEUS¶

proteus start --resume stops with a message naming columns the run's runtime_helpfile.csv does not carry:

Helpfile 'output/<run>/runtime_helpfile.csv' was written before 3 column(s)
of the current output schema existed: R_xuv, T_xuv, g_xuv.
Run this configuration again from t=0, or read this run with the PROTEUS
version that wrote it.

PROTEUS records one column per output quantity, and a resumed run reads its starting state from the last line of that file. A run paused before a quantity was added holds no value for it, so the run stops.

Either run the configuration again from t = 0, or check out the PROTEUS version the run was launched with and read it under that version. There is deliberately no option to continue anyway. The absent columns cannot be filled with a placeholder, because several modules decide what to do by testing whether a quantity is present at all: CALLIOPE refuses a run whose oxygen budget is missing, the dummy and boundary interiors fall back to a configured core size, and the atmosphere lower boundary moves to the solvus only when a solvus radius exists. Supplying a placeholder satisfies each of those tests and passes the placeholder to the solver behind it, which produces a result that looks ordinary and is wrong.

proteus observe and proteus offchem are held to a smaller set of columns, because reading a stored run to synthesise an observation or run offline chemistry touches only a small part of the schema. An archived run stays postprocessable after a schema addition it never used, and stops only when a column those commands actually read is absent. When that happens, the practical remedy for a run that is expensive or no longer possible to reproduce is the second one: check out the PROTEUS version the run was launched with and postprocess it under that version. Re-running from t = 0 under current code does not reproduce the archived run, it produces a different one.

To keep a long run resumable across an upgrade, note the PROTEUS version it was launched with and stay on it until the run finishes.

Resume stops: no Zalmoxis structure file matches the row¶

proteus start --resume on a Zalmoxis + Aragog run stops with status 20 and a message such as:

Resume: no Zalmoxis structure file matches the helpfile row at t = 6.020140e+02 yr
within max(1 m, 1e-9 of the mantle thickness); zalmoxis_output.dat: R_core +8.513e+03 m,
R_int -5.486e+04 m; ...

A structure solve rewrites data/zalmoxis_output.dat at once, while the helpfile and the snapshots are written only on snapshot iterations, so a stopped run can hold a structure newer than the row it resumes from. PROTEUS tries three files in this order: the copy saved with that row (data/<time>_zalmoxis.dat), data/zalmoxis_output.dat, and its backup data/zalmoxis_output.dat.prev. The run resumes on the first one whose inner and outer radii match the row within the tolerance in the message. When none of them matches, or none of them exists, run the configuration again from t = 0.

A resumed run and the uninterrupted run¶

A Zalmoxis + Aragog run solves the structure after the interior step of an iteration, and Aragog takes the new mesh at the start of the next iteration. A run that stops between the two leaves an interior snapshot on the earlier structure and a helpfile row on the new one. On resume, PROTEUS builds Aragog on the earlier structure, restores the snapshot, and then makes the same structure update as a running loop. The first resumed step then records the compression work of that update (step_dE_compression_J) and carries the entropy to the new mesh by mass parcel, as the uninterrupted run does.

This needs a mesh file of the earlier structure: the copy saved with its row (data/<time>_zalmoxis.dat), data/zalmoxis_output.dat.prev, or the copy saved with an older row on the same structure. With a snapshot on every iteration (params.out.write_mod = 1), one of them is present after a normal stop. With sparser snapshots, it is present only when the run stopped before the next structure solve, and the landing row of a giant impact keeps no copy. A resume that continues without the update gives one of two warnings:

Resume: the interior snapshot at t = 3.220037e+02 yr is on an earlier structure (...) and no row with a mesh file of that structure is left.
Resume: a structure update was pending at the stop. With core_bc = 'core_module' the solver is built on the structure of the row and the update is not replayed from the earlier mesh

The first appears when no mesh file of the earlier structure is left. The second appears for a core boundary condition other than energy_balance, and when a giant impact landed between the snapshot and the row. In both cases, the run continues on the structure of its row: the first resumed step records no compression work, and the entropy is handed over by node. The resumed run then differs from the uninterrupted one from that step on. A structure update that moves both radii by less than 1 m is not detected and is handled in the same way, without a warning. A snapshot with no entropy profile on the node count of the run gives neither warning: the run starts from a fresh initial condition on its row, as on any resume with such a snapshot. If the solver is built on the earlier structure and the snapshot entropy does not fit its cells, the resume stops with an error, and the run must start again from t = 0.

Cannot clone module, or Permission denied (publickey)¶

Have you added your SSH key to GitHub? See these pages for guidance:

• Checking for existing SSH keys
• Generating a new SSH key
• Adding a new SSH key to your Github account
• Testing your SSH connection

Out-of-date modules detected¶

You will see an error that looks like this:

[ INFO  ] Validating module versions
[ ERROR ] (MODULENAME) module is out of date: (INSTALLED VER) < (REQUIRED VER)
Uncaught exception
    ...
OSError: Out-of-date modules detected. Refer to the Troubleshooting guide on the wiki

This means one of the required PROTEUS modules is outdated on your machine. You need to update the named module before running PROTEUS again.

You can check which modules are out of date by running:

proteus doctor

If you have not manually installed this module, go to the PROTEUS folder and run:

python -m pip install -e .

If you have a developer installation of the module, go to the module's own folder and run:

git checkout main
git pull
python -m pip install -U -e .

An install refuses to replace a module checkout¶

An install script stops rather than delete a module checkout that holds work you would not get back:

ERROR: /path/to/aragog has uncommitted changes or commits not on a remote.
       Refusing to delete it. Commit and push your work, or run
       bash tools/get_aragog.sh --force  to discard the checkout.

Commit and push the work, and the next run replaces the checkout normally.

install.sh and proteus install-all never pass --force, so they cannot step over this for you. To discard the checkout deliberately, run the script the message names by hand, from the PROTEUS folder:

bash tools/get_aragog.sh --force

A second message reports a checkout git cannot read at all, from a truncated copy or a clone interrupted after it started writing refs:

ERROR: git could not report the state of /path/to/aragog, so whether
       it holds local work is unknown. Refusing to delete it.
       Inspect the checkout, or run
       bash tools/get_aragog.sh --force  to discard it.

The same --force discards it. A clone killed before it committed anything is not affected: there is nothing to lose, so the next install refreshes it without asking.

Data download errors or slow Zenodo downloads¶

PROTEUS downloads input data through fwl-io from Zenodo, without an API token. A file that Zenodo does not serve is fetched from its DataverseNL mirror (every dataset except the PHOENIX spectra has one), and every file is checked against the committed registry. When neither source serves a file, the error lists the failure from each source: a timeout, a connection error or an HTTP 429 or 5xx clears if you retry later; a 404 or a checksum mismatch from both sources does not, so report it as an issue. fwl-io check proteus lists what is missing or corrupt among the datasets PROTEUS declares in its own manifest.

PETSc complains about being in the wrong directory¶

First, check that you are in the correct directory when running make or ./configure. If you are, this could be caused by the environment variable PETSC_DIR remaining set after a previous PETSc installation. Run unset PETSC_DIR and try again.

libudev.so.1 not found¶

This happens when compiling SPIDER within a Python environment that is incompatible with PETSc. Try compiling SPIDER (and/or PETSc) in your system's base Python environment, and ensure that you are not using Python >= 3.13, as SPIDER is only supported for Python versions <= 3.12 currently.

Julia compatibility error¶

There are incompatibilities between Python and some versions of Julia. Supported Julia versions are 1.11.x, 1.12.x and 1.13.x. Support for 1.11 is deprecated and will be dropped in a future release; install.sh and proteus doctor warn when they find it. Newer releases are untested and may fail when juliacall resolves shared libraries.

You must use Python 3.12 with a supported Julia to avoid these problems:

juliaup add 1.13
juliaup default 1.13

macOS¶

PETSc compilation fails on Apple Silicon¶

The get_petsc.sh script automatically detects Apple Silicon and applies the necessary workarounds. If it still fails, check the following.

Prerequisites

Make sure you have installed GCC and OpenMPI via Homebrew:

brew install gcc open-mpi

Also ensure Xcode Command Line Tools are installed:

xcode-select --install

Common issues

1. Deprecated linker warnings (macOS 13+/clang 15+): PETSc's configure uses -bind_at_load and -multiply_defined flags that are deprecated on modern macOS. The script suppresses these with -Wl,-w.

2. C++ template errors (macOS 26+/clang 17+): PETSc 3.19's CUDA/CUPM headers trigger -Wmissing-template-arg-list-after-template-kw. The script avoids this by disabling C++ (--with-cxx=0), which is safe because SPIDER is pure C.

3. SDKROOT not set: On macOS Catalina and later, system headers are inside the SDK, not /usr/include. The script sets SDKROOT automatically via xcrun --show-sdk-path.

PETSc configure: "Downloaded sundials2 could not be used"¶

PETSc configure on macOS stops with:

Downloaded sundials2 could not be used. Please check install in .../arch-darwin-c-opt

and configure.log reports _CVDense and _CVSpgmr as undefined symbols.

A SUNDIALS 7 from Homebrew is in the linker search path ahead of the SUNDIALS 2.5 that PETSc downloads, and SUNDIALS 7 has no CVDense. get_petsc.sh passes no Homebrew library path, so on Apple Silicon, where Homebrew lives in /opt/homebrew, PETSc builds with a Homebrew SUNDIALS installed. On an Intel Mac, Homebrew installs into /usr/local/lib, which the linker searches by default, so a Homebrew SUNDIALS can still hide the downloaded one there. brew list --versions sundials shows whether Homebrew's SUNDIALS is installed.

PETSc tests error (network configuration)¶

Error when running the PETSc tests:

Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(467)...............:
MPID_Init(177).....................: channel initialization failed
MPIDI_CH3_Init(70).................:
MPID_nem_init(319).................:
MPID_nem_tcp_init(171).............:
MPID_nem_tcp_get_business_card(418):
MPID_nem_tcp_init(377).............: gethostbyname failed, localhost (errno 3)

This is a network configuration issue. To fix it, add the following to /etc/hosts, where computername is your hostname:

127.0.0.1   computername.local
127.0.0.1   computername

Then enable Remote Login in your Sharing settings and add your user to the allowed access list.

Errors during PETSc configuration or compilation (x86 header)¶

One of the following errors during PETSc configuration or compilation:

This header is only meant to be used on x86 and x64 architecture
#error "This header is only meant to be used on x86 and x64 architecture

Install Python via Homebrew and use pip to manage packages, rather than Conda.

Errors during the SOCRATES compilation¶

One of the following errors happen during the SOCRATES compilation:

1. clang (LLVM option parsing): Unknown command line argument '-x86-pad-for-align=false'. Try: 'clang (LLVM option parsing) --help'
2. clang (LLVM option parsing): Did you mean '--x86-slh-loads=false'?

There is an issue with your compiler, either the standard Apple clang or gcc installed by brew. Follow the steps provided at the StackOverflow thread:

sudo rm -rf /Library/Developer/CommandLineTools
sudo xcode-select --install

FastChem compilation with Apple Silicon¶

With Apple Silicon hardware (M1/M2/M3), the option -march=native sometimes causes issues.

Make sure to use the GNU version of g++, not the Apple one. The Apple version located at /usr/bin/gcc is actually a wrapper around clang.

The Homebrew version located at /opt/homebrew/bin/ works well. Find out which gcc version Homebrew installed (ls /opt/homebrew/bin/gcc-*), and edit the file make.globaloptions in the FastChem directory to use, e.g. g++-12 or g++-13 instead of g++.

Python netCDF library error¶

Error: Library not loaded: '@rpath/libcrypto.3.dylib'

Create a symlink in the local Python installation. See this guide for details.

brew install openssl

Follow the instructions at the end of the openssl installation (replace USERNAME with your own system username):

echo 'export PATH="/usr/local/opt/openssl@3/bin:$PATH"' >> /Users/USERNAME/.bash_profile
echo 'export LDFLAGS="-L/usr/local/opt/openssl@3/lib"' >>/Users/USERNAME/.bash_profile
echo 'export CPPFLAGS="-I/usr/local/opt/openssl@3/include"' >>/Users/USERNAME/.bash_profile
ln -s /usr/local/opt/openssl/lib/libcrypto.3.dylib /Users/USERNAME/opt/anaconda3/envs/proteus/lib/python3.10/site-packages/netCDF4/../../../
ln -s /usr/local/opt/openssl/lib/libssl.3.dylib /Users/$USER/opt/anaconda3/envs/proteus/lib/python3.10/site-packages/netCDF4/../../../

ModuleNotFoundError: No module named 'yaml'¶

Install yaml via pip rather than via conda:

python -m pip install pyyaml

Missing ifort compilers¶

If the SOCRATES installation steps complain about missing ifort compilers:

Install Intel compilers from the Intel oneAPI Toolkit page. First install Intel oneAPI Base Toolkit, then Intel oneAPI HPC Toolkit. Follow the instructions provided after the installation to set the locations of ifort in your environment.

ModuleNotFoundError: No module named '_tkinter'¶

Install the tkinter package using brew:

brew install python-tk

TkAgg backend / X-forwarding error¶

Error: ImportError: Cannot load backend 'TkAgg' which requires the 'tk' interactive framework, as 'headless' is currently running

If you are connecting to a computer over SSH, make sure to enable X-forwarding. This can be done in your SSH configuration file or as a command line parameter. Install XQuartz on your Mac.

YAML library error¶

Error: ld: unsupported tapi file type '!tapi-tbd' in YAML file '/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk/usr/lib/libSystem.tbd' for architecture arm64

There is an issue with your ld, potentially caused by an existing installation of anaconda. Delete all traces of anaconda by following the steps at the Anaconda uninstall guide, then install Python via brew.

gfortran cannot find system libraries¶

Error: ld library not found for -lsystem

This usually occurs when your operating system has been updated, but Xcode is not on the latest version. Try updating or reinstalling Xcode.

Alternatively, try adding the following flag to the gfortran compile and link steps:

-L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/

Linux¶

Cannot compile PETSc on compute cluster¶

There are various reasons that PETSc might have problems compiling. On a cluster, this could be caused by a mismatch between your MPICH version and SLURM. This is a known problem on Snellius.

Try loading the OpenMPI module:

module load mpi/openmpi-x86_64

Cannot compile PETSc, error moving libraries (Fedora/RHEL)¶

If you are using Fedora or RedHat Linux distributions, you might encounter an error:

Error moving
  /home/$USER/PROTEUS/petsc/arch-linux-c-opt/externalpackages/f2cblaslapack-3.8.0.q2
  libraries

To resolve, follow these steps:

1. Install libraries:

   sudo dnf install lapack lapack-devel lapack-static sundials-mpich \
       openmpi openmpi-devel f2c f2c-libs
   

2. Enable module utility: source /etc/profile.d/modules.sh
3. Load OpenMPI module: module load mpi/openmpi-x86_64
4. Add the commands from steps 2 and 3 to your ~/.bashrc file.
5. Run the get_petsc.sh script again.

netcdff.so not found¶

This happens when SOCRATES cannot find the NetCDF-Fortran library. Make sure that it's installed. You can also try adding the library location to your LD_LIBRARY_PATH variable. Find the location of the NetCDF-Fortran library using:

nf-config --flibs