HPC Runtime Doctor

Goal

Turn cluster symptoms into a resource-layout diagnosis, environment checklist, and safe retry plan.

Requirements

• Python 3.10+
• No external dependencies
• Works on Linux, macOS, and Windows

Inputs to Gather

Decision Guidance

• Check resource layout before changing physics settings.
• Confirm module/compiler/MPI/CUDA consistency before debugging solver behavior.
• Treat missing restart files and scratch cleanup as workflow failures, not physics failures.
• For GPU jobs, confirm the executable was built with the requested accelerator backend.

Script Outputs

scripts/hpc_runtime_doctor.py emits:

• resource_layout
• diagnoses
• environment_checks
• retry_plan
• scheduler_notes

Workflow

python3 skills/hpc-deployment/hpc-runtime-doctor/scripts/hpc_runtime_doctor.py \
  --scheduler slurm \
  --nodes 2 \
  --tasks 128 \
  --cpus-per-task 2 \
  --gpus 4 \
  --symptoms oom,slow-gpu \
  --uses-mpi \
  --uses-openmp \
  --uses-gpu \
  --json

Error Handling

Invalid resource counts stop with exit code 2. Unknown symptoms are preserved as custom items for human review.

Limitations

This skill does not query a live scheduler. It diagnoses from the submitted layout and symptoms.

Security

• Inputs are scalar CLI values and booleans only.
• The script does not execute scheduler commands or inspect environment variables.
• The skill uses Bash only to run its bundled script.

References

• See references/hpc_runtime_patterns.md for scheduler and runtime diagnosis patterns.

Version History

• 1.0.0: Initial HPC runtime diagnosis skill.