hardware-counters

REPRODUCING.md

Reproducing the results

Every number and figure in the report comes from this directory. This file says which script produces what, and in what order. Run everything from dissertation/.

Environment

On ARCHER2, the analysis needs a virtualenv that inherits the Cray Python packages. Installing scikit-learn with pip install --user instead pulls in a newer NumPy that shadows the Cray build and breaks pandas, with an error message that points at the standard library rather than the real cause:

module load cray-python/3.10.10
python3 -m venv --system-site-packages venv
source venv/bin/activate
pip install -r requirements.txt

Do not run the analysis scripts from /tmp. A stray /tmp/inspect.py shadows the stdlib inspect module and produces the same confusing failure.

1. Collect the data

Building and instrumenting all eight codes:

bash scripts/build/build_all.sh

The sweeps are Slurm jobs. Each runs one configuration five times, once per counter set, re-exporting PAT_RT_PERFCTR between srun invocations:

sbatch scripts/slurm/run_expand.slurm    # the main campaign, 8 codes
sbatch scripts/slurm/run_of.slurm        # OpenFOAM (pat_run, separate)
bash   scripts/cirrus/drip.sh            # Cirrus, drip-feeds past the QoS limit

Raw CrayPat experiment directories are about 17 GB and are not committed. They live on /work and are regenerable from these scripts. The distilled text output is in results/reports/.

2. Parse into tables

python scripts/analysis/parse_expand.py    # -> data/runs_expanded.csv
python scripts/analysis/parse_cirrus.py    # -> data/cirrus_persets.csv
python scripts/analysis/parse_weak.py      # -> data/cirrus_weak.csv

These merge the five counter-set runs into one row per configuration. That merge matters: treating the five runs as independent rows gives 815 pseudo-rows at 44% populated instead of 191 configurations at 97%.

3. Analysis

Script	Produces	Report section
`retrain2.py`	main LOAO results, the 1.05 headline	5.1, 5.4
`baseline_ladder.py`	the five-rung measurement ladder	5.1
`ablation2.py`	feature-group ablation and significance	5.1
`learning_curve.py`	rows-vs-applications learning curves	5.5
`noncircular.py`	projection with no target measurement	5.8
`matched_crossplatform.py`	the paired 166-configuration analysis	5.7
`weak_analysis.py`	rescaled Cirrus campaign	5.7
`importance_artefact.py`	feature importance under counter multiplexing	—
`stats_rigour.py`	significance testing for every comparison	5.3
`reconcile_rf.py`	how many digits a random forest median is worth	5.4

Results land in results/model/.

4. Figures

python scripts/analysis/make_simple_figs.py    # fig_ladder, fig_perapp
python scripts/analysis/make_clear_figs.py     # fig_importance_artefact, fig_weakscaling
python scripts/analysis/learning_curve.py      # fig_learning_curve

The remaining four figures in results/figures/ (fig_scaling_behaviour, fig_eta_vs_runtime, fig_pred_vs_actual_xplat, fig_ablation, fig_crossplatform) were produced by earlier versions of make_plots.py that have since been rewritten. The PNGs are committed and the underlying data is in data/ and results/model/, but those exact plotting calls are not recoverable from the current scripts. This is a genuine gap.

results/figures/captions.md gives each figure a full caption.

Scripts not in the table

scripts/analysis/ holds about 45 files. The ones above are the ones the report depends on. The rest are diagnostics and earlier versions (explore.py, holes.py, probe_sets.py, size_diag.py, xplat_diag.py, the train_nn*.py series), kept for reference; their outputs are in results/superseded/. Where a script has a numbered successor (retrain.py -> retrain2.py, ablation.py -> ablation2.py), the higher number is the current one.

← What hardware counters can and cannot tell us