reconcile_rf.txt
P4: why are there four different numbers for the SAME Random Forest?
============================================================================================
Values published for 'Random Forest, LOAO, ARCHER2, 191 merged
configurations, median error factor':
1.055 dissertation/README.md, docs/modelling.md (three places)
1.054 analysis/out/retrain2_summary.txt wins 125/191
1.0548 analysis/out/advanced_nn_summary.txt wins 124/191, p = 0.0002445
1.0554 analysis/out/counter_value_summary.txt p = 0.000247
--- step 1: are the inputs identical? ---
runs_expanded.csv md5 befa684050fc560801a2a56ff352c273
merged configurations 191
features 21
applications (folds) 8 ['comd', 'gromacs', 'hpcg', 'hpl', 'lulesh', 'minife', 'openfoam', 'stream']
advanced_nn.py and ablation2.py both do
from retrain2 import load_merged, build_features, fac
so the CSV, the merge, the 21 feature VALUES and the 8 LOAO folds
are byte-identical. The data are not the explanation.
--- step 2: is the estimator identical? ---
All three:
RandomForestRegressor(n_estimators=400, min_samples_leaf=2,
random_state=0, n_jobs=-1)
Identical, including the seed. The estimator is not the explanation.
--- step 3: is the forest deterministic at a fixed seed? ---
three identical repeats : 1.055414 1.055414 1.055414 spread 0.0e+00
varying n_jobs : n_jobs=1:1.055414 n_jobs=2:1.055414 n_jobs=8:1.055414 n_jobs=-1:1.055414
The forest is bit-reproducible and independent of thread count.
Thread non-determinism is NOT the explanation.
--- step 4: the two things that DO differ ---
C1 FEATURE COLUMN ORDER. retrain2.py and advanced_nn.py use
build_features(df) directly, so columns are in definition order.
ablation2.py rebuilds it as Xall[cols] with cols assembled from
its GROUPS dict in cumulative measurement-cost order:
definition order ...['arith_intensity', 'log_instr_per_rank', 'log_t_analytic']
ablation2 order ['log_ncore', 'log_nrank', 'log_nthread', 'log_t_analytic', 'ipc', 'log_instr_per_rank', 'flops_per_instr']...
Same 21 columns and same values, positions 4-21 permuted.
max_features='sqrt' draws candidate splits from a PRNG stream
indexed by column POSITION, so a permutation yields different
trees even at random_state=0.
C2 PREPROCESSING CHAIN, re-implemented in each script rather than
imported:
retrain2.py impute(median) -> sign(x)*log1p(|x|) -> StandardScaler
advanced_nn.py impute(median) -> RobustSmoothScaler
(x-med)/(IQR/1.349) then 3*tanh(z/3)
ablation2.py impute(median) -> StandardScaler (no log step)
A forest is invariant to strictly increasing per-feature maps in
exact arithmetic. In float64 it is not: 3*tanh(z/3) saturates,
squashing |z| > ~12 into a band narrower than 1e-5 around +/-3,
so values that were distinct collapse onto the same float and the
split that separated them vanishes.
--- step 5: the full 2x4 grid, every combination reproduced ---
scikit-learn in this interpreter: 1.9.0
(this interpreter; the ARCHER2 1.7.2 values that match the published
summaries exactly are tabulated immediately after)
column order preprocessing chain median p90 wins p vs const
-------------------------------------------------------------------------------------------------------
definition order slog + StandardScaler 1.055414 1.3878 124 0.0002085
definition order RobustSmoothScaler 1.055531 1.3849 123 0.000247
definition order StandardScaler only 1.055823 1.3924 124 0.0002063
definition order impute only, no scaler 1.052791 1.3685 129 0.0004104
ablation2 group order slog + StandardScaler 1.054660 1.3932 126 0.0002195
ablation2 group order RobustSmoothScaler 1.054647 1.3944 125 0.0002264
ablation2 group order StandardScaler only 1.054694 1.3930 125 0.0002139
ablation2 group order impute only, no scaler 1.053308 1.3682 127 0.0003925
THE PUBLISHED SUMMARIES WERE GENERATED ON ARCHER2, scikit-learn 1.7.2.
The same grid run there gives an EXACT match on median, win count and
p-value for all three summary files:
column order chain median p90 wins p vs const
---------------------------------------------------------------------------------
definition order slog + StandardScaler 1.054497 1.3901 125 0.0002162
definition order RobustSmoothScaler 1.054812 1.3887 124 0.0002445
definition order StandardScaler only 1.054497 1.3898 124 0.0002096
definition order impute only, no scaler 1.054497 1.3898 124 0.0002117
ablation2 group order slog + StandardScaler 1.055408 1.3910 125 0.0002495
ablation2 group order RobustSmoothScaler 1.054913 1.3917 125 0.0002432
ablation2 group order StandardScaler only 1.055408 1.3910 125 0.0002470
ablation2 group order impute only, no scaler 1.055408 1.3910 125 0.0002508
Matching against the published figures:
retrain2_summary.txt 1.054 125/191 p=0.0002162
== definition order + slog+StandardScaler 1.054497 125 0.0002162 EXACT
advanced_nn_summary.txt 1.0548 124/191 p=0.0002445
== definition order + RobustSmoothScaler 1.054812 124 0.0002445 EXACT
counter_value_summary.txt 1.0554 p=0.000247
== ablation2 order + StandardScaler only 1.055408 125 0.0002470 EXACT
README/modelling.md 1.055
== 1.055408 rounded to three decimals, i.e. the ablation2 variant
All four published figures are accounted for exactly, with no residual.
Reading the grid, on scikit-learn 1.7.2 the three scalers give the
IDENTICAL forest within a column order, except RobustSmoothScaler,
which differs because of the tanh saturation described in C2. So:
C1 (column order) explains 1.054497 -> 1.055408, i.e. the
retrain2/README gap. It is the dominant term.
C2 (tanh chain) explains 1.054497 -> 1.054812, i.e. the
advanced_nn value.
--- step 6: how large is the noise floor? ---
random_state=0 is an arbitrary choice. Sweeping it on the canonical
chain gives the band the four published figures sit inside:
seeds 0..9: 1.0554 1.0592 1.0643 1.0553 1.0497 1.0655 1.0531 1.0540 1.0580 1.0580
mean 1.0572 sd 0.0049 range [1.0497, 1.0655] spread 0.0158
Implementation spread across the whole 2x4 grid : 0.0030
Seed spread across 10 seeds, one implementation : 0.0158
The seed spread is the larger of the two. Every one of the four
published figures lies inside the seed band, so the differences
between them are not evidence of anything.
scikit-learn version is a third source: the same code gives 1.0545 on
ARCHER2 (1.7.2) and 1.0554 on Cirrus (1.9.0), because the tree
builder's tie-breaking changed between releases.
============================================================================================
VERDICT AND CANONICAL VALUE
There is no discrepancy of substance. One experiment was run through
three copy-pasted preprocessing chains and two feature orderings on
two scikit-learn versions, and reported to four decimal places, three
of which are noise. No conclusion changes: every variant beats the
constant baseline with wins in 123-126 of 191 and raw p ~ 2e-4.
CANONICAL: Random Forest, LOAO, ARCHER2, 191 merged configurations,
full 21-feature set, canonical pipeline = retrain2.py's (median
imputation, sign-log, StandardScaler) applied to build_features
output in definition order:
median error factor 1.05
seed-averaged point estimate 1.0572 (10 seeds, sd 0.0049)
bootstrap 95% CI on the median: see stats_corrected.txt
Report it as 1.05, with the CI. Do not quote 1.0548 or 1.0554: those
digits are seed, column order and library version.
REMEDIAL ACTION for the repository (not applied here, repo is owned
by another agent):
1. ablation2.py and advanced_nn.py should import the pipeline
constructor from retrain2.py instead of re-implementing it.
2. ablation2.py should index Xall with a column list built in
build_features order, or better, mask features rather than
reorder them.
3. Every reported forest median should be seed-averaged over at
least 10 seeds and quoted to two decimals with an interval.
4. Pin the scikit-learn version in the environment file.