importance_artefact.txt
Retraction analysis: the core_energy_frac importance claim
==============================================================================
Claim under test: `core_energy_frac` (PP0/PACKAGE) is the strongest
predictor of the efficiency factor eta, at 0.509 RF importance, and is
the project's clearest novelty claim.
--- 0. Provenance of the 0.509 figure ---
results/model/feature_importance.csv was written by train_model.py from
data/runs.csv: 40 rows, 17 features, 5 applications, one problem size each.
Reproduced here: core_energy_frac = 0.526 (file records 0.509), rank 1.
It was NOT computed on the 815-row per-counter-set table. That matters,
because it means three defects contributed, not one.
--- A. Defect one: no analytic time term ---
eta is dominated by run length: short runs carry a fixed MPI_Init, I/O
and instrumentation overhead. train_model.py gave the forest no term
for run length, so the split had to be made on a correlate of it.
same 40 rows, 17 features core_energy_frac 0.526 rank 1
+ log_t_analytic, log_instr_per_rank core_energy_frac 0.148 rank 2
the added term takes the top slot: log_t_analytic 0.582
adding two columns and changing nothing else removes 72% of the claimed importance.
--- B. Defect two: one problem size per application, n = 40 ---
In runs.csv every application appears at exactly one problem size, so
application identity, problem size and counter signature cannot be
separated. The association does not survive when sizes vary.
corr(core_energy_frac, log10 eta), 40 rows, one size per app : +0.440
corr, SAME five applications, merged, several sizes each : +0.088
corr, all eight applications, merged : +0.217
between-application variance share of core_energy_frac: 0.499 on the 40 rows, 0.295 merged
between-application variance share of the target itself: 0.290 (40 rows)
With five applications, one size each and eight rows apiece, a feature
that half tracks application identity can split the target almost as
well as a label would, and impurity importance rewards exactly that.
merged, same five applications, same 17 features: core_energy_frac 0.007, rank 15 (n = 117)
merged, subsampled to n=40 (30 draws), 17 features: core_energy_frac 0.028, rank 10
Small n alone inflates it a little; small n plus a single problem size
per application is what inflates it to the top of the table.
--- C. Defect three: structural missingness on the 815-row table ---
per-counter-set table : 815 rows, 44.1% of feature cells observed
merged configurations : 191 rows, 97.4% of feature cells observed
feature obs815 imp815 rk obs191 imp191 rk
log_t_analytic 100.0% 0.553 1 100.0% 0.375 1
energy_per_instr 42.7% 0.075 2 100.0% 0.031 8
core_energy_frac 75.0% 0.069 3 89.5% 0.007 15
flops_per_instr 24.7% 0.052 4 97.4% 0.132 2
log_ncore 100.0% 0.047 5 100.0% 0.004 19
l3_lat_per_miss 74.7% 0.028 6 90.6% 0.010 14
flops_per_cycle 24.7% 0.028 7 97.4% 0.030 9
arith_intensity 18.0% 0.025 8 88.0% 0.004 20
log_instr_per_rank 50.2% 0.024 9 100.0% 0.109 3
log_nthread 100.0% 0.019 10 100.0% 0.005 18
l3_miss_per_instr 40.7% 0.018 11 95.8% 0.045 6
log_nrank 100.0% 0.016 12 100.0% 0.001 21
stall_fp_frac 24.8% 0.013 13 97.9% 0.089 4
ipc 50.2% 0.012 14 100.0% 0.012 13
tlb_miss_per_instr 25.3% 0.009 15 100.0% 0.041 7
stall_load_frac 24.8% 0.005 16 97.9% 0.006 16
l2_hit_rate 25.0% 0.003 17 99.0% 0.015 11
stall_store_frac 24.8% 0.003 18 97.9% 0.006 17
l2_miss_per_instr 0.0% 0.000 20 99.0% 0.049 5
l1_access_per_instr 0.0% 0.000 20 99.0% 0.014 12
prefetch_l2_frac 0.0% 0.000 20 96.9% 0.016 10
If impurity importance measured signal it would be unrelated to how
often a feature happens to be recorded. It is not:
815 pseudo-rows Spearman rho = +0.567 (p = 0.00739), Pearson r = +0.442 (p = 0.045)
191 merged configs Spearman rho = +0.080 (p = 0.731), Pearson r = +0.230 (p = 0.315)
The association is present on the structurally-missing table and
absent on the merged one, which is the signature of the artefact.
fully observed on the 815-row table (4): log_ncore, log_nrank, log_nthread, log_t_analytic
summed importance there 0.635 (63% of the total, from 4/21 features)
the same features on the merged table 0.385 (38%)
never observed there (3): l2_miss_per_instr, l1_access_per_instr, prefetch_l2_frac
importance there 0.000, on the merged table 0.078, so real signal was being suppressed as well as
misattributed.
Control: re-impose the 815-row missingness on the merged data. Same
191 configurations, same signal, only the observation pattern copied
across, averaged over 20 random masks.
masked-merged core_energy_frac 0.018, rank 7 (unmasked 0.007, rank 15)
masked-merged importance vs 815-row observation frequency: Spearman rho = +0.798 (p = 1.48e-05)
masked-merged ranking vs the 815-row ranking: rho = +0.742
unmasked-merged ranking vs the 815-row ranking: rho = +0.109
The mask alone recreates most of the old ranking from data known to
carry the corrected signal, so the pattern, not the physics, produced it.
--- The corrected result ---
core_energy_frac, 40-row runs.csv (retracted) : 0.526, rank 1 of 17
core_energy_frac, 815 per-set rows : 0.069, rank 3 of 21
core_energy_frac, 191 merged configurations : 0.007, rank 15 of 21
corrected top five (merged, impurity):
log_t_analytic 0.375
flops_per_instr 0.132
log_instr_per_rank 0.109
stall_fp_frac 0.089
l2_miss_per_instr 0.049
--- Robustness: permutation importance on held-out folds ---
Impurity importance is measured on training data and is biased towards
high-cardinality features. Permutation importance is measured on held-
out folds grouped by application, matching the LOAO protocol, so it
cannot reward a feature the model merely memorised.
feature perm rk impurity rk
log_t_analytic 0.0150 1 0.3747 1
tlb_miss_per_instr 0.0059 2 0.0407 7
flops_per_instr 0.0029 3 0.1324 2
l2_miss_per_instr 0.0021 4 0.0487 5
ipc 0.0015 5 0.0118 13
l2_hit_rate 0.0009 6 0.0147 11
l1_access_per_instr 0.0007 7 0.0139 12
energy_per_instr 0.0006 8 0.0306 8
l3_miss_per_instr 0.0005 9 0.0453 6
log_nthread 0.0003 10 0.0049 18
log_instr_per_rank 0.0003 11 0.1086 3
l3_lat_per_miss 0.0001 12 0.0104 14
core_energy_frac 0.0000 13 0.0075 15
log_nrank 0.0000 14 0.0010 21
stall_load_frac 0.0000 15 0.0065 16
arith_intensity -0.0000 16 0.0036 20
log_ncore -0.0000 17 0.0043 19
stall_store_frac -0.0000 18 0.0058 17
stall_fp_frac -0.0001 19 0.0886 4
prefetch_l2_frac -0.0004 20 0.0156 10
flops_per_cycle -0.0009 21 0.0305 9
permutation vs impurity ranking agreement: Spearman rho = +0.464 (p = 0.0343)
The two measures agree on direction but not in detail, which is the
expected outcome and a reason not to over-read any single ranking.
Both place core_energy_frac in the bottom half: permutation 0.0000, rank 13 of 21.
permutation top four: log_t_analytic, tlb_miss_per_instr, flops_per_instr, l2_miss_per_instr
impurity top four : log_t_analytic, flops_per_instr, log_instr_per_rank, stall_fp_frac
Both agree that log_t_analytic leads and that the remaining signal is
in instruction mix and cache behaviour, not in energy. The features
they disagree about (stall_fp_frac, log_instr_per_rank) are the ones
where impurity importance is known to be optimistic.
--- Conclusion ---
The claim is retracted. `core_energy_frac` is not the strongest
predictor of eta; on the corrected 191-configuration table it is worth
0.007 (rank 15 of 21) by impurity and rank 13 by held-out permutation
importance. The leading terms are the analytic time scale, the
instruction mix and cache behaviour.
The 0.509 was produced by three compounding defects, none of them about
energy: a missing run-length covariate (A), a design in which each
application appeared at a single problem size at n = 40 (B), and, on the
later per-counter-set table, structural missingness that made the energy
pair the only fully observed feature (C). Each is demonstrated above by
changing one thing at a time.
What replaces the claim is a methodological result, and it generalises
beyond this study. Impurity importance is not interpretable on a feature
matrix assembled by counter multiplexing. Rotating counter sets is the
standard way to widen a feature space past a small hardware register
budget, and it produces structurally missing columns by construction;
imputation then makes those columns uninformative, and the importance
mass has to go somewhere. Any counter that sits on a separate register
and is therefore recorded in every set, RAPL energy being the usual
case, will absorb it and look like a discovery. The safeguards are to
merge to one row per configuration before interpreting importance, to
include an explicit scale covariate so run length is not attributed to a
correlate, to vary problem size within application so importance cannot
proxy application identity, and to confirm with held-out permutation
importance rather than training-set impurity.