learning_curve.txt
Learning curve: does 191 configurations saturate?
============================================================================================
dataset ARCHER2, 191 merged configurations, 21 features, 8 applications
outer loop leave-one-application-out (unchanged)
inner training set subsampled to 25/50/75/100%; TEST FOLD NEVER SUBSAMPLED,
so every point is scored on the same 191 configurations
repeats 20 random draws per point, summarised as median
with a percentile bootstrap 95% CI
schemes 'random' subsamples training ROWS (density);
'by-application' subsamples training APPLICATIONS (coverage)
constant baseline (full training data, does not vary with size):
median 1.0654 95% CI [1.0523, 1.0812]
--- scheme: random ---
model frac mean n_train median err 95% CI IQR over reps
--------------------------------------------------------------------------
rf 0.25 41.9 1.0871 [1.0817,1.1011] [1.0798,1.1020]
rf 0.50 83.9 1.0709 [1.0648,1.0780] [1.0632,1.0788]
rf 0.75 125.2 1.0622 [1.0579,1.0678] [1.0577,1.0696]
rf 1.00 167.1 1.0585 [1.0568,1.0601] [1.0563,1.0602]
gbq 0.25 41.9 1.0915 [1.0822,1.1073] [1.0808,1.1092]
gbq 0.50 83.9 1.0735 [1.0647,1.0793] [1.0632,1.0804]
gbq 0.75 125.2 1.0637 [1.0583,1.0720] [1.0578,1.0726]
gbq 1.00 167.1 1.0646 [1.0622,1.0696] [1.0616,1.0699]
const 0.25 41.9 1.0729 [1.0654,1.0825] [1.0640,1.0874]
const 0.50 83.9 1.0678 [1.0650,1.0690] [1.0650,1.0697]
const 0.75 125.2 1.0659 [1.0647,1.0685] [1.0647,1.0693]
const 1.00 167.1 1.0654 [1.0654,1.0654] [1.0654,1.0654]
--- scheme: by-application ---
model frac mean n_train median err 95% CI IQR over reps
--------------------------------------------------------------------------
rf 0.25 47.4 1.1333 [1.1183,1.1417] [1.1170,1.1448]
rf 0.50 95.3 1.0917 [1.0817,1.1050] [1.0806,1.1072]
rf 0.75 120.7 1.0784 [1.0709,1.0907] [1.0693,1.0914]
rf 1.00 167.1 1.0560 [1.0552,1.0580] [1.0549,1.0582]
gbq 0.25 47.4 1.1192 [1.1059,1.1372] [1.1047,1.1379]
gbq 0.50 95.3 1.0968 [1.0900,1.1098] [1.0867,1.1108]
gbq 0.75 120.7 1.0805 [1.0730,1.0870] [1.0699,1.0880]
gbq 1.00 167.1 1.0646 [1.0622,1.0696] [1.0616,1.0699]
const 0.25 47.4 1.0881 [1.0808,1.0992] [1.0798,1.1049]
const 0.50 95.3 1.0785 [1.0678,1.0930] [1.0658,1.0960]
const 0.75 120.7 1.0734 [1.0664,1.0817] [1.0662,1.0827]
const 1.00 167.1 1.0654 [1.0654,1.0654] [1.0654,1.0654]
============================================================================================
DOES IT SATURATE?
Test: compare the 75%->100% step against the 25%->50% step. If the
final step is a small fraction of the early step, and its CI includes
zero improvement, the curve has flattened.
random rf : 25%=1.0871 50%=1.0709 75%=1.0622 100%=1.0585
total gain 25->100% +0.0286 early step +0.0162 final step +0.0037
75% and 100% CIs OVERLAP -> no resolvable gain from the last quarter
random gbq : 25%=1.0915 50%=1.0735 75%=1.0637 100%=1.0646
total gain 25->100% +0.0269 early step +0.0180 final step -0.0009
75% and 100% CIs OVERLAP -> no resolvable gain from the last quarter
by-application rf : 25%=1.1333 50%=1.0917 75%=1.0784 100%=1.0560
total gain 25->100% +0.0772 early step +0.0416 final step +0.0223
75% and 100% CIs are DISJOINT -> the last quarter still helps
by-application gbq : 25%=1.1192 50%=1.0968 75%=1.0805 100%=1.0646
total gain 25->100% +0.0546 early step +0.0224 final step +0.0159
75% and 100% CIs are DISJOINT -> the last quarter still helps
VERDICT
Density (random rows). RF goes from 1.0871 at a quarter of the rows to 1.0585 at all of them, a total gain of +0.0286, and the
75% and 100% points sit inside each other's confidence intervals.
Adding more configurations of the SAME eight codes would buy
almost nothing. On this axis the dataset is saturated.
Coverage (random applications). RF goes from 1.1333 with two
training codes to 1.0560 with seven, a gain of +0.0772, and the
curve is still descending at the right-hand edge. The binding
constraint is the NUMBER OF APPLICATIONS, not the number of
configurations.
So the answer to 'would more data help?' is: more runs of the same
codes, no; more codes, yes. With eight applications, LOAO has
eight folds and every fold trains on seven codes, which is the
steep part of the coverage curve. That is also why the LOAO
variance is so large that a random forest median moves in the
third decimal place between seeds (see reconcile_rf.txt).
Design implication: the profiling budget was spent widening the
configuration sweep per application. The learning curve says it
should have been spent on more applications with a coarser sweep
each.
figure written to /work/project/project/user/analysis/out/fig_learning_curve.png