Validation · grouped cross-validation, seed 42
Model Lab
Three small logistic models, each tested on groups it never saw. Droplet and solid data are never pooled. One fails, and is shown failing.
Model
Positive class: test ended in Extinction (visible flame went out before the fuel was gone).
Headline AUC · leave-one-fuel-out
0.74
is the worse of grouped cross-validation (0.88) and the leave-one-fuel-out stress test (0.74). 0.5 is a coin flip; 1 is perfect.
Source: scripts/train_models.py → data/models/droplet_extinction.json; fold membership in data/cv_splits.json
0.88
grouped-CV ROC-AUC
95% interval 0.81 to 0.94 · cluster bootstrap over 46 groups
18%
false-positive rate at the working threshold
16 of 90 negatives called positive at threshold 0.68 (Youden's J)
275
rows after exclusions
185 positive (67%) · 46 groups · class weights not needed
Can it tell the two outcomes apart, and what does it get wrong?
Every point was scored by a model that never saw its group.
ROC curve, out-of-fold predictions only
- Model
- Coin flip
- Working threshold
- PR-AUC
- 0.94
- Brier score
- 0.126
- Majority-class accuracy
- 67% (the bar to beat)
- Sensitivity for “the visible flame did not go out” at 90% specificity
- 64%
- Row-wise CV AUC
- 0.89 (leaky; shown only for contrast)
Confusion matrices
58
actual 0, predicted 0
32
actual 0, predicted 1
19
actual 1, predicted 0
166
actual 1, predicted 1
74
actual 0, predicted 0
16
actual 0, predicted 1
39
actual 1, predicted 0
146
actual 1, predicted 1
The costly error is top right: 16 tests where the flame did not go out but the model said it would. A model like this must not be relied on by itself for a fire-protection decision.
Source: Pooled out-of-fold predictions, outer GroupKFold with 5 folds; inner GroupKFold with 3 folds chooses C from [0.01, 0.1, 1, 10] (chosen for the final fit: 1).
Are the probabilities honest, and does it hold up group by group?
On the dashed line means well calibrated. A blank AUC means the group has only one outcome.
Calibration, in ten bins
- Model, one dot per bin (bigger = more tests)
- Perfect calibration
Calibration slope 0.90 (1 is ideal; below 1 means over-confident), intercept 0.09. Bin sizes are small; read the shape, not single points.
Per group, and the leave-one-fuel-out stress test
| O2 0.12 / CO2 0.00 / He 0.00 @ 1 atm | 8 | 8 | — | 0.98 | 1.00 |
| O2 0.13 / CO2 0.00 / He 0.00 @ 1 atm | 8 | 8 | — | 0.96 | 1.00 |
| O2 0.13 / CO2 0.00 / He 0.40 @ 1 atm | 1 | 1 | — | 1.00 | 1.00 |
| O2 0.14 / CO2 0.00 / He 0.00 @ 1 atm | 7 | 6 | 1.00 | 0.88 | 0.86 |
| O2 0.14 / CO2 0.00 / He 0.35 @ 1 atm | 2 | 2 | — | 0.98 | 1.00 |
| O2 0.14 / CO2 0.36 / He 0.00 @ 1 atm | 2 | 2 | — | 0.99 | 1.00 |
| O2 0.15 / CO2 0.00 / He 0.00 @ 1 atm | 11 | 8 | 1.00 | 0.83 | 0.73 |
| O2 0.15 / CO2 0.00 / He 0.30 @ 1 atm | 2 | 2 | — | 1.00 | 1.00 |
| O2 0.15 / CO2 0.32 / He 0.00 @ 1 atm | 4 | 4 | — | 0.98 | 1.00 |
| O2 0.15 / CO2 0.50 / He 0.00 @ 0.7 atm | 4 | 4 | — | 0.97 | 1.00 |
| O2 0.16 / CO2 0.00 / He 0.00 @ 1 atm | 3 | 3 | — | 0.83 | 1.00 |
| O2 0.16 / CO2 0.00 / He 0.25 @ 1 atm | 3 | 3 | — | 0.90 | 1.00 |
| O2 0.16 / CO2 0.25 / He 0.00 @ 1 atm | 4 | 4 | — | 0.98 | 1.00 |
| O2 0.17 / CO2 0.00 / He 0.20 @ 1 atm | 6 | 4 | 0.75 | 0.88 | 0.67 |
| O2 0.17 / CO2 0.00 / He 0.45 @ 0.7 atm | 4 | 3 | 1.00 | 0.93 | 0.75 |
| O2 0.17 / CO2 0.20 / He 0.00 @ 1 atm | 4 | 4 | — | 0.93 | 1.00 |
| O2 0.17 / CO2 0.45 / He 0.00 @ 0.7 atm | 7 | 7 | — | 0.94 | 1.00 |
| O2 0.18 / CO2 0.00 / He 0.00 @ 1 atm | 7 | 6 | 1.00 | 0.82 | 0.86 |
| O2 0.18 / CO2 0.00 / He 0.15 @ 1 atm | 5 | 3 | 1.00 | 0.82 | 0.60 |
| O2 0.18 / CO2 0.00 / He 0.40 @ 0.7 atm | 5 | 5 | — | 0.91 | 1.00 |
| O2 0.18 / CO2 0.15 / He 0.00 @ 1 atm | 13 | 10 | 0.67 | 0.72 | 0.77 |
| O2 0.18 / CO2 0.40 / He 0.00 @ 0.7 atm | 6 | 6 | — | 0.91 | 1.00 |
| O2 0.19 / CO2 0.00 / He 0.10 @ 1 atm | 2 | 1 | 1.00 | 0.74 | 0.50 |
| O2 0.19 / CO2 0.10 / He 0.00 @ 1 atm | 5 | 4 | 1.00 | 0.66 | 0.80 |
| O2 0.20 / CO2 0.00 / He 0.00 @ 0.7 atm | 5 | 3 | 0.33 | 0.67 | 0.60 |
| O2 0.20 / CO2 0.00 / He 0.05 @ 1 atm | 4 | 2 | 1.00 | 0.64 | 0.50 |
| O2 0.20 / CO2 0.00 / He 0.35 @ 0.7 atm | 4 | 4 | — | 0.90 | 1.00 |
| O2 0.20 / CO2 0.05 / He 0.00 @ 1 atm | 12 | 9 | 1.00 | 0.65 | 0.75 |
| O2 0.20 / CO2 0.35 / He 0.00 @ 0.7 atm | 5 | 5 | — | 0.85 | 1.00 |
| O2 0.21 / CO2 0.00 / He 0.00 @ 1 atm | 29 | 14 | 0.66 | 0.51 | 0.48 |
| O2 0.21 / CO2 0.00 / He 0.15 @ 1 atm | 2 | 2 | — | 0.67 | 1.00 |
| O2 0.21 / CO2 0.00 / He 0.30 @ 0.7 atm | 5 | 5 | — | 0.78 | 1.00 |
| O2 0.21 / CO2 0.15 / He 0.00 @ 1 atm | 3 | 2 | 1.00 | 0.81 | 0.67 |
| O2 0.21 / CO2 0.30 / He 0.00 @ 0.7 atm | 6 | 4 | 0.13 | 0.86 | 0.67 |
| O2 0.21 / CO2 0.70 / He 0.00 @ 2 atm | 2 | 1 | 0.00 | 0.86 | 0.50 |
| O2 0.21 / CO2 0.70 / He 0.00 @ 3 atm | 7 | 6 | 1.00 | 0.62 | 0.86 |
| O2 0.23 / CO2 0.00 / He 0.25 @ 0.7 atm | 3 | 1 | 0.50 | 0.64 | 0.33 |
| O2 0.23 / CO2 0.20 / He 0.00 @ 0.7 atm | 4 | 1 | 0.33 | 0.84 | 0.25 |
| O2 0.23 / CO2 0.25 / He 0.00 @ 0.7 atm | 12 | 9 | 0.52 | 0.53 | 0.75 |
| O2 0.24 / CO2 0.00 / He 0.20 @ 0.7 atm | 3 | 2 | 0.50 | 0.49 | 0.67 |
| O2 0.24 / CO2 0.20 / He 0.00 @ 0.7 atm | 7 | 4 | 1.00 | 0.35 | 0.57 |
| O2 0.24 / CO2 0.20 / He 0.00 @ 1 atm | 9 | 1 | 1.00 | 0.14 | 0.11 |
| O2 0.25 / CO2 0.15 / He 0.00 @ 0.7 atm | 6 | 0 | — | 0.45 | 0.00 |
| O2 0.30 / CO2 0.00 / He 0.00 @ 0.7 atm | 8 | 0 | — | 0.09 | 0.00 |
| O2 0.30 / CO2 0.00 / He 0.00 @ 1 atm | 7 | 2 | 0.70 | 0.01 | 0.29 |
| O2 0.34 / CO2 0.00 / He 0.00 @ 0.7 atm | 9 | 0 | — | 0.02 | 0.00 |
Stress test: train on the others, test on the one left out
- Heptane60 of 119 positiveAUC 0.66
- Methanol125 of 156 positiveAUC 0.81
Source: Out-of-fold predictions per group; the stress test is reported separately and never averaged into the grouped-CV figure.
What moved the prediction?
Each bar is a change in log-odds for a one-standard-deviation increase, with a 95% bootstrap interval.
- raises the odds of the positive class
- lowers them
- 95% bootstrap interval
How to read this chart
Source: Final model refitted on all 275 rows; intervals from 1000 cluster-bootstrap refits.
Where does the model stop?
The ranges it was trained on, and a live check of how far any query sits from that data.
Training-data envelope
- Oxygen, %12.00 to 34.00 %
- Pressure70.17 to 309.29 kPa
- O2 partial pressure10.67 to 64.95 kPa
- CO2, %0.00 to 70.00 %
- Helium, %0.00 to 45.00 %
- Droplet diameter1.10 to 4.67 mm
Nearest-neighbour distance between training rows: median 0.13, 95th percentile 0.70 (standardised units).
Try a query
It starts outside the envelope on purpose.
Distance-to-data warning: this query is an extrapolation
Model output: 0% (95% interval 0% to 0%)DEMO
Nearest training rows: FLEX-039 (2.47), FLEX-023 (2.48), FLEX-024 (2.48), FLEX-037 (2.49)
How it was built
The protocol, the features left out, and the features added with their written reason.
Sensitivity check: leave the disrupted tests out
Source: Research specification §7.3, adapted where the data forced it; every adaptation is listed in DECISIONS.md in the repository.