Skip to content

Improve the regressor model: training window, tuning, commit messages, bug weighting and automatic risk bands - #7515

Draft
marco-c wants to merge 5 commits into
mozilla:masterfrom
marco-c:regressor-model-improvements
Draft

marco-c wants to merge 5 commits into
mozilla:masterfrom
marco-c:regressor-model-improvements

Conversation

@marco-c

@marco-c marco-c commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Improvements to the regressor model, one change per commit:

  1. Train on two years of data for both classes. Master took regressors from all history but non-regressors only from the last two years, so the model partly learned "old vs recent".
  2. Tune the XGBoost hyperparameters (600 trees, learning rate 0.03, depth 6, row/column subsampling).
  3. Use commit messages by default (interpretable=False, adding commit-message TF-IDF).
  4. Compute risk bands at training time. evaluation() computes the bands from the last 3 months of bugs and stores them in the model, replacing the manual REGRESSOR_RISK_BANDS secret. Old pickles fall back to the secret. There are now 4 bands (50/30/15/5% of bugs: low/average/high/very high), with tie-aware thresholds because isotonic calibration outputs few distinct values. The landings risk report and the changes UI get the new "very high" band.
  5. Weight training samples by bug instead of undersampling. Each bug's commits are weighted 1/n, and the classes are balanced with weights instead of RandomUnderSampler, so all non-regressor commits are used. The isotonic calibrator now accepts sample weights.

Evaluation

Temporal split: trained on commits pushed before 2026-07-01 and tested on the 4,331 bugs with patches pushed in Jul–Aug 2026 (404 caused regressions). Scores are at bug level (the max over the bug's commits).

master this PR
AUC 0.688 0.772
Average precision 0.173 0.273
Regressions caught in the riskiest 5% of bugs 51 83
Regressions caught in the riskiest 20% of bugs 164 223

End-to-end training with scripts.trainer regressor (DBs of 2026-11-14)

  • 31 min wall clock, peak memory 14.4 GB; model 2.1 MB (776 KB compressed).
  • Bands computed on the last 3 months (9,666 bugs):
Band Risk range Bugs Regression rate
low 0 – 0.42 5,003 2.8%
average 0.42 – 0.65 2,528 8.0%
high 0.65 – 0.80 1,630 13.6%
very high 0.80 – 1 505 28.9%

These rates are underestimated because recent regressions haven't all been filed yet; the ratios between bands are what matter.

After this lands, the REGRESSOR_RISK_BANDS secret is only needed until the next model is trained.

Regressors were taken from all history while clean commits only from the last two years, which
let the model learn "old-looking patch" (e.g. lower experience counts) instead of "risky patch".
Use the same two-year window for both classes.

On held-out windows this improves bug-level AUC by about +0.015/+0.027.
Use 600 trees with a 0.03 learning rate, depth 6, row/column subsampling and a minimum child
weight of 5 instead of the defaults.

On held-out windows this improves bug-level AUC by about +0.015/+0.02.
Enable the commit message TF-IDF features (interpretable=False) by default: the message is also
available when classifying patches under review.

On held-out windows this improves bug-level AUC by about +0.010/+0.016.
The risk band thresholds were set manually through the REGRESSOR_RISK_BANDS secret, and a gap
between bands made find_risk_band raise. Compute four contiguous bands (lower 50%, average 30%,
higher 15%, very high 5% of bugs) from the risk distribution of the evaluation period at training
time and store them in the model; the secret is only used for older models. Thresholds are
chosen among distinct risk values, as calibrated risks have many ties.

Show the new band in the changes UI and count it in the landings risk report (which also counted
average-risk changes under a band name that doesn't exist).
Labels are per bug and predictions are evaluated per bug, so bugs with many commits dominated
training. Give each bug the same total weight, split among its commits, and balance the classes
with weights instead of undersampling (which would not keep the weights aligned with the
samples). IsotonicRegressionCalibrator now accepts sample weights.

End to end on held-out windows this improves bug-level AUC by +0.04/+0.05 over the
undersampling version.
@marco-c
marco-c force-pushed the regressor-model-improvements branch from a0d3a90 to 73dfe56 Compare November 16, 2026 21:36

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant