Repository navigation
Conversation
Regressors were taken from all history while clean commits only from the last two years, which let the model learn "old-looking patch" (e.g. lower experience counts) instead of "risky patch". Use the same two-year window for both classes. On held-out windows this improves bug-level AUC by about +0.015/+0.027.
Use 600 trees with a 0.03 learning rate, depth 6, row/column subsampling and a minimum child weight of 5 instead of the defaults. On held-out windows this improves bug-level AUC by about +0.015/+0.02.
Enable the commit message TF-IDF features (interpretable=False) by default: the message is also available when classifying patches under review. On held-out windows this improves bug-level AUC by about +0.010/+0.016.
The risk band thresholds were set manually through the REGRESSOR_RISK_BANDS secret, and a gap between bands made find_risk_band raise. Compute four contiguous bands (lower 50%, average 30%, higher 15%, very high 5% of bugs) from the risk distribution of the evaluation period at training time and store them in the model; the secret is only used for older models. Thresholds are chosen among distinct risk values, as calibrated risks have many ties. Show the new band in the changes UI and count it in the landings risk report (which also counted average-risk changes under a band name that doesn't exist).
Labels are per bug and predictions are evaluated per bug, so bugs with many commits dominated training. Give each bug the same total weight, split among its commits, and balance the classes with weights instead of undersampling (which would not keep the weights aligned with the samples). IsotonicRegressionCalibrator now accepts sample weights. End to end on held-out windows this improves bug-level AUC by +0.04/+0.05 over the undersampling version.
marco-c
force-pushed
the
regressor-model-improvements
branch
from
November 16, 2026 21:36
a0d3a90 to
73dfe56
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improvements to the regressor model, one change per commit:
interpretable=False, adding commit-message TF-IDF).evaluation()computes the bands from the last 3 months of bugs and stores them in the model, replacing the manualREGRESSOR_RISK_BANDSsecret. Old pickles fall back to the secret. There are now 4 bands (50/30/15/5% of bugs: low/average/high/very high), with tie-aware thresholds because isotonic calibration outputs few distinct values. The landings risk report and the changes UI get the new "very high" band.RandomUnderSampler, so all non-regressor commits are used. The isotonic calibrator now accepts sample weights.Evaluation
Temporal split: trained on commits pushed before 2026-07-01 and tested on the 4,331 bugs with patches pushed in Jul–Aug 2026 (404 caused regressions). Scores are at bug level (the max over the bug's commits).
End-to-end training with
scripts.trainer regressor(DBs of 2026-11-14)These rates are underestimated because recent regressions haven't all been filed yet; the ratios between bands are what matter.
After this lands, the
REGRESSOR_RISK_BANDSsecret is only needed until the next model is trained.