Web interface

CANARY Web Console

Security-advisory forecasting for Jenkins plugins from public signals, evaluated honestly: four validation layers, one pre-registered criterion, and every number on this site traceable to the layer that produced it.

Honest evaluation — embargoed rolling backtests

The metrics on the Machine-learning tab are historical / diagnostic: they use the standard chronological protocol, which lets training labels share advisory events with the test window and inflates apparent performance. The numbers below are deployment-honest: the same evaluation is repeated at many points in time (an expanding-window rolling backtest), and at every fold the training labels are rebuilt using only advisories published before that fold's scoring date (the label embargo). No training label ever depends on an advisory the model is later tested on. Pooled metrics combine every fold's test predictions — hundreds of advisory outcomes across 2023–2025 rather than one two-month window.

Runs marked stored labels use the historical (leaky) labels on the same folds, shown for contrast. Produced by tools/rolling_backtest.py.

Feature-set names are the feature families a run was allowed to see — hover a name for what the family is made of, or expand the legend.

What the feature-set names mean
  • Activity-recency clocks ghclock_ — nine day-resolution clocks from GH Archive events, measured at month end: days since the last human push, any push (bots included), release, pull request opened, PR merged, PR review, issue opened and tag created, plus a has-events flag. 'Never' is encoded as a 3,650-day cap plus the flag, so a missing history can never look like recent activity.
  • Contributor dynamics ghdyn_ — nine columns on who is doing the work, from GH Archive actors over trailing windows: active contributors in the last 3 and 12 months, newcomers and departures over 12 months, turnover rate, event count, the top contributor's share of activity, a single-maintainer flag and a has-actors flag. Bot accounts are excluded.
  • Install base installs_ — ten columns from stats.jenkins.io monthly installation history: install count (raw and log10), share of all Jenkins installs, ecosystem percentile rank, 3- and 12-month growth (clamped), ratio to the plugin's own peak, 12-month rank change, months of data and a has-data flag. Month T uses figures published through T-1 — what was knowable at scoring time.
  • Advisory recurrence advhist_ — ten columns from the plugin's own advisory calendar: months since the last and first advisory, advisory months and advisory count to date, advisory months in the trailing 12 and 24 months, mean gap between advisories, size of the ecosystem batch the latest advisory arrived in, an exponential recency decay and a has-history flag.
  • Security vocabulary in project text ghtext_ — seven columns from the text of commits, pull requests and issues in GH Archive: days since the last security-vocabulary mention (CVE/CWE ids, 'vulnerability', XSS, CSRF, sanitize, disclosure, ...) and since the last CVE id, mention counts over 90 and 365 days, and has-mention flags. Generic 'fix' / 'bug' words are excluded.
  • Shared-maintainer contagion contagion_ — eight columns from a maintainer graph rebuilt as-of each month from GH Archive actors (push, PR merge, release; trailing 24 months): the plugin's active maintainers, maintainers shared with other plugins, number of neighbouring plugins, neighbours with an advisory in the last 12 and 24 months, months since the latest neighbour advisory, and has-neighbour flags.
  • Software Heritage visit deltas swhdelta_ — twelve columns comparing consecutive Software Heritage archive visits: visits to date, days since and between visits, commits added and commit rate per month, security-fix commits added, governance/tooling files adopted or dropped (SECURITY.md, Dependabot, GitHub Actions, Jenkinsfile, tests, ...), days since the last adoption, and presence flags. Changes between visits, not levels.
  • Advisory history (base panel) advisory_ — the advisory columns built into the monthly panel: count to date, count this month, CVE count, maximum and mean CVSS to date, count with CVSS >= 7, days since the first and latest advisory, and advisory span.
  • Advisory presence flags (base panel) advisories_ — whether the plugin has ever had an advisory and whether it had one in the trailing 365 days.
  • GH Archive activity (base panel) gharchive_ — the ~100 month-granularity GitHub activity columns of the base panel: event, push, pull-request, issue, release, tag, fork and watch counts over trailing 3- and 6-month windows, human vs bot splits, unique actors, PR merge and issue close rates and times, months since the last activity of each kind, and security / hotfix / dependency-bump keyword counts.
  • Software Heritage snapshot (base panel) swh_ — the ~37 archive-visit columns of the base panel, as of the most recent visit before the observation month: visit counts and archive age, commit count and days since the last commit, commit-hygiene rates (merge, conventional-commit, issue-reference, empty-message, weekend / late-night fractions), security-fix commit count, and the presence of governance and build files (README, SECURITY.md, Dependabot, GitHub Actions, Jenkinsfile, tests directory, ...).

The turning point

Same features, same test months, honest labels

Each pair is one configuration scored twice on identical test data. Stored labels let a training label be set by an advisory published inside the test window; embargoed rebuilds every training label from advisories known before the fold's forecast date. The gap is the leak. Hover a bar for the value.

ROC-AUC — separation of advisory-bound plugins from the rest; 0.5 is chance

stored labels (leaky)embargoed (honest)chance 0.50Jenkins · Advisory + Software Heritage (official config) · XGBoosttime split, 1 fold, 77 positivesJenkins · Advisory + Software Heritage (official config) · XGBoost stored: ROC-AUC 0.9300.930Jenkins · Advisory + Software Heritage (official config) · XGBoost embargoed: ROC-AUC 0.4820.482Jenkins · Advisory history (base panel) · XGBoostrolling, 13 folds, 760 positivesJenkins · Advisory history (base panel) · XGBoost stored: ROC-AUC 0.6020.602Jenkins · Advisory history (base panel) · XGBoost embargoed: ROC-AUC 0.5530.553PyPI · Advisory history (base panel) · XGBoostrolling, 13 folds, 2,143 positivesPyPI · Advisory history (base panel) · XGBoost stored: ROC-AUC 0.7730.773PyPI · Advisory history (base panel) · XGBoost embargoed: ROC-AUC 0.7200.720

Average precision — concentration of advisories at the top of the ranked list

stored labels (leaky)embargoed (honest)Jenkins · Advisory + Software Heritage (official config) · XGBoosttime split, 1 fold, 77 positivesJenkins · Advisory + Software Heritage (official config) · XGBoost stored: Average precision 0.5830.583Jenkins · Advisory + Software Heritage (official config) · XGBoost embargoed: Average precision 0.0190.019Jenkins · Advisory history (base panel) · XGBoostrolling, 13 folds, 760 positivesJenkins · Advisory history (base panel) · XGBoost stored: Average precision 0.0370.037Jenkins · Advisory history (base panel) · XGBoost embargoed: Average precision 0.0240.024PyPI · Advisory history (base panel) · XGBoostrolling, 13 folds, 2,143 positivesPyPI · Advisory history (base panel) · XGBoost stored: Average precision 0.2880.288PyPI · Advisory history (base panel) · XGBoost embargoed: Average precision 0.1870.187

Best embargoed development result

Activity-recency clocks + Contributor dynamics (Logistic Regression)

Pooled ROC-AUC0.638
Pooled avg precision0.0308
Lift over base rate2.16×
Fold ROC range0.479 – 0.824

13 folds, 53,378 test rows, 760 advisory outcomes.

Every fold, one criterion

ROC-AUC per fold across 16 forecast dates

Each point is one two-month test window scored by a model trained only on what was knowable at that date. The configurations were chosen on the development folds; the criterion was written down before the holdout months were collected; the holdout folds were run once. Larger markers are the recorded out-of-time folds. Hover a point for the fold's outcome count.

pre-registered holdoutdevelopment sweep0.400.450.500.550.600.650.700.750.800.85chance 0.50pre-registered criterion 0.552023-052023-072023-092023-112024-012024-032024-052024-072024-092024-112025-012025-032025-052025-072025-092025-11Activity-recency clocks + Contributor dynamics · Logistic Regression 2023-05 · ROC-AUC 0.700 · 132 advisory outcomesActivity-recency clocks + Contributor dynamics · Logistic Regression 2023-07 · ROC-AUC 0.668 · 90 advisory outcomesActivity-recency clocks + Contributor dynamics · Logistic Regression 2023-09 · ROC-AUC 0.658 · 67 advisory outcomesActivity-recency clocks + Contributor dynamics · Logistic Regression 2023-11 · ROC-AUC 0.652 · 55 advisory outcomesActivity-recency clocks + Contributor dynamics · Logistic Regression 2024-01 · ROC-AUC 0.666 · 40 advisory outcomesActivity-recency clocks + Contributor dynamics · Logistic Regression 2024-03 · ROC-AUC 0.652 · 22 advisory outcomesActivity-recency clocks + Contributor dynamics · Logistic Regression 2024-05 · ROC-AUC 0.824 · 23 advisory outcomesActivity-recency clocks + Contributor dynamics · Logistic Regression 2024-07 · ROC-AUC 0.794 · 32 advisory outcomesActivity-recency clocks + Contributor dynamics · Logistic Regression 2024-09 · ROC-AUC 0.699 · 42 advisory outcomesActivity-recency clocks + Contributor dynamics · Logistic Regression 2024-11 · ROC-AUC 0.615 · 39 advisory outcomesActivity-recency clocks + Contributor dynamics · Logistic Regression 2025-01 · ROC-AUC 0.497 · 66 advisory outcomesActivity-recency clocks + Contributor dynamics · Logistic Regression 2025-03 · ROC-AUC 0.519 · 75 advisory outcomesActivity-recency clocks + Contributor dynamics · Logistic Regression 2025-05 · ROC-AUC 0.479 · 77 advisory outcomesActivity-recency clocks + Contributor dynamics · Logistic Regression 2025-07 · ROC-AUC 0.589 · 40 advisory outcomesActivity-recency clocks + Contributor dynamics · Logistic Regression 2025-09 · ROC-AUC 0.632 · 31 advisory outcomesActivity-recency clocks + Contributor dynamics · Logistic Regression 2025-11 · ROC-AUC 0.758 · 56 advisory outcomesActivity-recency clocks · Logistic Regression 2023-05 · ROC-AUC 0.686 · 132 advisory outcomesActivity-recency clocks · Logistic Regression 2023-07 · ROC-AUC 0.654 · 90 advisory outcomesActivity-recency clocks · Logistic Regression 2023-09 · ROC-AUC 0.658 · 67 advisory outcomesActivity-recency clocks · Logistic Regression 2023-11 · ROC-AUC 0.648 · 55 advisory outcomesActivity-recency clocks · Logistic Regression 2024-01 · ROC-AUC 0.673 · 40 advisory outcomesActivity-recency clocks · Logistic Regression 2024-03 · ROC-AUC 0.628 · 22 advisory outcomesActivity-recency clocks · Logistic Regression 2024-05 · ROC-AUC 0.738 · 23 advisory outcomesActivity-recency clocks · Logistic Regression 2024-07 · ROC-AUC 0.743 · 32 advisory outcomesActivity-recency clocks · Logistic Regression 2024-09 · ROC-AUC 0.703 · 42 advisory outcomesActivity-recency clocks · Logistic Regression 2024-11 · ROC-AUC 0.661 · 39 advisory outcomesActivity-recency clocks · Logistic Regression 2025-01 · ROC-AUC 0.486 · 66 advisory outcomesActivity-recency clocks · Logistic Regression 2025-03 · ROC-AUC 0.499 · 75 advisory outcomesActivity-recency clocks · Logistic Regression 2025-05 · ROC-AUC 0.464 · 77 advisory outcomesActivity-recency clocks · Logistic Regression 2025-07 · ROC-AUC 0.580 · 40 advisory outcomesActivity-recency clocks · Logistic Regression 2025-09 · ROC-AUC 0.606 · 31 advisory outcomesActivity-recency clocks · Logistic Regression 2025-11 · ROC-AUC 0.717 · 56 advisory outcomesActivity-recency clocks + Install base · XGBoost 2023-05 · ROC-AUC 0.668 · 132 advisory outcomesActivity-recency clocks + Install base · XGBoost 2023-07 · ROC-AUC 0.624 · 90 advisory outcomesActivity-recency clocks + Install base · XGBoost 2023-09 · ROC-AUC 0.666 · 67 advisory outcomesActivity-recency clocks + Install base · XGBoost 2023-11 · ROC-AUC 0.632 · 55 advisory outcomesActivity-recency clocks + Install base · XGBoost 2024-01 · ROC-AUC 0.684 · 40 advisory outcomesActivity-recency clocks + Install base · XGBoost 2024-03 · ROC-AUC 0.711 · 22 advisory outcomesActivity-recency clocks + Install base · XGBoost 2024-05 · ROC-AUC 0.748 · 23 advisory outcomesActivity-recency clocks + Install base · XGBoost 2024-07 · ROC-AUC 0.674 · 32 advisory outcomesActivity-recency clocks + Install base · XGBoost 2024-09 · ROC-AUC 0.578 · 42 advisory outcomesActivity-recency clocks + Install base · XGBoost 2024-11 · ROC-AUC 0.649 · 39 advisory outcomesActivity-recency clocks + Install base · XGBoost 2025-01 · ROC-AUC 0.609 · 66 advisory outcomesActivity-recency clocks + Install base · XGBoost 2025-03 · ROC-AUC 0.582 · 75 advisory outcomesActivity-recency clocks + Install base · XGBoost 2025-05 · ROC-AUC 0.600 · 77 advisory outcomesActivity-recency clocks + Install base · XGBoost 2025-07 · ROC-AUC 0.611 · 40 advisory outcomesActivity-recency clocks + Install base · XGBoost 2025-09 · ROC-AUC 0.728 · 31 advisory outcomesActivity-recency clocks + Install base · XGBoost 2025-11 · ROC-AUC 0.601 · 56 advisory outcomesActivity-recency clocks + Contributor dynamics · Logistic RegressionActivity-recency clocks · Logistic RegressionActivity-recency clocks + Install base · XGBoost

Pre-registered out-of-time result

Configurations frozen before any post-boundary data was collected, each run once on months no modelling decision ever saw (3 folds, 12,318 test rows, 127 advisory outcomes). The first recorded run is the result; the development-era pooled ROC-AUC of the same configuration is shown for comparison.

Feature setModelOut-of-time ROC-AUCDevelopment ROC-AUCAP liftFold ROC range
Activity-recency clocks + Contributor dynamics
oot_champion		Logistic Regression0.6700.6383.81×0.589 – 0.758
Activity-recency clocks
oot_ghclock_logistic		Logistic Regression0.6380.6271.77×0.580 – 0.717
Activity-recency clocks + Install base
oot_runnerup		XGBoost0.6280.6322.03×0.601 – 0.728

What the holdout ranking looks like

Advisories caught vs plugins reviewed, out-of-time

Reviewing the top 20% of plugins by score would have caught 52% of the advisories that followed across the holdout folds (by fold: 42%, 52%, 61%), against a base rate of 1.0%. The thick line is the pooled ranking over every holdout fold; thin lines are the individual folds (12,318 plugin-months, 127 advisory outcomes). The dashed diagonal is random order. Right: the ROC curve of each fold for the first configuration.

0%0%20%20%40%40%60%60%80%80%100%100%random orderActivity-recency clocks + Contributor dynamics · Logistic Regression review top 1% of plugins → 6% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 2% of plugins → 11% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 3% of plugins → 20% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 5% of plugins → 28% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 8% of plugins → 35% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 10% of plugins → 39% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 15% of plugins → 45% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 20% of plugins → 52% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 25% of plugins → 54% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 30% of plugins → 57% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 40% of plugins → 63% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 50% of plugins → 69% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 60% of plugins → 72% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 70% of plugins → 78% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 80% of plugins → 86% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 90% of plugins → 96% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 100% of plugins → 100% of advisoriesActivity-recency clocks · Logistic Regression review top 1% of plugins → 5% of advisoriesActivity-recency clocks · Logistic Regression review top 2% of plugins → 6% of advisoriesActivity-recency clocks · Logistic Regression review top 3% of plugins → 6% of advisoriesActivity-recency clocks · Logistic Regression review top 5% of plugins → 9% of advisoriesActivity-recency clocks · Logistic Regression review top 8% of plugins → 13% of advisoriesActivity-recency clocks · Logistic Regression review top 10% of plugins → 18% of advisoriesActivity-recency clocks · Logistic Regression review top 15% of plugins → 31% of advisoriesActivity-recency clocks · Logistic Regression review top 20% of plugins → 41% of advisoriesActivity-recency clocks · Logistic Regression review top 25% of plugins → 48% of advisoriesActivity-recency clocks · Logistic Regression review top 30% of plugins → 57% of advisoriesActivity-recency clocks · Logistic Regression review top 40% of plugins → 65% of advisoriesActivity-recency clocks · Logistic Regression review top 50% of plugins → 70% of advisoriesActivity-recency clocks · Logistic Regression review top 60% of plugins → 76% of advisoriesActivity-recency clocks · Logistic Regression review top 70% of plugins → 80% of advisoriesActivity-recency clocks · Logistic Regression review top 80% of plugins → 85% of advisoriesActivity-recency clocks · Logistic Regression review top 90% of plugins → 96% of advisoriesActivity-recency clocks · Logistic Regression review top 100% of plugins → 100% of advisoriesActivity-recency clocks + Install base · XGBoost review top 1% of plugins → 2% of advisoriesActivity-recency clocks + Install base · XGBoost review top 2% of plugins → 7% of advisoriesActivity-recency clocks + Install base · XGBoost review top 3% of plugins → 9% of advisoriesActivity-recency clocks + Install base · XGBoost review top 5% of plugins → 11% of advisoriesActivity-recency clocks + Install base · XGBoost review top 8% of plugins → 13% of advisoriesActivity-recency clocks + Install base · XGBoost review top 10% of plugins → 17% of advisoriesActivity-recency clocks + Install base · XGBoost review top 15% of plugins → 27% of advisoriesActivity-recency clocks + Install base · XGBoost review top 20% of plugins → 36% of advisoriesActivity-recency clocks + Install base · XGBoost review top 25% of plugins → 41% of advisoriesActivity-recency clocks + Install base · XGBoost review top 30% of plugins → 46% of advisoriesActivity-recency clocks + Install base · XGBoost review top 40% of plugins → 59% of advisoriesActivity-recency clocks + Install base · XGBoost review top 50% of plugins → 72% of advisoriesActivity-recency clocks + Install base · XGBoost review top 60% of plugins → 77% of advisoriesActivity-recency clocks + Install base · XGBoost review top 70% of plugins → 83% of advisoriesActivity-recency clocks + Install base · XGBoost review top 80% of plugins → 91% of advisoriesActivity-recency clocks + Install base · XGBoost review top 90% of plugins → 95% of advisoriesActivity-recency clocks + Install base · XGBoost review top 100% of plugins → 100% of advisoriestop 10% → 39% caughttop 20% → 52% caughttop 30% → 57% caughtshare of plugins reviewedActivity-recency clocks + Contributor dynamics · Logistic RegressionActivity-recency clocks · Logistic RegressionActivity-recency clocks + Install base · XGBoost
0.00.00.20.20.40.40.60.60.80.81.01.0false-positive rate2025-07 (40 outcomes) · AUC 0.5892025-09 (31 outcomes) · AUC 0.6322025-11 (56 outcomes) · AUC 0.758

Which signals survive, by ecosystem

Pooled embargoed ROC-AUC per feature set

Best model per feature set on the thirteen development folds, training labels rebuilt at every fold. Bars at or below chance are shown on purpose: knowing which signals do not carry forward is part of the result. The same advisory-history features sit at the bottom of the Jenkins ladder and at the top of the PyPI one (outlined bars are the family present in both), which is why the method is the contribution rather than any one feature family.

Jenkins plugins0.40.50.60.70.80.91.0chance 0.50criterion 0.55Activity-recency clocks + Contributor dynamicsLogistic Regression · best combinationActivity-recency clocks + Contributor dynamics pooled ROC-AUC 0.6380.638Activity-recency clocksLogistic RegressionActivity-recency clocks pooled ROC-AUC 0.6270.627Contributor dynamicsXGBoostContributor dynamics pooled ROC-AUC 0.5740.574Install baseXGBoostInstall base pooled ROC-AUC 0.5680.568Advisory history (base panel)XGBoostAdvisory history (base panel) pooled ROC-AUC 0.5530.553Shared-maintainer contagionXGBoostShared-maintainer contagion pooled ROC-AUC 0.5510.551Security vocabulary in project textXGBoostSecurity vocabulary in project text pooled ROC-AUC 0.4910.491Software Heritage visit deltasXGBoostSoftware Heritage visit deltas pooled ROC-AUC 0.4740.474Advisory recurrenceXGBoostAdvisory recurrence pooled ROC-AUC 0.4350.435PyPI packages0.40.50.60.70.80.91.0chance 0.50criterion 0.55Advisory history (base panel)Logistic RegressionAdvisory history (base panel) pooled ROC-AUC 0.7780.778

All rolling backtest runs

Feature setModelProtocolWindowFoldsPositivesPooled ROC-AUCPooled APAP liftFold ROC mean (range)
Activity-recency clocks + Contributor dynamics
ghclock_ghdyn_logistic		Logistic Regressionembargoeddevelopment137600.6380.03082.16×0.648 (0.479–0.824)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.7000.06722.09×
2023-07 – 2023-082023-0890 / 41060.6680.04902.24×
2023-09 – 2023-102023-1067 / 41060.6580.03332.04×
2023-11 – 2023-122023-1255 / 41060.6520.04443.32×
2024-01 – 2024-022024-0240 / 41060.6660.01921.97×
2024-03 – 2024-042024-0422 / 41060.6520.063311.81×
2024-05 – 2024-062024-0623 / 41060.8240.090116.09×
2024-07 – 2024-082024-0832 / 41060.7940.093111.95×
2024-09 – 2024-102024-1042 / 41060.6990.07657.48×
2024-11 – 2024-122024-1239 / 41060.6150.02012.11×
2025-01 – 2025-022025-0266 / 41060.4970.01721.07×
2025-03 – 2025-042025-0475 / 41060.5190.02501.37×
2025-05 – 2025-062025-0677 / 41060.4790.02291.22×
Activity-recency clocks + Install base
ghclock_installs_xgb		XGBoostembargoeddevelopment137600.6320.02531.78×0.648 (0.578–0.748)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.6680.07162.23×
2023-07 – 2023-082023-0890 / 41060.6240.03911.78×
2023-09 – 2023-102023-1067 / 41060.6660.02951.81×
2023-11 – 2023-122023-1255 / 41060.6320.01891.41×
2024-01 – 2024-022024-0240 / 41060.6840.02222.28×
2024-03 – 2024-042024-0422 / 41060.7110.03346.24×
2024-05 – 2024-062024-0623 / 41060.7480.03295.87×
2024-07 – 2024-082024-0832 / 41060.6740.02993.83×
2024-09 – 2024-102024-1042 / 41060.5780.01931.88×
2024-11 – 2024-122024-1239 / 41060.6490.02342.47×
2025-01 – 2025-022025-0266 / 41060.6090.02491.55×
2025-03 – 2025-042025-0475 / 41060.5820.03471.90×
2025-05 – 2025-062025-0677 / 41060.6000.02961.58×
Activity-recency clocks + Contributor dynamics + Install base
ghclock_ghdyn_installs_xgb		XGBoostembargoeddevelopment137600.6310.02691.89×0.654 (0.591–0.760)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.6540.06061.88×
2023-07 – 2023-082023-0890 / 41060.6250.04351.98×
2023-09 – 2023-102023-1067 / 41060.6290.03131.92×
2023-11 – 2023-122023-1255 / 41060.6150.01951.46×
2024-01 – 2024-022024-0240 / 41060.6590.06676.84×
2024-03 – 2024-042024-0422 / 41060.7440.02224.14×
2024-05 – 2024-062024-0623 / 41060.7600.126822.64×
2024-07 – 2024-082024-0832 / 41060.7320.05096.53×
2024-09 – 2024-102024-1042 / 41060.6330.02472.41×
2024-11 – 2024-122024-1239 / 41060.6480.02712.85×
2025-01 – 2025-022025-0266 / 41060.6150.02491.55×
2025-03 – 2025-042025-0475 / 41060.5910.02731.49×
2025-05 – 2025-062025-0677 / 41060.5970.02541.35×
Activity-recency clocks
ghclock_only_logistic		Logistic Regressionembargoeddevelopment137600.6270.02731.92×0.634 (0.464–0.743)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.6860.06442.00×
2023-07 – 2023-082023-0890 / 41060.6540.04412.01×
2023-09 – 2023-102023-1067 / 41060.6580.04112.52×
2023-11 – 2023-122023-1255 / 41060.6480.02381.78×
2024-01 – 2024-022024-0240 / 41060.6730.02552.62×
2024-03 – 2024-042024-0422 / 41060.6280.01322.47×
2024-05 – 2024-062024-0623 / 41060.7380.01803.21×
2024-07 – 2024-082024-0832 / 41060.7430.02753.53×
2024-09 – 2024-102024-1042 / 41060.7030.02412.36×
2024-11 – 2024-122024-1239 / 41060.6610.02392.51×
2025-01 – 2025-022025-0266 / 41060.4860.01891.17×
2025-03 – 2025-042025-0475 / 41060.4990.01991.09×
2025-05 – 2025-062025-0677 / 41060.4640.01810.96×
Activity-recency clocks + Contributor dynamics + Install base
ghclock_ghdyn_installs_logistic		Logistic Regressionembargoeddevelopment137600.6270.02912.05×0.643 (0.448–0.815)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.6950.08152.53×
2023-07 – 2023-082023-0890 / 41060.6720.04942.25×
2023-09 – 2023-102023-1067 / 41060.6510.03302.02×
2023-11 – 2023-122023-1255 / 41060.6640.02531.89×
2024-01 – 2024-022024-0240 / 41060.7090.02242.30×
2024-03 – 2024-042024-0422 / 41060.7160.066212.36×
2024-05 – 2024-062024-0623 / 41060.8150.075913.55×
2024-07 – 2024-082024-0832 / 41060.7680.05186.64×
2024-09 – 2024-102024-1042 / 41060.6580.05145.02×
2024-11 – 2024-122024-1239 / 41060.6060.01691.78×
2025-01 – 2025-022025-0266 / 41060.4630.01641.02×
2025-03 – 2025-042025-0475 / 41060.4900.02321.27×
2025-05 – 2025-062025-0677 / 41060.4480.02121.13×
Activity-recency clocks
ghclock_only_xgb		XGBoostembargoeddevelopment137600.6210.02521.77×0.638 (0.554–0.773)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.5610.05151.60×
2023-07 – 2023-082023-0890 / 41060.5590.02891.32×
2023-09 – 2023-102023-1067 / 41060.6560.04242.60×
2023-11 – 2023-122023-1255 / 41060.6310.02091.56×
2024-01 – 2024-022024-0240 / 41060.6850.02512.58×
2024-03 – 2024-042024-0422 / 41060.6720.056210.49×
2024-05 – 2024-062024-0623 / 41060.7730.066411.85×
2024-07 – 2024-082024-0832 / 41060.7540.03284.21×
2024-09 – 2024-102024-1042 / 41060.6600.03713.63×
2024-11 – 2024-122024-1239 / 41060.6470.01952.06×
2025-01 – 2025-022025-0266 / 41060.5540.01881.17×
2025-03 – 2025-042025-0475 / 41060.5830.03992.18×
2025-05 – 2025-062025-0677 / 41060.5590.02411.28×
Activity-recency clocks
ghclock_only_lgb		LightGBMembargoeddevelopment137600.6210.02621.84×0.642 (0.547–0.773)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.5520.05941.85×
2023-07 – 2023-082023-0890 / 41060.5680.02771.27×
2023-09 – 2023-102023-1067 / 41060.6510.04352.66×
2023-11 – 2023-122023-1255 / 41060.6330.02191.63×
2024-01 – 2024-022024-0240 / 41060.6790.02412.47×
2024-03 – 2024-042024-0422 / 41060.7340.01823.39×
2024-05 – 2024-062024-0623 / 41060.7730.069312.37×
2024-07 – 2024-082024-0832 / 41060.7530.03053.91×
2024-09 – 2024-102024-1042 / 41060.6740.03573.49×
2024-11 – 2024-122024-1239 / 41060.6360.02422.55×
2025-01 – 2025-022025-0266 / 41060.5550.01851.15×
2025-03 – 2025-042025-0475 / 41060.5960.03952.16×
2025-05 – 2025-062025-0677 / 41060.5470.02301.23×
Activity-recency clocks + Install base
ghclock_installs_logistic		Logistic Regressionembargoeddevelopment137600.6200.02902.04×0.635 (0.447–0.782)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.6870.08432.62×
2023-07 – 2023-082023-0890 / 41060.6580.06272.86×
2023-09 – 2023-102023-1067 / 41060.6430.04782.93×
2023-11 – 2023-122023-1255 / 41060.6610.02511.87×
2024-01 – 2024-022024-0240 / 41060.7260.02612.68×
2024-03 – 2024-042024-0422 / 41060.7310.01392.60×
2024-05 – 2024-062024-0623 / 41060.7820.01763.14×
2024-07 – 2024-082024-0832 / 41060.7350.02102.70×
2024-09 – 2024-102024-1042 / 41060.6380.01961.92×
2024-11 – 2024-122024-1239 / 41060.6000.01481.56×
2025-01 – 2025-022025-0266 / 41060.4610.01771.10×
2025-03 – 2025-042025-0475 / 41060.4870.02211.21×
2025-05 – 2025-062025-0677 / 41060.4470.02071.11×
Activity-recency clocks + Contributor dynamics
ghclock_ghdyn_xgb		XGBoostembargoeddevelopment137600.6110.02541.78×0.636 (0.531–0.810)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.5790.05711.77×
2023-07 – 2023-082023-0890 / 41060.5410.04021.83×
2023-09 – 2023-102023-1067 / 41060.5780.02621.61×
2023-11 – 2023-122023-1255 / 41060.6130.03282.45×
2024-01 – 2024-022024-0240 / 41060.6460.02412.47×
2024-03 – 2024-042024-0422 / 41060.7960.02264.22×
2024-05 – 2024-062024-0623 / 41060.8100.062211.10×
2024-07 – 2024-082024-0832 / 41060.7480.04505.78×
2024-09 – 2024-102024-1042 / 41060.6650.03863.77×
2024-11 – 2024-122024-1239 / 41060.6250.01421.49×
2025-01 – 2025-022025-0266 / 41060.5820.02191.36×
2025-03 – 2025-042025-0475 / 41060.5600.02161.18×
2025-05 – 2025-062025-0677 / 41060.5310.02271.21×
Activity-recency clocks + Security vocabulary in project text
ghclock_ghtext_xgb		XGBoostembargoeddevelopment137600.6100.02641.85×0.637 (0.504–0.792)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.5310.05231.63×
2023-07 – 2023-082023-0890 / 41060.5040.02831.29×
2023-09 – 2023-102023-1067 / 41060.5670.02211.35×
2023-11 – 2023-122023-1255 / 41060.6140.02451.83×
2024-01 – 2024-022024-0240 / 41060.7400.02602.66×
2024-03 – 2024-042024-0422 / 41060.7050.04408.20×
2024-05 – 2024-062024-0623 / 41060.7920.03125.57×
2024-07 – 2024-082024-0832 / 41060.7560.05497.05×
2024-09 – 2024-102024-1042 / 41060.6990.04344.24×
2024-11 – 2024-122024-1239 / 41060.6360.05385.66×
2025-01 – 2025-022025-0266 / 41060.5800.02031.26×
2025-03 – 2025-042025-0475 / 41060.5990.02541.39×
2025-05 – 2025-062025-0677 / 41060.5510.02371.27×
Activity-recency clocks + Shared-maintainer contagion
ghclock_contagion_logistic		Logistic Regressionembargoeddevelopment137600.6070.02621.84×0.615 (0.450–0.712)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.7090.09072.82×
2023-07 – 2023-082023-0890 / 41060.6470.03761.72×
2023-09 – 2023-102023-1067 / 41060.6560.03352.05×
2023-11 – 2023-122023-1255 / 41060.6440.02351.75×
2024-01 – 2024-022024-0240 / 41060.6520.01831.88×
2024-03 – 2024-042024-0422 / 41060.6420.01112.06×
2024-05 – 2024-062024-0623 / 41060.7120.01422.54×
2024-07 – 2024-082024-0832 / 41060.6990.02633.37×
2024-09 – 2024-102024-1042 / 41060.5980.02262.21×
2024-11 – 2024-122024-1239 / 41060.5860.02542.67×
2025-01 – 2025-022025-0266 / 41060.5170.02151.34×
2025-03 – 2025-042025-0475 / 41060.4790.02361.29×
2025-05 – 2025-062025-0677 / 41060.4500.02051.09×
Activity-recency clocks + Security vocabulary in project text
ghclock_ghtext_logistic		Logistic Regressionembargoeddevelopment137600.6070.02071.45×0.613 (0.447–0.701)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.6570.05081.58×
2023-07 – 2023-082023-0890 / 41060.6100.02991.37×
2023-09 – 2023-102023-1067 / 41060.6330.02561.57×
2023-11 – 2023-122023-1255 / 41060.6250.01891.41×
2024-01 – 2024-022024-0240 / 41060.6340.01501.54×
2024-03 – 2024-042024-0422 / 41060.6350.01041.95×
2024-05 – 2024-062024-0623 / 41060.7010.01071.91×
2024-07 – 2024-082024-0832 / 41060.6610.01201.54×
2024-09 – 2024-102024-1042 / 41060.6690.02062.02×
2024-11 – 2024-122024-1239 / 41060.6470.01801.90×
2025-01 – 2025-022025-0266 / 41060.5350.02051.27×
2025-03 – 2025-042025-0475 / 41060.5100.02411.32×
2025-05 – 2025-062025-0677 / 41060.4470.01710.91×
Activity-recency clocks + Shared-maintainer contagion
ghclock_contagion_xgb		XGBoostembargoeddevelopment137600.6020.02191.54×0.595 (0.514–0.653)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.5740.05451.69×
2023-07 – 2023-082023-0890 / 41060.5140.02150.98×
2023-09 – 2023-102023-1067 / 41060.6150.03822.34×
2023-11 – 2023-122023-1255 / 41060.5770.01731.29×
2024-01 – 2024-022024-0240 / 41060.5960.01561.61×
2024-03 – 2024-042024-0422 / 41060.6400.02234.16×
2024-05 – 2024-062024-0623 / 41060.6530.02424.33×
2024-07 – 2024-082024-0832 / 41060.6130.01802.31×
2024-09 – 2024-102024-1042 / 41060.5440.01381.35×
2024-11 – 2024-122024-1239 / 41060.6430.02412.54×
2025-01 – 2025-022025-0266 / 41060.6150.02401.49×
2025-03 – 2025-042025-0475 / 41060.6100.03121.71×
2025-05 – 2025-062025-0677 / 41060.5430.02671.42×
Activity-recency clocks + Security vocabulary in project text + Shared-maintainer contagion + Contributor dynamics + Software Heritage visit deltas
honest_all_logistic		Logistic Regressionembargoeddevelopment137600.6020.02601.83×0.590 (0.481–0.704)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.7040.10223.18×
2023-07 – 2023-082023-0890 / 41060.6320.03771.72×
2023-09 – 2023-102023-1067 / 41060.5980.02641.62×
2023-11 – 2023-122023-1255 / 41060.5850.01991.48×
2024-01 – 2024-022024-0240 / 41060.5940.01461.50×
2024-03 – 2024-042024-0422 / 41060.6570.02063.84×
2024-05 – 2024-062024-0623 / 41060.6070.02584.61×
2024-07 – 2024-082024-0832 / 41060.5880.078210.03×
2024-09 – 2024-102024-1042 / 41060.5450.03723.63×
2024-11 – 2024-122024-1239 / 41060.5750.02092.20×
2025-01 – 2025-022025-0266 / 41060.5750.02021.26×
2025-03 – 2025-042025-0475 / 41060.5270.02191.20×
2025-05 – 2025-062025-0677 / 41060.4810.01921.02×
Contributor dynamics
ghdyn_xgb		XGBoostembargoeddevelopment137600.5740.02451.72×0.579 (0.441–0.663)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.5800.06121.90×
2023-07 – 2023-082023-0890 / 41060.5930.05612.56×
2023-09 – 2023-102023-1067 / 41060.6100.04272.62×
2023-11 – 2023-122023-1255 / 41060.6110.03792.83×
2024-01 – 2024-022024-0240 / 41060.5870.02512.57×
2024-03 – 2024-042024-0422 / 41060.6630.01452.71×
2024-05 – 2024-062024-0623 / 41060.5490.04137.37×
2024-07 – 2024-082024-0832 / 41060.5830.04665.98×
2024-09 – 2024-102024-1042 / 41060.6070.02392.34×
2024-11 – 2024-122024-1239 / 41060.5580.01461.54×
2025-01 – 2025-022025-0266 / 41060.6270.02511.56×
2025-03 – 2025-042025-0475 / 41060.5180.01891.03×
2025-05 – 2025-062025-0677 / 41060.4410.01650.88×
Activity-recency clocks + Software Heritage visit deltas
ghclock_swhdelta_logistic		Logistic Regressionembargoeddevelopment137600.5740.01901.34×0.553 (0.458–0.714)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.7140.06892.14×
2023-07 – 2023-082023-0890 / 41060.6500.03861.76×
2023-09 – 2023-102023-1067 / 41060.5780.02311.42×
2023-11 – 2023-122023-1255 / 41060.5540.01451.08×
2024-01 – 2024-022024-0240 / 41060.5680.01091.12×
2024-03 – 2024-042024-0422 / 41060.5490.00581.09×
2024-05 – 2024-062024-0623 / 41060.5760.00721.28×
2024-07 – 2024-082024-0832 / 41060.5580.00901.16×
2024-09 – 2024-102024-1042 / 41060.5270.01251.22×
2024-11 – 2024-122024-1239 / 41060.5290.01141.20×
2025-01 – 2025-022025-0266 / 41060.4620.01570.98×
2025-03 – 2025-042025-0475 / 41060.4680.01730.95×
2025-05 – 2025-062025-0677 / 41060.4580.01680.90×
Install base
installs_xgb		XGBoostembargoeddevelopment137600.5680.01841.29×0.589 (0.529–0.683)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.5960.04371.36×
2023-07 – 2023-082023-0890 / 41060.5780.03021.38×
2023-09 – 2023-102023-1067 / 41060.5940.02061.26×
2023-11 – 2023-122023-1255 / 41060.5720.01541.15×
2024-01 – 2024-022024-0240 / 41060.6120.04514.63×
2024-03 – 2024-042024-0422 / 41060.6620.01703.18×
2024-05 – 2024-062024-0623 / 41060.6830.02163.86×
2024-07 – 2024-082024-0832 / 41060.5740.01031.32×
2024-09 – 2024-102024-1042 / 41060.5420.01461.43×
2024-11 – 2024-122024-1239 / 41060.5780.01861.96×
2025-01 – 2025-022025-0266 / 41060.5660.02541.58×
2025-03 – 2025-042025-0475 / 41060.5290.02191.20×
2025-05 – 2025-062025-0677 / 41060.5750.02251.20×
Advisory recurrence + Activity-recency clocks
advhist_ghclock_clean_xgb		XGBoostembargoeddevelopment137600.5580.01951.37×0.562 (0.508–0.671)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.5380.04981.55×
2023-07 – 2023-082023-0890 / 41060.5130.02491.13×
2023-09 – 2023-102023-1067 / 41060.6010.02681.64×
2023-11 – 2023-122023-1255 / 41060.6000.02501.86×
2024-01 – 2024-022024-0240 / 41060.5730.04234.34×
2024-03 – 2024-042024-0422 / 41060.5630.01192.23×
2024-05 – 2024-062024-0623 / 41060.6710.01512.70×
2024-07 – 2024-082024-0832 / 41060.5740.00891.15×
2024-09 – 2024-102024-1042 / 41060.5470.01271.24×
2024-11 – 2024-122024-1239 / 41060.5710.01191.26×
2025-01 – 2025-022025-0266 / 41060.5080.01651.03×
2025-03 – 2025-042025-0475 / 41060.5380.02211.21×
2025-05 – 2025-062025-0677 / 41060.5100.01941.04×
Activity-recency clocks + Software Heritage visit deltas
ghclock_swhdelta_xgb		XGBoostembargoeddevelopment137600.5560.01891.33×0.530 (0.391–0.613)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.5860.05091.58×
2023-07 – 2023-082023-0890 / 41060.5730.02941.34×
2023-09 – 2023-102023-1067 / 41060.6120.02471.51×
2023-11 – 2023-122023-1255 / 41060.6130.02331.74×
2024-01 – 2024-022024-0240 / 41060.5550.01371.41×
2024-03 – 2024-042024-0422 / 41060.3910.00791.47×
2024-05 – 2024-062024-0623 / 41060.4770.00591.06×
2024-07 – 2024-082024-0832 / 41060.4180.00650.84×
2024-09 – 2024-102024-1042 / 41060.4480.00920.89×
2024-11 – 2024-122024-1239 / 41060.5930.01251.32×
2025-01 – 2025-022025-0266 / 41060.5490.01841.14×
2025-03 – 2025-042025-0475 / 41060.5450.02311.27×
2025-05 – 2025-062025-0677 / 41060.5230.02451.31×
all features in dataset
advisory_only_xgb		XGBoostembargoeddevelopment137600.5530.02451.72×0.565 (0.515–0.623)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.5530.06492.02×
2023-07 – 2023-082023-0890 / 41060.5250.04722.15×
2023-09 – 2023-102023-1067 / 41060.5950.02641.62×
2023-11 – 2023-122023-1255 / 41060.5580.01521.13×
2024-01 – 2024-022024-0240 / 41060.5830.01191.22×
2024-03 – 2024-042024-0422 / 41060.5680.01031.93×
2024-05 – 2024-062024-0623 / 41060.6080.098017.50×
2024-07 – 2024-082024-0832 / 41060.6230.04986.38×
2024-09 – 2024-102024-1042 / 41060.5750.04384.29×
2024-11 – 2024-122024-1239 / 41060.5760.02122.23×
2025-01 – 2025-022025-0266 / 41060.5250.02881.79×
2025-03 – 2025-042025-0475 / 41060.5400.03141.72×
2025-05 – 2025-062025-0677 / 41060.5150.03291.75×
Shared-maintainer contagion
contagion_xgb		XGBoostembargoeddevelopment137600.5510.02031.43×0.539 (0.451–0.632)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.6050.05971.86×
2023-07 – 2023-082023-0890 / 41060.5470.02881.31×
2023-09 – 2023-102023-1067 / 41060.5610.02551.56×
2023-11 – 2023-122023-1255 / 41060.5030.01481.10×
2024-01 – 2024-022024-0240 / 41060.5310.01101.12×
2024-03 – 2024-042024-0422 / 41060.5030.00651.22×
2024-05 – 2024-062024-0623 / 41060.4990.00991.77×
2024-07 – 2024-082024-0832 / 41060.4720.03394.35×
2024-09 – 2024-102024-1042 / 41060.4510.01431.40×
2024-11 – 2024-122024-1239 / 41060.6180.02312.43×
2025-01 – 2025-022025-0266 / 41060.6320.02291.42×
2025-03 – 2025-042025-0475 / 41060.5640.02111.16×
2025-05 – 2025-062025-0677 / 41060.5250.02081.11×
Activity-recency clocks + Security vocabulary in project text + Shared-maintainer contagion + Contributor dynamics + Software Heritage visit deltas
honest_all_xgb		XGBoostembargoeddevelopment137600.5510.01941.36×0.530 (0.408–0.649)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.5740.05351.67×
2023-07 – 2023-082023-0890 / 41060.5620.02901.32×
2023-09 – 2023-102023-1067 / 41060.5600.02611.60×
2023-11 – 2023-122023-1255 / 41060.5350.02261.69×
2024-01 – 2024-022024-0240 / 41060.4080.00840.86×
2024-03 – 2024-042024-0422 / 41060.4260.00811.52×
2024-05 – 2024-062024-0623 / 41060.5370.01442.58×
2024-07 – 2024-082024-0832 / 41060.4700.00750.96×
2024-09 – 2024-102024-1042 / 41060.4680.00960.94×
2024-11 – 2024-122024-1239 / 41060.6490.01631.72×
2025-01 – 2025-022025-0266 / 41060.6310.02341.46×
2025-03 – 2025-042025-0475 / 41060.5620.02421.32×
2025-05 – 2025-062025-0677 / 41060.5150.02501.33×
Advisory history (base panel) + Advisory presence flags (base panel) + Advisory recurrence + Activity-recency clocks
advhist_ghclock_xgb		XGBoostembargoeddevelopment137600.5320.01671.17×0.537 (0.477–0.652)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.5180.03791.18×
2023-07 – 2023-082023-0890 / 41060.4850.02150.98×
2023-09 – 2023-102023-1067 / 41060.5730.02491.53×
2023-11 – 2023-122023-1255 / 41060.5480.01791.34×
2024-01 – 2024-022024-0240 / 41060.5020.02202.26×
2024-03 – 2024-042024-0422 / 41060.5500.00781.46×
2024-05 – 2024-062024-0623 / 41060.6520.03526.29×
2024-07 – 2024-082024-0832 / 41060.5830.01191.53×
2024-09 – 2024-102024-1042 / 41060.5620.01281.25×
2024-11 – 2024-122024-1239 / 41060.5380.01061.12×
2025-01 – 2025-022025-0266 / 41060.4840.01510.94×
2025-03 – 2025-042025-0475 / 41060.5050.01921.05×
2025-05 – 2025-062025-0677 / 41060.4770.01850.99×
Security vocabulary in project text
ghtext_xgb		XGBoostembargoeddevelopment137600.4910.01581.11×0.529 (0.399–0.684)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.4790.03201.00×
2023-07 – 2023-082023-0890 / 41060.3990.01920.88×
2023-09 – 2023-102023-1067 / 41060.4780.02131.30×
2023-11 – 2023-122023-1255 / 41060.5730.01811.35×
2024-01 – 2024-022024-0240 / 41060.5980.01511.55×
2024-03 – 2024-042024-0422 / 41060.5500.00671.26×
2024-05 – 2024-062024-0623 / 41060.5590.01502.67×
2024-07 – 2024-082024-0832 / 41060.6840.02843.65×
2024-09 – 2024-102024-1042 / 41060.5610.02042.00×
2024-11 – 2024-122024-1239 / 41060.5460.01972.07×
2025-01 – 2025-022025-0266 / 41060.5190.01841.14×
2025-03 – 2025-042025-0475 / 41060.4810.01760.96×
2025-05 – 2025-062025-0677 / 41060.4450.01680.89×
Software Heritage visit deltas
swhdelta_xgb		XGBoostembargoeddevelopment137600.4740.01370.96×0.435 (0.301–0.522)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.5220.03861.20×
2023-07 – 2023-082023-0890 / 41060.4850.02100.96×
2023-09 – 2023-102023-1067 / 41060.5090.01911.17×
2023-11 – 2023-122023-1255 / 41060.5200.01471.10×
2024-01 – 2024-022024-0240 / 41060.3800.00790.81×
2024-03 – 2024-042024-0422 / 41060.3430.00440.81×
2024-05 – 2024-062024-0623 / 41060.3040.00430.76×
2024-07 – 2024-082024-0832 / 41060.3010.00540.69×
2024-09 – 2024-102024-1042 / 41060.3390.00740.72×
2024-11 – 2024-122024-1239 / 41060.5160.01081.13×
2025-01 – 2025-022025-0266 / 41060.4930.01500.93×
2025-03 – 2025-042025-0475 / 41060.4630.01670.91×
2025-05 – 2025-062025-0677 / 41060.4780.03101.65×
Advisory recurrence
advhist_only_xgb		XGBoostembargoeddevelopment137600.4350.01340.94×0.424 (0.334–0.493)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.4250.02850.89×
2023-07 – 2023-082023-0890 / 41060.4130.02110.96×
2023-09 – 2023-102023-1067 / 41060.4040.01360.84×
2023-11 – 2023-122023-1255 / 41060.4720.01290.96×
2024-01 – 2024-022024-0240 / 41060.4460.01081.10×
2024-03 – 2024-042024-0422 / 41060.3670.00450.84×
2024-05 – 2024-062024-0623 / 41060.4230.00490.88×
2024-07 – 2024-082024-0832 / 41060.3340.00570.73×
2024-09 – 2024-102024-1042 / 41060.3900.00830.81×
2024-11 – 2024-122024-1239 / 41060.4230.00870.91×
2025-01 – 2025-022025-0266 / 41060.4690.01560.97×
2025-03 – 2025-042025-0475 / 41060.4500.01660.91×
2025-05 – 2025-062025-0677 / 41060.4930.01891.01×
Advisory history (base panel) + Advisory presence flags (base panel)
advfull_xgb		XGBoostembargoeddevelopment137600.4260.01390.97×0.430 (0.363–0.545)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.4320.02790.87×
2023-07 – 2023-082023-0890 / 41060.3680.01670.76×
2023-09 – 2023-102023-1067 / 41060.3630.01240.76×
2023-11 – 2023-122023-1255 / 41060.3850.01160.87×
2024-01 – 2024-022024-0240 / 41060.3690.00870.89×
2024-03 – 2024-042024-0422 / 41060.3730.00480.89×
2024-05 – 2024-062024-0623 / 41060.5130.095116.97×
2024-07 – 2024-082024-0832 / 41060.4840.00961.24×
2024-09 – 2024-102024-1042 / 41060.5450.01301.27×
2024-11 – 2024-122024-1239 / 41060.4200.00810.85×
2025-01 – 2025-022025-0266 / 41060.4540.02511.56×
2025-03 – 2025-042025-0475 / 41060.4340.02591.42×
2025-05 – 2025-062025-0677 / 41060.4550.02141.14×
Advisory history (base panel) + Advisory presence flags (base panel) + Advisory recurrence
advhist_xgb		XGBoostembargoeddevelopment137600.4180.01300.91×0.404 (0.331–0.478)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.4440.03231.01×
2023-07 – 2023-082023-0890 / 41060.4110.02030.93×
2023-09 – 2023-102023-1067 / 41060.3700.01250.77×
2023-11 – 2023-122023-1255 / 41060.3910.01080.81×
2024-01 – 2024-022024-0240 / 41060.3570.00730.75×
2024-03 – 2024-042024-0422 / 41060.3310.00390.73×
2024-05 – 2024-062024-0623 / 41060.4340.091216.28×
2024-07 – 2024-082024-0832 / 41060.3610.00580.75×
2024-09 – 2024-102024-1042 / 41060.4100.00850.83×
2024-11 – 2024-122024-1239 / 41060.3860.00750.79×
2025-01 – 2025-022025-0266 / 41060.4450.01480.92×
2025-03 – 2025-042025-0475 / 41060.4370.01640.90×
2025-05 – 2025-062025-0677 / 41060.4780.01850.99×
Activity-recency clocks + Contributor dynamics
ghclock_ghdyn_logistic_asof		Logistic Regressionembargoeddevelopment sensitivity137600.6430.03102.17×0.652 (0.476–0.814)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.7130.07212.24×
2023-07 – 2023-082023-0890 / 41060.6820.04892.23×
2023-09 – 2023-102023-1067 / 41060.6830.03482.13×
2023-11 – 2023-122023-1255 / 41060.6710.04523.38×
2024-01 – 2024-022024-0240 / 41060.6760.01972.02×
2024-03 – 2024-042024-0422 / 41060.6520.063411.84×
2024-05 – 2024-062024-0623 / 41060.8140.095217.00×
2024-07 – 2024-082024-0832 / 41060.7730.099012.70×
2024-09 – 2024-102024-1042 / 41060.6990.07487.31×
2024-11 – 2024-122024-1239 / 41060.6280.02052.16×
2025-01 – 2025-022025-0266 / 41060.4990.01701.06×
2025-03 – 2025-042025-0475 / 41060.5130.02451.34×
2025-05 – 2025-062025-0677 / 41060.4760.02311.23×
Activity-recency clocks + Contributor dynamics
oot_champion		Logistic Regressionembargoedout-of-time31270.6700.03933.81×0.660 (0.589–0.758)
Per-fold detail (3 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2025-07 – 2025-082025-0840 / 41060.5890.02933.01×
2025-09 – 2025-102025-1031 / 41060.6320.03805.03×
2025-11 – 2025-122025-1256 / 41060.7580.07055.17×
Activity-recency clocks
oot_ghclock_logistic		Logistic Regressionembargoedout-of-time31270.6380.01831.77×0.634 (0.580–0.717)
Per-fold detail (3 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2025-07 – 2025-082025-0840 / 41060.5800.01681.73×
2025-09 – 2025-102025-1031 / 41060.6060.01361.80×
2025-11 – 2025-122025-1256 / 41060.7170.03392.48×
Activity-recency clocks + Install base
oot_runnerup		XGBoostembargoedout-of-time31270.6280.02092.03×0.647 (0.601–0.728)
Per-fold detail (3 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2025-07 – 2025-082025-0840 / 41060.6110.01681.72×
2025-09 – 2025-102025-1031 / 41060.7280.05667.50×
2025-11 – 2025-122025-1256 / 41060.6010.01771.30×
Activity-recency clocks + Contributor dynamics
oot_champion_asof		Logistic Regressionembargoedout-of-time sensitivity31270.6680.03723.61×0.659 (0.586–0.746)
Per-fold detail (3 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2025-07 – 2025-082025-0840 / 41060.5860.02922.99×
2025-09 – 2025-102025-1031 / 41060.6460.03774.99×
2025-11 – 2025-122025-1256 / 41060.7460.06664.88×
Activity-recency clocks + Contributor dynamics
full_curve_champion		Logistic Regressionembargoeddev + out-of-time168870.6480.03122.31×0.653 (0.487–0.825)
Per-fold detail (16 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-062023-06132 / 41060.7050.06732.09×
2023-07 – 2023-082023-0890 / 41060.6720.04912.24×
2023-09 – 2023-102023-1067 / 41060.6570.03322.04×
2023-11 – 2023-122023-1255 / 41060.6540.04453.32×
2024-01 – 2024-022024-0240 / 41060.6670.01911.96×
2024-03 – 2024-042024-0422 / 41060.6620.063411.83×
2024-05 – 2024-062024-0623 / 41060.8250.090116.08×
2024-07 – 2024-082024-0832 / 41060.7950.093211.96×
2024-09 – 2024-102024-1042 / 41060.7010.07667.49×
2024-11 – 2024-122024-1239 / 41060.6160.02022.13×
2025-01 – 2025-022025-0266 / 41060.5040.01731.08×
2025-03 – 2025-042025-0475 / 41060.5260.02511.37×
2025-05 – 2025-062025-0677 / 41060.4870.02311.23×
2025-07 – 2025-082025-0840 / 41060.5890.02933.01×
2025-09 – 2025-102025-1031 / 41060.6320.03805.03×
2025-11 – 2025-122025-1256 / 41060.7580.07055.17×
all features in dataset
advisory_only_xgb_leaky		XGBooststored labelsdevelopment137600.6020.03672.57×0.607 (0.521–0.684)
Per-fold detail (13 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2023-05 – 2023-06—132 / 41060.6310.09703.02×
2023-07 – 2023-08—90 / 41060.6310.08243.76×
2023-09 – 2023-10—67 / 41060.6410.03702.27×
2023-11 – 2023-12—55 / 41060.5660.01621.21×
2024-01 – 2024-02—40 / 41060.6420.01601.64×
2024-03 – 2024-04—22 / 41060.6680.03266.09×
2024-05 – 2024-06—23 / 41060.6090.098517.59×
2024-07 – 2024-08—32 / 41060.6840.085510.97×
2024-09 – 2024-10—42 / 41060.6290.08077.89×
2024-11 – 2024-12—39 / 41060.5840.02342.46×
2025-01 – 2025-02—66 / 41060.5270.02911.81×
2025-03 – 2025-04—75 / 41060.5620.03692.02×
2025-05 – 2025-06—77 / 41060.5210.04052.16×

How to read these numbers

ROC-AUC measures how well the model separates advisory-bound plugins from the rest across all thresholds; 0.5 is chance. Average-precision lift compares ranking concentration against random selection at the ~1.4% pooled base rate. Under the honest protocol the separation signal is real and consistent across model families, while top-of-list concentration remains modest — CANARY is a triage signal, not proof that a component is vulnerable. Feature sets whose pooled ROC-AUC sits at or below 0.5 (for example advisory-recurrence features) are shown deliberately: measuring which signals do not survive an honest evaluation is part of the research contribution.