Web interface

CANARY Web Console

Security-advisory forecasting for Jenkins plugins from public signals, evaluated honestly: four validation layers, one pre-registered criterion, and every number on this site traceable to the layer that produced it.

Explore

Pick a validation layer and a configuration

Configurations frozen before the months were collected, run once. The official test: nothing here was chosen after seeing the result.

Embargoed labels at every fold

Changing the layer reloads the configuration list; the fold filter applies to the case study below.

Readable metrics

Activity-recency clocks + Contributor dynamics (Logistic Regression)

oot_champion · 3 folds, 12,318 test rows, 127 advisory outcomes; pooled metrics combine every fold's predictions.

Pooled ROC-AUC0.670
Pooled avg precision0.0393
Lift over base rate3.81×
Fold ROC range0.589 – 0.758
Per-fold detail (3 folds)
Test windowLabels as-ofPositives / rowsROC-AUCAvg precisionAP lift
2025-07 – 2025-082025-0840 / 41060.5890.02933.01×
2025-09 – 2025-102025-1031 / 41060.6320.03805.03×
2025-11 – 2025-122025-1256 / 41060.7580.07055.17×

Feature drivers

What moved the score, and how consistently

Logistic-regression coefficients on standardized features, averaged over the 3 folds; the bar is the mean magnitude and the last column says how many folds agreed on the sign. A driver that flips sign between folds is not a stable signal. Features are hover-documented.

FeatureMagnitudeMean coefficientDirection
ghdyn_active_actors_12m+3.21raises risk in 3 of 3 folds
ghdyn_new_actors_12m-3.10lowers risk in 3 of 3 folds
ghclock_days_since_human_push-0.70lowers risk in 3 of 3 folds
ghclock_days_since_any_push+0.48raises risk in 3 of 3 folds
ghclock_days_since_tag_create-0.22lowers risk in 3 of 3 folds
ghdyn_active_actors_3m+0.21raises risk in 3 of 3 folds
ghclock_days_since_pr_opened-0.20lowers risk in 3 of 3 folds
ghdyn_has_actors-0.14lowers risk in 3 of 3 folds
ghdyn_top_actor_share_12m+0.14raises risk in 3 of 3 folds
ghclock_days_since_release+0.13raises risk in 3 of 3 folds
ghdyn_departed_actors_12m-0.12lowers risk in 3 of 3 folds
ghclock_days_since_pr_review+0.11raises risk in 3 of 3 folds
ghdyn_single_actor_12m-0.09lowers risk in 3 of 3 folds
ghclock_days_since_pr_merged+0.06raises risk in 3 of 3 folds
ghdyn_turnover_rate_12m+0.05raises risk in 3 of 3 folds
ghclock_has_events-0.05lowers risk in 3 of 3 folds
ghdyn_events_12m-0.01lowers risk in 2 of 3 folds
ghclock_days_since_issue_opened-0.01lowers risk in 3 of 3 folds

What the holdout ranking looks like

Advisories caught vs plugins reviewed, out-of-time

Reviewing the top 20% of plugins by score would have caught 52% of the advisories that followed across the holdout folds (by fold: 42%, 52%, 61%), against a base rate of 1.0%. The thick line is the pooled ranking over every holdout fold; thin lines are the individual folds (12,318 plugin-months, 127 advisory outcomes). The dashed diagonal is random order. Right: the ROC curve of each fold for the first configuration.

0%0%20%20%40%40%60%60%80%80%100%100%random orderActivity-recency clocks + Contributor dynamics · Logistic Regression review top 1% of plugins → 6% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 2% of plugins → 11% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 3% of plugins → 20% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 5% of plugins → 28% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 8% of plugins → 35% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 10% of plugins → 39% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 15% of plugins → 45% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 20% of plugins → 52% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 25% of plugins → 54% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 30% of plugins → 57% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 40% of plugins → 63% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 50% of plugins → 69% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 60% of plugins → 72% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 70% of plugins → 78% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 80% of plugins → 86% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 90% of plugins → 96% of advisoriesActivity-recency clocks + Contributor dynamics · Logistic Regression review top 100% of plugins → 100% of advisoriestop 10% → 39% caughttop 20% → 52% caughttop 30% → 57% caughtshare of plugins reviewedActivity-recency clocks + Contributor dynamics · Logistic Regression
0.00.00.20.20.40.40.60.60.80.81.01.0false-positive rate2025-07 (40 outcomes) · AUC 0.5892025-09 (31 outcomes) · AUC 0.6322025-11 (56 outcomes) · AUC 0.758

Out-of-time outcomes

Top-25 per holdout fold vs. what followed

Activity-recency clocks + Contributor dynamics · Logistic Regression · oot_champion. Each plugin's best month inside the fold is shown; training labels were rebuilt from advisories known before the fold's forecast date, and the folds were run once.

Pre-registered

Across the holdout folds shown, 5 of the 75 top-25 slots were followed by an advisory within six months. That is the honest precision at 25 for this configuration: a ranking signal with real lift over the base rate, not an operational triage list.

Holdout fold 2025-07 (1 of 25 confirmed)

Scored: 2025-07 and 2025-08  |  Labels known as of: 2025-08  |  Fold ROC-AUC: 0.589  |  Top-25 precision: 1/25 (4%)  |  Base rate: 1.0% (4.1×)
#PluginScoredScoreConfirmedSeverityAdvisory dateLead timeAdvisory
11git-client2025-0779.2%✓Medium (5.0)2025-12-10162 daysSECURITY-3614
The other 24 of the top 25
#PluginScoredScoreConfirmedSeverityAdvisory dateLead timeAdvisory
1ec22025-0888.7%–no advisory in the six-month window
2pipeline-graph-view2025-0887.0%–no advisory in the six-month window
3github-branch-source2025-0885.9%–no advisory in the six-month window
4cloudbees-folder2025-0785.6%–no advisory in the six-month window
5kubernetes2025-0884.8%–no advisory in the six-month window
6cloudbees-bitbucket-branch-source2025-0783.8%–no advisory in the six-month window
7google-compute-engine2025-0782.3%–no advisory in the six-month window
8git2025-0780.9%–no advisory in the six-month window
9datadog2025-0880.4%–no advisory in the six-month window
10junit2025-0779.7%–no advisory in the six-month window
12oic-auth2025-0878.5%–no advisory in the six-month window
13credentials2025-0778.3%–no advisory in the six-month window
14configuration-as-code2025-0778.2%–no advisory in the six-month window
15workflow-cps2025-0877.8%–no advisory in the six-month window
16branch-api2025-0777.3%–no advisory in the six-month window
17active-directory2025-0776.8%–no advisory in the six-month window
18workflow-job2025-0776.7%–no advisory in the six-month window
19pipeline-model-api2025-0776.6%–no advisory in the six-month window
20pipeline-model-declarative-agent2025-0776.6%–no advisory in the six-month window
21pipeline-model-definition2025-0776.6%–no advisory in the six-month window
22pipeline-model-extensions2025-0776.6%–no advisory in the six-month window
23pipeline-stage-tags-metadata2025-0776.6%–no advisory in the six-month window
24lockable-resources2025-0876.3%–no advisory in the six-month window
25badge2025-0775.8%–no advisory in the six-month window

Holdout fold 2025-09 (2 of 25 confirmed)

Scored: 2025-09 and 2025-10  |  Labels known as of: 2025-10  |  Fold ROC-AUC: 0.632  |  Top-25 precision: 2/25 (8%)  |  Base rate: 0.8% (10.6×)
#PluginScoredScoreConfirmedSeverityAdvisory dateLead timeAdvisory
2github-branch-source2025-1089.3%✓Medium (4.3)2026-04-29210 daysSECURITY-3702
16git-client2025-0975.5%✓Medium (5.0)2025-12-10100 daysSECURITY-3614
The other 23 of the top 25
#PluginScoredScoreConfirmedSeverityAdvisory dateLead timeAdvisory
1ec22025-1091.5%–no advisory in the six-month window
3pipeline-graph-view2025-1087.6%–no advisory in the six-month window
4cloudbees-folder2025-0984.7%–no advisory in the six-month window
5cloudbees-bitbucket-branch-source2025-0983.9%–no advisory in the six-month window
6google-compute-engine2025-0982.5%–no advisory in the six-month window
7datadog2025-0981.7%–no advisory in the six-month window
8kubernetes2025-0980.3%–no advisory in the six-month window
9workflow-cps2025-0978.9%–no advisory in the six-month window
10ldap2025-0978.3%–no advisory in the six-month window
11active-directory2025-0977.6%–no advisory in the six-month window
12git2025-0977.3%–no advisory in the six-month window
13oic-auth2025-1077.0%–no advisory in the six-month window
14junit2025-0976.7%–no advisory in the six-month window
15azure-vm-agents2025-1076.3%–no advisory in the six-month window
17branch-api2025-1075.5%–no advisory in the six-month window
18support-core2025-1074.9%–no advisory in the six-month window
19pipeline-model-api2025-0974.6%–no advisory in the six-month window
20pipeline-model-declarative-agent2025-0974.6%–no advisory in the six-month window
21pipeline-model-definition2025-0974.6%–no advisory in the six-month window
22pipeline-model-extensions2025-0974.6%–no advisory in the six-month window
23pipeline-stage-tags-metadata2025-0974.6%–no advisory in the six-month window
24lockable-resources2025-1074.1%–no advisory in the six-month window
25credentials2025-0973.9%–no advisory in the six-month window

Holdout fold 2025-11 (2 of 25 confirmed)

Scored: 2025-11 and 2025-12  |  Labels known as of: 2025-12  |  Fold ROC-AUC: 0.758  |  Top-25 precision: 2/25 (8%)  |  Base rate: 1.4% (5.9×)
#PluginScoredScoreConfirmedSeverityAdvisory dateLead timeAdvisory
2github-branch-source2025-1187.1%✓Medium (4.3)2026-04-29179 daysSECURITY-3702
6workflow-cps2025-1280.6%✓Medium (4.3)2026-06-24205 daysSECURITY-3677
The other 23 of the top 25
#PluginScoredScoreConfirmedSeverityAdvisory dateLead timeAdvisory
1ec22025-1191.5%–no advisory in the six-month window
3pipeline-graph-view2025-1284.7%–no advisory in the six-month window
4cloudbees-bitbucket-branch-source2025-1182.2%–no advisory in the six-month window
5cloudbees-folder2025-1181.3%–no advisory in the six-month window
7junit2025-1179.4%–no advisory in the six-month window
8kubernetes2025-1279.2%–no advisory in the six-month window
9google-compute-engine2025-1176.1%–no advisory in the six-month window
10datadog2025-1175.9%–no advisory in the six-month window
11pipeline-model-api2025-1275.8%–no advisory in the six-month window
12pipeline-model-declarative-agent2025-1275.8%–no advisory in the six-month window
13pipeline-model-definition2025-1275.8%–no advisory in the six-month window
14pipeline-model-extensions2025-1275.8%–no advisory in the six-month window
15pipeline-stage-tags-metadata2025-1275.8%–no advisory in the six-month window
16credentials2025-1175.1%–no advisory in the six-month window
17branch-api2025-1174.2%–no advisory in the six-month window
18pipeline-maven2025-1274.1%–no advisory in the six-month window
19pipeline-maven-api2025-1274.1%–no advisory in the six-month window
20pipeline-maven-database2025-1274.1%–no advisory in the six-month window
21gradle2025-1274.1%–no advisory in the six-month window
22azure-vm-agents2025-1273.6%–no advisory in the six-month window
23support-core2025-1273.5%–no advisory in the six-month window
24scm-api2025-1272.4%–no advisory in the six-month window
25oic-auth2025-1171.8%–no advisory in the six-month window

Advisory details come from the local Jenkins advisory dataset; a confirmed row without details is one whose stored label was positive. Lead time = days from the first of the scored month to publication; the window is the six calendar months that follow, so it can exceed 180 days. Plugin names link to their score page, which shows the plugin's rank at every forecast date.