CANARY Web Console

CANARY (Component Analytics & Near-term Advisory Risk Yardstick) predicts near-term security advisory risk for Jenkins plugins using publicly observable project signals. Rather than waiting for a vulnerability to be disclosed, CANARY estimates the likelihood that a plugin will appear in a Jenkins security advisory within the next six months — giving security teams a proactive prioritization signal.

This is a research prototype developed as part of a Doctor of Engineering praxis at The George Washington University. It is intended as a decision-support tool, not a replacement for security review.

1. Click the Score a plugin tab.
2. Type a Jenkins plugin name in the Plugin ID field (autocomplete is populated from the live registry).
3. Click Score plugin.
4. Review the score, its feature drivers, and the plugin's honest track record: where it ranked at every past forecast date and whether an advisory followed. Use Explain now (AI) for a plain-English summary.

The CANARY forecast is the frozen champion configuration applied to the newest month of the panel. Read it as a rank: where the plugin sits among every scored plugin, and what share of the panel scores above it. The raw probability is shown for completeness but is not calibrated to the roughly 1 percent monthly base rate. Across the pre-registered holdout, reviewing the top fifth of plugins by this ranking would have caught about half of the advisories that followed, so a high rank is a reason to look, not a finding that something is wrong.

Beneath the forecast, the honest track record shows the plugin's rank at every past forecast date and whether an advisory followed, and the current-state signals (maintenance, governance, SLSA, dependency advisories) describe the plugin today. Those signals are a practitioner's dashboard and are not part of the validated forecast.

CANARY uses only publicly observable data — no private telemetry or credentials are required. Under honest evaluation the signals that carry forward are about recency of human attention: activity clocks and contributor dynamics from GitHub Archive events, and the plugin's install base. Repository-snapshot and advisory-history signals, which looked strongest under the original protocol, did not survive the label embargo in Jenkins.

Every number on this site comes from one of four evaluation layers, and the tabs are organized around them. The pre-registered success criterion is the same at every layer: pooled ROC-AUC of at least 0.55 with training labels rebuilt from advisories known before each forecast date (the label embargo). 0.50 is chance.

1. Layer 1, chronological time split. One training/test cut with stored labels. This is where the project started and where a label leak inflated the headline (ROC-AUC 0.93 that fell to 0.48 once the labels were rebuilt). Kept on the Layer 1 diagnostics tab as the historical record.
2. Layer 2, group time split. Whole plugins withheld from training, the cold-start question. Also a Layer 1-era, stored-label result.
3. Layer 3, embargoed rolling backtest. Thirteen forecast dates from 2023 to 2025, training labels rebuilt at each one; the sweep on which the configurations were chosen. On the Results and Explore tabs.
4. Layer 4, pre-registered out-of-time holdout. Three later forecast dates whose months had not been collected when the configurations and the criterion were frozen; each run once, the first recorded run is the result. The official test, on the Results and Explore tabs.

The honest result is weak but measurable discrimination: the champion configuration held ROC-AUC 0.670 on the holdout, and reviewing the top fifth of plugins by score would have caught about half the advisories that followed. That is a triage signal, not proof that a component is vulnerable.

• CANARY is scoped to the Jenkins plugin ecosystem only — it does not score npm, PyPI, Maven, or other package registries. The PyPI cross-check tab replicates the validation protocol on PyPI to test whether the method travels; it is not a PyPI risk model.
• Scores reflect near-term advisory likelihood, not exploitability or severity in your specific environment.
• A low score does not mean a plugin is safe — it means CANARY sees no strong signal of an imminent advisory based on publicly observable data.
• Data is updated periodically, not in real time. Always consult the official Jenkins security advisories for the authoritative source.

The full source code, data pipeline, and research documentation are available on GitHub. The praxis document provides a detailed description of the methodology, ablation results, and future research directions.