Specialization drives detection.
On MSD, training raises recall from about 4% for untuned OpenJev to 95.6% at the native 0.5 threshold. The recipe also works from a plain base checkpoint.
SkillScreen
High-recall screening of malicious agent skills
with decision models.
Inside the scorer
Premise + hypothesis
SKILL.mdMD
Fixed hypothesis
This AI agent skill
is malicious.
SkillScreen · 2B
2.19M trainable parametersA continuous score
p(malicious)
τ
No alertReview
Threshold frozen
Set for 99% recall
on validation data
Catch the attacks.
Keep the review load down.
Agent skills can instruct assistants to read files, run commands and access credentials. Legitimate tools and malicious skills often use the same operations.
SkillScreen learns from labelled skills to judge the task in context. A compact decision model reduces false alarms at high recall. Three evaluation protocols test what it learns, how it transfers to new attack structures and what changes with unfamiliar sources.
Learn the task.
Test what transfers.
False positives at 99% recall · MaliciousSkillBench
Benign skills flagged for reviewLower is better
How to read these results
Each protocol trains a fresh model on its own training partition. These comparisons sweep held-out test scores to match a 99% recall target. Operational thresholds are selected separately using validation data and frozen before test scoring.
Both neural models average five seeds on SBR and MSD, with one epoch of training. On SD, SkillScreen averages five seeds and ModernBERT uses five converged runs from six attempts. The study reports every run and additional training budgets.
The paired FPR difference between ModernBERT and SkillScreen is 5.4 percentage points on SBR, with a 95% interval of [−0.1, 15.5]. On MSD, it is 7.8 points [−3.4, 22.1]. On SD, it is 24.1 points [14.4, 32.7]. Model performance depends on the evaluation protocol.
The SD evaluation also includes TypeSafe Jev 1.13 and SkillSpector-static on the same Markdown inputs. Both reach 99% recall only at 100% FPR. Their results are included in the download.
What makes
the difference?
The experiments test the training recipe and where it transfers.
The hypothesis matters at high recall.
Removing the task hypothesis increases FPR at 99% recall by 22.1 percentage points on MSD. Naming the task also helps when labels are scarce.
Benign source shift is the harder test.
SkillScreen retains 97.3% recall at the native threshold on unseen sources. Unfamiliar benign skills raise the false-positive rate to 36.0%, exposing the next challenge for screening.
Explore the work.
Inspect the model or reproduce the experiments.
The paper
Methods, results and ablations
SkillScreen-2B-SBR
Merged weights and the original adapter
Code and experiments
Table 1 reproduction and model reruns
Protocol checkpoints and the experimental 32K package model are available in the SkillScreen organization.