Introducing the unsafe-eval Leaderboard: comparing models for AI safety

Today we're opening the unsafe-eval leaderboard: 35 models in 43 configurations, evaluated as safety guardrails on 8 public benchmarks and under the same setting. It includes models from 300M-parameter classifiers to the latest frontier models from OpenAI, Anthropic and Google, and reports cost, latency and refusals next to accuracy.

An increasingly common way to build a guardrail is to have a separate AI model (with limited access to sensitive data) that analyzes each user request and decides whether that request is safe to pass to the main AI model (with complete access to the harness). It's quick to set up, but then how do you trust that the current configuration is the best one? And what does it mean to be the "best"? Accuracy is definitely important but, when using LLMs, SLMs and decision models, there are other factors to consider: What's the cost? What's the latency? Can you run it on premise? What's the refusal rate?

Unsafe-eval leaderboard aims to provide a partial answer to these questions, with several model families evaluated on over 10,000 prompts. Unsafe-eval is only a starting point of a larger plan: multilinguality, multi-turn scenarios, response safety, advanced safety policies, use case-specific instances, and much more is needed for a more comprehensive evaluation. However, if you are lost on what model to use as a prompt-safety guardrail, unsafe-eval is the best place to start, to the best of our knowledge.

The task: prompt safety classification

A prompt-safety guardrail sits between your users and your AI assistant, AI agent, or AI model. It reads every incoming request and decides whether it's safe to pass on, before the assistant processes the request and generates a response. On the leaderboard, we ask each model to do exactly that: classify user prompts as safe or unsafe under a safety policy, which can be a general one or specific to an application/domain. We are considering expanding unsafe-eval to other safety-related tasks in the future.

Current benchmarks in unsafe-eval

We chose eight public benchmarks to cover the different kinds of requests a guardrail might encounter. Aegis 2.0, OpenAI Moderation and WildGuardMix cover a broad range of harmful content, and WildGuardMix also includes adversarial prompts written to slip past a filter. ToxicChat collects real prompts that users sent to a public chatbot, so it's the closest to the traffic a guardrail sees in production. HarmBench and SimpleSafetyTests contain only harmful requests. CoSApien and DynaGuardrail add another consideration: whether a request is safe depends on the application's own policy. Describing a fight in detail is fine for a fiction-writing assistant, and out of place in a customer-support bot.

Reading the leaderboard

The Overview tab reports the following for each model configuration:

• Balanced accuracy (BA): The average of unsafe recall and safe recall, giving equal weight to catching harmful requests and allowing legitimate ones through. We report the mean across the six benchmarks containing both safe and unsafe prompts.
• Unsafe F1: Combines unsafe recall with precision, measuring how well a model catches harmful requests while avoiding false alarms. We average this score across the same six benchmarks.
• Refusal rate: The share of prompts the model declines to classify. We count refusals as unsafe predictions when computing accuracy and F1.
• Latency: The median time to return a verdict.
• Cost: The price per 1,000 prompts. We leave this blank for self-hosted models, where cost depends on specific hardware and setup.
• Output tokens: The total number of tokens generated during the evaluation, including reasoning tokens.
• Type: Whether the model is general-purpose or trained specifically for guardrailing.
• Parameters: The model's parameter count, where available.

Each configuration has its own row, with reasoning settings or requested outputs next to the model name. The table also shows weight availability, supports sorting and lets you filter for models with open weights.

The Balanced Accuracy tab breaks the overall score, labelled BA in the Overview, into separate columns for the six benchmarks containing both safe and unsafe prompts, alongside their average. Each score gives equal weight to catching unsafe requests and allowing safe ones through. This breakdown can reveal a weaker result hidden by the average, such as poor performance on ToxicChat's real user prompts.

Unsafe F1 combines unsafe recall with precision: of the prompts the model flags, how many are actually unsafe? A high F1 score requires catching harmful requests while keeping false alarms low. The Unsafe F1 tab reports this score for each of the same six benchmarks and their average, which also appears in the Overview. Unlike balanced accuracy, F1 depends on the proportion of unsafe prompts, so we'd compare models within each benchmark before drawing conclusions from differences across benchmarks.

The Unsafe Recall tab covers HarmBench and SimpleSafetyTests, with a column for each and their average. These benchmarks contain only unsafe prompts, so the scores tell us how many harmful requests each model catches. They can't tell us how often it would block a legitimate request.

We'd be reluctant to make users wait longer for a safety check unless it bought us better decisions. The Chart view makes that comparison easier to see. In the balanced accuracy plot below, models with similar scores spread across very different response times. Each dot is a configuration, with higher accuracy towards the top and lower latency towards the left. The time axis is logarithmic, so equal steps represent multiples of latency.

The line traces the Pareto frontier: among the configurations shown, moving to a more accurate model than one on this line means accepting a longer wait. These are useful candidates to consider first, though we'd still need to check their cost and performance on the traffic we expect. Colours distinguish model types, while filled dots show local runs and hollow dots show hosted APIs. The chart also supports unsafe F1 and unsafe recall for the same comparison.

Overall ranking: closed models lead on accuracy

If you look only at accuracy, closed models dominate the leaderboard, with eight of the top ten entries. The usual suspects are all there. GPT-5.6 takes three spots, and Claude Opus 5.5 ties for third. They get there with nothing more than a prompt: we describe which requests to allow and which to block, and ask the model to judge each incoming request.

The gaps at the top are small, though. The whole top ten spans just 1.5 points of balanced accuracy, and most of those differences aren't statistically significant. In practice, the top of the leaderboard is a tie, so choosing among these models comes down to everything else.

That's where the caveats come in. The most obvious one is cost. Opus 5.5 and GPT-5.6 Luna score the same, but Opus costs $5.35 per thousand prompts and Luna costs $0.15, about 36 times less. A guardrail checks every request, so that difference adds up quickly. Latency matters for the same reason: frontier models take between 1.3 and 3 seconds, and the user waits that long before the assistant can even start to respond. Then there are refusals. These are general-purpose models, so their providers decide what they'll agree to look at, and some of them decline to classify exactly the harmful requests a guardrail is there to catch. Opus 5.5 is a notable case, and we come back to it in the refusals section.

Decision models are emerging as a strong alternative. TypeSafe's Jev (closed) and Cloudflare's Clef 27B (open weights) match the frontier models on accuracy and answer in a fraction of the time, at 368 ms and 163 ms. Jev also costs $0.03 per thousand prompts, roughly 180 times less than Opus 5.5.

Our upcoming ScopeGuard-2 models belong to the group of state-of-the-art accuracy at a fraction of cost/latency. ScopeGuard-2 Pro takes first place, while ScopeGuard-2 4B comes seventh with a median latency of 34 ms, the fastest in the top ten. Pro will be available through our API (or licensed for on-prem deployments) and 4B as open weights. We'll cover both in the next post.

Cost: paying more doesn't always buy more accuracy

Suppose we're choosing a guardrail for an assistant that receives one million requests a day. Every one of those requests needs a safety check, including all the ordinary questions the assistant should simply answer. At that volume, even a small difference in the price per check can add up over a year.

* These estimates use individual prompts. Production requests may include conversation history and longer safety policies, increasing the cost per request.

If we looked only at balanced accuracy, GPT-5.6 Luna and Claude Opus 5.5 would be tied. But for our assistant, choosing Opus would take the annual guardrail bill from about $55K to $1.95M, roughly 36 times as much. We see a similar pattern within Anthropic's own models: Haiku 5.5 and Sonnet 5.5 have the same balanced accuracy, while Sonnet costs about 19 times more. In these comparisons, paying for the more expensive model doesn't improve accuracy.

Part of what we're paying for is the reasoning before the answer. Sometimes, we only need a safety label, but a model may generate many tokens before returning such label, and those tokens count towards the bill too. Gemini 3.8 Flash, for example, generates about 180 output tokens per prompt, with reasoning accounting for 95% of them. In other words, we would pay for roughly 170 tokens of thinking to get a one-word answer. Decision models return the label directly, which is a large part of why they are so much cheaper.

Latency: LLMs are prohibitive for real-time applications

Beyond accuracy and cost, another factor to consider is guardrail latency, i.e., the time the guardrail takes to check a request before the AI assistant can start answering. Sometimes latency even becomes the most important factor, e.g., in real-time applications. Consider a voice AI agent for customer support: international telecommunications standards recommend keeping one-way audio delay below 400 milliseconds (ms), and note that highly interactive conversations are affected by much lower delays [source]. In fact, those 400 ms also have to cover speech recognition, the assistant's reply and speech synthesis. Therefore, a "slow" guardrail can make the entire system feel slow, sometimes too slow.

Take Jev and Claude Opus 5.5 again. On accuracy they look interchangeable, but Jev's median response time is 368 ms and Opus's is 3 seconds, so Jev is about 8 times faster. Our ScopeGuard-2 models go further. Pro answers in 118 ms, roughly 3 times faster than Jev, with balanced accuracy 0.3 points higher. ScopeGuard-2 4B takes 34 ms, about 44 times faster than GPT-5.6 Luna, for a drop of 0.4 points in balanced accuracy.

We'd also want to know how long users might wait when a check takes longer than usual. The table reports p95, the time within which 95% of checks finish, alongside the median. Jev stays below a second at 626 ms, while Claude Opus 5.5 reaches 10.2 seconds. With Opus, about one in twenty checks took longer than 10.2 seconds, before the assistant could even start answering.

Opus isn't the only frontier model with these longer waits. Apart from Claude Haiku 4.5, which runs without reasoning and has a p95 of 1.1 seconds, every OpenAI, Anthropic or Google model we evaluated has a p95 of at least 3 seconds. So while several of these models look similar in the accuracy ranking, users could have quite a different experience depending on which one we choose.

These timings include the network round trip for hosted models, and they depend on how busy each provider was when we ran the evaluation. We ran the open-weight models on a single NVIDIA GeForce RTX 5090, so the timings may differ with your own hardware and traffic.

Refusals: newer frontier models are harder to use as guardrails

Beyond accuracy, cost and latency, another factor to consider is refusals, i.e., how often a model declines to classify a request at all. Hosted models sometimes refuse to process a prompt under their own safety policies, even when we're only asking them to identify harmful content. For our assistant, that leaves us without a clear indication of whether the request should go through. Therefore, the model provider interferes in what is considered safe or unsafe, even when we've supplied our own rules.

In fact, newer models refuse more often than their predecessors, especially at Anthropic. Claude Opus 5.5 refuses 10.1% of unsafe prompts, up from 7.8% for Opus 5. Claude Sonnet 5.5 refuses 6.1%, more than twice the 2.7% of Sonnet 5. Even Claude Haiku 5.5 refuses 1.3%, while Haiku 4.5 refused none. Gemini 3.8 Flash also refuses 3.7% of unsafe prompts, whereas GPT models stay at 1.3% or less. In other words, at least for Anthropic, each new generation is harder to use as a guardrail than the one before.

These refusals happen mostly on unsafe requests. With Claude Opus 5.5, 88% of refused prompts are labelled unsafe. On HarmBench, which contains explicit requests for harmful content, it refuses 30.6% of prompts, leaving us without a safe-or-unsafe label on nearly a third of the benchmark. Note that we're asking the model to recognize the harm in these requests, not to carry them out. Therefore, it won't tell us whether those requests are safe or unsafe.

To include those refusals in the ranking, we need to decide how to score them. We count a refusal as an unsafe prediction, on the assumption that an application would block a request its guardrail declined to classify. Since most refusals happen on unsafe prompts, the model still gets credit for blocking them. However, if we instead counted refusals as errors, Claude Opus 5.5's balanced accuracy would fall from 87.6 to 84.2, i.e., below Nemotron-3.5-Content-Safety (85.5), a 4B open-weight classifier.

Blocking an unsafe request is, of course, the outcome we want. But a refusal doesn't give us the safety label we asked for, nor does it tell us which rule in our policy the request violated. Moreover, refusals also happen on legitimate requests: Opus 5.5 refuses 0.9% of safe prompts. If we block those too, almost one user request in a hundred that should have gone through gets turned away.

This becomes particularly relevant when we define our own safety policy. DynaGuardrail supplies a custom policy with each prompt, yet Opus 5.5 refuses 5.3% of prompts, including 11.0% of those labelled unsafe. Supplying our rules therefore doesn't guarantee that we'll get a decision under those rules.

We see this as a serious limitation of using frontier models as guardrails, and, at least for Anthropic, it is getting worse with each release. Recognizing harmful content under the policy we've supplied is exactly the job we're asking them to do. When they refuse, we're left to handle the request without that assessment. If you're building an application around one of these models, the provider's willingness to classify your traffic deserves as much attention as its accuracy.

Real user traffic: task-specific models lead on chatbot conversations

Beyond the overall ranking, another question to consider is how these models handle the requests people actually send to a conversational assistant, e.g., a customer-facing chatbot. ToxicChat gives us a chance to look at that: its prompts are collected from real user conversations, and each user request is manually labeled as safe or unsafe. In fact, while the models are close in the overall ranking, on this benchmark the differences are much larger, and the order changes.

Here, specialized guardrail models are leading. Every one of them scores above 89 in balanced accuracy, from Gliguard-LLMGuardrails-300M, a 300M-parameter model, at 89.7, up to Qwen3Guard-Gen-4B at 92.6. Our ScopeGuard-2 models are also near the top, with Pro at 92.2 and 4B at 91.4. Hosted frontier models, on the other hand, score between 75.6 and 87.4, and even the best of them, GPT-5.6 Terra, falls below every specialized guardrail. Therefore, on real chatbot traffic, a model much smaller than any frontier LLM can do a better job than the models at the top of the overall ranking.

To understand what that gap means for our assistant, it helps to look at which requests the models let through. Claude Sonnet 5.5 allows 98 to 99% of safe prompts through, as do Gemini 3.8 Flash and Claude Opus 5.5. If we only looked at how often legitimate users get blocked, those results would look good. But we also need the guardrail to catch harmful requests, and here the frontier models fall behind: Sonnet catches 52.5% of unsafe prompts, Gemini 53.6% and Opus 70.7%, while the specialized guardrails catch above 84% of the unsafe prompts. In other words, Sonnet and Gemini label almost half of the unsafe requests as safe and pass them on to the assistant.

Moreover, Opus's 70.7% may overstate how well it classifies these requests. As explained in the refusals section, we count a refusal as an unsafe prediction, so part of that figure may come from Opus refusing to judge a prompt rather than identifying it as unsafe.

Model upgrades: newer isn't always better for guardrailing

A newer model isn't necessarily better at classifying requests as safe or unsafe. In fact, in the configurations we evaluated, GPT-6.1 Sol scores slightly below GPT-5.6 Sol in balanced accuracy (86.8 vs. 87.3), while taking twice as long to respond, i.e., 2.8 seconds at the median compared with 1.4 seconds, even though it costs about half as much. Similarly, Anthropic's newer models gain a little accuracy over their predecessors, e.g., Claude Opus 5.5 scores 87.6 against 87.1 for Opus 5, but, as we saw in the refusals section, they also refuse more often.

Note that reasoning settings also differ across some of these comparisons (we use the default setting recommended by the model provider for each model version), so the version number alone doesn't explain the results. Therefore, for our assistant, we'd evaluate an upgrade just as we would a new model, checking whether it improves the safety decisions and what it changes about cost, latency and refusals.

Choosing a guardrail

The leaderboard can help us choose a few models to try. For our assistant, we'd start with the cost and latency we can afford. Several models have similar accuracy, so there's room to choose one that fits those limits.

Then we'd try them on requests from our own users, in the languages they use and under our own safety policy. We'd want to see which harmful requests get through and look closely at the legitimate questions that get blocked. If those include common support questions, a model that scores well on the leaderboard could still turn away customers who need help.

The choice also depends on what we need back from the guardrail. A label might be enough to block a request, but someone reviewing that decision may need a category or an explanation. That's another thing to check when trying the models, along with how often they refuse to classify a request at all.

Limitations

We've evaluated how models classify incoming user prompts. These results don't tell us how well they would check an assistant's replies or judge a longer conversation. Most of the benchmarks are also in English, so we'd need more evaluation before drawing conclusions about other languages.

We use public benchmarks, which means some models may have seen these examples during training. For closed-source models, we can't inspect the training data to check for that overlap. Cost and latency also reflect the providers' prices and load when we ran the evaluations in October 2026, so the numbers may change over time.

What's next

We'll have more to share about ScopeGuard-2 soon, along with unsafe-eval, the framework we used to evaluate these models.

You can explore the results on the leaderboard. If there's a model you'd like us to evaluate, suggest it through the Request a model tab.