Introducing unsafe-eval: a framework for AI safety evaluation

TL;DR

• We're open-sourcing unsafe-eval on GitHub, a framework for testing how well LLMs work as safety guardrails, from purpose-built guardrail models to general-purpose LLMs.
• It comes with 8 widely used safety benchmarks, including HarmBench, ToxicChat, WildGuardTest, and policy-specific ones like CoSApien and DynaGuardrail. It's built on Inspect AI and designed to be extended with your own models and benchmarks.
• It measures quality and cost together. By default, every run reports balanced accuracy, unsafe F1, refusal rates, cost, and latency, so you can see what a guardrail costs you in time and money, not only how accurate it is.
• It works with the models you already use, whether they run locally (e.g., via vLLM) or through the main providers, such as OpenAI, Anthropic, and Google, as well as aggregators like OpenRouter.
• It's easy to run. Install it with pip, run an evaluation with one command, and generate an HTML report with another. You can also browse our results on the public leaderboard.

Star unsafe-eval on GitHub

Whenever a new model comes out, someone in our office wants to try it. At Principled Intelligence, we build guardrail models ourselves, so we're curious to see how a new model fares against existing ones, including our own. After trying a few prompts, we usually want to know whether that first impression holds up. Does it catch unsafe requests consistently? How often does it block legitimate ones? And if we used it to check every incoming request, how much time and money would those checks take? To help us answer those questions, we built unsafe-eval.

If you read our recent article on Jev for safety guardrailing, you've already seen unsafe-eval in use. We used it to compare Jev with other models across six safety benchmarks. We wanted to make it easier for others to run these comparisons too, so we're sharing the framework we built. You can see how we ran our experiments and try it with models you're curious about.

We use unsafe-eval to see how well different models work as guardrails. We ask each one to classify user prompts as safe or unsafe under a (potentially custom/domain-specific) safety policy. Asking every model to classify the same prompts sounds straightforward, but each model has its own way of doing the job. One might accept a safety policy as part of its instructions, while another uses a fixed set of categories. Some return a label; others give an explanation. So even when we want to ask the same question, we need to account for how each model receives it and how we read its answer.

A note for experts and researchers: unsafe-eval is not the first collection of safety benchmarks. We acknowledge that other efforts exist. Unsafe-eval is our opinionated way of evaluating guardrails (and general-purpose models as guardrails). More importantly, it's not just a collection: it's opinionated in the selection of benchmarks that are still relevant; it's opinionated in how we measure performance, cost, and latency; it's opinionated in how the task is framed. It's the most transparent way of showing how we think about (a small part of) AI Safety and our view of the current state of the field.

How unsafe-eval works

We wanted to avoid repeating that manual prompting and inspection whenever we tried another model. With unsafe-eval, we've brought the setup into a shared framework built on Inspect AI, an open-source tool for evaluating language models. Inspect runs the experiments and logs the results. We've added the model instructions and benchmark scoring rules needed to evaluate safety guardrails. It also provides ready-to-run configurations, making it easier to get started without assembling each evaluation from scratch. Each run records the data, prompts, and settings so revisiting and reproducing the experiment is also easy.

To get started, you can choose from the configurations we've included for eight safety benchmarks. You might begin with one model on one benchmark, then add other models to see how they compare. You can also extend unsafe-eval with your own models and benchmarks. These evaluations remain compatible with Inspect AI.

Benchmark coverage

We chose the benchmarks to cover different kinds of requests a guardrail might encounter. Some, such as HarmBench and SimpleSafetyTests, contain only harmful requests. ToxicChat includes real user prompts from chatbot interactions, giving us both safe and unsafe examples. CoSApien and DynaGuardrail introduce another consideration: whether a request is safe depends on a custom or domain-specific policy (e.g., suggesting a punch could be safe in a video game, while unsafe when giving advice on relationships).

Looking at the results across these benchmarks helps us understand where a model struggles, before we turn to its overall score.

Explore the leaderboard

If you're curious how the models performed, our leaderboard is a good place to start. We've published the results of our runs there, with scores for each benchmark alongside cost and latency. We walk through the main findings in our leaderboard announcement.

Suppose you're choosing a guardrail for a customer support assistant. The guardrail will check every incoming request, so its speed and cost matter across all the traffic you serve. You'll also want to see how often it misses unsafe requests or blocks customers who are simply asking for help. The leaderboard brings those measurements together so you can compare the models you're considering.

Your choice may also depend on what you need the model to return. A safety label might be enough to block a request, but someone reviewing that decision may need an explanation. That's why we include different configurations of the same model. With ScopeGuard, for example, we compare classification alone with outputs that include reasoning or evidence, measuring how those choices affect both classification performance and the time each check takes.

We'll keep adding results as we evaluate more models. If you're curious about one we haven't evaluated yet, you can run it on the same benchmarks with unsafe-eval and see how it compares.

Run your own comparison

We'll use ScopeGuard-2 4B, our newest guardrailing model, and compare it with GPT-6 Luna on ToxicChat.

For this run, we'll use vLLM to run ScopeGuard locally. You'll need an NVIDIA GPU and Python 3.11 or newer. If you don't have a GPU available, you can start with the hosted GPT example below. To install unsafe-eval with vLLM support, run:

pip install "unsafe-eval[vllm]"

We've included a configuration for this model and benchmark, so we can choose it by name. We'll save the results in runs/blog-example:

unsafe-eval run vllm/scope-guard-v2-4B-q-2609-class-only/toxic-chat-0124 --log-dir runs/blog-example

Once the run finishes, we can turn those results into an HTML report:

unsafe-eval report runs/blog-example --format html

That gives us our first set of results. To see how a general-purpose model compares, let's try GPT-6 Luna on the same benchmark through its hosted API. You can also start here if you're following along without a GPU. Set OPENAI_API_KEY in your environment or .env file, then install OpenAI support and run the evaluation:

pip install "unsafe-eval[openai]"

unsafe-eval run openai/gpt-6-luna --benchmark toxic-chat-0124 --log-dir runs/blog-example

We're saving this run in the same folder, so running the report command again will bring both models' results together if you've tried both. Let's look at what we found in our runs.

Read the results

On ToxicChat, ScopeGuard caught 89.5% of unsafe requests, compared with 67.4% for GPT-6 Luna. If we were choosing on unsafe detection alone, the choice would look pretty clear.

But we also care about the customers whose requests should go through. GPT correctly allowed 96.1% of safe prompts, while ScopeGuard allowed 93.2%. So ScopeGuard caught more harmful requests, at the expense of blocking slightly more legitimate ones.

In the report, these two metrics are called unsafe recall, the percentage of unsafe requests caught, and safe recall, the percentage of safe requests allowed through. We want a guardrail to do both well, so it helps to look at their average, balanced accuracy. That gives us 91.4% for ScopeGuard and 81.7% for GPT.

In our customer support example, ScopeGuard's median response time of 87 milliseconds, compared with nearly two seconds for GPT, would mean less waiting before the assistant starts answering. One thing to keep in mind, we ran ScopeGuard locally and GPT through its hosted API, so the timings may differ with your own hardware and traffic.

A bit more on the metrics

By the time you arrived at this point, you will have already taken a look at the leaderboard to check out which model is performing better. But better in what sense?

We have a clear preference for tracking balanced accuracy as our primary metric, because it explicitly balances two equally important capabilities of a prompt safety guardrail:

1. Detecting unsafe prompts correctly (true positive rate, or sensitivity).
2. Accepting safe prompts correctly (true negative rate, or specificity).

Both are essential: a guardrail should detect unsafe content without unnecessarily rejecting legitimate user requests.

Many prompt safety studies only report unsafe (or harmful) F1. While this metric does penalize false positives through precision, it has an important limitation: its value depends on the proportion of safe and unsafe prompts in the evaluation dataset. Consequently, unsafe F1 scores can be difficult to compare across benchmarks with different class distributions, and a high unsafe F1 does not necessarily imply a low false-positive rate.

Balanced accuracy addresses this limitation by giving equal weight to sensitivity and specificity, regardless of class prevalence. For example, a trivial guardrail that classifies every prompt as unsafe would achieve only 50% balanced accuracy, independently of the dataset's composition. Its unsafe F1, however, could be deceptively high on a dataset dominated by unsafe prompts.

For these reasons, we prioritize balanced accuracy while also reporting unsafe F1.

From a benchmark to your application

Whichever model you're considering, the next step is to try it on requests from your application. We'd want to see how reliably it catches unsafe requests and look more closely at the legitimate questions it blocks. If those include common support questions, we could end up turning away customers more often than its benchmark results suggest.

For that evaluation, we'd include everyday requests in the languages our customers use, alongside cases that have caused problems before. We'd label them according to our application's policy, including any limits on what the assistant is allowed to help with.

Try unsafe-eval

The next time a model comes out and someone at the office asks how it compares, we'll start with unsafe-eval. The leaderboard shows the experiments we've run; unsafe-eval gives you the tooling to run your own.

Explore the leaderboard · View the repository