Jev for Safety Guardrailing

Unless you have been living in a bubble, you will have heard of Jev by now, the new frontier model presented by TypeSafe AI. People have been buzzing about it in person, on Reddit and on LinkedIn, and yet we know surprisingly little about it: we don't even know its architecture.

This article is not another attempt to guess what is under the hood. It is about the one thing that is definite about Jev and that TypeSafe has decided, for reasons we will get to, not to disclose: how it performs on public safety benchmarks.

Jev and Benchmaxxing

Let's start with the basic question: what is Jev? There are plenty of good articles covering this, so we will just state what matters here: it is a new system that replaces LLMs on classification tasks at a fraction of their cost and latency. We will put actual numbers on that below.

The tricky part is how well it replaces them. Random guessing is faster than Jev too, but it is not like we can do much with it. This brings us to the core of this article: TypeSafe has explicitly decided not to release any performance score on public benchmarks.

This choice comes from their antibenchmaxxing policy. The argument there is that the field has spent years defining better as beating a score on a public benchmark, and that this has inevitably produced models that benchmax those resources rather than actually improve anything. Benchmarks have been broken for a while, as they put it. What TypeSafe proposes instead, since measurement and comparison still have to happen somehow, is to run your own private evals, treat public ones with a grain of salt, and don't amplify every number or plot you see.

Having worked in this research field, I am familiar with the issue and I share TypeSafe's position to some extent. However, public benchmarks are still important for at least two reasons:

1. They provide by no means perfect rankings, but things can still be read off them safely. For instance, a bad public score is enough to rule a model out, even if a good one does not prove much. Similarly, if two models score close to each other, you cannot conclude that one is definitely better, but you can expect to swap one for the other without major disruption.
2. Private evals are not free: they take data you may not have, annotation someone has to pay for and engineering time to keep them running. Most teams cannot do that every time a new model ships, so public numbers are what they go on in the meantime.

Safety Guardrailing

Jev is fast, cheap, accurate (spoiler here) and designed for classification tasks. That combination is exactly what guardrailing needs, and safety guardrailing in particular.

Take a customer support assistant handling a few hundred thousand messages a day: every single user turn has to clear the safety classifier before the assistant ever sees it, so the guardrail pays its cost and its latency on all of them, while the assistant itself only runs on the ones that pass. A guardrail that is slower and more expensive than it needs to be is a tax on every request you serve, including all the safe ones.

That is what makes public safety benchmarks the right place to look for Jev, and since TypeSafe is not going to publish those numbers, we ran them ourselves.

Evaluation Benchmarks

Every benchmark here evaluates the same task: the model reads a user prompt, with no model response and no surrounding conversation, and has to label it safe or unsafe. The datasets we report on are as follows:

These counts are the prompts we actually evaluate, not the raw dataset sizes: rows with redacted text, incomplete category labels or no annotator agreement are excluded.

Results

We test three more models along with Jev on these benchmarks:

1. Nemotron 3.5 Content Safety
2. GPT-5.6 Luna
3. GPT-5.6 Sol

The choice is intended: Nemotron is a small model that is close to Jev in price and latency, while GPT-5.6 Luna and GPT-5.6 Sol give the reference points for a lower- and a higher-tier LLM.

Accuracy

The score we report is balanced accuracy: the average of unsafe recall, that is the share of unsafe prompts the model flags, and safe recall, the share of safe prompts it lets through.

ChartTable

Benchmark	Jev 1.13	Nemotron 3.5 Content Safety	GPT-5.6 Luna	GPT-5.6 Sol
Aegis 2.0	0.8341	0.8416	0.8419	0.8186
CoSApien	0.9530	0.8840	0.9427	0.9427
DynaGuardrail	0.9087	0.8965	0.8917	0.9081
OpenAI Moderation	0.8282	0.7404	0.8545	0.8617
ToxicChat 0124	0.8473	0.9032	0.8137	0.8263
WildGuardMix test	0.9050	0.8620	0.8903	0.8937
Mean	0.8794	0.8546	0.8725	0.8752

Jev takes the best mean and comes first on more rows than any other model here, and we are not going to make much of that. The margins are in the third decimal: 0.8794 against 0.8752 for GPT-5.6 Sol. Reading a ranking out of numbers that close is the exact move we objected to earlier (and would call for statistical significance analysis, which we are not going to get into). The useful part of this table is the floor rather than the order: all four models land above 0.85 on the mean.

The only row worth a second look is ToxicChat 0124, where Jev is furthest from the top score and sits 5.6 points behind Nemotron. The gap comes from how the two models split their errors: Jev flags fewer safe prompts as unsafe, and in exchange it misses more of the unsafe ones. ToxicChat is 87% safe, and balanced accuracy weights the two classes equally, so with 362 unsafe prompts against 2,491 safe ones each miss on the unsafe side moves the score about seven times as much as each false alarm on the safe side. Which side of that trade you want is a product decision rather than a benchmark one: for a guardrail, over-refusal is a problem too, and depending on where you deploy it, it can be the worse one.

Cost and Latency

ChartTable

Model	Total cost ($)	Mean inference time (s)
Jev 1.13	0.2681	0.651
Nemotron 3.5 Content Safety	1.0748	0.842
GPT-5.6 Luna	1.7405	1.954
GPT-5.6 Sol	31.0207	3.251

This is where the picture stops being close. Running the suite costs $0.27 with Jev against $31.02 with GPT-5.6 Sol, and mean inference time goes the same way: 0.651 seconds against 3.251. Against Nemotron, the model that is actually in Jev's weight class, it is 4x cheaper and roughly 1.3x faster while scoring 2.5 points higher on the mean.

Guardrailing Limitations of Jev

Numbers like these make it tempting to read Jev as a silver bullet for guardrailing. It is not, and two limitations are worth stating plainly.

The first is reasoning. Some decisions genuinely require it: over a long policy document, over the earlier turns of a conversation, over a request that is only unsafe because of what the user asked three messages ago. A classifier like Jev is not going to get there, and an LLM is still the right tool for the job.

The second is that a label is often not the whole output you need. Go back to the support assistant: when the guardrail blocks a message, the user has to be told something, and whoever audits the decision next week has to understand why it was made. unsafe serves neither of them, while unsafe: the request asks for a way around an account lock rather than help with it serves both. Jev does not produce that second field, and bolting an LLM call onto the guardrail to generate it hands back most of the latency and the cost you just saved.

Both of these are problems we are working on at Principled Intelligence. If you are running a guardrail where either one is blocking you, we would like to hear about it: reach out at orbitals@principled-intelligence.com.