Updates & insights

News

The latest news, updates, and insights from our team on our mission to build AI systems that are safe, transparent, and aligned with human values.

3 posts tagged Evaluation

Articles

Guardrails

Introducing unsafe-eval: a framework for AI safety evaluation

When OpenAI, Anthropic, Google, or others release a new LLM, we're curious whether it makes sense to use it as a safety guardrail. Trying a few prompts out usually gives us a first impression; however, a proper comparison with other models takes more work. We built unsafe-eval to make those comparisons easier to run, and we're now sharing it.

11 min read
Guardrails

Introducing the unsafe-eval Leaderboard: comparing models for AI safety

We're introducing the unsafe-eval leaderboard to help compare models for AI safety tasks. Our first results examine frontier models and dedicated classifiers as safety guardrails. Several achieve similar accuracy, but differ substantially in cost, latency and how often they refuse to classify a request. We explore what those differences mean when choosing a model for an application.

21 min read
Guardrails

Jev for Safety Guardrailing

TypeSafe does not publish public benchmark scores for Jev, so we ran six public safety benchmarks ourselves. On accuracy, Jev sits in the same band as frontier LLMs; on cost and latency, it is nowhere near them.

9 min read