LLM reliability harnesses June – September 2026
Aegis
Aegis is a set of reliability harnesses for LLM apps. I test each one on held-out data under a protocol I freeze before the run, with confidence intervals and paired tests. The newest study asks whether Jev, a fast “System One” decision model, can replace an LLM judge as the guard in front of other models.
The Jev study
The full write-up, with the datasets, the protocol, the results and the problems I ran into. Open the PDF in a new tab, or read it below.
The question
LLM applications need guards: something has to screen incoming text for prompt injection, or decide which model a query should go to. Usually the guard is another LLM. That's accurate, but slow and expensive, and the guard can end up costing as much as the model it protects.
Jev, TypeSafe AI's “System One” model, doesn't generate text at all. It answers multiple-choice and rating questions about a text in about 140 milliseconds. I wanted to know if that's good enough for the job.
How it was tested
I tested jev-1.13.0 against Claude Haiku 4.5 and GPT-4o-mini judges, a trained local classifier (ProtectAI DeBERTa) and keyword rules, on 991 held-out texts. They include direct injections, injections hidden inside Wikipedia paragraphs, and NotInject, a set of harmless prompts full of trigger words that tend to set off false alarms. Separately, I had Jev route queries between Mixtral-8x7B and GPT-4 on RouterBench.
I froze the protocol before the test run and committed every cached model decision, so the numbers, tables and figures all rebuild without calling a model.
What it found
On screening, Jev came out ahead: AUROC 0.994, against 0.980 for the Claude Haiku 4.5 judge and 0.842 for GPT-4o-mini, with the lowest false-alarm rate on NotInject (1.6%). It ran 4× faster than the judge at 1/18 the cost. The local classifier was faster still, but it flagged 44% of the harmless prompts.
The problem is the threshold. At its default cut-off Jev catches only 69.9% of injections, and the cut-off that catches 97.8% also flags about half of NotInject. It ranks texts very well, but its scores aren’t calibrated.
Routing didn’t work. As a model router Jev did no better than picking at random, and worse than a TF-IDF baseline. The report gives that result as much space as the wins.
The wider project
Two other harnesses follow the same rules. A customer-service harness on tau2-bench checks an airline agent's actions against written policy, in code, before they run. With it, GPT-4o-mini solves 43.8% of tasks instead of 26.2%, about level with a stronger model. When I built the same check as an AI reviewer, it blocked three times as many actions, and many of those were correct.
The first one, a retrieval-QA harness (hybrid retrieval, reranking, NLI verification, and declining to answer rather than guessing), cut unsupported answers on HotpotQA from 12.7% to 3.3%. It also refused 39% of questions it could have answered, because its AI verifier rejected right answers about as often as wrong ones. That’s why the newer harnesses do their checks in code.