Parth Parekh
All work

OpenJEV-Flash

openjev's judge, rebuilt at 150M parameters

Year
2026
Kind
Models
Status
Finished

5.2x faster than openjev-0.8B

Model on Hugging Face

Alex Wortega's openjev judge, rebuilt on the 150M-parameter Ettin encoder and distilled from its 4B model. One cross-encoder call reranks, grades, checks for hallucinations or picks an option.

openjev-flash reads a premise and a hypothesis and returns contradiction, entailment or neutral, and one call covers reranking, grading, hallucination checks and multiple-choice decisions. It matches openjev-0.8B on NLI and grounded faithfulness at 4.4–5.2x its speed, and trails it on knowledge-heavy judging. The data mix, eval harness and teacher are openjev's. Every number on the model card comes from raw eval files published with it.

The problem

openjev, by Alex Wortega, turns Qwen models of 0.8B, 2B and 4B parameters into judges. A judge reads a premise and a hypothesis and answers contradiction, entailment or neutral, and that one call covers reranking, grading against a reference, hallucination checks and typed decisions. openjev-flash is the same judge made small enough to sit next to a generator on a small GPU: rebuilt on the 150M-parameter Ettin encoder and distilled from openjev's 4B model.

How it works

  1. 01

    Rebuild the training mix

    openjev's own data_mix.py rebuilds the data: 1,298,070 premise and hypothesis pairs, 359M tokens, from 23 sources including SNLI, MultiNLI, RAGTruth, FalseQA and agent trajectories, balanced by class. Every eval test split is on the script's ban list. The mix was built with --no-jevbench, so no JevBench item leaks in.

  2. 02

    Read both texts in one pass

    The model is a cross-encoder: premise and hypothesis go in together as [CLS] premise [SEP] hypothesis [SEP], so every layer can attend across both. The backbone is Ettin-150M, a 22-layer ModernBERT encoder, with a mean-pooled three-way head. Pairs train at up to 2,048 tokens, and only the premise is ever truncated.

  3. 03

    Learn from a 4B teacher

    openjev's qwen3.5-4b-nli-v5 labelled 1,161,975 training rows (89.5%) with its logits, agreeing with the gold label on 92.3% of them. The loss is half cross-entropy on the gold label and half KL divergence from the teacher's distribution, so the student learns how sure to be, not only which class wins. One epoch took 82 minutes on one RTX PRO 6000 Blackwell.

  4. 04

    Turn every task into entailment

    A small wrapper, openjev_flash.py, reuses openjev's task recipes. grade checks a candidate answer against a reference, and rerank picks the candidate a document best supports. decide writes one hypothesis per option and normalises P(entailment) across them. Nothing is generated and nothing is trained per task.

  5. 05

    Measure speed on one laptop GPU

    Every model ran through the same bench_speed.py on one RTX 5060 Laptop GPU: bf16, batch 16, pairs of 44, 105 and 300 tokens. At 44 tokens the 150M scores 1,425 pairs per second against 310 for openjev-0.8B. Reruns matched within 3%.

Results

4.4–5.2x
pairs per second over openjev-0.8B on the same GPU
87.9
MNLI matched accuracy, against 86.2 for openjev-0.8B
0.896
RAGTruth AUROC, against 0.899 for openjev-0.8B
0.41 GiB
peak VRAM at batch 16

Decisions

  • Ettin over Gemma 4 E2B

    Gemma 4 E2B was the first backbone considered. Its 2.3B parameters of per-layer embedding tables had to be offloaded to the CPU to fit in 8 GB, and it ran 2.7x slower than openjev-0.8B, so it was dropped.

  • One harness, raw files published

    All four models were scored with openjev's own eval scripts on one GPU. The harness reproduces openjev's published 0.8B numbers (MNLI 86.2 / 87.2, AggreFact 0.695), and the raw JSON behind every number ships with the model, along with the logs and code.

  • Keep the no-teacher model

    Distillation dropped the BullshitBench judge AUROC from 0.839 to 0.716. The repo also ships ettin150m-nli, trained on gold labels only, and the card says to use it for judging answers to false-premise questions.

What I learned

  • A 150M model reads well but knows little. It trails openjev-0.8B on all three adversarial ANLI rounds, is at chance on FinanceBench and DROP, and scores 0.32 on JevBench's hard tier against 0.49 for openjev-2B. Without context it names Melbourne as Australia's capital.
  • Distillation sharpened what the model already did, with JevBench's standard tier up 6.9 points and HaluBench up 3.7, but it added no knowledge.
  • Scores aren't calibrated across tasks (JevBench ECE 0.14 to 0.18), so thresholds have to be picked per task.