AI & Tech··1 sources verified
Hallucination Detector Rankings Invert Between Small Quantized LLMs, Audit Finds
Summarised by Relevant News AI · Read time: 3 min
A reproducible audit tested four hallucination detectors on Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct at 4-bit quantization on a single Colab T4 GPU, using closed-book TriviaQA. Detector rankings reversed between the two models (Spearman rho -0.80), and a cheap lexical n-gram baseline was statistically indistinguishable from sampling-based consistency methods. The authors also found that a standard TruthfulQA labeler produced near-chance AUROC, which they attribute to the labeler rather than the detectors, and they released their harness and per-item scores.
Why it matters: Teams shipping on-device small models cannot assume that a hallucination detector validated on 7B-70B research models will rank the same way on the 2-3B quantized models they actually deploy.
All sources