We tested GPT-4o-mini, GPT-4o, Llama 3 8B, and Llama 3 70B on document extraction. The results were surprising — and not in the way you'd expect.
I ran an experiment this week comparing four LLMs on structured document extraction: GPT-4o-mini, GPT-4o, Llama 3 8B, and Llama 3 70B. 165 documents across 13 categories — insurance policies, SEC filings, invoices, medical records, contracts, and more.
The headline: GPT-4o is worse than GPT-4o-mini at extraction. You pay 10x more for 2.3 percentage points less accuracy. Meanwhile, Llama 70B is 10pp better than Llama 8B, as you’d expect.
The mechanism: GPT-4o is more conservative — it returns null 25% of the time vs 21% for 4o-mini. It’s not extracting wrong values; it’s declining to extract at all, especially on arrays and free-text strings. On constrained types (enums, numbers), it’s fine or better.
I tested three hypotheses for why: prompt compliance (no), temperature sensitivity (no), and general scaling phenomenon (no — Llama scales normally). It’s something specific to OpenAI’s fine-tuning, not a universal property of larger models.
The full writeup with tables, methodology, and practical recommendations is on the Koji blog. The short version: test models on your actual extraction task before choosing, and don’t assume expensive = better.
This came out of the experiment infrastructure I built for an upcoming EMNLP paper on extraction pipeline techniques. More on that soon.