Quick Take I stood up a personal agentic AI benchmark using Inspect AI, with tasks running in Docker and OpenRouter as the default inference provider. A first run of six starter tasks against four free models already surprised me: ling-3.0-flash, a model I had never heard of, matched every score while using the fewest tokens.
New models ship constantly, and I have no good way of telling whether any of them matter for what I actually do. The published numbers don’t help; they’re benchmarks I don’t run, on tasks I don’t have. Meanwhile my own setup runs on auto-pilot: Claude Code with Opus 5 for coding, ChatGPT 5.6 Sol for adversarial reviews, Gemini 3.6 Flash for personal things. It works, but I couldn’t defend any of it with evidence.
I’ve wanted my own benchmark for a while. The closest thing I have is the tutorial test from my OCR pipeline post, where I ask a model to build a learning site from Breaking Down the Jelly Slider. But that only makes sense for frontier models, and lately I’ve been thinking a lot about smaller and even local models.
The Goal Link to heading
Create a small personal AI agent benchmark to evaluate models so I have a better understanding of what they excel at as I build agents and workflows. It will be a mix of coding and personal tasks I personally face and understand so the results actually mean something to me.
Where Do You Even Start? Link to heading
While it is always fun to just tell Claude to whip something up I figured I should do some research first to see what existing art I could use.
I started by telling Claude:
I want to come up with my own personal kind of AI benchmark evaluation, something that can test out local models and new models and get a good feel for what they're good for and what they're not good for. I primarily, probably like 80% of the time, use AI for coding with hard code but the other stuff I use is for my family vacations and whatever. What kinds of things should I be thinking about? How big does an evaluation suite really need to be?
We got into the weeds pretty quickly on two questions I’ve been chewing on for a while: how do you benchmark frontier models on things they can’t do yet, and how do you keep a suite fresh as knowledge cutoff dates move forward. Both deserve their own post once the framework actually deals with them, so I set them aside and got back to building.
Reeling It Back to Reality Link to heading
After I scratched my curiosity I decided to start small and at least get a skeleton up to have a good starting point. First I asked Claude to research what existing things I could use. It came back with this comparison.
| Inspect AI | promptfoo | Terminal-Bench / Harbor | lm-evaluation-harness | DeepEval | |
|---|---|---|---|---|---|
| Owner | UK AI Safety Institute | Promptfoo Inc. (OSS) | Stanford/Laude Institute | EleutherAI | Confident AI |
| OpenRouter | Native (openrouter/<model>) |
Native (openrouter:<model>) |
via provider strings / LiteLLM-ish | via OpenAI-compatible shim | via OpenAI-compatible shim |
| Sandboxed code execution | Yes (Docker, k8s, local) | No (you shell out yourself) | Yes (Docker is the unit of work) | No | No |
| Multi-turn agents + tools | Yes (first-class solvers) | Limited | Yes (that’s the point) | No | Limited |
| LLM-as-judge | Yes (model_graded_qa, custom scorers) |
Yes, very ergonomic | Not the model | No | Yes |
| Results UI | inspect view (rich trace viewer) |
Best-in-class local web UI / matrix | CSV + logs | CSV/JSON | SaaS-leaning |
| Best at | One harness for everything | Prompt A/B + regression gates | Hard end-to-end agent tasks | Academic multiple-choice/logprob | Unit-test-style assertions in CI |
Inspect AI was what Claude recommended and what I went with.
Standing Up the Skeleton Link to heading
With the framework picked I gave Claude one more prompt:
Setup this fresh repo to use `Inspect AI` as a basis for a coding and personal task agentic ai benchmark that runs in docker and uses openrouter as the default inference.
I asked Claude to summarize the tests it created to make sure things were working:
- slugify (implement): write a slugify function in /workspace/slugify.py that turns a string into a URL-friendly slug (lowercase, hyphen-separated, punctuation stripped). The sample seeds test_slugify.py, which defines the exact expected behavior.
- fix-median (debug): /workspace/stats.py is seeded with a buggy median function; the agent must diagnose and fix it so the seeded test_stats.py suite passes, without changing the public API or the tests.
- wordfreq-cli (build a tool): write
python wordfreq.py <file> <n>, a CLI that tokenizes a text file case-insensitively and prints the top-n words as word count lines, verified by a seeded test_wordfreq.py. - organize-files: /workspace is seeded with seven loose files (jpg, png, pdf, txt, csv); the agent must sort them into images/, documents/, and data/ subdirectories. A Python check script verifies every file ended up in the right folder and nothing is left in the root.
- expense-report: analyze a seeded expenses.csv and write report.md stating the total spent and the top spending category with its total. The check script verifies the report contains the correct figures ($412.45 total, Groceries at $174.80).
- reply-email: acting as an assistant for a persona named Jordan, read a property manager’s message in inbox.txt and draft a reply choosing the only slot compatible with Jordan’s Mon-Fri 9-to-5 schedule (Saturday 9 AM). This one is model-graded against criteria in target instead of using a check script; the grader checks the right slot was chosen and the faucet leak was referenced.
Proving Free Models Can Do the Basics Link to heading
I ran the suite of tests against four free models on OpenRouter:
- nvidia/nemotron-3-ultra-550b-a55b
- inclusionai/ling-3.0-flash
- poolside/laguna-s-2.1
- openai/gpt-oss-20b
| Model | Accuracy (6 samples) | Time (personal) | Time (coding) | Total Tokens | Output Tokens |
|---|---|---|---|---|---|
| inclusionai/ling-3.0-flash | 1.000 | 0:38 | 0:17 | 55,327 | 5,626 |
| nvidia/nemotron-3-ultra-550b | 1.000 | 1:25 | 1:20 | 57,246 | 5,915 |
| poolside/laguna-s-2.1 | 1.000 | 0:30 | 2:19 | 78,913 | 12,489 |
| openai/gpt-oss-20b | 0.667 | 3:09 | 8:18 | 141,634 | 24,052 |
I was a bit surprised that ling-3.0-flash did so well compared to laguna-s-2.1. Laguna is marketed as a strong coding model and I had never heard of Ling. Afterwards I looked up InclusionAI and it seems like one of their goals is token efficiency, which tracks with these results. I also found this blog post where they call the current ecosystem “a hackathon in the real world”, which also feels right.
The gpt-oss failures were on the organize-files test, where it sorted six of the seven files correctly but its final replies degenerated into random tokens and it hit the message limit before finishing. The other failure was on slugify where it didn’t do a proper implementation and then just started returning empty responses.
Lessons Learned Link to heading
Under-the-radar models can be quietly good. Laguna is the one marketed as a strong coding model, and I had never even heard of Ling before this run. On my suite they tied at 1.000, but Ling finished the coding tasks in 0:17 to Laguna’s 2:19 and used less than half the output tokens, 5,626 to 12,489. If I only paid attention to models with hype behind them I never would have tried it, and the only way I’ll keep finding models like it is by running my own tests.
Token efficiency as a design goal fits how I already work. Context engineering is a big part of how I structure agent work, trimming what goes in so each step stays lean, but I had only ever thought about it from the prompt side. InclusionAI targets it from the model side, and it showed: ling-3.0-flash finished with a perfect score on the fewest tokens of any model I ran, 55,327 total against Laguna’s 78,913. Finding a model whose design goal matches how I think about context is exactly the kind of thing this benchmark exists to surface.
What’s Next Link to heading
I’m going to work on adding some non-trivial tests and will probably publish benchmark results occasionally. Creating meaningful tests is a real challenge so I might supplement from existing frameworks till I can build out a proper suite. The benchmark repo is private for now.