LinkedIn·Wednesday, 19 August 2026·7d ago
Most of what you ask AI to code isn't that hard. So why run it through your most expensive model? Teams often route routine work (Terraform…
Dataminded
4,968 followers
Most of what you ask AI to code isn't that hard. So why run it through your most expensive model?
Teams often route routine work (Terraform modules, glue scripts, cleanup jobs,...) through the same frontier LLM they use for the hard stuff. The bill adds up.
We tested whether a smaller, self-hosted model could handle that routine work. Not with a public leaderboard, but with our own benchmark built from real tasks in our codebases.
Three things we learned:
- Model and harness matter equally: a strong model with a poor harness can perform worse than a weaker model with the right setup.
- Active parameters were a better predictor of performance than total parameters.
- Small models got surprisingly close to Sonnet 5. But not out of the box.
Niels Claeys breaks down the benchmark, where public leaderboards misled us, and which harness fixes actually closed the gap.
Read here: https://lnkd.in/eAHbSHXi
#LLM #AIEngineering #Benchmarking
♥ 11
View on LinkedIn Cross-referenced
Related on the wire
Last week we hosted our Agentic AI Skills for Data Practitioners workshop at our Antwerp office, and what an evening it was! 🙌 We were…
Many organisations started a Data Mesh journey. Far fewer reached the outcomes they hoped for. Why? In the latest episode of The Data…
Think your agent setup can hold up under real conditions? Prove it. Same model. Same tasks. Your harness, your context. Dataminded's data…
Time-series foundation models promise anomaly detection with no feature engineering. We tested the promise. The pitch is appealing: point a…
Most AI agent demos never touch a real pipeline. Ours does, and it gets scored on it. On 26 August in Antwerp, we're running a hackathon…
An AI agent scored 37 out of 100 on questions about our own data. The same model later scored 100 out of 100. Ask an AI agent a question…