How to read AI news without getting fooled

A new model drops every week and every one is "a breakthrough". Here is the five-question filter we use to decide whether a headline changes anything for a real business.

Every week brings a new model, a new benchmark record and a new wave of posts saying everything has changed. Most of it does not matter to a business that is trying to get work done. Some of it matters a lot. Telling the two apart is a skill, and it is learnable.

Here is the filter we run every headline through.

1. Is it a model, a product, or a demo?

A model is a capability. A product is that capability wrapped in something people can use, with pricing and a service agreement. A demo is a video.

Models and products can change what is possible for you. Demos cannot, until they become one of the other two. A surprising share of AI news is demos.

2. Did the price change, or only the score?

Benchmark scores creep up a few points at a time. Prices drop in halves. For most business use, a model that is slightly less clever and five times cheaper is the bigger news, because it changes what you can afford to run on every customer interaction instead of a few.

When you read "new state of the art", look for the price per million tokens in the same announcement. If it is not there, the score is the whole story, and the story is small.

3. Does it work on your kind of data?

Almost all public benchmarks test clean English text and code. If your business runs on scanned invoices, Telugu voice notes, half-filled spreadsheets and PDFs with tables, a benchmark win may not transfer. The only test that counts is one on your own documents, which is why we build that test first on every project.

4. Is the limitation mentioned?

Serious announcements name what the model is still bad at. Marketing does not. If a launch post has no "limitations" section and no failure examples, treat the claims as unverified until someone independent has tried it.

5. Can you switch to it in an afternoon?

If your software was designed around the AI properly, swapping the model underneath is a configuration change and an afternoon of re-running your evaluation set. If it was bolted on, every model change is a project. The news only helps you if your architecture lets you act on it.

What we do with this

We keep a short list of the announcements that passed all five questions in a given month. It is usually two or three items, out of dozens. Those are the ones we test on client data, and the ones we write about here. Everything else we read, enjoy, and let go.