EvaluationHowever Fast Agents Change, First Ask What Problem You're Solving

When I drafted L4, I listed Microsoft’s AutoGen as one of the main frameworks for building multi-agent systems. This month, while putting the series together and checking its sources again, I found that back in October 2025 its project page was already asking new users to move to Microsoft Agent Framework, its newer framework, and saying AutoGen itself would only get bug fixes and security patches. By October 2026, the page read: “This project has been archived and is no longer actively maintained by Microsoft.”¹ By the time I wrote it down, it was already out of date.

I’ve been writing this series since March 2026, and that wasn’t the only time something like this happened. So here is what changed over these months, and what didn’t.

What changed: Agents are getting more and more access

In the second half of this year, several companies released Agents that can work on someone’s behalf for long stretches. In August, xAI launched Grok Bot, billed as “AI teammates you can give real work to.” They have their own computer, work inside the same tools and apps you use, keep going around the clock, and come back to you only when something needs your approval.² In September, Meta launched Muse, which can connect to a user’s email, calendar and payments, and keeps working after the user leaves the app.³ The same month, OpenAI launched dots, which it describes as “always-on agents built to handle everything”: they keep working in the background toward goals the user sets, needing hardly any supervision.⁴ In October, Anthropic launched Claude Duet, an Agent that takes the initiative and keeps working after you step away.⁵

Not long ago, this was something engineers had to build themselves. Now you open an app and it’s there. I think LLM applications like these will only get bigger, and they’ll be given more and more access.

Measured with this series’ ruler, they’re already moving toward L5. In the series introduction, I described L5 (the Explorer) as “Experimental. High risk, high autonomy.” The quick selection chart labeled it “Caution in Production.” By L5 I mean this: you give the system a goal, and it works out its own route, correcting as it goes, with no end point written in advance. Half a year later, these products can already pursue a goal in the background for long stretches, deciding for themselves which apps to use and which steps to take.⁵ L5 arrived faster than I could write about it, and that’s why this post became a short note.

What hasn’t changed: how much to let go

As the access keeps widening, the question this series started with becomes more pressing: for any one task, how much should you actually hand over to AI?

These products are answering that question themselves. Before Claude Duet sends a message, makes a payment, signs a document or deletes anything, it shows you what it’s about to do and waits for you to approve. And there are a few things it won’t do even if you approve, such as trading securities or moving money between accounts.⁵ Grok Bot, for its part, says it learns over time when to check in with you and when to keep going.² So I wouldn’t call them full L5: when it comes to payments or sending messages, they still stop and wait for a person to say yes. When it comes down to it, they all do the same thing: draw the line, then let go.

Whatever changes, ask four questions first

The technology and the products will keep changing. Whatever happens, I think four questions need answers first:

1. What business problem are you actually trying to solve?
2. What do you want to solve it with, and how?
3. Which level of today’s technology is enough?
4. How do you make sure the results are reliable and worth trusting?

The first two questions are the step the series introduction put first: “Define the scenario.” The third is the L1-to-L5 ruler I put together myself. What it measures isn’t which product is strongest, but how much autonomy the task needs. For the fourth, my approach is to build up my own set of test questions.

The questions come from the real business. Twenty or thirty are enough, and each one spells out what counts as a right answer. Take the same e-commerce platform again. A customer asks, “How long is the warranty on this air fryer?” The answer has to be “one year on the whole unit, two years on the heating element,” with the source attached. A customer asks, “Can I get price protection on this order?” The answer can’t come from the courier’s shipping-insurance clause. Someone asks to look up another person’s order, and the system has to refuse.

From then on, every time you change the model, the framework or the prompt, or want to try a new product, run it through the same questions again. The total score matters less than three kinds of mistakes: a wrong answer given with confidence; a refusal where it should have answered; and a change made without a person confirming it first. The questions have to change with the business, too. When the after-sales policy changes, update the expected answers. Every time something goes wrong in production, turn that case into a new question.

Closing

Back to the archived framework from the start. It was already out of date when I wrote it down. This field moves that fast, and I can’t guess what it will look like next.

Guessing aside, I really do enjoy watching this technology evolve: every few months, something that couldn’t be done before becomes possible. But before using it, it’s still worth stopping to think: in your own business, which technology is the right one, how to use it well, and whether it really solves the problem in front of you.

This is where the series ends. If the ruler ever stops being useful, I’ll come back and update the note on the series page.