L1 & L2Teaching AI to Be Precise, Not Just Prolific

A customer asks the support bot, “How long is the warranty on this air fryer?” The bot answers, “Two years.” But the after-sales policy says the whole unit is covered for one year, and only the heating element for two. Every word of the answer can be found in the document. Put together, they’re wrong.

The series introduction argued that when a system only needs to talk, the task has clear boundaries, and the answer is due within 3 seconds, L1 and L2 are often the right choice. But choosing the right architecture doesn’t make the answers right on its own. This post covers the first two levels of the spectrum:

• L1 (Basic Responder): the LLM generates passively, and outside code controls the whole flow. The typical setup is Prompt + RAG Definition Retrieval-augmented generation. Put simply, before answering, the AI looks things up in a knowledge base you choose, then writes its answer from what it found instead of making it up. .
• L2 (Router): the LLM recognizes what the user wants and hands the request to a path defined in advance. The typical setup is Intent Classification + Router.

They go wrong most often in two places: retrieval that misses, and routing you can’t predict. Below I take the causes and the fixes first, then how the interface can help, and last their ceiling, which is also the point where it’s time for L3.

Precise definitions: what L1 and L2 do and don’t do

Before talking about improvements, it’s worth drawing the boundary of each level precisely, so that architecture decisions don’t start off in the wrong place.

L1 does passive generation, also called context-based generation. Its job is to write an answer inside the context it’s given. That context can come from what the model learned in training; inside a company it more often comes from external retrieval (RAG).

The boundary: L1 never changes anything in a business system. It doesn’t write to a database, doesn’t start a workflow, and doesn’t call an API that has side effects. It’s a read-only layer for handling information, and its range is reading, comparing, summarizing and presenting.

L2 does static routing. On top of L1 it adds intent recognition: it works out what the user wants and sends the request down a path defined in advance. Those paths are fixed. A return request takes path A, a delivery question takes path B, a price question calls interface C.

The boundary: L2’s planning is static. It can handle if-else branches, but as soon as a task needs to “plan the next step from an intermediate result” or “change strategy along the way,” L2 stops working. That is how L2 was designed to work, and a better implementation won’t change it.

A common mistake: matching the wrong level to the problem

Before getting to improvements, one mistake comes up often enough to name first:

graph LR
  A[System starts at L1] --> B[Precision falls short]
  B --> C[Pile on RAG tweaks]
  C --> D[Still not good enough]
  D --> E[Jump straight to multi-agent L4]

On this path, the trouble starts earlier than the jump between levels: giving up on L1 before its basic design is right. Bringing in a multi-agent system too early takes errors that could have been pinned down precisely and makes them bigger, while hiding them inside a complicated orchestration layer. Anthropic’s engineering team, summing up what they learned from many customer projects, gives plain advice: find the simplest solution that solves the problem, and add complexity only when it’s truly needed.¹

Where precision is what matters, the “simplest solution” is usually an L1 or L2 that has been designed properly. A more complex Agent system isn’t needed.

L1: three ways retrieval loses precision

RAG does most of the work in L1. It solves the problem of a model’s knowledge going out of date, but in business settings that demand precision it fails in three systematic ways. Knowing all three tells you where to start fixing.

Failure 1: chunking breaks the meaning apart

Standard RAG cuts documents into chunks Definition When a RAG system builds its knowledge base, it first cuts long documents into small pieces and stores them in an index. This is called chunking, and each piece is a chunk. At query time, the system looks for the most relevant chunks, not the whole document. , turns them into vectors and stores them in an index; at query time it pulls the most similar chunks and gives them to the model. The trouble is the cutting itself.

When text is cut at a fixed number of tokens, one clause of an after-sales policy can end up in pieces, spread across several chunks. That’s what happened with the air fryer: “one-year warranty on the whole unit” and “two years on the heating element” landed in two different chunks. Each piece is correct on its own, and only together do they say the whole thing. But at retrieval time a RAG system sees only fragments, and can’t see the logic that runs from one chunk to the next.

A controlled experiment on a question-answering system (30 questions) found that, with the rest of the RAG pipeline unchanged, switching from fixed-token chunking to Adaptive Chunking raised the share of answers rated “at least somewhat accurate” from 50% to 87%, and retrieval recall from 0.40 to 0.88.² The sample is small, but the gap makes the point: where precision matters, how you cut the documents is a design decision that visibly changes the results, and it deserves more thought than a setting picked in passing.

In practice there are several ways to improve it: semantic chunking (cutting where the meaning changes, whatever the word count), proposition chunking (breaking the text into standalone statements of fact, each complete on its own), and hierarchical indexing (adding a layer of section summaries above the small chunks, so retrieval works like a table of contents: find the section first, then read closely).³

Failure 2: general-purpose embeddings miss the domain’s meanings

Vector retrieval rests on one idea: text with similar meanings sits closer together in the vector space. But “similar meaning” is defined by the embedding model, and most general-purpose embedding models were trained on general-purpose text.

E-commerce has a good example, in Chinese. 价保 means price protection: if the price drops after you buy, you get the difference back. 保价 means declared-value shipping insurance: the courier pays out according to the value you declared for the parcel. The two terms are the same two characters in opposite order, almost identical on the page, and they mean completely different things. A general-purpose embedding model can easily treat them as one: a customer asks, “Can I get price protection on this order?” and the system pulls up the courier’s declared-value clause.

This is a systematic blind spot in RAG: the gap between the embedding model’s sense of meaning and your business domain. The way to address it is to fine-tune the embedding model on labeled data from your domain, so that its vectors follow the domain’s structure of meaning more closely. The Prompt Engineering Guide also notes that fine-tuning the embedding model may be needed when you work in a specialized domain.⁴

Failure 3: multi-hop reasoning needs more than one retrieval

The first two failures can be eased with better chunking and better embeddings. The third runs into a limit in the architecture of RAG itself.

Some questions can’t be answered with a single retrieval. They need reasoning that combines several documents or sections. For example: “Which clauses in this year’s after-sales policy have changed in substance since last year?”

To answer that, the system has to ① find last year’s relevant clauses, ② find this year’s, ③ understand how they differ, and ④ judge whether the difference is “substantive.” That’s a chain of reasoning with several hops, and the single retrieve-then-generate pass of standard RAG has no structure to support it.

I’ll come back to this failure at the end of the post, because it points straight at the ceiling of L1.

Improving L1: hybrid retrieval and traceable sources

Once the three failures are clear, so is where to work. The two strategies below have proven themselves where precision matters, with one caveat: they sit on top of RAG, and vector retrieval is still how candidates get pulled in quickly.

Strategy 1: hybrid retrieval

The idea is to combine two kinds of retrieval that cover each other’s gaps:

• Keyword search Definition Also called sparse retrieval (BM25): exact keyword matching. Like a traditional search engine, it finds documents that contain the same words; a similar meaning alone doesn’t count. It’s especially reliable for technical terms and exact numbers, such as product model numbers and SKUs.
• Dense retrieval: based on vector meaning. It works well when people ask for the same thing in many different words.

Where precision matters, the recommended approach is to use dense retrieval to pull in a set of relevant candidates quickly, then use BM25 or a structured index to check the exact terms and numbers. An experiment Anthropic ran in 2024 gives a sense of scale: they added to each chunk the context it had in the original document and combined vector retrieval with BM25, and the share of relevant chunks missing from the top 20 results fell from 5.7% to 2.9%; with a reranking step added, it fell to 1.9%.⁵ That includes the extra step of adding context, so not all of the drop belongs to hybrid retrieval. But one of their conclusions is put very directly: embeddings plus BM25 do better than embeddings alone.

For long, structured documents like product spec sheets and after-sales policies, you can build a hierarchical index above the chunk layer: store a summary at each node, and at retrieval time find the section first, then read its content closely. That’s much closer to how a person reads a document than searching blindly through fragments.³

Strategy 2: traceable sources

There’s one user behavior that’s easy to overlook: users don’t only need an answer; they need to know where it came from.

In areas that easily turn into disputes, such as after-sales, prices and compensation, the AI’s answer is only the first step. The support agent or the customer then needs to check the original clause, confirm the wording, and decide whether to accept the answer. If the system can’t tie each answer to a section of the document, or even a clause number, people have to go digging for themselves. That costs more than time: a wrong answer can start a dispute.

So traceability belongs in the basic requirements of an L1 system:

• Every conclusion comes with the source document’s name, section and page number (down to the paragraph or table number where possible).
• When a conclusion has several sources, the system marks whether they contradict each other.
• When no supporting source can be found, the system declines to answer instead of guessing.

This step turns an L1 system from “a model that gives answers” into a source of information people can verify. That change of role is the first step toward earning users’ trust.

L2: building routing you can predict

If L1 is about finding the right information, L2 is about handling the right request on the right path.

The hard part of L2 is “determinism,” but in engineering that word needs a precise definition, or it misleads.

Three kinds of determinism

• Reproducibility: the same input, whenever it’s asked, leads to the same document or data source. This depends on a stable index and on version control, not on the LLM.
• Routing consistency: the same user intent always takes the same logical path. This depends on a stable classifier and on routing rules that are written out explicitly.
• Structured output: even with temperature Definition Temperature is a setting that controls how random an LLM's output is. At 0 the output is, in theory, the most predictable; at 1 it is more creative but less stable. Even at temperature=0, different versions of a model can still word things differently. set to 0, an LLM’s raw output can drift from one model version to the next. Real determinism comes from constraining the output to a predefined structure (a JSON schema, a fixed set of options, a format template; for example, the AI may only conclude “yes,” “no” or “unsure,” never a free paragraph), plus a layer of rule checks downstream. Don’t count on the model to produce exactly the same words every time.

The two stages of L2 routing

In engineering terms, L2 routing is a pipeline with two stages, one after the other.

The first stage is the intent classifier. It maps what the user typed to one of a set of predefined intent categories. The design rule here is to use a lightweight classification model with highly predictable output, and not to hand intent recognition to the same generative LLM.

A typical set of categories might be:

• Open questions (such as “What’s the difference between these two air fryers?”) → go to the L1 retrieval flow
• Fixed-format queries (such as “Export my orders from last month”) → call the matching API directly
• Out-of-scope requests (such as looking up someone else’s order) → block them, and guide the user to clarify
• Sensitive requests (such as a complaint that asks for compensation, or a product question that involves personal safety) → always hand over to a person

The second stage is routing based on that classification, and it starts only once intent has been recognized. The rule for this stage: for the business paths that matter most, use hard-coded rules, and don’t let the LLM decide.

For example, when a customer asks for a product’s safety instructions (say, whether an electric blanket can stay on all night), L2 should route straight to the manual and after-sales documents that people have reviewed, and keep it out of the general retrieval flow. Information like this has to be authoritative, and that leaves no room for fuzzy retrieval.

graph TD
  Input([User input]) --> Classifier[Intent classifier]
  Classifier -- "Intent score / category" --> Router[Routing rules]
  
  Router -- "A" --> L1[Run the L1 retrieval flow]
  Router -- "B" --> API[Call the API directly]
  Router -- "C" --> Audit[Send to the human review queue]
  Router -- "Reject" --> Reject[Reply with guidance and ask the user for more detail]

Refusing to answer: the most underrated capability

At L2, the most important piece of control logic isn’t how to answer. It’s when not to answer.

Researchers call this abstention. A systematic survey of abstention in LLMs by Wen and colleagues, published in the journal TACL, defines it as an LLM declining to give an answer, a behavior seen as having the potential to reduce hallucinations and improve safety.⁶ On refunds, compensation or product safety, an overconfident wrong answer usually costs far more than “I can’t answer that.”

In engineering terms, an L2 system’s ability to refuse can be built from three layers stacked together.

The first layer is rule-based blocking: hard-code the boundary conditions you already know. For example, if a question involves someone else’s order, or a period the data doesn’t cover, block it right away and return a standard message (“This system can only look up orders from the last 12 months. Your request is outside that range; please check the dates.”).

This is the most predictable and cheapest way to refuse, and it suits boundaries that rules can list completely.

The second layer is a retrieval confidence threshold. In a RAG pipeline, every retrieval produces a similarity score. If even the top result scores below a set threshold, the system hasn’t found reliable support, and it should decline to answer instead of forcing an answer out of weak results.

retrieval_results = retrieve(query)
if retrieval_results[0].score < CONFIDENCE_THRESHOLD:
    return "Nothing in the knowledge base relates directly to this question.
            Please check how the question is worded, or ask for a human agent."
else:
    return generate(query, retrieval_results)

The third layer is a boundary classifier, which recognizes that “this question is beyond what the system can do.” For requests that rules can’t list but whose meaning can be recognized, train a dedicated classifier to judge whether a question is within the system’s scope. You can also constrain the model itself: an IBM experiment found that when there are diverse enough examples of out-of-scope questions, supervised fine-tuning (SFT) of the model works best; when the examples are less diverse, Circuit Breakers (a method that intervenes in the model’s internal representations) do well; combining the two usually gets you the benefits of both.⁷

UI/UX: constrain the input to cut uncertainty at its source

Determinism in the architecture needs help from the interface. For business users without a technical background, interface design comes down to reducing ambiguity in what they type. Instead of having the system guess what someone means, let the interface help them say it clearly.

Here are three design patterns that work in practice.

Pattern 1: intent-aware forms inside the conversation

When the intent classifier recognizes a highly structured intent, such as a return request or an invoice, the system opens a form to collect the details. The customer picks the order, the reason for the return and the refund method from drop-down menus, and nobody has to write a paragraph.

Each field on the form maps directly to a parameter slot in the routing rules. Submitting the form completes intent recognition and parameter extraction in one go, so the back end can go straight to L1 retrieval or an API call, with no step where the LLM has to infer the intent.

This fits tasks with a very predictable structure, users who mostly aren’t technical, and systems that only need to work within a fixed range of parameters.

Pattern 2: guiding users to standard terms (faceted search guidance)

As the user starts typing free text, the system recognizes in real time which kinds of terms they may be reaching for, and shows a list of suggestions next to the input box. That way they can pick the standard term as they type, and the final input doesn’t end up as a non-standard phrase or a variant spelling.

Once the input uses standard terms, the intent classifier has far less ambiguity to deal with, and vector retrieval at the L1 stage gets more precise as well (a standard term has a more stable vector than a free-form phrase).

This fits places full of terms and codes, such as product models, SKUs and after-sales clause numbers, where users know the standard names to very different degrees.

Pattern 3: designing the refusal (refusal UX)

When the system decides not to answer, the way that’s handled directly shapes how much users trust the system.

The wrong way: show “Unable to answer” and offer no next step.

The right way: a refusal contains three things. ① It says why (“This request is outside the data the system currently covers,” not a vague “I can’t answer that”). ② It helps the user fix the request (“Please choose an order from the last 12 months, or pick one of the question types below”). ③ It offers a way to reach a person (“If you need help, please contact a customer service agent”).

The logic behind this design is that refusing is not the same as failing. A system that can correctly tell where its own limits are is already showing you why it can be trusted.

The ceiling of L1/L2: three problems they can’t solve

All the improvements above work inside the L1/L2 framework. But in three kinds of business situation, no amount of refinement to an L1/L2 design will stop it from failing structurally. No implementation, however good, gets past this, because the limit comes from how the levels are defined.

These three are the main basis for judging whether you need to bring in L3.

Ceiling 1: multi-hop reasoning

Someone in operations asks: “What changed in this year’s after-sales policy compared with last year’s, and what does it mean for our home appliance category?”

L1 fails here because the question needs comparative reasoning across several documents and across time. The system first has to find the relevant clauses for both years, compare them, judge how “substantive” the differences are, and then map them onto a specific category. That’s a chain of at least four reasoning steps.

The single retrieve-then-generate flow of standard RAG supports, by its structure, one retrieval and one generation. It can’t retrieve again based on the output of a middle step, and it can’t break a question into sub-questions and solve them one by one. RAGFlow’s engineering review of the year also points out that “multi-hop questions” are one of the typical cases with an obvious semantic gap in RAG systems, where traditional retrieval methods have limited effect.⁸

What this needs is the ability to break a question into sub-tasks and work through them step by step. That’s what L3 (Tool Executor) does.

Ceiling 2: dynamic planning

Operations wants to list one product on the platform’s sites in three countries at the same time. Each country has different requirements for labels, certification and returns, and they’re tangled together: changing the packaging for one country may affect certification in another.

L2 fails because there’s no way to write the path for this task in advance. The system first needs to learn each country’s basic requirements, then find where they conflict, and then decide what to look up next based on those conflicts. In other words, what step N should be depends on what step N-1 returned, and that kind of dynamic planning is exactly what L2’s hard-coded routing can’t handle.

L2 can handle static logic like “if it’s a question about listing a product, take path A.” It can’t handle the dynamic logic of “look at what retrieval returned, then decide between path A and path B.”

What this needs is the ability to re-plan later steps during execution, based on intermediate results. That, too, is what L3 is for.

Ceiling 3: keeping state across systems

Operations needs to push the price changes for a sale event to three places: the product system, the membership system and the customer service knowledge base.

L1 and L2 fail here because, as defined above, L1 doesn’t change the state of outside systems, and L2’s paths are fixed in advance. Neither can handle a task that coordinates several outside systems, keeps them consistent with one another, and decides the next action from how each system responds.

This kind of task needs external tools, plus keeping track of state across several of them and handling partial failures (the product system updated, say, but the customer service knowledge base timed out). That is L3 (Tool Executor) work. Spanning several systems, on its own, doesn’t call for L4.

Together, these three draw the boundary of L1/L2: when a task needs reasoning that carries across steps, coordination across systems, or a path re-planned from intermediate results, L1/L2 has reached its limit. That is why L3 exists, and I’ll take it apart in the next post.

The business decision: when to stay at L1/L2, and when to move up

In real business decisions, these checks help you judge whether a use case needs to go beyond L1/L2.

If all of the following hold, L1/L2 is the right choice:

• The task has a clear standard of right and wrong (it doesn’t depend on subjective judgment)
• The system only needs to give out information, and doesn’t need to change the state of outside systems
• Each query stands alone, with no intermediate results to carry from one step to the next
• Answers are needed within 3 seconds

If any of the following signs show up, take a serious look at L3:

• Users’ questions can only be answered by combining several documents
• The system has to decide what to look up in the second step based on what the first step found
• A useful answer requires calling an external API and judging what it returns
• The situation calls for thinking twice, such as the system checking its own answer

Fix precision through the retrieval path first, not with a bigger model. If L1’s answers aren’t accurate enough, start by reviewing the chunking strategy, the quality of the embeddings and the retrieval logic, before switching to a larger model or jumping to L3.

Closing: saying it right matters more than saying more

Back to the air fryer. When the bot said “two years,” the problem lay outside the model: two clauses had been cut apart, retrieval didn’t bring them back together, and the system attached no source that would let someone check. Most of the work in L1 and L2 goes into exactly these places: cutting well, finding precisely, citing the source, and refusing when it should.

A system built this way doesn’t look especially clever, but it behaves consistently and its boundaries are clear, so users don’t have to check it every time. Every answer with a source adds a little trust; every answer forced out without support uses some up.

Being predictable also means having limits. When a question has to be broken into steps, when the next step depends on the result of the last one, or when the work means actually changing several systems, L1 and L2 can’t carry it. The next post is about L3, the level from which the model decides the next step.