DSPy in practice: what changes when the router is already an LLM but is not yet compilable

In the latest posts, I discussed data modeling in Redis Stack and DSPy as a framework that treats prompts as compilable code.

Today I want to show how these two subjects connect in practice, using e-commerce as the example domain and reproducing a real migration I carried out in a conversational assistant.

The point I want to make clear before we begin: the problem DSPy solves is not the absence of AI in routing. It is the absence of a contract for that AI's output.

The starting point: Agno, tools with DTOs, Redis as text#

The original assistant was built with the Agno framework. The architecture was elegant and worked: an agent with a detailed system prompt, a set of tools with typed DTOs, and Redis as state storage (key/value, text).

Each tool had its input DTO:

This works. The LLM reads the system prompt, understands which tool to use, builds the arguments, and the DTO validates them on entry. For a catalog with controlled volume and predictable queries, this architecture delivers.

Where the Agno architecture begins to show its limits#

As the catalog and query variety grew, three problems emerged. None is a bug. They are structural limitations of routing based on a free-form prompt.

Problem 1: the LLM output has no verifiable contract.

Agno lets the LLM decide which arguments to pass to the tool. The LLM is intelligent and usually gets it right. But “usually” is not auditable. When the user asks for “men's Nike sneakers up to 300 reais with free shipping” and the tool receives {"categoria": "tênis", "preco_maximo": 300} without frete_gratis=true, you have no systematic mechanism to detect it other than reviewing logs manually.

Problem 2: large prompt + model update = unpredictable behavior.

When the provider updated the model version, some classifications silently changed. Queries that previously went to BuscarProdutosTool started going to RankingCategoryTool, without an exception or a failing test. The behavior simply changed, and a user complaint revealed it.

Problem 3: improving routing is a manual trial-and-error process.

Adding a new use case (for example, searching by product compatibility) means editing the system prompt, manually testing the affected scenarios, and hoping that the order of instructions in the prompt does not break cases that already worked. There is no dataset, no metric, and no replay of historical cases.

The root cause of all three problems is the same: routing was correct in intent but opaque in execution.

What DSPy changes (and what it does not)#

Before showing the code, it is important to say what stays the same.

The LLM still does the routing. We did not go back to if/elif. We did not replace intelligence with rules. What changes is how the output of that decision is specified, validated, and improved over time.

The central change is this:

The Signature: specifying the contract#

The Signature is not a prompt. It has no “answer like this,” “be concise,” or “do not make things up.” It is a declarative contract: this goes in, this comes out, and these are the classification rules. DSPy automatically builds the actual prompt, including the compiled few-shot demos.

RouterOutput: typed output before reaching the tool#

In the Agno model, the LLM built the tool kwargs directly. If it forgot frete_gratis, the DTO received None and the search ignored the filter. No logging, no detection.

With RouterOutput, the LLM produces JSON that goes through coercion before any tool is called. A missing frete_gratis field becomes an explicit None, making it auditable, observable in the Langfuse trace, and trackable in the dataset for recompilation.

The asymmetric metric: where business priority becomes code#

For the optimizer to work, you define a metric. The temptation is to use simple accuracy: correct fields/total fields.

The problem is that this metric accepts something absurd. An example that gets scope wrong but the other 10 fields right receives 0.91 and is accepted as a demo by BootstrapFewShot. The model learns that the wrong scope with the right filters is acceptable.

It is not. The wrong scope means the wrong tool. The wrong tool means Redis receives a completely different query, and the user receives a silently incorrect answer.

This asymmetry is intentional and needs documentation:

scope is worth 2.50 because it is the routing decision
sort_direction is worth 0.50 because desc vs asc in a search has less impact than sending the query to the wrong scope

BootstrapFewShot rejects any demo with the wrong scope, regardless of how many fields were correct. It is the optimizer acting as automated QA with the rules you defined.

Output coercion: the layer the documentation ignores#

The LLM does not always deliver the same format. Sometimes clean JSON. Sometimes JSON inside a Markdown fence. Sometimes a scope field with a value slightly different from what was expected ("buscar_produtos" instead of "busca_catalogo").

In the Agno architecture, Pydantic caught this at the tool entry point and returned an error for the LLM to try again. With DSPy, coercion happens before any tool is called:

The important architectural difference: in the Agno model, a parsing failure could lead the LLM to retry with a reformulation, adding latency and consuming extra tokens. Here, the failure is absorbed during coercion and the graph continues with scope=geral. The worst case is a generic answer, not a user-visible error.

What changed in Redis alongside DSPy#

The migration to DSPy came with a move from text Redis to Redis Stack. The two changes are complementary.

In the previous model, the tool fetched products from the API, serialized the result as a JSON string, and saved it to Redis as a simple cache. The LLM received the raw JSON and formatted the answer.

With Redis Stack, the document is indexed at ingestion time:

The RouterOutput produced by DSPy feeds directly into the QueryBuilder that constructs the RediSearch query. The scope determines the index. Filters become predicates. The projection varies with intent (Slim for listings, Fat via JSON.GET for details).

The complete chain:

What LangGraph gained from this#

The router_node in LangGraph became declarative:

The graph makes no business decision. It reads a state field. DSPy made the decision, with a verifiable contract, a metric capturing business priority, and a reproducible improvement process.

When the provider updated the model, the response was not to edit 80 lines of system prompt and hope. It was to run the recompilation script with the new model as the target and validate the metric on the validation set before deployment.

What became more complex (but is worth it)#

This model has more components than the original Agno version:

Declarative Signature instead of a system prompt
Pydantic RouterOutput instead of free-form kwargs for the tool
A defensive coercion layer
An asymmetric metric with documented weights
An NDJSON dataset with supervised examples
A versioned recompilation script

This is real complexity. There is no point in downplaying it.

The trade-off that justifies this complexity: the router's behavior moved from “I believe the prompt is correct” to “I have evidence that the model correctly classifies 94.7% of validation-set queries, with zero tolerance for scope errors.”

For an assistant with low volume and stable use cases, Agno with a good prompt does the job. For an assistant that grows in volume, changes models periodically, and needs to track exactly where routing failed, DSPy solves what the prompt can no longer solve alone.