dspy.Refine: runtime self-correction without recompiling the model

In the posts about DSPy, I showed how BootstrapFewShot compiles a router that classifies intent and extracts structured parameters. That is DSPy in offline mode: you compile once, version the JSON, and use it in production.

There is another way to use DSPy that does not involve compilation: dspy.Refine. It provides runtime self-correction: the model generates an answer, evaluates it against a reward function, and tries again if it does not pass.

This post explains how the assistant's answer-generation module uses dspy.Refine to ensure output compliance without relying solely on the prompt.

The problem Refine solves#

The generation module receives the context returned by tools (catalog data, product details) and produces the final answer for the user.

With a simple dspy.Predict, the flow is: the LLM receives context → generates an answer → delivers it.

If the answer violates a business rule, such as giving a purchase directive, omitting a mandatory notice, or using data that is not in the context, the critique_node intervenes afterward. This adds latency and can still result in a fallback message if the violation is severe.

With dspy.Refine, the violation is detected and corrected within the generation module itself, before reaching the critique_node. The critique remains as a final guardrail, but receives answers that have already passed through a self-correction layer.

The reward function: compliance as a runtime metric#

The reward function is the heart of Refine. It evaluates the generated answer and returns a score between 0.0 and 1.0. If the score is below the configured threshold, Refine generates a new answer.

Enlarge ↗

Each rule contributes 0.25 to the score. If the total score is below threshold=1.0, Refine generates a new answer, with DSPy injecting automatic feedback about which rule failed.

The module: Refine over a Predictor with an attempt counter#

Enlarge ↗

The attempt counter exists for two reasons. The first is Langfuse: with advisor_refine_attempt and advisor_total_refine_attempts in the trace metadata, you know exactly what proportion of requests required Refine to make more than one call. The second is alerting: if total_attempts > 1 appears in more than 20% of traces, the Signature prompt needs adjustment.

The latency trade-off that needs monitoring#

dspy.Refine(N=3, threshold=1.0) can make up to 3 LLM calls per request.

In practice, most requests pass on the first attempt because the Signature prompt was already written to avoid the most common violations. But there are cases where the context returned by tools contains data that leads the LLM to generate sentences that fail one of the rules.

Monitoring in Langfuse filters by advisor_total_refine_attempts > 1:

• If this filter returns fewer than 5% of traces, Refine is working as a safety net rather than a routine correction mechanism, which is fine.

• If it returns between 5% and 20%, investigate which rules fail most often. The Signature probably needs more explicit examples for those rules.

• If it returns more than 20%, Refine is becoming a routine correction mechanism instead of a safety net. This signals that the Signature or prompt is misaligned with the data arriving from tools.

The answer to high rates is not increasing N, but improving the prompt so that the first attempt passes and using Refine only for the remaining cases.

Refine vs critique_node: different layers, not redundant ones#

A question naturally arises: if generation includes dspy.Refine, why is there still a critique_node?

Refine evaluates the generated answer against local rules, those in the reward function. The critique_node goes further:

• Detects hallucinations by comparing the answer with the actual context returned by tools

• Applies regex checks for violations Refine does not cover (for example, an inappropriate FGC disclaimer for specific products)

• Has its own Circuit Breaker for DSPy infrastructure failures

Refine is the first barrier, inside the generation module. critique_node is the final barrier, outside it. They are complementary layers, not redundant ones.

Refine improves the critique approval rate, and the critique ensures nothing gets through even when Refine fails or is bypassed.

Next week: QueryBuilder, how to build FT.SEARCH queries from a Pydantic model, with special-character escaping and fuzzy matching with tokenization rules.