The idea for this study emerged while I was taking an Anthropic learning path, part of which covers choosing between augmented LLMs, workflows, and agents based on two aspects that seem simple but have important consequences in production: the predictability of the execution path and the degree of autonomy given to the model.
That led me to a more specific question: what changes when the decision about the next step stops being entirely in application code and moves, to some extent, to the model? I decided to explore this by building the Controlled Autonomy Lab, an implementation that performs the same production-incident analysis task using six different architectures.
Throughout the project, the question eventually crystallized into one sentence: Who determines the next steps: deterministic code or the model?
The lab compares behavior, grounding, execution trajectory, tool use, latency, tokens, and causal authority across different combinations of OpenAI, Anthropic, and Groq. So far, the experimental record includes 234 cells distributed across three separate, frozen generations.
Enlarge ↗The six patterns#
All patterns receive the same task: analyze an incident from a controlled set of evidence. What changes is how the work is decomposed and who controls the execution trajectory.
Enlarge ↗The augmented LLM concentrates all the work in a single call.
In chaining, three calls execute sequential stages: fact extraction, severity assessment, and action recommendations.
Routing uses the model to classify the incident and select a path, but the code converts the result into a known enum and rejects routes outside the allowed set.
In parallelization, three specialists analyze metrics, recent changes, and dependencies simultaneously. A fourth model call then synthesizes the results. The synthesis content remains probabilistic; the fan-out and fan-in topology is defined by code.
The evaluator-optimizer works through generation, evaluation, and possible revisions until a quality criterion is met or an attempt limit is exhausted.
The bounded agent, meanwhile, receives five read-only tools and can decide which ones to consult and in what order:
get_service_metrics,get_recent_deployments,get_dependencies,search_runbook, andget_previous_incidents.
This autonomy has explicit limits: the code allows at most 6 steps and 8 tool calls, validates every tool name against an allowlist, and restricts access to the active incident. There are no tools for shell access, restart, rollback, configuration changes, or production writes. If the agent exceeds its budget, execution is interrupted and the partial state is preserved as benchmark evidence.
Why I implemented it without frameworks#
The project's pyproject.toml declares dependencies = []. Adapters for the Anthropic Messages API, OpenAI Responses API, and OpenAI-compatible APIs are implemented directly on the Python standard library, mainly using http.client, json, and urllib.parse. There is no LangChain, LangGraph, or provider SDK in the runtime.
This decision increased the amount of code needed for serialization, parsing, error handling, and tool calls. For the lab's purpose, however, that transparency is useful: I can follow the exact payload sent to each provider, how a tool call is represented, and how the response returns to the application domain.
The structure follows Clean Architecture, divided into domain, application, and adapters. CI checks these boundaries and blocks violations covered by the project's rules. The same quality gate runs Ruff, MyPy strict, tests with a minimum of 80% coverage, Bandit, pip-audit, and architectural validation.
This level of control was particularly useful because an important part of the experiment consists precisely of observing where application-defined behavior ends and model-introduced behavior begins.
The evaluation stack#
After the first runs, it became clear that comparing only latency, tokens, or a generic quality metric would be insufficient. The project ended up developing five evaluation layers, each answering a different question.
The Grounding Evaluation v1 is deterministic. It compares the response with the incident fixture through explicit rules, checking versions, timestamps, measurements, unit equivalences, associations between evidence, and strong causal language. If the response mentions a nonexistent version or associates a value with the wrong context, for example, that produces a hard failure. A subsequent semantic evaluation can recover a legitimate paraphrase that a lexical rule did not recognize, but it is not given authority to reverse those deterministic signals.
The Epistemic Evaluation v4.1 emerged after a limitation observed in the benchmarks themselves. An earlier version used an uncertainty-preservation indicator. In the breadth benchmark, this field was positive in all 59 successful cells, including responses that also contained causal overclaims. The metric had saturated. The next version began examining the relationship between the stance of the evidence and the degree of certainty used in the response.
The four incidents used by the lab were designed to represent different situations:
Enlarge ↗This makes it possible to treat phrases such as “may have caused” differently. In a correlational incident, that caution may be appropriate. In an incident whose cause is explicitly confirmed, the same wording may represent excessive hesitation.
The remaining layers complete the process. Claim Evaluation v2 separates facts, inferences, proposed actions, and unsupported claims. Semantic Escalation v2.1 handles cases in which a conservative deterministic rule failed to recognize a valid paraphrase. Independent Semantic Judge v2.2 makes it possible to decouple the model that generated the response from the one participating in semantic evaluation.
Enlarge ↗What the 234 experimental cells showed#
The lab maintains three separate generations:
Enlarge ↗Each generation has a frozen commit and evidence packs accompanied by SHA-256 checksums. Rate limits and provider errors remain recorded as availability data; quality metrics consider only cells that completed successfully. This separation matters because an API failure says something about availability in that execution context, but does not tell us the quality of a response that was never produced.
The first result that caught my attention appeared in the repeated benchmark. Chaining, with three sequential calls, showed the lowest specific grounding across all three provider bundles: 90.0% for OpenAI, 67.4% for Groq, and 82.1% for Anthropic. The augmented LLM, using only one call, reached 100.0%, 88.3%, and 95.3%, respectively.
The number of calls, therefore, did not show a monotonic relationship with grounding. One plausible hypothesis is that sequential handoffs create new points where unsupported inferences can appear and then propagate. The benchmark was not designed to prove that mechanism, so I prefer to treat it as a hypothesis for further experiments.
The breadth benchmark revealed another type of problem. Some cells achieved 100% specific grounding while also producing causal overclaims. This can happen when a response uses all available evidence correctly but turns a temporal correlation into a stronger causal claim than the data allows.
The distinction became quite concrete in the results:
Enlarge ↗Parallel is an interesting example: it maintained high grounding, at 94.9%, while accounting for seven overclaims across the ten observed cells. Fan-out expanded the analysis available to the synthesis stage, but the synthesis still had to decide how much causal authority to assign to that evidence.
At the other extreme, the bounded agent completed the ten observable breadth cells with 95.4% average grounding and no detected causal overclaims. Augmented and evaluator-optimizer had slightly higher grounding but registered three and four overclaims.
This result prompted new questions about the relationship between dynamic evidence search and causal restraint. Because the agent can consult new information before concluding, it has a means of evidence acquisition that the experiment's fixed workflows lack. Even so, ten cells are insufficient to confidently attribute the result to the architectural mechanism.
Enlarge ↗The Epistemic v4.1 generation helped test that signal with a more specific stance evaluation. There were 72 attempts and 70 completions. Among them, the evaluator classified 20 as aligned, 41 as overclaimed, 6 as no-position, and 3 as over-hedged. The incidents requiring greater restraint accounted for most overclaims: INC-001, with correlational evidence, and INC-004, inconclusive, totaled 29 of the 41 detected cases.
At the architectural level, the bounded agent showed 4 overclaims in 12 cells. Among fully observed patterns, this was the generation's lowest rate, at 33.3%. Augmented had 5/12, chaining 9/12, and routing 11/12. Routing was particularly useful for seeing the difference between factuality and stance. In the Epistemic v4.1 generation, it maintained average grounding of 95.1% and had 11 of its 12 responses classified as overclaimed.
Enlarge ↗These numbers also help put the agent's earlier result in perspective. The zero observed in breadth did not repeat in the next generation. The restraint signal continued to appear in relative terms, but now with four cases detected by the evaluator. It is exactly this kind of variation that makes preserving historical generations important, rather than choosing only the experiment that produces the most attractive result.
When the model becomes part of the control plane#
Routing revealed a finding of a different nature: in the breadth benchmark, OpenAI and Anthropic classified INC-001 into the deployment route, while Groq chose dependency. The fixture, architecture, and classification instruction were the same.
In a system where the model's classification decides which workflow will execute, this divergence affects application behavior beyond the response text. Changing a provider/model/configuration bundle can change the path the system follows.
For that reason, the project's routing converts the response into a known enum and fails closed on unexpected values. The model participates in the choice, but the code still defines the universe of accepted routes.
Enlarge ↗This behavior is also a reason to treat provider, model, API, and configuration as an experimental bundle. The observed differences cannot be attributed to the model alone because transport, output limits, tokenization, reasoning configuration, and service conditions also vary.
What these results do not yet tell us#
The breadth and epistemic benchmarks have an important limitation: there is only one run per cell. They describe observed behavior but do not allow variance or statistical significance to be estimated. The four incidents also belong to the same infrastructure domain. A legal-analysis, credit, customer-service, or software-development system may show completely different patterns. The fixtures are synthetic and controlled. That makes evaluation easier because there is a known reference for each piece of evidence, but real incidents contain noise, gaps, conflicting timestamps, and incomplete sources. Cost is not yet normalized in dollars either. The project records tokens and latency, but providers have different tokenizers, prices, and output limits. In breadth, for example, OpenAI and Anthropic used an output cap of 4,000 tokens, while Groq used 900.
These differences prevent reading the project as a model leaderboard. The repository maintains a specific section on limitations and non-claims, and frozen reports document threats to validity. This care became part of the methodology: if the goal is to evaluate how much certainty a response should express, conclusions about the benchmark itself must also respect the reach of the available evidence.
For those choosing between workflows and agents#
After these experiments, my way of thinking about autonomy became more concrete. There are tasks whose path is known in advance and can be represented as code. In those cases, workflows offer operational control that is straightforward to reconstruct, test, and observe. In other situations, the system needs to acquire new evidence during execution and decide what to investigate based on what it has just found. The ability to choose the next step begins to have value at that point. The lab's bounded agent explores exactly that second situation, but within a narrow boundary. It chooses which evidence to consult and in what order, while permissions, budgets, and the ability to act remain deterministically defined.
The benchmarks showed an interesting signal of causal restraint in this design. Breadth recorded zero overclaims across the ten observable cells; the Epistemic v4.1 generation recorded four in twelve, still the lowest rate among fully observed patterns. Grounding remained high in both generations. There is still plenty of room to test this behavior with more fixtures, repetitions, and other domains.
For architectural decisions today, I still find it more useful to start with the simplest solution that addresses the problem, measure actual behavior, and add autonomy when there is a concrete need that the current flow does not handle well.
This principle seems basic, but in practice it prevents buying future flexibility in advance at the cost of more possible states, more failure points, and greater difficulty auditing.
The Controlled Autonomy Lab continues to evolve precisely from these questions.
Repository and benchmark data: