Layering a CLI agent's models under a budget cap
TL;DR
Under a budget-capped subscription like opencode Go, picking a model isn't about token price but requests per 5 hours, the one quality source still alive (vals.ai), and tool-calling reliability — the decisive criterion no benchmark measures. The result: one model per preset, a frontier delegated outside the agent, and tools constrained rather than instructed.
Picking a model for a coding agent was easy while one dominated. In 2026, under a budget-capped subscription like opencode Go — a pool of open models for a fixed monthly fee — the question becomes: which model on which task, given a finite quota and half the models carrying documented failures? Here’s the method I used to layer my pi setup, with numbers measured in July 2026.
Why isn’t token price enough to pick a model?
Because a capped budget isn’t spent in tokens, it’s spent in requests. opencode Go caps at $60/month — which, smoothed over the rolling windows, is about $2.77 per effective working day. The token rate says nothing about what you’ll actually burn, because it ignores the model’s verbosity.
The decisive number, opencode Go publishes itself: requests per 5-hour window, which already folds in both price and verbosity. Measured in July 2026:
| Model | Req. / 5 h |
|---|---|
| Grok 4.5 | 120 |
| GLM-5.2 | 880 |
| DeepSeek V4 Pro | 4,300 |
| Hunyuan 3 (hy3) | 30,100 |
Grok 4.5 burns 36× more quota than DeepSeek V4 Pro, and 251× more than Hy3. A model that’s “cheap per token” but chatty can drain your quota in a morning. That’s the first filter: a model at 120 req/5 h can’t be your default — it’s reserved for sparing use.
How do you read the benchmarks that are left?
The 2026 trap: the independent, reproducible leaderboards froze or changed course, and what stays current is mostly self-reported. I detail that collapse — and why it calls for suspicion rather than surrender — in the dispatch Choosing a model when benchmarks can’t keep up. On the method side, hold two guardrails.
One source stays alive and independent: vals.ai (SWE-bench Verified, continuously updated). July 2026 excerpt:
| Model | Score | Access |
|---|---|---|
| Claude Opus 5 | 97.0 | Claude quota |
| Kimi K3 | 93.4 | Go |
| Grok 4.5 | 86.6 | Go |
| GLM-5.2 | 82.8 | Go |
| Kimi K2.7 Code | 78.2 | Go |
| DeepSeek V4 Pro | 77.4 | Go |
First guardrail: it saturates. The top five fit within four points, with ~2 points of standard error. Use it to eliminate the bottom, not to rank the top. Second: none of these numbers speak to your stack — no public data exists on the PHP/Symfony performance of these models (SWE-bench is Python, Terminal-Bench is shell). Any ranking applied to your code is a proxy extrapolation, hence the local test at the end.
What’s the real criterion for a CLI agent?
Tool-calling reliability — and no quality benchmark measures it. A CLI agent does nothing else: read, edit, run bash, rewrite. A model that scores high on generation but botches one tool call in five is unusable as an agent.
The only one whose tool-calling is independently verified is Kimi K2.7 Code (MCPMark Verified: 81.89%, ahead of Claude Opus 4.8 and DeepSeek V4 Pro). The tax: it’s the slowest of the pack (~51 t/s). For a role that must chain tools without derailing, it’s the most solid signal in the file.
Conversely, an opencode issue reports that DeepSeek V4 Pro/Flash and MiMo return a 500 error as soon as the tools parameter is present, on the Go backends. Careful: this claim is contested — two independent sources (OpenHands CI, a control group) don’t reproduce it, but both go through litellm, not the Go backend. Treat it as an unsettled risk, to verify yourself before handing these models any volume.
What does the layering look like once settled?
One model per preset, each assignment carrying its rationale. Here’s the layering deployed in July 2026 on pi via opencode Go:
| Preset | Model | Tools | Reason |
|---|---|---|---|
plan | glm-5.2 | read, bash, grep, find, ls | Best open IQ, 191 t/s |
implement | kimi-k2.7-code | read, bash, edit, write | Only independently verified tool-calling |
review | grok-4.5 | read, bash, grep, find, ls | Lowest verbosity; a review burns few requests, its only tenable use under 120 req/5 h |
bulk | deepseek-v4-flash | read, write | Cheapest, but tools restricted as a precaution |
fast | glm-5.2 | read, bash, edit, write | 191 t/s, TTFT 1.35 s |
vision | grok-4.5 | read, bash, grep, find, ls | Image input, but 120 req/5 h → sparing |
hard | kimi-k2.7-code high | all | Long-horizon: multi-file PRs, deep refactors |
The real config is public: presets.json (this table as JSON) and preset.ts (the extension that enforces it).
The point not in the table: the frontier doesn’t run through the agent. For a task that deserves the best model with all the guardrails, I don’t switch models inside pi — I delegate to interactive Claude Code. The reason is mechanical: calling a premium model from pi via a subprocess breaks interactive prompts and blinds supervision. The open layering covers the daily grind; the frontier lives elsewhere, deliberately.
Should you instruct the model or constrain it?
Constrain it. The bulk preset cuts its tools to ["read", "write"] instead of asking the model to avoid complex tool calls: an applied constraint holds where a prompt instruction gets forgotten on a long run. It’s the most transferable lesson of the layering — when a behavior matters, enforce it, don’t ask for it. The mechanism and exact config are in this TIL.
Frequently Asked Questions
- What is model layering in a coding agent?
- Assigning a different model to each type of task (plan, implement, review, handle volume) instead of one model for everything. A multi-provider agent like pi supports it natively; the point is to align each model's cost and reliability with what the task actually demands.
- Why not just pick the cheapest model per token?
- Because token price says nothing about real verbosity or caps. opencode Go publishes a more honest number: requests per 5-hour window. Grok 4.5 burns 36× more than DeepSeek V4 Pro at equal budget — a gap invisible on the token rate.
- Which LLM benchmarks are still reliable in 2026?
- Almost none independently. HAL is archived, Aider's official leaderboard hasn't moved since November 2025, Terminal-Bench holds only self-reported scores. SWE-bench and LiveCodeBench still run, but their public rankings are mostly self-declared. Only vals.ai remains an up-to-date independent harness — and even it saturates at the top.
- How do you know a model is reliable at tool-calling?
- No quality benchmark measures it directly. You cross field sources (harness GitHub issues, telemetry) and above all test it yourself on a real tool chain. For a CLI agent, that criterion outweighs the raw code score.