I typed a two-word prompt—"Run tests"—into my AI coding agent, and my API billing dashboard immediately registered 28,400 input tokens. I had not a...
✕
Wait! Don't Go Yet 🚀
Get our FREE eBook "10 Programming Tips That Changed Everything" when you subscribe!
No spam. Unsubscribe anytime.
Really good article, John! 😸🙌
I especially liked the point that “more tools” doesn’t automatically mean “more capable agent.”
It’s really easy to keep adding MCP servers because each one looks useful in isolation, but once all of those schemas and outputs start competing with the actual code and task context, the trade-off becomes much less obvious.
The distinction between focused micro-tools, small CLIs, and on-demand skills also makes a lot of sense to me.
“Small, sharp, single-purpose tools” feels very close to how I want to design developer tooling in general too. 😸
Thanks so much, nyaomaru! 🙌
That "looks useful in isolation" trap is exactly how bloat creeps into modern workflows. Keeping tools small, sharp, and single-purpose really is the sweet spot. Both for human developer experience and for keeping AI context windows clean.
Really glad this resonated with your philosophy on building developer tooling.
Good breakdown. One distinction I’d make in the table: 25k is per-request context occupancy; 250k over a 10-step loop is cumulative input processed, not 250k sitting in one context window. Both matter, but labeling them separately makes the cost-versus-window tradeoff much clearer.
Fair point - that is a really helpful distinction.
Separating single-turn window occupancy (which drives attention dilution) from cumulative input processed across the loop (which drives latency and API spend) definitely makes the tradeoff clearer.
Appreciate you calling that out for readers going through the numbers.
Glad that helped. The reason I started labeling them separately is pretty practical: the occupancy number is what I use to decide whether to unload a server mid-session, and the cumulative number is what I look at when a bill looks off. I mixed them up a couple of times and ended up fixing the wrong one.
This highlights an important design principle that I think will become increasingly relevant as AI agents evolve: more capabilities don't automatically translate into more intelligence. We often equip agents with every possible integration, assuming that greater access means better results. But there's a hidden trade-off: every additional tool competes for context, attention, and the model's ability to distinguish what actually matters.
I particularly like the shift toward focused micro-tools. It mirrors good software architecture: small, composable components with clearly defined responsibilities are often easier to maintain, understand, and optimize than massive systems trying to do everything.
I'd add one more consideration: tool selection should be dynamic, not just minimal. An agent working on a database issue shouldn't need to carry the entire GitHub, Slack, and Jira tool catalog throughout every reasoning step. Loading capabilities when they're actually needed could become just as important as optimizing the model itself.
We're entering an era where building better AI agents may involve giving them fewer tools, not more. The real measure of an agent isn't how much it can access, but how effectively it uses the context it has available.
Excellent reminder that context is an engineering resource, not an unlimited container.
The dynamic angle is a great addition — loading tools per task phase rather than per session feels like the natural next step after capping the count. The 78% to 99% first-try accuracy jump in the article already shows how much selection quality matters, and swapping tool sets as the task shifts could push that further. I wonder how you'd decide when to swap — a planner step up front, or letting the agent request a tool set as it goes?
Thanks so much, Marie. I love the framing that context is an engineering resource rather than an unlimited container.
Your point about dynamic capability loading really nails the next step here. An agent debugging a database issue has zero reason to carry Slack, Jira, or GitHub schemas through every reasoning loop, so loading tools just-in-time is quickly becoming just as critical as keeping individual tools small and composable.
Appreciate you bringing such a thoughtful architectural perspective to the thread.
The 28,400-token figure for a two-word prompt is a good concrete baseline. One thing I've run into that makes this worse than the schema count suggests: the same MCP server can expose overlapping tools (list_workflow_runs vs get_commit_status vs search_issues), and the model spends real tokens on the description of the near-miss before picking the right one. Trimming the server list to two or three tools per domain — or wrapping the server with a thin "search" tool that returns the schema for the real tool on demand — cut my per-request overhead by roughly the same 60-70% range you're implying.
Did you measure the hit on the model's actual picks, or only the token cost? With seven servers, I'd expect the bigger loss to be in the model calling a plausible-but-wrong tool from the list and having to recover from the error.
Your intuition is spot on. The recovery loops from picking a "plausible-but-wrong" tool hurt far more than the static schema tax itself.
In my benchmark runs, first-try tool selection accuracy dropped to 78% with the full 52-tool stack (compared to 99% with the 4-tool setup), and every misfire burned an extra full turn of context just to recover from validation errors or wrong endpoints. Wrapping servers behind a thin on-demand schema search tool is a great way to sidestep that overlap entirely.
Appreciate you sharing those numbers and insights from your own setup.
The schema overhead math tracks with what we see in document extraction pipelines too, though tool return payloads often dwarf the definition tax. An unguarded PDF or web-fetch MCP tool that dumps raw layout JSON or an entire 40-page document into stdout burns tens of thousands of tokens on the very first turn, pushing relevant code or instructions into the middle of the context window.
Returning a compact structural outline first (headings, table counts, and page ranges) and letting the agent request specific sections or page slices on demand keeps both the schema surface small and the tool outputs bounded.
Spot on. The output payload blowup is often the bigger silent killer in real-world pipelines.
Dumping raw layout JSON or a 40-page document on turn one immediately triggers that "lost in the middle" degradation. Returning a lightweight structural outline first and letting the agent fetch specific slices on demand is such a clean pattern for keeping context bounded on both ends.
Appreciate you sharing this practical insight from document extraction workflows.
This matches the tax we see when a client eagerly registers every capability up front: each MCP server's tool schemas are resent on nearly every turn, so a long server list compounds into a steady per-turn token bill even before the model acts. A practical mitigation is lazy, context-scoped tool exposure — register a narrow toolset for the current task and pull in the rest only when the workflow calls for it, and trim descriptions to the minimum the model needs to pick the right call. On the desktop side we apply the same rule: don't preload every module and permission surface at startup; load what the active view actually needs.
Useful breakdown, especially the split between the schema tax and the accuracy drop at 52 tools vs 4.
One thing I'd add to the dynamic-loading discussion above: tool definitions sit at the very start of the prompt, so swapping tools mid-session usually invalidates the prompt cache for everything after them. With a static toolset, the schema cost is mostly cached and cheap. With per-phase swapping you trade that for a cache miss each time. So "fewer tools" and "load on demand" have different cost profiles. Caching helps the bill and latency, but it does nothing for the tool-selection accuracy problem.
Did your benchmark run with caching on? Curious how much of the 3.4s vs 1.1s gap survives with a warm cache.
This is an excellent article, and I think you're highlighting one of the most underestimated problems in the current AI agent ecosystem: we've become so focused on expanding what our agents can do that we're forgetting to consider the cognitive and computational cost of giving them access to everything.
The example of a two-word prompt generating 28,400 input tokens before the model even begins responding is particularly striking. It illustrates how easily a seemingly harmless productivity improvement can become a hidden performance and cost problem. Connecting more integrations feels like adding capabilities, but every additional tool definition competes for valuable context that could otherwise be used for source code, error messages, architectural decisions, and actual reasoning.
What I find especially interesting is the distinction between token consumption and reasoning quality. The problem isn't simply that developers are paying for unnecessary input tokens. It's that excessive tool definitions introduce noise into an environment where context selection and attention are already critical. When an agent has dozens of overlapping tools available, choosing the correct one becomes another reasoning challenge. More capabilities can actually produce less predictable behavior.
I particularly appreciate your recommendation to use focused micro-tools and lightweight TypeScript CLIs. It reminds me of an important principle from traditional software engineering: good abstractions should reduce complexity, not merely move it somewhere else. An agent doesn't necessarily need access to an entire GitHub API when all it needs to do is inspect a pull request, summarize changed files, or retrieve a CI failure.
The same applies to database access, observability platforms, and project management integrations. A small tool that performs one well-defined operation and returns carefully filtered results can be significantly more useful than a massive integration exposing every possible endpoint.
I'd also emphasize that tool output optimization deserves as much attention as schema optimization. Even a perfectly designed MCP server can overwhelm an agent if it returns enormous JSON payloads, irrelevant metadata, or entire log histories. Keeping the input schema small is only half the battle; the output needs to be equally intentional.
One additional architectural approach worth exploring is dynamic tool discovery. Instead of registering every available tool at the beginning of every conversation, an agent could initially receive a small set of core capabilities and discover additional tools only when a task requires them. This would make tool availability contextual rather than global, potentially reducing both schema overhead and tool-selection ambiguity.
There is also an important security dimension here. Every additional integration potentially expands the agent's attack surface, permissions, and opportunities for unintended actions. Reducing the active toolset is therefore not just a token optimization strategy; it can also become a meaningful least-privilege security practice.
One small distinction I'd add to the benchmarking discussion: multiplying the schema footprint by ten agent steps is useful for illustrating cumulative request overhead, but it isn't necessarily equivalent to ten times that many simultaneously occupied context-window tokens. Context reuse, prompt caching, conversation history management, and provider-specific tool serialization all influence the actual cost. Separating cumulative token consumption from peak context occupancy would make an already compelling analysis even stronger.
Ultimately, I think your central argument points toward a broader principle for building effective AI agents:
The goal should not be to maximize the number of tools an agent can access, but to minimize the amount of information and capability it needs to accomplish a particular task reliably.
We're entering a phase where context engineering deserves the same architectural attention we've traditionally given to database optimization, API design, and resource management.
Excellent contribution. This is exactly the kind of practical discussion the AI developer community needs.
I'm curious about one thing: Do you think the future of MCP will move toward dynamic, task-aware tool loading, where agents discover and activate capabilities on demand, rather than keeping dozens of integrations permanently registered? I suspect that architectural shift could be just as important as the tools themselves.