The cheaper coding run still has to finish the job
In brief
TokenEconomizer makes a useful cost argument. Its research controller and its current plugin describe different stopping behavior, and that difference belongs beside the savings.
Originally published on Jack Gross.
TokenEconomizer makes a useful cost argument. Its research controller and its current plugin describe different stopping behavior, and that difference belongs beside the savings.
What I checked today
I followed an October 29 LinkedIn discussion back to TokenEconomizer, the project from researchers affiliated with UC Berkeley, Microsoft Research and Microsoft Security. The appeal is practical: work in the same repository repeats, so knowledge and tool choices from earlier sessions might make later sessions cheaper. This is my reading of the sources on October 31, not a local installation or a reproduction of the paper. I want to know what the bill counts, what still gets solved, and which implementation produced the result.
The denominator matters
The authors report 18.4-30.9% lower mean cost across five agent-model combinations on 277 SWE-Bench tasks from six repositories. They include TokenEconomizer model calls and cache pricing. Each task is a separate session, while tasks in a repository run in sequence from an initially empty learned state. That setup measures a history-dependent workflow. It does not establish the same saving on one unfamiliar repository or one isolated request. Pass@1 improved in four of five combinations; describing success as universally unchanged or better would erase the exception.
A research stop and a plugin nudge
The project research page describes redirecting repeated failures and stopping a run if they persist, including a forced-stop backstop. The current plugin README explicitly says its guard never stops a turn itself. It redirects, then tells the user how to interrupt. I cannot collapse these descriptions into one product claim. A research controller can end an expensive run; an unattended plugin session may keep spending after its warning. The measured study and the behavior of the version someone installs need separate records.
Remembering has a maintenance cost
The README describes micro-skills learned from repeated failures and a model proposer. Facts need confirmation across three sessions and can be dropped when contradicted. Micro-skills are off by default because proposing them costs model calls and their value depends on the work. Tool pruning and the loop guard are on by default. Those settings are part of the experiment, not footnotes. Turning everything on and calling the result the default would make a comparison hard to interpret.
An unused tool can still be needed
Tool pruning uses past non-use as evidence and changes the host settings. Its scope differs: individual Claude Code tools, Codex capability packages, or whole Copilot CLI MCP servers. Pi has no pruning; opencode has none yet and is experimental. The documented restore command returns removed entries, and restored tools are not removed again. My proposed test includes a rarely used tool on a later task. If restoring it takes a human several minutes, that time belongs in the acceptance record even when the token bill falls.
My proposed trial
Use an authorized disposable repository, pin the plugin and host versions, and run a fixed task sequence with the same model and billing assumptions. Include a build-command change after facts accumulate, a task needing a previously unused tool, and a repeated failure. Record solved tasks, billed cost, elapsed time, restore work and actual stop behavior. Repeat the baseline rather than treating one run as stable. The small record below is explanatory data, not executable enforcement or a benchmark result.
{"trial":"proposed_not_run","plugin_revision":"PIN_ME","host_version":"PIN_ME","settings":{"microskills":false,"toolprune":"propose"},"results":{"solved":null,"billed_cost":null,"restore_minutes":null,"stop_observed":null}}What would change my decision
I would accept a cost reduction only alongside the task-success and recovery evidence that produced it. The code is Apache 2.0 according to the LICENSE file, but model calls can still cost money. Learned state is stored locally according to the README; that is not proof that model-assisted processing never sends context to the configured provider. No production account, repository or credential was connected for this article. The unanswered question is whether the particular installed version saves work under the failures its users actually face.