We opened a month of call logs
After running our bots on a Windows device and a Mac mini for a while, a full month of model call logs had piled up. Opening them was a little funny. Even tiny chores like a one-line summary or a title length check were calling the biggest model. We had simply left the default alone because it was convenient.
We sorted the work into three tiers
We classified all 1,240 calls from four weeks, one by one.
• Light work: summaries, classification, format checks (71% of calls)
• Normal work: first drafts, translation polishing (22%)
• Heavy work: strategic judgment, final review of long documents (7%)
Then we assigned a model size to each tier and wrote it as a one-page rule.
light -> small model
normal -> mid-size model
heavy -> large model (called only for the final review)
The numbers confirmed it
Running the same rule for the next four weeks brought the cost index from 100 down to 44. On quality, which we had worried about, the rework rate moved from 8% to 9%, almost unchanged. Light work made up over 70% of all calls, so changing just that slice made a big difference.
Cutting cost was not about trimming. It was about placement.
Notes for small teams
• Change nothing in week one. Just log the calls. Savings without measurement are only a feeling.
• Do not retire the big model. Put it at the last gate.
• Keep the rule to one page. If it runs long, neither bots nor people read it.
In the early days of a startup, plugging leaks is often visible before growing revenue is. This experiment showed that with one small table.