Spend quotas are more useful than request limits
A thousand tiny requests and one huge context request are not the same operational event. Budget controls need to follow cost, not just count.
Rate limits are not budgets
Request limits are useful abuse protection. They protect edge capacity and stop accidental loops from overwhelming a service. They are not enough for shared LLM usage because model traffic is priced by token volume, model class, and provider-specific billing rules.
One request can be cheap. One request can also carry a large context window, stream a long answer, and cost more than thousands of small calls. Treating both as one unit hides the risk.
Spend is the control teams understand
Most teams reason about model usage in money. A monthly team budget, a daily user cap, and a per-key safety limit are easier to explain than a mesh of request-per-minute and token-per-minute values.
The gateway still supports rate and token guardrails, but the primary quota dimension is spend. The proxy estimates the request cost before forwarding, reserves budget, then reconciles the reservation when actual usage is known.
That reserve and reconcile flow is important. It avoids letting parallel requests slip past the same budget and gives usage reports a clean path from estimate to actual cost.
Quotas should apply where accountability lives
Budgets can make sense at several levels:
- Tenant or team, for monthly department control.
- User, for individual accountability.
- Key, for risky automation or CI jobs.
- Provider or model, for experimental routing.
The useful default is not to make every limit mandatory. The useful default is to make the commercial limit clear and let tighter operational limits be added where needed.
Refund paths are part of correctness
Quota systems often get the deny path right and the cleanup path wrong. If a request reserves spend and then fails before usage is recorded, the reservation must be refunded. If a panic, timeout, or client disconnect happens, the quota state still has to settle.
That is why quota code belongs near request lifecycle code, not as an afterthought in reporting. The accounting path needs to see forwarding, streaming, upstream errors, and final usage extraction.
Reports should explain the decision
A quota denial should not be a mystery. Operators need to know which scope was checked, what the limit was, what had already been spent, and why the request was rejected. The same usage rows that power billing should also make quota decisions auditable.
When spend is the primary dimension, quota reports become budget reports. That is the point: the system protects the thing the team actually cares about.