Usage
Token, request, latency, and cost reporting across teams, users, providers, models, and dates. Input and output tokens stay separate.
Issue 01 Capabilities Everything between
Everything between
teams and models.
The Self-Hosted LLM Gateway is a data-plane proxy, admin API, project-knowledge system, and dashboard for shared agent infrastructure. It governs the request path, then connects that traffic to the repositories and people behind it.
localhost:3333/overview LIVE
01 Access
Teams come before traffic
Create a team, add users, and issue scoped cpk_live keys before clients send traffic through the data plane.
- Keys rotate independently without redeploying applications
- Team and user records own budgets, keys, and history
- Revoked keys stop at the proxy before reaching upstream providers
02 Upstreams
Provider accounts hold the real secrets
Register upstream provider accounts in the dashboard. The gateway stores encrypted credential payloads and injects the right upstream auth at request time.
- OpenAI, Anthropic, and DeepSeek have dedicated data-plane routes
- Passthrough protocols cover configured providers such as Google, Mistral, Bedrock, Azure, and Codex-style endpoints
- Client proxy keys are stripped before upstream forwarding
03 Indirection
Routing is explicit and overridable
Built-in catalog routing handles known provider models. Manual aliases override that route for a public model name, team, or upstream provider account.
- Route by provider path and model catalog
- Override selected models globally or per team
- Keep callers stable while moving traffic to a new provider account
04 Guardrails
Quotas stop overruns at the door
Spend, token, request-rate, body-size, output-token, stream-duration, and concurrency policies are checked before or during the upstream call.
- Policy scopes include team, user, key, provider, and model
- Spend reservations are reconciled after usage is extracted
- Redis-backed counters keep enforcement distributed across replicas
05 Intelligence
Repositories become the unit of understanding
The gateway resolves normalized Git remotes into stable, tenant-scoped projects and joins each repository to participants, sessions, accepted model requests, cost, knowledge, and distillation coverage.
- SSH and HTTPS remotes converge without relying on checkout folder names
- New model traffic is attributed even when canonical session upload is disabled
- Knowledge-store failures remain visible instead of being reported as zero
06 Insight plugins In development
Insight logic can run outside the gateway
The in-progress plugin contract lets independently deployed gRPC processors generate insights while the gateway retains parsing, policy matching, tags, provenance, run history, and project materialization.
- Register services with gRPC target, transport, timeout, and a health test
- Run every matching global, tenant, or project configuration independently
- Persist configuration revision, input hash, results, warnings, and failures
07 Observability
See what happened—and what the system learned.
These surfaces keep lean metering, prompt-heavy evidence, generated knowledge, and client health distinct so each can be reviewed on its own terms.
Payloads
Captured request and response bodies are stored separately from usage rows, with retention controls for prompt-heavy history.
Sessions
Uploaded Pi session metadata and provider payloads are joined by conversation ID so a multi-turn run can be read back in order.
Insights
Agent observations and distillation output become attributed decisions, mistakes, lessons, and context inside each repository project.
Prompt feedback
Generated reflections highlight unclear requests and concrete ways to reduce rework without mixing prompt advice into operational memory.
Client health
Extension version metadata and aggregated failures show which client integration is active and where model, session, or project operations are failing.
Run it yourself.
Run proxy, admin API, UI, Postgres, and Redis. Start with governed model traffic, then enable the project intelligence features your team needs.