The Self-Hosted LLM Gateway sits between engineering teams, AI agents, and model
providers. It routes requests and controls spend, then connects that
traffic to repositories, sessions, durable insights, and shared
project memory—all on infrastructure you operate.
The runtime flow starts after administrators connect providers and
issue user keys. Every data-plane request is authenticated, checked,
rewritten, forwarded, and recorded.
01
Team key
Each user receives a scoped cpk_live key from the admin panel. Provider credentials stay inside the gateway.
02
Provider endpoint
Clients call provider-shaped routes such as /openai/v1/chat/completions, /anthropic/v1/messages, or /deepseek/v1/chat/completions.
03
Route & quota
The proxy resolves the model route, checks spend and rate guardrails, and rejects over-budget traffic before forwarding.
04
Credential inject
The gateway strips client auth, injects the configured upstream credential, and forwards only to an allowed provider path.
05
Meter & capture
Usage, cost, latency, and optional conversation payloads are recorded for the team, user, key, provider, and model.
02 Admin panel
Every team and every dollar, on one screen.
Live spend, token mix per model, and per-team budgets are visible in
the admin dashboard. Historical request payloads remain separate from
lean usage rows.
localhost:3333/overviewLIVE
The overview, metering live traffic
03 Project intelligence
From model traffic to repository-level insight.
Normalized Git remotes give each repository a stable identity. The gateway
brings its sessions, accepted model requests, contributors, cost,
memory, generated insights, and prompt feedback into one project view.
Repository identity converges across checkout paths and Git URL forms
Agent observations and distillation stay attributed to their source
Coverage and unavailable states are shown instead of hidden as zero
Keep provider credentials at the gateway and issue scoped team keys to callers.
agent observation · source linked Mistake
Do not infer project identity from a checkout folder; resolve the normalized remote.
session distillation · source linked
04 Context memory
The proxy learns from real agent sessions.
The gateway turns opted-in session evidence and agent observations into
project knowledge. The distiller extracts useful lessons, the compactor
keeps the brief focused, and server-side policy controls whether agents
can retrieve or receive that context.
Resolve the repository and retrieve relevant shared context when enabled.
during work
Record observations, failures, fixes, and code pointers.
after idle
Distill mistakes into memory so the next agent avoids them.
05 Insight plugins
Bring your insight logic. Keep the control plane.
The gateway is adding a plugin boundary for independently deployed,
stateless gRPC insight processors. Each matching policy can run its own
service while the gateway keeps canonical transcript parsing, tenant and
project scope, tag governance, provenance, and durable run history.
Generate insights behind a small versioned service contract.
02
Scoped configurations
Run matching global, tenant, and project policies independently.
03
Tag registry
Supply known tags and govern whether processors may propose more.
04
Auditable runs
Persist revisions, input hashes, results, usage, warnings, and failures.
06 Capabilities
One control plane, from access to insight.
Govern shared model access, investigate the request path, and understand
what agents are doing across repositories without stitching together
provider dashboards and local session files.
Teams
Create teams, users, scoped keys, and per-team budgets without putting provider secrets in app code.
Providers
Connect provider accounts from the dashboard. Built-in routes cover OpenAI, Anthropic, DeepSeek, and configured passthrough protocols.
Aliases
Use built-in model routing and manual aliases to move callers between upstream accounts without changing clients.
Quotas
Enforce spend, token, rate, body-size, and concurrency guardrails at team, user, key, provider, or model scope.
Usage reports
Report request count, input and output tokens, cost, team, user, provider, model, and date.
Projects
Resolve Git remotes into stable repository identities and see the people, sessions, requests, cost, and knowledge attached to each project.
Insights
Turn agent observations and session distillation into durable decisions, lessons, mistakes, and context with source attribution.
Prompt feedback
Review generated reflections on request clarity and concrete ways to reduce rework in future sessions.
Payloads
Review captured request and response bodies when conversation capture is enabled. Keep usage rows lean and prompts separate.
Sessions
Inspect uploaded Pi sessions and the captured provider calls that belong to each conversation.
Insight services
Run multiple insight policies through independently deployed gRPC processors while the gateway keeps transcript parsing, scope, tags, provenance, and run history.
Extension health
Track deployed client versions and aggregate extension failures by operation, project, conversation, and last-seen time.
Context memory
Compact repeated project lessons into curated memory and control retrieval or injection from the server side.
07 Cost governance
Cost visibility before the invoice.
Set team, user, key, provider, or model policies. Spend is reserved
before the upstream call and reconciled after usage is extracted.
Monthly and daily spend policies with live counters
Rate, token, body-size, and concurrency limits
Cost attributed to team, user, key, provider, and model
Monthly spend by team$80.58
Platform team$42.87
Research team$23.95
Support team$13.76
08 Usage reports
Tokens by team, model, and date.
Slice consumption by team, model, and date at daily or monthly
granularity. Input and output token counts are shown separately.
The gateway runs as a Go data-plane service, an admin API, and a static
dashboard. Postgres stores configuration and history; Redis backs
distributed quota and revocation checks.
# local services, then proxy + admin API + dashboard$ docker compose up -d postgres redis
$ make ui-install
$ make dev
# point a provider-compatible client at it$ export OPENAI_BASE_URL=\
http://localhost:8080/deepseek/v1
✓ data plane :8080 · admin API :8081 · UI :3333
