Skip to content
Gatewyse

Architecture Overview

Gatewyse sits between your applications and AI providers, scoping configuration and policy through a deployment to organization to department cascade. It provides unified API access, intelligent routing, security guards, budget enforcement, and observability across 49 AI providers.

Monorepo Structure

ai-gateway/
+-- packages/
|   +-- shared/       # Types, Zod schemas, utilities (consumed by all packages)
|   +-- license/      # Ed25519 license JWT verifier (EE distribution)
|   +-- server/       # Express 5 API server (gateway + admin API)
|   +-- worker/       # BullMQ background job processor
|   +-- admin/        # Nuxt 4 SPA dashboard (PrimeVue 4)
|   |   +-- e2e/      # Playwright E2E tests for the admin dashboard
|   +-- content/      # Shared content assets
|   +-- docs/         # Astro Starlight documentation site
|   +-- website/      # Astro marketing site
+-- docker/           # Docker Compose / Swarm stacks, Dockerfiles, nginx
+-- k8s/              # Kubernetes manifests
+-- tests/
|   +-- unit/         # Jest unit tests
|   +-- integration/  # Jest integration tests with Docker MongoDB/Redis
|   +-- smoke/        # Live server smoke tests (6 phases)

Key Technologies

ComponentTechnology
RuntimeNode.js 24+
API ServerExpress 5
Admin DashboardNuxt 4, PrimeVue 4, Pinia
DatabaseMongoDB (Mongoose ODM)
Cache / QueueRedis (ioredis)
Job QueueBullMQ
Package Managerpnpm 10+ (workspaces)
AuthJWT + RBAC + SSO (OIDC/SAML)

Request Lifecycle

Every gateway request follows this pipeline:

Client Request
      |
      v
[1] Authentication        -- Validate API key or JWT
      |
      v
[2] Scope Resolver        -- Merge deployment -> org -> department policy
      |
      v
[3] Capability Guard      -- Enforce per-key capability scoping
      |
      v
[4] Rate Limiting         -- Per-API-key and per-scope sliding window
      |
      v
[5] Validation            -- Zod schema check on request body
      |
      v
[6] Format Detection      -- Identify OpenAI vs Anthropic request shape
      |
      v
[7] Normalizer            -- Translate to UnifiedRequest internal format
      |
      v
[8] RAG Injection         -- Optionally inject retrieved context
      |
      v
[9] Prompt Guards         -- PII detection, injection defense, content filter,
      |                       toxicity scoring, token/cost limits, custom rules
      |  (blocked? return error)
      v
[10] Budget Check         -- Verify spending limits (deployment/org/dept/user)
      |
      v
[11] Semantic Cache       -- Lookup similar responses; short-circuit on hit
      |  (cache hit? return cached response)
      v
[12] Usage Tracking       -- Wire post-response tracking + audit logging
      |
      v
[13] Routing              -- Resolve provider chain using configured strategy
      |                       (one of 11 strategies, see Routing Strategies)
      v
[14] Provider Execution   -- Adapter translates to provider format, sends request
      |                       with retry/backoff on transient failures
      |  (failure? try next provider in chain)
      v
[15] Response Formatting  -- Adapter translates provider response to unified format
      |
      v
[16] Cache Write          -- Store response in semantic cache
      |
      v
[17] Usage + Audit        -- Record tokens, cost, latency + immutable audit log
      |
      v
Client Response

The order above matches packages/server/src/routes/gateway/chat.routes.ts. Other gateway routes use the same pipeline with their own validation schema and capability scope.

Data Flow Between Services

+------------+       +----------+       +-----------+
|   Client   | ----> |  Server  | ----> | Provider  |
| (your app) | <---- | (Express)| <---- | (OpenAI,  |
+------------+       +----+-----+       | Anthropic,|
                          |             | etc.)     |
                          v             +-----------+
                    +-----+------+
                    |  MongoDB   |  Configs, users, organizations,
                    |            |  audit logs, usage records
                    +-----+------+
                          |
                    +-----+------+
                    |   Redis    |  Cache, rate limits, routing
                    |            |  counters, session state, queues
                    +-----+------+
                          |
                    +-----+------+
                    |   Worker   |  Budget resets, SIEM export,
                    | (BullMQ)   |  backup jobs, health checks
                    +------------+

Admin Dashboard

The admin dashboard is a Nuxt 4 single-page application (SPA mode, ssr: false) that communicates with the server’s /api/admin/* endpoints. It provides management interfaces for:

  • Organizations, departments, users, and RBAC role management
  • Provider and model configuration
  • Routing rule management
  • Guard configuration
  • Budget creation and monitoring
  • Usage analytics and audit logs
  • Semantic cache management
  • System settings and backups

API Surface

The server exposes six groups of endpoints.

Gateway API (/v1/*) — OpenAI-compatible endpoints consumed by applications:

  • /v1/chat/completions — Chat completions (streaming and non-streaming)
  • /v1/messages — Anthropic Messages API
  • /v1/completions — Text completions
  • /v1/embeddings — Vector embeddings
  • /v1/audio/transcriptions, /v1/audio/translations, /v1/audio/speech — Audio
  • /v1/images/generations — Image generation
  • /v1/rerank — Document reranking
  • /v1/video/generations — Video generation
  • /v1/models — List available models
  • /v1/files — File registry with gateway-issued IDs
  • /v1/batches — Asynchronous batch jobs
  • /v1/responses — Responses API
  • /v1/vector_stores — Vector stores and file search
  • /v1/realtime — Realtime sessions, plus the WebSocket multiplexer at /v1/realtime/connect
  • /v1/usage, /v1/budget — API key self-service

Admin API (/api/admin/*) — Dashboard management endpoints for organizations, departments, users, roles, api-keys, providers and provider-overrides, models and model-overrides, routing, guards, budgets, usage, requests, audit logs, cache, settings, documents, backups, model intelligence and the licence.

Super-Admin API (/api/super-admin/*) — system/{health,metrics,config,reset,security/rotate-encryption-key} and queues/{status,:queueName/pause,:queueName/resume}.

SCIM 2.0 (/scim/v2) — Users and Groups provisioning, gated on the scim licence feature.

Public authentication (/api/auth/sso/*) — :slug/initiate (the organization slug, or system for the deployment’s own provider), callback (OIDC) and saml/acs. These are unauthenticated by design; they are how a session begins.

Infrastructure — /health (liveness), /ready (readiness, 503 when MongoDB or Redis is down), /metrics (Prometheus), and /openapi.json plus /docs (Swagger). The last two are withheld in production unless ENABLE_SWAGGER=true, and are gated on the same condition as each other: publishing a machine-readable API description while withholding the human-readable one beside it would be an oversight rather than a decision.

Next Steps