02 / Flagship Engineering
Shipped
LLM API Gateway
Provider-neutral LLM infrastructure with stable aliases, retries, fallback chains, usage accounting, and analytics.
FastAPIPythonOpenAIGeminiSQLiteDockerPytest
Case study
What this project demonstrates
Designed to make provider switching and failure recovery an infrastructure concern instead of leaking vendor-specific behavior into every application.
Architecture
Client → stable alias/router → provider adapter → retry/timeout policy → fallback provider → normalized response + usage ledger
Engineering highlights
- Stable model aliases decouple application code from provider-specific model names.
- Ordered fallback chains, retry/backoff with jitter, gateway timeouts, and normalized errors.
- Structured JSON output, embeddings, normalized token usage, cached-token accounting, and cost estimates.
- Persistent usage analytics with async tests, routing/fallback coverage, CI, CodeQL, and Dependabot.
Boundaries / limitations
Provider behavior and pricing still change externally; the gateway normalizes interfaces but cannot eliminate upstream availability or model-quality differences.