{"meta":{"title":"Reference architectures","subtitle":"Real inference stacks, not hypotheticals — official vendor blueprints and production case studies that named their framework, orchestration, and silicon in public.","as_of":"2026-08"},"count":9,"architectures":[{"id":"nvidia-rag-blueprint","title":"Enterprise RAG pipeline","source_type":"vendor","source_name":"NVIDIA","source_title":"NVIDIA RAG Blueprint","source_url":"https://build.nvidia.com/nvidia/build-an-enterprise-rag-pipeline","application":"Retrieval-augmented generation for enterprise search, knowledge assistants, and copilots","framework":[{"label":"NVIDIA NIM microservices (Nemotron LLMs + retrieval/reranking/extraction models)"}],"orchestration":[{"label":"Kubernetes + NIM","node_id":"k8snim"},{"label":"LangChain-based RAG orchestrator server"}],"silicon":[],"cloud":[{"label":"Docker, Helm/Kubernetes, or Helm/OpenShift — not tied to one cloud"}],"summary":"NVIDIA's official blueprint for enterprise RAG: NIM-packaged models behind a LangChain orchestrator, with a vector database (Milvus or Elasticsearch) for retrieval. Silicon isn't pinned in the blueprint itself — it's specified per-model in NIM's separate support matrix."},{"id":"aws-genai-inference","title":"Generative AI inference architecture and best practices","source_type":"vendor","source_name":"AWS","source_title":"Generative AI Inference Architecture and Best Practices on AWS (Prescriptive Guidance)","source_url":"https://docs.aws.amazon.com/prescriptive-guidance/latest/gen-ai-inference-architecture-and-best-practices-on-aws/introduction.html","application":"Real-time and batch inference for RAG, chatbot, and agentic workloads","framework":[{"label":"vLLM","node_id":"vllm"},{"label":"TensorRT-LLM","node_id":"trtllm"},{"label":"Triton Inference Server","node_id":"triton"}],"orchestration":[{"label":"Self-managed Kubernetes, Amazon ECS, or Amazon SageMaker"}],"silicon":[{"label":"NVIDIA GPUs"},{"label":"AWS Trainium","node_id":"trainium2"},{"label":"AWS Inferentia","node_id":"inferentia2"}],"cloud":[{"label":"EC2 g4dn/g5/g6, trn1/trn1n, inf1/inf2"}],"summary":"AWS's own prescriptive guidance covers the full decision space rather than one fixed stack — it explicitly compares vLLM, TensorRT-LLM, and Triton, and NVIDIA GPUs against AWS's own Trainium/Inferentia silicon, across self-managed Kubernetes, ECS, and SageMaker."},{"id":"gcp-gke-tpu-vllm","title":"Serving open LLMs on GKE with TPUs","source_type":"vendor","source_name":"Google Cloud","source_title":"Serve Open LLMs on GKE Using TPUs with a Pre-Configured Architecture","source_url":"https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/serve-open-models-tpu-terraform","application":"Real-time LLM serving (reference models: Gemma 3/4, Qwen 3 32B)","framework":[{"label":"vLLM","node_id":"vllm"}],"orchestration":[{"label":"Google Kubernetes Engine (Autopilot or Standard)"}],"silicon":[{"label":"Cloud TPU v5e","node_id":"tpuv5e"},{"label":"Cloud TPU v6e (Trillium)"}],"cloud":[{"label":"GKE, provisioned via Terraform + CLI wrappers for single-host TPU nodes"}],"summary":"Google's own pre-configured pattern for serving open-weight models: vLLM on GKE, backed by single-host TPU v5e or v6e nodes, provisioned with Terraform rather than clicked together by hand."},{"id":"azure-aks-dynamo-gb200","title":"Multi-node inference with NVIDIA Dynamo on AKS","source_type":"vendor","source_name":"Microsoft Azure","source_title":"Scaling Multi-Node LLM Inference with NVIDIA Dynamo and ND GB200 NVL72 GPUs on AKS","source_url":"https://blog.aks.azure.com/2025/10/24/dynamo-on-aks","source_date":"2025-10-24","application":"Large-scale LLM inference serving, benchmarked on GPT-OSS 120B","framework":[{"label":"NVIDIA Dynamo (disaggregated prefill/decode; vLLM, TensorRT-LLM, or SGLang backends)"}],"orchestration":[{"label":"Azure Kubernetes Service (AKS)"}],"silicon":[{"label":"GB200 NVL72 (Grace + Blackwell, NVLink-domain rack)"}],"cloud":[{"label":"Azure ND GB200-v6 VM series"}],"summary":"Azure's AKS engineering team benchmarked NVIDIA's Dynamo disaggregated-serving platform across a full GB200 NVL72 rack on AKS, reaching 1.2M tokens/sec on GPT-OSS 120B — one of the few public examples of rack-scale NVLink inference on a hyperscaler's managed Kubernetes."},{"id":"character-ai-amd-2026","title":"Doubling inference throughput on AMD MI325X","source_type":"case_study","source_name":"Character.AI","source_title":"Technical Deep Dive: How DigitalOcean and AMD Delivered a 2x Production Inference Performance Increase for Character.ai","source_url":"https://blog.character.ai/technical-deep-dive-how-digitalocean-and-amd-delivered-a-2x-production-inference-performance-increase-for-character-ai/","source_date":"2026-01-13","application":"Character.AI's production chat-companion serving","framework":[{"label":"Custom inference stack — AITER kernels, FP8 quantization, DP2/TP4/EP4 parallelism"}],"orchestration":[{"label":"DigitalOcean Kubernetes (DOKS)"}],"silicon":[{"label":"AMD Instinct MI325X"}],"cloud":[{"label":"DigitalOcean"}],"quote":"AMD Instinct MI325X platform on DigitalOcean achieved 2x production inference throughput.","summary":"Character.AI, DigitalOcean, and AMD jointly published the tuning behind a 2x throughput gain serving Qwen3-235B on MI325X — topology-aware GPU allocation plus FP8 and custom attention kernels, running on a non-NVIDIA cloud GPU host."},{"id":"character-ai-custom-2024","title":"A fully custom inference stack, from scratch","source_type":"case_study","source_name":"Character.AI","source_title":"Optimizing AI Inference at Character.AI","source_url":"https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/","source_date":"2024-06-20","application":"Character.AI's production chat-companion serving at 20,000+ queries/sec","framework":[{"label":"In-house inference engine — not vLLM or TensorRT-LLM"}],"orchestration":[],"silicon":[],"cloud":[],"quote":"Character.AI designs its model architecture, inference stack, and product from the ground up.","summary":"Rather than building on an off-the-shelf serving engine, Character.AI wrote their own — custom int8 attention kernels, cross-layer KV-cache sharing, and tree-structured inter-turn caching. A useful counterpoint: the popular frameworks aren't the only viable path at extreme scale."},{"id":"fireworks-fireattention-v3","title":"One engine, two silicon vendors","source_type":"case_study","source_name":"Fireworks AI","source_title":"FireAttention V3: Enabling AMD as a Viable Alternative for GPU Inference","source_url":"https://fireworks.ai/blog/fireattention-v3","source_date":"2024-10-15","application":"Fireworks AI's multi-tenant model-serving platform","framework":[{"label":"FireAttention (proprietary inference engine/kernels)"}],"orchestration":[],"silicon":[{"label":"NVIDIA H100","node_id":"h100"},{"label":"AMD MI300","node_id":"mi300x"}],"cloud":[],"quote":"Fireworks has been using AMD MI300 hardware in production since the launch of LLaMA 405B.","summary":"Fireworks built one proprietary engine (FireAttention) that runs production traffic on both NVIDIA H100 and AMD MI300 — a rare public example of a serving stack designed for silicon portability from day one, rather than a single-vendor bet."},{"id":"baseten-tensorrt","title":"TensorRT-LLM as the production serving layer","source_type":"case_study","source_name":"Baseten","source_title":"High Performance ML Inference with NVIDIA TensorRT","source_url":"https://www.baseten.co/blog/high-performance-ml-inference-with-nvidia-tensorrt/","source_date":"2025-12-11","application":"Baseten's hosted model-inference platform","framework":[{"label":"TensorRT-LLM","node_id":"trtllm"}],"orchestration":[],"silicon":[],"cloud":[],"quote":"We use both TensorRT and TensorRT-LLM in production to optimize model performance.","summary":"A straightforward, publicly confirmed data point: Baseten's platform runs NVIDIA's own TensorRT-LLM directly in production rather than an alternative serving engine — useful as a baseline for how the \"default\" NVIDIA-native stack performs at a real inference-hosting company."},{"id":"together-ai-blackwell","title":"Custom engine, rack-scale Blackwell","source_type":"case_study","source_name":"Together AI","source_title":"Foundational Research Powering Efficient Inference at Scale","source_url":"https://www.together.ai/blog/foundational-research-powering-efficient-inference-at-scale","source_date":"2026-05-04","application":"Together AI's inference platform for open-weight and frontier models","framework":[{"label":"In-house inference engine — dynamic batching and routing"}],"orchestration":[],"silicon":[{"label":"NVIDIA Blackwell (GB200 NVL72, HGX B200)","node_id":"b200"}],"cloud":[],"quote":"Together's inference engine handles this dynamically, extracting maximum efficiency from each GPU-hour.","summary":"Together AI's engine spans a 72-GPU NVLink mesh with NVFP4 quantization on Blackwell-generation hardware — a look at what a from-scratch serving engine looks like once it's tuned specifically for rack-scale NVLink domains rather than single 8-GPU nodes."}]}