Reference architectures
Real inference stacks, not hypotheticals — official vendor blueprints and production case studies that named their framework, orchestration, and silicon in public.
Enterprise RAG pipeline
Retrieval-augmented generation for enterprise search, knowledge assistants, and copilots
NVIDIA's official blueprint for enterprise RAG: NIM-packaged models behind a LangChain orchestrator, with a vector database (Milvus or Elasticsearch) for retrieval. Silicon isn't pinned in the blueprint itself — it's specified per-model in NIM's separate support matrix.
Generative AI inference architecture and best practices
Real-time and batch inference for RAG, chatbot, and agentic workloads
AWS's own prescriptive guidance covers the full decision space rather than one fixed stack — it explicitly compares vLLM, TensorRT-LLM, and Triton, and NVIDIA GPUs against AWS's own Trainium/Inferentia silicon, across self-managed Kubernetes, ECS, and SageMaker.
Serving open LLMs on GKE with TPUs
Real-time LLM serving (reference models: Gemma 3/4, Qwen 3 32B)
Google's own pre-configured pattern for serving open-weight models: vLLM on GKE, backed by single-host TPU v5e or v6e nodes, provisioned with Terraform rather than clicked together by hand.
Multi-node inference with NVIDIA Dynamo on AKS
Large-scale LLM inference serving, benchmarked on GPT-OSS 120B
Azure's AKS engineering team benchmarked NVIDIA's Dynamo disaggregated-serving platform across a full GB200 NVL72 rack on AKS, reaching 1.2M tokens/sec on GPT-OSS 120B — one of the few public examples of rack-scale NVLink inference on a hyperscaler's managed Kubernetes.
Doubling inference throughput on AMD MI325X
Character.AI's production chat-companion serving
Character.AI, DigitalOcean, and AMD jointly published the tuning behind a 2x throughput gain serving Qwen3-235B on MI325X — topology-aware GPU allocation plus FP8 and custom attention kernels, running on a non-NVIDIA cloud GPU host.
“AMD Instinct MI325X platform on DigitalOcean achieved 2x production inference throughput.”
A fully custom inference stack, from scratch
Character.AI's production chat-companion serving at 20,000+ queries/sec
Rather than building on an off-the-shelf serving engine, Character.AI wrote their own — custom int8 attention kernels, cross-layer KV-cache sharing, and tree-structured inter-turn caching. A useful counterpoint: the popular frameworks aren't the only viable path at extreme scale.
“Character.AI designs its model architecture, inference stack, and product from the ground up.”
One engine, two silicon vendors
Fireworks AI's multi-tenant model-serving platform
Fireworks built one proprietary engine (FireAttention) that runs production traffic on both NVIDIA H100 and AMD MI300 — a rare public example of a serving stack designed for silicon portability from day one, rather than a single-vendor bet.
“Fireworks has been using AMD MI300 hardware in production since the launch of LLaMA 405B.”
TensorRT-LLM as the production serving layer
Baseten's hosted model-inference platform
A straightforward, publicly confirmed data point: Baseten's platform runs NVIDIA's own TensorRT-LLM directly in production rather than an alternative serving engine — useful as a baseline for how the "default" NVIDIA-native stack performs at a real inference-hosting company.
“We use both TensorRT and TensorRT-LLM in production to optimize model performance.”
Custom engine, rack-scale Blackwell
Together AI's inference platform for open-weight and frontier models
Together AI's engine spans a 72-GPU NVLink mesh with NVFP4 quantization on Blackwell-generation hardware — a look at what a from-scratch serving engine looks like once it's tuned specifically for rack-scale NVLink domains rather than single 8-GPU nodes.
“Together's inference engine handles this dynamically, extracting maximum efficiency from each GPU-hour.”