Corso · 7 capitoli
Serving Open Models in Production
Self-hosting as an engineering discipline: vLLM, GPU math, OpenAI-compatible gateways, observability, the honest cost model, and hardening a shared GPU across tenants.
Cosa saprai fare
- Six chapters that take an open-weight checkpoint from a container that answers one request to a deployment that survives real traffic, a real budget, and more than one tenant.
- Why serving a model to many users at once is a different engineering problem than running one on your own machine.
- Turning weight count and context length into a real VRAM budget, instead of guessing and watching a container crash on startup.
- One stable address in front of two correctly sized backends, with routing, failover, and health checks doing the work no client should have to.
- Turning a vague sense that requests feel slow into a percentile target you can actually alert on, and tracing which hop is responsible when it's breached.
- Why the GPU's hourly rate is rarely the number that actually determines cost per token, and what chapter 4's SLO target quietly costs in reserved headroom.
Cosa contiene
- 1Serving Open Models: Start Here
Six chapters that take an open-weight checkpoint from a container that answers one request to a deployment that survives real traffic, a real budget, and more than one tenant.
- 2vLLM & the Serving Stack
Why serving a model to many users at once is a different engineering problem than running one on your own machine.
- 3GPU Sizing & Quantization at Scale
Turning weight count and context length into a real VRAM budget, instead of guessing and watching a container crash on startup.
- 4The OpenAI-Compatible Gateway
One stable address in front of two correctly sized backends, with routing, failover, and health checks doing the work no client should have to.
- 5Observability & SLOs for Inference
Turning a vague sense that requests feel slow into a percentile target you can actually alert on, and tracing which hop is responsible when it's breached.
- 6The Cost Model, Honestly
Why the GPU's hourly rate is rarely the number that actually determines cost per token, and what chapter 4's SLO target quietly costs in reserved headroom.
- 7Hardening & Multi-Tenancy
Turning chapter 3's virtual keys into an actual isolation guarantee, so one tenant's burst can never quietly spend another tenant's headroom.
Ottieni un certificato
Completa tutti i capitoli per ricevere il tuo certificato di completamento.