Cours · 7 chapitres
Serving Open Models in Production
Self-hosting as an engineering discipline: vLLM, GPU math, OpenAI-compatible gateways, observability, the honest cost model, and hardening a shared GPU across tenants.
Ce que tu sauras faire
- Six chapters that take an open-weight checkpoint from a container that answers one request to a deployment that survives real traffic, a real budget, and more than one tenant.
- Why serving a model to many users at once is a different engineering problem than running one on your own machine.
- Turning weight count and context length into a real VRAM budget, instead of guessing and watching a container crash on startup.
- One stable address in front of two correctly sized backends, with routing, failover, and health checks doing the work no client should have to.
- Turning a vague sense that requests feel slow into a percentile target you can actually alert on, and tracing which hop is responsible when it's breached.
- Why the GPU's hourly rate is rarely the number that actually determines cost per token, and what chapter 4's SLO target quietly costs in reserved headroom.
Ce qu’il contient
- 1Serving Open Models: Start Here
Six chapters that take an open-weight checkpoint from a container that answers one request to a deployment that survives real traffic, a real budget, and more than one tenant.
- 2vLLM & the Serving Stack
Why serving a model to many users at once is a different engineering problem than running one on your own machine.
- 3GPU Sizing & Quantization at Scale
Turning weight count and context length into a real VRAM budget, instead of guessing and watching a container crash on startup.
- 4The OpenAI-Compatible Gateway
One stable address in front of two correctly sized backends, with routing, failover, and health checks doing the work no client should have to.
- 5Observability & SLOs for Inference
Turning a vague sense that requests feel slow into a percentile target you can actually alert on, and tracing which hop is responsible when it's breached.
- 6The Cost Model, Honestly
Why the GPU's hourly rate is rarely the number that actually determines cost per token, and what chapter 4's SLO target quietly costs in reserved headroom.
- 7Hardening & Multi-Tenancy
Turning chapter 3's virtual keys into an actual isolation guarantee, so one tenant's burst can never quietly spend another tenant's headroom.
Obtiens un certificat
Termine tous les chapitres pour recevoir ton certificat de réussite.