People assume model hosting means one big GPU per model and a lot of idle time. We run several models per box and treat a short queue as normal. Here is how the scheduler works and where it stops being clever.
A 70B model on an A100 80GB takes most of the VRAM and most of the compute. A 7B model on the same card takes a small slice. If you pin one model per card, every 7B deployment owns a $1.45 an hour card and uses maybe a third of it. That is fine for a customer who wants isolation and bad for everyone else on the price list.
So we group small models onto one card and keep big ones alone. The scheduler decides the grouping when a deployment is created and rebalances it when models are added or removed.
Every customer would like zero queue. What no one wants is to pay for the idle GPU that makes zero queue possible. A model that never queues sits unused most of the day and bills someone for it. We run a short queue instead and let the spending cap decide how much waiting is acceptable. A few seconds of queue costs nothing and keeps the hourly price where it is. If a customer needs no waiting at all, they get a dedicated instance, which is the raw GPU product, and they pay for the headroom directly.
Weight updates are the annoying case. Loading new weights into a running model has to be atomic from the scheduler's point of view, otherwise a request mid-load sees a half-written model. We solve it by loading to a staging area and swapping the pointer, and requests that arrive during the swap wait a moment. It has been quiet for a year now, which is exactly what we want from an update path. Dario wrote up the details in why every hosted model runs on SGLang.
We do not do automatic model quantization. We do not pack mixed precision on one card beyond what the checkpoint already is. We do not try to squeeze a 70B onto a 48GB card with tricks, because tricks make the latency numbers lie. The scheduler trades capacity for predictability, and that is the whole design.