A shared GPU pool has competing customers: interactive requests, batch pipelines, development tests, and failover capacity. Maximizing utilization without distinguishing those workloads can make an expensive platform feel unreliable.

In Realtime and Batch Processing of GPU Workloads, Joseph Stein describes SS&C’s private-cloud AI platform, including priority queues across tenant and runtime environments, Valkey/Lua coordination, and backpressure based on inference-engine conditions. The interesting mechanism is coordinating admission with shared capacity, rather than treating each environment as an isolated hardware reservation.

Admit by expected work

Consider a hypothetical interactive request with a short prompt and a batch request with thousands of input tokens. Counting both as one request hides the difference in execution cost and memory pressure. Admission can consider estimated tokens, model identity, concurrency, and deadlines, then compare estimates with actual use.

Estimates will be imperfect. Output length is not known in advance, and requests can consume resources differently across models. Retain headroom and enforce maximum input and output sizes rather than assuming a predicted cost is exact.

A proposed scheduling loop inspired by the talk. Engine feedback informs admission before work reaches scarce GPU capacity.

A proposed scheduling loop inspired by the talk. Engine feedback informs admission before work reaches scarce GPU capacity.

Priority alone can starve batch work

Strict priority protects interactive traffic while it is busy, but may leave lower-priority queues waiting indefinitely. A proposed policy can combine minimum shares, weighted service, age-based promotion, and deadline checks. Each choice changes which guarantee a tenant receives.

Workload Useful scheduling objective
Interactive Bounded wait and acceptable first-token latency
Batch Completion by a declared deadline
Development Opportunistic capacity with explicit limits
Disaster recovery Reserved headroom or a tested eviction plan

Moving a queued request is easier than interrupting one already generating. A platform should distinguish admission priority from execution preemption and verify what its inference engine actually supports.

Make capacity loss a first-class test

A scheduler that works with a full fleet may fail during a zone outage. Queueing can grow while surviving GPUs look efficiently utilized. Check oldest queue age, deadline misses, rejected requests, and progress by workload class alongside utilization.

For an initial exercise, I would remove part of the capacity during a batch upload and confirm that interactive requests retain their intended service level. Then restore capacity and check that batch work catches up without a retry surge.

The platform contract should describe what happens when capacity is scarce. Average GPU utilization alone cannot express that contract.

Related reading: LLM inference architecture.

Archive note: added November 18, 2026, and filed under 2026-08 as part of the systems reading series.