Continuous batching
The serving technique where finished sequences exit a GPU batch and new requests join every decode step, instead of waiting for the whole batch to finish. Standard in vLLM-class servers; 2–4x throughput vs static batching.
The serving technique where finished sequences exit a GPU batch and new requests join every decode step, instead of waiting for the whole batch to finish. Standard in vLLM-class servers; 2–4x throughput vs static batching.