Hugging Face researchers Rémi Ouazan Reboul, Pedro Cuenca, and Aritra Roy Gosthipaty demonstrated that synchronous batching wastes nearly 24% of GPU runtime during LLM inference because the CPU and GPU take turns idling. Their solution, asynchronous batching, separates CPU batch preparation from GPU computation using CUDA streams, allowing both to run in parallel. This approach eliminates idle gaps without requiring new kernels or model changes, achieving a potential 24% speedup.
No score is assigned. Sources and their independence are shown in the citation chain below.