Hugging Face researchers surveyed sixteen open-source reinforcement learning libraries to guide the development of a new asynchronous trainer for TRL. The study compares these libraries across seven axes, including orchestration primitives, rollout buffer design, weight synchronization protocols, and staleness management. The team identifies GPU underutilization in synchronous training loops as a key motivation, noting that disaggregating inference from training onto separate GPU pools enables concurrent generation and gradient computation.
No score is assigned. Sources and their independence are shown in the citation chain below.