Reinforcement Learning from Human Feedback

Reinforcement Learning from Human Feedback (RLHF) is a technique that fine-tunes language models using human-generated preference data to align model outputs with desired behaviors. vLLM can be used to generate the completions for RLHF.

The following open-source RL libraries use vLLM for fast rollouts (sorted alphabetically and non-exhaustive):

• Cosmos-RL
• ms-swift
• NeMo-RL
• Open Instruct
• OpenRLHF
• PipelineRL
• Prime-RL
• SkyRL
• TRL
• Unsloth
• verl

For weight synchronization between training and inference, see the Weight Transfer documentation, which covers the pluggable backend system with NCCL (multi-GPU) and IPC (same-GPU) engines. To verify that an update landed on every rank, see the Weight Checker.

For pipelining generation and training to improve GPU utilization and throughput, see the Async Reinforcement Learning guide, which covers the pause/resume API for safely updating weights mid-flight.

See the following notebooks showing how to use vLLM for GRPO:

• Efficient Online Training with GRPO and vLLM in TRL
• Qwen-3 4B GRPO using Unsloth + vLLM