The "secret sauce" of recent AI breakthroughs: Post-training with RLVR (and RLHF) | Lex Fridman

In this podcast episode, Lex Fridman explores the recent breakthroughs in AI post-training, focusing on Reinforcement Learning with Verifiable Rewards (RLVR). The discussion reveals how this approach allows language models to improve their performance by iteratively solving problems with clear, verifiable outcomes, particularly in domains like mathematics and coding. The key innovation is the model's ability to generate step-by-step explanations and self-correct during problem-solving, which helps improve accuracy and build trust. The speakers highlight that RLVR differs from previous approaches like Reinforcement Learning from Human Feedback (RLHF) by using objective, measurable rewards instead of subjective human preferences. Importantly, the method doesn't necessarily teach new knowledge but helps 'unlock' existing capabilities within pre-trained models. The researchers note that RLVR shows promising scaling properties, with potential for logarithmic compute increases leading to linear performance improvements, unlike RLHF. The conversation also delves into the computational challenges of post-training, noting that RLVR runs can be as time-consuming as pre-training, but with different hardware requirements. Looking forward, researchers are exploring more sophisticated approaches like process reward models and value functions to further refine this technique.