Evolution of LLMs from 2019 to today | Lex Fridman Podcast

In this discussion, experts explore the evolution of Large Language Models (LLMs) from GPT-2 to current architectures, emphasizing that while the fundamental transformer architecture remains largely unchanged, significant advancements have occurred in training methodologies, computational efficiency, and model capabilities. The transformer architecture, originally derived from the 'Attention is All You Need' paper, continues to be the foundational framework for state-of-the-art language models. Key architectural innovations include mixture of experts layers, which allow models to selectively activate different neural network components based on input context, and slight modifications like group query attention and different normalization techniques. The most substantial improvements have been in training approaches, such as supervised fine-tuning, reinforcement learning with human feedback, and system-level optimizations like reduced precision training (FP8, FP4) that enable faster computational processing. While alternative model architectures like text diffusion models and Mamba are emerging, the auto-regressive transformer remains the predominant state-of-the-art approach. The rapid advancement in AI is less about radical architectural changes and more about incremental improvements in training methodologies, data quality, and computational efficiency.