TLDR
Some new training methods help big language models learn faster and more reliably than older methods, especially when working with huge amounts of data.
Summary
This content was automatically synthesized by Credo's AI models directly from the original source text.
AI summaries can make mistakes — double-check important details against the original source.
1 Study Aim
The paper aims to improve and test advanced training methods, called higher-order optimizers, for large language models (LLMs). The authors focus on making these optimizers, specifically SOAP (Second-Order Adaptive Preconditioner) and Muon, more stable and practical for use with very large datasets and models. They also compare these methods to the widely used AdamW optimizer to see which works best at large scales. The study wants to find out if new training methods can make big language models learn faster and more reliably than older ones.
2 Study Design
The researchers first identify problems with the SOAP optimizer when used with very large batch sizes (large groups of data processed at once). They then make changes to the algorithm, such as adding per-step QR orthogonalization (a mathematical process to keep calculations stable) and better preconditioning strategies (ways to adjust the optimizer for smoother learning). The team runs experiments on models with billions of parameters, training them on trillions of words. They compare SOAP, Muon, and AdamW using a method called update-RMS matching to ensure a fair comparison. They also develop a new distributed optimizer that works with Megatron-LM (a system for training large models), making it efficient for large-scale use. The researchers test new and old training methods on very large language models to see which ones work best and stay stable.
3 Findings
The study reveals that both SOAP and Muon optimizers consistently outperform AdamW when training very large language models, especially at batch sizes up to 100 million tokens. The authors show that their improvements to SOAP remove instability issues, such as sudden spikes in loss, and allow for stable training even with massive datasets. AdamW, in contrast, becomes less stable and effective at these scales. The new layer-wise distributed optimizer enables efficient and accurate training without sacrificing the benefits of the advanced optimizers. The authors also provide a public codebase to help others use and build on these methods. They recommend using SOAP or Muon for large-scale language model training to achieve better results and stability. The new training methods help big language models learn better and more reliably than older methods, especially when using huge amounts of data.