Issue 20: Meta's Llama 3.1, Gemini Flash Update and Mistral's 123B Powerhouse - July 29, 2024
This week saw major model releases from Meta and Mistral, Google's Gemini expansion in Europe, and groundbreaking research in AI training methods.
News
Meta Unleashes Llama 3.1 with 405B Parameters
Meta has released Llama 3.1, a powerful open model available in 8B, 70B, and 405B parameter versions. The 405B model boasts multilingual support for 8 languages, 128k context window, and performance rivaling GPT-4 on various benchmarks.
Mistral AI Introduces 123B Parameter Powerhouse
Mistral Large, a 123B parameter model, has been unveiled by Mistral AI. It supports 80+ coding languages, offers tool use and function calling capabilities, and achieves impressive scores on key benchmarks.
Google's Gemini 1.5 Flash Lights Up Europe
Google has made Gemini 1.5 Flash available for free testing in Europe, featuring an expanded 32k context window and support for 40+ languages. This update brings Google's version of SearchGPT to a wider audience without waitlisting.
Research
DPO Study Reveals Optimal Training Strategies
A new study on Direct Preference Optimization (DPO) explores the impact of KL-divergence constraints and reference models on AI performance. The research provides valuable insights for optimizing AI training processes.
HyPO Demonstrates Power of Online Data in RLHF
Hybrid Preference Optimization (HyPO) showcases the importance of online data in Reinforcement Learning from Human Feedback (RLHF). This method outperforms pure offline approaches, leading to better model generalization.
Google DeepMind Unveils J-BOND RLHF Method
Google DeepMind has revealed J-BOND, the RLHF method behind Gemma 1.1. This innovative approach uses a Best-of-N Distillation algorithm to optimize AI training, offering potential improvements over existing methods.
General
Hugging Face Hub Serves Massive AI Workloads
The Hugging Face Hub is now serving over 6 petabytes and nearly 1 billion requests daily, showcasing the platform's robust infrastructure and the growing demand for AI resources.
Mark Zuckerberg Advocates for Open-Source AI
Meta's CEO has shared his thoughts on open-source AI as the path forward, emphasizing its benefits for customization, ecosystem development, and democratization of AI technology.
I hope you enjoyed this newsletter. 🤗 If you have any questions or are interested in collaborating, feel free to contact me on Twitter or LinkedIn.
See you next week 👋🏻👋🏻