Issue 20: Meta's Llama 3.1, Gemini Flash Update and Mistral's 123B Powerhouse - July 29, 2024

Disclaimer: This content is generated by AI using my social media posts. Make sure to follow.

This week saw major model releases from Meta and Mistral, Google's Gemini expansion in Europe, and groundbreaking research in AI training methods.

News

Meta Unleashes Llama 3.1 with 405B Parameters

Meta has released Llama 3.1, a powerful open model available in 8B, 70B, and 405B parameter versions. The 405B model boasts multilingual support for 8 languages, 128k context window, and performance rivaling GPT-4 on various benchmarks.

Mistral AI Introduces 123B Parameter Powerhouse

Mistral Large, a 123B parameter model, has been unveiled by Mistral AI. It supports 80+ coding languages, offers tool use and function calling capabilities, and achieves impressive scores on key benchmarks.

Google's Gemini 1.5 Flash Lights Up Europe

Google has made Gemini 1.5 Flash available for free testing in Europe, featuring an expanded 32k context window and support for 40+ languages. This update brings Google's version of SearchGPT to a wider audience without waitlisting.

Research

DPO Study Reveals Optimal Training Strategies

A new study on Direct Preference Optimization (DPO) explores the impact of KL-divergence constraints and reference models on AI performance. The research provides valuable insights for optimizing AI training processes.

HyPO Demonstrates Power of Online Data in RLHF

Hybrid Preference Optimization (HyPO) showcases the importance of online data in Reinforcement Learning from Human Feedback (RLHF). This method outperforms pure offline approaches, leading to better model generalization.

Google DeepMind Unveils J-BOND RLHF Method

Google DeepMind has revealed J-BOND, the RLHF method behind Gemma 1.1. This innovative approach uses a Best-of-N Distillation algorithm to optimize AI training, offering potential improvements over existing methods.

General

Hugging Face Hub Serves Massive AI Workloads

The Hugging Face Hub is now serving over 6 petabytes and nearly 1 billion requests daily, showcasing the platform's robust infrastructure and the growing demand for AI resources.

Mark Zuckerberg Advocates for Open-Source AI

Meta's CEO has shared his thoughts on open-source AI as the path forward, emphasizing its benefits for customization, ecosystem development, and democratization of AI technology.


I hope you enjoyed this newsletter. 🤗 If you have any questions or are interested in collaborating, feel free to contact me on Twitter or LinkedIn.

See you next week 👋🏻👋🏻