Issue 14: Nemotron 4 vs. GPT-4, Apple's On-Device AI, and MoA Breakthrough - June 16, 2024
This week's highlights include Meta's new CRAG benchmark, NVIDIA's Nemotron 4, Microsoft's Samba model, and more exciting updates in AI and synthetic data generation.
News
Meta's CRAG Benchmark Unveiled
Discover Meta's new CRAG benchmark for factual question-answering, designed to test LLMs with 4,409 question-answer pairs. It features tasks like Retrieval Summarization, KG, and Web Retrieval Augmentation, offering a comprehensive way to assess and improve LLMs.
NVIDIA's Nemotron 4 vs. GPT-4
The Nemotron 4 by NVIDIA challenges the original GPT-4 with impressive metrics, especially for home use. It excels in HumanEval with a score of 73.2 compared to GPT-4's 67.0.
NVIDIA's Nemotron 4 340B Unveiled
NVIDIA's latest 340B dense LLM rivals GPT-4 for chat applications and synthetic data generation. Trained on 9 trillion tokens and supporting 50+ languages, it's commercially usable with a custom license.
Argilla Joins Hugging Face
Argilla's collaboration with Hugging Face brings tools like Distilable, simplifying synthetic data creation and enhancing the Enterprise Hub. Welcome to the team, Argilla! Discover more here.
Apple's On-Device AI Revolution
Apple's on-device LLM, running on Mac, iPhone, and iPad, outperforms many larger models using fine-tuned LoRA Adapters. This innovation promises faster, more efficient AI capabilities. Read more in their blog.
Research
Microsoft's Samba Model Surges Ahead
Meet Samba, a hybrid model combining Mamba and Transformers, outperforming Phi-3 mini on the same dataset. Samba achieves 3.64× faster decoding throughput than Llama-3. Check out the code.
Generate High-Quality Synthetic Datasets at Home
With Llama 3 70B, you can now create large-scale instruction datasets at home using self-synthesis, achieving results that rival GPT-4-generated data. This method generated 4 million pairs, filtered to 300k high-quality pairs.
Mixture-of-Agents (MoA) Takes the Lead
The MoA architecture uses multiple LLMs in layers to enhance generation quality, outperforming GPT-4 Omni on benchmarks like AlpacaEval 2.0. MoA-Lite is also cost-efficient, achieving superior performance with less resource use.
Introducing RLOO: A New Era in RLHF
RLOO simplifies RLHF by using a single action for model completion and REINFORCE, reducing memory needs and avoiding out-of-memory errors. It outperforms traditional methods like PPO.
ArmoRM: The Top Open Reward Model
ArmoRM from RLHFlow ranks #1 on Allen AI's Reward Bench, surpassing major competitors like Cohere and OpenAI. It's a versatile tool for model evaluation and synthetic data ranking.
General
Deploy Meta's Llama 3 70B on AWS
Learn to fine-tune and deploy Meta's Llama 3 70B with PyTorch FSDP, Q-Lora, and Flash Attention 2 on Amazon SageMaker, optimized for efficiency and cost-effectiveness.
Key Metrics for New Models
When evaluating new models, focus on MixEval, IFEval, and Arena-Hard for a comprehensive performance assessment. These benchmarks reflect real-world applications and user queries.
I hope you enjoyed this newsletter. 🤗 If you have any questions or are interested in collaborating, feel free to contact me on Twitter or LinkedIn.
See you next week 👋🏻👋🏻