AI in the Real World with Foundation Capital

Building the next generation of AI models: Rohan Taori, Researcher at Anthropic & Alpaca co-creator

Episode Notes

Welcome to AI in the Real World! In this episode, Foundation Capital Partner Jaya Gupta sits down with Rohan Taori, a researcher on Anthropic's multimodal pre-training team.

Within the AI community, Rohan is best known for co-creating Alpaca, a project that demonstrated how fine-tuning Meta's LLaMA model could achieve ChatGPT-level performance for under $600.

Rohan shares his journey from early work in computer vision at UC Berkeley to his Ph.D. at Stanford, where he explored methods for making AI more accessible.

He explains the technical breakthroughs behind Alpaca, including self-instruct, a method that uses a stronger language model (OpenAI's text-davinci-003) to generate synthetic data that is then used to fine-tune a weaker model (Llama). This approach, which underpins Alpaca and its follow-up projects AlpacaFarm and AlpacaEval, illustrates how small-scale post-training can significantly enhance model performance.

The conversation also covers:

• The promise and challenges of synthetic data for training AI models
• What it will take to build foundation models that are 100x better
• The future of multimodal AI and why it matters
• Why better evals are critical to the next wave of AI advances

Chapters:

• 00:00 Cold open
• 1:40 Rohan's journey into AI research
• 04:50 Transitioning from vision research to LLMs
• 06:18 The story behind Alpaca
• 08:55 How Alpaca works
• 10:45 The AI community's reception of Alpaca
• 12:26 The evolution of Alpaca related projects
• 14:22 The role of synthetic data
• 19:38 Challenges in multimodal AI
• 24:31 Future of foundation models
• 30:00 Importance of data in AI
• 34:48 Staying up to date with the latest AI research
• 36:12 Advice for founders