Topic 10: Inside DeepSeek Models

Introduction

The DeepSeek family of models presents a fascinating case study, particularly in open-source development. While much attention in the AI community has been focused on models like LLaMA and Mistral, DeepSeek has emerged as a significant player that deserves closer examination.

Coming from China, DeepSeek's technical innovations are turning heads in Silicon Valley. Their revolutionary approaches to attention mechanisms and the Mixture-of-Experts (MoE) technique have led to impressive efficiency gains. This is exemplified in their DeepSeek-V2 and DeepSeek-Coder-V2 models, with the latter widely regarded as one of the strongest open-source code models available. Another surprising thing is that DeepSeek small models often outperform various bigger models. These innovations highlight China's growing role in AI, challenging the notion that it only imitates rather than innovates, and signaling its ascent to global AI leadership. DeepSeek is also quite affordable. Let’s explore the specific models in the DeepSeek family and how they manage to do all the above.

In today’s episode, we will cover:

• In the race to beat the benchmarks

• New stage: DeepSeek innovates to beat challenges, not benchmarks

• Strategies behind DeepSeekMoE that make the difference

• DeepSeek-V2: How does it work?

• Advantages and limitations of DeepSeek-V2

• DeepSeek-Coder: What makes it highly efficient?

• Implementation of DeepSeek-Coder-V2

• Pricing

• Conclusion

• Bonus: Resources

In the race to beat the benchmarks

DeepSeek models quickly gained popularity upon release. Initially, DeepSeek created their first model with architecture similar to other open models like LLaMA, aiming to outperform benchmarks. This approach set the stage for a series of rapid model releases.

On November 2, 2023, DeepSeek began rapidly unveiling its models, starting with DeepSeek Coder. From the outset, it was free for commercial use and fully open-source.

Later, on November 29, 2023, DeepSeek launched DeepSeek LLM, described as the “next frontier of open-source LLMs,” scaled up to 67B parameters.

As we've already noted, DeepSeek LLM was developed to compete with other LLMs available at the time. DeepSeek LLM 67B Chat had already demonstrated significant performance, approaching that of GPT-4. But, like many models, it faced challenges in computational efficiency and scalability. This led the DeepSeek AI team to innovate further and develop their own approaches to solve these existing problems.

New stage: DeepSeek innovates to beat challenges, not benchmarks

In only two months, DeepSeek came up with something new and interesting.