Topic 20: What is Flow Matching?

Today, we’re exploring Flow Matching (FM), a concept that might sound complex but is more approachable than it seems. If it feels overwhelming at first, don’t worry – by the end of this episode, you’ll have a clear understanding of its key ideas and practical applications.

Why is Flow Matching worth discussing now? It’s gaining attention for its role in top generative models like Flux (text-to-image), F5-TTS and E2-TTS (text-to-speech), and Meta’s MovieGen (text-to-video). These models consistently achieve state-of-the-art results, and some experts argue that FM might even surpass diffusion models. But why is that the case?

FM enhances Continuous Normalizing Flows (CNFs), a framework for generating realistic samples of complex data – whether images, audio, or text – starting from structured noise. While powerful, CNFs face challenges such as long training times and intricate techniques for speeding up sampling. Flow Matching tackles these issues by optimizing the path from noise to structured outputs, streamlining CNFs and reducing the inefficiencies caused by differential equation computations. Put simply, FM focuses on learning how to match flows of probability distributions over time.

Still sounds tricky? Let’s break it all down, examine the details, and provide real-world examples of its implementation so you can see its potential in action. Let’s get started!

In today’s episode, we will cover:

• Continuous Normalizing Flows (CNFs) and their limitations

• Here comes Flow Matching

• How does Flow Matching work?

• How does Conditional Flow Matching (CFM) help?

• What about diffusion models?

• Advantages of Flow Matching

• Not without limitations

• Conclusion and implementation

• Bonus: Resources to dive deeper

Continuous Normalizing Flows (CNFs) and their limitations

Let’s start from the very beginning and make clear what are the Continuous Normalizing Flows (CNFs).

CNFs are a type of flexible framework or mathematical tool used in generative modeling to transform simple data (like random noise) into complex distributions (like realistic images or sounds). Unlike diffusion models, which slowly add and remove noise and have limited ways of data processing, CNFs can handle a broader range of data transformations. They do this by gradually and smoothly reshaping the data using a process guided by a vector field. Let’s break it down in simple terms.

Key concepts of CNFs:

• Data space: This is where the data “lives”. For example, an image with pixels could exist in a high-dimensional space with a fixed number of pixels.

• Probability density path: Probability density path describes how the data's probability distribution evolves over time. Imagine data as points in a space. To move from a simple noise to realistic data, we use a probability path, which is a gradual transformation.

• Vector field: Think of it as a map that guides how data points should move at every moment to realize the transformation.

• Flow: And finally, flow reshapes the data step by step over time, guided by the vector field. It defines a continuous transformation. But to be more accurate in terms, in CNFs, you need to solve an Ordinary Differential Equation (ODE), that tells how data moves based on the vector field. This turns the transformation into a flow of probability over time.

Instead of manually creating the vector field, researchers use a neural network to learn it. This neural network is like GPS – it takes data points as input and predicts where they should move to match the desired distribution. The network reshapes the data into a realistic and complex data distribution, such as a detailed image or piece of music, using a rule called the push-forward equation, which ensures that the transformation follows the rules of probability (the total probability remains 1 at all times).

This smooth, continuous transformation from noise to data is CNFs’ main advantage, which makes them a powerful tool for generative modeling. However, CNFs have a serious limitation – solving Ordinary Differential Equations during training is slow, difficult and computationally expensive.

Here comes Flow Matching