Topic 21: What is Natural Language Reinforcement Learning (NLRL)?

Recently we have spoken about Natural Language Reinforcement Learning (NLRL) on our social media, and it got a lot of feedback on Twitter, so we decided to expand the discussion of this interesting approach. We’ll talk about it in more detail, addressing the questions that our readers had.

So what is NLRL about and why should you know about it? NLRL is about adapting Reinforcement Learning (RL) concepts to work in a space where the key element is natural language. In NLRL, the core parts of RL like goals, strategies, and evaluation methods are redefined using natural language. Combined with LLMs, NLRL becomes practical and can be implemented either through simple prompts or by tweaking the model’s parameters. Let’s dive into what’s so special and revolutionary in NLRL and why it could be better than traditional RL.

In today’s episode, we will cover:

• Why isn't reinforcement learning always enough?

• Here comes NLRL

• How does NLRL work?

  • Redefining key concepts of RL with natural language

  • Methods to evaluate language policy

• LLMs as a good fit for NLRL

• How good is NLRL in practice?

• Advantages of NLRL

• The main question - limitations

• Conclusion

• Bonus: Resources to dive deeper

Why isn't Reinforcement Learning always enough?

Reinforcement Learning is a way of teaching machines to make decisions by framing problems as mathematical tasks, using a system called the Markov Decision Process (MDP). This method has led to breakthroughs in areas like gaming and robotics, but it has some issues. Traditional RL often struggles because it:

• lacks prior knowledge: it doesn’t start with helpful information about the task and requires a lot of trial and error to learn how things work;

• is hard to interpret: even advanced RL models like AlphaZero make decisions that are difficult to explain;

• has unstable training: RL relies on simple numeric rewards for feedback, which can be limiting, especially in real-world tasks where richer feedback, like text or visuals, is available.

In easy words, RL’s “rigid math” lacks the flexibility and interpretability of natural language. RL can help models reason in language (like Chain-of-Though), but a key puzzle is figuring out how to let the model judge its own progress in natural language. So the main questions are:

• How do you measure if the model is on the right path in its reasoning, using only words?

• How can this evaluation be done unsupervised, meaning without human-provided labels or examples?

Here comes NLRL