Topic 8: What is LSTM and xLSTM?

Be the first to get our detailed explanations and valuable collection of relevant resources straight to your inbox –>

Think of trying to recall a series of events from a few days ago while still remembering what happened today. For RNNs, updating their weights to learn from data is similar. They use backpropagation through time (BPTT) to adjust weights based on prediction errors. As the error signal is propagated back through many time steps, it can become so small that the network's weights barely change. This vanishing gradient problem means the network struggles to learn long-term dependencies and forgets important information from earlier time steps. What is not a problem for a human is a big challenge to RNNs.

Two German researchers, Jürgen Schmidhuber and his doctoral student Sepp Hochreiter, were determined to find a solution. In 1997, they introduced an improved RNN architecture called Long Short-Term Memory (LSTM). LSTMs were designed with a memory cell that could maintain information over long periods. This memory cell was controlled by three gates:

• the input gate

• the forget gate

• the output gate

These gates regulate the flow of information, allowing the network to keep important information for longer and forget what is no longer needed.

The initial reception of their work was lukewarm. But a few researchers kept building on the LSTM foundation. Schmidhuber himself didn’t plan to give up. In 2000, Felix Gers, Jürgen Schmidhuber, and Fred Cummins introduced peephole connections, which allowed the gates to access the cell state directly. This improvement helped LSTMs learn precise timings of events, enhancing their performance.

Popularization and success

Bidirectional LSTM (BiLSTM) (2005): Alex Graves and Jurgen Schmidhuber introduced BiLSTMs in 2005, comprising two LSTM layers running in opposite directions (forward and backward). This architecture captures both past and future context, improving performance in tasks like speech recognition and machine translation.

The rise of deep learning in the 2010s brought another wave of innovation. Researchers began stacking multiple LSTM layers, creating deep LSTM networks that could learn hierarchical features. This advancement made LSTMs even more powerful, enabling them to excel in a wide range of applications, from machine translation to speech recognition.

In 2014, Ilya Sutskever, Oriol Vinyals, and Quoc V. Le popularized LSTMs for machine translation with their sequence-to-sequence (Seq2Seq) models. These models used LSTMs for both encoding and decoding sequences, leading to notable improvements in translation quality.

In 2015, attention mechanisms (Bahdanau attention) were introduced by Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. These mechanisms allowed LSTMs to focus on specific parts of the input sequence, further boosting their performance in tasks like translation and summarization.

A lot changed in 2017 after Vaswani et al. introduced Transformer models in the paper "Attention Is All You Need." This marked a shift towards attention-based mechanisms. But why?

LSTM limitations and their overshadowing by Transformers

LSTM networks have been the backbone of many early successes in natural language processing (NLP). But despite their ability to capture long-range dependencies and solve the vanishing gradient problem, LSTMs have significant limitations that led to their gradual overshadowing by transformers.

Firstly, LSTMs struggle with parallelization. Their sequential nature means that each step depends on the previous one, making it challenging to process data efficiently on modern hardware. This results in slower training times and higher computational costs, which are less than ideal in an era where speed and efficiency are paramount.

Moreover, LSTMs often face difficulties in handling very long sequences. Although they are designed to remember information over long time periods, their effectiveness diminishes as the sequence length increases. This limitation is particularly problematic for tasks requiring the model to understand context from extensive input data, such as long-form text or complex temporal patterns.

Transformers addressed this limitations. Their efficiency, scalability, and superior performance have firmly established them as the new standard in the field, they outpaced LSTMs at scale.

Introducing xLSTM: Addressing the shortcomings

“Never give up” is probably the motto of LSTM aficionados. Remember Schmidhuber’s doctoral student Sepp Hochreiter? He became the head of the Linz Institute of Technology (LIT) AI Lab, a founding director of the Institute of Advanced Research in Artificial Intelligence (IARAI), and was awarded the IEEE CIS Neural Networks Pioneer Prize in 2021 for his work on LSTM networks. He continued to work on them. And in May 2024, he, along with eight other researchers from Linz, Austria, introduced Extended Long Short-Term Memory: xLSTM.

The researchers asked themselves a question: “How far can we get in language modeling by scaling LSTMs to billions of parameters, leveraging the latest techniques from modern LLMs, but mitigating the known limitations of LSTMs?”

They also argued that Transformers are powerful but lack the linear scaling with sequence length that LSTMs have.

The Architecture of xLSTM

Building on the solid foundation of traditional LSTMs, the researchers introduced two key enhancements:

1. Exponential Gating: This update equips LSTMs with a more flexible approach to managing information flow. It's like fine-tuning the mechanisms that control data processing, providing a more nuanced handling of inputs and memory.

2. Novel Memory Structures: xLSTM enhances memory in two significant ways:

   • Scalar LSTM (sLSTM): This version improves how memory is mixed and updated, allowing for more accurate data retention and processing.

   • Matrix LSTM (mLSTM): By transforming memory cells into a matrix structure, mLSTM enhances the network's ability to handle operations in parallel, which significantly speeds up processing.

The matrix structure in mLSTM not only expands memory capacity but also enhances the efficiency of information retrieval and storage. This structural change allows for better handling of tasks with complex data structures or long-range dependencies.

These enhancements are integrated into resilient block architectures, which essentially means they are stacked in a way that builds upon each previous layer's knowledge, creating a deep and powerful network.

With these improvements, xLSTM enhances its ability to handle complex sequence modeling tasks, which boosts its performance and applicability in real-world scenarios.

Evaluations

The researchers conducted a series of structured task evaluations, where xLTSM performed quite well. The models were also tested on the SlimPajama and PALOMA datasets, focusing on real-world scenarios. SlimPajama provided a platform to compare models like xLSTM, RWKV, and Llama, focusing on their performance metrics, such as perplexity, across extensive training sessions. These tests highlighted the influence of different model architectures on performance at scale.

The PALOMA dataset, comprising diverse data sources, allowed for testing across various natural language processing tasks, ranging from understanding internet slang to complex reasoning. This testing demonstrated how models like xLSTM perform in handling linguistic diversity, often showing lower perplexity, which indicates a stronger ability to manage varied language inputs.

These experiments on SlimPajama and PALOMA underline the practical strengths of xLSTM, showing its adaptability and potential for real-world applications in AI.

For all experiments, the researchers used Python 1 3.11 with PyTorch 2.2.0 2 , and CUDA 12.1 3 on NVIDIA A100 GPUs.

Applications across domains

Answering the question posed at the beginning, the researchers state that xLSTMs perform at least as well as current technologies like Transformers or State Space Models. xLSTM models demonstrate promising results in language modeling, comparable to advanced techniques like Transformers and State Space Models. Their scalability suggests they could effectively compete with major language models and potentially impact fields such as reinforcement learning, time series prediction, and physical system modeling.

Conclusion: The future of sequence modeling with xLSTM

xLSTM represents a significant evolution in the LSTM architecture, addressing past limitations and setting a new standard for sequence modeling. It opens up a whole new range of possibilities for tackling complex problems involving data over time. As this technology continues to mature, its integration into various domains promises to drive forward the capabilities of AI in processing complex, sequential data.

By leveraging modern deep learning innovations, xLSTM achieves scalability to billions of parameters, making it competitive with contemporary models like Transformers. However, xLSTM is unlikely to replace Transformers, which are superior in parallel processing and attention-based tasks. Instead, xLSTM will likely complement them by excelling in memory efficiency and long-sequence handling. Think of xLSTM as a new tool in the AI toolbox, one that can enhance our ability to understand and predict patterns in our world. It's especially exciting that research and breakthroughs are also happening in other areas, such as RNNs.

Thank you for reading, please feel free to share this article with your friends 🤍

Bonus: Resources

Implementation:

• Official repository of the xLSTM: https://github.com/NX-AI/xlstm

• A repository that contains resources about xLSTM: https://github.com/AI-Guru/xlstm-resources

Research papers, mentioned in the article or related to the topic:

• Long Short-Term Memory (the original paper) by Sepp Hochreiter and Jürgen Schmidhuber (2005)

• Framewise Phoneme Classification with Bidirectional LSTM Networks and Framewise phoneme classification with bidirectional LSTM and other neural network architectures by Alex Graves and Jürgen Schmidhuber (2005)

• Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition by Alex Graves , Santiago Fernandez, Jürgen Schmidhuber (2005)

• Sequence to Sequence Learning with Neural Networks by Ilya Sutskever, Oriol Vinyals, Quoc V. Le (2014)

• Neural Machine Translation by Jointly Learning to Align and Translate by Dzmitry Bahdanau, KyungHyun Cho, Yoshua Bengio (2015)

• Attention is All You Need by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin (2017)

• xLSTM: Extended Long Short-Term Memory by Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter (2024)

Good reads:

• LSTMs Got an Upgrade? xLSTM is Here to Challenge the Status Quo

• Interview With Jürgen Schmidhuber

Twitter:

• Sepp Hochreiter

• Jürgen Schmidhuber

Thank you for reading! Share this article with three friends and get a 1-month subscription free! 🤍