Overview

• RNNs solve the issue of dealing with sequential data, for traditional neural networks.
• RNNs are extremely versatile and are used in a wide range of applications that deal with sequential data:
• A common application of RNNs is Language Modelling.
• Simple RNNs face the following issues:
• These issues can be solved by the Long Short-Term Memory model

Mathematical Background

• RNNs utilise a recursive formula.
• The LSTM has internal gates, each with their own activation function.

Process of Operation

• So to summarise what happens in a simple RNN:
• To help understand how an LSTM RNN Network works, lets take a look at a Language Modelling RNN:

Training

• The simple RNN is trained using the Back Propagation Through Time (BPTT) algorithm:
• LSTMs are also trained using BPTT.

Citations

(n.d.). How LSTM networks solve the problem of vanishing gradients | by Nir Arbel | Data Driven Investor | Medium. Retrieved from https://medium.com/datadriveninvestor/how-do-lstm-networks-solve-the-problem-of-vanishing-gradients-a6784971a577

(n.d.). [ Back to Basics ] Deriving Back Propagation on simple RNN/LSTM (feat. Aidan Gomez) | by Jae Duk Seo | Towards Data Science. Retrieved from https://towardsdatascience.com/back-to-basics-deriving-back-propagation-on-simple-rnn-lstm-feat-aidan-gomez-c7f286ba973d

(n.d.). Loss Functions — ML Glossary documentation. Retrieved from https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html

(n.d.). CS 230 - Recurrent Neural Networks Cheatsheet. Retrieved from https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-recurrent-neural-networks