There are many neural network models that prove to be really helpful, but they all take input as a fixed size of array. This fixed size is one of the problem that we encounter when we start working on sequential data. The sequential data can vary in size.
What is Sequential Data
If there is a particular order in which related things follow each other, we call it as a sequence.
For example – i am a programmer OR am i a programmer.
Do both the sentences means the same. NO, that’s the importance of sequence. In sequence the position matters. The correct position gives it a correct meaning.
Just imagine, we are watching a movie. We can easily predict the next frame of the movie if we have already seen the movie. But if we haven’t seen the movie, can we still predict the next frame??? Ofcourse Not!!!
Ordinary neural networks can’t remember the positional relationship in sequences. They consider “ i am a programmer” and “am i a programmer” as the same.
Are we like them?. No!!!
Recurrent Neural Network(RNN)
Recurrent Neural Networks were created in the 1980’s but have just been recently gaining popularity from advances to the networks designs and increased computational power from graphic processing units. They’re especially useful with sequential data because each neuron or unit can use its internal memory to maintain information about the previous input. This is great because in cases of language, “I had washed my house” is much more different than “I had my house washed”. This allows the network to gain a deeper understanding of the statement.
The Feedforward Neural Network takes decisions based on only the current input, the RNN takes decisions based on current and previous inputs.
The Green Box represents a Neural Network. The arrows indicate memory or simply feedback to the next input.
The first figure shows the RNN. The Second figure shows the same RNN unrolled in time. Consider a sequence [i am a good programmer]. We can say that the sequence is arranged in time. At t=0, X0=“i” is given as the input . At t=1, X1=“am” is given as the input. The state from the first time step is remembered and given as input during the second time step along with the current input at that time step.
In a Feed Forward Neural Network, the Network is forward propagated only once per sample. But in RNN, the network is forward propagated equal to the number of time steps per sample.
Different Problems solved with RNN
Given a sequence of words we want to predict the probability of each word given the previous words.
Machine Translation is similar to language modeling in that our input is a sequence of words in our source language (e.g. German). We want to output a sequence of words in our target language (e.g. English).
Given an input sequence of acoustic signals from a sound wave, we can predict a sequence of phonetic segments together with their probabilities.
Generating Image Descriptions
Together with convolutional Neural Networks, RNNs have been used as part of a model to generate descriptions for unlabeled images.
Chatbots can give reply to your queries. When a sequence of words is given as the input, sequence of words will be generated at the output.
Recurrent Neural Networks have been becoming very popular as of recently and for a very good reason. They’re one of the most effective models out for natural language processing. New applications of these models are coming out all the time and its exciting to see what researchers come up with.
To play around with some RNN check out these awesome libraries
Tensorflow — Googles Machine Learning frameworks RNN example: https://www.tensorflow.org/versions/r0.10/tutorials/recurrent/index.html
Keras — a high level machine learning package that runs on top of Tensorflow or Theano: https://keras.io
Torch — Facebook machine learning framework in LUA: http://torch.ch