Recurrent Neural Network is on the most effective neural network model. It considers the position of the sequential data. In this tutorial we are going to explore the Recurrent Neural Network (RNN) with a practical example.

We are going to build a **Binary to Decimal** converter using RNN. We are going to build a RNN from scratch using python, no machine learning libraries are going to be used.

## Library

We need only one library ,i.e., **numpy**

```
import numpy as np
```

## Dataset

In a neural network, the dataset is one the primary thing we need. It the dataset only which is going to modify the weights, thats helps a neural network in correct prediction.

First of all we are going to built a function that will convert the a list (array) of binary numbers into decimal. For example –

```
[1,0,0,1] => 9
```

```
def bin2int(bin_list):
#bin_list <list> [0, 0, 0, 1]
int_val = ""
for k in bin_list:
int_val += str(int(k))
return int(int_val, 2)
```

This function takes the binary number as an input then it is converted into a string. After that we use python built-in functionality to convert that binary string into decimal value.

After that we have another function called **dataset, **that will help in building a dataset – it returns the X and Y for the dataset.

```
def dataset(num):
# num - no of samples
bin_len = 8
X = np.zeros((num, bin_len))
Y = np.zeros((num))
for i in range(num):
X[i] = np.around(np.random.rand(bin_len)).astype(int)
Y[i] = bin2int(X[i])
return X, Y
```

The **num** is the number of training data to be generated. The **bin_len** is the length of the binary number list to be generated. **X** contains the binary number and **Y** contains the decimal number of those binary number. If we run this function it will generate data like this.

```
no_of_smaples = 20
trainX, trainY = dataset(no_of_smaples)
testX, testY = dataset(5)
print trainX
print trainY
```

#### X

```
[
[ 0. 1. 0. 0. 0. 0. 0. 0.]
[ 0. 1. 0. 1. 0. 1. 0. 1.]
[ 0. 1. 0. 0. 1. 1. 0. 1.]
[ 1. 1. 0. 0. 0. 1. 0. 0.]
[ 1. 1. 1. 0. 1. 1. 0. 1.]
[ 1. 1. 0. 1. 0. 0. 1. 0.]
[ 0. 0. 0. 1. 0. 0. 0. 0.]
[ 1. 0. 1. 1. 0. 0. 1. 0.]
[ 0. 1. 1. 1. 1. 1. 1. 0.]
[ 0. 1. 0. 0. 1. 0. 1. 1.]
[ 1. 1. 1. 0. 0. 1. 0. 1.]
[ 1. 1. 0. 1. 0. 1. 0. 0.]
[ 1. 1. 0. 1. 0. 1. 1. 0.]
[ 1. 1. 0. 1. 1. 1. 0. 0.]
[ 1. 1. 0. 1. 1. 1. 1. 0.]
[ 0. 0. 0. 0. 1. 1. 1. 1.]
[ 0. 0. 1. 1. 1. 0. 1. 0.]
[ 1. 0. 1. 1. 1. 0. 0. 0.]
[ 0. 0. 0. 1. 1. 0. 0. 1.]
[ 1. 0. 0. 1. 0. 1. 1. 1.]
]
```

#### Y

```
[ 64. 85. 77. 196. 237. 210. 16. 178. 126. 75. 229. 212.
214. 220. 222. 15. 58. 184. 25. 151.]
```

## RNN from scratch

Now we have our dataset ready, now we need to build our RNN.

We will create a class named **RNN, **and giving it some default parameter values in its constructor.

```
class RNN:
def __init__(self):
self.W = [1, 1]
self.W_delta = [0.001, 0.001]
self.W_sign = [0, 0]
self.eta_p = 1.2
self.eta_n = 0.5
```

Here – **self.W** is the set of weights, one for the input, and another for recurrent state (previous state output). **self.W_delta** is the learning rate and rest are the values that are going to be used in the Resilient Back Propagation.

Calculating the next state.

```
def state(self, xk, sk):
return xk * self.W[0] + sk * self.W[1]
```

Calculating the forward states.

```
def forward_states(self, X):
S = np.zeros((X.shape[0], X.shape[1]+1))
for k in range(0, X.shape[1]):
next_state = self.state(X[:,k], S[:,k])
S[:,k+1] = next_state
return S
```

Compute the gradient of the MSE cost function with respect to the predicted (**guess**) output.

```
def output_gradient(self, guess, real):
return 2 * (guess - real) / no_of_smaples
```

Now we will backpropagate the gradient computed at the output through the network, then we will computate the gradient for Weights at each layer by addition.

```
def backward_gradient(self, X, S, grad_out):
grad_over_time = np.zeros(( X.shape[0], X.shape[1]+1 ))
grad_over_time[:,-1] = grad_out
wx_grad = 0
wr_grad = 0
for k in range(X.shape[1], 0, -1):
wx_grad += np.sum( grad_over_time[:, k] * X[:, k-1] )
wr_grad += np.sum( grad_over_time[:, k] * S[:, k-1] )
grad_over_time[:, k-1] = grad_over_time[:, k] * self.W[1]
return (wx_grad, wr_grad), grad_over_time
```

Now we will use Resilient Backpropagation to update the weights.

```
def update_rprop(self, X, Y, W_prev_sign, W_delta):
S = self.forward_states(X)
grad_out = self.output_gradient(S[:, -1], Y)
W_grads, _ = self.backward_gradient(X, S, grad_out)
self.W_sign = np.sign(W_grads)
for i, _ in enumerate(self.W):
if self.W_sign[i] == W_prev_sign[i]:
W_delta[i] *= self.eta_p
else:
W_delta[i] *= self.eta_n
self.W_delta = W_delta
```

Now we will use all the above functions for the training of the RNN.

```
def train(self, X, Y, training_epochs):
for epochs in range(training_epochs):
self.update_rprop(X, Y, self.W_sign, self.W_delta)
for i, _ in enumerate(self.W):
self.W[i] -= self.W_sign[i] * self.W_delta[i]
```

The RNN is completed. Now we will train and test it.

```
rnn = RNN()
rnn.train(trainX, trainY, 20000)
print "Weight: \t", rnn.W
print "Real: \t\t", testY
y = rnn.forward_states(testX)[:, -1]
print "Predicted: \t",y
```

Resources:

http://peterroelants.github.io/posts/rnn_implementation_part01/

Great work… Thanks for this post.