IDIOT DEVELOPER

Binary to Decimal using Recurrent Neural Network (RNN)
Binary to Decimal using Recurrent Neural Network
DOWNLOAD

Recurrent Neural Network is on the most effective neural network model. It considers the position of the sequential data. In this tutorial we are going to explore the Recurrent Neural Network (RNN) with a practical example.

 

We are going to build a Binary to Decimal converter using RNN. We are going to build a RNN from scratch using python, no machine learning libraries are going to be used.

 

Library

We need only one library ,i.e., numpy

 


import numpy as np

 

Dataset

In a neural network, the dataset is one the primary thing we need. It the dataset only which is going to modify the weights, thats helps a neural network in correct prediction.

 

First of all we are going to built a function that will convert the a list (array) of binary numbers into decimal. For example –

 


[1,0,0,1] => 9

 


def bin2int(bin_list):
  #bin_list <list> [0, 0, 0, 1]
  int_val = ""
  for k in bin_list:
    int_val += str(int(k))
  return int(int_val, 2)

 

This function takes the binary number as an input then it is converted into a string. After that we use python built-in  functionality to convert that binary string into decimal value.

 

After that we have another function called dataset, that will help in building a dataset – it returns the X and Y for the dataset.

 


def dataset(num):
  # num - no of samples
  bin_len = 8
  X = np.zeros((num, bin_len))
  Y = np.zeros((num))

  for i in range(num):
    X[i] = np.around(np.random.rand(bin_len)).astype(int)
    Y[i] = bin2int(X[i])
  return X, Y

 

The num is the number of training data to be generated. The bin_len is the length of the binary number list to be generated. X contains the binary number and Y contains the decimal number of those binary number. If we run this function it will generate data like this.

 


no_of_smaples = 20

trainX, trainY = dataset(no_of_smaples)
testX, testY = dataset(5)

print trainX
print trainY

 

X


[
[ 0. 1. 0. 0. 0. 0. 0. 0.]
[ 0. 1. 0. 1. 0. 1. 0. 1.]
[ 0. 1. 0. 0. 1. 1. 0. 1.]
[ 1. 1. 0. 0. 0. 1. 0. 0.]
[ 1. 1. 1. 0. 1. 1. 0. 1.]
[ 1. 1. 0. 1. 0. 0. 1. 0.]
[ 0. 0. 0. 1. 0. 0. 0. 0.]
[ 1. 0. 1. 1. 0. 0. 1. 0.]
[ 0. 1. 1. 1. 1. 1. 1. 0.]
[ 0. 1. 0. 0. 1. 0. 1. 1.]
[ 1. 1. 1. 0. 0. 1. 0. 1.]
[ 1. 1. 0. 1. 0. 1. 0. 0.]
[ 1. 1. 0. 1. 0. 1. 1. 0.]
[ 1. 1. 0. 1. 1. 1. 0. 0.]
[ 1. 1. 0. 1. 1. 1. 1. 0.]
[ 0. 0. 0. 0. 1. 1. 1. 1.]
[ 0. 0. 1. 1. 1. 0. 1. 0.]
[ 1. 0. 1. 1. 1. 0. 0. 0.]
[ 0. 0. 0. 1. 1. 0. 0. 1.]
[ 1. 0. 0. 1. 0. 1. 1. 1.]
]

 

Y


[ 64. 85. 77. 196. 237. 210. 16. 178. 126. 75. 229. 212.
214. 220. 222. 15. 58. 184. 25. 151.]

 

RNN from scratch

Now we have our dataset ready, now we need to build our RNN.

 

We will create a class named RNN, and giving it some default parameter values in its constructor.

 


class RNN:
  def __init__(self):
    self.W = [1, 1]
    self.W_delta = [0.001, 0.001]
    self.W_sign = [0, 0]

    self.eta_p = 1.2
    self.eta_n = 0.5

 

Here – self.W is the set of weights, one for the input, and another for recurrent state (previous state output). self.W_delta is the learning rate and rest are the values that are going to be used in the Resilient Back Propagation.

 

Calculating the next state.

 


def state(self, xk, sk):
  return xk * self.W[0] + sk * self.W[1]

 

Calculating the forward states.

 


def forward_states(self, X):
  S = np.zeros((X.shape[0], X.shape[1]+1))
  for k in range(0, X.shape[1]):
    next_state = self.state(X[:,k], S[:,k])
    S[:,k+1] = next_state
  return S

 

Compute the gradient of the MSE cost function with respect to the predicted (guess) output.

 


def output_gradient(self, guess, real):
  return 2 * (guess - real) / no_of_smaples

 

Now we will backpropagate the gradient computed at the output through the network, then we will computate the gradient for Weights at each layer by addition.

 


def backward_gradient(self, X, S, grad_out):
  grad_over_time = np.zeros(( X.shape[0], X.shape[1]+1 ))
  grad_over_time[:,-1] = grad_out

  wx_grad = 0
  wr_grad = 0
  for k in range(X.shape[1], 0, -1):
    wx_grad += np.sum( grad_over_time[:, k] * X[:, k-1] )
    wr_grad += np.sum( grad_over_time[:, k] * S[:, k-1] )

    grad_over_time[:, k-1] = grad_over_time[:, k] * self.W[1]
  return (wx_grad, wr_grad), grad_over_time

 

Now we will use Resilient Backpropagation to update the weights.

 


def update_rprop(self, X, Y, W_prev_sign, W_delta):
  S = self.forward_states(X)
  grad_out = self.output_gradient(S[:, -1], Y)
  W_grads, _ = self.backward_gradient(X, S, grad_out)
  self.W_sign = np.sign(W_grads)

  for i, _ in enumerate(self.W):
    if self.W_sign[i] == W_prev_sign[i]:
      W_delta[i] *= self.eta_p
    else:
      W_delta[i] *= self.eta_n
    self.W_delta = W_delta

 

Now we will use all the above functions for the training of the RNN.

 


def train(self, X, Y, training_epochs):
  for epochs in range(training_epochs):
    self.update_rprop(X, Y, self.W_sign, self.W_delta)

    for i, _ in enumerate(self.W):
      self.W[i] -= self.W_sign[i] * self.W_delta[i]

 

The RNN is completed. Now we will train and test it.

 


rnn = RNN()
rnn.train(trainX, trainY, 20000)
print "Weight: \t", rnn.W
print "Real: \t\t", testY

y = rnn.forward_states(testX)[:, -1]
print "Predicted: \t",y

 


RNN Preformance
 

Resources:

http://peterroelants.github.io/posts/rnn_implementation_part01/

 

One comment on “Binary to Decimal using Recurrent Neural Network (RNN)”

Leave a Reply

Your email address will not be published. Required fields are marked *