Recurrent Neural Networks

Introduction

RNNs are a good architecture for working with sequential data. Data is received by the model one member of the sequence at a time. The relationship between adjacent members is essential to how RNN processes information. RNNs are better suited for sequences with many members, and varied lengths than the perceptron. Due to their ability to retain information about previous states and process large sequences, RNNs a good option for time-series analysis and natural language processing.

Parameter sharing:

Terminology

• Parameter sharing: Instead of treating each value in the time index as a separate entity, parameter sharing allows for a specific piece of information to be shared in the same manner, irrespective of where in the sequence it occurs, as opposed to assigning separate parameters. Example- a given token "sister" carries the same information, no matter where in the sentence we see it. This allows us to process "I gave my sister a gift" and "I gave a gift to my sister" to extract the recipient of the gift without bothering with the order of the words.

The Architecture

Use Cases

manu-to-many, or many-to-one

RNNs for NLP

Embedding

Memory