rnn_python/README.md at master · Aightech/rnn_python · GitHub

RNN Network

Structure

Dimensions

$X\in \mathbb{R}^{n} $ \;,\; $Y\in \mathbb{R}^{n} $ \;,\; $S\in \mathbb{R}^{d} $ \;,\; $Z\in \mathbb{R}^{n} $ \;,\; $U\in \mathbb{R}^{d . n} $ \;,\; $V\in \mathbb{R}^{n . d} $ \;,\; $W\in \mathbb{R}^{d . d}$

Relations

$Y^t=V.S^t$ \;\; et \;\; $S^t=f(Z^t)$ \;\; avec \;\ $Z^t=U.X^t + W.S^{t-1}$ \;\; et \;\; $f = tanh$

Gradients

U

$\frac{\partial E^T}{\partial U} = \left ( \sum_{t=1}^{T} \frac{\partial E^t}{\partial u_{ij}}\right)_{ij}$

$\frac{\partial E^t}{\partial u_{ij}} = \sum_{k=1}^{n}\sum_{l=1}^{d}-2g(\hat{y}^t_k-y^t_k) . v_{kl} . \frac{\partial s^t_l}{\partial u_{ij}}$

$\frac{\partial s_l^0}{\partial u_{ij}} = 0 \;\;\;\; et \;\;\;\; \forall t \in [0,T], \;\frac{\partial s_l^t}{\partial u_{ij}} = f'(z^t_l).\left(x^t_j.\delta_{li} + \sum_{m=1}^{d}w_{lm}\frac{\partial s^t_m}{\partial u_{ij}} \right)$

V

$\frac{\partial E^T}{\partial V} = \left ( \sum_{t=1}^{T} \frac{\partial E^t}{\partial v_{ij}}\right)_{ij} = \left( \sum_{t=1}^{T}\frac{\partial E^t}{\partial y_i} . \frac{\partial y_i}{\partial v_{ij}} \right)_{ij}$

Results

Several Experts Only one expert

Several layers

Low level getes High level gates