04a Custom Activation Function.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Writing a custom activation function\n",
    "\n",
    "Activation functions (also called transforms in neon) are nonlinearities such as rectified linear, softmax, or hyperbolic tangent functions.  \n",
    "\n",
    "Generally these functions are bundled together with a neural network layer such as Affine or Convolution.\n",
    "\n",
    "We derive the Transform interface in neon, which specifies a __call__ and bprop function.  __call__ is used during forward pass computation, and bprop is used to compute the derivative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from neon.backends import gen_backend\n",
    "from neon.transforms.transform import Transform\n",
    "be = gen_backend('gpu', batch_size=128)\n",
    "\n",
    "class MySoftmax(Transform):\n",
    "    \"\"\"\n",
    "    SoftMax activation function. Ensures that the activation output sums to 1.\n",
    "    \"\"\"\n",
    "    def __init__(self, name=None):\n",
    "        super(MySoftmax, self).__init__(name)\n",
    "\n",
    "    def __call__(self, x):\n",
    "        \"\"\"\n",
    "        Implement softmax(x) = e^(x-max(x)) / sum(e^(x-max(x))). \n",
    "        \n",
    "        Input x has shape (# features, batch_size) \n",
    "        \n",
    "        Return softmax(x), with shape (# features, batch_size), but where the features sum to 1.\n",
    "                \n",
    "        \"\"\"\n",
    "\n",
    "        expx = self.be.exp(x - self.be.max(x, axis=0))\n",
    "        return expx / self.be.sum(expx, axis=0)\n",
    "    \n",
    "    def bprop(self, x):\n",
    "        \"\"\"\n",
    "        We take a shortcut here- the derivative cancels out with a term in the CrossEntropy derivative.\n",
    "        \"\"\"\n",
    "        return 1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test our softmax\n",
    "\n",
    "1) make some test data, on the host\n",
    "2) move the test data to the device (GPU)\n",
    "3) calculate softmax on our test data\n",
    "4) copy the result back to the host, and inspect that it's correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# generate some test data using numpy\n",
    "import numpy as np\n",
    "data_cpu = np.array([[1,1,1,1],\n",
    "                     [1,2,3,4],\n",
    "                     [1,3,5,7]])\n",
    "print data_cpu.shape\n",
    "\n",
    "# copy test data to the backend (GPU), and allocate an output buffer\n",
    "data = be.array(data_cpu)\n",
    "\n",
    "# test our softmax\n",
    "mysoftmax = MySoftmax()\n",
    "data[:] = mysoftmax(data)\n",
    "\n",
    "data_cpu = data.get()\n",
    "print data_cpu\n",
    "\n",
    "# validate that our output sums to one\n",
    "data_sum =  np.sum(data_cpu)\n",
    "assert 1 - data_sum < 0.0001"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}