program-a-neuron-master.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "coskmeJSP8rt"
      },
      "source": [
        "# Introduction to Neural Networks\n",
        "\n",
        "Neural networks are a type of algorithm designed to recognize patterns and make decisions. It's called a \"neural network\" because the basic building block of the algorithm, a \"neuron,\" is inspired by how neurons in the human brain work.\n",
        "\n",
        "Biological neurons have dendrites to receive input, a body for processing inputs (like a threshold for sodium levels), and an axon with terminal ends to pass along information.\n",
        "![](https://nickmccullum.com/images/python-deep-learning/understanding-neurons-deep-learning/neuron-anatomy.png)\n",
        "\n",
        "Information moves from one neuron to the next in a network by getting passed from the first neuron's axon to the second neuron's dendrites.\n",
        "![](https://i1.wp.com/www.brains-explained.com/wp-content/uploads/2015/06/synapse.jpg)\n",
        "\n",
        "In __artificial neural networks__, neurons can be modeled or represented as a sequence of mathematical functions. Each neuron has multiple inputs (dendrites) and an output that can connect to other neurons (axon terminals). The neuron multiplies each of its inputs by a specific weight and adds a fixed value called a bias, similar to point-slope formula for making a line in algebra. The sum of all the inputs is calculated, and then a special transformation called an \"activation function\" is used to convert the sum into a value between 0 and 1 (sort of like the signal processing that occurs in the body of a real-life neuron). The output can then be passed to the next layer of neurons in the network.\n",
        "![](https://miro.medium.com/v2/resize:fit:1302/format:webp/1*UA30b0mJUPYoPvN8yJr2iQ.jpeg)\n",
        "\n",
        "To build an artificial neural network, multiple neurons can be connected with the outputs of one neuron becoming the inputs of the next neuron. These connections are sort of like synapses between biological neurons in the brain.\n",
        "![](https://nickmccullum.com/images/python-deep-learning/understanding-neurons-deep-learning/neuron-functionality.png)\n",
        "\n",
        "\n",
        "Neurons can be connected in layers with different patterns that perform a sequence of mathematical operations on the input data. The structure of the layera and the order of the operations is what allows the network to learn how to find patterns in the data. The first layer of neurons that receives the data is the input layer. The last layer of neurons outputs the prediction of the network, so this is called the \"output layer.\" The layers in-between the input and output layers are called \"hidden layers.\" In this diagram, each circle is a neuron in the artificial neural network.\n",
        "\n",
        "![](https://nickmccullum.com/images/python-deep-learning/what-is-deep-learning/artificial-neural-net.png)\n",
        "\n",
        "For articial neural networks to make accurate predictions, they must be trained to find what the between combinations of weights and biases should be for each neuron in the network. The values of the weights and biases are usually found through \"forward passes\" and \"backpropagation.\"\n",
        "\n",
        "In the _forward pass_, input data is fed into the network, and the network calculates a prediction. The accuracy of that prediction is then check using a \"loss function.\" Based on how accurate or inaccurate the prediction was, the weights and biases for each of the neurons are updated to a new value in a process called _backpropagation_. To get the most accurate predictions, the forward passes and backpropagation need to be repeated with A LOT of data.\n",
        "\n",
        "This training process uses calculus and linear algebra, so for this class we are going to use another method to train the network and learn the best values for the weights of a neuron in a neural network."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YoKyI9R8PEnY"
      },
      "source": [
        "## Coding a Simple Neuron from Scratch\n",
        "Without using back propagation or (fancy) Python classes. Instead, we will use functions and grid search to find the best combinations of weights for the dendrite of one neuron in a network.\n",
        "\n",
        "For this project, we will use two python libraries:\n",
        "1. [`NumPy`](https://numpy.org/doc/stable/index.html) for fast numerical calculations\n",
        "2. [`Itertools`](https://docs.python.org/3/library/itertools.html), which is a built-in module that we can use to generate search grids\n",
        "\n",
        "From `itertools`, we will use the `product()` function, which calculates all possible combinations of items from a series of lists. For example, if we had two lists: `list_1 = [1, 2, 3]` and `list_2 = [5,6,7]`, then the output from `itertools.product()` would be:\n",
        "\n",
        "|   | 5 | 6 | 7 |\n",
        "|---|---|---|---|\n",
        "| **1** | 1,5 | 1,6 | 1,7 |\n",
        "| **2** | 2,5 | 2,6 | 2,7 |\n",
        "| **3** | 3,5 | 3,6 | 3,7 |\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "4LuNKoDwLvHW"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "from itertools import product"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uc1PSaO3O_l3"
      },
      "source": [
        "### Generate Some Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "AsuOrJ-AL2KO"
      },
      "outputs": [],
      "source": [
        "# generate random synthetic data for training\n",
        "np.random.seed(2024)\n",
        "X_train = np.random.rand(1000, 3)\n",
        "y_train = (X_train[:, 0] + X_train[:, 1] + X_train[:, 2] > 1.5).astype(int) # make two classes based on the sum of each feature in the data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IbtoWXc3PO_r"
      },
      "source": [
        "### Exercises\n",
        "#### Define Helper Functions\n",
        "To emulate a neuron in an artificial neural network, we'll write Python functions for the neuron's activation function, a function for the neuron to make predictions, a function to check the neuron's accuracy, and a function to perform a grid search and find the best weights for the neuron's dendrites.\n",
        "\n",
        "#### Exercise 1: Activation Function\n",
        "Write the activation function to use in the neuron. Let's start with a sigmoid activation function that forces the neuron's output to be between 0 and 1."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "FwchsCNPTLkG"
      },
      "outputs": [],
      "source": [
        "def sigmoid_activation(z):\n",
        "  return 1/(1 + np.exp(-z))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4nX3gR0NT2a-"
      },
      "source": [
        "#### Exercise 2: Prediction Function\n",
        "Write a function that uses the neuron's weights to make a prediction given an input data point."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "a3p5BHlMUHBG"
      },
      "outputs": [],
      "source": [
        "def predict(X, weights):\n",
        "  z = np.dot(X, weights)\n",
        "  return (sigmoid_activation(z) > 0.5).astype(int)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Narbfgu8UVaK"
      },
      "source": [
        "#### Exercise 3: Calculate the Accuracy of the Neuron\n",
        "Write a function to calculate the percent accuracy of the neuron's predictions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "aOZ5bJ7QUe32"
      },
      "outputs": [],
      "source": [
        "def accuracy_score(true_labels, pred_labels):\n",
        "  return sum(true_labels == pred_labels)/len(true_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-clvAgnxUg5y"
      },
      "source": [
        "#### Exercise 4: Grid Search\n",
        "Write a function to perform a grid search and optimize values for the neuron's weights."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "8U7O8Rf7MTIq"
      },
      "outputs": [],
      "source": [
        "def grid_search(X, y, weight_range):\n",
        "  best_weights = None\n",
        "  best_accuracy = 0\n",
        "\n",
        "  for w1, w2, w3 in product(weight_range, repeat=3):\n",
        "    weights = np.array([w1, w2, w3])\n",
        "    predictions = predict(X, weights)\n",
        "    accuracy = accuracy_score(y, predictions)\n",
        "    if accuracy > best_accuracy:\n",
        "      best_accuracy = accuracy\n",
        "      best_weights = weights\n",
        "  return best_weights, best_accuracy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zSH0HHWxPSB9"
      },
      "source": [
        "#### Exercise 5: Train the Neuron!\n",
        "To train the neuron with a grid search, we first need to establish the upper and lower limits or the search space."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "udv2ut5JMTvJ",
        "outputId": "25956a7c-9292-4791-fd1a-46c49f0378cd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Best weights: [-5.35353535  9.19191919  2.72727273]\n",
            "Best accuracy: 0.603\n"
          ]
        }
      ],
      "source": [
        "# define the bounds of our grid search for each parameter\n",
        "weight_range = np.linspace(-10, 10, 100)\n",
        "\n",
        "# obtain the weights and accuracy of our model by a grid search\n",
        "best_weights, best_accuracy = grid_search(X_train, y_train, weight_range)\n",
        "\n",
        "# let's see how we did!\n",
        "print(f\"Best weights: {best_weights}\")\n",
        "print(f\"Best accuracy: {best_accuracy}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2R9_QlqIPVl7"
      },
      "source": [
        "#### Exercise 6: Test the Neuron!\n",
        "Generate some new data, use the neuron to predict, and then check accuracy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "2BYREXoJOdK6"
      },
      "outputs": [],
      "source": [
        "# generate random synthetic data for testing\n",
        "np.random.seed(123)\n",
        "X_test = np.random.rand(100, 3) #\n",
        "y_test = (X_test[:, 0] + X_test[:, 1] + X_test[:, 2] > 1.5).astype(int)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iY_fUSBlQuM3",
        "outputId": "80fb3e2c-e3ed-47e1-c157-292940905624"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy on the test data: 56.00000000000001%\n"
          ]
        }
      ],
      "source": [
        "test_predictions = predict(X_test, best_weights)\n",
        "prediction_accuracy = accuracy_score(y_test, test_predictions)\n",
        "print(f\"Accuracy on the test data: {100*np.round(prediction_accuracy, 2)}%\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "34C0k2CrSy9F"
      },
      "source": [
        "## Extra Exercises\n",
        "1. Try an expanded grid search. (Easy)\n",
        "2. Try another activation function. (Medium)\n",
        "3. Add another parameter (weight) to learn. (Hard)\n",
        "4. Put more neurons together! Try making two neurons feed into a third neuron. (Very Hard!)\n",
        "\n",
        "### Exercise 2.1\n",
        "Try to make the neuron more accurate by increasing the range of potential values each of the weights can be. What changes do you notice when you try to train the neuron with the expanded grid search?\n",
        "\n",
        "**BONUS: How long does it take to train the neuron with different grid searches?**\n",
        "Look at the documentation for Python's [time](https://docs.python.org/3/library/time.html) module and see if you can figure out how long it will take with bigger and bigger search grids."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pSb9gzzad01K",
        "outputId": "2fed783e-4950-4f70-85b5-64ce66d950e2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Best weights: [-10.75376884  18.3919598    5.52763819]\n",
            "Best accuracy: 0.603\n"
          ]
        }
      ],
      "source": [
        "# define the bounds of our grid search for each parameter\n",
        "weight_range = np.linspace(-20, 20, 200)\n",
        "\n",
        "# obtain the weights and accuracy of our model by a grid search\n",
        "best_weights, best_accuracy = grid_search(X_train, y_train, weight_range)\n",
        "\n",
        "# let's see how we did!\n",
        "print(f\"Best weights: {best_weights}\")\n",
        "print(f\"Best accuracy: {best_accuracy}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u-iwKrYzdFg_"
      },
      "source": [
        "### Exercise 2.2\n",
        "Write a helper function (call it `relu_activation()`) for the ReLU (Rectified Linear Unit) activation function. Modify the prediction function (call it `predict_relu()`) from exercise 1.2 to use the new `relu_activation()` function. Then use the grid search function to train the neuron with the new activation function. Which activaiton function led to a more accurate neuron?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "02u_QL8HVVAg"
      },
      "outputs": [],
      "source": [
        "def relu_activation(z):\n",
        "  return max(0, x)\n",
        "\n",
        "def predict_relu(X, weights):\n",
        "  z = np.dot(X, weights)\n",
        "  return (relu_activation(z) > 0.5).astype(int)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-maM7BrGbe3v"
      },
      "source": [
        "#### Exercise 2.3\n",
        "Modify the data generation code to include another measurement about each data point. Then adjust the `grid_search()` function to learn the weight for the additional measurement. Re-train the neuron and see how well it predicts with more data!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "913A1yBtcKeY"
      },
      "outputs": [],
      "source": [
        "# generate random synthetic data for training\n",
        "np.random.seed(2024)\n",
        "X_train2 = np.random.rand(1000, 4)\n",
        "y_train2 = (X_train2[:, 0] + X_train2[:, 1] + X_train2[:, 2] + X_train2[:, 3] > 1.75).astype(int) # make two classes based on the sum of each feature in the data\n",
        "\n",
        "def grid_search_2(X, y, weight_range):\n",
        "  best_weights = None\n",
        "  best_accuracy = 0\n",
        "\n",
        "  for w1, w2, w3, w4 in product(weight_range, repeat=4):\n",
        "    weights = np.array([w1, w2, w3, w4])\n",
        "    predictions = predict(X, weights)\n",
        "    accuracy = accuracy_score(y, predictions)\n",
        "    if accuracy > best_accuracy:\n",
        "      best_accuracy = accuracy\n",
        "      best_weights = weights\n",
        "  return best_weights, best_accuracy\n",
        "\n",
        "# define the bounds of our grid search for each parameter\n",
        "weight_range = np.linspace(-10, 10, 100)\n",
        "\n",
        "# obtain the weights and accuracy of our model by a grid search\n",
        "best_weights, best_accuracy = grid_search(X_train2, y_train2, weight_range)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NSupyDHweF-C"
      },
      "source": [
        "### Exercise 2.4\n",
        "Let's make a neural _network_! Try training two neurons together, that feed into a third neuron to make the final prediction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Umiw_VNDeGfv"
      },
      "outputs": [],
      "source": [
        "# prompt: Make a simple neural network from scratch in python. It should have two neurons in the first layer, and one neuron in the second layer, with no hidden layers.\n",
        "\n",
        "import numpy as np\n",
        "from itertools import product\n",
        "\n",
        "def sigmoid_activation(z):\n",
        "  return 1/(1 + np.exp(-z))\n",
        "\n",
        "def predict(X, weights):\n",
        "  z = np.dot(X, weights)\n",
        "  return (sigmoid_activation(z) > 0.5).astype(int)\n",
        "\n",
        "def accuracy_score(true_labels, pred_labels):\n",
        "  return sum(true_labels == pred_labels)/len(true_labels)\n",
        "\n",
        "def grid_search(X, y, weight_range):\n",
        "  best_weights = None\n",
        "  best_accuracy = 0\n",
        "\n",
        "  for w1, w2, w3 in product(weight_range, repeat=3):\n",
        "    weights = np.array([w1, w2, w3])\n",
        "    predictions = predict(X, weights)\n",
        "    accuracy = accuracy_score(y, predictions)\n",
        "    if accuracy > best_accuracy:\n",
        "      best_accuracy = accuracy\n",
        "      best_weights = weights\n",
        "  return best_weights, best_accuracy\n",
        "\n",
        "# Generate some random data\n",
        "np.random.seed(2024)\n",
        "X_train = np.random.rand(1000, 2)\n",
        "y_train = (X_train[:, 0] + X_train[:, 1] > 1).astype(int)\n",
        "\n",
        "# Define the bounds of our grid search for each parameter\n",
        "weight_range = np.linspace(-10, 10, 100)\n",
        "\n",
        "# Train the first layer of neurons\n",
        "best_weights_1, best_accuracy_1 = grid_search(X_train, y_train, weight_range)\n",
        "\n",
        "# Generate the output of the first layer of neurons\n",
        "X_train_layer2 = sigmoid_activation(np.dot(X_train, best_weights_1))\n",
        "\n",
        "# Train the second layer of neurons\n",
        "best_weights_2, best_accuracy_2 = grid_search(X_train_layer2, y_train, weight_range)\n",
        "\n",
        "# Make predictions with the trained network\n",
        "y_pred = predict(X_train_layer2, best_weights_2)\n",
        "\n",
        "# Calculate the accuracy of the network\n",
        "accuracy = accuracy_score(y_train, y_pred)\n",
        "\n",
        "print(f\"Accuracy of the neural network: {accuracy}\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CXvaJ2SqfHpt"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "from itertools import product\n",
        "\n",
        "def sigmoid_activation(z):\n",
        "  return 1/(1 + np.exp(-z))\n",
        "\n",
        "def predict(X, weights):\n",
        "  z = np.dot(X, weights)\n",
        "  return (sigmoid_activation(z) > 0.5).astype(int)\n",
        "\n",
        "def accuracy_score(true_labels, pred_labels):\n",
        "  return sum(true_labels == pred_labels)/len(true_labels)\n",
        "\n",
        "def grid_search(X, y, weight_range):\n",
        "  best_weights = None\n",
        "  best_accuracy = 0\n",
        "\n",
        "  for w1, w2, w3 in product(weight_range, repeat=3):\n",
        "    weights = np.array([w1, w2, w3])\n",
        "    predictions = predict(X, weights)\n",
        "    accuracy = accuracy_score(y, predictions)\n",
        "    if accuracy > best_accuracy:\n",
        "      best_accuracy = accuracy\n",
        "      best_weights = weights\n",
        "  return best_weights, best_accuracy\n",
        "\n",
        "# Generate some random data\n",
        "np.random.seed(2024)\n",
        "X_train = np.random.rand(1000, 2)\n",
        "y_train = (X_train[:, 0] + X_train[:, 1] > 1).astype(int)\n",
        "\n",
        "# Define the bounds of our grid search for each parameter\n",
        "weight_range = np.linspace(-10, 10, 100)\n",
        "\n",
        "# Train the first layer of neurons\n",
        "best_weights_1, best_accuracy_1 = grid_search(X_train, y_train, weight_range)\n",
        "\n",
        "# Generate the output of the first layer of neurons\n",
        "X_train_layer2 = sigmoid_activation(np.dot(X_train, best_weights_1))\n",
        "\n",
        "# Train the second layer of neurons\n",
        "best_weights_2, best_accuracy_2 = grid_search(X_train_layer2, y_train, weight_range)\n",
        "\n",
        "# Make predictions with the trained network\n",
        "y_pred = predict(X_train_layer2, best_weights_2)\n",
        "\n",
        "# Calculate the accuracy of the network\n",
        "accuracy = accuracy_score(y_train, y_pred)\n",
        "\n",
        "print(f\"Accuracy of the neural network: {accuracy}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "10"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "np.sum(np.array([1,2,3,4]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}