Data_structures/Arrays_numpy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../figures/HeaDS_logo_large_withTitle.png\" width=\"300\">\n",
    "\n",
    "<img src=\"../figures/tsunami_logo.png\" width=\"600\">\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Center-for-Health-Data-Science/PythonTsunami/blob/intro/0_intro.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OJ2ayTn6J9oO",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Arrays\n",
    "\n",
    "## Numpy\n",
    "    \n",
    "<img src=\"https://numpy.org/images/logos/numpy.svg\" width=\"200\">\n",
    "    \n",
    "*Prepared by [Jakob Nybo Nissen](https://twitter.com/nybojakob), continued by [Henry Webel](https://twitter.com/Henrywebel)*\n",
    "    \n",
    "The first half of this Notebook contain a small introduction to `numpy`. You can also see a [the YouTube video](https://www.youtube.com/watch?v=8Mpc9ukltVA). If you feel comfortable using [`numpy`](https://numpy.org/), you can skip the introduction and go directly to the [exercises (click here)](#exercises). If you would like a recap, keep reading on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OJ2ayTn6J9oO",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Introduction\n",
    "\n",
    "[`numpy`](https://numpy.org/) is a Python package that provides a new type of object: The `ndarray`. This is an N-dimensional array, i.e. a \"list\" with any number of dimensions.\n",
    "`numpy` is one of the most fundamental Python packages. Almost all of scientific Python uses [`numpy`](https://numpy.org/) either directly, or indirectly through another package.\n",
    "\n",
    "[![Figure 2, Harris et. al. 2020](https://media.springernature.com/full/springer-static/image/art%3A10.1038%2Fs41586-020-2649-2/MediaObjects/41586_2020_2649_Fig2_HTML.png?as=webp)](https://www.nature.com/articles/s41586-020-2649-2/figures/2)\n",
    "\n",
    "It may not be immediately clear why [`numpy`](https://numpy.org/)'s `ndarrays` are so useful that they are everywhere. Do ALL scientific software really need N-dimensional arrays? As you will learn in these exercises, even for 1-dimensional data that *could* be placed in lists, `ndarrays` are generally useful for their convenience and speed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OJ2ayTn6J9oO",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Reference\n",
    "Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., del Río, J. F., Wiebe, M., Peterson, P., … Oliphant, T. E. (2020). Array programming with NumPy. Nature, 585(7825), 357–362. https://doi.org/10.1038/s41586-020-2649-2\n",
    "\n",
    "\n",
    "[![Figure1, Harris et. al. 2020](https://media.springernature.com/full/springer-static/image/art%3A10.1038%2Fs41586-020-2649-2/MediaObjects/41586_2020_2649_Fig1_HTML.png?as=webp)](https://www.nature.com/articles/s41586-020-2649-2/figures/1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OJ2ayTn6J9oO",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Outline\n",
    "Today, first start with 1-dimensionalal `ndarray`s and use some statistical operations on them. Then, we will have a look at multidimensional arrays by using matrices, i.e. 2-dimensional `ndarray`'s.\n",
    "\n",
    "Let's get started right away and import `numpy`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "executionInfo": {
     "elapsed": 846,
     "status": "ok",
     "timestamp": 1607165545393,
     "user": {
      "displayName": "Henry Webel",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjBXWWA4vUGwERb3tbUtvUVonuRuRodm6TbGPT5=s64",
      "userId": "10091992791924355375"
     },
     "user_tz": -60
    },
    "id": "9kPidT2uJ9oO",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# We can import like this to rename numpy \"np\"\n",
    "# and make the name shorter\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "executionInfo": {
     "elapsed": 1065,
     "status": "ok",
     "timestamp": 1607165547075,
     "user": {
      "displayName": "Henry Webel",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjBXWWA4vUGwERb3tbUtvUVonuRuRodm6TbGPT5=s64",
      "userId": "10091992791924355375"
     },
     "user_tz": -60
    },
    "id": "_Hr0fl9_0V7M",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# # You could accces your drive files by connecting your drive storage\n",
    "# from google.colab import drive\n",
    "# drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "F1pIGPVjJ9oO",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Instantiating (creating) `ndarray`s\n",
    "We begin with a normal Python list of numbers. *italicized text*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 677,
     "status": "ok",
     "timestamp": 1607165547464,
     "user": {
      "displayName": "Henry Webel",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjBXWWA4vUGwERb3tbUtvUVonuRuRodm6TbGPT5=s64",
      "userId": "10091992791924355375"
     },
     "user_tz": -60
    },
    "id": "zR_bgxWwJ9oO",
    "outputId": "5a3c78f9-aed2-44e7-96a0-bf48b716a4f8",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4, 9, 1, 0, 8, 3, 2, 2, 6, 5, 0]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "values = [4, 9, 1, 0, 8, 3, 2, 2, 6, 5, 0]\n",
    "values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7jDP3Mu2J9oO",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "---\n",
    "We can create an `ndarray` out of this list using the constructor function [`numpy.array`](https://numpy.org/doc/stable/reference/generated/numpy.array.html#numpy.array), which we pass some data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 860,
     "status": "ok",
     "timestamp": 1607165548307,
     "user": {
      "displayName": "Henry Webel",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjBXWWA4vUGwERb3tbUtvUVonuRuRodm6TbGPT5=s64",
      "userId": "10091992791924355375"
     },
     "user_tz": -60
    },
    "id": "-2_p90cFJ9oO",
    "outputId": "81fc3268-4e24-420f-84ad-f72db83ffdaf",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4 9 1 0 8 3 2 2 6 5 0]\n",
      "<class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "array = np.array(values)\n",
    "print(array)\n",
    "print(type(array))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zKsWAk7CJ9oP",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You notice the `ndarray` looks a lot like a `list`. You can also construct `ndarrays` from a tuple. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sXL5SHVQJ9oP",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "---\n",
    "There are many other ways to instantiate (i.e. create) `ndarray`s. Here are a few useful ones to create vector, i.e. `1d`-`ndarray`s. You might need them later!\n",
    "\n",
    "\n",
    "| function | Purpose |  Example |\n",
    "| -------- | ------- | ---------|\n",
    "| **`np.arange`**                  | Makes an array with all the integers between two values | np.arange(2, 7) |\n",
    "| **`np.linspace`**               | Makes a specific-length array |  np.linspace(2, 3, 10) |\n",
    "| **`np.zeros`**                    | Makes an array of all zeros | np.zeros(5) |\n",
    "| **`np.ones`**                     | Makes an array of all ones | np.ones(3) |\n",
    "| **`np.random.random`** | Makes an array of random numbers | np.random.random(100) |\n",
    "| **`np.random.randn`**     | Makes an array of normally-distributed random numbers | np.random.randn(100) |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 534,
     "status": "ok",
     "timestamp": 1606991863784,
     "user": {
      "displayName": "Henry Webel",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjBXWWA4vUGwERb3tbUtvUVonuRuRodm6TbGPT5=s64",
      "userId": "10091992791924355375"
     },
     "user_tz": -60
    },
    "id": "is4IYBIiJ9oP",
    "outputId": "07befe44-6dd6-4e52-a5ce-9d116fe138ae",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "np.zeros: [0 0 0 0 0 0 0 0 0 0]\n",
      "np.arange: [ 3  5  7  9 11 13]\n",
      "np.linspace: [ 3.   4.5  6.   7.5  9.  10.5 12.  13.5 15. ]\n",
      "np.random.random: [0.94478044 0.00878618 0.07009674 0.39555037 0.91589158]\n",
      "np.random.randint: [10  7  8  8 16 10 12 18 16]\n"
     ]
    }
   ],
   "source": [
    "# Create a vector 10 zeros, default dtype is np.float64\n",
    "print('np.zeros:', np.zeros(10, dtype=np.int64))\n",
    "\n",
    "# Create an array of a range, similar to the range object\n",
    "print('np.arange:', np.arange(3, 15, 2))\n",
    "\n",
    "# Create vector of 10 floats from 3 to 15 having 9\n",
    "# evenly spaced elements (linear spacing):\n",
    "print('np.linspace:', np.linspace(3, 15, 9))\n",
    "\n",
    "# Create a vector of random numbers in the interval [0.0, 1.0)\n",
    "print('np.random.random:', np.random.random(5))\n",
    "\n",
    "# Create 9 random integers from 7 to 18\n",
    "print('np.random.randint:', np.random.randint(7, 19, size=9))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1: Create Arrays\n",
    "1. Define an array called `things` that 10 integers.  The data is completely up to you.\n",
    "2. Define an array of 20 random floats from 0 to 100\n",
    "3. Define an evenly spaced array of 20 random floats from 0 to 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oh5bCq95jWbz",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Slicing\n",
    "We can access and assign individual numbers in an `ndarray` like we do with `list`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 848,
     "status": "ok",
     "timestamp": 1607165581228,
     "user": {
      "displayName": "Henry Webel",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjBXWWA4vUGwERb3tbUtvUVonuRuRodm6TbGPT5=s64",
      "userId": "10091992791924355375"
     },
     "user_tz": -60
    },
    "id": "WUWAYd-SJ9oP",
    "outputId": "4783beef-0fc1-444b-86dc-50f32c304f11",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4 9 1 0 8 3 2 2 6 5 0]\n",
      "1\n",
      "[1 0 8 3]\n",
      "[ 4  9  1 10  8  3  2  2  6  5  0]\n"
     ]
    }
   ],
   "source": [
    "print(array)\n",
    "print(array[2])\n",
    "print(array[2:6])\n",
    "array[3] = 10\n",
    "print(array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "opRfLr94J9oP",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "---\n",
    "We can also loop over the `ndarray`, reverse it, and so on. The full list of methods you find in the docs of [`numpy.ndarray`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html)\n",
    "\n",
    "There are two main limitations that `ndarrays` have compared to lists. We cannot use `append` or `pop`, or other operations that *changes* the array *in-place*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 163
    },
    "executionInfo": {
     "elapsed": 491,
     "status": "error",
     "timestamp": 1606992005152,
     "user": {
      "displayName": "Henry Webel",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjBXWWA4vUGwERb3tbUtvUVonuRuRodm6TbGPT5=s64",
      "userId": "10091992791924355375"
     },
     "user_tz": -60
    },
    "id": "LugFA4AUJ9oP",
    "outputId": "2eff65a2-e54b-40c6-fb5a-3dff2cffc246",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'numpy.ndarray' object has no attribute 'append'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-9-e9cb6c9fca85>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0marray\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m3\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m: 'numpy.ndarray' object has no attribute 'append'"
     ]
    }
   ],
   "source": [
    "array.append(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can however append several arrays together is slightly different way by using numpy.append(arr, values, axis=None) [`numpy.append`](https://numpy.org/doc/stable/reference/generated/numpy.append.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.append([1, 2, 3], [1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.append([1, 2, 3], [[4, 5, 6], [7, 8, 9]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2: Access items\n",
    "1. Print out the second element in your array `things`.\n",
    "2. Append `[4, 5, 22]` to `things`\n",
    "3. Remove `22` from `things` (you need to find this one)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ONbVG4LiJ9oP",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Data type of arrays (`dtype`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5-JwMXAxJ9oP",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The second disadvantage is that an `ndarray` can only contain one kind of data at a time, i.e. `float` or `int` etc. The data type is given by the `dtype` field of the array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 508,
     "status": "ok",
     "timestamp": 1606991880619,
     "user": {
      "displayName": "Henry Webel",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjBXWWA4vUGwERb3tbUtvUVonuRuRodm6TbGPT5=s64",
      "userId": "10091992791924355375"
     },
     "user_tz": -60
    },
    "id": "bsHqA_ScJ9oP",
    "outputId": "54b21794-b0b5-48f2-ccd8-0492e403fc86",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "array.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7tS1xGoOJ9oP",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "---\n",
    "This array contains `np.int64`, a type representing 64-bit integers. If we try to add a `float` to the array, it will be converted to an `np.int64`, or raise an error if that is not possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 214
    },
    "executionInfo": {
     "elapsed": 506,
     "status": "error",
     "timestamp": 1606991892783,
     "user": {
      "displayName": "Henry Webel",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjBXWWA4vUGwERb3tbUtvUVonuRuRodm6TbGPT5=s64",
      "userId": "10091992791924355375"
     },
     "user_tz": -60
    },
    "id": "r6ewCKVgJ9oP",
    "outputId": "db8fc94e-c6dc-401b-d0a1-2bbb3dba5d14",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# 1.7 can be \"converted\" by rounding down to 1\n",
    "# (i.e) removing the fraction part, keeping the interger part 1\n",
    "array[2] = 1.7\n",
    "print(array)\n",
    "\n",
    "# \"hello\" cannot be converted to an integer\n",
    "array[2] = \"hello\" # \"2\" would work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a type that allows mixing items of different type: object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "array2 = array.astype('object')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "array2[2] = \"hello\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "array2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TqSwL4LzJ9oP",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "---\n",
    "There are many possible datatypes - you can read more about them [on this link (click)](https://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html). However, we will only look at a few data types.\n",
    "\n",
    "When you construct an `ndarray`, `numpy` will try to automatically detect the input data type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JoUGrWpiJ9oP",
    "outputId": "6a1677f8-c07b-41db-dcf9-21febe455716",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "array = np.array([1, 2, 3]) # again: constructor utility function np.array\n",
    "print(array, array.dtype)\n",
    "\n",
    "array = np.array([1.0, 2.0, 3])\n",
    "print(array, array.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HOQkIfYYJ9oP",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "---\n",
    "Be careful when constructing `ndarrays` - for example if you try to construct an `ndarray` from a set object, which is not a list or a tuple, it simply create a 1-element `ndarray` containing the set as its only element, and the datatype will be `object`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hCkBzM6eJ9oP",
    "outputId": "b05399ef-3759-4970-ba77-39b2b0807030",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "array = np.array({1,2,3})\n",
    "print(\"array: \", array)\n",
    "print(\"dtype: \", array.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qTLAy0-vJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "---\n",
    "You can also manually specify the data type you want. Notice here that it converts `2.5` to `2`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "WDQvwRzQJ9oQ",
    "outputId": "5b826477-6934-49e2-8e7e-98e6ded42637",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "np.array([1.0, 2.5, 3.0], dtype=np.int64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1aEaOgwbJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "---\n",
    "And you can convert one numpy array to one with another dtype:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qbmECMehJ9oQ",
    "outputId": "9d166f04-ab6f-4a8b-f76d-536695694747",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "array = np.array([1, 2, 3])\n",
    "array.astype(np.float64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3ztUHTVjJ9oQ",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Vectorized operations\n",
    "\n",
    "Conveniently, for many operations like `+`, `-`, `*`, `/`, `**` and others apply to each element of an `ndarray` automatically. For example, if you write: `array + 5`, this will add 5 to **every element** in the array. This is termed vectorized operations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QcSqFxJHJ9oQ",
    "outputId": "f6601d85-09f2-4dbe-c7dc-9c1fff9db7b3",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "array = np.array([4, 8, 1, 2, 9, 3, 7])\n",
    "print(array)\n",
    "print(array + 5)\n",
    "print(array * 2)\n",
    "print(array / 2)\n",
    "print(array**2)\n",
    "print(0.4 * array**2 + 1.5 * array - 0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "> Think about the difference of what the `+`, `-`, `*`, `/` operators invoke in comparison to `str` or `list` built-in types."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BJEELYKYJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Because these vectorized operations actually execute code written in the programming language C, they are much, much faster than ordinary Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 822,
     "status": "ok",
     "timestamp": 1607166291379,
     "user": {
      "displayName": "Henry Webel",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjBXWWA4vUGwERb3tbUtvUVonuRuRodm6TbGPT5=s64",
      "userId": "10091992791924355375"
     },
     "user_tz": -60
    },
    "id": "fiJom0z_J9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "# Create a vector of 1 million random floats in [0.0, 1.0):\n",
    "python_list = [random.random() for i in range(1_000_000)]\n",
    "\n",
    "# Convert the Python list to a Numpy vector \n",
    "numpy_vector = np.array(python_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 5716,
     "status": "ok",
     "timestamp": 1607166297172,
     "user": {
      "displayName": "Henry Webel",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjBXWWA4vUGwERb3tbUtvUVonuRuRodm6TbGPT5=s64",
      "userId": "10091992791924355375"
     },
     "user_tz": -60
    },
    "id": "P4kAVPZHJ9oQ",
    "outputId": "d913a4e8-ec2a-4b44-bb6a-e883cd622ece",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%timeit [2*x + 3 for x in python_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 12126,
     "status": "ok",
     "timestamp": 1607166304265,
     "user": {
      "displayName": "Henry Webel",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjBXWWA4vUGwERb3tbUtvUVonuRuRodm6TbGPT5=s64",
      "userId": "10091992791924355375"
     },
     "user_tz": -60
    },
    "id": "DyDXVb0mJ9oQ",
    "outputId": "fb43716d-2aeb-4f2b-f4bf-6f139d0b6694",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# On Jacob's computer, this is >200x faster than Python \n",
    "# In colab (with the lastest Python version?) this is still ~70x faster\n",
    "# On my machine it is reduced to a 22x faster\n",
    "%timeit 2*numpy_vector + 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L-ec4YtjJ9oQ",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Masking\n",
    "Importantly, the comparison operators `>`, `>=`, `<`, `<=`, `==`, `!=` are also vectorized. Since these operators always return a `bool` (i.e. `True` or `False`), the result is an `ndarray` of data type `bool`. For example, we can check whether each element in `array` is greater than 3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ON9EBBLqJ9oQ",
    "outputId": "df52cc0c-4cbb-465f-bad8-e1002b3318c0",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "array = np.array([4, 8, 1, 2, 9, 3, 7])\n",
    "mask_above_3 = array > 3\n",
    "print(mask_above_3)  \n",
    "print(mask_above_3.dtype)\n",
    "array[mask_above_3] # explanation: see next section"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3: Masking\n",
    "1. Define an array of 100 random floats from 0 to 1000 and select those elements that are below 500\n",
    "3. In the same array, select those elements above 50 and below 300"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uZ-tvr0tJ9oQ",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Logical indexing using masking\n",
    "*Logical indexing* refers to indexing an `ndarray` using another `ndarray` of dtype `bool`. This selects every element of the `ndarray` paired with the value `True`: For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "A668kz_QJ9oQ",
    "outputId": "d3be0424-00a3-434d-e5cf-cdb7469b45d2",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "array: [ 1  4  6  5  0  2  9 10  0  2  7  5]\n",
      "nonzero: [ True  True  True  True False  True  True  True False  True  True  True]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([ 1,  4,  6,  5,  2,  9, 10,  2,  7,  5])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "array = np.array([1, 4, 6, 5, 0, 2, 9, 10, 0, 2, 7, 5])\n",
    "print('array:', array)\n",
    "mask_nonzero = array != 0\n",
    "print('nonzero:', mask_nonzero)\n",
    "\n",
    "# Now get all elements that are not zero:\n",
    "# Each element in array gets picked if nonzero is True\n",
    "array[mask_nonzero]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LW44xq4BJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "---\n",
    "You can also assign to an `ndarray` using logical indexing. This will change all the numbers where the boolean array is `True`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "l7__VYK6J9oQ",
    "outputId": "acc12161-c60f-4856-f8fd-ea47bcf97cbe",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[100 100 100 100   0 100 100 100   0 100 100 100]\n"
     ]
    }
   ],
   "source": [
    "array[mask_nonzero] = 100\n",
    "print(array) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To combine you can use the \"bit-wise\" operators `&` (logical and) and `|` (logical or), or one of the [logical operations](https://numpy.org/doc/stable/reference/routines.logic.html#logical-operations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CBVH3SVPJ9oQ",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## `ndarray` methods \n",
    "\n",
    "In addition to the vectorized opertaions, the `array` object also have some methods (i.e. functions of the object type or class `array`) to conveniently perform operations on the arrays. Like vectorized operations, these methods are very fast. The full list of methods you find in the docs of [`numpy.array`](https://numpy.org/doc/stable/reference/generated/numpy.array.html#numpy.array)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 491,
     "status": "ok",
     "timestamp": 1606992022239,
     "user": {
      "displayName": "Henry Webel",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjBXWWA4vUGwERb3tbUtvUVonuRuRodm6TbGPT5=s64",
      "userId": "10091992791924355375"
     },
     "user_tz": -60
    },
    "id": "mDYUldVNJ9oQ",
    "outputId": "94494096-d369-4c5d-c962-ee53c8e374b8",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Our array: [4 8 1 2 9 3 7]\n",
      "Calculate sum: 34\n",
      "Calculate mean: 4.857142857142857\n",
      "Calculate variance: 8.408163265306122\n",
      "Calculate standard deviation: 2.8996833043120627\n",
      "Calculate minimum/maximum: 1 9\n"
     ]
    }
   ],
   "source": [
    "array = np.array([4, 8, 1, 2, 9, 3, 7])\n",
    "print(\"Our array:\", array)\n",
    "print(\"Calculate sum:\", array.sum())\n",
    "print(\"Calculate mean:\", array.mean())\n",
    "print(\"Calculate variance:\", array.var())\n",
    "print(\"Calculate standard deviation:\", array.std())\n",
    "print(\"Calculate minimum/maximum:\", array.min(), array.max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The methods performed on the `ndarray` can also be performed on different datastructures than arrays using numpy functions with idential names to the methods of an `ndarray`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 491,
     "status": "ok",
     "timestamp": 1606992022239,
     "user": {
      "displayName": "Henry Webel",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjBXWWA4vUGwERb3tbUtvUVonuRuRodm6TbGPT5=s64",
      "userId": "10091992791924355375"
     },
     "user_tz": -60
    },
    "id": "mDYUldVNJ9oQ",
    "outputId": "94494096-d369-4c5d-c962-ee53c8e374b8",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "34"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum([4, 8, 1, 2, 9, 3, 7])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "D_9ljlyHJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "---\n",
    "Additional functions are not implemented as methods of `ndarray`, but can be found by typing `np.FUNCITON_NAME` such as:\n",
    "\n",
    "```python\n",
    "np.log\n",
    "np.sqrt\n",
    "np.any\n",
    "np.all,\n",
    "```\n",
    "and many, many more. In general, basic math operations, statistical operations and linear algebra is implemented in Numpy. Any other kind of number-crunching operation you might possibly need can be found in Scipy.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 510,
     "status": "ok",
     "timestamp": 1606992027028,
     "user": {
      "displayName": "Henry Webel",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjBXWWA4vUGwERb3tbUtvUVonuRuRodm6TbGPT5=s64",
      "userId": "10091992791924355375"
     },
     "user_tz": -60
    },
    "id": "yRNq7ZSIJ9oQ",
    "outputId": "d20d282f-08e4-41fa-d06d-142556e864db",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.38629436 2.07944154 0.         0.69314718 2.19722458 1.09861229\n",
      " 1.94591015]\n"
     ]
    }
   ],
   "source": [
    " print(np.log(array)) # ln"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Higher dimensional arrays\n",
    "\n",
    "The reason they are called `ndarray`s are that they are N-dimensional. \n",
    "In the exercises, you can work on matrices, i.e. `2d-array`s.\n",
    "\n",
    "First a little notation:\n",
    "* A 1-dimensional array is a vector. A `2d-array` is a matrix. An N-dimensional array is called a tensor or an array.\n",
    "* The number of dimensions of a tensor is its rank. So a matrix is a tensor of rank 2.\n",
    "* In `numpy`, the size of an array refers to the total number of elements. The shape of an array is a tuple with the length of the array in each of its dimensions. The length of an array is the length of the first dimension.\n",
    "* An axis is the same as a dimension.\n",
    "\n",
    "Many ways of instantiating `ndarrays` have an optional keyword that allows us to specify its shape:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Instantiate\n",
    "- list of list, etc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 5, 3, 9],\n",
       "       [9, 4, 2, 4],\n",
       "       [0, 6, 5, 1]])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = [[1, 5, 3, 9],\n",
    "        [9, 4, 2, 4],\n",
    "        [0, 6, 5, 1]]\n",
    "data = np.array(data)\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (3, 4)\n",
      "Dimensions (Axes): 2\n"
     ]
    }
   ],
   "source": [
    "print(\"Shape:\", data.shape)\n",
    "print(\"Dimensions (Axes):\", data.ndim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### reshape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 5, 3],\n",
       "       [9, 9, 4],\n",
       "       [2, 4, 0],\n",
       "       [6, 5, 1]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.reshape((4,-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Axis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 5 3 9]\n",
      " [9 4 2 4]\n",
      " [0 6 5 1]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "49"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(data)\n",
    "data.sum() # default along all axis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([10, 15, 10, 14])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.sum(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([18, 19, 12])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.sum(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### new axis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 4, 1)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[:,:, np.newaxis].shape # sometimes needed for broadcasting (next)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### broadcasting\n",
    "\n",
    "**Broadcasting** is a technique automatically matching arrays of different shape - if they are compatible. Read the [numpy guide on broadcasting](https://numpy.org/doc/stable/user/basics.broadcasting.html) (which is the default behaviour in many array libraries now)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 5, 25, 15, 45],\n",
       "       [45, 20, 10, 20],\n",
       "       [ 0, 30, 25,  5]])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data * 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.05555556, 0.27777778, 0.16666667, 0.5       ],\n",
       "       [0.47368421, 0.21052632, 0.10526316, 0.21052632],\n",
       "       [0.        , 0.5       , 0.41666667, 0.08333333]])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data / data.sum(axis=1)[:, np.newaxis]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ol5el84hJ9oQ",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Exercises\n",
    "\n",
    "In all of these exercises, do not loop over the `ndarray`s you create. All exercises can be solved using only vectorized operations on your arrays."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E8lcv9BwJ9oQ",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Simple vectorized operations\n",
    "\n",
    "You want to plot the mathematical function\n",
    "\n",
    "$f(x) = log(-1.3x^2 + 1.4^x + 7x + 50)$\n",
    "\n",
    "For the numbers in $[0, 20]$. To do this, you need to create a vector `xs` with lots of numbers between 0 and 20, and a vector `ys` with $f$ evaluated at every element of `xs`. A vector is a `1d-ndarray`.\n",
    "\n",
    "To get a hang of vectorized operations, solve the problem *without using any loops*:\n",
    "\n",
    "### Create a vector `xs` with 1000 evenly spaced points between 0 and 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E8lcv9BwJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Create a Python function $f$ as seen above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E8lcv9BwJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Evaluate `ys` = $f(x)$, i.e. $f$ of every element of `xs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E8lcv9BwJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### What is the mean and standard deviation of `ys`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E8lcv9BwJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### How many elements of. `ys` are below 0? Between 1 and 2, both exclusive?\n",
    "\n",
    "> Hint: You can use a comparison operator to get an array of dtype `bool`. To get the number of elements that are `True`, you can exploit the fact that `True` behaves similar to the number 1, and `False` similar to the number 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E8lcv9BwJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### What is the minimum and maximum value of `ys`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E8lcv9BwJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Create a vector `non_negatives`, which contain all the values of `ys` that are nonnegative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E8lcv9BwJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### *Extra*: Use `matplotlib` to plot `xs` vs `ys`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "24xnu_lLJ9oQ",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Species matrix\n",
    "\n",
    "Load in the data [`depths.csv`](https://github.com/pythontsunami/teaching/tree/intro/data/depths.csv). As you can see in drive preview, there are 11 columns, with columns 2-11 representing a sample from a human git microbiome. Each row represents a genome of a micro-organism, a so-called \"operational taxonomic unit at 97% sequence identity\" (OTU_97). The first row gives the name of the genome. The values in the matrix represents the relative abundance (or depth) of that micro-organism in that sample, i.e. how much of the micro-organism there is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 804,
     "status": "ok",
     "timestamp": 1607019616710,
     "user": {
      "displayName": "Henry Webel",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjBXWWA4vUGwERb3tbUtvUVonuRuRodm6TbGPT5=s64",
      "userId": "10091992791924355375"
     },
     "user_tz": -60
    },
    "id": "jzoZyd3jJ9oQ",
    "outputId": "55404f1c-5d7b-45aa-fbd6-8d6b07a4897a",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# You can use Bash commands by prepending them with !, this function works\n",
    "# on my computer because I have the correct path to the file depths.csv\n",
    "from pathlib import Path\n",
    "filepath = Path('data/depths.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Download in colab\n",
    "!mkdir -p data\n",
    "!wget https://raw.githubusercontent.com/pythontsunami/teaching/intro/data/depths.csv -P data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dgRBUUlvJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Load in the matrix in some data structure of your choice, but make sure the numbers in each row is an `ndarray`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dgRBUUlvJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### How many OTUs are there? Show how you figured it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dgRBUUlvJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Find the OTU \"OTU_97.41189.0\". What is the mean and standard deviations of the depths across the 10 samples of this OTU?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dgRBUUlvJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### How many samples have 0 depth of that OTU? (or rather, below detection limit?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dgRBUUlvJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### What is the mean and standard deviation if you exclude those samples?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dgRBUUlvJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### We are not interested in OTUs present in fewer than 4 samples. Remove all those OTUs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dgRBUUlvJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### How many OTUs did you remove?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dgRBUUlvJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### How many OTUs have a depth of > 5 in all 10 samples? (hint: `np.all`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dgRBUUlvJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Filtering and Normalization\n",
    "\n",
    "After discarding all OTUs present in fewer than 4 samples, sort the OTUs, do the following:\n",
    " - Calculate the mean depth across samples for each remaining OTU.\n",
    " - Normalize the remaining OTUs such that each row sum to 0 and have a standard deviation of 1 (so-called z-score normalization)\n",
    " - Print the remaining OTUs to a new file in descending order by their mean depth, with a 12th column giving the mean depth, and columns 1-11 being the normalized depth. Make sure that your file looks like the input file (except with the 12th column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZF8Ay1vl1f3Z",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0EjvKxCPJ9oQ",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##  Optional exercise: Sequence data as an array\n",
    "_Only do it if you have extra time._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0EjvKxCPJ9oQ",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### load data \n",
    "\n",
    "Create a function that loads a FASTA file into a `dict` where each key is the FASTA header and the value is an  `ndarray` corresponding to the sequence. Remember that you need to instantiate `ndarray` from a list, not a string. Test it using the `Yersinia pestis` fasta file. You can verify that your `ndarray` has the correct length by checking the `.size` attribute:\n",
    "  \n",
    "`array.size`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0EjvKxCPJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Check data\n",
    "Make your function verify that there are only 'A', 'C', 'G', 'T' and 'N' in the sequence. Convert lowercase letters to uppercase letters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0EjvKxCPJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### GC count\n",
    "Create a function that calculates GC content of your `ndarray` representing a DNA sequence - ignore Ns. Remember to still use vectorized operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0EjvKxCPJ9oQ",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Profile\n",
    "Use the Jupyter `%timeit` functionality to compare the time spent in your vectorized GC-content function with the GC-content function you created in exercise 2 last monday. See the use of `%timeit` in this Notebook for an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dukAju8VJ9oQ",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "---\n",
    "__Congratulations, you finished today's exercise!__"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Numpy.ipynb",
   "provenance": [
    {
     "file_id": "1RWt6QETnGQ9VKrSOsop56XanZavkoQM9",
     "timestamp": 1606991634276
    }
   ],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  },
  "rise": {
   "scroll": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "333.991px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  },
  "toc-autonumbering": true,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}