Part1_Datasets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Working with data in `PyTorch`\n",
    "<a id=part1></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part, we'll learn about the `Dataset` and `DataLoader` classes which are part of `PyTorch`'s `torch.util.data` package.\n",
    "These are highly useful abstractions that can greatly reduce the amount of boilerplate code you need to write in order to work with data.\n",
    "Knowing how to use these classes properly will prove useful in the coming assignments and course project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import unittest\n",
    "\n",
    "%matplotlib inline\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "plt.rcParams.update({'font.size': 12})\n",
    "torch.random.manual_seed(1904)\n",
    "test = unittest.TestCase()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Datasets\n",
    "<a id=part1_1></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Dataset` class is an abstraction over a sequence of python objects,\n",
    "each representing a sample (with or without a label). it's main purpose is\n",
    "to load a single (possibly labeled) sample from some soure (disk, web, etc) into memory,\n",
    "and transform it into a usuable representation (e.g. image to tensor).\n",
    "\n",
    "The `Dataset` abstracts away exactly when the data is loaded into memory: It can be on\n",
    "demand when each sample is accessed, all in advance or some combination using e.g. caching.\n",
    "This is implementation-specific.\n",
    "\n",
    "Lets create a demonstration `Dataset` that returns noise images. It should:\n",
    "- Return random tensors of size `CxWxH`.\n",
    "- Label each returned tensor with a class label, an integer between `0` and `num_classes-1`.\n",
    "- Initialize each returned tensor with a uniform distribution on `[0, 255]`.\n",
    "- Return a total of `num_samples` labeled images.\n",
    "\n",
    "**TODO** Implement the `RandomImageDataset` class in the `hw1/datasets.py` module.\n",
    "Use the code below to test your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test RandomImageDataset\n",
    "import cs236781.plot as plot\n",
    "import hw1.datasets as hw1datasets\n",
    "\n",
    "# Create the dataset\n",
    "num_samples = 1000\n",
    "num_classes = 10\n",
    "image_size = (3, 32, 32)\n",
    "ds = hw1datasets.RandomImageDataset(num_samples, num_classes, *image_size)\n",
    "\n",
    "# You can load individual items from the dataset by indexing\n",
    "img0, cls0 = ds[0]\n",
    "\n",
    "# Plot first N images from the dataset with a helper function\n",
    "fig, axes = plot.dataset_first_n(ds, 9, show_classes=True, nrows=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Built-in Datasets and Transforms\n",
    "<a id=part1_2></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've created a simple `Dataset` to see how they work, we'll load one of `pytorch`'s built-in datasets: CIFAR-10. This is a famous dataset consisting of 60,000 small `32x32` color images classified into 10 classes. You can read more about it [here](https://www.cs.toronto.edu/~kriz/cifar.html).\n",
    "\n",
    "The `torchvision` package has built-in `Dataset` classes that can download the data to a local folder,\n",
    "load it, transform it using arbitrary transform functions and iterate over the resulting samples.\n",
    "\n",
    "Run the following code block to download and create a CIFAR-10 `Dataset`. It won't be downloaded again if already present."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've loaded the entire CIFAR-10 dataset, we would want to work with a smaller subset\n",
    "from it to reduce runtime of the code in this notebook.\n",
    "A simple way to achieve this with `Datasets` is to wrap a `Dataset` in another `Dataset` that does this for us. This will make it easy to use our subset with `DataLoader`s as you will see later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following block to download CIFAR-10 and plot some random images from it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import torchvision\n",
    "import torchvision.transforms as tvtf\n",
    "\n",
    "cfar10_labels = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')\n",
    "data_root = os.path.expanduser('~/.pytorch-datasets')\n",
    "\n",
    "cifar10_train_ds = torchvision.datasets.CIFAR10(\n",
    "    root=data_root, download=True, train=True,\n",
    "    transform=tvtf.ToTensor()\n",
    ")\n",
    "\n",
    "print('Number of samples:', len(cifar10_train_ds))\n",
    "\n",
    "# Plot them with a helper function\n",
    "fig, axes = plot.dataset_first_n(cifar10_train_ds, 64,\n",
    "                                 show_classes=True, class_labels=cfar10_labels,\n",
    "                                 nrows=8, hspace=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's say we would like to only work with a small subset of the CIFAR-10 dataset.\n",
    "One way to easily achieve this is to wrap the built-in `Dataset` object with a custom dataset that exposes a different number of samples.\n",
    "\n",
    "**TODO** Complete the implementation of `SubsetDataset` in `hw1/datasets.py` and use the following code block to test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "subset_len = 5000\n",
    "subset_offset = 1234\n",
    "cifar10_train_subset_ds = hw1datasets.SubsetDataset(cifar10_train_ds, subset_len, subset_offset)\n",
    "\n",
    "dataset_x, dataset_y  = cifar10_train_ds[subset_offset + 10]\n",
    "subset_x, subset_y  = cifar10_train_subset_ds[10]\n",
    "\n",
    "# Tests\n",
    "test.assertEqual(len(cifar10_train_subset_ds), subset_len)\n",
    "test.assertTrue(torch.all(dataset_x == subset_x))\n",
    "test.assertEqual(dataset_y, subset_y)\n",
    "with test.assertRaises(IndexError, msg=\"Out of bounds index should raise IndexError\"):\n",
    "    tmp = cifar10_train_subset_ds[subset_len]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that when we initialized the `Dataset` instance for CIFAR-10, we provided a `transform` parameter.\n",
    "This is a way to specify an arbitrary transformation that should be run on each sample prior to returning it from the dataset.\n",
    "\n",
    "In the above, we used the `ToTensor()` transformation from `torchvision.transforms` to convert the\n",
    "images from a PIL (Python Imaging Library) image object which has a shape of `32x32x3` and values in range \\[0, 255\\] into a pytorch `Tensor` of shape `3x32x32` and values in range \\[0, 1\\].\n",
    "\n",
    "To demonstrate the use of transforms, we'll implement two custom transforms which invert the colors and flip the images along the height axis.\n",
    "\n",
    "**TODO** Complete the `InvertColors` and `FlipUpDown` classes in the `hw1/transforms.py` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import hw1.transforms as hw1transforms\n",
    "\n",
    "cifar10_inverted_ds = torchvision.datasets.CIFAR10(\n",
    "    root=data_root, download=True, train=True,\n",
    "    transform=tvtf.Compose([ # Compuse allows us to chain multiple transforms in a sequence\n",
    "        tvtf.ToTensor(), # Convert PIL image to pytorch Tensor (C,H,W) in range [0,1]\n",
    "        hw1transforms.InvertColors(),\n",
    "        hw1transforms.FlipUpDown(),\n",
    "    ])\n",
    ")\n",
    "\n",
    "fig, axes = plot.dataset_first_n(cifar10_inverted_ds, 64,\n",
    "                                 show_classes=True, class_labels=cfar10_labels,\n",
    "                                 nrows=8, hspace=0.5)\n",
    "\n",
    "test.assertTrue(torch.allclose(cifar10_train_ds[0][0], torch.flip(1.-cifar10_inverted_ds[0][0], [1])),\n",
    "               \"Wrong custom transform\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `DataLoader`s and `Sampler`s\n",
    "<a id=part1_3></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have seen that a `Dataset` is simply an iterable returning samples by index.\n",
    "Simple to implement, but not very powerful.\n",
    "The real benefit is when combining them with `DataLoader`.\n",
    "A `DataLoader` samples a batch of samples from the dataset according to logic defined by a `Sampler` object.\n",
    "The sampler decides how to partition the dataset into batches of `N` samples.\n",
    "The `DataLoader` additionally handles loading samples in parallel to speed up creation of a batch.\n",
    "\n",
    "A major motivation here is memory usage. When combining a `DataLoader` with a `Dataset` we can easily\n",
    "control memory constraints by simply setting the batch size. This is important since large\n",
    "datasets (e.g. ImageNet) may not fit in memory of most machines.\n",
    "Since a `Dataset` can lazily load samples from disk on access,\n",
    "and the `DataLoader` can sample random samples from it in parallel, we are provided with a simple\n",
    "yet high-performance mechanism to iterate over random batches from our dataset without needing to\n",
    "hold all of it in memory.\n",
    "\n",
    "Let's create a basic `DataLoader` for our CIFAR-10 dataset.\n",
    "Run the follwing code block multiple times and observe that different samples are shown each time in the first few batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a simple DataLoader that partitions the data into batches\n",
    "# of size N=8 in random order, using two background proceses\n",
    "cifar10_train_dl = torch.utils.data.DataLoader(\n",
    "    cifar10_train_ds, batch_size=8, shuffle=True, num_workers=2\n",
    ")\n",
    "\n",
    "# Iterate over batches sampled with our DataLoader\n",
    "num_batches_to_show = 5\n",
    "for idx, (images, classes) in enumerate(cifar10_train_dl):\n",
    "    # The DataLoader returns a tuple of:\n",
    "    # images: Tensor of size NxCxWxH\n",
    "    # classes: Tensor of size N\n",
    "    fig, axes = plot.tensors_as_images(images, figsize=(8, 1))\n",
    "    fig.suptitle(f'Batch #{idx+1}:', x=0, y=0.6)\n",
    "    if idx >= num_batches_to_show - 1:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training, Validation and Test Sets\n",
    "<a id=part1_4></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know about `DataLoaders` we can use them to do something useful: split a training dataset into **Training and Validation** sets.\n",
    "\n",
    "A common issue with machine learning models is abundance of hyperparameters that must be selected prior to training the model on data. More generally, multiple different models, or hypothesis classes could be fitted to the data.\n",
    "We would like to determine which model and/or hyperparameter selection of the same model can best fit the training data we have.\n",
    "\n",
    "How are such hyperparameters selected? How should their fitness be evaluated?\n",
    "While tempting, we can't use our test dataset to determine this. Doing so would be effectively equivalent to training with the test set, and may significantly bias our model towards overfitting, reducing it's generalization ability.\n",
    "\n",
    "A prevalent approach is therefore to split the training dataset into two parts:\n",
    "One for actual training, i.e. tuning model parameters e.g. weights in the case of neural nets,\n",
    "and another for validation, i.e. comparing one model or set of hyperparameters to another.\n",
    "After the best model is selected (by seeking the minimal validation error), it can be retrained with the entire training set.\n",
    "\n",
    "Crucially, test set performance is only evaluated once, at the end, after the best model has been selected and trained on the full training set. This provides us with an unbiased estimate of how our model will generalize to previously-unseen data.\n",
    "\n",
    "![img](https://cdn-images-1.medium.com/max/1600/1*Nv2NNALuokZEcV6hYEHdGA.png)\n",
    "\n",
    "**TODO** Implement the function `create_train_validation_loaders` in the `hw1/dataloaders.py` module.\n",
    "Use the following code block to check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Testing the train/validation split dataloaders\n",
    "import hw1.dataloaders as hw1dataloaders\n",
    "\n",
    "validation_ratio = 0.2\n",
    "dl_train, dl_valid = hw1dataloaders.create_train_validation_loaders(cifar10_train_ds, validation_ratio)\n",
    "\n",
    "train_idx = set(dl_train.sampler.indices)\n",
    "valid_idx = set(dl_valid.sampler.indices)\n",
    "train_size = len(train_idx)\n",
    "valid_size = len(valid_idx)\n",
    "print('Training set size: ', train_size)\n",
    "print('Validation set size: ', valid_size)\n",
    "\n",
    "# Tests\n",
    "test.assertEqual(train_size+valid_size, len(cifar10_train_ds), \"Incorrect total number of samples\")\n",
    "test.assertEqual(valid_size, validation_ratio * (train_size + valid_size), \"Incorrect ratio\")\n",
    "test.assertTrue(train_idx.isdisjoint(valid_idx), \"Train and validation sets are not disjoint\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}