master_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CE7454 Group 8 Report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Zhu Shien(G1801969B) & Mo Xiaoyun(G1702922E)\n",
    "### E-mail: SHIEN001@e.ntu.edu.sg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Project Description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our task is to do single objection detection on a image dataset captured by DJI drones. We need to find out both the bounding box which frames out the object, and the class of the object. Our goal is to maximize both the bounding box accuracy, the interaction over union (IoU), and classification accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img/goal.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The image dataset is provided by the 2018 DAC System Design Contest. In this contest, the teams competed for bounding box accuracy as well as computation performance, ignoring the classification accuracy. So it is different with our goal.\n",
    "\n",
    "The dataset consists of 96 folders, totoally 12 kinds of object. Each folder contains 400 to 1500 continuously captured images of the same object.\n",
    "\n",
    "We will try two types of models, namely Fast R-CNN and YOLOv2, to search the image to obtain the bounding box of the interested object, meanwhile classify the object and get a label. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook file list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. *master notebook*: project description, data exploration, data preprocessing and contribution of each student.\n",
    "2. *fast_rcnn_network.ipynb*: describe the solution based on Fast R-CNN.\n",
    "3. *yolov2_network.ipynb*: describe the solution based on Yolov2.\n",
    "4. *devide_data_notebook*: describe how we devide the dataset into training, testing, and inference sets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part, we explore the data to answer two questions of interest.\n",
    "dataset address: https://github.com/xyzxinyizhang/2018-DAC-System-Design-Contest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 1: How different are the images at the same folder? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each folder is expected to have some extent of redundancy, as they are all images capturing the same object continuously. Thus, we need to explore how different between these images.\n",
    "\n",
    "We use shannon entropy to represent the information of a image, and compare the entropy change between the *ith* image and the 1st image (images are sorted by the timestamps they were captured). The following gif shows an example of the folder \"boat5\", in which we pick the first image of every 10 images, totally 134 images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img/boat5_gif.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following figures show entropy change between the *ith* image and the first image of \"boat5\". The left figure shows the entropy calculated for the entire image, and the right figure shows the entropy calculated for the image inside the corresponding bounding box. We draw the conclusion at this part that although the images are captured continuously, they change quite a lot as time passes.There is redundancy between images, so we decided to pick the first image in every 10 continuous images, skipping 9 images in between."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img/data_exploration1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 2: How big are the objects in the images?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a first impression that the target object in the image seem to be small quite often. Thus, we investigate the bounding box(ground-truth) size (width * height) range of each folder(object). The following figure shows our result. We can see that there are quite a large percentage of object with a small size. For example, all the buildings, drones, groups are of small size.Among all the bounding boxes, we observed the smallest box of size 20px, and the largest box of size 73,000px. Small bounding box may make our task more difficult, because there are fewer object details, more background interference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img/size_range2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data pre-processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. We manually separated the folders into 80 classes. Although there are only 12 kinds of objects, but objects in different folders differ significantly. For example, the following 3 images are from 3 different folders. They are all \"person\", but as they differ quite a lot, we separate them into 3 different classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img/person.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. We separated the images into training set, small-scale testing set and testing set. We first extracted the 1st image for every 10 continuous images, then we randomly divided these images into 3 sets. \n",
    "\n",
    "  So, finally we have: 6000 images for training, 900 images for small-scale testing, and 2500 images for testing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Student Contributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Data acquisition: by Zhu Shien\n",
    "2. Data exploration: by Mo Xiaoyun\n",
    "3. Data preprocessing: by Zhu Shien and Mo Xiaoyun\n",
    "4. Solution 1 using Fast RCNN: by Mo Xiaoyun\n",
    "5. Solution 2 using Yolov2: by Zhu Shien\n",
    "6. Slides and report: by Zhu Shien and Mo Xiaoyun"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Github address\n",
    "https://github.com/yiweifengyan/CE7454_DeepLearningProject_Group8\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}