3_segmentation/00_introduction_to_image_segmentation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Image Segmentation Methods\n",
    "\n",
    "## What is _Image Segmentation_ ?\n",
    "\n",
    "segmenting an image means partitioning an image into different regions of homogenous pixels such that:\n",
    "\n",
    "- union of all regions = the image\n",
    "- each region is homogeneous. homogeneity can be based on criteria based on the problem one is trying to solve.\n",
    "- any pair of adjacent regions is not homogeneous\n",
    "- regions should have low internal variance\n",
    "- regions should be made of spatially contiguous pixels\n",
    "\n",
    "These points are derived from **[Pavlidis's definitions of segmentation](https://www2.cs.duke.edu/courses/spring06/cps296.1/handouts/Horowitz%20Pavlidis%201976.pdf)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at couple of examples of segmented images to understand what how do they look like and then discuss why are they useful.\n",
    "\n",
    "<br/>\n",
    "\n",
    "<figure>\n",
    "    <img src=\"https://devblogs.nvidia.com/wp-content/uploads/2016/11/figure9.png\" width = 700px/>\n",
    "    <figcaption style = \"text-align:center\">Example of a segmented image. Ref: \n",
    "        <a href=\"https://devblogs.nvidia.com/image-segmentation-using-digits-5/\">NVIDIA Dev Blog</a>\n",
    "    </figcaption>\n",
    "</figure>\n",
    "\n",
    "<br/>\n",
    "\n",
    "<figure>\n",
    "    <img src=\"https://1.bp.blogspot.com/-96qG-uqTaxQ/Xc7dxCaYZhI/AAAAAAAAA5U/LX-6HnwfTXghgUV1MVs6z3J3nkmil0XcACLcBGAsYHQ/s1600/bodypix2-twitter.gif\" width = 700px/>\n",
    "    <figcaption style = \"text-align:center\">Multi-person support and improved accuracy (based on ResNet50) in BodyPix 2.0. Ref: \n",
    "        <a href=\"https://blog.tensorflow.org/2019/11/updated-bodypix-2.html\">Tensorflow Blog</a>\n",
    "    </figcaption>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How does segmentation help us ?\n",
    "\n",
    "Segmenting an image gives us a class mapping for each pixel in that image. This enables us to have precise understanding and control of the scene. When self driving car sees a segmented camera feed it knows precisely which pixels belong to road and which pixel belong to human. \n",
    "\n",
    "Thus making the decision logic easier to implement - _Stay on the road pixels, stay away from human pixels and maintain a safe distance from other vehicle pixels_.\n",
    "\n",
    "Another usecase where segmentation proves to be extrememly useful is figuring out tumor areas in medical images. The image shown below shows tumor identification using image segmentation.\n",
    "\n",
    "<br/>\n",
    "\n",
    "<figure>\n",
    "    <img src=\"https://d3i71xaburhd42.cloudfront.net/005c6e8d0c34c8761897d3a0f5cf091c7c39f20a/3-Figure1-1.png\" width = 700px/>\n",
    "    <figcaption style = \"text-align:center\">Brain Tumor Segmentation, a very useful implementation of segmentation. Ref: \n",
    "        <a href=\"https://www.semanticscholar.org/paper/Brain-Tumor-Segmentation-in-Multi-modality-MRIs-and-Zhan-Zhan/005c6e8d0c34c8761897d3a0f5cf091c7c39f20a\">Brain Tumor Segmentation</a>\n",
    "    </figcaption>\n",
    "</figure>\n",
    "\n",
    "<br/>\n",
    "\n",
    "The bottom row showing the predictions. SOTA algorithms are capable of efficiently handling these usecases. Biomedical applications of segmentation algorithms are immensely useful to augment doctors in their decision making. These techniques also make handling large scale biomedical data possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Types of Segmentation Techniques\n",
    "\n",
    "**Interesting Read**: [Image Segmentation](https://www.cs.auckland.ac.nz/courses/compsci773s1c/lectures/ImageProcessing-html/topic3.htm#:~:text=Two%20basic%20approaches%20to%20contextual,signal%20changes%20across%20each%20boundary.)\n",
    "\n",
    "The following is a broad categorization of segmentation techniques we will be looking at in the following notebooks:\n",
    "\n",
    "- **Traditional Methods**\n",
    "    - _Non-Contextual_\n",
    "      - Image Thresholding\n",
    "          - Simple Thresholding\n",
    "          - Adaptive Thresholding\n",
    "          - Otsu's Threshold\n",
    "    - _Contextual_\n",
    "      - Region Growing\n",
    "- **Evaluating Segmentation Results**\n",
    "- **Deep Learning Method Concepts**\n",
    "    - _FCN_\n",
    "    - _UNet_\n",
    "- **Implementing a Deep Learning Model**\n",
    "\n",
    "Traditional methods help us understand how we can implement segmentation on images using ab-initio methods and then we see how we can evaluate these methods and get a sense of how well our algorithms are performing. \n",
    "\n",
    "After understanding how to evaluate these methods we investigate methods which are optimized to maximize or minimize the evaluation metrics. These are the deep learning methods and we end our discussion by implementing a deep learning model to solve a real life segmentation problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contextual & Non-Contextual Techniques\n",
    "\n",
    "In this discussion contextual and non-contextual refer to the spatial context taken into consideration while we are calculating a criterion to partition the image into segments."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}