Importing_data/Importing_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1f5e2b5a",
   "metadata": {},
   "source": [
    "<img src=\"../figures/HeaDS_logo_large_withTitle.png\" width=\"300\">\n",
    "\n",
    "<img src=\"../figures/tsunami_logo.PNG\" width=\"600\">\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Center-for-Health-Data-Science/PythonTsunami/blob/intro/Importing_data/Importing_data.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa888e38",
   "metadata": {},
   "source": [
    "# Importing Data into Python\n",
    "\n",
    "*Prepared by [Alberto Santos](https://heads.ku.d)*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f8ff87f",
   "metadata": {},
   "source": [
    "## Objectives\n",
    "\n",
    "*   Learn how to import data from different sources into Python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c048d038",
   "metadata": {},
   "source": [
    "#### *Note*: In colab, access to files is a little bit different than when you access directly in your computer\n",
    "\n",
    "We will need to 1st download the files to Colab using:\n",
    "\n",
    "##### create a data folder\n",
    "`!mkdir -p data`\n",
    "\n",
    "##### download file\n",
    "`!wget https://raw.githubusercontent.com/Center-for-Health-Data-Science/PythonTsunami/intro/data/file_name -P data`\n",
    "`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c80423f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Download data to Colab\n",
    "!mkdir -p data\n",
    "!wget https://raw.githubusercontent.com/Center-for-Health-Data-Science/PythonTsunami/intro/data/sample.txt -P data\n",
    "!wget https://raw.githubusercontent.com/Center-for-Health-Data-Science/PythonTsunami/intro/data/sample.csv -P data\n",
    "!wget https://raw.githubusercontent.com/Center-for-Health-Data-Science/PythonTsunami/intro/data/iris.tsv -P data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee5a88cc",
   "metadata": {},
   "source": [
    "## Importing Text Files\n",
    "\n",
    "A text file *(.txt)* is the most common file we will deal with. Text files are structured as a sequence of lines, where each line includes a sequence of characters.\n",
    "\n",
    "To import the contents of a text file, we will first need to define where the file is located: `pathfile`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2371b319",
   "metadata": {},
   "source": [
    "### Open file\n",
    "We first need to open the file. To do so, we use the `open()` built-in function. \n",
    "\n",
    "- `open()` has a required argument that is the path to the file and an argument to indicate the mode (i.e. 'r': open for reading; 'w': open for writing, 'a' for appending).  The following table lists the valid values of access mode parameters:\n",
    "\n",
    "| mode | format | read/write | create new file? | comments |\n",
    "| :--: | :--:   | :--:       | :--:             | :--:     |\n",
    "| `'r'`    | text   | read | no | Default mode |\n",
    "| `'rb'`   | binary | read | no | Raises I/O error if the file does not exist |\n",
    "| `'r+'`   | text   | read/write  | no | Raises I/O error if the file does not exist |\n",
    "| `'rb+'`   | binary | read/write | no | Raises I/O error if the file does not exist|\n",
    "| `'w'`    | text   | write | yes | Truncates and overwrites data if file exists |\n",
    "| `'wb'`    | binary   | write | yes | Truncates and overwrites data if file exists |\n",
    "| `'w+'`   | text   | read/write | yes | Truncates and overwrites data if file exists |\n",
    "| `'wb+'`    | binary   | write | yes | Truncates and overwrites data if file exists |\n",
    "| `'a'`    | text   | write | yes | Data is inserted at the end of the file |\n",
    "| `'ab'`    | binary   | write | no | Data is inserted at the end of the file |\n",
    "| `'a+'`   | text   | read/write | yes | Data is inserted at the end of the file |\n",
    "| `'ab+'`    | binary   | write | no | Data is inserted at the end of the file |\n",
    "\n",
    "- `open()` returns us a file object that we can then use to read, write or append content to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23322154",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data/sample.txt', 'r') as reader:\n",
    "    print(reader.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81007f0d",
   "metadata": {},
   "source": [
    "### Path to file\n",
    "\n",
    "We can also define a variable with the path where the file is located by using the module `pathlib` and the object `Path`. This can help avoid problems with Operating Systems using a different path structures (i.e Windows `/`, Unix `\\`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a5499ad5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "filepath = Path('data/sample.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fa9f572b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Country/Region\n",
      "Mainland China\n",
      "Japan\n",
      "Singapore\n",
      "Hong Kong\n",
      "Japan\n",
      "Thailand\n",
      "South Korea\n",
      "Malaysia\n",
      "Taiwan\n",
      "Germany\n",
      "Vietnam\n",
      "France\n",
      "Macau\n",
      "UK\n",
      "United Arab Emirates\n",
      "US\n",
      "Australia\n"
     ]
    }
   ],
   "source": [
    "with open(filepath, 'r') as reader:\n",
    "    print(reader.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb2c44c0",
   "metadata": {},
   "source": [
    "### Reading the file\n",
    "\n",
    "There are three methods to read content (i.e. `read()`, `readline()`, and `readlines()`) that can be called on this file object.\n",
    "\n",
    "- `read()` reads the content of the file -- Accepts as parameter the number of characters to be read\n",
    "- `readline()` reads one line -- Accepts as parameter the number of characters to be read of the line\n",
    "- `readlines()` reads all lines and stores them in a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "49817b06",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Country/Region\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open(filepath, 'r') as reader:\n",
    "    print(reader.readline())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7ede9574",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Country/Region\\n', 'Mainland China\\n', 'Japan\\n', 'Singapore\\n', 'Hong Kong\\n', 'Japan\\n', 'Thailand\\n', 'South Korea\\n', 'Malaysia\\n', 'Taiwan\\n', 'Germany\\n', 'Vietnam\\n', 'France\\n', 'Macau\\n', 'UK\\n', 'United Arab Emirates\\n', 'US\\n', 'Australia']\n"
     ]
    }
   ],
   "source": [
    "with open(filepath, 'r') as reader:\n",
    "    print(reader.readlines())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff8041a0",
   "metadata": {},
   "source": [
    "We can for instance read line by line using a loop (see [Loops](https://github.com/Center-for-Health-Data-Science/PythonTsunami/blob/intro/Loops/Loops.ipynb))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "dd9c487d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New line: Country/Region\n",
      "\n",
      "New line: Mainland China\n",
      "\n",
      "New line: Japan\n",
      "\n",
      "New line: Singapore\n",
      "\n",
      "New line: Hong Kong\n",
      "\n",
      "New line: Japan\n",
      "\n",
      "New line: Thailand\n",
      "\n",
      "New line: South Korea\n",
      "\n",
      "New line: Malaysia\n",
      "\n",
      "New line: Taiwan\n",
      "\n",
      "New line: Germany\n",
      "\n",
      "New line: Vietnam\n",
      "\n",
      "New line: France\n",
      "\n",
      "New line: Macau\n",
      "\n",
      "New line: UK\n",
      "\n",
      "New line: United Arab Emirates\n",
      "\n",
      "New line: US\n",
      "\n",
      "New line: Australia\n"
     ]
    }
   ],
   "source": [
    "with open(filepath, 'r') as reader:\n",
    "    for line in reader:\n",
    "        print(\"New line:\", line)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7574e17",
   "metadata": {},
   "source": [
    "## Importing CSV Files\n",
    "\n",
    "Comma Separated Value files (csv) are very common in biology and are used to store tabular data. In this type of files, every field on each line is separated by a delimiter, indicating where one field ends and the next field starts. \n",
    "\n",
    "These files are often either comma-separated (.csv)or tab-separated (.tsv or .txt).\n",
    "\n",
    "In principle, we can simply read them in the same way as text files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a83b41d4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Province/State,Country/Region,Last Update,Confirmed,Deaths,Recovered,City,Date_last_updated_AEDT,lat,lon\n",
      "Hubei,China,02/15/2020 23:00,56249,1596,5623,,2020-02-16 15:00:00,31.1517252,112.8783222\n",
      "Guangdong,China,02/15/2020 23:00,1316,2,442,,2020-02-16 15:00:00,23.1357694,113.1982688\n",
      "Henan,China,02/15/2020 23:00,1231,13,415,,2020-02-16 15:00:00,34.0000001,113.9999999\n"
     ]
    }
   ],
   "source": [
    "filepath = Path('data/sample.csv')\n",
    "with open(filepath, 'r') as reader:\n",
    "    print(reader.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2868d5e7",
   "metadata": {},
   "source": [
    "However, by doing so we lose the tabular structure in the data and we can not access it in any smart way, for instance printing the 3rd country in our table.\n",
    "\n",
    "\n",
    "One option is to include a bit of logic to break down each line using the string function `split` and specifying the delimiter. This will at least help us get list that we can then access by index (see [Lists.ipynb](https://github.com/Center-for-Health-Data-Science/PythonTsunami/blob/intro/Data_structures/Lists.ipynb)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "766bd920",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['Province/State', 'Country/Region', 'Last Update', 'Confirmed', 'Deaths', 'Recovered', 'City', 'Date_last_updated_AEDT', 'lat', 'lon\\n'], ['Hubei', 'China', '02/15/2020 23:00', '56249', '1596', '5623', '', '2020-02-16 15:00:00', '31.1517252', '112.8783222\\n'], ['Guangdong', 'China', '02/15/2020 23:00', '1316', '2', '442', '', '2020-02-16 15:00:00', '23.1357694', '113.1982688\\n'], ['Henan', 'China', '02/15/2020 23:00', '1231', '13', '415', '', '2020-02-16 15:00:00', '34.0000001', '113.9999999']]\n"
     ]
    }
   ],
   "source": [
    "my_lines = []\n",
    "with open(filepath, 'r') as reader:\n",
    "    for line in reader:\n",
    "        line_list = line.split(',')\n",
    "        my_lines.append(line_list)\n",
    "        \n",
    "print(my_lines)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "5d438b59",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'China'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 3rd country in our table\n",
    "my_lines[3][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e9db207",
   "metadata": {},
   "source": [
    "### csv library\n",
    "\n",
    "Python has a specific library for reading this type of files `csv`. The functionality in this library maintains the tabular structure (to some extent) and makes life easier to access the data afterwards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "f81976ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Import the library to be able to use it\n",
    "import csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "25541282",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Province/State', 'Country/Region', 'Last Update', 'Confirmed', 'Deaths', 'Recovered', 'City', 'Date_last_updated_AEDT', 'lat', 'lon']\n",
      "['Hubei', 'China', '02/15/2020 23:00', '56249', '1596', '5623', '', '2020-02-16 15:00:00', '31.1517252', '112.8783222']\n",
      "['Guangdong', 'China', '02/15/2020 23:00', '1316', '2', '442', '', '2020-02-16 15:00:00', '23.1357694', '113.1982688']\n",
      "['Henan', 'China', '02/15/2020 23:00', '1231', '13', '415', '', '2020-02-16 15:00:00', '34.0000001', '113.9999999']\n"
     ]
    }
   ],
   "source": [
    "with open(filepath,'r') as myFile:  \n",
    "    lines=csv.reader(myFile, delimiter=',')  \n",
    "    for line in lines:  \n",
    "        print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7fe7d99",
   "metadata": {},
   "source": [
    "## Remove some lines\n",
    "\n",
    "The easiest way to remove lines from a file is actually to keep only the ones you want.\n",
    "We will need to read the file, store the lines we want in a variable, and then open the file for writing and save the lines we wanted."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73063112",
   "metadata": {},
   "source": [
    "**1) We append 2 lines in the file data/sample.txt**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2745e1e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "filepath = 'data/sample.txt'\n",
    "\n",
    "with open(filepath, 'a') as f:\n",
    "    f.write('\\nThis is not a valid line\\n')\n",
    "    f.write('This is one either\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d4b278c",
   "metadata": {},
   "source": [
    "**2) We read the file and keep the first 18 lines only**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57bcb3f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_valid_lines = 18\n",
    "my_valid_lines = []\n",
    "i = 0\n",
    "with open(filepath, 'r') as f:\n",
    "    for line in f:\n",
    "        if i < num_valid_lines:\n",
    "            my_lines.append(line)\n",
    "        i += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08ea38b3",
   "metadata": {},
   "source": [
    "**3) We write back into the file only the valid lines**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "353b2a6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(filepath, 'w') as f:\n",
    "    f.write(\"\".join(my_lines))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa29cfeb",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "1) Read file **iris.tsv** using both approaches (text and csv)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9698a627",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3153da67",
   "metadata": {},
   "source": [
    "2) Open the sample.txt and write a new line with `Denmark/Zealand`.\n",
    "\n",
    "3) The, read again the file to see the new content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2d13515",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "exact-hamburg",
   "metadata": {},
   "source": [
    "## ----------------------------------------------------------------------------------------------------------------------------------\n",
    "<h2><center>Extra</center></h2>\n",
    "\n",
    "## ----------------------------------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "936b96a4",
   "metadata": {},
   "source": [
    "## Web Scraping\n",
    "\n",
    "Web scraping is a technique to automatically access and extract large amounts of information from a website.\n",
    "\n",
    "### Important notes about web scraping:\n",
    "\n",
    "- Read through the website’s Terms and Conditions whether or not you can use the data posted in the website.\n",
    "- Make sure you are not downloading data at too rapid rate, because this may break the website or you may potentially be blocked.\n",
    "\n",
    "In this case we will be scraping UniProt website to extract Gene Ontology terms associated with a specific protein."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "7eac6417",
   "metadata": {},
   "outputs": [],
   "source": [
    "#specify the url of the website you are interested in, in this case UniProt\n",
    "url = \"http://www.uniprot.org/uniprot/\"\n",
    "# Protein GTPase KRas\n",
    "protein = \"P01116\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "231a932b",
   "metadata": {},
   "source": [
    "Import the necessary libraries:\n",
    "\n",
    "- [urllib](https://docs.python.org/3/library/urllib.html)\n",
    "- [bs4](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "5cb52467",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "ea6f4976",
   "metadata": {},
   "outputs": [],
   "source": [
    "page = requests.get(url+protein)\n",
    "#Parse the html, store it in Beautiful Soup format\n",
    "bsf = BeautifulSoup(page.text, \"html.parser\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "a23f47f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "biological_process_go = []\n",
    "ext_data = bsf.find('ul', class_='noNumbering biological_process')\n",
    "if ext_data is not None:\n",
    "    for data in ext_data.find_all('li'):\n",
    "        cells = data.find(\"a\")\n",
    "        if cells is not None:\n",
    "            cells = cells.find(text=True).strip()\n",
    "            biological_process_go.append(cells)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "0a02494c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['actin cytoskeleton organization',\n",
       " 'cytokine-mediated signaling pathway',\n",
       " 'endocrine signaling',\n",
       " 'epithelial tube branching involved in lung morphogenesis',\n",
       " 'female pregnancy',\n",
       " 'forebrain astrocyte development',\n",
       " 'homeostasis of number of cells within a tissue',\n",
       " 'liver development',\n",
       " 'MAPK cascade',\n",
       " 'negative regulation of cell differentiation',\n",
       " 'negative regulation of neuron apoptotic process',\n",
       " 'positive regulation of cell population proliferation',\n",
       " 'Zimmermann G.',\n",
       " 'positive regulation of cellular senescence',\n",
       " 'positive regulation of gene expression',\n",
       " 'Oh Y.T.',\n",
       " 'positive regulation of MAP kinase activity',\n",
       " 'positive regulation of NF-kappaB transcription factor activity',\n",
       " 'positive regulation of nitric-oxide synthase activity',\n",
       " 'positive regulation of protein phosphorylation',\n",
       " 'Oh Y.T.',\n",
       " 'positive regulation of Rac protein signal transduction',\n",
       " 'Ras protein signal transduction',\n",
       " 'Gaudet P.',\n",
       " 'regulation of long-term neuronal synaptic plasticity',\n",
       " 'regulation of protein stability',\n",
       " 'Serra R.W.',\n",
       " 'regulation of synaptic transmission, GABAergic',\n",
       " 'response to glucocorticoid',\n",
       " 'response to isolation stress',\n",
       " 'response to mineralocorticoid',\n",
       " 'stimulatory C-type lectin receptor signaling pathway',\n",
       " 'striated muscle cell differentiation',\n",
       " 'visual learning']"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "biological_process_go"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cad37c5a",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "1) Can you do the same for GO Molecular Functions?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35aa8b02",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "9a8a45d9",
   "metadata": {},
   "source": [
    "2) Extract the pathway information for protein Erythropoietin (Human) (Look at the Html structure)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3721792",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "64e37537",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "- [An Overview Of Importing Data In Python](https://towardsdatascience.com/an-overview-of-importing-data-in-python-ac6aa46e0889)\n",
    "- [How to Web Scrape with Python in 4 Minutes](https://towardsdatascience.com/how-to-web-scrape-with-python-in-4-minutes-bc49186a8460)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}