4_pandas_02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Introduction to Pandas: Working with Different File types\n",
    "\n",
    "- prepared by [Rita Colaço](https://www.cpr.ku.dk/staff/?id=621366&vis=medarbejder), updated by [Henry Webel](https://twitter.com/Henrywebel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The Pandas library offers a wide range of possibilities for creating, writing and reading files. There are two types of files that can be handled in Python, normal text files and binary files.\n",
    "\n",
    "\n",
    "In this notebook we will learn more about working with these different formats: CSV, Excel, JSON, HTML, SQL, Pickle, Matlab .mat, and HDF5 files.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## File I/O Operations\n",
    "\n",
    "In Python, a physical file must be mapped to a built-in file object with the help of built-in function `open`.\n",
    "\n",
    "In this function, the first parameter is the **name of a file including its path**. The **access mode** parameter is an optional parameter, which refers to how the file will be used once its opened. The following table lists the valid values of access mode parameters:\n",
    "\n",
    "| mode | format | read/write | create new file? | comments |\n",
    "| :--: | :--:   | :--:       | :--:             | :--:     |\n",
    "| `'r'`    | text   | read | no | Default mode |\n",
    "| `'rb'`   | binary | read | no | Raises I/O error if the file does not exist |\n",
    "| `'r+'`   | text   | read/write  | no | Raises I/O error if the file does not exist |\n",
    "| `'rb+'`   | binary | read/write | no | Raises I/O error if the file does not exist|\n",
    "| `'w'`    | text   | write | yes | Truncates and overwrites data if file exists |\n",
    "| `'wb'`    | binary   | write | yes | Truncates and overwrites data if file exists |\n",
    "| `'w+'`   | text   | read/write | yes | Truncates and overwrites data if file exists |\n",
    "| `'wb+'`    | binary   | write | yes | Truncates and overwrites data if file exists |\n",
    "| `'a'`    | text   | write | yes | Data is inserted at the end of the file |\n",
    "| `'ab'`    | binary   | write | no | Data is inserted at the end of the file |\n",
    "| `'a+'`   | text   | read/write | yes | Data is inserted at the end of the file |\n",
    "| `'ab+'`    | binary   | write | no | Data is inserted at the end of the file |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Text Files\n",
    "\n",
    "In text files, each line of text is terminated with a special character called EOL (End Of Line), which is the new line character `'\\n'` by default. In this section we will learn how to open, close, read and write data in a text file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Writing to a File\n",
    "\n",
    "In order to write a physical file, you can run:\n",
    "\n",
    "```python\n",
    "with open('file.txt', 'w') as f:\n",
    "    f.write('Hello, World!')\n",
    "```\n",
    "\n",
    "This code opens the file *file.txt* in the write mode, or creates it if it doesn't exist, and assigns it to a variable `f`. To add data to the file, we use `write()`, and in the end, `close()` closes the file object.\n",
    "\n",
    "To avoid the manual closing of files, it is recommended to use a `ContextManager` notation, using the `with` keyword, which will automatically close the file\n",
    "\n",
    "```python\n",
    "with open('file.txt', 'w') as f:\n",
    "    f.write('Hello, World!')\n",
    "# after the intended block the file is closed\n",
    "\n",
    "```\n",
    "\n",
    "There is another method `writelines()` that saves the contents of a list object in a file. In this case, the newline character must be provided as part of the string:\n",
    "\n",
    "```python\n",
    "lines = ['Hello world.\\n', 'Welcome to the Python Tsunani.\\n']\n",
    "with open('file.txt', 'w') as f:\n",
    "    f.writelines(lines)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Reading from a File\n",
    "\n",
    "There are three different methods to read data from a file:\n",
    "\n",
    "1. `readline()`: reads the characters from the current position until the next newline character.\n",
    "\n",
    "2. `read()`: reads the specified number of characters.\n",
    "\n",
    "3. `readlines()`: reads all lines until the end of the file and returns a list.\n",
    "\n",
    "For example:\n",
    "\n",
    "```python\n",
    "with open('file.txt', 'r') as f:\n",
    "    line = f.readline()\n",
    "```\n",
    "\n",
    "This piece of code will read the first line of the *file.txt* file and assign it to the variable *line*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "If you can to iterate over all lines of the file and read it easily. You can use:\n",
    "\n",
    "```python\n",
    "with open('file.txt', 'r') as f:\n",
    "    for line in f:\n",
    "        print(line)\n",
    "    # Other relevant code\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Append Text to a File\n",
    "\n",
    "One of the particularities of the write mode `\"w\"` is that it will always treat the specified file as a new file. This means, if the file already exists and contains data, opening it with the `\"w\"` mode will make you lose its earlier contents.\n",
    "\n",
    "In order to add more data to and existing file, the better approach is to use the `\"a\"` or `\"a+\"` mode, like in the example:\n",
    "\n",
    "```python\n",
    "with open('file.txt', 'a+') as :\n",
    "    f.write('Hello! Learn Python with us.')\n",
    "    line = f.readline()\n",
    "```\n",
    "\n",
    "In this example, we use the mode `\"a+\"` in order to perform simultaneous read/append operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "lines = ['Hello world.\\n', 'Welcome to the Python Tsunani.\\n']\n",
    "with open('file.txt', 'w')as f:\n",
    "    f.write('Hello, World!\\n\\n')\n",
    "    f.writelines(lines)\n",
    "\n",
    "with open('file.txt', 'a+')as f:\n",
    "    f.write('Hello! Learn Python with us.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[file.txt](file.txt) is now saved locally to your disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.write('test') # file is closed after the with block"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### CSV (Comma-Separated Values) Files\n",
    "\n",
    "A CSV file is a plaintext file with a .csv extension that holds tabular data. This is one of the most popular file formats for storing large amounts of data. \n",
    "\n",
    "Each line of the file represents one record, and the fields are, by default, separated by commas, but you could change the separator to a semicolon, tab, space, or some other character. If the fields are labelled, the first line pf the file (referred to as \"header\") will contain the field names.\n",
    "\n",
    "Example of CSV file:\n",
    "```\n",
    "month,height,weight\n",
    "Jan,1.2,76\n",
    "Feb,1.21,77\n",
    "March,1.21,76\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To read CSV files, python comes with a csv reader that works quite well.\n",
    "\n",
    "```python\n",
    "import csv\n",
    "\n",
    "with open('file.csv', 'r') as f:\n",
    "    reader = csv.reader(f)\n",
    "    header = next(reader)\n",
    "    data = list(reader)\n",
    "```\n",
    "\n",
    "Once you have read the data, it can go to a DataFrame, for example:\n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame(data=data, columns=header)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You can also use the Pandas csv read function `pandas.read_csv()`, which can get the data into a DataFrame. This is what we usually use.\n",
    "\n",
    "The major advantage of this function is that it has a lot of options and does good file format and data format inference.\n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('file.csv')\n",
    "```\n",
    "\n",
    "The input `'file.csv'` can be any valid path, including URLs.\n",
    "\n",
    "You can read about all the options [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html).\n",
    "\n",
    "`read_csv` is accompained by the `to_csv` function, to write data from a `DataFrame` to disk in `csv`-format:\n",
    "\n",
    "```python\n",
    "df.to_csv('file.csv')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# using the covid 19 data from before\n",
    "import pandas as pd\n",
    "df = pd.read_csv(\"https://opendata.ecdc.europa.eu/covid19/casedistribution/csv/data.csv\", index_col='dateRep')\n",
    "sample = df.sample(10)\n",
    "sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's us create some sample data, containing 10 entires."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample.to_csv('testdata.txt')\n",
    "sample.to_csv('testdata.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And look at the create text-file: [testdata.csv](testdata.csv) (displayed nicely already, try the [testdata.txt](testdata.txt) file!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### JSON (Javascript Object Notation) Files\n",
    "\n",
    "The next file type we will look at is JSON. This is a popular format for transferring data over the web via APIs, and is also a plaintext file format.\n",
    "\n",
    "JSON is very similar to the text representation of a Python dictionary and lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "data = \"\"\"\n",
    "{\n",
    "\"day\": \"Saturday\",\n",
    "\"week\": 3,\n",
    "\"isSunny\": true,\n",
    "\"goals\": [\"eat breakfast\", \"write a book\", \"eat lunch\"]\n",
    "}\n",
    "\"\"\"\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The main downside with **hand-writing** JSON is that it is very picky about getting everything right. Even though it's very readable, it should not be considered human writable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Python and Pandas work well with JSON files, as Python's json library offers buit-in support for them.\n",
    "Tabular data can be stored in JSON in a variety of ways, called \"orientations\".\n",
    "\n",
    "- `'split'` : dict like {'index' -> [index], 'columns' -> [columns], 'data' -> [values]}\n",
    "- `'records'` : list like [{column -> value}, ... , {column -> value}]\n",
    "- `'index'` : dict like {index -> {column -> value}}\n",
    "- `'columns'` : dict like {column -> {index -> value}}\n",
    "- `'values'` : just the values array\n",
    "- `'table'` : dict like {'schema': {schema}, 'data': {data}}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You can save the data from your DataFrame to a JSON file with `to_json()` function:\n",
    "\n",
    "```python\n",
    "df.to_json('data.json', orient='index')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You can also load the data from a JSON file with `read_json()`.\n",
    "\n",
    "```python\n",
    "df = pd.read_json('data.json', orient='index')\n",
    "```\n",
    "\n",
    "In this case, the *orient* parameter is very important because it specifies how Pandas understands the structure of the file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Alternatively, you can use the **json module** to load (read) and dump (write) JSON files.\n",
    "This module has 4 main functions:\n",
    "    \n",
    "| function | read/write | file/string |\n",
    "| :---:    | :----:     |  :-----:    |\n",
    "| load()   |  read      | file        |\n",
    "| dump()   |  write     | file        |\n",
    "| loads()  |  read      | string      |\n",
    "| dumps()  |  write     | string      |\n",
    "\n",
    "\n",
    "To read the data example we created above, which means converting from JSON to Python:\n",
    "\n",
    "```python\n",
    "import json\n",
    "\n",
    "json.loads(data)\n",
    "```\n",
    "\n",
    "And to convert a Python object to JSON:\n",
    "\n",
    "```python\n",
    "import json\n",
    "\n",
    "json.dumps(data)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "sample_json_string = sample.to_json()\n",
    "sample_json_string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "sample_json_dict = json.loads(sample_json_string)\n",
    "sample_json_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "pprint(sample_json_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample.to_json('sample_data_json.txt')\n",
    "sample.to_json('sample_data.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And have a look [sample_data.json](sample_data.json) (or at the `txt` file - [sample_data_json.txt](sample_data_json.txt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### HTML Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "An HTML is a plaintext file that uses hypertext markup language to help browers render web pages. These files carry the extension *.html* and *htm*, and in order to work with them, you will need to install an HTML library like **lxml** or **html5lib**.\n",
    "\n",
    "Once you have these libraries, you can \n",
    "\n",
    "\n",
    "\n",
    "You can save your DataFrame as an HTML file with `to_html()`:\n",
    "\n",
    "```python\n",
    "df = pd.DataFrame(data=data).T\n",
    "df.to_html('data.html')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "sample.to_html('sample_data_html.txt')\n",
    "sample.to_html('sample_data.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And have a look [sample_data.html](sample_data.html) (or at the `txt` file - [sample_data_html.txt](sample_data_html.txt))  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Binary Files\n",
    "\n",
    "In binary files, there is no terminator for a line and the data is stored after converting it into machine understandable binary language. Unlike text files, binary files are not human readable, this means, if you try to open them in any text editor, it will either not open, or show the data in an unrecognizable format.\n",
    "\n",
    "Without documentation, proper software, and version management, these files can be difficult to work with.\n",
    "\n",
    "Below, we show you a very simple example of how you could read and write to a binary file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Reading and Writing to a Binary File\n",
    "\n",
    "Opening a file in binary format is very similar to opening a text file, just add `\"b\"` to the mode parameter. For example, `\"rb\"` mode opens the file in binary format for reading only.\n",
    "\n",
    "The following example stores a list of numbers in a binary file:\n",
    "\n",
    "```python\n",
    "with open('binfile.bin', 'wb') as f:\n",
    "    num = [5, 10, 15, 20, 25]\n",
    "    arr = bytearray(num)\n",
    "    f.write(arr)\n",
    "```\n",
    "\n",
    "The function `bytearray` converts the list into a byte representation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To read a binary file like the one shown above, the output of the `read()` function is turned back into a list:\n",
    "\n",
    "```python\n",
    "with open('binfile.bin', 'wb') as f:\n",
    "    num = list(f.read())\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "There are, of course, advantages to using binary file:\n",
    "\n",
    "- smaller file sizes\n",
    "- supports more features (compression, multiple dataset storage, self-description, etc)\n",
    "- quicker read/write times\n",
    "- entire ecosystems of supported software\n",
    "\n",
    "Due to this, the developers of Pandas have created a whole set of IO tools that allow not only to read/write text files, but also binary and even SQL file types.\n",
    "\n",
    "| format type | data | read | write |\n",
    "|    :---:    |:----:|:-----: | :---:  |\n",
    "| binary | MS Excel  | read_excel | to_excel |\n",
    "| binary | Python Pickle Format | read_pickle | to_pickle |\n",
    "| binary | HDF5 Format | read_hdf | to_hdf |\n",
    "| binary | SPSS | read_spss | |\n",
    "\n",
    "This table contains only a few examples but you can see all of the available IO tools [here](https://pandas.pydata.org/pandas-docs/dev/user_guide/io.html).\n",
    "\n",
    "In the next sections, we show you a few of these standards  for storing tabular data in binary formats."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Excel Files\n",
    "\n",
    "Microsoft Excel is probably the most widely-used spreadsheet software, and even though it is a binary file format, you can read and write Excel files in Pandas, similar to CSV files.\n",
    "\n",
    "An additional requirement however, depending on the Excel version you will work with, you will need to install other Python packages first.\n",
    "\n",
    "- **xlrd** to read Excel files *.xls* (Excel 2003)\n",
    "\n",
    "- **openpyxl** to read/write *.xlsx* files (Excel 2007+)\n",
    "\n",
    "- **pyxlsb** to read binary Excel *.xlsb*\n",
    "\n",
    "\n",
    "You can install them using **pip** with a single command:\n",
    "```python\n",
    "pip install xlrd openpyxl pyxlsb\n",
    "```\n",
    "\n",
    "Or using Conda:\n",
    "```python\n",
    "conda install xlrd openpyxl pyxlsb\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Once you have installed the neccessary packages, you can read an Excel file with read_excel():\n",
    "\n",
    "```python\n",
    "df = pd.read_excel('data.xlsx')\n",
    "```\n",
    "\n",
    "And save your DataFrame in an Excel file with to_excel():\n",
    "```python\n",
    "df.to_excel('data2.xlsx')\n",
    "``` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "sample.to_excel('sample_data.xlsx')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As it is a binary file, you cannot open [sample_data.xlsx](sample_data.xlsx) in your browser."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Pickle Files\n",
    "\n",
    "Pickling is the act of converting Python objects into byte streams, and unpickling is the inverse process. This format makes it easy to store any Python objects as binary files and keep the data and hierarchy of the object.\n",
    "\n",
    "However, you should remember that they will only read back correctly if the Python version and package versions of the readers are the same as the writer.\n",
    "\n",
    "The pickle module has the same interface as the json module:\n",
    "\n",
    "| function | read/write | file/string |\n",
    "| :---:    | :----:     |  :-----:    |\n",
    "| load()   |  read      | file        |\n",
    "| dump()   |  write     | file        |\n",
    "| loads()  |  read      | string      |\n",
    "| dumps()  |  write     | string      |\n",
    "\n",
    "\n",
    "The following command pickles the DataFrame *df* and saves it as *data.pickle*:\n",
    "\n",
    "```python\n",
    "import pickle\n",
    "with open('data.pickle', 'wb') as f:\n",
    "    pickle.dump(df, f)\n",
    "```    \n",
    "\n",
    "While, the following unpickles *data.pickle* and loads it as a pandas DataFrame:\n",
    "\n",
    "```python   \n",
    "with open('data.pickle', 'rb') as f:\n",
    "    data = pickle.load(f)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You can also use the Pandas built-in functionality for dealing with pickle files.\n",
    "\n",
    "```python\n",
    "df.to_pickle('data.pickle')   # Pickles df and saves it as data.pickle\n",
    "\n",
    "pd.read_pickle('data.pickle')   # Unpickles and reads data.pickle\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**As a word of caution, you should always beware of loading pickles from unstructured sources. When you unpickle an unstrustworthy file, it could execute arbitrary code on your machine, performing dangerous actions and exploiting your device.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "sample.to_pickle('sample_data.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### HDF5\n",
    "\n",
    "HDF5 (Hierarchical Data Format 5) is a file format that has become quite popular. It can store a large amount of data in a single file, has compression features, and can store many datasets. HDF5 file format has a filesystem-like organization inside it, which means you can store the datasets in their own \"folder sctructure\" inside the file.\n",
    "\n",
    "HDFStore is dictionary-like object for reading and writing pandas using the **PyTables** library.\n",
    "\n",
    "To get data into an hdf5 file, you need to specify the filename and the key/group of the dataset. If you don't give it a path, it will put the key in the root group, which is the \"root folder\" of the hdf5 file. \n",
    "\n",
    "```python\n",
    "df.to_hdf('store.h5', key='/data', format='table', mode='a')\n",
    "```\n",
    "\n",
    "And in order to access and read from the HDF5 file:\n",
    "\n",
    "```python\n",
    "pd.read_hdf('store.h5', key='/data')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Similarly to the `open()` method, `to_hdf()` takes the `mode` parameter (`\"a\"`, `\"w\"` or `\"r+\"`). `to_hdf()` also requires a format parameter. You can read more about the different options here: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_hdf.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample.to_hdf('sample_data.h5', key='sample')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "rise": {
   "scroll": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}