NLP/01_string_basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# [1] NLP Basics\n",
    "\n",
    "\n",
    "## 1.1) Strings and encodings \n",
    "\n",
    "- Unicode and ASCII. Limitations/Benefits of different encodings\n",
    "- Formatting strings\n",
    "- Basic methods for strings\n",
    "- Working with dates: **`datetime`** module\n",
    "\n",
    "\n",
    "## 1.2)  Regular expressions\n",
    "\n",
    "- Working with regular expressions\n",
    "- Exercise\n",
    "\n",
    "\n",
    "## 1.3) From strings to lists of words\n",
    "\n",
    "- Python methods for strings\n",
    "- Splitting sequences: **`nltk`** vs **`split`**\n",
    "\n",
    "\n",
    "## 1.4) Features for words\n",
    "\n",
    "- From words to features.\n",
    "   - Dictionary based approach\n",
    "   - Hash based approach (investigate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 1.1) Strings and encodings \n",
    "\n",
    "- 1.1.1) Unicode and ASCII. Limitations/Benefits of different encodings\n",
    "- 1.1.2) Formatting strings\n",
    "- 1.1.3) Basic methods for strings\n",
    "- 1.1.4) Working with dates\n",
    "\n",
    "\n",
    "\n",
    "### 1.1.1) Unicode and ASCII. Limitations/Benefits of different encodings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:27:09.089721Z",
     "start_time": "2022-02-16T16:27:09.079658Z"
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "import string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:27:36.396077Z",
     "start_time": "2022-02-16T16:27:36.384525Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This is a string'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'This is a string'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:27:37.122317Z",
     "start_time": "2022-02-16T16:27:37.118577Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This is a string'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"This is a string\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:28:54.174034Z",
     "start_time": "2022-02-16T16:28:54.169410Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This is also \\na longer string\\nthe teacher\\nhi \\nho\\n'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''This is also \n",
    "a longer string\n",
    "the teacher\n",
    "hi \n",
    "ho\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:29:20.409277Z",
     "start_time": "2022-02-16T16:29:20.403735Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'aoaisjdoaisjd asdabuasdhaisd '"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"aoaisjdoaisjd asd\" \\\n",
    "+ \"abuasdhaisd \""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:29:52.456024Z",
     "start_time": "2022-02-16T16:29:52.451106Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "john said \"look out\" \n"
     ]
    }
   ],
   "source": [
    "s = 'john said \"look out\" '\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:29:57.116717Z",
     "start_time": "2022-02-16T16:29:57.111914Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The man who said \"I'll never accept your resignation\" ended up firing me because I didn't know enough NLP.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "a = \"\"\"The man who said \"I'll never accept your resignation\" ended up firing me because I didn't know enough NLP.\n",
    "\"\"\"\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#[1,2,3,4]\n",
    "# a->  00000\n",
    "# b -> 00001\n",
    "# z -> 28\n",
    "# @ -> \n",
    "# ASCII\n",
    "# unicode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-15T16:47:49.559568Z",
     "start_time": "2021-02-15T16:47:49.556191Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(\"a\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:30:08.992444Z",
     "start_time": "2022-02-16T16:30:08.986634Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bytes"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(b\"This is a string\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:30:10.519869Z",
     "start_time": "2022-02-16T16:30:10.513947Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(u\"This is a string\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:30:41.886613Z",
     "start_time": "2022-02-16T16:30:41.883028Z"
    }
   },
   "outputs": [],
   "source": [
    "a = \"unicode string\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:30:42.312000Z",
     "start_time": "2022-02-16T16:30:42.309177Z"
    }
   },
   "outputs": [],
   "source": [
    "a = \" @ \""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ascii vs utf-8\n",
    "\n",
    "There are many ways to encode strings.\n",
    "\n",
    "At the end of the day all symbols are simply mapped to integers in binary. The more symbols we have the more bits we need to encode them. \n",
    "\n",
    "Historically since computer science began in English countries, ASCII was the standard. \n",
    "\n",
    "- ASCII didn't contain any accents.\n",
    "- Unicode\n",
    "   - UTF-8 contains ASCCII.\n",
    "\n",
    "There has to be a big memory constraint for you to want to use ASCII. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:33:16.833017Z",
     "start_time": "2022-02-16T16:33:16.829551Z"
    }
   },
   "outputs": [],
   "source": [
    "a = u\"This is a house.\"\n",
    "b = b\"This is a house.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:33:24.623560Z",
     "start_time": "2022-02-16T16:33:24.617473Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(16, 16)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(a), len(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:33:25.608102Z",
     "start_time": "2022-02-16T16:33:25.601966Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sys.getsizeof(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:33:26.895485Z",
     "start_time": "2022-02-16T16:33:26.889379Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "49"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sys.getsizeof(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Why not ASCII?? \n",
    "#### Languages have accents, and \"rare\" (non english) symbols: ñ, è, é, ä, â, "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-15T16:53:28.857938Z",
     "start_time": "2021-02-15T16:53:28.854666Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string.ascii_letters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-09T19:34:25.706449Z",
     "start_time": "2020-02-09T19:34:25.703690Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' \\t\\n\\r\\x0b\\x0c'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string.whitespace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:34:17.537998Z",
     "start_time": "2022-02-16T16:34:17.532528Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Españoles, Franco ha muerto'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u'Españoles, Franco ha muerto'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:34:24.856688Z",
     "start_time": "2022-02-16T16:34:24.853362Z"
    }
   },
   "outputs": [],
   "source": [
    "# Can't do it!!\n",
    "# b'Españoles, Franco ha muerto' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:34:45.103913Z",
     "start_time": "2022-02-16T16:34:45.100552Z"
    }
   },
   "outputs": [],
   "source": [
    "#b'Españoles, Franco ha muerto' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-15T16:54:15.738071Z",
     "start_time": "2021-02-15T16:54:15.736055Z"
    }
   },
   "outputs": [],
   "source": [
    "a = \"Demasiados $\".encode(\"ascii\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:35:14.254100Z",
     "start_time": "2022-02-16T16:35:14.152636Z"
    }
   },
   "outputs": [
    {
     "ename": "UnicodeEncodeError",
     "evalue": "'ascii' codec can't encode character '\\u20ac' in position 11: ordinal not in range(128)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mUnicodeEncodeError\u001b[0m                        Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/lh/njnz5hj934d2js334v28z1x00000gn/T/ipykernel_15875/4290972885.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0ma\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"Demasiados €\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"ascii\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mUnicodeEncodeError\u001b[0m: 'ascii' codec can't encode character '\\u20ac' in position 11: ordinal not in range(128)"
     ]
    }
   ],
   "source": [
    "a = \"Demasiados €\".encode(\"ascii\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:35:52.594976Z",
     "start_time": "2022-02-16T16:35:52.591506Z"
    }
   },
   "outputs": [],
   "source": [
    "#\"CastÇa!\".encode(\"ascii\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-09T19:34:28.591235Z",
     "start_time": "2020-02-09T19:34:28.585573Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "## Can't do this\n",
    "# a = \"Castaña!\".encode(\"ascii\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:36:30.018163Z",
     "start_time": "2022-02-16T16:36:30.014012Z"
    }
   },
   "outputs": [],
   "source": [
    "a8  = \"Café\".encode(\"utf8\")\n",
    "a16 = \"Café\".encode(\"utf16\")\n",
    "a32 = \"Café\".encode(\"utf32\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:36:31.701257Z",
     "start_time": "2022-02-16T16:36:31.696035Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'Caf\\xc3\\xa9'\n",
      "b'\\xff\\xfeC\\x00a\\x00f\\x00\\xe9\\x00'\n",
      "b'\\xff\\xfe\\x00\\x00C\\x00\\x00\\x00a\\x00\\x00\\x00f\\x00\\x00\\x00\\xe9\\x00\\x00\\x00'\n"
     ]
    }
   ],
   "source": [
    "print(a8)\n",
    "print(a16)\n",
    "print(a32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:36:39.762598Z",
     "start_time": "2022-02-16T16:36:39.757136Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(38, 43, 53)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sys.getsizeof(a8), sys.getsizeof(a16), sys.getsizeof(a32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-09T19:34:31.076758Z",
     "start_time": "2020-02-09T19:34:31.074036Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'Caf\\xc3\\xa9'\n"
     ]
    }
   ],
   "source": [
    "print(b'Caf\\xc3\\xa9')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Unicode data \n",
    "\n",
    "Some  important things to consider:\n",
    "\n",
    "- When reading data from files, expect bytes and decode then with **`b.decode('utf-8')`**.\n",
    "\n",
    "- When writting data back to a file, encode it with **`b.encode('utf-8')`**.\n",
    "\n",
    "- Avoid using **`str()`** or  **`bytes()`** without an encoding to convert between types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:37:27.205930Z",
     "start_time": "2022-02-16T16:37:27.201664Z"
    }
   },
   "outputs": [],
   "source": [
    "f = open(\"../data/spanish_line.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:37:28.740815Z",
     "start_time": "2022-02-16T16:37:28.737027Z"
    }
   },
   "outputs": [],
   "source": [
    "a = f.readline()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:37:34.156283Z",
     "start_time": "2022-02-16T16:37:34.150931Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'El hombre inglés de la montaña no puede leer esto ya que usa un encoding de los 70.\\n'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:37:42.121412Z",
     "start_time": "2022-02-16T16:37:42.116210Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "''"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.readline()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1.2) Formatting strings\n",
    "\n",
    "We can format strings using the `x.format` method. This method allow us to introduce information inside the string x. We write placeholders `{}` inside the string `x` in the positions where we want to put certain information.\n",
    "\n",
    "Let us see some examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:38:13.683313Z",
     "start_time": "2022-02-16T16:38:13.679190Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a= 23\n"
     ]
    }
   ],
   "source": [
    "a = 23\n",
    "print(\"a=\", a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:39:17.771560Z",
     "start_time": "2022-02-16T16:39:17.765179Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'num epoch=0'"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "epoch_num = 0\n",
    "f'num epoch={epoch_num}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:40:40.934360Z",
     "start_time": "2022-02-16T16:40:40.928925Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name: David\t age: 19\n",
      "name: Jaquim\t age: 30\n",
      "name: Michael\t age: 50\n"
     ]
    }
   ],
   "source": [
    "names = ['David', 'Jaquim', 'Michael']\n",
    "ages = [19, 30, 50]\n",
    "\n",
    "for name,age in zip(names, ages):\n",
    "    #print('name: {}\\t age: {}'.format(name,age))\n",
    "    print(f'name: {name}\\t age: {age}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:42:54.035128Z",
     "start_time": "2022-02-16T16:42:54.029468Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name: David\t age: 19\n",
      "name: Jaquim\t age: 30\n",
      "name: Michael\t age: 50\n"
     ]
    }
   ],
   "source": [
    "for k in range(min(len(names), len(ages))):\n",
    "    name = names[k]\n",
    "    age = ages[k]\n",
    "    print(f'name: {name}\\t age: {age}')  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:43:18.787638Z",
     "start_time": "2022-02-16T16:43:18.782087Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Model accuracy: 97%'"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Write number as percentage\n",
    "'Model accuracy: {}%'.format(97)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:43:37.736641Z",
     "start_time": "2022-02-16T16:43:37.731237Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Model accuracy: 97%'"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aux = 97\n",
    "f'Model accuracy: {aux}%'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-15T16:59:25.923840Z",
     "start_time": "2021-02-15T16:59:25.920368Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Model accuracy: 97%'"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acc = 97\n",
    "f'Model accuracy: {acc}%'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Placeholders and digit formatting\n",
    "\n",
    "We can speficy the number of digits used when formatting a number, or the maximum number of decimals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "big number 10000000000 hard to read\n"
     ]
    }
   ],
   "source": [
    "print('big number {} hard to read'.format(10**10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "big number 10,000,000,000 better with commas\n"
     ]
    }
   ],
   "source": [
    "print('big number {:,} better with commas'.format(10**10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "big number 10,000,000,000.000 also with decimal places\n"
     ]
    }
   ],
   "source": [
    "print('big number {:,.3f} also with decimal places'.format(10**10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:44:33.031807Z",
     "start_time": "2022-02-16T16:44:33.026284Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number 00005\n",
      "number 00006\n",
      "number 00007\n",
      "number 00008\n",
      "number 00009\n",
      "number 00010\n",
      "number 00011\n",
      "number 00012\n",
      "number 00013\n",
      "number 00014\n"
     ]
    }
   ],
   "source": [
    "for i in range(5,15):\n",
    "    print('number {:05}'.format(i))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:45:47.283985Z",
     "start_time": "2022-02-16T16:45:47.278269Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number 2 dedimals 1.67       number 3 decimals 1.667\n",
      "number 2 dedimals 2.00       number 3 decimals 2.000\n",
      "number 2 dedimals 2.33       number 3 decimals 2.333\n",
      "number 2 dedimals 2.67       number 3 decimals 2.667\n",
      "number 2 dedimals 3.00       number 3 decimals 3.000\n",
      "number 2 dedimals 3.33       number 3 decimals 3.333\n",
      "number 2 dedimals 3.67       number 3 decimals 3.667\n",
      "number 2 dedimals 4.00       number 3 decimals 4.000\n",
      "number 2 dedimals 4.33       number 3 decimals 4.333\n",
      "number 2 dedimals 4.67       number 3 decimals 4.667\n"
     ]
    }
   ],
   "source": [
    "# Decide the number of decimals\n",
    "for i in range(5,15):\n",
    "    print('number 2 dedimals {:.2f}'.format(i/3.), end=\"       \")\n",
    "    print('number 3 decimals {:.3f}'.format(i/3.))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1,234,567,890'"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Separate numbers with comma\n",
    "'{:,}'.format(1234567890)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Placeholders with integer values \n",
    "We can also use placeholders with integers inside, this can be usefull in a variety of situations. For example, if we want to print a repetead value inside a string we don't need to pass it to the format method several times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name: David\t name again: David \t age: 19\n",
      "name: Jaquim\t name again: Jaquim \t age: 30\n",
      "name: Michael\t name again: Michael \t age: 50\n"
     ]
    }
   ],
   "source": [
    "# We don't need to do this\n",
    "for name,age in zip(names,ages):\n",
    "    print('name: {}\\t name again: {} \\t age: {}'.format(name, name, age))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:47:19.523501Z",
     "start_time": "2022-02-16T16:47:19.517853Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name: David\t name again: David \t age: David\n",
      "name: Jaquim\t name again: Jaquim \t age: Jaquim\n",
      "name: Michael\t name again: Michael \t age: Michael\n"
     ]
    }
   ],
   "source": [
    "# We can simply use placeholders with integers inside to refer\n",
    "# to the potition of the input of the format method.\n",
    "for name,age in zip(names,ages):\n",
    "    print('name: {0}\\t name again: {0} \\t age: {1}'.format(name,age))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Placeholders with keyword arguments\n",
    "\n",
    "We can also use keyword arguments inside the format method. By doing so we don't need to take into account the order at which the inputs of `forward` are sent.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name: David\t \t age: 19\n",
      "name: Jaquim\t \t age: 30\n",
      "name: Michael\t \t age: 50\n"
     ]
    }
   ],
   "source": [
    "for n,a in zip(names,ages):\n",
    "    print('name: {name}\\t \\t age: {age}'.format(age=a, name=n))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'a, b, c'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'{0}, {1}, {2}'.format('a', 'b', 'c')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'c, b, a'"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'{2}, {1}, {0}'.format('a', 'b', 'c')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Coordinates: 37.24N, -115.81W'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'Coordinates: {latitude}, {longitude}'.format(latitude='37.24N', longitude='-115.81W')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Placeholders with dictionary inputs\n",
    "\n",
    "We can pass dictionaries in the `format` method and use the keys of the dictionaries inside the placeholders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:48:36.860689Z",
     "start_time": "2022-02-16T16:48:36.856164Z"
    }
   },
   "outputs": [],
   "source": [
    "names = ['David', 'Jaquim', 'Michael']\n",
    "ages = [19, 30, 50]\n",
    "d = []\n",
    "\n",
    "for n,a in zip(names,ages):\n",
    "    d.append({'name':n, 'age':a})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:48:38.117867Z",
     "start_time": "2022-02-16T16:48:38.113609Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'name': 'David', 'age': 19},\n",
       " {'name': 'Jaquim', 'age': 30},\n",
       " {'name': 'Michael', 'age': 50}]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name: David\t \t age: 19\n",
      "name: Jaquim\t \t age: 30\n",
      "name: Michael\t \t age: 50\n"
     ]
    }
   ],
   "source": [
    "for d_k in d:\n",
    "    print('name: {name}\\t \\t age: {age}'.format(**d_k))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Coordinates: 37.24N, -115.81W'"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coord = {'latitude': '37.24N', 'longitude': '-115.81W'}\n",
    "'Coordinates: {latitude}, {longitude}'.format(**coord)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Access object atributes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "c = 3-5j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:49:36.202003Z",
     "start_time": "2022-02-16T16:49:36.193341Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'c' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/lh/njnz5hj934d2js334v28z1x00000gn/T/ipykernel_15875/1137250948.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;34m' {0} is formed from the real part {0.real} and the imaginary part {0.imag}.'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'c' is not defined"
     ]
    }
   ],
   "source": [
    "' {0} is formed from the real part {0.real} and the imaginary part {0.imag}.'.format(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:49:38.962320Z",
     "start_time": "2022-02-16T16:49:38.958112Z"
    }
   },
   "outputs": [],
   "source": [
    "class Point(object):\n",
    "    def __init__(self, x, y):\n",
    "         self.x, self.y = x, y\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:50:03.305836Z",
     "start_time": "2022-02-16T16:50:03.300404Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<__main__.Point at 0x7fa308811b20>"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Point(1,2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:50:30.731164Z",
     "start_time": "2022-02-16T16:50:30.726459Z"
    }
   },
   "outputs": [],
   "source": [
    "class Point(object):\n",
    "    def __init__(self, x, y):\n",
    "         self.x, self.y = x, y\n",
    "    def __str__(self):\n",
    "         return 'Point({self.x}, {self.y})'.format(self=self)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:50:31.486950Z",
     "start_time": "2022-02-16T16:50:31.484142Z"
    }
   },
   "outputs": [],
   "source": [
    "aux = Point(4,2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:50:44.866217Z",
     "start_time": "2022-02-16T16:50:44.862828Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Point(4, 2)\n"
     ]
    }
   ],
   "source": [
    "print(aux)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4, 2)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aux.x, aux.y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Point(4, 2)\n"
     ]
    }
   ],
   "source": [
    "print(aux)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1.3) Basic methods for strings\n",
    "\n",
    "- `str.find(s,w)`\n",
    "   - returns the opsition of the first occurance of `w` in `s`\n",
    "\n",
    "\n",
    "- `str.replace(x, s1, s2)`\n",
    "   - Replaces each `s1` in `x` with by `s2`\n",
    "\n",
    "\n",
    "- `str.split(x,sep=None)`\n",
    "  - If `sep` is not specified or is `None`, any\n",
    "    whitespace string is a separator and empty strings are\n",
    "    removed from the result.\n",
    "\n",
    "\n",
    "- `str.upper(x)`\n",
    "  - Returns the string `x` with all characters uppercase.\n",
    "\n",
    "\n",
    "- `str.lower(x)`\n",
    "  - Returns the string `x` with all characters in lowercase.\n",
    "\n",
    "\n",
    "- `str.endswith(x,s)`\n",
    "    - Returns True if `x` ends with substring `s`\n",
    "    \n",
    "\n",
    "- `str.startswith(x,s)`\n",
    "    - Returns True if `x` ends with substring `s`\n",
    "  \n",
    "#### issomething methods\n",
    "  \n",
    "- `str.isupper(x)`\n",
    "  - Returns True if  `x` all its characters are uppercased.\n",
    "  \n",
    "  \n",
    "- `str.isdigit(x)`\n",
    "  - Returns True if  `x` all its characters are digits.\n",
    "  \n",
    "  \n",
    "- `str.isalpha(x)`\n",
    "    - Return True if all characters in x are alphabetic\n",
    "    \n",
    "    \n",
    "- `str.isalnum(x)`\n",
    "    - Return True if all characters in x are alphabetic\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:52:32.723512Z",
     "start_time": "2022-02-16T16:52:32.717668Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<method 'find' of 'str' objects>"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str.find"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-02-16T16:53:04.341744Z",
     "start_time": "2022-02-16T16:53:04.335692Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'hello world'"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"Hello world\".lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'trapped for more than two weeks in a cave in Thailand. '"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = \"Rescue divers will have to navigate flooded passages, with some openings as small as 3 feet\\\n",
    "     wide and 2 feet tall, to rescue 12 boys and their soccer coach who have been\\\n",
    "     trapped for more than two weeks in a cave in Thailand. \"\n",
    "\n",
    "w = \"trapped\"\n",
    "\n",
    "s[str.find(s,w):]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('The cat', 'Cats have sex leves')"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str.replace(\"The house\", \"house\", \"cat\"), str.replace(\"Cats have six lives\", \"i\", \"e\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string.ascii_letters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['h', 'use is big']"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Return a list of strings splitted by the sep character\n",
    "str.split(\"house is big\", sep=\"o\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['house', 'is', 'big']"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Return a list of strings splitted by the sep character\n",
    "str.split(\"house is big\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'House'"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# return string with first character uppercased\n",
    "str.capitalize(\"house\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'the ceo of apple resigned.'"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Return the string with lowered characters\n",
    "str.lower(\"The CEO of Apple resigned.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of non overlappeing occurencers of the query string in the input string\n",
    "str.count(\"this house is bigger than this car\",\"this\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check if input ends with a substring\n",
    "str.endswith(\"Encoding\",\"ing\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, False)"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check if input starts with substring\n",
    "str.startswith(\"Encoding\",\"Enc\"), str.startswith(\"Encoding\",\"enc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### issomething methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, False, False)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# verify if al characters are capitals\n",
    "'A'.isupper(),'Abc'.isupper(), 'a'.isupper()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, False, True)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# verify if all characters are digits\n",
    "'a'.isdigit(), '7eleven'.isdigit(), '723'.isdigit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, False, True, False, True)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check if all characters alphabetic\n",
    "str.isalpha(\"7eleven\"), str.isalpha(\"the house\"), str.isalpha(\"thehouse\"),str.isalpha(\"the1\"), str.isalpha(\"castaña\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check if all characters alphabetic OR numeric\n",
    "str.isalnum(\"7eleven\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### 1.1.4) Working with dates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "my_date = datetime.datetime(2019,10,5,10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.datetime(2019, 10, 5, 10, 0)"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_date"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2)  Regular expressions\n",
    "\n",
    "\n",
    "- Working with regular expressions\n",
    "- Exercise\n",
    "\n",
    "#### re.compile\n",
    "\n",
    "- Generate a regular expression using **`p = re.compile(r'our_regular_expression')`**.\n",
    "    - `p` will be a  `SRE_Pattern` object that we can apply to any string.\n",
    "\n",
    "\n",
    "- **`p.findall(s)`** returns a list containing all substrings within `s` that safisfy our regular expression.\n",
    "\n",
    "\n",
    "- **`p.finditer(s)`** returns a generator. Each element of the generator is a `SRE_Match`which contains \n",
    "    - **`span(x,y)`** indicating that the match starts at position `x` and ends in position `y` (of s).\n",
    "    - **`match='...'`** indicating the string that satisfies the regular expression.\n",
    "\n",
    "\n",
    "\n",
    "- **`p.sub(r'XXX', s)`** change all patterns by `XXX`\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "#### Special characters in regular expression\n",
    "\n",
    "- **`.`** Matches any character except new line.\n",
    "\n",
    "\n",
    "- **`\\d`** Matches any digit (0-9). Equivalent to `[0-9]`.\n",
    "\n",
    "\n",
    "- **`\\D`** Matches any NON digit. Equivalent to `[^0-9]`.\n",
    "\n",
    "\n",
    "- **`\\w`** Matches any \"word character\". Equivalent to `[a-zA-Z0-9_]`\n",
    "\n",
    "\n",
    "- **`\\W`** Matches any NON alphanumeric character. Equivalent to `[^a-zA-Z0-9_]`\n",
    "\n",
    "\n",
    "- **`\\s`** Matches any whitespace (space, tab, newline). Equivalent to `[\\t\\n\\r\\f\\v]` \n",
    "\n",
    "\n",
    "- **`\\S`** Matches any NON whitespace (space, tab, newline). Equivalent to `[^ \\t\\n\\r\\f\\v]` \n",
    "\n",
    "#### Anchers\n",
    "\n",
    "- **`\\b`** Matches any word boundary.\n",
    "\n",
    "\n",
    "- **`\\b`** Matches any NON word boundary.\n",
    "\n",
    "\n",
    "- **`^`** Matches beginning of a string\n",
    "\n",
    "\n",
    "- **`[^a-e]`** negates the character set. That is matches any character outside `a-e`.\n",
    "\n",
    "\n",
    "- **`$`** Matches a position that is end of a string."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "#### Working with character sets\n",
    "\n",
    "- **`[]`** allows us to specify sets of symbols.\n",
    "\n",
    "\n",
    "- **`[ab3-]`** matches any character `a`  or `b`or `3` or `-`.\n",
    "\n",
    "\n",
    "- **`[a-g]`** matches any character `a`  to `g` such as `b`,`c`,`d`,...,`g`\n",
    "\n",
    "\n",
    "- **`[A-G]`** matches any character `A`  to `G` such as `B`,`C`,`D`,...,`G`\n",
    "\n",
    "\n",
    "- **`[a-zA-G]`** matches any lowerchase character and any uppercase character from `A`  to `G`.\n",
    "\n",
    "\n",
    "- **`[d1-d2]`** matches any digit between `d1` and `d2`. For example `[0-5]` matches `0`,`1`,`2`,`3`,`4`,`5`\n",
    "\n",
    "\n",
    "For example:\n",
    "\n",
    "- `r'[abcd][abcd][a-z]'` matches any 3 character word that starts which contains a,b,c,d in the first 2 letters such as `cat`.\n",
    "\n",
    "\n",
    "#### Quantifiers \n",
    "\n",
    "Quantifiers are operators that are applied to the preceding symbol.\n",
    "\n",
    "\n",
    "- **`*`** previous symbol appears 0 or more matches.\n",
    "\n",
    "\n",
    "- **`+`** previous symbol appears 1 or more matches.\n",
    "\n",
    "\n",
    "- **`?`** previous symbol appears at most one (0 or 1).\n",
    "\n",
    "\n",
    "- **`{k}`** previous symbol repeated k exact matches.\n",
    "\n",
    "\n",
    "- **`{min,max}`** previous symbol appears  any number between min and max. For example `{2,8}` would be between 2 and 8 matches\n",
    "\n",
    "\n",
    "For example:\n",
    "\n",
    "\n",
    "\n",
    "#### Groups\n",
    "\n",
    "Groups are created between parenthesis\n",
    "\n",
    "- **`(abc)?`**  is the group `abc` which cap appear or not.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T22:05:51.015050Z",
     "start_time": "2021-02-14T22:05:51.009704Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"r'a.*'\""
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"r'a.*'\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T22:05:51.403327Z",
     "start_time": "2021-02-14T22:05:51.400296Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tSomething\n",
      "\\tSomething\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "a = '\\tSomething'   # Standard string\n",
    "b = r'\\tSomething'  # Raw string\n",
    "print(a)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T22:08:13.815363Z",
     "start_time": "2021-02-14T22:08:13.809848Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ack', 'cat', 'cat']"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aux =  \"The girl who loved the black cat ended up with a catwomen costume.\"\n",
    "p = re.compile(r'[abcd][abcd][a-z]')\n",
    "p.findall(aux)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T22:08:14.035181Z",
     "start_time": "2021-02-14T22:08:14.031414Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['cat', 'cat']"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = re.compile(r'cat')\n",
    "p.findall(aux)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T22:08:16.815577Z",
     "start_time": "2021-02-14T22:08:16.810121Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['cat ']"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = re.compile(r'cat\\s')\n",
    "p.findall(aux)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T22:08:42.646019Z",
     "start_time": "2021-02-14T22:08:42.640506Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['cat ', 'cat']"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = re.compile(r'cat\\s*')\n",
    "p.findall(aux)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T21:32:04.389241Z",
     "start_time": "2021-02-14T21:32:04.383894Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['cat ended up with a catwomen costume.']"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = re.compile(r'cat.*')\n",
    "p.findall(aux)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T21:34:34.753046Z",
     "start_time": "2021-02-14T21:34:34.747728Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['649070976']"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "phones = \"My phone number is 649070976, please don't call me, call 999.\"\n",
    "p = re.compile(r'\\d\\d\\d\\d\\d\\d\\d\\d\\d')\n",
    "p.findall(phones)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T21:34:51.712171Z",
     "start_time": "2021-02-14T21:34:51.706910Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "re.Pattern"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T21:31:00.807217Z",
     "start_time": "2021-02-14T21:31:00.801329Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['649', '070', '976', '999']"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "phones = \"My phone number is 649070976, please don't call me, call 999.\"\n",
    "p = re.compile(r'\\d\\d\\d')\n",
    "p.findall(phones)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T21:21:03.034351Z",
     "start_time": "2021-02-14T21:21:03.028503Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['649070976', '999']"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "phones = \"My phone number is 649070976, please don't call me, call 999.\"\n",
    "p = re.compile(r'\\d{3,10}')\n",
    "p.findall(phones)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T21:21:06.548797Z",
     "start_time": "2021-02-14T21:21:06.543538Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['649070976']"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "phones = \"My phone number is 649070976, please don't call me, call 999.\"\n",
    "p = re.compile(r'\\d{7,10}')\n",
    "p.findall(phones)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T21:58:33.899070Z",
     "start_time": "2021-02-14T21:58:33.895092Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['My ']"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = re.compile(r'My.')\n",
    "p.findall(phones)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T21:59:26.138599Z",
     "start_time": "2021-02-14T21:59:26.134810Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['mr.']"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "phones = \"Mr. Robert went to the moon\"\n",
    "phones = phones.lower()\n",
    "p = re.compile(r'mr\\.')\n",
    "p.findall(phones)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['643*234*1312', '645-312-3132', '654.425.2214']"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example = \"The phone 643*234*1312 will be picked up by the regular expression but we might\"+\\\n",
    "          \"want only numbers separated by . or _ such as 645-312-3132 or 654.425.2214.\"\n",
    "    \n",
    "p = re.compile(r'\\d\\d\\d.\\d\\d\\d.\\d\\d\\d\\d')\n",
    "p.findall(example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['645-312-3132', '654.425.2214']"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = re.compile(r'\\d\\d\\d[-.]\\d\\d\\d[-.]\\d\\d\\d\\d')\n",
    "p.findall(example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example = \"Notice that the character set tries to match one of the characters within for example\\\n",
    "        649--342-2332 will not be picked by our regular expression neither 649---342-2333\"\n",
    "p.findall(example)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['649--342-2332']"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = re.compile(r'\\d\\d\\d[-.][-.]\\d\\d\\d[-.]\\d\\d\\d\\d')\n",
    "p.findall(example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['649--342-2332', '649---342-2333']"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = re.compile(r'\\d\\d\\d[-.]*\\d\\d\\d[-.]\\d\\d\\d\\d')\n",
    "p.findall(example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['649--342-2332', '649---342-2333']"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = re.compile(r'\\d{3}[-.]{0,3}\\d{3}[.-]\\d{4}')\n",
    "p.findall(example)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Substitutions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-13T15:11:30.962538Z",
     "start_time": "2021-06-13T15:11:30.959942Z"
    }
   },
   "outputs": [],
   "source": [
    "urls = '''\n",
    "https://www.ub.edu is the web of the university\n",
    "The string ub.edu is not valid url\n",
    "www.upc.edu is the web of the university\n",
    "The scientists from https://nasa.gov are happy\n",
    "The url www.illinois.gov was broken\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-13T15:16:15.576968Z",
     "start_time": "2021-06-13T15:16:15.573461Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['www.ub.edu', 'www.upc.edu']\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "edu_regex = r'w+.\\w+.edu'\n",
    "gov_regex = r'{www}+.*.gov'\n",
    "import re\n",
    "pattern = re.compile(edu_regex)\n",
    "print(pattern.findall(urls))\n",
    "\n",
    "pattern = re.compile(gov_regex)\n",
    "print(pattern.findall(urls))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T22:00:18.109852Z",
     "start_time": "2021-02-14T22:00:18.104647Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(1, 19), match='https://www.ub.edu'>\n",
      "<re.Match object; span=(49, 68), match='https://youtube.com'>\n",
      "<re.Match object; span=(86, 106), match='https://www.nasa.gov'>\n",
      "\n",
      "group 1: www.\n",
      "group 1: None\n",
      "group 1: www.\n",
      "\n",
      "group 2: .edu\n",
      "group 2: .com\n",
      "group 2: .gov\n"
     ]
    }
   ],
   "source": [
    "pattern = re.compile(r'https?://(www\\.)?\\w+(\\.\\w+)')\n",
    "\n",
    "for match in pattern.finditer(urls):\n",
    "    print(match)\n",
    "\n",
    "print(\"\")\n",
    "for match in pattern.finditer(urls):\n",
    "    print('group 1:', match.group(1))\n",
    "    \n",
    "print(\"\")\n",
    "for match in pattern.finditer(urls):\n",
    "    print('group 2:', match.group(2))\n",
    "    \n",
    "#print(\"\")\n",
    "#for match in pattern.finditer(urls):\n",
    "#    print('group 3:', match.group(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T22:00:18.311084Z",
     "start_time": "2021-02-14T22:00:18.307398Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('www.', '.edu'), ('', '.com'), ('www.', '.gov')]"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pattern.findall(urls)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T22:00:18.600365Z",
     "start_time": "2021-02-14T22:00:18.594572Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(1, 19), match='https://www.ub.edu'>\n",
      "<re.Match object; span=(49, 68), match='https://youtube.com'>\n",
      "<re.Match object; span=(86, 106), match='https://www.nasa.gov'>\n",
      "\n",
      "group 1: www.\n",
      "group 1: None\n",
      "group 1: www.\n",
      "\n",
      "group 2: ub\n",
      "group 2: youtube\n",
      "group 2: nasa\n",
      "\n",
      "group 3: .edu\n",
      "group 3: .com\n",
      "group 3: .gov\n"
     ]
    }
   ],
   "source": [
    "pattern = re.compile(r'https?://(www\\.)?(\\w+)(\\.\\w+)')\n",
    "\n",
    "for match in pattern.finditer(urls):\n",
    "    print(match)\n",
    "    \n",
    "print(\"\")\n",
    "for match in pattern.finditer(urls):\n",
    "    print('group 1:', match.group(1))\n",
    "    \n",
    "print(\"\")\n",
    "for match in pattern.finditer(urls):\n",
    "    print('group 2:', match.group(2))\n",
    "    \n",
    "print(\"\")\n",
    "for match in pattern.finditer(urls):\n",
    "    print('group 3:', match.group(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T22:00:21.567378Z",
     "start_time": "2021-02-14T22:00:21.561344Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('www.', 'ub', '.edu'), ('', 'youtube', '.com'), ('www.', 'nasa', '.gov')]"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pattern.findall(urls)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Why groups? We can use them to change parts of string matches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "https://www.ub.edu is the web of the university\n",
      "https://youtube.com is from google !\n",
      "https://www.nasa.gov is where the geeks work\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(urls)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "***hiddenurl*** is the web of the university\n",
      "***hiddenurl*** is from google !\n",
      "***hiddenurl*** is where the geeks work\n",
      "\n"
     ]
    }
   ],
   "source": [
    "urls2   = pattern.sub(r'***hiddenurl***', urls)\n",
    "print(urls2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "ub.edu is the web of the university\n",
      "youtube.com is from google !\n",
      "nasa.gov is where the geeks work\n",
      "\n"
     ]
    }
   ],
   "source": [
    "pattern = re.compile(r'https?://(www\\.)?(\\w+)(\\.\\w+)')\n",
    "urls2   = pattern.sub(r'\\2\\3', urls)\n",
    "print(urls2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      ".edu is the web of the university\n",
      ".com is from google !\n",
      ".gov is where the geeks work\n",
      "\n"
     ]
    }
   ],
   "source": [
    "urls2   = pattern.sub(r'\\3', urls)\n",
    "print(urls2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise: Find all telephone numbers that start with 900 or 800"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['800-555-5669', '900-555-9340', '800-555-6771', '900-555-3205', '800-555-6089', '800-555-7100', '900-555-5118', '900-555-5428', '800-555-8810', '900-555-9598', '800-555-2420', '900-555-3567', '800-555-3216', '900-555-7755', '800-555-1372', '900-555-6426']\n"
     ]
    }
   ],
   "source": [
    "p = re.compile(r'[89]00.\\d\\d\\d.\\d\\d\\d\\d')\n",
    "with open(\"../data/people_data.txt\") as f:\n",
    "    info = f.read()\n",
    "    matches = p.findall(info)\n",
    "    print(matches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise: Find all telephone numbers that DO NOT start with 8 or 9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['615-555-7164', '560-555-5153', '714-555-7405', '783-555-4799', '516-555-4615', '127-555-1867', '608-555-4938', '568-555-6051', '292-555-1875', '614-555-1166', '530-555-2676', '470-555-2750', '777-555-8378', '196-555-5674', '203-555-3475', '195-555-2405', '321-555-9053', '133-555-1711', '760-555-7147', '391-555-6621', '609-555-7908', '149-555-7657', '130-555-9709', '143-555-9295', '574-555-3194', '496-555-7533', '210-555-3757', '669-555-7159', '152-555-7417', '217-555-7123', '786-555-6544', '780-555-2574', '372-555-9809', '670-555-3005', '509-555-5997', '721-555-5632', '147-555-6830', '582-555-3426', '400-555-1706', '525-555-1793', '317-555-6700', '746-555-4094', '711-555-4427', '355-555-1872', '691-555-5773', '332-555-5441', '379-555-3685', '127-555-9682', '789-555-7032', '783-555-5135', '315-555-6507', '481-555-5835', '365-555-8287', '681-555-2460', '274-555-9800', '300-555-7821', '133-555-3889', '705-555-6863', '215-555-9449', '623-555-3006', '192-555-4977', '178-555-4899']\n"
     ]
    }
   ],
   "source": [
    "p = re.compile(r'[0-7]\\d\\d.\\d\\d\\d.\\d\\d\\d\\d')\n",
    "with open(\"../data/people_data.txt\") as f:\n",
    "    info = f.read()\n",
    "    matches = p.findall(info)\n",
    "    print(matches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise: Find all mister references in the string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Mr.']"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = \"\"\"Mr. Robbinson and Mr David bought the company from Mr  JJ yet and Mr dani\"\"\"\n",
    "p = re.compile(r'Mr\\.')\n",
    "p.findall(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Mr. R', 'Mr  J']\n",
      "['Mr D']\n",
      "['Mr. R', 'Mr D', 'Mr  J']\n"
     ]
    }
   ],
   "source": [
    "## Why is this not pikking \"Mr David\" ?\n",
    "## Well because I am imposing (M,r,[. ],\\s,[A-Z])\n",
    "## And \"Mr D\" breaks this at the 4th character which is D and should be \\s\n",
    "p = re.compile(r'Mr[. ]\\s[A-Z]')\n",
    "print(p.findall(s))\n",
    "\n",
    "## If we remove \\s now  only 'Mr David' follows (M,r,[. ],[A-Z])\n",
    "p = re.compile(r'Mr[. ][A-Z]')\n",
    "print(p.findall(s))\n",
    "\n",
    "## If we add a symbol ? after \\s then we make optional the empty space after [. ].\n",
    "## Therefore any string matching (M,r,[. ],\\s,[A-Z]) or (M,r,[. ],[A-Z])\n",
    "p = re.compile(r'Mr[. ]?\\s[A-Z]')\n",
    "print(p.findall(s))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Mr. R', 'Mr D', 'Mr  J']\n",
      "['Mr. Robbinson and Mr David bought the company from Mr  JJ yet and M']\n",
      "['Mr. R', 'Mr D', 'Mr  J']\n",
      "['Mr. Robbinson', 'Mr David', 'Mr  JJ']\n",
      "['Mr. Robbinson', 'Mr David', 'Mr  JJ', 'Mr dani']\n"
     ]
    }
   ],
   "source": [
    "## Be carefull making expressions too general\n",
    "## This one can have any number of dots or empty spaces\n",
    "p = re.compile(r'Mr[. ]*[A-Z]')\n",
    "print(p.findall(s))\n",
    "\n",
    "## This one can have any number of times ANY character\n",
    "p = re.compile(r'Mr.*[A-Z]')\n",
    "print(p.findall(s))\n",
    "\n",
    "## This one can have  or a dot or an empty space 0 or more times\n",
    "p = re.compile(r'Mr[. ]*[A-Z]')\n",
    "print(p.findall(s))\n",
    "\n",
    "## If we want the whole name then we need to allow the regular expression to get any \n",
    "## character after the Capital letter any number of times. Since `dani` does not start\n",
    "## in capital it is not matched\n",
    "p = re.compile(r'Mr[. ]*[A-Z]\\w*')\n",
    "print(p.findall(s))\n",
    "\n",
    "## If we also want dani ... \n",
    "p = re.compile(r'Mr[. ]*\\w*')\n",
    "print(p.findall(s))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Mr. Robbinson', 'Mr David', 'Mr  JJ', 'Mr dani']\n"
     ]
    }
   ],
   "source": [
    "p = re.compile(r'Mr.?\\s\\w*')\n",
    "print(p.findall(s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise: Find all email adresses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-13T15:24:27.213620Z",
     "start_time": "2021-06-13T15:24:27.211222Z"
    }
   },
   "outputs": [],
   "source": [
    "s = \"\"\"Hei could you send the same information my .gmail account: miss.cute@gmail.com ?\n",
    "       If you want further info please contact cute_pikachu@hotmail.com but not  \n",
    "       churmander_norris66@mat.ub.edu\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-13T15:19:11.538564Z",
     "start_time": "2021-06-13T15:19:11.535351Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['miss.cute@gmail.com',\n",
       " 'cute_pikachu@hotmail.com',\n",
       " 'churmander_norris66@mat.ub.edu']"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = re.compile(r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+')\n",
    "p.findall(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-13T15:23:31.751872Z",
     "start_time": "2021-06-13T15:23:31.748428Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['miss.cute@gmail.com',\n",
       " 'cute_pikachu@hotmail.com',\n",
       " 'churmander_norris66@mat.ub.']"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = re.compile(r'[.\\w]+@[\\w\\.]+[.com]+')\n",
    "p.findall(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-13T15:31:57.972409Z",
     "start_time": "2021-06-13T15:31:57.969870Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['miss.cute@gmail.com', 'cute_pikachu@hotmail.com']\n"
     ]
    }
   ],
   "source": [
    "# do not show the .ub\n",
    "p = re.compile(r'[.\\w]+@[\\w]+.com')\n",
    "print(p.findall(s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "home = r\"We found the area of Turo Parc this beautiful ground floor of 220 m2 plus a terrace of 30 m2 the floor consists\\\n",
    "     of a large salo dining room overlooking the Turo Prac with large windows, kitchen with natural light with glass\\\n",
    "     to see the park in a back are the rooms where there are two double rooms one single with access to the terrace \\\n",
    "     and a suite room with bathroom, bathroom with shower, with the floor is included a parking space not located in\\\n",
    "     the same building. It is a flat in one of the most luxurious and residential areas of Barcelona. surrounded by\\\n",
    "     shops cinemas supermarkets shopping center Diagonal Island, market Galvany gyms. with public transport of \\\n",
    "     railways and buses\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['area', 'area']\n"
     ]
    }
   ],
   "source": [
    "pattern = re.compile(r'area')\n",
    "matches = pattern.findall(home)\n",
    "print(matches)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<_sre.SRE_Match object; span=(13, 17), match='area'>,\n",
       " <_sre.SRE_Match object; span=(541, 545), match='area'>]"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matches = pattern.finditer(home)\n",
    "matches_list = list(matches)\n",
    "matches_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_sre.SRE_Match object; span=(13, 17), match='area'>\n",
      "<_sre.SRE_Match object; span=(541, 545), match='area'>\n"
     ]
    }
   ],
   "source": [
    "matches = pattern.finditer(home)\n",
    "for m in matches:\n",
    "    print(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' sen'"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m1 = matches_list[0].span()\n",
    "s[m1[0]:m1[1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3) From strings to lists of words\n",
    "\n",
    "\n",
    "- **`str.split(\" \")`**: Is a bad approach for general text.\n",
    "\n",
    "\n",
    "- **`nltk.wordpunct_tokenize`**: Good string tokenizer that takes into account commas, empty spaces etc...\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /Users/qijunjin/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "#nltk.download()\n",
    "#nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = \"the man, who could not stop looking at the coffe shop,   decided to enter \\n and ask a latte.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['the', 'man', ',', 'who', 'could', 'not', 'stop', 'looking', 'at', 'the', 'coffe', 'shop', ',', 'decided', 'to', 'enter', 'and', 'ask', 'a', 'latte', '.']\n",
      "['the', 'man,', 'who', 'could', 'not', 'stop', 'looking', 'at', 'the', 'coffe', 'shop,', '', '', 'decided', 'to', 'enter', '\\n', 'and', 'ask', 'a', 'latte.']\n"
     ]
    }
   ],
   "source": [
    "print(nltk.word_tokenize(s))\n",
    "print(s.split(\" \"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I', \"'ll\", 'go', 'to', 'the', 'coffe', 'shop', '.']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.word_tokenize(\"I'll go to the coffe shop.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"I'lla\", 'go', 'to', 'the', 'coffe', 'shop', '.']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.word_tokenize(\"I'lla go to the coffe shop.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tokenizing in catalan/spanish..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "toktok = nltk.ToktokTokenizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"L'Amador\", 'té', 'un', 'gos', 'molt', 'pesat', '.']"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.word_tokenize(\"L'Amador té un gos molt pesat.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['L', \"'\", 'Amador', 'té', 'un', 'gos', 'molt', 'pesat', '.']"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toktok.tokenize(\"L'Amador té un gos molt pesat.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['¿', 'Dónde', 'estoy', '?', '¿', 'Puede', 'alguien', 'ayudarme', '?']"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toktok.tokenize(\"¿Dónde estoy? ¿Puede alguien ayudarme ?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Formatting strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T21:06:46.792703Z",
     "start_time": "2021-02-14T21:06:46.789483Z"
    }
   },
   "outputs": [],
   "source": [
    "col_formater = \"epoch: {0:3}, loss: {1:.3f}, acc: {2:4}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T21:06:46.995520Z",
     "start_time": "2021-02-14T21:06:46.992057Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:   1, loss: 2.312, acc: 96.4\n"
     ]
    }
   ],
   "source": [
    "ep = 1\n",
    "l = 2.31231213123\n",
    "acc = 96.4\n",
    "print(col_formater.format(ep,l,acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T21:06:47.190871Z",
     "start_time": "2021-02-14T21:06:47.187334Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:   2, loss: 2.312, acc: 96.4\n"
     ]
    }
   ],
   "source": [
    "ep = 2\n",
    "l = 2.31231213123\n",
    "acc = 96.4\n",
    "print(col_formater.format(ep,l,acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Controlling begining and end of strings: Placeholders for tables\n",
    "\n",
    "\n",
    "We can print strings in a table like format where we can control spaces with a very nice trick. We can simply use `{k:len_k}` to specify the starting position and the length of the column.\n",
    "\n",
    "```\n",
    "Index       Colum1       Column2\n",
    "value_c0_1  value_c1_2   value_c2_2\n",
    "value_c0_1  value_c1_2   value_c2_2\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-14T21:25:38.450754Z",
     "start_time": "2021-02-14T21:25:38.443939Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index Column2 Column1\n",
      "   12     1     5\n",
      "  324     4     2\n",
      "   23     9    -2\n"
     ]
    }
   ],
   "source": [
    "c0,c1,c2 = [12,324,23], [1,4,9], [5,2,-2]\n",
    "len_bin = 5\n",
    "col_formater = \"{0:len_bin} {1:len_bin} {2:len_bin}\".replace(\"len_bin\", str(len_bin))\n",
    "print(col_formater.format(\"Index\", \"Column2\", \"Column1\"))\n",
    "#print((\"-\"*(len_bin-1)+\" \")*3)\n",
    "for c0_k,c1_k,c2_k in zip(c0,c1,c2):\n",
    "    print(col_formater.format(c0_k,c1_k,c2_k))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "##  Features for words\n",
    "\n",
    "- From words to features.\n",
    "   - Dictionary based approach\n",
    "   \n",
    "- Tokenize a string\n",
    "- Encode each token as a one hot vector\n",
    "- Add all one hot vectors and generate a single sparse vector encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/david/anaconda/envs/py36/lib/python3.6/site-packages/sklearn/utils/deprecation.py:144: FutureWarning: The sklearn.datasets.base module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.datasets. Anything that cannot be imported from sklearn.datasets is now part of the private API.\n",
      "  warnings.warn(message, FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "import sklearn\n",
    "from sklearn import datasets\n",
    "X = sklearn.datasets.fetch_20newsgroups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From: lerxst@wam.umd.edu (where's my thing)\n",
      "Subject: WHAT car is this!?\n",
      "Nntp-Posting-Host: rac3.wam.umd.edu\n",
      "Organization: University of Maryland, College Park\n",
      "Lines: 15\n",
      "\n",
      " I was wondering if anyone out there could enlighten me on this car I saw\n",
      "the other day. It was a 2-door sports car, looked to be from the late 60s/\n",
      "early 70s. It was called a Bricklin. The doors were really small. In addition,\n",
      "the front bumper was separate from the rest of the body. This is \n",
      "all I know. If anyone can tellme a model name, engine specs, years\n",
      "of production, where this car is made, history, or whatever info you\n",
      "have on this funky looking car, please e-mail.\n",
      "\n",
      "Thanks,\n",
      "- IL\n",
      "   ---- brought to you by your neighborhood Lerxst ----\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "7\n"
     ]
    }
   ],
   "source": [
    "print(X[\"data\"][0]),\n",
    "print(X[\"target\"][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X[\"data\"], \n",
    "                                                                            X[\"target\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 11314/11314 [00:39<00:00, 285.08it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 38.7 s, sys: 696 ms, total: 39.4 s\n",
      "Wall time: 39.7 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "vocabulary = {}\n",
    "i = 0\n",
    "for x in tqdm(X[\"data\"]):\n",
    "    words = nltk.word_tokenize(x)\n",
    "    for w in words:\n",
    "        w = w.lower()\n",
    "        if w not in vocabulary:\n",
    "            vocabulary[w] = i\n",
    "            i = i+1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "187171"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocabulary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Doing this automatically"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "                dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "                lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "                ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "                strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                tokenizer=None, vocabulary=None)"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vectorizer = sklearn.feature_extraction.text.CountVectorizer()\n",
    "count_vectorizer.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "107015"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(count_vectorizer.vocabulary_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_encoded = count_vectorizer.transform(X_train)\n",
    "X_test_encoded = count_vectorizer.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<8485x107015 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 1337844 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_encoded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "scipy.sparse.csr.csr_matrix"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(X_train_encoded[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   310,   3054,   3457,   3509,   3838,   4177,   5379,   6748,\n",
       "         9123,  15337,  16979,  21722,  21753,  22098,  22211,  22611,\n",
       "        22945,  23357,  23723,  25731,  26506,  28519,  32910,  33251,\n",
       "        33292,  36847,  37233,  39881,  41110,  43857,  45573,  45644,\n",
       "        46263,  46810,  46887,  48648,  48987,  50804,  51084,  51167,\n",
       "        51323,  52027,  52730,  54745,  56436,  56627,  60149,  60314,\n",
       "        60425,  60647,  61335,  62450,  64725,  65054,  65492,  65645,\n",
       "        66712,  71198,  71756,  73158,  73567,  73577,  73883,  73987,\n",
       "        74453,  74745,  75040,  77990,  79051,  79351,  81712,  82884,\n",
       "        84269,  85028,  86761,  86920,  87191,  87341,  89261,  89490,\n",
       "        89955,  91705,  94378,  94392,  94705,  95253,  97475,  98610,\n",
       "       101094, 101674, 102345, 102593, 102845, 103253, 103388, 103391],\n",
       "      dtype=int32)"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_encoded[0].nonzero()[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/david/anaconda/envs/py36/lib/python3.6/site-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=0.1, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
       "                   multi_class='auto', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = sklearn.linear_model.LogisticRegression(C=0.1)\n",
    "m.fit(X_train_encoded, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.9990571596935769, 0.8670908448214917)"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.score(X_train_encoded, y_train), m.score(X_test_encoded, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}