01-intro/homework.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Homework\n",
    "\n",
    "The goal of this homework is to train a simple model for predicting the duration of a ride - similar to what we did in this module.\n",
    "I basically followed the process in the optional lecture [here](https://www.youtube.com/watch?v=iRunifGSHFc&list=PL3MmuxUbc_hIUISrluw_A7wDSmfOhErJK&index=5&pp=iAQB). There are certain things I could approached differently but they will skew the provided answers."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Used packages__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from pathlib import Path\n",
    "from typing import Dict, List"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = \"../data/raw_data\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Download Data__. I added it to central locaiton avoid litering data all over my space.\n",
    "We'll use [the same NYC taxi dataset](https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page),\n",
    "but instead of \"**Green** Taxi Trip Records\", we'll use \"**Yellow** Taxi Trip Records\". To down the data for January and February 2022 use\n",
    "\n",
    "```bash\n",
    "! wget https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2022-01.parquet -O raw-train.parquet\n",
    "! wget https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2022-01.parquet -O raw-test.parquet\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python 3.9.16\n"
     ]
    }
   ],
   "source": [
    "!python -V"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.feature_extraction import DictVectorizer\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.metrics import mean_squared_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_data(path: Path, file_name: str) -> pd.DataFrame:\n",
    "    \"\"\"Load parquet onjext from file into a DataFrame.\n",
    "\n",
    "    Args:\n",
    "        path (Path): The base path of the data\n",
    "        file_name (str): The parquet data file name\n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: DataFrame of  raw data\n",
    "    \"\"\"\n",
    " \n",
    "    return pd.read_parquet(Path(data_dir) / f\"{file_name}.parquet\")\n",
    "\n",
    "def add_trip_duration(\n",
    "    df: pd.DataFrame, pick_up_time: str=\"tpep_pickup_datetime\", drop_off_time: str =\"tpep_dropoff_datetime\"\n",
    ") -> pd.DataFrame:\n",
    "    \"\"\"Adds the column `duration` to the DataFrame.\n",
    "\n",
    "    The column duration is derived from the pickup and drop off time stamps.\n",
    "\n",
    "    Args:\n",
    "        df (pd.DataFrame): Raw data of the taxi trip data\n",
    "        pick_up_time (str, optional): The pickup time. Defaults to \"tpep_pickup_datetime\".\n",
    "        drop_off_time (str, optional): Dropoff time. Defaults to \"tpep_dropoff_datetime\".\n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: The taxi data with duration added\n",
    "    \"\"\"\n",
    "    df[pick_up_time] = pd.to_datetime(df[pick_up_time])\n",
    "    df[drop_off_time] = pd.to_datetime(df[drop_off_time])\n",
    "    df[\"duration\"] = df[drop_off_time] - df[pick_up_time] \n",
    "    df.duration = df.duration.apply(lambda td: td.total_seconds() / 60)\n",
    "\n",
    "    return df\n",
    "\n",
    "\n",
    "def remove_outliers(df: pd.DataFrame, strategy: Dict[str, Dict[str, float]])-> pd.DataFrame:\n",
    "    \"\"\"Removes outliers defined as a strategy.\n",
    "\n",
    "    The strategy is defined as the a dict with columns as keys and the key is another dict\n",
    "    with keys min and max whose values depict the cutoff of the outliers\n",
    "\n",
    "    Args:\n",
    "        df (pd.DataFrame): DataFrame with outliers present.\n",
    "        strategy (Dict[str, Dict[str, float]]): The strategy to remove outliers\n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: The processed data with ourliers removed.\n",
    "    \"\"\"\n",
    "    # mask = pd.Series(df.shape[0]*[True])\n",
    "    \n",
    "    for column, outlier in strategy.items():\n",
    "        # TODO: Make the following mask to only run on series and filter outside the loop\n",
    "        # print(column, outlier)\n",
    "        mask = (\n",
    "            (df[column] >= outlier.get(\"min\", df[column].min())) \n",
    "            & (df[column] <= outlier.get(\"max\", df[column].max()))\n",
    "            ) \n",
    "        # print(mask)\n",
    "        df = df[mask]\n",
    "\n",
    "    return df\n",
    "\n",
    "def preprocess_taxi_data(df: pd.DataFrame, pickup_dropoff: Dict[str, str],  strategy: Dict[str, Dict[str, float]], categorical_features: List[str])-> pd.DataFrame:\n",
    "    \"\"\"Adds duration data using `add_trip_duration` and removes outliers using `remove_outliers.`\n",
    "\n",
    "    Args:\n",
    "        df (pd.DataFrame): DataFrame with outliers present.\n",
    "        strategy (Dict[str, Dict[str, float]]): The strategy to remove outliers\n",
    "        categorical_features (List[str]): List of categorical features to pass to categoricla feature processing\n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: The processed data with ourliers removed.\n",
    "    \"\"\"\n",
    "    if not \"duration\" in df.columns:\n",
    "        df = add_trip_duration(df=df, pick_up_time=pickup_dropoff[\"pickup\"], drop_off_time=pickup_dropoff[\"dropoff\"])\n",
    "    df = remove_outliers(df=df, strategy=strategy)\n",
    "    df = categorial_feature_prepocessing(df, categorical_features=categorical_features)\n",
    "    \n",
    "    return df\n",
    "\n",
    "def categorial_feature_prepocessing(df: pd.DataFrame, categorical_features: List[str]) -> pd.DataFrame:\n",
    "    \"\"\"Preprocess categorical features.\n",
    "\n",
    "    Here we simply preprocess them by casting them as strings.\n",
    "\n",
    "    Args:\n",
    "        df (pd.DataFrame): Data with both numerical and categorical columns\n",
    "        categorical_features (List[str]): List of ategorical feature \n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: Process dataframe\n",
    "    \"\"\"\n",
    "    df[categorical_features] = df[categorical_features].astype(str)\n",
    "\n",
    "    return df \n",
    "\n",
    "def train_model(df_train:pd.DataFrame, df_test:pd.DataFrame, categorial_features: List[str], target: str = \"duration\"):\n",
    "    \"\"\"Train a Linear Regression and calculate the RMSE on the validation dataframe\n",
    "\n",
    "    Args:\n",
    "        df_train (pd.DataFrame): Training data\n",
    "        df_test (pd.DataFrame): Test data\n",
    "        categorial_features (List[str]): List of categorical features\n",
    "        target (str, optional): Target column. Defaults to \"duration\".\n",
    "\n",
    "    Returns:\n",
    "        Dict[str, float]: Dict of mse for train and test\n",
    "    \"\"\" \n",
    "    \n",
    "    dv = DictVectorizer()   \n",
    "    train_dicts = df_train[categorial_features].to_dict(orient='records')\n",
    "    test_dicts = df_test[categorial_features].to_dict(orient='records')\n",
    "    \n",
    "    X_train = dv.fit_transform(train_dicts)\n",
    "    X_test = dv.transform(test_dicts)\n",
    "    \n",
    "    y_train = df_train[target].values\n",
    "    y_test = df_test[target].values\n",
    "\n",
    "    lr = LinearRegression()\n",
    "    lr.fit(X_train, y_train)\n",
    "\n",
    "    y_pred_train = lr.predict(X_train)\n",
    "    y_pred_test = lr.predict(X_test)\n",
    "\n",
    "\n",
    "    mse = {\n",
    "        \"train-mse\": mean_squared_error(y_train, y_pred_train, squared=False),\n",
    "        \"test-mse\": mean_squared_error(y_test, y_pred_test, squared=False)\n",
    "        }\n",
    "    \n",
    "    return dv, mse "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>VendorID</th>\n",
       "      <th>tpep_pickup_datetime</th>\n",
       "      <th>tpep_dropoff_datetime</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>trip_distance</th>\n",
       "      <th>RatecodeID</th>\n",
       "      <th>store_and_fwd_flag</th>\n",
       "      <th>PULocationID</th>\n",
       "      <th>DOLocationID</th>\n",
       "      <th>payment_type</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>extra</th>\n",
       "      <th>mta_tax</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>improvement_surcharge</th>\n",
       "      <th>total_amount</th>\n",
       "      <th>congestion_surcharge</th>\n",
       "      <th>airport_fee</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2022-01-01 00:35:40</td>\n",
       "      <td>2022-01-01 00:53:29</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.8</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>142</td>\n",
       "      <td>236</td>\n",
       "      <td>1</td>\n",
       "      <td>14.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>3.65</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>21.95</td>\n",
       "      <td>2.5</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2022-01-01 00:33:43</td>\n",
       "      <td>2022-01-01 00:42:07</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>236</td>\n",
       "      <td>42</td>\n",
       "      <td>1</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>4.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>13.30</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   VendorID tpep_pickup_datetime tpep_dropoff_datetime  passenger_count   \n",
       "0         1  2022-01-01 00:35:40   2022-01-01 00:53:29              2.0  \\\n",
       "1         1  2022-01-01 00:33:43   2022-01-01 00:42:07              1.0   \n",
       "\n",
       "   trip_distance  RatecodeID store_and_fwd_flag  PULocationID  DOLocationID   \n",
       "0            3.8         1.0                  N           142           236  \\\n",
       "1            2.1         1.0                  N           236            42   \n",
       "\n",
       "   payment_type  fare_amount  extra  mta_tax  tip_amount  tolls_amount   \n",
       "0             1         14.5    3.0      0.5        3.65           0.0  \\\n",
       "1             1          8.0    0.5      0.5        4.00           0.0   \n",
       "\n",
       "   improvement_surcharge  total_amount  congestion_surcharge  airport_fee  \n",
       "0                    0.3         21.95                   2.5          0.0  \n",
       "1                    0.3         13.30                   0.0          0.0  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_train = load_data(Path(data_dir), file_name=\"yellow_tripdata_2022-01\") \n",
    "raw_train.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2463931, 19)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_train.shape"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q1. Downloading the data\n",
    "\n",
    "Read the data for January. How many columns are there?\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 19 columns in data for january\n"
     ]
    }
   ],
   "source": [
    "print(f'There are {len(raw_train.columns)} columns in data for january')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q2. Computing duration\n",
    "\n",
    "Now let's compute the `duration` variable. It should contain the duration of a ride in minutes. \n",
    "\n",
    "What's the standard deviation of the trips duration in January?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_train_with_duration = add_trip_duration(raw_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>VendorID</th>\n",
       "      <th>tpep_pickup_datetime</th>\n",
       "      <th>tpep_dropoff_datetime</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>trip_distance</th>\n",
       "      <th>RatecodeID</th>\n",
       "      <th>store_and_fwd_flag</th>\n",
       "      <th>PULocationID</th>\n",
       "      <th>DOLocationID</th>\n",
       "      <th>payment_type</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>extra</th>\n",
       "      <th>mta_tax</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>improvement_surcharge</th>\n",
       "      <th>total_amount</th>\n",
       "      <th>congestion_surcharge</th>\n",
       "      <th>airport_fee</th>\n",
       "      <th>duration</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2022-01-01 00:35:40</td>\n",
       "      <td>2022-01-01 00:53:29</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.8</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>142</td>\n",
       "      <td>236</td>\n",
       "      <td>1</td>\n",
       "      <td>14.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>3.65</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>21.95</td>\n",
       "      <td>2.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17.816667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2022-01-01 00:33:43</td>\n",
       "      <td>2022-01-01 00:42:07</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>236</td>\n",
       "      <td>42</td>\n",
       "      <td>1</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>4.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>13.30</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.400000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   VendorID tpep_pickup_datetime tpep_dropoff_datetime  passenger_count   \n",
       "0         1  2022-01-01 00:35:40   2022-01-01 00:53:29              2.0  \\\n",
       "1         1  2022-01-01 00:33:43   2022-01-01 00:42:07              1.0   \n",
       "\n",
       "   trip_distance  RatecodeID store_and_fwd_flag  PULocationID  DOLocationID   \n",
       "0            3.8         1.0                  N           142           236  \\\n",
       "1            2.1         1.0                  N           236            42   \n",
       "\n",
       "   payment_type  fare_amount  extra  mta_tax  tip_amount  tolls_amount   \n",
       "0             1         14.5    3.0      0.5        3.65           0.0  \\\n",
       "1             1          8.0    0.5      0.5        4.00           0.0   \n",
       "\n",
       "   improvement_surcharge  total_amount  congestion_surcharge  airport_fee   \n",
       "0                    0.3         21.95                   2.5          0.0  \\\n",
       "1                    0.3         13.30                   0.0          0.0   \n",
       "\n",
       "    duration  \n",
       "0  17.816667  \n",
       "1   8.400000  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_train_with_duration.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The standard deviation of trips in January is 46.4453\n"
     ]
    }
   ],
   "source": [
    "print(f\"The standard deviation of trips in January is {raw_train_with_duration['duration'].std():.4f}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q3. Dropping outliers\n",
    "\n",
    "Next, we need to check the distribution of the `duration` variable. There are some outliers. Let's remove them and keep only the records where the duration was between 1 and 60 minutes (inclusive).\n",
    "\n",
    "What fraction of the records left after you dropped the outliers?\n",
    "\n",
    "* 90%\n",
    "* 92%\n",
    "* 95%\n",
    "* 98%\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocess_train = remove_outliers(raw_train_with_duration, strategy={\"duration\":{\"min\": 1, \"max\": 60}})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>VendorID</th>\n",
       "      <th>tpep_pickup_datetime</th>\n",
       "      <th>tpep_dropoff_datetime</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>trip_distance</th>\n",
       "      <th>RatecodeID</th>\n",
       "      <th>store_and_fwd_flag</th>\n",
       "      <th>PULocationID</th>\n",
       "      <th>DOLocationID</th>\n",
       "      <th>payment_type</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>extra</th>\n",
       "      <th>mta_tax</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>improvement_surcharge</th>\n",
       "      <th>total_amount</th>\n",
       "      <th>congestion_surcharge</th>\n",
       "      <th>airport_fee</th>\n",
       "      <th>duration</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2022-01-01 00:35:40</td>\n",
       "      <td>2022-01-01 00:53:29</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.8</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>142</td>\n",
       "      <td>236</td>\n",
       "      <td>1</td>\n",
       "      <td>14.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>3.65</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>21.95</td>\n",
       "      <td>2.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17.816667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2022-01-01 00:33:43</td>\n",
       "      <td>2022-01-01 00:42:07</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>236</td>\n",
       "      <td>42</td>\n",
       "      <td>1</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>4.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>13.30</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.400000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   VendorID tpep_pickup_datetime tpep_dropoff_datetime  passenger_count   \n",
       "0         1  2022-01-01 00:35:40   2022-01-01 00:53:29              2.0  \\\n",
       "1         1  2022-01-01 00:33:43   2022-01-01 00:42:07              1.0   \n",
       "\n",
       "   trip_distance  RatecodeID store_and_fwd_flag  PULocationID  DOLocationID   \n",
       "0            3.8         1.0                  N           142           236  \\\n",
       "1            2.1         1.0                  N           236            42   \n",
       "\n",
       "   payment_type  fare_amount  extra  mta_tax  tip_amount  tolls_amount   \n",
       "0             1         14.5    3.0      0.5        3.65           0.0  \\\n",
       "1             1          8.0    0.5      0.5        4.00           0.0   \n",
       "\n",
       "   improvement_surcharge  total_amount  congestion_surcharge  airport_fee   \n",
       "0                    0.3         21.95                   2.5          0.0  \\\n",
       "1                    0.3         13.30                   0.0          0.0   \n",
       "\n",
       "    duration  \n",
       "0  17.816667  \n",
       "1   8.400000  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocess_train.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2463931, 20)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_train_with_duration.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Outlier removed is:1.00\n"
     ]
    }
   ],
   "source": [
    "print(f\"Outlier removed is:{ (preprocess_train.shape[1]/raw_train_with_duration.shape[1]):.2f}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fraction of records left after drop outliers is 98%"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q4. One-hot encoding\n",
    "\n",
    "Let's apply one-hot encoding to the pickup and dropoff location IDs. We'll use only these two features for our model. \n",
    "\n",
    "* Turn the dataframe into a list of dictionaries\n",
    "* Fit a dictionary vectorizer \n",
    "* Get a feature matrix from it\n",
    "\n",
    "What's the dimensionality of this matrix (number of columns)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical = ['PULocationID', 'DOLocationID']\n",
    "TARGET = 'duration'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PULocationID</th>\n",
       "      <th>DOLocationID</th>\n",
       "      <th>duration</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>142</td>\n",
       "      <td>236</td>\n",
       "      <td>17.816667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>236</td>\n",
       "      <td>42</td>\n",
       "      <td>8.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>166</td>\n",
       "      <td>166</td>\n",
       "      <td>8.966667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>114</td>\n",
       "      <td>68</td>\n",
       "      <td>10.033333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>68</td>\n",
       "      <td>163</td>\n",
       "      <td>37.533333</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  PULocationID DOLocationID   duration\n",
       "0          142          236  17.816667\n",
       "1          236           42   8.400000\n",
       "2          166          166   8.966667\n",
       "3          114           68  10.033333\n",
       "4           68          163  37.533333"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data = categorial_feature_prepocessing(preprocess_train[categorical + [TARGET]].copy(), categorical_features=categorical)\n",
    "train_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>VendorID</th>\n",
       "      <th>tpep_pickup_datetime</th>\n",
       "      <th>tpep_dropoff_datetime</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>trip_distance</th>\n",
       "      <th>RatecodeID</th>\n",
       "      <th>store_and_fwd_flag</th>\n",
       "      <th>PULocationID</th>\n",
       "      <th>DOLocationID</th>\n",
       "      <th>payment_type</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>extra</th>\n",
       "      <th>mta_tax</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>improvement_surcharge</th>\n",
       "      <th>total_amount</th>\n",
       "      <th>congestion_surcharge</th>\n",
       "      <th>airport_fee</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2022-02-01 00:06:58</td>\n",
       "      <td>2022-02-01 00:19:24</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.40</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>138</td>\n",
       "      <td>252</td>\n",
       "      <td>1</td>\n",
       "      <td>17.0</td>\n",
       "      <td>1.75</td>\n",
       "      <td>0.5</td>\n",
       "      <td>3.90</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.3</td>\n",
       "      <td>23.45</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2022-02-01 00:38:22</td>\n",
       "      <td>2022-02-01 00:55:55</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.40</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>138</td>\n",
       "      <td>41</td>\n",
       "      <td>2</td>\n",
       "      <td>21.0</td>\n",
       "      <td>1.75</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.55</td>\n",
       "      <td>0.3</td>\n",
       "      <td>30.10</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>2022-02-01 00:03:20</td>\n",
       "      <td>2022-02-01 00:26:59</td>\n",
       "      <td>1.0</td>\n",
       "      <td>12.50</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>138</td>\n",
       "      <td>200</td>\n",
       "      <td>2</td>\n",
       "      <td>35.5</td>\n",
       "      <td>1.75</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.55</td>\n",
       "      <td>0.3</td>\n",
       "      <td>44.60</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>2022-02-01 00:08:00</td>\n",
       "      <td>2022-02-01 00:28:05</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9.88</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>239</td>\n",
       "      <td>200</td>\n",
       "      <td>2</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.00</td>\n",
       "      <td>0.3</td>\n",
       "      <td>34.80</td>\n",
       "      <td>2.5</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>2022-02-01 00:06:48</td>\n",
       "      <td>2022-02-01 00:33:07</td>\n",
       "      <td>1.0</td>\n",
       "      <td>12.16</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>138</td>\n",
       "      <td>125</td>\n",
       "      <td>1</td>\n",
       "      <td>35.5</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.5</td>\n",
       "      <td>8.11</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.3</td>\n",
       "      <td>48.66</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1.25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   VendorID tpep_pickup_datetime tpep_dropoff_datetime  passenger_count   \n",
       "0         1  2022-02-01 00:06:58   2022-02-01 00:19:24              1.0  \\\n",
       "1         1  2022-02-01 00:38:22   2022-02-01 00:55:55              1.0   \n",
       "2         1  2022-02-01 00:03:20   2022-02-01 00:26:59              1.0   \n",
       "3         2  2022-02-01 00:08:00   2022-02-01 00:28:05              1.0   \n",
       "4         2  2022-02-01 00:06:48   2022-02-01 00:33:07              1.0   \n",
       "\n",
       "   trip_distance  RatecodeID store_and_fwd_flag  PULocationID  DOLocationID   \n",
       "0           5.40         1.0                  N           138           252  \\\n",
       "1           6.40         1.0                  N           138            41   \n",
       "2          12.50         1.0                  N           138           200   \n",
       "3           9.88         1.0                  N           239           200   \n",
       "4          12.16         1.0                  N           138           125   \n",
       "\n",
       "   payment_type  fare_amount  extra  mta_tax  tip_amount  tolls_amount   \n",
       "0             1         17.0   1.75      0.5        3.90          0.00  \\\n",
       "1             2         21.0   1.75      0.5        0.00          6.55   \n",
       "2             2         35.5   1.75      0.5        0.00          6.55   \n",
       "3             2         28.0   0.50      0.5        0.00          3.00   \n",
       "4             1         35.5   0.50      0.5        8.11          0.00   \n",
       "\n",
       "   improvement_surcharge  total_amount  congestion_surcharge  airport_fee  \n",
       "0                    0.3         23.45                   0.0         1.25  \n",
       "1                    0.3         30.10                   0.0         1.25  \n",
       "2                    0.3         44.60                   0.0         1.25  \n",
       "3                    0.3         34.80                   2.5         0.00  \n",
       "4                    0.3         48.66                   2.5         1.25  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_test = load_data(Path(data_dir), file_name=\"yellow_tripdata_2022-02\") \n",
    "raw_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_185176/102634147.py:94: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df[categorical_features] = df[categorical_features].astype(str)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>VendorID</th>\n",
       "      <th>tpep_pickup_datetime</th>\n",
       "      <th>tpep_dropoff_datetime</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>trip_distance</th>\n",
       "      <th>RatecodeID</th>\n",
       "      <th>store_and_fwd_flag</th>\n",
       "      <th>PULocationID</th>\n",
       "      <th>DOLocationID</th>\n",
       "      <th>payment_type</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>extra</th>\n",
       "      <th>mta_tax</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>improvement_surcharge</th>\n",
       "      <th>total_amount</th>\n",
       "      <th>congestion_surcharge</th>\n",
       "      <th>airport_fee</th>\n",
       "      <th>duration</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2022-02-01 00:06:58</td>\n",
       "      <td>2022-02-01 00:19:24</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>138</td>\n",
       "      <td>252</td>\n",
       "      <td>1</td>\n",
       "      <td>17.0</td>\n",
       "      <td>1.75</td>\n",
       "      <td>0.5</td>\n",
       "      <td>3.9</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.3</td>\n",
       "      <td>23.45</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.25</td>\n",
       "      <td>12.433333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2022-02-01 00:38:22</td>\n",
       "      <td>2022-02-01 00:55:55</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>138</td>\n",
       "      <td>41</td>\n",
       "      <td>2</td>\n",
       "      <td>21.0</td>\n",
       "      <td>1.75</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.55</td>\n",
       "      <td>0.3</td>\n",
       "      <td>30.10</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.25</td>\n",
       "      <td>17.550000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   VendorID tpep_pickup_datetime tpep_dropoff_datetime  passenger_count   \n",
       "0         1  2022-02-01 00:06:58   2022-02-01 00:19:24              1.0  \\\n",
       "1         1  2022-02-01 00:38:22   2022-02-01 00:55:55              1.0   \n",
       "\n",
       "   trip_distance  RatecodeID store_and_fwd_flag PULocationID DOLocationID   \n",
       "0            5.4         1.0                  N          138          252  \\\n",
       "1            6.4         1.0                  N          138           41   \n",
       "\n",
       "   payment_type  fare_amount  extra  mta_tax  tip_amount  tolls_amount   \n",
       "0             1         17.0   1.75      0.5         3.9          0.00  \\\n",
       "1             2         21.0   1.75      0.5         0.0          6.55   \n",
       "\n",
       "   improvement_surcharge  total_amount  congestion_surcharge  airport_fee   \n",
       "0                    0.3         23.45                   0.0         1.25  \\\n",
       "1                    0.3         30.10                   0.0         1.25   \n",
       "\n",
       "    duration  \n",
       "0  12.433333  \n",
       "1  17.550000  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data = preprocess_taxi_data(\n",
    "    df = raw_test, \n",
    "    pickup_dropoff = {\"pickup\": \"tpep_pickup_datetime\", \"dropoff\": \"tpep_dropoff_datetime\"},\n",
    "    strategy={\"duration\":{\"min\": 1, \"max\": 60}}, categorical_features=categorical\n",
    "    )\n",
    "test_data.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "dv, mse = train_model(train_data, test_data, categorial_features=categorical, target=TARGET)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "515"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dv.feature_names_)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q5. Training a model\n",
    "\n",
    "Now let's use the feature matrix from the previous step to train a model. \n",
    "\n",
    "* Train a plain linear regression model with default parameters \n",
    "* Calculate the RMSE of the model on the training data\n",
    "\n",
    "What's the RMSE on train?\n",
    "\n",
    "* 6.99\n",
    "* 11.99\n",
    "* 16.99\n",
    "* 21.99"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train mse is : 6.99\n"
     ]
    }
   ],
   "source": [
    "print(f'Train mse is : {mse[\"train-mse\"]:.2f}')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q6. Evaluating the model\n",
    "\n",
    "Now let's apply this model to the validation dataset (February 2022). \n",
    "\n",
    "What's the RMSE on validation?\n",
    "\n",
    "* 7.79\n",
    "* 12.79\n",
    "* 17.79\n",
    "* 22.79"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train mse is : 7.79\n"
     ]
    }
   ],
   "source": [
    "print(f'Train mse is : {mse[\"test-mse\"]:.2f}')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}