CLA08-NB01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring Ensemble Methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this assignment, we will explore the use of boosting. We will use the pre-implemented gradient boosted trees in Turi Create. You will:\n",
    "\n",
    "* Use SFrames to do some feature engineering.\n",
    "* Train a boosted ensemble of decision-trees (gradient boosted trees) on the LendingClub dataset.\n",
    "* Predict whether a loan will default along with prediction probabilities (on a validation set).\n",
    "* Evaluate the trained model and compare it with a baseline.\n",
    "* Find the most positive and negative loans using the learned model.\n",
    "* Explore how the number of trees influences classification performance.\n",
    "\n",
    "Let's get started!\n",
    "\n",
    "## Fire up Turi Create"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import turicreate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load LendingClub dataset\n",
    "\n",
    "We will be using the [LendingClub](https://www.lendingclub.com/) data. As discussed earlier, the [LendingClub](https://www.lendingclub.com/) is a peer-to-peer leading company that directly connects borrowers and potential lenders/investors. \n",
    "\n",
    "Just like we did in previous assignments, we will build a classification model to predict whether or not a loan provided by lending club is likely to default.\n",
    "\n",
    "Let us start by loading the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "loans = turicreate.SFrame('lending-club-data.sframe/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's quickly explore what the dataset looks like. First, let's print out the column names to see what features we have in this dataset. We have done this in previous assignments, so we won't belabor this here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['id',\n",
       " 'member_id',\n",
       " 'loan_amnt',\n",
       " 'funded_amnt',\n",
       " 'funded_amnt_inv',\n",
       " 'term',\n",
       " 'int_rate',\n",
       " 'installment',\n",
       " 'grade',\n",
       " 'sub_grade',\n",
       " 'emp_title',\n",
       " 'emp_length',\n",
       " 'home_ownership',\n",
       " 'annual_inc',\n",
       " 'is_inc_v',\n",
       " 'issue_d',\n",
       " 'loan_status',\n",
       " 'pymnt_plan',\n",
       " 'url',\n",
       " 'desc',\n",
       " 'purpose',\n",
       " 'title',\n",
       " 'zip_code',\n",
       " 'addr_state',\n",
       " 'dti',\n",
       " 'delinq_2yrs',\n",
       " 'earliest_cr_line',\n",
       " 'inq_last_6mths',\n",
       " 'mths_since_last_delinq',\n",
       " 'mths_since_last_record',\n",
       " 'open_acc',\n",
       " 'pub_rec',\n",
       " 'revol_bal',\n",
       " 'revol_util',\n",
       " 'total_acc',\n",
       " 'initial_list_status',\n",
       " 'out_prncp',\n",
       " 'out_prncp_inv',\n",
       " 'total_pymnt',\n",
       " 'total_pymnt_inv',\n",
       " 'total_rec_prncp',\n",
       " 'total_rec_int',\n",
       " 'total_rec_late_fee',\n",
       " 'recoveries',\n",
       " 'collection_recovery_fee',\n",
       " 'last_pymnt_d',\n",
       " 'last_pymnt_amnt',\n",
       " 'next_pymnt_d',\n",
       " 'last_credit_pull_d',\n",
       " 'collections_12_mths_ex_med',\n",
       " 'mths_since_last_major_derog',\n",
       " 'policy_code',\n",
       " 'not_compliant',\n",
       " 'status',\n",
       " 'inactive_loans',\n",
       " 'bad_loans',\n",
       " 'emp_length_num',\n",
       " 'grade_num',\n",
       " 'sub_grade_num',\n",
       " 'delinq_2yrs_zero',\n",
       " 'pub_rec_zero',\n",
       " 'collections_12_mths_zero',\n",
       " 'short_emp',\n",
       " 'payment_inc_ratio',\n",
       " 'final_d',\n",
       " 'last_delinq_none',\n",
       " 'last_record_none',\n",
       " 'last_major_derog_none']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loans.column_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modifying the target column\n",
    "\n",
    "The target column (label column) of the dataset that we are interested in is called `bad_loans`. In this column **1** means a risky (bad) loan **0** means a safe  loan.\n",
    "\n",
    "As in past assignments, in order to make this more intuitive and consistent with the lectures, we reassign the target to be:\n",
    "* **+1** as a safe  loan, \n",
    "* **-1** as a risky (bad) loan. \n",
    "\n",
    "We put this in a new column called `safe_loans`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)\n",
    "loans = loans.remove_column('bad_loans')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting features\n",
    "\n",
    "In this assignment, we will be using a subset of features (categorical and numeric). The features we will be using are **described in the code comments** below. If you are a finance geek, the [LendingClub](https://www.lendingclub.com/) website has a lot more details about these features.\n",
    "\n",
    "The features we will be using are described in the code comments below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = 'safe_loans'\n",
    "features = ['grade',                     # grade of the loan (categorical)\n",
    "            'sub_grade_num',             # sub-grade of the loan as a number from 0 to 1\n",
    "            'short_emp',                 # one year or less of employment\n",
    "            'emp_length_num',            # number of years of employment\n",
    "            'home_ownership',            # home_ownership status: own, mortgage or rent\n",
    "            'dti',                       # debt to income ratio\n",
    "            'purpose',                   # the purpose of the loan\n",
    "            'payment_inc_ratio',         # ratio of the monthly payment to income\n",
    "            'delinq_2yrs',               # number of delinquincies \n",
    "            'delinq_2yrs_zero',          # no delinquincies in last 2 years\n",
    "            'inq_last_6mths',            # number of creditor inquiries in last 6 months\n",
    "            'last_delinq_none',          # has borrower had a delinquincy\n",
    "            'last_major_derog_none',     # has borrower had 90 day or worse rating\n",
    "            'open_acc',                  # number of open credit accounts\n",
    "            'pub_rec',                   # number of derogatory public records\n",
    "            'pub_rec_zero',              # no derogatory public records\n",
    "            'revol_util',                # percent of available credit being used\n",
    "            'total_rec_late_fee',        # total late fees received to day\n",
    "            'int_rate',                  # interest rate of the loan\n",
    "            'total_rec_int',             # interest received to date\n",
    "            'annual_inc',                # annual income of borrower\n",
    "            'funded_amnt',               # amount committed to the loan\n",
    "            'funded_amnt_inv',           # amount committed by investors for the loan\n",
    "            'installment',               # monthly payment owed by the borrower\n",
    "           ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Skipping observations with missing values\n",
    "\n",
    "Recall from the lectures that one common approach to coping with missing values is to **skip** observations that contain missing values.\n",
    "\n",
    "We run the following code to do so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dropping 29 observations; keeping 122578 \n"
     ]
    }
   ],
   "source": [
    "loans, loans_with_na = loans[[target] + features].dropna_split()\n",
    "\n",
    "# Count the number of rows with missing data\n",
    "num_rows_with_na = loans_with_na.num_rows()\n",
    "num_rows = loans.num_rows()\n",
    "print('Dropping %s observations; keeping %s ' % (num_rows_with_na, num_rows))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fortunately, there are not too many missing values. We are retaining most of the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make sure the classes are balanced"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We saw in an earlier assignment that this dataset is also imbalanced. We will undersample the larger class (safe loans) in order to balance out our dataset. We used `seed=1` to make sure everyone gets the same results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage of safe loans                 : 0.5022471668494506\n",
      "Percentage of risky loans                : 0.4977528331505494\n",
      "Total number of loans in our new dataset : 46503\n"
     ]
    }
   ],
   "source": [
    "safe_loans_raw = loans[loans[target] == 1]\n",
    "risky_loans_raw = loans[loans[target] == -1]\n",
    "\n",
    "# Undersample the safe loans.\n",
    "percentage = len(risky_loans_raw)/float(len(safe_loans_raw))\n",
    "safe_loans = safe_loans_raw.sample(percentage, seed = 1)\n",
    "risky_loans = risky_loans_raw\n",
    "loans_data = risky_loans.append(safe_loans)\n",
    "\n",
    "print(\"Percentage of safe loans                 :\", len(safe_loans) / float(len(loans_data)))\n",
    "print(\"Percentage of risky loans                :\", len(risky_loans) / float(len(loans_data)))\n",
    "print(\"Total number of loans in our new dataset :\", len(loans_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Checkpoint:** You should now see that the dataset is balanced (approximately 50-50 safe vs risky loans)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** There are many approaches for dealing with imbalanced data, including some where we modify the learning algorithm. These approaches are beyond the scope of this course, but some of them are reviewed in this [paper](http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=5128907&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F69%2F5173046%2F05128907.pdf%3Farnumber%3D5128907 ). For this assignment, we use the simplest possible approach, where we subsample the overly represented class to get a more balanced dataset. In general, and especially when the data is highly imbalanced, we recommend using more advanced methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split data into training and validation sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split the data into training data and validation data. We used `seed=1` to make sure everyone gets the same results. We will use the validation data to help us select model parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data, validation_data = loans_data.random_split(.8, seed=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradient boosted tree classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient boosted trees are a powerful variant of boosting methods; they have been used to win many [Kaggle](https://www.kaggle.com/) competitions, and have been widely used in industry.  We will explore the predictive power of multiple decision trees as opposed to a single decision tree.\n",
    "\n",
    "**Additional reading:** If you are interested in gradient boosted trees, here is some additional reading material:\n",
    "* [Turi Create user guide](https://apple.github.io/turicreate/docs/api/generated/turicreate.boosted_trees_classifier.BoostedTreesClassifier.html)\n",
    "* [Advanced material on boosted trees](http://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf)\n",
    "\n",
    "\n",
    "We will now train models to predict `safe_loans` using the features above. In this section, we will experiment with training an ensemble of 5 trees. To cap the ensemble classifier at 5 trees, we call the function with **max_iterations=5** (recall that each iterations corresponds to adding a tree). We set `validation_set=None` to make sure everyone gets the same results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Boosted trees classifier:</pre>"
      ],
      "text/plain": [
       "Boosted trees classifier:"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>--------------------------------------------------------</pre>"
      ],
      "text/plain": [
       "--------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of examples          : 37219</pre>"
      ],
      "text/plain": [
       "Number of examples          : 37219"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of classes           : 2</pre>"
      ],
      "text/plain": [
       "Number of classes           : 2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of feature columns   : 24</pre>"
      ],
      "text/plain": [
       "Number of feature columns   : 24"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of unpacked features : 24</pre>"
      ],
      "text/plain": [
       "Number of unpacked features : 24"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+--------------+-------------------+-------------------+</pre>"
      ],
      "text/plain": [
       "+-----------+--------------+-------------------+-------------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| Iteration | Elapsed Time | Training Accuracy | Training Log Loss |</pre>"
      ],
      "text/plain": [
       "| Iteration | Elapsed Time | Training Accuracy | Training Log Loss |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+--------------+-------------------+-------------------+</pre>"
      ],
      "text/plain": [
       "+-----------+--------------+-------------------+-------------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 1         | 0.029014     | 0.657541          | 0.657139          |</pre>"
      ],
      "text/plain": [
       "| 1         | 0.029014     | 0.657541          | 0.657139          |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 2         | 0.060343     | 0.656976          | 0.636157          |</pre>"
      ],
      "text/plain": [
       "| 2         | 0.060343     | 0.656976          | 0.636157          |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 3         | 0.093539     | 0.664983          | 0.623206          |</pre>"
      ],
      "text/plain": [
       "| 3         | 0.093539     | 0.664983          | 0.623206          |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 4         | 0.124711     | 0.668476          | 0.613783          |</pre>"
      ],
      "text/plain": [
       "| 4         | 0.124711     | 0.668476          | 0.613783          |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 5         | 0.158634     | 0.673339          | 0.606229          |</pre>"
      ],
      "text/plain": [
       "| 5         | 0.158634     | 0.673339          | 0.606229          |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+--------------+-------------------+-------------------+</pre>"
      ],
      "text/plain": [
       "+-----------+--------------+-------------------+-------------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_5 = turicreate.boosted_trees_classifier.create(train_data, validation_set=None, \n",
    "        target = target, features = features, max_iterations = 5)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making predictions\n",
    "\n",
    "Just like we did in previous sections, let us consider a few positive and negative examples **from the validation set**. We will do the following:\n",
    "* Predict whether or not a loan is likely to default.\n",
    "* Predict the probability with which the loan is likely to default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">safe_loans</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">grade</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">sub_grade_num</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">short_emp</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">emp_length_num</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">home_ownership</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">dti</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">purpose</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">B</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MORTGAGE</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">29.44</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">credit_card</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">B</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12.19</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">credit_card</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">D</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">13.97</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MORTGAGE</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16.33</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">debt_consolidation</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">payment_inc_ratio</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">delinq_2yrs</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">delinq_2yrs_zero</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">inq_last_6mths</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">last_delinq_none</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">last_major_derog_none</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">open_acc</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6.30496</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">13.4952</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2.96736</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.90524</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">17</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">pub_rec</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">pub_rec_zero</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">revol_util</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">total_rec_late_fee</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">int_rate</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">total_rec_int</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">annual_inc</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">funded_amnt</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">funded_amnt_inv</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">93.9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">9.91</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">823.48</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">92000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">59.1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11.71</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1622.21</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">25000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8500</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">59.5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16.77</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">719.11</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">50004</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">62.1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8.9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">696.99</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">100000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5000</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">installment</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">483.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">281.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">123.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">158.77</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[4 rows x 25 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsafe_loans\tint\n",
       "\tgrade\tstr\n",
       "\tsub_grade_num\tfloat\n",
       "\tshort_emp\tint\n",
       "\temp_length_num\tint\n",
       "\thome_ownership\tstr\n",
       "\tdti\tfloat\n",
       "\tpurpose\tstr\n",
       "\tpayment_inc_ratio\tfloat\n",
       "\tdelinq_2yrs\tint\n",
       "\tdelinq_2yrs_zero\tint\n",
       "\tinq_last_6mths\tint\n",
       "\tlast_delinq_none\tint\n",
       "\tlast_major_derog_none\tint\n",
       "\topen_acc\tint\n",
       "\tpub_rec\tint\n",
       "\tpub_rec_zero\tint\n",
       "\trevol_util\tfloat\n",
       "\ttotal_rec_late_fee\tfloat\n",
       "\tint_rate\tfloat\n",
       "\ttotal_rec_int\tfloat\n",
       "\tannual_inc\tint\n",
       "\tfunded_amnt\tint\n",
       "\tfunded_amnt_inv\tint\n",
       "\tinstallment\tfloat\n",
       "\n",
       "Rows: 4\n",
       "\n",
       "Data:\n",
       "+------------+-------+---------------+-----------+----------------+----------------+\n",
       "| safe_loans | grade | sub_grade_num | short_emp | emp_length_num | home_ownership |\n",
       "+------------+-------+---------------+-----------+----------------+----------------+\n",
       "|     1      |   B   |      0.2      |     0     |       3        |    MORTGAGE    |\n",
       "|     1      |   B   |      0.6      |     1     |       1        |      RENT      |\n",
       "|     -1     |   D   |      0.4      |     0     |       3        |      RENT      |\n",
       "|     -1     |   A   |      1.0      |     0     |       11       |    MORTGAGE    |\n",
       "+------------+-------+---------------+-----------+----------------+----------------+\n",
       "+-------+--------------------+-------------------+-------------+------------------+\n",
       "|  dti  |      purpose       | payment_inc_ratio | delinq_2yrs | delinq_2yrs_zero |\n",
       "+-------+--------------------+-------------------+-------------+------------------+\n",
       "| 29.44 |    credit_card     |      6.30496      |      0      |        1         |\n",
       "| 12.19 |    credit_card     |      13.4952      |      0      |        1         |\n",
       "| 13.97 |       other        |      2.96736      |      3      |        0         |\n",
       "| 16.33 | debt_consolidation |      1.90524      |      0      |        1         |\n",
       "+-------+--------------------+-------------------+-------------+------------------+\n",
       "+----------------+------------------+-----------------------+----------+---------+\n",
       "| inq_last_6mths | last_delinq_none | last_major_derog_none | open_acc | pub_rec |\n",
       "+----------------+------------------+-----------------------+----------+---------+\n",
       "|       0        |        1         |           1           |    8     |    0    |\n",
       "|       0        |        1         |           1           |    8     |    0    |\n",
       "|       0        |        0         |           1           |    14    |    0    |\n",
       "|       0        |        1         |           1           |    17    |    0    |\n",
       "+----------------+------------------+-----------------------+----------+---------+\n",
       "+--------------+------------+--------------------+----------+-----+\n",
       "| pub_rec_zero | revol_util | total_rec_late_fee | int_rate | ... |\n",
       "+--------------+------------+--------------------+----------+-----+\n",
       "|      1       |    93.9    |        0.0         |   9.91   | ... |\n",
       "|      1       |    59.1    |        0.0         |  11.71   | ... |\n",
       "|      1       |    59.5    |        0.0         |  16.77   | ... |\n",
       "|      1       |    62.1    |        0.0         |   8.9    | ... |\n",
       "+--------------+------------+--------------------+----------+-----+\n",
       "[4 rows x 25 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select all positive and negative examples.\n",
    "validation_safe_loans = validation_data[validation_data[target] == 1]\n",
    "validation_risky_loans = validation_data[validation_data[target] == -1]\n",
    "\n",
    "# Select 2 examples from the validation set for positive & negative loans\n",
    "sample_validation_data_risky = validation_risky_loans[0:2]\n",
    "sample_validation_data_safe = validation_safe_loans[0:2]\n",
    "\n",
    "# Append the 4 examples into a single dataset\n",
    "sample_validation_data = sample_validation_data_safe.append(sample_validation_data_risky)\n",
    "sample_validation_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predicting on sample validation data\n",
    "\n",
    "For each row in the **sample_validation_data**, write code to make **model_5** predict whether or not the loan is classified as a **safe loan**.\n",
    "\n",
    "**Hint:** Use the `predict` method in `model_5` for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.75\n"
     ]
    }
   ],
   "source": [
    "incorrect = 0\n",
    "for i, it in enumerate(sample_validation_data):\n",
    "    prediction = model_5.predict(it)\n",
    "    #print(prediction, it)\n",
    "    if prediction[0] != it[target]:\n",
    "        incorrect+=1\n",
    "\n",
    "print(1 - (incorrect / len(sample_validation_data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** What percentage of the predictions on `sample_validation_data` did `model_5` get correct?\n",
    "\n",
    "### Prediction probabilities\n",
    "\n",
    "For each row in the **sample_validation_data**, what is the probability (according **model_5**) of a loan being classified as **safe**? \n",
    "\n",
    "**Hint:** Set `output_type='probability'` to make **probability** predictions using `model_5` on `sample_validation_data`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 [1] [0.7045905590057373]\n",
      "{'safe_loans': 1, 'grade': 'B', 'sub_grade_num': 0.2, 'short_emp': 0, 'emp_length_num': 3, 'home_ownership': 'MORTGAGE', 'dti': 29.44, 'purpose': 'credit_card', 'payment_inc_ratio': 6.30496, 'delinq_2yrs': 0, 'delinq_2yrs_zero': 1, 'inq_last_6mths': 0, 'last_delinq_none': 1, 'last_major_derog_none': 1, 'open_acc': 8, 'pub_rec': 0, 'pub_rec_zero': 1, 'revol_util': 93.9, 'total_rec_late_fee': 0.0, 'int_rate': 9.91, 'total_rec_int': 823.48, 'annual_inc': 92000, 'funded_amnt': 15000, 'funded_amnt_inv': 15000, 'installment': 483.38}\n",
      "1 [1] [0.5963408946990967]\n",
      "{'safe_loans': 1, 'grade': 'B', 'sub_grade_num': 0.6, 'short_emp': 1, 'emp_length_num': 1, 'home_ownership': 'RENT', 'dti': 12.19, 'purpose': 'credit_card', 'payment_inc_ratio': 13.4952, 'delinq_2yrs': 0, 'delinq_2yrs_zero': 1, 'inq_last_6mths': 0, 'last_delinq_none': 1, 'last_major_derog_none': 1, 'open_acc': 8, 'pub_rec': 0, 'pub_rec_zero': 1, 'revol_util': 59.1, 'total_rec_late_fee': 0.0, 'int_rate': 11.71, 'total_rec_int': 1622.21, 'annual_inc': 25000, 'funded_amnt': 8500, 'funded_amnt_inv': 8500, 'installment': 281.15}\n",
      "2 [-1] [0.44925159215927124]\n",
      "{'safe_loans': -1, 'grade': 'D', 'sub_grade_num': 0.4, 'short_emp': 0, 'emp_length_num': 3, 'home_ownership': 'RENT', 'dti': 13.97, 'purpose': 'other', 'payment_inc_ratio': 2.96736, 'delinq_2yrs': 3, 'delinq_2yrs_zero': 0, 'inq_last_6mths': 0, 'last_delinq_none': 0, 'last_major_derog_none': 1, 'open_acc': 14, 'pub_rec': 0, 'pub_rec_zero': 1, 'revol_util': 59.5, 'total_rec_late_fee': 0.0, 'int_rate': 16.77, 'total_rec_int': 719.11, 'annual_inc': 50004, 'funded_amnt': 5000, 'funded_amnt_inv': 5000, 'installment': 123.65}\n",
      "3 [1] [0.6119099855422974]\n",
      "{'safe_loans': -1, 'grade': 'A', 'sub_grade_num': 1.0, 'short_emp': 0, 'emp_length_num': 11, 'home_ownership': 'MORTGAGE', 'dti': 16.33, 'purpose': 'debt_consolidation', 'payment_inc_ratio': 1.90524, 'delinq_2yrs': 0, 'delinq_2yrs_zero': 1, 'inq_last_6mths': 0, 'last_delinq_none': 1, 'last_major_derog_none': 1, 'open_acc': 17, 'pub_rec': 0, 'pub_rec_zero': 1, 'revol_util': 62.1, 'total_rec_late_fee': 0.0, 'int_rate': 8.9, 'total_rec_int': 696.99, 'annual_inc': 100000, 'funded_amnt': 5000, 'funded_amnt_inv': 5000, 'installment': 158.77}\n"
     ]
    }
   ],
   "source": [
    "incorrect = 0\n",
    "for i, it in enumerate(sample_validation_data):\n",
    "    prediction = model_5.predict(it)\n",
    "    prob = model_5.predict(it, output_type='probability')\n",
    "    print(i, prediction, prob)\n",
    "    print(sample_validation_data[i])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** According to **model_5**, which loan is the least likely to be a safe loan?\n",
    "\n",
    "**A:** sample_validation_data[2] or 3rd\n",
    "\n",
    "**Checkpoint:** Can you verify that for all the predictions with `probability >= 0.5`, the model predicted the label **+1**?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the model on the validation data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that the accuracy is defined as follows:\n",
    "$$\n",
    "\\mbox{accuracy} = \\frac{\\mbox{# correctly classified examples}}{\\mbox{# total examples}}\n",
    "$$\n",
    "\n",
    "Evaluate the accuracy of the **model_5** on the **validation_data**.\n",
    "\n",
    "**Hint**: Use the `.evaluate()` method in the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.66813873330461,\n",
       " 'auc': 0.7246842336920359,\n",
       " 'confusion_matrix': Columns:\n",
       " \ttarget_label\tint\n",
       " \tpredicted_label\tint\n",
       " \tcount\tint\n",
       " \n",
       " Rows: 4\n",
       " \n",
       " Data:\n",
       " +--------------+-----------------+-------+\n",
       " | target_label | predicted_label | count |\n",
       " +--------------+-----------------+-------+\n",
       " |      -1      |        1        |  1618 |\n",
       " |      1       |        -1       |  1463 |\n",
       " |      -1      |        -1       |  3054 |\n",
       " |      1       |        1        |  3149 |\n",
       " +--------------+-----------------+-------+\n",
       " [4 rows x 3 columns],\n",
       " 'f1_score': 0.6715001599317625,\n",
       " 'log_loss': 0.6176131769693982,\n",
       " 'precision': 0.6605831760016782,\n",
       " 'recall': 0.6827840416305291,\n",
       " 'roc_curve': Columns:\n",
       " \tthreshold\tfloat\n",
       " \tfpr\tfloat\n",
       " \ttpr\tfloat\n",
       " \tp\tint\n",
       " \tn\tint\n",
       " \n",
       " Rows: 1001\n",
       " \n",
       " Data:\n",
       " +-----------+-----+-----+------+------+\n",
       " | threshold | fpr | tpr |  p   |  n   |\n",
       " +-----------+-----+-----+------+------+\n",
       " |    0.0    | 1.0 | 1.0 | 4612 | 4672 |\n",
       " |   0.001   | 1.0 | 1.0 | 4612 | 4672 |\n",
       " |   0.002   | 1.0 | 1.0 | 4612 | 4672 |\n",
       " |   0.003   | 1.0 | 1.0 | 4612 | 4672 |\n",
       " |   0.004   | 1.0 | 1.0 | 4612 | 4672 |\n",
       " |   0.005   | 1.0 | 1.0 | 4612 | 4672 |\n",
       " |   0.006   | 1.0 | 1.0 | 4612 | 4672 |\n",
       " |   0.007   | 1.0 | 1.0 | 4612 | 4672 |\n",
       " |   0.008   | 1.0 | 1.0 | 4612 | 4672 |\n",
       " |   0.009   | 1.0 | 1.0 | 4612 | 4672 |\n",
       " +-----------+-----+-----+------+------+\n",
       " [1001 rows x 5 columns]\n",
       " Note: Only the head of the SFrame is printed.\n",
       " You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_5.evaluate(validation_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the number of **false positives** made by the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1618\n"
     ]
    }
   ],
   "source": [
    "num_false_pos = 0\n",
    "for i, it in enumerate(validation_data):\n",
    "    prediction = model_5.predict(it)\n",
    "    #print(prediction, it)\n",
    "    if prediction[0] == 1 and prediction[0] != it[target]:\n",
    "        num_false_pos+=1\n",
    "\n",
    "print(num_false_pos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question**: What is the number of **false positives** on the **validation_data**?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the number of **false negatives** made by the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_false_neg=1463"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparison with decision trees\n",
    "\n",
    "In the earlier assignment, we saw that the prediction accuracy of the decision trees was around **0.64** (rounded). In this assignment, we saw that **model_5** has an accuracy of **0.67** (rounded).\n",
    "\n",
    "Here, we quantify the benefit of the extra 3% increase in accuracy of **model_5** in comparison with a single decision tree from the original decision tree assignment.\n",
    "\n",
    "As we explored in the earlier assignment, we calculated the cost of the mistakes made by the model. We again consider the same costs as follows:\n",
    "\n",
    "* **False negatives**: Assume a cost of \\$10,000 per false negative.\n",
    "* **False positives**: Assume a cost of \\$20,000 per false positive.\n",
    "\n",
    "Assume that the number of false positives and false negatives for the learned decision tree was\n",
    "\n",
    "* **False negatives**: 1936\n",
    "* **False positives**: 1503\n",
    "\n",
    "Using the costs defined above and the number of false positives and false negatives for the decision tree, we can calculate the total cost of the mistakes made by the decision tree model as follows:\n",
    "\n",
    "```\n",
    "cost = $10,000 * 1936  + $20,000 * 1503 = $49,420,000\n",
    "```\n",
    "\n",
    "The total cost of the mistakes of the model is $49.42M. That is a **lot of money**!.\n",
    "\n",
    "**Quiz Question**: Using the same costs of the false positives and false negatives, what is the cost of the mistakes made by the boosted tree model (**model_5**) as evaluated on the **validation_set**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "46990000\n"
     ]
    }
   ],
   "source": [
    "print(10000 * num_false_neg  + 20000 * num_false_pos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Reminder**: Compare the cost of the mistakes made by the boosted trees model with the decision tree model. The extra 3% improvement in prediction accuracy can translate to several million dollars!  And, it was so easy to get by simply boosting our decision trees."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Most positive & negative loans.\n",
    "\n",
    "In this section, we will find the loans that are most likely to be predicted **safe**. We can do this in a few steps:\n",
    "\n",
    "* **Step 1**: Use the **model_5** (the model with 5 trees) and make **probability predictions** for all the loans in the **validation_data**.\n",
    "* **Step 2**: Similar to what we did in the very first assignment, add the probability predictions as a column called **predictions** into the validation_data.\n",
    "* **Step 3**: Sort the data (in descreasing order) by the probability predictions.\n",
    "\n",
    "Start here with **Step 1** & **Step 2**. Make predictions using **model_5** for examples in the **validation_data**. Use `output_type = probability`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">safe_loans</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">grade</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">sub_grade_num</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">short_emp</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">emp_length_num</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">home_ownership</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">dti</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">purpose</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">payment_inc_ratio</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MORTGAGE</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4.21</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">credit_card</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.955726</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MORTGAGE</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12.76</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">car</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.7376</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MORTGAGE</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10.29</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">home_improvement</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3.22264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MORTGAGE</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10.02</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">wedding</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3.49357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MORTGAGE</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3.16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">home_improvement</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2.91713</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MORTGAGE</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">major_purchase</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.74268</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MORTGAGE</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.75</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">home_improvement</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.66994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4.76</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">major_purchase</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.6872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MORTGAGE</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3.33</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">major_purchase</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.64489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MORTGAGE</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2.4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">car</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2.49545</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">delinq_2yrs</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">delinq_2yrs_zero</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">inq_last_6mths</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">last_delinq_none</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">last_major_derog_none</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">open_acc</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">pub_rec</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">pub_rec_zero</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">revol_util</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">total_rec_late_fee</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">int_rate</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">total_rec_int</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">annual_inc</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">funded_amnt</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">funded_amnt_inv</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">installment</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7.9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6.39</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">179.18</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">146000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3800</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3650</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">116.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6.76</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">429.63</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">85000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">123.08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4.5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6.03</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">527.44</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">85000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7500</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7500</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">228.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7.9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6.03</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">161.9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">115000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">334.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7.14</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">505.27</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">85000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10400</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">9809</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">206.63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11.2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7.14</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">56.58</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">100000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">61.89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5.99</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">182.03</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">140987</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6450</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6450</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">196.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7.12</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">115.11</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">220000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">309.32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14.7</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6.92</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">381.04</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">135000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">185.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6.17</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">671.39</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">110000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7500</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7500</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">228.75</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">predictions</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.848508358001709</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.848508358001709</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.8412954211235046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.8412954211235046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.8412954211235046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.8412954211235046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.8412954211235046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.8412954211235046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.8412954211235046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.8412954211235046</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[9284 rows x 26 columns]<br/>Note: Only the head of the SFrame is printed.<br/>You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsafe_loans\tint\n",
       "\tgrade\tstr\n",
       "\tsub_grade_num\tfloat\n",
       "\tshort_emp\tint\n",
       "\temp_length_num\tint\n",
       "\thome_ownership\tstr\n",
       "\tdti\tfloat\n",
       "\tpurpose\tstr\n",
       "\tpayment_inc_ratio\tfloat\n",
       "\tdelinq_2yrs\tint\n",
       "\tdelinq_2yrs_zero\tint\n",
       "\tinq_last_6mths\tint\n",
       "\tlast_delinq_none\tint\n",
       "\tlast_major_derog_none\tint\n",
       "\topen_acc\tint\n",
       "\tpub_rec\tint\n",
       "\tpub_rec_zero\tint\n",
       "\trevol_util\tfloat\n",
       "\ttotal_rec_late_fee\tfloat\n",
       "\tint_rate\tfloat\n",
       "\ttotal_rec_int\tfloat\n",
       "\tannual_inc\tint\n",
       "\tfunded_amnt\tint\n",
       "\tfunded_amnt_inv\tint\n",
       "\tinstallment\tfloat\n",
       "\tpredictions\tfloat\n",
       "\n",
       "Rows: 9284\n",
       "\n",
       "Data:\n",
       "+------------+-------+---------------+-----------+----------------+----------------+\n",
       "| safe_loans | grade | sub_grade_num | short_emp | emp_length_num | home_ownership |\n",
       "+------------+-------+---------------+-----------+----------------+----------------+\n",
       "|     1      |   A   |      0.2      |     0     |       11       |    MORTGAGE    |\n",
       "|     1      |   A   |      0.4      |     0     |       4        |    MORTGAGE    |\n",
       "|     1      |   A   |      0.2      |     0     |       6        |    MORTGAGE    |\n",
       "|     1      |   A   |      0.2      |     0     |       8        |    MORTGAGE    |\n",
       "|     1      |   A   |      0.6      |     0     |       6        |    MORTGAGE    |\n",
       "|     1      |   A   |      0.6      |     0     |       5        |    MORTGAGE    |\n",
       "|     1      |   A   |      0.4      |     0     |       6        |    MORTGAGE    |\n",
       "|     1      |   A   |      0.6      |     0     |       3        |      RENT      |\n",
       "|     1      |   A   |      0.6      |     1     |       1        |    MORTGAGE    |\n",
       "|     1      |   A   |      0.6      |     0     |       11       |    MORTGAGE    |\n",
       "+------------+-------+---------------+-----------+----------------+----------------+\n",
       "+-------+------------------+-------------------+-------------+------------------+\n",
       "|  dti  |     purpose      | payment_inc_ratio | delinq_2yrs | delinq_2yrs_zero |\n",
       "+-------+------------------+-------------------+-------------+------------------+\n",
       "|  4.21 |   credit_card    |      0.955726     |      0      |        1         |\n",
       "| 12.76 |       car        |       1.7376      |      0      |        1         |\n",
       "| 10.29 | home_improvement |      3.22264      |      0      |        1         |\n",
       "| 10.02 |     wedding      |      3.49357      |      0      |        1         |\n",
       "|  3.16 | home_improvement |      2.91713      |      0      |        1         |\n",
       "|  5.2  |  major_purchase  |      0.74268      |      0      |        1         |\n",
       "|  5.75 | home_improvement |      1.66994      |      0      |        1         |\n",
       "|  4.76 |  major_purchase  |       1.6872      |      1      |        0         |\n",
       "|  3.33 |  major_purchase  |      1.64489      |      0      |        1         |\n",
       "|  2.4  |       car        |      2.49545      |      0      |        1         |\n",
       "+-------+------------------+-------------------+-------------+------------------+\n",
       "+----------------+------------------+-----------------------+----------+---------+\n",
       "| inq_last_6mths | last_delinq_none | last_major_derog_none | open_acc | pub_rec |\n",
       "+----------------+------------------+-----------------------+----------+---------+\n",
       "|       2        |        1         |           1           |    9     |    0    |\n",
       "|       2        |        1         |           1           |    11    |    0    |\n",
       "|       1        |        1         |           1           |    14    |    0    |\n",
       "|       0        |        1         |           1           |    14    |    0    |\n",
       "|       0        |        1         |           1           |    16    |    0    |\n",
       "|       1        |        1         |           1           |    7     |    0    |\n",
       "|       0        |        1         |           1           |    6     |    0    |\n",
       "|       0        |        0         |           1           |    14    |    0    |\n",
       "|       0        |        0         |           1           |    5     |    0    |\n",
       "|       0        |        1         |           1           |    6     |    0    |\n",
       "+----------------+------------------+-----------------------+----------+---------+\n",
       "+--------------+------------+--------------------+----------+-----+\n",
       "| pub_rec_zero | revol_util | total_rec_late_fee | int_rate | ... |\n",
       "+--------------+------------+--------------------+----------+-----+\n",
       "|      1       |    7.9     |        0.0         |   6.39   | ... |\n",
       "|      1       |    5.5     |        0.0         |   6.76   | ... |\n",
       "|      1       |    4.5     |        0.0         |   6.03   | ... |\n",
       "|      1       |    7.9     |        0.0         |   6.03   | ... |\n",
       "|      1       |    5.0     |        0.0         |   7.14   | ... |\n",
       "|      1       |    11.2    |        0.0         |   7.14   | ... |\n",
       "|      1       |    0.0     |        0.0         |   5.99   | ... |\n",
       "|      1       |    1.1     |        0.0         |   7.12   | ... |\n",
       "|      1       |    14.7    |        0.0         |   6.92   | ... |\n",
       "|      1       |    0.0     |        0.0         |   6.17   | ... |\n",
       "+--------------+------------+--------------------+----------+-----+\n",
       "[9284 rows x 26 columns]\n",
       "Note: Only the head of the SFrame is printed.\n",
       "You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns."
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "validation_data['predictions'] = model_5.predict(validation_data, output_type='probability')\n",
    "validation_data = validation_data.sort(['predictions'], ascending=False)\n",
    "\n",
    "validation_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Checkpoint:** For each row, the probabilities should be a number in the range **[0, 1]**. We have provided a simple check here to make sure your answers are correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Your loans      : [0.848508358001709, 0.848508358001709, 0.8412954211235046, 0.8412954211235046]\n",
      "\n",
      "Expected answer : [0.4492515948736132, 0.6119100103640573, 0.3835981314851436, 0.3693306705994325]\n"
     ]
    }
   ],
   "source": [
    "print(\"Your loans      : %s\\n\" % validation_data['predictions'].head(4))\n",
    "print(\"Expected answer : %s\" % [0.4492515948736132, 0.6119100103640573,\n",
    "                                0.3835981314851436, 0.3693306705994325])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are ready to go to **Step 3**. You can now use the `prediction` column to sort the loans in **validation_data** (in descending order) by prediction probability. Find the top 5 loans with the highest probability of being predicted as a **safe loan**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">safe_loans</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">grade</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">sub_grade_num</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">short_emp</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">emp_length_num</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">home_ownership</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">dti</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">purpose</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">30.17</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">debt_consolidation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">B</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">29.42</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">debt_consolidation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">D</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.8</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8.95</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">small_business</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.8</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MORTGAGE</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">17.37</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">home_improvement</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">RENT</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8.4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">credit_card</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">payment_inc_ratio</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">delinq_2yrs</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">delinq_2yrs_zero</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">inq_last_6mths</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">last_delinq_none</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">last_major_derog_none</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">open_acc</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">13.5391</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14.3733</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16.727</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12.5753</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11.8779</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">9</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">pub_rec</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">pub_rec_zero</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">revol_util</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">total_rec_late_fee</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">int_rate</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">total_rec_int</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">annual_inc</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">funded_amnt</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">funded_amnt_inv</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">80.7</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">19.7362</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14.33</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1632.01</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">35000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11500</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">11500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">57.5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20.9132</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14.09</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1891.71</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">35000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12250</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">12250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">41.6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16.7025</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15.2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1519.65</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">24000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">14000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">46.1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18.86</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15.31</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1089.84</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">36000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15750</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">60.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">34.64</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15.31</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2152.67</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">35000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">9950</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">9950</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">installment</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">predictions</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">394.89</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.13427510857582092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">419.22</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.13427510857582092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">334.54</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.13427510857582092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">377.26</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.13427510857582092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">346.44</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0.13427510857582092</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 26 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsafe_loans\tint\n",
       "\tgrade\tstr\n",
       "\tsub_grade_num\tfloat\n",
       "\tshort_emp\tint\n",
       "\temp_length_num\tint\n",
       "\thome_ownership\tstr\n",
       "\tdti\tfloat\n",
       "\tpurpose\tstr\n",
       "\tpayment_inc_ratio\tfloat\n",
       "\tdelinq_2yrs\tint\n",
       "\tdelinq_2yrs_zero\tint\n",
       "\tinq_last_6mths\tint\n",
       "\tlast_delinq_none\tint\n",
       "\tlast_major_derog_none\tint\n",
       "\topen_acc\tint\n",
       "\tpub_rec\tint\n",
       "\tpub_rec_zero\tint\n",
       "\trevol_util\tfloat\n",
       "\ttotal_rec_late_fee\tfloat\n",
       "\tint_rate\tfloat\n",
       "\ttotal_rec_int\tfloat\n",
       "\tannual_inc\tint\n",
       "\tfunded_amnt\tint\n",
       "\tfunded_amnt_inv\tint\n",
       "\tinstallment\tfloat\n",
       "\tpredictions\tfloat\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+------------+-------+---------------+-----------+----------------+----------------+\n",
       "| safe_loans | grade | sub_grade_num | short_emp | emp_length_num | home_ownership |\n",
       "+------------+-------+---------------+-----------+----------------+----------------+\n",
       "|     -1     |   C   |      0.2      |     0     |       5        |      RENT      |\n",
       "|     -1     |   B   |      1.0      |     0     |       5        |      RENT      |\n",
       "|     -1     |   D   |      0.8      |     0     |       3        |      RENT      |\n",
       "|     -1     |   C   |      0.8      |     1     |       0        |    MORTGAGE    |\n",
       "|     -1     |   C   |      0.4      |     0     |       4        |      RENT      |\n",
       "+------------+-------+---------------+-----------+----------------+----------------+\n",
       "+-------+--------------------+-------------------+-------------+------------------+\n",
       "|  dti  |      purpose       | payment_inc_ratio | delinq_2yrs | delinq_2yrs_zero |\n",
       "+-------+--------------------+-------------------+-------------+------------------+\n",
       "| 30.17 | debt_consolidation |      13.5391      |      0      |        1         |\n",
       "| 29.42 | debt_consolidation |      14.3733      |      0      |        1         |\n",
       "|  8.95 |   small_business   |       16.727      |      0      |        1         |\n",
       "| 17.37 |  home_improvement  |      12.5753      |      0      |        1         |\n",
       "|  8.4  |    credit_card     |      11.8779      |      0      |        1         |\n",
       "+-------+--------------------+-------------------+-------------+------------------+\n",
       "+----------------+------------------+-----------------------+----------+---------+\n",
       "| inq_last_6mths | last_delinq_none | last_major_derog_none | open_acc | pub_rec |\n",
       "+----------------+------------------+-----------------------+----------+---------+\n",
       "|       1        |        1         |           1           |    7     |    0    |\n",
       "|       0        |        0         |           1           |    14    |    0    |\n",
       "|       2        |        1         |           1           |    7     |    0    |\n",
       "|       0        |        1         |           1           |    8     |    0    |\n",
       "|       0        |        1         |           1           |    9     |    0    |\n",
       "+----------------+------------------+-----------------------+----------+---------+\n",
       "+--------------+------------+--------------------+----------+-----+\n",
       "| pub_rec_zero | revol_util | total_rec_late_fee | int_rate | ... |\n",
       "+--------------+------------+--------------------+----------+-----+\n",
       "|      1       |    80.7    |      19.7362       |  14.33   | ... |\n",
       "|      1       |    57.5    |      20.9132       |  14.09   | ... |\n",
       "|      1       |    41.6    |      16.7025       |   15.2   | ... |\n",
       "|      1       |    46.1    |       18.86        |  15.31   | ... |\n",
       "|      1       |    60.0    |       34.64        |  15.31   | ... |\n",
       "+--------------+------------+--------------------+----------+-----+\n",
       "[5 rows x 26 columns]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "validation_data[-5:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question**: What grades are the top 5 loans?\n",
    "\n",
    "**A:** All grade A\n",
    "\n",
    "Let us repeat this excercise to find the top 5 loans (in the **validation_data**) with the **lowest probability** of being predicted as a **safe loan**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Checkpoint:** You should expect to see 5 loans with the grade ['**D**', '**C**', '**C**', '**C**', '**B**'] or with ['**D**', '**C**', '**B**', '**C**', '**C**']."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Effect of adding more trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this assignment, we will train 5 different ensemble classifiers in the form of gradient boosted trees. We will train models with 10, 50, 100, 200, and 500 trees.  We use the **max_iterations** parameter in the boosted tree module. \n",
    "\n",
    "Let's get sarted with a model with **max_iterations = 10**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_10 = turicreate.boosted_trees_classifier.create(train_data, validation_set=None, \n",
    "        target = target, features = features, max_iterations = 10, verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, train 4 models with **max_iterations** to be:\n",
    "* `max_iterations = 50`, \n",
    "* `max_iterations = 100`\n",
    "* `max_iterations = 200`\n",
    "* `max_iterations = 500`. \n",
    "\n",
    "Let us call these models **model_50**, **model_100**, **model_200**, and **model_500**. You can pass in `verbose=False` in order to suppress the printed output.\n",
    "\n",
    "**Warning:** This could take a couple of minutes to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_50 = turicreate.boosted_trees_classifier.create(train_data, validation_set=None, target = target, features = features, max_iterations = 50, verbose=False)\n",
    "model_100 = turicreate.boosted_trees_classifier.create(train_data, validation_set=None, target = target, features = features, max_iterations = 100, verbose=False)\n",
    "model_200 = turicreate.boosted_trees_classifier.create(train_data, validation_set=None, target = target, features = features, max_iterations = 200, verbose=False)\n",
    "model_500 = turicreate.boosted_trees_classifier.create(train_data, validation_set=None, target = target, features = features, max_iterations = 500, verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare accuracy on entire validation set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will compare the predicitve accuracy of our models on the validation set. Evaluate the **accuracy** of the 10, 50, 100, 200, and 500 tree models on the **validation_data**. Use the `.evaluate` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6907582938388626\n",
      "0.6917277035760449\n",
      "0.684510986643688\n",
      "0.6718009478672986\n"
     ]
    }
   ],
   "source": [
    "print(model_50.evaluate(validation_data)['accuracy'])\n",
    "print(model_100.evaluate(validation_data)['accuracy'])\n",
    "print(model_200.evaluate(validation_data)['accuracy'])\n",
    "print(model_500.evaluate(validation_data)['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** Which model has the **best** accuracy on the **validation_data**?\n",
    "\n",
    "**A:** model_100\n",
    "\n",
    "**Quiz Question:** Is it always true that the model with the most trees will perform best on test data?\n",
    "\n",
    "**A:** no"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot the training and validation error vs. number of trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall from the lecture that the classification error is defined as\n",
    "\n",
    "$$\n",
    "\\mbox{classification error} = 1 - \\mbox{accuracy} \n",
    "$$\n",
    "\n",
    "In this section, we will plot the **training and validation errors versus the number of trees** to get a sense of how these models are performing. We will compare the 10, 50, 100, 200, and 500 tree models. You will need [matplotlib](http://matplotlib.org/downloads.html) in order to visualize the plots. \n",
    "\n",
    "First, make sure this block of code runs on your computer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "def make_figure(dim, title, xlabel, ylabel, legend):\n",
    "    plt.rcParams['figure.figsize'] = dim\n",
    "    plt.title(title)\n",
    "    plt.xlabel(xlabel)\n",
    "    plt.ylabel(ylabel)\n",
    "    if legend is not None:\n",
    "        plt.legend(loc=legend, prop={'size':15})\n",
    "    plt.rcParams.update({'font.size': 16})\n",
    "    plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to plot the classification errors (on the **train_data** and **validation_data**) versus the number of trees, we will need lists of these accuracies, which we get by applying the method `.evaluate`. \n",
    "\n",
    "**Steps to follow:**\n",
    "\n",
    "* **Step 1:** Calculate the classification error for model on the training data (**train_data**).\n",
    "* **Step 2:** Store the training errors into a list (called `training_errors`) that looks like this:\n",
    "```\n",
    "[train_err_10, train_err_50, ..., train_err_500]\n",
    "```\n",
    "* **Step 3:** Calculate the classification error of each model on the validation data (**validation_data**).\n",
    "* **Step 4:** Store the validation classification error into a list (called `validation_errors`) that looks like this:\n",
    "```\n",
    "[validation_err_10, validation_err_50, ..., validation_err_500]\n",
    "```\n",
    "Once that has been completed, the rest of the code should be able to evaluate correctly and generate the plot.\n",
    "\n",
    "\n",
    "Let us start with **Step 1**. Write code to compute the classification error on the **train_data** for models **model_10**, **model_50**, **model_100**, **model_200**, and **model_500**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_error(model, data):\n",
    "    return 1 - model.evaluate(data)['accuracy']\n",
    "\n",
    "train_err_10 = calc_error(model_10, train_data)\n",
    "train_err_50 = calc_error(model_50, train_data)\n",
    "train_err_100 = calc_error(model_100, train_data)\n",
    "train_err_200 = calc_error(model_200, train_data)\n",
    "train_err_500 = calc_error(model_500, train_data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us run **Step 2**. Save the training errors into a list called **training_errors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_errors = [train_err_10, train_err_50, train_err_100, \n",
    "                   train_err_200, train_err_500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, onto **Step 3**. Write code to compute the classification error on the **validation_data** for models **model_10**, **model_50**, **model_100**, **model_200**, and **model_500**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation_err_10 = calc_error(model_10, validation_data)\n",
    "validation_err_50 = calc_error(model_50, validation_data)\n",
    "validation_err_100 = calc_error(model_100, validation_data)\n",
    "validation_err_200 = calc_error(model_200, validation_data)\n",
    "validation_err_500 = calc_error(model_500, validation_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us run **Step 4**. Save the training errors into a list called **validation_errors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation_errors = [validation_err_10, validation_err_50, validation_err_100, \n",
    "                     validation_err_200, validation_err_500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will plot the **training_errors** and **validation_errors** versus the number of trees. We will compare the 10, 50, 100, 200, and 500 tree models. We provide some plotting code to visualize the plots within this notebook. \n",
    "\n",
    "Run the following code to visualize the plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAELCAYAAAAP/iu7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABBKElEQVR4nO3dd3wU5fb48c9JT2gBQhIgQJTQEqSjgCIdbIAiXkG84hd7u2JDRS+Ciu1eFSsKXtT7U0EQEC6oKAgWQOkCoQlIC0kIJbQUUp7fHzsJu8km2YRsNsme9+s1L3aemZ05OyR7MjPPPEeMMSillFLu5OPpAJRSSlV/mmyUUkq5nSYbpZRSbqfJRimllNtpslFKKeV2mmyUUkq5nSYbpSoZEbldRH714P7vE5FkETkjIvU9FYeqXjTZKLcSkX0ikm59ceVN73o6LuWciPgDbwADjTE1jTHHCiyPFhEjIn6eiVBVVfoDoyrCYGPM0pJWEhE/Y0x2gTZfY0yOqzsq7frVnbNjWoIIIAiIr8B9Ki+gZzbKY6zLRStF5E0ROQZMFJFPRGSqiHwjImeBPiLSRkRWiEiqiMSLyBC7bRRav8A+bhaRdQXaHhGRhdbra0Rkm4icFpEEEXm8mFh/FZF/i8gJEflLRK62W75PRPrbzU8Ukc+s13lnA/8nIget998rIl1FZLP1uQqe7YmIvCsiJ0Vkh4j0s1tQR0T+IyKJVswviohvUcfUyWcJFJEpInLYmqZYbS2BndZqqSLyo5ND8bPd8jMi0r2I/8dA61gdsC7JfSAiwXYxXCcim6zPvkpE2tkte9L6XKdFZKf9Z1dVmDFGJ53cNgH7gP5FLLsdyAYewnaWHQx8ApwELsf2x1AtYDcwHggA+gKngVbWNgquH1RgHyHW+i3s2tYCI6zXiUBP63VdoFMxsWYBdwG+wH3AYUCcfU5sX/KfWa+jAQN8gO2sYSCQAXwNhAONgSNArwLH5RHAH7jZ+oz1rOXzgQ+BGtb71wD3FHVMnXyW54HfrPc2AFYBLxSI1a+I41BoeRH/j28CC4F61v/h/4CXrfU7Wp/3MutYjraOXyDQCjgINLLbX3NP/xzrdOGTxwPQqXpP1pfIGSDVbrrLWnY7cKDA+p8A/7Wb7wkkAT52bTOBic7WLyKGz4AJ1usW2JJPiDV/ALgHqF3CNm4HdtvNh1hfupF2n7OkZNPYbvkx4Ga7+bnAWLt95Scyq20N8Hdsl7ky7ZMIMBJYXtQxdfJZ9gDX2M0PAvYViLW0yeaA3bwAZ+2TBNAd+Mt6PRUrudkt3wn0AmKwJaL+gL+nf351Kr9JL6OpinC9MSbUbpput+ygk/Xt2xoBB40xuXZt+7GdDRS3DXtfYPtCBrgF+NoYk2bN3whcA+wXkZ9EpHsx20nKe2H3/pol7Ntest3rdCfz9ttKMNa3sGU/tmPRDNvZTqJ1CSoV21lOuN26JR2PRtb2Cm77QtjvswG2ZLzeLsbvrHawfYbH8pZZy5tgO5vZDYzFlqyPiMgsEbnQ2FQloMlGeZqzYcft2w4DTUTE/me1KZBQwjbs/QA0EJEO2JLOF/lvNGatMWYoti/rr4HZLkfu6Cy2L9g8kWXcTp7GIiJ2802xHYuD2M5swuySd21jTJzduiUdj8PYvvALbtsVRW3bvv0otuQZZxdjHWNMXjI9CEwu8AdIiDFmJoAx5gtjzBVWjAZ41cXYVCWmyUZVdr8DacA4EfEXkd7AYGCWqxswxmQBc4B/YbuH8AOAiASIyCgRqWOtcwrILXpLxdoEjLBi7AIML+N28oQD/7C2dxPQBvjGGJMIfA+8LiK1RcRHRJqLSK9SbHsm8KyINBCRMGACtkuNrkjBdowuLmoF6yx0OvCmiIQDiEhjERlkrTIduFdELhObGiJyrYjUEpFWItJXRAKx3ddKp+z/J6oS0WSjKsL/xPE5m/muvtEYcw5bcrka21/M7wO3GWN2lDKGL7DdB5hjHLvl/h3YJyKngHuBUaXcbp5/As2BE8Ak7M6eyuh3bPeXjgKTgeHm/DMvt2HrLLHN2t9XQMNSbPtFYB2wGdgCbLDaSmRdPpwMrLQugXUrYtUnsXXs+M06tkux3fzHGLMOW0eLd634d2O77wO2TgKvYPvcSdiS7tOl+GyqkhLHy8JKKaVU+dMzG6WUUm6nyUYppZTbabJRSinldppslFJKuV2VG4gzLCzMREdHezoMpZRSTqxfv/6oMaZBwfYql2yio6NZt25dySsqpZSqcCKy31m7XkZTSinldppslFJKuZ0mG6WUUm6nyUYppZTbabJRSinldlWuN9oFOZ0ENSPAYeR2pZTyIsZAThZknYWsdDiXBll207k0uKgnBNUp1916T7LJyYJpfaBmOHS7D+JuAL9AT0ellFKOcnMhO92WCPK+/POTQTqcs5JEccmiyOXW+01O8THctRwadyrXj+U9yWb7Qjh92DbNvwe+/yd0vRO6jIGahZ4/Ukop53Ky7b7Iz55PCsUmhlIsz0739Ce0xVTOvCfZ/DbVcf7sEVjxEvzyb2g7HLrdCw3beyY2pVT5MAayM4v4ci/pr34Xl+dmefpTul9WWsnrlJJ3JJtzaRASBgiFqtrmnIM/vrBNzS63XWJrdQ34+HoiUqWqt9wcF/7qv4Azhaw0MFrYs0Q+fuBfA/yDISAE/POmYAioAcF1y32X3pFsAkLglllwbA+smQYbP4NzZwqvt3+lbQptCpfeAx1vheDQCg9XKY9w5caxw1/9ZbhElJPp6U9ZNfgFnU8AAVYScJocnC23/nVYXiCZ+PpX+EeqcpU6u3TpYi54bLSMk7Dxc1jzIZzYV/R6/jWgwy1w2b0QFnNh+1TqQuXmQnaGC1/ubrxxrACxvtBDXP9yL3K5k2ThHwI+VfepFBFZb4zpUqjdK5NNntwc2PWd7X7Ovl+KX7fFQFvSad5Xu04r51y6cVzaS0R2ycEN19GrJd+AUp4JlPJMwS9QvwOKocmmJElb4fepsHlO8af6DVrDZfdAuxG2H0pVdWRnWl/oeuPYo/yL+7IvzZlCEcnC1zvuDlRWmmxcdfYorPsY1n4EZ5KKXi8oFDrfDpfeBXWi3BePcmQMZJ6G9BOFp4xUu/nUwsuzMzwdfeUnvi5cIirl/QH79f2CqvQlIlUyTTaW42fPIUDdGgHFr5h9DrYtgN/eh8Mbil5PfCF2CFx2HzS5VE+vXZWTbbt35jRROJvslnvzfQW/IBfPCsp6iaiE3wulSuD1ycYYw9wNCUxevI3ercJ58+YOrr4RDq21JZ1tC4v/omvUEbrdD7HXe88vbVaGC4mi4JlGKmSe9HDg7iAXeLO4pGQRrF3yVaXn1cnm2JlMHpq5kVV7juW3/b87LqVni1KOHHDykO3y2rqPbV+wRakZaY1O8H9QI6x0+/AEY2xdwV1NFA6XpirB086u8vGzu0RU1pvFzpbrjWOl8nh1ssnKyWXwO7+yI+l0fluz+iEsGXslQf5l+EvxXBps/hJ+/wBSdhSzoth6xvj6277o8v718bfdxPTxt2vzK7Be3jJfx/Vc2obdMh9fyCwukViXsHKzS38cPMU/xPbQWf4UaruH5tBWcAqFgJqaDJRyM69ONgCbDqZyw/srsf+49/duzrirWpc9GGNg73L47QP4c0nZt+Otguo4JgRXEkZQKPgHeThwpVRRiko2XtNHsEOTUG7r1oxPV+/Pb5v2816GdGhE68jaZduoiO25m+Z94ehu20OiGz+3dY/1Fj5+pUsW+evW0fsPSnkRrzmzATidkcWAN34m6dT5LrCdmoby1b098PEpp8sr6am24XDWfwLH/iyfbVaEvEtT+QkjtPhkkTfppSmllB2vv4yW57utSdz72XqHtheub8vfuzW70NAKy8m2PeiXk2W7J5L3b25WgWVZttEM8l7nLbN/T/4ya12XtmutF1Cz+IShl6aUUuXEI5fRROQq4C3AF/jIGPNKgeX3Ag8AOcAZ4G5jzDZ3xnRV20gGxkbw/bbk/LbXvt3BwNgIImqX8xeur59t8g8u3+0qpVQV47ZHeUXEF3gPuBqIBUaKSGyB1b4wxlxijOkAvAa84a547E0aGkfNwPN59nRmNhMXxlfErpVSyiu5c9yIS4Hdxpi9xphzwCxgqP0KxphTdrM1KFRsxj0a1gnm8YEtHdq+3ZrEUruzHaWUUuXHncmmMXDQbv6Q1eZARB4QkT3Yzmz+4WxDInK3iKwTkXUpKSnlEtzfu0fTvkmoQ9uEBVs5m1mFnjdRSqkqwuMj4hlj3jPGNAeeBJ4tYp1pxpguxpguDRqU8qn/Ivj6CC/fcAm+dr3QDp/M4PXvd5XL9pVSSp3nzmSTADSxm4+y2ooyC7jejfEUEtuoNndecZFD2yer/mLLoeo4bpdSSnmOO5PNWqCFiFwkIgHACGCh/Qoi0sJu9lqgwh9Mebh/C6Lqnu8tlmvgqXmbyc7ROuZKKVVe3JZsjDHZwIPAEmA7MNsYEy8iz4vIEGu1B0UkXkQ2AY8Co90VT1FCAvx48fq2Dm3xh0/xyap9FR2KUkpVW173UGdRHpq5kf/9cTh/Ptjfl+8fuZIm9bQap1JKuaqohzo93kGgsphwXSy1g84/e5OelcOEBVupaslYKaUqI002lga1Ahl/TRuHtuU7U1i8JdFDESmlVPWhycbO37o04dLoeg5tk/63jZPpWR6KSCmlqgdNNnZ8fISXhrXF3/f8szcppzN59bviCqQppZQqiSabAmLCa3Ff7xiHti9+P8C6fcc9FJFSSlV9mmycuL93cy4Oq+HQ9vS8LZzL1mdvlFKqLDTZOBHk78vkGy5xaPvzyBmm/bzHQxEppVTVpsmmCN2b1+emzlEObW//uJu9KWc8FJFSSlVdmmyKMf6aNtSvEZA/fy47l2fm67M3SilVWppsilG3RgD/vM6x3tvqvceYu6G48USVUkoVpMmmBEM7NKJnizCHtsmLt3H87DkPRaSUUlWPJpsSiAgvXt+WQL/zh+pEWhYvLt7mwaiUUqpq0WTjgmb1a/Bw/xYObfM2JLBy91EPRaSUUlWLJhsX3dXzYlpH1nJoe2b+FjKycjwUkVJKVR2abFzk7+vDS8MuQc6PZMO+Y2m882OF13tTSqkqR5NNKXRqWpdbL2vm0PbhT3vZmXTaQxEppVTVoMmmlJ64qhXhtQLz57NzDePnbyE3V5+9UUqpomiyKaXaQf48PzTOoW39/hN8seaAhyJSSqnKT5NNGQyKi6R/mwiHtle/28GRUxkeikgppSo3TTZlICI8PzSOGgG++W2nM7KZ9D999kYppZzRZFNGjUKDeWxgK4e2xVsS+XFHsociUkqpykuTzQUY3SOadlF1HNr++XU8ZzOzPRSRUkpVTppsLoCvj/DSDZfg63P+4ZuE1HTe+GGXB6NSSqnKR5PNBWrbuA5jLo92aPt45V9sOXTSMwEppVQlVGyyERFfEfm8ooKpqh4Z0JLGocH587kGnp6/mewcLSOtlFJQQrIxxuQAzUQkoLj1vF1IgB8vXt/WoW1rwik+WbXPMwEppVQl48pltL3AShH5p4g8mje5O7Cqpk/rcK5r19Ch7Y0fdpGQmu6hiJRSqvJwJdnsARZZ69aym1QBEwbHUjvIL38+7VwOT83dzLlsvZymlPJuYoxrY3qJSE0AY8wZt0ZUgi5duph169Z5MoRiffH7AcbP3+LQdkVMGFNv7UStIH8PRaWUUhVDRNYbY7oUbC/xzEZE2orIRiAeiBeR9SISV9L7vNWIrk3oGl3Xoe3X3Ue56YPVJJ3U4WyUUt7Jlcto04BHjTHNjDHNgMeA6e4Nq+ry8RGmjOhIdP0Qh/YdSacZ9v5KdiVrOQKllPdxJdnUMMYsz5sxxqwAargtomqgcWgwc+/rQfsmoQ7th09mcOPUVazec8wzgSmllIe41BvN6okWbU3PYuuhpopRv2Ygs+7qRv824Q7tpzOyGT1jDQv/OOyhyJRSquK5kmzGAA2AecBcIMxqUyUIDvDlg1s7c2u3pg7t53Jy+cfMjUz7eQ+udtBQSqmqzK+4hSLiC8wzxvSpoHiqHT9fH14Y2pZGocG89t1Oh2UvfbODw6kZ/PO6WIfx1ZRSqrpxZQSBXBGpU9x6qngiwv29Y3jz5vb4+zomlU9W7eP+z9eTkZXjoeiUUsr9XLmMdgbYIiL/EZG38yZ3B1Yd3dAxik/+71JqBTqeUC6JT+aW6b9x/Ow5D0WmlFLu5UqymQf8E/gZWG83qTK4PCaM2fd2J7J2kEP7hgOpDJ+6igPH0jwUmVJKuU+xIwhY92yWVqZ7NpV9BAFXJZ5M5/YZa9lZ4LmbsJoBzLi9K+2iQj0TmFJKXYAyjSCg92zcp2GdYGbf253uF9d3aD965hw3f/gby3cc8VBkSilV/vSejQfVCfbnkzFdGdqhkUN7elYOd/53HTPXHPBQZEopVb6K7fpsmWdNyg0C/Xx5828daBQazNQVe/Lbc3INT8/bQmJqOo8MaImIdo1WSlVdJSYbY8ynIhIMNDXG7CxpfVV6Pj7Ck1e1plGdIJ5bGE+u3W20t3/cTUJqBq/ceAn+vlrFWylVNbky6vNgYBPwnTXfQUQWujkur/T37tF8cGtngvwd/1vmbjjEmE/Wcjojy0ORKaXUhXHlT+WJwKVAKoAxZhNwsdsi8nID4yL54q5u1KvhWIn7lz+PcvOHv5F8SssUKKWqHleSTZYx5mSBNi096UadmtZl7n09aFagTMG2xFMMe38Vf2qZAqVUFeNKsokXkVsAXxFpISLvAKvcHJfXuyishq1MQZRjr/OE1HRunLqK3/dqmQKlVNXhSrJ5CIgDMoEvgJPAWDfGpCxhNQOZeXc3+rV2LFNwKiObv/9nDYs2a5kCpVTVUGKyMcakGWOeMcZ0taZnjTF646CChAT48eHfO3PLZYXLFDz4xUY++mWvlilQSlV62pe2CvDz9WHy9W15YlCrQsteXLyd5xdtIydXE45SqvLSZFNFiAgP9Inhjb+1x69A7ZuPV+7jwS82aJkCpVSlpcmmihnWyVamoGaBMgXfbk3i1o9+54SWKVBKVUKuPNTZQETGi8g0EZmRN1VEcMq5K1qEMfue7kTUDnRoX7f/BDd+sIqDx7VMgVKqcnHlzGYBUAdYCiy2m5QHxTaqzfz7L6dlRE2H9r0pZ7nh/VVsOVTw0SillPIcV5JNiDHmSWPMbGPM3LzJ7ZGpEjUKDWbOvT3odnE9h/ajZzK5edpqlu/UMgVKqcrBlWSzSESucXskqkzqBPvz6ZhLGdzesUxB2rkc7vx0HV+u1TIFSinPcyXZPIwt4WSIyGlrOuXKxkXkKhHZKSK7ReQpJ8sfFZFtIrJZRJaJSLPSfgBlK1Pw1s0duKeX45B1ObmGJ+du4Y0fdpGrXaOVUh7kykOdtYwxPsaYIOt1LWNM7ZLeZ5WUfg+4GogFRopIbIHVNgJdjDHtgK+A10r/ERTYyhQ8fXUbJg2Jo2Dpm7eX/cmNH6xi22GX/kZQSqly51LXZxEZIiL/tqbrXNz2pcBuY8xeY8w5YBYw1H4FY8xyY0xe16nfgChXA1fOje4RzdRRnQn0c/yv3XgglcHv/srkxds4m5ntoeiUUt7Kla7Pr2C7lLbNmh4WkZdd2HZj4KDd/CGrrSh3AN8WEcPdIrJORNalpKS4sGvvdlVbW5mCuiH+Du05uYbpv/zFgDd+4vv4JA9Fp5TyRq6c2VwDDDDGzDDGzACuAq4tzyBE5FagC/AvZ8uNMdOMMV2MMV0aNGhQnruutjo3q8vif/RkYGxEoWWHT2Zw9/9bz13/XUdCaroHolNKeRtXRxAItXtdp6iVCkgAmtjNR1ltDkSkP/AMMMQYk+nitpULGoUGM+22Lky/rQuN6gQVWv7DtmT6v/4T037eQ1aOlihSSrmPK8nmZWCjiHwiIp8C64HJLrxvLdBCRC4SkQBgBOBQTlpEOgIfYks0+lCImwyIjeCHR3tx95UX41tgXLX0rBxe+mYHg9/5lQ0HTngoQqVUdSeuDE8vIg2BrtbsGmOMSxf8redzpgC+wAxjzGQReR5YZ4xZKCJLgUuAROstB4wxQ4rbZpcuXcy6detc2b1yYnviKZ6Zv4UNB1ILLROBkZc25clBralT4H6PUkq5QkTWG2O6FGovKtmISGtjzA4R6eRsuTFmQznH6BJNNhcuN9cwa+1BXvl2O6cyCvdMC6sZwLPXxjK0QyOkYD9qpZQqRlmSzTRjzN0istzJYmOM6VveQbpCk035STmdyUvfbGf+xkK30gC4PKY+Lwxty8UNajpdrpRSBZU62di9MahgZU5nbRVFk035W7n7KM9+vZW/jp4ttCzA14f7+zTn3l7NCfL39UB0SqmqpKhk40oHgVUutqkq6vKYML59uCdj+7cgwNfxR+JcTi5Tlv7JNW/9wsrdRz0UoVKqqisy2YhIpIh0BoJFpKOIdLKm3kBIRQWoKkaQvy9j+7fku7E9uTymfqHle4+eZdRHvzN21kZSTmsPdaVU6RR3z2Y0cDu2hy3tr1udBj4xxsxze3RO6GU09zPGsPCPw7ywaBtHzxSu/Fk7yI8nr27NyK5N8fHRDgRKqfMu5J7NjZWpfo0mm4pzMi2L15bs4Is1B3D2Y9KxaSgv3XAJbRqWOC6rUspLlDnZWG++FogD8h9DN8Y8X64RukiTTcXbcOAE4+dtYUfS6ULLfH2EO664iLH9WxAS4OeB6JRSlUmZOwiIyAfAzcBDgAA3AVp3xot0alqXRQ9dwTPXtCEkwLFHWk6uYdrPexnwxs/8sC3ZQxEqpSo7V3qj9TDG3AacMMZMAroDLd0blqps/Hx9uOvKi/nh0V4McDK4Z0JqOnf9dx13/Xcdh3VwT6VUAa4km7xvjjQRaQRkAQ3dF5KqzBqHBjP9ti5M+3vnogf3fOMnPvplL9k6uKdSyuJKslkkIqHYhv/fAOwDZroxJlUFDIyLLHJwz7RzOby4eDuD312pg3sqpQAXOwjkrywSCAQZY066L6TiaQeBymfb4VM88/UWNhYxuOctlzZl3FWtqROsg3sqVd1dSAeBB6wzG6x6Mz4icn/5h6iqqthGtZl7bw8m39CW2kGOPdKMgc9/P0C/139iwaYESvPHjVKq+nDlMtpdxpjUvBljzAngLrdFpKokHx9h1GXNWPZYb67v0KjQ8qNnMnl41ib+/p81TsdgU0pVb64kG1+xG2deRHyBAPeFpKqyBrUCmTKiI5/feRkXhdUotPzX3UcZNOVn3lr6J5nZOR6IUCnlCa4km++AL0Wkn4j0w9Y54Dv3hqWqurzBPR/u52Rwz+xc3ly6i6un/MIqHdxTKa/gynA1PsA9QD+r6QfgI2OMR/4s1Q4CVc/elDM8+/VWVu055nT5DR0b88y1bQirGVjBkSmlytsFDVdTmWiyqZqMMSzYdJgXFxc9uOdTV7dhRNcmOrinUlVYqXujichs698tIrK54OTOYFX1IyJc37Exyx7tzS2XNS20/FRGNuPnb2H4B6vYkXTKAxEqpdypuBIDjYwxh0XE6Thoxpj9bo2sCHpmUz2s33+CZ+YXPbjnnVdcxMM6uKdSVU5ZnrNZZP37ojFmf8HJPWEqb9G5WV3+99AVjL+mNcH+hQf3/NAa3HOpDu6pVLVQXLIJEJFbgB4iMqzgVFEBqurL39eHu69sztLHetG/jfPBPe/87zru+X86uKdSVV1xl9GuAEYBfwMWFlhsjDFj3BybU3oZrfr6Pj6JiQvjOXwyo9CykABfHh3Qktt7ROPn60qPfaWUJ1xIpc47jDH/cVtkpaTJpno7m5nNmz/s4uNV+8jJLfyzGduwNi8Nu4QOTUIrPjilVIlKnWxEpK8x5seiLpkZY+aVc4wu0WTjHeIPn+SZ+VvZdDC10DIRGHVZU54YpIN7KlXZlKWDQC/r38FOpuvKPUKl7MQ1qsO8+3rw4vVtqeVkcM/PftPBPZWqSvShTlXpHTmdweTF21mw6bDT5T1bhPHC0LZEOxmLTSlVsS6kxMDDIlJbbD4SkQ0iMtA9YSpVWHitIN4a0ZHP7riM6PohhZb/8udRBk75mbeX6eCeSlVWrnTrGWOMOQUMBOoDfwdecWtUSjlxRYswvht7ZZGDe77xwy6ufusXVu3RwT2VqmxcSTZ5A1VdA/zXGBNv16ZUhQry9+WRAS35dmxPejSvX2j53pSz3DL9dx79chNHz2R6IEKllDOuJJv1IvI9tmSzRERqAbnuDUup4jVvUJPP77yMN29uT/0ahcsrzduYQL/Xf2LmmgPkOulCrZSqWK6WGOgA7DXGpIpIPSDKGOORwTi1g4AqKDXtHK9+t5OZaw44Xd65WV0m39CW1pG1KzgypbxPmTsIAN2BnVaiuRV4FjhZ3gEqVVahIQG8POwS5t7XndaRtQotX7//BNe9/Ssvf7udtHPZHohQKeXKmc1moD3QDvgE+Aj4mzGmV3Hvcxc9s1HFycrJZcavfzFl6Z+kZxXumVYn2J9+bcIZFBfJlS0aEBzg62QrSqmyupDhajYYYzqJyAQgwRjzn7w2dwVbHE02yhWHTqQxcWE8S7cfKXKdIH8ferVswKC4SPq1jqBOiI5GoNSFupBk8xPwHfB/wJXAEeAPY8wl7gi0JJpslKuMMXy/LZmJC+NJdDK4pz0/H6HbxfUZ1DaSgbERRNQOqqAolapeLiTZRAK3AGuNMb+ISFOgtzHmv+4JtXiabFRpnc3M5u0f/+TLtQdJTcty6T0dm4YyKC6SQXGRXKQjEyjlsjInm8pGk40qq+ycXNbsO8738cksiU8q8WwnT8uImvmJJ65RbUT0MTOlinIhZzbdgHeANkAA4AucMcbUcUegJdFko8qDMYYtCSdZEp/Ekvhkdh8549L7GocGMzAugkFxkXSNroevjyYepexdSLJZB4wA5gBdgNuAlsaYp90RaEk02Sh32JNyJj/x/OGkrIEz9WoE0N/q2XZ5TBhB/tqzTakLSjbGmC4istkY085q22iM6eimWIulyUa5W+LJ9PxLbb//ddxpEbeCagT40ru1LfH0adWAWkHas015pwtJNj8D/bE9X5MEJAK3G2PauyPQkmiyURXpxNlzLNtxhCXxSfy8K4XM7JJHagrw9aFHTH0GxUXSv00EDWoFVkCkSlUOF5JsmmHr7uwPPALUAd43xux2R6Al0WSjPCXtXDY/70phSXwyy7Yncyqj5NEIRKBLs7r5HQya1CtcIkGp6kR7oylVjrJycvlt7zGWxCfxfXwyR067NsJ0m4a1GWR1MGgdWUt7tqlqp9TJRkS2AEVmorz7NxVNk42qbHJzDRsPpvJ9fBJL4pPYdyzNpfc1qx9infFE0LFJXXy0Z5uqBsqSbJoVt0FjzP5yiq1UNNmoyswYw67kvJ5tScQfPuXS+xrUCmRArO2Mp/vF9Qnwc2WMXKUqn7IkmxggwhizskD75UCSMWaPWyItgSYbVZUcPJ7G99tsPdvW7TuOK6V1agX50dfq2darZQNqBPq5P1ClyklZks0i4GljzJYC7ZcALxljBrsl0hJoslFV1dEzmSzbnsyS+GR+/fMo53JK7tkW6OdDzxZhDLR6ttVzUihOqcqkLMlmrTGmaxHLtuhAnEqV3ZnMbFbsPMJ3W5NYsTOFM5kl92zz9RG6Rp/v2dYoNLgCIlWqdMqSbP40xrQoYtluY0xMOcfoEk02qrrJzM5h1W5bz7YftiVz7Ow5l97XLqpOfgeDmPDCReOU8oSyJJuZwI/GmOkF2u8EBhhjbnZLpCXQZKOqs5xcw/r9J/I7GBw6ke7S+y5uUCP/jKd9VB3tUq08pizJJgKYD5wD1lvNXbANxnmDMSbJTbEWy5Vkc+rUKY4cOUJWlmvDyStVUfz9/QkPD6d27dolrmuMYVviKZbEJ/N9fBI7kk67tI/I2kH5g4VeelE9/H21Z5uqOBcygkAfoK01G2+M+dEN8bmspGRz6tQpkpOTady4McHBwfoXnqo0jDGkp6eTkJBARESESwnH3r6jZ/l+m22w0A0HTuDK89haBltVNK8ZQWD37t00atSIkBAdFkRVTmlpaRw+fJiYmLLf9jxyKiO/S/XqPcfIdqFPdbC/r60MdtsI+raOoE6wDhaqyl9RyabadeDPysoiOFh76ajKKzg4+IIv8YbXDuLWbs24tVszTqZnsdwaLHTFzhTSs3Kcvic9K4fv4pP4Lj4JPx+he/P6DIzTMtiqYrg12YjIVcBb2AqufWSMeaXA8iuBKUA7YIQx5qty2m95bEYptyjvn886wf5c37Ex13dsTEZWDr/8eZQl8Uks3Z5cZBns7FzDL38e5Zc/j/LPr7dqGWzldm5LNiLiC7wHDAAOAWtFZKExZpvdageA24HH3RWHUt4kyN+XAbERDIiNKFUZ7I0HUtl4IJVXvt2hZbCVW7jzzOZSYLcxZi+AiMwChgL5ycYYs89aVvKj1EqpUvHz9aFH8zB6NA/jucGxbD50Mr9L9Z6Us0W+b1fyGXYl7+adH3fTODQ4/1meLloGW10Ad/aJbAwctJs/ZLWVmojcLSLrRGRdSkpKuQRXWYlIidOKFSvKtO19+/YhIixatKhU71uxYgUiwtatW8u0X+V5IkL7JqGMu6o1yx7rzdJHe/HEoFa0j6pT7PsSUtOZsfIvbp72G5dOXsqTX23mxx3JZBRxX0ipolSJDgLGmGnANLD1RvNwOG61evXq/Nfp6en07duXZ599lmuvvTa/PTY2tkzbbtiwIatXr6Z169alel+nTp1YvXo1zZs3L9N+VeUTE16TmPAYHugT43IZ7GNnz/HluoN8ue6glsFWpebOZJMANLGbj7LaVDG6deuW//rMmTMANG/e3KHdXk5ODjk5OQQElDxAY2BgYJHbKU7t2rXL9L6KlpWVhY+PD76+vi61u6I0x7eqalgnmNE9ohndI9rlMthnz+WweHMiizcnOpTBHhAbQVhNLYOtCnPnZbS1QAsRuUhEAoARwEI37s8r3H777XTp0oWvv/6auLg4goKC+P3330lMTGTMmDFcfPHFBAcH07JlS5599lnOnTs/zpazy2jR0dE8/vjjvPnmm0RFRVG3bl1GjBhBampq/jrOLqOJCG+99Rbjx4+nQYMGhIeH88ADD5CZ6VixcsWKFbRr146goCC6du3KmjVrCAsLY+LEicV+ztzcXF555RViYmIIDAykZcuWfPrppw7r9O7dm+HDhzNt2jSaN29OUFAQhw8fLrI9JyeHiRMn0rRpUwIDA4mLi+OLL75w6fh6i7o1AhjeOYrpt3Vh44QBTB3Vies7NKJWUNF/l57LyWXFzhSenreFrpOXctMHq/jol70cPO5aETnlHdx2ZmOMyRaRB4El2Lo+zzDGxIvI88A6Y8xCEemKbUicusBgEZlkjIkrzziin1pcnpsrs32vXFvySq5ua98+xo0bx4QJE4iMjOSiiy7i6NGj1KtXjzfeeIO6deuya9cuJk6cSEpKCh9++GGx25s9ezbt2rVj2rRpHDp0iEcffZTx48fz/vvvF/u+119/nb59+/LZZ5+xefNmnn76aZo1a8a4ceMASEhI4JprrqFHjx689NJLJCUlMWrUKNLTSx7v66GHHuLTTz9lwoQJdOrUiR9++IExY8ZQv359rrvuuvz1Vq5cyZ49e3j11VcJCQmhTp06RbZPmDCB1157jeeee46uXbsyd+5cRo0ahYgwcuTIYo+vNwoJ8OPqSxpy9SUNOZdtVwZ7WzIpRZTBNgbW7jvB2n0neHHxdmIb1rZ1MGgbQasILYPtzdx6z8YY8w3wTYG2CXav12K7vKZK4dixYyxdupQOHTrkt0VFRfHvf/87f/7yyy+nRo0ajBkzhnfeeafYy0D+/v58/fXX+PnZfhy2bdvGrFmzSkw20dHRfPLJJwAMGjSIlStXMm/evPxkM2XKFEJCQvjf//6X/6Bt7dq1ufnm4sdw3b17N1OnTuXjjz9m9OjRAPTv35/ExEQmTZrkkGxSU1PZtGkTERERDtso2H78+HGmTJnCs88+y7PPPpsf86FDh5g4caJDsnF2fL1dgJ8PV7ZswJUtG/DC0LYul8HelniKbYmneHPpLi2D7eV0hL4qqHHjxoW+CI0xTJkyhdjYWIKDg/H392fUqFFkZmZy4MCBYrfXp0+f/EQDtg4IrgxkOnDgQIf52NhYDh06lD+/du1aBgwY4DCiw5AhQ0r6eCxbtgwfHx9uuOEGsrOz86d+/fqxadMmcnLO94Tq3LlzoUTjrH3r1q2kpaVx0003Oax38803s2vXLux7OTo7vuo8Hx+hc7O6PH1NG5Y/3pslY6/k0QEtiWtU/Fhv+4+lMe3nvdw4dTWXvbyM8fO38NOuFM4VcV9IVS9VojeacuTsy3XKlCk88cQTPPnkk/Tq1Yu6deuydu1aHnjgATIyin6YDyA0NNRhPiAgAGMMmZmZ+PsX3cvI2fvs95WUlES7du0c1gkKCqJmzZrFxnP06FFycnLyL4kVlJiYSFSU7YTY2bFw1p6YmOi03f7Mp0GDBsVuUxUmIrSKrEWryFr8o1+L82Wwtyaxdv/xIgcLTTmdyRe/H+CL3w9oGWwvUe3/V8vzXkll4ey695w5cxg+fDiTJ0/Ob9u2bVuh9SpSZGQkBZ+LysjIyO9lV5R69erh5+fHypUr8fEpfPIdHh6e/7qoewAF2xs2bAjAkSNHqF+/fn57cnJy/j5L2qYqWZN6IdxxxUXcccVFHD2TyVJrsNCVu48VWQb7dEY2CzYdZsGmw1YZ7AYMiougf5sI6moZ7Gqj2icbb5Genk5goGOX088//9xD0dh07dqVjz/+mPT09PxLaQsXltwhsW/fvuTk5HDy5EkGDBhQLrG0bduWkJAQ5syZw4QJ+bcNmT17Ni1btsw/q1HlJ6xmICMubcqIS5tyOiOLFTtT8gcLLaoMdmZ2Lku3J7N0ezK+PsKl0fUYFBfBQC2DXeVpsqkmBgwYwNtvv81ll11G8+bN+fzzz9m9e7dHYxo7dizvvfcegwcP5pFHHiEpKYlXXnmFkJAQp2cseVq1asW9997LiBEjGDduHF26dCEjI4P4+Hh27drFRx99VOpY6tWrx9ixY3nxxRfx8/OjS5cuzJs3j2+++YaZM2deyMdULqgV5M/g9o0Y3L6Ry2Wwc3INq/ceY/XeY0z83zYtg13FabKpJiZMmEBKSkp+T6thw4bx9ttvM3jwYI/F1LhxYxYvXszDDz/MsGHDaNOmDTNmzGDAgAElFg577733aNmyJdOnT2fChAnUrl2b2NhY7rjjjjLH8/zzz+Pn58fUqVNJTk4mJiaGzz77jBEjRpR5m6r0Av186dM6nD6tw5l8g+tlsDcfOsnmQyf515KdWga7Cqp2xdO2b99OmzZtKjAiVRq//vorPXv25Mcff6RPnz6eDsdj9Oe0sPwy2Ftt1Uh3JrtWBrthnSAGxp4vg+2nZbA9ymsqdeovceXy5JNP0rFjRyIjI9m5cycvvPAC9evXZ+PGjcVeSqvu9Oe0ZPuOns0/49lwINWl94SG+NOvdQSD4iK4smUDgvy1DHZF85pKnapyyczM5IknniA5OZlatWoxcOBA3njjDa9ONMo10WE1uKdXc+7p1dzlMtipaVnM3XCIuRsOaRnsSkbPbJTyAP05LTtXy2Dbsy+DPSg2gnAtg+02emajlKoWCpbB/nlXCkvik1m2w7Uy2BMWbKVjk/NlsKO1DHaF0GSjlKqygvx9GRgXycC4SFsZ7L+O5w8WWlQZbGNgw4FUNhxI5eVvd9Aqolb+szxaBtt9NNkopaoFP18fesSE0SMmjIlD4lwug70z+TQ7k0/z9o+7iaobzMBYLYPtDppslFLVTl4Z7LxS2LuPnLGd8cQn8cehk0W+79AJWxnsGSv/on6NAPq3iWBQ2wgujwkj0E97tl0ITTZKqWrPvgz24dR0qzxCMmv2la4M9lVxkfTWMthloslGKeVVGoUGc/vlF3H75Rdx4uw5lm5PZkl8Mr/86XoZ7MutMtj9tQy2y/Rhh0pm8ODBXHLJJUUuf/DBBwkNDS1UftmZoso5v/vuu8W+b9GiRYgI+/btczlugNdee40VK1YUandln0p5Qt0aAdzUpQkfjS5dGezlO1N4at4WLp28lL99sFrLYLtAz2wqmZEjRzJq1Ci2bdtGbGysw7KcnBy++uorhg0bVmiEZ1etXr3abWWOX3vtNR588EF69+5dYftUqryUpQx2roE1+46zZt9xLYNdAk02lczQoUMJCQlh5syZvPDCCw7Lli9fTnJyskMJ49Lq1q3bhYZYJfZZWvZlEFxpd0VWVhY+Pj74+uqN5aqmqDLY38Unsb8UZbCvsrpld2wS6vVlsPUyWiVTo0YNBg8ezJdffllo2axZswgPD6dv377s2LGDESNG0KRJE0JCQoiLi2PKlCnk5hZfYrfgJS1jDBMnTiQ8PJxatWpx2223cerUqULve+qpp7jkkkuoWbMmUVFRjBo1iqSkpPzl0dHRHDt2jEmTJiEiiEj+JTVnl9HeffddWrRoQWBgIDExMbz55psOyydOnEhYWBgbN26kW7duhISE0LFjR3755ZcSj2FGRgbjxo2jSZMmBAYG0r59e7755huHdaKjo3nsscd44YUXiIqKyh+Fuqj2tLQ0/vGPfxAZGUlQUBBdu3bl+++/d9hm7969GT58ONOmTaN58+YEBQVx+PDhEuNVlZt9GewVj/fmu7E9eaR/S2IbllwG+8Of93Lj1FV0e3kZz8zfws9eXAa7+p/ZTHReWrjCTSy6u2VBI0eO5Msvv2T9+vV07twZsP2VPG/ePEaNGoWvry8JCQm0atWKUaNGUatWLTZt2sRzzz1Heno6Tz/9tMv7evvtt3n++ecZP348PXv2ZN68eYwbN67QekeOHGH8+PE0atSIlJQUXn/9dfr27cvWrVvx8fFh/vz59OnTh+HDh3PnnXcCFLoMmGf69Ok89NBDPProowwaNIjly5fz2GOPkZmZyVNPPZW/XlpaGqNHj+aRRx4hMjKSSZMmMWzYMPbv309ISEiRn2n48OGsWbOGSZMm0bx5c2bPns2QIUNYt24dHTp0yF/viy++IC4ujvfff5/s7Oxi2++66y4WLlzISy+9RExMDNOnT+faa69l+fLlXHHFFfnvXblyJXv27OHVV18lJCSkyNLWqmoSEVpH1qZ1ZG0e7m8rg23rUp1cbBnsI6cz+fz3A3xulcHul1cGu1UDQgKq/9cweEOyqYKuvvpqQkNDmTVrVn6yWbJkCSdOnMi/hNavXz/69esH2M5OrrjiCtLS0pg+fbrLySYnJ4dXX32Ve+65hxdffBGAQYMGMWDAABISEhzWnTFjhsP7unfvTlRUFL/++itXXnklHTt2xM/Pj6ioqGIvm+Xm5jJx4kRuv/12Xn/9dQAGDhzIyZMnefnllxk7dixBQbZxq9LT05kyZQp9+/YFbKWdO3bsyM8//8xVV13ldPvLli1j8eLFrFixgl69euVvf9euXUyePJk5c+Y4rL9o0aL8/RXVvn37dmbOnMnHH3/M6NGj849Tu3bteOGFF1iyZEn++1JTU9m0aRMRERFFHgNVfTSpF8KdPS/mzp4Xl6oM9tebDvO1l5XB1stolVBAQADDhg1j9uzZ5A2U+uWXX9KsWTO6d+8O2C4VPffcc8TExBAYGIi/vz/PPPMMf/31l8Nf6cU5ePAgiYmJDB061KF92LBhhdb99ttv6dGjB3Xq1MlPKgC7du0q1Wc7dOgQhw8f5qabbnJov/nmmzl16hRbtmzJbwsICHDobJB3pnTo0KEit7906VIiIyO5/PLLyc7Ozp/69etHwQFc+/Xr5zTRFGxfu3YtxhiHmH18fLjpppv49ddfHd7buXNnTTReKq8M9sf/dynr/9mfd0Z25Lp2DakRUPQ9u7wy2E98tZkuk5cyctpvfLLyLw6nFl1ErqrSM5tKauTIkcyYMYPVq1fTqVMnFixYwP3335/fu+XJJ5/ko48+4rnnnqNTp06EhoayYMECXnzxRTIyMqhZs2aJ+8i75xIeHu7QXnB+7dq1DBkyhBtuuIGnnnqK8PBwRIRu3bqRkeF8/KmiJCYmAhT6Qs6bP378eH5brVq1HEoRBATY/uorbp9Hjx4lKSkJf//CD90VvFFfVFIo2J6YmEjNmjULXbqLiIggLS2NzMzM/N6BmmgUOC+D/d3WJJZud70MdvuoOrZRquMiiQkv+fe5sqv+yaYU90oqkz59+hAREcGsWbNITEzk9OnTDr3Q5syZw0MPPeRwf2Xx4sWl2kdkZCRgux9jr+D8/PnzadCgAV9++WV+stu/f3+p9pWnYcOGTveRnJwMQL169cq03Tz16tWjcePGfP311yWuW1S31ILtDRs25MyZM6SlpTkknOTkZEJCQhy6oWtXV1WQfRnsnFzDun3HWRJvu9yWUMwZzB+HTvKHVQa7uV0Z7HZVtAx29U82VZSvry9/+9vfmDNnDgkJCbRp04b27dvnL09PT3f4ksvJyWHWrFml2keTJk2IjIxkwYIFDvdA5s2b57Beeno6/v7+Dj/gn3/+eaHtBQQElHimExUVRaNGjZgzZw5XX311fvvs2bOpXbt2sQ+0uqJfv368/vrr1KxZk9atW1/QtvJ07doVEeGrr77itttuA2z3yb766iuHzgFKlcTXR7js4vpcdnF9/nldG+IPn8ofOqe4Mth7Us7y/oo9vL9iT5Utg63JphIbOXIk77zzDvPnz2fSpEkOywYMGMB7771HTEwM9erV47333nNpVAF7vr6+jBs3jscff5ywsDB69uzJ3Llz2b59e6F9TZkyhbFjxzJ48GBWrVrFZ599Vmh7rVu3ZvHixVx11VXUrFmTVq1aUatWLYd1fHx8mDhxIvfccw/169dnwIAB/PTTT0ydOpWXXnrJ6T2U0hgwYEB+J4cnn3ySuLg4Tp06xaZNm8jIyODll18u9TbbtGnDyJEjefDBBzl9+jTNmzdn+vTp7Nixg6lTp15QvMp7iQhtG9ehbeM6PDqwlctlsBNPZvDp6v18unp/lSqDXTVSopfq3r070dHRGGMKPcj5zjvv0LNnTx544AHGjBlD27ZtS9XlOc/YsWMZP348H3zwATfeeCNnzpzhtddec1jnmmuu4dVXX2Xu3LkMGTKEn376iUWLFhXa1r/+9S9q1KjBtddeS9euXVm/fr3Tfd5111289dZbzJ8/n+uuu46ZM2fy+uuvO3R7LisRYd68eYwZM4YpU6YwaNAg7rnnHlavXn1BZyHTp09n9OjRPP/88wwdOpT9+/ezaNEiPbNR5SavDPa8+y/n9/H9eOH6tvRsEYZfMQ+D5pXBvvv/rafj8z9w32fr+XpjAifTnReR8yQtC62UB+jPqXLVybQsftyZzJKtyfy0q3RlsAfFRTKwgstga1lopZSqguqE+HNDxyhu6BhVpjLY/6wkZbA12SilVBVRVBnsJfHJJJ0qXRnsQW0jiW1YcWWwNdkopVQVVF5lsPPOeDo3q+vWMtiabJRSqoorXAb7dP6zPJtLKIP9n1//4j+/2spgD7C6VPeIqV/uZbCrZbIxxlTJh56Ud6hqnXJU1RMTXouY8FqlLoM9a+1BZq09yE2do/jXTe2drldW1S7Z+Pv7k56eXuyowEp5Ut5DskpVhLKUwe7bOtxp+4WodskmPDychIQEGjduTHBwsJ7hqErDGEN6ejoJCQk6hpryiLwy2Dd1acLZzGyrZ1sSy3Yc4XSGbQDfQD8ferVqUO77rnbJJq/Y1eHDh8nKqnwPNinv5u/vT0RERP7PqVKeUiPQsQz2aqsMtjHGLTV2ql2yAVvC0V9mpZRyTYCfD71aNqBXy/I/o8mjw9UopZRyO002Siml3E6TjVJKKbfTZKOUUsrtNNkopZRyO002Siml3K7K1bMRkRRgvwurhgFH3RxOVaHH4jw9FjZ6HM7TY3FeeRyLZsaYQn2oq1yycZWIrHNWwMcb6bE4T4+FjR6H8/RYnOfOY6GX0ZRSSrmdJhullFJuV52TzTRPB1CJ6LE4T4+FjR6H8/RYnOe2Y1Ft79kopZSqPKrzmY1SSqlKQpONUkopt6t2yUZErhKRnSKyW0Se8nQ87iYiM0TkiIhstWurJyI/iMif1r91rXYRkbetY7NZRDp5LvLyJyJNRGS5iGwTkXgRedhq97rjISJBIrJGRP6wjsUkq/0iEfnd+sxfikiA1R5oze+2lkd79AOUMxHxFZGNIrLImvfK4wAgIvtEZIuIbBKRdVab239HqlWyERFf4D3gaiAWGCkisZ6Nyu0+Aa4q0PYUsMwY0wJYZs2D7bi0sKa7gakVFGNFyQYeM8bEAt2AB6z/f288HplAX2NMe6ADcJWIdANeBd40xsQAJ4A7rPXvAE5Y7W9a61UnDwPb7ea99Tjk6WOM6WD3TI37f0eMMdVmAroDS+zmnwae9nRcFfC5o4GtdvM7gYbW64bATuv1h8BIZ+tVxwlYAAzw9uMBhAAbgMuwPR3uZ7Xn/74AS4Du1ms/az3xdOzl9PmjrC/QvsAiQLzxONgdj31AWIE2t/+OVKszG6AxcNBu/pDV5m0ijDGJ1uskIK/gvdccH+vyR0fgd7z0eFiXjjYBR4AfgD1AqjEm21rF/vPmHwtr+UmgfoUG7D5TgHFArjVfH+88DnkM8L2IrBeRu602t/+OVMuy0Oo8Y4wREa/q3y4iNYG5wFhjzCkRyV/mTcfDGJMDdBCRUGA+0NqzEVU8EbkOOGKMWS8ivT0cTmVxhTEmQUTCgR9EZIf9Qnf9jlS3M5sEoIndfJTV5m2SRaQhgPXvEau92h8fEfHHlmg+N8bMs5q99ngAGGNSgeXYLheFikjeH5n2nzf/WFjL6wDHKjZSt7gcGCIi+4BZ2C6lvYX3HYd8xpgE698j2P4IuZQK+B2pbslmLdDC6mkSAIwAFno4Jk9YCIy2Xo/Gdu8ir/02q4dJN+Ck3alzlSe2U5j/ANuNMW/YLfK64yEiDawzGkQkGNu9q+3Yks5wa7WCxyLvGA0HfjTWRfqqzBjztDEmyhgTje374EdjzCi87DjkEZEaIlIr7zUwENhKRfyOePpmlRtufl0D7MJ2ffoZT8dTAZ93JpAIZGG7nnoHtmvMy4A/gaVAPWtdwdZbbw+wBeji6fjL+Vhcge169GZgkzVd443HA2gHbLSOxVZggtV+MbAG2A3MAQKt9iBrfre1/GJPfwY3HJPewCJvPg7W5/7DmuLzviMr4ndEh6tRSinldtXtMppSSqlKSJONUkopt9Nko5RSyu002SillHI7TTZKKaXcTpONqvZExIjI63bzj4vIxHLa9iciMrzkNS94PzeJyHYRWV6gPVpEbnH3/pW6UJpslDfIBIaJSJinA7Fn9wS7K+4A7jLG9CnQHg04TTal3L5SbqXJRnmDbGy11R8puKDgmYmInLH+7S0iP4nIAhHZKyKviMgosdWI2SIize02019E1onILmssrrxBMP8lImutOiD32G33FxFZCGxzEs9Ia/tbReRVq20CtgdW/yMi/yrwlleAnlZtkkdE5HYRWSgiPwLLrCfGZ1hxbxSRoSXE11BEfra2t1VEepbxmCvlQP/yUd7iPWCziLxWive0B9oAx4G9wEfGmEvFVpTtIWCstV40tvGlmgPLRSQGuA3b0B5dRSQQWCki31vrdwLaGmP+st+ZiDTCVj+lM7YaK9+LyPXGmOdFpC/wuDFmXYEYn7La85Lc7db22xljjovIS9iGXBljDV+zRkSWAqOKiG8YtuH2J4utPlRIKY6XUkXSZKO8grGN/vxf4B9AuotvW2uscaBEZA+Qlyy2APaXs2YbY3KBP0VkL7bRlQcC7ezOmupgK0B1DlhTMNFYugIrjDEp1j4/B64EvnYx3jw/GGOOW68HYhuI8nFrPghoWkx8a4EZ1oCmXxtjNpVy30o5pclGeZMp2IqIfWzXlo11OVlEfIAAu2WZdq9z7eZzcfzdKTjmk8E2ptRDxpgl9gvENsz92bIEXwr22xfgRmPMzgJxOI3PWnYlcC3wiYi8YYz5r1ujVV5B79kor2H9tT+b8yWAwVa1sLP1egjgX4ZN3yQiPtZ9nIuxVTNcAtxnnSEgIi2tUXaLswboJSJh1iWskcBPJbznNFCrmOVLgIes5IKIdLRrLxSfiDQDko0x04GPsF2SU+qC6ZmN8javAw/azU8HFojIH8B3lO2s4wC2RFEbuNcYkyEiH2G7l7PB+qJPAa4vbiPGmEQReQrb8PcCLDbGLCjuPdhGdc6x4v8E270eey9gO6PbbJ25/QVchy2ROIuvN/CEiGQBZ7Dde1Lqgumoz0oppdxOL6MppZRyO002Siml3E6TjVJKKbfTZKOUUsrtNNkopZRyO002Siml3E6TjVJKKbf7/0z7vx5/NEKRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot([10, 50, 100, 200, 500], training_errors, linewidth=4.0, label='Training error')\n",
    "plt.plot([10, 50, 100, 200, 500], validation_errors, linewidth=4.0, label='Validation error')\n",
    "\n",
    "make_figure(dim=(10,5), title='Error vs number of trees',\n",
    "            xlabel='Number of trees',\n",
    "            ylabel='Classification error',\n",
    "            legend='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question**: Does the training error reduce as the number of trees increases?\n",
    "\n",
    "**A:** yes\n",
    "\n",
    "**Quiz Question**: Is it always true that the validation error will reduce as the number of trees increases?\n",
    "\n",
    "**A:** no"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}