CLA08-NB02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Boosting a decision stump\n",
    "\n",
    "The goal of this notebook is to implement your own boosting module.\n",
    "\n",
    "**Brace yourselves**! This is going to be a fun and challenging assignment.\n",
    "\n",
    "\n",
    "* Use SFrames to do some feature engineering.\n",
    "* Modify the decision trees to incorporate weights.\n",
    "* Implement Adaboost ensembling.\n",
    "* Use your implementation of Adaboost to train a boosted decision stump ensemble.\n",
    "* Evaluate the effect of boosting (adding more decision stumps) on performance of the model.\n",
    "* Explore the robustness of Adaboost to overfitting.\n",
    "\n",
    "Let's get started!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fire up Turi Create"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure you have the latest version of Turi Create"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import turicreate\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting the data ready"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using the same [LendingClub](https://www.lendingclub.com/) dataset as in the previous assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "loans = turicreate.SFrame('lending-club-data.sframe/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting the target and the feature columns\n",
    "\n",
    "We will now repeat some of the feature processing steps that we saw in the previous assignment:\n",
    "\n",
    "First, we re-assign the target to have +1 as a safe (good) loan, and -1 as a risky (bad) loan.\n",
    "\n",
    "Next, we select four categorical features: \n",
    "1. grade of the loan \n",
    "2. the length of the loan term\n",
    "3. the home ownership status: own, mortgage, rent\n",
    "4. number of years of employment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = ['grade',              # grade of the loan\n",
    "            'term',               # the term of the loan\n",
    "            'home_ownership',     # home ownership status: own, mortgage or rent\n",
    "            'emp_length',         # number of years of employment\n",
    "           ]\n",
    "loans['safe_loans'] = loans['bad_loans'].apply(lambda x : +1 if x==0 else -1)\n",
    "loans.remove_column('bad_loans')\n",
    "target = 'safe_loans'\n",
    "loans = loans[features + [target]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Subsample dataset to make sure classes are balanced"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just as we did in the previous assignment, we will undersample the larger class (safe loans) in order to balance out our dataset. This means we are throwing away many data points. We use `seed=1` so everyone gets the same results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage of safe loans                 : 0.5022361744216048\n",
      "Percentage of risky loans                : 0.4977638255783951\n",
      "Total number of loans in our new dataset : 46508\n"
     ]
    }
   ],
   "source": [
    "safe_loans_raw = loans[loans[target] == 1]\n",
    "risky_loans_raw = loans[loans[target] == -1]\n",
    "\n",
    "# Undersample the safe loans.\n",
    "percentage = len(risky_loans_raw)/float(len(safe_loans_raw))\n",
    "risky_loans = risky_loans_raw\n",
    "safe_loans = safe_loans_raw.sample(percentage, seed=1)\n",
    "loans_data = risky_loans_raw.append(safe_loans)\n",
    "\n",
    "print(\"Percentage of safe loans                 :\", len(safe_loans) / float(len(loans_data)))\n",
    "print(\"Percentage of risky loans                :\", len(risky_loans) / float(len(loans_data)))\n",
    "print(\"Total number of loans in our new dataset :\", len(loans_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** There are many approaches for dealing with imbalanced data, including some where we modify the learning algorithm. These approaches are beyond the scope of this course, but some of them are reviewed in this [paper](http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=5128907&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F69%2F5173046%2F05128907.pdf%3Farnumber%3D5128907 ). For this assignment, we use the simplest possible approach, where we subsample the overly represented class to get a more balanced dataset. In general, and especially when the data is highly imbalanced, we recommend using more advanced methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transform categorical data into binary features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this assignment, we will work with **binary decision trees**. Since all of our features are currently categorical features, we want to turn them into binary features using 1-hot encoding. \n",
    "\n",
    "We can do so with the following code block (see the first assignments for more details):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "loans_data = risky_loans.append(safe_loans)\n",
    "for feature in features:\n",
    "    loans_data_one_hot_encoded = loans_data[feature].apply(lambda x: {x: 1})    \n",
    "    loans_data_unpacked = loans_data_one_hot_encoded.unpack(column_name_prefix=feature)\n",
    "    \n",
    "    # Change None's to 0's\n",
    "    for column in loans_data_unpacked.column_names():\n",
    "        loans_data_unpacked[column] = loans_data_unpacked[column].fillna(0)\n",
    "\n",
    "    loans_data = loans_data.remove_column(feature)\n",
    "    loans_data = loans_data.add_columns(loans_data_unpacked)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what the feature columns look like now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['grade.A',\n",
       " 'grade.B',\n",
       " 'grade.C',\n",
       " 'grade.D',\n",
       " 'grade.E',\n",
       " 'grade.F',\n",
       " 'grade.G',\n",
       " 'term. 36 months',\n",
       " 'term. 60 months',\n",
       " 'home_ownership.MORTGAGE',\n",
       " 'home_ownership.OTHER',\n",
       " 'home_ownership.OWN',\n",
       " 'home_ownership.RENT',\n",
       " 'emp_length.1 year',\n",
       " 'emp_length.10+ years',\n",
       " 'emp_length.2 years',\n",
       " 'emp_length.3 years',\n",
       " 'emp_length.4 years',\n",
       " 'emp_length.5 years',\n",
       " 'emp_length.6 years',\n",
       " 'emp_length.7 years',\n",
       " 'emp_length.8 years',\n",
       " 'emp_length.9 years',\n",
       " 'emp_length.< 1 year',\n",
       " 'emp_length.n/a']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features = loans_data.column_names()\n",
    "features.remove('safe_loans')  # Remove the response variable\n",
    "features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train-test split\n",
    "\n",
    "We split the data into training and test sets with 80% of the data in the training set and 20% of the data in the test set. We use `seed=1` so that everyone gets the same result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data, test_data = loans_data.random_split(0.8, seed=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Weighted decision trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's modify our decision tree code from Module 5 to support weighting of individual data points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Weighted error definition\n",
    "\n",
    "Consider a model with $N$ data points with:\n",
    "* Predictions $\\hat{y}_1 ... \\hat{y}_n$ \n",
    "* Target $y_1 ... y_n$ \n",
    "* Data point weights $\\alpha_1 ... \\alpha_n$.\n",
    "\n",
    "Then the **weighted error** is defined by:\n",
    "$$\n",
    "\\mathrm{E}(\\mathbf{\\alpha}, \\mathbf{\\hat{y}}) = \\frac{\\sum_{i=1}^{n} \\alpha_i \\times 1[y_i \\neq \\hat{y_i}]}{\\sum_{i=1}^{n} \\alpha_i}\n",
    "$$\n",
    "where $1[y_i \\neq \\hat{y_i}]$ is an indicator function that is set to $1$ if $y_i \\neq \\hat{y_i}$.\n",
    "\n",
    "\n",
    "### Write a function to compute weight of mistakes\n",
    "\n",
    "Write a function that calculates the weight of mistakes for making the \"weighted-majority\" predictions for a dataset. The function accepts two inputs:\n",
    "* `labels_in_node`: Targets $y_1 ... y_n$ \n",
    "* `data_weights`: Data point weights $\\alpha_1 ... \\alpha_n$\n",
    "\n",
    "We are interested in computing the (total) weight of mistakes, i.e.\n",
    "$$\n",
    "\\mathrm{WM}(\\mathbf{\\alpha}, \\mathbf{\\hat{y}}) = \\sum_{i=1}^{n} \\alpha_i \\times 1[y_i \\neq \\hat{y_i}].\n",
    "$$\n",
    "This quantity is analogous to the number of mistakes, except that each mistake now carries different weight. It is related to the weighted error in the following way:\n",
    "$$\n",
    "\\mathrm{E}(\\mathbf{\\alpha}, \\mathbf{\\hat{y}}) = \\frac{\\mathrm{WM}(\\mathbf{\\alpha}, \\mathbf{\\hat{y}})}{\\sum_{i=1}^{n} \\alpha_i}\n",
    "$$\n",
    "\n",
    "The function **intermediate_node_weighted_mistakes** should first compute two weights: \n",
    " * $\\mathrm{WM}_{-1}$: weight of mistakes when all predictions are $\\hat{y}_i = -1$ i.e $\\mathrm{WM}(\\mathbf{\\alpha}, \\mathbf{-1}$)\n",
    " * $\\mathrm{WM}_{+1}$: weight of mistakes when all predictions are $\\hat{y}_i = +1$ i.e $\\mbox{WM}(\\mathbf{\\alpha}, \\mathbf{+1}$)\n",
    " \n",
    " where $\\mathbf{-1}$ and $\\mathbf{+1}$ are vectors where all values are -1 and +1 respectively.\n",
    " \n",
    "After computing $\\mathrm{WM}_{-1}$ and $\\mathrm{WM}_{+1}$, the function **intermediate_node_weighted_mistakes** should return the lower of the two weights of mistakes, along with the class associated with that weight. We have provided a skeleton for you with `YOUR CODE HERE` to be filled in several places."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "def intermediate_node_weighted_mistakes(labels_in_node, data_weights):\n",
    "    # Sum the weights of all entries with label +1\n",
    "    total_weight_positive = sum(data_weights[labels_in_node == +1])\n",
    "    # Sum the weights of all entries with label -1\n",
    "    ### YOUR CODE HERE\n",
    "    total_weight_negative = sum(data_weights[labels_in_node == -1])\n",
    "    \n",
    "    # Weight of mistakes for predicting all -1's is equal to the sum above\n",
    "    ### YOUR CODE HERE\n",
    "    weighted_mistakes_all_negative = total_weight_positive\n",
    "    # Weight of mistakes for predicting all +1's is equal to the sum above\n",
    "    ### YOUR CODE HERE\n",
    "    weighted_mistakes_all_positive = total_weight_negative\n",
    "    \n",
    "    # Return the tuple (weight, class_label) representing the lower of the two weights\n",
    "    #    class_label should be an integer of value +1 or -1.\n",
    "    # If the two weights are identical, return (weighted_mistakes_all_positive,+1)\n",
    "    ### YOUR CODE HERE\n",
    "    result = None \n",
    "    if weighted_mistakes_all_negative < weighted_mistakes_all_positive:\n",
    "        result = (weighted_mistakes_all_negative, -1)\n",
    "    else:\n",
    "        result =(weighted_mistakes_all_positive, +1) \n",
    "    \n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Checkpoint:** Test your **intermediate_node_weighted_mistakes** function, run the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test passed!\n"
     ]
    }
   ],
   "source": [
    "example_labels = turicreate.SArray([-1, -1, 1, 1, 1])\n",
    "example_data_weights = turicreate.SArray([1., 2., .5, 1., 1.])\n",
    "if intermediate_node_weighted_mistakes(example_labels, example_data_weights) == (2.5, -1):\n",
    "    print('Test passed!')\n",
    "else:\n",
    "    print('Test failed... try again!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that the **classification error** is defined as follows:\n",
    "$$\n",
    "\\mbox{classification error} = \\frac{\\mbox{# mistakes}}{\\mbox{# all data points}}\n",
    "$$\n",
    "\n",
    "**Quiz Question:** If we set the weights $\\mathbf{\\alpha} = 1$ for all data points, how is the weight of mistakes $\\mbox{WM}(\\mathbf{\\alpha}, \\mathbf{\\hat{y}})$ related to the `classification error`?\n",
    "\n",
    "**A:** If all the weights are 1, WM is the count of the mistakes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Function to pick best feature to split on"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We continue modifying our decision tree code from the earlier assignment to incorporate weighting of individual data points. The next step is to pick the best feature to split on.\n",
    "\n",
    "The **best_splitting_feature** function is similar to the one from the earlier assignment with two minor modifications:\n",
    "  1. The function **best_splitting_feature** should now accept an extra parameter `data_weights` to take account of weights of data points.\n",
    "  2. Instead of computing the number of mistakes in the left and right side of the split, we compute the weight of mistakes for both sides, add up the two weights, and divide it by the total weight of the data.\n",
    "  \n",
    "Complete the following function. Comments starting with `DIFFERENT HERE` mark the sections where the weighted version differs from the original implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "def best_splitting_feature(data, features, target, data_weights):\n",
    "    \n",
    "    # These variables will keep track of the best feature and the corresponding error\n",
    "    best_feature = None\n",
    "    best_error = float('+inf') \n",
    "    num_points = float(len(data))\n",
    "\n",
    "    # Loop through each feature to consider splitting on that feature\n",
    "    for feature in features:\n",
    "        \n",
    "        # The left split will have all data points where the feature value is 0\n",
    "        # The right split will have all data points where the feature value is 1\n",
    "        left_split = data[data[feature] == 0]\n",
    "        right_split = data[data[feature] == 1]\n",
    "        \n",
    "        # Apply the same filtering to data_weights to create left_data_weights, right_data_weights\n",
    "        ## YOUR CODE HERE\n",
    "        left_data_weights = data_weights[data[feature] == 0]\n",
    "        right_data_weights = data_weights[data[feature] == 1]\n",
    "                    \n",
    "        # DIFFERENT HERE\n",
    "        # Calculate the weight of mistakes for left and right sides\n",
    "        ## YOUR CODE HERE\n",
    "        left_weighted_mistakes, left_class = intermediate_node_weighted_mistakes(left_split[target], left_data_weights)\n",
    "        right_weighted_mistakes, right_class = intermediate_node_weighted_mistakes(right_split[target], right_data_weights)\n",
    "        \n",
    "        # DIFFERENT HERE\n",
    "        # Compute weighted error by computing\n",
    "        #  ( [weight of mistakes (left)] + [weight of mistakes (right)] ) / [total weight of all data points]\n",
    "        ## YOUR CODE HERE\n",
    "        error = (left_weighted_mistakes + right_weighted_mistakes) / sum(data_weights)\n",
    "        \n",
    "        # If this is the best error we have found so far, store the feature and the error\n",
    "        if error < best_error:\n",
    "            best_feature = feature\n",
    "            best_error = error\n",
    "    \n",
    "    # Return the best feature we found\n",
    "    return best_feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Checkpoint:** Now, we have another checkpoint to make sure you are on the right track."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test passed!\n"
     ]
    }
   ],
   "source": [
    "example_data_weights = turicreate.SArray(len(train_data)* [1.5])\n",
    "if best_splitting_feature(train_data, features, target, example_data_weights) == 'term. 36 months':\n",
    "    print('Test passed!')\n",
    "else:\n",
    "    print('Test failed... try again!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**. If you get an exception in the line of \"the logical filter has different size than the array\", try upgradting your Turi Create installation to 1.8.3 or newer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Very Optional**. Relationship between weighted error and weight of mistakes\n",
    "\n",
    "By definition, the weighted error is the weight of mistakes divided by the weight of all data points, so\n",
    "$$\n",
    "\\mathrm{E}(\\mathbf{\\alpha}, \\mathbf{\\hat{y}}) = \\frac{\\sum_{i=1}^{n} \\alpha_i \\times 1[y_i \\neq \\hat{y_i}]}{\\sum_{i=1}^{n} \\alpha_i} = \\frac{\\mathrm{WM}(\\mathbf{\\alpha}, \\mathbf{\\hat{y}})}{\\sum_{i=1}^{n} \\alpha_i}.\n",
    "$$\n",
    "\n",
    "In the code above, we obtain $\\mathrm{E}(\\mathbf{\\alpha}, \\mathbf{\\hat{y}})$ from the two weights of mistakes from both sides, $\\mathrm{WM}(\\mathbf{\\alpha}_{\\mathrm{left}}, \\mathbf{\\hat{y}}_{\\mathrm{left}})$ and $\\mathrm{WM}(\\mathbf{\\alpha}_{\\mathrm{right}}, \\mathbf{\\hat{y}}_{\\mathrm{right}})$. First, notice that the overall weight of mistakes $\\mathrm{WM}(\\mathbf{\\alpha}, \\mathbf{\\hat{y}})$ can be broken into two weights of mistakes over either side of the split:\n",
    "$$\n",
    "\\mathrm{WM}(\\mathbf{\\alpha}, \\mathbf{\\hat{y}})\n",
    "= \\sum_{i=1}^{n} \\alpha_i \\times 1[y_i \\neq \\hat{y_i}]\n",
    "= \\sum_{\\mathrm{left}} \\alpha_i \\times 1[y_i \\neq \\hat{y_i}]\n",
    " + \\sum_{\\mathrm{right}} \\alpha_i \\times 1[y_i \\neq \\hat{y_i}]\\\\\n",
    "= \\mathrm{WM}(\\mathbf{\\alpha}_{\\mathrm{left}}, \\mathbf{\\hat{y}}_{\\mathrm{left}}) + \\mathrm{WM}(\\mathbf{\\alpha}_{\\mathrm{right}}, \\mathbf{\\hat{y}}_{\\mathrm{right}})\n",
    "$$\n",
    "We then divide through by the total weight of all data points to obtain $\\mathrm{E}(\\mathbf{\\alpha}, \\mathbf{\\hat{y}})$:\n",
    "$$\n",
    "\\mathrm{E}(\\mathbf{\\alpha}, \\mathbf{\\hat{y}})\n",
    "= \\frac{\\mathrm{WM}(\\mathbf{\\alpha}_{\\mathrm{left}}, \\mathbf{\\hat{y}}_{\\mathrm{left}}) + \\mathrm{WM}(\\mathbf{\\alpha}_{\\mathrm{right}}, \\mathbf{\\hat{y}}_{\\mathrm{right}})}{\\sum_{i=1}^{n} \\alpha_i}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building the tree\n",
    "\n",
    "With the above functions implemented correctly, we are now ready to build our decision tree. Recall from the previous assignments that each node in the decision tree is represented as a dictionary which contains the following keys:\n",
    "\n",
    "    { \n",
    "       'is_leaf'            : True/False.\n",
    "       'prediction'         : Prediction at the leaf node.\n",
    "       'left'               : (dictionary corresponding to the left tree).\n",
    "       'right'              : (dictionary corresponding to the right tree).\n",
    "       'features_remaining' : List of features that are posible splits.\n",
    "    }\n",
    "    \n",
    "Let us start with a function that creates a leaf node given a set of target values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_leaf(target_values, data_weights):\n",
    "    \n",
    "    # Create a leaf node\n",
    "    leaf = {'splitting_feature' : None,\n",
    "            'is_leaf': True}\n",
    "    \n",
    "    # Computed weight of mistakes.\n",
    "    weighted_error, best_class = intermediate_node_weighted_mistakes(target_values, data_weights)\n",
    "    # Store the predicted class (1 or -1) in leaf['prediction']\n",
    "    leaf['prediction'] = best_class\n",
    "    \n",
    "    return leaf "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We provide a function that learns a weighted decision tree recursively and implements 3 stopping conditions:\n",
    "1. All data points in a node are from the same class.\n",
    "2. No more features to split on.\n",
    "3. Stop growing the tree when the tree depth reaches **max_depth**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def weighted_decision_tree_create(data, features, target, data_weights, current_depth = 1, max_depth = 10):\n",
    "    remaining_features = features[:] # Make a copy of the features.\n",
    "    target_values = data[target]\n",
    "    print(\"--------------------------------------------------------------------\")\n",
    "    print(\"Subtree, depth = %s (%s data points).\" % (current_depth, len(target_values)))\n",
    "\n",
    "    # Stopping condition 1. Error is 0.\n",
    "    if intermediate_node_weighted_mistakes(target_values, data_weights)[0] <= 1e-15:\n",
    "        print(\"Stopping condition 1 reached.\")\n",
    "        return create_leaf(target_values, data_weights)\n",
    "    \n",
    "    # Stopping condition 2. No more features.\n",
    "    if remaining_features == []:\n",
    "        print(\"Stopping condition 2 reached.\")\n",
    "        return create_leaf(target_values, data_weights)    \n",
    "    \n",
    "    # Additional stopping condition (limit tree depth)\n",
    "    if current_depth > max_depth:\n",
    "        print(\"Reached maximum depth. Stopping for now.\")\n",
    "        return create_leaf(target_values, data_weights)\n",
    "    \n",
    "    splitting_feature = best_splitting_feature(data, features, target, data_weights)\n",
    "    remaining_features.remove(splitting_feature)\n",
    "        \n",
    "    left_split = data[data[splitting_feature] == 0]\n",
    "    right_split = data[data[splitting_feature] == 1]\n",
    "    \n",
    "    left_data_weights = data_weights[data[splitting_feature] == 0]\n",
    "    right_data_weights = data_weights[data[splitting_feature] == 1]\n",
    "    \n",
    "    print(\"Split on feature %s. (%s, %s)\" % (\\\n",
    "              splitting_feature, len(left_split), len(right_split)))\n",
    "\n",
    "    # Create a leaf node if the split is \"perfect\"\n",
    "    if len(left_split) == len(data):\n",
    "        print(\"Creating leaf node.\")\n",
    "        return create_leaf(left_split[target], data_weights)\n",
    "    if len(right_split) == len(data):\n",
    "        print(\"Creating leaf node.\")\n",
    "        return create_leaf(right_split[target], data_weights)\n",
    "    \n",
    "    # Repeat (recurse) on left and right subtrees\n",
    "    left_tree = weighted_decision_tree_create(\n",
    "        left_split, remaining_features, target, left_data_weights, current_depth + 1, max_depth)\n",
    "    right_tree = weighted_decision_tree_create(\n",
    "        right_split, remaining_features, target, right_data_weights, current_depth + 1, max_depth)\n",
    "    \n",
    "    return {'is_leaf'          : False, \n",
    "            'prediction'       : None,\n",
    "            'splitting_feature': splitting_feature,\n",
    "            'left'             : left_tree, \n",
    "            'right'            : right_tree}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a recursive function to count the nodes in your tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_nodes(tree):\n",
    "    if tree['is_leaf']:\n",
    "        return 1\n",
    "    return 1 + count_nodes(tree['left']) + count_nodes(tree['right'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following test code to check your implementation. Make sure you get **'Test passed'** before proceeding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature term. 36 months. (9223, 28001)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9223 data points).\n",
      "Split on feature grade.A. (9122, 101)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (9122 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (101 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (28001 data points).\n",
      "Split on feature grade.D. (23300, 4701)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (23300 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (4701 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "Test passed!\n"
     ]
    }
   ],
   "source": [
    "example_data_weights = turicreate.SArray([1.0 for i in range(len(train_data))])\n",
    "small_data_decision_tree = weighted_decision_tree_create(train_data, features, target,\n",
    "                                        example_data_weights, max_depth=2)\n",
    "if count_nodes(small_data_decision_tree) == 7:\n",
    "    print('Test passed!')\n",
    "else:\n",
    "    print('Test failed... try again!')\n",
    "    print('Number of nodes found:', count_nodes(small_data_decision_tree))\n",
    "    print('Number of nodes that should be there: 7' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us take a quick look at what the trained tree is like. You should get something that looks like the following\n",
    "\n",
    "```\n",
    "{'is_leaf': False,\n",
    "    'left': {'is_leaf': False,\n",
    "        'left': {'is_leaf': True, 'prediction': -1, 'splitting_feature': None},\n",
    "        'prediction': None,\n",
    "        'right': {'is_leaf': True, 'prediction': 1, 'splitting_feature': None},\n",
    "        'splitting_feature': 'grade.A'\n",
    "     },\n",
    "    'prediction': None,\n",
    "    'right': {'is_leaf': False,\n",
    "        'left': {'is_leaf': True, 'prediction': 1, 'splitting_feature': None},\n",
    "        'prediction': None,\n",
    "        'right': {'is_leaf': True, 'prediction': -1, 'splitting_feature': None},\n",
    "        'splitting_feature': 'grade.D'\n",
    "     },\n",
    "     'splitting_feature': 'term. 36 months'\n",
    "}```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'is_leaf': False,\n",
       " 'prediction': None,\n",
       " 'splitting_feature': 'term. 36 months',\n",
       " 'left': {'is_leaf': False,\n",
       "  'prediction': None,\n",
       "  'splitting_feature': 'grade.A',\n",
       "  'left': {'splitting_feature': None, 'is_leaf': True, 'prediction': -1},\n",
       "  'right': {'splitting_feature': None, 'is_leaf': True, 'prediction': 1}},\n",
       " 'right': {'is_leaf': False,\n",
       "  'prediction': None,\n",
       "  'splitting_feature': 'grade.D',\n",
       "  'left': {'splitting_feature': None, 'is_leaf': True, 'prediction': 1},\n",
       "  'right': {'splitting_feature': None, 'is_leaf': True, 'prediction': -1}}}"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "small_data_decision_tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making predictions with a weighted decision tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We give you a function that classifies one data point. It can also return the probability if you want to play around with that as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "def classify(tree, x, annotate = False):   \n",
    "    # If the node is a leaf node.\n",
    "    if tree['is_leaf']:\n",
    "        if annotate: \n",
    "            print(\"At leaf, predicting %s\" % tree['prediction'])\n",
    "        return tree['prediction'] \n",
    "    else:\n",
    "        # Split on feature.\n",
    "        split_feature_value = x[tree['splitting_feature']]\n",
    "        if annotate: \n",
    "            print(\"Split on %s = %s\" % (tree['splitting_feature'], split_feature_value))\n",
    "        if split_feature_value == 0:\n",
    "            return classify(tree['left'], x, annotate)\n",
    "        else:\n",
    "            return classify(tree['right'], x, annotate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating the tree\n",
    "\n",
    "Now, we will write a function to evaluate a decision tree by computing the classification error of the tree on the given dataset.\n",
    "\n",
    "Again, recall that the **classification error** is defined as follows:\n",
    "$$\n",
    "\\mbox{classification error} = \\frac{\\mbox{# mistakes}}{\\mbox{# all data points}}\n",
    "$$\n",
    "\n",
    "The function called **evaluate_classification_error** takes in as input:\n",
    "1. `tree` (as described above)\n",
    "2. `data` (an SFrame)\n",
    "\n",
    "The function does not change because of adding data point weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_classification_error(tree, data):\n",
    "    # Apply the classify(tree, x) to each row in your data\n",
    "    prediction = data.apply(lambda x: classify(tree, x))\n",
    "    \n",
    "    # Once you've made the predictions, calculate the classification error\n",
    "    return (prediction != data[target]).sum() / float(len(data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3981042654028436"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_classification_error(small_data_decision_tree, test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Training a weighted decision tree\n",
    "\n",
    "To build intuition on how weighted data points affect the tree being built, consider the following:\n",
    "\n",
    "Suppose we only care about making good predictions for the **first 10 and last 10 items** in `train_data`, we assign weights:\n",
    "* 1 to the last 10 items \n",
    "* 1 to the first 10 items \n",
    "* and 0 to the rest. \n",
    "\n",
    "Let us fit a weighted decision tree with `max_depth = 2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature home_ownership.RENT. (20514, 16710)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (20514 data points).\n",
      "Split on feature grade.F. (19613, 901)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (19613 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (901 data points).\n",
      "Stopping condition 1 reached.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (16710 data points).\n",
      "Split on feature grade.D. (13315, 3395)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (13315 data points).\n",
      "Stopping condition 1 reached.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (3395 data points).\n",
      "Stopping condition 1 reached.\n"
     ]
    }
   ],
   "source": [
    "# Assign weights\n",
    "example_data_weights = turicreate.SArray([1.] * 10 + [0.]*(len(train_data) - 20) + [1.] * 10)\n",
    "\n",
    "# Train a weighted decision tree model.\n",
    "small_data_decision_tree_subset_20 = weighted_decision_tree_create(train_data, features, target,\n",
    "                         example_data_weights, max_depth=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will compute the classification error on the `subset_20`, i.e. the subset of data points whose weight is 1 (namely the first and last 10 data points)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.05"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subset_20 = train_data.head(10).append(train_data.tail(10))\n",
    "evaluate_classification_error(small_data_decision_tree_subset_20, subset_20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us compare the classification error of the model `small_data_decision_tree_subset_20` on the entire test set `train_data`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.48124865678057166"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_classification_error(small_data_decision_tree_subset_20, train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model `small_data_decision_tree_subset_20` performs **a lot** better on `subset_20` than on `train_data`.\n",
    "\n",
    "So, what does this mean?\n",
    "* The points with higher weights are the ones that are more important during the training process of the weighted decision tree.\n",
    "* The points with zero weights are basically ignored during training.\n",
    "\n",
    "**Quiz Question**: Will you get the same model as `small_data_decision_tree_subset_20` if you trained a decision tree with only the 20 data points with non-zero weights from the set of points in `subset_20`?\n",
    "\n",
    "**A:** yes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (20 data points).\n",
      "Split on feature home_ownership.RENT. (11, 9)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (11 data points).\n",
      "Split on feature grade.F. (10, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (10 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1 data points).\n",
      "Stopping condition 1 reached.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9 data points).\n",
      "Split on feature grade.D. (8, 1)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (8 data points).\n",
      "Stopping condition 1 reached.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 3 (1 data points).\n",
      "Stopping condition 1 reached.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.05"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_data_weights_s20 = turicreate.SArray([1.0 for i in range(len(subset_20))])\n",
    "small_data_decision_tree_s20 = weighted_decision_tree_create(subset_20, features, target,\n",
    "                                        example_data_weights_s20, max_depth=2)\n",
    "evaluate_classification_error(small_data_decision_tree_s20, subset_20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementing your own Adaboost (on decision stumps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a weighted decision tree working, it takes only a bit of work to implement Adaboost. For the sake of simplicity, let us stick with **decision tree stumps** by training trees with **`max_depth=1`**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall from the lecture the procedure for Adaboost:\n",
    "\n",
    "1\\. Start with unweighted data with $\\alpha_j = 1$\n",
    "\n",
    "2\\. For t = 1,...T:\n",
    "  * Learn $f_t(x)$ with data weights $\\alpha_j$\n",
    "  * Compute coefficient $\\hat{w}_t$:\n",
    "     $$\\hat{w}_t = \\frac{1}{2}\\ln{\\left(\\frac{1- \\mbox{E}(\\mathbf{\\alpha}, \\mathbf{\\hat{y}})}{\\mbox{E}(\\mathbf{\\alpha}, \\mathbf{\\hat{y}})}\\right)}$$\n",
    "  * Re-compute weights $\\alpha_j$:\n",
    "     $$\\alpha_j \\gets \\begin{cases}\n",
    "     \\alpha_j \\exp{(-\\hat{w}_t)} & \\text{ if }f_t(x_j) = y_j\\\\\n",
    "     \\alpha_j \\exp{(\\hat{w}_t)} & \\text{ if }f_t(x_j) \\neq y_j\n",
    "     \\end{cases}$$\n",
    "  * Normalize weights $\\alpha_j$:\n",
    "      $$\\alpha_j \\gets \\frac{\\alpha_j}{\\sum_{i=1}^{N}{\\alpha_i}} $$\n",
    "  \n",
    "Complete the skeleton for the following code to implement **adaboost_with_tree_stumps**. Fill in the places with `YOUR CODE HERE`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "from math import log\n",
    "from math import exp\n",
    "\n",
    "def adaboost_with_tree_stumps(data, features, target, num_tree_stumps):\n",
    "    # start with unweighted data\n",
    "    alpha = turicreate.SArray([1.]*len(data))\n",
    "    weights = []\n",
    "    tree_stumps = []\n",
    "    target_values = data[target]\n",
    "    \n",
    "    for t in range(num_tree_stumps):\n",
    "        print('=====================================================')\n",
    "        print('Adaboost Iteration %d' % t)\n",
    "        print('=====================================================')\n",
    "        # Learn a weighted decision tree stump. Use max_depth=1\n",
    "        tree_stump = weighted_decision_tree_create(data, features, target, data_weights=alpha, max_depth=1)\n",
    "        tree_stumps.append(tree_stump)\n",
    "        \n",
    "        # Make predictions\n",
    "        predictions = data.apply(lambda x: classify(tree_stump, x))\n",
    "        \n",
    "        # Produce a Boolean array indicating whether\n",
    "        # each data point was correctly classified\n",
    "        is_correct = predictions == target_values\n",
    "        is_wrong   = predictions != target_values\n",
    "        \n",
    "        # Compute weighted error\n",
    "        # YOUR CODE HERE\n",
    "        weighted_error = sum(alpha[is_wrong == 1]) / sum(alpha)\n",
    "        \n",
    "        # Compute model coefficient using weighted error\n",
    "        # YOUR CODE HERE\n",
    "        weight = 1/2 * log((1-weighted_error)/weighted_error)\n",
    "        weights.append(weight)\n",
    "        \n",
    "        # Adjust weights on data point\n",
    "        adjustment = is_correct.apply(lambda is_correct : exp(-weight) if is_correct else exp(weight))\n",
    "        \n",
    "        # Scale alpha by multiplying by adjustment \n",
    "        # Then normalize data points weights\n",
    "        ## YOUR CODE HERE \n",
    "        alpha = alpha * adjustment / sum(alpha)\n",
    "    \n",
    "    return weights, tree_stumps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking your Adaboost code\n",
    "\n",
    "Train an ensemble of **two** tree stumps and see which features those stumps split on. We will run the algorithm with the following parameters:\n",
    "* `train_data`\n",
    "* `features`\n",
    "* `target`\n",
    "* `num_tree_stumps = 2`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=====================================================\n",
      "Adaboost Iteration 0\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature term. 36 months. (9223, 28001)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9223 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (28001 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 1\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.A. (32094, 5130)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (32094 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (5130 data points).\n",
      "Reached maximum depth. Stopping for now.\n"
     ]
    }
   ],
   "source": [
    "stump_weights, tree_stumps = adaboost_with_tree_stumps(train_data, features, target, num_tree_stumps=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_stump(tree):\n",
    "    split_name = tree['splitting_feature'] # split_name is something like 'term. 36 months'\n",
    "    if split_name is None:\n",
    "        print(\"(leaf, label: %s)\" % tree['prediction'])\n",
    "        return None\n",
    "    split_feature, split_value = split_name.split('.')\n",
    "    print('                       root')\n",
    "    print('         |---------------|----------------|')\n",
    "    print('         |                                |')\n",
    "    print('         |                                |')\n",
    "    print('         |                                |')\n",
    "    print('  [{0} == 0]{1}[{0} == 1]    '.format(split_name, ' '*(27-len(split_name))))\n",
    "    print('         |                                |')\n",
    "    print('         |                                |')\n",
    "    print('         |                                |')\n",
    "    print('    (%s)                 (%s)' \\\n",
    "        % (('leaf, label: ' + str(tree['left']['prediction']) if tree['left']['is_leaf'] else 'subtree'),\n",
    "           ('leaf, label: ' + str(tree['right']['prediction']) if tree['right']['is_leaf'] else 'subtree')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is what the first stump looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                       root\n",
      "         |---------------|----------------|\n",
      "         |                                |\n",
      "         |                                |\n",
      "         |                                |\n",
      "  [term. 36 months == 0]            [term. 36 months == 1]    \n",
      "         |                                |\n",
      "         |                                |\n",
      "         |                                |\n",
      "    (leaf, label: -1)                 (leaf, label: 1)\n"
     ]
    }
   ],
   "source": [
    "print_stump(tree_stumps[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is what the next stump looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                       root\n",
      "         |---------------|----------------|\n",
      "         |                                |\n",
      "         |                                |\n",
      "         |                                |\n",
      "  [grade.A == 0]                    [grade.A == 1]    \n",
      "         |                                |\n",
      "         |                                |\n",
      "         |                                |\n",
      "    (leaf, label: -1)                 (leaf, label: 1)\n"
     ]
    }
   ],
   "source": [
    "print_stump(tree_stumps[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.15802933659263743, 0.17682363293605327]\n"
     ]
    }
   ],
   "source": [
    "print(stump_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your Adaboost is correctly implemented, the following things should be true:\n",
    "\n",
    "* `tree_stumps[0]` should split on **term. 36 months** with the prediction -1 on the left and +1 on the right.\n",
    "* `tree_stumps[1]` should split on **grade.A** with the prediction -1 on the left and +1 on the right.\n",
    "* Weights should be approximately `[0.158, 0.177]` \n",
    "\n",
    "**Reminders**\n",
    "- Stump weights ($\\mathbf{\\hat{w}}$) and data point weights ($\\mathbf{\\alpha}$) are two different concepts.\n",
    "- Stump weights ($\\mathbf{\\hat{w}}$) tell you how important each stump is while making predictions with the entire boosted ensemble.\n",
    "- Data point weights ($\\mathbf{\\alpha}$) tell you how important each data point is while training a decision stump."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training a boosted ensemble of 10 stumps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us train an ensemble of 10 decision tree stumps with Adaboost. We run the **adaboost_with_tree_stumps** function with the following parameters:\n",
    "* `train_data`\n",
    "* `features`\n",
    "* `target`\n",
    "* `num_tree_stumps = 10`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=====================================================\n",
      "Adaboost Iteration 0\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature term. 36 months. (9223, 28001)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9223 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (28001 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 1\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.A. (32094, 5130)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (32094 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (5130 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 2\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.D. (30465, 6759)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (30465 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (6759 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 3\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (19846, 17378)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (19846 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (17378 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 4\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.B. (26858, 10366)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (26858 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (10366 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 5\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.E. (33815, 3409)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (33815 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (3409 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 6\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.A. (32094, 5130)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (32094 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (5130 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 7\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.F. (35512, 1712)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (35512 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (1712 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 8\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.A. (32094, 5130)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (32094 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (5130 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 9\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature emp_length.n/a. (35781, 1443)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (35781 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (1443 data points).\n",
      "Reached maximum depth. Stopping for now.\n"
     ]
    }
   ],
   "source": [
    "stump_weights, tree_stumps = adaboost_with_tree_stumps(train_data, features, \n",
    "                                target, num_tree_stumps=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making predictions\n",
    "\n",
    "Recall from the lecture that in order to make predictions, we use the following formula:\n",
    "$$\n",
    "\\hat{y} = sign\\left(\\sum_{t=1}^T \\hat{w}_t f_t(x)\\right)\n",
    "$$\n",
    "\n",
    "We need to do the following things:\n",
    "- Compute the predictions $f_t(x)$ using the $t$-th decision tree\n",
    "- Compute $\\hat{w}_t f_t(x)$ by multiplying the `stump_weights` with the predictions $f_t(x)$ from the decision trees\n",
    "- Sum the weighted predictions over each stump in the ensemble.\n",
    "\n",
    "Complete the following skeleton for making predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_adaboost(stump_weights, tree_stumps, data):\n",
    "    scores = turicreate.SArray([0.]*len(data))\n",
    "    \n",
    "    for i, tree_stump in enumerate(tree_stumps):\n",
    "        predictions = data.apply(lambda x: classify(tree_stump, x))\n",
    "        \n",
    "        # Accumulate predictions on scores array\n",
    "        # YOUR CODE HERE\n",
    "        scores = scores + (stump_weights[i] * predictions)\n",
    "        \n",
    "    return scores.apply(lambda score : +1 if score > 0 else -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of 10-component ensemble = 0.6203145196036192\n"
     ]
    }
   ],
   "source": [
    "predictions = predict_adaboost(stump_weights, tree_stumps, test_data)\n",
    "accuracy = turicreate.evaluation.accuracy(test_data[target], predictions)\n",
    "print('Accuracy of 10-component ensemble = %s' % accuracy )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us take a quick look what the `stump_weights` look like at the end of each iteration of the 10-stump ensemble:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.15802933659263743,\n",
       " 0.17682363293605327,\n",
       " 0.09311888971195705,\n",
       " 0.0728888552581495,\n",
       " 0.06706306914131716,\n",
       " 0.06456916961613322,\n",
       " 0.05456055779221647,\n",
       " 0.04351093673354489,\n",
       " 0.028988711500059067,\n",
       " 0.0259625096913776]"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stump_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question:** Are the weights monotonically decreasing, monotonically increasing, or neither?\n",
    "\n",
    "**A:** monotonically decreasing, well, actually, the first two are not, so neither\n",
    "\n",
    "**Reminder**: Stump weights ($\\mathbf{\\hat{w}}$) tell you how important each stump is while making predictions with the entire boosted ensemble."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Performance plots\n",
    "\n",
    "In this section, we will try to reproduce some of the performance plots dicussed in the lecture.\n",
    "\n",
    "### How does accuracy change with adding stumps to the ensemble?\n",
    "\n",
    "We will now train an ensemble with:\n",
    "* `train_data`\n",
    "* `features`\n",
    "* `target`\n",
    "* `num_tree_stumps = 30`\n",
    "\n",
    "Once we are done with this, we will then do the following:\n",
    "* Compute the classification error at the end of each iteration.\n",
    "* Plot a curve of classification error vs iteration.\n",
    "\n",
    "First, lets train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=====================================================\n",
      "Adaboost Iteration 0\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature term. 36 months. (9223, 28001)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9223 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (28001 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 1\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.A. (32094, 5130)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (32094 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (5130 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 2\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.D. (30465, 6759)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (30465 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (6759 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 3\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature home_ownership.MORTGAGE. (19846, 17378)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (19846 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (17378 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 4\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.B. (26858, 10366)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (26858 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (10366 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 5\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.E. (33815, 3409)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (33815 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (3409 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 6\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.A. (32094, 5130)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (32094 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (5130 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 7\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.F. (35512, 1712)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (35512 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (1712 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 8\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.A. (32094, 5130)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (32094 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (5130 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 9\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature emp_length.n/a. (35781, 1443)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (35781 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (1443 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 10\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.D. (30465, 6759)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (30465 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (6759 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 11\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.B. (26858, 10366)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (26858 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (10366 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 12\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature emp_length.n/a. (35781, 1443)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (35781 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (1443 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 13\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature emp_length.4 years. (34593, 2631)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (34593 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (2631 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 14\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature emp_length.n/a. (35781, 1443)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (35781 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (1443 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 15\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.C. (27812, 9412)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (27812 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9412 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 16\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.A. (32094, 5130)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (32094 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (5130 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 17\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.F. (35512, 1712)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (35512 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (1712 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 18\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature term. 36 months. (9223, 28001)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9223 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (28001 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 19\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.B. (26858, 10366)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (26858 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (10366 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 20\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature emp_length.n/a. (35781, 1443)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (35781 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (1443 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 21\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.D. (30465, 6759)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (30465 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (6759 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 22\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.F. (35512, 1712)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (35512 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (1712 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 23\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.A. (32094, 5130)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (32094 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (5130 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 24\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature emp_length.n/a. (35781, 1443)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (35781 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (1443 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 25\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature emp_length.2 years. (33652, 3572)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (33652 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (3572 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 26\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.F. (35512, 1712)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (35512 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (1712 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 27\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature home_ownership.OWN. (34149, 3075)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (34149 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (3075 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 28\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature emp_length.n/a. (35781, 1443)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (35781 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (1443 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "=====================================================\n",
      "Adaboost Iteration 29\n",
      "=====================================================\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 1 (37224 data points).\n",
      "Split on feature grade.C. (27812, 9412)\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (27812 data points).\n",
      "Reached maximum depth. Stopping for now.\n",
      "--------------------------------------------------------------------\n",
      "Subtree, depth = 2 (9412 data points).\n",
      "Reached maximum depth. Stopping for now.\n"
     ]
    }
   ],
   "source": [
    "# this may take a while... \n",
    "stump_weights, tree_stumps = adaboost_with_tree_stumps(train_data, \n",
    "                                 features, target, num_tree_stumps=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computing training error at the end of each iteration\n",
    "\n",
    "Now, we will compute the classification error on the **train_data** and see how it is reduced as trees are added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 1, training error = 0.4216365785514722\n",
      "Iteration 2, training error = 0.43343004513217276\n",
      "Iteration 3, training error = 0.4000376101439931\n",
      "Iteration 4, training error = 0.4000376101439931\n",
      "Iteration 5, training error = 0.3847249086610789\n",
      "Iteration 6, training error = 0.3846174511068128\n",
      "Iteration 7, training error = 0.3827638082957232\n",
      "Iteration 8, training error = 0.3846174511068128\n",
      "Iteration 9, training error = 0.3827638082957232\n",
      "Iteration 10, training error = 0.3844831291639802\n",
      "Iteration 11, training error = 0.3827369439071566\n",
      "Iteration 12, training error = 0.38144745325596385\n",
      "Iteration 13, training error = 0.3815280464216635\n",
      "Iteration 14, training error = 0.38056092843326883\n",
      "Iteration 15, training error = 0.3805071996561358\n",
      "Iteration 16, training error = 0.378223726627982\n",
      "Iteration 17, training error = 0.3782774554051149\n",
      "Iteration 18, training error = 0.3784117773479475\n",
      "Iteration 19, training error = 0.3780625402965828\n",
      "Iteration 20, training error = 0.3787610143993123\n",
      "Iteration 21, training error = 0.3795669460563078\n",
      "Iteration 22, training error = 0.3788953363421449\n",
      "Iteration 23, training error = 0.3788953363421449\n",
      "Iteration 24, training error = 0.3787610143993123\n",
      "Iteration 25, training error = 0.3788953363421449\n",
      "Iteration 26, training error = 0.3789759295078444\n",
      "Iteration 27, training error = 0.379110251450677\n",
      "Iteration 28, training error = 0.37892220073071137\n",
      "Iteration 29, training error = 0.3790296582849775\n",
      "Iteration 30, training error = 0.37873415001074573\n"
     ]
    }
   ],
   "source": [
    "error_all = []\n",
    "for n in range(1, 31):\n",
    "    predictions = predict_adaboost(stump_weights[:n], tree_stumps[:n], train_data)\n",
    "    error = 1.0 - turicreate.evaluation.accuracy(train_data[target], predictions)\n",
    "    error_all.append(error)\n",
    "    print(\"Iteration %s, training error = %s\" % (n, error_all[n-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing training error vs number of iterations\n",
    "\n",
    "We have provided you with a simple code snippet that plots classification error with the number of iterations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcAAAAFNCAYAAACXC791AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA/M0lEQVR4nO3deXxcdb3/8dcn+9J0SXe6QqG0pZRaUhE3FikgO4oCooKo6L0uoCACl1sLgoIKv7oAilwWEeWioIByRRCqUra2UOgCLW0p3dt0b5ZmmXx+f5yTdDJJJpM001nyfj4e85g537PM58yk8+n3e77n+zV3R0REpLfJSXUAIiIiqaAEKCIivZISoIiI9EpKgCIi0ispAYqISK+kBCgiIr2SEqCkhJkNNbN/mdkeM7st1fGkmpkVm9mTZrbLzP7QA8c73szWJbjtJWb2wv6+Z29jZnPM7EsdrBtrZm5meQc6LkmcEqAkzMxWm1mtmVWZ2WYzu9/M+nTzcJcBW4G+7n5lD4aZqc4DhgID3f1THW0UJis3s/MPXGg9L/xbOinVcUjvpgQoXXWmu/cBpgEVwPVd2dkCOcAYYKl3YySGLP1f9Rhgubs3drLdxcB24PPJD0kkuykBSre4+3rg/4DJAGb2ATN70cx2mtkbZnZ887ZhU9HNZjYXqAF+Q/BDfnVYmzzJzArNbLaZbQgfs82sMNz/eDNbZ2bfNbNNwH1mNsvM/mBmvw2bUReZ2Xgzu9bMtpjZWjM7OSqGL5jZW+G2q8zsK1Hrmo9/ZbjvRjP7QtT6YjO7zczeC5soXzCz4s7OO5aZTQw/i51mtsTMzgrLbwBmAueHn8cXO9h/DHAcQe35FDMbFhPj/Wa2w8yWAtNj9r3GzFaG57/UzM5te3j7RXh+b5vZx6JWHGRmT5jZdjNbYWZfjloX73sbZGZ/Cc93u5n928xyzOxBYDTwZHi+V3dwvmeY2cJw/xfNbErUutVmdpWZvRnG/L9mVhTvfaPO5VEzqzSzd83sm1HH7NLfVGicmb1qZrvN7HEzK+/gXPqZ2f+Ef1vrzewmM8ttb1s5gNxdDz0SegCrgZPC16OAJcD3gRHANuA0gv9UzQiXB4fbzgHWAEcAeUA+cD9wU9SxbwReBoYAg4EXge+H644HGoFbgUKgGJgF7AVOCY/5G+Bd4L/C438ZeDfq+KcD4wAjSCI1wLSY498Y7ntauH5AuP6O8BxGALnAB8M44p53zGeXD6wArgMKgBOBPcDh4fpZwG87+fz/G3g1fL0IuDJq3S3Av4Hy8LtZDKyLWv8p4KAwzvOBamB4uO6S8Py/FcZ5PrALKA/X/wu4EygCpgKVwIkJfG8/BH4ZHjMf+AhgsX9LHZzr+4AtwDHhZ35xuE9h1P6vhudUDrwFfDXe+4bnvoDgPxsFwCHAKuCUqO+gK39Tc4D1BP8JLAUebf4OgbGAA3nh8p+AX4XbDQlj/0qq/0339kfKA9Ajcx7hj04VsBN4L/xRLAa+CzwYs+3TwMXh6znAjTHr76d1AlwJnBa1fAqwOnx9PFAPFEWtnwU8E7V8ZhhbbrhcFv4A9e/gXP4MXB51/NrmH6uwbAvwgfBHsxY4qp1jxD3vmPKPAJuAnKiy3wOzos6nswT4DnBF+Ppa4I2odauAU6OWLyMqAbZzrIXA2eHrS4ANhMkpLHsV+BxBMo0AZVHrfgjcn8D3diPwOHBoB39L8RLgXYSJNKpsGXBc1P6fjVr3I+CX8d6XIJmuiSm7FrivO39TBH/Xt0RtP4ng7zSXqARIcG23DiiO2vZC4Plk/VvVI7GHmkClq85x9/7uPsbd/9PdawmuX30qbHLaaWY7gQ8Dw6P2W9vJcQ8iSKrN3gvLmlW6+96YfTZHva4Ftrp7JGoZoA+AmX3czF4Om8R2EtTaBkXtv81bX3+rCfcdRFDzWdlOzImcd/T5rXX3pphzHNHOtm2Y2YeAg4GHw6LfAUea2dTo48ccO3r/z0c1J+4kqLVEn/96D3+Zo/Y/KHxsd/c9HcQd73v7MUGt9+9hs/M1iZxraAxwZcxnO4rWfxObol43f1/x3ncMcFDMMa8jSFDNEv6bCsV+5vm0/lyb3zcf2Bj1vr8iqAlKCmVjZwI58NYS1IS+HGebzjq7bCD4oVgSLo8OyxLdv0PhNalHCTqOPO7uDWb2Z4Jmsc5sJWgWGwe8EbMukfNutgEYZWY5UUlwNLA8gX0haAI0YKGZxZYvBDayr1m6+dhAy7XDXwMfA15y94iZLaT1+Y8wM4tKgqOBJ8K4y82sLCoJjiZo+ms+r3a/t3D7KwkS2WTgOTOb5+7/oPPvcy1ws7vf3Ml2bXT0vuEx33X3w7p6zDhGRb0eDTQQ/M1El68lqAEO8s47OckBpBqg9ITfAmea2SlmlmtmRRZ0LBnZhWP8HrjezAab2SCC6zS/7aH4Cgiu2VUCjWb2cSC2M0O7wmR1L3B72IEi18yODZNqV877FYJaytVmlm9BZ5kz2Vej61DYuePTBM2aU6Me3wA+Y0Gv2EeAa81sQPj+34g6RClBwqkMj/cFws5LUYYA3wxj+xQwEXjK3dcSXNf7YXh+U4Avsu+76fB7CzuxHGpBxt5F0JTanPw3E1yD68ivga+a2TEWKDWz082sLIHPq6P3fRXYY0FnquLwO5tsZtPjHjC+z5rZJDMrIWh6/WNUjREAd98I/B24zcz6WtARaJyZHbcf7ys9QAlQ9lv4I3k2QXNSJcH/eL9D1/6+bgLmA28SdPB4LSzrifj2AN8kSBI7gM8Q1G4SdVUY0zyCWxBuJbiWl/B5u3s9QcL7OEEN4U7g8+7+dgLvfw5B89tv3H1T84MgMecBpwI3EDTBvUvwY/tg1HsvBW4DXiJIPEcCc2Pe4xXgsDC2m4Hz3H1buO5CgmtaGwg6c3zP3Z8N18X73g4DniW4jvYScKe7Px+u+yFB4txpZle183nNJ+h08guC72wFwbXKRLT7vmFiOoPgPw/vhud6D9AvweO250GC69mbCJrKv9nBdp8n+I/YUoLz+SPtN5XLAWTu3W5ZEhERyViqAYqISK+kBCgiIr2SEqCIiPRKSoAiItIrKQGKiEivlDU3wg8aNMjHjh2b6jBERCSNLFiwYKu7D25vXdYkwLFjxzJ//vxUhyEiImnEzN7raJ2aQEVEpFdSAhQRkV5JCVBERHolJUAREemVlABFRKRXUgIUEZFeKWtugxCR9LJ79262bNlCQ0NDqkORLJSfn8+QIUPo27dvt4+hBCgiPW737t1s3ryZESNGUFxcTMws9iL7xd2pra1l/fr1AN1OgmoC7aZIk/PIvLXc8OQSFq/flepwRNLKli1bGDFiBCUlJUp+0uPMjJKSEkaMGMGWLVu6fRzVALvp4Xlr+K8/LQbgd6+s4aVrP0Z5aUGKoxJJDw0NDRQXF6c6DMlyxcXF+9XErhpgN/3ptfUtr+sam5i7YmsKoxFJP6r5SbLt79+YEmA37G2I8Oa61s2e26rqUhSNiIh0hxJgNyxev4v6SFOrsu3V9SmKRkR6mpl1+pgzZ063jr169WrMjL/85S9d2m/OnDmYGYsXL+7W+0pbugbYDfNW72hTtk0JUCRrvPTSSy2va2trOfHEE7n++us5/fTTW8onTZrUrWMPHz6cl156iQkTJnRpv2nTpvHSSy8xbty4br2vtKUE2A3zV29vU6YaoEj2+MAHPtDyuqqqCoBx48a1Ko8WiUSIRCIUFHTeEa6wsLDD48TTt2/fbu13oDU0NJCTk0Nubm5C5YnoyufbFWoC7aKmJmfBGtUARXqzSy65hIqKCv785z9zxBFHUFRUxCuvvMLGjRu59NJLOeSQQyguLmb8+PFcf/311Nfv+31orwl07NixXHXVVfy///f/GDlyJAMGDOCCCy5g586dLdu01wRqZvz0pz/luuuuY/DgwQwZMoSvfe1r1NW17pMwZ84cpkyZQlFREdOnT+fVV19l0KBBzJo1K+55NjU1ccstt3DooYdSWFjI+PHjeeCBB1ptc/zxx3Peeedx9913M27cOIqKitiwYUOH5ZFIhFmzZjF69GgKCws54ogj+N3vfpfQ59vTVAPsopWVVeysadvtVjVAkY6NveavqQ4BgNW3nN75Rokea/Vqrr76ambOnMmwYcM4+OCD2bp1K+Xl5dx+++0MGDCA5cuXM2vWLCorK/nVr34V93iPPPIIU6ZM4e6772bdunV8+9vf5rrrruPOO++Mu99tt93GiSeeyG9/+1vefPNNrr32WsaMGcPVV18NwPr16znttNP44Ac/yA9+8AM2bdrERRddRG1tbafn+I1vfIMHHniAmTNnMm3aNJ555hkuvfRSBg4cyBlnnNGy3dy5c1m5ciW33norJSUl9OvXr8PymTNn8qMf/Yjvfe97TJ8+nUcffZSLLroIM+PCCy+M+/n2NCXALpr/XtvaHygBivQ227Zt49lnn2Xq1KktZSNHjuQnP/lJy/KHPvQhSktLufTSS/n5z38etwkvPz+fP//5z+TlBT/LS5cu5eGHH+40AY4dO5b7778fgFNOOYW5c+fy2GOPtSTA2bNnU1JSwpNPPtlyb2bfvn05//zz4x53xYoV3HXXXdx3331cfPHFAJx00kls3LiRG264oVUC3LlzJwsXLmTo0KGtjhFbvn37dmbPns3111/P9ddf3xLzunXrmDVrVqsE2N7n29PUBNpF89q5/gewo6aeSJMf4GhEJFVGjBjR5sfZ3Zk9ezaTJk2iuLiY/Px8LrroIurq6lizZk3c451wwgktyQ+CTjaJjKV68sknt1qeNGkS69ata1meN28eM2bMaDUwwVlnndXZ6fGPf/yDnJwczj33XBobG1seH/vYx1i4cCGRSKRl26OPPrpN8muvfPHixdTU1PCpT32q1Xbnn38+y5cvp7KysqWsvc+3pykBdtGCDmqA7rCzRrVAkd6ivR/82bNnc9VVV3Huuefy+OOP8+qrr3LHHXcAsHfv3rjH69+/f6vlgoIC3L3N9bxE9ot+r02bNjF48OBW2xQVFdGnT5+4x926dSuRSIR+/fqRn5/f8rjkkktobGxk48aNLdu291m0V968T2x5dA2xs2P2JDWBdsGWPXt5b1tNh+u3V9czsE/hAYxIJDP05LW3dNHeKCR/+MMfOO+887j55ptbypYuXXogw2pj2LBhrWpWECTj5t6tHSkvLycvL4+5c+eSk9O2rjRkyJCW1x2NyBJbPnz4cCAYK3bgwIEt5Zs3b255z86O2ZOUALtgQTv3/0XbVl3PYQcoFhFJP7W1tRQWtv5P8EMPPZSiaALTp0/nvvvuo7a2tqUZ9Iknnuh0vxNPPJFIJMKuXbuYMWNGj8QyefJkSkpK+MMf/sDMmTNbyh955BHGjx/fpqaabElNgGZ2KvBTIBe4x91v6WC7TwJ/BKa7+3wzez9wd/NqYJa7/ymZsSaivRvgo6kjjEjvNmPGDH72s59xzDHHMG7cOB566CFWrFiR0piuuOIK7rjjDs4880y+9a1vsWnTJm655RZKSkrardk1O/zww/nqV7/KBRdcwNVXX01FRQV79+5lyZIlLF++nHvuuafLsZSXl3PFFVdw0003kZeXR0VFBY899hhPPfUUv//97/fnNLslaQnQzHKBO4AZwDpgnpk94e5LY7YrAy4Hom/yWAxUuHujmQ0H3jCzJ929MVnxJmLBe607wAwsLWh1/5/uBRTp3WbOnEllZWVLD8dPfOIT/OxnP+PMM89MWUwjRozgr3/9K5dffjmf+MQnmDhxIvfeey8zZszodB69O+64g/Hjx/PrX/+amTNn0rdvXyZNmsQXv/jFbsdz4403kpeXx1133cXmzZs59NBD+e1vf8sFF1zQ7WN2l7knp+eimR1LUHM7JVy+FsDdfxiz3WzgGeA7wFXuPj9m/cHAy8CIeAmwoqLC58+f39Hq/VZT38iRs/7eqqfn+RWj+N/5a1uWv3XSeC4/SY2gIm+99RYTJ05MdRjSgRdeeIGPfOQjPPfcc5xwwgmpDme/dPa3ZmYL3L2ivXXJbAIdAayNWl4HHBMT2DRglLv/1cy+E7PuGOBeYAzwuVTX/hau2dkq+Y0dWMJhQ1v3otperRkhRCT9fPe73+V973sfw4YNY9myZXz/+99nypQpHHfccakOLaVS1gnGzHKA24FL2lvv7q8AR5jZROABM/s/d2/Vj9jMLgMuAxg9enRS4429Ab5ibDkD+7S+qVVNoCKSjurq6vjOd77D5s2bKSsr4+STT+b222+Pew2wN0hmAlwPjIpaHhmWNSsDJgNzwu6uw4AnzOys6GZQd3/LzKrCbVu1cbr73YSdZSoqKpJ6F3rsDfAVYwZQXtq6t5c6wYhIOpo9ezazZ89OdRhpJ5npfx5wmJkdbGYFwAVAS99bd9/l7oPcfay7jyW4zndW2Av0YDPLAzCzMcAEYHUSY40r0uS8vmZnq7KKseUMLG1dA1QCFBHJHEmrAYY9OL8OPE1wG8S97r7EzG4E5rt7vBtRPgxcY2YNQBPwn+6+NVmxdubtTbupqtt3CXJAST7jBpeycVfrkR3UBCoikjmSeg3Q3Z8Cnoopm9nBtsdHvX4QeDCZsXVF7PBnR48px8woj6kB7qiux90PyAgGIulO/xYk2fb3LobefQU0QbE3wE8fOwCAovxc+hTu+z9EY5OzuzalnVVF0kJ+fn5C0+2I7I/a2lry8/O7vb8SYAIWxHaACRMg0KYWuE23QogwZMgQ1q9fT01NzX7/L10klrtTU1PD+vXrW41J2lUaC7QT63fWsiHqWl9BXg6TR/RrWS4vLWDN9n0DZG+vrueQAzucnUjaaR5hZMOGDZ1O5yPSHfn5+QwdOrTT0WziUQLsxPyY2t9RI/tRmJfbshzbE1QdYUQCffv23a8fJ5FkUxNoJ+avbnsDfLTYJlDdCiEikhmUADsRewP89KjrfwDlfZQARUQykRJgHLv3NrBs855WZdNGt06AbZpAq5QARUQygRJgHK+9t4PoDmzjh/ahf0nrhNd2ODT1AhURyQRKgHG0dwN8LHWCERHJTEqAcXR2/Q/auQ9QTaAiIhlBCbADDZEmFq7d2aqsop0aoHqBiohkJiXADizZsJu9DU0ty0PKChlVXtxmu9g5AbeH44GKiEh6UwLsQOwN8NPHlrc7sG9JQR5F+fs+xvpIU6uZI0REJD0pAXYg9gb4o8e0vf7XbKAmxhURyThKgO1wd+a/FzsDRNvrf83aDoitBCgiku6UANvx3rYatlbtu5+vpCCXicPLOty+TUcY9QQVEUl7SoDtiL394X2j+5OX2/FHFXsvoJpARUTSnxJgOxK5AT6amkBFRDKPEmA7ErkBPlrbAbE1HJqISLpTAoyxvbqelZXVLcs5Bu8bHT8Bajg0EZHMowQYI7b5c+LwvvQpjD9vcNsBsZUARUTSnRJgjPnvtb0BvjMaDk1EJPMoAcboyg3wzTQnoIhI5lECjLK3IcKidbtalVV00gEGNCu8iEgmUgKMsmj9Luoj+wbAHtG/mOH92g6AHausMI/83H3jhNY2RKitjyQlRhER6RlKgFG6evtDMzNr515A3QohIpLOlACjLIi9/pdAB5hm6gkqIpJZlABDTU3tDYCdWA0QdC+giEimUQIMraysYldtQ8tyWVEe44d0PAB2LA2ILSKSWZQAQ/Pauf0hJ6ftBLgd0b2AIiKZRQkw1J0b4KOpCVREJLMoAYa6cwN8NA2ILSKSWZQAgS2797Jme03Lcn6ucdTI/l06huYEFBHJLEqAQE19hI9PHsbgsuBWhiMO6kdxQW6XjhF7G4SaQEVE0lv8aQ56ibGDSrnrs0fj7qzZXtOqN2ii1AlGRCSzJLUGaGanmtkyM1thZtfE2e6TZuZmVhEuzzCzBWa2KHw+MZlxRsXBmIGlTOli8yfAoNhrgLoNQkQkrSWtBmhmucAdwAxgHTDPzJ5w96Ux25UBlwOvRBVvBc509w1mNhl4GhiRrFh7Qt+ifHJzjEiTA7CnrpG6xgiFeV1rShURkQMjmTXA9wMr3H2Vu9cDDwNnt7Pd94Fbgb3NBe7+urtvCBeXAMVmVtjOvmkjJ8cYUKJmUBGRTJHMBDgCWBu1vI6YWpyZTQNGuftf4xznk8Br7p729xVoXkARkcyRsk4wZpYD3A5cEmebIwhqhyd3sP4y4DKA0aNH93yQXaSOMCIimSOZNcD1wKio5ZFhWbMyYDIwx8xWAx8AnojqCDMS+BPweXdf2d4buPvd7l7h7hWDBw9Owil0jSbGFRHJHMlMgPOAw8zsYDMrAC4Anmhe6e673H2Qu49197HAy8BZ7j7fzPoDfwWucfe5SYyxR2k4NBGRzJG0BOjujcDXCXpwvgU84u5LzOxGMzurk92/DhwKzDSzheFjSLJi7Sltm0DT/rKliEivldRrgO7+FPBUTNnMDrY9Pur1TcBNyYwtGTQcmohI5tBQaD2ozXBo6gUqIpK2lAB7kHqBiohkDiXAHjRQvUBFRDKGEmAPiq0BqheoiEj6UgLsQQNKCjDbt7yrtoGGSFPqAhIRkQ4pAfag3Byjf3F+q7IdNaoFioikIyXAHqaOMCIimUEJsIcNjLkVQvMCioikJyXAHqaOMCIimUEJsIdpQGwRkcygBNjDNCC2iEhmUALsYRoQW0QkMygB9jD1AhURyQxxE6CZ5ZrZQwcqmGwQ2wtUA2KLiKSnuAnQ3SPAmHBCW0mAaoAiIpkhkfkAVwFzzewJoLq50N1vT1pUGUwDYouIZIZEEuDK8JEDlCU3nMw3oKR1AtxRU09Tk5OTYx3sISIiqdBpAnT3GwDMrE+4XJXsoDJZQV4OZUV57NnbCECTw87ahjZNoyIiklqd9gI1s8lm9jqwBFhiZgvM7Ijkh5a5Yu8F1K0QIiLpJ5HbIO4Gvu3uY9x9DHAl8OvkhpXZ2gyHpp6gIiJpJ5EEWOruzzcvuPscoDRpEWWB8tgBsdURRkQk7STUC9TM/ht4MFz+LEHPUOmAhkMTEUl/idQALwUGA48BjwKDwjLpQOyA2GoCFRFJP3FrgGaWCzzm7iccoHiygjrBiIikv0RGgmkys34HKJ6soDkBRUTSXyLXAKuARWb2DK1Hgvlm0qLKcBoOTUQk/SWSAB8LH5KgQX3UC1REJN0lcg3wEl0D7Bo1gYqIpD9dA0yC2AS4o7oed09RNCIi0h5dA0yCovxcSgtyqa6PANDY5OyubaRfSX6KIxMRkWa6Bpgk5X0KqN5e27K8rbpOCVBEJI0kMhvEA2ZWDIx292UHIKasUF5ayNqoBLi9up5DBqcwIBERaSWR2SDOBBYCfwuXp4aT40ocGg5NRCS9JTIU2izg/cBOAHdfCByStIiyhO4FFBFJb4kkwAZ33xVT1pSMYLJJ2+HQlABFRNJJIp1glpjZZ4BcMzsM+CbwYnLDynyaE1BEJL0lUgP8BnAEUAf8DtgFXJHIwc3sVDNbZmYrzOyaONt90szczCrC5YFm9ryZVZnZLxJ5r3TTtglUA2KLiKSTRHqB1gD/FT4SFo4icwcwA1gHzDOzJ9x9acx2ZcDlwCtRxXuB/wYmh4+MMzB2SiQ1gYqIpJVEaoDd9X5ghbuvcvd64GHg7Ha2+z5wK0HSA8Ddq939heiyTKNZ4UVE0lsyE+AIYG3U8rqwrIWZTQNGuftfkxhHSqgTjIhIektmAozLzHKA24Er9+MYl5nZfDObX1lZ2XPB9YD2BsTWeKAiIumj02uAZjYY+DIwNnp7d7+0k13XA6OilkeGZc3KCK7vzTEzgGHAE2Z2lrvPTyR4d78buBugoqIirbJLSUEuhXk51DUGd4zUNzZRXR+hT2EiHW9FRCTZEvk1fhz4N/AsEOnCsecBh5nZwQSJ7wLgM80rw3sLBzUvm9kc4KpEk1+6MzMGlhawYde+y5jbq+qVAEVE0kQiv8Yl7v7drh7Y3RvN7OvA00AucK+7LzGzG4H57h53ODUzWw30BQrM7Bzg5NgepOmuvE/rBLituo7RA0tSGJGIiDRLJAH+xcxOc/enunrwcJ+nYspmdrDt8THLY7v6fulGPUFFRNJXIp1gLidIgnvNbE/42J3swLKBBsQWEUlfidwIX3YgAslGGhBbRCR9JdQjw8zOAj4aLs5x978kL6TsoQQoIpK+EpkP8BaCZtCl4eNyM/thsgPLBm2aQDUgtohI2kikBngaMNXdmwDM7AHgdeDaZAaWDTQgtohI+kp0JJj+Ua/7JSGOrBQ7ILaaQEVE0kciNcAfAq+b2fOAEVwL7HBqI9kn9jaIrWoCFRFJG4n0Av19OErL9LDou+6+KalRZQl1ghERSV8dNoGa2YTweRownGA2h3XAQWGZdKJvUR75udayXNsQoba+K6PJiYhIssSrAX4buAy4rZ11DpyYlIiyiJkxoKSALXv2dX7ZVl3HyAINhyYikmodJkB3vyx8+XF3bzUxrZkVJTWqLFJe2joBbq+uZ+QAJUARkVRLpBfoiwmWSTtie4JqODQRkfTQYQ3QzIYRzOBebGbvI+gBCsEMDarCJKjNgNjqCSoikhbiXQM8BbiEYCLb26PK9wDXJTGmrBI7Gox6goqIpId41wAfAB4ws0+6+6MHMKasEnsrhJpARUTSQyL3AT5qZqcDRwBFUeU3JjOwbNF2NBgNhyYikg4SGQz7l8D5wDcIrgN+ChiT5LiyhppARUTSUyK9QD/o7p8Hdrj7DcCxwPjkhpU9YjvBqAlURCQ9JJIAa8PnGjM7CGggGBlGEqDh0ERE0lMig2H/xcz6Az8GXiMYBeaeZAaVTdo0geo2CBGRtJBIJ5jvhy8fNbO/AEXuviu5YWWPfsX55OYYkSYHYE9dI3WNEQrzclMcmYhI75ZIJ5ivhTVA3L0OyDGz/0x2YNkiJ8cYUJLfqmxHdUOKohERkWaJXAP8srvvbF5w9x3Al5MWURZqey+gboUQEUm1RBJgrpm1zOljZrlAQZztJYY6woiIpJ9EOsH8DfhfM/tVuPyVsEwSNDB2PFAlQBGRlEskAX6XIOn9R7j8DOoF2iVtmkDVE1REJOUS6QXaBNwVPqQb1AQqIpJ+4k2H9Ii7f9rMFhHc+9eKu09JamRZRHMCioikn3g1wCvC5zMOQBxZrW0NUL1ARURSLV4C/AswDbjJ3T93gOLJSmoCFRFJP/ESYIGZfQb4oJl9Inaluz+WvLCyS2wvUDWBioikXrwE+FXgIqA/cGbMOgeUABOkGqCISPqJNyP8C8ALZjbf3f/nAMaUdWKHQttZ00BjpIm83ETGIRARkWSI1wv0RHd/DtihJtD9k5ebQ/+SfHbW7BsDdEdNA4PLCuPsJSIiyRSvCfQ44DnaNn+CmkC7rLy0oFUC3FZdpwQoIpJC8ZpAvxc+f+HAhZO9BpYWsKqyumVZ8wKKiKRWItMhXW5mfS1wj5m9ZmYnJ3JwMzvVzJaZ2QozuybOdp80Mzeziqiya8P9lpnZKYmdTvpqOyOEEqCISColMhbope7+0zAJDQQ+BzwI/D3eTuGsEXcAM4B1wDwze8Ldl8ZsVwZcDrwSVTYJuAA4AjgIeNbMxrt7JOEzSzPlMbdC3Df3XeYsq+x0PzOYOLwvnz92DPnqNCMi0mMSSYDNUyGdBvzG3ZdET48Ux/uBFe6+CsDMHgbOBpbGbPd94FbgO1FlZwMPhxPwvmtmK8LjvZTA+6algTE1wNfW7OS1NTsT3n/z7r1cd9rEHo5KRKT3SqRKscDM/k6QAJ8Oa2xNCew3AlgbtbwuLGthZtOAUe7+167um2mG9Svar/3/b/HGHopEREQgsQT4ReAaYLq71wD5wH53jDGzHOB24Mr9OMZlZjbfzOZXVnbenJhKJx8xlCH70etz7fZaquoaezAiEZHeLZEm0GOBhe5ebWafJRgf9KcJ7LceGBW1PDIsa1YGTAbmhC2qw4AnzOysBPYFwN3vBu4GqKioaDNjRToZUlbE/13+EV5atY3a+sQuZc5+9h3W76xtWV62aQ9HjxmQrBBFRHqVRBLgXcBRZnYUQW3tHuA3BPcJxjMPOMzMDiZIXhcAn2le6e67gEHNy2Y2B7jK3eebWS3wOzO7naATzGHAq4meVLoa2KeQM6YclPD2c5ZXtkqAb2/arQQoItJDEmkCbXR3J+iY8gt3v4Og9haXuzcCXweeBt4CHgk70NwY1vLi7bsEeISgw8zfgK9lcg/Q7po4rPXH/PbGPSmKREQk+yRSA9xjZtcCnwU+Gl67y+9kHwDc/SngqZiymR1se3zM8s3AzYm8T7aaMKxvq+W3N+1OUSQiItknkRrg+UAd8EV330RwPe7HSY1KAJgwvG0NMKiMi4jI/uo0Abr7Jne/3d3/HS6vcfffJD80GdG/mLLCfZX0PXWNra4JiohI9yUyFNoHzGyemVWZWb2ZRcxs14EIrrczMw6PuQ64bJOuA4qI9IREmkB/AVwIvAMUA18C7kxmULJPm2ZQJUARkR6R0OCS7r4CyHX3iLvfB5ya3LCkWWxHmLc2qiOMiEhPSKQXaI2ZFQALzexHwEYSTJyy/yaqBigikhSJJLLPAbkE9/RVE4zQ8slkBiX7jB/aOgGuqqxib0OvuyVSRKTHdVoDdPf3wpe1wA3JDUdilRXlM6q8mLXbg96fTQ4rtlQxeUS/FEcmIpLZOkyAZrYI6PCmM3efkpSIpI0Jw/q2JEAIrgMqAYqI7J94NcAzDlgUEteEYWU8s3Rzy7JuhRAR2X/xEmA+MNTd50YXmtmHgE1JjUpaaTskmhKgiMj+itcJZjbQXp/73eE6OUDa3guoWyFERPZXvAQ41N0XxRaGZWOTFpG0MXZgKYV5+76qrVX1VO6pS2FEIiKZL14C7B9nXXEPxyFx5Oa0HRJNtUARkf0TLwHON7Mvxxaa2ZeABckLSdozQXMDioj0qHidYK4A/mRmF7Ev4VUABcC5SY5LYrQZEk01QBGR/dJhAnT3zcAHzewEYHJY/Fd3f+6ARCattDc3oIiIdF8iI8E8Dzx/AGKROGJrgCu2VNEYaSIvV8Oyioh0h349M0R5aQFDygpblusjTby7tTqFEYmIZDYlwAwyYXjsdUA1g4qIdJcSYAaZ2KYnqDrCiIh0lxJgBtHs8CIiPUcJMIO0GRNUNUARkW5TAswg4wb3IS/HWpY37NrLrpqGFEYkIpK5lAAzSEFeDuMG92lVpiHRRES6Rwkww8ReB1y2WdcBRUS6Qwkww7QZEk0jwoiIdIsSYIbR3IAiIj1DCTDDTIypAS7btIemJk9RNCIimUsJMMMM7VtI/5L8luWa+ghrd9SkMCIRkcykBJhhzKzN3IC6Digi0nVKgBmozQ3xug4oItJlSoAZKLYGuExDoomIdJkSYAaKnRVCY4KKiHSdEmAGGj+0D7ZvRDRWb6umpr4xdQGJiGSgpCZAMzvVzJaZ2Qozu6ad9V81s0VmttDMXjCzSWF5gZndF657w8yOT2acmaakII+xA0tblt1h+eaqFEYkIpJ5kpYAzSwXuAP4ODAJuLA5wUX5nbsf6e5TgR8Bt4flXwZw9yOBGcBtZqbaapTY64CaGUJEpGuSmVTeD6xw91XuXg88DJwdvYG7R/9qlwLNd3RPAp4Lt9kC7AQqkhhrxmnbE1TXAUVEuiKZCXAEsDZqeV1Y1oqZfc3MVhLUAL8ZFr8BnGVmeWZ2MHA0MCqJsWac2CHR3lINUESkS1LerOjud7j7OOC7wPVh8b0ECXM+MBt4EYjE7mtml5nZfDObX1lZeYAiTg9tmkA37cFdQ6KJiCQqmQlwPa1rbSPDso48DJwD4O6N7v4td5/q7mcD/YHlsTu4+93uXuHuFYMHD+6xwDPBqAEllBTktizvqm1g8+66FEYkIpJZkpkA5wGHmdnBZlYAXAA8Eb2BmR0WtXg68E5YXmJmpeHrGUCjuy9NYqwZJyfHODx2SDSNCCMikrCkJUB3bwS+DjwNvAU84u5LzOxGMzsr3OzrZrbEzBYC3wYuDsuHAK+Z2VsETaOfS1acmaxNRxiNCSoikrC8ZB7c3Z8Cnoopmxn1+vIO9lsNHJ7M2LLBRM0NKCLSbSnvBCPdpxqgiEj3KQFmsNhrgCsrq6hrbNNZVkRE2qEEmMH6Feczon9xy3Jjk7NyS3UKIxIRyRxKgBkuthao64AiIolRAsxwmhtQRKR7lAAzXOzcgG8pAYqIJEQJMMNN1KwQIiLdogSY4Q4eVEpB7r6vccueOrZVaUg0EZHOKAFmuLzcHA4b2qdVma4Dioh0TgkwC8TeEK/rgCIinVMCzAJthkTTdUARkU4pAWaB2HsBl21WDVBEpDNKgFkgtgl02aY9RJo0Oa6ISDxKgFlgcFkhg/oUtCzXNTaxepuGRBMRiUcJMEtoZggRka5RAswSsUOiaUxQEZH4kjohrhw4sUOizVu9ncXrd1GUn0txQS7F+cGjMC+HnBxLUZQiIulDCTBLxNYAX161nTN+/kK72xbl51BSkEdxfi5F+TkMLivkE+8byXlHj+zR5OjuPLVoE39csJbh/Yv5j+PGMaq8pMeO32xvQ4TKPXVJObaIZC8lwCxx6JA+5OZYQr0/9zY0sbehvmV5ZWU1L6/azuNvrOfWT05h5ID9TyRbq+r4rz8t4uklm1vKHn99PdefMYkLpo/CbP8TbUOkifvmvstPn32H6voIU0f15wfnHsmkg/p2vrOI9Hrmnh3d5SsqKnz+/PmpDiOl/vOhBTy1aNN+HaNPYR7Xnz6R8/cjST21aCPX/3kx26vr211/3PjB3PLJIxner7jd9Yl4fc0OrvvTYt6Kuek/N8f48kcO4YqTDqMoP7fbxxeR7GBmC9y9ot11SoDZY2tVHffNfZc31u6itiFCbX2EvQ2R4HW4XNfYlNCxjj98MLd+cgpD+xYl/P47quuZ+cQSnnxjQ6fblhXlMevMI/jEtBFdSrR79jbw46eX8eDL7xHvT3fMwBJ+cO6RfOjQQQkfW0SyjxKgtGhqcvY2BsmwtiHCjuoGbvnbW8xdsa3Ntn2L8rjh7CM4Z2rnSerZpZu59k+LqNzTdiaKCcPKWLO9hpr6SJt1J00cyg8+MZkhZfETrbvzt8WbmPXkEjbvTny2i09OG8n1p09kQGlB5xuLSNZRApS4mpqch155jx889Ta1DW2T1MmThnLzuUcyuKywzbpdtQ3c+ORSHn1tXZt1+bnGFSeN5ysfPYQNO/dy1R/f4NV3t7fZrn9JPt8/ezJnHnVQu/Gt31nL9x5fzLNvbWl3/TlTD+KECUO45f/eZuOuvW3WDywtYOaZkzjrqIN65NqjiGQOJUBJyHvbqvnOH97k1dVtk9SAknxuOudITp8yvKXsn8sruebRN9tNOpOG9+W2Tx/FxKjbM5qanPtfXM2tf3u73abY048czo1nH8HAPkGibYw0cf+Lq7n9meXt1h7HDCzhpnMm85HDBgNB8+hPnl7GbzpoHj1u/GBuOmeyeouK9CJKgJKwSJNz39x3+fHTy9pNUmdMGc53T53AnXNW8vtX17RZn5tjfO2EQ/n6CYdSkNf+OAurKqu48g9v8PqanW3WDepTwE3nHMlB/Yu49rFFLNnQ9ob+vBzjK8cdwjdObL+jy4L3dnDtY2+yfHNVm3XF+blcefJ4vvChg8nV/ZAiWU8JULpsxZYqrvrDGyxcuzPhfcYP7cNtn5rKkSP7dbptpMn59b9Xcfvfl1MfaZtozWi3Fnf0mAH88BNHMn5oWduVUeobm/jVP1fy8+dWtHv8KSP78YNzj2TyiM5jFZHMpQQo3dIYaeLuf69i9jPvtJtEmuUYXPbRcXxrxmEU5nXt1oPlm/dw5SNvsGj9rrjblRXlce3HJ3LB9FFdull/ZWUV1z62qN1rjwAfmzCE/zh+HBVjy7sUt4hkBiVA2S/LNu3h248sbLc58pBBpfzk00cxbfSAbh+/IdLEXXNW8rN/vENjOzfyn3nUQfz3GRM77SnakaYm53/nr+UHT73Fnr2N7W7z/rHl/Mfx4zj+8MHqKCOSRZQAZb81RJq44/kV/OK5FTQ2OWZw6YcO5qqTD6e4oGduOF+yYRdXPvIGb28KZrIYVV7M98+ezPGHD+mR42/ZvZdZTy6JO1jAhGFl/Mfx4zj9yOHk5WqseJFMpwQoPWbNthpeXLmVirEDOHRI/Otw3VHf2MQ/l1fSGGni+MOH9Fhyjfb821uY/Y93eCPO9c3R5SVc9tFDOO/okRpRRiSDKQGKxHB3Xl61nTvnrODf72ztcLtBfQq59MNj+ewHxtC3KD/h4zeFtWQ1p4qklhKgSByL1+/irn+u5KlFGzscXq2sMI9jDhlIQ6SJ+sYm6hoj1EeaqGtoanmua4yE65pobHL6l+Rz0sShnDN1BMeOG6jbLkRSQAlQJAHvbq3m7n+t5NEF6+P2eu2OwWWFnDFlOGdPHcFRI/upZihygCgBinTB5t17+Z8X3uWhl9+jup0RaPbX2IElnDV1BGdPPYhxg/v0+PFFZB8lQJFu2FXTwIMvr+beuas7nNppf00e0ZezjxrBmUcdxLB+3bvNQ0Q6pgQosh9q6yPMf2871XWNFOblUpiXQ0FeDoV5ueFzTstzYX4uBbk55Bi8tGobjy/cwN8Wb6Kqrv37D5uZwbTRAygvLSA/18jPzSEvJ4f8XCMv16Je55CfEzyXFeVx8hHDGNG/+/MqimS7lCVAMzsV+CmQC9zj7rfErP8q8DUgAlQBl7n7UjPLB+4BphHMWv8bd/9hvPdSApR0tbchwnNvb+Hxhet5/u3KHr2+mJdj3HD2EVx0zJgeO6ZINklJAjSzXGA5MANYB8wDLnT3pVHb9HX33eHrs4D/dPdTzewzwFnufoGZlQBLgePdfXVH76cEKJlgV00Df1uykccXbuClVdviTurbFZd8cCzXnz5RN++LxIiXAPOS+L7vB1a4+6owiIeBswmSGQDNyS9UCjT/HDhQamZ5QDFQD7Qdh0skw/Qryef86aM5f/poNu3ay1/e3MDjCzd0OhZqZ+5/cTUrK6v4xYXT6FeS+P2KIr1ZMhPgCGBt1PI64JjYjczsa8C3gQLgxLD4jwTJciNQAnzL3dsfzVgkQw3rV8SXPnIIX/rIIazeWs3KyioaIk00RJzGpvC51evg/sKGSBO19REefPm9VvMk/vudrZx751z+55LpHDyoNIVnlh4iTc6KLVU4zqGD+6h2LG0kMwEmxN3vAO4Imz2vBy4mqD1GgIOAAcC/zezZ5tpkMzO7DLgMYPTo0Qc0bpGeNHZQKWO7mLTOnjqCLz0wjw1RExKv2lrNOXfM5c6LpvGhQwf1dJhpb1dtA/9aXsnzb29hzvLKlt67fQrzqBg7gGMOHsgxh5Rz5Ih+5Csh9nrJvAZ4LDDL3U8Jl68F6Kgzi5nlADvcvZ+Z3QG87O4PhuvuBf7m7o909H66Bii9UeWeOr7y4Hxei5lcODfHmHXmJD537NiUxHWguAe1vOfe3sJzb29h/ns7iLQzo0is4vxcjh4zgGMOLuf9B5dz1Kj+vWbMV3entiFCdV2E6rpGqusbqWtsoqwwj37F+fQtzs+qzyJV1wDnAYeZ2cHAeuAC4DMxgR3m7u+Ei6cDza/XEDSHPmhmpcAHgNlJjFUkIw0uK+R3X/4A1z22iMdeX99SHmly/vvxJbyzpYqZZ0zKqua/vQ0RXl61jeff3sI/3t7Cuh21XT5GbUOEF1Zs5YUVwTiwBXk5vG9Uf44Jk2FBXuKfl2EUF+RSWphLaUEepYV5lBQEt8t0dcSfSJNTXd8YJKboBBUOtVcXDrVX1xg1JF87y7UNTdTUNVJV10hN/b7jVNdFqK5v7LTzVVF+Dv2LC+hXnE+/knz6FefTvzh8LgmSZFOTU9vQRG1DhL0NEWrrI9Q2BI+9Ua9r64P1hXm5DOtXxLC+RQzrV8TwfkUMDZ+H9y2mb3HeAR8hKdm3QZxGkLhygXvd/WYzuxGY7+5PmNlPgZOABmAH8HV3X2JmfYD7gEmAAfe5+4/jvZdqgNKbuTu//OcqfvT0221+3D586CDu+Ezmdo5xd5ZvruLFlVuZu2Irc1dso7YhsRF6yoryyM/NSdpABvHk5RglBbn0KcyjpDBIjKUFuZQU5FEfaT9B7W3o2SH4MklRfg7D+xUztG8hw/sVtyTLEycMYVR5SbePqxvhRXqJZ5Zu5vKHX2/VOQbg4EGl3HNxRVKGXnN36hqbqK5rxIHykgJy9mPgb3dnZWU1L63axssrt/Hyqm1s60ICO2xIH06cMIQTJgzh6DEDyMsxVmyp4uV3t/Pqu9t5ZdU2tuyp63Z8cmDd94XpnLAfc4IqAYr0Im9t3M2XHpjP+p2tmwb7FuVx/RmT6Fec36Z3aUNT2Ms04jQ0Bc+NkSb2NjYFtZS6RqrqItQ0N88111rC19HX3Qrychg5oJiRA0oYNaCYUeUljBxQzKgBJYwqL2FASX6rpi53Z832Gl5auY2XVm3jpZVdS1AFeTkce8hATpwwJKHagruzelsNr6zaFiTEd7e3+ayyXWFeTlgzDZptC/Ny2LO3kV21DeysbUjoOuqB8rcrPsKEYX27vb8SoEgvs7Wqjq8+uID57+1IdShtlBbkMnJAkBRLC/OYv3p7q56siRjWt4gTJgzhYxOG8MFDB1JSsH/dGdZur+HVsIbY1WQYaWruVBI0Z1aF/zFo7EYSMYPSgryoptMgQRXl57Yaaq8wP6fluTC3bXlRfm5Lk2tpYV5wfbIwj5KCoCzeNWF3p7o+ws6aenbWNLA7TIq7ahvYWRM876ptID/XKM7PpSg/l+KCXIrzg0dRQS4lYVlRc1l+DtV1ETbuqmXT7r1s2hU8Nu7ay+bdwXNHzdoLZ86gf0lBlz/LfZ+pEqBIr1PXGOG6xxbz6GvrUh3KfutTmMf7Dy7n2EMG8qFDBzFxeFnaTylV1xihpi6y7zpfVOeWwrycls4yzYmuT2EeRXm5+9V8nKncnd17G8OkWNuSFCv31HHTOZP367tOVS9QEUmhwrxcfvKpKYwf2odb/ta2c0xPKsjNoaQwl0jE2dPJwN+JKM7PpWLsAI4dN5APjhvE5IP6ZlxP1mDg9FwGlHa/9tJbmFnQ47Q4n8OHlR2w91UCFMliZsZXjhvH5BH9ePCl99jbGGlnZonWs0zk5Rr5OcFzYV4ufaKaz6JrKy21l4K8VrcN7KptYO32GtbtqGHdjtrwdS1rd9Swdnttu01dhXk5HD1mAMceMpBjxw1kysiu3Yog0h1KgCK9wIcOHXTARobpV5xPvxH9mDyiX5t17s726nrW7qhl3Y4adtQ0cOjgPrxvdO+5EV3ShxKgiBwwZsbAPoUM7FPI1FH9Ux2O9HJqYxARkV5JCVBERHolJUAREemVlABFRKRXUgIUEZFeSQlQRER6JSVAERHplZQARUSkV1ICFBGRXilrZoMws0rgvXZWDQK2HuBwUk3n3Hv0xvPWOfcOPXXOY9x9cHsrsiYBdsTM5nc0FUa20jn3Hr3xvHXOvcOBOGc1gYqISK+kBCgiIr1Sb0iAd6c6gBTQOfcevfG8dc69Q9LPOeuvAYqIiLSnN9QARURE2sjaBGhmp5rZMjNbYWbXpDqeA8XMVpvZIjNbaGbzUx1PMpjZvWa2xcwWR5WVm9kzZvZO+DwglTH2tA7OeZaZrQ+/64VmdloqY+xpZjbKzJ43s6VmtsTMLg/Ls/a7jnPO2f5dF5nZq2b2RnjeN4TlB5vZK+Hv+P+aWUGPvm82NoGaWS6wHJgBrAPmARe6+9KUBnYAmNlqoMLds/aeITP7KFAF/MbdJ4dlPwK2u/st4X94Brj7d1MZZ0/q4JxnAVXu/pNUxpYsZjYcGO7ur5lZGbAAOAe4hCz9ruOc86fJ7u/agFJ3rzKzfOAF4HLg28Bj7v6wmf0SeMPd7+qp983WGuD7gRXuvsrd64GHgbNTHJP0EHf/F7A9pvhs4IHw9QMEPxpZo4NzzmruvtHdXwtf7wHeAkaQxd91nHPOah6oChfzw4cDJwJ/DMt7/LvO1gQ4AlgbtbyOXvBHFHLg72a2wMwuS3UwB9BQd98Yvt4EDE1lMAfQ183szbCJNGuaAmOZ2VjgfcAr9JLvOuacIcu/azPLNbOFwBbgGWAlsNPdG8NNevx3PFsTYG/2YXefBnwc+FrYdNareNCun31t+23dBYwDpgIbgdtSGk2SmFkf4FHgCnffHb0uW7/rds45679rd4+4+1RgJEEr3oRkv2e2JsD1wKio5ZFhWdZz9/Xh8xbgTwR/SL3B5vD6SfN1lC0pjifp3H1z+KPRBPyaLPyuw+tBjwIPuftjYXFWf9ftnXNv+K6buftO4HngWKC/meWFq3r8dzxbE+A84LCwB1EBcAHwRIpjSjozKw0vnGNmpcDJwOL4e2WNJ4CLw9cXA4+nMJYDojkJhM4ly77rsGPE/wBvufvtUauy9rvu6Jx7wXc92Mz6h6+LCTowvkWQCM8LN+vx7zore4EChN2EZwO5wL3ufnNqI0o+MzuEoNYHkAf8LhvP28x+DxxPMFr8ZuB7wJ+BR4DRBLOCfNrds6bTSAfnfDxBk5gDq4GvRF0by3hm9mHg38AioCksvo7gmlhWftdxzvlCsvu7nkLQySWXoGL2iLvfGP6mPQyUA68Dn3X3uh5732xNgCIiIvFkaxOoiIhIXEqAIiLSKykBiohIr6QEKCIivZISoIiI9EpKgCI9zMx+aGYnmNk5ZnZtF/cdHI5+/7qZfSRm3T1mNil8fV0Px3yJmR3U3nuJZCvdBiHSw8zsOeB04AfAH919bhf2vQA4yd2/1Ml2Ve7ep4tx5bp7pIN1c4Cr3D0rp9ASaY9qgCI9xMx+bGZvAtOBl4AvAXeZ2cx2th1rZs+Fgxv/w8xGm9lU4EfA2eGcb8Ux+8wxswozuwUoDrd5KFz32XA+tYVm9qtwSjDMrMrMbjOzN4BjzWymmc0zs8VmdrcFzgMqgIea37f5vcJjXGjBHJOLzezWqHiqzOxmC+Zwe9nMhoblnwq3fcPM/tXjH7RIT3F3PfTQo4ceBMnv5wTTucyNs92TwMXh60uBP4evLwF+0cE+cwjmeoRgbrjm8onh8fLD5TuBz4evnWCklOZty6NePwicGXvs6GXgIGANMJhgdKHngHOijt28/4+A68PXi4AR4ev+qf5O9NCjo4dqgCI9axrwBsFI9m/F2e5Y4Hfh6weBD+/He34MOBqYF04n8zHgkHBdhGBg5WYnhNcYFxHMtXZEJ8eeDsxx90oPpqV5CGieYaQe+Ev4egEwNnw9F7jfzL5MMLSVSFrK63wTEelM2Hx5P8GI9VuBkqDYFgLHunttMt8eeMDd2+tws9fD635mVkRQO6xw97UWzChftB/v2+DuzZ0IIoS/J+7+VTM7huA66AIzO9rdt+3H+4gkhWqAIj3A3Rd6MJfZcmASQVPhKe4+tYPk9yLBLCUAFxEMgNwVDeG0OQD/AM4zsyEAZlZuZmPa2ac52W0N55s7L2rdHqCsnX1eBY4zs0HhdcULgX/GC8zMxrn7K+4+E6ik9dRkImlDNUCRHmJmg4Ed7t5kZhPcfWmczb8B3Gdm3yFIEl/o4tvdDbxpZq+5+0Vmdj3wdzPLARqArxHMlNDC3Xea2a8JptLZRDBtWLP7gV+aWS1B82zzPhvN7BqCaWkM+Ku7dzYlzY/N7LBw+38QNAmLpB3dBiEiIr2SmkBFRKRXUgIUEZFeSQlQRER6JSVAERHplZQARUSkV1ICFBGRXkkJUEREeiUlQBER6ZX+PyzEFtfPiwnaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 504x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.rcParams['figure.figsize'] = 7, 5\n",
    "plt.plot(range(1,31), error_all, '-', linewidth=4.0, label='Training error')\n",
    "plt.title('Performance of Adaboost ensemble')\n",
    "plt.xlabel('# of iterations')\n",
    "plt.ylabel('Classification error')\n",
    "plt.legend(loc='best', prop={'size':15})\n",
    "\n",
    "plt.rcParams.update({'font.size': 16})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question**: Which of the following best describes a **general trend in accuracy** as we add more and more components? Answer based on the 30 components learned so far.\n",
    "\n",
    "1. Training error goes down monotonically, i.e. the training error reduces with each iteration but never increases.\n",
    "2. Training error goes down in general, with some ups and downs in the middle. XXX\n",
    "3. Training error goes up in general, with some ups and downs in the middle. \n",
    "4. Training error goes down in the beginning, achieves the best error, and then goes up sharply.\n",
    "5. None of the above\n",
    "\n",
    "\n",
    "### Evaluation on the test data\n",
    "\n",
    "Performing well on the training data is cheating, so lets make sure it works on the `test_data` as well. Here, we will compute the classification error on the `test_data` at the end of each iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 1, test error = 0.4233089185695821\n",
      "Iteration 2, test error = 0.4284791038345541\n",
      "Iteration 3, test error = 0.3981042654028436\n",
      "Iteration 4, test error = 0.3981042654028436\n",
      "Iteration 5, test error = 0.37990090478242133\n",
      "Iteration 6, test error = 0.38000861697544164\n",
      "Iteration 7, test error = 0.37925463162429984\n",
      "Iteration 8, test error = 0.38000861697544164\n",
      "Iteration 9, test error = 0.37925463162429984\n",
      "Iteration 10, test error = 0.37968548039638084\n",
      "Iteration 11, test error = 0.37925463162429984\n",
      "Iteration 12, test error = 0.37796208530805686\n",
      "Iteration 13, test error = 0.37925463162429984\n",
      "Iteration 14, test error = 0.37785437311503667\n",
      "Iteration 15, test error = 0.37850064627315816\n",
      "Iteration 16, test error = 0.37785437311503667\n",
      "Iteration 17, test error = 0.37796208530805686\n",
      "Iteration 18, test error = 0.37785437311503667\n",
      "Iteration 19, test error = 0.37817750969409736\n",
      "Iteration 20, test error = 0.3768849633778544\n",
      "Iteration 21, test error = 0.37753123653597587\n",
      "Iteration 22, test error = 0.37677725118483407\n",
      "Iteration 23, test error = 0.37677725118483407\n",
      "Iteration 24, test error = 0.3768849633778544\n",
      "Iteration 25, test error = 0.37677725118483407\n",
      "Iteration 26, test error = 0.3765618267987936\n",
      "Iteration 27, test error = 0.3764541146057734\n",
      "Iteration 28, test error = 0.37699267557087457\n",
      "Iteration 29, test error = 0.37677725118483407\n",
      "Iteration 30, test error = 0.37677725118483407\n"
     ]
    }
   ],
   "source": [
    "test_error_all = []\n",
    "for n in range(1, 31):\n",
    "    predictions = predict_adaboost(stump_weights[:n], tree_stumps[:n], test_data)\n",
    "    error = 1.0 - turicreate.evaluation.accuracy(test_data[target], predictions)\n",
    "    test_error_all.append(error)\n",
    "    print(\"Iteration %s, test error = %s\" % (n, test_error_all[n-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize both the training and test errors\n",
    "\n",
    "Now, let us plot the training & test error with the number of iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeMAAAFTCAYAAAAKvWRNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABj0ElEQVR4nO3dd3gc1dXA4d9Rl9xlW5Z771Rj04sxMSTU0EISQgkJARIIhPDRYsD0EooDAYKBAKEk1NADhGICxhQDBlxw71WyJMtWL+f7485Kq9XsalVWq3Le59lHO3fuzNzZXe3ZW+aOqCrGGGOMiZ+EeBfAGGOM6ewsGBtjjDFxZsHYGGOMiTMLxsYYY0ycWTA2xhhj4syCsTHGGBNnFoxNWCIyXkReF5FtIqIisiDeZTKRicgxIvKZiBR679msVjz2VO+YM+O5D9N+iMjZ3vt9diO2URGZE7tSxUdSvAtgwhORYcDqkORyYDPwPnCzqq6M0bETgX8Dw4B/AJuALbE4lmkZIjISeAnYDswGdgGfNmL744BXvcUDVXVeixeygxKRx4GzgOGquia+pTHtkQXj9mE58Iz3vDswFfglcKKI7Keqy2JwzBHAWOAhVT0/Bvs3LW8akAJcqqr/asL25wAKiPfcgrExrcSCcfuwTFVnBhZERIDHcL/E/+T9bWn9vb9WG24/mvyeiUgWcAzwX28/p4nIxapa3ILlM8aEYX3G7ZC6OUwf8BYnB9JFJEFEzvX6DHd5j09E5KTQfYjI417fy0gR+T8RWSIiZSIyS0TWAB96Wa/z8tXp1xGRPUXkRRHJ8bZbLiI3i0jXkOMM87Z9XEQmisgrIpLnpfUUkZne86ki8isRWSQiJSKyVETO8PaRKiK3icg6ESkVkS9E5ACfc5omIo+JyDIRKRKRnd75n+aTN7hco0Tk3yKS7233rojs6ffae3kf9cpSJiKbReRtETk+JF/U70UkIpIlIn8VkbUiUu4d7wkRGR56LsD1XtIHQe/ZsCgPdQaQDDwFPAl0A04NU6YkEZkhIqu992OJiJwX4RxOFJFnRWSV997me6/xDxo496ki8pH32m0XkSdFpH+YvCd6eXd67+F8Efl1mLxdvc/qcu89zPE+y/XecxEZ4L3+K4LKvlBE7hORZC/PGmp/EK8Oeu3nRDq/oGOM9D63G7z3eIOIPCAifUPyNfozKyJjROQfIrLGO9ftIvKViFzvkzdbRO713qcyEdkqIk8Ff9aC8qqIzBGRQd57myciO0TkeRHp5+U5SEQ+8N6THBG5J/CahXkdThaRL73XebP3GveI5jVsbPnbJFW1Rxt94PprFXjdZ92+3rqF3rIAz3ppi4D7vccaL+3ikO0f99L/A+QCTwB3AOcDlwStnwPM9B57edseBhQDZbgv7tuAz7z884F0n3P4GCgE/gfcSe0X/kxv/Su4vs6/e+XO9dKPAV4DlgH3edtVAgVAz5BzesvLFyjTbFwtUYFLwry2c7xjfQjcBbzspecB/UK2ORTYCVTh+lZvBR4GvgVeDsrXqPciwvufhRszoMA73vFeAqq9Mo/z8vX0Xsc5Xt7Hg96znlEeayGuj7kLMMA7xw/D5P2Hd5xlwJ+917kQeN1LnxmSfwmwANeac6tXvgLvGCeF5J3q7eNt3OfrReAW7/wVWAX0Dtnmcm/dNuCvwN3AWi/t/pC86bjPqOL602/F/QApx32mDw3K28V7z8px4yduw30G3/LK1tXLd4l3fgrMCnrtz47idT8A2OHt73nc/+Cr3nu8Eshs6mcWGOi9zkW4bq5bgAeBD4CtIeUYDWz03pPXvPf1X9655wAjQ/Ir8A3u8/mRV473gl7Xg73jvuCtW+ytuz5kP2d76a9T+31yq7cPBT4HUnyOPac55W+Lj7gXwB4R3pzIwfjv3rrHvOXzvOX7gcSgfF1wgbIMGBCU/riXfy0wyGf/U/H/Yk30viSqgcOC0iVon9f5nIMC1/ocZ6a3LgcYGpS+j5ee7335BAf4P3rrLgt9vXz238X70tgBZIQp1xUh29zopV8ZlJbm/bNXAlN9jjMw6Hmj3osI73/g9bw2JD3Qt/tBmNeyXvkaOM5+3nZPBqW9473HoV/C06j9kkwLSp8AlIb5zAz3OWY/YAOwIsznToFzwrwv9wWljfLek41AdlB6N9wPDA35nAZeo0dD9n2El74CSPDSjifMjyegVyBfyHtV7zMY4XVPwf3/5QHjQ9ad6u3vr834zP7eSzvB59ihP2jmeZ/LQ0PSDwAqCPkOCirHHSHpr1L7f3tMyGd/s3euyUHpZwft64igdKH2R98ffY49pznlb4uPuBfAHhHenNp/vmXU/tq+m9pf9nnAaC/vt7hfwSk++znWy39hUNrjoWkh20zF/4v1MC/9FZ9tBnj/EKt8zmFT8D9h0PqZ3vprfNat8NYdEpI+yEt/IsrX8VJCglRQuVYR9KUasu7FoLTTvLRHojheo96LMPtIBUqArQQFPW+dAN95+xni81pObaiMIft7yNvuqKC0M7y0m0PyBn4EHuuzn9l+n5kIx72XkAAW9Ln7HpCQ/F28z3x+YB1wnZf/Dz77P8Vb9/egtNXeZzTbJ3+gZn+otxwIxr+J4lweDz2XKLY5ydvm8jDr5wM5zfjMBoLxkQ2UY5KX74Ew61/A1Th7BKUprpUoIyTvL7x17/ns5xFv3fCgtLO9tHd88g/B/dD6JiS9TjBuSvnb4sMGcLUPo3FfOuB+5W3CfSnepKqrRSQD2A1YD1wtIqHbB/qexvnse34jy7Kn9/fD0BWquklElgMTRaSbqu4MWv2tqlZE2O83PmmbgZE+6wIDlAYEJ4pId1yT5Qm40eAZIdv59TcuUNXqkLQN3t+eQWlTvL/v+OwjuAzNeS+CjcXVxv+rqqXBK1RVReRD7zh7Ausa2Fek8qYDP8W9pu8GrXoJ16R5pohcE/QaBd7/j3129zFwrs8xsoGrgB8Bg73zCtYf1xwcbK5636QBqlok7lr3w4Gh3jZhP4+4FpWaMnufj2HAd6rqN8htDq5bZE9cd8qHuNflfq9/+y3gf6q6wmfbptjP+ztR/K+rTgf6iEgfVc0NSo/2M/sarsn33yLyHG5w3v9UdUPItoFyDAxTjv648UWjqft9sVzrD/Db7P31+38O/r9dHbLuo9DMqrpORNYBu4lIoqpW+eyzOeVvUywYtw9vqOqxEdb3wtWWhlAbtP108Unb2siydG9guy3ARC9fcDBu6DiFPmlVAKpaZ52qVnpBrmYwiIik4L489wK+xNVU8rx97IUL0KnRHDdo/4lByT28v5saOI/mvBfBonmdg/M11SnePh4N/rLzAt+/cTWdo3BjC8C9DhWqWuCzr3plFZFMXJP2IFyw/g+uy6AaVws+DP/3ZVuY8gaOEXg/wr5OqporIpVBeRr1mqrqDnEDBW8AjsMb0CYiy3B9n8/47iV6md7fMxvI1wXXRxwQ1WfW+6F+AK7F5Ce4Wigi8hVwlaoGflgGynG894hUjmBh/2fDrKv0/voN4or0fg8HuuI+N36aWv42xYJxxxD44H+mqvs3clttOIvvsfqFWd8vJF9Tj9NYJ+CC7sOq+pvgFSJyhbe+OQq8vwMiZaJ574Xffhr7OjfWOd7fP4jIHyLkCQTjHUCyiPT0Cch+Zf0Vrjb8J1W9JXiFiDyIC8Z+ssKkB44R+GIOfp02huy/N+47rtAnb6R917ym6ibwOFPcJDh7AT8ELgaeFpEtqvp+mH1FI3CcH6nqW83YT1iq+i1wkvdjdQqu5n8R8KqI7KWq3weV4wJV/VssyhGFSO93NW5wYThtofzNZpc2dQBec/D3wAQR6Rbjwy3w/h4aukLcZSdjcH3GO0PXx9hI7++rPusOaoH9f+H9PTJSphZ8L5biBkTtKyJ+NcfA6+/XHBgVERmBC4abgEfDPLYDx3uBLfh4B/vs0i/N930RV42rd3lakIMkpI1fRLrgAmIBbuATRPg8UhvoF0BNC8tqYEzg8ptI+YOpapWqfqmqNwOBH3vHBWUJ1AgTid7n3t/m/GiLiqqWq+pcVb0auAbXGnFUa5cjgkNCE0RkCK6FaWGEJmpoG+VvNgvGHcd9uBGkfxOR0D45xF3jG+7XZ2N8jBtAcryIhH753owbIfqPFjhOYwX6TesEXnHX9R5XP3ujvYoLWmeLyNTQlSIyMGix2e+FqpbhLo/qhxs9Hrz9WcAeuEEsTe4vxs3iJrgRu7/2e+DGJqTgmqvBXQYEcG3wuYnIBPybW33fF9zlQHsS3livfMGuxHUDPBXUn/wMLhBeFvyairvefaa3GPx5/AcuEN0YvGPvPT0Wd6XA3MA5hXmfAoE8uC8/z/s7OMI5hXoZN7bg/0SkXiARkXQR2a/eVlESkUleP3moOuVX1c9wAe0MEfmxz36Sff7XW9p0ETkiJO1G3I+biN8nbaT8zWbN1B3Hg8CBwOnAISLyPq4PrD+wO7A3riYSrm8mKqpaLSLn4AazvOsNDNmI6//bH9dfe0dzjtFEr+G++K8QkYm42ulEXLPiv4ETm7NzVS0VkZ8BbwLvicgbuEtnMnEDSNYCP/ayt9R7cTmutnaziBwKfIULUifiaqwXNPV8RCQBN1FFNZG/7B4D/g8XGP+iqu+LyJO40dbfiMiruD7Wn+EGgB0Tsv2TwBXAX72AtxE3Uc3+wBs++QPeAR4UkWNwrQSTgem4mu3MQCZVXSEiVwO3A9+KyPO4QY4n4gZrPaCqwYO7bscF3XNFZDfcOINBuD7VUtzlVIHBUUcCd4jIx7gpafNwLT/H4WrnjwTt933gMmC2iLyIu8Z2rao+Geb8UNUyETkV1wXwiYi8g7seN8kr+2G4S3Z+GG4fDTjTO8+PcD8yduJ+xP0Q97/yXFDen+OuP/63d75f4/p4h+JqrXk0POiwOd4A3vS+TzbgBunthxtwdV8U28e7/M0X7+Hc9gj/IMJ1xhG2OR33oczHXcKxDjeBwgVAl6B8jxPhUgzCXNoUtH4vam9KUI67DOkWvIkQfM7h8TD7mUmYy3HwJrEIs12dyxu8tJG4wJuD++L5CNcUd7aX/+xGlKve/r30sdTeOCNw047/EHRNZWPfiwbezyzcRBbrvONt8Y7vd+1u2NfSJ+9RXt63osgbmIBhH285CdfUucY7ryW4yWJ8PzO4Hx//9V6HHd5rMMWvvMH78J5/hAtsebjA7nt9Nu4yoY9xfYvFuB+F54bJ2837rK7wXtPtuM/yniH5xuMm8fjKy1PibfO3MK///+EuQywP9/kJU54h3nu80ns983GXrt0HTGnqZxYXzB7C/Wgs8F7HJbgfy1k+2/fGjb5e7J1roZf/UYKuAW7g/8P3MxDu80nQ/yZwsve+BS7p+ys+lyNFOHbU5W+Lj8C1esYYY4yJE+szNsYYY+LMgrExxhgTZxaMjTHGmDizYGyMMcbEmV3a5KNPnz46bNiweBfDGGNMB/Pll1/mqmrf0HQLxj6GDRvG/Pltdj5xY4wx7ZSIrPVLt2ZqY4wxJs4sGBtjjDFxZsHYGGOMiTMLxsYYY0ycWTA2xhhj4syCsTHGGBNndmmTMaZTKCwsZNu2bVRUVMS7KKYDSk5OJisri+7d/W4h3TALxsaYDq+wsJCtW7cycOBA0tPTEZF4F8l0IKpKSUkJGzduBGhSQLZm6jaiqKySv76/nBteW8z6vOJ4F8eYDmXbtm0MHDiQjIwMC8SmxYkIGRkZDBw4kG3btjVpH1YzbiNueXMJT3+2DoB3Fm/hf/93OAkJ9qVhTEuoqKggPT093sUwHVx6enqTu0GsZtwGVFRV8/LXG2uWN+SXsDJnVxxLZEzHYzViE2vN+YxZMG4Dvtu4g6LyqjppOTvL4lQaY4wxrc2CcRswb+X2emk5uywYG2NMZ2HBuA34ZGVuvTSrGRtjAkSkwcecOXOatO81a9YgIrz++uuN2m7OnDmICAsXLmzScU1dNoArzkorqpi/Jr9eeu6u8jiUxhjTFs2bN6/meUlJCdOmTWPGjBkcc8wxNekTJkxo0r779+/PvHnzGDduXKO2mzRpEvPmzWPkyJFNOq6py4JxnH21Lp+yyup66VYzNsYE7L///jXPd+1ygztHjhxZJz1YVVUVVVVVpKSkNLjv1NTUsPuJpHv37k3arrVVVFSQkJBAYmJiVOnRaMzrGy1rpo4zv/5igFzrMzbGROnss89m8uTJvPzyy0ycOJG0tDQ+++wzNm/ezDnnnMOIESNIT09nzJgxzJgxg/Ly2pY3v2bqYcOGcdlll3HPPfcwaNAgevXqxU9/+lMKCgpq8vg1U4sIf/nLX7j66qvp27cvWVlZ/O53v6OsrO732Zw5c9hjjz1IS0tjypQpfP755/Tp04eZM2dGPM/q6mpuu+02Ro0aRWpqKmPGjOGJJ56ok2fq1KmccsopzJ49m5EjR5KWlsamTZvCpldVVTFz5kyGDBlCamoqEydO5Jlnnonq9W1JVjOOs08sGBvT6oZd+Ua8iwDAmtuOaThTtPtas4bLL7+ca6+9luzsbIYPH05ubi6ZmZncfffd9OrVi2XLljFz5kxycnJ46KGHIu7vueeeY4899mD27Nls2LCBSy+9lKuvvpoHHngg4nZ33XUX06ZN46mnnuLbb7/lqquuYujQoVx++eUAbNy4kaOPPpoDDzyQW265hS1btnD66adTUlLS4DledNFFPPHEE1x77bVMmjSJ//73v5xzzjn07t2bY489tibf3LlzWblyJbfffjsZGRn06NEjbPq1117LHXfcwXXXXceUKVN48cUXOf300xERfvazn0V8fVuSBeM42lVWyTfrC3zXWTO1MaYxtm/fzrvvvstee+1VkzZo0CDuvPPOmuWDDjqILl26cM4553DfffdFbGZNTk7m5ZdfJinJhYnFixfzr3/9q8FgPGzYMB5//HEAjjrqKObOnctLL71UE4xnzZpFRkYGr732Ws1ELN27d+e0006LuN8VK1bw4IMP8thjj3HWWWcB8IMf/IDNmzdz/fXX1wnGBQUFLFiwgH79+tXZR2h6Xl4es2bNYsaMGcyYMaOmzBs2bGDmzJl1grHf69uSrJk6jr5Yk0dltfqu215UTnWYdcYYE2rgwIH1AoWqMmvWLCZMmEB6ejrJycmcfvrplJWVsW7duoj7O/zww2sCMbgBYtHcaOPII4+sszxhwgQ2bNhQs/zFF18wffr0OjOiHX/88Q2dHu+99x4JCQmceOKJVFZW1jyOOOIIFixYQFVV7VwN++yzT71A7Je+cOFCiouLOfXUU+vkO+2001i2bBk5OTk1aX6vb0tq9WAsIoNF5AUR2SEihSLykogMacJ+rhQRFZGPQ9K7ichzIrJCRIpEpEBEPheRX7TcWbSMcP3FAFXVSkGJ3V3GGBMdv+Aza9YsLrvsMk488UReeeUVPv/8c+6//34ASktLI+6vZ8+edZZTUlJQ1Xr9v9FsF3ysLVu20Ldv3zp50tLS6Nq1a8T95ubmUlVVRY8ePUhOTq55nH322VRWVrJ58+aavH6vhV96YJvQ9OCac0P7bCmt2kwtIhnA+0AZcBagwE3AByKyh6oWRbmfEcAMwG9G7hSgErgVWAOkAqcBT4pIX1W9p7nn0VL8ri8OlrurjMwuLTdazxjjtGRfbVvhNxXj888/zymnnMLNN99ck7Z48eLWLFY92dnZdWqc4H4YBEaJh5OZmUlSUhJz584lIaF+PTIrK6vmebhpKUPT+/fvD7gbifTu3bsmfevWrTXHbGifLaW1+4zPBUYAY1V1BYCIfAssB84D7o5yPw8CTwNjCTkHVd0O/Dwk/5siMgY4B2gTwbiguJxFmwrrpI3K6sqKbbUfyJydZYzp1621i2aM6SBKSkpITU2tk/b000/HqTTOlClTeOyxxygpKalpqn711Vcb3G7atGlUVVWxY8cOpk+f3iJl2W233cjIyOD555/n2muvrUl/7rnnGDNmTL0afCy1djA+Hvg0EIgBVHW1iMwFTiCKYCwiPwcmAT8DXmrEsbcDaY0rbux8uioPDeoSHt+/O8P7ZNQJxjai2hjTHNOnT+fee+9lv/32Y+TIkTz99NOsWLGi4Q1j6JJLLuH+++/nuOOO4w9/+ANbtmzhtttuIyMjw7fGGzB27FjOP/98fvrTn3L55ZczefJkSktLWbRoEcuWLeORRx5pdFkyMzO55JJLuOmmm0hKSmLy5Mm89NJLvPnmm/zzn/9szmk2WmsH44nAKz7pi4BTfdLrEJFeuJrt5aqaF6nZQNzKRKAHcDJwFPCrJpQ5JuaFNFEfOLI3FVV1J/+wEdXGmOa49tprycnJqRkpfNJJJ3Hvvfdy3HHHxa1MAwcO5I033uDiiy/mpJNOYvz48fz9739n+vTpdO/ePeK2999/P2PGjOHhhx/m2muvpXv37kyYMIFf/arpX+033HADSUlJPPjgg2zdupVRo0bx1FNP8dOf/rTJ+2wKUW29EbsiUg7crapXhqTfBFypqhF/HIjII7im6UNVVUVkDpCkqgf75L0QuM9brAAuUdWwY/JF5DfAbwCGDBmyz9q1a6M/sSaYfveHLA+qBT961mQWbyrkrv8uq0k777ARXPWj8TEthzGdwZIlSxg/3v6X2qqPP/6YQw45hPfff5/DDz883sVploY+ayLypapODk1vN9cZi8ghwJnAJI3uF8SzwKdAH1zz+H0iUqWqvle6q+psYDbA5MmTY/oLZdvO0jqBODFB2Hd4JttCasK5O21+amNMx3PFFVew9957k52dzdKlS7nxxhvZY489OOyww+JdtLhp7WCcD/TySc/01kXyEPAosEFEenppSUCit1yiqjXRTFVzgMCQvbe8kdx3isjfVTWu1wyFXtK0+8AedEtLpk/XugMtrM/YGNMRlZWV8X//939s3bqVbt26ceSRR3L33XdH7DPu6Fo7GC/C9RuHmgA0NN5+vPc432ddPvAHYFaE7efjLqfqB2yIkC/mQoPxgSPdkPq+3eoGY+szNsZ0RLNmzWLWrFnxLkab0to/Q14F9veuEwZARIYBB3nrIjnc5/ENsNB7/kID2x8G7ML/2uRWFTof9YEj+8CuHMZ89HueSb6J/RPc7xKrGRtjTOfQ2jXjh4ELgVdEZAZu0o8bgfW4ZmgARGQosBK4QVVvAFDVOaE7E5EC3ACuOUFp5wH7A+/iasC9gZ8Ap+AGicW1I3Z9XjHr8oprllMSE9hnSA/4x1FkbPySAxNhTMIGDiq7l+1FQnW1kpAQ24vNjTHGxFer1oy9GbamAcuAJ3ETd6wGpqlq8PQrgcuSmlK+73BN0XcC7+BGVPcBjlXV25te+pYxb1XdWvHeQ3qSvuifsPHLmrQ+Usgo2URVtZJfbIO4jDGmo2v10dSqug533W+kPGtwAbmhfU31SfsEOLqJxYu50P7iqUNT4d3r6+XLknwW6TByd5XTO2RglzHGmI6l8w5diwNVrTcf9Yk7/gHF9eeozpICwPqNjTGmM7Bg3IpW5RaxtbA2uO6evIl+3//DN28/70ovG1FtjDEdnwXjVlR3FLVyW8ZTiFb55s0SF4ytZmyMEZEGH3PmzGnWMWbPns3LL7/cIuU1jdduZuDqCILno/5RwudMLFsQNm8/r5k6x4KxMZ3evHnzap6XlJQwbdo0ZsyYwTHH1N4KcsKECc06xuzZs9ltt9348Y9/3Kz9mKaxYNxKqqu1ZvBWGmX8KTnkNmbd+sPO2ptjB2rG1kxtjNl///1rngfu+zty5Mg66W1d8C0To0mPRkVFBQkJCSQmJja3eHFnzdSt5PstO8kvdrNwXpD0GoMkaNBWQhIcd2+d/LUDuOzSJmNMwx555BEmTpxIamoqQ4cO5Y477qizftGiRfzwhz8kMzOTLl26MH78eO6//34Apk6dypdffskTTzxR0+z9+OOPhz1WaWkpl19+OYMHDyY1NZU999yTN998s06eYcOG8cc//pEbb7yRQYMG1dyRKVx6cXExv//978nOziYtLY0pU6bwzjvv1Nnn1KlTOeWUU5g9ezYjR44kLS2NTZs2NfelaxOsZtxKAqOoB8k2zk98re7K/c6H4YfWSepLAQlUk2s1Y2Na3swe8S6BM3NHi+zmz3/+M1dffTWXX355TWC95ppryMjI4MILLwTguOOOY/z48Tz11FOkpqaydOlSCgsLAXjggQc4+eSTGTFiBNdccw3gat7hnHLKKXz++edcf/31jBw5kueee47jjz+e+fPns9dee9Xke+aZZ5g4cSIPPPAAlZWVEdPPPfdcXn31VW655RZGjRrFww8/zDHHHMMHH3zAwQfX3phv7ty5rFy5kttvv52MjAx69Ggj72UzWTBuJYEm6hlJT5MqQfep6JIFh10OyWmQ1hNKCwBIkmp6U0jOrqY13xhjOofCwkKuv/56ZsyYwXXXXQfA9OnTKS4u5qabbuKCCy4gPz+f1atX88orr7D77rsDcMQRR9TsY8KECXTp0oW+ffs22PT93nvv8cYbbzBnzpyauywdeeSRLFu2jJtvvpnnn3++Tv7XX3+dtLS0evsJTl+yZAn//Oc/eeyxxzjrrLMAOOqoo9hjjz248cYbefvtt2u2KygoYMGCBfTr16+xL1WbZs3UraCyqprPVudxcMJ3/DDxi7orfzAT0rxfdt2y66zKkgLyisqprm69e04bY9qXefPmUVRUxKmnnkplZWXNY9q0aWzdupUNGzaQmZnJ4MGDOf/883n22WfZtq3pU/S/++67ZGdnc9BBB9U53hFHHMH8+fPr5D3iiCN8A3Fo+hdffIGqcuqpp9akJSQkcOqpp/Lxxx/X2XafffbpcIEYLBi3iu827qC0rJSZSU/UXTFwMuz5s9rlesE436bENMZElJvrusAmTpxIcnJyzePwww8HYP369SQkJPDOO++QnZ3NOeecQ3Z2Nocccghff/11k463ZcuWOsdKTk5m5syZrF+/vk7ecEEzNH3z5s107dqVjIyMevmKi4spKysLu21HYc3UreCTlds5K/FtRiUEDzQQOPoOCL5/Z7f+dbbLCrq8yabENKYFtVBfbVuQmZkJuGZfv0A1duxYAMaNG8eLL75IRUUFH330EVdccQXHHHMMGzZsaNR9hDMzMxk4cGBU1ySL+M9qHJrev39/du3aRXFxcZ2AvHXrVjIyMkhNTQ27bUdhwbgVLF62nNuSXqqbuPcvYOA+ddO61v1HCszClbuzHOpWmo0xBoADDjiA9PR0Nm3aVOe643CSk5OZNm0al156KT//+c8pKCggMzOTlJQUSktLG9z+iCOO4K677qJr166MGzeuJU6BKVOmICK88MILnHnmmYCbPviFF16oM3irI7NgHGNllVVM2/gg3RJKatKqU7qTcMR19TOH1Iz72SxcxpgG9OzZk5kzZ3LxxRezdu1aDj30UKqrq1m2bBkffPAB//73v/n222+57LLLOO200xgxYgT5+fncfvvt7LnnnjU163HjxvH222/z9ttv07t3b4YPH07v3r3rHW/69OkcddRRTJ8+nSuuuIKJEydSWFjIggULKC0t5dZbb230OYwfP56f/exnXHjhhezcuZORI0fy8MMP8/333/Pggw82+zVqDywYx9jyLz/g5IQP66TJ4VdB1771M/sM4AKb+MMYE9nll1/OgAEDuOeee7jrrrtIS0tjzJgxnHbaaQBkZ2fTr18/br75ZjZt2kTPnj05/PDDuf322rvKzpgxg3Xr1vGTn/yEwsJCHnvsMc4+++x6xxIRXnrpJW655RZmzZrFunXryMzMZK+99uKiiy5q8jk8/PDDXHHFFdxwww0UFBSw++678/rrr3eamrGo2kjdUJMnT9bQUYFNUl3NlrsPJHvXkpqkzanD6X/5F5CYXD//uk/h70fVLH5TPYITym/ivENHcNXR45tfHmM6qSVLljB+vP0Pmdhr6LMmIl+q6uTQdBtNHUsLnqoTiAGWT5rhH4ihXs040Ext81MbY0zHZsE4VkoK0Hevr5P0RtW+jNovwgCLrnWDcR92uFm4bEpMY4zp0CwYx8qc25Di2vmnSzSFJ7udy4CeEWbUCszC5amZhcv6jI0xpkOzYBwLWxfD57PrJD1YeTwjRkfRZ+VzrbGNpjbGmI7NgnFLU4W3rgCtqklaX92Xh6qO5cCR9S8TqKdb3WuNsySfvKJyqmxKTGOaxQarmlhrzmfMgnFLy10G6z6rk3RT5S8oI4X9R0QTjOtfa2xTYhrTPMnJyZSUlDSc0ZhmKCkpITk5zADdBlgwbml9x8LvPmXbgGkA/K9qd96unsy47G70iWZKy9BrjSkAbOIPY5ojKyuLjRs3UlxcbDVk0+JUleLiYjZu3EhWVlaT9mGTfsRC5ggeyL6J1WteYYP2AYQDommihnojqmtm4bIpMY1pssAN7Ddt2kRFRUUDuY1pvOTkZPr161fzWWusBoOxiCQDRwPfqurqJh2lE/pkZS7LqvesWT5wZJ/oNvS5cxNAzq6G54w1xoTXvXv3Jn9RGhNrDTZTq2oF8BwwLOal6SBydpaxbOuumuUEgX2HZ0a3cZg7N+XutD5jY4zpqKLtM14FNK0hvBOat2p7neXdB/agR3qUnfoho6ntZhHGGNPxRRuM7wD+JCI+dzcwoeatzK2zfEC0TdRQr8+4LwUkUG0TfxhjTAcW7QCuaUAmsFpEPgU2A8FDElVVz2rpwrVXn6ysWzOO6vrigOQ0SO8FJa5GnCjqZuGymrExxnRY0Qbjg4EKIAcY6T2C2bUCnqpq5eRJg5i7Ipev1xWgKJOH9WrcTrpm1wRjcIO4bH5qY4zpuKIKxqo6vKUOKCKDgXuA6YAA7wKXqOq6Ru7nSuBWYK6qHhyUPgb4HXA4MALYCXwBXKOq37TISUSQmCD8/ojR/P6I0ZSUV7F0604yUhp5BVm3bMipvdtTlhTwnTVTG2NMh9Wqk36ISAbwPjAOOAs4AxgNfCAiXRqxnxHADGCbz+ojcYH4CeA44LdAX+BTEdmnWSfQSOkpiew1uGfjN/S5lWJeUZlNiWmMMR1U1FU2L5CeAxyG6z/OAz4AHlPVaOeZOxdXWx2rqiu8/X4LLAfOA+6Ocj8PAk8DY33O4V/A/Ro0zY6IvA+sAS4GzozyGPETGozJp1ohv7g8ulm8jDHGtCtR1YxFJBv4CrgXmAxkeH//CnwlIv0ibB7seODTQCAG8CYSmQucEGVZfg5MAq7yW6+quRoy352q7gCWAQOjLGd8hbnW2EZUG2NMx9SYS5t6AYeo6nBVPcDrRz4Y6AncHuV+JgILfdIXARMa2lhEeuH6my9X1bwoj4mIZAK7AUsaytsmdK1/5yawa42NMaajijYY/wi4SlXnBieq6ie4vttjotxPJpDvk56HC/YN+TOuhvt4lMcLuA83WGxWuAwi8hsRmS8i83Nychq5+xbmc+cmsGBsjDEdVbTBuCuwKcy6Dd76mBKRQ3D9vReENkM3sN1VwM+BC4Obx0Op6mxVnayqk/v2jfPcJvXmpy4ArJnaGGM6qmiD8VLcyGc/vwC+j3I/+fjXgMPVmIM9BDwKbBCRniLSEzd4K9FbrjeySUTOB24BZqjq36MsY/yFNFMHZuGya42NMaZjinY09Z3AP7yBWs/gZuDKBn4K/IDwgTrUIly/cagJwOIGth3vPc73WZcP/IGgZmgROQN4ALhLVW+OsnxtQ5hZuHKtZmyMMR1StJN+POVd2nQD8EjQqq3A+ar6TJTHexW4U0RGqOoqABEZBhwEXNnAtof7pM0CEoGLgJomaBE5EXgMeERVL4uybG1Lt/71ZuGyKTGNMaZjiuZ+xom4kcj/xgXisdReZ7xUVasbcbyHgQuBV0RkBm4azRuB9bhm6MAxhwIrgRtU9QYAVZ3jU7YCICl4nYgcCvwT+AZ4XET2D9qkTFW/bkR546drP9hW21jQT/LZZDVjY4zpkKKpGSswHzhGVd+hGZcHqWqRiEzDXZ70JG6E83u46TB3BWUVXI23KTOETQNScdcizw1Zt5b2cl9mn2uNv7U+Y2OM6ZAaDMaqWi0i64Gop6tsYH/rgJMbyLMGF5Ab2tdUn7SZwMwmFa4tCb2vMbVTYiYmNPjSGGOMaUeirXk+BFwiIimxLIwJ4nOtcbVCXpHVjo0xpqOJdjR1N9xtE1eJyFv438/4upYuXKcWcq1xX+9a49xdZfTtZvNTG2NMRxJtML466Pk5PusVsGDckrrWv3MT2CxcxhjTEUV7aVOr3mrRYLNwGWNMJ9JgkBWRFBH5t3fJkGktoc3UNbNwWTA2xpiOpsFgrKrluFm2rHbcmpJS3SxcHjcL1w6bEtMYYzqgaAPsXGD/BnOZluVzrbE1UxtjTMcTbTD+I/ArEblQRAaJSKKIJAQ/YlnITqtb/UFc1kxtjDEdT7RB9DvcpU1/wc1iVQ5UBD2s7TQWfEZUW83YGGM6nmgvbbqButcVm9YQOqKaAqsZG2NMBxTtpU0zY1wO48enmTqvqNymxDTGmA6m0X29ItJVRIaKSHIsCmSC+MzCZVNiGmNMxxN1MBaRY0XkK2AHsArY3Ut/RER+HqPydW4+81ODTfxhjDEdTVTBWER+DLwC5AJXUPeOSquBs1q8ZMbd0ziITYlpjDEdU7Q14+uAx1T1SGBWyLqFwG4tWSjjCWmm7sMOm4XLGGM6oGiD8XjgWe956KjqfKB3i5XI1EpKhfTMmsXALFzWTG2MMR1LtMG4EOgTZt0wIKdFSmPqs4k/jDGmw4s2GP8XuEpEegalqYikAhcC/2npghmPz92bbH5qY4zpWKKd9ONPwOfAUuBNXFP1lcAeQA/gx7EonMF3RPV6a6Y2xpgOJaqasaquASYBrwPTgSrgUOBTYD9V3RSrAnZ6PiOqrZnaGGM6lmhrxqjqBuBXMSyL8RN65yabEtMYYzocu9tSW1evzzif7UXlVFZVx6lAxhhjWpoF47bOZwCXKuQV2yAuY4zpKCwYt3U+lzYB5O60YGyMMR2FBeO2LmQAV2AWrhzrNzbGmA7DgnFbF2YWrly7vMkYYzoMC8btgc+1xjai2hhjOo6oL20SkRHAT4AhQFrIalVVu+wpVrr1g22Lahb7Sb7NT22MMR1IVMHYu4Xic7ia9DYgNBKE3jzCtKTQa43FrjU2xpiOJNpm6huBOUB/VR2gqsNDHiOiPaCIDBaRF0Rkh4gUishLIjKksQUXkStFREXkY591l4rIayKy2cszs7H7b1N8bxZho6mNMaajiDYYjwDuVNVm3Z1JRDKA94FxwFnAGcBo4AMR6dKI/YwAZuBq6X7OBbKAl5tT3jaja8i1xjYLlzHGdCjR9hl/T8vcs/hcXGAfq6orAETkW2A5cB5wd5T7eRB4GhiL/zlMVNVqEUkCzm92qePNZxYu6zM2xpiOI9qa8eXA1V6NtDmOBz4NBGIAVV0NzAVOiGYHIvJz3E0rrgqXR1U71lyRPqOp84ptSkxjjOkooq0Zz8TVjJeIyHIgL2S9quphUexnIvCKT/oi4NSGNhaRXsA9wOWqmiciURyyA+hWd+KP4Ckxs7qFDmw3xhjT3kQbjKtw9zJurkwg3yc9D+gVxfZ/BpYBj7dAWeoQkd8AvwEYMqTR48liy2cWrkSqyNlZZsHYGGM6gKiCsapOjXE5GiQihwBnApNUtcUvpVLV2cBsgMmTJ7etS7UCs3CVuAYJNwtXoY2oNsaYDqK1Z+DKx78GHK7GHOwh4FFgg4j0FJGeuB8Tid5yaouWtK2pd61xvk2JaYwxHUTUwVhE+ovInSLyhYis9P7eISLZDW9dYxGu3zjUBGBxA9uOx42Mzg96HATs7z2/oBHlaH98rjW2m0UYY0zHEO0MXGOAj3C12rnACiAbuBg4U0QOUdXlUezqVeBOERmhqqu8fQ/DBdUrG9j2cJ+0WUAicJFXpo7L577GVjM2xpiOIdoBXLcDhcB+qromkCgiQ4F3vPUnRbGfh4ELgVdEZAZuGs0bgfW4Zujg/a4EblDVGwBUdU7ozkSkAEgKXScik4Fh1Nb8J4jIKd7zN1W1OIqyti0+NeM1VjM2xpgOIdpm6sOBa4IDMYCqrsVd9uRXa61HVYuAabgR0U/iJu5YDUxT1V1BWQVX421qn/aFwPPAs97yqd7y87iZudqf0D5jrJnaGGM6imhrxinAzjDrdnrro6Kq64CTG8izBheQG9rX1DDpZwNnR1umdqFr/WuNc3faaGpjjOkIoq15LgAuEpE6+cXNuvFbb72JJbunsTHGdFjR1oxvAF7HzcD1LLAZN4DrVNyNHo6JTfFMjXp9xgU1U2ImJbb2FWrGGGNaUrSTfrwlIscCNwF/wjUhK/AlcKyqvhO7IhqgXjN1b3aQoFXkFZWT1d1m4TLGmPYs2poxqvoW8JZ3G8ReQH67HJXcXiWlQEZvKN4O1M7ClbOrzIKxMca0c1EH4wAvAFsQjoeu2TXBGOxWisYY01GEDcYici3wiKpu8p5Hoqp6Y8sWzdTTLRu2LapZdIO4bES1Mca0d5FqxjOBt4BN3vNIApN3mFiqN6K6wEZUG2NMBxA2GKtqgt9zE0f17mtszdTGGNMRRBVkRWSIiCSHWZckIm3sBsAdlM8sXFYzNsaY9i/aGu9qYO8w6/b01ptY87nW2IKxMca0f9EG40hTUyYD1S1QFtOQrqF3brJmamOM6QgijabuCWQGJQ0UkREh2dKBs4AtLV80U49vzdhGUxtjTHsXaTT1xcB1uJHSCrwQJp94+UyshczC1YcdFBaXUFFVTbJNiWmMMe1WpGD8MrAGF2z/jpsKc2VInjJgsap+G4vCmRAhs3AliJKpheQVldPPZuEyxph2K9KlTd8A3wCIiAKvq+r2cPlNK+nWv84sXP28fmMLxsYY035F1bapqk9YIG4j6t3X2C5vMsaY9i7qualFZCLwa2AsEFoNU1U9oiULZsLwnYXLBnEZY0x7FlUwFpH9gA9xfcijgW9xd24aAmwAVsSofCZUvRHVdnmTMca0d9EOwb0FeAmYiBvQ9StVHQb8AEjEDe4yrSEkGPe1WbiMMabdizYY7wE8hbvECVwARlXfxwXiW1u+aMaXzcJljDEdTrTBOAUoUtVqIA8I7rhcCuzW0gUzYdTrM7ZmamOMae+iDcYrgIHe82+Bc0QkQUQSgF9iM3C1nnqjqa1mbIwx7V20o6lfA6YCz+D6j98ACoEqoCvw+1gUzvjwmYUrf2dJnApjjDGmJUQVjFV1ZtDzd0Vkf+BkIAN4S1XfiU3xTD1JKWhGH6Q4F3CzcElxrk2JaYwx7VjU1xkHU9Wvga9buCwmStItG7xgDK7f2KbENMaY9iuqqpSI7C8iPwmz7lTvOmTTWuxaY2OM6VCibde8FXeNsZ/x2KVNravefY0LyLFBXMYY025FG4z3BD4Ns+5z3HXIprX41IxzrWZsjDHtVrTBOC1C3kSgS8sUx0QlJBhnkW/zUxtjTDsWbTBeAhwfZt3xuIk/oiIig0XkBRHZISKFIvKSiAyJdvug/VwpIioiH/usSxCRq0RkjYiUisg3InJyY4/RZoUGYymwPmNjjGnHog3GfwPOFZE/i8gYEckQkdEi8mfgV8AD0exERDKA94FxwFnAGbgbT3wgIlHXrkVkBDAD2BYmy43ATOCvwI9wTezPi8jR0R6jTfOZhcsm/jDGmPYr2uuMHxaRscAfgEuDVwH3qOrsKI93LjACGKuqKwBE5FtgOXAecHeU+3kQeBp3O8c65yAiWcBlwG2qeqeX/IGIjAJuA96M8hhtl0/NeMW2Xfx38daoNk9MgMnDMumelhyL0hljjGmkqK8zVtXLRORB3J2aegO5wLuquqoRxzse+DQQiL39rhaRucAJRBGMReTnwCTgZ7g7SYU6CjeX9lMh6U8BfxeR4aq6uhFlbnu6ZNVZ7MMOlm7O59x/zI96FylJCTx/3gHsObhnCxfOGGNMYzVqyiZVXamqD6nqLao6u5GBGNzlUQt90hcBExraWER6AfcAl6tqXoRjlFH/HsuLvL8NHqfNS0qhOr13zWKCKH3Y0ahdlFdWc+97y1u6ZMYYY5ogbM3YG1S1WVUrohlgparrojheJpDvk54H9Ipi+z8Dy4DHGzhGgapqSHpe0Pp6ROQ3wG8Ahgxp9HiyVifd+0PJ9prlLClgq/qeWlgL1hegqohISxfPGGNMI0Rqpl4D7I+7jngNtfcyDiexZYrkT0QOAc4EJvkE2mbz+r1nA0yePLnF99/SpFs2bK1tZDhycDX9Qpqv/fxveS7lldUAbC8qZ/OOUgb0TI9ZOY0xxjQsUjD+JbDSe34ODQfjaOTjXwMOV2MO9hDwKLBBRHp6aUlAordcoqpl3n56ioiEBO1AtTFc83b7EjKI66IpXWHKlAY3O/nBT/hybe1LvXDjDgvGxhgTZ5GCcQ9qa7vv4zVZN/N4i/CfVnMCsLiBbcd7j/N91uXjRnrP8o6RCoykbr9xoK+4oeO0DyFTYrIrupHUuw/sUS8YHzkxO8IWxhhjYi3SAK57gGHe89XA3i1wvFeB/b3rhAEQkWHAQd66SA73eXyDGxB2OPCCl+8toAI4PWT7XwAL2/1I6oCQmjE7N0e12W4De9RZ/m5j4wZ+GWOMaXmRasYFQOAbX2iZZuqHgQuBV0RkhrfPG4H1uGZodzCRobgm8htU9QYAVZ0TujMRKQCSgtep6jYRuRu4SkR2Al8BpwHTCD+LWPsTMvEH21fCFr+B6nXtk1bEGFnPGs2mnGS+21hog7iMMSbOIgXjucATIvKNt/ygiBSGyauqekRDB1PVIhGZhqt1P4kL8u8Bl6jqrqCsgmsib9SlV0H+BOwCLsb9oFgK/ERVX2/i/tqe0Jrx2rnwt4Ma3Gw48E4q5GlXzii/mkW7hrG1sIzsHnYvZGOMiZdIwfhc4Drc1JXq5W32lE3eJVAR54lW1TW4gNzQvqaGSa8CbvIeHVNoMG6kTNnFBUmvcmHF7/lu4w4LxsYYE0dhg7GqbgV+CyAi1cBvVPXz1iqYaUC3AdBjMOxY3+RdTBTXff7dxh1Mn9CvpUpmjDGmkaKdDnM4EN0IIdM6EhLgtCfh/ZugsBFvzbZFNU+HyjZSKWeRDeIyxpi4ivZGEWtjXRDTBAP2hl+82LhtZu0OBW6ytARRRskmvtvYLQaFM8YYE62wA6REpEpE9vWeV3vL4R6VrVdk0yx9x9dZHCUb2LazjG2FpXEqkDHGmEg14xuADUHP2/wUkSYKWeNg+ds1i2MSNkC16zc+orsN4jLGmHiINIDr+qDnM1ulNCb2QmrGY2Qj4AXj8TaIyxhj4qGp1/EiIpkiso+IpLZkgUyM9R1bZ3G0uMaPhTaIyxhj4iaqYCwiM0Tk1qDlQ3F3cvocWC4io2NTPNPiQoLxEG9E9cKN4eZzMcYYE2vR1ox/AawKWr4dNy/0j4GtuCktTXuQ0gV6Dq1ZDIyo3lJYSs7OsjgWzBhjOq9og/FAYDmAiPQF9gWuUdXXgNuAQ2JTPBMTWXX7ja2p2hhj4ivaYFwFpHjPDwVKcXNXA+RQe69g0x6ENFWPSXDB2O7gZIwx8RFtMF4E/EJEugLnAB8G3dt4MLAtFoUzMdLXv2ZswdgYY+Ij2ukwbwBewd0juAI4Kmjd0bjbFJr2ImtcncXR3uVN1kxtjDHxEe10mG+LyHhgErBAVVcGrf4fbjCXaS/6jCX4FtVDZBtplLF5B+TuKqNPV7tazRhjWlPU1xmr6mpVfTEkEKOqD6nqpy1fNBMzKRnQq+6I6pGyCbDasTHGxEO01xmfICK/DFoeKiLzRGSniLzg9SWb9qRv3abqMTai2hhj4ibamvEMoG/Q8t3AIGA2bnT1zJYtlom5kGA8OqF2WkxjjDGtK9pgPBL4FkBE0nGDti5V1T8CVwMnxqZ4JmbCXmtsM3EZY0xrizYYpwEl3vMDcQO/3vGWlwIDWrhcJtbCNFNvLCghr6g8HiUyxphOK9pgvAY42Ht+AvClqgbaM7MAa9tsb/qMwY2odgZLDmm46TCtqdoYY1pXtMH4IWCmiMwHfgs8GrTuAGBxSxfMxJjPiOpRdr2xMcbERVTBWFX/ApwNzAPOUdWHg1Z3Ax5r+aKZmKs3E5cFY2OMiYdoZ+BCVZ8GnvZJP69FS2RaT9Y4WPafmsUxCRug2pqpjTGmtUU96YfpgMLMUb0hv4R8G8RljDGtJupgLCK/EZGvRaRYRKpCH7EspImR0Ls3ecEYYOEmqx0bY0xriXYGrjOB+4AvcJc5PQY8BRQCK3E3kjDtTciI6kGSSzqlgDVVG2NMa4q2ZnwJcCtwgbf8gKqeBYzAXX+8veWLZmIuJQN6DatZtDmqjTEmPqINxqNxd2eq9h4pAKqaD9wMXByT0pnYC5mJa4zNxGWMMa0u2mBcAiSoqgJbcDXigF3YDFztV2i/sTdH9bq8YnYUV8SjRMYY0+lEG4y/A0Z5zz8CrhaRA0RkCu4mEd9He0ARGezd6WmHiBSKyEsiMiSK7YaKyCsislZESkQkV0Q+FJGjffIO945RICJFIvKBiEyOtoydSsiI6lE2iMsYY1pdtMF4NtDLe34N0BX4GPgUGAP8MZqdiEgG8D4wDjgLOAPXBP6BiHRpYPOuQC7uDlJHA78CdgJviMhJQcfo7ZVtN+A84Kfeqg9EpG7kMe5a4yDBI6ptEJcxxrSOqCb9UNVng56vEJGJuGkwM4BPVDU3yuOdi2viHquqKwBE5FtgOS5w3h2hDItwAbiGiLwBrAZ+CbzkJV8A9AMOVdWVXr73gVXA9cBPoixr59BnDEgCaDUAQxJySKeUEtIsGBtjTCtp0qQfqlqkqu+q6quNCMQAxwOfBgKxt6/VwFzcDSgaW45K3E0qKoOS9weWBwJxoLy45vVjRSTqWcc6heT0OiOqAUbZiGpjjGlVYQNTNP24wVR1XRTZJgKv+KQvAk6N5jgikoD7EdEH+A2umTx4NHcV4Dd9VBmQjrs389JojtVp9B0HeatqFsfIBr7TEazdXsyOkgp6pCfHsXDGGNPxRaolrgG0EftKjCJPJpDvk55HbZ90Q+6gto96F/BTVX0vaP1SYLqI9FbV7VATwPcNKkM9IvIbXHBnyJBG/Q5p//qOg6Vv1iyOTtjoLmADFm3awYEj+8SpYMYY0zlECsbn0Lhg3FpmAf8CsoEzgWdE5BRVfd1b/zfg98A/ROT3QDHwJ2C4t77ab6eqOhs3UI3Jkye3xfOOnSz/OarBNVVbMDbGmNgKG4xV9fEYHC8f/xpwuBpzPaq6AQhEi9dFZA5wJ/C6t36ViJwO3A8E+qa/Au4BLgM2N7XwHVbfSCOqbfIPY4yJtbADuMQ5TkR2i5BndxE5rhHHW4TrNw41AVjciP0Em0/tNdAAqOqLwEBvv6NUdR/cpVHro+zb7lz6jHYjqj2DE3LI8OaotkFcxhgTe5FGU58B/BMoipBnJ/BPEflZlMd7FdhfRGpm8BKRYcBB3rpG8fqCD8bdrKIOVa1S1SWqulJEBgCnAQ829hidgs+I6sAc1atziygstZm4jDEmliIF418Aj3mXHvlS1TXAo7gJPKLxMG5g2CsicoKIHI8bXb0eeCiQyZttq1JErg1Kmyki94rIaSJymIicBryFG5h1XVC+ZBG5R0R+LCLTROQiXO15EXBXlOXsfPr6z1ENsMiaqo0xJqYiBeNJwDtR7ONdIKqpJr3rfacBy4Angadxk3ZMU9VdQVkFNzo7uHxf4WbVus8r1x1AKXCIqv4r+DC4Wb0eAv6Du+PU34GjVNXvkicD9WbiGp0QFIxtWkxjjImpSKOpuxHdoKp8L29UvD7bkxvIs4bgG+26tFeJoinbmwjk2GjLYzwRasY2E5cxxsRWpJpxLjA0in0M8fKa9qze3ZssGBtjTGuJFIw/Jrq+4LO9vKY9C8xR7RkkuTUjqlfnFrGrrDLclsYYY5opUjCeBRzhDYZKCV3pDZSahesDvic2xTOtJjkNeg2vkzRK3L2NVWGR1Y6NMSZmIk36MU9E/ogbgXy6iLwDrPVWDwWmA72BP6rqpzEvqYm9rPGQV3uV2JiEDXxbNRJwTdX7jegdr5IZY0yHFvEORqo6S0S+Aq4ATsTdaAGgBJgD3KaqH8W0hKb19B0H379esxg6LaYxxpjYaPB2gqr6P+B/3gQbgUmKt6tqVUxLZlpfyLSYo71maoCFm+xaY2OMiZWo7+2rqtXAthiWxcRbyLXGwSOqV+bsoqiski6pdjtoY4xpaZEGcJnOpvfosCOqVWHxZqsdG2NMLFgwNrWS0yBzRJ2k4H7j7zZYv7ExxsSCBWNTV+jtFBNsEJcxxsSaBWNTV0gwHhU0iMtm4jLGmNiwYGzqygo/R/XKnF0Ul9tMXMYY09JsaKypK6RmPD5pE3i3M65WmHbnh3RJTSQtOfBIIC3JPU9NTnBpSS69b7dUjpqYzYCe6T4Har68onL+u3gLvTJSmDYui6RE+21pjGmfLBibuvp4I6q1GoBszaELJRR5871sKSxt1O5u+8/3XDRtFOceOoLUpMQWKWJ1tfLPL9Zxx1tL2VHifimM79+dm368G/sM7dUixwhYsW0ns95dzsKNOzhkdF9+f8Ro+nZLbdFjGGOMqGq8y9DmTJ48WefPnx/vYsTPffvA9hU1iyeU3cA3OqpZuxzRpws3nLAbB4/u03DmCBZu3MGfXl7IN+sLfNf/dMpgrvjhOHp1qTedeqPkFZXzl3eX8dRn66iqrv0f6ZqaxIXTRvHLg4a12I8LY0znISJfqurk0HRr1zP1hTRVnzx4JwkSJm+UVuUW8YtHP+Oif37N1kbWrgF2lFRw3SsLOf6vH4cNxAD/+mI90+6aw7NfrKO6uvE/NMsrq3nko1VM/fMHPDFvbZ1ADLCrrJLb/vM9R97zP95etAX7MWuMaQlWM/bR6WvG798E//tz7fIBF1I09XqKyiopraimtLKK0ooq97zCe17pnpd56TtKKnjm83XkFZXX233X1CQunT6GMw8Y2mA/r6ryyoJN3PTGEnJ3ldVbn5KUQHllte+2k4b05KYf786EAd0bPGVV5e1FW7ntP0tYs724wfwBB47szbXHTWBcdsPHMMaYcDVjC8Y+On0w/u4FePFXtcujfgC/eLHRuykoLuf2t5byry/W4fcxa6ifd8W2ncx4eSGfrsrzXf/Didlcc9wE1m4v4pqXF7Iyp6hengSBsw8czh+mj6ZbWrLvfhZu3MFNbywOe5zBmekcu8cAnv50LYWl9UeTJwj8bN8hXDp9DL27Wn+yMSY8C8aN0OmD8ZaF8LeDape7D4JLFzV5d1+ty+ealxeyKMzNJkL7eYvLK7nv/RU88tEqKqrqfz6HZGZw/fETOXxcVk1aeWU1j3y8invfW05pRf2acla3VK45dgLH7tEfEdfmvrWwlD+/vZQXv9rg+2Ohm9c/fLbXP5xXVM49/13G05+txa8FvFtaEhcfMZozDxhGSpL1ABlj6rNg3AidPhhXlsHN/SH4xlxXroe0pjfFVlZV89Sna7nrnWXsLKtfu+yVkcyVPxpHr4wUrn9tMRsLSurlSUlM4PypI/nt1JGkJfsPnlqfV8wNry/mv4u3+q4/eFQfrjp6HO8t2cbfPlxJcXn9m48lCPx8vyH84Qf+Nd2lW3Zy4+uL+XhFru8xRvTpwp+OGc+0cVk1gd8YY8CCcaN0+mAMcN9k2L68dvnX78Ggep+fRttWWMrNby7hlQWbGrXdIaP7cMMJuzG8T5eo8r+7eCszX1vEhvz6QT2SQ8f0ZcYx4xnTr1vEfKrKe0u2cdMbi8P2MR8yug+XHTmWPQf3bFQZjDEdlwXjRrBgDDz7C1jyWu3y8X+FSWe02O4/WZHLjFcWssqnnzdYv+6pXHvsRI7ePbvRtcyS8iru/2AFD/1vpW9zd7BRWV350zHjOXxsVsR8ocorq3nikzXc+95y3xo/wJ6De3L2gUM5evf+djmUMZ2cBeNGsGAMvH8z/O+O2uUDLoSjbm7RQ5RXVvPwR6u47/36/byJCcIvDxzGJdPH0LWZ91BesW0X1726kLkrttdb1ysjmUunj+Fn+w5p1gxeubvKuOudZWEHqwH06ZrCz/Ydws/3G0L/HrGZlcwY07ZZMG4EC8a02IjqaKzPK+b61xbz7hLXz7vP0F7c9OPdGN+/5S4XUlVe+3YzN76+mJydZSQnCr88aDi/O3wUPdL9R1k3xeJNhdzw+qKwI7PB/dD44cRszjxgKPsOz7R+ZWM6EQvGjWDBGNi6CB48sHa5mSOqo7E6t4iS8irG9+8WswBVWlHFdxt3MLJvVzKbOUtXOKrKnGU5PD53DR8uy4mYd1x2N846cBg/3msg6SnWhG1MR2fBuBEsGBOTEdWd0aqcXTz56VpemL8hbJ8yQPe0JE6bMphj9hiAAOVV1ZRXeo+Q5xXeclllNZVVypDe6Rw+NoueGbH5cWGMaTkWjBvBgrHnr1Mgd1nt8q/ehcFT4leedmxXWSX//noj//hkDcu37Wrx/ScmCPuPyOSHE7M5cmI2/bqntfgxjDHNZ8G4ESwYe+qNqL4PJp0Zv/J0AKrKvJXbefyTNby7ZKvv5CEtYa/BPfnhbtkcNTE76svBjDGxFy4Y2y0UTXh9x9cNxjlL41eWDkJEOHBUHw4c1YcN+cU89ek6/vXFOgqKK1r0OAvWF7BgfQG3/ed7xvbrxlET+3HUbtlM6N/dBowZ0wa1ejAWkcHAPcB0QIB3gUtUdV0D2w0F7gX2ArKAImARcLuqvhmSdwhwI3A40BdYDzwH3KqqkS9sNbWy6t69iW1L4lOODmpQrwyu/NE4LvnBaF79ZhOvLthEzs4yUpISSElKIDlRSElKJCUxgdSaZbcuJTGR5CShskr5aHkOy7aGb/peunUnS7fu5N73VzCoVzpHTcxmSGYGSYlCUoKQmOD2nZggJCUkkJQg3rqEmjyZXVIY0bdrK746xnQurRqMRSQDeB8oA84CFLgJ+EBE9mggUHYFcoEZwAagO3Au8IaInKyqL3nH6IIL8MnANcA6YApwPTAaOC0Gp9Yx9R1fdznn+/iUo4NLS07kJ5MH85PJg5u8j1U5u3h70VbeXrSFBRFuMbkhv4RHP17dpGPsObgn9/xkTwvKxsRAq/YZi8jFwN3AWFVd4aUNB5YDl6vq3Y3cXxKwGligqsd5aUcCbwNHqeo7QXlvAy4DuqtqxHvkWZ+xp7Icbs4OGVG9DtJ6hN+mqgIqStwjIQm69I5tGct2QWIyJNndkgI27yjhv4u38tbCLXy2Oq/ePZmbo0tKIreevAfH7zmgxfZpTGfSVvqMjwc+DQRiAFVdLSJzgRNwgTpqqlopIjuA4GtGAtd3hN4iqABIwDWNm2gkpUDvkXVHVD95EkgCVJZARSlUlrrAG/irITde6DEYhh0MQw9yf3sNg+b0WRZugjVzYe3HsOZj2L7CBf2B+9QeY/B+kNp5a2/9e6Rz5gHDOPOAYeQXlfPe99t4a+EWPlqeQ1mYez9Hq6i8it//82s+W7Wda46dEPaGHcaYxmntmvEW4BVVPS8k/QHgVFXtG8U+EnBBtQ/wG1yz9Y9U9T1vfRrwLbAZuADXTL0v8DTwb1X9bUPHsJpxkGfPgCWvttz+ug+sG5wzR0QOzjs21A2+easaPkZCEgzY2zvGITBkP0iNfOOHGqpQvB22r3THCjyKcqD7AMgcCZnDXbl7j4zcStDGFJVV8uGyHBasL6C0ooqKKqWq2l2rXFmtVNZ5rlRWueXC0gq+37Kz3v7G9+/OA6dPstHaIaqrlVW5u6ioUkb27Wq30zR1tIlLm0SkHLhbVa8MSb8JuFJVG6ypi8idwB+9xV3AWYH+4qA8WcCLwMFByY8A56mqb9VARH6DC+4MGTJkn7Vr10Z3Uh3dh3fABy07J3Ud3frXBuZhB7vm5uDgm7+m+ceQRBiwV93gXFlWP+DmrYS81VDmf99lXxm9XWDOHOn99R69R0B6r+aXvQ1QVZ76bB03vraY8qq6/z5dUhK57eQ9OK6TN1uvzytm7opcPl6Ryycrt5NXVA5AalICuw/swd5DejJpSC/2HtKL7B52DXhn1pGC8SAg23uciWv6PkVVX/fWpwH/AQbgRlQHasbXAk+r6gUNHcNqxkEK1sHsw6HY/9699UgiJKdDUpoLalXlsS0fghsH2AZ1yYKJJ8K+v4E+o1p23zlL4fPZsOxtV2Ofci5M/LHrP4+RhRt38LtnvmKtzy0jT99vSKdqts4vKueTlduZuzKXuStyfV+TcPr3SPMCc0/2HtKLiQO6d5rXrbyympLyKorKKykur6KkvIriwPOKKlISE+ienkz39CS6pyXTPT2ZLimJHepyvLYSjLcCLzenmdpnn3OAbFUd5y3/DvgrMEpVVwblOxeYDeylqt9E2qcF4xBFubDhC9eEm5wGSeku4AaCbvDf4GBQUQob57sa7pqP3T4qS5tXloQkGDAJhgX1D5cWwtq5tcfJW9nwflrbyCNcUB59JCQ0sdmyugqWvQWfPQSrP6y/vscQOOC3sPcZMeszLyyt4KoXv+ON7zbXWzehf3fu76DN1qUVVXyxJo+PV7jgu2hTYdi7czVWSmICEwZ0Z+8hPRnYM73RgSctOYGMlETSk5PokppY8zwjJZGM1EQyUpLISE4kIaH+fiurqimucEGxqKw2KLpAWUlRWRXFFVWUllfVTssaND1rzdSsPmklFYFg64JvSXkVlU0YTJgg0C0tKEAHPe+SmkRldTUl5dWUVlZRVlFFaUU1pRVVlFYGPa+odusqq6hW6N0lhazuqfTtmkpWtzSyuqeS1S2VvnWep8bklqdtJRi/D6So6sEh6XO8shzWhH3eibtOOclb/hvwE1XNDMm3J7AA+Jmq/ivSPi0Yx0hlGWz8sjZorv/cDQSLJCHZDc4KBN9B+zYcaAo31w3O25c3rpzJGV5z8/Da5ueuWa7/Om91bZN2/prG1/x7DXO12L1/Aek9o9umOA++fhK+eMS1VDQkrSfsey7sex50bfTv2wapKk99upYbX19Sr9m6a2oSt560e7tvti4oLuerdfnMX5PP/LX5LFhfQHkjBr91S02iS2oSWwqb+eOzBbmgnURKYgKllVUUl1XVe/9MXT0zksnq5gJ2326pXDRtVLMv7WsrwfgS4E5gjKqu8tKG4S5tulJV72rk/hKAT4BeqjrWS5sJXAeMDh617fUJPwQcqqofRdqvBeNWUlkOm76qG5yrK2Dg5LrBNyWjecfZudXrg57rgnTO95DStXYgVnB/b++R0LVfdCO+q6ugcKMLzjX9z6tr+6CrysJvm5wBe5zmasv9Jvjn2fKdqwV/93zTWhQSU2Gvn8OBF7nzaoyKEnf8jV+6R1EO9B7tfhgNmgyZI1m4eSe/ffor1uXVb6L9xf5DmHFM85qtq6uVkoraWlVxUJNmoImzpKKKHunJDOiZzoAeafTpmupbA4xEVVmzvZj5a/JqAnBj5w9PThT2GdqLg0f14aBRfdh9YA+SEhPYvKOEr9cV8NXafL5eX8B3G3c0KqibtuXN3x/ChAHNu1lOWwnGXYBvgBLcKGjF9et2A/ZQ1V1evqHASuAGVb3BS5sJZAJzgS24PuNfAT8Afh6o7XrB/Vsvz824PuPJuAlAlgH7hhvEFWDBuIOrqoSExOZdYtWQynJY/Ap8/pBrno9k2CEuKI89GlA3BennD8O6TyJvN3h/N1f4xvnw9dMRgr/A+OPgoEtg0D71V1dXu9aDjV/Chvnu79aFUB3+LlOk9YABkyjL3pu/r87k0dW9yaXuyPKx/boxvn83KqqVqipvtHa1eiO23Ujtiuq6I7pLK2qDbmlF44NWcqKQ3SON/j3SGdgznf490hjYPYlhqTsZmFBAX/JI0TKWJo1mbn4mX67N56t1+eTuavzYhgn9u3PwaBd8pwzrRUZKw1eKllVWsWTzzprg/NXafDYWNNA61IEkJohrPk/xms+95+kpSaQnJ1BWWU1hSQU7SyspLK2gsKSSkoqqhnfcSubP+AF9ujZvToM2EYy9ggyh7nSY7+GamdcE5RmGm8zjelWd6aUdD1wC7Ab0wAXbb3DTYc4NOcYEYCZwAO4SqPXAq8DNqprfUBktGJsWtfErN9hq4YuRm7W7DwKthp2bwudJTIXdT3XN0AP2qk3ftc0d4/OHobQg/PZDD4YDfgdobeDd9HXjRpCHsVH78HX1SL6pHsmC6lGs1v4kU0malJOG95ByUiknPWjZrasgTcpRoERTKSXFPTSZUlIoq1lOoSToeTlJZMpOsiWPbPLpJ3n0lzyyxT3Plnz6sIMEqf8993HVRJ6oOor3qidRTcP9+IN6pXOIF3wPGNGb3s38Ug7YVljKV+sK+G5jAcXljQs8qi7AF5V5rQUVrp+3pLyK4opKimvS/fcrAhnJiWSkJnl9zYl0CXru+p2TSEtKJDU5gWRvataUxISgaVsDU7R667y0QLN4bcB1U7s2tk+8vLKanaUVFJZWhgTqCnaVVZKSlFBTvrTkRPdICnoeSPfyAOTsLGPbzjJydpbWPN9WWMa2naXu+c4ytu8qq3Mjl6QEYdlNP2p0y0v917yNBOP2wIKxiYldOfDVEzD/7655uzG6D4Ipv4JJZ0We1axsl+tfnnc/7FjfvPJ2Euur+/Jk1Q94tupwduD6A0VczX7ysF5MHprJPkN7MTizmd0lcVRdrZR6Qbu8qrom0KYmNT44dhZV1cr2Ihekc3aWUVhawQl7DWz2fi0YN4IFYxNTVZXw/euuJrt2buS8ww6B/c6DMT+CxEZMmFdVAYv+DXP/4pqcm6LnENd/P3Af6DEoqA/5Kyjb0bR9tmHlpLA060eU7/Nrxux5AN3SGnmZWGUZbFnoXqPNC1zawH3ce9hndGy7RUy7YcG4ESwYm1azZaELyt8+VzuyPDkD9vypG3UdbnBXtFRh5fsuKPtdDhWQ1tMFjsAArQGTwo/Erq52o8kDg7s2zHeBurplbwPZknYk9GQrvdlQ2ZMeFLJPQgMj7Icc6LoCxh/nf+22qhu0t/FL12e/8Uv3GoTrhuiS5QYlBiae6TvWgnMnZcG4ESwYm1ZXnOdqy5IA446N/rKnxtj0tWu+3rTADcAKBN6B+zQ8LWlDamqF82uDdPF275r0NJ+/af7XqasGzXvu3XCk5rnf31J3Lt0HQvf+bka37gPco9sAl9Y1282zjhs5XVZZTVqu9yPouxcij1TvNgAmn+MmVNm+su75lTajdSCjjxecvZnn+o5r+vXnnY2qm6N+6yIoWOs+N+mZbsa74EdSSsP7igMLxo1gwdiYTqI4z/Xjf/FofPvY0zNdcM4cQePuZaPuh1DgTmnhbuBSWQoVxW5ddYUbCBj2R1JG/XUpGV6Q8wl6sQx8pYXuPurbFsHWxS4Ab1sU3Q+h5C6Qkel+2AaXNbW7+9HbFIf8EdI6wKVN7YUFY2M6mapKWPYfV1te/b+m76f7IBg4ybU4VFe6a9vXfQoVkW7V3gH4Bb7UbuFbRpIz6reKJCRB/moXcLcudkE3mkluWtOl37vWlmZoK7dQNMaYticxyfUPjz/O1cQ+nw3f/MvVJsNJ7e7uDhZo6h+4D3TLrpvnkD+6wXSbv4E1H3nBeR6UN25SkTavogh2FNkI/mawmrEPqxkbYygpgAXPwNdPuSCTOaJu4O09umn9vFWVsOUbb+Y5Lzi3wHXenUpKV8iaAH3GuBaIknwoyfP+eo/Iczs1TQxrxhaMfVgwNsa0muoq2PItrP+iaTXmpLQw/b9hmoQTk10/s2+/ckj/cqAPunyX+3ESGvBiGfjA3QWu9yh3VUHWROg30T3vMSTyD6Hqaijf6cpWHFLmsvr35o7avudGf2/0MKyZ2hhj2qKERNfcPWDv1jtmUirQvIFINaqrXc0+NECX72pgJHzoYLMydzldv4le4J0Afca6HxSNlZDgRtmn9XA3Z2kHLBgbY4xpuoQEb+BWT2B4nAvTftmFbcYYY0ycWTA2xhhj4syCsTHGGBNnFoyNMcaYOLNgbIwxxsSZBWNjjDEmziwYG2OMMXFmwdgYY4yJMwvGxhhjTJzZ3NQ+RCQHWOuzqg+Q28rFaSs687mDnb+dv52/nX/LGKqqfUMTLRg3gojM95vguzPozOcOdv52/nb+dv6xPX9rpjbGGGPizIKxMcYYE2cWjBtndrwLEEed+dzBzt/Ov3Oz848x6zM2xhhj4sxqxsYYY0ycWTA2xhhj4syCcQNEZLCIvCAiO0SkUEReEpEh8S5XaxCRqSKiPo+CeJetpYnIIBG5T0TmiUixd57DfPKlicifRWSziJR4+Q+NQ5FbVCPO3+/zoCKyV+uXumWIyCki8qKIrPXe06UicquIdAvJ10tEHhGRXBEpEpF3RWT3eJW7pURz/iIyLMJ73zOOxW82ETlKRN4XkS0iUiYiG0TkORGZEJIvprEgqaV21BGJSAbwPlAGnAUocBPwgYjsoapF8SxfK/o98EXQcmW8ChJDo4CfAF8CHwFHhsn3KHAM8H/AKuB3wNsicoCqLmiFcsZKtOcP8DjwUEjastgUq1VcBqwDrgY2AHsDM4HDReRAVa0WEQFeA4YBFwH5wFW474K9VHVDPAreQho8/6C8twKvhmy/szUKGUOZuM/9A0AOMAS4EvhURHZX1bWtEgtU1R5hHsDFQBUwKihtOC4YXRrv8rXC+U/1PnQ/iHdZWuFcE4Ke/9o772Ehefb00n8ZlJYELAVejfc5xPr8vXUK3BTv8rbwuff1STvTO9dp3vIJ3vLhQXl6AHnAvfE+h1Y4/2He8q/jXd5Wek3Geuf7R2855rHAmqkjOx74VFVXBBJUdTUwF/fPaToIrfvrP5zjgQrg2aDtKoF/AUeJSGqMihdzUZ5/h6SqOT7JgZaggd7f44FNqvpB0HY7cLXldv1dEOX5dzbbvb+BVsCYxwILxpFNBBb6pC8CJvikd1RPi0iViGwXkWc6S5+5j4nAalUtDklfBKTgmno7gwu8vrVir6/tkHgXKAYO8/4u8f5G+i4YIiJdW6VUrSf0/ANuFZFKr9/01Y7QZx4gIokikiIio3HdMFuAf3qrYx4LrM84skxc31CoPKBXK5clHnYAdwEfAoW4vqSrgXkisreqbotn4eIg0uchsL6jewp4HdgEDMX1nb8vItNVdU48C9ZSRGQgcAPwrqrO95IzgTU+2QPvfS9gV+xLF3thzr8MF6DewfWrjsN9F3wiIvuqamjQbo8+A/bxnq/ANdEHvuNiHgssGJuwVPVr4OugpA9F5H/A57hBXTPiUjATN6p6RtDiRyLyCq7GcBNwcHxK1XK8Gu4ruObJX8a5OK0u3Pmr6mbg/KCsH4nIW7ia4Z+AX7RmOWPkDKA7MAI3qO2/InKwqq5pjYNbM3Vk+fj/6gn3K6nDU9WvcCNnp8S7LHEQ6fMAtbWkTkNVdwJv0AE+DyKSjusDHgEcpXVHSDf03rf774MGzr8eVV0PfEwHeO8BVHWJqn6mqv8EjgC64kZVQyvEAgvGkS3C9RWEmgAsbuWytDWdcR7VRcBw7zKHYBOAclzTVmfVrj8PIpIMvABMBo5W1e9CskT6Llinqu26iTqK84+kXb/3flS1APf/HBgHEvNYYME4sleB/UVkRCDBmwjhIOpfa9cpiMhk3LD/z+Ndljh4DUgGTg0kiEgScBrwjqqWxatg8SIi3YFjacefBxFJAJ4GpgE/VtVPfbK9CgwUkcOCtusOHEc7/y6I8vz9thuC65pot+99OCLSD9cvvtJLinkssBtFRCAiXYBvgBJc/6gCNwLdgD3a+6/hhojI08Bq4CugADeA6yqgGJikqrnxK13LE5FTvKdH4PrHfosbrJKjqh96ef4FHIUbuLQauAAXjA70mvDbrYbOX0Quw/0Q+4DaAVyBtCNU9aPWL3XziciDuPO9GTc4LdgGVd3gBayPgcG49z4w6ccewJ5ek227FOX534WrvM3DfSbG4s6/B7Cfqi5txSK3KBH5N+477lvcQNUxwB+AbGBfVV3WKrEg3hdXt/UHbjaWF703aSfwMj6TIXTEB+6f7VvcqOoKYD3uVmL94122GJ2vhnnMCcqTDtyNu+yhFDcCc2q8y94a54+rBc4Fcr3Pw3ZcrWDfeJe9mee9JsK5zwzKlwn8HTc2oBh4DxeI434OsT5/4Bzctcf53nu/BXgGGBvv8rfA+V+Bm4GrwHtfl+JGjg8LyRfTWGA1Y2OMMSbOrM/YGGOMiTMLxsYYY0ycWTA2xhhj4syCsTHGGBNnFoyNMcaYOLNgbIwxxsSZBWNjWpGInCEi64KWF4vIb1v4GAeIyGciUiQiKiJ7hck3U0Q0aLmnlzapJcvTGCKyl1eGenfA8s5lZhyKZUzMWTA2pnXtg5tgIHCHnLGB5Rb0KO6ObMcBB+Bu7OHnEW99QE/gOiBuwRjYyyuD3+0oD8CV2ZgOx26haEzr2gd423s+CajGTbPXIrxpG8cCN6vq+5HyqrsrT8Q787RAeQRIVtXy5u5Lo5wz2Zj2yGrGxrQSL1DuRW1NeDKwWFVLo9y+u4j8VUQ2iUiZiCwVkT94AQ8RORuowv1fX+M1666JsL+aZmpv0vvV3qqHvW3V22cg/0ki8qmIFItIgYg8790sIHifa0TkKRE5R0S+x93N6hhv3fUi8pWIFIpIroi8LyL7B217NvCYt7g8qAzDvPX1mqlF5IciMk9ESkRkh4i8LCJjQ/LMEZGPReQH3vGLRWShiJwYkm+MiPxbRLaJSKmIrPPO0SotJuYsGBsTY16AUlyg7Aq86S3fBewRGnTC7CMBd9/gX3rbHQe8hZsn+2Yv2xu4u+iAa6o+ADiR6GwGTvKe3+pte4C3T0TkfNy8vIuBU4DzgN2AD0WkW8i+DgcuBa4Hfoib3xxgIHAPcAJwNrAN+J+I7B5U/pu856cGlWGzX4FF5IfeNrtwd866wCvTxyIyMCT7SOAvuNfrJG+fz4vIqKA8b3hlvAB3M5ArgTLse9K0hnhP0m0Pe3T0B+6ep3vhAsEi7/leuAnn/xC0nBJhH8fiJu4/OyT9EVzA6OMtJxFyg4MI+5zpvgJqlod52/46JF9X3M1C/h6SPhxX870kKG0NbrL97AaOneiVdSnwl6D0s70yjPLZJvTGDfOB5UBSSJkqgLuD0uZ4aaOD0rJwP46u9pb7ePs/Pt6fF3t0zof94jMmxlR1saouwN1+b473vAh3+7XnVXWB94jUr3oorn/5mZD0p4AU6g7EamkHAN2Bp0UkKfDA3cXre69swT5V1S2hO/GaiT8Qke1AJS5AjsH1cTeKd0u7ScCzqloZSFfV1bg7Sx0WsslyVV0elG8brmYeaGbfDqwCbhORc0VkdGPLZExzWDA2JoZEJDEoeB0EzPOeHwJsBLZ466WBXWUCeT4Be0vQ+ljJ8v6+iwugwY/dgd4h+es1K3uXS72Ja1L+FbA/MAU3eC2tCWXqBYjfsXCvSejrkeeTryxwbFVVYDqutn0rsExEVonIBU0omzGNZgMTjImt96hbS3vSewRUeH8PxzWnhpMHZIpISkhAzg5aHyvbvb9n45rZQ+0MWfa7L+vJuNrwSaoaOGdEpBfuPrKNle8dJ9tnXTZNeD1UdRVwpvfDaE/gQuABEVmjqv9pQhmNiZrVjI2JrfNwNcA7gRXe8ylADjAjaLmha40/xP2/nhqSfjqu33ZeC5S1zPubHpL+CS7gjlLV+T6PpVHsOwPXRxs8ycg0apuJGypDHapahHvNThWRxKB9DgUOJPIPm4jUWYAbhAZuUJgxMWU1Y2NiKBCoROQa4A1Vne9detMHeNSvbzWM/wAfA38Tkb64GurRwK+BW1U1twWKuxVXC/6piHyL69derarbReT/gPu9Y/8HN6BrIK7WP0dVQ/uyQ70FXAI8LiKP4fqKr8E11Qdb7P39nYg8gWs5+DZMf/o1uBHQr4vIA7iBZtd7Zbsr+tMGEdkDN9r6WdyPpkRcS0AlEPF6bWNagtWMjYkxEUkBjsAFJIAfAV83IhCjqtW463WfAK7ABaFjcLW3P7VEOb1j/BrXH/su8AXuEipU9SHgeNxgqydx/b8zcT/oF0Sx77eB3+P6zV8HzgHOxAW+4HzfePs9Dvfj4wtgQJh9voV7DXoCzwF/A5YAB6vqpqhOutYWYB3u9XwV+Kd33GNVtaVnSDOmHnHjFowxxhgTL1YzNsYYY+LMgrExxhgTZxaMjTHGmDizYGyMMcbEmQVjY4wxJs4sGBtjjDFxZsHYGGOMiTMLxsYYY0yc/T+TqaCfnfAaaAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 504x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.rcParams['figure.figsize'] = 7, 5\n",
    "plt.plot(range(1,31), error_all, '-', linewidth=4.0, label='Training error')\n",
    "plt.plot(range(1,31), test_error_all, '-', linewidth=4.0, label='Test error')\n",
    "\n",
    "plt.title('Performance of Adaboost ensemble')\n",
    "plt.xlabel('# of iterations')\n",
    "plt.ylabel('Classification error')\n",
    "plt.rcParams.update({'font.size': 16})\n",
    "plt.legend(loc='best', prop={'size':15})\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Quiz Question:** From this plot (with 30 trees), is there massive overfitting as the # of iterations increases?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'no' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-117-49dddb3b048c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mno\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'no' is not defined"
     ]
    }
   ],
   "source": [
    "no"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}