1st Part (Dataset A).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ομάδα #75 \n",
    "## Νικόλαος Μανιάτης (ΑΜ: 03400097) - Φίλιππος Μαυρεπής (ΑΜ: 03400098)\n",
    "\n",
    "##### Datasets: S12,\tB04,\t119"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset B04"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:12:28.026133Z",
     "iopub.status.busy": "2021-03-14T20:12:28.026133Z",
     "iopub.status.idle": "2021-03-14T20:12:28.042201Z",
     "shell.execute_reply": "2021-03-14T20:12:28.041204Z",
     "shell.execute_reply.started": "2021-03-14T20:12:28.026133Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from statsmodels.stats.outliers_influence import variance_inflation_factor\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LinearRegression,Ridge,Lasso\n",
    "from sklearn.svm import SVR\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.neural_network import MLPRegressor\n",
    "from sklearn.ensemble import ExtraTreesRegressor\n",
    "import xgboost as xgb\n",
    "from xgboost import XGBRegressor\n",
    "from sklearn.model_selection import cross_val_score,KFold\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "import seaborn as sns\n",
    "from sklearn import preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Dataset Abstract**\n",
    "NASA data set, obtained from a series of aerodynamic and acoustic tests of two and three-dimensional airfoil blade sections conducted in an anechoic wind tunnel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Attribute Information:**\n",
    "\n",
    "This problem has the following inputs:\n",
    "1. Frequency, in Hertz.\n",
    "2. Angle of attack, in degrees.\n",
    "3. Chord length, in meters.\n",
    "4. Free-stream velocity, in meters per second.\n",
    "5. Suction side displacement thickness, in meters.\n",
    "\n",
    "The only output is:  \n",
    "6. Scaled sound pressure level, in decibels.\n",
    "\n",
    "It is obvious all attributes are numerical."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:07:17.383464Z",
     "iopub.status.busy": "2021-03-14T20:07:17.383464Z",
     "iopub.status.idle": "2021-03-14T20:07:17.398545Z",
     "shell.execute_reply": "2021-03-14T20:07:17.398545Z",
     "shell.execute_reply.started": "2021-03-14T20:07:17.383464Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1503, 6)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>freq</th>\n",
       "      <th>angle</th>\n",
       "      <th>length</th>\n",
       "      <th>velocity</th>\n",
       "      <th>thickness</th>\n",
       "      <th>pressure</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>800</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3048</td>\n",
       "      <td>71.3</td>\n",
       "      <td>0.002663</td>\n",
       "      <td>126.201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3048</td>\n",
       "      <td>71.3</td>\n",
       "      <td>0.002663</td>\n",
       "      <td>125.201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1250</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3048</td>\n",
       "      <td>71.3</td>\n",
       "      <td>0.002663</td>\n",
       "      <td>125.951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1600</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3048</td>\n",
       "      <td>71.3</td>\n",
       "      <td>0.002663</td>\n",
       "      <td>127.591</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3048</td>\n",
       "      <td>71.3</td>\n",
       "      <td>0.002663</td>\n",
       "      <td>127.461</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   freq  angle  length  velocity  thickness  pressure\n",
       "0   800    0.0  0.3048      71.3   0.002663   126.201\n",
       "1  1000    0.0  0.3048      71.3   0.002663   125.201\n",
       "2  1250    0.0  0.3048      71.3   0.002663   125.951\n",
       "3  1600    0.0  0.3048      71.3   0.002663   127.591\n",
       "4  2000    0.0  0.3048      71.3   0.002663   127.461"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features = ['freq','angle','length','velocity','thickness','pressure']\n",
    "df = pd.read_table('./datasets/airfoil_self_noise.dat',names = features )\n",
    "print(df.shape)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are no null values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:07:17.634104Z",
     "iopub.status.busy": "2021-03-14T20:07:17.634104Z",
     "iopub.status.idle": "2021-03-14T20:07:17.650367Z",
     "shell.execute_reply": "2021-03-14T20:07:17.650367Z",
     "shell.execute_reply.started": "2021-03-14T20:07:17.634104Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "freq         0\n",
       "angle        0\n",
       "length       0\n",
       "velocity     0\n",
       "thickness    0\n",
       "pressure     0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Correlation Matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:07:17.916375Z",
     "iopub.status.busy": "2021-03-14T20:07:17.916375Z",
     "iopub.status.idle": "2021-03-14T20:07:17.936973Z",
     "shell.execute_reply": "2021-03-14T20:07:17.936001Z",
     "shell.execute_reply.started": "2021-03-14T20:07:17.916375Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>freq</th>\n",
       "      <th>angle</th>\n",
       "      <th>length</th>\n",
       "      <th>velocity</th>\n",
       "      <th>thickness</th>\n",
       "      <th>pressure</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.272765</td>\n",
       "      <td>-0.003661</td>\n",
       "      <td>0.133664</td>\n",
       "      <td>-0.230107</td>\n",
       "      <td>-0.390711</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>angle</th>\n",
       "      <td>-0.272765</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.504868</td>\n",
       "      <td>0.058760</td>\n",
       "      <td>0.753394</td>\n",
       "      <td>-0.156108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>length</th>\n",
       "      <td>-0.003661</td>\n",
       "      <td>-0.504868</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.003787</td>\n",
       "      <td>-0.220842</td>\n",
       "      <td>-0.236162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>velocity</th>\n",
       "      <td>0.133664</td>\n",
       "      <td>0.058760</td>\n",
       "      <td>0.003787</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.003974</td>\n",
       "      <td>0.125103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>thickness</th>\n",
       "      <td>-0.230107</td>\n",
       "      <td>0.753394</td>\n",
       "      <td>-0.220842</td>\n",
       "      <td>-0.003974</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.312670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pressure</th>\n",
       "      <td>-0.390711</td>\n",
       "      <td>-0.156108</td>\n",
       "      <td>-0.236162</td>\n",
       "      <td>0.125103</td>\n",
       "      <td>-0.312670</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               freq     angle    length  velocity  thickness  pressure\n",
       "freq       1.000000 -0.272765 -0.003661  0.133664  -0.230107 -0.390711\n",
       "angle     -0.272765  1.000000 -0.504868  0.058760   0.753394 -0.156108\n",
       "length    -0.003661 -0.504868  1.000000  0.003787  -0.220842 -0.236162\n",
       "velocity   0.133664  0.058760  0.003787  1.000000  -0.003974  0.125103\n",
       "thickness -0.230107  0.753394 -0.220842 -0.003974   1.000000 -0.312670\n",
       "pressure  -0.390711 -0.156108 -0.236162  0.125103  -0.312670  1.000000"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train-Test-Split (70\\% Train - 30\\% Test) and Min-Max Scaling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Πρώτα εκτελούμε το διαχωρισμό test-train και μετά το min-max scaling ώστε να μην έχουμε διαρροή πληροφορίας από το test set μέσα στο train set (π.χ. πληροφορία από το μέσο όρο)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:15:28.214291Z",
     "iopub.status.busy": "2021-03-14T20:15:28.214291Z",
     "iopub.status.idle": "2021-03-14T20:15:28.231782Z",
     "shell.execute_reply": "2021-03-14T20:15:28.230843Z",
     "shell.execute_reply.started": "2021-03-14T20:15:28.214291Z"
    }
   },
   "outputs": [],
   "source": [
    "features = df.copy()\n",
    "features = features.drop(['pressure'],axis=1)\n",
    "output = df[['pressure']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:15:28.558229Z",
     "iopub.status.busy": "2021-03-14T20:15:28.558229Z",
     "iopub.status.idle": "2021-03-14T20:15:28.573425Z",
     "shell.execute_reply": "2021-03-14T20:15:28.573425Z",
     "shell.execute_reply.started": "2021-03-14T20:15:28.558229Z"
    }
   },
   "outputs": [],
   "source": [
    "X_train_unsc, X_test_unsc, y_train_unsc, y_test_unsc = train_test_split(features,output, test_size=0.3, random_state=75)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:19:51.351304Z",
     "iopub.status.busy": "2021-03-14T20:19:51.351304Z",
     "iopub.status.idle": "2021-03-14T20:19:51.376150Z",
     "shell.execute_reply": "2021-03-14T20:19:51.376150Z",
     "shell.execute_reply.started": "2021-03-14T20:19:51.351304Z"
    }
   },
   "outputs": [],
   "source": [
    "scaler_x = preprocessing.MinMaxScaler()\n",
    "scaler_y = preprocessing.MinMaxScaler()\n",
    "\n",
    "train_X = scaler_x.fit_transform(X_train_unsc)\n",
    "test_X = scaler_x.fit_transform(X_test_unsc)\n",
    "\n",
    "train_y = scaler_y.fit_transform(np.array(y_train_unsc).reshape(-1,1))\n",
    "train_y = train_y.reshape(-1)\n",
    "test_y = scaler_y.fit_transform(np.array(y_test_unsc).reshape(-1,1))\n",
    "test_y = test_y.reshape(-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:19:51.759681Z",
     "iopub.status.busy": "2021-03-14T20:19:51.758681Z",
     "iopub.status.idle": "2021-03-14T20:19:51.767913Z",
     "shell.execute_reply": "2021-03-14T20:19:51.766960Z",
     "shell.execute_reply.started": "2021-03-14T20:19:51.759681Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X train shape: (1052, 5)\n",
      "X test shape: (451, 5)\n",
      "y train shape: (1052,)\n",
      "y test shape: (451,)\n"
     ]
    }
   ],
   "source": [
    "print('X train shape:',train_X.shape)\n",
    "print('X test shape:',test_X.shape)\n",
    "print('y train shape:',train_y.shape)\n",
    "print('y test shape:',test_y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Θα ξεκινήσουμε με δύο απλούς Regressors, τον Ordinary Least Squares και τον Polynomial Regressor. Αυτοί δεν επιδέχονται κάποιου είδους optimization (εκτός από την τάξη του πολυωνύμου) οπότε τους παρουσιάζουμε στην αρχή, πριν συνεχίσουμε με πιο πολύπλοκους αλγόριθμους.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ordinary Least Squares"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:20:11.707068Z",
     "iopub.status.busy": "2021-03-14T20:20:11.707068Z",
     "iopub.status.idle": "2021-03-14T20:20:11.726551Z",
     "shell.execute_reply": "2021-03-14T20:20:11.725548Z",
     "shell.execute_reply.started": "2021-03-14T20:20:11.707068Z"
    }
   },
   "outputs": [],
   "source": [
    "reg = LinearRegression().fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:20:12.361368Z",
     "iopub.status.busy": "2021-03-14T20:20:12.361368Z",
     "iopub.status.idle": "2021-03-14T20:20:12.372249Z",
     "shell.execute_reply": "2021-03-14T20:20:12.371355Z",
     "shell.execute_reply.started": "2021-03-14T20:20:12.361368Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.48701529018157896"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reg.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Polynomial Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:20:21.161880Z",
     "iopub.status.busy": "2021-03-14T20:20:21.161208Z",
     "iopub.status.idle": "2021-03-14T20:20:21.177043Z",
     "shell.execute_reply": "2021-03-14T20:20:21.176025Z",
     "shell.execute_reply.started": "2021-03-14T20:20:21.161880Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import PolynomialFeatures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:25:57.431016Z",
     "iopub.status.busy": "2021-03-14T20:25:57.431016Z",
     "iopub.status.idle": "2021-03-14T20:25:57.459003Z",
     "shell.execute_reply": "2021-03-14T20:25:57.458086Z",
     "shell.execute_reply.started": "2021-03-14T20:25:57.431016Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6223336446282257"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = Pipeline([('poly', PolynomialFeatures(degree=2)),\n",
    "...                   ('linear', LinearRegression(fit_intercept=False))])\n",
    "# fit to an order-3 polynomial data\n",
    "model = model.fit(X_train,y_train)\n",
    "model.named_steps['linear'].coef_\n",
    "model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Εύρεση βέλτιστου βαθμού πολυωνύμου:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:07:20.426406Z",
     "iopub.status.busy": "2021-03-14T20:07:20.426406Z",
     "iopub.status.idle": "2021-03-14T20:07:20.607263Z",
     "shell.execute_reply": "2021-03-14T20:07:20.606262Z",
     "shell.execute_reply.started": "2021-03-14T20:07:20.426406Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "degree of polynomial = 1 and score is 0.48701529018065426\n",
      "degree of polynomial = 2 and score is 0.6223336446282257\n",
      "degree of polynomial = 3 and score is 0.7209720262016102\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAm6klEQVR4nO3dd3xV9f3H8deHECAMGTJkDyEMmRIQxYGKgNYq1bbiaOtoqQOttdKqv9YOq6K0UlxVflTbWsAJuBBUHLghhE0CsklYYYdA9uf3x734S+MNBMjNvbn3/Xw8eHDPvJ+Ey33f8znfe465OyIiImXViHQBIiISnRQQIiISkgJCRERCUkCIiEhICggREQmpZqQLqExNmzb1Dh06RLoMEZFqY+HChTvdvVmoZTEVEB06dCA1NTXSZYiIVBtmtrG8ZWoxiYhISAoIEREJSQEhIiIhKSBERCQkBYSIiIQUU6OYRETiycxFWYyfs4otew/RqlESY4d3ZWS/1pW2fwWEiEg1NHNRFvdOX8ahwmIAsvYe4t7pywAqLSTUYhIRqYbGz1n1TTgcdqiwmPFzVlXacyggRESqoay9h0LO31LO/OOhFpOISDWybV8ej71X/lFCq0ZJlfZcCggRkWogJ6+QZz9ex+RP11Fc4pyX3Iyv1u0ir6jkm3WSEhMYO7xrpT2nAkJEJIoVFJUwbf4mJs79mt25BVzWpxVjh3elbZO6GsUkIhKP3J3Zy7fxyOwMNuw6yKBOTbjvku70btPom3VG9mtdqYFQlgJCRCTKLNiwm4dmpbNo016SW9Tn+esHMKRrM8ysSutQQIiIRIm12Qd45J0M3l25neYNavPIlb248vQ21EyIzIBTBYSISIRl5+Qzce5qps3fTFJiAncPS+bGsztSt1Zk36IVECIiEZKbX8TkT9bz7Ly1FBSVcO0Z7bjjwi40rV870qUBCggRkSpXVFzCy6mZTHh/Ndk5+Vzc8xTGDu9Kp2b1I13af1FAiIhUEXdnbvoOxs3OYM2OA/Rv35hnrjud/u2bRLq0kBQQIiJVYPHmvTw0K53563fTqWk9nrmuP8NPa1HlI5OOhQJCRCSMNu7K5dE5q3h76Vaa1q/FAyN7MmpAWxIjNDLpWCggRETCYHduAU988DX/+XIjNWvU4I4LOjP6vFOpX7v6vO1Wn0pFRKqBvMJinvtsPX//cC25BUVcNaAtdw5NpsVJdSJd2jELa0CY2QhgIpAATHb3cWWWjwWuLVVLd6CZu+8+2rYiItGkuMSZnpbJY++tZuu+PIZ2b85vRnSjS4sGkS7tuIUtIMwsAXgKuAjIBBaY2RvuvvLwOu4+HhgfXP+7wC+D4XDUbUVEooG7M+/rnTw8K52MbTn0adOQCVf1ZVCnkyNd2gkL5xHEQGCNu68DMLMXgcuB8t7krwamHee2IiJVbnnWPsa9k8Gna3bStkkST1zdj0t7t4zqkUnHIpwB0RrYXGo6Ezgj1IpmVhcYAYw5jm1HA6MB2rVrd2IVi4hUQOaeg/z13dXMWJRFo7qJ3H9pD64d1I7aNRMiXVqlCmdAhIpQL2fd7wKfufvuY93W3ScBkwBSUlLK27+IyAnbd7CQpz9aw/OfbwDg5vNO5ZYhp9IwKTGyhYVJOAMiE2hbaroNsKWcdUfx/+2lY91WRCSs8ouKeeGLjTzxwRr25xVyRb823DUsmdaVeHvPaBTOgFgAdDGzjkAWgRC4puxKZtYQOA+47li3FREJp5IS582lWxg/ZxWZew5xbnIz7hnRjR6tTop0aVUibAHh7kVmNgaYQ2Co6nPuvsLMbg4ufya46veAd90992jbhqtWEZGyPl+7k4dnZbAsax89Wp7ECzf14pwuzSJdVpUy99hp26ekpHhqamqkyxCRamzVthzGvZPOh6uyadWwDncP78rIvq2pUSM2RiaVZWYL3T0l1DJ9k1pEBNi2L48J763mlYWbqVe7Jvde3I2fnNWBOomxNTLpWCggRCSu5eQV8uzH65j86TqKS5wbBndkzPmdaVyvVqRLizgFhIjEpYKiEqbN38TEuV+zO7eAy/q0YuzwrrRtUjfSpUUNBYSIxBV3Z/bybTwyO4MNuw4yqFMT7rukO73bNIp0aVFHASEicWPBht08NCudRZv2ktyiPs9fP4AhXZvFzKUxKpsCQkRi3trsAzzyTgbvrtxO8wa1eeTKXlx5ehtqVoOb9kSSAkJEYlZ2Tj4T565m2vzNJCUmcPewZG48uyN1a+mtryL0WxKRmJObX8TkT9bz7Ly1FBSVcO0Z7bjjwi40rV870qVVKwoIEYkZRcUlvJyayYT3V5Odk8/FPU9h7PCudGpWP9KlVUsKCBGp9tyduek7GDc7gzU7DtC/fWOeua4//ds3jnRp1ZoCQkSqtcWb9/LQrHTmr99Np6b1ePZH/RnWo4VGJlUCBYSIVEsbd+Xy6JxVvL10K03r1+KBkT0ZNaAtiRqZVGkUECJSrezOLeCJD77mP19upGaNGtxxQWdGn3cq9Wvr7ayy6TcqItVCXmExz322nr9/uJbcgiKuGtCWO4cm0+KkOpEuLWYpIEQkqhWXONPTMnnsvdVs3ZfH0O7N+c2IbnRp0SDSpcU8BYSIRCV3Z97XO3l4VjoZ23Lo06YhE67qy6BOJ0e6tLihgBCRqLM8ax/j3sng0zU7adskiSeu7selvVtqZFIVU0CISNTI3HOQx95dzYzFWTRMSuT+S3tw7aB21K4ZvzftiSQFhIhE3L6DhTz90Rqe/3wDAD8/91RuGXIqDZMSI1tYnFNAiEjE5BcV88IXG3nigzXszyvkin5tuGtYMq0bJUW6NEEBISIRUFLivLl0C+PnrCJzzyHOTW7GPSO60aPVSZEuTUpRQIhIlfp87U4enpXBsqx99Gh5Ei/c1ItzujSLdFkSggJCRKrEqm05jHsnnQ9XZdOqYR0e+2EfRvZtTY0aGpkUrRQQIhJW2/blMeG91byycDP1atfk3ou78ZOzOlAnUSOTop0CQkTCIievkGc/XsfkT9dRXOLcMLgjY87vTON6tSJdmlSQAkJEKlVBUQnT5m9i4tyv2Z1bwGV9WjF2eFfaNqkb6dLkGCkgRKRSuDuzl2/jkdkZbNh1kEGdmnDfJd3p3aZRpEuT46SAEJETtmDDbh6alc6iTXtJblGf568fwJCuzXRpjGpOASEix21t9gEeeSeDd1dup3mD2jxyZS+uPL0NNXXTnpiggBCRY5adk8/EuauZNn8zSYkJ3D0smRvP7kjdWnpLiSX61xSRCsvNL2LyJ+t5dt5aCopKuPaMdtxxYRea1q8d6dIkDBQQInJURcUlvJyayYT3V5Odk8/FPU9h7PCudGpWP9KlSRgpIESkXO7O3PQdjJudwZodB+jfvjHPXNef/u0bR7o0qQJhDQgzGwFMBBKAye4+LsQ6Q4C/AYnATnc/Lzh/A5ADFANF7p4SzlpF5L8t3ryXh2alM3/9bjo1rcezP+rPsB4tNDIpjoQtIMwsAXgKuAjIBBaY2RvuvrLUOo2Ap4ER7r7JzJqX2c357r4zXDWKyLdt3JXL+DmreGvpVprWr8UDI3syakBbEjUyKe6E8whiILDG3dcBmNmLwOXAylLrXANMd/dNAO6+I4z1iMgR7M4t4IkPvuY/X26kZo0a3HFhF0af24n6tdWJjlfh/JdvDWwuNZ0JnFFmnWQg0cw+AhoAE93938FlDrxrZg486+6TQj2JmY0GRgO0a9eu8qoXiRN5hcU899l6/v7hWnILirhqQFt+OTSZ5ifViXRpEmHhDIhQjUoP8fz9gQuBJOALM/vS3VcDg919S7Dt9J6ZZbj7vG/tMBAckwBSUlLK7l9EylFc4kxPy+Sx91azdV8eQ7s35zcjutGlRYNIlyZRIpwBkQm0LTXdBtgSYp2d7p4L5JrZPKAPsNrdt0Cg7WRmMwi0rL4VECJybNydeV/v5OFZ6WRsy6FPm4ZMuKovgzqdHOnSJMqEMyAWAF3MrCOQBYwicM6htNeBJ82sJlCLQAtqgpnVA2q4e07w8TDgT2GsVSQuLM/ax7h3Mvh0zU7aNkniiav7cWnvlhqZJCGFLSDcvcjMxgBzCAxzfc7dV5jZzcHlz7h7upnNBpYCJQSGwi43s07AjOCLtiYw1d1nh6tWkViXuecgj727mhmLs2iYlMj9l/bg2kHtqF1TN+2R8pl77LTtU1JSPDU1NdJliESNfQcLefqjNTz/+QYAbhzckVuGnErDpMTIFiZRw8wWlvc9M41fE4lB+UXFvPDFRp74YA378wq5ol8b7hqWTOtGSZEuTaoRBYRIDCkpcd5cuoXxc1aRuecQ5yY3454R3ejR6qRIlybVkAJCJEZ8vnYnD8/KYFnWPnq0PIkXburFOV2aRbosqcYUECLV3KptOYx7J50PV2XTqmEdHvthH0b2bU2NGhqZJCdGASFSTW3bl8eE91bzysLN1Ktdk3sv7sZPzupAnUSNTJLKoYAQqWZy8gp59uN1TP50HcUlzg2DOzLm/M40rlcr0qVJjFFAiFQThcUlTP1qExPnfs3u3AIu69OKscO70rZJ3UiXJjFKASES5dyd2cu38eicVazfmcugTk2475Lu9G7TKNKlSYxTQIhEsdQNu3lwVjqLNu0luUV9nr9+AEO6NtOlMaRKKCBEotDa7AM8OjuDOSu207xBbR65shdXnt6Gmrppj1ShowaEmSUDfwdauHtPM+sNXObufw57dSJxJjsnn4lzVzNt/maSEhO4e1gyN57dkbq19FlOql5FXnX/C4wFngVw96VmNhVQQIhUktz8IiZ/sp5J89aSX1TCtWe0444Lu9C0fu1IlyZxrCIBUdfd55fpeRaFqR6RuFJUXMLLqZlMeH812Tn5XNzzFMYO70qnZvUjXZpIhQJip5mdSvBucGb2fWBrWKsSiXHuztz0HYybncGaHQfo374xz1zXn/7tG0e6NJFvVCQgbiNwS89uZpYFrAeuDWtVIjFs8ea9PDQrnfnrd9OpaT2e/VF/hvVooZFJEnWOGBBmlgDc4u5DS9/lrWpKE6n+Zi7KYvycVWzZe4jmDWrTqlESizbvpWn9WjwwsiejBrQlUSOTJEodMSDcvdjM+gcf51ZNSSKxYeaiLO6dvoxDhcUAbM/JZ3tOPsNPa8Fff9iX+rU1MkmiW0VeoYvM7A3gFeCbkHD36WGrSiQGjJ+z6ptwKG151n6Fg1QLFXmVNgF2AReUmueAAkKkHMUlTtbeQyGXbSlnvki0OWpAuPsNVVGISKzYkZPHnS8uLnd5K932U6qJo54dM7M2ZjbDzHaY2XYze83M2lRFcSLVzedrdnLJxE9J27SHqwe2JanMvRmSEhMYO7xrhKoTOTYVGT7xPPAG0ApoDbwZnCciQcUlzt/eX821//iKhkk1ef22s3n4it48fEUvWjdKwoDWjZJ4+IpejOzXOtLlilSIufuRVzBb7O59jzYvGqSkpHhqamqky5A4k52Tz50vLeKzNbv4Xr/W/HlkT+rpJLRUE2a20N1TQi2r6DeprwOmBaevJnDSWiTufb52J794cTH7DxXy6JW9+UFKG33hTWJGRQLiRuBJYAKB0UufB+eJxK3iEuepD9fwt/dX07FpPV64aSDdTjkp0mWJVKqKjGLaBFxWBbWIVAvZOfn88qXFfLpmJyP7tuLB7/VSS0liUkVGMf3LzBqVmm5sZs+FtSqRKPXF2l1c8vgnLNiwm0eu7MWEq/oqHCRmVeSV3dvd9x6ecPc9ZtYvfCWJRJ+SYEtpwvur6XByPf5940C6t1RLSWJbRQKihpk1dvc9AGbWpILbicSEnQcCLaVPvt7J5cGWki6VIfGgIq/yvwKfm9mrwekfAA+GrySR6PHlul3cMW0Rew8V8vAVvRg1oK1GKUncqMhJ6n+bWSqBazEZcIW7rwx7ZSIRVFLiPP3RGh57L9BS+ucNA+nRSi0liS9HDYjg3eTWuvtKMxsCDDWzLaXPS4jEkl0H8rkz2FK6rE8rHrpCLSWJTxW51MZrQLGZdQYmAx2BqWGtSiRCvloXGKX01frdPPS9Xkwcpfs2SPyqSECUuHsRcAUw0d1/CbSsyM7NbISZrTKzNWZ2TznrDDGzxWa2wsw+PpZtRSrL4VFKV//vl9StVZOZtw7mmjPa6XyDxLWKfDQqNLOrgR8D3w3OSzzaRsHblT4FXARkAgvM7I3S5y+C3694Ghjh7pvMrHlFtxWpLLsO5PPLl5cwb3U2l/ZuycNX9KJBnaO+xEViXkUC4gbgZuBBd19vZh2B/1Rgu4HAGndfB2BmLwKXA6Xf5K8Bpge/rY277ziGbUVO2IINu7l96iJ2Hyzgwe/15JqBOmoQOawio5hWAneUml4PjKvAvlsDm0tNZwJnlFknGUg0s4+ABgRaWP+u4LYAmNloYDRAu3btKlCWSKCl9My8tfz13dW0bZzEjFvP4rRWDSNdlkhUCefZt1Afw8peW7wm0B+4EEgCvjCzLyu4bWCm+yRgEgQu933c1Urc2J1bwC9fWszHq7P5Tu+WjFNLSSSkcAZEJtC21HQbYEuIdXa6ey6Qa2bzgD4V3FbkmKVu2M2YqYvYnVvAAyN7cp1ORIuUqyKjmI7XAqCLmXU0s1rAKAJ3pivtdeAcM6tpZnUJtJHSK7itSIWVlDjPfLyWqyZ9Se3EGky/9Sx+NKi9wkHkCMo9ggiOJPopgU/vs939s1LLfuvufz7Sjt29yMzGAHOABOA5d19hZjcHlz/j7ulmNhtYCpQAk919efA5vrXtifygEr925xbwq5cX8+GqbL7TqyXjrlRLSaQiyr3lqJlNBuoC84EfAR+7+13BZWnufnqVVVlBuuWolJW6YTe3T1vErgMF/O7S7lynowaR/3K8txwd6O69gzt4EnjazKYTuOWo/odJVCspcSZ9so7xc1bRulES0289i56tNUpJ5FgcKSBqHX4Q/Cb1aDO7H/gAqB/uwkSO157cAn71yhI+yNjBJb1OYdyVvTlJLSWRY3akgEg1sxHuPvvwDHf/k5ltAf4e/tJEjt3CjXu4fWoaOw8U8KfLT9OJaJETUG5AuPt15cyfTOCifSJRo6TEmfzpOh6dvYqWjerw2i1n0auNWkoiJ6Iil/tOcPfiqihG5HjsPVjAr15ewtyMHVzcM9BSapiklpLIiTpiQJhZA2AacGnVlCNybNI27eH2qYvYkZPHHy87jR+fqZaSSGU50vcgWgIz0e1FJQq5O5M/Wc8jszO+aSn1btMo0mWJxJQjHUF8Aox1d32DWaLK3oMF3P3KEt5P38Hw01rw6Pf7qKUkEgZHCog9BK6qKhI1SreUfv/dHlx/Vge1lETC5EgBMQR42czc3Z+qonpEQnJ3/vHpesa9k8EpDevw6s1n0adto0iXJRLTjjTMNdfMLgOercJ6RL5l38FCfvXKEt5P366WkkgVOuIopuDw1p9WUS0i37Jo0x7GBFtK91/agxsGq6UkUlWO+XLfZpZgZteGoxiRww63lH747BcAvHLzWdx4dkeFg0gVOtIw15OA2wicqH4DeA8YA9wNLAamVEF9Eof2HSxk7KtLeHfldi7q0YK/fL8PDeuqpSRS1Y7UYnqBwEimLwi0mcYSuIDf5e6+OPylSTxavHkvY6amsW1fHr+7tAc3qqUkEjFHCohO7t4Lvrk3xE6gnbvnVEllElfcnec/28DD76TTvEEdXrn5TPq1axzpskTi2pECovDwA3cvNrP1CgcJh32HCvn1q0uYs2I7Q7u34C8/6E2jurWOvqGIhNWRAqKPme0PPjYgKThtgLv7SWGvTmLeks17GTMtja178/jtd7pzk05Ei0SNI30PIqEqC5H44u786/MNPDgr0FJ6+eYzOV0tJZGoctTLfYtUtn2HCvnNq0uZvWIbQ7s35y8/6KOWkkgUUkBIlVqauZfbpgZaSv9zSXd+eo5aSiLRSgEhVcLd+fcXG3nw7XSa1q/FSz8/k/7t1VISiWYKCAm7/XmBltI7y7dxQbfm/PUHfWhcTy0lkWingJCwWpa5j9umppG19xD3XtyNn53TiRo11FISqQ4UEBIW7s4LX27kz2+lc3L9Wrz880H0b98k0mWJyDFQQEil259XyL2vLePtZVs5v2szHvthX7WURKohBYRUquVZgZZS5h61lESqOwWEVAp35z9fbuSBt9JpUq8WL40eREoHtZREqjMFhJywnLxC7pm+jLeXbmVIsKXURC0lkWpPASEnZHnWPsZMTWPznkP8ZkQ3fn6uWkoisUIBIcfF3Zny1Sb+9NZKmtStxYujBzFALSWRmKKAkGOWk1fIvdOX8dbSrZyX3IzHftiHk+vXjnRZIlLJFBByTFZs2ceYqYvYtPsgvx7RlZvPPVUtJZEYpYCQCnF3ps7fxB/fXEnjuolM+9kgBnZUS0kkloU1IMxsBDARSAAmu/u4MsuHAK8D64Ozprv7n4LLNgA5QDFQ5O4p4axVyncgv4j7pi/jjSVbODe5GRPUUhKJC2ELCDNLAJ4CLgIygQVm9oa7ryyz6ifufmk5uznf3XeGq0Y5upVb9nPb1DQ27spl7PCu3HKeWkoi8SKcRxADgTXuvg7AzF4ELgfKBoREIXdn2vzN/OHNFTRKCrSUzuh0cqTLEpEqVCOM+24NbC41nRmcV9aZZrbEzN4xs9NKzXfgXTNbaGajy3sSMxttZqlmlpqdnV05lce5A/lF3PnSYu6bsYwzOjZh1i/OUTiIxKFwHkGE6kN4mek0oL27HzCzS4CZQJfgssHuvsXMmgPvmVmGu8/71g7dJwGTAFJSUsruX45R+tb93DYljQ27crl7WDK3DumslpJInArnEUQm0LbUdBtgS+kV3H2/ux8IPp4FJJpZ0+D0luDfO4AZBFpWEibuzovzNzHyqc84kF/E1J8NYswFXRQOInEsnEcQC4AuZtYRyAJGAdeUXsHMTgG2u7ub2UACgbXLzOoBNdw9J/h4GPCnMNYa13Lzi/ifGcuYuXgLZ3duyoSr+tKsgUYpicS7sAWEuxeZ2RhgDoFhrs+5+wozuzm4/Bng+8AtZlYEHAJGBcOiBTAjeDP7msBUd58drlrjWca2/dw6JY0NO3P51UXJ3Hp+ZxJ01CAigLnHTts+JSXFU1NTI11GteDuvJy6mftfX8FJSYk8PqofZ56qE9Ei8cbMFpb3PTN9kzoO5eYX8duZy5mxKIvBnU/mb1f1U0tJRL5FARFnVm3L4dYpC1m3M5dfDk1mzAVqKYlIaAqIOOHuvJKayf1vLKd+7USm3HQGZ3VuGumyRCSKKSDiwMGCQEtpelqgpTThqr40b1An0mWJSJRTQMS4VdtyuG1qGmuzD3Dn0C7cfkEXtZREpEIUEDEsMEop0FL6z01nMFgtJRE5BgqIGHSwoIjfzVzBa2mZnNnpZCZerZaSiBw7BUSM+Xp7DrdOSWNN9gF+cWEX7rhQLSUROT4KiBjy6sJMfjdzOfVqJ/DCjWdwdhe1lETk+CkgYsDBgiLuf30Fry7MZFCnJjw+qh/NT1JLSUROjAKimivdUrrjwi78Qi0lEakkCohq7LWFmfx25nLq1krg3zcO5JwuzSJdkojEEAVENXSooJjfv7Gcl1MzOaNjEx6/uh8t1FISkUqmgKhm1uwItJS+3nGA2y/ozC8u7ELNhHDe90lE4pUCohqZnpbJ/8wItJT+dcNAzk1WS0lEwkcBUQ0cKijmD2+s4KXUzQzs2IQn1FISkSqggIhya3Yc4LYpaazansOY8ztz51C1lESkaiggotjMRVncN2MZdRIT+NeNAzlPLSURqUIKiCiUVxhoKb24YDMDOwRGKZ3SUC0lEalaCogoszY70FLK2JbDrUNO5a6LktVSEpGIUEBEkdcXZ3Hv9GXUrlmDf94wgCFdm0e6JBGJYwqIKJBXWMwf31zBtPmbGdChMY9f3Y+WDZMiXZaIxDkFRIStyz7ArcGW0i1DTuVXaimJSJRQQETQ64uzuG/6MmrVrMHzNwzgfLWURCSKKCAiIK+wmD+9tZKpX22if/vGPHF1P1o1UktJRKKLAqKKrcs+wG1TF5G+dT83n3cqvxqWTKJaSiIShRQQVejNJVu457WlJNaswfPXD+D8bmopiUj0UkBUgbzCYh54ayVTvtrE6e0a8eQ1p6ulJCJRTwERZut35nLblDRWbt3Pz8/txN3Du6qlJCLVggIijN5csoV7py+jZoLxj5+kcGH3FpEuSUSkwhQQYZBXWMyf317Jf74MtJSeuOZ0WqulJCLVjAKikm3YmcttU9NYsWU/o8/txFi1lESkmlJAVKK3l27lN68tJaGGMfnHKQztoZaSiFRfYf1oa2YjzGyVma0xs3tCLB9iZvvMbHHwz/0V3Taa5BUWc//ry7ltahqdm9fn7TvOVjiISLUXtiMIM0sAngIuAjKBBWb2hruvLLPqJ+5+6XFuG3EbdwVaSsuz9vOzczoydng3atVUS0lEqr9wtpgGAmvcfR2Amb0IXA5U5E3+RLatMrOWbeU3ry7FDP73xylcpKMGEYkh4fyo2xrYXGo6MzivrDPNbImZvWNmpx3jtpjZaDNLNbPU7Ozsyqj7qPKLivn968u5dUoanZrX5+07zlE4iEjMCecRhIWY52Wm04D27n7AzC4BZgJdKrhtYKb7JGASQEpKSsh1KtPGXbmMmbqIZVn7+OnZHfn1CLWURCQ2hTMgMoG2pabbAFtKr+Du+0s9nmVmT5tZ04psGwnvLNvKr4MtpUk/6s+w006JdEkiImETzoBYAHQxs45AFjAKuKb0CmZ2CrDd3d3MBhJoee0C9h5t26qUX1TMw7My+OfnG+jTthFPXt2Ptk3qRqocEZEqEbaAcPciMxsDzAESgOfcfYWZ3Rxc/gzwfeAWMysCDgGj3N2BkNuGq9Yj2bTrIGOmpbE0cx83Du7IPRerpSQi8cEC78exISUlxVNTUyttf7OXb2Xsq0sB+MsP+jBcLSURiTFmttDdU0It0zepQygoKuGhWemBllKbhjx5zelqKYlI3FFAlLF590HGTE1jSeY+bhjcgXsv7q6WkojEpbgPiJmLshg/ZxVb9h6icd1aHCospmaC8cx1pzOiZ8tIlyciEjFxHRAzF2Vx7/RlHCosBmD3wQLM4H+GdVc4iEjci+veyfg5q74Jh8Pc4fnPNkSmIBGRKBLXAbFl76Fjmi8iEk/iOiBalXOXt/Lmi4jEk7gOiLHDu5KUmPBf85ISExg7vGuEKhIRiR5xfZJ6ZL/ABWIPj2Jq1SiJscO7fjNfRCSexXVAQCAkFAgiIt8W1y0mEREpnwJCRERCUkCIiEhICggREQlJASEiIiHF1P0gzCwb2HicmzcFdlZiOSKl6fUl4XQir6/27t4s1IKYCogTYWap5d00Q+RE6fUl4RSu15daTCIiEpICQkREQlJA/L9JkS5AYppeXxJOYXl96RyEiIiEpCMIEREJSQEhIiIhxX1AmNlzZrbDzJZHuhaJPWbW1sw+NLN0M1thZr+IdE0SO8ysjpnNN7MlwdfXHyt1//F+DsLMzgUOAP92956Rrkdii5m1BFq6e5qZNQAWAiPdfWWES5MYYGYG1HP3A2aWCHwK/MLdv6yM/cf9EYS7zwN2R7oOiU3uvtXd04KPc4B0QDcgkUrhAQeCk4nBP5X2qT/uA0KkqphZB6Af8FWES5EYYmYJZrYY2AG85+6V9vpSQIhUATOrD7wG3Onu+yNdj8QOdy92975AG2CgmVVaq1wBIRJmwd7wa8AUd58e6XokNrn7XuAjYERl7VMBIRJGwZOI/wDS3f2xSNcjscXMmplZo+DjJGAokFFZ+4/7gDCzacAXQFczyzSzmyJdk8SUwcCPgAvMbHHwzyWRLkpiRkvgQzNbCiwgcA7ircraedwPcxURkdDi/ghCRERCU0CIiEhICggREQlJASEiIiEpIEREJCQFhEQlMysODgldEbxS5V1mFtWv1+CY9K/MbJGZnVNm2Udmtir4s3xmZl2PsJ8Okbq6sJldZmb3HGWd683syaqqSSKnZqQLECnHoeDlAzCz5sBUoCHw+xPdsZkluHvxie4nhAuBDHf/STnLr3X3VDMbDYwHLgtDDSfE3d8A3oh0HRIdovoTmQiAu+8ARgNjLCDBzMab2QIzW2pmPwcwsxpm9nTwqOMtM5tlZt8PLttgZveb2afAD8xsmJl9YWZpZvZK8FpJmFl/M/vYzBaa2Zzg5br/i5m1N7O5weeea2btzKwv8ChwSfDIJ+kIP9I8oHPwZxlvZsvNbJmZXRXiuT4J7vvw9Gdm1tvM/hC8l8lHZrbOzO4otc5dwX0uN7M7g/M6mFmGmU0Ozp9iZkOD+/vazAYG1/vm6MDMvlvqiOh9M2txLP9uUv0pIKRacPd1BF6vzYGbgH3uPgAYAPzMzDoCVwAdgF7AT4Ezy+wmz93PBt4HfgsMdffTgVTgruA1k54Avu/u/YHngAdDlPMkgfuH9AamAI+7+2LgfuAld+/r7oeO8ON8F1gWrLcv0IfAJRLGhwikycD1AGaWDNR296XBZd2A4cBA4Pdmlmhm/YEbgDOAQcHfTb/g+p2BiUDv4LbXAGcDdwP3hajzU2CQu/cDXgR+fYSfSWKQWkxSnVjw72FA78NHBwRaT10IvNm94u4lwDYz+7DM9i8F/x4E9AA+C1wqiVoEL7cC9ATeC85PALaGqONMAm/uAC8QOHKoiClmdgjYANwO3AVMC7a7tpvZxwQCb2mpbV4BfmdmY4EbgX+WWva2u+cD+Wa2A2gR/B3McPdcADObDpxDoG203t2XBeevAOa6u5vZMgLBWlYb4KVgaNUC1lfw55QYoYCQasHMOgHFBK55b8Dt7j6nzDrfOcpucg+vSuCaNVeX2b4XsMLdyx55HE1Fr1dzrbunlno+O9LKAO5+0MzeAy4HfgiklFqcX+pxMYH/z0faZ+n1S0pNlxD6veAJ4DF3f8PMhgB/OFq9ElvUYpKoZ2bNgGeAJz1w8bA5wC3BlhBmlmxm9Qi0RK4MnotoAQwpZ5dfAoPNrHNw+7rB9s0qoJmZnRmcn2hmp4XY/nNgVPDxtcHnPR7zgKuC51SaAecC80OsNxl4HFjg7ke7++E8YGTwZ6oHfA/45DjrawhkBR+Xd+JdYpiOICRaJVngLlmJQBGBVs7hy2VPJtASSQt+Cs8GRhK458KFwHJgNYE7t+0ru2N3zzaz64FpZlY7OPu37r462LZ63MwaEvj/8TdgRZld3AE8F2z7ZBPo+R+PGQTaVUsIHIX82t23WeDOc6XrXWhm+4Hnj7bD4L2v/8n/B81kd19Udp8V9AfgFTPLIhCqHY9jH1KN6WquElPMrH7wBu4nE3iTHOzu2yJd14kws1YEbgTTLXh+RaRK6AhCYs1bFriBSi3ggRgIhx8TGEl1l8JBqpqOIEREJCSdpBYRkZAUECIiEpICQkREQlJAiIhISAoIEREJ6f8A47VCic0I/8oAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "scores = []\n",
    "ais = []\n",
    "for i in range(1,4):\n",
    "    model = Pipeline([('poly', PolynomialFeatures(degree=i)),\n",
    "    ('linear', LinearRegression(fit_intercept=False))])\n",
    "    # fit to an order-3 polynomial data\n",
    "    model = model.fit(X_train,y_train)\n",
    "    model.named_steps['linear'].coef_\n",
    "    print('degree of polynomial = {} and score is {}'.format(i,model.score(X_test, y_test)))\n",
    "    scores.append(model.score(X_test,y_test))\n",
    "    ais.append(i)\n",
    "plt.plot(ais,scores)\n",
    "plt.scatter(ais,scores)\n",
    "plt.ylabel('R^2 score')\n",
    "plt.xlabel('Degree of Polynomial')\n",
    "plt.xticks([1,2,3])\n",
    "plt.yticks()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Παρατηρούμε ότι βέλτιστος βαθμός πολυωνύμου είναι $n=3$ με $R^2 = 72.1 \\%$.  \n",
    "Αυτό είναι ένα πολύ καλό αποτέλεσμα αν λάβουμε υπόψη τη διαστατικότητα των δεδομένων μας (5 μεταβλητές) για ένα τόσο απλό μοντέλο."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Main Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Οι Regressors με τους οποίους θα ασχοληθούμε είναι οι εξής:\n",
    "- Linear Regressor (LR)\n",
    "- Ridge Regressor (RD)\n",
    "- Lasso Regressor (LAS)\n",
    "-  Decision Trees Regressor (DT)\n",
    "-  Multi Layer Perceptron Regressor (MLP)\n",
    "-  Multi Layer Perceptron Regressor (but with hidden layers) (DMLP)\n",
    "-  Support Vector Regressor (SVR)\n",
    "-  Extra Trees Regressor (ET)\n",
    "-  XGBoost Regressor (XGB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:38:33.856411Z",
     "iopub.status.busy": "2021-03-14T20:38:33.856411Z",
     "iopub.status.idle": "2021-03-14T20:38:33.865516Z",
     "shell.execute_reply": "2021-03-14T20:38:33.864516Z",
     "shell.execute_reply.started": "2021-03-14T20:38:33.856411Z"
    }
   },
   "outputs": [],
   "source": [
    "#instantiate all random components with team number as seed - reproducible results\n",
    "models = []\n",
    "models.append(('LR', LinearRegression()))\n",
    "models.append(('RD', Ridge()))\n",
    "models.append(('LAS', Lasso()))\n",
    "models.append(('DT', DecisionTreeRegressor(random_state=75)))\n",
    "models.append(('MLP', MLPRegressor(random_state=75)))#Default Hidden Layers\n",
    "models.append(('DMLP', MLPRegressor(random_state=75, hidden_layer_sizes=(100,100))))#With 1 extra hidden layer, default is (100,). (Deep Multi Layer Perceptron).\n",
    "models.append(('SVR',SVR()))\n",
    "models.append(('ET',ExtraTreesRegressor(random_state=75)))\n",
    "models.append(('XGB', XGBRegressor(random_state=75)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T21:38:21.685414Z",
     "iopub.status.busy": "2021-03-14T21:38:21.685414Z",
     "iopub.status.idle": "2021-03-14T21:38:21.699515Z",
     "shell.execute_reply": "2021-03-14T21:38:21.698741Z",
     "shell.execute_reply.started": "2021-03-14T21:38:21.685414Z"
    }
   },
   "outputs": [],
   "source": [
    "#helper function to plot models in train and test sets for easier comparison\n",
    "def compare_models(trainX,trainY,testX,testY,models):\n",
    "    print('\\nCompare Multiple Regressors:')\n",
    "    print('\\nK-Fold Cross-Validation Accuracy:\\n')\n",
    "    \n",
    "    names = []\n",
    "    tempdf = pd.DataFrame(columns=['Name','Dataset','Score','St.Dev'])\n",
    "    for name, model in models:\n",
    "        model.fit(trainX,trainY)\n",
    "        kfold = KFold(n_splits=2,shuffle=True,random_state = 75)\n",
    "        train_accuracy_results = cross_val_score(model, trainX,trainY, cv=kfold)\n",
    "        test_accuracy_results = cross_val_score(model,testX,testY,cv=kfold)\n",
    "        tempdf = tempdf.append({'Name':name,'Dataset':'Train','Score':train_accuracy_results.mean(),\n",
    "                               'St.Dev':train_accuracy_results.std()},ignore_index=True)\n",
    "        tempdf = tempdf.append({'Name':name,'Dataset':'Test','Score':test_accuracy_results.mean(),\n",
    "                               'St.Dev':test_accuracy_results.std()},ignore_index=True)\n",
    "    g = sns.catplot(x = 'Name', y='Score',hue = 'Dataset',data=tempdf, kind='bar',height=7.5, aspect=11.7/7.5)\n",
    "    g.set_xticklabels(rotation=30)\n",
    "    return tempdf,plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T21:38:23.110839Z",
     "iopub.status.busy": "2021-03-14T21:38:23.109471Z",
     "iopub.status.idle": "2021-03-14T21:38:28.416637Z",
     "shell.execute_reply": "2021-03-14T21:38:28.415273Z",
     "shell.execute_reply.started": "2021-03-14T21:38:23.110839Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Compare Multiple Regressors:\n",
      "\n",
      "K-Fold Cross-Validation Accuracy:\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA5EAAAIsCAYAAABr8S2gAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA8cUlEQVR4nO3dfdzX8/3//3snR2dkJZXTNScjRpg53YbFikgkRJOTZURrMjnNnC0nDRmzOW0MhYxVm4WxfDZ12QkzZ5WPs9DoRHSeOo7j/fvDz/FZX8aLjvfeHble//J6v96934/j2eHouL1fr/f71ahUKpUCAAAABTSu9AAAAAA0HCISAACAwkQkAAAAhYlIAAAAChORAAAAFCYiAQAAKKxppQdYFW+/vSi1ta5QAgAArLr27VtXeoQGwZFIAAAAChORAAAAFCYiAQAAKExEAgAAUJiIBAAAoDARCQAAQGEiEgAAgMJEJAAAAIWJSAAAAAoTkQAAABQmIgEAAChMRAIAAFCYiAQAAKAwEQkAAEBhIhIAAIDCRCQAAACFiUgAAAAKE5EAAAAUJiIBAAAoTEQCAABQmIgEAACgsKaVHgAAgM+P1uu0SIvmVZUeo86y91Zk4YJllR4DGhQRCQDAf02L5lU56ow7Kz1GndEj+mVhRCR8Gk5nBQAAoDARCQAAQGEiEgAAgMJEJAAAAIX5YB0AAFhD+PRb/htEJAAArCF8+i3/DU5nBQAAoDARCQAAQGEiEgAAgMJEJAAAAIWJSAAAAAoTkQAAABQmIgEAAChMRAIAAFCYiAQAAKAwEQkAAEBhIhIAAIDCRCQAAACFiUgAAAAKE5EAAAAUJiIBAAAoTEQCAABQmIgEAACgMBEJAABAYSISAACAwkQkAAAAhYlIAAAAChORAAAAFCYiAQAAKExEAgAAUJiIBAAAoDARCQAAQGEiEgAAgMJEJAAAAIWJSAAAAAoTkQAAABQmIgEAAChMRAIAAFCYiAQAAKAwEQkAAEBhIhIAAIDCyhqREyZMSI8ePdKtW7fceeedH9r/3HPP5dBDD81BBx2UE088MQsWLCjnOAAAAKyiskXkrFmzMnLkyIwePTq/+c1vcvfdd+fFF19c6T7Dhw/P4MGDM378+Gy66aa55ZZbyjUOAAAA9aBsETl58uTstttuadOmTVq1apXu3btn4sSJK92ntrY2ixcvTpIsXbo0LVq0KNc4AAAA1IOm5Xrg2bNnp3379nXbHTp0yNNPP73Sfc4666wcf/zxueSSS9KyZcvcc889n+o52rVbu15mBQDg86t9+9aVHmGNZn3XPGWLyNra2jRq1Khuu1QqrbS9bNmynHvuubn11lvTpUuX/PKXv8yZZ56ZG2+8sfBzvP32otTWlup1bgAAymd1DIo5cxZWeoR6Y31Xzeq4fqujskXk+uuvn7///e9123PmzEmHDh3qtl944YU0b948Xbp0SZIcccQR+elPf1qucQAA4ENqq1esNuFQvfy9vDN/eaXHgE9UtojcY489cu2112bevHlp2bJlHnrooVx88cV1+zt16pS33norL7/8cjbbbLM88sgj2W677co1DgAAfEjjplV5YsSASo+RJNnpjJuTiEhWf2WLyI4dO2bIkCHp379/VqxYkT59+qRLly454YQTMnjw4Gy33Xa59NJLc+qpp6ZUKqVdu3a55JJLyjUOAAAA9aBsEZkkPXv2TM+ePVe67aabbqr777322it77bVXOUcAAACgHpXtEh8AAACsecp6JBIAAPj88sFFayYRCQAAlIUPLlozOZ0VAACAwkQkAAAAhYlIAAAAChORAAAAFCYiAQAAKExEAgAAUJiIBAAAoDARCQAAQGEiEgAAgMJEJAAAAIWJSAAAAAoTkQAAABQmIgEAAChMRAIAAFCYiAQAAKAwEQkAAEBhIhIAAIDCRCQAAACFiUgAAAAKE5EAAAAUJiIBAAAoTEQCAABQmIgEAACgMBEJAABAYSISAACAwkQkAAAAhYlIAAAAChORAAAAFCYiAQAAKKxppQcAqITW67RIi+ZVlR4jSbLsvRVZuGBZpccAAChERAKfSy2aV+WoM+6s9BhJktEj+mVhRCQA0DCISACAf7M6namQOFsBWP2ISACAf7M6namQOFsBWP34YB0AAAAKE5EAAAAUJiIBAAAoTEQCAABQmIgEAACgMBEJAABAYSISAACAwkQkAAAAhYlIAAAAChORAAAAFCYiAQAAKExEAgAAUJiIBAAAoDARCQAAQGEiEgAAgMJEJAAAAIWJSAAAAAoTkQAAABQmIgEAAChMRAIAAFCYiAQAAKAwEQkAAEBhIhIAAIDCRCQAAACFiUgAAAAKE5EAAAAUJiIBAAAoTEQCAABQmIgEAACgMBEJAABAYSISAACAwkQkAAAAhYlIAAAAChORAAAAFCYiAQAAKKxppQcAAD6d1uu0SIvmVZUeI0my7L0VWbhgWaXHAOC/SEQCQAPTonlVjjrjzkqPkSQZPaJfFkZEAnyeOJ0VAACAwkQkAAAAhTmdFQBgNVZbvSLt27eu9BhJkurl7+Wd+csrPQZQYSISAGA11rhpVZ4YMaDSYyRJdjrj5iQiEj7vnM4KAABAYSISAACAwkQkAAAAhYlIAAAAChORAAAAFCYiAQAAKExEAgAAUJiIBAAAoDARCQAAQGEiEgAAgMJEJAAAAIWJSAAAAAoTkQAAABQmIgEAACisrBE5YcKE9OjRI926dcudd975of0vv/xyjj766Bx00EH57ne/m/nz55dzHAAAAFZR2SJy1qxZGTlyZEaPHp3f/OY3ufvuu/Piiy/W7S+VShk4cGBOOOGEjB8/PltvvXVuvPHGco0DAABAPShbRE6ePDm77bZb2rRpk1atWqV79+6ZOHFi3f7nnnsurVq1yp577pkkOemkk9KvX79yjQMAAEA9aFquB549e3bat29ft92hQ4c8/fTTdduvvfZa1ltvvZxzzjmZOnVqNttss5x33nmf6jnatVu73uYFqKT27VtXegT4zHz/fr74+y4v61te1rd+lC0ia2tr06hRo7rtUqm00nZ1dXX++te/5o477sh2222Xq6++Opdddlkuu+yyws/x9tuLUltbqte5gc+H1e0fkTlzFlZ6BBoQ37/ltbqt7+pmVf++re/Hs77l9Unra/2KKdvprOuvv37mzJlTtz1nzpx06NChbrt9+/bp1KlTtttuuyTJgQceuNKRSgAAAFY/ZYvIPfbYI1OmTMm8efOydOnSPPTQQ3Xvf0ySHXfcMfPmzcu0adOSJI8++mi+8pWvlGscAAAA6kHZTmft2LFjhgwZkv79+2fFihXp06dPunTpkhNOOCGDBw/Odtttl+uuuy7Dhg3L0qVLs/7662fEiBHlGgcAAIB6ULaITJKePXumZ8+eK91200031f339ttvn3vvvbecIwAAZVRbvWK1eg9R9fL38s785ZUeA2CNVtaIBADWbI2bVuWJEQMqPUadnc64OYmIBCinsr0nEgAAgDWPiAQAAKAwEQkAAEBhIhIAAIDCRCQAAACFiUgAAAAKE5EAAAAU5jqRANS71uu0SIvmVZUeI0my7L0VWbhgWaXHAIA1hogEoN61aF6Vo864s9JjJElGj+iXhRGRAFBfnM4KAABAYSISAACAwkQkAAAAhYlIAAAACvPBOgAVVlu9Iu3bt670GHWql7+Xd+Yvr/QYAMBqSkQCVFjjplV5YsSASo9RZ6czbk4iIgGAj+Z0VgAAAAoTkQAAABQmIgEAAChMRAIAAFCYiAQAAKAwEQkAAEBhIhIAAIDCRCQAAACFiUgAAAAKE5EAAAAUJiIBAAAoTEQCAABQmIgEAACgMBEJAABAYSISAACAwkQkAAAAhTWt9ADl1HqdFmnRvKrSYyRJlr23IgsXLKv0GAAAAKtkjY7IFs2rctQZd1Z6jCTJ6BH9sjBrVkSKdAAA+PxZoyOS8hLpAADw+eM9kQAAABQmIgEAAChMRAIAAFCYiAQAAKAwEQkAAEBhIhIAAIDCRCQAAACFiUgAAAAKE5EAAAAUJiIBAAAoTEQCAABQmIgEAACgMBEJAABAYSISAACAwkQkAAAAhYlIAAAAChORAAAAFCYiAQAAKKxppQf4vKitXpH27VtXeowkSfXy9/LO/OWVHgMAAGiAROR/SeOmVXlixIBKj5Ek2emMm5OISAAA4NNzOisAAACFiUgAAAAKE5EAAAAU5j2RrBF8cBEAAPx3iEjWCD64CAAA/jsKnc66ePHiXHjhhTnmmGPy7rvv5kc/+lEWL15c7tkAAABYzRSKyB//+MdZZ5118vbbb6d58+ZZtGhRfvSjH5V7NgAAAFYzhSJy6tSpGTJkSJo2bZqWLVvmiiuuyNSpU8s9GwAAAKuZQhHZuPHKd6upqfnQbQAAAKz5Cn2wzs4775yf/OQnWbZsWf70pz/lzjvvzK677lru2QAAAFjNFDqcePrpp6dVq1Zp3bp1Ro4cma222ipnnHFGuWcDAABgNVPoSOQ111yTH/7whznllFPKPQ8AAACrsUJHIidNmlTmMQAAAGgICh2J3HjjjXP88cfnq1/9atZaa62624877riyDQYA9aG2ekXat29d6THqVC9/L+/MX17pMQDgMysUkW3atEmSzJw5s5yzAEC9a9y0Kk+MGFDpMersdMbNSUQkAA1XoYi89NJLk7wfkdXV1enUqVNZhwIAAGD1VCgiZ8yYkZNPPjmzZ89ObW1t2rZtmxtuuCGbb755uecDAABgNVLog3UuuuiiDBgwIH/729/yxBNPZODAgbnwwgvLPRsAAACrmUIR+fbbb+eQQw6p2z700EPzzjvvlG0oAAAAVk+FIrKmpibvvvtu3fa8efPKNQ8AAACrsULvifzOd76TI444Ivvvv38aNWqUBx54IMccc0y5ZwMAAGA1UygijzjiiHTq1Cl/+tOfUltbmwsuuCC77757uWcDAABgNVPodNZZs2Zl4sSJGTp0aA477LDcfvvtmTNnTrlnAwAAaFDeeOONbL311unVq1d69eqVnj17pm/fvnnggQc+8c/+7Gc/yx/+8IeyzHX88cfX29sSCx2JPPPMM9O1a9ckyUYbbZRddtkl55xzTm666aZ6GQIAAGBN0aJFi4wbN65ue+bMmTn22GPTpEmTdO/e/T/+ub/85S/ZYostyjLT448/Xm+PVSgi33nnnfTv3z9J0rx58xx77LH5zW9+U29DAAAArKk22mijDB48OLfccku23HLLXHTRRVm8eHHmzJmTzp075+qrr869996bZ599NiNGjEiTJk2yxRZbfOT9mjdvnmuuuSYPP/xwqqqq0rZt21x66aXp0KFDXnrppQwfPjzvvvtuampqcvTRR6dPnz45++yzkyTHHHNMbrzxxmywwQar9PUUisiamprMmjUrHTt2TJLMnTs3pVJplZ4YAADg86Jz58554YUXcs899+Tggw9Or169smLFivTu3TuTJk1Kv379MnHixPTr1y/f/va3c/nll3/k/bp06ZLbbrstU6ZMSbNmzTJq1Kg8/fTT2XvvvTN48OCMGDEiX/nKV7Jw4cIcccQR2WKLLXLppZfmvvvuy2233ZZ11113lb+WQhF57LHH5uCDD843v/nNJMmUKVNyxhlnrPKTAwAAfB40atQoLVq0yNChQ/P444/npptuyquvvprZs2dnyZIlH7r/f7pfx44d07lz5xxyyCHZc889s+eee2b33XfPiy++mNdeey3nnHNO3WMsW7Yszz//fHbYYYd6/Vo+MSJLpVIOPvjgbLvttvnDH/6Qxo0b57vf/W622mqreh0EAABgTfXMM89kyy23zGmnnZaamprsv//+2XvvvfPmm29+5Fme/+l+jRs3zh133JFnnnkmU6ZMySWXXJJvfvOb6dWrV1q3br3SezHnzp2b1q1b1/vX8rGfzvriiy9mn332yZ/+9Kd86Utfym9/+9tMmDAhAwYMqNc3ZgIAAKypXnnllfz85z/P8ccfnz//+c855ZRT0qNHjyTJP//5z9TU1CRJmjRpkurq6iT5j/ebNm1aDjzwwGy++eY58cQTc+yxx+aZZ57JpptuutIH+rz55ps58MAD8+yzz37osVfVxx6JHDFiRE499dR861vfyq9//es0atQov/vd7zJr1qwMGTIkX//61+tlCAAAgDXFsmXL0qtXryRJ48aN07x585x22mnZe++9M2TIkJxyyilp1apV1l577ey888557bXXkiRdu3bNVVddlRUrVvzH+x122GHZf//9c+ihh6ZVq1Zp0aJFhg0blmbNmuXnP/95hg8fnptvvjnV1dX5wQ9+kJ122ilJst9+++Xoo4/Otddemy233HKVvr6Pjcg333wzBx10UJL3P252n332SePGjbPBBhtk0aJFq/TEAAAAa5qNN944U6dO/Y/7+/Xrl379+n3kvv79+9ddFeOD+36UQYMGZdCgQR+6vXPnzrn99ts/8s9cddVVHzf2p/Kxp7M2bvx/u//xj39k5513rtt+77336m0IAAAAGoaPPRL5hS98IdOmTcuiRYsyZ86cuoh88skn6y73AQAAwOfHx0bkaaedlmOPPTaLFi3K6aefnlatWuWWW27J9ddfn+uuu+6/NSMAAACriY+NyB122CH/8z//k2XLlmWdddZJkuy4444ZO3ZsvvSlL/035gMAAGA18onXiWzWrFmaNWtWt/3Vr361rAMBAACw+vrYD9ZZVRMmTEiPHj3SrVu33Hnnnf/xfpMmTUrXrl3LOQoAAAD14BOPRH5Ws2bNysiRI3PfffelWbNm6du3b3bddddsscUWK91v7ty5ufzyy8s1BgAAQL1ZvqImzaqaVORxL7zwwjz55JNZsWJFXnvttWy++eZJ3r80yKGHHvqJz9GrV6+MGzdulWctW0ROnjw5u+22W9q0aZMk6d69eyZOnPih65kMGzYsgwYNypVXXlmuUQAAAOpFs6omOeqM/3yW5Wc1esRHXxPy351//vlJkjfeeCP9+/f/1EFYHwGZlDEiZ8+enfbt29dtd+jQIU8//fRK9/nVr36VbbbZJttvv325xgAAAFijde3aNV26dMnUqVMzevTo/OpXv8qUKVMyf/78dOjQISNHjsx6662XrbbaKtOnT8+1116bWbNmZcaMGZk5c2YOO+ywDBw4sPDzlS0ia2tr06hRo7rtUqm00vYLL7yQhx56KLfeemveeuutz/Qc7dqtvcpzfl61b9+60iOs0awvDZ3v4fKyvuVlfcvL+paX9S2vNXl999xzz1x99dWZMWNGXn755dx1111p3LhxzjjjjIwfPz7HH3/8SvefPn167rzzzixcuDD77rtv+vXrV3dFjk9Stohcf/318/e//71ue86cOenQoUPd9sSJEzNnzpwceuihWbFiRWbPnp2jjjoqo0ePLvwcb7+9KLW1pf+4f03+JllVc+YsXOXHsL7/WX2sL+Xl+/fjrer3sPX9eNa3vKxveVnf8rK+5fVJ69uQ1++Dszs7deqUM888M2PHjs0rr7ySp556Kl/84hc/dP9dd901zZo1S7t27dKmTZssXLiwcESW7dNZ99hjj0yZMiXz5s3L0qVL89BDD2XPPfes2z948OA8+OCDGTduXG688cZ06NDhUwUkAAAA72vevHmS5Nlnn813v/vd1NbWpnv37tl3331TKn34wNsH90+SRo0afeR9/pOyRWTHjh0zZMiQ9O/fPwcffHAOPPDAdOnSJSeccEKeeeaZcj0tAADA59bf/va37LLLLjnyyCPzpS99KZMmTUpNTU29PkfZTmdNkp49e6Znz54r3XbTTTd96H4bb7xxHn300XKOAgAAsMbr0aNHBg0aVNdh2267bd544416fY6yRiQAAMCaZPmKmkKX4/gsj1v0+pP/70G4f//vjh07ZuzYsR/556ZPn54k+f73v7/S7Z/2gF7ZTmcFAABY0xQNvdXlcctBRAIAAFCYiAQAAKAwEQkAAEBhIhIAAIDCRCQAAACFiUgAAAAKc51IAACAgmqrV6Rx06qKPO6FF16YJ598MitWrMhrr72WzTffPEnSv3//HHrooZ/4HAsXLsxZZ52V6667bpVmFZEAAAAFNW5alSdGDKj3x93pjJs/8T7nn39+kuSNN95I//79M27cuE/1HPPnz8/UqVM/03z/zumsAAAADdSMGTNy3HHH5ZBDDsmRRx6Z559/PkkyYcKE9OrVK717987gwYPz3nvv5cc//nFmz56dU045ZZWeU0QCAAA0UGeeeWaGDh2a+++/PxdffHGGDBmSJLn66qszatSo3Hfffdloo43y8ssvZ9iwYenQoYPTWQEAAD6PFi9enGeffTZnn3123W1LlizJO++8k29961s58sgjs++++6Z79+7Zeuut88Ybb9TL84pIAACABqi2tjbNmjVb6b2Rb731Vtq0aZNhw4Zl2rRpeeyxxzJ06NAMGjQoO+20U708r9NZAQAAGqDWrVvnS1/6Ul1EPv744+nXr1+qq6vTrVu3tG3bNieeeGJ69eqVqVOnpmnTpqmurl7l53UkEgAAoKDa6hWFPkn1szzuZ7l0yE9+8pNccMEFufnmm1NVVZWRI0emqqoqgwcPzvHHH5/mzZunXbt2ueyyy7LOOutkww03zNFHH53bb7/9M88qIgEAAAoqxzUiP+3jbrzxxnn00UeTJJtvvvlHBuGBBx6YAw888EO333XXXZ99yP+f01kBAAAoTEQCAABQmIgEAACgMBEJAABAYSISAACAwkQkAAAAhYlIAAAAChORAAAAFCYiAQAAKExEAgAAUJiIBAAAoDARCQAAQGEiEgAAgMJEJAAAAIWJSAAAAAoTkQAAABQmIgEAAChMRAIAAFCYiAQAAKAwEQkAAEBhIhIAAIDCRCQAAACFiUgAAAAKE5EAAAAUJiIBAAAoTEQCAABQmIgEAACgMBEJAABAYSISAACAwkQkAAAAhYlIAAAAChORAAAAFCYiAQAAKExEAgAAUJiIBAAAoDARCQAAQGEiEgAAgMJEJAAAAIWJSAAAAAoTkQAAABQmIgEAAChMRAIAAFCYiAQAAKAwEQkAAEBhIhIAAIDCRCQAAACFiUgAAAAKE5EAAAAUJiIBAAAoTEQCAABQmIgEAACgMBEJAABAYSISAACAwkQkAAAAhYlIAAAAChORAAAAFCYiAQAAKExEAgAAUJiIBAAAoDARCQAAQGEiEgAAgMJEJAAAAIWJSAAAAAoTkQAAABQmIgEAAChMRAIAAFCYiAQAAKAwEQkAAEBhIhIAAIDCRCQAAACFiUgAAAAKE5EAAAAUJiIBAAAoTEQCAABQmIgEAACgsLJG5IQJE9KjR49069Ytd95554f2/+EPf0ivXr1y0EEH5eSTT878+fPLOQ4AAACrqGwROWvWrIwcOTKjR4/Ob37zm9x999158cUX6/YvWrQoF1xwQW688caMHz8+W221Va699tpyjQMAAEA9KFtETp48ObvttlvatGmTVq1apXv37pk4cWLd/hUrVuT8889Px44dkyRbbbVV3nzzzXKNAwAAQD1oWq4Hnj17dtq3b1+33aFDhzz99NN1223bts23v/3tJMmyZcty44035uijj/5Uz9Gu3dr1M+znUPv2rSs9whrN+tLQ+R4uL+tbXta3vKxveVnf8rK+9aNsEVlbW5tGjRrVbZdKpZW2P7Bw4cKccsop6dy5cw455JBP9Rxvv70otbWl/7jfN8l/NmfOwlV+DOv7n9XH+lJevn8/3qp+D1vfj2d9y8v6lpf1LS/rW16ftL7Wr5iync66/vrrZ86cOXXbc+bMSYcOHVa6z+zZs3PUUUdlq622yvDhw8s1CgAAAPWkbBG5xx57ZMqUKZk3b16WLl2ahx56KHvuuWfd/pqampx00knZf//9c+65537kUUoAAABWL2U7nbVjx44ZMmRI+vfvnxUrVqRPnz7p0qVLTjjhhAwePDhvvfVWnn/++dTU1OTBBx9Mkmy77baOSAIAAKzGyhaRSdKzZ8/07NlzpdtuuummJMl2222XadOmlfPpAQAAqGdlO50VAACANY+IBAAAoDARCQAAQGEiEgAAgMJEJAAAAIWJSAAAAAoTkQAAABQmIgEAAChMRAIAAFCYiAQAAKAwEQkAAEBhIhIAAIDCRCQAAACFiUgAAAAKE5EAAAAUJiIBAAAoTEQCAABQmIgEAACgMBEJAABAYSISAACAwkQkAAAAhYlIAAAAChORAAAAFCYiAQAAKExEAgAAUJiIBAAAoDARCQAAQGEiEgAAgMJEJAAAAIWJSAAAAAoTkQAAABQmIgEAAChMRAIAAFCYiAQAAKAwEQkAAEBhIhIAAIDCRCQAAACFiUgAAAAKE5EAAAAUJiIBAAAoTEQCAABQmIgEAACgMBEJAABAYSISAACAwkQkAAAAhYlIAAAAChORAAAAFCYiAQAAKExEAgAAUJiIBAAAoDARCQAAQGEiEgAAgMJEJAAAAIWJSAAAAAoTkQAAABQmIgEAAChMRAIAAFCYiAQAAKAwEQkAAEBhIhIAAIDCRCQAAACFiUgAAAAKE5EAAAAUJiIBAAAoTEQCAABQmIgEAACgMBEJAABAYSISAACAwkQkAAAAhYlIAAAAChORAAAAFCYiAQAAKExEAgAAUJiIBAAAoDARCQAAQGEiEgAAgMJEJAAAAIWJSAAAAAoTkQAAABQmIgEAAChMRAIAAFCYiAQAAKAwEQkAAEBhIhIAAIDCRCQAAACFiUgAAAAKE5EAAAAUJiIBAAAoTEQCAABQmIgEAACgMBEJAABAYWWNyAkTJqRHjx7p1q1b7rzzzg/tnzp1anr37p3u3bvn3HPPTXV1dTnHAQAAYBWVLSJnzZqVkSNHZvTo0fnNb36Tu+++Oy+++OJK9xk6dGh+9KMf5cEHH0ypVMo999xTrnEAAACoB2WLyMmTJ2e33XZLmzZt0qpVq3Tv3j0TJ06s2z9z5swsW7YsO+ywQ5Kkd+/eK+0HAABg9dOoVCqVyvHAN9xwQ5YsWZIhQ4YkScaOHZunn346F198cZLkH//4R0aMGJExY8YkSWbMmJHvfe97efDBB+tthuUratKsqkm9Pd6qqFmxPE2qmlV6jCRJbfWKNG5atcqPY30/mvUtL+tbfvWxxtb3P7O+5bWmrW+yeq2x9S0v61te9fU7BEnTcj1wbW1tGjVqVLddKpVW2v6k/UW8/fai1NaWpYHL5L1KD/BvllV6gDJYs9a3ffvWOeqMD7+XuBJGj+iXJ0YMqPQYSZKdzrg5c+YsrPQYZbA6ff8ma97PCOtbXta3/FanNba+5WV9y+vj17d9+9b/pTkatrJF5Prrr5+///3vddtz5sxJhw4dVto/Z86cuu25c+eutB9YfdRWr8hOZ9xc6TGSJNXLV6d/iAAAPn/KFpF77LFHrr322sybNy8tW7bMQw89VHcqa5JstNFGad68eZ544onstNNOGTduXPbcc89yjQMNzrL3VmT0iH6VHiPJ+7MsfGdNPPoHAMCnVbb3RCbvX+LjhhtuyIoVK9KnT5+ccMIJOeGEEzJ48OBst912mTZtWoYNG5ZFixblK1/5Si699NI0a1b8nOmGdzorAACwunI6azFljchyE5EAAEB9EZHFlO0SHwAAAKx5RCQAAACFiUgAAAAKE5EAAAAUJiIBAAAoTEQCAABQmIgEAACgMBEJAABAYSISAACAwkQkAAAAhYlIAAAAChORAAAAFCYiAQAAKExEAgAAUJiIBAAAoDARCQAAQGEiEgAAgMJEJAAAAIU1rfQAq6Jx40aVHgEAAOBzpVGpVCpVeggAAAAaBqezAgAAUJiIBAAAoDARCQAAQGEiEgAAgMJEJAAAAIWJSAAAAAoTkQAAABQmIgEAAChMRAIAAFCYiKyAUqlU6REAAPg3CxYsqPQI0GCIyP+SpUuX5oknnsiSJUvSqFGjSo+zxlqyZEmlR4DP7PXXX8/TTz9d6TGgrLyQyupm6dKlufzyy3PJJZdk0aJFlR5njVdTU1PpEagHTSs9wOfBXXfdlRtvvDEbbrhhmjRpkptvvjlVVVWVHmuNsnTp0lx55ZX517/+lY022ii77757unbtWumx1lgTJkxIp06d0rZt22yyySYplUpeHFkFS5cuzTXXXJPHH388Z599dqXHWeO9++67ue222/LlL385u+66a9q1a+d7uIxmzZqV0aNHZ5tttsnXv/71rL322pUeqUF7991384tf/CKbb7559tprr3Ts2DE1NTVp0qRJpUdrkMaMGZNRo0Zl3333zXnnnZe11lqr0iOtsd59991cc801+cIXvpC99torO+ywQ6VHYhU0KnlJsGz+9Kc/5bbbbktVVVWGDBmSzTffPF/5yldy2223Zdddd630eGuM22+/PQ888EC22267HHjggfnjH/+Y0aNH58orr8xuu+2Wpk29VlJfJk2alJ///Odp27Zt2rRpk6qqqlx44YV+eVkFY8aMyXXXXZd1110348ePr/Q4a7xbb70148ePT+fOnbNw4cK8+uqrmTBhQqXHWmP94he/yMMPP5xtt902M2fOzNKlSzN69OhKj9Vg3XrrrRk3blw6d+6cBQsW5O23385dd91V6bEapOeffz4XXXRRamtrM3PmzDz22GNp2rRpamtr07ixE/Xq2y233JIHH3wwX/va17LWWmvlt7/9bSZMmOB3tAbM31yZTJs2LSeccEIuueSS9O7dO8n7p6odddRR2WKLLSo83Zqhuro648ePz/Dhw/PYY4+lY8eOSZIuXbpkyZIlGTt2bDbccMNsttlmFZ50zfD666/ntttuy8knn5y99947K1asyLx58+oC0pGcT+edd97Jeeedl1KplB/84Af55S9/mUWLFmXttdd2VKEMXnrppQwaNCjbbrttrrnmmmy88cZJkp133jl/+MMfsu+++1Z4wjXLM888k9NOOy277bZbbrzxxqy33npZunRpdt111/z973/P1772tUqP2KA8+eSTOeecc7LjjjvmqquuyqabbpqJEyfm97//fZYsWZKWLVv6+VvQ66+/nl/+8pfp3Llzvve976Vr164588wzM3z48Jx//vkCsp4tX748V199dcaMGZNHHnkk6667bpLkf/7nf/L6669n0003TRLx3gCJyDIolUrp3LlzDjjggLz55ptJkssuuyz3339/Ntpoo7zzzjtp165dkvhl8TN4/fXXc+2116Zt27bp27dvvvrVr2b69Onp2LFjlixZklatWuXEE0/MgAED8uKLL2azzTYTOPXg2WefzWabbZa99947NTU1eeyxx7JkyZJUV1fn61//el3E8/FmzpyZ559/Pl/60pdywgknZPvtt0/y/i+JF110UUaMGOEf0jJo27Zt5s2bl+9+97t1AZkk3/jGN7y9oAzWWWedLFiwIAMHDsx6662XJGnZsmW++c1vVniyhmXu3Ll57rnn8r//+79p06ZNLr300iTJq6++moceeii77757WrVqVeEpG4aampr87Gc/y2OPPZauXbvm8MMPr9t3yimn5Nhjj83TTz+dLl26VHDKNccHL4o2a9Yse++9d6ZPn55GjRrlrbfeytChQzN37txcfPHF2XbbbTNw4MC0bNmy0iPzKTmdtZ5Mnjw5y5YtS9euXevC8KWXXsqxxx6bli1bplu3bvnyl7+cZ555JpMnT862226b8847L61atRKRBX3wD8CkSZPy7W9/O88//3wWLFiQPn365Kabbqo7JW3FihWpqqrKiBEjMmPGjFx33XUVnrxhmjZtWjbbbLM0a9YsSfL000/nzDPPTKdOnfLss89m6623zjvvvJOampo0btw4v/71rys88ertg+/f3//+99l2220zY8aMbLrpprngggvSqlWrvP766zn++OMzYsSI7Ljjjl5gqkcfrOX111+fv/71rxk1alSWLVuWkSNH5rnnnsuVV17pRZB69MERhcsvvzwzZ87MNddckwULFuSKK67IzJkzc8UVV6Rt27aVHrNBGDZsWLbYYovss88+ue2229KhQ4fMnz8/EydOzNtvv52DDjooe++9d77xjW+kWbNmjub8B5MmTcqwYcPSq1evHHPMMenQoUOS99+v265duzRt2jRXXXVVXnjhhfziF7/wovMqWLJkSa688sq88sor6dixY/bdd9/ss88++dnPfpZf//rXWW+99XLIIYfkqKOOyvjx4zNx4sR861vfymGHHVbp0fmU/KSpB7W1tfnf//3fjBo1KitWrEiTJk1SW1ubzTffPEcccUTWXnvtnH766enVq1eGDRuW8847LzNnzkyfPn3y0EMPVXr8BmHSpEnZa6+98t577+X666/PySefnFNOOSVNmjTJN7/5zbRv3z6/+MUvkrz/95Ek6667bt2pw14r+XRuvPHGHHrooRk5cmTdbV26dMmIESPyjW98IxdddFGGDBmS66+/Pvfff39mzZrlU0U/xgffv7W1tbnnnntyxRVX5Lrrrss///nP/OpXv8r8+fOzySabpHfv3rnqqquSRECugnvvvbfuRaXq6uq6X6pPOumkzJ07N0OHDs3RRx+d2tra3HDDDQJyFd1+++155JFHkry/3h/8Aj5kyJBMnz49Z555Zo4//viss846uf766wVkAdXV1UmSrbfeOn/+85+zySabpEuXLhkzZkzeeuutPPLII7n//vuz2Wab5fzzz89JJ52Uv/zlL3V/jpUtXbo07733XoYOHZoOHTrkb3/7W4444ojce++9db8znHrqqXnjjTcyceLECk/bcE2aNCmHH354GjdunHPPPTfbb799XnjhhSTJ/vvvn44dO6Zv37456qijkiQHHXRQmjRp4sOMGqoS9WLOnDml008/vXTzzTeXSqVSqaamplQqlUpvv/12qWfPnqXf/e53pVKpVKquri6VSqXSkiVLSk8++WRlhm2AHnjggdLXvva1uu2//vWvpYEDB5auu+66UqlUKk2ZMqX0rW99q7R48eJSqVQqPfvss6V+/fqVHnnkkYrM29CNGzeudPHFF5f69+9fOv3000vPPfdc3b4lS5asdN/p06eXzjrrrNLSpUv/22M2GA888EBp5513rttetGhRqVQqlX73u9+V+vfvX5o2bVqpVCqVFi5cWDrssMNKd999d6lU+r+fIxQzderU0llnnVX68Y9/XNpuu+3qbq+tra372Ttp0qTSLrvsUvrb3/5Wt/+DfXw6//jHP0qlUqk0ZsyY0le/+tW62/99vX/729+Wdtttt9L06dPr9lvvj/biiy+Wxo0bVyqV/u///X/+85+lc889t7RgwYLSzJkzS5deemnpF7/4RalUen+dS6VS6YUXXih9//vfL5111lkf+vnM//n+979fOuecc0oXX3xx6fDDDy9NmDChbt8H633vvfeW9tprr7qf0RQze/bsUqlUKp1//vmlP/7xjx/a/69//as0e/bs0r333lv67ne/W3f7P/7xj1Lv3r1Lf/nLX/5bo1KPHImsJ+3atcv++++fSZMmZebMmWncuHGWL1+eddddN4MGDcr111+fJHVHKVu2bJkdd9yxwlM3HPvvv3923333DBs2LBdffHHOO++8vPbaa3n88cczduzYbLLJJjnooINy+umn59JLL80FF1yQI444wmU+PqPGjRtn+vTp+elPf5qWLVvmoosuyl/+8pckyaOPPppjjjkmU6ZMyWWXXZZTTz0122yzTVq0aFHhqVdf+++/f3bbbbe69zM1b948SdKjR48sW7Ysf/7zn5Mka6+9dvr27Vv3Ka1OS/t05syZk/vvvz9bbrll1l9//fzkJz9J8v7ZCR8c2d1rr72y7bbb5h//+MeH9lHc5MmT07dv34wePTp9+/bNJptsUvfWgX8/FfuAAw5Ip06drPfHmD17dp566qk8/vjjufDCC/PII4/UXfN4+fLlmTlzZlq3bp0NN9wwO++8c1588cVMnjw5jRo1Sm1tbb785S/nmmuuyaWXXup9ZR/j5JNPzgMPPJCamprcfffdOfDAA+v2ffCz9tBDD82gQYMcGfsUHnjggVx11VWZN29e/vznP6dNmzZ1+2bMmJEf/vCH6dWrV/7617/ma1/7Wjp27Jjhw4fnggsuyGWXXZaBAwdml112qdwXwGfmN5R60qhRo+y6667p3LlzbrnlliSpey9ZkyZN8q9//SuPPfZYEr8YflYnn3xyfve736WmpiYTJ07Mr3/96xx22GEZP358jj766Lz33nt5/PHH07p164wdOzY9e/as9MirvcmTJ+fRRx9NsvLFf/fee+/Mnz8/tbW1ueiii7JixYqcffbZGTNmTA444IB06dIlv//977Nw4cLccccdOfrooyv1JTQYAwcOzMMPP5xXXnklTZs2rbug9Te/+c1MnTq17n69e/fOHXfcUakxG5RSqbTSqerNmzfPpptumtmzZ+fwww/PqFGjMnv27DRp0iTLly+vu+/QoUPzs5/9LC+99JKfx59R69at88UvfjHTpk3La6+9lssuuyw///nPs2TJkjRt2jTLly+vu+9pp52Wyy67LG+99Zb1/jc1NTW5+uqrc8wxx+Taa6/N/Pnzc9JJJ+X3v/99fvrTnyZJvva1r2Xx4sWZNGlSkmT77bfPFltskXvvvTcrVqyoW8+St2x8os6dO6dv376ZP3/+SrdPnDgxgwcPzrPPPpsk6dOnTyXGa7CmTJmSrbbaKo0aNcoGG2yw0unq1157bbbffvsMHTo0d911Vxo3bpw999wzv/3tb7Phhhvmrrvu8snYDZif5vVorbXWSs+ePfPqq69mxowZefnll3PAAQfk+uuvz6hRo7LXXntVesQG7YN/ABYsWJAkqaqqysEHH5xRo0ala9euqaqqysMPP5xBgwZVeNKG4aPey/vBLyLvvvtuvvzlL+euu+5K3759s8466+SAAw7InXfemcGDB+eHP/xhzj///AwfPrzu47r5eFtvvXW6d++eyy+/PEnqLrj+1ltvZb/99qvkaA3SH/7wh/Tu3Tvjxo2ru23HHXdMhw4dMm3atLRr1y477LBDhg0bluT9F/UaNWqUO+64I+3bt8+VV16Z9ddfv1LjNziPPfZYXn/99brttm3bplmzZqmurs6jjz6azp07Z999982ZZ56Z5P9eRB01alS+8pWv+Fnx/1i+fHkuvPDCzJ07N2PHjs0NN9yQr3/96znmmGNy4okn5pVXXsmPf/zjzJo1K926dauLxfXWWy877rhj9thjj5XC0QfBFDNgwID87//+b5566qnMmjUrAwcOzF133ZX+/ftn2223rfR4DcIDDzyQMWPGZN68eUne/11im222Sdu2bfPOO+/kt7/9bd19L7744vTv3z9du3ZN06ZNs9Zaa6Vr16555JFH8r3vfa9SXwL1RETWs6233jq77LJLunfvniOPPDKHHnpoxo4d6yOj68mAAQPywgsv5LHHHqs7ZbiqqirDhg3LaaedVveJa3yyxo0b54ADDsgGG2yQX/3qV0n+79XsjTfeOG+++WZGjRqVAQMG5Je//GV++MMf5rrrrkv37t2T+OCXz2LAgAGZOXNmnn766bz11lsZOHBgZs2ala233rrSozU41dXVmTp1akaOHJkpU6Zk7ty5qaqqSrdu3bLzzjvnySefzGGHHZbJkyfnpZdeyrRp07L//vvnV7/6Vd59993su+++Tlkr6NZbb82JJ56Y8847L0uWLEltbW023njjfPvb306TJk3ywgsv5Kmnnsrw4cMzadKkzJo1K3//+9+zzz775P7778/8+fPTo0ePurAkWbBgQV544YX8+Mc/ztprr52ZM2emWbNmeeaZZ/LFL34xV1xxRZYvX56bbropEydOrDtzIUl22WWX9OnTx3p+Bu3atUvfvn3Tt2/fDBo0KPvss09uvfVW1y0tqLa2NgsXLsyDDz6YY489NhMmTMhzzz1X9yLGqaeemjFjxuRPf/pTFi1alJYtW+aJJ57IoEGDstNOO+ULX/hCqqqqXJZmDeE6kfWsqqoqBxxwQJo1a5Z+/frVvfeJ+tGuXbsceeSRueCCC/LHP/7RP6Kr6IP38v7yl7/Mfvvtl4022qjuWpvdunXLhhtuWHeqyfLly9OpU6d06tSpwlM3XB/8AnP44YenS5cuOfzww5069Rl169Yt/fr1y6RJkzJ9+vRMmjQpZ599dmbMmJHevXunZcuWmTt3bvbbb78ccMABWXfddfP9738/Rx55ZKVHbxBmzpyZf/7zn9lmm22y3377ZcqUKXniiScyZsyYbLrppunatWvWWWed7LnnnnnooYfy8MMPZ4cddshJJ52UvfbaK+uvv35OPfXUHHzwwZX+UlZL6623Xho1apQf/OAHWbp0aRYtWpTmzZvnqaeeym677ZaBAwfmoosuyvjx4zN27Nj87W9/S48ePZI46riqPriURJ8+ffyO9ik1btw4RxxxRI444oiMHz8+zz33XKZNm5bf/e536dSpU/bdd9+6F6Bra2vTsWPHvPTSS/nOd76TQw45pNLjU89cJ5IGZ/ny5Rk7dmyOPPLINGrUyD+oq2jx4sW5+uqrU1NTkx/96Ed1t5911llZZ511cuqpp3rVsB598P3rF5hVN3369AwaNCiXXHJJ7rjjjuyyyy5ZsGBBFi5cmO9///s5/fTTs++++2bJkiXp169fpcdtEP7f65m++uqr2WabbdK9e/ecd955Of300zNmzJice+65eeSRR/KFL3wh++yzTy699NJ069YtBx54YO6777707t270l/Kau+VV17JmDFjsmzZsuyzzz5Za621suGGG+aWW27JJptskmOPPTbJ++8523333Ss7LHyExYsX57jjjkurVq2yZMmSbLnlljnppJPSvHnzzJ8/Py+++KK3a6zBHImkwfngKC/144P38l599dWZMWNGqqurM3To0CxevDhXXnmlgKxnvn/rz1ZbbZWuXbvmvvvuyyWXXJKrrroqs2fPTrNmzdKyZcscf/zx2XLLLdO6detKj9ogfHBB9kMPPTT33HNP1llnncyePTvf+c530rFjx7Rp0yYLFy7MiSeemJtuuilbbbVVnnrqqXznO9/JlltuWXeNQgFZzKabbppzzjnnQ7cvWbJkpffnCUhWV6+//nqWLl2ae+65J6+//npGjBiRAQMGZODAgenVq1fdtbpZM4lIYKX38n7hC1/IiSeemOOPP77SY8En+t73vpejjz46zz//fM4777wMHz48L7/8cmpqarLTTjtVerwGZenSpVm+fHmGDBmS5P2jDB06dMipp56aX//61+nbt29uuOGGjBs3Lk2aNMkNN9yQWbNmJUlOPPHEVFVVVXL8BunZZ5/Nww8/nJ133jnrrrturrzyyjRu3DibbLJJpUeDj1UqlVJbW5uNNtooS5cuzSabbJKRI0dmwYIFPkTrc8LprECS919RfPjhh72XlwbnjjvuyC233JI//vGPlR6lwRs8eHA22GCDnH322amurk7Tpu+/1tynT58cd9xxeemll/Laa6/liiuuqHvf5Afv1ePTW7p0aW699dY888wzmTlzZo488sj07du30mNBIR98Uuvtt99e6VGoABEJQIP27++TTlyLd1VMnTo1p5xySm655ZZsuummWbRoUdZee+1cc801+de//pWTTjopAwYMyG233ZaNNtqo0uOuMRYtWpQWLVrURTs0BGPGjEmjRo3St2/flEoln1HxOSMiAYA6l19+eV555ZVcf/31dbede+652XvvvfPtb3+7LiyBz7fa2lov2n2O+ZsHAOp81PVMZ8+enW222SZJBCSQxFkfn3fOmwAA6rieKQCfREQCACtxQXYAPo73RAIAAFCYk5kBAAAoTEQCAABQmIgEAACgMBEJAABAYSISgNXeG2+8ka222ipjx45d6fZbbrklZ511VoWmAoDPJxEJQIPQuHHjXH755Xn55ZcrPQoAfK65TiQADUKLFi1y3HHH5fTTT89dd92VZs2a1e175ZVXctFFF2Xx4sWZM2dOOnfunKuvvjrNmzfPdtttl+OOOy6TJ0/OkiVLMmjQoEycODEvvPBCOnTokOuvvz6tWrXKSy+9lOHDh+fdd99NTU1Njj766PTp06eCXzEArJ4ciQSgwRg4cGBatWqVkSNHrnT7Pffck4MPPjj33HNPHnroobzxxhuZNGlSkmT58uVZb731cu+99+bggw/OsGHDcu655+aBBx7IokWL8sgjj6S6ujqDBw/OD3/4w9x333254447MmrUqDz11FP//S8SAFZzjkQC0GA0btw4P/nJT3LwwQfnG9/4Rt3tQ4cOzeOPP56bbropr776ambPnp0lS5bU7e/evXuS5Itf/GK23HLLdOzYMUmy8cYbZ/78+Xn11Vfz2muv5Zxzzqn7M8uWLcvzzz+fHXbY4b/zxQFAAyEiAWhQNthgg1x44YU588wzc/DBBydJTjvttNTU1GT//ffP3nvvnTfffDOlUqnuz1RVVX3kf3+gpqYmrVu3zrhx4+pumzt3blq3bl2+LwQAGiinswLQ4Oy3337Zc889c9tttyVJ/vznP+eUU05Jjx49kiT//Oc/U1NTU/jxNt1007Ro0aIuIt98880ceOCBefbZZ+t/eABo4ByJBKBBGjZsWJ544okkyZAhQ3LKKaekVatWWXvttbPzzjvntddeK/xYzZo1y89//vMMHz48N998c6qrq/ODH/wgO+20U7nGB4AGq1Hp38/3AQAAgI/hdFYAAAAKE5EAAAAUJiIBAAAoTEQCAABQmIgEAACgMBEJAABAYSISAACAwkQkAAAAhf1/+1hBjcyjRRgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 906.875x540 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Name Dataset     Score    St.Dev\n",
      "0     LR   Train  0.524293  0.022378\n",
      "1     LR    Test  0.483958  0.010032\n",
      "2     RD   Train  0.522606  0.019062\n",
      "3     RD    Test  0.473472  0.020209\n",
      "4    LAS   Train -0.004726  0.000553\n",
      "5    LAS    Test -0.003199  0.000080\n",
      "6     DT   Train  0.724711  0.018531\n",
      "7     DT    Test  0.561512  0.030118\n",
      "8    MLP   Train  0.573983  0.033982\n",
      "9    MLP    Test  0.452736  0.026096\n",
      "10  DMLP   Train  0.645791  0.020936\n",
      "11  DMLP    Test  0.590723  0.002015\n",
      "12   SVR   Train  0.719682  0.008204\n",
      "13   SVR    Test  0.638348  0.003131\n",
      "14    ET   Train  0.891462  0.015224\n",
      "15    ET    Test  0.795614  0.013143\n",
      "16   XGB   Train  0.880487  0.006803\n",
      "17   XGB    Test  0.786158  0.009417\n",
      "Wall time: 5.29 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "models_df,_ = compare_models(train_X, train_y, test_X, test_y, models)\n",
    "print(models_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**About LASSO having a negative score on both sets**:  \n",
    "It's not impossible to get an $R^2$ value less than 0.  \n",
    "$R^2$ is a metric used to measure the performance of a regressor. The optimal score of $R^2$ metric is 1. If any regressor predicts a constant value, you should expect an $R^2$ score of 0 for that regressor, but unexpectedly, you can get worse performance than that.  \n",
    "$R^2$ is negative only when the chosen model does not follow the trend of the data, so fits worse than a horizontal line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimization for each regressor individually"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Στα πιο απλά μοντέλα, όπου τα fit είναι πιο 'εύκολα' υπολογιστικά, χρησιμοποιούμε 10-fold cross validation στο grid search, ενώ στα πιο περίπλοκα 2-folds ώστε να είναι πιο οικονομικό."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Σε όλα τα Grid Search που εκτελούμε, φροντίζουμε να εμπεριέχονται οι default υπερπαράμετροι του κάθε Regressor, ώστε το υποτιθέμενο βέλτιστο μοντέλο να μην έχει χειρότερη απόδοση από ότι το default.  \n",
    "**Σημαντική Σημείωση**: Όταν επιστρέφουμε τον best estimator μετά από κάθε grid search, αυτό μας επιστρέφει μόνο τις τιμές οι οποίες διαφοροποιούνται από τις default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:42:27.616299Z",
     "iopub.status.busy": "2021-03-14T20:42:27.615540Z",
     "iopub.status.idle": "2021-03-14T20:42:27.632460Z",
     "shell.execute_reply": "2021-03-14T20:42:27.631460Z",
     "shell.execute_reply.started": "2021-03-14T20:42:27.616299Z"
    }
   },
   "outputs": [],
   "source": [
    "models_opt = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Linear Regression: Has no hyperparameters to optimize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:42:30.306349Z",
     "iopub.status.busy": "2021-03-14T20:42:30.306349Z",
     "iopub.status.idle": "2021-03-14T20:42:30.316239Z",
     "shell.execute_reply": "2021-03-14T20:42:30.315238Z",
     "shell.execute_reply.started": "2021-03-14T20:42:30.306349Z"
    }
   },
   "outputs": [],
   "source": [
    "LR = LinearRegression()\n",
    "models_opt.append(('LD*',LR))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Ridge Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:42:35.629459Z",
     "iopub.status.busy": "2021-03-14T20:42:35.629400Z",
     "iopub.status.idle": "2021-03-14T20:42:35.945888Z",
     "shell.execute_reply": "2021-03-14T20:42:35.944888Z",
     "shell.execute_reply.started": "2021-03-14T20:42:35.629459Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 293 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=KFold(n_splits=2, random_state=75, shuffle=True),\n",
       "             estimator=Ridge(),\n",
       "             param_grid={'alpha': [0.0, 0.05, 0.1, 0.15000000000000002, 0.2,\n",
       "                                   0.25, 0.30000000000000004,\n",
       "                                   0.35000000000000003, 0.4, 0.45, 0.5, 0.55,\n",
       "                                   0.6000000000000001, 0.65, 0.7000000000000001,\n",
       "                                   0.75, 0.8, 0.8500000000000001, 0.9,\n",
       "                                   0.9500000000000001],\n",
       "                         'solver': ('svd', 'cholesky', 'lsqr')})"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "Ridge_ = GridSearchCV(Ridge(), param_grid = {'alpha':[i for i in np.arange(0,1,0.05)],\\\n",
    "                                                'solver':('svd', 'cholesky', 'lsqr')},cv=KFold(n_splits=2,shuffle=True,random_state = 75))\n",
    "Ridge_.fit(train_X,train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:42:38.252274Z",
     "iopub.status.busy": "2021-03-14T20:42:38.252274Z",
     "iopub.status.idle": "2021-03-14T20:42:38.258835Z",
     "shell.execute_reply": "2021-03-14T20:42:38.258835Z",
     "shell.execute_reply.started": "2021-03-14T20:42:38.252274Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ridge(alpha=0.2, solver='svd')"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Ridge_.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Σημείωση:** Από εδώ και πέρα θα ορίζουμε ξανά τον best estimator, χρησιμοποιώντας τις βέλτιστες υπερπαραμέτρους. Έτσι, όταν θέλουμε να ξαναδουλέψουμε στο πρόβλημα, να έχουμε ήδη ορατή την πληροφορία για το σύνολο τον βέλτιστων υπερπαραμέτρων και να μην χρειάζεται να ξανατρέξουμε το grid search που πολλές φορές παίρνει αρκετή ώρα."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:58:39.371523Z",
     "iopub.status.busy": "2021-03-14T20:58:39.371523Z",
     "iopub.status.idle": "2021-03-14T20:58:39.392574Z",
     "shell.execute_reply": "2021-03-14T20:58:39.391615Z",
     "shell.execute_reply.started": "2021-03-14T20:58:39.371523Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ridge(alpha=0.2, solver='svd')"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Ridge_opt = Ridge(alpha=0.2,solver = 'svd')\n",
    "Ridge_opt.fit(train_X,train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:42:52.080129Z",
     "iopub.status.busy": "2021-03-14T20:42:52.080129Z",
     "iopub.status.idle": "2021-03-14T20:42:52.085520Z",
     "shell.execute_reply": "2021-03-14T20:42:52.084577Z",
     "shell.execute_reply.started": "2021-03-14T20:42:52.080129Z"
    }
   },
   "outputs": [],
   "source": [
    "models_opt.append(('RD*',Ridge_opt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Σημαντικότερη παράμετρος: $\\alpha$ καθώς είναι το μέτρο της ποινής (L1) που εφαρμόζει στο regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Lasso Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:56:36.660200Z",
     "iopub.status.busy": "2021-03-14T20:56:36.660200Z",
     "iopub.status.idle": "2021-03-14T20:56:41.909020Z",
     "shell.execute_reply": "2021-03-14T20:56:41.908428Z",
     "shell.execute_reply.started": "2021-03-14T20:56:36.660200Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 10 folds for each of 1134 candidates, totalling 11340 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 12 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  17 tasks      | elapsed:    2.3s\n",
      "[Parallel(n_jobs=-1)]: Done 492 tasks      | elapsed:    2.7s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 5.24 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done 11072 tasks      | elapsed:    5.0s\n",
      "[Parallel(n_jobs=-1)]: Done 11340 out of 11340 | elapsed:    5.0s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=KFold(n_splits=10, random_state=75, shuffle=True),\n",
       "             estimator=Lasso(), n_jobs=-1,\n",
       "             param_grid={'alpha': [0.0001, 0.0002, 0.001, 0.01, 0.1, 0.2, 0.5,\n",
       "                                   0.8, 1],\n",
       "                         'max_iter': [1000, 1500, 2000, 2500, 3000, 3500, 4000,\n",
       "                                      4500, 5000, 5500, 6000, 6500, 7000, 7500,\n",
       "                                      8000, 8500, 9000, 9500],\n",
       "                         'tol': [0.01, 0.001, 0.0001, 1e-05, 1e-06, 1e-07,\n",
       "                                 1e-08]},\n",
       "             verbose=2)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "Lasso_ = GridSearchCV(Lasso(), param_grid = {'alpha':[0.0001, 0.0002,0.001,0.01,0.1,0.2,0.5,0.8,1],\\\n",
    "                                            'max_iter':[k for k in np.arange(1000,10000,500)],\n",
    "                                            'tol':[1e-2,1e-3,1e-4,1e-5,1e-6,1e-7,1e-8]},cv=KFold(n_splits=10,shuffle=True,random_state = 75),verbose=2,n_jobs=-1)\n",
    "Lasso_.fit(train_X,train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:56:41.909767Z",
     "iopub.status.busy": "2021-03-14T20:56:41.909767Z",
     "iopub.status.idle": "2021-03-14T20:56:41.923588Z",
     "shell.execute_reply": "2021-03-14T20:56:41.923588Z",
     "shell.execute_reply.started": "2021-03-14T20:56:41.909767Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Lasso(alpha=0.0001, tol=1e-08)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Lasso_.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:44:14.223015Z",
     "iopub.status.busy": "2021-03-14T20:44:14.223015Z",
     "iopub.status.idle": "2021-03-14T20:44:14.244615Z",
     "shell.execute_reply": "2021-03-14T20:44:14.243670Z",
     "shell.execute_reply.started": "2021-03-14T20:44:14.223015Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Lasso(alpha=0.0001, tol=1e-08)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Lasso_opt = Lasso(alpha=0.0001,tol=1e-8)\n",
    "Lasso_opt.fit(train_X,train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T20:44:18.558439Z",
     "iopub.status.busy": "2021-03-14T20:44:18.558439Z",
     "iopub.status.idle": "2021-03-14T20:44:18.564690Z",
     "shell.execute_reply": "2021-03-14T20:44:18.563224Z",
     "shell.execute_reply.started": "2021-03-14T20:44:18.558439Z"
    }
   },
   "outputs": [],
   "source": [
    "models_opt.append(('LAS*',Lasso_opt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Σημαντικότερη παράμετρος: $\\alpha$ καθώς είναι το μέτρο της ποινής (L2) που εφαρμόζει στο regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Decision Trees Regressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T21:00:45.163658Z",
     "iopub.status.busy": "2021-03-14T21:00:45.163658Z",
     "iopub.status.idle": "2021-03-14T21:00:54.156973Z",
     "shell.execute_reply": "2021-03-14T21:00:54.156248Z",
     "shell.execute_reply.started": "2021-03-14T21:00:45.163658Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 10 folds for each of 704 candidates, totalling 7040 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 12 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  17 tasks      | elapsed:    0.0s\n",
      "[Parallel(n_jobs=-1)]: Done 1320 tasks      | elapsed:    0.7s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 8.97 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done 7040 out of 7040 | elapsed:    8.8s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=KFold(n_splits=10, random_state=75, shuffle=True),\n",
       "             estimator=DecisionTreeRegressor(), n_jobs=-1,\n",
       "             param_grid={'criterion': ['mse', 'mae'],\n",
       "                         'max_depth': [None, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],\n",
       "                         'min_samples_leaf': range(1, 5),\n",
       "                         'min_samples_split': range(2, 10)},\n",
       "             verbose=2)"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "DT_ = GridSearchCV(DecisionTreeRegressor(), param_grid = {'criterion': ['mse', 'mae'],\n",
    "                                                          'max_depth': [None,1,2,3,4,5,6,7,8,9,10],\n",
    "                                                         \"min_samples_split\":range(2,10),\n",
    "                                                         \"min_samples_leaf\":range(1,5)}\n",
    "                                                          ,n_jobs = -1,cv=KFold(n_splits=10,shuffle=True,random_state = 75),verbose=2)\n",
    "DT_.fit(train_X, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T21:00:58.460309Z",
     "iopub.status.busy": "2021-03-14T21:00:58.460309Z",
     "iopub.status.idle": "2021-03-14T21:00:58.472728Z",
     "shell.execute_reply": "2021-03-14T21:00:58.471799Z",
     "shell.execute_reply.started": "2021-03-14T21:00:58.460309Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(criterion='mae')"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DT_.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T21:01:06.498313Z",
     "iopub.status.busy": "2021-03-14T21:01:06.498313Z",
     "iopub.status.idle": "2021-03-14T21:01:06.533101Z",
     "shell.execute_reply": "2021-03-14T21:01:06.532248Z",
     "shell.execute_reply.started": "2021-03-14T21:01:06.498313Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(criterion='mae')"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DT_opt = DecisionTreeRegressor(criterion='mae')\n",
    "DT_opt.fit(train_X,train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T21:01:12.139664Z",
     "iopub.status.busy": "2021-03-14T21:01:12.139664Z",
     "iopub.status.idle": "2021-03-14T21:01:12.155107Z",
     "shell.execute_reply": "2021-03-14T21:01:12.154295Z",
     "shell.execute_reply.started": "2021-03-14T21:01:12.139664Z"
    }
   },
   "outputs": [],
   "source": [
    "models_opt.append(('DT*',DT_opt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Σημαντικότεροι παράμετροι: το min_samples_split, το max_depth και το κριτήριο mse ή mae."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5(a). Multi Layer Perceptron Regressor (no-hidden layers)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-13T13:21:03.908509Z",
     "iopub.status.busy": "2021-03-13T13:21:03.908509Z",
     "iopub.status.idle": "2021-03-13T13:21:37.791053Z",
     "shell.execute_reply": "2021-03-13T13:21:37.790316Z",
     "shell.execute_reply.started": "2021-03-13T13:21:03.908509Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 33.9 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=KFold(n_splits=2, random_state=75, shuffle=True),\n",
       "             estimator=MLPRegressor(random_state=75), n_jobs=-1,\n",
       "             param_grid={'activation': ['tanh', 'relu'],\n",
       "                         'alpha': [1e-05, 0.0001, 0.001],\n",
       "                         'learning_rate': ['constant', 'adaptive'],\n",
       "                         'learning_rate_init': [0.0001, 0.001, 0.003, 0.005,\n",
       "                                                0.01],\n",
       "                         'max_iter': [100, 200, 1000],\n",
       "                         'solver': ['sgd', 'adam']})"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "MLP_ = GridSearchCV(MLPRegressor(random_state=75), param_grid={'activation' : ['tanh','relu'],\n",
    "                                                'solver':['sgd','adam'],\n",
    "                                                'alpha':[1e-5,1e-4,1e-3],\n",
    "                                                'learning_rate':['constant','adaptive'],\n",
    "                                                'max_iter':[100,200,1000],\n",
    "                                                'learning_rate_init':[0.0001,0.001,0.003,0.005,0.01]},\n",
    "                                                 n_jobs = -1,cv=KFold(n_splits=2,shuffle=True,random_state = 75))\n",
    "MLP_.fit(train_X,train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-13T13:24:33.275391Z",
     "iopub.status.busy": "2021-03-13T13:24:33.275391Z",
     "iopub.status.idle": "2021-03-13T13:24:33.293883Z",
     "shell.execute_reply": "2021-03-13T13:24:33.293883Z",
     "shell.execute_reply.started": "2021-03-13T13:24:33.275391Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MLPRegressor(learning_rate_init=0.01, max_iter=100, random_state=75)"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MLP_.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T21:03:14.769773Z",
     "iopub.status.busy": "2021-03-14T21:03:14.769773Z",
     "iopub.status.idle": "2021-03-14T21:03:15.055930Z",
     "shell.execute_reply": "2021-03-14T21:03:15.055930Z",
     "shell.execute_reply.started": "2021-03-14T21:03:14.769773Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MLPRegressor(learning_rate_init=0.01, max_iter=100, random_state=75)"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MLP_opt = MLPRegressor(random_state=75, max_iter=100,learning_rate_init=0.01)\n",
    "MLP_opt.fit(train_X,train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T21:03:17.500301Z",
     "iopub.status.busy": "2021-03-14T21:03:17.500301Z",
     "iopub.status.idle": "2021-03-14T21:03:17.511267Z",
     "shell.execute_reply": "2021-03-14T21:03:17.510468Z",
     "shell.execute_reply.started": "2021-03-14T21:03:17.500301Z"
    }
   },
   "outputs": [],
   "source": [
    "models_opt.append(('MLP*',MLP_opt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Το MLP όντας νευρωνικό δίκτυο, έχει πολλές σημαντικές παραμέτρους οι οποίες πρέπει να γίνουν fine tuning ταυτόχρονα για τη μέγιστη απόδοση. Όπως το learning rate, ο solver, η activation function κτλ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5(b). Multi Layer Perceptron (with hidden layers)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Εδώ κρατάμε τις βέλτιστες υπερπαραμέτρους που βρέθηκαν από το παραπάνω Grid Search και τις χρησιμοποιούμε για να κάνουμε κι'άλλο Grid Search, αυτή τη φορά μαζί με hidden layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-13T13:27:08.400227Z",
     "iopub.status.busy": "2021-03-13T13:27:08.400227Z",
     "iopub.status.idle": "2021-03-13T13:27:42.807150Z",
     "shell.execute_reply": "2021-03-13T13:27:42.806197Z",
     "shell.execute_reply.started": "2021-03-13T13:27:08.400227Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 34.4 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=KFold(n_splits=2, random_state=75, shuffle=True),\n",
       "             estimator=MLPRegressor(random_state=75), n_jobs=-1,\n",
       "             param_grid={'hidden_layer_sizes': [(256, 256, 256),\n",
       "                                                (512, 256, 128), (64, 32, 8)],\n",
       "                         'learning_rate_init': [0.0001, 0.001, 0.005],\n",
       "                         'max_iter': [100, 200, 1000]})"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "MLP_2 = GridSearchCV(MLPRegressor(random_state=75), \n",
    "                                 param_grid={'max_iter':[100,200,1000],'learning_rate_init':[0.0001,0.001,0.005],\n",
    "                                'hidden_layer_sizes':[(256, 256, 256),(512,256,128),(64,32,8)]},n_jobs = -1,\n",
    "                                    cv=KFold(n_splits=2,shuffle=True,random_state = 75))\n",
    "MLP_2.fit(train_X,train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-13T13:27:52.234233Z",
     "iopub.status.busy": "2021-03-13T13:27:52.234233Z",
     "iopub.status.idle": "2021-03-13T13:27:52.252693Z",
     "shell.execute_reply": "2021-03-13T13:27:52.251854Z",
     "shell.execute_reply.started": "2021-03-13T13:27:52.234233Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MLPRegressor(hidden_layer_sizes=(512, 256, 128), learning_rate_init=0.005,\n",
       "             max_iter=100, random_state=75)"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MLP_2.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T21:03:30.205103Z",
     "iopub.status.busy": "2021-03-14T21:03:30.204263Z",
     "iopub.status.idle": "2021-03-14T21:03:35.921232Z",
     "shell.execute_reply": "2021-03-14T21:03:35.921232Z",
     "shell.execute_reply.started": "2021-03-14T21:03:30.205103Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MLPRegressor(hidden_layer_sizes=(512, 256, 128), learning_rate_init=0.005,\n",
       "             max_iter=100, random_state=75)"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MLP_2_opt = MLPRegressor(hidden_layer_sizes=(512, 256, 128), learning_rate_init=0.005,\n",
    "             max_iter=100, random_state=75)\n",
    "MLP_2_opt.fit(train_X,train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T21:03:39.900240Z",
     "iopub.status.busy": "2021-03-14T21:03:39.900240Z",
     "iopub.status.idle": "2021-03-14T21:03:39.909652Z",
     "shell.execute_reply": "2021-03-14T21:03:39.908305Z",
     "shell.execute_reply.started": "2021-03-14T21:03:39.900240Z"
    }
   },
   "outputs": [],
   "source": [
    "models_opt.append(('DMLP*',MLP_2_opt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**6. Support Vector Regression** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-13T15:28:42.530466Z",
     "iopub.status.busy": "2021-03-13T15:28:42.530466Z",
     "iopub.status.idle": "2021-03-13T15:28:52.191500Z",
     "shell.execute_reply": "2021-03-13T15:28:52.190493Z",
     "shell.execute_reply.started": "2021-03-13T15:28:42.530466Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 2 folds for each of 48 candidates, totalling 96 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 12 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  17 tasks      | elapsed:    0.0s\n",
      "[Parallel(n_jobs=-1)]: Done  73 out of  96 | elapsed:    0.2s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 9.65 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  96 out of  96 | elapsed:    9.5s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=KFold(n_splits=2, random_state=75, shuffle=True),\n",
       "             estimator=SVR(), n_jobs=-1,\n",
       "             param_grid={'C': [1, 5], 'coef0': [0.01, 0.001], 'degree': [3, 8],\n",
       "                         'gamma': ['scale', 'auto'],\n",
       "                         'kernel': ('poly', 'rbf', 'sigmoid')},\n",
       "             verbose=2)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "param = {'kernel' : ('poly', 'rbf','sigmoid'),'C' : [1,5],'degree' : [3,8],'coef0' : [0.01,0.001],'gamma' : ['scale','auto']}\n",
    "#defaults: kernel=rbf, C=1.0, degree=3, coef0=0.0, gamma=scale\n",
    "SVR_ = GridSearchCV(SVR(),param_grid = param,cv = KFold(n_splits=2,shuffle=True,random_state = 75),n_jobs=-1,verbose=2)\n",
    "SVR_.fit(train_X,train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-13T15:28:55.556589Z",
     "iopub.status.busy": "2021-03-13T15:28:55.555595Z",
     "iopub.status.idle": "2021-03-13T15:28:55.570591Z",
     "shell.execute_reply": "2021-03-13T15:28:55.569577Z",
     "shell.execute_reply.started": "2021-03-13T15:28:55.556589Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVR(C=5, coef0=0.01)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SVR_.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Τώρα ξέρουμε ότι ο καλύτερος kernel είναι ο RBF οπότε θα κάνουμε άλλο ένα GridSearch πιο ειδικό."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-13T15:35:26.759924Z",
     "iopub.status.busy": "2021-03-13T15:35:26.759924Z",
     "iopub.status.idle": "2021-03-13T15:35:27.883740Z",
     "shell.execute_reply": "2021-03-13T15:35:27.883740Z",
     "shell.execute_reply.started": "2021-03-13T15:35:26.759924Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 2 folds for each of 252 candidates, totalling 504 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 12 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  17 tasks      | elapsed:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 1.11 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done 504 out of 504 | elapsed:    1.0s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=KFold(n_splits=2, random_state=75, shuffle=True),\n",
       "             estimator=SVR(), n_jobs=-1,\n",
       "             param_grid={'C': [1, 5, 5.5, 6, 7, 7.5, 8],\n",
       "                         'coef0': [10, 5, 1, 0.5, 0.1, 0.05], 'degree': [3, 8],\n",
       "                         'tol': [0.01, 0.001, 0.0001]},\n",
       "             verbose=2)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "param = {'C' : [1,5,5.5,6,7,7.5,8],'degree' : [3,8],'coef0' : [10,5,1,0.5,0.1,0.05],'tol':[1e-2,1e-3,1e-4]}\n",
    "#defaults: kernel=rbf, C=1.0, degree=3, coef0=0.0, gamma=scale\n",
    "SVR_ = GridSearchCV(SVR(),param_grid = param,cv = KFold(n_splits=2,shuffle=True,random_state = 75),n_jobs=-1,verbose=2)\n",
    "SVR_.fit(train_X,train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-13T15:36:28.815809Z",
     "iopub.status.busy": "2021-03-13T15:36:28.815809Z",
     "iopub.status.idle": "2021-03-13T15:36:28.821475Z",
     "shell.execute_reply": "2021-03-13T15:36:28.821475Z",
     "shell.execute_reply.started": "2021-03-13T15:36:28.815809Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVR(C=7, coef0=10)"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SVR_.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T21:04:22.959741Z",
     "iopub.status.busy": "2021-03-14T21:04:22.958768Z",
     "iopub.status.idle": "2021-03-14T21:04:22.999925Z",
     "shell.execute_reply": "2021-03-14T21:04:22.998925Z",
     "shell.execute_reply.started": "2021-03-14T21:04:22.958768Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVR(C=7, coef0=10)"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SVR_opt = SVR(C=7, coef0 = 10)\n",
    "SVR_opt.fit(train_X,train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T21:04:27.442205Z",
     "iopub.status.busy": "2021-03-14T21:04:27.442205Z",
     "iopub.status.idle": "2021-03-14T21:04:27.458558Z",
     "shell.execute_reply": "2021-03-14T21:04:27.457190Z",
     "shell.execute_reply.started": "2021-03-14T21:04:27.442205Z"
    }
   },
   "outputs": [],
   "source": [
    "models_opt.append(('SVR*',SVR_opt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Μέγιστη σημασία στο support vector regression έχει η επιλογή κατάλληλου Kernel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**7. Extra Trees Regressor**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-13T15:50:31.530333Z",
     "iopub.status.busy": "2021-03-13T15:50:31.529296Z",
     "iopub.status.idle": "2021-03-13T15:50:55.940615Z",
     "shell.execute_reply": "2021-03-13T15:50:55.939683Z",
     "shell.execute_reply.started": "2021-03-13T15:50:31.530333Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 10 folds for each of 54 candidates, totalling 540 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 12 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  17 tasks      | elapsed:    0.3s\n",
      "[Parallel(n_jobs=-1)]: Done 252 tasks      | elapsed:    2.9s\n",
      "[Parallel(n_jobs=-1)]: Done 509 tasks      | elapsed:   21.7s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 24.4 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done 540 out of 540 | elapsed:   24.1s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=KFold(n_splits=10, random_state=75, shuffle=True),\n",
       "             estimator=ExtraTreesRegressor(random_state=75), n_jobs=-1,\n",
       "             param_grid={'criterion': ['mse', 'mae'], 'max_depth': [None, 1, 2],\n",
       "                         'min_samples_leaf': [1, 2, 3],\n",
       "                         'min_samples_split': [1, 2, 3]},\n",
       "             verbose=2)"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "ET_ = GridSearchCV(ExtraTreesRegressor(random_state=75), param_grid = {'criterion': ['mse', 'mae'],\n",
    "                                                          'max_depth': [None,1,2],\n",
    "                                                         \"min_samples_split\":[1,2,3],\n",
    "                                                         \"min_samples_leaf\":[1,2,3]},\n",
    "                                                           cv=KFold(n_splits=10,shuffle=True,random_state = 75),n_jobs=-1,verbose=2)\n",
    "ET_.fit(train_X, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-13T15:51:06.532794Z",
     "iopub.status.busy": "2021-03-13T15:51:06.531782Z",
     "iopub.status.idle": "2021-03-13T15:51:06.540780Z",
     "shell.execute_reply": "2021-03-13T15:51:06.539814Z",
     "shell.execute_reply.started": "2021-03-13T15:51:06.532794Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ExtraTreesRegressor(random_state=75)"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ET_.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Παρατηρούμε ότι ο Extra Trees Regressor προτιμάει τις default παραμέτρους του."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T21:04:40.205148Z",
     "iopub.status.busy": "2021-03-14T21:04:40.205148Z",
     "iopub.status.idle": "2021-03-14T21:04:40.420529Z",
     "shell.execute_reply": "2021-03-14T21:04:40.419932Z",
     "shell.execute_reply.started": "2021-03-14T21:04:40.205148Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ExtraTreesRegressor(random_state=75)"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ET_opt = ExtraTreesRegressor(random_state=75)\n",
    "ET_opt.fit(train_X,train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T21:04:44.321174Z",
     "iopub.status.busy": "2021-03-14T21:04:44.321174Z",
     "iopub.status.idle": "2021-03-14T21:04:44.339074Z",
     "shell.execute_reply": "2021-03-14T21:04:44.337537Z",
     "shell.execute_reply.started": "2021-03-14T21:04:44.321174Z"
    }
   },
   "outputs": [],
   "source": [
    "models_opt.append(('ET*',ET_opt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ομοίως με τα Decision Trees μεγάλο ρόλο παίζει η παράμετρος με την οποία γίνεται η απόφαση (gini ή entropy) καθώς και min_samples_split και το max_depth."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**8. XG Boost Regressor**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-13T15:55:49.864888Z",
     "iopub.status.busy": "2021-03-13T15:55:49.863891Z",
     "iopub.status.idle": "2021-03-13T16:00:14.535498Z",
     "shell.execute_reply": "2021-03-13T16:00:14.534499Z",
     "shell.execute_reply.started": "2021-03-13T15:55:49.864888Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 2 folds for each of 360 candidates, totalling 720 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 12 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  17 tasks      | elapsed:    2.9s\n",
      "[Parallel(n_jobs=-1)]: Done 138 tasks      | elapsed:   24.5s\n",
      "[Parallel(n_jobs=-1)]: Done 341 tasks      | elapsed:  1.2min\n",
      "[Parallel(n_jobs=-1)]: Done 624 tasks      | elapsed:  3.7min\n",
      "[Parallel(n_jobs=-1)]: Done 720 out of 720 | elapsed:  4.4min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 4min 24s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=KFold(n_splits=2, random_state=75, shuffle=True),\n",
       "             estimator=XGBRegressor(base_score=None, booster=None,\n",
       "                                    colsample_bylevel=None,\n",
       "                                    colsample_bynode=None,\n",
       "                                    colsample_bytree=None, gamma=None,\n",
       "                                    gpu_id=None, importance_type='gain',\n",
       "                                    interaction_constraints=None,\n",
       "                                    learning_rate=None, max_delta_step=None,\n",
       "                                    max_depth=None, min_child_weight=None,\n",
       "                                    missing=nan, monot...\n",
       "                                    n_estimators=100, n_jobs=-1,\n",
       "                                    num_parallel_tree=None, random_state=75,\n",
       "                                    reg_alpha=None, reg_lambda=None,\n",
       "                                    scale_pos_weight=None, subsample=None,\n",
       "                                    tree_method=None, validate_parameters=None,\n",
       "                                    verbosity=None),\n",
       "             n_jobs=-1,\n",
       "             param_grid={'booster': ['gbtree', 'dart'],\n",
       "                         'gamma': [0, 1, 2, 3, 4], 'max_depth': [5, 6, 7],\n",
       "                         'min_child_weight': [1, 5],\n",
       "                         'n_estimators': [100, 500, 1000],\n",
       "                         'subsample': [0.5, 1]},\n",
       "             verbose=2)"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "XGB_ = GridSearchCV(XGBRegressor(random_state=75,n_jobs=-1),\n",
    "                                                            param_grid  = {'booster':['gbtree','dart'],\n",
    "                                                            'min_child_weight':[1,5], \n",
    "                                                            'gamma':[0,1,2,3,4],  \n",
    "                                                            'subsample':[0.5,1],\n",
    "                                                            'max_depth': [5,6,7],\n",
    "                                                             'n_estimators':[100,500,1000]},\n",
    "                                                            cv=KFold(n_splits=2,shuffle=True,random_state = 75),n_jobs=-1,verbose=2)\n",
    "XGB_.fit(train_X, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-13T16:00:23.783004Z",
     "iopub.status.busy": "2021-03-13T16:00:23.783004Z",
     "iopub.status.idle": "2021-03-13T16:00:23.791002Z",
     "shell.execute_reply": "2021-03-13T16:00:23.790012Z",
     "shell.execute_reply.started": "2021-03-13T16:00:23.783004Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "             colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n",
       "             importance_type='gain', interaction_constraints='',\n",
       "             learning_rate=0.300000012, max_delta_step=0, max_depth=5,\n",
       "             min_child_weight=1, missing=nan, monotone_constraints='()',\n",
       "             n_estimators=1000, n_jobs=-1, num_parallel_tree=1, random_state=75,\n",
       "             reg_alpha=0, reg_lambda=1, scale_pos_weight=1, subsample=0.5,\n",
       "             tree_method='exact', validate_parameters=1, verbosity=None)"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "XGB_.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-13T16:00:55.873327Z",
     "iopub.status.busy": "2021-03-13T16:00:55.873327Z",
     "iopub.status.idle": "2021-03-13T16:00:55.891328Z",
     "shell.execute_reply": "2021-03-13T16:00:55.890326Z",
     "shell.execute_reply.started": "2021-03-13T16:00:55.873327Z"
    }
   },
   "outputs": [],
   "source": [
    "XGB_opt = XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
    "             colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n",
    "             importance_type='gain', interaction_constraints='',\n",
    "             learning_rate=0.300000012, max_delta_step=0, max_depth=5,\n",
    "             min_child_weight=1,monotone_constraints='()',\n",
    "             n_estimators=1000, n_jobs=-1, num_parallel_tree=1, random_state=75,\n",
    "             reg_alpha=0, reg_lambda=1, scale_pos_weight=1, subsample=0.5,\n",
    "             tree_method='exact', validate_parameters=1, verbosity=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-13T16:08:58.736110Z",
     "iopub.status.busy": "2021-03-13T16:08:58.735108Z",
     "iopub.status.idle": "2021-03-13T16:09:16.701465Z",
     "shell.execute_reply": "2021-03-13T16:09:16.700465Z",
     "shell.execute_reply.started": "2021-03-13T16:08:58.736110Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 2 folds for each of 320 candidates, totalling 640 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 12 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  17 tasks      | elapsed:    0.6s\n",
      "[Parallel(n_jobs=-1)]: Done 138 tasks      | elapsed:    2.9s\n",
      "[Parallel(n_jobs=-1)]: Done 341 tasks      | elapsed:    8.0s\n",
      "[Parallel(n_jobs=-1)]: Done 640 out of 640 | elapsed:   17.7s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 17.9 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=KFold(n_splits=2, random_state=75, shuffle=True),\n",
       "             estimator=XGBRegressor(base_score=None, booster=None,\n",
       "                                    colsample_bylevel=None,\n",
       "                                    colsample_bynode=None,\n",
       "                                    colsample_bytree=None, gamma=None,\n",
       "                                    gpu_id=None, importance_type='gain',\n",
       "                                    interaction_constraints=None,\n",
       "                                    learning_rate=None, max_delta_step=None,\n",
       "                                    max_depth=None, min_child_weight=None,\n",
       "                                    missing=nan, monotone_constraints=None,\n",
       "                                    n_estimators=100, n_jobs=-1,\n",
       "                                    num_parallel_tree=None, random_state=75,\n",
       "                                    reg_alpha=None, reg_lambda=None,\n",
       "                                    scale_pos_weight=None, subsample=None,\n",
       "                                    tree_method=None, validate_parameters=None,\n",
       "                                    verbosity=None),\n",
       "             n_jobs=-1,\n",
       "             param_grid={'max_depth': [3, 4, 5, 6],\n",
       "                         'reg_alpha': [0, 0.1, 0.5, 1],\n",
       "                         'reg_lambda': [1, 1.5, 3, 5],\n",
       "                         'subsample': [0.2, 0.3, 0.4, 0.5, 0.6]},\n",
       "             verbose=2)"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "XGB_ = GridSearchCV(XGBRegressor(random_state=75,n_jobs=-1),\n",
    "                                                            param_grid  = {'subsample':[0.2,0.3,0.4,0.5,0.6],\n",
    "                                                            'max_depth': [3,4,5,6],\n",
    "                                                            'reg_alpha':[0,0.1,0.5,1],\n",
    "                                                             'reg_lambda':[1,1.5,3,5]},\n",
    "                                                            cv=KFold(n_splits=2,shuffle=True,random_state = 75),n_jobs=-1,verbose=2)\n",
    "XGB_.fit(train_X, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-13T16:09:16.703464Z",
     "iopub.status.busy": "2021-03-13T16:09:16.703464Z",
     "iopub.status.idle": "2021-03-13T16:09:16.717472Z",
     "shell.execute_reply": "2021-03-13T16:09:16.716492Z",
     "shell.execute_reply.started": "2021-03-13T16:09:16.703464Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "             colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n",
       "             importance_type='gain', interaction_constraints='',\n",
       "             learning_rate=0.300000012, max_delta_step=0, max_depth=6,\n",
       "             min_child_weight=1, missing=nan, monotone_constraints='()',\n",
       "             n_estimators=100, n_jobs=-1, num_parallel_tree=1, random_state=75,\n",
       "             reg_alpha=0, reg_lambda=5, scale_pos_weight=1, subsample=0.5,\n",
       "             tree_method='exact', validate_parameters=1, verbosity=None)"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "XGB_.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T21:33:49.255126Z",
     "iopub.status.busy": "2021-03-14T21:33:49.255126Z",
     "iopub.status.idle": "2021-03-14T21:33:49.408601Z",
     "shell.execute_reply": "2021-03-14T21:33:49.408601Z",
     "shell.execute_reply.started": "2021-03-14T21:33:49.255126Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 138 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "             colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n",
       "             importance_type='gain', interaction_constraints='',\n",
       "             learning_rate=0.300000012, max_delta_step=0, max_depth=6,\n",
       "             min_child_weight=1, missing=nan, monotone_constraints='()',\n",
       "             n_estimators=100, n_jobs=-1, num_parallel_tree=1, random_state=75,\n",
       "             reg_alpha=0, reg_lambda=5, scale_pos_weight=1, subsample=0.5,\n",
       "             tree_method='exact', validate_parameters=1, verbosity=None)"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "XGB_opt = XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
    "             colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n",
    "             importance_type='gain', interaction_constraints='',\n",
    "             learning_rate=0.300000012, max_delta_step=0, max_depth=6,\n",
    "             min_child_weight=1, monotone_constraints='()',\n",
    "             n_estimators=100, n_jobs=-1, num_parallel_tree=1, random_state=75,\n",
    "             reg_alpha=0, reg_lambda=5, scale_pos_weight=1, subsample=0.5,\n",
    "             tree_method='exact', validate_parameters=1, verbosity=None)\n",
    "XGB_opt.fit(train_X,train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T21:33:50.597055Z",
     "iopub.status.busy": "2021-03-14T21:33:50.596054Z",
     "iopub.status.idle": "2021-03-14T21:33:50.607414Z",
     "shell.execute_reply": "2021-03-14T21:33:50.606100Z",
     "shell.execute_reply.started": "2021-03-14T21:33:50.597055Z"
    }
   },
   "outputs": [],
   "source": [
    "models_opt.append(('XGB*',XGB_opt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "O XGBoost καθώς είναι επικίνδυνος στο overfitting, μεγάλη σημασία έχουν οι παράμετροι max_depth, min_child_weight και gamma διότι ελέγχουν την πολυπλοκότητα του μοντέλου."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T22:06:34.348702Z",
     "iopub.status.busy": "2021-03-14T22:06:34.347701Z",
     "iopub.status.idle": "2021-03-14T22:06:34.362717Z",
     "shell.execute_reply": "2021-03-14T22:06:34.361716Z",
     "shell.execute_reply.started": "2021-03-14T22:06:34.348702Z"
    }
   },
   "outputs": [],
   "source": [
    "def compare_models_final(trainX,trainY,testX,testY,models,modelsopt,tempdf):\n",
    "    print('\\nCompare Multiple Regressors:')\n",
    "    print('\\nK-Fold Cross-Validation Accuracy:\\n')\n",
    "    \n",
    "#     tempdf = pd.DataFrame(columns=['Name','Dataset','Score','St.Dev'])\n",
    "    for model, modelopt in zip(models,modelsopt):\n",
    "        modelopt[1].fit(trainX,trainY)\n",
    "        kfold = KFold(n_splits=2,shuffle=True,random_state = 75)\n",
    "        train_accuracy_results = cross_val_score(modelopt[1], trainX,trainY, cv=kfold)\n",
    "        test_accuracy_results = cross_val_score(modelopt[1],testX,testY,cv=kfold)\n",
    "        tempdf = tempdf.append({'Name':model[0],'Dataset':'Train-opt','Score':train_accuracy_results.mean(),\n",
    "                               'St.Dev':train_accuracy_results.std()},ignore_index=True)\n",
    "        tempdf = tempdf.append({'Name':model[0],'Dataset':'Test-opt','Score':test_accuracy_results.mean(),\n",
    "                               'St.Dev':test_accuracy_results.std()},ignore_index=True)\n",
    "    tempdf['Percent Score(%)'] = 100*tempdf['Score']\n",
    "    print('Models with default hyperparameters:')\n",
    "    print(tempdf.loc[:17,:])\n",
    "    print('Models after hyperparameter optimization:')\n",
    "    print(tempdf.loc[18:,:])\n",
    "    g = sns.catplot(x = 'Name', y='Percent Score(%)',hue = 'Dataset',data=tempdf, kind='bar',height=8.27, aspect=11.7/8.27)\n",
    "    g.set_xticklabels(rotation=30)\n",
    "    g.set(yticks=np.arange(0,90,10))\n",
    "    return tempdf,plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T22:06:35.410111Z",
     "iopub.status.busy": "2021-03-14T22:06:35.410111Z",
     "iopub.status.idle": "2021-03-14T22:06:54.950979Z",
     "shell.execute_reply": "2021-03-14T22:06:54.950979Z",
     "shell.execute_reply.started": "2021-03-14T22:06:35.410111Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Compare Multiple Regressors:\n",
      "\n",
      "K-Fold Cross-Validation Accuracy:\n",
      "\n",
      "Models with default hyperparameters:\n",
      "    Name Dataset     Score    St.Dev  Percent Score(%)\n",
      "0     LR   Train  0.524293  0.022378         52.429282\n",
      "1     LR    Test  0.483958  0.010032         48.395821\n",
      "2     RD   Train  0.522606  0.019062         52.260580\n",
      "3     RD    Test  0.473472  0.020209         47.347213\n",
      "4    LAS   Train -0.004726  0.000553         -0.472610\n",
      "5    LAS    Test -0.003199  0.000080         -0.319950\n",
      "6     DT   Train  0.724711  0.018531         72.471080\n",
      "7     DT    Test  0.561512  0.030118         56.151249\n",
      "8    MLP   Train  0.573983  0.033982         57.398279\n",
      "9    MLP    Test  0.452736  0.026096         45.273615\n",
      "10  DMLP   Train  0.645791  0.020936         64.579134\n",
      "11  DMLP    Test  0.590723  0.002015         59.072253\n",
      "12   SVR   Train  0.719682  0.008204         71.968240\n",
      "13   SVR    Test  0.638348  0.003131         63.834776\n",
      "14    ET   Train  0.891462  0.015224         89.146186\n",
      "15    ET    Test  0.795614  0.013143         79.561411\n",
      "16   XGB   Train  0.880487  0.006803         88.048735\n",
      "17   XGB    Test  0.786158  0.009417         78.615801\n",
      "Models after hyperparameter optimization:\n",
      "    Name    Dataset     Score    St.Dev  Percent Score(%)\n",
      "18    LR  Train-opt  0.524293  0.022378         52.429282\n",
      "19    LR   Test-opt  0.483958  0.010032         48.395821\n",
      "20    RD  Train-opt  0.524113  0.020208         52.411255\n",
      "21    RD   Test-opt  0.480662  0.017326         48.066199\n",
      "22   LAS  Train-opt  0.524347  0.022057         52.434671\n",
      "23   LAS   Test-opt  0.483742  0.011260         48.374202\n",
      "24    DT  Train-opt  0.732810  0.017792         73.281007\n",
      "25    DT   Test-opt  0.555588  0.093985         55.558811\n",
      "26   MLP  Train-opt  0.596438  0.012980         59.643784\n",
      "27   MLP   Test-opt  0.530174  0.000154         53.017413\n",
      "28  DMLP  Train-opt  0.802352  0.021276         80.235244\n",
      "29  DMLP   Test-opt  0.649542  0.017360         64.954159\n",
      "30   SVR  Train-opt  0.745700  0.000580         74.570020\n",
      "31   SVR   Test-opt  0.620144  0.019027         62.014429\n",
      "32    ET  Train-opt  0.891462  0.015224         89.146186\n",
      "33    ET   Test-opt  0.795614  0.013143         79.561411\n",
      "34   XGB  Train-opt  0.898965  0.000867         89.896470\n",
      "35   XGB   Test-opt  0.822630  0.015809         82.263014\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6QAAAJjCAYAAAAbJdxlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABQ/UlEQVR4nO3deUBU5eL/8Q/LgBumIuB6ySyzrmVmLnwrMTNckHAX3FKz1NwzNbfMLZcyXMrMXCo1vWKayDXKq0E39bZobrlU5p4imguIbDP8/vDn3LyJoMzwDPh+/dXMnHnmM4/TwIfznHPcsrOzswUAAAAAQAFzNx0AAAAAAHBnopACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIzwNB3gVp07lyKbjRMDAwAAAHAcPz8f0xHuSOwhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGOFpOgAAAACAoqf0Xd7y9vLK1xjpGRm6dDHdQYngiiikAAAAABzO28tLPZYMztcYH/acLYlCWpSxZBcAAAAAYASFFAAAAABgBEt2AQAAUCB8ShdTMW9LvsdJS89U8qW0fI3hiOMbJY5xBPKLQgoAAIACUczbos4jlud7nE9mdFGy8ldIHXF8o8QxjkB+sWQXAAAAAGAEhRQAAAAAYASFFAAAAABgBIUUAAAAAGAEhRQAAAAAYARn2QUAAACKAFe6rA6QVxRSAAAAoAhwpcvqAHnFkl0AAAAAgBEUUgAAAACAERRSAAAAAIARFFIAAAAAgBEUUgAAAACAERRSAAAAAIARFFIAAAAAgBEUUgAAAACAERRSAAAAAIARFFIAAAAAgBEUUgAAAACAERRSAAAAAIARFFIAAAAAgBEUUgAAAACAERRSAAAAAIARFFIAAAAAgBEUUgAAAACAERRSAAAAAIARFFIAAAAAgBEUUgAAAACAERRSAAAAAIARFFIAAAAAgBEUUgAAAACAERRSAAAAAIARFFIAAAAAgBEUUgAAAACAERRSAAAAAIARFFIAAAAAgBEUUgAAAACAERRSAAAAAIARnqYDAAAAAMCNWDMy5efnk+9xMtPSdSE5wwGJ4GgUUgAAAAAuycPLog3de+Z7nJYfL5EopC6JJbsAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACOcWkjXrVun0NBQhYaGavr06ZKkrVu3KiwsTCEhIYqKinLmywMAAAAAXJjTCumVK1c0ZcoULV26VOvWrdMPP/ygzZs3a/To0Zo3b542bNigvXv3KiEhwVkRAAAAAAAuzNNZA1utVtlsNl25ckUlSpRQVlaWSpUqpcDAQFWtWlWSFBYWpri4OAUHBzsrBgAAcJLSd3nL28srX2OkZ2To0sV0ByUCABQ2TiukpUqV0uDBg9WiRQsVL15c9erV05kzZ+Tn52ffxt/fX4mJibc0rq9vKUdHBQAAt6nHksH5ev6HPWfLzy9/pRZ3Jj8/H9MR7Fwpi6PwnlBQnFZIDxw4oE8//VRfffWVfHx89Morr+jIkSNyc3Ozb5OdnX3d7bw4dy5FNlu2o+MCAIBb5Khf7pKSkh0yDlyfIwtBfj83rpTFUVztPblaAcztPbla3juF0wrpN998o6CgIPn6+kqS2rZtq0WLFsnDw8O+TVJSkvz9/Z0VAQAAAEWQLSvTZcqDNcMxWTLT0nUhOcMBiYDCxWmFtGbNmnrzzTeVmpqq4sWLa/Pmzapdu7bWr1+vo0ePqkqVKoqNjVW7du2cFQEAAABFkLunRdtn9M7XGHVHLHRIFg8vizZ075nvcVp+vESikOIO5LRC+sQTT2jfvn1q27atLBaLHnroIQ0cOFCPP/64Bg4cqPT0dAUHB6t58+bOigAAAAAAcGFOK6SS9OKLL+rFF1+87r6goCDFxMQ482UBAAAAAIWA065DCgAAAADAzVBIAQAAAABGOHXJLgAAAIDCxZXOYoyij0IKAAAAwM4RZzGWHHcmYxRtLNkFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGOFpOgAAAAD+qvRd3vL28sr3OOkZGbp0Md0BiQDA8SikAAAALsjby0s9lgzO9zgf9pwtiUIKwDWxZBcAAAAAYASFFAAAAABgBIUUAAAAAGAEhRQAAAAAYASFFAAAAABgBGfZBQBDuKQDAAC401FIAcAQLukAAADudCzZBQAAAAAYQSEFAAAAABhBIQUAAAAAGMExpADuKD6li6mYtyVfY6SlZyr5UpqDEgEAANy5KKQA7ijFvC3qPGJ5vsb4ZEYXJYtCCgAAkF8s2QUAAAAAGMEeUgAAgP+PZf0AULAopAAAAP8fy/oBoGCxZBcAAAAAYASFFAAAAABgBIUUAAAAAGAEhRQAAAAAYASFFAAAAABghNPOshsdHa1ly5bZb584cULh4eFq2rSppk6dqvT0dLVo0UJDhw51VgQAAAAAgAtzWiHt0KGDOnToIEn65Zdf1L9/f73wwguKjIzU0qVLVbFiRfXp00cJCQkKDg52VgwAAAAAgIsqkCW7r7/+uoYOHarjx48rMDBQVatWlaenp8LCwhQXF1cQEQAAAAAALsZpe0iv2bp1q9LS0tSiRQvFxsbKz8/P/pi/v78SExNvaTxf31KOjggAt8zPz8d0hOu4Wh7gVhTFz6+rvSdXy4Mb49/JuZhf1+T0Qrpy5Ur17NlTkmSz2eTm5mZ/LDs7+7rbeXHuXIpstmyHZgRw53DUD6OkpOR8j+HIH4yOyAPcKlf6/8lRXOk9FcXvCArBzeX334n5vbnc5pf5M8OpS3YzMjL0/fffq0mTJpKkChUqKCkpyf54UlKS/P39nRkBAAAAAOCinFpIDx48qLvvvlslSpSQJNWuXVuHDx/W0aNHZbVaFRsbq0aNGjkzAgAAAADARTl1ye7x48dVoUIF+21vb29NmzZNAwcOVHp6uoKDg9W8eXNnRgAAAAAAuCinFtKWLVuqZcuW190XFBSkmJgYZ74sAAAAAKAQKJDLvgAAAAAA8L8opAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACM8TQcAAAB3LmtGpvz8fPI9TmZaui4kZzggEQCgIFFIAQCAMR5eFm3o3jPf47T8eIlEIQWAQodCCgBAIeFTupiKeVvyPU5aeqaSL6U5IBEAAPlDIQUAoJAo5m1R5xHL8z3OJzO6KFkUUgCAeZzUCAAAAABgBIUUAAAAAGAEhRQAAAAAYATHkAIAADiQLcsxl7IBgDsBhRQAAMCB3D0t2j6jd77HqTtioQPSAIBrY8kuAAAAAMAICikAAAAAwAgKKQAAAADACAopAAAAAMAICikAAAAAwAgKKQAAAADACAopAAAAAMAICikAAAAAwAhP0wEAAADgPNaMTPn5+eRrjMy0dF1IznBQIgD4LwopAABAEebhZdGG7j3zNUbLj5dIFFIATsCSXQAAAACAERRSAAAAAIARFFIAAAAAgBEUUgAAAACAERRSAAAAAIARFFIAAAAAgBFOLaSbN29W27Zt1aJFC02ePFmStHXrVoWFhSkkJERRUVHOfHkAAAAAgAtzWiE9fvy4xo8fr3nz5ikmJkb79u1TQkKCRo8erXnz5mnDhg3au3evEhISnBUBAAAAAODCnFZIN27cqJYtW6pChQqyWCyKiopS8eLFFRgYqKpVq8rT01NhYWGKi4tzVgQAAAAAgAvzdNbAR48elcViUd++fXXq1Ck1btxY9913n/z8/Ozb+Pv7KzEx8ZbG9fUt5eioAHDL/Px8TEe4jqvlgesrip+ZovieXAnz63zMsXMxv67JaYXUarXqhx9+0NKlS1WiRAn169dPxYoVk5ubm32b7Ozs627nxblzKbLZsh0dF8AdwlE/jJKSkvM9hiN/MDoiD1yfq31mXO2Xu6L4nlwJ8+t8+Z1j5vfmcptf5s8MpxXS8uXLKygoSOXKlZMkNW3aVHFxcfLw8LBvk5SUJH9/f2dFAADcwUrf5S1vL698j5OekaFLF9MdkAgAAPwvpxXSp556SiNHjtSlS5dUsmRJ/fvf/1bz5s21YMECHT16VFWqVFFsbKzatWvnrAgAgDuYt5eXeiwZnO9xPuw5WxKFFAAAZ3BaIa1du7Z69+6tzp07KzMzU48//rgiIyN1zz33aODAgUpPT1dwcLCaN2/urAgAAAAAABfmtEIqSe3bt1f79u2vuy8oKEgxMTHOfFkAAHATtqxMjpUCALgEpxZSAADgetw9Ldo+o3e+x6k7YqED0gAA7mROuw4pAAAAAAA3QyEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYwXVIAQAO41O6mIp5W/I9Tlp6ppIvpTkgEQAAcGUUUgCAwxTztqjziOX5HueTGV2ULAopAABFHUt2AQAAAABGUEgBAAAAAEZQSAEAAAAARlBIAQAAAABGUEgBAAAAAEZwll0AuEW2rEz5+fmYjgEAAFDoUUgB4Ba5e1q0fUbvfI9Td8RCB6QBAAAovFiyCwAAAAAwgj2kAFDIWTPyv4Q4My1dF5IzHJQIAAAgbyikAFDIeXhZtKF7z3yN0fLjJRKFFAAAFDCW7AIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIzwNB0AAABXZs3IlJ+fT77HyUxL14XkDAckAgCg6Mi1kGZlZWnHjh06fPiw3N3ddc8996hu3boFkQ0AAOM8vCza0L1nvsdp+fESiUIKAMB1blpIly9frvfff19lypRRlSpV5OnpqRUrVuj8+fN64YUXFBkZKTc3t4LKCgAAAAAoQnIspAMHDtQDDzyg1atXy9/f/7rHzp07pxUrVqh///6aN2+e00MCAAAAAIqeHAvpmDFjVKFChRs+5uvrqwEDBuj06dNOC3a7fEoXUzFvS77GSEvPVPKltHxnKX2Xt7y9vPI9TnpGhi5dTM/3OEUN8+t8jphj5hcAAAA5ybGQ3qiMZmRkKDMzUyVLlsxxG9OKeVvUecTyfI3xyYwuSlb+C6m3l5d6LBmc73E+7Dlbkmv8Qu+Iwi85pvQXxfl1NY6YY+YXAAAAOcnzWXbXr1+vBQsWyGazKSwsTH379nVmLrgoRxR+yXGlHwAAAEDhleN1SC9cuHDd7ZiYGMXExCg2NlZr1651di4AAAAAQBGX4x7Svn37KiwsTJ07d5abm5v8/Pw0f/58eXh4qEyZMgUYEQAAAABQFOW4h3TZsmVKS0tT165dtX37dr322mvy9fVViRIlNH/+/ILMCAAAAAAognLcQ+rp6annn39ezz77rGbMmCE3NzcNHz5cfn5+BZkPAAAAAFBE5VhIU1JStG7dOnl5eWny5Mnas2ePBg8erKefflrPPfecPD3zfD4kAAAAAAD+Isclu/369dPly5d14sQJvfrqq3rssce0bNkyeXt7q2vXrgWZEQAAAABQBOW4mzMxMVEvvvii0tPT1a5dO0mSu7u7unbtqtDQ0AILCAAAAAAomnIspC1btlRoaKiys7PVpUuX6x4rW7Zsngbv1q2b/vjjD/vy3okTJ+ry5cuaOnWq0tPT1aJFCw0dOjQf8QEAAAAAhVWOhXTIkCHq3bu3PD09VaxYsVseODs7W0eOHNFXX31lL6RpaWlq3ry5li5dqooVK6pPnz5KSEhQcHDw7b8DAAAAAEChlOMxpNHR0SpVqtRNy+g//vGPHB/77bffJEm9evXSs88+q2XLlmn37t0KDAxU1apV5enpqbCwMMXFxeUjPgAAAACgsMpxD2lWVpY6deqkZ599Vk899ZQqVaokSTp58qS+/vprrV69Wm3atMlx4EuXLikoKEjjxo1TZmamunfvrt69e1932Rh/f38lJibeUmBf31K3tP3t8vPzKZDXyStXy+MIrvSeXClLUcT8Fg6u9u/kankcoSi+J1fC/DoX8+t8zLFzMb+uKcdCGhkZqeDgYL3//vt65513lJycLEkqXbq0QkJCNHv2bFWpUiXHgevUqaM6derYb7dv315z5sxR3bp17fdlZ2fLzc3tlgKfO5cimy07x8cd9UFLSkrO9xiO/NA7Io8juNJ7cqUsRZUr/f/kKPwwyllR+95zxX/rovieXImrfYaLGubX+fiOcK7c5pf5M+OmFxOtVKmSJkyYoAkTJuj8+fNyc3NTmTJl8jTwDz/8oMzMTAUFBUm6Wj4rV66spKQk+zZJSUny9/e//fQAAAAAgEIrx2NI/ywuLk4ffvihvL29FRsbm6eBk5OTNWPGDKWnpyslJUVr167Vyy+/rMOHD+vo0aOyWq2KjY1Vo0aN8vUGAAAAAACF0033kErSggULtGXLFp0+fVo9evTQO++8o6NHj6p///43fd5TTz2lXbt2qXXr1rLZbOrcubPq1KmjadOmaeDAgUpPT1dwcLCaN2/usDcDAAAAACg8ci2k//znPxUdHa2OHTuqbNmyWrVqlTp16pRrIZWuXjpmyJAh190XFBSkmJiY2w4MAAAAACgacl2y6+npKS8vL/vt0qVL268rCgAAAADA7cq1WVasWFHx8fFyc3NTRkaGFi1apMqVKxdENgAAAABAEZZrIR03bpxGjBihgwcP6pFHHlHt2rU1c+bMgsgGAAAAACjCci2ke/bs0UcffaQrV67IarWqVKlSBZHLKFtWpktdh8iakf88mWnpupCc4aBEQN454vMr8RkGAAAoinItpFFRUWratKmKFy9eEHlcgrunRdtn9M73OHVHLHRAGsnDy6IN3Xvma4yWHy+R+GUeBjji8yvxGQYAACiKci2kNWrU0HvvvafHHntMJUqUsN//97//3anBgILCHjwAAADAjFwL6a5du7Rr1y5FR0fb73Nzc9OmTZucGgwoKOzBAwAAAMzItZBu3ry5IHIAAAAAAO4wuRbS1NRUzZgxQ19//bWysrL0+OOPa8yYMXfEyY0AAAAAAM6TayGdOnWqrFar3n33XVmtVn3yySeaNGmSpk+fXhD5UES52pmMAQAAABS8PB1DGhMTY789efJkhYaGOjUUij5HnMnYUWcxBgAAAGCGe24bWK1W2Ww2+22bzSYPDw+nhgIAAAAAFH257iENCgrSkCFDFBkZKUlasWKF6tev7/RgAAAAAFAUnThxQs8884xq1Kgh6epOv5IlS6p79+5q2bLlTZ/7zjvvqGbNmmratKnDc/Xq1UtvvfWWypUr5/Cxc5JrIX311Vf13nvv6e2335bValWjRo3Ur1+/gsgGAAAAAEVSsWLFtG7dOvvtkydPqkePHvLw8FCzZs1yfN63336re++91ymZtmzZ4pRxbybXQipJgYGBio6OVlJSkv75z3/KYrE4OxcAAAAA3DEqV66sQYMGadGiRapRo4YmTpyoy5cvKykpSTVr1tSsWbO0evVq7d27VzNmzJCHh4fuvffeG27n7e2tOXPmaOPGjbJYLCpbtqymTp0qf39/HTp0SFOmTNGFCxdktVrVrVs3tW/fXqNGjZIkPffcc1qwYIEqVqxYIO8712NIX3/9dcXHx1/d2N1d27dv1xtvvOHsXAAAAABwR6lZs6Z+/vlnrVq1Sq1bt9aqVav05Zdf6sSJE4qPj1eXLl1Uq1YtjRgxQs8880yO2506dUofffSRPv30U61Zs0aPP/64du/eraysLA0aNEjDhg3TmjVrtGzZMi1evFg7d+7U1KlTJUkfffRRgZVRKQ97SHfu3KnY2FhJkq+vr2bPnq3w8HCnBwMAAACAO4mbm5uKFSum4cOHa8uWLfrggw905MgRnTlzRqmpqX/ZPqftAgICVLNmTbVp00aNGjVSo0aNFBQUpF9//VXHjh3T6NGj7WOkpaVp3759euSRRwrwnf5XroU0MzNTGRkZ8vLykiRlZWU5PRQAAAAA3Gn27NmjGjVq6OWXX5bValWLFi3UuHFjnTp1StnZ2X/ZPqft3N3dtWzZMu3Zs0fbtm3TG2+8oSeffFLh4eHy8fG57tjVs2fPysfHpyDf5nVyXbLbuHFjPf/881q9erU+/fRT9enTR8HBwQWRDQAAAADuCIcPH9a8efPUq1cvffPNN+rfv7/9jLu7du2S1WqVJHl4eNh3Eua03YEDB9SqVStVr15dffr0UY8ePbRnzx5Vq1btupMpnTp1Sq1atdLevXv/MnZByXUP6YgRI7R8+XJt2rRJnp6eeuaZZxQREVEQ2QAAAACgSEpLS7MfCunu7i5vb2+9/PLLaty4sYYOHar+/furRIkSKlWqlOrVq6djx45Jkpo0aaK3335bmZmZOW7XoUMHtWjRQu3atVOJEiVUrFgxjR07Vl5eXpo3b56mTJmihQsXKisrS4MHD1bdunUlSc2bN1e3bt00d+5c+yVpnC3XQurh4aHu3bure/fuunDhgu666y65ubkVRDYAwB3KlpUpPz9zy4cAAHCmKlWqaP/+/Tk+3qVLF3Xp0uWGj13rZn/e9kYGDBigAQMG/OX+mjVraunSpTd8zttvv32z2E6RYyFNSUnR66+/ro4dO6p+/fp6+eWXFRcXp0qVKmnRokUKDAwsyJwAgDuIu6dF22f0ztcYdUcsdFAaAADgLDkeQzp9+nSVLFlS9957rxISErRt2zZt2rRJ48aN0/Tp0wsyIwAAAACgCMpxD+nOnTsVExMjNzc3ff3113rmmWdUsWJFVaxYkUIKAAAAAMi3HPeQenh42I8V/fHHH1W/fn37Yzc65TAAAAAAALcixz2k7u7uSk5OVmpqqg4ePKgGDRpIkhITE2WxWAosIAAAAACgaMqxkHbt2lVt2rRRdna2WrRoIT8/P23evFkzZ85U165dCzIjAAAAAKAIyrGQtm3bVvfdd5+SkpLUqFEjSdL58+fVu3dvtWnTpsACAgAAAICryMi0ysviYWTcCRMmaMeOHcrMzNSxY8dUvXp1SVcvBdOuXbtcXyM8PFzr1q1zSF5Huel1SB966KHrbuflTQIAAABAUeVl8VDnEcsdPu4nM258PdE/Gz9+vCTpxIkT6t69+y2XS1cro1IuhRQAAAAA4NqaNGmihx9+WPv379cnn3yijz/+WNu2bdPFixfl7++vqKgolS9fXvfff78OHjyouXPnKjExUUePHtXJkyfVoUMH9evXz0j2HM+yCwAAAAAoHBo1aqQvvvhCKSkp+u2337Ry5Up98cUXqlixomJiYv6y/cGDB7Vo0SJFR0drwYIFunTpkoHUeSikiYmJf7nv119/dUoYAAAAAMCtq127tiQpMDBQI0eOVHR0tKZNm6adO3cqNTX1L9s3aNBAXl5e8vX1VZkyZZScnFzQkSXdpJBeuHBBFy5c0AsvvKCLFy/ab589e1YDBgwoyIwAAAAAgJvw9vaWJO3du1fPP/+8bDabmjVrpqZNmyo7OzvH7SXJzc3thtsUhByPIR02bJi2bNkiSfZrkEqSp6enmjVr5vxkAAAAAIBb8v3336t+/fqKjIzU+fPnFR8fr5CQENOxcpRjIV20aJEkadSoUZo6dWqBBQIAAAAA3J6WLVtqwIABCgsLkyTVqlVLJ06cMJwqZ7meZXfq1Kk6efKkLl68eN1u3L///e9ODQYAAAAAriYj05qnS7Tczrh5vb5plSpVtHnzZvvtP/93QECAoqOjb/i8gwcPSpIGDhx43f1/fn5By7WQzpkzR4sWLZKvr6/9Pjc3N23atMmpwQAAAADA1eS1NLrKuK4u10L62Wef6csvv1RAQEBB5AEAAAAA3CFyvexLxYoVKaMAAAAAAIfLdQ9pUFCQZsyYoaefflrFihWz388xpAAAAACA/Mi1kK5Zs0aSFBcXZ7+PY0gBAAAAAPmVayE1ecYlAAAAAEDRlesxpJcvX9bEiRP13HPP6cKFC3rttdd0+fLlgsgGAAAAAC7FlpVZqMZ1dbnuIZ08ebL8/f117tw5eXt7KyUlRa+99ppmzpxZEPkAAAAAwGW4e1q0fUZvh49bd8TCXLeZMGGCduzYoczMTB07dkzVq1eXJHXv3l3t2rXL9fnJycl69dVX9e677+Y7r6PkWkj379+vqVOnKiEhQcWLF9dbb72lVq1aFUQ2AAAAAMD/N378eEnSiRMn1L17d61bt+6Wnn/x4kXt37/fGdFuW65Ldt3dr9/EarX+5T4AAAAAQME7evSoevbsqTZt2igyMlL79u2TJK1fv17h4eFq27atBg0apPT0dE2ePFlnzpxR//79Daf+r1ybZb169fTmm28qLS1N//73vzVgwAA1aNAgzy8wffp0vfrqq5KkrVu3KiwsTCEhIYqKirr91AAAAAAAjRw5UsOHD9fatWs1adIkDR06VJI0a9YsLV68WGvWrFHlypX122+/aezYsfL393epJbu5FtJXXnlFJUqUkI+Pj6KiolSzZk2NGDEiT4Nv27ZNa9eulSSlpaVp9OjRmjdvnjZs2KC9e/cqISEhf+kBAAAA4A51+fJl7d27V6NGjVJ4eLiGDRum1NRUnT9/Xk899ZQiIyM1Y8YMNWvWTA888IDpuDeU6zGkFotF9evXV//+/XXhwgX98MMP8vb2znXgCxcuKCoqSn379tWBAwe0e/duBQYGqmrVqpKksLAwxcXFKTg4OP/vAgAAAADuMDabTV5eXtcdS3r69GmVKVNGY8eO1YEDB5SQkKDhw4drwIABqlu3rsG0N5ZrIY2KitKOHTu0dOlSpaWlacGCBfr555/10ksv3fR5r732moYOHapTp05Jks6cOSM/Pz/74/7+/kpMTLzlwL6+pW75ObjKz8/HdIQijzl2LubXuZhf52OOnYv5dS7m1/mYY+cqivPr4+Oju+++W+vWrVN4eLi2bNmi1157TXFxcQoNDdXSpUvVp08fZWZmav/+/WrQoIGysrJMx75OroV006ZN9mW3FSpU0LJly9S2bdubFtLo6GhVrFhRQUFBWrNmjaSr7d3Nzc2+TXZ29nW38+rcuRTZbNk5Pl4UP2iOkpSUnO8xmN+bc8QcuxJX+/fmM+xczK/z5XeOmd+b4zPsXMyv8/Ed4Vy5zW9e58+WlZmnS7TcKltWptw9Lbf8vDfffFOvv/66Fi5cKIvFoqioKFksFg0aNEi9evWSt7e3fH19NW3aNJUuXVqVKlVSt27dtHTpUoe/h9uRayHNzMyUxfLfibFYLLkWyQ0bNigpKUnh4eG6ePGiUlNTdfLkSXl4eNi3SUpKkr+/fz6iAwAAAEDBup3S6Ohxq1Spos2bN0uSqlevfsNy2apVqxternPlypW3H9IJci2kjz76qIYNG6b27dvLzc1Nn332mWrXrn3T5yxZssT+32vWrNF3332nCRMmKCQkREePHlWVKlUUGxubp4u3AgAAAACKplwL6bhx4zRnzhxNnTpVnp6eCgoK0oABA275hby9vTVt2jQNHDhQ6enpCg4OVvPmzW8rNAAAAACg8Mu1kL733nv264jejrZt26pt27aSpKCgIMXExNz2WAAAAACAoiPX65DGx8cXQAwAAAAAwJ0m1z2kVapUUa9evfToo4+qZMmS9vt79uzp1GAAAAAAgKIt10JapkwZSdLJkyednQUAAAAAcAfJtZBOnTpVknTp0iWVLl3a6YEAAAAAwFVlZGXKywmXfsnLuBMmTNCOHTuUmZmpY8eOqXr16pKk7t275+kKJuHh4Vq3bp1D8t7I8ePH9d577+mNN97I83NyLaSHDx9W//79lZycrNWrV6tHjx5655137G8eAAAAAO4UXp4W9Vgy2OHjfthzdq7bjB8/XpJ04sQJde/e/ZbLpTPLqCT9/vvvOn78+C09J9eTGk2aNEljxoyRr6+vAgIC1LVrV7322mu3HRIAAAAA4DhNmjTRkCFD1KxZM507d05RUVHq2LGjmjVrpm7duuns2bOSpPvvv1+SNHfuXI0dO1bdunVTkyZN9N57791w3LNnz6pPnz4KCwtTmzZt9PXXX9ufP2rUKHXs2FHPPPOMFi5cKEmaPHmy9u7dqwkTJuQ5e66F9MKFC3r88cftt7t06aKUlJQ8vwAAAAAAwLkaNWqkL774QikpKfrtt9+0cuVKffHFF6pYseINL7158OBBLVq0SNHR0VqwYIEuXbr0l20mTZqkhg0bav369ZozZ45Gjx5tL7d79+7VkiVLtGbNGv3jH//QTz/9pLFjx6pWrVr2Pbl5kWshlaT09HS5ublJkpKSkmSz2fL8AgAAAAAA56pdu7YkKTAwUCNHjlR0dLSmTZumnTt3KjU19S/bN2jQQF5eXvL19VWZMmWUnJz8l23+85//qH379pKkqlWrqnbt2tq1a5ckqVWrVipZsqR8fHzUpEkT/ec//7mt3LkW0s6dO+v555/XuXPnNHPmTHXq1EmRkZG39WIAAAAAAMfz9vaWdHXP5fPPPy+bzaZmzZqpadOmys7OznF7SXJzc1N2drbGjBmj8PBwhYeHa8+ePX95XnZ2tqxWqyTJw8PDfr/NZrvu9q3ItZC2b99egwcPVlhYmLKysjRp0iR17tz5tl4MAAAAAOA833//verXr6/IyEjdfffdio+Pt5fI3EyZMkXr1q3TunXr9NBDD6lhw4ZavXq1pKtn0N2xY4ceeeQRSdK//vUvZWRk6OLFi/rqq6/0xBNPyMPDQ1lZWbeU96Zn2f3555915MgR1a5dW8OHD7+lgQEAAAAABatly5YaMGCAwsLCJEm1atXSiRMnbmusMWPG6LXXXtOaNWskXT1pkb+/v6Sre1g7d+6slJQU9enTR/fee6/Onz+v5ORkDR8+XG+++WaeXiPHQvrpp59q+vTpCgwM1LFjxzRz5kw98cQTt/VGAAAAAKAoyMjKzNMlWm5n3Lxe37RKlSravHmz/faf/zsgIEDR0dE3fN7BgwclSQMHDrzu/j8//88CAgL0/vvv3/Cxxx577C/jlC1bVrGxsbm/gT/JsZAuXbpU69evV0BAgH788UdFRUVRSAEAAADc0fJaGl1lXFd30yW7AQEBkqQ6dero/PnzBRIIAAAAAOC6/nfPaH7keFKja5d5ueZ2z5oEAAAAAMCN5Ok6pNJfCyoAAAAAAPmR45LdgwcP6tFHH7XfTktL06OPPqrs7Gy5ublpx44dBRIQAAAAAFA05VhIN27cWJA5AAAAAAB3mBwLaeXKlQsyBwAAAAC4PGtGpjy8HH9GXGeN6+puepZdAAAAAMB/eXhZtKF7T4eP2/LjJbluM2HCBO3YsUOZmZk6duyYqlevLknq3r272rVrl+vzk5OT9eqrr+rdd9/Nd15J6tatm5YuXZqvMSikAAAAAFAIjB8/XpJ04sQJde/eXevWrbul51+8eFH79+93WJ7vvvsu32Pk+Sy7AAAAAADXcvToUfXs2VNt2rRRZGSk9u3bJ0lav369wsPD1bZtWw0aNEjp6emaPHmyzpw5o/79+/9lHJvNpsmTJys0NFStWrXSggULJEnffvutnnvuOT3//PNq1qyZhg8froyMDE2ePFmS1KFDh3zlp5ACAAAAQCE1cuRIDR8+XGvXrtWkSZM0dOhQSdKsWbO0ePFirVmzRpUrV9Zvv/2msWPHyt/f/4ZLdlesWKFTp04pJiZG0dHR+vLLLxUfHy9J+vHHHzVmzBjFxcUpPT1dy5cv19ixYyVJ0dHR+crPkl0AAAAAKIQuX76svXv3atSoUfb7UlNTdf78eT311FOKjIxU06ZN1axZMz3wwAM6ceJEjmN9++23atOmjTw8PFS8eHGFhYVp27ZtatKkierVq6d77rlHkhQeHq5Vq1apZ0/HHEdLIQUAAACAQshms8nLy+u6Y0lPnz6tMmXKaOzYsTpw4IASEhI0fPhwDRgwQHXr1rVvt2LFCq1cuVKSFBERIZvNdt3Y2dnZslqtkiQPD4/r7v/z7fxiyS4AAAAAFEI+Pj66++677YV0y5Yt6tKli7KyshQSEqKyZcuqT58+Cg8P1/79++Xp6amsrCxJUmRkpNatW6d169YpMjJSDRs21GeffSar1aorV65o/fr1atCggSRp+/btSkxMlM1m02effaZGjRpJulpUr413u9hDCgAAAAB5ZM3IzNMlWm5n3Nu5Dumbb76p119/XQsXLpTFYlFUVJQsFosGDRqkXr16ydvbW76+vpo2bZpKly6tSpUq3fByLZ06ddKRI0cUHh6uzMxMhYWF6ZlnntG3334rf39/jRgxQomJiXr88cftJzJ6+umnFR4erjVr1sjb2/u23jeFFAAAAADy6HZKo6PHrVKlijZv3ixJql69+g2vBdqqVSu1atXqL/dfW6b7vywWi/1ERf+rfPny+uijj/5y/9y5c/OcOScs2QUAAAAAGMEeUgAAAADADTVo0MB+LKkzsIcUAAAAAGAEhRQAAAAAYASFFAAAAABgBIUUAAAAAGAEhRQAAAAAYASFFAAAAABgBIUUAAAAAGAEhRQAAAAAYASFFAAAAABgBIUUAAAAAGAEhRQAAAAAYASFFAAAAABgBIUUAAAAAGAEhRQAAAAAYASFFAAAAABgBIUUAAAAAGAEhRQAAAAAYIRTC+ns2bPVsmVLhYaGasmSJZKkrVu3KiwsTCEhIYqKinLmywMAAAAAXJinswb+7rvv9J///EcxMTHKyspSy5YtFRQUpNGjR2vp0qWqWLGi+vTpo4SEBAUHBzsrBgAAAADARTltD2n9+vX18ccfy9PTU+fOnZPVatWlS5cUGBioqlWrytPTU2FhYYqLi3NWBAAAAACAC3PaHlJJslgsmjNnjhYvXqzmzZvrzJkz8vPzsz/u7++vxMTEWxrT17eUo2PeMfz8fExHKPKYY+difp2L+XU+5ti5mF/nYn6djzl2LubXNTm1kErSoEGD9MILL6hv3746cuSI3Nzc7I9lZ2dfdzsvzp1Lkc2WnePjfNBylpSUnO8xmN+bc8QcuxJX+/fmM+xczK/z5XeOmd+b4zPsXMyv8/Ed4Vy5zS/zZ4bTluweOnRI+/fvlyQVL15cISEh+vbbb5WUlGTfJikpSf7+/s6KAAAAAABwYU4rpCdOnNDYsWOVkZGhjIwMbdq0SRERETp8+LCOHj0qq9Wq2NhYNWrUyFkRAAAAAAAuzGlLdoODg7V79261bt1aHh4eCgkJUWhoqMqVK6eBAwcqPT1dwcHBat68ubMiAAAAAABcmFOPIR04cKAGDhx43X1BQUGKiYlx5ssCAAAAAAoBpy3ZBQAAAADgZiikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAIyikAAAAAAAjKKQAAAAAACMopAAAAAAAI5xaSN955x2FhoYqNDRUM2bMkCRt3bpVYWFhCgkJUVRUlDNfHgAAAADgwpxWSLdu3apvvvlGa9eu1WeffaaffvpJsbGxGj16tObNm6cNGzZo7969SkhIcFYEAAAAAIALc1oh9fPz06uvviovLy9ZLBZVr15dR44cUWBgoKpWrSpPT0+FhYUpLi7OWREAAAAAAC7M01kD33ffffb/PnLkiD7//HN17dpVfn5+9vv9/f2VmJh4S+P6+pZyWMY7jZ+fj+kIRR5z7FzMr3Mxv87HHDsX8+tczK/zMcfOxfy6JqcV0mt++eUX9enTRyNGjJCHh4eOHDlifyw7O1tubm63NN65cymy2bJzfJwPWs6SkpLzPQbze3OOmGNX4mr/3nyGnYv5db78zjHze3N8hp2L+XU+viOcK7f5Zf7McOpJjbZv364ePXpo2LBhatOmjSpUqKCkpCT740lJSfL393dmBAAAAACAi3JaIT116pT69++vt956S6GhoZKk2rVr6/Dhwzp69KisVqtiY2PVqFEjZ0UAAAAAALgwpy3ZXbRokdLT0zVt2jT7fREREZo2bZoGDhyo9PR0BQcHq3nz5s6KAAAAAABwYU4rpGPHjtXYsWNv+FhMTIyzXhYAAAAAUEg49RhSAAAAAAByQiEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAYQSEFAAAAABhBIQUAAAAAGEEhBQAAAAAY4dRCmpKSolatWunEiROSpK1btyosLEwhISGKiopy5ksDAAAAAFyc0wrprl27FBkZqSNHjkiS0tLSNHr0aM2bN08bNmzQ3r17lZCQ4KyXBwAAAAC4OKcV0lWrVmn8+PHy9/eXJO3evVuBgYGqWrWqPD09FRYWpri4OGe9PAAAAADAxXk6a+ApU6Zcd/vMmTPy8/Oz3/b391diYuItj+vrWyrf2e5Ufn4+piMUecyxczG/zsX8Oh9z7FzMr3Mxv87HHDsX8+uanFZI/5fNZpObm5v9dnZ29nW38+rcuRTZbNk5Ps4HLWdJScn5HoP5vTlHzLErcbV/bz7DzsX8Ol9+55j5vTk+w87F/Dof3xHOldv8Mn9mFNhZditUqKCkpCT77aSkJPtyXgAAAADAnafACmnt2rV1+PBhHT16VFarVbGxsWrUqFFBvTwAAAAAwMUU2JJdb29vTZs2TQMHDlR6erqCg4PVvHnzgnp5AAAAAICLcXoh3bx5s/2/g4KCFBMT4+yXBAAAAAAUAgW2ZBcAAAAAgD+jkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjDBSSNevX6+WLVsqJCREy5cvNxEBAAAAAGCYZ0G/YGJioqKiorRmzRp5eXkpIiJCDRo00L333lvQUQAAAAAABhX4HtKtW7eqYcOGKlOmjEqUKKFmzZopLi6uoGMAAAAAAAxzy87Ozi7IF3z//feVmpqqoUOHSpKio6O1e/duTZo0ySHjZ2Ra5WXxyNcY1swMeVi88p8lK0Nenvkfx5qRIQ+v/I1jzciUh5cl31kcMb+SY+bYleb36jj5n2NXml/JMXPsSvMr8R2R8xiuM78S3xE5caX5lfiOyDGLC82vxHdEjllcaH6vjsN3xA2zuND8wjkKfMmuzWaTm5ub/XZ2dvZ1t3Nz7lyKbLaC6NDpRXCcNAeM4UiOeE+uNL9SfufYz89HnUfk/7jqZW90zPcYkuRuy1ZSUrIDRnKN+XUsV/vs8R3hvDEcOY4rzbErzQ3zWzjGcaX5lVzrs8dn2LnjFMz8+vn5OOh1cCsKvJBWqFBBP/zwg/12UlKS/P39CzoG4JLS0jP1yYwuDhkn+bwjiiQAAADgPAVeSP/v//5Pc+fO1R9//KHixYvryy+/dNhyXaCwS76UpmSX+usoAAAA4DwFXkgDAgI0dOhQde/eXZmZmWrfvr0efvjhgo4BAAAAADCswE9qlF8FdwwpAAAAgDsFx5CaUeCXfQEAAAAAQKKQAgAAAAAMoZACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACM8DQd4Fa5u7uZjgAAAAAAcAC37OzsbNMhAAAAAAB3HpbsAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJACAAAAAIygkAIAAAAAjKCQAgAAAACMoJAWkOzsbNMRALio48ePa/fu3aZjAE7Fz0G4skuXLpmOcEewWq2mI8AFUUid6MqVK9q+fbtSU1Pl5uZmOk6RlZqaajrCHWH9+vXavXu3jh8/LolfLh3hypUrmj59uvr376/Lly+bjlPkXbhwQbNnz9aGDRt07tw5SXyOnSkxMVFRUVH64osvlJKSws/BfLpw4YKmTp2qVatWKTExURK/3DvCte/hN954QykpKabjFFkXLlzQxIkT9c4772jnzp2m48DFuGXz09gpVq5cqQULFqhSpUry8PDQwoULZbFYTMcqUq5cuaKZM2fq999/V+XKlRUUFKQmTZqYjlXkxMfHa968eSpbtqzKlCkji8WiCRMmyMPDw3S0Qm3FihV69913Va5cOcXExJiOU+R9+OGHiomJUc2aNZWcnKwjR45o/fr1pmMVWe+99542btyoWrVq6eTJk7py5Yo++eQT07EKrQ8//FDr1q1TzZo1denSJZ07d04rV640HavQW7FihRYvXqymTZtqwIABKlmypOlIRdKiRYv0xRdf6LHHHlPJkiUVGxur9evXy9PT03Q0uAg+CQ7273//Wx999JEsFovmz5+v6tWr6+9//7t27NihBg0amI5XZCxdulQbNmzQQw89pL59++qrr77SqFGjNHPmTDVs2JAvOQc5fvy4PvroI7300ktq3LixMjMz9ccff9jLaHZ2Nns9btH58+c1btw4ZWdna/DgwVqyZIlSUlJUqlQpWa1Wir6DHTp0SAMGDFCtWrU0Z84cValSRZJUr149/etf/1LTpk0NJyxa9uzZo5dfflkNGzbUggULVL58eV25ckUNGjTQDz/8oMcee8x0xEJlx44dGj16tOrUqaO3335b1apVU1xcnD7//HOlpqaqePHifAffhn379mnixImy2WxKTU3VsGHD5OnpKZvNJnd3Fg86SkZGhmbNmqUVK1Zo06ZNKleunCTp66+/1vHjx1WtWjVJYt5BIXWkAwcO6IUXXtAbb7yhtm3bSrr6C33nzp117733Gk5XNGRlZSkmJkZTpkxRQkKCAgICJEkPP/ywUlNTFR0drUqVKumee+4xnLRo2Lt3r+655x41btxYVqtVCQkJSk1NVVZWlh5//HH7/CN3J0+e1L59+3T33XfrhRdeUO3atSVd/YVz4sSJmjFjBj+QnaBs2bL6448/9Pzzz9vLqCQ98cQTrFpxgtKlS+vSpUvq16+fypcvL0kqXry4nnzyScPJCpezZ8/qp59+0i+//KIyZcpo6tSpkqQjR47oyy+/VFBQkEqUKGE4ZeFz/PhxLVmyRDVr1tSLL76oJk2aaOTIkZoyZYrGjx/Pd7CDXPsjq5eXlxo3bqyDBw/Kzc1Np0+f1vDhw3X27FlNmjRJtWrVUr9+/VS8eHHTkWEYhdRBsrOzVbNmTYWGhurUqVOSpGnTpmnt2rWqXLmyzp8/L19fX0liL8htOH78uObOnauyZcsqIiJCjz76qA4ePKiAgAClpqaqRIkS6tOnj3r37q1ff/1V99xzD3vvbsOBAwd0zz33yMvLS5JUuXJlzZkzR3379tXevXv1wAMP6Pz587JarVq+fLk+/fRTw4ldn9Vq1TvvvKPPP/9ctWrV0tGjR1WtWjXdd999KlGihF566SX16tVLP/74o+rUqcP3gwNZrVaVK1dOPXv21IwZM7R48WKlpaUpKipKSUlJqlmzpumIRYrNZlNgYKDatm2radOmac6cObp06ZLeeustpaWlqXr16qYjFhqzZs3Svffeq2bNmun06dNasGCBLl68qLi4OJ07d06lSpXS5s2b9cQTT8jLy4s9TLm49j2ckJCgJk2aqGPHjvbH+vfvrx49emj37t16+OGHDaYs/FJTUzVz5kwdPnxYAQEBatq0qZ5++ml99913atu2rcqXL682bdqoc+fOiomJUVxcnGJjY9WhQwfT0WEY3175sHXrVm3evFnS1R/EkvTSSy9p5cqVCgkJkaenp0aPHq1HH31UgwYN0ogRI5ScnGwycqFjtVo1e/ZsDRo0SHfffbdOnjyp8ePHKyIiQm+++aYkqUSJEsrMzFS5cuXUsGFDrVu3TpIoo7dowYIFateunaKiouz3Pfzww5oxY4aeeOIJTZw4UUOHDtX8+fO1du1aJSYmcmbYXMTHxys4OFg2m02rVq3SW2+9pXfffVe7du3Sxx9/rIsXL6pq1apq27at3n77bUmijObD6tWr7ceFZmVl2X9B79u3r86ePavhw4erW7dustlsev/999nDn09Lly7Vpk2bJF2d72vfuUOHDtXBgwc1cuRI9erVS6VLl9b8+fNVtmxZk3ELhaysLEnSAw88oG+++UZVq1bVww8/rBUrVuj06dPatGmT1q5dq3vuuUfjx49X37599e2339qfh7+69j2ckZGh+fPna8CAAZKunnQrKytLf/vb39SqVSvNmzePk5zlQ3x8vDp27Ch3d3eNGTNGtWvX1s8//yxJatGihQICAhQREaHOnTtLkp599ll5eHhw3C4kUUhvm81m0y+//KLFixcrMzNTHh4estlsql69ujp16qRSpUrplVdeUXh4uMaOHatx48bp5MmTat++vb788kvT8QuFaz9E0tPTNX/+fL300kvq37+/PDw89OSTT8rPz0/vvfeepP/+QaBcuXL25dH8YLk1FSpUUGRkpPbt26fhw4dr3759kqSHHnpI7dq1U5MmTfTggw+qfPny+vnnn/Xkk0+qRo0ahlO7titXrigjI0NDhw5V6dKldfnyZfn7+2vw4MHatm2bTp8+LUnq1q2b0tPTtWrVKkn//Twjbw4cOKBRo0bp4MGDGjNmjCTZjyO/dhbSYcOG6euvv9bIkSM1ZswYlSxZkjOU3qZrZ8i0WCwaMWKEpOvn28vLS4MGDdLXX3+tN954Q6+88oosFgvznYNDhw7ZT2x27Y8oDz30kCpUqKDk5GQ99thjatasme677z5J0t13360ePXpo8eLFKlWqlD777DPm9iauXLmi9PR0DR8+XP7+/vr+++/VqVMnrV692v5dO2TIEJ04cUJxcXGG0xY+SUlJkq7+zvbKK69ozJgxql69uiIiItSvXz+dOnVKpUuXVocOHfT555/bn7dz5079/vvv9qX9uLNRSG+Tu7u7QkNDVbFiRX388cfXPda5c2dlZWVpw4YNkq7+gA4KCtLChQs1bdo0tWjRwkTkQufaD5ERI0YoICBA33//vebOnat69eqpbNmyevHFFxUdHa3U1FR5e3vrp59+Unx8vP3YPPaQ3hp3d3cdPHhQs2fPVvHixTVx4kR9++23kqTNmzfrueee07Zt2zRt2jQNGTJEDz74oIoVK2Y4tWtr0aKFGjZsaD/+y9vbW5LUsmVLpaWl6ZtvvpEklSpVShEREX/5pRR5k5SUpLVr16pGjRqqUKGCffWEzWaz73EODg5WrVq19OOPP/7lMeTd1q1bFRERoU8++UQRERGqWrWq3n33XUnXH44SGhqqwMBA5vsmzpw5o507d2rLli2aMGGCNm3aZL+MWUZGhk6ePCkfHx9VqlRJ9erV06+//qqtW7fKzc1NNptN9913n+bMmaOpU6dyDN5NtGjRQkFBQRozZowmT56st956S926dVP//v2vW+7cs2dPTZ8+nUtw3YINGzbo7bff1h9//KFvvvlGZcqUsT929OhRDRs2TOHh4fruu+/02GOPKSAgQFOmTNHrr7+uadOmqV+/fqpfv765NwCXwW89+eDr66sWLVooPj5eJ0+elLu7uzIyMlSuXDkNGDBA8+fPlyT73tPixYurTp06hlMXHtd+iIwdO1aTJk3SuHHjdOzYMW3ZskXR0dGqWrWqnn32Wb3yyiuaOnWqXn/9dXXq1IlLv+Tiz0vN//xX9caNG+vixYuy2WyaOHGiMjMzNWrUKK1YsUKhoaF6+OGH9fnnnys5OVnLli1Tt27dTL2FQqVfv37auHGjDh8+LE9PT/t17p588knt37/fvl3btm21bNkyUzELlezs7OtWQHh7e6tatWo6c+aMOnbsqMWLF+vMmTPy8PBQRkaGfdvhw4frnXfe0aFDhyj9t8nHx0d/+9vfdODAAR07dkzTpk3TvHnzlJqaKk9PT2VkZNi3ffnllzVt2jSdPn2a+f4Tq9WqWbNm6bnnntPcuXN18eJF9e3bV59//rlmz54tSXrsscd0+fJlxcfHS5Jq166te++9V6tXr1ZmZqZ9PlkJlDcvvfSSNmzYIKvVqn/84x9q1aqV/bFrc9muXTsu/XKLtm3bpvvvv19ubm6qWLHidcvy586dq9q1a2v48OFauXKl3N3d1ahRI8XGxqpSpUpauXIlZzmHHT8h8sHNzU0NGjRQzZo1tWjRIkmynwzGw8NDv//+uxISEiSxx+N2vfTSS/rnP/8pq9WquLg4ffrpp+rQoYNiYmLsyxy3bNkiHx8fRUdHKywszHRkl3ajpebXfqG5cOGC7rvvPq1cuVIREREqXbq0QkNDtXz5cg0aNEjDhg3T+PHjNWXKFPup25G7Bx54QM2aNdP06dMlXd0bKkmnT59W8+bNTUYrlP71r3+pbdu29mPFJalOnTry9/fXgQMH5Ovrq0ceeURjx46VdPU72c3NTcuWLZOfn59mzpypChUqmIpf6CQkJOj48eP222XLlpWXl5eysrK0efNm1axZU02bNtXIkSMl/fdn4OLFi/X3v/+d74v/kZGRoQkTJujs2bOKjo7W+++/r8cff1zPPfec+vTpo8OHD2vy5MlKTExUSEiI/XeH8uXLq06dOvq///u/60ooK4HypmbNmoqIiNDFixevuz8uLk6DBg3S3r17JUnt27c3Ea/Q2LBhg1asWKE//vhD0tXfKR588EGVLVtW58+fV2xsrH3bSZMmqXv37mrSpIk8PT1VsmRJNWnSRJs2bdKLL75o6i3ARdGS8qlkyZIKCwvTkSNHdPToUf32228KDQ3V/PnztXjxYgUHB5uOWKhd+yFy6dIlSVePWWrdurUWL16sJk2ayGKxaOPGjfaTFODmbrTU/NovN1WqVNGpU6e0ePFi9e7dW0uWLNGwYcP07rvvqlmzZpI44c7t6t27t06ePKndu3fr9OnT6tevnxITE/XAAw+YjlboZGVlaf/+/YqKitK2bdt09uxZWSwWhYSEqF69etqxY4c6dOigrVu36tChQzpw4IBatGihjz/+WBcuXFDTpk3ZA5JHH374ofr06aNx48YpNTVVNptNVapU0TPPPCMPDw/9/PPP2rlzp6ZMmaL4+HglJibqhx9+0NNPP621a9fq4sWLatmypb2kQrp06ZJ+/vlnTZ48WaVKldLJkyfl5eWlPXv26G9/+5veeustZWRk6IMPPlBcXJx9RYUk1a9fX+3bt2c+b1Pv3r31yy+/aOfOnUpMTFS/fv20cuVKde/eXbVq1TIdz+XZbDYlJyfriy++UI8ePbR+/Xr99NNP9j+KDBkyRCtWrNC///1vpaSkqHjx4tq+fbsGDBigunXr6q677pLFYuFyRbghLvviAA888IDq16+vZs2a6a677lKfPn3Uq1cv07GKjN69e+u5555TQkKC/Ux5Xl5e9j0guDXXlpovWbJEzZs3V+XKle2XzgkJCVGlSpXsy2gyMjIUGBiowMBAw6kLN19fX0VERKhjx456+OGH1bFjR/4Sf5tCQkLUpUsXxcfH6+DBg4qPj9eoUaN09OhRtW3bVsWLF9fZs2fVvHlzhYaGqly5cho4cKAiIyNNRy8UTp48qV27dunBBx9U8+bNtW3bNm3fvl0rVqxQtWrV1KRJE5UuXVqNGjXSl19+qY0bN+qRRx5R3759FRwcrAoVKmjIkCFq3bq16bfiksqXLy83NzcNHjxYV65cUUpKiry9vbVz5041bNhQ/fr108SJExUTE6Po6Gh9//33atmypST2hubXte/hiIgIPfTQQ+rUqRPfw7fA3d1dnTp1UqdOnRQTE6OffvpJBw4c0D//+U8FBgaqadOm9j9q22w2BQQE6NChQ+ratavatGljOj5cnFs2ByA4xPHjx7Vx40Z16dLFfuISOM7y5cu1cOFCffXVV6ajFAmXL1/WrFmzZLVa9dprr9nvf/XVV1W6dGkNGTKEv2I6WEZGhqKjo9W+fXu+I/Lp4MGDGjBggN544w0tW7ZM9evX16VLl5ScnKyBAwfqlVdeUdOmTZWamqouXbqYjlso/O/1co8cOaIHH3xQzZo107hx4/TKK69oxYoVGjNmjDZt2qS77rpLTz/9tKZOnaqQkBC1atVKa9asUdu2bU2/FZd3+PBhrVixQmlpaXr66adVsmRJVapUSYsWLVLVqlXVo0cPSVePzwsKCjIbtojhe9hxLl++rJ49e6pEiRJKTU1VjRo11LdvX3l7e+vixYv69ddfOSwFecYeUgepWrUqe0Wd6NpFk202m9zc3PhLcT5dW2o+a9YsHT16VFlZWRo+fLguX76smTNnUkadwMvLi3LkIPfff7+aNGmiNWvW6I033tDbb7+tM2fOyMvLS8WLF1evXr1Uo0YN+fj4mI5aKMTHx2vs2LFq166dVq1apdKlS+vMmTPq2rWrAgICVKZMGSUnJ6tPnz764IMPdP/992vnzp3q2rWratSoYb8GJmU0b6pVq6bRo0f/5f7U1NTrlo5SRh2P72HHOX78uK5cuaJVq1bp+PHjmjFjhnr37q1+/fopPDzcfgk+IC8opCgU+CHieCw1R2H24osvqlu3btq3b5/GjRunKVOm6LfffpPValXdunVNxytU/ny9XEn26+UOGTJEn376qSIiIvT+++9r3bp18vDw0Pvvv6/ExERJUp8+fWSxWEzGL5T27t2rjRs3ql69eipXrpxmzpwpd3d3Va1a1XQ0IFfZ2dmy2WyqXLmyrly5oqpVqyoqKkqXLl3iJGa4LSzZBe5gLDVHYbZs2TItWrSIpfwOMGjQIFWsWFGjRo1SVlaWPD2v/r26ffv26tmzpw4dOqRjx47prbfesh9neu3YRty6K1eu6MMPP9SePXt08uRJRUZGKiIiwnQsIM+unXF36dKlpqOgCKCQAgAKpWvHg107YRGX17p9+/fvV//+/bVo0SJVq1ZNKSkpKlWqlObMmaPff/9dffv2Ve/evfXRRx+pcuXKpuMWGSkpKSpWrJj9DwBAYbFixQq5ubkpIiJC2dnZHEqFfKGQAgAATZ8+XYcPH9b8+fPt940ZM0aNGzfWM888Yy+pAGCz2fgjIByGTxIAALjh9XLPnDmjBx98UJIoowDsKKNwJNaIAAAArpcLADCCQgoAACT99xJbXKcRAFBQOIYUAAAAAGAEC8ABAAAAAEZQSAEAAAAARlBIAQAAAABGUEgBAAAAAEZQSAEAAAAARlBIAQAu68SJE7r//vsVHR193f2LFi3Sq6++aigVAABwFAopAMClubu7a/r06frtt99MRwEAAA7maToAAAA3U6xYMfXs2VOvvPKKVq5cKS8vL/tjhw8f1sSJE3X58mUlJSWpZs2amjVrlry9vfXQQw+pZ8+e2rp1q1JTUzVgwADFxcXp559/lr+/v+bPn68SJUro0KFDmjJlii5cuCCr1apu3bqpffv2Bt8xAAB3DvaQAgBcXr9+/VSiRAlFRUVdd/+qVavUunVrrVq1Sl9++aVOnDih+Ph4SVJGRobKly+v1atXq3Xr1ho7dqzGjBmjDRs2KCUlRZs2bVJWVpYGDRqkYcOGac2aNVq2bJkWL16snTt3FvybBADgDsQeUgCAy3N3d9ebb76p1q1b64knnrDfP3z4cG3ZskUffPCBjhw5ojNnzig1NdX+eLNmzSRJf/vb31SjRg0FBARIkqpUqaKLFy/qyJEjOnbsmEaPHm1/Tlpamvbt26dHHnmkYN4cAAB3MAopAKBQqFixoiZMmKCRI0eqdevWkqSXX35ZVqtVLVq0UOPGjXXq1CllZ2fbn2OxWG7439dYrVb5+Pho3bp19vvOnj0rHx8f570RAABgx5JdAECh0bx5czVq1EgfffSRJOmbb75R//791bJlS0nSrl27ZLVa8zxetWrVVKxYMXshPXXqlFq1aqW9e/c6PjwAAPgL9pACAAqVsWPHavv27ZKkoUOHqn///ipRooRKlSqlevXq6dixY3key8vLS/PmzdOUKVO0cOFCZWVlafDgwapbt66z4gMAgD9xy/7z2iYAAAAAAAoIS3YBAAAAAEZQSAEAAAAARlBIAQAAAABGUEgBAAAAAEZQSAEAAAAARlBIAQAAAABGUEgBAAAAAEb8P1/e4WJytUhVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 926x595.44 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 19.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "final_df,_ = compare_models_final(train_X,train_y,test_X,test_y,models,models_opt,models_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T22:28:26.770025Z",
     "iopub.status.busy": "2021-03-14T22:28:26.769025Z",
     "iopub.status.idle": "2021-03-14T22:28:26.835643Z",
     "shell.execute_reply": "2021-03-14T22:28:26.834730Z",
     "shell.execute_reply.started": "2021-03-14T22:28:26.770025Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Dataset</th>\n",
       "      <th>ScoreChange</th>\n",
       "      <th>Per</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>LR</td>\n",
       "      <td>Train</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>LR</td>\n",
       "      <td>Test</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>RD</td>\n",
       "      <td>Train</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RD</td>\n",
       "      <td>Test</td>\n",
       "      <td>0.01</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>LAS</td>\n",
       "      <td>Train</td>\n",
       "      <td>0.53</td>\n",
       "      <td>53.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>LAS</td>\n",
       "      <td>Test</td>\n",
       "      <td>0.49</td>\n",
       "      <td>49.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>DT</td>\n",
       "      <td>Train</td>\n",
       "      <td>0.01</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>DT</td>\n",
       "      <td>Test</td>\n",
       "      <td>-0.01</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>MLP</td>\n",
       "      <td>Train</td>\n",
       "      <td>0.02</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>MLP</td>\n",
       "      <td>Test</td>\n",
       "      <td>0.08</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>DMLP</td>\n",
       "      <td>Train</td>\n",
       "      <td>0.16</td>\n",
       "      <td>16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>DMLP</td>\n",
       "      <td>Test</td>\n",
       "      <td>0.06</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>SVR</td>\n",
       "      <td>Train</td>\n",
       "      <td>0.03</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>SVR</td>\n",
       "      <td>Test</td>\n",
       "      <td>-0.02</td>\n",
       "      <td>-2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>ET</td>\n",
       "      <td>Train</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>ET</td>\n",
       "      <td>Test</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>XGB</td>\n",
       "      <td>Train</td>\n",
       "      <td>0.02</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>XGB</td>\n",
       "      <td>Test</td>\n",
       "      <td>0.04</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Name Dataset  ScoreChange   Per\n",
       "0     LR   Train         0.00   0.0\n",
       "1     LR    Test         0.00   0.0\n",
       "2     RD   Train         0.00   0.0\n",
       "3     RD    Test         0.01   1.0\n",
       "4    LAS   Train         0.53  53.0\n",
       "5    LAS    Test         0.49  49.0\n",
       "6     DT   Train         0.01   1.0\n",
       "7     DT    Test        -0.01  -1.0\n",
       "8    MLP   Train         0.02   2.0\n",
       "9    MLP    Test         0.08   8.0\n",
       "10  DMLP   Train         0.16  16.0\n",
       "11  DMLP    Test         0.06   6.0\n",
       "12   SVR   Train         0.03   3.0\n",
       "13   SVR    Test        -0.02  -2.0\n",
       "14    ET   Train         0.00   0.0\n",
       "15    ET    Test         0.00   0.0\n",
       "16   XGB   Train         0.02   2.0\n",
       "17   XGB    Test         0.04   4.0"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "change = pd.DataFrame(columns=['Name','Dataset','ScoreChange'])\n",
    "for i in range(0,18):\n",
    "    change = change.append({'Name':final_df.Name[i],'Dataset':final_df.Dataset[i],'ScoreChange':round(final_df.Score[i+18]-final_df.Score[i],2)},ignore_index=True)\n",
    "change['Per'] = 100*change['ScoreChange']\n",
    "change"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Πίνακας μεταβολής επίδοσης των αλγορίθμων μετά τη βελτιστοποίησή τους\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-14T22:03:01.451188Z",
     "iopub.status.busy": "2021-03-14T22:03:01.451188Z",
     "iopub.status.idle": "2021-03-14T22:03:01.873374Z",
     "shell.execute_reply": "2021-03-14T22:03:01.872338Z",
     "shell.execute_reply.started": "2021-03-14T22:03:01.451188Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0.98, 'Score Change after hyperparameter optimization for each Regressor')"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmIAAAF+CAYAAAAhqnabAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABUxUlEQVR4nO3deVxN+f8H8Ff7orKkbGOyVkTKHkY0tKhkLClkNxhkG2OyjMEgWSLLDGNnjAkhhjC2GTJ2ESqVtCBXQvt2P78/fLs/jS10O8Xr+Xh4PNx7zv2c9znn3nNfnc/nnKsihBAgIiIiolKnKnUBRERERJ8qBjEiIiIiiTCIEREREUmEQYyIiIhIIgxiRERERBJhECMiIiKSCIOYkuXn52Pt2rVwcHBAkyZNYGNjg0mTJuHu3btSl/aSI0eOYMCAAWjRogXatGmD4cOH49q1a4rp586dg5mZGTIyMiSssmSFhITgiy++QLNmzXDy5ElERETg33//LbH2P8ZtVhaU9H4qCefPn8etW7cAAImJiTAzM0NUVNR7teXl5YWFCxd+cE1CCAQGBiInJwcAEBQUhDZt2nxwu6/z389TWfY++8jOzg5mZmZF/jVv3hz9+/cvcqwkehcMYkq2dOlSBAYGwsfHByEhIVizZg0yMjLQv39/PHv2TOryFJYvX47p06fD0dERgYGB2LJlC2rUqIEBAwbg4sWLUpenNEuXLsUXX3yBgwcPom3btvjmm28QHR0tdVn0FmVxP3l5eeHBgwcAgBo1auD06dOoV6/ee7W1YsUKjBkz5oNrunDhAmbOnIn8/HwAQLdu3XDw4MEPbvd1/vt5+hhNmjQJp0+fxunTp/HPP/9g06ZN0NDQwIgRI/gHF70XBjEl2717N8aOHYtOnTrhs88+g6WlJZYtW4aMjAz89ddfUpcHALh69Sp+/vlnrFy5EgMGDED9+vVhZmaGuXPnok2bNvD19ZW6RKV59uwZWrRogVq1akFbW1vqcugjoaamBiMjI6irq7/X6ytVqgQ9Pb0PruO/9+vW1taGoaHhB7f7Op/C56lChQowMjKCkZERjI2NYWlpCV9fXzx58qTMnaWl8oFBTMlUVFRw7tw5xV+kAKCjo4O9e/eia9euiue2b98OBwcHNGvWDL179y5yFur06dPo06cPrKys0LlzZ6xbt05xgF2xYgWGDx+OYcOGoUWLFtizZw8AYP369ejcuTOsra3h6emJq1evvrbGoKAgWFpavrLL4ocffsCCBQuKPBccHAw7OztYWlpiyJAhuH//vmLaP//8A3d3d1haWqJZs2YYOHAgYmJiAPx/V0BISAicnJxgbW0NLy8vxMbGKl4fEREBT09PWFpaws3NDRs3boSdnZ1i+p07dzBs2DA0a9YMnTt3hp+fH3Jzc1+7bm+qx8zMDKmpqZg2bRrs7Ozg5eWFpKQkzJ07F15eXgAAmUyG8ePHw9raGh06dMD06dORlpZWZH1Wr16NNm3aYMSIEa+t43XbbMSIEZgyZUqReQsDMfC8K2Tjxo3o27cvLC0t0bdvX9y4cUMxb15eHhYvXoz27dujRYsWGDZsWJHtaWdnBz8/P3Tq1Am2trZ4+vQpzMzMsHPnTri4uMDKygpDhw5FQkKC4jVxcXEYNWoUWrZsiSZNmsDFxQXHjh1TTDczM8OyZcvQrl07dO/eHQUFBcXa78ePH4e9vT0sLS0xatQoPHjwABMnToSVlRW6du2K06dPK5bxpu2urP10/fp1DBw4UNHGokWLkJeXB+B5F3PLli0RFBSEDh06oGXLlpg5cyays7MV2xkARo0ahe+///6lbi87Ozv8/vvv6NevH5o2bYru3bsjKioKK1asQJs2bdC2bVv8+uuvilpe7Jr8b1dY4b+kpCQAwKZNmxRDH1q3bo0pU6YgIyMDiYmJGDhwIACgefPmCAoKeqlrMi4uDqNHj0arVq3Qpk0bzJgxA+np6UW225s+ry/67+cJAB4+fIgpU6bAxsYGzZs3x/jx4/Hw4cN32i9hYWHw8PBA06ZNYW9vj19//RVyuVwxPTg4GK6urmjSpAmaN2+OUaNGQSaTKab/+++/6Nu3L5o1a4YuXbogMDCwSPunT5+Gs7MzmjZtij59+rxXd7KmpiaA5wG80JuOwUIILF++HO3atUOLFi0wf/58eHl5ISgoCMDz/f/jjz/CyckJNjY2iImJQXp6OmbOnInWrVujTZs28Pb2RnJysqLNo0ePwsXFBU2bNoWdnR3WrVunmHbx4kX07t0blpaW6NChA/z8/FBQUKCoZcuWLXBwcEDTpk3h5uaGU6dOKV77qlqohAlSqjVr1ghTU1PRvn174ePjI/bt2ydSUlKKzLNr1y5haWkpdu7cKeLi4oSfn59o3ry5ePz4sTh//rxo1KiRWLVqlYiNjRX79u0T1tbWYtu2bUIIIQICAoSpqalYvXq1iI6OFikpKeL3338XHTt2FCdOnBB37twRP//8s7C0tBTx8fGvrLF3795izpw5b12Xf//9V5iamgp3d3dx/fp1ce3aNdGlSxcxceJEIYQQiYmJwsLCQqxfv17Ex8eLK1euiO7du4uRI0cKIYRISEgQpqamwtnZWZw/f16EhYUJBwcHxfRnz54JGxsb8f3334vo6GixZ88eYWVlJTp37iyEECI7O1t07txZzJo1S8TExIjz588LFxcXMXPmzFfW+7Z6Hj58KFq3bi02bdokUlJSRGpqqujYsaP45ZdfRGpqqhBCiL59+4px48aJyMhIce3aNTFgwAAxbNiwIuvTp08fERsbK6Kiot55mwUHBwtra2uRlZWleI2jo6PYsWOHEEKIzp07i2bNmolt27aJ6OhoMXXqVNG6dWtFfYsXLxYuLi7i3LlzIjo6WsydO1e0a9dOPHv2TPH6li1birCwMHHt2jUhhBCmpqaidevW4s8//xSRkZFi6NChwt7eXuTm5gq5XC4cHBzEpEmTRHR0tIiOjhYTJ04UrVu3Fjk5OYrX29nZiaioKHHz5s1i7/fu3buLsLAwceHCBWFlZSVatWolNm/eLKKjo8W4ceOEra2tYhu8absrYz/FxsaKZs2aidmzZ4vo6Ghx/Phx0b59e+Hr66vYj40aNRLdunUTly5dEufPnxd2dnbi+++/F0IIkZKSIkxNTcXevXvFs2fPFMuMjIxU7IfWrVuLo0ePiujoaNGjRw/RqlUrMXXqVBEdHS1++eUXYWZmJuLi4oQQQgwYMECx7IcPHyr+xcfHC3t7e8X7Z//+/aJ58+bi+PHjIjExUfz111/C2tparF+/XuTn54vDhw8LU1NTER8fL7KyssTu3btF69atFdvRxsZGsd3OnTsnHB0dxbhx44pst9d9Xv/rv5+n3Nxc4eTkJPr37y+uXbsmrl27Jvr27St69+4t5HJ5sfbLo0ePRIsWLcTKlSvFnTt3xMmTJ0WnTp3EmjVrhBBCXLp0SVhYWIg9e/aIxMREERoaKmxtbRXHs+joaGFhYSEWLlwoYmNjxf79+4WFhYU4ffq0Yvn29vbi/PnzIioqSvTp00e4u7u/cv0K9+PWrVuLPPf48WMxevRo0b59e5GWliaEEG89Bq9Zs0a0adNGHDt2TERGRorhw4cLMzMzsXv3bsX+t7CwEH///bcICwsTQggxceJE0b9/fxEWFiYiIyOFt7e3cHFxEXl5eUImkwkLCwsRGBgoEhMTxaFDh4SFhYUIDQ0V+fn5onXr1mLFihUiISFBnDlzRrRs2VIEBgYKIYRYvXq1aNmypThw4ICIjY0VAQEBolGjRuLWrVuvrYVKFoNYKQgJCRGDBg0SFhYWwtTUVDRu3Fj89NNPIj8/XwghRM+ePcW8efMU88vlcuHn5yfi4uLEuHHjxNdff12kvTVr1ogOHToIIZ4HMUtLS1FQUKCY3qlTJ7Fv374irxkyZIjiwP5f9vb2YunSpW9dj8JQER4ernhu1apVwsnJSQghxN27d8WWLVuKvGb9+vXiyy+/FEL8/4H9wIEDiumbN28W7du3F0IIsWPHDtGuXTvFF74QQixatEgRxHbt2iXs7e2FXC5XTL906ZIwNzdXHABf9LZ6hBCidevWioOfEEUPtGfPnhVWVlZF6nnw4IHiC/ZV6/Ou2ywzM1NYWVmJQ4cOCSGECA8PFxYWFoqA0blzZ+Hj46N4bU5OjmjXrp347bffRFZWlmjSpIm4ePFikWXa29sr1qFz584vBVVTU1OxcuVKxePCg/jJkydFZmamWLdunWL5Qghx/fp1YWpqKu7du6d4feEXoRDF3+8HDx5UTB8+fLjw8PB4aTulpaW9dbsXrldJ7idfX1/h4uJS5L31559/CgsLC5GRkaGo78KFC4rpR48eFRYWFor3nqmpqTh+/HiRdX6x3h9//FHx2rVr1woLCwtFAM/Pzxfm5ubi2LFjQoiiQexFkyZNEq6uriIzM1MIIcS5c+fEkSNHiswzcuRIxXumsO709HQhhCgSxLZu3SratGlT5I+Aq1evClNTUxEbG/vWz+urvPh5On78uLCwsBAPHjxQTL9//75o3LhxkSD0pv2yfPlyMXjw4CLPBQcHK9bhxo0bIigoqMj0OXPmiIEDBwohnu/XHj16FJm+bdu2Iss/fPiwYtrevXuFpaXla+vp3LmzsLCwEFZWVsLKykpYWloKCwsLMWjQoCJB8m3H4A4dOojNmzcrpqWmpopmzZoVCWJDhw5VTI+PjxempqZFtmVOTo6wsrISx48fFzdu3BCmpqbin3/+UUy/cOGCkMlkIjU1VZiamoo//vhDMS0sLEwkJiYKuVwu2rRpI9atW1ek1mHDholJkya9shYqee83gEFCKSnpkMvL1++UN2/eDs2bt0NmZiauXLmEkJA/sWXLFujoGMDLazCio6PRs6cHZLI0xWsGDx4FAIiIiIS9vVORafXrN8LDhw8RG3sPGRk5qF69JlJSng8SzczMxL179zBjxgz88MMPitfk5uZCCNUi7RTS0zNAcnLKK6e96MmTTACArm4VxbyqqprIzMyCTJYGHZ3KaNGiHfz9V+DOnVjEx8chOvo2DA2rQiZLw+PHz2usWNFY8Xoh1JGXlweZLA1Xr15HvXoN8PRpDoDnV3nVq2eGgoIDkMnScO3aTSQkJMDa2lpRkxACcrkcV67chLl5oyL1vq2ewtenpWUrHhcUyJGenv2/esKRlZWF1q1bv7QtwsJuwszs+fL09Q1fu+3ets0AoGPHztizZx9atGiPwMAgtG3bDnl5apDJ0lBQIIeZWZMi7detWx/Xrt1E/fqNkJubi6FDh0JFRUUxPTc3FzdvRipeb2hY7aX6GjRo/MJzWqhWrQauXg1H48bN0bWrK3bvDkZExE0kJMQjKioCACCTPYO6ut7/9qGR4vXF3e8GBlVf2AbqMDKqrnicmfm8m+T+/dS3bvfKlWuU+H66cSMCZmaN8ehRuuK5OnXMkJeXh6tXbyItLRNqamqoVau+oo1ateohLy8Ply+Ho1EjCwDA06dZRdY5NTXjlfuhoEAFlStXQVpaHtLSnnd/qqur49Gjp5DJ0pCbm4/MzNwi9W7fvgWnTv2Ndeu2ID09H+npaahbtxEiIm5i3ryFiI+P+9/2vwtHR2fIZGmK99+jR+nIzJQjLS0bQgjIZGm4fv0m6tdvWKSG6tXrQENDA5cvh6NhQ9P/7etXf15f5cXP09Wr4ahWrTpUVXUV86upVUD16s/fa/r6VYuxX27h3LlzsLKyUjwnl8uRk5OD6OgEGBnVRlZWAfz8liIu7g7u3o3DnTsxaNq0GWSyNNy6FYn69c2KtG9v3x0AcP/+PQCAnp7hC+ungezs7NfWU1Agx4ABg+Hg0A25ubnYs2cX/v77OAYMGIZKlZ6/n992DL59OwEPHz5E7dr1X1iOGj77rLZi2+Xm5qNWLRPF9EuXrgMAHBwcitSTnZ2N69cjYGHRAvb2Thg2bBhq1qyFdu06wNHRGUJoIS8P8PIagpkzZ2L58gDY2LRHly4OqFGjLqKi7iI1NRV16pgWWWdz8yY4ceLYK2t5H0ZG+u/92k9BuQti5Ul09G0EBwdh0qSpAABdXV20b/8F2rf/ArNm+eDcuVB4eQ2GuroGXvgeLUJLS+ul5wrH3wohf2mewrET06fPhqmp2VvbAoDGjS1w/XrYK6ddvHgeu3btwMyZcxTPqaoWHVoo/ldQbGw0Ro4cilat2sDKqjlcXXvg5s1wBAXtLDK/hob6K1+vrq5eZOzHfxUUFMDCoimmTZv10jQjI6OXnituPW9aXvXqNeDvv+qlaVWqVMHTp08BvH67vuh12wwAHBycMHXqZGRmZuD48aMYO3ZikXlfHHcCAHK5gJqaqmKMh7//KlSuXKXIPBUqVFD8/1X1qan9dx/IoaqqiqysLHz99SBoaWmjY8dO6NDBFjo6Ohg3bmSR+V9ss7jb+b8D1/+7TQq9bbu/6/zF2U+v/pw930eFf/ipqKgUqbnwvaqqqvbSa1+luOv/Khcu/Iu1a1djwYIlqFXrM8XzBw/ux+LFvujWzRVt27bDoEHDsWHDmmK1+brtIYRQHFuA139e397+qwfrF/7x9LY6gOf7tlMnO4wY8c1L0ypU0MOFC+fw3XcT8OWX9rC2bo6+ffvhyJFDiI6+DQBvPLYWUlN7t6HSlSpVwmef1QYATJ48FU+epGLq1InYvPl3VKtW/a3H4ML3wZuOdYXzFiooKIC6ujo2bPityB9dAGBgYAAVFRX88MNceHoOwD//nMLZs2ewZ88u+Pj8AEdHZ4wcOQZOTi44ffr5tIkTx2D48FHo1cv9lcv+73ugOMc4en8crK9EcnkBgoJ24sqVSy9N09XVQ6VKlQEAtWt/jqioyCLTBwxwx8mTx2BiUhc3blwvMi08PAyVK1eBvr7BS+3q6enB0NAQjx49xGef1Vb82707EOfOnX1lnY6OzoiMjMDlyy/fpuKPP37Dw4fJqFDh7VdwHTr0J0xNzTB//iK4u3vCyqo57t27V+wDd7169REbG6MYIA0At27dVPzfxKQOEhMTYGRkrFivZ8+e4pdfViAvL/+l9t6nnhcPciYmdfHokQy6urqK5amra2DlSn+kpqYWa52Ko0WL1jAwMMD27VuRkZGO9u2/KDK98IwUAOTk5CAm5jYaNGiIWrVqQ01NDampqYr6atX6DBs2rMXNmzf+u5jXtpmS8gjJyQ/QoIEprly5hMTEBKxa9SsGDhyKdu064PHjxwBe/wX8ofv9v4qz3Ut6P5mY1MGNG9eL1Hz9+jWoq6srgk9+fj5iY/9/oPKtWzegqamFzz83ea/1LK6kpET88MM0DBkyAjY27YtMCwraiX79vPDtt9/DxaUHGjRoiISEBMV6/PdL+0UmJnURHX0bOTnZiuciIm4iPz8fn39e54PrrlOnDh48uI9Hjx4pnnv0SIYHD+7DxKR47ZuY1MXdu3eLHMvi4u5g48Zfoaqqin37gmBn1xUzZsxGjx690bhxEyQm/v+FJ686ti5c+BNWrPD/4PUrNHny91BXV8OSJc+vLn/bMVhPTw/GxtUQGfn/n8H09PQidf9XnTp1kJ+fj+zsLEV7hoZVsWrVcsTHx+P27SisWLEUDRuaYejQr/Hrr5vRuXMX/PXXYTx48ACLFy9A9eo10K/fQKxYsQb9+w/CkSOHUKGCHqpWNUJ4+H+/Y66VyHuAiodBTIlMTc1ha9sZM2d+j/379yIpKRG3b0di+/atOHr0EDw8+gMA+vbth717d+HIkUNISkrEL7+sRGpqCpo1a45+/bxw7txZbNq0DgkJ8Th27Ai2bt2E3r37vvYg26/fQGzcuA7Hjh1BUlIiNm1ah6CgwNd+YZiamsPdvR98fCZj9+5AJCTEIyLiJubMmYlLly5i0qTvi7W+BgYVcfduHMLCriApKRF//PEb9u/fUyRYvUmXLo4AgMWLF+Du3TgcO3YUu3b9oVhPB4duUFVVxbx5PyI2Nhrh4dcwf/4cZGVlv/JS//epR0dHB3FxcUhNfYxWrdqgbt16mDVrGiIibuH27SjMnTsTSUmJqF69RrHWqThUVVXRpYsDtm/fCltbu5f++ty3LwhHjoTg7t04+PnNg7q6OuzsukJXVxdffdUHy5cvxtmzZ5CYmIClS/3wzz8nUadO3Tcuc/Pm9Th79jRiYqLx00+zULduPVhbt4CBQUXk5eXh+PGjePDgPk6dOoGAgCUA8Nrt9qH7/b+Ks91Lej/16uWO+/fvw9/fD3fvxuHs2TNYtWoZnJxciry3Fi78SfFHy8qVy+Dq6gYdHZ3/1aSL2NgYPHv29L3W+1WysrLg4zMZTZs2Q/fuPZGS8kjxLycnGwYGFXH58kXcuROL2NgYLFz4E+LiYpGXl6uoCQAiI28hMzOzSNv29k7Q1NTE3Lk/IDY2GmFhV+DrOxctW7ZG3brvd/+zF7Vs2QYNGpjixx+f75eIiFv48cfpqF3bBC1bFu+msj179kFiYjyWLVuE+Pg4XLhwDosWzYeOjg5UVVVRsWJF3Lp1A5GREYiPv4s1a1bh339DFVdS9+zZB7Gx0fj115+RmJiAo0dDcPjwQbRr1+GD169QpUqVMGrUWISGnsY//5wE8PZjsLu7JzZvXo8zZ/7BnTuxmD9/NrKysl57TP/88zro0KEjfvppFsLCruDu3Tj89NMPuHnzOkxM6sDAwAB79uzGpk3rcO9eEq5du4obN553mVeuXAknTx5DQMASJCYmIDIyAhcunIO5eWMAwIABg7Blywb89ddhJCTEY9OmdTh//l/07t23xLYRvRm7JpVs1qx52L59CwIDt2PZskVQVVVD48ZNsGTJCjRpYgkA6NLFAY8fp2Dt2tVITX0MU1MzLF4cgMqVK6Ny5cqYN88Pv/76CzZvXg8jI2MMHjwMffv2f+0y+/TxRE5ODlavDkBq6mPUrv055s5dCEtLq9e+ZuzYCTAxqYO9e3djzZpV0NTUQKNGFli9et1LY69ev1wPxMTcxtSpEwGowMzMHFOm+GDhwnl4+DD5ra/X1taGn98yLFmyAIMHe6JOnbpwcemOs2fPAHj+5bt06UqsWLEUI0YMgra2Ntq1+wLjxk16r3qMjau94jWeWLHCH+HhYdi4cTt8fZdi+fLF8PYeBVVVFVhbt8TMmXNf6i78UF27OmLHjm3o2tXxpWkuLm7444/fEBcXiyZNLLF8+c/Q1X3e9fjNN95QU1PDggVzkJGRjgYNTLF4cUCR7qtX6d79KwQELMWjRzK0bNkGixbNhpqaGpo0aYqvv/4Gv/yyAhkZGahd+3OMGTMB/v5+iIy89cozGR+63/9LVVX1rdu9pPdT1apGWLIkAKtXB2DwYE9UrFgJTk4uGDr06yLzdelij8mTx0IIAScnV4waNVYxzdNzADZt+hU3b4bD2/vV78l3devWDcTGxiA2Ngaurl2LTJs2bRYmTJiMhQvnYfhwL+jp6aFly9bw8hqCEyee326kfv0GaNfuC0yaNBYjR46BgUFFxeu1tbWxZMkKBAQswfDhg6Crq4NOnb7EN994l0jtKioqWLBgMZYvX4xx40ZCTU0Vbdq0w+zZ86GhoVGsNoyNq2HJkpX4+ecADB7cD/r6+vjyS3uMGjUOADBs2EgsWDAHY8eOgJaWFpo0scSYMROwceOvyMnJQfXqNbBw4VL8/PMKbN++BdWr18DUqTPQokUrxRixkuDs7IY//9yP5cuXoFWrtm89Bru798PjxymYN+9HFBTkw82tJ6pXr/HG+85Nnz4bK1YshY/Pt8jLy0Pjxk3g778K+vr60NfX/9/3xGps3boRFSrooWtXBwwcOBQaGhpYuHAZVq70x5Ah/aCuroEvvrCFt/dkAECvXn2RlZWF1asD8ORJKurVa4CFC/3RrJn1a2uhkqUi3rf/QCLlcbA+Fc+9e0l48OA+mjdvqXhu+/Yt+PffUAQE/CJhZcoXGnoafn7zEBT0Z5GxQ717u8LTcwB69Sq5v047dGiJhQv9X+oCpde7fPkivL1H4ciRv6Grqyt1OVTOnT17Gqam5jA0fH7BQn5+PlxcusDXdymsrJpLXF3J42D9N+MZMSozMjIyMHHiGEyf/iMsLa0QH38Xf/yx/aWzEh+Te/eSEBFxCxs3rkWPHr3eaQA3EZVPBw7sQ3r6dnh7T4KmpiYCA3+Hnp4+GjduInVpJAEGMSozGjY0xXffTcfGjb8iOfkBDA2rom/ffuje/SupS1Oahw+TsWDBbDRtagUPjwFSl0NEpWDixKnw91+IMWNGoKCgAE2bNsPSpSsVd+inTwu7JomIiEhp2DX5ZuwHISIiIpIIgxgRERGRRBjEiIiIiCTCIEZEREQkEQYxIiIiIonw9hUkiQp6WtDVKflLtTOzcpGRnlPi7RIRESkDgxhJQldHE5VNJ5R4u6lRyxjEiIio3GDXJBEREZFEGMSIiIiIJMIgRkRERCQRBjEiIiIiiTCIEREREUmEQYyIiIhIIgxiRERERBJhECMiIiKSCIMYERERkUQYxIiIiIgkotSfOPLy8sLjx4+hrv58MXPmzEFGRgYWLFiAnJwcODk5YeLEicosgYiIiKjMUloQE0IgLi4OJ06cUASx7OxsODo6YuvWrahRowZGjhyJU6dOwdbWVlllEBEREZVZSgtisbGxAIChQ4fiyZMncHd3h6mpKUxMTFC7dm0AgKurK0JCQhjEiIiI6JOktDFiz549g42NDVatWoVNmzZhx44duHfvHoyMjBTzGBsbIzk5WVklEBEREZVpSjsjZm1tDWtra8Xj3r17IyAgAC1atFA8J4SAiorKO7VraKhXYjXSx8nISF/qEoiIiIpFaUHs4sWLyMvLg42NDYDnoatWrVqQyWSKeWQyGYyNjd+p3ZSUdMjlokRrpdKnzLAkk6UprW0iIno3/OP4zZTWNZmWlgY/Pz/k5OQgPT0de/bswaRJk3Dnzh3cvXsXBQUFOHDgADp27KisEoiIiIjKNKWdEevcuTPCwsLQo0cPyOVy9OvXD9bW1vD19cW4ceOQk5MDW1tbODo6KqsEIiIiojJNRQhRrvr52DX5cTAy0kdl0wkl3m5q1DJ2TRIRlSHsmnwzpd7Qlai05efmKOVDn5udhadp+SXeLhERfdoYxOijoq6phQ1O9Uq83aGHYoE0nmkjIqKSxd+aJCIiIpIIgxgRERGRRBjEiIiIiCTCIEZEREQkEQYxIiIiIokwiBERERFJhEGMiIiISCIMYkREREQSYRAjIiIikgiDGBEREZFEGMSIiIiIJMIgRkRERCQRBjEiIiIiiTCIEREREUmEQYyIiIhIIurKXsDChQuRmpoKX19f+Pj44NKlS9DR0QEAjB07Fl27dlV2CURERERlklKD2NmzZ7Fnzx506tQJABAeHo5t27bB2NhYmYslIiIiKheU1jX55MkT+Pv7Y9SoUQCArKws3Lt3D9OmTYOrqysCAgIgl8uVtXgiIiKiMk9pQeyHH37AxIkTYWBgAAB49OgR2rZti/nz5yMwMBAXL17Erl27lLV4IiIiojJPKV2TO3fuRI0aNWBjY4OgoCAAQO3atbFq1SrFPF5eXti7dy/c3d3fqW1DQ70SrZWouIyM9KUugYiIPjJKCWIHDx6ETCaDm5sbnj59iszMTIwZMwbdu3eHg4MDAEAIAXX1d198Sko65HJR0iVTKSuPoUYmS5O6BCKicqc8Hu9Lk1KC2MaNGxX/DwoKwvnz5zF48GCMHDkSbdu2ha6uLv744w989dVXylg8ERERUbmg9NtXFDI3N8fXX38NT09P5Ofnw97eHi4uLqW1eCIiIqIyR0UIUa76+dg1+XEwMtJHZdMJJd5uatQybHCqV+LtDj0Uy65JIqL3wK7JN+Od9YmIiIgkwiBGREREJBEGMSIiIiKJMIgRERERSYRBjIiIiEgiDGJEREREEmEQIyIiIpIIgxgRERGRRBjEiIiIiCTCIEZEREQkEQYxIiIiIokwiBERERFJhEGMiIiISCIMYkREREQSYRAjIiIikgiDGBEREZFEGMSIiIiIJKL0ILZw4UJ8//33AIDQ0FC4urrC3t4e/v7+yl40ERERUZmm1CB29uxZ7NmzBwCQnZ2NadOmYfXq1Th48CDCw8Nx6tQpZS6eiIiIqExTWhB78uQJ/P39MWrUKADAtWvXYGJigtq1a0NdXR2urq4ICQlR1uKJiIiIyjylBbEffvgBEydOhIGBAQDg4cOHMDIyUkw3NjZGcnKyshZPREREVOapK6PRnTt3okaNGrCxsUFQUBAAQC6XQ0VFRTGPEKLI4+IyNNQrsTqJ3oWRkb7UJRAR0UdGKUHs4MGDkMlkcHNzw9OnT5GZmYmkpCSoqakp5pHJZDA2Nn7ntlNS0iGXi5IslyRQHkONTJYmdQlEROVOeTzelyalBLGNGzcq/h8UFITz589j9uzZsLe3x927d/HZZ5/hwIED6NWrlzIWT0RERFQuKCWIvYqWlhZ8fX0xbtw45OTkwNbWFo6OjqW1eCIiIqIyR+lBrGfPnujZsycAwMbGBsHBwcpeJBEREVG5wDvrExEREUmEQYyIiIhIIgxiRERERBJhECMiIiKSCIMYERERkUQYxIiIiIgkwiBGREREJBEGMSIiIiKJMIgRERERSYRBjIiIiEgiDGJEREREEmEQIyIiIpIIgxgRERGRRBjEiIiIiCTCIEZEREQkEQYxIiIiIokwiBERERFJRF2ZjS9fvhyHDx+GiooKevfujSFDhsDHxweXLl2Cjo4OAGDs2LHo2rWrMssgIiIiKpOUFsTOnz+Pf//9F8HBwcjPz0e3bt1ga2uL8PBwbNu2DcbGxspaNBEREVG5oLSuydatW2PLli1QV1dHSkoKCgoKoK2tjXv37mHatGlwdXVFQEAA5HK5skogIiIiKtOUOkZMQ0MDAQEBcHZ2ho2NDfLz89G2bVvMnz8fgYGBuHjxInbt2qXMEoiIiIjKLKWOEQMAb29vjBgxAqNGjcLZs2exatUqxTQvLy/s3bsX7u7uxW7P0FBPGWUSvZWRkb7UJRAR0UdGaUEsJiYGubm5aNSoEXR0dGBvb4+DBw+iUqVKcHBwAAAIIaCu/m4lpKSkQy4XyiiZSlF5DDUyWZrUJRARlTvl8XhfmpTWNZmYmIgZM2YgNzcXubm5OHbsGFq1aoX58+fj6dOnyMvLwx9//MErJomIiOiTpbQzYra2trh27Rp69OgBNTU12NvbY+zYsahcuTI8PT2Rn58Pe3t7uLi4KKsEIiIiojJNRQhRrvr52DX5cTAy0kdl0wkl3m5q1DJscKpX4u0OPRTLrkkiovfArsk34531iYiIiCTCIEZEREQkEQYxIiIiIokwiBERERFJhEGMiIiISCIMYkREREQSYRAjIiIikgiDGBEREZFEGMSIiIiIJMIgRkRERCQRBjEiIiIiiTCIEREREUmEQYyIiIhIIgxiRERERBJhECMiIiKSCIMYERERkUQYxIiIiIgkotQgtnz5cnTr1g3Ozs7YuHEjACA0NBSurq6wt7eHv7+/MhdPREREVKapK6vh8+fP499//0VwcDDy8/PRrVs32NjYYNq0adi6dStq1KiBkSNH4tSpU7C1tVVWGURERERlltLOiLVu3RpbtmyBuro6UlJSUFBQgGfPnsHExAS1a9eGuro6XF1dERISoqwSiIiIiMo0pXZNamhoICAgAM7OzrCxscHDhw9hZGSkmG5sbIzk5GRllkBERERUZimta7KQt7c3RowYgVGjRiEuLg4qKiqKaUKIIo+Lw9BQr6RLJCoWIyN9qUsgIqKPjNKCWExMDHJzc9GoUSPo6OjA3t4eISEhUFNTU8wjk8lgbGz8Tu2mpKRDLhclXS6VsvIYamSyNKlLICIqd8rj8b40Ka1rMjExETNmzEBubi5yc3Nx7NgxeHh44M6dO7h79y4KCgpw4MABdOzYUVklEBEREZVpSjsjZmtri2vXrqFHjx5QU1ODvb09nJ2dUaVKFYwbNw45OTmwtbWFo6OjskogIiIiKtNUhBDlqp+PXZMfByMjfVQ2nVDi7aZGLcMGp3ol3u7QQ7HsmiQieg/smnwz3lmfiIiISCIMYkREREQSYRAjIiIikgiDGBEREZFEGMSIiIiIJMIgRkRERCQRBjEiIiIiiTCIEREREUmEQYyIiIhIIgxiRERERBJhECMiIiKSSLGC2JIlS5RdBxEREdEnp1hB7OTJk0oug4iIiOjTo16cmT777DMMHToUzZs3R4UKFRTPDxkyRGmFEREREX3sihXEKlWqBABISkpSZi1EREREn5RiBbEFCxYAAJ49ewYDAwOlFkRERET0qSjWGLE7d+6gW7ducHZ2RnJyMpycnBATE6Ps2oiIiIg+asUKYnPnzsX06dNhaGiIatWqYcCAAfjhhx/e+rqVK1fC2dkZzs7O8PPzAwD4+PjA3t4ebm5ucHNzw9GjRz9sDYiIiIjKqWJ1TT558gTt27fHokWLAAD9+/dHYGDgG18TGhqK06dPY8+ePVBRUcHw4cNx9OhRhIeHY9u2bTA2Nv7w6omIiIjKsWLf0DUnJwcqKioAAJlMBrlc/sb5jYyM8P3330NTUxMaGhqoX78+7t27h3v37mHatGlwdXVFQEDAW9shIiKiT1NiYiIaNWqk6EVzdXWFh4cHDh48+NbXrly5En/99ZdS6ho6dCgeP35cIm0V64xYv379MGzYMKSkpGDJkiX4888/MXz48De+pmHDhor/x8XF4dChQ/jtt99w/vx5zJo1C/r6+hg5ciR27doFd3f3D1sLIiIi+ihpa2tj3759isdJSUkYPHgw1NTU4ODg8NrXnTt3Dg0aNFBKTWfOnCmxtt4axKKiomBgYIDx48fj5MmTyM/Px9y5c9G+fftiLeD27dsYOXIkvvvuO9SrVw+rVq1STPPy8sLevXvfKYgZGuoVe16ikmRkpC91CUREn7xatWrB29sb69evh6mpKebMmYOMjAzIZDKYm5tj2bJl2LVrF8LDw+Hn5wc1NTU0aNDglfNpaWkhICAAR48ehYaGBipXrowFCxbA2NgYMTExmDdvHp48eYKCggJ4eXmhd+/e8PHxAQAMGjQIa9euRY0aNT5ofd4YxHbv3o2FCxfCxMQE8fHxWLx4Mb744otiN37p0iV4e3tj2rRpcHZ2RmRkJOLi4hQJVggBdfVinZRTSElJh1wu3uk1VPaUx1Ajk6VJXQIRUbmjjOO9ubk5oqKiEBgYiB49esDNzQ15eXno2bMnTp48if79+yMkJAT9+/dH165dsXDhwlfOZ2lpic2bN+Ps2bPQ1NTEhg0bcO3aNXTq1Ane3t7w8/ODhYUF0tLS0LdvXzRo0AALFixAUFAQNm/ejCpVqnzwurwxBW3duhX79+9HtWrVcOXKFfj7+xc7iN2/fx9jxoyBv78/bGxsADwPXvPnz0fbtm2hq6uLP/74A1999dUHrwQRERF9OlRUVKCtrY0pU6bgzJkz+PXXXxEXF4eHDx8iMzPzpflfN1+1atVgbm6Or776Ch07dkTHjh1hY2OD6OhoxMfHY9q0aYo2srOzcfPmTVhZWZXourz1dFS1atUAANbW1khNTS12w+vXr0dOTg58fX0Vz3l4eODrr7+Gp6cn8vPzYW9vDxcXl/com4iIiD5V169fh6mpKSZNmoSCggI4OTmhU6dOuH//PoR4udfsdfOpqqpi27ZtuH79Os6ePYv58+fjiy++gJubG/T19YuMTXv06BH09Uv+7N4bg1jhVZKF1NTUit3wjBkzMGPGjFdO69+/f7HbISIiIip0584drF69GtOnT8e3336Lbdu2wdzcHLdv30ZYWBicnJwAPM8s+fn5AIDTp0+/cr6IiAhMnjwZgYGBsLS0RNWqVbF3715MmDBBcZGAm5sb7t+/j6+++gqrVq1CixYtirT9od5pgNZ/gxkRERGRMmVnZ8PNzQ0AoKqqCi0tLUyaNAmdOnXCxIkTMWbMGOjq6kJPTw+tWrVCfHw8AMDOzg5Lly5FXl7ea+fr06cPnJyc0KtXL+jq6kJbWxszZsyApqYmVq9ejXnz5mHdunXIz8/H+PHj0aJFCwCAo6MjvLy8sGLFCpiamn7Q+qmIV53D+5/GjRtDW1u7yMbQ1taGEAIqKiq4fPnyBy38fXCw/sfByEgflU0nlHi7qVHLsMGpXom3O/RQLAfrExG9h/J4cVZpeuMZMf78EBEREZHyvDGI1apVq7TqICIiIvrkFPsnjoiIiIioZDGIEREREUmEQYyIiIhIIgxiRERERBJ5tx96JCIiIioh2Tl50NbSkKTd2bNn4/Lly8jLy0N8fDzq168PABg4cCB69er11mW4ubkVufP++2IQIyIiIkloa2ko7Z6SbzNr1iwAQGJiIgYOHPjOoaokQhjAIEZERESkYGdnB0tLS9y6dQvbt2/Hli1bcPbsWTx9+hTGxsbw9/dH1apVYWZmhsjISKxYsQLJycm4e/cukpKS0KdPH4wePbrYy+MYMSIiIqIXdOzYEYcPH0Z6ejpiY2OxY8cOHD58GDVq1EBwcPBL80dGRmL9+vXYuXMn1q5di2fPnhV7WTwjRkRERPSCZs2aAQBMTEwwdepU7Ny5E3fu3MHVq1fx+eefvzR/mzZtoKmpCUNDQ1SqVAlpaWkwMDAo1rJ4RoyIiIjoBVpaWgCA8PBwDBs2DHK5HA4ODujSpQte9RPdhfMDgIqKyivneR0GMSIiIqJXuHDhAlq3bg1PT0/UqVMHJ0+eREFBQYkug12TRERERK/QrVs3jB07Fq6urgCAJk2aIDExsUSXoSLe5fxZGZCSkg65vFyVTK9gZKSvtEuWNzjVK/F2hx6KhUyWVuLtEhF97IyM9F87Tcr7iJUVSu2aXLlyJZydneHs7Aw/Pz8AQGhoKFxdXWFvbw9/f39lLp6IiIjKMGWFpfISwgAlBrHQ0FCcPn0ae/bswd69e3Hjxg0cOHAA06ZNw+rVq3Hw4EGEh4fj1KlTyiqBiIiIqExTWhAzMjLC999/D01NTWhoaKB+/fqIi4uDiYkJateuDXV1dbi6uiIkJERZJRARERGVaUoLYg0bNoSVlRUAIC4uDocOHYKKigqMjIwU8xgbGyM5OVlZJRARERGVaUq/avL27dsYOXIkvvvuO6ipqSEuLk4xTQgBFRWVd2rP0FCvhCskKp43DTglIiJ6H0oNYpcuXYK3tzemTZsGZ2dnnD9/HjKZTDFdJpPB2Nj4ndrkVZMfh/IYanjVJBHRuyuPx/vSpLSuyfv372PMmDFYvHgxnJ2dATz/yYA7d+7g7t27KCgowIEDB9CxY0dllUBERERUpintjNj69euRk5MDX19fxXMeHh7w9fXFuHHjkJOTA1tbWzg6OiqrBCIiIirD8nNzoK6p9fYZldDu7NmzcfnyZeTl5SE+Ph7169cHAAwcOBC9evV66zLS0tLw/fffY9WqVR9UK2/oSpLgDV2JiD4Nb+uaVNYxu7gSExMxcOBAHD9+/J2W8b6v+y/+1iQRERERgLt372LIkCH46quv4OnpiZs3bwIA9u/fDzc3N/Ts2RPe3t7IycnBTz/9hIcPH2LMmDEftEwGMSIiIiIAU6dOxZQpU7Bnzx7MnTsXEydOBAAsW7YMGzZsQFBQEGrVqoXY2FjMmDEDxsbGH9w1yR/9JiIiok9eRkYGwsPD4ePjo3guMzMTqamp6Ny5Mzw9PdGlSxc4ODigUaNGJfbj3wxiRERE9MmTy+XQ1NTEvn37FM89ePAAlSpVwowZMxAREYFTp05hypQpGDt2LFq0aFEiy2XXJBEREX3y9PX1UadOHUUQO3PmDPr374/8/HzY29ujcuXKGDlyJNzc3HDr1i2oq6sjPz//g5fLM2JEREQkifzcnHe6wvFd2n2f22IsWrQIP/74I9atWwcNDQ34+/tDQ0MD3t7eGDp0KLS0tGBoaAhfX18YGBigZs2a8PLywtatW9+7Vt6+giTB21cQEX0aeGf9N2PXJBEREZFEGMSIiIiIJMIgRkRERCQRBjEiIiIiiTCIEREREUmEQYyIiIhIIgxiRERERBJhECMiIiKSCO+sT0T0PxX0tKCro1ni7WZm5SIjPafE2yWi8o9BjIjof3R1NJX2iw8MYkT0KkrtmkxPT4eLiwsSExMBAD4+PrC3t4ebmxvc3Nxw9OhRZS6eiIiIqExT2hmxsLAwzJgxA3FxcYrnwsPDsW3bNhgbGytrsURERETlhtLOiAUGBmLWrFmK0JWVlYV79+5h2rRpcHV1RUBAAORyubIWT0RERFTmKS2IzZs3Dy1btlQ8fvToEdq2bYv58+cjMDAQFy9exK5du5S1eCIiIqIyr9QG69euXRurVq1SPPby8sLevXvh7u7+Tu0YGuqVdGlExWJkpC91CVSO8f1DRK9SakEsMjIScXFxcHBwAAAIIaCu/u6LT0lJh1wuSro8KmXl8UtJJkuTugRSMmW+L/n+oU9VeTzel6ZSu6GrEALz58/H06dPkZeXhz/++ANdu3YtrcUTERERlTmldkbM3NwcX3/9NTw9PZGfnw97e3u4uLiU1uKJiIiIyhylB7Hjx48r/t+/f3/0799f2YskIiIiKhf4W5NEREREEmEQIyIiIpIIgxgRERGRRBjEiIiIiCTCIEZEREQkEQYxIiIiIokwiBERERFJhEGMiIiISCIMYkREREQSYRAjIiIikgiDGBEREZFEGMSIiIiIJMIgRkRERCQRBjEiIiIiiTCIEREREUmEQYyIiIhIIgxiRERERBJRahBLT0+Hi4sLEhMTAQChoaFwdXWFvb09/P39lbloIiIiojJPaUEsLCwMnp6eiIuLAwBkZ2dj2rRpWL16NQ4ePIjw8HCcOnVKWYsnIiIiKvOUFsQCAwMxa9YsGBsbAwCuXbsGExMT1K5dG+rq6nB1dUVISIiyFk9ERERU5qkrq+F58+YVefzw4UMYGRkpHhsbGyM5OVlZiyciIiIq85QWxP5LLpdDRUVF8VgIUeRxcRka6pVkWUTFZmSkL3UJVI7x/UNEr1JqQax69eqQyWSKxzKZTNFt+S5SUtIhl4uSLI0kUB6/lGSyNKlLICVT5vuS7x/6VJXH431pKrXbVzRr1gx37tzB3bt3UVBQgAMHDqBjx46ltXgiIiKiMqfUzohpaWnB19cX48aNQ05ODmxtbeHo6FhaiyciIiIqc5QexI4fP674v42NDYKDg5W9SCIiIqJygXfWJyIiIpIIgxgRERGRRBjEiIiIiCTCIEZEREQkEQYxIiIiIomU2u0riIjepKK+OjS1dUq83dzsLDxNyy/xdomISgKDGBGVCZraOtjgVK/E2x16KBZI413tiahsYtckERERkUQYxIiIiIgkwiBGREREJBEGMSIiIiKJMIgRERERSYRBjIiIiEgiDGJEREREEmEQIyIiIpIIb+hKRKRk+bk5MDLSL/F2+asBROUfgxgRkZKpa2rxVwOI6JUkCWJeXl54/Pgx1NWfL37OnDlo1qyZFKUQERERSabUg5gQAnFxcThx4oQiiBERERF9ikp9sH5sbCwAYOjQoejevTu2bdtW2iUQERERlQmlfkrq2bNnsLGxwcyZM5GXl4eBAweibt26aN++fWmXQkRERCSpUg9i1tbWsLa2Vjzu3bs3Tp06VewgZmiop6zSiN5IGVe9Uen4mPfdx7xuRJ+CUg9iFy9eRF5eHmxsbAA8HzP2LmPFUlLSIZcLZZVHpaQ8fnnIZLw6TZmU+Z4o7r7j+5Ko5JXHz1VpKvUxYmlpafDz80NOTg7S09OxZ88edO3atbTLICIiIpJcqZ8R69y5M8LCwtCjRw/I5XL069evSFclERER0adCkvtHTJgwARMmTJBi0URERERlBm/kRURE9ImpqK8OTW2dEm+XP7v17hjEiIiIPjGa2jr82a0yotQH6xMRERHRcwxiRERERBJhECMiIiKSCMeIERF9AiroaUFXR7PE283MykVGek6Jt0v0qWAQIyL6BOjqaKKy6YQSbzc1ahmDGNEHYNckERERkUQYxIiIiIgkwiBGREREJBGOESOiYlPWgG8iejV+5j5+DGJEVGzKGvANPB/0TURFKfMiCyob2DVJREREJBEGMSIiIiKJMIgRERERSYRjxD5BFfXVoamtU+Lt5mZn4Wlafom3S89xvxERfXwYxD5Bmto62OBUr8TbHXooFkhLK/F26TnuNyKij48kXZP79+9Ht27dYG9vj99++02KEoiIiIgkV+pnxJKTk+Hv74+goCBoamrCw8MDbdq0QYMGDUq7FCIiIiJJlfoZsdDQULRt2xaVKlWCrq4uHBwcEBISUtplEBEREUmu1IPYw4cPYWRkpHhsbGyM5OTk0i6DiIiISHIqQghRmgv8+eefkZOTgwkTJgAAAgMDER4ejjlz5pRmGZ+0/NwcqGtqSdpudk4etLU0Sr6GnGyoa2mXfLtK2mZloYaysN+AsrHvPub35ce8bh8z7rePX6mPEatevTouXryoeCyTyWBsbFzs16ekpEMuL9Xs+JHKlbzdNGQrqYY8JbWrrG32Lj7m/QaUhX33Mb8vlbFuRkb6SruaVybj1bxA+X9PGhnpK2k5H4dS75ps164dzp49i8ePHyMrKwtHjhxBx44dS7sMIiIiIsmV+hmxatWqYeLEiRg4cCDy8vLQu3dvWFpalnYZRERERJKT5Iaurq6ucHV1lWLRRERERGUGf2uSiIiISCIMYkREREQSYRAjIiIikgiDGBEREZFEGMSIiIiIJFLqd9b/ULyhKxFR2VFRXx2a2jol3m5udhaepuWXeLtU+nhD1zeT5PYVRET0cXialg+k8Q74RO+LXZNEREREEmEQIyIiIpIIgxgRERGRRBjEiIiIiCTCIEZEREQkEQYxIiIiIokwiBERERFJhEGMiIiISCIMYkREREQSYRAjIiIikgiDGBEREZFEyt1vTaqqqkhdAhEREVGJUBFCCKmLICIiIvoUsWuSiIiISCIMYkREREQSYRAjIiIikgiDGBEREZFEGMSIiIiIJMIgRkRERCQRBjEiIiIiiTCIEREREUmEQYyIiIhIIgxixcQfIKCyIiEhAdeuXZO6DFISHmvKnmfPnkldAn3EGMTeICsrC5cuXUJmZiZUVD6+37jMzMyUugSl279/P65du4aEhAQA5ftLLisrCwsXLsSYMWOQkZEhdTlK9eTJEyxfvhwHDx5ESkoKgPK9794kOTkZ/v7+OHz4MNLT08v1sebJkydYsGABAgMDkZycDAAoKCiQuKr3V/iZmz9/PtLT06UuR6nK834q7/hbk6+xY8cOrF27FjVr1oSamhrWrVsHDQ0NqcsqEVlZWViyZAnu3buHWrVqwcbGBnZ2dlKXVaJOnjyJ1atXo3LlyqhUqRI0NDQwe/ZsqKmpSV3ae/n999+xatUqVKlSBcHBwVKXo1SbNm1CcHAwzM3NkZaWhri4OOzfv1/qspTi559/xtGjR9GkSRMkJSUhKysL27dvl7qs97Jp0ybs27cP5ubmePbsGVJSUrBjxw6py3pvv//+OzZs2IAuXbpg7NixqFChgtQlKcWTJ08QEBCAihUrwtbWFlZWVlKX9MlRl7qAsuaff/7B5s2boaGhgV9++QX169eHhYUFLl++jDZt2khd3gfbunUrDh48iKZNm2LUqFE4ceIEfHx8sGTJErRt2xbq6uX/LZGQkIDNmzfjm2++QadOnZCXl4fHjx8rQpgQotycdUhNTcXMmTMhhMD48eOxceNGpKenQ09PDwUFBeU2WL5KTEwMxo4diyZNmiAgIACfffYZAKBVq1b466+/0KVLF4krLDnXr1/HpEmT0LZtW6xduxZVq1ZFVlYW2rRpg4sXL6Jly5ZSl1hsly9fxrRp02BtbY2lS5eibt26CAkJwaFDh5CZmQkdHZ1y83kDgJs3b2LOnDmQy+XIzMzE5MmToa6uDrlcDlXVj6sTaf369Th8+DBatmwJdXV1+Pj4YP/+/R/F90B5wq39goiICIwYMQLz589Hz549ATz/Uu/Xrx8aNGggcXUfJj8/H8HBwZg3bx5OnTqFatWqAQAsLS2RmZmJnTt3ombNmqhXr57ElX648PBw1KtXD506dUJBQQFOnTqFzMxM5Ofno3379op1L8uSkpJw8+ZN1KlTByNGjECzZs0APP/SmzNnDvz8/D66L4XKlSvj8ePHGDZsmCKEAUCHDh0+mrPRhQwMDPDs2TOMHj0aVatWBQDo6Ojgiy++kLiy4nv06BFu3LiB27dvo1KlSliwYAEAIC4uDkeOHIGNjQ10dXUlrrL4EhISsHHjRpibm+Prr7+GnZ0dpk6dinnz5mHWrFkf1ectNzcXy5Ytw++//45jx46hSpUqAIC///4bCQkJqFu3LgB8lOGzLOIW/h8hBMzNzeHs7Iz79+8DAHx9fdG7d29cvXoVqampinnLU196QkICvvvuOyxatAjW1tZo3rw5IiMjAfz/GLGRI0ciISEB0dHRAMrfWJyIiAjk5uYqHteqVQuhoaEYNWoUbG1t8ccff2DLli3YunUrvvnmGwkrfbuCggIsX74cw4YNw+HDhzFt2jT89ttvin31zTff4MqVK7hy5QpUVFTK1XvxTQoKClClShUMGTIEfn5+AIDs7GwsWLAAMpkM5ubmEldYcuRyOUxMTNCzZ0/4+voCeD4Y/IcffkB2djbq168vcYXFs2zZMty5cwcODg5o0qQJ1q5di0WLFmHYsGE4fvw4bt68iePHjys+m3K5XOKKX63wMzd+/HhUqVIF7u7uiqEaY8aMwalTpz6ai2MKx7lpamqiU6dOaN68OVRUVPDgwQN4eXnh0aNHmDt3LpYuXYqsrCyGsFLySY8RCw0NRXZ2Nuzs7BTdPDExMRg8eDB0dHRgb2+Phg0b4vr16wgNDUWTJk0wc+ZM6OrqlvkuoYKCAqxcuRInT55E165dcfPmTTx79gy9e/fGr7/+qhhzk5eXBw0NDfj5+eHu3btYtWqVxJW/m7Vr12L58uUYOHAgpk6dqnj++vXrCAsLQ82aNVG9enUYGxujatWq6NChA1avXg1LS0sJq361kydPYsaMGejVqxeGDRsGAwMDPHz4EF5eXvjqq6/g6emJihUr4ueff0ZoaCi2bt0qdcnvbdeuXdDS0oKrqyvy8/Ohpqam6L7q3r07zMzMEBcXBysrK0yYMKFcj8/ZunUratasiS+//LLIuubm5sLV1RVWVlaIiYlB27ZtMX78+DJ/9i8/Px/q6ur47bffcOLECaxbtw7BwcHw9/dH8+bNsWTJEty5cwenTp3C+vXr0bBhQ4wcORLW1tbQ1NSUuvwiCj9zbm5uGDRoEIyNjQE8v4DC0NAQ6urqWLp0KaKiovDzzz+Xqy7WF2VmZir2S7Vq1dClSxd8+eWXWLlyJXbv3o2qVaviq6++Qr9+/RAcHIyQkBB07twZffr0kbr0T8InG3flcjlu376NDRs2IC8vD2pqapDL5ahfvz769u0LPT09fPvtt3Bzc8OMGTMwc+ZMJCUloXfv3jhy5IjU5b/RyZMnYWtri5ycHPzyyy/45ptvMGbMGKipqeGLL76AkZERfv75ZwD//1dqlSpVFN2v5SmbV69eHZ6enrh58yamTJmCmzdvAgCaNm2KXr16wc7ODo0bN0bVqlURFRWFL774AqamphJX/WpZWVnIzc3FxIkTYWBggIyMDBgbG2P8+PE4e/YsHjx4AADw8vJCTk4OAgMDAZTdMw2vEhERAR8fH0RGRmL69OkAoBiPUnh2b/Lkyfj7778xdepUTJ8+HRUqVCiXZ/6uXr0KANDQ0MB3330HoOi6ampqwtvbG3///Tfmz5+Pb7/9FhoaGmVyXWNiYhQXiRSeJWnatCmqV6+OtLQ0tGzZEg4ODmjYsCEAoE6dOhg8eDA2bNgAPT097N27t0yuV1ZWFnJycjBlyhQYGxvjwoUL6Nu3L3bt2qX4XE2YMAGJiYkICQmRuNr3c/LkSbi7u0NVVRXTp09Hs2bNEBUVBQBwcnJCtWrV4OHhgX79+gF4/oeQmppauf7jp7z5ZIOYqqoqnJ2dUaNGDWzZsqXItH79+iE/Px8HDx4E8PygaWNjg3Xr1sHX1xdOTk5SlFxshQeX7777DtWqVcOFCxewYsUKtGrVCpUrV8bXX3+NnTt3IjMzE1paWrhx4wZOnjypGIdUnv7qU1VVRWRkJJYvXw4dHR3MmTMH586dAwAcP34cgwYNwtmzZ+Hr64sJEyagcePG0NbWlrjqV3NyckLbtm0VY220tLQAAN26dUN2djZOnz4NANDT04OHh8dLX4zlgUwmw549e2Bqaorq1atj0aJFAJ6HycKzzLa2tmjSpAmuXLny0rTyIjQ0FB4eHti+fTs8PDxQu3ZtxdnmFy+ycHZ2homJSZld14cPH+Lq1as4c+YMZs+ejWPHjim6yXNzc5GUlAR9fX3UrFkTrVq1QnR0NEJDQ6GiogK5XI6GDRsiICAACxYsgI6OjsRr8zInJyfY2Nhg+vTp+Omnn7B48WJ4eXlhzJgx0NTUVIyRGjJkCBYuXFiubhsjk8kAPA9i3377LaZPn4769evDw8MDo0ePxv3792FgYIA+ffrg0KFDitddvXoV9+7dU4xdJOX7pLsmhRA4ceIENm7cCF9fX9SqVQu5ubnQ1NTEkSNHsHLlSsWXXXkbtOjt7Q0DAwNoaWnhzJkzUFdXR8WKFdGjRw+0a9cOO3fuRFRUFGrXro3Lly9j4MCBcHV1lbrs13pVNzLwfMxDv379sGnTJlSpUgW9evVCamoqRowYAU9PTyxZsgRPnz5FQUEBJk+erBiUWlbdunULY8aMwfr161G3bl3FFZIrV65EXFwcFi9eLHWJ76Tw8FIY7s+fP49Zs2bBxcUFWlpaWLJkCU6dOgVjY2Pk5uZCQ0MDKioqiIiIQN++fREUFFRuxky96Pr165g8eTLatm2L4cOHIzMzE7169cKFCxegq6urOM4Az7fJyJEjcejQIVSvXl3iyp8rKCjAihUrcPjwYdSsWRNWVlbQ1tZGZGQkKleurDib6e7urrg6+dGjR9i1axeioqKwcOFCRRdrWb9KOSIiAp6enujRowdmzZr12vl27dqF3r17l2Jl7+/gwYP4559/MGXKFLi7u2Px4sWK21LcvXsXAQEB+OeffzBr1izF+D5dXV3k5eUhIiICw4cP/6iuUi7ryk+yUAIVFRW0adMG5ubmWL9+PQAoDo5qamq4d+8eTp06BaB8nXUAng/q/vPPP1FQUICQkBDs3r0bffr0QXBwsKJr68yZM9DX18fOnTvLdAh7VTdy4Rf8kydP0LBhQ+zYsQMeHh4wMDCAs7MzfvvtN3h7e2Py5MmYNWsW5s2bV+ZDGAA0atQIDg4OWLhwIYDnZ78A4MGDB3B0dJSytHf2119/oWfPnti3b5/iOWtraxgbGyMiIgKGhoawsrLCjBkzADz/7KmoqGDbtm0wMjLCkiVLykwweZtTp04pbhoMPL8CVFNTE/n5+Th+/DjMzc3RpUsXxTjGwuPMhg0bYGFhUaben7m5uZg9ezYePXqEnTt3Ys2aNWjfvj0GDRqEkSNH4s6dO/jpp5+QnJwMe3t7xbGxatWqsLa2Rrt27YoMbyjLIQwAzM3N4eHhgadPnxZ5PiQkBN7e3ggPDweAchPCAODs2bMwMzODiooKatSogcqVKyumrVixAs2aNcOUKVOwY8cOqKqqomPHjjhw4ABq1qyJHTt2MISVNkEiLCxMDBkyRMTFxYmYmBjRrVs30bt3bxEWFiZ1aR/E19dXTJw4UQghREFBgRBCiNzcXDF37lyxZMkSkZycLGV570Qmk4lvv/1WrFu3Tgjx/+sjhBCenp6iRYsW4ujRo4rn4uLixIEDB0q9zpLw6NEj4eLiIsLCwsT9+/fFqFGjxPDhw0ViYqLUpb2TQ4cOCTMzM9GxY0cRGhoqZDKZEEKIbdu2ic2bN4sZM2aI3bt3CwsLCxEdHS1u3bolHB0dRdeuXUV0dLTE1Rffxo0bhZmZmRg0aJDIyMhQvDeXLVsmZsyYIXx8fMSVK1dEWlqaaNKkiXjw4IG4cOGCsLOzEy4uLiIpKUniNShKJpOJvn37Kh7HxcWJ69evi4sXL4rs7GyRmpoqZs6cKebOnSt69eol/vzzT8W8crlcipI/WOFn7sqVK+LBgwdi1KhRYtCgQeLChQtSl1Ysf/75p9i+fbtISUkRQggxbdo0ce7cOSGEEM7OzmLlypWKeTMzM4UQz9d58ODBIiUlReTm5oqMjIzSL5yEEELwPmJ4fhaidevWcHBwQMWKFTFy5EgMHTpU6rI+2PDhwzFo0CCcOnUKtra2iu6QwjMQ5YmhoSGcnJywceNGODo6olatWsjMzISuri7s7e1Rs2ZNxV9xubm5MDExgYmJicRVvx9DQ0N4eHjA3d0dlpaWcHd3L1d/jReyt7dH//79cfLkSURGRuLkyZPw8fHB3bt30bNnT+jo6ODRo0dwdHSEs7MzqlSpgnHjxsHT01Pq0t8qKSkJYWFhaNy4MRwdHXH27FlcunQJv//+O+rWrQs7OzsYGBigY8eOOHLkCI4ePQorKyvFLVWqV6+OCRMmoEePHlKvykuqVq0KFRUVjB8/HllZWUhPT4eWlhauXr2Ktm3bYvTo0ZgzZw6Cg4Oxc+dOXLhwAd26dQNQ9s9+vU7hZ87DwwNNmzZF3759y81nTi6XIy0tDYcPH8bvv/+OESNG4MaNG4r31oQJE/Djjz/C0tIS1tbW0NPTw6VLl7B48WK0b98eFStWhJqaWpm/WvdjxiCG51c1OTs7Q1NTE/3791cMki7vDA0N4enpiR9//BEnTpwoc5eOv4vCbuSzZ89i/fr1+OGHHxQ3i4yIiECVKlUUwaw8r2ehwsvGe/fuXW7fj6qqqnB3d8fff/8NCwsLXLlyBb/99hsqV66M4OBgjBs3Dt9++y26dOkCa2tr9O/fX+qS36rwtjCHDh1CkyZNsGHDBjRu3BgDBw7E7du3UaNGDWzcuBE1a9ZEZmYmbty4gYEDB2LBggU4cOAAxowZgxo1aihuGF1WzZ8/H7///juys7Px5ZdfokKFCqhZsybWr1+Py5cvw9LSEt27d4eRkRFsbGykLrdElNfPnKqqKvr27Yu+ffsiODgYN27cQEREBP7880+YmJigS5cuuH//PjZs2AC5XI5q1aohJiYGAwYMwFdffSV1+YRPfLD+pyA3Nxc7d+6Ep6cnVFRUyu1frIWuXbuGZcuWYdasWcjPz8eUKVOQkZGBJUuWoEmTJlKXR6+wYMECPHv2DDNmzMDSpUvx8OFDaGpqYsmSJbh06RJMTU2hr68vdZlv9br7vA0YMABubm44duwY+vbti1q1amHPnj0wMzNDVFQUFi9ejJUrV+Kzzz4rk2fA3oWPjw969epVrn6C6VOTkZGBIUOGQFdXF5mZmTA1NcWoUaOgpaWFp0+fIjo6utyNN/3Y8YzYR67wLN/H4mPtRv6Yff311/Dy8sLNmzcxc+ZMzJs3D7GxsSgoKECLFi2kLq/YXrzPGwDFfd4mTJiA3bt3w8PDA2vWrMG+ffugpqaGNWvWIDk5GcDzX68ob10/4eHhOHr0KFq1aoUqVapgyZIlUFVVRe3ataUujd4gISEBWVlZCAwMREJCAvz8/DB8+HCMHj0abm5u5f7n+j5GPCNG5U5CQgKOHj36UXUjf+y2bduG9evX48SJE1KX8kG8vb1Ro0YN+Pj4KO4wDzzvzhoyZAhiYmIQHx+PxYsXK8aRFY6fKm+ysrKwadMmXL9+HUlJSfD09ISHh4fUZdEbCCFw69YtBAQEwN/fHzo6OsjPz8ezZ8/KzFW59DKeEaNyp3bt2jwLVs64u7srbvIJlL/bwRQaPXo0xowZAw8PjyL3eevYsSP++ecfjBo1CsOHD0dSUhJq1aqFWrVqSV3ye9PR0cHo0aORnp4ObW1tReiksktFRQVxcXHIyMhQ3EBXXV2dIayMK59HQyIqVwq7yFVVVcttCANef5+35ORkfPnll6hTpw727t1brgPYf+np6TGElSNPnz6Fs7MzgPL1c3WfMn66iIjewfDhwzF48GBcu3YNxsbGmD17NvLz89G4cWMA/x/OiKTQt29fxR875f3irE8FgxgR0Tv4WO7zRh+n8nzG+VPFIEZE9I7K6z2niKjs4VWTRERERBLhOUwiIiIiiTCIEREREUmEQYyIiIhIIgxiRERERBJhECMiIiKSCIMYEZWIxMREmJmZYefOnUWeX79+Pb7//nuJqiIiKtsYxIioxKiqqmLhwoWIjY2VuhQionKBN3QlohKjra2NIUOG4Ntvv8WOHTugqampmHbnzh3MmTMHGRkZkMlkMDc3x7Jly6ClpYWmTZtiyJAhCA0NRWZmJsaOHYuQkBBERUXB2NgYv/zyC3R1dRETE4N58+bhyZMnKCgogJeXF+9qT0TlGs+IEVGJGj16NHR1deHv71/k+cDAQPTo0QOBgYE4cuQIEhMTcfLkSQBAbm4uqlatil27dqFHjx6YMWMGpk+fjoMHDyI9PR3Hjh1Dfn4+vL29MXnyZAQFBWHbtm3YsGEDrl69WvorSURUQnhGjIhKlKqqKhYtWoQePXqgQ4cOiuenTJmCM2fO4Ndff0VcXBwePnyIzMxMxXQHBwcAwOeffw5TU1NUq1YNAPDZZ5/h6dOniIuLQ3x8PKZNm6Z4TXZ2Nm7evAkrK6vSWTkiohLGIEZEJa5GjRqYPXs2pk6dih49egAAJk2ahIKCAjg5OaFTp064f/8+XvyFNQ0NjVf+v1BBQQH09fWxb98+xXOPHj2Cvr6+8laEiEjJ2DVJRErh6OiIjh07YvPmzQCA06dPY8yYMejWrRsAICwsDAUFBcVur27dutDW1lYEsfv378PFxQXh4eElXzwRUSnhGTEiUpoZM2bg0qVLAICJEydizJgx0NXVhZ6eHlq1aoX4+Phit6WpqYnVq1dj3rx5WLduHfLz8zF+/Hi0aNFCWeUTESmdinixb4CIiIiISg27JomIiIgkwiBGREREJBEGMSIiIiKJMIgRERERSYRBjIiIiEgiDGJEREREEmEQIyIiIpIIgxgRERGRRP4PjXvH4ogqBh0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 604.475x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g = sns.catplot(x = 'Name', y='Per',hue = 'Dataset',data=change, kind='bar',height=5,aspect=1.5, palette='dark')\n",
    "g.set_xticklabels(rotation=30)\n",
    "g.set(yticks=np.arange(0,53,5))\n",
    "g.fig.suptitle('Score Change after hyperparameter optimization for each Regressor')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Σχόλια"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - Παρατηρούμε ότι στις περιπτώσεις Ridge και Lasso Regression το γεγονός ότι $\\alpha\\rightarrow 0$, σημαίνει ότι το πρόβλημά μας τείνει σε αυτό της απλής γραμμικής παλινδρόμησης επομένως παίρνουμε τους ίδιους συντελεστές με αυτό, γιαυτό και οι 2 τιμές του $R^2$ συγκλίνουν προς αυτό της γραμμικής παλινδρόμησης.\n",
    " - Στα Decision Trees βλέπουμε μια μικρή αύξηση της τάξης του 1\\% στο test set μετά το optimization.\n",
    " - Το Multi Layer Perceptron χωρίς τα hidden layers, εμφανίζει μια 2\\% αύξηση στο train set και 8\\% αύξηση στο test set, δηλαδή μπορεί και γενικεύει καλύτερα.  \n",
    " - Το Default Multi Layer Perceptron όμως με 1 παραπάνω hidden layer, εμφανίζει ήδη by default λίγο μεγαλύτερη ακρίβεια από το αρχικό, λόγω της ικανότητας του να αποθηκεύει περισσότερη πληροφορία, και μετά το optimization όπου έχουμε 3 hidden layers με 512, 256 και 128 παραμέτρους αντίστοιχα, έχουμε 16\\% παραπάνω accuracy στο train set και 6\\% στο test set.\n",
    " - Το MLP με τα hidden layers έχει 10\\% παραπάνω απόδοση από ότι αυτό χωρίς.\n",
    " - O SVR μετά το optimization εμφανίζει 3\\% αύξηση στο train set και 2\\% μείωση στο test set. Αυτό είναι μια περίπτωση overfitting.\n",
    " - O Extra Trees Regressor μετά το Grid Search δείχνει ότι προτιμάει τις default υπερπαραμέτρους του, επομένως δεν έχει καμία αλλαγή. Αυτό δεν σημαίνει ότι αυτές είναι οι βέλτιστοι υπερπαράμετροι, για να βρεθούν όμως, πρέπει να επεκταθεί το εύρος των υπερπαραμέτρων που ψάχνουμε.\n",
    " - Ο XGBoost Regressor μετά την βελτιστοποίηση δείχνει 2\\% αύξηση στο train set και 4\\% αύξηση στο test set, και είναι ο πιο ακριβής regressor στο σύνολο όλων, με test set accuracy 82\\%.\n",
    " - Στα περισσότερα κελιά που υπάρχει ενδιαφέρον για το χρόνο εκτέλεσης έχει προστεθεί η built-in magic command `%%time` ώστε να φαίνεται ο χρόνος εκτέλεσης. Για όλους τους υπολογισμούς χρησιμοποιήθηκαν jupyter notebooks σε local runtime με επεξεργαστή i7-9th gen. Στις πιο απαιτητικές εργασίες, δηλαδή στα Grid Search, προστέθηκε η εντολή `n_jobs=-1` ώστε να εκτελούνται υπολογισμοί εν παραλλήλω, αξιοποιώντας και τα 12 threads του επεξεργαστή. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}