divisao_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Projeto Introdução a Redes Neurais\n",
    "\n",
    "**Equipe:**\n",
    "- Eduardo Luiz Silva - <els6@cin.ufpe.br>\n",
    "- Ianní Muliterno - <iwmb1@de.ufpe.br>\n",
    "- Tu Chin Hung - <tch1@de.ufpe.br>\n",
    "- Wellington Barbosa de Almeida - <wba@cin.ufpe.br>\n",
    "\n",
    "**Professor:** Germano Crispim Vasconcelos - <gcv@cin.ufpe.br>\n",
    "\n",
    "**Objetivo:**\n",
    "> Realizar um estudo experimental sobre a aplicação de modelos de redes neurais em um problema do mundo real.\n",
    "\n",
    "**Problema**\n",
    "> Análise de Risco de Crédito, Com base no perfil de clientes, decidir a quem conceder crédito (risco de inadimplência) \n",
    "\n",
    "## Etapa 1 - Divisão da base de dados:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Base de dados\n",
    "\n",
    "[Análise de Risco de Crédito](http://www.cin.ufpe.br/~gcv/web_lci/TRN) (+-400K Registros para Treinamento e +-130K Registros para Teste). Tamanho: 731 MB "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importando os modulos básicos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import scipy as sp\n",
    "import pandas as pd\n",
    "import csv\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import accuracy_score, recall_score, precision_score, f1_score\n",
    "from sklearn.metrics import roc_auc_score, average_precision_score\n",
    "\n",
    "import matplotlib\n",
    "matplotlib.use('nbagg')\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Antes de seguir para o próximo passo, lembre de realizar o download da base de dados na pasta `/data` e executar o script python para converte-la para `csv`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convertendo em um [dataframe pandas](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_set = pd.read_csv(\"data/trn.csv\", sep=',', header=0, low_memory=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>INDEX</th>\n",
       "      <th>UF_1</th>\n",
       "      <th>UF_2</th>\n",
       "      <th>UF_3</th>\n",
       "      <th>UF_4</th>\n",
       "      <th>UF_5</th>\n",
       "      <th>UF_6</th>\n",
       "      <th>UF_7</th>\n",
       "      <th>IDADE</th>\n",
       "      <th>SEXO_1</th>\n",
       "      <th>...</th>\n",
       "      <th>CEP4_7</th>\n",
       "      <th>CEP4_8</th>\n",
       "      <th>CEP4_9</th>\n",
       "      <th>CEP4_10</th>\n",
       "      <th>CEP4_11</th>\n",
       "      <th>CEP4_12</th>\n",
       "      <th>CEP4_13</th>\n",
       "      <th>CEP4_14</th>\n",
       "      <th>IND_BOM_1_1</th>\n",
       "      <th>IND_BOM_1_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.135098</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.273504</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.281910</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.225741</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.480403</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 246 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   INDEX  UF_1  UF_2  UF_3  UF_4  UF_5  UF_6  UF_7     IDADE  SEXO_1  \\\n",
       "0      0     1     1     1     0     0     0     0  0.135098       1   \n",
       "1      1     1     0     1     0     0     1     0  0.273504       1   \n",
       "2      2     1     0     1     0     0     1     0  0.281910       0   \n",
       "3      3     1     1     1     0     0     0     0  0.225741       0   \n",
       "4      4     1     1     0     0     0     1     0  0.480403       0   \n",
       "\n",
       "      ...       CEP4_7  CEP4_8  CEP4_9  CEP4_10  CEP4_11  CEP4_12  CEP4_13  \\\n",
       "0     ...            0       0       1        1        0        1        1   \n",
       "1     ...            0       1       0        1        1        0        0   \n",
       "2     ...            1       1       0        0        0        0        1   \n",
       "3     ...            1       1       0        1        1        0        1   \n",
       "4     ...            1       1       1        0        0        1        0   \n",
       "\n",
       "   CEP4_14  IND_BOM_1_1  IND_BOM_1_2  \n",
       "0        1            0            1  \n",
       "1        0            1            0  \n",
       "2        0            1            0  \n",
       "3        0            1            0  \n",
       "4        1            1            0  \n",
       "\n",
       "[5 rows x 246 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_set.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Estatísticas sobre as variáveis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>INDEX</th>\n",
       "      <th>UF_1</th>\n",
       "      <th>UF_2</th>\n",
       "      <th>UF_3</th>\n",
       "      <th>UF_4</th>\n",
       "      <th>UF_5</th>\n",
       "      <th>UF_6</th>\n",
       "      <th>UF_7</th>\n",
       "      <th>IDADE</th>\n",
       "      <th>SEXO_1</th>\n",
       "      <th>...</th>\n",
       "      <th>CEP4_7</th>\n",
       "      <th>CEP4_8</th>\n",
       "      <th>CEP4_9</th>\n",
       "      <th>CEP4_10</th>\n",
       "      <th>CEP4_11</th>\n",
       "      <th>CEP4_12</th>\n",
       "      <th>CEP4_13</th>\n",
       "      <th>CEP4_14</th>\n",
       "      <th>IND_BOM_1_1</th>\n",
       "      <th>IND_BOM_1_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>10000.00000</td>\n",
       "      <td>10000.00000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.00000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>1.000000e+04</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>4999.50000</td>\n",
       "      <td>0.89240</td>\n",
       "      <td>0.683100</td>\n",
       "      <td>0.472600</td>\n",
       "      <td>0.294400</td>\n",
       "      <td>0.25370</td>\n",
       "      <td>0.219500</td>\n",
       "      <td>0.184300</td>\n",
       "      <td>4.565028e-01</td>\n",
       "      <td>0.522300</td>\n",
       "      <td>...</td>\n",
       "      <td>0.428300</td>\n",
       "      <td>0.420500</td>\n",
       "      <td>0.422100</td>\n",
       "      <td>0.462100</td>\n",
       "      <td>0.440000</td>\n",
       "      <td>0.430600</td>\n",
       "      <td>0.434100</td>\n",
       "      <td>0.426700</td>\n",
       "      <td>0.658800</td>\n",
       "      <td>0.341200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2886.89568</td>\n",
       "      <td>0.30989</td>\n",
       "      <td>0.465291</td>\n",
       "      <td>0.499274</td>\n",
       "      <td>0.455795</td>\n",
       "      <td>0.43515</td>\n",
       "      <td>0.413929</td>\n",
       "      <td>0.387748</td>\n",
       "      <td>2.542518e-01</td>\n",
       "      <td>0.499527</td>\n",
       "      <td>...</td>\n",
       "      <td>0.494857</td>\n",
       "      <td>0.493664</td>\n",
       "      <td>0.493919</td>\n",
       "      <td>0.498586</td>\n",
       "      <td>0.496412</td>\n",
       "      <td>0.495185</td>\n",
       "      <td>0.495663</td>\n",
       "      <td>0.494623</td>\n",
       "      <td>0.474136</td>\n",
       "      <td>0.474136</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.506237e-16</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>2499.75000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.515817e-01</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>4999.50000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.368994e-01</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>7499.25000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6.596725e-01</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>9999.00000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 246 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             INDEX         UF_1          UF_2          UF_3          UF_4  \\\n",
       "count  10000.00000  10000.00000  10000.000000  10000.000000  10000.000000   \n",
       "mean    4999.50000      0.89240      0.683100      0.472600      0.294400   \n",
       "std     2886.89568      0.30989      0.465291      0.499274      0.455795   \n",
       "min        0.00000      0.00000      0.000000      0.000000      0.000000   \n",
       "25%     2499.75000      1.00000      0.000000      0.000000      0.000000   \n",
       "50%     4999.50000      1.00000      1.000000      0.000000      0.000000   \n",
       "75%     7499.25000      1.00000      1.000000      1.000000      1.000000   \n",
       "max     9999.00000      1.00000      1.000000      1.000000      1.000000   \n",
       "\n",
       "              UF_5          UF_6          UF_7         IDADE        SEXO_1  \\\n",
       "count  10000.00000  10000.000000  10000.000000  1.000000e+04  10000.000000   \n",
       "mean       0.25370      0.219500      0.184300  4.565028e-01      0.522300   \n",
       "std        0.43515      0.413929      0.387748  2.542518e-01      0.499527   \n",
       "min        0.00000      0.000000      0.000000  5.506237e-16      0.000000   \n",
       "25%        0.00000      0.000000      0.000000  2.515817e-01      0.000000   \n",
       "50%        0.00000      0.000000      0.000000  4.368994e-01      1.000000   \n",
       "75%        1.00000      0.000000      0.000000  6.596725e-01      1.000000   \n",
       "max        1.00000      1.000000      1.000000  1.000000e+00      1.000000   \n",
       "\n",
       "           ...             CEP4_7        CEP4_8        CEP4_9       CEP4_10  \\\n",
       "count      ...       10000.000000  10000.000000  10000.000000  10000.000000   \n",
       "mean       ...           0.428300      0.420500      0.422100      0.462100   \n",
       "std        ...           0.494857      0.493664      0.493919      0.498586   \n",
       "min        ...           0.000000      0.000000      0.000000      0.000000   \n",
       "25%        ...           0.000000      0.000000      0.000000      0.000000   \n",
       "50%        ...           0.000000      0.000000      0.000000      0.000000   \n",
       "75%        ...           1.000000      1.000000      1.000000      1.000000   \n",
       "max        ...           1.000000      1.000000      1.000000      1.000000   \n",
       "\n",
       "            CEP4_11       CEP4_12       CEP4_13       CEP4_14   IND_BOM_1_1  \\\n",
       "count  10000.000000  10000.000000  10000.000000  10000.000000  10000.000000   \n",
       "mean       0.440000      0.430600      0.434100      0.426700      0.658800   \n",
       "std        0.496412      0.495185      0.495663      0.494623      0.474136   \n",
       "min        0.000000      0.000000      0.000000      0.000000      0.000000   \n",
       "25%        0.000000      0.000000      0.000000      0.000000      0.000000   \n",
       "50%        0.000000      0.000000      0.000000      0.000000      1.000000   \n",
       "75%        1.000000      1.000000      1.000000      1.000000      1.000000   \n",
       "max        1.000000      1.000000      1.000000      1.000000      1.000000   \n",
       "\n",
       "        IND_BOM_1_2  \n",
       "count  10000.000000  \n",
       "mean       0.341200  \n",
       "std        0.474136  \n",
       "min        0.000000  \n",
       "25%        0.000000  \n",
       "50%        0.000000  \n",
       "75%        1.000000  \n",
       "max        1.000000  \n",
       "\n",
       "[8 rows x 246 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_set.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pre-processamento e limpeza da base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove instancias duplicadas\n",
    "data_set.drop_duplicates(inplace=True)\n",
    "\n",
    "# merge das duas ultimas colunas na coluna 'y'\n",
    "data_set = data_set.drop('IND_BOM_1_2', 1)\n",
    "data_set = data_set.rename(columns={'IND_BOM_1_1': 'y'})\n",
    "\n",
    "# renomeia algumas colunas por conveniência\n",
    "data_set.rename(columns={'INDEX': 'index'}, inplace=True)\n",
    "\n",
    "# encontra as variavies categoricas\n",
    "for col in data_set.columns:\n",
    "    if ((data_set[col].quantile() == 1.0) | (data_set[col].quantile() == 0.0)):\n",
    "        data_set[col] = data_set[col].astype('category')\n",
    "\n",
    "# Embaralha o dataset\n",
    "data_set = data_set.sample(frac=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Computa quantidade de exemplos da cada classe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Quantidade de instancias da classe 1: 6588\n",
      "Quantidade de instancias da classe 2: 3412\n"
     ]
    }
   ],
   "source": [
    "# Classe 1\n",
    "A = data_set.loc[data_set.y == 1]\n",
    "size_A = A['index'].count()\n",
    "\n",
    "# Classe 2\n",
    "B = data_set.loc[data_set.y == 0]\n",
    "size_B = B['index'].count()\n",
    "\n",
    "print(\"Quantidade de instancias da classe 1: {}\".format(A['index'].count()))\n",
    "print(\"Quantidade de instancias da classe 2: {}\".format(B['index'].count()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Particionamento dos Dados com K-folds\n",
    "Utiliza *10-fold Cross Validation*\n",
    "\n",
    "Em cada rodada:\n",
    "- 1 fold para test\n",
    "- 1 fold para validação\n",
    "- 8 folds para treino"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TO DO: 4-fold\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "take_nd() got an unexpected keyword argument 'axis'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-9-1057f49e4290>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     18\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdata_set\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0miloc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m \u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_test\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_test_split\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtest_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;36m4\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrandom_state\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m42\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstratify\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     21\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     22\u001b[0m \u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX_val\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_val\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_test_split\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtest_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrandom_state\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m42\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstratify\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0my_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/wba/.local/lib/python3.5/site-packages/sklearn/model_selection/_split.py\u001b[0m in \u001b[0;36mtrain_test_split\u001b[0;34m(*arrays, **options)\u001b[0m\n\u001b[1;32m   2057\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2058\u001b[0m     return list(chain.from_iterable((safe_indexing(a, train),\n\u001b[0;32m-> 2059\u001b[0;31m                                      safe_indexing(a, test)) for a in arrays))\n\u001b[0m\u001b[1;32m   2060\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2061\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/wba/.local/lib/python3.5/site-packages/sklearn/model_selection/_split.py\u001b[0m in \u001b[0;36m<genexpr>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m   2057\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2058\u001b[0m     return list(chain.from_iterable((safe_indexing(a, train),\n\u001b[0;32m-> 2059\u001b[0;31m                                      safe_indexing(a, test)) for a in arrays))\n\u001b[0m\u001b[1;32m   2060\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2061\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/wba/.local/lib/python3.5/site-packages/sklearn/utils/__init__.py\u001b[0m in \u001b[0;36msafe_indexing\u001b[0;34m(X, indices)\u001b[0m\n\u001b[1;32m    158\u001b[0m                                    indices.dtype.kind == 'i'):\n\u001b[1;32m    159\u001b[0m             \u001b[0;31m# This is often substantially faster than X[indices]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 160\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtake\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mindices\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    161\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    162\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mindices\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: take_nd() got an unexpected keyword argument 'axis'"
     ]
    }
   ],
   "source": [
    "\n",
    "'''\n",
    "# Classe A\n",
    "X = A.iloc[:, :-1]\n",
    "y = A.iloc[:, -1]\n",
    "AX_train, AX_test, Ay_train, Ay_test = train_test_split(X, y, test_size=1/4, random_state=42, stratify=y)\n",
    "\n",
    "AX_train, AX_val, Ay_train, Ay_val = train_test_split(AX_train, Ay_train, test_size=1/3, random_state=42, stratify=Ay_train)\n",
    "\n",
    "# Classe B\n",
    "X = B.iloc[:, :-1]\n",
    "y = B.iloc[:, -1]\n",
    "BX_train, BX_test, By_train, By_test = train_test_split(X, y, test_size=1/4, random_state=42, stratify=y)\n",
    "\n",
    "BX_train, BX_val, By_train, By_val = train_test_split(BX_train, By_train, test_size=1/3, random_state=42, stratify=By_train)\n",
    "'''\n",
    "X = data_set.iloc[:, :-1].values\n",
    "y = data_set.iloc[:, -1].values\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=1/4, random_state=42, stratify=y)\n",
    "\n",
    "X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=1/3, random_state=42, stratify=y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Classe 1(A): X Treino: {}\".format(AX_train['index'].count()))\n",
    "print(\"Classe 1(A): X validação: {}\".format(AX_val['index'].count()))\n",
    "print(\"Classe 1(A): X test: {}\".format(AX_test['index'].count()))\n",
    "print(\"sum: {}\".format(AX_train['index'].count() + AX_val['index'].count() + AX_test['index'].count()))\n",
    "\n",
    "print(\"Classe 0(B): X Treino: {}\".format(BX_train['index'].count()))\n",
    "print(\"Classe 0(B): X validação: {}\".format(BX_val['index'].count()))\n",
    "print(\"Classe 0(B): X test: {}\".format(BX_test['index'].count()))\n",
    "print(\"sum: {}\".format(BX_train['index'].count() + BX_val['index'].count() + BX_test['index'].count()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Replicação (oversampling) da Classe Minoritária (classe 1) com [SMOTE](http://contrib.scikit-learn.org/imbalanced-learn/stable/generated/imblearn.over_sampling.SMOTE.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "3412 - 1647 -> B_test\n",
    "\n",
    "1765 -> 75% -> SMOTE(1323) -> 3294\n",
    "1765 -> 25% -> SMOTE(441) -> 1647\n",
    "\n",
    "\n",
    "\n",
    "1765 -> 75% -> SMOTE(1323) -> 3294\n",
    "1765 -> 25% -> SMOTE(441) -> 1647\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "# Alternativa 1: reduz a classe majoritaria até a minoritaria\n",
    "if(size_A < size_B):\n",
    "    B = B.iloc[:size_A]\n",
    "else:\n",
    "    A = A.iloc[:size_B]\n",
    "'''\n",
    "'''\n",
    "# Alternativa 2: SMOTE\n",
    "from collections import Counter\n",
    "from sklearn.datasets import make_classification\n",
    "from imblearn.over_sampling import SMOTE\n",
    "sm = SMOTE(random_state=42)\n",
    "\n",
    "X_train_to_smote = pd.concat([AX_train, BX_train]).values\n",
    "y_train_to_smote = pd.concat([Ay_train, By_train]).values\n",
    "\n",
    "X_val_to_smote = pd.concat([AX_val, BX_val]).values\n",
    "y_val_to_smote = pd.concat([Ay_val, By_val]).values\n",
    "\n",
    "X_train_smoted, y_train_smoted = sm.fit_sample(X_train_to_smote, y_train_to_smote)\n",
    "X_val_smoted, y_val_smoted = sm.fit_sample(X_val_to_smote, y_val_to_smote)\n",
    "\n",
    "print(\"Total com smote: X Treino: {}\".format(np.count_nonzero(y_train_smoted)))\n",
    "print(\"Total com smote: X validação: {}\".format(np.count_nonzero(X_val_smoted)))\n",
    "\n",
    "pd.DataFrame(X_train_smoted)\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Normaliza os dados"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(pd.concat([AX_train,BX_train]))\n",
    "y_train = pd.concat([Ay_train,By_train])\n",
    "X_val = scaler.transform(pd.concat([AX_val,BX_val]))\n",
    "y_val = pd.concat([Ay_val,By_val])\n",
    "X_test = scaler.transform(pd.concat([AX_test,BX_test]))\n",
    "y_test = pd.concat([Ay_test,By_test])\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Treino"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "input_dim = X_train.shape[1]\n",
    "\n",
    "classifier = Sequential()\n",
    "\n",
    "classifier.add(Dense(16, activation='tanh', input_dim=input_dim))\n",
    "\n",
    "classifier.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "classifier.compile(optimizer='adam', loss='mean_squared_error')\n",
    "\n",
    "# Treino\n",
    "history = classifier.fit(X_train, y_train, batch_size=64, epochs=100000, callbacks=[EarlyStopping(patience=3)], validation_data=(X_val, y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def extract_final_losses(history):\n",
    "    \"\"\"Função para extrair o melhor loss de treino e validação.\n",
    "    \n",
    "    Argumento(s):\n",
    "    history -- Objeto retornado pela função fit do keras.\n",
    "    \n",
    "    Retorno:\n",
    "    Dicionário contendo o melhor loss de treino e de validação baseado \n",
    "    no menor loss de validação.\n",
    "    \"\"\"\n",
    "    train_loss = history.history['loss']\n",
    "    val_loss = history.history['val_loss']\n",
    "    idx_min_val_loss = np.argmin(val_loss)\n",
    "    return {'train_loss': train_loss[idx_min_val_loss], 'val_loss': val_loss[idx_min_val_loss]}\n",
    "\n",
    "def plot_training_error_curves(history):\n",
    "    \"\"\"Função para plotar as curvas de erro do treinamento da rede neural.\n",
    "    \n",
    "    Argumento(s):\n",
    "    history -- Objeto retornado pela função fit do keras.\n",
    "    \n",
    "    Retorno:\n",
    "    A função gera o gráfico do treino da rede e retorna None.\n",
    "    \"\"\"\n",
    "    train_loss = history.history['loss']\n",
    "    val_loss = history.history['val_loss']\n",
    "    \n",
    "    fig, ax = plt.subplots()\n",
    "    ax.plot(train_loss, label='Train')\n",
    "    ax.plot(val_loss, label='Validation')\n",
    "    ax.set(title='Training and Validation Error Curves', xlabel='Epochs', ylabel='Loss (MSE)')\n",
    "    ax.legend()\n",
    "    plt.show()\n",
    "\n",
    "def compute_performance_metrics(y, y_pred_class, y_pred_scores=None):\n",
    "    accuracy = accuracy_score(y, y_pred_class)\n",
    "    recall = recall_score(y, y_pred_class)\n",
    "    precision = precision_score(y, y_pred_class)\n",
    "    f1 = f1_score(y, y_pred_class)\n",
    "    performance_metrics = (accuracy, recall, precision, f1)\n",
    "    if y_pred_scores is not None:\n",
    "        auroc = roc_auc_score(y, y_pred_scores)\n",
    "        aupr = average_precision_score(y, y_pred_scores)\n",
    "        performance_metrics = performance_metrics + (auroc, aupr)\n",
    "    return performance_metrics\n",
    "\n",
    "def print_metrics_summary(accuracy, recall, precision, f1, auroc=None, aupr=None):\n",
    "    print()\n",
    "    print(\"{metric:<18}{value:.4f}\".format(metric=\"Accuracy:\", value=accuracy))\n",
    "    print(\"{metric:<18}{value:.4f}\".format(metric=\"Recall:\", value=recall))\n",
    "    print(\"{metric:<18}{value:.4f}\".format(metric=\"Precision:\", value=precision))\n",
    "    print(\"{metric:<18}{value:.4f}\".format(metric=\"F1:\", value=f1))\n",
    "    if auroc is not None:\n",
    "        print(\"{metric:<18}{value:.4f}\".format(metric=\"AUROC:\", value=auroc))\n",
    "    if aupr is not None:\n",
    "        print(\"{metric:<18}{value:.4f}\".format(metric=\"AUPR:\", value=aupr))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_training_error_curves(history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}