Insuranceclaim.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Insuranceclaim.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyP3OYoMMrMJjFGRQQTVroO3",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/AbdulmalikGiwa/Insurance-Claim-Prediction/blob/master/Insuranceclaim.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gQnR8CcvFOh7",
        "colab_type": "text"
      },
      "source": [
        "# Introduction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pafhrj0E7wuC",
        "colab_type": "text"
      },
      "source": [
        "A predictive model to determine if a building will have an insurance claim during a certain period or not. A few features were provided which will be explored to select the important and best feaures to build the model after analysis is done"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eDyA--26FH3r",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "0e10b6c9-da9b-49ca-b1eb-b7aa4d02b04b"
      },
      "source": [
        "#importing all libraries needed\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import keras\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "from sklearn import preprocessing\n",
        "from sklearn.linear_model import LinearRegression\n",
        "from sklearn.preprocessing import  LabelEncoder"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-fMUWcj2F2WU",
        "colab_type": "code",
        "outputId": "03730a03-9949-4cb0-eaf9-f9a19bc18e19",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "#importing train set\n",
        "train = pd.read_csv('/content/train_data.csv')\n",
        "test = pd.read_csv('/content/test_data.csv')\n",
        "#Creating an entire column to fill train and test locations with tags to make seperating them easier later on\n",
        "train['col'] = 'train' \n",
        "test['col'] = 'test'\n",
        "#new dataframe combining train and test\n",
        "total = pd.concat([train,test], ignore_index=True) \n",
        "print(total.shape, train.shape, test.shape)"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(10229, 15) (7160, 15) (3069, 14)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0lSsivsaAy7l",
        "colab_type": "text"
      },
      "source": [
        "# Exploratory Analysis and Data Wrangling"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0w4kLY6j1h9O",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "outputId": "df0697be-0f6a-479e-d838-6d6e4e33339b"
      },
      "source": [
        "train['Claim'].value_counts().plot(kind='pie', labels=['No Claim','Claim'],autopct='%1.0f%%')"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7f3c6dfc5b38>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPUAAADnCAYAAADGrxD1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAZGklEQVR4nO3dd5xcZb3H8c9vSzaVhBBIIYQTwNCDUqSLqNgGvRApAkqIkCsgNgQ5yuUyKMpYQJEigoUiXqRJ8chLkSJNDISSTQNCPIlAgNQJKZst89w/ztnUze7M7sw85zzze79e80p2dmbPN5BvTpnnPI8YY1BKuaPOdgClVHlpqZVyjJZaKcdoqZVyjJZaKcdoqZVyjJZaKcdoqZVyjJZaKcdoqZVyjJZaKcdoqZVyjJZaKcdoqZVyjJZaKcdoqZVyjJZaKcdoqZVyjJZaKcdoqZVyjJZaKcdoqZVyjJZaKcdoqZVyjJZaKcdoqZVyjJZaKcc02A6gys/zg0ZgF2AMMBoYFf/a+fvBQD+gcbNHHbAWWAOsAvLAcmAZsAj4NzAfmB/mMour9ydSpRBdIC+9PD8QYDywL7BP/NgXmEBU0kp6j6jk84AXgGnAc2Eus6LC21U90FKniOcH9cCBwFHAkcDhwLZWQ23KEJV8Wvx4PMxlZtiNVHu01Ann+cFo4Djg48CHgWFWA5XuDeAv8ePvYS6z2nIe52mpE8jzg5HACcBJwBG4c0FzHfAEcD9wp56XV4aWOiE8PxgGnAKcTHRo7UqRt6Yd+CtwK3B/mMuss5zHGVpqyzw/2Bf4KnAaMNByHFuWAb8HbgxzmVm2w6SdltoCzw8agEnAeUR7ZbXBQ8CPw1zmcdtB0kpLXUWeHzQBZwMXAjtajpN004AfAfeFuUzBdpg00VJXQTwY5EzgYmCs5Thp8yqQA27RchdHS11B8efKpwOXEA0SUb3XDHwrzGUeth0k6bTUFeL5waeBq4DdbWdxzEPABWEuM9t2kKTSUpeZ5wfjgZ8Dn7WdxWEdwK+BS/Sz7i1pqcskvqJ9AfC/wADLcWrFEuCrYS5zh+0gSaKlLgPPDw4EfgNMtJ2lRt0LnBvmMu/YDpIEWuo+8PygDvg28D0qf1eU6t5S4OthLnO77SC2aal7yfODMcBtwEdsZ1GbuA84M8xlltkOYouWuhc8P/gM8FtghO0sqksLgZPCXOZftoPYoKUuQXwx7Cqisdoq2dqIPte+xnaQatNSFym+i+pu4KO2s6iS3AKcHeYyLbaDVEuvb+8TESMiV2709QUiki3xZ3xKRJ4Xkdki8mLnzxORrIhc0MN7zxaR03sVvkSeH+wKPIsWOo0mA0/Fk03UhL7cs7sOmCQivTqvFJF9gGuBLxhj9iKapmdese83xtxgjLm1N9suhecHRwL/QkeGpdkBwNOeH7zPdpBq6Eup24EbgW9u/g0R8UTkURGZISKPiMi4Lt7/beAHxpi5AMaYDmPML7v4WVNF5DkReVlE7hGRgfHz6/fmIvK4iPws3uvPEZGDROReEXlNRC7v7R/Q84PTgb8D2/X2Z6jEGE9U7ANtB6m0vs6ucR1wmogM3ez5a4BbjDETgduBX3Tx3n2A6UVs415jzEHGmP2AOUR3O3Wl1RhzIHAD0XQ5X4m3cYaIlFxKzw/OIzof61fqe1VibQ885vnBx2wHqaQ+ldoYs5JoOpqvbfatQ4E/xL+/jWierd7aR0SeFJFmotlB9t7K6x6If20GZhljFhlj1hHNU71TKRv0/OAbRP8wKfcMBgLPDz5vO0illGMerJ8T7T0Hlfi+WUTnOj25GTjPGLMvcBnQfyuv65zjqrDR7zu/LnrRAs8PvgX8rNjXq1TqB9zuarH7XGpjzDLgTjY9LH4G6PwPdhrwZBdv/QnwXRGZACAidSJydhevGwIsEpHG+GdVjOcHFwE/reQ2VGLUAbfFA4mcUq4ZK69k09FVXwWmiMgM4IvA1zd/gzFmBvAN4P9EZA4wk2ipmM1dQnT1+WlgbpnybsHzg28TzbChakcDcKfnB04N9dXBJ4DnB5OJDvNVbVoFHBPmMs/aDlIONV9qzw8+CTyILhZY65YDR4W5TLPtIH1V06X2/GAi0WH9YNtZVCIsAA4Mc5kltoP0heurQGyV5wejgD+jhVYb7AzcHd+4k1o1WWrPD/oRDVAp6fNrVROOAq62HaIvarLURB9bfdB2CJVY53p+MNV2iN6quXNqzw8mAffYzqESrxX4SJjLPG07SKlqqtTx9L0vkL41npUdbwATw1xmue0gpaiZw+946Zs70EKr4o0FfmU7RKlqptREo8X0PFqV6kTPD86wHaIUNXH47fnBYcBTgNjOolIpD+wb5jL/sR2kGM7vqePD7hvRQqveG0q0zE8qOF9q4CK2fg+2UsX6uOcHU2yHKIbTh9+eH0wAZgBNtrMoJ7wLTAhzmbztIN1xdk/t+YEQHXZroVW57ABcajtET5wtNdGECkfZDqGcc57nB3vaDtEdJ0sdj+3+vu0cykmNJHxsuJOlBs4GPNshlLOO8fzgONshtsa5C2WeHwwmmkF0e9tZlNPmAPuEuUzBdpDNubin/hZaaFV5ewKfsx2iK07tqT0/2B54nWgGUqUqbQbw/jCXSVSJUj3DQxfOp8qFblv6Bosf+NH6r9tXvM2wI77Aurfm0rbsDQAKLaup6z+IMVOuoeWN2Sz72/VIfQMjPnMhjcN3pNCyisX3/4gdTroMERcPnpw1Efgs0YQbieFMqeNz6a7mDa+oxu3GMmZKtJiHKXTwxvWTGTjhULY56L/Wv2bZo7+mrila62Dlc39ihxOytK98h/deeojhHzmL/DN/ZOihJ2qh0+kSElZql/4WTcXybZUtC16mcdhoGobusP45Ywxr5j7FoD0/BIDUNWDa12Ha1iF1DbQtX0T7e0voP26irdiqbw7w/OBTtkNszIlSe35QR7SAgFWr5zzBwLi8nda9MYv6QcNoHL4jAEMPOZElf76K/LN3MWT/Y1nxxK0MO/ILNuKq8tlisQqbnCg1cCzRUqXWmI421s6bxqA9Nl0LcPXsf6zfSwP0G7kLo0+/klGnXEF7/m3qBw8HYPH9P2LJgz+lY3WqJtlQkWM8P/Bsh+jkSqmt76XXzp9Ov5G7Uj9o2/XPmUIHa179JwP3+NAWrzfGROfSh32eFU//gW0/PIXB+32CldMfrGZsVR51wFm2Q3RKfanjecc+ajvH5ntkgJbwJRq3G0vDNiO2fP3MRxmwy4HUDxiCaVsHIiAS/V6l0ZSkzBee+lIDp2J5AoRCawst4UsM3P2wTZ5fPeeJLYoOUGhrYdXMvzNk/wwA2xx0HO/elWX5Izcx5P2JuuaiijeG6DTQutQPPvH8YBawl+0cSgEPhbnMp22HSHWpPT/YD3jJdg6lYh3AmDCXeddmiLQffp9qO4BSG6kHjrcdIrWljmc2+bztHEptxvpNHqktNXAwMM52CKU2c7TnB1ZHNqa51J+wHUCpLjQAn7QZIM2lPsZ2AKW2wupHW6kstecHQ4gOv5VKIqtHkaksNfBhHLptVDlnhOcHu9vaeFpLrYfeKukOt7XhtJba+lhvpXpwRM8vqYzUlTo+n070ZOpKoXvqkuyHrmCpkm9CPBFm1aWx1O+3HUCpIh1qY6NaaqUqZx8bG01jqT9gO4BSRdrDxkZTVep4ZgldQF6lhZXPqlNVamBXdL1plR5a6iJYnTFUqRIN9fxgZLU3WtRQSxEZBpxOtDzs+vcYY75WmVhb5VV5e0r11e7AO9XcYLHjp/8CPAs0AzaX7tT7p1XajAeeqOYGiy11f2PM+RVNUpwxtgMoVaIt54eusGLPqW8TkakiMlpEhnc+Kpqsa1pqlTZVL3Wxe+pW4CfAxUDn9KMG2KUSoboxqsrbU6qvtqv2Bost9beA3YwxSyoZpgjbWN6+UqVK7OH3PGBNJYMUaYDtAEqVKLF76tXASyLyGLB+sScLH2lpqVXaJLbU98UP27TUKm36V3uDRZXaGHNLpYP0JB73rfOSqbSpr/YGuy2JiNxpjDlJRJrZcNV7PWPMxIol25LupVUaJavUwNfjX5OwRKfupSukgfa2V5smr8DCX0DXFZCVsLyq2+y2KMaYRfGvC6oTp1trbQdwVTsNjQWkrUEKVqbfcVkdZmX1t1kEETlERJ4TkVUi0ioiHSJS1bBhLtOC3XHnTlvJIKvLrzqsvdobLPZz6muBU4DXiM5tzwKuq1SobiThs3InvWFGvGc7g6MSW2qMMfOAemNMhzHmd9hZBExLXSGvmp06bGdwVNX/sSy21GtEpB/RAJQfi8g3S3hvOa22sM2a0FzwdEaZyqjqvdRQfDG/SHRl9DyiYu2EncW1tdQVMrMwflvbGRxV9VIXO/ik8+r3WuCyysXp0WKL23baK2Ynva21Mt6u9gZ7GnzS5aCTTlUefAKwsMrbqxmrGLhNwcjSOjFVH6vsuMTtqScBI4H/bPb8Tlj4FwgtdUWtpv/bQ1irpS6vxJ1T/wzIG2MWbPwA8vH3qk1LXUFvm+F52xkclLhSjzTGNG/+ZPycV5FE3dNSV9BrZsdW2xkclLhSD+vmezZusNBSV9Csgqfj68urDZhf7Y32VOrnRWTq5k+KyFnA9MpE6tYCdKhoxTSb8TpdVHnNJZuv+tFPT/8yfwP4k4icxoYSHwj0A46vZLCuhLnMWs8P5gO7VXvbtWBuYafRtjM4ZoaNjfZ0l9Y7wGEicjQbluUMjDGPVjzZ1r2Mlroi3mX49sawSoTBtrM44mUbGy128MljwGMVzlKsl7Ezmq0mtNDvrQG0TrCdwxFWSp22BfLAzrl8zVhshi6zncEhVg6/01jqabYDuOzfZvS6nl+livAu2byNAVrpK3WYyywBQts5XDXb7Cy2MzjiBVsbTl2pY/+wHcBVzYVd9CJZefzd1obTWuq/2Q7gqtlmnM5TVh5/tbXhtJb6Ybq5e0z13kIzcowxtNnOkXJvks3PtLXxVJY6zGUWAy/azuGiAnX1bTS8aTtHyj1sc+OpLHVMD8ErZDmDba9umnbWDr1BS626sNDsoBM89l4B3VP32tNUe+mDGjG3ME6vV/TeC2TzS20GSG2pw1ymFbjbdg4XzTC76LplvXeX7QCpLXXs97YDuGhWwdMpjXqnA7jNdoi0l/pJdOKEsnvdjNnRGP3IsBceJptfZDtEqksd5jIG+IPtHK5ZR7/+BeoqPg3Pf/IFjr5lNXtdt4q9r1/F1c9Gw84vebSFib9cxftvWMXHb1vNW+9F82LcM7uNva9fxZG/W83SNdFzry8rcPLdibmud7PtAJDyUsf0ELwCVjKw4jcjNNTBlR/vz+yvDObZMwdx3XNtzF7cwYWHNzHjnMG8dPZgjp3QwPf+EZX9mmmtPDd1EF8+oJE/NEdLVP3PYy1cfnQiFhdZAdxnOwQ4UOowl5mF3o5Zdm9WYcG80UPq2H90tCT2kCZhz+3reHOlYZumDfeUrG6Fzq/qBNa1w5o2aKyHJxe0M2pQHe/bLhHLat9BNp+IO9xcmWjuauBW2yFc8qoZW9inijfDhSsKvLiog4PHRgW9+JEWbp3RxtAm4bHJAwH4zhFNfOy21YwZUsfvjx/AiXet4Y4TBlYtYw9uth2gkxiT/ushnh80Et2OqUvHlMmU+of+eWnjbYdWY1urWg1H3byai49sYtKejZt874on19HSbrjs6P6bPH/ry60sW2s4ZGw9P32mlW37C1d/qj8DG63cOfoi2fz+NjbcldQffgOEuUwb0RraqkxmFsZ3Nz102bR1GD535xpO27dxi0IDnDaxkXvmbLrE85o2w80vtfGVg/px6ePruOW4ARwxrp7bZ1i7D+WHtjbcFSdKHbsBXRWzbF4xYyt+1GOM4cwHWthzRD3nH7rhYtdrSzcslX3/3Hb2GLHpX9OfPN3K1w7uR2O9sLYNRKLz7TVtVo465wL32tjw1jhx+N3J84PrgHNt53DF/KZTl9cJFVvi9qmF7Rz5uzXsu0MddfFR8w8/2sRvXmzjlSUF6gR2HlbHDZn+7LhNVOy33isw9cEWglOjc+m7ZrWR/cc6hvUX7jt5ANsPqvp+ajLZfKKu57hW6l2BV4jW0lZ91Nx05qwhsnZv2zkS7N/ABLL59h5fWUUuHX4T5jKvA7fYzuGKd8y2umBe936ctEKDY6WOXQYk4vPCtHvdjNEF87buLeB3tkN0xblSh7nMQuCXtnO4YGbB2/JytOp0eVIGm2zOuVLHvo/ea91nzWaXIbYzJNSLwK9sh9gaJ0sd5jLLgMtt50i7uYVxI21nSCADnEc2n9jVV50sdexaoivhqpfeZvhIY0jMLVAJcSvZ/DO2Q3TH2VLHM6NMRacS7pMWGnVm0Q3ywEW2Q/TE2VIDhLnMkyT43CcNluiCeRu7lGy+4veZ95XTpY5dBOjeppdCM7rFdoaEaAausx2iGM6XOsxlVgLn2M6RVrPNOF0wD1qJhoMmbqBJV5wvNUCYyzwI/NF2jjTSBfMA8MnmU7MiTE2UOnYOsMB2iLSZZXYeYTuDZQ8BP7cdohQ1U+owl1kOnAS6+Fsp4gXzUnHYWQFvA2eQzafqE5SaKTVAmMtMAy60nSNNOqhvaKe+Fi80GqLz6HdtBylVTZUaIMxlriZhN7UnXY0umHcV2Xwq12uruVLHvgTMtx0iLRaakbU2o8xTwHdth+itmix1mMvkgeOIRgipHswt7JSqc8o+mg8cTzaf2ttOa7LUAGEu0wxMQi+c9WimGd+/51c5IQ8cSzaf6tONmi01QJjLPEp0KF5Le6KSzayNBfNagc+Rzc+xHaSvarrUAGEu83vgYts5kmye2dH1BfM6r3Q/YjtIOdR8qQHCXOYKoimGVRdaaBpQQBJ/I0MfnE82f0cxLxSRUSJyh4i8LiLTReQvIjJBRGb28L4xIlKV9dS11BucB9xuO0RSrWRg6j6vLdKlZPNFjRgTEQH+BDxujNnVGHMA8B2gx8kkjDFvGWNO6FvU4mipY2Eu0wFMRovdpbfMiJW2M1TABWTz3yvh9UcDbcaY9Ud1xpiXgf90fi0inog8KSIvxI/DNnp+Zvz7M0TkPhF5WERCETlPRM4XkRdF5FkRGd6XP5SWeiMbFTtRk7MnwatmrEtDRQ1wLtn8lSW+bx96XmH1XeAYY8z+wMnAL7r5WZOAg4AfAGuMMR8A/gmcXmKuTWipNxMX+wx0RtJNzCyM72c7Q5lE/3+z+Ur9/20EbhKRZuAuYK+tvO4xY8x7xpjFRB+lPRg/3wx4fQngylK2ZRXmMgY41/ODPODbzpMEMwteVRbMq7A24DSy+bt6+f5ZQE/nxd8E3gH2I9ppbm2SiY2nFy5s9HWBPvZS99TdCHOZ7wBnoQNUmGvGpX2Z4FXApD4UGuBRoElE/rvzCRGZCOy00WuGAouMMQXgi1hYAkpL3YMwl/kNcAyw1HYWm/IMHlYwrLCdo5fmAYeQzf+5Lz/ERAvPHQ98LP5IaxZwBdEtmp2uByaLyMvAHlhYidWpBfIqKV5870FgT9tZbJnZ9KXZg6Vla+eISfUQcCrZfFr/QSqZ7qmLFC++dyjwV9tZbEnZgnmGaDH4Y2up0KClLkl8d9eniQYc1Nx59utmTCLXjurCKuAEsvmLk7ySRqVoqUsU5jKFMJfJAYcBr9nOU02zCl4aPi15CTiYbL5mJ8LQUvdSmMs8D3wA+K3tLNXSbMYnecG8NuBS4INk87Nth7FJL5SVgecHJxANVnF65s0xLHn7mf5fG2U7RxemA1PI5pttB0kC3VOXQZjL3A3sDtyIw/dmv8V2I41hre0cG2klum32EC30BrqnLjPPDz5ItNfe33aWSpjbNHlef2nbzXYOojHSU8nmZ9kOkjS6py6zeBrig4hu5XTuo5SlbGN7wbxXia5sH6aF7pqWugLiK+TXAROIVndIy0dBPQoLo2wdfi8Czgb2Jpu/x1KGVNBSV1CYyywOc5lvAu8DbsKBz7ZnmZ2rvWDeSuASYDey+V+lZZE6m/Scuoo8P9iZaODKFCCVtzJ+pu6Z56/pd+2BVdjUUqILj1elfXbPatNSW+D5wRjgy8BUYLTlOCXZVd5c8EjThTtXcBMziCYWuJ1sXtfG7gUttUWeHzQQ3fXzFeAoy3GK0kB722tNp4tIWe/F7wAeAH5BNv94GX9uTdJSJ4TnB3sRXQg6iSImsrPptaYvLmyUjnHl+FHAncBNZPO6zHCZaKkTxvODOuBDRDNsTCKBh+fPNZ0zfXvJH9DLt88B7gbuJpufUcZYKqalTjDPDwQ4nKjgnyC66d66e/v97xP71837UJEvN0TnyfcQFTn1K2AkXRruuqlZ8VxpT8UPPD8YSbQXPyp+7A1U+yMm5hbGmf3r5m3t2y3A88DTRLmfIZu3PWClpuieOsU8PxhBdAvofsDE+LErFZ4X65T6R/51ReNvDiYaVPMaMBeYRlTi6WleMdIFWmrHeH7Qj6jYuwNjgVFEF942fowgmsq2ga4HIK0hGuKaj39dASwHFgLh3vLveUHTxfOBBbU4CUHSaalV50drnY91YS6T+pFvtUxLrZRjdOy3Uo7RUivlGC21Uo7RUivlGC21Uo7RUivlGC21Uo7RUivlGC21Uo7RUivlGC21Uo7RUivlGC21Uo7RUivlGC21Uo7RUivlGC21Uo7RUivlGC21Uo7RUivlGC21Uo7RUivlGC21Uo7RUivlGC21Uo7RUivlGC21Uo75f0ltPq2h5lcOAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dHuL8RGM-jvk",
        "colab_type": "text"
      },
      "source": [
        "From this pie chart, it can be seen that 77% of Customers from the train data did not make an insurance claim during the specified period. The most popular case here is of No claim. The main aim of this project is to predict future Claim or No claim situation to give the company a foresight as to which customers will make a claim or not "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ySEWYIVQrc4o",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        },
        "outputId": "3c808074-d079-47e6-f26c-995ec3fed232"
      },
      "source": [
        "total.head()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Customer Id</th>\n",
              "      <th>YearOfObservation</th>\n",
              "      <th>Insured_Period</th>\n",
              "      <th>Residential</th>\n",
              "      <th>Building_Painted</th>\n",
              "      <th>Building_Fenced</th>\n",
              "      <th>Garden</th>\n",
              "      <th>Settlement</th>\n",
              "      <th>Building Dimension</th>\n",
              "      <th>Building_Type</th>\n",
              "      <th>Date_of_Occupancy</th>\n",
              "      <th>NumberOfWindows</th>\n",
              "      <th>Geo_Code</th>\n",
              "      <th>Claim</th>\n",
              "      <th>col</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>H14663</td>\n",
              "      <td>2013</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "      <td>N</td>\n",
              "      <td>V</td>\n",
              "      <td>V</td>\n",
              "      <td>U</td>\n",
              "      <td>290.0</td>\n",
              "      <td>1</td>\n",
              "      <td>1960.0</td>\n",
              "      <td>.</td>\n",
              "      <td>1053</td>\n",
              "      <td>0.0</td>\n",
              "      <td>train</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>H2037</td>\n",
              "      <td>2015</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "      <td>V</td>\n",
              "      <td>N</td>\n",
              "      <td>O</td>\n",
              "      <td>R</td>\n",
              "      <td>490.0</td>\n",
              "      <td>1</td>\n",
              "      <td>1850.0</td>\n",
              "      <td>4</td>\n",
              "      <td>1053</td>\n",
              "      <td>0.0</td>\n",
              "      <td>train</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>H3802</td>\n",
              "      <td>2014</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "      <td>N</td>\n",
              "      <td>V</td>\n",
              "      <td>V</td>\n",
              "      <td>U</td>\n",
              "      <td>595.0</td>\n",
              "      <td>1</td>\n",
              "      <td>1960.0</td>\n",
              "      <td>.</td>\n",
              "      <td>1053</td>\n",
              "      <td>0.0</td>\n",
              "      <td>train</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>H3834</td>\n",
              "      <td>2013</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "      <td>V</td>\n",
              "      <td>V</td>\n",
              "      <td>V</td>\n",
              "      <td>U</td>\n",
              "      <td>2840.0</td>\n",
              "      <td>1</td>\n",
              "      <td>1960.0</td>\n",
              "      <td>.</td>\n",
              "      <td>1053</td>\n",
              "      <td>0.0</td>\n",
              "      <td>train</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>H5053</td>\n",
              "      <td>2014</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "      <td>V</td>\n",
              "      <td>N</td>\n",
              "      <td>O</td>\n",
              "      <td>R</td>\n",
              "      <td>680.0</td>\n",
              "      <td>1</td>\n",
              "      <td>1800.0</td>\n",
              "      <td>3</td>\n",
              "      <td>1053</td>\n",
              "      <td>0.0</td>\n",
              "      <td>train</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  Customer Id  YearOfObservation  Insured_Period  ...  Geo_Code Claim    col\n",
              "0      H14663               2013             1.0  ...      1053   0.0  train\n",
              "1       H2037               2015             1.0  ...      1053   0.0  train\n",
              "2       H3802               2014             1.0  ...      1053   0.0  train\n",
              "3       H3834               2013             1.0  ...      1053   0.0  train\n",
              "4       H5053               2014             1.0  ...      1053   0.0  train\n",
              "\n",
              "[5 rows x 15 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ewb5MlNtGGIo",
        "colab_type": "code",
        "outputId": "d314104b-7b8f-4962-878e-5b8996341402",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        }
      },
      "source": [
        "total.isna().sum() #checking for missing values in the dataset"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Customer Id              0\n",
              "YearOfObservation        0\n",
              "Insured_Period           0\n",
              "Residential              0\n",
              "Building_Painted         0\n",
              "Building_Fenced          0\n",
              "Garden                  11\n",
              "Settlement               0\n",
              "Building Dimension     119\n",
              "Building_Type            0\n",
              "Date_of_Occupancy     1236\n",
              "NumberOfWindows          0\n",
              "Geo_Code               115\n",
              "Claim                 3069\n",
              "col                      0\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e3llxMCWfC9s",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        },
        "outputId": "4c2f9118-9d42-4da7-aab9-74ebdde8ffbe"
      },
      "source": [
        "total.nunique()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Customer Id           10229\n",
              "YearOfObservation         5\n",
              "Insured_Period          467\n",
              "Residential               2\n",
              "Building_Painted          2\n",
              "Building_Fenced           2\n",
              "Garden                    2\n",
              "Settlement                2\n",
              "Building Dimension     2463\n",
              "Building_Type             4\n",
              "Date_of_Occupancy       139\n",
              "NumberOfWindows          11\n",
              "Geo_Code               1525\n",
              "Claim                     2\n",
              "col                       2\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bGXtrjODn_5D",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        },
        "outputId": "9bae15e2-b177-43bc-f65c-f246e707f075"
      },
      "source": [
        "total.describe() #brief statistics of the data"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>YearOfObservation</th>\n",
              "      <th>Insured_Period</th>\n",
              "      <th>Residential</th>\n",
              "      <th>Building Dimension</th>\n",
              "      <th>Building_Type</th>\n",
              "      <th>Date_of_Occupancy</th>\n",
              "      <th>Claim</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>10229.000000</td>\n",
              "      <td>10229.000000</td>\n",
              "      <td>10229.000000</td>\n",
              "      <td>10110.000000</td>\n",
              "      <td>10229.000000</td>\n",
              "      <td>8993.000000</td>\n",
              "      <td>7160.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>2013.646789</td>\n",
              "      <td>0.913672</td>\n",
              "      <td>0.281064</td>\n",
              "      <td>1818.152918</td>\n",
              "      <td>2.238049</td>\n",
              "      <td>1965.061715</td>\n",
              "      <td>0.228212</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>1.380660</td>\n",
              "      <td>0.233806</td>\n",
              "      <td>0.449540</td>\n",
              "      <td>2272.111284</td>\n",
              "      <td>0.961589</td>\n",
              "      <td>33.548619</td>\n",
              "      <td>0.419709</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>2012.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1545.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>2012.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>500.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>1960.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>2013.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1002.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>1970.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>2015.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2190.000000</td>\n",
              "      <td>3.000000</td>\n",
              "      <td>1980.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>2016.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>30745.000000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>2016.000000</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       YearOfObservation  Insured_Period  ...  Date_of_Occupancy        Claim\n",
              "count       10229.000000    10229.000000  ...        8993.000000  7160.000000\n",
              "mean         2013.646789        0.913672  ...        1965.061715     0.228212\n",
              "std             1.380660        0.233806  ...          33.548619     0.419709\n",
              "min          2012.000000        0.000000  ...        1545.000000     0.000000\n",
              "25%          2012.000000        1.000000  ...        1960.000000     0.000000\n",
              "50%          2013.000000        1.000000  ...        1970.000000     0.000000\n",
              "75%          2015.000000        1.000000  ...        1980.000000     0.000000\n",
              "max          2016.000000        1.000000  ...        2016.000000     1.000000\n",
              "\n",
              "[8 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K-laNkicpKPp",
        "colab_type": "text"
      },
      "source": [
        "From these figures and tables above, there are a few noticeable things to take note of :\n",
        "\n",
        "*   The minimum value in the Building Dimension is 1m^2 which is completely impossible, no habitable building can be 1m^2\n",
        "*   The minimum date of occupancy is 1545, this is a possible but extremely unlikely scenario. Asides from looking like a highly unusual figure, it deviates greatly from the mean.\n",
        "\n",
        "\n",
        "*   The NumberofWindows has missing values that can be seen from the first five rows, and these values do not reflect as Nan\n",
        "*   List item\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qx_EguVqBU_e",
        "colab_type": "text"
      },
      "source": [
        "## Creating Useful Tools"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k7h1AHaaBlQZ",
        "colab_type": "text"
      },
      "source": [
        "This section consists of a class 'DataCleaner' in which I have defined two very useful methods that will be used for data cleaning processes subsequently in this analysis. The method 'handle_missing_categorical' will be used to replace NaN values in columns of categorical data while the 'handle_missing_continous' will be used for replacing NaN values in columns of continuous data. There are two different methods due to the few differences that exist while fitting models to predict continuous and categorical variables. Both methods take in data-related arguments and a final argument which is an instance variable of the selected ML algorithm to be used in predicting the missing values.\n",
        "\n",
        "The other tools are the 'tweaks' function and 'best_neighbors' function. The tweaks function applies LabelEncoder and get_dummies on selected columns, LabelEncoder converts column labels of categorical data into numeric data to enable the ML algorithms fit the data, get_dummies converts categorical variables into indicator variables and creates new columns, it is mostly used when there are more than two categories in the column.'best_neighbors' helps to select the best n_neigbors for a KNeighborsClassifier"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dtj8UMsNHZMb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#LabelEncoder encodes categorical variablees with integers, ML algorithms cannot work with string values\n",
        "\n",
        "\n",
        "le = LabelEncoder() \n",
        "#function to get dummies and encode\n",
        "def tweaks(table,cols) : \n",
        "\n",
        "  table = pd.get_dummies(table, columns=['Building_Type'])\n",
        " \n",
        "  #cols.append(extra)\n",
        "  for i in cols :\n",
        "    table[i] = le.fit_transform(table[i])\n",
        "\n",
        "  return table \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gt_3rdqTW-g9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# dummy variables creates  new columns for categorical variables and encodes with 1 and 0\n",
        "\n",
        "class DataCleaner :\n",
        "\n",
        "  def handle_missing_categorical(self,dum_gen,col_in_view,col_not_used,model,og_frame,cols):\n",
        "    #This function will help to sort missing values in columns and return a new dataframe with missing values in\n",
        "    #the selected column(columns of categorical data) replaced by values predicted by a chosen ML algorithm\n",
        "    in_ytrain = dum_gen.loc[dum_gen[col_in_view].notnull(), col_in_view]\n",
        "    in_xtrain = dum_gen.loc[dum_gen[col_in_view].notnull()]\n",
        "    in_xtrain = in_xtrain.drop(col_not_used, axis=1)\n",
        "    in_xtest = dum_gen.loc[dum_gen[col_in_view].isnull(), in_xtrain.columns]\n",
        "\n",
        "    #in_ytrain = le.fit_transform(in_ytrain)\n",
        "    in_xtrain= preprocessing.StandardScaler().fit(in_xtrain).transform(in_xtrain) #normalizing data\n",
        "    in_xtest= preprocessing.StandardScaler().fit(in_xtest).transform(in_xtest)\n",
        "\n",
        "    model.fit(in_xtrain, in_ytrain) #fit the model\n",
        "\n",
        "    pred = model.predict(in_xtest)\n",
        "    pred= pd.DataFrame(pred)\n",
        "\n",
        "    samp = dum_gen.loc[dum_gen[col_in_view].isnull()]\n",
        "    samp2 = samp[['Customer Id']].copy()\n",
        "    samp2.reset_index(drop=True, inplace=True)\n",
        "    pred.reset_index(drop=True,inplace=True)\n",
        "\n",
        "    fixed = pd.concat([samp2, pred], axis=1)\n",
        "    fixed.columns = ['Customer Id', col_in_view]\n",
        "\n",
        "    og = og_frame.loc[og_frame[col_in_view].isnull()]\n",
        "    og.reset_index(drop=True, inplace=True)\n",
        "    \n",
        "    merged = pd.merge(og[cols], fixed)\n",
        "    \n",
        "\n",
        "    og_copy = og_frame.drop(og_frame.loc[og_frame[col_in_view].isnull()].index)\n",
        "    \n",
        "    final_table = pd.concat([og_copy,merged])\n",
        "\n",
        "    return final_table\n",
        "\n",
        "\n",
        "  def handle_missing_continuous(self,dum_gen,col_in_view,col_not_used,model,og_frame,cols):\n",
        "    #This function will help to sort missing values in columns and return a new dataframe with missing values in\n",
        "    #the selected column(column of continuous data) replaced by values predicted by a chosen ML algorithm\n",
        "    in_ytrain = dum_gen.loc[dum_gen[col_in_view].notnull(), col_in_view]\n",
        "    in_xtrain = dum_gen.loc[dum_gen[col_in_view].notnull()]\n",
        "    in_xtrain = in_xtrain.drop(col_not_used, axis=1)\n",
        "    in_xtest = dum_gen.loc[dum_gen[col_in_view].isnull(), in_xtrain.columns]\n",
        "\n",
        "    #in_ytrain = le.fit_transform(in_ytrain)\n",
        "  \n",
        "\n",
        "    model.fit(in_xtrain, in_ytrain) #fit the model\n",
        "\n",
        "    pred = model.predict(in_xtest)\n",
        "    pred= pd.DataFrame(pred) #missing values predicted\n",
        "\n",
        "    samp = dum_gen.loc[dum_gen[col_in_view].isnull()]\n",
        "    samp2 = samp[['Customer Id']].copy()\n",
        "    samp2.reset_index(drop=True,inplace=True)\n",
        "    pred.reset_index(drop=True,inplace=True)\n",
        "\n",
        "    fixed = pd.concat([samp2, pred], axis=1)\n",
        "    fixed.columns = ['Customer Id', col_in_view] #customerId of missing values\n",
        "\n",
        "    \n",
        "    og = og_frame.loc[og_frame[col_in_view].isnull()]\n",
        "    og.reset_index(drop=True, inplace=True)\n",
        "    \n",
        "    merged = pd.merge(og[cols], fixed)\n",
        "    \n",
        "\n",
        "    og_copy = og_frame.drop(og_frame.loc[og_frame[col_in_view].isnull()].index)\n",
        "    \n",
        "    final_table = pd.concat([og_copy,merged])\n",
        "\n",
        "    return final_table\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        " "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OgoQuiF62Ftk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#This is to select the best no of neighbors for the KNN classifier\n",
        "def best_neighbors(x,y,samp_num=10) :\n",
        "  Ks = samp_num\n",
        "  jacc= np.zeros((Ks-1))\n",
        "\n",
        "  for n in range(1,Ks) :\n",
        "      neigh = KNeighborsClassifier(n_neighbors=n).fit(x,y)\n",
        "      \n",
        "      jacc[n-1] = neigh.score(x,y)\n",
        "  return jacc"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jFYpHKayXftF",
        "colab_type": "text"
      },
      "source": [
        "## Missing values in 'Garden' column\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ryJV9K6pZ8vr",
        "colab_type": "text"
      },
      "source": [
        "There are four(4) columns with missing values, In this section I will deal with the values mising in the column of buildings with/without a garden. A K-Nearest Neighbor classifier will be used to predict the missing values. The missing values will then be filled by the predicted values and all of this is already made easier by a predefined method. \n",
        "The column is a categorical variable one since there are just two options, and hence the use of the 'handle_missing_categorical'"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LnmxKdhP2ofm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "colnot = ['Geo_Code', 'Claim', 'col', 'NumberOfWindows', 'Date_of_Occupancy', 'Customer Id',\n",
        "                        'Building Dimension','Garden']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M7G1cVcqoptM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "samp = total.copy()\n",
        "col_g = ['Building_Painted', 'Building_Fenced', 'Settlement']\n",
        "samp1= tweaks(samp,col_g)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DJdd3eUfo3iq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "eceea956-d57e-432f-e285-1f2ec2cbff6f"
      },
      "source": [
        "all_colg= ['Customer Id', 'YearOfObservation', 'Insured_Period', 'Residential',\n",
        "  'Building_Painted', 'Building_Fenced','Settlement',\n",
        "  'Building Dimension', 'Date_of_Occupancy', 'NumberOfWindows',\n",
        "  'Geo_Code', 'Claim', 'col', 'Building_Type']\n",
        "sampneigh = KNeighborsClassifier(n_neighbors=7)\n",
        "clean1 = DataCleaner()\n",
        "clean1 = clean1.handle_missing_categorical(samp1,'Garden',colnot,sampneigh,samp,all_colg)\n",
        "print(clean1.shape)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(10229, 15)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1WVH_r37q5yC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        },
        "outputId": "f6bebe00-197e-49e6-d39b-54a643a0663a"
      },
      "source": [
        "clean1.isna().sum()"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Customer Id              0\n",
              "YearOfObservation        0\n",
              "Insured_Period           0\n",
              "Residential              0\n",
              "Building_Painted         0\n",
              "Building_Fenced          0\n",
              "Garden                   0\n",
              "Settlement               0\n",
              "Building Dimension     119\n",
              "Building_Type            0\n",
              "Date_of_Occupancy     1236\n",
              "NumberOfWindows          0\n",
              "Geo_Code               115\n",
              "Claim                 3069\n",
              "col                      0\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9p1ftMVh-A13",
        "colab_type": "text"
      },
      "source": [
        "Column 'Garden' has been successfully handled"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BLc-yKqeZpZz",
        "colab_type": "text"
      },
      "source": [
        "## Missing values in 'Building Dimension'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pwTiGjWFeCa0",
        "colab_type": "text"
      },
      "source": [
        "As noticed earlier on that the smallest value in building dimension column is 1m^2 which should be impossible, no building is possibly that small, I'll have to do a little bit of research to understand the smallest possible dimension for a building then use this to restructure the column because the problem in this column exceeds just NaN values"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P8sRn7DLrtjv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "clean1.loc[clean1['Building Dimension']<30, 'Building Dimension']= np.nan #replaces all values less than 30 with Nan\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NnHTtjItKy7N",
        "colab_type": "text"
      },
      "source": [
        "All dimensions lesser than 30m^2 are considered as missing  values and this is because from some research I did through sources online, the minimum dimension for a building is expected to fall within the range of 30m^2 to 37m^2, for this reason I'll take 30 as the minimum acceptable dimension therefore every row with values of Building Dimension being lesser than 30  will be replaced by NaN. After sorting the irregularity in the dimension, A Linear Regression algorithm will be used to predict all the unknown values "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JvklekGC_ze9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "colnotbd = ['Geo_Code', 'Claim', 'col', 'NumberOfWindows', 'Date_of_Occupancy', 'Customer Id',\n",
        "                        'Building Dimension'] #columns to  be dropped\n",
        "all_colbd= ['Customer Id', 'YearOfObservation', 'Insured_Period', 'Residential',\n",
        "                'Building_Painted', 'Building_Fenced','Settlement',\n",
        "                'Garden', 'Date_of_Occupancy', 'NumberOfWindows',\n",
        "                'Geo_Code', 'Claim', 'col', 'Building_Type']#columns used in merge"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eoPZet97-Jb3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "bbd3296b-7095-4b84-e43a-394122e9a94a"
      },
      "source": [
        "lr = LinearRegression()\n",
        "col_bd = ['Building_Painted', 'Building_Fenced', 'Settlement','Garden']\n",
        "samp2= tweaks(clean1,col_bd)\n",
        "sampp=  clean1.copy()\n",
        "clean2 = DataCleaner()\n",
        "clean2 = clean2.handle_missing_continuous(samp2,'Building Dimension',colnotbd,lr,sampp,all_colbd)\n",
        "print(clean2.shape)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(10229, 15)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xv8NYt1_RqT3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        },
        "outputId": "d91ae881-9343-494a-bc93-00470e0ebf63"
      },
      "source": [
        "clean2.isna().sum()"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Customer Id              0\n",
              "YearOfObservation        0\n",
              "Insured_Period           0\n",
              "Residential              0\n",
              "Building_Painted         0\n",
              "Building_Fenced          0\n",
              "Garden                   0\n",
              "Settlement               0\n",
              "Building Dimension       0\n",
              "Building_Type            0\n",
              "Date_of_Occupancy     1236\n",
              "NumberOfWindows          0\n",
              "Geo_Code               115\n",
              "Claim                 3069\n",
              "col                      0\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g-QIeFvWCpIH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        },
        "outputId": "c37731f2-76c2-4cbe-e572-a61c81b45e18"
      },
      "source": [
        "clean2.describe()"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>YearOfObservation</th>\n",
              "      <th>Insured_Period</th>\n",
              "      <th>Residential</th>\n",
              "      <th>Building Dimension</th>\n",
              "      <th>Building_Type</th>\n",
              "      <th>Date_of_Occupancy</th>\n",
              "      <th>Claim</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>10229.000000</td>\n",
              "      <td>10229.000000</td>\n",
              "      <td>10229.000000</td>\n",
              "      <td>10229.000000</td>\n",
              "      <td>10229.000000</td>\n",
              "      <td>8993.000000</td>\n",
              "      <td>7160.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>2013.646789</td>\n",
              "      <td>0.913672</td>\n",
              "      <td>0.281064</td>\n",
              "      <td>1821.141656</td>\n",
              "      <td>2.238049</td>\n",
              "      <td>1965.061715</td>\n",
              "      <td>0.228212</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>1.380660</td>\n",
              "      <td>0.233806</td>\n",
              "      <td>0.449540</td>\n",
              "      <td>2256.806453</td>\n",
              "      <td>0.961589</td>\n",
              "      <td>33.548619</td>\n",
              "      <td>0.419709</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>2012.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>30.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1545.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>2012.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>516.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>1960.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>2013.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1045.000000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>1970.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>2015.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2169.000000</td>\n",
              "      <td>3.000000</td>\n",
              "      <td>1980.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>2016.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>30745.000000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>2016.000000</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       YearOfObservation  Insured_Period  ...  Date_of_Occupancy        Claim\n",
              "count       10229.000000    10229.000000  ...        8993.000000  7160.000000\n",
              "mean         2013.646789        0.913672  ...        1965.061715     0.228212\n",
              "std             1.380660        0.233806  ...          33.548619     0.419709\n",
              "min          2012.000000        0.000000  ...        1545.000000     0.000000\n",
              "25%          2012.000000        1.000000  ...        1960.000000     0.000000\n",
              "50%          2013.000000        1.000000  ...        1970.000000     0.000000\n",
              "75%          2015.000000        1.000000  ...        1980.000000     0.000000\n",
              "max          2016.000000        1.000000  ...        2016.000000     1.000000\n",
              "\n",
              "[8 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "niOHJKi3d9g4",
        "colab_type": "text"
      },
      "source": [
        "## 'Date_of_Occupancy' Column"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LdeiFQBHbrbn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        },
        "outputId": "4fd399d8-1af1-446f-bd85-81e9cf3f0fa0"
      },
      "source": [
        "clean2.loc[clean2['Date_of_Occupancy']<=1800].head()"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Customer Id</th>\n",
              "      <th>YearOfObservation</th>\n",
              "      <th>Insured_Period</th>\n",
              "      <th>Residential</th>\n",
              "      <th>Building_Painted</th>\n",
              "      <th>Building_Fenced</th>\n",
              "      <th>Garden</th>\n",
              "      <th>Settlement</th>\n",
              "      <th>Building Dimension</th>\n",
              "      <th>Building_Type</th>\n",
              "      <th>Date_of_Occupancy</th>\n",
              "      <th>NumberOfWindows</th>\n",
              "      <th>Geo_Code</th>\n",
              "      <th>Claim</th>\n",
              "      <th>col</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>H5053</td>\n",
              "      <td>2014</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0</td>\n",
              "      <td>V</td>\n",
              "      <td>N</td>\n",
              "      <td>O</td>\n",
              "      <td>R</td>\n",
              "      <td>680.0</td>\n",
              "      <td>1</td>\n",
              "      <td>1800.0</td>\n",
              "      <td>3</td>\n",
              "      <td>1053</td>\n",
              "      <td>0.0</td>\n",
              "      <td>train</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>H15676</td>\n",
              "      <td>2013</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0</td>\n",
              "      <td>V</td>\n",
              "      <td>N</td>\n",
              "      <td>O</td>\n",
              "      <td>R</td>\n",
              "      <td>842.0</td>\n",
              "      <td>1</td>\n",
              "      <td>1550.0</td>\n",
              "      <td>4</td>\n",
              "      <td>1305</td>\n",
              "      <td>0.0</td>\n",
              "      <td>train</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>H16563</td>\n",
              "      <td>2016</td>\n",
              "      <td>0.997268</td>\n",
              "      <td>0</td>\n",
              "      <td>V</td>\n",
              "      <td>N</td>\n",
              "      <td>O</td>\n",
              "      <td>R</td>\n",
              "      <td>2400.0</td>\n",
              "      <td>1</td>\n",
              "      <td>1700.0</td>\n",
              "      <td>3</td>\n",
              "      <td>2408</td>\n",
              "      <td>0.0</td>\n",
              "      <td>train</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>H16387</td>\n",
              "      <td>2012</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0</td>\n",
              "      <td>V</td>\n",
              "      <td>N</td>\n",
              "      <td>O</td>\n",
              "      <td>R</td>\n",
              "      <td>480.0</td>\n",
              "      <td>2</td>\n",
              "      <td>1800.0</td>\n",
              "      <td>4</td>\n",
              "      <td>4088</td>\n",
              "      <td>0.0</td>\n",
              "      <td>train</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>187</th>\n",
              "      <td>H19770</td>\n",
              "      <td>2015</td>\n",
              "      <td>0.260274</td>\n",
              "      <td>0</td>\n",
              "      <td>V</td>\n",
              "      <td>N</td>\n",
              "      <td>O</td>\n",
              "      <td>R</td>\n",
              "      <td>100.0</td>\n",
              "      <td>4</td>\n",
              "      <td>1800.0</td>\n",
              "      <td>3</td>\n",
              "      <td>6030</td>\n",
              "      <td>0.0</td>\n",
              "      <td>train</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    Customer Id  YearOfObservation  Insured_Period  ...  Geo_Code Claim    col\n",
              "4         H5053               2014        1.000000  ...      1053   0.0  train\n",
              "12       H15676               2013        1.000000  ...      1305   0.0  train\n",
              "21       H16563               2016        0.997268  ...      2408   0.0  train\n",
              "34       H16387               2012        1.000000  ...      4088   0.0  train\n",
              "187      H19770               2015        0.260274  ...      6030   0.0  train\n",
              "\n",
              "[5 rows x 15 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RXCkz1FNnXTy",
        "colab_type": "text"
      },
      "source": [
        "There are a number of unusual values in this column, with years ranging from mid 1500s to early 1700s and these values do not seem normal considering the fact that even the first storey building in Nigeria was built in 1845, any building that old and still being inhabited or insured either has some historic origin or has been completely renovated or rebuilt, This column is populated with a lot of these values and what makes the most sense at this point is to drop the entire column, although if the model produces a poor performance after then there may be need to carry out proper investigation in this column in order to understand why exactly these values are missing and decide the best step to take. The date of occupancy is an important feature since older buildings are most likely to have a reason to make a claim but this feature is of no use if there are so many errors that would  be difficult to correct."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c5WyCIe6ccT0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "clean2.drop(['Date_of_Occupancy'], axis=1, inplace=True) #dropping column"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mqjm0WqXjDSH",
        "colab_type": "text"
      },
      "source": [
        "## 'NumberOfWindows' column"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e8bTLC2EeN1D",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "outputId": "7f5296b0-70d4-449e-aec0-bf772f2d0a37"
      },
      "source": [
        "clean2['NumberOfWindows'].value_counts()"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   .    5791\n",
              "4       1133\n",
              "3       1071\n",
              "5        790\n",
              "2        433\n",
              "6        376\n",
              "7        265\n",
              "8        142\n",
              "1         91\n",
              ">=10      78\n",
              "9         59\n",
              "Name: NumberOfWindows, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZzVCBd4bjyfm",
        "colab_type": "text"
      },
      "source": [
        "There are multiple rows with values of '  .' which are just missing values in short, it can be seen that the values missing are more than 50% of the entire dataset and the best thing to do here is to drop the entire column"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uTmrqEvFgLQW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "clean2.drop(['NumberOfWindows'], axis=1,inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sgar5a_ghbub",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 255
        },
        "outputId": "df79f52f-5d4c-4b9b-88ea-00530e684554"
      },
      "source": [
        "clean2.nunique()"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Customer Id           10229\n",
              "YearOfObservation         5\n",
              "Insured_Period          467\n",
              "Residential               2\n",
              "Building_Painted          2\n",
              "Building_Fenced           2\n",
              "Garden                    2\n",
              "Settlement                2\n",
              "Building Dimension     2469\n",
              "Building_Type             4\n",
              "Geo_Code               1525\n",
              "Claim                     2\n",
              "col                       2\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0L5mssi_p1dC",
        "colab_type": "text"
      },
      "source": [
        "## GeoCode column"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zm-Av22Ckp7a",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "outputId": "85364dfc-583b-45d4-d467-1f0c636b3da4"
      },
      "source": [
        "clean2['Geo_Code'].value_counts()"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "6088     238\n",
              "6083     226\n",
              "13206    166\n",
              "33063    162\n",
              "13205    133\n",
              "        ... \n",
              "6059       1\n",
              "84087      1\n",
              "76613      1\n",
              "17093      1\n",
              "21166      1\n",
              "Name: Geo_Code, Length: 1525, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HFcutns-rApg",
        "colab_type": "text"
      },
      "source": [
        "There are 115 missing values in this column and the most likely reason fo geocode value to be missing(in a small number like that) would be because there was some error in collection of geo_code for certain errors, fromt the value counts done above, it can be seen that there are geocodes common in multiple rows with the highest even being over double of the 115 missing, for this reason what I'll be doing here is to represent the missing values as an entirely different category "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DpDjI-scuRtm",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        },
        "outputId": "a955b19b-313c-42ed-e93b-d8ddc585e2b4"
      },
      "source": [
        "clean2.loc[clean2['Geo_Code']=='1234'] #checking for random number available"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Customer Id</th>\n",
              "      <th>YearOfObservation</th>\n",
              "      <th>Insured_Period</th>\n",
              "      <th>Residential</th>\n",
              "      <th>Building_Painted</th>\n",
              "      <th>Building_Fenced</th>\n",
              "      <th>Garden</th>\n",
              "      <th>Settlement</th>\n",
              "      <th>Building Dimension</th>\n",
              "      <th>Building_Type</th>\n",
              "      <th>Geo_Code</th>\n",
              "      <th>Claim</th>\n",
              "      <th>col</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "Empty DataFrame\n",
              "Columns: [Customer Id, YearOfObservation, Insured_Period, Residential, Building_Painted, Building_Fenced, Garden, Settlement, Building Dimension, Building_Type, Geo_Code, Claim, col]\n",
              "Index: []"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cP9sx-Q0lPrA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "clean2['Geo_Code'].replace({np.nan:'1234'},inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TzZgBF49u6Ud",
        "colab_type": "text"
      },
      "source": [
        "## Splitting back into train/test + some final touches"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lVKXm2hlydU5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def tweaks2(table) : \n",
        "\n",
        "  table = pd.get_dummies(table, columns=['Building_Type','Geo_Code'])\n",
        " \n",
        "  cols = ['Residential','Building_Painted', 'Building_Fenced', 'Garden', 'Settlement']\n",
        "  for i in cols :\n",
        "    table[i] = le.fit_transform(table[i])\n",
        "\n",
        "  return table "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5V-rAzpqzYvX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "clean2 = tweaks2(clean2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MyaqjGDLusgJ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "outputId": "1b114e73-7204-4e8d-cd6e-faf726a2dd35"
      },
      "source": [
        "clean2.isna().sum()"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Customer Id          0\n",
              "YearOfObservation    0\n",
              "Insured_Period       0\n",
              "Residential          0\n",
              "Building_Painted     0\n",
              "                    ..\n",
              "Geo_Code_95563       0\n",
              "Geo_Code_95582       0\n",
              "Geo_Code_95585       0\n",
              "Geo_Code_95598       0\n",
              "Geo_Code_95607       0\n",
              "Length: 1541, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TXlj6vG7xtJK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 270
        },
        "outputId": "ee78443b-f660-42f2-c738-e82b4af71186"
      },
      "source": [
        "clean2.head()"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Customer Id</th>\n",
              "      <th>YearOfObservation</th>\n",
              "      <th>Insured_Period</th>\n",
              "      <th>Residential</th>\n",
              "      <th>Building_Painted</th>\n",
              "      <th>Building_Fenced</th>\n",
              "      <th>Garden</th>\n",
              "      <th>Settlement</th>\n",
              "      <th>Building Dimension</th>\n",
              "      <th>Claim</th>\n",
              "      <th>col</th>\n",
              "      <th>Building_Type_1</th>\n",
              "      <th>Building_Type_2</th>\n",
              "      <th>Building_Type_3</th>\n",
              "      <th>Building_Type_4</th>\n",
              "      <th>Geo_Code_10033</th>\n",
              "      <th>Geo_Code_10081</th>\n",
              "      <th>Geo_Code_10297</th>\n",
              "      <th>Geo_Code_10333</th>\n",
              "      <th>Geo_Code_10343</th>\n",
              "      <th>Geo_Code_10362</th>\n",
              "      <th>Geo_Code_10387</th>\n",
              "      <th>Geo_Code_1053</th>\n",
              "      <th>Geo_Code_11069</th>\n",
              "      <th>Geo_Code_11076</th>\n",
              "      <th>Geo_Code_11170</th>\n",
              "      <th>Geo_Code_11202</th>\n",
              "      <th>Geo_Code_11262</th>\n",
              "      <th>Geo_Code_11266</th>\n",
              "      <th>Geo_Code_1143</th>\n",
              "      <th>Geo_Code_11441</th>\n",
              "      <th>Geo_Code_1160</th>\n",
              "      <th>Geo_Code_1173</th>\n",
              "      <th>Geo_Code_11910</th>\n",
              "      <th>Geo_Code_12052</th>\n",
              "      <th>Geo_Code_12084</th>\n",
              "      <th>Geo_Code_12096</th>\n",
              "      <th>Geo_Code_12145</th>\n",
              "      <th>Geo_Code_12174</th>\n",
              "      <th>Geo_Code_12176</th>\n",
              "      <th>...</th>\n",
              "      <th>Geo_Code_94074</th>\n",
              "      <th>Geo_Code_94076</th>\n",
              "      <th>Geo_Code_94078</th>\n",
              "      <th>Geo_Code_94079</th>\n",
              "      <th>Geo_Code_94080</th>\n",
              "      <th>Geo_Code_94081</th>\n",
              "      <th>Geo_Code_95014</th>\n",
              "      <th>Geo_Code_95018</th>\n",
              "      <th>Geo_Code_95042</th>\n",
              "      <th>Geo_Code_95051</th>\n",
              "      <th>Geo_Code_95052</th>\n",
              "      <th>Geo_Code_95060</th>\n",
              "      <th>Geo_Code_95063</th>\n",
              "      <th>Geo_Code_95127</th>\n",
              "      <th>Geo_Code_95176</th>\n",
              "      <th>Geo_Code_95183</th>\n",
              "      <th>Geo_Code_95197</th>\n",
              "      <th>Geo_Code_95199</th>\n",
              "      <th>Geo_Code_95203</th>\n",
              "      <th>Geo_Code_95205</th>\n",
              "      <th>Geo_Code_95210</th>\n",
              "      <th>Geo_Code_95219</th>\n",
              "      <th>Geo_Code_95252</th>\n",
              "      <th>Geo_Code_95280</th>\n",
              "      <th>Geo_Code_95288</th>\n",
              "      <th>Geo_Code_95306</th>\n",
              "      <th>Geo_Code_95313</th>\n",
              "      <th>Geo_Code_95394</th>\n",
              "      <th>Geo_Code_95424</th>\n",
              "      <th>Geo_Code_95426</th>\n",
              "      <th>Geo_Code_95428</th>\n",
              "      <th>Geo_Code_95488</th>\n",
              "      <th>Geo_Code_95500</th>\n",
              "      <th>Geo_Code_95539</th>\n",
              "      <th>Geo_Code_95555</th>\n",
              "      <th>Geo_Code_95563</th>\n",
              "      <th>Geo_Code_95582</th>\n",
              "      <th>Geo_Code_95585</th>\n",
              "      <th>Geo_Code_95598</th>\n",
              "      <th>Geo_Code_95607</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>H14663</td>\n",
              "      <td>2013</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>290.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>train</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>H2037</td>\n",
              "      <td>2015</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>490.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>train</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>H3802</td>\n",
              "      <td>2014</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>595.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>train</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>H3834</td>\n",
              "      <td>2013</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2840.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>train</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>H5053</td>\n",
              "      <td>2014</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>680.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>train</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 1541 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "  Customer Id  YearOfObservation  ...  Geo_Code_95598  Geo_Code_95607\n",
              "0      H14663               2013  ...               0               0\n",
              "1       H2037               2015  ...               0               0\n",
              "2       H3802               2014  ...               0               0\n",
              "3       H3834               2013  ...               0               0\n",
              "4       H5053               2014  ...               0               0\n",
              "\n",
              "[5 rows x 1541 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I3sBQhaPux8v",
        "colab_type": "text"
      },
      "source": [
        "The data cleaning is complete so now the next thing would be to split the train and test sets back into train and test, a column was created at the beginning of this project to mark the rows into train and test, this column will be used here to seperate them back"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mmx8qX_FuvZ8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train = clean2.loc[clean2['col']=='train'].reset_index(drop=True)\n",
        "test = clean2.loc[clean2['col']=='test'].reset_index(drop=True)\n",
        "\n",
        "test.drop(['Claim','col'], axis=1, inplace=True)\n",
        "train.drop(['col'], axis=1, inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kUZUMgzqww4W",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "outputId": "d6b8ad9a-44dd-4cda-81e1-e9371b586bcf"
      },
      "source": [
        "train.isna().sum() "
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Customer Id          0\n",
              "YearOfObservation    0\n",
              "Insured_Period       0\n",
              "Residential          0\n",
              "Building_Painted     0\n",
              "                    ..\n",
              "Geo_Code_95563       0\n",
              "Geo_Code_95582       0\n",
              "Geo_Code_95585       0\n",
              "Geo_Code_95598       0\n",
              "Geo_Code_95607       0\n",
              "Length: 1540, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6yj6qNEww4wT",
        "colab_type": "text"
      },
      "source": [
        "Train set is clean :)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hfnfDGNrwz6G",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "outputId": "9230fa31-694c-4d1a-9a07-5b8440ff7046"
      },
      "source": [
        "test.isna().sum()"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Customer Id          0\n",
              "YearOfObservation    0\n",
              "Insured_Period       0\n",
              "Residential          0\n",
              "Building_Painted     0\n",
              "                    ..\n",
              "Geo_Code_95563       0\n",
              "Geo_Code_95582       0\n",
              "Geo_Code_95585       0\n",
              "Geo_Code_95598       0\n",
              "Geo_Code_95607       0\n",
              "Length: 1539, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u76362mexCpq",
        "colab_type": "text"
      },
      "source": [
        "Test is clean too :) "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h-olQo6_xQuC",
        "colab_type": "text"
      },
      "source": [
        "# Model Development"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sw8cqPw1Fz_L",
        "colab_type": "text"
      },
      "source": [
        "I'll be creating a function to fit the data with different classification algorithms, make predictions and return the scores of the predictions using three selected evaluation metrics\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BWIOs1Jmw_Rr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.metrics import accuracy_score,jaccard_similarity_score, f1_score, roc_auc_score\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "from sklearn import svm\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "\n",
        "def apply_metrics(train_X,train_y,test_X,test_y):\n",
        "  #all four classifiers used\n",
        "  KNC = KNeighborsClassifier(n_neighbors=4)\n",
        "  RFC = RandomForestClassifier()\n",
        "  DTC = DecisionTreeClassifier()\n",
        "  SVM = svm.SVC(kernel='rbf')\n",
        "\n",
        "  train_X= preprocessing.StandardScaler().fit(train_X).transform(train_X)\n",
        "  test_X= preprocessing.StandardScaler().fit(test_X).transform(test_X)\n",
        "\n",
        "  KNC.fit(train_X,train_y)\n",
        "  RFC.fit(train_X,train_y)\n",
        "  DTC.fit(train_X,train_y)\n",
        "  SVM.fit(train_X,train_y)\n",
        "\n",
        "  KNC_pred = KNC.predict(test_X)\n",
        "  RFC_pred = RFC.predict(test_X)\n",
        "  DTC_pred = DTC.predict(test_X)\n",
        "  SVC_pred = SVM.predict(test_X)\n",
        "\n",
        "  #for KNeighbors\n",
        "  print(\n",
        "      'Accuracy Score for KNeighborsClassifier is ' + str(accuracy_score(test_y,KNC_pred))\n",
        "  )\n",
        "  print(\n",
        "      'Jaccard Similarity Score for KNeighborsClassifier is ' + str(jaccard_similarity_score(test_y,KNC_pred))\n",
        "  )\n",
        "  print(\n",
        "      'F1 Score for KNeighborsClassifier is ' + str(f1_score(test_y,KNC_pred))\n",
        "  )\n",
        "  print(\n",
        "      'Roc_Auc Score for KNeighborsClassifier is ' + str(roc_auc_score(test_y,KNC_pred))\n",
        "  )\n",
        "\n",
        "  print('')\n",
        "  \n",
        "  #for RandomForest\n",
        "  print(\n",
        "      'Accuracy Score for RandomForestClassifier is' + str(accuracy_score(test_y,RFC_pred))\n",
        "  )\n",
        "  print(\n",
        "      'Jaccard Similarity Score for RandomForestClassifier is' + str(jaccard_similarity_score(test_y,RFC_pred))\n",
        "  )\n",
        "  print(\n",
        "      'F1 Score for RandomForestClassifier is' + str(f1_score(test_y,RFC_pred))\n",
        "  )\n",
        "  print(\n",
        "      'Roc_Auc Score for RandomForestClassifier is' + str(roc_auc_score(test_y,RFC_pred))\n",
        "  )\n",
        "  print('')\n",
        "\n",
        "  #for DecisionTree\n",
        "\n",
        "  print(\n",
        "      'Accuracy Score for DecisionTreeClassifier is' + str(accuracy_score(test_y,DTC_pred))\n",
        "  )\n",
        "  print(\n",
        "      'Jaccard Similarity Score for DecisionTreeClassifier is' + str(jaccard_similarity_score(test_y,DTC_pred))\n",
        "  )\n",
        "  print(\n",
        "      'F1 Score for DecisionTreeClassifier is' + str(f1_score(test_y,DTC_pred))\n",
        "  )\n",
        "  print(\n",
        "      'Roc_Auc Score for DecisionTreeClassifier is' + str(roc_auc_score(test_y,DTC_pred))\n",
        "  )\n",
        "\n",
        "  print('')\n",
        "  # for SVC\n",
        "  print(\n",
        "      'Accuracy Score for SVC is' + str(accuracy_score(test_y,SVC_pred))\n",
        "  )\n",
        "  print(\n",
        "      'Jaccard Similarity Score for SVC is' + str(jaccard_similarity_score(test_y,SVC_pred))\n",
        "  )\n",
        "  print(\n",
        "      'F1 Score for SVC is' + str(f1_score(test_y,SVC_pred))\n",
        "  )\n",
        "  print(\n",
        "      'Roc_Auc Score for SVC is' + str(roc_auc_score(test_y,SVC_pred))\n",
        "  )"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2r45LjaWx0_3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 496
        },
        "outputId": "c98ae6dc-301a-448d-cf7b-006f8b71ceb3"
      },
      "source": [
        "xx= train.drop(['Customer Id'],axis=1)\n",
        "yy= xx['Claim'].values\n",
        "xx.drop(['Claim'],axis=1,inplace=True)\n",
        "\n",
        "xx_train,xx_test,yy_train,yy_test = train_test_split(xx,yy,test_size=0.3)\n",
        "apply_metrics(xx_train,yy_train,xx_test,yy_test)"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy Score for KNeighborsClassifier is 0.7746741154562383\n",
            "Jaccard Similarity Score for KNeighborsClassifier is 0.7746741154562383\n",
            "F1 Score for KNeighborsClassifier is 0.15972222222222224\n",
            "Roc_Auc Score for KNeighborsClassifier is 0.5308749369802057\n",
            "\n",
            "Accuracy Score for RandomForestClassifier is0.7825884543761639\n",
            "Jaccard Similarity Score for RandomForestClassifier is0.7825884543761639\n",
            "F1 Score for RandomForestClassifier is0.3280575539568345\n",
            "Roc_Auc Score for RandomForestClassifier is0.5881819948161854\n",
            "\n",
            "Accuracy Score for DecisionTreeClassifier is0.7374301675977654\n",
            "Jaccard Similarity Score for DecisionTreeClassifier is0.7374301675977654\n",
            "F1 Score for DecisionTreeClassifier is0.36629213483146067\n",
            "Roc_Auc Score for DecisionTreeClassifier is0.5969425399167695\n",
            "\n",
            "Accuracy Score for SVC is0.7797951582867784\n",
            "Jaccard Similarity Score for SVC is0.7797951582867784\n",
            "F1 Score for SVC is0.05964214711729622\n",
            "Roc_Auc Score for SVC is0.5103333413761618\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:664: FutureWarning: jaccard_similarity_score has been deprecated and replaced with jaccard_score. It will be removed in version 0.23. This implementation has surprising behavior for binary and multiclass classification tasks.\n",
            "  FutureWarning)\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:664: FutureWarning: jaccard_similarity_score has been deprecated and replaced with jaccard_score. It will be removed in version 0.23. This implementation has surprising behavior for binary and multiclass classification tasks.\n",
            "  FutureWarning)\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:664: FutureWarning: jaccard_similarity_score has been deprecated and replaced with jaccard_score. It will be removed in version 0.23. This implementation has surprising behavior for binary and multiclass classification tasks.\n",
            "  FutureWarning)\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:664: FutureWarning: jaccard_similarity_score has been deprecated and replaced with jaccard_score. It will be removed in version 0.23. This implementation has surprising behavior for binary and multiclass classification tasks.\n",
            "  FutureWarning)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fWq224VUOJ-a",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}