AML_Attack_and_Defense_Dataset_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Case Study: Automated Machine Learning (AutoML) for Autonomous Intrusion Detection System Development \n",
    "This is the code for the paper entitled \"**[Enabling AutoML for Zero-Touch Network Security: Use-Case Driven Analysis](https://ieeexplore.ieee.org/document/10472316)**\" published in *IEEE Transactions on Network and Service Management* (IF:5.3).<br>\n",
    "Authors: Li Yang (liyanghart@gmail.com), Mirna El Rajab, Abdallah Shami, and Sami Muhaidat<br>\n",
    "\n",
    "L. Yang, M. E. Rajab, A. Shami, and S. Muhaidat, \"Enabling AutoML for Zero-Touch Network Security: Use-Case Driven Analysis,\" IEEE Transactions on Network and Service Management, pp. 1-28, 2024, doi: https://doi.org/10.1109/TNSM.2024.3376631."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code Part 3: Adversarial Machine Learning (AML) Attack and Defense\n",
    "As many network services and functionalities rely on AI/ML models, they are becoming increasingly vulnerable to AML attacks. AML attacks exploit the weaknesses and vulnerabilities of ML models by generating adversarial inputs that can deceive or manipulate the models into making incorrect predictions. In networks, AML attacks pose a significant threat to overall network security and reliability.  \n",
    "This case study aims to demonstrate the detrimental impact that AML attacks can have on ML models in networks and presents basic defense strategies to mitigate these attacks, thereby ensuring the accuracy of the ML-based IDS. In this case study, three common types of adversarial attacks ((i.e., DTA, FGSM, and BIM)) are used to generate adversarial samples to probe the vulnerability of the IDS.  \n",
    "Subsequently, basic defense mechanisms, including adversarial sample detection and filtering, are devised to safeguard the AutoML-based IDS against AML attacks.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset 2: 5G-NIDD\n",
    "A subset of the network traffic data randomly sampled from the [5G-NIDD dataset](https://ieee-dataport.org/documents/5g-nidd-comprehensive-network-intrusion-detection-dataset-generated-over-5g-wireless).  \n",
    "\n",
    "The 5G-NIDD dataset, created in December 2022, is a fully labeled resource constructed on a functional 5G test network for researchers and practitioners evaluating AI/ML solutions in the context of 5G/6G security [87]. 5G-NIDD encompasses data extracted from a 5G testbed connected to the 5G Test Network (5GTN) at the University of Oulu, Finland. The dataset is derived from two base stations, each featuring an attacker node and multiple benign 5G users. The attacker nodes target a server deployed within the 5GTN MEC environment. The attack scenarios captured in the dataset primarily include DoS attacks and port scans."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split,cross_val_score\n",
    "import lightgbm as lgb\n",
    "from sklearn.metrics import classification_report,confusion_matrix,accuracy_score, precision_score, recall_score, f1_score\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import classification_report,confusion_matrix,accuracy_score\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from scipy.stats import shapiro\n",
    "from imblearn.over_sampling import SMOTE\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings \n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read the sampled 5G-NIDD dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"Data/5gnidd_0.01_pre-processed.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Seq</th>\n",
       "      <th>Dur</th>\n",
       "      <th>RunTime</th>\n",
       "      <th>Mean</th>\n",
       "      <th>Sum</th>\n",
       "      <th>Min</th>\n",
       "      <th>Max</th>\n",
       "      <th>Proto</th>\n",
       "      <th>sTos</th>\n",
       "      <th>dTos</th>\n",
       "      <th>...</th>\n",
       "      <th>SrcWin</th>\n",
       "      <th>DstWin</th>\n",
       "      <th>sVid</th>\n",
       "      <th>dVid</th>\n",
       "      <th>SrcTCPBase</th>\n",
       "      <th>DstTCPBase</th>\n",
       "      <th>TcpRtt</th>\n",
       "      <th>SynAck</th>\n",
       "      <th>AckDat</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000496</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.388654</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.002442</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.388654</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.003295</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.388654</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.003317</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.388654</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.003375</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.388669</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12154</th>\n",
       "      <td>0.003688</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12155</th>\n",
       "      <td>0.004104</td>\n",
       "      <td>2.000005e-07</td>\n",
       "      <td>2.000005e-07</td>\n",
       "      <td>2.000005e-07</td>\n",
       "      <td>2.000005e-07</td>\n",
       "      <td>2.000005e-07</td>\n",
       "      <td>2.000005e-07</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>0.00025</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.292755</td>\n",
       "      <td>0.030151</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12156</th>\n",
       "      <td>0.005234</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12157</th>\n",
       "      <td>0.000015</td>\n",
       "      <td>4.079091e-02</td>\n",
       "      <td>4.079091e-02</td>\n",
       "      <td>4.079091e-02</td>\n",
       "      <td>4.079091e-02</td>\n",
       "      <td>4.079091e-02</td>\n",
       "      <td>4.079091e-02</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.830357</td>\n",
       "      <td>0.215054</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12158</th>\n",
       "      <td>0.000015</td>\n",
       "      <td>4.088011e-02</td>\n",
       "      <td>4.088011e-02</td>\n",
       "      <td>4.088011e-02</td>\n",
       "      <td>4.088011e-02</td>\n",
       "      <td>4.088011e-02</td>\n",
       "      <td>4.088011e-02</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.830357</td>\n",
       "      <td>0.215054</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12159 rows × 49 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            Seq           Dur       RunTime          Mean           Sum  \\\n",
       "0      0.000496  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   \n",
       "1      0.002442  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   \n",
       "2      0.003295  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   \n",
       "3      0.003317  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   \n",
       "4      0.003375  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   \n",
       "...         ...           ...           ...           ...           ...   \n",
       "12154  0.003688  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   \n",
       "12155  0.004104  2.000005e-07  2.000005e-07  2.000005e-07  2.000005e-07   \n",
       "12156  0.005234  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   \n",
       "12157  0.000015  4.079091e-02  4.079091e-02  4.079091e-02  4.079091e-02   \n",
       "12158  0.000015  4.088011e-02  4.088011e-02  4.088011e-02  4.088011e-02   \n",
       "\n",
       "                Min           Max  Proto      sTos      dTos  ...    SrcWin  \\\n",
       "0      0.000000e+00  0.000000e+00   0.75  0.000000  0.000000  ...  0.000002   \n",
       "1      0.000000e+00  0.000000e+00   0.75  0.000000  0.000000  ...  0.000000   \n",
       "2      0.000000e+00  0.000000e+00   0.75  0.000000  0.000000  ...  0.000000   \n",
       "3      0.000000e+00  0.000000e+00   0.75  0.000000  0.000000  ...  0.000000   \n",
       "4      0.000000e+00  0.000000e+00   0.75  0.000000  0.000000  ...  0.000000   \n",
       "...             ...           ...    ...       ...       ...  ...       ...   \n",
       "12154  0.000000e+00  0.000000e+00   1.00  0.000000  0.000000  ...  0.000000   \n",
       "12155  2.000005e-07  2.000005e-07   0.75  0.000000  0.000000  ...  0.000008   \n",
       "12156  0.000000e+00  0.000000e+00   1.00  0.000000  0.000000  ...  0.000000   \n",
       "12157  4.079091e-02  4.079091e-02   0.50  0.830357  0.215054  ...  0.000000   \n",
       "12158  4.088011e-02  4.088011e-02   0.50  0.830357  0.215054  ...  0.000000   \n",
       "\n",
       "        DstWin  sVid  dVid  SrcTCPBase  DstTCPBase  TcpRtt  SynAck  AckDat  \\\n",
       "0      0.00000   0.0   0.0    0.388654    0.000000     0.0     0.0     0.0   \n",
       "1      0.00000   0.0   0.0    0.388654    0.000000     0.0     0.0     0.0   \n",
       "2      0.00000   0.0   0.0    0.388654    0.000000     0.0     0.0     0.0   \n",
       "3      0.00000   0.0   0.0    0.388654    0.000000     0.0     0.0     0.0   \n",
       "4      0.00000   0.0   0.0    0.388669    0.000000     0.0     0.0     0.0   \n",
       "...        ...   ...   ...         ...         ...     ...     ...     ...   \n",
       "12154  0.00000   1.0   0.0    0.000000    0.000000     0.0     0.0     0.0   \n",
       "12155  0.00025   0.0   0.0    0.292755    0.030151     0.0     0.0     0.0   \n",
       "12156  0.00000   1.0   0.0    0.000000    0.000000     0.0     0.0     0.0   \n",
       "12157  0.00000   0.0   1.0    0.000000    0.000000     0.0     0.0     0.0   \n",
       "12158  0.00000   0.0   1.0    0.000000    0.000000     0.0     0.0     0.0   \n",
       "\n",
       "       Label  \n",
       "0          1  \n",
       "1          1  \n",
       "2          1  \n",
       "3          1  \n",
       "4          1  \n",
       "...      ...  \n",
       "12154      0  \n",
       "12155      0  \n",
       "12156      0  \n",
       "12157      0  \n",
       "12158      0  \n",
       "\n",
       "[12159 rows x 49 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Automated Data Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated Transformation/Encoding\n",
    "Automatically identify and transform string/text features into numerical features to make the data more readable by ML models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the automated data encoding function\n",
    "def Auto_Encoding(df):\n",
    "    cat_features=[x for x in df.columns if df[x].dtype==\"object\"] ## Find string/text features\n",
    "    le=LabelEncoder()\n",
    "    for col in cat_features:\n",
    "        if col in df.columns:\n",
    "            i = df.columns.get_loc(col)\n",
    "            # Transform to numerical features\n",
    "            df.iloc[:,i] = df.apply(lambda i:le.fit_transform(i.astype(str)), axis=0, result_type='expand')\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=Auto_Encoding(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated Imputation\n",
    "Detect and impute missing values to improve data quality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the automated data imputation function\n",
    "def Auto_Imputation(df):\n",
    "    if df.isnull().values.any() or np.isinf(df).values.any(): # if there is any empty or infinite values\n",
    "        df.replace([np.inf, -np.inf], np.nan, inplace=True)\n",
    "        df.fillna(0, inplace = True)  # Replace empty values with zeros; there are other imputation methods discussed in the paper\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=Auto_Imputation(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated normalization\n",
    "Normalize the range of features to a similar scale to improve data quality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def Auto_Normalization(df):\n",
    "    stat, p = shapiro(df)\n",
    "    print('Statistics=%.3f, p=%.3f' % (stat, p))\n",
    "    # interpret\n",
    "    alpha = 0.05\n",
    "    numeric_features = df.drop(['Label'],axis = 1).dtypes[df.dtypes != 'object'].index\n",
    "    \n",
    "    # check if the data distribution follows a Gaussian/normal distribution\n",
    "    # If so, select the Z-score normalization method; otherwise, select the min-max normalization\n",
    "    # Details are in the paper\n",
    "    if p > alpha:\n",
    "        print('Sample looks Gaussian (fail to reject H0)')\n",
    "        df[numeric_features] = df[numeric_features].apply(\n",
    "            lambda x: (x - x.mean()) / (x.std()))\n",
    "        print('Z-score normalization is automatically chosen and used')\n",
    "    else:\n",
    "        print('Sample does not look Gaussian (reject H0)')\n",
    "        df[numeric_features] = df[numeric_features].apply(\n",
    "            lambda x: (x - x.min()) / (x.max()-x.min()))\n",
    "        print('Min-max normalization is automatically chosen and used')\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Statistics=0.563, p=0.000\n",
      "Sample does not look Gaussian (reject H0)\n",
      "Min-max normalization is automatically chosen and used\n"
     ]
    }
   ],
   "source": [
    "df=Auto_Normalization(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=Auto_Imputation(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Train-test split\n",
    "Split the dataset into the training and the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop(['Label'],axis=1)\n",
    "y = df['Label']\n",
    "\n",
    "# Here we used the 80%/20% split, it can be changed based on specific tasks\n",
    "#X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2, shuffle=False,random_state = 0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2,random_state = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated data balancing\n",
    "Generate minority class samples to solve class-imbalance and improve data quality.  \n",
    "Synthetic Minority Over-sampling Technique (SMOTE) method is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    5927\n",
       "0    3800\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_train).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For binary data (can be modified for multi-class data with the same logic)\n",
    "def Auto_Balancing(X_train, y_train):\n",
    "    number0 = pd.Series(y_train).value_counts().iloc[0]\n",
    "    number1 = pd.Series(y_train).value_counts().iloc[1]\n",
    "    \n",
    "    if number0 > number1:\n",
    "        nlarge = number0\n",
    "    else:\n",
    "        nlarge = number1\n",
    "    \n",
    "    # evaluate whether the incoming dataset is imbalanced (the abnormal/normal ratio is smaller than a threshold (e.g., 50%)) \n",
    "    if (number1/number0 > 1.5) or (number0/number1 > 1.5):\n",
    "        smote=SMOTE(n_jobs=-1,sampling_strategy={0:nlarge, 1:nlarge})\n",
    "        X_train, y_train = smote.fit_sample(X_train, y_train)\n",
    "        \n",
    "    return X_train, y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train = Auto_Balancing(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    5927\n",
       "0    5927\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_train).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Automated Feature Engineering\n",
    "Feature selection method 1: **Information Gain (IG)**, used to remove irrelevant features to improve model efficiency  \n",
    "Feature selection method 2: **Pearson Correlation**, used to remove redundant features to improve model efficiency and accuracy  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove irrelevant features and select important features\n",
    "def Feature_Importance_IG(data):\n",
    "    features = data.drop(['Label'],axis=1).values  # \"Label\" should be changed to the target class variable name if different\n",
    "    labels = data['Label'].values\n",
    "    \n",
    "    # Extract feature names\n",
    "    feature_names = list(data.drop(['Label'],axis=1).columns)\n",
    "\n",
    "    # Empty array for feature importances\n",
    "    feature_importance_values = np.zeros(len(feature_names))\n",
    "    model = lgb.LGBMRegressor(verbose = -1)\n",
    "    model.fit(features, labels)\n",
    "    feature_importances = pd.DataFrame({'feature': feature_names, 'importance': model.feature_importances_})\n",
    "\n",
    "    # Sort features according to importance\n",
    "    feature_importances = feature_importances.sort_values('importance', ascending = False).reset_index(drop = True)\n",
    "\n",
    "    # Normalize the feature importances to add up to one\n",
    "    feature_importances['normalized_importance'] = feature_importances['importance'] / feature_importances['importance'].sum()\n",
    "    feature_importances['cumulative_importance'] = np.cumsum(feature_importances['normalized_importance'])\n",
    "    \n",
    "    cumulative_importance=0.90 # Only keep the important features with cumulative importance scores>=90%. It can be changed.\n",
    "\n",
    "    # Make sure most important features are on top\n",
    "    feature_importances = feature_importances.sort_values('cumulative_importance')\n",
    "\n",
    "    # Identify the features not needed to reach the cumulative_importance\n",
    "    record_low_importance = feature_importances[feature_importances['cumulative_importance'] > cumulative_importance]\n",
    "\n",
    "    to_drop = list(record_low_importance['feature'])\n",
    "#     print(feature_importances.drop(['importance'],axis=1))\n",
    "    return to_drop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove redundant features\n",
    "def Feature_Redundancy_Pearson(data):\n",
    "    correlation_threshold=0.90 # Only remove features with the redundancy>90%. It can be changed\n",
    "    features = data.drop(['Label'],axis=1)\n",
    "    corr_matrix = features.corr()\n",
    "\n",
    "    # Extract the upper triangle of the correlation matrix\n",
    "    upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k = 1).astype(np.bool))\n",
    "\n",
    "    # Select the features with correlations above the threshold\n",
    "    # Need to use the absolute value\n",
    "    to_drop = [column for column in upper.columns if any(upper[column].abs() > correlation_threshold)]\n",
    "\n",
    "    # Dataframe to hold correlated pairs\n",
    "    record_collinear = pd.DataFrame(columns = ['drop_feature', 'corr_feature', 'corr_value'])\n",
    "\n",
    "    # Iterate through the columns to drop\n",
    "    for column in to_drop:\n",
    "\n",
    "        # Find the correlated features\n",
    "        corr_features = list(upper.index[upper[column].abs() > correlation_threshold])\n",
    "\n",
    "        # Find the correlated values\n",
    "        corr_values = list(upper[column][upper[column].abs() > correlation_threshold])\n",
    "        drop_features = [column for _ in range(len(corr_features))]    \n",
    "\n",
    "        # Record the information (need a temp df for now)\n",
    "        temp_df = pd.DataFrame.from_dict({'drop_feature': drop_features,\n",
    "                                         'corr_feature': corr_features,\n",
    "                                         'corr_value': corr_values})\n",
    "        record_collinear = record_collinear.append(temp_df, ignore_index = True)\n",
    "#     print(record_collinear)\n",
    "    return to_drop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Auto_Feature_Engineering(df):\n",
    "    drop1 = Feature_Importance_IG(df)\n",
    "    dfh1 = df.drop(columns = drop1)\n",
    "    \n",
    "    drop2 = Feature_Redundancy_Pearson(dfh1)\n",
    "    dfh2 = dfh1.drop(columns = drop2)\n",
    "    \n",
    "    return dfh2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Seq</th>\n",
       "      <th>Dur</th>\n",
       "      <th>Proto</th>\n",
       "      <th>sTtl</th>\n",
       "      <th>dTtl</th>\n",
       "      <th>sHops</th>\n",
       "      <th>TotBytes</th>\n",
       "      <th>Offset</th>\n",
       "      <th>sMeanPktSz</th>\n",
       "      <th>dMeanPktSz</th>\n",
       "      <th>pLoss</th>\n",
       "      <th>Rate</th>\n",
       "      <th>SrcWin</th>\n",
       "      <th>DstWin</th>\n",
       "      <th>SrcTCPBase</th>\n",
       "      <th>TcpRtt</th>\n",
       "      <th>SynAck</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000496</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.211765</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.357143</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>0.000201</td>\n",
       "      <td>0.042208</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.388654</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.002442</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.145098</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.964286</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>0.000850</td>\n",
       "      <td>0.042208</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.388654</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.003295</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.149020</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.928571</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>0.001133</td>\n",
       "      <td>0.042208</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.388654</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.003317</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.211765</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.357143</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>0.001140</td>\n",
       "      <td>0.042208</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.388654</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.003375</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.211765</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.357143</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>0.001160</td>\n",
       "      <td>0.042208</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.388669</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12154</th>\n",
       "      <td>0.003688</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.976471</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.000014</td>\n",
       "      <td>0.003075</td>\n",
       "      <td>0.053852</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12155</th>\n",
       "      <td>0.004104</td>\n",
       "      <td>2.000005e-07</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.203922</td>\n",
       "      <td>0.250980</td>\n",
       "      <td>0.428571</td>\n",
       "      <td>0.000158</td>\n",
       "      <td>0.003462</td>\n",
       "      <td>0.039298</td>\n",
       "      <td>0.053791</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>0.00025</td>\n",
       "      <td>0.292755</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12156</th>\n",
       "      <td>0.005234</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.976471</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.000014</td>\n",
       "      <td>0.004298</td>\n",
       "      <td>0.053852</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12157</th>\n",
       "      <td>0.000015</td>\n",
       "      <td>4.079091e-02</td>\n",
       "      <td>0.50</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.980392</td>\n",
       "      <td>0.035714</td>\n",
       "      <td>0.000145</td>\n",
       "      <td>0.004546</td>\n",
       "      <td>0.069862</td>\n",
       "      <td>0.060603</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000003</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12158</th>\n",
       "      <td>0.000015</td>\n",
       "      <td>4.088011e-02</td>\n",
       "      <td>0.50</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.980392</td>\n",
       "      <td>0.035714</td>\n",
       "      <td>0.000145</td>\n",
       "      <td>0.004757</td>\n",
       "      <td>0.069862</td>\n",
       "      <td>0.060603</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000003</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12159 rows × 18 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            Seq           Dur  Proto      sTtl      dTtl     sHops  TotBytes  \\\n",
       "0      0.000496  0.000000e+00   0.75  0.211765  0.000000  0.357143  0.000007   \n",
       "1      0.002442  0.000000e+00   0.75  0.145098  0.000000  0.964286  0.000007   \n",
       "2      0.003295  0.000000e+00   0.75  0.149020  0.000000  0.928571  0.000007   \n",
       "3      0.003317  0.000000e+00   0.75  0.211765  0.000000  0.357143  0.000007   \n",
       "4      0.003375  0.000000e+00   0.75  0.211765  0.000000  0.357143  0.000007   \n",
       "...         ...           ...    ...       ...       ...       ...       ...   \n",
       "12154  0.003688  0.000000e+00   1.00  0.976471  0.000000  0.250000  0.000014   \n",
       "12155  0.004104  2.000005e-07   0.75  0.203922  0.250980  0.428571  0.000158   \n",
       "12156  0.005234  0.000000e+00   1.00  0.976471  0.000000  0.250000  0.000014   \n",
       "12157  0.000015  4.079091e-02   0.50  1.000000  0.980392  0.035714  0.000145   \n",
       "12158  0.000015  4.088011e-02   0.50  1.000000  0.980392  0.035714  0.000145   \n",
       "\n",
       "         Offset  sMeanPktSz  dMeanPktSz  pLoss      Rate    SrcWin   DstWin  \\\n",
       "0      0.000201    0.042208    0.000000    0.0  0.000000  0.000002  0.00000   \n",
       "1      0.000850    0.042208    0.000000    0.0  0.000000  0.000000  0.00000   \n",
       "2      0.001133    0.042208    0.000000    0.0  0.000000  0.000000  0.00000   \n",
       "3      0.001140    0.042208    0.000000    0.0  0.000000  0.000000  0.00000   \n",
       "4      0.001160    0.042208    0.000000    0.0  0.000000  0.000000  0.00000   \n",
       "...         ...         ...         ...    ...       ...       ...      ...   \n",
       "12154  0.003075    0.053852    0.000000    0.0  0.000000  0.000000  0.00000   \n",
       "12155  0.003462    0.039298    0.053791    0.0  1.000000  0.000008  0.00025   \n",
       "12156  0.004298    0.053852    0.000000    0.0  0.000000  0.000000  0.00000   \n",
       "12157  0.004546    0.069862    0.060603    0.0  0.000003  0.000000  0.00000   \n",
       "12158  0.004757    0.069862    0.060603    0.0  0.000003  0.000000  0.00000   \n",
       "\n",
       "       SrcTCPBase  TcpRtt  SynAck  Label  \n",
       "0        0.388654     0.0     0.0      1  \n",
       "1        0.388654     0.0     0.0      1  \n",
       "2        0.388654     0.0     0.0      1  \n",
       "3        0.388654     0.0     0.0      1  \n",
       "4        0.388669     0.0     0.0      1  \n",
       "...           ...     ...     ...    ...  \n",
       "12154    0.000000     0.0     0.0      0  \n",
       "12155    0.292755     0.0     0.0      0  \n",
       "12156    0.000000     0.0     0.0      0  \n",
       "12157    0.000000     0.0     0.0      0  \n",
       "12158    0.000000     0.0     0.0      0  \n",
       "\n",
       "[12159 rows x 18 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfh2 = Auto_Feature_Engineering(df)\n",
    "dfh2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Split & Balancing (After Feature Engineering)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = dfh2.drop(['Label'],axis=1)\n",
    "y = dfh2['Label']\n",
    "\n",
    "#X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2, shuffle=False,random_state = 0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2,random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train = Auto_Balancing(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Automated Model Selection\n",
    "Select the best-performing model among five common machine learning models (Naive Bayes, KNN, random forest, LightGBM, and ANN/MLP) by evaluating their learning performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model learning (for Comparison)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.79400000000001%\n",
      "Precision: 99.732%\n",
      "Recall: 99.933%\n",
      "F1-score: 99.832%\n",
      "Time: 4.10164\n",
      "Wall time: 322 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lg = lgb.LGBMClassifier(verbose = -1)\n",
    "lg.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = lg.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.79400000000001%\n",
      "Precision: 99.799%\n",
      "Recall: 99.866%\n",
      "F1-score: 99.832%\n",
      "Time: 10.25318\n",
      "Wall time: 753 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = RandomForestClassifier()\n",
    "rf.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 4. Hyperparameter Optimization\n",
    "Optimize the best performing machine learning model (lightGBM) by tuning its hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hold-out validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████| 50/50 [00:22<00:00,  2.18trial/s, best loss: -0.998766447368421]\n",
      "LightGBM: Hyperopt estimated optimum {'learning_rate': 0.2326246024181119, 'max_depth': 16.0, 'min_child_samples': 50.0, 'n_estimators': 200.0, 'num_leaves': 200.0}\n"
     ]
    }
   ],
   "source": [
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "\n",
    "# Define the objective function\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'n_estimators': int(params['n_estimators']), \n",
    "        'max_depth': int(params['max_depth']),\n",
    "        'learning_rate': abs(float(params['learning_rate'])),\n",
    "        \"num_leaves\": int(params['num_leaves']),\n",
    "        \"min_child_samples\": int(params['min_child_samples']),\n",
    "    }\n",
    "    clf = lgb.LGBMClassifier( **params)\n",
    "    clf.fit(X_train,y_train)\n",
    "    predictions = clf.predict(X_test)\n",
    "    score = accuracy_score(y_test,predictions)\n",
    "    return {'loss':-score, 'status': STATUS_OK }\n",
    "\n",
    "# Define the hyperparameter configuration space\n",
    "space = {\n",
    "    'n_estimators': hp.quniform('n_estimators', 50, 500, 20),\n",
    "    'max_depth': hp.quniform('max_depth', 5, 50, 1),\n",
    "    \"learning_rate\":hp.uniform('learning_rate', 0, 1),\n",
    "    \"num_leaves\":hp.quniform('num_leaves',100,2000,100),\n",
    "    \"min_child_samples\":hp.quniform('min_child_samples',10,50,5),\n",
    "}\n",
    "\n",
    "# Detect the optimal hyperparameter values\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=50)\n",
    "print(\"LightGBM: Hyperopt estimated optimum {}\".format(best))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.87700000000001%\n",
      "Precision: 99.799%\n",
      "Recall: 100.0%\n",
      "F1-score: 99.899%\n",
      "Wall time: 199 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = lgb.LGBMClassifier(max_depth=21, learning_rate= 0.6251262379739384, n_estimators = 60, \n",
    "                         num_leaves = 1900, min_child_samples = 30)\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "After hyperparameter optimization, the hold-out accuracy has been improved from 99.806% to 99.841%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████| 20/20 [00:39<00:00,  1.95s/trial, best loss: -0.9991776315789473]\n",
      "RandomForest: Hyperopt estimated optimum {'criterion': 1, 'max_depth': 37.0, 'min_samples_leaf': 2.0, 'min_samples_split': 6.0, 'n_estimators': 200.0}\n"
     ]
    }
   ],
   "source": [
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# Define the objective function\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'n_estimators': int(params['n_estimators']), \n",
    "        'max_depth': int(params['max_depth']),\n",
    "        'min_samples_split': int(params['min_samples_split']),\n",
    "        'min_samples_leaf': int(params['min_samples_leaf']),\n",
    "        'criterion': str(params['criterion'])\n",
    "    }\n",
    "    clf = RandomForestClassifier(**params)\n",
    "    clf.fit(X_train,y_train)\n",
    "    predictions = clf.predict(X_test)\n",
    "    score = accuracy_score(y_test,predictions)\n",
    "    return {'loss':-score, 'status': STATUS_OK }\n",
    "\n",
    "# Define the hyperparameter configuration space\n",
    "space = {\n",
    "    'n_estimators': hp.quniform('n_estimators', 50, 500, 20),\n",
    "    'max_depth': hp.quniform('max_depth', 5, 50, 1),\n",
    "    'min_samples_split': hp.quniform('min_samples_split', 2, 11, 1),\n",
    "    'min_samples_leaf': hp.quniform('min_samples_leaf', 1, 11, 1),\n",
    "    'criterion': hp.choice('criterion', ['gini', 'entropy'])\n",
    "}\n",
    "\n",
    "# Detect the optimal hyperparameter values\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=20)\n",
    "print(\"RandomForest: Hyperopt estimated optimum {}\".format(best))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.836%\n",
      "Precision: 99.866%\n",
      "Recall: 99.866%\n",
      "F1-score: 99.866%\n",
      "Wall time: 466 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = RandomForestClassifier(max_depth=40, n_estimators = 60, min_samples_split = 4,\n",
    "                         min_samples_leaf = 1, criterion = 'entropy')\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AML"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experiment 1\n",
    "* 1. Original ML model for IDS\n",
    "* 2. generate adversarial samples using DecisionTreeAttack or other attacks\n",
    "* 3. test ML model under attack\n",
    "* 4. develop adversarial sample detection model\n",
    "* 5. remove adversarial samples from the training set\n",
    "* 6. re-train IDS model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DecisionTreeAttack"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Original ML model for IDS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.87700000000001%\n",
      "Precision: 99.799%\n",
      "Recall: 100.0%\n",
      "F1-score: 99.899%\n",
      "Wall time: 247 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = lgb.LGBMClassifier(max_depth=21, learning_rate= 0.6251262379739384, n_estimators = 60, \n",
    "                         num_leaves = 1900, min_child_samples = 30)\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: generate adversarial samples using DecisionTreeAttack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Decision tree attack: 100%|██████████| 11854/11854 [00:03<00:00, 2980.75it/s]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.datasets import load_digits\n",
    "from matplotlib import pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "from art.attacks.evasion import DecisionTreeAttack\n",
    "from art.estimators.classification import SklearnClassifier\n",
    "\n",
    "clf = DecisionTreeClassifier()\n",
    "clf.fit(X_train,y_train)\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "y_train1 = y_train.values.reshape(-1, 1)\n",
    "y_train1_AML = OneHotEncoder().fit_transform(y_train1).toarray()\n",
    "\n",
    "clf_art = SklearnClassifier(clf)\n",
    "attack = DecisionTreeAttack(clf_art)\n",
    "\n",
    "# Generate adversarial examples\n",
    "x_adv = attack.generate(X_train.values,y_train1_AML)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: test ML model under attack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 62.458999999999996%\n",
      "Precision: 68.947%\n",
      "Recall: 70.383%\n",
      "F1-score: 69.658%\n",
      "Time: 2.87043\n",
      "Wall time: 224 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#DTA\n",
    "rf = lgb.LGBMClassifier(max_depth=21, learning_rate= 0.6251262379739384, n_estimators = 60, \n",
    "                         num_leaves = 1900, min_child_samples = 30)\n",
    "rf.fit(x_adv,y_train)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: develop adversarial sample detection model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new = np.concatenate([x_adv,X_train])\n",
    "y1 = pd.Series(np.ones(y_train.shape[0]))\n",
    "y2 = pd.Series(np.zeros(y_train.shape[0]))\n",
    "y_new = np.concatenate([y1,y2])\n",
    "\n",
    "\n",
    "X_train_d, X_test_d, y_train_d, y_test_d = train_test_split(X_new,y_new, train_size = 0.1, test_size = 0.9,random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.822%\n",
      "Precision: 99.72%\n",
      "Recall: 99.925%\n",
      "F1-score: 99.822%\n",
      "Time: 1.35512\n",
      "Wall time: 234 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = lgb.LGBMClassifier(max_depth=21, learning_rate= 0.6251262379739384, n_estimators = 60, \n",
    "                         num_leaves = 1900, min_child_samples = 30)\n",
    "rf.fit(X_train_d,y_train_d)\n",
    "t1=time.time()\n",
    "predictions2 = rf.predict(X_test_d)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test_d)*1000000,5)))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: remove adversarial samples from the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "detection_results = rf.predict(X_new)\n",
    "X_new1 = X_new\n",
    "y_new1 = np.concatenate([y_train,y_train])\n",
    "\n",
    "indices_to_remove = [i for i in range(len(detection_results)) if detection_results[i] == 1]\n",
    "for i in reversed(indices_to_remove):\n",
    "    X_new1 = np.delete(X_new1, i, axis=0)\n",
    "    y_new1 = np.delete(y_new1, i)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: re-train IDS model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.87700000000001%\n",
      "Precision: 99.799%\n",
      "Recall: 100.0%\n",
      "F1-score: 99.899%\n",
      "Time: 2.87053\n",
      "Wall time: 180 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#DTA\n",
    "rf = lgb.LGBMClassifier(max_depth=21, learning_rate= 0.6251262379739384, n_estimators = 60, \n",
    "                         num_leaves = 1900, min_child_samples = 30)\n",
    "rf.fit(X_new1,y_new1)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FastGradientMethod"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Original ML model for IDS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.87700000000001%\n",
      "Precision: 99.799%\n",
      "Recall: 100.0%\n",
      "F1-score: 99.899%\n",
      "Wall time: 275 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = lgb.LGBMClassifier(max_depth=21, learning_rate= 0.6251262379739384, n_estimators = 60, \n",
    "                         num_leaves = 1900, min_child_samples = 30)\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: generate adversarial samples using DecisionTreeAttack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Input,Dense,Dropout,BatchNormalization,Activation\n",
    "from art.estimators.classification import KerasClassifier\n",
    "from keras import Model\n",
    "import keras.backend as K\n",
    "import keras.callbacks as kcallbacks\n",
    "from keras import optimizers\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "from art.attacks import evasion\n",
    "from art.attacks.evasion import FastGradientMethod\n",
    "from art.estimators.classification import SklearnClassifier\n",
    "\n",
    "def ANN(optimizer = 'sgd',neurons=16,batch_size=1024,epochs=80,activation='relu',patience=8,loss='binary_crossentropy'):\n",
    "    inputs=Input(shape=(X.shape[1],))\n",
    "    x=Dense(1000)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.3)(x)\n",
    "    x=Dense(256)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.25)(x)\n",
    "    x=Dense(2,activation='softmax')(x)\n",
    "    model=Model(inputs=inputs,outputs=x,name='base_nlp')\n",
    "    model.compile(optimizer='adam',loss='categorical_crossentropy')\n",
    "#     model.compile(optimizer=Adam(lr = 0.01),loss='categorical_crossentropy',metrics=['accuracy'])\n",
    "    early_stopping = EarlyStopping(monitor=\"loss\", patience = patience)# early stop patience\n",
    "    history = model.fit(X, pd.get_dummies(y).values,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              callbacks = [early_stopping],\n",
    "              verbose=0) #verbose set to 1 will show the training process\n",
    "    return model\n",
    "\n",
    "# Create a KerasClassifier\n",
    "classifier = KerasClassifier(model=ANN(), clip_values=(0, 1))\n",
    "\n",
    "# Create the FastGradientMethod attack\n",
    "attack = FastGradientMethod(estimator=classifier, eps=0.1)\n",
    "\n",
    "# Generate adversarial examples\n",
    "x_adv = attack.generate(X_train.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: test ML model under attack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 40.583999999999996%\n",
      "Precision: 97.82600000000001%\n",
      "Recall: 3.0220000000000002%\n",
      "F1-score: 5.863%\n",
      "Time: 5.74126\n",
      "Wall time: 188 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#FGM\n",
    "rf = lgb.LGBMClassifier(max_depth=21, learning_rate= 0.6251262379739384, n_estimators = 60, \n",
    "                         num_leaves = 1900, min_child_samples = 30)\n",
    "rf.fit(x_adv,y_train)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: develop adversarial sample detection model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new = np.concatenate([x_adv,X_train])\n",
    "y1 = pd.Series(np.ones(y_train.shape[0]))\n",
    "y2 = pd.Series(np.zeros(y_train.shape[0]))\n",
    "y_new = np.concatenate([y1,y2])\n",
    "\n",
    "\n",
    "X_train_d, X_test_d, y_train_d, y_test_d = train_test_split(X_new,y_new, train_size = 0.1, test_size = 0.9,random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.663%\n",
      "Precision: 99.775%\n",
      "Recall: 99.55000000000001%\n",
      "F1-score: 99.66199999999999%\n",
      "Time: 1.02829\n",
      "Wall time: 183 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = lgb.LGBMClassifier(max_depth=21, learning_rate= 0.6251262379739384, n_estimators = 60, \n",
    "                         num_leaves = 1900, min_child_samples = 30)\n",
    "rf.fit(X_train_d,y_train_d)\n",
    "t1=time.time()\n",
    "predictions2 = rf.predict(X_test_d)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test_d)*1000000,5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: remove adversarial samples from the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "detection_results = rf.predict(X_new)\n",
    "X_new1 = X_new\n",
    "y_new1 = np.concatenate([y_train,y_train])\n",
    "\n",
    "indices_to_remove = [i for i in range(len(detection_results)) if detection_results[i] == 1]\n",
    "for i in reversed(indices_to_remove):\n",
    "    X_new1 = np.delete(X_new1, i, axis=0)\n",
    "    y_new1 = np.delete(y_new1, i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: re-train IDS model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.87700000000001%\n",
      "Precision: 99.799%\n",
      "Recall: 100.0%\n",
      "F1-score: 99.899%\n",
      "Time: 3.28335\n",
      "Wall time: 404 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#FGM\n",
    "rf = lgb.LGBMClassifier(max_depth=21, learning_rate= 0.6251262379739384, n_estimators = 60, \n",
    "                         num_leaves = 1900, min_child_samples = 30)\n",
    "rf.fit(X_new1,y_new1)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BasicIterativeMethod"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Original ML model for IDS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.87700000000001%\n",
      "Precision: 99.799%\n",
      "Recall: 100.0%\n",
      "F1-score: 99.899%\n",
      "Wall time: 224 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = lgb.LGBMClassifier(max_depth=21, learning_rate= 0.6251262379739384, n_estimators = 60, \n",
    "                         num_leaves = 1900, min_child_samples = 30)\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: generate adversarial samples using FastGradientMethod"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Input,Dense,Dropout,BatchNormalization,Activation\n",
    "from art.estimators.classification import KerasClassifier\n",
    "from keras import Model\n",
    "import keras.backend as K\n",
    "import keras.callbacks as kcallbacks\n",
    "from keras import optimizers\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "from art.attacks import evasion\n",
    "from art.attacks.evasion import FastGradientMethod\n",
    "from art.estimators.classification import SklearnClassifier\n",
    "\n",
    "def ANN(optimizer = 'sgd',neurons=16,batch_size=1024,epochs=80,activation='relu',patience=8,loss='binary_crossentropy'):\n",
    "    inputs=Input(shape=(X.shape[1],))\n",
    "    x=Dense(1000)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.3)(x)\n",
    "    x=Dense(256)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.25)(x)\n",
    "    x=Dense(2,activation='softmax')(x)\n",
    "    model=Model(inputs=inputs,outputs=x,name='base_nlp')\n",
    "    model.compile(optimizer='adam',loss='categorical_crossentropy')\n",
    "#     model.compile(optimizer=Adam(lr = 0.01),loss='categorical_crossentropy',metrics=['accuracy'])\n",
    "    early_stopping = EarlyStopping(monitor=\"loss\", patience = patience)# early stop patience\n",
    "    history = model.fit(X, pd.get_dummies(y).values,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              callbacks = [early_stopping],\n",
    "              verbose=0) #verbose set to 1 will show the training process\n",
    "    return model\n",
    "\n",
    "# Create a KerasClassifier\n",
    "classifier = KerasClassifier(model=ANN(), clip_values=(0, 1))\n",
    "\n",
    "# Create the FastGradientMethod attack\n",
    "attack = evasion.BasicIterativeMethod(estimator=classifier, eps=0.1, eps_step=0.1, max_iter=200, batch_size=32, verbose = False)\n",
    "\n",
    "# Generate adversarial examples\n",
    "x_adv = attack.generate(X_train.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: test ML model under attack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 38.734%\n",
      "Precision: 33.333%\n",
      "Recall: 0.067%\n",
      "F1-score: 0.134%\n",
      "Time: 2.46036\n",
      "Wall time: 205 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#BIM\n",
    "rf = lgb.LGBMClassifier(max_depth=21, learning_rate= 0.6251262379739384, n_estimators = 60, \n",
    "                         num_leaves = 1900, min_child_samples = 30)\n",
    "rf.fit(x_adv,y_train)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: develop adversarial sample detection model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new = np.concatenate([x_adv,X_train])\n",
    "y1 = pd.Series(np.ones(y_train.shape[0]))\n",
    "y2 = pd.Series(np.zeros(y_train.shape[0]))\n",
    "y_new = np.concatenate([y1,y2])\n",
    "\n",
    "\n",
    "X_train_d, X_test_d, y_train_d, y_test_d = train_test_split(X_new,y_new, train_size = 0.1, test_size = 0.9,random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.634%\n",
      "Precision: 99.579%\n",
      "Recall: 99.691%\n",
      "F1-score: 99.63499999999999%\n",
      "Time: 0.88804\n",
      "Wall time: 184 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = lgb.LGBMClassifier(max_depth=21, learning_rate= 0.6251262379739384, n_estimators = 60, \n",
    "                         num_leaves = 1900, min_child_samples = 30)\n",
    "rf.fit(X_train_d,y_train_d)\n",
    "t1=time.time()\n",
    "predictions2 = rf.predict(X_test_d)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test_d,predictions2),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test_d)*1000000,5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: remove adversarial samples from the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "detection_results = rf.predict(X_new)\n",
    "X_new1 = X_new\n",
    "y_new1 = np.concatenate([y_train,y_train])\n",
    "\n",
    "indices_to_remove = [i for i in range(len(detection_results)) if detection_results[i] == 1]\n",
    "for i in reversed(indices_to_remove):\n",
    "    X_new1 = np.delete(X_new1, i, axis=0)\n",
    "    y_new1 = np.delete(y_new1, i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: re-train IDS model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.836%\n",
      "Precision: 99.732%\n",
      "Recall: 100.0%\n",
      "F1-score: 99.866%\n",
      "Time: 2.87122\n",
      "Wall time: 195 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#BIM\n",
    "rf = lgb.LGBMClassifier(max_depth=21, learning_rate= 0.6251262379739384, n_estimators = 60, \n",
    "                         num_leaves = 1900, min_child_samples = 30)\n",
    "rf.fit(X_new1,y_new1)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}