fraud-detection-demo-with-p2p.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Exploring Fraud Detection With Neo4j & Graph Data Science\n",
    "\n",
    "This analysis uses [Neo4j and Graph Data Science (GDS)](https://neo4j.com/docs/graph-data-science/current/) to explore an anonymized data sample from a Peer-to-Peer (P2P) payment platform.  The notebook is split up into the following sections to cover various stages of the graph data science workflow:\n",
    "\n",
    "- Notebook Setup\n",
    "- Part 1: Exploring Connected Fraud Data\n",
    "- Part 2: Resolving Fraud Communities using Entity Resolution and Community Detection\n",
    "- Part 3: Recommending Suspicious Accounts With Centrality & Node Similarity\n",
    "- Part 4: Predicting Fraud Risk Accounts with Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook Setup <a name=\"p0\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import configparser\n",
    "import os\n",
    "pd.set_option('display.width', 0)\n",
    "pd.set_option('display.max_colwidth', 500)\n",
    "pd.set_option('display.max_rows', 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neo4j Settings\n",
    "The `NEO4J_PROPERTIES_FILE` is an ini configuration file for Neo4j properties so this notebook can connect to\n",
    "your Neo4j instance and load data. The ini file should be formatted as follows\n",
    "```\n",
    "[NEO4J]\n",
    "PASSWORD=<password>\n",
    "USERNAME=<database name, is 'neo4j' by default>\n",
    "HOST=<host uri>\n",
    "```\n",
    "\n",
    "Set `NEO4J_PROPERTIES_FILE` to None or any non-existent location to use the below defaults\n",
    "```\n",
    "HOST = 'neo4j://localhost'\n",
    "USERNAME = 'neo4j'\n",
    "PASSWORD = 'password'\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "NEO4J_PROPERTIES_FILE = '/Users/zachblumenfeld/devtools/auth/aura-p2p-fd.ini'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using custom database properties\n"
     ]
    }
   ],
   "source": [
    "## Using an ini file for credentials, otherwise providing defaults\n",
    "HOST = 'neo4j://localhost'\n",
    "USERNAME = 'neo4j'\n",
    "PASSWORD = 'password'\n",
    "\n",
    "if NEO4J_PROPERTIES_FILE is not None and os.path.exists(NEO4J_PROPERTIES_FILE):\n",
    "    config = configparser.RawConfigParser()\n",
    "    config.read(NEO4J_PROPERTIES_FILE)\n",
    "    HOST = config['NEO4J']['HOST']\n",
    "    USERNAME = config['NEO4J']['USERNAME']\n",
    "    PASSWORD = config['NEO4J']['PASSWORD']\n",
    "    print('Using custom database properties')\n",
    "else:\n",
    "    print('Could not find database properties file, using defaults')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect to Graph Data Science"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from graphdatascience import GraphDataScience\n",
    "\n",
    "# Use Neo4j URI and credentials according to your setup\n",
    "gds = GraphDataScience(HOST, auth=(USERNAME, PASSWORD), aura_ds=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clear_graph_by_name(g_name):\n",
    "    if gds.graph.exists(g_name).exists:\n",
    "        g = gds.graph.get(g_name)\n",
    "        gds.graph.drop(g)\n",
    "\n",
    "def clear_all_graphs():\n",
    "    g_names = gds.graph.list().graphName.tolist()\n",
    "    for g_name in g_names:\n",
    "        g = gds.graph.get(g_name)\n",
    "        gds.graph.drop(g)\n",
    "\n",
    "def identifier_degrees(user_label, degree_property):\n",
    "    g_name = 'id-projection'\n",
    "    clear_graph_by_name(g_name)\n",
    "    g, _  = gds.graph.project(g_name, [user_label, 'Card', 'Device', 'IP'],{\n",
    "        'HAS_CC': {'orientation': 'REVERSE'},\n",
    "        'HAS_IP': {'orientation': 'REVERSE'},\n",
    "        'USED': {'orientation': 'REVERSE'}\n",
    "    })\n",
    "    gds.degree.mutate(g, mutateProperty=degree_property)\n",
    "    gds.graph.writeNodeProperties(g, [degree_property], ['Card', 'Device', 'IP'])\n",
    "    g.drop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Exploring Connected Fraud Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset Introduction\n",
    "\n",
    "We will be using an anonymized sample of user accounts and transactions from a real-world Peer-to-Peer (P2P) platform. Prior to ingesting the data into graph, the original identification numbers were removed and categorical values were masked. Each user account has a unique 128-bit identifier, while the other nodes, representing unique credit cards, devices, and ip addresses have been assigned random UUIDs. These identifier are stored as the guid property in the graph schema.\n",
    "\n",
    "Below is a visualization of the graph schema:\n",
    "\n",
    "<img src=\"img/schema.png\" width=\"500\" >\n",
    "\n",
    "Each user node has an indicator variable for money transfer fraud (named MoneyTransferFraud) that is 1 for known fraud and 0 otherwise. This indicator is determined by a combination of credit card chargeback events and manual review. A chargeback is an action taken by a bank to reverse electronic payments. It involves reversing a payment and triggering a dispute resolution process, often for billing errors and unauthorized credit use. In short, a user must have at least one chargeback to be considered fraudulent. Only a small proportion of the user accounts, roughly 0.7 %, are flagged for fraud.\n",
    "\n",
    "Each user node has an indicator variable for money transfer fraud (named `MoneyTransferFraud`) that is 1 for known fraud and 0 otherwise. This indicator is determined by a combination of credit card chargeback events and manual review. A chargeback is an action taken by a bank to reverse electronic payments. It involves reversing a payment and triggering a dispute resolution process, often for billing errors and unauthorized credit use. In short, a user must have at least one chargeback to be considered fraudulent. Only a small proportion of the user accounts, roughly 0.7 %, are flagged for fraud."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Below is a breakdown of high-level counts by labels and relationships as well as the flagged accounts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nodeLabel</th>\n",
       "      <th>nodeCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>User</td>\n",
       "      <td>33732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Device</td>\n",
       "      <td>51451</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Card</td>\n",
       "      <td>118818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>IP</td>\n",
       "      <td>585855</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  nodeLabel  nodeCount\n",
       "0      User      33732\n",
       "1    Device      51451\n",
       "2      Card     118818\n",
       "3        IP     585855"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# total node counts\n",
    "gds.run_cypher('''\n",
    "    CALL apoc.meta.stats()\n",
    "    YIELD labels\n",
    "    UNWIND keys(labels) AS nodeLabel\n",
    "    RETURN nodeLabel, labels[nodeLabel] AS nodeCount\n",
    "''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>relationshipType</th>\n",
       "      <th>relationshipCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>USED</td>\n",
       "      <td>55026</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>HAS_IP</td>\n",
       "      <td>1488949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>HAS_CC</td>\n",
       "      <td>128066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>REFERRED</td>\n",
       "      <td>1870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>P2P</td>\n",
       "      <td>102832</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  relationshipType  relationshipCount\n",
       "0             USED              55026\n",
       "1           HAS_IP            1488949\n",
       "2           HAS_CC             128066\n",
       "3         REFERRED               1870\n",
       "4              P2P             102832"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# total relationship counts\n",
    "gds.run_cypher('''\n",
    "    CALL apoc.meta.stats()\n",
    "    YIELD relTypesCount\n",
    "    UNWIND keys(relTypesCount) AS relationshipType\n",
    "    RETURN relationshipType, relTypesCount[relationshipType] AS relationshipCount\n",
    "''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fraudMoneyTransfer</th>\n",
       "      <th>cnt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>33491</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>241</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fraudMoneyTransfer    cnt\n",
       "0                   0  33491\n",
       "1                   1    241"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#fraud money transfer flags\n",
    "gds.run_cypher('MATCH(u:User) RETURN u.fraudMoneyTransfer AS fraudMoneyTransfer, count(u) AS cnt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Closer Look at Cards and Devices\n",
    "\n",
    "Upon an initial investigation of aggregate statistics, the labeled fraud accounts do not look well connected which would naturally make one skeptical of a graph based approach.  This is exemplified well by the lack of discrimination in key identifiers.\n",
    "\n",
    "As a first step, I break out the ratio of fraud vs non-fraud users connected to credit cards and devices below. I find that very few cards or devices are centered well around flagged accounts. This is a bit surprising for me. If a card or device is used by a flagged account, I would expect the other accounts that use the card/device to also be mostly flagged for fraud as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count(u)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>241</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   count(u)\n",
       "0       241"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Setting a label for flagged users will enable faster lookups in cypher and faster gds projections\n",
    "gds.run_cypher('MATCH(u:User) WHERE u.fraudMoneyTransfer=1 SET u:FlaggedUser RETURN count(u)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use GDS degree centrality to count the number of Users connected to each identifier type - Card, Device, IP\n",
    "identifier_degrees('User', 'degree')\n",
    "# Use GDS degree centrality to count the number of FLAGGED Users connected to each identifier type - Card, Device, IP\n",
    "identifier_degrees('FlaggedUser', 'flaggedDegree')\n",
    "\n",
    "# Calculate the ratio of flagged users to total users\n",
    "gds.run_cypher('''\n",
    "    MATCH(n) WHERE n:Card OR n:Device OR n:IP\n",
    "    SET n.flaggedRatio = toFloat(n.flaggedDegree)/toFloat(n.degree)\n",
    "''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Flagged User Ratio for Card Count\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>flaggedUserRatio</th>\n",
       "      <th>count</th>\n",
       "      <th>percentCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>8185</td>\n",
       "      <td>0.964</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>0.004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Between 0-1</td>\n",
       "      <td>274</td>\n",
       "      <td>0.032</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  flaggedUserRatio  count  percentCount\n",
       "0                0   8185         0.964\n",
       "1                1     31         0.004\n",
       "2      Between 0-1    274         0.032"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Flagged User Ratio for Card Count')\n",
    "gds.run_cypher('''\n",
    "    MATCH(n:Card) WHERE n.degree > 1\n",
    "    WITH toFloat(count(n)) AS total\n",
    "    MATCH(n:Card) WHERE n.degree > 1\n",
    "    WITH n, total, CASE\n",
    "        WHEN n.flaggedRatio=0 THEN '0'\n",
    "        WHEN n.flaggedRatio=1  THEN '1'\n",
    "        ELSE 'Between 0-1' END AS flaggedUserRatio\n",
    "    RETURN flaggedUserRatio, count(n) AS count, round(toFloat(count(n))/total,3) AS percentCount \n",
    "    ORDER BY flaggedUserRatio\n",
    "''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Flagged User Ratio for Device Count\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>flaggedUserRatio</th>\n",
       "      <th>count</th>\n",
       "      <th>percentCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>2655</td>\n",
       "      <td>0.967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Between 0-1</td>\n",
       "      <td>88</td>\n",
       "      <td>0.032</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  flaggedUserRatio  count  percentCount\n",
       "0                0   2655         0.967\n",
       "1                1      2         0.001\n",
       "2      Between 0-1     88         0.032"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Flagged User Ratio for Device Count')\n",
    "gds.run_cypher('''\n",
    "    MATCH(n:Device) WHERE n.degree > 1\n",
    "    WITH toFloat(count(n)) AS total\n",
    "    MATCH(n:Device) WHERE n.degree > 1\n",
    "    WITH n, total, CASE\n",
    "        WHEN n.flaggedRatio=0 THEN '0'\n",
    "        WHEN n.flaggedRatio=1  THEN '1'\n",
    "        ELSE 'Between 0-1' END AS flaggedUserRatio\n",
    "    RETURN flaggedUserRatio, count(n) AS count, round(toFloat(count(n))/total,3) AS percentCount \n",
    "    ORDER BY flaggedUserRatio\n",
    "''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exploring Potential Fraud Patterns with Community Detection\n",
    "\n",
    "In this example, it is unclear to me what exactly is taking place, and I have a hunch that the fraud activity is not completely labeled given the lack of connectivity and the limited chargeback logic used to flag fraud. At the same time I do not want to simply label every user that shares a card or device with another flagged account, since it is possible that a benign user's device and or card was used fraudulently by another. Since fraudsters are actively avoiding being identified, actors committing fraud are often not just represented by a single user account, but rather by multiple accounts and identifiers which, hopefully for us, share some connections and similarities.\n",
    "\n",
    "In a graph, we can attempt to roughly identify these fragmented identities with Community Detection, a large set of methods that attempt to partition graphs into well connected groups a.k.a. Communities, where the connectivity in the communities is significantly higher than outside the community. There are multiple forms of community detection. We will use a couple in this post, starting with Louvain."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using Louvain Community Detection\n",
    "\n",
    "[Louvain](https://neo4j.com/docs/graph-data-science/current/algorithms/louvain/) is useful for exploratory analysis of communities because it uses a form of modularity scoring to split up the graph into hierarchical clusters. This means that your theories around fraud patterns and graph structure don't need to be exact for it to provide informative communities and insights.\n",
    "\n",
    "I will leverage P2P transactions, cards, and devices for Louvain. We will leave out IP addresses for now since they have some super node issues (we will leverage IP addresses in subsequent parts ). Below are the queries for running Louvain in Neo4j GDS and aggregating community statistics. The last query below orders communities by the count of flagged users so we can further examine some of the more concentrated flagged communities.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "writeMillis                                                                                                                                                                                                                                                                                                                                                                    411\n",
       "nodePropertiesWritten                                                                                                                                                                                                                                                                                                                                                       204001\n",
       "modularity                                                                                                                                                                                                                                                                                                                                                                0.968496\n",
       "modularities                                                                                                                                                                                                                                                                                                          [0.8468774527867388, 0.9542424101443094, 0.9684960238545887]\n",
       "ranLevels                                                                                                                                                                                                                                                                                                                                                                        3\n",
       "communityCount                                                                                                                                                                                                                                                                                                                                                               13636\n",
       "communityDistribution                                                                                                                                                                                                                                      {'p99': 89, 'min': 1, 'max': 5238, 'mean': 14.960472279260781, 'p90': 26, 'p50': 10, 'p999': 353, 'p95': 36, 'p75': 16}\n",
       "postProcessingMillis                                                                                                                                                                                                                                                                                                                                                            18\n",
       "preProcessingMillis                                                                                                                                                                                                                                                                                                                                                              0\n",
       "computeMillis                                                                                                                                                                                                                                                                                                                                                                 1347\n",
       "configuration            {'maxIterations': 10, 'writeConcurrency': 4, 'seedProperty': None, 'consecutiveIds': False, 'maxLevels': 10, 'relationshipWeightProperty': None, 'concurrency': 4, 'writeProperty': 'louvainCommunityId', 'includeIntermediateCommunities': False, 'nodeLabels': ['*'], 'sudo': False, 'relationshipTypes': ['*'], 'tolerance': 0.0001, 'username': None}\n",
       "Name: 0, dtype: object"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# clear the graph if it exists beforehand\n",
    "clear_graph_by_name('comm-projection')\n",
    "\n",
    "g, _ = gds.graph.project('comm-projection', ['User','Card', 'Device'], {\n",
    "    'HAS_CC': {'orientation': 'UNDIRECTED'},\n",
    "    'USED': {'orientation': 'UNDIRECTED'},\n",
    "    'P2P': {'orientation': 'NATURAL', 'aggregation': 'SINGLE'}\n",
    "})\n",
    "\n",
    "df = gds.louvain.write(g, writeProperty='louvainCommunityId')\n",
    "g.drop()\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Louvain Communities Ordered by count of Flagged Users\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>community</th>\n",
       "      <th>cnt</th>\n",
       "      <th>flaggedCount</th>\n",
       "      <th>flaggedRatio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>89885</td>\n",
       "      <td>27</td>\n",
       "      <td>5</td>\n",
       "      <td>0.185185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>182733</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>0.571429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>179565</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>188313</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>201222</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>0.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>170654</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>172960</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>94447</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>0.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>194173</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>183139</td>\n",
       "      <td>21</td>\n",
       "      <td>1</td>\n",
       "      <td>0.047619</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    community  cnt  flaggedCount  flaggedRatio\n",
       "0       89885   27             5      0.185185\n",
       "1      182733    7             4      0.571429\n",
       "2      179565    3             3      1.000000\n",
       "3      188313    6             3      0.500000\n",
       "4      201222    7             3      0.428571\n",
       "..        ...  ...           ...           ...\n",
       "95     170654    2             1      0.500000\n",
       "96     172960    2             1      0.500000\n",
       "97      94447    5             1      0.200000\n",
       "98     194173    3             1      0.333333\n",
       "99     183139   21             1      0.047619\n",
       "\n",
       "[100 rows x 4 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Louvain Communities Ordered by count of Flagged Users\")\n",
    "gds.run_cypher('''\n",
    "    MATCH (u:User)\n",
    "    WITH u.louvainCommunityId AS community,\n",
    "        count(u) AS cnt,\n",
    "        sum(u.fraudMoneyTransfer) as flaggedCount\n",
    "    RETURN community,\n",
    "        cnt,\n",
    "        flaggedCount,\n",
    "        toFloat(flaggedCount)/toFloat(cnt) AS flaggedRatio\n",
    "    ORDER BY flaggedCount DESC LIMIT 100\n",
    "''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can view communities of users and connecting identifiers in [Neo4j Bloom](https://neo4j.com/docs/bloom-user-guide/current/) or Neo4j Browser with queries of the form:\n",
    "\n",
    "```cypher\n",
    "MATCH(u1:User{louvainCommunityId: $id})-[r1:HAS_CC|HAS_IP|USED]->(n)<-[r2:HAS_CC|HAS_IP|USED]-(u2:User{louvainCommunityId: $id})\n",
    "WITH *\n",
    "OPTIONAL MATCH (u1)-[:P2P]-(u2)\n",
    "RETURN *\n",
    "```\n",
    "Below are a couple examples of these communities. Flagged users are colored red with caption=1 with other users colored orange and caption=0. We will see an interesting pattern of flagged users sending money to non-flagged users with which they share devices, cards, and IPs. This behavior is suspicious because it suggests that the same individual or group of individuals may be using multiple accounts to withdraw cash from credit transactions prior to a chargeback being executed on the cards. The shared credit card examples (such as example 2 below) are some of the most indicative.\n",
    "\n",
    "Example 1:\n",
    "<img src=\"img/louvain-community-11.png\" width=\"900\" >\n",
    "\n",
    "Example 2:\n",
    "<img src=\"img/louvain-community-12.png\" width=\"900\" >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This pattern indicates that the receiving user accounts are fraud risk accounts. In the next section we will use these patterns to inform resolution of fraud communities and the labeling of additional fraud risk accounts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Resolving Fraud Communities using Entity Resolution and Community Detection\n",
    "\n",
    "Identifying communities that reflect underlying groups of individuals is often a key step to fraud detection.  We already explored with Louvain; in this section, in this part, we will provide more formal definitions for resolving entities that will allow us to partition well defined communities in a scalable manner.\n",
    "\n",
    "To accomplish this, we will define some Entity Resolution (ER) rules that will allow us to draw relationships between users which we believe belong to the same underlying community. We will then use the [Weakly Connected Components (WCC)](https://neo4j.com/docs/graph-data-science/current/algorithms/wcc/) algorithm to resolve the communities. Lastly, we will label all users in communities that include flagged accounts as fraud risks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Entity Resolution Business Rules\n",
    "\n",
    "We will now use Entity Resolution (ER) to resolve groups of individuals behind sets of user accounts. For this analysis, we will use some pretty straightforward ER business logic. If either of the two below conditions are true, we will resolve two user accounts by linking them together with a new relationship type.\n",
    "\n",
    "1. One user sent money to another user that shares the same credit card\n",
    "2. Two users share a card or device connected to less than or equal to 10 total accounts, and those two users also share at least two other identifiers of type credit card, device, or IP address\n",
    "\n",
    "You could switch out or add different rules to the above, these are just examples. In a real-world scenario these business rules would pass by SMEs and possibly be backed by further supervised machine learning on manually labeled data. More advanced techniques for this type of ER are possible in graph and we describe them in [this whitepaper](https://neo4j.com/whitepapers/graph-data-science-use-cases-entity-resolution/) and [this blog](https://neo4j.com/developer-blog/exploring-supervised-entity-resolution-in-neo4j/).\n",
    "\n",
    "For a P2P dataset, we do not necessarily want to label all senders/receivers of flagged user transactions as fraudulent since some fraud schemes involve transactions with victims. Furthermore, additional identifiers such as IP may be inexact and cards + devices can be fraudulently controlled/used without the owners permission. Hence I used somewhat stringent rules that aligned with the patterns noted in part 1. We can apply relationships to reflect these business rules using cypher:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cnt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6240</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    cnt\n",
       "0  6240"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# P2P with shared card rule\n",
    "gds.run_cypher('''\n",
    "    MATCH (u1:User)-[r:P2P]->(u2)\n",
    "    WITH u1, u2, count(r) AS cnt\n",
    "    MATCH (u1)-[:HAS_CC]->(n)<-[:HAS_CC]-(u2)\n",
    "    WITH u1, u2, count(DISTINCT n) AS cnt\n",
    "    MERGE(u1)-[s:P2P_WITH_SHARED_CARD]->(u2)\n",
    "    RETURN count(DISTINCT s) AS cnt\n",
    "''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count(DISTINCT s)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5316</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   count(DISTINCT s)\n",
       "0               5316"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Shared ids rule\n",
    "gds.run_cypher('''\n",
    "    MATCH (u1:User)-[:HAS_CC|USED]->(n)<-[:HAS_CC|USED]-(u2)\n",
    "    WHERE n.degree <= 10 AND id(u1) < id(u2)\n",
    "    WITH u1, u2, count(DISTINCT n) as cnt\n",
    "    MATCH (u1)-[:HAS_CC|USED|HAS_IP]->(m)<-[:HAS_CC|USED|HAS_IP]-(u2)\n",
    "    WITH u1, u2, count(DISTINCT m) as cnt\n",
    "    WHERE cnt > 2\n",
    "    MERGE(u1)-[s:SHARED_IDS]->(u2)\n",
    "    RETURN count(DISTINCT s)\n",
    "''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Weakly Connected Components (WCC) to Resolve Communities\n",
    "\n",
    "[Weakly Connected Components (WCC)](https://neo4j.com/docs/graph-data-science/current/algorithms/wcc/) is a practical and highly scalable community detection algorithm. It is also deterministic and very explainable. It defines a community simply as a set of nodes connected by a subset of relationship types in the graph. This makes WCC a good choice for formal community assignment in production fraud detection settings.\n",
    "\n",
    "Below we run WCC on users via the ER relationships created above:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "writeMillis                                                                                                                                                                                                                                                                      79\n",
       "nodePropertiesWritten                                                                                                                                                                                                                                                         33732\n",
       "componentCount                                                                                                                                                                                                                                                                28203\n",
       "componentDistribution                                                                                                                                                {'p99': 3, 'min': 1, 'max': 175, 'mean': 1.196042974151686, 'p90': 2, 'p50': 1, 'p999': 8, 'p95': 2, 'p75': 1}\n",
       "postProcessingMillis                                                                                                                                                                                                                                                             13\n",
       "preProcessingMillis                                                                                                                                                                                                                                                               0\n",
       "computeMillis                                                                                                                                                                                                                                                                    11\n",
       "configuration            {'writeConcurrency': 4, 'seedProperty': None, 'consecutiveIds': False, 'writeProperty': 'wccId', 'threshold': 0.0, 'relationshipWeightProperty': None, 'nodeLabels': ['*'], 'sudo': False, 'relationshipTypes': ['*'], 'username': None, 'concurrency': 4}\n",
       "Name: 0, dtype: object"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# clear the graph if it exists beforehand\n",
    "clear_graph_by_name('comm-projection')\n",
    "\n",
    "g, _ = gds.graph.project('comm-projection', ['User'], {\n",
    "    'SHARED_IDS': {'orientation': 'UNDIRECTED'},\n",
    "    'P2P_WITH_SHARED_CARD': {'orientation': 'UNDIRECTED'}\n",
    "})\n",
    "\n",
    "df = gds.wcc.write(g, writeProperty='wccId')\n",
    "g.drop()\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Labeling Fraud Risk User Accounts\n",
    "\n",
    "As these communities are meant to label underlying groups of individuals, if even one flagged account is in the community, we will label all user accounts in the group as fraud risks:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count(u)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>452</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   count(u)\n",
       "0       452"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gds.run_cypher('''\n",
    "    MATCH (f:FlaggedUser)\n",
    "    WITH collect(DISTINCT f.wccId) AS flaggedCommunities\n",
    "    MATCH(u:User) WHERE u.wccId IN flaggedCommunities\n",
    "    SET u:FraudRiskUser\n",
    "    SET u.fraudRisk=1\n",
    "    RETURN count(u)\n",
    "''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count(u)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>33280</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   count(u)\n",
       "0     33280"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gds.run_cypher('''\n",
    "    MATCH (u:User) WHERE NOT u:FraudRiskUser\n",
    "    SET u.fraudRisk=0\n",
    "    RETURN count(u)\n",
    "''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### WCC Community Statistics\n",
    "\n",
    "The breakdown of communities by size is listed below. The majority are single user communities. Only a small portion have multiple users and of those, community sizes are mostly 2 and 3. Larger communities are rare. However, if we look at the fraudUser accounts we will see that the majority reside in multi-user communities. The 118 fraud accounts in single user communities are flagged users (via original chargeback logic) that have yet to be resolved to a community."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>componentSize</th>\n",
       "      <th>numberOfComponents</th>\n",
       "      <th>totalUserCount</th>\n",
       "      <th>fraudUserCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>24283</td>\n",
       "      <td>24283</td>\n",
       "      <td>118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>3165</td>\n",
       "      <td>6330</td>\n",
       "      <td>144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>481</td>\n",
       "      <td>1443</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4-10</td>\n",
       "      <td>258</td>\n",
       "      <td>1246</td>\n",
       "      <td>68</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11-50</td>\n",
       "      <td>15</td>\n",
       "      <td>255</td>\n",
       "      <td>59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>&gt;50</td>\n",
       "      <td>1</td>\n",
       "      <td>175</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  componentSize  numberOfComponents  totalUserCount  fraudUserCount\n",
       "0             1               24283           24283             118\n",
       "1             2                3165            6330             144\n",
       "2             3                 481            1443              63\n",
       "3          4-10                 258            1246              68\n",
       "4         11-50                  15             255              59\n",
       "5           >50                   1             175               0"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gds.run_cypher( '''\n",
    "    MATCH (u:User)\n",
    "    WITH u.wccId AS community, count(u) AS cSize, sum(u.fraudRisk) AS cFraudSize\n",
    "    WITH community, cSize, cFraudSize,\n",
    "    CASE\n",
    "        WHEN cSize=1 THEN ' 1'\n",
    "        WHEN cSize=2 THEN ' 2'\n",
    "        WHEN cSize=3 THEN ' 3'\n",
    "        WHEN cSize>3 AND cSize<=10 THEN ' 4-10'\n",
    "        WHEN cSize>10 AND cSize<=50 THEN '11-50'\n",
    "        WHEN cSize>10 THEN '>50' END AS componentSize\n",
    "    RETURN componentSize, \n",
    "        count(*) AS numberOfComponents, \n",
    "        sum(cSize) AS totalUserCount, \n",
    "        sum(cFraudSize) AS fraudUserCount \n",
    "    ORDER BY componentSize\n",
    "''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to Louvain we can view communities of users and connecting identifiers in [Neo4j Bloom](https://neo4j.com/docs/bloom-user-guide/current/) or Browser with queries of the form:\n",
    "\n",
    "```cypher\n",
    "MATCH(u1:User{wccId: $id})-[r1:HAS_CC|HAS_IP|USED]->(n)<-[r2:HAS_CC|HAS_IP|USED]-(u2:User{wccId: $id})\n",
    "WITH *\n",
    "OPTIONAL MATCH (u1)-[r3:P2P]-(u2)\n",
    "RETURN *\n",
    "```\n",
    "Below are a couple examples. Users that were flagged via initial chargeback logic are colored red with caption=1, while other users are colored orange with caption=0. Overall, you will notice a high degree of overlapping connectivity of identifiers and P2P transactions between users, which we should expect given our ER rules.\n",
    "\n",
    "Example 1:\n",
    "<img src=\"img/wcc-community-12.png\" width=\"900\" >\n",
    "\n",
    "Example 2:\n",
    "<img src=\"img/wcc-community-11.png\" width=\"900\" >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outcomes of Fraud Risk Labeling\n",
    "Fraud Risk labeling helped identify an additional 211 new fraud risk user accounts, nearly doubling the number of known fraud users (87.5% increase). We also see that 65% of the money going to/from previously flagged accounts and other users can be attributed to the newly identified risk accounts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.652"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gds.run_cypher('''\n",
    "   MATCH (:FlaggedUser)-[r:P2P]-(u)  WHERE NOT u:FlaggedUser\n",
    "   WITH toFloat(sum(r.totalAmount)) AS p2pTotal\n",
    "   MATCH (u:FraudRiskUser)-[r:P2P]-(:FlaggedUser) WHERE NOT u:FlaggedUser\n",
    "   WITH p2pTotal,  toFloat(sum(r.totalAmount)) AS fraudRiskP2pTotal\n",
    "   RETURN round((fraudRiskP2pTotal)/p2pTotal,3) AS p\n",
    "''').p[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, while the newly identified 211 accounts represents less than 1% of total users in the sample, 12.7% of the total P2P amount in the sample involved the newly identified accounts as senders or receivers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.127"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gds.run_cypher('''\n",
    "   MATCH (:User)-[r:P2P]->()\n",
    "   WITH toFloat(sum(r.totalAmount)) AS p2pTotal\n",
    "   MATCH (u:FraudRiskUser)-[r:P2P]-() WHERE NOT u:FlaggedUser\n",
    "   WITH p2pTotal, toFloat(sum(r.totalAmount)) AS fraudRiskP2pTotal\n",
    "   RETURN round((fraudRiskP2pTotal)/p2pTotal,3) AS p\n",
    "''').p[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can see an improvement in card and device discrimination with many more cards and devices being used by fraud risk accounts exclusively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "identifier_degrees('FraudRiskUser', 'fraudRiskDegree')\n",
    "\n",
    "gds.run_cypher('''\n",
    "    MATCH(n) WHERE n:Card OR n:Device OR n:IP\n",
    "    SET n.fraudRiskRatio = toFloat(n.fraudRiskDegree)/toFloat(n.degree)\n",
    "''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fraudRiskRatio</th>\n",
       "      <th>count</th>\n",
       "      <th>percentCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>8114</td>\n",
       "      <td>0.956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>351</td>\n",
       "      <td>0.041</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Between 0-1</td>\n",
       "      <td>25</td>\n",
       "      <td>0.003</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  fraudRiskRatio  count  percentCount\n",
       "0              0   8114         0.956\n",
       "1              1    351         0.041\n",
       "2    Between 0-1     25         0.003"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gds.run_cypher('''\n",
    "    MATCH(n:Card) WHERE n.degree > 1\n",
    "    WITH toFloat(count(n)) AS total\n",
    "    MATCH(n:Card) WHERE n.degree > 1\n",
    "    WITH n, total, CASE\n",
    "        WHEN n.fraudRiskRatio=0 THEN '0'\n",
    "        WHEN n.fraudRiskRatio=1  THEN '1'\n",
    "        ELSE 'Between 0-1' END AS fraudRiskRatio\n",
    "    RETURN fraudRiskRatio, count(n) AS count, round(toFloat(count(n))/total,3) AS percentCount \n",
    "    ORDER BY fraudRiskRatio\n",
    "''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fraudRiskRatio</th>\n",
       "      <th>count</th>\n",
       "      <th>percentCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>2605</td>\n",
       "      <td>0.949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>92</td>\n",
       "      <td>0.034</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Between 0-1</td>\n",
       "      <td>48</td>\n",
       "      <td>0.017</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  fraudRiskRatio  count  percentCount\n",
       "0              0   2605         0.949\n",
       "1              1     92         0.034\n",
       "2    Between 0-1     48         0.017"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gds.run_cypher('''\n",
    "    MATCH(n:Device) WHERE n.degree > 1\n",
    "    WITH toFloat(count(n)) AS total\n",
    "    MATCH(n:Device) WHERE n.degree > 1\n",
    "    WITH n, total, CASE\n",
    "        WHEN n.fraudRiskRatio=0 THEN '0'\n",
    "        WHEN n.fraudRiskRatio=1  THEN '1'\n",
    "        ELSE 'Between 0-1' END AS fraudRiskRatio\n",
    "    RETURN fraudRiskRatio, count(n) AS count, round(toFloat(count(n))/total,3) AS percentCount \n",
    "    ORDER BY fraudRiskRatio\n",
    "''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The aggregate P2P statistics combined with improvements in Card and Device metrics are significant given the limited scope of the previously flagged fraud which focused on chargebacks.  These results strongly imply that there are more sophisticated networks of fraudulent money flows behind the chargebacks rather than the chargebacks being isolated occurrences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Recommending Suspicious Accounts With Centrality & Node Similarity\n",
    "\n",
    "In parts 1 & 2 we explored the graph and identified high risk fraud communities. At this stage, we may want to expand beyond our business logic to automatically identify other users that are suspiciously similar to the fraud risks already identified. Neo4j and GDS makes it simple to triage and recommend such suspect users in a matter of seconds. We can leverage both centrality and similarity algorithms for this.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Weighted Degree Centrality to Recommend Potential High Risk Accounts\n",
    "\n",
    "We can quickly and easily generate a ranked list of suspicious user accounts with weighted degree centrality. Specifically, we can calculate the degree centrality of users in respect to their identifiers (Devices, Cards, and IPs) weighted by the fraudRiskRatios we made in part 2. In this case, a simple Cypher query suffices.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uid</th>\n",
       "      <th>totalIdFraudRisk</th>\n",
       "      <th>numberFraudRiskIds</th>\n",
       "      <th>averageFraudIdRisk</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>50fd11c0</td>\n",
       "      <td>7.405512</td>\n",
       "      <td>28</td>\n",
       "      <td>0.264483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b41b9e3b</td>\n",
       "      <td>6.487978</td>\n",
       "      <td>22</td>\n",
       "      <td>0.294908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>a982e8fc</td>\n",
       "      <td>3.928543</td>\n",
       "      <td>13</td>\n",
       "      <td>0.302196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60b71ab9</td>\n",
       "      <td>2.020481</td>\n",
       "      <td>7</td>\n",
       "      <td>0.288640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>bc1eb835</td>\n",
       "      <td>2.010309</td>\n",
       "      <td>5</td>\n",
       "      <td>0.402062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1b1a78a9</td>\n",
       "      <td>1.925455</td>\n",
       "      <td>5</td>\n",
       "      <td>0.385091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>f2b8a068</td>\n",
       "      <td>1.693794</td>\n",
       "      <td>6</td>\n",
       "      <td>0.282299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1c773976</td>\n",
       "      <td>1.594978</td>\n",
       "      <td>5</td>\n",
       "      <td>0.318996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>a2f748e3</td>\n",
       "      <td>1.570455</td>\n",
       "      <td>4</td>\n",
       "      <td>0.392614</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>7524a090</td>\n",
       "      <td>1.527338</td>\n",
       "      <td>5</td>\n",
       "      <td>0.305468</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        uid  totalIdFraudRisk  numberFraudRiskIds  averageFraudIdRisk\n",
       "0  50fd11c0          7.405512                  28            0.264483\n",
       "1  b41b9e3b          6.487978                  22            0.294908\n",
       "2  a982e8fc          3.928543                  13            0.302196\n",
       "3  60b71ab9          2.020481                   7            0.288640\n",
       "4  bc1eb835          2.010309                   5            0.402062\n",
       "5  1b1a78a9          1.925455                   5            0.385091\n",
       "6  f2b8a068          1.693794                   6            0.282299\n",
       "7  1c773976          1.594978                   5            0.318996\n",
       "8  a2f748e3          1.570455                   4            0.392614\n",
       "9  7524a090          1.527338                   5            0.305468"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gds.run_cypher('''\n",
    "    MATCH(f:FraudRiskUser)-[:HAS_CC|HAS_IP|USED]->(n)\n",
    "    WITH DISTINCT n\n",
    "    MATCH(u:User)-[:HAS_CC|HAS_IP|USED]->(n) WHERE NOT u:FraudRiskUser\n",
    "    WITH left(u.guid,8) as uid,\n",
    "        sum(n.fraudRiskRatio) AS totalIdFraudRisk,\n",
    "        count(n) AS numberFraudRiskIds\n",
    "    WITH uid, totalIdFraudRisk,\n",
    "        numberFraudRiskIds,\n",
    "        totalIdFraudRisk/toFloat(numberFraudRiskIds) AS averageFraudIdRisk\n",
    "    WHERE averageFraudIdRisk >= 0.25\n",
    "    RETURN uid, totalIdFraudRisk, numberFraudRiskIds, averageFraudIdRisk\n",
    "    ORDER BY totalIdFraudRisk DESC LIMIT 10\n",
    "''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Users in the above result list are sorted by how much identifying information they share with previously labeled fraud risks, the ones with the most being at the top. Technically speaking, these users are ranked by their total Id fraud risk, which is equal to the sum of the fraudRiskRatios from the Identifiers they are connected to. In the query I also implement a limit on the average fraud risk to avoid users that just have a lot of high-degree identifiers (likely proxy ip addresses shared by only a small fraction of fraud risk users). This sort of filtering can be tweaked by use case to get the right balance between total risk vs average risk.\n",
    "\n",
    "In a real-world fraud detection use case, these results can be triaged by analysts to label more fraud accounts and grow labeled fraud communities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Node Similarity to Expand on Fraud Communities\n",
    "\n",
    "Simple calculations like weighted degree centrality work well for identifying suspicious users over the whole graph, but what if we are interested in how users are related to a specific fraud risk community or set of communities? Perhaps we hypothesize that communities of fraud risk users are actually bigger than currently represented but we don't have exact business rules to apply. We can leverage similarity algorithms to help us score and recommend users for this.\n",
    "\n",
    "GDS offers multiple algorithms for similarity. In this analysis I will focus on the aptly named  Node Similarity algorithm – you can read more about it [here](https://neo4j.com/docs/graph-data-science/current/algorithms/node-similarity/). Node similarity parallelizes well and is explainable. It identifies pairs of similar nodes based on a straightforward Jaccard similarity calculation. So while other ML-based similarity approaches like FastRP + KNN covered [in this post](https://neo4j.com/developer-blog/exploring-practical-recommendation-systems-in-neo4j/) scale well for running globally on very large graphs, Node Similarity is a good choice where explainability is important and you can narrow down the universe of comparisons to a subset of your data. Examples of narrowing down include focusing on just single communities, newly added users, or users within a specific proximity to suspect accounts. In this analysis we will take the third approach, filtering to just those Cards, Devices, and IP addresses that connect to at least one fraud risk account from part 2.\n",
    "\n",
    "\n",
    "Below I apply three queries to calculate node similarity. The first query enables the identifier filtering via setting a new label on the Card, Device, and IP address nodes that connect to fraud risk accounts. The first query also weights relationships by the inverse of degree centrality, essentially downplaying the importance of identifiers proportional to the number of other users they connect to. This is important as some identifiers, particularly IP addresses, can connect to hundreds or thousands of users, in which case the identifier may be very generic (like a proxy IP address) and not as relevant to true user identity. The second and third query project the graph and write relationships back to the database with a score to represent similarity strength between user node pairs. You will notice that I use a similarity cutoff of 0.01 in the third query, which is intended to rule out weak associations and keep the similarities relevant.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count(DISTINCT n)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18182</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   count(DISTINCT n)\n",
       "0              18182"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# label identifiers and users that are close to fraud risk users and assign inverse degree weight\n",
    "gds.run_cypher('''\n",
    "    MATCH(f:FraudRiskUser)-[:HAS_CC|HAS_IP|USED]->(n)\n",
    "    WITH DISTINCT n\n",
    "    MATCH(n)<-[r:HAS_CC|HAS_IP|USED]-(u)\n",
    "    SET n:FraudSharedId\n",
    "    SET r.inverseDegreeWeight = 1.0/(n.degree-1.0)\n",
    "    RETURN count(DISTINCT n)\n",
    "''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "preProcessingMillis                                                                                                                                                                                                                                                                                                                                                                                                           0\n",
       "computeMillis                                                                                                                                                                                                                                                                                                                                                                                                             71320\n",
       "writeMillis                                                                                                                                                                                                                                                                                                                                                                                                                6398\n",
       "postProcessingMillis                                                                                                                                                                                                                                                                                                                                                                                                         -1\n",
       "nodesCompared                                                                                                                                                                                                                                                                                                                                                                                                             32188\n",
       "relationshipsWritten                                                                                                                                                                                                                                                                                                                                                                                                     314504\n",
       "similarityDistribution    {'p1': 0.02386075258255005, 'max': 1.0000075697898865, 'p5': 0.06616824865341187, 'p90': 0.5131835341453552, 'p50': 0.18831056356430054, 'p95': 0.8225058913230896, 'p10': 0.08741754293441772, 'p75': 0.3100394606590271, 'p99': 1.0000075697898865, 'p25': 0.12364381551742554, 'p100': 1.0000075697898865, 'min': 0.010002970695495605, 'mean': 0.2599237197617545, 'stdDev': 0.21765269370986462}\n",
       "configuration                                {'topK': 10, 'writeConcurrency': 4, 'similarityMetric': 'JACCARD', 'bottomK': 10, 'bottomN': 0, 'relationshipWeightProperty': 'inverseDegreeWeight', 'topN': 0, 'concurrency': 4, 'writeProperty': 'score', 'degreeCutoff': 1, 'writeRelationshipType': 'SIMILAR_IDS', 'similarityCutoff': 0.01, 'nodeLabels': ['*'], 'sudo': False, 'relationshipTypes': ['*'], 'username': None}\n",
       "Name: 0, dtype: object"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# clear the graph if it exists beforehand\n",
    "clear_graph_by_name('similarity-projection')\n",
    "\n",
    "g, _ = gds.graph.project('similarity-projection', ['User', 'FraudSharedId'], ['HAS_CC', 'USED', 'HAS_IP'],\n",
    "                         relationshipProperties=['inverseDegreeWeight'])\n",
    "\n",
    "\n",
    "\n",
    "df = gds.nodeSimilarity.write(g, writeRelationshipType='SIMILAR_IDS', writeProperty='score',\n",
    "                              similarityCutoff=0.01, relationshipWeightProperty='inverseDegreeWeight')\n",
    "g.drop()\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From there, we can run a Cypher query to rank users by how similar they are to known fraud risk communities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>totalScore</th>\n",
       "      <th>closeToCommunityIds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0b3f278ff6b348fb1a599479d9321cd9</td>\n",
       "      <td>0.427332</td>\n",
       "      <td>[4616, 4932]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>c0d80bb57c0bbb0b68f647c7adc83274</td>\n",
       "      <td>0.298319</td>\n",
       "      <td>[4932]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>99d857cf888723db4ab2c19e42ac2b4c</td>\n",
       "      <td>0.260633</td>\n",
       "      <td>[4616]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6b41e66c7667df06c10eab6c6b712fec</td>\n",
       "      <td>0.201825</td>\n",
       "      <td>[4932]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>55447b3000b48335d3b18557c43ba70a</td>\n",
       "      <td>0.136346</td>\n",
       "      <td>[4616]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>8c7fb38c22da0e7ad8d91a8551f62e4c</td>\n",
       "      <td>0.136169</td>\n",
       "      <td>[433]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>f0ec0c4c8cf74ebdf1ace4c366e58873</td>\n",
       "      <td>0.111970</td>\n",
       "      <td>[4616]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>f3c3fa16c8d095c257452f3287991cef</td>\n",
       "      <td>0.082760</td>\n",
       "      <td>[4616]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>48ab157818d5d6e31d4808e624a93846</td>\n",
       "      <td>0.077805</td>\n",
       "      <td>[4616]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>a2f748e3e52915b9aff50f9af628a9ee</td>\n",
       "      <td>0.075465</td>\n",
       "      <td>[433]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>6ab1e28cd734867b6533ecb88e076a63</td>\n",
       "      <td>0.068820</td>\n",
       "      <td>[4932]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>60fb2a28448dd7c445c95d0bd753ed6a</td>\n",
       "      <td>0.067419</td>\n",
       "      <td>[4932]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>9eb2b34590af0895010065f9442c02b6</td>\n",
       "      <td>0.066657</td>\n",
       "      <td>[4932]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>f76d815c0754d255b6f350c8304f83ea</td>\n",
       "      <td>0.066657</td>\n",
       "      <td>[4932]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>ac066690cdd0b366221fc994bf8e447f</td>\n",
       "      <td>0.066657</td>\n",
       "      <td>[4932]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>50248d7d3a3cfeec8b4cf7fb692f6061</td>\n",
       "      <td>0.066442</td>\n",
       "      <td>[4932]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>798e715bb86c758e7d524e06459ee607</td>\n",
       "      <td>0.066148</td>\n",
       "      <td>[4932]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>78d14471b7158be6c5bfa64afd11c317</td>\n",
       "      <td>0.062817</td>\n",
       "      <td>[433]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>c56baa4531f583130f2f70863436a0d6</td>\n",
       "      <td>0.062650</td>\n",
       "      <td>[433]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>576dbfad2ec4237bf2b702eeb9264374</td>\n",
       "      <td>0.062368</td>\n",
       "      <td>[433]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0430d475a583353d935fc836810234c9</td>\n",
       "      <td>0.060690</td>\n",
       "      <td>[433]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>af6d286829353ec5d9caa0d51dd6f71c</td>\n",
       "      <td>0.058621</td>\n",
       "      <td>[433]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>99549170b6977f2a74e092e188d650b7</td>\n",
       "      <td>0.057703</td>\n",
       "      <td>[433]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>633c3879567088e3f241603a2fc5dcf4</td>\n",
       "      <td>0.057160</td>\n",
       "      <td>[433]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>73331fc3ac27ce4e60190b19311c417a</td>\n",
       "      <td>0.056775</td>\n",
       "      <td>[4616]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>1f070c1dddb70c0859c2befc1865b02e</td>\n",
       "      <td>0.056002</td>\n",
       "      <td>[4616]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>10fdc8365059f18487887bbe0ecedc73</td>\n",
       "      <td>0.055848</td>\n",
       "      <td>[433]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>9d428858704b6a1f4c618146ae4176da</td>\n",
       "      <td>0.053725</td>\n",
       "      <td>[4616]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>24431095fc157f1da870a8d44dbd63f8</td>\n",
       "      <td>0.053603</td>\n",
       "      <td>[433]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>d4809c7afb2413f5e245bea5bfb086c6</td>\n",
       "      <td>0.048996</td>\n",
       "      <td>[4616]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>ce506e3eee79867033640712679b332a</td>\n",
       "      <td>0.046605</td>\n",
       "      <td>[433]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>ccbc680c06f973d90153af0822758630</td>\n",
       "      <td>0.038643</td>\n",
       "      <td>[1189]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>d63ef30f4731cc7c261a0e0ee003d06e</td>\n",
       "      <td>0.033884</td>\n",
       "      <td>[1189]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>e44c064d39dbf0060887f63d75a82d7f</td>\n",
       "      <td>0.032981</td>\n",
       "      <td>[1189]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>b894d789371e310344ade21b889e2a78</td>\n",
       "      <td>0.026043</td>\n",
       "      <td>[1189]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>b5874f163dba44abaef09b5687c3ff74</td>\n",
       "      <td>0.025851</td>\n",
       "      <td>[1189]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>bdf90e92a3d13ba1124f1cf6d9b85d3f</td>\n",
       "      <td>0.017435</td>\n",
       "      <td>[1189]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              userId  totalScore closeToCommunityIds\n",
       "0   0b3f278ff6b348fb1a599479d9321cd9    0.427332        [4616, 4932]\n",
       "1   c0d80bb57c0bbb0b68f647c7adc83274    0.298319              [4932]\n",
       "2   99d857cf888723db4ab2c19e42ac2b4c    0.260633              [4616]\n",
       "3   6b41e66c7667df06c10eab6c6b712fec    0.201825              [4932]\n",
       "4   55447b3000b48335d3b18557c43ba70a    0.136346              [4616]\n",
       "5   8c7fb38c22da0e7ad8d91a8551f62e4c    0.136169               [433]\n",
       "6   f0ec0c4c8cf74ebdf1ace4c366e58873    0.111970              [4616]\n",
       "7   f3c3fa16c8d095c257452f3287991cef    0.082760              [4616]\n",
       "8   48ab157818d5d6e31d4808e624a93846    0.077805              [4616]\n",
       "9   a2f748e3e52915b9aff50f9af628a9ee    0.075465               [433]\n",
       "10  6ab1e28cd734867b6533ecb88e076a63    0.068820              [4932]\n",
       "11  60fb2a28448dd7c445c95d0bd753ed6a    0.067419              [4932]\n",
       "12  9eb2b34590af0895010065f9442c02b6    0.066657              [4932]\n",
       "13  f76d815c0754d255b6f350c8304f83ea    0.066657              [4932]\n",
       "14  ac066690cdd0b366221fc994bf8e447f    0.066657              [4932]\n",
       "15  50248d7d3a3cfeec8b4cf7fb692f6061    0.066442              [4932]\n",
       "16  798e715bb86c758e7d524e06459ee607    0.066148              [4932]\n",
       "17  78d14471b7158be6c5bfa64afd11c317    0.062817               [433]\n",
       "18  c56baa4531f583130f2f70863436a0d6    0.062650               [433]\n",
       "19  576dbfad2ec4237bf2b702eeb9264374    0.062368               [433]\n",
       "20  0430d475a583353d935fc836810234c9    0.060690               [433]\n",
       "21  af6d286829353ec5d9caa0d51dd6f71c    0.058621               [433]\n",
       "22  99549170b6977f2a74e092e188d650b7    0.057703               [433]\n",
       "23  633c3879567088e3f241603a2fc5dcf4    0.057160               [433]\n",
       "24  73331fc3ac27ce4e60190b19311c417a    0.056775              [4616]\n",
       "25  1f070c1dddb70c0859c2befc1865b02e    0.056002              [4616]\n",
       "26  10fdc8365059f18487887bbe0ecedc73    0.055848               [433]\n",
       "27  9d428858704b6a1f4c618146ae4176da    0.053725              [4616]\n",
       "28  24431095fc157f1da870a8d44dbd63f8    0.053603               [433]\n",
       "29  d4809c7afb2413f5e245bea5bfb086c6    0.048996              [4616]\n",
       "30  ce506e3eee79867033640712679b332a    0.046605               [433]\n",
       "31  ccbc680c06f973d90153af0822758630    0.038643              [1189]\n",
       "32  d63ef30f4731cc7c261a0e0ee003d06e    0.033884              [1189]\n",
       "33  e44c064d39dbf0060887f63d75a82d7f    0.032981              [1189]\n",
       "34  b894d789371e310344ade21b889e2a78    0.026043              [1189]\n",
       "35  b5874f163dba44abaef09b5687c3ff74    0.025851              [1189]\n",
       "36  bdf90e92a3d13ba1124f1cf6d9b85d3f    0.017435              [1189]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#get nodes similar to the high risk ones\n",
    "gds.run_cypher('''\n",
    "    MATCH (f:FraudRiskUser)\n",
    "    WITH f.wccId AS componentId, count(*) AS numberOfUsers, collect(f) AS users\n",
    "    UNWIND users AS f\n",
    "    MATCH (f)-[s:SIMILAR_IDS]->(u:User) WHERE NOT u:FraudRiskUser AND numberOfUsers > 2\n",
    "    RETURN u.guid AS userId, sum(s.score) AS totalScore, collect(DISTINCT componentId) AS closeToCommunityIds \n",
    "    ORDER BY totalScore DESC\n",
    "''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s take a look at the first user in the list, user `0b3f278f`, in [Bloom](https://neo4j.com/docs/bloom-user-guide/current/) . This user account seems interesting in the sense that they connect to two different communities.\n",
    "\n",
    "<img src=\"img/node-similarity-1.png\" width=\"700\" >\n",
    "\n",
    "You can see how this user connects to the two fraud risk communities and how the similarity relationships were based on shared IP addresses. This user seems to act as a sort of bridge between the two communities, suggesting not only that the user is likely part of the fraud communities but also that the two communities may actually reflect one-in-the same."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Overall, centrality and similarity metrics like these are fast and easy to implement with Neo4j and GDS. They can help advance your Data Science approach by introducing automated and semi-supervised processes to assist in targeted triage and identification of suspicious user accounts based on previously labeled data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: Predicting Fraud Risk Accounts with Machine Learning\n",
    "\n",
    "In real-world scenarios we often don't know which user accounts are fraudulent ahead of time. There will be cases, like with this dataset, where some accounts get flagged due to business rules, for example chargeback history, or via user reporting mechanisms. However, as we saw in parts 1 & 2 above, those flags don't tell the whole story. The community detection and recommendation approaches we previously covered can go a very long way in helping us understand this story and label additional fraud risk users and patterns. That said, there are multiple reasons why we may want to add supervised Machine Learning to predict the additional fraud risks:\n",
    "\n",
    " - __Proactive Detection:__ We can train a model to identify fraudulent actors ahead of time (such as before additional chargebacks or system flags) and better identify new communities that aren't connected to older known fraud accounts.\n",
    " - __Measurable Performance:__ Supervised learning models produce clear performance metrics that enable us to evaluate and adjust as needed\n",
    " - __Automation:__ supervised Machine Learning automates the prediction of fraud risk accounts.\n",
    "\n",
    "In the below sections we will walk through how to engineer graph features for ML, export those features to python, then train and evaluate an ML model for fraud classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Engineering\n",
    "If we want a machine learning model to successfully classify fraud risk user accounts, we need to supply features that will be informative for the task. The below commands engineer graph features using GDS.  This includes features from [WCC](https://neo4j.com/docs/graph-data-science/current/algorithms/wcc/) community sizes, [pageRank](https://neo4j.com/docs/graph-data-science/current/algorithms/page-rank/), and [degree centrality](https://neo4j.com/docs/graph-data-science/current/algorithms/degree-centrality/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Community Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gds.run_cypher('''\n",
    "    MATCH (u:User)\n",
    "    WITH u.wccId AS componentId, count(*) AS communitySize, collect(u) AS users\n",
    "    WITH communitySize, toInteger(communitySize > 1) AS partOfCommunity, users\n",
    "    UNWIND users as u\n",
    "    SET u.communitySize = communitySize\n",
    "    SET u.partOfCommunity = partOfCommunity;\n",
    "''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### P2P With Cards and Id Sharing Centrality Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# clear the graph if it exists beforehand\n",
    "clear_graph_by_name('p2p-features')\n",
    "\n",
    "# Payment Centrality & Influence Features\n",
    "g, _ = gds.graph.project('p2p-features', ['User'], {\n",
    "    'P2P': {'type': 'P2P', 'orientation': 'NATURAL', 'aggregation': 'SUM', 'properties': ['totalAmount']},\n",
    "    'P2P_REVERSE': {'type': 'P2P', 'orientation': 'REVERSE', 'aggregation': 'SUM', 'properties': ['totalAmount']},\n",
    "    'SHARED_IDS': {'type': 'SHARED_IDS', 'orientation': 'UNDIRECTED'},\n",
    "    'P2P_WITH_SHARED_CARD': {'type': 'P2P_WITH_SHARED_CARD', 'orientation': 'NATURAL'}\n",
    "})\n",
    "gds.degree.write(g, relationshipTypes=['SHARED_IDS'], writeProperty='sharedIdsDegree')\n",
    "gds.pageRank.write(g, relationshipTypes=['P2P_WITH_SHARED_CARD'], maxIterations=1000,\n",
    "                   writeProperty='p2pSharedCardPageRank')\n",
    "gds.pageRank.write(g, relationshipTypes=['P2P'], maxIterations=1000, writeProperty='p2pSentPageRank')\n",
    "gds.pageRank.write(g, relationshipTypes=['P2P_REVERSE'], maxIterations=1000, relationshipWeightProperty='totalAmount',\n",
    "                   writeProperty='p2pReceivedWeightedPageRank')\n",
    "gds.degree.write(g, relationshipTypes=['P2P_REVERSE'], relationshipWeightProperty='totalAmount',\n",
    "                 writeProperty='p2pReceivedWeightedDegree')\n",
    "g.drop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ID Centrality Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# clear the graph if it exists beforehand\n",
    "clear_graph_by_name('p2p-features')\n",
    "\n",
    "# User Id Centrality Features\n",
    "g, _ = gds.graph.project('user-centrality-features', ['User', 'Card', 'Device', 'IP'], ['HAS_CC', 'HAS_IP', 'USED'])\n",
    "gds.degree.mutate(g, nodeLabels=['User', 'Card'], relationshipTypes=['HAS_CC'], mutateProperty='cardDegree')\n",
    "gds.degree.mutate(g, nodeLabels=['User', 'Device'], relationshipTypes=['USED'], mutateProperty='deviceDegree')\n",
    "gds.degree.mutate(g, nodeLabels=['User', 'IP'], relationshipTypes=['HAS_IP'], mutateProperty='ipDegree')\n",
    "gds.graph.writeNodeProperties(g, ['cardDegree', 'deviceDegree', 'ipDegree'], ['User'])\n",
    "g.drop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Machine Learning Training & Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get and Prepare Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>guid</th>\n",
       "      <th>wccId</th>\n",
       "      <th>fraudRisk</th>\n",
       "      <th>fraudMoneyTransfer</th>\n",
       "      <th>sharedIdsDegree</th>\n",
       "      <th>p2pSharedCardPageRank</th>\n",
       "      <th>p2pSentPageRank</th>\n",
       "      <th>p2pReceivedWeightedPageRank</th>\n",
       "      <th>p2pReceivedWeightedDegree</th>\n",
       "      <th>ipDegree</th>\n",
       "      <th>cardDegree</th>\n",
       "      <th>deviceDegree</th>\n",
       "      <th>communitySize</th>\n",
       "      <th>partOfCommunity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>be0222d26897046dc92ae2929d3427e7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.175456</td>\n",
       "      <td>0.414277</td>\n",
       "      <td>10.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>59515432f0e2e79b83e9524c94f7f86e</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.310548</td>\n",
       "      <td>0.150422</td>\n",
       "      <td>10.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ba1f4324c66ee093478d41a6826a3649</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.277500</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>160.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8a18e1625e35a9f66abfd7bd2d04f0b3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.230174</td>\n",
       "      <td>0.319779</td>\n",
       "      <td>100.0</td>\n",
       "      <td>135.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9f705a0107acc25a57f9e6aa8436a08e</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.391426</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>2703.6</td>\n",
       "      <td>169.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33727</th>\n",
       "      <td>f3693169dcc9064114a171c5c2f485b5</td>\n",
       "      <td>33727</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.297005</td>\n",
       "      <td>0.451011</td>\n",
       "      <td>5.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33728</th>\n",
       "      <td>10f7f048645c602de3fae7c2cf81b254</td>\n",
       "      <td>27913</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.999999</td>\n",
       "      <td>0.371382</td>\n",
       "      <td>1.632799</td>\n",
       "      <td>150.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33729</th>\n",
       "      <td>d3a4469957800636802a8de4874af1bc</td>\n",
       "      <td>33729</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.277500</td>\n",
       "      <td>0.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33730</th>\n",
       "      <td>4698ccb2e4b07b1974741f634115e49b</td>\n",
       "      <td>33730</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.181054</td>\n",
       "      <td>0.170451</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33731</th>\n",
       "      <td>8eea038c1a3408651f5980631768deaf</td>\n",
       "      <td>33731</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.185296</td>\n",
       "      <td>0.401513</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>33732 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   guid  wccId  fraudRisk  fraudMoneyTransfer  \\\n",
       "0      be0222d26897046dc92ae2929d3427e7      0          0                   0   \n",
       "1      59515432f0e2e79b83e9524c94f7f86e      1          0                   0   \n",
       "2      ba1f4324c66ee093478d41a6826a3649      2          0                   0   \n",
       "3      8a18e1625e35a9f66abfd7bd2d04f0b3      3          0                   0   \n",
       "4      9f705a0107acc25a57f9e6aa8436a08e      4          0                   0   \n",
       "...                                 ...    ...        ...                 ...   \n",
       "33727  f3693169dcc9064114a171c5c2f485b5  33727          0                   0   \n",
       "33728  10f7f048645c602de3fae7c2cf81b254  27913          0                   0   \n",
       "33729  d3a4469957800636802a8de4874af1bc  33729          0                   0   \n",
       "33730  4698ccb2e4b07b1974741f634115e49b  33730          0                   0   \n",
       "33731  8eea038c1a3408651f5980631768deaf  33731          0                   0   \n",
       "\n",
       "       sharedIdsDegree  p2pSharedCardPageRank  p2pSentPageRank  \\\n",
       "0                  0.0               0.150000         0.175456   \n",
       "1                  0.0               0.150000         0.310548   \n",
       "2                  0.0               0.150000         0.277500   \n",
       "3                  0.0               0.150000         0.230174   \n",
       "4                  0.0               0.150000         0.391426   \n",
       "...                ...                    ...              ...   \n",
       "33727              0.0               0.150000         0.297005   \n",
       "33728              1.0               0.999999         0.371382   \n",
       "33729              0.0               0.150000         0.150000   \n",
       "33730              0.0               0.150000         0.181054   \n",
       "33731              0.0               0.150000         0.185296   \n",
       "\n",
       "       p2pReceivedWeightedPageRank  p2pReceivedWeightedDegree  ipDegree  \\\n",
       "0                         0.414277                       10.0      21.0   \n",
       "1                         0.150422                       10.0       7.0   \n",
       "2                         0.150000                      160.0       8.0   \n",
       "3                         0.319779                      100.0     135.0   \n",
       "4                         0.150000                     2703.6     169.0   \n",
       "...                            ...                        ...       ...   \n",
       "33727                     0.451011                        5.0      14.0   \n",
       "33728                     1.632799                      150.0       5.0   \n",
       "33729                     0.277500                        0.0      38.0   \n",
       "33730                     0.170451                       10.0       1.0   \n",
       "33731                     0.401513                        5.0       2.0   \n",
       "\n",
       "       cardDegree  deviceDegree  communitySize  partOfCommunity  \n",
       "0             3.0           4.0              1                0  \n",
       "1             1.0           1.0              1                0  \n",
       "2             2.0           1.0              1                0  \n",
       "3            12.0           2.0              1                0  \n",
       "4             9.0           2.0              1                0  \n",
       "...           ...           ...            ...              ...  \n",
       "33727         2.0           2.0              1                0  \n",
       "33728         4.0           4.0              2                1  \n",
       "33729         2.0           0.0              1                0  \n",
       "33730         1.0           0.0              1                0  \n",
       "33731         1.0           0.0              1                0  \n",
       "\n",
       "[33732 rows x 14 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = gds.run_cypher('''\n",
    "    MATCH(u:User)\n",
    "    RETURN u.guid AS guid,\n",
    "        u.wccId AS wccId,\n",
    "        u.fraudRisk AS fraudRisk,\n",
    "        u.fraudMoneyTransfer AS fraudMoneyTransfer,\n",
    "        u.sharedIdsDegree AS sharedIdsDegree,\n",
    "        u.p2pSharedCardPageRank AS p2pSharedCardPageRank,\n",
    "        u.p2pSentPageRank AS p2pSentPageRank,\n",
    "        u.p2pReceivedWeightedPageRank AS p2pReceivedWeightedPageRank,\n",
    "        u.p2pReceivedWeightedDegree AS p2pReceivedWeightedDegree,\n",
    "        u.ipDegree AS ipDegree,\n",
    "        u.cardDegree AS cardDegree,\n",
    "        u.deviceDegree AS deviceDegree,\n",
    "        u.communitySize AS communitySize,\n",
    "        u.partOfCommunity AS partOfCommunity\n",
    "''')\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop(columns=['fraudRisk', 'fraudMoneyTransfer', 'wccId', 'guid'])\n",
    "y = df.fraudRisk - df.fraudMoneyTransfer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sharedIdsDegree</th>\n",
       "      <th>p2pSharedCardPageRank</th>\n",
       "      <th>p2pSentPageRank</th>\n",
       "      <th>p2pReceivedWeightedPageRank</th>\n",
       "      <th>p2pReceivedWeightedDegree</th>\n",
       "      <th>ipDegree</th>\n",
       "      <th>cardDegree</th>\n",
       "      <th>deviceDegree</th>\n",
       "      <th>communitySize</th>\n",
       "      <th>partOfCommunity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.175456</td>\n",
       "      <td>0.414277</td>\n",
       "      <td>10.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.310548</td>\n",
       "      <td>0.150422</td>\n",
       "      <td>10.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.277500</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>160.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.230174</td>\n",
       "      <td>0.319779</td>\n",
       "      <td>100.0</td>\n",
       "      <td>135.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.391426</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>2703.6</td>\n",
       "      <td>169.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33727</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.297005</td>\n",
       "      <td>0.451011</td>\n",
       "      <td>5.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33728</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.999999</td>\n",
       "      <td>0.371382</td>\n",
       "      <td>1.632799</td>\n",
       "      <td>150.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33729</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.277500</td>\n",
       "      <td>0.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33730</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.181054</td>\n",
       "      <td>0.170451</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33731</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.185296</td>\n",
       "      <td>0.401513</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>33732 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       sharedIdsDegree  p2pSharedCardPageRank  p2pSentPageRank  \\\n",
       "0                  0.0               0.150000         0.175456   \n",
       "1                  0.0               0.150000         0.310548   \n",
       "2                  0.0               0.150000         0.277500   \n",
       "3                  0.0               0.150000         0.230174   \n",
       "4                  0.0               0.150000         0.391426   \n",
       "...                ...                    ...              ...   \n",
       "33727              0.0               0.150000         0.297005   \n",
       "33728              1.0               0.999999         0.371382   \n",
       "33729              0.0               0.150000         0.150000   \n",
       "33730              0.0               0.150000         0.181054   \n",
       "33731              0.0               0.150000         0.185296   \n",
       "\n",
       "       p2pReceivedWeightedPageRank  p2pReceivedWeightedDegree  ipDegree  \\\n",
       "0                         0.414277                       10.0      21.0   \n",
       "1                         0.150422                       10.0       7.0   \n",
       "2                         0.150000                      160.0       8.0   \n",
       "3                         0.319779                      100.0     135.0   \n",
       "4                         0.150000                     2703.6     169.0   \n",
       "...                            ...                        ...       ...   \n",
       "33727                     0.451011                        5.0      14.0   \n",
       "33728                     1.632799                      150.0       5.0   \n",
       "33729                     0.277500                        0.0      38.0   \n",
       "33730                     0.170451                       10.0       1.0   \n",
       "33731                     0.401513                        5.0       2.0   \n",
       "\n",
       "       cardDegree  deviceDegree  communitySize  partOfCommunity  \n",
       "0             3.0           4.0              1                0  \n",
       "1             1.0           1.0              1                0  \n",
       "2             2.0           1.0              1                0  \n",
       "3            12.0           2.0              1                0  \n",
       "4             9.0           2.0              1                0  \n",
       "...           ...           ...            ...              ...  \n",
       "33727         2.0           2.0              1                0  \n",
       "33728         4.0           4.0              2                1  \n",
       "33729         2.0           0.0              1                0  \n",
       "33730         1.0           0.0              1                0  \n",
       "33731         1.0           0.0              1                0  \n",
       "\n",
       "[33732 rows x 10 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0        0\n",
      "1        0\n",
      "2        0\n",
      "3        0\n",
      "4        0\n",
      "        ..\n",
      "33727    0\n",
      "33728    0\n",
      "33729    0\n",
      "33730    0\n",
      "33731    0\n",
      "Length: 33732, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Training and Evaluation\n",
    "\n",
    "For purposes of this demo I am going to use a random forest classifier. Other classifiers including logistic regression, SVM, Neural Nets and Boosting variants could work as well. Going into the exact pros and cons of these models is out of scope here. Overall, I like exploring classification with Random Forests since they are relatively robust to feature scaling and collinearity issues and require minimal tuning to get working well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(class_weight='balanced', max_depth=5, n_estimators=500,\n",
       "                       random_state=0)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "clf = RandomForestClassifier(n_estimators=500, random_state=0, max_depth=5, bootstrap=True, class_weight='balanced')\n",
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of random forrest classifier on test set: 0.85\n",
      "\n",
      "Confusion Matrix: \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQMAAAEGCAYAAABhHPB4AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAASfElEQVR4nO3dfXBV9Z3H8fc3D1gWgkoJI9zASgAhgMQpAQQXsNDKw9a60DILsjOVqkD3AbrbbludnVqtdWxTp04Xyypqu24trI50wdKqTGsBtZTwEEgQKJGIJBEV0VIpU0j87h/3l5hASC7KOYeEz2vmTM7D79zzvTlzP/fc3z3nHnN3RESyki5ARM4PCgMRARQGIhIoDEQEUBiISJCTdAHNZWVleVaW8qkjKS4uTroEOQsHDhzg8OHD1tqy8y0MyMvLS7oMOQsvvfRS0iXIWRg/fvwZl+ltWEQAhYGIBAoDEQEUBiISKAxEBFAYiEigMBARQGEgIoHCQEQAhYGIBAoDEQEUBiISKAxEBFAYiEigMBARQGEgIoHCQEQAhYGIBAoDEQEUBiISKAxEBFAYiEigMBARQGEgIoHCQEQAhYGIBAoDEQEUBiISKAxEBFAYiEigMBARQGEgIoHCQEQAhYGIBAoDEQEUBiISKAxEBFAYiEigMBARQGEgIoHCQEQAhYGIBAoDEQEUBiISKAxEBFAYiEigMBARQGEgIoHCQEQAhcFHMmXKFDZv3szWrVv58pe/fNryHj16sGLFCjZu3MhLL73EjTfe2GJ5VlYW69evZ+XKlTFVLM899xxXXnklw4YNo7S09LTle/fuZdKkSfTo0YMf/OAHLZa9++67zJ07l5EjR1JcXMymTZviKjsWkYaBmU0zs71mVmVm34hyW3HLysqitLSU2bNnc/XVV/O5z32OIUOGtGhzyy23sHfvXiZMmMD111/P3XffTW5ubtPyRYsW8Yc//CHu0i9YDQ0NLFmyhNWrV1NeXs4TTzzB7t27W7S59NJLue+++1oN96985St8+tOfZufOnZSVlTF06NCYKo9HZGFgZtnAA8B0YBgw18yGRbW9uI0aNYr9+/dz4MABTp48yapVq5gxY0aLNu5O9+7dAejWrRvvvPMO9fX1APTt25frrruOxx57LPbaL1RlZWUMHDiQwsJCunTpwuzZs3n66adbtOnduzclJSUtQhvg6NGjvPDCC8yfPx+ALl26cMkll8RVeiyiPDIYA1S5+353PwGsBG6IcHux6tOnD7W1tU3TdXV19OnTp0Wb5cuXc8UVV7B7925efPFFbrvtNtwdgHvuuYc77riD999/P9a6L2R1dXUUFBQ0TadSKerq6jJat7q6mvz8fG699VbGjh3LokWLOHbsWFSlJiLKMEgBB5tN14R5LZjZAjPbYmZbOtILw8xOm9f4Qm80efJkKioqKCoqYuLEiXzve98jLy+PqVOncvjwYXbs2BFXucLp+wda34+tqa+vZ/v27SxYsIDf//73dOvWrdU+h44syjBo7b982t5w94fcvcTdS7KyOk5/Zl1dHanUB9nWt29fDh061KLNvHnz+MUvfgGk31kOHDjA4MGDGTt2LNOmTWPHjh088sgjTJgwgQcffDDW+i9EqVSKmpqapuna2trTjubaWjeVSjFmzBgAZs6cSXl5eRRlJibKV18N0K/ZdAGQ2TFZB7Bt2zYGDhxI//79yc3NZdasWfzqV79q0aampoaJEycCkJ+fz6BBg3j11Ve56667GDFiBMXFxdx8881s3LiRhQsXJvE0LiglJSVUVVVRXV3NiRMnePLJJ/nMZz6T0bqXXXYZBQUFTR2+zz//PEVFRVGWG7ucCB+7DBhsZgOAWmAOcGPbq3QcDQ0NfO1rX+Opp54iOzubxx9/nD179jR1MP34xz+mtLSUBx54gBdffBEz48477+TIkSMJV37hysnJ4f777+f666+noaGBL3zhCwwbNozly5cDcOutt3Lo0CGuueYajh49SlZWFkuXLmX79u1NXzXedNNNnDhxggEDBvDQQw8l/IzOLWvtc9Q5e3CzGcD9QDbwqLt/p632OTk5npeXF1k9cu6d+tFIzm/jx49n69atrXaURHlkgLv/EvhllNsQkXOj4/TYiUikFAYiAigMRCRQGIgIoDAQkUBhICKAwkBEAoWBiAAKAxEJFAYiAigMRCRQGIgIoDAQkUBhICKAwkBEAoWBiAAKAxEJFAYiAigMRCRQGIgIoDAQkUBhICKAwkBEAoWBiAAKAxEJFAYiArRxezUz+09auYV6I3dfHElFIpKItu61uCW2KkQkcWcMA3f/7+bTZtbN3Y9FX5KIJKHdPgMzG2dmLwO7w3Sxmf0o8spEJFaZdCDeD0wF3gZw9x3AxAhrEpEEZPRtgrsfPGVWQwS1iEiC2upAbHTQzMYDbmZdgMWEjwwi0nlkcmSwCPgnIAXUAleFaRHpRNo9MnD3w8C8GGoRkQRl8m1CoZk9bWZvmdmbZrbazArjKE5E4pPJx4SfAU8AfYC+wJPAiiiLEpH4ZRIG5u7/4+71YfgpbZymLCIdU1vXJvQMo8+b2TeAlaRD4O+BtTHUJiIxaqsDcSvpF7+F6YXNljnw7aiKEpH4tXVtwoA4CxGRZGVy0hFmNgIYBnyscZ67PxZVUSISv3bDwMzuAK4lHQa/BKYDLwAKA5FOJJNvEz4PTAEOuft8oBi4KNKqRCR2mYTBcXd/H6g3sx7Am4BOOhLpZDLpM9hiZpcAy0l/w/AesDnKokQkfplcm/CPYfS/zOwZoIe774y2LBGJW1snHX2irWXuvi2akkQkCebe+pnFZvZ8G+u5u08+18WUlJT4li36HdaOxMzabyTnFXdvdae1ddLRJ6MrR0TON7qJiogACgMRCRQGIgJk9ktHZmb/YGbfDNP9zWxM9KWJSJwyOTL4ETAOmBum/wQ8EFlFIpKITM5AHOvunzCz7QDu/k74yXQR6UQyOTI4aWbZhJ86M7N84P1IqxKR2GUSBj8Efg70NrPvkL58+Z5IqxKR2GVybcLjZraV9GXMBvydu+uOSiKdTCY/btIf+DPwdPN57v5alIWJSLwy6UBcywc/jPoxYACwFxgeYV0iErNMPiZc2Xw6XM248AzNRaSDOuszEMOly6MjqEVEEpRJn8G/NZvMAj4BvBVZRSKSiEz6DPKajdeT7kN4KppyRCQpbYZBONmou7v/e0z1iEhCzthnYGY57t5A+mOBiHRybR0ZbCYdBOVmtob0rdiPNS5091UR1yYiMcqkz6An8DYwmQ/ON3BAYSDSibQVBr3DNwmVtLwbM2FaRDqRtsIgG+hOyxBopDAQ6WTaCoPX3f2u2CoRkUS1dQaifhBf5ALSVhhMia0KEUncGcPA3Y/EWYiIJEs/lS4igMJARAKFgYgACgMRCRQGIgIoDEQkUBiICKAwEJFAYSAigMJARAKFgYgACgMRCRQGIgIoDEQkUBiICKAwEJFAYSAigMJARAKFgYgACgMRCRQGIgIoDEQkUBiICKAw+EieeeYZhgwZwqBBg7j33ntPW75nzx7GjRvHRRddxPe///0Wy774xS/Su3dvRowYEVe5AkydOpU9e/awb98+vv71r5+2vEePHqxZs4by8nIqKyu56aabmpYtXryYiooKKisrWbJkSYxVx8TdIxmAR4E3gcpM1xk1apR3FPX19V5YWOivvPKK/+Uvf/GRI0f6rl27WrR54403fPPmzX777bd7aWlpi2Xr16/3rVu3+vDhw+Ms+5wjfRPeDjFkZWV5VVWVDxgwwHNzc728vNyLiopatLntttv83nvvdcB79erlb7/9tufm5vrw4cO9oqLCu3bt6tnZ2b5u3TofNGhQ4s/pwwx+htdflEcGPwGmRfj4idq8eTODBg2isLCQLl26MGfOHFavXt2iTe/evRk9ejS5ubmnrT9x4kR69uwZV7kCjBkzhqqqKqqrqzl58iQrV67khhtuaNHG3cnLywOge/fuHDlyhPr6eoqKiti0aRPHjx+noaGB9evXM3PmzCSeRmQiCwN33wB02lu01dbW0q9fv6bpgoICamtrE6xI2pNKpTh48GDTdE1NDalUqkWbpUuXUlRURF1dHRUVFSxZsgR3p7KysinAu3btyowZM1rs/86grVuyx8LMFgALAPr3759wNZlLHyG3ZKYbV5/PWts/p+7HqVOnUl5ezuTJkxk4cCDr1q2juLiYPXv28N3vfpd169bx3nvvsWPHDurr6+MqPRaJdyC6+0PuXuLuJfn5+UmXk7GCgoLT3mX69u2bYEXSnpqamtOO5urq6lq0mT9/PqtWrQLglVdeobq6mqFDhwLw6KOPMmrUKCZNmsSRI0fYt29ffMXHIPEw6KhGjx7Nvn37qK6u5sSJE6xcuZLPfvazSZclbSgrK2Pw4MFcfvnl5ObmMmfOHNasWdOizWuvvcaUKVOAdJ/PkCFD2L9/PwCNb1b9+vVj1qxZrFixIt4nELUz9SyeiwG4nE76bYK7+9q1a33w4MFeWFjod999t7u7L1u2zJctW+bu7q+//rqnUinPy8vziy++2FOplP/xj390d/c5c+b4ZZdd5jk5OZ5Kpfzhhx9O7Hl8FJwHveNnM0yfPt337t3rVVVVfvvttzvgCxcu9IULFzrgffr08WeffdZ37tzpFRUVPm/evKZ1N2zY4Lt27fLy8nKfPHly4s/lww5+htefeSuffc8FM1sBXAv0At4A7nD3R9pap6SkxLds2RJJPRIN9ZN0PO7e6k6LrAPR3edG9dgicu6pz0BEAIWBiAQKAxEBFAYiEigMRARQGIhIoDAQEUBhICKBwkBEAIWBiAQKAxEBFAYiEigMRARQGIhIoDAQEUBhICKBwkBEAIWBiAQKAxEBFAYiEigMRARQGIhIoDAQEUBhICKBwkBEAIWBiAQKAxEBFAYiEigMRARQGIhIoDAQEUBhICKBwkBEAIWBiAQKAxEBFAYiEigMRARQGIhIoDAQEUBhICKBwkBEAIWBiAQKAxEBFAYiEigMRARQGIhIoDAQEUBhICKBwkBEADB3T7qGJmb2FnAg6Toi0As4nHQRclY66z77a3fPb23BeRUGnZWZbXH3kqTrkMxdiPtMHxNEBFAYiEigMIjHQ0kXIGftgttn6jMQEUBHBiISKAxEBFAYRMrMppnZXjOrMrNvJF2PtM/MHjWzN82sMula4qYwiIiZZQMPANOBYcBcMxuWbFWSgZ8A05IuIgkKg+iMAarcfb+7nwBWAjckXJO0w903AEeSriMJCoPopICDzaZrwjyR85LCIDrWyjx9jyvnLYVBdGqAfs2mC4C6hGoRaZfCIDplwGAzG2BmXYA5wJqEaxI5I4VBRNy9Hvhn4FlgN/CEu+9Ktippj5mtAH4HDDGzGjO7Oema4qLTkUUE0JGBiAQKAxEBFAYiEigMRARQGIhIoDDoQMyswczKzazSzJ40s7/6CI/1EzP7fBh/uK2LqMzsWjMb/yG28aqZ9cp0/ilt3jvLbX3LzL56tjXKBxQGHctxd7/K3UcAJ4BFzReGKyXPmrvf4u4vt9HkWuCsw0A6FoVBx7URGBTetZ83s58BFWaWbWalZlZmZjvNbCGApS01s5fNbC3Qu/GBzOy3ZlYSxqeZ2TYz22Fmvzazy0mHzr+Go5IJZpZvZk+FbZSZ2TVh3Y+b2XNmtt3MHqT16zNaMLP/M7OtZrbLzBacsuy+UMuvzSw/zBtoZs+EdTaa2dBz8t8UcHcNHWQA3gt/c4DVwJdIv2sfAwaEZQuA/wjjFwFbgAHALGAdkA30Bd4FPh/a/RYoAfJJX2nZ+Fg9w99vAV9tVsfPgL8J4/2B3WH8h8A3w/jfkr4wq1crz+PVxvnNttEVqAQ+HqYdmBfGvwksDeO/BgaH8bHAb1qrUcPZDzkfLkIkIV3NrDyMbwQeIX34vtndq8P864CRjf0BwMXAYGAisMLdG4A6M/tNK49/NbCh8bHc/UzX9X8KGGbW9Mbfw8zywjZmhXXXmtk7GTynxWY2M4z3C7W+DbwP/G+Y/1NglZl1D8/3yWbbviiDbUgGFAYdy3F3v6r5jPCiONZ8FvAv7v7sKe1m0P4l1JZBG0h/vBzn7sdbqSXj89vN7FrSwTLO3f9sZr8FPnaG5h62++6p/wM5N9Rn0Pk8C3zJzHIBzOwKM+sGbADmhD6FPsAnW1n3d8AkMxsQ1u0Z5v8JyGvW7jnSF2ER2l0VRjcA88K86cCl7dR6MfBOCIKhpI9MGmUBjUc3NwIvuPtRoNrMZodtmJkVt7MNyZDCoPN5GHgZ2BZ+1PNB0keAPwf2ARXAMmD9qSu6+1uk+xxWmdkOPjhMfxqY2diBCCwGSkIH5ct88K3GncBEM9tG+uPKa+3U+gyQY2Y7gW8Dm5otOwYMN7OtwGTgrjB/HnBzqG8X+im5c0ZXLYoIoCMDEQkUBiICKAxEJFAYiAigMBCRQGEgIoDCQESC/weBabj6Wjg0ygAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import ConfusionMatrixDisplay\n",
    "print('Accuracy of random forrest classifier on test set: {:.2f}'.format(clf.score(X_test, y_test)))\n",
    "print('\\nConfusion Matrix: ')\n",
    "disp = ConfusionMatrixDisplay.from_predictions(y_test, clf.predict(X_test), display_labels=clf.classes_,\n",
    "                                               normalize='true', cmap='Greys', colorbar=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAqa0lEQVR4nO3deZgU1b3/8fdHBEFFNiFRkIAIGlxAHSUaF9wVxeWGiBjXxBhvNOr1xsgvi1Fvbq5evcZgVEQ0GKNgXFA0xjXuCwIREXAJwqgoKhJFXBH8/v6omrFpZqZrhukeevrzep55pk/Vqapv9UCfPkudo4jAzMwq1zotHYCZmbUsFwRmZhXOBYGZWYVzQWBmVuFcEJiZVTgXBGZmFc4FgZlZhXNBYK2KpGpJn0r6SNLbkiZI2jAvz66S/i5pmaSlku6SNDAvz0aSLpP0enqueWl643quK0mnS5ot6WNJCyXdImnbYt6vWXNwQWCt0fCI2BAYDGwP/L+aHZJ2Ae4H7gQ2BfoCzwNPSto8zdMOeAjYGjgQ2AjYFVgC7FzPNX8PnAGcDnQFBgB3AAc3NnhJ6zb2GLM1IT9ZbK2JpGrgpIh4ME3/L7B1RBycph8HXoiIH+cd9zdgcUQcJ+kk4L+BfhHxUYZr9gdeAnaJiGfryfMI8OeIGJ+mT0jj3C1NB3AacCawLnAf8FFE/DTnHHcCj0bEpZI2BS4H9gA+An4XEWMKv0Nmq3ONwFotSb2Ag4B5aXp9km/2t9SR/S/AfunrfYF7sxQCqX2AhfUVAo1wODAEGAjcBIyUJABJXYD9gUmS1gHuIqnJ9Eyvf6akA9bw+lahXBBYa3SHpGXAG8C7wK/T7V1J/s0vquOYRUBN+3+3evLUp7H56/M/EfGviPgUeBwIYPd03wjg6Yh4C9gJ6B4RF0TE8oiYD1wDHNUMMVgFckFgrdHhEdERGApsxVcf8O8DXwKb1HHMJsB76esl9eSpT2Pz1+eNmheRtNlOAkalm44GbkxffwPYVNIHNT/Az4GvNUMMVoFcEFirFRGPAhOAS9L0x8DTwHfryH4kSQcxwIPAAZI2yHiph4BekqoayPMxsH5O+ut1hZyXngiMkPQNkiaj29LtbwALIqJzzk/HiBiWMV6zVbggsNbuMmA/SYPT9Gjg+HSoZ0dJXST9BtgFOD/NcwPJh+1tkraStI6kbpJ+Lmm1D9uI+CdwJTBR0lBJ7SS1l3SUpNFptpnAv0laX9IWwA8KBR4RzwGLgfHAfRHxQbrrWeBDSedI6iCpjaRtJO3U2DfHDFwQWCsXEYuBPwG/StNPAAcA/0bSrv8ayRDT3dIPdCLic5IO45eAB4APST58Nwam1nOp04E/AFcAHwCvAkeQdOoC/A5YDrwDXM9XzTyFTExjuSnnnlYCw0mGxy4gadIaD3TKeE6zVXj4qJlZhXONwMyswrkgMDOrcC4IzMwqnAsCM7MKV3aTW2288cbRp0+flg7DzKyszJgx472I6F7XvrIrCPr06cP06dNbOgwzs7Ii6bX69rlpyMyswrkgMDOrcC4IzMwqnAsCM7MK54LAzKzCFa0gkHSdpHclza5nvySNSRcFnyVph2LFYmZm9StmjWACycLf9TkI6J/+nAxcVcRYzMysHkV7jiAiHpPUp4EshwF/SldiekZSZ0mbRERzLPlnGd009XXunPlmS4dhZhkM3HQjfj1862Y/b0v2EfQkZ2k+YGG6bTWSTpY0XdL0xYsXlyS4SnHnzDeZu+jDlg7DzFpQSz5ZrDq21bk4QkSMA8YBVFVVeQGFZjZwk424+Ue7tHQYZtZCWrJGsBDYLCfdC3irhWIxM6tYLVkQTAGOS0cPfQtY6v4BM7PSK1rTkKSJwFBgY0kLgV8DbQEiYixwDzAMmAd8ApxYrFjMzKx+xRw1NKrA/gBOLdb1rWE1o4XmLvqQgZts1NLhmFkL8pPFFSq3EDhscJ2DtcysQpTdegTWfDxayMzANQIzs4rngsDMrMK5aagVa2j6CHcSm1kN1whasYamj3AnsZnVyFQjkLQOMAjYFPgUmBMR7xQzMGse7hA2s0IaLAgk9QPOAfYF/gksBtoDAyR9AlwNXB8RXxY7UDMzK45CNYLfkKwT8KP0AbBaknoARwPHAtcXJzwzMyu2BguChp4Ojoh3gcuaOyAzMyutJo8akrRfRDzQnMFY09U1Qsgjg8wsizUZNXRts0Vha6yuEUIeGWRmWRTqLJ5S3y6gW/OHY2vCI4TMrCkKNQ3tDhwDfJS3XcDORYnIzMxKqlBB8AzwSUQ8mr9D0svFCcnMzEqp0KihgxrYt0fzh2O5GpoiIp87hs2sqTzFxFqsoSki8rlj2MyaypPOreXcAWxmxeYagZlZhXNBYGZW4TIXBJLOayhtZmblqTF9BDMKpC2jrKOBPBLIzEohc40gIu5qKG3ZZR0N5JFAZlYKhaaYuByI+vZHxOnNHlGF8GggM1tbFGoaml6SKMzMrMUUerJ4lQVnJG0QER8XNyQzMyulrGsW70Iy7fSGQG9Jg0hWLftxMYNrLfI7h90JbGZrk6ydxZcBBwBLACLiecBzDWWU3znsTmAzW5tkHj4aEW9Iyt20svnDab3cOWxma6usBcEbknYFQlI74HTgxeKFZWZmpZK1aegU4FSgJ/AmMDhNm5lZmctUEETEexHxvYj4WkR0j4hjImJJoeMkHSjpZUnzJI2uY38nSXdJel7SHEknNuUmzMys6TIVBJI2Tz+wF0t6V9KdkjYvcEwb4ArgIGAgMErSwLxspwJzI2IQMBT4v7TpqVW4aerrjLz66cxrCpiZtYSsTUM3AX8BNgE2BW4BJhY4ZmdgXkTMj4jlwCTgsLw8AXRU0gu9IfAvYEXGmNZ6NaOFPErIzNZmWTuLFRE35KT/LOm0Asf0BN7ISS8EhuTl+QMwBXgL6AiMjIgvV7u4dDJwMkDv3r0zhrx28GghM1vbNVgjkNRVUlfgYUmjJfWR9A1JPwP+WuDcqmNb/rxFBwAzSWoZg4E/SFrtSauIGBcRVRFR1b179wKXNTOzxihUI5hB8uFd86H+o5x9AfxXA8cuBDbLSfci+eaf60TgwogIYJ6kBcBWwLMF4jIzs2ZSaK6hvmtw7mlAf0l9SYacHgUcnZfndWAf4HFJXwO2BOavwTVbRH3rC3gqCTMrB5mfLJa0Dcnon/Y12yLiT/Xlj4gVaT/CfUAb4LqImCPplHT/WJIaxQRJL5DUOs6JiPeadCctKLdTOJc7ic2sHGSddO7XJMM7BwL3kAwJfQKotyAAiIh70vy528bmvH4L2L9REa+l3ClsZuUq6/DRESRNOG9HxInAIGC9okVlZmYlk7Ug+DQd1rkiHdXzLtDgA2VmZlYesvYRTJfUGbiGZCTRR3hkj5lZq5CpIMhZgGaspHuBjSJiVvHCKh83TX2dqQv+xZC+XVs6FDOzJim0eP0ODe2LiH80f0jlpWbYqEcHmVm5KlQj+L8G9gWwdzPGUraG9O3K0UPKa+oLM7MahR4o26tUgZiZWcvIOmrIzMxaKRcEa6Cmo9jMrJy5IFgD7ig2s9Yg6wplknSMpHPTdG9JOxc3tPLgjmIzK3dZawRXArsAo9L0MpJlKM3MrMxlfbJ4SETsIOk5gIh4vzWtLWxmVsmy1gi+SBejDwBJ3YHVlpQ0M7Pyk7VGMAaYDPSQ9N8ks5H+smhRrUXqW3QGvPCMmbUOWecaulHSDJKpqAUcHhEvFjWytUR9i86AF54xs9Yh68I0vwdujoiK7CD2ojNm1ppl7SP4B/BLSfMkXSypqphBmZlZ6WQqCCLi+ogYBuwMvAJcJOmfRY3MzMxKorFPFm8BbAX0AV5q9mjWMp5CwswqQdYni2tqABcAc4AdI2J4USNbC3gKCTOrBFmHjy4AdomI94oZzNrIU0iYWWtXaIWyrSLiJZL1iXtLWuUT0SuUmZmVv0I1grOAk6l7pTKvUGZm1goUWqHs5PTlQRHxWe4+Se2LFpWZmZVM1lFDT2XcZmZmZaZQH8HXgZ5AB0nbk0wvAbARsH6RYzMzsxIo1EdwAHAC0Au4NGf7MuDnRYrJzMxKqFAfwfXA9ZK+ExG3lSgmMzMroUJNQ8dExJ+BPpLOyt8fEZfWcZiZmZWRQp3FG6S/NwQ61vHTIEkHSno5naxudD15hkqaKWmOpEcbEXvR3DT1dUZe/TRzF33Y0qGYmRVdoaahq9Pf5zf2xOmKZlcA+wELgWmSpkTE3Jw8nUnWQz4wIl6X1KOx1ymG3DUIPL2EmbV2Weca+l9JG0lqK+khSe9JOqbAYTsD8yJifkQsByYBh+XlORq4PSJeB4iIdxt7A8VSswaBp5cws9Yu63ME+0fEh8AhJN/uBwBnFzimJ/BGTnphui3XAKCLpEckzZB0XF0nknSypOmSpi9evDhjyGZmlkXWgqBt+nsYMDEisszNrDq2RV56XWBH4GCSoaq/kjRgtYMixkVEVURUde/ePWPIZmaWRdbZR++S9BLwKfBjSd2BzwocsxDYLCfdC3irjjzvRcTHwMeSHgMGkSx+Y2ZmJZB1hbLRwC5AVUR8AXzM6u39+aYB/SX1ldQOOAqYkpfnTmB3SetKWh8YArzYmBswM7M1k3Xx+rbAscAekgAeBcY2dExErJB0GnAf0Aa4LiLmSDol3T82Il6UdC8wC/gSGB8Rs5t8N2Zm1mhZm4auIuknuDJNH5tuO6mhgyLiHuCevG1j89IXAxdnjMPMzJpZ1oJgp4gYlJP+u6TnixGQmZmVVtZRQysl9atJSNocWFmckMzMrJSy1gjOBh6WNJ9kWOg3gBOLFlULuGnq67WL1dc8VWxmVgkKFgTpUNGlJE8K9yApCF6KiM+LHFtJ5U4r4aklzKySFJp99CTgt8CrQF/g5IjIHwLaatRMK2FmVkkK1QjOBLaOiMVpv8CNrP4sgJmZlbFCncXLI2IxQETMB9YrfkhmZlZKhWoEvSSNqS8dEacXJywzMyuVQgVB/gyjM4oViJmZtYwsaxabmVkr1mAfgaRxkrapZ98Gkr4v6XvFCc3MzEqhUNPQlcC5krYFZgOLgfZAf2Aj4DqSkURmZlamCjUNzQSOlLQhUAVsQrImwYsR8XLxwzMzs2LLNMVERHwEPFLcUFpGzdQSnlbCzCpV1knnWq3cQsDTSphZJco66Vyr5qklzKySNapGIGmDYgViZmYtI1NBIGlXSXNJ1xOWNEjSlQUOMzOzMpC1RvA74ABgCUBEPA/sUaygzMysdDI3DUXEG3mbvEKZmVkrkLWz+A1JuwIhqR1wOmkzkZmZlbesNYJTgFOBnsBCYDDw4yLFZGZmJZS1RrBlRKwyp5CkbwNPNn9IZmZWSllrBJdn3GZmZmWm0JrFuwC7At0lnZWzayOgTTEDK4Wbpr7O1AX/Ykjfri0diplZiynUNNQO2DDN1zFn+4fAiGIFVSp3znwTwFNLmFlFKzT76KPAo5ImRMRrJYqppIb07crRQ3q3dBhmZi0ma2fxJ5IuBrYmWY8AgIjYuyhRmZlZyWTtLL4ReAnoC5wPVAPTihSTmZmVUNaCoFtEXAt8ERGPRsT3gW8VMS4zMyuRrE1DX6S/F0k6GHgL6FWckMzMrJSy1gh+I6kT8J/AT4HxwJmFDpJ0oKSXJc2TNLqBfDtJWimp7EcimZmVm6xLVd6dvlwK7AW1TxbXS1Ib4ApgP5JpKaZJmhIRc+vIdxFwX+NCNzOz5tBgjUBSG0mjJP1U0jbptkMkPQX8ocC5dwbmRcT8iFgOTAIOqyPfT4DbgHcbH76Zma2pQjWCa4HNgGeBMZJeA3YBRkfEHQWO7QnkTl29EBiSm0FST+AIYG9gp/pOJOlk4GSA3r095t/MrDkVKgiqgO0i4ktJ7YH3gC0i4u0M51Yd2yIvfRlwTkSslOrKnh4UMQ4YB1BVVZV/jibx9BJmZolCBcHyiPgSICI+k/RKxkIAkhrAZjnpXiSjjXJVAZPSQmBjYJikFRlqG2vM00uYmSUKFQRbSZqVvhbQL00LiIjYroFjpwH9JfUF3gSOAo7OzRARfWteS5oA3F2KQqCGp5cwMytcEHyzqSeOiBWSTiMZDdQGuC4i5kg6Jd0/tqnnNjOz5lNo0rk1mmguIu4B7snbVmcBEBEnrMm1zMysaTIvXm9mZq2TCwIzswqXuSCQ1EHSlsUMxszMSi9TQSBpODATuDdND5Y0pYhxmZlZiWStEZxHMmXEBwARMRPoU4yAzMystLIWBCsiYmlRIzEzsxaRdT2C2ZKOBtpI6g+cDjxVvLDMzKxUstYIfkKyXvHnwE0k01GfWaSYzMyshLLWCLaMiF8AvyhmMGZmVnpZawSXSnpJ0n9J2rqoEZmZWUllKggiYi9gKLAYGCfpBUm/LGZgZmZWGpkfKIuItyNiDHAKyTMF5xYrKDMzK52sD5R9U9J5kmaTLFH5FMn6AmWpZlEaMzPL3ln8R2AisH9E5C8uU3a8KI2Z2VcyFQQR8a1iB1JqXpTGzCzRYEEg6S8RcaSkF1h1veEsK5SZmVkZKFQjOCP9fUixAzEzs5bRYGdxRCxKX/44Il7L/QF+XPzwzMys2LIOH92vjm0HNWcgZmbWMgr1Efw7yTf/zSXNytnVEXiymIGZmVlpFOojuAn4G/A/wOic7csiwgPxzcxagUIFQUREtaRT83dI6urCwMys/GWpERwCzCAZPqqcfQFsXqS4iqbmqeIhfbu2dChmZmuFBguCiDgk/d23NOEUn58qNjNbVda5hr4taYP09TGSLpVUto/l+qliM7OvZB0+ehXwiaRBwM+A14AbihaVmZmVTGMWrw/gMOD3EfF7kiGkZmZW5rLOPrpM0v8DjgV2l9QGaFu8sMzMrFSy1ghGkixc//2IeBvoCVxctKjMzKxksi5V+TZwI9BJ0iHAZxHxp6JGZmZmJZF11NCRwLPAd4EjgamSRmQ47kBJL0uaJ2l0Hfu/J2lW+vNU2hltZmYllLWP4BfAThHxLoCk7sCDwK31HZD2I1xBMmHdQmCapCkRMTcn2wJgz4h4X9JBwDhgSONvw8zMmiprH8E6NYVAakmGY3cG5kXE/IhYDkwiGXVUKyKeioj30+QzlPE6yGZm5SprQXCvpPsknSDpBOCvwD0FjukJvJGTXphuq88PSCa4W42kkyVNlzR98eLFGUNenRetNzNbXdY1i8+W9G/AbiTzDY2LiMkFDlMd26KObUjai6Qg2K2e648jaTaiqqqqznNk4eklzMxWV2g9gv7AJUA/4AXgpxHxZsZzLwQ2y0n3At6q4xrbAeOBgyJiScZzN5mnlzAzW1WhpqHrgLuB75DMQHp5I849Degvqa+kdsBRwJTcDOl8RbcDx0bEK404t5mZNZNCTUMdI+Ka9PXLkv6R9cQRsULSacB9QBvguoiYI+mUdP9Y4FygG3ClJEimsqhq7E2YmVnTFSoI2kvanq/a+zvkpiOiwYIhIu4hr1M5LQBqXp8EnNTYoM3MrPkUKggWAZfmpN/OSQewdzGCMjOz0im0MM1epQrEzMxaRtbnCMzMrJVyQWBmVuFcEJiZVbiss48qXav43DTdW9LOxQ3NzMxKIWuN4EpgF2BUml5GMrOomZmVuazTUA+JiB0kPQeQThvdrohxmZlZiWStEXyRri8QULsewZdFi8rMzEoma0EwBpgM9JD038ATwG+LFpWZmZVM1mmob5Q0A9iHZHqJwyPixaJGZmZmJZGpIEhnCf0EuCt3W0S8XqzAzMysNLJ2Fv+VpH9AQHugL/AysHWR4jIzsxLJ2jS0bW5a0g7Aj4oSkZmZlVSTnixOp5/eqZljMTOzFpC1j+CsnOQ6wA5A01eRNzOztUbWGkHHnJ/1SPoMDitWUMVw09TXmbrgXy0dhpnZWqdgjSB9kGzDiDi7BPEUzZ0z3wTgsME9WzgSM7O1S4M1AknrRsRKkqagsjekb1eOHtK7pcMwM1urFKoRPEtSCMyUNAW4Bfi4ZmdE3F7E2MzMrASyPkfQFVhCskZxzfMEAbggMDMrc4UKgh7piKHZfFUA1IiiRWVWAb744gsWLlzIZ5991tKhWCvSvn17evXqRdu2bTMfU6ggaANsyKoFQA0XBGZrYOHChXTs2JE+ffog1fVfzKxxIoIlS5awcOFC+vbtm/m4QgXBooi4YM1CM7O6fPbZZy4ErFlJolu3bixe3LjHvAo9R+B/oWZF5ELAmltT/k0VKgj2aVooZmZWLhosCCLCj+KatWJt2rRh8ODBbLPNNgwfPpwPPvgAgOrqajp06MDgwYNrf5YvX77KsY888giSuPbaa2u3Pffcc0jikksuyRxDdXU122yzTZPzLFq0iEMOOWSVbWeccQY9e/bkyy+/WkjxvPPOWy2uPn368N577wHw9ttvc9RRR9GvXz8GDhzIsGHDeOWVVzLfR10+//xzRo4cyRZbbMGQIUOorq6uM9/NN9/Mdtttx9Zbb83Pfvaz2u0TJkyge/futX+D8ePHA7B48WIOPPDANYotV5MmnTOz1qFDhw7MnDmT2bNn07VrV6644oraff369WPmzJm1P+3arb5M+bbbbsvNN99cm540aRKDBg0qSew1Lr30Un74wx/Wpr/88ksmT57MZpttxmOPPZbpHBHBEUccwdChQ3n11VeZO3cuv/3tb3nnnXfWKLZrr72WLl26MG/ePP7jP/6Dc845Z7U8S5Ys4eyzz+ahhx5izpw5vPPOOzz00EO1+0eOHFn7NzjppJMA6N69O5tssglPPvnkGsVXI+tzBGZWROffNYe5b33YrOccuOlG/Hp49iVDdtllF2bNmtWoa/Tu3ZsPP/yQd955hx49enDvvfcybNiw2v0zZ87klFNO4ZNPPqFfv35cd911dOnShRkzZvD973+f9ddfn9122602/8qVKxk9ejSPPPIIn3/+Oaeeeio/+lHDM97fdttt/OY3v6lNP/zww2yzzTaMHDmSiRMnMnTo0IL38fDDD9O2bVtOOeWU2m2DBw/O/kbU48477+S8884DYMSIEZx22mlExCrt+PPnz2fAgAF0794dgH333ZfbbruNffZpuGX+8MMP58Ybb+Tb3/72GsfpGoGZsXLlSh566CEOPfTQ2m2vvvpqbZPEqaeeWu+xI0aM4JZbbuGpp55ihx12YL311qvdd9xxx3HRRRcxa9Ystt12W84//3wATjzxRMaMGcPTTz+9yrmuvfZaOnXqxLRp05g2bRrXXHMNCxYsqPfaCxYsoEuXLqtcc+LEiYwaNYojjjiCu+++my+++KLg/c+ePZsdd9yxYD6A3XfffZUms5qfBx98cLW8b775JpttthkA6667Lp06dWLJkiWr5Nliiy146aWXqK6uZsWKFdxxxx288cYbtftvu+02tttuO0aMGLHK9qqqKh5//PFMMRfiGoHZWqAx39yb06effsrgwYOprq5mxx13ZL/99qvdV9M0VMiRRx7JyJEjeemllxg1ahRPPfUUAEuXLuWDDz5gzz33BOD444/nu9/97mrbjz32WP72t78BcP/99zNr1ixuvfXW2nP885//ZMCAAXVee9GiRbXfpAGWL1/OPffcw+9+9zs6duzIkCFDuP/++zn44IPrHU3T2FE2jfnwjVj9cav863Xp0oWrrrqKkSNHss4667Drrrsyf/58AIYPH86oUaNYb731GDt2LMcffzx///vfAejRowdvvfVWo2KvT1FrBJIOlPSypHmSRtexX5LGpPtnpSufmVmJ1PQRvPbaayxfvnyVPoKsvv71r9O2bVseeOCBgs0ZwGpNI/n7Lr/88to28QULFrD//vs3GH/uk9n33nsvS5cuZdttt6VPnz488cQTTJw4EYBu3brx/vvvr3L8smXL6Ny5M1tvvTUzZszIcruNqhH06tWr9lv8ihUrWLp0KV27dl0t3/Dhw5k6dSpPP/00W265Jf3796+Nuaa288Mf/nCVGD/77DM6dOiQKeZCilYQpNNXXwEcBAwERkkamJftIKB/+nMycFWx4jGz+nXq1IkxY8ZwySWXZGpKyXfBBRdw0UUX0aZNm1XO2aVLl9pv0DfccAN77rknnTt3plOnTjzxxBMA3HjjjbXHHHDAAVx11VW1Mbzyyit8/PHH1GfAgAGrjMSZOHEi48ePp7q6murqahYsWMD999/PJ598wh577MGUKVNYtmwZALfffjuDBg2iTZs27L333nz++edcc801teeaNm0ajz766GrXfPzxx1fpRK/52XfffVfLe+ihh3L99dcDcOutt7L33nvXWQi+++67ALz//vtceeWVtZ3CixYtqs0zZcoUvvnNb9amX3nllYKjrbIqZtPQzsC8iJgPIGkSyWI2c3PyHAb8KZL60zOSOkvaJCIWrX46Myum7bffnkGDBjFp0iR23333Rh2766671rn9+uuvr+0s3nzzzfnjH/8IwB//+MfazuIDDjigNv9JJ51EdXU1O+ywAxFB9+7dueOOO+q97gYbbEC/fv2YN28em266Kffddx9XX331Kvt322037rrrLkaOHMlpp53GbrvthiR69OhROxxTEpMnT+bMM8/kwgsvpH379vTp04fLLrusUe9Dvh/84Acce+yxbLHFFnTt2pVJkybV7hs8eHBt09sZZ5zB888/D8C5555b2xQ2ZswYpkyZwrrrrkvXrl2ZMGFC7fEPP/wwBx988BrFV0N1tWE1y4mlEcCBEXFSmj4WGBIRp+XkuRu4MCKeSNMPAedExPS8c51MUmOgd+/eO7722muNjuf8u+YALdcWa5bvxRdfXOUbnjXN5MmTmTFjxiojhyrBHnvswZ133kmXLl1W21fXvy1JMyKiqq5zFbNGkGWiukyT2UXEOGAcQFVVVZNKLhcAZq3TEUccsdpInNZu8eLFnHXWWXUWAk1RzM7ihcBmOeleQH4Xd5Y8ZmYNqmlTrxTdu3fn8MMPb7bzFbMgmAb0l9RXUjvgKGBKXp4pwHHp6KFvAUvdP2CVpFhNs1a5mvJvqmhNQxGxQtJpwH0k6xpcFxFzJJ2S7h8L3AMMA+YBnwAnFises7VN+/btWbJkCd26dfMspNYsatYjaN++faOOK1pncbFUVVXF9OnTC2c0W8t5hTIrhvpWKGupzmIza0Dbtm0btYqUWbF4riEzswrngsDMrMK5IDAzq3Bl11ksaTHQ+EeLExsD7zVjOOXA91wZfM+VYU3u+RsR0b2uHWVXEKwJSdPr6zVvrXzPlcH3XBmKdc9uGjIzq3AuCMzMKlylFQTjWjqAFuB7rgy+58pQlHuuqD4CMzNbXaXVCMzMLI8LAjOzCtcqCwJJB0p6WdI8SaPr2C9JY9L9syTt0BJxNqcM9/y99F5nSXpK0qCWiLM5FbrnnHw7SVqZrppX1rLcs6ShkmZKmiNp9UV3y0yGf9udJN0l6fn0nst6FmNJ10l6V9LsevY3/+dXRLSqH5Ipr18FNgfaAc8DA/PyDAP+RrJC2reAqS0ddwnueVegS/r6oEq455x8fyeZ8nxES8ddgr9zZ5J1wXun6R4tHXcJ7vnnwEXp6+7Av4B2LR37GtzzHsAOwOx69jf751drrBHsDMyLiPkRsRyYBByWl+cw4E+ReAboLGmTUgfajArec0Q8FRHvp8lnSFaDK2dZ/s4APwFuA94tZXBFkuWejwZuj4jXASKi3O87yz0H0FHJog4bkhQEK0obZvOJiMdI7qE+zf751RoLgp7AGznphem2xuYpJ429nx+QfKMoZwXvWVJP4AhgbAnjKqYsf+cBQBdJj0iaIem4kkVXHFnu+Q/AN0mWuX0BOCMivixNeC2i2T+/WuN6BHUt9ZQ/RjZLnnKS+X4k7UVSEOxW1IiKL8s9XwacExErW8kKYFnueV1gR2AfoAPwtKRnIuKVYgdXJFnu+QBgJrA30A94QNLjEfFhkWNrKc3++dUaC4KFwGY56V4k3xQam6ecZLofSdsB44GDImJJiWIrliz3XAVMSguBjYFhklZExB0libD5Zf23/V5EfAx8LOkxYBBQrgVBlns+Ebgwkgb0eZIWAFsBz5YmxJJr9s+v1tg0NA3oL6mvpHbAUcCUvDxTgOPS3vdvAUsjYlGpA21GBe9ZUm/gduDYMv52mKvgPUdE34joExF9gFuBH5dxIQDZ/m3fCewuaV1J6wNDgBdLHGdzynLPr5PUgJD0NWBLYH5JoyytZv/8anU1gohYIek04D6SEQfXRcQcSaek+8eSjCAZBswDPiH5RlG2Mt7zuUA34Mr0G/KKKOOZGzPec6uS5Z4j4kVJ9wKzgC+B8RFR5zDEcpDx7/xfwARJL5A0m5wTEWU7PbWkicBQYGNJC4FfA22heJ9fnmLCzKzCtcamITMzawQXBGZmFc4FgZlZhXNBYGZW4VwQmJlVOBcEFSCdeXNmzk+fBvJ+1AzXmyBpQXqtf0japQnnGC9pYPr653n7nlrTGNPz1Lwvs9PZKzsXyD9Y0rAmXGcTSXenr4dKWirpOUkvSvp1E853aM0snJIOr3mf0vQFkvZt7DnruMYEFZitNZ3GIvMQ5PTe786Qr87ZNyVdImnvrNez7FwQVIZPI2Jwzk91Ca55dkQMBkYDVzf24Ig4KSLmpsmf5+3bdc3DA756X7YhmeTr1AL5B5OM326ss4BrctKPR8T2JE8+HyNpx8acLCKmRMSFafJwYGDOvnMj4sEmxLg2mQAcWMf2y0n+PVkzc0FQgSRtKOmh9Nv6C5JWm7Uz/Rb7WM435t3T7ftLejo99hZJGxa43GPAFumxZ6Xnmi3pzHTbBpL+qmQu+dmSRqbbH5FUJelCoEMax43pvo/S3zfnfkNPv8V+R1IbSRdLmqZkvvYfZXhbniaduEvSzkrWbHgu/b1l+lTrBcDINJaRaezXpdd5rq73MfUd4N78jek0EDOAfmlt45k03smSuqSxnC5pbrp9UrrtBEl/kLQrcChwcRpTv5pv8pIOkvSXnPdmqKS70teN+htKOje9x9mSxkmrTNx0TPoezZa0c5o/6/tSp/pm34yI14Bukr7emPNZBqWaY9s/LfcDrCSZlGsmMJnkifKN0n0bkzyhWPNw4Ufp7/8EfpG+bgN0TPM+BmyQbj8HOLeO600gnfsf+C4wlWQitBeADUimCp4DbE/yIXlNzrGd0t+PAFW5MeXkqYnxCOD69HU7khkZOwAnA79Mt68HTAf61hHnRzn3dwtwYJreCFg3fb0vcFv6+gTgDznH/xY4Jn3dmWQ+nw3yrtEXmJGTHgrcnb7uBlQDW5M8Cbxnuv0C4LL09VvAejXXyI8j973OTad/49dz/lZXAcc08W/YNWf7DcDwnL/RNenrPUjnz6/vfcm79yqSp57r+zfbhzrm4yepWX2npf9PtbafVjfFhNXp00iaaQCQ1Bb4raQ9SKYh6Al8DXg755hpwHVp3jsiYqakPUmaIZ5MvxS2I/kmXZeLJf0SWEwy2+k+wORIvgUj6XZgd5JvypdIuojkQ+LxRtzX34AxktYjaUp4LCI+lbQ/sF1OG3cnoD+wIO/4DpJmknzozAAeyMl/vaT+JLM6tq3n+vsDh0r6aZpuD/Rm1bl9Nknfg1y7S3qO5L2/kGQSsc4RUbOa2PUkBRMkBcSNku4A7qgnjtVEMjXDvcBwSbcCBwM/AxrzN6yxl6SfAesDXUkK8bvSfRPT6z0maSMl/Sz1vS+58U0HTsp6PzneBTZtwnHWABcElel7JCs57RgRX0iqJvnPWiv9j70HyQfIDZIuBt4HHoiIURmucXZE3FqTUD0dmBHxStpGPgz4H0n3R8QFWW4iIj6T9AjJNMQjST+USOab+UlE3FfgFJ9GxGBJnYC7SfoIxpDMXfNwRByhpGP9kXqOF8m305cbugZ57y1JH8EhtSdJrl+fg0m+bR8K/ErS1g3kzXczyT39C5gWEcvSZp2sf0MktQeuJKmdvSHpPFa9n/w5aoJ63hclE8KtqfYk76k1I/cRVKZOwLtpIbAX8I38DJK+kea5BriWZOm8Z4BvS6pp819f0oCM13wMODw9ZgOSZp3HJW0KfBIRfwYuSa+T74u0ZlKXSSSTbu1OMjEZ6e9/rzlG0oD0mnWKiKXA6cBP02M6AW+mu0/IybqMpImsxn3AT2razCVtX8fpXyGpcdQrvf77SvthgGOBRyWtA2wWEQ+TfJvvTNKslis/plyPkLyfPyQpFKDxf8OaD/330r6E/JFENX06u5HMgrmUbO9LUw0AynYSvbWVC4LKdCNQJWk6Se3gpTryDAVmpk0Y3wF+HxGLST4YJ0qaRfKhslWWC0bEP0janZ8l6TMYHxHPAdsCz6ZNNL8AflPH4eOAWUo7i/PcT/KN+cFIljKEZM2FucA/lAxBvJoCtd80ludJpjn+X5LayZMk/Qc1HgYG1nQWk9Qc2qaxzU7T+ef9GHi15oO3AceTNKfNIhmddEF67T8rmVXzOeB3EfFB3nGTgLPTTtl+eddeSVLTOSj9TWP/hun1riHp37mDpMkw1/tKhvOOJWkChAzvi5KBAOPruqaS2TefBraUtFDSD9LtbUkGHkyvL15rGs8+alZkko4gaYb7ZUvHUs7S93GHiPhVS8fS2riPwKzIImKypG4tHUcrsC7wfy0dRGvkGoGZWYVzH4GZWYVzQWBmVuFcEJiZVTgXBGZmFc4FgZlZhfv/doME+joWaI0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import RocCurveDisplay\n",
    "\n",
    "display = RocCurveDisplay.from_estimator(clf, X_test, y_test, name=\"RF Model\")\n",
    "_ = display.ax_.set_title(\"ROC Curve\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAuwElEQVR4nO3deZgU1bnH8e/LsA7bIIvKJjiAiuyOCwIB3HBBUaOyJBqNG0aj11yJ3iQuJLmJ0auJRhI0iBpjwN0AF3EBUbwuLIooIIgIMi6Ag+zrDO/9o2omw9g9XTNM99DTv8/zzENXndNV7xmg365zqs4xd0dERDJXreoOQEREqpcSgYhIhlMiEBHJcEoEIiIZTolARCTDKRGIiGQ4JQJJG2b2AzN7OUK98WZ2aypiSgUzW2Vmp4Sv7zCzf1R3TFKzKBFIlQg/rHaY2VYzW2tmj5hZo6o8h7s/4e6nRag32t1/U5XnLmZmbmbbwnZ+YWb3mllWMs5VGWbWxMz+ZGafhzGuCLdbVHdscuBSIpCqdLa7NwL6AMcCvypbwcxqpzyqqtczbOdAYDjw42qOBwAzqwvMBI4GTgeaACcCBcBxlTheTfi7kgiUCKTKufsXwItANyj5Fn2tmX0CfBLuG2pmC81so5m9ZWY9it9vZu3M7DkzW29mBWb2QLj/UjN7M3xtZvZHM1tnZpvMbJGZFZ/vUTP7banjXRl+M95gZlPMrHWpMjez0Wb2iZl9a2bjzMwitnMF8H9Ar1LHq0y7cs1sVrjvGzN7wsxyKvhrB7gEaA+c5+5L3H2vu69z99+4+/RS7e1UKqaS35WZDTKzfDO72cy+Bh4xs6VmNrRU/dphjH3C7RPCdm40sw/MbFAl4pZqpkQgVc7M2gFnAu+X2n0ucDzQNfwQmQhcDTQHHgSmmFm9sJtlGrAa6AC0ASbHOM1pwPeALkAOwTfzghixnAT8HrgIODQ8btnjDSW4gukZ1hsSsZ1HAgOAFeF2ZdtlYYytgaOAdsAdUWIo4xRghrtvrcR7ix0CHAQcBlwFTAJGliofAnzj7u+ZWRvgf4Hfhu+5CXjWzFrux/mlGigRSFV6wcw2Am8CrwO/K1X2e3ff4O47gCuBB939XXcvcvfHgF3ACQRdGK2BMe6+zd13uvubMc61B2gMHAmYuy91969i1PsBMNHd33P3XcB/AX3NrEOpOne6+0Z3/xx4jVLf8ON4z8y2AUuB2cBfwv2Vape7r3D3V9x9l7uvB+4l6HaqqOZArN9BRewFbg9j2QH8EzjHzLLD8lHhPoAfAtPdfXp49fEKMJ/gS4CkESUCqUrnunuOux/m7j8JP0iKrSn1+jDgP8PuhI1h8mhH8EHZDljt7oXlncjdZwEPAOOAtWb2kJk1iVG1NcG38OL3bSW4cmhTqs7XpV5vBxoBmNnicMB1q5kNKFWnT1hnOMFVTsP9aZeZtTKzyeHg82bgH0BlBncLCK569sd6d99ZvBF2fy0Fzg6TwTn8OxEcBlxYpr39qyAGSTElAkmV0tPcrgH+O0waxT/Z7j4pLGsfZaDS3e9392MIBke7AGNiVPuS4AMLADNrSPDN+YsIxz/a3RuFP3PKlLm7PwW8Ddy2n+36PcHvp4e7NyH4ph1pnKKMV4EhYRvj2Q5kl9o+pEx5rOmIi7uHhgFLwuQAQZseL9Pehu5+ZyVil2qkRCDV4W/AaDM7Phz0bWhmZ5lZY2AuQffGneH++mbWr+wBzOzY8P11gG3ATqAoxrn+CVxmZr3MrB5Bd9W77r6qitpyJ3CVmR2yH+1qDGwFNob97rESWhSPE3w4P2tmR5pZLTNrbma/MLPi7pqFwCgzyzKz04nWBTWZYEzmGv59NQDBlcvZZjYkPF79cMC5bSXjl2qiRCAp5+7zCfrTHwC+JRhsvTQsKwLOBjoBnwP5BF0wZTUh+OD9lqDrpwD4nxjnmgncCjxL8EGcC4yowrZ8SDAeMmY/2jWWoLtpE8Hg63OVjGUXwYDxx8ArwGaCBNQCeDesdkMYx0aC8ZMXIhz3K4IrnxOBJ0vtX0NwlfALYD1BEhqDPlfSjmlhGhGRzKbMLSKS4ZQIREQynBKBiEiGUyIQEclwaTepVIsWLbxDhw7VHYaISFpZsGDBN+4ec/qPtEsEHTp0YP78+dUdhohIWjGz1fHK1DUkIpLhlAhERDKcEoGISIZTIhARyXBKBCIiGS5picDMJlqwjOBHccrNzO63YAnBRcVL34mISGol84rgUYIFtOM5A+gc/lwF/DWJsYiISBxJSwTu/gawoZwqw4C/hwt8vAPkmFnSVjZa9vUW7n15Gd9s3ZWsU4iIpKXqHCNow77LF+az7/KBJczsKjObb2bz169fX6mTrVi3lftnrWDDtt2Ver+ISE1VnYkg1lJ8MRdHcPeH3D3P3fNatoz5hLSIiFRSdSaCfIIFvYu1JVhfVkREUqg6E8EU4JLw7qETgE3hkngiIpJCSZt0zswmAYOAFmaWD9wO1AFw9/HAdOBMgnVdtwOXJSsWERGJL2mJwN1HJih34NpknV9ERKLRk8UiIhlOiUBEJMMpEYiIZDglAhGRDKdEICKS4ZQIREQynBKBiEiGUyIQEclwSgQiIhlOiUBEJMMpEYiIZDglAhGRDKdEICKS4ZQIREQynBKBiEiGUyIQEclwSgQiIhlOiUBEJMMpEYiIZDglAhGRDKdEICKS4ZQIREQynBKBiEiGUyIQEclwSgQiIhlOiUBEJMMpEYiIZDglAhGRDFc7SiUzawX0A1oDO4CPgPnuvjeJsYmISAqUmwjMbDBwC3AQ8D6wDqgPnAvkmtkzwD3uvjnJcYqISJIkuiI4E7jS3T8vW2BmtYGhwKnAs0mITUREUqDcMQJ3HxMrCYRlhe7+grvHTQJmdrqZLTOzFWZ2S4zypmY21cw+MLPFZnZZxZsgIiL7o9KDxYk+tM0sCxgHnAF0BUaaWdcy1a4Flrh7T2AQcI+Z1a1sTCIiUnH7c9fQ2ATlxwEr3H2lu+8GJgPDytRxoLGZGdAI2AAU7kdMIiJSQYkGixfFKwIOTnDsNsCaUtv5wPFl6jwATAG+BBoDw2PdiWRmVwFXAbRv3z7BaUVEpCISDRYfDAwBvi2z34C3ErzXYuzzMttDgIXASUAu8IqZzSl7F5K7PwQ8BJCXl1f2GCIish8SJYJpQCN3X1i2wMxmJ3hvPtCu1HZbgm/+pV0G3OnuDqwws8+AI4G5CY4tIiJVJNFdQ5e7+5txykYlOPY8oLOZdQwHgEcQdAOV9jlwMoCZHQwcAayMEriIiFSNSE8WV4a7F5rZdcBLQBYw0d0Xm9nosHw88BvgUTP7kKAr6WZ3/yZZMYmIyHclLREAuPt0YHqZfeNLvf4SOC2ZMYiISPk06ZyISIZTIhARyXCRu4bM7CF3vyredqbYuH03e4ri38Fay6B5o3opjEhEZP9UZIzgwQTbNd6Mj75i9D/eS1jvjrO7cmm/jimISERk/0VOBO6+oLztTLB28y4A/uuMI8muF/tXd8eUxXwd1hMRSQeJppiYynefBi7h7udUeURp4MK8dhzUMPbceL+ZtiTF0YiI7J9EVwT/k5IoRESk2pSbCNz99eLXZtYAaO/uy5IelYiIpEyk20fN7GyCyeFmhNu9zKzsdBEiIpKGoj5HcAfB+gIbAcJJ6DokIyAREUmtqImg0N03JTUSERGpFlFvH/3IzEYBWWbWGbiexOsRiIhIGoh6RfBT4GhgFzAJ2Az8R5JiEhGRFIp0ReDu24Ffmtkfgk3fktywREQkVaLeNXRsuGbAIuBDM/vAzI5JbmgiIpIKUccIHgZ+4u5zAMysP/AI0CNZgYmISGpEHSPYUpwEAMLlK9U9JCJSAySaa6hP+HKumT1IMFDswHBgdnJDExGRVEjUNXRPme3bS72OPym/iIikjURzDQ1OVSAHgqfmrWHeqg1xyz9ZtzWF0YiIpEZFVig7i+BZgvrF+9z918kIqrrcN/MTNmzbTbPsOnHr9GmfQ+P6FVnPR0TkwBbpE83MxgPZwGBgAnABMDeJcVWbs3ocyv9c2LO6wxARSZmodw2d6O6XAN+6+1igL9AueWGJiEiqRE0EO8I/t5tZa2APoEV5RURqgKid3dPMLAe4G3iP4I6hCckKSkREUifqXEO/CV8+a2bTgPqallpEpGZI9EDZ+eWU4e7PVX1IIiKSSomuCM4up8wBJQIRkTSX6IGyy1IViIiIVI+odw2JiEgNldREYGanm9kyM1thZrfEqTPIzBaa2WIzez2Z8YiIyHclba4EM8sCxgGnAvnAPDOb4u5LStXJAf4CnO7un5tZq2TFIyIisUVdoSzbzG41s7+F253NbGiCtx0HrHD3le6+G5gMDCtTZxTwnLt/DuDu6yoWvoiI7K+oXUOPECxc3zfczgd+m+A9bYA1pbbzw32ldQGamdlsM1tgZpfEOpCZXWVm881s/vr16yOGLCIiUURNBLnufhfB1BK4+w7AErwnVnnZNQxqA8cAZwFDgFvNrMt33uT+kLvnuXtey5YtI4YsIiJRRB0j2G1mDQg/yM0sl+AKoTz57DsxXVvgyxh1vnH3bcA2M3sD6AksjxiXiIjsp6hXBHcAM4B2ZvYEMBP4eYL3zAM6m1lHM6sLjACmlKnzL2CAmdU2s2zgeGBp1OBFRGT/RZ1r6GUzWwCcQNDlc4O7f5PgPYVmdh3wEpAFTHT3xWY2Oiwf7+5LzWwGsAjYC0xw94/2oz0iIlJBURemmUKwcP2UsBsnEnefDkwvs298me27CWY1FRGRahC1a+geYACwxMyeNrMLzKx+ojeJiMiBL2rX0OvA6+FDYicBVwITgSZJjE1ERFKgIovXNyCYjXQ40Ad4LFlBiYhI6kQdI3iS4I6eGQTTRsx2973JDExERFIj6hXBI8Aody9KZjAiIpJ6iVYoO8ndZwHZwDCzfR8W1gplIiLpL9EVwUBgFrFXKtMKZSIiNUCiFcpuD1/+2t0/K11mZh2TFpWIiKRM1OcIno2x75mqDERERKpHojGCI4GjgaZmdn6poiaAHigTEakBEo0RHAEMBXLYd5xgC8FDZSIikuYSjRH8C/iXmfV197dTFJOIiKRQoq6hn4cL0owys5Fly939+qRFJiIiKZGoa6h4bYD5yQ4k2dZt2QnANf9YQJeDG8esU7At0Vo7IiI1T6KuoanhnyXzCplZLaCRu29OcmxVav6qbwH4dP02smrFXmWz/UHZ9O/UIpVhiYhUu6hzDf0TGA0UAQsI7iK6N1xLIK08MKo3Q3u0ru4wREQOGFGfI+gaXgGcS7DQTHvg4mQFJSIiqRM1EdQxszoEieBf7r6HcCF7ERFJb1ETwYPAKqAh8IaZHQak1RiBiIjEFnWFsvuB+0vtWm1mg5MTkoiIpFKkKwIza2pm95rZ/PDnHoKrAxERSXNRu4YmEkwrcVH4s5lgsRoREUlzUVcoy3X375faHmtmC5MQj4iIpFjUK4IdZta/eMPM+gE7khOSiIikUtQrgtHA382sabj9LfCj5ISU3nYX7mX865/y6fqtcesccXBjbhpyRAqjEhGJL2EiMLPeQC4wAvgCIN2ml6gO+d/GvmBav2UnM5euVSIQkQNGotlHbwN+SDCtxF3A7939b6kILN29eMOAmPvveXkZ415bkeJoRETiS3RFMBzo5e7bzaw5MANQIhARqUESDRbvdPftAO5eEKG+iIikmURXBLlmNiV8bWW2cfdzkhaZiIikRKJEMKzM9v8kKxAREakeiRameX1/Dm5mpwP3AVnABHe/M069Y4F3gOHu/sz+nFNERCqm3D5/M5tqZmeHU1CXLTvczH5tZj+O894sYBxwBtAVGGlmXePU+wPwUmUaICIi+ydR19CVwM+AP5nZBmA9UB/oAHwKPODu/4rz3uOAFe6+EsDMJhN0NS0pU++nwLPAsZVpQE1TWLSXC8a/zRcby39w+8Tc5tw3oneKohKRmixR19DXwM+Bn5tZB+BQgqkllhffTVSONsCaUtv5wPGlK5hZG+A84CTKSQRmdhVwFUD79u0TnDa9bd9TxMI1G+ndPocjD2kSs867nxXwzsqCFEcmIjVV1CkmcPdVBIvTRBVrhfiyq5r9CbjZ3YvMYi8oH577IeAhgLy8vIxYGe2s7odyxYDDY5bd8uwiXlu2LsURiUhNFTkRVEI+0K7UdlvgyzJ18oDJYRJoAZxpZoXu/kIS46pWry1bx16H7931WszyLTv3ADBv1Ya4iUBEpColMxHMAzqbWUeCOYpGAKNKV3D3jsWvzexRYFpNTgIAH30RTNN0zGHNYpYXbNvNG8vX82H+plSGJSIZLGmJwN0Lzew6gruBsoCJ7r7YzEaH5eOTde508MfhvWLu/2LjDvrdOSu1wYhIRouUCML1B+4ADgvfY4C7e7l9F+4+HZheZl/MBODul0aJ5UB3fu82XHRsu8QVRUQOEFGvCB4GbiSYhbQoeeGkv3vjfNMXETlQRU0Em9z9xaRGIiIi1SJqInjNzO4GngN2Fe909/eSElUGy2kQPMTdv3OLao5ERDJF1ERQ/CBYXql9TvAgmFShhvVqs+rOs6o7DBHJIJESgbsPTnYgIiJSPSItNGNmTc3sXjObH/7cU2ohexERSWNRVxybCGwBLgp/NgOPJCsoERFJnaiJINfdb3f3leHPWEDzH1TCL888qrpDEBHZR9REsMPM+hdvhA+YlT9PssR05fcO12CwiBxQot41dA3wWDguYMAG4NJkBSUiIqkT9a6hhUBPM2sSbm9OZlAiIpI65SYCM/uhu//DzH5WZj8A7n5vEmMTEZEUSHRF0DD8s3GyAxERkeqRaKnKB8M/x6YmHBERSbWoD5TdZWZNzKyOmc00s2/M7IfJDk5ERJIv6u2jp4UDxEMJlqDsAoxJWlQiIpIyURNBnfDPM4FJ7r4hSfGIiEiKRX2OYKqZfUzwENlPzKwlsDN5YYmISKpEuiJw91uAvkCeu+8BtgHDkhmYiIikRqLnCE5y91lmdn6pfaWrPJeswEREJDUSdQ0NBGYBZ8coc5QIRETSXqLnCG4P/7wsNeGIiEiqRRosNrPfAXe5+8Zwuxnwn+7+qyTGJnFs213E2s27ePvTgrh16tWpRa+2OdSqZXHriIhA9LuGznD3XxRvuPu3ZnYmoERQDaZ+8CUAI//2Trn1JlySxyldD05FSCKSxqImgiwzq+fuuwDMrAFQL3lhSRSTrjwh5v6vNu3gZ099wLbdhSmOSETSUdRE8A9gppk9QjBI/GPgsaRFJZH0zW0ec//K9VuB4MqhzF1e++jWugmHt2yUlNhEJH1EXY/gLjNbBJxCsDDNb9z9paRGJpX25cbgWb9Xl67j1aXr4tbr0z6H537SL1VhicgBKuoVAcBSoNDdXzWzbDNr7O5bkhWYVN6uwiIAjm7dhPtG9I5Z51cvfMiWneo6EpHodw1dCVwFHATkAm2A8cDJyQtN9tfBTerTqVXsrp9G9WorEYgIEP2K4FrgOOBdAHf/xMxaJS0qKdfogbk8917+fh1jx54iFn+5mY++2BS3TsN6tenYomHcchGpGaImgl3uvrt44NHMahMMGpfLzE4H7gOygAnufmeZ8h8AN4ebW4Fr3P2DiDFlrFvOOJJbzjhyv47xfyuCZxCG/vnNcutN+2l/urVpul/nEpEDW9RE8LqZ/QJoYGanAj8Bppb3BjPLAsYBpxKsYTDPzKa4+5JS1T4DBobPJZwBPAQcX9FGyL76dWoBwH+c0jlh3YcuPibm/k/WbeXul5axeceeKo1NRA48URPBzcAVwIfA1cB0YEKC9xwHrHD3lQBmNplgxtKSRODub5Wq/w7QNmI8Uo76dbJYdedZkeqedvQhMfc3bRD/qWURqVkSJgIzqwUscvduwN8qcOw2wJpS2/mU/23/cuDFODFcRTBYTfv27SsQgoiIJJJwPQJ33wt8YGYV/QSO9SRTzHEFMxtMkAhujlXu7g+5e56757Vs2bKCYYiISHmidg0dCiw2s7kEi9IA4O7nlPOefKBdqe22wJdlK5lZD4JupjPcXf0RKRK160hEar6oiWBsJY49D+hsZh2BL4ARwKjSFcKrjOeAi919eSXOISIi+ynRCmX1gdFAJ4KB4ofdPdJTSO5eaGbXAS8R3D460d0Xm9nosHw8cBvQHPhLeGtqobvnVbYxIiJScYmuCB4D9gBzgDOArsANUQ/u7tMJ7jAqvW98qddXENyNJCIi1SRRIujq7t0BzOxhYG7yQxIRkVRKdNdQydNEUbuEREQkvSS6IuhpZpvD10bwZPHm8LW7e5OkRifVZlP4RPGPHplL39wWcet9v08bhvVqk6qwRCQJEi1en5WqQOTAsuzrYIbxPUUed5qJ5Wu3YKBEIJLmKrIegWSg6wZ34qYhR8Qs6/hf/8vry9czYc7KmOX53+7g0bdWkXdYM8pZKI3L+x/O6d1iT3UhIsmnRCAxNa5fe58/Y/HwOfHf/u/Sco+1fO2WuDOYzl/9La8uXRs3ERQW7eXDLzZRuDf+ZLe1DLq1aUq92rqAFakMJQKJ6ZK+HVi2dgtXDDg8Yd0P7zgt5v4bJi9k1sfr+N353Rnao3XMOv3unFXusV9Y+CU3PZ14ZvIxQ47g2sGdEtYTke9SIpCYatUyfn9+j0h1G9evE3P/6IG5zPp4Hf3KGWz+YuMOnlmQzydrY696+kF+sHDOA6N6k9Ogbsw6lz4yt9zV1tyd/G93sNfjX1Vk1TLa5DTAyuvDEqmhlAik0qZc16/c7pjjOh4UeU6jZg1jf8gXG3xEKxrWi/3PtVat8j+8H31rFWOnLim3DkDLxvU4r3f8ge+ebXM4q8ehCY8jkm6UCKTSerTNqbJjPXrZcZV+7+7CvYx//VMWrvk2ZvmaDTsAuPeinjHLv9m6i99N/5j1W3bx+NurY9bZsacIgN9NbxA3jnp1avGXH/ThyEN0V7WkFyUCqVa/Ousouraumg/OeOPJbZo14KQjW3F+n9jrHm3avoffTf+Yfp2a88QVJ8Ss0+GW/wXghMObxyzfsnMPLy9Zy7KvtygRSNpRIpBqFWUwOop2BzXgqav7Vuq9TbPrMPW6/nQ+uFHcOhflteWp+fncE+eqYuX6rby8ZG2lzi9S3ZQIJO0t+NUpcQeso+reNvbtrcXuuqAnd10QOwnAv7uObpi8kL/O/jRuvdO7HcJ/nNKlckGKJIkSgaS95o3qVXcIfL1pZ8nr9gdlx6zz8pK1fPz1Fp549/O4x2lYN4t/XHE8bZvFPoZIMigRiFSBAZ2DJVRfvvF7dDm4ccw6xeMMpxx1cMzy9Vt28urSdXxesF2JQFJKiUCkCtStXSvhrbJ/Gt6Lb7buijsu8u7KAl5dui4Z4YmUS4lAJEXOLecZBYDtu4Nxhv94ciEn5sa+O+mFhcGy38d1PCjucXIa1OH+kb2pX0dTbkg0SgQiB4ilXwczvq/bsov312wst268Z+gKtu5m7mcbeGP5+rhdVOu27KJor5MV5yBFe521m3fStln8ZybMjO5tmlK3dqIlTSQdKBGIHCAuPbEDf539KW//18k0ivMU9XPv5dOwXm2GHB17kr5fvfAhn6zbylWPL0hmqIDmd6pJlAhEDhDZdWvz4R1Dyq0T76G4Ysd3bM4/3vmc/zy1C20Piv2N/sYnPyC7bhZ/uyQvZvlT89fwr4Vf8vCP8uJ+47/skXnlzu8k6UWJQKQGObtna87uGXum12Ln9S4/mfTr1IL7RvQut06i+Z0kvdSIRLBnzx7y8/PZuXNn3Dqjjsji3I6H0tw3sHTpphRGJweq+vXr07ZtW+rU2b+H0UTSXY1IBPn5+TRu3JgOHTrEnUZ4dcE2Nu3YQ/uDssnJLn+mS6n53J2CggLy8/Pp2LFjdYeTdhJN9Ldh225WF2zntrO7xj3GC+9/Qfc2OeUuftSjbVNOjvPchVSdGpEIdu7cWW4SECnLzGjevDnr16+v7lDSWryJ/pav3QrAL5//qNz3z1sVO5EUa920vhJBCtSIRAAoCUiF6d9M5X1w+2lk182iTlbsweSlX23m72+v5sZTOsc9xnPvf8HwvHZx16IY8dDbvLNyA3+ZvSLuMVZ/s53Tjj447loVALktG9GycfVPQ3IgqzGJQERSp2mD8sdVjjq0Cb8/v3u5dUYPzC23/J2VGwC4a8aycus9OX9NueUQ3JobT3bdLK47qRPZdTP34zBzW17FsrKy6N69O4WFhXTs2JHHH3+cnJwcVq1axVFHHcURRxxRUnfu3LnUrfvvb0GzZ89m8ODBTJgwgcsvvxyA999/nz59+nD33Xdz0003RYph1apVDB06lI8+in85Xl6dr776iiuvvJJp06aV7Lvhhht45plnWLNmDbVqBd/+Hn30UcaMGUObNm3YvXs3N954I1deeWWkGOP57LPPGDFiBBs2bKBPnz48/vjj+/yOip1++um888479O/ff584BwwYwJYtwXKX69at47jjjuOFF15g2rRpzJs3j7Fjx+5XfJJ6b4wZzMpvtsZdAwLgzPvncOMpXWjeKPZVxai/vQvA8+9/EbO8sGgv23YX0b1NU/rGeZobgsRXk68glQiqSIMGDVi4cCEAP/rRjxg3bhy//OUvAcjNzS0pi6d79+48+eSTJYlg8uTJ9OwZf9rjZLj33nv3+UDfu3cvzz//PO3ateONN95g0KBBJWXDhw/ngQceYN26dRx99NGcc845HHxw5ftyb775Zm688UZGjBjB6NGjefjhh7nmmmu+U2/MmDFs376dBx98cJ/9c+bMKXn9/e9/n2HDhgFw1llnceutt3LzzTeTna2J3NJJ++bZtG9e/t/ZrP8cVG75R2OHUFTkNM2OfQXz1Lw1/PzZRVzzxHvlHmfkce0TXuGksxqXCMZOXcySLzd/Z/+uwiIKi5x6dbKoXcF7oLu2bsLtZx8duX7fvn1ZtGhRhc7Rvn17Nm/ezNq1a2nVqhUzZszgzDPPLClfuHAho0ePZvv27eTm5jJx4kSaNWvGggUL+PGPf0x2djb9+/cvqV9UVMQtt9zC7Nmz2bVrF9deey1XX311uTE8++yz/Pa3vy3Zfu211+jWrRvDhw9n0qRJ+ySCYq1atSI3N5fVq1dXOhG4O7NmzeKf//wnECTSO+64I2YiOPnkk5k9e3bcY23ZsoVZs2bxyCOPAME4wKBBg5g2bRoXXXRRpeKT9BXvCe1iZ/Y4lJ8/u4ifndol7t1LY6cuYdLcz5k0N/704RB0hx3UMHbCWbRmE8cffhAnHRn7/8iOPUUs/3oL/Tu3KPccuS0bVdmKfqXVuERQ3YqKipg5c2bJN3uATz/9lF69egHQr18/xo0bF/O9F1xwAU8//TS9e/emT58+1Kv37wGuSy65hD//+c8MHDiQ2267jbFjx/KnP/2Jyy67rGT/mDFjSuo//PDDNG3alHnz5rFr1y769evHaaedFvfy9rPPPqNZs2b7nHPSpEmMHDmSYcOG8Ytf/II9e/Z85577lStXsnLlSjp12neqgWXLljF8+PCY55o9ezY5OTkl2wUFBeTk5FC7dvDPsW3btnzxRexL+USef/55Tj75ZJo0+fd/lry8PObMmaNEIN/RqF7thLPGLsrfxCtL1nJ5/9i3Gb+2bB2L8jexc08Ru/bEnuhvy65CXl26LuHssonGO0YPzFUiiCLeN/dkP0ewY8cOevXqxapVqzjmmGM49dRTS8qidA0BXHTRRQwfPpyPP/6YkSNH8tZbbwGwadMmNm7cyMCBA4HgG/OFF174nf0XX3wxL774IgAvv/wyixYt4plnnik5xieffEKXLrFXx/rqq69o2bJlyfbu3buZPn06f/zjH2ncuDHHH388L7/8MmedFfynefLJJ3nzzTepV68eDz74IAcdtO9smEcccUSkNkNwRVBWZftjJ02axBVXXLHPvlatWvHll19W6ngifxzeq9zyG09NvOLcyvVb2V20l2ZxPnt27C7i7ZUFHNuhWbnHSdYzUElNBGZ2OnAfkAVMcPc7y5RbWH4msB241N3L76w7QBWPEWzatImhQ4cybtw4rr/++god45BDDqFOnTq88sor3HfffSWJIB53j/uB6e78+c9/ZsiQfeeuWbVqVdz4Sz+ZPWPGDDZt2kT37kG/6Pbt28nOzi5JBMVjBPFU5IqgRYsWbNy4kcLCQmrXrk1+fj6tW5c/TUIsBQUFzJ07l+eff36f/Tt37qRBg/gzaYok2+Et46+HXaxDi4YpiCS2pCUCM8sCxgGnAvnAPDOb4u5LSlU7A+gc/hwP/DX8M201bdqU+++/n2HDhsXs407k17/+NevWrSMr69+XmE2bNqVZs2bMmTOHAQMG8PjjjzNw4EBycnJo2rQpb775Jv379+eJJ54oec+QIUP461//ykknnUSdOnVYvnw5bdrEnw+/S5cu+ySJSZMmMWHCBEaOHAnAtm3b6NixI9u3b4/UjopcEZgZgwcP5plnnmHEiBE89thjJYO9FfH0008zdOhQ6tevv8/+5cuX061btwofTyRTJHMy8eOAFe6+0t13A5OBsv+7hwF/98A7QI6ZHZrEmFKid+/e9OzZk8mTJ1f4vSeeeCLnnnvud/Y/9thjjBkzhh49erBw4UJuu+02AB555BGuvfZa+vbtu8+33iuuuIKuXbvSp08funXrxtVXX01hYfzZIhs2bEhubi4rVqxg+/btvPTSSyXf/ovL+/fvz9SpUyvcpij+8Ic/cO+999KpUycKCgpKxljmz5+/T1fPgAEDuPDCC5k5cyZt27blpZdeKimbPHlySeIq7bXXXtunLSKyL4vVP1slBza7ADjd3a8Ity8Gjnf360rVmQbc6e5vhtszgZvdfX6ZY10FXAXQvn37Y1avXr3PuZYuXcpRRx1Vbjy79hSxbO0WurVpSq0afD/w/nj++edZsGDBPncOpbu1a9cyatQoZs6cGbM8yr8dkZrAzBa4e8y5x5N5RRDr07Zs1olSB3d/yN3z3D2v9IBmRdSrk0WPtjlKAuU477zz6NChQ3WHUaU+//xz7rnnnuoOQ+SAlszB4nygXanttkDZWzei1JEUKnvHTbo79thjqzsEkQNeMq8I5gGdzayjmdUFRgBTytSZAlxigROATe7+VWVOlqwuLqm59G9GJJC0KwJ3LzSz64CXCG4fnejui81sdFg+HphOcOvoCoLbRy+rzLnq169PQUEBzZs3r9HzgUjVKV6PoOwdRiKZKGmDxcmSl5fn8+fvM5YcaYUykbK0QplkkvIGi2vEk8V16tTRKlMiIpWUzDECERFJA0oEIiIZTolARCTDpd1gsZmtB1YnrBhbC+CbKgwnHajNmUFtzgz70+bD3D3mE7lplwj2h5nNjzdqXlOpzZlBbc4MyWqzuoZERDKcEoGISIbLtETwUHUHUA3U5sygNmeGpLQ5o8YIRETkuzLtikBERMpQIhARyXA1MhGY2elmtszMVpjZLTHKzczuD8sXmVmf6oizKkVo8w/Cti4ys7fMrGd1xFmVErW5VL1jzawoXDUvrUVps5kNMrOFZrbYzF5PdYxVLcK/7aZmNtXMPgjbXKlZjA8UZjbRzNaZ2Udxyqv+88vda9QPwZTXnwKHA3WBD4CuZeqcCbxIsELaCcC71R13Ctp8ItAsfH1GJrS5VL1ZBFOeX1Ddcafg7zkHWAK0D7dbVXfcKWjzL4A/hK9bAhuAutUd+360+XtAH+CjOOVV/vlVE68IjgNWuPtKd98NTAaGlakzDPi7B94Bcszs0FQHWoUSttnd33L3b8PNdwhWg0tnUf6eAX4KPAusS2VwSRKlzaOA59z9cwB3T/d2R2mzA40tWIykEUEiKExtmFXH3d8gaEM8Vf75VRMTQRtgTant/HBfReukk4q253KCbxTpLGGbzawNcB4wPoVxJVOUv+cuQDMzm21mC8zskpRFlxxR2vwAcBTBMrcfAje4+97UhFctqvzzq0asR1BGrCXKyt4jG6VOOoncHjMbTJAI+ic1ouSL0uY/ATe7e1ENWbkuSptrA8cAJwMNgLfN7B13X57s4JIkSpuHAAuBk4Bc4BUzm+Pum5McW3Wp8s+vmpgI8oF2pbbbEnxTqGiddBKpPWbWA5gAnOHuBSmKLVmitDkPmBwmgRbAmWZW6O4vpCTCqhf13/Y37r4N2GZmbwA9gXRNBFHafBlwpwcd6CvM7DPgSGBuakJMuSr//KqJXUPzgM5m1tHM6gIjgCll6kwBLglH308ANrn7V6kOtAolbLOZtQeeAy5O42+HpSVss7t3dPcO7t4BeAb4SRonAYj2b/tfwAAzq21m2cDxwNIUx1mVorT5c4IrIMzsYOAIYGVKo0ytKv/8qnFXBO5eaGbXAS8R3HEw0d0Xm9nosHw8wR0kZwIrgO0E3yjSVsQ23wY0B/4SfkMu9DSeuTFim2uUKG1296VmNgNYBOwFJrh7zNsQ00HEv+ffAI+a2YcE3SY3u3vaTk9tZpOAQUALM8sHbgfqQPI+vzTFhIhIhquJXUMiIlIBSgQiIhlOiUBEJMMpEYiIZDglAhGRDKdEIEkXzvy50Mw+CmeJzKni468ysxbh661x6jQws9fNLMvMOpjZjjCmJWY23swq9H/BzPLM7P7w9SAzO7FU2eiqmNrBzO4ws5sS1Hm0IrOqhm1PeDupmf23ma0p+/s0s+vSfXZP+S4lAkmFHe7ey927EUymdW01xPBjgsnYisLtT929F9AD6AqcW5GDuft8d78+3BxEMLtrcdl4d//7/gZczaYSTPhW1kTg+hj7JY0pEUiqvU04QZaZ5ZrZjHBytDlmdmS4/2Azez6cX/6D4m/bZvZCWHexmV1VwfP+gOCp2324eyHwFtDJzA4zs5nhHO8zw6exMbMLw6uZD8IpG4qvAqaZWQdgNHBjeIUxoPibvJkdZWYl0xyE38YXha+PCa9QFpjZS5Zg9kgzu9LM5oUxPBs+NVzslPD3t9zMhob1s8zs7vA9i8zs6or8stz9nVhPq7r7dmCVmcVKEpKmlAgkZcwsi2AqgOIpAh4CfuruxwA3AX8J998PvO7uPQnmZV8c7v9xWDcPuN7Mmkc8b13gcHdfFaMsO4zpQ4JZLP/u7j2AJ8I4IHgqe0gYzzml3x8eczzwx/CqZ06psqVAXTM7PNw1HHjKzOoAfyZYH+EYgm/Z/52gGc+5+7FhDEsJJg4s1gEYCJwFjDez+mH5Jnc/FjgWuNLMOpZpe2szm57gvLHMBwZU4n1ygKpxU0zIAamBmS0k+MBaQDA7ZCOC7pSn7d8zg9YL/zwJuAQg7MrZFO6/3szOC1+3AzoDUSbPawFsLLMvN4zJgX+5+4tm9jhwflj+OHBX+Pr/CKYweIpgvqaKeAq4CLiTIBEMJ5gLpxvB7wGCqRMSzRXTzcx+S7DwTCOCKRdKzhFOu/yJma0kmHDtNKBHqfGDpgS/r5J5ptz9S4KpCipqXXgOqSGUCCQVdrh7LzNrCkwjGCN4FNgY9tMnZGaDgFOAvu6+3cxmA/Wjnj9G3U8jnNsB3H20mR1P8I17oZlFijn0JEGyey44lH9iZt2Bxe7etwLHeRQ4190/MLNLCcYl9omzzLYRXG2VThiEXVn7qz7B71RqCHUNScq4+yaCgcabCD5IPjOzC6FkHdbidZRnAteE+7PMrAnBN9pvwyRwJMESfVHP+y2QFXaZlOctgtktIRhTeDOMIdfd33X324Bv2HcKYIAtQOM45/4UKAJuJUgKAMuAlmbWNzx+HTM7OkFsjYGvwm6lH5Qpu9DMaplZLsGSjssIrhiuCetjZl3MrGGCc0TVBUjbiezku5QIJKXc/X2CdWdHEHygXW5mHxCMAxQvQXgDMNiC2SQXAEcDM4Da4WDrbwiW26yIl0m8GM/1wGXhOS4O4wC428w+DG+7fCOMv7SpwHnFg8Uxjvsk8EOCbiLCJRcvAP4Qtn0hpe46iuNW4F3gFeDjMmXLgNcJVp0b7e47CdadWAK8F8b9IGV6AMobIzCzuyyY+TLbzPLN7I5Sxf2AVxPEK2lEs49KRjCz3sDP3P3i6o4lnen3WDPpikAyQngl8lp455JUXguCqxOpQXRFICKS4XRFICKS4ZQIREQynBKBiEiGUyIQEclwSgQiIhnu/wEWpEJ656zgiAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import PrecisionRecallDisplay\n",
    "\n",
    "y_prob = clf.predict_proba(X_test)\n",
    "display = PrecisionRecallDisplay.from_predictions(y_test, y_prob[:, 1], name=\"RF Model\")\n",
    "_ = display.ax_.set_title(\"Precision-Recall Curve\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is a ranked list of the most influential features. Among the most important are the community sizes and the shared ids degree and p2p shared card pageRank."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>communitySize</th>\n",
       "      <td>0.106526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>partOfCommunity</th>\n",
       "      <td>0.096765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sharedIdsDegree</th>\n",
       "      <td>0.018892</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p2pSharedCardPageRank</th>\n",
       "      <td>0.016861</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ipDegree</th>\n",
       "      <td>0.008471</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cardDegree</th>\n",
       "      <td>0.004484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>deviceDegree</th>\n",
       "      <td>0.002431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p2pReceivedWeightedPageRank</th>\n",
       "      <td>0.002268</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p2pSentPageRank</th>\n",
       "      <td>0.002031</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p2pReceivedWeightedDegree</th>\n",
       "      <td>0.001319</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    0\n",
       "communitySize                0.106526\n",
       "partOfCommunity              0.096765\n",
       "sharedIdsDegree              0.018892\n",
       "p2pSharedCardPageRank        0.016861\n",
       "ipDegree                     0.008471\n",
       "cardDegree                   0.004484\n",
       "deviceDegree                 0.002431\n",
       "p2pReceivedWeightedPageRank  0.002268\n",
       "p2pSentPageRank              0.002031\n",
       "p2pReceivedWeightedDegree    0.001319"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.inspection import permutation_importance\n",
    "result = permutation_importance(clf, X_train, y_train, random_state=0)\n",
    "pd.DataFrame(abs(result['importances_mean']),index=X_train.columns).sort_values(0, ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Investigating Unlabeled High-Probability Fraud Risk Predictions\n",
    "The labeling from part 2 wasn't perfect. Now that we have trained a machine learning model, investigating user accounts that were predicted as high probability fraud risks despite not being labeled as such by us (ostensible false positives), will bring further insights.\n",
    "\n",
    "The below commands will isolate some cases from the test set so we can visualize in [Neo4j Bloom](https://neo4j.com/product/bloom/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cls</th>\n",
       "      <th>predictedProbability</th>\n",
       "      <th>guid</th>\n",
       "      <th>wccId</th>\n",
       "      <th>communitySize</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3724</th>\n",
       "      <td>0</td>\n",
       "      <td>0.881712</td>\n",
       "      <td>50ad7d70bb174a8f95d06d5067db1bba</td>\n",
       "      <td>2391</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10344</th>\n",
       "      <td>0</td>\n",
       "      <td>0.880321</td>\n",
       "      <td>bf39047c8f385a0a6657217c1b052858</td>\n",
       "      <td>363</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21326</th>\n",
       "      <td>0</td>\n",
       "      <td>0.896839</td>\n",
       "      <td>c4d3c05ebf06a6b7b2f10833a51a0b70</td>\n",
       "      <td>16635</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       cls  predictedProbability                              guid  wccId  \\\n",
       "3724     0              0.881712  50ad7d70bb174a8f95d06d5067db1bba   2391   \n",
       "10344    0              0.880321  bf39047c8f385a0a6657217c1b052858    363   \n",
       "21326    0              0.896839  c4d3c05ebf06a6b7b2f10833a51a0b70  16635   \n",
       "\n",
       "       communitySize  \n",
       "3724               5  \n",
       "10344              7  \n",
       "21326              4  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Retrieve High Probability predictions for non-fraud risk labeled data in the testset\n",
    "y_test_df = y_test.to_frame(name='cls')\n",
    "y_test_df['predictedProbability']=y_prob[:, 1]\n",
    "test_prob_df = y_test_df[(y_test_df.predictedProbability > 0.88) & (y_test_df.cls == 0)] \\\n",
    "    .join(df[['guid','wccId', 'communitySize']])\n",
    "test_prob_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Write back to database for investigation in Bloom\n",
    "for index, row in test_prob_df.iterrows():\n",
    "    gds.run_cypher('''\n",
    "        MATCH(u:User) WHERE u.guid = $guid\n",
    "        SET u.predictedProbability = $predictedProbability\n",
    "    ''', params = row.to_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below are a couple examples from the test set visualized in [Bloom](https://neo4j.com/docs/bloom-user-guide/current/) . The larger, leftmost, green nodes with user icons represent the high probability nodes of interest.\n",
    "\n",
    "Example 1\n",
    "<img src=\"img/predictions-in-bloom-1.png\" width=\"700\" >\\n\n",
    "\n",
    "Example 2\n",
    "<img src=\"img/predictions-in-bloom-2.png\" width=\"700\" >\n",
    "\n",
    "\n",
    "Perhaps unsurprisingly, they exhibit the P2P with shared card behavior, they also have a relatively large number of credit cards (the median degree centrality on cards is 3). This could potentially be a sign of fraud, though it is hard to know on an anonymized dataset like this. This is where subject matter expert review and iteration comes in. If this behavior turns out to be a clear indicator of fraud, it means we are predicting fraud more proactively before chargebacks take place which is the ideal. In this case, if we re-label these users appropriately and re-train our ML model as more data comes in, we will further improve predictive performance. If, on the other hand, it turns out that some of this behavior is benign, we can adjust the feature engineer and model so the ML learns to rule out such cases which will likewise improve predictive performance and increase our understanding of fraud patterns. Either way, it is a win."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean Up\n",
    "This section will help clean all the additional graph elements and properties created in the above workflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# delete created relationships\n",
    "gds.run_cypher('MATCH (:User)-[r:SHARED_IDS]->() DELETE r')\n",
    "gds.run_cypher('MATCH (:User)-[r:P2P_WITH_SHARED_CARD]->() DELETE r')\n",
    "gds.run_cypher('MATCH (:User)-[r:SIMILAR_IDS]->() DELETE r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove created node Labels\n",
    "gds.run_cypher('MATCH (u:FlaggedUser) REMOVE u:FlaggedUser')\n",
    "gds.run_cypher('MATCH (u:FraudRiskUser) REMOVE u:FraudRiskUser')\n",
    "gds.run_cypher('MATCH (u:FraudSharedId) REMOVE u:FraudSharedId')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove created node properties\n",
    "gds.run_cypher('''\n",
    "    MATCH (n)\n",
    "    REMOVE n.wccId,\n",
    "        n.sharedIdsDegree,\n",
    "        n.predictedProbability,\n",
    "        n.partOfCommunity,\n",
    "        n.p2pSharedCardPageRank,\n",
    "        n.p2pSharedCardDegree,\n",
    "        n.p2pSentWeightedPageRank,\n",
    "        n.p2pSentWeightedDegree,\n",
    "        n.p2pSentPageRank,\n",
    "        n.p2pSentDegree,\n",
    "        n.p2pReversedSharedCardPageRank,\n",
    "        n.p2pReversedSharedCardDegree,\n",
    "        n.p2pReceivedWeightedPageRank,\n",
    "        n.p2pReceivedWeightedDegree,\n",
    "        n.p2pReceivedPageRank,\n",
    "        n.p2pReceivedDegree,\n",
    "        n.louvainCommunityId,\n",
    "        n.ipDegree,\n",
    "        n.fraudRiskRatio,\n",
    "        n.fraudRiskDegree,\n",
    "        n.fraudRisk,\n",
    "        n.flaggedRatio,\n",
    "        n.flaggedDegree,\n",
    "        n.deviceDegree,\n",
    "        n.degree,\n",
    "        n.communitySize,\n",
    "        n.cardDegree\n",
    "''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove created relationship properties\n",
    "gds.run_cypher('MATCH ()-[r]->() REMOVE r.inverseDegreeWeight')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}