CS224W_Colab_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XuXWJLEm2UWS"
   },
   "source": [
    "# **CS224W - Colab 2**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8gzsP50bF6Gb"
   },
   "source": [
    "In this Colab, we will construct our own graph neural network by using PyTorch Geometric (PyG) and apply the model on two of Open Graph Benchmark (OGB) datasets. Those two datasets are used to benchmark the model performance on two different graph-related tasks. One is node property prediction, predicting properties of single nodes. Another one is graph property prediction, predicting the entire graphs or subgraphs.\n",
    "\n",
    "At first, we will learn how PyTorch Geometric stores the graphs in PyTorch tensor.\n",
    "\n",
    "We will then load and take a quick look on one of the Open Graph Benchmark (OGB) datasets by using the `ogb` package. OGB is a collection of realistic, large-scale, and diverse benchmark datasets for machine learning on graphs. The `ogb` package not only provides the data loader of the dataset but also the evaluator.\n",
    "\n",
    "At last, we will build our own graph neural networks by using PyTorch Geometric. And then apply and evaluate the models on node property prediction and grpah property prediction tasks.\n",
    "\n",
    "**Note**: Make sure to **sequentially run all the cells in each section**, so that the intermediate variables / packages will carry over to the next cell\n",
    "\n",
    "Have fun on Colab 2 :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZGKqVEbbMEzf"
   },
   "source": [
    "# Device\n",
    "You might need to use GPU for this Colab.\n",
    "\n",
    "Please click `Runtime` and then `Change runtime type`. Then set the `hardware accelerator` to **GPU**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "B0NiFL6OLpaJ"
   },
   "source": [
    "# Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Nwwq0nSdmsOL"
   },
   "source": [
    "# 1 PyTorch Geometric (Datasets and Data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Sf7vUmdNKCjA"
   },
   "source": [
    "PyTorch Geometric generally has two classes for storing or transforming the graphs into tensor format. One is the `torch_geometric.datasets`, which contains a variety of common graph datasets. Another one is `torch_geometric.data` that provides the data handling of graphs in PyTorch tensors.\n",
    "\n",
    "In this section, we will learn how to use the `torch_geometric.datasets` and `torch_geometric.data`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ic-o1P3r6hr2"
   },
   "source": [
    "## PyG Datasets\n",
    "\n",
    "The `torch_geometric.datasets` has many common graph datasets. Here we will explore the usage by using one example dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "zT5qca3x6XpG"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ENZYMES(600)\n"
     ]
    }
   ],
   "source": [
    "from torch_geometric.datasets import TUDataset\n",
    "\n",
    "root = './enzymes'\n",
    "name = 'ENZYMES'\n",
    "\n",
    "# The ENZYMES dataset\n",
    "pyg_dataset= TUDataset('./enzymes', 'ENZYMES')\n",
    "\n",
    "# You can find that there are 600 graphs in this dataset\n",
    "print(pyg_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NLm5vVYMAP2x"
   },
   "source": [
    "## Question 1: What is the number of classes and number of features in the ENZYMES dataset? (5 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "8iF_Kyqr_JbY"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ENZYMES dataset has 6 classes\n",
      "ENZYMES dataset has 3 features\n"
     ]
    }
   ],
   "source": [
    "def get_num_classes(pyg_dataset):\n",
    "    # TODO: Implement this function that takes a PyG dataset object\n",
    "    # and return the number of classes for that dataset.\n",
    "\n",
    "    num_classes = 0\n",
    "\n",
    "    ############# Your code here ############\n",
    "    ## (~1 line of code)\n",
    "    ## Note\n",
    "    ## 1. Colab autocomplete functionality might be useful.\n",
    "    num_classes = pyg_dataset.num_classes\n",
    "    #########################################\n",
    "\n",
    "    return num_classes\n",
    "\n",
    "def get_num_features(pyg_dataset):\n",
    "    # TODO: Implement this function that takes a PyG dataset object\n",
    "    # and return the number of features for that dataset.\n",
    "\n",
    "    num_features = 0\n",
    "\n",
    "    ############# Your code here ############\n",
    "    ## (~1 line of code)\n",
    "    ## Note\n",
    "    ## 1. Colab autocomplete functionality might be useful.\n",
    "    num_features = pyg_dataset.num_features\n",
    "    #########################################\n",
    "\n",
    "    return num_features\n",
    "\n",
    "# You may find that some information need to be stored in the dataset level,\n",
    "# specifically if there are multiple graphs in the dataset\n",
    "\n",
    "num_classes = get_num_classes(pyg_dataset)\n",
    "num_features = get_num_features(pyg_dataset)\n",
    "print(\"{} dataset has {} classes\".format(name, num_classes))\n",
    "print(\"{} dataset has {} features\".format(name, num_features))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rwKbzhHUAckZ"
   },
   "source": [
    "## PyG Data\n",
    "\n",
    "Each PyG dataset usually stores a list of `torch_geometric.data.Data` objects. Each `torch_geometric.data.Data` object usually represents a graph. You can easily get the `Data` object by indexing on the dataset.\n",
    "\n",
    "For more information such as what will be stored in `Data` object, please refer to the [documentation](https://pytorch-geometric.readthedocs.io/en/latest/modules/data.html#torch_geometric.data.Data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7sCV3xJWCddX"
   },
   "source": [
    "## Question 2: What is the label of the graph (index 100 in the ENZYMES dataset)? (5 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "LIis9oTZAfs3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data(edge_index=[2, 168], x=[37, 3], y=[1])\n",
      "Graph with index 100 has label tensor([4])\n"
     ]
    }
   ],
   "source": [
    "def get_graph_class(pyg_dataset, idx):\n",
    "    # TODO: Implement this function that takes a PyG dataset object,\n",
    "    # the index of the graph in dataset, and returns the class/label \n",
    "    # of the graph (in integer).\n",
    "\n",
    "    label = -1\n",
    "\n",
    "    ############# Your code here ############\n",
    "    ## (~1 line of code)\n",
    "    label = pyg_dataset[idx].y\n",
    "    #########################################\n",
    "\n",
    "    return label\n",
    "\n",
    "# Here pyg_dataset is a dataset for graph classification\n",
    "graph_0 = pyg_dataset[0]\n",
    "print(graph_0)\n",
    "idx = 100\n",
    "label = get_graph_class(pyg_dataset, idx)\n",
    "print('Graph with index {} has label {}'.format(idx, label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fKhcVeAhCwoY"
   },
   "source": [
    "## Question 3: What is the number of edges for the graph (index 200 in the ENZYMES dataset)? (5 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "f5m2DOfhBtWv"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Graph with index 200 has 53 edges\n"
     ]
    }
   ],
   "source": [
    "def get_graph_num_edges(pyg_dataset, idx):\n",
    "    # TODO: Implement this function that takes a PyG dataset object,\n",
    "    # the index of the graph in dataset, and returns the number of \n",
    "    # edges in the graph (in integer). You should not count an edge \n",
    "    # twice if the graph is undirected. For example, in an undirected \n",
    "    # graph G, if two nodes v and u are connected by an edge, this edge\n",
    "    # should only be counted once.\n",
    "\n",
    "    num_edges = 0\n",
    "\n",
    "    ############# Your code here ############\n",
    "    ## Note:\n",
    "    ## 1. You can't return the data.num_edges directly\n",
    "    ## 2. We assume the graph is undirected\n",
    "    ## (~4 lines of code)\n",
    "    num_edges = pyg_dataset[idx].num_edges // 2\n",
    "    #########################################\n",
    "\n",
    "    return num_edges\n",
    "\n",
    "idx = 200\n",
    "num_edges = get_graph_num_edges(pyg_dataset, idx)\n",
    "print('Graph with index {} has {} edges'.format(idx, num_edges))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AXa7yIG4E0Fp"
   },
   "source": [
    "# 2 Open Graph Benchmark (OGB)\n",
    "\n",
    "The Open Graph Benchmark (OGB) is a collection of realistic, large-scale, and diverse benchmark datasets for machine learning on graphs. Its datasets are automatically downloaded, processed, and split using the OGB Data Loader. The model performance can also be evaluated by using the OGB Evaluator in a unified manner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HnazPGGAJAZN"
   },
   "source": [
    "## Dataset and Data\n",
    "\n",
    "OGB also supports the PyG dataset and data. Here we take a look on the `ogbn-arxiv` dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "Gpc6bTm3GF02"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The ogbn-arxiv dataset has 1 graph\n",
      "Data(adj_t=[169343, 169343, nnz=1166243], node_year=[169343, 1], x=[169343, 128], y=[169343, 1])\n"
     ]
    }
   ],
   "source": [
    "import torch_geometric.transforms as T\n",
    "from ogb.nodeproppred import PygNodePropPredDataset\n",
    "\n",
    "dataset_name = 'ogbn-arxiv'\n",
    "# Load the dataset and transform it to sparse tensor\n",
    "dataset = PygNodePropPredDataset(name=dataset_name,\n",
    "                                 transform=T.ToSparseTensor())\n",
    "print('The {} dataset has {} graph'.format(dataset_name, len(dataset)))\n",
    "\n",
    "# Extract the graph\n",
    "data = dataset[0]\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Cw0xZJKZI-n3"
   },
   "source": [
    "## Question 4: What is the number of features in the ogbn-arxiv graph? (5 points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "ZP844_nT2ZJl"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The graph has 128 features\n"
     ]
    }
   ],
   "source": [
    "def graph_num_features(data):\n",
    "    # TODO: Implement this function that takes a PyG data object,\n",
    "    # and returns the number of features in the graph (in integer).\n",
    "\n",
    "    num_features = 0\n",
    "\n",
    "    ############# Your code here ############\n",
    "    ## (~1 line of code)\n",
    "    num_features = data.x.shape[1]\n",
    "    #########################################\n",
    "\n",
    "    return num_features\n",
    "\n",
    "num_features = graph_num_features(data)\n",
    "print('The graph has {} features'.format(num_features))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9DP_yEQZ0NVW"
   },
   "source": [
    "# 3 GNN: Node Property Prediction\n",
    "\n",
    "In this section we will build our first graph neural network by using PyTorch Geometric and apply it on node property prediction (node classification).\n",
    "\n",
    "We will build the graph neural network by using GCN operator ([Kipf et al. (2017)](https://arxiv.org/pdf/1609.02907.pdf)).\n",
    "\n",
    "You should use the PyG built-in `GCNConv` layer directly. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "O4CcOUEoInjD"
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "-DCtgcHpGIpd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.8.1\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "print(torch.__version__)\n",
    "\n",
    "# The PyG built-in GCNConv\n",
    "from torch_geometric.nn import GCNConv\n",
    "\n",
    "import torch_geometric.transforms as T\n",
    "from ogb.nodeproppred import PygNodePropPredDataset, Evaluator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0IK9z0wQIwzQ"
   },
   "source": [
    "## Load and Preprocess the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "0ibJ0ieoIwQM"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device: cuda\n"
     ]
    }
   ],
   "source": [
    "dataset_name = 'ogbn-arxiv'\n",
    "dataset = PygNodePropPredDataset(name=dataset_name,\n",
    "                                 transform=T.ToSparseTensor())\n",
    "data = dataset[0]\n",
    "\n",
    "# Make the adjacency matrix to symmetric\n",
    "data.adj_t = data.adj_t.to_symmetric()\n",
    "\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "\n",
    "# If you use GPU, the device should be cuda\n",
    "print('Device: {}'.format(device))\n",
    "\n",
    "data = data.to(device)\n",
    "split_idx = dataset.get_idx_split()\n",
    "train_idx = split_idx['train'].to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Data(adj_t=[169343, 169343, nnz=2315598], node_year=[169343, 1], x=[169343, 128], y=[169343, 1])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OgUA815bNJ8w"
   },
   "source": [
    "## GCN Model\n",
    "\n",
    "Now we will implement our GCN model!\n",
    "\n",
    "Please follow the figure below to implement your `forward` function.\n",
    "\n",
    "\n",
    "![test](https://drive.google.com/uc?id=128AuYAXNXGg7PIhJJ7e420DoPWKb-RtL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "IgspXTYpNJLA"
   },
   "outputs": [],
   "source": [
    "class GCN(torch.nn.Module):\n",
    "    def __init__(self, input_dim, hidden_dim, output_dim, num_layers,\n",
    "                 dropout, return_embeds=False):\n",
    "        # TODO: Implement this function that initializes self.convs, \n",
    "        # self.bns, and self.softmax.\n",
    "\n",
    "        super(GCN, self).__init__()\n",
    "\n",
    "        # A list of GCNConv layers\n",
    "        self.convs = None\n",
    "\n",
    "        # A list of 1D batch normalization layers\n",
    "        self.bns = None\n",
    "\n",
    "        # The log softmax layer\n",
    "        self.softmax = None\n",
    "        \n",
    "        self.num_layers = num_layers\n",
    "\n",
    "        ############# Your code here ############\n",
    "        ## Note:\n",
    "        ## 1. You should use torch.nn.ModuleList for self.convs and self.bns\n",
    "        ## 2. self.convs has num_layers GCNConv layers\n",
    "        ## 3. self.bns has num_layers - 1 BatchNorm1d layers\n",
    "        ## 4. You should use torch.nn.LogSoftmax for self.softmax\n",
    "        ## 5. The parameters you can set for GCNConv include 'in_channels' and \n",
    "        ## 'out_channels'. More information please refer to the documentation:\n",
    "        ## https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.conv.GCNConv\n",
    "        ## 6. The only parameter you need to set for BatchNorm1d is 'num_features'\n",
    "        ## More information please refer to the documentation: \n",
    "        ## https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html\n",
    "        ## (~10 lines of code)\n",
    "        \n",
    "        self.convs = nn.ModuleList([GCNConv(input_dim, hidden_dim)])\n",
    "        self.convs.extend([GCNConv(hidden_dim, hidden_dim) for i in range(num_layers-2)])\n",
    "        self.convs.extend([GCNConv(hidden_dim, output_dim)])\n",
    "        \n",
    "        self.bns = nn.ModuleList([nn.BatchNorm1d(hidden_dim) for i in range(num_layers-1)])\n",
    "        \n",
    "        self.softmax = nn.LogSoftmax(dim=1)\n",
    "        \n",
    "        #########################################\n",
    "\n",
    "        # Probability of an element to be zeroed\n",
    "        self.dropout = dropout\n",
    "\n",
    "        # Skip classification layer and return node embeddings\n",
    "        self.return_embeds = return_embeds\n",
    "\n",
    "    def reset_parameters(self):\n",
    "        for conv in self.convs:\n",
    "            conv.reset_parameters()\n",
    "        for bn in self.bns:\n",
    "            bn.reset_parameters()\n",
    "\n",
    "    def forward(self, x, adj_t):\n",
    "        # TODO: Implement this function that takes the feature tensor x,\n",
    "        # edge_index tensor adj_t and returns the output tensor as\n",
    "        # shown in the figure.\n",
    "\n",
    "        out = None\n",
    "\n",
    "        ############# Your code here ############\n",
    "        ## Note:\n",
    "        ## 1. Construct the network as showing in the figure\n",
    "        ## 2. torch.nn.functional.relu and torch.nn.functional.dropout are useful\n",
    "        ## More information please refer to the documentation:\n",
    "        ## https://pytorch.org/docs/stable/nn.functional.html\n",
    "        ## 3. Don't forget to set F.dropout training to self.training\n",
    "        ## 4. If return_embeds is True, then skip the last softmax layer\n",
    "        ## (~7 lines of code)\n",
    "        for i in range(self.num_layers):\n",
    "            \n",
    "            # Last Conv layer pass\n",
    "            if (i == self.num_layers-1):\n",
    "                x = self.convs[i](x, adj_t)\n",
    "                if (self.return_embeds):\n",
    "                    return x\n",
    "                x = self.softmax(x)\n",
    "                out = x\n",
    "                \n",
    "            else:\n",
    "                x = self.convs[i](x, adj_t)\n",
    "                x = self.bns[i](x)\n",
    "                x = F.relu(x)\n",
    "                x = F.dropout(x, p=self.dropout, training=self.training)\n",
    "        #########################################\n",
    "\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "FF1hnHUhO81e"
   },
   "outputs": [],
   "source": [
    "def train(model, data, train_idx, optimizer, loss_fn):\n",
    "    # TODO: Implement this function that trains the model by \n",
    "    # using the given optimizer and loss_fn.\n",
    "    model.train()\n",
    "    loss = 0\n",
    "\n",
    "    ############# Your code here ############\n",
    "    ## Note:\n",
    "    ## 1. Zero grad the optimizer\n",
    "    ## 2. Feed the data into the model\n",
    "    ## 3. Slicing the model output and label by train_idx\n",
    "    ## 4. Feed the sliced output and label to loss_fn\n",
    "    ## (~4 lines of code)\n",
    "    optimizer.zero_grad()\n",
    "    out = model(data.x, data.adj_t)\n",
    "    loss = loss_fn(out[train_idx], data.y[train_idx].squeeze(1))\n",
    "\n",
    "    #########################################\n",
    "\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    return loss.item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "aJdlrJQhPBsK"
   },
   "outputs": [],
   "source": [
    "# Test function here\n",
    "@torch.no_grad()\n",
    "def test(model, data, split_idx, evaluator):\n",
    "    # TODO: Implement this function that tests the model by \n",
    "    # using the given split_idx and evaluator.\n",
    "    model.eval()\n",
    "\n",
    "    # The output of model on all data\n",
    "    out = None\n",
    "\n",
    "    ############# Your code here ############\n",
    "    ## (~1 line of code)\n",
    "    ## Note:\n",
    "    ## 1. No index slicing here\n",
    "    out = model(data.x, data.adj_t)\n",
    "    \n",
    "\n",
    "    #########################################\n",
    "\n",
    "    y_pred = out.argmax(dim=-1, keepdim=True)\n",
    "\n",
    "    train_acc = evaluator.eval({\n",
    "        'y_true': data.y[split_idx['train']],\n",
    "        'y_pred': y_pred[split_idx['train']],\n",
    "    })['acc']\n",
    "    valid_acc = evaluator.eval({\n",
    "        'y_true': data.y[split_idx['valid']],\n",
    "        'y_pred': y_pred[split_idx['valid']],\n",
    "    })['acc']\n",
    "    test_acc = evaluator.eval({\n",
    "        'y_true': data.y[split_idx['test']],\n",
    "        'y_pred': y_pred[split_idx['test']],\n",
    "    })['acc']\n",
    "\n",
    "    return train_acc, valid_acc, test_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "o7F46xkuLiOL"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'device': 'cuda',\n",
       " 'num_layers': 3,\n",
       " 'hidden_dim': 256,\n",
       " 'dropout': 0.5,\n",
       " 'lr': 0.01,\n",
       " 'epochs': 100}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Please do not change the args\n",
    "args = {\n",
    "    'device': device,\n",
    "    'num_layers': 3,\n",
    "    'hidden_dim': 256,\n",
    "    'dropout': 0.5,\n",
    "    'lr': 0.01,\n",
    "    'epochs': 100,\n",
    "}\n",
    "args"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "dT8RyM2cPGxM"
   },
   "outputs": [],
   "source": [
    "model = GCN(data.num_features, args['hidden_dim'],\n",
    "            dataset.num_classes, args['num_layers'],\n",
    "            args['dropout']).to(device)\n",
    "evaluator = Evaluator(name='ogbn-arxiv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "qd5O5cnPPdVF",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01, Loss: 4.3668, Train: 13.34%, Valid: 23.49% Test: 21.86%\n",
      "Epoch: 02, Loss: 2.2613, Train: 36.90%, Valid: 46.01% Test: 47.74%\n",
      "Epoch: 03, Loss: 1.9286, Train: 35.51%, Valid: 36.39% Test: 38.45%\n",
      "Epoch: 04, Loss: 1.7386, Train: 35.81%, Valid: 32.19% Test: 32.74%\n",
      "Epoch: 05, Loss: 1.6397, Train: 34.87%, Valid: 21.98% Test: 20.00%\n",
      "Epoch: 06, Loss: 1.5598, Train: 36.24%, Valid: 20.53% Test: 16.88%\n",
      "Epoch: 07, Loss: 1.5045, Train: 35.08%, Valid: 20.55% Test: 17.60%\n",
      "Epoch: 08, Loss: 1.4609, Train: 34.25%, Valid: 24.59% Test: 24.89%\n",
      "Epoch: 09, Loss: 1.4203, Train: 35.61%, Valid: 30.65% Test: 33.87%\n",
      "Epoch: 10, Loss: 1.3823, Train: 37.21%, Valid: 33.83% Test: 39.11%\n",
      "Epoch: 11, Loss: 1.3547, Train: 38.34%, Valid: 35.83% Test: 40.88%\n",
      "Epoch: 12, Loss: 1.3282, Train: 40.14%, Valid: 37.14% Test: 41.37%\n",
      "Epoch: 13, Loss: 1.3022, Train: 42.05%, Valid: 37.88% Test: 42.03%\n",
      "Epoch: 14, Loss: 1.2827, Train: 43.79%, Valid: 39.06% Test: 43.16%\n",
      "Epoch: 15, Loss: 1.2656, Train: 45.52%, Valid: 41.58% Test: 45.74%\n",
      "Epoch: 16, Loss: 1.2475, Train: 47.48%, Valid: 44.90% Test: 49.12%\n",
      "Epoch: 17, Loss: 1.2265, Train: 48.88%, Valid: 46.58% Test: 51.10%\n",
      "Epoch: 18, Loss: 1.2090, Train: 50.78%, Valid: 48.48% Test: 53.05%\n",
      "Epoch: 19, Loss: 1.2043, Train: 52.35%, Valid: 49.37% Test: 54.02%\n",
      "Epoch: 20, Loss: 1.1878, Train: 54.34%, Valid: 51.32% Test: 55.92%\n",
      "Epoch: 21, Loss: 1.1726, Train: 56.42%, Valid: 54.08% Test: 58.31%\n",
      "Epoch: 22, Loss: 1.1674, Train: 59.03%, Valid: 58.20% Test: 61.33%\n",
      "Epoch: 23, Loss: 1.1589, Train: 61.54%, Valid: 61.37% Test: 63.68%\n",
      "Epoch: 24, Loss: 1.1475, Train: 63.36%, Valid: 63.31% Test: 65.18%\n",
      "Epoch: 25, Loss: 1.1409, Train: 64.60%, Valid: 64.51% Test: 65.94%\n",
      "Epoch: 26, Loss: 1.1307, Train: 65.35%, Valid: 65.46% Test: 66.51%\n",
      "Epoch: 27, Loss: 1.1244, Train: 65.33%, Valid: 65.46% Test: 66.44%\n",
      "Epoch: 28, Loss: 1.1139, Train: 64.76%, Valid: 64.63% Test: 65.90%\n",
      "Epoch: 29, Loss: 1.1106, Train: 64.32%, Valid: 63.86% Test: 65.32%\n",
      "Epoch: 30, Loss: 1.1049, Train: 64.80%, Valid: 64.35% Test: 65.73%\n",
      "Epoch: 31, Loss: 1.1000, Train: 65.74%, Valid: 65.46% Test: 66.55%\n",
      "Epoch: 32, Loss: 1.0858, Train: 67.02%, Valid: 66.83% Test: 67.42%\n",
      "Epoch: 33, Loss: 1.0820, Train: 67.77%, Valid: 67.80% Test: 67.48%\n",
      "Epoch: 34, Loss: 1.0779, Train: 68.15%, Valid: 68.17% Test: 67.49%\n",
      "Epoch: 35, Loss: 1.0717, Train: 68.44%, Valid: 68.41% Test: 67.85%\n",
      "Epoch: 36, Loss: 1.0646, Train: 68.60%, Valid: 68.62% Test: 68.17%\n",
      "Epoch: 37, Loss: 1.0612, Train: 68.58%, Valid: 68.32% Test: 68.30%\n",
      "Epoch: 38, Loss: 1.0568, Train: 68.55%, Valid: 68.12% Test: 68.45%\n",
      "Epoch: 39, Loss: 1.0524, Train: 68.88%, Valid: 68.52% Test: 68.85%\n",
      "Epoch: 40, Loss: 1.0464, Train: 69.36%, Valid: 69.08% Test: 69.27%\n",
      "Epoch: 41, Loss: 1.0413, Train: 69.62%, Valid: 69.40% Test: 69.57%\n",
      "Epoch: 42, Loss: 1.0374, Train: 69.95%, Valid: 69.70% Test: 69.69%\n",
      "Epoch: 43, Loss: 1.0357, Train: 70.29%, Valid: 69.98% Test: 69.57%\n",
      "Epoch: 44, Loss: 1.0293, Train: 70.61%, Valid: 70.02% Test: 69.39%\n",
      "Epoch: 45, Loss: 1.0266, Train: 70.84%, Valid: 70.30% Test: 69.67%\n",
      "Epoch: 46, Loss: 1.0238, Train: 70.85%, Valid: 70.20% Test: 70.05%\n",
      "Epoch: 47, Loss: 1.0189, Train: 70.73%, Valid: 70.14% Test: 70.13%\n",
      "Epoch: 48, Loss: 1.0205, Train: 70.95%, Valid: 70.36% Test: 70.32%\n",
      "Epoch: 49, Loss: 1.0180, Train: 71.22%, Valid: 70.55% Test: 70.34%\n",
      "Epoch: 50, Loss: 1.0068, Train: 71.45%, Valid: 70.44% Test: 69.47%\n",
      "Epoch: 51, Loss: 1.0081, Train: 71.32%, Valid: 70.07% Test: 68.71%\n",
      "Epoch: 52, Loss: 1.0063, Train: 71.28%, Valid: 70.06% Test: 68.72%\n",
      "Epoch: 53, Loss: 1.0024, Train: 71.38%, Valid: 70.61% Test: 69.74%\n",
      "Epoch: 54, Loss: 0.9991, Train: 71.49%, Valid: 70.83% Test: 70.19%\n",
      "Epoch: 55, Loss: 0.9944, Train: 71.73%, Valid: 70.88% Test: 69.80%\n",
      "Epoch: 56, Loss: 0.9956, Train: 71.90%, Valid: 70.69% Test: 69.22%\n",
      "Epoch: 57, Loss: 0.9908, Train: 72.03%, Valid: 70.82% Test: 69.45%\n",
      "Epoch: 58, Loss: 0.9883, Train: 72.00%, Valid: 70.96% Test: 69.71%\n",
      "Epoch: 59, Loss: 0.9893, Train: 72.03%, Valid: 70.91% Test: 69.61%\n",
      "Epoch: 60, Loss: 0.9858, Train: 72.03%, Valid: 70.23% Test: 68.37%\n",
      "Epoch: 61, Loss: 0.9804, Train: 71.92%, Valid: 69.79% Test: 67.67%\n",
      "Epoch: 62, Loss: 0.9816, Train: 71.94%, Valid: 70.44% Test: 68.93%\n",
      "Epoch: 63, Loss: 0.9793, Train: 72.18%, Valid: 71.04% Test: 69.89%\n",
      "Epoch: 64, Loss: 0.9752, Train: 72.39%, Valid: 71.20% Test: 69.81%\n",
      "Epoch: 65, Loss: 0.9745, Train: 72.33%, Valid: 70.39% Test: 68.36%\n",
      "Epoch: 66, Loss: 0.9729, Train: 72.22%, Valid: 70.12% Test: 68.09%\n",
      "Epoch: 67, Loss: 0.9702, Train: 72.49%, Valid: 70.71% Test: 69.33%\n",
      "Epoch: 68, Loss: 0.9641, Train: 72.43%, Valid: 71.04% Test: 70.30%\n",
      "Epoch: 69, Loss: 0.9646, Train: 72.41%, Valid: 70.90% Test: 70.09%\n",
      "Epoch: 70, Loss: 0.9638, Train: 72.56%, Valid: 70.46% Test: 68.68%\n",
      "Epoch: 71, Loss: 0.9594, Train: 72.40%, Valid: 69.74% Test: 67.34%\n",
      "Epoch: 72, Loss: 0.9564, Train: 72.71%, Valid: 70.54% Test: 68.46%\n",
      "Epoch: 73, Loss: 0.9573, Train: 72.84%, Valid: 71.53% Test: 70.54%\n",
      "Epoch: 74, Loss: 0.9537, Train: 72.66%, Valid: 71.49% Test: 70.92%\n",
      "Epoch: 75, Loss: 0.9504, Train: 72.76%, Valid: 71.16% Test: 70.04%\n",
      "Epoch: 76, Loss: 0.9522, Train: 72.87%, Valid: 70.75% Test: 69.14%\n",
      "Epoch: 77, Loss: 0.9496, Train: 72.98%, Valid: 71.05% Test: 69.69%\n",
      "Epoch: 78, Loss: 0.9491, Train: 73.06%, Valid: 71.44% Test: 70.27%\n",
      "Epoch: 79, Loss: 0.9467, Train: 72.93%, Valid: 71.31% Test: 70.25%\n",
      "Epoch: 80, Loss: 0.9418, Train: 72.76%, Valid: 71.19% Test: 70.58%\n",
      "Epoch: 81, Loss: 0.9423, Train: 72.76%, Valid: 71.25% Test: 70.94%\n",
      "Epoch: 82, Loss: 0.9393, Train: 73.03%, Valid: 71.71% Test: 71.31%\n",
      "Epoch: 83, Loss: 0.9392, Train: 73.25%, Valid: 71.53% Test: 70.40%\n",
      "Epoch: 84, Loss: 0.9389, Train: 72.96%, Valid: 70.43% Test: 68.46%\n",
      "Epoch: 85, Loss: 0.9342, Train: 73.09%, Valid: 71.19% Test: 69.91%\n",
      "Epoch: 86, Loss: 0.9333, Train: 73.23%, Valid: 71.69% Test: 71.03%\n",
      "Epoch: 87, Loss: 0.9315, Train: 73.24%, Valid: 71.64% Test: 71.02%\n",
      "Epoch: 88, Loss: 0.9311, Train: 73.34%, Valid: 71.62% Test: 71.03%\n",
      "Epoch: 89, Loss: 0.9299, Train: 73.46%, Valid: 71.70% Test: 70.94%\n",
      "Epoch: 90, Loss: 0.9313, Train: 73.64%, Valid: 71.45% Test: 70.09%\n",
      "Epoch: 91, Loss: 0.9258, Train: 73.66%, Valid: 71.22% Test: 69.85%\n",
      "Epoch: 92, Loss: 0.9266, Train: 73.56%, Valid: 71.66% Test: 70.66%\n",
      "Epoch: 93, Loss: 0.9230, Train: 73.36%, Valid: 71.65% Test: 70.89%\n",
      "Epoch: 94, Loss: 0.9224, Train: 73.62%, Valid: 71.66% Test: 70.15%\n",
      "Epoch: 95, Loss: 0.9212, Train: 73.58%, Valid: 71.38% Test: 69.72%\n",
      "Epoch: 96, Loss: 0.9193, Train: 73.65%, Valid: 71.80% Test: 70.87%\n",
      "Epoch: 97, Loss: 0.9165, Train: 73.75%, Valid: 71.77% Test: 70.84%\n",
      "Epoch: 98, Loss: 0.9155, Train: 73.84%, Valid: 71.87% Test: 70.71%\n",
      "Epoch: 99, Loss: 0.9156, Train: 73.79%, Valid: 72.00% Test: 71.11%\n",
      "Epoch: 100, Loss: 0.9119, Train: 73.59%, Valid: 71.76% Test: 71.00%\n"
     ]
    }
   ],
   "source": [
    "import copy\n",
    "\n",
    "# reset the parameters to initial random value\n",
    "model.reset_parameters()\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=args['lr'])\n",
    "loss_fn = F.nll_loss\n",
    "\n",
    "best_model = None\n",
    "best_valid_acc = 0\n",
    "\n",
    "for epoch in range(1, 1 + args[\"epochs\"]):\n",
    "    loss = train(model, data, train_idx, optimizer, loss_fn)\n",
    "    result = test(model, data, split_idx, evaluator)\n",
    "    train_acc, valid_acc, test_acc = result\n",
    "    if valid_acc > best_valid_acc:\n",
    "        best_valid_acc = valid_acc\n",
    "        best_model = copy.deepcopy(model)\n",
    "    print(f'Epoch: {epoch:02d}, '\n",
    "        f'Loss: {loss:.4f}, '\n",
    "        f'Train: {100 * train_acc:.2f}%, '\n",
    "        f'Valid: {100 * valid_acc:.2f}% '\n",
    "        f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "EqcextqOL2FX"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best model: Train: 73.79%, Valid: 72.00% Test: 71.11%\n"
     ]
    }
   ],
   "source": [
    "best_result = test(best_model, data, split_idx, evaluator)\n",
    "train_acc, valid_acc, test_acc = best_result\n",
    "print(f'Best model: '\n",
    "      f'Train: {100 * train_acc:.2f}%, '\n",
    "      f'Valid: {100 * valid_acc:.2f}% '\n",
    "      f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "duMEg-olLjbJ"
   },
   "source": [
    "## Question 5: What are your `best_model` validation and test accuracy? Please report them on Gradescope. For example, for an accuracy such as 50.01%, just report 50.01 and please don't include the percent sign. (20 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "R8pOD6y80TyI"
   },
   "source": [
    "# 4 GNN: Graph Property Prediction\n",
    "\n",
    "In this section we will create a graph neural network for graph property prediction (graph classification)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vRg5VOEdQTa4"
   },
   "source": [
    "## Load and preprocess the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "LXb-O5QUIgTH"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device: cuda\n",
      "Task type: binary classification\n"
     ]
    }
   ],
   "source": [
    "from ogb.graphproppred import PygGraphPropPredDataset, Evaluator\n",
    "from torch_geometric.data import DataLoader\n",
    "from tqdm import tqdm\n",
    "\n",
    "# Load the dataset \n",
    "dataset = PygGraphPropPredDataset(name='ogbg-molhiv')\n",
    "\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "print('Device: {}'.format(device))\n",
    "\n",
    "split_idx = dataset.get_idx_split()\n",
    "\n",
    "# Check task type\n",
    "print('Task type: {}'.format(dataset.task_type))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "7cHHbgW1c5hi"
   },
   "outputs": [],
   "source": [
    "# Load the data sets into dataloader\n",
    "# We will train the graph classification task on a batch of 32 graphs\n",
    "# Shuffle the order of graphs for training set\n",
    "train_loader = DataLoader(dataset[split_idx[\"train\"]], batch_size=32, shuffle=True, num_workers=0)\n",
    "valid_loader = DataLoader(dataset[split_idx[\"valid\"]], batch_size=32, shuffle=False, num_workers=0)\n",
    "test_loader = DataLoader(dataset[split_idx[\"test\"]], batch_size=32, shuffle=False, num_workers=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "AYrSnOj0Y4DK",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'device': 'cuda',\n",
       " 'num_layers': 5,\n",
       " 'hidden_dim': 256,\n",
       " 'dropout': 0.5,\n",
       " 'lr': 0.001,\n",
       " 'epochs': 30}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Please do not change the args\n",
    "args = {\n",
    "    'device': device,\n",
    "    'num_layers': 5,\n",
    "    'hidden_dim': 256,\n",
    "    'dropout': 0.5,\n",
    "    'lr': 0.001,\n",
    "    'epochs': 30,\n",
    "}\n",
    "args"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "41127"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7WLhguSTeazy"
   },
   "source": [
    "## Graph Prediction Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u05Z14TRYPGn"
   },
   "source": [
    "Now we will implement our GCN Graph Prediction model!\n",
    "\n",
    "We will reuse the existing GCN model to generate `node_embeddings` and use  Global Pooling on the nodes to predict properties for the whole graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "3_Kq3zyjeZ22"
   },
   "outputs": [],
   "source": [
    "from ogb.graphproppred.mol_encoder import AtomEncoder\n",
    "from torch_geometric.nn import global_add_pool, global_mean_pool\n",
    "\n",
    "### GCN to predict graph property\n",
    "class GCN_Graph(torch.nn.Module):\n",
    "    def __init__(self, hidden_dim, output_dim, num_layers, dropout, pool):\n",
    "        super(GCN_Graph, self).__init__()\n",
    "\n",
    "        # Load encoders for Atoms in molecule graphs\n",
    "        self.node_encoder = AtomEncoder(hidden_dim)\n",
    "\n",
    "        # Node embedding model\n",
    "        # Note that the input_dim and output_dim are set to hidden_dim\n",
    "        self.gnn_node = GCN(hidden_dim, hidden_dim,\n",
    "            hidden_dim, num_layers, dropout, return_embeds=True)\n",
    "\n",
    "        self.pool = None\n",
    "\n",
    "        ############# Your code here ############\n",
    "        ## Note:\n",
    "        ## 1. Initialize the self.pool to global mean pooling layer\n",
    "        ## More information please refer to the documentation:\n",
    "        ## https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#global-pooling-layers\n",
    "        ## (~1 line of code)\n",
    "#         self.pool = global_mean_pool\n",
    "        self.pool = pool\n",
    "\n",
    "        #########################################\n",
    "\n",
    "        # Output layer\n",
    "        self.linear = torch.nn.Linear(hidden_dim, output_dim)\n",
    "\n",
    "\n",
    "    def reset_parameters(self):\n",
    "        self.gnn_node.reset_parameters()\n",
    "        self.linear.reset_parameters()\n",
    "\n",
    "    def forward(self, batched_data):\n",
    "        # TODO: Implement this function that takes the input tensor batched_data,\n",
    "        # returns a batched output tensor for each graph.\n",
    "        x, edge_index, batch = batched_data.x, batched_data.edge_index, batched_data.batch\n",
    "        embed = self.node_encoder(x)\n",
    "\n",
    "        out = None\n",
    "\n",
    "        ############# Your code here ############\n",
    "        ## Note:\n",
    "        ## 1. Construct node embeddings using existing GCN model\n",
    "        ## 2. Use global pooling layer to construct features for the whole graph\n",
    "        ## More information please refer to the documentation:\n",
    "        ## https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#global-pooling-layers\n",
    "        ## 3. Use a linear layer to predict the graph property \n",
    "        ## (~3 lines of code)\n",
    "        out = self.gnn_node(embed, edge_index)\n",
    "        out = self.pool(out, batch)\n",
    "        out = self.linear(out)\n",
    "\n",
    "        #########################################\n",
    "\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "FJjnGuMSbjX0"
   },
   "outputs": [],
   "source": [
    "def train(model, device, data_loader, optimizer, loss_fn):\n",
    "    # TODO: Implement this function that trains the model by \n",
    "    # using the given optimizer and loss_fn.\n",
    "    model.train()\n",
    "    loss = 0\n",
    "\n",
    "    for step, batch in enumerate(tqdm(data_loader, desc=\"Iteration\")):\n",
    "        batch = batch.to(device)\n",
    "\n",
    "        if batch.x.shape[0] == 1 or batch.batch[-1] == 0:\n",
    "            pass\n",
    "        else:\n",
    "            ## ignore nan targets (unlabeled) when computing training loss.\n",
    "            is_labeled = batch.y == batch.y\n",
    "\n",
    "            ############# Your code here ############\n",
    "            ## Note:\n",
    "            ## 1. Zero grad the optimizer\n",
    "            ## 2. Feed the data into the model\n",
    "            ## 3. Use `is_labeled` mask to filter output and labels\n",
    "            ## 4. You might change the type of label\n",
    "            ## 5. Feed the output and label to loss_fn\n",
    "            ## (~3 lines of code)\n",
    "            optimizer.zero_grad()\n",
    "            out = model(batch)\n",
    "            \n",
    "            loss = loss_fn(out[is_labeled], batch.y[is_labeled].type_as(out))\n",
    "            #########################################\n",
    "\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "    return loss.item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "ztPHXq_Gzn7U"
   },
   "outputs": [],
   "source": [
    "# The evaluation function\n",
    "def eval(model, device, loader, evaluator):\n",
    "    model.eval()\n",
    "    y_true = []\n",
    "    y_pred = []\n",
    "\n",
    "    for step, batch in enumerate(tqdm(loader, desc=\"Iteration\")):\n",
    "        batch = batch.to(device)\n",
    "\n",
    "        if batch.x.shape[0] == 1:\n",
    "            pass\n",
    "        else:\n",
    "            with torch.no_grad():\n",
    "                pred = model(batch)\n",
    "\n",
    "            y_true.append(batch.y.view(pred.shape).detach().cpu())\n",
    "            y_pred.append(pred.detach().cpu())\n",
    "\n",
    "    y_true = torch.cat(y_true, dim = 0).numpy()\n",
    "    y_pred = torch.cat(y_pred, dim = 0).numpy()\n",
    "\n",
    "    input_dict = {\"y_true\": y_true, \"y_pred\": y_pred}\n",
    "\n",
    "    return evaluator.eval(input_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "MR1wQ4hMZeMw"
   },
   "outputs": [],
   "source": [
    "model = GCN_Graph(args['hidden_dim'],\n",
    "            dataset.num_tasks, args['num_layers'],\n",
    "            args['dropout'], global_mean_pool).to(device)\n",
    "evaluator = Evaluator(name='ogbg-molhiv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "qJGTNZiuZy0A",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Iteration:   0%|          | 0/1029 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:16<00:00, 61.78it/s]\n",
      "Iteration:   1%|▏         | 14/1029 [00:00<00:07, 133.59it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 187.28it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 147.43it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 130.35it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:15, 67.48it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01, Loss: 0.0267, Train: 70.73%, Valid: 67.49% Test: 68.56%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.60it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:06, 161.33it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 194.05it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.61it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.91it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 74.31it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 02, Loss: 0.0385, Train: 75.17%, Valid: 71.40% Test: 72.46%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 74.32it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:06, 163.30it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 189.02it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.24it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.80it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 73.82it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 03, Loss: 0.0803, Train: 75.40%, Valid: 75.24% Test: 70.74%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 73.31it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 151.43it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 189.87it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.28it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.79it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 77.91it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 04, Loss: 0.6573, Train: 77.65%, Valid: 76.32% Test: 69.34%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.76it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 159.25it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 177.05it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 182.36it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 180.23it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 75.93it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 05, Loss: 0.0296, Train: 78.00%, Valid: 73.36% Test: 69.13%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.48it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:06, 163.86it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 190.16it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 190.47it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.42it/s]\n",
      "Iteration:   1%|          | 9/1029 [00:00<00:12, 80.37it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 06, Loss: 0.0206, Train: 78.65%, Valid: 76.26% Test: 71.15%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.56it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 171.47it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 190.43it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.91it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.11it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 70.96it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 07, Loss: 0.0219, Train: 77.11%, Valid: 72.67% Test: 71.00%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.10it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 152.20it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 191.00it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.65it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.68it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 69.91it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 08, Loss: 0.0196, Train: 79.24%, Valid: 76.99% Test: 71.43%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 70.74it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 152.27it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 193.62it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 196.56it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 186.04it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 72.06it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 09, Loss: 0.0233, Train: 79.10%, Valid: 76.78% Test: 73.78%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.96it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 176.50it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 190.21it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 198.54it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 198.72it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 73.24it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 10, Loss: 0.0171, Train: 80.21%, Valid: 77.53% Test: 72.11%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 73.71it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 176.21it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 193.30it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 196.83it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 198.87it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 77.27it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 11, Loss: 0.0399, Train: 79.78%, Valid: 73.99% Test: 71.59%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.67it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 176.18it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 193.78it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 198.69it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 198.82it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 77.88it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 12, Loss: 0.2214, Train: 81.07%, Valid: 78.15% Test: 73.69%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 74.08it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 154.29it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 193.84it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.98it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 195.87it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:15, 67.49it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 13, Loss: 0.0322, Train: 80.54%, Valid: 75.52% Test: 74.91%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 74.32it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:06, 161.78it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 200.85it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 202.63it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 202.69it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 72.98it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 14, Loss: 0.0260, Train: 80.48%, Valid: 78.41% Test: 74.59%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:12<00:00, 79.93it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 177.02it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 197.26it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 201.33it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 201.85it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 75.85it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 15, Loss: 0.0310, Train: 81.38%, Valid: 78.08% Test: 74.31%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.82it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 178.50it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 193.42it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 196.03it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 199.20it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 78.06it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 16, Loss: 0.0330, Train: 81.08%, Valid: 77.69% Test: 73.89%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:12<00:00, 80.23it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 178.36it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 195.01it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 198.59it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 200.06it/s]\n",
      "Iteration:   1%|          | 9/1029 [00:00<00:12, 80.54it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 17, Loss: 0.0245, Train: 81.78%, Valid: 76.83% Test: 74.45%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 75.93it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 179.16it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 198.01it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 201.05it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 202.59it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 78.06it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 18, Loss: 0.6059, Train: 81.98%, Valid: 77.78% Test: 72.35%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 73.51it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 176.25it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 196.92it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 201.15it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 200.18it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 73.46it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 19, Loss: 0.0307, Train: 82.35%, Valid: 78.94% Test: 76.21%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:12<00:00, 80.81it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 179.58it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 193.74it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 172.57it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 178.44it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:12, 78.88it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 20, Loss: 0.0236, Train: 81.49%, Valid: 76.86% Test: 73.69%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 70.01it/s]\n",
      "Iteration:   1%|▏         | 15/1029 [00:00<00:07, 144.58it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 190.21it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.26it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.83it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:14, 69.77it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 21, Loss: 0.0181, Train: 82.25%, Valid: 78.15% Test: 72.16%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.59it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:05, 169.15it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 189.31it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.39it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.42it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 76.92it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 22, Loss: 0.0378, Train: 82.42%, Valid: 77.98% Test: 75.54%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.97it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 172.20it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 190.62it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.84it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.45it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:14, 68.85it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 23, Loss: 0.0226, Train: 82.38%, Valid: 77.92% Test: 75.63%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.81it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 170.85it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 188.22it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.60it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 190.90it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 70.93it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 24, Loss: 0.4324, Train: 83.01%, Valid: 79.20% Test: 74.78%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.48it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 156.26it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 188.99it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.14it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.83it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:14, 69.98it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 25, Loss: 0.2804, Train: 83.28%, Valid: 77.73% Test: 74.56%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.65it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:06, 164.78it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 189.79it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.37it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.65it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 73.39it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 26, Loss: 0.0155, Train: 83.68%, Valid: 79.63% Test: 74.72%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.86it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 159.65it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 188.83it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.17it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 190.87it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 72.41it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 27, Loss: 0.0323, Train: 83.37%, Valid: 77.81% Test: 76.27%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.78it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 173.77it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 193.11it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.46it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.71it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 74.83it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 28, Loss: 0.0241, Train: 83.39%, Valid: 77.64% Test: 76.13%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.74it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 173.17it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 193.62it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 196.13it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.24it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:14, 68.93it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 29, Loss: 0.0192, Train: 84.52%, Valid: 78.58% Test: 76.31%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.38it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 171.29it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 193.55it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.89it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.41it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 30, Loss: 0.4564, Train: 84.47%, Valid: 78.84% Test: 76.22%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "import copy\n",
    "\n",
    "model.reset_parameters()\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=args['lr'])\n",
    "loss_fn = torch.nn.BCEWithLogitsLoss()\n",
    "\n",
    "best_model = None\n",
    "best_valid_acc = 0\n",
    "\n",
    "for epoch in range(1, 1 + args[\"epochs\"]):\n",
    "    print('Training...')\n",
    "    loss = train(model, device, train_loader, optimizer, loss_fn)\n",
    "\n",
    "    print('Evaluating...')\n",
    "    train_result = eval(model, device, train_loader, evaluator)\n",
    "    val_result = eval(model, device, valid_loader, evaluator)\n",
    "    test_result = eval(model, device, test_loader, evaluator)\n",
    "\n",
    "    train_acc, valid_acc, test_acc = train_result[dataset.eval_metric], val_result[dataset.eval_metric], test_result[dataset.eval_metric]\n",
    "    if valid_acc > best_valid_acc:\n",
    "        best_valid_acc = valid_acc\n",
    "        best_model = copy.deepcopy(model)\n",
    "    print(f'Epoch: {epoch:02d}, '\n",
    "        f'Loss: {loss:.4f}, '\n",
    "        f'Train: {100 * train_acc:.2f}%, '\n",
    "        f'Valid: {100 * valid_acc:.2f}% '\n",
    "        f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "Oq5QaG21dOOO"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 189.66it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 188.78it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 189.81it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best model: Train: 83.68%, Valid: 79.63% Test: 74.72%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "train_acc = eval(best_model, device, train_loader, evaluator)[dataset.eval_metric]\n",
    "valid_acc = eval(best_model, device, valid_loader, evaluator)[dataset.eval_metric]\n",
    "test_acc = eval(best_model, device, test_loader, evaluator)[dataset.eval_metric]\n",
    "\n",
    "print(f'Best model: '\n",
    "      f'Train: {100 * train_acc:.2f}%, '\n",
    "      f'Valid: {100 * valid_acc:.2f}% '\n",
    "      f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-uKs6j6t1ah3"
   },
   "source": [
    "## Question 6: What are your `best_model` validation and test ROC-AUC score? Please report them on Gradescope. For example, for an ROC-AUC score such as 50.01%, just report 50.01 and please don't include the percent sign. (20 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gBi_t8n0iZ4P"
   },
   "source": [
    "## Question 7 (Optional): Experiment with other two global pooling layers other than mean pooling in Pytorch Geometric."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Max Pooling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch_geometric.nn import global_max_pool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = GCN_Graph(args['hidden_dim'],\n",
    "            dataset.num_tasks, args['num_layers'],\n",
    "            args['dropout'], global_max_pool).to(device)\n",
    "evaluator = Evaluator(name='ogbg-molhiv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "taxEEWyh1_jq"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 72.16it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.05it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:06, 164.75it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 190.75it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 187.24it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 187.13it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:14, 69.55it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01, Loss: 0.0449, Train: 71.42%, Valid: 64.69% Test: 68.83%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.92it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 157.11it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 193.66it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 196.01it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 196.76it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 71.04it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 02, Loss: 0.0312, Train: 76.17%, Valid: 67.81% Test: 72.80%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 70.88it/s]\n",
      "Iteration:   1%|▏         | 15/1029 [00:00<00:07, 143.78it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 189.29it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.37it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 190.83it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:14, 69.56it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 03, Loss: 0.0266, Train: 78.08%, Valid: 73.02% Test: 73.27%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.95it/s]\n",
      "Iteration:   1%|▏         | 15/1029 [00:00<00:06, 145.48it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 187.28it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 184.65it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 184.83it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:14, 69.70it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 04, Loss: 0.0150, Train: 78.71%, Valid: 73.60% Test: 74.18%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.06it/s]\n",
      "Iteration:   1%|▏         | 15/1029 [00:00<00:06, 149.44it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 188.23it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 190.05it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.53it/s]\n",
      "Iteration:   1%|          | 9/1029 [00:00<00:12, 82.86it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 05, Loss: 0.0403, Train: 79.50%, Valid: 70.09% Test: 75.74%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.48it/s]\n",
      "Iteration:   1%|▏         | 15/1029 [00:00<00:06, 149.84it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 186.27it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 187.21it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 185.91it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 71.38it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 06, Loss: 0.9828, Train: 79.24%, Valid: 71.30% Test: 72.55%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 70.94it/s]\n",
      "Iteration:   1%|▏         | 15/1029 [00:00<00:06, 146.03it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 190.66it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.05it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.17it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 72.57it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 07, Loss: 0.0232, Train: 81.01%, Valid: 73.00% Test: 72.99%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.71it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:06, 167.67it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 192.46it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.46it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.19it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 76.85it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 08, Loss: 0.0497, Train: 80.81%, Valid: 75.43% Test: 75.13%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.96it/s]\n",
      "Iteration:   1%|▏         | 15/1029 [00:00<00:06, 148.77it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 191.94it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 180.79it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.68it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:14, 68.84it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 09, Loss: 0.0318, Train: 80.88%, Valid: 69.83% Test: 75.55%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 73.13it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:05, 169.09it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 191.01it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.94it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.42it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:15, 66.30it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 10, Loss: 0.0255, Train: 82.49%, Valid: 75.25% Test: 76.06%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.07it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 152.62it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 186.23it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.64it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.15it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 78.31it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 11, Loss: 0.0503, Train: 81.83%, Valid: 77.03% Test: 74.90%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 73.52it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 152.23it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 187.51it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 190.93it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.21it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 71.57it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 12, Loss: 0.0224, Train: 82.32%, Valid: 77.99% Test: 76.34%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.06it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:06, 166.08it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 186.84it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 187.69it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 189.78it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 76.39it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 13, Loss: 0.0704, Train: 83.18%, Valid: 77.18% Test: 74.45%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.02it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 171.13it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 189.38it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 190.69it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 190.72it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 69.80it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 14, Loss: 0.3094, Train: 83.40%, Valid: 78.15% Test: 77.58%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 70.86it/s]\n",
      "Iteration:   1%|▏         | 15/1029 [00:00<00:06, 144.89it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 186.40it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 186.71it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 187.93it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 76.34it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 15, Loss: 0.0781, Train: 83.16%, Valid: 79.30% Test: 77.16%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.38it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 173.14it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 188.43it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 190.43it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.71it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:15, 66.72it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 16, Loss: 0.3612, Train: 84.11%, Valid: 77.65% Test: 76.86%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 70.85it/s]\n",
      "Iteration:   1%|▏         | 15/1029 [00:00<00:06, 149.00it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 191.90it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.63it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.00it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 70.52it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 17, Loss: 0.0831, Train: 84.06%, Valid: 77.08% Test: 75.69%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 70.82it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:06, 167.39it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 188.16it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 188.95it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 190.69it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 71.37it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 18, Loss: 0.9567, Train: 84.48%, Valid: 78.28% Test: 76.63%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.19it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 172.37it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 186.74it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 187.93it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 188.77it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 73.67it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 19, Loss: 0.0907, Train: 84.97%, Valid: 77.61% Test: 78.71%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.15it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 172.03it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 187.44it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 188.12it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 188.29it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 71.71it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 20, Loss: 0.0974, Train: 85.35%, Valid: 77.69% Test: 75.97%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 77.50it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:05, 169.83it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 184.36it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 187.63it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 185.99it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 71.80it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 21, Loss: 0.0140, Train: 84.57%, Valid: 77.85% Test: 76.71%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 70.88it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 153.36it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 188.43it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 189.60it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 187.56it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 76.13it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 22, Loss: 0.0353, Train: 84.99%, Valid: 79.71% Test: 77.83%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 70.38it/s]\n",
      "Iteration:   1%|▏         | 15/1029 [00:00<00:06, 145.56it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 189.84it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.67it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.46it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 74.72it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 23, Loss: 0.0321, Train: 85.71%, Valid: 81.01% Test: 78.51%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.87it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 171.87it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 191.39it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.68it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 196.01it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 71.49it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 24, Loss: 0.0068, Train: 84.99%, Valid: 78.44% Test: 75.88%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.49it/s]\n",
      "Iteration:   1%|▏         | 15/1029 [00:00<00:06, 146.53it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 189.63it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.63it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.02it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 74.39it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 25, Loss: 0.0157, Train: 85.85%, Valid: 77.61% Test: 76.57%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.25it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 152.83it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 190.01it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.24it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.80it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 73.15it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 26, Loss: 0.0188, Train: 86.36%, Valid: 78.65% Test: 78.33%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 75.00it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 152.50it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 191.07it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.34it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.23it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 70.33it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 27, Loss: 0.0491, Train: 86.32%, Valid: 77.73% Test: 78.51%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.72it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 154.67it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 189.71it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.39it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.17it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 74.55it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 28, Loss: 0.0121, Train: 86.42%, Valid: 78.78% Test: 77.46%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.60it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 151.81it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 186.19it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 188.91it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 188.62it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 73.19it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 29, Loss: 1.1591, Train: 87.06%, Valid: 79.67% Test: 78.20%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.26it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 171.26it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 187.14it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 190.52it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 187.33it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 30, Loss: 0.0343, Train: 87.24%, Valid: 82.47% Test: 78.92%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "import copy\n",
    "\n",
    "model.reset_parameters()\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=args['lr'])\n",
    "loss_fn = torch.nn.BCEWithLogitsLoss()\n",
    "\n",
    "best_model = None\n",
    "best_valid_acc = 0\n",
    "\n",
    "for epoch in range(1, 1 + args[\"epochs\"]):\n",
    "    print('Training...')\n",
    "    loss = train(model, device, train_loader, optimizer, loss_fn)\n",
    "\n",
    "    print('Evaluating...')\n",
    "    train_result = eval(model, device, train_loader, evaluator)\n",
    "    val_result = eval(model, device, valid_loader, evaluator)\n",
    "    test_result = eval(model, device, test_loader, evaluator)\n",
    "\n",
    "    train_acc, valid_acc, test_acc = train_result[dataset.eval_metric], val_result[dataset.eval_metric], test_result[dataset.eval_metric]\n",
    "    if valid_acc > best_valid_acc:\n",
    "        best_valid_acc = valid_acc\n",
    "        best_model = copy.deepcopy(model)\n",
    "    print(f'Epoch: {epoch:02d}, '\n",
    "        f'Loss: {loss:.4f}, '\n",
    "        f'Train: {100 * train_acc:.2f}%, '\n",
    "        f'Valid: {100 * valid_acc:.2f}% '\n",
    "        f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 186.62it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 187.73it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 190.43it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best model: Train: 87.24%, Valid: 82.47% Test: 78.92%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "train_acc = eval(best_model, device, train_loader, evaluator)[dataset.eval_metric]\n",
    "valid_acc = eval(best_model, device, valid_loader, evaluator)[dataset.eval_metric]\n",
    "test_acc = eval(best_model, device, test_loader, evaluator)[dataset.eval_metric]\n",
    "\n",
    "print(f'Best model: '\n",
    "      f'Train: {100 * train_acc:.2f}%, '\n",
    "      f'Valid: {100 * valid_acc:.2f}% '\n",
    "      f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sum Pooling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch_geometric.nn import global_add_pool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = GCN_Graph(args['hidden_dim'],\n",
    "            dataset.num_tasks, args['num_layers'],\n",
    "            args['dropout'], global_add_pool).to(device)\n",
    "evaluator = Evaluator(name='ogbg-molhiv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration:   1%|          | 7/1029 [00:00<00:16, 62.67it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.69it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:06, 160.33it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 190.72it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.58it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 195.64it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:14, 69.13it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01, Loss: 0.0147, Train: 56.90%, Valid: 56.18% Test: 55.92%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.35it/s]\n",
      "Iteration:   1%|▏         | 15/1029 [00:00<00:06, 146.64it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 191.99it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 197.71it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 197.62it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 75.22it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 02, Loss: 0.5428, Train: 66.91%, Valid: 67.29% Test: 65.73%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.90it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 153.36it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 191.00it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.84it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.66it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 78.19it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 03, Loss: 1.6086, Train: 72.21%, Valid: 64.05% Test: 70.15%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.15it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:05, 168.88it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 193.42it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 197.23it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 196.32it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 75.70it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 04, Loss: 0.0593, Train: 67.93%, Valid: 67.21% Test: 57.69%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 77.45it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:06, 162.80it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 193.18it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.79it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.80it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:14, 69.83it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 05, Loss: 0.0269, Train: 68.62%, Valid: 61.72% Test: 60.02%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.91it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:06, 166.48it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 192.63it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.26it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.98it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:14, 69.66it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 06, Loss: 0.0310, Train: 73.05%, Valid: 61.68% Test: 63.85%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.91it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 170.59it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 191.92it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 195.94it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.55it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 71.54it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 07, Loss: 0.0209, Train: 72.76%, Valid: 65.93% Test: 65.84%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.61it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 174.79it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 194.54it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 198.11it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 198.54it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 71.77it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 08, Loss: 0.0356, Train: 74.55%, Valid: 73.41% Test: 69.49%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 76.90it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 179.11it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 192.41it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 188.48it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 195.10it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 75.44it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 09, Loss: 0.0186, Train: 76.20%, Valid: 63.34% Test: 68.33%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.66it/s]\n",
      "Iteration:   1%|▏         | 15/1029 [00:00<00:06, 149.74it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 187.62it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 190.54it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.12it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 74.49it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 10, Loss: 0.2734, Train: 77.05%, Valid: 70.05% Test: 71.35%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 73.19it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:06, 165.92it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 188.20it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 190.18it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 190.72it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:14, 69.60it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 11, Loss: 0.0377, Train: 76.03%, Valid: 75.34% Test: 73.97%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.31it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 153.75it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 193.55it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 198.38it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 197.42it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 74.34it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 12, Loss: 0.0382, Train: 76.85%, Valid: 75.52% Test: 73.77%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 75.50it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 176.64it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 194.38it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 189.71it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 184.46it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 71.99it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 13, Loss: 0.1107, Train: 77.44%, Valid: 76.43% Test: 72.85%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.40it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 158.75it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 186.14it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.46it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.49it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 74.70it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 14, Loss: 0.8132, Train: 78.73%, Valid: 74.96% Test: 74.65%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.89it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:06, 165.64it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 194.83it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 197.17it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 197.45it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:14, 69.20it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 15, Loss: 0.0268, Train: 79.04%, Valid: 73.85% Test: 75.07%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.36it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 175.74it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 191.96it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 195.17it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.45it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:12, 79.92it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 16, Loss: 0.0510, Train: 79.59%, Valid: 78.97% Test: 74.49%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 73.55it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:06, 167.41it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 193.32it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 196.67it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 195.35it/s]\n",
      "Iteration:   1%|          | 9/1029 [00:00<00:12, 80.52it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 17, Loss: 0.0168, Train: 78.73%, Valid: 78.73% Test: 76.70%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 73.98it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 174.13it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 193.14it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.39it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.76it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 76.53it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 18, Loss: 0.0276, Train: 79.62%, Valid: 78.13% Test: 74.75%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 74.02it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 174.49it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 191.58it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.30it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 195.04it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 73.61it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 19, Loss: 0.4068, Train: 80.19%, Valid: 77.68% Test: 75.77%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 77.53it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 175.91it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 194.57it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 198.38it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 198.94it/s]\n",
      "Iteration:   1%|          | 9/1029 [00:00<00:12, 81.26it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 20, Loss: 0.0251, Train: 80.83%, Valid: 77.63% Test: 74.74%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 74.14it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 176.70it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 196.47it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 196.92it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 198.62it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:14, 71.61it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 21, Loss: 0.0275, Train: 80.86%, Valid: 80.52% Test: 75.55%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.63it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:06, 165.06it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 194.50it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 199.16it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 198.95it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 76.77it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 22, Loss: 0.0501, Train: 81.59%, Valid: 78.48% Test: 75.26%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 75.83it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 153.40it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 192.49it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.72it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 192.11it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 76.12it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 23, Loss: 0.4086, Train: 80.47%, Valid: 76.58% Test: 76.29%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 72.33it/s]\n",
      "Iteration:   2%|▏         | 16/1029 [00:00<00:06, 153.10it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 191.53it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.23it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.57it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 75.17it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 24, Loss: 0.0281, Train: 80.89%, Valid: 75.17% Test: 71.96%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.33it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 172.89it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 189.53it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.34it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.54it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:14, 69.72it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 25, Loss: 0.2534, Train: 81.91%, Valid: 79.47% Test: 74.23%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 74.39it/s]\n",
      "Iteration:   2%|▏         | 17/1029 [00:00<00:05, 169.63it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 188.59it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.27it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 193.85it/s]\n",
      "Iteration:   1%|          | 7/1029 [00:00<00:15, 66.50it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 26, Loss: 0.0188, Train: 80.77%, Valid: 70.30% Test: 71.85%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 71.31it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 176.23it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 194.06it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 197.05it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 197.17it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 77.94it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 27, Loss: 0.0974, Train: 81.35%, Valid: 80.77% Test: 75.49%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 73.74it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 173.02it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 194.13it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 195.30it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 195.87it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 74.03it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 28, Loss: 0.6375, Train: 82.35%, Valid: 80.93% Test: 77.75%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:13<00:00, 74.20it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 174.24it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 194.47it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 195.28it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.33it/s]\n",
      "Iteration:   1%|          | 8/1029 [00:00<00:13, 75.24it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 29, Loss: 0.0269, Train: 82.44%, Valid: 79.72% Test: 77.22%\n",
      "Training...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:14<00:00, 73.12it/s]\n",
      "Iteration:   2%|▏         | 18/1029 [00:00<00:05, 173.46it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 192.02it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 195.31it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 194.93it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 30, Loss: 0.0304, Train: 83.29%, Valid: 80.09% Test: 75.82%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "import copy\n",
    "\n",
    "model.reset_parameters()\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=args['lr'])\n",
    "loss_fn = torch.nn.BCEWithLogitsLoss()\n",
    "\n",
    "best_model = None\n",
    "best_valid_acc = 0\n",
    "\n",
    "for epoch in range(1, 1 + args[\"epochs\"]):\n",
    "    print('Training...')\n",
    "    loss = train(model, device, train_loader, optimizer, loss_fn)\n",
    "\n",
    "    print('Evaluating...')\n",
    "    train_result = eval(model, device, train_loader, evaluator)\n",
    "    val_result = eval(model, device, valid_loader, evaluator)\n",
    "    test_result = eval(model, device, test_loader, evaluator)\n",
    "\n",
    "    train_acc, valid_acc, test_acc = train_result[dataset.eval_metric], val_result[dataset.eval_metric], test_result[dataset.eval_metric]\n",
    "    if valid_acc > best_valid_acc:\n",
    "        best_valid_acc = valid_acc\n",
    "        best_model = copy.deepcopy(model)\n",
    "    print(f'Epoch: {epoch:02d}, '\n",
    "        f'Loss: {loss:.4f}, '\n",
    "        f'Train: {100 * train_acc:.2f}%, '\n",
    "        f'Valid: {100 * valid_acc:.2f}% '\n",
    "        f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration: 100%|██████████| 1029/1029 [00:05<00:00, 188.74it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 191.28it/s]\n",
      "Iteration: 100%|██████████| 129/129 [00:00<00:00, 190.57it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best model: Train: 82.35%, Valid: 80.93% Test: 77.75%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "train_acc = eval(best_model, device, train_loader, evaluator)[dataset.eval_metric]\n",
    "valid_acc = eval(best_model, device, valid_loader, evaluator)[dataset.eval_metric]\n",
    "test_acc = eval(best_model, device, test_loader, evaluator)[dataset.eval_metric]\n",
    "\n",
    "print(f'Best model: '\n",
    "      f'Train: {100 * train_acc:.2f}%, '\n",
    "      f'Valid: {100 * valid_acc:.2f}% '\n",
    "      f'Test: {100 * test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e7JXsMTBgeOI"
   },
   "source": [
    "# Submission\n",
    "\n",
    "In order to get credit, you must go submit your answers on Gradescope.\n",
    "\n",
    "Also, you need to submit the `ipynb` file of Colab 2, by clicking `File` and `Download .ipynb`. Please make sure that your output of each cell is available in your `ipynb` file."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "machine_shape": "hm",
   "name": "CS224W - Colab 2.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python [conda env:Graph_env] *",
   "language": "python",
   "name": "conda-env-Graph_env-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}