PyTorch-基本功能.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ohBsx6xbr-H"
      },
      "source": [
        "# PyTorch-基本功能\n",
        "\n",
        "## 教學目標\n",
        "\n",
        "這份教學的目標是介紹 PyTorch，撰寫深度學習模型的函式庫。\n",
        "\n",
        "## 適用對象\n",
        "\n",
        "已經有基本的機器學習知識，且擁有 python、`numpy`、`matplotlib` 基礎的學生。\n",
        "\n",
        "若沒有先學過 python，請參考 [python-入門語法](./python-入門語法.ipynb) 教學。\n",
        "\n",
        "若沒有先學過 `numpy`，請參考 [numpy-基本功能](./numpy-基本功能.ipynb) 教學。\n",
        "\n",
        "若沒有先學過 `matplotlib`，請參考 [matplotlib-資料視覺化](./matplotlib-資料視覺化.ipynb) 教學。\n",
        "\n",
        "## 執行時間\n",
        "\n",
        "本教學全部執行時間約為 4.376506090164185 秒。\n",
        "\n",
        "|測試環境|名稱|\n",
        "|-|-|\n",
        "|主機板|X570 AORUS ELITE|\n",
        "|處理器|AMD Ryzen 7 3700X 8-Core Processor|\n",
        "|記憶體|Kingston KHX3200C16D4/16GX|\n",
        "|硬碟|Seagate ST1000DM003-1ER1|\n",
        "|顯示卡|GeForce RTX 3060|\n",
        "|作業系統|Ubuntu 22.04 LTS|"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "sJDTx3HZbr-I"
      },
      "source": [
        "## 大綱\n",
        "\n",
        "- [簡介](#簡介)\n",
        "- [安裝](#安裝)\n",
        "- 🍓 [張量宣告](#張量宣告)\n",
        "- [張量取值](#張量取值)\n",
        "- [張量運算](#張量運算)\n",
        "- 🍓 [創造張量](#創造張量)\n",
        "- 🍓[高維張量運算](#高維張量運算)\n",
        "- 🍓[維度運算](#維度運算)\n",
        "- 🍓🍓 [使用 GPU 運算](#使用-GPU-運算)\n",
        "- 🍓🍓🍓 [深度學習](#深度學習)\n",
        "- [練習](#練習)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_5Hk28KHbr-I"
      },
      "source": [
        "## 簡介\n",
        "\n",
        "根據 [PyTorch 官方網站](https://pytorch.org/)（穩定版 v2.0.1）：\n",
        "\n",
        "> PyTorch is an open source machine learning framework that accelerates the path from research prototyping to production deployment.\n",
        "> \n",
        "> PyTorch 是一個開源的機器學習框架，能夠幫助加速從研究原型到商業應用的轉換過程。\n",
        "\n",
        "![PyTorch usage statistics](https://thegradient.pub/content/images/2019/10/ratio_medium-1.png)\n",
        "\n",
        "根據[統計](https://thegradient.pub/state-of-ml-frameworks-2019-pytorch-dominates-research-tensorflow-dominates-industry/)，PyTorch 在各大機器學習會議使用率逐年上升，使用者選擇 PyTorch 的原因為：\n",
        "\n",
        "- 簡單（Simplicity）\n",
        "    - 使用 `python` 作為介面\n",
        "    - 操作方法與 `numpy` 相似\n",
        "- 好用的介面（Great API）\n",
        "    - 沒有過多的抽象化\n",
        "- 效能（Performance）"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "OwS9cuvXbr-I"
      },
      "source": [
        "## 安裝\n",
        "\n",
        "請參考 [PyTorch 官方網站](https://pytorch.org/get-started/locally/#start-locally)，並選擇適合的環境選項與安裝方法。\n",
        "有 GPU 的人請用 `nvidia-smi` 檢查一下 cuda driver 版本，因為最重要的是裝對符合你 cuda driver 的 PyTorch，你才能用 GPU 加速。\n",
        "\n",
        "本教學使用 `pip` 安裝 `torch`，選項如下。\n",
        "|選項|描述|選擇|\n",
        "|-|-|-|\n",
        "|PyTorch Build|請選**穩定版**避免未知錯誤|`Stable(2.0.1)`|\n",
        "|Your OS|依照**作業系統**來選擇|`Linux`|\n",
        "|Package|安裝 **PyTorch** 使用的方法|`Pip`|\n",
        "|Language|當前執行 **Python** 版本|`Python 3.10.12`|\n",
        "|CUDA|電腦上是否有 **GPU** 且支援 **CUDA 架構**|`11.7`|\n",
        "\n",
        "得到以下安裝指令：\n",
        "\n",
        "```sh\n",
        "pip3 install torch torchvision torchaudio\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A8q-VOUcbr-J",
        "outputId": "4ec36342-b15b-44a4-9e43-056826034819"
      },
      "outputs": [],
      "source": [
        "# 匯入 PyTorch 套件\n",
        "# 在 python 中的介面名稱為 torch\n",
        "import torch\n",
        "# 匯入 numpy 與 matplotlib\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "print(\n",
        "    # 確認 torch 的版本\n",
        "    f'PyTorch version {torch.__version__}\\n' +\n",
        "    # 確認是否有 GPU 裝置\n",
        "    f'GPU-enabled installation? {torch.cuda.is_available()}'\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5okJuKPxbr-J"
      },
      "source": [
        "## 張量宣告\n",
        "\n",
        "在 `torch` 中陣列稱為張量（Tensor），創造張量的語法為 `torch.tensor([value1, value2, ...])`。\n",
        "\n",
        "- 每個 `torch.Tensor` 都有不同的**數值型態屬性** `torch.Tensor.dtype`\n",
        "    - 必須透過 `torch.Tensor.dtype` 取得，無法透過 `type()` 取得\n",
        "- 可以指定型態\n",
        "    - 透過參數 `dtype` 指定型態\n",
        "    - 透過 `torch.LongTensor` 創造整數，預設為 `torch.int64`\n",
        "    - 透過 `torch.FloatTensor` 創造浮點數，預設為 `torch.float32`\n",
        "\n",
        "|`torch` 型態|`numpy` 型態|C 型態|範圍|\n",
        "|-|-|-|-|\n",
        "|`torch.int8`|`numpy.int8`|`int_8`|-128~127|\n",
        "|`torch.int16`|`numpy.int16`|`int_16`|-32768~32767|\n",
        "|`torch.int32`|`numpy.int32`|`int_32`|-2147483648~2147483647|\n",
        "|`torch.int64`|`numpy.int64`|`int_64`|-9223372036854775808~9223372036854775807|\n",
        "|`torch.float32`|`numpy.float32`|`float`||\n",
        "|`torch.float64`|`numpy.float64`|`double`||\n",
        "\n",
        "- 每個 `torch.Tensor` 都有**維度屬性** `torch.Size`\n",
        "    - 呼叫 `torch.Tensor.size()` 來取得維度屬性\n",
        "    - `torch.Tensor.size` 本質是 `tuple`\n",
        "    - 張量維度愈高，`len(torch.Tensor.size)` 數字愈大\n",
        "- 可以使用 `torch.Tensor.reshape` 或 `torch.Tensor.view` 進行維度變更\n",
        "    - 變更後的維度必須要與變更前的維度乘積相同\n",
        "    - 變更後的內容為 **shallow copy**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8KGb8WQKbr-J",
        "outputId": "a05f52cc-4cf6-4ce6-baa0-73ce6a7ce6c0"
      },
      "outputs": [],
      "source": [
        "# 張量宣告\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t1 = torch.tensor([1, 2, 3])                           \n",
        "# 輸出 Tensor\n",
        "print(t1)                                              \n",
        "# 輸出 True\n",
        "print(type(t1) == torch.Tensor)                        \n",
        "# 輸出 torch.int64\n",
        "print(t1.dtype)                                        \n",
        "print()\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t2 = torch.tensor([1., 2., 3.])                        \n",
        "# 輸出 Tensor\n",
        "print(t2)                                              \n",
        "# 輸出 True\n",
        "print(type(t2) == torch.Tensor)                        \n",
        "# 輸出 torch.float32\n",
        "print(t2.dtype)                                        \n",
        "print()\n",
        "\n",
        "# 各種 dtype\n",
        "# 輸出 torch.int8\n",
        "print(torch.tensor([1, 2], dtype=torch.int8).dtype)\n",
        "x = torch.tensor([1, 2], dtype=torch.int8)\n",
        "print(x)\n",
        "try:\n",
        "    x[0] = 128\n",
        "except Exception as e:\n",
        "    # int8 的範圍為 -128 ~ 127\n",
        "    # RuntimeError: value cannot be converted to type int8_t without overflow\n",
        "    print(e)\n",
        "print()\n",
        "\n",
        "# 輸出 torch.int16\n",
        "print(torch.tensor([1, 2], dtype=torch.int16).dtype)   \n",
        "# 輸出 torch.int32\n",
        "print(torch.tensor([1, 2], dtype=torch.int32).dtype)   \n",
        "# 輸出 torch.int64\n",
        "print(torch.tensor([1, 2], dtype=torch.int64).dtype)   \n",
        "# 輸出 torch.float32\n",
        "print(torch.tensor([1, 2], dtype=torch.float32).dtype) \n",
        "# 輸出 torch.float64\n",
        "print(torch.tensor([1, 2], dtype=torch.float64).dtype) \n",
        "print()\n",
        "\n",
        "# 宣告 LongTensor 變數 -> 通常 label 會使用這種型態，因為 label 通常是整數，在 torch 訓練的錯誤時可能會看到類似這種錯誤：\n",
        "# RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' \n",
        "# 此時就要記得檢查是否有使用到 LongTensor\n",
        "t3 = torch.LongTensor([1, 2, 3])                       \n",
        "# 輸出 torch.int64\n",
        "print(t3.dtype)                                        \n",
        "\n",
        "# 宣告 FloatTensor 變數\n",
        "t4 = torch.FloatTensor([1, 2, 3])                      \n",
        "# 輸出 torch.float32\n",
        "print(t4.dtype)                                        "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Kg9RgMObbr-K",
        "outputId": "7ce67e53-f074-4cad-f00f-3958a192bef4",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "# size 屬性\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t5 = torch.tensor([               \n",
        "    [1, 2, 3],\n",
        "    [4, 5, 6],\n",
        "    [7, 8, 9],\n",
        "    [10, 11, 12],\n",
        "])\n",
        "\n",
        "# 輸出 Tensor\n",
        "print(t5)                         \n",
        "# 輸出 t5.size (4, 3)\n",
        "print(t5.size())                  \n",
        "print()\n",
        "\n",
        "# 重新更改 t5.size\n",
        "print(t5.reshape(3, 4))           \n",
        "# 輸出更改後的維度 (3, 4)\n",
        "print(t5.reshape(3, 4).size())    \n",
        "print()\n",
        "# 重新更改 t5.size\n",
        "print(t5.view(3, 4))              \n",
        "# 輸出更改後的維度 (3, 4)\n",
        "print(t5.view(3, 4).size())       \n",
        "print()\n",
        "\n",
        "# 重新更改 t5.size\n",
        "print(t5.reshape(2, 6))           \n",
        "# 輸出更改後的維度 (2, 6)\n",
        "print(t5.reshape(2, 6).size())    \n",
        "print()\n",
        "# 重新更改 t5.size\n",
        "print(t5.view(2, 6))              \n",
        "# 輸出更改後的維度 (2, 6)\n",
        "print(t5.view(2, 6).size())       \n",
        "print()\n",
        "\n",
        "# 重新更改 t5.size\n",
        "print(t5.reshape(2, 3, 2))        \n",
        "# 輸出更改後的維度 (2, 3, 2)\n",
        "print(t5.reshape(2, 3, 2).size()) \n",
        "print()\n",
        "# 重新更改 t5.size\n",
        "print(t5.view(2, 3, 2))           \n",
        "# 輸出更改後的維度 (2, 3, 2)\n",
        "print(t5.view(2, 3, 2).size())  \n",
        "# 自動計算第一個維度\n",
        "print(t5.view(-1, 3, 2).size())  \n",
        "print()\n",
        "\n",
        "# 對 t5 進行轉置\n",
        "print(t5.transpose(1, 0))\n",
        "# 輸出轉置後的維度 (3, 4) 元素的順序會改變\n",
        "print(t5.transpose(1, 0).is_contiguous()) \n",
        "# reshape 可以對轉置後的 Tensor 進行操作\n",
        "print(t5.transpose(1, 0).reshape(6, 2))\n",
        "print()\n",
        "# view 不能對轉置後的 Tensor 進行操作\n",
        "try:\n",
        "    print(t5.transpose(1, 0).view(6, 2))\n",
        "except Exception as e:\n",
        "    print(e)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "CaPUOTC7dhxd"
      },
      "source": [
        "## view 和 reshape \n",
        "- 參考[官方網站連結](https://pytorch.org/docs/stable/tensor_view.html)，PyTorch allows a tensor to be a View of an existing tensor. `View` tensor shares the same underlying data with its base tensor. Supporting View avoids explicit data copy, thus allows us to do fast and memory efficient reshaping, slicing and element-wise operations.\n",
        "\n",
        "- `torch.Tensor.view(*shape)-> Tensor`: \n",
        "Returns a **new tensor with the same data** as the self tensor but of a different shape. The returned tensor shares the same data and must have the same number of elements, but may have a different size. For a tensor to be viewed, the new view size must be compatible with its original size and stride, i.e., each new view dimension must either be a subspace of an original dimension, or only span across original dimensions $d,d+1,…,d+k$ that satisfy the following contiguity-like condition that $\\forall i = d, ... d+k-1$\n",
        "$$\n",
        "stride[i] = stride[i+1] \\times size[i+1]\n",
        "$$\n",
        "\n",
        "- `*torch.Tensor.reshape(shape)->Tensor`: \n",
        "Returns a tensor with the same data and number of elements as self but with the specified shape. **This method returns a view if shape is compatible with the current shape. See `torch.Tensor.view()` on when it is possible to return a view.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uy1L8FOqbr-K"
      },
      "source": [
        "## 張量取值\n",
        "\n",
        "與 `numpy` 語法概念相似。\n",
        "\n",
        "- 使用 `torch.Tensor[位置]` 來取得 `torch.Tensor` 中指定位置的值\n",
        "    - 若為**多個維度**的張量，則使用 `tuple` 來取得指定位置的值\n",
        "    - 若位置為**負數**，則等同於反向取得指定位置的值\n",
        "    - 取出的值會以 `torch.Tensor.dtype` 的形式保留\n",
        "- 使用 `torch.Tensor[起始位置:結束位置]` 來取得 `torch.Tensor` 中的部分**連續**值\n",
        "    - **包含起始位置**的值\n",
        "    - **不包含結束位置**的值\n",
        "    - 取出的值會以 `torch.Tensor` 的形式保留\n",
        "- 使用 `torch.Tensor[iterable]`（例如 `list`, `tuple` 等）來取得**多個** `torch.Tensor` 中的值\n",
        "    - 取出的值會以 `torch.Tensor` 的形式保留\n",
        "- 使用判斷式來取得 `torch.Tensor` 中的部份資料\n",
        "    - 經由判斷式所得結果也為 `torch.Tensor`\n",
        "    - 判斷式所得結果之 `torch.Tensor.dtype` 為**布林值** `bool`（`True` 或 `False`）\n",
        "    - 取出的值會以 `torch.Tensor` 的形式保留"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "214IwVJEbr-K",
        "outputId": "daaee408-eeb4-4836-92c5-31eb8a5ffa20"
      },
      "outputs": [],
      "source": [
        "# 張量取值\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t6 = torch.tensor([ \n",
        "    [0, 1, 2],\n",
        "    [3, 4, 5],\n",
        "    [6, 7, 8],\n",
        "    [9, 10, 11],\n",
        "])\n",
        "\n",
        "# 輸出張量 t6 中的第 0 個位置的值 [0, 1, 2]\n",
        "print(t6[0])        \n",
        "# 輸出張量 t6 中的第 1 個位置的值 [3, 4, 5]\n",
        "print(t6[1])        \n",
        "# 輸出張量 t6 中的第 1 個位置的值 [6, 7, 8]\n",
        "print(t6[2])        \n",
        "# 輸出張量 t6 中的第 -2 個位置的值 [6, 7, 8]\n",
        "print(t6[-2])       \n",
        "# 輸出張量 t6 中的第 -1 個位置的值 [9, 10, 11]\n",
        "print(t6[-1])       \n",
        "print()\n",
        "\n",
        "# 輸出張量 t6 中的第 [0, 0] 個位置的值 0\n",
        "print(t6[0, 0])     \n",
        "# 輸出張量 t6 中的第 [0, 1] 個位置的值 1\n",
        "print(t6[0, 1])     \n",
        "# 輸出張量 t6 中的第 [1, 1] 個位置的值 4\n",
        "print(t6[1, 1])     \n",
        "# 輸出張量 t6 中的第 [1, 2] 個位置的值 5\n",
        "print(t6[1, 2])     \n",
        "# 輸出張量 t6 中的第 [-1, -1] 個位置的值 11\n",
        "print(t6[-1, -1])   \n",
        "# 輸出張量 t6 中的第 [-1, -2] 個位置的值 10\n",
        "print(t6[-1, -2])   \n",
        "# 輸出張量 t6 中的第 [-2, -1] 個位置的值 8\n",
        "print(t6[-2, -1])   "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3TTmmlRLbr-K",
        "outputId": "4e0a0eee-0766-44c6-badb-1432de18717d"
      },
      "outputs": [],
      "source": [
        "# 取連續值\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t7 = torch.tensor([ \n",
        "    0, 10, 20, 30, 40, \n",
        "    50, 60, 70, 80, 90\n",
        "])\n",
        "\n",
        "# 輸出張量 t7 位置 0, 1, 2 但是不含位置 3 的值 [0, 10, 20]\n",
        "print(t7[0:3])      \n",
        "# 輸出張量 t7 位置 7, 8, 9 的值 [70, 80, 90]\n",
        "print(t7[7:])       \n",
        "# 輸出張量 t7 位置 0, 1 但是不含位置 2 的值 [0, 10]\n",
        "print(t7[:2])       \n",
        "# 輸出張量 t7 所有位置的值 [0, 10, 20, 30, 40, 50, 60, 70, 80, 90]\n",
        "print(t7[:])        \n",
        "print()\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t8 = torch.tensor([ \n",
        "    [0, 1, 2],\n",
        "    [3, 4, 5],\n",
        "    [6, 7, 8],\n",
        "    [9, 10, 11],\n",
        "])\n",
        "\n",
        "# 輸出張量 t8 位置 0, 1, 但是不含位置 2 的值 [[0, 1, 2], [3, 4, 5]]\n",
        "print(t8[0:2])      \n",
        "print()\n",
        "\n",
        "# 輸出張量 t8 位置 1, 2, 3 的值 [[3, 4, 5], [6, 7, 8], [9, 10, 11]]\n",
        "print(t8[1:])       \n",
        "print()\n",
        "\n",
        "# 輸出張量 t8 位置 0 但是不含位置 1 的值 [[0, 1, 2]]\n",
        "print(t8[:1])       \n",
        "print()\n",
        "\n",
        "# 輸出張量 t8 位置 0 但是不含位置 1 的值 [[0, 1, 2], [3, 4, 5], [6, 7, 8], [9, 10, 11]]\n",
        "print(t8[:])        "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6q5w2A_hbr-K",
        "outputId": "68f5efa5-01ab-4fd1-93b8-6eae16e59691"
      },
      "outputs": [],
      "source": [
        "# 使用 iterable 取得多個值\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t9 = torch.tensor([        \n",
        "    0, 10, 20, 30, 40, \n",
        "    50, 60, 70, 80, 90\n",
        "])\n",
        "\n",
        "# 輸出張量 t9 中偶數位置的值 [0, 20, 40, 60, 80]\n",
        "print(t9[[0, 2, 4, 6, 8]]) \n",
        "print()\n",
        "# 輸出張量 t9 中奇數位置的值 [10, 30, 50, 70, 90]\n",
        "print(t9[[1, 3, 5, 7, 9]]) \n",
        "print()\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t10 = torch.tensor([       \n",
        "    [1, 2, 3, 4],\n",
        "    [5, 6, 7, 8],\n",
        "    [9, 10, 11, 12]\n",
        "])\n",
        "\n",
        "# 輸出張量 t10[0] 與 t10[1] 的值 [[1, 2, 3, 4] [5, 6, 7, 8]]\n",
        "print(t10[[0, 1]])         \n",
        "print()\n",
        "# 輸出張量 t10[0, 2] 與 t10[1, 3] 的值 [3, 8]\n",
        "print(t10[[0, 1], [2, 3]]) "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tJBfUCkxbr-K",
        "outputId": "13257ce4-76d5-42cb-9b7f-7a45331e1c8d"
      },
      "outputs": [],
      "source": [
        "# 判斷式取值\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t11 = torch.tensor([      \n",
        "    0, 10, 20, 30, 40, \n",
        "    50, 60, 70, 80, 90\n",
        "])\n",
        "\n",
        "# 輸出每個值是否大於 50 的 `torch.Tensor`\n",
        "print(t11 > 50)           \n",
        "# 輸出 torch.bool\n",
        "print((t11 > 50).dtype)   \n",
        "# 輸出大於 50 的值 [60, 70, 80, 90]\n",
        "print(t11[t11 > 50])      \n",
        "# 輸出除以 20 餘數為 0 的值 [0, 20, 40, 60, 80]\n",
        "print(t11[t11 % 20 == 0]) "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "x72LPvvtbr-K"
      },
      "source": [
        "## 張量運算\n",
        "\n",
        "### 純量運算（Scalar Operation）\n",
        "\n",
        "對張量內所有數值與單一純量（Scalar）進行相同計算。\n",
        "\n",
        "|符號|意義|\n",
        "|-|-|\n",
        "|`torch.Tensor + scalar`|張量中的每個數值加上 `scalar`|\n",
        "|`torch.Tensor - scalar`|張量中的每個數值減去 `scalar`|\n",
        "|`torch.Tensor * scalar`|張量中的每個數值乘上 `scalar`|\n",
        "|`torch.Tensor / scalar`|張量中的每個數值除以 `scalar`|\n",
        "|`torch.Tensor // scalar`|張量中的每個數值除以 `scalar` 所得之商|\n",
        "|`torch.Tensor % scalar`|張量中的每個數值除以 `scalar` 所得之餘數|\n",
        "|`torch.Tensor ** scalar`|張量中的每個數值取 `scalar` 次方|\n",
        "\n",
        "### 個別數值運算（Element-wise Operation）\n",
        "\n",
        "若兩個張量想要進行運算，則兩個張量的**維度必須相同**（即兩張量之 `torch.size()` 相同）。\n",
        "\n",
        "|符號|意義|\n",
        "|-|-|\n",
        "|`A + B`|張量 `A` 中的每個數值加上張量 `B` 中相同位置的數值|\n",
        "|`A - B`|張量 `A` 中的每個數值減去張量 `B` 中相同位置的數值|\n",
        "|`A * B`|張量 `A` 中的每個數值乘上張量 `B` 中相同位置的數值|\n",
        "|`A / B`|張量 `A` 中的每個數值除以張量 `B` 中相同位置的數值|\n",
        "|`A // B`|張量 `A` 中的每個數值除以張量 `B` 中相同位置的數值所得之商|\n",
        "|`A % B`|張量 `A` 中的每個數值除以張量 `B` 中相同位置的數值所得之餘數|\n",
        "|`A ** B`|張量 `A` 中的每個數值取張量 `B` 中相同位置的數值之次方|\n",
        "\n",
        "### 個別數值函數運算（Element-wise Functional Operation）\n",
        "\n",
        "若想對張量中的**所有數值**進行**相同函數運算**，必須透過 `torch` 提供的介面進行。\n",
        "\n",
        "|函數|意義|\n",
        "|-|-|\n",
        "|`torch.sin`|張量中的每個數值 $x$ 計算 $\\sin(x)$|\n",
        "|`torch.cos`|張量中的每個數值 $x$ 計算 $\\cos(x)$|\n",
        "|`torch.tan`|張量中的每個數值 $x$ 計算 $\\tan(x)$|\n",
        "|`torch.exp`|張量中的每個數值 $x$ 計算 $e^{x}$|\n",
        "|`torch.log`|張量中的每個數值 $x$ 計算 $\\log x$\n",
        "|`torch.ceil`|張量中的每個數值 $x$ 計算 $\\left\\lceil x \\right\\rceil$\n",
        "|`torch.floor`|張量中的每個數值 $x$ 計算 $\\left\\lfloor x \\right\\rfloor$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NR54kMUdbr-L",
        "outputId": "be0f0dd1-9f3c-46ea-f7b9-3b97415423e1"
      },
      "outputs": [],
      "source": [
        "# 純量運算(Scalar Operation)\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t12 = torch.tensor([ \n",
        "    [0, 10, 20],\n",
        "    [30, 40, 50],\n",
        "    [60, 70, 80],\n",
        "    [90, 100, 110],\n",
        "])\n",
        "\n",
        "# 輸出張量 t12\n",
        "print(t12)           \n",
        "print()\n",
        "# 對張量 t12 所有數值加 5\n",
        "print(t12 + 5)       \n",
        "print()\n",
        "# 對張量 t12 所有數值減 4\n",
        "print(t12 - 4)       \n",
        "print()\n",
        "# 對張量 t12 所有數值乘 3\n",
        "print(t12 * 3)       \n",
        "print()\n",
        "# 對張量 t12 所有數值除以 10\n",
        "print(t12 / 10)      \n",
        "print()\n",
        "# 對張量 t12 所有數值除以 10 所得整數部份\n",
        "print(t12 // 10)     \n",
        "print()\n",
        "# 對張量 t12 所有數值除以 7 得到餘數\n",
        "print(t12 % 7)       \n",
        "print()\n",
        "# 對張量 t12 所有數值取 2 次方\n",
        "print(t12 ** 2)      "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "j0xQnAE0br-L",
        "outputId": "15ee2e34-64e9-4f31-b6eb-145ce11aac74"
      },
      "outputs": [],
      "source": [
        "# 個別數值運算\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t13 = torch.tensor([ \n",
        "    [1, 2, 3],\n",
        "    [4, 5, 6]\n",
        "])\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t14 = torch.tensor([ \n",
        "    [6, 5, 4],\n",
        "    [3, 2, 1]\n",
        "])\n",
        "\n",
        "# 張量相加\n",
        "print(t13 + t14)     \n",
        "print()\n",
        "# 張量相減\n",
        "print(t13 - t14)     \n",
        "print()\n",
        "# 張量相乘\n",
        "print(t13 * t14)     \n",
        "print()\n",
        "# 張量相除\n",
        "print(t13 / t14)     \n",
        "print()\n",
        "# 張量相除取商\n",
        "print(t13 // t14)    \n",
        "print()\n",
        "# 張量相除取餘數\n",
        "print(t13 % t14)     \n",
        "print()\n",
        "# 張量 A 取張量 B 次方\n",
        "print(t13 ** t14)   "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GenMDYK4br-L",
        "outputId": "0c79e347-6861-4d68-af5d-12c23500b19d"
      },
      "outputs": [],
      "source": [
        "# 個別數值函數運算\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t15 = torch.tensor([               \n",
        "    [0,     np.pi / 4,     np.pi / 2,     np.pi / 4 * 3],\n",
        "    [np.pi, np.pi / 4 * 5, np.pi / 2 * 3, np.pi / 4 * 7]\n",
        "])\n",
        "\n",
        "# 張量所有數值計算 sine\n",
        "print(torch.sin(t15))              \n",
        "print()\n",
        "# 張量所有數值計算 cosine\n",
        "print(torch.cos(t15))              \n",
        "print()\n",
        "# 張量所有數值計算 tangent\n",
        "print(torch.tan(t15))              \n",
        "print()\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t16 = torch.tensor([               \n",
        "    [1., 2., 3.],\n",
        "    [4., 5., 6.]\n",
        "])\n",
        "\n",
        "# 張量所有數值取指數\n",
        "print(torch.exp(t16))              \n",
        "print()\n",
        "# 張量所有數值取對數\n",
        "print(torch.log(t16))              \n",
        "print()\n",
        "# 張量所有數值取對數後無條件進位\n",
        "print(torch.ceil(torch.log(t16)))  \n",
        "print()\n",
        "# 張量所有數值取對數後無條件捨去\n",
        "print(torch.floor(torch.log(t16))) "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "RqUbDIOiv4Uk"
      },
      "source": [
        "\n",
        "### 🚧 **張量自動擴充（Broadcasting)**\n",
        "\n",
        "若張量 `A` 的維度為 `(a1, a2, ..., an)`（即 `A.size() == (a1, a2, ..., an)`），則張量 `B` 在滿足以下其中一種條件時即可與張量 `A` 進行運算：\n",
        "\n",
        "- 張量 `B` 與張量 `A` 維度完全相同（即 `B.size() == (a1, a2, ..., an)`）\n",
        "- 張量 `B` 為純量（即 `B.size() == (1,)`）\n",
        "- 張量 `B` 的維度為 `(b1, b2, ..., bn)`，若 `ai != bi`，則 `ai == 1` 或 `bi == 1`\n",
        "    - 從**最後**一個維度開始比較\n",
        "    - 如果有任何一個維度無法滿足前述需求，則會得到 `ValueError`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# 張量自動擴充\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t17 = torch.tensor([ \n",
        "    [\n",
        "        [1, 2],\n",
        "        [3, 4],\n",
        "        [5, 6],\n",
        "    ],\n",
        "    [\n",
        "        [7, 8],\n",
        "        [9 ,10],\n",
        "        [11, 12]\n",
        "    ]\n",
        "])\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t18 = torch.tensor([ \n",
        "    [\n",
        "        [1],\n",
        "        [1],\n",
        "        [1]\n",
        "    ],\n",
        "    [\n",
        "        [2],\n",
        "        [2],\n",
        "        [2]\n",
        "    ],\n",
        "])\n",
        "\n",
        "# 輸出張量 t17 維度\n",
        "print(t17.size())    \n",
        "# 輸出張量 t18 維度\n",
        "print(t18.size())    \n",
        "print()\n",
        "# 張量 t17 與張量 t17 維度相同，所以可以直接運算\n",
        "print(t17 + t17)     \n",
        "print()\n",
        "# 張量 t17 與張量 t18 可以擴充成相同維度，所以可以運算\n",
        "print(t17 + t18) "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5ryiNnF9wDOr",
        "outputId": "49c0c912-b68d-4ac4-db99-6d22f6e7c9c5"
      },
      "outputs": [],
      "source": [
        "# 張量 t17 與張量 t18 可以擴充成相同維度，所以可以運算\n",
        "print(t17 + t18)  \n",
        "t18_like = torch.tensor([ \n",
        "    [\n",
        "        [1, 1],\n",
        "        [1, 1],\n",
        "        [1, 1]\n",
        "    ],\n",
        "    [\n",
        "        [2, 2],\n",
        "        [2, 2],\n",
        "        [2, 2]\n",
        "    ],\n",
        "])\n",
        "print()\n",
        "\n",
        "print(t18_like.size())\n",
        "print()\n",
        "\n",
        "# 輸出 True，因為 t18_like 與 t18 擴充後維度相同\n",
        "print(torch.equal(t18 + t17, t18_like + t17))\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "pdmIDslGbr-L"
      },
      "source": [
        "## 創造張量\n",
        "\n",
        "### 賦值（Assignment）\n",
        "\n",
        "使用 `=` 賦與指定位置數值。可以使用 `iterable` 一次指定多個位置。\n",
        "\n",
        "|符號|意義|\n",
        "|-|-|\n",
        "|`=`|賦值|\n",
        "|`+=`|進行加法後賦值|\n",
        "|`-=`|進行減法後賦值|\n",
        "|`*=`|進行乘法後賦值|\n",
        "\n",
        "### 隨機（Random）\n",
        "\n",
        "創造出新的張量，所有數值皆為**隨機決定**，必須**事先指定張量維度**。\n",
        "\n",
        "|函數|意義|用途|備註|\n",
        "|-|-|-|-|\n",
        "|`torch.empty`|創造隨機未初始化張量|已確認維度，尚未確認數值|無法控制隨機|\n",
        "|`torch.rand`|創造隨機浮點數張量，並符合均勻分佈|需要隨機浮點數時|透過均勻分佈決定亂數，範圍介於 0 到 1之間|\n",
        "|`torch.randn`|創造隨機浮點數張量，並符合常態分佈|需要符合常態分佈的隨機浮點數時|透過常態分佈決定亂數，$\\mu = 0$ 且 $\\sigma = 1$|\n",
        "|`torch.randint`|創造隨機整數張量|需要隨機整數時|透過均勻分佈決定亂數，可以控制隨機範圍|\n",
        "\n",
        "### 指定數值（Filled In）\n",
        "\n",
        "**快速創造**擁有特定數值的張量，必須**事先指定張量維度**。\n",
        "\n",
        "|函數|意義|用途|\n",
        "|-|-|-|\n",
        "|`torch.zeros`|創造指定維度大小的張量，所有數值初始化為 0|快速初始化|\n",
        "|`torch.zeros_like`|複製指定張量的維度，創造出新的張量，所有數值初始化為 0|複製張量並初始化|\n",
        "|`torch.ones`|創造指定維度大小的張量，所有數值初始化為 1|快速初始化|\n",
        "|`torch.ones_like`|複製指定張量的維度，創造出新的張量，所有數值初始化為 1|複製張量並初始化|\n",
        "|`torch.full`|創造指定維度大小的張量，所有數值初始化為指定數值|快速初始化|\n",
        "|`torch.full_like`|複製指定張量的維度，創造出新的張量，所有數值初始化為指定數值|複製張量並初始化|\n",
        "|`torch.eye`|創造單位矩陣|矩陣微分|\n",
        "|`torch.arange`|列舉數字|等同於 `list(range(value))`|\n",
        "\n",
        "### 從 numpy 轉換\n",
        "\n",
        "可以使用 `torch.tensor()` 將 `numpy.ndarray` 轉換成 `torch.Tensor`；\n",
        "使用 `torch.numpy()` 將 `torch.Tensor` 轉換成 `numpy.ndarray`。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YvxSd3X1br-L",
        "outputId": "99dc0132-e0ca-4054-8228-691b67b01261"
      },
      "outputs": [],
      "source": [
        "# 賦值\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t19 = torch.tensor([     \n",
        "    [1, 2, 3],\n",
        "    [4, 5, 6],\n",
        "    [7, 8, 9],\n",
        "    [10, 11, 12]\n",
        "])\n",
        "print(t19)\n",
        "print()\n",
        "\n",
        "# 將張量 t19 位置 0 的所有數值改成 1995\n",
        "t19[0] = 1995            \n",
        "print(t19)\n",
        "print()\n",
        "\n",
        "# 將張量 t19 位置 [0, 1] 的所有數值改成 10\n",
        "t19[0, 1] = 10           \n",
        "print(t19)\n",
        "print()\n",
        "\n",
        "# 將張量 t19 位置 [2, 1] 與 [0, 2] 的所有數值改成 12\n",
        "t19[[2, 0], [1, 2]] = 12 \n",
        "print(t19)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d6LzdLlgbr-L",
        "outputId": "86617d59-d3f7-4d96-a078-62240a2313da"
      },
      "outputs": [],
      "source": [
        "# 宣告 Tensor 變數\n",
        "t20 = torch.tensor([      \n",
        "    [1, 2, 3],\n",
        "    [4, 5, 6],\n",
        "    [7, 8, 9],\n",
        "    [10, 11, 12]\n",
        "])\n",
        "print(t20)\n",
        "print()\n",
        "\n",
        "# 將張量 t20 位置 0 的所有數值加上 1995\n",
        "t20[0] += 1995            \n",
        "print(t20)\n",
        "print()\n",
        "\n",
        "# 將張量 t20 位置 [0, 1] 的所有數值減掉 10\n",
        "t20[0, 1] -= 10           \n",
        "print(t20)\n",
        "print()\n",
        "\n",
        "# 將張量 t20 位置 [2, 1] 與 [0, 2] 的所有數值乘上 12\n",
        "t20[[2, 0], [1, 2]] *= 12\n",
        "print(t20)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "M2p5aidybr-L",
        "outputId": "8c79bb15-c76b-44b0-f85d-ac06766fa457"
      },
      "outputs": [],
      "source": [
        "# 隨機\n",
        "\n",
        "# 隨機創造維度為 (2, 3) 的張量，數值為無法控制範圍的浮點\n",
        "print(torch.empty((2, 3)))               \n",
        "print()                                  \n",
        "\n",
        "# 隨機創造維度為 (2, 3) 的張量，數值為介於 0 到 1 之間的浮點\n",
        "print(torch.rand(2, 3))                  \n",
        "print()                                  \n",
        "\n",
        "# 隨機創造維度為 (2, 3) 的張量，數值為介於 0 到 10 之間的浮點\n",
        "print(torch.rand(2, 3) * 10)             \n",
        "print()                                  \n",
        "\n",
        "# 隨機創造維度為 (2, 3) 的張量，數值為介於 -5 到 5 之間的浮點數\n",
        "print(torch.rand(2, 3) * 10 - 5)         \n",
        "print()                                  \n",
        "\n",
        "# 隨機創造維度為 (2, 3) 的張量，分佈為平均值為 0 標準差為 1 的常態分佈\n",
        "print(torch.randn(2, 3))                 \n",
        "print()                                  \n",
        "\n",
        "# 隨機創造維度為 (2, 3) 的張量，數值為介於 -5 到 5 之間的浮點數\n",
        "print(torch.randint(-5, 5, size=(2, 3))) \n",
        "                                         "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rmziKBTmbr-L",
        "outputId": "e573b226-6f80-4367-e988-6473282df5f4"
      },
      "outputs": [],
      "source": [
        "# 指定數值\n",
        "# 創造維度為 (2, 3) 的張量，並初始化為 0\n",
        "print(torch.zeros((2, 3)))      \n",
        "print()\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t21 = torch.tensor([            \n",
        "    [1, 2, 3],\n",
        "    [4, 5, 6],\n",
        "])\n",
        "# 複製張量 t21 的維度，創造出新的張量，並初始化為 0\n",
        "print(torch.zeros_like(t21))    \n",
        "print()\n",
        "\n",
        "# 創造維度為 (3, 4) 的張量，並初始化為 1\n",
        "print(torch.ones((3, 4)))       \n",
        "print()\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t22 = torch.tensor([            \n",
        "    [1, 2, 3, 4],\n",
        "    [5, 6, 7, 8],\n",
        "    [9, 10, 11, 12]\n",
        "])\n",
        "# 複製張量 t22 的維度，創造出新的張量，並初始化為 1\n",
        "# like 的概念就是「模仿我丟給你的這個 tensor 的維度」\n",
        "print(torch.ones_like(t22))     \n",
        "print()\n",
        "\n",
        "# 創造維度為 (5, 6) 的張量，並初始化為 420\n",
        "print(torch.full((5, 6), 420))  \n",
        "print()\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t23 = torch.tensor([            \n",
        "    [1, 2, 3, 4, 5, 6],\n",
        "    [7, 8, 9, 10, 11, 12],\n",
        "    [13, 14, 15, 16, 17, 18],\n",
        "    [19, 20, 21, 22, 23, 24],\n",
        "    [25, 26, 27, 28, 29, 30]\n",
        "])\n",
        "# 複製張量 t23 的維度，創造出新的張量，並初始化為 69\n",
        "print(torch.full_like(t23, 69)) "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eJgdH-vabr-L",
        "outputId": "ffab6382-383b-4dde-cddb-b4cd1e29b8c6"
      },
      "outputs": [],
      "source": [
        "# 創造 3x3 單位矩陣\n",
        "print(torch.eye(3))           \n",
        "print()\n",
        "\n",
        "# 從 0 列舉至 10，但不包含 10\n",
        "print(torch.arange(10))       \n",
        "print()\n",
        "\n",
        "# 從 6 列舉至 9，但不包含 9\n",
        "print(torch.arange(6, 9))     \n",
        "print()\n",
        "\n",
        "# 從 4 遞增至 20，但不包含 20，每次遞增 7\n",
        "print(torch.arange(4, 20, 7)) "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BvkT3EX7br-L",
        "outputId": "16ae7f16-37a4-46b8-d714-99fa73b6999d"
      },
      "outputs": [],
      "source": [
        "# 從 numpy 轉換\n",
        "\n",
        "# 宣告 ndarray 變數\n",
        "arr1 = np.array([1., 2., 3.]) \n",
        "# 將 numpy.ndarray 轉換為 torch.Tensor\n",
        "t24 = torch.tensor(arr1)      \n",
        "# 將 torch.Tensor 轉換為 numpy.ndarray\n",
        "arr2 = t24.numpy()            \n",
        "\n",
        "print((\n",
        "    f'original numpy.ndarray: {arr1}, dtype: {arr1.dtype}\\n' + \n",
        "    f'converted torch.Tensor: {t24}, dtype: {t24.dtype}\\n' +\n",
        "    f'converted numpy.ndarray: {arr2}, dtype: {arr2.dtype}'\n",
        "))"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "STyE59Zgbr-M"
      },
      "source": [
        "## 高維張量運算\n",
        "\n",
        "矩陣等同於是維度為 2 的張量。\n",
        "而高維度的張量運算等同於**固定大部分的維度**，只使用**其中的兩個維度進行計算**。\n",
        "\n",
        "### 張量乘法（Tensor Multiplication）\n",
        "\n",
        "令 $A$ 與 $B$ 為兩張量，$A.\\text{size}() = (a_1, a_2, ..., a_{n - 1}, a_n)$, $B.\\text{size}() = (b_1, b_2, ..., b_{n - 1}, b_n)$。定義 $A \\times B$ 如下：\n",
        "\n",
        "$$\n",
        "\\begin{align*}\n",
        "a_i &= b_i \\forall i \\in \\{1, \\dots, n - 2\\} \\\\\n",
        "a_n &= b_{n - 1} \\\\\n",
        "(A \\times B).\\text{size}() &= (d_1, d_2, \\dots, d_{n - 2}, a_{n - 1}, b_n) \\\\\n",
        "&, \\text{where } d_i = a_i = b_i \\forall i \\in \\{1, 2, \\dots, n - 2\\} \\\\\n",
        "(A \\times B)_{d_1, d_2, \\dots, d_{n - 2}, i, j} &=\n",
        "\\begin{cases}\n",
        "\\sum_{k = 1}^{b_{n - 1}} A_{i, k} \\times B_{k, j} & \\text{if } n = 2 \\\\\n",
        "\\sum_{k = 1}^{b_{n - 1}} A_{d_1, d_2, \\dots, d_{n - 2}, i, k} \\times B_{d_1, d_2, \\dots, d_{n - 2}, k, j} & \\text{if } n > 2\n",
        "\\end{cases} \\\\\n",
        "&, \\forall i \\in \\{1, \\dots, a_1\\}, j \\in \\{1, \\dots, b_2\\}\n",
        "\\end{align*}\n",
        "$$\n",
        "例如：以 $A.\\text{size}() = (4, 3)$ 與 $B.\\text{size}() = (3, 2)$ 來說，$(A \\times B).\\text{size}() = (4, 2)$。\n",
        "\n",
        "例如：以 $A.\\text{size}() = (5, 4, 3)$ 與 $B.\\text{size}() = (5, 3, 2)$ 來說，$(A \\times B).\\text{size}() = (5, 4, 2)$。\n",
        "\n",
        "例如：以 $A.\\text{size}() = (1995, 10, 12, 5, 4, 3)$ 與 $B.\\text{size}() = (1995, 10, 12, 5, 3, 2)$ 來說，$(A \\times B).\\text{size}() = (1995, 10, 12, 5, 4, 2)$。\n",
        "\n",
        "在 `torch` 中張量乘法為 `torch.matmul(A, B)`。\n",
        "\n",
        "### 張量轉置（Tensor Transpose）\n",
        "\n",
        "令 $A$ 兩張量，$A.\\text{size}() = (a_1, a_2, ..., a_{n - 1}, a_n)$。定義 $A^{\\top}$ 如下：\n",
        "\n",
        "$$\n",
        "\\begin{align*}\n",
        "A^{\\top} &= (A_{a_1, a_2, \\dots, a_{n - 2}, a_{n - 1}, a_n})^{\\top} \\\\\n",
        "&= A_{a_1, a_2, \\dots, a_{n - 2}, a_n, a_{n - 1}}\n",
        "\\end{align*}\n",
        "$$\n",
        "\n",
        "即交換張量 $A$ 的最後兩個維度。若想要指定不同的維度 $i, j$ 進行轉置，則定義 $A^{\\top_{i, j}}$ 如下：\n",
        "\n",
        "$$\n",
        "\\begin{align*}\n",
        "A^{\\top_{i, j}} &= (A_{a_1, a_2, \\dots, a_i, \\dots, a_j, \\dots, a_n})^{\\top_i, j} \\\\\n",
        "&= A_{a_1, a_2, \\dots, a_j, \\dots, a_i, \\dots, a_n}\n",
        "\\end{align*}\n",
        "$$\n",
        "\n",
        "例如：以 $A.\\text{size}() = (5, 4, 3)$ 來說，$A^{\\top_{1, 2}}.\\text{size}() = (5, 3, 4)$。\n",
        "\n",
        "例如：以 $A.\\text{size}() = (1995, 10, 12, 5, 4, 3)$ 來說，$A^{\\top_{3, 4}}.\\text{size}() = (1995, 10, 12, 4, 5, 3)$。\n",
        "\n",
        "在 `torch` 中張量轉置為 `torch.transpose(A, i, j)`。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O0ntJ0Jzbr-M",
        "outputId": "5faaf6fb-06c5-4cce-e0b8-49bc569ab0f8"
      },
      "outputs": [],
      "source": [
        "# 張量乘法\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t25 = torch.ones(5, 4, 3)    \n",
        "# 宣告 Tensor 變數\n",
        "t26 = torch.ones(5, 3, 2)    \n",
        "# 進行張量乘法\n",
        "t27 = torch.matmul(t25, t26) \n",
        "\n",
        "# 輸出張量 t25 的維度\n",
        "print(t25.size())            \n",
        "# 輸出張量 t26 的維度\n",
        "print(t26.size())            \n",
        "# 輸出張量 t27 的維度\n",
        "print(t27.size())            "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iELV2wlbbr-M",
        "outputId": "c9c89622-150c-4b50-9273-c7ddd4265988"
      },
      "outputs": [],
      "source": [
        "# 張量轉置\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t28 = torch.ones(5, 4, 3)                \n",
        "\n",
        "# 輸出轉置維度 1 與 2 後的維度\n",
        "print(torch.transpose(t28, 1, 2).size()) \n",
        "# 輸出轉置維度 1 與 2 後的維度\n",
        "print(t28.transpose(1, 2).size())        \n",
        "\n",
        "# 輸出轉置維度 0 與 2 後的維度\n",
        "print(torch.transpose(t28, 0, 2).size()) \n",
        "# 輸出轉置維度 0 與 2 後的維度\n",
        "print(t28.transpose(0, 2).size())        "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "uwt4npXsbr-M"
      },
      "source": [
        "## 維度運算\n",
        "\n",
        "### 降維函數（Dimension Decreasing Function）\n",
        "\n",
        "以下函數將會使**輸出**張量維度**小於輸入**張量維度。\n",
        "在機器學習中你幾乎必定會用到 `torch.argmax()`。\n",
        "\n",
        "|函數|意義|\n",
        "|-|-|\n",
        "|`torch.sum`|將所有數值相加|\n",
        "|`torch.max`|取出所有數值中最大者|\n",
        "|`torch.min`|取出所有數值中最小者|\n",
        "|`torch.argmax`|取出所有數值中最大者的位置|\n",
        "|`torch.argmin`|取出所有數值中最小者的位置|\n",
        "|`torch.mean`|取出所有數值的平均值|\n",
        "|`torch.var`|取出所有數值的變異數|\n",
        "|`torch.std`|取出所有數值的標準差|\n",
        "|`torch.squeeze`|移除數字為 1 的維度|\n",
        "\n",
        "### 增維函數（Dimension Increasing Function）\n",
        "\n",
        "以下函數將會使**輸出**張量維度**大於輸入**張量維度。\n",
        "\n",
        "|函數|意義|\n",
        "|-|-|\n",
        "|`torch.cat`|串接多個相同維度的張量|\n",
        "|`torch.unsqueeze`|在指定的維度間增加 1 維度|"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TuN13HnNbr-M",
        "outputId": "c9c9fa52-3349-4cf1-f40e-a77be5b2befa"
      },
      "outputs": [],
      "source": [
        "# 降維函數\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t29 = torch.tensor([            \n",
        "    [1, 2, 3, 4],\n",
        "    [5, 6, 7, 8],\n",
        "    [9, 10, 11, 12]\n",
        "])\n",
        "\n",
        "# 將張量 t29 中所有值相加\n",
        "print(torch.sum(t29))           \n",
        "# 將張量 t29 中所有值相加\n",
        "print(t29.sum())                \n",
        "\n",
        "# 將張量 t29 中依照維度 0 將所有值相加\n",
        "print(torch.sum(t29, dim=0))    \n",
        "# 將張量 t29 中依照維度 0 將所有值相加\n",
        "print(t29.sum(dim=0))           \n",
        "# 將張量 t29 中依照維度 1 將所有值相加\n",
        "print(torch.sum(t29, dim=1))    \n",
        "# 將張量 t29 中依照維度 1 將所有值相加\n",
        "print(t29.sum(dim=1))           \n",
        "\n",
        "# 找出張量 t29 中最大值\n",
        "print(torch.max(t29))          \n",
        "# 找出張量 t29 中最大值\n",
        "print(t29.max())                \n",
        "print()\n",
        "\n",
        "# 依照維度 0 找出張量 t29 中最大值，並回傳最大值與對應位置\n",
        "print(torch.max(t29, dim=0))    \n",
        "print()                         \n",
        "# 依照維度 0 找出張量 t29 中最大值，並回傳最大值與對應位置\n",
        "print(t29.max(dim=0))           \n",
        "print()                         \n",
        "# 依照維度 0 找出張量 t29 中最大值\n",
        "print(torch.max(t29, dim=0)[0]) \n",
        "# 依照維度 0 找出張量 t29 中最大值位置\n",
        "print(torch.max(t29, dim=0)[1]) \n",
        "print()\n",
        "\n",
        "# 找出張量 t29 中最小值\n",
        "print(torch.min(t29))           \n",
        "# 找出張量 t29 中最小值\n",
        "print(t29.min())                \n",
        "print()\n",
        "\n",
        "# 依照維度 1 找出張量 t29 中最小值，並回傳最小值與對應位置\n",
        "print(torch.min(t29, dim=1))    \n",
        "print()                         \n",
        "# 依照維度 1 找出張量 t29 中最小值，並回傳最小值與對應位置\n",
        "print(t29.min(dim=1))           \n",
        "print()                         \n",
        "# 依照維度 1 找出張量 t29 中最小值\n",
        "print(torch.min(t29, dim=1)[0]) \n",
        "# 依照維度 1 找出張量 t29 中最小值位置\n",
        "print(torch.min(t29, dim=1)[1]) "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### `torch.argmax(input, dim, keepdim=False) → LongTensor`\n",
        "[PyTorch Official Documentation](https://pytorch.org/docs/stable/generated/torch.argmax.html)\n",
        "- Parameters:\n",
        "    - input (Tensor) – the input tensor.\n",
        "    - dim (int) – the dimension to reduce. If None, the argmax of the flattened input is returned.\n",
        "    - keepdim (bool) – whether the output tensor has dim retained or not. Ignored if dim=None."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AmWn28FWbr-M",
        "outputId": "00eb6da5-fc1d-4366-92a9-955a6a11ac4f"
      },
      "outputs": [],
      "source": [
        "# 宣告 Tensor 變數\n",
        "t30 = torch.tensor([            \n",
        "    [1, 2, 3, 4],\n",
        "    [5, 6, 7, 8],\n",
        "    [9, 10, 11, 12]\n",
        "])\n",
        "print('shape:', t30.shape)\n",
        "print('=== argmax ===')\n",
        "# 找出張量 t30 中最大值的位置\n",
        "print('- flattened argmax:')\n",
        "print(torch.argmax(t30))        \n",
        "# 找出張量 t30 中最大值的位置\n",
        "# dim (int) – the dimension to reduce. If None, the argmax of the flattened input is returned.\n",
        "print(t30.argmax())             \n",
        "print(\"- reduced along dim 0:\")\n",
        "# 依照維度 0 找出張量 t30 中最大值的位置，\n",
        "# 維度 0: dimension TO REDUCE，代表沿著維度 0 會被攤平，shape (3,4) 的 tensor 會變成 shape (4,) 的 tensor\n",
        "# 沿著 [1,5,9], [2, 6, 10], ... 找最大值的 index。\n",
        "print(torch.argmax(t30, dim=0)) \n",
        "# 依照維度 0 找出張量 t30 中最大值的位置\n",
        "print(t30.argmax(dim=0))        \n",
        "\n",
        "print(\"- reduced along dim 1:\")\n",
        "# 依照維度 1 找出張量 t30 中最大值的位置\n",
        "print(torch.argmax(t30, dim=1)) \n",
        "# 依照維度 1 找出張量 t30 中最大值的位置\n",
        "print(t30.argmax(dim=1))        \n",
        "\n",
        "print('=== argmin ===')\n",
        "# 找出張量 t30 中最小值的位置\n",
        "print(torch.argmin(t30))        \n",
        "# 找出張量 t30 中最小值的位置\n",
        "print(t30.argmin())                    "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YJwYUYHgbr-M",
        "outputId": "1cb7e250-de49-49f5-bf82-7c5a950c23b5"
      },
      "outputs": [],
      "source": [
        "# 宣告 Tensor 變數\n",
        "t31 = torch.tensor([            \n",
        "    [1., 2., 3., 4.],\n",
        "    [5., 6., 7., 8.],\n",
        "    [9., 10., 11., 12.]\n",
        "])\n",
        "\n",
        "# 計算張量 t31 中所有值的平均數\n",
        "print(torch.mean(t31))         \n",
        "# 計算張量 t31 中所有值的平均數\n",
        "print(t31.mean())              \n",
        "\n",
        "# 依照維度 0 計算張量 t31 中所有值的平均數\n",
        "print(torch.mean(t31, axis=0)) \n",
        "# 依照維度 0 計算張量 t31 中所有值的平均數\n",
        "print(t31.mean(axis=0))        \n",
        "# 依照維度 1 計算張量 t31 中所有值的平均數\n",
        "print(torch.mean(t31, axis=1)) \n",
        "# 依照維度 1 計算張量 t31 中所有值的平均數\n",
        "print(t31.mean(axis=1))        \n",
        "\n",
        "# 計算張量 t31 中所有值的變異數\n",
        "print(torch.var(t31))          \n",
        "# 計算張量 t31 中所有值的變異數\n",
        "print(t31.var())               \n",
        "\n",
        "# 依照維度 0 計算張量 t31 中所有值的變異數\n",
        "print(torch.var(t31, axis=0))  \n",
        "# 依照維度 0 計算張量 t31 中所有值的變異數\n",
        "print(t31.var(axis=0))         \n",
        "# 依照維度 1 計算張量 t31 中所有值的變異數\n",
        "print(torch.var(t31, axis=1))  \n",
        "# 依照維度 1 計算張量 t31 中所有值的變異數\n",
        "print(t31.var(axis=1))         \n",
        "\n",
        "# 計算張量 t31 中所有值的標準差\n",
        "print(torch.std(t31))          \n",
        "# 計算張量 t31 中所有值的標準差\n",
        "print(t31.std())               \n",
        "\n",
        "# 依照維度 0 計算張量 t31 中所有值的標準差\n",
        "print(torch.std(t31, axis=0))  \n",
        "# 依照維度 0 計算張量 t31 中所有值的標準差\n",
        "print(t31.std(axis=0))         \n",
        "# 依照維度 1 計算張量 t31 中所有值的標準差\n",
        "print(torch.std(t31, axis=1))  \n",
        "# 依照維度 1 計算張量 t31 中所有值的標準差\n",
        "print(t31.std(axis=1))         "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LfXHwo9-br-M",
        "outputId": "bfaa7ffd-e9da-4a36-96f6-ffd3acfdfdac"
      },
      "outputs": [],
      "source": [
        "# 宣告 Tensor 變數\n",
        "t32 = torch.tensor([        \n",
        "    [1, 2, 3]\n",
        "])\n",
        "\n",
        "# 移除張量 t32 中多餘的維度(為 1 的維度)\n",
        "t32_sq = torch.squeeze(t32) \n",
        "\n",
        "# 輸出張量 t32\n",
        "print(t32)                  \n",
        "# 輸出張量 t32 的維度\n",
        "print('- before squeezing:', t32.size())           \n",
        "# 輸出移除維度後的張量 t32\n",
        "print(t32_sq)               \n",
        "# 輸出移除維度後張量 t32 的維度\n",
        "print('- after squeezing:',t32_sq.size())        "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ROt_wvYWbr-M",
        "outputId": "4a354c6d-1ba0-40b2-e3de-a27452320b0e"
      },
      "outputs": [],
      "source": [
        "# 增維函數\n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t33 = torch.tensor([                  \n",
        "    [1, 2, 3]\n",
        "])\n",
        "\n",
        "# 串接多個張量 t33\n",
        "t33_cat = torch.cat([                 \n",
        "    t33,\n",
        "    t33,\n",
        "    t33,\n",
        "    t33\n",
        "]) \n",
        "\n",
        "# 輸出串接後的張量 t33_cat\n",
        "print(t33_cat)                        \n",
        "# 輸出串接後的張量 t33_cat 維度\n",
        "print(t33_cat.size())                 \n",
        "\n",
        "# 宣告 Tensor 變數\n",
        "t34 = torch.tensor([                  \n",
        "    [1, 2, 3],\n",
        "    [4, 5, 6]\n",
        "])\n",
        "\n",
        "print(t34)\n",
        "print(t34.size())\n",
        "\n",
        "# 對張量 t34 維度 0 增加 1 維\n",
        "t34_usq = torch.unsqueeze(t34, dim=0) \n",
        "\n",
        "# 輸出張量 t34 維度 0 增加 1 維後的結果\n",
        "print(t34_usq)                        \n",
        "# 輸出張量 t34 維度 0 增加 1 維後的維度\n",
        "print(t34_usq.size())                 "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Torch Tensor\n",
        "程式語言框架通常有其主要的資料型態，像是 numpy 中的 ndarray。在 PyTorch 內則是叫做 tensor（張量）的一種資料型態。PyTorch 的所有操作和 numpy 都很相似，但重要的是 tensor 支援 CUDA 的硬體加速（GPU），使得 GPU 深度學習變得簡單可行。tensor 可以在GPU/CPU上傳輸：只要使用 `tensor.cuda(device_id)`\n",
        "即可以將 tensor 移動到第 `device_id` 個（0-indexed）的 GPU 核心上。或者 `tensor.cpu()`可以將 tensor 移動回到 CPU 上。另外一個更通用的方法是 `tensor.to(device)`。\n",
        "\n",
        "--- \n",
        "### 額外補充\n",
        "#### 什麼是 GPU ？\n",
        "GPU全稱為圖形處理器（Graphics Processing Unit），是一種專門進行繪圖運算工作的微處理器。儘管GPU在遊戲中以3D渲染而聞名，但GPU相較於「傳統的專為通用計算而設計的CPU，具有數百或數千個核心，經過優化，可並行運行大量計算，對運行深度學習和機器學習算法尤其有用。GPU允許某些計算機比傳統CPU上運行相同的計算速度快10-100倍。\n",
        "\n",
        "#### 什麼是CUDA？\n",
        "\n",
        "CUDA全稱為計算統一設備架構（Compute Unified Device Architecture），是NVIDIA（輝達）創建的平行計算平臺和應用程序編程接口模型。CUDA 平臺是一個軟件層，可直接訪問GPU的虛擬指令集和並行計算元素，以執行計算內核。因此，如果我們想利用 GPU 加速運行深度學習算法，那麼 CUDA 就是一個不可或缺的中間層，它代替我們直接和GPU硬體打交道，並對外開放接口。而 PyTorch 則對這層接口再次進行封裝，以方便程式設計人員使用。\n",
        "\n",
        "#### 什麼是cuDNN？\n",
        "\n",
        "cuDNN 全稱為 CUDA 深度神經網絡庫（CUDA Deep Neural Network library），是 NVIDIA 打造的針對深度神經網絡的加速庫，是一個用於深層神經網絡的 GPU 加速庫。如果你要使用 GPU 訓練模型，cuDNN 不是必須的，但一般會採用這個加速庫。\n",
        "\n",
        "### 參考資料\n",
        "- [知乎：什麼是張量？& 深度學習](https://zhuanlan.zhihu.com/p/48982978)\n",
        "- [CUDA 入門, nvidia 公司官方網站](https://blogs.nvidia.com.tw/2020/10/27/cuda-refresher-getting-started-with-cuda/)\n",
        "- [CUDA、cuDNN、pytorch 安装分析](https://blog.csdn.net/weixin_38481963/article/details/105313471) \n",
        "- [NQU Tensor 介紹 slides](https://www.nqu.edu.tw/upload/educsie/attachment/529fa35c91b055e7da3c8dc7a9bc975e.pdf)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dpkIiN9Vbr-M"
      },
      "source": [
        "## 使用 GPU 運算\n",
        "\n",
        "上述的所有教學都是在 CPU 上進行運算，而大多數的深度學習框架都會提供操作 GPU 的介面幫助平型化運算。\n",
        "而 `torch` 與大部分的深度學習框架相同，使用 Nvidia 開發的 CUDA（Compute Unified Device Architecture）幫助使用 GPU 進行深度學習的運算（cuDNN）。\n",
        "\n",
        "使用 CUDA 操作平型化運算的流程為：\n",
        "\n",
        "1. 宣告 GPU 運算所需要佔用的記憶體（`cudaMalloc`）\n",
        "2. 定義每個平型化運算節點的運算內容\n",
        "3. 在主記憶體上創造資料（`malloc`）\n",
        "4. 將資料搬移至 GPU 的記憶體（`cudaMemcpy`）\n",
        "5. 每個節點獨立運算\n",
        "6. 將計算結果搬回至主記憶體（`memcpy`）\n",
        "7. 釋放 GPU 的記憶體（`cudaFree`）\n",
        "\n",
        "而在 `torch` 中將以上流程簡化成以下兩種方法\n",
        "\n",
        "- 宣告 `torch.Tensor` 變數時使用 `device='cuda:0'` 參數將變數宣告於 GPU 記憶體第0顆（0-indexed）。\n",
        "- 對已經創造於主記憶體的 `torch.Tensor` 變數使用 `torch.to('cuda:0')` 搬移至 GPU 記憶體第0顆（0-indexed）。\n",
        "```python\n",
        "torch.tensor([1., 2., 3.], device='cuda:0') # 使用 device 參數將變數宣告於 GPU 記憶體\n",
        "torch.tensor([1., 2., 3.]).to('cuda:0')     # 使用 to 將變數搬移至 GPU 記憶體\n",
        "```\n",
        "\n",
        "宣告於 GPU 或搬移至 GPU 後，之後所有的運算便會在 GPU 上進行。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "djuqIr7vbr-M",
        "outputId": "9c908a0d-17f7-4119-eba8-2b31a63bc5ae"
      },
      "outputs": [],
      "source": [
        "# 使用 GPU 運算\n",
        "\n",
        "if torch.cuda.is_available():\n",
        "    # 使用 device 參數創造張量於 GPU 上\n",
        "    t35 = torch.tensor([1., 2., 3.], device='cuda:0') \n",
        "else:\n",
        "    # 如果不支援 cuda 則出現 error\n",
        "    print('torch not compiled with CUDA enabled')     \n",
        "    t35 = None\n",
        "    \n",
        "# 輸出張量 t35\n",
        "print(t35)                                            "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bKtEO_4Nbr-M",
        "outputId": "bc81130a-0364-46f8-f285-4e9dc15e71c7"
      },
      "outputs": [],
      "source": [
        "if torch.cuda.is_available():\n",
        "    # 使用 to 將張量搬移至 GPU 上\n",
        "    t36 = torch.tensor([1., 2., 3.]).to('cuda:0') \n",
        "else:\n",
        "    # 如果不支援 cuda 則出現 error\n",
        "    print('torch not compiled with CUDA enabled') \n",
        "    t36 = None\n",
        "    \n",
        "# 輸出張量 t36\n",
        "print(t36)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "14dZ9N1Ybr-M",
        "outputId": "cfbd5e50-4101-4edb-8ed4-4f57532f2ad0"
      },
      "outputs": [],
      "source": [
        "# 如果有可用 GPU 時採用 GPU cuda:0\n",
        "if torch.cuda.is_available():                   \n",
        "    device = torch.device('cuda:0')\n",
        "# 若無 GPU 可用則使用 CPU\n",
        "else:                                           \n",
        "    device = torch.device('cpu')\n",
        "\n",
        "print(device)\n",
        "\n",
        "# 根據 device 創造張量\n",
        "t37 = torch.tensor([1., 2., 3.], device=device) \n",
        "# 使用 to 搬移張量至指定的裝置\n",
        "t38 = torch.tensor([1., 2., 3.]).to(device)     \n",
        "\n",
        "# 輸出張量 t37\n",
        "print(t37)                                      \n",
        "# 輸出張量 t38\n",
        "print(t38)                                      "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "TSLKGH_vbr-M"
      },
      "source": [
        "## 深度學習\n",
        "\n",
        "![Deep Learning](https://miro.medium.com/max/1000/1*51D0MqtqHu3h2vTE5oJ-7g.png)\n",
        "![Deep Learning vs. Machine Learning](https://learn.microsoft.com/zh-tw/azure/machine-learning/media/concept-deep-learning-vs-machine-learning/ai-vs-machine-learning-vs-deep-learning.png)\n",
        "### 深度學習要幹嘛？\n",
        "上課教的 SVM, Bayes 等等的模型算是 （傳統）Machine Learning 的範疇，比較偏向建立於統計的假設的數學統計模型。而深度學習則是機器學習內的一個子集合（subset），他是以類神經網路來模擬人類大腦的神經元之間的互動。\n",
        "\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "![](https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/03-pytorch-computer-vision-workflow.png)\n",
        "\n",
        "使用 `torch` 進行深度學習主要包含以下步驟：\n",
        "1. 將資料轉換成 `torch.Tensor`，使用 Dataset 和 Dataloader 包裝管理資料。\n",
        "2. 使用 `torch.nn` 建立深度學習模型架構。\n",
        "3. 從 `torch.optim` 選擇最佳化工具。\n",
        "4. 選擇目標函數。\n",
        "5. 訓練深度學習模型（train）。\n",
        "6. 測試深度學習模型（test, inference）。\n",
        "以下範例：使用兩層全連接層做 Polynomial Regression 任務。"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "### 資料集\n",
        "1. 使用 `torch.utils.data.Dataset` 將資料集轉換成 `torch.Tensor`\n",
        "2. 使用 `torch.utils.data.DataLoader` 將資料集以批次（mini-batch）取出\n",
        "3. （Optional）額外定義 `collate_fn` 將抽樣的資料整理成固定的格式\n",
        "\n",
        "\n",
        "我們的資料長這樣：$$y = 2x^2 + 3x + 17$$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TFiFnsh5br-N",
        "outputId": "912e0668-6434-4d31-8111-bb111208753d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "10\n",
            "(tensor(0.7652), tensor(20.4665))\n"
          ]
        }
      ],
      "source": [
        "# 資料集\n",
        "\n",
        "# 匯入資料集 base class\n",
        "from torch.utils.data import Dataset \n",
        "\n",
        "# 繼承 base class 創造資料集\n",
        "class MyDataset(Dataset):            \n",
        "    # 給予資料集大小，並隨機創造資料\n",
        "    def __init__(self, size:int):        \n",
        "        self.x = torch.rand(size) \n",
        "        self.y = 2 * self.x ** 2 + 3 * self.x + 17\n",
        "        \n",
        "    # 定義總資料數\n",
        "    def __len__(self):               \n",
        "        return len(self.x)\n",
        "    \n",
        "    # 定義取出單一資料的方法\n",
        "    def __getitem__(self, index):    \n",
        "        return self.x[index], self.y[index]\n",
        "    \n",
        "# 創造資料集\n",
        "my_dataset = MyDataset(10)           \n",
        "# 取得總資料數\n",
        "print(len(my_dataset))               \n",
        "# 取出單一資料\n",
        "print(my_dataset[0])                 "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### batch (mini-batch)\n",
        "<img src=\"https://miro.medium.com/v2/resize:fit:825/0*FKi8Z3JT-Y2C1yiw\"  width=\"400\">\n",
        "\n",
        "每一次更新參數時，模型以整個 batch 中`batch_size` 筆的資料，看過一遍，然後更新一次參數。一個 epoch 代表模型看過整個資料集一次。\n",
        "最極端的兩個狀況：batch_size = 1, batch_size = 資料集大小\n",
        "- batch_size 小：計算成本昂貴，gradient 噪音多，但對最佳化的方向較為準確。\n",
        "- batch_size 大：計算成本低，gradient 噪音少，但神奇的是（？）實驗結果說對最佳化的方向較不準確。\n",
        "- batch_size 怎麼設？看任務內容（調參技術之一）。\n",
        "相關參考影片:\n",
        "[【機器學習2021】類神經網路訓練不起來怎麼辦 (二)： 批次 (batch) 與動量 (momentum)](https://www.youtube.com/watch?v=zzbr1h9sF54)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 53,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kWP1moMObr-N",
        "outputId": "703e9283-8e0f-4f81-e5c4-8e008071aed2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "torch.Size([3, 1]) torch.Size([3, 1])\n",
            "torch.Size([3, 1]) torch.Size([3, 1])\n",
            "torch.Size([3, 1]) torch.Size([3, 1])\n",
            "torch.Size([1, 1]) torch.Size([1, 1])\n"
          ]
        }
      ],
      "source": [
        "# 匯入資料集抽樣工具\n",
        "from torch.utils.data import DataLoader \n",
        "\n",
        "# 定義格式化的方法\n",
        "def collate_fn(batch):                  \n",
        "    x_list = []\n",
        "    y_list = []\n",
        "    \n",
        "    for x, y in batch:\n",
        "        x_list.append([x])              \n",
        "        y_list.append([y])              \n",
        "    # 最終回傳的維度為 [(batch_size, 1), (batch_size, 1)]\n",
        "    # 最終回傳的維度為 [(batch_size, n_features), (batch_size, label_dim)]\n",
        "    return [torch.tensor(x_list), torch.tensor(y_list)] \n",
        "\n",
        "# 創造 DataLoader 實例\n",
        "batch_size = 3\n",
        "my_data_loader = DataLoader(            \n",
        "    my_dataset,                 # 對資料集 my_dataset 進行抽樣\n",
        "    batch_size=batch_size,      # 設定每次抽樣的數量                 \n",
        "    shuffle=True,               # 設定隨機抽樣                  \n",
        "    collate_fn=collate_fn       # 指定格式化的方法    \n",
        ")\n",
        "\n",
        "# 透過 my_data_loader 對資料集 my_dataset 進行抽樣\n",
        "for x, y in my_data_loader:             \n",
        "    print(x.size(), y.size())\n",
        "    "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Q. 為什麼要將每個 `x` 轉換成 `[x]`？\n",
        "在此方法中，將每個 `x` 轉換成 `[x]` 的形式是因為我們希望每個 `x` 在 tensor 中都有一個單獨的維度，以便後續能夠進行批次運算。\n",
        "如果直接將每個 `x` append 到列表中，則會產生一個沒有額外維度的列表，\n",
        "這樣在後續批次處理時會出現維度不一致的問題，無法進行運算。\n",
        "Try append x, y alone and you will see `RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x3 and 1x10)` in `model.forward()`。\n",
        "#### Q. 為什麼最後一個 batch 內的形狀是 `torch.Size([1, 1]) torch.Size([1, 1])`? `batch_size` 不是 3 嗎？\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "bqeXVb2e0tlT"
      },
      "source": [
        "### 建立模型\n",
        "\n",
        "可以使用 `torch.nn` 現成的模型進行深度學習，``torch`` 提供很多「神經層」（layer），可以用來建立模型的架構（architecture）。\n",
        "- 面對哪個任務應該用什麼模型？\n",
        "- 模型的輸入輸出的形式要如何定義？\n",
        "- 模型要怎麼架構？\n",
        "- 沒辦法一一的講，在[NN中文文本分類](NN-中文文本分類.ipynb)會講怎麼架構一顆 RNN 來做文本分類任務。其他神經層可以去[torch.nn](https://pytorch.org/docs/stable/nn.html)看看，怎麼架構只能先了解原理然後多查多練習：[PyTorch Tutorial](https://pytorch.org/tutorials/)。\n",
        "\n",
        "|模型介面|名稱|常見用途|\n",
        "|-|-|-|\n",
        "|`torch.nn.Linear`|線性層（Linear Layer）|轉換特徵|\n",
        "|`torch.nn.Embedding`|嵌入層（Embedding Layer）|學習特徵向量表達法|\n",
        "|`torch.nn.Conv1d`|1 維卷積層（1-Dimensional Convolution Layer）|抽取連續資料區域特徵|\n",
        "|`torch.nn.Conv2d`|2 維卷積層（2-Dimensional Convolution Layer）|抽取平面圖片區域特徵|\n",
        "|`torch.nn.Conv3d`|3 維卷積層（3-Dimensional Convolution Layer）|抽取立體圖片區域特徵|\n",
        "|`torch.nn.RNN`|循環神經網路（Recurrent Neural Network）|壓縮動態長度文字|\n",
        "|`torch.nn.LSTM`|長短期記憶神經網路（Long Short-Term Memory）|有效壓縮動態長度文字|\n",
        "|`torch.nn.Transformer`|多面向自我注意力機制模型（Multi-Head Self-Attention）|機器翻譯|\n",
        "\n",
        "如果需要使用深度學習模型，必須透過繼承 `torch.nn.Module` 來定義**模型結構**與**運算流程**：\n",
        "使用 `nn.Module` 定義模型時必須要記得以下規則：\n",
        "\n",
        "- 在類別方法 `__init__` 中定義模型結構\n",
        "    - 必須要執行 `super(MyModel, self).__init__()`\n",
        "    - 為什麼需要：[colab link](https://colab.research.google.com/drive/1LL1RVaoGoGYE68HCaMOEVSCR8bEPgqqc?usp=sharing)\n",
        "- 必須透過定義類別方法 `forward` 才能定義計算流程"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "QWrkuLw005aB"
      },
      "source": [
        "\n",
        "### 啟動函數（Activation Functions）\n",
        "\n",
        "若模型需要啟動函數，可以使用 `torch.nn.functional` 中事先定義好的啟動函數：\n",
        "常見的啟動函數包含：\n",
        "\n",
        "|啟動函數|名稱|定義|數值範圍|\n",
        "|-|-|-|-|\n",
        "|`torch.nn.functional.relu`|ReLU|$$f(x_i) = \\max(0, x_i)$$|$$\\mathbb{R}^+$$|\n",
        "|`torch.nn.functional.softmax`|Softmax|$$f(x_i) = \\frac{e^{x_i}}{\\sum_{j = 0}^n e^{x_j}}$$|$$[0, 1]$$|\n",
        "|`torch.nn.functional.sigmoid`|Sigmoid|$$f(x_i) = \\frac{1}{1 + e^{-x_i}}$$|$$[0, 1]$$|\n",
        "|`torch.nn.functional.tanh`|Hyperbolic Tangent|$$f(x_i) = \\frac{e^{x_i} - e^{-x_i}}{e^{x_i} + e^{-x_i}}$$|$$[-1, 1]$$|\n",
        "\n",
        "<img src=\"https://i.imgur.com/2T2K61V.png\"  width=\"600\" height=\"300\">\n",
        "\n",
        "#### Why do we need Activation Functions? \n",
        "所有 Activation functions 的大略概念都是 \"creates a non-linearity in your output\"，它們的存在使模型\n",
        "能夠 fit 在非線性更為複雜的資料上。Sigmoid 是最初始的，但是因為它的數值範圍落在 0 到 0.25，gradient 在後向傳播時會消失，因此漸漸少用（相關影片：[Sigmoid and Vanishing Gradient](https://www.youtube.com/watch?v=kGAo32JgY48)）。ReLU (Rectified Linear Units) 則取而代之，它是把負數先轉為零，正數就什麼都不做直接離開 node。用意是把負值關係排除掉，可以抵抗 gradient vanishing。ReLU 有各種變形：LeakyReLU、alpha ReLU 等等。Tanh 是 Sigmoid 的變形，它的數值範圍是 -1 到 1，因此它的 gradient 不會消失，但是它的輸出是 zero-centered，因此在訓練時會比較難收斂。Softmax $softmax(\\overrightarrow{a}) = \\frac{e^{a_i}}{\\sum_k{e^{a_k}}}$是用來把輸出轉為機率分佈，讓輸出的數值總和為 1，可以用來做（單一/多）分類的機率預測。\n",
        "\n",
        "#### The reasons we need `torch.argmax()`.\n",
        "<img src=\"https://img-blog.csdn.net/20180902220822202?watermark/2/text/aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JpdGNhcm1hbmxlZQ==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70\"  width=\"400\">\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "#### `torch.nn.Linear`\n",
        "![](https://cdn.analyticsvidhya.com/wp-content/uploads/2020/02/Screenshot-from-2020-02-03-22-14-21-300x195.png)\n",
        "- $X$: input \n",
        "- $W, b$: the parameters we would like to fit. \n",
        "- $Z$: the output we would like to get. \n",
        "- If multiple layers are stacked, $Z$ will be the input of the next layer. In general, we would like the FINAL layer's output to be close \n",
        "to the ground truth $Y$. \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OmmUqpKLbr-N",
        "outputId": "8c4252de-75a9-4bac-e425-84fb8e3e4017"
      },
      "outputs": [],
      "source": [
        "# 建立模型\n",
        "\n",
        "# 匯入神經網路模型\n",
        "import torch.nn as nn                   \n",
        "# 匯入啟動函數\n",
        "import torch.nn.functional as F         \n",
        "\n",
        "# 模型需要繼承自 nn.Module\n",
        "class MyModel(nn.Module):               \n",
        "    # 定義模型結構, 輸入層維度, 隱藏層維度, 輸出層維度\n",
        "    def __init__(self,                  \n",
        "                 in_dim,                \n",
        "                 hid_dim,               \n",
        "                 out_dim):              \n",
        "\n",
        "        # 繼承 nn.Module 所有屬性\n",
        "        super(MyModel, self).__init__() \n",
        "        \n",
        "        # 創造線性層 self.layer1\n",
        "        self.layer1 = nn.Linear(        \n",
        "            # 設定線性層輸入維度\n",
        "            in_features=in_dim,         \n",
        "            # 設定線性層輸出維度\n",
        "            out_features=hid_dim        \n",
        "        )\n",
        "        # 創造線性層 self.layer2\n",
        "        self.layer2 = nn.Linear(        \n",
        "            # 設定線性層輸入維度\n",
        "            in_features=hid_dim,        \n",
        "            # 設定線性層輸出維度\n",
        "            out_features=out_dim      \n",
        "        )\n",
        "        \n",
        "    # [Important] 定義運算流程\n",
        "    def forward(self, batch_x): \n",
        "        # Why use ReLU?       \n",
        "        # 使用線性層 self.layer1 輸入 batch_x 計算得到 h\n",
        "                                     # batch_x's shape: (batch_size, in_dim)\n",
        "        h = self.layer1(batch_x)     # h = Wx + b, h's shape: (batch_size, hid_dim)   \n",
        "                                    # 使用 ReLU 啟動函數輸入 h 得到 a\n",
        "        a = F.relu(h)                # a = ReLU(h), a's shape: (batch_size, hid_dim)        \n",
        "                                    # 使用線性層 self.layer2 輸入 a 計算得到 y\n",
        "        y = self.layer2(a)           # y = Wa + b, y's shape: (batch_size, out_dim) \n",
        "        \n",
        "        \n",
        "        # 輸出 y\n",
        "        return y                        \n",
        "    \n",
        "# 創造 MyModel 模型實例\n",
        "my_model = MyModel(                     \n",
        "    # 設定輸入層維度\n",
        "    in_dim=1,                           \n",
        "    # 設定隱藏層維度\n",
        "    hid_dim=10,                         \n",
        "    # 設定輸出層維度\n",
        "    out_dim=1                           \n",
        ")\n",
        "\n",
        "# 透過 my_data_loader 對資料集 my_dataset 進行抽樣\n",
        "for batch_x, batch_y in my_data_loader: \n",
        "    print('batch_x shape:', batch_x.shape)\n",
        "    print('batch_y shape:', batch_y.shape) \n",
        "    pred_y = my_model(batch_x)   # this part calls my_model.forward(batch_x)       \n",
        "    print('pred_y shape:', pred_y.shape) # should be the same as batch_y in most of the cases, sometimes this should be postprocessed \n",
        "    # to match the shape of batch_y, HERE is the simplest case that the shape of pred_y is the same as batch_y\n",
        "    break"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "ACOTX8Jw1Ci2"
      },
      "source": [
        "### 目標函數（Objective Functions）\n",
        "\n",
        "使用 `torch.nn` 中事先定義好的目標函數進行模型最佳化，計算模型預測結果與標記訓練資料的誤差值，並透過向後傳播（Back Propagation）演算法取得相對於誤差值的梯度（Gradient）。\n",
        "- 基本上就是去計算每個參數的偏微分。\n",
        "- 和 gradient descent 是完全相同的概念，只是在神經網路中 neurons 數量龐大，因此需要有有效率的演算法去計算 gradient。這個演算法就是 back propagation。\n",
        "- 使用了 Chain Rule 的概念。計算每個參數內對 cost function 的偏微分。這個 cost function 就是「正確答案與預測答案的距離」，更適當的名字是 loss function。\n",
        "- 對一個 weight matrix $W \\in R^{a \\times b}$ 內，$W_{ij}$ 是 $l$層$i$ 神經元與 $l-1$ 層 $j$ 神經元之間的權重。\n",
        "    - ![](https://i.imgur.com/fk5HCm4.png)\n",
        "- 想更新這個權重值，就要計算 $\\frac{\\partial C}{\\partial W_{ij}}$，其中 $C$ 是 loss function。使用 Chain Rule，可以拆解成 $\\frac{\\partial C}{\\partial W_{ij}} =  \\frac{\\partial C}{\\partial Z_{i}} \\frac{\\partial Z_{i}}{\\partial W_{ij}}$，$Z_i$ 是 $i$ 所在的神經元的輸入值，$a^i$ 是 $i$ 神經元的輸出值。（下圖上標意義解釋：上標 $l$ 是你的層數（你現在更新的是哪一層的權重矩陣？實際上每一層都需要算），$r$ 是第$r$組資料）。所以這兩個 terms 分別要怎麼計算？請去看下面第一個影片。\n",
        "    - ![](https://i.imgur.com/3EihDh7.png)\n",
        "- Resources (a lot of maths): \n",
        "    - [DNN Back propagation](http://speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2015_2/Lecture/DNN%20backprop.ecm.mp4/index.html)\n",
        "    - [Computational Graph 的角度去看 Back propagation](https://www.youtube.com/watch?v=-yhm3WdGFok)\n",
        "\n",
        "```python\n",
        "# 匯入神經網路模型\n",
        "import torch.nn as nn             \n",
        "\n",
        "# 創造均方誤差計算工具\n",
        "criterion = nn.MSELoss()          \n",
        "\n",
        "# 計算 batch_x 得到 pred_y\n",
        "pred_y = my_model(batch_x)        \n",
        "# 計算 pred_y 與 batch_y 的均方誤差\n",
        "loss = criterion(pred_y, batch_y) \n",
        "\n",
        "# 使用向後傳播計算梯度\n",
        "loss.backward()                   \n",
        "```\n",
        "\n",
        "常見的目標函數包含：\n",
        "\n",
        "|目標函數|名稱|\n",
        "|-|-|\n",
        "|`torch.nn.MSELoss`|均方誤差（Mean Square Error）|\n",
        "|`torch.nn.CrossEntropyLoss`|交叉熵（Cross Entropy）|\n",
        "|`torch.nn.BCELoss`|二元交叉熵（Binary Cross Entropy）|\n",
        "|`torch.nn.NLLLoss`|負對數似然（Negative Log Likelihood）|\n",
        "\n",
        "## MSELoss \n",
        "- [PyTorch Documentation](https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html)\n",
        "<img src=\"https://i.imgur.com/dqKaK25.png\"  width=\"400\">\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9LdZoW-abr-N",
        "outputId": "cbd3f16c-31cd-4ced-f4c4-1772b491c74f"
      },
      "outputs": [],
      "source": [
        "# 目標函數\n",
        "\n",
        "# 創造均方誤差計算工具\n",
        "criterion = nn.MSELoss()                \n",
        "# 透過 my_data_loader 對資料集 my_dataset 進行抽樣\n",
        "for batch_x, batch_y in my_data_loader: \n",
        "    \n",
        "    # 自動呼叫 forward 計算 batch_x 得到 pred_y\n",
        "    pred_y = my_model(batch_x)          \n",
        "    \n",
        "    # 計算 pred_y 與 batch_y 的均方誤差\n",
        "    loss = criterion(pred_y, batch_y)   \n",
        "    print(loss)\n",
        "    \n",
        "    # 使用向後傳播計算梯度\n",
        "    loss.backward()                     "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "igh-yFNr1K3o"
      },
      "source": [
        "### 最佳化（Optimization）\n",
        "\n",
        "![Gradient Descent](https://img-blog.csdnimg.cn/20181110102438617.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpX2tfeQ==,size_16,color_FFFFFF,t_70)\n",
        "\n",
        "使用 `torch.optim` 中的不同的最佳化策略進行梯度下降（Gradient Descent）演算法：\n",
        "\n",
        "$$\n",
        "\\theta_t = \\theta_{t - 1} - \\text{lr} \\cdot \\nabla \\mathcal{L}(x)\n",
        "$$\n",
        "\n",
        "進行最佳化時需要注意以下事項：\n",
        "\n",
        "- 創造最佳化工具時必須指定哪些參數被更新\n",
        "    - 使用 `model.parameters()` 取得模型中所有可以被更新的參數\n",
        "    - 學習率（Learning Rate）負責決定模型參數更新的幅度，可以透過 `lr` 參數設定\n",
        "- 必須先計算誤差並且透過誤差向後傳播（`loss.backward()`），才能執行梯度下降更新參數（`optimizer.step()`）\n",
        "\n",
        "#### Stochastic Gradient Descent (SGD)\n",
        "<img src=\"https://www.samvitjain.com/blog/assets/gradient-descent/comparison.png\"  width=\"350\">\n",
        "\n",
        "- SGD 的核心概念就是用 小batch 訓練， GD 可以說就是 batch_size = dataset_size 的狀態。\n",
        "- SGD 比 GD 更適合用於深度學習模型的訓練，因為它可以更快地收斂、更有效地處理大資料集，並且因為具有更大的隨機性更容易跳出 local minima。不過，SGD 也有一些缺點，例如可能會出現收斂不穩定、震盪等問題。為了解決這些問題，有許多基於SGD的改進算法被提出，例如 Momentum、Adagrad、RMSprop、Adam 等。\n",
        "- [(PyTorch Documentation) Stochastic Gradient Descent](https://pytorch.org/docs/stable/optim.html#torch.optim.SGD)\n",
        "- [NTU EE ML Lecture 3-1: Gradient Descent](https://www.youtube.com/watch?v=yKKNr-QKz2Q&list=PLJV_el3uVTsPy9oCRY30oBPNLCo89yu49&index=6)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FqC3vN9gbr-N"
      },
      "outputs": [],
      "source": [
        "# 最佳化\n",
        "\n",
        "# 匯入計算梯度下降演算法的工具\n",
        "from torch.optim import SGD             \n",
        "\n",
        "# 創造計算隨機梯度下降的工具\n",
        "optimizer = SGD(                        \n",
        "    # 設定計算梯度下降的目標\n",
        "    my_model.parameters(),              \n",
        "    # 設定學習率\n",
        "    lr=0.0001                           \n",
        ")\n",
        "\n",
        "# 透過 my_data_loader 對資料集 my_dataset 進行抽樣\n",
        "for batch_x, batch_y in my_data_loader: \n",
        "    \n",
        "    # 自動呼叫 forward 計算 batch_x 得到 pred_y\n",
        "    pred_y = my_model(batch_x)          \n",
        "    \n",
        "    # 計算 pred_y 與 batch_y 的均方誤差\n",
        "    loss = criterion(pred_y, batch_y)   \n",
        "    # 使用向後傳播計算梯度\n",
        "    loss.backward()\n",
        "    \n",
        "    # 使用梯度下降更新模型參數\n",
        "    optimizer.step()\n",
        "    \n",
        "    # 清空計算過後的梯度值\n",
        "    optimizer.zero_grad()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TB1tspVHbr-N",
        "outputId": "51d33a4c-331e-4c32-82bc-471739c4065b"
      },
      "outputs": [],
      "source": [
        "# 驗證\n",
        "\n",
        "# 如果有可用 GPU 時採用 GPU cuda:0\n",
        "if torch.cuda.is_available():                   \n",
        "    device = torch.device('cuda:0')\n",
        "# 若無 GPU 可用則使用 CPU\n",
        "else:                                           \n",
        "    device = torch.device('cpu')\n",
        "\n",
        "# 創造訓練資料集\n",
        "train_dataset = MyDataset(1000)         \n",
        "# 創造測試資料集\n",
        "test_dataset = MyDataset(500)           \n",
        "\n",
        "# 設定超參數\n",
        "\n",
        "# 設定每次抽樣的數量\n",
        "batch_size = 50                         \n",
        "# 設定資料集總訓練次數\n",
        "n_epoch = 5                          \n",
        "# 設定隱藏層維度\n",
        "hid_dim = 100                            \n",
        "\n",
        "# 創造 DataLoader 實例\n",
        "train_data_loader = DataLoader(         \n",
        "    # 對資料集 train_dataset 進行抽樣\n",
        "    train_dataset,                      \n",
        "    # 設定每次抽樣的數量\n",
        "    batch_size=batch_size,              \n",
        "    # 設定隨機抽樣\n",
        "    shuffle=True,                       \n",
        "    # 指定格式化的方法\n",
        "    collate_fn=collate_fn               \n",
        ")\n",
        "# 創造 DataLoader 實例\n",
        "test_data_loader = DataLoader(          \n",
        "    test_dataset,                       \n",
        "    batch_size=batch_size,              \n",
        "    shuffle=True,                       \n",
        "    collate_fn=collate_fn               \n",
        ")\n",
        "\n",
        "# 創造 MyModel 模型實例\n",
        "model = MyModel(                        \n",
        "    # 設定輸入層維度\n",
        "    in_dim=1,                           \n",
        "    # 設定隱藏層維度\n",
        "    hid_dim=hid_dim,                    \n",
        "    # 設定輸出層維度\n",
        "    out_dim=1                           \n",
        ")\n",
        "# 將模型搬移至 GPU\n",
        "model = model.to(device)                \n",
        "\n",
        "# 創造均方誤差計算工具\n",
        "criterion = nn.MSELoss()                \n",
        "\n",
        "# 創造計算隨機梯度下降的工具\n",
        "optimizer = SGD(                        \n",
        "    model.parameters(),                 \n",
        "    lr=0.0001                           \n",
        ")\n",
        "\n",
        "# 總共訓練 n_epoch 次\n",
        "for epoch in range(n_epoch):            \n",
        "    for batch_x, batch_y in train_data_loader:\n",
        "        # 將訓練資料搬移至 GPU\n",
        "        batch_x = batch_x.to(device)    \n",
        "        # 將訓練資料標記搬移至 GPU\n",
        "        batch_y = batch_y.to(device)    \n",
        "        \n",
        "        # 自動呼叫 forward 計算 batch_x 得到 pred_y\n",
        "        pred_y = model(batch_x)  # shape: (batch_size, 1)      \n",
        "        # 計算 pred_y (預測標記）與 batch_y （真實標記）的均方誤差\n",
        "        loss = criterion(pred_y, batch_y) \n",
        "        \n",
        "        # 使用向後傳播計算梯度\n",
        "        loss.backward()                 \n",
        "        # 使用梯度下降更新模型參數\n",
        "        optimizer.step()\n",
        "        # 清空計算過後的梯度值\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "    # 此區塊不會計算梯度\n",
        "    with torch.no_grad():               \n",
        "        # 統計訓練資料誤差\n",
        "        total_loss = 0                  \n",
        "        for batch_x, batch_y in train_data_loader:\n",
        "            batch_x = batch_x.to(device)\n",
        "            batch_y = batch_y.to(device)\n",
        "            \n",
        "            pred_y = model(batch_x)\n",
        "            loss = criterion(pred_y, batch_y)\n",
        "            \n",
        "            total_loss += float(loss) / len(train_data_loader)\n",
        "        \n",
        "        print('Epoch {}, training loss: {}'.format(epoch, total_loss))\n",
        "        \n",
        "        # 統計測試資料誤差\n",
        "        total_loss = 0                  \n",
        "        for batch_x, batch_y in test_data_loader:\n",
        "            batch_x = batch_x.to(device)\n",
        "            batch_y = batch_y.to(device)\n",
        "            \n",
        "            pred_y = model(batch_x)\n",
        "            loss = criterion(pred_y, batch_y)\n",
        "            \n",
        "            total_loss += float(loss) / len(test_data_loader)\n",
        "            \n",
        "        print('Epoch {}, testing loss: {}'.format(epoch, total_loss))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 887
        },
        "id": "JkmZGlVEbr-O",
        "outputId": "cdebfe36-079d-4e4f-a458-0b29bd72d253"
      },
      "outputs": [],
      "source": [
        "with torch.no_grad():\n",
        "    for batch_x, batch_y in train_data_loader:\n",
        "        batch_x = batch_x.to(device)\n",
        "        batch_y = batch_y.to(device)\n",
        "\n",
        "        pred_y = model(batch_x)\n",
        "        \n",
        "        batch_x = batch_x.to('cpu')\n",
        "        batch_y = batch_y.to('cpu')\n",
        "        pred_y = pred_y.to('cpu')\n",
        "        # 畫出訓練資料答案分佈\n",
        "        plt.scatter(batch_x, batch_y, color='red') \n",
        "        # 畫出訓練資料預測分佈\n",
        "        plt.scatter(batch_x, pred_y, color='blue') \n",
        "        \n",
        "    plt.title('Training data performance')\n",
        "    plt.show()\n",
        "    \n",
        "    for batch_x, batch_y in test_data_loader:\n",
        "        batch_x = batch_x.to(device)\n",
        "        batch_y = batch_y.to(device)\n",
        "\n",
        "        pred_y = model(batch_x)\n",
        "        \n",
        "        batch_x = batch_x.to('cpu')\n",
        "        batch_y = batch_y.to('cpu')\n",
        "        pred_y = pred_y.to('cpu')\n",
        "        # 畫出測試資料答案分佈\n",
        "        plt.scatter(batch_x, batch_y, color='red') \n",
        "        # 畫出測試資料預測分佈\n",
        "        plt.scatter(batch_x, pred_y, color='blue') \n",
        "    \n",
        "    plt.title('Testing data performance')\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lMbkzFUU1S2K"
      },
      "source": [
        "### 測試\n",
        "\n",
        "深度學習模型在訓練時會自動計算梯度，若於分析模型在目標函數的表現時不想花多餘資源計算梯度可以使用 `with torch.no_grad():`：\n",
        "\n",
        "### 儲存 & 載入模型\n",
        "\n",
        "使用 `torch.save()` 配合 `model.state_dict()` 儲存訓練後的模型參數；\n",
        "使用 `model.load_state_dict()` 配合 `torch.load()` 載入儲存的訓練過的模型參數。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Pnq25QSFbr-O"
      },
      "outputs": [],
      "source": [
        "# 儲存 & 載入模型\n",
        "\n",
        "# 儲存模型參數\n",
        "# torch.save(model.state_dict(), './data/model.ckpt')    \n",
        "# 載入模型參數\n",
        "# model.load_state_dict(torch.load('./data/model.ckpt')) "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8VHNeeFnbr-O"
      },
      "source": [
        "## 練習\n",
        "\n",
        "### 練習 1：調整超參數\n",
        "\n",
        "請試著更改前述範例中的超參數讓模型表現變好：\n",
        "\n",
        "- 增加訓練次數 `n_epoch`\n",
        "- 增大單一訓練資料次數 `batch_size`\n",
        "- 增大隱藏層的維度 `hid_dim`\n",
        "- 更改啟動函數 `F.relu`\n",
        "\n",
        "### 練習 2：加深模型\n",
        "\n",
        "請試著更改前述範例中的模型深度讓模型表現變好：\n",
        "\n",
        "- 增加 1 個或多個 `nn.Linear`"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 簡單的機器學習任務練習\n",
        "- MNIST 資料集：手寫數字圖像分類任務 (Image Classification)\n",
        "  - ![](https://thumbs.gfycat.com/AdorableJoyfulLemming-max-1mb.gif)\n",
        "  - MNIST 一筆 data $\\in \\mathbb{R}^{784}$ （784 維的 feature vector）\n",
        "  - [datahacker/cnn/#005 PyTorch - Convolutional Neural Network on MNIST Handwritten Digit Recognition in PyTorch 1.3.ipynb](https://github.com/maticvl/dataHacker/blob/master/CNN/%23005%20PyTorch%20-%20Convolutional%20Neural%20Network%20on%20MNIST%20Handwritten%20Digit%20Recognition%20in%20PyTorch%201.3.ipynb)\n",
        "- Kaggle Titanic Survival Prediction (Feature Classification)\n",
        "    - [Example Notebook: Introduction to Pytorch (a very gentle start)](https://www.kaggle.com/code/frtgnn/introduction-to-pytorch-a-very-gentle-start)\n",
        "- Real-and-Fake Text Classification (Text Classifcation)\n",
        "    - [A step-by-step guide: LSTM Text Classification Using Pytorch](https://towardsdatascience.com/lstm-text-classification-using-pytorch-2c6c657f8fc0)\n",
        "- 分類問題是最簡單也是最核心的，所有任務（文字生成 text-generation，文字翻譯 translation，問答 QA，序列標注 sequence labeling 等等）全都是分類問題的變形而已，只是因為資料形式不同，這些任務的資料前處理與模型預測後處理會比分類問題更加複雜和不直覺。建議從最直觀的分類問題下手練習。\n",
        "- 例子都是提供 PyTorch 架構的範例。\n",
        "\n",
        "### 推薦深度學習或自然語言處理學習資源\n",
        "  - 比較輕鬆入門深度學習的方式(老師很好笑，數學很硬）：\n",
        "    - [Hung-Yi Lee 's NTU ML Course Website](https://speech.ee.ntu.edu.tw/~hylee/ml/2022-spring.php)\n",
        "      - 如果想要更加熟悉，可以寫裡面的作業（大概寫 Regression, Classifcation, CNN, Self-attention, Transformer, BERT 就能夠 cover 這堂課需要的所有技能，比較有趣的作業可以寫看看 Explainable AI）。\n",
        "    - [Hung Yi Lee's Youtube Channel](https://www.youtube.com/@HungyiLeeNTU)\n",
        "  - 史丹佛的教科書/NTU 資訊所 NLP 課程用書：[Stanford Textbook: Speech & language Processing](https://web.stanford.edu/~jurafsky/slp3/) \n",
        "這本書寫很仔細，從 sequence models 和 word embeddings 開始講，後面的章節是分成多個不同 NLP 任務講解。想要知道一些 NLP 的基礎，詳讀以下章節應該可以收穫良多：\n",
        "    - 6:\tVector Semantics and Embeddings\t6: Vector Semantics\t[new in this edition]\n",
        "    - 7:\tNeural Networks and Neural Language Models\t7: Neural Networks [new in this edition]\n",
        "    - 9:\tRNNs and LSTMs\t\t[new in this edition]\n",
        "    - 10:\tTransformers and Pretrained Language Models\n",
        "  - Andrew Ng's Machine Learning Resources\n",
        "    - [Sequence Models](https://www.coursera.org/learn/nlp-sequence-models)\n",
        "  - [PyTorch Introduction by NTU EE](https://colab.research.google.com/drive/1Xed5YSpLsLfkn66OhhyNzr05VE89enng#scrollTo=jifMOIcNMTh5)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.12"
    },
    "vscode": {
      "interpreter": {
        "hash": "5d79344b50f250bbd6ad2de2adfaeedd1b2740d625477f4cf63f23f68cf7a998"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}