准确率、召回率、f1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准确率、召回率、`f1`\n",
    "***\n",
    "***\n",
    "Time: 2020-09-20\n",
    "Author: dsy\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 真正例(True Positive,TP):一个样本的真实类别为c并且模型正确地预测为类别c。这类样本数量记为：\n",
    "$$\n",
    "TP_c = \\sum_{n=1}^{N} I(y^{(n)} = \\hat{y}^{(n)} =c)\n",
    "$$\n",
    "2. 假负例（False Negative，FN）：一个样本的真实类别为c，模型错误地预测为其它类。这类样本数量记为:\n",
    "$$\n",
    "FN_c = \\sum_{n=1}^N I( y^{(n)} =c \\land \\hat{y}^{(n)}  \\ne c   )\n",
    "$$\n",
    "3. 假正例（False Positive，FP）一个样本的真实类别为其它类，模型错误地预测为类别c。这类样本数量记为:\n",
    "$$\n",
    "FP_c = \\sum_{n=1}^N I( y^{(n)} \\ne c \\land \\hat{y}^{(n)}  = c   )\n",
    "$$\n",
    "4. 真负例（True Negative，TN）：一个样本的真实类别为其它类，模型也预测为其它类。这类样本数量记为T Nc。对于类别c来说，这种情况一般不需要关注。\n",
    "***\n",
    "5. 查准率（Precision），也叫精确率或精度，类别c的查准率是所有预测为类别c的样本中，预测正确的比例。\n",
    "$$\n",
    "P_c = \\frac{TP_c}{TP_c+FP_c}\n",
    "$$\n",
    "6. 查全率（Recall），也叫召回率，类别c的查全率是所有真实标签为类别c的样本中，预测正确的比例。\n",
    "$$\n",
    "R_c = \\frac{TP_c}{TP_c+FN_c}\n",
    "$$\n",
    "7. F值（F Measure）是一个综合指标，为查准率和查全率的调和平均。\n",
    "$$\n",
    "F_c = \\frac{(1+\\beta^2) \\times P_c \\times R_c}{\\beta^2 \\times P_c + R_c}\n",
    "$$\n",
    "其中 β 用于平衡查全率和查准率的重要性，一般取值为 1。β = 1 时的 F 值称为F1值，是查准率和查全率的调和平均。\n",
    "8. 宏平均和微平均 为了计算分类算法在所有类别上的总体查准率、查全率和 F1值，经常使用两种平均方法，分别称为宏平均（Macro Average）和微平均（Micro Average）,宏平均是每一类的性能指标的算术平均值:\n",
    "$$\n",
    "\\begin{array}{cl}\n",
    "P_{macro} & = \\frac{1}{C} \\sum_{c=1}^C P_c \\\\\n",
    "R_{macro} & = \\frac{1}{C} \\sum_{c=1}^C R_c \\\\\n",
    "F1_{macro} & = \\frac{2 \\times P_{macro} \\times R_{macro}}{P_{macro} + R_{macro}}\n",
    "\\end{array}\n",
    "$$\n",
    "微平均是每一个样本的性能指标的算术平均值。对于单个样本而言，它的查\n",
    "准率和查全率是相同的（要么都是1，要么都是0）。因此查准率的微平均和查全\n",
    "率的微平均是相同的。同理，F1值的微平均指标是相同的。当不同类别的样本数\n",
    "量不均衡时，使用宏平均会比微平均更合理些。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import recall_score,precision_score,f1_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_true = np.array([4, 5, 6, 4, 5, 6])\n",
    "y_pred = np.array([4, 6, 5, 4, 4, 5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 根据公式自定义计算"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels = np.union1d(y_pred,y_true)\n",
    "c = labels[0]\n",
    "c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TP = (y_pred[y_true == c] == c).sum()\n",
    "TP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FN = (y_pred[ y_true == c] != c).sum()\n",
    "FN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FP = (y_pred[y_true != c] == c).sum()\n",
    "FP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2222222222222222"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pmacro = 1/len(labels) * TP / (TP+FP)\n",
    "pmacro"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3333333333333333"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rmacro = 1/len(labels) * TP / (TP+FN)\n",
    "rmacro"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.26666666666666666"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1macro = 2 * pmacro * rmacro / (pmacro + rmacro)\n",
    "f1macro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用`sklearn`进行计算`precision`,`recall`,`f1`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2222222222222222"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "P_macro = precision_score(y_true,y_pred,average=\"macro\")\n",
    "P_macro"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3333333333333333"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "R_macro = recall_score(y_true,y_pred,average=\"macro\")\n",
    "R_macro"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.26666666666666666"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1_marco = f1_score(y_true,y_pred,average=\"macro\")\n",
    "f1_marco"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.26666666666666666"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "2 * P_macro * R_macro / ( P_macro + R_macro)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 最后附上二维数组的`precision`、`recall`、`f1`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def precision_recall_f1(true,pred):\n",
    "    m,n = true.shape\n",
    "    precision = np.zeros((m,))\n",
    "    recall = np.zeros((m,))\n",
    "    f1 = np.zeros((m,))\n",
    "    \n",
    "    for i in range(m):\n",
    "        precision[i] = precision_score(true[i],pred[i],average=\"macro\")\n",
    "        recall[i] = recall_score(true[i],pred[i],average=\"macro\")\n",
    "        f1[i] = f1_score(true[i],pred[i],average=\"macro\")\n",
    "        \n",
    "    return (precision.mean(),recall.mean(),f1.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "true = np.array([np.array([4, 5, 6, 4, 5, 6]) for _ in range(10)])\n",
    "pred = np.array([np.array([4, 6, 5, 4, 4, 5]) for _ in range(10)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.22222222222222224, 0.33333333333333337, 0.26666666666666666)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision_recall_f1(true,pred)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}