re_ranking_topic_diversification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Re-Ranking for Topic Diversification\n",
    "\n",
    "Recommender systems (RecSys) are a class of machine learning models built to recommend the most relevant things to the user - an ad that a customer is most likely to click on, a product that a customer is most likely to buy, and so on. In doing so, RecSys often exploit known preferences of the users to ensure that suggestions from it are relevant. This leads to an of echo chamber of sorts, for example, if you were to purchase a summer dress from a retail website, you might be shown other summer dresses everytime you visit the website because the RecSys knows that you were interested in summer dresses in the past.\n",
    "\n",
    "Understandably, this leads to some obvious issues like:\n",
    "- bad user experience from constantly seeing a product that might not be relevant to them anymore\n",
    "- not exposing other products from the catalog that user might have bought leading to lost revenues for the company\n",
    "aling with a number of nuanced issues.\n",
    "\n",
    "One can think of a few ways to resolve these issues like:\n",
    "- Ensure that the output of the model recommends a diverse list of topics by changes to features or type of model\n",
    "- Re-ranking the items recommended by the RecSys to ensure it is diverse\n",
    "\n",
    "For the purpose of this exercise, we will look at the latter, the problem of re-ranking. We have a dataset of movie ratings ([MovieLens Dataset](https://grouplens.org/datasets/movielens/1m/ \"MovieLens 1M dataset\")), a stable benchmark dataset for recommender systems. It has 1 million ratings from 6000 users on 4000 movies (or 4.16% user-movie interactions covered).\n",
    "\n",
    "\n",
    "## Sections\n",
    "1. [Methodology](#Methodology) \n",
    "2. [Building a baseline RecSys](#Building-a-baseline-RecSys)\n",
    "3. [Diversification as a Set Cover Problem // A Greedy Solution](#Diversification-as-a-Set-Cover-Problem-//-A-Greed-Solution)\n",
    "4. [Diversification as a Set Cover Problem // A Mixed Integer Programming Solution](#Diversification-as-a-Set-Cover-Problem-//-A-Mixed-Integer-Programming-Solution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Methodology\n",
    "We will build two recommender systems:  \n",
    "1. **Baseline** - Recommend 10 top-rated movies from this dataset.  \n",
    "2. **Version 1** - Diversified recommender system using a greedy algorithm\n",
    "3. **Version 2** - Diversified recommender system using a mixed linear programming approach\n",
    "\n",
    "Reference:\n",
    "- The problem is inspired by the blog here: https://nbviewer.jupyter.org/github/david-cortes/datascienceprojects/blob/master/machine_learning/topic_diversification.ipynb\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a baseline RecSys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing data\n",
    "import pandas as pd\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "movies = pd.read_csv('data/ml-1m/movies.dat', sep='::', engine='python')\n",
    "ratings = pd.read_csv('data/ml-1m/ratings.dat', sep='::', engine='python')\n",
    "users = pd.read_csv('data/ml-1m/users.dat', sep='::', engine='python')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data in movies = ['MovieID', 'Title', 'Genres']\n",
      "Data in ratings = ['UserID', 'MovieID', 'Rating', 'Timestamp']\n",
      "Data in users = ['UserID', 'Gender', 'Age', 'Occupation', 'Zip-code']\n"
     ]
    }
   ],
   "source": [
    "print(f\"Data in movies = {list(movies.columns)}\")\n",
    "print(f\"Data in ratings = {list(ratings.columns)}\")\n",
    "print(f\"Data in users = {list(users.columns)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AvgRating</th>\n",
       "      <th>UserCount</th>\n",
       "      <th>score</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MovieID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2858</th>\n",
       "      <td>4.317386</td>\n",
       "      <td>3428</td>\n",
       "      <td>14800.0</td>\n",
       "      <td>American Beauty (1999)</td>\n",
       "      <td>Comedy|Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>260</th>\n",
       "      <td>4.453694</td>\n",
       "      <td>2991</td>\n",
       "      <td>13321.0</td>\n",
       "      <td>Star Wars: Episode IV - A New Hope (1977)</td>\n",
       "      <td>Action|Adventure|Fantasy|Sci-Fi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1196</th>\n",
       "      <td>4.292977</td>\n",
       "      <td>2990</td>\n",
       "      <td>12836.0</td>\n",
       "      <td>Star Wars: Episode V - The Empire Strikes Back...</td>\n",
       "      <td>Action|Adventure|Drama|Sci-Fi|War</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1210</th>\n",
       "      <td>4.022893</td>\n",
       "      <td>2883</td>\n",
       "      <td>11598.0</td>\n",
       "      <td>Star Wars: Episode VI - Return of the Jedi (1983)</td>\n",
       "      <td>Action|Adventure|Romance|Sci-Fi|War</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2028</th>\n",
       "      <td>4.337354</td>\n",
       "      <td>2653</td>\n",
       "      <td>11507.0</td>\n",
       "      <td>Saving Private Ryan (1998)</td>\n",
       "      <td>Action|Drama|War</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1198</th>\n",
       "      <td>4.477725</td>\n",
       "      <td>2514</td>\n",
       "      <td>11257.0</td>\n",
       "      <td>Raiders of the Lost Ark (1981)</td>\n",
       "      <td>Action|Adventure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>593</th>\n",
       "      <td>4.351823</td>\n",
       "      <td>2578</td>\n",
       "      <td>11219.0</td>\n",
       "      <td>Silence of the Lambs, The (1991)</td>\n",
       "      <td>Drama|Thriller</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2571</th>\n",
       "      <td>4.315830</td>\n",
       "      <td>2590</td>\n",
       "      <td>11178.0</td>\n",
       "      <td>Matrix, The (1999)</td>\n",
       "      <td>Action|Sci-Fi|Thriller</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2762</th>\n",
       "      <td>4.406263</td>\n",
       "      <td>2459</td>\n",
       "      <td>10835.0</td>\n",
       "      <td>Sixth Sense, The (1999)</td>\n",
       "      <td>Thriller</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>589</th>\n",
       "      <td>4.058513</td>\n",
       "      <td>2649</td>\n",
       "      <td>10751.0</td>\n",
       "      <td>Terminator 2: Judgment Day (1991)</td>\n",
       "      <td>Action|Sci-Fi|Thriller</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         AvgRating  UserCount    score  \\\n",
       "MovieID                                  \n",
       "2858      4.317386       3428  14800.0   \n",
       "260       4.453694       2991  13321.0   \n",
       "1196      4.292977       2990  12836.0   \n",
       "1210      4.022893       2883  11598.0   \n",
       "2028      4.337354       2653  11507.0   \n",
       "1198      4.477725       2514  11257.0   \n",
       "593       4.351823       2578  11219.0   \n",
       "2571      4.315830       2590  11178.0   \n",
       "2762      4.406263       2459  10835.0   \n",
       "589       4.058513       2649  10751.0   \n",
       "\n",
       "                                                     Title  \\\n",
       "MovieID                                                      \n",
       "2858                                American Beauty (1999)   \n",
       "260              Star Wars: Episode IV - A New Hope (1977)   \n",
       "1196     Star Wars: Episode V - The Empire Strikes Back...   \n",
       "1210     Star Wars: Episode VI - Return of the Jedi (1983)   \n",
       "2028                            Saving Private Ryan (1998)   \n",
       "1198                        Raiders of the Lost Ark (1981)   \n",
       "593                       Silence of the Lambs, The (1991)   \n",
       "2571                                    Matrix, The (1999)   \n",
       "2762                               Sixth Sense, The (1999)   \n",
       "589                      Terminator 2: Judgment Day (1991)   \n",
       "\n",
       "                                      Genres  \n",
       "MovieID                                       \n",
       "2858                            Comedy|Drama  \n",
       "260          Action|Adventure|Fantasy|Sci-Fi  \n",
       "1196       Action|Adventure|Drama|Sci-Fi|War  \n",
       "1210     Action|Adventure|Romance|Sci-Fi|War  \n",
       "2028                        Action|Drama|War  \n",
       "1198                        Action|Adventure  \n",
       "593                           Drama|Thriller  \n",
       "2571                  Action|Sci-Fi|Thriller  \n",
       "2762                                Thriller  \n",
       "589                   Action|Sci-Fi|Thriller  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Getting the top-rated movies\n",
    "import numpy as np\n",
    "\n",
    "topRated = ratings.assign(raters=ratings.Rating>=0)\\\n",
    "    .groupby('MovieID')\\\n",
    "    .agg({'Rating': 'mean','UserID': 'count'})\\\n",
    "    .rename(columns={'UserID': 'UserCount', 'Rating': 'AvgRating'})\\\n",
    "    .assign(score=lambda x: x.AvgRating*x.UserCount)\\\n",
    "    .sort_values('score',ascending=False)\\\n",
    "    .head(50)\n",
    "\n",
    "topRated.join(movies.set_index('MovieID')).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observation** - Only 8 of 18 genres are present in the top 10 movies that have been recommended by the RecSys. Likely that users have a preference for many types of movies, and using this recommendation might not satisfy every user."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assessing the diversity of the solution\n",
    "We want to understand what the universe looks like, and how do we measure the diversity in the context of genres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of genres in the dataset = 18\n"
     ]
    }
   ],
   "source": [
    "def get_unique_items_in_column(dataset, colname):\n",
    "    \"\"\"Finds unique items in a column\n",
    "    For a given Pandas DataFrame, the function finds all the unique\n",
    "    items in the column. The function expects the column to contain\n",
    "    either one value or a list of values.\n",
    "\n",
    "    Args:\n",
    "        dataset (obj): Pandas DataFrame with data\n",
    "        colname (str): Column that needs to be scanned\n",
    "\n",
    "    Returns:\n",
    "        obj : A set object with unique items\n",
    "    \"\"\"\n",
    "    uniqueItems = set()\n",
    "\n",
    "    for index, data in dataset.iterrows():\n",
    "        if isinstance(data[colname], list):\n",
    "            uniqueItems.update(data[colname])\n",
    "        else:\n",
    "            uniqueItems.add(data[colname])\n",
    "    return uniqueItems\n",
    "\n",
    "movies['Genres'] = movies['Genres'].apply(lambda x : x.strip().split('|'))\n",
    "genresUniverse = get_unique_items_in_column(movies, 'Genres')\n",
    "print(f\"Number of genres in the dataset = {len(genresUniverse)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Diversification as a Set Cover Problem // A Greedy Solution\n",
    "Now that we have a list of top movies, one way to introduce more genres, can be to reshuffle the movies in the top 50 so that all genres are represented in the top 10 recommendations. In doing so, we can use the maximum coverage formulation, where movies that have more coverage across genres, are represented and give us a better chance of getting more genres in.\n",
    "\n",
    "For this exercise, we are going to use the greedy algorithm for set cover problems. This is a common algorithm is computer science, and it's a polynomial time approximation of the NP-hard problem of finding a subset of items that covers all elements in the universe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Finding the movies to rank\n",
    "moviesToRerank = topRated.join(movies.set_index('MovieID'))\n",
    "# Finding the genres that are covered in the top N movies, so that we can inlude all genres\n",
    "genresInList = get_unique_items_in_column(moviesToRerank, 'Genres')\n",
    "\n",
    "# Setting constants, and initializing variables\n",
    "uncoveredGenres = set()\n",
    "movieIndex = 0\n",
    "recListSize = 10\n",
    "reRankedMovies = pd.DataFrame()\n",
    "\n",
    "while genresInList.difference(uncoveredGenres) and movieIndex < recListSize: \n",
    "\n",
    "    # Get the number of genres covered by the movies\n",
    "    moviesToRerank['nGenres'] = moviesToRerank['Genres'].apply(lambda x: len(set(x).difference(uncoveredGenres)) )\n",
    "    # Order movies by descending order or Genres covered in dataset\n",
    "    moviesToRerank = moviesToRerank.sort_values('nGenres', ascending=False)\n",
    "    moviesToRerank = moviesToRerank.reset_index(drop=True)\n",
    "\n",
    "    # Adding the top movie to the list\n",
    "    topMovie = moviesToRerank.iloc[0]\n",
    "    reRankedMovies = reRankedMovies.append(topMovie)\n",
    "    uncoveredGenres.update(topMovie['Genres'])\n",
    "    movieIndex += 1\n",
    "\n",
    "    # Dropping the row\n",
    "    moviesToRerank = moviesToRerank.drop([0])\n",
    "    \n",
    "# Add movies to the list, if we don't have a full list yet\n",
    "moviesToRerank = moviesToRerank.sort_values('score', ascending=False)\n",
    "moviesToRerank = moviesToRerank.reset_index(drop=True)\n",
    "\n",
    "for index, movie in moviesToRerank.iterrows():\n",
    "    if movieIndex >= recListSize:\n",
    "        break\n",
    "    elif movieIndex < recListSize and movie.Title not in list(reRankedMovies.Title):\n",
    "        reRankedMovies = reRankedMovies.append(movie)\n",
    "        movieIndex += 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AvgRating</th>\n",
       "      <th>Genres</th>\n",
       "      <th>Title</th>\n",
       "      <th>UserCount</th>\n",
       "      <th>nGenres</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.292977</td>\n",
       "      <td>[Action, Adventure, Drama, Sci-Fi, War]</td>\n",
       "      <td>Star Wars: Episode V - The Empire Strikes Back...</td>\n",
       "      <td>2990.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>12836.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.219406</td>\n",
       "      <td>[Crime, Film-Noir, Mystery, Thriller]</td>\n",
       "      <td>L.A. Confidential (1997)</td>\n",
       "      <td>2288.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>9654.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.146846</td>\n",
       "      <td>[Animation, Children's, Comedy]</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>2077.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>8613.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.953029</td>\n",
       "      <td>[Comedy, Romance]</td>\n",
       "      <td>Groundhog Day (1993)</td>\n",
       "      <td>2278.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9005.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.409778</td>\n",
       "      <td>[Action, Adventure, Fantasy, Sci-Fi]</td>\n",
       "      <td>Star Wars: Episode I - The Phantom Menace (1999)</td>\n",
       "      <td>2250.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>7672.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.159585</td>\n",
       "      <td>[Action, Horror, Sci-Fi, Thriller]</td>\n",
       "      <td>Alien (1979)</td>\n",
       "      <td>2024.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>8419.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.247963</td>\n",
       "      <td>[Adventure, Children's, Drama, Musical]</td>\n",
       "      <td>Wizard of Oz, The (1939)</td>\n",
       "      <td>1718.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>7298.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.317386</td>\n",
       "      <td>[Comedy, Drama]</td>\n",
       "      <td>American Beauty (1999)</td>\n",
       "      <td>3428.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14800.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.453694</td>\n",
       "      <td>[Action, Adventure, Fantasy, Sci-Fi]</td>\n",
       "      <td>Star Wars: Episode IV - A New Hope (1977)</td>\n",
       "      <td>2991.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13321.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.022893</td>\n",
       "      <td>[Action, Adventure, Romance, Sci-Fi, War]</td>\n",
       "      <td>Star Wars: Episode VI - Return of the Jedi (1983)</td>\n",
       "      <td>2883.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11598.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   AvgRating                                     Genres  \\\n",
       "0   4.292977    [Action, Adventure, Drama, Sci-Fi, War]   \n",
       "0   4.219406      [Crime, Film-Noir, Mystery, Thriller]   \n",
       "0   4.146846            [Animation, Children's, Comedy]   \n",
       "0   3.953029                          [Comedy, Romance]   \n",
       "0   3.409778       [Action, Adventure, Fantasy, Sci-Fi]   \n",
       "0   4.159585         [Action, Horror, Sci-Fi, Thriller]   \n",
       "0   4.247963    [Adventure, Children's, Drama, Musical]   \n",
       "0   4.317386                            [Comedy, Drama]   \n",
       "1   4.453694       [Action, Adventure, Fantasy, Sci-Fi]   \n",
       "2   4.022893  [Action, Adventure, Romance, Sci-Fi, War]   \n",
       "\n",
       "                                               Title  UserCount  nGenres  \\\n",
       "0  Star Wars: Episode V - The Empire Strikes Back...     2990.0      5.0   \n",
       "0                           L.A. Confidential (1997)     2288.0      4.0   \n",
       "0                                   Toy Story (1995)     2077.0      3.0   \n",
       "0                               Groundhog Day (1993)     2278.0      1.0   \n",
       "0   Star Wars: Episode I - The Phantom Menace (1999)     2250.0      1.0   \n",
       "0                                       Alien (1979)     2024.0      1.0   \n",
       "0                           Wizard of Oz, The (1939)     1718.0      1.0   \n",
       "0                             American Beauty (1999)     3428.0      0.0   \n",
       "1          Star Wars: Episode IV - A New Hope (1977)     2991.0      0.0   \n",
       "2  Star Wars: Episode VI - Return of the Jedi (1983)     2883.0      0.0   \n",
       "\n",
       "     score  \n",
       "0  12836.0  \n",
       "0   9654.0  \n",
       "0   8613.0  \n",
       "0   9005.0  \n",
       "0   7672.0  \n",
       "0   8419.0  \n",
       "0   7298.0  \n",
       "0  14800.0  \n",
       "1  13321.0  \n",
       "2  11598.0  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reRankedMovies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Diversification as a Set Cover Problem // A Mixed Integer Programming Solution\n",
    "Another way of solving the problem above comes from the field of operations research. We can used mixed integer programming to "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Edit Metadata",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}