movie-recommendation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3cb6aca9",
   "metadata": {},
   "source": [
    "# Movie Recommendation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cbb6126",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/myscale/examples/blob/main/movie-recommendation.ipynb\" style=\"padding-left: 0.5rem;\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg?style=plastic)](https://colab.research.google.com/github/myscale/examples/blob/main/movie-recommendation.ipynb)\"></a>\n",
    "<a href=\"https://github.com/myscale/examples/blob/main/movie-recommendation.ipynb\" style=\"padding-left: 0.5rem;\"><img src=\"https://img.shields.io/badge/Open-Github-blue.svg?logo=github&style=plastic)](https://github.com/myscale/examples/blob/main/movie-recommendation.ipynb)\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f247abe2",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "A recommendation system is a subclass of information filtering system that provides suggestions for items that are most pertinent to a particular user. To suggest the most suitable option from a range of possibilities, these systems use algorithms. Various techniques and algorithms such as collaborative filtering, matrix factorization, and deep learning are utilized to implement a recommendation system.\n",
    "\n",
    "This guide will demonstrate how to construct a basic recommendation system using MyScale. The process comprises several stages, including the construction of user and item vectors based on the NMF model, the insertion of datasets into MyScale, the retrieval of top K recommended items for a user, and the use of an SVD model to predict user ratings for MyScale's suggested items.\n",
    "\n",
    "If you are more interested in exploring the capabilities of MyScale, you may skip the [Building Datasets](#building-datasets) section and dive right into the [Populating data to MyScale section](#populating-data-to-myscale)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bc4af86",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "To begin with, certain dependencies must be installed, including [clickhouse python client](https://clickhouse.com/docs/en/integrations/language-clients/python/intro/), scikit-learn, and other relevant tools."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7926cb5b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: clickhouse-connect in /opt/anaconda3/lib/python3.8/site-packages (0.5.17)\n",
      "Requirement already satisfied: scikit-learn in /opt/anaconda3/lib/python3.8/site-packages (1.2.2)\n",
      "Requirement already satisfied: urllib3>=1.26 in /opt/anaconda3/lib/python3.8/site-packages (from clickhouse-connect) (1.26.7)\n",
      "Requirement already satisfied: lz4 in /opt/anaconda3/lib/python3.8/site-packages (from clickhouse-connect) (4.3.2)\n",
      "Requirement already satisfied: zstandard in /opt/anaconda3/lib/python3.8/site-packages (from clickhouse-connect) (0.20.0)\n",
      "Requirement already satisfied: certifi in /opt/anaconda3/lib/python3.8/site-packages (from clickhouse-connect) (2021.10.8)\n",
      "Requirement already satisfied: pytz in /opt/anaconda3/lib/python3.8/site-packages (from clickhouse-connect) (2021.3)\n",
      "Requirement already satisfied: joblib>=1.1.1 in /opt/anaconda3/lib/python3.8/site-packages (from scikit-learn) (1.2.0)\n",
      "Requirement already satisfied: numpy>=1.17.3 in /opt/anaconda3/lib/python3.8/site-packages (from scikit-learn) (1.21.2)\n",
      "Requirement already satisfied: scipy>=1.3.2 in /opt/anaconda3/lib/python3.8/site-packages (from scikit-learn) (1.7.3)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/anaconda3/lib/python3.8/site-packages (from scikit-learn) (3.0.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install -U clickhouse-connect scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7baf365",
   "metadata": {},
   "source": [
    "## Building Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6859c9e",
   "metadata": {},
   "source": [
    "### Downloading and processing data\n",
    "\n",
    "For this example, we will employ the small datasets at [MovieLens Latest Datasets](https://grouplens.org/datasets/movielens/latest/) to provide movie recommendations. The dataset comprises 100,000 ratings applied to 9,000 movies by 600 users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "46721b9c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100  955k  100  955k    0     0   368k      0  0:00:02  0:00:02 --:--:--  368k\n"
     ]
    }
   ],
   "source": [
    "!wget https://files.grouplens.org/datasets/movielens/ml-latest-small.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "aa882036",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  ml-latest-small.zip\n",
      "   creating: ml-latest-small/\n",
      "  inflating: ml-latest-small/links.csv  \n",
      "  inflating: ml-latest-small/tags.csv  \n",
      "  inflating: ml-latest-small/ratings.csv  \n",
      "  inflating: ml-latest-small/README.txt  \n",
      "  inflating: ml-latest-small/movies.csv  \n"
     ]
    }
   ],
   "source": [
    "!unzip ml-latest-small.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13bd73b9",
   "metadata": {},
   "source": [
    "Let's read data into pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e619a300",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>tmdbId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>[Adventure, Animation, Children, Comedy, Fantasy]</td>\n",
       "      <td>862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji (1995)</td>\n",
       "      <td>[Adventure, Children, Fantasy]</td>\n",
       "      <td>8844</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men (1995)</td>\n",
       "      <td>[Comedy, Romance]</td>\n",
       "      <td>15602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale (1995)</td>\n",
       "      <td>[Comedy, Drama, Romance]</td>\n",
       "      <td>31357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II (1995)</td>\n",
       "      <td>[Comedy]</td>\n",
       "      <td>11862</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movieId                               title  \\\n",
       "0        1                    Toy Story (1995)   \n",
       "1        2                      Jumanji (1995)   \n",
       "2        3             Grumpier Old Men (1995)   \n",
       "3        4            Waiting to Exhale (1995)   \n",
       "4        5  Father of the Bride Part II (1995)   \n",
       "\n",
       "                                              genres tmdbId  \n",
       "0  [Adventure, Animation, Children, Comedy, Fantasy]    862  \n",
       "1                     [Adventure, Children, Fantasy]   8844  \n",
       "2                                  [Comedy, Romance]  15602  \n",
       "3                           [Comedy, Drama, Romance]  31357  \n",
       "4                                           [Comedy]  11862  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# obtain movie metadata\n",
    "original_movie_metadata = pd.read_csv('ml-latest-small/movies.csv')\n",
    "movie_metadata = original_movie_metadata[['movieId', 'title', 'genres']]\n",
    "movie_metadata['genres'] = movie_metadata['genres'].str.split('|', expand=False)\n",
    "\n",
    "# add tmdbId to movie metadata dataframe\n",
    "original_movie_links = pd.read_csv('ml-latest-small/links.csv')\n",
    "movie_info = pd.merge(movie_metadata, original_movie_links, on=[\"movieId\"])[['movieId', 'title', 'genres', 'tmdbId']]\n",
    "\n",
    "# filter tmdb valid movies\n",
    "movie_info = movie_info[movie_info['tmdbId'].notnull()]\n",
    "movie_info['tmdbId'] = movie_info['tmdbId'].astype(int).astype(str)\n",
    "movie_info.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "281b0a0a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating\n",
       "0       1        1     4.0\n",
       "1       1        3     4.0\n",
       "2       1        6     4.0\n",
       "3       1       47     5.0\n",
       "4       1       50     5.0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get movie user rating info\n",
    "movie_user_rating = pd.read_csv('ml-latest-small/ratings.csv')\n",
    "\n",
    "# remove ratings of movies which don't have tmdbId\n",
    "movie_user_rating = movie_user_rating[movie_user_rating['movieId'].isin(movie_info['movieId'])]\n",
    "movie_user_rating = movie_user_rating[[\"userId\", \"movieId\", \"rating\"]]\n",
    "movie_user_rating.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0ee58a2c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "userId      610\n",
       "movieId    9716\n",
       "rating       10\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_user_rating.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce67f2cf",
   "metadata": {},
   "source": [
    "### Generating user and movie vectors\n",
    "\n",
    "Non-negative Matrix Factorization (NMF) is a matrix factorization technique that decomposes a non-negative matrix `R` into two non-negative matrices `W` and `H`, where `R ≈ WH`. NMF is a commonly used technique in recommender systems for extracting latent features from high-dimensional sparse data such as user-item interaction matrices.\n",
    "\n",
    "In a recommendation system context, NMF can be utilized to factorize the user-item interaction matrix into two low-rank non-negative matrices: one matrix represents users' preferences for the latent features, and the other matrix represents how each item is related to these latent features. Given a user-item interaction matrix `R` of size `m` x `n`, we can factorize it into two non-negative matrices W and H, such that R is approximated by their product: `R ≈ W * H`. The factorization is achieved by minimizing the distance between `R` and `W * H`, subject to the non-negative constraints on `W` and `H`.\n",
    "\n",
    "The `W` and `H` matrices correspond to the user vector matrix and item vector matrix, respectively, and can be used as vector indices for queries later."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "338f703d",
   "metadata": {},
   "source": [
    "Let's start with creating a user-item matrix for movie ratings first, where each row represents a user and each column represents a movie. Each cell in the matrix represents the corresponding user rating for that movie. If a user has not rated a particular movie, the cell value will be set to 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0c86633d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import NMF\n",
    "from sklearn.preprocessing import MaxAbsScaler\n",
    "from scipy.sparse import csr_matrix\n",
    "\n",
    "user_indices, user_ids = pd.factorize(movie_user_rating['userId'])\n",
    "item_indices, movie_ids = pd.factorize(movie_user_rating['movieId'])\n",
    "rating_sparse_matrix = csr_matrix((movie_user_rating['rating'], (user_indices, item_indices)))\n",
    "\n",
    "# normalize matrix with MaxAbsScaler\n",
    "max_abs_scaler = MaxAbsScaler()\n",
    "rating_sparse_matrix = max_abs_scaler.fit_transform(rating_sparse_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "038538b3",
   "metadata": {},
   "source": [
    "After building our user-item matrix, we can fit NMF model with the matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d7a6a06b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reconstruction error:  31.316179423055555\n"
     ]
    }
   ],
   "source": [
    "# create NMF model with settings\n",
    "dimension = 512\n",
    "nmf_model = NMF(n_components=dimension, init='nndsvd', max_iter=500)\n",
    "\n",
    "# rating sparse matrix decomposition with NMF\n",
    "user_vectors = nmf_model.fit_transform(rating_sparse_matrix)\n",
    "item_vectors = nmf_model.components_.T\n",
    "\n",
    "error = nmf_model.reconstruction_err_\n",
    "print(\"Reconstruction error: \", error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f5dfdc3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate user vector matrix, containing userIds and user vectors\n",
    "user_vector_df = pd.DataFrame(zip(user_ids, user_vectors), columns=['userId', 'user_rating_vector']).reset_index(drop=True)\n",
    "\n",
    "# generate movie vector matrix, containing movieIds and movie vectors\n",
    "movie_rating_vector_df = pd.DataFrame(zip(movie_ids, item_vectors), columns=['movieId', 'movie_rating_vector'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e2b2991",
   "metadata": {},
   "source": [
    "### Creating Datasets\n",
    "\n",
    "We now have four dataframes: movie metadata, user movie ratings, user vectors and movie vectors. We will merge the relevant dataframes into a single dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ab1d0b8c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>tmdbId</th>\n",
       "      <th>movie_rating_vector</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>[Adventure, Animation, Children, Comedy, Fantasy]</td>\n",
       "      <td>862</td>\n",
       "      <td>[0.43727969455469096, 0.5435853936931008, 0.0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji (1995)</td>\n",
       "      <td>[Adventure, Children, Fantasy]</td>\n",
       "      <td>8844</td>\n",
       "      <td>[0.3812919324223999, 0.0, 0.0, 0.0, 0.13988881...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men (1995)</td>\n",
       "      <td>[Comedy, Romance]</td>\n",
       "      <td>15602</td>\n",
       "      <td>[0.6578283373638811, 0.0, 0.0, 0.9922804828007...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale (1995)</td>\n",
       "      <td>[Comedy, Drama, Romance]</td>\n",
       "      <td>31357</td>\n",
       "      <td>[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II (1995)</td>\n",
       "      <td>[Comedy]</td>\n",
       "      <td>11862</td>\n",
       "      <td>[0.31088958180836535, 0.0, 0.0, 0.0, 0.0699447...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movieId                               title  \\\n",
       "0        1                    Toy Story (1995)   \n",
       "1        2                      Jumanji (1995)   \n",
       "2        3             Grumpier Old Men (1995)   \n",
       "3        4            Waiting to Exhale (1995)   \n",
       "4        5  Father of the Bride Part II (1995)   \n",
       "\n",
       "                                              genres tmdbId  \\\n",
       "0  [Adventure, Animation, Children, Comedy, Fantasy]    862   \n",
       "1                     [Adventure, Children, Fantasy]   8844   \n",
       "2                                  [Comedy, Romance]  15602   \n",
       "3                           [Comedy, Drama, Romance]  31357   \n",
       "4                                           [Comedy]  11862   \n",
       "\n",
       "                                 movie_rating_vector  \n",
       "0  [0.43727969455469096, 0.5435853936931008, 0.0,...  \n",
       "1  [0.3812919324223999, 0.0, 0.0, 0.0, 0.13988881...  \n",
       "2  [0.6578283373638811, 0.0, 0.0, 0.9922804828007...  \n",
       "3  [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...  \n",
       "4  [0.31088958180836535, 0.0, 0.0, 0.0, 0.0699447...  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_rating_df = movie_user_rating.reset_index(drop=True)\n",
    "# add movie vectors into movie metadata and remove movies without movie vector\n",
    "movie_info_df = pd.merge(movie_info, movie_rating_vector_df, on=[\"movieId\"]).reset_index(drop=True)\n",
    "movie_info_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "66a7351b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyarrow as pa\n",
    "import pyarrow.parquet as pq\n",
    "\n",
    "# create table objects from the data and schema\n",
    "movie_table = pa.Table.from_pandas(movie_info_df)\n",
    "user_table = pa.Table.from_pandas(user_vector_df)\n",
    "rating_table = pa.Table.from_pandas(user_rating_df)\n",
    "\n",
    "# write the table to parquet files\n",
    "pq.write_table(movie_table, 'movie.parquet')\n",
    "pq.write_table(user_table, 'user.parquet')\n",
    "pq.write_table(rating_table, 'rating.parquet')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e858761",
   "metadata": {},
   "source": [
    "## Populating Data to MyScale\n",
    "\n",
    "###  Loading data\n",
    "\n",
    "To populate data to MyScale, first, we load data from the HuggingFace Dataset [myscale/recommendation-examples](https://huggingface.co/datasets/myscale/recommendation-examples) created in the previous section. The following code snippet shows how to load data and transform them into panda DataFrames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ed7ce4fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset parquet (/Users/guanxiaoyao/.cache/huggingface/datasets/myscale___parquet/myscale--recommendation-examples-cada9fb4a7f7e92a/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)\n",
      "Found cached dataset parquet (/Users/guanxiaoyao/.cache/huggingface/datasets/myscale___parquet/myscale--recommendation-examples-a4e48372fb027471/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)\n",
      "Found cached dataset parquet (/Users/guanxiaoyao/.cache/huggingface/datasets/myscale___parquet/myscale--recommendation-examples-4a978546f49558d3/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)\n"
     ]
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "movie = load_dataset(\"myscale/recommendation-examples\", data_files=\"movie.parquet\", split=\"train\")\n",
    "user = load_dataset(\"myscale/recommendation-examples\", data_files=\"user.parquet\", split=\"train\")\n",
    "rating = load_dataset(\"myscale/recommendation-examples\", data_files=\"rating.parquet\", split=\"train\")\n",
    "\n",
    "# transform datasets to panda Dataframe\n",
    "movie_info_df = movie.to_pandas()\n",
    "user_vector_df = user.to_pandas()\n",
    "user_rating_df = rating.to_pandas()\n",
    "\n",
    "# convert embedding vectors from np array to list\n",
    "movie_info_df['movie_rating_vector'] = movie_info_df['movie_rating_vector'].apply(lambda x: x.tolist())\n",
    "user_vector_df['user_rating_vector'] = user_vector_df['user_rating_vector'].apply(lambda x: x.tolist())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14c7d491",
   "metadata": {},
   "source": [
    "### Creating table\n",
    "\n",
    "Next, we'll create tables in MyScale. \n",
    "\n",
    "Before you begin, you will need to retrieve your cluster host, username, and password information from the MyScale console. The following code snippet creates three tables, for movie metadata, user vectors, and user movie ratings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0ac169ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "import clickhouse_connect\n",
    "\n",
    "# initialize client\n",
    "client = clickhouse_connect.get_client(\n",
    "    host='YOUR_CLUSTER_HOST', \n",
    "    port=8443, \n",
    "    username='YOUR_USERNAME', \n",
    "    password='YOUR_CLUSTER_PASSWORD'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2126d737",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "''"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.command(\"DROP TABLE IF EXISTS default.myscale_movies\")\n",
    "client.command(\"DROP TABLE IF EXISTS default.myscale_users\")\n",
    "client.command(\"DROP TABLE IF EXISTS default.myscale_ratings\")\n",
    "\n",
    "# create table for movies\n",
    "client.command(f\"\"\"\n",
    "CREATE TABLE default.myscale_movies\n",
    "(\n",
    "    movieId Int64,\n",
    "    title String,\n",
    "    genres Array(String),\n",
    "    tmdbId String,\n",
    "    movie_rating_vector Array(Float32),\n",
    "    CONSTRAINT vector_len CHECK length(movie_rating_vector) = 512\n",
    ")\n",
    "ORDER BY movieId\n",
    "\"\"\")\n",
    "\n",
    "# create table for user vectors\n",
    "client.command(f\"\"\"\n",
    "CREATE TABLE default.myscale_users\n",
    "(\n",
    "    userId Int64,\n",
    "    user_rating_vector Array(Float32),\n",
    "    CONSTRAINT vector_len CHECK length(user_rating_vector) = 512\n",
    ")\n",
    "ORDER BY userId\n",
    "\"\"\")\n",
    "\n",
    "# create table for user movie ratings\n",
    "client.command(\"\"\"\n",
    "CREATE TABLE default.myscale_ratings\n",
    "(\n",
    "    userId Int64,\n",
    "    movieId Int64,\n",
    "    rating Float64\n",
    ")\n",
    "ORDER BY userId\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3063b00",
   "metadata": {},
   "source": [
    "### Uploading Data\n",
    "\n",
    "After creating the tables, we insert data loaded from the datasets into tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "10a1b6fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "movies count: 9716\n",
      "users count: 610\n",
      "ratings count: 100823\n"
     ]
    }
   ],
   "source": [
    "client.insert(\"default.myscale_movies\", movie_info_df.to_records(index=False).tolist(), column_names=movie_info_df.columns.tolist())\n",
    "client.insert(\"default.myscale_users\", user_vector_df.to_records(index=False).tolist(), column_names=user_vector_df.columns.tolist())\n",
    "client.insert(\"default.myscale_ratings\", user_rating_df.to_records(index=False).tolist(), column_names=user_rating_df.columns.tolist())\n",
    "\n",
    "# check count of inserted data\n",
    "print(f\"movies count: {client.command('SELECT count(*) FROM default.myscale_movies')}\")\n",
    "print(f\"users count: {client.command('SELECT count(*) FROM default.myscale_users')}\")\n",
    "print(f\"ratings count: {client.command('SELECT count(*) FROM default.myscale_ratings')}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3ba4fe3d",
   "metadata": {},
   "source": [
    "### Building Index\n",
    "\n",
    "Now, our datasets are uploaded to MyScale. We will create a vector index to accelerate vector search after inserting datasets.\n",
    "\n",
    "We used `MSTG` as our vector search algorithm. For configuration details please refer to [Vector Reference](https://docs.myscale.com/en/vector-reference/).\n",
    "\n",
    "The inner product is used as the distance metric here. Specifically, the inner product between the query vector(representing a user's preferences) and the item vectors(representing movie features) yields the cell values in the matrix `R`, which can be approximated by the product of the matrices `W` and `H` as mentioned in the section [Generating user and movie vectors](#generating-user-and-movie-vectors)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cdb7fcda",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "''"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create vector index with cosine\n",
    "client.command(\"\"\"\n",
    "ALTER TABLE default.myscale_movies \n",
    "ADD VECTOR INDEX movie_rating_vector_index movie_rating_vector\n",
    "TYPE MSTG('metric_type=IP')\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "225dcd72",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "index build status: Built\n"
     ]
    }
   ],
   "source": [
    " # check the status of the vector index, make sure vector index is ready with 'Built' status\n",
    "get_index_status=\"SELECT status FROM system.vector_indices WHERE name='movie_rating_vector_index'\"\n",
    "print(f\"index build status: {client.command(get_index_status)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b94f487a",
   "metadata": {},
   "source": [
    "## Query MyScale\n",
    "\n",
    "### Performing query for movie recommendation\n",
    "\n",
    "Randomly selecting a user as the target user for whom we recommend movies, and obtain the user rating histogram, which exhibits the distribution of user ratings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "463e4046",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "currently selected user id=526 for movie recommendation\n",
      "\n",
      "Distribution of ratings for user 526:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAYv0lEQVR4nO3de5xV9X3u8c8jo/ECajiAh4s4sV4gohLFW/VQ1EAUjRFtVYJGCtaTU623xNQmaarmIEg1R0+1pSTykqgHk2isCIZCCWqlJgQvIMFYE0EZvCA1CHiHfM8fa0F2hrnsuey99vB73q/XvGbv3157rYc1+sya3157bUUEZmaWjl2KDmBmZtXl4jczS4yL38wsMS5+M7PEuPjNzBLj4jczS4yL35IiaZyk+QVuf5qkv+2kdQ2UtFlSt/z+Y5Iu6Yx15+v7iaSLO2t9Vjtc/NYqSSHpoEZj10u6twrbvlvSR3nBvS1pgaRBZT63Ps9et20sIu6LiFEVyrpa0vuSNknaIOk/JH1Z0vb/zyLiyxHx7TLX9dmWlomIVyOie0Rs7YTsO/w8I+L0iJjZ0XVb7XHxW80oLehGpkZEd6A/sBa4q3qp2uzzEdEDOACYAvw1Fcjbwr4ya5WL3zpMUi9Jc/Kj3Lcl/fu2o1xJ/SQ9KOktSaskXVHyvOslPSDpXkkbgfEtbSci3gd+CAwtWccZkp6VtFHSGknXlzzlifz7hvwvhhMkjZf0ZMnzIz8qf0nSbyXdKUn5Y90k3SppfZ798sZ/QbSQ9Z2ImA2cD1wsaUi+zrsl/e+W9puke4CBwCN57q+V/PUyUdKrwE+b+osG+CNJSyS9I+lhST3zbY2Q1NDo57Za0mclnQZ8HTg/396y/PHtU0d5rm9KekXSOknfl7RP/ti2HBdLejXfX99obR9ZcVz81hm+AjQAvYH9yEok8vJ/BFhGdrR+KnCVpM+VPPcLwAPAvsB9LW1E0l7AWODXJcPvAl/Kn38G8L8knZ0/Njz/vm8+JfJUM6s+EzgGOBI4D9iW7y+A08l+0RwFnN3Ec1sUEUvI9s3/aOLhJvdbRFwEvEr210P3iJha8pw/AQaXZGzsS8AEoB+wBfi/ZWScB9wE/CDf3pFNLDY+/zoZOBDoDtzRaJmTgEPJfs7fkjS4tW1bMVz81hk+BvoCB0TExxHx75FdBOoYoHdE3BgRH0XEy8B3gQtKnvtURPxLRPwuP6JvylclbQA2kZXLRdseiIjHIuL5/PnLgVlk5dgWUyJiQ0S8Cizi939RnAfcHhENEfFbsqmb9ngN6NnEeHP7rSXXR8S7LeyreyJiRUS8C/wtcN62F387aBzwnYh4OSI2A38DXNDor40bIuL9iFhG9su+qV8gVgNc/FaOrcCujcZ2JSsugL8nOwqfL+llSdfl4wcA/fKpjA15eX+d7Oh2mzVlbP+WiNgXqAfeJzuqBEDScZIW5VNJ7wBfBnq15R8HvFFy+z2yo1nIjppL85WTtSn9gbebGG9uv7WktQylj79C9nNq6/5oSr98faXrruMPf5bN7UerMS5+K8erZKVb6lPkRRARmyLiKxFxIPB54BpJp5KV0KqI2Lfkq0dEjC5ZT9mXh82PyK8Ebpe0Rz78/4DZwP4RsQ8wDVBb192M14EBJff3b+sKJB1DVvxPNn6shf0GzWdv7d9UmnEg2S/n9WRTYnuW5OpGNsVU7npfI/tFXrruLcCbrTzPapCL38rxA+CbkgbkL/J9lqyoHgCQdKakg/IXRTeS/YWwFVgCbJT015L2yF8sHZKXYbtExAKyEro0H+oBvB0RH0g6FvhiyeJvAb8jm5Nujx8CV0rqL2lfsjN0yiJpb0lnAvcD90bE800s09x+g6xQ25P7QkmflrQncCPwQH66538Cu+cvhu8KfBP4RMnz3gTqVXLqaSOzgKslfUpSd37/msCWdmS0grn4rRw3Av9BdtT6W2AqMC4iVuSPHwz8G7AZeAr4x3zufSvZL4ihwCqyI8/vAft0MM/fA1+T9AngL4EbJW0CvkVW1gBExHvAJGBxPtV0fBu3811gPrAceBZ4lOwot6Xz5h/Js6wBvgF8B/jzZpZtcr/lj00m+2W7QdJX25D5HuBusmmX3YErIDvLiGxffY/slNh3yV5Y3uZH+ff/kvRME+udka/7CbKf5QfAX7Uhl9UQ+YNYzMoj6XRgWkQc0OrCZjXMR/xmzcinp0ZLqpPUH/g74KGic5l1lI/4zZqRz5M/DgwiO5toLnBlRGwsNJhZB7n4zcwS46keM7PEdIkLPfXq1Svq6+uLjmFm1qU8/fTT6yOid+PxLlH89fX1LF26tOgYZmZdiqRXmhr3VI+ZtduaNWs4+eSTGTx4MIcddhi33347ANdeey2DBg3iiCOOYMyYMWzYsKHYoPYHXPxm1m51dXXceuutvPDCC/zsZz/jzjvvZOXKlYwcOZIVK1awfPlyDjnkECZPnlx0VCvh4jezduvbty9HHXUUAD169GDw4MGsXbuWUaNGUVeXzSQff/zxNDQ0tLQaqzIXv5l1itWrV/Pss89y3HHH/cH4jBkzOP300wtKZU1x8ZtZh23evJlzzz2X2267jb333nv7+KRJk6irq2PcuHEFprPGusRZPWZWuz7++GPOPfdcxo0bxznnnLN9fObMmcyZM4eFCxeSf5ql1QgXv5m1W0QwceJEBg8ezDXXXLN9fN68edx88808/vjj7Lnnni2swYrg4jezdlu8eDH33HMPhx9+OEOHDgXgpptu4oorruDDDz9k5MiRQPYC77Rp0wpMaqVc/GbWbieddBJNXe9r9OjRTSxttcLFb2bNqr9ubtW2tXrKGVXbVup8Vo+ZWWJc/GZmiXHxm5klxsVvZpYYF7+ZWWJc/GZmiXHxm5klxsVvZpYYF7+ZWWJc/GZmiXHxm5klxsVvZpYYF7+ZWWJc/GZmialY8UvaX9IiSS9I+qWkK/PxnpIWSHop//7JSmUwM7MdVfKIfwvwlYgYDBwPXCbp08B1wMKIOBhYmN83M7MqqVjxR8TrEfFMfnsT8ALQH/gCMDNfbCZwdqUymJnZjqoyxy+pHvgM8HNgv4h4HbJfDkCfamQwM7NMxYtfUnfgQeCqiNjYhuddKmmppKVvvfVW5QKamSWmosUvaVey0r8vIn6cD78pqW/+eF9gXVPPjYjpETEsIob17t27kjHNzJJSybN6BNwFvBAR3yl5aDZwcX77YuDhSmUwM7Md1VVw3ScCFwHPS3ouH/s6MAX4oaSJwKvAn1Uwg5mZNVKx4o+IJwE18/CpldqumZm1zO/cNTNLjIvfzCwxLn4zs8S4+M3MEuPiNzNLjIvfzCwxLn4zs8S4+M3MEuPiNzNLjIvfzCwxLn4zs8S4+M3MEuPiNzNLjIvfzCwxLn4zs8S4+M3MEuPiNzNLjIvfzCwxLn4zs8S4+M3MEuPiNzNLjIvfzCwxLn4zs8S4+M3MEuPiNzNLjIvfzCwxLn4zs8S4+M3MEuPiNzNLjIvfzCwxLn4zs8S4+M3MEuPiNzNLjIvfzCwxLn4zs8S4+M3MEuPiNzNLjIvfzCwxFSt+STMkrZO0omTseklrJT2Xf42u1PbNzKxplTzivxs4rYnx/xMRQ/OvRyu4fTMza0LFij8ingDertT6zcysfYqY479c0vJ8KuiTBWzfzCxp1S7+fwL+CBgKvA7c2tyCki6VtFTS0rfeeqtK8cysq5owYQJ9+vRhyJAhOzx2yy23IIn169cXkKz2VLX4I+LNiNgaEb8Dvgsc28Ky0yNiWEQM6927d/VCmlmXNH78eObNm7fD+Jo1a1iwYAEDBw4sIFVtqmrxS+pbcncMsKK5Zc3M2mL48OH07Nlzh/Grr76aqVOnIqmAVLWprlIrljQLGAH0ktQA/B0wQtJQIIDVwP+s1PbNzGbPnk3//v058sgji45SUypW/BExtonhuyq1PTOzUu+99x6TJk1i/vz5RUepOX7nrpntlH7zm9+watUqjjzySOrr62loaOCoo47ijTfeKDpa4Sp2xG9mVqTDDz+cdevWbb9fX1/P0qVL6dWrV4GpaoOP+M1spzB27FhOOOEEXnzxRQYMGMBdd3lmuTk+4jezncKsWbNafHz16tXVCdIFuPjNrObVXze3qttbPeWMqm6v2jzVY2aWGBe/mVliXPxmZolx8ZuZJcbFb2aWGBe/mVliXPxmZolx8ZuZJcbFb2aWGBe/mVknauojIK+99loGDRrEEUccwZgxY9iwYUNxASmz+CWdWM6YmVnqmvoIyJEjR7JixQqWL1/OIYccwuTJkwtKlyn3iP8fyhwzM0taUx8BOWrUKOrqskujHX/88TQ0NBQRbbsWL9Im6QTgj4Hekq4peWhvoFslg5mZ7YxmzJjB+eefX2iG1q7OuRvQPV+uR8n4RuBPKxXKzGxnNGnSJOrq6hg3blyhOVos/oh4HHhc0t0R8UqVMpmZ7XRmzpzJnDlzWLhwIZIKzVLu9fg/IWk6UF/6nIg4pRKhzMx2JvPmzePmm2/m8ccfZ8899yw6TtnF/yNgGvA9YGvl4piZdW1jx47lscceY/369QwYMIAbbriByZMn8+GHHzJy5Egge4F32rRphWUst/i3RMQ/VTSJmdlOoKmPgJw4cWIBSZpXbvE/IukvgYeAD7cNRsTbFUllZlajdoaPgSy3+C/Ov19bMhbAgZ0bx8zMKq2s4o+IT1U6iJmZVUdZxS/pS02NR8T3OzeOmZlVWrlTPceU3N4dOBV4BnDxm5l1MeVO9fxV6X1J+wD3VCSRmZlVVHsvy/wecHBnBjEzs+ood47/EbKzeCC7ONtg4IeVCmVmZpVT7hz/LSW3twCvRESx1xU1M7N2KWuqJ79Y26/IrtD5SeCjSoYyM7PKKfcTuM4DlgB/BpwH/FySL8tsZtYFlTvV8w3gmIhYByCpN/BvwAOVCmZmZpVR7lk9u2wr/dx/teG5ZmZWQ8o94p8n6V+BbZedOx94tDKRzMysklr7zN2DgP0i4lpJ5wAnAQKeAu6rQj4zM+tkrU3X3AZsAoiIH0fENRFxNdnR/m2VjWZmZpXQWvHXR8TyxoMRsZTsYxibJWmGpHWSVpSM9ZS0QNJL+fdPtiu1mZm1W2vFv3sLj+3RynPvBk5rNHYdsDAiDgYW5vfNzKyKWiv+X0j6i8aDkiYCT7f0xIh4Amj8CV1fAGbmt2cCZ5cX08zMOktrZ/VcBTwkaRy/L/phwG7AmHZsb7+IeB0gIl6X1Ke5BSVdClwKMHDgwHZsyszMmtJi8UfEm8AfSzoZGJIPz42In1Y6WERMB6YDDBs2LFpZ3MzMylTu9fgXAYs6YXtvSuqbH+33Bda1+gwzM+tU1X737Wx+/8HtFwMPV3n7ZmbJq1jxS5pF9kavQyU15C8ITwFGSnoJGJnfNzOzKir3kg1tFhFjm3no1Ept08zMWucLrZmZJcbFb2aWGBe/mVliXPxmZolx8ZuZJcbFb2aWGBe/mVliXPxmZolx8ZuZJcbFb2aWGBe/mVliXPxmZolx8ZuZJcbFb2aWGBe/mVliXPxmZolx8ZuZJcbFb2aWGBe/mVliXPxmZolx8ZuZJcbFb2aWGBe/mVliXPxmZolx8ZuZJcbFb2aWGBe/mVliXPxmZolx8ZuZJcbFb2aWGBe/mVliXPxmZompKzqAmXVcfX09PXr0oFu3btTV1bF06dKiI1kNc/Gb7SQWLVpEr169io5hXYCneszMEuPiN9sJSGLUqFEcffTRTJ8+veg4VuM81WO2E1i8eDH9+vVj3bp1jBw5kkGDBjF8+PCiY1mN8hG/2U6gX79+APTp04cxY8awZMmSghNZLSuk+CWtlvS8pOck+fQDsw5499132bRp0/bb8+fPZ8iQIQWnslpW5FTPyRGxvsDtm+0U3nzzTcaMGQPAli1b+OIXv8hpp51WcCqrZZ7jN+viDjzwQJYtW1Z0DOtCiir+AOZLCuCfI2KH0xAkXQpcCjBw4MAqxzMrz9atWxk2bBj9+/dnzpw5nbLO+uvmdsp6yrV6yhlV3Z4Vr6gXd0+MiKOA04HLJO1w+kFETI+IYRExrHfv3tVPaFaG22+/ncGDBxcdw6xNCin+iHgt/74OeAg4togcZh3R0NDA3LlzueSSS4qOYtYmVS9+SXtJ6rHtNjAKWFHtHGYdddVVVzF16lR22cVnRVvXUsR/sfsBT0paBiwB5kbEvAJymLXbnDlz6NOnD0cffXTRUczarOov7kbEy8CR1d6uWWdavHgxs2fP5tFHH+WDDz5g48aNXHjhhdx7771FRzNrlf9GNWuHyZMn09DQwOrVq7n//vs55ZRTXPrWZbj4zcwS4zdwmXXQiBEjGDFiRNExzMrm4jfLVfONU37TlBXJUz1mZolx8ZuZJcbFb2aWGBe/mVliXPxmZolx8ZuZJcbFb2aWGBe/mVliXPxmZolx8ZuZJcbFb2aWGBe/mVliXPxmZolx8ZuZJcbFb2aWGBe/mVliXPxmZolx8ZuZJcbFb13KhAkT6NOnD0OGDCk6ilmX5eK3LmX8+PHMmzev6BhmXZqL37qU4cOH07Nnz6JjmHVpLn4zs8S4+M3MEuPiNzNLjIvfzCwxLn7rUsaOHcsJJ5zAiy++yIABA7jrrruKjmTW5dQVHcCsLWbNmlV0BLMuz8Vvhaq/bm5Vt7d6yhlV3Z5ZLfJUTyPz5s3j0EMP5aCDDmLKlCnOUoNZzKxjXPwltm7dymWXXcZPfvITVq5cyaxZs1i5cqWz1FAWM+s4F3+JJUuWcNBBB3HggQey2267ccEFF/Dwww87Sw1lMbOOc/GXWLt2Lfvvv//2+wMGDGDt2rXOUkNZzKzjXPwlImKHMUkFJHEWM6scF3+JAQMGsGbNmu33Gxoa6Nevn7PUUBYz67hCil/SaZJelPRrSdcVkaEpxxxzDC+99BKrVq3io48+4v777+ess85ylhrKYmYdV/Xz+CV1A+4ERgINwC8kzY6Iwk8Tqaur44477uBzn/scW7duZcKECRx22GHOUkNZzKzjingD17HAryPiZQBJ9wNfAAovfoDRo0czevToomMAzmJmlaGmXrir6AalPwVOi4hL8vsXAcdFxOWNlrsUuDS/eyjwYlWDQi9gfZW3Weu8T3bkfdI075cdFbFPDoiI3o0Hizjib+p0kB1++0TEdGB65eM0TdLSiBhW1PZrkffJjrxPmub9sqNa2idFvLjbAOxfcn8A8FoBOczMklRE8f8COFjSpyTtBlwAzC4gh5lZkqo+1RMRWyRdDvwr0A2YERG/rHaOMhQ2zVTDvE925H3SNO+XHdXMPqn6i7tmZlYsv3PXzCwxLn4zs8S4+BuRNEPSOkkris5SKyTtL2mRpBck/VLSlUVnKpqk3SUtkbQs3yc3FJ2pVkjqJulZSXOKzlILJK2W9Lyk5yQtLToPeI5/B5KGA5uB70fEkKLz1AJJfYG+EfGMpB7A08DZtXCZjaIouzzpXhGxWdKuwJPAlRHxs4KjFU7SNcAwYO+IOLPoPEWTtBoYFhE184Y2H/E3EhFPAG8XnaOWRMTrEfFMfnsT8ALQv9hUxYrM5vzurvlX8kdRkgYAZwDfKzqLNc/Fb20iqR74DPDzgqMULp/SeA5YByyIiOT3CXAb8DXgdwXnqCUBzJf0dH4pmsK5+K1skroDDwJXRcTGovMULSK2RsRQsnefHysp6alBSWcC6yLi6aKz1JgTI+Io4HTgsnw6uVAufitLPo/9IHBfRPy46Dy1JCI2AI8BpxWbpHAnAmflc9r3A6dIurfYSMWLiNfy7+uAh8iuUFwoF7+1Kn8h8y7ghYj4TtF5aoGk3pL2zW/vAXwW+FWhoQoWEX8TEQMiop7sUiw/jYgLC45VKEl75SdEIGkvYBRQ+BmDLv5GJM0CngIOldQgaWLRmWrAicBFZEdwz+VfqV+cvy+wSNJysutPLYgIn75oje0HPClpGbAEmBsR8wrO5NM5zcxS4yN+M7PEuPjNzBLj4jczS4yL38wsMS5+M7PEuPgteZK25qeorpD0yLbz81tYfmjp6aySzpJ0XcWDmnUSn85pyZO0OSK657dnAv8ZEZNaWH482dUWL69SRLNOVfXP3DWrcU8BRwBIOpbsomN7AO8Dfw6sAm4E9pB0EjA5f3xYRFwu6W5gI9llif878LWIeEDSLsAdwJ/k69iF7POmH6jeP80s46kes5ykbsCpwOx86FfA8Ij4DPAt4KaI+Ci//YOIGBoRP2hiVX2Bk4AzgSn52DlAPXA4cAlwQqX+HWat8RG/WXb0/hxZMT8NLMjH9wFmSjqY7NK6u5a5vn+JiN8BKyXtl4+dBPwoH39D0qLOCm/WVj7iN4P388srHwDsBlyWj38bWJR/Etvngd3LXN+HJbfV6LtZ4Vz8ZrmIeAe4AvhqfhnqfYC1+cPjSxbdBPRo4+qfBM6VtEv+V8CIjqU1az8Xv1mJiHgWWEZ2WeGpwGRJi4FuJYstAj6dnwJ6fpmrfhBoILsk7z+TfYLZO50W3KwNfDqnWZVI6p5/OPt/I7tE74kR8UbRuSw9fnHXrHrm5G8O2w34tkvfiuIjfjOzxHiO38wsMS5+M7PEuPjNzBLj4jczS4yL38wsMf8fYSQ6SDiWvrQAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "random_user = client.query(\"SELECT * FROM default.myscale_users ORDER BY rand() LIMIT 1\")\n",
    "assert random_user.row_count == 1\n",
    "target_user_id = random_user.first_item[\"userId\"]\n",
    "target_user_vector = random_user.first_item[\"user_rating_vector\"]\n",
    "\n",
    "print(\"currently selected user id={} for movie recommendation\\n\".format(target_user_id))\n",
    "\n",
    "# user rating plot\n",
    "target_user_ratings = user_rating_df.loc[user_rating_df['userId'] == target_user_id]['rating'].tolist()\n",
    "bins = np.arange(1.0, 6, 0.5)\n",
    "\n",
    "# Compute the histogram\n",
    "hist, _ = np.histogram(target_user_ratings, bins=bins)\n",
    "\n",
    "print(\"Distribution of ratings for user {}:\".format(target_user_id))\n",
    "plt.bar(bins[:-1], hist, width=0.4)\n",
    "plt.xlabel('Rating')\n",
    "plt.ylabel('Count')\n",
    "plt.title('User Rating Distribution')\n",
    "for i in range(len(hist)):\n",
    "    plt.text(bins[i], hist[i], str(hist[i]), ha='center', va='bottom')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf49c790",
   "metadata": {},
   "source": [
    "Next, let's recommend movies for the user.\n",
    "\n",
    "As described in [Generating user and movie vectors](#generating-user-and-movie-vectors) and [Building Index](#building-index) sections, our user and movie vectors are extracted from the NMF model, and the inner products of vectors serve as our vector distance metrics. The formula of inner product of two vectors can be  simplified as follows:\n",
    "\n",
    "$$\\langle \\mathbf{u}, \\mathbf{v} \\rangle = \\sum_{i=1}^{n} u_i v_i = u_1 v_1 + u_2 v_2 + \\cdots + u_n v_n$$\n",
    "\n",
    "More specifically, we can obtain an approximated user-rating matrix using the inner product of the user vector matrix and the movie vector matrix based on NMF model. The value of the cell located at `(i, j)` represents the estimated rating of user `i` to movie `j`. Therefore, the distances between user vectors and movie vectors, represented by their inner products, can be used to recommend movies to users. Higher distances correspond to higher estimated movie ratings. \n",
    "\n",
    "However, since we normalized the rating matrix in previous sections, we still need to scale the distances to the new rating scale `(0, 5)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7374f660",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 10 movie recommandations with estimated ratings for user 526\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>title</th>\n",
       "      <th>estimated_rating</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>158966</td>\n",
       "      <td>Captain Fantastic (2016)</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>[Drama]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>79702</td>\n",
       "      <td>Scott Pilgrim vs. the World (2010)</td>\n",
       "      <td>4.930944</td>\n",
       "      <td>[Action, Comedy, Fantasy, Musical, Romance]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>4.199992</td>\n",
       "      <td>[Adventure, Animation, Children, Comedy, Fantasy]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8874</td>\n",
       "      <td>Shaun of the Dead (2004)</td>\n",
       "      <td>4.021980</td>\n",
       "      <td>[Comedy, Horror]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>68157</td>\n",
       "      <td>Inglourious Basterds (2009)</td>\n",
       "      <td>3.808410</td>\n",
       "      <td>[Action, Drama, War]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>44191</td>\n",
       "      <td>V for Vendetta (2006)</td>\n",
       "      <td>3.678385</td>\n",
       "      <td>[Action, Sci-Fi, Thriller, IMAX]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>6539</td>\n",
       "      <td>Pirates of the Caribbean: The Curse of the Bla...</td>\n",
       "      <td>3.654729</td>\n",
       "      <td>[Action, Adventure, Comedy, Fantasy]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8636</td>\n",
       "      <td>Spider-Man 2 (2004)</td>\n",
       "      <td>3.571647</td>\n",
       "      <td>[Action, Adventure, Sci-Fi, IMAX]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>6333</td>\n",
       "      <td>X2: X-Men United (2003)</td>\n",
       "      <td>3.458405</td>\n",
       "      <td>[Action, Adventure, Sci-Fi, Thriller]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>8360</td>\n",
       "      <td>Shrek 2 (2004)</td>\n",
       "      <td>3.417371</td>\n",
       "      <td>[Adventure, Animation, Children, Comedy, Music...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movieId                                              title  \\\n",
       "0   158966                           Captain Fantastic (2016)   \n",
       "1    79702                 Scott Pilgrim vs. the World (2010)   \n",
       "2        1                                   Toy Story (1995)   \n",
       "3     8874                           Shaun of the Dead (2004)   \n",
       "4    68157                        Inglourious Basterds (2009)   \n",
       "5    44191                              V for Vendetta (2006)   \n",
       "6     6539  Pirates of the Caribbean: The Curse of the Bla...   \n",
       "7     8636                                Spider-Man 2 (2004)   \n",
       "8     6333                            X2: X-Men United (2003)   \n",
       "9     8360                                     Shrek 2 (2004)   \n",
       "\n",
       "   estimated_rating                                             genres  \n",
       "0          5.000000                                            [Drama]  \n",
       "1          4.930944        [Action, Comedy, Fantasy, Musical, Romance]  \n",
       "2          4.199992  [Adventure, Animation, Children, Comedy, Fantasy]  \n",
       "3          4.021980                                   [Comedy, Horror]  \n",
       "4          3.808410                               [Action, Drama, War]  \n",
       "5          3.678385                   [Action, Sci-Fi, Thriller, IMAX]  \n",
       "6          3.654729               [Action, Adventure, Comedy, Fantasy]  \n",
       "7          3.571647                  [Action, Adventure, Sci-Fi, IMAX]  \n",
       "8          3.458405              [Action, Adventure, Sci-Fi, Thriller]  \n",
       "9          3.417371  [Adventure, Animation, Children, Comedy, Music...  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_k = 10\n",
    "# query the database to find the top K recommende\n",
    "# d movies\n",
    "recommended_results = client.query(f\"\"\"\n",
    "SELECT movieId, title, genres, tmdbId, distance(movie_rating_vector, {target_user_vector}) AS dist\n",
    "FROM default.myscale_movies\n",
    "WHERE movieId not in (\n",
    "    SELECT movieId\n",
    "    from default.myscale_ratings\n",
    "    where userId = {target_user_id}\n",
    "    )\n",
    "ORDER BY dist DESC\n",
    "LIMIT {top_k}\n",
    "\"\"\")\n",
    "\n",
    "recommended_movies = pd.DataFrame.from_records(recommended_results.named_results())\n",
    "rated_score_scale = client.query(f\"\"\"\n",
    "SELECT max(rating) AS max, min(rating) AS min\n",
    "FROM default.myscale_ratings\n",
    "WHERE userId = {target_user_id}\n",
    "\"\"\")\n",
    "max_rated_score = rated_score_scale.first_row[0]\n",
    "min_rated_score = rated_score_scale.first_row[1]\n",
    "\n",
    "print(\"Top 10 movie recommandations with estimated ratings for user {}\".format(target_user_id))\n",
    "max_dist = recommended_results.first_row[4]\n",
    "recommended_movies['estimated_rating'] = min_rated_score + ((max_rated_score - min_rated_score) / max_dist) * recommended_movies['dist']\n",
    "recommended_movies[['movieId', 'title', 'estimated_rating', 'genres']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4db98795",
   "metadata": {},
   "source": [
    "Compile a list of genres for the top 10 highest-rated movies and top 10 recommended movies to ascertain the similarity of their movie genres."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b375c183",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Genres of top 10 highest-rated and recommended movies for user 526:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>occurrence_in_rated_movie</th>\n",
       "      <th>occurrence_in_recommended_movie</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Drama</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Comedy</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Romance</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>War</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Adventure</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    category  occurrence_in_rated_movie  occurrence_in_recommended_movie\n",
       "1      Drama                          8                                2\n",
       "0     Comedy                          5                                5\n",
       "2    Romance                          3                                2\n",
       "3        War                          2                                1\n",
       "4  Adventure                          1                                5"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# count rated movies\n",
    "rated_count = len(user_rating_df[user_rating_df[\"userId\"] == target_user_id])\n",
    "\n",
    "# query the database to find the top K recommended watched movies for user\n",
    "rated_results = client.query(f\"\"\"\n",
    "SELECT movieId, genres, tmdbId, dist, rating\n",
    "FROM (SELECT * FROM default.myscale_ratings WHERE userId = {target_user_id}) AS ratings\n",
    "INNER JOIN (\n",
    "    SELECT movieId, genres, tmdbId, distance(movie_rating_vector, {target_user_vector}) AS dist\n",
    "    FROM default.myscale_movies\n",
    "    WHERE movieId in ( SELECT movieId FROM default.myscale_ratings WHERE userId = {target_user_id} )\n",
    "    ORDER BY dist DESC\n",
    "    LIMIT {rated_count}\n",
    ") AS movie_info\n",
    "ON ratings.movieId = movie_info.movieId\n",
    "WHERE rating >= (\n",
    "    SELECT MIN(rating) FROM (\n",
    "        SELECT least(rating) AS rating FROM default.myscale_ratings WHERE userId = {target_user_id} ORDER BY rating DESC LIMIT {top_k})\n",
    "    )\n",
    "ORDER BY dist DESC\n",
    "LIMIT {top_k}\n",
    "\"\"\")\n",
    "\n",
    "print(\"Genres of top 10 highest-rated and recommended movies for user {}:\".format(target_user_id))\n",
    "rated_genres = {}\n",
    "for r in rated_results.named_results():\n",
    "    for tag in r['genres']:\n",
    "        rated_genres[tag] = rated_genres.get(tag, 0) + 1\n",
    "rated_tags = pd.DataFrame(rated_genres.items(), columns=['category', 'occurrence_in_rated_movie'])\n",
    "\n",
    "recommended_genres = {}\n",
    "for r in recommended_results.named_results():\n",
    "    for tag in r['genres']:\n",
    "        recommended_genres[tag] = recommended_genres.get(tag, 0) + 1\n",
    "recommended_tags = pd.DataFrame(recommended_genres.items(), columns=['category', 'occurrence_in_recommended_movie'])\n",
    "inner_join_tags = pd.merge(rated_tags, recommended_tags, on='category', how='inner')\n",
    "inner_join_tags = inner_join_tags.sort_values('occurrence_in_rated_movie', ascending=False)\n",
    "inner_join_tags"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3664099",
   "metadata": {},
   "source": [
    "Additionally, we can retrieve the top 10 rated movies with their actual rating scores and predicted scores, to observe the similarity between the user ratings and our estimated ratings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e7bc16e1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 10 highest-rated movies along with their respective user scores and predicted ratings for the user 526\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>estimated_rating</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2324</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>[Comedy, Drama, Romance, War]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>90430</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.925842</td>\n",
       "      <td>[Comedy, Drama]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>128620</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.925816</td>\n",
       "      <td>[Crime, Drama, Romance]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>63876</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.925714</td>\n",
       "      <td>[Drama]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6807</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.925266</td>\n",
       "      <td>[Comedy]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>3967</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.924646</td>\n",
       "      <td>[Drama]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3448</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.923244</td>\n",
       "      <td>[Comedy, Drama, War]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>4027</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.922347</td>\n",
       "      <td>[Adventure, Comedy, Crime]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>215</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.922092</td>\n",
       "      <td>[Drama, Romance]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>112290</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.918183</td>\n",
       "      <td>[Drama]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movieId  rating  estimated_rating                         genres\n",
       "0     2324     5.0          5.000000  [Comedy, Drama, Romance, War]\n",
       "1    90430     5.0          4.925842                [Comedy, Drama]\n",
       "2   128620     5.0          4.925816        [Crime, Drama, Romance]\n",
       "3    63876     5.0          4.925714                        [Drama]\n",
       "4     6807     5.0          4.925266                       [Comedy]\n",
       "5     3967     5.0          4.924646                        [Drama]\n",
       "6     3448     5.0          4.923244           [Comedy, Drama, War]\n",
       "7     4027     5.0          4.922347     [Adventure, Comedy, Crime]\n",
       "8      215     5.0          4.922092               [Drama, Romance]\n",
       "9   112290     5.0          4.918183                        [Drama]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rated_movies = pd.DataFrame.from_records(rated_results.named_results())\n",
    "print(\"Top 10 highest-rated movies along with their respective user scores and predicted ratings for the user {}\".format(target_user_id))\n",
    "max_dist = rated_results.first_row[3]\n",
    "rated_movies['estimated_rating'] = min_rated_score + ((max_rated_score - min_rated_score) / max_dist) * rated_movies['dist']\n",
    "rated_movies[['movieId', 'rating', 'estimated_rating', 'genres']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7de7e45",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "40d3a090f54c6569ab1632332b64b2c03c39dcf918b08424e98f38b5ae0af88f"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}