ML Model - Predict Sentiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Classes\n",
    "\n",
    "This section here was to get some practice using classes. Of course I could have written all of this without classes but i think this made it a little easier to use and understand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "class Sentiment:\n",
    "    NEGATIVE = \"NEGATIVE\"\n",
    "    NEUTRAL = \"NEUTRAL\"\n",
    "    POSITIVE = \"POSITIVE\"\n",
    "\n",
    "class Review:\n",
    "    def __init__(self, text, score):\n",
    "        self.text = text\n",
    "        self.score = score\n",
    "        self.sentiment = self.get_sentiment()\n",
    "        \n",
    "    def get_sentiment(self):\n",
    "        if self.score <= 2:\n",
    "            return Sentiment.NEGATIVE\n",
    "        elif self.score == 3:\n",
    "            return Sentiment.NEUTRAL\n",
    "        else: #Score of 4 or greater\n",
    "            return Sentiment.POSITIVE\n",
    "        \n",
    "class ReviewContainer:\n",
    "    def __init__(self, reviews):\n",
    "        self.reviews = reviews\n",
    "        \n",
    "    def get_text(self):\n",
    "        return [x.text for x in self.reviews]\n",
    "    \n",
    "    def get_sentiment(self):\n",
    "        return [x.sentiment for x in self.reviews]\n",
    "        \n",
    "    def evenly_distribute(self):\n",
    "        negative = list(filter(lambda x: x.sentiment == Sentiment.NEGATIVE, self.reviews))\n",
    "        positive = list(filter(lambda x: x.sentiment == Sentiment.POSITIVE, self.reviews))\n",
    "        positive_shrunk = positive[:len(negative)]\n",
    "        self.reviews = negative + positive_shrunk\n",
    "        random.shuffle(self.reviews)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Excellent book Rick was a great believable character very romantic I want to read about stem next I hope she writes about him'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "import os\n",
    "\n",
    "file_name = os.path.join('data', 'sentiment', 'Books_small_10000.json')\n",
    "\n",
    "reviews = []\n",
    "\n",
    "with open(file_name) as file:\n",
    "    for line in file:\n",
    "        review = json.loads(line)\n",
    "        reviews.append(Review(review['reviewText'],review['overall']))\n",
    "        \n",
    "reviews[88].text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prep Data\n",
    "\n",
    "At this point in time I haven't use the Sci-Kit Learn library very much so I'm not 100% sure if there's a neater way to do this. This section takes all of the data and passes it into the *test_train_split* method from sklearn. This one step allowed me to take the data and split it into a training send and a test set. I decided to use 66% of the data for training and the test for testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "# This will split our data into a training set and a testing set.\n",
    "training, test = train_test_split(reviews, test_size=0.33, random_state = 42)\n",
    "\n",
    "train_container = ReviewContainer(training)\n",
    "\n",
    "test_container = ReviewContainer(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This will split the training and test variables into the review text and its sentiment value\n",
    "\n",
    "train_container.evenly_distribute()\n",
    "training_x = train_container.get_text()\n",
    "training_y = train_container.get_sentiment()\n",
    "\n",
    "test_container.evenly_distribute()\n",
    "test_x = test_container.get_text()\n",
    "test_y = test_container.get_sentiment()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bag of Words Vectorization\n",
    "\n",
    "Bag of words is a cool technique that I learned that takes the the occurance of every word in the entire dataset and adds it to a dictionary and then counts the number of times a word is used. For example:\n",
    "The phrase: John likes to watch movies. Mary likes movies too.\n",
    "Would be converted to {\"too\":1,\"Mary\":1,\"movies\":2,\"John\":1,\"watch\":1,\"likes\":2,\"to\":1}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<872x8906 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 53647 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer #Use Tfidf instead next time\n",
    "\n",
    "vectorizer = CountVectorizer()\n",
    "train_x_vectors = vectorizer.fit_transform(training_x)\n",
    "test_x_vectors = vectorizer.transform(test_x)\n",
    "\n",
    "train_x_vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification\n",
    "\n",
    "In this section I wanted to see how different classifications would perform when predicting with the test set. They all seemed to get the job done but I needed to know if one was more accururate than than the others."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Linear SVM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['POSITIVE'], dtype='<U8')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import svm #support vector machine\n",
    "\n",
    "clf_svm = svm.SVC(kernel='linear')\n",
    "\n",
    "clf_svm.fit(train_x_vectors, training_y)\n",
    "\n",
    "test_x[0]\n",
    "\n",
    "clf_svm.predict(test_x_vectors[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['NEGATIVE'], dtype='<U8')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "clf_dec = DecisionTreeClassifier()\n",
    "clf_dec.fit(train_x_vectors, training_y)\n",
    "\n",
    "clf_dec.predict(test_x_vectors[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['POSITIVE'], dtype='<U8')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "clf_log = LogisticRegression()\n",
    "clf_log.fit(train_x_vectors, training_y)\n",
    "\n",
    "clf_log.predict(test_x_vectors[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "After testing the accuracy I realized that they were all pretty close with SVM and Logistic Regression performing similarily. For our purposes in this project I think SVM will do just fine. When testing the F1 score I relaized that the value was pretty good POSITIVE but was less than 0.10 for NEGATIVE. I realized this was because the data is heavily biased towards POSITIVE reviews since they made up the majority of the data. This lead to me having to go back and remake my training and test set and downsample my positive reviews so that they more closely matched the number of negative and now they perform evenly.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7980769230769231\n",
      "0.6634615384615384\n",
      "0.8149038461538461\n"
     ]
    }
   ],
   "source": [
    "# Check to see how accurate these classifcations are at predicting the correct values\n",
    "\n",
    "# Mean accuracy\n",
    "print(clf_svm.score(test_x_vectors, test_y))\n",
    "print(clf_dec.score(test_x_vectors, test_y))\n",
    "print(clf_log.score(test_x_vectors, test_y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.8028169 , 0.79310345])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# F1 Scores\n",
    "from sklearn.metrics import f1_score\n",
    "\n",
    "f1_score(test_y, clf_svm.predict(test_x_vectors), average = None, labels = [Sentiment.POSITIVE, Sentiment.NEGATIVE])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing With New Data\n",
    "\n",
    "At this point the training is complete and the model can be testing with new data it has never seen before!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['NEGATIVE', 'NEGATIVE', 'NEGATIVE'], dtype='<U8')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_set = ['not great book', 'probably would not purchase this again', 'really disliked it']\n",
    "new_test = vectorizer.transform(test_set)\n",
    "\n",
    "clf_svm.predict(new_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Ways to Improve\n",
    "\n",
    "There are definily a few things I could change to make this model better. For example the data set came from book reviews so the model does very well when the string contains the word book or something similar. If I passed in \"I really enjoyed the video\" it returns NEGATIVE sometimes because the model has never seen the word video and does understand its sentimenent. This can be fixed in many ways, either by having a larger more generalized dataset or by designing the model to deconstruct sentences and identify verbs, nouns, and participles to better understand language. \n",
    "\n",
    "Another imporovement that might be made is changing the vertorizor from Bag of Words to a term frequency–inverse document frequency model so that common words like \"the, and, is, or\" arent weighted equally as more relevent words like \"great, aweful, brilliant, distasteful\". I did try TFIDF in this model but with this dataset the results were negligable. More research is required."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}