project2_1.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "# Initialize Otter\n",
    "import otter\n",
    "grader = otter.Notebook(\"project2.ipynb\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project 2: Movie Classification\n",
    "Welcome to the second project of Data 8! You will build a classification model that guesses whether a movie is a comedy or a thriller by using only the number of times chosen words appear in the movies's screenplay. By the end of the project, you should know how to:\n",
    "\n",
    "1. Build a k-nearest-neighbors classifier.\n",
    "2. Test a classifier on data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistics\n",
    "\n",
    "\n",
    "**Deadline.** Sunday 8/8 at 11:59pm P.T.\n",
    "\n",
    "**Checkpoint.** For full credit on the checkpoint, you must complete the first 2 sections, pass all public autograder tests for those sections, and submit to the Gradescope Checkpoint assignment by 11:59pm P.T. on Friday, 8/6. **This is worth 5% of your entire project grade**. After you've submitted the checkpoint, you may still change your answers before the project deadline - only your final submission, to the Project 1 assignment, will be graded for correctness. We recommend that you start the project as soon as possible.\n",
    "\n",
    "**Partners.** You may work with one other partner; your partner must be from your assigned lab section. **Only one of you is allowed to submit the project. If both partners submit, you will be docked 10% of your project grade, and we will use whichever submission yields you fewer points.** \n",
    "\n",
    "On Gradescope, the person who submits should also designate their partner so that both of you receive credit. Feel free to split up the work between your partner, share what you complete with each other in a google document, and share your screen in a Zoom call as you work together.\n",
    "\n",
    "**[Watch this tutorial video](https://drive.google.com/file/d/19hdMiqeO6svyPxOtAfHMDOj99HorcO5_/view?usp=sharing) on how to add a project partner on Gradescope.**\n",
    "\n",
    "**Rules.** Don't share your code with anybody but your partner. You are welcome to discuss questions with other students, but don't share the answers. The experience of solving the problems in this project will prepare you for exams (and life). If someone asks you for the answer, resist! Instead, you can demonstrate how you would solve a similar problem.\n",
    "\n",
    "**Support.** You are not alone! Come to office hours, post on Piazza, and talk to your classmates. If you want to ask about the details of your solution to a problem, make a private Piazza post and the staff will respond. If you're ever feeling overwhelmed or don't know how to make progress, email your TA or tutor for help. You can find contact information for the staff on the [course website](http://data8.org/su21/staff.html).\n",
    "\n",
    "**Tests.** The tests that are given are **not comprehensive** and passing the tests for a question **does not** mean that you answered the question correctly. Tests usually only check that your table has the correct column labels. However, more tests will be applied to verify the correctness of your submission in order to assign your final score, so be careful and check your work! You might want to create your own checks along the way to see if your answers make sense. Additionally, before you submit, make sure that none of your cells take a very long time to run (several minutes).\n",
    "\n",
    "**Free Response Questions:** Make sure that you put the answers to the written questions in the indicated cell we provide. **Every free response question should include an explanation** that adequately answers the question.\n",
    "\n",
    "**Advice.** Develop your answers incrementally. To perform a complicated table manipulation, break it up into steps, perform each step on a different line, give a new name to each result, and check that each intermediate result is what you expect. You can add any additional names or functions you want to the provided cells. Make sure that you are using distinct and meaningful variable names throughout the notebook. Along that line, **DO NOT** reuse the variable names that we use when we grade your answers. For example, in Question 1 of the New Technology section we ask you to assign an answer to `sc`. Do not reassign the variable name `sc` to anything else in your notebook, otherwise there is the chance that our tests grade against what `sc` was reassigned to.\n",
    "\n",
    "**You should have scratch paper or some other place to work out problems before trying them in the notebook. This will greatly improve your ability to solve some of the more difficult questions.**\n",
    "\n",
    "You **never** have to use just one line in this project or any others. Use intermediate variables and multiple lines as much as you would like!\n",
    "\n",
    "All of the concepts necessary for this project are found in the textbook. If you are stuck on a particular problem, reading through the relevant textbook section often will help clarify the concept.\n",
    "\n",
    "To get started, load `datascience`, `numpy`, `plots`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Don't change this cell; just run it. \n",
    "# When you log-in please hit return (not shift + return) after typing in your email\n",
    "\n",
    "from datascience import *\n",
    "import numpy as np\n",
    "import math\n",
    "\n",
    "# These lines set up the plotting functionality and formatting.\n",
    "import matplotlib\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plots\n",
    "plots.style.use('fivethirtyeight')\n",
    "import warnings\n",
    "warnings.simplefilter(action=\"ignore\", category=FutureWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Run this cell to set up the notebook, but please don't change it.\n",
    "import numpy as np\n",
    "import math\n",
    "import datascience\n",
    "from datascience import *\n",
    "\n",
    "# These lines set up the plotting functionality and formatting.\n",
    "import matplotlib\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plots\n",
    "plots.style.use('fivethirtyeight')\n",
    "import warnings\n",
    "warnings.simplefilter(action=\"ignore\", category=FutureWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: The Dataset\n",
    "\n",
    "In this project, we are exploring movie screenplays. We'll be trying to predict each movie's genre from the text of its screenplay. In particular, we have compiled a list of 5,000 words that occur in conversations between movie characters. For each movie, our dataset tells us the frequency with which each of these words occurs in certain conversations in its screenplay. All words have been converted to lowercase.\n",
    "\n",
    "Run the cell below to read the `movies` table. **It may take up to a minute to load.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies = Table.read_table('movies.csv')\n",
    "movies.where(\"Title\", \"runaway bride\").select(0, 1, 2, 3, 4, 14, 49, 1042, 4004)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above cell prints a few columns of the row for the comedy movie *Runaway Bride*.  The movie contains 4895 words. The word \"it\" appears 115 times, as it makes up  $\\frac{115}{4895} \\approx 0.0234092$ of the words in the movie. The word \"england\" doesn't appear at all.\n",
    "This numerical representation of a body of text, one that describes only the frequencies of individual words, is called a bag-of-words representation. This is a model that is often used in [NLP](https://en.wikipedia.org/wiki/Natural_language_processing). A lot of information is discarded in this representation: the order of the words, the context of each word, who said what, the cast of characters and actors, etc. However, a bag-of-words representation is often used for machine learning applications as a reasonable starting point, because a great deal of information is also retained and expressed in a convenient and compact format. In this project, we will investigate whether this representation is sufficient to build an accurate genre classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All movie titles are unique. The `row_for_title` function provides fast access to the one row for each title. \n",
    "\n",
    "*Note: All movies in our dataset have their titles lower-cased.* "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "title_index = movies.index_by('Title')\n",
    "def row_for_title(title):\n",
    "    \"\"\"Return the row for a title, similar to the following expression (but faster)\n",
    "    \n",
    "    movies.where('Title', title).row(0)\n",
    "    \"\"\"\n",
    "    return title_index.get(title)[0]\n",
    "\n",
    "row_for_title('toy story')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, the fastest way to find the frequency of \"fun\" in the movie *Toy Story* is to access the `'fun'` item from its row. Check the original table to see if this worked for you!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "row_for_title('toy story').item('fun') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 1.0\n",
    "Set `expected_row_sum` to the number that you __expect__ will result from summing all proportions in each row, excluding the first five columns. Think about what any one row adds up to. **(1 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q1_0\n",
    "points:\n",
    " - 0\n",
    " - 1\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "# Set row_sum to a number that's the (approximate) sum of each row of word proportions.\n",
    "expected_row_sum = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q1_0\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset was extracted from [a dataset from Cornell University](http://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html). After transforming the dataset (e.g., converting the words to lowercase, removing the naughty words, and converting the counts to frequencies), we created this new dataset containing the frequency of 5000 common words in each movie."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Words with frequencies:', movies.drop(np.arange(5)).num_columns) \n",
    "print('Movies with genres:', movies.num_rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1. Word Stemming\n",
    "The columns other than \"Title\", \"Year\", \"Rating\", \"Genre\", and \"# Words\" in the `movies` table are all words that appear in some of the movies in our dataset.  These words have been *stemmed*, or abbreviated heuristically, in an attempt to make different [inflected](https://en.wikipedia.org/wiki/Inflection) forms of the same base word into the same string.  For example, the column \"manag\" is the sum of proportions of the words \"manage\", \"manager\", \"managed\", and \"managerial\" (and perhaps others) in each movie. This is a common technique used in machine learning and natural language processing.\n",
    "\n",
    "Stemming makes it a little tricky to search for the words you want to use, so we have provided another table that will let you see examples of unstemmed versions of each stemmed word.  Run the code below to load it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Just run this cell.\n",
    "vocab_mapping = Table.read_table('stem.csv')\n",
    "stemmed = np.take(movies.labels, np.arange(3, len(movies.labels)))\n",
    "vocab_table = Table().with_column('Stem', stemmed).join('Stem', vocab_mapping)\n",
    "vocab_table.take(np.arange(1100, 1110))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "only_one = vocab_table.group(0).where('count', are.equal_to(1))\n",
    "only_one"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 1.1.1\n",
    "Assign `stemmed_message` to the stemmed version of the word \"elements\". **(3 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q1_1_1\n",
    "points:\n",
    " - 0\n",
    " - 0\n",
    " - 3\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stemmed_message = vocab_table.where(1, are.equal_to('elements')).column(0).item(0)\n",
    "stemmed_message"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q1_1_1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 1.1.2\n",
    "What stem in the dataset has the most words that are shortened to it? Assign `most_stem` to that stem. **(4 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q1_1_2\n",
    "points:\n",
    " - 0\n",
    " - 4\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_len = vocab_table.apply(len, 1)\n",
    "stem_len = vocab_table.apply(len, 0)\n",
    "new_wlen = vocab_table.with_column('wlen', word_len)\n",
    "most_stem = new_wlen.sort('wlen', descending=True).column(0).item(0)\n",
    "most_stem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q1_1_2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 1.1.3\n",
    "What is the longest word in the dataset whose stem wasn't shortened? Assign that to `longest_uncut`. Break ties alphabetically from Z to A (so if your options are \"cat\" or \"bat\", you should pick \"cat\"). **(6 Points)**\n",
    "\n",
    "*Hint #1:* `vocab_table` has 2 columns: one for stems and one for the unstemmed (normal) word. Find the longest word that wasn't cut at all (same length as stem).\n",
    "\n",
    "*Hint #2:* There is a table function that allows you to compute a function on every element in a column. Check Python Reference if you aren't sure which one.\n",
    "\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q1_1_3\n",
    "points:\n",
    " - 0\n",
    " - 5\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "for_assignment_type": "student"
   },
   "outputs": [],
   "source": [
    "# In our solution, we found it useful to first add columns with\n",
    "# the length of the word and the length of the stem,\n",
    "# and then to add a column with the difference between those lengths.\n",
    "# What will the difference be if the word is not shortened?\n",
    "\n",
    "word_len = vocab_table.apply(len, 1)\n",
    "stem_len = vocab_table.apply(len, 0)\n",
    "new_wlen = vocab_table.with_column('wlen', word_len)\n",
    "tbl_with_lens = new_wlen.with_column('slen', stem_len)\n",
    "tbl_with_dif = tbl_with_lens.with_column('the dif', tbl_with_lens.column(3)-tbl_with_lens.column(2))\n",
    "zero_dif = tbl_with_dif.where(4, are.equal_to(0)).sort(2, descending=True)\n",
    "\n",
    "max_len = np.max(zero_dif.column(2))\n",
    "longest_uncut = zero_dif.where(2, are.equal_to(max_len)).sort(0, descending=True).column(0).item(0)\n",
    "longest_uncut"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q1_1_3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 1.1.4\n",
    "How many stems have only one word that is shortened to them? For example, if the stem \"book\" only maps to the word \"books\" and if the stem \"a\" only maps to the word \"a,\" both should be counted as stems that map only to a single word.\n",
    "\n",
    "Assign `count_single_stems` to the count of stems that map to one word only. **(3 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q1_1_4\n",
    "points:\n",
    " - 0\n",
    " - 3\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "count_single_stems = vocab_table.group(0).where('count', are.equal_to(1)).num_rows\n",
    "count_single_stems"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q1_1_4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2. Exploratory Data Analysis: Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's explore our dataset before trying to build a classifier. To start, we'll look at the relationship between words in proportions. \n",
    "\n",
    "The first association we'll investigate is the association between the proportion of words that are \"outer\" and the proportion of words that are \"space\". \n",
    "\n",
    "As usual, we'll investigate our data visually before performing any numerical analysis.\n",
    "\n",
    "Run the cell below to plot a scatter diagram of space proportions vs outer proportions and to create the `outer_space` table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Just run this cell!\n",
    "outer_space = movies.select(\"outer\", \"space\")\n",
    "outer_space.scatter(\"outer\", \"space\")\n",
    "plots.axis([-0.001, 0.0025, -0.001, 0.005]);\n",
    "plots.xticks(rotation=45);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 1.2.1\n",
    "Looking at that chart it is difficult to see if there is an association. Calculate the correlation coefficient for the potential linear association between proportion of words that are \"outer\" and the proportion of words that are \"space\" for every movie in the dataset, and assign it to `outer_space_r`. **(2 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q1_2_1\n",
    "points:\n",
    " - 0\n",
    " - 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Our solution took multiple lines\n",
    "# these two arrays should make your code cleaner!\n",
    "outer = movies.column(\"outer\")\n",
    "space = movies.column(\"space\")\n",
    "\n",
    "outer_su = (outer - np.mean(outer))/np.std(outer)\n",
    "space_su = (space - np.mean(space))/np.std(space)\n",
    "\n",
    "outer_space_r = np.mean(outer_su*space_su)\n",
    "outer_space_r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q1_2_1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "#### Question 1.2.2\n",
    "Choose two *different* words in the dataset with a magnitude (absolute value) of correlation higher than 0.2 that are not \"outer\" and \"space\" and plot a scatter plot with a line of best fit for them. The code to plot the scatter plot and line of best fit is given for you, you just need to calculate the correct values to `r`, `slope` and `intercept`. **(2 Points)**\n",
    "\n",
    "*Hint: It's easier to think of words with a positive correlation, i.e. words that are often mentioned together*.\n",
    "\n",
    "*Hint 2: Try to think of common phrases or idioms*.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q1_2_2\n",
    "manual: true\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_x = 'san'\n",
    "word_y = 'diego'\n",
    "\n",
    "# These arrays should make your code cleaner!\n",
    "arr_x = movies.column(word_x)\n",
    "arr_y = movies.column(word_y)\n",
    "\n",
    "x_su = (arr_x - np.mean(arr_x))/np.std(arr_x)\n",
    "y_su = (arr_y - np.mean(arr_y))/np.std(arr_y)\n",
    "\n",
    "r = np.mean(x_su*y_su)\n",
    "\n",
    "slope = r*np.std(arr_y)/np.std(arr_x)\n",
    "intercept = np.mean(arr_y)-slope*np.mean(arr_x)\n",
    "\n",
    "# DON'T CHANGE THESE LINES OF CODE\n",
    "movies.scatter(word_x, word_y)\n",
    "max_x = max(movies.column(word_x))\n",
    "plots.title(f\"Correlation: {r}, magnitude greater than .2: {abs(r) >= 0.2}\")\n",
    "plots.plot([0, max_x * 1.3], [intercept, intercept + slope * (max_x*1.3)], color='gold');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "#### Question 1.2.3\n",
    "Imagine that you picked the words \"san\" and \"francisco\" as the two words that you would expect to be correlated because they compose the city name San Francisco. Assign `unexpected` to either the number 1 or the number 2 according to which statement is true regarding the correlation between \"san\" and \"francisco.\" **(2 Points)**\n",
    "\n",
    "1. \"san\" can also preceed other city names like San Diego and San Jose. This might lead to \"san\" appearing in movies without \"francisco,\" and would reduce the correlation between \"san\" and \"francisco.\"\n",
    "2. \"san\" can also preceed other city names like San Diego and San Jose. The fact that \"san\" could appear more often in front of different cities and without \"francisco\" would increase the correlation between \"san\" and \"francisco.\" \n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q1_2_3\n",
    "points:\n",
    " - 0\n",
    " - 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "unexpected = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q1_2_3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3. Splitting the dataset\n",
    "We're going to use our `movies` dataset for two purposes.\n",
    "\n",
    "1. First, we want to *train* movie genre classifiers.\n",
    "2. Second, we want to *test* the performance of our classifiers.\n",
    "\n",
    "Hence, we need two different datasets: *training* and *test*.\n",
    "\n",
    "The purpose of a classifier is to classify unseen data that is similar to the training data. Therefore, we must ensure that there are no movies that appear in both sets. We do so by splitting the dataset randomly. The dataset has already been permuted randomly, so it's easy to split.  We just take the top for training and the rest for test. \n",
    "\n",
    "Run the code below (without changing it) to separate the datasets into two tables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here we have defined the proportion of our data\n",
    "# that we want to designate for training as 17/20ths\n",
    "# of our total dataset.  3/20ths of the data is\n",
    "# reserved for testing.\n",
    "\n",
    "training_proportion = 17/20\n",
    "\n",
    "num_movies = movies.num_rows\n",
    "num_train = int(num_movies * training_proportion)\n",
    "num_test = num_movies - num_train\n",
    "\n",
    "train_movies = movies.take(np.arange(num_train))\n",
    "test_movies = movies.take(np.arange(num_train, num_movies))\n",
    "\n",
    "print(\"Training: \",   train_movies.num_rows, \";\",\n",
    "      \"Test: \",       test_movies.num_rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "#### Question 1.3.1\n",
    "Draw a horizontal bar chart with two bars that show the proportion of Comedy movies in each dataset. Complete the function `comedy_proportion` first; it should help you create the bar chart. **(3 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q1_3_1\n",
    "manual: true\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train1 = train_movies.group('Genre').select('Genre', 'count')\n",
    "train1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "for_assignment_type": "solution"
   },
   "outputs": [],
   "source": [
    "def comedy_proportion(table):\n",
    "    # Return the proportion of movies in a table that have the Comedy genre.\n",
    "    genre_only = table.group('Genre').select('Genre', 'count')\n",
    "    genre_prop = genre_only.with_column('proportion', genre_only.column('count')/table.num_rows)\n",
    "    return genre_prop.barh('Genre', 'proportion')\n",
    "\n",
    "# The staff solution took multiple lines.  Start by creating a table.\n",
    "# If you get stuck, think about what sort of table you need for barh to work\n",
    "comedy_proportion(train_movies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "\n",
    "\n",
    "# Part 2: K-Nearest Neighbors - A Guided Example\n",
    "\n",
    "K-Nearest Neighbors (k-NN) is a classification algorithm.  Given some numerical *attributes* (also called *features*) of an unseen example, it decides whether that example belongs to one or the other of two categories based on its similarity to previously seen examples. Predicting the category of an example is called *labeling*, and the predicted category is also called a *label*.\n",
    "\n",
    "An attribute (feature) we have about each movie is *the proportion of times a particular word appears in the movies*, and the labels are two movie genres: comedy and thriller.  The algorithm requires many previously seen examples for which both the attributes and labels are known: that's the `train_movies` table.\n",
    "\n",
    "To build understanding, we're going to visualize the algorithm instead of just describing it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comedy_proportion(movies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1. Classifying a movie\n",
    "\n",
    "In k-NN, we classify a movie by finding the `k` movies in the *training set* that are most similar according to the features we choose. We call those movies with similar features the *nearest neighbors*.  The k-NN algorithm assigns the movie to the most common category among its `k` nearest neighbors.\n",
    "\n",
    "Let's limit ourselves to just 2 features for now, so we can plot each movie.  The features we will use are the proportions of the words \"water\" and \"feel\" in the movie.  Taking the movie *Monty Python and the Holy Grail* (in the test set), 0.000804074 of its words are \"water\" and 0.0010721 are \"feel\". This movie appears in the test set, so let's imagine that we don't yet know its genre.\n",
    "\n",
    "First, we need to make our notion of similarity more precise.  We will say that the *distance* between two movies is the straight-line distance between them when we plot their features in a scatter diagram. \n",
    "\n",
    "**This distance is called the Euclidean (\"yoo-KLID-ee-un\") distance, whose formula is $\\sqrt{(x_1 - x_2)^2 + (y_1 - y_2)^2}$.**\n",
    "\n",
    "For example, in the movie *Clerks.* (in the training set), 0.00016293 of all the words in the movie are \"water\" and 0.00154786 are \"feel\".  Its distance from *Monty Python and the Holy Grail* on this 2-word feature set is $\\sqrt{(0.000804074 - 0.000162933)^2 + (0.0010721 - 0.00154786)^2} \\approx 0.000798379$.  (If we included more or different features, the distance could be different.)\n",
    "\n",
    "A third movie, *The Avengers* (in the training set), is 0 \"water\" and 0.00103173 \"feel\".\n",
    "\n",
    "The function below creates a plot to display the \"water\" and \"feel\" features of a test movie and some training movies. As you can see in the result, *Monty Python and the Holy Grail* is more similar to \"Clerks.\" than to the *The Avengers* based on these features, which is makes sense as both movies are comedy movies, while *The Avengers* is a thriller."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Just run this cell.\n",
    "def plot_with_two_features(test_movie, training_movies, x_feature, y_feature):\n",
    "    \"\"\"Plot a test movie and training movies using two features.\"\"\"\n",
    "    test_row = row_for_title(test_movie)\n",
    "    distances = Table().with_columns(\n",
    "            x_feature, [test_row.item(x_feature)],\n",
    "            y_feature, [test_row.item(y_feature)],\n",
    "            'Color',   ['unknown'],\n",
    "            'Title',   [test_movie]\n",
    "        )\n",
    "    for movie in training_movies:\n",
    "        row = row_for_title(movie)\n",
    "        distances.append([row.item(x_feature), row.item(y_feature), row.item('Genre'), movie])\n",
    "    distances.scatter(x_feature, y_feature, group='Color', labels='Title', s=30)\n",
    "    \n",
    "training = [\"clerks.\", \"the avengers\"] \n",
    "plot_with_two_features(\"monty python and the holy grail\", training, \"water\", \"feel\")\n",
    "plots.axis([-0.001, 0.0011, -0.004, 0.008]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 2.1.1\n",
    "\n",
    "Compute the Euclidean distance (defined in the section above) between the two movies, *Monty Python and the Holy Grail* and *The Avengers*, using the `water` and `feel` features only.  Assign it the name `one_distance`. **(2 Points)**\n",
    "\n",
    "**Note:** If you have a row, you can use `item` to get a value from a column by its name.  For example, if `r` is a row, then `r.item(\"Genre\")` is the value in column `\"Genre\"` in row `r`.\n",
    "\n",
    "*Hint*: Remember the function `row_for_title`, redefined for you below.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q2_1_1\n",
    "points:\n",
    " - 0\n",
    " - 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "title_index = movies.index_by('Title')\n",
    "python = row_for_title(\"monty python and the holy grail\") \n",
    "avengers = row_for_title(\"the avengers\") \n",
    "\n",
    "one_distance = np.sqrt((python.item('water')-avengers.item('water'))**2 + (python.item('feel')-avengers.item('feel'))**2)\n",
    "one_distance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q2_1_1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we've added a third training movie, *The Silence of the Lambs*. Before, the point closest to *Monty Python and the Holy Grail* was *Clerks.*, a comedy movie. However, now the closest point is *The Silence of the Lambs*, a thriller movie."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training = [\"clerks.\", \"the avengers\", \"the silence of the lambs\"] \n",
    "plot_with_two_features(\"monty python and the holy grail\", training, \"water\", \"feel\") \n",
    "plots.axis([-0.001, 0.0011, -0.004, 0.008]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 2.1.2\n",
    "Complete the function `distance_two_features` that computes the Euclidean distance between any two movies, using two features. The last two lines call your function to show that *Monty Python and the Holy Grail* is closer to *The Silence of the Lambs* than it is to *Clerks*. **(3 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q2_1_2\n",
    "points:\n",
    " - 1\n",
    " - 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "def distance_two_features(title0, title1, x_feature, y_feature):\n",
    "    \"\"\"Compute the distance between two movies with titles title0 and title1\n",
    "    \n",
    "    Only the features named x_feature and y_feature are used when computing the distance.\n",
    "    \"\"\"\n",
    "    row0 = row_for_title(title0)\n",
    "    row1 = row_for_title(title1)\n",
    "    diff_x = row0.item(x_feature)-row1.item(x_feature)\n",
    "    diff_y = row0.item(y_feature)-row1.item(y_feature)\n",
    "    return np.sqrt(diff_x**2 + diff_y**2)\n",
    "\n",
    "for movie in make_array(\"clerks.\", \"the silence of the lambs\"):\n",
    "    movie_distance = distance_two_features(movie, \"monty python and the holy grail\", \"water\", \"feel\")\n",
    "    print(movie, 'distance:\\t', movie_distance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q2_1_2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 2.1.3\n",
    "Define the function `distance_from_python` so that it works as described in its documentation. **(3 Points)**\n",
    "\n",
    "**Note:** Your solution should not use arithmetic operations directly. Instead, it should make use of existing functionality above!\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q2_1_3\n",
    "points:\n",
    " - 0\n",
    " - 3\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "def distance_from_python(title):\n",
    "    \"\"\"The distance between the given movie and \"monty python and the holy grail\", \n",
    "    based on the features \"water\" and \"feel\".\n",
    "    \n",
    "    This function takes a single argument:\n",
    "      title: A string, the name of a movie.\n",
    "    \"\"\"\n",
    "    return distance_two_features(title, \"monty python and the holy grail\", \"water\", \"feel\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q2_1_3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 2.1.4\n",
    "\n",
    "Using the features `\"water\"` and `\"feel\"`, what are the names and genres of the 5 movies in the **training set** closest to *Monty Python and the Holy Grail*?  To answer this question, make a table named `close_movies` containing those 5 movies with columns `\"Title\"`, `\"Genre\"`, `\"water\"`, and `\"feel\"`, as well as a column called `\"distance from python\"` that contains the distance from *Monty Python and the Holy Grail*.  The table should be **sorted in ascending order by `distance from python`**. **(2 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q2_1_4\n",
    "points:\n",
    " - 0\n",
    " - 0\n",
    " - 0\n",
    " - 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "for_assignment_type": "solution"
   },
   "outputs": [],
   "source": [
    "title_arr = train_movies.column('Title')\n",
    "python_distance = make_array()\n",
    "\n",
    "for title in title_arr:\n",
    "    new_distance = distance_from_python(title)\n",
    "    python_distance = np.append(python_distance, new_distance)\n",
    "\n",
    "train_movies1 = train_movies.with_column('distance from python', python_distance)\n",
    "train_movies2 = train_movies1.select('Title','Genre','water','feel','distance from python')\n",
    "\n",
    "# The staff solution took multiple lines.\n",
    "close_movies = train_movies2.sort(4, descending=False).take(0,1,2,3,4)\n",
    "close_movies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q2_1_4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 2.1.5\n",
    "Next, we'll clasify *Monty Python and the Holy Grail* based on the genres of the closest movies. \n",
    "\n",
    "To do so, define the function `most_common` so that it works as described in its documentation below. **(4 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q2_1_5\n",
    "points:\n",
    " - 4\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def most_common(label, table):\n",
    "    \"\"\"The most common element in a column of a table.\n",
    "    \n",
    "    This function takes two arguments:\n",
    "      label: The label of a column, a string.\n",
    "      table: A table.\n",
    "     \n",
    "    It returns the most common value in that column of that table.\n",
    "    In case of a tie, it returns any one of the most common values\n",
    "    \"\"\"\n",
    "    t1 = table.group(label).sort('count', descending=True)\n",
    "    return t1.column(label).item(0)\n",
    "    \n",
    "\n",
    "# Calling most_common on your table of 5 nearest neighbors classifies\n",
    "# \"monty python and the holy grail\" as a thriller movie, 3 votes to 2. \n",
    "most_common('Genre', close_movies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q2_1_5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations are in order -- you've classified your first movie! However, we can see that the classifier doesn't work too well since it categorized *Monty Python and the Holy Grail* as a thriller movie (unless you count the thrilling holy hand grenade scene). Let's see if we can do better!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Checkpoint (due Friday 8/06 by 11:59 PM PT)\n",
    "#### Congratulations, you have reached the checkpoint! Run the following cells and submit to the gradescope assignment corresponding to the checkpoint:  <u>Project 2 Checkpoint.</u> \n",
    "\n",
    "### NOTE: this checkpoint does not represent the halfway point of the project. You are **highly** encouraged to continue on to the next section early.\n",
    "\n",
    "\n",
    "To double-check your work, the cell below will rerun all of the autograder tests for sections 1 and 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint_tests = [\"q1_0\", \"q1_1_1\",\"q1_1_2\",\"q1_1_3\",\"q1_1_4\",\"q1_2_1\",\"q1_2_3\",\"q2_1_1\",\"q2_1_2\",\"q2_1_3\",\"q2_1_4\",\"q2_1_5\",]\n",
    "for test in checkpoint_tests:\n",
    "    display(grader.check(test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submission\n",
    "\n",
    "Make sure you have run all cells in your notebook in order before running the cell below, so that all images/graphs appear in the output. The cell below will generate a zip file for you to submit. **Please save before exporting!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save your notebook first, then run this cell to export your submission.\n",
    "grader.export(pdf=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3: Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we're going to extend our classifier to consider more than two features at a time.\n",
    "\n",
    "Euclidean distance still makes sense with more than two features. For `n` different features, we compute the difference between corresponding feature values for two movies, square each of the `n`  differences, sum up the resulting numbers, and take the square root of the sum."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 3.0\n",
    "Write a function called `distance` to compute the Euclidean distance between two **arrays** of **numerical** features (e.g. arrays of the proportions of times that different words appear). The function should be able to calculate the Euclidean distance between two arrays of arbitrary (but equal) length. **(2 Points)**\n",
    "\n",
    "Next, use the function you just defined to compute the distance between the first and second movie in the training set *using all of the features*.  (Remember that the first five columns of your tables are not features.)\n",
    "\n",
    "**Note:** To convert rows to arrays, use `np.array`. For example, if `t` was a table, `np.array(t.row(0))` converts row 0 of `t` into an array.\n",
    "\n",
    "**Note:** If you're working offline: Depending on the versions of your packages, you may need to convert rows to arrays using the following instead: `np.array(list(t.row(0))`\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3_0\n",
    "points:\n",
    " - 0\n",
    " - 0\n",
    " - 0\n",
    " - 0.5\n",
    " - 0.5\n",
    " - 0.5\n",
    " - 0.5\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_arr = make_array(1,3,8,9)\n",
    "my_sum = np.sum(my_arr)\n",
    "my_sum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "def distance(features_array1, features_array2):\n",
    "    \"\"\"The Euclidean distance between two arrays of feature values.\"\"\"\n",
    "    n1 = len(features_array1)\n",
    "    square_diff = make_array()\n",
    "    for i in np.arange(n1):\n",
    "        the_diff = features_array1.item(i)-features_array2.item(i)\n",
    "        new_square = the_diff**2\n",
    "        square_diff = np.append(square_diff, new_square)\n",
    "    return np.sqrt(np.sum(square_diff))\n",
    "\n",
    "new_train = train_movies.drop(0,1,2,3,4)\n",
    "distance_first_to_second = distance(np.array(new_train.row(0)), np.array(new_train.row(1)) )\n",
    "distance_first_to_second"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q3_0\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1. Creating your own feature set\n",
    "\n",
    "Unfortunately, using all of the features has some downsides.  One clear downside is the lack of *computational efficiency* -- computing Euclidean distances just takes a long time when we have lots of features.  You might have noticed that in the last question!\n",
    "\n",
    "So we're going to select just 20.  We'd like to choose features that are very *discriminative*. That is, features which lead us to correctly classify as much of the test set as possible.  This process of choosing features that will make a classifier work well is sometimes called *feature selection*, or, more broadly, *feature engineering*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 3.1.1\n",
    "In this question, we will help you get started on selecting more effective features for distinguishing comedy from thriller movies. The plot below (generated for you) shows the average number of times each word occurs in a comedy movie on the horizontal axis and the average number of times it occurs in an thriller movie on the vertical axis. \n",
    "\n",
    "\n",
    "*Note: The line graphed is the line of best fit, NOT a y=x*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](word_plot.png \"Title\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "The following questions ask you to interpret the plot above. For each question, select one of the following choices and assign its number to the provided name.\n",
    "    1. The word is common in both comedy and thriller movies \n",
    "    2. The word is uncommon in comedy movies and common in thriller movies\n",
    "    3. The word is common in comedy movies and uncommon in thriller movies\n",
    "    4. The word is uncommon in both comedy and thriller movies\n",
    "    5. It is not possible to say from the plot \n",
    "    \n",
    "What properties does a word in the bottom left corner of the plot have? Your answer should be a single integer from 1 to 5, corresponding to the correct statement from the choices above. **(2 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3_1_1\n",
    "points:\n",
    " - 0\n",
    " - 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "bottom_left = 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q3_1_1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "**Question 3.1.2**\n",
    "\n",
    "What properties does a word in the bottom right corner have? **(2 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3_1_2\n",
    "points:\n",
    " - 0\n",
    " - 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "bottom_right = 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q3_1_2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "**Question 3.1.3**\n",
    "\n",
    "What properties does a word in the top right corner have? **(2 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3_1_3\n",
    "points:\n",
    " - 0\n",
    " - 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "top_right = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q3_1_3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "**Question 3.1.4**\n",
    "\n",
    "What properties does a word in the top left corner have? **(2 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3_1_4\n",
    "points:\n",
    " - 0\n",
    " - 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "top_left = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q3_1_4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "**Question 3.1.5**\n",
    "\n",
    "If we see a movie with a lot of words that are common for comedy movies but uncommon for thriller movies, what would be a reasonable guess about the genre of the movie? Assign `movie_genre` to the number corresponding to your answer:\n",
    "    1. It is a thriller movie.\n",
    "    2. It is a comedy movie.\n",
    "\n",
    "**(2 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3_1_5\n",
    "points:\n",
    " - 0\n",
    " - 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "movie_genre_guess = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q3_1_5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 3.1.6\n",
    "Using the plot above, make an array of at least 10 common words that you think might let you **distinguish** between comedy and thriller movies. Make sure to choose words that are **frequent enough** that every movie contains at least one of them. Don't just choose the most frequent words, though--you can do much better. **(2 Points)**\n",
    "\n",
    "You might want to come back to this question later to improve your list, once you've seen how to evaluate your classifier.  \n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3_1_6\n",
    "points:\n",
    " - 0\n",
    " - 0\n",
    " - 1\n",
    " - 1\n",
    " - 0\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "# Set my_features to an array of at least 10 features (strings that are column labels)\n",
    "\n",
    "my_features = make_array('a','and','my', 'well', 'oh', 'he', 'there','where','kill', 'dead')\n",
    "\n",
    "# Select the 10 features of interest from both the train and test sets\n",
    "train_my_features = train_movies.select(my_features)\n",
    "test_my_features = test_movies.select(my_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q3_1_6\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This test makes sure that you have chosen words such that at least one appears in each movie. If you can't find words that satisfy this test just through intuition, try writing code to print out the titles of movies that do not contain any words from your list, then look at the words they do contain."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "#### Question 3.1.7\n",
    "In two sentences or less, describe how you selected your features. **(4 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3_1_7\n",
    "manual: True\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I choose the words which far enough from the red dash line. Particularly, I choose 5 words common in comedy but uncommon in thriller, and 5 other words common in thriller but uncommon in comedy. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "\n",
    "\n",
    "Next, let's classify the first movie from our test set using these features.  You can examine the movie by running the cells below. Do you think it will be classified correctly?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Movie:\")\n",
    "test_movies.take(0).select('Title', 'Genre').show()\n",
    "print(\"Features:\")\n",
    "test_my_features.take(0).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, we want to look for the movies in the training set that are most like our test movie.  We will calculate the Euclidean distances from the test movie (using `my_features`) to all movies in the training set.  You could do this with a `for` loop, but to make it computationally faster, we have provided a function, `fast_distances`, to do this for you.  Read its documentation to make sure you understand what it does.  (You don't need to understand the code in its body unless you want to.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Just run this cell to define fast_distances.\n",
    "\n",
    "def fast_distances(test_row, train_table):\n",
    "    \"\"\"Return an array of the distances between test_row and each row in train_table.\n",
    "\n",
    "    Takes 2 arguments:\n",
    "      test_row: A row of a table containing features of one\n",
    "        test movie (e.g., test_my_features.row(0)).\n",
    "      train_table: A table of features (for example, the whole\n",
    "        table train_my_features).\"\"\"\n",
    "    assert train_table.num_columns < 50, \"Make sure you're not using all the features of the movies table.\"\n",
    "    assert type(test_row) != datascience.tables.Table, \"Make sure you are passing in a row object to fast_distances.\"\n",
    "    assert len(test_row) == len(train_table.row(0)), \"Make sure the length of test row is the same as the length of a row in train_table.\"\n",
    "    counts_matrix = np.asmatrix(train_table.columns).transpose()\n",
    "    diff = np.tile(np.array(list(test_row)), [counts_matrix.shape[0], 1]) - counts_matrix\n",
    "    np.random.seed(0) # For tie breaking purposes\n",
    "    distances = np.squeeze(np.asarray(np.sqrt(np.square(diff).sum(1))))\n",
    "    eps = np.random.uniform(size=distances.shape)*1e-10 #Noise for tie break\n",
    "    distances = distances + eps\n",
    "    return distances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 3.1.8\n",
    "Use the `fast_distances` function provided above to compute the distance from the first movie in the test set to all the movies in the training set, **using your set of features**.  Make a new table called `genre_and_distances` with one row for each movie in the training set and two columns:\n",
    "* The `\"Genre\"` of the training movie\n",
    "* The `\"Distance\"` from the first movie in the test set \n",
    "\n",
    "Ensure that `genre_and_distances` is **sorted in ascending order by distance to the first test movie**. **(3 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3_1_8\n",
    "points:\n",
    " - 0\n",
    " - 1\n",
    " - 1\n",
    " - 1\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "for_assignment_type": "solution"
   },
   "outputs": [],
   "source": [
    "# The staff solution took multiple lines of code.\n",
    "firt_movie_distances = fast_distances(test_my_features.row(0), train_my_features)\n",
    "genre_and_distances1 = train_movies.with_column('Distance', firt_movie_distances).select('Genre', 'Distance')\n",
    "genre_and_distances = genre_and_distances1.sort('Distance', descending=False)\n",
    "genre_and_distances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q3_1_8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 3.1.9\n",
    "Now compute the 7-nearest neighbors classification of the first movie in the test set.  That is, decide on its genre by finding the most common genre among its 7 nearest neighbors in the training set, according to the distances you've calculated.  Then check whether your classifier chose the right genre.  (Depending on the features you chose, your classifier might not get this movie right, and that's okay.) **(2 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3_1_9\n",
    "points:\n",
    " - 1\n",
    " - 1\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "# Set my_assigned_genre to the most common genre among these.\n",
    "seven_nearest = genre_and_distances.take(np.arange(7)).group(0).sort('count', descending=True)\n",
    "my_assigned_genre = seven_nearest.column('Genre').item(0)\n",
    "\n",
    "# Set my_assigned_genre_was_correct to True if my_assigned_genre\n",
    "# matches the actual genre of the first movie in the test set, False otherwise.\n",
    "my_assigned_genre_was_correct = True\n",
    "\n",
    "print(\"The assigned genre, {}, was{}correct.\".format(my_assigned_genre, \" \" if my_assigned_genre_was_correct else \" not \"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q3_1_9\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2. A classifier function\n",
    "\n",
    "Now we can write a single function that encapsulates the whole process of classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 3.2.1\n",
    "Write a function called `classify`.  It should take the following four arguments:\n",
    "* A row of features for a movie to classify (e.g., `test_my_features.row(0)`).\n",
    "* A table with a column for each feature (e.g., `train_my_features`).\n",
    "* An array of classes (e.g. the labels \"comedy\" or \"thriller\") that has as many items as the previous table has rows, and in the same order.\n",
    "* `k`, the number of neighbors to use in classification.\n",
    "\n",
    "It should return the class a `k`-nearest neighbor classifier picks for the given row of features (the string `'comedy'` or the string `'thriller'`). **(7 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3_2_1\n",
    "points:\n",
    " - 3\n",
    " - 3\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "def classify(test_row, train_rows, train_labels, k):\n",
    "    \"\"\"Return the most common class among k nearest neigbors to test_row.\"\"\"\n",
    "    distances = fast_distances(test_row, train_rows)\n",
    "    genre_and_distances1 = train_movies.with_column('Distance', distances).select('Genre', 'Distance')\n",
    "    genre_and_distances = genre_and_distances1.sort('Distance', descending=False)\n",
    "    k_nearest = genre_and_distances.take(np.arange(k)).group(0).sort('count', descending=True)\n",
    "    the_assigned_genre = k_nearest.column('Genre').item(0)\n",
    "    return the_assigned_genre"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q3_2_1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 3.2.2\n",
    "\n",
    "Assign `godzilla_genre` to the genre predicted by your classifier for the movie \"godzilla\" in the test set, using **15 neighbors** and using your 10 features. **(4 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3_2_2\n",
    "points:\n",
    " - 4\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "# The staff solution first defined a row called godzilla_features.\n",
    "godzilla_row = test_movies.where('Title', 'godzilla')\n",
    "godzilla_features = godzilla_row.select(my_features)\n",
    "godzilla_genre = classify(godzilla_features.row(0), train_my_features, train_my_features.labels, 15)\n",
    "godzilla_genre"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q3_2_2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, when we evaluate our classifier, it will be useful to have a classification function that is specialized to use a fixed training set and a fixed value of `k`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### Question 3.2.3\n",
    "Create a classification function that takes as its argument a row containing your 10 features and classifies that row using the 15-nearest neighbors algorithm with `train_my_features` as its training set. **(4 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3_2_3\n",
    "points:\n",
    " - 4\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "def classify_feature_row(row):\n",
    "    row_genre = classify(row, train_my_features, train_my_features.labels, 15)\n",
    "    return row_genre\n",
    "\n",
    "# When you're done, this should produce 'Thriller' or 'Comedy'.\n",
    "classify_feature_row(test_my_features.row(0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q3_2_3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3. Evaluating your classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "Now that it's easy to use the classifier, let's see how accurate it is on the whole test set.\n",
    "\n",
    "**Question 3.3.1.** Use `classify_feature_row` and `apply` to classify every movie in the test set.  Assign these guesses as an array to `test_guesses`.  Then, compute the proportion of correct classifications. **(4 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3_3_1\n",
    "points:\n",
    " - 0\n",
    " - 4\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "test_guesses = test_my_features.apply(classify_feature_row)\n",
    "test_genres = test_movies.column('Genre')\n",
    "proportion_correct = np.count_nonzero(test_guesses==test_genres)/len(test_genres)\n",
    "proportion_correct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q3_3_1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "**Question 3.3.2.** An important part of evaluating your classifiers is figuring out where they make mistakes. Assign the name `test_movie_correctness` to a table with three columns, `'Title'`, `'Genre'`, and `'Was correct'`. **(4 Points)** \n",
    "\n",
    "- The `'Genre'` column should contain the original genres, not the ones you predicted. \n",
    "- The `'Was correct'` column should contain `True` or `False` depending on whether or not the movie was classified correctly.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3_3_2\n",
    "points:\n",
    " - 0\n",
    " - 0\n",
    " - 4\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "for_assignment_type": "student",
    "manual_problem_id": "test_movie_correctness"
   },
   "outputs": [],
   "source": [
    "# Feel free to use multiple lines of code\n",
    "# but make sure to assign test_movie_correctness to the proper table!\n",
    "title_and_genre = test_movies.select('Title', 'Genre')\n",
    "binary_test = test_guesses==test_genres\n",
    "test_movie_correctness = title_and_genre.with_column('Was correct', binary_test)\n",
    "test_movie_correctness.sort('Was correct', descending = False).show(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q3_3_2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "**Question 3.3.3.** Do you see a pattern in the types of movies your classifier misclassifies? In two sentences or less, describe any patterns you see in the results or any other interesting findings from the table above. If you need some help, try looking up the movies that your classifier got wrong on Wikipedia. **(4 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3_3_3\n",
    "manual: true\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# I count what is the proportion of False guess in each Genre\n",
    "test_movie_correctness.pivot('Was correct','Genre' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The classifier made more mistake in the comedy genre rather than the thriller. \n",
    "From wikipedia I found that most of the missclassified movies are old movies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "\n",
    "\n",
    "At this point, you've gone through one cycle of classifier design.  Let's summarize the steps:\n",
    "1. From available data, select test and training sets.\n",
    "2. Choose an algorithm you're going to use for classification.\n",
    "3. Identify some features.\n",
    "4. Define a classifier function using your features and the training set.\n",
    "5. Evaluate its performance (the proportion of correct classifications) on the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4: Explorations\n",
    "Now that you know how to evaluate a classifier, it's time to build a better one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "**Question 4.1.** Develop a classifier with better test-set accuracy than `classify_feature_row`. Your new function should have the same arguments as `classify_feature_row` and return a classification.  Name it `another_classifier`. Then, output your accuracy using code from earlier to compare the new classifier to your old one. \n",
    "\n",
    "You can use more or different features, or you can try different values of `k`. (Of course, you still have to use `train_movies` as your training set!) Move on to the next question once you have built a classifier with better accuracy.\n",
    "\n",
    "**Make sure you don't reassign any previously used variables here**, such as `proportion_correct` from the previous question. **(2 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q4_1\n",
    "points:\n",
    " - 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "# To start you off, here's a list of possibly-useful features\n",
    "# Feel free to add or change this array to improve your classifier\n",
    "new_features = make_array(\"laugh\", \"marri\", \"dead\", \"heart\", \"cop\", \"oh\", \"kill\", \"love\")\n",
    "\n",
    "train_new = train_movies.select(new_features)\n",
    "test_new = test_movies.select(new_features)\n",
    "\n",
    "def another_classifier(row):\n",
    "    row_genre = classify(row, train_new, train_new.labels, 20)\n",
    "    return row_genre\n",
    "\n",
    "new_guesses = test_new.apply(another_classifier)\n",
    "test_genres = test_movies.column('Genre')\n",
    "new_proportion_correct = np.count_nonzero(new_guesses==test_genres)/len(test_genres)\n",
    "new_proportion_correct\n",
    "title_and_genre = test_movies.select('Title', 'Genre')\n",
    "new_binary_test = new_guesses==test_genres\n",
    "new_movie_correctness = title_and_genre.with_column('Was correct', new_binary_test)\n",
    "new_movie_correctness.sort('Was correct', descending = True).show(10)\n",
    "new_movie_correctness.pivot('Was correct','Genre' )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_proportion_correct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q4_1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "**Question 4.2.** Do you see a pattern in the mistakes your new classifier makes? What about in the improvement from your first classifier to the second one? Describe in two sentences or less. **(2 Points)**\n",
    "\n",
    "*Hint:* You may not be able to see a pattern.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q4_2\n",
    "manual: true\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I cannot see the pattern in my new classifier mistakes.\n",
    "The proportion of correct guess raising from 0.71 to 0.76."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "**Question 4.3.** Briefly describe what you tried to improve your classifier. **(3 Points)**\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q4_3\n",
    "manual: true\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I tried to change the feature word using right bottom words and left top words of the scatter plot, to make a better distinguisher.\n",
    "I also tried to raise the k from 15 to 20."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "\n",
    "\n",
    "# Part 5: Other Classification Methods (OPTIONAL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: Everything below is **OPTIONAL**. Please only work on this part after you have finished and submitted the project. If you create new cells below, do NOT reassign variables defined in previous parts of the project.\n",
    "\n",
    "Now that you've finished your k-NN classifier, you might be wondering what else you could do to improve your accuracy on the test set. Classification is one of many machine learning tasks, and there are plenty of other classification algorithms! If you feel so inclined, we encourage you to try any methods you feel might help improve your classifier. \n",
    "\n",
    "We've compiled a list of blog posts with some more information about classification and machine learning. Create as many cells as you'd like below--you can use them to import new modules or implement new algorithms. \n",
    "\n",
    "Blog posts: \n",
    "\n",
    "* [Classification algorithms/methods](https://medium.com/@sifium/machine-learning-types-of-classification-9497bd4f2e14)\n",
    "* [Train/test split and cross-validation](https://towardsdatascience.com/train-test-split-and-cross-validation-in-python-80b61beca4b6)\n",
    "* [More information about k-nearest neighbors](https://medium.com/@adi.bronshtein/a-quick-introduction-to-k-nearest-neighbors-algorithm-62214cea29c7)\n",
    "* [Overfitting](https://elitedatascience.com/overfitting-in-machine-learning)\n",
    "\n",
    "In future data science classes, such as Data Science 100, you'll learn about some about some of the algorithms in the blog posts above, including logistic regression. You'll also learn more about overfitting, cross-validation, and approaches to different kinds of machine learning problems.\n",
    "\n",
    "There's a lot to think about, so we encourage you to find more information on your own!\n",
    "\n",
    "Modules to think about using:\n",
    "\n",
    "* [Scikit-learn tutorial](http://scikit-learn.org/stable/tutorial/basic/tutorial.html)\n",
    "* [TensorFlow information](https://www.tensorflow.org/tutorials/)\n",
    "\n",
    "...and many more!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Congrats, you have finished Project 2!\n",
    "\n",
    "Using your statistics and machine learning skills you have:\n",
    "- Investigated a movie script dataset\n",
    "- Identified word associations\n",
    "- Built a machine learning model to classify movies by their scripts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "---\n",
    "\n",
    "To double-check your work, the cell below will rerun all of the autograder tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check_all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "## Submission\n",
    "\n",
    "Make sure you have run all cells in your notebook in order before running the cell below, so that all images/graphs appear in the output. The cell below will generate a zip file for you to submit. **Please save before exporting!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "# Save your notebook first, then run this cell to export your submission.\n",
    "grader.export(pdf=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}