2. ElasticSearch-Tricks-and-Tips.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ElasticSearch Process and Tips (Indexing, Analyzing, Querying)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Things About Your Local ES Installation\n",
    "\n",
    "There is a config directory where you have ES installed.  This is where you would edit the config yml file, or add files for stopwords or synonyms.  (This is important information you will need, so make sure you know where your install lives.)\n",
    "\n",
    "To start up: Run elasticsearch (bin/elasticsearch), then run bin/kibana inside the kibana directory.  This establishes the plugins you might want, like Sense.  See below re Sense."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Sense Plugin to Kibana and ES is Useful:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install Kibana, and then add plugins (they become \"apps\" under the tab in the Kibana toolbar. (The little grid icon to the right of the \"Sense\" name in the pic below.)\n",
    "\n",
    "Sense plugin: https://github.com/bleskes/sense  (Note, it used to be part of Marvel.)\n",
    "Install directions: https://www.elastic.co/guide/en/sense/current/installing.html\n",
    "\n",
    "**Example: in the ES docs**:\n",
    "\n",
    "    curl 'localhost:9200/_cat/indices?v'\n",
    "\n",
    "**In Sense**:\n",
    "\n",
    "    GET /_cat/indices?v\n",
    "\n",
    "<img src=\"images/sense_ui.png\">\n",
    "\n",
    "In the github repo, there is a text file of Sense queries that will work after you index your data using the steps below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Python...  Slightly different API:\n",
    "\n",
    "We will be loading the data from the stored dataframe, and then indexing the data in ES.  We can do queries against it from Python or using the Sense plugin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from elasticsearch import Elasticsearch, client\n",
    "\n",
    "#es = Elasticsearch(hosts=[{'host': 'elasticsearch.aws.blahblah.com', 'port': '9200'}])\n",
    "localEs = Elasticsearch()\n",
    "localclient = client.IndicesClient(localEs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyzers, Defaults, and Preventing Analysis\n",
    "\n",
    "Analysis is the process of chopping up your text and storing it in a form that can be searched efficiently against.\n",
    "\n",
    "#### Read this:\n",
    "\n",
    "https://www.elastic.co/guide/en/elasticsearch/guide/current/custom-analyzers.html\n",
    "\n",
    "An Analyzer, is in order, a sequence of optional\n",
    "* character filters\n",
    "* tokenizers\n",
    "* token filters\n",
    "\n",
    "To prevent analysis, you can specify \"not_analyzed\" on the index itself.  The Interwebs also suggest \"keyword\" as the analyzer for a field, but some folks claim it does some simple analyis.\n",
    "\n",
    "The default analyzer (if unspecified!) for string fields is \"standard.\"  In a custom analyzer, it would be defined:\n",
    "\n",
    "    {\n",
    "        \"type\":      \"custom\",\n",
    "        \"tokenizer\": \"standard\",\n",
    "        \"filter\":  [ \"lowercase\", \"stop\" ]\n",
    "    }\n",
    "    \n",
    "More on default analysis from the docs (https://www.elastic.co/guide/en/elasticsearch/guide/current/_controlling_analysis.html):\n",
    "\n",
    ">While we can specify an analyzer at the field level, how do we determine which analyzer is used for a field if none is specified at the field level?\n",
    ">\n",
    ">Analyzers can be specified at several levels. Elasticsearch works through each level until it finds an analyzer that it can use. At index time, the order is as follows:\n",
    ">\n",
    ">1. The analyzer defined in the field mapping, else\n",
    ">2. The analyzer named default in the index settings, which defaults to\n",
    ">3. The standard analyzer\n",
    ">\n",
    ">...At search time, the sequence is slightly different:...\n",
    ">\n",
    ">1. The analyzer defined in the query itself, else\n",
    ">2. The search_analyzer defined in the field mapping, else\n",
    ">3. The analyzer defined in the field mapping, else\n",
    ">4. The analyzer named default_search in the index settings, which defaults to\n",
    ">5. The analyzer named default in the index settings, which defaults to\n",
    ">6. The standard analyzer\n",
    "    \n",
    "#### We can inspect analysis with the \"analyze\" function (or \"_analyze\" in the curl style)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'acknowledged': True}"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#localEs.indices.delete(index='my_index')\n",
    "localEs.indices.create(index='my_index')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'tokens': [{u'end_offset': 2,\n",
       "   u'position': 0,\n",
       "   u'start_offset': 0,\n",
       "   u'token': u'my',\n",
       "   u'type': u'<ALPHANUM>'},\n",
       "  {u'end_offset': 8,\n",
       "   u'position': 1,\n",
       "   u'start_offset': 3,\n",
       "   u'token': u'kitty',\n",
       "   u'type': u'<ALPHANUM>'},\n",
       "  {u'end_offset': 12,\n",
       "   u'position': 2,\n",
       "   u'start_offset': 9,\n",
       "   u'token': u'cat',\n",
       "   u'type': u'<ALPHANUM>'},\n",
       "  {u'end_offset': 15,\n",
       "   u'position': 3,\n",
       "   u'start_offset': 13,\n",
       "   u'token': u'is',\n",
       "   u'type': u'<ALPHANUM>'},\n",
       "  {u'end_offset': 24,\n",
       "   u'position': 4,\n",
       "   u'start_offset': 16,\n",
       "   u'token': u'adorable',\n",
       "   u'type': u'<ALPHANUM>'}]}"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this is the default analyzer ES will use if you don't specify one! Specify one!\n",
    "\n",
    "localclient.analyze(index='my_index', analyzer='standard',\n",
    "                                        text='My kitty-cat is adorable.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# A utility to make analysis results easier to read:\n",
    "\n",
    "def get_analyzer_tokens(result):\n",
    "    ''' Utility to combine tokens in an analyzer result. '''\n",
    "    tokens = result[u'tokens']\n",
    "    print tokens\n",
    "    return ' '.join([token['token'] for token in tokens])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{u'end_offset': 2, u'token': u'my', u'type': u'<ALPHANUM>', u'start_offset': 0, u'position': 0}, {u'end_offset': 8, u'token': u'kitty', u'type': u'<ALPHANUM>', u'start_offset': 3, u'position': 1}, {u'end_offset': 14, u'token': u\"cat's\", u'type': u'<ALPHANUM>', u'start_offset': 9, u'position': 2}, {u'end_offset': 16, u'token': u'a', u'type': u'<ALPHANUM>', u'start_offset': 15, u'position': 3}, {u'end_offset': 21, u'token': u'pain', u'type': u'<ALPHANUM>', u'start_offset': 17, u'position': 4}, {u'end_offset': 24, u'token': u'in', u'type': u'<ALPHANUM>', u'start_offset': 22, u'position': 5}, {u'end_offset': 28, u'token': u'the', u'type': u'<ALPHANUM>', u'start_offset': 25, u'position': 6}, {u'end_offset': 33, u'token': u'neck', u'type': u'<ALPHANUM>', u'start_offset': 29, u'position': 7}]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "u\"my kitty cat's a pain in the neck\""
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_analyzer_tokens(localclient.analyze(index='my_index',\n",
    "                                        analyzer=\"standard\",\n",
    "                                        text='My kitty-cat\\'s a pain in the neck.'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NB: Prevent analysis with \"keyword\" analyzer, or set the index itself as \"not_analyzed\" in settings.**\n",
    "\n",
    "But if you do this, you need to match on EXACT field contents to search for it.  Best to keep an analyzed copy too, if it's meant to be english searchable text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{u'end_offset': 34, u'token': u\"My kitty-cat's a pain in the neck.\", u'type': u'word', u'start_offset': 0, u'position': 0}]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "u\"My kitty-cat's a pain in the neck.\""
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_analyzer_tokens(localclient.analyze(index='my_index',\n",
    "                                        analyzer='keyword',\n",
    "                                        text='My kitty-cat\\'s a pain in the neck.'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Built-In ES \"English\" Analyzer:\n",
    "### A useful analyzer for text is the built-in English one, which does this, approximately:\n",
    "\n",
    "https://www.elastic.co/guide/en/elasticsearch/guide/current/language-intro.html\n",
    "\n",
    "See: \n",
    "https://simpsora.wordpress.com/2014/05/02/customizing-elasticsearch-english-analyzer/\n",
    "\n",
    ">Tokenizer: Standard tokenizer\n",
    "\n",
    ">TokenFilters:\n",
    ">* Standard token filter\n",
    ">* English possessive filter, which removes trailing 's from words\n",
    ">* Lowercase token filter\n",
    ">* Stop token filter\n",
    ">* Keyword marker filter, which protects certain tokens from modification by stemmers\n",
    ">* Porter stemmer filter, which reduces words down to a base form (“stem”)\n",
    "\n",
    "\n",
    "These are the stop-words defined for English:\n",
    "\n",
    "    a, an, and, are, as, at, be, but, by, for, if, in, into, is, it,\n",
    "    no, not, of, on, or, such, that, the, their, then, there, these,\n",
    "    they, this, to, was, will, with\n",
    "    \n",
    "If you want to customize you can create a new filter yourself or use a file in your config directory for ES.    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{u'end_offset': 2, u'token': u'my', u'type': u'<ALPHANUM>', u'start_offset': 0, u'position': 0}, {u'end_offset': 8, u'token': u'kitti', u'type': u'<ALPHANUM>', u'start_offset': 3, u'position': 1}, {u'end_offset': 14, u'token': u'cat', u'type': u'<ALPHANUM>', u'start_offset': 9, u'position': 2}, {u'end_offset': 21, u'token': u'pain', u'type': u'<ALPHANUM>', u'start_offset': 17, u'position': 4}, {u'end_offset': 33, u'token': u'neck', u'type': u'<ALPHANUM>', u'start_offset': 29, u'position': 7}]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "u'my kitti cat pain neck'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Try it on some text and see...\n",
    "get_analyzer_tokens(localclient.analyze(index='my_index', analyzer='english',\n",
    "                                        text='My kitty-cat\\'s a pain in the neck.'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you wanted to customize the 'english' analyzer with your own special rules (extra stopwords etc), see here: https://www.elastic.co/guide/en/elasticsearch/guide/current/configuring-language-analyzers.html\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzers and Custom Analyzers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You want to make sure you are explicit about types in your data, so that ES doesn't just guess and maybe get it wrong. Also, this is how you set explicit analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Create a setting for the index:\n",
    "\n",
    "    PUT /my_index\n",
    "    {\n",
    "        \"settings\": {\n",
    "            \"analysis\": {\n",
    "                \"char_filter\": { ... custom character filters ... },\n",
    "                \"tokenizer\":   { ...    custom tokenizers     ... },\n",
    "                \"filter\":      { ...   custom token filters   ... },\n",
    "                \"analyzer\":    { ...    custom analyzers referring to the definitions above ... }\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "    \n",
    "For example - this saves a bunch of analysis components into an analyzer called 'my_analyzer':\n",
    "\n",
    "    PUT /my_index\n",
    "    {\n",
    "        \"settings\": {\n",
    "            \"analysis\": {\n",
    "                \"char_filter\": {\n",
    "                    \"&_to_and\": {\n",
    "                        \"type\":       \"mapping\",\n",
    "                        \"mappings\": [ \"&=> and \"]\n",
    "                }},\n",
    "                \"filter\": {\n",
    "                    \"my_stopwords\": {\n",
    "                        \"type\":       \"stop\",\n",
    "                        \"stopwords\": [ \"the\", \"a\" ]\n",
    "                }},\n",
    "                \"analyzer\": {\n",
    "                    \"my_analyzer\": {\n",
    "                        \"type\":         \"custom\",\n",
    "                        \"char_filter\":  [ \"html_strip\", \"&_to_and\" ],\n",
    "                        \"tokenizer\":    \"standard\",\n",
    "                        \"filter\":       [ \"lowercase\", \"my_stopwords\" ]\n",
    "                }}\n",
    "    }}}\n",
    "    \n",
    " Then you **use it**, by referring to it in a mapping for a document in this index:\n",
    " \n",
    "     PUT /my_index/_mapping/my_type\n",
    "    {\n",
    "        \"properties\": {\n",
    "            \"title\": {\n",
    "                \"type\":      \"string\",\n",
    "                \"analyzer\":  \"my_analyzer\"\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "    \n",
    "#### Remember: If you don't assign it to a field in a mapping, you aren't using it.\n",
    "\n",
    "In Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "MY_SETTINGS = {\n",
    "    \"settings\": {\n",
    "        \"analysis\": {\n",
    "            \"char_filter\": {\n",
    "                \"&_to_and\": {\n",
    "                    \"type\":       \"mapping\",\n",
    "                    \"mappings\": [ \"&=> and \"]\n",
    "            }},\n",
    "            \"filter\": {\n",
    "                \"my_stopwords\": {\n",
    "                    \"type\":       \"stop\",\n",
    "                    \"stopwords\": [ \"the\", \"a\" ]\n",
    "            }},\n",
    "            \"analyzer\": {\n",
    "                \"my_analyzer\": {\n",
    "                    \"type\":         \"custom\",\n",
    "                    \"char_filter\":  [ \"html_strip\", \"&_to_and\" ],\n",
    "                    \"tokenizer\":    \"standard\",\n",
    "                    \"filter\":       [ \"lowercase\", \"my_stopwords\" ]\n",
    "            }}\n",
    "        }}\n",
    "}\n",
    "\n",
    "MAPPING = {\n",
    "    \"my_doc_type\": {\n",
    "        \"properties\": {\n",
    "            \"title\": {\n",
    "                \"type\":      \"string\",\n",
    "                \"analyzer\":  \"my_analyzer\"\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stopwords Note\n",
    "\n",
    "The default list of stopwords is indicated thusly:\n",
    "    \n",
    ">\"stopwords\": \"\\_english\\_\"\n",
    "\n",
    "So you can specify both that filter and a custom stopwords list, if you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'acknowledged': True}"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "#localEs.indices.delete(index='my_index')  # uncomment is this index exists and you need to remake it\n",
    "localEs.indices.create(index='my_index', body=json.dumps(MY_SETTINGS))\n",
    "localEs.indices.put_mapping(index='my_index', doc_type=\"my_doc_type\", body=json.dumps(MAPPING))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'my_index': {u'mappings': {u'my_doc_type': {u'properties': {u'title': {u'analyzer': u'my_analyzer',\n",
       "      u'type': u'string'}}}}}}"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check that your mapping looks right!\n",
    "\n",
    "localclient.get_mapping(index='my_index')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "res = localclient.analyze(index='my_index', analyzer='my_analyzer',\\\n",
    "                          text=\"<p>This is the title & a Capitalized Word!</p>\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{u'end_offset': 7, u'token': u'this', u'type': u'<ALPHANUM>', u'start_offset': 3, u'position': 0}, {u'end_offset': 10, u'token': u'is', u'type': u'<ALPHANUM>', u'start_offset': 8, u'position': 1}, {u'end_offset': 20, u'token': u'title', u'type': u'<ALPHANUM>', u'start_offset': 15, u'position': 3}, {u'end_offset': 22, u'token': u'and', u'type': u'<ALPHANUM>', u'start_offset': 21, u'position': 4}, {u'end_offset': 36, u'token': u'capitalized', u'type': u'<ALPHANUM>', u'start_offset': 25, u'position': 6}, {u'end_offset': 41, u'token': u'word', u'type': u'<ALPHANUM>', u'start_offset': 37, u'position': 7}]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "u'this is title and capitalized word'"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_analyzer_tokens(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenizers vs. Analyzers - Be Careful.\n",
    "\n",
    "Some of the names in ES are confusing.  There is a **\"standard\" analyzer** and a **\"standard\" tokenizer**. https://www.elastic.co/guide/en/elasticsearch/guide/current/standard-tokenizer.html#standard-tokenizer\n",
    "\n",
    "Check them out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{u'end_offset': 2, u'token': u'my', u'type': u'<ALPHANUM>', u'start_offset': 0, u'position': 0}, {u'end_offset': 8, u'token': u'kitty', u'type': u'<ALPHANUM>', u'start_offset': 3, u'position': 1}, {u'end_offset': 14, u'token': u\"cat's\", u'type': u'<ALPHANUM>', u'start_offset': 9, u'position': 2}, {u'end_offset': 18, u'token': u'not', u'type': u'<ALPHANUM>', u'start_offset': 15, u'position': 3}, {u'end_offset': 20, u'token': u'a', u'type': u'<ALPHANUM>', u'start_offset': 19, u'position': 4}, {u'end_offset': 25, u'token': u'pain', u'type': u'<ALPHANUM>', u'start_offset': 21, u'position': 5}, {u'end_offset': 28, u'token': u'in', u'type': u'<ALPHANUM>', u'start_offset': 26, u'position': 6}, {u'end_offset': 32, u'token': u'the', u'type': u'<ALPHANUM>', u'start_offset': 29, u'position': 7}, {u'end_offset': 38, u'token': u'neck', u'type': u'<ALPHANUM>', u'start_offset': 34, u'position': 8}]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "u\"my kitty cat's not a pain in the neck\""
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_analyzer_tokens(localclient.analyze(index='my_index',\n",
    "                                        analyzer='standard',\n",
    "                                        text='My kitty-cat\\'s not a pain in the \\'neck\\'!'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{u'end_offset': 2, u'token': u'My', u'type': u'<ALPHANUM>', u'start_offset': 0, u'position': 0}, {u'end_offset': 8, u'token': u'kitty', u'type': u'<ALPHANUM>', u'start_offset': 3, u'position': 1}, {u'end_offset': 14, u'token': u\"cat's\", u'type': u'<ALPHANUM>', u'start_offset': 9, u'position': 2}, {u'end_offset': 18, u'token': u'not', u'type': u'<ALPHANUM>', u'start_offset': 15, u'position': 3}, {u'end_offset': 20, u'token': u'a', u'type': u'<ALPHANUM>', u'start_offset': 19, u'position': 4}, {u'end_offset': 25, u'token': u'pain', u'type': u'<ALPHANUM>', u'start_offset': 21, u'position': 5}, {u'end_offset': 28, u'token': u'in', u'type': u'<ALPHANUM>', u'start_offset': 26, u'position': 6}, {u'end_offset': 32, u'token': u'the', u'type': u'<ALPHANUM>', u'start_offset': 29, u'position': 7}, {u'end_offset': 38, u'token': u'neck', u'type': u'<ALPHANUM>', u'start_offset': 34, u'position': 8}]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "u\"My kitty cat's not a pain in the neck\""
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#  The difference is subtle but there.\n",
    "get_analyzer_tokens(localclient.analyze(index='my_index',\n",
    "                                        tokenizer=\"standard\",\n",
    "                                        text='My kitty-cat\\'s not a pain in the \\'neck\\'!'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{u'end_offset': 2, u'token': u'my', u'type': u'<ALPHANUM>', u'start_offset': 0, u'position': 0}, {u'end_offset': 8, u'token': u'kitti', u'type': u'<ALPHANUM>', u'start_offset': 3, u'position': 1}, {u'end_offset': 14, u'token': u'cat', u'type': u'<ALPHANUM>', u'start_offset': 9, u'position': 2}, {u'end_offset': 25, u'token': u'pain', u'type': u'<ALPHANUM>', u'start_offset': 21, u'position': 5}, {u'end_offset': 38, u'token': u'neck', u'type': u'<ALPHANUM>', u'start_offset': 34, u'position': 8}]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "u'my kitti cat pain neck'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# However, if you use the english analyzer it will override that uppercase and also remove the negation, \n",
    "# because \"not\" is in the stopwords list:\n",
    "get_analyzer_tokens(localclient.analyze(index='my_index',\n",
    "                                        analyzer=\"english\",\n",
    "                                        tokenizer=\"standard\",\n",
    "                                        text='My kitty-cat\\'s not a pain in the \\'neck\\'!'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Indexing Yelp Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_msgpack(\"data/yelp_df_forES.msg\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>business_id</th>\n",
       "      <th>date</th>\n",
       "      <th>review_id</th>\n",
       "      <th>stars</th>\n",
       "      <th>text</th>\n",
       "      <th>user_id</th>\n",
       "      <th>tokens</th>\n",
       "      <th>text_length</th>\n",
       "      <th>net_sentiment</th>\n",
       "      <th>sent_per_token</th>\n",
       "      <th>fake_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9yKzy9PApeiPPOUJEtnvkg</td>\n",
       "      <td>2011-01-26</td>\n",
       "      <td>fWKvX83p0-ka4JS3dc6E5A</td>\n",
       "      <td>5</td>\n",
       "      <td>My wife took me here on my birthday for breakf...</td>\n",
       "      <td>rLtl8ZkDX5vH5nAx9C3q5Q</td>\n",
       "      <td>wife took birthday breakfast excellent weather...</td>\n",
       "      <td>542</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0.064576</td>\n",
       "      <td>The Our Saturday</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6oRAC4uyJCsJl1X0WZpVSA</td>\n",
       "      <td>2012-06-14</td>\n",
       "      <td>IESLBzqUCLdSzSqm0eCSxQ</td>\n",
       "      <td>4</td>\n",
       "      <td>love the gyro plate. Rice is so good and I als...</td>\n",
       "      <td>0hT2KtfLiobPvh6cDC8JQg</td>\n",
       "      <td>love gyro plate rice good also dig candy selec...</td>\n",
       "      <td>50</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.120000</td>\n",
       "      <td>Rice</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>_1QQZuf4zZOyFCvXc0o6Vg</td>\n",
       "      <td>2010-05-27</td>\n",
       "      <td>G-WvGaISbqqaMHlNnByodA</td>\n",
       "      <td>5</td>\n",
       "      <td>Rosie, Dakota, and I LOVE Chaparral Dog Park!!...</td>\n",
       "      <td>uZetl9T0NcROGOyFfughhg</td>\n",
       "      <td>rosie dakota love chaparral dog park convenien...</td>\n",
       "      <td>280</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.035714</td>\n",
       "      <td>Rosie Dakota LOVE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6ozycU1RpktNG2-1BroVtw</td>\n",
       "      <td>2012-01-05</td>\n",
       "      <td>1uJFq2r5QfJG_6ExMRCaGw</td>\n",
       "      <td>5</td>\n",
       "      <td>General Manager Scott Petello is a good egg!!!...</td>\n",
       "      <td>vYmM4KTsC8ZfQBg-j5MWkw</td>\n",
       "      <td>general manager scott petello good egg detail ...</td>\n",
       "      <td>248</td>\n",
       "      <td>13.0</td>\n",
       "      <td>0.052419</td>\n",
       "      <td>General Manager Scott</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>hW0Ne_HTHEAgGF1rAdmR-g</td>\n",
       "      <td>2012-07-12</td>\n",
       "      <td>JL7GXJ9u4YMx7Rzs05NfiQ</td>\n",
       "      <td>4</td>\n",
       "      <td>Luckily, I didn't have to travel far to make m...</td>\n",
       "      <td>1ieuYcKS7zeAv_U15AB13A</td>\n",
       "      <td>luckily travel far make connecting flight than...</td>\n",
       "      <td>162</td>\n",
       "      <td>13.0</td>\n",
       "      <td>0.080247</td>\n",
       "      <td>Luckily And Phoenix</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              business_id        date               review_id  stars  \\\n",
       "0  9yKzy9PApeiPPOUJEtnvkg  2011-01-26  fWKvX83p0-ka4JS3dc6E5A      5   \n",
       "2  6oRAC4uyJCsJl1X0WZpVSA  2012-06-14  IESLBzqUCLdSzSqm0eCSxQ      4   \n",
       "3  _1QQZuf4zZOyFCvXc0o6Vg  2010-05-27  G-WvGaISbqqaMHlNnByodA      5   \n",
       "4  6ozycU1RpktNG2-1BroVtw  2012-01-05  1uJFq2r5QfJG_6ExMRCaGw      5   \n",
       "7  hW0Ne_HTHEAgGF1rAdmR-g  2012-07-12  JL7GXJ9u4YMx7Rzs05NfiQ      4   \n",
       "\n",
       "                                                text                 user_id  \\\n",
       "0  My wife took me here on my birthday for breakf...  rLtl8ZkDX5vH5nAx9C3q5Q   \n",
       "2  love the gyro plate. Rice is so good and I als...  0hT2KtfLiobPvh6cDC8JQg   \n",
       "3  Rosie, Dakota, and I LOVE Chaparral Dog Park!!...  uZetl9T0NcROGOyFfughhg   \n",
       "4  General Manager Scott Petello is a good egg!!!...  vYmM4KTsC8ZfQBg-j5MWkw   \n",
       "7  Luckily, I didn't have to travel far to make m...  1ieuYcKS7zeAv_U15AB13A   \n",
       "\n",
       "                                              tokens  text_length  \\\n",
       "0  wife took birthday breakfast excellent weather...          542   \n",
       "2  love gyro plate rice good also dig candy selec...           50   \n",
       "3  rosie dakota love chaparral dog park convenien...          280   \n",
       "4  general manager scott petello good egg detail ...          248   \n",
       "7  luckily travel far make connecting flight than...          162   \n",
       "\n",
       "   net_sentiment  sent_per_token              fake_name  \n",
       "0           35.0        0.064576       The Our Saturday  \n",
       "2            6.0        0.120000                   Rice  \n",
       "3           10.0        0.035714      Rosie Dakota LOVE  \n",
       "4           13.0        0.052419  General Manager Scott  \n",
       "7           13.0        0.080247    Luckily And Phoenix  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# test with a small sample if you want\n",
    "dfshort = df.query('stars >= 5 and net_sentiment > 35')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "358"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dfshort)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# filter out any rows with a nan for sent_per_token, which breaks bulk load:\n",
    "df = df[df.sent_per_token.isnull() != True]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "MAPPING = {\n",
    "\t        'review': {\n",
    "\t            'properties': {\n",
    "\t                'business_id': {'index': 'not_analyzed', 'type': 'string'},\n",
    "\t                'date': {'index': 'not_analyzed', 'format': 'dateOptionalTime', 'type': 'date'},\n",
    "\t                'review_id': {'index': 'not_analyzed', 'type': 'string'},\n",
    "                    'stars': {'index': 'not_analyzed', 'type': 'integer'},\n",
    "\t                'text': {'index': 'analyzed',\n",
    "                             'analyzer': 'english',\n",
    "                             'store': 'yes',\n",
    "                             \"term_vector\": \"with_positions_offsets_payloads\",\n",
    "                             'type': 'string'},\n",
    "                    'fake_name': {'index':'not_analyzed', 'type': 'string'},\n",
    "                    'text_orig': {'index': 'not_analyzed', 'type': 'string'},\n",
    "                    'user_id': {'index': 'not_analyzed', 'type': 'string'},\n",
    "                    'net_sentiment': {'index': 'not_analyzed', 'type': 'integer'},\n",
    "                    'sent_per_token': {'index': 'not_analyzed', 'type': 'float'}\n",
    "\t            }}\n",
    "\t}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'acknowledged': True}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "localEs.indices.delete(index='yelp')  # nb: this errors the first time you run it. comment out.\n",
    "localEs.indices.create(index='yelp')\n",
    "localEs.indices.put_mapping(index='yelp', doc_type='review', body=json.dumps(MAPPING))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Bulk data is structured as alternating opt_dict and data dicts.\n",
    "\n",
    "bulk_data = []\n",
    "\n",
    "for index, row in df.iterrows():\n",
    "    data_dict = {}\n",
    "    data_dict['text_orig'] = row['text']\n",
    "    data_dict['text']= row['text']\n",
    "    data_dict['net_sentiment'] = row['net_sentiment'] \n",
    "    data_dict['sent_per_token'] = row['sent_per_token']\n",
    "    data_dict['stars'] = row['stars']\n",
    "    data_dict['fake_name'] = row['fake_name']\n",
    "    data_dict['user_id'] = row['user_id']\n",
    "    data_dict['business_id'] = row['business_id']\n",
    "    data_dict['date'] = row['date']\n",
    "    data_dict['review_id'] = row['review_id']\n",
    "    op_dict = {\n",
    "            \"index\": {\n",
    "            \"_index\": 'yelp',\n",
    "            \"_type\": 'review',\n",
    "            \"_id\": row['review_id']\n",
    "            }\n",
    "    }\n",
    "    bulk_data.append(op_dict)\n",
    "    bulk_data.append(data_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'index': {'_id': u'fWKvX83p0-ka4JS3dc6E5A',\n",
       "  '_index': 'yelp',\n",
       "  '_type': 'review'}}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bulk_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "344884"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(bulk_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# May time out with a large bulk_data bump or error and fail without any reason.  Mine did, so see below.\n",
    "res = localEs.bulk(index = 'yelp', body = bulk_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# In order to find the error, I did them one-by-one, with a try.\n",
    "for ind, obj in enumerate(bulk_data):\n",
    "    # every other one is the data, so use those to do it one by one\n",
    "    if ind % 2 != 0:\n",
    "        try:\n",
    "            localEs.index(index='yelp', doc_type='review', id=obj['review_id'], body=json.dumps(obj))\n",
    "        except:\n",
    "            print obj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'_shards': {u'failed': 0, u'successful': 5, u'total': 5},\n",
       " u'hits': {u'hits': [{u'_id': u'mkNmB283CAXzUfpQm0hr5Q',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 2.1016731,\n",
       "    u'_source': {u'business_id': u'8Hn5X1AqgmSLHRG2KgBJBg',\n",
       "     u'date': u'2012-03-09',\n",
       "     u'fake_name': u'Probably Phoenix There',\n",
       "     u'net_sentiment': 5.0,\n",
       "     u'review_id': u'mkNmB283CAXzUfpQm0hr5Q',\n",
       "     u'sent_per_token': 0.16129032258064516,\n",
       "     u'stars': 4,\n",
       "     u'text': u'Great pizza, better pizza cookie.',\n",
       "     u'text_orig': u'Great pizza, better pizza cookie.',\n",
       "     u'user_id': u'4TZARz8AjNWfukK2UfZitA'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'lxrcTePTWl6GpssJOPl1Sg',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 1.7622974,\n",
       "    u'_source': {u'business_id': u'ioGIaoswvbbeY8P965IYRA',\n",
       "     u'date': u'2012-05-19',\n",
       "     u'fake_name': u'Oregano Flagstaff NAU',\n",
       "     u'net_sentiment': 5.0,\n",
       "     u'review_id': u'lxrcTePTWl6GpssJOPl1Sg',\n",
       "     u'sent_per_token': 0.0625,\n",
       "     u'stars': 5,\n",
       "     u'text': u'Save room at the end for a pizza cookie! I even converted my sweets-hating husband to a pizza cookie fan!',\n",
       "     u'text_orig': u'Save room at the end for a pizza cookie! I even converted my sweets-hating husband to a pizza cookie fan!',\n",
       "     u'user_id': u'sih5J4HDcRMdcp8MehHAnA'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'LTOulb_HY2CSNFXx9fqukA',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 1.5776026,\n",
       "    u'_source': {u'business_id': u'kFVapB4z_5VoOnYazF_ERA',\n",
       "     u'date': u'2012-01-06',\n",
       "     u'fake_name': u'The Saturday Declining',\n",
       "     u'net_sentiment': 0.0,\n",
       "     u'review_id': u'LTOulb_HY2CSNFXx9fqukA',\n",
       "     u'sent_per_token': 0.0,\n",
       "     u'stars': 4,\n",
       "     u'text': u\"Two words: cookie pizza...  That's all you need to know.\",\n",
       "     u'text_orig': u\"Two words: cookie pizza...  That's all you need to know.\",\n",
       "     u'user_id': u'f01B5tRtqf3k6ak1pPSAWw'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'tEdvJNJrcfQk1UyQmz9Wng',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 1.5743153,\n",
       "    u'_source': {u'business_id': u'fKXwK5K5iCAMPHgqVPDhiQ',\n",
       "     u'date': u'2012-10-21',\n",
       "     u'fake_name': u'Good Staff',\n",
       "     u'net_sentiment': 13.0,\n",
       "     u'review_id': u'tEdvJNJrcfQk1UyQmz9Wng',\n",
       "     u'sent_per_token': 0.203125,\n",
       "     u'stars': 4,\n",
       "     u'text': u'Good pizza, good cookie dough, good coupons! Staff was very helpful and friendly.',\n",
       "     u'text_orig': u'Good pizza, good cookie dough, good coupons! Staff was very helpful and friendly.',\n",
       "     u'user_id': u'kE2Dw1sKMeaDOpyZNVQKJQ'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'fmn5yGrPChOYMR2vGOIrYA',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 1.2620821,\n",
       "    u'_source': {u'business_id': u'FRiT2RI6Y1nds1jjZqvnXQ',\n",
       "     u'date': u'2012-08-21',\n",
       "     u'fake_name': u'Airport Very',\n",
       "     u'net_sentiment': 10.0,\n",
       "     u'review_id': u'fmn5yGrPChOYMR2vGOIrYA',\n",
       "     u'sent_per_token': 0.08771929824561403,\n",
       "     u'stars': 4,\n",
       "     u'text': u'Very impressive airport fare.  One of the better thin crust pizzas I have ever had.  Recommend pizza with caramelized onion.\\n\\nAlso good large $1 cookie!',\n",
       "     u'text_orig': u'Very impressive airport fare.  One of the better thin crust pizzas I have ever had.  Recommend pizza with caramelized onion.\\n\\nAlso good large $1 cookie!',\n",
       "     u'user_id': u'YkUWDC6iML9burjYuaiRKQ'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'90y9fw-LTefWZYZO9aPmoA',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 1.2620821,\n",
       "    u'_source': {u'business_id': u'ioGIaoswvbbeY8P965IYRA',\n",
       "     u'date': u'2012-10-19',\n",
       "     u'fake_name': u'Oregano Flagstaff NAU',\n",
       "     u'net_sentiment': 4.0,\n",
       "     u'review_id': u'90y9fw-LTefWZYZO9aPmoA',\n",
       "     u'sent_per_token': 0.06666666666666667,\n",
       "     u'stars': 5,\n",
       "     u'text': u'Chicago style pan is worth the 45 min wait to make!  Save room for the pizza cookie!',\n",
       "     u'text_orig': u'Chicago style pan is worth the 45 min wait to make!  Save room for the pizza cookie!',\n",
       "     u'user_id': u'0fWyq8bc9TYp7WAgYh2CeA'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'wP5vRvMGbwEtOFoHrSfU0g',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 1.2594522,\n",
       "    u'_source': {u'business_id': u'8Hn5X1AqgmSLHRG2KgBJBg',\n",
       "     u'date': u'2012-11-07',\n",
       "     u'fake_name': u'Probably Phoenix There',\n",
       "     u'net_sentiment': 1.0,\n",
       "     u'review_id': u'wP5vRvMGbwEtOFoHrSfU0g',\n",
       "     u'sent_per_token': 0.012195121951219513,\n",
       "     u'stars': 5,\n",
       "     u'text': u'mini chicken pesto pizza for lunch is to die for! may look small but will be enough for two people! the cookie is amazing to!',\n",
       "     u'text_orig': u'mini chicken pesto pizza for lunch is to die for! may look small but will be enough for two people! the cookie is amazing to!',\n",
       "     u'user_id': u'uFXrezdNbph5QbmvYVmhEA'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'dGs1PA79NITmi9KNTVyYJg',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 1.2594522,\n",
       "    u'_source': {u'business_id': u'LMG0zsAkUSscIvmV9vvm3A',\n",
       "     u'date': u'2011-12-30',\n",
       "     u'fake_name': u'Christmas Oregano Italian',\n",
       "     u'net_sentiment': 6.0,\n",
       "     u'review_id': u'dGs1PA79NITmi9KNTVyYJg',\n",
       "     u'sent_per_token': 0.08450704225352113,\n",
       "     u'stars': 4,\n",
       "     u'text': u'Beefstro salad is the best.  I get it every time.  The pizza cookie is hands down the best dessert in town.',\n",
       "     u'text_orig': u'Beefstro salad is the best.  I get it every time.  The pizza cookie is hands down the best dessert in town.',\n",
       "     u'user_id': u'CNR3bzad-Cgc8s-wojn3EQ'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'AlNt1MA0SbwxCFFYcSKhsg',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 1.2594522,\n",
       "    u'_source': {u'business_id': u'kFVapB4z_5VoOnYazF_ERA',\n",
       "     u'date': u'2010-01-20',\n",
       "     u'fake_name': u'The Saturday Declining',\n",
       "     u'net_sentiment': 0.0,\n",
       "     u'review_id': u'AlNt1MA0SbwxCFFYcSKhsg',\n",
       "     u'sent_per_token': 0.0,\n",
       "     u'stars': 5,\n",
       "     u'text': u'again....the freaking pizza cookie. how can you not just fade away into bliss when eating this thing....it it it it is ridiculous!',\n",
       "     u'text_orig': u'again....the freaking pizza cookie. how can you not just fade away into bliss when eating this thing....it it it it is ridiculous!',\n",
       "     u'user_id': u'ST8Yzlk2MqKlcaLqL2djBg'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'ji0werJjmnuYhFpRZZ_eNQ',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 1.2461324,\n",
       "    u'_source': {u'business_id': u'6v0vQm3wXzaFoEywvpbnqg',\n",
       "     u'date': u'2012-10-26',\n",
       "     u'fake_name': u'Best The Jennifer',\n",
       "     u'net_sentiment': 16.0,\n",
       "     u'review_id': u'ji0werJjmnuYhFpRZZ_eNQ',\n",
       "     u'sent_per_token': 0.125,\n",
       "     u'stars': 4,\n",
       "     u'text': u'Fun atmosphere-good pizza, good size small appetizers, good tomato and mozzarella salad and Excellent pie-zookie (warm cookie with ice cream on top).',\n",
       "     u'text_orig': u'Fun atmosphere-good pizza, good size small appetizers, good tomato and mozzarella salad and Excellent pie-zookie (warm cookie with ice cream on top).',\n",
       "     u'user_id': u'5c4p-A1sLBlpS1cPaMpdBg'},\n",
       "    u'_type': u'review'}],\n",
       "  u'max_score': 2.1016731,\n",
       "  u'total': 10985},\n",
       " u'timed_out': False,\n",
       " u'took': 2138}"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "localEs.search(index='yelp', doc_type='review', q='pizza-cookie')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that score relevancy results are based on the indexed TF-IDF for the doc and docs:\n",
    "    https://www.elastic.co/guide/en/elasticsearch/guide/current/relevance-intro.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Want to explain why something matched?  You need the id of the matched doc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'_id': u'fmn5yGrPChOYMR2vGOIrYA',\n",
       " u'_index': u'yelp',\n",
       " u'_type': u'review',\n",
       " u'explanation': {u'description': u'sum of:',\n",
       "  u'details': [{u'description': u'sum of:',\n",
       "    u'details': [{u'description': u'weight(_all:pizza in 10428) [PerFieldSimilarity], result of:',\n",
       "      u'details': [{u'description': u'score(doc=10428,freq=2.0), product of:',\n",
       "        u'details': [{u'description': u'queryWeight, product of:',\n",
       "          u'details': [{u'description': u'idf(docFreq=1896, maxDocs=36713)',\n",
       "            u'details': [],\n",
       "            u'value': 3.9628572},\n",
       "           {u'description': u'queryNorm',\n",
       "            u'details': [],\n",
       "            u'value': 0.13838264}],\n",
       "          u'value': 0.5483907},\n",
       "         {u'description': u'fieldWeight in 10428, product of:',\n",
       "          u'details': [{u'description': u'tf(freq=2.0), with freq of:',\n",
       "            u'details': [{u'description': u'termFreq=2.0',\n",
       "              u'details': [],\n",
       "              u'value': 2.0}],\n",
       "            u'value': 1.4142135},\n",
       "           {u'description': u'idf(docFreq=1896, maxDocs=36713)',\n",
       "            u'details': [],\n",
       "            u'value': 3.9628572},\n",
       "           {u'description': u'fieldNorm(doc=10428)',\n",
       "            u'details': [],\n",
       "            u'value': 0.125}],\n",
       "          u'value': 0.7005408}],\n",
       "        u'value': 0.38417003}],\n",
       "      u'value': 0.38417003},\n",
       "     {u'description': u'weight(_all:cookie in 10428) [PerFieldSimilarity], result of:',\n",
       "      u'details': [{u'description': u'score(doc=10428,freq=2.0), product of:',\n",
       "        u'details': [{u'description': u'queryWeight, product of:',\n",
       "          u'details': [{u'description': u'idf(docFreq=236, maxDocs=36713)',\n",
       "            u'details': [],\n",
       "            u'value': 6.042826},\n",
       "           {u'description': u'queryNorm',\n",
       "            u'details': [],\n",
       "            u'value': 0.13838264}],\n",
       "          u'value': 0.83622223},\n",
       "         {u'description': u'fieldWeight in 10428, product of:',\n",
       "          u'details': [{u'description': u'tf(freq=2.0), with freq of:',\n",
       "            u'details': [{u'description': u'termFreq=2.0',\n",
       "              u'details': [],\n",
       "              u'value': 2.0}],\n",
       "            u'value': 1.4142135},\n",
       "           {u'description': u'idf(docFreq=236, maxDocs=36713)',\n",
       "            u'details': [],\n",
       "            u'value': 6.042826},\n",
       "           {u'description': u'fieldNorm(doc=10428)',\n",
       "            u'details': [],\n",
       "            u'value': 0.125}],\n",
       "          u'value': 1.0682309}],\n",
       "        u'value': 0.8932784}],\n",
       "      u'value': 0.8932784}],\n",
       "    u'value': 1.2774484},\n",
       "   {u'description': u'match on required clause, product of:',\n",
       "    u'details': [{u'description': u'# clause', u'details': [], u'value': 0.0},\n",
       "     {u'description': u'_type:review, product of:',\n",
       "      u'details': [{u'description': u'boost', u'details': [], u'value': 1.0},\n",
       "       {u'description': u'queryNorm', u'details': [], u'value': 0.13838264}],\n",
       "      u'value': 0.13838264}],\n",
       "    u'value': 0.0}],\n",
       "  u'value': 1.2774484},\n",
       " u'matched': True}"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "localEs.explain(index='yelp', doc_type='review', q='pizza-cookie', id=u'fmn5yGrPChOYMR2vGOIrYA')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More Like This\n",
    "\n",
    "A variety of options for finding similar documents, including term counts and custom stop words: https://www.elastic.co/guide/en/elasticsearch/reference/2.3/query-dsl-mlt-query.html\n",
    "  \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'My wife took me here on my birthday for breakfast and it was excellent.  The weather was perfect which made sitting outside overlooking their grounds an absolute pleasure.  Our waitress was excellent and our food arrived quickly on the semi-busy Saturday morning.  It looked like the place fills up pretty quickly so the earlier you get here the better.\\n\\nDo yourself a favor and get their Bloody Mary.  It was phenomenal and simply the best I\\'ve ever had.  I\\'m pretty sure they only use ingredients from their garden and blend them fresh when you order it.  It was amazing.\\n\\nWhile EVERYTHING on the menu looks excellent, I had the white truffle scrambled eggs vegetable skillet and it was tasty and delicious.  It came with 2 pieces of their griddled bread with was amazing and it absolutely made the meal complete.  It was the best \"toast\" I\\'ve ever had.\\n\\nAnyway, I can\\'t wait to go back!'"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = df.iloc[0].text\n",
    "text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "QUERY = {\n",
    "      \"query\": { \n",
    "        \"more_like_this\" : {\n",
    "            \"fields\" : [\"text\"],\n",
    "            \"like_text\": text,\n",
    "            \"analyzer\" : \"english\",\n",
    "            \"min_term_freq\" : 2\n",
    "            }\n",
    "        }\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'_shards': {u'failed': 0, u'successful': 5, u'total': 5},\n",
       " u'hits': {u'hits': [{u'_id': u'fWKvX83p0-ka4JS3dc6E5A',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 1.8600082,\n",
       "    u'_source': {u'business_id': u'9yKzy9PApeiPPOUJEtnvkg',\n",
       "     u'date': u'2011-01-26',\n",
       "     u'fake_name': u'The Our Saturday',\n",
       "     u'net_sentiment': 35.0,\n",
       "     u'review_id': u'fWKvX83p0-ka4JS3dc6E5A',\n",
       "     u'sent_per_token': 0.06457564575645756,\n",
       "     u'stars': 5,\n",
       "     u'text': u'My wife took me here on my birthday for breakfast and it was excellent.  The weather was perfect which made sitting outside overlooking their grounds an absolute pleasure.  Our waitress was excellent and our food arrived quickly on the semi-busy Saturday morning.  It looked like the place fills up pretty quickly so the earlier you get here the better.\\n\\nDo yourself a favor and get their Bloody Mary.  It was phenomenal and simply the best I\\'ve ever had.  I\\'m pretty sure they only use ingredients from their garden and blend them fresh when you order it.  It was amazing.\\n\\nWhile EVERYTHING on the menu looks excellent, I had the white truffle scrambled eggs vegetable skillet and it was tasty and delicious.  It came with 2 pieces of their griddled bread with was amazing and it absolutely made the meal complete.  It was the best \"toast\" I\\'ve ever had.\\n\\nAnyway, I can\\'t wait to go back!',\n",
       "     u'text_orig': u'My wife took me here on my birthday for breakfast and it was excellent.  The weather was perfect which made sitting outside overlooking their grounds an absolute pleasure.  Our waitress was excellent and our food arrived quickly on the semi-busy Saturday morning.  It looked like the place fills up pretty quickly so the earlier you get here the better.\\n\\nDo yourself a favor and get their Bloody Mary.  It was phenomenal and simply the best I\\'ve ever had.  I\\'m pretty sure they only use ingredients from their garden and blend them fresh when you order it.  It was amazing.\\n\\nWhile EVERYTHING on the menu looks excellent, I had the white truffle scrambled eggs vegetable skillet and it was tasty and delicious.  It came with 2 pieces of their griddled bread with was amazing and it absolutely made the meal complete.  It was the best \"toast\" I\\'ve ever had.\\n\\nAnyway, I can\\'t wait to go back!',\n",
       "     u'user_id': u'rLtl8ZkDX5vH5nAx9C3q5Q'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'8iGqxj3mxtFx_sI0aFcEAQ',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 0.6409701,\n",
       "    u'_source': {u'business_id': u'Rg4dHuszzF0Mb4yqtoHSiA',\n",
       "     u'date': u'2011-07-22',\n",
       "     u'fake_name': u'Still Los Tacos',\n",
       "     u'net_sentiment': 29.0,\n",
       "     u'review_id': u'8iGqxj3mxtFx_sI0aFcEAQ',\n",
       "     u'sent_per_token': 0.06473214285714286,\n",
       "     u'stars': 5,\n",
       "     u'text': u\"If you are looking for a GREAT family owned place to get some excellent Mexican food look no further!  This place is awesome from the authentic selection of meats to fill your tacos and burritos with to the handmade chunky guacamole that is made for EACH person as they order it right in front of you!  The owner Rudy himself is always nice and welcomes you with a smile.  The restaurant is staffed with his family members and they are always quick and very helpful.  The carne asada is some of the best I have ever had!  Not much seating inside and they do have some outside but we usually just get it to go and enjoy it at home.  They have AMAZING tamales but don't always have them in stock so don't forget to ask!  Overall this is one of if not my absolute favorite place to get a burrito and some handmade guac!\",\n",
       "     u'text_orig': u\"If you are looking for a GREAT family owned place to get some excellent Mexican food look no further!  This place is awesome from the authentic selection of meats to fill your tacos and burritos with to the handmade chunky guacamole that is made for EACH person as they order it right in front of you!  The owner Rudy himself is always nice and welcomes you with a smile.  The restaurant is staffed with his family members and they are always quick and very helpful.  The carne asada is some of the best I have ever had!  Not much seating inside and they do have some outside but we usually just get it to go and enjoy it at home.  They have AMAZING tamales but don't always have them in stock so don't forget to ask!  Overall this is one of if not my absolute favorite place to get a burrito and some handmade guac!\",\n",
       "     u'user_id': u'Pj4NfwbcszVybCrqibna1g'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'9PSk0p0cUOllZYlLFCcQsw',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 0.63211685,\n",
       "    u'_source': {u'business_id': u'MPziC5MO2QQiYVAMM3fdSw',\n",
       "     u'date': u'2011-09-06',\n",
       "     u'fake_name': u'Saturday Our The',\n",
       "     u'net_sentiment': 21.0,\n",
       "     u'review_id': u'9PSk0p0cUOllZYlLFCcQsw',\n",
       "     u'sent_per_token': 0.043209876543209874,\n",
       "     u'stars': 4,\n",
       "     u'text': u\"I went here with a party of 6 on Saturday night.  Our table was ready and waiting when we got there which was nice.  It is an older place with tons of charm.  The place was pretty busy and you could tell that the waiters/servers/busers were clearly over worked.\\n\\nAfter sitting for about 10 minutes we had to flag someone down to get bread and place our appetizer order.  After that we only saw our waiter every 20 minutes or so so it was difficult to order additional items or to get soda refills.\\n\\nThe service was pretty blah but the food was excellent.  Everything I tried was top notch.  Everything was freshly made and delicious  Probably some of the best Italian cuisine around.  Definitely some of the best gnocchi I've ever had.\\n\\nI would definitely return but hope the service is better.\",\n",
       "     u'text_orig': u\"I went here with a party of 6 on Saturday night.  Our table was ready and waiting when we got there which was nice.  It is an older place with tons of charm.  The place was pretty busy and you could tell that the waiters/servers/busers were clearly over worked.\\n\\nAfter sitting for about 10 minutes we had to flag someone down to get bread and place our appetizer order.  After that we only saw our waiter every 20 minutes or so so it was difficult to order additional items or to get soda refills.\\n\\nThe service was pretty blah but the food was excellent.  Everything I tried was top notch.  Everything was freshly made and delicious  Probably some of the best Italian cuisine around.  Definitely some of the best gnocchi I've ever had.\\n\\nI would definitely return but hope the service is better.\",\n",
       "     u'user_id': u'rLtl8ZkDX5vH5nAx9C3q5Q'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'l_QJ2rWSXVeK6e86dQkruw',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 0.61993265,\n",
       "    u'_source': {u'business_id': u'3Zsjlum5kl5N5KV712aTMQ',\n",
       "     u'date': u'2010-10-06',\n",
       "     u'fake_name': u'This FAVORITE Phoenix',\n",
       "     u'net_sentiment': 33.0,\n",
       "     u'review_id': u'l_QJ2rWSXVeK6e86dQkruw',\n",
       "     u'sent_per_token': 0.08991825613079019,\n",
       "     u'stars': 4,\n",
       "     u'text': u\"Been here twice now.  The views are absolutely amazing.  Even if you don't want to come for dinner, it has a great patio lounge that you can come to for drinks.  Its not a bar scene, but more of a place to go and get drinks with good friends.  The food has ranged from excellent to good.  I had possibly the best lemon drop martini of my life here.  The seafood dishes are excellent.  I also ordered a brussel sprout salad which was amazing.  The steak was ok.  I've had better.  Would definitely recommend this place for a quiet dinner with good friends.  Try to get a table that faces outward.  Its really all about the view.\",\n",
       "     u'text_orig': u\"Been here twice now.  The views are absolutely amazing.  Even if you don't want to come for dinner, it has a great patio lounge that you can come to for drinks.  Its not a bar scene, but more of a place to go and get drinks with good friends.  The food has ranged from excellent to good.  I had possibly the best lemon drop martini of my life here.  The seafood dishes are excellent.  I also ordered a brussel sprout salad which was amazing.  The steak was ok.  I've had better.  Would definitely recommend this place for a quiet dinner with good friends.  Try to get a table that faces outward.  Its really all about the view.\",\n",
       "     u'user_id': u'19BunjXk7zhd811rF42U7w'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'fBjrZHbgeTwtzIv2saIocw',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 0.6185121,\n",
       "    u'_source': {u'business_id': u'lk-6NXfpSgvNw7zm0xtUVw',\n",
       "     u'date': u'2012-11-04',\n",
       "     u'fake_name': u'Huge Enough Fresh',\n",
       "     u'net_sentiment': 6.0,\n",
       "     u'review_id': u'fBjrZHbgeTwtzIv2saIocw',\n",
       "     u'sent_per_token': 0.0759493670886076,\n",
       "     u'stars': 4,\n",
       "     u'text': u\"Great tacos/burritos made to order...absolutely the best menudo I've ever had. Pops and I have been regulars here for years.\",\n",
       "     u'text_orig': u\"Great tacos/burritos made to order...absolutely the best menudo I've ever had. Pops and I have been regulars here for years.\",\n",
       "     u'user_id': u'6LYoJJptmLKIziAMyu_WtQ'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'tNy8Tv2NfrES7UcKwj914w',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 0.6159888,\n",
       "    u'_source': {u'business_id': u'K8pM6qQdYu5h6buRE1-_sw',\n",
       "     u'date': u'2008-03-03',\n",
       "     u'fake_name': u'Every Phoenix The',\n",
       "     u'net_sentiment': 17.0,\n",
       "     u'review_id': u'tNy8Tv2NfrES7UcKwj914w',\n",
       "     u'sent_per_token': 0.056291390728476824,\n",
       "     u'stars': 5,\n",
       "     u'text': u\"Lo-Lo's has some of the best food you can find in Phoenix, and they honestly make the best Southern food I've ever had.  Their fried chicken is absolutely fantastic - it's just so moist and flavorful.  Their Mac 'N Cheese is also excellent.  Honestly, everything here is just really good - the only thing that disappointed me was that the cheese grits weren't quite as cheesy as I had expected them to be.  Still though, they were delicious.  If you're looking for good Southern food, this is absolutely the right place to go.\",\n",
       "     u'text_orig': u\"Lo-Lo's has some of the best food you can find in Phoenix, and they honestly make the best Southern food I've ever had.  Their fried chicken is absolutely fantastic - it's just so moist and flavorful.  Their Mac 'N Cheese is also excellent.  Honestly, everything here is just really good - the only thing that disappointed me was that the cheese grits weren't quite as cheesy as I had expected them to be.  Still though, they were delicious.  If you're looking for good Southern food, this is absolutely the right place to go.\",\n",
       "     u'user_id': u'cXhs6BawP6L_V59zKb9dOw'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'WTzaQMBx8xaqg5JWFVbzqg',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 0.5924493,\n",
       "    u'_source': {u'business_id': u'O-Xa9GCFWI65YiBD5Jw_hA',\n",
       "     u'date': u'2009-10-19',\n",
       "     u'fake_name': u'The Caesar',\n",
       "     u'net_sentiment': 22.0,\n",
       "     u'review_id': u'WTzaQMBx8xaqg5JWFVbzqg',\n",
       "     u'sent_per_token': 0.05513784461152882,\n",
       "     u'stars': 5,\n",
       "     u'text': u\"Delicious!!!  Everything we had was absolutely amazing ... best New York steak I've ever had - very juicy and flavorful.  We also ordered 3 sides - sauteed broccoli, garlic mash and mac and cheese, all of which were also excellent!  Our server suggested adding a jalapeno into the mac and cheese to give it some kick, but some people in our group don't eat spicy food, so we didn't.  But I imagine that would have made it even better.  \\n\\nService was outstanding and I also love the ambiance.  We went on a Saturday night and they had live music in the bar area.  I'd definitely save this for a special occasion though as it was quite pricey.\",\n",
       "     u'text_orig': u\"Delicious!!!  Everything we had was absolutely amazing ... best New York steak I've ever had - very juicy and flavorful.  We also ordered 3 sides - sauteed broccoli, garlic mash and mac and cheese, all of which were also excellent!  Our server suggested adding a jalapeno into the mac and cheese to give it some kick, but some people in our group don't eat spicy food, so we didn't.  But I imagine that would have made it even better.  \\n\\nService was outstanding and I also love the ambiance.  We went on a Saturday night and they had live music in the bar area.  I'd definitely save this for a special occasion though as it was quite pricey.\",\n",
       "     u'user_id': u'mQZea1qUUUOqCBkpCUMTRA'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'fF-ReMS064kGw04cG4wCmQ',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 0.57324606,\n",
       "    u'_source': {u'business_id': u'Zdi-Mrwk_JNnhopAAbxRcQ',\n",
       "     u'date': u'2010-02-21',\n",
       "     u'fake_name': u'This During This',\n",
       "     u'net_sentiment': 28.0,\n",
       "     u'review_id': u'fF-ReMS064kGw04cG4wCmQ',\n",
       "     u'sent_per_token': 0.05622489959839357,\n",
       "     u'stars': 5,\n",
       "     u'text': u\"This has got to be one of the best churches I've been to.  During my travels I definitely get a little nervous about visiting another church for mass.  I cannot stand judgmental churches and was looking for a place near our hotel that would not be.  \\n\\nThis place was far from judgmental.  While the congregation may appear to be older and more conservative, they were incredibly lively, welcoming, and friendly.  Before the mass starts, they welcome visitors and everyone greets each other beforehand.  The music is amazing and they even had one lector who was a radio announcer (made for an excellent reading).  \\n\\nWhile there, they had a presiding priest who gave a great and funny homily.  \\n\\nAll in all a wonderful and welcoming congregation.  If I'm ever in the area again and needing to go to mass.  I'll definitely put this place at the top of my list.\",\n",
       "     u'text_orig': u\"This has got to be one of the best churches I've been to.  During my travels I definitely get a little nervous about visiting another church for mass.  I cannot stand judgmental churches and was looking for a place near our hotel that would not be.  \\n\\nThis place was far from judgmental.  While the congregation may appear to be older and more conservative, they were incredibly lively, welcoming, and friendly.  Before the mass starts, they welcome visitors and everyone greets each other beforehand.  The music is amazing and they even had one lector who was a radio announcer (made for an excellent reading).  \\n\\nWhile there, they had a presiding priest who gave a great and funny homily.  \\n\\nAll in all a wonderful and welcoming congregation.  If I'm ever in the area again and needing to go to mass.  I'll definitely put this place at the top of my list.\",\n",
       "     u'user_id': u'pXTM0MEmK24EO1NQK8zKMw'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'wB83ASXnZ8lzRVGxRTjycw',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 0.57247144,\n",
       "    u'_source': {u'business_id': u'e8SKP917u2g0MTAaoGkCnA',\n",
       "     u'date': u'2010-08-18',\n",
       "     u'fake_name': u'This Taco Bell',\n",
       "     u'net_sentiment': 13.0,\n",
       "     u'review_id': u'wB83ASXnZ8lzRVGxRTjycw',\n",
       "     u'sent_per_token': 0.05803571428571429,\n",
       "     u'stars': 4,\n",
       "     u'text': u\"This Taco Bell/Long John Silvers is awesome! My coworkers and I frequent this location pretty often and we always get good service, our food is made quickly, and the workers here are nice. I'd have to say this is probably the cleanest Taco Bell I've ever been to. If I'm going to eat food that's kinda bad for me, I'd at least like to have a pleasant experience doing so...\",\n",
       "     u'text_orig': u\"This Taco Bell/Long John Silvers is awesome! My coworkers and I frequent this location pretty often and we always get good service, our food is made quickly, and the workers here are nice. I'd have to say this is probably the cleanest Taco Bell I've ever been to. If I'm going to eat food that's kinda bad for me, I'd at least like to have a pleasant experience doing so...\",\n",
       "     u'user_id': u'3Zi1udoY2iT9s8Q3X6tgZQ'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'ffFWuZbyrs0qDemT9YzX5g',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 0.5604533,\n",
       "    u'_source': {u'business_id': u'PuHucejs4LG3hSXnrwi9-g',\n",
       "     u'date': u'2012-01-06',\n",
       "     u'fake_name': u'The',\n",
       "     u'net_sentiment': 15.0,\n",
       "     u'review_id': u'ffFWuZbyrs0qDemT9YzX5g',\n",
       "     u'sent_per_token': 0.14018691588785046,\n",
       "     u'stars': 5,\n",
       "     u'text': u\"Very cute restaurant... our server was great and the food was some of the best I've ever had. The French Onion soup was very good, and the beef bourguignon was absolutely amazing.   I can't wait to go again!\",\n",
       "     u'text_orig': u\"Very cute restaurant... our server was great and the food was some of the best I've ever had. The French Onion soup was very good, and the beef bourguignon was absolutely amazing.   I can't wait to go again!\",\n",
       "     u'user_id': u'z1M-HXnX4WQYvRM_WXuRxQ'},\n",
       "    u'_type': u'review'}],\n",
       "  u'max_score': 1.8600082,\n",
       "  u'total': 43413},\n",
       " u'timed_out': False,\n",
       " u'took': 3670}"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Result is not brilliant, though.  You could limit the hits unless a score threshold is hit.\n",
    "localEs.search(index='yelp', doc_type='review', body=json.dumps(QUERY))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Suggestions: For Mispellings\n",
    "\n",
    "Can be added to queries too, to help if there are no matches.  Still in development, though. See: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html#search-suggesters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "SUGGESTION = {\n",
    "    \"my-suggestion\":\n",
    "        {\"text\" : \"cheese piza\",\n",
    "         \"term\" : { \"field\" : \"text\" }\n",
    "        }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'_shards': {u'failed': 0, u'successful': 5, u'total': 5},\n",
       " u'my-suggestion': [{u'length': 6,\n",
       "   u'offset': 0,\n",
       "   u'options': [],\n",
       "   u'text': u'chees'},\n",
       "  {u'length': 4,\n",
       "   u'offset': 7,\n",
       "   u'options': [{u'freq': 4320, u'score': 0.75, u'text': u'pizza'},\n",
       "    {u'freq': 647, u'score': 0.75, u'text': u'pita'},\n",
       "    {u'freq': 21, u'score': 0.75, u'text': u'pima'},\n",
       "    {u'freq': 18, u'score': 0.75, u'text': u'pina'},\n",
       "    {u'freq': 3, u'score': 0.75, u'text': u'pizz'}],\n",
       "   u'text': u'piza'}]}"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# I don't love the results, tbh.  Fail on cheese.\n",
    "\n",
    "localEs.suggest(index='yelp', body=SUGGESTION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reminders:\n",
    "* check your mapping on your fields\n",
    "* check your analyzer results - they can be mysterious and hidden; if you configure wrong, it will use defaults...\n",
    "* check your document tokenization\n",
    "* use multi-fields to be sure of matches that may need stopwords too"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Let's Index the Businesses too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "biz = pd.read_msgpack(\"data/biz_stats_df.msg\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11479"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(biz)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>business_id</th>\n",
       "      <th>reviews</th>\n",
       "      <th>sent_per_token_median</th>\n",
       "      <th>net_sentiment_median</th>\n",
       "      <th>stars_median</th>\n",
       "      <th>stars_mean</th>\n",
       "      <th>stars_std</th>\n",
       "      <th>text_length_median</th>\n",
       "      <th>ids</th>\n",
       "      <th>fake_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>--5jkZ3-nUPZxUvtcbr8Uw</td>\n",
       "      <td>11</td>\n",
       "      <td>0.046753</td>\n",
       "      <td>10.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.545455</td>\n",
       "      <td>0.687552</td>\n",
       "      <td>327.0</td>\n",
       "      <td>--5jkZ3-nUPZxUvtcbr8Uw</td>\n",
       "      <td>George The Tzatziki</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>--BlvDO_RG2yElKu9XA1_g</td>\n",
       "      <td>29</td>\n",
       "      <td>0.041667</td>\n",
       "      <td>10.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.103448</td>\n",
       "      <td>1.291312</td>\n",
       "      <td>242.0</td>\n",
       "      <td>--BlvDO_RG2yElKu9XA1_g</td>\n",
       "      <td>Hawaii Hawaii That</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0D_CYhlD2ILkmLR0pBmnA</td>\n",
       "      <td>3</td>\n",
       "      <td>0.023339</td>\n",
       "      <td>13.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.333333</td>\n",
       "      <td>1.154701</td>\n",
       "      <td>298.0</td>\n",
       "      <td>-0D_CYhlD2ILkmLR0pBmnA</td>\n",
       "      <td>Saw Olive Gourmet</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              business_id  reviews  sent_per_token_median  \\\n",
       "0  --5jkZ3-nUPZxUvtcbr8Uw       11               0.046753   \n",
       "1  --BlvDO_RG2yElKu9XA1_g       29               0.041667   \n",
       "2  -0D_CYhlD2ILkmLR0pBmnA        3               0.023339   \n",
       "\n",
       "   net_sentiment_median  stars_median  stars_mean  stars_std  \\\n",
       "0                  10.0           5.0    4.545455   0.687552   \n",
       "1                  10.0           5.0    4.103448   1.291312   \n",
       "2                  13.0           4.0    3.333333   1.154701   \n",
       "\n",
       "   text_length_median                     ids            fake_name  \n",
       "0               327.0  --5jkZ3-nUPZxUvtcbr8Uw  George The Tzatziki  \n",
       "1               242.0  --BlvDO_RG2yElKu9XA1_g   Hawaii Hawaii That  \n",
       "2               298.0  -0D_CYhlD2ILkmLR0pBmnA    Saw Olive Gourmet  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "biz.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "B_MAPPING = {\n",
    "\t        'business': {\n",
    "\t            'properties': {\n",
    "\t                'business_id': {'index': 'not_analyzed', 'type': 'string'},\n",
    "\t                'reviews': {'index': 'not_analyzed', 'type': 'integer'},\n",
    "\t                'stars_median': {'index': 'not_analyzed', 'type': 'float'},\n",
    "                    'stars_mean': {'index': 'not_analyzed', 'type': 'float'},\n",
    "                    'text_length_median': {'index': 'not_analyzed', 'type': 'float'},\n",
    "                    'fake_name': {'index':'not_analyzed', 'type': 'string'},\n",
    "                    'net_sentiment_median': {'index': 'not_analyzed', 'type': 'float'},\n",
    "                    'sent_per_token_median': {'index': 'not_analyzed', 'type': 'float'}\n",
    "\t            }}\n",
    "\t}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'acknowledged': True}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#localEs.indices.delete(index='yelp')  # nb: this errors the first time you run it. comment out.\n",
    "#localEs.indices.create(index='yelp')  # do not do this is you already made the reviews!\n",
    "localEs.indices.put_mapping(index='yelp', doc_type='business', body=json.dumps(B_MAPPING))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Bulk data is structured as alternating opt_dict and data dicts.\n",
    "\n",
    "bulk_data = []\n",
    "\n",
    "for index, row in biz.iterrows():\n",
    "    data_dict = {}\n",
    "    data_dict['net_sentiment_median'] = row['net_sentiment_median'] \n",
    "    data_dict['sent_per_token_median'] = row['sent_per_token_median']\n",
    "    data_dict['stars_median'] = row['stars_median']\n",
    "    data_dict['stars_mean'] = row['stars_mean']\n",
    "    data_dict['fake_name'] = row['fake_name']\n",
    "    data_dict['text_length_median'] = row['text_length_median']\n",
    "    data_dict['business_id'] = row['business_id']\n",
    "    data_dict['reviews'] = row['reviews']\n",
    "    op_dict = {\n",
    "            \"index\": {\n",
    "            \"_index\": 'yelp',\n",
    "            \"_type\": 'business',\n",
    "            \"_id\": row['business_id']\n",
    "            }\n",
    "    }\n",
    "    bulk_data.append(op_dict)\n",
    "    bulk_data.append(data_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# May time out with a large bulk_data bump or error and fail without any reason.  Mine did, so see below.\n",
    "res = localEs.bulk(index = 'yelp', body = bulk_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'_shards': {u'failed': 0, u'successful': 5, u'total': 5},\n",
       " u'hits': {u'hits': [{u'_id': u'JokKtdXU7zXHcr20Lrk29A',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 2.140637,\n",
       "    u'_source': {u'business_id': u'JokKtdXU7zXHcr20Lrk29A',\n",
       "     u'fake_name': u'This The The',\n",
       "     u'net_sentiment_median': 11.0,\n",
       "     u'reviews': 571,\n",
       "     u'sent_per_token_median': 0.05241935483870968,\n",
       "     u'stars_mean': 4.352014010507881,\n",
       "     u'stars_median': 4.0,\n",
       "     u'text_length_median': 231.0},\n",
       "    u'_type': u'business'}],\n",
       "  u'max_score': 2.140637,\n",
       "  u'total': 1},\n",
       " u'timed_out': False,\n",
       " u'took': 6}"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "localEs.search(index='yelp', doc_type='business', q='JokKtdXU7zXHcr20Lrk29A')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aggregate Queries to get Business ID's and More\n",
    "\n",
    "\n",
    "Here we are using the operator \"and\" to make sure all words in the search match, and then getting counts of matching business id's."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "QUERY = {\n",
    "    \"query\": {\n",
    "        \"match\": {\n",
    "            \"text\": {      \n",
    "                \"query\":    \"good pizza\",\n",
    "                \"operator\": \"and\"\n",
    "            }\n",
    "        }\n",
    "    },\n",
    "    \"aggs\" : {\n",
    "      \"businesses\" : {\n",
    "            \"terms\" : {\n",
    "              \"field\" : \"business_id\"\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'_shards': {u'failed': 0, u'successful': 5, u'total': 5},\n",
       " u'aggregations': {u'businesses': {u'buckets': [{u'doc_count': 184,\n",
       "     u'key': u'VVeogjZya58oiTxK7qUjAQ'},\n",
       "    {u'doc_count': 123, u'key': u'pwpl-rxwNRQdgqFz_-qMPg'},\n",
       "    {u'doc_count': 120, u'key': u'V1nEpIRmEa1768oj_tuxeQ'},\n",
       "    {u'doc_count': 93, u'key': u'dcd3C1gWv-vVdQ9XYV8Ubw'},\n",
       "    {u'doc_count': 69, u'key': u'7SO_rX1F6rQEl-5s3wZxgQ'},\n",
       "    {u'doc_count': 68, u'key': u'VY_tvNUCCXGXQeSvJl757Q'},\n",
       "    {u'doc_count': 52, u'key': u'8Hn5X1AqgmSLHRG2KgBJBg'},\n",
       "    {u'doc_count': 52, u'key': u'LMG0zsAkUSscIvmV9vvm3A'},\n",
       "    {u'doc_count': 46, u'key': u'XkNQVTkCEzBrq7OlRHI11Q'},\n",
       "    {u'doc_count': 46, u'key': u'YQvg0JCGRFUkb6reMMf3Iw'}],\n",
       "   u'doc_count_error_upper_bound': 23,\n",
       "   u'sum_other_doc_count': 4053}},\n",
       " u'hits': {u'hits': [{u'_id': u'csIoVmNtuhxrQvNd0zPi6g',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 2.7316122,\n",
       "    u'_source': {u'business_id': u'e_riFHMoJ1Yguvr0KtOkDQ',\n",
       "     u'date': u'2012-10-27',\n",
       "     u'fake_name': u'Very This The',\n",
       "     u'net_sentiment': 3.0,\n",
       "     u'review_id': u'csIoVmNtuhxrQvNd0zPi6g',\n",
       "     u'sent_per_token': 0.3,\n",
       "     u'stars': 3,\n",
       "     u'text': u'Good pizza',\n",
       "     u'text_orig': u'Good pizza',\n",
       "     u'user_id': u'i-HvYHgHJaXlYcUai16KVQ'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'H7hrHeJAFoCuX-rwPjZq4g',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 2.4152129,\n",
       "    u'_source': {u'business_id': u's5BD7v8zO-YCjm9xnOLo8g',\n",
       "     u'date': u'2011-05-20',\n",
       "     u'fake_name': u'Rosatis Baseline The',\n",
       "     u'net_sentiment': 9.0,\n",
       "     u'review_id': u'H7hrHeJAFoCuX-rwPjZq4g',\n",
       "     u'sent_per_token': 0.45,\n",
       "     u'stars': 3,\n",
       "     u'text': u'Good pizza not the best! But good',\n",
       "     u'text_orig': u'Good pizza not the best! But good',\n",
       "     u'user_id': u'6wpEHXevnaIZp2kUTUfhxQ'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'_IhV8F_zfxwMaot4FuMf9g',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 2.215016,\n",
       "    u'_source': {u'business_id': u'Xk3inJcuBGLhavUcdlINZA',\n",
       "     u'date': u'2012-08-29',\n",
       "     u'fake_name': u'Offered Somewhat Had',\n",
       "     u'net_sentiment': 7.0,\n",
       "     u'review_id': u'_IhV8F_zfxwMaot4FuMf9g',\n",
       "     u'sent_per_token': 0.35,\n",
       "     u'stars': 3,\n",
       "     u'text': u'good pizza, fun place',\n",
       "     u'text_orig': u'good pizza, fun place',\n",
       "     u'user_id': u'nM3vfxr6fcnN_nXdzwbVCw'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'KiGrPpMK3yXrdPB7vgqT6g',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 2.215016,\n",
       "    u'_source': {u'business_id': u'yKbXWo_6VlS3mD_RcPquog',\n",
       "     u'date': u'2012-12-31',\n",
       "     u'fake_name': u'Love Their HUGE',\n",
       "     u'net_sentiment': 3.0,\n",
       "     u'review_id': u'KiGrPpMK3yXrdPB7vgqT6g',\n",
       "     u'sent_per_token': 0.3,\n",
       "     u'stars': 4,\n",
       "     u'text': u'Very good pizza!!',\n",
       "     u'text_orig': u'Very good pizza!!',\n",
       "     u'user_id': u'5UVxIS3gWIPtd9H8bMG0aQ'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'NYMqags7lLJLMke4SVpmSQ',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 2.215016,\n",
       "    u'_source': {u'business_id': u'MagW3D2UDQaP6KHzhauUHw',\n",
       "     u'date': u'2011-08-27',\n",
       "     u'fake_name': u'Here The Hudson',\n",
       "     u'net_sentiment': 3.0,\n",
       "     u'review_id': u'NYMqags7lLJLMke4SVpmSQ',\n",
       "     u'sent_per_token': 0.17647058823529413,\n",
       "     u'stars': 4,\n",
       "     u'text': u'Very good pizza slices.',\n",
       "     u'text_orig': u'Very good pizza slices.',\n",
       "     u'user_id': u'zzVpQ6IBxeW3ZDdg0Ep_wg'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'7GvQrxmGZJN3YHUcf4hP6Q',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 2.193916,\n",
       "    u'_source': {u'business_id': u'rFgSvsgo0BWnS03NFgI7Sg',\n",
       "     u'date': u'2012-06-20',\n",
       "     u'fake_name': u'Horrendously The',\n",
       "     u'net_sentiment': 3.0,\n",
       "     u'review_id': u'7GvQrxmGZJN3YHUcf4hP6Q',\n",
       "     u'sent_per_token': 0.13043478260869565,\n",
       "     u'stars': 5,\n",
       "     u'text': u'Surprisingly good pizza!',\n",
       "     u'text_orig': u'Surprisingly good pizza!',\n",
       "     u'user_id': u'40d7fEqD1ddLGpkMDcP5jA'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'WyidyA4lfV53X9N_tQ2brA',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 2.1852899,\n",
       "    u'_source': {u'business_id': u'L9UYbtAUOcfTgZFimehlXw',\n",
       "     u'date': u'2009-11-02',\n",
       "     u'fake_name': u'Joe That Phoenix',\n",
       "     u'net_sentiment': 3.0,\n",
       "     u'review_id': u'WyidyA4lfV53X9N_tQ2brA',\n",
       "     u'sent_per_token': 0.17647058823529413,\n",
       "     u'stars': 5,\n",
       "     u'text': u'really good pizza.',\n",
       "     u'text_orig': u'really good pizza.',\n",
       "     u'user_id': u'6jiCkha-eL2gOhwl6k0-Ow'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'jYFgOoEYejwdrHPDp-OSUg',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 2.1852899,\n",
       "    u'_source': {u'business_id': u'x4L42igQPv4TFlqGR2Wthg',\n",
       "     u'date': u'2009-08-23',\n",
       "     u'fake_name': u'None',\n",
       "     u'net_sentiment': 4.0,\n",
       "     u'review_id': u'jYFgOoEYejwdrHPDp-OSUg',\n",
       "     u'sent_per_token': 0.23529411764705882,\n",
       "     u'stars': 3,\n",
       "     u'text': u'Pretty good pizza',\n",
       "     u'text_orig': u'Pretty good pizza',\n",
       "     u'user_id': u'hIWlVkf-N1Yzj7I-dfF7qA'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'37iXAeCVWaoeZIyorwLkyQ',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 2.1679807,\n",
       "    u'_source': {u'business_id': u'cUaaIk_3UdeSJ54CLvrW8A',\n",
       "     u'date': u'2008-12-30',\n",
       "     u'fake_name': u'What Actually The',\n",
       "     u'net_sentiment': 3.0,\n",
       "     u'review_id': u'37iXAeCVWaoeZIyorwLkyQ',\n",
       "     u'sent_per_token': 0.1875,\n",
       "     u'stars': 3,\n",
       "     u'text': u'Good pizza. Cheap.',\n",
       "     u'text_orig': u'Good pizza. Cheap.',\n",
       "     u'user_id': u'H0nYlvGjMA9szTaqIwTwpg'},\n",
       "    u'_type': u'review'},\n",
       "   {u'_id': u'yUyLzJekCRc9xj1PB1hlxg',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 2.1487236,\n",
       "    u'_source': {u'business_id': u'9Wy2CxVwnKoT5X4GrGzfyw',\n",
       "     u'date': u'2011-08-18',\n",
       "     u'fake_name': u'Great Chicago Good',\n",
       "     u'net_sentiment': 6.0,\n",
       "     u'review_id': u'yUyLzJekCRc9xj1PB1hlxg',\n",
       "     u'sent_per_token': 0.14634146341463414,\n",
       "     u'stars': 5,\n",
       "     u'text': u'Best pizza in the West Valley.  As good as Chicago pizza',\n",
       "     u'text_orig': u'Best pizza in the West Valley.  As good as Chicago pizza',\n",
       "     u'user_id': u'8B26BN2WQkw9i-DcU7Kavw'},\n",
       "    u'_type': u'review'}],\n",
       "  u'max_score': 2.7316122,\n",
       "  u'total': 4906},\n",
       " u'timed_out': False,\n",
       " u'took': 2417}"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "localEs.search(index=\"yelp\", doc_type=\"review\", body=json.dumps(QUERY))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# exact match on field: https://www.elastic.co/guide/en/elasticsearch/guide/master/_finding_exact_values.html\n",
    "# requires not indexed field for the match\n",
    "QUERY = {\n",
    "  \"query\": {\n",
    "    \"constant_score\" : { \n",
    "        \"filter\" : {\n",
    "            \"term\" : { \n",
    "                      \"business_id\" : \"VVeogjZya58oiTxK7qUjAQ\"\n",
    "                    }\n",
    "            }\n",
    "        }\n",
    "  }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'_shards': {u'failed': 0, u'successful': 5, u'total': 5},\n",
       " u'hits': {u'hits': [{u'_id': u'VVeogjZya58oiTxK7qUjAQ',\n",
       "    u'_index': u'yelp',\n",
       "    u'_score': 1.0,\n",
       "    u'_source': {u'business_id': u'VVeogjZya58oiTxK7qUjAQ',\n",
       "     u'fake_name': u'Pizzeria Bianco Useless',\n",
       "     u'net_sentiment_median': 9.0,\n",
       "     u'reviews': 486,\n",
       "     u'sent_per_token_median': 0.034909109373413225,\n",
       "     u'stars_mean': 3.9526748971193415,\n",
       "     u'stars_median': 4.0,\n",
       "     u'text_length_median': 279.5},\n",
       "    u'_type': u'business'}],\n",
       "  u'max_score': 1.0,\n",
       "  u'total': 1},\n",
       " u'timed_out': False,\n",
       " u'took': 656}"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "localEs.search(index=\"yelp\", doc_type=\"business\", body=json.dumps(QUERY))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Now Move to The JS App\n",
    "\n",
    "Now that you have the data indexed and searchable, we can build a small app to iterate on for the eventual ui you might want.  Use the \"web\" folder in my repo for that.\n",
    "\n",
    "### Some Other Reference Materials:\n",
    "\n",
    "* Tutorial slides from PyCon 2015, repo here (may be out of date!): https://github.com/erikrose/elasticsearch-tutorial\n",
    "* Docs for the Python Lib: https://elasticsearch-py.readthedocs.org/en/master/index.html\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}