examples/pattern_matching.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebooks illustrates how to write regex like patterns to match sequences of events. While, I write up more about the API, look at the original inspiration for this library [here](https://observablehq.com/@mikpanko/sequence-pattern-matching)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import awkward as ak\n",
    "import pyarrow as pa\n",
    "import pyarrow.parquet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from seqmatcher.codegen import clear_cache\n",
    "from seqmatcher.matching_jitted import match_pattern, extract_pattern\n",
    "from seqmatcher.parsing import parse_match_pattern\n",
    "from seqmatcher.utilities import get_types_from_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We serialize the generated code to disk for debugging, and because it is easier to compile \n",
    "# Numba code that way. \n",
    "clear_cache()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the shot by shot tennis dataset. We view it as an `awkward array` since that lets us run jitted \n",
    "Numba code against it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = pa.parquet.read_table(\"../datasets/tennis_shot_by_shot.parquet\")\n",
    "awk_data = ak.from_arrow(dataset)\n",
    "\n",
    "# collect the python types of the columns in the dataset, since we need those \n",
    "# to parse the pattern.\n",
    "seq_type_map, events_type_map = get_types_from_dataset(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we go. Lets look at how many rallies were serve faults. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "pat_str = \"|-(:serve_fault)-(:serve_fault)\"\n",
    "pat = parse_match_pattern(pat_str)\n",
    "pat.type_properties(seq_type_map, events_type_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Array [{seq_id: 59, evt_indices: [0, ... 1]}] type='14972 * {\"seq_id\": int64, \"...'>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = match_pattern(pat, awk_data)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output result is a bunch of numpy arrays. Using those, we can sub-select from the input list of sequences to get the sequences that match the pattern. \n",
    "\n",
    "The `extract_pattern` routine can be used to do this in one go. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Array [{matchId: 0, pointNumber: 60, ... ] type='14972 * {\"matchId\": ?int64, \"p...'>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = extract_pattern(pat, awk_data)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another ex - if a rally starts with a serve fault, is the player more likely to lose the point? Lets find out. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# player1 ends up as a winner\n",
    "pat_str = \"|-(:serve_fault)-[..]()-(:winner|:ace {isPlayer1=True})-|\"\n",
    "pat = parse_match_pattern(pat_str)\n",
    "pat.type_properties(seq_type_map, events_type_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Array [{seq_id: 4, evt_indices: [0, ... 1]}] type='20136 * {\"seq_id\": int64, \"e...'>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = match_pattern(pat, awk_data)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, the number of times they lose. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# player1 ends up losing the point\n",
    "pat_str = \"|-(:serve_fault)-[..]()-(!:winner|:ace {isPlayer1=True})-|\"\n",
    "pat = parse_match_pattern(pat_str)\n",
    "pat.type_properties(seq_type_map, events_type_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Array [{seq_id: 2, evt_indices: [0, ... 1]}] type='82896 * {\"seq_id\": int64, \"e...'>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = match_pattern(pat, awk_data)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That doesn't quite seem right - a 4x difference in points made when you fault on the first serve!\n",
    "\n",
    "I wasn't identifying the rallies which player 1 wins correctly, since we leave out rallies which ended with the other player making the last shot! So, one way to correct this would be to also match agains the rallies which end with player 2. And sum up the two results. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we could also write some custom code to do it in one go!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# player 1 winning\n",
    "\n",
    "pat_str = \"|-(:serve_fault)-[..]()-(@end)-|\"\n",
    "code_str = \"(@end.isPlayer1==True and @end._eventName in ('winner', 'ace')) or (@end.isPlayer1==False and @end._eventName not in ('winner', 'ace'))\"\n",
    "pat = parse_match_pattern(pat_str, code_str)\n",
    "pat.type_properties(seq_type_map, events_type_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Array [{seq_id: 0, evt_indices: [0, ... 1]}] type='82393 * {\"seq_id\": int64, \"e...'>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = match_pattern(pat, awk_data)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, calculating the number of times they lost the point. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "pat_str = \"|-(:serve_fault)-[..]()-(@end)-|\"\n",
    "code_str = \"(@end.isPlayer1==True and @end._eventName not in ('winner', 'ace')) or (@end.isPlayer1==False and @end._eventName in ('winner', 'ace'))\"\n",
    "pat = parse_match_pattern(pat_str, code_str)\n",
    "pat.type_properties(seq_type_map, events_type_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Array [{seq_id: 2, evt_indices: [0, ... 1]}] type='82209 * {\"seq_id\": int64, \"e...'>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = match_pattern(pat, awk_data)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Almost the same amount. So, starting with a serve fault doesn't really matter for the point much. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.5 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "bee13da5dfd8cc2ab21d09493523ce7a17f20d31f50093ee24dd9a0766090f17"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}