pratical-rl/week2_model_based/practice_vi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Markov decision process\n",
    "\n",
    "This week's methods are all built to solve __M__arkov __D__ecision __P__rocesses. In the broadest sense, an MDP is defined by how it changes states and how rewards are computed.\n",
    "\n",
    "State transition is defined by $P(s' |s,a)$ - how likely areare you to end at state $s'$ if you take action $a$ from state $s$. Now there's more than one way to define rewards, but we'll use $r(s,a,s')$ function for convenience."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For starters, let's define a simple MDP from this picture:\n",
    "<img src='https://upload.wikimedia.org/wikipedia/commons/thumb/a/ad/Markov_Decision_Process.svg/800px-Markov_Decision_Process.svg.png' width=300px>\n",
    "_img by MistWiz (Own work) [Public domain], via Wikimedia Commons_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "transition_probs = {\n",
    "  's0':{\n",
    "    'a0': {'s0': 0.5, 's2': 0.5},\n",
    "    'a1': {'s2': 1}\n",
    "  },\n",
    "  's1':{\n",
    "    'a0': {'s0': 0.7, 's1': 0.1, 's2': 0.2},\n",
    "    'a1': {'s1': 0.95, 's2': 0.05}\n",
    "  },\n",
    "  's2':{\n",
    "    'a0': {'s0': 0.4, 's1': 0.6},\n",
    "    'a1': {'s0': 0.3, 's1': 0.3, 's2':0.4}\n",
    "  }\n",
    "}\n",
    "rewards = {\n",
    "  's1': {'a0': {'s0': +5}},\n",
    "  's2': {'a1': {'s0': -1}}\n",
    "}\n",
    "\n",
    "from mdp import MDP\n",
    "mdp = MDP(transition_probs, rewards, initial_state='s0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use MDP just as any other gym environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "initial state = s0\n",
      "next_state = s2, reward = 0.0, done = False\n"
     ]
    }
   ],
   "source": [
    "print('initial state =', mdp.reset())\n",
    "next_state, reward, done, info = mdp.step('a1')\n",
    "print('next_state = %s, reward = %s, done = %s' % (next_state, reward, done))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "but it also has other methods that you'll need for Value Iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mdp.get_all_states = ('s0', 's1', 's2')\n",
      "mdp.get_possible_actions('s1') =  ('a0', 'a1')\n",
      "mdp.get_next_states('s1', 'a0') =  {'s0': 0.7, 's1': 0.1, 's2': 0.2}\n",
      "mdp.get_reward('s1', 'a0', 's0') =  5\n",
      "mdp.get_transition_prob('s1', 'a0', 's0') =  0.7\n"
     ]
    }
   ],
   "source": [
    "print(\"mdp.get_all_states =\", mdp.get_all_states())\n",
    "print(\"mdp.get_possible_actions('s1') = \", mdp.get_possible_actions('s1'))\n",
    "print(\"mdp.get_next_states('s1', 'a0') = \", mdp.get_next_states('s1', 'a0'))\n",
    "print(\"mdp.get_reward('s1', 'a0', 's0') = \", mdp.get_reward('s1', 'a0', 's0'))\n",
    "print(\"mdp.get_transition_prob('s1', 'a0', 's0') = \", mdp.get_transition_prob('s1', 'a0', 's0'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Value Iteration\n",
    "\n",
    "Now let's build something to solve this MDP. The simplest algorithm so far is __V__alue __I__teration\n",
    "\n",
    "Here's the pseudo-code for VI:\n",
    "\n",
    "---\n",
    "\n",
    "`1.` Initialize $V^{(0)}(s)=0$, for all $s$\n",
    "\n",
    "`2.` For $i=0, 1, 2, \\dots$\n",
    " \n",
    "`3.` $ \\quad V_{(i+1)}(s) = \\max_a \\sum_{s'} P(s' | s,a) \\cdot [ r(s,a,s') + \\gamma V_{i}(s')]$, for all $s$\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's write a function to compute the state-action value function $Q^{\\pi}$, defined as follows\n",
    "\n",
    "$$Q_i(s, a) = \\sum_{s'} P(s' | s,a) \\cdot [ r(s,a,s') + \\gamma V_{i}(s')]$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_action_value(mdp, state_values, state, action, gamma):\n",
    "    \"\"\" Computes Q(s,a) as in formula above \"\"\"\n",
    "#     <YOUR CODE>\n",
    "    next_states_dict = mdp.get_next_states(state, action)\n",
    "    Q = 0\n",
    "    for n_s, prob in next_states_dict.items():\n",
    "        reward = mdp.get_reward(state, action, n_s)\n",
    "        Q += prob * (reward + gamma * state_values[n_s])\n",
    "    return Q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "test_Vs = {s : i for i, s in enumerate(mdp.get_all_states())}\n",
    "assert np.allclose(get_action_value(mdp, test_Vs, 's2', 'a1', 0.9), 0.69)\n",
    "assert np.allclose(get_action_value(mdp, test_Vs, 's1', 'a0', 0.9), 3.95)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using $Q(s,a)$ we can now define the \"next\" V(s) for value iteration.\n",
    " $$V_{(i+1)}(s) = \\max_a \\sum_{s'} P(s' | s,a) \\cdot [ r(s,a,s') + \\gamma V_{i}(s')] = \\max_a Q_i(s,a)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_new_state_value(mdp, state_values, state, gamma):\n",
    "    \"\"\" Computes next V(s) as per formula above. Please do not change state_values in process. \"\"\"\n",
    "    if mdp.is_terminal(state): return 0\n",
    "    \n",
    "    actions = mdp.get_possible_actions(state)\n",
    "    Qs = [get_action_value(mdp, state_values, state, action, gamma) for action in actions]\n",
    "    \n",
    "    return max(Qs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_Vs_vopy = dict(test_Vs)\n",
    "assert np.allclose(get_new_state_value(mdp, test_Vs, 's0', 0.9), 1.8)\n",
    "assert np.allclose(get_new_state_value(mdp, test_Vs, 's2', 0.9), 0.69)\n",
    "assert test_Vs == test_Vs_vopy, \"please do not change state_values in get_new_state_value\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's combine everything we wrote into a working value iteration algo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter    0   |   diff: 3.50000   |   V(s0) = 0.000   V(s1) = 0.000   V(s2) = 0.000\n",
      "\n",
      "iter    1   |   diff: 1.89000   |   V(s0) = 0.000   V(s1) = 3.500   V(s2) = 0.000\n",
      "\n",
      "iter    2   |   diff: 1.70100   |   V(s0) = 0.000   V(s1) = 3.815   V(s2) = 1.890\n",
      "\n",
      "iter    3   |   diff: 1.13542   |   V(s0) = 1.701   V(s1) = 4.184   V(s2) = 2.060\n",
      "\n",
      "iter    4   |   diff: 0.73024   |   V(s0) = 1.854   V(s1) = 5.319   V(s2) = 2.871\n",
      "\n",
      "iter    5   |   diff: 0.61135   |   V(s0) = 2.584   V(s1) = 5.664   V(s2) = 3.540\n",
      "\n",
      "iter    6   |   diff: 0.54664   |   V(s0) = 3.186   V(s1) = 6.275   V(s2) = 3.989\n",
      "\n",
      "iter    7   |   diff: 0.49198   |   V(s0) = 3.590   V(s1) = 6.790   V(s2) = 4.535\n",
      "\n",
      "iter    8   |   diff: 0.42210   |   V(s0) = 4.082   V(s1) = 7.189   V(s2) = 4.959\n",
      "\n",
      "iter    9   |   diff: 0.36513   |   V(s0) = 4.463   V(s1) = 7.611   V(s2) = 5.352\n",
      "\n",
      "iter   10   |   diff: 0.32862   |   V(s0) = 4.816   V(s1) = 7.960   V(s2) = 5.717\n",
      "\n",
      "iter   11   |   diff: 0.29262   |   V(s0) = 5.145   V(s1) = 8.280   V(s2) = 6.032\n",
      "\n",
      "iter   12   |   diff: 0.26189   |   V(s0) = 5.429   V(s1) = 8.572   V(s2) = 6.323\n",
      "\n",
      "iter   13   |   diff: 0.23503   |   V(s0) = 5.691   V(s1) = 8.830   V(s2) = 6.584\n",
      "\n",
      "iter   14   |   diff: 0.21124   |   V(s0) = 5.925   V(s1) = 9.065   V(s2) = 6.817\n",
      "\n",
      "iter   15   |   diff: 0.19012   |   V(s0) = 6.135   V(s1) = 9.276   V(s2) = 7.028\n",
      "\n",
      "iter   16   |   diff: 0.17091   |   V(s0) = 6.325   V(s1) = 9.465   V(s2) = 7.218\n",
      "\n",
      "iter   17   |   diff: 0.15366   |   V(s0) = 6.496   V(s1) = 9.636   V(s2) = 7.388\n",
      "\n",
      "iter   18   |   diff: 0.13830   |   V(s0) = 6.649   V(s1) = 9.790   V(s2) = 7.542\n",
      "\n",
      "iter   19   |   diff: 0.12445   |   V(s0) = 6.788   V(s1) = 9.928   V(s2) = 7.680\n",
      "\n",
      "iter   20   |   diff: 0.11200   |   V(s0) = 6.912   V(s1) = 10.052   V(s2) = 7.805\n",
      "\n",
      "iter   21   |   diff: 0.10079   |   V(s0) = 7.024   V(s1) = 10.164   V(s2) = 7.917\n",
      "\n",
      "iter   22   |   diff: 0.09071   |   V(s0) = 7.125   V(s1) = 10.265   V(s2) = 8.017\n",
      "\n",
      "iter   23   |   diff: 0.08164   |   V(s0) = 7.216   V(s1) = 10.356   V(s2) = 8.108\n",
      "\n",
      "iter   24   |   diff: 0.07347   |   V(s0) = 7.297   V(s1) = 10.437   V(s2) = 8.190\n",
      "\n",
      "iter   25   |   diff: 0.06612   |   V(s0) = 7.371   V(s1) = 10.511   V(s2) = 8.263\n",
      "\n",
      "iter   26   |   diff: 0.05951   |   V(s0) = 7.437   V(s1) = 10.577   V(s2) = 8.329\n",
      "\n",
      "iter   27   |   diff: 0.05356   |   V(s0) = 7.496   V(s1) = 10.636   V(s2) = 8.389\n",
      "\n",
      "iter   28   |   diff: 0.04820   |   V(s0) = 7.550   V(s1) = 10.690   V(s2) = 8.442\n",
      "\n",
      "iter   29   |   diff: 0.04338   |   V(s0) = 7.598   V(s1) = 10.738   V(s2) = 8.491\n",
      "\n",
      "iter   30   |   diff: 0.03904   |   V(s0) = 7.641   V(s1) = 10.782   V(s2) = 8.534\n",
      "\n",
      "iter   31   |   diff: 0.03514   |   V(s0) = 7.681   V(s1) = 10.821   V(s2) = 8.573\n",
      "\n",
      "iter   32   |   diff: 0.03163   |   V(s0) = 7.716   V(s1) = 10.856   V(s2) = 8.608\n",
      "\n",
      "iter   33   |   diff: 0.02846   |   V(s0) = 7.747   V(s1) = 10.887   V(s2) = 8.640\n",
      "\n",
      "iter   34   |   diff: 0.02562   |   V(s0) = 7.776   V(s1) = 10.916   V(s2) = 8.668\n",
      "\n",
      "iter   35   |   diff: 0.02306   |   V(s0) = 7.801   V(s1) = 10.941   V(s2) = 8.694\n",
      "\n",
      "iter   36   |   diff: 0.02075   |   V(s0) = 7.824   V(s1) = 10.964   V(s2) = 8.717\n",
      "\n",
      "iter   37   |   diff: 0.01867   |   V(s0) = 7.845   V(s1) = 10.985   V(s2) = 8.738\n",
      "\n",
      "iter   38   |   diff: 0.01681   |   V(s0) = 7.864   V(s1) = 11.004   V(s2) = 8.756\n",
      "\n",
      "iter   39   |   diff: 0.01513   |   V(s0) = 7.881   V(s1) = 11.021   V(s2) = 8.773\n",
      "\n",
      "iter   40   |   diff: 0.01361   |   V(s0) = 7.896   V(s1) = 11.036   V(s2) = 8.788\n",
      "\n",
      "iter   41   |   diff: 0.01225   |   V(s0) = 7.909   V(s1) = 11.049   V(s2) = 8.802\n",
      "\n",
      "iter   42   |   diff: 0.01103   |   V(s0) = 7.922   V(s1) = 11.062   V(s2) = 8.814\n",
      "\n",
      "iter   43   |   diff: 0.00992   |   V(s0) = 7.933   V(s1) = 11.073   V(s2) = 8.825\n",
      "\n",
      "iter   44   |   diff: 0.00893   |   V(s0) = 7.943   V(s1) = 11.083   V(s2) = 8.835\n",
      "\n",
      "iter   45   |   diff: 0.00804   |   V(s0) = 7.952   V(s1) = 11.092   V(s2) = 8.844\n",
      "\n",
      "iter   46   |   diff: 0.00724   |   V(s0) = 7.960   V(s1) = 11.100   V(s2) = 8.852\n",
      "\n",
      "iter   47   |   diff: 0.00651   |   V(s0) = 7.967   V(s1) = 11.107   V(s2) = 8.859\n",
      "\n",
      "iter   48   |   diff: 0.00586   |   V(s0) = 7.973   V(s1) = 11.113   V(s2) = 8.866\n",
      "\n",
      "iter   49   |   diff: 0.00527   |   V(s0) = 7.979   V(s1) = 11.119   V(s2) = 8.872\n",
      "\n",
      "iter   50   |   diff: 0.00475   |   V(s0) = 7.984   V(s1) = 11.125   V(s2) = 8.877\n",
      "\n",
      "iter   51   |   diff: 0.00427   |   V(s0) = 7.989   V(s1) = 11.129   V(s2) = 8.882\n",
      "\n",
      "iter   52   |   diff: 0.00384   |   V(s0) = 7.993   V(s1) = 11.134   V(s2) = 8.886\n",
      "\n",
      "iter   53   |   diff: 0.00346   |   V(s0) = 7.997   V(s1) = 11.137   V(s2) = 8.890\n",
      "\n",
      "iter   54   |   diff: 0.00311   |   V(s0) = 8.001   V(s1) = 11.141   V(s2) = 8.893\n",
      "\n",
      "iter   55   |   diff: 0.00280   |   V(s0) = 8.004   V(s1) = 11.144   V(s2) = 8.896\n",
      "\n",
      "iter   56   |   diff: 0.00252   |   V(s0) = 8.007   V(s1) = 11.147   V(s2) = 8.899\n",
      "\n",
      "iter   57   |   diff: 0.00227   |   V(s0) = 8.009   V(s1) = 11.149   V(s2) = 8.902\n",
      "\n",
      "iter   58   |   diff: 0.00204   |   V(s0) = 8.011   V(s1) = 11.152   V(s2) = 8.904\n",
      "\n",
      "iter   59   |   diff: 0.00184   |   V(s0) = 8.014   V(s1) = 11.154   V(s2) = 8.906\n",
      "\n",
      "iter   60   |   diff: 0.00166   |   V(s0) = 8.015   V(s1) = 11.155   V(s2) = 8.908\n",
      "\n",
      "iter   61   |   diff: 0.00149   |   V(s0) = 8.017   V(s1) = 11.157   V(s2) = 8.909\n",
      "\n",
      "iter   62   |   diff: 0.00134   |   V(s0) = 8.019   V(s1) = 11.159   V(s2) = 8.911\n",
      "\n",
      "iter   63   |   diff: 0.00121   |   V(s0) = 8.020   V(s1) = 11.160   V(s2) = 8.912\n",
      "\n",
      "iter   64   |   diff: 0.00109   |   V(s0) = 8.021   V(s1) = 11.161   V(s2) = 8.913\n",
      "\n",
      "iter   65   |   diff: 0.00098   |   V(s0) = 8.022   V(s1) = 11.162   V(s2) = 8.915\n",
      "\n",
      "Terminated\n"
     ]
    }
   ],
   "source": [
    "# parameters\n",
    "gamma = 0.9            # discount for MDP\n",
    "num_iter = 100         # maximum iterations, excluding initialization\n",
    "min_difference = 0.001 # stop VI if new values are this close to old values (or closer)\n",
    "\n",
    "# initialize V(s)\n",
    "state_values = {s : 0 for s in mdp.get_all_states()}\n",
    "\n",
    "\n",
    "for i in range(num_iter):\n",
    "    \n",
    "    # Compute new state values using the functions you defined above. It must be a dict {state : new_V(state)}\n",
    "    states = mdp.get_all_states()\n",
    "    new_state_values = { state : get_new_state_value(mdp, state_values, state, gamma) for state in states }\n",
    "    \n",
    "    assert isinstance(new_state_values, dict)\n",
    "    \n",
    "    # Compute difference\n",
    "    diff = max(abs(new_state_values[s] - state_values[s]) for s in mdp.get_all_states())\n",
    "    print(\"iter %4i   |   diff: %6.5f   |   \"%(i, diff), end=\"\")\n",
    "    print('   '.join(\"V(%s) = %.3f\"%(s, v) for s,v in state_values.items()), end='\\n\\n')\n",
    "    state_values = new_state_values\n",
    "    \n",
    "    if diff < min_difference:\n",
    "        print(\"Terminated\"); break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final state values: {'s0': 8.023123818663871, 's1': 11.163174814980803, 's2': 8.915559364985523}\n"
     ]
    }
   ],
   "source": [
    "print(\"Final state values:\", state_values)\n",
    "\n",
    "assert abs(state_values['s0'] - 8.032)  < 0.01\n",
    "assert abs(state_values['s1'] - 11.169) < 0.01\n",
    "assert abs(state_values['s2'] - 8.921)  < 0.01"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's use those $V^{*}(s)$ to find optimal actions in each state\n",
    "\n",
    " $$\\pi^*(s) = argmax_a \\sum_{s'} P(s' | s,a) \\cdot [ r(s,a,s') + \\gamma V_{i}(s')] = argmax_a Q_i(s,a)$$\n",
    " \n",
    "The only difference vs V(s) is that here we take not max but argmax: find action such with maximum Q(s,a)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_optimal_action(mdp, state_values, state, gamma=0.9):\n",
    "    \"\"\" Finds optimal action using formula above. \"\"\"\n",
    "    if mdp.is_terminal(state): return None\n",
    "    \n",
    "    actions = mdp.get_possible_actions(state)\n",
    "    action_value_dict = {\n",
    "        action : get_action_value(mdp, state_values, state, action, gamma) for action in actions\n",
    "    }\n",
    "    \n",
    "    return max(action_value_dict, key=action_value_dict.get)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert get_optimal_action(mdp, state_values, 's0', gamma) == 'a1'\n",
    "assert get_optimal_action(mdp, state_values, 's1', gamma) == 'a0'\n",
    "assert get_optimal_action(mdp, state_values, 's2', gamma) == 'a0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "average reward:  0.9225\n"
     ]
    }
   ],
   "source": [
    "# Measure agent's average reward\n",
    "\n",
    "s = mdp.reset()\n",
    "rewards = []\n",
    "for _ in range(10000):\n",
    "    s, r, done, _ = mdp.step(get_optimal_action(mdp, state_values, s, gamma))\n",
    "    rewards.append(r)\n",
    "    \n",
    "print(\"average reward: \", np.mean(rewards))\n",
    "\n",
    "assert(0.85 < np.mean(rewards) < 1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Frozen lake"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*FFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from mdp import FrozenLakeEnv\n",
    "mdp = FrozenLakeEnv(slip_chance=0)\n",
    "\n",
    "mdp.render()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def value_iteration(mdp, state_values=None, gamma = 0.9, num_iter = 1000, min_difference = 1e-5):\n",
    "    \"\"\" performs num_iter value iteration steps starting from state_values. Same as before but in a function \"\"\"\n",
    "    state_values = state_values or {s : 0 for s in mdp.get_all_states()}\n",
    "    for i in range(num_iter):\n",
    "\n",
    "        # Compute new state values using the functions you defined above. It must be a dict {state : new_V(state)}\n",
    "        states = mdp.get_all_states()\n",
    "        new_state_values = { state : get_new_state_value(mdp, state_values, state, gamma) for state in states }\n",
    "        assert isinstance(new_state_values, dict)\n",
    "\n",
    "        # Compute difference\n",
    "        diff = max(abs(new_state_values[s] - state_values[s]) for s in mdp.get_all_states())\n",
    "        print(\"iter %4i   |   diff: %6.5f   |   V(start): %.3f \"%(i, diff, new_state_values[mdp._initial_state]))\n",
    "        \n",
    "        state_values = new_state_values\n",
    "        if diff < min_difference:\n",
    "            print(\"Terminated\"); break\n",
    "            \n",
    "    return state_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter    0   |   diff: 1.00000   |   V(start): 0.000 \n",
      "iter    1   |   diff: 0.90000   |   V(start): 0.000 \n",
      "iter    2   |   diff: 0.81000   |   V(start): 0.000 \n",
      "iter    3   |   diff: 0.72900   |   V(start): 0.000 \n",
      "iter    4   |   diff: 0.65610   |   V(start): 0.000 \n",
      "iter    5   |   diff: 0.59049   |   V(start): 0.590 \n",
      "iter    6   |   diff: 0.00000   |   V(start): 0.590 \n",
      "Terminated\n"
     ]
    }
   ],
   "source": [
    "state_values = value_iteration(mdp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*FFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "\n",
      "down\n",
      "\n",
      "SFFF\n",
      "*HFH\n",
      "FFFH\n",
      "HFFG\n",
      "\n",
      "down\n",
      "\n",
      "SFFF\n",
      "FHFH\n",
      "*FFH\n",
      "HFFG\n",
      "\n",
      "right\n",
      "\n",
      "SFFF\n",
      "FHFH\n",
      "F*FH\n",
      "HFFG\n",
      "\n",
      "down\n",
      "\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H*FG\n",
      "\n",
      "right\n",
      "\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF*G\n",
      "\n",
      "right\n",
      "\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF*\n",
      "\n"
     ]
    }
   ],
   "source": [
    "s = mdp.reset()\n",
    "mdp.render()\n",
    "for t in range(100):\n",
    "    a = get_optimal_action(mdp, state_values, s, gamma)\n",
    "    print(a, end='\\n\\n')\n",
    "    s, r, done, _ = mdp.step(a)\n",
    "    mdp.render()\n",
    "    if done: break\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's visualize!\n",
    "\n",
    "It's usually interesting to see what your algorithm actually learned under the hood. To do so, we'll plot state value functions and optimal actions at each VI step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "def draw_policy(mdp, state_values):\n",
    "    plt.figure(figsize=(3,3))\n",
    "    h,w = mdp.desc.shape\n",
    "    states = sorted(mdp.get_all_states())\n",
    "    V = np.array([state_values[s] for s in states])\n",
    "    Pi = {s: get_optimal_action(mdp, state_values, s, gamma) for s in states}\n",
    "    plt.imshow(V.reshape(w,h), cmap='gray', interpolation='none', clim=(0,1))\n",
    "    ax = plt.gca()\n",
    "    ax.set_xticks(np.arange(h)-.5)\n",
    "    ax.set_yticks(np.arange(w)-.5)\n",
    "    ax.set_xticklabels([])\n",
    "    ax.set_yticklabels([])\n",
    "    Y, X = np.mgrid[0:4, 0:4]\n",
    "    a2uv = {'left': (-1, 0), 'down':(0, -1), 'right':(1,0), 'up':(-1, 0)}\n",
    "    for y in range(h):\n",
    "        for x in range(w):\n",
    "            plt.text(x, y, str(mdp.desc[y,x].item()),\n",
    "                     color='g', size=12,  verticalalignment='center',\n",
    "                     horizontalalignment='center', fontweight='bold')\n",
    "            a = Pi[y, x]\n",
    "            if a is None: continue\n",
    "            u, v = a2uv[a]\n",
    "            plt.arrow(x, y,u*.3, -v*.3, color='m', head_width=0.1, head_length=0.1) \n",
    "    plt.grid(color='b', lw=2, ls='-')\n",
    "    plt.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after iteration 0\n",
      "iter    0   |   diff: 1.00000   |   V(start): 0.000 \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAALgAAAC4CAYAAABQMybHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAACyRJREFUeJzt3X1sVfUdx/H36b20a3lqDRUUeWzGQMNACkpkumimuPAg0S0CIbiIQjGGYEx0Ek0gN8Y5dZCYOJkhEP/QGiVkiXYYHdXM6RIRdD5VpCC6qNAVhc260nrP/ri9pZRy7+ntebrffl7NDfT2/u799PixPaf0nK/jui4iVpVEHUAkSCq4mKaCi2kquJimgotpKriYpoKLaSq4mKaCi2nJfA9wHGc1sDrz3tBamBpwJBEv3sF1XSffo5z+/FO948x24Z0BxQpO9vPI+zlHSBn95KXg2kUR01RwMU0FF9NUcDFNBRfTVHAxTQUX01RwMU0FF9NUcDFNBRfTVHAxTQUX01RwMS3v74MHogq4DhgPlAFtwDHgJeCbSBKdaT1Q2cf9TwJfh5zlXOKeMSb5oin4zcAY4BDQCowAJgDDiUfBsz7hzDzfRRUkh7hnjDjfgAvu4DCPeSxnOZvZzKd8mntBOZlyfw883eP+BIHtMJVSyiIWcTVXcy/38p3XrbwfaAomU29VVLGCFZzHeWxik/eFIWacyERWs5r3eI/neM7bohDz9aXggmeLXUcdVVSRJMkEJuQveHvXrRyoAw4DR4BmoKPQNH0rpZTFLGYlK0mSJEGCYQzzXvBLgYk93t/tbz44XewFLMDBoZ32/j1BCBmzxZ7FLIYwhAQJ7wUPIV8uBZ+ydhM3cSd3FvSijZc08tiix/juRz2K9l/gGeDLgp6Svk61SpHiCq6gpJ/fGpauX8rRyqNnf2Bjodmyzs64k52MZCQJEv16prAyjmY09dSTJt2v7RhcvtO8nLJW8FfwV3iFUYxiCUtwujbGC7zAG7yRf/GHMPaTsRyccJD0hDTMAoYBPweeLTTR2baylTRpLudyEiRwcUmRooWWnOtaaQVgTP0Yvm4K9ogoRYq1rGUc4yinnA46WMe6vOuyGavqq/imKbgDlxZaeIiHuJ3bqeh6a6aZR3nUU76x9WNpa2rjm6gOrlzX9XyDWpfM/+LdtxGMcNewxm2gwa3t4+Nn3UpwGd/rvrm4bMTlljxrc97crtvZHxvHOHcTm9yXeMmtpDL/c63vyjN1IHn6l3EmM92tbHV3sMPbc4WcsYQS9zquc5/nefdu7o4w3+mbl876dlZ9CSWkSed/klJgA9ACfEVmv3saUAH8Ffib5zi9ZD+Pc3/X8pwx+yOuenw+QCr+jNHnOy3QXZTePH3SAJ3AW2QOPH4MDAFOAm8Df/crTd88Z4xQ3DPGPV9vui5KqJTRT7ouigx6KriYpoKLaSq4mKaCi2kquJimgotpKriYpoKLaSq4mKaCi2kquJimgotpKriYVsDvg3v/9dpoxD0fKKMfvP06b96v4I7jrHYcZ6/jOHvJcy6jSNzohIdQKaOfdMKDDHoquJimgotpKriYpoKLaSq4mKaCi2kquJimgotpKriYpoKLaSq4mKaCi2nhjxGMyfzEc+rrwu0Tgd8A/wN+F0mqs2k7ehLNnEyIfH6iGdqOOflW8CRJOun0viDi+Ylxpe3orwEXPDvncSELuZ/7eZu3vS2MeH5iXj3zjQj+5WYzm7WsJUmSW7jF+0Jtx5wKLvhIRrKSlSxgAQAuLnOYwylO5Vz3ER/RQQf8pNcH4vYfpne+gNRSyx3cwQVcQDnl/MAPzGBG3nXajt4UXPBruIYbufGM+37d9ZbLUpZylKOk6lPsadpDI42FRghWXwdHAbiP+6iksnsQbIIEW9iSd13P7bipaVP/dmvCFNJ2PJeCC76LXXzFV9RRRzXVJEnyCI/wKq96Wv8ADxT60qbcyq0sYxlLWEIJJbTTzmIWe16v7ZjbgPbB/9H1Npe5LGc5hznsV65B4yQn2cpWnuVZlrGMaqqjjmSKLz9FyRZdCpctuvhLl40IlTL6SZeNkEFPBRfTVHAxTQUX01RwMU0FF9NUcDFNBRfTVHAxTQUX01RwMU0FF9NUcDFNBRfTVHAxTYNgI6GMA6dBsCI6oydcyugnndEjg54KLqap4GKaCi6mqeBimgoupqngYpoKLqap4GKaCi6mqeBimgoupqngYpoGwfZFGQdOg2CJ/wBTZSx6vhV8ClM4yEHSpL0tCHmAaYIEk5jEQQ56X1QMQ1aLIWOEBlzwOcyhjjomM5kUKfawx9vCkAaYJkgwn/ncxm1UUcXN3MwxjnlbHPchqxD/jMU6CLaGGjawoXuAaTvtrGc913JtznUP8zDf8m0oA0yv5Eru4i7KKKOCCgA2spETnIhNxgGLe8ZiHQR7ERcxnvHduyRJkpRRxlzm5lxXRhkAqfoUu5p2sY99hUbIazrTGcYwSnr8sGga0/Ku65nx8abHvX/Fj0CqPsUzTc/wMR9HHaVvxToI9nVe5wM+YCUrmc98EiR4kAdjNQj2CZ7gZV5mDWuYznSSJFnBCo5y1NP6YhmyWkpp1BFia0D74K20spnNPM3TLGIR7/KuX7l800wz93APNdQwj3kc53jUkSREvvwUpZVWdrDDj6cKTHPXmwwyrut6vkGtS+a6AjG8uV23qHOEm7GRRncGM2KdMaibl87qn+rFNBVcTFPBxTQVXExTwcU0FVxMU8GL1FCGMprRAFRTTRVVESeKJ10+OVT+ZdzCFqYxjVJK6aSTDjpYzGI66RzgMxfDdszQ5ZMNa6Sxu8wODvvZ70O57VHBi1QDDXTQAUAHHTzFUxEniicVvEh10MF2tpMmzT728RmfRR0plqI7J1MGrIEGaqllG9uijhJbOsgMlTL6SQeZMuip4GKaBsFGQhkHToNgRXSQGa5Mxn5s8tA53ZsvztsxQweZMuip4GKaCi6mqeBimgoupqngYpoKLqap4GKaCi6mqeBimgoupqngYpoKLqaFe05mTIaD5hX3IavAxC0TOXLiyFn371+zn5ljZkaQqA9jgZ8B44ByoA04BuyFsEYK6aTjXIpgyOrCKQupqarpfr+6ojrCND1cDPyKzD5CC3AAKCNT+umo4LHQzyGroxjFVVzFbnbTRltgsXpadekqlkxd4vnxJ946wffN33P+0vMpSQa0hzoEWEim3O8Du6B7PrADjArmZfsSTcEjHg7qWT+HrM5jHmtZyypWUU89O9kZeNG37d/Ga5+91v3+luu35Hz85w99zvG/HOfQvYeY/PDkYIo+DrrGksLrcMbwa5dQTwyLpuARDwf1rFfOxt2NeZd00kkFFSxnOctYxgY2BDp97sUDL57x/pJfevtqfurLUxyoO8Ch3x5i9juzKR3t4yjCoT3+/m3Xn78gsz+etdG/l8slmoJHPBzUq+r6alJNqe45lIc5nPPxk5jUPRg3TZpWWjnJyUAzbh+3nYv/cPHpOy7J/fi2D7u+ozhAGiqmVuCU+nx6Ws9jlRHAceBz4J/AT/19qXy0D55DCy3UUef58TdwA+tYxxd8wZM8yZu8GWC6jMorKrns1ss8P/79G96n9cVWKq+upOb3NQyfNdz/UF+Q+YlJBXAl8GcyB5knUcGL2W52c4QjsRyImzXlj1M4tfEUwy8NoNhZHUADcCOZ45gLgH8BI4N7yXNRwX3UTnusyw1QdmEZZReWBf9CH5D5ij2PzEHnTDK7LgeBD4N/+SxdNiJUumyEn3TZCBn0VHAxTQUX01RwMU0FF9NUcDFNBRfTVHAxTQUX01RwMU0FF9NUcDFNBRfTVHAxTQUX0/r5++DOf8hcLSSuRgH/jjpEHsrojwmu6+a9CEx/z+j5xHXd2QUGCpzjOHvjnA+UMWzaRRHTVHAxrb8F/1MgKfwT93ygjKHq10GmSLHRLoqYpoKLaSq4mKaCi2kquJj2fyEM3HQ7FTfGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 216x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after iteration 1\n",
      "iter    0   |   diff: 0.90000   |   V(start): 0.000 \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAALgAAAC4CAYAAABQMybHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAC7tJREFUeJzt3X1sFHUex/H3dLfda7Fsi+WxFnqgiE8n8iQBzUWjeAZQgt4doMHLEbEmxnAxF04OL+DGeD7iH3dyGIycMbZEDZqg59NRjRpMRFFRryCUA4UCtRQRyrWlnftjuy1tt7vT7ezO7I/Pq9nY3e7sfjp+2M60O/O1bNtGxFQ5XgcQSScVXIymgovRVHAxmgouRlPBxWgquBhNBRejqeBitGCyO1iWtRRYGr02aDJMSHMkESc+xbZtK9m9rP78qd6yptjw6YBipU/s+0j6PXtIGd3kpODaRBGjqeBiNBVcjKaCi9FUcDGaCi5GU8HFaCq4GE0FF6Op4GI0FVyMpoKL0VRwMZoKLkZL+n7wtCgGZgGjgRDQBBwBXgcaPUnU3TKgKM7t/wAOZThLX/ye0Sf5vCn4b4ERQC3QAAwGxgCF+KPgMTvpnuekV0ES8HtGj/MNuOAWFjOZySIWsYY1fMu3iRfIJ1ruU8DzZ9weIG0bTHnkMZe5XMM1LGc5J52u5e1ATXoy9VRMMbdzO0MYwmpWO18wgxnLKWcpS/mCL9jIRmcLZTBfPCkXPFbsCiooppggQcYwJnnBmzsu+UAFsBfYB+wBWlNNE18eedzETSxmMUGCBAhwDuc4L/gVQPkZ1990Nx90FXs2s7GwaKa5fw+QgYyxYk9iErnkEiDgvOAZyJdIyoes3cIt3MM9KT1p9SXVPDH3CU7+7IyinQBeBA6m9JDEO9QqQoQZzCCnnz8aFixbwOGiw72/sCrVbDG9M77CK4QJEyDQr0fKVMbhDKeKKtpp79d6TF++Lk4OWUv5Ffwd3qGEEuYxD6tjZbzMy3zIh8kX/hpKd5aye8xu2se0wyTgHOCXQGWqiXpbxzraaedKriRAABubCBHqqU+4XAMNAIyoGsGhmvTuEUWIcDd3U0YZ+eTTSiv3cm/S5WIZi6uKaaxJ345LPfU8zMPcyZ0UdHzsYQ+P87ijfKVVpTTVNNHo1c6VbduOLzDZJvpPvPMymMH2Xdxlv8Eb9uQ4X+91ycFmdI/bpmOzCps7kiyb8GJ3XHp/rYwyezWr7dd53S6iKPljLevIM2EgefqXcSIT7XWsszewwdljZThjDjn2LGbZL/GSfR/3eZiv6+Kks64dVZ9DDu20J3+QPGAFUA/UEd3uvggoAP4NfOA4Tg+x76Pvn1qOM8Z+xVWFyztI2Z/R+3xd0rqJ0pOjbxrgNLCV6I7HBUAucBz4BPjIrTTxOc7oIb9n9Hu+nnRelIxSRjfpvChy1lPBxWgquBhNBRejqeBiNBVcjKaCi9FUcDGaCi5GU8HFaCq4GE0FF6Op4GI0FVyMlsL7wZ2/vdYbfs8HyugGZ2/nTfoKblnWUsuytlmWtY0kxzKK+I0OeMgoZXSTDniQs54KLkZTwcVoKrgYTQUXo6ngYjQVXIymgovRVHAxmgouRlPBxWgquBhNBRejZX6MoE/mJ/Yp3onby4HfAf8D/upJqt60Hh3xZk4meD4/0Rhajwm5VvAgQU5z2vkCHs9PNIbWY0IDLnhszuMc5rCSlXzCJ84W9Hh+YlJn5hvsYY5ktB4TSrngYcIsZjGzmQ2Ajc1UptJCS8LlvuEbWmmFC3t8wW//Y3rm8yutx4RSLvi1XMt85ne77dcdH4ksYAGHOUykKsKWmi1UU51qhPSKt3PkQ5GqCOtr1rOPfV5Hic/j9ZhywTexiTrqqKCCoQwlSJDHeIx3edfR8g/wQKpPLT0UUeTfgntsQNvgH3d8TGc6i1jEXva6lUvEFa78FiVWdBHfGegob/9c+h6T7Z+L+xmrqbYv53JfZ0zXxUln9ad6MZoKLkZTwcVoKrgYTQUXo6ngYjQVXIym0ydnlHsZV7KSaUyjkELaaOMAB1jCkv69ZTmubFiPUTp9ssHqqSdECIAAAU53fEh3KniWqqSSdtoBOMUp1rLW40T+pIJnqeMc51VepY026qhjG9u8juRLKngWq6SSYxzjaZ72OopvaSczo5TRTdrJlLOeCi5G0yBYTyjjwGkQrIh2MjMrmvHgwTqPc/Rt1KiRHZ/5eT1GaSdTznoquBhNBRejqeBiNBVcjKaCi9FUcDGaCi5GU8HFaCq4GE0FF6Op4GI0FVyMpkGw8WRBxmkvTuP7E9/3uv3t+W9zacmlHiTqQYNg8f8A0yzIeN3o6ygfXN55/dz8c70L40OuFXw849nN7s5zdSSVDQNMsyDjwgkLubH8Rq9j+NaACz6VqVRQwVjGEiHCFrY4W9DvA0whKzJW1lSy9eDWzusPznjQwzRxZOsg2HGMYwUrGMlI8smnmWaWsYzruT7hco/wCMc45v8BppAVGd/d331so+8Knq2DYM/jPEYzunOTJEiQECGmMz3hcrHz6UWqImyq2cRnfJZqhLSLVEVorGnkSZ70OkqfIlURFj22iPzJ+V5HiS9bB8G+z/t8xVcsZjE3cAMBAjzEQ8YNgh3KUK8jJGU3+/0IeO8MaBu8gQbWsIbneZ65zOVzPncrl4grXPktSgMNbGCDGw8l4iqdNiKBaqr5mI+5n/tdekT3Txuxc9ROyl4uo2BGgSuPp9NGiGQRFVyMpoKL0VRwMZoKLkZTwcVoKngcueQynOEAhAkzjGEeJ+qt7Xgbrd+1AtBa18rpeo0QjEcFj2MhC3mBFwC4iIvYyEbO53yPU3V34I4D1F5dC8ChPxyidnotdov+ZN+TCh7HR3zUbahqI43sZa+HiXorvLkQK9jxd452KLiqACvP/3+cyTQVPI497GEHO2ijjSaaWM962mjzOlY34YVhrFC00FaexdAV/n9TmBdU8D48wzO0004zzbzFW17H6SUnlEPJH0sgBwquLiB0YcjrSL7k3TGZPreb3bzGa3zJl7579Y4JLwzT9EETJctLvI7iW3qzVUZpRo+b9GYrOeup4GI0DYL1QNdmgJ/5fT1qEKyIeTuZ2bAD149VnnFW5wujdjJFfE8FF6Op4GI0FVyMpoKL0VRwMZoKLkZTwcVoKrgYTQUXo6ngYjQVXIymgovRMntMpk+Ggybj+yGrQPlT5ez7cV+v27fftZ2JIyZ6kCiOUuAqoAzIB5qAI8A24D+ZiaCDjhPIhiGrc8bPYVzxuM7rQwt8cvqIi4FbiW4j1AO7gBDR0l+GCu4H/R2y2lrXyk+bfyK8IEygMJDGZF2WXLGEeRPmOb7/j1t/5NSeUwxbMIycYJq2UHOBOUTLvQPYBJ3zgS0ggycB8KbgHg8Hdaq/Q1ZPvHWC+gfr+eHRHxhy9xCK7yxOe9Gf3f4s7/33vc7rT/3qqYT33//wfo7+6yi1y2sZ+8jY9BS9DIhNVHkfug2/tsnogWHeFNzj4aBO9RyyetuttyVfKAD2SZujfzvK0b8fpfSfpQy6alCaEsLmXZu7XZ93o7NX85aDLeyq2EXtn2qZ8ukU8obnuRfqzG/3WMd/ryO6PR6zyr2nS8Sbgns8HNSpdZes47LIZV1zKJP8w2zZ2RL9sdwGBCA4LEhgSHpfwZ8re46Ln7y464ZLEt+/6eum6CcW0XMaTkjDOQ1PnvH5YOAosB/4EviFu0+VjLbBEwiWBil/s9zx/Rs3NHLkz0fI/Xkuw/4yjEGzBmFZ6T22sWhGEdN+P83x/XfcvIOGzQ0UXVPEuEfHUTip0P1Q3xH9jUkBcDXwGtGdzOOo4Nks/JswoQtC5M/IT3uxUzV+7XhaVrVQeEUaih3TCrwBzCe6vzUS+B4Ip+8p+6KCuyinIIeCme7Mq0yX0KgQoVEZOFHnV0RfsWcS3emcSHTTZTfwdfqfPkanjcggnTbCXTpthJz1VHAxmgouRlPBxWgquBhNBRejqeBiNBVcjKaCi9FUcDGaCi5GU8HFaCq4GE0FF6Op4GK0fr4f3PoJ2Jm+OANWAvzgdYgklNEdY2zbTnoSmP4e0bPTtu0pKQZKO8uytvk5HyhjpmkTRYymgovR+lvwZ9KSwj1+zwfKmFH92skUyTbaRBGjqeBiNBVcjKaCi9FUcDHa/wFaHQy4jc/FPgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 216x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after iteration 2\n",
      "iter    0   |   diff: 0.81000   |   V(start): 0.000 \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAALgAAAC4CAYAAABQMybHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADG1JREFUeJzt3X9s1PUdx/Hn93rt0RYolVKQiq02AuPHpFiRgLpBiBFFaVA2QMPmD5AlLpMtDrPpIHYJm+j0D5dZweiWKDUh0SVsUyTgnAszYDrG0IJQx88C5UctUKCl990f15b+uPau1/ve9/v98Ho0DXfHfe/e/fbFl8/37j6ft2XbNiKmCrhdgIiTFHAxmgIuRlPAxWgKuBhNARejKeBiNAVcjKaAi9GCse5gWdZSYGnkWvYtMNbhkkTi8Tm2bVux7mX15a16yyq14fN+leWctp8j5s/sItWYTPEEXEMUMZoCLkZTwMVoCrgYTQEXoyngYjQFXIymgIvRFHAxmgIuRlPAxWgKuBhNARejKeBitJifB3dELnAXcD0QAhqBE8BfgDOuVNTZU8CQKLe/BhxLcS098XqNHqnPnYB/HxgB1ACngMFAITAIbwS8zR4613PerUJ64fUaXa6v3wG3sJjOdBaxiJd5ma/4qvcNMomE+wLwpw63p+HYgCmDDO7jPmYwgxWs4Hy8e7kKqHampqRJYY1FFLGUpexkJ+/ybnwbubwPEw54W7CXsYxccgkSpJDC2AG/1PqdCSwDvgYOAPuB5kSriS6DDO7nfhazmCBB0khjIAPjD3gJUNTh+gfJrS8pUlBjW7AnM5l00kkjLf6Au7wPE56y9gAP8CRPJvSkW8dv5aX7XuL8gA5BOwe8AxxN6CGJNtWqnHKmMY1AH/9rWPDUAo4POd79L1YlWlubJE4H62mMu6q/D9y5xuEMp5JKwoT7tB+d24dXxDNlLeEj+Ed8RB55lFGG1bozNrCBT/k09sa7oWBPAfsK9xEuDMNkYCDwHWB9ohV1V0EFYcLcxm2kkYaNTTnl1FHX63anOAXAiMoRHKv2whlbz8ory9lQvYGd7HTk8euoYzWrWcISslq/9rOfF3mx1+3a9mFBZQGN1Y2ccenkKuGAN9BABRWsZz0LWchc5lJFFdWxBlwB4DrgIJFhyX4ir6LcDWQkWk10hznMSlYyilE8zuOUUsoudlFPfVzbH/PEyxHuChNmE5vYzGZmMYslLOFLvoz9e251hCMOV9i7fp9ktgV9LWsJE47vGR8F6oBaIuPub7X+XU1/q4nuEIdYyUoCBOKrUbrpGHQ/7cOkvUwY9w99GdhG5MTjJiAdaAC2A/9MVjXR+ekX41W+24e2bcf9DbfYRM5CPPhtt367XUdqa9zKVvtmbvZ0jU59x5NZvVUvRlPAxWgKuBhNARejKeBiNAVcjObOx2Wl3+Yzn1JKAVjOcnazmxd5ERvb5cq8RQH3qQlMaA94IYUMZjABArTQ4nJl3qIhik+tYx3NrZ8vbqSRdaxTuKNQwH3qEIf4jM8IE+YSl/iQD90uyZMUcB9bx7r2P3X0jk49elIq+TXmk89JTibxQ1B+2I8Rjk54EG84wQm3S/A0DVHEaAkcwb3+OqvX6wPVmAzxDaFiHsEty1pqWdYOy7J2EGMuo4jX6CQzpSI1VlX92+U6elZSMqn1kpf3Y4QawcpVTwEXoyngYjQFXIymgIvRFHAxmgIuRlPAxWgKuBhNARejKeBiNAVcjKaAi9FSP6PHI/0Te9RWXyVXuoMVAT8ELgK/caWqbu7ZdA+1F2q73V753UrG5IxxoaIuPLIf3Zuy5vX+jj5x5/A7uS77uvbruRm5LlbjPUkLeJAgl7kc/wZ+6EHpA2WFZcy4dobbZXhWvwOeSy4P8zBzmMOzPMt2tse3odd7UHasb7CLdcTw/oH32XFyR/v1pyc+7WI1Ubi8HxMOeA45LGYx93IvADY2t3IrTTT1ut0XfBFZkanrMNFrAffAMDYenxz/pNN1zwXc5f2YcMBnMpN5zOt02/zWr94sYAHHOU55ZTlbqrewla2JluCsaCdHHlReWc7sX88mrTjN7VKic3k/Jhzw93iPWmpZxjKGMYwgQdawhs1sjmv753gu0aeWLsJnwqTh0YC7rF9j8H+1fk1lKotYxNd8nay6RJIiKa+itAVdxGu0bERKJX/ZiPqSerLXZpNemp6Ux9OyESI+ooCL0RRwMZoCLkZTwMVoCrgYTQEXoyngPnX+mfPU31kfufzEeRrKGrCbvb5ofeop4D4VyA9EZsYAhIF01HEpCgXcp0KPhGj/fFUmZP40E8vy/ruPqaaA+1QgN0BofggCECgIEJyqw3c0CriPhR4JYeVaZP5MR++e6J+9jwVyA+RsznG7DE/TEVyMpoCL0RL4PPiO2HcUcZyVnM+DqxGs+JlxM3r80GT16NHuS655xciR17Ze8v6rMprRI1c9BVyMpoCL0RRwMZoCLkZTwMVoCrgYTQEXoyngYjQFXIymgIvRFHAxmgIuRlMj2Cg832QVmPLOFA6fO9zt9k3zNjEhb4ILFXWhRrB4vhGsH5qszrp+FkWDi9qvD80c6l4xHuRewBNoBDua0exjH2HCjpTUlR+arC4cu5DZRbPdLsOz3At4HxvB3sANVFBBHXW8zutsYYvjQfd8k1VgffV6th3d1n79+WnPu1hNFH5tBNtvXYayqz9Y3evdhzOcMGGGMYzlLGcpS1nDmvg7KyfA801Wgc0HO7dt9FzA/doItr+GVg5lQ/WGhLbNaP0axzhHA15eWc7M8TPJejbLsefor/LKchatWUTmLZlulxKdXxvB9tcpTjGD+Me3xRTzGq/RQgvb2c5a1nKQgw5WGBE+lprxfn/Yl7SqbE98s7LVEY7wNm+zhS0pCbaYwTcBv8hF3uItt8sQn9GyEb2oL6knOD3IwFcHJuXxnFg2Ys/IPYzaMIqsack5T9CyESI+ooCL0RRwMZoCLkZTwMVoCrgYTQGPwr5k03K0JXK53iZc6713M1saWmg+1AxAc20zl+suu1yRNyngUVx88yJn7z8LQMvuFhruaeBytbcCdOQHR6i5owaAY8uPUTO1BrtJb9l3pYBHkT4jvdN7vFauRVpxWs8buGDQ3EFYwdb3OcKQdXsWVob335xJNQU8iuCYIMGSYGTvZMKAHw/ASvdWeHIW5mCFIjVZGRbDfjHM5Yq8SQHvwYCfDIAAWAMsMuZkuF1ON4FQgLyn8yAAWXdkERoTcrskT/LNh61SLTg2SMb8DIKTg547erfJWZhD4z8ayVuR53YpnqWA9yLr596d6ACRo3jBGwVul+FpGqKI0RRwMZoawYpPqRGsiHkzevzQZLUPuzzlrPZjojdfOepIM3rkqqeAi9EUcDGaAi5GU8DFaAq4GE0BF6Mp4GI0BVyMpoCL0RRwMZoCLkZTwMVoqZ2y5pHmoLF4vskqUPRKEQe+OdDt9qonqpg0YpILFUVRANwOjAIygUbgBJEpBV+mpgTNyeyFH5qszhk9h+Lc4vbrw7I8snzEOOBBImOEOmAvECIS+oko4F7Q1yarzbXNnN14lpwFOaQNSs1CQY+VPEbZ2LK47//Ntm+4sP8C+QvyCQQdGqGmA3OIhHsX8B60tzS1gBQuAuBOwF1uDhqvvjZZPffhOeqer+PkCye55kfXkLsk1/Ggv1H1Bh//7+P266/c/Uqv9z+4+iCn/3aamhU13PjbG50J+iigbUGCv0Onfr02KZ0Y5k7AXW4OGq+uTVYfevCh2BulgX3e5vSrpzn9+9MU/LGA7NuzHaoQNu7d2Ol62ez4juZNR5vYu2wvNc/UUPp5KRnDk7i4Uccft771z1lExuNtViXv6XrjTsBdbg4ar4rxFUwsn3ilD2WMf5hNe5oi/y23AGkQzA+Sdo2zR/A3R73JuN+Nu3LD+N7v37i7MXLBIrKm4VgH1jQ83+HyYOA0cBD4D/Dt5D5VLBqD9yJYEKTog6K473/mrTOc+OUJ0m9IJ/9X+WTflY1lOTu3cci0IUx5dErc9981dxenNp5iyIwhFL9QzKDJg5Jf1CEir5hkAXcAfyZyktmAAu5nOd/LIXRTiMxpmY4HO1Gj/zCaplVNDCpxINhtmoG/AvOInG9dCxwGcpx7yp4o4EkUyAqQNd3by72FRoYIjUzBQp3/JXLEnk7kpHMSkaHLPmC380/fRstGpJCWjUguLRshVz0FXIymgIvRFHAxmgIuRlPAxWgKuBhNARejKeBiNAVcjKaAi9EUcDGaAi5GU8DFaAq4GK2Pnwe3zgJ7nCun3/KAk24XEYNqTI5C27ZjLgLT1xk9e2zbLk2wIMdZlrXDy/WBakw1DVHEaAq4GK2vAX/dkSqSx+v1gWpMqT6dZIr4jYYoYjQFXIymgIvRFHAxmgIuRvs/bJg/5kgS9FwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 216x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after iteration 3\n",
      "iter    0   |   diff: 0.72900   |   V(start): 0.000 \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAALgAAAC4CAYAAABQMybHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADN5JREFUeJzt3X1wFPUdx/H3Xi65PBRDJERNFIIRRMVqAB9G1IpCBhwtjA8t4DBWHRBm6NTSsTptpzKmM7RWK+PUQWvVKjMSH2bqA1WMzGh9GEcLpUJRUEkVFSSYgEFQktxt/9gkhOSSuyS3+9v78Xk5DrfHLvu95cNmb+9+v6/jui4itoqYLkDETwq4WE0BF6sp4GI1BVyspoCL1RRwsZoCLlZTwMVq0VQrOI6zEFjoLRVNgvE+lySSjg24ruukWssZyEf1jjPZhQ1DKss/na8j5Ws2yKuxvv4Vw3X0raZmesejMB9HTzoB1yWKWE0BF6sp4GI1BVyspoCL1RRwsZoCLlZTwMVqCrhYTQEXqyngYjUFXKymgIvVFHCxWsrvg/uiBKgBRgEx4CDQCPwD2GukoiPdAgxP8vwDwJcB19KH+Rvms/vQ7l7PrzxrJVVFVQYq6iEkx9BMwH8MHA80AE3AMcBoYBjhCHinbRxZzwFThfTtvJLzKM8v71oujhYbrCYJw8dwyAF3cJjCFOYxj3u5l4/4qP8NCvDC/S3weLfncwjfBdNGYGv6q09gAotZzGM8xru861tZ3c0om8GUEVMC2VcllSxkIe/xHk/yZHobDfAYZtqgA94Z7EUsooQSokQZzejUAT/U8X8BsAj4H/ApsB1oG2w1PqkGKrstr+1/9SqqGMc47uAOGmlkJSt9D/raxrVsatnUtbx4zOKM76Mz2BOZSC655JCTfsAHeAwzbdBD1q7mapawZFA7ffWMV7nnyns4kN/t59U3wBPAzkH9kWR0yFof14+vLns15aYJEkS6/ShaznLqqe9YytyQtb6uwesvqE+ydvp6Dlk7juOoo67X60plzi1z2D28d30sG1J5R0hnyNqgz+Cv8AqllDKb2TgdB+MZnuFN3ky98Rao2FbBx6M/JjE6AROB7wE/AFYPtqLMi9ZFOWXrKV3Li+n/7Did6VzJlbTSiovL0zzNW7zla421dbWcf9P5uGf5Mw32HvawnOUsYAGFHf9tZzt3c3e/2zXRBEBFXQUHtx5kr6E3V4MOeAstPMiDrGY1c5nLLGaxkY1sTXXBFQFOBHbgXZZsx7uLMgPIG2w1/minPfXr6WYkI5nJTJ7maZ7iKQ6E8V3pACVIUE8961jHNKaxgAV8wAdpH5cv+MLnCvs35DeZnUF/iIdIkEhvjzcCe4BdeNfdp3X8XsNQqzHrDd7gLd5K7zhkme5Bz6bXl7HbhGm/6Hbgbbw3HmOBXKAF+Bf4/NM8ENn0lz8Y2fb6gr8PngBeDnyvA7PCdAGprZq0CoDcZbm00264miRCcgzDdudZJKMUcLGaAi5WU8DFagq4WE0BF6uZ+bqsDFnkmQjOBu8rEjn35ZA4LUFiaUKnrB4U8CzlbHFw/u0F3PnMIdISIZFQwHvS4chS8Rvi3qfAgJvvEr8xrtNVEgp4thoF7jkuruNCDNzp/nybMNsp4FksfkPc+1Vn7z7psGSzUdC+qh1GmC4kvBTwbFdmuoBw0yWKWG0QYzLX+1iOSLqczLQRdBxnoeM46x3HWe8NwxHJHmoEGyivxo0b/2O4jr5VV5/d8SjMx9GjRrBy1FPAxWoKuFhNARerKeBiNQVcrKaAi9UUcLGaAi5WU8DFagq4WE0BF6sp4GK14Ef0hKR/Yp8666vjcHewSuAnwHfA741U1cvl9Zez69tdvZ6vu6SOU4tPNVBRDyE5juaGrGVBD8pscPFxF3Ni0YldyyV5JQarCR9zATfcP9EWs0fPZuoJU02XEVrmAm64f2JK3es7xmAdKTz76bOs/+rwMMJbz7zVYDVJGD6O5gLe8zIxbAEPwWVsOl7f/foRy6ELuOHjaCzgNXU1zNg6g6UsNVVC/5K9OQqh2rpaZv5uJjlVOaZLSc7wcTR2m3Ac46im2tTurZLYm12dz4Kk++BiNQVcrGZs2ohHeIQxjGEqmbrFdXROG7Gveh9FDxWROzk3I3+epo0QySIKuFhNARerKeBiNQVcrKaAi9UUcLFa4AEfy1ie4znGMAaA53mey7gs6DKy3oHbD7Dv4n3e45sP0DK7BbdNndZ6CjzgzTRTQEHXch55NNIYdBlZL1IW8UbGACTwemaq41IvgQe8iSZe4iVaaSVBgu1sZzObgy4j68VuiEHnFwgLoGBpAY4T/k8fg2bkGvxxHsfFpZVWHuABEyVkvUhJhNi1MYhApCJC9HydvpMxEvAmmniBF3if93X2HoLYDTGcEoeCX+js3Rdj/+zv535Tu7ZGpCRC8bpi02WEmm4TitUUcLGaGsFKllIjWJGBn8Hr65f7WM7g1dRMB7KjyerOnb2nXAuL8vITOh6F/66MRvTIUU8BF6sp4GI1BVyspoCL1RRwsZoCLlZTwMVqCrhYTQEXqyngYjUFXKymgIvVAh+yNn/DfHYf2t3r+ZVnraSqqCrocpIKfZNV4NwnzuXzbz7v9Xz9VfVMKJ1goKIejvZGsOeVnEd5fnnXcnE0fGMLs6HJ6rRR06g8prJreUTBCHPFhJCxgM8om8GUEVPS38AFPgKqODwfiM+yocnq3PFzmVk503QZoWUs4Gsb17KpZVPX8uIxi/vf4BPIXZKLO8IlflMcd6rre9BD32QVWL11NW/vfLtr+c4L7jRYTRJHayPYd/a+c8Tykr8u6Xd9p9HBdVycJoec+3LgYYgvjeOe4998fKFvsgqs27HuiOXQBfxobQR7R+kdXLLkksFt3Aq0gfOB42vAa+tqufSMSyn8TaFv+xiq2rpa5v1xHgWTClKvbILhRrDm5vsaAW31bemvvx2iS6KQA+4k7zKFUf6V1ynxZfibrLqHNKtsX7JnQrtySMxJkJiaCCTYYofsCXgBJK4P/9lUwiXwgK+atCroXQ7YizUvArBv2T4YwJ3MIL07710Ati3bZriSPqxI8twnwLJgy9BH9WI1BVyspoCL1RRwsZoCLlZTwMVqCngS7iGX+M6493ifS2JX+O6/x1vitH3mfRLctquN9j3thisKJwU8ie8e/Y79P9wPQHxLnJbLW2jfGq4AfXH9FzRc1ADAlz//kobzG3Bb9ZF9Twp4ErlTc4/4CMwpccipCuhL6GkaNmsYTrRjeuwEFF5YiJMX/jm9g6aAJxE9NUq0OuodnQLI/2k+Tm64wlM8txgn5tXk5DmM/NVIwxWFkwLeh/yf5UMEnHyHvCvyTJfTSyQWofTWUohA4UWFxE6NmS4plLLny1YBi46PkndtHtGJ0dCdvTsVzy3m4BsHKb2t1HQpoaWA96Pwl+Ed6ADeWbzi4QrTZYSaLlHEagq4WE2NYCVLqRGsyGDO4Bt8LGcovNeRDU1WB3DIA+d0nRPDeeeoOzWClaOeAi5WU8DFagq4WE0BF6sp4GI1BVyspoCL1RRwsZoCLlZTwMVqCrhYTQEXqwU7ZC0kzUFTCX2TVaByRSWffv1pr+c33ryRs48/20BFSVQAFwInAQXAQaARb0jBB8GUoDGZ/ciGJqtXjLuCqpLDHaJHFoZk+ojTgWvwrhH2AB8CMbzQn4kCHgYDbbLatquN/Wv2UzynmJxhwUwUdFP1TcwePzvt9b9++2u+3f4tZXPKiER9ukLNBa7AC/dm4O9A5+x3DhDgJABmAm64OWi6Btpk9ZuXv2HPnXv46q6vOHbxsZQsKPE96A9vfJjXPnmta3nFjGS9Qw7bsXwHzS8103BbAyf/4WR/gn4S0DkhwT85HG7wxqUEODDMTMANNwdNV88mq9ddc13qjXLAPeDS/Odmmu9vpuKxCoouLPKpQljz4ZojlmfPTO9s3rqzlQ8XfUjD7Q1M3jCZvOMyOLlR95e7r+PXaXjX452WZW53/TETcMPNQdP14BkPcmbtmYf7UKb4h9m6rdX7sRwHciBaFiXnWH/P4I+e9Cin/+n0w0+c0f/6B7cc9B44eHMajvdhTsMD3R4fAzQDO4BNwPczu6tUdA3ej2hFlMq1lWmvv/dve2n8dSO5Y3Ip+20ZRTVFOI6/YxuHXzCcc288N+31N8/aTNOaJoZPHU7VXVUMmzgs80V9hnfHpBC4CHgO701mCwp4Niv+UTGxsTEKLijwPdiDNW7lOFqXtTKs2odgd2oDXgSuwnu/dQLwOVDs3y77ooBnUKQwQuGUcE/3FiuPESsPYKLO/+Kdsafgvek8G+/S5WNgi/+776RpIwKkaSMyS9NGyFFPARerKeBiNQVcrKaAi9UUcLGaAi5WU8DFagq4WE0BF6sp4GI1BVyspoCL1RRwsZoCLlYb4PfBnf3ANv/KGbJS4CvTRaSgGjNjtOu6KSeBGeiInm2u604eZEG+cxxnfZjrA9UYNF2iiNUUcLHaQAP+F1+qyJyw1weqMVADepMpkm10iSJWU8DFagq4WE0BF6sp4GK1/wOk8GDB0JiwtAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 216x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after iteration 4\n",
      "iter    0   |   diff: 0.65610   |   V(start): 0.000 \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAALgAAAC4CAYAAABQMybHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADVpJREFUeJzt3X9wF3V+x/Hn7vf7zTc/CN+ExIhBIVxOpXBWAUELinBVBqi9YxyvFa1DldaTGZy72ipXbHsZ0xn12uk4d3UYnXPsHR3JHzdz1471OM7pcHe9cbiCdKB4YiUqKEiAEBPy85vvd/vHkoSQb/JNSHY/+/3weswwZJf9zr6zvPLJfve7+3k7nuchYivXdAEiQVLAxWoKuFhNARerKeBiNQVcrKaAi9UUcLGaAi5Wi+fbwHGcx4DH/KWyxTAv4JJE8ps27QgdHR1Ovu2ciXxU7zi3erB/UoUFx/8+du5sMlzH6DZseACA3bt/briS0a1efQ8Q7eMIsG3bNpqbm/MGXKcoYjUFXKymgIvVFHCxmgIuVlPAxWoKuFhNARerKeBiNQVcrKaAi9UUcLGaAi5WU8DFannvBw9EJbAamA0kgS6gBfgP4JyRioZ54ugTnOk/M2L9c3Oeo664LvyCcnh4/8Oc6j01Yv32m7dTX1ZvoKLhonIMzQT8j4GZQDNwFpgOzAHKiUTABywqW0RNUc3g8vTYdIPV5HZb5W3UFtcOLqfiKYPVjGT6GIYf8BL8cHcDP7xofYzInTCtTK1kSfmScW+feC9B+b+Wc/7+8/Td3Ad5b8efvDU1a1hetTz4HQHx43Gm7ZxG37w+ur7SNa7XTPQYTrXwA9574U8J8DjwIfAxcBRIh17NmPZ8vod3u98dXN5Ys3HM7ePH4iQ+TFDxYgWZ6gwdf9IReNB3teziYPvBweXNczdP+T7ix+NMe30aycNJ//8ow7gDPtFjONXCD3gW+HeI/2Gc/pn9/mj+e8B54HXgROgVjeqdznegc2h56ze25n2N53i4vS7upy4zXphB2+Nt9NzVE1iNe8/tHbY81QF3T7tUP12Nh4dz4Se1+GAxMzfMHPN1sW/GoGLkMbQ/4ACH4aUjL9E2p42tc7bCImAacBew00hFOT1Z8yTLzi4bXD7beHbM7Yt/VUzpf5aSdbPgQOe6TnqX9AZaY2NTI7dvuh3v5mCmwc5WZWl7vI3ypnKcbge31yV9XZr2x9rHfl0mC8DT7tMsvmox2YpsIPXlE37AXeBaSBxLsPToUv/UpAtYAxSFXs3Y4pD+4vjPm9xWl5JflNC5rpOuP+jCK7Ng7nUXeu7qoeeOHor/q5jypnLSX0znPS7eUQ/6oX9mP9lyM+EGEwGPA4/CM6efYf7J+f453e9c+Lfm0KuZUr1Le2lZ3OK/YbZNbCjoUbsYMJbwA94PvA2JugR7r98LCaAd+G/g16FXM/VsDPfFCuz7M/Mm82fQQANzmcsqVoVeQj7fq/+e6RLy2rF4BwCJhgT99BuuZqSoHMMC+mUjMnEKuFhNARerKeBiNQVcrKaAi9VCD3g11TzP88xlLgAv8AILWBB2GQXP/ZFL7K/9i9Kx78Zw/9H1L8HKMKEHPEWKJQzdPrmIRVzN1WGXUfCcww7OO/7NT85xB3evAp5L6AE/ylH2s58MGQBaaWUPe8Iuo+BlHsn4nwIDXrFH5tGMqVvnIs3IOfgrvEKaNN108zIvk9XQM3GzwVvi4TkeJMG7x4IbuwJgJOAf8AGHOEQ77Rq9JyHziP9bUKP36IwdlgYaiBPX6D0Zs6F/Rz9UmS4kuowFvIvxPfIkedTk3+RKpuvgYrXLaCO4L8ByRMbLwfO8ybcRdBznMcdx9jmOsw9OT01tIiGZ8Ai+c+dfBVjO5RtoshrKZCSXzT/WBw78j+E6Rrdw4S0XvorycfRNyQguUsgUcLGaAi5WU8DFagq4WE0BF6sp4GI1BVyspoCL1RRwsZoCLlZTwMVqCrhYLfQneqLSP3FU3wQqgCbgvQvr6oA/BXqA541UNcK63es42X1yxPqmlU3cmLrRQEWXiMhxNPbImun+ibZYcfUKri27dnC5sqjSYDXRYyzgpvsn2mL9nPWsuiZ6TQSiwljATfdPzGsh/q9U8DsxR9RPPv4J+84MPUb41E1PGawmB8PH0VjATfdPzCsCp7Hj8ctTvxy2HLmAGz6OxgK+7YNtrHhnBef+NkLN6S+W681RBDU2NbL279cSq49odyjDx9HYZcLEhwmS7yZN7d4q2XOaPGk0ug4uVlPAxWqhn4MP9E+s+n5EJ9R7Mce6j4CGcMvI583VbwLQ1tBmuJJRROQ4agQXqyngYjUFXKymgIvVFHCxmgIuVlPAxWqhBzz+YZyaP6sh8YnfA69mUw3Fvy4Ou4yC1/mtTtpW+NfAO7/eSfv6dry0Oq1dKvSAZ1NZnN6haZ2dtEOmKhN2GQXPrXH9J2PAbwCbQJ3Wcgg/4DOydN/VjRf38PBIz0mTnpcOu4yCl3wkCQM3EJZAyZMlOE70J60Pm5Fz8PP3nfcbCBRBx0MdJkooeG6lS/JrSXDBneUSv13Ddy5GAp6dkaXr97vou75Po/ckJB9J4lQ6lPylRu/RGPux79iokXuy3EqX1Fsp02VEmi4TitUUcLGaGsFKgVIjWJGJj+C7dz8XYDmXb/Xqe4DCaLJ64sTIKdeiorb2mgtfRf+qjBrByhVPARerKeBiNQVcrKaAi9UUcLGaAi5WU8DFagq4WE0BF6sp4GI1BVyspoCL1UJ/ZO3h/Q9zqvfUiPXbb95OfVl92OXkFPkmq8DS15fyyflPRqzffd9uvlT9JQMVXeJKbwR7W+Vt1BbXDi6n4tF7trAQmqzePftu6qbXDS5XlUS0sYAhxgK+pmYNy6uWj/8FHvB/QD1D84EErBCarG6Yt4G1dWtNlxFZxgK+q2UXB9sPDi5vnrt57Bd8BIktCbwqj8ymDN4qL/CgR77JKrDzvZ28feLtweVnlz1rsJocrtRGsHvP7R22vOX7W8bc3mlx8BwP56xD7LsxeBUyT2bwlgQ3H1/km6wCbx17a9hy5AJ+pTaC/Xb1t1m5ZeXlvbgPSIPzWyfQgDc2NfLlBV+m9G9KA9vHZDU2NfLgPzxIyeIS06XkZrgRrLn5vqogvXsCs1odhfiWOMTAW+yfpjA7uPIGZD+LfpNVr1ezyo6mcCa0q4XsA1myq7KhBFvsUDgBL4HsxuiPphItoQd8x+IdYe9ywoY1WZ3Alcww/ebB3wBwpOGI4UpGoUawIsFTwMVqCrhYTQEXqyngYjUFXKymgOfg9XpkTvitDb02j+zJ6F1/z7RnSB/3PwlOn0zTf7rfcEXRpIDn0PNaDx1f8XsIZQ5naF/XTv970QrQpxs/pfnOZgA++4vPaL69Ga9PH9lfSgHPIbEqMewjMKfSIVYf0k3o41T+1XKc+IXpsbNQekcpTlH05/QOmwKeQ/zGOPGFcf/olEDxE8U4iWiFJ7UhhZP0a3KKHK7adpXhiqJJAR9F8TeKwQWn2KHo3iLT5YzgJl2qn6oGF0rvLCV5Y9J0SZFUODdbhSw+L07R14qIL4pHbvQekNqQoutXXVRvrTZdSmQp4GMofTq6DzqAP4rPenWW6TIiTacoYjUFXKymRrBSoNQIVuRyRvD9AZYzGf73UQhNVidwyEPnDI6J0bxydDE1gpUrngIuVlPAxWoKuFhNARerKeBiNQVcrKaAi9UUcLGaAi5WU8DFagq4WE0BF6uF+8haRJqD5hP5JqtA3Yt1fPz5xyPWH/j6AW6ZeYuBinKYBdwBXAeUAF1AC/4jBb8NpwQ9kzmGQmiyeu8N91JfOdQh+qrSiEwfMR+4H/8c4TTwPpDED/1NKOBRMNEmq+mTaTre6CD1QIpYeTgTBW1auIn189aPe/vP3/6c7qPd1DxQgxsP6Aw1AdyLH+5DwI+BgdnvHCDESQDMBNxwc9DxmmiT1fM/O8/pZ09z5jtnmLF5BpV/Xhl40F898Cp7PtozuPzimly9Q4Yce+4YrT9tpXlrM1944QvBBP06YGBCgl8wFG7wn0sJ8cEwMwE33Bx0vC5tsvrQ/Q/lf1EMvE6P1n9upfWlVmb9YBZld5QFVCG88f4bw5bXrx3faN53oo/3H3+f5m81c+v+Wym6egonN7r422278Pfd+OfjAxqmbndjMRNww81Bx+vlBS9zU+NNQ30o8/xg9h3p838tZ4AYxGvixGYEO4K/dt1rzP+n+UMrFoy9fdfhLv8LB39Ow3kBzGnYedHX04FW4BhwEPjdqd1VPjoHH0N8Vpy6XXXj3v7cv5yj5ZkWEnMT1PxdDWWry3CcYJ9trFhWwdJHl457+0NfPcTZN85SsaqC+u/UU76ofOqLOo5/xaQUuBP4N/w3me0o4IUs9UcpktcnKVlWEniwL9cN22+gr6GP8oUBBHtAGngTuA///dY1wCdAKrhdjkYBn0JuqUvp8mhP95asTZKsDWGizv/FH7GX47/pvAX/1OUD4HDwux+gaSNCpGkjppamjZArngIuVlPAxWoKuFhNARerKeBiNQVcrKaAi9UUcLGaAi5WU8DFagq4WE0BF6sp4GI1BVysNsH7wZ0O4Ehw5UxaNXDGdBF5qMapMcfzvLyTwEz0iZ4jnufdepkFBc5xnH1Rrg9UY9h0iiJWU8DFahMN+CuBVDF1ol4fqMZQTehNpkih0SmKWE0BF6sp4GI1BVyspoCL1f4fmu6KxEARLygAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 216x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after iteration 5\n",
      "iter    0   |   diff: 0.59049   |   V(start): 0.590 \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAALgAAAC4CAYAAABQMybHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADcpJREFUeJzt3X1wXNV5x/Hvubur3dXbSpa8AdkGObKxAqFghA1jg18CeGyXIS7jZGyoxyVOiJUxQ0oLpO5LPKgTmqTpMKEdD5kQ2nEHayY0TWcY13VJQ5IyjDMGM6bGNmDZ+BVLtiTrXdqX2z+uJSFrpdVKuvfcPX4+/2jPzt3ZR3d+Orp7997zKNu2EcJUlu4ChHCTBFwYTQIujCYBF0aTgAujScCF0STgwmgScGE0CbgwWjDbBkqpx4HHnVFRHdS6XJIQ2RUXH6Ozs1Nl207l8lW9UnfaO3d+fUqFuaW+fisAu3c3aq5kbBs3bgBg377/1lzJ2FategDw934E2L59O01NTVkDLocowmgScGE0CbgwmgRcGE0CLowmARdGk4ALo0nAhdEk4MJoEnBhNAm4MJoEXBhNAi6MJgEXRst6PbgbWpIt/KLrF3w88DF9dh/FVjFVwSo2lGxgZnCmjpJGeOL4E1xMXhz1/PM3Pk91pNr7gjLY9M4mLvRfGPX8ztt2UlNUo6GikfyyD7UE/KXLL3E2eZYFBQuIB+K0p9r5KPERl9OXmYn+gA+6o+gO4gXxoXFpoFRjNZndVX4XVZGqoXEsGNNYzWi696HnAe9Od3M2eZaoivJk2ZMo5VyznrAT2PhrncQVsRUsKlk04e1DR0OU/GsJXeu7GLhtALJejj91q+OrWVqx1P03AoKngxTvLmagdoCeh3om9Jpc9+F08zzgERUhrML02r18r/V7LChYwLzQPL4Q/gJhFfa6nHG9eflNPuj9YGi8Ob553O2Dp4KEToQoe6GMVGWKzj/udD3oe5v3cqjj0NC4fm79tL9H8HSQ4leLCR8OQwJIMeGA57oPp5vnAQ+oAJtKN/Fq26ucSZ7hTPIMv+JXlFql1JfVUx2q9rqkMb3b/S50D4+fffLZrK+xlY3Vb2GdtZjx/Rm0b22nb3mfazXub9s/YjzdAbdaLCqfqcTGRl35S40cinDdxuvGfV3g2wEoG70PjQ84QF2kjod+9hBHwkf49dZf81bvW3SkO9jTtYdvlX9LR0kZPRV/iiWXlgyNLzVcGnf7yO8iFP5PIWkrDQq613bTv6jf1RobGhu4e8vd2Le5c3iXrkjTvrWdksYSVK/C6rdIzEnQ8XjH+K9LpQF4xnqGupl1pMvSrtSXjecBT9kpTiROMDc1l8XHF1NaXEqxKua1rtfot90NQ86CkJiXmPDmVqtF9DdRutd20/OHPdhF/vpMMSkW9C3vo++ePiL/G6GksYTEvETW/WIftyEJyeuSpEv0hBs0BDxhJ/hR24/4+Vd/zvzT8+nr6OO9vvcAqA3n95IU/Yv7aa5rhoDuSlwQGA56Pn174nnAQyrEfYX3cfLiSfbP309fbx/lgXKWR5azqnCV1+VMPxPD/Vl59vtp+ZC5vmQ9c/9tLuHzYY7uPOp1CVm9WPOi7hKy2lW3C4DQjhBJkpqrGc0v+zCP/tkIkTsJuDCaBFwYTQIujCYBF0aTgAujeR7wYFuQ2S/OJnzeubBq9ouziRyPeF1G3rNeswj8hXNSOvDjANbfW6DvC0Pf8jzgga4ARUeKhsZFx4oItYa8LiPvqcMK9a5z8ZM6rbD2S8Az8Tzg/XP66a7txlbOdRrJkiSddZ1el5H3Uo+l4Mq8YEdsUl9Labp0zt+0HIO3/FELdtAmXZCm+eFm+SQwGTeAvch2Joow2A8YcGGXC7REq39OPz3zekgVpWT2noLUYynnp8zeY9K2W8594xwqpWT2noobILkrCRW6C/EvbQFPR+UT0bSIZ9/kWibzpzBazm0E4YCL5QgxUQrbtqfeRlAp9bhS6oBS6gC0TE9tQngk5xl89+4/d7GcyRtssurJYiST5uzrgwff01zH2BYuvP3KIz/vR8e0zOBC5DMJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmgScGE0CbgwmgRcGE0CLowmARdG8/yOHr/0TxzTt4EyoBEYXNm5GvgToA/4Oy1VjbJ231rO954f9XzjikYWxBZoqOgqPtmP2m5Z090/0RTLPreM2UWzh8blBeUaq/EfbQHX3T/RFOtuXMfK61fqLsO3tAVcd//ErBbi/EsF8PE/l19+8ksOXBy+jfDpW5/WWE0GmvejtoDr7p+YlQ8OYyfitxd+O2Lsu4Br3o/aAr794+0se3cZbX/dpquE8WX6cORDDY0NrPnbNQRqfNodSvN+1HaaMHQiRPgDf7XuzlfpNlljZixyHlwYTQIujOb5Mfhg/8SKn/p0Qb0XMjx3EtjhbRnZ7Fm1B4D2He2aKxmDT/ajzODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujed8I9kSQ+NfjhM44PfDiW+JE3pJGsLnq/k437cucc+Dd3+ymY10HdkI6rV3N84CnY2lU//CyziqhSFWkvC4j71lxy7kzBpwGsCGk01oG3gd8Rpre5b3YQRsbm8SNCRK1Ca/LyHvhx8IweAFhFKJPRVHK/4vWe03LMXjXw11OA4EC6HxU+mROhlVuEf5KGCywZlkE75bpOxMtAU/PSNNzXw8D8wdk9p6C8GNhVLki+mcye49F259952aZuafKKreIvRHTXYavyWlCYTQJuDCaNIIVeUoawQqR+wy+b9/zLpYzeatWPQDkR5PVc+dGL7nmF1VV11955P+zMtIIVlzzJODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmgScGE0CbgwmgRcGM3zW9Y2vbOJC/0XRj2/87ad1BTVeF1ORr5vsgosfnUxZ7rOjHp+38P7+GLlFzVUdJVrvRHsXeV3URWpGhrHgv67tzAfmqzef8P9VJdWD40roj5tLKCJtoCvjq9macXSib/ABj4CahheD8Rl+dBkdWPtRtZUr9Fdhm9pC/je5r0c6jg0NK6fWz/+C05CaFsIu8ImtSWFvdJ2Pei+b7IK7D66m7fPvT00fm7JcxqryeBabQS7v23/iPG2n24bd3vVrLCVjbqkCPw4AC9D6qkU9iL31uPzfZNV4I1Tb4wY+y7g12oj2O9WfpcV21ZM7sUDQALUEeVqwBsaG/jSLV+i8K8KXXuPqWpobOCRHz5CtC6qu5TMNDeC1bfeVwUk9uWwqtVxCG4LQgDsOucwhRvcK29Q+lP/N1m1+2VV2bHkz4J2VZDekCa9Mu1JsIUZ8ifgUUhv9v9sKvzF84Dvqtvl9VvmbEST1RzOZHrp94/8HoBjO45prmQM0ghWCPdJwIXRJODCaBJwYTQJuDCaBFwYTQKegd1vkzrntDa0223S5/13/j3VkSJx2vkmOHE+QbIlqbkif5KAZ9D3Sh+dDzk9hFKHU3Ss7SB51F8BOrv5LE33NgHw6Z9+StPdTdgD8pX91STgGYRWhkZ8BabKFYEajy5Cn6CSL5eggleWx05D4T2FqAL/r+ntNQl4BsEFQYILg87eiULkiQgq5K/wxDbGUGGnJlWgmLl9puaK/EkCPobIkxGwQEUUBQ8W6C5nFCtsUfl0JVhQeG8h4QVh3SX5Uv5cbOWxYG2Qgq8UELwj6LvZe1BsY4ye3/VQ+Wyl7lJ8SwI+jsJn/HujAziz+KyXZ+kuw9fkEEUYTQIujCaNYEWekkawQkxmBn/HxXKmwvk98qHJag673HNqaE7055mjz5JGsOKaJwEXRpOAC6NJwIXRJODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmje3rLmk+ag2fi+ySpQ/UI1n1z+ZNTzB795kNuvu11DRRnMAu4B5gBRoAdoxrml4Ig3Jcg9mePIhyarD970IDXlwx2iZxb6ZPmIm4H1OMcILcCHQBgn9LciAfeDXJusJs4n6Hy9k9iGGIESbxYK2rJwC+tq1014+8tvX6b3eC/xDXGsoEtHqCHgQZxwvw/8OzC4+p0CPFwEQE/ANTcHnahcm6x2/VcXLc+1cPEHF5lRP4Pyb5S7HvSXD77MmyffHBq/sDpT75Bhp54/Ret/ttL0bBOf//7n3Qn6HGBwQYLfMBxucO5L8fDGMD0B19wcdKKubrL66PpHs78oAHa3Tes/ttL6T63M+pdZFN1T5FKF8PqHr48Yr1szsdl84NwAH279kKbvNHHnO3dS8LlpXNzos79u+5Wf9+Mcjw/aMX1vNx49AdfcHHSiXrrlJW5tuHW4D2WWP8yBYwPOv+UUEIBgPEhghrsz+CtzXuHmf7h5+Ilbxt++53CP80DhrGlY68Kaht2feVwKtAKngEPAH0zvW2Ujx+DjCM4KUr23esLbt/1zG81/2Uxoboj438QpWlWEUu7e21i2pIzFX1s84e3f//L7XHr9EmUry6j5QQ0ld5RMf1Gncc6YFAL3Av+B8yGzAwl4Pot9NUZ4fpjokqjrwZ6sm3bexMCOAUoWuhDsQQlgD/Awzuet64EzQMy9txyLBHwaWYUWhUv9vdxbuCpMuMqDhTr/D2fGXorzofN2nEOXj4HD7r/9IFk2wkOybMT0kmUjxDVPAi6MJgEXRpOAC6NJwIXRJODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmgScGG0HK8HV53AMffKmbJK4KLuIrKQGqfHjbZtZ10EJtc7eo7Ztn3nJAtynVLqgJ/rA6nRa3KIIowmARdGyzXgP3Gliunj9/pAavRUTh8yhcg3cogijCYBF0aTgAujScCF0STgwmj/D6950SHWwvgbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 216x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after iteration 6\n",
      "iter    0   |   diff: 0.00000   |   V(start): 0.590 \n",
      "Terminated\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAALgAAAC4CAYAAABQMybHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADcpJREFUeJzt3X1wXNV5x/Hvubur3dXbSpa8AdkGObKxAqFghA1jg18CeGyXIS7jZGyoxyVOiJUxQ0oLpO5LPKgTmqTpMKEdD5kQ2nEHayY0TWcY13VJQ5IyjDMGM6bGNmDZ+BVLtiTrXdqX2z+uJSFrpdVKuvfcPX4+/2jPzt3ZR3d+Orp7997zKNu2EcJUlu4ChHCTBFwYTQIujCYBF0aTgAujScCF0STgwmgScGE0CbgwWjDbBkqpx4HHnVFRHdS6XJIQ2RUXH6Ozs1Nl207l8lW9UnfaO3d+fUqFuaW+fisAu3c3aq5kbBs3bgBg377/1lzJ2FategDw934E2L59O01NTVkDLocowmgScGE0CbgwmgRcGE0CLowmARdGk4ALo0nAhdEk4MJoEnBhNAm4MJoEXBhNAi6MJgEXRst6PbgbWpIt/KLrF3w88DF9dh/FVjFVwSo2lGxgZnCmjpJGeOL4E1xMXhz1/PM3Pk91pNr7gjLY9M4mLvRfGPX8ztt2UlNUo6GikfyyD7UE/KXLL3E2eZYFBQuIB+K0p9r5KPERl9OXmYn+gA+6o+gO4gXxoXFpoFRjNZndVX4XVZGqoXEsGNNYzWi696HnAe9Od3M2eZaoivJk2ZMo5VyznrAT2PhrncQVsRUsKlk04e1DR0OU/GsJXeu7GLhtALJejj91q+OrWVqx1P03AoKngxTvLmagdoCeh3om9Jpc9+F08zzgERUhrML02r18r/V7LChYwLzQPL4Q/gJhFfa6nHG9eflNPuj9YGi8Ob553O2Dp4KEToQoe6GMVGWKzj/udD3oe5v3cqjj0NC4fm79tL9H8HSQ4leLCR8OQwJIMeGA57oPp5vnAQ+oAJtKN/Fq26ucSZ7hTPIMv+JXlFql1JfVUx2q9rqkMb3b/S50D4+fffLZrK+xlY3Vb2GdtZjx/Rm0b22nb3mfazXub9s/YjzdAbdaLCqfqcTGRl35S40cinDdxuvGfV3g2wEoG70PjQ84QF2kjod+9hBHwkf49dZf81bvW3SkO9jTtYdvlX9LR0kZPRV/iiWXlgyNLzVcGnf7yO8iFP5PIWkrDQq613bTv6jf1RobGhu4e8vd2Le5c3iXrkjTvrWdksYSVK/C6rdIzEnQ8XjH+K9LpQF4xnqGupl1pMvSrtSXjecBT9kpTiROMDc1l8XHF1NaXEqxKua1rtfot90NQ86CkJiXmPDmVqtF9DdRutd20/OHPdhF/vpMMSkW9C3vo++ePiL/G6GksYTEvETW/WIftyEJyeuSpEv0hBs0BDxhJ/hR24/4+Vd/zvzT8+nr6OO9vvcAqA3n95IU/Yv7aa5rhoDuSlwQGA56Pn174nnAQyrEfYX3cfLiSfbP309fbx/lgXKWR5azqnCV1+VMPxPD/Vl59vtp+ZC5vmQ9c/9tLuHzYY7uPOp1CVm9WPOi7hKy2lW3C4DQjhBJkpqrGc0v+zCP/tkIkTsJuDCaBFwYTQIujCYBF0aTgAujeR7wYFuQ2S/OJnzeubBq9ouziRyPeF1G3rNeswj8hXNSOvDjANbfW6DvC0Pf8jzgga4ARUeKhsZFx4oItYa8LiPvqcMK9a5z8ZM6rbD2S8Az8Tzg/XP66a7txlbOdRrJkiSddZ1el5H3Uo+l4Mq8YEdsUl9Labp0zt+0HIO3/FELdtAmXZCm+eFm+SQwGTeAvch2Joow2A8YcGGXC7REq39OPz3zekgVpWT2noLUYynnp8zeY9K2W8594xwqpWT2noobILkrCRW6C/EvbQFPR+UT0bSIZ9/kWibzpzBazm0E4YCL5QgxUQrbtqfeRlAp9bhS6oBS6gC0TE9tQngk5xl89+4/d7GcyRtssurJYiST5uzrgwff01zH2BYuvP3KIz/vR8e0zOBC5DMJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmgScGE0CbgwmgRcGE0CLowmARdG8/yOHr/0TxzTt4EyoBEYXNm5GvgToA/4Oy1VjbJ231rO954f9XzjikYWxBZoqOgqPtmP2m5Z090/0RTLPreM2UWzh8blBeUaq/EfbQHX3T/RFOtuXMfK61fqLsO3tAVcd//ErBbi/EsF8PE/l19+8ksOXBy+jfDpW5/WWE0GmvejtoDr7p+YlQ8OYyfitxd+O2Lsu4Br3o/aAr794+0se3cZbX/dpquE8WX6cORDDY0NrPnbNQRqfNodSvN+1HaaMHQiRPgDf7XuzlfpNlljZixyHlwYTQIujOb5Mfhg/8SKn/p0Qb0XMjx3EtjhbRnZ7Fm1B4D2He2aKxmDT/ajzODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujed8I9kSQ+NfjhM44PfDiW+JE3pJGsLnq/k437cucc+Dd3+ymY10HdkI6rV3N84CnY2lU//CyziqhSFWkvC4j71lxy7kzBpwGsCGk01oG3gd8Rpre5b3YQRsbm8SNCRK1Ca/LyHvhx8IweAFhFKJPRVHK/4vWe03LMXjXw11OA4EC6HxU+mROhlVuEf5KGCywZlkE75bpOxMtAU/PSNNzXw8D8wdk9p6C8GNhVLki+mcye49F259952aZuafKKreIvRHTXYavyWlCYTQJuDCaNIIVeUoawQqR+wy+b9/zLpYzeatWPQDkR5PVc+dGL7nmF1VV11955P+zMtIIVlzzJODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmgScGE0CbgwmgRcGM3zW9Y2vbOJC/0XRj2/87ad1BTVeF1ORr5vsgosfnUxZ7rOjHp+38P7+GLlFzVUdJVrvRHsXeV3URWpGhrHgv67tzAfmqzef8P9VJdWD40roj5tLKCJtoCvjq9macXSib/ABj4CahheD8Rl+dBkdWPtRtZUr9Fdhm9pC/je5r0c6jg0NK6fWz/+C05CaFsIu8ImtSWFvdJ2Pei+b7IK7D66m7fPvT00fm7JcxqryeBabQS7v23/iPG2n24bd3vVrLCVjbqkCPw4AC9D6qkU9iL31uPzfZNV4I1Tb4wY+y7g12oj2O9WfpcV21ZM7sUDQALUEeVqwBsaG/jSLV+i8K8KXXuPqWpobOCRHz5CtC6qu5TMNDeC1bfeVwUk9uWwqtVxCG4LQgDsOucwhRvcK29Q+lP/N1m1+2VV2bHkz4J2VZDekCa9Mu1JsIUZ8ifgUUhv9v9sKvzF84Dvqtvl9VvmbEST1RzOZHrp94/8HoBjO45prmQM0ghWCPdJwIXRJODCaBJwYTQJuDCaBFwYTQKegd1vkzrntDa0223S5/13/j3VkSJx2vkmOHE+QbIlqbkif5KAZ9D3Sh+dDzk9hFKHU3Ss7SB51F8BOrv5LE33NgHw6Z9+StPdTdgD8pX91STgGYRWhkZ8BabKFYEajy5Cn6CSL5eggleWx05D4T2FqAL/r+ntNQl4BsEFQYILg87eiULkiQgq5K/wxDbGUGGnJlWgmLl9puaK/EkCPobIkxGwQEUUBQ8W6C5nFCtsUfl0JVhQeG8h4QVh3SX5Uv5cbOWxYG2Qgq8UELwj6LvZe1BsY4ye3/VQ+Wyl7lJ8SwI+jsJn/HujAziz+KyXZ+kuw9fkEEUYTQIujCaNYEWekkawQkxmBn/HxXKmwvk98qHJag673HNqaE7055mjz5JGsOKaJwEXRpOAC6NJwIXRJODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmje3rLmk+ag2fi+ySpQ/UI1n1z+ZNTzB795kNuvu11DRRnMAu4B5gBRoAdoxrml4Ig3Jcg9mePIhyarD970IDXlwx2iZxb6ZPmIm4H1OMcILcCHQBgn9LciAfeDXJusJs4n6Hy9k9iGGIESbxYK2rJwC+tq1014+8tvX6b3eC/xDXGsoEtHqCHgQZxwvw/8OzC4+p0CPFwEQE/ANTcHnahcm6x2/VcXLc+1cPEHF5lRP4Pyb5S7HvSXD77MmyffHBq/sDpT75Bhp54/Ret/ttL0bBOf//7n3Qn6HGBwQYLfMBxucO5L8fDGMD0B19wcdKKubrL66PpHs78oAHa3Tes/ttL6T63M+pdZFN1T5FKF8PqHr48Yr1szsdl84NwAH279kKbvNHHnO3dS8LlpXNzos79u+5Wf9+Mcjw/aMX1vNx49AdfcHHSiXrrlJW5tuHW4D2WWP8yBYwPOv+UUEIBgPEhghrsz+CtzXuHmf7h5+Ilbxt++53CP80DhrGlY68Kaht2feVwKtAKngEPAH0zvW2Ujx+DjCM4KUr23esLbt/1zG81/2Uxoboj438QpWlWEUu7e21i2pIzFX1s84e3f//L7XHr9EmUry6j5QQ0ld5RMf1Gncc6YFAL3Av+B8yGzAwl4Pot9NUZ4fpjokqjrwZ6sm3bexMCOAUoWuhDsQQlgD/Awzuet64EzQMy9txyLBHwaWYUWhUv9vdxbuCpMuMqDhTr/D2fGXorzofN2nEOXj4HD7r/9IFk2wkOybMT0kmUjxDVPAi6MJgEXRpOAC6NJwIXRJODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmgScGG0HK8HV53AMffKmbJK4KLuIrKQGqfHjbZtZ10EJtc7eo7Ztn3nJAtynVLqgJ/rA6nRa3KIIowmARdGyzXgP3Gliunj9/pAavRUTh8yhcg3cogijCYBF0aTgAujScCF0STgwmj/D6950SHWwvgbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 216x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after iteration 7\n",
      "iter    0   |   diff: 0.00000   |   V(start): 0.590 \n",
      "Terminated\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAALgAAAC4CAYAAABQMybHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADcpJREFUeJzt3X1wXNV5x/Hvubur3dXbSpa8AdkGObKxAqFghA1jg18CeGyXIS7jZGyoxyVOiJUxQ0oLpO5LPKgTmqTpMKEdD5kQ2nEHayY0TWcY13VJQ5IyjDMGM6bGNmDZ+BVLtiTrXdqX2z+uJSFrpdVKuvfcPX4+/2jPzt3ZR3d+Orp7997zKNu2EcJUlu4ChHCTBFwYTQIujCYBF0aTgAujScCF0STgwmgScGE0CbgwWjDbBkqpx4HHnVFRHdS6XJIQ2RUXH6Ozs1Nl207l8lW9UnfaO3d+fUqFuaW+fisAu3c3aq5kbBs3bgBg377/1lzJ2FategDw934E2L59O01NTVkDLocowmgScGE0CbgwmgRcGE0CLowmARdGk4ALo0nAhdEk4MJoEnBhNAm4MJoEXBhNAi6MJgEXRst6PbgbWpIt/KLrF3w88DF9dh/FVjFVwSo2lGxgZnCmjpJGeOL4E1xMXhz1/PM3Pk91pNr7gjLY9M4mLvRfGPX8ztt2UlNUo6GikfyyD7UE/KXLL3E2eZYFBQuIB+K0p9r5KPERl9OXmYn+gA+6o+gO4gXxoXFpoFRjNZndVX4XVZGqoXEsGNNYzWi696HnAe9Od3M2eZaoivJk2ZMo5VyznrAT2PhrncQVsRUsKlk04e1DR0OU/GsJXeu7GLhtALJejj91q+OrWVqx1P03AoKngxTvLmagdoCeh3om9Jpc9+F08zzgERUhrML02r18r/V7LChYwLzQPL4Q/gJhFfa6nHG9eflNPuj9YGi8Ob553O2Dp4KEToQoe6GMVGWKzj/udD3oe5v3cqjj0NC4fm79tL9H8HSQ4leLCR8OQwJIMeGA57oPp5vnAQ+oAJtKN/Fq26ucSZ7hTPIMv+JXlFql1JfVUx2q9rqkMb3b/S50D4+fffLZrK+xlY3Vb2GdtZjx/Rm0b22nb3mfazXub9s/YjzdAbdaLCqfqcTGRl35S40cinDdxuvGfV3g2wEoG70PjQ84QF2kjod+9hBHwkf49dZf81bvW3SkO9jTtYdvlX9LR0kZPRV/iiWXlgyNLzVcGnf7yO8iFP5PIWkrDQq613bTv6jf1RobGhu4e8vd2Le5c3iXrkjTvrWdksYSVK/C6rdIzEnQ8XjH+K9LpQF4xnqGupl1pMvSrtSXjecBT9kpTiROMDc1l8XHF1NaXEqxKua1rtfot90NQ86CkJiXmPDmVqtF9DdRutd20/OHPdhF/vpMMSkW9C3vo++ePiL/G6GksYTEvETW/WIftyEJyeuSpEv0hBs0BDxhJ/hR24/4+Vd/zvzT8+nr6OO9vvcAqA3n95IU/Yv7aa5rhoDuSlwQGA56Pn174nnAQyrEfYX3cfLiSfbP309fbx/lgXKWR5azqnCV1+VMPxPD/Vl59vtp+ZC5vmQ9c/9tLuHzYY7uPOp1CVm9WPOi7hKy2lW3C4DQjhBJkpqrGc0v+zCP/tkIkTsJuDCaBFwYTQIujCYBF0aTgAujeR7wYFuQ2S/OJnzeubBq9ouziRyPeF1G3rNeswj8hXNSOvDjANbfW6DvC0Pf8jzgga4ARUeKhsZFx4oItYa8LiPvqcMK9a5z8ZM6rbD2S8Az8Tzg/XP66a7txlbOdRrJkiSddZ1el5H3Uo+l4Mq8YEdsUl9Labp0zt+0HIO3/FELdtAmXZCm+eFm+SQwGTeAvch2Joow2A8YcGGXC7REq39OPz3zekgVpWT2noLUYynnp8zeY9K2W8594xwqpWT2noobILkrCRW6C/EvbQFPR+UT0bSIZ9/kWibzpzBazm0E4YCL5QgxUQrbtqfeRlAp9bhS6oBS6gC0TE9tQngk5xl89+4/d7GcyRtssurJYiST5uzrgwff01zH2BYuvP3KIz/vR8e0zOBC5DMJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmgScGE0CbgwmgRcGE0CLowmARdG8/yOHr/0TxzTt4EyoBEYXNm5GvgToA/4Oy1VjbJ231rO954f9XzjikYWxBZoqOgqPtmP2m5Z090/0RTLPreM2UWzh8blBeUaq/EfbQHX3T/RFOtuXMfK61fqLsO3tAVcd//ErBbi/EsF8PE/l19+8ksOXBy+jfDpW5/WWE0GmvejtoDr7p+YlQ8OYyfitxd+O2Lsu4Br3o/aAr794+0se3cZbX/dpquE8WX6cORDDY0NrPnbNQRqfNodSvN+1HaaMHQiRPgDf7XuzlfpNlljZixyHlwYTQIujOb5Mfhg/8SKn/p0Qb0XMjx3EtjhbRnZ7Fm1B4D2He2aKxmDT/ajzODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujed8I9kSQ+NfjhM44PfDiW+JE3pJGsLnq/k437cucc+Dd3+ymY10HdkI6rV3N84CnY2lU//CyziqhSFWkvC4j71lxy7kzBpwGsCGk01oG3gd8Rpre5b3YQRsbm8SNCRK1Ca/LyHvhx8IweAFhFKJPRVHK/4vWe03LMXjXw11OA4EC6HxU+mROhlVuEf5KGCywZlkE75bpOxMtAU/PSNNzXw8D8wdk9p6C8GNhVLki+mcye49F259952aZuafKKreIvRHTXYavyWlCYTQJuDCaNIIVeUoawQqR+wy+b9/zLpYzeatWPQDkR5PVc+dGL7nmF1VV11955P+zMtIIVlzzJODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmgScGE0CbgwmgRcGM3zW9Y2vbOJC/0XRj2/87ad1BTVeF1ORr5vsgosfnUxZ7rOjHp+38P7+GLlFzVUdJVrvRHsXeV3URWpGhrHgv67tzAfmqzef8P9VJdWD40roj5tLKCJtoCvjq9macXSib/ABj4CahheD8Rl+dBkdWPtRtZUr9Fdhm9pC/je5r0c6jg0NK6fWz/+C05CaFsIu8ImtSWFvdJ2Pei+b7IK7D66m7fPvT00fm7JcxqryeBabQS7v23/iPG2n24bd3vVrLCVjbqkCPw4AC9D6qkU9iL31uPzfZNV4I1Tb4wY+y7g12oj2O9WfpcV21ZM7sUDQALUEeVqwBsaG/jSLV+i8K8KXXuPqWpobOCRHz5CtC6qu5TMNDeC1bfeVwUk9uWwqtVxCG4LQgDsOucwhRvcK29Q+lP/N1m1+2VV2bHkz4J2VZDekCa9Mu1JsIUZ8ifgUUhv9v9sKvzF84Dvqtvl9VvmbEST1RzOZHrp94/8HoBjO45prmQM0ghWCPdJwIXRJODCaBJwYTQJuDCaBFwYTQKegd1vkzrntDa0223S5/13/j3VkSJx2vkmOHE+QbIlqbkif5KAZ9D3Sh+dDzk9hFKHU3Ss7SB51F8BOrv5LE33NgHw6Z9+StPdTdgD8pX91STgGYRWhkZ8BabKFYEajy5Cn6CSL5eggleWx05D4T2FqAL/r+ntNQl4BsEFQYILg87eiULkiQgq5K/wxDbGUGGnJlWgmLl9puaK/EkCPobIkxGwQEUUBQ8W6C5nFCtsUfl0JVhQeG8h4QVh3SX5Uv5cbOWxYG2Qgq8UELwj6LvZe1BsY4ye3/VQ+Wyl7lJ8SwI+jsJn/HujAziz+KyXZ+kuw9fkEEUYTQIujCaNYEWekkawQkxmBn/HxXKmwvk98qHJag673HNqaE7055mjz5JGsOKaJwEXRpOAC6NJwIXRJODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmje3rLmk+ag2fi+ySpQ/UI1n1z+ZNTzB795kNuvu11DRRnMAu4B5gBRoAdoxrml4Ig3Jcg9mePIhyarD970IDXlwx2iZxb6ZPmIm4H1OMcILcCHQBgn9LciAfeDXJusJs4n6Hy9k9iGGIESbxYK2rJwC+tq1014+8tvX6b3eC/xDXGsoEtHqCHgQZxwvw/8OzC4+p0CPFwEQE/ANTcHnahcm6x2/VcXLc+1cPEHF5lRP4Pyb5S7HvSXD77MmyffHBq/sDpT75Bhp54/Ret/ttL0bBOf//7n3Qn6HGBwQYLfMBxucO5L8fDGMD0B19wcdKKubrL66PpHs78oAHa3Tes/ttL6T63M+pdZFN1T5FKF8PqHr48Yr1szsdl84NwAH279kKbvNHHnO3dS8LlpXNzos79u+5Wf9+Mcjw/aMX1vNx49AdfcHHSiXrrlJW5tuHW4D2WWP8yBYwPOv+UUEIBgPEhghrsz+CtzXuHmf7h5+Ilbxt++53CP80DhrGlY68Kaht2feVwKtAKngEPAH0zvW2Ujx+DjCM4KUr23esLbt/1zG81/2Uxoboj438QpWlWEUu7e21i2pIzFX1s84e3f//L7XHr9EmUry6j5QQ0ld5RMf1Gncc6YFAL3Av+B8yGzAwl4Pot9NUZ4fpjokqjrwZ6sm3bexMCOAUoWuhDsQQlgD/Awzuet64EzQMy9txyLBHwaWYUWhUv9vdxbuCpMuMqDhTr/D2fGXorzofN2nEOXj4HD7r/9IFk2wkOybMT0kmUjxDVPAi6MJgEXRpOAC6NJwIXRJODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmgScGG0HK8HV53AMffKmbJK4KLuIrKQGqfHjbZtZ10EJtc7eo7Ztn3nJAtynVLqgJ/rA6nRa3KIIowmARdGyzXgP3Gliunj9/pAavRUTh8yhcg3cogijCYBF0aTgAujScCF0STgwmj/D6950SHWwvgbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 216x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after iteration 8\n",
      "iter    0   |   diff: 0.00000   |   V(start): 0.590 \n",
      "Terminated\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAALgAAAC4CAYAAABQMybHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADcpJREFUeJzt3X1wXNV5x/Hvubur3dXbSpa8AdkGObKxAqFghA1jg18CeGyXIS7jZGyoxyVOiJUxQ0oLpO5LPKgTmqTpMKEdD5kQ2nEHayY0TWcY13VJQ5IyjDMGM6bGNmDZ+BVLtiTrXdqX2z+uJSFrpdVKuvfcPX4+/2jPzt3ZR3d+Orp7997zKNu2EcJUlu4ChHCTBFwYTQIujCYBF0aTgAujScCF0STgwmgScGE0CbgwWjDbBkqpx4HHnVFRHdS6XJIQ2RUXH6Ozs1Nl207l8lW9UnfaO3d+fUqFuaW+fisAu3c3aq5kbBs3bgBg377/1lzJ2FategDw934E2L59O01NTVkDLocowmgScGE0CbgwmgRcGE0CLowmARdGk4ALo0nAhdEk4MJoEnBhNAm4MJoEXBhNAi6MJgEXRst6PbgbWpIt/KLrF3w88DF9dh/FVjFVwSo2lGxgZnCmjpJGeOL4E1xMXhz1/PM3Pk91pNr7gjLY9M4mLvRfGPX8ztt2UlNUo6GikfyyD7UE/KXLL3E2eZYFBQuIB+K0p9r5KPERl9OXmYn+gA+6o+gO4gXxoXFpoFRjNZndVX4XVZGqoXEsGNNYzWi696HnAe9Od3M2eZaoivJk2ZMo5VyznrAT2PhrncQVsRUsKlk04e1DR0OU/GsJXeu7GLhtALJejj91q+OrWVqx1P03AoKngxTvLmagdoCeh3om9Jpc9+F08zzgERUhrML02r18r/V7LChYwLzQPL4Q/gJhFfa6nHG9eflNPuj9YGi8Ob553O2Dp4KEToQoe6GMVGWKzj/udD3oe5v3cqjj0NC4fm79tL9H8HSQ4leLCR8OQwJIMeGA57oPp5vnAQ+oAJtKN/Fq26ucSZ7hTPIMv+JXlFql1JfVUx2q9rqkMb3b/S50D4+fffLZrK+xlY3Vb2GdtZjx/Rm0b22nb3mfazXub9s/YjzdAbdaLCqfqcTGRl35S40cinDdxuvGfV3g2wEoG70PjQ84QF2kjod+9hBHwkf49dZf81bvW3SkO9jTtYdvlX9LR0kZPRV/iiWXlgyNLzVcGnf7yO8iFP5PIWkrDQq613bTv6jf1RobGhu4e8vd2Le5c3iXrkjTvrWdksYSVK/C6rdIzEnQ8XjH+K9LpQF4xnqGupl1pMvSrtSXjecBT9kpTiROMDc1l8XHF1NaXEqxKua1rtfot90NQ86CkJiXmPDmVqtF9DdRutd20/OHPdhF/vpMMSkW9C3vo++ePiL/G6GksYTEvETW/WIftyEJyeuSpEv0hBs0BDxhJ/hR24/4+Vd/zvzT8+nr6OO9vvcAqA3n95IU/Yv7aa5rhoDuSlwQGA56Pn174nnAQyrEfYX3cfLiSfbP309fbx/lgXKWR5azqnCV1+VMPxPD/Vl59vtp+ZC5vmQ9c/9tLuHzYY7uPOp1CVm9WPOi7hKy2lW3C4DQjhBJkpqrGc0v+zCP/tkIkTsJuDCaBFwYTQIujCYBF0aTgAujeR7wYFuQ2S/OJnzeubBq9ouziRyPeF1G3rNeswj8hXNSOvDjANbfW6DvC0Pf8jzgga4ARUeKhsZFx4oItYa8LiPvqcMK9a5z8ZM6rbD2S8Az8Tzg/XP66a7txlbOdRrJkiSddZ1el5H3Uo+l4Mq8YEdsUl9Labp0zt+0HIO3/FELdtAmXZCm+eFm+SQwGTeAvch2Joow2A8YcGGXC7REq39OPz3zekgVpWT2noLUYynnp8zeY9K2W8594xwqpWT2noobILkrCRW6C/EvbQFPR+UT0bSIZ9/kWibzpzBazm0E4YCL5QgxUQrbtqfeRlAp9bhS6oBS6gC0TE9tQngk5xl89+4/d7GcyRtssurJYiST5uzrgwff01zH2BYuvP3KIz/vR8e0zOBC5DMJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmgScGE0CbgwmgRcGE0CLowmARdG8/yOHr/0TxzTt4EyoBEYXNm5GvgToA/4Oy1VjbJ231rO954f9XzjikYWxBZoqOgqPtmP2m5Z090/0RTLPreM2UWzh8blBeUaq/EfbQHX3T/RFOtuXMfK61fqLsO3tAVcd//ErBbi/EsF8PE/l19+8ksOXBy+jfDpW5/WWE0GmvejtoDr7p+YlQ8OYyfitxd+O2Lsu4Br3o/aAr794+0se3cZbX/dpquE8WX6cORDDY0NrPnbNQRqfNodSvN+1HaaMHQiRPgDf7XuzlfpNlljZixyHlwYTQIujOb5Mfhg/8SKn/p0Qb0XMjx3EtjhbRnZ7Fm1B4D2He2aKxmDT/ajzODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujed8I9kSQ+NfjhM44PfDiW+JE3pJGsLnq/k437cucc+Dd3+ymY10HdkI6rV3N84CnY2lU//CyziqhSFWkvC4j71lxy7kzBpwGsCGk01oG3gd8Rpre5b3YQRsbm8SNCRK1Ca/LyHvhx8IweAFhFKJPRVHK/4vWe03LMXjXw11OA4EC6HxU+mROhlVuEf5KGCywZlkE75bpOxMtAU/PSNNzXw8D8wdk9p6C8GNhVLki+mcye49F259952aZuafKKreIvRHTXYavyWlCYTQJuDCaNIIVeUoawQqR+wy+b9/zLpYzeatWPQDkR5PVc+dGL7nmF1VV11955P+zMtIIVlzzJODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmgScGE0CbgwmgRcGM3zW9Y2vbOJC/0XRj2/87ad1BTVeF1ORr5vsgosfnUxZ7rOjHp+38P7+GLlFzVUdJVrvRHsXeV3URWpGhrHgv67tzAfmqzef8P9VJdWD40roj5tLKCJtoCvjq9macXSib/ABj4CahheD8Rl+dBkdWPtRtZUr9Fdhm9pC/je5r0c6jg0NK6fWz/+C05CaFsIu8ImtSWFvdJ2Pei+b7IK7D66m7fPvT00fm7JcxqryeBabQS7v23/iPG2n24bd3vVrLCVjbqkCPw4AC9D6qkU9iL31uPzfZNV4I1Tb4wY+y7g12oj2O9WfpcV21ZM7sUDQALUEeVqwBsaG/jSLV+i8K8KXXuPqWpobOCRHz5CtC6qu5TMNDeC1bfeVwUk9uWwqtVxCG4LQgDsOucwhRvcK29Q+lP/N1m1+2VV2bHkz4J2VZDekCa9Mu1JsIUZ8ifgUUhv9v9sKvzF84Dvqtvl9VvmbEST1RzOZHrp94/8HoBjO45prmQM0ghWCPdJwIXRJODCaBJwYTQJuDCaBFwYTQKegd1vkzrntDa0223S5/13/j3VkSJx2vkmOHE+QbIlqbkif5KAZ9D3Sh+dDzk9hFKHU3Ss7SB51F8BOrv5LE33NgHw6Z9+StPdTdgD8pX91STgGYRWhkZ8BabKFYEajy5Cn6CSL5eggleWx05D4T2FqAL/r+ntNQl4BsEFQYILg87eiULkiQgq5K/wxDbGUGGnJlWgmLl9puaK/EkCPobIkxGwQEUUBQ8W6C5nFCtsUfl0JVhQeG8h4QVh3SX5Uv5cbOWxYG2Qgq8UELwj6LvZe1BsY4ye3/VQ+Wyl7lJ8SwI+jsJn/HujAziz+KyXZ+kuw9fkEEUYTQIujCaNYEWekkawQkxmBn/HxXKmwvk98qHJag673HNqaE7055mjz5JGsOKaJwEXRpOAC6NJwIXRJODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmje3rLmk+ag2fi+ySpQ/UI1n1z+ZNTzB795kNuvu11DRRnMAu4B5gBRoAdoxrml4Ig3Jcg9mePIhyarD970IDXlwx2iZxb6ZPmIm4H1OMcILcCHQBgn9LciAfeDXJusJs4n6Hy9k9iGGIESbxYK2rJwC+tq1014+8tvX6b3eC/xDXGsoEtHqCHgQZxwvw/8OzC4+p0CPFwEQE/ANTcHnahcm6x2/VcXLc+1cPEHF5lRP4Pyb5S7HvSXD77MmyffHBq/sDpT75Bhp54/Ret/ttL0bBOf//7n3Qn6HGBwQYLfMBxucO5L8fDGMD0B19wcdKKubrL66PpHs78oAHa3Tes/ttL6T63M+pdZFN1T5FKF8PqHr48Yr1szsdl84NwAH279kKbvNHHnO3dS8LlpXNzos79u+5Wf9+Mcjw/aMX1vNx49AdfcHHSiXrrlJW5tuHW4D2WWP8yBYwPOv+UUEIBgPEhghrsz+CtzXuHmf7h5+Ilbxt++53CP80DhrGlY68Kaht2feVwKtAKngEPAH0zvW2Ujx+DjCM4KUr23esLbt/1zG81/2Uxoboj438QpWlWEUu7e21i2pIzFX1s84e3f//L7XHr9EmUry6j5QQ0ld5RMf1Gncc6YFAL3Av+B8yGzAwl4Pot9NUZ4fpjokqjrwZ6sm3bexMCOAUoWuhDsQQlgD/Awzuet64EzQMy9txyLBHwaWYUWhUv9vdxbuCpMuMqDhTr/D2fGXorzofN2nEOXj4HD7r/9IFk2wkOybMT0kmUjxDVPAi6MJgEXRpOAC6NJwIXRJODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmgScGG0HK8HV53AMffKmbJK4KLuIrKQGqfHjbZtZ10EJtc7eo7Ztn3nJAtynVLqgJ/rA6nRa3KIIowmARdGyzXgP3Gliunj9/pAavRUTh8yhcg3cogijCYBF0aTgAujScCF0STgwmj/D6950SHWwvgbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 216x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after iteration 9\n",
      "iter    0   |   diff: 0.00000   |   V(start): 0.590 \n",
      "Terminated\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAALgAAAC4CAYAAABQMybHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADcpJREFUeJzt3X1wXNV5x/Hvubur3dXbSpa8AdkGObKxAqFghA1jg18CeGyXIS7jZGyoxyVOiJUxQ0oLpO5LPKgTmqTpMKEdD5kQ2nEHayY0TWcY13VJQ5IyjDMGM6bGNmDZ+BVLtiTrXdqX2z+uJSFrpdVKuvfcPX4+/2jPzt3ZR3d+Orp7997zKNu2EcJUlu4ChHCTBFwYTQIujCYBF0aTgAujScCF0STgwmgScGE0CbgwWjDbBkqpx4HHnVFRHdS6XJIQ2RUXH6Ozs1Nl207l8lW9UnfaO3d+fUqFuaW+fisAu3c3aq5kbBs3bgBg377/1lzJ2FategDw934E2L59O01NTVkDLocowmgScGE0CbgwmgRcGE0CLowmARdGk4ALo0nAhdEk4MJoEnBhNAm4MJoEXBhNAi6MJgEXRst6PbgbWpIt/KLrF3w88DF9dh/FVjFVwSo2lGxgZnCmjpJGeOL4E1xMXhz1/PM3Pk91pNr7gjLY9M4mLvRfGPX8ztt2UlNUo6GikfyyD7UE/KXLL3E2eZYFBQuIB+K0p9r5KPERl9OXmYn+gA+6o+gO4gXxoXFpoFRjNZndVX4XVZGqoXEsGNNYzWi696HnAe9Od3M2eZaoivJk2ZMo5VyznrAT2PhrncQVsRUsKlk04e1DR0OU/GsJXeu7GLhtALJejj91q+OrWVqx1P03AoKngxTvLmagdoCeh3om9Jpc9+F08zzgERUhrML02r18r/V7LChYwLzQPL4Q/gJhFfa6nHG9eflNPuj9YGi8Ob553O2Dp4KEToQoe6GMVGWKzj/udD3oe5v3cqjj0NC4fm79tL9H8HSQ4leLCR8OQwJIMeGA57oPp5vnAQ+oAJtKN/Fq26ucSZ7hTPIMv+JXlFql1JfVUx2q9rqkMb3b/S50D4+fffLZrK+xlY3Vb2GdtZjx/Rm0b22nb3mfazXub9s/YjzdAbdaLCqfqcTGRl35S40cinDdxuvGfV3g2wEoG70PjQ84QF2kjod+9hBHwkf49dZf81bvW3SkO9jTtYdvlX9LR0kZPRV/iiWXlgyNLzVcGnf7yO8iFP5PIWkrDQq613bTv6jf1RobGhu4e8vd2Le5c3iXrkjTvrWdksYSVK/C6rdIzEnQ8XjH+K9LpQF4xnqGupl1pMvSrtSXjecBT9kpTiROMDc1l8XHF1NaXEqxKua1rtfot90NQ86CkJiXmPDmVqtF9DdRutd20/OHPdhF/vpMMSkW9C3vo++ePiL/G6GksYTEvETW/WIftyEJyeuSpEv0hBs0BDxhJ/hR24/4+Vd/zvzT8+nr6OO9vvcAqA3n95IU/Yv7aa5rhoDuSlwQGA56Pn174nnAQyrEfYX3cfLiSfbP309fbx/lgXKWR5azqnCV1+VMPxPD/Vl59vtp+ZC5vmQ9c/9tLuHzYY7uPOp1CVm9WPOi7hKy2lW3C4DQjhBJkpqrGc0v+zCP/tkIkTsJuDCaBFwYTQIujCYBF0aTgAujeR7wYFuQ2S/OJnzeubBq9ouziRyPeF1G3rNeswj8hXNSOvDjANbfW6DvC0Pf8jzgga4ARUeKhsZFx4oItYa8LiPvqcMK9a5z8ZM6rbD2S8Az8Tzg/XP66a7txlbOdRrJkiSddZ1el5H3Uo+l4Mq8YEdsUl9Labp0zt+0HIO3/FELdtAmXZCm+eFm+SQwGTeAvch2Joow2A8YcGGXC7REq39OPz3zekgVpWT2noLUYynnp8zeY9K2W8594xwqpWT2noobILkrCRW6C/EvbQFPR+UT0bSIZ9/kWibzpzBazm0E4YCL5QgxUQrbtqfeRlAp9bhS6oBS6gC0TE9tQngk5xl89+4/d7GcyRtssurJYiST5uzrgwff01zH2BYuvP3KIz/vR8e0zOBC5DMJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmgScGE0CbgwmgRcGE0CLowmARdG8/yOHr/0TxzTt4EyoBEYXNm5GvgToA/4Oy1VjbJ231rO954f9XzjikYWxBZoqOgqPtmP2m5Z090/0RTLPreM2UWzh8blBeUaq/EfbQHX3T/RFOtuXMfK61fqLsO3tAVcd//ErBbi/EsF8PE/l19+8ksOXBy+jfDpW5/WWE0GmvejtoDr7p+YlQ8OYyfitxd+O2Lsu4Br3o/aAr794+0se3cZbX/dpquE8WX6cORDDY0NrPnbNQRqfNodSvN+1HaaMHQiRPgDf7XuzlfpNlljZixyHlwYTQIujOb5Mfhg/8SKn/p0Qb0XMjx3EtjhbRnZ7Fm1B4D2He2aKxmDT/ajzODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujed8I9kSQ+NfjhM44PfDiW+JE3pJGsLnq/k437cucc+Dd3+ymY10HdkI6rV3N84CnY2lU//CyziqhSFWkvC4j71lxy7kzBpwGsCGk01oG3gd8Rpre5b3YQRsbm8SNCRK1Ca/LyHvhx8IweAFhFKJPRVHK/4vWe03LMXjXw11OA4EC6HxU+mROhlVuEf5KGCywZlkE75bpOxMtAU/PSNNzXw8D8wdk9p6C8GNhVLki+mcye49F259952aZuafKKreIvRHTXYavyWlCYTQJuDCaNIIVeUoawQqR+wy+b9/zLpYzeatWPQDkR5PVc+dGL7nmF1VV11955P+zMtIIVlzzJODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmgScGE0CbgwmgRcGM3zW9Y2vbOJC/0XRj2/87ad1BTVeF1ORr5vsgosfnUxZ7rOjHp+38P7+GLlFzVUdJVrvRHsXeV3URWpGhrHgv67tzAfmqzef8P9VJdWD40roj5tLKCJtoCvjq9macXSib/ABj4CahheD8Rl+dBkdWPtRtZUr9Fdhm9pC/je5r0c6jg0NK6fWz/+C05CaFsIu8ImtSWFvdJ2Pei+b7IK7D66m7fPvT00fm7JcxqryeBabQS7v23/iPG2n24bd3vVrLCVjbqkCPw4AC9D6qkU9iL31uPzfZNV4I1Tb4wY+y7g12oj2O9WfpcV21ZM7sUDQALUEeVqwBsaG/jSLV+i8K8KXXuPqWpobOCRHz5CtC6qu5TMNDeC1bfeVwUk9uWwqtVxCG4LQgDsOucwhRvcK29Q+lP/N1m1+2VV2bHkz4J2VZDekCa9Mu1JsIUZ8ifgUUhv9v9sKvzF84Dvqtvl9VvmbEST1RzOZHrp94/8HoBjO45prmQM0ghWCPdJwIXRJODCaBJwYTQJuDCaBFwYTQKegd1vkzrntDa0223S5/13/j3VkSJx2vkmOHE+QbIlqbkif5KAZ9D3Sh+dDzk9hFKHU3Ss7SB51F8BOrv5LE33NgHw6Z9+StPdTdgD8pX91STgGYRWhkZ8BabKFYEajy5Cn6CSL5eggleWx05D4T2FqAL/r+ntNQl4BsEFQYILg87eiULkiQgq5K/wxDbGUGGnJlWgmLl9puaK/EkCPobIkxGwQEUUBQ8W6C5nFCtsUfl0JVhQeG8h4QVh3SX5Uv5cbOWxYG2Qgq8UELwj6LvZe1BsY4ye3/VQ+Wyl7lJ8SwI+jsJn/HujAziz+KyXZ+kuw9fkEEUYTQIujCaNYEWekkawQkxmBn/HxXKmwvk98qHJag673HNqaE7055mjz5JGsOKaJwEXRpOAC6NJwIXRJODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmje3rLmk+ag2fi+ySpQ/UI1n1z+ZNTzB795kNuvu11DRRnMAu4B5gBRoAdoxrml4Ig3Jcg9mePIhyarD970IDXlwx2iZxb6ZPmIm4H1OMcILcCHQBgn9LciAfeDXJusJs4n6Hy9k9iGGIESbxYK2rJwC+tq1014+8tvX6b3eC/xDXGsoEtHqCHgQZxwvw/8OzC4+p0CPFwEQE/ANTcHnahcm6x2/VcXLc+1cPEHF5lRP4Pyb5S7HvSXD77MmyffHBq/sDpT75Bhp54/Ret/ttL0bBOf//7n3Qn6HGBwQYLfMBxucO5L8fDGMD0B19wcdKKubrL66PpHs78oAHa3Tes/ttL6T63M+pdZFN1T5FKF8PqHr48Yr1szsdl84NwAH279kKbvNHHnO3dS8LlpXNzos79u+5Wf9+Mcjw/aMX1vNx49AdfcHHSiXrrlJW5tuHW4D2WWP8yBYwPOv+UUEIBgPEhghrsz+CtzXuHmf7h5+Ilbxt++53CP80DhrGlY68Kaht2feVwKtAKngEPAH0zvW2Ujx+DjCM4KUr23esLbt/1zG81/2Uxoboj438QpWlWEUu7e21i2pIzFX1s84e3f//L7XHr9EmUry6j5QQ0ld5RMf1Gncc6YFAL3Av+B8yGzAwl4Pot9NUZ4fpjokqjrwZ6sm3bexMCOAUoWuhDsQQlgD/Awzuet64EzQMy9txyLBHwaWYUWhUv9vdxbuCpMuMqDhTr/D2fGXorzofN2nEOXj4HD7r/9IFk2wkOybMT0kmUjxDVPAi6MJgEXRpOAC6NJwIXRJODCaBJwYTQJuDCaBFwYTQIujCYBF0aTgAujScCF0STgwmgScGG0HK8HV53AMffKmbJK4KLuIrKQGqfHjbZtZ10EJtc7eo7Ztn3nJAtynVLqgJ/rA6nRa3KIIowmARdGyzXgP3Gliunj9/pAavRUTh8yhcg3cogijCYBF0aTgAujScCF0STgwmj/D6950SHWwvgbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 216x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "state_values = {s : 0 for s in mdp.get_all_states()}\n",
    "\n",
    "for i in range(10):\n",
    "    print(\"after iteration %i\"%i)\n",
    "    state_values = value_iteration(mdp, state_values, num_iter=1)\n",
    "    draw_policy(mdp, state_values)\n",
    "# please ignore iter 0 at each step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after iteration 29\n",
      "iter    0   |   diff: 0.00000   |   V(start): 0.198 \n",
      "Terminated\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAALgAAAC4CAYAAABQMybHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztnXl8VNXd/9+zZWay7xsEkhCWsBN2iAQExAVw37U/d0sXtfrz6aPPU2uhL221au3jY7WupbVirT/rghYtKIgIhD0BkpCwJJB93yaZzPL745oZhpkkc2fO6DC579fLl/fce+eTO9zvPXPuOedzviq73Y6CQqii/r4vQEEhkCgBrhDSKAGuENIoAa4Q0igBrhDSKAGuENIoAa4Q0igBrhDSKAGuENJohzpBpVLdA9wjlSJmwoQAX5KCwtAYDIcxmUyqoc5TyRmqV6lm2fPyxAzt79u3F4B58+YL0du58xsACgoWC9ED2Lr1SwBWrLhYiN6mTf8C4IorrhSiB/DPf74PwI033iRE7+23/wbAXXfdLUTv1VdfAeD++x8QotfP22+/TV1d3ZABrjRRFEIaJcAVQholwBVCmiFfMr2lN7yX05NO0xXfhVVrRWvWYuwwknEoA323XrbevoJ9mI1mt/1Tvp5CREeET9e4c95Oeo29bvtnFs4ksjNStt7WGVvpMfS47Z9/cD7R3dGy9TZN3IQpzOS2f3HpYmJNsbL1AD7M/pAuXZfb/otPXkxcb5xsvQ2pG+jUdrrtv7LuShL6Eny6xtejX6dD0+G2/6b2m0iyJvmk2Y+wAD8++zimGBNRDVHou/SYDWY6EzrpM/T5FOD9xNbHYug2OMo6s87va41vjMdoMgrTTGpOwtjr1AuzhPmll9KWQoTZ+RDrLb7/+/WT3plOpNn5EOut/mlmmDKItjgfYoPVMMjZ3pFlziLGFuMoG23GQc72DiEBbtFZMMWY0Jg15HyTgwrp5damtsGQ77mDk3w6mfj6eAFX6SStJo3ExkRheiPqR5DSkiJMb3TzaNLb0oXpAYxpG8PIzpHC9MZ3jSezJ1OYHsAk8yTG9I0RqikkwDUWDWqLGmuYlZKCEqIao4hsiiSqIQqNVeM4T2VXYVfJ62asH1lPe3y7o5xZkumXHkBNWg2tsa2Ock55jrTRLyXzoTyTfIbmmGZHOfdkrvOgXb7eqfhTNEY2OspTz0z1Sw+gIqaCOmOdozyzYaZfmqURpdToaxzl+W3+d/ceDjvMae1pR7nAVOC3ppAAV9lVjD4wmspplZhiTJhiTNSPqSfMFMas7bOIbY5lWss0FjYs5LO0z9iZuNNr7dbkVpfy3H1zATBYDNxecjtH447yWcZnsq63ObHZpbzgwAIAVlSsILY3ls+yP6M0odRrvYb4BpfyoqJFAKS1pbGsZBl7Ru1hV9Yur/XqYupcykuOLHFsX7njSpqimtg+eTsNsQ3nfnRAqiOrXcrLji0DILc8l+zKbAqnFVKWVea1XpWxyqUsIsBPhJ1wKQdNgAPEVccRUxtDqi6VRGMiG/M20hLZAqPg4a8fdpx35ekrqTHWsNtL3UmFk3hh4wsDHp/aNBWNXcOnMq71rk/u4ubdN3s8ZsfO9Yev54n8J7zWW7J3CY999NiAxxecWECnvpMPvdSbc3wOv13/2wGPG3uNXLT3It5a+pbX13jLrlu489M7BzxesLvA7UEdjJuKbuLazdey8ZqNXn9mKNZtWEfnwk56It1f3H1FSIDbVXa64rqIbI6kvree+t56Ik5G0DK5hUMJh3go7yEmtE1gWe0yPkn/hONRx73W7tP08ci8R9z2h1nDuOPoHRTHF7MzdSfs/qnXmt9kfMMx4zG3/Rcev5AEUwKbszZj0Vi81muJaOE3K37jtj+5PZmLD1/MzuydlCWXQYmXgip4/ornPR66fMflNMQ0sHfsXq+vD+DUyFP86cY/ue3POZHD+OPj2T19N81xzR4+6ZmY1hhiW33r2RmMsJ6w4Atwm9pGWX4Zhg4DxjYjaqua1lSpaRHdIL1pl8SUUBLj7R0eGrPGzEuTXxKmB7Ale4tQvfroetbPXy9U84MFHwjVK88qpzyrXKhmMCEkwNU2NckVyXQkdtCe3I5NYyOsJ4ykk0mklIvrXVBQkIuwl8yRh8V1QQHkbc0Tqgcwb+c8oXoF+/1/CTqbFUdWCNUDWH18tVC9G2pvAGDGmRnCNO9ovwOA/JJ8tswU+yuqDNUrhDRKgCvIxthlZGzJWACm7J0iRHP++1I348L/t9A5HiEAJcAVZNNr6KXH0INVbaUpqUmIZmtyK3aVndakVr9Hv89GCXAF2dg0NvbO20tdWh3VGdVDf8ALymeW05bYxuH8w0L0+pH9kqlWi30mgl0PQKsVNh4GgMHg/8SkcwkPDxeqFxk5+OzKxumNNE5vJBLvZmFGRw8xuzIa9v1w37ebQ8/E1Gg0Q54DPljWYI/X5ysoBIq0tHSqq6v9t6ypVKp7VCrVHpVKtQe8H8pVUAgGZNfgs2aJeQPYs6cQgAULFgrR27HjawCWLLlQiB7AF19IfbKXXbZSiN7GjR8DcMMNNwrRA9iw4W0A7rzzLiF6r732KgAPPPAzIXq///1zAPziFwPP1fGFV199VUwNrqBwPqMEuEJIowS4QkgjrP/r0IWHMIe7m4QnbptIeLv8Lqy9F+z1aBCe9s00n03H38z9xqNJeNaeWUR1RcnW2zJ1Cya9u0k4vzifGFOMh08MzodjPqRb1+22f8WJFT4ZhAHeSXvHo0n4itorfDIJvxb9Gh1qDwbhjptItib7dI1/UP+BNlWb2/67rXeTSqpPmv2I7eAFYupi0Hc5Da1as39/Iq4hzsV07K8eQEJTgovpOKzPP5Nwcmsy4T3Oh9hfk7BogzB4MAnb/OuLz+pzNQiH2/zvhx9rH0uc3fkgh+O/pvAAT6xMJK7Ot9rGE8mnk0lo8G05goFIq0kjqcm/5QjOJqMhg9RW/2qas8luzRZqEAYY1zWOTFOmML1J5knk9OUI0wOYbpvOBMFrXwoP8MZRjXQkOH/CRh0ZBUB6dzrTmqexLWWbx3U6BuJc03FWaRYAGpuGZZXLOBp/lMroSlnXeK7peGyFNHFoUu0kwi3h7Evbh1Vj9VqvKqmKpijnnIxJVZMAiDJFMfPETPZn7qct3P0neCCOxx6nPrzeUc6rd04dzivOoymuiVPpp2TN2SiLKKNWX+soz2uVpg4n1SSRVp3G0clH6dP3ea13rkF4sWmx9xczAAfUBzhlP+Uor7D7P31YeIC3pbjeyHfWv+NSXlS3iBcmvOD1eGhLUotL+W9v/M2lvLB6IbtTd7NDxjU2JbhOEPrLq39xbNuxU3CygOcWPOe1Xn1svUv5jZfecCnnnchj44yNeOtePNcg/PIfXnYpW1VWapJq2Hih937Ic03CLzzj6nOdWDSRj6/62Gu9EzpXg7CIAD+mOuby0K6wBmGAT9o5iZv23oTaLnXQ9DvoJ7dOJtISyeHYw7SGtQ4m4cL4/eNZdXAVSSapSbE7RbIrGywGpjZNpU3fxtH4o1A7mIoriwoXsXq/0wiwN13yN86snolVZWV/6n5sKpvXenNK53DNrmscC/4cHHUQgNTWVFLaUzgTf4b66PrBJFzIP51PweEC0uultVGOjDniODaxYiJmnVmWAx7gsqrLWP31alR2KYKOjZc8qVnlWWitWk5lnaLX4P5SPxA/3fFTrvrsKj54QJyFbt2GdegW6ehK8f4XfiiEB3iPpof3R7/vtv99+/tEm6Np1Xsf3P3sSPdcP2/q2USbvk322ijNxmY2jnev/f495t+o7WpMOveekcGwqq1szd3qfsAOMaYYWc2TfipGV1AxusJt/95Je+nR92DTeP8AAvQYe9i90H0tg8L5heh79Jgi5H3n8wXhAT4QNpXNp+AejFaDWL1erfc1mFeo8Cm4B6M73L0b0R9sGlvIBjcoAz0KIY6wGnzqlqlDnySDmV/NHPokmczfJSabRD8XHhI3sQtgdYVYgzDA9TXXC9W7s11aPGha9TRhmvfZ7gMk0/GuRd6vAOYNSg2uIBu1RU3GkQwAEqrEjFGk7U8DYOQesf3/SoAryCasJwyNTXLUJJwRE+ApxdL6OYmliYrpWOH7pSeyh1MTT2HRWjg+w/tl+AajYqnUY1RxYYVQ0/F31ouiEFoUFxRTOrcUi977NRwHoyO9g50/3IkpQWyPjuLJVDgvGTkyg6qqKsWTqTC8kV2DT5/u/SSkwThwYD8As2bNFqIn2uMJTp+nuEah9G/9gx/8H0F6sH79nwFYs+ZHQvT++McXAXjssV8K0Vu79lcAPPPMs0L0+nnuuefE1OAKCuczSoArhDRKgCuENMK6CQ9fdJi+cPcJ8+O+GEd4m3zrkWiPJwj2eT4AxAIbcKYmyQRuA3oA94wmQ/JexnsezSArT68k3uxbKsW/Jv7VY5LVa5uuJdEiP5Xi86rnPfon77Hd47N/8tedv6bF3uK2/8HwBxmhGeGTZj/C+8Gja6MJ63J6HLW9/v0J0R5PCIzPUyQju0YSZXGaoEUkWR3dO1qoJ3OsfSzxOB86Ef7JiZqJJKidI6ORKvnZp89F+J2NPxVPbI245ESiPZ4QGJ+nSHI6chjVPUqoZq4pl6zeLGF6M+wzhPsn5+jmMEUnZr3xfoQHePPoZjoTncsUjCySJs+o7WrizHE06eWtJz2Qx9MfBvJ5+sQMpKYJ4MWiqF5RHlXukrR1dpOzKzWiM4IeY48szyjAUeNRzujOOMr5nfmANHHKYDLQHSVvnvl+1X5OIdY/ubtvNxVWp8njCsMVfmsKD/D21HaX8s8//jkAk9smY7AZKIou4t2Md73WO9fjKSLAz/V5+hXg4/28GA+cjjjtUn7wgwcd2znHc+gN66Uwr5Bj49xTIQ7EKf0pOGv1iX7NzLJM6XjOKfYt2ue13jGV698WEeBHrEfgrOc2KAP8gi8uYO3WtQMen9Axgag+7xfZuWjLRTyy7REenvXw0Cd7yboN6yhTl1ER624Jk42nl0w/WVyzmDXvrSG50fNCOlqLlqxTWbIC/JYjt3Dn3wdOBDvi5AiK5xZ7rffj4h9zzT+u4YvHvvD6M0OxbsM64q6IozdJnLNKeIBXhVfxs+nuK5OmmdKY2jqVr5K+olvr/c+hiBes71JXCCrYeKlnx/zkosm0xLdwJv2Mx+MD0ZjeyN/X/N1tf0JNAsnVyRybcgxLmPcTp3TdOll/31tUFoFTCfkOZxPWGGuoMdZ8V38uZCme4n0t6w1NaU00pYnJsxOMKAM9CiGNsBp80meTREkBTo9nTq245cH6fZ75Jfn8bfzfhjh7CH7vYd9J4HHfJa+uutr3Dw/ALY23CNW7334/2GFc7Thhmv8d+d8A5JXkcWTlkSHOlkfQ1+CxZqlPPdwiJslShFkasYzplb/6q4ITQ5v0DqPp8S4Z1FDo2qQ2fViLfwuhnktQB7jRYuSC+gsAWF0pxnF+dblUS1528jK0tuAawTxfMLQYSKiQBsoyt2UK0cx+IxuArL9kgbw1jQYlqAPcpDVRHlWODRuFiYVCNPek7MGOneL4YixqMXar4UZPbA+dSZ3YVXZqp8pYM28QGuc1YsdO88xmoVEZ9FXYBxkfMK1lGhVRAvqsgaPxR9mevt2xxqGCD6igdFUpsSdj6UoVs45g0+wmDLUG6i6sG/pkGSieTIXzkszMLE6cOKF4MhWGN997nsz8/AuE6G3f/hUAy5YtF6IH8O9/fw7A6tWXC9H78ENpqeHbb79DiB7AG2+8/u2WWN/os896vz76YDz4oDSq/eabfxai18/jjz8upgZXUDifUQJcIaRRAlwhpFECXCGkCdpEsIULCz0ahKfvnE5kp29eve2ztntMBDt3/1yfEsF+Pulzj4lgC44W+JQI9t0R73pM2rq6erVPSVsDYYxe17HOo0H4oYiHfDYIP1T1EE1W9xmNv0r7FaP1o33S7Oe8SAR7dtJWXZ//85ATmxOFaqa0pRDR63Tl9yej8pWM7gxX07GfBuFAMFHrahCOUPmWffpsphmnkax1mjyiNf57AIM+EWxqdapwg3B6bTrJzb6lnfbEqMZRpLWlCdMb2zmW0Sb/aq5AM1c3V7hBeFHkImZGiM3s8Z0lgs1tzWV59XI+Hfkpx6K8t1rVptfSFuf0ZWaXSZNy9BY9txXfxpGEI+xM3ynrGqtTq2mJcf7Mjj8hGSuXHFtCQncCX+R8QVOE9yaAysRKl0Swk09PBiCpLYmL9l9E4dhCjqV7/52PRR6j1uCc4zG3Za7Xn/VIAIzRu/p2UW4td5SvNFwJgLZdy5hXxtA0u4mm+fKMFNs6t1HSU+Io35xws9/X+Z0ngr2n7B5eGveSrESwLTiD8a3X33I5nt6ZzojOEchxBjbGN7qU17+y3rFtx864hnE8teQpr/XqYlznT7z24msu5cv2XsYXvV/gbUbJqnDXpK1+B3gAjNFHLK7zttc+6urDNZ4xYqwzIoeDpoMu5aAM8LG7x5Jdme0o/2ay9CYzpWUKBXUFbErfxMnIk17r5R7MJfN0pmNq67MzpVVKDVYDtxXfRkl8CVtGbYGmH3utmVecR3a18xr/d8H/ArC8bDlxpjg2j90sa6bh7PLZTKhxrhHy+lJpdDGlNYWlB5eyb8w+jmQcAS/dZpeeupSH33iY9bevH/pkbwiAMfp24+3kdeahskqDiYcfPQxI87qzX8+mJa+F2uW18I33mus2rCP75mz6MrxPKT4UwgPcprLRZHD/afoy7Uu+TP3SpxHlDr370mMAT8590ic9i9pCa7h7js13p327nIVcTRW0RbgvZ9YW0UZZepnQlBzBRF+ceyCaE80U/booaL7zdztdVvSXDna9QGkGO0H0nZWBHoWQJmgTwc7+Wkzmh7PJ35MvVG/5YXEzFwGuPXMtAJHt/i86CQTEGP2LqF/4/uEBeCbjGQBGl4ymmmqh2koNHmzY4YJt0hTiBV8t+J4v5rsj/g1ppdrEVxKHjydzWKIC7FJ3pcoeRI3ZQGOTvrMdu9A2vBLgQcje2XuxaC0cmHHg+76U74y2K9qwG+y03tCqJIINdepT63n71re/78v4TrEmWKl6qWroE2WimI4VzktycsZy7NgxxXSsMLyRXYNPmyZmsZyDB6X25XBMBPvQQ/9XkB4888zvAFi7dp0Qvccek7oBX3rpZSF6P/zhvQB89NHHQvT6+dnPfiamBldQOJ9RAlwhpFECXCGkEdZNeOSiI/RFeEgEu2UcxjZ584JheCaCfSXiFdrV7W77b+26lWSbbw6kZ23P0or7zMk1qjWkqeS7kB6tf5RmW7Pb/v9K+C8ydBk+XeOdRXdSb6532/987vNkh2d7+IT3iE8EW6MkgvWXbEs2sTZnrlGjXX4FcS7jGOeSuDUC/zyUU/RTSNIkOcqRav/nz8yOmU2a3vnQxWj9X8M9IIlgY2rELS4/HBPBTu6bzFjLWKGaM1UzyVXlCtNbaFzIdMN0YXoAyxOXMz92vlDNgCeCHVEkLSUQY44hpzOH/XH7sanOmU1jh9y2XFp1rdREuCaq8jURbFxPHBntGRxKOuTWyxfsiWCLdcWc1jhzZS7pXeLYTi1LpTOhk84E9+UlBmOvfS8n7Ccc5UvVlwJgaDIQWRNJ48RGWW9kX5u+psxc5ihfF32dtGGB6N3RdE3uwhotL1nt542fU9zhtD3dnXG3rM97Qnwi2DTXNuS7f3VN+np5zeW8MOYFR1lv1XPfkftI7nVtY96QfwN14XU+JYJdfmo5C6sXorPruL78eqfmnBuoM9YFfSLY49rjLuVnnnjGpWzVWDmTe4ZDKw55rVlGmUv5yceedCn3xPRw8C5XT+RgFPUWuZR/8R+u02htOhv117q3qwejsM01yUFQBvj0bdN55OtH0FuldnObVgrQGIvUbOnSdGFROQeL7NhpD2t3BHi7rt2xH+A/3/1PKs2VlMW43qDB6NJ1YVfZwQ7d2m6sKquL5g8+/QEV6grpHH8JgN9xdfdqVm5dSUax9NLWE+FcrMjQJb079EbIS5Z6e/PtXLf+OtQWqZo2R0ov8GGd0vtSX0Qfdo33/x5rItdw6fpLMZyUrscSLd1TbbsUUnatHWuEvBp83YZ15P04D1umuPmywgO8NayVJyY84bbfaDUywjSC8ohylyaDWWPm5fEvM7JrJJ3aTlr1rQ4dgHcz35XdBt+RvoN9yftI60rjRIzzZ7lNLz1su1J3BXUbHBWU5pdSml/qdijuTBzdsd2yA7wnoYfCB93TwOg6dRibjLSPdu+9GQy71s6ZH3lIRmuD8KPhmMaasIfZ4RVZssL5zroPTBoT5ZHlAx4/Nz+7v/Roe1yCO1RoGeG+bJo/9EX20RcpzsWOGroneZ/JOtAoAz0KIY2wGnziZxNFSQHiPZ7gTAQrhAD4He/u8v+l6lweVD8oVO+JZPfmp7+8NkVaKCny8Ui6EVv7KzW4QlCgKZISymr2iUks248S4ApBQdibUm9O2DthiulYIfQw3yx1W5qvMw+vRLAKwwPrDCumR01YZ8jrOx8KxZOpcF6SmzuRI0eOKJ5MheGN7Bp8wgQxuclLSo72qwrR6/c7il35UdK8994fClF7+eWXAHjkkUeF6AE8+aTUbff8838Qonf//fd9uyX2vuzZs1eQnsStt94qpgZXUDifUQJcIaRRAlwhpBHWTVi+qhxLhPuaKZn/ysTQKjMNXgD8joHQfCvpLY95La9uuJpES6JsvRfDXqRN5Z4p4g7zHaTYU+RfIPCrtl959FA+HPUwI7Uj5YkF4r4AqzavosZU47b/rQveYnyMfxPuhfeDR5yJcMwxBtD0ih16DUZG9Ywi2uq08xht/nkoc6w5xNqdnsxwu28m67OZpJtEotr50InwUIrmguQLGBnhfOjiwvy3KgoP8NjjsUSdkZ81+HxmQvcEsnr9cAWdwzTbNMbZxgnTA5gXNo+pYeInsInk8lGXszh1sVBNnwNca9N6zETWmt1Kd7JzRljKft9+WoGA+B0DoVkSXkK13pmZYGG7c/k4tUWNTStvcsVB9UFOqU45ysutzkwSaosam8Ymuxdvp3kn5RbnfPyrwq+SJ3A2gbgvwAeVH7C3ydmd+NCkh/zWlB3gcX1xXN5wOXPb5/JR4kfUhEltpypbFRYsdI3oogtnX7lfAR4Av2MgNCsNlS7l6w9IPtCE+gRyD+ZSMaGCotlFnj7qkXKNqzHk5sPOfJFTP5hKZ2InpctKaRntvfnhcN9hl7JfAR6I+wJ8Vf+VS/l7CfBZ7bOY0z4HgFWNqxz7t1i30EUX6zasY3PvZmr1tQNJeE8A/I6B0Lz6xNX85M8/GfD4mJIxNCV5n/X3avPVrHlqDRqL5/eXqIYoxm8ez847vM/w/MsPf8nYzLE0FcjLPuyRQNwXJE/miqdXoE4T17knO8A/j/+c0/rTLGlZwoaUDTSHSW/oTTrpH+6lES8NuzZ4R0wHG+7d4LY/qiWKGTtnUDKthPr0evjKw4c9oYLND2/2eGjyh5PpSOmgKk/eYvHabi3hleE0ISDAA4i9S4AR/Czkt8FVcDTyKEcjjw597jCnI66DbZdsE6pZvNrLdMkKgDLQoxDiCOsmzPkoR5RUQPyOgdC8ueHmoU+SwY/MPxKqB/DLmF8CMLVkKi2z/HTkB+K+AB8t/QiAjsc9p2z3B6UGHw5820upsgRvWkJ737dtb/cFhf1CCfBhQPK/pFXDYg/EEtYYNsTZ3w/da7od/7fbxL1oKgE+DOga14VNY6Mvqg9znOAqUhDa+VpQg2a2BpVa3C+N4skcBnTldNGa10r75HYI0qlBYTeEYd1rxfATmRPzhkAJ8GHC6VvELo0nGlWEivAX/Z9U5qarmI4VzkemT5/B/v37FdOxwvBGdg0+fry8zAIDUVraP5lBrLn1kksuFaQHn376CQDXXXf9EGd6x9///s63W+KN0f/4x3tC1K655upvt8Tel7Y2ecszD0VBQYGYGlxB4XxGCXCFkEYJcIWQRlg3YcWqCiyR7g6f0Z+ODg7TMfDl9C8x6U1u+xcWLSS6W7415eOcj+kOc1/PennFcuJ6ffATBuB7ryldQ0Ofe+fA02OeJsso02YXoPsy5bUpVLZXuu3/6uavmJrsn81uWJqOk1qSCO9x9rmG9fk3fJ3WkUak2Wni7U/AFUzMjJpJaliqoxytFeg1E8TFWReTFet86BLD5a9McC7CAzymIiboDQ8ZDRmktPhhpTuH7NZsRnSMEKYXCJbGLWVO9Jzv+zIG5dbJt7IyZ6VQTeEB3jamDVOKsxmQvM+3HOtAwMytVUlVNEU5nS0TK6X0K9NPTSeyN5LCrEJ6da5ZzGLbY8k7msfBcQdpinN1xRyPPU59uDMn5Iy6Gf5dYAC+9+aWzRzucvoyb0+73XexAN2XvxT/he2ntzvKv1nsY5vnLIQH+LmmY78CPEDm1oY41zbpmy+/6di2Y2fWyVn8z9L/cewbWTuSldtWokLF+FPOi9r6wFa6w7qpiXJdtMbvAA/A997b4br4pV8BHqD78q8T/3IpB2WAr9uwjvST6Tw1+in/xQJkbl10cBEFRwsc5S0TtgCwoHwBADvG7MCqdi7E3hTbRElmCbknc6lKqaIyVXoh6q/lb9x1IyqjCovW/SXbJwLwvddtWMfs5Nl03idgoC6ApuO7X78b4yT/Fk46m4BMtlIJHakTT6exkz1Z7nNq9o/ajwoVFo1roJoMJr6c8yU7pu/ArDM7BvnMOmnqaWV6ZdC3wYcrymzCs7BqBk+fYQ4LzrnUCgOjDPQohDTCavAxH40BIL80nxMGP1NoB8jcuvjAYv8EzmFludgurUB87z+O/yMACSUJ9CbLy2/vRoDuS9Gd0qpf+x7f55+QB4TW4Nnd2QBk9WQRZQnuvvDhhG6vDgD9Nj24D+QGBbW/l1ZCq/lNDXJmuA6F0AA/e03scJt4d4aCb2hqpNFku8qOyhScHQCdO6Tene793c50SwIQGuCFUYW0aFvYH7mfurA6kdIKftCzvAdbhI3eZb3Y48UujSaKEY+PABWMWDsieE3HdpWdtZlrsagE9QcriEGAGfszAAAFN0lEQVQPrS+0YtcHZ3ADGCcamVI6BW2S2I49xZOpcJ6iwm63K55MheGN7Bp86tQ+IX/40KGDAMyaNVuI3p49hQDk518gRA9g+/b+9Y7F+hM/+uhjQXqwapXUVVlUJGbV2SlTJgPiPJQxMf2zscS/3AqpwRUUzmeUAFcIaZQAVwhphPXJHF1xlL4I9/b52M1jMbbJn/546MJDmMPdJzdN3DaR8HbfBpEKFxbSa3Qfrp6+czqRnTLzRgbAn3hn0Z3Um+vd9j+f+zzZ4dnyBYEVn66gurvabf+7S99lQuwE2XoB8U+OAPKBDMAIdAP1SB12fiYSET6bMKomCn2X05Oo7fXvT8TUxbjqmf2/5LiGOIwm50On69P5rSmS2TGzSdOnOcox2hi/NQtSC8iIzHCU4/T+JVkV5p+cCFyD1JZoAMoAPVLQTyH4Ajz+ZDwxNf7fkH4SKxOJq/M/4+3ZpFanktCQIFRTJMsTlzM/dr5QzSszr2TpiKXC9IT4J3XASqTgLgLex7FYPyrAf8+x+ABvzmymK8lpWUs/lA6AzqYjpSeF0+HyVjltHNVIR4IztcWoI6Mc2yO7RlJrrPWYkFZn1ZFoSqQm0j0Hem16LW1xzpzw2WW+/fwDAfEnft74OcUdzm6/uzPudmyrT6qxJdpAZovq/ZPvs6fROUj382k/B8DWYcNWbUM7Xl4oDOSftNvtdO/pJnx6OCrdEL14GUB/a3MrzuAGqUdVwLCL8ADvSOugA2dAPv3e0wBkmjIBOBF+gjdHvem1XltKm0u5X89gNZDck0y3ppv3Rr/nMr6aV5vHJScuwWA1UBNR45g6UDK7hF5jLy1JLbTgzFfjV4AHwJ9Y2FboUr7v9/c5tjVlGux6O+YbzfRd7f2YxNbarS7le397LwCWIunfRjdPR9RT3s8APdc/edva2wDoq+mjr7oPbZKWUS+OYtCnPuKs7dZv/78MqT3ez+NeX5JHhAf4vC/m8eTWJwc8ntqTisHm/UJA43eNZ+2WtST3ejYvh9nCSOtOc9mX2pWK1iZ9tbQu5zGNXZpV98A/H+BA+AGPNb9sAuBPfDTrUZa8sARt4QC3xwrqY/I6wH4X/jtm/sdMR9mC63e3lFiwm7wf9Fu/ZD1Z12ZhqZd0unFdAMnaaqW3bIj5511nbUcDzUAlcAjwb70fB8IDvMZQw/1T73fbn9CbQG5HLrvid9Gn9r7msagtPD3labf9apua+Q3zKY0ppdHQCLW/dhz7ZMwn7EndQ2ZbJoVphdhV0o1r1UvVxKasTUHdBkcNPY/1eDyk/VyLbbQN2zibx+MDoZ2qJbHIvVHbV9aH5ZAFw+WGoZsUZ6GJ1DD1mHsU2kw2Gl5vIHZVLPpRevivQUSqkHpMwoELgA+QXjLbCd4AH4gmfRPb9duHPtFLbGobX6d8PeDx+oh66iPcu9zOdyzLxc7U1I3ToRsnrhdJbVST8mMvF1XqAz4BrkJ6l0kDTgPi+igU07HC90wxUo29EOmlczpS06UcODzI57xEWIDnbsoVJQXA1C2CfqPOYvbXYiZ2AQHxJ7425TXfPzwAmy7ZJFSv3z8plMpv/wsAylC9QkijBLhCSKMEuEJIowS4QkijBLhCSCPTsqbqAEoF/v1EoDGI9QKhORyvMRDfebTdbk8a6iS53YSldrt9lo8X5IZKpdoTzHqB0ByO1xiI7+wtShNFIaRRAlwhpJEb4H8S/PeDXS8QmsPxGgPxnb1C1kumgsL5htJEUQhplABXCGmUAFcIaZQAVwhplABXCGn+P94mjRGy6KGCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 216x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import clear_output\n",
    "from time import sleep\n",
    "mdp = FrozenLakeEnv(map_name='8x8',slip_chance=0.1)\n",
    "state_values = {s : 0 for s in mdp.get_all_states()}\n",
    "\n",
    "for i in range(30):\n",
    "    clear_output(True)\n",
    "    print(\"after iteration %i\"%i)\n",
    "    state_values = value_iteration(mdp, state_values, num_iter=1)\n",
    "    draw_policy(mdp, state_values)\n",
    "    sleep(0.5)\n",
    "# please ignore iter 0 at each step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Massive tests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter    0   |   diff: 1.00000   |   V(start): 0.000 \n",
      "iter    1   |   diff: 0.90000   |   V(start): 0.000 \n",
      "iter    2   |   diff: 0.81000   |   V(start): 0.000 \n",
      "iter    3   |   diff: 0.72900   |   V(start): 0.000 \n",
      "iter    4   |   diff: 0.65610   |   V(start): 0.000 \n",
      "iter    5   |   diff: 0.59049   |   V(start): 0.590 \n",
      "iter    6   |   diff: 0.00000   |   V(start): 0.590 \n",
      "Terminated\n",
      "average reward:  1.0\n",
      "Well done!\n"
     ]
    }
   ],
   "source": [
    "mdp = FrozenLakeEnv(slip_chance=0)\n",
    "state_values = value_iteration(mdp)\n",
    "\n",
    "total_rewards = []\n",
    "for game_i in range(1000):\n",
    "    s = mdp.reset()\n",
    "    rewards = []\n",
    "    for t in range(100):\n",
    "        s, r, done, _ = mdp.step(get_optimal_action(mdp, state_values, s, gamma))\n",
    "        rewards.append(r)\n",
    "        if done: break\n",
    "    total_rewards.append(np.sum(rewards))\n",
    "    \n",
    "print(\"average reward: \", np.mean(total_rewards))\n",
    "assert(1.0 <= np.mean(total_rewards) <= 1.0)\n",
    "print(\"Well done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter    0   |   diff: 0.90000   |   V(start): 0.000 \n",
      "iter    1   |   diff: 0.72900   |   V(start): 0.000 \n",
      "iter    2   |   diff: 0.62330   |   V(start): 0.000 \n",
      "iter    3   |   diff: 0.50487   |   V(start): 0.000 \n",
      "iter    4   |   diff: 0.40894   |   V(start): 0.000 \n",
      "iter    5   |   diff: 0.34868   |   V(start): 0.349 \n",
      "iter    6   |   diff: 0.06529   |   V(start): 0.410 \n",
      "iter    7   |   diff: 0.05832   |   V(start): 0.468 \n",
      "iter    8   |   diff: 0.01139   |   V(start): 0.480 \n",
      "iter    9   |   diff: 0.00764   |   V(start): 0.487 \n",
      "iter   10   |   diff: 0.00164   |   V(start): 0.489 \n",
      "iter   11   |   diff: 0.00094   |   V(start): 0.490 \n",
      "iter   12   |   diff: 0.00022   |   V(start): 0.490 \n",
      "iter   13   |   diff: 0.00011   |   V(start): 0.490 \n",
      "iter   14   |   diff: 0.00003   |   V(start): 0.490 \n",
      "iter   15   |   diff: 0.00001   |   V(start): 0.490 \n",
      "iter   16   |   diff: 0.00000   |   V(start): 0.490 \n",
      "Terminated\n",
      "average reward:  0.87\n",
      "Well done!\n"
     ]
    }
   ],
   "source": [
    "# Measure agent's average reward\n",
    "mdp = FrozenLakeEnv(slip_chance=0.1)\n",
    "state_values = value_iteration(mdp)\n",
    "\n",
    "total_rewards = []\n",
    "for game_i in range(1000):\n",
    "    s = mdp.reset()\n",
    "    rewards = []\n",
    "    for t in range(100):\n",
    "        s, r, done, _ = mdp.step(get_optimal_action(mdp, state_values, s, gamma))\n",
    "        rewards.append(r)\n",
    "        if done: break\n",
    "    total_rewards.append(np.sum(rewards))\n",
    "    \n",
    "print(\"average reward: \", np.mean(total_rewards))\n",
    "assert(0.8 <= np.mean(total_rewards) <= 0.95)\n",
    "print(\"Well done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter    0   |   diff: 0.75000   |   V(start): 0.000 \n",
      "iter    1   |   diff: 0.50625   |   V(start): 0.000 \n",
      "iter    2   |   diff: 0.39867   |   V(start): 0.000 \n",
      "iter    3   |   diff: 0.26910   |   V(start): 0.000 \n",
      "iter    4   |   diff: 0.18164   |   V(start): 0.000 \n",
      "iter    5   |   diff: 0.14013   |   V(start): 0.140 \n",
      "iter    6   |   diff: 0.07028   |   V(start): 0.199 \n",
      "iter    7   |   diff: 0.06030   |   V(start): 0.260 \n",
      "iter    8   |   diff: 0.02594   |   V(start): 0.285 \n",
      "iter    9   |   diff: 0.01918   |   V(start): 0.305 \n",
      "iter   10   |   diff: 0.00858   |   V(start): 0.313 \n",
      "iter   11   |   diff: 0.00560   |   V(start): 0.319 \n",
      "iter   12   |   diff: 0.00260   |   V(start): 0.321 \n",
      "iter   13   |   diff: 0.00159   |   V(start): 0.323 \n",
      "iter   14   |   diff: 0.00076   |   V(start): 0.324 \n",
      "iter   15   |   diff: 0.00045   |   V(start): 0.324 \n",
      "iter   16   |   diff: 0.00022   |   V(start): 0.324 \n",
      "iter   17   |   diff: 0.00012   |   V(start): 0.325 \n",
      "iter   18   |   diff: 0.00006   |   V(start): 0.325 \n",
      "iter   19   |   diff: 0.00003   |   V(start): 0.325 \n",
      "iter   20   |   diff: 0.00002   |   V(start): 0.325 \n",
      "iter   21   |   diff: 0.00001   |   V(start): 0.325 \n",
      "Terminated\n",
      "average reward:  0.669\n",
      "Well done!\n"
     ]
    }
   ],
   "source": [
    "# Measure agent's average reward\n",
    "mdp = FrozenLakeEnv(slip_chance=0.25)\n",
    "state_values = value_iteration(mdp)\n",
    "\n",
    "total_rewards = []\n",
    "for game_i in range(1000):\n",
    "    s = mdp.reset()\n",
    "    rewards = []\n",
    "    for t in range(100):\n",
    "        s, r, done, _ = mdp.step(get_optimal_action(mdp, state_values, s, gamma))\n",
    "        rewards.append(r)\n",
    "        if done: break\n",
    "    total_rewards.append(np.sum(rewards))\n",
    "    \n",
    "print(\"average reward: \", np.mean(total_rewards))\n",
    "assert(0.6 <= np.mean(total_rewards) <= 0.7)\n",
    "print(\"Well done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter    0   |   diff: 0.80000   |   V(start): 0.000 \n",
      "iter    1   |   diff: 0.57600   |   V(start): 0.000 \n",
      "iter    2   |   diff: 0.41472   |   V(start): 0.000 \n",
      "iter    3   |   diff: 0.29860   |   V(start): 0.000 \n",
      "iter    4   |   diff: 0.24186   |   V(start): 0.000 \n",
      "iter    5   |   diff: 0.19349   |   V(start): 0.000 \n",
      "iter    6   |   diff: 0.15325   |   V(start): 0.000 \n",
      "iter    7   |   diff: 0.12288   |   V(start): 0.000 \n",
      "iter    8   |   diff: 0.09930   |   V(start): 0.000 \n",
      "iter    9   |   diff: 0.08037   |   V(start): 0.000 \n",
      "iter   10   |   diff: 0.06426   |   V(start): 0.000 \n",
      "iter   11   |   diff: 0.05129   |   V(start): 0.000 \n",
      "iter   12   |   diff: 0.04330   |   V(start): 0.000 \n",
      "iter   13   |   diff: 0.03802   |   V(start): 0.033 \n",
      "iter   14   |   diff: 0.03332   |   V(start): 0.058 \n",
      "iter   15   |   diff: 0.02910   |   V(start): 0.087 \n",
      "iter   16   |   diff: 0.01855   |   V(start): 0.106 \n",
      "iter   17   |   diff: 0.01403   |   V(start): 0.120 \n",
      "iter   18   |   diff: 0.00810   |   V(start): 0.128 \n",
      "iter   19   |   diff: 0.00555   |   V(start): 0.133 \n",
      "iter   20   |   diff: 0.00321   |   V(start): 0.137 \n",
      "iter   21   |   diff: 0.00247   |   V(start): 0.138 \n",
      "iter   22   |   diff: 0.00147   |   V(start): 0.139 \n",
      "iter   23   |   diff: 0.00104   |   V(start): 0.140 \n",
      "iter   24   |   diff: 0.00058   |   V(start): 0.140 \n",
      "iter   25   |   diff: 0.00036   |   V(start): 0.141 \n",
      "iter   26   |   diff: 0.00024   |   V(start): 0.141 \n",
      "iter   27   |   diff: 0.00018   |   V(start): 0.141 \n",
      "iter   28   |   diff: 0.00012   |   V(start): 0.141 \n",
      "iter   29   |   diff: 0.00007   |   V(start): 0.141 \n",
      "iter   30   |   diff: 0.00004   |   V(start): 0.141 \n",
      "iter   31   |   diff: 0.00003   |   V(start): 0.141 \n",
      "iter   32   |   diff: 0.00001   |   V(start): 0.141 \n",
      "iter   33   |   diff: 0.00001   |   V(start): 0.141 \n",
      "Terminated\n",
      "average reward:  0.738\n",
      "Well done!\n"
     ]
    }
   ],
   "source": [
    "# Measure agent's average reward\n",
    "mdp = FrozenLakeEnv(slip_chance=0.2, map_name='8x8')\n",
    "state_values = value_iteration(mdp)\n",
    "\n",
    "total_rewards = []\n",
    "for game_i in range(1000):\n",
    "    s = mdp.reset()\n",
    "    rewards = []\n",
    "    for t in range(100):\n",
    "        s, r, done, _ = mdp.step(get_optimal_action(mdp, state_values, s, gamma))\n",
    "        rewards.append(r)\n",
    "        if done: break\n",
    "    total_rewards.append(np.sum(rewards))\n",
    "    \n",
    "print(\"average reward: \", np.mean(total_rewards))\n",
    "assert(0.6 <= np.mean(total_rewards) <= 0.8)\n",
    "print(\"Well done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submit to coursera"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from submit import submit_assigment\n",
    "submit_assigment(\n",
    "    get_action_value, \n",
    "    get_new_state_value, \n",
    "    get_optimal_action, \n",
    "    value_iteration, \n",
    "    <EMAIL>, \n",
    "    <TOKEN>)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}