Tennis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Collaboration and Competition\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will learn how to use the Unity ML-Agents environment for the third project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program.\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Tennis.app\"`\n",
    "- **Windows** (x86): `\"path/to/Tennis_Windows_x86/Tennis.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Tennis_Windows_x86_64/Tennis.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Tennis_Linux/Tennis.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Tennis_Linux/Tennis.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Tennis_Linux_NoVis/Tennis.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Tennis_Linux_NoVis/Tennis.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Tennis.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Tennis.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: TennisBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 8\n",
      "        Number of stacked Vector Observation: 3\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 2\n",
      "        Vector Action descriptions: , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name=\"D:/DRL/deep-reinforcement-learning-master/p3_collab-compet/Tennis_Windows_x86_64/Tennis_Windows_x86_64/Tennis.exe\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "In this environment, two agents control rackets to bounce a ball over a net. If an agent hits the ball over the net, it receives a reward of +0.1.  If an agent lets a ball hit the ground or hits the ball out of bounds, it receives a reward of -0.01.  Thus, the goal of each agent is to keep the ball in play.\n",
    "\n",
    "The observation space consists of 8 variables corresponding to the position and velocity of the ball and racket. Two continuous actions are available, corresponding to movement toward (or away from) the net, and jumping. \n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 2\n",
      "Size of each action: 2\n",
      "There are 2 agents. Each observes a state with length: 24\n",
      "The state for the first agent looks like: [ 0.          0.          0.          0.          0.          0.\n",
      "  0.          0.          0.          0.          0.          0.\n",
      "  0.          0.          0.          0.         -6.65278625 -1.5\n",
      " -0.          0.          6.83172083  6.         -0.          0.        ]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents \n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agents and receive feedback from the environment.\n",
    "\n",
    "Once this cell is executed, you will watch the agents' performance, if they select actions at random with each time step.  A window should pop up that allows you to observe the agents.\n",
    "\n",
    "Of course, as part of the project, you'll have to change the code so that the agents are able to use their experiences to gradually choose better actions when interacting with the environment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score (max over agents) from episode 1: 0.09000000171363354\n",
      "Score (max over agents) from episode 2: 0.0\n",
      "Score (max over agents) from episode 3: 0.10000000149011612\n",
      "Score (max over agents) from episode 4: 0.0\n",
      "Score (max over agents) from episode 5: 0.0\n"
     ]
    }
   ],
   "source": [
    "for i in range(1, 6):                                      # play game for 5 episodes\n",
    "    env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    \n",
    "    states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "    scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "    while True:\n",
    "        actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "        actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "        env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "        next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "        rewards = env_info.rewards                         # get reward (for each agent)\n",
    "        dones = env_info.local_done                        # see if episode finished\n",
    "        scores += env_info.rewards                         # update the score (for each agent)\n",
    "        states = next_states                               # roll over states to next time step\n",
    "        if np.any(dones):                                  # exit loop if episode finished\n",
    "            break\n",
    "    print('Score (max over agents) from episode {}: {}'.format(i, np.max(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MADDPG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I will try a multi agent Deep Deterministic Policy Gradient approach. Therefore the DDPG agent from the previous project will be adjusted to learn in a multi agent environment, in which agents act on a shared environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: TennisBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 8\n",
      "        Number of stacked Vector Observation: 3\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 2\n",
      "        Vector Action descriptions: , \n"
     ]
    }
   ],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "\n",
    "#multiple agents\n",
    "env = UnityEnvironment(file_name=\"D:/DRL/deep-reinforcement-learning-master/p3_collab-compet/Tennis_Windows_x86_64/Tennis_Windows_x86_64/Tennis.exe\")\n",
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import random\n",
    "import torch\n",
    "import numpy as np\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from mddpg_agent import Actor, Critic, OUNoise, ReplayBuffer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Init agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 2\n",
      "Number of actions: 2\n",
      "There are 2 agents. Each observes a state with length: 24\n",
      "The state for the first agent looks like:  [ 0.          0.          0.          0.          0.          0.\n",
      "  0.          0.          0.          0.          0.          0.\n",
      "  0.          0.          0.          0.         -6.65278625 -1.5\n",
      " -0.          0.          6.83172083  6.         -0.          0.        ]\n"
     ]
    }
   ],
   "source": [
    "# environment information\n",
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "# number of agents in the environment\n",
    "n_agents = len(env_info.agents)\n",
    "print('Number of agents:', n_agents)\n",
    "# number of actions\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Number of actions:', action_size)\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like: ', states[0])\n",
    "\n",
    "#device to put NN into memory\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# hyperparameters\n",
    "BATCH_SIZE = 128        # minibatch size / Learning will occur with a batch of 128\n",
    "GAMMA = 0.99            # discount: discounting of future rewards\n",
    "BUFFER_SIZE = int(1e6)  # replay buffer size\n",
    "TAU = 1e-1              # soft update factor for target NN (fixed Q-targets)\n",
    "LR_ACTOR = 1e-4         # learning rate actor optimizer\n",
    "LR_CRITIC = 1e-4        # learning rate critic optimizer\n",
    "WEIGHT_DECAY = 0        # L2 weight decay: L2 regularization against overfitting: not used here\n",
    "ADD_NOISE = True        #add OUNoise to increase exploration\n",
    "SEED = 2                # random seed\n",
    "\n",
    "#init OU noise in shape action_Size with random seed\n",
    "noise = OUNoise(action_size, 2)\n",
    "\n",
    "#init buffer to store experience tuples SARS: Experience Replay\n",
    "memory = ReplayBuffer(device, action_size, BUFFER_SIZE, BATCH_SIZE, SEED)\n",
    "\n",
    "#AGent ID\n",
    "ACTOR_0_Number = 0\n",
    "ACTOR_1_Number = 1\n",
    "\n",
    "#initialize one Actor object from ACTOR class. This object has the actor NN, act, step function\n",
    "actor_0 = Actor(device, ACTOR_0_Number, state_size, action_size, SEED, memory, noise, LR_ACTOR, WEIGHT_DECAY)\n",
    "actor_1 = Actor(device, ACTOR_1_Number, state_size, action_size, SEED, memory, noise, LR_ACTOR, WEIGHT_DECAY)\n",
    "\n",
    "#initialize the Critic object. We only need 1 critic object. It has critic NN , step, learn, soft_update functions\n",
    "critic = Critic(device, state_size, action_size, SEED, GAMMA, TAU, LR_CRITIC, WEIGHT_DECAY)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HMM das ist eigentlich schon ein MULTI-Agent DDPG programm... Unterschied zum \"Continuous control\" ist wohl, dass hier das Environment bei einer action eines agents, sich fuer alle agents aendert. Somit sollte das auch so schon funktionieren... mal sehen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Episode 0\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "\r",
      "Episode 0\tAverage Score: 0.00\n",
      "\r",
      "Episode 1\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Mir\\Desktop\\Jupyter_temp\\deep-reinforcement-learning-master\\p3_collab-compet\\mddpg_agent.py:131: UserWarning: torch.nn.utils.clip_grad_norm is now deprecated in favor of torch.nn.utils.clip_grad_norm_.\n",
      "  torch.nn.utils.clip_grad_norm(self.local.parameters(), 1) #gradient clipping\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 2\tAverage Score: 0.03Current Score: [-0.01  0.1 ]\n",
      "Episode 3\tAverage Score: 0.03Current Score: [ 0.   -0.01]\n",
      "Episode 4\tAverage Score: 0.02Current Score: [ 0.   -0.01]\n",
      "Episode 5\tAverage Score: 0.02Current Score: [-0.01  0.  ]\n",
      "Episode 6\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 7\tAverage Score: 0.01Current Score: [-0.01  0.  ]\n",
      "Episode 8\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 9\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 10\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 11\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 12\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 13\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 14\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 15\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 16\tAverage Score: 0.01Current Score: [-0.01  0.  ]\n",
      "Episode 17\tAverage Score: 0.01Current Score: [-0.01  0.  ]\n",
      "Episode 18\tAverage Score: 0.01Current Score: [-0.01  0.  ]\n",
      "Episode 19\tAverage Score: 0.01Current Score: [-0.01  0.  ]\n",
      "Episode 20\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 21\tAverage Score: 0.01Current Score: [0.   0.09]\n",
      "Episode 22\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 23\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 24\tAverage Score: 0.01Current Score: [-0.01  0.  ]\n",
      "Episode 25\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 26\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 27\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 28\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 29\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 30\tAverage Score: 0.01Current Score: [-0.01  0.  ]\n",
      "Episode 31\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 32\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 33\tAverage Score: 0.01Current Score: [-0.01  0.  ]\n",
      "Episode 34\tAverage Score: 0.01Current Score: [-0.01  0.  ]\n",
      "Episode 35\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 36\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 37\tAverage Score: 0.01Current Score: [-0.01  0.  ]\n",
      "Episode 38\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 39\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 40\tAverage Score: 0.00Current Score: [-0.01  0.  ]\n",
      "Episode 41\tAverage Score: 0.00Current Score: [-0.01  0.  ]\n",
      "Episode 42\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 43\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 44\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 45\tAverage Score: 0.00Current Score: [-0.01  0.  ]\n",
      "Episode 46\tAverage Score: 0.00Current Score: [-0.01  0.  ]\n",
      "Episode 47\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 48\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 49\tAverage Score: 0.00Current Score: [-0.01  0.  ]\n",
      "Episode 50\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 51\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 52\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 53\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 54\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 55\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 56\tAverage Score: 0.00Current Score: [-0.01  0.  ]\n",
      "Episode 57\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 58\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 59\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 60\tAverage Score: 0.00Current Score: [-0.01  0.  ]\n",
      "Episode 61\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 62\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 63\tAverage Score: 0.00Current Score: [-0.01  0.  ]\n",
      "Episode 64\tAverage Score: 0.00Current Score: [-0.01  0.  ]\n",
      "Episode 65\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 66\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 67\tAverage Score: 0.00Current Score: [ 0.   -0.01]\n",
      "Episode 68\tAverage Score: 0.00Current Score: [ 0.1  -0.01]\n",
      "Episode 69\tAverage Score: 0.00Current Score: [-0.01  0.  ]\n",
      "Episode 70\tAverage Score: 0.00Current Score: [-0.01  0.  ]\n",
      "Episode 71\tAverage Score: 0.01Current Score: [ 0.1  -0.01]\n",
      "Episode 72\tAverage Score: 0.01Current Score: [-0.01  0.  ]\n",
      "Episode 73\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 74\tAverage Score: 0.01Current Score: [-0.01  0.  ]\n",
      "Episode 75\tAverage Score: 0.01Current Score: [0.09 0.2 ]\n",
      "Episode 76\tAverage Score: 0.01Current Score: [ 0.1  -0.01]\n",
      "Episode 77\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 78\tAverage Score: 0.01Current Score: [0.1  0.09]\n",
      "Episode 79\tAverage Score: 0.01Current Score: [ 0.1  -0.01]\n",
      "Episode 80\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 81\tAverage Score: 0.01Current Score: [-0.01  0.  ]\n",
      "Episode 82\tAverage Score: 0.01Current Score: [ 0.   -0.01]\n",
      "Episode 83\tAverage Score: 0.01Current Score: [-0.01  0.1 ]\n",
      "Episode 84\tAverage Score: 0.01Current Score: [ 0.1  -0.01]\n",
      "Episode 85\tAverage Score: 0.01Current Score: [0.   0.09]\n",
      "Episode 86\tAverage Score: 0.01Current Score: [0.   0.09]\n",
      "Episode 87\tAverage Score: 0.02Current Score: [-0.01  0.1 ]\n",
      "Episode 88\tAverage Score: 0.02Current Score: [0.1  0.09]\n",
      "Episode 89\tAverage Score: 0.02Current Score: [0.   0.09]\n",
      "Episode 90\tAverage Score: 0.02Current Score: [ 0.   -0.01]\n",
      "Episode 91\tAverage Score: 0.02Current Score: [-0.01  0.  ]\n",
      "Episode 92\tAverage Score: 0.02Current Score: [-0.01  0.  ]\n",
      "Episode 93\tAverage Score: 0.02Current Score: [ 0.   -0.01]\n",
      "Episode 94\tAverage Score: 0.02Current Score: [-0.01  0.  ]\n",
      "Episode 95\tAverage Score: 0.02Current Score: [0.   0.09]\n",
      "Episode 96\tAverage Score: 0.02Current Score: [-0.01  0.  ]\n",
      "Episode 97\tAverage Score: 0.02Current Score: [ 0.   -0.01]\n",
      "Episode 98\tAverage Score: 0.02Current Score: [0.   0.09]\n",
      "Episode 99\tAverage Score: 0.02Current Score: [0.   0.09]\n",
      "Episode 100\tAverage Score: 0.02Current Score: [-0.01  0.1 ]\n",
      "Episode 100\tAverage Score: 0.02\n",
      "Episode 101\tAverage Score: 0.02Current Score: [0.1  0.09]\n",
      "Episode 102\tAverage Score: 0.02Current Score: [0.   0.09]\n",
      "Episode 103\tAverage Score: 0.02Current Score: [ 0.1  -0.01]\n",
      "Episode 104\tAverage Score: 0.02Current Score: [-0.01  0.  ]\n",
      "Episode 105\tAverage Score: 0.02Current Score: [-0.01  0.  ]\n",
      "Episode 106\tAverage Score: 0.02Current Score: [0.   0.09]\n",
      "Episode 107\tAverage Score: 0.02Current Score: [0.   0.09]\n",
      "Episode 108\tAverage Score: 0.02Current Score: [ 0.   -0.01]\n",
      "Episode 109\tAverage Score: 0.02Current Score: [ 0.1  -0.01]\n",
      "Episode 110\tAverage Score: 0.02Current Score: [0.   0.09]\n",
      "Episode 111\tAverage Score: 0.03Current Score: [ 0.1  -0.01]\n",
      "Episode 112\tAverage Score: 0.03Current Score: [0.   0.09]\n",
      "Episode 113\tAverage Score: 0.03Current Score: [-0.01  0.  ]\n",
      "Episode 114\tAverage Score: 0.03Current Score: [0.   0.09]\n",
      "Episode 115\tAverage Score: 0.03Current Score: [ 0.1  -0.01]\n",
      "Episode 116\tAverage Score: 0.03Current Score: [ 0.1  -0.01]\n",
      "Episode 117\tAverage Score: 0.03Current Score: [ 0.1  -0.01]\n",
      "Episode 118\tAverage Score: 0.03Current Score: [0.09 0.1 ]\n",
      "Episode 119\tAverage Score: 0.03Current Score: [ 0.   -0.01]\n",
      "Episode 120\tAverage Score: 0.03Current Score: [ 0.1  -0.01]\n",
      "Episode 121\tAverage Score: 0.03Current Score: [0.   0.09]\n",
      "Episode 122\tAverage Score: 0.03Current Score: [0.09 0.1 ]\n",
      "Episode 123\tAverage Score: 0.03Current Score: [0.   0.09]\n",
      "Episode 124\tAverage Score: 0.04Current Score: [ 0.1  -0.01]\n",
      "Episode 125\tAverage Score: 0.04Current Score: [ 0.1  -0.01]\n",
      "Episode 126\tAverage Score: 0.04Current Score: [0.   0.09]\n",
      "Episode 127\tAverage Score: 0.04Current Score: [-0.01  0.  ]\n",
      "Episode 128\tAverage Score: 0.04Current Score: [0.1  0.09]\n",
      "Episode 129\tAverage Score: 0.04Current Score: [0.   0.09]\n",
      "Episode 130\tAverage Score: 0.04Current Score: [0.   0.09]\n",
      "Episode 131\tAverage Score: 0.04Current Score: [ 0.1  -0.01]\n",
      "Episode 132\tAverage Score: 0.04Current Score: [-0.01  0.1 ]\n",
      "Episode 133\tAverage Score: 0.04Current Score: [ 0.   -0.01]\n",
      "Episode 134\tAverage Score: 0.04Current Score: [0.1  0.09]\n",
      "Episode 135\tAverage Score: 0.04Current Score: [ 0.1  -0.01]\n",
      "Episode 136\tAverage Score: 0.05Current Score: [ 0.1  -0.01]\n",
      "Episode 137\tAverage Score: 0.05Current Score: [-0.01  0.  ]\n",
      "Episode 138\tAverage Score: 0.05Current Score: [ 0.   -0.01]\n",
      "Episode 139\tAverage Score: 0.05Current Score: [ 0.1  -0.01]\n",
      "Episode 140\tAverage Score: 0.05Current Score: [ 0.1  -0.01]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 141\tAverage Score: 0.05Current Score: [-0.01  0.1 ]\n",
      "Episode 142\tAverage Score: 0.05Current Score: [ 0.1  -0.01]\n",
      "Episode 143\tAverage Score: 0.05Current Score: [ 0.1  -0.01]\n",
      "Episode 144\tAverage Score: 0.05Current Score: [ 0.1  -0.01]\n",
      "Episode 145\tAverage Score: 0.05Current Score: [ 0.1  -0.01]\n",
      "Episode 146\tAverage Score: 0.06Current Score: [0.3  0.19]\n",
      "Episode 147\tAverage Score: 0.06Current Score: [0.2  0.09]\n",
      "Episode 148\tAverage Score: 0.06Current Score: [-0.01  0.  ]\n",
      "Episode 149\tAverage Score: 0.06Current Score: [ 0.   -0.01]\n",
      "Episode 150\tAverage Score: 0.06Current Score: [-0.01  0.1 ]\n",
      "Episode 151\tAverage Score: 0.06Current Score: [-0.01  0.  ]\n",
      "Episode 152\tAverage Score: 0.06Current Score: [-0.01  0.1 ]\n",
      "Episode 153\tAverage Score: 0.06Current Score: [ 0.1  -0.01]\n",
      "Episode 154\tAverage Score: 0.06Current Score: [ 0.1  -0.01]\n",
      "Episode 155\tAverage Score: 0.06Current Score: [ 0.1  -0.01]\n",
      "Episode 156\tAverage Score: 0.06Current Score: [ 0.1  -0.01]\n",
      "Episode 157\tAverage Score: 0.06Current Score: [0.   0.09]\n",
      "Episode 158\tAverage Score: 0.07Current Score: [0.1  0.09]\n",
      "Episode 159\tAverage Score: 0.07Current Score: [-0.01  0.1 ]\n",
      "Episode 160\tAverage Score: 0.07Current Score: [-0.01  0.1 ]\n",
      "Episode 161\tAverage Score: 0.07Current Score: [0.2  0.19]\n",
      "Episode 162\tAverage Score: 0.07Current Score: [-0.01  0.1 ]\n",
      "Episode 163\tAverage Score: 0.07Current Score: [ 0.1  -0.01]\n",
      "Episode 164\tAverage Score: 0.07Current Score: [-0.01  0.1 ]\n",
      "Episode 165\tAverage Score: 0.07Current Score: [ 0.1  -0.01]\n",
      "Episode 166\tAverage Score: 0.07Current Score: [0.   0.09]\n",
      "Episode 167\tAverage Score: 0.08Current Score: [0.09 0.1 ]\n",
      "Episode 168\tAverage Score: 0.08Current Score: [-0.01  0.1 ]\n",
      "Episode 169\tAverage Score: 0.08Current Score: [0.2  0.09]\n",
      "Episode 170\tAverage Score: 0.08Current Score: [-0.01  0.1 ]\n",
      "Episode 171\tAverage Score: 0.08Current Score: [ 0.1  -0.01]\n",
      "Episode 172\tAverage Score: 0.08Current Score: [0.09 0.1 ]\n",
      "Episode 173\tAverage Score: 0.08Current Score: [ 0.1  -0.01]\n",
      "Episode 174\tAverage Score: 0.08Current Score: [0.   0.09]\n",
      "Episode 175\tAverage Score: 0.08Current Score: [ 0.1  -0.01]\n",
      "Episode 176\tAverage Score: 0.08Current Score: [0.   0.09]\n",
      "Episode 177\tAverage Score: 0.08Current Score: [ 0.   -0.01]\n",
      "Episode 178\tAverage Score: 0.08Current Score: [ 0.1  -0.01]\n",
      "Episode 179\tAverage Score: 0.08Current Score: [0.09 0.1 ]\n",
      "Episode 180\tAverage Score: 0.08Current Score: [0.09 0.1 ]\n",
      "Episode 181\tAverage Score: 0.08Current Score: [-0.01  0.1 ]\n",
      "Episode 182\tAverage Score: 0.08Current Score: [0.1  0.09]\n",
      "Episode 183\tAverage Score: 0.08Current Score: [0.   0.09]\n",
      "Episode 184\tAverage Score: 0.08Current Score: [ 0.1  -0.01]\n",
      "Episode 185\tAverage Score: 0.08Current Score: [-0.01  0.  ]\n",
      "Episode 186\tAverage Score: 0.08Current Score: [0.09 0.1 ]\n",
      "Episode 187\tAverage Score: 0.08Current Score: [-0.01  0.1 ]\n",
      "Episode 188\tAverage Score: 0.08Current Score: [0.   0.09]\n",
      "Episode 189\tAverage Score: 0.08Current Score: [-0.01  0.1 ]\n",
      "Episode 190\tAverage Score: 0.08Current Score: [0.2  0.19]\n",
      "Episode 191\tAverage Score: 0.09Current Score: [ 0.1  -0.01]\n",
      "Episode 192\tAverage Score: 0.09Current Score: [0.09 0.1 ]\n",
      "Episode 193\tAverage Score: 0.09Current Score: [0.09 0.2 ]\n",
      "Episode 194\tAverage Score: 0.09Current Score: [-0.01  0.1 ]\n",
      "Episode 195\tAverage Score: 0.09Current Score: [-0.01  0.1 ]\n",
      "Episode 196\tAverage Score: 0.09Current Score: [0.2  0.09]\n",
      "Episode 197\tAverage Score: 0.09Current Score: [0.1  0.09]\n",
      "Episode 198\tAverage Score: 0.09Current Score: [0.19 0.3 ]\n",
      "Episode 199\tAverage Score: 0.09Current Score: [ 0.1  -0.01]\n",
      "Episode 200\tAverage Score: 0.09Current Score: [-0.01  0.  ]\n",
      "Episode 200\tAverage Score: 0.09\n",
      "Episode 201\tAverage Score: 0.09Current Score: [0.1  0.09]\n",
      "Episode 202\tAverage Score: 0.09Current Score: [0.1  0.09]\n",
      "Episode 203\tAverage Score: 0.09Current Score: [ 0.1  -0.01]\n",
      "Episode 204\tAverage Score: 0.09Current Score: [-0.01  0.1 ]\n",
      "Episode 205\tAverage Score: 0.10Current Score: [-0.01  0.1 ]\n",
      "Episode 206\tAverage Score: 0.10Current Score: [-0.01  0.1 ]\n",
      "Episode 207\tAverage Score: 0.10Current Score: [ 0.1  -0.01]\n",
      "Episode 208\tAverage Score: 0.10Current Score: [-0.01  0.1 ]\n",
      "Episode 209\tAverage Score: 0.10Current Score: [0.09 0.1 ]\n",
      "Episode 210\tAverage Score: 0.10Current Score: [0.09 0.1 ]\n",
      "Episode 211\tAverage Score: 0.10Current Score: [-0.01  0.1 ]\n",
      "Episode 212\tAverage Score: 0.10Current Score: [0.09 0.2 ]\n",
      "Episode 213\tAverage Score: 0.10Current Score: [-0.01  0.1 ]\n",
      "Episode 214\tAverage Score: 0.10Current Score: [0.1  0.09]\n",
      "Episode 215\tAverage Score: 0.10Current Score: [0.1  0.09]\n",
      "Episode 216\tAverage Score: 0.10Current Score: [-0.01  0.1 ]\n",
      "Episode 217\tAverage Score: 0.10Current Score: [0.2  0.09]\n",
      "Episode 218\tAverage Score: 0.10Current Score: [0.1  0.09]\n",
      "Episode 219\tAverage Score: 0.10Current Score: [ 0.   -0.01]\n",
      "Episode 220\tAverage Score: 0.10Current Score: [0.1  0.09]\n",
      "Episode 221\tAverage Score: 0.10Current Score: [ 0.1  -0.01]\n",
      "Episode 222\tAverage Score: 0.10Current Score: [0.   0.09]\n",
      "Episode 223\tAverage Score: 0.10Current Score: [0.1  0.09]\n",
      "Episode 224\tAverage Score: 0.10Current Score: [-0.01  0.  ]\n",
      "Episode 225\tAverage Score: 0.10Current Score: [0.50000001 0.49000001]\n",
      "Episode 226\tAverage Score: 0.10Current Score: [0.   0.09]\n",
      "Episode 227\tAverage Score: 0.10Current Score: [-0.01  0.1 ]\n",
      "Episode 228\tAverage Score: 0.10Current Score: [0.   0.09]\n",
      "Episode 229\tAverage Score: 0.11Current Score: [0.2  0.09]\n",
      "Episode 230\tAverage Score: 0.11Current Score: [-0.01  0.1 ]\n",
      "Episode 231\tAverage Score: 0.11Current Score: [0.1  0.09]\n",
      "Episode 232\tAverage Score: 0.11Current Score: [0.2  0.09]\n",
      "Episode 233\tAverage Score: 0.11Current Score: [0.09 0.1 ]\n",
      "Episode 234\tAverage Score: 0.11Current Score: [ 0.1  -0.01]\n",
      "Episode 235\tAverage Score: 0.11Current Score: [ 0.1  -0.01]\n",
      "Episode 236\tAverage Score: 0.11Current Score: [ 0.1  -0.01]\n",
      "Episode 237\tAverage Score: 0.11Current Score: [ 0.1  -0.01]\n",
      "Episode 238\tAverage Score: 0.11Current Score: [0.50000001 0.39000001]\n",
      "Episode 239\tAverage Score: 0.11Current Score: [ 0.1  -0.01]\n",
      "Episode 240\tAverage Score: 0.11Current Score: [ 0.1  -0.01]\n",
      "Episode 241\tAverage Score: 0.11Current Score: [0.2  0.09]\n",
      "Episode 242\tAverage Score: 0.11Current Score: [0.1  0.09]\n",
      "Episode 243\tAverage Score: 0.11Current Score: [ 0.1  -0.01]\n",
      "Episode 244\tAverage Score: 0.11Current Score: [-0.01  0.1 ]\n",
      "Episode 245\tAverage Score: 0.11Current Score: [0.09 0.1 ]\n",
      "Episode 246\tAverage Score: 0.11Current Score: [0.1  0.09]\n",
      "Episode 247\tAverage Score: 0.11Current Score: [0.3  0.29]\n",
      "Episode 248\tAverage Score: 0.12Current Score: [0.1  0.19]\n",
      "Episode 249\tAverage Score: 0.12Current Score: [0.1  0.09]\n",
      "Episode 250\tAverage Score: 0.12Current Score: [0.09 0.1 ]\n",
      "Episode 251\tAverage Score: 0.12Current Score: [0.29       0.40000001]\n",
      "Episode 252\tAverage Score: 0.12Current Score: [ 0.   -0.01]\n",
      "Episode 253\tAverage Score: 0.12Current Score: [-0.01  0.  ]\n",
      "Episode 254\tAverage Score: 0.12Current Score: [0.29 0.3 ]\n",
      "Episode 255\tAverage Score: 0.12Current Score: [0.09 0.1 ]\n",
      "Episode 256\tAverage Score: 0.13Current Score: [0.79000001 0.90000001]\n",
      "Episode 257\tAverage Score: 0.13Current Score: [0.09 0.2 ]\n",
      "Episode 258\tAverage Score: 0.13Current Score: [-0.01  0.1 ]\n",
      "Episode 259\tAverage Score: 0.13Current Score: [-0.01  0.1 ]\n",
      "Episode 260\tAverage Score: 0.13Current Score: [ 0.1  -0.01]\n",
      "Episode 261\tAverage Score: 0.14Current Score: [1.30000002 1.29000002]\n",
      "Episode 262\tAverage Score: 0.14Current Score: [0.3  0.29]\n",
      "Episode 263\tAverage Score: 0.14Current Score: [ 0.1  -0.01]\n",
      "Episode 264\tAverage Score: 0.14Current Score: [-0.01  0.  ]\n",
      "Episode 265\tAverage Score: 0.14Current Score: [ 0.1  -0.01]\n",
      "Episode 266\tAverage Score: 0.14Current Score: [-0.01  0.1 ]\n",
      "Episode 267\tAverage Score: 0.14Current Score: [ 0.   -0.01]\n",
      "Episode 268\tAverage Score: 0.14Current Score: [0.2  0.29]\n",
      "Episode 269\tAverage Score: 0.14Current Score: [0.09 0.1 ]\n",
      "Episode 270\tAverage Score: 0.14Current Score: [0.3  0.19]\n",
      "Episode 271\tAverage Score: 0.14Current Score: [0.1  0.08]\n",
      "Episode 272\tAverage Score: 0.14Current Score: [0.09 0.2 ]\n",
      "Episode 273\tAverage Score: 0.14Current Score: [0.09 0.1 ]\n",
      "Episode 274\tAverage Score: 0.14Current Score: [0.1  0.09]\n",
      "Episode 275\tAverage Score: 0.14Current Score: [0.09 0.1 ]\n",
      "Episode 276\tAverage Score: 0.14Current Score: [-0.01  0.1 ]\n",
      "Episode 277\tAverage Score: 0.15Current Score: [0.1  0.09]\n",
      "Episode 278\tAverage Score: 0.14Current Score: [-0.01  0.  ]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 279\tAverage Score: 0.14Current Score: [-0.01  0.  ]\n",
      "Episode 280\tAverage Score: 0.14Current Score: [ 0.1  -0.01]\n",
      "Episode 281\tAverage Score: 0.14Current Score: [-0.01  0.  ]\n",
      "Episode 282\tAverage Score: 0.14Current Score: [0.1  0.09]\n",
      "Episode 283\tAverage Score: 0.14Current Score: [ 0.1  -0.01]\n",
      "Episode 284\tAverage Score: 0.14Current Score: [ 0.   -0.01]\n",
      "Episode 285\tAverage Score: 0.14Current Score: [0.09 0.2 ]\n",
      "Episode 286\tAverage Score: 0.15Current Score: [0.79000001 0.90000001]\n",
      "Episode 287\tAverage Score: 0.15Current Score: [0.   0.09]\n",
      "Episode 288\tAverage Score: 0.15Current Score: [0.1  0.09]\n",
      "Episode 289\tAverage Score: 0.15Current Score: [ 0.1  -0.01]\n",
      "Episode 290\tAverage Score: 0.15Current Score: [0.1  0.09]\n",
      "Episode 291\tAverage Score: 0.15Current Score: [-0.01  0.  ]\n",
      "Episode 292\tAverage Score: 0.15Current Score: [0.39000001 0.3       ]\n",
      "Episode 293\tAverage Score: 0.15Current Score: [ 0.1  -0.01]\n",
      "Episode 294\tAverage Score: 0.15Current Score: [-0.01  0.1 ]\n",
      "Episode 295\tAverage Score: 0.15Current Score: [0.3  0.19]\n",
      "Episode 296\tAverage Score: 0.16Current Score: [0.40000001 0.39000001]\n",
      "Episode 297\tAverage Score: 0.16Current Score: [ 0.1  -0.01]\n",
      "Episode 298\tAverage Score: 0.16Current Score: [0.19 0.3 ]\n",
      "Episode 299\tAverage Score: 0.15Current Score: [-0.01  0.  ]\n",
      "Episode 300\tAverage Score: 0.16Current Score: [0.1  0.09]\n",
      "Episode 300\tAverage Score: 0.16\n",
      "Episode 301\tAverage Score: 0.16Current Score: [0.39000001 0.50000001]\n",
      "Episode 302\tAverage Score: 0.16Current Score: [0.1  0.09]\n",
      "Episode 303\tAverage Score: 0.16Current Score: [0.2  0.19]\n",
      "Episode 304\tAverage Score: 0.16Current Score: [0.   0.09]\n",
      "Episode 305\tAverage Score: 0.16Current Score: [0.2  0.19]\n",
      "Episode 306\tAverage Score: 0.16Current Score: [0.09 0.1 ]\n",
      "Episode 307\tAverage Score: 0.16Current Score: [0.2  0.09]\n",
      "Episode 308\tAverage Score: 0.16Current Score: [-0.01  0.1 ]\n",
      "Episode 309\tAverage Score: 0.16Current Score: [0.09 0.2 ]\n",
      "Episode 310\tAverage Score: 0.16Current Score: [0.1  0.09]\n",
      "Episode 311\tAverage Score: 0.16Current Score: [0.1  0.09]\n",
      "Episode 312\tAverage Score: 0.16Current Score: [0.3  0.29]\n",
      "Episode 313\tAverage Score: 0.16Current Score: [-0.01  0.  ]\n",
      "Episode 314\tAverage Score: 0.17Current Score: [0.3  0.19]\n",
      "Episode 315\tAverage Score: 0.17Current Score: [ 0.1  -0.01]\n",
      "Episode 316\tAverage Score: 0.17Current Score: [0.09 0.1 ]\n",
      "Episode 317\tAverage Score: 0.17Current Score: [0.2  0.19]\n",
      "Episode 318\tAverage Score: 0.17Current Score: [0.09 0.1 ]\n",
      "Episode 319\tAverage Score: 0.17Current Score: [0.3  0.19]\n",
      "Episode 320\tAverage Score: 0.17Current Score: [-0.01  0.1 ]\n",
      "Episode 321\tAverage Score: 0.17Current Score: [ 0.1  -0.01]\n",
      "Episode 322\tAverage Score: 0.17Current Score: [0.3  0.19]\n",
      "Episode 323\tAverage Score: 0.17Current Score: [-0.01  0.1 ]\n",
      "Episode 324\tAverage Score: 0.17Current Score: [ 0.1  -0.01]\n",
      "Episode 325\tAverage Score: 0.17Current Score: [0.2  0.19]\n",
      "Episode 326\tAverage Score: 0.17Current Score: [0.29       0.40000001]\n",
      "Episode 327\tAverage Score: 0.17Current Score: [-0.01  0.  ]\n",
      "Episode 328\tAverage Score: 0.18Current Score: [0.59000001 0.60000001]\n",
      "Episode 329\tAverage Score: 0.18Current Score: [0.39000001 0.50000001]\n",
      "Episode 330\tAverage Score: 0.18Current Score: [0.   0.09]\n",
      "Episode 331\tAverage Score: 0.18Current Score: [0.29 0.3 ]\n",
      "Episode 332\tAverage Score: 0.18Current Score: [0.2  0.29]\n",
      "Episode 333\tAverage Score: 0.18Current Score: [-0.01  0.1 ]\n",
      "Episode 334\tAverage Score: 0.18Current Score: [0.40000001 0.29      ]\n",
      "Episode 335\tAverage Score: 0.18Current Score: [0.09 0.1 ]\n",
      "Episode 336\tAverage Score: 0.18Current Score: [ 0.   -0.01]\n",
      "Episode 337\tAverage Score: 0.18Current Score: [-0.01  0.1 ]\n",
      "Episode 338\tAverage Score: 0.18Current Score: [0.2  0.19]\n",
      "Episode 339\tAverage Score: 0.18Current Score: [0.40000001 0.29      ]\n",
      "Episode 340\tAverage Score: 0.18Current Score: [0.1  0.09]\n",
      "Episode 341\tAverage Score: 0.18Current Score: [0.19 0.3 ]\n",
      "Episode 342\tAverage Score: 0.18Current Score: [-0.01  0.1 ]\n",
      "Episode 343\tAverage Score: 0.19Current Score: [0.1  0.19]\n",
      "Episode 344\tAverage Score: 0.19Current Score: [0.1  0.09]\n",
      "Episode 345\tAverage Score: 0.19Current Score: [0.18 0.2 ]\n",
      "Episode 346\tAverage Score: 0.20Current Score: [1.69000003 1.80000003]\n",
      "Episode 347\tAverage Score: 0.21Current Score: [0.39000001 0.50000001]\n",
      "Episode 348\tAverage Score: 0.21Current Score: [0.3  0.19]\n",
      "Episode 349\tAverage Score: 0.21Current Score: [ 0.1  -0.01]\n",
      "Episode 350\tAverage Score: 0.21Current Score: [0.09 0.1 ]\n",
      "Episode 351\tAverage Score: 0.20Current Score: [0.1  0.09]\n",
      "Episode 352\tAverage Score: 0.21Current Score: [0.3  0.29]\n",
      "Episode 353\tAverage Score: 0.21Current Score: [0.19 0.2 ]\n",
      "Episode 354\tAverage Score: 0.21Current Score: [0.2  0.19]\n",
      "Episode 355\tAverage Score: 0.21Current Score: [0.80000001 0.69000001]\n",
      "Episode 356\tAverage Score: 0.21Current Score: [0.1  0.19]\n",
      "Episode 357\tAverage Score: 0.21Current Score: [0.39000001 0.50000001]\n",
      "Episode 358\tAverage Score: 0.21Current Score: [0.29 0.3 ]\n",
      "Episode 359\tAverage Score: 0.21Current Score: [-0.01  0.1 ]\n",
      "Episode 360\tAverage Score: 0.22Current Score: [0.78000001 0.80000001]\n",
      "Episode 361\tAverage Score: 0.21Current Score: [0.19 0.3 ]\n",
      "Episode 362\tAverage Score: 0.21Current Score: [0.80000001 0.79000001]\n",
      "Episode 363\tAverage Score: 0.22Current Score: [0.60000001 0.49000001]\n",
      "Episode 364\tAverage Score: 0.22Current Score: [ 0.   -0.01]\n",
      "Episode 365\tAverage Score: 0.22Current Score: [0.09 0.1 ]\n",
      "Episode 366\tAverage Score: 0.22Current Score: [ 0.   -0.01]\n",
      "Episode 367\tAverage Score: 0.22Current Score: [0.3  0.19]\n",
      "Episode 368\tAverage Score: 0.22Current Score: [0.1  0.19]\n",
      "Episode 369\tAverage Score: 0.22Current Score: [0.09 0.1 ]\n",
      "Episode 370\tAverage Score: 0.22Current Score: [0.29 0.3 ]\n",
      "Episode 371\tAverage Score: 0.22Current Score: [0.19 0.2 ]\n",
      "Episode 372\tAverage Score: 0.22Current Score: [0.3  0.29]\n",
      "Episode 373\tAverage Score: 0.22Current Score: [0.1  0.09]\n",
      "Episode 374\tAverage Score: 0.22Current Score: [ 0.   -0.01]\n",
      "Episode 375\tAverage Score: 0.22Current Score: [0.29 0.3 ]\n",
      "Episode 376\tAverage Score: 0.22Current Score: [0.1  0.09]\n",
      "Episode 377\tAverage Score: 0.22Current Score: [ 0.   -0.01]\n",
      "Episode 378\tAverage Score: 0.23Current Score: [0.19 0.3 ]\n",
      "Episode 379\tAverage Score: 0.23Current Score: [0.2  0.19]\n",
      "Episode 380\tAverage Score: 0.23Current Score: [0.60000001 0.49000001]\n",
      "Episode 381\tAverage Score: 0.23Current Score: [0.19 0.2 ]\n",
      "Episode 382\tAverage Score: 0.24Current Score: [0.3  0.19]\n",
      "Episode 383\tAverage Score: 0.24Current Score: [0.39000001 0.40000001]\n",
      "Episode 384\tAverage Score: 0.24Current Score: [0.29       0.40000001]\n",
      "Episode 385\tAverage Score: 0.25Current Score: [0.40000001 0.29      ]\n",
      "Episode 386\tAverage Score: 0.24Current Score: [0.1  0.09]\n",
      "Episode 387\tAverage Score: 0.24Current Score: [0.50000001 0.49000001]\n",
      "Episode 388\tAverage Score: 0.24Current Score: [0.09 0.1 ]\n",
      "Episode 389\tAverage Score: 0.24Current Score: [0.2  0.09]\n",
      "Episode 390\tAverage Score: 0.25Current Score: [0.50000001 0.39000001]\n",
      "Episode 391\tAverage Score: 0.26Current Score: [0.99000002 1.00000001]\n",
      "Episode 392\tAverage Score: 0.26Current Score: [0.89000001 0.90000001]\n",
      "Episode 393\tAverage Score: 0.26Current Score: [0.2  0.09]\n",
      "Episode 394\tAverage Score: 0.27Current Score: [0.39000001 0.40000001]\n",
      "Episode 395\tAverage Score: 0.28Current Score: [1.30000002 1.29000002]\n",
      "Episode 396\tAverage Score: 0.27Current Score: [0.3  0.19]\n",
      "Episode 397\tAverage Score: 0.28Current Score: [0.50000001 0.69000001]\n",
      "Episode 398\tAverage Score: 0.28Current Score: [0.40000001 0.29      ]\n",
      "Episode 399\tAverage Score: 0.28Current Score: [ 0.1  -0.01]\n",
      "Episode 400\tAverage Score: 0.31Current Score: [2.50000004 2.39000004]\n",
      "Episode 400\tAverage Score: 0.31\n",
      "Episode 401\tAverage Score: 0.30Current Score: [0.1  0.09]\n",
      "Episode 402\tAverage Score: 0.30Current Score: [-0.01  0.  ]\n",
      "Episode 403\tAverage Score: 0.30Current Score: [-0.01  0.1 ]\n",
      "Episode 404\tAverage Score: 0.30Current Score: [0.3  0.29]\n",
      "Episode 405\tAverage Score: 0.30Current Score: [ 0.   -0.01]\n",
      "Episode 406\tAverage Score: 0.31Current Score: [0.80000001 0.79000001]\n",
      "Episode 407\tAverage Score: 0.31Current Score: [0.40000001 0.29      ]\n",
      "Episode 408\tAverage Score: 0.32Current Score: [1.00000001 1.09000002]\n",
      "Episode 409\tAverage Score: 0.32Current Score: [0.60000001 0.49000001]\n",
      "Episode 410\tAverage Score: 0.32Current Score: [0.2  0.19]\n",
      "Episode 411\tAverage Score: 0.33Current Score: [0.40000001 0.39000001]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 412\tAverage Score: 0.33Current Score: [0.29 0.3 ]\n",
      "Episode 413\tAverage Score: 0.34Current Score: [0.89000001 1.10000002]\n",
      "Episode 414\tAverage Score: 0.34Current Score: [0.2  0.19]\n",
      "Episode 415\tAverage Score: 0.34Current Score: [ 0.1  -0.01]\n",
      "Episode 416\tAverage Score: 0.34Current Score: [0.70000001 0.59000001]\n",
      "Episode 417\tAverage Score: 0.35Current Score: [0.60000001 0.49000001]\n",
      "Episode 418\tAverage Score: 0.36Current Score: [1.20000002 1.09000002]\n",
      "Episode 419\tAverage Score: 0.36Current Score: [0.59000001 0.60000001]\n",
      "Episode 420\tAverage Score: 0.37Current Score: [0.80000001 0.79000001]\n",
      "Episode 421\tAverage Score: 0.37Current Score: [0.40000001 0.49000001]\n",
      "Episode 422\tAverage Score: 0.37Current Score: [ 0.1  -0.01]\n",
      "Episode 423\tAverage Score: 0.38Current Score: [1.49000002 1.50000002]\n",
      "Episode 424\tAverage Score: 0.39Current Score: [0.69000001 0.80000001]\n",
      "Episode 425\tAverage Score: 0.39Current Score: [0.   0.09]\n",
      "Episode 426\tAverage Score: 0.39Current Score: [0.2  0.19]\n",
      "Episode 427\tAverage Score: 0.40Current Score: [0.90000001 0.89000001]\n",
      "Episode 428\tAverage Score: 0.39Current Score: [0.1  0.09]\n",
      "Episode 429\tAverage Score: 0.39Current Score: [0.2  0.09]\n",
      "Episode 430\tAverage Score: 0.39Current Score: [0.19 0.2 ]\n",
      "Episode 431\tAverage Score: 0.39Current Score: [0.1  0.09]\n",
      "Episode 432\tAverage Score: 0.39Current Score: [0.2  0.09]\n",
      "Episode 433\tAverage Score: 0.39Current Score: [0.60000001 0.59000001]\n",
      "Episode 434\tAverage Score: 0.40Current Score: [1.20000002 1.09000002]\n",
      "Episode 435\tAverage Score: 0.40Current Score: [ 0.   -0.01]\n",
      "Episode 436\tAverage Score: 0.41Current Score: [0.89000001 0.90000001]\n",
      "Episode 437\tAverage Score: 0.41Current Score: [0.40000001 0.29      ]\n",
      "Episode 438\tAverage Score: 0.41Current Score: [0.40000001 0.49000001]\n",
      "Episode 439\tAverage Score: 0.42Current Score: [1.20000002 1.09000002]\n",
      "Episode 440\tAverage Score: 0.42Current Score: [0.1  0.09]\n",
      "Episode 441\tAverage Score: 0.42Current Score: [0.09 0.2 ]\n",
      "Episode 442\tAverage Score: 0.42Current Score: [ 0.   -0.01]\n",
      "Episode 443\tAverage Score: 0.43Current Score: [0.79000001 0.90000001]\n",
      "Episode 444\tAverage Score: 0.43Current Score: [-0.01  0.  ]\n",
      "Episode 445\tAverage Score: 0.44Current Score: [1.49000002 1.60000002]\n",
      "Episode 446\tAverage Score: 0.43Current Score: [0.39000001 0.40000001]\n",
      "Episode 447\tAverage Score: 0.42Current Score: [0.2  0.09]\n",
      "Episode 448\tAverage Score: 0.42Current Score: [0.2  0.09]\n",
      "Episode 449\tAverage Score: 0.42Current Score: [-0.01  0.1 ]\n",
      "Episode 450\tAverage Score: 0.43Current Score: [0.50000001 0.39000001]\n",
      "Episode 451\tAverage Score: 0.43Current Score: [0.1  0.09]\n",
      "Episode 452\tAverage Score: 0.43Current Score: [0.90000001 0.99000002]\n",
      "Episode 453\tAverage Score: 0.44Current Score: [0.3        0.39000001]\n",
      "Episode 454\tAverage Score: 0.44Current Score: [0.40000001 0.49000001]\n",
      "Episode 455\tAverage Score: 0.44Current Score: [0.99000002 1.00000001]\n",
      "Episode 456\tAverage Score: 0.44Current Score: [0.1  0.09]\n",
      "Episode 457\tAverage Score: 0.43Current Score: [-0.01  0.  ]\n",
      "Episode 458\tAverage Score: 0.44Current Score: [0.60000001 0.69000001]\n",
      "Episode 459\tAverage Score: 0.44Current Score: [0.59000001 0.70000001]\n",
      "Episode 460\tAverage Score: 0.44Current Score: [0.80000001 0.79000001]\n",
      "Episode 461\tAverage Score: 0.44Current Score: [-0.01  0.  ]\n",
      "Episode 462\tAverage Score: 0.43Current Score: [0.1  0.09]\n",
      "Episode 463\tAverage Score: 0.44Current Score: [0.80000001 0.69000001]\n",
      "Episode 464\tAverage Score: 0.45Current Score: [1.20000002 1.29000002]\n",
      "Episode 465\tAverage Score: 0.45Current Score: [0.2  0.09]\n",
      "Episode 466\tAverage Score: 0.45Current Score: [ 0.1  -0.01]\n",
      "Episode 467\tAverage Score: 0.45Current Score: [0.50000001 0.49000001]\n",
      "Episode 468\tAverage Score: 0.46Current Score: [0.50000001 0.49000001]\n",
      "Episode 469\tAverage Score: 0.47Current Score: [1.00000001 0.99000002]\n",
      "Episode 470\tAverage Score: 0.46Current Score: [0.19 0.2 ]\n",
      "Episode 471\tAverage Score: 0.46Current Score: [ 0.   -0.01]\n",
      "Episode 472\tAverage Score: 0.47Current Score: [0.49000001 0.60000001]\n",
      "Episode 473\tAverage Score: 0.47Current Score: [1.00000001 0.99000002]\n",
      "Episode 474\tAverage Score: 0.48Current Score: [0.2  0.19]\n",
      "Episode 475\tAverage Score: 0.48Current Score: [0.2  0.09]\n",
      "Episode 476\tAverage Score: 0.48Current Score: [0.2  0.09]\n",
      "Episode 477\tAverage Score: 0.48Current Score: [0.2  0.29]\n",
      "Episode 478\tAverage Score: 0.48Current Score: [0.39000001 0.40000001]\n",
      "Episode 479\tAverage Score: 0.48Current Score: [-0.01  0.1 ]\n",
      "Episode 480\tAverage Score: 0.47Current Score: [0.1  0.19]\n",
      "Episode 481\tAverage Score: 0.48Current Score: [0.29 0.3 ]\n",
      "Episode 482\tAverage Score: 0.47Current Score: [-0.01  0.  ]\n",
      "Episode 483\tAverage Score: 0.48Current Score: [1.39000002 1.40000002]\n",
      "Episode 484\tAverage Score: 0.48Current Score: [ 0.   -0.01]\n",
      "Episode 485\tAverage Score: 0.48Current Score: [0.50000001 0.59000001]\n",
      "Episode 486\tAverage Score: 0.48Current Score: [0.2  0.09]\n",
      "Episode 487\tAverage Score: 0.48Current Score: [0.50000001 0.39000001]\n",
      "Episode 488\tAverage Score: 0.48Current Score: [0.09 0.1 ]\n",
      "Episode 489\tAverage Score: 0.48Current Score: [0.09 0.2 ]\n",
      "Episode 490\tAverage Score: 0.50Current Score: [2.60000004 2.60000004]\n",
      "\n",
      "Environment solved after 490 episodes!\tMean Score: 0.50\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJztnXecHNWV739neoJGo6wZkFBAgEQ0SYggw9pEEx84YAPrgNOyGNvAW6/95LAE2xh2jQETbGAXsLFZgsFEiSCQiAaEEopIGlAaxYkaTZ7uPu+Pqlt9q7qqurp7qnq653w/n/l0d9Xtqnu7e+65J15iZgiCIAgCAJQVugOCIAjC4EGEgiAIgmAhQkEQBEGwEKEgCIIgWIhQEARBECxEKAiCIAgWIhQEQRAECxEKgiAIgoUIBUEQBMGivNAdyJba2lqeNm1aobshCIJQVCxZsqSJmesytSs6oTBt2jQsXry40N0QBEEoKohoc5B2Yj4SBEEQLEQoCIIgCBYiFARBEAQLEQqCIAiChQgFQRAEwUKEgiAIgmAhQkEQBEGwEKEgCIJQBPz+1Q14c31j6PcRoSAIglAE3LOwHv/4uDn0+4hQEARBKAIYDKLw7yNCQRAEoQhgBiKQCSIUBEEQioWi1hSIaAoRLSSitUS0moiucWlzKhHtIaLl5t91YfVHEAShmOGI7hNmldQ4gB8x81IiGglgCRHNZ+Y1jnZvMfMFIfZDEASh6GFmUAQGpNA0BWbewcxLzed7AawFMCms+wmCIJQyjCI3H+kQ0TQAxwJ43+X0bCL6kIheJKIjouiPIAilx+UPLsLdCzYUuhuhUTKOZiIaAeApANcyc7vj9FIA+zPz0QDuAvCMxzWuIKLFRLS4sTH85A1BEIqPN9Y34tZX1he6G+ESgaoQqlAgogoYAuERZv678zwztzNzh/l8HoAKIqp1aXc/M89i5ll1dRl3kxMEQSgpmA03c1FrCkREAB4AsJaZb/NoM8FsByI6wexP+Cl7giAIRYQpEyLxKYQZfXQygK8DWElEy81jPwMwFQCY+V4AFwP4HhHFAXQDuJSVSBQEQRBsRBF9FJpQYOa3kUHbYea7AdwdVh8EQRBKgShXypLRLAiCMMixfAqlEpIqCIIg5I7SFIra0SwIgiAMDFE6mkUoCIIgDHIYynxU5HkKgiAIQv5EGZMpQkEQBKFIEPORIAiCYFHUVVIFQRCEgUHMR4IgCIJFytEc/r1EKAiCIAxyrJDUCO4lQkEQBGGQYyWviaYgCIIgpEpni6NZEARhyCOagiAIgmAh0UeCIAhCGlLmQhAEQYh0QwURCoIgCIMcK08hgnuJUBAEQRjkSOlsQRAEwUI22REEQRAsUttxiqNZEARhyCN5CoIgCIKF1D4SBEEQCoIIBUEQhEGOCkmNwn4kQkEQBGGwI+YjQRAEQSGOZkEQBMEi5WgW85EgCMKQR7bjFARBECxKIiSViKYQ0UIiWktEq4noGpc2RER3ElE9Ea0goplh9UcQBKFYidKnUB7iteMAfsTMS4loJIAlRDSfmddobc4FMMP8OxHAH81HQRAEoQCEpikw8w5mXmo+3wtgLYBJjmYXAXiYDd4DMIaIJobVJ0EQhGKk5PZoJqJpAI4F8L7j1CQAW7XXDUgXHIIgCL5wlPtVFgCOsExq6EKBiEYAeArAtczc7jzt8pa0b5eIriCixUS0uLGxMYxuCoJQxJS4TLAoakczABBRBQyB8Agz/92lSQOAKdrryQC2Oxsx8/3MPIuZZ9XV1YXTWUEQhEFKapOdIjYfkdH7BwCsZebbPJo9B+AbZhTSSQD2MPOOsPokCEJpUuqKQpTbcYYZfXQygK8DWElEy81jPwMwFQCY+V4A8wCcB6AeQBeAb4XYH0EQSpSh4lMo6pBUZn4bGQQbG9/k98PqgyAIQ4PSFgkpJKNZEAQhACWuKEQq9EQoCIJQ9HCJ6woll6cgCIIQJkNFUxDzkSAIghCp0BOhIAiCMOhRpbPFfCQIgpCRkjcflULpbEEQhKgoeUez+Sg+BUEQhACUuqagkOgjQRCEAJS6TBBHsyAIRcPWli7c+vK6gpaaKPkyF7JHsyAIxcL3HlmCuxfWo353R8H6UNoiQRzNgiAUEb39SQBAsoAzc4krCpEWxBOhIAhCXkQxUQ11UtFV4mgWBEHIjGgKA4YIBUEQip5Sz1NQiE9BEISioZATc6n7FBRS5kIQhEFPFAlVmSh1mSB5CoIgCFkwZPIUIriXCAVBEAaEQs7LpS0SxNEsCEIRMRhCUktcUZCCeIIgCEIK2Y5TEISio7Dmo9JWFazRiaYgCIIQgNKWCVL7SBAEIRtKXCZYSJ6CIAhFw1BLXnv342Y0tHZFdLfoBlge2Z0EQShJoli9ZqIQAumy/34PFTHChpvOC/1eYj4SBKHoKKijuUD37k9Ec2MJSRUEQciCUvcppDQF8SkIglAklHoCWSGx8hSKWVMgogeJaDcRrfI4fyoR7SGi5ebfdWH1RRCE8FDzVFL2aA6N6LbYCdfR/CcAdwN42KfNW8x8QYh9EAQhIgorFAp260jgCKVCaJoCM78JoCWs6wuCMLgo8Xl5UDAUfAqziehDInqRiI4ocF8EQcgBZecupAknn1u/sGI7fv70yoHrTAhEGXJbSKGwFMD+zHw0gLsAPOPVkIiuIKLFRLS4sbExsg4KghCcYq199I+PmzFv5Y4B7E0IDIXS2czczswd5vN5ACqIqNaj7f3MPIuZZ9XV1UXaT0EQgpEs0jwFZiBRyM4HIEpHc8GEAhFNIDMVkohOMPvSXKj+CIKQG2r1WkhHcz4wc0EFWhBSm+yELxYCRx8R0SkAZjDzQ0RUB2AEM2/0af8ogFMB1BJRA4DrAVQAADPfC+BiAN8jojiAbgCXcqnHlQlCCVOsO68Vh6YQXZ5CIKFARNcDmAXgEAAPwZjc/wrgZK/3MPNlftdk5rthhKwKglACFNbRnPu9k8xIDPL16GCsffQFABcC6AQAZt4OYGRYnRIEofgo5LSal6YAIDnoNQWDweRo7jNNOwwARFQTXpcEQSgmVOx8sSavDUZNgZmxsanT5czgyVN4gojuAzCGiP4FwKsA/ju8bgmCUGwUdrGdT/iRIVSyMUGFbSr7y3ubcdqtr2PJ5tZI7qcTyKfAzLcS0VkA2mH4Fa5j5vmh9kwQhKKiWJPXlIaTSDLKY8FW4mEPddmWNgDA5uZOHLf/2EjNRxmFAhHFALzMzGcCEEEgCIIrRRt9ZD4mmAOHY0Y+1MHkaGbmBIAuIhodQX8EQShSCrkdZz4os1cymc17oh1rKiR18OQp9ABYSUTzYUYgAQAzXx1KrwRBKBqs5LUsJtWBJr+MZuPN2Uz0UWtFUYakBhUKc80/QRAEG4NiP4U8tBTV7WwikKLWilIZzeHfK6ij+c9EVAngYPPQOmbuD69bgiAUGwXNU8gr+MjUFLIIn4pcUzAfB03pbCI6FcAGAPcA+AOA9UT0mRD7JQglSW88gd/MW4uO3nihuzLgFCL66H/e+gRrd7TnF31kmr2yKXVRKKVoMCWv/Q7A55j5s8z8GQBnA7g9vG4JQmnyt8UNuP/NT3DH/PWF7srAQSp5Lfpb/3ruWpz7+7fyMx+Z7w3TfNTRG8esX8/He5/kVvMzSoEbVChUMPM69YKZ18MsbicIQnDiCWNZ2pcooFc2JAoakppXnoL5mFX0UXb3WLVtD5o6+nDbK7ktBqL8aIM6mhcT0QMA/mK+/iqAJeF0SRBKlyhCCgtF5GGaA3S/nBzNWd5bfTa5fv2DztEM4HsAvg/gahjBBm/C8C0IgpADg6zUzoAQ9ZAGylxlhaRm41PI+h7GY6ws11ndFCoROJqDCoVyAL9n5tsAK8u5KrReCUKJUoqKghpS1I5mXTPJL/rIIExHs7p2WY4/gCg1haA+hdcAVGuvq2EUxRMEQQAQvfnIJhTy0FOs2kchmo/Utcty1BQGY+nsYWo/ZQAwnw8Pp0uCIBQjUZvEdMdwvns0G9cLT1NQQiRX61Eqo3mQ5CkA6CSimeoFEc2CsYWmIAg5UKx1gvyIOiTVrilkx8V//AeO+9V823XcNIXTb30dZ932BgDgmw8twsG/eDGn+ykBFisC+2FQn8K1AP5GRNthfB77AbgktF4JQoky+KeE3CmsTyG7ey829ynQcfMpfKJtdPP6ukbXewchFX2Uq/kouj2afTUFIjqeiCYw8wcADgXwOIA4gJcAbAy/e4IgDHbURFVQ81E+17Gij4K/J9uxJgfMfBQ+mcxH9wHoM5/PBvAzGKUuWgHcH2K/BEEoEgpVEG/Aoo8iKIinlJBcQ1IH0yY7MWZuMZ9fAuB+Zn4KwFNEtDzcrglC6SJ5CvkzUEJI33ktKLlrCrmGpKZK4oVNJk0hRkRKcJwBYIF2Lqg/QhAEhTkplKBMiFxTsK/s8y+dHeZ+Ckre5LvSHwyawqMA3iCiJhjRRm8BABFNB7An5L4JQslR2o7mwt1vQMxHWWU0Z2k+GqjktZzenR2+QoGZbyKi1wBMBPAKp3SYMgA/DLtzglCqlJL5SEXUFDT6KED7Vdv2YFhFGabvM9J2PJf9FLINv1V9zd2nkF/0UjZkNAEx83sux0qo7q8gCANB1HkK+sreSx499M5GHD5xFE48cDwuuOttAMCmW863tVGXCbcgnvFYBGkK4hcQhCgphkkhV6LWFOzmI/d73/j8GgDpgsB+nQgczUVkPgqa0SwIguBLMWU066j3hukoj5sfTq4ZzYOxIJ4gCANKCTkVTKLPU3A/nqtpJ5t9j7Ida8LMjMu7IN4gCEnNGSJ6kIh2E9Eqj/NERHcSUT0RrdBrKwmCUDwUyiIWxKcQiAjMR3HLfJTd+1L3GyRlLvLkTwDO8Tl/LoAZ5t8VAP4YYl8EYVAQxUqvUBRy5zU9RDTXHIKs8hSyuwXiiTx9Cjm9KzdCEwrM/CaAFp8mFwF4mA3eAzCGiCaG1R9BEMIl8tpHHrlr2XZDCZRsNIVsBWC+mgKGiE9hEoCt2usG85gglDylladgPEblaH6nvglfue9d9GtOAJt8yOLDTSbZKoSXb0YzM4OZccer63HDc6tt53SfAjPjK/e+i1fX7MLzH27H1x94P/P9BlOeQoi4jc71WyGiK2CYmDB16tQw+yQIoVKoiqJhokxiUZmPfvTEh9jZ3oPtbaktXThHTSGeTBmestEU3O5yzWPL8dyH263XN1x4hO0+gPFZ9caTWLSpBcsb2tAX9/duF+J3UkhNoQHAFO31ZADb3Roy8/3MPIuZZ9XV1UXSOUEIg9L1KETH6OoKAEBbV791zM2nECRDOWmu7oFszUfpx3SB4ET5FNQ9gWC/BfWuoZKn8ByAb5hRSCcB2MPMOwrYH0EQ8iCbMhH5oIRCc2efdcyuKZiTfIBldjzJngXx/MxQuUYfJZlTWoM2w3vdSx0fTKWzc4aIHgVwKoBaImoAcD2ACgBg5nsBzANwHoB6AF0AvhVWXwRhsGCt/EowTyGqEY2qNqatpo5e33ZBVv6JJKc22XE093t7tt+f8ikwMxIukUjM7hO+khVR7tEcmlBg5ssynGcA3w/r/oIwGIk6bDMSLEdzNGMbNczQFHShYHc0B+9PwsenEPfZii2bXdqMa6UET9yl5EWCGWUuE77q3aDZjlMQhIGlFGUCrEk4mtuNHJauKbiZX7LXFOztvSZ+Zs5aU1A+hSSz1S99fvcSYMk0TSF8RCgIQsgwMzY3d1rP86G9px8tmi19MGBNkBFJPBWW2bg3g6YQYDWfSLL15kSSsbWly5q0dU1h554e63mSgS3NXVn12a4pJM1xpM57CyDzUR0QTUEQip+/vr8Fn/3t61i2pTVt5ZctJ9z0Kmb+av7AdW4A4Ig1BTWptnSmoo/syWv+jmbdIZ7glKbQ0NqNf/qvhfjty+vMdqn3nHTza9bzFQ1t+N4jS7Pqc8K6mKYpOMxHrn11HC/q2keCIBgs3dwKAPiksTNvTaGnP0tjdgSoiSsq57kyxfTFE9Yxt5BUL/ORPgEnEql37mw3tIF/fNxk3Mdj+b7J1Ppy6XMyCfRbjubUeS/zkXU0QrujCAVBCBm77bhg3QgNNaToNAXjRv2JdEFg74+HUPDQFFRUEGnn3MjWyaz3Wfcp6I5mr3DeQoSkilAQhAgpxeijXDa+zwc3m7/bhjuemoIuFJLJlGbh6L/X+3MZZyKDT8HrXukhqeEjQkEQIoLh4jgsATjiQblpCm4EMh8lvTOgM03U2RDX8xRcfApeWpZlmuP094SFCAVBCBs9c1XFnZeQVMhkrhloEi72G3Z5rvfH5lzWhEk8mbQmXCVs1BLeUyjkIP30kNRUHaQUnj4Fh7wVTUEQSghmLk2fQrQRqa4agm1vBRdHsz7pJmzCwns/BW/zUdZdtoekumQ0e+cp2BcRRV3mQhCEdErSp4DUhKfz3IfbUVtTiU9Pr8Xr63Zj7Y69KCPgXz97UF73c5us2fEikWQrtBRwmoxSz+9asMH6TuKJoEIh+He4ZHML6nd3aD4FtkxJZT4+BWeUWkpTKOIyF4IgGOj/yClzQOkIB6/9CK5+dBkAYNMt5+ObD31gHf/nE6dipFmqIhfibkLBFn3EeKe+CS+u2pnWR8A+Ab+yZpc1OSvBkTH6KIuv7kt/fBcAcNKB46x+uvoUPCKa0gSQRB8JQulgOJpLRxgosh1Rpj0EMuHmU4AjTyGtZIWHpmCcMx+dtY88HNlpq/gA36kuON2qpKYlqZknU6a56H43IhQEIWTs//yF60dYqAkrqFmluz+RuZEPbpO1M08h5tj3UtcuvMxCzsna086fZurJ1OOU1pHUqqTaQlIzJa+ZSJ6CIJQYJelT8HDUetHdl59QyORTYGbEHLOnvnWnm/nJOJ50vA5mPgpajVW9161KqpcmkOZoznin/BGhIAhRwdqEUkKyIdsw23w1hf4A6pYznl83WXlN4v1xh1kpoKPZqzf6RK/nG6QczVrto4wF8SRPQRBKBtu/cZamlmIg24J4XXlrCi55ChnMR72aUPDyFShtQr3TS1NwfnVe32WnNk6lKdgczVpb70Q50RQEoWRhpPIUSsm3kJoUC+hTcDiaHTLBZj7ymsT7HMv1wJqCx7BbOlIlzm0hqS4+hcDJa+JTEITix+2fP8j+wcWCGknQQnE9IfgUmmx7K3Da5Kmbjy64623X61qagvlmT03B8dprQl/e0JbWJukVkpoh/DWlKYj5SBBKitQ/eekIhdTOa8HGlL/5KP0+Nzy/xtYfZ5PeAGGwzkzpoHsceA37DwvrU9eyJa+lmxCDltQQTUEQSghmLft38G2LkDOp/RSCkb+j2f/DY6SbfoLkRvQ72iQ88xTsr72E4d6eOABjItdX/MrRbC/D4bwH245HmewoQkEQQkap/Izoy0xHgWU+iiokNUN1VOb0Vb7TX+BGn8PR7KUpeCW/OUlVRk09130KXrWZgPTNdaL8uYhQEISISCTZWsHm62j2coIWgiz9zPk7mjOMncFpmphTC3Cj3yE4guzHAHibAnWHuF4l1TIl6ULBK8vacWkxHwlCCZHUqqTm61MYTJpGqiCeHgHk3b8wfApOnJ9PEE3BedngQsH9evo9lcDRk9dsWdYepTNSOSAqjFUczYJQ9KjVXSLJrhNoLgym6KVUXR/tmE/3esLWFNzMR1nUW9K/ryD39/oudU1BObGNTXZcfAqO7jlzP6IsnS1CQRAiIpHkrBO9vAjbUb2trRtLNrdk9R59SH6r+a6+eMZrxRNJvLRqh6vGEc+w6jd8N7kLBYWnUHDmM2TwKQCZNYV0n4Ldl6DOSvKaIJQAeoG1VLz64DYfnX37m1bZ50y4FcTz619Pf+YJ+r43P8GVf12Kl7Ty14rMmgKnlY3ojQfXTpSJJqim4GUq6/fwKajnug/DqdkoeXLvGx9jS3OXpimI+UgQSoZEMr3AWc7XClkodPQaq/lg9nvzidbUb+IO0vftbd0AgKbOvrRzmfqkl5JQOPMUfnH+Ydhw07m+1/EKfU3zKfhcQ83hfS6agi4U0jfVSb3+l4cXp67n2+OBQYSCIEREknnAQlI5ojyH1q70SdmJm5/Eb+LOFFLqey8t+StTOx2nUCgjQrmzFoYDrxpJQX0KAFBR5phiNZ+Crkk4NRv9Fj3xhOQpCEJpkTJHqH/2ICtwP8LWFCpiRp9bXFbqTtwEna9QyKPvQT42N0ezUyiUxwhEhMqYyxRoygpniKrCWZDPr0/qc9Tbugk1v4gmfTFR9I5mIjqHiNYRUT0RzXE5/00iaiSi5ebfd8PsjyAUEsPRnLv5iANOugPB6Gpju8zmjsxCwRkhA/j3L58cC+eeB16km4/sPgVVtro85j3LOstepPoQzKcAABXl9ilWz1Pwu4at7HZSL4hXxHs0E1EMwD0AzgLQAOADInqOmdc4mj7OzD8Iqx+CUGh0R2w+jmZ9Mgm7dtLo6go0dfQF0hRgmY9SR/yEQhDzj+d7zYm6MlbmmXvA4DSh64w+UqYjNxOSVTrb4/pOs5K+infet7zMKRTchU1anoLt+hxpSnOYmsIJAOqZ+RNm7gPwGICLQryfIAxK1ARpD0nNQSjomkIEQgEAWjp7M7TU56tg/Qsydq8WSqBUlXtPXUEczWWmMKhwMx+ZeG3mk17mwnjttoavdGgiep5CkGsCZogtojEdAeEKhUkAtmqvG8xjTr5ERCuI6EkimhJifwQhME8va8AP/nep5/kFH+3C5Q8uCnQty4/A+e2noM8lfu9fvX0PLrrnnUD5AF6MUuajAJqCVQ5cz9DVVsPXP7vK1t7LgZuJqx5ZgmeXbwMAVPoJBaQLnt5+d03BVyh4aQouPoVfPr/G9Tspd1x/e1s3nljckN5n9n69Y08PHnx7YySRR0C4QsFtDM6P7XkA05j5KACvAviz64WIriCixUS0uLGxcYC7KQjp/N/HP8QLK3Z4nv/2nxbjjfWNWZVcSNp8CvlpCn52+ZvmrsWHW9uwbEubZ5tMqH/eIMXrVE9s0TRaX59aus3WPoiW4zZ5zFu5E9c9uxoAMKwi5t0fzUyncPoU1M5sbj4FtSIPbj5iPPjORte2Tkdze4+7oPbTFABjF7co/AlAuEKhAYC+8p8MYLvegJmbmVnpp/8N4Di3CzHz/cw8i5ln1dXVhdJZQcgFr9WkTsp8ZN9sJVv8qmq6kY+FKRVLH/w+umaim0icGstAOMmHVfhrCunJa/YDMR+fgiKoo9k/+si9n4fsO9JxjdwEZRiEKRQ+ADCDiA4gokoAlwJ4Tm9ARBO1lxcCWBtifwRhwAkiFHTnMjuOZYOuHYQdfaT6FyTaR2k93ZqJJuFj6grS95Sbwl2zqq700xRczEdOoWCuup17Oet4jT1olVTAWyhUVaRHJfm9jpLQoo+YOU5EPwDwMoAYgAeZeTUR/RLAYmZ+DsDVRHQhgDiAFgDfDKs/gpALzOyrtgexj+s293x8CjbzUZCVZR5Ly1QphuCaQremEfgJk2wEWr9HP6p9zEeAi/mo38N85Ewu0+8d99IUcs9TUAwrt/ffubZw+3qjcjSHJhQAgJnnAZjnOHad9vynAH4aZh8EIR8SSfaPZQ+wklZNEpqtOxefgq3+fsgLSdXPIJqQ6oq+T4LfxxJEKKhPXN3fORH7+RSM+9vv4QxfVULBTVNQbw1e5sJ7PE5Hs8LpKA+iKURRNhuQjGZB8CXTSjmIpqBW+LqjOe+Q1JClglXJM4hQUOYjzSk9UJqCl2/DT1MwMprtx5zRR36O5qQ19oA+BZ+PyDVjGukhtYH2fS6BkFRBKHoyaQKeE0ciiR17jKJuScvRnMpTyGVS93M09yeS2Lmnx3YsV7t0S2cf2rv7jeua92zu6EWnWSRvT1c/9pjnjfsYjzZNwefeQaKPVIue/gQWb2pJK+Pt61NAuqawubnT9loJhTIXm4zqn5dgc37nvrWPPLTMNJ9CkhFPJK1CgK7mI8+7DCwiFATBh0yagFdW7W/mfYTZNy9Ac0dvyqdgMx9l3xdbnoL5/K0Njdixpxs/eXIFTrr5NVvmbq75ADN/NR8fN3aa1zCud9yvX8V5d74FADj6l6/g6Btfsdor80l/glPmHp97BwrjNds8umgLLr73XXz7T4tt5zNpCm4hnTp+0UeW+SigpuA3Hi/zUZXTp8DAr+euxadvWYDWzj5Xk1QpJK8JQtGTyXzitZpc8NEuAEZcuppbkjZHc57mI/P51x9YhPPvfBtPLzNyAfR4/CD+gEzok/vm5i7XNsypyVVpC37aQBChoIRtk0ftJd88BbB1/zd+fKprG7/oo6Rlskp9fhNHD8MT/zobQHpBPGdk02ETR1nPA5uPkox5K428mK7+hKvPSHwKgjAI8Cp1oPBaEau3EVI29wTrdZCy74uX+UivT2TTFAbA75Bp/IAhFIab5pwec0XuWyU1wDUzaTn+yWupiX3i6GrXNsqX4OpTUOYjrQ+1I6owoqrctW/KrAYA15wxA3POPTTtPoBdK3FzNKu9qxMJdg1EEE1BEAYBmTQFr9W4WxnpZJ61j2w7myXZNavZbbP4fAjkaAZjeKUxYXYNlFDQVuOjhqUHSWY2HxnPvdIQyixNoSytXcJFUyACVPSqU9i296T8K7Eysl1Lz1PQBYRTU0gwo9MM6e1LuGsKUSFCQRB8yBR95HVezd/xZNJWEC8fn4JdU3B3guuaQpAcAydOQRPEL8EMDK8yJmnLfJRnlVS9726r/epKv4zmVHlqr+Q0lZ9Q7pKvkErcs/dTCRLn2No1p3sZpUxTgF0o6BvuODUFvRBqbzzpGuQqjmZBGARkyuj1Wkkr9b8/oUUccZ4+BUdGs9ukb3c0Z68p9DhqBPUnkxn3P0gyoyYLTSHI2PVV+oTRw9LO+yevqaRD7/0H1Pzslq+QcjRrmgJS2oRTA9PrGRGR7Z569JHSFMooPWlO/7z64kkP85H4FASh4GRaKXvZ3NXheIJteQr57KdgK6fM7Drp607PIP4AJ12OKJ14gm0mKdcNYqD5FIJoCgGEVdymKaQLhUw+hQSza7ipIl1T0ISClhthXYII+g56OsHNR2XWo1ODSRcK6X0WTUFz+7bUAAAfC0lEQVQoWhZtbMEzy7ZlbhgSrZ19uG3++gFJ8Mpkl3dOcE98sBXLt7ZZE/jyhjYs2dwKwFmmwn6dJZtb8fel6SWVdWyaArtrCje/uNaa2N0m3492tuO+Nz7GrS+vQ188iXgiid+9sg4NrV343Svr0OGo4rly2x48umiL9VrPRXj43U0A7I7m7r4E3qlvwvMrbLUvbSQZaNzbizteXY91O/finoX1+O3LH2FLcxfuXrABa3e04+36Jqv9PqNcNAWfPIXGvb24Z+HHnv4EAFBzdZlLElvK0Zy0RQ+p6zkjq9q747Y2+oRvNx+R9egUCrpmsHxrG1Zu25Pe6VIocyEMTb5y37sAgM8f67Z9Rvjc8PxqPLt8O46dMganHbpPXtfK7FOwT7w/eWoFAKBuZBUA4D+eSe0loPsUdDPP1Y8uw3MfGpPoF2dO9ryX03ntJrDeqW/27BsAnHPHW9bzSWOrUVNVjrsW1OOuBfUA3JO5bnw+tVmiXvH0umdX42sn7g8AGG5G5nT1J/Ddh+05BTqz9h+LVdv34PrnVmHeyp2449UN1rl7Fn4MALj1lfW299SOqLS9JkqFeg6vjGHa+Bqs2dFunb/28eXm+L2/u5hDU4hp5pyE5lOoLC9Dbzxpmo+Mtk7tUdcUjP55mY+Me1SUl6VFEuka3s0vfuTaZ9EUhKIn7C0jvVAhgl6JZdmQOfrIy9GcfjzJbCWd6StuJRAyoWsXzAFMWxnOd/cl0grF6ZnKbvT02T8P5YOocYSkunHv12bi+APGIZnMLrFuXI1dKFRq5pdjpozBvGv+yXbeawx/+OpM67kzT8FuPjIe++IpTYFIEwoOP1O7I7s7U/RReVmZzRlNZGg3mYjqv0mEghAaHb257/w1WMiYp5B0t7e7vc2pKWRr3kpzNOdYgsPteopMm+p09du/085eo30qJNX7Oy8jQowI8WQSNVXBjRRpQqHcO4rHj+GaySkWswsDPYNYLxuuh46qeVz/3IZXxrBXM7klHb4MtzyFihilmZh27bWXKHEj7HpXChEKQmgE2/Q9PAZCUclGU8hU+yeZtK/2evrTJ2A/H4a9Sqq7+UgnY+SUm1Bw6ZPtvENoKCFg+RT6ve8ZM23pSfb3CTgZX1Nle11VXmaZ37wyht0YoQmimCNPQf+61OQbT7BN6Cj/g/6dD68st5mPkkm2Tfh6/yzndsweoVReRtjVnllTGIhkxCCIUBBCI8j+vmESZIOYTGRTJVWfMN0EUoLtmarOSB/Af1J27qeQr/kokUyPh8+0r7NTKChNQYWIdvtpCpqD1bmfgB9jaypsrytjZZZpsCpDaKqObsrxr32USl5TQkEPSdWpqbJrComkPfPY5miOKUdzGfRE6hgRdrVn1hRyCTHOBREKQmi0eNStiYog+wtnIpNg0Vfr+v3cVP2klqcAGJqC0/fgZ5O3m48y+0wyaRKJpN3hDQCtXf4+BafQ6jbNSWVlhOqKmK9QixFp1Ul9b2Nj7PB081FvDpqCbtZx5ifo34JeEK/C8imQqxO+uiJm8yk4Q2HL3RzNsTJL6zCuDZtg8SKqLGcRCkJoFNp8lMkUEoTMq3FNKGj3c/On6HkKgKEpOCd2N+3Ben+WmkKmlWWCOU1wZvrOnONSmgKRYULy639M0xSCfjejqyvStrTUhYKzBLUf5Fidqz4Bds2uP5E0wnU9fAo6NVXltsghp/nI5mjWwl/dBFRUtY0yIUKhCPj50ysxxwx1LAbUj7uly3uCeXN9I6bNmYutLanKm6+s3olpc+YGUqUV79Q3Ydqcufj1C2swbc5czHlqBV5duxuAv6Zw9aPLMPvm1wAA//XSR5g2Z65ru5bOPkybMxe/mbcW0+bMxdItrbbzv567Fsff9CqAzKaXuFb7CDAmRudK/dRbX0dzh7t92b7vsXvyms6f392MaXPm2v50/vf9Lbhpnn1b9CaPeyvaHJrEh1vbABgVPIdVxHw/c+VoBoJrcfuMrEo7Fisrw5hqw6Q0ZezwQNcx3qdNxDGn+Sj1xeztiePgX7yIfs2noIek6jgd5oamkHqth6Sqa5U78hTU8clj3Yv3RY3kKRQBq7e3D4gpJApsNnOf6KPHP9gKAFi6pRVTxhn/2H95bzMAYM32duzrkrDkxt8WG9f5n7c3AgAeM68L+K+69TDQP7xuxMf3xhNpde7V5iz3v/kJAGDuih2YOXWsrY0KJ8y0+t3T3Y/xNZWoMle6XX3xNKEAAPW7OzB+RPpk2ONwZOcbcusmADKZMZwC+3fzjZyCUdXlGF4ZQ5tpSjl0wkicdOB4/Okfm1ARI/Qn2FNTmDSmGtvMzWUUleVlePDy4zHSpRheTWUMFxw1EUTAuZ+amHGcC//9VLR392NUdco34acp6IzW3uO2kP/RWQfjzfWN1utkkh15Cql19xjTDNbVl7BpIL+9+Gis2dGOY6eMwSX3v5d2j88fsx+eWR4sbHkgEE2hCOjuSxTcaRuUXi1F329S9mOgQu/conv87uVmOnGLeXerBdSfSGYU3C2dfUhyanXZ059uPgK8fQV2n0Xum+jkw04PLW58TSWqK2PW7/SS46fghguPwPs/OwOXnTAVgJFFrCZh/bdxyvRa7OcoZXHyQeNxyoxaHD1lTNq9Rg4rBxHhgqP28yx4p3NAbQ2OnjIG47XQVqej2euTVOGwep6Cjr53AmD8nrwK4qkkvPaeflt47GcOrsOVnz0IJx443rUPZx0+AYB7tdgwEKFQBHT3J9Da1ZexMNlgQJ+Ig9iN3WzvbRkSqIISRCi1aSauZtMxrms7jY7VNMG9OmlrZ1/G8Xb1JdDVF0eNWVG0qy+Rtncw4G3Xd4a8DkR0VbY4t/xUjKupQnVFDK1m39Wkt++oYVZkUhm5awrVlbG0EFW//AN9xZ8Ner2klKPZfwrUndxuQsF5yOlo1s1HKst9b0/ct3aTE+WsznXc2SJCoQjo6ksgkeRAEQqFRp+I/VbOKlnILUKppTNzzLbCT056TdL6pK9PwK2mgNCv6cw07YknXFfozZ19gYRQc2efVVG028XRDKSEkxP9+skko68QmoKnUDA0BfV56pOemvB185G+eCgjchEK3pPmqGH5T45KwVATrleklqUpgEAus6VTTDB7h6TWmSbBrr6ElewXBCVYBmLcQRChUASof6DmLCbLQtEdUFNQAk43i6kojmxMZc66M7a+eEzS+uSq30tNaPoK3CkUWjr7XIVCS2dfIHNVW1e/luiV7mjW++FE/zwTDkdzEDPKQLBjT49rbP/4mkoMr4xZmp8+6blpCvp30NUXTyuF7RdqOqo6fzMKOXwKXgsuK5vaw3zkPJbwiT6q1ZzmmUp/6yhtZiDGHQQRCoMcZrYmg0KHeAah2/bP7j1JqtVwi8uknE1+g99n4iWU9PfojlPVJ93P4Izbb+7oczUf6ZqC0z7uRPkUuvsSafv7qmu5oSeGJR21j2IRxTN29ycwZnj6inV0dYVdO/DSFFyijzp646h2rJz9zEcjB3DFnOlzG1ujm4/Szxu+htRrX/ORFjyQTUa3+qwGctx+DLnoo9fX7cbx08ZlVXslahau240Tpo3DWxuacOohddYk9eKqnfi4sQNfmjkZa3fsxYptbTjniAmYv2YXDp4wMi0qBjDU4meXb8fo6grEE0mc86kJICJsae7Cht17EU8yPnf4vnjvkxas2dGOGAEH1I0w6/UzGjt6ceiEkQCMMgHb9/SgaW8vKsvLsHtvL04/dB9sbOrAxiYjtFQPMe3uT2D+ml0ojxE+aTSiePYZWYW+eNKKNlm9fQ8eeHsjZuwzAvW7OwAY5ZrVserKGKorYnh/YwsAY/IZURXDsIoYduzpwbZWe9SKTkNrF/70zkY4F/Y796Teo5f4/mBTC8pjZIs4cbKpuRN/NaOkdF5atcOK2d939DBs9zCzAKkJc9mWNqxoSC+RvHxrGx54eyNGVMVw7pETsWRzK1o6+vDy6l1Wmz//Y5MtgqWsDMAABaipaCEvRlVXoMkhuFXymkLfGU0dj5WRZa7Ro406e+OoduQbOLertN1/IIVCBg1LZV4TDBOSEyLCqOoKK1Q3mXSGpGqfQ2W6oAzCXlMbjsp8NHhnxhDY0tyFbz70AS46Zj/8/tJjM7bf092PO1/bgB+ffUhWjqF82NTUiW899AGIDPvkj8462Dr3gBl2uf/4Gtz84kf4cGsbfv60UZq5IkZ4+qqT8alJo9HS2Yc/vl6Pi46ZhB8/uQJrtbLCd112LF5atRNzV+6wjp12SB1eX9/oGZanlxVwsnRzK15avdO2uq6IESaPHY5FG1uwyJzMvfi4sRO/emGN7dhHO/emHcuF9bs6cMPz7tdRn+/CdY0oI2Pl/eKqnXhx1U7fa+5q78Udr26w3q+Yt9J437Txw3H05DFYtqXN9r6Lj5uMV9fuQltXP/YfPxzVFTG8tNr9Xmt3tFvjv/O1etsEWlMZQ2dfwhKgik8fVItFG1sGpAjhIRNGYtW21G/miP1GYfX21OvZB463hDwAK6rnwLoRAIxV/n5jUjH30/cZgTHDK1A3ogrrtFX0YRNHYe2OdnzpuMlY8NFuWx+cmsJlJ0y19nWYfVB6lM4PT59ulf/246snTsUj76f2hzh5ei0AIzKoqaMPh04YiY927sX5R03E/DW7bIXy9FW/ztWnz8Avze/rvCMn2vquCwVV3fWCoyZ6mo/2Hz8cXX0Jm9ny2KlGBNaXZkZTin5ICQUVXrhu595A7e9esAEPvL0RB9TW4Gsn7R9m1yyUjVxNOPWNHWltGvf2osn80Rw+cRTOPmICbn91PV5duwufmjQat7y4Fk8sbsBrH+3GJ42duOrUg/Dooi1o7erHA29vxPKt9glr4bpGVFfE8OZPTsNpt75uTSwH1NbgomP2s9W833dUla141/pde5FIMn5+3mH4yvFTABirvH97Yjk2NqUmju+fdhCu+KeDcPur61E3sgpfn70/RlaVo70njobWLtz68jr84PTpOHbKWOztjWNzcyd+/cJaLNpkCJXzj5qI33zhSDy2aIut3vznj9kPt3zpKCSSjGEVMfTFk6iujOHfHl+Ovy/bhtHVFXjzJ6elfYZV5WUoI0J3fwIVMcLn73kH63elf9ZTxlXjK8dNwbdOOQA1lTHs7Y2D2QhlrCovs+LvlalqeGUM5WWEfz/7EJz0m9fQ0RvHtWfOwLVnHozeeAI9/UmMGlaOeSt32ib7F354Cj41aTSSScbe3jg27NqLi+99Ny1+v6aqHKfMqMUp02vxH8+uBgAs+tkZqB1RhbIywsm3LMC2tm788qIjcFDdCDy5pAG3feVoEBGSSUZDazc+89uFAIBVN55tFYnr6I2jxsxIfuidjVi1rR3nHDEBHb1x3P+N47C9rQdn3vaG8ZvbbxQ23XJ+2mf1nVMOwJdnTUZlrMy2iDp26lgsv+5zAOy1hiaPrcaLZtnr5Q4h6vQp3PzFI3HzF49Mu6fiR587BBcdsx/OvO1NzzYAcNMXjsRNX0hd55AJI7HplvOxqakTv3hmFe79+nG2wnn/MDf7ITLKVKy+8Wwccf3Ltmt++5QD8O1TDrBeJ20LpNQ4xtVU4uPfnAfAO1z6jR+fhm1t3Tj5lgXWsen7jHT9vMNiSAmFVp8MWzeUjdjNGRgWThu2m12+pbMPzZ29+O4pB+AXFxwOAHjwnY1WMlKn+Z4dbT0YX1OJn5xzKC44aj+cd+dbWL8rJRDPOHQfjBxWjmeWb8e+o6pQN7IKY2sq0NEbx5WfPQhzzj0UAHDfG5+guz+Ba86YgfOPmojP3W7849WOqMIGc8W635hqm9mlusL+05o4uhqjh1fghguPsB0fXV2B0dWj8dC3TrAdO2ryGDxx5WzMvvk17NjTg31HDsPo6gr862cPwsPvbsa2tm5879SD8P/OOdR2PaWWKwfh+BGVvuYgtapToYcXHr0fdrb3YNHGFtzyxSNxqRljr3Cq8CpIxrmyHVFVbkU5qXNV5TErOW5cTaVtwldtysoM85VK6HPSG0/ivq/PQm88YQmFUdUVVi0dJdDH1VTi5Om11krYurbmD9AnP/W8pqrccsJOq62xfgN6jP94RylrnUwmDr3mjz4xjhvhXR47KONq0hP+gjKttgZ//e6JnueV6SiIxUAfoy7c9NwEP/NYNvWcwmBIOZqzddQqTTeX/XRzxRmO6VY6YVtbN3r6k7Z/pLqRVWjaa4xP/SS7+xOoNZ1b40eksikV42oqrfPqUf0g9X98tbobP6LSVtt+xj4jbNfS0W3Kzutlg5qsxmtjVROG3zXVZzM8oO1WXX9cTUqI5BvRo341bv/kTmets42zCJxCOR31zGt99a0LBTf8JiOFcpTqobu6YM1n8nXrK5D+XTrrHQVhTARx/Nn+JvSCeHqmM/k4uHMRiANJqHcnonOIaB0R1RPRHJfzVUT0uHn+fSKaFmZ/ss0K7o9zTu/LB2eMultc+AZzta//Ixk2UUOg6DKsdqTRxm2SGTei0gqTG+5wvOuTilr5jKuptF1nxr4poTDesdJz2ky9JqlMxLR7O/G75ngtvjwIlvDRhEK+mdVqMeE2EbsVeXO+divx4Obb0Scq1WfnHgTWdQNMtupy+uj11W+u36XzOvp+0M7fZ288e695WZ5C3A31GeQa3JWLcAsiuMMktLsTUQzAPQDOBXA4gMuI6HBHs+8AaGXm6QBuB/CfYfUHSK3CM5UUVqi8gKYAW+UNFE5tpsElukb5GfQVW+2IKkso6KUZLA3AZZKpqSy3wuTUqlD9E+haiPpfG1dTaZuApmomjnRNwX4vp9AIilpRZSsUsl3NKuExZniFJRQybU2ZCSVTgqz83CaCoNqV26rTuQeBIsjEaWnIHkIxV60PsJtibZqC4/fRMUgSNdUCK3ehkP0bS9l8dAKAemb+hJn7ADwG4CJHm4sA/Nl8/iSAM8hPr8oTNeFmqhmvaDRX7ZkqRw4kTqHgjGOfNKYaW1sMQaFPinUjq6wwQb2/emy0czWWZPa0t+v/+Mqc4Fx91vpc26kpeJlDMqH+P9wEgNdq2LhfbqYEIrLs4n6JcUFw+hT8cGuTz4o8188bSH3fXoqSn48mE7qg1Sd+pxAvha1cgdQeCtkQhsaTDWE6micB2Kq9bgDg9ORYbZg5TkR7AIwH0DTQnXljfSMeXWR0p6WzD2eZkRR+bDIrZL77SXOg9gOBVxkBxfR9RlgOSrv5qAodvXGcedsb2KRF/ehZlMPMWPC6kVVoNHMN1GSktAg1Ieq2+LE1lWju7EubDPRJy2lrddryx+Q4Sal76qsnVRjML9ZbOQRHBMxHGW7WI6osL7NMbpl2LstEVXkM/Yk4KmPp/XTmyQy0UMjFbKFQWovXKjefSUtf5MS06zuFeJR+PD9UaSRn9dxMjKupREtnX0Dj5eAiTKHg9nk4v+kgbUBEVwC4AgCmTp2a9oYgjKgqx3lHTsCUscOxfU8PEgGKiR2870hMGTccW1o6M7YdKGbsO8Lq4+Sx1djS3IVxNZW49IQpeHtDEw6bOAo1VTHsM3KYLULl3E9NwLqdexFPJnGI2e/tbd04+4gJVptrzjgYr6zZiatOnY4nl2zFNz89DZWxMlz52YPw7VOmATDyGP62pAEH1aX8Bf/zjVl47kMjQgkAHrh8Fnr6k5i5/1h8+bjJaZUiAeC0Q/bB0mNaMWXscIx1mJ2y4dYvH42/vrcZR04abR27+59n4sklDTiorsbzfYdPHIUfnj7dqtCZiWvPPBiVsTJcdMx+IBC2tHThqlOn59RnxfX/53C8+3EzTjhgnOu5SWOq8flj98PCjxpdJ52vnbQ/qivLccD44Thm6hg0tHbjqMmpqqFPX/VprNLyBwDgyStnWxFhXtz0hU+5fmeKrxw/BQ2t3fjhGTNsxx+74iSrlHiufPm4ydja0oXhlTFb2evR1RX4yTmH4IxD98XTy7bhis8cmNP1H7viJGxp7sKo6vK8BKPipAPG4wenTcfln55mHfvDV2diY1MnanwWJU9eORsvrtqJmqpy/OL8w2xRYIp7vzbTs4+/uugItPfEbSbaqCDndoADdmGi2QBuYOazzdc/BQBmvllr87LZ5l0iKgewE0Ad+3Rq1qxZvHjx4lD6LAiCUKoQ0RJmnpWpXZg+hQ8AzCCiA4ioEsClAJ5ztHkOwOXm84sBLPATCIIgCEK4hGY+Mn0EPwDwMoAYgAeZeTUR/RLAYmZ+DsADAP5CRPUAWmAIDkEQBKFAhJrRzMzzAMxzHLtOe94D4Mth9kEQBEEIzpDKaBYEQRD8EaEgCIIgWIhQEARBECxEKAiCIAgWIhQEQRAEi9CS18KCiBoBpO+HGIxahFBCowiQcQ8tZNxDi6Dj3p+Z6zI1KjqhkA9EtDhIRl+pIeMeWsi4hxYDPW4xHwmCIAgWIhQEQRAEi6EmFO4vdAcKhIx7aCHjHloM6LiHlE9BEARB8GeoaQqCIAiCD0NCKBDROUS0jojqiWhOofsz0BDRg0S0m4hWacfGEdF8ItpgPo41jxMR3Wl+FiuIaGbhep47RDSFiBYS0VoiWk1E15jHS3rcAEBEw4hoERF9aI79RvP4AUT0vjn2x82S9SCiKvN1vXl+WiH7nw9EFCOiZUT0gvm65McMAES0iYhWEtFyIlpsHgvlt17yQoGIYgDuAXAugMMBXEZEhxe2VwPOnwCc4zg2B8BrzDwDwGvma8D4HGaYf1cA+GNEfRxo4gB+xMyHATgJwPfN77XUxw0AvQBOZ+ajARwD4BwiOgnAfwK43Rx7K4DvmO2/A6CVmacDuN1sV6xcA2Ct9noojFlxGjMfo4WfhvNbZ+aS/gMwG8DL2uufAvhpofsVwjinAVilvV4HYKL5fCKAdebz+wBc5taumP8APAvgrCE47uEAlsLY/7wJQLl53Prdw9jTZLb5vNxsR4Xuew5jnWxOfqcDeAHGdr4lPWZt7JsA1DqOhfJbL3lNAcAkAFu11w3msVJnX2beAQDm4z7m8ZL7PEzTwLEA3scQGbdpRlkOYDeA+QA+BtDGzHGziT4+a+zm+T0Axkfb4wHhDgA/AaA2WB+P0h+zggG8QkRLzD3rgZB+66FusjNIcNsxfiiHXJXU50FEIwA8BeBaZm4nchue0dTlWNGOm5kTAI4hojEAngZwmFsz87Hox05EFwDYzcxLiOhUddilacmM2cHJzLydiPYBMJ+IPvJpm9fYh4Km0ABgivZ6MoDtBepLlOwiookAYD7uNo+XzOdBRBUwBMIjzPx383DJj1uHmdsAvA7DrzKGiNRCTx+fNXbz/GgY298WEycDuJCINgF4DIYJ6Q6U9pgtmHm7+bgbxiLgBIT0Wx8KQuEDADPMKIVKGPtAP1fgPkXBcwAuN59fDsPmro5/w4xQOAnAHqWCFhNkqAQPAFjLzLdpp0p63ABARHWmhgAiqgZwJgzn60IAF5vNnGNXn8nFABawaWwuFpj5p8w8mZmnwfgfXsDMX0UJj1lBRDVENFI9B/A5AKsQ1m+90A6UiJw05wFYD8Pu+vNC9yeE8T0KYAeAfhirhO/AsJ++BmCD+TjObEsworE+BrASwKxC9z/HMZ8CQyVeAWC5+XdeqY/bHMtRAJaZY18F4Drz+IEAFgGoB/A3AFXm8WHm63rz/IGFHkOe4z8VwAtDZczmGD80/1arOSys37pkNAuCIAgWQ8F8JAiCIAREhIIgCIJgIUJBEARBsBChIAiCIFiIUBAEQRAsRCgIQwYiSphVJtWfb8VcIrqSiL4xAPfdRES1ObzvbCK6gYjGEtG8fPshCEEYCmUuBEHRzczHBG3MzPeG2ZkA/BOM5KzPAHinwH0RhggiFIQhj1k64XEAp5mH/pmZ64noBgAdzHwrEV0N4EoYJbvXMPOlRDQOwIMwkou6AFzBzCuIaDyMhMI6GIlTpN3rawCuBlAJo4DfVWzUMdL7cwmMar4HArgIwL4A2onoRGa+MIzPQBAUYj4ShhLVDvPRJdq5dmY+AcDdMGrqOJkD4FhmPgqGcACAGwEsM4/9DMDD5vHrAbzNzMfCKDkwFQCI6DAAl8AobnYMgASArzpvxMyPA5gJoxT6kTCylo8VgSBEgWgKwlDCz3z0qPZ4u8v5FQAeIaJnADxjHjsFwJcAgJkXENF4IhoNw9zzRfP4XCJqNdufAeA4AB+Y1VyrkSpi5mQGjDIFADCcmfcGGJ8g5I0IBUEwYI/nivNhTPYXAvgPIjoC/iWK3a5BAP7MzD/164i53WItgHIiWgNgorl3wg+Z+S3/YQhCfoj5SBAMLtEe39VPEFEZgCnMvBDGJi9jAIwA8CZM849Z47+Jmdsdx88FMNa81GsALjZr4qs9dvd3doSN7RbnwvAn/BeMAmjHiEAQokA0BWEoUW2uuBUvMbMKS60iovdhLJQuc7wvBuCvpmmIYOwJ3GY6oh8iohUwHM2qjPGNAB4loqUA3gCwBQCYeQ0R/QLGDlplMKrafh/AZpe+zoThkL4KwG0u5wUhFKRKqjDkMaOPZjFzU6H7IgiFRsxHgiAIgoVoCoIgCIKFaAqCIAiChQgFQRAEwUKEgiAIgmAhQkEQBEGwEKEgCIIgWIhQEARBECz+PzGoNR/2UmXuAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def mddpg(n_episodes=2000, print_every=100):\n",
    "    scores = []\n",
    "    \n",
    "    scores_deque = deque(maxlen=print_every)\n",
    "    \n",
    "\n",
    "    for episode in range(n_episodes):\n",
    "        env_info = env.reset(train_mode=True)[brain_name]            # reset environment\n",
    "        states = env_info.vector_observations                        # get starting states\n",
    "        actor_0.reset()                                              # reset the agent noise\n",
    "        actor_1.reset()                                              \n",
    "                                              \n",
    "        score = np.zeros(n_agents)\n",
    "        \n",
    "        while True:\n",
    "            action_0 = actor_0.act(states[ACTOR_0_Number], ADD_NOISE)#predict actions using current actor NN (local NN is used)\n",
    "            action_1 = actor_1.act(states[ACTOR_1_Number], ADD_NOISE)#predict actions of second agent using current actor NN (local NN is used)\n",
    "            actions = np.concatenate((action_0, action_1)) #add actions together\n",
    "        \n",
    "            env_info = env.step(actions)[brain_name]              # act on environment to move a timestep ahead                           \n",
    "            next_states = env_info.vector_observations               # get state from environment since its now in next state        \n",
    "            rewards = env_info.rewards                               # rewards for both agents       \n",
    "            dones = env_info.local_done                              # check if any of the agents is \"done\"       \n",
    "\n",
    "            #learning step preparation: add SARS tuple to experience replay buffer\n",
    "            actor_0.step(states[ACTOR_0_Number], action_0, rewards[ACTOR_0_Number], next_states[ACTOR_0_Number], dones[ACTOR_0_Number])\n",
    "            actor_1.step(states[ACTOR_1_Number], action_1, rewards[ACTOR_1_Number], next_states[ACTOR_1_Number], dones[ACTOR_1_Number])\n",
    "            \n",
    "            #actual learning step, in which the individual actor, that was passed along is trained, together with the critic\n",
    "            critic.step(actor_0, memory)\n",
    "            critic.step(actor_1, memory)\n",
    "\n",
    "            score += rewards       # update total score in this episode     \n",
    "            states = next_states   # prepare for next time sequence        \n",
    "                                                        \n",
    "            if np.any( dones ):    # break if any agent is done       \n",
    "                break                                        \n",
    "\n",
    "        \n",
    "\n",
    "        scores.append(np.max(score))\n",
    "        scores_deque.append(np.max(score))\n",
    "        \n",
    "        print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(episode, np.mean(scores_deque)), end=\"\")\n",
    "        print('Current Score: ' + str(score))\n",
    "        torch.save(actor_0.target.state_dict(), 'checkpoint_actor_0.pth')\n",
    "        torch.save(actor_1.target.state_dict(), 'checkpoint_actor_1.pth')\n",
    "        torch.save(critic.target.state_dict(), 'checkpoint_critic.pth')\n",
    "        if episode % print_every == 0:\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(episode, np.mean(scores_deque)))\n",
    "            \n",
    "        if np.mean(scores_deque) >= 0.5: #mean of last 100 scores above 0.5: solved\n",
    "            print('\\nEnvironment solved after {:d} episodes!\\tMean Score: {:.2f}'.format(episode, np.mean(scores_deque)))\n",
    "            break   \n",
    "\n",
    "    plt.plot(np.arange(1, len(scores)+1), scores)\n",
    "    plt.ylabel('Score')\n",
    "    plt.xlabel('Episode #')\n",
    "    plt.show()    \n",
    "\n",
    "\n",
    "# train the agent\n",
    "mddpg()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "DRL2",
   "language": "python",
   "name": "drl2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}