easy-rl/projects/notebooks/1.QLearning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1、定义算法\n",
    "强化学习算法的模式都比较固定，一般包括sample（即训练时采样动作），predict（测试时预测动作），update（算法更新）以及保存模型和加载模型等几个方法，其中对于每种算法samle和update的方式是不相同，而其他方法就大同小异。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import math\n",
    "import torch\n",
    "from collections import defaultdict\n",
    "\n",
    "class QLearning(object):\n",
    "    def __init__(self,n_states,\n",
    "                 n_actions,cfg):\n",
    "        self.n_actions = n_actions \n",
    "        self.lr = cfg.lr  # 学习率\n",
    "        self.gamma = cfg.gamma  \n",
    "        self.epsilon = cfg.epsilon_start\n",
    "        self.sample_count = 0  \n",
    "        self.epsilon_start = cfg.epsilon_start\n",
    "        self.epsilon_end = cfg.epsilon_end\n",
    "        self.epsilon_decay = cfg.epsilon_decay\n",
    "        self.Q_table  = defaultdict(lambda: np.zeros(n_actions)) # 用嵌套字典存放状态->动作->状态-动作值（Q值）的映射，即Q表\n",
    "    def sample(self, state):\n",
    "        ''' 采样动作，训练时用\n",
    "        '''\n",
    "        self.sample_count += 1\n",
    "        self.epsilon = self.epsilon_end + (self.epsilon_start - self.epsilon_end) * \\\n",
    "            math.exp(-1. * self.sample_count / self.epsilon_decay) # epsilon是会递减的，这里选择指数递减\n",
    "        # e-greedy 策略\n",
    "        if np.random.uniform(0, 1) > self.epsilon:\n",
    "            action = np.argmax(self.Q_table[str(state)]) # 选择Q(s,a)最大对应的动作\n",
    "        else:\n",
    "            action = np.random.choice(self.n_actions) # 随机选择动作\n",
    "        return action\n",
    "    def predict(self,state):\n",
    "        ''' 预测或选择动作，测试时用\n",
    "        '''\n",
    "        action = np.argmax(self.Q_table[str(state)])\n",
    "        return action\n",
    "    def update(self, state, action, reward, next_state, done):\n",
    "        Q_predict = self.Q_table[str(state)][action] \n",
    "        if done: # 终止状态\n",
    "            Q_target = reward  \n",
    "        else:\n",
    "            Q_target = reward + self.gamma * np.max(self.Q_table[str(next_state)]) \n",
    "        self.Q_table[str(state)][action] += self.lr * (Q_target - Q_predict)\n",
    "    def save(self,path):\n",
    "        import dill\n",
    "        torch.save(\n",
    "            obj=self.Q_table,\n",
    "            f=path+\"Qleaning_model.pkl\",\n",
    "            pickle_module=dill\n",
    "        )\n",
    "        print(\"保存模型成功！\")\n",
    "    def load(self, path):\n",
    "        import dill\n",
    "        self.Q_table =torch.load(f=path+'Qleaning_model.pkl',pickle_module=dill)\n",
    "        print(\"加载模型成功！\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2、定义训练\n",
    "强化学习算法的训练方式也比较固定，如下：\n",
    "```python\n",
    "for i_ep in range(train_eps): # 遍历每个回合\n",
    "    state = env.reset()  # 重置环境,即开始新的回合\n",
    "    while True: # 对于一些比较复杂的游戏可以设置每回合最大的步长，例如while ep_step<100，就是每回合最大步长为100。\n",
    "        action = agent.sample(state)  # 根据算法采样一个动作\n",
    "        next_state, reward, done, _ = env.step(action)  # 与环境进行一次动作交互\n",
    "        agent.memory.push(state, action, reward, next_state, done) # 记录memory\n",
    "        agent.update(state, action, reward, next_state, done)  # 算法更新\n",
    "        state = next_state  # 更新状态\n",
    "        if done:\n",
    "            break\n",
    "```\n",
    "首先对于每个回合，回合开始时环境需要重置，好比我们每次开一把游戏需要从头再来一样。我们可以设置智能体在每回合数的最大步长，尤其是对于比较复杂的游戏，这样做的好处之一就是帮助智能体在训练中快速收敛，比如我们先验地知道最优解的大概步数，那么理论上智能体收敛时也应该是这个步数附近，设置最大步数可以方便智能体接近这个最优解。在每个回合中，智能体首先需要采样（sample），或者说采用探索策略例如常见的$\\varepsilon$-greedy策略或者UCB探索策略等等。采样的过程是将当前的状态state作为输入，智能体采样输出动作action。然后环境根据采样出来的动作反馈出下一个状态以及相应的reward等信息。接下来对于具有memory的智能体例如包含replay memory的DQN来说，需要将相应的transition（记住这个词，中文不好翻译，通常是状态、动作、奖励等信息）。紧接着就是智能体更新，对于深度强化学习此时一般从memory中随机采样一些transition进行更新，对于Q learning一般是采样上一次的transition。更新公式是比较关键的部分，但是也很通用，一般基于值的算法更新公式都是一个套路如下：\n",
    "$$\n",
    "y_{j}= \\begin{cases}r_{j} & \\text { for terminal } s_{t+1} \\\\ r_{j}+\\gamma \\max _{a^{\\prime}} Q\\left(s_{t+1}, a^{\\prime} ; \\theta\\right) & \\text { for non-terminal } s_{t+1}\\end{cases}\n",
    "$$\n",
    "智能体更新完之后，通常需要更新状态，即```state = next_state```，然后会检查是否完成了这一回合的游戏，即```done==True```，注意完成并不代表这回合成功，也有可能是失败的太离谱，等同学们有了自定义强化学习环境的经验就知道了（等你长大就知道了XD）。\n",
    "如果需要记录奖励、损失等等的话可以再加上，如下方代码，实际项目中更多地使用tensorboard来记录相应的数据，甚至于笔者就在这些教学代码中使用过，但是看起来有些繁琐，容易给大家增加不必要的学习难度，因此学有余力以及需要在项目研究中做强化学习的可以去看看，也很简单。\n",
    "此外稍微复杂一些的强化学习不是一次性写完代码就能收敛的，这时需要我们做一个调参侠。为了检查我们参数调得好不好，可以在终端print出奖励、损失以及epsilon等随着回合数的变化，这点说明一下强化学习的训练过程一般都是先探索然后收敛的，官方的话就是权衡exploration and exploitation。e-greedy策略的做法就是前期探索，然后逐渐减小探索率至慢慢收敛，也就是这个epsilon。这个值越大比如0.9就说明智能体90%的概率在随机探索，通常情况下会设置三个值，epsilon_start、epsilon_end以及epsilon_decay，即初始值、终止值和衰减率，其中初始值一般是0.95不变，终止值是0.01，也就是说即使在收敛阶段也让智能体保持很小概率的探索，这样做的原因就是智能体已经学出了一个不错的策略，但是保不齐还有更好的策略，好比我们知道要出人头地学历高比较重要，但是“人还是要有梦想的，万一实现了呢”，总是存在意外的可能，对吧。回归正题，比较关键的是epsilon_decay这个衰减率，这个epsilon衰减太快了学来的策略往往过拟合，好比一条只能选择一朵花的花道上，你早早选择了一朵看起来还可以的花，却错过了后面更多的好花。但是衰减的太慢会影响收敛的速度，好比你走过了花道的尽头也还没选出一朵花来，相比前者不如更甚。当然强化学习的调参相比于深度学习只能说是有过之无不及，比较复杂，不止epsilon这一个，这就需要同学们的耐心学习了。\n",
    "强化学习测试的代码跟训练基本上是一样的，因此我放到同一个代码段里。相比于训练代码，测试代码主要有以下几点不同：1、测试模型的过程是不需要更新的，这个是不言而喻的；2、测试代码不需要采样（sample）动作，相比之代替的是预测（sample）动作，其区别就是采样动作时可能会使用各种策略例如$\\varepsilon$-greedy策略，而预测动作不需要，只需要根据训练时学习好的Q表或者网络模型代入状态得到动作即可；3、测试过程终端一般只需要看奖励，不需要看epislon等，反正它在测试中也是无意义的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(cfg,env,agent):\n",
    "    print('开始训练！')\n",
    "    print(f'环境:{cfg.env_name}, 算法:{cfg.algo_name}, 设备:{cfg.device}')\n",
    "    rewards = []  # 记录奖励\n",
    "    for i_ep in range(cfg.train_eps):\n",
    "        ep_reward = 0  # 记录每个回合的奖励\n",
    "        state = env.reset()  # 重置环境,即开始新的回合\n",
    "        while True:\n",
    "            action = agent.sample(state)  # 根据算法采样一个动作\n",
    "            next_state, reward, done, _ = env.step(action)  # 与环境进行一次动作交互\n",
    "            agent.update(state, action, reward, next_state, done)  # Q学习算法更新\n",
    "            state = next_state  # 更新状态\n",
    "            ep_reward += reward\n",
    "            if done:\n",
    "                break\n",
    "        rewards.append(ep_reward)\n",
    "        print(f\"回合：{i_ep+1}/{cfg.train_eps}，奖励：{ep_reward:.1f}，Epsilon：{agent.epsilon}\")\n",
    "    print('完成训练！')\n",
    "    return {\"rewards\":rewards}\n",
    "def test(cfg,env,agent):\n",
    "    print('开始测试！')\n",
    "    print(f'环境：{cfg.env_name}, 算法：{cfg.algo_name}, 设备：{cfg.device}')\n",
    "    rewards = []  # 记录所有回合的奖励\n",
    "    for i_ep in range(cfg.test_eps):\n",
    "        ep_reward = 0  # 记录每个episode的reward\n",
    "        state = env.reset()  # 重置环境, 重新开一局（即开始新的一个回合）\n",
    "        while True:\n",
    "            action = agent.predict(state)  # 根据算法选择一个动作\n",
    "            next_state, reward, done, _ = env.step(action)  # 与环境进行一个交互\n",
    "            state = next_state  # 更新状态\n",
    "            ep_reward += reward\n",
    "            if done:\n",
    "                break\n",
    "        rewards.append(ep_reward)\n",
    "        print(f\"回合数：{i_ep+1}/{cfg.test_eps}, 奖励：{ep_reward:.1f}\")\n",
    "    print('完成测试！')\n",
    "    return {\"rewards\":rewards}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3、定义环境\n",
    "\n",
    "OpenAI Gym中其实集成了很多强化学习环境，足够大家学习了，但是在做强化学习的应用中免不了要自己创建环境，比如在本项目中其实不太好找到Qlearning能学出来的环境，Qlearning实在是太弱了，需要足够简单的环境才行，因此本项目写了一个环境，大家感兴趣的话可以看一下，一般环境接口最关键的部分即使reset和step。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import turtle\n",
    "import numpy as np\n",
    "\n",
    "# turtle tutorial : https://docs.python.org/3.3/library/turtle.html\n",
    "\n",
    "def GridWorld(gridmap=None, is_slippery=False):\n",
    "    if gridmap is None:\n",
    "        gridmap = ['SFFF', 'FHFH', 'FFFH', 'HFFG']\n",
    "    env = gym.make(\"FrozenLake-v0\", desc=gridmap, is_slippery=False)\n",
    "    env = FrozenLakeWapper(env)\n",
    "    return env\n",
    "\n",
    "\n",
    "class FrozenLakeWapper(gym.Wrapper):\n",
    "    def __init__(self, env):\n",
    "        gym.Wrapper.__init__(self, env)\n",
    "        self.max_y = env.desc.shape[0]\n",
    "        self.max_x = env.desc.shape[1]\n",
    "        self.t = None\n",
    "        self.unit = 50\n",
    "\n",
    "    def draw_box(self, x, y, fillcolor='', line_color='gray'):\n",
    "        self.t.up()\n",
    "        self.t.goto(x * self.unit, y * self.unit)\n",
    "        self.t.color(line_color)\n",
    "        self.t.fillcolor(fillcolor)\n",
    "        self.t.setheading(90)\n",
    "        self.t.down()\n",
    "        self.t.begin_fill()\n",
    "        for _ in range(4):\n",
    "            self.t.forward(self.unit)\n",
    "            self.t.right(90)\n",
    "        self.t.end_fill()\n",
    "\n",
    "    def move_player(self, x, y):\n",
    "        self.t.up()\n",
    "        self.t.setheading(90)\n",
    "        self.t.fillcolor('red')\n",
    "        self.t.goto((x + 0.5) * self.unit, (y + 0.5) * self.unit)\n",
    "\n",
    "    def render(self):\n",
    "        if self.t == None:\n",
    "            self.t = turtle.Turtle()\n",
    "            self.wn = turtle.Screen()\n",
    "            self.wn.setup(self.unit * self.max_x + 100,\n",
    "                          self.unit * self.max_y + 100)\n",
    "            self.wn.setworldcoordinates(0, 0, self.unit * self.max_x,\n",
    "                                        self.unit * self.max_y)\n",
    "            self.t.shape('circle')\n",
    "            self.t.width(2)\n",
    "            self.t.speed(0)\n",
    "            self.t.color('gray')\n",
    "            for i in range(self.desc.shape[0]):\n",
    "                for j in range(self.desc.shape[1]):\n",
    "                    x = j\n",
    "                    y = self.max_y - 1 - i\n",
    "                    if self.desc[i][j] == b'S':  # Start\n",
    "                        self.draw_box(x, y, 'white')\n",
    "                    elif self.desc[i][j] == b'F':  # Frozen ice\n",
    "                        self.draw_box(x, y, 'white')\n",
    "                    elif self.desc[i][j] == b'G':  # Goal\n",
    "                        self.draw_box(x, y, 'yellow')\n",
    "                    elif self.desc[i][j] == b'H':  # Hole\n",
    "                        self.draw_box(x, y, 'black')\n",
    "                    else:\n",
    "                        self.draw_box(x, y, 'white')\n",
    "            self.t.shape('turtle')\n",
    "\n",
    "        x_pos = self.s % self.max_x\n",
    "        y_pos = self.max_y - 1 - int(self.s / self.max_x)\n",
    "        self.move_player(x_pos, y_pos)\n",
    "\n",
    "\n",
    "class CliffWalkingWapper(gym.Wrapper):\n",
    "    def __init__(self, env):\n",
    "        gym.Wrapper.__init__(self, env)\n",
    "        self.t = None\n",
    "        self.unit = 50\n",
    "        self.max_x = 12\n",
    "        self.max_y = 4\n",
    "\n",
    "    def draw_x_line(self, y, x0, x1, color='gray'):\n",
    "        assert x1 > x0\n",
    "        self.t.color(color)\n",
    "        self.t.setheading(0)\n",
    "        self.t.up()\n",
    "        self.t.goto(x0, y)\n",
    "        self.t.down()\n",
    "        self.t.forward(x1 - x0)\n",
    "\n",
    "    def draw_y_line(self, x, y0, y1, color='gray'):\n",
    "        assert y1 > y0\n",
    "        self.t.color(color)\n",
    "        self.t.setheading(90)\n",
    "        self.t.up()\n",
    "        self.t.goto(x, y0)\n",
    "        self.t.down()\n",
    "        self.t.forward(y1 - y0)\n",
    "\n",
    "    def draw_box(self, x, y, fillcolor='', line_color='gray'):\n",
    "        self.t.up()\n",
    "        self.t.goto(x * self.unit, y * self.unit)\n",
    "        self.t.color(line_color)\n",
    "        self.t.fillcolor(fillcolor)\n",
    "        self.t.setheading(90)\n",
    "        self.t.down()\n",
    "        self.t.begin_fill()\n",
    "        for i in range(4):\n",
    "            self.t.forward(self.unit)\n",
    "            self.t.right(90)\n",
    "        self.t.end_fill()\n",
    "\n",
    "    def move_player(self, x, y):\n",
    "        self.t.up()\n",
    "        self.t.setheading(90)\n",
    "        self.t.fillcolor('red')\n",
    "        self.t.goto((x + 0.5) * self.unit, (y + 0.5) * self.unit)\n",
    "\n",
    "    def render(self):\n",
    "        if self.t == None:\n",
    "            self.t = turtle.Turtle()\n",
    "            self.wn = turtle.Screen()\n",
    "            self.wn.setup(self.unit * self.max_x + 100,\n",
    "                          self.unit * self.max_y + 100)\n",
    "            self.wn.setworldcoordinates(0, 0, self.unit * self.max_x,\n",
    "                                        self.unit * self.max_y)\n",
    "            self.t.shape('circle')\n",
    "            self.t.width(2)\n",
    "            self.t.speed(0)\n",
    "            self.t.color('gray')\n",
    "            for _ in range(2):\n",
    "                self.t.forward(self.max_x * self.unit)\n",
    "                self.t.left(90)\n",
    "                self.t.forward(self.max_y * self.unit)\n",
    "                self.t.left(90)\n",
    "            for i in range(1, self.max_y):\n",
    "                self.draw_x_line(\n",
    "                    y=i * self.unit, x0=0, x1=self.max_x * self.unit)\n",
    "            for i in range(1, self.max_x):\n",
    "                self.draw_y_line(\n",
    "                    x=i * self.unit, y0=0, y1=self.max_y * self.unit)\n",
    "\n",
    "            for i in range(1, self.max_x - 1):\n",
    "                self.draw_box(i, 0, 'black')\n",
    "            self.draw_box(self.max_x - 1, 0, 'yellow')\n",
    "            self.t.shape('turtle')\n",
    "\n",
    "        x_pos = self.s % self.max_x\n",
    "        y_pos = self.max_y - 1 - int(self.s / self.max_x)\n",
    "        self.move_player(x_pos, y_pos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "def env_agent_config(cfg,seed=1):\n",
    "    '''创建环境和智能体\n",
    "    Args:\n",
    "        cfg ([type]): [description]\n",
    "        seed (int, optional): 随机种子. Defaults to 1.\n",
    "    Returns:\n",
    "        env [type]: 环境\n",
    "        agent : 智能体\n",
    "    '''    \n",
    "    env = gym.make(cfg.env_name)  \n",
    "    env = CliffWalkingWapper(env)\n",
    "    env.seed(seed) # 设置随机种子\n",
    "    n_states = env.observation_space.n # 状态维度\n",
    "    n_actions = env.action_space.n # 动作维度\n",
    "    agent = QLearning(n_states,n_actions,cfg)\n",
    "    return env,agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4、设置参数\n",
    "\n",
    "到这里所有qlearning模块就算完成了，下面需要设置一些参数，方便大家“炼丹”，其中默认的是笔者已经调好的～。另外为了定义了一个画图函数，用来描述奖励的变化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "import argparse\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "def get_args():\n",
    "    \"\"\" \n",
    "    \"\"\"\n",
    "    curr_time = datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\")  # 获取当前时间\n",
    "    parser = argparse.ArgumentParser(description=\"hyperparameters\")      \n",
    "    parser.add_argument('--algo_name',default='Q-learning',type=str,help=\"name of algorithm\")\n",
    "    parser.add_argument('--env_name',default='CliffWalking-v0',type=str,help=\"name of environment\")\n",
    "    parser.add_argument('--train_eps',default=400,type=int,help=\"episodes of training\") # 训练的回合数\n",
    "    parser.add_argument('--test_eps',default=20,type=int,help=\"episodes of testing\") # 测试的回合数\n",
    "    parser.add_argument('--gamma',default=0.90,type=float,help=\"discounted factor\") # 折扣因子\n",
    "    parser.add_argument('--epsilon_start',default=0.95,type=float,help=\"initial value of epsilon\") #  e-greedy策略中初始epsilon\n",
    "    parser.add_argument('--epsilon_end',default=0.01,type=float,help=\"final value of epsilon\") # e-greedy策略中的终止epsilon\n",
    "    parser.add_argument('--epsilon_decay',default=300,type=int,help=\"decay rate of epsilon\") # e-greedy策略中epsilon的衰减率\n",
    "    parser.add_argument('--lr',default=0.1,type=float,help=\"learning rate\")\n",
    "    parser.add_argument('--device',default='cpu',type=str,help=\"cpu or cuda\")                \n",
    "    args = parser.parse_args([])                          \n",
    "    return args\n",
    "curr_time = datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\") # 获取当前时间\n",
    "\n",
    "def smooth(data, weight=0.9):  \n",
    "    '''用于平滑曲线，类似于Tensorboard中的smooth\n",
    "\n",
    "    Args:\n",
    "        data (List):输入数据\n",
    "        weight (Float): 平滑权重，处于0-1之间，数值越高说明越平滑，一般取0.9\n",
    "\n",
    "    Returns:\n",
    "        smoothed (List): 平滑后的数据\n",
    "    '''\n",
    "    last = data[0]  # First value in the plot (first timestep)\n",
    "    smoothed = list()\n",
    "    for point in data:\n",
    "        smoothed_val = last * weight + (1 - weight) * point  # 计算平滑值\n",
    "        smoothed.append(smoothed_val)                    \n",
    "        last = smoothed_val                                \n",
    "    return smoothed\n",
    "\n",
    "def plot_rewards(rewards,cfg, tag='train'):\n",
    "    sns.set()\n",
    "    plt.figure()  # 创建一个图形实例，方便同时多画几个图\n",
    "    plt.title(\"learning curve on {} of {} for {}\".format(\n",
    "        cfg.device, cfg.algo_name, cfg.env_name))\n",
    "    plt.xlabel('epsiodes')\n",
    "    plt.plot(rewards, label='rewards')\n",
    "    plt.plot(smooth(rewards), label='smoothed')\n",
    "    plt.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5、我准备好了！\n",
    "\n",
    "到现在我们真的可以像海绵宝宝那样大声说出来“我准备好了！“，跟着注释来看下效果吧～。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "开始训练！\n",
      "环境:CliffWalking-v0, 算法:Q-learning, 设备:cpu\n",
      "回合：1/400，奖励：-1668.0，Epsilon：0.3771901652370099\n",
      "回合：2/400，奖励：-2328.0，Epsilon：0.03210668110464856\n",
      "回合：3/400，奖励：-152.0，Epsilon：0.02331928797825333\n",
      "回合：4/400，奖励：-296.0，Epsilon：0.014965661602689185\n",
      "回合：5/400，奖励：-168.0，Epsilon：0.012836430915462094\n",
      "回合：6/400，奖励：-149.0，Epsilon：0.011726126490407173\n",
      "回合：7/400，奖励：-274.0，Epsilon：0.010963239247691907\n",
      "回合：8/400，奖励：-127.0，Epsilon：0.010630787152305933\n",
      "回合：9/400，奖励：-356.0，Epsilon：0.010267816440118822\n",
      "回合：10/400，奖励：-105.0，Epsilon：0.0101887270555826\n",
      "回合：11/400，奖励：-162.0，Epsilon：0.01010998036181645\n",
      "回合：12/400，奖励：-124.0，Epsilon：0.010072745604688937\n",
      "回合：13/400，奖励：-125.0，Epsilon：0.010047956858279448\n",
      "回合：14/400，奖励：-69.0，Epsilon：0.010038103335373512\n",
      "回合：15/400，奖励：-146.0，Epsilon：0.010023421049147612\n",
      "回合：16/400，奖励：-99.0，Epsilon：0.010016837948094095\n",
      "回合：17/400，奖励：-102.0，Epsilon：0.010011984751749595\n",
      "回合：18/400，奖励：-114.0，Epsilon：0.010008195909220538\n",
      "回合：19/400，奖励：-95.0，Epsilon：0.010005971322860786\n",
      "回合：20/400，奖励：-50.0，Epsilon：0.010005054615675078\n",
      "回合：21/400，奖励：-179.0，Epsilon：0.010002783294099886\n",
      "回合：22/400，奖励：-51.0，Epsilon：0.010002348167306314\n",
      "回合：23/400，奖励：-53.0，Epsilon：0.010001967902958245\n",
      "回合：24/400，奖励：-126.0，Epsilon：0.01000129300438042\n",
      "回合：25/400，奖励：-105.0，Epsilon：0.010000911164786836\n",
      "回合：26/400，奖励：-55.0，Epsilon：0.010000758536131584\n",
      "回合：27/400，奖励：-112.0，Epsilon：0.010000522203364875\n",
      "回合：28/400，奖励：-81.0，Epsilon：0.01000039863934062\n",
      "回合：29/400，奖励：-187.0，Epsilon：0.010000297294659517\n",
      "回合：30/400，奖励：-176.0，Epsilon：0.01000022999489198\n",
      "回合：31/400，奖励：-71.0，Epsilon：0.010000181524464132\n",
      "回合：32/400，奖励：-77.0，Epsilon：0.010000140432053464\n",
      "回合：33/400，奖励：-82.0，Epsilon：0.010000106846201706\n",
      "回合：34/400，奖励：-95.0，Epsilon：0.010000077845318887\n",
      "回合：35/400，奖励：-53.0，Epsilon：0.010000065238977184\n",
      "回合：36/400，奖励：-30.0，Epsilon：0.010000059030667672\n",
      "回合：37/400，奖励：-122.0，Epsilon：0.010000039306520976\n",
      "回合：38/400，奖励：-37.0，Epsilon：0.010000034745744355\n",
      "回合：39/400，奖励：-100.0，Epsilon：0.01000002489641374\n",
      "回合：40/400，奖励：-201.0，Epsilon：0.010000017720528442\n",
      "回合：41/400，奖励：-62.0，Epsilon：0.010000014411941012\n",
      "回合：42/400，奖励：-61.0，Epsilon：0.010000011760233133\n",
      "回合：43/400，奖励：-57.0，Epsilon：0.010000009725232207\n",
      "回合：44/400，奖励：-73.0，Epsilon：0.0100000076246806\n",
      "回合：45/400，奖励：-39.0，Epsilon：0.010000006695197199\n",
      "回合：46/400，奖励：-71.0，Epsilon：0.010000005284213373\n",
      "回合：47/400，奖励：-77.0，Epsilon：0.010000004088005098\n",
      "回合：48/400，奖励：-53.0，Epsilon：0.010000003425989836\n",
      "回合：49/400，奖励：-88.0，Epsilon：0.010000002555012459\n",
      "回合：50/400，奖励：-65.0，Epsilon：0.01000000205729175\n",
      "回合：51/400，奖励：-41.0，Epsilon：0.010000001794495218\n",
      "回合：52/400，奖励：-67.0，Epsilon：0.010000001435323749\n",
      "回合：53/400，奖励：-38.0，Epsilon：0.010000001264559407\n",
      "回合：54/400，奖励：-50.0，Epsilon：0.010000001070426428\n",
      "回合：55/400，奖励：-35.0，Epsilon：0.010000000952552966\n",
      "回合：56/400，奖励：-74.0，Epsilon：0.010000000744325952\n",
      "回合：57/400，奖励：-75.0，Epsilon：0.010000000579681634\n",
      "回合：58/400，奖励：-31.0，Epsilon：0.010000000522772152\n",
      "回合：59/400，奖励：-38.0，Epsilon：0.010000000460576537\n",
      "回合：60/400，奖励：-51.0，Epsilon：0.01000000038857222\n",
      "回合：61/400，奖励：-64.0，Epsilon：0.010000000313922366\n",
      "回合：62/400，奖励：-78.0，Epsilon：0.010000000242050338\n",
      "回合：63/400，奖励：-41.0，Epsilon：0.010000000211131054\n",
      "回合：64/400，奖励：-62.0，Epsilon：0.010000000171710922\n",
      "回合：65/400，奖励：-58.0，Epsilon：0.010000000141525377\n",
      "回合：66/400，奖励：-34.0，Epsilon：0.010000000126361357\n",
      "回合：67/400，奖励：-52.0，Epsilon：0.010000000106251867\n",
      "回合：68/400，奖励：-28.0，Epsilon：0.010000000096783744\n",
      "回合：69/400，奖励：-57.0，Epsilon：0.010000000080036202\n",
      "回合：70/400，奖励：-39.0，Epsilon：0.010000000070279423\n",
      "回合：71/400，奖励：-55.0，Epsilon：0.01000000005850696\n",
      "回合：72/400，奖励：-33.0，Epsilon：0.010000000052412531\n",
      "回合：73/400，奖励：-62.0，Epsilon：0.010000000042626625\n",
      "回合：74/400，奖励：-56.0，Epsilon：0.010000000035368174\n",
      "回合：75/400，奖励：-34.0，Epsilon：0.01000000003157858\n",
      "回合：76/400，奖励：-37.0，Epsilon：0.010000000027914485\n",
      "回合：77/400，奖励：-149.0，Epsilon：0.0100000000236291\n",
      "回合：78/400，奖励：-46.0，Epsilon：0.010000000020270076\n",
      "回合：79/400，奖励：-28.0，Epsilon：0.010000000018463805\n",
      "回合：80/400，奖励：-37.0，Epsilon：0.010000000016321432\n",
      "回合：81/400，奖励：-64.0，Epsilon：0.01000000001318587\n",
      "回合：82/400，奖励：-52.0，Epsilon：0.010000000011087433\n",
      "回合：83/400，奖励：-22.0，Epsilon：0.010000000010303453\n",
      "回合：84/400，奖励：-32.0，Epsilon：0.010000000009261004\n",
      "回合：85/400，奖励：-74.0，Epsilon：0.010000000007236559\n",
      "回合：86/400，奖励：-33.0，Epsilon：0.010000000006482756\n",
      "回合：87/400，奖励：-39.0，Epsilon：0.010000000005692478\n",
      "回合：88/400，奖励：-40.0，Epsilon：0.010000000004981906\n",
      "回合：89/400，奖励：-33.0，Epsilon：0.010000000004462961\n",
      "回合：90/400，奖励：-47.0，Epsilon：0.010000000003815783\n",
      "回合：91/400，奖励：-45.0，Epsilon：0.010000000003284274\n",
      "回合：92/400，奖励：-28.0，Epsilon：0.010000000002991612\n",
      "回合：93/400，奖励：-45.0，Epsilon：0.010000000002574904\n",
      "回合：94/400，奖励：-56.0，Epsilon：0.010000000002136451\n",
      "回合：95/400，奖励：-31.0，Epsilon：0.010000000001926707\n",
      "回合：96/400，奖励：-38.0，Epsilon：0.010000000001697481\n",
      "回合：97/400，奖励：-50.0，Epsilon：0.010000000001436887\n",
      "回合：98/400，奖励：-41.0，Epsilon：0.010000000001253341\n",
      "回合：99/400，奖励：-41.0，Epsilon：0.01000000000109324\n",
      "回合：100/400，奖励：-13.0，Epsilon：0.010000000001046878\n",
      "回合：101/400，奖励：-45.0，Epsilon：0.010000000000901057\n",
      "回合：102/400，奖励：-19.0，Epsilon：0.01000000000084576\n",
      "回合：103/400，奖励：-44.0，Epsilon：0.010000000000730383\n",
      "回合：104/400，奖励：-23.0，Epsilon：0.010000000000676478\n",
      "回合：105/400，奖励：-40.0，Epsilon：0.010000000000592037\n",
      "回合：106/400，奖励：-52.0，Epsilon：0.010000000000497817\n",
      "回合：107/400，奖励：-38.0，Epsilon：0.010000000000438592\n",
      "回合：108/400，奖励：-24.0，Epsilon：0.01000000000040487\n",
      "回合：109/400，奖励：-32.0，Epsilon：0.010000000000363909\n",
      "回合：110/400，奖励：-38.0，Epsilon：0.010000000000320614\n",
      "回合：111/400，奖励：-52.0，Epsilon：0.01000000000026959\n",
      "回合：112/400，奖励：-22.0，Epsilon：0.010000000000250527\n",
      "回合：113/400，奖励：-38.0，Epsilon：0.010000000000220721\n",
      "回合：114/400，奖励：-33.0，Epsilon：0.01000000000019773\n",
      "回合：115/400，奖励：-29.0，Epsilon：0.010000000000179511\n",
      "回合：116/400，奖励：-56.0，Epsilon：0.010000000000148944\n",
      "回合：117/400，奖励：-20.0，Epsilon：0.010000000000139338\n",
      "回合：118/400，奖励：-31.0，Epsilon：0.010000000000125658\n",
      "回合：119/400，奖励：-33.0，Epsilon：0.01000000000011257\n",
      "回合：120/400，奖励：-39.0，Epsilon：0.010000000000098846\n",
      "回合：121/400，奖励：-26.0，Epsilon：0.010000000000090641\n",
      "回合：122/400，奖励：-31.0，Epsilon：0.010000000000081742\n",
      "回合：123/400，奖励：-40.0，Epsilon：0.010000000000071538\n",
      "回合：124/400，奖励：-33.0，Epsilon：0.010000000000064086\n",
      "回合：125/400，奖励：-46.0，Epsilon：0.010000000000054977\n",
      "回合：126/400，奖励：-28.0，Epsilon：0.010000000000050078\n",
      "回合：127/400，奖励：-23.0，Epsilon：0.010000000000046382\n",
      "回合：128/400，奖励：-30.0，Epsilon：0.010000000000041968\n",
      "回合：129/400，奖励：-24.0，Epsilon：0.010000000000038742\n",
      "回合：130/400，奖励：-36.0，Epsilon：0.01000000000003436\n",
      "回合：131/400，奖励：-28.0，Epsilon：0.010000000000031298\n",
      "回合：132/400，奖励：-28.0，Epsilon：0.01000000000002851\n",
      "回合：133/400，奖励：-35.0，Epsilon：0.01000000000002537\n",
      "回合：134/400，奖励：-27.0，Epsilon：0.010000000000023187\n",
      "回合：135/400，奖励：-30.0，Epsilon：0.01000000000002098\n",
      "回合：136/400，奖励：-35.0，Epsilon：0.01000000000001867\n",
      "回合：137/400，奖励：-31.0，Epsilon：0.010000000000016837\n",
      "回合：138/400，奖励：-27.0，Epsilon：0.010000000000015387\n",
      "回合：139/400，奖励：-48.0，Epsilon：0.010000000000013113\n",
      "回合：140/400，奖励：-23.0，Epsilon：0.010000000000012145\n",
      "回合：141/400，奖励：-29.0，Epsilon：0.010000000000011026\n",
      "回合：142/400，奖励：-21.0，Epsilon：0.01000000000001028\n",
      "回合：143/400，奖励：-22.0，Epsilon：0.010000000000009553\n",
      "回合：144/400，奖励：-42.0，Epsilon：0.010000000000008306\n",
      "回合：145/400，奖励：-21.0，Epsilon：0.010000000000007744\n",
      "回合：146/400，奖励：-141.0，Epsilon：0.010000000000006733\n",
      "回合：147/400，奖励：-43.0，Epsilon：0.010000000000005834\n",
      "回合：148/400，奖励：-44.0，Epsilon：0.010000000000005038\n",
      "回合：149/400，奖励：-18.0，Epsilon：0.010000000000004745\n",
      "回合：150/400，奖励：-23.0，Epsilon：0.010000000000004394\n",
      "回合：151/400，奖励：-24.0，Epsilon：0.010000000000004056\n",
      "回合：152/400，奖励：-30.0，Epsilon：0.010000000000003671\n",
      "回合：153/400，奖励：-27.0，Epsilon：0.010000000000003355\n",
      "回合：154/400，奖励：-15.0，Epsilon：0.01000000000000319\n",
      "回合：155/400，奖励：-19.0，Epsilon：0.010000000000002994\n",
      "回合：156/400，奖励：-50.0，Epsilon：0.010000000000002535\n",
      "回合：157/400，奖励：-22.0，Epsilon：0.010000000000002356\n",
      "回合：158/400，奖励：-28.0，Epsilon：0.010000000000002146\n",
      "回合：159/400，奖励：-27.0，Epsilon：0.010000000000001962\n",
      "回合：160/400，奖励：-13.0，Epsilon：0.010000000000001879\n",
      "回合：161/400，奖励：-33.0，Epsilon：0.010000000000001683\n",
      "回合：162/400，奖励：-24.0，Epsilon：0.010000000000001553\n",
      "回合：163/400，奖励：-30.0，Epsilon：0.010000000000001405\n",
      "回合：164/400，奖励：-19.0，Epsilon：0.010000000000001319\n",
      "回合：165/400，奖励：-22.0，Epsilon：0.010000000000001227\n",
      "回合：166/400，奖励：-32.0，Epsilon：0.010000000000001102\n",
      "回合：167/400，奖励：-35.0，Epsilon：0.01000000000000098\n",
      "回合：168/400，奖励：-32.0，Epsilon：0.010000000000000881\n",
      "回合：169/400，奖励：-21.0，Epsilon：0.010000000000000822\n",
      "回合：170/400，奖励：-27.0，Epsilon：0.010000000000000751\n",
      "回合：171/400，奖励：-22.0，Epsilon：0.010000000000000698\n",
      "回合：172/400，奖励：-22.0，Epsilon：0.010000000000000649\n",
      "回合：173/400，奖励：-34.0，Epsilon：0.01000000000000058\n",
      "回合：174/400，奖励：-22.0，Epsilon：0.010000000000000538\n",
      "回合：175/400，奖励：-27.0，Epsilon：0.010000000000000491\n",
      "回合：176/400，奖励：-13.0，Epsilon：0.01000000000000047\n",
      "回合：177/400，奖励：-29.0，Epsilon：0.010000000000000427\n",
      "回合：178/400，奖励：-20.0，Epsilon：0.010000000000000401\n",
      "回合：179/400，奖励：-22.0，Epsilon：0.010000000000000371\n",
      "回合：180/400，奖励：-33.0，Epsilon：0.010000000000000333\n",
      "回合：181/400，奖励：-20.0，Epsilon：0.010000000000000312\n",
      "回合：182/400，奖励：-26.0，Epsilon：0.010000000000000286\n",
      "回合：183/400，奖励：-22.0，Epsilon：0.010000000000000266\n",
      "回合：184/400，奖励：-29.0，Epsilon：0.010000000000000241\n",
      "回合：185/400，奖励：-25.0，Epsilon：0.010000000000000222\n",
      "回合：186/400，奖励：-16.0，Epsilon：0.01000000000000021\n",
      "回合：187/400，奖励：-28.0，Epsilon：0.010000000000000191\n",
      "回合：188/400，奖励：-23.0，Epsilon：0.010000000000000177\n",
      "回合：189/400，奖励：-31.0，Epsilon：0.01000000000000016\n",
      "回合：190/400，奖励：-17.0，Epsilon：0.010000000000000151\n",
      "回合：191/400，奖励：-22.0，Epsilon：0.01000000000000014\n",
      "回合：192/400，奖励：-18.0，Epsilon：0.010000000000000132\n",
      "回合：193/400，奖励：-34.0，Epsilon：0.010000000000000118\n",
      "回合：194/400，奖励：-32.0，Epsilon：0.010000000000000106\n",
      "回合：195/400，奖励：-14.0，Epsilon：0.0100000000000001\n",
      "回合：196/400，奖励：-23.0，Epsilon：0.010000000000000094\n",
      "回合：197/400，奖励：-23.0，Epsilon：0.010000000000000087\n",
      "回合：198/400，奖励：-28.0，Epsilon：0.01000000000000008\n",
      "回合：199/400，奖励：-24.0，Epsilon：0.010000000000000073\n",
      "回合：200/400，奖励：-21.0，Epsilon：0.010000000000000068\n",
      "回合：201/400，奖励：-15.0，Epsilon：0.010000000000000064\n",
      "回合：202/400，奖励：-16.0，Epsilon：0.010000000000000061\n",
      "回合：203/400，奖励：-22.0，Epsilon：0.010000000000000057\n",
      "回合：204/400，奖励：-28.0，Epsilon：0.010000000000000052\n",
      "回合：205/400，奖励：-25.0，Epsilon：0.010000000000000049\n",
      "回合：206/400，奖励：-16.0，Epsilon：0.010000000000000045\n",
      "回合：207/400，奖励：-13.0，Epsilon：0.010000000000000044\n",
      "回合：208/400，奖励：-31.0，Epsilon：0.01000000000000004\n",
      "回合：209/400，奖励：-25.0，Epsilon：0.010000000000000037\n",
      "回合：210/400，奖励：-21.0，Epsilon：0.010000000000000033\n",
      "回合：211/400，奖励：-26.0，Epsilon：0.010000000000000031\n",
      "回合：212/400，奖励：-13.0，Epsilon：0.01000000000000003\n",
      "回合：213/400，奖励：-15.0，Epsilon：0.010000000000000028\n",
      "回合：214/400，奖励：-23.0，Epsilon：0.010000000000000026\n",
      "回合：215/400，奖励：-23.0，Epsilon：0.010000000000000024\n",
      "回合：216/400，奖励：-13.0，Epsilon：0.010000000000000023\n",
      "回合：217/400，奖励：-21.0，Epsilon：0.010000000000000021\n",
      "回合：218/400，奖励：-28.0，Epsilon：0.01000000000000002\n",
      "回合：219/400，奖励：-24.0，Epsilon：0.010000000000000018\n",
      "回合：220/400，奖励：-20.0，Epsilon：0.010000000000000018\n",
      "回合：221/400，奖励：-13.0，Epsilon：0.010000000000000016\n",
      "回合：222/400，奖励：-15.0，Epsilon：0.010000000000000016\n",
      "回合：223/400，奖励：-27.0，Epsilon：0.010000000000000014\n",
      "回合：224/400，奖励：-18.0，Epsilon：0.010000000000000014\n",
      "回合：225/400，奖励：-20.0，Epsilon：0.010000000000000012\n",
      "回合：226/400，奖励：-27.0，Epsilon：0.010000000000000012\n",
      "回合：227/400，奖励：-18.0，Epsilon：0.01000000000000001\n",
      "回合：228/400，奖励：-15.0，Epsilon：0.01000000000000001\n",
      "回合：229/400，奖励：-19.0，Epsilon：0.010000000000000009\n",
      "回合：230/400，奖励：-20.0，Epsilon：0.010000000000000009\n",
      "回合：231/400，奖励：-13.0，Epsilon：0.010000000000000009\n",
      "回合：232/400，奖励：-28.0，Epsilon：0.010000000000000007\n",
      "回合：233/400，奖励：-38.0，Epsilon：0.010000000000000007\n",
      "回合：234/400，奖励：-17.0，Epsilon：0.010000000000000007\n",
      "回合：235/400，奖励：-22.0，Epsilon：0.010000000000000005\n",
      "回合：236/400，奖励：-13.0，Epsilon：0.010000000000000005\n",
      "回合：237/400，奖励：-20.0，Epsilon：0.010000000000000005\n",
      "回合：238/400，奖励：-18.0，Epsilon：0.010000000000000005\n",
      "回合：239/400，奖励：-14.0，Epsilon：0.010000000000000005\n",
      "回合：240/400，奖励：-13.0，Epsilon：0.010000000000000005\n",
      "回合：241/400，奖励：-28.0，Epsilon：0.010000000000000004\n",
      "回合：242/400，奖励：-13.0，Epsilon：0.010000000000000004\n",
      "回合：243/400，奖励：-23.0，Epsilon：0.010000000000000004\n",
      "回合：244/400，奖励：-17.0，Epsilon：0.010000000000000004\n",
      "回合：245/400，奖励：-14.0，Epsilon：0.010000000000000004\n",
      "回合：246/400，奖励：-22.0，Epsilon：0.010000000000000004\n",
      "回合：247/400，奖励：-15.0，Epsilon：0.010000000000000004\n",
      "回合：248/400，奖励：-19.0，Epsilon：0.010000000000000004\n",
      "回合：249/400，奖励：-17.0，Epsilon：0.010000000000000004\n",
      "回合：250/400，奖励：-27.0，Epsilon：0.010000000000000002\n",
      "回合：251/400，奖励：-21.0，Epsilon：0.010000000000000002\n",
      "回合：252/400，奖励：-23.0，Epsilon：0.010000000000000002\n",
      "回合：253/400，奖励：-15.0，Epsilon：0.010000000000000002\n",
      "回合：254/400，奖励：-15.0，Epsilon：0.010000000000000002\n",
      "回合：255/400，奖励：-13.0，Epsilon：0.010000000000000002\n",
      "回合：256/400，奖励：-15.0，Epsilon：0.010000000000000002\n",
      "回合：257/400，奖励：-13.0，Epsilon：0.010000000000000002\n",
      "回合：258/400，奖励：-28.0，Epsilon：0.010000000000000002\n",
      "回合：259/400，奖励：-13.0，Epsilon：0.010000000000000002\n",
      "回合：260/400，奖励：-13.0，Epsilon：0.010000000000000002\n",
      "回合：261/400，奖励：-23.0，Epsilon：0.010000000000000002\n",
      "回合：262/400，奖励：-13.0，Epsilon：0.010000000000000002\n",
      "回合：263/400，奖励：-24.0，Epsilon：0.010000000000000002\n",
      "回合：264/400，奖励：-17.0，Epsilon：0.010000000000000002\n",
      "回合：265/400，奖励：-19.0，Epsilon：0.010000000000000002\n",
      "回合：266/400，奖励：-13.0，Epsilon：0.010000000000000002\n",
      "回合：267/400，奖励：-25.0，Epsilon：0.010000000000000002\n",
      "回合：268/400，奖励：-15.0，Epsilon：0.01\n",
      "回合：269/400，奖励：-15.0，Epsilon：0.01\n",
      "回合：270/400，奖励：-21.0，Epsilon：0.01\n",
      "回合：271/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：272/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：273/400，奖励：-22.0，Epsilon：0.01\n",
      "回合：274/400，奖励：-15.0，Epsilon：0.01\n",
      "回合：275/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：276/400，奖励：-19.0，Epsilon：0.01\n",
      "回合：277/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：278/400，奖励：-18.0，Epsilon：0.01\n",
      "回合：279/400，奖励：-14.0，Epsilon：0.01\n",
      "回合：280/400，奖励：-126.0，Epsilon：0.01\n",
      "回合：281/400，奖励：-15.0，Epsilon：0.01\n",
      "回合：282/400，奖励：-19.0，Epsilon：0.01\n",
      "回合：283/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：284/400，奖励：-25.0，Epsilon：0.01\n",
      "回合：285/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：286/400，奖励：-119.0，Epsilon：0.01\n",
      "回合：287/400，奖励：-15.0，Epsilon：0.01\n",
      "回合：288/400，奖励：-15.0，Epsilon：0.01\n",
      "回合：289/400，奖励：-14.0，Epsilon：0.01\n",
      "回合：290/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：291/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：292/400，奖励：-15.0，Epsilon：0.01\n",
      "回合：293/400，奖励：-33.0，Epsilon：0.01\n",
      "回合：294/400，奖励：-19.0，Epsilon：0.01\n",
      "回合：295/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：296/400，奖励：-15.0，Epsilon：0.01\n",
      "回合：297/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：298/400，奖励：-132.0，Epsilon：0.01\n",
      "回合：299/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：300/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：301/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：302/400，奖励：-14.0，Epsilon：0.01\n",
      "回合：303/400，奖励：-15.0，Epsilon：0.01\n",
      "回合：304/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：305/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：306/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：307/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：308/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：309/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：310/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：311/400，奖励：-15.0，Epsilon：0.01\n",
      "回合：312/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：313/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：314/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：315/400，奖励：-15.0，Epsilon：0.01\n",
      "回合：316/400，奖励：-14.0，Epsilon：0.01\n",
      "回合：317/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：318/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：319/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：320/400，奖励：-21.0，Epsilon：0.01\n",
      "回合：321/400，奖励：-19.0，Epsilon：0.01\n",
      "回合：322/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：323/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：324/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：325/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：326/400，奖励：-14.0，Epsilon：0.01\n",
      "回合：327/400，奖励：-15.0，Epsilon：0.01\n",
      "回合：328/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：329/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：330/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：331/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：332/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：333/400，奖励：-14.0，Epsilon：0.01\n",
      "回合：334/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：335/400，奖励：-113.0，Epsilon：0.01\n",
      "回合：336/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：337/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：338/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：339/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：340/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：341/400，奖励：-15.0，Epsilon：0.01\n",
      "回合：342/400，奖励：-23.0，Epsilon：0.01\n",
      "回合：343/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：344/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：345/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：346/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：347/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：348/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：349/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：350/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：351/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：352/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：353/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：354/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：355/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：356/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：357/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：358/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：359/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：360/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：361/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：362/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：363/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：364/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：365/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：366/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：367/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：368/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：369/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：370/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：371/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：372/400，奖励：-14.0，Epsilon：0.01\n",
      "回合：373/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：374/400，奖励：-15.0，Epsilon：0.01\n",
      "回合：375/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：376/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：377/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：378/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：379/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：380/400，奖励：-117.0，Epsilon：0.01\n",
      "回合：381/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：382/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：383/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：384/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：385/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：386/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：387/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：388/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：389/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：390/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：391/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：392/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：393/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：394/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：395/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：396/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：397/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：398/400，奖励：-15.0，Epsilon：0.01\n",
      "回合：399/400，奖励：-13.0，Epsilon：0.01\n",
      "回合：400/400，奖励：-13.0，Epsilon：0.01\n",
      "完成训练！\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEcCAYAAAA2g5hwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA/CElEQVR4nO3deZgcVbn48W9Vd8++ZjIzySSZ7DkQQiAhYd9BEdn1KvsiXlkEkStcwSsKKCCKKILyE0UB2QTEHdkVkS0hYYdwAtn3zExm37u7fn+c6pmeTvV0z9ILmffzPHnSU1Vd9XZ1db11ljplOY6DEEIIMRg70wEIIYTIfpIshBBCJCTJQgghREKSLIQQQiQkyUIIIURCkiyEEEIk5M90AOmilFoL/LfW+rk0b/cQ4G6ttUrndsXoUErlA48ChwLPaK2/MErrPQ9zPB48Gusb4rafBH6vtb4vBeu+GLgOKASmaq0bRnsbMdu7DpiltT5LKVULfACUaq1DSqlq4DFgAfAr4Ergt8DJwEda631HMY57gY1a62uUUocDD2itJ3ss94k9H4yZZJEpWuv/AJ+4A0P0+S+gGqjQWge9FlBKzQVuBg7DlNZfB/5Pa/1a2qIcAq31salYr1IqAPwE2F9r/fYorvcM4BvAbkAr8BZwo9b6pejltNbrgaKoSRcA9UCJ1tpxT9SfAiZrrduVUhr4rtb6EXc7BwEvAafFTHsKKI/3/Q9FJs8HSqn/Aa4CCoA/ABdrrbuTfb9UQ42QUsqX6RhGalf4DCk0FVg5SKKYCbwMvAtMB2qAPwPPKqVG7co1WUqpTF4AVgN5wPtDfaNSylJK7XQ+Ukp9A7gNuMldfy1wJ3BSEqudCnygtXai/l6rtW53/34RU2KMOBT40GPaq6ORKDJJKXUMcDVwFGY/zACuH8o6xmTJwj0ovwl8BSgDngcu0lrvcOc/BhwC5ANvYzLw++68e4FOzA4/DDhJKXU38HPgHHf6U8C5Wuuu2CKpWx3muaw7/5vA/wAO8F3g18BsrfXHHp9jHHArcIwb67+11id7VXEopZzIejw+w/fcq45JWuuQu/wpwPVa6/mJ9pdHXF/BXMGMw1ypXaS13hwVx8XAFUAl8CBwadQPOno9Pnc9XwaqgJXAyVrrDe56vg5cDpQA9wBXaa3D0VUT7nqmAWuAgNePXim1O/D/gL2BTcC3tNZ/VUpdD3wLsJRSJwNf11r/Jubt12FOJt+Omna7u84fAkd47SOPGHYD7gD2AeqA72itH3XnHQfcAMwEmoHfaK2vi/ls/w1cC6xVSv3W/fs1d981AV/VWj/pvucFzDF5d+RYGWTZ6cB9mKqcJYDGVPOcFRP/HOBN988mpdRSrfWRSqkDgZ8BczDf39e11q9ExfEycDiwENgT+DhqnaXA94Avaa3/GLW5v7n/YvdhZF8EgLuBMwFHKXU55nj7GRBQSrVhfjcvYo7riEMw39kVMdNedNcf97wwGKXUZcBFwKeBWaTgfKCU2g/4C3F+w8C5mOMmch77Pua3d3Wi+CPGasnia5h6y8MwV4KNwC+i5j8JzMacoN7A7NRoZwA3AsWYkyHAF4HPYK4u5wPnDbJ9z2WVUp/BFLePxhxUhyf4HPdjipR7uLH+NMHy8T7Dz4B24MiY+Q+5rxPtrz5KqSOBH2A+40RgHfD7mMWOBxZjPvsXMcnOyzeA04HPYhLC+UBH1PxTgEWYE81J7vwhcatO/gY8g9mHXwMeVEoprfW1mCvaR7TWRR6JAky1xmMe0x8FDlFK5SURQyHwLGZ/VwGnAXe61VtgvptzMIn6OOBiN3lFOwzYnf59uR/mxD4e+BHwG6WUFSeEwZZ9CFgKVGAS49leK9Bar8QchwBlbqIYBzwB3O6+/yfAE0qpiqi3no2pLirGHCvRDsCUVP4UJ+64tNbnYX63P3K/u7swJ+xX3b+vxSSBPZRS49wLokXAI0BZ1LSD3OUg8XlhJ0qp72J+34dprTfGWWzE5wOt9RIG/w3vgUlwEW8D1THfxaDGZMkCc9BcGvny3CvR9Uqps7XWQa31byMLuvMalVKlWutmd/JftNYvu6+7lFIAt0ddPf8Nc5UaT7xlvwjcE5X9r8NcHe1EKTUROBZTl97oTv53Up/e+zM8jDkxP6uUKsacoK905w+6v2LWeybwW631G+6y38Lsv2la67XuMjdrrZswV6D/cj//Ux4x/jfwTa21dv+OrQf/oVu62aGUus2N/+4h7AOA/TH13DdrrcPAP5VSf3fXdV0S7x8PbPGYvgXwYUpXmxOs43hM9cg97t9vKqUeB76AuTJ8IWrZd9zv6jBMdVfEdZHqFfd4XKe1/rX7932YqptqYKvH9j2XVUrlYJL6UVrrHuAlpdRfE3yWaMdhGpLvd/9+2L3KPgG415127yBX5xVAfaqqgLTW65RS6zGlhfVurJ1KqZejpuVgSlQkcV6IZimlfgLsCxwRZ5mIEZ8PXIP9hoswpdKIyOtiIKlOCGM1WUwF/qSUCkdNC2F+IFsxV9xfwFSTRJYZT/8O3uCxzugfYQfmCjyeeMvWAMui5nltJ2IKsCMqUQxV7LofAl5RpjfL54A3tNaRK724+wtTbROtBnPVBYDWuk0p1QBMAta6k2M/f3SjZLQpwKokP8M6Bt/n8dQAG9xEEb2uSUm+vx5Tgoo1EVN10OA2rD4ZWbfWeo+YZacC+ymlmqKm+TElx0gVw83APMzJK5edSzOx32ffPtZad7gJJN5+jrfseMwxFl2a24D5XpJRw86lhdh9O9gx3gCMV0r5U9hmEGm3WA/8x532UtS0pVrrbrdKNNF5IVoZpsR0aoJEAcM4H6j+nl8AaK2LGPw33IYpnUdEXrcmiK3PWE0WG4Dzo66s+yilzsZUaRyNObmVYqpdoovwqRqqdwsQ3d1usB/lBmCcUqrMvUqP1o6pngJAKTXB4/0DPoPW+gOl1DpMaSW6+BrZluf+8rAZc/KLbLsQc4UYm1SSsQFTT/9enPlT6G9MraX/Cn7A5we8Pn90vFOUUnZUwqjF1K8n4znMCeSemOlfBF5ze5v8h/gnajCf899a60/Fmf8Qpl77WLcd7DbMSSpaKo7JLZhjrCAqYSSbKCDmWHDVMrAUOVjcrwLdmCrQPwxhu0PxInAhJolFvsP/YOr419FfBXUGic8L0RqBs4BHlVKnJPnbiRX3fODR8yvRb/h9YC9M9Sju6216CF2bx2qy+CVwo1LqXLcoWgkcqLX+C6ZY1o25qinA1Fmny6PAb5VS92MO1O/EW1BrvUWZ/vJ3KqUuwVw5HKC1fhFTXbOHUmpvTO+O65Lc/kOYRuP9GVjcHWx/xXoYU93wELACs/+WRFVBDcXdwPeVUh9gGj73BDZFHeD/q5RagvnRfB1TJw6ma+VV7tVXM6aROp4lmKu5byqlbsXUUZ+AqX5JxvXA60qpGzGNpr2YOucvYaqXkvF34Gb3QiXSvrM30Ka1XoE5Jne4iWJfzIngmSTXPWzud70MuE4pdQ2m8f0EPBqX4/gHcIcyXV8fBT4PzMV83mS23+zW+f9CKRXEfOZezAn7CK31NwddQXJexBzfUzHVntDfs20GcJc7bcjnBa31C0qpM4E/KqVO0FovHWJsSZ8PosT7Df8OuFcp9SAmiV9Df1VgUsZqA/fPgL8CzyilWjE9QfZz5/0O88VswhTz0tZX3u2BcjvwL8zJMbLteH2hz8b8eD4EtmN6BkUaG7+Huer9iP5G+EQideH/1FrXR00fbH/FfobnMAf145gro5mYBtvh+AnmB/MM0AL8BtMTJeIvwHJMcnjCnY/W+llMQ+U77vy4Jye3Lv4EzNVYPaa+/hyt9YfJBKi1/gg4GHOlthbTm+j7wCk6yRtAtdatmJ4yp2F+yFsxvXJy3UW+iumx1orpEfOo13pS5ExMQ3MDpkfWI8Q/Hgdwk/rxmN5FDZieR8fHHFuJ1nErppH3GkwvsQ3ApQxsrxk297dSB2yNlNDdEuZSTFXNK+6iwzovuMfi+cDflFILhxjbUM8HEOc3rLV+CtN54V+Y6rV1mN5zSbPk4UfZS5nul+8BuSmss/1EUlFdgTMdSzSl1GTMj/raOL2nPtGUUo8AH7q9iUQaZfp8MFarobKW2zf6H5ii7g+Bv0mi+OTQWm9USh2Luf+mSGvdlumYRkIptRjYgbl/4dOYevubMxrUGJJN5wNJFtnnQkxdYgjTFfarGY1GDJnW+l1MvfeuYALwR0wnhY2YG9HeHPwtYhRlzflAqqGEEEIkNFYbuIUQQgzBrloNlYvp+rgFU3wTQgiRmA9zQ+nrxPS62lWTxWL678YUQggxNIcQ0+V+V00WWwAaG9sJh4feJlNRUURDQ/Z1YpG4hi5bY5O4hkbiGprhxmXbFuXlheAx3tmumixCAOGwM6xkEXlvNpK4hi5bY5O4hkbiGpoRxrVT9b00cAshhEhIkoUQQoiEsroaSpmnb92HuSGoATNmz0eZjUoIIcaebC9Z/BL4hdZ6DubJbHclWF4IIUQKZG2yUEpVYR6X+bA76WFgoTs8thBCiDTK5mqoKZhnF4QAtNYhpdRmd3pdRiMTn3iO42BZ8Z5bM/LlR9NIth373uGsy3EcHMC2rEHfH3Yc7GFuK96ysdPDHsMT2ZbVNz2y/eiYvdYTmR8tMjd6euT9XtsFt8flCIdMio5/sGmJeMU/mrI5WYxYRcVgDycbXGVl8ShGMnpSGVdjSxclRbn47OR+4B1dvVx48/OccthMaiqLmDdzPAW5fmzbIhR2+tbT0NxJYX6AvBxzuC1bsY3dp42jMD+w0zpDYYeOrl4K8gLYFnFPNqGwQ29viA3bW7Eti7LiXBwHxpfl896qejZsb+PAPScSCjs88Jxp5jrvuLk4wPqtLdz5h3f49vn7Ul1ewHOvr2fezPFMm1hCd2+INZubKS/Oo6W9m8lVxaze1MzPHnmT4w6azvjSfObOGMeHaxuZXlNCXo6fVZuaeOWdLZz5md1obe/hyZc/piw3zN671fDR1i6Wv7eRs46eQW1VPss/3Mq/lq7Fbzscf+BUSvN9PPzke3S2t3HRqftR39LDa+9s5MM19ey7exUdnV1s2tbMqUfNxnJCPPXyappbO9l7VgU2ISaOy2fNpkaaWzoYX5LDus1N7Lf3NNpCAd5d10ZjUxuzawrZVt9CV1c31WV5dHR0UV7op6OzG7/lUF7opzDfz6Sa8XzYEmLN+nqCXe3k0ouPEHmFRTQ1t1NZEqCrq4fS0kK6e8O0tndTmOsjHA4TDIbo7e0lx2eR43PwWQ7hcJiADT0FlXxUdTSrN7cS7unACvYwc/ZUPtrYTFtHD+1dQQiHOHSviYwv8fPKkg8pooOy2pksX2PulcoN+MjLsehpaaLU7qDU7qDM7qDQ6qYkJ4wV6iaHIEWBED7COKEgFg5lhX7CwSA9Pb34LBOX33JwwmEswtg42Jb5v9lfwZOhA+lsbaHE7qTU6qSiEOjthlAPAYLkWCFyrCABK0gO5rWfcN86zL8wthX1GgfbcrCiTuOR19FHd2RaNwFeL/k0b2wKUWJ1UGJ3Umh1k2f1kmv1kuf+y7WC+KwQPhz8hPDbDj7C9OAnfN71TJyS7JOBk5O1Awm61VArgQq3VOHDNHLP1lonKllMA9Y0NLQNq69xZWUxdXVJP5p22JZ9uJ2q8nxqq5NLANFxdfeEyM3x8ebKOqbXlFBWlDtg2cbWbhpaulj6wTZOO3o24bDD1oYOJlcVUd/cSWmhWf7Rf37M4t2rqBlfyGU/+w/TJxZz6efmA1CQ66e+pYvyolwefm4lJxw0jcbWbgrzAkyuKuK1D7byq79+MGC7n148heMPnMb197zO9JoSzvvMblx624tMrizkiAU11DV28NTrG6kucDjvmFk8vWQdUyrzOWiPKnoI8PCzmsYdTRRY3UwqsRhfAD4rDKEg5UV+yoryWLO1jZADdTvaKLS6ybd66cHH+vAE/ufyM7jyzldobO3GRwg1Lgyt2yi322kN55Fn9VJkdzHebsNnhbFwKHDXUV1RRHNHkJaOID4rjI8wAdvBckKU2h0ECLo/egb88KNZOASs/sd5Bx0bvxX2XDZbhB3zaXxW/2cKY9Fr5dIdgjyrlxA+ehwfDhZ+twt+5PRnWTbBMPj9PrqCYPt8hLAJhc2V9yR/I42hAgrsHnItM7r2Bz01rA1WMr+ii7JgPQW9O7Bj9umOUCFLemYyv7yT8lAdeb0t2Oy8L4OWn7CdQ8jOpdvxE7b9WLaP3hA0dQSxbZuSkgJz6nagN2zh9/uxfT5zercsmtt7mN2zwvO7Cjo2jh0g7AsQsgKE7QAhO4ewFYBADr1hG8cyqQHL6nvtWJb7v42DBZbt7rcIkyo6ekLkBnz4LIvuYJiihg+o8TftFEfIChC0cwj5cgnZuW4MPsKWD8fy0RM22/MVlrDP6RfS2jmkwwAwN+W5F9nTMQ/z6o82W5MFgFLqBeBurfUDSqmzgC9rrY9I4q3TyPJk0dLRw+W3m7vp83N9/ODCAygpyAFMkffef3zIwfMnMmdKGQDbGzt4f30zem0DK9Y10trRy1VnLOCHD5nRom+5+EDyc/08vXQ9h+1dw5V3vtK3rdqqIna0dtPW2ctln5/P7Y+/Q0Gun2P3r+Xxf68G4POHzeh7nZfjo6snRH6Ohb+3g5kVNm1NjcyszmPtNnNVN3/GOFasbWCSXU+R3U2pe5WXYwXx+2yCIXOydbAIY1FodeO3wvQ4PgByrNEbsqvH8ZFjhQg5FptK9mJ7fTNzCpooDjXFPal3kUdX2AYsfPlFNPf46O4xSSLXb5GXl0tXEJo7QwQCAfJKy2nosBhXksdHm1ooyAswviyfprYeivMD5AZs6pq7aO8KMmfGBEK+PJZ/sIkiu4f5u02mvddma1M3tRNLmVZTTkFRPv96YzNt3SEW7jaRZR+3sL2ukXnTyugJWSyeO4GN9Z04lk1FeSHPvL6JynGFLJ47kbBj88+3NrPv3Bpe/6ieWZPHsceMSrqCDlsbu7jtoaXkWz2ce/RUZk6pwPHlEAgEWLmplbdWN7Hv3AmMH1eEZfsIhi3KS/JZ+sFW1m9pYt/dxlE1rojcvHwsy+Ld1Q2883EDpx41i9WbW8gN+AiGw8ysKWXNlhZaO3qYN6OCto5eSgpz2NLQTmVZPj7bwrIsXnx7M6v++SdmB7YxsXYK+eXjee2NVXwq/z3CDvhKq/CVT4KyGp5avo1AToBPHzqP59/YxOLmZwgQwiqtpmjyTHpzyrEKzT/b/d/KK8ayfXGPjeb2Hory/fjswZtn//nGRl54/lUm+Ro5/MC5zJozFaugjC4nwJYd3cycVOr5vtE+VzS39/Dtnz/P3MAmaqdUc+wR87EKShN+ztGK65OcLHbDdJ0txzwA/RyttU7irdNIc7Lo6Q1x55/fY/bkUlZvbuGIBZOYOqGYzfXtqNrynZZ/eul6Hvln/0PeLjllHvuoKjq6gqzc0MTtj79DbsDHJafMIxR2WLpiO6++v3XAOvaeNZ63PjZPTtxjWjm7TxvHH15YRVF+gK7OLip9LUz27aDYLcaOs9vpcvyU2p0ErBB/61hIdV4X1eHtBLGp8bcwya6n1cmj0m6lyE789MxucmgO5bIjXESPlUdb0FQ11VYXUVpcwHsfb8OyHCoqq8gryOXjNdsBOOLgeazY2EZxYR6ObbNMNxDsamdKdSmH7zcbJ5DPii09zJpeRV1LiJLifIKORUtbN2+trOPldzdx1md2Z+G8abz03nZ+/9TbnF/0byb6mgg6NuXT5pBTWUvppKm0WyX8/rUG9EcbOOmouSycNx3Hn09Hd5AityqsNxjiw/VNlBflMrnKVF+Gww7bGjuYWFE44DO3dvRQlB9IWMe+bmsrpUU5O5X6YOdjbDTbRFasa6S+qZND9qoZ8ntH++T35so67vijebTHZZ+fz96zx/OTR99i3ZpNlJQW8/2LDu1btrM7SMBv4/fZ3P+05pU31xLG4udXHkXNxLKUXsAt+3A7d/75PQC+ffY+cZNDrNHeX6FwmAt+9AIOppR+2lGzh7WeVCSLrG6zcJ+D7Pms50zr6gkSDDl9J5uVG5p4Z1UD76xqAKCjK0hXT4h121r55RWHkRMYeFUQWS6ipaOXWx95iw/XNRJyE1x3b4ifPPo2ADn+/iujfXevYtmKrbz98Xb2zNvGibWNbNu8lVBjHpcVt5Br9VKd10Ig6uo95FjsCBeRa/XSZRdQFG7jitJ/9M3zWQ5dvmLWh8Zj9XYSrNmLnMmTaA3lsqM7wKQp1dz/7GoIB/n8Zxfx6gfbOXLhZMaVVTAuN4/a7h7A4r3VDSxZsY0jj9udgN/Hu0vX09zRw6cOn4XjOPz4h/9izxkVnLjPXuy7T//nn3skbN3RQUVJHgH3s+5Va+YVVvUvVwVMmVrDhMkTWTBvApZtccheNdQ1d/KLV8xJ+ZunL6B2qknQxZXFdNW1ctzRteSUjmf+/BlYfhsL+r47gIDfx54zKgZ8J7Zt7ZQoAIrdEqCX6BP+1AnJty+NZuP57lPLYerOFyiZUFTQv4+L3dclBTm0OvlUFRYMWDY/t/90VJgfoJsAOQGbgD/5K+rhKinM8Xydbj7bpjA/QFtnb0bj8JLVySKbffc3S6lv7uK3Vx8JmBNdtIaWLuqbuwBYtamZ2VPKeOxfq3hnVT1fPHIWa7a0DFj+o41NvL9mR9zt9QTDVJfnMtdazeftJZw5LqqA1ZKPnVeIP7iDFn8x/oIKrMkLyJs0g66SqTh5xRSXlLB+TSPvb21h/szx3Pa7Z1GBLUzfYw/yauawbGU9XzlhLm89s5KX393Kjw8/kNySPHKB8e5mLvjyHvT2hsnN8XHi5Nq+zVeOK6CuziSmBXMqWTCnv3fzMfv2L2dZFnd+49C4VQITxhV4To+VG/Bx8PyJA6YV5fWflCaO3/kEX1aUyxeOmJXU+sXoiU7IkcQRnTQSva/IoxNEKpRmSbKIbL+ts7dvP2ULSRbDFEkEkeqDdVsHFvkaW/urcG75/VsctXAy67a1sq2xkzseN8XyCeMK+pLMms0meVx88jwcxyHgt3lnVQMrPlzLuFAd+xVvZZ/AKpzeLsLt1TTWHsmqLa1Uz5jN7gcfQXdbkBt+t5zTjpzF/ntM6Nt29OE2f2YF82dW0NrRw7ZwGdu6y5hWOZv95tWw3zxTZXHkwsmUF+cxriRvp89sWxa5OSO7yov0iBptA65g03SCEYlFl8KK883rSJIYrIRWlO93/0/PdxlJELk5PnIDqS/JDCZy/A62fzJBksUQBENhHMfBjupaes8/PmT25FI+2tQ8YNlQTFvJ26vqsW2LfVQlKzc00drRy4I543nytfUAbGs0XRcK8vzMnVpOcNUSVOhFQgUrAAfHsinc/TBCE/fCV7sXRZZFbdT6x5cG+OmlByVVnVGYF101MPCAnD6xhOkTS5LZHVkl+qRiJ9n1V6ReQZ4fyzIXGvm55iQcSewlhfETQZGbWNKVLPJyfAT8NqVZcIIuLhyYVLOFJIshuPa3S9ne2Mn3vrxv37SX3t3CS+8OHPq9tCiH5raeAdNqxhfy0UbTgPr9L+/Hu6sbKC3M6UsWEcU9dXQ+fR+h9W9jlVSRs/BEfDW74auopWryhEEbrZKt944+mQ72g/0k8bpnQ2SebVkU5Qew3d5RkGzJIr3VUJZlUVKQk/EqKICSvmq67DqmJVkkaUtDO1saTJXRfU8N7JA1fWIxXT0hxhXn8v7aRqZWF/NOm2nAPnqfyTy3fCMt7T10docoLjQH5EF7TkSvbwQc5vi3EsKiytdC2YsPEwrkkLvfqQTmH4NlpXZElmz4cYyGdJ1UxNAVF+QMuLO7JIkr50g1VDovAmZPKR3QdpEp40ry8NlWXwkjW0iySMIbK+v4udv9D0zPp2jXnLOIUNjhnn+sAKC2uph3VjVw9D6TOeNTc2hu7+HtVaaLa/TVwvRxNl+b9BazOvvXTfVcCo+6ELsgua57I5VtRd3hkmSRvWoqCgaUZqdOKObMT81h79nj476nr2SRl77v9YIT9kjbtgZzxIJJ7D61PONtJ7EkWcTRGwzh99ls3dHBkg+27TT/jssP4Wu3mcd8W5aF32f1tQWUF+Vw5zcO7fuySwpy6Ok1d4ZGit7htgZ6/ngds7paaa7ZnzdWtdAULuCMz16E7Uvf15I3wgbrbBHpdjmxIrkeVSJ9Ljhx4EnYtiyO2mfyoO8pyAvwhcNnDuhZN1bk5/qzst1QkoWHru4gF/743xy7Xy1PLlnvuUxhXoAjF04acPNOpMhckBcY0OunOKpdoKQgBycUpPO5O3FCveR/9kq2M5k/v/cGAOekKVEsnFPJGyvrMjY43mizLYtvnbWQ6iS734r08fuGV5V67P5TRzkSMRKSLDz0BE0pIF6i2HuWKT6f9Wk1YHpBXqSedeBuHdB9MNeh85nbCW9fRd7Rl+CfPI+SpmEM4jJCl5wyL2WjU2bK7MllmQ5BiF2WJAsPg/W83Hf3Kr583FzPeZH61cKYetZIO4WFQ+GyewlteJfcg88lMGOxOz/97QaWZbFrlCmEEOmQtQ8/yqTBrrijh6OItefMCo7dr5YpVQOHRi8rNsNQnD55Pc6Gt8g94HRy5vaPhzjSG92EECLVpGThYbCxFeMlCjA9OLyGlJhebnPdId2Ur3gZ/9QFBOZ9ajTCFEKItJFk4WGwkXhjBwRMJNzeSMfj36W8qxW7rIbcw873bFS+4tS9pfunECJrSbJIIPbxhoOVLGI5jkPXf+7F6e0m/4Rv4ZswJ27voz2mjxtxrEIIkSrSZuEhumBRVZ4/YF7OEJJFcPVSQuvfJnfx5/FPVLtMN1UhxNgjycJD9KPcd04WyVVDOY5Dz5t/wy6fLG0UQohPPEkWXqJKFpVlA5NFstVQoY3vEd6xkZz5x2AleKSjEEJkOzmLeYhu3o4daC8nkHiXOU6Y7tcfxyqqwD9r/1GOTggh0k+ShYfo3lB5Ob4B49gk84jH4JplhOvXkrvoc1g+6eEkhPjkk2ThIbqBOzfg48xPzSHSNp1MA3fPm3/HLp+Ef9YBKYpQCCHSS5KFh0iyKMoPcID7iNLI4BiJ7rMIt2wn3LCegDpU2iqEELsMOZt5iPSG+q/DZ/Y1aCdbsgiuWQ6Af/rC1AUohBBpJsnCi1uy8LorIlFvqN61y7ErarGLx944/EKIXZckCw99TRZR2aKvZDFINVS4o4nwtlX4p++TstiEECITJFl4iPSGGjiIt3kdGORBLr0rXwIc/NMXpTA6IYRIPxkbahDRo3NcdeYCXnlva9z7LJxQL73vPoNv8jx85ZPSFKEQQqSHJAsPXoPOzqwpZWZN6c4zXKFtH+N0thCIek6FEELsKqQaykOkN9RQxv0Lbf4QLAv/xN1SFJUQQmSOJAsvfb2hks8Woc0rsMdPw8otTFFQQgiROZIsPHj1hhp0+WA3oe2r8EmpQgixi5Jk4aG/N1RyQls/gnAIf83uqQtKCCEySJKFh74G7iSzhWmvsPFNmJ2ymIQQIpMkWQwi2TaL4OYV2JXTsXLyEy8shBCfQJIsPPRVQyWRK5yeTsJ1a6QKSgixS0vpfRZKqXuBo4F6d9JjWusb3XnVwP3ANKATuEBrvSTRvHTwuM0irtDWj8AJ45NkIYTYhaWjZHGz1npv99+NUdN/ALyotZ4DXAI8oJSykpiXepGus0kULULbV4Fl4auemeKghBAiczJZDfVF4JcAWuuXgG5gURLzUm4o7duh+rXYZTVYgbxUhiSEEBmVjuE+vqGUuhBYBXxLa71CKVUBWFrr+qjl1gNTlFKr480DXh/KhisqioYVcNvmZgBKS/OprCyOu5zjOKxvWEfBjL0HXW40pWs7Q5WtcUH2xiZxDY3ENTSjHdeIkoVS6g2gNs7sauDbwBatdVgpdQ7wlFJqxki2ORQNDW2Ew0NpgTAiXWdbWrqoq2uNu1y4vZFQexO9xZMGXW60VFYWp2U7Q5WtcUH2xiZxDY3ENTTDjcu2rbgX2SNKFlrrRI+D2xS17O+UUj8FJmut1ymlUEqNjypB1AIbtNYN8eaNJNahSLY3VKhuDQC+8dNSHJEQQmRWStsslFKTol4fA4ToTyCPARe58w4G8oHlScxLuWTbLML1a8GysMfHK1wJIcSuIdVtFve53WDDQAtwotY66M67GtPL6VxM99iztdbhJOalXpLZIlS3FrtsEpY/N+UhCSFEJqU0WWitjx5k3lbMPRhDmpcOfUOUD5ItHMchXL8W35T56QpLCCEyRu7g9pDM2FBOeyNOZ4u0VwghxgRJFoMYrBYqVO82bldOS0ssQgiRSZIsPCTTGypctxYsG7tCGreFELs+SRYe+u/MiJ8tQg3rzZ3b/px0hCSEEBklycJL39hQ8RcJN2/DLpuQnniEECLDJFl4cPqewR1nfjiI01KHXSrJQggxNkiy8OAweLZwWurBCWGXTUxfUEIIkUGSLDz0lyy8s0W4eSsAdml1ukISQoiMkmQxmDgli/5kIdVQQoixQZKFh76us3Hmh5u3YuUWYeUNbwh0IYT4pJFk4SHRDdzhpq1Y0hNKCDGGSLLw0pct4rdZSBWUEGIskWThIdIbyvbIFU5vF05HkyQLIcSYIsnCgzPIYOjh5m2A9IQSQowtkiw89A1R7lENFW7ZDoBdUpXWmIQQIpMkWXhwBnlst9NqnvRql1SmKRohhMg8SRYeBmvfDrfWQW4hVk5BWmMSQohMkmThZZA7uMOtddjFUqoQQowtkiw8DDY2VLilDrt4fHoDEkKIDJNk4SHeqLOOE8ZprZfGbSHEmCPJYjAx2cJpb4JwEEtKFkKIMUaShYf+saEGZotwax2AtFkIIcYcSRYe4vWGkm6zQoixSpKFlzj3WZgb8iysooq0hiOEEJkmycJDXzVUTMki3FqPVViO5QtkICohhMgcSRYe+ocoH5gtnLZ66TYrhBiTJFl4iDfcR7i9EauwPL3BCCFEFpBk4WnnaijHcXAkWQghxihJFh48Sxbd7RDqxS6QZCGEGHskWXjo7zrbX7QIdzSaaUWSLIQQY48kCy+R4T6iq6HaTbKQkoUQYiySZOEh3HcHd9Q0N1lIm4UQYiySZOHF4xZup20HYGEVlGUiIiGEyCj/SFeglDoL+CYwF7hca/3zqHkFwD3APkAQuFJr/feRzEuHvseqRk0Lt2zHKhqH5RvxLhNCiE+c0ShZvAWcBjzkMe9KoEVrPQs4AbhbKVU0wnkp53i0WYRb62RociHEmDXiZKG1fk9r/QEQ9ph9KnCXu9xHwDLg2BHOSzmvnrNOy3YZQFAIMWalus2iFlgX9fd6YMoI56Ve39hQpmjh9HTidLZgFUvJQggxNiWsgFdKvYE5eXup1lqHRjek0VNRMbyaK2d9k3n/uEIqK4vo3tZAG1A2uZaiyuLRC3AYKjO8/XiyNS7I3tgkrqGRuIZmtONKmCy01gtHsP71wFSgzv27FvjXCOclraGhjXA4zkBPg4i0WexobCeAQ3DjBgDawgV01rUOeX2jpbKymLoMbj+ebI0Lsjc2iWtoJK6hGW5ctm3FvchOdTXUY8CFAEqp2cBi4KkRzkuDgb2hwh3N5m/pNiuEGKNGnCyUUqcrpTYCXwC+r5TaqJSa686+BShTSn0M/B24QGvdOsJ5KefE3GfhRJJFfkm6QhBCiKwy4psGtNYPAw/HmdeOSSKjNi8d+rrORv7ubIacAix/TqZCEkKIjJI7uD0NrIZyOpqxC0ozF44QQmSYJAsP/dVQ7t+dLVj5kiyEEGOXJAsPsY9VDXc0Y0nJQggxhkmy8BA73IfT2SwlCyHEmCbJwlP/vRlObxf0dknJQggxpkmy8NBfsrBwOlsAsKXbrBBiDJNk4SH6Nou+eyykZCGEGMMkWXhwop6UF+6M3JAnyUIIMXZJsvAQfQe3lCyEEEKShbeokoXT2QyWhZUnbRZCiLFLkoWHvr5QbpuFlVeMZcuuEkKMXXIG9BA9NpTckCeEEJIsPDmRsaHcrrNSBSWEGOskWXiJel6S092OlZedT8ISQoh0kWThYcB9Ft1tWLmFGY1HCCEyTZKFh76us+EwdHdg5UmyEEKMbZIsPLnZorcTcLByvZ9JK4QQY4UkCw99vaF62s3/Ug0lhBjjJFl46KuG6naTRZ6ULIQQY5skC09utuhpA6RkIYQQkiw89JcsOgCkzUIIMeZJsvDQd5tFj1RDCSEESLLwFBminO42wIKcgozGI4QQmSbJYjDd7ZBbIIMICiHGPDkLeojuDSWN20IIIcnCU6QaypFkIYQQgCQLT9EN3NK4LYQQkiw8RaqhnC4ZRFAIIUCShbcB1VBSshBCCEkWHhzAIgw9HVKyEEIIJFl4chwosHoAuSFPCCFAkoUnB6c/WUjJQgghJFl4cqDI7gZkXCghhABJFp4coMByk4VUQwkhBP6RrkApdRbwTWAucLnW+udR8+4Fjgbq3UmPaa1vdOdVA/cD04BO4AKt9ZJE89LBcRwKbamGEkKIiNEoWbwFnAY8FGf+zVrrvd1/N0ZN/wHwotZ6DnAJ8IBSykpiXloU2lKyEEKIiBEnC631e1rrD4DwEN/6ReCX7jpeArqBRUnMSznTG6obM+Jsfro2K4QQWSsdbRbfUEq9q5T6s1JqdwClVAVgaa3ro5ZbD0wZbF4aYgXcaiir24w4a0mzjhBCJGyzUEq9AdTGmV2ttQ4N8vZvA1u01mGl1DnAU0qpGcOIc1gqKoZfhVRg9xAoLKGysngUIxq5bIsnIlvjguyNTeIaGolraEY7roTJQmu9cLgr11pvinr9O6XUT4HJWut1SimUUuOjShC1wAatdUO8eUPdfkNDG+Gwk3jBGJFqqJA/n7q61iG/P1UqK4uzKp6IbI0Lsjc2iWtoJK6hGW5ctm3FvchOaR2LUmpS1OtjgBAQSSCPARe58w4G8oHlScxLuUjXWbnHQgghjNHoOns6cAtQDpyklLoa+LTb6H2f2w02DLQAJ2qtg+5br8b0cjoX0z32bK11OIl5Kec4jpsspNusEELAKCQLrfXDwMNx5h09yPu2Yu7BGNK8dCmwuqXbrBBCuKSrj5dwiHyrV6qhhBDCJcnCgy/YCYCVW5DhSIQQIjtIsvDgC7t3b+dIshBCCJBk4ckX7DIvcvIyG4gQQmQJSRYe/CG3ZBGQoT6EEAIkWXiSaighhBhIkoUHX8hUQ1lSDSWEEIAkC0+RkgVSshBCCECSxU7WvfcWk7e8AIAVkJKFEEKAJIud1L+/hFK7k5BjYflzMh2OEEJkBUkWMTr9ZQD4rKGPViuEELsqSRYxOnPKMx2CEEJkHUkWMbpyx2U6BCGEyDqSLGL05pRmOgQhhMg6kixi2b5MRyCEEFlnxM+z2NXYFtzSfBx5BQX8X6aDEUKILCHJIobPttgYqqDUkW6zQggRIdVQMWzbAsCRnrNCCNFHkkWM/mQh2UIIISIkWcSwLSlZCCFELEkWMSIli3BYsoUQQkRIsojRV7JAkoUQQkRIsojh6ytZZDgQIYTIIpIsYkgDtxBC7EySRYxINVRYkoUQQvSRZBHDdveI5AohhOgnySKG9IYSQoidSbKI0d8bSgghRIQkixiRZCGEEKKfJIsYka6zQggh+kmyiGFJshBCiJ1IsoghJQshhNiZJIsY0mYhhBA7k2QRw5aShRBC7GTET8pTSv0COAroBtqAr2utl7nzqoH7gWlAJ3CB1nrJSOalmuQKIYTY2WiULJ4E9tRa7wX8AHgkat4PgBe11nOAS4AHlFLWCOellM+WwpYQQsQacclCa/33qD9fBSYrpWytdRj4IqZ0gNb6JaVUN7AIeH0E81LKklwhRNo4jkNbWzOdnW2Ew6G4y23fbhPOwqGgP6lx+f05lJdX4vMlnwJGnCxiXAo8obUOK6UqAEtrXR81fz0wRSm1ejjzGGKyqKgoGvIHaO7qP2ArK4uH/P5Uy8aYIHvjguyNTeKCdevW4Thhqqom4vP5saSDSco5jkNrazMdHY3MmDEj6fclTBZKqTeA2jizq7XWIXe504AzgEOT3nqKNTS0DXmMp+bmjr7XdXWtox3SiFRWFmddTJC9cUH2xiZxGS0tbVRXTwZsQiGHeAPt+P02wWD2XcF/UuPKzy9m27bGnb5r27biXmQnTBZa64WJllFKnQLcCByltd7mvq9BKYVSanxUKaEW2DDceYniGA3SG0qIdHKwpO437YZTghvxt6SUOh74CXCM1nptzOzHgIvc5Q4G8oHlI5yXUnKfhRAiW9x443U8/vgjiRdMg9Fos7gH6AH+oJSKTDtKa90AXI3pyXQupgvs2W7DNyOYl1JSshBCRASDQfz+0W7azfy2hmM0ekNVDjJvK3D0aM5LNUkWQoxtBx+8iC996Su8+urL7LffAZxxxtncccdPWbXqI3p6eliwYBFf+9r/sGnTBr797W9y//2PEgwGOe64ozj33C9zxhnn8Pzzz/Kf/7zAddfdyMMPP8Dzzz9DKBQkJyeXK6+8mtmzlee2Tj7589xww7U0NNQzYcJE7Kiu/H/5yx959NGHCARycJww3/vezUydOi1t+yV701iGSK4QInNefncLL72zZafpljXyp1cePH8iB+05Mallc3Nzufvu3wFw883fZ++9F3L11d8hHA5z/fXX8MQTf+XEE0+hvb2d+vp6tm7dzPTpM1m27HXOOOMcli9fyqJFiwH4zGeO4/TTzwLg9deXcMstP+BXv7rXc1vf/vb/stdeCzj//AvYtGkj5513BvvtdwAAd975Mx588HHGjx9PT09P2rvsSrKIITflCSGOPfb4vtcvvfQiK1a8z+9//yAAXV1dVFVVA7Bo0WKWL1/Kli2bOemkz/Hgg7+jt7eXZcuWctZZ5wGg9Qruv/8eWlqasW2bDRvWx93WG28s5/LL/xeASZMm9yUcgIULF3Pjjddy0EGHcMABBzNp0uSUfPZ4JFnEkJKFEJlz0J7eV//p7qKan18Q9ZfDTTf92PPkvM8+i1m+/HU2b97Ed7/7fd566w2ee+5pHAdqaibR29vLd75zFT//+a9Rajfq6+s4+eRjB9lWfDfddAsrVrzP8uXLuOyyi7jyym9xwAEHjeRjDolcRseQNgshRLSDDjqUBx64j1DI3LDb1NTE5s2bAFi8eF+WLHmV1tZWqqqqWbRoX37zm7v6SgQ9Pd2EQqG+ksgf//jYoNvaZ59FPPHEXwHYvHkTy5aZ+5CDwSCbN29i7tx5nH32eey77/589JFOyeeNR0oWMSRZCCGiff3rV3Dnnbdz3nmnY1kWgUAOl112BTU1k6iqqqagoID58/cGTElj27atLFy4CIDCwiK+/OUL+cpXzqGkpJQjjjgqwbau5IYbruW5555m4sQaFizYB4BwOMyNN15HW1srlmVTXV3NRRddmtLPHctyRtpqlJ2mAWuGcwd3Z3eQS376IgC/vfrI0Y9sBOSu36HL1tgkLmPr1nVMmDA14XKf1DulMyWZuLz2fdQd3NOBtQPmjW6In3xSshBCiJ1Jsoghj1UVQoidSbKIIcN9CCHEziRZxJBcIYQQO5NkEUPG0xdCiJ1JshBCCJGQJAshhBAJSbIQQogM+81v7qK3t7fv79F6jsWll17Ayy//Z8TrAUkWQgiRcffc8+sBySIbyXAfQgjh6urq4oYbrmXt2tX4fH5qa6dyyin/xc9+ditz5+7B+++/i9/v55prvsc99/yaNWtWUVVVzY033kJ+fj4dHR3cdpsZ8A/M8ORnnnkuABs3buCWW26iqakRn8/HBRdcwv77H8itt/4QgIsvPh/LsrnjjrsAWL16FZdddhHbt29jjz325JprrseyLNrb2zyfr+Hz+VizZjU33XQ9XV2dzJgxk56enlHbN5IshBBZo3fly/TqF3eablkWIx2aKKAOJTBn8FFalyx5lY6Odh54wAz419LSwscfr2Tt2tVcc811XHXVNdx66w+54oqvcddd91BTM5HLL7+U5557mhNOOJl7772bcDjM7373CB0d7Vx44fnMmDGLAw44iOuvv4aTTjqF448/mTVrVnPppV/hgQf+wBVXXMWf/vQY/+///ZaCgv4RaFevXsVtt92Jbdt86UtnsmzZEhYv3p877vhp3OdrfP/73+ULXziNE044kbfeepuvfvXLI9pn0SRZCCGEa9as2axdu4Zbb/0hCxbsw4EHHgxAbe3UvqfbKaXYtm1L30iySu3Oxo0bAFi2bClf//qVWJZFYWERRx/9aZYtW8pee+3Nxx+v5LOfPRGA6dNnMGuW4v333+Xggw/1jOWQQw4nNze3b5ubNm1k8eL4z9dob29jzZpVHHPMZwGYN29PZsyYNWr7RpKFECJrBOYc5Hn1n64B+yZNmswDDzzKsmWv89prL/OrX/2Cyy//X3JycvuWsW0fOTk5UX/bfcOXj6bc3Oht+KK24f18jfb2tlGPIZo0cAshhGv79m3Yto9DDz2cyy67gqamRlpaWpJ+/6JF+/LEE3/BcRw6Otp5/vlnWLx4PwoKCpk1aw5PPvl3ANauXcOqVSvZY489ASgoKEz6ZB/v+RqFhUXMmDGLZ599CoAPPniP1as/HsrHH5SULIQQwrVq1cf88pc/ByAcDnHWWecxfvz4pN9/3nn/zU9/+iPOOedUAI455rPsv/+BAFx77Q3ccstNPProQ/h8Pq655nuUl5cDcNppZ3LZZReRm5vX18Adz2DP17jmmuu56abrefDB+5g+fSa77TZ3OLvBkzzPwsP5N/8TkOdZJCtb44LsjU3iMuR5Fqkhz7NIo6qy/EyHIIQQWUOqoTz84ebj2dGQ2sYiIYT4JJGShYfcgA+/T3aNEEJEyBlRCJFBFo6TfXX+u7rhtFVLshBCZExOTh5NTfUEg70jvkNbJMdxHNrbW/D7cxIvHEXaLIQQGVNeXklbWzM7dmwjHI5/Y5tt24TD2VcC+aTG5ffnUF5eOaR1SrIQQmSMZVkUF5dRXFw26HLS1XhoUhGXVEMJIYRISJKFEEKIhHbVaigfmLsRh2sk700liWvosjU2iWtoJK6hGU5cUe/xxc7bVYf7OBgYnWcJCiHE2HMI8FL0hF01WeQCi4EtwOiPHSyEELsmHzAReB3ojp6xqyYLIYQQo0gauIUQQiQkyUIIIURCkiyEEEIkJMlCCCFEQpIshBBCJCTJQgghREKSLIQQQiS0qw73MSxKqTnAfUAF0ACco7X+KEOxrAW63H8AV2mtn1ZK7Q/cBeRjHqh+ltZ6ewrj+DHweWAasKfW+j13etx9lY79OEhca/HYb+68lO87pVQFcD8wE+gBPgIu1FrXDbb9VMeWIC4HeBeIjGl9ttb6Xfd9JwC3YM4Vy4Evaa07Risudxt/Bqa7228Dvqa1fivTx1iC2NaSwePM3c61wHW4x3+qjy8pWQz0S+AXWus5wC8wOzeT/ktrvbf772mllA08AFzixvgicHOKY/gzcCiwLmb6YPsqHfsxXlwQs98A0rjvHOBHWmultd4TWAXcPNj20xSbZ1xR8w+M2meRRFEE/Bo4QWs9C2gFrhzluADO1VrvpbVeAPwY+K07PdPH2GCxQQaPM6XUQmB/3OM/HceXJAuXUqoKWAg87E56GFiolBraE0JSax+gS2sdGbPll8AXU7lBrfVLWusN0dMG21fp2o9ecSWQln2ntd6htX4hatJrwNQE2095bIPENZhjgWVRV+y/BE4dzbjc2Jqj/iwFwtlwjMWLLcFbUv5dKqVyMQny4iS3OyoxSbLoNwXYpLUOAbj/b3anZ8qDSql3lFJ3KqXKgFqirqS11vWArZQal+a4BttX2bAfY/cbZGDfuVd0FwN/TbD9tMYWE1fEC0qpt5RSP3BPRsTGBawnRd+jUupupdR64EbgXLLoGPOILSJTx9n3gAe01mujpqX8+JJkkb0O0VrvhRkQ0QJ+nuF4Pimyab/dgannzrbvLjauWq31Iky13lzgO+kOSGv931rrWuD/MG0kWSNObBk5zpRSBwCLgDvTsb1okiz6bQAmKaV8AO7/Ne70tItUsWituzEHxkGYK7u+qgOl1HggrLXekebwBttXGd2PcfYbpHnfuQ3ws4FTtdbhBNtPW2wecUXvsxbgbuLsM8wVakq/R631/cARwEay7BiLxKaUqsjgcXYYsDuwxm1knww8DcwaZLujEpMkC5fbM+At4HR30unAm1rrunTHopQqVEqVuq8t4DQ3tuVAvlLqYHfRi4DH0h3fYPsqk/txkP0Gadx3SqmbMPXEJ7snk0TbT0tsXnEppcqVUvnuaz/wX/Tvs6eAxUqp2VFxPTrKMRUppaZE/X0CsAPI+DE2SGxdmTrOtNY3a61rtNbTtNbTMEn1GEyJJ6XHlwxRHkUptRumO1450IjpjqczEMcM4HHM2PI+4APgMq31FqXUgZieH3n0d4HblsJYbgc+B0wA6oEGrfUeg+2rdOxHr7iAE4iz39z3pHzfKaX2AN4DVgKd7uQ1WutTBtt+qmOLFxfwI3e7DhAAXgEu11q3ue87yV3GB7wJnKe1bh/FuKqBvwCFmGfP7ACu1Fq/kQXHmGdsQBMZPs6iYlwLHK9N19mUHl+SLIQQQiQk1VBCCCESkmQhhBAiIUkWQgghEpJkIYQQIiFJFkIIIRKSZCFECiml/k8pdfcw33uvUuqG0Y5JiOGQIcqFSCGt9U2ZjkGI0SAlCyGEEAlJyUKIKEqpGsxAe4diBtv7qdb6dqXUdcA8zJ28n8U8POhLWuu33fddBVwGlGBGQP2q1vp5932ztNZnucudCPwAmIQZIuJirfUKd94C4DeYsZv+gbmrOjq244EbMA98+gC4SGv9zmDbH929I8YyKVkI4XKH7v4b8DbmZH4UcLlS6hh3kZMwY+qMAx4C/qyUCiilFHApsFhrXYwZq2etx/rnYJ69cDlQiUkIf1NK5SilcjAPdLrfXf9jmCcBRt67APPgnQsxT4e7C/irUio32e0LMRJSshCi32KgUmv9Pffv1UqpX2MGilsHLNda/wFAKfUT4ArM08q2ALnAXKVUXcxzBqKdCjyhtX7WXcePga8DB2IeqhMAbtNaO8AflFLfiHrvBcBdWusl7t/3KaX+z93+piS3L8SwSclCiH5TgRqlVFPkH+YZBtXu/L4hsN3hvTcCNVrrjzGlheuA7Uqp37vVWbFqGPgQmrC7zknuvE1uooiIfvDQVOCKmNimDHH7QgyblCyE6LcBM0Ls7NgZbttD9HDVNuZZApsBtNYPAQ8ppUowVUQ/BM6OWc1mYM+odVjuOjdh2icmKaWsqIRRi3lWdiS2G7XWN3oFnuT2hRg2SRZC9FsKtLqNxbcDPZgHzeS78/dRSn0O8zjSy4Bu4DW3zWAS8DLQhRkC3Oex/keBq5VSRwEvYqqgujHDggMEgcuUUndihlvfF/iXO+/XwJ+UUs+5cRYAh7vrqUly+0IMm1RDCeFyn+V8PLA35lkP9Zgnx5W6i/wF0+7QiLlq/5zWuhfTXnCzu/xWoAr4lsf6NXAWprdVPSYhnKC17tFa92Cez3Ee5rkJpwJ/jHrvMuArmMd3NgIfu8uS7PaFGAl5noUQSYjtAivEWCMlCyGEEAlJshBCCJGQVEMJIYRISEoWQgghEpJkIYQQIiFJFkIIIRKSZCGEECIhSRZCCCESkmQhhBAiof8PwOWbZ1y9wrIAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "开始测试！\n",
      "环境：CliffWalking-v0, 算法：Q-learning, 设备：cpu\n",
      "回合数：1/20, 奖励：-13.0\n",
      "回合数：2/20, 奖励：-13.0\n",
      "回合数：3/20, 奖励：-13.0\n",
      "回合数：4/20, 奖励：-13.0\n",
      "回合数：5/20, 奖励：-13.0\n",
      "回合数：6/20, 奖励：-13.0\n",
      "回合数：7/20, 奖励：-13.0\n",
      "回合数：8/20, 奖励：-13.0\n",
      "回合数：9/20, 奖励：-13.0\n",
      "回合数：10/20, 奖励：-13.0\n",
      "回合数：11/20, 奖励：-13.0\n",
      "回合数：12/20, 奖励：-13.0\n",
      "回合数：13/20, 奖励：-13.0\n",
      "回合数：14/20, 奖励：-13.0\n",
      "回合数：15/20, 奖励：-13.0\n",
      "回合数：16/20, 奖励：-13.0\n",
      "回合数：17/20, 奖励：-13.0\n",
      "回合数：18/20, 奖励：-13.0\n",
      "回合数：19/20, 奖励：-13.0\n",
      "回合数：20/20, 奖励：-13.0\n",
      "完成测试！\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEcCAYAAADdtCNzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA1KElEQVR4nO3de1wU5f4H8M/uAl4JKQEBT5oiBJoCgqgoClqAgFxMjxiaWnS0hPRoR7x0CjWvFSmVhnfDe4kGoqVk3o+K92NoQlnIVREERNmVfX5/eNyfyHAdLpaf9+vl67W788wz3x2enc/OzDqjEEIIEBERPUbZ1AUQEdGTiQFBRESSGBBERCSJAUFERJIYEEREJIkBQUREkv7SAeHh4YFjx441+nKTk5Ph6enZ6Mul+nHv3j1MmDABPXv2RHh4eL31u2PHDgQHB9dbf7Xx5ptvIi4urkH63rRpE/r27QsHBwfk5+c3yDIeFR0djWnTpgEAMjMz4eDggLKyMgDAzZs38dprr8HBwQELFy6EEAIzZsyAs7MzXn311XqtIyIiAlFRUQCAEydOwM3NTbLdn3l78JcOiKbi5OSE77//vqnLoDrau3cvbt68iRMnTmDZsmWSbVJTU3Uh4uDggDFjxuDcuXONW2gtrFq1CoGBgfXer0ajwcKFC7FmzRqcPXsWxsbG9dJvfHw8goKC4ODggH79+uHNN99EcnJyhXYWFhY4e/YsVCoVAGDr1q0wNjbGmTNnEBERgdOnT+Po0aM4ePAgvvnmG3h6eiIxMVE3/+nTp2FjY1PhNQcHB9y/f79e3ktTbg/WrVsHV1dXODo6YsaMGVCr1bWanwFRBw+/rfyZ/RXeQ0PJzMxEx44doaenJzn9jz/+QHBwMGxsbJCUlITDhw9j8ODBGDduHC5cuNDI1aLeNmR1kZeXh9LSUlhZWdV6XiEEtFpthdfXrl2L+fPnY8KECTh69CgOHDiAUaNGISkpqdo+MzMz0blzZygUCgBARkYGLC0t0bJlSwCAs7MzTp06pWufnJyMTp06VXjNwcGh0r//n8Xhw4cRExODdevW4cCBA7h+/XqlX3gq89QEhFarRUxMDAYPHgwXFxe8++67KCgo0E0PDw+Hq6srevbsiddeew1Xr17VTYuIiMAHH3yA0NBQ2Nvb48SJE/Dw8MDq1avh5+eHnj17YvLkySgtLQVQcXezqrYAsHLlSvTr1w/9+vXD9u3bYWNjg99//13yfRQUFGDGjBno168fnJ2d8fbbbwOQPnzxaD+Pv4fVq1fD1dW1XFDs27cPfn5+NVpfj9u2bRtefvll9OrVCxMmTEBOTk65OjZv3oxXXnkFTk5OiIyMRGX/gb+srAwrVqzA4MGD4eDggKCgIGRlZen62bBhAwYNGgQXFxcsWrRIt4F59LADAFy/fh02NjaVbjzT0tIwevRoODk5wcfHR7fxWbZsGb788kvs2bMHDg4O2L59e4V5o6OjYW9vjylTpqBNmzZo3bo1xowZg6FDh2LJkiWVriOpGsaNG4devXpV+Gb7008/ISAgAI6OjhgwYACio6MrvLft27dj4MCBeP3113V//0WLFsHZ2RkeHh44ePCgbp7Ro0fr3kt1bdPT03WHacaOHYvIyMhy6/ah3377DV5eXgAebHjHjBkDADhz5gyGDRuGnj17YtiwYThz5ky5OqKiojBy5Ej06NED6enp5fosKirCsmXL8O9//xuvvPIKWrZsCX19fXh4eGD69OkVanj07xwREYGdO3di9erVcHBwwJYtWzB79mycO3cODg4OWLZsGZycnMrtiSQnJyM0NLTCa05OTgCq3i5UZcOGDRgyZAiys7MbbHtw/vz5Kj/DO3fuxKuvvoouXbrAyMgIb7/9du0PM4q/MHd3d3H06FEhhBDr1q0Tw4cPF1lZWaK0tFS8//77YsqUKbq227dvF0VFRaK0tFTMmzdPDB06VDdt+vTpwtHRUSQnJ4uysjJx79494e7uLoYNGyays7NFfn6+8PLyEps2bRJCCPGf//xH9O/fv1wdlbU9ePCg6Nu3r/jll19ESUmJmDp1qrC2thbXrl2TfE+hoaHi3XffFQUFBUKtVosTJ04IIYT49ttvxciRI8u1fbQfqfcwaNAgceTIEV37sLAw8dVXX9VofT3q2LFjolevXuK///2vKC0tFXPmzBGjRo0qV8dbb70lbt++LTIyMoSLi4s4ePCgZF8rV64Uvr6+Ii0tTWi1WpGSkiJu3bql6yckJETk5+eLjIwM8corr4ht27YJIYRYtmyZmDp1qq6f9PR0YW1tLTQaTYVlqNVqMXjwYLF8+XJRWloqjh07Juzt7UVaWppkX4/r27ev+Oabbyq8fvz4cWFrayvu3bsnOd+jf6M7d+4INzc38c033wiNRiMuXbokevXqJa5evSqEeDCGLl++LMrKykRKSoro06eP2LdvX7n39t5774k7d+6Iu3fvim+//VbY2dmJrVu3ivv374uNGzcKV1dXodVqhRBChISE6NZVdW1HjBghFi5cKEpLS8WpU6eEg4NDpevj8fWcn58vnJycRFxcnNBoNCI+Pl44OTnp/oYhISFiwIAB4pdffhEajUao1epy/R08eFDY2tpK/t0eevTv8/jyp0+fLj799FPJdS6EENevXxc2NjYiPz9flJWVid69e4u7d+8KNzc33WuOjo7i5MmTQojqtwsPl/XoZz46OloEBASIvLy8CtOEqN/tQVWfYT8/P7F7927dtLy8PGFtba37W9TEU7MHsWXLFkyZMgXt2rWDgYEBJk2ahO+//173DfPVV19F69atYWBggLCwMFy+fBlFRUW6+QcNGoSePXtCqVSiWbNmAB58GzIzM0ObNm3g7u6OlJSUSpdfWds9e/YgKCgIXbp0QYsWLRAWFlZpH7m5uTh06BAiIyNhZGQEfX199OrVq8br4PH34OPjg4SEBABAcXExDh06BB8fnxqtr0fFx8dj2LBh6Nq1KwwMDPDPf/4T586dw/Xr13VtQkND8cwzz8DCwgIuLi64fPmyZI3bt2/Hu+++i06dOkGhUODFF18sd1w7NDQUbdq0gYWFBcaMGaOrvzbOnz+PkpISvPXWWzAwMECfPn3g7u6O3bt312j+/Px8mJiYVHjdxMQEZWVlVe5pPfTTTz/B0tISw4YNg56eHuzs7ODp6Ym9e/cCAFxcXGBjYwOlUokXX3wRPj4+OHnyZLk+wsLC0LJlSzRv3hzAg+PxI0aMgEqlQmBgIG7cuIGbN29KLr+ytpmZmbh48SLCw8NhYGAAJycneHh41Gi9PHxfHTp0QEBAAPT09ODr64tOnTrhwIEDujaBgYHo0qUL9PT0oK+vX27+goICGBsbN9jhHUtLS1hYWCA5ORmXL19Ghw4d0Lx5czg6Oupe02g06NGjB4DqtwuPEkJgwYIFOHr0KDZs2IBnn3220jrqY3sAoMrPcElJCVq3bq1ra2hoCAC4c+dODdcW8Oc+yFYLmZmZeOedd6BU/n8mKpVK5OXloW3btoiKisLevXtx69YtXZv8/HzdSjU3N6/Q56MbiRYtWiA3N7fS5VfWNjc3F926ddNNk1rOQ9nZ2TAyMoKRkVF1b1fS4337+flh5MiRiIyMxL59+2BnZwdLS0sAVa8vMzOzcv3k5uaia9euuuetWrVCmzZtkJOTg/bt2wOo+P4rG6TZ2dl4/vnna/QeLC0tq1znlcnNzUW7du3KvTcLC4tyh8WqYmxsjBs3blR4/caNG1AoFDA2NtYdunjY9+Phk5GRgQsXLugOZQAPDq8NHToUwIMQ+/jjj3H16lVoNBqo1Wrd4ZyH2rVrV+5527ZtdY9btGgB4MFGQkplbfPz82FkZKR7DXiwzh8e5qtObm4uLCwsyr32+Lqtaoy3adMG+fn5uH//foOFxMPDTObm5rr137NnT91r3bt3h4GBAcrKyqrdLjyqqKgI27ZtQ1RUlOT0R9Vle5CZmanb+APA2bNnq/wMt2zZEsXFxbr2Dx+3atWqZisKT1FAtGvXDvPnz0fPnj0rTNu5cyeSkpKwdu1atG/fHkVFRXB2dq70OHl9MjU1LffhqeqD2K5dO9y+fRuFhYV45plnyk1r0aIF7t27p3sutQF7nJWVFSwsLHDo0CEkJCTA19e33LIqW19S7yEjI0P3vKSkBAUFBRWCpCbatWuHP/74A9bW1pLTs7Ky0KVLFwAPPjCmpqYAKr7/yr45P6w3OzsbWq1W96HPyspCx44da1Rjnz59sHfvXgwbNqzc63v27IG9vb3um/fZs2cr7cPc3BzOzs5Yu3at5PSpU6ciJCQEq1atQrNmzfDRRx9V+AnpwxOx9cnExAS3b9/G3bt3dSFR03AAHqzbzMzMcq9lZWWhf//+uudV1e3g4AADAwPs37+/QiDWF2dnZ2zZsgWWlpYICgoC8CA04uLiYGlpqQuN+Pj4Wm0XnnnmGSxZsgSTJ0/G559/XqPPzuOq2h48/MXWo6r6DHfp0gVXrlzBkCFDAACXL19G27Zta/VLs6fmEFNwcDA+++wz3Ybs1q1b2L9/P4AHu1wGBgYwNjbG3bt38emnnzZaXV5eXtixYwfS0tJw9+5dfPnll5W2NTU1hZubGyIjI3H79m1oNBrdry9efPFFXL16FSkpKSgtLS13UrMqvr6+WL9+PU6dOlXuA1nV+pLqY8eOHUhJSYFarcann36K7t276/YeamP48OFYunQprl27BiEELl++XG7DuHr1aty+fRtZWVm6E4EAYGtri1OnTiEzMxNFRUX46quvKl1G9+7d0bx5c6xatQoajQYnTpzAjz/+qOurOpMmTcLZs2cRFRWFgoICFBcX4+uvv8aOHTtq/P8mBg4ciGvXrmHnzp3QaDTQaDS4cOEC0tLSADwYk0ZGRmjWrBkuXLhQp0NpdWFpaYlu3bohOjoaarUaZ8+eLXd4qDoDBgzAtWvXEB8fj/v37yMxMRGpqakYOHBgjeY3NDREeHg45syZg/379+Pu3bvQaDQ4ePAgFi9eXMd3VZ6TkxNSUlJw6tQpODo6AgCsra1x/fp1nDhxAs7OzgDqtl1wcXHBxx9/jLCwsDr9oq0224OHKvsM+/v745tvvkFqaioKCwuxfPnyWv/U+akJiDFjxsDDwwPjx4+Hg4MDRowYofsDBgQEwMLCAv3794ePjw/s7e0bra4BAwZg9OjRGDNmDF5++WXdsU8DAwPJ9osXL4aenh68vb3Rt29frF+/HgDwwgsv4J133sHYsWPxyiuv1Pjbi6+vL06dOoXevXuXO2Za1fp6XN++ffHuu+8iLCwM/fr1Q3p6uu4/ENXWuHHj4O3tjfHjx8PR0RGzZs0q9wuPQYMGISgoCAEBARg4cKDuPz+5urpiyJAhGDp0KIKCguDu7l7pMgwMDLBixQocOnQIvXv3RmRkJBYvXozOnTvXqMaOHTti06ZNuHz5Mjw8PODs7IylS5fi888/R9++fWvUR+vWrbF69WokJiaif//+6NevHz7++GPd79Q/+OADLFu2DA4ODvjiiy/g7e1do37rw8cff4xz587BxcUFn332GYYMGVLpeHycsbExVqxYgbVr18LFxQWrVq3CihUrqjwe/7jx48cjIiICX375Jfr06YOBAwdi48aNGDx4cF3fUjkvvPACnn32WbRt21a3J65UKtG9e3cUFxfDwcEBQN23C66urrqf6V66dKlWtdV2ewBU/hl2c3PDm2++iTFjxmDgwIGwtLSs9X/8VIjGOI5CNZaWlgZfX19cvHjxT/877PpmY2ODH374AR06dGjqUsrJzs7GiBEjEBYWhuHDhzd1OfVu8uTJ6NSpU73+r3KqmabeHjw1exBPsn379kGtVuP27dtYsmQJ3N3dGQ5/Iu3atcPKlStx48aNWv1C5El14cIF/PHHH9BqtTh06BCSkpLq7ds7Ve9J2h5wK/QE2LJlCyIiIqBSqeDs7IwPPvigqUuiWrKxsYGNjU1Tl1Evbt68ibCwMBQUFKBdu3b48MMPYWdn19RlPTWepO0BDzEREZEkHmIiIiJJDAgiIpLEgCAiIkl/qZPU+fl3oNXW7ZTKc8+1Rl5ecfUNmwjrk4f1ycP65HlS61MqFTA2rvzSG3+pgNBqRZ0D4uH8TzLWJw/rk4f1yfOk1yeFh5iIiEgSA4KIiCT9pQ4xEdGTTwiB/PwbUKvvAaifwy65uUrJ25c+KZq6PpVKD61bt0GLFjW/1DfAgCCiRlZcfBsKhQJmZu2hUNTPQQw9PSXu339yA6Ip6xNCQKNRo6DgwS0AahMSPMRERI3q7t1iGBq2qbdwoKopFAoYGDRDmzYmKC4uqNW8/AsRUaPSasugUvHgRWPT1zdAWVnFWwZXhQFBRI2uIe6GR1WryzpnQBARPSE++uhDfPvt1qYuQ4cBQUT0P/fv1+4QzJ9lWXXFA4FE9FTr188J48aF4vjxo3Bx6YNRo0YjOjoKaWlXoVar4eDghLCwKcjISMfMmf9CbOw23L9/Hz4+g/D6629g1KgxSErah8OHf8KHH36EzZtjkZT0A8rK7sPAoBmmTYuAra2t5LICAoZh3rwPkJd3E+3amUOp/P/v7Lt27cC2bZugr28AIbSYM2chOnTo2KjrhgFBRE3m6MUsHLmQJbsfhQJ4/M42/bqbw/Ul8xrN36xZM6xatQEAsHDhXNjbOyIi4n1otVpERs7G7t3fYejQQJSU3MHNmzeRnZ2JF17ojOTkUxg1agxOnz4JJydnAICXlw+Cg0MAAKdOncCSJQuwZs0GyWXNmvUeevRwwPjxbyEj4zrGjh0FF5c+AIAvv1yKjRu/Rdu2baFWq5vk/1HUS0Ds2rULq1atQlpaGmbOnImQkBDdtMjISBw/fhwGBgZo2bIlZs2ahZdeeqnSvm7dugVfX184OTlh2bJl9VEeEVGVvL19dY+PHDmElJRL2LJlIwDg3r17MDU1AwA4Ojrh9OmTyMrKhL9/EDZu3ACNRoPk5JMICRkLALhyJQVff70WhYW3oVQqkZ7+R6XLOnPmNCZPfg8AYGnZXhcyD5bljI8++gCurv3Rp08/WFq2b5D3XpV6CQhbW1tERUUhJiamwjQ3NzfMnDkT+vr6OHDgAKZMmYL9+/dX2teHH36IAQMG/CXu7UtEVXN9qebf8qsi9z+itWjR8pFnAvPnfyy5Qe7Z0xmnT59CZmYG/v3vuTh37gz27/8eQgAWFpbQaDR4//3p+PzzlbCxeRE3b95AQIB3Fcuq3Pz5S5CScgmnTycjPHwCpk2bgT59XOv8HuuiXk5SW1tbw8rKqtzxs4fc3d2hr68PALC3t0d2dnalu0rfffcd2rZtC2dnZ8npREQNzdXVDbGx61FWVgYAKCgoQGZmBoAHAXHixHEUFRXB1NQMTk69sHr1V7pv/mp1KcrKynR7HDt2bK9yWT17OmH37u8AAJmZGUhOPgXgwQnszMwM2Nl1w+jRY9GrV29cvXqlQd5vVRr1HMTGjRsxcOBAySDJycnBunXr8PXXX+P777+vU//PPddaVn0mJoay5m9orE8e1idPfdWXm6uEnl79/4BSTp96ev9f0z//+R4+/3wpxo0bBYVCAX19fUyePA3PP/83WFiYo1WrVrC3t4eenhK9erlgzpz34ezcC3p6ShgZPYPQ0AkIDR0DIyMjeHgMrlBf+WX9C3PmvI+QkO9hYWEJR8eeUCoVUCqB+fM/RHFx8f8uS2KGSZPCZa83pVJZq7+jQojHT+1UFBgYiMzMTMlpx44dg0qlAgBERESgW7du5c5BPLR7924sW7YMGzduRNu2bStMf+uttzBu3Dj06dMHO3bswE8//VTrcxB5ecV1vua6iYkhbtwoqtO8jYH1ycP65KnP+rKzf0e7dh3qpa+HeC2mmnl83SuViiq/WNdoDyIuLk5WUfv27UNUVBTWrVsnGQ4AcO7cOcyaNQsAcOfOHZSWliI0NBQrV66UtWwiIqqbBj/EdODAASxYsABr165F+/aVn4U/efKk7nFd9yCIiKj+1MuBwISEBLi5uWHv3r1YunQp3NzckJqaCgCYMWMGNBoNwsPD4e/vD39/f+Tn5wMAZs2ahaSkpPoogYiI6lmNzkH8WfAcRNNhffI8TfXxHETTqe05CF6LiYiIJDEgiIhIEgOCiIgkMSCIiJrA6tVfQaPR6J7X170gJk16C0ePHpbdD8CAICJqEmvXriwXEE8iXu6biJqM5pej0Fw5JLsfhUKBx3+QqW/jBn3r6i9ud+/ePcyb9wGuXfsVKpUenn++AwIDX8XSpZ/Azq4rLl26CD09PcyePQdr167Eb7+lwdTUDB99tAQtWrRASUkJPvvswYX1gAeX+37ttdcBANevp2PJkvkoKMiHSqXCW2+9g969++KTTxYBACZOHA+FQono6K8AAL/+mobw8AnIzc1B164vYfbsSCgUCty5Uyx5jwqVSoXffvsV8+dH4u7du+jcuTPUarXs9fkQ9yCI6Kl24sRxlJTcQWzsdqxfvxnvvTcTAHDt2q8IChqODRu2omvX7pg6NQxhYVMQG7sdSqUS+/c/uGbcunWroNVqsWHDVqxYsQZ79uzG8eNHAQCRkbPx8sue2LhxG95/fy7mzn0f+fn5mDp1OgBg+fI1WLduEwwNH1wf6ddf07BkyVJ8/fU2XLlyGcnJJwAA0dFRsLd3xMqVG7B27Sbk59/SXeRv7tx/IyhoOGJjt2H48FG4fPnnels33IMgoiajb+1ao2/51ZHz/wysrLrg2rXf8Mkni+Dg0BN9+/YDADz/fAd06WIDALCxsUFOTpbuKq02Nra4fj0dAJCcfBLvvjsNCoUCrVq1xuDBryA5+SR69LBHauovGDJkKADghRc6wcrKBpcuXUS/fm6StfTvPxDNmjXTLTMj4zqcnSu/R8WdO8X47bc0eHoOAQB06/YSOnWyqtN6kMKAIKKnmqVle8TGbkNy8in85z9HERPzBSZPfg8GBs10bZRKFQwMDB55rtRdDrw+NWv26DJUjyxD+h4Vd+4U13sNj+IhJiJ6quXm5kCpVMHNbSDCw6eioCAfhYWFNZ7fyakXdu/eBSEESkruICnpBzg7u6Bly1awsrLGnj0JAIBr135DWtov6Nr1wR01W7ZsVeMNfGX3qGjVqjU6dbLCvn17AQA///xf/Ppram3efpW4B0FET7W0tFSsWPE5AECrLUNIyNhKrzotZezYNxEVtRhjxvwdAODpOQS9e/cFAHzwwTwsWTIf27ZtgkqlwuzZc2BsbAwAGDnyNYSHT0CzZs11J6kr8+67U/Hll8swdmzw/+5RYYDw8KmwsLDE7NmRmD8/ErGx69CpkxVefNGuLqtBEq/F9D9P07VwGgLrk+dpqo/XYmo6vBYTERHVCwYEERFJYkAQUaP7Cx3Z/tOoyzpnQBBRo3rw8837TV3GU0ejUUOlqt3vkhgQRNSoWrRojaKiAgjR9CdtnwZCCKjVpSgouIHWrdvUal7+zJWIGlXr1kbIz7+BnJzrAOrnUJNSqYRW++QGTlPXp1LpwdDQGC1atKrVfAwIImpUCoUCzz5rWq99Pk0/E25MPMRERESSGBBERCRJdkDs2rULfn5+sLOzQ2xsbLlpkZGR8PLywtChQzFy5EhcvHix0n6OHz+OoKAg+Pj4wMfHB5cvX5ZbGhERySD7HIStrS2ioqIQExNTYZqbmxtmzpwJfX19HDhwAFOmTMH+/fsrtMvJycGsWbOwatUqdOrUCffu3cP9+/wZHBFRU5IdENbW1gAenKV/nLu7u+6xvb09srOzodVqK7TdtGkT/P390alTJwBA8+bN5ZZFREQyNdqvmDZu3IiBAwdKBklqaiosLS0xZswYFBYWwsXFBVOnTi13/XUiImpc1QZEYGAgMjMzJacdO3YMKpWq2oXs3r0b8fHx2Lhxo+T0srIynDlzBmvXrkWzZs0wbdo0xMTEYNKkSdX2/aiqrkpYEyYmhrLmb2isTx7WJw/rk+dJr09KtQERFxcnawH79u1DVFQU1q1bV+k11i0sLNCtWzfdfVm9vLywa9euWi+Ll/tuOqxPHtYnD+urmya93PeBAwewYMECrF69Gu3bt6+0na+vL06cOAG1Wg0hBI4cOYIXX3yxIUsjIqJqyA6IhIQEuLm5Ye/evVi6dCnc3NyQmvrglnczZsyARqNBeHg4/P394e/vj/z8fADArFmzkJSUBABwdHRE//79ERAQgKFDh6KsrAz/+Mc/5JZGREQy8I5y//Ok7gI+xPrkYX3ysD55ntT6eEc5IiKqEwYEERFJYkAQEZEkBgQREUliQBARkSQGBBERSWJAEBGRJAYEERFJYkAQEZEkBgQREUliQBARkSQGBBERSWJAEBGRJAYEERFJYkAQEZEkBgQREUliQBARkSQGBBERSWJAEBGRJAYEERFJYkAQEZEkBgQREUmSHRC7du2Cn58f7OzsEBsbW25aZGQkvLy8MHToUIwcORIXL16U7OPu3buYOnUqfH194ePjg8mTJ6O4uFhuaUREJIPsgLC1tUVUVBR8fX0rTHNzc0N8fDy+++47/OMf/8CUKVMk+9i6dSs0Gg3i4+ORkJAArVaLzZs3yy2NiIhk0JPbgbW1NQBAqayYNe7u7rrH9vb2yM7OhlarrdBWoVDg3r170Gg0AICSkhK0a9dObmlERCSD7ICoqY0bN2LgwIGSQTJy5EicO3cOrq6uAIB+/frBz8+v1st47rnWsmo0MTGUNX9DY33ysD55WJ88T3p9UqoNiMDAQGRmZkpOO3bsGFQqVbUL2b17N+Lj47Fx48ZK+wGAI0eOAACmTp2K1atX44033qi270fl5RVDqxW1muchExND3LhRVKd5GwPrk4f1ycP65HlS61MqFVV+sa42IOLi4mQVsG/fPkRFRWHdunVo27atZJstW7bA398fzZo1AwAMGTIEO3furHVAEBFR/WnQn7keOHAACxYswOrVq9G+fftK27Vv3x5HjhyBEAJarRaHDx9Gly5dGrI0IiKqhuyASEhIgJubG/bu3YulS5fCzc0NqampAIAZM2ZAo9EgPDwc/v7+8Pf3R35+PgBg1qxZSEpKAgC88847KCwshK+vL/z8/KBWqzFx4kS5pRERkQwKIUTdDto/gXgOoumwPnlYnzysr26qOwfB/0lNRESSGBBERCSJAUFERJIYEEREJIkBQUREkhgQREQkiQFBRESSGBBERCSJAUFERJIYEEREJIkBQUREkhgQREQkiQFBRESSGBBERCSJAUFERJIYEEREJIkBQUREkhgQREQkiQFBRESSGBBERCSJAUFERJJkB8SuXbvg5+cHOzs7xMbGlpu2fPly+Pn5ISAgAP7+/khMTKy0n23btuHll1/G4MGDMWfOHGi1WrmlERGRDHpyO7C1tUVUVBRiYmIqTAsJCcHEiRMBADk5OfD29oarqyuMjIzKtUtPT8fnn3+OnTt3ok2bNggNDcV3332HgIAAueUREVEdyd6DsLa2hpWVFZTKil0ZGhrqHpeUlEChUEjuGXz//fcYPHgwnn32WSiVSgwfPrzKvQ0iImp4svcgqrN582asX78e2dnZmD9/PoyNjSu0ycrKgoWFhe65hYUFsrKyGro0nUs/7obeteMQWtFoy6ytK0oF65OB9cnD+uRp6PrKOvVFVw+feu+32oAIDAxEZmam5LRjx45BpVJVOX9wcDCCg4Nx5coVTJs2DX369JEMifrw3HOt6zRfi+b60ABQKBX1W1A9Y33ysD55WJ88DVlfi+b6MDExrL5hLVUbEHFxcfWyIBsbG5iamuLkyZPw9PQsN83c3LxcCGVmZsLc3LzWy8jLK4a2Dindqe8rMPEfhhs3imo9b2MxMTFkfTKwPnlYnzyNUV9d+lcqFVV+sW7Qn7mmpqbqHqenpyMlJQVWVlYV2nl6emL//v24desWtFottm/fDm9v74YsjYiIqiH7HERCQgIWL16MwsJCJCUlISYmBmvWrIGVlRWio6ORmpoKPT09qFQqzJ49G507dwYALF26FKampggODsbf/vY3vP322xgxYgQAwNXVFUOHDpVbGhERyaAQQjy5Z3Zqqa6HmADuosrF+uRhffKwvrpp0kNMRET058WAICIiSQwIIiKSxIAgIiJJDAgiIpLEgCAiIkkMCCIiksSAICIiSQwIIiKSxIAgIiJJDAgiIpLEgCAiIkkMCCIiksSAICIiSQwIIiKSxIAgIiJJDAgiIpLEgCAiIkkMCCIiksSAICIiSQwIIiKSJDsgdu3aBT8/P9jZ2SE2NrbctOXLl8PPzw8BAQHw9/dHYmKiZB/79+9HUFAQfH194ePjgzVr1sgti4iIZNKT24GtrS2ioqIQExNTYVpISAgmTpwIAMjJyYG3tzdcXV1hZGRUrp2JiQmWL18OMzMzFBUVISgoCN27d4eTk5Pc8oiIqI5kB4S1tTUAQKmsuDNiaGioe1xSUgKFQgGtVluhXY8ePcrN07lzZ2RkZDAgiIiaUIOfg9i8eTO8vLwQGBiIuXPnwtjYuMr2aWlpOHfuHHr37t3QpRERURUUQghRVYPAwEBkZmZKTjt27BhUKhUAICIiAt26dUNISIhk2ytXrmDatGnYsGFDpSGRm5uL0aNHY/LkyfD29q7N+yAionpW7SGmuLi4elmQjY0NTE1NcfLkSXh6elaYnpeXh3HjxuHNN9+sczjk5RVDq60y7yplYmKIGzeK6jRvY2B98rA+eVifPE9qfUqlAs8917ry6Q258NTUVN3j9PR0pKSkwMrKqkK7/Px8jBs3Dq+99hqGDx/ekCUREVENyT5JnZCQgMWLF6OwsBBJSUmIiYnBmjVrYGVlhejoaKSmpkJPTw8qlQqzZ89G586dAQBLly6FqakpgoODERMTg2vXrmHr1q3YunUrAGDMmDEYNmyY3PKIiKiOqj0H8WfCQ0xNh/XJw/rkYX1106SHmIiI6M+LAUFERJIYEEREJIkBQUREkhgQREQkiQFBRESSGBBERCSJAUFERJIYEEREJIkBQUREkhgQREQkiQFBRESSGBBERCSJAUFERJIYEEREJIkBQUREkhgQREQkiQFBRESSGBBERCSJAUFERJIYEEREJIkBQUREkmQHxK5du+Dn5wc7OzvExsaWm7Z8+XL4+fkhICAA/v7+SExMrLKv0tJS+Pj4ICgoSG5ZREQkk57cDmxtbREVFYWYmJgK00JCQjBx4kQAQE5ODry9veHq6gojIyPJvqKiotCjRw9cvnxZbllERCST7D0Ia2trWFlZQams2JWhoaHucUlJCRQKBbRarWQ/ycnJuHbtGvz9/eWWRERE9UD2HkR1Nm/ejPXr1yM7Oxvz58+HsbFxhTYlJSWYP38+li9fjmvXrtV5Wc8911pGpYCJiWH1jZoQ65OH9cnD+uR50uuTUm1ABAYGIjMzU3LasWPHoFKpqpw/ODgYwcHBuHLlCqZNm4Y+ffpUCInFixdj1KhRMDMzkxUQeXnF0GpFneY1MTHEjRtFdV52Q2N98rA+eVifPE9qfUqlosov1tUGRFxcXL0UYmNjA1NTU5w8eRKenp7lpp0+fRqHDh3Cl19+idLSUty+fRt+fn6Ij4+vl2UTEVHtNeghptTUVFhZWQEA0tPTkZKSonv+qEeD4MSJE1i0aBF27NjRkKUREVE1ZAdEQkICFi9ejMLCQiQlJSEmJgZr1qyBlZUVoqOjkZqaCj09PahUKsyePRudO3cGACxduhSmpqYIDg6W/SaIiKj+KYQQdTto/wTiOYimw/rkYX3ysL66qe4cBP8nNRERSWJAEBGRJAYEERFJYkAQEZEkBgQREUliQBARkSQGBBERSWJAEBGRJAYEERFJYkAQEZEkBgQREUliQBARkSQGBBERSWJAEBGRJAYEERFJYkAQEZEkBgQREUliQBARkSQGBBERSWJAEBGRJAYEERFJkh0Qu3btgp+fH+zs7BAbG1tu2vLly+Hn54eAgAD4+/sjMTGx0n5SUlLw2muvYciQIRgyZAgOHjwotzQiIpJBT24Htra2iIqKQkxMTIVpISEhmDhxIgAgJycH3t7ecHV1hZGRUbl2JSUlmDRpEj755BPY29vj/v37KCoqklsaERHJIDsgrK2tAQBKZcWdEUNDQ93jkpISKBQKaLXaCu0SEhLQs2dP2NvbPyhKTw/GxsZySyMiIhlkB0R1Nm/ejPXr1yM7Oxvz58+X3PCnpqZCT08PoaGhyM3NRdeuXTF9+vQKexpERNR4FEIIUVWDwMBAZGZmSk47duwYVCoVACAiIgLdunVDSEiIZNsrV65g2rRp2LBhQ4WQmDdvHn788Uds2bIFbdu2xYIFC1BcXIwFCxbU5T0REVE9qHYPIi4url4WZGNjA1NTU5w8eRKenp7lppmbm8PFxQWmpqYAAD8/P8ycObPWy8jLK4ZWW2XeVcrExBA3bjy55z1YnzysTx7WJ8+TWp9SqcBzz7WufHpDLjw1NVX3OD09HSkpKbCysqrQztvbGxcuXEBxcTEA4NChQ7CxsWnI0oiIqBqyz0EkJCRg8eLFKCwsRFJSEmJiYrBmzRpYWVkhOjpad35BpVJh9uzZ6Ny5MwBg6dKlMDU1RXBwMCwsLBAaGoqRI0dCoVCgffv2mDt3ruw3R0REdVftOYg/Ex5iajqsTx7WJw/rq5smPcRERER/XgwIIiKSxIAgIiJJDAgiIpLEgCAiIkkMCCIiksSAICIiSQwIIiKSxIAgIiJJDAgiIpLEgCAiIkkNfsOgxqRUKpp0/obG+uRhffKwPnmexPqqq+kvdbE+IiKqPzzEREREkhgQREQkiQFBRESSGBBERCSJAUFERJIYEEREJIkBQUREkhgQREQkiQFBRESS/lKX2qjOb7/9hoiICBQUFKBNmzZYtGgROnbsWK5NWVkZ5s2bh8OHD0OhUOCtt97C8OHDG7y2/Px8/Otf/8Iff/wBAwMDdOjQAXPmzMGzzz5brl1ERASOHTsGY2NjAICXlxcmTpzY4PUBgIeHBwwMDNCsWTMAwLRp09C/f/9ybe7evYsZM2bg0qVLUKlUmD59Otzd3Ru8tuvXr+Odd97RPS8qKkJxcTFOnjxZrl10dDQ2bdoEU1NTAICjoyM++OCDBqlp0aJF+P7775GRkYH4+HhYW1sDqNk4BBp+LErVV9NxCDT8WKxs/dVkHAINPxal6qvpOAQadyzWmXiKjB49WuzcuVMIIcTOnTvF6NGjK7SJi4sT48ePF2VlZSIvL0/0799fpKenN3ht+fn54j//+Y/u+cKFC8WMGTMqtJs+fbr4+uuvG7weKe7u7uLKlStVtomOjhazZs0SQgjx22+/ib59+4ri4uLGKK+cefPmicjIyAqvL1u2TCxcuLBRajh16pTIzMyssN5qMg6FaPixKFVfTcehEA0/FitbfzUZh0I0/FisrL5HVTYOhWjcsVhXT80hpry8PPz888/w9fUFAPj6+uLnn3/GrVu3yrVLTEzE8OHDoVQq8eyzz2Lw4MHYu3dvg9fXpk0buLi46J7b29sjMzOzwZdb3/bs2YO///3vAICOHTuiW7duOHToUKPWoFarER8fj2HDhjXqch/n5OQEc3Pzcq/VdBwCDT8Wpep7ksahVH210dBjsbr6npRxKMdTExBZWVkwMzODSqUCAKhUKpiamiIrK6tCOwsLC91zc3NzZGdnN2qtWq0WmzdvhoeHh+T0tWvXws/PD2+//TbS0tIatbZp06bBz88PH374IQoLCytMz8zMhKWlpe55U6y/H3/8EWZmZujatavk9N27d8PPzw/jx4/H2bNnG7W2mo7Dh22bcixWNw6BphuL1Y1DoOnHYnXjEGjasVgTT01A/JnMnTsXLVu2REhISIVpU6ZMwb59+xAfH49XXnkFb775JsrKyhqlro0bN+K7777Dt99+CyEE5syZ0yjLra1vv/220m9tI0eORFJSEuLj4/HGG2/g7bffRn5+fiNX+OdQ1TgEmm4s/hXGIfDnGItPTUCYm5sjJydHN4DLysqQm5tbYRfR3Ny83C51VlYW2rVr12h1Llq0CL///js+++wzKJUV/zxmZma61wMCAlBSUtJo34oerisDAwOMGjUKZ86cqdDGwsICGRkZuueNvf5ycnJw6tQp+Pn5SU43MTGBvr4+AMDV1RXm5ua4evVqo9VX03H4sG1TjcXqxiHQdGOxJuMQaNqxWN04BJp+LNbEUxMQzz33HGxtbZGQkAAASEhIgK2tbYVfZ3h5eWH79u3QarW4desW9u/fD09Pz0ap8dNPP8V///tffPHFFzAwMJBsk5OTo3t8+PBhKJVKmJmZNXhtJSUlKCoqAgAIIZCYmAhbW9sK7by8vLB161YAwLVr13Dx4kXJX5g0lLi4OAwYMED3y5rHPbr+UlJSkJGRgRdeeKGxyqvxOASabizWZBwCTTMWazoOgaYdi9WNQ6Dpx2JNPFU3DEpLS0NERAQKCwvxzDPPYNGiRejUqRNCQ0MRHh6Ol156CWVlZZgzZw6OHj0KAAgNDdWd6GpIV69eha+vLzp27IjmzZsDANq3b48vvvgC/v7+iImJgZmZGcaOHYu8vDwoFAq0bt0a//rXv2Bvb9/g9aWnpyMsLAxlZWXQarXo3LkzZs+eDVNT03L1lZSUICIiAikpKVAqlXjvvfcwePDgBq/vIU9PT8yaNQtubm661x79+06fPh2XLl2CUqmEvr4+wsPDMWDAgAapZd68efjhhx9w8+ZNGBsbo02bNti9e3el4/DxWht6LErV99lnn1U6DgE06liUqm/FihWVjsPH62vosVjZ3xeQHodA043FunqqAoKIiGruqTnEREREtcOAICIiSQwIIiKSxIAgIiJJDAgiIpLEgCCqZytWrMCsWbPqNG9ERASioqLquSKiunmqLvdN1BgmTJjQ1CUQ1QvuQRARkSQGBD31cnJyEBYWht69e8PDwwMbNmwA8OCGLuHh4Zg8eTIcHBwQGBiIy5cv6+aLiYlB//794eDgAE9PTxw/flw337Rp03TtkpKS4OPjAycnJ4wePbrcVU9//vlnBAYGwsHBAZMnT0ZpaWm52g4cOAB/f384OTlh5MiRNVo+Ub1pultREDW9srIyERgYKKKjo0Vpaan4448/hIeHhzh06JBYtmyZsLOzE3v27BFqtVqsWrVKuLu7C7VaLdLS0oSbm5vIzs4WQgiRnp4ufv/9dyHEgxvBTJ06VQghxK+//ip69Oghjhw5ItRqtYiJiRGDBw8WpaWlorS0VAwcOFCsXbtWqNVqsWfPHmFnZyc+/fRTIYQQly5dEr179xbnzp0T9+/fFzt27BDu7u6itLS0yuUT1RfuQdBT7eLFi7h16xYmTZoEAwMD/O1vf8OIESOQmJgIAOjatSu8vLygr6+PcePGQa1W4/z581CpVFCr1UhLS4NGo0H79u3x/PPPV+g/MTERAwYMgKurK/T19fHGG2/g3r17OHv2LM6fPw+NRoPXX38d+vr68PLywksvvaSbd+vWrfj73/+OHj16QKVSITAwEPr6+jh37lyNl08kB09S01MtIyMDubm5cHJy0r1WVlYGJycnWFhYlLs89MOrlT5sP3PmTERHRyM1NRX9+vVDREREhauZ5ubmlrvpj1Kp1F3yW6VSwczMDAqFQjf90baZmZnYuXMnYmNjda9pNBrk5uaiV69eNVo+kRzcg6Cnmrm5Odq3b4/k5GTdv7Nnz2LlypUAUO7+BlqtFjk5Oborh/r5+WHz5s04cOAAFAoFPv744wr9m5qalrungxBCd1c5ExMT5OTkQDxyvcxH25qbm2PChAnlajt//rzudqU1WT6RHAwIeqp1794drVq1QkxMDO7du4eysjL88ssvuHDhAgDg0qVL+OGHH3D//n2sX78eBgYG6NGjB3799VccP34carUaBgYGaNasmeSNdby9vXHw4EEcP34cGo0Ga9asgYGBARwcHGBvbw89PT1s2LABGo0GP/zwAy5evKibd/jw4diyZQvOnz8PIQRKSkrw008/obi4uMbLJ5KDh5joqaZSqbBixQosWrQIgwYNglqtxgsvvIDJkycDAAYNGoTExERMnz4dHTp0QHR0NPT19aFWq/HJJ58gLS0N+vr6cHBwkLz1ZadOnbBkyRLMnTsXOTk5sLW1xYoVK3Q34omOjsb777+Pzz77DAMGDMDLL7+sm/ell17C3LlzMWfOHPz+++9o3rw5HB0d4eTkVOPlE8nB+0EQVSI6Ohq///47D93QU4v7pEREJIkBQUREkniIiYiIJHEPgoiIJDEgiIhIEgOCiIgkMSCIiEgSA4KIiCQxIIiISNL/AQrd+XrWqhsMAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 获取参数\n",
    "cfg = get_args() \n",
    "# 训练\n",
    "env, agent = env_agent_config(cfg)\n",
    "res_dic = train(cfg, env, agent)\n",
    " \n",
    "plot_rewards(res_dic['rewards'], cfg, tag=\"train\")  \n",
    "# 测试\n",
    "res_dic = test(cfg, env, agent)\n",
    "plot_rewards(res_dic['rewards'], cfg, tag=\"test\")  # 画出结果"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.13 ('easyrl')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "8994a120d39b6e6a2ecc94b4007f5314b68aa69fc88a7f00edf21be39b41f49c"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}