diff --git a/projects/.gitignore b/projects/.gitignore deleted file mode 100644 index d1fc2b1..0000000 --- a/projects/.gitignore +++ /dev/null @@ -1,10 +0,0 @@ -.DS_Store -.ipynb_checkpoints -__pycache__ -.vscode -test.py -pseudocodes.aux -pseudocodes.log -pseudocodes.synctex.gz -pseudocodes.out -pseudocodes.toc \ No newline at end of file diff --git a/projects/LICENSE b/projects/LICENSE deleted file mode 100644 index 673d927..0000000 --- a/projects/LICENSE +++ /dev/null @@ -1,21 +0,0 @@ -MIT License - -Copyright (c) 2020 John Jim - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. diff --git a/projects/README.md b/projects/README.md deleted file mode 100644 index bb6196a..0000000 --- a/projects/README.md +++ /dev/null @@ -1,115 +0,0 @@ -## 0. 写在前面 - -本项目用于学习RL基础算法，主要面向对象为RL初学者、需要结合RL的非专业学习者，尽量做到: **注释详细**，**结构清晰**。 - -注意本项目为实战内容，建议首先掌握相关算法的一些理论基础，再来享用本项目，理论教程参考本人参与编写的[蘑菇书](https://github.com/datawhalechina/easy-rl)。 - -未来开发计划包括但不限于：多智能体算法、强化学习Python包以及强化学习图形化编程平台等等。 - -## 1. 项目说明 - -项目内容主要包含以下几个部分： -* [Jupyter Notebook](./notebooks/)：使用Notebook写的算法，有比较详细的实战引导，推荐新手食用 -* [codes](./codes/)：这些是基于Python脚本写的算法，风格比较接近实际项目的写法，推荐有一定代码基础的人阅读，下面会说明其具体的一些架构 -* [附件](./assets/)：目前包含强化学习各算法的中文伪代码 - - -[codes](./assets/)结构主要分为以下几个脚本： -* ```[algorithm_name].py```：即保存算法的脚本，例如```dqn.py```，每种算法都会有一定的基础模块，例如```Replay Buffer```、```MLP```(多层感知机)等等； -* ```task.py```: 即保存任务的脚本，基本包括基于```argparse```模块的参数，训练以及测试函数等等，其中训练函数即```train```遵循伪代码而设计，想读懂代码可从该函数入手； -* ```utils.py```：该脚本用于保存诸如存储结果以及画图的软件，在实际项目或研究中，推荐大家使用```Tensorboard```来保存结果，然后使用诸如```matplotlib```以及```seabron```来进一步画图。 -## 2. 算法列表 - -注：点击对应的名称会跳到[codes](./codes/)下对应的算法中，其他版本还请读者自行翻阅 - -| 算法名称 | 参考文献 | 作者 | 备注 | -| :-------------------------------------: | :----------------------------------------------------------: | :--------------------------------------------------: | :--: | -| [Policy Gradient](codes/PolicyGradient) | [Policy Gradient paper](https://proceedings.neurips.cc/paper/1999/file/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf) | [johnjim0816](https://github.com/johnjim0816) | | -| [Monte Carlo](codes/MonteCarlo) | | [johnjim0816](https://github.com/johnjim0816) | | -| [DQN](codes/DQN) | | [johnjim0816](https://github.com/johnjim0816) | | -| DQN-CNN | | | 待更 | -| [PER_DQN](codes/PER_DQN) | [PER DQN Paper](https://arxiv.org/abs/1511.05952) | [wangzhongren](https://github.com/wangzhongren-code) | | -| [DoubleDQN](codes/DoubleDQN) | [Double DQN Paper](https://arxiv.org/abs/1509.06461) | [johnjim0816](https://github.com/johnjim0816) | | -| [SoftQ](codes/SoftQ) | [Soft Q-learning paper](https://arxiv.org/abs/1702.08165) | [johnjim0816](https://github.com/johnjim0816) | | -| [SAC](codes/SAC) | [SAC paper](https://arxiv.org/pdf/1812.05905.pdf) | | | -| [SAC-Discrete](codes/SAC) | [SAC-Discrete paper](https://arxiv.org/pdf/1910.07207.pdf) | | | -| SAC-S | [SAC-S paper](https://arxiv.org/abs/1801.01290) | | | -| DSAC | [DSAC paper](https://paperswithcode.com/paper/addressing-value-estimation-errors-in) | | 待更 | - -## 3. 算法环境 - -算法环境说明请跳转[env](./codes/envs/README.md) - -## 4. 运行环境 - -主要依赖：Python 3.7、PyTorch 1.10.0、Gym 0.25.2。 - -### 4.1. 创建Conda环境 -```bash -conda create -n easyrl python=3.7 -conda activate easyrl # 激活环境 -``` -### 4.2. 安装Torch - -安装CPU版本： -```bash -conda install pytorch==1.10.0 torchvision==0.11.0 torchaudio==0.10.0 cpuonly -c pytorch -``` -安装CUDA版本： -```bash -conda install pytorch==1.10.0 torchvision==0.11.0 torchaudio==0.10.0 cudatoolkit=11.3 -c pytorch -c conda-forge -``` -如果安装Torch需要镜像加速的话，点击[清华镜像链接](https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/)，选择对应的操作系统，如```win-64```，然后复制链接，执行： -```bash -conda install pytorch==1.10.0 torchvision==0.11.0 torchaudio==0.10.0 cudatoolkit=11.3 -c https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/win-64/ -``` -也可以使用PiP镜像安装（仅限CUDA版本）： -```bash -pip install torch==1.10.0+cu113 torchvision==0.11.0+cu113 torchaudio==0.10.0 --extra-index-url https://download.pytorch.org/whl/cu113 -``` -### 4.3. 检验CUDA版本Torch安装 - -CPU版本Torch请忽略此步，执行如下Python脚本，如果返回True说明CUDA版本安装成功: -```python -import torch -print(torch.cuda.is_available()) -``` -### 4.4. 安装Gym - -```bash -pip install gym==0.25.2 -``` -如需安装Atari环境，则需另外安装 - -```bash -pip install gym[atari,accept-rom-license]==0.25.2 -``` - -### 4.5. 安装其他依赖 - -项目根目录下执行： -```bash -pip install -r requirements.txt -``` - -## 6.使用说明 - -对于[codes](./codes/)，`cd`到对应的算法目录下，例如`DQN`： - -```bash -python task_0.py -``` - -或者加载配置文件： - -```bash -python task0.py --yaml configs/CartPole-v1_DQN_Train.yaml -``` - -对于[Jupyter Notebook](./notebooks/)： - -* 直接运行对应的ipynb文件就行 - -## 6. 友情说明 - -推荐使用VS Code做项目，入门可参考[VSCode上手指南](https://blog.csdn.net/JohnJim0/article/details/126366454) \ No newline at end of file diff --git a/projects/assets/pseudocodes/pseudocodes.pdf b/projects/assets/pseudocodes/pseudocodes.pdf deleted file mode 100644 index 5232181..0000000 Binary files a/projects/assets/pseudocodes/pseudocodes.pdf and /dev/null differ diff --git a/projects/assets/pseudocodes/pseudocodes.tex b/projects/assets/pseudocodes/pseudocodes.tex deleted file mode 100644 index 0033ae8..0000000 --- a/projects/assets/pseudocodes/pseudocodes.tex +++ /dev/null @@ -1,359 +0,0 @@ -\documentclass[11pt]{ctexart} -\usepackage{ctex} -\usepackage{algorithm} -\usepackage{algorithmic} -\usepackage{amssymb} -\usepackage{amsmath} -\usepackage{hyperref} -% \usepackage[hidelinks]{hyperref} 去除超链接的红色框 -\usepackage{setspace} -\usepackage{titlesec} -\usepackage{float} % 调用该包能够使用[H] -% \pagestyle{plain} % 去除页眉，但是保留页脚编号，都去掉plain换empty - -% 更改脚注为圆圈 -\usepackage{pifont} -\makeatletter -\newcommand*{\circnum}[1]{% - \expandafter\@circnum\csname c@#1\endcsname -} -\newcommand*{\@circnum}[1]{% - \ifnum#1<1 % - \@ctrerr - \else - \ifnum#1>20 % - \@ctrerr - \else - \ding{\the\numexpr 171+(#1)\relax}% - \fi - \fi -} -\makeatother - -\renewcommand*{\thefootnote}{\circnum{footnote}} - -\begin{document} -\tableofcontents % 目录，注意要运行两下或者vscode保存两下才能显示 -% \singlespacing -\clearpage -\section{模版备用} -\begin{algorithm}[H] % [H]固定位置 - \floatname{algorithm}{{算法}\footnotemark[1]} - \renewcommand{\thealgorithm}{} % 去掉算法标号 - \caption{} - \begin{algorithmic}[1] % [1]显示步数 - \STATE 测试 - \end{algorithmic} -\end{algorithm} -\footnotetext[1]{脚注} -\clearpage -\section{Q learning算法} -\begin{algorithm}[H] % [H]固定位置 - \floatname{algorithm}{{Q-learning算法}\footnotemark[1]} - \renewcommand{\thealgorithm}{} % 去掉算法标号 - \caption{} - \begin{algorithmic}[1] % [1]显示步数 - \STATE 初始化Q表$Q(s,a)$为任意值，但其中$Q(s_{terminal},)=0$，即终止状态对应的Q值为0 - \FOR {回合数 = $1,M$} - \STATE 重置环境，获得初始状态$s_1$ - \FOR {时步 = $1,T$} - \STATE 根据$\varepsilon-greedy$策略采样动作$a_t$ - \STATE 环境根据$a_t$反馈奖励$r_t$和下一个状态$s_{t+1}$ - \STATE {\bfseries 更新策略：} - \STATE $Q(s_t,a_t) \leftarrow Q(s_t,a_t)+\alpha[r_t+\gamma\max _{a}Q(s_{t+1},a)-Q(s_t,a_t)]$ - \STATE 更新状态$s_{t+1} \leftarrow s_t$ - \ENDFOR - \ENDFOR - \end{algorithmic} -\end{algorithm} -\footnotetext[1]{Reinforcement Learning: An Introduction} -\clearpage -\section{Sarsa算法} -\begin{algorithm}[H] % [H]固定位置 - \floatname{algorithm}{{Sarsa算法}\footnotemark[1]} - \renewcommand{\thealgorithm}{} % 去掉算法标号 - \caption{} - \begin{algorithmic}[1] % [1]显示步数 - \STATE 初始化Q表$Q(s,a)$为任意值，但其中$Q(s_{terminal},)=0$，即终止状态对应的Q值为0 - \FOR {回合数 = $1,M$} - \STATE 重置环境，获得初始状态$s_1$ - \STATE 根据$\varepsilon-greedy$策略采样初始动作$a_1$ - \FOR {时步 = $1,t$} - \STATE 环境根据$a_t$反馈奖励$r_t$和下一个状态$s_{t+1}$ - \STATE 根据$\varepsilon-greedy$策略$s_{t+1}$和采样动作$a_{t+1}$ - \STATE {\bfseries 更新策略：} - \STATE $Q(s_t,a_t) \leftarrow Q(s_t,a_t)+\alpha[r_t+\gamma Q(s_{t+1},a_{t+1})-Q(s_t,a_t)]$ - \STATE 更新状态$s_{t+1} \leftarrow s_t$ - \STATE 更新动作$a_{t+1} \leftarrow a_t$ - \ENDFOR - \ENDFOR - \end{algorithmic} -\end{algorithm} -\footnotetext[1]{Reinforcement Learning: An Introduction} -\clearpage - -\section{DQN算法} -\begin{algorithm}[H] % [H]固定位置 - \floatname{algorithm}{{DQN算法}\footnotemark[1]} - \renewcommand{\thealgorithm}{} % 去掉算法标号 - \caption{} - \renewcommand{\algorithmicrequire}{\textbf{输入:}} - \renewcommand{\algorithmicensure}{\textbf{输出:}} - \begin{algorithmic}[1] - % \REQUIRE $n \geq 0 \vee x \neq 0$ % 输入 - % \ENSURE $y = x^n$ % 输出 - \STATE 初始化策略网络参数$\theta$ % 初始化 - \STATE 复制参数到目标网络$\hat{Q} \leftarrow Q$ - \STATE 初始化经验回放$D$ - \FOR {回合数 = $1,M$} - \STATE 重置环境，获得初始状态$s_t$ - \FOR {时步 = $1,t$} - \STATE 根据$\varepsilon-greedy$策略采样动作$a_t$ - \STATE 环境根据$a_t$反馈奖励$r_t$和下一个状态$s_{t+1}$ - \STATE 存储transition即$(s_t,a_t,r_t,s_{t+1})$到经验回放$D$中 - \STATE 更新环境状态$s_{t+1} \leftarrow s_t$ - \STATE {\bfseries 更新策略：} - \STATE 从$D$中采样一个batch的transition - \STATE 计算实际的$Q$值，即$y_{j}$\footnotemark[2] - \STATE 对损失 $L(\theta)=\left(y_{i}-Q\left(s_{i}, a_{i} ; \theta\right)\right)^{2}$关于参数$\theta$做随机梯度下降\footnotemark[3] - \ENDFOR - \STATE 每$C$个回合复制参数$\hat{Q}\leftarrow Q$\footnotemark[4]] - \ENDFOR - \end{algorithmic} -\end{algorithm} -\footnotetext[1]{Playing Atari with Deep Reinforcement Learning} -\footnotetext[2]{$y_{i}= \begin{cases}r_{i} & \text {对于终止状态} s_{i+1} \\ r_{i}+\gamma \max _{a^{\prime}} Q\left(s_{i+1}, a^{\prime} ; \theta\right) & \text {对于非终止状态} s_{i+1}\end{cases}$} -\footnotetext[3]{$\theta_i \leftarrow \theta_i - \lambda \nabla_{\theta_{i}} L_{i}\left(\theta_{i}\right)$} -\footnotetext[4]{此处也可像原论文中放到小循环中改成每$C$步，但没有每$C$个回合稳定} -\clearpage - - -\section{PER\_DQN算法} -\begin{algorithm}[H] % [H]固定位置 - \floatname{algorithm}{{PER\_DQN算法}\footnotemark[1]} - \renewcommand{\thealgorithm}{} % 去掉算法标号 - \caption{} - \renewcommand{\algorithmicrequire}{\textbf{输入:}} - \renewcommand{\algorithmicensure}{\textbf{输出:}} - \begin{algorithmic}[1] - % \REQUIRE $n \geq 0 \vee x \neq 0$ % 输入 - % \ENSURE $y = x^n$ % 输出 - \STATE 初始化策略网络参数$\theta$ % 初始化 - \STATE 复制参数到目标网络$\hat{Q} \leftarrow Q$ - \STATE 初始化经验回放$D$ - \FOR {回合数 = $1,M$} - \STATE 重置环境，获得初始状态$s_t$ - \FOR {时步 = $1,t$} - \STATE 根据$\varepsilon-greedy$策略采样动作$a_t$ - \STATE 环境根据$a_t$反馈奖励$r_t$和下一个状态$s_{t+1}$ - \STATE 存储transition即$(s_t,a_t,r_t,s_{t+1})$到经验回放$D$，并根据TD-error损失确定其优先级$p_t$ - \STATE 更新环境状态$s_{t+1} \leftarrow s_t$ - \STATE {\bfseries 更新策略：} - \STATE 按照经验回放中的优先级别，每个样本采样概率为$P(j)=p_j^\alpha / \sum_i p_i^\alpha$，从$D$中采样一个大小为batch的transition - \STATE 计算各个样本重要性采样权重 $w_j=(N \cdot P(j))^{-\beta} / \max _i w_i$ - \STATE 计算TD-error $\delta_j$ ; 并根据TD-error更新优先级$p_j$ - \STATE 计算实际的$Q$值，即$y_{j}$\footnotemark[2] - \STATE 根据重要性采样权重调整损失 $L(\theta)=\left(y_{j}-Q\left(s_{j}, a_{j} ; \theta\right)\cdot w_j \right)^{2}$，并将其关于参数$\theta$做随机梯度下降\footnotemark[3] - \ENDFOR - \STATE 每$C$个回合复制参数$\hat{Q}\leftarrow Q$\footnotemark[4]] - \ENDFOR - \end{algorithmic} -\end{algorithm} -\footnotetext[1]{Playing Atari with Deep Reinforcement Learning} -\footnotetext[2]{$y_{i}= \begin{cases}r_{i} & \text {对于终止状态} s_{i+1} \\ r_{i}+\gamma \max _{a^{\prime}} Q\left(s_{i+1}, a^{\prime} ; \theta\right) & \text {对于非终止状态} s_{i+1}\end{cases}$} -\footnotetext[3]{$\theta_i \leftarrow \theta_i - \lambda \nabla_{\theta_{i}} L_{i}\left(\theta_{i}\right)$} -\footnotetext[4]{此处也可像原论文中放到小循环中改成每$C$步，但没有每$C$个回合稳定} -\clearpage - - -\section{Policy Gradient算法} -\begin{algorithm}[H] % [H]固定位置 - \floatname{algorithm}{{REINFORCE算法：Monte-Carlo Policy Gradient}\footnotemark[1]} - \renewcommand{\thealgorithm}{} % 去掉算法标号 - \caption{} - \begin{algorithmic}[1] % [1]显示步数 - \STATE 初始化策略参数$\boldsymbol{\theta} \in \mathbb{R}^{d^{\prime}}($ e.g., to $\mathbf{0})$ - \FOR {回合数 = $1,M$} - \STATE 根据策略$\pi(\cdot \mid \cdot, \boldsymbol{\theta})$采样一个(或几个)回合的transition - \FOR {时步 = $0,1,2,...,T-1$} - \STATE 计算回报$G \leftarrow \sum_{k=t+1}^{T} \gamma^{k-t-1} R_{k}$ - \STATE 更新策略$\boldsymbol{\theta} \leftarrow {\boldsymbol{\theta}+\alpha \gamma^{t}} G \nabla \ln \pi\left(A_{t} \mid S_{t}, \boldsymbol{\theta}\right)$ - \ENDFOR - \ENDFOR - \end{algorithmic} -\end{algorithm} -\footnotetext[1]{Reinforcement Learning: An Introduction} -\clearpage -\section{Advantage Actor Critic算法} -\begin{algorithm}[H] % [H]固定位置 - \floatname{algorithm}{{Q Actor Critic算法}} - \renewcommand{\thealgorithm}{} % 去掉算法标号 - \caption{} - \begin{algorithmic}[1] % [1]显示步数 - \STATE 初始化Actor参数$\theta$和Critic参数$w$ - \FOR {回合数 = $1,M$} - \STATE 根据策略$\pi_{\theta}(a|s)$采样一个(或几个)回合的transition - \STATE {\bfseries 更新Critic参数\footnotemark[1]} - \FOR {时步 = $t+1,1$} - \STATE 计算Advantage，即$ \delta_t = r_t + \gamma Q_w(s_{t+1},a_{t+1})-Q_w(s_t,a_t)$ - \STATE $w \leftarrow w+\alpha_{w} \delta_{t} \nabla_{w} Q_w(s_t,a_t)$ - \STATE $a_t \leftarrow a_{t+1}$,$s_t \leftarrow s_{t+1}$ - \ENDFOR - \STATE 更新Actor参数$\theta \leftarrow \theta+\alpha_{\theta} Q_{w}(s, a) \nabla_{\theta} \log \pi_{\theta}(a \mid s)$ - \ENDFOR - \end{algorithmic} -\end{algorithm} -\footnotetext[1]{这里结合TD error的特性按照从$t+1$到$1$计算法Advantage更方便} - -\clearpage - -\section{PPO-Clip算法} -\begin{algorithm}[H] % [H]固定位置 - \floatname{algorithm}{{PPO-Clip算法}\footnotemark[1]\footnotemark[2]} - \renewcommand{\thealgorithm}{} % 去掉算法标号 - \caption{} - \begin{algorithmic}[1] % [1]显示步数 - \STATE 初始化策略网络(Actor)参数$\theta$和价值网络(Critic)参数$\phi$ - \STATE 初始化Clip参数$\epsilon$ - \STATE 初始化epoch数量$K$ - \STATE 初始化经验回放$D$ - \STATE 初始化总时步数$c=0$ - \FOR {回合数 = $1,2,\cdots,M$} - \STATE 重置环境，获得初始状态$s_0$ - \FOR {时步 $t = 1,2,\cdots,T$} - \STATE 计数总时步$c \leftarrow c+1$ - \STATE 根据策略$\pi_{\theta}$选择$a_t$ - \STATE 环境根据$a_t$反馈奖励$r_t$和下一个状态$s_{t+1}$ - \STATE 存储$(s_t,a_t,r_t,s_{t+1})$到经验回放$D$中 - \IF{$c$被$C$整除\footnotemark[3]} - \FOR {$k= 1,2,\cdots,K$} - \STATE 测试 - \ENDFOR - \STATE 清空经验回放$D$ - \ENDIF - \ENDFOR - \ENDFOR - \end{algorithmic} -\end{algorithm} -\footnotetext[1]{Proximal Policy Optimization Algorithms} -\footnotetext[2]{https://spinningup.openai.com/en/latest/algorithms/ppo.html} -\footnotetext[3]{\bfseries 即每$C$个时步更新策略} -\clearpage -\section{DDPG算法} -\begin{algorithm}[H] % [H]固定位置 - \floatname{algorithm}{{DDPG算法}\footnotemark[1]} - \renewcommand{\thealgorithm}{} % 去掉算法标号 - \caption{} - \begin{algorithmic}[1] % [1]显示步数 - \STATE 初始化critic网络$Q\left(s, a \mid \theta^Q\right)$和actor网络$\mu(s|\theta^{\mu})$的参数$\theta^Q$和$\theta^{\mu}$ - \STATE 初始化对应的目标网络参数，即$\theta^{Q^{\prime}} \leftarrow \theta^Q, \theta^{\mu^{\prime}} \leftarrow \theta^\mu$ - \STATE 初始化经验回放$R$ - \FOR {回合数 = $1,M$} - \STATE 选择动作$a_t=\mu\left(s_t \mid \theta^\mu\right)+\mathcal{N}_t$，$\mathcal{N}_t$为探索噪声 - \STATE 环境根据$a_t$反馈奖励$s_t$和下一个状态$s_{t+1}$ - \STATE 存储transition$(s_t,a_t,r_t,s_{t+1})$到经验回放$R$中 - \STATE 更新环境状态$s_{t+1} \leftarrow s_t$ - \STATE {\bfseries 更新策略：} - \STATE 从$R$中取出一个随机批量的$(s_i,a_i,r_i,s_{i+1})$ - \STATE 求得$y_i=r_i+\gamma Q^{\prime}\left(s_{i+1}, \mu^{\prime}\left(s_{i+1} \mid \theta^{\mu^{\prime}}\right) \mid \theta^{Q^{\prime}}\right)$ - \STATE 更新critic参数，其损失为：$L=\frac{1}{N} \sum_i\left(y_i-Q\left(s_i, a_i \mid \theta^Q\right)\right)^2$ - \STATE 更新actor参数：$\left.\left.\nabla_{\theta^\mu} J \approx \frac{1}{N} \sum_i \nabla_a Q\left(s, a \mid \theta^Q\right)\right|_{s=s_i, a=\mu\left(s_i\right)} \nabla_{\theta^\mu} \mu\left(s \mid \theta^\mu\right)\right|_{s_i}$ - \STATE 软更新目标网络：$\theta^{Q^{\prime}} \leftarrow \tau \theta^Q+(1-\tau) \theta^{Q^{\prime}}$， - $\theta^{\mu^{\prime}} \leftarrow \tau \theta^\mu+(1-\tau) \theta^{\mu^{\prime}}$ - \ENDFOR - \end{algorithmic} -\end{algorithm} -\footnotetext[1]{Continuous control with deep reinforcement learning} -\clearpage -\section{SoftQ算法} -\begin{algorithm}[H] - \floatname{algorithm}{{SoftQ算法}} - \renewcommand{\thealgorithm}{} % 去掉算法标号 - \caption{} - \begin{algorithmic}[1] - \STATE 初始化参数$\theta$和$\phi$% 初始化 - \STATE 复制参数$\bar{\theta} \leftarrow \theta, \bar{\phi} \leftarrow \phi$ - \STATE 初始化经验回放$D$ - \FOR {回合数 = $1,M$} - \FOR {时步 = $1,t$} - \STATE 根据$\mathbf{a}_{t} \leftarrow f^{\phi}\left(\xi ; \mathbf{s}_{t}\right)$采样动作，其中$\xi \sim \mathcal{N}(\mathbf{0}, \boldsymbol{I})$ - \STATE 环境根据$a_t$反馈奖励$s_t$和下一个状态$s_{t+1}$ - \STATE 存储transition即$(s_t,a_t,r_t,s_{t+1})$到经验回放$D$中 - \STATE 更新环境状态$s_{t+1} \leftarrow s_t$ - \STATE {\bfseries 更新soft Q函数参数：} - \STATE 对于每个$s^{(i)}_{t+1}$采样$\left\{\mathbf{a}^{(i, j)}\right\}_{j=0}^{M} \sim q_{\mathbf{a}^{\prime}}$ - \STATE 计算empirical soft values $V_{\mathrm{soft}}^{\theta}\left(\mathbf{s}_{t}\right)$\footnotemark[1] - \STATE 计算empirical gradient $J_{Q}(\theta)$\footnotemark[2] - \STATE 根据$J_{Q}(\theta)$使用ADAM更新参数$\theta$ - \STATE {\bfseries 更新策略：} - \STATE 对于每个$s^{(i)}_{t}$采样$\left\{\xi^{(i, j)}\right\}_{j=0}^{M} \sim \mathcal{N}(\mathbf{0}, \boldsymbol{I})$ - \STATE 计算$\mathbf{a}_{t}^{(i, j)}=f^{\phi}\left(\xi^{(i, j)}, \mathbf{s}_{t}^{(i)}\right)$ - \STATE 使用经验估计计算$\Delta f^{\phi}\left(\cdot ; \mathbf{s}_{t}\right)$\footnotemark[3] - \STATE 计算经验估计$\frac{\partial J_{\pi}\left(\phi ; \mathbf{s}_{t}\right)}{\partial \phi} \propto \mathbb{E}_{\xi}\left[\Delta f^{\phi}\left(\xi ; \mathbf{s}_{t}\right) \frac{\partial f^{\phi}\left(\xi ; \mathbf{s}_{t}\right)}{\partial \phi}\right]$，即$\hat{\nabla}_{\phi} J_{\pi}$ - \STATE 根据$\hat{\nabla}_{\phi} J_{\pi}$使用ADAM更新参数$\phi$ - \STATE - \ENDFOR - \STATE 每$C$个回合复制参数$\bar{\theta} \leftarrow \theta, \bar{\phi} \leftarrow \phi$ - \ENDFOR - \end{algorithmic} -\end{algorithm} -\footnotetext[1]{$V_{\mathrm{soft}}^{\theta}\left(\mathbf{s}_{t}\right)=\alpha \log \mathbb{E}_{q_{\mathbf{a}^{\prime}}}\left[\frac{\exp \left(\frac{1}{\alpha} Q_{\mathrm{soft}}^{\theta}\left(\mathbf{s}_{t}, \mathbf{a}^{\prime}\right)\right)}{q_{\mathbf{a}^{\prime}}\left(\mathbf{a}^{\prime}\right)}\right]$} -\footnotetext[2]{$J_{Q}(\theta)=\mathbb{E}_{\mathbf{s}_{t} \sim q_{\mathbf{s}_{t}}, \mathbf{a}_{t} \sim q_{\mathbf{a}_{t}}}\left[\frac{1}{2}\left(\hat{Q}_{\mathrm{soft}}^{\bar{\theta}}\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)-Q_{\mathrm{soft}}^{\theta}\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)\right)^{2}\right]$} -\footnotetext[3]{$\begin{aligned} \Delta f^{\phi}\left(\cdot ; \mathbf{s}_{t}\right)=& \mathbb{E}_{\mathbf{a}_{t} \sim \pi^{\phi}}\left[\left.\kappa\left(\mathbf{a}_{t}, f^{\phi}\left(\cdot ; \mathbf{s}_{t}\right)\right) \nabla_{\mathbf{a}^{\prime}} Q_{\mathrm{soft}}^{\theta}\left(\mathbf{s}_{t}, \mathbf{a}^{\prime}\right)\right|_{\mathbf{a}^{\prime}=\mathbf{a}_{t}}\right.\\ &\left.+\left.\alpha \nabla_{\mathbf{a}^{\prime}} \kappa\left(\mathbf{a}^{\prime}, f^{\phi}\left(\cdot ; \mathbf{s}_{t}\right)\right)\right|_{\mathbf{a}^{\prime}=\mathbf{a}_{t}}\right] \end{aligned}$} -\clearpage -\section{SAC-S算法} -\begin{algorithm}[H] % [H]固定位置 - \floatname{algorithm}{{SAC-S算法}\footnotemark[1]} - \renewcommand{\thealgorithm}{} % 去掉算法标号 - \caption{} - \begin{algorithmic}[1] % [1]显示步数 - \STATE 初始化参数$\psi, \bar{\psi}, \theta, \phi$ - \FOR {回合数 = $1,M$} - \FOR {时步 = $1,t$} - \STATE 根据$\boldsymbol{a}_{t} \sim \pi_{\phi}\left(\boldsymbol{a}_{t} \mid \mathbf{s}_{t}\right)$采样动作$a_t$ - \STATE 环境反馈奖励和下一个状态，$\mathbf{s}_{t+1} \sim p\left(\mathbf{s}_{t+1} \mid \mathbf{s}_{t}, \mathbf{a}_{t}\right)$ - \STATE 存储transition到经验回放中，$\mathcal{D} \leftarrow \mathcal{D} \cup\left\{\left(\mathbf{s}_{t}, \mathbf{a}_{t}, r\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right), \mathbf{s}_{t+1}\right)\right\}$ - \STATE 更新环境状态$s_{t+1} \leftarrow s_t$ - \STATE {\bfseries 更新策略：} - \STATE $\psi \leftarrow \psi-\lambda_{V} \hat{\nabla}_{\psi} J_{V}(\psi)$ - \STATE $\theta_{i} \leftarrow \theta_{i}-\lambda_{Q} \hat{\nabla}_{\theta_{i}} J_{Q}\left(\theta_{i}\right)$ for $i \in\{1,2\}$ - \STATE $\phi \leftarrow \phi-\lambda_{\pi} \hat{\nabla}_{\phi} J_{\pi}(\phi)$ - \STATE $\bar{\psi} \leftarrow \tau \psi+(1-\tau) \bar{\psi}$ - \ENDFOR - \ENDFOR - \end{algorithmic} -\end{algorithm} -\footnotetext[1]{Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor} -\clearpage -\section{SAC算法} -\begin{algorithm}[H] % [H]固定位置 - \floatname{algorithm}{{SAC算法}\footnotemark[1]} - \renewcommand{\thealgorithm}{} % 去掉算法标号 - \caption{} - \begin{algorithmic}[1] - \STATE 初始化网络参数$\theta_1,\theta_2$以及$\phi$ % 初始化 - \STATE 复制参数到目标网络$\bar{\theta_1} \leftarrow \theta_1,\bar{\theta_2} \leftarrow \theta_2,$ - \STATE 初始化经验回放$D$ - \FOR {回合数 = $1,M$} - \STATE 重置环境，获得初始状态$s_t$ - \FOR {时步 = $1,t$} - \STATE 根据$\boldsymbol{a}_{t} \sim \pi_{\phi}\left(\boldsymbol{a}_{t} \mid \mathbf{s}_{t}\right)$采样动作$a_t$ - \STATE 环境反馈奖励和下一个状态，$\mathbf{s}_{t+1} \sim p\left(\mathbf{s}_{t+1} \mid \mathbf{s}_{t}, \mathbf{a}_{t}\right)$ - \STATE 存储transition到经验回放中，$\mathcal{D} \leftarrow \mathcal{D} \cup\left\{\left(\mathbf{s}_{t}, \mathbf{a}_{t}, r\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right), \mathbf{s}_{t+1}\right)\right\}$ - \STATE 更新环境状态$s_{t+1} \leftarrow s_t$ - \STATE {\bfseries 更新策略：} - \STATE 更新$Q$函数，$\theta_{i} \leftarrow \theta_{i}-\lambda_{Q} \hat{\nabla}_{\theta_{i}} J_{Q}\left(\theta_{i}\right)$ for $i \in\{1,2\}$\footnotemark[2]\footnotemark[3] - \STATE 更新策略权重，$\phi \leftarrow \phi-\lambda_{\pi} \hat{\nabla}_{\phi} J_{\pi}(\phi)$ \footnotemark[4] - \STATE 调整temperature，$\alpha \leftarrow \alpha-\lambda \hat{\nabla}_{\alpha} J(\alpha)$ \footnotemark[5] - \STATE 更新目标网络权重，$\bar{\theta}_{i} \leftarrow \tau \theta_{i}+(1-\tau) \bar{\theta}_{i}$ for $i \in\{1,2\}$ - \ENDFOR - \ENDFOR - \end{algorithmic} -\end{algorithm} -\footnotetext[2]{Soft Actor-Critic Algorithms and Applications} -\footnotetext[2]{$J_{Q}(\theta)=\mathbb{E}_{\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right) \sim \mathcal{D}}\left[\frac{1}{2}\left(Q_{\theta}\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)-\left(r\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)+\gamma \mathbb{E}_{\mathbf{s}_{t+1} \sim p}\left[V_{\bar{\theta}}\left(\mathbf{s}_{t+1}\right)\right]\right)\right)^{2}\right]$} -\footnotetext[3]{$\hat{\nabla}_{\theta} J_{Q}(\theta)=\nabla_{\theta} Q_{\theta}\left(\mathbf{a}_{t}, \mathbf{s}_{t}\right)\left(Q_{\theta}\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)-\left(r\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)+\gamma\left(Q_{\bar{\theta}}\left(\mathbf{s}_{t+1}, \mathbf{a}_{t+1}\right)-\alpha \log \left(\pi_{\phi}\left(\mathbf{a}_{t+1} \mid \mathbf{s}_{t+1}\right)\right)\right)\right)\right.$} -\footnotetext[4]{$\hat{\nabla}_{\phi} J_{\pi}(\phi)=\nabla_{\phi} \alpha \log \left(\pi_{\phi}\left(\mathbf{a}_{t} \mid \mathbf{s}_{t}\right)\right)+\left(\nabla_{\mathbf{a}_{t}} \alpha \log \left(\pi_{\phi}\left(\mathbf{a}_{t} \mid \mathbf{s}_{t}\right)\right)-\nabla_{\mathbf{a}_{t}} Q\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)\right) \nabla_{\phi} f_{\phi}\left(\epsilon_{t} ; \mathbf{s}_{t}\right)$,$\mathbf{a}_{t}=f_{\phi}\left(\epsilon_{t} ; \mathbf{s}_{t}\right)$} -\footnotetext[5]{$J(\alpha)=\mathbb{E}_{\mathbf{a}_{t} \sim \pi_{t}}\left[-\alpha \log \pi_{t}\left(\mathbf{a}_{t} \mid \mathbf{s}_{t}\right)-\alpha \overline{\mathcal{H}}\right]$} -\clearpage -\end{document} \ No newline at end of file diff --git a/projects/codes/A2C/README.md b/projects/codes/A2C/README.md deleted file mode 100644 index 5252838..0000000 --- a/projects/codes/A2C/README.md +++ /dev/null @@ -1,7 +0,0 @@ -## 脚本描述 - -* `task0.py`：离散动作任务 - -* `task1.py`：离散动作任务，与`task0.py`唯一的区别就是Actor的激活函数是tanh而不是relu，在`CartPole-v1`上效果更好 - -* `task2.py`：连续动作任务，#TODO待调试 \ No newline at end of file diff --git a/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/config.yaml b/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/config.yaml deleted file mode 100644 index 865f5bb..0000000 --- a/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/config.yaml +++ /dev/null @@ -1,24 +0,0 @@ -general_cfg: - algo_name: A2C - device: cuda - env_name: CartPole-v1 - eval_eps: 10 - load_checkpoint: true - load_path: Train_CartPole-v1_A2C_20221030-211435 - max_steps: 200 - mode: test - save_fig: true - seed: 1 - show_fig: false - test_eps: 20 - train_eps: 1000 -algo_cfg: - actor_hidden_dim: 256 - actor_lr: 0.0003 - batch_size: 64 - buffer_size: 100000 - critic_hidden_dim: 256 - critic_lr: 0.001 - gamma: 0.99 - hidden_dim: 256 - target_update: 4 diff --git a/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/logs/log.txt b/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/logs/log.txt deleted file mode 100644 index 0ecfa0a..0000000 --- a/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/logs/log.txt +++ /dev/null @@ -1,23 +0,0 @@ -2022-10-30 21:25:53 - r - INFO: - n_states: 4, n_actions: 2 -2022-10-30 21:25:55 - r - INFO: - Start testing! -2022-10-30 21:25:55 - r - INFO: - Env: CartPole-v1, Algorithm: A2C, Device: cuda -2022-10-30 21:25:56 - r - INFO: - Episode: 1/20, Reward: 200.0, Step: 200 -2022-10-30 21:25:56 - r - INFO: - Episode: 2/20, Reward: 200.0, Step: 200 -2022-10-30 21:25:56 - r - INFO: - Episode: 3/20, Reward: 200.0, Step: 200 -2022-10-30 21:25:56 - r - INFO: - Episode: 4/20, Reward: 200.0, Step: 200 -2022-10-30 21:25:56 - r - INFO: - Episode: 5/20, Reward: 200.0, Step: 200 -2022-10-30 21:25:56 - r - INFO: - Episode: 6/20, Reward: 200.0, Step: 200 -2022-10-30 21:25:56 - r - INFO: - Episode: 7/20, Reward: 200.0, Step: 200 -2022-10-30 21:25:56 - r - INFO: - Episode: 8/20, Reward: 200.0, Step: 200 -2022-10-30 21:25:56 - r - INFO: - Episode: 9/20, Reward: 200.0, Step: 200 -2022-10-30 21:25:56 - r - INFO: - Episode: 10/20, Reward: 200.0, Step: 200 -2022-10-30 21:25:57 - r - INFO: - Episode: 11/20, Reward: 200.0, Step: 200 -2022-10-30 21:25:57 - r - INFO: - Episode: 12/20, Reward: 190.0, Step: 190 -2022-10-30 21:25:57 - r - INFO: - Episode: 13/20, Reward: 200.0, Step: 200 -2022-10-30 21:25:57 - r - INFO: - Episode: 14/20, Reward: 200.0, Step: 200 -2022-10-30 21:25:57 - r - INFO: - Episode: 15/20, Reward: 96.0, Step: 96 -2022-10-30 21:25:57 - r - INFO: - Episode: 16/20, Reward: 200.0, Step: 200 -2022-10-30 21:25:57 - r - INFO: - Episode: 17/20, Reward: 200.0, Step: 200 -2022-10-30 21:25:57 - r - INFO: - Episode: 18/20, Reward: 200.0, Step: 200 -2022-10-30 21:25:57 - r - INFO: - Episode: 19/20, Reward: 112.0, Step: 112 -2022-10-30 21:25:57 - r - INFO: - Episode: 20/20, Reward: 200.0, Step: 200 diff --git a/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/models/actor_checkpoint.pt b/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/models/actor_checkpoint.pt deleted file mode 100644 index 89d0854..0000000 Binary files a/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/models/actor_checkpoint.pt and /dev/null differ diff --git a/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/models/critic_checkpoint.pt b/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/models/critic_checkpoint.pt deleted file mode 100644 index 720f388..0000000 Binary files a/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/models/critic_checkpoint.pt and /dev/null differ diff --git a/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/results/learning_curve.png b/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/results/learning_curve.png deleted file mode 100644 index bfee34b..0000000 Binary files a/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/results/res.csv b/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/results/res.csv deleted file mode 100644 index ce0e7d1..0000000 --- a/projects/codes/A2C/Test_CartPole-v1_A2C_20221030-212553/results/res.csv +++ /dev/null @@ -1,21 +0,0 @@ -episodes,rewards,steps -0,200.0,200 -1,200.0,200 -2,200.0,200 -3,200.0,200 -4,200.0,200 -5,200.0,200 -6,200.0,200 -7,200.0,200 -8,200.0,200 -9,200.0,200 -10,200.0,200 -11,190.0,190 -12,200.0,200 -13,200.0,200 -14,96.0,96 -15,200.0,200 -16,200.0,200 -17,200.0,200 -18,112.0,112 -19,200.0,200 diff --git a/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/config.yaml b/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/config.yaml deleted file mode 100644 index 709a1e3..0000000 --- a/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/config.yaml +++ /dev/null @@ -1,25 +0,0 @@ -general_cfg: - algo_name: A2C - device: cuda - env_name: CartPole-v1 - eval_eps: 10 - eval_per_episode: 5 - load_checkpoint: true - load_path: Train_CartPole-v1_A2C_20221031-232138 - max_steps: 200 - mode: test - save_fig: true - seed: 1 - show_fig: false - test_eps: 20 - train_eps: 1000 -algo_cfg: - actor_hidden_dim: 256 - actor_lr: 0.0003 - batch_size: 64 - buffer_size: 100000 - critic_hidden_dim: 256 - critic_lr: 0.001 - gamma: 0.99 - hidden_dim: 256 - target_update: 4 diff --git a/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/logs/log.txt b/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/logs/log.txt deleted file mode 100644 index d84edb2..0000000 --- a/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/logs/log.txt +++ /dev/null @@ -1,28 +0,0 @@ -2022-10-31 23:33:16 - r - INFO: - n_states: 4, n_actions: 2 -2022-10-31 23:33:16 - r - INFO: - Actor model name: ActorSoftmaxTanh -2022-10-31 23:33:16 - r - INFO: - Critic model name: Critic -2022-10-31 23:33:16 - r - INFO: - ACMemory memory name: PGReplay -2022-10-31 23:33:16 - r - INFO: - agent name: A2C -2022-10-31 23:33:17 - r - INFO: - Start testing! -2022-10-31 23:33:17 - r - INFO: - Env: CartPole-v1, Algorithm: A2C, Device: cuda -2022-10-31 23:33:18 - r - INFO: - Episode: 1/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:18 - r - INFO: - Episode: 2/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:18 - r - INFO: - Episode: 3/20, Reward: 186.0, Step: 186 -2022-10-31 23:33:18 - r - INFO: - Episode: 4/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:18 - r - INFO: - Episode: 5/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:19 - r - INFO: - Episode: 6/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:19 - r - INFO: - Episode: 7/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:19 - r - INFO: - Episode: 8/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:19 - r - INFO: - Episode: 9/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:19 - r - INFO: - Episode: 10/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:19 - r - INFO: - Episode: 11/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:19 - r - INFO: - Episode: 12/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:19 - r - INFO: - Episode: 13/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:19 - r - INFO: - Episode: 14/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:19 - r - INFO: - Episode: 15/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:19 - r - INFO: - Episode: 16/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:19 - r - INFO: - Episode: 17/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:19 - r - INFO: - Episode: 18/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:19 - r - INFO: - Episode: 19/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:20 - r - INFO: - Episode: 20/20, Reward: 200.0, Step: 200 -2022-10-31 23:33:20 - r - INFO: - Finish testing! diff --git a/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/models/actor_checkpoint.pt b/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/models/actor_checkpoint.pt deleted file mode 100644 index 05bd7b6..0000000 Binary files a/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/models/actor_checkpoint.pt and /dev/null differ diff --git a/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/models/critic_checkpoint.pt b/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/models/critic_checkpoint.pt deleted file mode 100644 index 720f388..0000000 Binary files a/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/models/critic_checkpoint.pt and /dev/null differ diff --git a/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/results/learning_curve.png b/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/results/learning_curve.png deleted file mode 100644 index 33274af..0000000 Binary files a/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/results/res.csv b/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/results/res.csv deleted file mode 100644 index 571b1e6..0000000 --- a/projects/codes/A2C/Test_CartPole-v1_A2C_20221031-233316/results/res.csv +++ /dev/null @@ -1,21 +0,0 @@ -episodes,rewards,steps -0,200.0,200 -1,200.0,200 -2,186.0,186 -3,200.0,200 -4,200.0,200 -5,200.0,200 -6,200.0,200 -7,200.0,200 -8,200.0,200 -9,200.0,200 -10,200.0,200 -11,200.0,200 -12,200.0,200 -13,200.0,200 -14,200.0,200 -15,200.0,200 -16,200.0,200 -17,200.0,200 -18,200.0,200 -19,200.0,200 diff --git a/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/config.yaml b/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/config.yaml deleted file mode 100644 index 7dde5b7..0000000 --- a/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/config.yaml +++ /dev/null @@ -1,23 +0,0 @@ -general_cfg: - algo_name: A2C - device: cuda - env_name: CartPole-v1 - eval_eps: 10 - load_checkpoint: false - load_path: tasks - max_steps: 200 - mode: train - save_fig: true - seed: 1 - show_fig: false - test_eps: 20 - train_eps: 1000 -algo_cfg: - actor_hidden_dim: 256 - actor_lr: 0.0003 - batch_size: 64 - buffer_size: 100000 - critic_hidden_dim: 256 - critic_lr: 0.001 - gamma: 0.99 - hidden_dim: 256 diff --git a/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/logs/log.txt b/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/logs/log.txt deleted file mode 100644 index b13b335..0000000 --- a/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/logs/log.txt +++ /dev/null @@ -1,1066 +0,0 @@ -2022-10-30 21:14:35 - r - INFO: - n_states: 4, n_actions: 2 -2022-10-30 21:14:35 - r - INFO: - Start training! -2022-10-30 21:14:35 - r - INFO: - Env: CartPole-v1, Algorithm: A2C, Device: cuda -2022-10-30 21:14:37 - r - INFO: - Episode: 1/1000, Reward: 25.0, Step: 25 -2022-10-30 21:14:38 - r - INFO: - Current episode 1 has the best eval reward: 29.2 -2022-10-30 21:14:38 - r - INFO: - Episode: 2/1000, Reward: 13.0, Step: 13 -2022-10-30 21:14:38 - r - INFO: - Episode: 3/1000, Reward: 58.0, Step: 58 -2022-10-30 21:14:38 - r - INFO: - Episode: 4/1000, Reward: 10.0, Step: 10 -2022-10-30 21:14:38 - r - INFO: - Episode: 5/1000, Reward: 39.0, Step: 39 -2022-10-30 21:14:38 - r - INFO: - Episode: 6/1000, Reward: 39.0, Step: 39 -2022-10-30 21:14:38 - r - INFO: - Episode: 7/1000, Reward: 25.0, Step: 25 -2022-10-30 21:14:39 - r - INFO: - Episode: 8/1000, Reward: 22.0, Step: 22 -2022-10-30 21:14:39 - r - INFO: - Episode: 9/1000, Reward: 21.0, Step: 21 -2022-10-30 21:14:39 - r - INFO: - Episode: 10/1000, Reward: 27.0, Step: 27 -2022-10-30 21:14:39 - r - INFO: - Episode: 11/1000, Reward: 35.0, Step: 35 -2022-10-30 21:14:40 - r - INFO: - Episode: 12/1000, Reward: 26.0, Step: 26 -2022-10-30 21:14:40 - r - INFO: - Episode: 13/1000, Reward: 38.0, Step: 38 -2022-10-30 21:14:40 - r - INFO: - Episode: 14/1000, Reward: 29.0, Step: 29 -2022-10-30 21:14:40 - r - INFO: - Episode: 15/1000, Reward: 50.0, Step: 50 -2022-10-30 21:14:40 - r - INFO: - Episode: 16/1000, Reward: 20.0, Step: 20 -2022-10-30 21:14:40 - r - INFO: - Episode: 17/1000, Reward: 52.0, Step: 52 -2022-10-30 21:14:41 - r - INFO: - Current episode 17 has the best eval reward: 32.9 -2022-10-30 21:14:41 - r - INFO: - Episode: 18/1000, Reward: 12.0, Step: 12 -2022-10-30 21:14:41 - r - INFO: - Episode: 19/1000, Reward: 20.0, Step: 20 -2022-10-30 21:14:41 - r - INFO: - Episode: 20/1000, Reward: 38.0, Step: 38 -2022-10-30 21:14:41 - r - INFO: - Current episode 20 has the best eval reward: 38.9 -2022-10-30 21:14:41 - r - INFO: - Episode: 21/1000, Reward: 22.0, Step: 22 -2022-10-30 21:14:41 - r - INFO: - Episode: 22/1000, Reward: 36.0, Step: 36 -2022-10-30 21:14:42 - r - INFO: - Episode: 23/1000, Reward: 20.0, Step: 20 -2022-10-30 21:14:42 - r - INFO: - Episode: 24/1000, Reward: 35.0, Step: 35 -2022-10-30 21:14:42 - r - INFO: - Episode: 25/1000, Reward: 90.0, Step: 90 -2022-10-30 21:14:42 - r - INFO: - Episode: 26/1000, Reward: 29.0, Step: 29 -2022-10-30 21:14:42 - r - INFO: - Episode: 27/1000, Reward: 16.0, Step: 16 -2022-10-30 21:14:43 - r - INFO: - Episode: 28/1000, Reward: 25.0, Step: 25 -2022-10-30 21:14:43 - r - INFO: - Episode: 29/1000, Reward: 46.0, Step: 46 -2022-10-30 21:14:43 - r - INFO: - Episode: 30/1000, Reward: 33.0, Step: 33 -2022-10-30 21:14:43 - r - INFO: - Episode: 31/1000, Reward: 11.0, Step: 11 -2022-10-30 21:14:43 - r - INFO: - Episode: 32/1000, Reward: 27.0, Step: 27 -2022-10-30 21:14:44 - r - INFO: - Episode: 33/1000, Reward: 32.0, Step: 32 -2022-10-30 21:14:44 - r - INFO: - Current episode 33 has the best eval reward: 39.2 -2022-10-30 21:14:44 - r - INFO: - Episode: 34/1000, Reward: 21.0, Step: 21 -2022-10-30 21:14:44 - r - INFO: - Episode: 35/1000, Reward: 11.0, Step: 11 -2022-10-30 21:14:44 - r - INFO: - Episode: 36/1000, Reward: 21.0, Step: 21 -2022-10-30 21:14:44 - r - INFO: - Episode: 37/1000, Reward: 51.0, Step: 51 -2022-10-30 21:14:44 - r - INFO: - Episode: 38/1000, Reward: 29.0, Step: 29 -2022-10-30 21:14:45 - r - INFO: - Current episode 38 has the best eval reward: 41.7 -2022-10-30 21:14:45 - r - INFO: - Episode: 39/1000, Reward: 50.0, Step: 50 -2022-10-30 21:14:45 - r - INFO: - Current episode 39 has the best eval reward: 48.5 -2022-10-30 21:14:45 - r - INFO: - Episode: 40/1000, Reward: 19.0, Step: 19 -2022-10-30 21:14:45 - r - INFO: - Episode: 41/1000, Reward: 41.0, Step: 41 -2022-10-30 21:14:45 - r - INFO: - Episode: 42/1000, Reward: 28.0, Step: 28 -2022-10-30 21:14:46 - r - INFO: - Episode: 43/1000, Reward: 71.0, Step: 71 -2022-10-30 21:14:46 - r - INFO: - Episode: 44/1000, Reward: 45.0, Step: 45 -2022-10-30 21:14:46 - r - INFO: - Episode: 45/1000, Reward: 42.0, Step: 42 -2022-10-30 21:14:46 - r - INFO: - Current episode 45 has the best eval reward: 49.6 -2022-10-30 21:14:46 - r - INFO: - Episode: 46/1000, Reward: 39.0, Step: 39 -2022-10-30 21:14:47 - r - INFO: - Episode: 47/1000, Reward: 21.0, Step: 21 -2022-10-30 21:14:47 - r - INFO: - Episode: 48/1000, Reward: 14.0, Step: 14 -2022-10-30 21:14:47 - r - INFO: - Episode: 49/1000, Reward: 23.0, Step: 23 -2022-10-30 21:14:47 - r - INFO: - Episode: 50/1000, Reward: 21.0, Step: 21 -2022-10-30 21:14:47 - r - INFO: - Episode: 51/1000, Reward: 34.0, Step: 34 -2022-10-30 21:14:48 - r - INFO: - Episode: 52/1000, Reward: 14.0, Step: 14 -2022-10-30 21:14:48 - r - INFO: - Episode: 53/1000, Reward: 41.0, Step: 41 -2022-10-30 21:14:48 - r - INFO: - Episode: 54/1000, Reward: 99.0, Step: 99 -2022-10-30 21:14:48 - r - INFO: - Episode: 55/1000, Reward: 21.0, Step: 21 -2022-10-30 21:14:49 - r - INFO: - Episode: 56/1000, Reward: 52.0, Step: 52 -2022-10-30 21:14:49 - r - INFO: - Episode: 57/1000, Reward: 34.0, Step: 34 -2022-10-30 21:14:49 - r - INFO: - Episode: 58/1000, Reward: 73.0, Step: 73 -2022-10-30 21:14:49 - r - INFO: - Episode: 59/1000, Reward: 21.0, Step: 21 -2022-10-30 21:14:49 - r - INFO: - Episode: 60/1000, Reward: 27.0, Step: 27 -2022-10-30 21:14:50 - r - INFO: - Episode: 61/1000, Reward: 51.0, Step: 51 -2022-10-30 21:14:50 - r - INFO: - Episode: 62/1000, Reward: 46.0, Step: 46 -2022-10-30 21:14:50 - r - INFO: - Episode: 63/1000, Reward: 21.0, Step: 21 -2022-10-30 21:14:50 - r - INFO: - Episode: 64/1000, Reward: 20.0, Step: 20 -2022-10-30 21:14:51 - r - INFO: - Episode: 65/1000, Reward: 44.0, Step: 44 -2022-10-30 21:14:51 - r - INFO: - Episode: 66/1000, Reward: 16.0, Step: 16 -2022-10-30 21:14:51 - r - INFO: - Episode: 67/1000, Reward: 39.0, Step: 39 -2022-10-30 21:14:51 - r - INFO: - Episode: 68/1000, Reward: 30.0, Step: 30 -2022-10-30 21:14:51 - r - INFO: - Episode: 69/1000, Reward: 37.0, Step: 37 -2022-10-30 21:14:52 - r - INFO: - Episode: 70/1000, Reward: 20.0, Step: 20 -2022-10-30 21:14:52 - r - INFO: - Episode: 71/1000, Reward: 21.0, Step: 21 -2022-10-30 21:14:52 - r - INFO: - Episode: 72/1000, Reward: 13.0, Step: 13 -2022-10-30 21:14:52 - r - INFO: - Episode: 73/1000, Reward: 65.0, Step: 65 -2022-10-30 21:14:53 - r - INFO: - Episode: 74/1000, Reward: 45.0, Step: 45 -2022-10-30 21:14:53 - r - INFO: - Episode: 75/1000, Reward: 45.0, Step: 45 -2022-10-30 21:14:53 - r - INFO: - Episode: 76/1000, Reward: 46.0, Step: 46 -2022-10-30 21:14:53 - r - INFO: - Episode: 77/1000, Reward: 13.0, Step: 13 -2022-10-30 21:14:53 - r - INFO: - Episode: 78/1000, Reward: 33.0, Step: 33 -2022-10-30 21:14:54 - r - INFO: - Episode: 79/1000, Reward: 30.0, Step: 30 -2022-10-30 21:14:54 - r - INFO: - Episode: 80/1000, Reward: 52.0, Step: 52 -2022-10-30 21:14:54 - r - INFO: - Episode: 81/1000, Reward: 27.0, Step: 27 -2022-10-30 21:14:54 - r - INFO: - Episode: 82/1000, Reward: 30.0, Step: 30 -2022-10-30 21:14:55 - r - INFO: - Episode: 83/1000, Reward: 47.0, Step: 47 -2022-10-30 21:14:55 - r - INFO: - Episode: 84/1000, Reward: 56.0, Step: 56 -2022-10-30 21:14:55 - r - INFO: - Episode: 85/1000, Reward: 19.0, Step: 19 -2022-10-30 21:14:55 - r - INFO: - Episode: 86/1000, Reward: 33.0, Step: 33 -2022-10-30 21:14:56 - r - INFO: - Episode: 87/1000, Reward: 25.0, Step: 25 -2022-10-30 21:14:56 - r - INFO: - Episode: 88/1000, Reward: 41.0, Step: 41 -2022-10-30 21:14:56 - r - INFO: - Episode: 89/1000, Reward: 20.0, Step: 20 -2022-10-30 21:14:56 - r - INFO: - Episode: 90/1000, Reward: 58.0, Step: 58 -2022-10-30 21:14:56 - r - INFO: - Episode: 91/1000, Reward: 35.0, Step: 35 -2022-10-30 21:14:57 - r - INFO: - Episode: 92/1000, Reward: 23.0, Step: 23 -2022-10-30 21:14:57 - r - INFO: - Episode: 93/1000, Reward: 12.0, Step: 12 -2022-10-30 21:14:57 - r - INFO: - Episode: 94/1000, Reward: 20.0, Step: 20 -2022-10-30 21:14:57 - r - INFO: - Episode: 95/1000, Reward: 10.0, Step: 10 -2022-10-30 21:14:57 - r - INFO: - Episode: 96/1000, Reward: 49.0, Step: 49 -2022-10-30 21:14:58 - r - INFO: - Episode: 97/1000, Reward: 29.0, Step: 29 -2022-10-30 21:14:58 - r - INFO: - Episode: 98/1000, Reward: 35.0, Step: 35 -2022-10-30 21:14:58 - r - INFO: - Episode: 99/1000, Reward: 36.0, Step: 36 -2022-10-30 21:14:58 - r - INFO: - Current episode 99 has the best eval reward: 53.4 -2022-10-30 21:14:58 - r - INFO: - Episode: 100/1000, Reward: 36.0, Step: 36 -2022-10-30 21:14:59 - r - INFO: - Episode: 101/1000, Reward: 16.0, Step: 16 -2022-10-30 21:14:59 - r - INFO: - Episode: 102/1000, Reward: 36.0, Step: 36 -2022-10-30 21:14:59 - r - INFO: - Current episode 102 has the best eval reward: 70.3 -2022-10-30 21:14:59 - r - INFO: - Episode: 103/1000, Reward: 30.0, Step: 30 -2022-10-30 21:15:00 - r - INFO: - Episode: 104/1000, Reward: 76.0, Step: 76 -2022-10-30 21:15:00 - r - INFO: - Episode: 105/1000, Reward: 52.0, Step: 52 -2022-10-30 21:15:00 - r - INFO: - Episode: 106/1000, Reward: 39.0, Step: 39 -2022-10-30 21:15:00 - r - INFO: - Episode: 107/1000, Reward: 52.0, Step: 52 -2022-10-30 21:15:01 - r - INFO: - Episode: 108/1000, Reward: 69.0, Step: 69 -2022-10-30 21:15:01 - r - INFO: - Episode: 109/1000, Reward: 27.0, Step: 27 -2022-10-30 21:15:01 - r - INFO: - Episode: 110/1000, Reward: 14.0, Step: 14 -2022-10-30 21:15:01 - r - INFO: - Episode: 111/1000, Reward: 28.0, Step: 28 -2022-10-30 21:15:01 - r - INFO: - Episode: 112/1000, Reward: 12.0, Step: 12 -2022-10-30 21:15:02 - r - INFO: - Episode: 113/1000, Reward: 26.0, Step: 26 -2022-10-30 21:15:03 - r - INFO: - Episode: 114/1000, Reward: 50.0, Step: 50 -2022-10-30 21:15:03 - r - INFO: - Episode: 115/1000, Reward: 25.0, Step: 25 -2022-10-30 21:15:03 - r - INFO: - Episode: 116/1000, Reward: 53.0, Step: 53 -2022-10-30 21:15:03 - r - INFO: - Episode: 117/1000, Reward: 19.0, Step: 19 -2022-10-30 21:15:04 - r - INFO: - Episode: 118/1000, Reward: 33.0, Step: 33 -2022-10-30 21:15:04 - r - INFO: - Episode: 119/1000, Reward: 34.0, Step: 34 -2022-10-30 21:15:04 - r - INFO: - Episode: 120/1000, Reward: 41.0, Step: 41 -2022-10-30 21:15:04 - r - INFO: - Episode: 121/1000, Reward: 25.0, Step: 25 -2022-10-30 21:15:05 - r - INFO: - Episode: 122/1000, Reward: 18.0, Step: 18 -2022-10-30 21:15:05 - r - INFO: - Episode: 123/1000, Reward: 114.0, Step: 114 -2022-10-30 21:15:05 - r - INFO: - Episode: 124/1000, Reward: 25.0, Step: 25 -2022-10-30 21:15:05 - r - INFO: - Episode: 125/1000, Reward: 46.0, Step: 46 -2022-10-30 21:15:06 - r - INFO: - Episode: 126/1000, Reward: 22.0, Step: 22 -2022-10-30 21:15:06 - r - INFO: - Episode: 127/1000, Reward: 71.0, Step: 71 -2022-10-30 21:15:06 - r - INFO: - Episode: 128/1000, Reward: 30.0, Step: 30 -2022-10-30 21:15:07 - r - INFO: - Episode: 129/1000, Reward: 130.0, Step: 130 -2022-10-30 21:15:07 - r - INFO: - Episode: 130/1000, Reward: 65.0, Step: 65 -2022-10-30 21:15:07 - r - INFO: - Episode: 131/1000, Reward: 55.0, Step: 55 -2022-10-30 21:15:07 - r - INFO: - Episode: 132/1000, Reward: 37.0, Step: 37 -2022-10-30 21:15:08 - r - INFO: - Episode: 133/1000, Reward: 46.0, Step: 46 -2022-10-30 21:15:08 - r - INFO: - Episode: 134/1000, Reward: 65.0, Step: 65 -2022-10-30 21:15:08 - r - INFO: - Episode: 135/1000, Reward: 31.0, Step: 31 -2022-10-30 21:15:08 - r - INFO: - Episode: 136/1000, Reward: 33.0, Step: 33 -2022-10-30 21:15:09 - r - INFO: - Episode: 137/1000, Reward: 39.0, Step: 39 -2022-10-30 21:15:09 - r - INFO: - Episode: 138/1000, Reward: 73.0, Step: 73 -2022-10-30 21:15:09 - r - INFO: - Episode: 139/1000, Reward: 78.0, Step: 78 -2022-10-30 21:15:10 - r - INFO: - Episode: 140/1000, Reward: 36.0, Step: 36 -2022-10-30 21:15:10 - r - INFO: - Episode: 141/1000, Reward: 56.0, Step: 56 -2022-10-30 21:15:10 - r - INFO: - Episode: 142/1000, Reward: 12.0, Step: 12 -2022-10-30 21:15:10 - r - INFO: - Episode: 143/1000, Reward: 36.0, Step: 36 -2022-10-30 21:15:11 - r - INFO: - Episode: 144/1000, Reward: 13.0, Step: 13 -2022-10-30 21:15:11 - r - INFO: - Episode: 145/1000, Reward: 85.0, Step: 85 -2022-10-30 21:15:11 - r - INFO: - Episode: 146/1000, Reward: 34.0, Step: 34 -2022-10-30 21:15:11 - r - INFO: - Episode: 147/1000, Reward: 16.0, Step: 16 -2022-10-30 21:15:12 - r - INFO: - Episode: 148/1000, Reward: 68.0, Step: 68 -2022-10-30 21:15:12 - r - INFO: - Episode: 149/1000, Reward: 94.0, Step: 94 -2022-10-30 21:15:12 - r - INFO: - Episode: 150/1000, Reward: 17.0, Step: 17 -2022-10-30 21:15:13 - r - INFO: - Episode: 151/1000, Reward: 64.0, Step: 64 -2022-10-30 21:15:13 - r - INFO: - Episode: 152/1000, Reward: 33.0, Step: 33 -2022-10-30 21:15:13 - r - INFO: - Episode: 153/1000, Reward: 63.0, Step: 63 -2022-10-30 21:15:13 - r - INFO: - Episode: 154/1000, Reward: 39.0, Step: 39 -2022-10-30 21:15:14 - r - INFO: - Episode: 155/1000, Reward: 72.0, Step: 72 -2022-10-30 21:15:14 - r - INFO: - Episode: 156/1000, Reward: 39.0, Step: 39 -2022-10-30 21:15:14 - r - INFO: - Episode: 157/1000, Reward: 37.0, Step: 37 -2022-10-30 21:15:14 - r - INFO: - Episode: 158/1000, Reward: 18.0, Step: 18 -2022-10-30 21:15:15 - r - INFO: - Episode: 159/1000, Reward: 55.0, Step: 55 -2022-10-30 21:15:15 - r - INFO: - Episode: 160/1000, Reward: 21.0, Step: 21 -2022-10-30 21:15:15 - r - INFO: - Episode: 161/1000, Reward: 54.0, Step: 54 -2022-10-30 21:15:15 - r - INFO: - Episode: 162/1000, Reward: 46.0, Step: 46 -2022-10-30 21:15:16 - r - INFO: - Episode: 163/1000, Reward: 21.0, Step: 21 -2022-10-30 21:15:16 - r - INFO: - Episode: 164/1000, Reward: 26.0, Step: 26 -2022-10-30 21:15:16 - r - INFO: - Episode: 165/1000, Reward: 70.0, Step: 70 -2022-10-30 21:15:17 - r - INFO: - Episode: 166/1000, Reward: 20.0, Step: 20 -2022-10-30 21:15:17 - r - INFO: - Episode: 167/1000, Reward: 41.0, Step: 41 -2022-10-30 21:15:17 - r - INFO: - Episode: 168/1000, Reward: 77.0, Step: 77 -2022-10-30 21:15:17 - r - INFO: - Episode: 169/1000, Reward: 13.0, Step: 13 -2022-10-30 21:15:18 - r - INFO: - Episode: 170/1000, Reward: 66.0, Step: 66 -2022-10-30 21:15:18 - r - INFO: - Episode: 171/1000, Reward: 72.0, Step: 72 -2022-10-30 21:15:18 - r - INFO: - Episode: 172/1000, Reward: 28.0, Step: 28 -2022-10-30 21:15:19 - r - INFO: - Episode: 173/1000, Reward: 68.0, Step: 68 -2022-10-30 21:15:19 - r - INFO: - Episode: 174/1000, Reward: 124.0, Step: 124 -2022-10-30 21:15:19 - r - INFO: - Episode: 175/1000, Reward: 41.0, Step: 41 -2022-10-30 21:15:20 - r - INFO: - Episode: 176/1000, Reward: 54.0, Step: 54 -2022-10-30 21:15:20 - r - INFO: - Episode: 177/1000, Reward: 33.0, Step: 33 -2022-10-30 21:15:20 - r - INFO: - Episode: 178/1000, Reward: 92.0, Step: 92 -2022-10-30 21:15:20 - r - INFO: - Episode: 179/1000, Reward: 23.0, Step: 23 -2022-10-30 21:15:21 - r - INFO: - Episode: 180/1000, Reward: 76.0, Step: 76 -2022-10-30 21:15:21 - r - INFO: - Episode: 181/1000, Reward: 47.0, Step: 47 -2022-10-30 21:15:22 - r - INFO: - Episode: 182/1000, Reward: 89.0, Step: 89 -2022-10-30 21:15:22 - r - INFO: - Episode: 183/1000, Reward: 84.0, Step: 84 -2022-10-30 21:15:22 - r - INFO: - Episode: 184/1000, Reward: 75.0, Step: 75 -2022-10-30 21:15:23 - r - INFO: - Episode: 185/1000, Reward: 64.0, Step: 64 -2022-10-30 21:15:23 - r - INFO: - Episode: 186/1000, Reward: 35.0, Step: 35 -2022-10-30 21:15:23 - r - INFO: - Episode: 187/1000, Reward: 44.0, Step: 44 -2022-10-30 21:15:24 - r - INFO: - Episode: 188/1000, Reward: 46.0, Step: 46 -2022-10-30 21:15:24 - r - INFO: - Episode: 189/1000, Reward: 67.0, Step: 67 -2022-10-30 21:15:25 - r - INFO: - Episode: 190/1000, Reward: 82.0, Step: 82 -2022-10-30 21:15:25 - r - INFO: - Episode: 191/1000, Reward: 55.0, Step: 55 -2022-10-30 21:15:25 - r - INFO: - Episode: 192/1000, Reward: 26.0, Step: 26 -2022-10-30 21:15:26 - r - INFO: - Episode: 193/1000, Reward: 116.0, Step: 116 -2022-10-30 21:15:26 - r - INFO: - Episode: 194/1000, Reward: 116.0, Step: 116 -2022-10-30 21:15:26 - r - INFO: - Episode: 195/1000, Reward: 119.0, Step: 119 -2022-10-30 21:15:27 - r - INFO: - Episode: 196/1000, Reward: 50.0, Step: 50 -2022-10-30 21:15:27 - r - INFO: - Episode: 197/1000, Reward: 43.0, Step: 43 -2022-10-30 21:15:27 - r - INFO: - Episode: 198/1000, Reward: 47.0, Step: 47 -2022-10-30 21:15:28 - r - INFO: - Episode: 199/1000, Reward: 71.0, Step: 71 -2022-10-30 21:15:28 - r - INFO: - Episode: 200/1000, Reward: 53.0, Step: 53 -2022-10-30 21:15:28 - r - INFO: - Current episode 200 has the best eval reward: 86.0 -2022-10-30 21:15:29 - r - INFO: - Episode: 201/1000, Reward: 137.0, Step: 137 -2022-10-30 21:15:29 - r - INFO: - Episode: 202/1000, Reward: 82.0, Step: 82 -2022-10-30 21:15:30 - r - INFO: - Episode: 203/1000, Reward: 120.0, Step: 120 -2022-10-30 21:15:30 - r - INFO: - Current episode 203 has the best eval reward: 92.8 -2022-10-30 21:15:30 - r - INFO: - Episode: 204/1000, Reward: 69.0, Step: 69 -2022-10-30 21:15:31 - r - INFO: - Episode: 205/1000, Reward: 55.0, Step: 55 -2022-10-30 21:15:31 - r - INFO: - Episode: 206/1000, Reward: 62.0, Step: 62 -2022-10-30 21:15:31 - r - INFO: - Episode: 207/1000, Reward: 64.0, Step: 64 -2022-10-30 21:15:32 - r - INFO: - Episode: 208/1000, Reward: 49.0, Step: 49 -2022-10-30 21:15:32 - r - INFO: - Episode: 209/1000, Reward: 32.0, Step: 32 -2022-10-30 21:15:33 - r - INFO: - Episode: 210/1000, Reward: 42.0, Step: 42 -2022-10-30 21:15:33 - r - INFO: - Episode: 211/1000, Reward: 50.0, Step: 50 -2022-10-30 21:15:33 - r - INFO: - Episode: 212/1000, Reward: 93.0, Step: 93 -2022-10-30 21:15:34 - r - INFO: - Episode: 213/1000, Reward: 60.0, Step: 60 -2022-10-30 21:15:34 - r - INFO: - Episode: 214/1000, Reward: 54.0, Step: 54 -2022-10-30 21:15:35 - r - INFO: - Episode: 215/1000, Reward: 68.0, Step: 68 -2022-10-30 21:15:35 - r - INFO: - Episode: 216/1000, Reward: 84.0, Step: 84 -2022-10-30 21:15:35 - r - INFO: - Current episode 216 has the best eval reward: 94.6 -2022-10-30 21:15:36 - r - INFO: - Episode: 217/1000, Reward: 55.0, Step: 55 -2022-10-30 21:15:36 - r - INFO: - Episode: 218/1000, Reward: 70.0, Step: 70 -2022-10-30 21:15:37 - r - INFO: - Episode: 219/1000, Reward: 115.0, Step: 115 -2022-10-30 21:15:37 - r - INFO: - Episode: 220/1000, Reward: 149.0, Step: 149 -2022-10-30 21:15:38 - r - INFO: - Episode: 221/1000, Reward: 68.0, Step: 68 -2022-10-30 21:15:38 - r - INFO: - Episode: 222/1000, Reward: 50.0, Step: 50 -2022-10-30 21:15:38 - r - INFO: - Current episode 222 has the best eval reward: 95.5 -2022-10-30 21:15:39 - r - INFO: - Episode: 223/1000, Reward: 56.0, Step: 56 -2022-10-30 21:15:39 - r - INFO: - Episode: 224/1000, Reward: 61.0, Step: 61 -2022-10-30 21:15:39 - r - INFO: - Episode: 225/1000, Reward: 117.0, Step: 117 -2022-10-30 21:15:40 - r - INFO: - Episode: 226/1000, Reward: 66.0, Step: 66 -2022-10-30 21:15:41 - r - INFO: - Episode: 227/1000, Reward: 127.0, Step: 127 -2022-10-30 21:15:41 - r - INFO: - Episode: 228/1000, Reward: 66.0, Step: 66 -2022-10-30 21:15:42 - r - INFO: - Episode: 229/1000, Reward: 48.0, Step: 48 -2022-10-30 21:15:42 - r - INFO: - Episode: 230/1000, Reward: 36.0, Step: 36 -2022-10-30 21:15:42 - r - INFO: - Episode: 231/1000, Reward: 79.0, Step: 79 -2022-10-30 21:15:43 - r - INFO: - Episode: 232/1000, Reward: 49.0, Step: 49 -2022-10-30 21:15:43 - r - INFO: - Episode: 233/1000, Reward: 55.0, Step: 55 -2022-10-30 21:15:43 - r - INFO: - Episode: 234/1000, Reward: 41.0, Step: 41 -2022-10-30 21:15:43 - r - INFO: - Episode: 235/1000, Reward: 20.0, Step: 20 -2022-10-30 21:15:44 - r - INFO: - Episode: 236/1000, Reward: 40.0, Step: 40 -2022-10-30 21:15:44 - r - INFO: - Episode: 237/1000, Reward: 120.0, Step: 120 -2022-10-30 21:15:44 - r - INFO: - Episode: 238/1000, Reward: 27.0, Step: 27 -2022-10-30 21:15:45 - r - INFO: - Episode: 239/1000, Reward: 51.0, Step: 51 -2022-10-30 21:15:45 - r - INFO: - Episode: 240/1000, Reward: 35.0, Step: 35 -2022-10-30 21:15:45 - r - INFO: - Episode: 241/1000, Reward: 43.0, Step: 43 -2022-10-30 21:15:46 - r - INFO: - Episode: 242/1000, Reward: 54.0, Step: 54 -2022-10-30 21:15:46 - r - INFO: - Episode: 243/1000, Reward: 52.0, Step: 52 -2022-10-30 21:15:46 - r - INFO: - Episode: 244/1000, Reward: 47.0, Step: 47 -2022-10-30 21:15:46 - r - INFO: - Episode: 245/1000, Reward: 63.0, Step: 63 -2022-10-30 21:15:47 - r - INFO: - Episode: 246/1000, Reward: 29.0, Step: 29 -2022-10-30 21:15:47 - r - INFO: - Episode: 247/1000, Reward: 36.0, Step: 36 -2022-10-30 21:15:47 - r - INFO: - Episode: 248/1000, Reward: 58.0, Step: 58 -2022-10-30 21:15:48 - r - INFO: - Episode: 249/1000, Reward: 63.0, Step: 63 -2022-10-30 21:15:48 - r - INFO: - Episode: 250/1000, Reward: 49.0, Step: 49 -2022-10-30 21:15:48 - r - INFO: - Episode: 251/1000, Reward: 70.0, Step: 70 -2022-10-30 21:15:49 - r - INFO: - Episode: 252/1000, Reward: 114.0, Step: 114 -2022-10-30 21:15:49 - r - INFO: - Episode: 253/1000, Reward: 62.0, Step: 62 -2022-10-30 21:15:50 - r - INFO: - Episode: 254/1000, Reward: 73.0, Step: 73 -2022-10-30 21:15:50 - r - INFO: - Current episode 254 has the best eval reward: 96.7 -2022-10-30 21:15:50 - r - INFO: - Episode: 255/1000, Reward: 62.0, Step: 62 -2022-10-30 21:15:51 - r - INFO: - Episode: 256/1000, Reward: 61.0, Step: 61 -2022-10-30 21:15:51 - r - INFO: - Episode: 257/1000, Reward: 115.0, Step: 115 -2022-10-30 21:15:52 - r - INFO: - Episode: 258/1000, Reward: 50.0, Step: 50 -2022-10-30 21:15:52 - r - INFO: - Episode: 259/1000, Reward: 128.0, Step: 128 -2022-10-30 21:15:53 - r - INFO: - Current episode 259 has the best eval reward: 104.8 -2022-10-30 21:15:53 - r - INFO: - Episode: 260/1000, Reward: 200.0, Step: 200 -2022-10-30 21:15:53 - r - INFO: - Episode: 261/1000, Reward: 75.0, Step: 75 -2022-10-30 21:15:54 - r - INFO: - Episode: 262/1000, Reward: 64.0, Step: 64 -2022-10-30 21:15:54 - r - INFO: - Episode: 263/1000, Reward: 33.0, Step: 33 -2022-10-30 21:15:55 - r - INFO: - Episode: 264/1000, Reward: 90.0, Step: 90 -2022-10-30 21:15:55 - r - INFO: - Current episode 264 has the best eval reward: 107.6 -2022-10-30 21:15:56 - r - INFO: - Episode: 265/1000, Reward: 117.0, Step: 117 -2022-10-30 21:15:56 - r - INFO: - Current episode 265 has the best eval reward: 119.4 -2022-10-30 21:15:56 - r - INFO: - Episode: 266/1000, Reward: 60.0, Step: 60 -2022-10-30 21:15:57 - r - INFO: - Episode: 267/1000, Reward: 177.0, Step: 177 -2022-10-30 21:15:57 - r - INFO: - Episode: 268/1000, Reward: 39.0, Step: 39 -2022-10-30 21:15:58 - r - INFO: - Episode: 269/1000, Reward: 40.0, Step: 40 -2022-10-30 21:15:58 - r - INFO: - Episode: 270/1000, Reward: 109.0, Step: 109 -2022-10-30 21:15:59 - r - INFO: - Episode: 271/1000, Reward: 100.0, Step: 100 -2022-10-30 21:16:00 - r - INFO: - Episode: 272/1000, Reward: 99.0, Step: 99 -2022-10-30 21:16:00 - r - INFO: - Episode: 273/1000, Reward: 136.0, Step: 136 -2022-10-30 21:16:01 - r - INFO: - Episode: 274/1000, Reward: 62.0, Step: 62 -2022-10-30 21:16:01 - r - INFO: - Episode: 275/1000, Reward: 100.0, Step: 100 -2022-10-30 21:16:02 - r - INFO: - Current episode 275 has the best eval reward: 120.1 -2022-10-30 21:16:02 - r - INFO: - Episode: 276/1000, Reward: 73.0, Step: 73 -2022-10-30 21:16:03 - r - INFO: - Episode: 277/1000, Reward: 166.0, Step: 166 -2022-10-30 21:16:03 - r - INFO: - Episode: 278/1000, Reward: 74.0, Step: 74 -2022-10-30 21:16:04 - r - INFO: - Current episode 278 has the best eval reward: 121.8 -2022-10-30 21:16:04 - r - INFO: - Episode: 279/1000, Reward: 126.0, Step: 126 -2022-10-30 21:16:05 - r - INFO: - Episode: 280/1000, Reward: 111.0, Step: 111 -2022-10-30 21:16:06 - r - INFO: - Episode: 281/1000, Reward: 198.0, Step: 198 -2022-10-30 21:16:07 - r - INFO: - Episode: 282/1000, Reward: 106.0, Step: 106 -2022-10-30 21:16:07 - r - INFO: - Episode: 283/1000, Reward: 80.0, Step: 80 -2022-10-30 21:16:08 - r - INFO: - Episode: 284/1000, Reward: 74.0, Step: 74 -2022-10-30 21:16:08 - r - INFO: - Episode: 285/1000, Reward: 114.0, Step: 114 -2022-10-30 21:16:09 - r - INFO: - Episode: 286/1000, Reward: 69.0, Step: 69 -2022-10-30 21:16:09 - r - INFO: - Episode: 287/1000, Reward: 98.0, Step: 98 -2022-10-30 21:16:10 - r - INFO: - Episode: 288/1000, Reward: 63.0, Step: 63 -2022-10-30 21:16:10 - r - INFO: - Episode: 289/1000, Reward: 61.0, Step: 61 -2022-10-30 21:16:11 - r - INFO: - Episode: 290/1000, Reward: 49.0, Step: 49 -2022-10-30 21:16:11 - r - INFO: - Episode: 291/1000, Reward: 89.0, Step: 89 -2022-10-30 21:16:12 - r - INFO: - Episode: 292/1000, Reward: 114.0, Step: 114 -2022-10-30 21:16:13 - r - INFO: - Episode: 293/1000, Reward: 103.0, Step: 103 -2022-10-30 21:16:13 - r - INFO: - Episode: 294/1000, Reward: 103.0, Step: 103 -2022-10-30 21:16:14 - r - INFO: - Episode: 295/1000, Reward: 93.0, Step: 93 -2022-10-30 21:16:14 - r - INFO: - Episode: 296/1000, Reward: 137.0, Step: 137 -2022-10-30 21:16:15 - r - INFO: - Episode: 297/1000, Reward: 97.0, Step: 97 -2022-10-30 21:16:16 - r - INFO: - Episode: 298/1000, Reward: 124.0, Step: 124 -2022-10-30 21:16:16 - r - INFO: - Episode: 299/1000, Reward: 147.0, Step: 147 -2022-10-30 21:16:17 - r - INFO: - Episode: 300/1000, Reward: 125.0, Step: 125 -2022-10-30 21:16:18 - r - INFO: - Episode: 301/1000, Reward: 105.0, Step: 105 -2022-10-30 21:16:18 - r - INFO: - Current episode 301 has the best eval reward: 148.8 -2022-10-30 21:16:18 - r - INFO: - Episode: 302/1000, Reward: 113.0, Step: 113 -2022-10-30 21:16:19 - r - INFO: - Current episode 302 has the best eval reward: 150.8 -2022-10-30 21:16:19 - r - INFO: - Episode: 303/1000, Reward: 120.0, Step: 120 -2022-10-30 21:16:20 - r - INFO: - Episode: 304/1000, Reward: 159.0, Step: 159 -2022-10-30 21:16:21 - r - INFO: - Episode: 305/1000, Reward: 190.0, Step: 190 -2022-10-30 21:16:22 - r - INFO: - Current episode 305 has the best eval reward: 183.4 -2022-10-30 21:16:22 - r - INFO: - Episode: 306/1000, Reward: 119.0, Step: 119 -2022-10-30 21:16:23 - r - INFO: - Episode: 307/1000, Reward: 200.0, Step: 200 -2022-10-30 21:16:24 - r - INFO: - Episode: 308/1000, Reward: 148.0, Step: 148 -2022-10-30 21:16:25 - r - INFO: - Episode: 309/1000, Reward: 200.0, Step: 200 -2022-10-30 21:16:26 - r - INFO: - Episode: 310/1000, Reward: 79.0, Step: 79 -2022-10-30 21:16:27 - r - INFO: - Episode: 311/1000, Reward: 115.0, Step: 115 -2022-10-30 21:16:28 - r - INFO: - Episode: 312/1000, Reward: 147.0, Step: 147 -2022-10-30 21:16:29 - r - INFO: - Episode: 313/1000, Reward: 112.0, Step: 112 -2022-10-30 21:16:29 - r - INFO: - Episode: 314/1000, Reward: 125.0, Step: 125 -2022-10-30 21:16:30 - r - INFO: - Episode: 315/1000, Reward: 184.0, Step: 184 -2022-10-30 21:16:31 - r - INFO: - Episode: 316/1000, Reward: 193.0, Step: 193 -2022-10-30 21:16:32 - r - INFO: - Episode: 317/1000, Reward: 117.0, Step: 117 -2022-10-30 21:16:33 - r - INFO: - Episode: 318/1000, Reward: 153.0, Step: 153 -2022-10-30 21:16:34 - r - INFO: - Episode: 319/1000, Reward: 125.0, Step: 125 -2022-10-30 21:16:35 - r - INFO: - Episode: 320/1000, Reward: 184.0, Step: 184 -2022-10-30 21:16:36 - r - INFO: - Episode: 321/1000, Reward: 173.0, Step: 173 -2022-10-30 21:16:36 - r - INFO: - Episode: 322/1000, Reward: 117.0, Step: 117 -2022-10-30 21:16:37 - r - INFO: - Episode: 323/1000, Reward: 47.0, Step: 47 -2022-10-30 21:16:38 - r - INFO: - Episode: 324/1000, Reward: 107.0, Step: 107 -2022-10-30 21:16:38 - r - INFO: - Episode: 325/1000, Reward: 104.0, Step: 104 -2022-10-30 21:16:39 - r - INFO: - Episode: 326/1000, Reward: 114.0, Step: 114 -2022-10-30 21:16:39 - r - INFO: - Episode: 327/1000, Reward: 90.0, Step: 90 -2022-10-30 21:16:40 - r - INFO: - Episode: 328/1000, Reward: 112.0, Step: 112 -2022-10-30 21:16:41 - r - INFO: - Episode: 329/1000, Reward: 70.0, Step: 70 -2022-10-30 21:16:41 - r - INFO: - Episode: 330/1000, Reward: 74.0, Step: 74 -2022-10-30 21:16:42 - r - INFO: - Episode: 331/1000, Reward: 159.0, Step: 159 -2022-10-30 21:16:42 - r - INFO: - Episode: 332/1000, Reward: 39.0, Step: 39 -2022-10-30 21:16:43 - r - INFO: - Episode: 333/1000, Reward: 129.0, Step: 129 -2022-10-30 21:16:44 - r - INFO: - Episode: 334/1000, Reward: 50.0, Step: 50 -2022-10-30 21:16:44 - r - INFO: - Episode: 335/1000, Reward: 74.0, Step: 74 -2022-10-30 21:16:44 - r - INFO: - Episode: 336/1000, Reward: 31.0, Step: 31 -2022-10-30 21:16:45 - r - INFO: - Episode: 337/1000, Reward: 57.0, Step: 57 -2022-10-30 21:16:45 - r - INFO: - Episode: 338/1000, Reward: 71.0, Step: 71 -2022-10-30 21:16:46 - r - INFO: - Episode: 339/1000, Reward: 43.0, Step: 43 -2022-10-30 21:16:46 - r - INFO: - Episode: 340/1000, Reward: 41.0, Step: 41 -2022-10-30 21:16:46 - r - INFO: - Episode: 341/1000, Reward: 64.0, Step: 64 -2022-10-30 21:16:47 - r - INFO: - Episode: 342/1000, Reward: 38.0, Step: 38 -2022-10-30 21:16:47 - r - INFO: - Episode: 343/1000, Reward: 45.0, Step: 45 -2022-10-30 21:16:48 - r - INFO: - Episode: 344/1000, Reward: 120.0, Step: 120 -2022-10-30 21:16:48 - r - INFO: - Episode: 345/1000, Reward: 40.0, Step: 40 -2022-10-30 21:16:48 - r - INFO: - Episode: 346/1000, Reward: 46.0, Step: 46 -2022-10-30 21:16:48 - r - INFO: - Episode: 347/1000, Reward: 57.0, Step: 57 -2022-10-30 21:16:49 - r - INFO: - Episode: 348/1000, Reward: 29.0, Step: 29 -2022-10-30 21:16:49 - r - INFO: - Episode: 349/1000, Reward: 29.0, Step: 29 -2022-10-30 21:16:49 - r - INFO: - Episode: 350/1000, Reward: 50.0, Step: 50 -2022-10-30 21:16:50 - r - INFO: - Episode: 351/1000, Reward: 38.0, Step: 38 -2022-10-30 21:16:50 - r - INFO: - Episode: 352/1000, Reward: 51.0, Step: 51 -2022-10-30 21:16:50 - r - INFO: - Episode: 353/1000, Reward: 49.0, Step: 49 -2022-10-30 21:16:50 - r - INFO: - Episode: 354/1000, Reward: 30.0, Step: 30 -2022-10-30 21:16:51 - r - INFO: - Episode: 355/1000, Reward: 40.0, Step: 40 -2022-10-30 21:16:51 - r - INFO: - Episode: 356/1000, Reward: 45.0, Step: 45 -2022-10-30 21:16:51 - r - INFO: - Episode: 357/1000, Reward: 68.0, Step: 68 -2022-10-30 21:16:52 - r - INFO: - Episode: 358/1000, Reward: 27.0, Step: 27 -2022-10-30 21:16:52 - r - INFO: - Episode: 359/1000, Reward: 18.0, Step: 18 -2022-10-30 21:16:52 - r - INFO: - Episode: 360/1000, Reward: 26.0, Step: 26 -2022-10-30 21:16:52 - r - INFO: - Episode: 361/1000, Reward: 15.0, Step: 15 -2022-10-30 21:16:52 - r - INFO: - Episode: 362/1000, Reward: 65.0, Step: 65 -2022-10-30 21:16:53 - r - INFO: - Episode: 363/1000, Reward: 38.0, Step: 38 -2022-10-30 21:16:53 - r - INFO: - Episode: 364/1000, Reward: 41.0, Step: 41 -2022-10-30 21:16:53 - r - INFO: - Episode: 365/1000, Reward: 61.0, Step: 61 -2022-10-30 21:16:54 - r - INFO: - Episode: 366/1000, Reward: 113.0, Step: 113 -2022-10-30 21:16:54 - r - INFO: - Episode: 367/1000, Reward: 39.0, Step: 39 -2022-10-30 21:16:54 - r - INFO: - Episode: 368/1000, Reward: 60.0, Step: 60 -2022-10-30 21:16:55 - r - INFO: - Episode: 369/1000, Reward: 134.0, Step: 134 -2022-10-30 21:16:56 - r - INFO: - Episode: 370/1000, Reward: 122.0, Step: 122 -2022-10-30 21:16:56 - r - INFO: - Episode: 371/1000, Reward: 34.0, Step: 34 -2022-10-30 21:16:57 - r - INFO: - Episode: 372/1000, Reward: 129.0, Step: 129 -2022-10-30 21:16:57 - r - INFO: - Episode: 373/1000, Reward: 40.0, Step: 40 -2022-10-30 21:16:58 - r - INFO: - Episode: 374/1000, Reward: 128.0, Step: 128 -2022-10-30 21:16:59 - r - INFO: - Episode: 375/1000, Reward: 200.0, Step: 200 -2022-10-30 21:17:00 - r - INFO: - Episode: 376/1000, Reward: 108.0, Step: 108 -2022-10-30 21:17:01 - r - INFO: - Episode: 377/1000, Reward: 108.0, Step: 108 -2022-10-30 21:17:02 - r - INFO: - Episode: 378/1000, Reward: 151.0, Step: 151 -2022-10-30 21:17:03 - r - INFO: - Episode: 379/1000, Reward: 79.0, Step: 79 -2022-10-30 21:17:03 - r - INFO: - Episode: 380/1000, Reward: 105.0, Step: 105 -2022-10-30 21:17:04 - r - INFO: - Episode: 381/1000, Reward: 87.0, Step: 87 -2022-10-30 21:17:05 - r - INFO: - Episode: 382/1000, Reward: 94.0, Step: 94 -2022-10-30 21:17:06 - r - INFO: - Episode: 383/1000, Reward: 112.0, Step: 112 -2022-10-30 21:17:07 - r - INFO: - Episode: 384/1000, Reward: 200.0, Step: 200 -2022-10-30 21:17:08 - r - INFO: - Episode: 385/1000, Reward: 184.0, Step: 184 -2022-10-30 21:17:08 - r - INFO: - Episode: 386/1000, Reward: 124.0, Step: 124 -2022-10-30 21:17:09 - r - INFO: - Episode: 387/1000, Reward: 200.0, Step: 200 -2022-10-30 21:17:11 - r - INFO: - Episode: 388/1000, Reward: 200.0, Step: 200 -2022-10-30 21:17:12 - r - INFO: - Episode: 389/1000, Reward: 109.0, Step: 109 -2022-10-30 21:17:12 - r - INFO: - Episode: 390/1000, Reward: 88.0, Step: 88 -2022-10-30 21:17:13 - r - INFO: - Episode: 391/1000, Reward: 104.0, Step: 104 -2022-10-30 21:17:14 - r - INFO: - Episode: 392/1000, Reward: 200.0, Step: 200 -2022-10-30 21:17:15 - r - INFO: - Episode: 393/1000, Reward: 84.0, Step: 84 -2022-10-30 21:17:16 - r - INFO: - Episode: 394/1000, Reward: 187.0, Step: 187 -2022-10-30 21:17:17 - r - INFO: - Episode: 395/1000, Reward: 182.0, Step: 182 -2022-10-30 21:17:18 - r - INFO: - Episode: 396/1000, Reward: 148.0, Step: 148 -2022-10-30 21:17:19 - r - INFO: - Episode: 397/1000, Reward: 86.0, Step: 86 -2022-10-30 21:17:20 - r - INFO: - Episode: 398/1000, Reward: 200.0, Step: 200 -2022-10-30 21:17:21 - r - INFO: - Episode: 399/1000, Reward: 199.0, Step: 199 -2022-10-30 21:17:22 - r - INFO: - Episode: 400/1000, Reward: 200.0, Step: 200 -2022-10-30 21:17:23 - r - INFO: - Episode: 401/1000, Reward: 92.0, Step: 92 -2022-10-30 21:17:23 - r - INFO: - Episode: 402/1000, Reward: 112.0, Step: 112 -2022-10-30 21:17:24 - r - INFO: - Episode: 403/1000, Reward: 86.0, Step: 86 -2022-10-30 21:17:25 - r - INFO: - Episode: 404/1000, Reward: 114.0, Step: 114 -2022-10-30 21:17:26 - r - INFO: - Episode: 405/1000, Reward: 90.0, Step: 90 -2022-10-30 21:17:26 - r - INFO: - Episode: 406/1000, Reward: 101.0, Step: 101 -2022-10-30 21:17:27 - r - INFO: - Episode: 407/1000, Reward: 111.0, Step: 111 -2022-10-30 21:17:28 - r - INFO: - Episode: 408/1000, Reward: 107.0, Step: 107 -2022-10-30 21:17:28 - r - INFO: - Episode: 409/1000, Reward: 120.0, Step: 120 -2022-10-30 21:17:29 - r - INFO: - Episode: 410/1000, Reward: 114.0, Step: 114 -2022-10-30 21:17:30 - r - INFO: - Episode: 411/1000, Reward: 97.0, Step: 97 -2022-10-30 21:17:30 - r - INFO: - Episode: 412/1000, Reward: 95.0, Step: 95 -2022-10-30 21:17:31 - r - INFO: - Episode: 413/1000, Reward: 126.0, Step: 126 -2022-10-30 21:17:32 - r - INFO: - Episode: 414/1000, Reward: 111.0, Step: 111 -2022-10-30 21:17:33 - r - INFO: - Episode: 415/1000, Reward: 120.0, Step: 120 -2022-10-30 21:17:33 - r - INFO: - Episode: 416/1000, Reward: 178.0, Step: 178 -2022-10-30 21:17:34 - r - INFO: - Episode: 417/1000, Reward: 97.0, Step: 97 -2022-10-30 21:17:35 - r - INFO: - Episode: 418/1000, Reward: 144.0, Step: 144 -2022-10-30 21:17:36 - r - INFO: - Episode: 419/1000, Reward: 200.0, Step: 200 -2022-10-30 21:17:36 - r - INFO: - Episode: 420/1000, Reward: 190.0, Step: 190 -2022-10-30 21:17:37 - r - INFO: - Episode: 421/1000, Reward: 29.0, Step: 29 -2022-10-30 21:17:38 - r - INFO: - Episode: 422/1000, Reward: 200.0, Step: 200 -2022-10-30 21:17:38 - r - INFO: - Episode: 423/1000, Reward: 116.0, Step: 116 -2022-10-30 21:17:39 - r - INFO: - Episode: 424/1000, Reward: 200.0, Step: 200 -2022-10-30 21:17:40 - r - INFO: - Episode: 425/1000, Reward: 107.0, Step: 107 -2022-10-30 21:17:41 - r - INFO: - Episode: 426/1000, Reward: 128.0, Step: 128 -2022-10-30 21:17:41 - r - INFO: - Episode: 427/1000, Reward: 164.0, Step: 164 -2022-10-30 21:17:42 - r - INFO: - Episode: 428/1000, Reward: 30.0, Step: 30 -2022-10-30 21:17:42 - r - INFO: - Episode: 429/1000, Reward: 122.0, Step: 122 -2022-10-30 21:17:43 - r - INFO: - Episode: 430/1000, Reward: 110.0, Step: 110 -2022-10-30 21:17:44 - r - INFO: - Episode: 431/1000, Reward: 105.0, Step: 105 -2022-10-30 21:17:44 - r - INFO: - Episode: 432/1000, Reward: 137.0, Step: 137 -2022-10-30 21:17:45 - r - INFO: - Episode: 433/1000, Reward: 110.0, Step: 110 -2022-10-30 21:17:45 - r - INFO: - Episode: 434/1000, Reward: 111.0, Step: 111 -2022-10-30 21:17:46 - r - INFO: - Episode: 435/1000, Reward: 33.0, Step: 33 -2022-10-30 21:17:46 - r - INFO: - Episode: 436/1000, Reward: 100.0, Step: 100 -2022-10-30 21:17:47 - r - INFO: - Episode: 437/1000, Reward: 131.0, Step: 131 -2022-10-30 21:17:48 - r - INFO: - Episode: 438/1000, Reward: 99.0, Step: 99 -2022-10-30 21:17:48 - r - INFO: - Episode: 439/1000, Reward: 118.0, Step: 118 -2022-10-30 21:17:49 - r - INFO: - Episode: 440/1000, Reward: 98.0, Step: 98 -2022-10-30 21:17:49 - r - INFO: - Episode: 441/1000, Reward: 119.0, Step: 119 -2022-10-30 21:17:50 - r - INFO: - Episode: 442/1000, Reward: 41.0, Step: 41 -2022-10-30 21:17:50 - r - INFO: - Episode: 443/1000, Reward: 107.0, Step: 107 -2022-10-30 21:17:51 - r - INFO: - Episode: 444/1000, Reward: 41.0, Step: 41 -2022-10-30 21:17:52 - r - INFO: - Episode: 445/1000, Reward: 113.0, Step: 113 -2022-10-30 21:17:52 - r - INFO: - Episode: 446/1000, Reward: 113.0, Step: 113 -2022-10-30 21:17:53 - r - INFO: - Episode: 447/1000, Reward: 117.0, Step: 117 -2022-10-30 21:17:54 - r - INFO: - Episode: 448/1000, Reward: 140.0, Step: 140 -2022-10-30 21:17:54 - r - INFO: - Episode: 449/1000, Reward: 133.0, Step: 133 -2022-10-30 21:17:55 - r - INFO: - Episode: 450/1000, Reward: 108.0, Step: 108 -2022-10-30 21:17:56 - r - INFO: - Episode: 451/1000, Reward: 117.0, Step: 117 -2022-10-30 21:17:57 - r - INFO: - Episode: 452/1000, Reward: 40.0, Step: 40 -2022-10-30 21:17:57 - r - INFO: - Episode: 453/1000, Reward: 108.0, Step: 108 -2022-10-30 21:17:58 - r - INFO: - Episode: 454/1000, Reward: 140.0, Step: 140 -2022-10-30 21:17:59 - r - INFO: - Episode: 455/1000, Reward: 133.0, Step: 133 -2022-10-30 21:18:00 - r - INFO: - Episode: 456/1000, Reward: 115.0, Step: 115 -2022-10-30 21:18:00 - r - INFO: - Episode: 457/1000, Reward: 30.0, Step: 30 -2022-10-30 21:18:01 - r - INFO: - Episode: 458/1000, Reward: 119.0, Step: 119 -2022-10-30 21:18:02 - r - INFO: - Episode: 459/1000, Reward: 160.0, Step: 160 -2022-10-30 21:18:02 - r - INFO: - Episode: 460/1000, Reward: 125.0, Step: 125 -2022-10-30 21:18:03 - r - INFO: - Episode: 461/1000, Reward: 161.0, Step: 161 -2022-10-30 21:18:04 - r - INFO: - Episode: 462/1000, Reward: 139.0, Step: 139 -2022-10-30 21:18:05 - r - INFO: - Episode: 463/1000, Reward: 190.0, Step: 190 -2022-10-30 21:18:06 - r - INFO: - Episode: 464/1000, Reward: 149.0, Step: 149 -2022-10-30 21:18:07 - r - INFO: - Episode: 465/1000, Reward: 173.0, Step: 173 -2022-10-30 21:18:08 - r - INFO: - Current episode 465 has the best eval reward: 187.6 -2022-10-30 21:18:08 - r - INFO: - Episode: 466/1000, Reward: 165.0, Step: 165 -2022-10-30 21:18:09 - r - INFO: - Episode: 467/1000, Reward: 82.0, Step: 82 -2022-10-30 21:18:10 - r - INFO: - Episode: 468/1000, Reward: 197.0, Step: 197 -2022-10-30 21:18:11 - r - INFO: - Current episode 468 has the best eval reward: 195.0 -2022-10-30 21:18:12 - r - INFO: - Episode: 469/1000, Reward: 200.0, Step: 200 -2022-10-30 21:18:13 - r - INFO: - Episode: 470/1000, Reward: 200.0, Step: 200 -2022-10-30 21:18:14 - r - INFO: - Current episode 470 has the best eval reward: 199.4 -2022-10-30 21:18:14 - r - INFO: - Episode: 471/1000, Reward: 200.0, Step: 200 -2022-10-30 21:18:16 - r - INFO: - Episode: 472/1000, Reward: 182.0, Step: 182 -2022-10-30 21:18:17 - r - INFO: - Episode: 473/1000, Reward: 118.0, Step: 118 -2022-10-30 21:18:18 - r - INFO: - Episode: 474/1000, Reward: 200.0, Step: 200 -2022-10-30 21:18:19 - r - INFO: - Episode: 475/1000, Reward: 200.0, Step: 200 -2022-10-30 21:18:20 - r - INFO: - Episode: 476/1000, Reward: 93.0, Step: 93 -2022-10-30 21:18:21 - r - INFO: - Episode: 477/1000, Reward: 200.0, Step: 200 -2022-10-30 21:18:23 - r - INFO: - Episode: 478/1000, Reward: 200.0, Step: 200 -2022-10-30 21:18:24 - r - INFO: - Episode: 479/1000, Reward: 167.0, Step: 167 -2022-10-30 21:18:25 - r - INFO: - Episode: 480/1000, Reward: 200.0, Step: 200 -2022-10-30 21:18:26 - r - INFO: - Episode: 481/1000, Reward: 200.0, Step: 200 -2022-10-30 21:18:27 - r - INFO: - Episode: 482/1000, Reward: 200.0, Step: 200 -2022-10-30 21:18:28 - r - INFO: - Episode: 483/1000, Reward: 190.0, Step: 190 -2022-10-30 21:18:29 - r - INFO: - Episode: 484/1000, Reward: 86.0, Step: 86 -2022-10-30 21:18:30 - r - INFO: - Episode: 485/1000, Reward: 166.0, Step: 166 -2022-10-30 21:18:31 - r - INFO: - Episode: 486/1000, Reward: 200.0, Step: 200 -2022-10-30 21:18:32 - r - INFO: - Episode: 487/1000, Reward: 200.0, Step: 200 -2022-10-30 21:18:33 - r - INFO: - Episode: 488/1000, Reward: 172.0, Step: 172 -2022-10-30 21:18:34 - r - INFO: - Episode: 489/1000, Reward: 200.0, Step: 200 -2022-10-30 21:18:35 - r - INFO: - Episode: 490/1000, Reward: 102.0, Step: 102 -2022-10-30 21:18:36 - r - INFO: - Episode: 491/1000, Reward: 194.0, Step: 194 -2022-10-30 21:18:37 - r - INFO: - Episode: 492/1000, Reward: 200.0, Step: 200 -2022-10-30 21:18:38 - r - INFO: - Episode: 493/1000, Reward: 179.0, Step: 179 -2022-10-30 21:18:39 - r - INFO: - Episode: 494/1000, Reward: 187.0, Step: 187 -2022-10-30 21:18:40 - r - INFO: - Episode: 495/1000, Reward: 200.0, Step: 200 -2022-10-30 21:18:41 - r - INFO: - Episode: 496/1000, Reward: 89.0, Step: 89 -2022-10-30 21:18:41 - r - INFO: - Episode: 497/1000, Reward: 169.0, Step: 169 -2022-10-30 21:18:42 - r - INFO: - Episode: 498/1000, Reward: 28.0, Step: 28 -2022-10-30 21:18:43 - r - INFO: - Episode: 499/1000, Reward: 160.0, Step: 160 -2022-10-30 21:18:44 - r - INFO: - Episode: 500/1000, Reward: 140.0, Step: 140 -2022-10-30 21:18:44 - r - INFO: - Episode: 501/1000, Reward: 37.0, Step: 37 -2022-10-30 21:18:45 - r - INFO: - Episode: 502/1000, Reward: 32.0, Step: 32 -2022-10-30 21:18:45 - r - INFO: - Episode: 503/1000, Reward: 129.0, Step: 129 -2022-10-30 21:18:46 - r - INFO: - Episode: 504/1000, Reward: 22.0, Step: 22 -2022-10-30 21:18:46 - r - INFO: - Episode: 505/1000, Reward: 124.0, Step: 124 -2022-10-30 21:18:46 - r - INFO: - Episode: 506/1000, Reward: 24.0, Step: 24 -2022-10-30 21:18:47 - r - INFO: - Episode: 507/1000, Reward: 115.0, Step: 115 -2022-10-30 21:18:47 - r - INFO: - Episode: 508/1000, Reward: 24.0, Step: 24 -2022-10-30 21:18:48 - r - INFO: - Episode: 509/1000, Reward: 38.0, Step: 38 -2022-10-30 21:18:49 - r - INFO: - Episode: 510/1000, Reward: 24.0, Step: 24 -2022-10-30 21:18:49 - r - INFO: - Episode: 511/1000, Reward: 23.0, Step: 23 -2022-10-30 21:18:49 - r - INFO: - Episode: 512/1000, Reward: 125.0, Step: 125 -2022-10-30 21:18:49 - r - INFO: - Episode: 513/1000, Reward: 22.0, Step: 22 -2022-10-30 21:18:50 - r - INFO: - Episode: 514/1000, Reward: 24.0, Step: 24 -2022-10-30 21:18:50 - r - INFO: - Episode: 515/1000, Reward: 20.0, Step: 20 -2022-10-30 21:18:50 - r - INFO: - Episode: 516/1000, Reward: 25.0, Step: 25 -2022-10-30 21:18:50 - r - INFO: - Episode: 517/1000, Reward: 31.0, Step: 31 -2022-10-30 21:18:50 - r - INFO: - Episode: 518/1000, Reward: 23.0, Step: 23 -2022-10-30 21:18:51 - r - INFO: - Episode: 519/1000, Reward: 30.0, Step: 30 -2022-10-30 21:18:51 - r - INFO: - Episode: 520/1000, Reward: 101.0, Step: 101 -2022-10-30 21:18:51 - r - INFO: - Episode: 521/1000, Reward: 25.0, Step: 25 -2022-10-30 21:18:52 - r - INFO: - Episode: 522/1000, Reward: 22.0, Step: 22 -2022-10-30 21:18:52 - r - INFO: - Episode: 523/1000, Reward: 20.0, Step: 20 -2022-10-30 21:18:52 - r - INFO: - Episode: 524/1000, Reward: 16.0, Step: 16 -2022-10-30 21:18:53 - r - INFO: - Episode: 525/1000, Reward: 104.0, Step: 104 -2022-10-30 21:18:53 - r - INFO: - Episode: 526/1000, Reward: 17.0, Step: 17 -2022-10-30 21:18:53 - r - INFO: - Episode: 527/1000, Reward: 108.0, Step: 108 -2022-10-30 21:18:53 - r - INFO: - Episode: 528/1000, Reward: 121.0, Step: 121 -2022-10-30 21:18:54 - r - INFO: - Episode: 529/1000, Reward: 29.0, Step: 29 -2022-10-30 21:18:54 - r - INFO: - Episode: 530/1000, Reward: 29.0, Step: 29 -2022-10-30 21:18:54 - r - INFO: - Episode: 531/1000, Reward: 43.0, Step: 43 -2022-10-30 21:18:55 - r - INFO: - Episode: 532/1000, Reward: 105.0, Step: 105 -2022-10-30 21:18:55 - r - INFO: - Episode: 533/1000, Reward: 130.0, Step: 130 -2022-10-30 21:18:55 - r - INFO: - Episode: 534/1000, Reward: 30.0, Step: 30 -2022-10-30 21:18:56 - r - INFO: - Episode: 535/1000, Reward: 31.0, Step: 31 -2022-10-30 21:18:56 - r - INFO: - Episode: 536/1000, Reward: 30.0, Step: 30 -2022-10-30 21:18:56 - r - INFO: - Episode: 537/1000, Reward: 37.0, Step: 37 -2022-10-30 21:18:57 - r - INFO: - Episode: 538/1000, Reward: 115.0, Step: 115 -2022-10-30 21:18:58 - r - INFO: - Episode: 539/1000, Reward: 110.0, Step: 110 -2022-10-30 21:18:58 - r - INFO: - Episode: 540/1000, Reward: 112.0, Step: 112 -2022-10-30 21:18:59 - r - INFO: - Episode: 541/1000, Reward: 33.0, Step: 33 -2022-10-30 21:18:59 - r - INFO: - Episode: 542/1000, Reward: 120.0, Step: 120 -2022-10-30 21:19:00 - r - INFO: - Episode: 543/1000, Reward: 109.0, Step: 109 -2022-10-30 21:19:01 - r - INFO: - Episode: 544/1000, Reward: 122.0, Step: 122 -2022-10-30 21:19:01 - r - INFO: - Episode: 545/1000, Reward: 115.0, Step: 115 -2022-10-30 21:19:02 - r - INFO: - Episode: 546/1000, Reward: 34.0, Step: 34 -2022-10-30 21:19:02 - r - INFO: - Episode: 547/1000, Reward: 28.0, Step: 28 -2022-10-30 21:19:03 - r - INFO: - Episode: 548/1000, Reward: 29.0, Step: 29 -2022-10-30 21:19:03 - r - INFO: - Episode: 549/1000, Reward: 113.0, Step: 113 -2022-10-30 21:19:04 - r - INFO: - Episode: 550/1000, Reward: 100.0, Step: 100 -2022-10-30 21:19:04 - r - INFO: - Episode: 551/1000, Reward: 26.0, Step: 26 -2022-10-30 21:19:04 - r - INFO: - Episode: 552/1000, Reward: 24.0, Step: 24 -2022-10-30 21:19:05 - r - INFO: - Episode: 553/1000, Reward: 26.0, Step: 26 -2022-10-30 21:19:05 - r - INFO: - Episode: 554/1000, Reward: 102.0, Step: 102 -2022-10-30 21:19:05 - r - INFO: - Episode: 555/1000, Reward: 18.0, Step: 18 -2022-10-30 21:19:06 - r - INFO: - Episode: 556/1000, Reward: 107.0, Step: 107 -2022-10-30 21:19:06 - r - INFO: - Episode: 557/1000, Reward: 27.0, Step: 27 -2022-10-30 21:19:06 - r - INFO: - Episode: 558/1000, Reward: 87.0, Step: 87 -2022-10-30 21:19:07 - r - INFO: - Episode: 559/1000, Reward: 29.0, Step: 29 -2022-10-30 21:19:07 - r - INFO: - Episode: 560/1000, Reward: 31.0, Step: 31 -2022-10-30 21:19:07 - r - INFO: - Episode: 561/1000, Reward: 112.0, Step: 112 -2022-10-30 21:19:08 - r - INFO: - Episode: 562/1000, Reward: 112.0, Step: 112 -2022-10-30 21:19:09 - r - INFO: - Episode: 563/1000, Reward: 108.0, Step: 108 -2022-10-30 21:19:09 - r - INFO: - Episode: 564/1000, Reward: 98.0, Step: 98 -2022-10-30 21:19:10 - r - INFO: - Episode: 565/1000, Reward: 104.0, Step: 104 -2022-10-30 21:19:10 - r - INFO: - Episode: 566/1000, Reward: 116.0, Step: 116 -2022-10-30 21:19:11 - r - INFO: - Episode: 567/1000, Reward: 123.0, Step: 123 -2022-10-30 21:19:12 - r - INFO: - Episode: 568/1000, Reward: 105.0, Step: 105 -2022-10-30 21:19:12 - r - INFO: - Episode: 569/1000, Reward: 133.0, Step: 133 -2022-10-30 21:19:13 - r - INFO: - Episode: 570/1000, Reward: 116.0, Step: 116 -2022-10-30 21:19:14 - r - INFO: - Episode: 571/1000, Reward: 128.0, Step: 128 -2022-10-30 21:19:15 - r - INFO: - Episode: 572/1000, Reward: 130.0, Step: 130 -2022-10-30 21:19:15 - r - INFO: - Episode: 573/1000, Reward: 113.0, Step: 113 -2022-10-30 21:19:16 - r - INFO: - Episode: 574/1000, Reward: 143.0, Step: 143 -2022-10-30 21:19:17 - r - INFO: - Episode: 575/1000, Reward: 145.0, Step: 145 -2022-10-30 21:19:18 - r - INFO: - Episode: 576/1000, Reward: 159.0, Step: 159 -2022-10-30 21:19:19 - r - INFO: - Episode: 577/1000, Reward: 150.0, Step: 150 -2022-10-30 21:19:19 - r - INFO: - Episode: 578/1000, Reward: 130.0, Step: 130 -2022-10-30 21:19:20 - r - INFO: - Episode: 579/1000, Reward: 145.0, Step: 145 -2022-10-30 21:19:21 - r - INFO: - Episode: 580/1000, Reward: 173.0, Step: 173 -2022-10-30 21:19:22 - r - INFO: - Episode: 581/1000, Reward: 154.0, Step: 154 -2022-10-30 21:19:23 - r - INFO: - Episode: 582/1000, Reward: 131.0, Step: 131 -2022-10-30 21:19:24 - r - INFO: - Episode: 583/1000, Reward: 163.0, Step: 163 -2022-10-30 21:19:25 - r - INFO: - Episode: 584/1000, Reward: 160.0, Step: 160 -2022-10-30 21:19:26 - r - INFO: - Episode: 585/1000, Reward: 181.0, Step: 181 -2022-10-30 21:19:27 - r - INFO: - Episode: 586/1000, Reward: 161.0, Step: 161 -2022-10-30 21:19:28 - r - INFO: - Episode: 587/1000, Reward: 169.0, Step: 169 -2022-10-30 21:19:29 - r - INFO: - Episode: 588/1000, Reward: 150.0, Step: 150 -2022-10-30 21:19:30 - r - INFO: - Episode: 589/1000, Reward: 176.0, Step: 176 -2022-10-30 21:19:31 - r - INFO: - Episode: 590/1000, Reward: 157.0, Step: 157 -2022-10-30 21:19:32 - r - INFO: - Episode: 591/1000, Reward: 167.0, Step: 167 -2022-10-30 21:19:33 - r - INFO: - Episode: 592/1000, Reward: 168.0, Step: 168 -2022-10-30 21:19:34 - r - INFO: - Episode: 593/1000, Reward: 135.0, Step: 135 -2022-10-30 21:19:35 - r - INFO: - Episode: 594/1000, Reward: 157.0, Step: 157 -2022-10-30 21:19:35 - r - INFO: - Episode: 595/1000, Reward: 138.0, Step: 138 -2022-10-30 21:19:36 - r - INFO: - Episode: 596/1000, Reward: 139.0, Step: 139 -2022-10-30 21:19:37 - r - INFO: - Episode: 597/1000, Reward: 146.0, Step: 146 -2022-10-30 21:19:38 - r - INFO: - Episode: 598/1000, Reward: 121.0, Step: 121 -2022-10-30 21:19:39 - r - INFO: - Episode: 599/1000, Reward: 140.0, Step: 140 -2022-10-30 21:19:40 - r - INFO: - Episode: 600/1000, Reward: 124.0, Step: 124 -2022-10-30 21:19:41 - r - INFO: - Episode: 601/1000, Reward: 124.0, Step: 124 -2022-10-30 21:19:42 - r - INFO: - Episode: 602/1000, Reward: 115.0, Step: 115 -2022-10-30 21:19:42 - r - INFO: - Episode: 603/1000, Reward: 129.0, Step: 129 -2022-10-30 21:19:43 - r - INFO: - Episode: 604/1000, Reward: 107.0, Step: 107 -2022-10-30 21:19:44 - r - INFO: - Episode: 605/1000, Reward: 118.0, Step: 118 -2022-10-30 21:19:44 - r - INFO: - Episode: 606/1000, Reward: 108.0, Step: 108 -2022-10-30 21:19:45 - r - INFO: - Episode: 607/1000, Reward: 102.0, Step: 102 -2022-10-30 21:19:46 - r - INFO: - Episode: 608/1000, Reward: 105.0, Step: 105 -2022-10-30 21:19:46 - r - INFO: - Episode: 609/1000, Reward: 103.0, Step: 103 -2022-10-30 21:19:47 - r - INFO: - Episode: 610/1000, Reward: 96.0, Step: 96 -2022-10-30 21:19:47 - r - INFO: - Episode: 611/1000, Reward: 116.0, Step: 116 -2022-10-30 21:19:48 - r - INFO: - Episode: 612/1000, Reward: 51.0, Step: 51 -2022-10-30 21:19:48 - r - INFO: - Episode: 613/1000, Reward: 100.0, Step: 100 -2022-10-30 21:19:49 - r - INFO: - Episode: 614/1000, Reward: 121.0, Step: 121 -2022-10-30 21:19:50 - r - INFO: - Episode: 615/1000, Reward: 109.0, Step: 109 -2022-10-30 21:19:50 - r - INFO: - Episode: 616/1000, Reward: 85.0, Step: 85 -2022-10-30 21:19:51 - r - INFO: - Episode: 617/1000, Reward: 111.0, Step: 111 -2022-10-30 21:19:52 - r - INFO: - Episode: 618/1000, Reward: 91.0, Step: 91 -2022-10-30 21:19:52 - r - INFO: - Episode: 619/1000, Reward: 127.0, Step: 127 -2022-10-30 21:19:53 - r - INFO: - Episode: 620/1000, Reward: 117.0, Step: 117 -2022-10-30 21:19:53 - r - INFO: - Episode: 621/1000, Reward: 104.0, Step: 104 -2022-10-30 21:19:54 - r - INFO: - Episode: 622/1000, Reward: 119.0, Step: 119 -2022-10-30 21:19:55 - r - INFO: - Episode: 623/1000, Reward: 111.0, Step: 111 -2022-10-30 21:19:56 - r - INFO: - Episode: 624/1000, Reward: 132.0, Step: 132 -2022-10-30 21:19:56 - r - INFO: - Episode: 625/1000, Reward: 130.0, Step: 130 -2022-10-30 21:19:57 - r - INFO: - Episode: 626/1000, Reward: 140.0, Step: 140 -2022-10-30 21:19:58 - r - INFO: - Episode: 627/1000, Reward: 95.0, Step: 95 -2022-10-30 21:19:58 - r - INFO: - Episode: 628/1000, Reward: 106.0, Step: 106 -2022-10-30 21:19:59 - r - INFO: - Episode: 629/1000, Reward: 120.0, Step: 120 -2022-10-30 21:20:00 - r - INFO: - Episode: 630/1000, Reward: 111.0, Step: 111 -2022-10-30 21:20:00 - r - INFO: - Episode: 631/1000, Reward: 114.0, Step: 114 -2022-10-30 21:20:01 - r - INFO: - Episode: 632/1000, Reward: 126.0, Step: 126 -2022-10-30 21:20:02 - r - INFO: - Episode: 633/1000, Reward: 100.0, Step: 100 -2022-10-30 21:20:03 - r - INFO: - Episode: 634/1000, Reward: 111.0, Step: 111 -2022-10-30 21:20:03 - r - INFO: - Episode: 635/1000, Reward: 104.0, Step: 104 -2022-10-30 21:20:04 - r - INFO: - Episode: 636/1000, Reward: 103.0, Step: 103 -2022-10-30 21:20:04 - r - INFO: - Episode: 637/1000, Reward: 111.0, Step: 111 -2022-10-30 21:20:05 - r - INFO: - Episode: 638/1000, Reward: 110.0, Step: 110 -2022-10-30 21:20:06 - r - INFO: - Episode: 639/1000, Reward: 131.0, Step: 131 -2022-10-30 21:20:06 - r - INFO: - Episode: 640/1000, Reward: 90.0, Step: 90 -2022-10-30 21:20:07 - r - INFO: - Episode: 641/1000, Reward: 97.0, Step: 97 -2022-10-30 21:20:08 - r - INFO: - Episode: 642/1000, Reward: 104.0, Step: 104 -2022-10-30 21:20:09 - r - INFO: - Episode: 643/1000, Reward: 91.0, Step: 91 -2022-10-30 21:20:09 - r - INFO: - Episode: 644/1000, Reward: 97.0, Step: 97 -2022-10-30 21:20:10 - r - INFO: - Episode: 645/1000, Reward: 109.0, Step: 109 -2022-10-30 21:20:10 - r - INFO: - Episode: 646/1000, Reward: 112.0, Step: 112 -2022-10-30 21:20:11 - r - INFO: - Episode: 647/1000, Reward: 97.0, Step: 97 -2022-10-30 21:20:11 - r - INFO: - Episode: 648/1000, Reward: 32.0, Step: 32 -2022-10-30 21:20:12 - r - INFO: - Episode: 649/1000, Reward: 94.0, Step: 94 -2022-10-30 21:20:13 - r - INFO: - Episode: 650/1000, Reward: 107.0, Step: 107 -2022-10-30 21:20:13 - r - INFO: - Episode: 651/1000, Reward: 61.0, Step: 61 -2022-10-30 21:20:14 - r - INFO: - Episode: 652/1000, Reward: 97.0, Step: 97 -2022-10-30 21:20:14 - r - INFO: - Episode: 653/1000, Reward: 99.0, Step: 99 -2022-10-30 21:20:15 - r - INFO: - Episode: 654/1000, Reward: 76.0, Step: 76 -2022-10-30 21:20:15 - r - INFO: - Episode: 655/1000, Reward: 38.0, Step: 38 -2022-10-30 21:20:15 - r - INFO: - Episode: 656/1000, Reward: 96.0, Step: 96 -2022-10-30 21:20:16 - r - INFO: - Episode: 657/1000, Reward: 96.0, Step: 96 -2022-10-30 21:20:16 - r - INFO: - Episode: 658/1000, Reward: 65.0, Step: 65 -2022-10-30 21:20:17 - r - INFO: - Episode: 659/1000, Reward: 45.0, Step: 45 -2022-10-30 21:20:17 - r - INFO: - Episode: 660/1000, Reward: 91.0, Step: 91 -2022-10-30 21:20:18 - r - INFO: - Episode: 661/1000, Reward: 78.0, Step: 78 -2022-10-30 21:20:18 - r - INFO: - Episode: 662/1000, Reward: 90.0, Step: 90 -2022-10-30 21:20:19 - r - INFO: - Episode: 663/1000, Reward: 92.0, Step: 92 -2022-10-30 21:20:19 - r - INFO: - Episode: 664/1000, Reward: 94.0, Step: 94 -2022-10-30 21:20:20 - r - INFO: - Episode: 665/1000, Reward: 101.0, Step: 101 -2022-10-30 21:20:20 - r - INFO: - Episode: 666/1000, Reward: 111.0, Step: 111 -2022-10-30 21:20:21 - r - INFO: - Episode: 667/1000, Reward: 109.0, Step: 109 -2022-10-30 21:20:22 - r - INFO: - Episode: 668/1000, Reward: 99.0, Step: 99 -2022-10-30 21:20:22 - r - INFO: - Episode: 669/1000, Reward: 115.0, Step: 115 -2022-10-30 21:20:23 - r - INFO: - Episode: 670/1000, Reward: 112.0, Step: 112 -2022-10-30 21:20:23 - r - INFO: - Episode: 671/1000, Reward: 113.0, Step: 113 -2022-10-30 21:20:24 - r - INFO: - Episode: 672/1000, Reward: 110.0, Step: 110 -2022-10-30 21:20:25 - r - INFO: - Episode: 673/1000, Reward: 108.0, Step: 108 -2022-10-30 21:20:26 - r - INFO: - Episode: 674/1000, Reward: 112.0, Step: 112 -2022-10-30 21:20:26 - r - INFO: - Episode: 675/1000, Reward: 125.0, Step: 125 -2022-10-30 21:20:27 - r - INFO: - Episode: 676/1000, Reward: 122.0, Step: 122 -2022-10-30 21:20:28 - r - INFO: - Episode: 677/1000, Reward: 114.0, Step: 114 -2022-10-30 21:20:28 - r - INFO: - Episode: 678/1000, Reward: 127.0, Step: 127 -2022-10-30 21:20:29 - r - INFO: - Episode: 679/1000, Reward: 125.0, Step: 125 -2022-10-30 21:20:30 - r - INFO: - Episode: 680/1000, Reward: 112.0, Step: 112 -2022-10-30 21:20:30 - r - INFO: - Episode: 681/1000, Reward: 111.0, Step: 111 -2022-10-30 21:20:31 - r - INFO: - Episode: 682/1000, Reward: 124.0, Step: 124 -2022-10-30 21:20:32 - r - INFO: - Episode: 683/1000, Reward: 113.0, Step: 113 -2022-10-30 21:20:33 - r - INFO: - Episode: 684/1000, Reward: 103.0, Step: 103 -2022-10-30 21:20:33 - r - INFO: - Episode: 685/1000, Reward: 119.0, Step: 119 -2022-10-30 21:20:34 - r - INFO: - Episode: 686/1000, Reward: 120.0, Step: 120 -2022-10-30 21:20:35 - r - INFO: - Episode: 687/1000, Reward: 95.0, Step: 95 -2022-10-30 21:20:35 - r - INFO: - Episode: 688/1000, Reward: 100.0, Step: 100 -2022-10-30 21:20:36 - r - INFO: - Episode: 689/1000, Reward: 29.0, Step: 29 -2022-10-30 21:20:36 - r - INFO: - Episode: 690/1000, Reward: 119.0, Step: 119 -2022-10-30 21:20:37 - r - INFO: - Episode: 691/1000, Reward: 107.0, Step: 107 -2022-10-30 21:20:38 - r - INFO: - Episode: 692/1000, Reward: 117.0, Step: 117 -2022-10-30 21:20:38 - r - INFO: - Episode: 693/1000, Reward: 78.0, Step: 78 -2022-10-30 21:20:38 - r - INFO: - Episode: 694/1000, Reward: 35.0, Step: 35 -2022-10-30 21:20:39 - r - INFO: - Episode: 695/1000, Reward: 101.0, Step: 101 -2022-10-30 21:20:40 - r - INFO: - Episode: 696/1000, Reward: 98.0, Step: 98 -2022-10-30 21:20:40 - r - INFO: - Episode: 697/1000, Reward: 94.0, Step: 94 -2022-10-30 21:20:41 - r - INFO: - Episode: 698/1000, Reward: 102.0, Step: 102 -2022-10-30 21:20:41 - r - INFO: - Episode: 699/1000, Reward: 90.0, Step: 90 -2022-10-30 21:20:42 - r - INFO: - Episode: 700/1000, Reward: 86.0, Step: 86 -2022-10-30 21:20:42 - r - INFO: - Episode: 701/1000, Reward: 81.0, Step: 81 -2022-10-30 21:20:43 - r - INFO: - Episode: 702/1000, Reward: 105.0, Step: 105 -2022-10-30 21:20:43 - r - INFO: - Episode: 703/1000, Reward: 72.0, Step: 72 -2022-10-30 21:20:44 - r - INFO: - Episode: 704/1000, Reward: 100.0, Step: 100 -2022-10-30 21:20:44 - r - INFO: - Episode: 705/1000, Reward: 96.0, Step: 96 -2022-10-30 21:20:45 - r - INFO: - Episode: 706/1000, Reward: 111.0, Step: 111 -2022-10-30 21:20:45 - r - INFO: - Episode: 707/1000, Reward: 27.0, Step: 27 -2022-10-30 21:20:46 - r - INFO: - Episode: 708/1000, Reward: 107.0, Step: 107 -2022-10-30 21:20:47 - r - INFO: - Episode: 709/1000, Reward: 87.0, Step: 87 -2022-10-30 21:20:47 - r - INFO: - Episode: 710/1000, Reward: 114.0, Step: 114 -2022-10-30 21:20:48 - r - INFO: - Episode: 711/1000, Reward: 111.0, Step: 111 -2022-10-30 21:20:48 - r - INFO: - Episode: 712/1000, Reward: 88.0, Step: 88 -2022-10-30 21:20:49 - r - INFO: - Episode: 713/1000, Reward: 112.0, Step: 112 -2022-10-30 21:20:50 - r - INFO: - Episode: 714/1000, Reward: 108.0, Step: 108 -2022-10-30 21:20:50 - r - INFO: - Episode: 715/1000, Reward: 108.0, Step: 108 -2022-10-30 21:20:51 - r - INFO: - Episode: 716/1000, Reward: 103.0, Step: 103 -2022-10-30 21:20:52 - r - INFO: - Episode: 717/1000, Reward: 120.0, Step: 120 -2022-10-30 21:20:52 - r - INFO: - Episode: 718/1000, Reward: 116.0, Step: 116 -2022-10-30 21:20:53 - r - INFO: - Episode: 719/1000, Reward: 112.0, Step: 112 -2022-10-30 21:20:54 - r - INFO: - Episode: 720/1000, Reward: 99.0, Step: 99 -2022-10-30 21:20:54 - r - INFO: - Episode: 721/1000, Reward: 118.0, Step: 118 -2022-10-30 21:20:55 - r - INFO: - Episode: 722/1000, Reward: 114.0, Step: 114 -2022-10-30 21:20:56 - r - INFO: - Episode: 723/1000, Reward: 104.0, Step: 104 -2022-10-30 21:20:56 - r - INFO: - Episode: 724/1000, Reward: 99.0, Step: 99 -2022-10-30 21:20:57 - r - INFO: - Episode: 725/1000, Reward: 102.0, Step: 102 -2022-10-30 21:20:57 - r - INFO: - Episode: 726/1000, Reward: 106.0, Step: 106 -2022-10-30 21:20:58 - r - INFO: - Episode: 727/1000, Reward: 31.0, Step: 31 -2022-10-30 21:20:58 - r - INFO: - Episode: 728/1000, Reward: 91.0, Step: 91 -2022-10-30 21:20:59 - r - INFO: - Episode: 729/1000, Reward: 32.0, Step: 32 -2022-10-30 21:20:59 - r - INFO: - Episode: 730/1000, Reward: 96.0, Step: 96 -2022-10-30 21:20:59 - r - INFO: - Episode: 731/1000, Reward: 20.0, Step: 20 -2022-10-30 21:21:00 - r - INFO: - Episode: 732/1000, Reward: 33.0, Step: 33 -2022-10-30 21:21:00 - r - INFO: - Episode: 733/1000, Reward: 23.0, Step: 23 -2022-10-30 21:21:00 - r - INFO: - Episode: 734/1000, Reward: 80.0, Step: 80 -2022-10-30 21:21:01 - r - INFO: - Episode: 735/1000, Reward: 35.0, Step: 35 -2022-10-30 21:21:01 - r - INFO: - Episode: 736/1000, Reward: 88.0, Step: 88 -2022-10-30 21:21:01 - r - INFO: - Episode: 737/1000, Reward: 28.0, Step: 28 -2022-10-30 21:21:01 - r - INFO: - Episode: 738/1000, Reward: 26.0, Step: 26 -2022-10-30 21:21:02 - r - INFO: - Episode: 739/1000, Reward: 70.0, Step: 70 -2022-10-30 21:21:02 - r - INFO: - Episode: 740/1000, Reward: 86.0, Step: 86 -2022-10-30 21:21:02 - r - INFO: - Episode: 741/1000, Reward: 28.0, Step: 28 -2022-10-30 21:21:02 - r - INFO: - Episode: 742/1000, Reward: 39.0, Step: 39 -2022-10-30 21:21:03 - r - INFO: - Episode: 743/1000, Reward: 65.0, Step: 65 -2022-10-30 21:21:03 - r - INFO: - Episode: 744/1000, Reward: 52.0, Step: 52 -2022-10-30 21:21:03 - r - INFO: - Episode: 745/1000, Reward: 43.0, Step: 43 -2022-10-30 21:21:04 - r - INFO: - Episode: 746/1000, Reward: 97.0, Step: 97 -2022-10-30 21:21:04 - r - INFO: - Episode: 747/1000, Reward: 27.0, Step: 27 -2022-10-30 21:21:05 - r - INFO: - Episode: 748/1000, Reward: 89.0, Step: 89 -2022-10-30 21:21:05 - r - INFO: - Episode: 749/1000, Reward: 34.0, Step: 34 -2022-10-30 21:21:05 - r - INFO: - Episode: 750/1000, Reward: 35.0, Step: 35 -2022-10-30 21:21:06 - r - INFO: - Episode: 751/1000, Reward: 28.0, Step: 28 -2022-10-30 21:21:06 - r - INFO: - Episode: 752/1000, Reward: 96.0, Step: 96 -2022-10-30 21:21:07 - r - INFO: - Episode: 753/1000, Reward: 97.0, Step: 97 -2022-10-30 21:21:07 - r - INFO: - Episode: 754/1000, Reward: 108.0, Step: 108 -2022-10-30 21:21:08 - r - INFO: - Episode: 755/1000, Reward: 45.0, Step: 45 -2022-10-30 21:21:09 - r - INFO: - Episode: 756/1000, Reward: 103.0, Step: 103 -2022-10-30 21:21:10 - r - INFO: - Episode: 757/1000, Reward: 97.0, Step: 97 -2022-10-30 21:21:10 - r - INFO: - Episode: 758/1000, Reward: 114.0, Step: 114 -2022-10-30 21:21:11 - r - INFO: - Episode: 759/1000, Reward: 103.0, Step: 103 -2022-10-30 21:21:12 - r - INFO: - Episode: 760/1000, Reward: 116.0, Step: 116 -2022-10-30 21:21:12 - r - INFO: - Episode: 761/1000, Reward: 127.0, Step: 127 -2022-10-30 21:21:13 - r - INFO: - Episode: 762/1000, Reward: 122.0, Step: 122 -2022-10-30 21:21:14 - r - INFO: - Episode: 763/1000, Reward: 112.0, Step: 112 -2022-10-30 21:21:14 - r - INFO: - Episode: 764/1000, Reward: 112.0, Step: 112 -2022-10-30 21:21:15 - r - INFO: - Episode: 765/1000, Reward: 120.0, Step: 120 -2022-10-30 21:21:16 - r - INFO: - Episode: 766/1000, Reward: 129.0, Step: 129 -2022-10-30 21:21:17 - r - INFO: - Episode: 767/1000, Reward: 127.0, Step: 127 -2022-10-30 21:21:18 - r - INFO: - Episode: 768/1000, Reward: 125.0, Step: 125 -2022-10-30 21:21:19 - r - INFO: - Episode: 769/1000, Reward: 124.0, Step: 124 -2022-10-30 21:21:20 - r - INFO: - Episode: 770/1000, Reward: 126.0, Step: 126 -2022-10-30 21:21:20 - r - INFO: - Episode: 771/1000, Reward: 129.0, Step: 129 -2022-10-30 21:21:21 - r - INFO: - Episode: 772/1000, Reward: 129.0, Step: 129 -2022-10-30 21:21:22 - r - INFO: - Episode: 773/1000, Reward: 43.0, Step: 43 -2022-10-30 21:21:22 - r - INFO: - Episode: 774/1000, Reward: 121.0, Step: 121 -2022-10-30 21:21:23 - r - INFO: - Episode: 775/1000, Reward: 40.0, Step: 40 -2022-10-30 21:21:24 - r - INFO: - Episode: 776/1000, Reward: 116.0, Step: 116 -2022-10-30 21:21:24 - r - INFO: - Episode: 777/1000, Reward: 117.0, Step: 117 -2022-10-30 21:21:25 - r - INFO: - Episode: 778/1000, Reward: 113.0, Step: 113 -2022-10-30 21:21:26 - r - INFO: - Episode: 779/1000, Reward: 117.0, Step: 117 -2022-10-30 21:21:26 - r - INFO: - Episode: 780/1000, Reward: 108.0, Step: 108 -2022-10-30 21:21:27 - r - INFO: - Episode: 781/1000, Reward: 108.0, Step: 108 -2022-10-30 21:21:28 - r - INFO: - Episode: 782/1000, Reward: 119.0, Step: 119 -2022-10-30 21:21:28 - r - INFO: - Episode: 783/1000, Reward: 109.0, Step: 109 -2022-10-30 21:21:29 - r - INFO: - Episode: 784/1000, Reward: 116.0, Step: 116 -2022-10-30 21:21:29 - r - INFO: - Episode: 785/1000, Reward: 114.0, Step: 114 -2022-10-30 21:21:30 - r - INFO: - Episode: 786/1000, Reward: 45.0, Step: 45 -2022-10-30 21:21:31 - r - INFO: - Episode: 787/1000, Reward: 116.0, Step: 116 -2022-10-30 21:21:31 - r - INFO: - Episode: 788/1000, Reward: 116.0, Step: 116 -2022-10-30 21:21:32 - r - INFO: - Episode: 789/1000, Reward: 110.0, Step: 110 -2022-10-30 21:21:32 - r - INFO: - Episode: 790/1000, Reward: 105.0, Step: 105 -2022-10-30 21:21:33 - r - INFO: - Episode: 791/1000, Reward: 110.0, Step: 110 -2022-10-30 21:21:34 - r - INFO: - Episode: 792/1000, Reward: 112.0, Step: 112 -2022-10-30 21:21:34 - r - INFO: - Episode: 793/1000, Reward: 104.0, Step: 104 -2022-10-30 21:21:35 - r - INFO: - Episode: 794/1000, Reward: 120.0, Step: 120 -2022-10-30 21:21:36 - r - INFO: - Episode: 795/1000, Reward: 110.0, Step: 110 -2022-10-30 21:21:36 - r - INFO: - Episode: 796/1000, Reward: 113.0, Step: 113 -2022-10-30 21:21:37 - r - INFO: - Episode: 797/1000, Reward: 33.0, Step: 33 -2022-10-30 21:21:37 - r - INFO: - Episode: 798/1000, Reward: 111.0, Step: 111 -2022-10-30 21:21:38 - r - INFO: - Episode: 799/1000, Reward: 31.0, Step: 31 -2022-10-30 21:21:38 - r - INFO: - Episode: 800/1000, Reward: 139.0, Step: 139 -2022-10-30 21:21:39 - r - INFO: - Episode: 801/1000, Reward: 110.0, Step: 110 -2022-10-30 21:21:40 - r - INFO: - Episode: 802/1000, Reward: 124.0, Step: 124 -2022-10-30 21:21:41 - r - INFO: - Episode: 803/1000, Reward: 120.0, Step: 120 -2022-10-30 21:21:41 - r - INFO: - Episode: 804/1000, Reward: 112.0, Step: 112 -2022-10-30 21:21:42 - r - INFO: - Episode: 805/1000, Reward: 116.0, Step: 116 -2022-10-30 21:21:43 - r - INFO: - Episode: 806/1000, Reward: 105.0, Step: 105 -2022-10-30 21:21:43 - r - INFO: - Episode: 807/1000, Reward: 125.0, Step: 125 -2022-10-30 21:21:44 - r - INFO: - Episode: 808/1000, Reward: 103.0, Step: 103 -2022-10-30 21:21:45 - r - INFO: - Episode: 809/1000, Reward: 122.0, Step: 122 -2022-10-30 21:21:45 - r - INFO: - Episode: 810/1000, Reward: 109.0, Step: 109 -2022-10-30 21:21:46 - r - INFO: - Episode: 811/1000, Reward: 118.0, Step: 118 -2022-10-30 21:21:47 - r - INFO: - Episode: 812/1000, Reward: 124.0, Step: 124 -2022-10-30 21:21:48 - r - INFO: - Episode: 813/1000, Reward: 115.0, Step: 115 -2022-10-30 21:21:48 - r - INFO: - Episode: 814/1000, Reward: 26.0, Step: 26 -2022-10-30 21:21:49 - r - INFO: - Episode: 815/1000, Reward: 118.0, Step: 118 -2022-10-30 21:21:49 - r - INFO: - Episode: 816/1000, Reward: 118.0, Step: 118 -2022-10-30 21:21:50 - r - INFO: - Episode: 817/1000, Reward: 31.0, Step: 31 -2022-10-30 21:21:50 - r - INFO: - Episode: 818/1000, Reward: 99.0, Step: 99 -2022-10-30 21:21:51 - r - INFO: - Episode: 819/1000, Reward: 122.0, Step: 122 -2022-10-30 21:21:52 - r - INFO: - Episode: 820/1000, Reward: 102.0, Step: 102 -2022-10-30 21:21:52 - r - INFO: - Episode: 821/1000, Reward: 111.0, Step: 111 -2022-10-30 21:21:53 - r - INFO: - Episode: 822/1000, Reward: 110.0, Step: 110 -2022-10-30 21:21:54 - r - INFO: - Episode: 823/1000, Reward: 113.0, Step: 113 -2022-10-30 21:21:54 - r - INFO: - Episode: 824/1000, Reward: 117.0, Step: 117 -2022-10-30 21:21:55 - r - INFO: - Episode: 825/1000, Reward: 113.0, Step: 113 -2022-10-30 21:21:56 - r - INFO: - Episode: 826/1000, Reward: 109.0, Step: 109 -2022-10-30 21:21:57 - r - INFO: - Episode: 827/1000, Reward: 122.0, Step: 122 -2022-10-30 21:21:57 - r - INFO: - Episode: 828/1000, Reward: 117.0, Step: 117 -2022-10-30 21:21:58 - r - INFO: - Episode: 829/1000, Reward: 127.0, Step: 127 -2022-10-30 21:21:59 - r - INFO: - Episode: 830/1000, Reward: 113.0, Step: 113 -2022-10-30 21:21:59 - r - INFO: - Episode: 831/1000, Reward: 118.0, Step: 118 -2022-10-30 21:22:00 - r - INFO: - Episode: 832/1000, Reward: 107.0, Step: 107 -2022-10-30 21:22:01 - r - INFO: - Episode: 833/1000, Reward: 108.0, Step: 108 -2022-10-30 21:22:01 - r - INFO: - Episode: 834/1000, Reward: 103.0, Step: 103 -2022-10-30 21:22:02 - r - INFO: - Episode: 835/1000, Reward: 126.0, Step: 126 -2022-10-30 21:22:03 - r - INFO: - Episode: 836/1000, Reward: 131.0, Step: 131 -2022-10-30 21:22:03 - r - INFO: - Episode: 837/1000, Reward: 106.0, Step: 106 -2022-10-30 21:22:04 - r - INFO: - Episode: 838/1000, Reward: 116.0, Step: 116 -2022-10-30 21:22:05 - r - INFO: - Episode: 839/1000, Reward: 24.0, Step: 24 -2022-10-30 21:22:05 - r - INFO: - Episode: 840/1000, Reward: 107.0, Step: 107 -2022-10-30 21:22:06 - r - INFO: - Episode: 841/1000, Reward: 124.0, Step: 124 -2022-10-30 21:22:07 - r - INFO: - Episode: 842/1000, Reward: 125.0, Step: 125 -2022-10-30 21:22:07 - r - INFO: - Episode: 843/1000, Reward: 110.0, Step: 110 -2022-10-30 21:22:08 - r - INFO: - Episode: 844/1000, Reward: 112.0, Step: 112 -2022-10-30 21:22:09 - r - INFO: - Episode: 845/1000, Reward: 105.0, Step: 105 -2022-10-30 21:22:09 - r - INFO: - Episode: 846/1000, Reward: 104.0, Step: 104 -2022-10-30 21:22:10 - r - INFO: - Episode: 847/1000, Reward: 134.0, Step: 134 -2022-10-30 21:22:11 - r - INFO: - Episode: 848/1000, Reward: 107.0, Step: 107 -2022-10-30 21:22:11 - r - INFO: - Episode: 849/1000, Reward: 128.0, Step: 128 -2022-10-30 21:22:12 - r - INFO: - Episode: 850/1000, Reward: 113.0, Step: 113 -2022-10-30 21:22:13 - r - INFO: - Episode: 851/1000, Reward: 138.0, Step: 138 -2022-10-30 21:22:14 - r - INFO: - Episode: 852/1000, Reward: 118.0, Step: 118 -2022-10-30 21:22:14 - r - INFO: - Episode: 853/1000, Reward: 142.0, Step: 142 -2022-10-30 21:22:15 - r - INFO: - Episode: 854/1000, Reward: 118.0, Step: 118 -2022-10-30 21:22:16 - r - INFO: - Episode: 855/1000, Reward: 122.0, Step: 122 -2022-10-30 21:22:16 - r - INFO: - Episode: 856/1000, Reward: 130.0, Step: 130 -2022-10-30 21:22:17 - r - INFO: - Episode: 857/1000, Reward: 126.0, Step: 126 -2022-10-30 21:22:18 - r - INFO: - Episode: 858/1000, Reward: 111.0, Step: 111 -2022-10-30 21:22:19 - r - INFO: - Episode: 859/1000, Reward: 114.0, Step: 114 -2022-10-30 21:22:19 - r - INFO: - Episode: 860/1000, Reward: 128.0, Step: 128 -2022-10-30 21:22:20 - r - INFO: - Episode: 861/1000, Reward: 126.0, Step: 126 -2022-10-30 21:22:21 - r - INFO: - Episode: 862/1000, Reward: 143.0, Step: 143 -2022-10-30 21:22:22 - r - INFO: - Episode: 863/1000, Reward: 132.0, Step: 132 -2022-10-30 21:22:22 - r - INFO: - Episode: 864/1000, Reward: 123.0, Step: 123 -2022-10-30 21:22:23 - r - INFO: - Episode: 865/1000, Reward: 111.0, Step: 111 -2022-10-30 21:22:24 - r - INFO: - Episode: 866/1000, Reward: 129.0, Step: 129 -2022-10-30 21:22:25 - r - INFO: - Episode: 867/1000, Reward: 121.0, Step: 121 -2022-10-30 21:22:25 - r - INFO: - Episode: 868/1000, Reward: 114.0, Step: 114 -2022-10-30 21:22:26 - r - INFO: - Episode: 869/1000, Reward: 110.0, Step: 110 -2022-10-30 21:22:27 - r - INFO: - Episode: 870/1000, Reward: 118.0, Step: 118 -2022-10-30 21:22:27 - r - INFO: - Episode: 871/1000, Reward: 120.0, Step: 120 -2022-10-30 21:22:28 - r - INFO: - Episode: 872/1000, Reward: 109.0, Step: 109 -2022-10-30 21:22:29 - r - INFO: - Episode: 873/1000, Reward: 106.0, Step: 106 -2022-10-30 21:22:29 - r - INFO: - Episode: 874/1000, Reward: 118.0, Step: 118 -2022-10-30 21:22:30 - r - INFO: - Episode: 875/1000, Reward: 104.0, Step: 104 -2022-10-30 21:22:30 - r - INFO: - Episode: 876/1000, Reward: 98.0, Step: 98 -2022-10-30 21:22:31 - r - INFO: - Episode: 877/1000, Reward: 115.0, Step: 115 -2022-10-30 21:22:31 - r - INFO: - Episode: 878/1000, Reward: 34.0, Step: 34 -2022-10-30 21:22:32 - r - INFO: - Episode: 879/1000, Reward: 96.0, Step: 96 -2022-10-30 21:22:33 - r - INFO: - Episode: 880/1000, Reward: 108.0, Step: 108 -2022-10-30 21:22:33 - r - INFO: - Episode: 881/1000, Reward: 105.0, Step: 105 -2022-10-30 21:22:34 - r - INFO: - Episode: 882/1000, Reward: 33.0, Step: 33 -2022-10-30 21:22:34 - r - INFO: - Episode: 883/1000, Reward: 105.0, Step: 105 -2022-10-30 21:22:35 - r - INFO: - Episode: 884/1000, Reward: 111.0, Step: 111 -2022-10-30 21:22:35 - r - INFO: - Episode: 885/1000, Reward: 112.0, Step: 112 -2022-10-30 21:22:36 - r - INFO: - Episode: 886/1000, Reward: 101.0, Step: 101 -2022-10-30 21:22:36 - r - INFO: - Episode: 887/1000, Reward: 25.0, Step: 25 -2022-10-30 21:22:37 - r - INFO: - Episode: 888/1000, Reward: 35.0, Step: 35 -2022-10-30 21:22:37 - r - INFO: - Episode: 889/1000, Reward: 99.0, Step: 99 -2022-10-30 21:22:38 - r - INFO: - Episode: 890/1000, Reward: 105.0, Step: 105 -2022-10-30 21:22:38 - r - INFO: - Episode: 891/1000, Reward: 36.0, Step: 36 -2022-10-30 21:22:38 - r - INFO: - Episode: 892/1000, Reward: 92.0, Step: 92 -2022-10-30 21:22:39 - r - INFO: - Episode: 893/1000, Reward: 104.0, Step: 104 -2022-10-30 21:22:39 - r - INFO: - Episode: 894/1000, Reward: 111.0, Step: 111 -2022-10-30 21:22:40 - r - INFO: - Episode: 895/1000, Reward: 106.0, Step: 106 -2022-10-30 21:22:41 - r - INFO: - Episode: 896/1000, Reward: 109.0, Step: 109 -2022-10-30 21:22:41 - r - INFO: - Episode: 897/1000, Reward: 108.0, Step: 108 -2022-10-30 21:22:42 - r - INFO: - Episode: 898/1000, Reward: 101.0, Step: 101 -2022-10-30 21:22:42 - r - INFO: - Episode: 899/1000, Reward: 100.0, Step: 100 -2022-10-30 21:22:43 - r - INFO: - Episode: 900/1000, Reward: 33.0, Step: 33 -2022-10-30 21:22:44 - r - INFO: - Episode: 901/1000, Reward: 119.0, Step: 119 -2022-10-30 21:22:44 - r - INFO: - Episode: 902/1000, Reward: 112.0, Step: 112 -2022-10-30 21:22:45 - r - INFO: - Episode: 903/1000, Reward: 112.0, Step: 112 -2022-10-30 21:22:45 - r - INFO: - Episode: 904/1000, Reward: 126.0, Step: 126 -2022-10-30 21:22:46 - r - INFO: - Episode: 905/1000, Reward: 123.0, Step: 123 -2022-10-30 21:22:47 - r - INFO: - Episode: 906/1000, Reward: 125.0, Step: 125 -2022-10-30 21:22:47 - r - INFO: - Episode: 907/1000, Reward: 107.0, Step: 107 -2022-10-30 21:22:48 - r - INFO: - Episode: 908/1000, Reward: 128.0, Step: 128 -2022-10-30 21:22:49 - r - INFO: - Episode: 909/1000, Reward: 119.0, Step: 119 -2022-10-30 21:22:50 - r - INFO: - Episode: 910/1000, Reward: 142.0, Step: 142 -2022-10-30 21:22:50 - r - INFO: - Episode: 911/1000, Reward: 117.0, Step: 117 -2022-10-30 21:22:51 - r - INFO: - Episode: 912/1000, Reward: 125.0, Step: 125 -2022-10-30 21:22:52 - r - INFO: - Episode: 913/1000, Reward: 141.0, Step: 141 -2022-10-30 21:22:53 - r - INFO: - Episode: 914/1000, Reward: 134.0, Step: 134 -2022-10-30 21:22:53 - r - INFO: - Episode: 915/1000, Reward: 131.0, Step: 131 -2022-10-30 21:22:54 - r - INFO: - Episode: 916/1000, Reward: 131.0, Step: 131 -2022-10-30 21:22:55 - r - INFO: - Episode: 917/1000, Reward: 140.0, Step: 140 -2022-10-30 21:22:56 - r - INFO: - Episode: 918/1000, Reward: 115.0, Step: 115 -2022-10-30 21:22:56 - r - INFO: - Episode: 919/1000, Reward: 142.0, Step: 142 -2022-10-30 21:22:57 - r - INFO: - Episode: 920/1000, Reward: 142.0, Step: 142 -2022-10-30 21:22:58 - r - INFO: - Episode: 921/1000, Reward: 128.0, Step: 128 -2022-10-30 21:22:59 - r - INFO: - Episode: 922/1000, Reward: 139.0, Step: 139 -2022-10-30 21:23:00 - r - INFO: - Episode: 923/1000, Reward: 133.0, Step: 133 -2022-10-30 21:23:01 - r - INFO: - Episode: 924/1000, Reward: 129.0, Step: 129 -2022-10-30 21:23:01 - r - INFO: - Episode: 925/1000, Reward: 124.0, Step: 124 -2022-10-30 21:23:02 - r - INFO: - Episode: 926/1000, Reward: 131.0, Step: 131 -2022-10-30 21:23:03 - r - INFO: - Episode: 927/1000, Reward: 125.0, Step: 125 -2022-10-30 21:23:04 - r - INFO: - Episode: 928/1000, Reward: 146.0, Step: 146 -2022-10-30 21:23:04 - r - INFO: - Episode: 929/1000, Reward: 118.0, Step: 118 -2022-10-30 21:23:05 - r - INFO: - Episode: 930/1000, Reward: 126.0, Step: 126 -2022-10-30 21:23:06 - r - INFO: - Episode: 931/1000, Reward: 134.0, Step: 134 -2022-10-30 21:23:07 - r - INFO: - Episode: 932/1000, Reward: 155.0, Step: 155 -2022-10-30 21:23:07 - r - INFO: - Episode: 933/1000, Reward: 134.0, Step: 134 -2022-10-30 21:23:08 - r - INFO: - Episode: 934/1000, Reward: 136.0, Step: 136 -2022-10-30 21:23:09 - r - INFO: - Episode: 935/1000, Reward: 146.0, Step: 146 -2022-10-30 21:23:10 - r - INFO: - Episode: 936/1000, Reward: 150.0, Step: 150 -2022-10-30 21:23:11 - r - INFO: - Episode: 937/1000, Reward: 167.0, Step: 167 -2022-10-30 21:23:12 - r - INFO: - Episode: 938/1000, Reward: 135.0, Step: 135 -2022-10-30 21:23:13 - r - INFO: - Episode: 939/1000, Reward: 197.0, Step: 197 -2022-10-30 21:23:14 - r - INFO: - Episode: 940/1000, Reward: 190.0, Step: 190 -2022-10-30 21:23:15 - r - INFO: - Episode: 941/1000, Reward: 170.0, Step: 170 -2022-10-30 21:23:16 - r - INFO: - Episode: 942/1000, Reward: 179.0, Step: 179 -2022-10-30 21:23:17 - r - INFO: - Episode: 943/1000, Reward: 192.0, Step: 192 -2022-10-30 21:23:18 - r - INFO: - Episode: 944/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:19 - r - INFO: - Current episode 944 has the best eval reward: 199.5 -2022-10-30 21:23:20 - r - INFO: - Episode: 945/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:21 - r - INFO: - Episode: 946/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:22 - r - INFO: - Episode: 947/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:23 - r - INFO: - Current episode 947 has the best eval reward: 200.0 -2022-10-30 21:23:23 - r - INFO: - Episode: 948/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:25 - r - INFO: - Episode: 949/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:25 - r - INFO: - Current episode 949 has the best eval reward: 200.0 -2022-10-30 21:23:26 - r - INFO: - Episode: 950/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:26 - r - INFO: - Current episode 950 has the best eval reward: 200.0 -2022-10-30 21:23:27 - r - INFO: - Episode: 951/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:28 - r - INFO: - Current episode 951 has the best eval reward: 200.0 -2022-10-30 21:23:28 - r - INFO: - Episode: 952/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:29 - r - INFO: - Current episode 952 has the best eval reward: 200.0 -2022-10-30 21:23:29 - r - INFO: - Episode: 953/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:30 - r - INFO: - Current episode 953 has the best eval reward: 200.0 -2022-10-30 21:23:31 - r - INFO: - Episode: 954/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:31 - r - INFO: - Current episode 954 has the best eval reward: 200.0 -2022-10-30 21:23:32 - r - INFO: - Episode: 955/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:33 - r - INFO: - Current episode 955 has the best eval reward: 200.0 -2022-10-30 21:23:33 - r - INFO: - Episode: 956/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:34 - r - INFO: - Current episode 956 has the best eval reward: 200.0 -2022-10-30 21:23:34 - r - INFO: - Episode: 957/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:35 - r - INFO: - Current episode 957 has the best eval reward: 200.0 -2022-10-30 21:23:36 - r - INFO: - Episode: 958/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:36 - r - INFO: - Current episode 958 has the best eval reward: 200.0 -2022-10-30 21:23:37 - r - INFO: - Episode: 959/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:37 - r - INFO: - Current episode 959 has the best eval reward: 200.0 -2022-10-30 21:23:38 - r - INFO: - Episode: 960/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:39 - r - INFO: - Episode: 961/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:40 - r - INFO: - Current episode 961 has the best eval reward: 200.0 -2022-10-30 21:23:40 - r - INFO: - Episode: 962/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:41 - r - INFO: - Current episode 962 has the best eval reward: 200.0 -2022-10-30 21:23:42 - r - INFO: - Episode: 963/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:42 - r - INFO: - Current episode 963 has the best eval reward: 200.0 -2022-10-30 21:23:43 - r - INFO: - Episode: 964/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:43 - r - INFO: - Current episode 964 has the best eval reward: 200.0 -2022-10-30 21:23:44 - r - INFO: - Episode: 965/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:45 - r - INFO: - Episode: 966/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:46 - r - INFO: - Current episode 966 has the best eval reward: 200.0 -2022-10-30 21:23:46 - r - INFO: - Episode: 967/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:47 - r - INFO: - Current episode 967 has the best eval reward: 200.0 -2022-10-30 21:23:48 - r - INFO: - Episode: 968/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:48 - r - INFO: - Current episode 968 has the best eval reward: 200.0 -2022-10-30 21:23:49 - r - INFO: - Episode: 969/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:50 - r - INFO: - Current episode 969 has the best eval reward: 200.0 -2022-10-30 21:23:50 - r - INFO: - Episode: 970/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:51 - r - INFO: - Episode: 971/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:52 - r - INFO: - Current episode 971 has the best eval reward: 200.0 -2022-10-30 21:23:52 - r - INFO: - Episode: 972/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:53 - r - INFO: - Current episode 972 has the best eval reward: 200.0 -2022-10-30 21:23:54 - r - INFO: - Episode: 973/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:54 - r - INFO: - Current episode 973 has the best eval reward: 200.0 -2022-10-30 21:23:55 - r - INFO: - Episode: 974/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:55 - r - INFO: - Current episode 974 has the best eval reward: 200.0 -2022-10-30 21:23:56 - r - INFO: - Episode: 975/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:57 - r - INFO: - Current episode 975 has the best eval reward: 200.0 -2022-10-30 21:23:57 - r - INFO: - Episode: 976/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:58 - r - INFO: - Current episode 976 has the best eval reward: 200.0 -2022-10-30 21:23:58 - r - INFO: - Episode: 977/1000, Reward: 200.0, Step: 200 -2022-10-30 21:23:59 - r - INFO: - Current episode 977 has the best eval reward: 200.0 -2022-10-30 21:24:00 - r - INFO: - Episode: 978/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:01 - r - INFO: - Episode: 979/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:01 - r - INFO: - Current episode 979 has the best eval reward: 200.0 -2022-10-30 21:24:02 - r - INFO: - Episode: 980/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:03 - r - INFO: - Current episode 980 has the best eval reward: 200.0 -2022-10-30 21:24:03 - r - INFO: - Episode: 981/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:04 - r - INFO: - Episode: 982/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:05 - r - INFO: - Episode: 983/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:06 - r - INFO: - Current episode 983 has the best eval reward: 200.0 -2022-10-30 21:24:07 - r - INFO: - Episode: 984/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:08 - r - INFO: - Episode: 985/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:09 - r - INFO: - Episode: 986/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:10 - r - INFO: - Episode: 987/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:11 - r - INFO: - Current episode 987 has the best eval reward: 200.0 -2022-10-30 21:24:12 - r - INFO: - Episode: 988/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:12 - r - INFO: - Current episode 988 has the best eval reward: 200.0 -2022-10-30 21:24:13 - r - INFO: - Episode: 989/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:14 - r - INFO: - Episode: 990/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:15 - r - INFO: - Episode: 991/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:16 - r - INFO: - Current episode 991 has the best eval reward: 200.0 -2022-10-30 21:24:16 - r - INFO: - Episode: 992/1000, Reward: 198.0, Step: 198 -2022-10-30 21:24:17 - r - INFO: - Current episode 992 has the best eval reward: 200.0 -2022-10-30 21:24:18 - r - INFO: - Episode: 993/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:18 - r - INFO: - Current episode 993 has the best eval reward: 200.0 -2022-10-30 21:24:19 - r - INFO: - Episode: 994/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:19 - r - INFO: - Current episode 994 has the best eval reward: 200.0 -2022-10-30 21:24:20 - r - INFO: - Episode: 995/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:21 - r - INFO: - Episode: 996/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:22 - r - INFO: - Episode: 997/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:23 - r - INFO: - Episode: 998/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:24 - r - INFO: - Current episode 998 has the best eval reward: 200.0 -2022-10-30 21:24:25 - r - INFO: - Episode: 999/1000, Reward: 200.0, Step: 200 -2022-10-30 21:24:26 - r - INFO: - Episode: 1000/1000, Reward: 200.0, Step: 200 diff --git a/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/models/actor_checkpoint.pt b/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/models/actor_checkpoint.pt deleted file mode 100644 index 89d0854..0000000 Binary files a/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/models/actor_checkpoint.pt and /dev/null differ diff --git a/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/models/critic_checkpoint.pt b/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/models/critic_checkpoint.pt deleted file mode 100644 index 720f388..0000000 Binary files a/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/models/critic_checkpoint.pt and /dev/null differ diff --git a/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/results/learning_curve.png b/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/results/learning_curve.png deleted file mode 100644 index 8bbfcde..0000000 Binary files a/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/results/res.csv b/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/results/res.csv deleted file mode 100644 index 6f853b9..0000000 --- a/projects/codes/A2C/Train_CartPole-v1_A2C_20221030-211435/results/res.csv +++ /dev/null @@ -1,1001 +0,0 @@ -episodes,rewards,steps -0,25.0,25 -1,13.0,13 -2,58.0,58 -3,10.0,10 -4,39.0,39 -5,39.0,39 -6,25.0,25 -7,22.0,22 -8,21.0,21 -9,27.0,27 -10,35.0,35 -11,26.0,26 -12,38.0,38 -13,29.0,29 -14,50.0,50 -15,20.0,20 -16,52.0,52 -17,12.0,12 -18,20.0,20 -19,38.0,38 -20,22.0,22 -21,36.0,36 -22,20.0,20 -23,35.0,35 -24,90.0,90 -25,29.0,29 -26,16.0,16 -27,25.0,25 -28,46.0,46 -29,33.0,33 -30,11.0,11 -31,27.0,27 -32,32.0,32 -33,21.0,21 -34,11.0,11 -35,21.0,21 -36,51.0,51 -37,29.0,29 -38,50.0,50 -39,19.0,19 -40,41.0,41 -41,28.0,28 -42,71.0,71 -43,45.0,45 -44,42.0,42 -45,39.0,39 -46,21.0,21 -47,14.0,14 -48,23.0,23 -49,21.0,21 -50,34.0,34 -51,14.0,14 -52,41.0,41 -53,99.0,99 -54,21.0,21 -55,52.0,52 -56,34.0,34 -57,73.0,73 -58,21.0,21 -59,27.0,27 -60,51.0,51 -61,46.0,46 -62,21.0,21 -63,20.0,20 -64,44.0,44 -65,16.0,16 -66,39.0,39 -67,30.0,30 -68,37.0,37 -69,20.0,20 -70,21.0,21 -71,13.0,13 -72,65.0,65 -73,45.0,45 -74,45.0,45 -75,46.0,46 -76,13.0,13 -77,33.0,33 -78,30.0,30 -79,52.0,52 -80,27.0,27 -81,30.0,30 -82,47.0,47 -83,56.0,56 -84,19.0,19 -85,33.0,33 -86,25.0,25 -87,41.0,41 -88,20.0,20 -89,58.0,58 -90,35.0,35 -91,23.0,23 -92,12.0,12 -93,20.0,20 -94,10.0,10 -95,49.0,49 -96,29.0,29 -97,35.0,35 -98,36.0,36 -99,36.0,36 -100,16.0,16 -101,36.0,36 -102,30.0,30 -103,76.0,76 -104,52.0,52 -105,39.0,39 -106,52.0,52 -107,69.0,69 -108,27.0,27 -109,14.0,14 -110,28.0,28 -111,12.0,12 -112,26.0,26 -113,50.0,50 -114,25.0,25 -115,53.0,53 -116,19.0,19 -117,33.0,33 -118,34.0,34 -119,41.0,41 -120,25.0,25 -121,18.0,18 -122,114.0,114 -123,25.0,25 -124,46.0,46 -125,22.0,22 -126,71.0,71 -127,30.0,30 -128,130.0,130 -129,65.0,65 -130,55.0,55 -131,37.0,37 -132,46.0,46 -133,65.0,65 -134,31.0,31 -135,33.0,33 -136,39.0,39 -137,73.0,73 -138,78.0,78 -139,36.0,36 -140,56.0,56 -141,12.0,12 -142,36.0,36 -143,13.0,13 -144,85.0,85 -145,34.0,34 -146,16.0,16 -147,68.0,68 -148,94.0,94 -149,17.0,17 -150,64.0,64 -151,33.0,33 -152,63.0,63 -153,39.0,39 -154,72.0,72 -155,39.0,39 -156,37.0,37 -157,18.0,18 -158,55.0,55 -159,21.0,21 -160,54.0,54 -161,46.0,46 -162,21.0,21 -163,26.0,26 -164,70.0,70 -165,20.0,20 -166,41.0,41 -167,77.0,77 -168,13.0,13 -169,66.0,66 -170,72.0,72 -171,28.0,28 -172,68.0,68 -173,124.0,124 -174,41.0,41 -175,54.0,54 -176,33.0,33 -177,92.0,92 -178,23.0,23 -179,76.0,76 -180,47.0,47 -181,89.0,89 -182,84.0,84 -183,75.0,75 -184,64.0,64 -185,35.0,35 -186,44.0,44 -187,46.0,46 -188,67.0,67 -189,82.0,82 -190,55.0,55 -191,26.0,26 -192,116.0,116 -193,116.0,116 -194,119.0,119 -195,50.0,50 -196,43.0,43 -197,47.0,47 -198,71.0,71 -199,53.0,53 -200,137.0,137 -201,82.0,82 -202,120.0,120 -203,69.0,69 -204,55.0,55 -205,62.0,62 -206,64.0,64 -207,49.0,49 -208,32.0,32 -209,42.0,42 -210,50.0,50 -211,93.0,93 -212,60.0,60 -213,54.0,54 -214,68.0,68 -215,84.0,84 -216,55.0,55 -217,70.0,70 -218,115.0,115 -219,149.0,149 -220,68.0,68 -221,50.0,50 -222,56.0,56 -223,61.0,61 -224,117.0,117 -225,66.0,66 -226,127.0,127 -227,66.0,66 -228,48.0,48 -229,36.0,36 -230,79.0,79 -231,49.0,49 -232,55.0,55 -233,41.0,41 -234,20.0,20 -235,40.0,40 -236,120.0,120 -237,27.0,27 -238,51.0,51 -239,35.0,35 -240,43.0,43 -241,54.0,54 -242,52.0,52 -243,47.0,47 -244,63.0,63 -245,29.0,29 -246,36.0,36 -247,58.0,58 -248,63.0,63 -249,49.0,49 -250,70.0,70 -251,114.0,114 -252,62.0,62 -253,73.0,73 -254,62.0,62 -255,61.0,61 -256,115.0,115 -257,50.0,50 -258,128.0,128 -259,200.0,200 -260,75.0,75 -261,64.0,64 -262,33.0,33 -263,90.0,90 -264,117.0,117 -265,60.0,60 -266,177.0,177 -267,39.0,39 -268,40.0,40 -269,109.0,109 -270,100.0,100 -271,99.0,99 -272,136.0,136 -273,62.0,62 -274,100.0,100 -275,73.0,73 -276,166.0,166 -277,74.0,74 -278,126.0,126 -279,111.0,111 -280,198.0,198 -281,106.0,106 -282,80.0,80 -283,74.0,74 -284,114.0,114 -285,69.0,69 -286,98.0,98 -287,63.0,63 -288,61.0,61 -289,49.0,49 -290,89.0,89 -291,114.0,114 -292,103.0,103 -293,103.0,103 -294,93.0,93 -295,137.0,137 -296,97.0,97 -297,124.0,124 -298,147.0,147 -299,125.0,125 -300,105.0,105 -301,113.0,113 -302,120.0,120 -303,159.0,159 -304,190.0,190 -305,119.0,119 -306,200.0,200 -307,148.0,148 -308,200.0,200 -309,79.0,79 -310,115.0,115 -311,147.0,147 -312,112.0,112 -313,125.0,125 -314,184.0,184 -315,193.0,193 -316,117.0,117 -317,153.0,153 -318,125.0,125 -319,184.0,184 -320,173.0,173 -321,117.0,117 -322,47.0,47 -323,107.0,107 -324,104.0,104 -325,114.0,114 -326,90.0,90 -327,112.0,112 -328,70.0,70 -329,74.0,74 -330,159.0,159 -331,39.0,39 -332,129.0,129 -333,50.0,50 -334,74.0,74 -335,31.0,31 -336,57.0,57 -337,71.0,71 -338,43.0,43 -339,41.0,41 -340,64.0,64 -341,38.0,38 -342,45.0,45 -343,120.0,120 -344,40.0,40 -345,46.0,46 -346,57.0,57 -347,29.0,29 -348,29.0,29 -349,50.0,50 -350,38.0,38 -351,51.0,51 -352,49.0,49 -353,30.0,30 -354,40.0,40 -355,45.0,45 -356,68.0,68 -357,27.0,27 -358,18.0,18 -359,26.0,26 -360,15.0,15 -361,65.0,65 -362,38.0,38 -363,41.0,41 -364,61.0,61 -365,113.0,113 -366,39.0,39 -367,60.0,60 -368,134.0,134 -369,122.0,122 -370,34.0,34 -371,129.0,129 -372,40.0,40 -373,128.0,128 -374,200.0,200 -375,108.0,108 -376,108.0,108 -377,151.0,151 -378,79.0,79 -379,105.0,105 -380,87.0,87 -381,94.0,94 -382,112.0,112 -383,200.0,200 -384,184.0,184 -385,124.0,124 -386,200.0,200 -387,200.0,200 -388,109.0,109 -389,88.0,88 -390,104.0,104 -391,200.0,200 -392,84.0,84 -393,187.0,187 -394,182.0,182 -395,148.0,148 -396,86.0,86 -397,200.0,200 -398,199.0,199 -399,200.0,200 -400,92.0,92 -401,112.0,112 -402,86.0,86 -403,114.0,114 -404,90.0,90 -405,101.0,101 -406,111.0,111 -407,107.0,107 -408,120.0,120 -409,114.0,114 -410,97.0,97 -411,95.0,95 -412,126.0,126 -413,111.0,111 -414,120.0,120 -415,178.0,178 -416,97.0,97 -417,144.0,144 -418,200.0,200 -419,190.0,190 -420,29.0,29 -421,200.0,200 -422,116.0,116 -423,200.0,200 -424,107.0,107 -425,128.0,128 -426,164.0,164 -427,30.0,30 -428,122.0,122 -429,110.0,110 -430,105.0,105 -431,137.0,137 -432,110.0,110 -433,111.0,111 -434,33.0,33 -435,100.0,100 -436,131.0,131 -437,99.0,99 -438,118.0,118 -439,98.0,98 -440,119.0,119 -441,41.0,41 -442,107.0,107 -443,41.0,41 -444,113.0,113 -445,113.0,113 -446,117.0,117 -447,140.0,140 -448,133.0,133 -449,108.0,108 -450,117.0,117 -451,40.0,40 -452,108.0,108 -453,140.0,140 -454,133.0,133 -455,115.0,115 -456,30.0,30 -457,119.0,119 -458,160.0,160 -459,125.0,125 -460,161.0,161 -461,139.0,139 -462,190.0,190 -463,149.0,149 -464,173.0,173 -465,165.0,165 -466,82.0,82 -467,197.0,197 -468,200.0,200 -469,200.0,200 -470,200.0,200 -471,182.0,182 -472,118.0,118 -473,200.0,200 -474,200.0,200 -475,93.0,93 -476,200.0,200 -477,200.0,200 -478,167.0,167 -479,200.0,200 -480,200.0,200 -481,200.0,200 -482,190.0,190 -483,86.0,86 -484,166.0,166 -485,200.0,200 -486,200.0,200 -487,172.0,172 -488,200.0,200 -489,102.0,102 -490,194.0,194 -491,200.0,200 -492,179.0,179 -493,187.0,187 -494,200.0,200 -495,89.0,89 -496,169.0,169 -497,28.0,28 -498,160.0,160 -499,140.0,140 -500,37.0,37 -501,32.0,32 -502,129.0,129 -503,22.0,22 -504,124.0,124 -505,24.0,24 -506,115.0,115 -507,24.0,24 -508,38.0,38 -509,24.0,24 -510,23.0,23 -511,125.0,125 -512,22.0,22 -513,24.0,24 -514,20.0,20 -515,25.0,25 -516,31.0,31 -517,23.0,23 -518,30.0,30 -519,101.0,101 -520,25.0,25 -521,22.0,22 -522,20.0,20 -523,16.0,16 -524,104.0,104 -525,17.0,17 -526,108.0,108 -527,121.0,121 -528,29.0,29 -529,29.0,29 -530,43.0,43 -531,105.0,105 -532,130.0,130 -533,30.0,30 -534,31.0,31 -535,30.0,30 -536,37.0,37 -537,115.0,115 -538,110.0,110 -539,112.0,112 -540,33.0,33 -541,120.0,120 -542,109.0,109 -543,122.0,122 -544,115.0,115 -545,34.0,34 -546,28.0,28 -547,29.0,29 -548,113.0,113 -549,100.0,100 -550,26.0,26 -551,24.0,24 -552,26.0,26 -553,102.0,102 -554,18.0,18 -555,107.0,107 -556,27.0,27 -557,87.0,87 -558,29.0,29 -559,31.0,31 -560,112.0,112 -561,112.0,112 -562,108.0,108 -563,98.0,98 -564,104.0,104 -565,116.0,116 -566,123.0,123 -567,105.0,105 -568,133.0,133 -569,116.0,116 -570,128.0,128 -571,130.0,130 -572,113.0,113 -573,143.0,143 -574,145.0,145 -575,159.0,159 -576,150.0,150 -577,130.0,130 -578,145.0,145 -579,173.0,173 -580,154.0,154 -581,131.0,131 -582,163.0,163 -583,160.0,160 -584,181.0,181 -585,161.0,161 -586,169.0,169 -587,150.0,150 -588,176.0,176 -589,157.0,157 -590,167.0,167 -591,168.0,168 -592,135.0,135 -593,157.0,157 -594,138.0,138 -595,139.0,139 -596,146.0,146 -597,121.0,121 -598,140.0,140 -599,124.0,124 -600,124.0,124 -601,115.0,115 -602,129.0,129 -603,107.0,107 -604,118.0,118 -605,108.0,108 -606,102.0,102 -607,105.0,105 -608,103.0,103 -609,96.0,96 -610,116.0,116 -611,51.0,51 -612,100.0,100 -613,121.0,121 -614,109.0,109 -615,85.0,85 -616,111.0,111 -617,91.0,91 -618,127.0,127 -619,117.0,117 -620,104.0,104 -621,119.0,119 -622,111.0,111 -623,132.0,132 -624,130.0,130 -625,140.0,140 -626,95.0,95 -627,106.0,106 -628,120.0,120 -629,111.0,111 -630,114.0,114 -631,126.0,126 -632,100.0,100 -633,111.0,111 -634,104.0,104 -635,103.0,103 -636,111.0,111 -637,110.0,110 -638,131.0,131 -639,90.0,90 -640,97.0,97 -641,104.0,104 -642,91.0,91 -643,97.0,97 -644,109.0,109 -645,112.0,112 -646,97.0,97 -647,32.0,32 -648,94.0,94 -649,107.0,107 -650,61.0,61 -651,97.0,97 -652,99.0,99 -653,76.0,76 -654,38.0,38 -655,96.0,96 -656,96.0,96 -657,65.0,65 -658,45.0,45 -659,91.0,91 -660,78.0,78 -661,90.0,90 -662,92.0,92 -663,94.0,94 -664,101.0,101 -665,111.0,111 -666,109.0,109 -667,99.0,99 -668,115.0,115 -669,112.0,112 -670,113.0,113 -671,110.0,110 -672,108.0,108 -673,112.0,112 -674,125.0,125 -675,122.0,122 -676,114.0,114 -677,127.0,127 -678,125.0,125 -679,112.0,112 -680,111.0,111 -681,124.0,124 -682,113.0,113 -683,103.0,103 -684,119.0,119 -685,120.0,120 -686,95.0,95 -687,100.0,100 -688,29.0,29 -689,119.0,119 -690,107.0,107 -691,117.0,117 -692,78.0,78 -693,35.0,35 -694,101.0,101 -695,98.0,98 -696,94.0,94 -697,102.0,102 -698,90.0,90 -699,86.0,86 -700,81.0,81 -701,105.0,105 -702,72.0,72 -703,100.0,100 -704,96.0,96 -705,111.0,111 -706,27.0,27 -707,107.0,107 -708,87.0,87 -709,114.0,114 -710,111.0,111 -711,88.0,88 -712,112.0,112 -713,108.0,108 -714,108.0,108 -715,103.0,103 -716,120.0,120 -717,116.0,116 -718,112.0,112 -719,99.0,99 -720,118.0,118 -721,114.0,114 -722,104.0,104 -723,99.0,99 -724,102.0,102 -725,106.0,106 -726,31.0,31 -727,91.0,91 -728,32.0,32 -729,96.0,96 -730,20.0,20 -731,33.0,33 -732,23.0,23 -733,80.0,80 -734,35.0,35 -735,88.0,88 -736,28.0,28 -737,26.0,26 -738,70.0,70 -739,86.0,86 -740,28.0,28 -741,39.0,39 -742,65.0,65 -743,52.0,52 -744,43.0,43 -745,97.0,97 -746,27.0,27 -747,89.0,89 -748,34.0,34 -749,35.0,35 -750,28.0,28 -751,96.0,96 -752,97.0,97 -753,108.0,108 -754,45.0,45 -755,103.0,103 -756,97.0,97 -757,114.0,114 -758,103.0,103 -759,116.0,116 -760,127.0,127 -761,122.0,122 -762,112.0,112 -763,112.0,112 -764,120.0,120 -765,129.0,129 -766,127.0,127 -767,125.0,125 -768,124.0,124 -769,126.0,126 -770,129.0,129 -771,129.0,129 -772,43.0,43 -773,121.0,121 -774,40.0,40 -775,116.0,116 -776,117.0,117 -777,113.0,113 -778,117.0,117 -779,108.0,108 -780,108.0,108 -781,119.0,119 -782,109.0,109 -783,116.0,116 -784,114.0,114 -785,45.0,45 -786,116.0,116 -787,116.0,116 -788,110.0,110 -789,105.0,105 -790,110.0,110 -791,112.0,112 -792,104.0,104 -793,120.0,120 -794,110.0,110 -795,113.0,113 -796,33.0,33 -797,111.0,111 -798,31.0,31 -799,139.0,139 -800,110.0,110 -801,124.0,124 -802,120.0,120 -803,112.0,112 -804,116.0,116 -805,105.0,105 -806,125.0,125 -807,103.0,103 -808,122.0,122 -809,109.0,109 -810,118.0,118 -811,124.0,124 -812,115.0,115 -813,26.0,26 -814,118.0,118 -815,118.0,118 -816,31.0,31 -817,99.0,99 -818,122.0,122 -819,102.0,102 -820,111.0,111 -821,110.0,110 -822,113.0,113 -823,117.0,117 -824,113.0,113 -825,109.0,109 -826,122.0,122 -827,117.0,117 -828,127.0,127 -829,113.0,113 -830,118.0,118 -831,107.0,107 -832,108.0,108 -833,103.0,103 -834,126.0,126 -835,131.0,131 -836,106.0,106 -837,116.0,116 -838,24.0,24 -839,107.0,107 -840,124.0,124 -841,125.0,125 -842,110.0,110 -843,112.0,112 -844,105.0,105 -845,104.0,104 -846,134.0,134 -847,107.0,107 -848,128.0,128 -849,113.0,113 -850,138.0,138 -851,118.0,118 -852,142.0,142 -853,118.0,118 -854,122.0,122 -855,130.0,130 -856,126.0,126 -857,111.0,111 -858,114.0,114 -859,128.0,128 -860,126.0,126 -861,143.0,143 -862,132.0,132 -863,123.0,123 -864,111.0,111 -865,129.0,129 -866,121.0,121 -867,114.0,114 -868,110.0,110 -869,118.0,118 -870,120.0,120 -871,109.0,109 -872,106.0,106 -873,118.0,118 -874,104.0,104 -875,98.0,98 -876,115.0,115 -877,34.0,34 -878,96.0,96 -879,108.0,108 -880,105.0,105 -881,33.0,33 -882,105.0,105 -883,111.0,111 -884,112.0,112 -885,101.0,101 -886,25.0,25 -887,35.0,35 -888,99.0,99 -889,105.0,105 -890,36.0,36 -891,92.0,92 -892,104.0,104 -893,111.0,111 -894,106.0,106 -895,109.0,109 -896,108.0,108 -897,101.0,101 -898,100.0,100 -899,33.0,33 -900,119.0,119 -901,112.0,112 -902,112.0,112 -903,126.0,126 -904,123.0,123 -905,125.0,125 -906,107.0,107 -907,128.0,128 -908,119.0,119 -909,142.0,142 -910,117.0,117 -911,125.0,125 -912,141.0,141 -913,134.0,134 -914,131.0,131 -915,131.0,131 -916,140.0,140 -917,115.0,115 -918,142.0,142 -919,142.0,142 -920,128.0,128 -921,139.0,139 -922,133.0,133 -923,129.0,129 -924,124.0,124 -925,131.0,131 -926,125.0,125 -927,146.0,146 -928,118.0,118 -929,126.0,126 -930,134.0,134 -931,155.0,155 -932,134.0,134 -933,136.0,136 -934,146.0,146 -935,150.0,150 -936,167.0,167 -937,135.0,135 -938,197.0,197 -939,190.0,190 -940,170.0,170 -941,179.0,179 -942,192.0,192 -943,200.0,200 -944,200.0,200 -945,200.0,200 -946,200.0,200 -947,200.0,200 -948,200.0,200 -949,200.0,200 -950,200.0,200 -951,200.0,200 -952,200.0,200 -953,200.0,200 -954,200.0,200 -955,200.0,200 -956,200.0,200 -957,200.0,200 -958,200.0,200 -959,200.0,200 -960,200.0,200 -961,200.0,200 -962,200.0,200 -963,200.0,200 -964,200.0,200 -965,200.0,200 -966,200.0,200 -967,200.0,200 -968,200.0,200 -969,200.0,200 -970,200.0,200 -971,200.0,200 -972,200.0,200 -973,200.0,200 -974,200.0,200 -975,200.0,200 -976,200.0,200 -977,200.0,200 -978,200.0,200 -979,200.0,200 -980,200.0,200 -981,200.0,200 -982,200.0,200 -983,200.0,200 -984,200.0,200 -985,200.0,200 -986,200.0,200 -987,200.0,200 -988,200.0,200 -989,200.0,200 -990,200.0,200 -991,198.0,198 -992,200.0,200 -993,200.0,200 -994,200.0,200 -995,200.0,200 -996,200.0,200 -997,200.0,200 -998,200.0,200 -999,200.0,200 diff --git a/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/config.yaml b/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/config.yaml deleted file mode 100644 index 49d9701..0000000 --- a/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/config.yaml +++ /dev/null @@ -1,24 +0,0 @@ -general_cfg: - algo_name: A2C - device: cuda - env_name: CartPole-v1 - eval_eps: 10 - eval_per_episode: 5 - load_checkpoint: false - load_path: tasks - max_steps: 200 - mode: train - save_fig: true - seed: 1 - show_fig: false - test_eps: 20 - train_eps: 1000 -algo_cfg: - actor_hidden_dim: 256 - actor_lr: 0.0003 - batch_size: 64 - buffer_size: 100000 - critic_hidden_dim: 256 - critic_lr: 0.001 - gamma: 0.99 - hidden_dim: 256 diff --git a/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/logs/log.txt b/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/logs/log.txt deleted file mode 100644 index 18436c8..0000000 --- a/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/logs/log.txt +++ /dev/null @@ -1,1086 +0,0 @@ -2022-10-31 23:21:38 - r - INFO: - n_states: 4, n_actions: 2 -2022-10-31 23:21:38 - r - INFO: - Actor model name: ActorSoftmaxTanh -2022-10-31 23:21:38 - r - INFO: - Critic model name: Critic -2022-10-31 23:21:38 - r - INFO: - ACMemory memory name: PGReplay -2022-10-31 23:21:38 - r - INFO: - agent name: A2C -2022-10-31 23:21:38 - r - INFO: - Start training! -2022-10-31 23:21:38 - r - INFO: - Env: CartPole-v1, Algorithm: A2C, Device: cuda -2022-10-31 23:21:40 - r - INFO: - Episode: 1/1000, Reward: 25.0, Step: 25 -2022-10-31 23:21:40 - r - INFO: - Episode: 2/1000, Reward: 11.0, Step: 11 -2022-10-31 23:21:41 - r - INFO: - Episode: 3/1000, Reward: 32.0, Step: 32 -2022-10-31 23:21:41 - r - INFO: - Episode: 4/1000, Reward: 11.0, Step: 11 -2022-10-31 23:21:41 - r - INFO: - Episode: 5/1000, Reward: 14.0, Step: 14 -2022-10-31 23:21:41 - r - INFO: - Current episode 5 has the best eval reward: 19.90 -2022-10-31 23:21:41 - r - INFO: - Episode: 6/1000, Reward: 11.0, Step: 11 -2022-10-31 23:21:41 - r - INFO: - Episode: 7/1000, Reward: 23.0, Step: 23 -2022-10-31 23:21:41 - r - INFO: - Episode: 8/1000, Reward: 27.0, Step: 27 -2022-10-31 23:21:41 - r - INFO: - Episode: 9/1000, Reward: 10.0, Step: 10 -2022-10-31 23:21:41 - r - INFO: - Episode: 10/1000, Reward: 21.0, Step: 21 -2022-10-31 23:21:41 - r - INFO: - Episode: 11/1000, Reward: 15.0, Step: 15 -2022-10-31 23:21:41 - r - INFO: - Episode: 12/1000, Reward: 26.0, Step: 26 -2022-10-31 23:21:41 - r - INFO: - Episode: 13/1000, Reward: 22.0, Step: 22 -2022-10-31 23:21:41 - r - INFO: - Episode: 14/1000, Reward: 14.0, Step: 14 -2022-10-31 23:21:41 - r - INFO: - Episode: 15/1000, Reward: 14.0, Step: 14 -2022-10-31 23:21:41 - r - INFO: - Episode: 16/1000, Reward: 21.0, Step: 21 -2022-10-31 23:21:41 - r - INFO: - Episode: 17/1000, Reward: 10.0, Step: 10 -2022-10-31 23:21:42 - r - INFO: - Episode: 18/1000, Reward: 19.0, Step: 19 -2022-10-31 23:21:42 - r - INFO: - Episode: 19/1000, Reward: 18.0, Step: 18 -2022-10-31 23:21:42 - r - INFO: - Episode: 20/1000, Reward: 26.0, Step: 26 -2022-10-31 23:21:42 - r - INFO: - Current episode 20 has the best eval reward: 21.50 -2022-10-31 23:21:42 - r - INFO: - Episode: 21/1000, Reward: 29.0, Step: 29 -2022-10-31 23:21:42 - r - INFO: - Episode: 22/1000, Reward: 40.0, Step: 40 -2022-10-31 23:21:42 - r - INFO: - Episode: 23/1000, Reward: 35.0, Step: 35 -2022-10-31 23:21:42 - r - INFO: - Episode: 24/1000, Reward: 33.0, Step: 33 -2022-10-31 23:21:42 - r - INFO: - Episode: 25/1000, Reward: 47.0, Step: 47 -2022-10-31 23:21:43 - r - INFO: - Current episode 25 has the best eval reward: 31.90 -2022-10-31 23:21:43 - r - INFO: - Episode: 26/1000, Reward: 50.0, Step: 50 -2022-10-31 23:21:43 - r - INFO: - Episode: 27/1000, Reward: 21.0, Step: 21 -2022-10-31 23:21:43 - r - INFO: - Episode: 28/1000, Reward: 30.0, Step: 30 -2022-10-31 23:21:43 - r - INFO: - Episode: 29/1000, Reward: 26.0, Step: 26 -2022-10-31 23:21:43 - r - INFO: - Episode: 30/1000, Reward: 40.0, Step: 40 -2022-10-31 23:21:43 - r - INFO: - Current episode 30 has the best eval reward: 56.70 -2022-10-31 23:21:43 - r - INFO: - Episode: 31/1000, Reward: 31.0, Step: 31 -2022-10-31 23:21:43 - r - INFO: - Episode: 32/1000, Reward: 54.0, Step: 54 -2022-10-31 23:21:43 - r - INFO: - Episode: 33/1000, Reward: 59.0, Step: 59 -2022-10-31 23:21:44 - r - INFO: - Episode: 34/1000, Reward: 50.0, Step: 50 -2022-10-31 23:21:44 - r - INFO: - Episode: 35/1000, Reward: 26.0, Step: 26 -2022-10-31 23:21:44 - r - INFO: - Episode: 36/1000, Reward: 34.0, Step: 34 -2022-10-31 23:21:44 - r - INFO: - Episode: 37/1000, Reward: 25.0, Step: 25 -2022-10-31 23:21:44 - r - INFO: - Episode: 38/1000, Reward: 166.0, Step: 166 -2022-10-31 23:21:44 - r - INFO: - Episode: 39/1000, Reward: 35.0, Step: 35 -2022-10-31 23:21:44 - r - INFO: - Episode: 40/1000, Reward: 25.0, Step: 25 -2022-10-31 23:21:45 - r - INFO: - Episode: 41/1000, Reward: 110.0, Step: 110 -2022-10-31 23:21:45 - r - INFO: - Episode: 42/1000, Reward: 22.0, Step: 22 -2022-10-31 23:21:45 - r - INFO: - Episode: 43/1000, Reward: 57.0, Step: 57 -2022-10-31 23:21:45 - r - INFO: - Episode: 44/1000, Reward: 45.0, Step: 45 -2022-10-31 23:21:45 - r - INFO: - Episode: 45/1000, Reward: 35.0, Step: 35 -2022-10-31 23:21:45 - r - INFO: - Episode: 46/1000, Reward: 45.0, Step: 45 -2022-10-31 23:21:45 - r - INFO: - Episode: 47/1000, Reward: 51.0, Step: 51 -2022-10-31 23:21:46 - r - INFO: - Episode: 48/1000, Reward: 32.0, Step: 32 -2022-10-31 23:21:46 - r - INFO: - Episode: 49/1000, Reward: 67.0, Step: 67 -2022-10-31 23:21:46 - r - INFO: - Episode: 50/1000, Reward: 46.0, Step: 46 -2022-10-31 23:21:46 - r - INFO: - Episode: 51/1000, Reward: 61.0, Step: 61 -2022-10-31 23:21:46 - r - INFO: - Episode: 52/1000, Reward: 49.0, Step: 49 -2022-10-31 23:21:46 - r - INFO: - Episode: 53/1000, Reward: 47.0, Step: 47 -2022-10-31 23:21:46 - r - INFO: - Episode: 54/1000, Reward: 37.0, Step: 37 -2022-10-31 23:21:46 - r - INFO: - Episode: 55/1000, Reward: 32.0, Step: 32 -2022-10-31 23:21:47 - r - INFO: - Current episode 55 has the best eval reward: 85.50 -2022-10-31 23:21:47 - r - INFO: - Episode: 56/1000, Reward: 31.0, Step: 31 -2022-10-31 23:21:47 - r - INFO: - Episode: 57/1000, Reward: 33.0, Step: 33 -2022-10-31 23:21:47 - r - INFO: - Episode: 58/1000, Reward: 93.0, Step: 93 -2022-10-31 23:21:47 - r - INFO: - Episode: 59/1000, Reward: 60.0, Step: 60 -2022-10-31 23:21:48 - r - INFO: - Episode: 60/1000, Reward: 128.0, Step: 128 -2022-10-31 23:21:48 - r - INFO: - Episode: 61/1000, Reward: 200.0, Step: 200 -2022-10-31 23:21:48 - r - INFO: - Episode: 62/1000, Reward: 47.0, Step: 47 -2022-10-31 23:21:48 - r - INFO: - Episode: 63/1000, Reward: 47.0, Step: 47 -2022-10-31 23:21:49 - r - INFO: - Episode: 64/1000, Reward: 63.0, Step: 63 -2022-10-31 23:21:49 - r - INFO: - Episode: 65/1000, Reward: 68.0, Step: 68 -2022-10-31 23:21:49 - r - INFO: - Episode: 66/1000, Reward: 45.0, Step: 45 -2022-10-31 23:21:49 - r - INFO: - Episode: 67/1000, Reward: 101.0, Step: 101 -2022-10-31 23:21:49 - r - INFO: - Episode: 68/1000, Reward: 47.0, Step: 47 -2022-10-31 23:21:49 - r - INFO: - Episode: 69/1000, Reward: 49.0, Step: 49 -2022-10-31 23:21:50 - r - INFO: - Episode: 70/1000, Reward: 54.0, Step: 54 -2022-10-31 23:21:50 - r - INFO: - Episode: 71/1000, Reward: 42.0, Step: 42 -2022-10-31 23:21:50 - r - INFO: - Episode: 72/1000, Reward: 77.0, Step: 77 -2022-10-31 23:21:50 - r - INFO: - Episode: 73/1000, Reward: 67.0, Step: 67 -2022-10-31 23:21:50 - r - INFO: - Episode: 74/1000, Reward: 41.0, Step: 41 -2022-10-31 23:21:51 - r - INFO: - Episode: 75/1000, Reward: 89.0, Step: 89 -2022-10-31 23:21:51 - r - INFO: - Episode: 76/1000, Reward: 51.0, Step: 51 -2022-10-31 23:21:51 - r - INFO: - Episode: 77/1000, Reward: 54.0, Step: 54 -2022-10-31 23:21:51 - r - INFO: - Episode: 78/1000, Reward: 37.0, Step: 37 -2022-10-31 23:21:51 - r - INFO: - Episode: 79/1000, Reward: 49.0, Step: 49 -2022-10-31 23:21:51 - r - INFO: - Episode: 80/1000, Reward: 46.0, Step: 46 -2022-10-31 23:21:52 - r - INFO: - Episode: 81/1000, Reward: 31.0, Step: 31 -2022-10-31 23:21:52 - r - INFO: - Episode: 82/1000, Reward: 43.0, Step: 43 -2022-10-31 23:21:52 - r - INFO: - Episode: 83/1000, Reward: 60.0, Step: 60 -2022-10-31 23:21:52 - r - INFO: - Episode: 84/1000, Reward: 41.0, Step: 41 -2022-10-31 23:21:52 - r - INFO: - Episode: 85/1000, Reward: 40.0, Step: 40 -2022-10-31 23:21:52 - r - INFO: - Episode: 86/1000, Reward: 28.0, Step: 28 -2022-10-31 23:21:52 - r - INFO: - Episode: 87/1000, Reward: 50.0, Step: 50 -2022-10-31 23:21:53 - r - INFO: - Episode: 88/1000, Reward: 159.0, Step: 159 -2022-10-31 23:21:53 - r - INFO: - Episode: 89/1000, Reward: 30.0, Step: 30 -2022-10-31 23:21:53 - r - INFO: - Episode: 90/1000, Reward: 34.0, Step: 34 -2022-10-31 23:21:53 - r - INFO: - Episode: 91/1000, Reward: 70.0, Step: 70 -2022-10-31 23:21:53 - r - INFO: - Episode: 92/1000, Reward: 22.0, Step: 22 -2022-10-31 23:21:53 - r - INFO: - Episode: 93/1000, Reward: 39.0, Step: 39 -2022-10-31 23:21:53 - r - INFO: - Episode: 94/1000, Reward: 50.0, Step: 50 -2022-10-31 23:21:53 - r - INFO: - Episode: 95/1000, Reward: 40.0, Step: 40 -2022-10-31 23:21:54 - r - INFO: - Episode: 96/1000, Reward: 37.0, Step: 37 -2022-10-31 23:21:54 - r - INFO: - Episode: 97/1000, Reward: 121.0, Step: 121 -2022-10-31 23:21:54 - r - INFO: - Episode: 98/1000, Reward: 26.0, Step: 26 -2022-10-31 23:21:54 - r - INFO: - Episode: 99/1000, Reward: 40.0, Step: 40 -2022-10-31 23:21:54 - r - INFO: - Episode: 100/1000, Reward: 30.0, Step: 30 -2022-10-31 23:21:55 - r - INFO: - Episode: 101/1000, Reward: 35.0, Step: 35 -2022-10-31 23:21:55 - r - INFO: - Episode: 102/1000, Reward: 40.0, Step: 40 -2022-10-31 23:21:55 - r - INFO: - Episode: 103/1000, Reward: 28.0, Step: 28 -2022-10-31 23:21:55 - r - INFO: - Episode: 104/1000, Reward: 29.0, Step: 29 -2022-10-31 23:21:55 - r - INFO: - Episode: 105/1000, Reward: 42.0, Step: 42 -2022-10-31 23:21:55 - r - INFO: - Episode: 106/1000, Reward: 54.0, Step: 54 -2022-10-31 23:21:55 - r - INFO: - Episode: 107/1000, Reward: 25.0, Step: 25 -2022-10-31 23:21:55 - r - INFO: - Episode: 108/1000, Reward: 47.0, Step: 47 -2022-10-31 23:21:55 - r - INFO: - Episode: 109/1000, Reward: 32.0, Step: 32 -2022-10-31 23:21:55 - r - INFO: - Episode: 110/1000, Reward: 50.0, Step: 50 -2022-10-31 23:21:56 - r - INFO: - Episode: 111/1000, Reward: 30.0, Step: 30 -2022-10-31 23:21:56 - r - INFO: - Episode: 112/1000, Reward: 58.0, Step: 58 -2022-10-31 23:21:56 - r - INFO: - Episode: 113/1000, Reward: 32.0, Step: 32 -2022-10-31 23:21:56 - r - INFO: - Episode: 114/1000, Reward: 43.0, Step: 43 -2022-10-31 23:21:56 - r - INFO: - Episode: 115/1000, Reward: 57.0, Step: 57 -2022-10-31 23:21:56 - r - INFO: - Episode: 116/1000, Reward: 20.0, Step: 20 -2022-10-31 23:21:57 - r - INFO: - Episode: 117/1000, Reward: 48.0, Step: 48 -2022-10-31 23:21:57 - r - INFO: - Episode: 118/1000, Reward: 45.0, Step: 45 -2022-10-31 23:21:57 - r - INFO: - Episode: 119/1000, Reward: 47.0, Step: 47 -2022-10-31 23:21:57 - r - INFO: - Episode: 120/1000, Reward: 69.0, Step: 69 -2022-10-31 23:21:57 - r - INFO: - Episode: 121/1000, Reward: 34.0, Step: 34 -2022-10-31 23:21:57 - r - INFO: - Episode: 122/1000, Reward: 22.0, Step: 22 -2022-10-31 23:21:57 - r - INFO: - Episode: 123/1000, Reward: 22.0, Step: 22 -2022-10-31 23:21:57 - r - INFO: - Episode: 124/1000, Reward: 38.0, Step: 38 -2022-10-31 23:21:57 - r - INFO: - Episode: 125/1000, Reward: 36.0, Step: 36 -2022-10-31 23:21:58 - r - INFO: - Episode: 126/1000, Reward: 41.0, Step: 41 -2022-10-31 23:21:58 - r - INFO: - Episode: 127/1000, Reward: 28.0, Step: 28 -2022-10-31 23:21:58 - r - INFO: - Episode: 128/1000, Reward: 35.0, Step: 35 -2022-10-31 23:21:58 - r - INFO: - Episode: 129/1000, Reward: 48.0, Step: 48 -2022-10-31 23:21:58 - r - INFO: - Episode: 130/1000, Reward: 51.0, Step: 51 -2022-10-31 23:21:58 - r - INFO: - Episode: 131/1000, Reward: 51.0, Step: 51 -2022-10-31 23:21:58 - r - INFO: - Episode: 132/1000, Reward: 36.0, Step: 36 -2022-10-31 23:21:59 - r - INFO: - Episode: 133/1000, Reward: 45.0, Step: 45 -2022-10-31 23:21:59 - r - INFO: - Episode: 134/1000, Reward: 27.0, Step: 27 -2022-10-31 23:21:59 - r - INFO: - Episode: 135/1000, Reward: 40.0, Step: 40 -2022-10-31 23:21:59 - r - INFO: - Episode: 136/1000, Reward: 43.0, Step: 43 -2022-10-31 23:21:59 - r - INFO: - Episode: 137/1000, Reward: 64.0, Step: 64 -2022-10-31 23:21:59 - r - INFO: - Episode: 138/1000, Reward: 43.0, Step: 43 -2022-10-31 23:21:59 - r - INFO: - Episode: 139/1000, Reward: 37.0, Step: 37 -2022-10-31 23:21:59 - r - INFO: - Episode: 140/1000, Reward: 38.0, Step: 38 -2022-10-31 23:22:00 - r - INFO: - Episode: 141/1000, Reward: 69.0, Step: 69 -2022-10-31 23:22:00 - r - INFO: - Episode: 142/1000, Reward: 36.0, Step: 36 -2022-10-31 23:22:00 - r - INFO: - Episode: 143/1000, Reward: 28.0, Step: 28 -2022-10-31 23:22:00 - r - INFO: - Episode: 144/1000, Reward: 58.0, Step: 58 -2022-10-31 23:22:00 - r - INFO: - Episode: 145/1000, Reward: 43.0, Step: 43 -2022-10-31 23:22:00 - r - INFO: - Episode: 146/1000, Reward: 50.0, Step: 50 -2022-10-31 23:22:01 - r - INFO: - Episode: 147/1000, Reward: 30.0, Step: 30 -2022-10-31 23:22:01 - r - INFO: - Episode: 148/1000, Reward: 42.0, Step: 42 -2022-10-31 23:22:01 - r - INFO: - Episode: 149/1000, Reward: 42.0, Step: 42 -2022-10-31 23:22:01 - r - INFO: - Episode: 150/1000, Reward: 35.0, Step: 35 -2022-10-31 23:22:01 - r - INFO: - Episode: 151/1000, Reward: 67.0, Step: 67 -2022-10-31 23:22:01 - r - INFO: - Episode: 152/1000, Reward: 45.0, Step: 45 -2022-10-31 23:22:01 - r - INFO: - Episode: 153/1000, Reward: 28.0, Step: 28 -2022-10-31 23:22:01 - r - INFO: - Episode: 154/1000, Reward: 59.0, Step: 59 -2022-10-31 23:22:02 - r - INFO: - Episode: 155/1000, Reward: 64.0, Step: 64 -2022-10-31 23:22:02 - r - INFO: - Episode: 156/1000, Reward: 67.0, Step: 67 -2022-10-31 23:22:02 - r - INFO: - Episode: 157/1000, Reward: 41.0, Step: 41 -2022-10-31 23:22:02 - r - INFO: - Episode: 158/1000, Reward: 81.0, Step: 81 -2022-10-31 23:22:02 - r - INFO: - Episode: 159/1000, Reward: 76.0, Step: 76 -2022-10-31 23:22:02 - r - INFO: - Episode: 160/1000, Reward: 91.0, Step: 91 -2022-10-31 23:22:03 - r - INFO: - Episode: 161/1000, Reward: 119.0, Step: 119 -2022-10-31 23:22:03 - r - INFO: - Episode: 162/1000, Reward: 47.0, Step: 47 -2022-10-31 23:22:03 - r - INFO: - Episode: 163/1000, Reward: 64.0, Step: 64 -2022-10-31 23:22:03 - r - INFO: - Episode: 164/1000, Reward: 178.0, Step: 178 -2022-10-31 23:22:04 - r - INFO: - Episode: 165/1000, Reward: 97.0, Step: 97 -2022-10-31 23:22:04 - r - INFO: - Current episode 165 has the best eval reward: 104.10 -2022-10-31 23:22:04 - r - INFO: - Episode: 166/1000, Reward: 181.0, Step: 181 -2022-10-31 23:22:05 - r - INFO: - Episode: 167/1000, Reward: 166.0, Step: 166 -2022-10-31 23:22:05 - r - INFO: - Episode: 168/1000, Reward: 79.0, Step: 79 -2022-10-31 23:22:05 - r - INFO: - Episode: 169/1000, Reward: 141.0, Step: 141 -2022-10-31 23:22:06 - r - INFO: - Episode: 170/1000, Reward: 119.0, Step: 119 -2022-10-31 23:22:06 - r - INFO: - Current episode 170 has the best eval reward: 119.50 -2022-10-31 23:22:06 - r - INFO: - Episode: 171/1000, Reward: 81.0, Step: 81 -2022-10-31 23:22:06 - r - INFO: - Episode: 172/1000, Reward: 124.0, Step: 124 -2022-10-31 23:22:07 - r - INFO: - Episode: 173/1000, Reward: 150.0, Step: 150 -2022-10-31 23:22:07 - r - INFO: - Episode: 174/1000, Reward: 98.0, Step: 98 -2022-10-31 23:22:07 - r - INFO: - Episode: 175/1000, Reward: 164.0, Step: 164 -2022-10-31 23:22:08 - r - INFO: - Current episode 175 has the best eval reward: 132.00 -2022-10-31 23:22:08 - r - INFO: - Episode: 176/1000, Reward: 200.0, Step: 200 -2022-10-31 23:22:09 - r - INFO: - Episode: 177/1000, Reward: 115.0, Step: 115 -2022-10-31 23:22:09 - r - INFO: - Episode: 178/1000, Reward: 116.0, Step: 116 -2022-10-31 23:22:09 - r - INFO: - Episode: 179/1000, Reward: 160.0, Step: 160 -2022-10-31 23:22:09 - r - INFO: - Episode: 180/1000, Reward: 103.0, Step: 103 -2022-10-31 23:22:10 - r - INFO: - Current episode 180 has the best eval reward: 134.00 -2022-10-31 23:22:10 - r - INFO: - Episode: 181/1000, Reward: 181.0, Step: 181 -2022-10-31 23:22:11 - r - INFO: - Episode: 182/1000, Reward: 185.0, Step: 185 -2022-10-31 23:22:11 - r - INFO: - Episode: 183/1000, Reward: 93.0, Step: 93 -2022-10-31 23:22:11 - r - INFO: - Episode: 184/1000, Reward: 110.0, Step: 110 -2022-10-31 23:22:12 - r - INFO: - Episode: 185/1000, Reward: 200.0, Step: 200 -2022-10-31 23:22:12 - r - INFO: - Current episode 185 has the best eval reward: 155.50 -2022-10-31 23:22:13 - r - INFO: - Episode: 186/1000, Reward: 141.0, Step: 141 -2022-10-31 23:22:13 - r - INFO: - Episode: 187/1000, Reward: 150.0, Step: 150 -2022-10-31 23:22:13 - r - INFO: - Episode: 188/1000, Reward: 121.0, Step: 121 -2022-10-31 23:22:13 - r - INFO: - Episode: 189/1000, Reward: 110.0, Step: 110 -2022-10-31 23:22:14 - r - INFO: - Episode: 190/1000, Reward: 115.0, Step: 115 -2022-10-31 23:22:14 - r - INFO: - Episode: 191/1000, Reward: 114.0, Step: 114 -2022-10-31 23:22:14 - r - INFO: - Episode: 192/1000, Reward: 45.0, Step: 45 -2022-10-31 23:22:15 - r - INFO: - Episode: 193/1000, Reward: 125.0, Step: 125 -2022-10-31 23:22:15 - r - INFO: - Episode: 194/1000, Reward: 142.0, Step: 142 -2022-10-31 23:22:15 - r - INFO: - Episode: 195/1000, Reward: 54.0, Step: 54 -2022-10-31 23:22:16 - r - INFO: - Episode: 196/1000, Reward: 62.0, Step: 62 -2022-10-31 23:22:16 - r - INFO: - Episode: 197/1000, Reward: 122.0, Step: 122 -2022-10-31 23:22:16 - r - INFO: - Episode: 198/1000, Reward: 58.0, Step: 58 -2022-10-31 23:22:16 - r - INFO: - Episode: 199/1000, Reward: 88.0, Step: 88 -2022-10-31 23:22:16 - r - INFO: - Episode: 200/1000, Reward: 141.0, Step: 141 -2022-10-31 23:22:17 - r - INFO: - Episode: 201/1000, Reward: 113.0, Step: 113 -2022-10-31 23:22:18 - r - INFO: - Episode: 202/1000, Reward: 200.0, Step: 200 -2022-10-31 23:22:18 - r - INFO: - Episode: 203/1000, Reward: 136.0, Step: 136 -2022-10-31 23:22:18 - r - INFO: - Episode: 204/1000, Reward: 114.0, Step: 114 -2022-10-31 23:22:18 - r - INFO: - Episode: 205/1000, Reward: 102.0, Step: 102 -2022-10-31 23:22:19 - r - INFO: - Episode: 206/1000, Reward: 176.0, Step: 176 -2022-10-31 23:22:20 - r - INFO: - Episode: 207/1000, Reward: 150.0, Step: 150 -2022-10-31 23:22:20 - r - INFO: - Episode: 208/1000, Reward: 105.0, Step: 105 -2022-10-31 23:22:20 - r - INFO: - Episode: 209/1000, Reward: 200.0, Step: 200 -2022-10-31 23:22:21 - r - INFO: - Episode: 210/1000, Reward: 200.0, Step: 200 -2022-10-31 23:22:21 - r - INFO: - Episode: 211/1000, Reward: 167.0, Step: 167 -2022-10-31 23:22:22 - r - INFO: - Episode: 212/1000, Reward: 104.0, Step: 104 -2022-10-31 23:22:22 - r - INFO: - Episode: 213/1000, Reward: 124.0, Step: 124 -2022-10-31 23:22:22 - r - INFO: - Episode: 214/1000, Reward: 96.0, Step: 96 -2022-10-31 23:22:23 - r - INFO: - Episode: 215/1000, Reward: 200.0, Step: 200 -2022-10-31 23:22:24 - r - INFO: - Episode: 216/1000, Reward: 199.0, Step: 199 -2022-10-31 23:22:24 - r - INFO: - Episode: 217/1000, Reward: 200.0, Step: 200 -2022-10-31 23:22:24 - r - INFO: - Episode: 218/1000, Reward: 132.0, Step: 132 -2022-10-31 23:22:25 - r - INFO: - Episode: 219/1000, Reward: 188.0, Step: 188 -2022-10-31 23:22:25 - r - INFO: - Episode: 220/1000, Reward: 132.0, Step: 132 -2022-10-31 23:22:26 - r - INFO: - Episode: 221/1000, Reward: 151.0, Step: 151 -2022-10-31 23:22:26 - r - INFO: - Episode: 222/1000, Reward: 125.0, Step: 125 -2022-10-31 23:22:26 - r - INFO: - Episode: 223/1000, Reward: 42.0, Step: 42 -2022-10-31 23:22:27 - r - INFO: - Episode: 224/1000, Reward: 200.0, Step: 200 -2022-10-31 23:22:27 - r - INFO: - Episode: 225/1000, Reward: 159.0, Step: 159 -2022-10-31 23:22:28 - r - INFO: - Episode: 226/1000, Reward: 171.0, Step: 171 -2022-10-31 23:22:28 - r - INFO: - Episode: 227/1000, Reward: 122.0, Step: 122 -2022-10-31 23:22:29 - r - INFO: - Episode: 228/1000, Reward: 189.0, Step: 189 -2022-10-31 23:22:29 - r - INFO: - Episode: 229/1000, Reward: 129.0, Step: 129 -2022-10-31 23:22:29 - r - INFO: - Episode: 230/1000, Reward: 106.0, Step: 106 -2022-10-31 23:22:30 - r - INFO: - Episode: 231/1000, Reward: 107.0, Step: 107 -2022-10-31 23:22:30 - r - INFO: - Episode: 232/1000, Reward: 200.0, Step: 200 -2022-10-31 23:22:31 - r - INFO: - Episode: 233/1000, Reward: 200.0, Step: 200 -2022-10-31 23:22:31 - r - INFO: - Episode: 234/1000, Reward: 200.0, Step: 200 -2022-10-31 23:22:32 - r - INFO: - Episode: 235/1000, Reward: 200.0, Step: 200 -2022-10-31 23:22:32 - r - INFO: - Current episode 235 has the best eval reward: 169.70 -2022-10-31 23:22:33 - r - INFO: - Episode: 236/1000, Reward: 158.0, Step: 158 -2022-10-31 23:22:33 - r - INFO: - Episode: 237/1000, Reward: 200.0, Step: 200 -2022-10-31 23:22:33 - r - INFO: - Episode: 238/1000, Reward: 192.0, Step: 192 -2022-10-31 23:22:34 - r - INFO: - Episode: 239/1000, Reward: 179.0, Step: 179 -2022-10-31 23:22:34 - r - INFO: - Episode: 240/1000, Reward: 102.0, Step: 102 -2022-10-31 23:22:35 - r - INFO: - Episode: 241/1000, Reward: 125.0, Step: 125 -2022-10-31 23:22:35 - r - INFO: - Episode: 242/1000, Reward: 138.0, Step: 138 -2022-10-31 23:22:36 - r - INFO: - Episode: 243/1000, Reward: 189.0, Step: 189 -2022-10-31 23:22:36 - r - INFO: - Episode: 244/1000, Reward: 41.0, Step: 41 -2022-10-31 23:22:36 - r - INFO: - Episode: 245/1000, Reward: 97.0, Step: 97 -2022-10-31 23:22:36 - r - INFO: - Episode: 246/1000, Reward: 49.0, Step: 49 -2022-10-31 23:22:37 - r - INFO: - Episode: 247/1000, Reward: 86.0, Step: 86 -2022-10-31 23:22:37 - r - INFO: - Episode: 248/1000, Reward: 121.0, Step: 121 -2022-10-31 23:22:37 - r - INFO: - Episode: 249/1000, Reward: 117.0, Step: 117 -2022-10-31 23:22:37 - r - INFO: - Episode: 250/1000, Reward: 43.0, Step: 43 -2022-10-31 23:22:38 - r - INFO: - Episode: 251/1000, Reward: 72.0, Step: 72 -2022-10-31 23:22:38 - r - INFO: - Episode: 252/1000, Reward: 34.0, Step: 34 -2022-10-31 23:22:38 - r - INFO: - Episode: 253/1000, Reward: 83.0, Step: 83 -2022-10-31 23:22:38 - r - INFO: - Episode: 254/1000, Reward: 83.0, Step: 83 -2022-10-31 23:22:38 - r - INFO: - Episode: 255/1000, Reward: 38.0, Step: 38 -2022-10-31 23:22:38 - r - INFO: - Episode: 256/1000, Reward: 34.0, Step: 34 -2022-10-31 23:22:39 - r - INFO: - Episode: 257/1000, Reward: 99.0, Step: 99 -2022-10-31 23:22:39 - r - INFO: - Episode: 258/1000, Reward: 45.0, Step: 45 -2022-10-31 23:22:39 - r - INFO: - Episode: 259/1000, Reward: 47.0, Step: 47 -2022-10-31 23:22:39 - r - INFO: - Episode: 260/1000, Reward: 44.0, Step: 44 -2022-10-31 23:22:39 - r - INFO: - Episode: 261/1000, Reward: 26.0, Step: 26 -2022-10-31 23:22:39 - r - INFO: - Episode: 262/1000, Reward: 37.0, Step: 37 -2022-10-31 23:22:39 - r - INFO: - Episode: 263/1000, Reward: 26.0, Step: 26 -2022-10-31 23:22:39 - r - INFO: - Episode: 264/1000, Reward: 43.0, Step: 43 -2022-10-31 23:22:40 - r - INFO: - Episode: 265/1000, Reward: 27.0, Step: 27 -2022-10-31 23:22:40 - r - INFO: - Episode: 266/1000, Reward: 24.0, Step: 24 -2022-10-31 23:22:40 - r - INFO: - Episode: 267/1000, Reward: 42.0, Step: 42 -2022-10-31 23:22:40 - r - INFO: - Episode: 268/1000, Reward: 86.0, Step: 86 -2022-10-31 23:22:40 - r - INFO: - Episode: 269/1000, Reward: 23.0, Step: 23 -2022-10-31 23:22:40 - r - INFO: - Episode: 270/1000, Reward: 32.0, Step: 32 -2022-10-31 23:22:40 - r - INFO: - Episode: 271/1000, Reward: 57.0, Step: 57 -2022-10-31 23:22:40 - r - INFO: - Episode: 272/1000, Reward: 25.0, Step: 25 -2022-10-31 23:22:41 - r - INFO: - Episode: 273/1000, Reward: 98.0, Step: 98 -2022-10-31 23:22:41 - r - INFO: - Episode: 274/1000, Reward: 29.0, Step: 29 -2022-10-31 23:22:41 - r - INFO: - Episode: 275/1000, Reward: 25.0, Step: 25 -2022-10-31 23:22:41 - r - INFO: - Episode: 276/1000, Reward: 29.0, Step: 29 -2022-10-31 23:22:41 - r - INFO: - Episode: 277/1000, Reward: 39.0, Step: 39 -2022-10-31 23:22:41 - r - INFO: - Episode: 278/1000, Reward: 20.0, Step: 20 -2022-10-31 23:22:41 - r - INFO: - Episode: 279/1000, Reward: 92.0, Step: 92 -2022-10-31 23:22:41 - r - INFO: - Episode: 280/1000, Reward: 28.0, Step: 28 -2022-10-31 23:22:42 - r - INFO: - Episode: 281/1000, Reward: 78.0, Step: 78 -2022-10-31 23:22:42 - r - INFO: - Episode: 282/1000, Reward: 25.0, Step: 25 -2022-10-31 23:22:42 - r - INFO: - Episode: 283/1000, Reward: 31.0, Step: 31 -2022-10-31 23:22:42 - r - INFO: - Episode: 284/1000, Reward: 88.0, Step: 88 -2022-10-31 23:22:42 - r - INFO: - Episode: 285/1000, Reward: 85.0, Step: 85 -2022-10-31 23:22:43 - r - INFO: - Episode: 286/1000, Reward: 37.0, Step: 37 -2022-10-31 23:22:43 - r - INFO: - Episode: 287/1000, Reward: 26.0, Step: 26 -2022-10-31 23:22:43 - r - INFO: - Episode: 288/1000, Reward: 19.0, Step: 19 -2022-10-31 23:22:43 - r - INFO: - Episode: 289/1000, Reward: 40.0, Step: 40 -2022-10-31 23:22:43 - r - INFO: - Episode: 290/1000, Reward: 27.0, Step: 27 -2022-10-31 23:22:43 - r - INFO: - Episode: 291/1000, Reward: 17.0, Step: 17 -2022-10-31 23:22:43 - r - INFO: - Episode: 292/1000, Reward: 27.0, Step: 27 -2022-10-31 23:22:43 - r - INFO: - Episode: 293/1000, Reward: 26.0, Step: 26 -2022-10-31 23:22:43 - r - INFO: - Episode: 294/1000, Reward: 82.0, Step: 82 -2022-10-31 23:22:43 - r - INFO: - Episode: 295/1000, Reward: 36.0, Step: 36 -2022-10-31 23:22:44 - r - INFO: - Episode: 296/1000, Reward: 24.0, Step: 24 -2022-10-31 23:22:44 - r - INFO: - Episode: 297/1000, Reward: 30.0, Step: 30 -2022-10-31 23:22:44 - r - INFO: - Episode: 298/1000, Reward: 20.0, Step: 20 -2022-10-31 23:22:44 - r - INFO: - Episode: 299/1000, Reward: 34.0, Step: 34 -2022-10-31 23:22:44 - r - INFO: - Episode: 300/1000, Reward: 30.0, Step: 30 -2022-10-31 23:22:44 - r - INFO: - Episode: 301/1000, Reward: 23.0, Step: 23 -2022-10-31 23:22:44 - r - INFO: - Episode: 302/1000, Reward: 36.0, Step: 36 -2022-10-31 23:22:44 - r - INFO: - Episode: 303/1000, Reward: 29.0, Step: 29 -2022-10-31 23:22:44 - r - INFO: - Episode: 304/1000, Reward: 34.0, Step: 34 -2022-10-31 23:22:44 - r - INFO: - Episode: 305/1000, Reward: 25.0, Step: 25 -2022-10-31 23:22:45 - r - INFO: - Episode: 306/1000, Reward: 42.0, Step: 42 -2022-10-31 23:22:45 - r - INFO: - Episode: 307/1000, Reward: 88.0, Step: 88 -2022-10-31 23:22:45 - r - INFO: - Episode: 308/1000, Reward: 26.0, Step: 26 -2022-10-31 23:22:45 - r - INFO: - Episode: 309/1000, Reward: 85.0, Step: 85 -2022-10-31 23:22:45 - r - INFO: - Episode: 310/1000, Reward: 89.0, Step: 89 -2022-10-31 23:22:46 - r - INFO: - Episode: 311/1000, Reward: 48.0, Step: 48 -2022-10-31 23:22:46 - r - INFO: - Episode: 312/1000, Reward: 83.0, Step: 83 -2022-10-31 23:22:46 - r - INFO: - Episode: 313/1000, Reward: 109.0, Step: 109 -2022-10-31 23:22:46 - r - INFO: - Episode: 314/1000, Reward: 42.0, Step: 42 -2022-10-31 23:22:46 - r - INFO: - Episode: 315/1000, Reward: 93.0, Step: 93 -2022-10-31 23:22:47 - r - INFO: - Episode: 316/1000, Reward: 85.0, Step: 85 -2022-10-31 23:22:47 - r - INFO: - Episode: 317/1000, Reward: 100.0, Step: 100 -2022-10-31 23:22:47 - r - INFO: - Episode: 318/1000, Reward: 106.0, Step: 106 -2022-10-31 23:22:47 - r - INFO: - Episode: 319/1000, Reward: 28.0, Step: 28 -2022-10-31 23:22:48 - r - INFO: - Episode: 320/1000, Reward: 108.0, Step: 108 -2022-10-31 23:22:48 - r - INFO: - Episode: 321/1000, Reward: 112.0, Step: 112 -2022-10-31 23:22:48 - r - INFO: - Episode: 322/1000, Reward: 88.0, Step: 88 -2022-10-31 23:22:49 - r - INFO: - Episode: 323/1000, Reward: 108.0, Step: 108 -2022-10-31 23:22:49 - r - INFO: - Episode: 324/1000, Reward: 108.0, Step: 108 -2022-10-31 23:22:49 - r - INFO: - Episode: 325/1000, Reward: 90.0, Step: 90 -2022-10-31 23:22:50 - r - INFO: - Episode: 326/1000, Reward: 112.0, Step: 112 -2022-10-31 23:22:50 - r - INFO: - Episode: 327/1000, Reward: 113.0, Step: 113 -2022-10-31 23:22:50 - r - INFO: - Episode: 328/1000, Reward: 94.0, Step: 94 -2022-10-31 23:22:50 - r - INFO: - Episode: 329/1000, Reward: 99.0, Step: 99 -2022-10-31 23:22:51 - r - INFO: - Episode: 330/1000, Reward: 45.0, Step: 45 -2022-10-31 23:22:51 - r - INFO: - Episode: 331/1000, Reward: 121.0, Step: 121 -2022-10-31 23:22:51 - r - INFO: - Episode: 332/1000, Reward: 102.0, Step: 102 -2022-10-31 23:22:52 - r - INFO: - Episode: 333/1000, Reward: 111.0, Step: 111 -2022-10-31 23:22:52 - r - INFO: - Episode: 334/1000, Reward: 54.0, Step: 54 -2022-10-31 23:22:52 - r - INFO: - Episode: 335/1000, Reward: 198.0, Step: 198 -2022-10-31 23:22:53 - r - INFO: - Episode: 336/1000, Reward: 83.0, Step: 83 -2022-10-31 23:22:53 - r - INFO: - Episode: 337/1000, Reward: 107.0, Step: 107 -2022-10-31 23:22:53 - r - INFO: - Episode: 338/1000, Reward: 101.0, Step: 101 -2022-10-31 23:22:54 - r - INFO: - Episode: 339/1000, Reward: 129.0, Step: 129 -2022-10-31 23:22:54 - r - INFO: - Episode: 340/1000, Reward: 88.0, Step: 88 -2022-10-31 23:22:54 - r - INFO: - Episode: 341/1000, Reward: 86.0, Step: 86 -2022-10-31 23:22:55 - r - INFO: - Episode: 342/1000, Reward: 199.0, Step: 199 -2022-10-31 23:22:55 - r - INFO: - Episode: 343/1000, Reward: 95.0, Step: 95 -2022-10-31 23:22:55 - r - INFO: - Episode: 344/1000, Reward: 103.0, Step: 103 -2022-10-31 23:22:56 - r - INFO: - Episode: 345/1000, Reward: 100.0, Step: 100 -2022-10-31 23:22:56 - r - INFO: - Episode: 346/1000, Reward: 89.0, Step: 89 -2022-10-31 23:22:56 - r - INFO: - Episode: 347/1000, Reward: 87.0, Step: 87 -2022-10-31 23:22:57 - r - INFO: - Episode: 348/1000, Reward: 110.0, Step: 110 -2022-10-31 23:22:57 - r - INFO: - Episode: 349/1000, Reward: 127.0, Step: 127 -2022-10-31 23:22:57 - r - INFO: - Episode: 350/1000, Reward: 97.0, Step: 97 -2022-10-31 23:22:57 - r - INFO: - Episode: 351/1000, Reward: 34.0, Step: 34 -2022-10-31 23:22:58 - r - INFO: - Episode: 352/1000, Reward: 123.0, Step: 123 -2022-10-31 23:22:58 - r - INFO: - Episode: 353/1000, Reward: 49.0, Step: 49 -2022-10-31 23:22:58 - r - INFO: - Episode: 354/1000, Reward: 96.0, Step: 96 -2022-10-31 23:22:58 - r - INFO: - Episode: 355/1000, Reward: 90.0, Step: 90 -2022-10-31 23:22:59 - r - INFO: - Episode: 356/1000, Reward: 110.0, Step: 110 -2022-10-31 23:22:59 - r - INFO: - Episode: 357/1000, Reward: 93.0, Step: 93 -2022-10-31 23:22:59 - r - INFO: - Episode: 358/1000, Reward: 102.0, Step: 102 -2022-10-31 23:23:00 - r - INFO: - Episode: 359/1000, Reward: 128.0, Step: 128 -2022-10-31 23:23:00 - r - INFO: - Episode: 360/1000, Reward: 125.0, Step: 125 -2022-10-31 23:23:01 - r - INFO: - Episode: 361/1000, Reward: 92.0, Step: 92 -2022-10-31 23:23:01 - r - INFO: - Episode: 362/1000, Reward: 109.0, Step: 109 -2022-10-31 23:23:01 - r - INFO: - Episode: 363/1000, Reward: 114.0, Step: 114 -2022-10-31 23:23:01 - r - INFO: - Episode: 364/1000, Reward: 111.0, Step: 111 -2022-10-31 23:23:02 - r - INFO: - Episode: 365/1000, Reward: 38.0, Step: 38 -2022-10-31 23:23:02 - r - INFO: - Episode: 366/1000, Reward: 55.0, Step: 55 -2022-10-31 23:23:02 - r - INFO: - Episode: 367/1000, Reward: 106.0, Step: 106 -2022-10-31 23:23:02 - r - INFO: - Episode: 368/1000, Reward: 115.0, Step: 115 -2022-10-31 23:23:03 - r - INFO: - Episode: 369/1000, Reward: 103.0, Step: 103 -2022-10-31 23:23:03 - r - INFO: - Episode: 370/1000, Reward: 50.0, Step: 50 -2022-10-31 23:23:03 - r - INFO: - Episode: 371/1000, Reward: 110.0, Step: 110 -2022-10-31 23:23:04 - r - INFO: - Episode: 372/1000, Reward: 102.0, Step: 102 -2022-10-31 23:23:04 - r - INFO: - Episode: 373/1000, Reward: 110.0, Step: 110 -2022-10-31 23:23:04 - r - INFO: - Episode: 374/1000, Reward: 29.0, Step: 29 -2022-10-31 23:23:04 - r - INFO: - Episode: 375/1000, Reward: 35.0, Step: 35 -2022-10-31 23:23:04 - r - INFO: - Episode: 376/1000, Reward: 42.0, Step: 42 -2022-10-31 23:23:05 - r - INFO: - Episode: 377/1000, Reward: 62.0, Step: 62 -2022-10-31 23:23:05 - r - INFO: - Episode: 378/1000, Reward: 119.0, Step: 119 -2022-10-31 23:23:05 - r - INFO: - Episode: 379/1000, Reward: 33.0, Step: 33 -2022-10-31 23:23:05 - r - INFO: - Episode: 380/1000, Reward: 31.0, Step: 31 -2022-10-31 23:23:05 - r - INFO: - Episode: 381/1000, Reward: 97.0, Step: 97 -2022-10-31 23:23:06 - r - INFO: - Episode: 382/1000, Reward: 192.0, Step: 192 -2022-10-31 23:23:06 - r - INFO: - Episode: 383/1000, Reward: 179.0, Step: 179 -2022-10-31 23:23:07 - r - INFO: - Episode: 384/1000, Reward: 89.0, Step: 89 -2022-10-31 23:23:07 - r - INFO: - Episode: 385/1000, Reward: 32.0, Step: 32 -2022-10-31 23:23:07 - r - INFO: - Episode: 386/1000, Reward: 33.0, Step: 33 -2022-10-31 23:23:07 - r - INFO: - Episode: 387/1000, Reward: 52.0, Step: 52 -2022-10-31 23:23:07 - r - INFO: - Episode: 388/1000, Reward: 31.0, Step: 31 -2022-10-31 23:23:07 - r - INFO: - Episode: 389/1000, Reward: 22.0, Step: 22 -2022-10-31 23:23:08 - r - INFO: - Episode: 390/1000, Reward: 118.0, Step: 118 -2022-10-31 23:23:08 - r - INFO: - Episode: 391/1000, Reward: 24.0, Step: 24 -2022-10-31 23:23:08 - r - INFO: - Episode: 392/1000, Reward: 115.0, Step: 115 -2022-10-31 23:23:08 - r - INFO: - Episode: 393/1000, Reward: 20.0, Step: 20 -2022-10-31 23:23:08 - r - INFO: - Episode: 394/1000, Reward: 33.0, Step: 33 -2022-10-31 23:23:08 - r - INFO: - Episode: 395/1000, Reward: 40.0, Step: 40 -2022-10-31 23:23:08 - r - INFO: - Episode: 396/1000, Reward: 27.0, Step: 27 -2022-10-31 23:23:08 - r - INFO: - Episode: 397/1000, Reward: 26.0, Step: 26 -2022-10-31 23:23:09 - r - INFO: - Episode: 398/1000, Reward: 24.0, Step: 24 -2022-10-31 23:23:09 - r - INFO: - Episode: 399/1000, Reward: 19.0, Step: 19 -2022-10-31 23:23:09 - r - INFO: - Episode: 400/1000, Reward: 22.0, Step: 22 -2022-10-31 23:23:09 - r - INFO: - Episode: 401/1000, Reward: 24.0, Step: 24 -2022-10-31 23:23:09 - r - INFO: - Episode: 402/1000, Reward: 18.0, Step: 18 -2022-10-31 23:23:09 - r - INFO: - Episode: 403/1000, Reward: 23.0, Step: 23 -2022-10-31 23:23:09 - r - INFO: - Episode: 404/1000, Reward: 27.0, Step: 27 -2022-10-31 23:23:09 - r - INFO: - Episode: 405/1000, Reward: 20.0, Step: 20 -2022-10-31 23:23:09 - r - INFO: - Episode: 406/1000, Reward: 27.0, Step: 27 -2022-10-31 23:23:09 - r - INFO: - Episode: 407/1000, Reward: 17.0, Step: 17 -2022-10-31 23:23:09 - r - INFO: - Episode: 408/1000, Reward: 27.0, Step: 27 -2022-10-31 23:23:09 - r - INFO: - Episode: 409/1000, Reward: 25.0, Step: 25 -2022-10-31 23:23:09 - r - INFO: - Episode: 410/1000, Reward: 25.0, Step: 25 -2022-10-31 23:23:09 - r - INFO: - Episode: 411/1000, Reward: 24.0, Step: 24 -2022-10-31 23:23:10 - r - INFO: - Episode: 412/1000, Reward: 24.0, Step: 24 -2022-10-31 23:23:10 - r - INFO: - Episode: 413/1000, Reward: 18.0, Step: 18 -2022-10-31 23:23:10 - r - INFO: - Episode: 414/1000, Reward: 20.0, Step: 20 -2022-10-31 23:23:10 - r - INFO: - Episode: 415/1000, Reward: 27.0, Step: 27 -2022-10-31 23:23:10 - r - INFO: - Episode: 416/1000, Reward: 28.0, Step: 28 -2022-10-31 23:23:10 - r - INFO: - Episode: 417/1000, Reward: 30.0, Step: 30 -2022-10-31 23:23:10 - r - INFO: - Episode: 418/1000, Reward: 28.0, Step: 28 -2022-10-31 23:23:10 - r - INFO: - Episode: 419/1000, Reward: 33.0, Step: 33 -2022-10-31 23:23:10 - r - INFO: - Episode: 420/1000, Reward: 24.0, Step: 24 -2022-10-31 23:23:10 - r - INFO: - Episode: 421/1000, Reward: 96.0, Step: 96 -2022-10-31 23:23:11 - r - INFO: - Episode: 422/1000, Reward: 26.0, Step: 26 -2022-10-31 23:23:11 - r - INFO: - Episode: 423/1000, Reward: 29.0, Step: 29 -2022-10-31 23:23:11 - r - INFO: - Episode: 424/1000, Reward: 25.0, Step: 25 -2022-10-31 23:23:11 - r - INFO: - Episode: 425/1000, Reward: 38.0, Step: 38 -2022-10-31 23:23:11 - r - INFO: - Episode: 426/1000, Reward: 33.0, Step: 33 -2022-10-31 23:23:11 - r - INFO: - Episode: 427/1000, Reward: 23.0, Step: 23 -2022-10-31 23:23:11 - r - INFO: - Episode: 428/1000, Reward: 39.0, Step: 39 -2022-10-31 23:23:11 - r - INFO: - Episode: 429/1000, Reward: 28.0, Step: 28 -2022-10-31 23:23:11 - r - INFO: - Episode: 430/1000, Reward: 97.0, Step: 97 -2022-10-31 23:23:12 - r - INFO: - Episode: 431/1000, Reward: 30.0, Step: 30 -2022-10-31 23:23:12 - r - INFO: - Episode: 432/1000, Reward: 29.0, Step: 29 -2022-10-31 23:23:12 - r - INFO: - Episode: 433/1000, Reward: 103.0, Step: 103 -2022-10-31 23:23:12 - r - INFO: - Episode: 434/1000, Reward: 36.0, Step: 36 -2022-10-31 23:23:12 - r - INFO: - Episode: 435/1000, Reward: 32.0, Step: 32 -2022-10-31 23:23:12 - r - INFO: - Episode: 436/1000, Reward: 41.0, Step: 41 -2022-10-31 23:23:13 - r - INFO: - Episode: 437/1000, Reward: 111.0, Step: 111 -2022-10-31 23:23:13 - r - INFO: - Episode: 438/1000, Reward: 48.0, Step: 48 -2022-10-31 23:23:13 - r - INFO: - Episode: 439/1000, Reward: 24.0, Step: 24 -2022-10-31 23:23:13 - r - INFO: - Episode: 440/1000, Reward: 49.0, Step: 49 -2022-10-31 23:23:14 - r - INFO: - Episode: 441/1000, Reward: 116.0, Step: 116 -2022-10-31 23:23:14 - r - INFO: - Episode: 442/1000, Reward: 118.0, Step: 118 -2022-10-31 23:23:14 - r - INFO: - Episode: 443/1000, Reward: 94.0, Step: 94 -2022-10-31 23:23:14 - r - INFO: - Episode: 444/1000, Reward: 132.0, Step: 132 -2022-10-31 23:23:14 - r - INFO: - Episode: 445/1000, Reward: 41.0, Step: 41 -2022-10-31 23:23:15 - r - INFO: - Episode: 446/1000, Reward: 105.0, Step: 105 -2022-10-31 23:23:15 - r - INFO: - Episode: 447/1000, Reward: 116.0, Step: 116 -2022-10-31 23:23:16 - r - INFO: - Episode: 448/1000, Reward: 136.0, Step: 136 -2022-10-31 23:23:16 - r - INFO: - Episode: 449/1000, Reward: 137.0, Step: 137 -2022-10-31 23:23:16 - r - INFO: - Episode: 450/1000, Reward: 45.0, Step: 45 -2022-10-31 23:23:17 - r - INFO: - Episode: 451/1000, Reward: 157.0, Step: 157 -2022-10-31 23:23:17 - r - INFO: - Episode: 452/1000, Reward: 116.0, Step: 116 -2022-10-31 23:23:17 - r - INFO: - Episode: 453/1000, Reward: 125.0, Step: 125 -2022-10-31 23:23:18 - r - INFO: - Episode: 454/1000, Reward: 120.0, Step: 120 -2022-10-31 23:23:18 - r - INFO: - Episode: 455/1000, Reward: 150.0, Step: 150 -2022-10-31 23:23:19 - r - INFO: - Episode: 456/1000, Reward: 114.0, Step: 114 -2022-10-31 23:23:19 - r - INFO: - Episode: 457/1000, Reward: 44.0, Step: 44 -2022-10-31 23:23:19 - r - INFO: - Episode: 458/1000, Reward: 138.0, Step: 138 -2022-10-31 23:23:19 - r - INFO: - Episode: 459/1000, Reward: 133.0, Step: 133 -2022-10-31 23:23:20 - r - INFO: - Episode: 460/1000, Reward: 141.0, Step: 141 -2022-10-31 23:23:20 - r - INFO: - Episode: 461/1000, Reward: 124.0, Step: 124 -2022-10-31 23:23:21 - r - INFO: - Episode: 462/1000, Reward: 143.0, Step: 143 -2022-10-31 23:23:21 - r - INFO: - Episode: 463/1000, Reward: 123.0, Step: 123 -2022-10-31 23:23:21 - r - INFO: - Episode: 464/1000, Reward: 134.0, Step: 134 -2022-10-31 23:23:22 - r - INFO: - Episode: 465/1000, Reward: 152.0, Step: 152 -2022-10-31 23:23:23 - r - INFO: - Episode: 466/1000, Reward: 140.0, Step: 140 -2022-10-31 23:23:23 - r - INFO: - Episode: 467/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:23 - r - INFO: - Episode: 468/1000, Reward: 168.0, Step: 168 -2022-10-31 23:23:24 - r - INFO: - Episode: 469/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:24 - r - INFO: - Episode: 470/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:25 - r - INFO: - Current episode 470 has the best eval reward: 199.80 -2022-10-31 23:23:25 - r - INFO: - Episode: 471/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:26 - r - INFO: - Episode: 472/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:26 - r - INFO: - Episode: 473/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:27 - r - INFO: - Episode: 474/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:27 - r - INFO: - Episode: 475/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:28 - r - INFO: - Current episode 475 has the best eval reward: 200.00 -2022-10-31 23:23:28 - r - INFO: - Episode: 476/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:29 - r - INFO: - Episode: 477/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:29 - r - INFO: - Episode: 478/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:30 - r - INFO: - Episode: 479/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:30 - r - INFO: - Episode: 480/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:31 - r - INFO: - Current episode 480 has the best eval reward: 200.00 -2022-10-31 23:23:31 - r - INFO: - Episode: 481/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:32 - r - INFO: - Episode: 482/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:32 - r - INFO: - Episode: 483/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:33 - r - INFO: - Episode: 484/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:33 - r - INFO: - Episode: 485/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:34 - r - INFO: - Current episode 485 has the best eval reward: 200.00 -2022-10-31 23:23:34 - r - INFO: - Episode: 486/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:35 - r - INFO: - Episode: 487/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:35 - r - INFO: - Episode: 488/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:36 - r - INFO: - Episode: 489/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:36 - r - INFO: - Episode: 490/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:37 - r - INFO: - Current episode 490 has the best eval reward: 200.00 -2022-10-31 23:23:37 - r - INFO: - Episode: 491/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:38 - r - INFO: - Episode: 492/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:38 - r - INFO: - Episode: 493/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:38 - r - INFO: - Episode: 494/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:39 - r - INFO: - Episode: 495/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:40 - r - INFO: - Current episode 495 has the best eval reward: 200.00 -2022-10-31 23:23:40 - r - INFO: - Episode: 496/1000, Reward: 169.0, Step: 169 -2022-10-31 23:23:40 - r - INFO: - Episode: 497/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:41 - r - INFO: - Episode: 498/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:41 - r - INFO: - Episode: 499/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:42 - r - INFO: - Episode: 500/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:42 - r - INFO: - Current episode 500 has the best eval reward: 200.00 -2022-10-31 23:23:43 - r - INFO: - Episode: 501/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:43 - r - INFO: - Episode: 502/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:44 - r - INFO: - Episode: 503/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:44 - r - INFO: - Episode: 504/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:45 - r - INFO: - Episode: 505/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:45 - r - INFO: - Current episode 505 has the best eval reward: 200.00 -2022-10-31 23:23:46 - r - INFO: - Episode: 506/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:46 - r - INFO: - Episode: 507/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:47 - r - INFO: - Episode: 508/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:47 - r - INFO: - Episode: 509/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:48 - r - INFO: - Episode: 510/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:49 - r - INFO: - Episode: 511/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:49 - r - INFO: - Episode: 512/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:50 - r - INFO: - Episode: 513/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:50 - r - INFO: - Episode: 514/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:51 - r - INFO: - Episode: 515/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:52 - r - INFO: - Episode: 516/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:52 - r - INFO: - Episode: 517/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:53 - r - INFO: - Episode: 518/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:53 - r - INFO: - Episode: 519/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:54 - r - INFO: - Episode: 520/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:55 - r - INFO: - Current episode 520 has the best eval reward: 200.00 -2022-10-31 23:23:55 - r - INFO: - Episode: 521/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:55 - r - INFO: - Episode: 522/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:56 - r - INFO: - Episode: 523/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:56 - r - INFO: - Episode: 524/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:57 - r - INFO: - Episode: 525/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:58 - r - INFO: - Current episode 525 has the best eval reward: 200.00 -2022-10-31 23:23:58 - r - INFO: - Episode: 526/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:59 - r - INFO: - Episode: 527/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:59 - r - INFO: - Episode: 528/1000, Reward: 200.0, Step: 200 -2022-10-31 23:23:59 - r - INFO: - Episode: 529/1000, Reward: 186.0, Step: 186 -2022-10-31 23:24:00 - r - INFO: - Episode: 530/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:00 - r - INFO: - Current episode 530 has the best eval reward: 200.00 -2022-10-31 23:24:01 - r - INFO: - Episode: 531/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:01 - r - INFO: - Episode: 532/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:02 - r - INFO: - Episode: 533/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:02 - r - INFO: - Episode: 534/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:03 - r - INFO: - Episode: 535/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:04 - r - INFO: - Current episode 535 has the best eval reward: 200.00 -2022-10-31 23:24:04 - r - INFO: - Episode: 536/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:04 - r - INFO: - Episode: 537/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:05 - r - INFO: - Episode: 538/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:05 - r - INFO: - Episode: 539/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:06 - r - INFO: - Episode: 540/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:07 - r - INFO: - Current episode 540 has the best eval reward: 200.00 -2022-10-31 23:24:07 - r - INFO: - Episode: 541/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:08 - r - INFO: - Episode: 542/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:08 - r - INFO: - Episode: 543/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:08 - r - INFO: - Episode: 544/1000, Reward: 84.0, Step: 84 -2022-10-31 23:24:09 - r - INFO: - Episode: 545/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:09 - r - INFO: - Current episode 545 has the best eval reward: 200.00 -2022-10-31 23:24:10 - r - INFO: - Episode: 546/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:10 - r - INFO: - Episode: 547/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:11 - r - INFO: - Episode: 548/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:11 - r - INFO: - Episode: 549/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:12 - r - INFO: - Episode: 550/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:13 - r - INFO: - Episode: 551/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:13 - r - INFO: - Episode: 552/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:14 - r - INFO: - Episode: 553/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:14 - r - INFO: - Episode: 554/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:15 - r - INFO: - Episode: 555/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:16 - r - INFO: - Episode: 556/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:16 - r - INFO: - Episode: 557/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:17 - r - INFO: - Episode: 558/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:17 - r - INFO: - Episode: 559/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:17 - r - INFO: - Episode: 560/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:18 - r - INFO: - Current episode 560 has the best eval reward: 200.00 -2022-10-31 23:24:19 - r - INFO: - Episode: 561/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:19 - r - INFO: - Episode: 562/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:20 - r - INFO: - Episode: 563/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:20 - r - INFO: - Episode: 564/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:20 - r - INFO: - Episode: 565/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:21 - r - INFO: - Current episode 565 has the best eval reward: 200.00 -2022-10-31 23:24:21 - r - INFO: - Episode: 566/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:22 - r - INFO: - Episode: 567/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:22 - r - INFO: - Episode: 568/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:23 - r - INFO: - Episode: 569/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:23 - r - INFO: - Episode: 570/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:24 - r - INFO: - Current episode 570 has the best eval reward: 200.00 -2022-10-31 23:24:24 - r - INFO: - Episode: 571/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:25 - r - INFO: - Episode: 572/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:25 - r - INFO: - Episode: 573/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:26 - r - INFO: - Episode: 574/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:26 - r - INFO: - Episode: 575/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:28 - r - INFO: - Episode: 576/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:28 - r - INFO: - Episode: 577/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:28 - r - INFO: - Episode: 578/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:29 - r - INFO: - Episode: 579/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:29 - r - INFO: - Episode: 580/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:30 - r - INFO: - Current episode 580 has the best eval reward: 200.00 -2022-10-31 23:24:31 - r - INFO: - Episode: 581/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:31 - r - INFO: - Episode: 582/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:32 - r - INFO: - Episode: 583/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:32 - r - INFO: - Episode: 584/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:33 - r - INFO: - Episode: 585/1000, Reward: 199.0, Step: 199 -2022-10-31 23:24:34 - r - INFO: - Episode: 586/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:34 - r - INFO: - Episode: 587/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:35 - r - INFO: - Episode: 588/1000, Reward: 178.0, Step: 178 -2022-10-31 23:24:35 - r - INFO: - Episode: 589/1000, Reward: 200.0, Step: 200 -2022-10-31 23:24:36 - r - INFO: - Episode: 590/1000, Reward: 188.0, Step: 188 -2022-10-31 23:24:36 - r - INFO: - Episode: 591/1000, Reward: 156.0, Step: 156 -2022-10-31 23:24:37 - r - INFO: - Episode: 592/1000, Reward: 165.0, Step: 165 -2022-10-31 23:24:37 - r - INFO: - Episode: 593/1000, Reward: 131.0, Step: 131 -2022-10-31 23:24:37 - r - INFO: - Episode: 594/1000, Reward: 157.0, Step: 157 -2022-10-31 23:24:38 - r - INFO: - Episode: 595/1000, Reward: 170.0, Step: 170 -2022-10-31 23:24:39 - r - INFO: - Episode: 596/1000, Reward: 123.0, Step: 123 -2022-10-31 23:24:39 - r - INFO: - Episode: 597/1000, Reward: 109.0, Step: 109 -2022-10-31 23:24:39 - r - INFO: - Episode: 598/1000, Reward: 124.0, Step: 124 -2022-10-31 23:24:39 - r - INFO: - Episode: 599/1000, Reward: 113.0, Step: 113 -2022-10-31 23:24:39 - r - INFO: - Episode: 600/1000, Reward: 38.0, Step: 38 -2022-10-31 23:24:40 - r - INFO: - Episode: 601/1000, Reward: 107.0, Step: 107 -2022-10-31 23:24:40 - r - INFO: - Episode: 602/1000, Reward: 115.0, Step: 115 -2022-10-31 23:24:41 - r - INFO: - Episode: 603/1000, Reward: 101.0, Step: 101 -2022-10-31 23:24:41 - r - INFO: - Episode: 604/1000, Reward: 113.0, Step: 113 -2022-10-31 23:24:41 - r - INFO: - Episode: 605/1000, Reward: 100.0, Step: 100 -2022-10-31 23:24:42 - r - INFO: - Episode: 606/1000, Reward: 109.0, Step: 109 -2022-10-31 23:24:42 - r - INFO: - Episode: 607/1000, Reward: 119.0, Step: 119 -2022-10-31 23:24:42 - r - INFO: - Episode: 608/1000, Reward: 117.0, Step: 117 -2022-10-31 23:24:43 - r - INFO: - Episode: 609/1000, Reward: 108.0, Step: 108 -2022-10-31 23:24:43 - r - INFO: - Episode: 610/1000, Reward: 101.0, Step: 101 -2022-10-31 23:24:43 - r - INFO: - Episode: 611/1000, Reward: 110.0, Step: 110 -2022-10-31 23:24:44 - r - INFO: - Episode: 612/1000, Reward: 59.0, Step: 59 -2022-10-31 23:24:44 - r - INFO: - Episode: 613/1000, Reward: 112.0, Step: 112 -2022-10-31 23:24:44 - r - INFO: - Episode: 614/1000, Reward: 104.0, Step: 104 -2022-10-31 23:24:44 - r - INFO: - Episode: 615/1000, Reward: 45.0, Step: 45 -2022-10-31 23:24:44 - r - INFO: - Episode: 616/1000, Reward: 29.0, Step: 29 -2022-10-31 23:24:44 - r - INFO: - Episode: 617/1000, Reward: 42.0, Step: 42 -2022-10-31 23:24:45 - r - INFO: - Episode: 618/1000, Reward: 74.0, Step: 74 -2022-10-31 23:24:45 - r - INFO: - Episode: 619/1000, Reward: 79.0, Step: 79 -2022-10-31 23:24:45 - r - INFO: - Episode: 620/1000, Reward: 50.0, Step: 50 -2022-10-31 23:24:45 - r - INFO: - Episode: 621/1000, Reward: 30.0, Step: 30 -2022-10-31 23:24:45 - r - INFO: - Episode: 622/1000, Reward: 43.0, Step: 43 -2022-10-31 23:24:46 - r - INFO: - Episode: 623/1000, Reward: 77.0, Step: 77 -2022-10-31 23:24:46 - r - INFO: - Episode: 624/1000, Reward: 36.0, Step: 36 -2022-10-31 23:24:46 - r - INFO: - Episode: 625/1000, Reward: 61.0, Step: 61 -2022-10-31 23:24:46 - r - INFO: - Episode: 626/1000, Reward: 36.0, Step: 36 -2022-10-31 23:24:46 - r - INFO: - Episode: 627/1000, Reward: 30.0, Step: 30 -2022-10-31 23:24:46 - r - INFO: - Episode: 628/1000, Reward: 43.0, Step: 43 -2022-10-31 23:24:46 - r - INFO: - Episode: 629/1000, Reward: 27.0, Step: 27 -2022-10-31 23:24:46 - r - INFO: - Episode: 630/1000, Reward: 88.0, Step: 88 -2022-10-31 23:24:47 - r - INFO: - Episode: 631/1000, Reward: 42.0, Step: 42 -2022-10-31 23:24:47 - r - INFO: - Episode: 632/1000, Reward: 40.0, Step: 40 -2022-10-31 23:24:47 - r - INFO: - Episode: 633/1000, Reward: 59.0, Step: 59 -2022-10-31 23:24:47 - r - INFO: - Episode: 634/1000, Reward: 81.0, Step: 81 -2022-10-31 23:24:47 - r - INFO: - Episode: 635/1000, Reward: 85.0, Step: 85 -2022-10-31 23:24:48 - r - INFO: - Episode: 636/1000, Reward: 55.0, Step: 55 -2022-10-31 23:24:48 - r - INFO: - Episode: 637/1000, Reward: 40.0, Step: 40 -2022-10-31 23:24:48 - r - INFO: - Episode: 638/1000, Reward: 99.0, Step: 99 -2022-10-31 23:24:48 - r - INFO: - Episode: 639/1000, Reward: 104.0, Step: 104 -2022-10-31 23:24:49 - r - INFO: - Episode: 640/1000, Reward: 117.0, Step: 117 -2022-10-31 23:24:49 - r - INFO: - Episode: 641/1000, Reward: 112.0, Step: 112 -2022-10-31 23:24:49 - r - INFO: - Episode: 642/1000, Reward: 43.0, Step: 43 -2022-10-31 23:24:50 - r - INFO: - Episode: 643/1000, Reward: 96.0, Step: 96 -2022-10-31 23:24:50 - r - INFO: - Episode: 644/1000, Reward: 105.0, Step: 105 -2022-10-31 23:24:50 - r - INFO: - Episode: 645/1000, Reward: 115.0, Step: 115 -2022-10-31 23:24:51 - r - INFO: - Episode: 646/1000, Reward: 99.0, Step: 99 -2022-10-31 23:24:51 - r - INFO: - Episode: 647/1000, Reward: 123.0, Step: 123 -2022-10-31 23:24:51 - r - INFO: - Episode: 648/1000, Reward: 123.0, Step: 123 -2022-10-31 23:24:51 - r - INFO: - Episode: 649/1000, Reward: 40.0, Step: 40 -2022-10-31 23:24:51 - r - INFO: - Episode: 650/1000, Reward: 100.0, Step: 100 -2022-10-31 23:24:52 - r - INFO: - Episode: 651/1000, Reward: 124.0, Step: 124 -2022-10-31 23:24:52 - r - INFO: - Episode: 652/1000, Reward: 106.0, Step: 106 -2022-10-31 23:24:53 - r - INFO: - Episode: 653/1000, Reward: 122.0, Step: 122 -2022-10-31 23:24:53 - r - INFO: - Episode: 654/1000, Reward: 127.0, Step: 127 -2022-10-31 23:24:53 - r - INFO: - Episode: 655/1000, Reward: 121.0, Step: 121 -2022-10-31 23:24:54 - r - INFO: - Episode: 656/1000, Reward: 121.0, Step: 121 -2022-10-31 23:24:54 - r - INFO: - Episode: 657/1000, Reward: 125.0, Step: 125 -2022-10-31 23:24:54 - r - INFO: - Episode: 658/1000, Reward: 127.0, Step: 127 -2022-10-31 23:24:55 - r - INFO: - Episode: 659/1000, Reward: 132.0, Step: 132 -2022-10-31 23:24:55 - r - INFO: - Episode: 660/1000, Reward: 142.0, Step: 142 -2022-10-31 23:24:56 - r - INFO: - Episode: 661/1000, Reward: 134.0, Step: 134 -2022-10-31 23:24:56 - r - INFO: - Episode: 662/1000, Reward: 147.0, Step: 147 -2022-10-31 23:24:57 - r - INFO: - Episode: 663/1000, Reward: 175.0, Step: 175 -2022-10-31 23:24:57 - r - INFO: - Episode: 664/1000, Reward: 180.0, Step: 180 -2022-10-31 23:24:57 - r - INFO: - Episode: 665/1000, Reward: 183.0, Step: 183 -2022-10-31 23:24:58 - r - INFO: - Episode: 666/1000, Reward: 167.0, Step: 167 -2022-10-31 23:24:59 - r - INFO: - Episode: 667/1000, Reward: 179.0, Step: 179 -2022-10-31 23:24:59 - r - INFO: - Episode: 668/1000, Reward: 173.0, Step: 173 -2022-10-31 23:25:00 - r - INFO: - Episode: 669/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:00 - r - INFO: - Episode: 670/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:01 - r - INFO: - Episode: 671/1000, Reward: 184.0, Step: 184 -2022-10-31 23:25:02 - r - INFO: - Episode: 672/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:02 - r - INFO: - Episode: 673/1000, Reward: 193.0, Step: 193 -2022-10-31 23:25:03 - r - INFO: - Episode: 674/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:03 - r - INFO: - Episode: 675/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:04 - r - INFO: - Current episode 675 has the best eval reward: 200.00 -2022-10-31 23:25:04 - r - INFO: - Episode: 676/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:05 - r - INFO: - Episode: 677/1000, Reward: 199.0, Step: 199 -2022-10-31 23:25:05 - r - INFO: - Episode: 678/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:06 - r - INFO: - Episode: 679/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:06 - r - INFO: - Episode: 680/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:07 - r - INFO: - Current episode 680 has the best eval reward: 200.00 -2022-10-31 23:25:08 - r - INFO: - Episode: 681/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:08 - r - INFO: - Episode: 682/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:09 - r - INFO: - Episode: 683/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:09 - r - INFO: - Episode: 684/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:10 - r - INFO: - Episode: 685/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:10 - r - INFO: - Current episode 685 has the best eval reward: 200.00 -2022-10-31 23:25:11 - r - INFO: - Episode: 686/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:11 - r - INFO: - Episode: 687/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:12 - r - INFO: - Episode: 688/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:12 - r - INFO: - Episode: 689/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:13 - r - INFO: - Episode: 690/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:14 - r - INFO: - Episode: 691/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:14 - r - INFO: - Episode: 692/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:15 - r - INFO: - Episode: 693/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:15 - r - INFO: - Episode: 694/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:15 - r - INFO: - Episode: 695/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:17 - r - INFO: - Episode: 696/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:17 - r - INFO: - Episode: 697/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:17 - r - INFO: - Episode: 698/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:18 - r - INFO: - Episode: 699/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:18 - r - INFO: - Episode: 700/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:19 - r - INFO: - Episode: 701/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:20 - r - INFO: - Episode: 702/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:20 - r - INFO: - Episode: 703/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:21 - r - INFO: - Episode: 704/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:21 - r - INFO: - Episode: 705/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:22 - r - INFO: - Current episode 705 has the best eval reward: 200.00 -2022-10-31 23:25:22 - r - INFO: - Episode: 706/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:23 - r - INFO: - Episode: 707/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:23 - r - INFO: - Episode: 708/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:24 - r - INFO: - Episode: 709/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:24 - r - INFO: - Episode: 710/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:25 - r - INFO: - Current episode 710 has the best eval reward: 200.00 -2022-10-31 23:25:26 - r - INFO: - Episode: 711/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:26 - r - INFO: - Episode: 712/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:26 - r - INFO: - Episode: 713/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:27 - r - INFO: - Episode: 714/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:27 - r - INFO: - Episode: 715/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:28 - r - INFO: - Current episode 715 has the best eval reward: 200.00 -2022-10-31 23:25:28 - r - INFO: - Episode: 716/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:29 - r - INFO: - Episode: 717/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:29 - r - INFO: - Episode: 718/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:30 - r - INFO: - Episode: 719/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:30 - r - INFO: - Episode: 720/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:31 - r - INFO: - Current episode 720 has the best eval reward: 200.00 -2022-10-31 23:25:31 - r - INFO: - Episode: 721/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:32 - r - INFO: - Episode: 722/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:32 - r - INFO: - Episode: 723/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:33 - r - INFO: - Episode: 724/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:33 - r - INFO: - Episode: 725/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:34 - r - INFO: - Current episode 725 has the best eval reward: 200.00 -2022-10-31 23:25:34 - r - INFO: - Episode: 726/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:35 - r - INFO: - Episode: 727/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:35 - r - INFO: - Episode: 728/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:35 - r - INFO: - Episode: 729/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:36 - r - INFO: - Episode: 730/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:37 - r - INFO: - Current episode 730 has the best eval reward: 200.00 -2022-10-31 23:25:37 - r - INFO: - Episode: 731/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:37 - r - INFO: - Episode: 732/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:38 - r - INFO: - Episode: 733/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:38 - r - INFO: - Episode: 734/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:39 - r - INFO: - Episode: 735/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:39 - r - INFO: - Current episode 735 has the best eval reward: 200.00 -2022-10-31 23:25:40 - r - INFO: - Episode: 736/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:40 - r - INFO: - Episode: 737/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:41 - r - INFO: - Episode: 738/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:41 - r - INFO: - Episode: 739/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:42 - r - INFO: - Episode: 740/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:42 - r - INFO: - Current episode 740 has the best eval reward: 200.00 -2022-10-31 23:25:43 - r - INFO: - Episode: 741/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:43 - r - INFO: - Episode: 742/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:44 - r - INFO: - Episode: 743/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:44 - r - INFO: - Episode: 744/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:44 - r - INFO: - Episode: 745/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:45 - r - INFO: - Current episode 745 has the best eval reward: 200.00 -2022-10-31 23:25:46 - r - INFO: - Episode: 746/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:46 - r - INFO: - Episode: 747/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:46 - r - INFO: - Episode: 748/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:47 - r - INFO: - Episode: 749/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:47 - r - INFO: - Episode: 750/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:48 - r - INFO: - Current episode 750 has the best eval reward: 200.00 -2022-10-31 23:25:48 - r - INFO: - Episode: 751/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:49 - r - INFO: - Episode: 752/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:49 - r - INFO: - Episode: 753/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:50 - r - INFO: - Episode: 754/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:50 - r - INFO: - Episode: 755/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:51 - r - INFO: - Current episode 755 has the best eval reward: 200.00 -2022-10-31 23:25:51 - r - INFO: - Episode: 756/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:52 - r - INFO: - Episode: 757/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:52 - r - INFO: - Episode: 758/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:53 - r - INFO: - Episode: 759/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:53 - r - INFO: - Episode: 760/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:54 - r - INFO: - Current episode 760 has the best eval reward: 200.00 -2022-10-31 23:25:54 - r - INFO: - Episode: 761/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:55 - r - INFO: - Episode: 762/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:55 - r - INFO: - Episode: 763/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:55 - r - INFO: - Episode: 764/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:56 - r - INFO: - Episode: 765/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:57 - r - INFO: - Current episode 765 has the best eval reward: 200.00 -2022-10-31 23:25:57 - r - INFO: - Episode: 766/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:57 - r - INFO: - Episode: 767/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:58 - r - INFO: - Episode: 768/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:58 - r - INFO: - Episode: 769/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:59 - r - INFO: - Episode: 770/1000, Reward: 200.0, Step: 200 -2022-10-31 23:25:59 - r - INFO: - Current episode 770 has the best eval reward: 200.00 -2022-10-31 23:26:00 - r - INFO: - Episode: 771/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:00 - r - INFO: - Episode: 772/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:01 - r - INFO: - Episode: 773/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:01 - r - INFO: - Episode: 774/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:02 - r - INFO: - Episode: 775/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:02 - r - INFO: - Current episode 775 has the best eval reward: 200.00 -2022-10-31 23:26:03 - r - INFO: - Episode: 776/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:03 - r - INFO: - Episode: 777/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:04 - r - INFO: - Episode: 778/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:04 - r - INFO: - Episode: 779/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:04 - r - INFO: - Episode: 780/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:06 - r - INFO: - Episode: 781/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:06 - r - INFO: - Episode: 782/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:06 - r - INFO: - Episode: 783/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:07 - r - INFO: - Episode: 784/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:07 - r - INFO: - Episode: 785/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:08 - r - INFO: - Episode: 786/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:09 - r - INFO: - Episode: 787/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:09 - r - INFO: - Episode: 788/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:10 - r - INFO: - Episode: 789/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:10 - r - INFO: - Episode: 790/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:11 - r - INFO: - Current episode 790 has the best eval reward: 200.00 -2022-10-31 23:26:11 - r - INFO: - Episode: 791/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:12 - r - INFO: - Episode: 792/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:12 - r - INFO: - Episode: 793/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:12 - r - INFO: - Episode: 794/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:13 - r - INFO: - Episode: 795/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:14 - r - INFO: - Current episode 795 has the best eval reward: 200.00 -2022-10-31 23:26:14 - r - INFO: - Episode: 796/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:14 - r - INFO: - Episode: 797/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:15 - r - INFO: - Episode: 798/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:15 - r - INFO: - Episode: 799/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:16 - r - INFO: - Episode: 800/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:16 - r - INFO: - Current episode 800 has the best eval reward: 200.00 -2022-10-31 23:26:17 - r - INFO: - Episode: 801/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:17 - r - INFO: - Episode: 802/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:18 - r - INFO: - Episode: 803/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:18 - r - INFO: - Episode: 804/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:19 - r - INFO: - Episode: 805/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:19 - r - INFO: - Current episode 805 has the best eval reward: 200.00 -2022-10-31 23:26:20 - r - INFO: - Episode: 806/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:20 - r - INFO: - Episode: 807/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:21 - r - INFO: - Episode: 808/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:21 - r - INFO: - Episode: 809/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:21 - r - INFO: - Episode: 810/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:22 - r - INFO: - Current episode 810 has the best eval reward: 200.00 -2022-10-31 23:26:23 - r - INFO: - Episode: 811/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:23 - r - INFO: - Episode: 812/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:23 - r - INFO: - Episode: 813/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:24 - r - INFO: - Episode: 814/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:24 - r - INFO: - Episode: 815/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:25 - r - INFO: - Current episode 815 has the best eval reward: 200.00 -2022-10-31 23:26:25 - r - INFO: - Episode: 816/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:26 - r - INFO: - Episode: 817/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:26 - r - INFO: - Episode: 818/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:27 - r - INFO: - Episode: 819/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:27 - r - INFO: - Episode: 820/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:28 - r - INFO: - Current episode 820 has the best eval reward: 200.00 -2022-10-31 23:26:28 - r - INFO: - Episode: 821/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:29 - r - INFO: - Episode: 822/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:29 - r - INFO: - Episode: 823/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:30 - r - INFO: - Episode: 824/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:30 - r - INFO: - Episode: 825/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:31 - r - INFO: - Current episode 825 has the best eval reward: 200.00 -2022-10-31 23:26:31 - r - INFO: - Episode: 826/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:32 - r - INFO: - Episode: 827/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:32 - r - INFO: - Episode: 828/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:32 - r - INFO: - Episode: 829/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:33 - r - INFO: - Episode: 830/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:34 - r - INFO: - Current episode 830 has the best eval reward: 200.00 -2022-10-31 23:26:34 - r - INFO: - Episode: 831/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:34 - r - INFO: - Episode: 832/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:35 - r - INFO: - Episode: 833/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:35 - r - INFO: - Episode: 834/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:36 - r - INFO: - Episode: 835/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:36 - r - INFO: - Current episode 835 has the best eval reward: 200.00 -2022-10-31 23:26:37 - r - INFO: - Episode: 836/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:37 - r - INFO: - Episode: 837/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:38 - r - INFO: - Episode: 838/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:38 - r - INFO: - Episode: 839/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:38 - r - INFO: - Episode: 840/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:39 - r - INFO: - Current episode 840 has the best eval reward: 200.00 -2022-10-31 23:26:40 - r - INFO: - Episode: 841/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:40 - r - INFO: - Episode: 842/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:40 - r - INFO: - Episode: 843/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:41 - r - INFO: - Episode: 844/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:41 - r - INFO: - Episode: 845/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:42 - r - INFO: - Current episode 845 has the best eval reward: 200.00 -2022-10-31 23:26:42 - r - INFO: - Episode: 846/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:43 - r - INFO: - Episode: 847/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:43 - r - INFO: - Episode: 848/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:44 - r - INFO: - Episode: 849/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:44 - r - INFO: - Episode: 850/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:45 - r - INFO: - Current episode 850 has the best eval reward: 200.00 -2022-10-31 23:26:45 - r - INFO: - Episode: 851/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:46 - r - INFO: - Episode: 852/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:46 - r - INFO: - Episode: 853/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:47 - r - INFO: - Episode: 854/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:47 - r - INFO: - Episode: 855/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:48 - r - INFO: - Current episode 855 has the best eval reward: 200.00 -2022-10-31 23:26:48 - r - INFO: - Episode: 856/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:49 - r - INFO: - Episode: 857/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:49 - r - INFO: - Episode: 858/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:49 - r - INFO: - Episode: 859/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:50 - r - INFO: - Episode: 860/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:51 - r - INFO: - Current episode 860 has the best eval reward: 200.00 -2022-10-31 23:26:51 - r - INFO: - Episode: 861/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:51 - r - INFO: - Episode: 862/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:52 - r - INFO: - Episode: 863/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:52 - r - INFO: - Episode: 864/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:53 - r - INFO: - Episode: 865/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:53 - r - INFO: - Current episode 865 has the best eval reward: 200.00 -2022-10-31 23:26:54 - r - INFO: - Episode: 866/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:54 - r - INFO: - Episode: 867/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:55 - r - INFO: - Episode: 868/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:55 - r - INFO: - Episode: 869/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:56 - r - INFO: - Episode: 870/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:56 - r - INFO: - Current episode 870 has the best eval reward: 200.00 -2022-10-31 23:26:57 - r - INFO: - Episode: 871/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:57 - r - INFO: - Episode: 872/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:58 - r - INFO: - Episode: 873/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:58 - r - INFO: - Episode: 874/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:59 - r - INFO: - Episode: 875/1000, Reward: 200.0, Step: 200 -2022-10-31 23:26:59 - r - INFO: - Current episode 875 has the best eval reward: 200.00 -2022-10-31 23:27:00 - r - INFO: - Episode: 876/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:00 - r - INFO: - Episode: 877/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:01 - r - INFO: - Episode: 878/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:01 - r - INFO: - Episode: 879/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:01 - r - INFO: - Episode: 880/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:02 - r - INFO: - Current episode 880 has the best eval reward: 200.00 -2022-10-31 23:27:03 - r - INFO: - Episode: 881/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:03 - r - INFO: - Episode: 882/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:03 - r - INFO: - Episode: 883/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:04 - r - INFO: - Episode: 884/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:04 - r - INFO: - Episode: 885/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:05 - r - INFO: - Current episode 885 has the best eval reward: 200.00 -2022-10-31 23:27:05 - r - INFO: - Episode: 886/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:06 - r - INFO: - Episode: 887/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:06 - r - INFO: - Episode: 888/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:07 - r - INFO: - Episode: 889/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:07 - r - INFO: - Episode: 890/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:08 - r - INFO: - Current episode 890 has the best eval reward: 200.00 -2022-10-31 23:27:08 - r - INFO: - Episode: 891/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:09 - r - INFO: - Episode: 892/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:09 - r - INFO: - Episode: 893/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:10 - r - INFO: - Episode: 894/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:10 - r - INFO: - Episode: 895/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:11 - r - INFO: - Current episode 895 has the best eval reward: 200.00 -2022-10-31 23:27:11 - r - INFO: - Episode: 896/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:12 - r - INFO: - Episode: 897/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:12 - r - INFO: - Episode: 898/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:12 - r - INFO: - Episode: 899/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:13 - r - INFO: - Episode: 900/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:14 - r - INFO: - Current episode 900 has the best eval reward: 200.00 -2022-10-31 23:27:14 - r - INFO: - Episode: 901/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:15 - r - INFO: - Episode: 902/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:15 - r - INFO: - Episode: 903/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:16 - r - INFO: - Episode: 904/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:16 - r - INFO: - Episode: 905/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:17 - r - INFO: - Current episode 905 has the best eval reward: 200.00 -2022-10-31 23:27:17 - r - INFO: - Episode: 906/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:18 - r - INFO: - Episode: 907/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:18 - r - INFO: - Episode: 908/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:19 - r - INFO: - Episode: 909/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:19 - r - INFO: - Episode: 910/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:20 - r - INFO: - Current episode 910 has the best eval reward: 200.00 -2022-10-31 23:27:20 - r - INFO: - Episode: 911/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:21 - r - INFO: - Episode: 912/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:21 - r - INFO: - Episode: 913/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:21 - r - INFO: - Episode: 914/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:22 - r - INFO: - Episode: 915/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:23 - r - INFO: - Current episode 915 has the best eval reward: 200.00 -2022-10-31 23:27:23 - r - INFO: - Episode: 916/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:23 - r - INFO: - Episode: 917/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:24 - r - INFO: - Episode: 918/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:24 - r - INFO: - Episode: 919/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:25 - r - INFO: - Episode: 920/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:25 - r - INFO: - Current episode 920 has the best eval reward: 200.00 -2022-10-31 23:27:26 - r - INFO: - Episode: 921/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:26 - r - INFO: - Episode: 922/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:27 - r - INFO: - Episode: 923/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:27 - r - INFO: - Episode: 924/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:28 - r - INFO: - Episode: 925/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:29 - r - INFO: - Episode: 926/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:29 - r - INFO: - Episode: 927/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:30 - r - INFO: - Episode: 928/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:30 - r - INFO: - Episode: 929/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:30 - r - INFO: - Episode: 930/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:31 - r - INFO: - Episode: 931/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:32 - r - INFO: - Episode: 932/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:32 - r - INFO: - Episode: 933/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:33 - r - INFO: - Episode: 934/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:33 - r - INFO: - Episode: 935/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:34 - r - INFO: - Episode: 936/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:35 - r - INFO: - Episode: 937/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:35 - r - INFO: - Episode: 938/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:36 - r - INFO: - Episode: 939/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:36 - r - INFO: - Episode: 940/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:37 - r - INFO: - Episode: 941/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:38 - r - INFO: - Episode: 942/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:38 - r - INFO: - Episode: 943/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:38 - r - INFO: - Episode: 944/1000, Reward: 153.0, Step: 153 -2022-10-31 23:27:39 - r - INFO: - Episode: 945/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:40 - r - INFO: - Episode: 946/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:40 - r - INFO: - Episode: 947/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:41 - r - INFO: - Episode: 948/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:41 - r - INFO: - Episode: 949/1000, Reward: 150.0, Step: 150 -2022-10-31 23:27:41 - r - INFO: - Episode: 950/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:43 - r - INFO: - Episode: 951/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:43 - r - INFO: - Episode: 952/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:43 - r - INFO: - Episode: 953/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:44 - r - INFO: - Episode: 954/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:44 - r - INFO: - Episode: 955/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:45 - r - INFO: - Episode: 956/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:46 - r - INFO: - Episode: 957/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:46 - r - INFO: - Episode: 958/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:47 - r - INFO: - Episode: 959/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:47 - r - INFO: - Episode: 960/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:48 - r - INFO: - Current episode 960 has the best eval reward: 200.00 -2022-10-31 23:27:48 - r - INFO: - Episode: 961/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:49 - r - INFO: - Episode: 962/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:49 - r - INFO: - Episode: 963/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:49 - r - INFO: - Episode: 964/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:50 - r - INFO: - Episode: 965/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:51 - r - INFO: - Current episode 965 has the best eval reward: 200.00 -2022-10-31 23:27:51 - r - INFO: - Episode: 966/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:51 - r - INFO: - Episode: 967/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:52 - r - INFO: - Episode: 968/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:52 - r - INFO: - Episode: 969/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:53 - r - INFO: - Episode: 970/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:53 - r - INFO: - Current episode 970 has the best eval reward: 200.00 -2022-10-31 23:27:54 - r - INFO: - Episode: 971/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:54 - r - INFO: - Episode: 972/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:55 - r - INFO: - Episode: 973/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:55 - r - INFO: - Episode: 974/1000, Reward: 161.0, Step: 161 -2022-10-31 23:27:55 - r - INFO: - Episode: 975/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:57 - r - INFO: - Episode: 976/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:57 - r - INFO: - Episode: 977/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:57 - r - INFO: - Episode: 978/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:58 - r - INFO: - Episode: 979/1000, Reward: 200.0, Step: 200 -2022-10-31 23:27:58 - r - INFO: - Episode: 980/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:00 - r - INFO: - Episode: 981/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:00 - r - INFO: - Episode: 982/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:01 - r - INFO: - Episode: 983/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:01 - r - INFO: - Episode: 984/1000, Reward: 111.0, Step: 111 -2022-10-31 23:28:01 - r - INFO: - Episode: 985/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:03 - r - INFO: - Episode: 986/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:03 - r - INFO: - Episode: 987/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:04 - r - INFO: - Episode: 988/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:04 - r - INFO: - Episode: 989/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:04 - r - INFO: - Episode: 990/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:05 - r - INFO: - Current episode 990 has the best eval reward: 200.00 -2022-10-31 23:28:06 - r - INFO: - Episode: 991/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:06 - r - INFO: - Episode: 992/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:07 - r - INFO: - Episode: 993/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:07 - r - INFO: - Episode: 994/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:07 - r - INFO: - Episode: 995/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:09 - r - INFO: - Episode: 996/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:09 - r - INFO: - Episode: 997/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:09 - r - INFO: - Episode: 998/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:10 - r - INFO: - Episode: 999/1000, Reward: 154.0, Step: 154 -2022-10-31 23:28:10 - r - INFO: - Episode: 1000/1000, Reward: 200.0, Step: 200 -2022-10-31 23:28:11 - r - INFO: - Finish training! diff --git a/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/models/actor_checkpoint.pt b/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/models/actor_checkpoint.pt deleted file mode 100644 index 05bd7b6..0000000 Binary files a/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/models/actor_checkpoint.pt and /dev/null differ diff --git a/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/models/critic_checkpoint.pt b/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/models/critic_checkpoint.pt deleted file mode 100644 index 720f388..0000000 Binary files a/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/models/critic_checkpoint.pt and /dev/null differ diff --git a/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/results/learning_curve.png b/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/results/learning_curve.png deleted file mode 100644 index 841a786..0000000 Binary files a/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/results/res.csv b/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/results/res.csv deleted file mode 100644 index ee82c68..0000000 --- a/projects/codes/A2C/Train_CartPole-v1_A2C_20221031-232138/results/res.csv +++ /dev/null @@ -1,1001 +0,0 @@ -episodes,rewards,steps -0,25.0,25 -1,11.0,11 -2,32.0,32 -3,11.0,11 -4,14.0,14 -5,11.0,11 -6,23.0,23 -7,27.0,27 -8,10.0,10 -9,21.0,21 -10,15.0,15 -11,26.0,26 -12,22.0,22 -13,14.0,14 -14,14.0,14 -15,21.0,21 -16,10.0,10 -17,19.0,19 -18,18.0,18 -19,26.0,26 -20,29.0,29 -21,40.0,40 -22,35.0,35 -23,33.0,33 -24,47.0,47 -25,50.0,50 -26,21.0,21 -27,30.0,30 -28,26.0,26 -29,40.0,40 -30,31.0,31 -31,54.0,54 -32,59.0,59 -33,50.0,50 -34,26.0,26 -35,34.0,34 -36,25.0,25 -37,166.0,166 -38,35.0,35 -39,25.0,25 -40,110.0,110 -41,22.0,22 -42,57.0,57 -43,45.0,45 -44,35.0,35 -45,45.0,45 -46,51.0,51 -47,32.0,32 -48,67.0,67 -49,46.0,46 -50,61.0,61 -51,49.0,49 -52,47.0,47 -53,37.0,37 -54,32.0,32 -55,31.0,31 -56,33.0,33 -57,93.0,93 -58,60.0,60 -59,128.0,128 -60,200.0,200 -61,47.0,47 -62,47.0,47 -63,63.0,63 -64,68.0,68 -65,45.0,45 -66,101.0,101 -67,47.0,47 -68,49.0,49 -69,54.0,54 -70,42.0,42 -71,77.0,77 -72,67.0,67 -73,41.0,41 -74,89.0,89 -75,51.0,51 -76,54.0,54 -77,37.0,37 -78,49.0,49 -79,46.0,46 -80,31.0,31 -81,43.0,43 -82,60.0,60 -83,41.0,41 -84,40.0,40 -85,28.0,28 -86,50.0,50 -87,159.0,159 -88,30.0,30 -89,34.0,34 -90,70.0,70 -91,22.0,22 -92,39.0,39 -93,50.0,50 -94,40.0,40 -95,37.0,37 -96,121.0,121 -97,26.0,26 -98,40.0,40 -99,30.0,30 -100,35.0,35 -101,40.0,40 -102,28.0,28 -103,29.0,29 -104,42.0,42 -105,54.0,54 -106,25.0,25 -107,47.0,47 -108,32.0,32 -109,50.0,50 -110,30.0,30 -111,58.0,58 -112,32.0,32 -113,43.0,43 -114,57.0,57 -115,20.0,20 -116,48.0,48 -117,45.0,45 -118,47.0,47 -119,69.0,69 -120,34.0,34 -121,22.0,22 -122,22.0,22 -123,38.0,38 -124,36.0,36 -125,41.0,41 -126,28.0,28 -127,35.0,35 -128,48.0,48 -129,51.0,51 -130,51.0,51 -131,36.0,36 -132,45.0,45 -133,27.0,27 -134,40.0,40 -135,43.0,43 -136,64.0,64 -137,43.0,43 -138,37.0,37 -139,38.0,38 -140,69.0,69 -141,36.0,36 -142,28.0,28 -143,58.0,58 -144,43.0,43 -145,50.0,50 -146,30.0,30 -147,42.0,42 -148,42.0,42 -149,35.0,35 -150,67.0,67 -151,45.0,45 -152,28.0,28 -153,59.0,59 -154,64.0,64 -155,67.0,67 -156,41.0,41 -157,81.0,81 -158,76.0,76 -159,91.0,91 -160,119.0,119 -161,47.0,47 -162,64.0,64 -163,178.0,178 -164,97.0,97 -165,181.0,181 -166,166.0,166 -167,79.0,79 -168,141.0,141 -169,119.0,119 -170,81.0,81 -171,124.0,124 -172,150.0,150 -173,98.0,98 -174,164.0,164 -175,200.0,200 -176,115.0,115 -177,116.0,116 -178,160.0,160 -179,103.0,103 -180,181.0,181 -181,185.0,185 -182,93.0,93 -183,110.0,110 -184,200.0,200 -185,141.0,141 -186,150.0,150 -187,121.0,121 -188,110.0,110 -189,115.0,115 -190,114.0,114 -191,45.0,45 -192,125.0,125 -193,142.0,142 -194,54.0,54 -195,62.0,62 -196,122.0,122 -197,58.0,58 -198,88.0,88 -199,141.0,141 -200,113.0,113 -201,200.0,200 -202,136.0,136 -203,114.0,114 -204,102.0,102 -205,176.0,176 -206,150.0,150 -207,105.0,105 -208,200.0,200 -209,200.0,200 -210,167.0,167 -211,104.0,104 -212,124.0,124 -213,96.0,96 -214,200.0,200 -215,199.0,199 -216,200.0,200 -217,132.0,132 -218,188.0,188 -219,132.0,132 -220,151.0,151 -221,125.0,125 -222,42.0,42 -223,200.0,200 -224,159.0,159 -225,171.0,171 -226,122.0,122 -227,189.0,189 -228,129.0,129 -229,106.0,106 -230,107.0,107 -231,200.0,200 -232,200.0,200 -233,200.0,200 -234,200.0,200 -235,158.0,158 -236,200.0,200 -237,192.0,192 -238,179.0,179 -239,102.0,102 -240,125.0,125 -241,138.0,138 -242,189.0,189 -243,41.0,41 -244,97.0,97 -245,49.0,49 -246,86.0,86 -247,121.0,121 -248,117.0,117 -249,43.0,43 -250,72.0,72 -251,34.0,34 -252,83.0,83 -253,83.0,83 -254,38.0,38 -255,34.0,34 -256,99.0,99 -257,45.0,45 -258,47.0,47 -259,44.0,44 -260,26.0,26 -261,37.0,37 -262,26.0,26 -263,43.0,43 -264,27.0,27 -265,24.0,24 -266,42.0,42 -267,86.0,86 -268,23.0,23 -269,32.0,32 -270,57.0,57 -271,25.0,25 -272,98.0,98 -273,29.0,29 -274,25.0,25 -275,29.0,29 -276,39.0,39 -277,20.0,20 -278,92.0,92 -279,28.0,28 -280,78.0,78 -281,25.0,25 -282,31.0,31 -283,88.0,88 -284,85.0,85 -285,37.0,37 -286,26.0,26 -287,19.0,19 -288,40.0,40 -289,27.0,27 -290,17.0,17 -291,27.0,27 -292,26.0,26 -293,82.0,82 -294,36.0,36 -295,24.0,24 -296,30.0,30 -297,20.0,20 -298,34.0,34 -299,30.0,30 -300,23.0,23 -301,36.0,36 -302,29.0,29 -303,34.0,34 -304,25.0,25 -305,42.0,42 -306,88.0,88 -307,26.0,26 -308,85.0,85 -309,89.0,89 -310,48.0,48 -311,83.0,83 -312,109.0,109 -313,42.0,42 -314,93.0,93 -315,85.0,85 -316,100.0,100 -317,106.0,106 -318,28.0,28 -319,108.0,108 -320,112.0,112 -321,88.0,88 -322,108.0,108 -323,108.0,108 -324,90.0,90 -325,112.0,112 -326,113.0,113 -327,94.0,94 -328,99.0,99 -329,45.0,45 -330,121.0,121 -331,102.0,102 -332,111.0,111 -333,54.0,54 -334,198.0,198 -335,83.0,83 -336,107.0,107 -337,101.0,101 -338,129.0,129 -339,88.0,88 -340,86.0,86 -341,199.0,199 -342,95.0,95 -343,103.0,103 -344,100.0,100 -345,89.0,89 -346,87.0,87 -347,110.0,110 -348,127.0,127 -349,97.0,97 -350,34.0,34 -351,123.0,123 -352,49.0,49 -353,96.0,96 -354,90.0,90 -355,110.0,110 -356,93.0,93 -357,102.0,102 -358,128.0,128 -359,125.0,125 -360,92.0,92 -361,109.0,109 -362,114.0,114 -363,111.0,111 -364,38.0,38 -365,55.0,55 -366,106.0,106 -367,115.0,115 -368,103.0,103 -369,50.0,50 -370,110.0,110 -371,102.0,102 -372,110.0,110 -373,29.0,29 -374,35.0,35 -375,42.0,42 -376,62.0,62 -377,119.0,119 -378,33.0,33 -379,31.0,31 -380,97.0,97 -381,192.0,192 -382,179.0,179 -383,89.0,89 -384,32.0,32 -385,33.0,33 -386,52.0,52 -387,31.0,31 -388,22.0,22 -389,118.0,118 -390,24.0,24 -391,115.0,115 -392,20.0,20 -393,33.0,33 -394,40.0,40 -395,27.0,27 -396,26.0,26 -397,24.0,24 -398,19.0,19 -399,22.0,22 -400,24.0,24 -401,18.0,18 -402,23.0,23 -403,27.0,27 -404,20.0,20 -405,27.0,27 -406,17.0,17 -407,27.0,27 -408,25.0,25 -409,25.0,25 -410,24.0,24 -411,24.0,24 -412,18.0,18 -413,20.0,20 -414,27.0,27 -415,28.0,28 -416,30.0,30 -417,28.0,28 -418,33.0,33 -419,24.0,24 -420,96.0,96 -421,26.0,26 -422,29.0,29 -423,25.0,25 -424,38.0,38 -425,33.0,33 -426,23.0,23 -427,39.0,39 -428,28.0,28 -429,97.0,97 -430,30.0,30 -431,29.0,29 -432,103.0,103 -433,36.0,36 -434,32.0,32 -435,41.0,41 -436,111.0,111 -437,48.0,48 -438,24.0,24 -439,49.0,49 -440,116.0,116 -441,118.0,118 -442,94.0,94 -443,132.0,132 -444,41.0,41 -445,105.0,105 -446,116.0,116 -447,136.0,136 -448,137.0,137 -449,45.0,45 -450,157.0,157 -451,116.0,116 -452,125.0,125 -453,120.0,120 -454,150.0,150 -455,114.0,114 -456,44.0,44 -457,138.0,138 -458,133.0,133 -459,141.0,141 -460,124.0,124 -461,143.0,143 -462,123.0,123 -463,134.0,134 -464,152.0,152 -465,140.0,140 -466,200.0,200 -467,168.0,168 -468,200.0,200 -469,200.0,200 -470,200.0,200 -471,200.0,200 -472,200.0,200 -473,200.0,200 -474,200.0,200 -475,200.0,200 -476,200.0,200 -477,200.0,200 -478,200.0,200 -479,200.0,200 -480,200.0,200 -481,200.0,200 -482,200.0,200 -483,200.0,200 -484,200.0,200 -485,200.0,200 -486,200.0,200 -487,200.0,200 -488,200.0,200 -489,200.0,200 -490,200.0,200 -491,200.0,200 -492,200.0,200 -493,200.0,200 -494,200.0,200 -495,169.0,169 -496,200.0,200 -497,200.0,200 -498,200.0,200 -499,200.0,200 -500,200.0,200 -501,200.0,200 -502,200.0,200 -503,200.0,200 -504,200.0,200 -505,200.0,200 -506,200.0,200 -507,200.0,200 -508,200.0,200 -509,200.0,200 -510,200.0,200 -511,200.0,200 -512,200.0,200 -513,200.0,200 -514,200.0,200 -515,200.0,200 -516,200.0,200 -517,200.0,200 -518,200.0,200 -519,200.0,200 -520,200.0,200 -521,200.0,200 -522,200.0,200 -523,200.0,200 -524,200.0,200 -525,200.0,200 -526,200.0,200 -527,200.0,200 -528,186.0,186 -529,200.0,200 -530,200.0,200 -531,200.0,200 -532,200.0,200 -533,200.0,200 -534,200.0,200 -535,200.0,200 -536,200.0,200 -537,200.0,200 -538,200.0,200 -539,200.0,200 -540,200.0,200 -541,200.0,200 -542,200.0,200 -543,84.0,84 -544,200.0,200 -545,200.0,200 -546,200.0,200 -547,200.0,200 -548,200.0,200 -549,200.0,200 -550,200.0,200 -551,200.0,200 -552,200.0,200 -553,200.0,200 -554,200.0,200 -555,200.0,200 -556,200.0,200 -557,200.0,200 -558,200.0,200 -559,200.0,200 -560,200.0,200 -561,200.0,200 -562,200.0,200 -563,200.0,200 -564,200.0,200 -565,200.0,200 -566,200.0,200 -567,200.0,200 -568,200.0,200 -569,200.0,200 -570,200.0,200 -571,200.0,200 -572,200.0,200 -573,200.0,200 -574,200.0,200 -575,200.0,200 -576,200.0,200 -577,200.0,200 -578,200.0,200 -579,200.0,200 -580,200.0,200 -581,200.0,200 -582,200.0,200 -583,200.0,200 -584,199.0,199 -585,200.0,200 -586,200.0,200 -587,178.0,178 -588,200.0,200 -589,188.0,188 -590,156.0,156 -591,165.0,165 -592,131.0,131 -593,157.0,157 -594,170.0,170 -595,123.0,123 -596,109.0,109 -597,124.0,124 -598,113.0,113 -599,38.0,38 -600,107.0,107 -601,115.0,115 -602,101.0,101 -603,113.0,113 -604,100.0,100 -605,109.0,109 -606,119.0,119 -607,117.0,117 -608,108.0,108 -609,101.0,101 -610,110.0,110 -611,59.0,59 -612,112.0,112 -613,104.0,104 -614,45.0,45 -615,29.0,29 -616,42.0,42 -617,74.0,74 -618,79.0,79 -619,50.0,50 -620,30.0,30 -621,43.0,43 -622,77.0,77 -623,36.0,36 -624,61.0,61 -625,36.0,36 -626,30.0,30 -627,43.0,43 -628,27.0,27 -629,88.0,88 -630,42.0,42 -631,40.0,40 -632,59.0,59 -633,81.0,81 -634,85.0,85 -635,55.0,55 -636,40.0,40 -637,99.0,99 -638,104.0,104 -639,117.0,117 -640,112.0,112 -641,43.0,43 -642,96.0,96 -643,105.0,105 -644,115.0,115 -645,99.0,99 -646,123.0,123 -647,123.0,123 -648,40.0,40 -649,100.0,100 -650,124.0,124 -651,106.0,106 -652,122.0,122 -653,127.0,127 -654,121.0,121 -655,121.0,121 -656,125.0,125 -657,127.0,127 -658,132.0,132 -659,142.0,142 -660,134.0,134 -661,147.0,147 -662,175.0,175 -663,180.0,180 -664,183.0,183 -665,167.0,167 -666,179.0,179 -667,173.0,173 -668,200.0,200 -669,200.0,200 -670,184.0,184 -671,200.0,200 -672,193.0,193 -673,200.0,200 -674,200.0,200 -675,200.0,200 -676,199.0,199 -677,200.0,200 -678,200.0,200 -679,200.0,200 -680,200.0,200 -681,200.0,200 -682,200.0,200 -683,200.0,200 -684,200.0,200 -685,200.0,200 -686,200.0,200 -687,200.0,200 -688,200.0,200 -689,200.0,200 -690,200.0,200 -691,200.0,200 -692,200.0,200 -693,200.0,200 -694,200.0,200 -695,200.0,200 -696,200.0,200 -697,200.0,200 -698,200.0,200 -699,200.0,200 -700,200.0,200 -701,200.0,200 -702,200.0,200 -703,200.0,200 -704,200.0,200 -705,200.0,200 -706,200.0,200 -707,200.0,200 -708,200.0,200 -709,200.0,200 -710,200.0,200 -711,200.0,200 -712,200.0,200 -713,200.0,200 -714,200.0,200 -715,200.0,200 -716,200.0,200 -717,200.0,200 -718,200.0,200 -719,200.0,200 -720,200.0,200 -721,200.0,200 -722,200.0,200 -723,200.0,200 -724,200.0,200 -725,200.0,200 -726,200.0,200 -727,200.0,200 -728,200.0,200 -729,200.0,200 -730,200.0,200 -731,200.0,200 -732,200.0,200 -733,200.0,200 -734,200.0,200 -735,200.0,200 -736,200.0,200 -737,200.0,200 -738,200.0,200 -739,200.0,200 -740,200.0,200 -741,200.0,200 -742,200.0,200 -743,200.0,200 -744,200.0,200 -745,200.0,200 -746,200.0,200 -747,200.0,200 -748,200.0,200 -749,200.0,200 -750,200.0,200 -751,200.0,200 -752,200.0,200 -753,200.0,200 -754,200.0,200 -755,200.0,200 -756,200.0,200 -757,200.0,200 -758,200.0,200 -759,200.0,200 -760,200.0,200 -761,200.0,200 -762,200.0,200 -763,200.0,200 -764,200.0,200 -765,200.0,200 -766,200.0,200 -767,200.0,200 -768,200.0,200 -769,200.0,200 -770,200.0,200 -771,200.0,200 -772,200.0,200 -773,200.0,200 -774,200.0,200 -775,200.0,200 -776,200.0,200 -777,200.0,200 -778,200.0,200 -779,200.0,200 -780,200.0,200 -781,200.0,200 -782,200.0,200 -783,200.0,200 -784,200.0,200 -785,200.0,200 -786,200.0,200 -787,200.0,200 -788,200.0,200 -789,200.0,200 -790,200.0,200 -791,200.0,200 -792,200.0,200 -793,200.0,200 -794,200.0,200 -795,200.0,200 -796,200.0,200 -797,200.0,200 -798,200.0,200 -799,200.0,200 -800,200.0,200 -801,200.0,200 -802,200.0,200 -803,200.0,200 -804,200.0,200 -805,200.0,200 -806,200.0,200 -807,200.0,200 -808,200.0,200 -809,200.0,200 -810,200.0,200 -811,200.0,200 -812,200.0,200 -813,200.0,200 -814,200.0,200 -815,200.0,200 -816,200.0,200 -817,200.0,200 -818,200.0,200 -819,200.0,200 -820,200.0,200 -821,200.0,200 -822,200.0,200 -823,200.0,200 -824,200.0,200 -825,200.0,200 -826,200.0,200 -827,200.0,200 -828,200.0,200 -829,200.0,200 -830,200.0,200 -831,200.0,200 -832,200.0,200 -833,200.0,200 -834,200.0,200 -835,200.0,200 -836,200.0,200 -837,200.0,200 -838,200.0,200 -839,200.0,200 -840,200.0,200 -841,200.0,200 -842,200.0,200 -843,200.0,200 -844,200.0,200 -845,200.0,200 -846,200.0,200 -847,200.0,200 -848,200.0,200 -849,200.0,200 -850,200.0,200 -851,200.0,200 -852,200.0,200 -853,200.0,200 -854,200.0,200 -855,200.0,200 -856,200.0,200 -857,200.0,200 -858,200.0,200 -859,200.0,200 -860,200.0,200 -861,200.0,200 -862,200.0,200 -863,200.0,200 -864,200.0,200 -865,200.0,200 -866,200.0,200 -867,200.0,200 -868,200.0,200 -869,200.0,200 -870,200.0,200 -871,200.0,200 -872,200.0,200 -873,200.0,200 -874,200.0,200 -875,200.0,200 -876,200.0,200 -877,200.0,200 -878,200.0,200 -879,200.0,200 -880,200.0,200 -881,200.0,200 -882,200.0,200 -883,200.0,200 -884,200.0,200 -885,200.0,200 -886,200.0,200 -887,200.0,200 -888,200.0,200 -889,200.0,200 -890,200.0,200 -891,200.0,200 -892,200.0,200 -893,200.0,200 -894,200.0,200 -895,200.0,200 -896,200.0,200 -897,200.0,200 -898,200.0,200 -899,200.0,200 -900,200.0,200 -901,200.0,200 -902,200.0,200 -903,200.0,200 -904,200.0,200 -905,200.0,200 -906,200.0,200 -907,200.0,200 -908,200.0,200 -909,200.0,200 -910,200.0,200 -911,200.0,200 -912,200.0,200 -913,200.0,200 -914,200.0,200 -915,200.0,200 -916,200.0,200 -917,200.0,200 -918,200.0,200 -919,200.0,200 -920,200.0,200 -921,200.0,200 -922,200.0,200 -923,200.0,200 -924,200.0,200 -925,200.0,200 -926,200.0,200 -927,200.0,200 -928,200.0,200 -929,200.0,200 -930,200.0,200 -931,200.0,200 -932,200.0,200 -933,200.0,200 -934,200.0,200 -935,200.0,200 -936,200.0,200 -937,200.0,200 -938,200.0,200 -939,200.0,200 -940,200.0,200 -941,200.0,200 -942,200.0,200 -943,153.0,153 -944,200.0,200 -945,200.0,200 -946,200.0,200 -947,200.0,200 -948,150.0,150 -949,200.0,200 -950,200.0,200 -951,200.0,200 -952,200.0,200 -953,200.0,200 -954,200.0,200 -955,200.0,200 -956,200.0,200 -957,200.0,200 -958,200.0,200 -959,200.0,200 -960,200.0,200 -961,200.0,200 -962,200.0,200 -963,200.0,200 -964,200.0,200 -965,200.0,200 -966,200.0,200 -967,200.0,200 -968,200.0,200 -969,200.0,200 -970,200.0,200 -971,200.0,200 -972,200.0,200 -973,161.0,161 -974,200.0,200 -975,200.0,200 -976,200.0,200 -977,200.0,200 -978,200.0,200 -979,200.0,200 -980,200.0,200 -981,200.0,200 -982,200.0,200 -983,111.0,111 -984,200.0,200 -985,200.0,200 -986,200.0,200 -987,200.0,200 -988,200.0,200 -989,200.0,200 -990,200.0,200 -991,200.0,200 -992,200.0,200 -993,200.0,200 -994,200.0,200 -995,200.0,200 -996,200.0,200 -997,200.0,200 -998,154.0,154 -999,200.0,200 diff --git a/projects/codes/A2C/a2c.py b/projects/codes/A2C/a2c.py deleted file mode 100644 index f822451..0000000 --- a/projects/codes/A2C/a2c.py +++ /dev/null @@ -1,103 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-08-16 23:05:25 -LastEditor: JiangJi -LastEditTime: 2022-11-01 00:33:49 -Discription: -''' -import torch -import numpy as np -from torch.distributions import Categorical,Normal - - -class A2C: - def __init__(self,models,memories,cfg): - self.n_actions = cfg.n_actions - self.gamma = cfg.gamma - self.device = torch.device(cfg.device) - self.continuous = cfg.continuous - if hasattr(cfg,'action_bound'): - self.action_bound = cfg.action_bound - self.memory = memories['ACMemory'] - self.actor = models['Actor'].to(self.device) - self.critic = models['Critic'].to(self.device) - self.actor_optim = torch.optim.Adam(self.actor.parameters(), lr=cfg.actor_lr) - self.critic_optim = torch.optim.Adam(self.critic.parameters(), lr=cfg.critic_lr) - def sample_action(self,state): - # state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(dim=0) - # dist = self.actor(state) - # self.entropy = - np.sum(np.mean(dist.detach().cpu().numpy()) * np.log(dist.detach().cpu().numpy())) - # value = self.critic(state) # note that 'dist' need require_grad=True - # self.value = value.detach().cpu().numpy().squeeze(0)[0] - # action = np.random.choice(self.n_actions, p=dist.detach().cpu().numpy().squeeze(0)) # shape(p=(n_actions,1) - # self.log_prob = torch.log(dist.squeeze(0)[action]) - if self.continuous: - state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(dim=0) - mu, sigma = self.actor(state) - dist = Normal(self.action_bound * mu.view(1,), sigma.view(1,)) - action = dist.sample() - value = self.critic(state) - # self.entropy = - np.sum(np.mean(dist.detach().cpu().numpy()) * np.log(dist.detach().cpu().numpy())) - self.value = value.detach().cpu().numpy().squeeze(0)[0] # detach() to avoid gradient - self.log_prob = dist.log_prob(action).squeeze(dim=0) # Tensor([0.]) - self.entropy = dist.entropy().cpu().detach().numpy().squeeze(0) # detach() to avoid gradient - return action.cpu().detach().numpy() - else: - state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(dim=0) - probs = self.actor(state) - dist = Categorical(probs) - action = dist.sample() # Tensor([0]) - value = self.critic(state) - self.value = value.detach().cpu().numpy().squeeze(0)[0] # detach() to avoid gradient - self.log_prob = dist.log_prob(action).squeeze(dim=0) # Tensor([0.]) - self.entropy = dist.entropy().cpu().detach().numpy().squeeze(0) # detach() to avoid gradient - return action.cpu().numpy().item() - @torch.no_grad() - def predict_action(self,state): - if self.continuous: - state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(dim=0) - mu, sigma = self.actor(state) - dist = Normal(self.action_bound * mu.view(1,), sigma.view(1,)) - action = dist.sample() - return action.cpu().detach().numpy() - else: - state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(dim=0) - dist = self.actor(state) - # value = self.critic(state) # note that 'dist' need require_grad=True - # value = value.detach().cpu().numpy().squeeze(0)[0] - action = np.random.choice(self.n_actions, p=dist.detach().cpu().numpy().squeeze(0)) # shape(p=(n_actions,1) - return action - def update(self,next_state,entropy): - value_pool,log_prob_pool,reward_pool = self.memory.sample() - value_pool = torch.tensor(value_pool, device=self.device) - log_prob_pool = torch.stack(log_prob_pool) - next_state = torch.tensor(next_state, device=self.device, dtype=torch.float32).unsqueeze(dim=0) - next_value = self.critic(next_state) - returns = np.zeros_like(reward_pool) - for t in reversed(range(len(reward_pool))): - next_value = reward_pool[t] + self.gamma * next_value # G(s_{t},a{t}) = r_{t+1} + gamma * V(s_{t+1}) - returns[t] = next_value - returns = torch.tensor(returns, device=self.device) - advantages = returns - value_pool - actor_loss = (-log_prob_pool * advantages).mean() - critic_loss = 0.5 * advantages.pow(2).mean() - tot_loss = actor_loss + critic_loss + 0.001 * entropy - self.actor_optim.zero_grad() - self.critic_optim.zero_grad() - tot_loss.backward() - self.actor_optim.step() - self.critic_optim.step() - self.memory.clear() - def save_model(self, path): - from pathlib import Path - # create path - Path(path).mkdir(parents=True, exist_ok=True) - torch.save(self.actor.state_dict(), f"{path}/actor_checkpoint.pt") - torch.save(self.critic.state_dict(), f"{path}/critic_checkpoint.pt") - - def load_model(self, path): - self.actor.load_state_dict(torch.load(f"{path}/actor_checkpoint.pt")) - self.critic.load_state_dict(torch.load(f"{path}/critic_checkpoint.pt")) \ No newline at end of file diff --git a/projects/codes/A2C/a2c_2.py b/projects/codes/A2C/a2c_2.py deleted file mode 100644 index e29acdc..0000000 --- a/projects/codes/A2C/a2c_2.py +++ /dev/null @@ -1,65 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-09-19 14:48:16 -LastEditor: JiangJi -LastEditTime: 2022-10-30 01:21:50 -Discription: #TODO，待更新模版 -''' -import torch -import numpy as np - -class A2C_2: - def __init__(self,models,memories,cfg): - self.n_actions = cfg.n_actions - self.gamma = cfg.gamma - self.device = torch.device(cfg.device) - self.memory = memories['ACMemory'] - self.ac_net = models['ActorCritic'].to(self.device) - self.ac_optimizer = torch.optim.Adam(self.ac_net.parameters(), lr = cfg.lr) - def sample_action(self,state): - state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(dim=0) - value, dist = self.ac_net(state) # note that 'dist' need require_grad=True - value = value.detach().numpy().squeeze(0)[0] - action = np.random.choice(self.n_actions, p=dist.detach().numpy().squeeze(0)) # shape(p=(n_actions,1) - return action,value,dist - def predict_action(self,state): - ''' predict can be all wrapped with no_grad(), then donot need detach(), or you can just copy contents of 'sample_action' - ''' - with torch.no_grad(): - state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(dim=0) - value, dist = self.ac_net(state) - value = value.numpy().squeeze(0)[0] # shape(value) = (1,) - action = np.random.choice(self.n_actions, p=dist.numpy().squeeze(0)) # shape(p=(n_actions,1) - return action,value,dist - def update(self,next_state,entropy): - value_pool,log_prob_pool,reward_pool = self.memory.sample() - next_state = torch.tensor(next_state, device=self.device, dtype=torch.float32).unsqueeze(dim=0) - next_value,_ = self.ac_net(next_state) - returns = np.zeros_like(reward_pool) - for t in reversed(range(len(reward_pool))): - next_value = reward_pool[t] + self.gamma * next_value # G(s_{t},a{t}) = r_{t+1} + gamma * V(s_{t+1}) - returns[t] = next_value - returns = torch.tensor(returns, device=self.device) - value_pool = torch.tensor(value_pool, device=self.device) - advantages = returns - value_pool - log_prob_pool = torch.stack(log_prob_pool) - actor_loss = (-log_prob_pool * advantages).mean() - critic_loss = 0.5 * advantages.pow(2).mean() - ac_loss = actor_loss + critic_loss + 0.001 * entropy - self.ac_optimizer.zero_grad() - ac_loss.backward() - self.ac_optimizer.step() - self.memory.clear() - def save_model(self, path): - from pathlib import Path - # create path - Path(path).mkdir(parents=True, exist_ok=True) - torch.save(self.ac_net.state_dict(), f"{path}/a2c_checkpoint.pt") - - def load_model(self, path): - self.ac_net.load_state_dict(torch.load(f"{path}/a2c_checkpoint.pt")) - - \ No newline at end of file diff --git a/projects/codes/A2C/config/CartPole-v1_A2C_Test.yaml b/projects/codes/A2C/config/CartPole-v1_A2C_Test.yaml deleted file mode 100644 index d148bb0..0000000 --- a/projects/codes/A2C/config/CartPole-v1_A2C_Test.yaml +++ /dev/null @@ -1,21 +0,0 @@ -general_cfg: - algo_name: A2C - device: cuda - env_name: CartPole-v1 - mode: test - load_checkpoint: true - load_path: Train_CartPole-v1_A2C_20221031-232138 - max_steps: 200 - save_fig: true - seed: 1 - show_fig: false - test_eps: 20 - train_eps: 1000 -algo_cfg: - continuous: false - batch_size: 64 - buffer_size: 100000 - gamma: 0.99 - actor_lr: 0.0003 - critic_lr: 0.001 - target_update: 4 diff --git a/projects/codes/A2C/config/CartPole-v1_A2C_Train.yaml b/projects/codes/A2C/config/CartPole-v1_A2C_Train.yaml deleted file mode 100644 index f79f148..0000000 --- a/projects/codes/A2C/config/CartPole-v1_A2C_Train.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: A2C - device: cuda - env_name: CartPole-v1 - mode: train - load_checkpoint: false - load_path: Train_CartPole-v1_DQN_20221026-054757 - max_steps: 200 - save_fig: true - seed: 1 - show_fig: false - test_eps: 20 - train_eps: 600 -algo_cfg: - continuous: false - batch_size: 64 - buffer_size: 100000 - gamma: 0.0003 - lr: 0.001 diff --git a/projects/codes/A2C/config/Pendulum-v1_A2C_Train.yaml b/projects/codes/A2C/config/Pendulum-v1_A2C_Train.yaml deleted file mode 100644 index a1680c9..0000000 --- a/projects/codes/A2C/config/Pendulum-v1_A2C_Train.yaml +++ /dev/null @@ -1,21 +0,0 @@ -general_cfg: - algo_name: A2C - device: cuda - env_name: Pendulum-v1 - mode: train - eval_per_episode: 200 - load_checkpoint: false - load_path: Train_CartPole-v1_DQN_20221026-054757 - max_steps: 200 - save_fig: true - seed: 1 - show_fig: false - test_eps: 20 - train_eps: 1000 -algo_cfg: - continuous: true - batch_size: 64 - buffer_size: 100000 - gamma: 0.0003 - actor_lr: 0.0003 - critic_lr: 0.001 diff --git a/projects/codes/A2C/config/config.py b/projects/codes/A2C/config/config.py deleted file mode 100644 index a552d38..0000000 --- a/projects/codes/A2C/config/config.py +++ /dev/null @@ -1,38 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-10-30 00:53:03 -LastEditor: JiangJi -LastEditTime: 2022-11-01 00:17:55 -Discription: default parameters of A2C -''' -from common.config import GeneralConfig,AlgoConfig - -class GeneralConfigA2C(GeneralConfig): - def __init__(self) -> None: - self.env_name = "CartPole-v1" # name of environment - self.algo_name = "A2C" # name of algorithm - self.mode = "train" # train or test - self.seed = 1 # random seed - self.device = "cuda" # device to use - self.train_eps = 1000 # number of episodes for training - self.test_eps = 20 # number of episodes for testing - self.max_steps = 200 # max steps for each episode - self.load_checkpoint = False - self.load_path = "tasks" # path to load model - self.show_fig = False # show figure or not - self.save_fig = True # save figure or not - -class AlgoConfigA2C(AlgoConfig): - def __init__(self) -> None: - self.continuous = False # continuous or discrete action space - self.hidden_dim = 256 # hidden_dim for MLP - self.gamma = 0.99 # discount factor - self.actor_lr = 3e-4 # learning rate of actor - self.critic_lr = 1e-3 # learning rate of critic - self.actor_hidden_dim = 256 # hidden_dim for actor MLP - self.critic_hidden_dim = 256 # hidden_dim for critic MLP - self.buffer_size = 100000 # size of replay buffer - self.batch_size = 64 # batch size \ No newline at end of file diff --git a/projects/codes/A2C/main2.py b/projects/codes/A2C/main2.py deleted file mode 100644 index 60bd7c2..0000000 --- a/projects/codes/A2C/main2.py +++ /dev/null @@ -1,130 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-09-19 14:48:16 -LastEditor: JiangJi -LastEditTime: 2022-10-30 01:21:15 -Discription: #TODO，待更新模版 -''' -import sys,os -os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE" # avoid "OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized." -curr_path = os.path.dirname(os.path.abspath(__file__)) # current path -parent_path = os.path.dirname(curr_path) # parent path -sys.path.append(parent_path) # add path to system path - -import datetime -import argparse -import gym -import torch -import numpy as np -from common.utils import all_seed -from common.launcher import Launcher -from common.memories import PGReplay -from common.models import ActorCriticSoftmax -from envs.register import register_env -from a2c_2 import A2C_2 - -class Main(Launcher): - def get_args(self): - curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # obtain current time - parser = argparse.ArgumentParser(description="hyperparameters") - parser.add_argument('--algo_name',default='A2C',type=str,help="name of algorithm") - parser.add_argument('--env_name',default='CartPole-v0',type=str,help="name of environment") - parser.add_argument('--train_eps',default=2000,type=int,help="episodes of training") - parser.add_argument('--test_eps',default=20,type=int,help="episodes of testing") - parser.add_argument('--ep_max_steps',default = 100000,type=int,help="steps per episode, much larger value can simulate infinite steps") - parser.add_argument('--gamma',default=0.99,type=float,help="discounted factor") - parser.add_argument('--lr',default=3e-4,type=float,help="learning rate") - parser.add_argument('--actor_hidden_dim',default=256,type=int) - parser.add_argument('--critic_hidden_dim',default=256,type=int) - parser.add_argument('--device',default='cpu',type=str,help="cpu or cuda") - parser.add_argument('--seed',default=10,type=int,help="seed") - parser.add_argument('--show_fig',default=False,type=bool,help="if show figure or not") - parser.add_argument('--save_fig',default=True,type=bool,help="if save figure or not") - args = parser.parse_args() - default_args = {'result_path':f"{curr_path}/outputs/{args.env_name}/{curr_time}/results/", - 'model_path':f"{curr_path}/outputs/{args.env_name}/{curr_time}/models/", - } - args = {**vars(args),**default_args} # type(dict) - return args - def env_agent_config(self,cfg): - ''' create env and agent - ''' - register_env(cfg['env_name']) - env = gym.make(cfg['env_name']) - if cfg['seed'] !=0: # set random seed - all_seed(env,seed=cfg["seed"]) - try: # state dimension - n_states = env.observation_space.n # print(hasattr(env.observation_space, 'n')) - except AttributeError: - n_states = env.observation_space.shape[0] # print(hasattr(env.observation_space, 'shape')) - n_actions = env.action_space.n # action dimension - print(f"n_states: {n_states}, n_actions: {n_actions}") - cfg.update({"n_states":n_states,"n_actions":n_actions}) # update to cfg paramters - models = {'ActorCritic':ActorCriticSoftmax(cfg['n_states'],cfg['n_actions'], actor_hidden_dim = cfg['actor_hidden_dim'],critic_hidden_dim=cfg['critic_hidden_dim'])} - memories = {'ACMemory':PGReplay()} - agent = A2C_2(models,memories,cfg) - return env,agent - def train(self,cfg,env,agent): - print("Start training!") - print(f"Env: {cfg['env_name']}, Algorithm: {cfg['algo_name']}, Device: {cfg['device']}") - rewards = [] # record rewards for all episodes - steps = [] # record steps for all episodes - - for i_ep in range(cfg['train_eps']): - ep_reward = 0 # reward per episode - ep_step = 0 # step per episode - ep_entropy = 0 - state = env.reset() # reset and obtain initial state - - for _ in range(cfg['ep_max_steps']): - action, value, dist = agent.sample_action(state) # sample action - next_state, reward, done, _ = env.step(action) # update env and return transitions - log_prob = torch.log(dist.squeeze(0)[action]) - entropy = -np.sum(np.mean(dist.detach().numpy()) * np.log(dist.detach().numpy())) - agent.memory.push((value,log_prob,reward)) # save transitions - state = next_state # update state - ep_reward += reward - ep_entropy += entropy - ep_step += 1 - if done: - break - agent.update(next_state,ep_entropy) # update agent - rewards.append(ep_reward) - steps.append(ep_step) - if (i_ep+1)%10==0: - print(f'Episode: {i_ep+1}/{cfg["train_eps"]}, Reward: {ep_reward:.2f}, Steps:{ep_step}') - print("Finish training!") - return {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - def test(self,cfg,env,agent): - print("Start testing!") - print(f"Env: {cfg['env_name']}, Algorithm: {cfg['algo_name']}, Device: {cfg['device']}") - rewards = [] # record rewards for all episodes - steps = [] # record steps for all episodes - for i_ep in range(cfg['test_eps']): - ep_reward = 0 # reward per episode - ep_step = 0 - state = env.reset() # reset and obtain initial state - for _ in range(cfg['ep_max_steps']): - action,_,_ = agent.predict_action(state) # predict action - next_state, reward, done, _ = env.step(action) - state = next_state - ep_reward += reward - ep_step += 1 - if done: - break - rewards.append(ep_reward) - steps.append(ep_step) - print(f"Episode: {i_ep+1}/{cfg['test_eps']}, Steps:{ep_step}, Reward: {ep_reward:.2f}") - print("Finish testing!") - return {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - -if __name__ == "__main__": - main = Main() - main.run() - - - - diff --git a/projects/codes/A2C/task0.py b/projects/codes/A2C/task0.py deleted file mode 100644 index 4a3208a..0000000 --- a/projects/codes/A2C/task0.py +++ /dev/null @@ -1,142 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-10-30 01:19:43 -LastEditor: JiangJi -LastEditTime: 2022-11-01 01:21:06 -Discription: -''' -import sys,os -os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE" # avoid "OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized." -curr_path = os.path.dirname(os.path.abspath(__file__)) # current path -parent_path = os.path.dirname(curr_path) # parent path -sys.path.append(parent_path) # add path to system path - -import gym -from common.utils import all_seed,merge_class_attrs -from common.launcher import Launcher -from common.memories import PGReplay -from common.models import ActorSoftmax,Critic -from envs.register import register_env -from a2c import A2C -from config.config import GeneralConfigA2C,AlgoConfigA2C - -class Main(Launcher): - def __init__(self) -> None: - super().__init__() - self.cfgs['general_cfg'] = merge_class_attrs(self.cfgs['general_cfg'],GeneralConfigA2C()) - self.cfgs['algo_cfg'] = merge_class_attrs(self.cfgs['algo_cfg'],AlgoConfigA2C()) - def env_agent_config(self,cfg,logger): - ''' create env and agent - ''' - register_env(cfg.env_name) - env = gym.make(cfg.env_name,new_step_api=True) # create env - if cfg.seed !=0: # set random seed - all_seed(env,seed = cfg.seed) - try: # state dimension - n_states = env.observation_space.n # print(hasattr(env.observation_space, 'n')) - except AttributeError: - n_states = env.observation_space.shape[0] # print(hasattr(env.observation_space, 'shape')) - n_actions = env.action_space.n # action dimension - logger.info(f"n_states: {n_states}, n_actions: {n_actions}") # print info - # update to cfg paramters - setattr(cfg, 'n_states', n_states) - setattr(cfg, 'n_actions', n_actions) - models = {'Actor':ActorSoftmax(n_states,n_actions, hidden_dim = cfg.actor_hidden_dim),'Critic':Critic(n_states,1,hidden_dim=cfg.critic_hidden_dim)} - memories = {'ACMemory':PGReplay()} - agent = A2C(models,memories,cfg) - for k,v in models.items(): - logger.info(f"{k} model name: {type(v).__name__}") - for k,v in memories.items(): - logger.info(f"{k} memory name: {type(v).__name__}") - logger.info(f"agent name: {type(agent).__name__}") - return env,agent - def train_one_episode(self, env, agent, cfg): - ep_reward = 0 # reward per episode - ep_step = 0 # step per episode - ep_entropy = 0 # entropy per episode - state = env.reset() # reset and obtain initial state - for _ in range(cfg.max_steps): - action = agent.sample_action(state) # sample action - next_state, reward, terminated, truncated , info = env.step(action) # update env and return transitions - agent.memory.push((agent.value,agent.log_prob,reward)) # save transitions - state = next_state # update state - ep_reward += reward - ep_entropy += agent.entropy - ep_step += 1 - if terminated: - break - agent.update(next_state,ep_entropy) # update agent - return agent,ep_reward,ep_step - def test_one_episode(self, env, agent, cfg): - ep_reward = 0 # reward per episode - ep_step = 0 # step per episode - state = env.reset() # reset and obtain initial state - for _ in range(cfg.max_steps): - action = agent.predict_action(state) # predict action - next_state, reward, terminated, truncated , info = env.step(action) - state = next_state - ep_reward += reward - ep_step += 1 - if terminated: - break - return agent,ep_reward,ep_step - # def train(self,cfg,env,agent,logger): - # logger.info("Start training!") - # logger.info(f"Env: {cfg.env_name}, Algorithm: {cfg.algo_name}, Device: {cfg.device}") - # rewards = [] # record rewards for all episodes - # steps = [] # record steps for all episodes - # for i_ep in range(cfg.train_eps): - # ep_reward = 0 # reward per episode - # ep_step = 0 # step per episode - # ep_entropy = 0 - # state = env.reset() # reset and obtain initial state - # for _ in range(cfg.max_steps): - # action = agent.sample_action(state) # sample action - # next_state, reward, terminated, truncated , info = env.step(action) # update env and return transitions - # agent.memory.push((agent.value,agent.log_prob,reward)) # save transitions - # state = next_state # update state - # ep_reward += reward - # ep_entropy += agent.entropy - # ep_step += 1 - # if terminated: - # break - # agent.update(next_state,ep_entropy) # update agent - # rewards.append(ep_reward) - # steps.append(ep_step) - # logger.info(f"Episode: {i_ep+1}/{cfg.train_eps}, Reward: {ep_reward:.2f}, Steps:{ep_step}") - # logger.info("Finish training!") - # return {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - # def test(self,cfg,env,agent,logger): - # logger.info("Start testing!") - # logger.info(f"Env: {cfg.env_name}, Algorithm: {cfg.algo_name}, Device: {cfg.device}") - # rewards = [] # record rewards for all episodes - # steps = [] # record steps for all episodes - # for i_ep in range(cfg.test_eps): - # ep_reward = 0 # reward per episode - # ep_step = 0 - # state = env.reset() # reset and obtain initial state - # for _ in range(cfg.max_steps): - # action = agent.predict_action(state) # predict action - # next_state, reward, terminated, truncated , info = env.step(action) - # state = next_state - # ep_reward += reward - # ep_step += 1 - # if terminated: - # break - # rewards.append(ep_reward) - # steps.append(ep_step) - # logger.info(f"Episode: {i_ep+1}/{cfg.test_eps}, Reward: {ep_reward:.2f}, Steps:{ep_step}") - # logger.info("Finish testing!") - # env.close() - # return {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - -if __name__ == "__main__": - main = Main() - main.run() - - - - diff --git a/projects/codes/A2C/task1.py b/projects/codes/A2C/task1.py deleted file mode 100644 index ff7c86f..0000000 --- a/projects/codes/A2C/task1.py +++ /dev/null @@ -1,142 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-10-30 01:19:43 -LastEditor: JiangJi -LastEditTime: 2022-11-01 01:21:12 -Discription: continuous action space -''' -import sys,os -os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE" # avoid "OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized." -curr_path = os.path.dirname(os.path.abspath(__file__)) # current path -parent_path = os.path.dirname(curr_path) # parent path -sys.path.append(parent_path) # add path to system path - -import gym -from common.utils import all_seed,merge_class_attrs -from common.launcher import Launcher -from common.memories import PGReplay -from common.models import ActorSoftmaxTanh,Critic -from envs.register import register_env -from a2c import A2C -from config.config import GeneralConfigA2C,AlgoConfigA2C - -class Main(Launcher): - def __init__(self) -> None: - super().__init__() - self.cfgs['general_cfg'] = merge_class_attrs(self.cfgs['general_cfg'],GeneralConfigA2C()) - self.cfgs['algo_cfg'] = merge_class_attrs(self.cfgs['algo_cfg'],AlgoConfigA2C()) - def env_agent_config(self,cfg,logger): - ''' create env and agent - ''' - register_env(cfg.env_name) - env = gym.make(cfg.env_name,new_step_api=True) # create env - if cfg.seed !=0: # set random seed - all_seed(env,seed = cfg.seed) - try: # state dimension - n_states = env.observation_space.n # print(hasattr(env.observation_space, 'n')) - except AttributeError: - n_states = env.observation_space.shape[0] # print(hasattr(env.observation_space, 'shape')) - n_actions = env.action_space.n # action dimension - logger.info(f"n_states: {n_states}, n_actions: {n_actions}") # print info - # update to cfg paramters - setattr(cfg, 'n_states', n_states) - setattr(cfg, 'n_actions', n_actions) - models = {'Actor':ActorSoftmaxTanh(n_states,n_actions, hidden_dim = cfg.actor_hidden_dim),'Critic':Critic(n_states,1,hidden_dim=cfg.critic_hidden_dim)} - memories = {'ACMemory':PGReplay()} - agent = A2C(models,memories,cfg) - for k,v in models.items(): - logger.info(f"{k} model name: {type(v).__name__}") - for k,v in memories.items(): - logger.info(f"{k} memory name: {type(v).__name__}") - logger.info(f"agent name: {type(agent).__name__}") - return env,agent - def train_one_episode(self, env, agent, cfg): - ep_reward = 0 # reward per episode - ep_step = 0 # step per episode - ep_entropy = 0 # entropy per episode - state = env.reset() # reset and obtain initial state - for _ in range(cfg.max_steps): - action = agent.sample_action(state) # sample action - next_state, reward, terminated, truncated , info = env.step(action) # update env and return transitions - agent.memory.push((agent.value,agent.log_prob,reward)) # save transitions - state = next_state # update state - ep_reward += reward - ep_entropy += agent.entropy - ep_step += 1 - if terminated: - break - agent.update(next_state,ep_entropy) # update agent - return agent,ep_reward,ep_step - def test_one_episode(self, env, agent, cfg): - ep_reward = 0 # reward per episode - ep_step = 0 # step per episode - state = env.reset() # reset and obtain initial state - for _ in range(cfg.max_steps): - action = agent.predict_action(state) # predict action - next_state, reward, terminated, truncated , info = env.step(action) - state = next_state - ep_reward += reward - ep_step += 1 - if terminated: - break - return agent,ep_reward,ep_step - # def train(self,cfg,env,agent,logger): - # logger.info("Start training!") - # logger.info(f"Env: {cfg.env_name}, Algorithm: {cfg.algo_name}, Device: {cfg.device}") - # rewards = [] # record rewards for all episodes - # steps = [] # record steps for all episodes - # for i_ep in range(cfg.train_eps): - # ep_reward = 0 # reward per episode - # ep_step = 0 # step per episode - # ep_entropy = 0 - # state = env.reset() # reset and obtain initial state - # for _ in range(cfg.max_steps): - # action = agent.sample_action(state) # sample action - # next_state, reward, terminated, truncated , info = env.step(action) # update env and return transitions - # agent.memory.push((agent.value,agent.log_prob,reward)) # save transitions - # state = next_state # update state - # ep_reward += reward - # ep_entropy += agent.entropy - # ep_step += 1 - # if terminated: - # break - # agent.update(next_state,ep_entropy) # update agent - # rewards.append(ep_reward) - # steps.append(ep_step) - # logger.info(f"Episode: {i_ep+1}/{cfg.train_eps}, Reward: {ep_reward:.2f}, Steps:{ep_step}") - # logger.info("Finish training!") - # return {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - # def test(self,cfg,env,agent,logger): - # logger.info("Start testing!") - # logger.info(f"Env: {cfg.env_name}, Algorithm: {cfg.algo_name}, Device: {cfg.device}") - # rewards = [] # record rewards for all episodes - # steps = [] # record steps for all episodes - # for i_ep in range(cfg.test_eps): - # ep_reward = 0 # reward per episode - # ep_step = 0 - # state = env.reset() # reset and obtain initial state - # for _ in range(cfg.max_steps): - # action = agent.predict_action(state) # predict action - # next_state, reward, terminated, truncated , info = env.step(action) - # state = next_state - # ep_reward += reward - # ep_step += 1 - # if terminated: - # break - # rewards.append(ep_reward) - # steps.append(ep_step) - # logger.info(f"Episode: {i_ep+1}/{cfg.test_eps}, Reward: {ep_reward:.2f}, Steps:{ep_step}") - # logger.info("Finish testing!") - # env.close() - # return {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - -if __name__ == "__main__": - main = Main() - main.run() - - - - diff --git a/projects/codes/A2C/task2.py b/projects/codes/A2C/task2.py deleted file mode 100644 index 96c1cc2..0000000 --- a/projects/codes/A2C/task2.py +++ /dev/null @@ -1,149 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-10-30 01:19:43 -LastEditor: JiangJi -LastEditTime: 2022-11-01 00:08:22 -Discription: the only difference from task0.py is that the actor here we use ActorSoftmaxTanh instead of ActorSoftmax with ReLU -''' -import sys,os -os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE" # avoid "OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized." -curr_path = os.path.dirname(os.path.abspath(__file__)) # current path -parent_path = os.path.dirname(curr_path) # parent path -sys.path.append(parent_path) # add path to system path - -import gym -import torch -import numpy as np -from common.utils import all_seed,merge_class_attrs -from common.launcher import Launcher -from common.memories import PGReplay -from common.models import ActorNormal,Critic -from envs.register import register_env -from a2c import A2C -from config.config import GeneralConfigA2C,AlgoConfigA2C - -class Main(Launcher): - def __init__(self) -> None: - super().__init__() - self.cfgs['general_cfg'] = merge_class_attrs(self.cfgs['general_cfg'],GeneralConfigA2C()) - self.cfgs['algo_cfg'] = merge_class_attrs(self.cfgs['algo_cfg'],AlgoConfigA2C()) - def env_agent_config(self,cfg,logger): - ''' create env and agent - ''' - register_env(cfg.env_name) - env = gym.make(cfg.env_name,new_step_api=True) # create env - if cfg.seed !=0: # set random seed - all_seed(env,seed = cfg.seed) - try: # state dimension - n_states = env.observation_space.n # print(hasattr(env.observation_space, 'n')) - except AttributeError: - n_states = env.observation_space.shape[0] # print(hasattr(env.observation_space, 'shape')) - try: - n_actions = env.action_space.n # action dimension - except AttributeError: - n_actions = env.action_space.shape[0] - logger.info(f"action bound: {abs(env.action_space.low.item())}") - setattr(cfg, 'action_bound', abs(env.action_space.low.item())) - logger.info(f"n_states: {n_states}, n_actions: {n_actions}") # print info - # update to cfg paramters - setattr(cfg, 'n_states', n_states) - setattr(cfg, 'n_actions', n_actions) - models = {'Actor':ActorNormal(n_states,n_actions, hidden_dim = cfg.actor_hidden_dim),'Critic':Critic(n_states,1,hidden_dim=cfg.critic_hidden_dim)} - memories = {'ACMemory':PGReplay()} - agent = A2C(models,memories,cfg) - for k,v in models.items(): - logger.info(f"{k} model name: {type(v).__name__}") - for k,v in memories.items(): - logger.info(f"{k} memory name: {type(v).__name__}") - logger.info(f"agent name: {type(agent).__name__}") - return env,agent - def train_one_episode(self, env, agent, cfg): - ep_reward = 0 # reward per episode - ep_step = 0 # step per episode - ep_entropy = 0 # entropy per episode - state = env.reset() # reset and obtain initial state - for _ in range(cfg.max_steps): - action = agent.sample_action(state) # sample action - next_state, reward, terminated, truncated , info = env.step(action) # update env and return transitions - agent.memory.push((agent.value,agent.log_prob,reward)) # save transitions - state = next_state # update state - ep_reward += reward - ep_entropy += agent.entropy - ep_step += 1 - if terminated: - break - agent.update(next_state,ep_entropy) # update agent - return agent,ep_reward,ep_step - def test_one_episode(self, env, agent, cfg): - ep_reward = 0 # reward per episode - ep_step = 0 # step per episode - state = env.reset() # reset and obtain initial state - for _ in range(cfg.max_steps): - action = agent.predict_action(state) # predict action - next_state, reward, terminated, truncated , info = env.step(action) - state = next_state - ep_reward += reward - ep_step += 1 - if terminated: - break - return agent,ep_reward,ep_step - # def train(self,cfg,env,agent,logger): - # logger.info("Start training!") - # logger.info(f"Env: {cfg.env_name}, Algorithm: {cfg.algo_name}, Device: {cfg.device}") - # rewards = [] # record rewards for all episodes - # steps = [] # record steps for all episodes - # for i_ep in range(cfg.train_eps): - # ep_reward = 0 # reward per episode - # ep_step = 0 # step per episode - # ep_entropy = 0 - # state = env.reset() # reset and obtain initial state - # for _ in range(cfg.max_steps): - # action = agent.sample_action(state) # sample action - # next_state, reward, terminated, truncated , info = env.step(action) # update env and return transitions - # agent.memory.push((agent.value,agent.log_prob,reward)) # save transitions - # state = next_state # update state - # ep_reward += reward - # ep_entropy += agent.entropy - # ep_step += 1 - # if terminated: - # break - # agent.update(next_state,ep_entropy) # update agent - # rewards.append(ep_reward) - # steps.append(ep_step) - # logger.info(f"Episode: {i_ep+1}/{cfg.train_eps}, Reward: {ep_reward:.2f}, Steps:{ep_step}") - # logger.info("Finish training!") - # return {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - # def test(self,cfg,env,agent,logger): - # logger.info("Start testing!") - # logger.info(f"Env: {cfg.env_name}, Algorithm: {cfg.algo_name}, Device: {cfg.device}") - # rewards = [] # record rewards for all episodes - # steps = [] # record steps for all episodes - # for i_ep in range(cfg.test_eps): - # ep_reward = 0 # reward per episode - # ep_step = 0 - # state = env.reset() # reset and obtain initial state - # for _ in range(cfg.max_steps): - # action = agent.predict_action(state) # predict action - # next_state, reward, terminated, truncated , info = env.step(action) - # state = next_state - # ep_reward += reward - # ep_step += 1 - # if terminated: - # break - # rewards.append(ep_reward) - # steps.append(ep_step) - # logger.info(f"Episode: {i_ep+1}/{cfg.test_eps}, Reward: {ep_reward:.2f}, Steps:{ep_step}") - # logger.info("Finish testing!") - # env.close() - # return {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - -if __name__ == "__main__": - main = Main() - main.run() - - - - diff --git a/projects/codes/A3C/README.md b/projects/codes/A3C/README.md deleted file mode 100644 index 5856b80..0000000 --- a/projects/codes/A3C/README.md +++ /dev/null @@ -1,5 +0,0 @@ -## A2C - - - -https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f \ No newline at end of file diff --git a/projects/codes/A3C/a3c.py b/projects/codes/A3C/a3c.py deleted file mode 100644 index ba0ed7c..0000000 --- a/projects/codes/A3C/a3c.py +++ /dev/null @@ -1,56 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2021-05-03 22:16:08 -LastEditor: JiangJi -LastEditTime: 2022-07-20 23:54:40 -Discription: -Environment: -''' -import torch -import torch.optim as optim -import torch.nn as nn -import torch.nn.functional as F -from torch.distributions import Categorical - -class ActorCritic(nn.Module): - ''' A2C网络模型，包含一个Actor和Critic - ''' - def __init__(self, input_dim, output_dim, hidden_dim): - super(ActorCritic, self).__init__() - self.critic = nn.Sequential( - nn.Linear(input_dim, hidden_dim), - nn.ReLU(), - nn.Linear(hidden_dim, 1) - ) - - self.actor = nn.Sequential( - nn.Linear(input_dim, hidden_dim), - nn.ReLU(), - nn.Linear(hidden_dim, output_dim), - nn.Softmax(dim=1), - ) - - def forward(self, x): - value = self.critic(x) - probs = self.actor(x) - dist = Categorical(probs) - return dist, value -class A2C: - ''' A2C算法 - ''' - def __init__(self,n_states,n_actions,cfg) -> None: - self.gamma = cfg.gamma - self.device = torch.device(cfg.device) - self.model = ActorCritic(n_states, n_actions, cfg.hidden_size).to(self.device) - self.optimizer = optim.Adam(self.model.parameters()) - - def compute_returns(self,next_value, rewards, masks): - R = next_value - returns = [] - for step in reversed(range(len(rewards))): - R = rewards[step] + self.gamma * R * masks[step] - returns.insert(0, R) - return returns \ No newline at end of file diff --git a/projects/codes/A3C/outputs/CartPole-v0/20220713-221850/results/params.json b/projects/codes/A3C/outputs/CartPole-v0/20220713-221850/results/params.json deleted file mode 100644 index 2773964..0000000 --- a/projects/codes/A3C/outputs/CartPole-v0/20220713-221850/results/params.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "algo_name": "A2C", - "env_name": "CartPole-v0", - "n_envs": 8, - "max_steps": 20000, - "n_steps": 5, - "gamma": 0.99, - "lr": 0.001, - "hidden_dim": 256, - "deivce": "cpu", - "result_path": "C:\\Users\\24438\\Desktop\\rl-tutorials/outputs/CartPole-v0/20220713-221850/results/", - "model_path": "C:\\Users\\24438\\Desktop\\rl-tutorials/outputs/CartPole-v0/20220713-221850/models/", - "save_fig": true -} \ No newline at end of file diff --git a/projects/codes/A3C/outputs/CartPole-v0/20220713-221850/results/train_ma_rewards.npy b/projects/codes/A3C/outputs/CartPole-v0/20220713-221850/results/train_ma_rewards.npy deleted file mode 100644 index 66091a2..0000000 Binary files a/projects/codes/A3C/outputs/CartPole-v0/20220713-221850/results/train_ma_rewards.npy and /dev/null differ diff --git a/projects/codes/A3C/outputs/CartPole-v0/20220713-221850/results/train_rewards.npy b/projects/codes/A3C/outputs/CartPole-v0/20220713-221850/results/train_rewards.npy deleted file mode 100644 index 5e6ea3f..0000000 Binary files a/projects/codes/A3C/outputs/CartPole-v0/20220713-221850/results/train_rewards.npy and /dev/null differ diff --git a/projects/codes/A3C/outputs/CartPole-v0/20220713-221850/results/train_rewards_curve.png b/projects/codes/A3C/outputs/CartPole-v0/20220713-221850/results/train_rewards_curve.png deleted file mode 100644 index b8c0921..0000000 Binary files a/projects/codes/A3C/outputs/CartPole-v0/20220713-221850/results/train_rewards_curve.png and /dev/null differ diff --git a/projects/codes/A3C/task0.py b/projects/codes/A3C/task0.py deleted file mode 100644 index 09dcceb..0000000 --- a/projects/codes/A3C/task0.py +++ /dev/null @@ -1,137 +0,0 @@ -import sys,os -curr_path = os.path.dirname(os.path.abspath(__file__)) # current path -parent_path = os.path.dirname(curr_path) # parent path -sys.path.append(parent_path) # add to system path - -import gym -import numpy as np -import torch -import torch.optim as optim -import datetime -import argparse -from common.multiprocessing_env import SubprocVecEnv -from a3c import ActorCritic -from common.utils import save_results, make_dir -from common.utils import plot_rewards, save_args - - -def get_args(): - """ Hyperparameters - """ - curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # Obtain current time - parser = argparse.ArgumentParser(description="hyperparameters") - parser.add_argument('--algo_name',default='A2C',type=str,help="name of algorithm") - parser.add_argument('--env_name',default='CartPole-v0',type=str,help="name of environment") - parser.add_argument('--n_envs',default=8,type=int,help="numbers of environments") - - parser.add_argument('--max_steps',default=20000,type=int,help="episodes of training") - parser.add_argument('--n_steps',default=5,type=int,help="episodes of testing") - parser.add_argument('--gamma',default=0.99,type=float,help="discounted factor") - parser.add_argument('--lr',default=1e-3,type=float,help="learning rate") - parser.add_argument('--hidden_dim',default=256,type=int) - parser.add_argument('--device',default='cpu',type=str,help="cpu or cuda") - parser.add_argument('--result_path',default=curr_path + "/outputs/" + parser.parse_args().env_name + \ - '/' + curr_time + '/results/' ) - parser.add_argument('--model_path',default=curr_path + "/outputs/" + parser.parse_args().env_name + \ - '/' + curr_time + '/models/' ) # path to save models - parser.add_argument('--save_fig',default=True,type=bool,help="if save figure or not") - args = parser.parse_args() - return args - -def make_envs(env_name): - def _thunk(): - env = gym.make(env_name) - env.seed(2) - return env - return _thunk -def test_env(env,model,vis=False): - state = env.reset() - if vis: env.render() - done = False - total_reward = 0 - while not done: - state = torch.FloatTensor(state).unsqueeze(0).to(cfg.device) - dist, _ = model(state) - next_state, reward, done, _ = env.step(dist.sample().cpu().numpy()[0]) - state = next_state - if vis: env.render() - total_reward += reward - return total_reward - -def compute_returns(next_value, rewards, masks, gamma=0.99): - R = next_value - returns = [] - for step in reversed(range(len(rewards))): - R = rewards[step] + gamma * R * masks[step] - returns.insert(0, R) - return returns - - -def train(cfg,envs): - print('Start training!') - print(f'Env:{cfg.env_name}, Algorithm:{cfg.algo_name}, Device:{cfg.device}') - env = gym.make(cfg.env_name) # a single env - env.seed(10) - n_states = envs.observation_space.shape[0] - n_actions = envs.action_space.n - model = ActorCritic(n_states, n_actions, cfg.hidden_dim).to(cfg.device) - optimizer = optim.Adam(model.parameters()) - step_idx = 0 - test_rewards = [] - test_ma_rewards = [] - state = envs.reset() - while step_idx < cfg.max_steps: - log_probs = [] - values = [] - rewards = [] - masks = [] - entropy = 0 - # rollout trajectory - for _ in range(cfg.n_steps): - state = torch.FloatTensor(state).to(cfg.device) - dist, value = model(state) - action = dist.sample() - next_state, reward, done, _ = envs.step(action.cpu().numpy()) - log_prob = dist.log_prob(action) - entropy += dist.entropy().mean() - log_probs.append(log_prob) - values.append(value) - rewards.append(torch.FloatTensor(reward).unsqueeze(1).to(cfg.device)) - masks.append(torch.FloatTensor(1 - done).unsqueeze(1).to(cfg.device)) - state = next_state - step_idx += 1 - if step_idx % 100 == 0: - test_reward = np.mean([test_env(env,model) for _ in range(10)]) - print(f"step_idx:{step_idx}, test_reward:{test_reward}") - test_rewards.append(test_reward) - if test_ma_rewards: - test_ma_rewards.append(0.9*test_ma_rewards[-1]+0.1*test_reward) - else: - test_ma_rewards.append(test_reward) - # plot(step_idx, test_rewards) - next_state = torch.FloatTensor(next_state).to(cfg.device) - _, next_value = model(next_state) - returns = compute_returns(next_value, rewards, masks) - log_probs = torch.cat(log_probs) - returns = torch.cat(returns).detach() - values = torch.cat(values) - advantage = returns - values - actor_loss = -(log_probs * advantage.detach()).mean() - critic_loss = advantage.pow(2).mean() - loss = actor_loss + 0.5 * critic_loss - 0.001 * entropy - optimizer.zero_grad() - loss.backward() - optimizer.step() - print('Finish training！') - return {'rewards':test_rewards,'ma_rewards':test_ma_rewards} -if __name__ == "__main__": - cfg = get_args() - envs = [make_envs(cfg.env_name) for i in range(cfg.n_envs)] - envs = SubprocVecEnv(envs) - # training - res_dic = train(cfg,envs) - make_dir(cfg.result_path,cfg.model_path) - save_args(cfg) - save_results(res_dic, tag='train', - path=cfg.result_path) - plot_rewards(res_dic['rewards'], res_dic['ma_rewards'], cfg, tag="train") # 画出结果 diff --git a/projects/codes/DDPG/ddpg.py b/projects/codes/DDPG/ddpg.py deleted file mode 100644 index 246966b..0000000 --- a/projects/codes/DDPG/ddpg.py +++ /dev/null @@ -1,96 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -@Author: John -@Email: johnjim0816@gmail.com -@Date: 2020-06-09 20:25:52 -@LastEditor: John -LastEditTime: 2022-09-27 15:43:21 -@Discription: -@Environment: python 3.7.7 -''' -import random -import numpy as np -import torch -import torch.nn as nn -import torch.optim as optim - -class DDPG: - def __init__(self, models,memories,cfg): - self.device = torch.device(cfg['device']) - self.critic = models['critic'].to(self.device) - self.target_critic = models['critic'].to(self.device) - self.actor = models['actor'].to(self.device) - self.target_actor = models['actor'].to(self.device) - # copy weights from critic to target_critic - for target_param, param in zip(self.target_critic.parameters(), self.critic.parameters()): - target_param.data.copy_(param.data) - # copy weights from actor to target_actor - for target_param, param in zip(self.target_actor.parameters(), self.actor.parameters()): - target_param.data.copy_(param.data) - self.critic_optimizer = optim.Adam(self.critic.parameters(), lr=cfg['critic_lr']) - self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=cfg['actor_lr']) - self.memory = memories['memory'] - self.batch_size = cfg['batch_size'] - self.gamma = cfg['gamma'] - self.tau = cfg['tau'] - - def sample_action(self, state): - state = torch.FloatTensor(state).unsqueeze(0).to(self.device) - action = self.actor(state) - return action.detach().cpu().numpy()[0, 0] - @torch.no_grad() - def predict_action(self, state): - ''' predict action - ''' - state = torch.FloatTensor(state).unsqueeze(0).to(self.device) - action = self.actor(state) - return action.cpu().numpy()[0, 0] - - def update(self): - if len(self.memory) < self.batch_size: # when memory size is less than batch size, return - return - # sample a random minibatch of N transitions from R - state, action, reward, next_state, done = self.memory.sample(self.batch_size) - # convert to tensor - state = torch.FloatTensor(np.array(state)).to(self.device) - next_state = torch.FloatTensor(np.array(next_state)).to(self.device) - action = torch.FloatTensor(np.array(action)).to(self.device) - reward = torch.FloatTensor(reward).unsqueeze(1).to(self.device) - done = torch.FloatTensor(np.float32(done)).unsqueeze(1).to(self.device) - - policy_loss = self.critic(state, self.actor(state)) - policy_loss = -policy_loss.mean() - next_action = self.target_actor(next_state) - target_value = self.target_critic(next_state, next_action.detach()) - expected_value = reward + (1.0 - done) * self.gamma * target_value - expected_value = torch.clamp(expected_value, -np.inf, np.inf) - - value = self.critic(state, action) - value_loss = nn.MSELoss()(value, expected_value.detach()) - - self.actor_optimizer.zero_grad() - policy_loss.backward() - self.actor_optimizer.step() - self.critic_optimizer.zero_grad() - value_loss.backward() - self.critic_optimizer.step() - # soft update - for target_param, param in zip(self.target_critic.parameters(), self.critic.parameters()): - target_param.data.copy_( - target_param.data * (1.0 - self.tau) + - param.data * self.tau - ) - for target_param, param in zip(self.target_actor.parameters(), self.actor.parameters()): - target_param.data.copy_( - target_param.data * (1.0 - self.tau) + - param.data * self.tau - ) - def save_model(self,path): - from pathlib import Path - # create path - Path(path).mkdir(parents=True, exist_ok=True) - torch.save(self.actor.state_dict(), f"{path}/actor_checkpoint.pt") - - def load_model(self,path): - self.actor.load_state_dict(torch.load(f"{path}/actor_checkpoint.pt")) \ No newline at end of file diff --git a/projects/codes/DDPG/env.py b/projects/codes/DDPG/env.py deleted file mode 100644 index 89445cf..0000000 --- a/projects/codes/DDPG/env.py +++ /dev/null @@ -1,56 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -@Author: John -@Email: johnjim0816@gmail.com -@Date: 2020-06-10 15:28:30 -@LastEditor: John -LastEditTime: 2021-09-16 00:52:30 -@Discription: -@Environment: python 3.7.7 -''' -import gym -import numpy as np - -class NormalizedActions(gym.ActionWrapper): - ''' 将action范围重定在[0.1]之间 - ''' - def action(self, action): - low_bound = self.action_space.low - upper_bound = self.action_space.high - action = low_bound + (action + 1.0) * 0.5 * (upper_bound - low_bound) - action = np.clip(action, low_bound, upper_bound) - return action - - def reverse_action(self, action): - low_bound = self.action_space.low - upper_bound = self.action_space.high - action = 2 * (action - low_bound) / (upper_bound - low_bound) - 1 - action = np.clip(action, low_bound, upper_bound) - return action - -class OUNoise(object): - '''Ornstein–Uhlenbeck噪声 - ''' - def __init__(self, action_space, mu=0.0, theta=0.15, max_sigma=0.3, min_sigma=0.3, decay_period=100000): - self.mu = mu # OU噪声的参数 - self.theta = theta # OU噪声的参数 - self.sigma = max_sigma # OU噪声的参数 - self.max_sigma = max_sigma - self.min_sigma = min_sigma - self.decay_period = decay_period - self.n_actions = action_space.shape[0] - self.low = action_space.low - self.high = action_space.high - self.reset() - def reset(self): - self.obs = np.ones(self.n_actions) * self.mu - def evolve_obs(self): - x = self.obs - dx = self.theta * (self.mu - x) + self.sigma * np.random.randn(self.n_actions) - self.obs = x + dx - return self.obs - def get_action(self, action, t=0): - ou_obs = self.evolve_obs() - self.sigma = self.max_sigma - (self.max_sigma - self.min_sigma) * min(1.0, t / self.decay_period) # sigma会逐渐衰减 - return np.clip(action + ou_obs, self.low, self.high) # 动作加上噪声后进行剪切 \ No newline at end of file diff --git a/projects/codes/DDPG/main.py b/projects/codes/DDPG/main.py deleted file mode 100644 index 8da5d29..0000000 --- a/projects/codes/DDPG/main.py +++ /dev/null @@ -1,152 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -@Author: John -@Email: johnjim0816@gmail.com -@Date: 2020-06-11 20:58:21 -@LastEditor: John -LastEditTime: 2022-09-27 15:50:12 -@Discription: -@Environment: python 3.7.7 -''' -import sys,os -curr_path = os.path.dirname(os.path.abspath(__file__)) # current path -parent_path = os.path.dirname(curr_path) # parent path -sys.path.append(parent_path) # add to system path - -import datetime -import gym -import torch -import argparse -import torch.nn as nn -import torch.nn.functional as F -from env import NormalizedActions,OUNoise -from ddpg import DDPG -from common.utils import all_seed -from common.memories import ReplayBufferQue -from common.launcher import Launcher -from envs.register import register_env - -class Actor(nn.Module): - def __init__(self, n_states, n_actions, hidden_dim, init_w=3e-3): - super(Actor, self).__init__() - self.linear1 = nn.Linear(n_states, hidden_dim) - self.linear2 = nn.Linear(hidden_dim, hidden_dim) - self.linear3 = nn.Linear(hidden_dim, n_actions) - - self.linear3.weight.data.uniform_(-init_w, init_w) - self.linear3.bias.data.uniform_(-init_w, init_w) - - def forward(self, x): - x = F.relu(self.linear1(x)) - x = F.relu(self.linear2(x)) - x = torch.tanh(self.linear3(x)) - return x -class Critic(nn.Module): - def __init__(self, n_states, n_actions, hidden_dim, init_w=3e-3): - super(Critic, self).__init__() - - self.linear1 = nn.Linear(n_states + n_actions, hidden_dim) - self.linear2 = nn.Linear(hidden_dim, hidden_dim) - self.linear3 = nn.Linear(hidden_dim, 1) - # 随机初始化为较小的值 - self.linear3.weight.data.uniform_(-init_w, init_w) - self.linear3.bias.data.uniform_(-init_w, init_w) - - def forward(self, state, action): - # 按维数1拼接 - x = torch.cat([state, action], 1) - x = F.relu(self.linear1(x)) - x = F.relu(self.linear2(x)) - x = self.linear3(x) - return x -class Main(Launcher): - def get_args(self): - """ hyperparameters - """ - curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # obtain current time - parser = argparse.ArgumentParser(description="hyperparameters") - parser.add_argument('--algo_name',default='DDPG',type=str,help="name of algorithm") - parser.add_argument('--env_name',default='Pendulum-v1',type=str,help="name of environment") - parser.add_argument('--train_eps',default=300,type=int,help="episodes of training") - parser.add_argument('--test_eps',default=20,type=int,help="episodes of testing") - parser.add_argument('--max_steps',default=100000,type=int,help="steps per episode, much larger value can simulate infinite steps") - parser.add_argument('--gamma',default=0.99,type=float,help="discounted factor") - parser.add_argument('--critic_lr',default=1e-3,type=float,help="learning rate of critic") - parser.add_argument('--actor_lr',default=1e-4,type=float,help="learning rate of actor") - parser.add_argument('--memory_capacity',default=8000,type=int,help="memory capacity") - parser.add_argument('--batch_size',default=128,type=int) - parser.add_argument('--target_update',default=2,type=int) - parser.add_argument('--tau',default=1e-2,type=float) - parser.add_argument('--critic_hidden_dim',default=256,type=int) - parser.add_argument('--actor_hidden_dim',default=256,type=int) - parser.add_argument('--device',default='cpu',type=str,help="cpu or cuda") - parser.add_argument('--seed',default=1,type=int,help="random seed") - parser.add_argument('--show_fig',default=False,type=bool,help="if show figure or not") - parser.add_argument('--save_fig',default=True,type=bool,help="if save figure or not") - args = parser.parse_args() - default_args = {'result_path':f"{curr_path}/outputs/{args.env_name}/{curr_time}/results/", - 'model_path':f"{curr_path}/outputs/{args.env_name}/{curr_time}/models/", - } - args = {**vars(args),**default_args} # type(dict) - return args - - def env_agent_config(self,cfg): - register_env(cfg['env_name']) - env = gym.make(cfg['env_name']) - env = NormalizedActions(env) # decorate with action noise - if cfg['seed'] !=0: # set random seed - all_seed(env,seed=cfg["seed"]) - n_states = env.observation_space.shape[0] - n_actions = env.action_space.shape[0] - print(f"n_states: {n_states}, n_actions: {n_actions}") - cfg.update({"n_states":n_states,"n_actions":n_actions}) # update to cfg paramters - models = {"actor":Actor(n_states,n_actions,hidden_dim=cfg['actor_hidden_dim']),"critic":Critic(n_states,n_actions,hidden_dim=cfg['critic_hidden_dim'])} - memories = {"memory":ReplayBufferQue(cfg['memory_capacity'])} - agent = DDPG(models,memories,cfg) - return env,agent - def train(self,cfg, env, agent): - print('Start training!') - ou_noise = OUNoise(env.action_space) # noise of action - rewards = [] # record rewards for all episodes - for i_ep in range(cfg['train_eps']): - state = env.reset() - ou_noise.reset() - ep_reward = 0 - for i_step in range(cfg['max_steps']): - action = agent.sample_action(state) - action = ou_noise.get_action(action, i_step+1) - next_state, reward, done, _ = env.step(action) - ep_reward += reward - agent.memory.push((state, action, reward, next_state, done)) - agent.update() - state = next_state - if done: - break - if (i_ep+1)%10 == 0: - print(f"Env:{i_ep+1}/{cfg['train_eps']}, Reward:{ep_reward:.2f}") - rewards.append(ep_reward) - print('Finish training!') - return {'rewards':rewards} - - def test(self,cfg, env, agent): - print('Start testing!') - rewards = [] # record rewards for all episodes - for i_ep in range(cfg['test_eps']): - state = env.reset() - ep_reward = 0 - for i_step in range(cfg['max_steps']): - action = agent.predict_action(state) - next_state, reward, done, _ = env.step(action) - ep_reward += reward - state = next_state - if done: - break - rewards.append(ep_reward) - print(f"Episode:{i_ep+1}/{cfg['test_eps']}, Reward:{ep_reward:.1f}") - print('Finish testing!') - return {'rewards':rewards} -if __name__ == "__main__": - main = Main() - main.run() - diff --git a/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/models/actor_checkpoint.pt b/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/models/actor_checkpoint.pt deleted file mode 100644 index e65e7ca..0000000 Binary files a/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/models/actor_checkpoint.pt and /dev/null differ diff --git a/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/results/params.json b/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/results/params.json deleted file mode 100644 index c3825cf..0000000 --- a/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/results/params.json +++ /dev/null @@ -1,25 +0,0 @@ -{ - "algo_name": "DDPG", - "env_name": "Pendulum-v1", - "train_eps": 300, - "test_eps": 20, - "max_steps": 100000, - "gamma": 0.99, - "critic_lr": 0.001, - "actor_lr": 0.0001, - "memory_capacity": 8000, - "batch_size": 128, - "target_update": 2, - "tau": 0.01, - "critic_hidden_dim": 256, - "actor_hidden_dim": 256, - "device": "cpu", - "seed": 1, - "show_fig": false, - "save_fig": true, - "result_path": "/Users/jj/Desktop/rl-tutorials/codes/DDPG/outputs/Pendulum-v1/20220927-155053/results/", - "model_path": "/Users/jj/Desktop/rl-tutorials/codes/DDPG/outputs/Pendulum-v1/20220927-155053/models/", - "n_states": 3, - "n_actions": 1, - "training_time": 358.8142900466919 -} \ No newline at end of file diff --git a/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/results/testing_curve.png b/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/results/testing_curve.png deleted file mode 100644 index 44e53e2..0000000 Binary files a/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/results/testing_curve.png and /dev/null differ diff --git a/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/results/testing_results.csv b/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/results/testing_results.csv deleted file mode 100644 index 590c141..0000000 --- a/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/results/testing_results.csv +++ /dev/null @@ -1,21 +0,0 @@ -rewards --116.045416124376 --126.18022935469217 --231.46338228458293 --246.40481094689758 --304.69493818839186 --124.39609191913091 --1.060003582878406 --114.19659653048288 --348.9745708742037 --116.10811133324769 --117.20146333694844 --118.66206784602966 --235.17836229762355 --356.14054913290624 --118.38579118156366 --351.9415915140771 --114.50877866098972 --124.775484599685 --226.47062962476875 --121.48872909193936 diff --git a/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/results/training_curve.png b/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/results/training_curve.png deleted file mode 100644 index b0b95fe..0000000 Binary files a/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/results/training_curve.png and /dev/null differ diff --git a/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/results/training_results.csv b/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/results/training_results.csv deleted file mode 100644 index 2fa54ec..0000000 --- a/projects/codes/DDPG/outputs/Pendulum-v1/20220927-155053/results/training_results.csv +++ /dev/null @@ -1,301 +0,0 @@ -rewards --1557.8518596631177 --1354.7599369723537 --1375.5732016629706 --1493.8609739040871 --1426.7116204537845 --1235.7920755027762 --1339.1647620443073 --1544.2379906560486 --1539.6232758780877 --1549.5690058648204 --1446.9193195793853 --1520.2666688767558 --1525.0116707122581 --1379.136573640111 --1532.702831768523 --1484.7552963941637 --1359.6699201737677 --1349.6805649166854 --1510.869999766432 --1515.8398785434708 --1447.4648656578254 --1537.3822077872178 --1249.6517039877456 --1350.0302666965736 --1529.4363372505607 --1320.28204807604 --1502.9248141320654 --1545.4861772197075 --1579.928789692619 --1413.296070504152 --1242.4673258663781 --1403.8672028946078 --1452.7199002523635 --871.6071114009982 --1324.1789316121412 --1313.3348146041249 --1059.8722927418046 --1054.232673559123 --973.8956270782459 --972.9936641224186 --972.9477399905655 --947.0613443333731 --737.3866328989184 --958.6068164634295 --739.6973395350705 --886.8383108399455 --775.1430379821574 --937.3115016337417 --700.875502951337 --829.9396339144109 --271.1629773396998 --493.5460684734584 --485.9321719313203 --858.3735607086766 --1145.3440084994113 --1121.1338201339777 --1191.5640831332366 --1350.0425368846784 --249.25438665107953 --727.9051714734406 --368.5579316240395 --392.0611344939354 --955.3231703741553 --488.27956192035265 --362.2734695759137 --949.5440839122496 --496.8460016912189 --726.6871514929877 --424.48641462866266 --954.7075428204689 --608.9650086409792 --848.6059768900151 --866.7052398755033 --856.9846415044439 --751.0342976129083 --749.5118249469103 --509.882299129811 --506.56154097018043 --906.0964475820368 --1318.3941416286855 --1422.2017011876615 --1523.1661091894277 --1209.2850593747999 --1415.0972750475833 --1533.2263827605834 --1405.8345530072663 --1244.3384723384913 --1237.4704845061992 --949.3394417935086 --981.1855396112669 --1241.224568444032 --1033.118364799829 --1017.2403725619487 --981.9727804516916 --853.1877724775591 --869.0652369861646 --1069.8265343327998 --371.73173813891884 --735.5887912713665 --1262.050240428957 --1242.985056062197 --1191.6867713427482 --1328.5323118458034 --1015.5308653784714 --895.3066515461381 --994.1114862316568 --761.4710321387583 --717.6979056272868 --782.302146467708 --640.4913147345328 --725.6469893076355 --497.5346232085584 --1027.1192149202325 --950.0117149822681 --956.1343737377374 --708.9489626669097 --964.5003064113283 --611.9111516886613 --612.3182791021098 --1100.0047939174613 --984.9262458612923 --858.7106075590494 --842.305917848386 --745.9043991922597 --741.2168858394704 --1143.0750387284456 --755.5257242325362 --745.8440029056219 --387.8717950334138 --764.6628701051523 --486.7967495537958 --485.13357559164814 --313.5415216767419 --611.3450529954782 --611.1570544377465 --507.6456747676814 --615.2032627013064 --242.37988821149764 --603.85498620892 --352.2672241055367 --155.99874664988383 --615.4003063516313 --384.9811293551548 --498.80727354456315 --407.6898591217813 --1213.6383844696395 --1122.2425748913884 --592.4819308883913 --478.2046833075051 --891.0254788311132 --482.40204115385 --339.34676196677407 --582.9985110154428 --213.38243627478826 --928.8434951613825 --1545.5433749195483 --1179.5016285049896 --1211.9549773601925 --1396.8082561792166 --1318.073128824395 --597.3837225413702 --564.7793352410449 --723.744223659601 --653.0145534050461 --847.6138123247009 --385.62784320332867 --245.25250602651928 --117.55094416757835 --864.0064774069044 --124.30221387458867 --244.4014050243669 --1148.861754008653 --914.4047868424254 --765.9394408203351 --124.05114610943177 --605.7641303826842 --616.3595829453579 --375.5024692962698 --253.51874076866997 --240.08405245866714 --503.96565579077225 --606.7646526173963 --502.6512112729435 --746.404013238678 --718.8658110051653 --125.65808359856703 --247.62256797883364 --363.69852213666803 --249.21801061415547 --491.7724416523124 --235.37050442527357 --609.6026403583944 --236.05731608228092 --381.19853850450454 --298.7683201867404 --127.64145601534942 --233.4300138495176 --129.11243486763516 --390.0092951263507 --1000.7729892969854 --249.60445310459787 --253.02347910759622 --129.04269174391223 --360.6321251486308 --377.26297602576534 --124.98466986009481 --245.47913567739212 --127.0885254550411 --118.11013006825459 --128.8682755001942 --497.3015586531096 --340.77352433313484 --514.4945799737978 --503.24077308842783 --627.9068157464455 --511.39396524392146 --763.8866112068075 --741.7885082408757 --617.4945380476306 --950.3176437519387 --643.4791402436576 --511.9377874351982 --573.6219349516633 --564.1297823875693 --242.06399233336583 --496.4020380325518 --360.56387982880364 --495.4590728336022 --503.7263345016764 --122.47964616802327 --254.16543926263168 --614.5335268729743 --234.3718017676852 --301.27514663062874 --387.64758894986204 --368.74492411716415 --364.43559131093593 --160.6845848115533 --504.1948947975429 --246.51676032967683 --251.5732500220603 --600.1463819723879 --247.17476928471288 --381.924164337607 --377.4773226068174 --378.511830774651 --126.69199895843033 --365.0506645811703 --130.45052114802874 --374.37400288581813 --502.37678159638887 --374.43552658473055 --241.157211525502 --388.9597456642503 --249.4412385534861 --114.71395078439846 --864.6882327286056 --626.8144095971478 --732.9226896140248 --368.24767905020394 --369.7425524469132 --398.07832598184626 --906.7113918582257 --252.2343258180765 --370.4258473086036 --736.0203154396909 --609.4605173515027 --661.1255920773486 --489.9605291008584 --364.1671188501402 --644.4029089587781 --477.9510457677364 --128.78294672880136 --373.74382001694886 --380.69931133982936 --372.60275628381805 --743.0410655515724 --597.558847789258 --387.94245652694394 --725.3939448944484 --409.1301313430852 --491.8442467896486 --123.0638156839621 --377.9292326597324 --489.27209762667974 --255.63227821371257 --379.5885382060625 --370.2312967024669 --250.94061817008688 --131.2125308195906 --600.3312016651868 --130.84444772735733 --312.6287688438562 --382.4144610039701 --259.03558003697265 --224.92206667096863 --376.81390821359685 --382.39993489751646 --380.25599578593636 --610.1016672243638 diff --git a/projects/codes/DQN/Test_CartPole-v1_DQN_20221031-001343/config.yaml b/projects/codes/DQN/Test_CartPole-v1_DQN_20221031-001343/config.yaml deleted file mode 100644 index 5e3ad4e..0000000 --- a/projects/codes/DQN/Test_CartPole-v1_DQN_20221031-001343/config.yaml +++ /dev/null @@ -1,25 +0,0 @@ -general_cfg: - algo_name: DQN - device: cuda - env_name: CartPole-v1 - eval_eps: 10 - eval_per_episode: 5 - load_checkpoint: true - load_path: Train_CartPole-v1_DQN_20221031-001201 - max_steps: 200 - mode: test - save_fig: true - seed: 0 - show_fig: false - test_eps: 10 - train_eps: 100 -algo_cfg: - batch_size: 64 - buffer_size: 100000 - epsilon_decay: 500 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - hidden_dim: 256 - lr: 0.0001 - target_update: 4 diff --git a/projects/codes/DQN/Test_CartPole-v1_DQN_20221031-001343/logs/log.txt b/projects/codes/DQN/Test_CartPole-v1_DQN_20221031-001343/logs/log.txt deleted file mode 100644 index 44f28cb..0000000 --- a/projects/codes/DQN/Test_CartPole-v1_DQN_20221031-001343/logs/log.txt +++ /dev/null @@ -1,14 +0,0 @@ -2022-10-31 00:13:43 - r - INFO: - n_states: 4, n_actions: 2 -2022-10-31 00:13:44 - r - INFO: - Start testing! -2022-10-31 00:13:44 - r - INFO: - Env: CartPole-v1, Algorithm: DQN, Device: cuda -2022-10-31 00:13:45 - r - INFO: - Episode: 1/10, Reward: 200.0, Step: 200 -2022-10-31 00:13:45 - r - INFO: - Episode: 2/10, Reward: 200.0, Step: 200 -2022-10-31 00:13:45 - r - INFO: - Episode: 3/10, Reward: 200.0, Step: 200 -2022-10-31 00:13:45 - r - INFO: - Episode: 4/10, Reward: 200.0, Step: 200 -2022-10-31 00:13:45 - r - INFO: - Episode: 5/10, Reward: 200.0, Step: 200 -2022-10-31 00:13:45 - r - INFO: - Episode: 6/10, Reward: 200.0, Step: 200 -2022-10-31 00:13:45 - r - INFO: - Episode: 7/10, Reward: 200.0, Step: 200 -2022-10-31 00:13:45 - r - INFO: - Episode: 8/10, Reward: 200.0, Step: 200 -2022-10-31 00:13:45 - r - INFO: - Episode: 9/10, Reward: 200.0, Step: 200 -2022-10-31 00:13:45 - r - INFO: - Episode: 10/10, Reward: 200.0, Step: 200 -2022-10-31 00:13:45 - r - INFO: - Finish testing! diff --git a/projects/codes/DQN/Test_CartPole-v1_DQN_20221031-001343/models/checkpoint.pt b/projects/codes/DQN/Test_CartPole-v1_DQN_20221031-001343/models/checkpoint.pt deleted file mode 100644 index 722eb69..0000000 Binary files a/projects/codes/DQN/Test_CartPole-v1_DQN_20221031-001343/models/checkpoint.pt and /dev/null differ diff --git a/projects/codes/DQN/Test_CartPole-v1_DQN_20221031-001343/results/learning_curve.png b/projects/codes/DQN/Test_CartPole-v1_DQN_20221031-001343/results/learning_curve.png deleted file mode 100644 index 046009a..0000000 Binary files a/projects/codes/DQN/Test_CartPole-v1_DQN_20221031-001343/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/DQN/Test_CartPole-v1_DQN_20221031-001343/results/res.csv b/projects/codes/DQN/Test_CartPole-v1_DQN_20221031-001343/results/res.csv deleted file mode 100644 index cbbcf2e..0000000 --- a/projects/codes/DQN/Test_CartPole-v1_DQN_20221031-001343/results/res.csv +++ /dev/null @@ -1,11 +0,0 @@ -episodes,rewards,steps -0,200.0,200 -1,200.0,200 -2,200.0,200 -3,200.0,200 -4,200.0,200 -5,200.0,200 -6,200.0,200 -7,200.0,200 -8,200.0,200 -9,200.0,200 diff --git a/projects/codes/DQN/Train_Acrobot-v1_DQN_20221026-094645/config.yaml b/projects/codes/DQN/Train_Acrobot-v1_DQN_20221026-094645/config.yaml deleted file mode 100644 index 7416aec..0000000 --- a/projects/codes/DQN/Train_Acrobot-v1_DQN_20221026-094645/config.yaml +++ /dev/null @@ -1,23 +0,0 @@ -general_cfg: - algo_name: DQN - device: cuda - env_name: Acrobot-v1 - load_checkpoint: false - load_path: Train_CartPole-v1_DQN_20221026-054757 - max_steps: 100000 - mode: train - save_fig: true - seed: 1 - show_fig: false - test_eps: 10 - train_eps: 100 -algo_cfg: - batch_size: 128 - buffer_size: 200000 - epsilon_decay: 500 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - hidden_dim: 256 - lr: 0.002 - target_update: 4 diff --git a/projects/codes/DQN/Train_Acrobot-v1_DQN_20221026-094645/logs/log.txt b/projects/codes/DQN/Train_Acrobot-v1_DQN_20221026-094645/logs/log.txt deleted file mode 100644 index e745c8c..0000000 --- a/projects/codes/DQN/Train_Acrobot-v1_DQN_20221026-094645/logs/log.txt +++ /dev/null @@ -1,104 +0,0 @@ -2022-10-26 09:46:45 - r - INFO: - n_states: 6, n_actions: 3 -2022-10-26 09:46:48 - r - INFO: - Start training! -2022-10-26 09:46:48 - r - INFO: - Env: Acrobot-v1, Algorithm: DQN, Device: cuda -2022-10-26 09:46:50 - r - INFO: - Episode: 1/100, Reward: -861.00: Epislon: 0.178 -2022-10-26 09:46:50 - r - INFO: - Episode: 2/100, Reward: -252.00: Epislon: 0.111 -2022-10-26 09:46:50 - r - INFO: - Episode: 3/100, Reward: -196.00: Epislon: 0.078 -2022-10-26 09:46:51 - r - INFO: - Episode: 4/100, Reward: -390.00: Epislon: 0.041 -2022-10-26 09:46:52 - r - INFO: - Episode: 5/100, Reward: -371.00: Epislon: 0.025 -2022-10-26 09:46:52 - r - INFO: - Episode: 6/100, Reward: -237.00: Epislon: 0.019 -2022-10-26 09:46:52 - r - INFO: - Episode: 7/100, Reward: -227.00: Epislon: 0.016 -2022-10-26 09:46:53 - r - INFO: - Episode: 8/100, Reward: -228.00: Epislon: 0.014 -2022-10-26 09:46:53 - r - INFO: - Episode: 9/100, Reward: -305.00: Epislon: 0.012 -2022-10-26 09:46:54 - r - INFO: - Episode: 10/100, Reward: -234.00: Epislon: 0.011 -2022-10-26 09:46:54 - r - INFO: - Episode: 11/100, Reward: -204.00: Epislon: 0.011 -2022-10-26 09:46:55 - r - INFO: - Episode: 12/100, Reward: -277.00: Epislon: 0.010 -2022-10-26 09:46:55 - r - INFO: - Episode: 13/100, Reward: -148.00: Epislon: 0.010 -2022-10-26 09:46:56 - r - INFO: - Episode: 14/100, Reward: -372.00: Epislon: 0.010 -2022-10-26 09:46:56 - r - INFO: - Episode: 15/100, Reward: -273.00: Epislon: 0.010 -2022-10-26 09:46:56 - r - INFO: - Episode: 16/100, Reward: -105.00: Epislon: 0.010 -2022-10-26 09:46:56 - r - INFO: - Episode: 17/100, Reward: -79.00: Epislon: 0.010 -2022-10-26 09:46:57 - r - INFO: - Episode: 18/100, Reward: -112.00: Epislon: 0.010 -2022-10-26 09:46:57 - r - INFO: - Episode: 19/100, Reward: -276.00: Epislon: 0.010 -2022-10-26 09:46:57 - r - INFO: - Episode: 20/100, Reward: -148.00: Epislon: 0.010 -2022-10-26 09:46:58 - r - INFO: - Episode: 21/100, Reward: -201.00: Epislon: 0.010 -2022-10-26 09:46:58 - r - INFO: - Episode: 22/100, Reward: -173.00: Epislon: 0.010 -2022-10-26 09:46:58 - r - INFO: - Episode: 23/100, Reward: -226.00: Epislon: 0.010 -2022-10-26 09:46:59 - r - INFO: - Episode: 24/100, Reward: -154.00: Epislon: 0.010 -2022-10-26 09:46:59 - r - INFO: - Episode: 25/100, Reward: -269.00: Epislon: 0.010 -2022-10-26 09:46:59 - r - INFO: - Episode: 26/100, Reward: -191.00: Epislon: 0.010 -2022-10-26 09:47:00 - r - INFO: - Episode: 27/100, Reward: -177.00: Epislon: 0.010 -2022-10-26 09:47:00 - r - INFO: - Episode: 28/100, Reward: -209.00: Epislon: 0.010 -2022-10-26 09:47:00 - r - INFO: - Episode: 29/100, Reward: -116.00: Epislon: 0.010 -2022-10-26 09:47:00 - r - INFO: - Episode: 30/100, Reward: -117.00: Epislon: 0.010 -2022-10-26 09:47:01 - r - INFO: - Episode: 31/100, Reward: -121.00: Epislon: 0.010 -2022-10-26 09:47:01 - r - INFO: - Episode: 32/100, Reward: -208.00: Epislon: 0.010 -2022-10-26 09:47:01 - r - INFO: - Episode: 33/100, Reward: -147.00: Epislon: 0.010 -2022-10-26 09:47:02 - r - INFO: - Episode: 34/100, Reward: -104.00: Epislon: 0.010 -2022-10-26 09:47:02 - r - INFO: - Episode: 35/100, Reward: -161.00: Epislon: 0.010 -2022-10-26 09:47:02 - r - INFO: - Episode: 36/100, Reward: -144.00: Epislon: 0.010 -2022-10-26 09:47:02 - r - INFO: - Episode: 37/100, Reward: -131.00: Epislon: 0.010 -2022-10-26 09:47:03 - r - INFO: - Episode: 38/100, Reward: -226.00: Epislon: 0.010 -2022-10-26 09:47:03 - r - INFO: - Episode: 39/100, Reward: -117.00: Epislon: 0.010 -2022-10-26 09:47:03 - r - INFO: - Episode: 40/100, Reward: -344.00: Epislon: 0.010 -2022-10-26 09:47:04 - r - INFO: - Episode: 41/100, Reward: -123.00: Epislon: 0.010 -2022-10-26 09:47:04 - r - INFO: - Episode: 42/100, Reward: -232.00: Epislon: 0.010 -2022-10-26 09:47:04 - r - INFO: - Episode: 43/100, Reward: -190.00: Epislon: 0.010 -2022-10-26 09:47:05 - r - INFO: - Episode: 44/100, Reward: -176.00: Epislon: 0.010 -2022-10-26 09:47:05 - r - INFO: - Episode: 45/100, Reward: -139.00: Epislon: 0.010 -2022-10-26 09:47:06 - r - INFO: - Episode: 46/100, Reward: -410.00: Epislon: 0.010 -2022-10-26 09:47:06 - r - INFO: - Episode: 47/100, Reward: -115.00: Epislon: 0.010 -2022-10-26 09:47:06 - r - INFO: - Episode: 48/100, Reward: -118.00: Epislon: 0.010 -2022-10-26 09:47:06 - r - INFO: - Episode: 49/100, Reward: -113.00: Epislon: 0.010 -2022-10-26 09:47:07 - r - INFO: - Episode: 50/100, Reward: -355.00: Epislon: 0.010 -2022-10-26 09:47:07 - r - INFO: - Episode: 51/100, Reward: -110.00: Epislon: 0.010 -2022-10-26 09:47:07 - r - INFO: - Episode: 52/100, Reward: -148.00: Epislon: 0.010 -2022-10-26 09:47:08 - r - INFO: - Episode: 53/100, Reward: -135.00: Epislon: 0.010 -2022-10-26 09:47:08 - r - INFO: - Episode: 54/100, Reward: -220.00: Epislon: 0.010 -2022-10-26 09:47:08 - r - INFO: - Episode: 55/100, Reward: -157.00: Epislon: 0.010 -2022-10-26 09:47:09 - r - INFO: - Episode: 56/100, Reward: -130.00: Epislon: 0.010 -2022-10-26 09:47:09 - r - INFO: - Episode: 57/100, Reward: -150.00: Epislon: 0.010 -2022-10-26 09:47:09 - r - INFO: - Episode: 58/100, Reward: -254.00: Epislon: 0.010 -2022-10-26 09:47:10 - r - INFO: - Episode: 59/100, Reward: -148.00: Epislon: 0.010 -2022-10-26 09:47:10 - r - INFO: - Episode: 60/100, Reward: -108.00: Epislon: 0.010 -2022-10-26 09:47:10 - r - INFO: - Episode: 61/100, Reward: -152.00: Epislon: 0.010 -2022-10-26 09:47:10 - r - INFO: - Episode: 62/100, Reward: -107.00: Epislon: 0.010 -2022-10-26 09:47:10 - r - INFO: - Episode: 63/100, Reward: -110.00: Epislon: 0.010 -2022-10-26 09:47:11 - r - INFO: - Episode: 64/100, Reward: -266.00: Epislon: 0.010 -2022-10-26 09:47:11 - r - INFO: - Episode: 65/100, Reward: -344.00: Epislon: 0.010 -2022-10-26 09:47:12 - r - INFO: - Episode: 66/100, Reward: -93.00: Epislon: 0.010 -2022-10-26 09:47:12 - r - INFO: - Episode: 67/100, Reward: -113.00: Epislon: 0.010 -2022-10-26 09:47:12 - r - INFO: - Episode: 68/100, Reward: -191.00: Epislon: 0.010 -2022-10-26 09:47:12 - r - INFO: - Episode: 69/100, Reward: -102.00: Epislon: 0.010 -2022-10-26 09:47:13 - r - INFO: - Episode: 70/100, Reward: -187.00: Epislon: 0.010 -2022-10-26 09:47:13 - r - INFO: - Episode: 71/100, Reward: -158.00: Epislon: 0.010 -2022-10-26 09:47:13 - r - INFO: - Episode: 72/100, Reward: -166.00: Epislon: 0.010 -2022-10-26 09:47:14 - r - INFO: - Episode: 73/100, Reward: -202.00: Epislon: 0.010 -2022-10-26 09:47:14 - r - INFO: - Episode: 74/100, Reward: -179.00: Epislon: 0.010 -2022-10-26 09:47:14 - r - INFO: - Episode: 75/100, Reward: -150.00: Epislon: 0.010 -2022-10-26 09:47:14 - r - INFO: - Episode: 76/100, Reward: -170.00: Epislon: 0.010 -2022-10-26 09:47:15 - r - INFO: - Episode: 77/100, Reward: -149.00: Epislon: 0.010 -2022-10-26 09:47:15 - r - INFO: - Episode: 78/100, Reward: -119.00: Epislon: 0.010 -2022-10-26 09:47:15 - r - INFO: - Episode: 79/100, Reward: -115.00: Epislon: 0.010 -2022-10-26 09:47:15 - r - INFO: - Episode: 80/100, Reward: -97.00: Epislon: 0.010 -2022-10-26 09:47:16 - r - INFO: - Episode: 81/100, Reward: -153.00: Epislon: 0.010 -2022-10-26 09:47:16 - r - INFO: - Episode: 82/100, Reward: -97.00: Epislon: 0.010 -2022-10-26 09:47:16 - r - INFO: - Episode: 83/100, Reward: -211.00: Epislon: 0.010 -2022-10-26 09:47:16 - r - INFO: - Episode: 84/100, Reward: -195.00: Epislon: 0.010 -2022-10-26 09:47:17 - r - INFO: - Episode: 85/100, Reward: -125.00: Epislon: 0.010 -2022-10-26 09:47:17 - r - INFO: - Episode: 86/100, Reward: -155.00: Epislon: 0.010 -2022-10-26 09:47:17 - r - INFO: - Episode: 87/100, Reward: -151.00: Epislon: 0.010 -2022-10-26 09:47:18 - r - INFO: - Episode: 88/100, Reward: -194.00: Epislon: 0.010 -2022-10-26 09:47:18 - r - INFO: - Episode: 89/100, Reward: -188.00: Epislon: 0.010 -2022-10-26 09:47:18 - r - INFO: - Episode: 90/100, Reward: -195.00: Epislon: 0.010 -2022-10-26 09:47:19 - r - INFO: - Episode: 91/100, Reward: -141.00: Epislon: 0.010 -2022-10-26 09:47:19 - r - INFO: - Episode: 92/100, Reward: -132.00: Epislon: 0.010 -2022-10-26 09:47:19 - r - INFO: - Episode: 93/100, Reward: -127.00: Epislon: 0.010 -2022-10-26 09:47:19 - r - INFO: - Episode: 94/100, Reward: -195.00: Epislon: 0.010 -2022-10-26 09:47:20 - r - INFO: - Episode: 95/100, Reward: -152.00: Epislon: 0.010 -2022-10-26 09:47:20 - r - INFO: - Episode: 96/100, Reward: -145.00: Epislon: 0.010 -2022-10-26 09:47:20 - r - INFO: - Episode: 97/100, Reward: -123.00: Epislon: 0.010 -2022-10-26 09:47:20 - r - INFO: - Episode: 98/100, Reward: -176.00: Epislon: 0.010 -2022-10-26 09:47:21 - r - INFO: - Episode: 99/100, Reward: -180.00: Epislon: 0.010 -2022-10-26 09:47:21 - r - INFO: - Episode: 100/100, Reward: -124.00: Epislon: 0.010 -2022-10-26 09:47:21 - r - INFO: - Finish training! diff --git a/projects/codes/DQN/Train_Acrobot-v1_DQN_20221026-094645/models/checkpoint.pt b/projects/codes/DQN/Train_Acrobot-v1_DQN_20221026-094645/models/checkpoint.pt deleted file mode 100644 index 5448aca..0000000 Binary files a/projects/codes/DQN/Train_Acrobot-v1_DQN_20221026-094645/models/checkpoint.pt and /dev/null differ diff --git a/projects/codes/DQN/Train_Acrobot-v1_DQN_20221026-094645/results/learning_curve.png b/projects/codes/DQN/Train_Acrobot-v1_DQN_20221026-094645/results/learning_curve.png deleted file mode 100644 index 7f1054d..0000000 Binary files a/projects/codes/DQN/Train_Acrobot-v1_DQN_20221026-094645/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/DQN/Train_Acrobot-v1_DQN_20221026-094645/results/res.csv b/projects/codes/DQN/Train_Acrobot-v1_DQN_20221026-094645/results/res.csv deleted file mode 100644 index 1758be2..0000000 --- a/projects/codes/DQN/Train_Acrobot-v1_DQN_20221026-094645/results/res.csv +++ /dev/null @@ -1,101 +0,0 @@ -episodes,rewards,steps -0,-861.0,862 -1,-252.0,253 -2,-196.0,197 -3,-390.0,391 -4,-371.0,372 -5,-237.0,238 -6,-227.0,228 -7,-228.0,229 -8,-305.0,306 -9,-234.0,235 -10,-204.0,205 -11,-277.0,278 -12,-148.0,149 -13,-372.0,373 -14,-273.0,274 -15,-105.0,106 -16,-79.0,80 -17,-112.0,113 -18,-276.0,277 -19,-148.0,149 -20,-201.0,202 -21,-173.0,174 -22,-226.0,227 -23,-154.0,155 -24,-269.0,270 -25,-191.0,192 -26,-177.0,178 -27,-209.0,210 -28,-116.0,117 -29,-117.0,118 -30,-121.0,122 -31,-208.0,209 -32,-147.0,148 -33,-104.0,105 -34,-161.0,162 -35,-144.0,145 -36,-131.0,132 -37,-226.0,227 -38,-117.0,118 -39,-344.0,345 -40,-123.0,124 -41,-232.0,233 -42,-190.0,191 -43,-176.0,177 -44,-139.0,140 -45,-410.0,411 -46,-115.0,116 -47,-118.0,119 -48,-113.0,114 -49,-355.0,356 -50,-110.0,111 -51,-148.0,149 -52,-135.0,136 -53,-220.0,221 -54,-157.0,158 -55,-130.0,131 -56,-150.0,151 -57,-254.0,255 -58,-148.0,149 -59,-108.0,109 -60,-152.0,153 -61,-107.0,108 -62,-110.0,111 -63,-266.0,267 -64,-344.0,345 -65,-93.0,94 -66,-113.0,114 -67,-191.0,192 -68,-102.0,103 -69,-187.0,188 -70,-158.0,159 -71,-166.0,167 -72,-202.0,203 -73,-179.0,180 -74,-150.0,151 -75,-170.0,171 -76,-149.0,150 -77,-119.0,120 -78,-115.0,116 -79,-97.0,98 -80,-153.0,154 -81,-97.0,98 -82,-211.0,212 -83,-195.0,196 -84,-125.0,126 -85,-155.0,156 -86,-151.0,152 -87,-194.0,195 -88,-188.0,189 -89,-195.0,196 -90,-141.0,142 -91,-132.0,133 -92,-127.0,128 -93,-195.0,196 -94,-152.0,153 -95,-145.0,146 -96,-123.0,124 -97,-176.0,177 -98,-180.0,181 -99,-124.0,125 diff --git a/projects/codes/DQN/Train_CartPole-v1_DQN_20221031-001201/config.yaml b/projects/codes/DQN/Train_CartPole-v1_DQN_20221031-001201/config.yaml deleted file mode 100644 index 33950ad..0000000 --- a/projects/codes/DQN/Train_CartPole-v1_DQN_20221031-001201/config.yaml +++ /dev/null @@ -1,25 +0,0 @@ -general_cfg: - algo_name: DQN - device: cuda - env_name: CartPole-v1 - eval_eps: 10 - eval_per_episode: 5 - load_checkpoint: false - load_path: tasks - max_steps: 200 - mode: train - save_fig: true - seed: 1 - show_fig: false - test_eps: 10 - train_eps: 100 -algo_cfg: - batch_size: 64 - buffer_size: 100000 - epsilon_decay: 500 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - hidden_dim: 256 - lr: 0.0001 - target_update: 800 diff --git a/projects/codes/DQN/Train_CartPole-v1_DQN_20221031-001201/logs/log.txt b/projects/codes/DQN/Train_CartPole-v1_DQN_20221031-001201/logs/log.txt deleted file mode 100644 index 5b084be..0000000 --- a/projects/codes/DQN/Train_CartPole-v1_DQN_20221031-001201/logs/log.txt +++ /dev/null @@ -1,116 +0,0 @@ -2022-10-31 00:12:01 - r - INFO: - n_states: 4, n_actions: 2 -2022-10-31 00:12:01 - r - INFO: - Start training! -2022-10-31 00:12:01 - r - INFO: - Env: CartPole-v1, Algorithm: DQN, Device: cuda -2022-10-31 00:12:04 - r - INFO: - Episode: 1/100, Reward: 18.0, Step: 18 -2022-10-31 00:12:04 - r - INFO: - Episode: 2/100, Reward: 35.0, Step: 35 -2022-10-31 00:12:04 - r - INFO: - Episode: 3/100, Reward: 13.0, Step: 13 -2022-10-31 00:12:04 - r - INFO: - Episode: 4/100, Reward: 32.0, Step: 32 -2022-10-31 00:12:04 - r - INFO: - Episode: 5/100, Reward: 16.0, Step: 16 -2022-10-31 00:12:04 - r - INFO: - Current episode 5 has the best eval reward: 15.30 -2022-10-31 00:12:04 - r - INFO: - Episode: 6/100, Reward: 12.0, Step: 12 -2022-10-31 00:12:04 - r - INFO: - Episode: 7/100, Reward: 13.0, Step: 13 -2022-10-31 00:12:04 - r - INFO: - Episode: 8/100, Reward: 15.0, Step: 15 -2022-10-31 00:12:04 - r - INFO: - Episode: 9/100, Reward: 11.0, Step: 11 -2022-10-31 00:12:04 - r - INFO: - Episode: 10/100, Reward: 15.0, Step: 15 -2022-10-31 00:12:04 - r - INFO: - Episode: 11/100, Reward: 9.0, Step: 9 -2022-10-31 00:12:04 - r - INFO: - Episode: 12/100, Reward: 13.0, Step: 13 -2022-10-31 00:12:04 - r - INFO: - Episode: 13/100, Reward: 13.0, Step: 13 -2022-10-31 00:12:04 - r - INFO: - Episode: 14/100, Reward: 10.0, Step: 10 -2022-10-31 00:12:04 - r - INFO: - Episode: 15/100, Reward: 9.0, Step: 9 -2022-10-31 00:12:04 - r - INFO: - Episode: 16/100, Reward: 24.0, Step: 24 -2022-10-31 00:12:04 - r - INFO: - Episode: 17/100, Reward: 8.0, Step: 8 -2022-10-31 00:12:04 - r - INFO: - Episode: 18/100, Reward: 10.0, Step: 10 -2022-10-31 00:12:04 - r - INFO: - Episode: 19/100, Reward: 11.0, Step: 11 -2022-10-31 00:12:04 - r - INFO: - Episode: 20/100, Reward: 13.0, Step: 13 -2022-10-31 00:12:04 - r - INFO: - Episode: 21/100, Reward: 12.0, Step: 12 -2022-10-31 00:12:04 - r - INFO: - Episode: 22/100, Reward: 11.0, Step: 11 -2022-10-31 00:12:04 - r - INFO: - Episode: 23/100, Reward: 9.0, Step: 9 -2022-10-31 00:12:04 - r - INFO: - Episode: 24/100, Reward: 21.0, Step: 21 -2022-10-31 00:12:05 - r - INFO: - Episode: 25/100, Reward: 14.0, Step: 14 -2022-10-31 00:12:05 - r - INFO: - Episode: 26/100, Reward: 12.0, Step: 12 -2022-10-31 00:12:05 - r - INFO: - Episode: 27/100, Reward: 9.0, Step: 9 -2022-10-31 00:12:05 - r - INFO: - Episode: 28/100, Reward: 11.0, Step: 11 -2022-10-31 00:12:05 - r - INFO: - Episode: 29/100, Reward: 12.0, Step: 12 -2022-10-31 00:12:05 - r - INFO: - Episode: 30/100, Reward: 13.0, Step: 13 -2022-10-31 00:12:05 - r - INFO: - Episode: 31/100, Reward: 10.0, Step: 10 -2022-10-31 00:12:05 - r - INFO: - Episode: 32/100, Reward: 13.0, Step: 13 -2022-10-31 00:12:05 - r - INFO: - Episode: 33/100, Reward: 18.0, Step: 18 -2022-10-31 00:12:05 - r - INFO: - Episode: 34/100, Reward: 9.0, Step: 9 -2022-10-31 00:12:05 - r - INFO: - Episode: 35/100, Reward: 10.0, Step: 10 -2022-10-31 00:12:05 - r - INFO: - Episode: 36/100, Reward: 9.0, Step: 9 -2022-10-31 00:12:05 - r - INFO: - Episode: 37/100, Reward: 10.0, Step: 10 -2022-10-31 00:12:05 - r - INFO: - Episode: 38/100, Reward: 10.0, Step: 10 -2022-10-31 00:12:05 - r - INFO: - Episode: 39/100, Reward: 10.0, Step: 10 -2022-10-31 00:12:05 - r - INFO: - Episode: 40/100, Reward: 8.0, Step: 8 -2022-10-31 00:12:06 - r - INFO: - Episode: 41/100, Reward: 9.0, Step: 9 -2022-10-31 00:12:06 - r - INFO: - Episode: 42/100, Reward: 9.0, Step: 9 -2022-10-31 00:12:06 - r - INFO: - Episode: 43/100, Reward: 20.0, Step: 20 -2022-10-31 00:12:06 - r - INFO: - Episode: 44/100, Reward: 16.0, Step: 16 -2022-10-31 00:12:06 - r - INFO: - Episode: 45/100, Reward: 17.0, Step: 17 -2022-10-31 00:12:06 - r - INFO: - Current episode 45 has the best eval reward: 17.50 -2022-10-31 00:12:06 - r - INFO: - Episode: 46/100, Reward: 17.0, Step: 17 -2022-10-31 00:12:06 - r - INFO: - Episode: 47/100, Reward: 17.0, Step: 17 -2022-10-31 00:12:06 - r - INFO: - Episode: 48/100, Reward: 18.0, Step: 18 -2022-10-31 00:12:06 - r - INFO: - Episode: 49/100, Reward: 25.0, Step: 25 -2022-10-31 00:12:06 - r - INFO: - Episode: 50/100, Reward: 31.0, Step: 31 -2022-10-31 00:12:06 - r - INFO: - Current episode 50 has the best eval reward: 24.80 -2022-10-31 00:12:06 - r - INFO: - Episode: 51/100, Reward: 22.0, Step: 22 -2022-10-31 00:12:06 - r - INFO: - Episode: 52/100, Reward: 39.0, Step: 39 -2022-10-31 00:12:06 - r - INFO: - Episode: 53/100, Reward: 36.0, Step: 36 -2022-10-31 00:12:06 - r - INFO: - Episode: 54/100, Reward: 26.0, Step: 26 -2022-10-31 00:12:07 - r - INFO: - Episode: 55/100, Reward: 33.0, Step: 33 -2022-10-31 00:12:07 - r - INFO: - Current episode 55 has the best eval reward: 38.70 -2022-10-31 00:12:07 - r - INFO: - Episode: 56/100, Reward: 56.0, Step: 56 -2022-10-31 00:12:07 - r - INFO: - Episode: 57/100, Reward: 112.0, Step: 112 -2022-10-31 00:12:07 - r - INFO: - Episode: 58/100, Reward: 101.0, Step: 101 -2022-10-31 00:12:08 - r - INFO: - Episode: 59/100, Reward: 69.0, Step: 69 -2022-10-31 00:12:08 - r - INFO: - Episode: 60/100, Reward: 75.0, Step: 75 -2022-10-31 00:12:08 - r - INFO: - Episode: 61/100, Reward: 182.0, Step: 182 -2022-10-31 00:12:09 - r - INFO: - Episode: 62/100, Reward: 52.0, Step: 52 -2022-10-31 00:12:09 - r - INFO: - Episode: 63/100, Reward: 67.0, Step: 67 -2022-10-31 00:12:09 - r - INFO: - Episode: 64/100, Reward: 53.0, Step: 53 -2022-10-31 00:12:09 - r - INFO: - Episode: 65/100, Reward: 119.0, Step: 119 -2022-10-31 00:12:10 - r - INFO: - Current episode 65 has the best eval reward: 171.90 -2022-10-31 00:12:10 - r - INFO: - Episode: 66/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:10 - r - INFO: - Episode: 67/100, Reward: 74.0, Step: 74 -2022-10-31 00:12:11 - r - INFO: - Episode: 68/100, Reward: 138.0, Step: 138 -2022-10-31 00:12:11 - r - INFO: - Episode: 69/100, Reward: 149.0, Step: 149 -2022-10-31 00:12:12 - r - INFO: - Episode: 70/100, Reward: 144.0, Step: 144 -2022-10-31 00:12:12 - r - INFO: - Current episode 70 has the best eval reward: 173.70 -2022-10-31 00:12:13 - r - INFO: - Episode: 71/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:13 - r - INFO: - Episode: 72/100, Reward: 198.0, Step: 198 -2022-10-31 00:12:14 - r - INFO: - Episode: 73/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:14 - r - INFO: - Episode: 74/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:15 - r - INFO: - Episode: 75/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:16 - r - INFO: - Current episode 75 has the best eval reward: 200.00 -2022-10-31 00:12:16 - r - INFO: - Episode: 76/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:17 - r - INFO: - Episode: 77/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:17 - r - INFO: - Episode: 78/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:18 - r - INFO: - Episode: 79/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:19 - r - INFO: - Episode: 80/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:19 - r - INFO: - Current episode 80 has the best eval reward: 200.00 -2022-10-31 00:12:20 - r - INFO: - Episode: 81/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:20 - r - INFO: - Episode: 82/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:21 - r - INFO: - Episode: 83/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:21 - r - INFO: - Episode: 84/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:22 - r - INFO: - Episode: 85/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:23 - r - INFO: - Current episode 85 has the best eval reward: 200.00 -2022-10-31 00:12:23 - r - INFO: - Episode: 86/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:24 - r - INFO: - Episode: 87/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:25 - r - INFO: - Episode: 88/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:25 - r - INFO: - Episode: 89/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:26 - r - INFO: - Episode: 90/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:27 - r - INFO: - Current episode 90 has the best eval reward: 200.00 -2022-10-31 00:12:27 - r - INFO: - Episode: 91/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:28 - r - INFO: - Episode: 92/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:28 - r - INFO: - Episode: 93/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:29 - r - INFO: - Episode: 94/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:29 - r - INFO: - Episode: 95/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:30 - r - INFO: - Current episode 95 has the best eval reward: 200.00 -2022-10-31 00:12:31 - r - INFO: - Episode: 96/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:31 - r - INFO: - Episode: 97/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:32 - r - INFO: - Episode: 98/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:32 - r - INFO: - Episode: 99/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:33 - r - INFO: - Episode: 100/100, Reward: 200.0, Step: 200 -2022-10-31 00:12:33 - r - INFO: - Current episode 100 has the best eval reward: 200.00 -2022-10-31 00:12:33 - r - INFO: - Finish training! diff --git a/projects/codes/DQN/Train_CartPole-v1_DQN_20221031-001201/models/checkpoint.pt b/projects/codes/DQN/Train_CartPole-v1_DQN_20221031-001201/models/checkpoint.pt deleted file mode 100644 index 722eb69..0000000 Binary files a/projects/codes/DQN/Train_CartPole-v1_DQN_20221031-001201/models/checkpoint.pt and /dev/null differ diff --git a/projects/codes/DQN/Train_CartPole-v1_DQN_20221031-001201/results/learning_curve.png b/projects/codes/DQN/Train_CartPole-v1_DQN_20221031-001201/results/learning_curve.png deleted file mode 100644 index 331f645..0000000 Binary files a/projects/codes/DQN/Train_CartPole-v1_DQN_20221031-001201/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/DQN/Train_CartPole-v1_DQN_20221031-001201/results/res.csv b/projects/codes/DQN/Train_CartPole-v1_DQN_20221031-001201/results/res.csv deleted file mode 100644 index 3bf53a3..0000000 --- a/projects/codes/DQN/Train_CartPole-v1_DQN_20221031-001201/results/res.csv +++ /dev/null @@ -1,101 +0,0 @@ -episodes,rewards,steps -0,18.0,18 -1,35.0,35 -2,13.0,13 -3,32.0,32 -4,16.0,16 -5,12.0,12 -6,13.0,13 -7,15.0,15 -8,11.0,11 -9,15.0,15 -10,9.0,9 -11,13.0,13 -12,13.0,13 -13,10.0,10 -14,9.0,9 -15,24.0,24 -16,8.0,8 -17,10.0,10 -18,11.0,11 -19,13.0,13 -20,12.0,12 -21,11.0,11 -22,9.0,9 -23,21.0,21 -24,14.0,14 -25,12.0,12 -26,9.0,9 -27,11.0,11 -28,12.0,12 -29,13.0,13 -30,10.0,10 -31,13.0,13 -32,18.0,18 -33,9.0,9 -34,10.0,10 -35,9.0,9 -36,10.0,10 -37,10.0,10 -38,10.0,10 -39,8.0,8 -40,9.0,9 -41,9.0,9 -42,20.0,20 -43,16.0,16 -44,17.0,17 -45,17.0,17 -46,17.0,17 -47,18.0,18 -48,25.0,25 -49,31.0,31 -50,22.0,22 -51,39.0,39 -52,36.0,36 -53,26.0,26 -54,33.0,33 -55,56.0,56 -56,112.0,112 -57,101.0,101 -58,69.0,69 -59,75.0,75 -60,182.0,182 -61,52.0,52 -62,67.0,67 -63,53.0,53 -64,119.0,119 -65,200.0,200 -66,74.0,74 -67,138.0,138 -68,149.0,149 -69,144.0,144 -70,200.0,200 -71,198.0,198 -72,200.0,200 -73,200.0,200 -74,200.0,200 -75,200.0,200 -76,200.0,200 -77,200.0,200 -78,200.0,200 -79,200.0,200 -80,200.0,200 -81,200.0,200 -82,200.0,200 -83,200.0,200 -84,200.0,200 -85,200.0,200 -86,200.0,200 -87,200.0,200 -88,200.0,200 -89,200.0,200 -90,200.0,200 -91,200.0,200 -92,200.0,200 -93,200.0,200 -94,200.0,200 -95,200.0,200 -96,200.0,200 -97,200.0,200 -98,200.0,200 -99,200.0,200 diff --git a/projects/codes/DQN/config/Acrobot-v1_DQN_Test.yaml b/projects/codes/DQN/config/Acrobot-v1_DQN_Test.yaml deleted file mode 100644 index d6e8d84..0000000 --- a/projects/codes/DQN/config/Acrobot-v1_DQN_Test.yaml +++ /dev/null @@ -1,22 +0,0 @@ -general_cfg: - algo_name: DQN - device: cuda - env_name: Acrobot-v1 - mode: test - load_checkpoint: true - load_path: Train_Acrobot-v1_DQN_20221026-094645 - max_steps: 100000 - save_fig: true - seed: 1 - show_fig: false - test_eps: 10 - train_eps: 100 -algo_cfg: - batch_size: 128 - buffer_size: 200000 - epsilon_decay: 500 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - lr: 0.002 - target_update: 4 diff --git a/projects/codes/DQN/config/Acrobot-v1_DQN_Train.yaml b/projects/codes/DQN/config/Acrobot-v1_DQN_Train.yaml deleted file mode 100644 index 0b18e79..0000000 --- a/projects/codes/DQN/config/Acrobot-v1_DQN_Train.yaml +++ /dev/null @@ -1,22 +0,0 @@ -general_cfg: - algo_name: DQN - device: cuda - env_name: Acrobot-v1 - mode: train - load_checkpoint: false - load_path: Train_CartPole-v1_DQN_20221026-054757 - max_steps: 100000 - save_fig: true - seed: 1 - show_fig: false - test_eps: 10 - train_eps: 100 -algo_cfg: - batch_size: 128 - buffer_size: 200000 - epsilon_decay: 500 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - lr: 0.002 - target_update: 4 diff --git a/projects/codes/DQN/config/CartPole-v1_DQN_Test.yaml b/projects/codes/DQN/config/CartPole-v1_DQN_Test.yaml deleted file mode 100644 index baa98f0..0000000 --- a/projects/codes/DQN/config/CartPole-v1_DQN_Test.yaml +++ /dev/null @@ -1,22 +0,0 @@ -general_cfg: - algo_name: DQN - device: cuda - env_name: CartPole-v1 - mode: test - load_checkpoint: true - load_path: Train_CartPole-v1_DQN_20221031-001201 - max_steps: 200 - save_fig: true - seed: 0 - show_fig: false - test_eps: 10 - train_eps: 100 -algo_cfg: - batch_size: 64 - buffer_size: 100000 - epsilon_decay: 500 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - lr: 0.0001 - target_update: 4 diff --git a/projects/codes/DQN/config/CartPole-v1_DQN_Train.yaml b/projects/codes/DQN/config/CartPole-v1_DQN_Train.yaml deleted file mode 100644 index 14297b5..0000000 --- a/projects/codes/DQN/config/CartPole-v1_DQN_Train.yaml +++ /dev/null @@ -1,22 +0,0 @@ -general_cfg: - algo_name: DQN - device: cuda - env_name: CartPole-v1 - mode: train - load_checkpoint: false - load_path: Train_CartPole-v1_DQN_20221026-054757 - max_steps: 200 - save_fig: true - seed: 0 - show_fig: false - test_eps: 10 - train_eps: 200 -algo_cfg: - batch_size: 64 - buffer_size: 100000 - epsilon_decay: 500 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - lr: 0.0001 - target_update: 4 diff --git a/projects/codes/DQN/config/config.py b/projects/codes/DQN/config/config.py deleted file mode 100644 index 2653c8d..0000000 --- a/projects/codes/DQN/config/config.py +++ /dev/null @@ -1,38 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-10-30 00:37:33 -LastEditor: JiangJi -LastEditTime: 2022-10-31 00:11:57 -Discription: default parameters of DQN -''' -from common.config import GeneralConfig,AlgoConfig -class GeneralConfigDQN(GeneralConfig): - def __init__(self) -> None: - self.env_name = "CartPole-v1" # name of environment - self.algo_name = "DQN" # name of algorithm - self.mode = "train" # train or test - self.seed = 1 # random seed - self.device = "cuda" # device to use - self.train_eps = 100 # number of episodes for training - self.test_eps = 10 # number of episodes for testing - self.max_steps = 200 # max steps for each episode - self.load_checkpoint = False - self.load_path = "tasks" # path to load model - self.show_fig = False # show figure or not - self.save_fig = True # save figure or not - -class AlgoConfigDQN(AlgoConfig): - def __init__(self) -> None: - # set epsilon_start=epsilon_end can obtain fixed epsilon=epsilon_end - self.epsilon_start = 0.95 # epsilon start value - self.epsilon_end = 0.01 # epsilon end value - self.epsilon_decay = 500 # epsilon decay rate - self.hidden_dim = 256 # hidden_dim for MLP - self.gamma = 0.95 # discount factor - self.lr = 0.0001 # learning rate - self.buffer_size = 100000 # size of replay buffer - self.batch_size = 64 # batch size - self.target_update = 800 # target network update frequency per steps diff --git a/projects/codes/DQN/dqn.py b/projects/codes/DQN/dqn.py deleted file mode 100644 index 761d25f..0000000 --- a/projects/codes/DQN/dqn.py +++ /dev/null @@ -1,130 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -@Author: John -@Email: johnjim0816@gmail.com -@Date: 2020-06-12 00:50:49 -@LastEditor: John -LastEditTime: 2022-10-31 00:07:19 -@Discription: -@Environment: python 3.7.7 -''' -'''off-policy -''' - -import torch -import torch.nn as nn -import torch.optim as optim -import random -import math -import numpy as np - -class DQN: - def __init__(self,model,memory,cfg): - - self.n_actions = cfg.n_actions - self.device = torch.device(cfg.device) - self.gamma = cfg.gamma - ## e-greedy parameters - self.sample_count = 0 # sample count for epsilon decay - self.epsilon = cfg.epsilon_start - self.sample_count = 0 - self.epsilon_start = cfg.epsilon_start - self.epsilon_end = cfg.epsilon_end - self.epsilon_decay = cfg.epsilon_decay - self.batch_size = cfg.batch_size - self.target_update = cfg.target_update - self.policy_net = model.to(self.device) - self.target_net = model.to(self.device) - ## copy parameters from policy net to target net - for target_param, param in zip(self.target_net.parameters(),self.policy_net.parameters()): - target_param.data.copy_(param.data) - # self.target_net.load_state_dict(self.policy_net.state_dict()) # or use this to copy parameters - self.optimizer = optim.Adam(self.policy_net.parameters(), lr=cfg.lr) - self.memory = memory - self.update_flag = False - - def sample_action(self, state): - ''' sample action with e-greedy policy - ''' - self.sample_count += 1 - # epsilon must decay(linear,exponential and etc.) for balancing exploration and exploitation - self.epsilon = self.epsilon_end + (self.epsilon_start - self.epsilon_end) * \ - math.exp(-1. * self.sample_count / self.epsilon_decay) - if random.random() > self.epsilon: - with torch.no_grad(): - state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(dim=0) - q_values = self.policy_net(state) - action = q_values.max(1)[1].item() # choose action corresponding to the maximum q value - else: - action = random.randrange(self.n_actions) - return action - # @torch.no_grad() - # def sample_action(self, state): - # ''' sample action with e-greedy policy - # ''' - # self.sample_count += 1 - # # epsilon must decay(linear,exponential and etc.) for balancing exploration and exploitation - # self.epsilon = self.epsilon_end + (self.epsilon_start - self.epsilon_end) * \ - # math.exp(-1. * self.sample_count / self.epsilon_decay) - # if random.random() > self.epsilon: - # state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(dim=0) - # q_values = self.policy_net(state) - # action = q_values.max(1)[1].item() # choose action corresponding to the maximum q value - # else: - # action = random.randrange(self.n_actions) - # return action - def predict_action(self,state): - ''' predict action - ''' - with torch.no_grad(): - state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(dim=0) - q_values = self.policy_net(state) - action = q_values.max(1)[1].item() # choose action corresponding to the maximum q value - return action - def update(self): - if len(self.memory) < self.batch_size: # when transitions in memory donot meet a batch, not update - return - else: - if not self.update_flag: - print("Begin to update!") - self.update_flag = True - # sample a batch of transitions from replay buffer - state_batch, action_batch, reward_batch, next_state_batch, done_batch = self.memory.sample( - self.batch_size) - state_batch = torch.tensor(np.array(state_batch), device=self.device, dtype=torch.float) # shape(batchsize,n_states) - action_batch = torch.tensor(action_batch, device=self.device).unsqueeze(1) # shape(batchsize,1) - reward_batch = torch.tensor(reward_batch, device=self.device, dtype=torch.float).unsqueeze(1) # shape(batchsize,1) - next_state_batch = torch.tensor(np.array(next_state_batch), device=self.device, dtype=torch.float) # shape(batchsize,n_states) - done_batch = torch.tensor(np.float32(done_batch), device=self.device).unsqueeze(1) # shape(batchsize,1) - # print(state_batch.shape,action_batch.shape,reward_batch.shape,next_state_batch.shape,done_batch.shape) - # compute current Q(s_t,a), it is 'y_j' in pseucodes - q_value_batch = self.policy_net(state_batch).gather(dim=1, index=action_batch) # shape(batchsize,1),requires_grad=True - # print(q_values.requires_grad) - # compute max(Q(s_t+1,A_t+1)) respects to actions A, next_max_q_value comes from another net and is just regarded as constant for q update formula below, thus should detach to requires_grad=False - next_max_q_value_batch = self.target_net(next_state_batch).max(1)[0].detach().unsqueeze(1) - # print(q_values.shape,next_q_values.shape) - # compute expected q value, for terminal state, done_batch[0]=1, and expected_q_value=rewardcorrespondingly - expected_q_value_batch = reward_batch + self.gamma * next_max_q_value_batch* (1-done_batch) - # print(expected_q_value_batch.shape,expected_q_value_batch.requires_grad) - loss = nn.MSELoss()(q_value_batch, expected_q_value_batch) # shape same to - # backpropagation - self.optimizer.zero_grad() - loss.backward() - # clip to avoid gradient explosion - for param in self.policy_net.parameters(): - param.grad.data.clamp_(-1, 1) - self.optimizer.step() - if self.sample_count % self.target_update == 0: # target net update, target_update means "C" in pseucodes - self.target_net.load_state_dict(self.policy_net.state_dict()) - - def save_model(self, fpath): - from pathlib import Path - # create path - Path(fpath).mkdir(parents=True, exist_ok=True) - torch.save(self.target_net.state_dict(), f"{fpath}/checkpoint.pt") - - def load_model(self, fpath): - self.target_net.load_state_dict(torch.load(f"{fpath}/checkpoint.pt")) - for target_param, param in zip(self.target_net.parameters(), self.policy_net.parameters()): - param.data.copy_(target_param.data) diff --git a/projects/codes/DQN/task0.py b/projects/codes/DQN/task0.py deleted file mode 100644 index e69ed45..0000000 --- a/projects/codes/DQN/task0.py +++ /dev/null @@ -1,138 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-10-12 11:09:54 -LastEditor: JiangJi -LastEditTime: 2022-10-31 00:13:31 -Discription: CartPole-v1,Acrobot-v1 -''' -import sys,os -curr_path = os.path.dirname(os.path.abspath(__file__)) # current path -parent_path = os.path.dirname(curr_path) # parent path -sys.path.append(parent_path) # add to system path -import gym -from common.utils import all_seed,merge_class_attrs -from common.models import MLP -from common.memories import ReplayBuffer -from common.launcher import Launcher -from envs.register import register_env -from dqn import DQN -from config.config import GeneralConfigDQN,AlgoConfigDQN -class Main(Launcher): - def __init__(self) -> None: - super().__init__() - self.cfgs['general_cfg'] = merge_class_attrs(self.cfgs['general_cfg'],GeneralConfigDQN()) - self.cfgs['algo_cfg'] = merge_class_attrs(self.cfgs['algo_cfg'],AlgoConfigDQN()) - def env_agent_config(self,cfg,logger): - ''' create env and agent - ''' - register_env(cfg.env_name) - env = gym.make(cfg.env_name,new_step_api=True) # create env - if cfg.seed !=0: # set random seed - all_seed(env,seed=cfg.seed) - try: # state dimension - n_states = env.observation_space.n # print(hasattr(env.observation_space, 'n')) - except AttributeError: - n_states = env.observation_space.shape[0] # print(hasattr(env.observation_space, 'shape')) - n_actions = env.action_space.n # action dimension - logger.info(f"n_states: {n_states}, n_actions: {n_actions}") # print info - # update to cfg paramters - setattr(cfg, 'n_states', n_states) - setattr(cfg, 'n_actions', n_actions) - # cfg.update({"n_states":n_states,"n_actions":n_actions}) # update to cfg paramters - model = MLP(n_states,n_actions,hidden_dim=cfg.hidden_dim) - memory = ReplayBuffer(cfg.buffer_size) # replay buffer - agent = DQN(model,memory,cfg) # create agent - return env, agent - def train_one_episode(self, env, agent, cfg): - ep_reward = 0 # reward per episode - ep_step = 0 - state = env.reset() # reset and obtain initial state - for _ in range(cfg.max_steps): - ep_step += 1 - action = agent.sample_action(state) # sample action - next_state, reward, terminated, truncated , info = env.step(action) # update env and return transitions under new_step_api of OpenAI Gym - agent.memory.push(state, action, reward, - next_state, terminated) # save transitions - agent.update() # update agent - state = next_state # update next state for env - ep_reward += reward # - if terminated: - break - return agent,ep_reward,ep_step - def test_one_episode(self, env, agent, cfg): - ep_reward = 0 # reward per episode - ep_step = 0 - state = env.reset() # reset and obtain initial state - for _ in range(cfg.max_steps): - ep_step += 1 - action = agent.predict_action(state) # sample action - next_state, reward, terminated, truncated , info = env.step(action) # update env and return transitions under new_step_api of OpenAI Gym - state = next_state # update next state for env - ep_reward += reward # - if terminated: - break - return agent,ep_reward,ep_step - # def train(self,env, agent,cfg,logger): - # ''' 训练 - # ''' - # logger.info("Start training!") - # logger.info(f"Env: {cfg.env_name}, Algorithm: {cfg.algo_name}, Device: {cfg.device}") - # rewards = [] # record rewards for all episodes - # steps = [] # record steps for all episodes - # for i_ep in range(cfg.train_eps): - # ep_reward = 0 # reward per episode - # ep_step = 0 - # state = env.reset() # reset and obtain initial state - # for _ in range(cfg.max_steps): - # ep_step += 1 - # action = agent.sample_action(state) # sample action - # next_state, reward, terminated, truncated , info = env.step(action) # update env and return transitions under new_step_api of OpenAI Gym - # agent.memory.push(state, action, reward, - # next_state, terminated) # save transitions - # state = next_state # update next state for env - # agent.update() # update agent - # ep_reward += reward # - # if terminated: - # break - # if (i_ep + 1) % cfg.target_update == 0: # target net update, target_update means "C" in pseucodes - # agent.target_net.load_state_dict(agent.policy_net.state_dict()) - # steps.append(ep_step) - # rewards.append(ep_reward) - # logger.info(f'Episode: {i_ep+1}/{cfg.train_eps}, Reward: {ep_reward:.2f}: Epislon: {agent.epsilon:.3f}') - # logger.info("Finish training!") - # env.close() - # res_dic = {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - # return res_dic - - # def test(self,cfg, env, agent,logger): - # logger.info("Start testing!") - # logger.info(f"Env: {cfg.env_name}, Algorithm: {cfg.algo_name}, Device: {cfg.device}") - # rewards = [] # record rewards for all episodes - # steps = [] # record steps for all episodes - # for i_ep in range(cfg.test_eps): - # ep_reward = 0 # reward per episode - # ep_step = 0 - # state = env.reset() # reset and obtain initial state - # for _ in range(cfg.max_steps): - # ep_step+=1 - # action = agent.predict_action(state) # predict action - # next_state, reward, terminated, _, _ = env.step(action) - # state = next_state - # ep_reward += reward - # if terminated: - # break - # steps.append(ep_step) - # rewards.append(ep_reward) - # logger.info(f"Episode: {i_ep+1}/{cfg.test_eps}, Reward: {ep_reward:.2f}") - # logger.info("Finish testing!") - # env.close() - # return {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - - -if __name__ == "__main__": - main = Main() - main.run() - diff --git a/projects/codes/DQN/task1.py b/projects/codes/DQN/task1.py deleted file mode 100644 index 590d0c2..0000000 --- a/projects/codes/DQN/task1.py +++ /dev/null @@ -1,221 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-10-24 08:21:31 -LastEditor: JiangJi -LastEditTime: 2022-10-26 09:50:49 -Discription: Not finished -''' -import sys,os -os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE" # avoid "OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized." -curr_path = os.path.dirname(os.path.abspath(__file__)) # current path -parent_path = os.path.dirname(curr_path) # parent path -sys.path.append(parent_path) # add path to system path - -import gym -import torch -import datetime -import numpy as np -import argparse -from common.utils import all_seed -from common.models import MLP -from common.memories import ReplayBuffer -from common.launcher import Launcher -from envs.register import register_env -from dqn import DQN -import torch.nn as nn -import torch.nn.functional as F -import torchvision.transforms as T -from PIL import Image -resize = T.Compose([T.ToPILImage(), - T.Resize(40, interpolation=Image.CUBIC), - T.ToTensor()]) - -# xvfb-run -s "-screen 0 640x480x24" python main1.py -def get_cart_location(env,screen_width): - world_width = env.x_threshold * 2 - scale = screen_width / world_width - return int(env.state[0] * scale + screen_width / 2.0) # MIDDLE OF CART - -def get_screen(env): - # Returned screen requested by gym is 400x600x3, but is sometimes larger - # such as 800x1200x3. Transpose it into torch order (CHW). - screen = env.render().transpose((2, 0, 1)) - # Cart is in the lower half, so strip off the top and bottom of the screen - _, screen_height, screen_width = screen.shape - screen = screen[:, int(screen_height*0.4):int(screen_height * 0.8)] - view_width = int(screen_width * 0.6) - cart_location = get_cart_location(env,screen_width) - if cart_location < view_width // 2: - slice_range = slice(view_width) - elif cart_location > (screen_width - view_width // 2): - slice_range = slice(-view_width, None) - else: - slice_range = slice(cart_location - view_width // 2, - cart_location + view_width // 2) - # Strip off the edges, so that we have a square image centered on a cart - screen = screen[:, :, slice_range] - # Convert to float, rescale, convert to torch tensor - # (this doesn't require a copy) - screen = np.ascontiguousarray(screen, dtype=np.float32) / 255 - screen = torch.from_numpy(screen) - # Resize, and add a batch dimension (BCHW) - return resize(screen) - - -class CNN(nn.Module): - - def __init__(self, h, w, outputs): - super(CNN, self).__init__() - self.conv1 = nn.Conv2d(3, 16, kernel_size=5, stride=2) - self.bn1 = nn.BatchNorm2d(16) - self.conv2 = nn.Conv2d(16, 32, kernel_size=5, stride=2) - self.bn2 = nn.BatchNorm2d(32) - self.conv3 = nn.Conv2d(32, 32, kernel_size=5, stride=2) - self.bn3 = nn.BatchNorm2d(32) - - # Number of Linear input connections depends on output of conv2d layers - # and therefore the input image size, so compute it. - def conv2d_size_out(size, kernel_size = 5, stride = 2): - return (size - (kernel_size - 1) - 1) // stride + 1 - convw = conv2d_size_out(conv2d_size_out(conv2d_size_out(w))) - convh = conv2d_size_out(conv2d_size_out(conv2d_size_out(h))) - linear_input_size = convw * convh * 32 - self.head = nn.Linear(linear_input_size, outputs) - - # Called with either one element to determine next action, or a batch - # during optimization. Returns tensor([[left0exp,right0exp]...]). - def forward(self, x): - x = F.relu(self.bn1(self.conv1(x))) - x = F.relu(self.bn2(self.conv2(x))) - x = F.relu(self.bn3(self.conv3(x))) - return self.head(x.view(x.size(0), -1)) -class Main(Launcher): - def get_args(self): - """ hyperparameters - """ - curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # obtain current time - parser = argparse.ArgumentParser(description="hyperparameters") - parser.add_argument('--algo_name',default='DQN',type=str,help="name of algorithm") - parser.add_argument('--env_name',default='CartPole-v1',type=str,help="name of environment") - parser.add_argument('--train_eps',default=800,type=int,help="episodes of training") - parser.add_argument('--test_eps',default=20,type=int,help="episodes of testing") - parser.add_argument('--ep_max_steps',default = 100000,type=int,help="steps per episode, much larger value can simulate infinite steps") - parser.add_argument('--gamma',default=0.999,type=float,help="discounted factor") - parser.add_argument('--epsilon_start',default=0.95,type=float,help="initial value of epsilon") - parser.add_argument('--epsilon_end',default=0.01,type=float,help="final value of epsilon") - parser.add_argument('--epsilon_decay',default=500,type=int,help="decay rate of epsilon, the higher value, the slower decay") - parser.add_argument('--lr',default=0.0001,type=float,help="learning rate") - parser.add_argument('--memory_capacity',default=100000,type=int,help="memory capacity") - parser.add_argument('--batch_size',default=128,type=int) - parser.add_argument('--target_update',default=4,type=int) - parser.add_argument('--hidden_dim',default=256,type=int) - parser.add_argument('--device',default='cuda',type=str,help="cpu or cuda") - parser.add_argument('--seed',default=10,type=int,help="seed") - parser.add_argument('--show_fig',default=False,type=bool,help="if show figure or not") - parser.add_argument('--save_fig',default=True,type=bool,help="if save figure or not") - # please manually change the following args in this script if you want - parser.add_argument('--result_path',default=curr_path + "/outputs/" + parser.parse_args().env_name + \ - '/' + curr_time + '/results' ) - parser.add_argument('--model_path',default=curr_path + "/outputs/" + parser.parse_args().env_name + \ - '/' + curr_time + '/models' ) - args = parser.parse_args() - args = {**vars(args)} # type(dict) - return args - - def env_agent_config(self,cfg): - ''' create env and agent - ''' - env = gym.make('CartPole-v1', new_step_api=True, render_mode='single_rgb_array').unwrapped - if cfg['seed'] !=0: # set random seed - all_seed(env,seed=cfg["seed"]) - try: # state dimension - n_states = env.observation_space.n # print(hasattr(env.observation_space, 'n')) - except AttributeError: - n_states = env.observation_space.shape[0] # print(hasattr(env.observation_space, 'shape')) - n_actions = env.action_space.n # action dimension - print(f"n_states: {n_states}, n_actions: {n_actions}") - cfg.update({"n_states":n_states,"n_actions":n_actions}) # update to cfg paramters - env.reset() - init_screen = get_screen(env) - _, screen_height, screen_width = init_screen.shape - model = CNN(screen_height, screen_width, n_actions) - memory = ReplayBuffer(cfg["memory_capacity"]) # replay buffer - agent = DQN(model,memory,cfg) # create agent - return env, agent - - def train(self,cfg, env, agent): - ''' 训练 - ''' - print("Start training!") - print(f"Env: {cfg['env_name']}, Algorithm: {cfg['algo_name']}, Device: {cfg['device']}") - rewards = [] # record rewards for all episodes - steps = [] - for i_ep in range(cfg["train_eps"]): - ep_reward = 0 # reward per episode - ep_step = 0 - state = env.reset() # reset and obtain initial state - last_screen = get_screen(env) - current_screen = get_screen(env) - state = current_screen - last_screen - for _ in range(cfg['ep_max_steps']): - ep_step += 1 - action = agent.sample_action(state) # sample action - _, reward, done, _,_ = env.step(action) # update env and return transitions - last_screen = current_screen - current_screen = get_screen(env) - next_state = current_screen - last_screen - agent.memory.push(state.cpu().numpy(), action, reward, - next_state.cpu().numpy(), done) # save transitions - state = next_state # update next state for env - agent.update() # update agent - ep_reward += reward # - if done: - break - if (i_ep + 1) % cfg["target_update"] == 0: # target net update, target_update means "C" in pseucodes - agent.target_net.load_state_dict(agent.policy_net.state_dict()) - steps.append(ep_step) - rewards.append(ep_reward) - if (i_ep + 1) % 10 == 0: - print(f'Episode: {i_ep+1}/{cfg["train_eps"]}, Reward: {ep_reward:.2f}, step: {ep_step:d}, Epislon: {agent.epsilon:.3f}') - print("Finish training!") - env.close() - res_dic = {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - return res_dic - - def test(self,cfg, env, agent): - print("Start testing!") - print(f"Env: {cfg['env_name']}, Algorithm: {cfg['algo_name']}, Device: {cfg['device']}") - rewards = [] # record rewards for all episodes - steps = [] - for i_ep in range(cfg['test_eps']): - ep_reward = 0 # reward per episode - ep_step = 0 - state = env.reset() # reset and obtain initial state - last_screen = get_screen(env) - current_screen = get_screen(env) - state = current_screen - last_screen - for _ in range(cfg['ep_max_steps']): - ep_step+=1 - action = agent.predict_action(state) # predict action - _, reward, done, _,_ = env.step(action) - last_screen = current_screen - current_screen = get_screen(env) - next_state = current_screen - last_screen - state = next_state - ep_reward += reward - if done: - break - steps.append(ep_step) - rewards.append(ep_reward) - print(f"Episode: {i_ep+1}/{cfg['test_eps']}，Reward: {ep_reward:.2f}") - print("Finish testing!") - env.close() - return {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - - -if __name__ == "__main__": - main = Main() - main.run() diff --git a/projects/codes/DoubleDQN/README.md b/projects/codes/DoubleDQN/README.md deleted file mode 100644 index 714bd26..0000000 --- a/projects/codes/DoubleDQN/README.md +++ /dev/null @@ -1,39 +0,0 @@ -食用本篇之前，需要有DQN算法的基础，参考[DQN算法实战](../DQN)。 - -## 原理简介 - -Double-DQN是2016年提出的算法，灵感源自2010年的Double-Qlearning，可参考论文[Deep Reinforcement Learning with Double Q-learning](https://arxiv.org/abs/1509.06461)。 -跟Nature DQN一样，Double-DQN也用了两个网络，一个当前网络(对应用$Q$表示)，一个目标网络(对应一般用$Q'$表示，为方便区分，以下用$Q_{tar}$代替)。我们先回忆一下，对于非终止状态，目标$Q_{tar}$值计算如下 -![在这里插入图片描述](assets/20201222145725907.png) - -而在Double-DQN中，不再是直接从目标$Q_{tar}$网络中选择各个动作中的最大$Q_{tar}$值，而是先从当前$Q$网络选择$Q$值最大对应的动作，然后代入到目标网络中计算对应的值： -![在这里插入图片描述](assets/20201222150225327.png) -Double-DQN的好处是Nature DQN中使用max虽然可以快速让Q值向可能的优化目标靠拢，但是很容易过犹不及，导致过度估计(Over Estimation)，所谓过度估计就是最终我们得到的算法模型有很大的偏差(bias)。为了解决这个问题， DDQN通过解耦目标Q值动作的选择和目标Q值的计算这两步，来达到消除过度估计的问题，感兴趣可以阅读原论文。 - -伪代码如下： -![在这里插入图片描述](assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70.png) -当然也可以两个网络可以同时为当前网络和目标网络，如下： -![在这里插入图片描述](assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210328110837146.png) -或者这样更好理解如何同时为当前网络和目标网络： -![在这里插入图片描述](assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210328110837157.png) - -## 代码实战 -完整程序见[github](https://github.com/JohnJim0816/reinforcement-learning-tutorials/tree/master/DoubleDQN)。结合上面的原理，其实Double DQN改进来很简单，基本只需要在```update```中修改几行代码，如下： -```python -'''以下是Nature DQN的q_target计算方式 -next_q_state_value = self.target_net( -next_state_batch).max(1)[0].detach() # # 计算所有next states的Q'(s_{t+1})的最大值，Q'为目标网络的q函数,比如tensor([ 0.0060, -0.0171,...,]) -#计算 q_target -#对于终止状态，此时done_batch[0]=1, 对应的expected_q_value等于reward -q_target = reward_batch + self.gamma * next_q_state_value * (1-done_batch[0]) -''' -'''以下是Double DQNq_target计算方式，与NatureDQN稍有不同''' -next_target_values = self.target_net( -next_state_batch) -#选出Q(s_t‘, a)对应的action，代入到next_target_values获得target net对应的next_q_value，即Q’(s_t|a=argmax Q(s_t‘, a)) -next_target_q_value = next_target_values.gather(1, torch.max(next_q_values, 1)[1].unsqueeze(1)).squeeze(1) -q_target = reward_batch + self.gamma * next_target_q_value * (1-done_batch[0]) -``` -reward变化结果如下： -![在这里插入图片描述](assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210328110837128.png) -其中下边蓝色和红色分别表示Double DQN和Nature DQN在训练中的reward变化图，而上面蓝色和绿色则表示Double DQN和Nature DQN在测试中的reward变化图。 \ No newline at end of file diff --git a/projects/codes/DoubleDQN/assets/20201222145725907.png b/projects/codes/DoubleDQN/assets/20201222145725907.png deleted file mode 100644 index d2cbb2d..0000000 Binary files a/projects/codes/DoubleDQN/assets/20201222145725907.png and /dev/null differ diff --git a/projects/codes/DoubleDQN/assets/20201222150225327.png b/projects/codes/DoubleDQN/assets/20201222150225327.png deleted file mode 100644 index 20b79be..0000000 Binary files a/projects/codes/DoubleDQN/assets/20201222150225327.png and /dev/null differ diff --git a/projects/codes/DoubleDQN/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210328110837128.png b/projects/codes/DoubleDQN/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210328110837128.png deleted file mode 100644 index 427a903..0000000 Binary files a/projects/codes/DoubleDQN/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210328110837128.png and /dev/null differ diff --git a/projects/codes/DoubleDQN/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210328110837146.png b/projects/codes/DoubleDQN/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210328110837146.png deleted file mode 100644 index d95f900..0000000 Binary files a/projects/codes/DoubleDQN/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210328110837146.png and /dev/null differ diff --git a/projects/codes/DoubleDQN/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210328110837157.png b/projects/codes/DoubleDQN/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210328110837157.png deleted file mode 100644 index ddeda96..0000000 Binary files a/projects/codes/DoubleDQN/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210328110837157.png and /dev/null differ diff --git a/projects/codes/DoubleDQN/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70.png b/projects/codes/DoubleDQN/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70.png deleted file mode 100644 index dec19e5..0000000 Binary files a/projects/codes/DoubleDQN/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70.png and /dev/null differ diff --git a/projects/codes/DoubleDQN/double_dqn.py b/projects/codes/DoubleDQN/double_dqn.py deleted file mode 100644 index b7f4e97..0000000 --- a/projects/codes/DoubleDQN/double_dqn.py +++ /dev/null @@ -1,106 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -@Author: John -@Email: johnjim0816@gmail.com -@Date: 2020-06-12 00:50:49 -@LastEditor: John -LastEditTime: 2022-08-29 23:34:20 -@Discription: -@Environment: python 3.7.7 -''' -'''off-policy -''' - - -import torch -import torch.nn as nn -import torch.optim as optim -import torch.nn.functional as F -import random -import math -import numpy as np -class DoubleDQN: - def __init__(self,models, memories, cfg): - self.n_actions = cfg['n_actions'] - self.device = torch.device(cfg['device']) - self.gamma = cfg['gamma'] - ## e-greedy parameters - self.sample_count = 0 # sample count for epsilon decay - self.epsilon_start = cfg['epsilon_start'] - self.epsilon_end = cfg['epsilon_end'] - self.epsilon_decay = cfg['epsilon_decay'] - self.batch_size = cfg['batch_size'] - self.policy_net = models['Qnet'].to(self.device) - self.target_net = models['Qnet'].to(self.device) - # target_net copy from policy_net - for target_param, param in zip(self.target_net.parameters(), self.policy_net.parameters()): - target_param.data.copy_(param.data) - # self.target_net.eval() # donnot use BatchNormalization or Dropout - # the difference between parameters() and state_dict() is that parameters() require_grad=True - self.optimizer = optim.Adam(self.policy_net.parameters(), lr=cfg['lr']) - self.memory = memories['Memory'] - self.update_flag = False - - def sample_action(self, state): - ''' sample action - ''' - self.sample_count += 1 - self.epsilon = self.epsilon_end + (self.epsilon_start - self.epsilon_end) * math.exp(-1. * self.sample_count / self.epsilon_decay) - if random.random() > self.epsilon: - with torch.no_grad(): - state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(0) - q_value = self.policy_net(state) - action = q_value.max(1)[1].item() - else: - action = random.randrange(self.n_actions) - return action - def predict_action(self, state): - ''' predict action - ''' - with torch.no_grad(): - state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(0) - q_value = self.policy_net(state) - action = q_value.max(1)[1].item() - return action - def update(self): - if len(self.memory) < self.batch_size: # when transitions in memory donot meet a batch, not update - return - else: - if not self.update_flag: - print("Begin to update!") - self.update_flag = True - # sample a batch of transitions from replay buffer - state_batch, action_batch, reward_batch, next_state_batch, done_batch = self.memory.sample(self.batch_size) - # convert to tensor - state_batch = torch.tensor(np.array(state_batch), device=self.device, dtype=torch.float) - action_batch = torch.tensor(action_batch, device=self.device).unsqueeze(1) # shape(batchsize,1) - reward_batch = torch.tensor(reward_batch, device=self.device, dtype=torch.float).unsqueeze(1) # shape(batchsize,1) - next_state_batch = torch.tensor(np.array(next_state_batch), device=self.device, dtype=torch.float) - done_batch = torch.tensor(np.float32(done_batch), device=self.device).unsqueeze(1) # shape(batchsize,1) - # compute current Q(s_t|a=a_t) - q_value_batch = self.policy_net(state_batch).gather(dim=1, index=action_batch) # shape(batchsize,1),requires_grad=True - next_q_value_batch = self.policy_net(next_state_batch) - '''the following is the way of computing Double DQN expected_q_value，a bit different from Nature DQN''' - next_target_value_batch = self.target_net(next_state_batch) - # choose action a from Q(s_t‘, a), next_target_values obtain next_q_value，which is Q’(s_t|a=argmax Q(s_t‘, a)) - next_target_q_value_batch = next_target_value_batch.gather(1, torch.max(next_q_value_batch, 1)[1].unsqueeze(1)) # shape(batchsize,1) - expected_q_value_batch = reward_batch + self.gamma * next_target_q_value_batch * (1-done_batch) - loss = nn.MSELoss()(q_value_batch , expected_q_value_batch) - self.optimizer.zero_grad() - loss.backward() - # clip to avoid gradient explosion - for param in self.policy_net.parameters(): - param.grad.data.clamp_(-1, 1) - self.optimizer.step() - - def save_model(self,path): - from pathlib import Path - # create path - Path(path).mkdir(parents=True, exist_ok=True) - torch.save(self.target_net.state_dict(), path+'checkpoint.pth') - - def load_model(self,path): - self.target_net.load_state_dict(torch.load(path+'checkpoint.pth')) - for target_param, param in zip(self.target_net.parameters(), self.policy_net.parameters()): - param.data.copy_(target_param.data) diff --git a/projects/codes/DoubleDQN/main.py b/projects/codes/DoubleDQN/main.py deleted file mode 100644 index a66025e..0000000 --- a/projects/codes/DoubleDQN/main.py +++ /dev/null @@ -1,129 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2021-11-07 18:10:37 -LastEditor: JiangJi -LastEditTime: 2022-08-29 23:33:31 -Discription: -''' -import sys,os -curr_path = os.path.dirname(os.path.abspath(__file__)) # current path -parent_path = os.path.dirname(curr_path) # parent path -sys.path.append(parent_path) # add to system path - -import gym -import datetime -import argparse - -from common.utils import all_seed -from common.models import MLP -from common.memories import ReplayBufferQue -from DoubleDQN.double_dqn import DoubleDQN -from common.launcher import Launcher -from envs.register import register_env -class Main(Launcher): - def get_args(self): - ''' hyperparameters - ''' - curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # obtain current time - parser = argparse.ArgumentParser(description="hyperparameters") - parser.add_argument('--algo_name',default='DoubleDQN',type=str,help="name of algorithm") - parser.add_argument('--env_name',default='CartPole-v0',type=str,help="name of environment") - parser.add_argument('--train_eps',default=200,type=int,help="episodes of training") - parser.add_argument('--test_eps',default=20,type=int,help="episodes of testing") - parser.add_argument('--ep_max_steps',default = 100000,type=int,help="steps per episode, much larger value can simulate infinite steps") - parser.add_argument('--gamma',default=0.95,type=float,help="discounted factor") - parser.add_argument('--epsilon_start',default=0.95,type=float,help="initial value of epsilon") - parser.add_argument('--epsilon_end',default=0.01,type=float,help="final value of epsilon") - parser.add_argument('--epsilon_decay',default=500,type=int,help="decay rate of epsilon") - parser.add_argument('--lr',default=0.0001,type=float,help="learning rate") - parser.add_argument('--memory_capacity',default=100000,type=int,help="memory capacity") - parser.add_argument('--batch_size',default=64,type=int) - parser.add_argument('--target_update',default=4,type=int) - parser.add_argument('--hidden_dim',default=256,type=int) - parser.add_argument('--device',default='cpu',type=str,help="cpu or cuda") - parser.add_argument('--seed',default=1,type=int,help="seed") - parser.add_argument('--show_fig',default=False,type=bool,help="if show figure or not") - parser.add_argument('--save_fig',default=True,type=bool,help="if save figure or not") - args = parser.parse_args() - default_args = {'result_path':f"{curr_path}/outputs/{args.env_name}/{curr_time}/results/", - 'model_path':f"{curr_path}/outputs/{args.env_name}/{curr_time}/models/", - } - args = {**vars(args),**default_args} # type(dict) - return args - def env_agent_config(self,cfg): - ''' create env and agent - ''' - register_env(cfg['env_name']) - env = gym.make(cfg['env_name']) - if cfg['seed'] !=0: # set random seed - all_seed(env,seed=cfg["seed"]) - try: # state dimension - n_states = env.observation_space.n # print(hasattr(env.observation_space, 'n')) - except AttributeError: - n_states = env.observation_space.shape[0] # print(hasattr(env.observation_space, 'shape')) - n_actions = env.action_space.n # action dimension - print(f"n_states: {n_states}, n_actions: {n_actions}") - cfg.update({"n_states":n_states,"n_actions":n_actions}) # update to cfg paramters - models = {'Qnet':MLP(n_states,n_actions,hidden_dim=cfg['hidden_dim'])} - memories = {'Memory':ReplayBufferQue(cfg['memory_capacity'])} - agent = DoubleDQN(models,memories,cfg) - return env,agent - - def train(self,cfg,env,agent): - print("Start training!") - print(f"Env: {cfg['env_name']}, Algorithm: {cfg['algo_name']}, Device: {cfg['device']}") - rewards = [] # record rewards for all episodes - steps = [] - for i_ep in range(cfg["train_eps"]): - ep_reward = 0 # reward per episode - ep_step = 0 - state = env.reset() # reset and obtain initial state - for _ in range(cfg['ep_max_steps']): - action = agent.sample_action(state) - next_state, reward, done, _ = env.step(action) - ep_reward += reward - agent.memory.push((state, action, reward, next_state, done)) - state = next_state - agent.update() - if done: - break - if i_ep % cfg['target_update'] == 0: - agent.target_net.load_state_dict(agent.policy_net.state_dict()) - steps.append(ep_step) - rewards.append(ep_reward) - if (i_ep+1)%10 == 0: - print(f'Episode: {i_ep+1}/{cfg["train_eps"]}, Reward: {ep_reward:.2f}: Epislon: {agent.epsilon:.3f}') - print("Finish training!") - env.close() - res_dic = {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - return res_dic - - def test(self,cfg,env,agent): - print("Start testing!") - print(f"Env: {cfg['env_name']}, Algorithm: {cfg['algo_name']}, Device: {cfg['device']}") - rewards = [] # record rewards for all episodes - steps = [] - for i_ep in range(cfg['test_eps']): - ep_reward = 0 # reward per episode - ep_step = 0 - state = env.reset() # reset and obtain initial state - for _ in range(cfg['ep_max_steps']): - action = agent.predict_action(state) - next_state, reward, done, _ = env.step(action) - state = next_state - ep_reward += reward - if done: - break - steps.append(ep_step) - rewards.append(ep_reward) - print(f"Episode: {i_ep+1}/{cfg['test_eps']}，Reward: {ep_reward:.2f}") - print("Finish testing!") - env.close() - return {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - -if __name__ == "__main__": - main = Main() - main.run() diff --git a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/models/checkpoint.pth b/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/models/checkpoint.pth deleted file mode 100644 index d402ba1..0000000 Binary files a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/models/checkpoint.pth and /dev/null differ diff --git a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/results/params.json b/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/results/params.json deleted file mode 100644 index 91df006..0000000 --- a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/results/params.json +++ /dev/null @@ -1 +0,0 @@ -{"algo_name": "DoubleDQN", "env_name": "CartPole-v0", "train_eps": 200, "test_eps": 20, "ep_max_steps": 100000, "gamma": 0.95, "epsilon_start": 0.95, "epsilon_end": 0.01, "epsilon_decay": 500, "lr": 0.0001, "memory_capacity": 100000, "batch_size": 64, "target_update": 4, "hidden_dim": 256, "device": "cpu", "seed": 1, "show_fig": false, "save_fig": true, "result_path": "c:\\Users\\24438\\Desktop\\rl-tutorials\\codes\\DoubleDQN/outputs/CartPole-v0/20220829-233435/results/", "model_path": "c:\\Users\\24438\\Desktop\\rl-tutorials\\codes\\DoubleDQN/outputs/CartPole-v0/20220829-233435/models/", "n_states": 4, "n_actions": 2} \ No newline at end of file diff --git a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/results/testing_curve.png b/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/results/testing_curve.png deleted file mode 100644 index fe21c95..0000000 Binary files a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/results/testing_curve.png and /dev/null differ diff --git a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/results/testing_results.csv b/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/results/testing_results.csv deleted file mode 100644 index 2a504ee..0000000 --- a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/results/testing_results.csv +++ /dev/null @@ -1,21 +0,0 @@ -episodes,rewards,steps -0,145.0,0 -1,166.0,0 -2,171.0,0 -3,200.0,0 -4,139.0,0 -5,200.0,0 -6,200.0,0 -7,141.0,0 -8,200.0,0 -9,187.0,0 -10,166.0,0 -11,172.0,0 -12,121.0,0 -13,200.0,0 -14,200.0,0 -15,149.0,0 -16,128.0,0 -17,200.0,0 -18,178.0,0 -19,185.0,0 diff --git a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/results/training_curve.png b/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/results/training_curve.png deleted file mode 100644 index a8475ea..0000000 Binary files a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/results/training_curve.png and /dev/null differ diff --git a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/results/training_results.csv b/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/results/training_results.csv deleted file mode 100644 index 8f87049..0000000 --- a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233435/results/training_results.csv +++ /dev/null @@ -1,201 +0,0 @@ -episodes,rewards,steps -0,19.0,0 -1,16.0,0 -2,17.0,0 -3,11.0,0 -4,10.0,0 -5,27.0,0 -6,16.0,0 -7,9.0,0 -8,20.0,0 -9,21.0,0 -10,15.0,0 -11,10.0,0 -12,14.0,0 -13,37.0,0 -14,12.0,0 -15,10.0,0 -16,27.0,0 -17,33.0,0 -18,19.0,0 -19,13.0,0 -20,26.0,0 -21,15.0,0 -22,29.0,0 -23,11.0,0 -24,20.0,0 -25,23.0,0 -26,23.0,0 -27,26.0,0 -28,17.0,0 -29,33.0,0 -30,16.0,0 -31,48.0,0 -32,48.0,0 -33,69.0,0 -34,58.0,0 -35,24.0,0 -36,18.0,0 -37,28.0,0 -38,12.0,0 -39,12.0,0 -40,18.0,0 -41,12.0,0 -42,13.0,0 -43,21.0,0 -44,30.0,0 -45,32.0,0 -46,22.0,0 -47,18.0,0 -48,12.0,0 -49,12.0,0 -50,20.0,0 -51,32.0,0 -52,15.0,0 -53,100.0,0 -54,26.0,0 -55,25.0,0 -56,18.0,0 -57,15.0,0 -58,35.0,0 -59,12.0,0 -60,65.0,0 -61,27.0,0 -62,29.0,0 -63,22.0,0 -64,83.0,0 -65,24.0,0 -66,28.0,0 -67,15.0,0 -68,43.0,0 -69,13.0,0 -70,22.0,0 -71,46.0,0 -72,14.0,0 -73,32.0,0 -74,44.0,0 -75,53.0,0 -76,31.0,0 -77,51.0,0 -78,61.0,0 -79,30.0,0 -80,36.0,0 -81,30.0,0 -82,48.0,0 -83,26.0,0 -84,27.0,0 -85,43.0,0 -86,20.0,0 -87,87.0,0 -88,71.0,0 -89,43.0,0 -90,57.0,0 -91,40.0,0 -92,37.0,0 -93,43.0,0 -94,31.0,0 -95,45.0,0 -96,47.0,0 -97,52.0,0 -98,48.0,0 -99,98.0,0 -100,49.0,0 -101,98.0,0 -102,68.0,0 -103,70.0,0 -104,74.0,0 -105,73.0,0 -106,127.0,0 -107,92.0,0 -108,70.0,0 -109,97.0,0 -110,66.0,0 -111,112.0,0 -112,138.0,0 -113,81.0,0 -114,74.0,0 -115,153.0,0 -116,113.0,0 -117,88.0,0 -118,138.0,0 -119,200.0,0 -120,84.0,0 -121,123.0,0 -122,158.0,0 -123,171.0,0 -124,137.0,0 -125,143.0,0 -126,170.0,0 -127,127.0,0 -128,118.0,0 -129,200.0,0 -130,189.0,0 -131,149.0,0 -132,137.0,0 -133,115.0,0 -134,153.0,0 -135,136.0,0 -136,140.0,0 -137,169.0,0 -138,187.0,0 -139,200.0,0 -140,196.0,0 -141,200.0,0 -142,200.0,0 -143,137.0,0 -144,200.0,0 -145,185.0,0 -146,200.0,0 -147,164.0,0 -148,200.0,0 -149,143.0,0 -150,143.0,0 -151,112.0,0 -152,192.0,0 -153,200.0,0 -154,144.0,0 -155,188.0,0 -156,200.0,0 -157,133.0,0 -158,200.0,0 -159,143.0,0 -160,158.0,0 -161,161.0,0 -162,169.0,0 -163,176.0,0 -164,200.0,0 -165,149.0,0 -166,156.0,0 -167,200.0,0 -168,200.0,0 -169,200.0,0 -170,134.0,0 -171,171.0,0 -172,200.0,0 -173,200.0,0 -174,200.0,0 -175,194.0,0 -176,200.0,0 -177,138.0,0 -178,159.0,0 -179,187.0,0 -180,200.0,0 -181,192.0,0 -182,200.0,0 -183,200.0,0 -184,200.0,0 -185,173.0,0 -186,200.0,0 -187,178.0,0 -188,176.0,0 -189,196.0,0 -190,200.0,0 -191,195.0,0 -192,158.0,0 -193,156.0,0 -194,200.0,0 -195,200.0,0 -196,200.0,0 -197,200.0,0 -198,193.0,0 -199,200.0,0 diff --git a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/models/checkpoint.pth b/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/models/checkpoint.pth deleted file mode 100644 index 01e8c46..0000000 Binary files a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/models/checkpoint.pth and /dev/null differ diff --git a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/results/params.json b/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/results/params.json deleted file mode 100644 index 2d2c2ca..0000000 --- a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/results/params.json +++ /dev/null @@ -1 +0,0 @@ -{"algo_name": "DoubleDQN", "env_name": "CartPole-v0", "train_eps": 200, "test_eps": 20, "ep_max_steps": 100000, "gamma": 0.95, "epsilon_start": 0.95, "epsilon_end": 0.01, "epsilon_decay": 500, "lr": 0.0001, "memory_capacity": 100000, "batch_size": 64, "target_update": 4, "hidden_dim": 256, "device": "cuda", "seed": 1, "show_fig": false, "save_fig": true, "result_path": "C:\\Users\\24438\\Desktop\\rl-tutorials\\codes\\DoubleDQN/outputs/CartPole-v0/20220829-233635/results/", "model_path": "C:\\Users\\24438\\Desktop\\rl-tutorials\\codes\\DoubleDQN/outputs/CartPole-v0/20220829-233635/models/", "n_states": 4, "n_actions": 2} \ No newline at end of file diff --git a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/results/testing_curve.png b/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/results/testing_curve.png deleted file mode 100644 index 288ee92..0000000 Binary files a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/results/testing_curve.png and /dev/null differ diff --git a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/results/testing_results.csv b/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/results/testing_results.csv deleted file mode 100644 index 6e8adb7..0000000 --- a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/results/testing_results.csv +++ /dev/null @@ -1,21 +0,0 @@ -episodes,rewards,steps -0,200.0,0 -1,200.0,0 -2,200.0,0 -3,200.0,0 -4,191.0,0 -5,200.0,0 -6,200.0,0 -7,179.0,0 -8,200.0,0 -9,200.0,0 -10,200.0,0 -11,190.0,0 -12,147.0,0 -13,197.0,0 -14,200.0,0 -15,200.0,0 -16,167.0,0 -17,200.0,0 -18,200.0,0 -19,200.0,0 diff --git a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/results/training_curve.png b/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/results/training_curve.png deleted file mode 100644 index 544de6e..0000000 Binary files a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/results/training_curve.png and /dev/null differ diff --git a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/results/training_results.csv b/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/results/training_results.csv deleted file mode 100644 index 67bdb9e..0000000 --- a/projects/codes/DoubleDQN/outputs/CartPole-v0/20220829-233635/results/training_results.csv +++ /dev/null @@ -1,201 +0,0 @@ -episodes,rewards,steps -0,19.0,0 -1,16.0,0 -2,17.0,0 -3,11.0,0 -4,10.0,0 -5,27.0,0 -6,55.0,0 -7,17.0,0 -8,23.0,0 -9,9.0,0 -10,17.0,0 -11,14.0,0 -12,17.0,0 -13,12.0,0 -14,14.0,0 -15,16.0,0 -16,27.0,0 -17,36.0,0 -18,17.0,0 -19,17.0,0 -20,21.0,0 -21,23.0,0 -22,13.0,0 -23,12.0,0 -24,17.0,0 -25,26.0,0 -26,25.0,0 -27,17.0,0 -28,10.0,0 -29,16.0,0 -30,14.0,0 -31,19.0,0 -32,23.0,0 -33,37.0,0 -34,29.0,0 -35,22.0,0 -36,29.0,0 -37,15.0,0 -38,16.0,0 -39,18.0,0 -40,23.0,0 -41,16.0,0 -42,26.0,0 -43,13.0,0 -44,24.0,0 -45,39.0,0 -46,23.0,0 -47,32.0,0 -48,123.0,0 -49,18.0,0 -50,39.0,0 -51,17.0,0 -52,28.0,0 -53,34.0,0 -54,26.0,0 -55,61.0,0 -56,28.0,0 -57,16.0,0 -58,45.0,0 -59,41.0,0 -60,49.0,0 -61,18.0,0 -62,40.0,0 -63,24.0,0 -64,37.0,0 -65,26.0,0 -66,51.0,0 -67,17.0,0 -68,152.0,0 -69,17.0,0 -70,29.0,0 -71,37.0,0 -72,15.0,0 -73,55.0,0 -74,152.0,0 -75,23.0,0 -76,45.0,0 -77,30.0,0 -78,39.0,0 -79,20.0,0 -80,53.0,0 -81,49.0,0 -82,71.0,0 -83,115.0,0 -84,41.0,0 -85,52.0,0 -86,52.0,0 -87,36.0,0 -88,84.0,0 -89,122.0,0 -90,49.0,0 -91,200.0,0 -92,67.0,0 -93,87.0,0 -94,183.0,0 -95,132.0,0 -96,76.0,0 -97,200.0,0 -98,200.0,0 -99,200.0,0 -100,200.0,0 -101,200.0,0 -102,106.0,0 -103,192.0,0 -104,111.0,0 -105,95.0,0 -106,200.0,0 -107,200.0,0 -108,148.0,0 -109,200.0,0 -110,97.0,0 -111,200.0,0 -112,200.0,0 -113,105.0,0 -114,135.0,0 -115,200.0,0 -116,144.0,0 -117,156.0,0 -118,200.0,0 -119,200.0,0 -120,166.0,0 -121,200.0,0 -122,200.0,0 -123,200.0,0 -124,200.0,0 -125,200.0,0 -126,200.0,0 -127,158.0,0 -128,139.0,0 -129,200.0,0 -130,200.0,0 -131,200.0,0 -132,200.0,0 -133,122.0,0 -134,200.0,0 -135,188.0,0 -136,200.0,0 -137,183.0,0 -138,200.0,0 -139,200.0,0 -140,200.0,0 -141,200.0,0 -142,200.0,0 -143,158.0,0 -144,200.0,0 -145,200.0,0 -146,200.0,0 -147,191.0,0 -148,200.0,0 -149,194.0,0 -150,178.0,0 -151,200.0,0 -152,200.0,0 -153,200.0,0 -154,162.0,0 -155,200.0,0 -156,200.0,0 -157,128.0,0 -158,200.0,0 -159,184.0,0 -160,194.0,0 -161,200.0,0 -162,200.0,0 -163,200.0,0 -164,200.0,0 -165,160.0,0 -166,163.0,0 -167,200.0,0 -168,200.0,0 -169,200.0,0 -170,141.0,0 -171,200.0,0 -172,200.0,0 -173,200.0,0 -174,200.0,0 -175,200.0,0 -176,200.0,0 -177,157.0,0 -178,164.0,0 -179,200.0,0 -180,200.0,0 -181,200.0,0 -182,200.0,0 -183,200.0,0 -184,200.0,0 -185,193.0,0 -186,182.0,0 -187,200.0,0 -188,200.0,0 -189,200.0,0 -190,200.0,0 -191,200.0,0 -192,174.0,0 -193,178.0,0 -194,200.0,0 -195,200.0,0 -196,200.0,0 -197,200.0,0 -198,200.0,0 -199,200.0,0 diff --git a/projects/codes/DuelingDQN/assets/task0_train_20211112021954.png b/projects/codes/DuelingDQN/assets/task0_train_20211112021954.png deleted file mode 100644 index 2529311..0000000 Binary files a/projects/codes/DuelingDQN/assets/task0_train_20211112021954.png and /dev/null differ diff --git a/projects/codes/DuelingDQN/task0_train.ipynb b/projects/codes/DuelingDQN/task0_train.ipynb deleted file mode 100644 index efa485f..0000000 --- a/projects/codes/DuelingDQN/task0_train.ipynb +++ /dev/null @@ -1,418 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "import math, random\n", - "import gym\n", - "import numpy as np\n", - "import torch\n", - "import torch.nn as nn\n", - "import torch.optim as optim\n", - "import torch.autograd as autograd \n", - "import torch.nn.functional as F\n", - "from IPython.display import clear_output # 清空单元格输出区域\n", - "import matplotlib.pyplot as plt\n", - "# %matplotlib inline\n" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [], - "source": [ - "USE_CUDA = torch.cuda.is_available()\n", - "Variable = lambda *args, **kwargs: autograd.Variable(*args, **kwargs).cuda() if USE_CUDA else autograd.Variable(*args, **kwargs)" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "from collections import deque\n", - "\n", - "class ReplayBuffer(object):\n", - " def __init__(self, capacity):\n", - " self.buffer = deque(maxlen=capacity)\n", - " \n", - " def push(self, state, action, reward, next_state, done):\n", - " state = np.expand_dims(state, 0)\n", - " next_state = np.expand_dims(next_state, 0)\n", - " \n", - " self.buffer.append((state, action, reward, next_state, done))\n", - " \n", - " def sample(self, batch_size):\n", - " state, action, reward, next_state, done = zip(*random.sample(self.buffer, batch_size))\n", - " return np.concatenate(state), action, reward, np.concatenate(next_state), done\n", - " \n", - " def __len__(self):\n", - " return len(self.buffer)" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "env_name = \"CartPole-v0\"\n", - "env = gym.make(env_name)" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "epsilon_start = 1.0\n", - "epsilon_final = 0.01\n", - "epsilon_decay = 500\n", - "\n", - "epsilon_by_frame = lambda frame_idx: epsilon_final + (epsilon_start - epsilon_final) * math.exp(-1. * frame_idx / epsilon_decay)" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "[]" - ] - }, - "execution_count": 6, - "metadata": {}, - "output_type": "execute_result" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZ9UlEQVR4nO3dfXQd9X3n8ff3XulK1pP1aNmWbWQb24lJAgaFmIeT0IQQYFvc3W0b3GVDSFL6RFua7u6Bk57Qsv80TdvdZuOS0DakTSgOoWniEojbJpA2FFMLHMAPGGSDbQkbyY+yLcuypO/+MSNzLWTr2rrSaGY+r3Pu0cxvRvd+RyN/PPrNb2bM3RERkfjLRF2AiIgUhwJdRCQhFOgiIgmhQBcRSQgFuohIQpRE9cGNjY3e2toa1ceLiMTS888/v9/dm8ZaFlmgt7a20t7eHtXHi4jEkpntOtsydbmIiCSEAl1EJCEU6CIiCaFAFxFJCAW6iEhCjBvoZvY1M+s2s81nWW5m9iUz6zCzl8zs8uKXKSIi4ynkCP3rwI3nWH4TsCR83Qk8MPGyRETkfI0b6O7+r8DBc6yyCvhbD2wAas1sTrEKHG3jGwf5wg9eQbf9FRE5UzH60FuAPXnznWHbO5jZnWbWbmbtPT09F/RhL+45zANP76D3xOAFfb+ISFJN6UlRd3/Q3dvcva2pacwrV8fVUJUD4MDxk8UsTUQk9ooR6F3A/Lz5eWHbpKivLAPg4PGByfoIEZFYKkagrwM+EY52WQkccfe9RXjfMTVUjhyhK9BFRPKNe3MuM3sEuA5oNLNO4D6gFMDdvwI8AdwMdAB9wB2TVSxAfRjoOkIXETnTuIHu7qvHWe7AbxatonEo0EVExha7K0XLS7NU5rIcOKZAFxHJF7tAB6irzHFQo1xERM4Qy0BvqMzppKiIyCixDPT6ypz60EVERolpoJcp0EVERolloDdUBUfoup+LiMjbYhno9ZU5Tg4O0zcwFHUpIiLTRmwDHTQWXUQkXywDXZf/i4i8UywD/e0jdI1FFxEZEctAbwjvuKirRUVE3hbLQK+vUh+6iMhosQz0ylyWXElGgS4ikieWgW5muvxfRGSUWAY66PJ/EZHRYh3oOkIXEXlbbAO9QbfQFRE5Q3wDvapMwxZFRPLENtAbq8roGxiib2Aw6lJERKaF2AZ6U3VwcdH+ozpKFxGBBAR6z7H+iCsREZkeYhvojeHVoj1HdWJURARiHOinj9AV6CIiQIwDvaGyjIxBj0a6iIgAMQ70bMaoryzTEbqISCi2gQ5BP7oCXUQkEOtAb6ouo+eYAl1EBBIQ6Pt1hC4iAsQ90KuCI3R3j7oUEZHIxTvQq8sYGBymt1+X/4uIxD7QAfarH11EpLBAN7MbzWy7mXWY2T1jLF9gZk+Z2SYze8nMbi5+qe/UVKWLi0RERowb6GaWBdYANwHLgdVmtnzUar8PPOruK4Bbgb8odqFjadTVoiIipxVyhH4l0OHuO919AFgLrBq1jgM14fRM4M3ilXh2OkIXEXlbIYHeAuzJm+8M2/L9AXCbmXUCTwC/NdYbmdmdZtZuZu09PT0XUO6ZZs4opTRr6kMXEaF4J0VXA19393nAzcA3zOwd7+3uD7p7m7u3NTU1TfhDMxmjQZf/i4gAhQV6FzA/b35e2Jbv08CjAO7+LFAONBajwPHoalERkUAhgb4RWGJmC80sR3DSc92odXYDHwEws3cTBPrE+1QK0FRdRnevAl1EZNxAd/dB4C5gPbCNYDTLFjO738xuCVf7PeBXzOxF4BHgkz5Fl28215TRrS4XERFKClnJ3Z8gONmZ3/b5vOmtwDXFLa0wzTXlHDh+klNDw5RmY32dlIjIhMQ+AZtrynFHR+kiknqxD/TZNeUA7Duih0WLSLrFPtCbw0B/q1eBLiLpFvtAnz1TR+giIpCAQK+rKCVXktERuoikXuwD3cxoriljnwJdRFIu9oEOwYlRHaGLSNolItCba8p5S1eLikjKJSLQZ9eUs+9Iv54tKiKplohAb64p58SpIT1bVERSLRmBPlNj0UVEEhHoulpURCRpga4jdBFJsUQE+qya4Nmi3Qp0EUmxRAR6eWmWuopSHaGLSKolItAhGOmiPnQRSbPEBHpL7QzePKxAF5H0Skygz62dQdfhE1GXISISmUQF+pETpzh2UhcXiUg6JSbQW+pmALBXR+kiklLJCfTaYCx6pwJdRFIqMYE+tzY4Qn9TgS4iKZWYQJ9VXU5Jxug6pEAXkXRKTKBnM8bsmeU6QheR1EpMoIPGootIuiUu0DUWXUTSKlGBPrd2Bvt6+xkcGo66FBGRKZeoQG+pm8HQsPPWUT1fVETSJ1GBrqGLIpJmiQr0kYuLNHRRRNIoUYE+coSuE6MikkYFBbqZ3Whm282sw8zuOcs6v2RmW81si5n9XXHLLExFroS6ilI6dYQuIilUMt4KZpYF1gAfBTqBjWa2zt235q2zBLgXuMbdD5nZrMkqeDwL6ivoPNQX1ceLiESmkCP0K4EOd9/p7gPAWmDVqHV+BVjj7ocA3L27uGUWbn59BbsPKtBFJH0KCfQWYE/efGfYlm8psNTMnjGzDWZ241hvZGZ3mlm7mbX39PRcWMXjuKihgq5DJzQWXURSp1gnRUuAJcB1wGrgL82sdvRK7v6gu7e5e1tTU1ORPvpMC+orGBx29ur5oiKSMoUEehcwP29+XtiWrxNY5+6n3P114FWCgJ9yC+orAdh1QN0uIpIuhQT6RmCJmS00sxxwK7Bu1DrfJTg6x8waCbpgdhavzMItaKgAUD+6iKTOuIHu7oPAXcB6YBvwqLtvMbP7zeyWcLX1wAEz2wo8BfxPdz8wWUWfy+yacnLZDLsOHo/i40VEIjPusEUAd38CeGJU2+fzph34bPiKVDZjzKubwR4doYtIyiTqStERCxo0dFFE0ieZgV5fwa4DfQR/OIiIpENiA/1o/yBHTpyKuhQRkSmT2EAHDV0UkXRJZqBr6KKIpFAyA/30EbqGLopIeiQy0CtyJcyuKWfnfgW6iKRHIgMdYFFTJTt7FOgikh6JDfSFjZXs7DmmoYsikhqJDfRFTVX09g9y4PhA1KWIiEyJBAd6cNdFdbuISFokNtAXN1YB8Pr+YxFXIiIyNRIb6C11M8iVZHSELiKpkdhAz2aM1oYKdijQRSQlEhvoAIsaq9ipLhcRSYlkB3pTJbsP9HFKD4wWkRRIeKBXMTjsetiFiKRCwgNdQxdFJD0SHeiLm4Khi691qx9dRJIv0YE+c0Ypc2aW8+pbR6MuRURk0iU60AGWNlezfZ8CXUSSL/GBvmx2NR09xxjUSBcRSbjkB3pzNQODw+zSSBcRSbjkB/rsagB1u4hI4iU+0C+eVYWZAl1Eki/xgV5emqW1oVIjXUQk8RIf6ABLm6vYrkAXkYRLRaAva67mjf3H6T81FHUpIiKTJhWBvnR2NcMOHbpiVEQSLBWB/u45NQBs29sbcSUiIpMnFYG+sKGSylyWLW8q0EUkuQoKdDO70cy2m1mHmd1zjvX+q5m5mbUVr8SJy2SM5XNr2Nx1JOpSREQmzbiBbmZZYA1wE7AcWG1my8dYrxr4HeC5YhdZDJfMncnWvb0MDXvUpYiITIpCjtCvBDrcfae7DwBrgVVjrPe/gS8A/UWsr2je0zKTvoEhXt+ve6OLSDIVEugtwJ68+c6w7TQzuxyY7+7fP9cbmdmdZtZuZu09PT3nXexEvKclODG65U11u4hIMk34pKiZZYA/A35vvHXd/UF3b3P3tqampol+9HlZ3FRFriSjfnQRSaxCAr0LmJ83Py9sG1ENvAd42szeAFYC66bbidHSbIZ3z65mc5dGuohIMhUS6BuBJWa20MxywK3AupGF7n7E3RvdvdXdW4ENwC3u3j4pFU/AJS0z2fzmEdx1YlREkmfcQHf3QeAuYD2wDXjU3beY2f1mdstkF1hM75k7k6P9g+zWvdFFJIFKClnJ3Z8AnhjV9vmzrHvdxMuaHJfNrwXgp3sOc1FDZbTFiIgUWSquFB2xtLmKilyWF3YdiroUEZGiS1Wgl2QzXDqvlhd2H466FBGRoktVoAOsWFDLtr29nBjQrXRFJFlSF+iXL6hjcNh5WePRRSRhUhfoly2oBWDTbvWji0iypC7QG6vKuKihghcU6CKSMKkLdIAV84MTo7rASESSJJWBfsVFdfQcPcmegyeiLkVEpGhSGegrFzUAsGHngYgrEREpnlQG+sWzqmisyvGsAl1EEiSVgW5mfGBRAxt2HlA/uogkRioDHYJul71H+nWjLhFJjNQG+lWL6gH1o4tIcqQ20Bc3Bf3oG3YejLoUEZGiSG2gj/SjP7tD/egikgypDXSAay9uZF9vP691H4u6FBGRCUt1oH9oafCg6h9v74m4EhGRiUt1oM+tncHS5iqefrU76lJERCYs1YEOcN2yWWx8/RDHTw5GXYqIyISkPtA/tLSJgaFhnt2h4YsiEm+pD/S21joqcll1u4hI7KU+0MtKsly9uIGnXunR8EURibXUBzrADctn03X4BFve7I26FBGRC6ZAB65f3kzG4Aeb90VdiojIBVOgA/WVOT6wsIEnN++NuhQRkQumQA/d9N7Z7Og5Tkf30ahLERG5IAr00A3LZwPw5MvqdhGReFKgh2bPLGfFglq+/7K6XUQknhToeX7+shZe2XeUbXs12kVE4keBnufnLp1LScb4h01dUZciInLeFOh56itzXLdsFt/d1MXQsC4yEpF4KSjQzexGM9tuZh1mds8Yyz9rZlvN7CUz+6GZXVT8UqfGf7m8he6jJ3mmY3/UpYiInJdxA93MssAa4CZgObDazJaPWm0T0Obu7wMeA/642IVOlQ+/axY15SU89nxn1KWIiJyXQo7QrwQ63H2nuw8Aa4FV+Su4+1Pu3hfObgDmFbfMqVNemuU/r2jhB5v3ceDYyajLEREpWCGB3gLsyZvvDNvO5tPAk2MtMLM7zazdzNp7eqbvU4JuW3kRA0PDPNquo3QRiY+inhQ1s9uANuCLYy139wfdvc3d25qamor50UW1pLmalYvqefi5XTo5KiKxUUigdwHz8+bnhW1nMLPrgc8Bt7h77Psq/vvKVjoPneDHuk+6iMREIYG+EVhiZgvNLAfcCqzLX8HMVgBfJQjzRCTgDZc0M6u6jIeeeSPqUkRECjJuoLv7IHAXsB7YBjzq7lvM7H4zuyVc7YtAFfBtM/upma07y9vFRmk2w6euXci/vbaflzuPRF2OiMi4LKqn9LS1tXl7e3skn12oo/2nuPqPfsS1FzfywG1XRF2OiAhm9ry7t421TFeKnkN1eSmfuOoifrBlHx3dx6IuR0TknBTo47jjmoXkshn+4umOqEsRETknBfo4GqvKuP3qVv5hUxfb9+nhFyIyfSnQC/Ab1y2mqqyEL65/JepSRETOSoFegNqKHL/2ocX8y7ZuNr5xMOpyRETGpEAv0KeuWUhzTRl/+I9bdPWoiExLCvQCzchl+f3/tJzNXb18c8OuqMsREXkHBfp5+Nn3zeHaixv5k/Xb6e7tj7ocEZEzKNDPg5lx/6pLODk4zH3rthDVRVkiImNRoJ+nRU1V3P3RJTy5eR/feUHPHhWR6UOBfgF+9YOLubK1nvvWbWHPwb7xv0FEZAoo0C9ANmP86S9dCsBvr93EycGhiCsSEVGgX7D59RV88Rfex6bdh7nve+pPF5HoKdAn4Kb3zuE3f2Yxazfu4ZvP7Y66HBFJuZKoC4i7z350Gdv2HuW+721mVnUZH7tkdtQliUhK6Qh9grIZ48u/vIJL59fyW49s4t937I+6JBFJKQV6EVTkSnjok++ntaGCz/xNO890KNRFZOop0IuktiLHNz/zAebXVXDHQxv5py37oi5JRFJGgV5Es6rL+davrmT53Bp+/eEX+NpPXtfoFxGZMgr0IqutyPHwZz7AR941i/sf38r/+PZL9J/SOHURmXwK9ElQWVbCV267gruvX8Lfv9DJqi8/w5Y3j0RdlogknAJ9kmQyxt3XL+WhO97Pwb4Bfn7NM/y/H76mq0pFZNIo0CfZzyybxT/d/UFuuGQ2f/rPr/Kx//Ov/OiVt6IuS0QSSIE+Beoqc6z55cv5+h3vJ5MxPvX1dlY/uIFndxyIujQRSRCLahRGW1ubt7e3R/LZURoYHOabG3bxwI930HP0JFe21nPHNa1cv7yZ0qz+fxWRczOz5929bcxlCvRo9J8aYu1/7OYv/+11ug6fYFZ1GR9//3xWXTaXi2dVR12eiExTCvRpbGjYeeqVbh5+bhdPv9qDOyxrrubm987hw++axSVza8hkLOoyRWSaUKDHxFu9/Tz58l6+//Je2ncdwh3qKkq5enEjVy1u4LL5tSybXa2uGZEUU6DHUPfRfv694wA/6djPT17bz77wodS5kgyXzK3hvS0zuXhWFYubgldzTRlmOpIXSToFesy5O3sOnuDFzsO81HmYF/ccYeveXo6dHDy9TlVZCfPqZjC3dgZza8uZMzP42lxdTl1ljvrKHLUVpZSVZCPcEhGZqHMFuu6HHgNmxoKGChY0VPBzl84FgpDvPnqSHd3H6Og5xo7uY3QeOsGbR/p5YfchDvedGvO9qspKqKsspa4iR2WuhMqyEirLssHX3MjXEirKspSVZCnNGmUlGXIlGXLZYD4XzpeVZCjNBq9sxsiYkc0YWTMyGU63jbRnDP0VITKJCgp0M7sR+HMgC/yVu//RqOVlwN8CVwAHgI+7+xvFLVXymRnNNeU015Rz9cWN71jeNzDI3iP9vNXbz+G+Uxw8PsCh4wMc6jvFob4BDh4f4PjJQboOn6BvYJDjJwc5fnKIE5N835mM8Y7wtzDoR7Lewu0bif6g3U5P57fbmO2W933nXm/a/fcyzQqaZuVMuwOCC63mtz+y5PTBWTGNG+hmlgXWAB8FOoGNZrbO3bfmrfZp4JC7X2xmtwJfAD5e9GqlYBW5ktP96+djaNjpGxikb2CIgcFhTg4Oc2pomIHBYQZGvo5qPzU0zJA7w8PO0LAz5ATTHsy7O0PDvL3OGes67py+K6VDME847zDSKRisktceLnA8b/rM7+eM7/cz3mu63Qdzut2Zc3pVw7QryCdQ0MwZpUWs5G2FHKFfCXS4+04AM1sLrALyA30V8Afh9GPAl83MfLr9hsq4shmjuryU6vLJ+YUTkclTyPi3FmBP3nxn2DbmOu4+CBwBGka/kZndaWbtZtbe09NzYRWLiMiYpnRAs7s/6O5t7t7W1NQ0lR8tIpJ4hQR6FzA/b35e2DbmOmZWAswkODkqIiJTpJBA3wgsMbOFZpYDbgXWjVpnHXB7OP0LwI/Ufy4iMrXGPSnq7oNmdhewnmDY4tfcfYuZ3Q+0u/s64K+Bb5hZB3CQIPRFRGQKFTQO3d2fAJ4Y1fb5vOl+4BeLW5qIiJwP3eVJRCQhFOgiIgkR2c25zKwH2HWB394I7C9iOXGgbU4HbXM6TGSbL3L3Mcd9RxboE2Fm7We721hSaZvTQducDpO1zepyERFJCAW6iEhCxDXQH4y6gAhom9NB25wOk7LNsexDFxGRd4rrEbqIiIyiQBcRSYjYBbqZ3Whm282sw8zuibqeC2Vm883sKTPbamZbzOx3wvZ6M/tnM3st/FoXtpuZfSnc7pfM7PK897o9XP81M7v9bJ85XZhZ1sw2mdnj4fxCM3su3LZvhTeBw8zKwvmOcHlr3nvcG7ZvN7OPRbQpBTGzWjN7zMxeMbNtZnZV0vezmf1u+Hu92cweMbPypO1nM/uamXWb2ea8tqLtVzO7wsxeDr/nS2YFPH/P3WPzIrg52A5gEZADXgSWR13XBW7LHODycLoaeBVYDvwxcE/Yfg/whXD6ZuBJgscYrgSeC9vrgZ3h17pwui7q7Rtn2z8L/B3weDj/KHBrOP0V4NfD6d8AvhJO3wp8K5xeHu77MmBh+DuRjXq7zrG9fwN8JpzOAbVJ3s8ED7x5HZiRt38/mbT9DHwQuBzYnNdWtP0K/Ee4roXfe9O4NUX9QznPH+BVwPq8+XuBe6Ouq0jb9j2C57ZuB+aEbXOA7eH0V4HVeetvD5evBr6a137GetPtRXA//R8CHwYeD39Z9wMlo/cxwR0+rwqnS8L1bPR+z19vur0Ing3wOuEAhNH7L4n7mbefYFYf7rfHgY8lcT8DraMCvSj7NVz2Sl77Geud7RW3LpdCHocXO+GfmCuA54Bmd98bLtoHNIfTZ9v2uP1M/i/wv4DhcL4BOOzBowvhzPrP9mjDOG3zQqAHeCjsZvorM6skwfvZ3buAPwF2A3sJ9tvzJHs/jyjWfm0Jp0e3n1PcAj1xzKwK+HvgbnfvzV/mwX/NiRlXamY/C3S7+/NR1zKFSgj+LH/A3VcAxwn+FD8tgfu5juDB8QuBuUAlcGOkRUUgiv0at0Av5HF4sWFmpQRh/rC7fydsfsvM5oTL5wDdYfvZtj1OP5NrgFvM7A1gLUG3y58DtRY8uhDOrP9sjzaM0zZ3Ap3u/lw4/xhBwCd5P18PvO7uPe5+CvgOwb5P8n4eUaz92hVOj24/p7gFeiGPw4uF8Iz1XwPb3P3P8hblP87vdoK+9ZH2T4Rny1cCR8I/7dYDN5hZXXhkdEPYNu24+73uPs/dWwn23Y/c/b8BTxE8uhDeuc1jPdpwHXBrODpiIbCE4ATStOPu+4A9ZrYsbPoIsJUE72eCrpaVZlYR/p6PbHNi93OeouzXcFmvma0Mf4afyHuvs4v6pMIFnIS4mWBEyA7gc1HXM4HtuJbgz7GXgJ+Gr5sJ+g5/CLwG/AtQH65vwJpwu18G2vLe61NAR/i6I+ptK3D7r+PtUS6LCP6hdgDfBsrC9vJwviNcvijv+z8X/iy2U8DZ/4i39TKgPdzX3yUYzZDo/Qz8IfAKsBn4BsFIlUTtZ+ARgnMEpwj+Evt0Mfcr0Bb+/HYAX2bUifWxXrr0X0QkIeLW5SIiImehQBcRSQgFuohIQijQRUQSQoEuIpIQCnQRkYRQoIuIJMT/B3B4SePGsjO/AAAAAElFTkSuQmCC", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "plt.plot([epsilon_by_frame(i) for i in range(10000)])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Dueling DQN 网络\n", - "\n", - "DQN等算法中使用的是一个简单的三层神经网络：一个输入层，一个隐藏层和一个输出层。如下左图：\n", - "\n", - "\"image-20211112022028670\"\n", - "\n", - "而在Dueling DQN中，我们在后面加了两个子网络结构，分别对应上面上到价格函数网络部分和优势函数网络部分。对应上面右图所示。最终Q网络的输出由价格函数网络的输出和优势函数网络的输出线性组合得到。\n", - "\n", - "我们可以直接使用上一节的价值函数的组合公式得到我们的动作价值，但是这个式子无法辨识最终输出里面$V(S, w, \\alpha)$和$A(S, A, w, \\beta)$各自的作用，为了可以体现这种可辨识性(identifiability),实际使用的组合公式如下：\n", - "\n", - "$$\n", - "Q(S, A, w, \\alpha, \\beta)=V(S, w, \\alpha)+\\left(A(S, A, w, \\beta)-\\frac{1}{\\mathcal{A}} \\sum_{a^{\\prime} \\in \\mathcal{A}} A\\left(S, a^{\\prime}, w, \\beta\\right)\\right)\n", - "$$" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [], - "source": [ - "class DuelingNet(nn.Module):\n", - " def __init__(self, n_states, n_actions,hidden_size=128):\n", - " super(DuelingNet, self).__init__()\n", - " \n", - " # 隐藏层\n", - " self.hidden = nn.Sequential(\n", - " nn.Linear(n_states, hidden_size),\n", - " nn.ReLU()\n", - " )\n", - " \n", - " # 优势函数\n", - " self.advantage = nn.Sequential(\n", - " nn.Linear(hidden_size, hidden_size),\n", - " nn.ReLU(),\n", - " nn.Linear(hidden_size, n_actions)\n", - " )\n", - " \n", - " # 价值函数\n", - " self.value = nn.Sequential(\n", - " nn.Linear(hidden_size, hidden_size),\n", - " nn.ReLU(),\n", - " nn.Linear(hidden_size, 1)\n", - " )\n", - " \n", - " def forward(self, x):\n", - " x = self.hidden(x)\n", - " advantage = self.advantage(x)\n", - " value = self.value(x)\n", - " return value + advantage - advantage.mean()\n", - " \n", - " def act(self, state, epsilon):\n", - " if random.random() > epsilon:\n", - " with torch.no_grad():\n", - " state = Variable(torch.FloatTensor(state).unsqueeze(0))\n", - " q_value = self.forward(state)\n", - " action = q_value.max(1)[1].item()\n", - " else:\n", - " action = random.randrange(env.action_space.n)\n", - " return action" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "metadata": {}, - "outputs": [ - { - "ename": "SyntaxError", - "evalue": "unexpected EOF while parsing (, line 1)", - "output_type": "error", - "traceback": [ - "\u001b[0;36m File \u001b[0;32m\"\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m class DuelingDQN:\u001b[0m\n\u001b[0m ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m unexpected EOF while parsing\n" - ] - } - ], - "source": [ - "class DuelingDQN:\n", - " def __init__(self,n_states,n_actions,cfg) -> None:\n", - " self.batch_size = cfg.batch_size\n", - " self.device = cfg.device\n", - " self.loss_history = [] # 记录loss的变化\n", - " self.frame_idx = 0 # 用于epsilon的衰减计数\n", - " self.epsilon = lambda frame_idx: cfg.epsilon_end + \\\n", - " (cfg.epsilon_start - cfg.epsilon_end) * \\\n", - " math.exp(-1. * frame_idx / cfg.epsilon_decay)\n", - " self.policy_net = DuelingNet(n_states, n_actions,hidden_dim=cfg.hidden_dim).to(self.device)\n", - " self.target_net = DuelingNet(n_states, n_actions,hidden_dim=cfg.hidden_dim).to(self.device)\n", - " for target_param, param in zip(self.target_net.parameters(),self.policy_net.parameters()): # 复制参数到目标网络targe_net\n", - " target_param.data.copy_(param.data)\n", - " self.optimizer = optim.Adam(self.policy_net.parameters(), lr=cfg.lr) # 优化器\n", - " self.memory = ReplayBuffer(cfg.memory_capacity) \n", - " def choose_action(self,state):\n", - " self.frame_idx += 1\n", - " if random.random() > self.epsilon(self.frame_idx):\n", - " with torch.no_grad():\n", - " state = torch.tensor([state], device=self.device, dtype=torch.float32)\n", - " q_values = self.policy_net(state)\n", - " action = q_values.max(1)[1].item() # 选择Q值最大的动作\n", - " else:\n", - " action = random.randrange(self.n_actions)\n", - " return action\n", - " def update(self):\n", - " if len(self.memory) < self.batch_size: # 当memory中不满足一个批量时，不更新策略\n", - " return\n", - " state, action, reward, next_state, done = self.memory.sample(batch_size)\n", - " state = torch.tensor(state, device=self.device, dtype=torch.float)\n", - " action = torch.tensor(action, device=self.device).unsqueeze(1) \n", - " reward = torch.tensor(reward, device=self.device, dtype=torch.float) \n", - " next_state = torch.tensor(next_state, device=self.device, dtype=torch.float)\n", - " done = torch.tensor(np.float32(done), device=self.device)\n", - " q_values = self.policy_net(state)\n", - " next_q_values = self.target_net(next_state)\n", - "\n", - " q_value = q_values.gather(1, action.unsqueeze(1)).squeeze(1)\n", - " next_q_value = next_q_values.max(1)[0]\n", - " expected_q_value = reward + gamma * next_q_value * (1 - done)\n", - " \n", - " loss = (q_value - expected_q_value.detach()).pow(2).mean()\n", - " self.loss_history.append(loss)\n", - " self.optimizer.zero_grad()\n", - " loss.backward()\n", - " self.optimizer.step()" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [], - "source": [ - "current_model = DuelingNet(env.observation_space.shape[0], env.action_space.n)\n", - "target_model = DuelingNet(env.observation_space.shape[0], env.action_space.n)\n", - "\n", - "if USE_CUDA:\n", - " current_model = current_model.cuda()\n", - " target_model = target_model.cuda()\n", - " \n", - "optimizer = optim.Adam(current_model.parameters())\n", - "\n", - "replay_buffer = ReplayBuffer(1000)" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": {}, - "outputs": [], - "source": [ - "\n", - "def update_target(current_model, target_model):\n", - " target_model.load_state_dict(current_model.state_dict())" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [], - "source": [ - "\n", - "update_target(current_model, target_model)" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [], - "source": [ - "def compute_td_loss(batch_size):\n", - " state, action, reward, next_state, done = replay_buffer.sample(batch_size)\n", - "\n", - " state = Variable(torch.FloatTensor(np.float32(state)))\n", - " next_state = Variable(torch.FloatTensor(np.float32(next_state)))\n", - " action = Variable(torch.LongTensor(action))\n", - " reward = Variable(torch.FloatTensor(reward))\n", - " done = Variable(torch.FloatTensor(done))\n", - "\n", - " q_values = current_model(state)\n", - " next_q_values = target_model(next_state)\n", - "\n", - " q_value = q_values.gather(1, action.unsqueeze(1)).squeeze(1)\n", - " next_q_value = next_q_values.max(1)[0]\n", - " expected_q_value = reward + gamma * next_q_value * (1 - done)\n", - " \n", - " loss = (q_value - expected_q_value.detach()).pow(2).mean()\n", - " \n", - " optimizer.zero_grad()\n", - " loss.backward()\n", - " optimizer.step()\n", - " \n", - " return loss" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": {}, - "outputs": [], - "source": [ - "def plot(frame_idx, rewards, losses):\n", - " clear_output(True) # 清空单元格输出区域，因为多次打印，每次需要清楚前面打印的图片\n", - " plt.figure(figsize=(20,5))\n", - " plt.subplot(131)\n", - " plt.title('frame %s. reward: %s' % (frame_idx, np.mean(rewards[-10:])))\n", - " plt.plot(rewards)\n", - " plt.subplot(132)\n", - " plt.title('loss')\n", - " plt.plot(losses)\n", - " plt.show()" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvwAAAE/CAYAAAA6zBcIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABvqUlEQVR4nO3dd3xkd33v/9dninrf1fZdb/e6FxbbYJtiuklCuQmB5IIhJA43cC+B3CQQcgNplOSSEFLIj9674eKAMcUFMDa2123ttXe93du0klZdmtG07++Pc85oRhpJI2mkkTTv5+Ohx0pn2nfKzvmcz/l8P19zziEiIiIiIstTqNwDEBERERGR+aOAX0RERERkGVPALyIiIiKyjCngFxERERFZxhTwi4iIiIgsYwr4RURERESWMQX8y4SZnW9mj5rZoJn9r3KPR+aPmb3ZzO4p9zhERJYbMztmZi8u9zhESk0B//LxZ8BdzrlG59zHyz2Y8czsk2Z2wMwyZvbmApe/y8w6zGzAzD5rZtU5l202s7vMbMTM9o//Mp7LbSvBVK+9f/CQNrOhnJ8X5Fx+l5l1+a/tY2b2qike54fj7idhZo/P2xMTERGRoijgXz7OA/ZNdqGZhRdwLIU8BvwR8PD4C8zsZcB7gBfhPY+twF/nXOVrwCPACuB9wLfNrH2ut50JM4vM9DalUKLHnfS1993nnGvI+bk757J3Amudc03AzcCXzWxtoTtxzr0i936Ae4FvlWD8IiIiMgcK+JcBM7sTeCHwb35mdaeZfd7MPmFmt5nZMPBCM3ulmT3iZ2tPmNkHcu5js5k5M3uLf1mvmb3NzJ5tZnvNrM/M/m3c4/6emT3lX/dHZnbeZGN0zv27c+4OIF7g4puAzzjn9jnneoG/Bd7sP8ZO4Erg/c65mHPuFuBx4L+V4LbTva7HzOzPzWwvMGxmETO7xszu9V+Px4JsuJm9MDebbWY/MbMHc/7+hZm92v/9PWZ22C+/etLMXpNzvTeb2S/N7J/N7BzwATNbYWa3+u/bA8C2YsYfmOa1n+62e51zqeBPIApsnO52ZrYZuB744kwfU0Sk3Mys2sw+Zman/Z+PBWePzWylmX3f3w/0+N/vIf+yPzezU/73+wEze1F5n4mIRwH/MuCcuwH4BfAOP7v6tH/R7wB/DzQC9wDDwJuAFuCVwP8IgtAcVwM7gN8GPoaXFX8xcBHwOjN7PoBf2vEXwGuBdv/xvzbLp3ARXhY68Biw2sxW+Jcdcc4Njrv8ohLcthhvwHutWoDVwA+AvwPagP8N3OKfMfgVsMPfEUSBS4F1ZtZoZrXAbrzXCOAwXjDcjHc2YnzW/GrgiP94fw/8O16wvhb4Pf8ny9/xvGcGz2m8K8ys28yeNrP/M/6sgn//ceB+4G5gTxH3+SbgF865Y3MYl4hIubwPuAa4HLgMuAr4S/+yPwFO4u37VuPtC52ZnQ+8A3i2c64ReBlwbEFHLTIJBfzL2/ecc790zmWcc3Hn3N3Oucf9v/fiBejPH3ebv/Wv+2O8A4SvOec6nXOn8ALWK/zrvQ34kHPuKT8D/EHg8qmy/FNoAPpz/g5+byxwWXB5YwluW4yPO+dOOOdiwH8HbnPO3ea/hj/BC35v9C9/EHge8Cy8A4tfAtfi7TQOOufOATjnvuWcO+3fxzeAg3g7k8Bp59y/+q9rAu+MxF8554adc08AX8gdoHPu15xzH57Bc8r1c+BiYJX/OG8A/nT8/eO9ZjcCP3bOZYq43zcBn5/lmEREyu13gb/x939deMmZN/qXJfESMOc555LOuV845xyQBqqBC80s6pw75pw7XJbRi4yjgH95O5H7h5ldnTMJsx8vaF857jZnc36PFfi7wf/9POBf/FOafUAPYMD6WYxzCGjK+Tv4fbDAZcHlQdZ+LrctRu5reB7wW8Fz9p/3dXhf/AA/A16AF/T/DC8b/nz/52fBnZjZm8zrqBTcx8Xkvw+5j9kORMZtOz6D8U/JOXfEOXfUP/h4HPgb4DcLXC/pnPsh8FIz+42p7tPMrgPWAN8u1ThFRBbYOvK/a4/72wD+ETgE/NjMjgRnWJ1zh4A/Bj4AdJrZ181sHSKLgAL+5c2N+/urwK3ARudcM/CfeEH6bJwA/tA515LzU+ucu3cW97UP75Rp4DLgrJ8R3wdsNbPGcZfvK8Fti5H7Gp4AvjTuOdfnZNfHB/w/Y1zA758B+RTead8VzrkW4Any34fcx+wCUuTXzW+awfhnyjH1ZyLC9HMIbgK+45wbKtmoREQW1mm8JE9gk78N59ygc+5PnHNbgd8A3h3U6jvnvuqcu86/rQM+srDDFilMAX9laQR6nHNxM7sKr8Z/tv4TeK+ZXQRgZs1m9luTXdnMqsysBi+YjJpZTTDJCW9i51vN7EIza8Grk/w8gD8f4VHg/f5tXoNXH39LCW47U18Gft3MXmZmYf8+X2BmG/zL7wXOxyvPecA5tw/vS/9qvNIZgHq8nUCX/7q8BS/DX5BzLg18B2/ybp2ZXYgXUBdtqtfezF5hZqv933cB/wf4XvC3f3mtmUXN7L8zdjAz2WPVAq9D5TwisrR9DfhLM2s3s5XAX+HtAzCzXzOz7WZmeGWiaSBj3no4N/iTe+N4Z8WLKYEUmXcK+CvLHwF/Y2aDeF9e35ztHTnnvouXufi6mQ3gZalfMcVNfoz35fdc4JP+78/z7+t24B+Au4Bn8E6dvj/ntq/Hm/TaC3wY+E2/pnJOtzWz3zWzorP9zrkTQDBZuQsv4/+n+P+PnHPDeK0v9znnEv7N7gOOO+c6/es8CXzU334WuASv1n8q78ArperAC6Q/l3uhef3v/2KK20/62uO1M91rXien2/AOLj4Y3DX+qWn/+b4T+G3n3MP+415vZuOz+K8G+vDeDxGRperv8OZo7cXr7vawvw28xhY/xSsbvQ/4D+fcXXj1+x8GuvG+r1cB713YYYsUZt48ExERERERWY6U4RcRERERWcYU8IuIiIiILGMK+EVEREREljEF/CIiIiIiy5gCfhERERGRZSxS7gEArFy50m3evLncwxARWZQeeuihbudce7nHUU7aT4iIFFbMPmJRBPybN29mz5495R6GiMiiZGbHyz2GctN+QkSksGL2ESrpERERERFZxhTwi4iIiIgsYwr4RURERESWMQX8IiIiIiLLmAJ+EREREZFlTAG/iIiIiMgypoBfRERERGQZmzbgN7ONZnaXmT1pZvvM7J3+9jYz+4mZHfT/bfW3m5l93MwOmdleM7tyvp+EiIiIiIgUVkyGPwX8iXPuQuAa4O1mdiHwHuAO59wO4A7/b4BXADv8n5uBT5R81CIiIiIiUpRpV9p1zp0Bzvi/D5rZU8B64FXAC/yrfQG4G/hzf/sXnXMO+JWZtZjZWv9+ZImJJ9Pc9vgZRlOZaa8bDhkvu2gNzbXRBRiZyMJ7+uwg9dUR1rfUTrhs78k+NrbW0Vpflbd9JJHiu4+c4uotK9i+qmGhhioiIovMLw52ce22lYRCtuCPPW3An8vMNgNXAPcDq3OC+A5gtf/7euBEzs1O+tvyAn4zuxnvDACbNm2a6bhlgdx9oIt3f/Oxoq8/EEvy+9dvnccRiZTP//raI+xY3ci/vuGKCZe94ZO/4q3Xb+XdL9mZt71vJMn7vvsEH37tJQr4RUQq1E+fPMvvf3EPf3HjLm5+3rYFf/yiA34zawBuAf7YOTdgNnZ04pxzZuZm8sDOuU8CnwTYvXv3jG4rC2cwngTglv/x3IJZzYDD8ZwP3clAPLVQQxNZcF2Do7Q3Vk/YnkhlGE6kGR6d+PlPpb2vt0hYPRJERCrVmYE4AMfPjZTl8YsK+M0sihfsf8U59x1/89mgVMfM1gKd/vZTwMacm2/wt8kSFPdLeTa11RUMdHLVREPEk+kJ24dHU7z7m4/yV79+0ZQHDSKLmXOOvliy4Gc85m9LZybmLpIZ7/9QNLzwp3BFRESguC49BnwGeMo59085F90K3OT/fhPwvZztb/K79VwD9Kt+f+mKJ7xApiY6fXayNhomlpgYDB3uGuJH+87yi6e7Sj4+kYUyNJoinXGMFPiMB5/7ZHriXJdshj+kDL+IiJRHMRn+a4E3Ao+b2aP+tr8APgx808zeChwHXudfdhtwI3AIGAHeUsoBy8IKMpc10fC0162NhrPXzxUESKf6YqUdnMgC6hvxytsKHdSOJLxSniC4zxUcBESU4RcRkTIppkvPPcBke6oXFbi+A94+x3HJIhFPpomEjGgR9cc1VeEpyx1O9Srgl6WrP+YF/IUy/MG2oHwnV8ov81FJj4iIlIvOMcuUYsk0tUVk9wFqIoUD/rgy/LIMBAF/obNYwee+UIY/FWT4VdIjIiJloj2QTCmezFBdZMBfW1W4pCeb4VfAL0vY1CU9fsBfIMOfzHbpUYZfRKTSlastpQJ+mVI8maa2qriPyWSTdoOAv6M/XrCLichS0BdLAJBIZ7JZ+0C2pKdAhj/4zCvDLyJSucqd8tEeSKYUn0lJTzRMLDkxwxnLZj8dZ/0+tCJLTZDhBxgZdyYrlgwm7RbI8Gc0aVdERMpLAb9MKZZMF9WhB7ySnoI1/DnbVNYjS1VQww9j81ICsYQX1KcKnMEK6vqjyvCLiEiZaA8kU4olZhDwR0NTlvSAOvXI0tU3ksj+Pr5Tz1RtOVNqyykiImWmgF+mFE9lZhDwh4mnCk9orPLbeirDL0tVboZ/fMAfm2rSrtpyiohImSnglynFE2lqi1hlF/wa/gIZ/ngyTXNdlLb6KgX8smTl1vAHNfuBoKa/0KRdteUUEZFy0x5IphRPFV/SUxMNM5rKkBlXxxxLeBN/17fUqqRHlqz+WJLWuigwswx/Sm05RUSkzBTwy5SCYL0YtVXe9caX9QSLd61vqVWGX5asvpEka5trgYm9+LMBf4EMf9Clp5jVqkVEZHlzZepOrj2QTCk+ky49/vUmBEPJDDVVYda3ehl+V65Pu8gc9MUSrG2uASautjtW0jNFhj+kDL+ISKWyMu8CFPDLlOLJmU3ahYnBUDyRps7P8MeSaXpzaqFFloJ4Mk08mWFtixfwTyzp8bv0FGrLqYW3RESkzLQHkkmlM45EOlP8wltBSc+4xbdiyTS1foYf1JpTlp4Bv0NPUNIzsS3n5CU9asspIiLlpoBfJhUsmFVTbJeeSCjvdoHcGn6AU30jJRylyPzrywb8Xoa/0GccJpm0m9GkXRERKS8F/DKpIIgJJuNOJ7je+JKeYPGusYA/XsJRisy/oCVne2M1IRtbaCsw5aRdP8OvlXZFRKRctAeSSWUz/JG5TtpNU1sVoqUuSl1VWCU9suQEq+y21lVRVxWZtKRnskm7IYOQJu2KiEiZKOCXSWUD/iIz/DWTTNoNWnuamd+aUyU9srQEJT3NtVFqqyYuMJet4S8waTeZyRBRS04RESkj7YVkUsHk2xn34c8J+J1z2Rp+wGvNqV78ssT0+yU9wVmqCZ2oklNN2nVEld0XEREAytOaXAG/TCo2w0m7QVCfG/CPpryDhuAswTqttitLUF8sQThkNFRHqI2G80p6nHPZmv5kwZV2leEXEal0RnkTP9oLyaSCwL3otpwFaviD3+uCDH9LLb0jyQmTHkUWs/5YkpbaKGY2oaRnNJUh46A6EsI5r51trmTGadEtEREpKwX8MqkgqJn5wltjWc7xnX42qBe/LEF9I0maa6MA1FWF8w5Yg/8nTf7l4yfuptNOLTlFRKSsFPDLpMZKeooL+Kv9Pvy59c3j72OsNacCflk6+mNJmuu8gL42mt+lZ8T/jDfVRICJE3eTmcyyX2XXzDaa2V1m9qSZ7TOzd/rb28zsJ2Z20P+31d9uZvZxMztkZnvN7MryPgMRkeVtee+FZE5G/Ux9sTX8oZBREw3l1fAH2c/cSbuggF+Wlr4Rr6QHvAx/oc94kOFPj5u4m0o7oss/w58C/sQ5dyFwDfB2M7sQeA9wh3NuB3CH/zfAK4Ad/s/NwCcWfsgiIpVj2kjOzD5rZp1m9kTOtm+Y2aP+zzEze9TfvtnMYjmX/ec8jl3mWWyGNfzBdfNq+MeV9KxqrCESMpX0yJLSF0vQUlcFMGHSbvB5b6zxS3rGTdxNVUBbTufcGefcw/7vg8BTwHrgVcAX/Kt9AXi1//urgC86z6+AFjNbu7CjFhGpHJEirvN54N+ALwYbnHO/HfxuZh8F+nOuf9g5d3mJxidlFJ9hSQ94wdBUGf5wyFjTXLPkMvwPP9PLZRtaCGvyZUXKreEfP2k3qOdvDEp6xmX4k+nKmrRrZpuBK4D7gdXOuTP+RR3Aav/39cCJnJud9LedydmGmd2MdwaATZs2zd+gRUQWiCtPV87pM/zOuZ8DPYUuMzMDXgd8rcTjkkVgpjX8wXWnquEHWNdcS0d/vESjnH+HOod47X/cy0+fOlvuoUgZpDOOwXiKlrqcSbvJNM7/1h6r4S88aTeVzhBd5hn+gJk1ALcAf+ycG8i9zHkv2Ix2dc65Tzrndjvndre3t5dwpCIilWWue6HrgbPOuYM527aY2SNm9jMzu36O9y9lFE9mqIqEZpTVrhmX4Q9+r8tZrbehJsLwEmrLebR7GICuwdEyj0TKYcBfZTe3hj+dcST9TH48W8NfeNJuKlMZXXrMLIoX7H/FOfcdf/PZoFTH/7fT334K2Jhz8w3+NhGRZc3KtDuYa8D/BvKz+2eATc65K4B3A181s6ZCNzSzm81sj5nt6erqmuMwZD7Ek2lqIjP7iNSOW4U0W9KTE/DXVYUZGU1PuO1iddovPxqML52DFCmdPj/gz3bpqfIC++CzHdTzBxn+1LgMfzKdIbr8u/QY8BngKefcP+VcdCtwk//7TcD3cra/ye/Wcw3Qn1P6IyIiJTbrvZCZRYDXAt8ItjnnRp1z5/zfHwIOAzsL3V6nahe/eDKdF6gXY9JJuzklPQ3VEYZGl07wHMw3GIgnyzwSmS+3P3Fm0sXg+kYSALTUjk3aBRhJpvx/89tyJsfV8KczrhLmflwLvBG4Iadpw43Ah4GXmNlB4MX+3wC3AUeAQ8CngD8qw5hFRCpGMZN2J/NiYL9z7mSwwczagR7nXNrMtuK1XDsyxzFKmcSS6RnV74NX0nNuOJF3H8H2QF1Vfh/zxS7oKBSUdsjycqJnhLd9+WE+9NpLeMNVEyeGjs/wB+VpwWc45h8oBG05U5nxGX5HTXR5B/zOuXtg0nXjX1Tg+g54+7wOSkREsoppy/k14D7gfDM7aWZv9S96PRMn6z4P2Ou36fw28DbnXMEJv7L4xRLpGbXkBK90Z3RcSY/Z2KJcAPXVYYYTqeykx8XuZDbDv3TOSkjxuoa8uRlnJplI3j+SX8MfnPWKZQN+L8BvnGThrVSmcibtiojI4jRtht8594ZJtr+5wLZb8CZtyTIQT2WonmmGPxKaUMNfGw1jObNU6qsjOOdl/+uq5nKSaWEow7+89fpnpLoGCwf82ZIevw9/kOEPPucjyRRVkRDVEW/7+LacqQpryykiIouP0k4yqXgiTW2Rq+wGJkzaTU48S1DvB0zD8zRx997D3fzyUHdJ7iueTNPtZ4AHVcO/LPX4Af/ZgcJdmIKSnqBGP1vDn83wp6mrCmfr9AtO2lWGX0REWMR9+KVyxVOzKOkpMGl3/DyA+movcJpskuRcfewnB/nI7ftLcl9Bhx4zlfQsV71+Br9zkgx/fyxJY00ku1ruWEmPP2nXP4sV9VtvJiu0LaeIiEyuXO04Awr4ZVKxxOwm7Y6mMmT8oCeeTOf14AeyZTzz1alnJJmic5Js7UwFHXo2r6hXSc8y1TPsva+TfWb6c1bZhbHPb26Gv7YqTMRvvTk+w++V9OirVkREykd7IZnUrDL8fnAfT+UHQ7nqq/NLIkptJJGma2iUdGbu582CDP+uNY1qy7lMBTX83ZN8ZvpiyewquzCxhj/mH9QGWfzxbTm9kh5l+EVEpHwU8MukYomZT9oNDhDiSS/LOVVJz/A8ZfhjiTTpjMvWZs/Fqd4YIYOdqxuJJzMkUpnpbzRHtz1+hu88fHL6K0pJ9PglPRkH54YnZvn7RhLZHvwwsUvPSCJFXTSSrdMf35ZTJT0iIlJuCvhlUvECE26nU+NP8h3LfmYKTNoNAv75yfAHjz1ZTfZMnOyLsbqphhUNXsC3EBN3P3PPUf7zZ4fn/XHE05tzYFiorKcvlsz24IfCk3a9kp5g0u74Lj0ZlfSIiEhZaS8kk4on09kAvlhBNj+WsyjR+IA/KIkYnqdJu0Eg1jk49zr+U70x1rfU0lTjBXwLMXG3czA+aU94Kb2ekQTrmmsA6CrwmekfSWZ78ANEwyGiYct+zsYm7Xr/V5Lja/gzasspIiLlpYBfCkqmM6QyblZdesA7WAC/Lee4Gv6GoEvPPJT0pDMuW3bTVYKJu6f6Yqxvrc0uqjTfE3edc3QNjjIYT81bFyPJ1zeS5Pw1jcDEs0LOOfrH1fCD9zmPT1LDP34eQCrtsh1+REREykF7ISkoCGbGB+vTqR0/oTGRmVDDX1cdZPhLX9KTuwbAXEt60hlHR3/cy/DXBhn+qQP+j9y+n//3yKlZP+bQaCo7/6FDWf55l844+kYS7AwC/nEHicOJNKmMy6vhB69TT3BANr5Lz/i2nMmMJu2KiIhHffhlUQkC59lO2g1KegrNA6gKh4iEbF4m7eZmxeda0nN2IE4q41jfOlbSMzhNSc83HzzB1x98ZtaPmVtS0jGggH++DcSSZBysaaqhpS7K2XEHicEqu83jM/xV4bySnrq8Gv6xkp50xuEcquEXEalw5U77aC8kBY36WeaZT9odK+lxzmXLHXKZGfXVkXkJ+OOJsWBrrr34gx78XoZ/+pKeoPzjydMDuFkewucepJxVwD/vgg49bfVVrGqsnvCZ6Rvx3u/cPvwwtsBcJuOyq0kHJT25k3aDen516RERkXJSwC8FBRn+WU/aTaZJph3pjCtYFlRfFZ6Xkp6R5NhBxPhs7UwFPfg3tNbSWDN9SU8s6ZV/DMRTnOyNzeox8zL8/aVZPEwmF3Toaa2rYnVTzYSzQudyLs9V52f4g/UmaqvG2nImc9pypvzyHpX0iIhIOSngl4KyNfyzXXgrmc45aJh4H3XVkXmZlBqUErXWReec4Q+C9nUttdRXhQkZDMQmH3N/TvZ/3+mBWT1mEPBHQqYM/wII1mpoq6+ivbF6Qpeew51DAGxtr8/bXlsVJpZMZz9vdZO05QzKe1TSIyIi5aS9kBQUBDKFgvWp5NbwT3XQUF8dYWge+vAH4960op6uwdFZl9aAV9LTWhelriqCmdFUG50yw597MPDkmVkG/EOjRMPGeSvqONM/u7MEUrxev6Sntb6KVY01Ez4zBzsHaa2LsqJ+YoY/lkhn6/hrq8KEC9TwB6vuqqRHRETKSQG/FDRVdn4q2YA/mckJhiZ+zOqrwvPSljN4zM0r6kikM3lZ95k61eu15Aw01USnnLSb+1hPnu6f1WN2DozS3lDN2uZaOkrQVlSm1jPsvWdtdV4NfyKdydbtAxw8O8SO1Y2Y5QfstdEwI8lU9v9JXVUYMyMatrwuPUGLTmX4RUSknLQXkoLis5y0Wx0ZW2k3yLYXuo+6qsi8tuU8b4VXgjGXTj2n+rxFtwJNtZEpJ+0Gl21dWc+TBUp6ijnb0DU0SntTDaubajirtpzzrnckQXUkRG1VmFVN1cDYZ8Y5x9NnB9mxqmHC7WqrIvkZfv8zHgmF8vrwa9KuiIjkcpSnL6cCfikoPstJu6GQURMNMTpNDX9DdXheuvTEcjL8MPtOPc45f5Xduuy2xuqpS3qCDP8121Zwuj+enRAajOu6j9zFH33loSnPOnQNehn+Nc3VdA2NTljESUqrZzhBm1+us6rRW203WL+ha3CUgXiKnasbJ9wuKOmJJfLXq4iELW+lXU3aFRERACvzbkABvxQ024W3wAvwY8mxGv66qsiE68zbpN0JGf7ZZcl7R5LEkun8kp7ayJSTdoODgedsXQHk1/HfdaCTU30xbnu8g1//13t44lThkp+uwVHaG6tZ01RDOuPoHlJZz3zqHU5kO/CsavQz/P5B4tNnvQm7hTL8dVVhRpJpYn5XqOAzHgmZJu2KiMiio72QFJTNzkdmHvAHPcqnKumprwozPA+TdoMSi/P8DP/ZWWb4T+f04A801RSZ4fcD/n05dfw/ePwMbfVVfPMPn0MyneG1/3Ev39pzIu/2qXSGc8N+wN/sPa5W251fPSM5GX6/pCdo53qwcxCA7asnBvw10TDOQa8/B6Aum+EPkcpMnLSrDL+ISGX68b4OTvSMlHsYCvilsGwN/ywy/LV+hj+WnGLSbnWEWDJd8pKVmH/WoK2uivqq8Kwz/EFLzg15Gf7pJ+02Vkdob6xmbXNNto4/lkhz1/5OXnbRGq7a0sZt/+t6Lt3QzAdu3Ucm5/n3DCdwzss0r2nyykvOKOCfV73DCVr9gL+uKkJjdSQvw99SF6W9oXrC7YIA/9ywd93goDYasmyQD2SDf2X4RUQq081feoiX/vPPyz0MBfxSWBCsB5NwZ6ImGp62D3+9XwJR6rKeYNXTUMhYVWAhpWKdKpDhb6yJMDSaymu7mGsglqLJX5H1wrVN2ZKenz3dyUgizSsvWQt4LSBfc+V6hhNpOnJ67QdjbW+sZnWzn21WL/551TOcoK1ubBXd9qaxXvyHOr0Ju+M79EBuwO/N06jNzfCrLaeIiOQI4iHw5nZ1lmHfroBfCoon09REQwWDnekEixJN1Ye/rtrbNlLiTj0jiXQ2GGtvrKZrliU9p3pj1EbDtOQEg03+artDk0w27o8lswH/ReuaONw1TDyZ5gePd9BWX8U1W9uy193W7pWJHO4aym7rGhoL+FfWVxMJWd4BgZRWMp1hIJ7KZvjBO7vSORj3O/R4LTkLqfUPWHuGvIC/LnfSbmZiDX+wCq+IiFS27zx8iqs+eAf9I7NvGz4b2gtJQV7AP/NyHvAC/HheH/5CXXq8gGmy4Hm2YjnjXt1UM+uSnoOdg2xtr8874AmC+ckm7g7EkzTVeM/rwnVNpDOOx070ccdTZ3nZRauJ5AR92YC/Myfg9w9O2huqvTMUjdVlbc15pj/GZ+45OqfFyxazoN9+W17A750V6hoapT+WLDhhF6Aump/hD+a6REP5Gf6gZC1YlEtERARgcHSRBfxm9lkz6zSzJ3K2fcDMTpnZo/7PjTmXvdfMDpnZATN72XwNXOZXLJGecQ/+QM24SbuFJv4GXU1GSjxxN5aT4feytbPL8B/oGOT8NfnZ3SCYn2zi7kAsSXM2w98MwCd+dpiRRJob/XKewMqGKppqIhzuGs5uy83wA6xprilrhv/7j53hb7//JHtPzm4RscUuu8pu3bgM/8AoB7MdeibL8PsB/9AoNdEQIT+gj4Tzu/Qk1ZZTREQWgWIy/J8HXl5g+z875y73f24DMLMLgdcDF/m3+Q8zm13UKGUVT2XmEPCHiPslPdWRsWAoV70fMA2XuIZ/JJHOBmOrGqsZSaRnfBahdzhB5+Aou8YH/EGGf4qAP7jOhtZaGmsi3H2gi9a6aLZVZ8DM2NrekF/SMzhKU00ke4ai3AF/EBDfub+zbGOYTz1+dj4vw99UTSyZ5tETfQDsLNChB8YC/u6hRF7bWa9Lj9pyiojI4jLtXsg593Ogp8j7exXwdefcqHPuKHAIuGoO45MyiSXSVM+hpCfo0lM3SZefer+kp9SLbwWTdiGnzeIMg+b9HV47xl1rmvK2NwYZ/klKevpzMvxmxoVrvdu/7KI1eeU8gW0FAv4guw9eSVJHf7xsJTW9fsnLXQeWZ8AfLIyWn+H3uiPdc7CbpppI3vuRK/hc9wwn8g6MIyEr2JZTk3ZFRKSc5pJ2eoeZ7fVLflr9beuB3ObiJ/1tssSMptLUznCV3UAwaXeqsqD66iDDP58lPf7KqTOcuLu/w+uuMyHDXzN5hj+VzjCcSGevA14dP8ArxpXzBLatqufswCiD/v11DsbzAsw1TTWMJNIMzsOKxMXoj3kB8d6T/fPSUeBw1xAfuHXfpF2P5ltvoRp+/yDxoeO97FzdOOmk9bqod/A3/qA2MklbTk3aFRGRcprtXugTwDbgcuAM8NGZ3oGZ3Wxme8xsT1dX1yyHIfMllpjbpN1Ywsvw10yS4R+r4Z+HDH9OSQ/MfLXdAx2DtNZFJ2R3xybtTgz4B/z+/M21Y+Udr758Pa+9Yj3P3bZiwvVhbOLuEb+O38vw12QvX9Ps/Z47cXfvyT5O9i7MAh59I0la/S5Fdx8o/f/RWx89zefvPcaDx3pLft/FCEqWcjsxBQeJiXSGHZOU8wDU5KwtkRvwR8e15Qzq+SOatCsiImU0q4DfOXfWOZd2zmWATzFWtnMK2Jhz1Q3+tkL38Unn3G7n3O729vbZDEPmUTw1t0m7oymvS8/kGf556tKTSFPrZ1+D4K1rhhN39/sTdsdndxurI5hRcPGtYJXd4KAA4LKNLfzTb18+aXZ3fGvOrsHRvEWeVvuLbwV1/LFEmt/51P186Lb9M3o+s9U7kuRZ57WytrmGO/afLfn9B8+7XCVDPcMJ6qvCeQe2QYYfJp+wC+TV7efePhK2vBr+pNpyiojIIjCrvZCZ5dYovAYIOvjcCrzezKrNbAuwA3hgbkOUcphTht/PePaNJCYP+Kvmqw9/Kruyb1NthKpIaEadejIZx9NnByfU7wOEQkZDVaRgSU+Q9W/OCfinc96KOiIh43DXEMOjKYYT6byAc62f4e/wM/w/fOIMQ6MpDpwdLPox5qJ/JEFLXRU37FrFPQe7GU2V9r0Kzmzc8VTpDyaKkbvKbqCxOkKNX8o2VYY/93OdX9ITGlfSoxp+EREpv2Lacn4NuA8438xOmtlbgX8ws8fNbC/wQuBdAM65fcA3gSeB24G3O+dKGyXIgognM7MO+Gv81Xl7hhMFe/CD182kOhIqeZcer6bay76aGaubqmdUf36yN8ZIIj2hfj/QVBstOGm3fxYBfzQcYtOKOo50DWfPQhTK8AeTjr+15yQAx7qHSaTmv+69L5akpTbKDbtWMZxI8+DR0pXeZDKOI91DNPqtSZ85tzBlSrl6RhJ59fvgfWaCM0NTZfjDIcuuQp2b7Y+GbVxJj7r0iIhI+RXTpecNzrm1zrmoc26Dc+4zzrk3Oucucc5d6pz7DefcmZzr/71zbptz7nzn3A/nd/gyX+LJdDZTPlO1OR1MpjpoqK+OlLRLTybjiCfz24kGCykV6yl/wu74HvyBxppJMvzxiSU9xQg69YzvwQ9eqUhLXZSOgTgneka478g5dq5uIJVxHDs3PNldlsRoKs1IIk1rfRXP3baS6kiopGU9p/tjxJMZfvfq8wC4cx5KhqbTO5zI69ATWNVYTWNNhNVNhTv0BILMfu5B7YS2nBnV8IuICBjl3Q8o7SQFxZLpggtmFSMI8gfiqSnnAdRVhUu68FYsOXFl35kuvnXAb8m5c/VUGf6JAf9sMvzgBfzHukc445ftjJ8ovKapho7+Ub790EnM4E9ftguAp+e5rCdY8ru5NkptVZjnblvBnfs7S9YiNFhw7IZdq9jaXs8dZej1XyjDD3D9jnZeecnaSTv0BILPdu5nPDquLWdKbTlFRGQRUMAvEzjn/Az/7Lv0BCbrww/QUB0paUlPEPDXjQv4Z9KH/0DHIJva6rKTisdrqokWnLQblPnktuUsxrb2ehLpDA8f782ON9fqphrO9Me45eGTXLttJdfvWEnIyK4EO1/6/AOYoIPNDbtWcfzcCEe6S3Nm4XCnN/5t7fW8aNcq7j/SU/I1GabTO5wsmOF/54t38OH/dum0tw/+f+TV8E9YaVeTdkVEBH5xqLusj6+9kEyQSGfIOOY8aRemvo+6qjDDpczw+xOA87uu1DAYTxFPFvc4+zsGJq3fB2iapKSnP5akKhzKTvgs1rZV3sTQXx05RzhkEwLQtc017Ds9wMneGL+1ewM10TCb2uo42Dm/Gf7xi1K9cNcqoHTtOQ93DdFSF6WtvooX7lpFIp3hngX8MhxNeSswt9XP7AAtV1C7n/t5D4+ftKu2nCIiAvzXY6fz/v7h4x1c++E7F2wtmsJpTKlo8aT34ZtLH/7s71Nk+OurIyVty1kowx+UyLzjq49kg/HfuWoTz92+csLt48k0R7uHeeUkC2XB5CU9A/EkTbWRactAxtu20gv493cMsrqpmtC4wDCYuNtYE+FlF60BYMfqxgXL8AclShta62hvrOaAP8dhrg53DbGtvQEz49mb22isjnDX/s7sc5xvfX7J0vguPTNRKMMfDY8v6fF+DyvgFxGRHH9/21OA1+p7LvuiYinDLxME2fCZZqsDuQcKU9Xw11dFSlrDH7T4zA3Anr25jQvXNnGke4gnzwzw4yfP8sX7jhe8/aHOITIOzi/QkjPQVBNhcDRFJpNfy94fS854wi5Ac12UlX5nnvH1+zC2+NavX7Yu+7ruWNXA0e7hbI/3+RDU8OcuSrVlRT1HS1XS0zXMtvZ6wCt3ed7O9rw5Aoe7hrj3UPe8ZT56/DMYbQVKeoqVreHP6dITCYXGlfQ4omGb8YGgiIhIKSnDLxMEAf9cFt4KTDlptzpc0gz/iD8fIPfxt6ys57Z3Xp/9+82fe4CTfYVbQO73J+xO1qEHvAy/czCcSNGYU68/EEvOuH4/sLW9nu6h/EW3AuevaSQSMn7nqk3ZbTuCTj3dw+yYZHLxXAWr0OaWGG1ZWV+STj39sSRdg6PZhcfAKxn6weNn+L8/PsC9h8/xyDN9AKxsqOY1V6zjtVduyK5LANBYEy0qa+6cKxhsZ0uW5pBVyXbpieZn+JPj2nKqJaeIiJSbAn6ZIDbHgD+vhn+qkp6qSDZIL4V4tqRn8o/1htZaHj3RV/CyAx0DVEVCbF5RN+ntG2u8+x6ITwz4W2aZLd7W3sADR3uy/d9zXbmplcfe/9K8ScRBf/iDnUPzFvD3xZJEw5Z3tmRLez3dexJe+dIsD24AjnQFE3bHAv4XnN9OyODf7zrMztUNvO/GC9jYVst3HznF5355jE/94mjefVy9pY1v/OFzCt5/z3CC7z16im8/dJJTfTHu+fMbaBg3CbvHP6Ap1KWnWJNO2s1badepQ4+IiJSdAn6ZoNDk15moLbakpzoy60m7P9rXwb5T/bz7pedntwUlPVM95obWOvpGkgzGk3kBO3gZ/p2rG4hM0VElCHQHYknWt9Rmt/fHkpy3on5WzyUobSlU0gNM6Bjk1b57rTlvnGK+wVz0jSRorq3Ky45v9p/fse5hLt3QMuv7Dlpybm0fe71WNlTzpbdeTWNNhEvWN2cf9+UXr+Xc0Ch3PNWZ7ei092Q/333kFAfPDk444PnI7fv59C+OkEw71jTV0DeS5Fj3MBevb8673vhJybNRsA9/KEQ647JnFlKZjCbsiohI2elcs0xQykm7U7XlrK8Kk0hnZrVq7Cd/foTP3Xssb1usQA3/eBtbvez9yd7YhMv2dwxy/urJ6/dhbGGt8RN3B+Ipmmpnd/wcdOqZLOAfr7Yq6NQzfxN3+0aStNblHxAFAfpc6/gPdw0RDRsb2/LPpFy7fSWXbmiZUIKzoqGa1z17I2+5dgtvuXYLf/nKC4iEjG89dHLC/X7i7sO8aNdqbv/j6/nUm3YDhd/rnuGJcxRmKjiTVDeupAfIdupJZ9yUB5AiIiILQXsimWCuk3arI2O3m7qG3wuYgkC9WIPxJI+e6GMwnsqrly608NZ4G1q9rPz4ILBvJEHX4Cjnr2kodLOsbIY/pxe/c47+WHLGi24FLl3fzNrmGi7d0Dz9lX07VjVwaB479fSNJCcEw5va6jArHPB3D40WvSjX4c4hzltRP+ve9Csaqrlh1yq+8/CpvPf/c788SlU4xN+++mJ2rWlivf9en+qbGPB3DsZpqYvOqT9+TbRAht+/v7Rf1pNMO6LK8IuISJkp4JcJ4kUEzlMJhSwb9E91lqCh2rtsaIZ1/Pcf6ckGVMHkUii2pMcLAk/05E/cPezXlW9fNU3A72fxB3N68Y8k0qQzbtZ17SsaqrnvvS/iik2tRd9m+6pGjnQPzVunnl6/pCdXTTTM+pbaCQH/yd4RnvOhO/j+3jNF3bfXknN25U+B39q9ke6hUX7mrwvQN5LglodO8arL12XPlLTWRamNhjlVIMN/ojfGprbJ52oUo65QDb8f3AcLbqXSGWX4RUSk7LQnkgmCTHlNZHYBP4wdLEx10BCURIzMsFNP7gJNvcNjgXesiIC/rb6KuqrwhAz/4U4viM2dSFpIY83Ekp7+cT3rF8LO1Q0k047j5wp3HJqr/tjEkh7wOvWMD/gfONpDMu344RPTB/zJdIZnekamfZ2n84Lz21nZUMW3HjoBwFcfeIZYMs1br9+SvY6Zsb61llMFujKd6BmZUFI0U/XZgD+3LacX8AetOZOZypi0a2afNbNOM3siZ9sHzOyUmT3q/9yYc9l7zeyQmR0ws5eVZ9QiIpVDAb9MENTwzzbDDzk9yqectOtdNjzDkp5fHurOdl0J+qmDd6BSHQlNWLwql5mxobWWk70TM/xV4RAbWqcOAnO79ASClXdn04d/trKdes7Oz4q7hUp6wA/4u4bzyncefqYXgJ8/3T3tfIwTPSMk027OAX80HOLVl6/njqc6OTsQ54v3Hufa7SvYNW4NhfUttZzui+dtS2ccJ3tHsvM5ZuvlF6/lfTdekNcuNMjmB+sHpNIZopXRlvPzwMsLbP9n59zl/s9tAGZ2IfB64CL/Nv9hZrP/shERkWlVxJ5IZiab4Z/lpF0oMuD3M6PDM8jwnx2Ic7BziJdeuBrIL+mJJdJTTtgNbGitm5jh7xpiy8r6aXu7R8Mh6qrC+Rn+kYXP8G9b5ZXEzMfE3XgyTSyZLthmdMvKegZHU5zLOdB65Jk+6qq8NRX2HOuZ8r6DDj3bpimdKsZv7d5IKuN4+1cepmMgzu9ft3XCdda11E6o4e8YiJNMuzmX9LQ3VvMHz9uaN8k4O2nXLzlLVUhbTufcz4Gp3/wxrwK+7pwbdc4dBQ4BV83b4ERERAG/TDTXSbvebb3Au6Zq8vsI2k3OJOC/97BXzvPrl60D8jP8I4n0lD34AxtaazkxLsN/pGs4G0RPp6kmms3qw1i2fy696WeqrirCxrZanp6HDH9QolQow795ZX6nnpFEiv0dg7zhqk1URULcsb9zyvsO5kpsnWMNP3iLkl26oZk9x3vZ2l7P83e2T7jOhtZaeoYTees9POOXQc014C8kWGQryPAn1aXnHWa21y/5CSaprAdO5FznpL9NRETmSUXviaSweDJNyKBqDoFKbYFVSMcLsvEjMyjpuefgOVrrojxn2wpgrJ86QCyZKuogZWNrHYPxVDawTaQyHO8ZYevK4rLOjTURBnNKespRww9eWc+hecjw9/lnLFpqJ2b4twYBv5+p33uyn3TGce32FTxn6wrumi7g7xxiVWN1yQ6OfutZGwD4vWu3FCzlCtZKOJ2T5Q8mbM9LwD+uLadX0rP8M/yT+ASwDbgcOAN8dKZ3YGY3m9keM9vT1dVV4uGJiJRfgcXg54UCfpkglkhTEw1P6Ic+E0HgPVXAH9ThDxWZ4XfO8ctD3Tx3+0pqomEaqyPZFVODcReb4QeydfzP9AyTzrjiM/y10WyQD2UM+Fc3cKRruOjXL/C9R0/x9QeemfTyoEyqUIZ/fUst0bBx9JwX8Af1+1dsbOWGXas40j08ZZ/+Q11Dc67fz/W6Z2/kI//tEl63e2PBy9cXaMP6TM8I4ZCxtmXiysZzFbT5TAVdejJu2jKx5co5d9Y5l3bOZYBPMVa2cwrIfcM2+NsK3ccnnXO7nXO729snnsEREZHiKOCXCeKp9JSBejFqo2GqwqEpyxmCPvwjRbblPNw1TMdAnGu3rQSgtb4qL8M/kihu3MHE3BM9XhB4qMgOPYHz2uo40DFIxq/TDur5G2oWduHql1ywmrTzathn0p7z3+86xL/eeWjSy7MZ/gIBfyQcYlNbXTbD//DxPrasrKe1voobdq0C4M5Jsvydg3H2nuznik0tRY91OtWRML/9bK+cqJAgw59bx/9MzwjrWmrm1IN/MuO79KTSmXl5nKXAzHKXgX4NEHTwuRV4vZlVm9kWYAfwwEKPT0SkklTmnkimFEtk5jRhF7wa/unKa4IVSodHiyvp+aXfjvO67WMBf8/IWKY9nkwX1VloY1t+hn+srry4gP+521dybjjBAb9+vj+WpLEmsuCZ3N2b2/j7V1/Mz57u4n3ffbyoha+GR1Mc6hziVF8sbx5Crv5YkOGfWNIDY605nXM8eqI3G8BvbKtjx6qGSct6vvvwKdIZx2/6ZTgLYXVTDZGQ5fXiP9E7Mi/lPJCb4XfZfyth0q6ZfQ24DzjfzE6a2VuBfzCzx81sL/BC4F0Azrl9wDeBJ4Hbgbc752bWqktERGZkYVOSsiTEU+k5TdgFaK2rmjRgDIRCRl1VuOhJu/cc6mZjWy2bVnjBWltdlO6h/Az/upbpA/7m2igN1ZFsmcfhriHWNNVkS4ymc+12b/7ALw91c8HaJgbiyQWdsJvr9Vdt4nR/nI/fcZC1zbW86yU7p7z+k2cG8GNRDnQM8uzNbROu05ut4S/8nLasrOcXB7s5fm6E7qEEV+YsGHbDrlV89pdHGRpN5b2ezjm+uecEzzqvtegDq1IIh4w1zTUTavhf4nd5mo/Hg5xJu2mXnci7nDnn3lBg82emuP7fA38/fyMSEZFcy39PJDMW92v45+KdL97BZ27aPe316qoiRfXhT6Uz/OrIuWx2H/wM/yxKesb34j88gw49AGuba9nWXp9dAGwgllzw+v1c73rxDn7rWRv4lzsO8l+PnZ7yuo+d6Mv+vr+jcIefvpEkVX770UK2rGxgNJXhNn+hrdwSnRt2rSKZdtxzMH+C5SMn+jjcNZydZLuQcltzDo+m6B5KzHnRrckUnLRbARl+ERFZ3BTwywSlqOFf2VDNjtWN016vvjpcVA3/fUfOMRhP8YLzV2W3tdVV5fXhL7akB8Z68TvnONI584mk121fyf1HekikMgzEUjTVlu9kmZnxwddewiXrm/nwD/czmpr8AOrxU/2saaqhsSbCgY6BgtfpjyVorotOOml780ovWL7loZPUVYU5P+d9ftZ5rTTVRPjJk/llPd/ac5KaaIhXXrqWhbahpTZb0hO0Y53/kp6xSbsV3pZTREQWAe2JZIKBWGrBJqDWV0WKKum57fEz1FeF83qtt9ZXMZJIZ9cNKDbDD/gZ/hidg6MMjqZmHPBfu30lsWSaR57ppb/MGX7wAs33vGIXp/pifOVXk3fgefxkP5duaOb81Y0cmCTD3zucnLScB8i2Lz3cNcylG5rzAtpIOMQrL13Hdx45ye1PdABe96TvP3aaGy9eS2MZSp/Wt9b6i21l5rUHP0yctJus7LacIiIyjcv/5icc6Sp9i+3xFPDLBGcH4qxuLH3LwkLqq8PTTtpNpTP8aN9ZXnTB6rxSo7Z6b45A70gC5xyxZHEr7YIX8A+Npnj4uNdWcqYB/9VbVxAyr46/nDX8ua7dvpJrt6/g3+46VLBV50A8yZHuYS7b2ML5axrZ3zFYcKJvXyxB6xTzL1Y3VWcPrHLr9wN/9WsXctmGFt759Ud46HgPP9rXweBoit/cvfDlPOB16sk46OiP88w89uCHsQx/0DWpUlbaFRGR2fuhnyCbT9MG/P4KiZ1m9kTOtn80s/3+CorfNbMWf/tmM4uZ2aP+z3/O49hlHqTSGbqHRlndVL0gj1dXFZm2pOdXR3roGU5w4yVr8rYHQWnPcIJ40guwaovoww9ka7jvPuDVms+khh+8ib+XbmjhnkPdiyLDH/izl+2iZzjBp39xZMJlT5zqB+CS9c3sWtvEYDzF6f74hOv1jSRpLtCSM2Bm2RV3rygQ8NdWhfnMTbtZ21zDW7+wh0/+/AgbWmu5ZsuK2T6tOQl68Z/qi3GiZ4TGmsi8vV9BcD/WpSejkh4RESm7YvZEnwdePm7bT4CLnXOXAk8D78257LBz7nL/522lGaYslHPDCTIOVjUtTIa/oTqSl43+wr3HuNefDBu47Ykz1FWF8+r3ISfDP5zMHjTUFtldKFh862dPd1FXFWbNLJ7vddtX8tjJfkYSaZoWScB/2cYWXnHxGj79i6OcGxrNu2zvyZyAf41Xd1+ojr9vZOqSHhhbcXeynvorGqr5wu9dRdiMJ88M8N+u3FBwJdyFkO3F3xvjmR6vJedcFpWbStCRJ5vhz7hsmY+IiEi5TBsdOed+DvSM2/Zj51wQpf0Kb6VEWQbODngZ39ULFPDXVYUZ8bv0PPxML++/dR9v+/JD2XGk0hl+9EQHN+xaNaFzUFu9F5T2jCSI+XX8xay0C2OLb3UMxNnW3jCrAPDa7StJ+5ncxZLhB/iTl+5kJJHiP+4+nLf98ZP9bGyrpbW+ip3+RNtCnXr6Ygla66duqfrrl63jDVdtYmXD5GeCzltRz2ff/GxefMEqfufqTbN4JqWxLmfxrSDgny9BR57gc5GqkLacIiKyuJViT/R7wA9z/t5iZo+Y2c/M7PrJbmRmN5vZHjPb09XVNdnVZIGdHfCywgtV0lNf7U3adc7xD7fvp7Uuymgqk11I6oGjPZwbTvDKSyZ2dwlKevpGEsT8g4Ziu/Q010Zp8icmb2ufWTlP4MrzWrLrFSymgH/7qkZ+81kb+NJ9x/NWmN17qo9L17cA3njXt9Sy/0x+wB9PpoknM9M+n5dfvIYPvfaSacdy2cYWPn3TsxfsALKQmmiYlQ3VnOwd4URvbF4D/nChSbuq4RcRkTKbU8BvZu8DUsBX/E1ngE3OuSuAdwNfNbOmQrd1zn3SObfbObe7vb290FWkDBY6w19fHWY4keYXB7v51ZEe3vmiHfzpy87np0918r1HT/ODx89QG51YzgNe0Grm1fAHGf6ZtBMNsvwznbAbqI6EucqvSy9nW85C3vninWDwsZ88DUDvcIITPTEu3dCcvc75ayZ26ukLFt2aooZ/KVrfUsMjz/SRSGXmrQc/5EzazWvLqYBfRETKa9YBv5m9Gfg14Hed3+rDOTfqnDvn//4QcBiYeulPWVQ6B+KEDFZMU9JRKnVVEdIZxwdve4oNrbW84epNvOXaLVy5qYX337qPHz7RwQ0XrCqYuY+EQzTXRukdTmTLgort0gNjdfzbVs1+5dfr/FV3F1OGH7y69Tddcx63PHySg2cHeTyYsDsu4D/cNUQilclu64t56xpM1aVnKVrfWsvBTq/t2Xxm+HPbcjrnSGdU0iMiIuU3qz2Rmb0c+DPgN5xzIznb280s7P++FdgBTGwXIotWx0Cc9sbqBess0lDtZcb3dwzyrhfvpDoSJhwy/vG3LiOWTNMzSTlPoK2uip6RZLakp2YGAX+Q6Z1thh/g1Ves5w1XbeLCtc3TX3mB/dELt1NXFeGjP36avSf7ALh4/dg4d61pJJVxHOke6//bO+xn+BfZAcxcBRN3gXnN8Edy2nIGq+2qpEdERMqtmLacXwPuA843s5Nm9lbg34BG4Cfj2m8+D9hrZo8C3wbe5pzrKXS/sjidHRhd0HrrICO/c3UDr75ifXb7tvYG3nfjBWxsq+UF509e8tVaX0XvcO6k3eID/qu3tLGtvT67cuxsrGqs4UOvvaTouQMLqa2+ij+4fiu37+vglodPsXVlfd56AbvWeNV2uXX8/X6Gf6q2nEtREPCb5Qf/pRbNacsZrLartpwiIsvD8XPDPHair9zDmJVpC4+dc28osPkzk1z3FuCWuQ5KyufsQDxb274Qgi4vf/qyXdkJj4GbnruZm567ecrbt9ZFOd0XHyvpiRZfS//Si9bw0ovWTH/FJeyt12/hi/cd42j3MK+6fF3eZVvb64mGLa9TT1DDv/xKerzP9LrmWqoi8xeAB+U7qZwMv9pyiogsD8//x7sBOPbhV5Z3ILOg1JPk6RxcuEW3AJ63s50f/K/reMmFq2d1+9a6KnpHEsT8Pvw1VfpI52qojvD2F24HvP77uaLhENvaG/J68fcu20m7XlZ/Y9v8ZfdhLMOfTDtSfi9+BfwiIlJui6u1iJTVaMqrmV/Ikp5wyLho3ezr39vqq+jJm7Srj/R4v3vNJkYSKV6TUzIV2LWmkQeOjlXd9cUSVIVDM+p2tBQEq+3O54Rd8FYhDoeMdMZle/GrpEdERMpNeyLJ6hpc2B78pdBaX8VoKkPPsFd7vtwC1VKojoR5xw07WFFgkazz1zRxuj9Ov5/Z7x9J0lIXnbeVaMulqSbCyy9aw4svmN2ZpJmIhIxkJkMyo0m7IiKyOCjgl6xg0a1VZVwkaaba/FrzU30xqiKhCfMAZGpXbGoB4H9+/RHODY3SO5JYduU84GXe//ONz1qQORuRkJHKK+nR16yIiJSX9kSS1RksutW4dAL+1vqxgH8mHXrEc/WWNv7u1RfzqyPneOXH7+HJMwO01C6vCbsLLRIO5U/aVYZfRETKTAG/ZI2tsrt0Snra6r1s9KnemMp5ZsHM+O/XnMd3/sdzqY6GONETW5YZ/oUUDRvJnLacUdXwi4hImWlPJFlnB0eJhm1JtWQMxto1NLooe+EvFRevb+b7//M63vzczbz2yg3lHs6SFgl5Gf6U2nKKiMgioZYmknW2P86qxhpCSyhAafNLepzThN25aqyJ8oHfuKjcw1jyImGvhj+ZVoZfRGQ5iifTnOwdYfuqxnIPpWjaE0nW2cH4kirnAWiqiRIcn6iGXxaDaDjkl/Sohl9EZDl659cf4cX/9HNG/DWAlgIF/JJ1dmB0QXvwl0IoNFaCVKse/LIIeF16xkp61DlKRGR5ue/wOQCSKVfmkRRPAb9knR2IL7mAH8Y69dRG9XGW8ouEQ6Q0aVdERBYR7YkEgJFEisF4ilVLrKQHxnrxa5VdWQzGZ/g1aVdERMpNAb8A0OkvurWUevAHWv3WnDWatCuLQCRspDKatCsiIouH9kQC5PbgX3oBf9CpR5N2ZTGIhkIk0xlN2hURkUVDAb8AXg9+WFqLbgVa6xTwy+Ixvi1nJKSvWRERKS/tiQSATj/Dv2oJZ/hV0iOLQSRoy+nX8EeV4RcRkTJTwC8AdPTHqYmGaKpZehNfleGXxSQaTNr1u/REVMMvIiJlpj2RAF5Jz+qmGsyWXjayLduWUwG/lN9YSY+f4VeXHhGRZalraLTcQyiaAn4Blm4Pfsjpw68MvywCXh/+DOmMFt4SEVmOguToi//pZ9x7qLvMoymOAn4BvBr+pRrwX7C2kZuecx7Xbl9Z7qGIeH34c9pyqqRHRGT5+p1P38+JnpFyD2Na2hMJzjnODoyyunHpdegBqI6E+etXXczKhqU5flleIqEQqbTLtuXUpF0RkeWtbyQ5p9s750o0kskp4BcGR1PEkuklm+EXWUyiYfP68Kstp4hIRVuIQL5Y2hMJZ/uDlpzKkIvM1dhKu8rwi4jI9EZTmXl/jKICfjP7rJl1mtkTOdvazOwnZnbQ/7fV325m9nEzO2Rme83syvkavJTGke5hAM5bUV/mkYgsfZHsSrsZwiFbkp2vRERk4ZwbTsz7YxSb4f888PJx294D3OGc2wHc4f8N8Apgh/9zM/CJuQ9T5tOhziEAtq9qKPNIRJa+qN+WM5V2RNShR0REFoGiAn7n3M+BnnGbXwV8wf/9C8Crc7Z/0Xl+BbSY2doSjFXmydNnB1nfUktD9dJbdEtksQnacibTjqg69IiILDuLqTa/WHPZG612zp3xf+8AVvu/rwdO5FzvpL9NFqmDZ4eU3RcpkWjISKZdtqRHRESk3EqSfnLeoc6MDnfM7GYz22Nme7q6ukoxDJmFdMZxuGuInasV8IuUQtB3P5HKaMKuiMgyVOzcrMV0ImAuAf/ZoFTH/7fT334K2JhzvQ3+tjzOuU8653Y753a3t7fPYRgyFyd6RhhNZdixqrHcQxFZFoKsfjyZVktOERFZFOayN7oVuMn//Sbgeznb3+R367kG6M8p/ZFF5mAwYVcZfpGSCLL6sWSaiDL8IiIyjYXYUxQ1S9PMvga8AFhpZieB9wMfBr5pZm8FjgOv869+G3AjcAgYAd5S4jFLCT19dhCAHarhFymJIKsfS2Y0aVdERKa1EJU/RQX8zrk3THLRiwpc1wFvn8ugZOEc6hxibXMNjTXRcg9FZFkIMvzxRFptOUVEKsD/+MpDbF5Rz5d//+pZ3X4hav2VfqpwT58dZMdq1e+LlEowadcr6amMr1gtzigilexkb4x7DnWXexhTqoy9kRSUzjgOdQ6pnEekhIKsfiyZrqQuPZ9HizOKiORZRE16FPBXslO9Mb9DjwJ+kVIJ6vZjFVTSo8UZRUQWNwX8FSw7YVclPSIlE3TmGU1VfFtOLc4oIrJIVPTeqNJlW3Iqwy9SMpFsH/6M2nL6ZrM4I2iBRhFZnIpcd6toP3myg83v+QHDo6nS3nEOBfwV7ODZQdY01dBcqw49IqUSZPVHEqmKmbQ7iTktzghaoFFEKkP3UAKA4+dG5u0xKnpvVOkOdg6xQwtuiZRUkNXPOIhWSA3/JLQ4o4hUpN7hBJvf8wO+v/d0uYeSpYC/QmX8Dj0q5xEprdzFtiqlpMdfnPE+4HwzO+kvyPhh4CVmdhB4sf83eIszHsFbnPFTwB+VYcgiIvPmcJdXMv25Xx4r70ByFLXwliw/p/pixJJpdmrCrkhJ5XbmqZSSHi3OKCIykdpyStkd7PQ79CjDL1JSuUF+hZf0iIjIIqGAv0I9fdY73bRjlTL8IqWUu9hWpWT4RUQqiVtMqfsiaW9UoY52DbOyoZrmOnXoESml3N77FbTSroiILGIK+CtUXyzBivqqcg9DZNnJDfLDKukREZFFQAF/hRqIpWis0ZxtkVLLDfIrfKVdEZFlqdQLby0E7Y0q1OBokiYtuCVScrltOVXSIyJSwRZRsb8C/gqlDL/I/Iho0q6IyLK2iOL4omlvVKEG40kF/CLzIG/Srmr4RUSWnf5YstxDmDEF/BXIOcdgPEVTjUp6REpNbTlFRGSx0d6oAsWSaVIZR6MCfpGSyw3yI6rhFxGRRUABfwUajKcAaKpVSY9IqUVyynii6tIjIiKLgPZGFWjArz1Thl+k9HK79KgPv4hI5VpMc3sV8FeggSDDr0m7IiWXG+OrLaeISOX47iMnyz2ESSngr0ADcWX4ReaLmWUDfU3aFRGpHO/6xmO4Rdqzc9YpXjM7H/hGzqatwF8BLcAfAF3+9r9wzt0228eR0gtq+JtVwy8yLyKhEMl0Oq+eX0REpFxmHfE55w4AlwOYWRg4BXwXeAvwz865/1uKAUrpDSrDLzKvImGDZH49v4iISLmUam/0IuCwc+54ie5P5tFAzMvwa+EtkfkRBPpqyykiIotBqQL+1wNfy/n7HWa218w+a2atJXoMKZHBeJJIyKiNhss9FJFlKSjliagtp4hIxVpM5fxz3huZWRXwG8C3/E2fALbhlfucAT46ye1uNrM9Zranq6ur0FVkngzEkzTWRDBT9lFkPgQZfnXpERGRxaAU6adXAA87584COOfOOufSzrkM8CngqkI3cs590jm32zm3u729vQTDkGINxlM01ap+X2S+RNSlR0SkIi2mrH6uUuyN3kBOOY+Zrc257DXAEyV4DCmhgVhS9fsi8yicLelRhl9EpFI9fqq/3EPImlPUZ2b1wEuAP8zZ/A9mdjneAmPHxl0mi8BgPEWTOvSIzJuoX7uvgF9ERBaDOQX8zrlhYMW4bW+c04hk3g3GU2xeWVfuYYgsWyrpERGRxUR7owrkTdpVhl9kvkQ0aVdERBYRBfwVSCU9IvMrqracIiKyiGhvVGHSGcfQaEqTdkXmUVDSowy/iEhlWaRNehTwV5qhuLfKrtpyisyfsZV29RUrIiLlp71RhRmIJwGU4ReZRxG15RQRkUVEAX+FCQJ+1fCLzJ+xSbv6ihURkfLT3qjCDAYlPcrwi8ybILMfVoZfREQWAQX8FWYg5mf4VcMvMm/UllNERBYTBfwVJsjwq4ZfZP5k23KqpEdEpKI4tzj79GhvVGHGJu0qwy8yX7Ir7aqkR0REFgEF/BVGGX6R+adJuyIisphob1RhBuNJaqNhBSIi8ygaMsw0aVdERBYHRX0VZiCWoqlW2X2R+RQJh4iG9PUqIiKLgyK/CjM4mlT9vsg8e9Z5rZzqjZV7GCIissAW55RdBfwVZyCWUg9+kXl24yVrufGSteUehoiICKCSnoozGFeGX0RERKSSKOCvMAPxlDr0iIiIiFQQBfwVZjCe1Cq7IiIiIhVEAX+FUYZfREREpLIo4K8g8WSaRCpDk2r4RURERErOLdI2PQr4K0iwyq669IiIiIhUDgX8FWQgngRQDb+IiIhIBVGqt4IEGX7V8IvIQjGzY8AgkAZSzrndZtYGfAPYDBwDXuec6y3XGEVEljtl+CvIQMzL8KsPv4gssBc65y53zu32/34PcIdzbgdwh/+3iIjMkzkH/GZ2zMweN7NHzWyPv63NzH5iZgf9f1vnPlSZq7EafgX8IlJWrwK+4P/+BeDV5RuKiMjyV6oMv7I3S8BgPMjwq6RHRBaMA35sZg+Z2c3+ttXOuTP+7x3A6vIMTUSktByLs03PfEV+rwJe4P/+BeBu4M/n6bGkSJq0KyJlcJ1z7pSZrQJ+Ymb7cy90zjkzK7iH9A8QbgbYtGnT/I9URGSZKkWGf1bZGzO72cz2mNmerq6uEgxDpjMYTxEyqK8Kl3soIlIhnHOn/H87ge8CVwFnzWwtgP9v5yS3/aRzbrdzbnd7e/tCDVlEZNkpRcB/nXPuSuAVwNvN7Hm5FzrnHEw8v6Ev8oU3EEvSWBPFzMo9FBGpAGZWb2aNwe/AS4EngFuBm/yr3QR8rzwjFBGpDHMu6cnN3phZXvbGOXdmquyNLKzBeEr1+yKykFYD3/WTDBHgq865283sQeCbZvZW4DjwujKOUURk2ZtT9OdnbELOucGc7M3fMJa9+TDK3iwaA/GUOvSIyIJxzh0BLiuw/RzwooUfkYhIZZprulfZmyVkIJ5Uhl9ERERknrjF2aRnbgG/sjdLy2A8xfqW2nIPQ0REREQWkFbarSADsSRNtcrwi4iIiFQSBfwVZDCeVA2/iIiISIVRwF8hMhnH4GiKJtXwi4iIiFQUBfwV4txwAuegpa6q3EMRERERkQWkgL9C7D3ZB8DF65vLOxARERERWVAK+CvEw8/0EgkZlyjgFxEREakoCvgrxMPH+7hgbRO1VeFyD0VEREREFpAC/gqQzjgeO9nHlZtayj0UEREREVlgCvgrwIGOQUYSaa7Y1FruoYiIiIjIAlPAXwEeOdELwJUK+EVEREQqjgL+CvDw8T5W1Fexsa223EMRERERWbacK/cIClPAXwEeOdHLFZtaMbNyD0VEREREFpgC/mWubyTBka5hrjyvpdxDEREREZEyUMC/zD1yog+AKzaqfl9ERESkEingX+YeOd5LyOCyjVpwS0RERKQSKeBf4tIZx0PHeya9/JETfexa00RdVWQBRyUiIiIii4UC/iXuR/s6+G+fuI9DnYMTLstkHI8+06f6fRERWZTiyTSn+2LlHoZIyTgWZ5seBfxL3PFzIwAc7hqecNnBziEGR1Oq3xcRkUXjTH+Mv/mvJ0lnHG/53IM898N3lntIIsue6jyWuI5+LzNyomdkwmWPPOMvuHWeAn4RkUqy/S9uo6Uuyp6/fEm5hwJAMp3h+LlhVjfV8M6vP8oDR3t43s6V3HfkXLmHJlIRFPCXQCbjuPWx07zy0rVEwwt70uR0fxyAZwoE/AfODlJfFWbziroFHZOIiJRXKuPoHkqUexhZH7ptP5/95dG8bW/+3INlGo1I5VFJzwz0jSTIZCbWZj14rIc//saj3PHU2QUfU8cUAf/R7mE2r6zXglsiIjJv9p7sY/N7fsCx7omlpYE9UzSXEJH5p4C/SAPxJM/98J3c8vDJCZcF9fNBPf1COjNNwL9lZf1CD0lERJa50VSaD9y6j76RBLc85O0X7z7Qmb3cOceP9nXw/b2nyzVEEclRkQH/P/5oP/9x96EZ3eZw5xAjiTSPneybcNnR7iGgcNA9W1+9/xnec8veKa8zmkrTPTRKOGSc7InlnX1IpDKc6BlhqwJ+ERGZgf/z/57gvsNebX3u/LBUOsObP/cADx3v5ZaHTvH5e4/xodv2F7yPL//qOH/4pYd4x1cfWZAxiywWbnE26Zl9wG9mG83sLjN70sz2mdk7/e0fMLNTZvao/3Nj6YZbGrc8dIov3XccN4N35ah/qvJw58RTlke7vS/EUgb8/++RU3z9wRN0DsQnvU7nwCgAF69vJpHOcHZw7LonekfIONisgF9ERGbgS786zhs+9Sve/Y1Huf4f7sqWq57qi3H3gS7e9Jn7+fe7vKTZNx86UfA+7j7QtWDjFZHpzSXDnwL+xDl3IXAN8HYzu9C/7J+dc5f7P7fNeZSz0DeS4FSB3r4jiRQdA3HO9Mc52Vt879+gNvFw19CEy4IMf6FOObPhnGN/xwAw9ZdmUM5z9ZY2AJ7JKSk66pcZqaRHRESKlc45U/ydR04B8OTpgbzrDCfS2f3rYs1miki+WQf8zrkzzrmH/d8HgaeA9aUaWLGGRlMFt7//1n288TP3T9h+rHssKH7gaPGTiI74AX/n4CgD8WR2eyqd4ZmeEUIGJ3tjeV+Ws9UxEGcg7j2vO/ZPPhH4jN+S86rNfsCfc8ARnJFQwC8iIsV64lT/pJcZxTeAUK8IkcWlJDX8ZrYZuAIIIux3mNleM/usmc1bE/iP/vgAV//9TwsG2Y+e6ONI1zAjifwDgqM5XQRmEvAf7R6mym+5eSRnkavTfXGSacflG1tIZVw2CJ+L/R3eqrk7Vzdwz8FuRlPpgtcLMvy7N7cSMjiRc8biSPcwbfVVtNRVzXk8IiIik1GSX2Txm3PAb2YNwC3AHzvnBoBPANuAy4EzwEcnud3NZrbHzPZ0dc2u1u+8FfUMJ9LZkprAQDyZ7ZhzqDP/smPnvGD92u0reOBYcQG/c45j3cNcs20F4E3gDRzxH/v5O1cBpanj33/GC/jf9vxtDCfSPHi0t+D1OvrjNNZEaKmrYm1zbV5J0bHuYfXfFxGRGSmUmZ8uoP/ifcfnZSwiUjpzCvjNLIoX7H/FOfcdAOfcWedc2jmXAT4FXFXots65Tzrndjvndre3t8/q8S/b0AzA3pP5pyCDgBng4Nn8gP9I1zBrmmp4/s52jnYP0zk4+aTYQNfgKMOJNM/f2U4kZHl1/MEZg+ftXAmUpo7/QMcAa5treMXFa6mOhCYt6zndF2Ntcw0Am9rqJpT0bFnZMOexiIjI0rbvdD/ffLDw5NpijKbSXP3Bn3LnFCWmE6mmRyrTYj3jNZcuPQZ8BnjKOfdPOdvX5lztNcATsx/e1La2N1BXFZ4Q8O873e+PBZ7uHMy77Gj3EFtW1nPVFi9bP1n2PFdQv79zdQObVtTlBfzHuodprI5wyfpmIiErTYa/Y5Dz1zRSWxXmOdtWcOf+zoIdhToG4qxprgXyA/7hUW9i8tZ21e+LiFS6V378Hv5smjbPUzndF+fswCh/+4OnSjgqEVlIc8nwXwu8EbhhXAvOfzCzx81sL/BC4F2lGGgh4ZBx8bpm9o7rjf/k6QFWNlSxc1Ujh8Zl+I92D7OlvZ6L1jVRGw3zwNFz0z5O0KFn84p6trU3ZBfaAu9gYEt7PZFwiPWttTzTM7ca/mQ6w+GuIc5f0wjAi3at4vi5kexBR64z/XHWBRn+FXV0DY4SS6SzZUubVyjgFxFZjjr64wzmNJCYTDw5Ngfsdf/ffXzl/qnLb2YyMVdElo65dOm5xzlnzrlLc1twOufe6Jy7xN/+G865M6Uc8HiXbGhm3+kBUulMdtu+0wNcsLaJHasb8jL8fSMJekeSbFlRTzQc4lnntXJ/ERN3j3YPUxUJsa6llm3tDRw/N5x9vKPdw9nAenxZzWwc6RommXZcsKYJgBfu8uYG3LW/M+96iVSG7qFR1vgB/8Y2r17/RO9IthOROvSIiCxP13zoDl72zz8veFkiNbY/3PV/bs/+/sDRHt733alPuheq4S/2EMBybqwuPSKLy5JfaffSDc2MpjI87WfyE6kMBzsHuWhdMztWNXKyN5bt1DO+VeVVW9o4cHaQ/pGpsyRH/Amw4ZCxrb2eZNpxojfGaMrrRRzc38a2ujnX8Af994MM/4bWOs5f3cid4wL+swNxnCOvhh+8XvzBJObNKzVpV0RkuTrtd2o72TuSl+0/0j1xvZiFMJPFLEVkYUXKPYC5unRDCwCPn+rjwnVNHOocIpl2XLiuiWjIcM5bHfeSDc1jAX/7WMDvHOw53sOLLlgNQOdAnKbaKDXRcPYxjnUPZ4P6bau8ibCHO4dIpTM4R7ZWflNbHT3DCQbjSRprorN6Pgc6BomEjG3tYxNuX7hrFZ/+xREG4kma/Pvt8Ffgza3hB69L0JHuYdY211BXteTfXhERyfHth07SVDP23X6qL8Z1H7kLgOt3rGRDax1vvOa8cg1PpOIt1gPfJZ/hP6+tjsaaSHbibjBh96J1XkkPwEG/rOdY9zAhg42tXnB8+cYWomHjgaM9OOf4zD1HufYjd/J3P3gye//pjOP4uZHsQcI2v/PN4a6hbF19cDAQBN0n5lDHf6BjkK3t9VRFxt6aG3atIpVx/PJgd3Zb0IM/qOFvrYvSUB3hmZ6RvDIjERFZPv73tx7j5i89lP37n3/ydPb3Xxzs5msPPMMHbt1XjqHll/SUZQQiMpklH/CHQsYl65t53F8d8MkzA9RGw2xeUc95K+qJho2Dft/8I93DbGyrywbTNdEwl21o4ecHu3nblx/ib7//JNWRMN/fe4akX6N/ui9GIp1hqx/UN9dFWdlQzeGuobHJvEFJT+tYln229ncMssuv3w9csamF+qowvzycE/D7y5oHNfxmxobWWk72jnhnJNShR0Rk2SsUWPeOJBZ8HJCf2VQNv8jisuQDfvDKep46M8BoKu1P2G0kHDKi4RBbVtZz8KyX4T+aU5oTuGpLG0+dGeCOpzr5y1dewEdfdxl9I0nuO3wuexvI73izrb2ew13DHO0eZmVDVbbMZizDP7uAfyCe5FRfLFu/H4iGQ1yzdQW/PDTWUehMf5yG6khe6dCmtjr2nuyndySZPUAREZGl6XO/PMrm9/yAZDpDIpXh4vf/aMJ1vvXQyZI+5mwW3hKRxW+ZBPzNJNOO/WcGeer0ABeuG8uQ71jVyMHOoexqueNLXV5zxXqu276Sb/zhc/j967fy/J3t1FeFue1xr7nQ+Lp/8Or4D3V6JT25BxDNdVGaaiKzzvA/3eEdmOwaF/ADXLt9JUe7hznZ6913R388O2E3sKmtjs7BUUAtOUVElqpEKsNgPMlHf+yV64yMptn5lz9kaDRV1O1LHaArWS+y9C2LgP+S9d6Ku7c9fobB0RQXrWvOXrZjdQPP9IzwTM8Iw4n0hMWodqxu5Mu/fzXPOq8V8Mp8XnTBan60r4NkOsPR7mEaqiO0N1Rnb7OtvYH+WJInTvVPCKw3rZjYmjOdKe7rd38Q8K9tmnDZdTu8lXzv9bP8Z/pj2XKe3McOqKRHRGTp6I8liSXS7Dvdz86//CGXfODH2QA/WFtlscuv4ddhgshisiwC/g2ttbTWRfm2f2rzwrX5GX7n4KdPeW0ti+lNf+Mla+kdSfKrI+e8lpwr6/K+yLb5wfRIIj0hsN40rjXn7U+c4Yq/+TH35Ey4ncyBjkEaayLZibi5dqxqoL2xmnsOefdzpkCGP+jFHw5Zdj6BiIgsbt995CSX/fWPueCvbueVH79nzvc3my4hqXSGWCI9/RVL/LiFfOT2/dmz7CJLzWItgVsWAb+ZccmGFs4NJwiHLK8GPujU8+N9HUBxpS4vOD8o6+nwW3I25F2e2zJzfK38xrY6TvbGSGcciVSGD962n4F4ird9+SGePD0w5ePu7xjg/NWNeQcXuc/xuu0r+eWhbhKpDF1Do9mWnIFgDsHG1tq8Lj8iIrI4dA7E+c7D+XX37/rGY1PeJjPDQHo2AcdbPv8gF/zV7dNfcR79eF8H8WSaT9x9mD/6ysO8+t9/yU+ePMvPnu4q67hEloNlExVetsEr49nWXp/XQ3/zinoiIePBYz3Z1XKnUxMNc8MFq7n9iTOc7B2ZcFZgfUst1X5AvXnlxAx/Ip3h7ECcbzz4DM/0jPDB11xCQ3WEt3z+AU753XUyGcevjpzj3+86xL/deZB/u/MgT50ZnDBhN9e121dybjjBLw524RwTzgSsb6nFbOKYRERkcfjvn7mfd3/zMfpjUy/4mKvIqtA5+YV/FvpjPz044bInz0ydrAoUSlYB/Pq/3pNtnT2Zx070cfOXHuKv/2uspeijJ/r4gy/u4abPPlDU44vI5JbNykxBHX9u/T5AVSTE5pX1HOocyq6WW4xXXrKG/3rsNABbxq1YGwoZW1bWs79jcGINv59lP9AxyMfvPMRVm9t4w1UbufK8Fn7rE/fx5s8+wCsvXcstD5+c0K8/ZF5QP5lrt68AyJYuja/hr4mGuWbLCq6b4j5ERKQ8vnDvseyq8NFw8TXuMy6VmebqqXSGSNhLWqUzjhs+enf2sp88eXbC9YP5ZcXMR0tnHO+/9QlO9o2VtgZts6fS47cSnW4dmxM9IwyNpugbSfLwM728/YXbOdMfY23z9Mk8kfGcc8SS6ZIuVHrpB37Mf/73K0t2f6WybAL+yze2EA5ZNtOfa+fqBj/gLz7z/YLzV1FXFfbq9MeV9IA3T2A0lck7mwBjAf+HfvgUXYOjfOJ3r8TM2LWmif/vjc/ips89wMd+epBrt6/g3S/ZyUsuXJM9W2CQ/RIuZG1zLVvb6/npU2ezf4/3tZuvKfo5iohI6R3tHqZnOJFtBhH46I8PZH+PhELce6ibZ29pm/b+fvM/75vR408Xlm9/3w951eXr+N6jp2d0v9N5/637eP8sF/16y+ceBMjOUxvv7ECcbz54go/mLDQG3r7/dz99Px/77ct59RXrZ/XYsvR9f+9p3vfdJ9jzly8mOkUcNd7XHjjBX3z3cX72py/gvBJ2NzzoH9gvJssm4F/VVMP3/+d1efX1ge2rGoGOGXWuqYmGuWHXKr6/9wxbCnwI/s+vXViwRdq6llpCBk+fHeJFu1axe/PYl/lzt6/kB//reuqqwmyY5aTa67av5Iv3HQdgbcvEyb0iIkuFmb0c+BcgDHzaOffhMg9pWrFEmrRzPHGqn6v9YD23lMU5xwv/790AfOy3L6e9sZrVTdWMpjJURcKAt9/Y+Zc/XOih5yl1sD/f/vBLD/Hoib4J23/30/cD8MffeJTLN7bw06fO0t5YzeUbWyYEcLFEms/+8ig3P2/rjILCSvPgsR7WNtcwmvIWHZ2sVKtYwRmq6e7nZO8IdVUR2uqrJlx2qi9GPJkuGOMBvOOrjwBw/l/+kIyDb7/tOXnx172Hu3nqzCArG6p4wc5VNNd5axjd7s/vPNI9zP6OQdb5idX66rHweDSV5t/uPMQfvWB70c95/IFpseKpuU2cn8qyCfgBLijQzhK8DD9MnGA7nT9+8U6evbkt+8HI1VpfRWuBD2U07M0TONUX43+/7PwCY5m8Rr8Y1/oBf31VmMbqZfX2iUgFMbMw8O/AS4CTwINmdqtz7slSPs6eYz0c7R7mhl2r6B5KcP6aRk71xejoj/Gs89roHIjzjQdP8Ecv3M63HzrBn9/yOPv/9uV88Lan+OJ9x/nAr1/I1VtX8Ip/+cWMH/uPv/FoKZ9K0frKtNLufCoU7I/3Av9AK3D5xhYuWNtE30iCR0/0caY/DsA//ugAH3/DFVy4ton+WJJnnddKOuPIODerA4FkOkPGOaoj4emvPI2fPnmW3ZtbSWUcVZFQdmHPYsSTaQ51DpHKOD5421P89u6NHO8Z4Q+u30JjTZRTfTGqIyFW5rQZzzUYT7LnWC9v+fyD2W037FrF+3/9QgbjKZpro3z1gWf4xN2HaayJ8FvP2shf/fqFE+7n0784wt/94Cm+/NaruW7HSra/74fZcrC/e/XFtDdW87KL1tA3kqChOsK+0wM88kwvH/ivJ6mKhPj337mScAhu2LWaze/5Qd59/+2rL2aDP1+xvbGabe0NeZUWQdXZR27fz5d//2qioRAO+J1P3Z93P1/9g6t57raVZHLK1P7wSw9lf3/ni3bwrpfsBOBr9z/Dv955iH+981AR78LcfOLuw3zqTbvn5b6tVG205mL37t1uz54983b/nYNx/uCLD/Gvr78ir1f9fPnQD58C4L2vuKDk990fS3LF3/yYLSvrueNPXlDy+xeRxcfMHnLOzc9eoEzM7DnAB5xzL/P/fi+Ac+5Dha4/2/3En397L9995BSJdGYuwxUBYGVDNd1Do/zGZeu4YG0TH7l9/4KPYUV9FeeGl99BnXiLwf7zb18+49sVs4+oiBTxqsYavvf2axfs8eYj0A8010a5essKWgqcdRARWULWAydy/j4JXJ17BTO7GbgZYNOmTbN6kFddsY5v7Dkx/RVFitA95K1mf+tjp7n1sfKURSnYXx6u37Ey2x0r8N4bd83b41VEwL/cfOqm3RTZbEhEZMlyzn0S+CR4Gf7Z3Mdzt63kF3/2QvpjScy8s6SGsba5hlQmQ/dQgvUttTx4rIdLNzQTDoUYHk0RCRvntdVzbniUvpEkG9vq6B9JMpxIsaK+CgckUhma66IMxJKkM46WuiriyTQjiTTtjdX0jSSIJzM01UQYSaQJmbGmuYah0RSxZJqmmghDoynCZjTVRjnRM8JIIs3qphp6hhP0DCfYsbqB4+dGSGccjTXeLjvjHA3VEc4OjDKaSnPeinrq/SYTo6kMPcMJNrbVsr6lluPnRmioidBWV8WZgThV4RCRkDEYT9E1FGdNcy01kRDJtKOlLkoskSaWTNNc6yWV4sk0tVVhUhlHPJFmZUM1wwmvS87qphqiYWMkkSaeTBNPZchknP/aOlL+ejTpjKOuKkw0HPJLX0I453XmaayJ+OU0UBUOEU+l6RnytrfUVXG0e5j66jDnhhJ0DY1y0bomIqEQ54ZGaaqNMhhP0VQT4ZETfWxqqyNkRm00zOBokjN9cUaSada31HLPwW6ef347kZBxuGuIuqoI9dVh/3WN0tEfpypiZDIQjYTo6I+xvqWOptoIITPODScwoLYq7L/XUXqHkyT8BcvCIfPLU/pprImwfZX3vtVXR0ilvY/uqb4Rdq5upDoSJpnOMDyaIpHOkEhlcEBbXRXrWmqJJdOc7B1hXUstNZEw3cOjpNOOrqFRVtRXEQ4Z6YyjOhomkcpwbmiU+uoIDTURcN7nI5HK0FgTxcwr1VnVVENVOMShziFqoiEG4ilqo2F6RxJcsr6ZWDJN2Iwj3cOsbqomkcqwvqWOkUSK0VSGzsFRVjdVMxhPsaG1liNdw6xqquZY9wiRkNFcF8U5r94dvPdyS3s9Q/EUybQjnkxTFQkRDYcYjCeJJdNUhUNUR8MMxpM45y0aOppKMxBLcYFfblUdCZHKZKiNep/9zkGvLKutvopoOEQ4ZKTSjqHRFOetqGNoNEV/LMloMs3QaJpVjdX0DCdY1VhNU200+9rEkxmOdg+xvrWWdMbr6BhLpGiqiTI4mmJTWx1Hu4epjniPEUukWdlYzZGuIX7jsvUc7xlmRX017Y3VE7podQ8l6OiPU18dZsvKeoZGUzTWRHHOzXk+xFxUREmPiMhSppIe7SdERCZTzD5C09RFRKQcHgR2mNkWM6sCXg/cWuYxiYgsSyrpERGRBeecS5nZO4Af4bXl/KxzbnZN3EVEZEoK+EVEpCycc7cBt5V7HCIiy51KekREREREljEF/CIiIiIiy9i8Bfxm9nIzO2Bmh8zsPfP1OCIiIiIiMrl5Cfhzlkx/BXAh8AYzm7j+soiIiIiIzKv5yvBfBRxyzh1xziWArwOvmqfHEhERERGRScxXwF9oyfT18/RYIiIiIiIyibJN2jWzm81sj5nt6erqKtcwRERERESWtfnqw38K2Jjz9wZ/W5Zz7pPAJwHMrMvMjs/ysVYC3bO87XKl1ySfXo+J9JpMtJhfk/PKPYBye+ihh7q1nyianu/ypue7vM3m+U67jzDn3OyGM9WdmkWAp4EX4QX6DwK/Mx+rKJrZHufc7lLf71Km1ySfXo+J9JpMpNdk+aq091bPd3nT813e5uv5zkuGX0umi4iIiIgsDvNV0qMl00VEREREFoHlsNLuJ8s9gEVIr0k+vR4T6TWZSK/J8lVp762e7/Km57u8zcvznZcafhERERERWRyWQ4ZfREREREQmsaQDfjN7uZkdMLNDZvaeco9noZnZRjO7y8yeNLN9ZvZOf3ubmf3EzA76/7aWe6wLzczCZvaImX3f/3uLmd3vf1a+YWZV5R7jQjGzFjP7tpntN7OnzOw5lf4ZMbN3+f9nnjCzr5lZTSV/Rpaz5bCfmOl3vXk+7j/nvWZ2Zc593eRf/6CZ3VSu51SMYr/Hzaza//uQf/nmnPt4r7/9gJm9rExPZVoz+Z5eDu/vTL6Dl+L7a2afNbNOM3siZ1vJ3k8ze5aZPe7f5uNmZtMOyjm3JH/wuv8cBrYCVcBjwIXlHtcCvwZrgSv93xvxWqFeCPwD8B5/+3uAj5R7rGV4bd4NfBX4vv/3N4HX+7//J/A/yj3GBXwtvgD8vv97FdBSyZ8RvFW/jwK1OZ+NN1fyZ2S5/iyX/cRMv+uBG4EfAgZcA9zvb28Djvj/tvq/t5b7+U3xvIv6Hgf+CPhP//fXA9/wf7/Qf8+rgS3+ZyFc7uc1yXMt+nt6qb+/M/0OXorvL/A84ErgiZxtJXs/gQf865p/21dMN6alnOG/CjjknDvinEsAXwdeVeYxLSjn3Bnn3MP+74PAU3j/kV6F9+WB/++ryzLAMjGzDcArgU/7fxtwA/Bt/yoV85qYWTPeF89nAJxzCedcHxX+GcHrUFZr3pohdcAZKvQzsswti/3ELL7rXwV80Xl+BbSY2VrgZcBPnHM9zrle4CfAyxfumRRvht/jua/Dt4EX+dd/FfB159yoc+4ocAjvM7GozOJ7esm/v8zsO3jJvb/OuZ8DPeM2l+T99C9rcs79ynnR/xcpYn+1lAP+9cCJnL9P+tsqkn+K6wrgfmC1c+6Mf1EHsLpc4yqTjwF/BmT8v1cAfc65lP93JX1WtgBdwOf8U+OfNrN6Kvgz4pw7Bfxf4Bm8nUw/8BCV+xlZzpbdfqLI7/rJnvdSej0+RvHf49nn5V/e719/qTzfmX5PL+n3dxbfwUv9/Q2U6v1c7/8+fvuUlnLALz4zawBuAf7YOTeQe5l/9FcxrZjM7NeATufcQ+UeyyIRwTut+Ann3BXAMN6pxKwK/Iy04mVUtgDrgHoWbxZMJKtSvusr8Hu8or6n9R1cnvdzKQf8p4CNOX9v8LdVFDOL4u0AvuKc+46/+ax/ygf/385yja8MrgV+w8yO4Z2+vwH4F7xTZMFCc5X0WTkJnHTO3e///W28HUslf0ZeDBx1znU555LAd/A+N5X6GVnOls1+Yobf9ZM976Xyesz0ezz7vPzLm4FzLJ3nO9Pv6aX+/s70O3ipv7+BUr2fp/zfx2+f0lIO+B8EdvizuqvwJnLcWuYxLSi/hu0zwFPOuX/KuehWIJjNfRPwvYUeW7k4597rnNvgnNuM95m40zn3u8BdwG/6V6uY18Q51wGcMLPz/U0vAp6kgj8jeKeRrzGzOv//UPCaVORnZJlbFvuJWXzX3wq8ye/+cQ3Q75cS/Ah4qZm1+lnWl/rbFpVZfI/nvg6/6V/f+dtf73d52QLswJvsuKjM4nt6Sb+/zPw7eEm/vzlK8n76lw2Y2TX+6/cmitlfzXTm8WL6wZvZ/DTezOz3lXs8ZXj+1+GdEtoLPOr/3IhX23YHcBD4KdBW7rGW6fV5AWPdHbbifREcAr4FVJd7fAv4OlwO7PE/J/8Pb7Z/RX9GgL8G9gNPAF/C6/JQsZ+R5fyzHPYTM/2ux+vc8e/+c34c2J1zX7/nf8YPAW8p93Mr4rlP+z0O1Ph/H/Iv35pz+/f5r8MBiuhkUsbnWfT39HJ4f2fyHbwU31/ga3jzE5J4Z3DeWsr3E9jtv3aHgX/DX0h3qh+ttCsiIiIisowt5ZIeERERERGZhgJ+EREREZFlTAG/iIiIiMgypoBfRERERGQZU8AvIiIiIrKMKeAXEREREVnGFPCLiIiIiCxjCvhFRERERJax/x/69NfX8sNkFgAAAABJRU5ErkJggg==", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "num_frames = 10000\n", - "batch_size = 32\n", - "gamma = 0.99\n", - "\n", - "losses = []\n", - "all_rewards = []\n", - "ep_reward = 0\n", - "\n", - "state = env.reset()\n", - "for frame_idx in range(1, num_frames + 1):\n", - " epsilon = epsilon_by_frame(frame_idx)\n", - " action = current_model.act(state, epsilon)\n", - " next_state, reward, done, _ = env.step(action)\n", - " replay_buffer.push(state, action, reward, next_state, done)\n", - " \n", - " state = next_state\n", - " ep_reward += reward\n", - " \n", - " if done:\n", - " state = env.reset()\n", - " all_rewards.append(ep_reward)\n", - " ep_reward = 0\n", - " \n", - " if len(replay_buffer) > batch_size:\n", - " loss = compute_td_loss(batch_size)\n", - " losses.append(loss.item())\n", - " \n", - " if frame_idx % 200 == 0:\n", - " plot(frame_idx, all_rewards, losses)\n", - " \n", - " if frame_idx % 100 == 0:\n", - " update_target(current_model, target_model)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 参考\n", - "\n", - "[强化学习(十二) Dueling DQN](https://www.cnblogs.com/pinard/p/9923859.html)" - ] - } - ], - "metadata": { - "interpreter": { - "hash": "fe38df673a99c62a9fea33a7aceda74c9b65b12ee9d076c5851d98b692a4989a" - }, - "kernelspec": { - "display_name": "Python 3.7.10 64-bit ('py37': conda)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.10" - }, - "orig_nbformat": 4 - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/projects/codes/GAE/task0_train.py b/projects/codes/GAE/task0_train.py deleted file mode 100644 index 961816c..0000000 --- a/projects/codes/GAE/task0_train.py +++ /dev/null @@ -1,167 +0,0 @@ -import math -import random - -import gym -import numpy as np - -import torch -import torch.nn as nn -import torch.optim as optim -import torch.nn.functional as F -from torch.distributions import Normal -import matplotlib.pyplot as plt -import seaborn as sns -import sys,os -curr_path = os.path.dirname(os.path.abspath(__file__)) # 当前文件所在绝对路径 -parent_path = os.path.dirname(curr_path) # 父路径 -sys.path.append(parent_path) # 添加父路径到系统路径sys.path - -use_cuda = torch.cuda.is_available() -device = torch.device("cuda" if use_cuda else "cpu") - -from common.multiprocessing_env import SubprocVecEnv - -num_envs = 16 -env_name = "Pendulum-v0" - -def make_env(): - def _thunk(): - env = gym.make(env_name) - return env - - return _thunk - -envs = [make_env() for i in range(num_envs)] -envs = SubprocVecEnv(envs) - -env = gym.make(env_name) - -def init_weights(m): - if isinstance(m, nn.Linear): - nn.init.normal_(m.weight, mean=0., std=0.1) - nn.init.constant_(m.bias, 0.1) - -class ActorCritic(nn.Module): - def __init__(self, num_inputs, num_outputs, hidden_size, std=0.0): - super(ActorCritic, self).__init__() - - self.critic = nn.Sequential( - nn.Linear(num_inputs, hidden_size), - nn.ReLU(), - nn.Linear(hidden_size, 1) - ) - - self.actor = nn.Sequential( - nn.Linear(num_inputs, hidden_size), - nn.ReLU(), - nn.Linear(hidden_size, num_outputs), - ) - self.log_std = nn.Parameter(torch.ones(1, num_outputs) * std) - - self.apply(init_weights) - - def forward(self, x): - value = self.critic(x) - mu = self.actor(x) - std = self.log_std.exp().expand_as(mu) - dist = Normal(mu, std) - return dist, value - - -def plot(frame_idx, rewards): - plt.figure(figsize=(20,5)) - plt.subplot(131) - plt.title('frame %s. reward: %s' % (frame_idx, rewards[-1])) - plt.plot(rewards) - plt.show() - -def test_env(vis=False): - state = env.reset() - if vis: env.render() - done = False - total_reward = 0 - while not done: - state = torch.FloatTensor(state).unsqueeze(0).to(device) - dist, _ = model(state) - next_state, reward, done, _ = env.step(dist.sample().cpu().numpy()[0]) - state = next_state - if vis: env.render() - total_reward += reward - return total_reward - -def compute_gae(next_value, rewards, masks, values, gamma=0.99, tau=0.95): - values = values + [next_value] - gae = 0 - returns = [] - for step in reversed(range(len(rewards))): - delta = rewards[step] + gamma * values[step + 1] * masks[step] - values[step] - gae = delta + gamma * tau * masks[step] * gae - returns.insert(0, gae + values[step]) - return returns - -num_inputs = envs.observation_space.shape[0] -num_outputs = envs.action_space.shape[0] - -#Hyper params: -hidden_size = 256 -lr = 3e-2 -num_steps = 20 - -model = ActorCritic(num_inputs, num_outputs, hidden_size).to(device) -optimizer = optim.Adam(model.parameters()) - -max_frames = 100000 -frame_idx = 0 -test_rewards = [] - -state = envs.reset() - -while frame_idx < max_frames: - - log_probs = [] - values = [] - rewards = [] - masks = [] - entropy = 0 - - for _ in range(num_steps): - state = torch.FloatTensor(state).to(device) - dist, value = model(state) - - action = dist.sample() - next_state, reward, done, _ = envs.step(action.cpu().numpy()) - - log_prob = dist.log_prob(action) - entropy += dist.entropy().mean() - - log_probs.append(log_prob) - values.append(value) - rewards.append(torch.FloatTensor(reward).unsqueeze(1).to(device)) - masks.append(torch.FloatTensor(1 - done).unsqueeze(1).to(device)) - - state = next_state - frame_idx += 1 - - if frame_idx % 1000 == 0: - test_rewards.append(np.mean([test_env() for _ in range(10)])) - print(test_rewards[-1]) - # plot(frame_idx, test_rewards) - - next_state = torch.FloatTensor(next_state).to(device) - _, next_value = model(next_state) - returns = compute_gae(next_value, rewards, masks, values) - - log_probs = torch.cat(log_probs) - returns = torch.cat(returns).detach() - values = torch.cat(values) - - advantage = returns - values - - actor_loss = -(log_probs * advantage.detach()).mean() - critic_loss = advantage.pow(2).mean() - - loss = actor_loss + 0.5 * critic_loss - 0.001 * entropy - - optimizer.zero_grad() - loss.backward() - optimizer.step() diff --git a/projects/codes/HierarchicalDQN/README.md b/projects/codes/HierarchicalDQN/README.md deleted file mode 100644 index 383cdd0..0000000 --- a/projects/codes/HierarchicalDQN/README.md +++ /dev/null @@ -1,13 +0,0 @@ -# Hierarchical DQN - -## 原理简介 - -Hierarchical DQN是一种分层强化学习方法，与DQN相比增加了一个meta controller， - -![image-20210331153115575](assets/image-20210331153115575.png) - -即学习时，meta controller每次会生成一个goal，然后controller或者说下面的actor就会达到这个goal，直到done为止。这就相当于给agent增加了一个队长，队长擅长制定局部目标，指导agent前行，这样应对一些每回合步数较长或者稀疏奖励的问题会有所帮助。 - -## 伪代码 - -![image-20210331153542314](assets/image-20210331153542314.png) \ No newline at end of file diff --git a/projects/codes/HierarchicalDQN/agent.py b/projects/codes/HierarchicalDQN/agent.py deleted file mode 100644 index 91428cc..0000000 --- a/projects/codes/HierarchicalDQN/agent.py +++ /dev/null @@ -1,154 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: John -Email: johnjim0816@gmail.com -Date: 2021-03-24 22:18:18 -LastEditor: John -LastEditTime: 2021-05-04 22:39:34 -Discription: -Environment: -''' -import torch -import torch.nn as nn -import torch.optim as optim -import torch.nn.functional as F -import numpy as np -import random,math - -class ReplayBuffer: - def __init__(self, capacity): - self.capacity = capacity # 经验回放的容量 - self.buffer = [] # 缓冲区 - self.position = 0 - - def push(self, state, action, reward, next_state, done): - ''' 缓冲区是一个队列，容量超出时去掉开始存入的转移(transition) - ''' - if len(self.buffer) < self.capacity: - self.buffer.append(None) - self.buffer[self.position] = (state, action, reward, next_state, done) - self.position = (self.position + 1) % self.capacity - - def sample(self, batch_size): - batch = random.sample(self.buffer, batch_size) # 随机采出小批量转移 - state, action, reward, next_state, done = zip(*batch) # 解压成状态，动作等 - return state, action, reward, next_state, done - - def __len__(self): - ''' 返回当前存储的量 - ''' - return len(self.buffer) -class MLP(nn.Module): - def __init__(self, input_dim,output_dim,hidden_dim=128): - """ 初始化q网络，为全连接网络 - input_dim: 输入的特征数即环境的状态维度 - output_dim: 输出的动作维度 - """ - super(MLP, self).__init__() - self.fc1 = nn.Linear(input_dim, hidden_dim) # 输入层 - self.fc2 = nn.Linear(hidden_dim,hidden_dim) # 隐藏层 - self.fc3 = nn.Linear(hidden_dim, output_dim) # 输出层 - - def forward(self, x): - # 各层对应的激活函数 - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - return self.fc3(x) - -class HierarchicalDQN: - def __init__(self,n_states,n_actions,cfg): - self.n_states = n_states - self.n_actions = n_actions - self.gamma = cfg.gamma - self.device = cfg.device - self.batch_size = cfg.batch_size - self.frame_idx = 0 # 用于epsilon的衰减计数 - self.epsilon = lambda frame_idx: cfg.epsilon_end + (cfg.epsilon_start - cfg.epsilon_end ) * math.exp(-1. * frame_idx / cfg.epsilon_decay) - self.policy_net = MLP(2*n_states, n_actions,cfg.hidden_dim).to(self.device) - self.meta_policy_net = MLP(n_states, n_states,cfg.hidden_dim).to(self.device) - self.optimizer = optim.Adam(self.policy_net.parameters(),lr=cfg.lr) - self.meta_optimizer = optim.Adam(self.meta_policy_net.parameters(),lr=cfg.lr) - self.memory = ReplayBuffer(cfg.memory_capacity) - self.meta_memory = ReplayBuffer(cfg.memory_capacity) - self.loss_numpy = 0 - self.meta_loss_numpy = 0 - self.losses = [] - self.meta_losses = [] - def to_onehot(self,x): - oh = np.zeros(self.n_states) - oh[x - 1] = 1. - return oh - def set_goal(self,state): - if random.random() > self.epsilon(self.frame_idx): - with torch.no_grad(): - state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(0) - goal = self.meta_policy_net(state).max(1)[1].item() - else: - goal = random.randrange(self.n_states) - return goal - def choose_action(self,state): - self.frame_idx += 1 - if random.random() > self.epsilon(self.frame_idx): - with torch.no_grad(): - state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(0) - q_value = self.policy_net(state) - action = q_value.max(1)[1].item() - else: - action = random.randrange(self.n_actions) - return action - def update(self): - self.update_policy() - self.update_meta() - def update_policy(self): - if self.batch_size > len(self.memory): - return - state_batch, action_batch, reward_batch, next_state_batch, done_batch = self.memory.sample(self.batch_size) - state_batch = torch.tensor(state_batch,device=self.device,dtype=torch.float) - action_batch = torch.tensor(action_batch,device=self.device,dtype=torch.int64).unsqueeze(1) - reward_batch = torch.tensor(reward_batch,device=self.device,dtype=torch.float) - next_state_batch = torch.tensor(next_state_batch,device=self.device, dtype=torch.float) - done_batch = torch.tensor(np.float32(done_batch),device=self.device) - q_values = self.policy_net(state_batch).gather(dim=1, index=action_batch).squeeze(1) - next_state_values = self.policy_net(next_state_batch).max(1)[0].detach() - expected_q_values = reward_batch + 0.99 * next_state_values * (1-done_batch) - loss = nn.MSELoss()(q_values, expected_q_values) - self.optimizer.zero_grad() - loss.backward() - for param in self.policy_net.parameters(): # clip防止梯度爆炸 - param.grad.data.clamp_(-1, 1) - self.optimizer.step() - self.loss_numpy = loss.detach().cpu().numpy() - self.losses.append(self.loss_numpy) - def update_meta(self): - if self.batch_size > len(self.meta_memory): - return - state_batch, action_batch, reward_batch, next_state_batch, done_batch = self.meta_memory.sample(self.batch_size) - state_batch = torch.tensor(state_batch,device=self.device,dtype=torch.float) - action_batch = torch.tensor(action_batch,device=self.device,dtype=torch.int64).unsqueeze(1) - reward_batch = torch.tensor(reward_batch,device=self.device,dtype=torch.float) - next_state_batch = torch.tensor(next_state_batch,device=self.device, dtype=torch.float) - done_batch = torch.tensor(np.float32(done_batch),device=self.device) - q_values = self.meta_policy_net(state_batch).gather(dim=1, index=action_batch).squeeze(1) - next_state_values = self.meta_policy_net(next_state_batch).max(1)[0].detach() - expected_q_values = reward_batch + 0.99 * next_state_values * (1-done_batch) - meta_loss = nn.MSELoss()(q_values, expected_q_values) - self.meta_optimizer.zero_grad() - meta_loss.backward() - for param in self.meta_policy_net.parameters(): # clip防止梯度爆炸 - param.grad.data.clamp_(-1, 1) - self.meta_optimizer.step() - self.meta_loss_numpy = meta_loss.detach().cpu().numpy() - self.meta_losses.append(self.meta_loss_numpy) - - def save(self, path): - torch.save(self.policy_net.state_dict(), path+'policy_checkpoint.pth') - torch.save(self.meta_policy_net.state_dict(), path+'meta_checkpoint.pth') - - def load(self, path): - self.policy_net.load_state_dict(torch.load(path+'policy_checkpoint.pth')) - self.meta_policy_net.load_state_dict(torch.load(path+'meta_checkpoint.pth')) - - - - \ No newline at end of file diff --git a/projects/codes/HierarchicalDQN/assets/image-20210331153115575.png b/projects/codes/HierarchicalDQN/assets/image-20210331153115575.png deleted file mode 100644 index 5bb9251..0000000 Binary files a/projects/codes/HierarchicalDQN/assets/image-20210331153115575.png and /dev/null differ diff --git a/projects/codes/HierarchicalDQN/assets/image-20210331153542314.png b/projects/codes/HierarchicalDQN/assets/image-20210331153542314.png deleted file mode 100644 index 6db2d82..0000000 Binary files a/projects/codes/HierarchicalDQN/assets/image-20210331153542314.png and /dev/null differ diff --git a/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/models/meta_checkpoint.pth b/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/models/meta_checkpoint.pth deleted file mode 100644 index 02f3f7c..0000000 Binary files a/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/models/meta_checkpoint.pth and /dev/null differ diff --git a/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/models/policy_checkpoint.pth b/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/models/policy_checkpoint.pth deleted file mode 100644 index 9d906ea..0000000 Binary files a/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/models/policy_checkpoint.pth and /dev/null differ diff --git a/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/test_ma_rewards.npy b/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/test_ma_rewards.npy deleted file mode 100644 index 14dd955..0000000 Binary files a/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/test_ma_rewards.npy and /dev/null differ diff --git a/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/test_rewards.npy b/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/test_rewards.npy deleted file mode 100644 index e815222..0000000 Binary files a/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/test_rewards.npy and /dev/null differ diff --git a/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/test_rewards_curve.png b/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/test_rewards_curve.png deleted file mode 100644 index 645b21a..0000000 Binary files a/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/test_rewards_curve.png and /dev/null differ diff --git a/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/train_ma_rewards.npy b/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/train_ma_rewards.npy deleted file mode 100644 index bf58391..0000000 Binary files a/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/train_ma_rewards.npy and /dev/null differ diff --git a/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/train_rewards.npy b/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/train_rewards.npy deleted file mode 100644 index f4d20ff..0000000 Binary files a/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/train_rewards.npy and /dev/null differ diff --git a/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/train_rewards_curve.png b/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/train_rewards_curve.png deleted file mode 100644 index 20ccbc5..0000000 Binary files a/projects/codes/HierarchicalDQN/outputs/CartPole-v0/20211221-200119/results/train_rewards_curve.png and /dev/null differ diff --git a/projects/codes/HierarchicalDQN/task0.py b/projects/codes/HierarchicalDQN/task0.py deleted file mode 100644 index b2cf312..0000000 --- a/projects/codes/HierarchicalDQN/task0.py +++ /dev/null @@ -1,88 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: John -Email: johnjim0816@gmail.com -Date: 2021-03-29 10:37:32 -LastEditor: John -LastEditTime: 2021-05-04 22:35:56 -Discription: -Environment: -''' -import sys -import os -curr_path = os.path.dirname(os.path.abspath(__file__)) # 当前文件所在绝对路径 -parent_path = os.path.dirname(curr_path) # 父路径 -sys.path.append(parent_path) # 添加路径到系统路径 - -import datetime -import numpy as np -import torch -import gym - -from common.utils import save_results,make_dir -from common.utils import plot_rewards -from HierarchicalDQN.agent import HierarchicalDQN -from HierarchicalDQN.train import train,test - -curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # 获取当前时间 -algo_name = "Hierarchical DQN" # 算法名称 -env_name = 'CartPole-v0' # 环境名称 -class HierarchicalDQNConfig: - def __init__(self): - self.algo_name = algo_name # 算法名称 - self.env_name = env_name # 环境名称 - self.device = torch.device( - "cuda" if torch.cuda.is_available() else "cpu") # 检测GPU - self.train_eps = 300 # 训练的episode数目 - self.test_eps = 50 # 测试的episode数目 - self.gamma = 0.99 - self.epsilon_start = 1 # start epsilon of e-greedy policy - self.epsilon_end = 0.01 - self.epsilon_decay = 200 - self.lr = 0.0001 # learning rate - self.memory_capacity = 10000 # Replay Memory capacity - self.batch_size = 32 - self.target_update = 2 # 目标网络的更新频率 - self.hidden_dim = 256 # 网络隐藏层 -class PlotConfig: - ''' 绘图相关参数设置 - ''' - - def __init__(self) -> None: - self.algo_name = algo_name # 算法名称 - self.env_name = env_name # 环境名称 - self.device = torch.device( - "cuda" if torch.cuda.is_available() else "cpu") # 检测GPU - self.result_path = curr_path + "/outputs/" + self.env_name + \ - '/' + curr_time + '/results/' # 保存结果的路径 - self.model_path = curr_path + "/outputs/" + self.env_name + \ - '/' + curr_time + '/models/' # 保存模型的路径 - self.save = True # 是否保存图片 - -def env_agent_config(cfg,seed=1): - env = gym.make(cfg.env_name) - env.seed(seed) - n_states = env.observation_space.shape[0] - n_actions = env.action_space.n - agent = HierarchicalDQN(n_states,n_actions,cfg) - return env,agent - -if __name__ == "__main__": - cfg = HierarchicalDQNConfig() - plot_cfg = PlotConfig() - # 训练 - env, agent = env_agent_config(cfg, seed=1) - rewards, ma_rewards = train(cfg, env, agent) - make_dir(plot_cfg.result_path, plot_cfg.model_path) # 创建保存结果和模型路径的文件夹 - agent.save(path=plot_cfg.model_path) # 保存模型 - save_results(rewards, ma_rewards, tag='train', - path=plot_cfg.result_path) # 保存结果 - plot_rewards(rewards, ma_rewards, plot_cfg, tag="train") # 画出结果 - # 测试 - env, agent = env_agent_config(cfg, seed=10) - agent.load(path=plot_cfg.model_path) # 导入模型 - rewards, ma_rewards = test(cfg, env, agent) - save_results(rewards, ma_rewards, tag='test', path=plot_cfg.result_path) # 保存结果 - plot_rewards(rewards, ma_rewards, plot_cfg, tag="test") # 画出结果 - diff --git a/projects/codes/HierarchicalDQN/train.py b/projects/codes/HierarchicalDQN/train.py deleted file mode 100644 index 3dc8aa3..0000000 --- a/projects/codes/HierarchicalDQN/train.py +++ /dev/null @@ -1,77 +0,0 @@ -import sys -import os -curr_path = os.path.dirname(os.path.abspath(__file__)) # 当前文件所在绝对路径 -parent_path = os.path.dirname(curr_path) # 父路径 -sys.path.append(parent_path) # 添加路径到系统路径 - -import numpy as np - -def train(cfg, env, agent): - print('开始训练!') - print(f'环境：{cfg.env_name}, 算法：{cfg.algo_name}, 设备：{cfg.device}') - rewards = [] # 记录所有回合的奖励 - ma_rewards = [] # 记录所有回合的滑动平均奖励 - for i_ep in range(cfg.train_eps): - state = env.reset() - done = False - ep_reward = 0 - while not done: - goal = agent.set_goal(state) - onehot_goal = agent.to_onehot(goal) - meta_state = state - extrinsic_reward = 0 - while not done and goal != np.argmax(state): - goal_state = np.concatenate([state, onehot_goal]) - action = agent.choose_action(goal_state) - next_state, reward, done, _ = env.step(action) - ep_reward += reward - extrinsic_reward += reward - intrinsic_reward = 1.0 if goal == np.argmax( - next_state) else 0.0 - agent.memory.push(goal_state, action, intrinsic_reward, np.concatenate( - [next_state, onehot_goal]), done) - state = next_state - agent.update() - if (i_ep+1)%10 == 0: - print(f'回合：{i_ep+1}/{cfg.train_eps}，奖励：{ep_reward}，Loss:{agent.loss_numpy:.2f}， Meta_Loss:{agent.meta_loss_numpy:.2f}') - agent.meta_memory.push(meta_state, goal, extrinsic_reward, state, done) - rewards.append(ep_reward) - if ma_rewards: - ma_rewards.append( - 0.9*ma_rewards[-1]+0.1*ep_reward) - else: - ma_rewards.append(ep_reward) - print('完成训练！') - return rewards, ma_rewards - -def test(cfg, env, agent): - print('开始测试!') - print(f'环境：{cfg.env_name}, 算法：{cfg.algo_name}, 设备：{cfg.device}') - rewards = [] # 记录所有回合的奖励 - ma_rewards = [] # 记录所有回合的滑动平均奖励 - for i_ep in range(cfg.train_eps): - state = env.reset() - done = False - ep_reward = 0 - while not done: - goal = agent.set_goal(state) - onehot_goal = agent.to_onehot(goal) - extrinsic_reward = 0 - while not done and goal != np.argmax(state): - goal_state = np.concatenate([state, onehot_goal]) - action = agent.choose_action(goal_state) - next_state, reward, done, _ = env.step(action) - ep_reward += reward - extrinsic_reward += reward - state = next_state - agent.update() - if (i_ep+1)%10 == 0: - print(f'回合：{i_ep+1}/{cfg.train_eps}，奖励：{ep_reward}，Loss:{agent.loss_numpy:.2f}， Meta_Loss:{agent.meta_loss_numpy:.2f}') - rewards.append(ep_reward) - if ma_rewards: - ma_rewards.append( - 0.9*ma_rewards[-1]+0.1*ep_reward) - else: - ma_rewards.append(ep_reward) - print('完成训练！') - return rewards, ma_rewards \ No newline at end of file diff --git a/projects/codes/MonteCarlo/README.md b/projects/codes/MonteCarlo/README.md deleted file mode 100644 index 91ff767..0000000 --- a/projects/codes/MonteCarlo/README.md +++ /dev/null @@ -1,5 +0,0 @@ -# *On-Policy First-Visit MC Control* - -### 伪代码 - -![mc_control_algo](assets/mc_control_algo.png) \ No newline at end of file diff --git a/projects/codes/MonteCarlo/Train_Racetrack-v0_FirstVisitMC_20221106-010504/config.yaml b/projects/codes/MonteCarlo/Train_Racetrack-v0_FirstVisitMC_20221106-010504/config.yaml deleted file mode 100644 index 326f84e..0000000 --- a/projects/codes/MonteCarlo/Train_Racetrack-v0_FirstVisitMC_20221106-010504/config.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: FirstVisitMC - device: cpu - env_name: Racetrack-v0 - eval_eps: 10 - eval_per_episode: 5 - load_checkpoint: false - load_path: tasks - max_steps: 200 - mode: train - save_fig: true - seed: 1 - show_fig: false - test_eps: 20 - train_eps: 200 -algo_cfg: - epsilon: 0.15 - gamma: 0.9 - lr: 0.1 diff --git a/projects/codes/MonteCarlo/Train_Racetrack-v0_FirstVisitMC_20221106-010504/logs/log.txt b/projects/codes/MonteCarlo/Train_Racetrack-v0_FirstVisitMC_20221106-010504/logs/log.txt deleted file mode 100644 index 993059b..0000000 --- a/projects/codes/MonteCarlo/Train_Racetrack-v0_FirstVisitMC_20221106-010504/logs/log.txt +++ /dev/null @@ -1,210 +0,0 @@ -2022-11-06 01:05:04 - r - INFO: - n_states: 4, n_actions: 9 -2022-11-06 01:05:04 - r - INFO: - Start training! -2022-11-06 01:05:04 - r - INFO: - Env: Racetrack-v0, Algorithm: FirstVisitMC, Device: cpu -2022-11-06 01:05:40 - r - INFO: - Episode: 1/200, Reward: -760.000, Step: 200 -2022-11-06 01:05:58 - r - INFO: - Episode: 2/200, Reward: -560.000, Step: 200 -2022-11-06 01:05:59 - r - INFO: - Episode: 3/200, Reward: -156.000, Step: 66 -2022-11-06 01:06:17 - r - INFO: - Episode: 4/200, Reward: -500.000, Step: 200 -2022-11-06 01:06:38 - r - INFO: - Episode: 5/200, Reward: -600.000, Step: 200 -2022-11-06 01:06:38 - r - INFO: - Current episode 5 has the best eval reward: -208.000 -2022-11-06 01:06:52 - r - INFO: - Episode: 6/200, Reward: -350.000, Step: 200 -2022-11-06 01:07:07 - r - INFO: - Episode: 7/200, Reward: -430.000, Step: 200 -2022-11-06 01:07:10 - r - INFO: - Episode: 8/200, Reward: -206.000, Step: 96 -2022-11-06 01:07:31 - r - INFO: - Episode: 9/200, Reward: -460.000, Step: 200 -2022-11-06 01:07:45 - r - INFO: - Episode: 10/200, Reward: -410.000, Step: 200 -2022-11-06 01:07:45 - r - INFO: - Current episode 10 has the best eval reward: -204.000 -2022-11-06 01:07:58 - r - INFO: - Episode: 11/200, Reward: -400.000, Step: 200 -2022-11-06 01:08:08 - r - INFO: - Episode: 12/200, Reward: -380.000, Step: 200 -2022-11-06 01:08:09 - r - INFO: - Episode: 13/200, Reward: -155.000, Step: 75 -2022-11-06 01:08:24 - r - INFO: - Episode: 14/200, Reward: -400.000, Step: 200 -2022-11-06 01:08:37 - r - INFO: - Episode: 15/200, Reward: -350.000, Step: 200 -2022-11-06 01:08:37 - r - INFO: - Current episode 15 has the best eval reward: -203.000 -2022-11-06 01:08:51 - r - INFO: - Episode: 16/200, Reward: -400.000, Step: 200 -2022-11-06 01:09:05 - r - INFO: - Episode: 17/200, Reward: -360.000, Step: 200 -2022-11-06 01:09:23 - r - INFO: - Episode: 18/200, Reward: -420.000, Step: 200 -2022-11-06 01:09:37 - r - INFO: - Episode: 19/200, Reward: -430.000, Step: 200 -2022-11-06 01:09:48 - r - INFO: - Episode: 20/200, Reward: -360.000, Step: 200 -2022-11-06 01:09:48 - r - INFO: - Current episode 20 has the best eval reward: -187.300 -2022-11-06 01:10:08 - r - INFO: - Episode: 21/200, Reward: -420.000, Step: 200 -2022-11-06 01:10:19 - r - INFO: - Episode: 22/200, Reward: -390.000, Step: 200 -2022-11-06 01:10:19 - r - INFO: - Episode: 23/200, Reward: -59.000, Step: 49 -2022-11-06 01:10:33 - r - INFO: - Episode: 24/200, Reward: -390.000, Step: 200 -2022-11-06 01:10:33 - r - INFO: - Episode: 25/200, Reward: 2.000, Step: 8 -2022-11-06 01:10:36 - r - INFO: - Episode: 26/200, Reward: -217.000, Step: 117 -2022-11-06 01:10:43 - r - INFO: - Episode: 27/200, Reward: -287.000, Step: 167 -2022-11-06 01:10:47 - r - INFO: - Episode: 28/200, Reward: -248.000, Step: 118 -2022-11-06 01:11:04 - r - INFO: - Episode: 29/200, Reward: -370.000, Step: 200 -2022-11-06 01:11:19 - r - INFO: - Episode: 30/200, Reward: -390.000, Step: 200 -2022-11-06 01:11:32 - r - INFO: - Episode: 31/200, Reward: -370.000, Step: 200 -2022-11-06 01:11:39 - r - INFO: - Episode: 32/200, Reward: -360.000, Step: 200 -2022-11-06 01:11:57 - r - INFO: - Episode: 33/200, Reward: -420.000, Step: 200 -2022-11-06 01:12:16 - r - INFO: - Episode: 34/200, Reward: -430.000, Step: 200 -2022-11-06 01:12:34 - r - INFO: - Episode: 35/200, Reward: -430.000, Step: 200 -2022-11-06 01:12:55 - r - INFO: - Episode: 36/200, Reward: -430.000, Step: 200 -2022-11-06 01:13:09 - r - INFO: - Episode: 37/200, Reward: -380.000, Step: 200 -2022-11-06 01:13:27 - r - INFO: - Episode: 38/200, Reward: -420.000, Step: 200 -2022-11-06 01:13:40 - r - INFO: - Episode: 39/200, Reward: -350.000, Step: 200 -2022-11-06 01:13:55 - r - INFO: - Episode: 40/200, Reward: -370.000, Step: 200 -2022-11-06 01:14:09 - r - INFO: - Episode: 41/200, Reward: -400.000, Step: 200 -2022-11-06 01:14:26 - r - INFO: - Episode: 42/200, Reward: -410.000, Step: 200 -2022-11-06 01:14:40 - r - INFO: - Episode: 43/200, Reward: -360.000, Step: 200 -2022-11-06 01:14:40 - r - INFO: - Episode: 44/200, Reward: -16.000, Step: 16 -2022-11-06 01:14:40 - r - INFO: - Episode: 45/200, Reward: -23.000, Step: 13 -2022-11-06 01:14:52 - r - INFO: - Episode: 46/200, Reward: -390.000, Step: 200 -2022-11-06 01:15:08 - r - INFO: - Episode: 47/200, Reward: -390.000, Step: 200 -2022-11-06 01:15:09 - r - INFO: - Episode: 48/200, Reward: -109.000, Step: 79 -2022-11-06 01:15:22 - r - INFO: - Episode: 49/200, Reward: -300.000, Step: 200 -2022-11-06 01:15:39 - r - INFO: - Episode: 50/200, Reward: -370.000, Step: 200 -2022-11-06 01:15:55 - r - INFO: - Episode: 51/200, Reward: -460.000, Step: 200 -2022-11-06 01:16:11 - r - INFO: - Episode: 52/200, Reward: -350.000, Step: 200 -2022-11-06 01:16:23 - r - INFO: - Episode: 53/200, Reward: -320.000, Step: 200 -2022-11-06 01:16:32 - r - INFO: - Episode: 54/200, Reward: -310.000, Step: 200 -2022-11-06 01:16:47 - r - INFO: - Episode: 55/200, Reward: -390.000, Step: 200 -2022-11-06 01:17:01 - r - INFO: - Episode: 56/200, Reward: -370.000, Step: 200 -2022-11-06 01:17:19 - r - INFO: - Episode: 57/200, Reward: -390.000, Step: 200 -2022-11-06 01:17:34 - r - INFO: - Episode: 58/200, Reward: -350.000, Step: 200 -2022-11-06 01:17:35 - r - INFO: - Episode: 59/200, Reward: -123.000, Step: 73 -2022-11-06 01:17:39 - r - INFO: - Episode: 60/200, Reward: -204.000, Step: 124 -2022-11-06 01:17:40 - r - INFO: - Episode: 61/200, Reward: -39.000, Step: 29 -2022-11-06 01:17:41 - r - INFO: - Episode: 62/200, Reward: -155.000, Step: 85 -2022-11-06 01:17:42 - r - INFO: - Episode: 63/200, Reward: -108.000, Step: 58 -2022-11-06 01:17:49 - r - INFO: - Episode: 64/200, Reward: -249.000, Step: 169 -2022-11-06 01:17:51 - r - INFO: - Episode: 65/200, Reward: -170.000, Step: 100 -2022-11-06 01:17:51 - r - INFO: - Current episode 65 has the best eval reward: -181.800 -2022-11-06 01:17:51 - r - INFO: - Episode: 66/200, Reward: 1.000, Step: 9 -2022-11-06 01:17:51 - r - INFO: - Episode: 67/200, Reward: -23.000, Step: 23 -2022-11-06 01:17:52 - r - INFO: - Episode: 68/200, Reward: -104.000, Step: 74 -2022-11-06 01:17:56 - r - INFO: - Episode: 69/200, Reward: -223.000, Step: 123 -2022-11-06 01:18:11 - r - INFO: - Episode: 70/200, Reward: -350.000, Step: 200 -2022-11-06 01:18:13 - r - INFO: - Episode: 71/200, Reward: -124.000, Step: 104 -2022-11-06 01:18:13 - r - INFO: - Episode: 72/200, Reward: -20.000, Step: 20 -2022-11-06 01:18:26 - r - INFO: - Episode: 73/200, Reward: -360.000, Step: 200 -2022-11-06 01:18:26 - r - INFO: - Episode: 74/200, Reward: -67.000, Step: 37 -2022-11-06 01:18:40 - r - INFO: - Episode: 75/200, Reward: -360.000, Step: 200 -2022-11-06 01:18:41 - r - INFO: - Episode: 76/200, Reward: -71.000, Step: 41 -2022-11-06 01:18:41 - r - INFO: - Episode: 77/200, Reward: -23.000, Step: 23 -2022-11-06 01:18:41 - r - INFO: - Episode: 78/200, Reward: -41.000, Step: 21 -2022-11-06 01:18:41 - r - INFO: - Episode: 79/200, Reward: -1.000, Step: 11 -2022-11-06 01:18:50 - r - INFO: - Episode: 80/200, Reward: -270.000, Step: 200 -2022-11-06 01:18:50 - r - INFO: - Current episode 80 has the best eval reward: -163.100 -2022-11-06 01:19:02 - r - INFO: - Episode: 81/200, Reward: -330.000, Step: 200 -2022-11-06 01:19:10 - r - INFO: - Episode: 82/200, Reward: -290.000, Step: 200 -2022-11-06 01:19:11 - r - INFO: - Episode: 83/200, Reward: -2.000, Step: 12 -2022-11-06 01:19:25 - r - INFO: - Episode: 84/200, Reward: -300.000, Step: 200 -2022-11-06 01:19:37 - r - INFO: - Episode: 85/200, Reward: -380.000, Step: 200 -2022-11-06 01:19:37 - r - INFO: - Episode: 86/200, Reward: -47.000, Step: 37 -2022-11-06 01:19:53 - r - INFO: - Episode: 87/200, Reward: -350.000, Step: 200 -2022-11-06 01:20:04 - r - INFO: - Episode: 88/200, Reward: -308.000, Step: 188 -2022-11-06 01:20:21 - r - INFO: - Episode: 89/200, Reward: -370.000, Step: 200 -2022-11-06 01:20:27 - r - INFO: - Episode: 90/200, Reward: -214.000, Step: 154 -2022-11-06 01:20:43 - r - INFO: - Episode: 91/200, Reward: -290.000, Step: 200 -2022-11-06 01:21:00 - r - INFO: - Episode: 92/200, Reward: -370.000, Step: 200 -2022-11-06 01:21:01 - r - INFO: - Episode: 93/200, Reward: -32.000, Step: 22 -2022-11-06 01:21:21 - r - INFO: - Episode: 94/200, Reward: -400.000, Step: 200 -2022-11-06 01:21:25 - r - INFO: - Episode: 95/200, Reward: -217.000, Step: 127 -2022-11-06 01:21:41 - r - INFO: - Episode: 96/200, Reward: -330.000, Step: 200 -2022-11-06 01:21:55 - r - INFO: - Episode: 97/200, Reward: -380.000, Step: 200 -2022-11-06 01:22:16 - r - INFO: - Episode: 98/200, Reward: -320.000, Step: 200 -2022-11-06 01:22:32 - r - INFO: - Episode: 99/200, Reward: -300.000, Step: 200 -2022-11-06 01:22:46 - r - INFO: - Episode: 100/200, Reward: -350.000, Step: 200 -2022-11-06 01:23:00 - r - INFO: - Episode: 101/200, Reward: -400.000, Step: 200 -2022-11-06 01:23:11 - r - INFO: - Episode: 102/200, Reward: -330.000, Step: 200 -2022-11-06 01:23:29 - r - INFO: - Episode: 103/200, Reward: -360.000, Step: 200 -2022-11-06 01:23:45 - r - INFO: - Episode: 104/200, Reward: -380.000, Step: 200 -2022-11-06 01:24:06 - r - INFO: - Episode: 105/200, Reward: -400.000, Step: 200 -2022-11-06 01:24:16 - r - INFO: - Episode: 106/200, Reward: -290.000, Step: 200 -2022-11-06 01:24:19 - r - INFO: - Episode: 107/200, Reward: -203.000, Step: 103 -2022-11-06 01:24:19 - r - INFO: - Episode: 108/200, Reward: -74.000, Step: 54 -2022-11-06 01:24:36 - r - INFO: - Episode: 109/200, Reward: -330.000, Step: 200 -2022-11-06 01:24:54 - r - INFO: - Episode: 110/200, Reward: -380.000, Step: 200 -2022-11-06 01:25:03 - r - INFO: - Episode: 111/200, Reward: -263.000, Step: 173 -2022-11-06 01:25:20 - r - INFO: - Episode: 112/200, Reward: -290.000, Step: 200 -2022-11-06 01:25:34 - r - INFO: - Episode: 113/200, Reward: -340.000, Step: 200 -2022-11-06 01:25:34 - r - INFO: - Episode: 114/200, Reward: -86.000, Step: 66 -2022-11-06 01:25:50 - r - INFO: - Episode: 115/200, Reward: -340.000, Step: 200 -2022-11-06 01:25:52 - r - INFO: - Episode: 116/200, Reward: -160.000, Step: 110 -2022-11-06 01:26:07 - r - INFO: - Episode: 117/200, Reward: -340.000, Step: 200 -2022-11-06 01:26:15 - r - INFO: - Episode: 118/200, Reward: -320.000, Step: 200 -2022-11-06 01:26:29 - r - INFO: - Episode: 119/200, Reward: -320.000, Step: 200 -2022-11-06 01:26:43 - r - INFO: - Episode: 120/200, Reward: -360.000, Step: 200 -2022-11-06 01:26:56 - r - INFO: - Episode: 121/200, Reward: -330.000, Step: 200 -2022-11-06 01:27:09 - r - INFO: - Episode: 122/200, Reward: -350.000, Step: 200 -2022-11-06 01:27:25 - r - INFO: - Episode: 123/200, Reward: -300.000, Step: 200 -2022-11-06 01:27:38 - r - INFO: - Episode: 124/200, Reward: -320.000, Step: 200 -2022-11-06 01:27:39 - r - INFO: - Episode: 125/200, Reward: -70.000, Step: 40 -2022-11-06 01:27:39 - r - INFO: - Episode: 126/200, Reward: -59.000, Step: 39 -2022-11-06 01:27:55 - r - INFO: - Episode: 127/200, Reward: -340.000, Step: 200 -2022-11-06 01:27:56 - r - INFO: - Episode: 128/200, Reward: -87.000, Step: 77 -2022-11-06 01:28:13 - r - INFO: - Episode: 129/200, Reward: -330.000, Step: 200 -2022-11-06 01:28:22 - r - INFO: - Episode: 130/200, Reward: -260.000, Step: 200 -2022-11-06 01:28:38 - r - INFO: - Episode: 131/200, Reward: -290.000, Step: 200 -2022-11-06 01:28:57 - r - INFO: - Episode: 132/200, Reward: -330.000, Step: 200 -2022-11-06 01:29:07 - r - INFO: - Episode: 133/200, Reward: -340.000, Step: 200 -2022-11-06 01:29:08 - r - INFO: - Episode: 134/200, Reward: -78.000, Step: 48 -2022-11-06 01:29:23 - r - INFO: - Episode: 135/200, Reward: -390.000, Step: 200 -2022-11-06 01:29:33 - r - INFO: - Episode: 136/200, Reward: -320.000, Step: 200 -2022-11-06 01:29:51 - r - INFO: - Episode: 137/200, Reward: -360.000, Step: 200 -2022-11-06 01:30:06 - r - INFO: - Episode: 138/200, Reward: -340.000, Step: 200 -2022-11-06 01:30:10 - r - INFO: - Episode: 139/200, Reward: -185.000, Step: 115 -2022-11-06 01:30:26 - r - INFO: - Episode: 140/200, Reward: -340.000, Step: 200 -2022-11-06 01:30:43 - r - INFO: - Episode: 141/200, Reward: -250.000, Step: 200 -2022-11-06 01:30:57 - r - INFO: - Episode: 142/200, Reward: -347.000, Step: 197 -2022-11-06 01:31:11 - r - INFO: - Episode: 143/200, Reward: -320.000, Step: 200 -2022-11-06 01:31:25 - r - INFO: - Episode: 144/200, Reward: -330.000, Step: 200 -2022-11-06 01:31:37 - r - INFO: - Episode: 145/200, Reward: -270.000, Step: 200 -2022-11-06 01:31:55 - r - INFO: - Episode: 146/200, Reward: -380.000, Step: 200 -2022-11-06 01:32:10 - r - INFO: - Episode: 147/200, Reward: -320.000, Step: 200 -2022-11-06 01:32:27 - r - INFO: - Episode: 148/200, Reward: -340.000, Step: 200 -2022-11-06 01:32:38 - r - INFO: - Episode: 149/200, Reward: -310.000, Step: 200 -2022-11-06 01:32:57 - r - INFO: - Episode: 150/200, Reward: -290.000, Step: 200 -2022-11-06 01:33:10 - r - INFO: - Episode: 151/200, Reward: -380.000, Step: 200 -2022-11-06 01:33:21 - r - INFO: - Episode: 152/200, Reward: -281.000, Step: 181 -2022-11-06 01:33:21 - r - INFO: - Episode: 153/200, Reward: -30.000, Step: 30 -2022-11-06 01:33:33 - r - INFO: - Episode: 154/200, Reward: -280.000, Step: 200 -2022-11-06 01:33:45 - r - INFO: - Episode: 155/200, Reward: -300.000, Step: 200 -2022-11-06 01:33:59 - r - INFO: - Episode: 156/200, Reward: -300.000, Step: 200 -2022-11-06 01:34:10 - r - INFO: - Episode: 157/200, Reward: -300.000, Step: 200 -2022-11-06 01:34:28 - r - INFO: - Episode: 158/200, Reward: -370.000, Step: 200 -2022-11-06 01:34:45 - r - INFO: - Episode: 159/200, Reward: -320.000, Step: 200 -2022-11-06 01:34:52 - r - INFO: - Episode: 160/200, Reward: -250.000, Step: 200 -2022-11-06 01:35:04 - r - INFO: - Episode: 161/200, Reward: -370.000, Step: 200 -2022-11-06 01:35:16 - r - INFO: - Episode: 162/200, Reward: -290.000, Step: 200 -2022-11-06 01:35:31 - r - INFO: - Episode: 163/200, Reward: -320.000, Step: 200 -2022-11-06 01:35:41 - r - INFO: - Episode: 164/200, Reward: -290.000, Step: 200 -2022-11-06 01:35:41 - r - INFO: - Episode: 165/200, Reward: -44.000, Step: 44 -2022-11-06 01:35:53 - r - INFO: - Episode: 166/200, Reward: -216.000, Step: 196 -2022-11-06 01:36:06 - r - INFO: - Episode: 167/200, Reward: -340.000, Step: 200 -2022-11-06 01:36:23 - r - INFO: - Episode: 168/200, Reward: -360.000, Step: 200 -2022-11-06 01:36:38 - r - INFO: - Episode: 169/200, Reward: -310.000, Step: 200 -2022-11-06 01:36:51 - r - INFO: - Episode: 170/200, Reward: -320.000, Step: 200 -2022-11-06 01:37:08 - r - INFO: - Episode: 171/200, Reward: -280.000, Step: 200 -2022-11-06 01:37:17 - r - INFO: - Episode: 172/200, Reward: -290.000, Step: 200 -2022-11-06 01:37:33 - r - INFO: - Episode: 173/200, Reward: -280.000, Step: 200 -2022-11-06 01:37:45 - r - INFO: - Episode: 174/200, Reward: -300.000, Step: 200 -2022-11-06 01:38:02 - r - INFO: - Episode: 175/200, Reward: -350.000, Step: 200 -2022-11-06 01:38:17 - r - INFO: - Episode: 176/200, Reward: -320.000, Step: 200 -2022-11-06 01:38:31 - r - INFO: - Episode: 177/200, Reward: -320.000, Step: 200 -2022-11-06 01:38:47 - r - INFO: - Episode: 178/200, Reward: -320.000, Step: 200 -2022-11-06 01:39:03 - r - INFO: - Episode: 179/200, Reward: -300.000, Step: 200 -2022-11-06 01:39:04 - r - INFO: - Episode: 180/200, Reward: -117.000, Step: 87 -2022-11-06 01:39:06 - r - INFO: - Episode: 181/200, Reward: -158.000, Step: 88 -2022-11-06 01:39:23 - r - INFO: - Episode: 182/200, Reward: -300.000, Step: 200 -2022-11-06 01:39:34 - r - INFO: - Episode: 183/200, Reward: -290.000, Step: 200 -2022-11-06 01:39:51 - r - INFO: - Episode: 184/200, Reward: -350.000, Step: 200 -2022-11-06 01:40:09 - r - INFO: - Episode: 185/200, Reward: -310.000, Step: 200 -2022-11-06 01:40:10 - r - INFO: - Episode: 186/200, Reward: -58.000, Step: 38 -2022-11-06 01:40:26 - r - INFO: - Episode: 187/200, Reward: -290.000, Step: 200 -2022-11-06 01:40:42 - r - INFO: - Episode: 188/200, Reward: -310.000, Step: 200 -2022-11-06 01:40:57 - r - INFO: - Episode: 189/200, Reward: -350.000, Step: 200 -2022-11-06 01:41:12 - r - INFO: - Episode: 190/200, Reward: -300.000, Step: 200 -2022-11-06 01:41:32 - r - INFO: - Episode: 191/200, Reward: -380.000, Step: 200 -2022-11-06 01:41:37 - r - INFO: - Episode: 192/200, Reward: -230.000, Step: 200 -2022-11-06 01:41:37 - r - INFO: - Episode: 193/200, Reward: -26.000, Step: 26 -2022-11-06 01:41:56 - r - INFO: - Episode: 194/200, Reward: -340.000, Step: 200 -2022-11-06 01:42:09 - r - INFO: - Episode: 195/200, Reward: -280.000, Step: 200 -2022-11-06 01:42:10 - r - INFO: - Episode: 196/200, Reward: -106.000, Step: 66 -2022-11-06 01:42:10 - r - INFO: - Episode: 197/200, Reward: -7.000, Step: 17 -2022-11-06 01:42:20 - r - INFO: - Episode: 198/200, Reward: -248.000, Step: 178 -2022-11-06 01:42:22 - r - INFO: - Episode: 199/200, Reward: -161.000, Step: 101 -2022-11-06 01:42:22 - r - INFO: - Episode: 200/200, Reward: -3.000, Step: 13 -2022-11-06 01:42:22 - r - INFO: - Finish training! diff --git a/projects/codes/MonteCarlo/Train_Racetrack-v0_FirstVisitMC_20221106-010504/models/Q_table b/projects/codes/MonteCarlo/Train_Racetrack-v0_FirstVisitMC_20221106-010504/models/Q_table deleted file mode 100644 index 3231a0b..0000000 Binary files a/projects/codes/MonteCarlo/Train_Racetrack-v0_FirstVisitMC_20221106-010504/models/Q_table and /dev/null differ diff --git a/projects/codes/MonteCarlo/Train_Racetrack-v0_FirstVisitMC_20221106-010504/results/learning_curve.png b/projects/codes/MonteCarlo/Train_Racetrack-v0_FirstVisitMC_20221106-010504/results/learning_curve.png deleted file mode 100644 index 3799635..0000000 Binary files a/projects/codes/MonteCarlo/Train_Racetrack-v0_FirstVisitMC_20221106-010504/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/MonteCarlo/Train_Racetrack-v0_FirstVisitMC_20221106-010504/results/res.csv b/projects/codes/MonteCarlo/Train_Racetrack-v0_FirstVisitMC_20221106-010504/results/res.csv deleted file mode 100644 index 214239b..0000000 --- a/projects/codes/MonteCarlo/Train_Racetrack-v0_FirstVisitMC_20221106-010504/results/res.csv +++ /dev/null @@ -1,201 +0,0 @@ -episodes,rewards,steps -0,-760,200 -1,-560,200 -2,-156,66 -3,-500,200 -4,-600,200 -5,-350,200 -6,-430,200 -7,-206,96 -8,-460,200 -9,-410,200 -10,-400,200 -11,-380,200 -12,-155,75 -13,-400,200 -14,-350,200 -15,-400,200 -16,-360,200 -17,-420,200 -18,-430,200 -19,-360,200 -20,-420,200 -21,-390,200 -22,-59,49 -23,-390,200 -24,2,8 -25,-217,117 -26,-287,167 -27,-248,118 -28,-370,200 -29,-390,200 -30,-370,200 -31,-360,200 -32,-420,200 -33,-430,200 -34,-430,200 -35,-430,200 -36,-380,200 -37,-420,200 -38,-350,200 -39,-370,200 -40,-400,200 -41,-410,200 -42,-360,200 -43,-16,16 -44,-23,13 -45,-390,200 -46,-390,200 -47,-109,79 -48,-300,200 -49,-370,200 -50,-460,200 -51,-350,200 -52,-320,200 -53,-310,200 -54,-390,200 -55,-370,200 -56,-390,200 -57,-350,200 -58,-123,73 -59,-204,124 -60,-39,29 -61,-155,85 -62,-108,58 -63,-249,169 -64,-170,100 -65,1,9 -66,-23,23 -67,-104,74 -68,-223,123 -69,-350,200 -70,-124,104 -71,-20,20 -72,-360,200 -73,-67,37 -74,-360,200 -75,-71,41 -76,-23,23 -77,-41,21 -78,-1,11 -79,-270,200 -80,-330,200 -81,-290,200 -82,-2,12 -83,-300,200 -84,-380,200 -85,-47,37 -86,-350,200 -87,-308,188 -88,-370,200 -89,-214,154 -90,-290,200 -91,-370,200 -92,-32,22 -93,-400,200 -94,-217,127 -95,-330,200 -96,-380,200 -97,-320,200 -98,-300,200 -99,-350,200 -100,-400,200 -101,-330,200 -102,-360,200 -103,-380,200 -104,-400,200 -105,-290,200 -106,-203,103 -107,-74,54 -108,-330,200 -109,-380,200 -110,-263,173 -111,-290,200 -112,-340,200 -113,-86,66 -114,-340,200 -115,-160,110 -116,-340,200 -117,-320,200 -118,-320,200 -119,-360,200 -120,-330,200 -121,-350,200 -122,-300,200 -123,-320,200 -124,-70,40 -125,-59,39 -126,-340,200 -127,-87,77 -128,-330,200 -129,-260,200 -130,-290,200 -131,-330,200 -132,-340,200 -133,-78,48 -134,-390,200 -135,-320,200 -136,-360,200 -137,-340,200 -138,-185,115 -139,-340,200 -140,-250,200 -141,-347,197 -142,-320,200 -143,-330,200 -144,-270,200 -145,-380,200 -146,-320,200 -147,-340,200 -148,-310,200 -149,-290,200 -150,-380,200 -151,-281,181 -152,-30,30 -153,-280,200 -154,-300,200 -155,-300,200 -156,-300,200 -157,-370,200 -158,-320,200 -159,-250,200 -160,-370,200 -161,-290,200 -162,-320,200 -163,-290,200 -164,-44,44 -165,-216,196 -166,-340,200 -167,-360,200 -168,-310,200 -169,-320,200 -170,-280,200 -171,-290,200 -172,-280,200 -173,-300,200 -174,-350,200 -175,-320,200 -176,-320,200 -177,-320,200 -178,-300,200 -179,-117,87 -180,-158,88 -181,-300,200 -182,-290,200 -183,-350,200 -184,-310,200 -185,-58,38 -186,-290,200 -187,-310,200 -188,-350,200 -189,-300,200 -190,-380,200 -191,-230,200 -192,-26,26 -193,-340,200 -194,-280,200 -195,-106,66 -196,-7,17 -197,-248,178 -198,-161,101 -199,-3,13 diff --git a/projects/codes/MonteCarlo/agent.py b/projects/codes/MonteCarlo/agent.py deleted file mode 100644 index c426527..0000000 --- a/projects/codes/MonteCarlo/agent.py +++ /dev/null @@ -1,78 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: John -Email: johnjim0816@gmail.com -Date: 2021-03-12 16:14:34 -LastEditor: John -LastEditTime: 2022-11-06 01:04:57 -Discription: -Environment: -''' -import numpy as np -from collections import defaultdict -import torch -import dill - -class FisrtVisitMC: - ''' On-Policy First-Visit MC Control - ''' - def __init__(self,cfg): - self.n_actions = cfg.n_actions - self.epsilon = cfg.epsilon - self.gamma = cfg.gamma - self.Q_table = defaultdict(lambda: np.zeros(cfg.n_actions)) - self.returns_sum = defaultdict(float) # 保存return之和 - self.returns_count = defaultdict(float) - - def sample_action(self,state): - state = str(state) - if state in self.Q_table.keys(): - best_action = np.argmax(self.Q_table[state]) - action_probs = np.ones(self.n_actions, dtype=float) * self.epsilon / self.n_actions - action_probs[best_action] += (1.0 - self.epsilon) - action = np.random.choice(np.arange(len(action_probs)), p=action_probs) - else: - action = np.random.randint(0,self.n_actions) - return action - def predict_action(self,state): - state = str(state) - if state in self.Q_table.keys(): - best_action = np.argmax(self.Q_table[state]) - action_probs = np.ones(self.n_actions, dtype=float) * self.epsilon / self.n_actions - action_probs[best_action] += (1.0 - self.epsilon) - action = np.argmax(self.Q_table[state]) - else: - action = np.random.randint(0,self.n_actions) - return action - def update(self,one_ep_transition): - # Find all (state, action) pairs we've visited in this one_ep_transition - # We convert each state to a tuple so that we can use it as a dict key - sa_in_episode = set([(str(x[0]), x[1]) for x in one_ep_transition]) - for state, action in sa_in_episode: - sa_pair = (state, action) - # Find the first occurence of the (state, action) pair in the one_ep_transition - - first_occurence_idx = next(i for i,x in enumerate(one_ep_transition) - if str(x[0]) == state and x[1] == action) - # Sum up all rewards since the first occurance - G = sum([x[2]*(self.gamma**i) for i,x in enumerate(one_ep_transition[first_occurence_idx:])]) - # Calculate average return for this state over all sampled episodes - self.returns_sum[sa_pair] += G - self.returns_count[sa_pair] += 1.0 - self.Q_table[state][action] = self.returns_sum[sa_pair] / self.returns_count[sa_pair] - def save_model(self,path=None): - '''把 Q表格 的数据保存到文件中 - ''' - from pathlib import Path - Path(path).mkdir(parents=True, exist_ok=True) - torch.save( - obj=self.Q_table, - f=path+"Q_table", - pickle_module=dill - ) - - def load_model(self, path=None): - '''从文件中读取数据到 Q表格 - ''' - self.Q_table =torch.load(f=path+"Q_table",pickle_module=dill) \ No newline at end of file diff --git a/projects/codes/MonteCarlo/assets/mc_control_algo.png b/projects/codes/MonteCarlo/assets/mc_control_algo.png deleted file mode 100644 index 0b436fa..0000000 Binary files a/projects/codes/MonteCarlo/assets/mc_control_algo.png and /dev/null differ diff --git a/projects/codes/MonteCarlo/config/config.py b/projects/codes/MonteCarlo/config/config.py deleted file mode 100644 index d255547..0000000 --- a/projects/codes/MonteCarlo/config/config.py +++ /dev/null @@ -1,32 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-11-06 00:31:35 -LastEditor: JiangJi -LastEditTime: 2022-11-06 00:45:44 -Discription: parameters of MonteCarlo -''' -from common.config import GeneralConfig,AlgoConfig - -class GeneralConfigMC(GeneralConfig): - def __init__(self) -> None: - self.env_name = "Racetrack-v0" # name of environment - self.algo_name = "FirstVisitMC" # name of algorithm - self.mode = "train" # train or test - self.seed = 1 # random seed - self.device = "cpu" # device to use - self.train_eps = 200 # number of episodes for training - self.test_eps = 20 # number of episodes for testing - self.max_steps = 200 # max steps for each episode - self.load_checkpoint = False - self.load_path = "tasks" # path to load model - self.show_fig = False # show figure or not - self.save_fig = True # save figure or not - -class AlgoConfigMC(AlgoConfig): - def __init__(self) -> None: - self.gamma = 0.90 # discount factor - self.epsilon = 0.15 # epsilon greedy - self.lr = 0.1 # learning rate \ No newline at end of file diff --git a/projects/codes/MonteCarlo/task0.py b/projects/codes/MonteCarlo/task0.py deleted file mode 100644 index 75e52e1..0000000 --- a/projects/codes/MonteCarlo/task0.py +++ /dev/null @@ -1,125 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: John -Email: johnjim0816@gmail.com -Date: 2021-03-11 14:26:44 -LastEditor: John -LastEditTime: 2022-11-08 23:35:18 -Discription: -Environment: -''' -import sys,os -os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE" # avoid "OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized." -curr_path = os.path.dirname(os.path.abspath(__file__)) # current path -parent_path = os.path.dirname(curr_path) # parent path -sys.path.append(parent_path) # add path to system path - -import datetime -import gym -from envs.wrappers import CliffWalkingWapper -from envs.register import register_env -from common.utils import merge_class_attrs,all_seed -from common.launcher import Launcher -from MonteCarlo.agent import FisrtVisitMC -from MonteCarlo.config.config import GeneralConfigMC,AlgoConfigMC - -class Main(Launcher): - def __init__(self) -> None: - super().__init__() - self.cfgs['general_cfg'] = merge_class_attrs(self.cfgs['general_cfg'],GeneralConfigMC()) - self.cfgs['algo_cfg'] = merge_class_attrs(self.cfgs['algo_cfg'],AlgoConfigMC()) - def env_agent_config(self,cfg,logger): - ''' create env and agent - ''' - register_env(cfg.env_name) - env = gym.make(cfg.env_name,new_step_api=False) # create env - if cfg.env_name == 'CliffWalking-v0': - env = CliffWalkingWapper(env) - if cfg.seed !=0: # set random seed - all_seed(env,seed=cfg.seed) - try: # state dimension - n_states = env.observation_space.n # print(hasattr(env.observation_space, 'n')) - except AttributeError: - n_states = env.observation_space.shape[0] # print(hasattr(env.observation_space, 'shape')) - n_actions = env.action_space.n # action dimension - logger.info(f"n_states: {n_states}, n_actions: {n_actions}") # print info - # update to cfg paramters - setattr(cfg, 'n_states', n_states) - setattr(cfg, 'n_actions', n_actions) - agent = FisrtVisitMC(cfg) - return env,agent - def train_one_episode(self, env, agent, cfg): - ep_reward = 0 # reward per episode - ep_step = 0 - state = env.reset() # reset and obtain initial state - one_ep_transition = [] - for _ in range(cfg.max_steps): - ep_step += 1 - action = agent.sample_action(state) # sample action - next_state, reward, terminated, info = env.step(action) # update env and return transitions under new_step_api of OpenAI Gym - one_ep_transition.append((state, action, reward)) # save transitions - agent.update(one_ep_transition) # update agent - state = next_state # update next state for env - ep_reward += reward # - if terminated: - break - return agent,ep_reward,ep_step - def test_one_episode(self, env, agent, cfg): - ep_reward = 0 # reward per episode - ep_step = 0 - state = env.reset() # reset and obtain initial state - for _ in range(cfg.max_steps): - ep_step += 1 - action = agent.predict_action(state) # sample action - next_state, reward, terminated, info = env.step(action) # update env and return transitions under new_step_api of OpenAI Gym - state = next_state # update next state for env - ep_reward += reward # - if terminated: - break - return agent,ep_reward,ep_step - -def train(cfg, env, agent): - print("开始训练！") - print(f"环境：{cfg.env_name}，算法：{cfg.algo_name}，设备：{cfg.device}") - rewards = [] - for i_ep in range(cfg.train_eps): - state = env.reset() - ep_reward = 0 - one_ep_transition = [] - while True: - action = agent.sample(state) - next_state, reward, done = env.step(action) - ep_reward += reward - one_ep_transition.append((state, action, reward)) - state = next_state - if done: - break - rewards.append(ep_reward) - agent.update(one_ep_transition) - if (i_ep+1) % 10 == 0: - print(f"Episode:{i_ep+1}/{cfg.train_eps}: Reward:{ep_reward}") - print("完成训练") - return {'rewards':rewards} - -def test(cfg, env, agent): - print("开始测试！") - print(f"环境：{cfg.env_name}, 算法：{cfg.algo_name}, 设备：{cfg.device}") - rewards = [] - for i_ep in range(cfg.test_eps): - state = env.reset() - ep_reward = 0 - while True: - action = agent.predict(state) - next_state, reward, done = env.step(action) - ep_reward += reward - state = next_state - if done: - break - rewards.append(ep_reward) - print(f'回合：{i_ep+1}/{cfg.test_eps}，奖励：{ep_reward:.2f}') - return {'rewards':rewards} - -if __name__ == "__main__": - main = Main() - main.run() \ No newline at end of file diff --git a/projects/codes/NoisyDQN/noisy_dqn.py b/projects/codes/NoisyDQN/noisy_dqn.py deleted file mode 100644 index 45cc5d2..0000000 --- a/projects/codes/NoisyDQN/noisy_dqn.py +++ /dev/null @@ -1,52 +0,0 @@ -import torch -import torch.nn as nn - -class NoisyLinear(nn.Module): - def __init__(self, input_dim, output_dim, std_init=0.4): - super(NoisyLinear, self).__init__() - - self.input_dim = input_dim - self.output_dim = output_dim - self.std_init = std_init - - self.weight_mu = nn.Parameter(torch.FloatTensor(output_dim, input_dim)) - self.weight_sigma = nn.Parameter(torch.FloatTensor(output_dim, input_dim)) - self.register_buffer('weight_epsilon', torch.FloatTensor(output_dim, input_dim)) - - self.bias_mu = nn.Parameter(torch.FloatTensor(output_dim)) - self.bias_sigma = nn.Parameter(torch.FloatTensor(output_dim)) - self.register_buffer('bias_epsilon', torch.FloatTensor(output_dim)) - - self.reset_parameters() - self.reset_noise() - - def forward(self, x): - if self.training: - weight = self.weight_mu + self.weight_sigma.mul( (self.weight_epsilon)) - bias = self.bias_mu + self.bias_sigma.mul(Variable(self.bias_epsilon)) - else: - weight = self.weight_mu - bias = self.bias_mu - - return F.linear(x, weight, bias) - - def reset_parameters(self): - mu_range = 1 / math.sqrt(self.weight_mu.size(1)) - - self.weight_mu.data.uniform_(-mu_range, mu_range) - self.weight_sigma.data.fill_(self.std_init / math.sqrt(self.weight_sigma.size(1))) - - self.bias_mu.data.uniform_(-mu_range, mu_range) - self.bias_sigma.data.fill_(self.std_init / math.sqrt(self.bias_sigma.size(0))) - - def reset_noise(self): - epsilon_in = self._scale_noise(self.input_dim) - epsilon_out = self._scale_noise(self.output_dim) - - self.weight_epsilon.copy_(epsilon_out.ger(epsilon_in)) - self.bias_epsilon.copy_(self._scale_noise(self.output_dim)) - - def _scale_noise(self, size): - x = torch.randn(size) - x = x.sign().mul(x.abs().sqrt()) - return x \ No newline at end of file diff --git a/projects/codes/NoisyDQN/task0_train.ipynb b/projects/codes/NoisyDQN/task0_train.ipynb deleted file mode 100644 index ecd0092..0000000 --- a/projects/codes/NoisyDQN/task0_train.ipynb +++ /dev/null @@ -1,25 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import sys\n", - "from pathlib import Path\n", - "curr_path = str(Path().absolute()) # 当前路径\n", - "parent_path = str(Path().absolute().parent) # 父路径\n", - "sys.path.append(parent_path) # 添加路径到系统路径" - ] - } - ], - "metadata": { - "language_info": { - "name": "python" - }, - "orig_nbformat": 4 - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/projects/codes/PER_DQN/Test_CartPole-v1_PER_DQN_20221114-104649/config.yaml b/projects/codes/PER_DQN/Test_CartPole-v1_PER_DQN_20221114-104649/config.yaml deleted file mode 100644 index 39f8743..0000000 --- a/projects/codes/PER_DQN/Test_CartPole-v1_PER_DQN_20221114-104649/config.yaml +++ /dev/null @@ -1,25 +0,0 @@ -general_cfg: - algo_name: PER_DQN - device: cpu - env_name: CartPole-v1 - eval_eps: 10 - eval_per_episode: 5 - load_checkpoint: true - load_path: Train_CartPole-v1_PER_DQN_20221113-162804 - max_steps: 200 - mode: test - save_fig: true - seed: 0 - show_fig: false - test_eps: 10 - train_eps: 200 -algo_cfg: - batch_size: 64 - buffer_size: 100000 - epsilon_decay: 500 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - hidden_dim: 256 - lr: 0.0001 - target_update: 4 diff --git a/projects/codes/PER_DQN/Test_CartPole-v1_PER_DQN_20221114-104649/logs/log.txt b/projects/codes/PER_DQN/Test_CartPole-v1_PER_DQN_20221114-104649/logs/log.txt deleted file mode 100644 index 9fe5454..0000000 --- a/projects/codes/PER_DQN/Test_CartPole-v1_PER_DQN_20221114-104649/logs/log.txt +++ /dev/null @@ -1,14 +0,0 @@ -2022-11-14 10:46:49 - r - INFO: - n_states: 4, n_actions: 2 -2022-11-14 10:46:49 - r - INFO: - Start testing! -2022-11-14 10:46:49 - r - INFO: - Env: CartPole-v1, Algorithm: PER_DQN, Device: cpu -2022-11-14 10:46:49 - r - INFO: - Episode: 1/10, Reward: 200.000, Step: 200 -2022-11-14 10:46:49 - r - INFO: - Episode: 2/10, Reward: 200.000, Step: 200 -2022-11-14 10:46:49 - r - INFO: - Episode: 3/10, Reward: 200.000, Step: 200 -2022-11-14 10:46:49 - r - INFO: - Episode: 4/10, Reward: 200.000, Step: 200 -2022-11-14 10:46:49 - r - INFO: - Episode: 5/10, Reward: 200.000, Step: 200 -2022-11-14 10:46:49 - r - INFO: - Episode: 6/10, Reward: 200.000, Step: 200 -2022-11-14 10:46:49 - r - INFO: - Episode: 7/10, Reward: 200.000, Step: 200 -2022-11-14 10:46:49 - r - INFO: - Episode: 8/10, Reward: 200.000, Step: 200 -2022-11-14 10:46:49 - r - INFO: - Episode: 9/10, Reward: 200.000, Step: 200 -2022-11-14 10:46:49 - r - INFO: - Episode: 10/10, Reward: 200.000, Step: 200 -2022-11-14 10:46:49 - r - INFO: - Finish testing! diff --git a/projects/codes/PER_DQN/Test_CartPole-v1_PER_DQN_20221114-104649/models/checkpoint.pt b/projects/codes/PER_DQN/Test_CartPole-v1_PER_DQN_20221114-104649/models/checkpoint.pt deleted file mode 100644 index 06d607b..0000000 Binary files a/projects/codes/PER_DQN/Test_CartPole-v1_PER_DQN_20221114-104649/models/checkpoint.pt and /dev/null differ diff --git a/projects/codes/PER_DQN/Test_CartPole-v1_PER_DQN_20221114-104649/results/learning_curve.png b/projects/codes/PER_DQN/Test_CartPole-v1_PER_DQN_20221114-104649/results/learning_curve.png deleted file mode 100644 index f1e8056..0000000 Binary files a/projects/codes/PER_DQN/Test_CartPole-v1_PER_DQN_20221114-104649/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/PER_DQN/Test_CartPole-v1_PER_DQN_20221114-104649/results/res.csv b/projects/codes/PER_DQN/Test_CartPole-v1_PER_DQN_20221114-104649/results/res.csv deleted file mode 100644 index cbbcf2e..0000000 --- a/projects/codes/PER_DQN/Test_CartPole-v1_PER_DQN_20221114-104649/results/res.csv +++ /dev/null @@ -1,11 +0,0 @@ -episodes,rewards,steps -0,200.0,200 -1,200.0,200 -2,200.0,200 -3,200.0,200 -4,200.0,200 -5,200.0,200 -6,200.0,200 -7,200.0,200 -8,200.0,200 -9,200.0,200 diff --git a/projects/codes/PER_DQN/Train_CartPole-v1_PER_DQN_20221113-162804/config.yaml b/projects/codes/PER_DQN/Train_CartPole-v1_PER_DQN_20221113-162804/config.yaml deleted file mode 100644 index bd4f2bd..0000000 --- a/projects/codes/PER_DQN/Train_CartPole-v1_PER_DQN_20221113-162804/config.yaml +++ /dev/null @@ -1,25 +0,0 @@ -general_cfg: - algo_name: PER_DQN - device: cuda - env_name: CartPole-v1 - eval_eps: 10 - eval_per_episode: 5 - load_checkpoint: false - load_path: tasks - max_steps: 200 - mode: train - save_fig: true - seed: 1 - show_fig: false - test_eps: 10 - train_eps: 200 -algo_cfg: - batch_size: 64 - buffer_size: 100000 - epsilon_decay: 500 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - hidden_dim: 256 - lr: 0.0001 - target_update: 4 diff --git a/projects/codes/PER_DQN/Train_CartPole-v1_PER_DQN_20221113-162804/logs/log.txt b/projects/codes/PER_DQN/Train_CartPole-v1_PER_DQN_20221113-162804/logs/log.txt deleted file mode 100644 index 1cea48c..0000000 --- a/projects/codes/PER_DQN/Train_CartPole-v1_PER_DQN_20221113-162804/logs/log.txt +++ /dev/null @@ -1,224 +0,0 @@ -2022-11-13 16:28:04 - r - INFO: - n_states: 4, n_actions: 2 -2022-11-13 16:28:19 - r - INFO: - Start training! -2022-11-13 16:28:19 - r - INFO: - Env: CartPole-v1, Algorithm: PER_DQN, Device: cuda -2022-11-13 16:28:23 - r - INFO: - Episode: 1/200, Reward: 18.000, Step: 18 -2022-11-13 16:28:24 - r - INFO: - Episode: 2/200, Reward: 35.000, Step: 35 -2022-11-13 16:28:24 - r - INFO: - Episode: 3/200, Reward: 13.000, Step: 13 -2022-11-13 16:28:24 - r - INFO: - Episode: 4/200, Reward: 20.000, Step: 20 -2022-11-13 16:28:24 - r - INFO: - Episode: 5/200, Reward: 24.000, Step: 24 -2022-11-13 16:28:24 - r - INFO: - Current episode 5 has the best eval reward: 9.100 -2022-11-13 16:28:24 - r - INFO: - Episode: 6/200, Reward: 10.000, Step: 10 -2022-11-13 16:28:24 - r - INFO: - Episode: 7/200, Reward: 20.000, Step: 20 -2022-11-13 16:28:24 - r - INFO: - Episode: 8/200, Reward: 19.000, Step: 19 -2022-11-13 16:28:25 - r - INFO: - Episode: 9/200, Reward: 30.000, Step: 30 -2022-11-13 16:28:25 - r - INFO: - Episode: 10/200, Reward: 10.000, Step: 10 -2022-11-13 16:28:25 - r - INFO: - Current episode 10 has the best eval reward: 9.200 -2022-11-13 16:28:25 - r - INFO: - Episode: 11/200, Reward: 16.000, Step: 16 -2022-11-13 16:28:25 - r - INFO: - Episode: 12/200, Reward: 16.000, Step: 16 -2022-11-13 16:28:25 - r - INFO: - Episode: 13/200, Reward: 12.000, Step: 12 -2022-11-13 16:28:25 - r - INFO: - Episode: 14/200, Reward: 28.000, Step: 28 -2022-11-13 16:28:25 - r - INFO: - Episode: 15/200, Reward: 22.000, Step: 22 -2022-11-13 16:28:25 - r - INFO: - Current episode 15 has the best eval reward: 9.300 -2022-11-13 16:28:25 - r - INFO: - Episode: 16/200, Reward: 14.000, Step: 14 -2022-11-13 16:28:25 - r - INFO: - Episode: 17/200, Reward: 9.000, Step: 9 -2022-11-13 16:28:26 - r - INFO: - Episode: 18/200, Reward: 13.000, Step: 13 -2022-11-13 16:28:26 - r - INFO: - Episode: 19/200, Reward: 19.000, Step: 19 -2022-11-13 16:28:26 - r - INFO: - Episode: 20/200, Reward: 10.000, Step: 10 -2022-11-13 16:28:26 - r - INFO: - Episode: 21/200, Reward: 10.000, Step: 10 -2022-11-13 16:28:26 - r - INFO: - Episode: 22/200, Reward: 12.000, Step: 12 -2022-11-13 16:28:26 - r - INFO: - Episode: 23/200, Reward: 9.000, Step: 9 -2022-11-13 16:28:26 - r - INFO: - Episode: 24/200, Reward: 12.000, Step: 12 -2022-11-13 16:28:26 - r - INFO: - Episode: 25/200, Reward: 11.000, Step: 11 -2022-11-13 16:28:26 - r - INFO: - Current episode 25 has the best eval reward: 9.800 -2022-11-13 16:28:26 - r - INFO: - Episode: 26/200, Reward: 11.000, Step: 11 -2022-11-13 16:28:26 - r - INFO: - Episode: 27/200, Reward: 13.000, Step: 13 -2022-11-13 16:28:26 - r - INFO: - Episode: 28/200, Reward: 11.000, Step: 11 -2022-11-13 16:28:27 - r - INFO: - Episode: 29/200, Reward: 13.000, Step: 13 -2022-11-13 16:28:27 - r - INFO: - Episode: 30/200, Reward: 20.000, Step: 20 -2022-11-13 16:28:27 - r - INFO: - Current episode 30 has the best eval reward: 12.200 -2022-11-13 16:28:27 - r - INFO: - Episode: 31/200, Reward: 16.000, Step: 16 -2022-11-13 16:28:27 - r - INFO: - Episode: 32/200, Reward: 9.000, Step: 9 -2022-11-13 16:28:27 - r - INFO: - Episode: 33/200, Reward: 16.000, Step: 16 -2022-11-13 16:28:27 - r - INFO: - Episode: 34/200, Reward: 15.000, Step: 15 -2022-11-13 16:28:27 - r - INFO: - Episode: 35/200, Reward: 12.000, Step: 12 -2022-11-13 16:28:27 - r - INFO: - Current episode 35 has the best eval reward: 12.500 -2022-11-13 16:28:27 - r - INFO: - Episode: 36/200, Reward: 12.000, Step: 12 -2022-11-13 16:28:27 - r - INFO: - Episode: 37/200, Reward: 16.000, Step: 16 -2022-11-13 16:28:28 - r - INFO: - Episode: 38/200, Reward: 13.000, Step: 13 -2022-11-13 16:28:28 - r - INFO: - Episode: 39/200, Reward: 18.000, Step: 18 -2022-11-13 16:28:28 - r - INFO: - Episode: 40/200, Reward: 18.000, Step: 18 -2022-11-13 16:28:28 - r - INFO: - Current episode 40 has the best eval reward: 20.400 -2022-11-13 16:28:28 - r - INFO: - Episode: 41/200, Reward: 48.000, Step: 48 -2022-11-13 16:28:29 - r - INFO: - Episode: 42/200, Reward: 52.000, Step: 52 -2022-11-13 16:28:29 - r - INFO: - Episode: 43/200, Reward: 33.000, Step: 33 -2022-11-13 16:28:29 - r - INFO: - Episode: 44/200, Reward: 15.000, Step: 15 -2022-11-13 16:28:29 - r - INFO: - Episode: 45/200, Reward: 18.000, Step: 18 -2022-11-13 16:28:29 - r - INFO: - Episode: 46/200, Reward: 22.000, Step: 22 -2022-11-13 16:28:29 - r - INFO: - Episode: 47/200, Reward: 19.000, Step: 19 -2022-11-13 16:28:30 - r - INFO: - Episode: 48/200, Reward: 19.000, Step: 19 -2022-11-13 16:28:30 - r - INFO: - Episode: 49/200, Reward: 11.000, Step: 11 -2022-11-13 16:28:30 - r - INFO: - Episode: 50/200, Reward: 9.000, Step: 9 -2022-11-13 16:28:30 - r - INFO: - Episode: 51/200, Reward: 10.000, Step: 10 -2022-11-13 16:28:30 - r - INFO: - Episode: 52/200, Reward: 10.000, Step: 10 -2022-11-13 16:28:30 - r - INFO: - Episode: 53/200, Reward: 10.000, Step: 10 -2022-11-13 16:28:30 - r - INFO: - Episode: 54/200, Reward: 10.000, Step: 10 -2022-11-13 16:28:30 - r - INFO: - Episode: 55/200, Reward: 9.000, Step: 9 -2022-11-13 16:28:30 - r - INFO: - Episode: 56/200, Reward: 17.000, Step: 17 -2022-11-13 16:28:31 - r - INFO: - Episode: 57/200, Reward: 75.000, Step: 75 -2022-11-13 16:28:31 - r - INFO: - Episode: 58/200, Reward: 28.000, Step: 28 -2022-11-13 16:28:31 - r - INFO: - Episode: 59/200, Reward: 30.000, Step: 30 -2022-11-13 16:28:32 - r - INFO: - Episode: 60/200, Reward: 54.000, Step: 54 -2022-11-13 16:28:32 - r - INFO: - Current episode 60 has the best eval reward: 34.600 -2022-11-13 16:28:32 - r - INFO: - Episode: 61/200, Reward: 22.000, Step: 22 -2022-11-13 16:28:32 - r - INFO: - Episode: 62/200, Reward: 28.000, Step: 28 -2022-11-13 16:28:32 - r - INFO: - Episode: 63/200, Reward: 26.000, Step: 26 -2022-11-13 16:28:33 - r - INFO: - Episode: 64/200, Reward: 32.000, Step: 32 -2022-11-13 16:28:33 - r - INFO: - Episode: 65/200, Reward: 30.000, Step: 30 -2022-11-13 16:28:33 - r - INFO: - Episode: 66/200, Reward: 29.000, Step: 29 -2022-11-13 16:28:34 - r - INFO: - Episode: 67/200, Reward: 28.000, Step: 28 -2022-11-13 16:28:34 - r - INFO: - Episode: 68/200, Reward: 38.000, Step: 38 -2022-11-13 16:28:34 - r - INFO: - Episode: 69/200, Reward: 28.000, Step: 28 -2022-11-13 16:28:34 - r - INFO: - Episode: 70/200, Reward: 22.000, Step: 22 -2022-11-13 16:28:34 - r - INFO: - Current episode 70 has the best eval reward: 36.700 -2022-11-13 16:28:35 - r - INFO: - Episode: 71/200, Reward: 40.000, Step: 40 -2022-11-13 16:28:35 - r - INFO: - Episode: 72/200, Reward: 27.000, Step: 27 -2022-11-13 16:28:35 - r - INFO: - Episode: 73/200, Reward: 24.000, Step: 24 -2022-11-13 16:28:35 - r - INFO: - Episode: 74/200, Reward: 47.000, Step: 47 -2022-11-13 16:28:36 - r - INFO: - Episode: 75/200, Reward: 127.000, Step: 127 -2022-11-13 16:28:37 - r - INFO: - Episode: 76/200, Reward: 48.000, Step: 48 -2022-11-13 16:28:37 - r - INFO: - Episode: 77/200, Reward: 27.000, Step: 27 -2022-11-13 16:28:37 - r - INFO: - Episode: 78/200, Reward: 65.000, Step: 65 -2022-11-13 16:28:38 - r - INFO: - Episode: 79/200, Reward: 75.000, Step: 75 -2022-11-13 16:28:38 - r - INFO: - Episode: 80/200, Reward: 47.000, Step: 47 -2022-11-13 16:28:38 - r - INFO: - Current episode 80 has the best eval reward: 37.200 -2022-11-13 16:28:39 - r - INFO: - Episode: 81/200, Reward: 34.000, Step: 34 -2022-11-13 16:28:39 - r - INFO: - Episode: 82/200, Reward: 38.000, Step: 38 -2022-11-13 16:28:39 - r - INFO: - Episode: 83/200, Reward: 24.000, Step: 24 -2022-11-13 16:28:39 - r - INFO: - Episode: 84/200, Reward: 47.000, Step: 47 -2022-11-13 16:28:40 - r - INFO: - Episode: 85/200, Reward: 35.000, Step: 35 -2022-11-13 16:28:40 - r - INFO: - Current episode 85 has the best eval reward: 66.900 -2022-11-13 16:28:41 - r - INFO: - Episode: 86/200, Reward: 103.000, Step: 103 -2022-11-13 16:28:41 - r - INFO: - Episode: 87/200, Reward: 64.000, Step: 64 -2022-11-13 16:28:42 - r - INFO: - Episode: 88/200, Reward: 59.000, Step: 59 -2022-11-13 16:28:43 - r - INFO: - Episode: 89/200, Reward: 200.000, Step: 200 -2022-11-13 16:28:44 - r - INFO: - Episode: 90/200, Reward: 200.000, Step: 200 -2022-11-13 16:28:46 - r - INFO: - Current episode 90 has the best eval reward: 200.000 -2022-11-13 16:28:47 - r - INFO: - Episode: 91/200, Reward: 200.000, Step: 200 -2022-11-13 16:28:48 - r - INFO: - Episode: 92/200, Reward: 200.000, Step: 200 -2022-11-13 16:28:50 - r - INFO: - Episode: 93/200, Reward: 200.000, Step: 200 -2022-11-13 16:28:51 - r - INFO: - Episode: 94/200, Reward: 200.000, Step: 200 -2022-11-13 16:28:52 - r - INFO: - Episode: 95/200, Reward: 200.000, Step: 200 -2022-11-13 16:28:54 - r - INFO: - Current episode 95 has the best eval reward: 200.000 -2022-11-13 16:28:55 - r - INFO: - Episode: 96/200, Reward: 200.000, Step: 200 -2022-11-13 16:28:56 - r - INFO: - Episode: 97/200, Reward: 200.000, Step: 200 -2022-11-13 16:28:58 - r - INFO: - Episode: 98/200, Reward: 200.000, Step: 200 -2022-11-13 16:28:59 - r - INFO: - Episode: 99/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:00 - r - INFO: - Episode: 100/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:02 - r - INFO: - Current episode 100 has the best eval reward: 200.000 -2022-11-13 16:29:04 - r - INFO: - Episode: 101/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:05 - r - INFO: - Episode: 102/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:06 - r - INFO: - Episode: 103/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:07 - r - INFO: - Episode: 104/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:09 - r - INFO: - Episode: 105/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:10 - r - INFO: - Current episode 105 has the best eval reward: 200.000 -2022-11-13 16:29:11 - r - INFO: - Episode: 106/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:13 - r - INFO: - Episode: 107/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:14 - r - INFO: - Episode: 108/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:16 - r - INFO: - Episode: 109/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:17 - r - INFO: - Episode: 110/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:20 - r - INFO: - Episode: 111/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:21 - r - INFO: - Episode: 112/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:22 - r - INFO: - Episode: 113/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:23 - r - INFO: - Episode: 114/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:25 - r - INFO: - Episode: 115/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:26 - r - INFO: - Current episode 115 has the best eval reward: 200.000 -2022-11-13 16:29:27 - r - INFO: - Episode: 116/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:29 - r - INFO: - Episode: 117/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:30 - r - INFO: - Episode: 118/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:31 - r - INFO: - Episode: 119/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:33 - r - INFO: - Episode: 120/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:34 - r - INFO: - Current episode 120 has the best eval reward: 200.000 -2022-11-13 16:29:35 - r - INFO: - Episode: 121/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:37 - r - INFO: - Episode: 122/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:38 - r - INFO: - Episode: 123/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:39 - r - INFO: - Episode: 124/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:41 - r - INFO: - Episode: 125/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:43 - r - INFO: - Episode: 126/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:45 - r - INFO: - Episode: 127/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:46 - r - INFO: - Episode: 128/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:47 - r - INFO: - Episode: 129/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:49 - r - INFO: - Episode: 130/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:51 - r - INFO: - Episode: 131/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:53 - r - INFO: - Episode: 132/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:54 - r - INFO: - Episode: 133/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:55 - r - INFO: - Episode: 134/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:57 - r - INFO: - Episode: 135/200, Reward: 200.000, Step: 200 -2022-11-13 16:29:59 - r - INFO: - Episode: 136/200, Reward: 200.000, Step: 200 -2022-11-13 16:30:01 - r - INFO: - Episode: 137/200, Reward: 185.000, Step: 185 -2022-11-13 16:30:02 - r - INFO: - Episode: 138/200, Reward: 193.000, Step: 193 -2022-11-13 16:30:03 - r - INFO: - Episode: 139/200, Reward: 192.000, Step: 192 -2022-11-13 16:30:04 - r - INFO: - Episode: 140/200, Reward: 200.000, Step: 200 -2022-11-13 16:30:07 - r - INFO: - Episode: 141/200, Reward: 200.000, Step: 200 -2022-11-13 16:30:08 - r - INFO: - Episode: 142/200, Reward: 200.000, Step: 200 -2022-11-13 16:30:10 - r - INFO: - Episode: 143/200, Reward: 200.000, Step: 200 -2022-11-13 16:30:11 - r - INFO: - Episode: 144/200, Reward: 191.000, Step: 191 -2022-11-13 16:30:12 - r - INFO: - Episode: 145/200, Reward: 200.000, Step: 200 -2022-11-13 16:30:15 - r - INFO: - Episode: 146/200, Reward: 184.000, Step: 184 -2022-11-13 16:30:17 - r - INFO: - Episode: 147/200, Reward: 198.000, Step: 198 -2022-11-13 16:30:18 - r - INFO: - Episode: 148/200, Reward: 200.000, Step: 200 -2022-11-13 16:30:19 - r - INFO: - Episode: 149/200, Reward: 200.000, Step: 200 -2022-11-13 16:30:21 - r - INFO: - Episode: 150/200, Reward: 192.000, Step: 192 -2022-11-13 16:30:23 - r - INFO: - Episode: 151/200, Reward: 186.000, Step: 186 -2022-11-13 16:30:25 - r - INFO: - Episode: 152/200, Reward: 200.000, Step: 200 -2022-11-13 16:30:26 - r - INFO: - Episode: 153/200, Reward: 194.000, Step: 194 -2022-11-13 16:30:27 - r - INFO: - Episode: 154/200, Reward: 199.000, Step: 199 -2022-11-13 16:30:29 - r - INFO: - Episode: 155/200, Reward: 183.000, Step: 183 -2022-11-13 16:30:32 - r - INFO: - Episode: 156/200, Reward: 173.000, Step: 173 -2022-11-13 16:30:33 - r - INFO: - Episode: 157/200, Reward: 197.000, Step: 197 -2022-11-13 16:30:34 - r - INFO: - Episode: 158/200, Reward: 200.000, Step: 200 -2022-11-13 16:30:36 - r - INFO: - Episode: 159/200, Reward: 200.000, Step: 200 -2022-11-13 16:30:37 - r - INFO: - Episode: 160/200, Reward: 196.000, Step: 196 -2022-11-13 16:30:40 - r - INFO: - Episode: 161/200, Reward: 200.000, Step: 200 -2022-11-13 16:30:42 - r - INFO: - Episode: 162/200, Reward: 200.000, Step: 200 -2022-11-13 16:30:43 - r - INFO: - Episode: 163/200, Reward: 194.000, Step: 194 -2022-11-13 16:30:44 - r - INFO: - Episode: 164/200, Reward: 185.000, Step: 185 -2022-11-13 16:30:45 - r - INFO: - Episode: 165/200, Reward: 173.000, Step: 173 -2022-11-13 16:30:48 - r - INFO: - Episode: 166/200, Reward: 192.000, Step: 192 -2022-11-13 16:30:49 - r - INFO: - Episode: 167/200, Reward: 164.000, Step: 164 -2022-11-13 16:30:50 - r - INFO: - Episode: 168/200, Reward: 188.000, Step: 188 -2022-11-13 16:30:52 - r - INFO: - Episode: 169/200, Reward: 189.000, Step: 189 -2022-11-13 16:30:53 - r - INFO: - Episode: 170/200, Reward: 197.000, Step: 197 -2022-11-13 16:30:55 - r - INFO: - Episode: 171/200, Reward: 187.000, Step: 187 -2022-11-13 16:30:57 - r - INFO: - Episode: 172/200, Reward: 200.000, Step: 200 -2022-11-13 16:30:58 - r - INFO: - Episode: 173/200, Reward: 195.000, Step: 195 -2022-11-13 16:30:59 - r - INFO: - Episode: 174/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:01 - r - INFO: - Episode: 175/200, Reward: 195.000, Step: 195 -2022-11-13 16:31:03 - r - INFO: - Episode: 176/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:05 - r - INFO: - Episode: 177/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:06 - r - INFO: - Episode: 178/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:07 - r - INFO: - Episode: 179/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:09 - r - INFO: - Episode: 180/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:11 - r - INFO: - Episode: 181/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:13 - r - INFO: - Episode: 182/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:14 - r - INFO: - Episode: 183/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:15 - r - INFO: - Episode: 184/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:17 - r - INFO: - Episode: 185/200, Reward: 173.000, Step: 173 -2022-11-13 16:31:19 - r - INFO: - Episode: 186/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:21 - r - INFO: - Episode: 187/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:22 - r - INFO: - Episode: 188/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:23 - r - INFO: - Episode: 189/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:24 - r - INFO: - Episode: 190/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:26 - r - INFO: - Current episode 190 has the best eval reward: 200.000 -2022-11-13 16:31:27 - r - INFO: - Episode: 191/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:29 - r - INFO: - Episode: 192/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:30 - r - INFO: - Episode: 193/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:31 - r - INFO: - Episode: 194/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:33 - r - INFO: - Episode: 195/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:34 - r - INFO: - Current episode 195 has the best eval reward: 200.000 -2022-11-13 16:31:35 - r - INFO: - Episode: 196/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:37 - r - INFO: - Episode: 197/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:38 - r - INFO: - Episode: 198/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:39 - r - INFO: - Episode: 199/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:40 - r - INFO: - Episode: 200/200, Reward: 200.000, Step: 200 -2022-11-13 16:31:42 - r - INFO: - Current episode 200 has the best eval reward: 200.000 -2022-11-13 16:31:42 - r - INFO: - Finish training! diff --git a/projects/codes/PER_DQN/Train_CartPole-v1_PER_DQN_20221113-162804/models/checkpoint.pt b/projects/codes/PER_DQN/Train_CartPole-v1_PER_DQN_20221113-162804/models/checkpoint.pt deleted file mode 100644 index acaef5b..0000000 Binary files a/projects/codes/PER_DQN/Train_CartPole-v1_PER_DQN_20221113-162804/models/checkpoint.pt and /dev/null differ diff --git a/projects/codes/PER_DQN/Train_CartPole-v1_PER_DQN_20221113-162804/results/learning_curve.png b/projects/codes/PER_DQN/Train_CartPole-v1_PER_DQN_20221113-162804/results/learning_curve.png deleted file mode 100644 index 6f666e3..0000000 Binary files a/projects/codes/PER_DQN/Train_CartPole-v1_PER_DQN_20221113-162804/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/PER_DQN/Train_CartPole-v1_PER_DQN_20221113-162804/results/res.csv b/projects/codes/PER_DQN/Train_CartPole-v1_PER_DQN_20221113-162804/results/res.csv deleted file mode 100644 index 1c3339f..0000000 --- a/projects/codes/PER_DQN/Train_CartPole-v1_PER_DQN_20221113-162804/results/res.csv +++ /dev/null @@ -1,201 +0,0 @@ -episodes,rewards,steps -0,18.0,18 -1,35.0,35 -2,13.0,13 -3,20.0,20 -4,24.0,24 -5,10.0,10 -6,20.0,20 -7,19.0,19 -8,30.0,30 -9,10.0,10 -10,16.0,16 -11,16.0,16 -12,12.0,12 -13,28.0,28 -14,22.0,22 -15,14.0,14 -16,9.0,9 -17,13.0,13 -18,19.0,19 -19,10.0,10 -20,10.0,10 -21,12.0,12 -22,9.0,9 -23,12.0,12 -24,11.0,11 -25,11.0,11 -26,13.0,13 -27,11.0,11 -28,13.0,13 -29,20.0,20 -30,16.0,16 -31,9.0,9 -32,16.0,16 -33,15.0,15 -34,12.0,12 -35,12.0,12 -36,16.0,16 -37,13.0,13 -38,18.0,18 -39,18.0,18 -40,48.0,48 -41,52.0,52 -42,33.0,33 -43,15.0,15 -44,18.0,18 -45,22.0,22 -46,19.0,19 -47,19.0,19 -48,11.0,11 -49,9.0,9 -50,10.0,10 -51,10.0,10 -52,10.0,10 -53,10.0,10 -54,9.0,9 -55,17.0,17 -56,75.0,75 -57,28.0,28 -58,30.0,30 -59,54.0,54 -60,22.0,22 -61,28.0,28 -62,26.0,26 -63,32.0,32 -64,30.0,30 -65,29.0,29 -66,28.0,28 -67,38.0,38 -68,28.0,28 -69,22.0,22 -70,40.0,40 -71,27.0,27 -72,24.0,24 -73,47.0,47 -74,127.0,127 -75,48.0,48 -76,27.0,27 -77,65.0,65 -78,75.0,75 -79,47.0,47 -80,34.0,34 -81,38.0,38 -82,24.0,24 -83,47.0,47 -84,35.0,35 -85,103.0,103 -86,64.0,64 -87,59.0,59 -88,200.0,200 -89,200.0,200 -90,200.0,200 -91,200.0,200 -92,200.0,200 -93,200.0,200 -94,200.0,200 -95,200.0,200 -96,200.0,200 -97,200.0,200 -98,200.0,200 -99,200.0,200 -100,200.0,200 -101,200.0,200 -102,200.0,200 -103,200.0,200 -104,200.0,200 -105,200.0,200 -106,200.0,200 -107,200.0,200 -108,200.0,200 -109,200.0,200 -110,200.0,200 -111,200.0,200 -112,200.0,200 -113,200.0,200 -114,200.0,200 -115,200.0,200 -116,200.0,200 -117,200.0,200 -118,200.0,200 -119,200.0,200 -120,200.0,200 -121,200.0,200 -122,200.0,200 -123,200.0,200 -124,200.0,200 -125,200.0,200 -126,200.0,200 -127,200.0,200 -128,200.0,200 -129,200.0,200 -130,200.0,200 -131,200.0,200 -132,200.0,200 -133,200.0,200 -134,200.0,200 -135,200.0,200 -136,185.0,185 -137,193.0,193 -138,192.0,192 -139,200.0,200 -140,200.0,200 -141,200.0,200 -142,200.0,200 -143,191.0,191 -144,200.0,200 -145,184.0,184 -146,198.0,198 -147,200.0,200 -148,200.0,200 -149,192.0,192 -150,186.0,186 -151,200.0,200 -152,194.0,194 -153,199.0,199 -154,183.0,183 -155,173.0,173 -156,197.0,197 -157,200.0,200 -158,200.0,200 -159,196.0,196 -160,200.0,200 -161,200.0,200 -162,194.0,194 -163,185.0,185 -164,173.0,173 -165,192.0,192 -166,164.0,164 -167,188.0,188 -168,189.0,189 -169,197.0,197 -170,187.0,187 -171,200.0,200 -172,195.0,195 -173,200.0,200 -174,195.0,195 -175,200.0,200 -176,200.0,200 -177,200.0,200 -178,200.0,200 -179,200.0,200 -180,200.0,200 -181,200.0,200 -182,200.0,200 -183,200.0,200 -184,173.0,173 -185,200.0,200 -186,200.0,200 -187,200.0,200 -188,200.0,200 -189,200.0,200 -190,200.0,200 -191,200.0,200 -192,200.0,200 -193,200.0,200 -194,200.0,200 -195,200.0,200 -196,200.0,200 -197,200.0,200 -198,200.0,200 -199,200.0,200 diff --git a/projects/codes/PER_DQN/config/CartPole-v1_PER_DQN_Test.yaml b/projects/codes/PER_DQN/config/CartPole-v1_PER_DQN_Test.yaml deleted file mode 100644 index a1db2ab..0000000 --- a/projects/codes/PER_DQN/config/CartPole-v1_PER_DQN_Test.yaml +++ /dev/null @@ -1,22 +0,0 @@ -general_cfg: - algo_name: PER_DQN - device: cpu - env_name: CartPole-v1 - mode: test - load_checkpoint: true - load_path: Train_CartPole-v1_PER_DQN_20221113-162804 - max_steps: 200 - save_fig: true - seed: 0 - show_fig: false - test_eps: 10 - train_eps: 200 -algo_cfg: - batch_size: 64 - buffer_size: 100000 - epsilon_decay: 500 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - lr: 0.0001 - target_update: 4 diff --git a/projects/codes/PER_DQN/config/CartPole-v1_PER_DQN_Train.yaml b/projects/codes/PER_DQN/config/CartPole-v1_PER_DQN_Train.yaml deleted file mode 100644 index 553622f..0000000 --- a/projects/codes/PER_DQN/config/CartPole-v1_PER_DQN_Train.yaml +++ /dev/null @@ -1,22 +0,0 @@ -general_cfg: - algo_name: PER_DQN - device: cuda - env_name: CartPole-v1 - mode: train - load_checkpoint: false - load_path: Train_CartPole-v1_PER_DQN_20221026-054757 - max_steps: 200 - save_fig: true - seed: 0 - show_fig: false - test_eps: 10 - train_eps: 200 -algo_cfg: - batch_size: 64 - buffer_size: 100000 - epsilon_decay: 500 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - lr: 0.0001 - target_update: 4 diff --git a/projects/codes/PER_DQN/config/config.py b/projects/codes/PER_DQN/config/config.py deleted file mode 100644 index a92c7e0..0000000 --- a/projects/codes/PER_DQN/config/config.py +++ /dev/null @@ -1,38 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-10-30 00:37:33 -LastEditor: JiangJi -LastEditTime: 2022-10-30 01:19:08 -Discription: default parameters of DQN -''' -from common.config import GeneralConfig,AlgoConfig -class GeneralConfigDQN(GeneralConfig): - def __init__(self) -> None: - self.env_name = "CartPole-v1" # name of environment - self.algo_name = "PER_DQN" # name of algorithm - self.mode = "train" # train or test - self.seed = 1 # random seed - self.device = "cuda" # device to use - self.train_eps = 200 # number of episodes for training - self.test_eps = 10 # number of episodes for testing - self.max_steps = 200 # max steps for each episode - self.load_checkpoint = False - self.load_path = "tasks" # path to load model - self.show_fig = False # show figure or not - self.save_fig = True # save figure or not - -class AlgoConfigDQN(AlgoConfig): - def __init__(self) -> None: - # set epsilon_start=epsilon_end can obtain fixed epsilon=epsilon_end - self.epsilon_start = 0.95 # epsilon start value - self.epsilon_end = 0.01 # epsilon end value - self.epsilon_decay = 500 # epsilon decay rate - self.hidden_dim = 256 # hidden_dim for MLP - self.gamma = 0.95 # discount factor - self.lr = 0.0001 # learning rate - self.buffer_size = 100000 # size of replay buffer - self.batch_size = 64 # batch size - self.target_update = 4 # target network update frequency diff --git a/projects/codes/PER_DQN/per_dqn.py b/projects/codes/PER_DQN/per_dqn.py deleted file mode 100644 index 6fbf651..0000000 --- a/projects/codes/PER_DQN/per_dqn.py +++ /dev/null @@ -1,139 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: DingLi -Email: wangzhongren@sjtu.edu.cn -Date: 2022-10-31 22:54:00 -LastEditor: DingLi -LastEditTime: 2022-11-14 10:43:18 -Discription: CartPole-v1 -''' - -''' -@Author: John -@Email: johnjim0816@gmail.com -@Date: 2020-06-12 00:50:49 -@LastEditor: John -LastEditTime: 2022-10-26 07:50:24 -@Discription: -@Environment: python 3.7.7 -''' -'''off-policy -''' - -import torch -import torch.nn as nn -import torch.optim as optim -import random -import math -import numpy as np - -class PER_DQN: - def __init__(self,model,memory,cfg): - - self.n_actions = cfg.n_actions - self.device = torch.device(cfg.device) - self.gamma = cfg.gamma - ## e-greedy parameters - self.sample_count = 0 # sample count for epsilon decay - self.epsilon = cfg.epsilon_start - self.sample_count = 0 - self.epsilon_start = cfg.epsilon_start - self.epsilon_end = cfg.epsilon_end - self.epsilon_decay = cfg.epsilon_decay - self.batch_size = cfg.batch_size - self.policy_net = model.to(self.device) - self.target_net = model.to(self.device) - ## copy parameters from policy net to target net - for target_param, param in zip(self.target_net.parameters(),self.policy_net.parameters()): - target_param.data.copy_(param.data) - # self.target_net.load_state_dict(self.policy_net.state_dict()) # or use this to copy parameters - self.optimizer = optim.Adam(self.policy_net.parameters(), lr=cfg.lr) - self.memory = memory - self.update_flag = False - - def sample_action(self, state): - ''' sample action with e-greedy policy - ''' - self.sample_count += 1 - # epsilon must decay(linear,exponential and etc.) for balancing exploration and exploitation - self.epsilon = self.epsilon_end + (self.epsilon_start - self.epsilon_end) * \ - math.exp(-1. * self.sample_count / self.epsilon_decay) - if random.random() > self.epsilon: - with torch.no_grad(): - state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(dim=0) - q_values = self.policy_net(state) - action = q_values.max(1)[1].item() # choose action corresponding to the maximum q value - else: - action = random.randrange(self.n_actions) - return action - # @torch.no_grad() - # def sample_action(self, state): - # ''' sample action with e-greedy policy - # ''' - # self.sample_count += 1 - # # epsilon must decay(linear,exponential and etc.) for balancing exploration and exploitation - # self.epsilon = self.epsilon_end + (self.epsilon_start - self.epsilon_end) * \ - # math.exp(-1. * self.sample_count / self.epsilon_decay) - # if random.random() > self.epsilon: - # state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(dim=0) - # q_values = self.policy_net(state) - # action = q_values.max(1)[1].item() # choose action corresponding to the maximum q value - # else: - # action = random.randrange(self.n_actions) - # return action - def predict_action(self,state): - ''' predict action - ''' - with torch.no_grad(): - state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(dim=0) - q_values = self.policy_net(state) - action = q_values.max(1)[1].item() # choose action corresponding to the maximum q value - return action - def update(self): - if len(self.memory) < self.batch_size: # when transitions in memory donot meet a batch, not update - # print ("self.batch_size = ", self.batch_size) - return - else: - if not self.update_flag: - print("Begin to update!") - self.update_flag = True - # sample a batch of transitions from replay buffer - (state_batch, action_batch, reward_batch, next_state_batch, done_batch), idxs_batch, is_weights_batch = self.memory.sample( - self.batch_size) - state_batch = torch.tensor(np.array(state_batch), device=self.device, dtype=torch.float) # shape(batchsize,n_states) - action_batch = torch.tensor(action_batch, device=self.device).unsqueeze(1) # shape(batchsize,1) - reward_batch = torch.tensor(reward_batch, device=self.device, dtype=torch.float).unsqueeze(1) # shape(batchsize,1) - next_state_batch = torch.tensor(np.array(next_state_batch), device=self.device, dtype=torch.float) # shape(batchsize,n_states) - done_batch = torch.tensor(np.float32(done_batch), device=self.device).unsqueeze(1) # shape(batchsize,1) - q_value_batch = self.policy_net(state_batch).gather(dim=1, index=action_batch) # shape(batchsize,1),requires_grad=True - next_max_q_value_batch = self.target_net(next_state_batch).max(1)[0].detach().unsqueeze(1) - expected_q_value_batch = reward_batch + self.gamma * next_max_q_value_batch* (1-done_batch) - - loss = torch.mean(torch.pow((q_value_batch - expected_q_value_batch) * torch.from_numpy(is_weights_batch).cuda(), 2)) - # loss = nn.MSELoss()(q_value_batch, expected_q_value_batch) # shape same to - - abs_errors = np.sum(np.abs(q_value_batch.cpu().detach().numpy() - expected_q_value_batch.cpu().detach().numpy()), axis=1) - self.memory.batch_update(idxs_batch, abs_errors) - - # backpropagation - self.optimizer.zero_grad() - loss.backward() - # clip to avoid gradient explosion - for param in self.policy_net.parameters(): - param.grad.data.clamp_(-1, 1) - self.optimizer.step() - if self.sample_count % self.target_update == 0: # target net update, target_update means "C" in pseucodes - self.target_net.load_state_dict(self.policy_net.state_dict()) - - def save_model(self, fpath): - from pathlib import Path - # create path - Path(fpath).mkdir(parents=True, exist_ok=True) - torch.save(self.target_net.state_dict(), f"{fpath}/checkpoint.pt") - - def load_model(self, fpath): - checkpoint = torch.load(f"{fpath}/checkpoint.pt",map_location=self.device) - self.target_net.load_state_dict(checkpoint) - for target_param, param in zip(self.target_net.parameters(), self.policy_net.parameters()): - param.data.copy_(target_param.data) diff --git a/projects/codes/PER_DQN/task0.py b/projects/codes/PER_DQN/task0.py deleted file mode 100644 index 8b6247b..0000000 --- a/projects/codes/PER_DQN/task0.py +++ /dev/null @@ -1,104 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: DingLi -Email: wangzhongren@sjtu.edu.cn -Date: 2022-10-31 22:54:00 -LastEditor: DingLi -LastEditTime: 2022-11-14 10:45:11 -Discription: CartPole-v1 -''' - -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-10-12 11:09:54 -LastEditor: JiangJi -LastEditTime: 2022-10-30 01:29:25 -Discription: CartPole-v1,Acrobot-v1 -''' -import sys,os -curr_path = os.path.dirname(os.path.abspath(__file__)) # current path -parent_path = os.path.dirname(curr_path) # parent path -sys.path.append(parent_path) # add to system path -import gym -import torch - -from common.utils import all_seed,merge_class_attrs -from common.models import MLP -from common.memories import ReplayBuffer, ReplayTree -from common.launcher import Launcher -from envs.register import register_env -from per_dqn import PER_DQN -from config.config import GeneralConfigDQN,AlgoConfigDQN -class Main(Launcher): - def __init__(self) -> None: - super().__init__() - self.cfgs['general_cfg'] = merge_class_attrs(self.cfgs['general_cfg'],GeneralConfigDQN()) - self.cfgs['algo_cfg'] = merge_class_attrs(self.cfgs['algo_cfg'],AlgoConfigDQN()) - def env_agent_config(self,cfg,logger): - ''' create env and agent - ''' - register_env(cfg.env_name) - env = gym.make(cfg.env_name,new_step_api=True) # create env - all_seed(env,seed=cfg.seed) # set random seed - try: # state dimension - n_states = env.observation_space.n # print(hasattr(env.observation_space, 'n')) - except AttributeError: - n_states = env.observation_space.shape[0] # print(hasattr(env.observation_space, 'shape')) - n_actions = env.action_space.n # action dimension - logger.info(f"n_states: {n_states}, n_actions: {n_actions}") # print info - # update to cfg paramters - setattr(cfg, 'n_states', n_states) - setattr(cfg, 'n_actions', n_actions) - # cfg.update({"n_states":n_states,"n_actions":n_actions}) # update to cfg paramters - model = MLP(n_states,n_actions,hidden_dim=cfg.hidden_dim) - memory = ReplayTree(cfg.buffer_size) # replay SumTree - agent = PER_DQN(model,memory,cfg) # create agent - return env, agent - - def train_one_episode(self,env, agent, cfg): - ''' train one episode - ''' - ep_step = 0 - state = env.reset() # reset and obtain initial state - for _ in range(cfg.max_steps): - ep_step += 1 - action = agent.sample_action(state) # sample action - next_state, reward, terminated, truncated , info = env.step(action) # update env and return transitions under new_step_api of OpenAI Gym - - policy_val = agent.policy_net(torch.tensor(state, device = cfg.device))[action] - target_val = agent.target_net(torch.tensor(next_state, device = cfg.device)) - - if terminated: - error = abs(policy_val - reward) - else: - error = abs(policy_val - reward - cfg.gamma * torch.max(target_val)) - agent.memory.push(error.cpu().detach().numpy(), (state, action, reward, - next_state, terminated)) # save transitions - state = next_state # update next state for env - agent.update() # update agent - ep_reward += reward # - if terminated: - break - return agent, ep_reward, ep_step - - def test_one_episode(self, env, agent, cfg): - ep_reward = 0 # reward per episode - ep_step = 0 - state = env.reset() # reset and obtain initial state - for _ in range(cfg.max_steps): - ep_step+=1 - action = agent.predict_action(state) # predict action - next_state, reward, terminated, _, _ = env.step(action) - state = next_state - ep_reward += reward - if terminated: - break - return agent, ep_reward, ep_step - - -if __name__ == "__main__": - main = Main() - main.run() - diff --git a/projects/codes/PPO/README.md b/projects/codes/PPO/README.md deleted file mode 100644 index 125ef51..0000000 --- a/projects/codes/PPO/README.md +++ /dev/null @@ -1,142 +0,0 @@ -## 原理简介 - -PPO是一种on-policy算法，具有较好的性能，其前身是TRPO算法，也是policy gradient算法的一种，它是现在 OpenAI 默认的强化学习算法，具体原理可参考[PPO算法讲解](https://datawhalechina.github.io/easy-rl/#/chapter5/chapter5)。PPO算法主要有两个变种，一个是结合KL penalty的，一个是用了clip方法，本文实现的是后者即```PPO-clip```。 -## 伪代码 -要实现必先了解伪代码，伪代码如下： -![在这里插入图片描述](assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70.png) -这是谷歌找到的一张比较适合的图，本人比较懒就没有修改，上面的```k```就是第```k```个episode，第六步是用随机梯度下降的方法优化，这里的损失函数(即```argmax```后面的部分)可能有点难理解，可参考[PPO paper](https://arxiv.org/abs/1707.06347)，如下： -![在这里插入图片描述](assets/20210323154236878.png) -第七步就是一个平方损失函数，即实际回报与期望回报的差平方。 -## 代码实战 -[点击查看完整代码](https://github.com/JohnJim0816/rl-tutorials/tree/master/PPO) -### PPOmemory -首先第三步需要搜集一条轨迹信息，我们可以定义一个```PPOmemory```来存储相关信息： -```python -class PPOMemory: - def __init__(self, batch_size): - self.states = [] - self.probs = [] - self.vals = [] - self.actions = [] - self.rewards = [] - self.dones = [] - self.batch_size = batch_size - def sample(self): - batch_step = np.arange(0, len(self.states), self.batch_size) - indices = np.arange(len(self.states), dtype=np.int64) - np.random.shuffle(indices) - batches = [indices[i:i+self.batch_size] for i in batch_step] - return np.array(self.states),\ - np.array(self.actions),\ - np.array(self.probs),\ - np.array(self.vals),\ - np.array(self.rewards),\ - np.array(self.dones),\ - batches - - def push(self, state, action, probs, vals, reward, done): - self.states.append(state) - self.actions.append(action) - self.probs.append(probs) - self.vals.append(vals) - self.rewards.append(reward) - self.dones.append(done) - - def clear(self): - self.states = [] - self.probs = [] - self.actions = [] - self.rewards = [] - self.dones = [] - self.vals = [] -``` -这里的push函数就是将得到的相关量放入memory中，sample就是随机采样出来，方便第六步的随机梯度下降。 -### PPO model -model就是actor和critic两个网络了： -```python -import torch.nn as nn -from torch.distributions.categorical import Categorical -class Actor(nn.Module): - def __init__(self,n_states, n_actions, - hidden_dim=256): - super(Actor, self).__init__() - - self.actor = nn.Sequential( - nn.Linear(n_states, hidden_dim), - nn.ReLU(), - nn.Linear(hidden_dim, hidden_dim), - nn.ReLU(), - nn.Linear(hidden_dim, n_actions), - nn.Softmax(dim=-1) - ) - def forward(self, state): - dist = self.actor(state) - dist = Categorical(dist) - return dist - -class Critic(nn.Module): - def __init__(self, n_states,hidden_dim=256): - super(Critic, self).__init__() - self.critic = nn.Sequential( - nn.Linear(n_states, hidden_dim), - nn.ReLU(), - nn.Linear(hidden_dim, hidden_dim), - nn.ReLU(), - nn.Linear(hidden_dim, 1) - ) - def forward(self, state): - value = self.critic(state) - return value -``` -这里Actor就是得到一个概率分布(Categorica，也可以是别的分布，可以搜索torch distributionsl)，critc根据当前状态得到一个值，这里的输入维度可以是```n_states+n_actions```，即将action信息也纳入critic网络中，这样会更好一些，感兴趣的小伙伴可以试试。 - -### PPO update -定义一个update函数主要实现伪代码中的第六步和第七步： -```python -def update(self): - for _ in range(self.n_epochs): - state_arr, action_arr, old_prob_arr, vals_arr,\ - reward_arr, dones_arr, batches = \ - self.memory.sample() - values = vals_arr - ### compute advantage ### - advantage = np.zeros(len(reward_arr), dtype=np.float32) - for t in range(len(reward_arr)-1): - discount = 1 - a_t = 0 - for k in range(t, len(reward_arr)-1): - a_t += discount*(reward_arr[k] + self.gamma*values[k+1]*\ - (1-int(dones_arr[k])) - values[k]) - discount *= self.gamma*self.gae_lambda - advantage[t] = a_t - advantage = torch.tensor(advantage).to(self.device) - ### SGD ### - values = torch.tensor(values).to(self.device) - for batch in batches: - states = torch.tensor(state_arr[batch], dtype=torch.float).to(self.device) - old_probs = torch.tensor(old_prob_arr[batch]).to(self.device) - actions = torch.tensor(action_arr[batch]).to(self.device) - dist = self.actor(states) - critic_value = self.critic(states) - critic_value = torch.squeeze(critic_value) - new_probs = dist.log_prob(actions) - prob_ratio = new_probs.exp() / old_probs.exp() - weighted_probs = advantage[batch] * prob_ratio - weighted_clipped_probs = torch.clamp(prob_ratio, 1-self.policy_clip, - 1+self.policy_clip)*advantage[batch] - actor_loss = -torch.min(weighted_probs, weighted_clipped_probs).mean() - returns = advantage[batch] + values[batch] - critic_loss = (returns-critic_value)**2 - critic_loss = critic_loss.mean() - total_loss = actor_loss + 0.5*critic_loss - self.actor_optimizer.zero_grad() - self.critic_optimizer.zero_grad() - total_loss.backward() - self.actor_optimizer.step() - self.critic_optimizer.step() - self.memory.clear() -``` -该部分首先从memory中提取搜集到的轨迹信息，然后计算gae，即advantage，接着使用随机梯度下降更新网络，最后清除memory以便搜集下一条轨迹信息。 - -最后实现效果如下： -![在这里插入图片描述](assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210405110725113.png) \ No newline at end of file diff --git a/projects/codes/PPO/assets/20210323154236878.png b/projects/codes/PPO/assets/20210323154236878.png deleted file mode 100644 index 0e8d796..0000000 Binary files a/projects/codes/PPO/assets/20210323154236878.png and /dev/null differ diff --git a/projects/codes/PPO/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210405110725113.png b/projects/codes/PPO/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210405110725113.png deleted file mode 100644 index e1b61f4..0000000 Binary files a/projects/codes/PPO/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210405110725113.png and /dev/null differ diff --git a/projects/codes/PPO/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70.png b/projects/codes/PPO/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70.png deleted file mode 100644 index 944c7a6..0000000 Binary files a/projects/codes/PPO/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70.png and /dev/null differ diff --git a/projects/codes/PPO/config/config.py b/projects/codes/PPO/config/config.py deleted file mode 100644 index b8f9870..0000000 --- a/projects/codes/PPO/config/config.py +++ /dev/null @@ -1,37 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-10-30 11:30:56 -LastEditor: JiangJi -LastEditTime: 2022-10-31 00:33:15 -Discription: default parameters of PPO -''' -from common.config import GeneralConfig,AlgoConfig - -class GeneralConfigPPO(GeneralConfig): - def __init__(self) -> None: - self.env_name = "CartPole-v0" - self.algo_name = "PPO" - self.seed = 1 - self.device = "cuda" - self.train_eps = 100 # number of episodes for training - self.test_eps = 10 # number of episodes for testing - self.max_steps = 200 # max steps for each episode - -class AlgoConfigPPO(AlgoConfig): - def __init__(self) -> None: - self.gamma = 0.99 # discount factor - self.continuous = False # continuous action space or not - self.policy_clip = 0.2 # clip range of policy - self.n_epochs = 10 # number of epochs - self.gae_lambda = 0.95 # gae lambda - self.actor_lr = 0.0003 # learning rate of actor - self.critic_lr = 0.0003 # learning rate of critic - self.actor_hidden_dim = 256 # - self.critic_hidden_dim = 256 - self.n_epochs = 4 # epochs - self.batch_size = 5 # - self.policy_clip = 0.2 - self.update_fre = 20 # frequency of updating agent diff --git a/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/models/ppo_actor.pt b/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/models/ppo_actor.pt deleted file mode 100644 index e7660b4..0000000 Binary files a/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/models/ppo_actor.pt and /dev/null differ diff --git a/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/models/ppo_critic.pt b/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/models/ppo_critic.pt deleted file mode 100644 index f0ec0d4..0000000 Binary files a/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/models/ppo_critic.pt and /dev/null differ diff --git a/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/results/params.json b/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/results/params.json deleted file mode 100644 index 15097c6..0000000 --- a/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/results/params.json +++ /dev/null @@ -1,25 +0,0 @@ -{ - "algo_name": "PPO", - "env_name": "CartPole-v0", - "continuous": false, - "train_eps": 200, - "test_eps": 20, - "gamma": 0.99, - "batch_size": 5, - "n_epochs": 4, - "actor_lr": 0.0003, - "critic_lr": 0.0003, - "gae_lambda": 0.95, - "policy_clip": 0.2, - "update_fre": 20, - "actor_hidden_dim": 256, - "critic_hidden_dim": 256, - "device": "cpu", - "seed": 10, - "show_fig": false, - "save_fig": true, - "result_path": "c:\\Users\\24438\\Desktop\\rl-tutorials\\codes\\PPO/outputs/CartPole-v0/20220920-213310/results/", - "model_path": "c:\\Users\\24438\\Desktop\\rl-tutorials\\codes\\PPO/outputs/CartPole-v0/20220920-213310/models/", - "n_states": 4, - "n_actions": 2 -} \ No newline at end of file diff --git a/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/results/testing_curve.png b/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/results/testing_curve.png deleted file mode 100644 index badf029..0000000 Binary files a/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/results/testing_curve.png and /dev/null differ diff --git a/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/results/testing_results.csv b/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/results/testing_results.csv deleted file mode 100644 index fb73fd6..0000000 --- a/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/results/testing_results.csv +++ /dev/null @@ -1,21 +0,0 @@ -episodes,rewards -0,200.0 -1,200.0 -2,200.0 -3,200.0 -4,200.0 -5,200.0 -6,200.0 -7,200.0 -8,200.0 -9,200.0 -10,200.0 -11,200.0 -12,200.0 -13,200.0 -14,200.0 -15,200.0 -16,200.0 -17,200.0 -18,200.0 -19,200.0 diff --git a/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/results/training_curve.png b/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/results/training_curve.png deleted file mode 100644 index 1bc6604..0000000 Binary files a/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/results/training_curve.png and /dev/null differ diff --git a/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/results/training_results.csv b/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/results/training_results.csv deleted file mode 100644 index 7836df5..0000000 --- a/projects/codes/PPO/outputs/CartPole-v0/20220920-213310/results/training_results.csv +++ /dev/null @@ -1,201 +0,0 @@ -episodes,rewards -0,34.0 -1,12.0 -2,47.0 -3,29.0 -4,20.0 -5,23.0 -6,33.0 -7,25.0 -8,11.0 -9,30.0 -10,18.0 -11,16.0 -12,15.0 -13,25.0 -14,33.0 -15,19.0 -16,50.0 -17,23.0 -18,21.0 -19,42.0 -20,60.0 -21,64.0 -22,30.0 -23,31.0 -24,90.0 -25,43.0 -26,54.0 -27,74.0 -28,30.0 -29,82.0 -30,50.0 -31,53.0 -32,25.0 -33,27.0 -34,145.0 -35,118.0 -36,141.0 -37,148.0 -38,200.0 -39,191.0 -40,71.0 -41,105.0 -42,100.0 -43,120.0 -44,80.0 -45,40.0 -46,104.0 -47,39.0 -48,89.0 -49,60.0 -50,30.0 -51,24.0 -52,20.0 -53,23.0 -54,30.0 -55,32.0 -56,20.0 -57,12.0 -58,25.0 -59,25.0 -60,24.0 -61,29.0 -62,200.0 -63,62.0 -64,200.0 -65,58.0 -66,81.0 -67,200.0 -68,52.0 -69,140.0 -70,200.0 -71,74.0 -72,200.0 -73,29.0 -74,124.0 -75,129.0 -76,200.0 -77,194.0 -78,175.0 -79,117.0 -80,200.0 -81,186.0 -82,114.0 -83,200.0 -84,166.0 -85,150.0 -86,135.0 -87,200.0 -88,200.0 -89,133.0 -90,111.0 -91,200.0 -92,90.0 -93,200.0 -94,147.0 -95,30.0 -96,137.0 -97,200.0 -98,200.0 -99,179.0 -100,167.0 -101,186.0 -102,169.0 -103,200.0 -104,200.0 -105,171.0 -106,200.0 -107,181.0 -108,125.0 -109,200.0 -110,200.0 -111,122.0 -112,200.0 -113,124.0 -114,95.0 -115,102.0 -116,118.0 -117,91.0 -118,64.0 -119,124.0 -120,122.0 -121,76.0 -122,68.0 -123,40.0 -124,52.0 -125,51.0 -126,50.0 -127,49.0 -128,37.0 -129,76.0 -130,83.0 -131,76.0 -132,92.0 -133,113.0 -134,94.0 -135,157.0 -136,92.0 -137,200.0 -138,123.0 -139,200.0 -140,200.0 -141,200.0 -142,140.0 -143,200.0 -144,200.0 -145,200.0 -146,200.0 -147,200.0 -148,200.0 -149,200.0 -150,200.0 -151,78.0 -152,200.0 -153,200.0 -154,200.0 -155,200.0 -156,200.0 -157,200.0 -158,200.0 -159,200.0 -160,200.0 -161,200.0 -162,107.0 -163,187.0 -164,200.0 -165,200.0 -166,200.0 -167,200.0 -168,200.0 -169,200.0 -170,200.0 -171,200.0 -172,200.0 -173,200.0 -174,200.0 -175,200.0 -176,200.0 -177,200.0 -178,200.0 -179,200.0 -180,200.0 -181,200.0 -182,200.0 -183,200.0 -184,200.0 -185,200.0 -186,200.0 -187,200.0 -188,200.0 -189,200.0 -190,200.0 -191,200.0 -192,200.0 -193,200.0 -194,200.0 -195,200.0 -196,200.0 -197,200.0 -198,200.0 -199,200.0 diff --git a/projects/codes/PPO/ppo2.py b/projects/codes/PPO/ppo2.py deleted file mode 100644 index 5d399b8..0000000 --- a/projects/codes/PPO/ppo2.py +++ /dev/null @@ -1,119 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-09-26 16:11:36 -LastEditor: JiangJi -LastEditTime: 2022-10-31 00:36:37 -Discription: PPO-clip -''' - -import os -import numpy as np -import torch -import torch.optim as optim -from torch.distributions.categorical import Categorical - - -class PPO: - def __init__(self, models,memory,cfg): - self.gamma = cfg.gamma - self.continuous = cfg.continuous - self.policy_clip = cfg.policy_clip - self.n_epochs = cfg.n_epochs - self.batch_size = cfg.batch_size - self.gae_lambda = cfg.gae_lambda - self.device = torch.device(cfg.device) - self.actor = models['Actor'].to(self.device) - self.critic = models['Critic'].to(self.device) - self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=cfg.actor_lr) - self.critic_optimizer = optim.Adam(self.critic.parameters(), lr=cfg.critic_lr) - self.memory = memory - self.loss = 0 - - def sample_action(self, state): - state = np.array([state]) # 先转成数组再转tensor更高效 - state = torch.tensor(state, dtype=torch.float).to(self.device) - probs = self.actor(state) - dist = Categorical(probs) - value = self.critic(state) - action = dist.sample() - probs = torch.squeeze(dist.log_prob(action)).item() - if self.continuous: - action = torch.tanh(action) - else: - action = torch.squeeze(action).item() - value = torch.squeeze(value).item() - return action, probs, value - @torch.no_grad() - def predict_action(self, state): - state = np.array([state]) # 先转成数组再转tensor更高效 - state = torch.tensor(state, dtype=torch.float).to(self.device) - dist = self.actor(state) - value = self.critic(state) - action = dist.sample() - probs = torch.squeeze(dist.log_prob(action)).item() - if self.continuous: - action = torch.tanh(action) - else: - action = torch.squeeze(action).item() - value = torch.squeeze(value).item() - return action, probs, value - - def update(self): - for _ in range(self.n_epochs): - state_arr, action_arr, old_prob_arr, vals_arr,reward_arr, dones_arr, batches = self.memory.sample() - values = vals_arr[:] - ### compute advantage ### - advantage = np.zeros(len(reward_arr), dtype=np.float32) - for t in range(len(reward_arr)-1): - discount = 1 - a_t = 0 - for k in range(t, len(reward_arr)-1): - a_t += discount*(reward_arr[k] + self.gamma*values[k+1]*\ - (1-int(dones_arr[k])) - values[k]) - discount *= self.gamma*self.gae_lambda - advantage[t] = a_t - advantage = torch.tensor(advantage).to(self.device) - ### SGD ### - values = torch.tensor(values).to(self.device) - for batch in batches: - states = torch.tensor(state_arr[batch], dtype=torch.float).to(self.device) - old_probs = torch.tensor(old_prob_arr[batch]).to(self.device) - actions = torch.tensor(action_arr[batch]).to(self.device) - dist = self.actor(states) - critic_value = self.critic(states) - critic_value = torch.squeeze(critic_value) - new_probs = dist.log_prob(actions) - prob_ratio = new_probs.exp() / old_probs.exp() - weighted_probs = advantage[batch] * prob_ratio - weighted_clipped_probs = torch.clamp(prob_ratio, 1-self.policy_clip, - 1+self.policy_clip)*advantage[batch] - actor_loss = -torch.min(weighted_probs, weighted_clipped_probs).mean() - returns = advantage[batch] + values[batch] - critic_loss = (returns-critic_value)**2 - critic_loss = critic_loss.mean() - total_loss = actor_loss + 0.5*critic_loss - self.loss = total_loss - self.actor_optimizer.zero_grad() - self.critic_optimizer.zero_grad() - total_loss.backward() - self.actor_optimizer.step() - self.critic_optimizer.step() - self.memory.clear() - def save_model(self,path): - from pathlib import Path - # create path - Path(path).mkdir(parents=True, exist_ok=True) - actor_checkpoint = os.path.join(path, 'ppo_actor.pt') - critic_checkpoint= os.path.join(path, 'ppo_critic.pt') - torch.save(self.actor.state_dict(), actor_checkpoint) - torch.save(self.critic.state_dict(), critic_checkpoint) - def load_model(self,path): - actor_checkpoint = os.path.join(path, 'ppo_actor.pt') - critic_checkpoint= os.path.join(path, 'ppo_critic.pt') - self.actor.load_state_dict(torch.load(actor_checkpoint)) - self.critic.load_state_dict(torch.load(critic_checkpoint)) - - diff --git a/projects/codes/PPO/task0.py b/projects/codes/PPO/task0.py deleted file mode 100644 index dbf0e7a..0000000 --- a/projects/codes/PPO/task0.py +++ /dev/null @@ -1,159 +0,0 @@ -import sys,os -os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE" # avoid "OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized." -curr_path = os.path.dirname(os.path.abspath(__file__)) # current path -parent_path = os.path.dirname(curr_path) # parent path -sys.path.append(parent_path) # add path to system path - -import gym -import torch -import datetime -import numpy as np -import argparse -import torch.nn as nn - - -from common.utils import all_seed,merge_class_attrs -from common.models import ActorSoftmax, Critic -from common.memories import PGReplay -from common.launcher import Launcher -from envs.register import register_env -from ppo2 import PPO -from config,config import GeneralConfigPPO,AlgoConfigPPO -class PPOMemory: - def __init__(self, batch_size): - self.states = [] - self.probs = [] - self.vals = [] - self.actions = [] - self.rewards = [] - self.terminateds = [] - self.batch_size = batch_size - def sample(self): - batch_step = np.arange(0, len(self.states), self.batch_size) - indices = np.arange(len(self.states), dtype=np.int64) - np.random.shuffle(indices) - batches = [indices[i:i+self.batch_size] for i in batch_step] - return np.array(self.states),np.array(self.actions),np.array(self.probs),\ - np.array(self.vals),np.array(self.rewards),np.array(self.terminateds),batches - - def push(self, state, action, probs, vals, reward, terminated): - self.states.append(state) - self.actions.append(action) - self.probs.append(probs) - self.vals.append(vals) - self.rewards.append(reward) - self.terminateds.append(terminated) - - def clear(self): - self.states = [] - self.probs = [] - self.actions = [] - self.rewards = [] - self.terminateds = [] - self.vals = [] - - -class Main(Launcher): - def __init__(self) -> None: - super().__init__() - self.cfgs['general_cfg'] = merge_class_attrs(self.cfgs['general_cfg'],GeneralConfigPPO()) - self.cfgs['algo_cfg'] = merge_class_attrs(self.cfgs['algo_cfg'],AlgoConfigPPO()) - def env_agent_config(self,cfg,logger): - ''' create env and agent - ''' - register_env(cfg.env_name) - env = gym.make(cfg.env_name,new_step_api=False) # create env - if cfg.seed !=0: # set random seed - all_seed(env,seed=cfg.seed) - try: # state dimension - n_states = env.observation_space.n # print(hasattr(env.observation_space, 'n')) - except AttributeError: - n_states = env.observation_space.shape[0] # print(hasattr(env.observation_space, 'shape')) - n_actions = env.action_space.n # action dimension - logger.info(f"n_states: {n_states}, n_actions: {n_actions}") # print info - # update to cfg paramters - setattr(cfg, 'n_states', n_states) - setattr(cfg, 'n_actions', n_actions) - models = {'Actor':ActorSoftmax(n_states,n_actions, hidden_dim = cfg.actor_hidden_dim),'Critic':Critic(n_states,1,hidden_dim=cfg.critic_hidden_dim)} - memory = PGReplay # replay buffer - agent = PPO(models,memory,cfg) # create agent - return env, agent - def train_one_episode(self, env, agent, cfg): - ep_reward = 0 # reward per episode - ep_step = 0 # step per episode - state = env.reset() - for _ in range(cfg.max_steps): - action, prob, val = agent.sample_action(state) - next_state, reward, terminated, _ = env.step(action) - ep_reward += reward - ep_step += 1 - agent.memory.push((state, action, prob, val, reward, terminated)) - if ep_step % cfg['update_fre'] == 0: - agent.update() - state = next_state - if terminated: - break - return agent, ep_reward, ep_step - def test_one_episode(self, env, agent, cfg): - ep_reward = 0 # reward per episode - ep_step = 0 # step per episode - state = env.reset() - for _ in range(cfg.max_steps): - action, prob, val = agent.sample_action(state) - next_state, reward, terminated, _ = env.step(action) - ep_reward += reward - ep_step += 1 - state = next_state - if terminated: - break - return agent, ep_reward, ep_step - def train(self,cfg,env,agent): - ''' train agent - ''' - print("Start training!") - print(f"Env: {cfg['env_name']}, Algorithm: {cfg['algo_name']}, Device: {cfg['device']}") - rewards = [] # record rewards for all episodes - steps = 0 - for i_ep in range(cfg['train_eps']): - state = env.reset() - ep_reward = 0 - while True: - action, prob, val = agent.sample_action(state) - next_state, reward, terminated, _ = env.step(action) - steps += 1 - ep_reward += reward - agent.memory.push(state, action, prob, val, reward, terminated) - if steps % cfg['update_fre'] == 0: - agent.update() - state = next_state - if terminated: - break - rewards.append(ep_reward) - if (i_ep+1)%10==0: - print(f"Episode: {i_ep+1}/{cfg['train_eps']}, Reward: {ep_reward:.2f}") - print("Finish training!") - return {'episodes':range(len(rewards)),'rewards':rewards} - def test(self,cfg,env,agent): - ''' test agent - ''' - print("Start testing!") - print(f"Env: {cfg['env_name']}, Algorithm: {cfg['algo_name']}, Device: {cfg['device']}") - rewards = [] # record rewards for all episodes - for i_ep in range(cfg['test_eps']): - state = env.reset() - ep_reward = 0 - while True: - action, prob, val = agent.predict_action(state) - next_state, reward, terminated, _ = env.step(action) - ep_reward += reward - state = next_state - if terminated: - break - rewards.append(ep_reward) - print(f"Episode: {i_ep+1}/{cfg['test_eps']}, Reward: {ep_reward:.2f}") - print("Finish testing!") - return {'episodes':range(len(rewards)),'rewards':rewards} - -if __name__ == "__main__": - main = Main() - main.run() \ No newline at end of file diff --git a/projects/codes/PPO/task1.py b/projects/codes/PPO/task1.py deleted file mode 100644 index d664770..0000000 --- a/projects/codes/PPO/task1.py +++ /dev/null @@ -1,77 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-09-19 14:48:16 -LastEditor: JiangJi -LastEditTime: 2022-10-30 00:45:14 -Discription: -''' -import sys,os -curr_path = os.path.dirname(os.path.abspath(__file__)) # 当前文件所在绝对路径 -parent_path = os.path.dirname(curr_path) # 父路径 -sys.path.append(parent_path) # 添加路径到系统路径 - -import gym -import torch -import datetime -from common.utils import plot_rewards -from common.utils import save_results,make_dir -from ppo2 import PPO - -curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # 获取当前时间 - -class PPOConfig: - def __init__(self) -> None: - self.algo = "PPO" # 算法名称 - self.env_name = 'Pendulum-v1' # 环境名称 - self.continuous = True # 环境是否为连续动作 - self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # 检测GPU - self.train_eps = 200 # 训练的回合数 - self.test_eps = 20 # 测试的回合数 - self.batch_size = 5 - self.gamma=0.99 - self.n_epochs = 4 - self.actor_lr = 0.0003 - self.critic_lr = 0.0003 - self.gae_lambda=0.95 - self.policy_clip=0.2 - self.hidden_dim = 256 - self.update_fre = 20 # frequency of agent update - -class PlotConfig: - def __init__(self) -> None: - self.algo = "PPO" # 算法名称 - self.env_name = 'Pendulum-v1' # 环境名称 - self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # 检测GPU - self.result_path = curr_path+"/outputs/" + self.env_name + \ - '/'+curr_time+'/results/' # 保存结果的路径 - self.model_path = curr_path+"/outputs/" + self.env_name + \ - '/'+curr_time+'/models/' # 保存模型的路径 - self.save = True # 是否保存图片 - -def env_agent_config(cfg,seed=1): - env = gym.make(cfg.env_name) - env.seed(seed) - n_states = env.observation_space.shape[0] - n_actions = env.action_space.shape[0] - agent = PPO(n_states,n_actions,cfg) - return env,agent - - -cfg = PPOConfig() -plot_cfg = PlotConfig() -# 训练 -env,agent = env_agent_config(cfg,seed=1) -rewards, ma_rewards = train(cfg, env, agent) -make_dir(plot_cfg.result_path, plot_cfg.model_path) # 创建保存结果和模型路径的文件夹 -agent.save(path=plot_cfg.model_path) -save_results(rewards, ma_rewards, tag='train', path=plot_cfg.result_path) -plot_rewards(rewards, ma_rewards, plot_cfg, tag="train") -# 测试 -env,agent = env_agent_config(cfg,seed=10) -agent.load(path=plot_cfg.model_path) -rewards,ma_rewards = eval(cfg,env,agent) -save_results(rewards,ma_rewards,tag='eval',path=plot_cfg.result_path) -plot_rewards(rewards,ma_rewards,plot_cfg,tag="eval") \ No newline at end of file diff --git a/projects/codes/PolicyGradient/README.md b/projects/codes/PolicyGradient/README.md deleted file mode 100644 index 956cdbf..0000000 --- a/projects/codes/PolicyGradient/README.md +++ /dev/null @@ -1,27 +0,0 @@ -# Policy Gradient - - -Policy-based方法是强化学习中与Value-based(比如Q-learning)相对的方法，其目的是对策略本身进行梯度下降，相关基础知识参考[Datawhale-Policy Gradient](https://datawhalechina.github.io/leedeeprl-notes/#/chapter4/chapter4)。 -其中REINFORCE是一个最基本的Policy Gradient方法，主要解决策略梯度无法直接计算的问题，具体原理参考[CSDN-REINFORCE和Reparameterization Trick](https://blog.csdn.net/JohnJim0/article/details/110230703) - -## 伪代码 - -结合REINFORCE原理，其伪代码如下： - -image-20211016004808604 - -https://pytorch.org/docs/stable/distributions.html - -加负号的原因是，在公式中应该是实现的梯度上升算法，而loss一般使用随机梯度下降的，所以加个负号保持一致性。 - -![img](assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210428001336032.png) - -## 实现 - -## 参考 - -[REINFORCE和Reparameterization Trick](https://blog.csdn.net/JohnJim0/article/details/110230703) - -[Policy Gradient paper](https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf) - -[REINFORCE](https://towardsdatascience.com/policy-gradient-methods-104c783251e0) \ No newline at end of file diff --git a/projects/codes/PolicyGradient/assets/image-20211016004808604.png b/projects/codes/PolicyGradient/assets/image-20211016004808604.png deleted file mode 100644 index b0a56b5..0000000 Binary files a/projects/codes/PolicyGradient/assets/image-20211016004808604.png and /dev/null differ diff --git a/projects/codes/PolicyGradient/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210428001336032.png b/projects/codes/PolicyGradient/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210428001336032.png deleted file mode 100644 index 44c1874..0000000 Binary files a/projects/codes/PolicyGradient/assets/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0pvaG5KaW0w,size_16,color_FFFFFF,t_70-20210428001336032.png and /dev/null differ diff --git a/projects/codes/PolicyGradient/main.py b/projects/codes/PolicyGradient/main.py deleted file mode 100644 index 3473c38..0000000 --- a/projects/codes/PolicyGradient/main.py +++ /dev/null @@ -1,131 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: John -Email: johnjim0816@gmail.com -Date: 2020-11-22 23:21:53 -LastEditor: John -LastEditTime: 2022-08-27 00:04:08 -Discription: -Environment: -''' -import sys,os -curr_path = os.path.dirname(os.path.abspath(__file__)) # current path -parent_path = os.path.dirname(curr_path) # parent path -sys.path.append(parent_path) # add to system path - -import gym -import torch -import datetime -import argparse -from itertools import count -import torch.nn.functional as F -from pg import PolicyGradient -from common.utils import save_results, make_dir,all_seed,save_args,plot_rewards -from common.models import MLP -from common.memories import PGReplay -from common.launcher import Launcher -from envs.register import register_env - - -class PGNet(MLP): - ''' instead of outputing action, PG Net outputs propabilities of actions, we can use class inheritance from MLP here - ''' - def forward(self, x): - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = torch.sigmoid(self.fc3(x)) - return x - -class Main(Launcher): - def get_args(self): - """ Hyperparameters - """ - curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # Obtain current time - parser = argparse.ArgumentParser(description="hyperparameters") - parser.add_argument('--algo_name',default='PolicyGradient',type=str,help="name of algorithm") - parser.add_argument('--env_name',default='CartPole-v0',type=str,help="name of environment") - parser.add_argument('--train_eps',default=200,type=int,help="episodes of training") - parser.add_argument('--test_eps',default=20,type=int,help="episodes of testing") - parser.add_argument('--ep_max_steps',default = 100000,type=int,help="steps per episode, much larger value can simulate infinite steps") - parser.add_argument('--gamma',default=0.99,type=float,help="discounted factor") - parser.add_argument('--lr',default=0.01,type=float,help="learning rate") - parser.add_argument('--update_fre',default=8,type=int) - parser.add_argument('--hidden_dim',default=36,type=int) - parser.add_argument('--device',default='cpu',type=str,help="cpu or cuda") - parser.add_argument('--seed',default=1,type=int,help="seed") - parser.add_argument('--save_fig',default=True,type=bool,help="if save figure or not") - parser.add_argument('--show_fig',default=False,type=bool,help="if show figure or not") - args = parser.parse_args() - default_args = {'result_path':f"{curr_path}/outputs/{args.env_name}/{curr_time}/results/", - 'model_path':f"{curr_path}/outputs/{args.env_name}/{curr_time}/models/", - } - args = {**vars(args),**default_args} # type(dict) - return args - def env_agent_config(self,cfg): - register_env(cfg['env_name']) - env = gym.make(cfg['env_name']) - if cfg['seed'] !=0: # set random seed - all_seed(env,seed=cfg['seed']) - n_states = env.observation_space.shape[0] - n_actions = env.action_space.n # action dimension - print(f"state dim: {n_states}, action dim: {n_actions}") - cfg.update({"n_states":n_states,"n_actions":n_actions}) # update to cfg paramters - model = PGNet(n_states,1,hidden_dim=cfg['hidden_dim']) - memory = PGReplay() - agent = PolicyGradient(model,memory,cfg) - return env,agent - def train(self,cfg,env,agent): - print("Start training!") - print(f"Env: {cfg['env_name']}, Algorithm: {cfg['algo_name']}, Device: {cfg['device']}") - rewards = [] - for i_ep in range(cfg['train_eps']): - state = env.reset() - ep_reward = 0 - for _ in range(cfg['ep_max_steps']): - action = agent.sample_action(state) # sample action - next_state, reward, done, _ = env.step(action) - ep_reward += reward - if done: - reward = 0 - agent.memory.push((state,float(action),reward)) - state = next_state - if done: - break - if (i_ep+1) % 10 == 0: - print(f"Episode：{i_ep+1}/{cfg['train_eps']}, Reward:{ep_reward:.2f}") - if (i_ep+1) % cfg['update_fre'] == 0: - agent.update() - rewards.append(ep_reward) - print('Finish training!') - env.close() # close environment - res_dic = {'episodes':range(len(rewards)),'rewards':rewards} - return res_dic - - def test(self,cfg,env,agent): - print("Start testing!") - print(f"Env: {cfg['env_name']}, Algorithm: {cfg['algo_name']}, Device: {cfg['device']}") - rewards = [] - for i_ep in range(cfg['test_eps']): - state = env.reset() - ep_reward = 0 - for _ in range(cfg['ep_max_steps']): - action = agent.predict_action(state) - next_state, reward, done, _ = env.step(action) - ep_reward += reward - if done: - reward = 0 - state = next_state - if done: - break - print(f"Episode: {i_ep+1}/{cfg['test_eps']}，Reward: {ep_reward:.2f}") - rewards.append(ep_reward) - print("Finish testing!") - env.close() - return {'episodes':range(len(rewards)),'rewards':rewards} - -if __name__ == "__main__": - main = Main() - main.run() - - diff --git a/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/models/checkpoint.pt b/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/models/checkpoint.pt deleted file mode 100644 index 7b98cda..0000000 Binary files a/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/models/checkpoint.pt and /dev/null differ diff --git a/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/results/params.json b/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/results/params.json deleted file mode 100644 index 4dfae79..0000000 --- a/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/results/params.json +++ /dev/null @@ -1 +0,0 @@ -{"algo_name": "PolicyGradient", "env_name": "CartPole-v0", "train_eps": 200, "test_eps": 20, "ep_max_steps": 100000, "gamma": 0.99, "lr": 0.01, "update_fre": 8, "hidden_dim": 36, "device": "cpu", "seed": 1, "save_fig": true, "show_fig": false, "result_path": "c:\\Users\\24438\\Desktop\\rl-tutorials\\codes\\PolicyGradient/outputs/CartPole-v0/20220827-000433/results/", "model_path": "c:\\Users\\24438\\Desktop\\rl-tutorials\\codes\\PolicyGradient/outputs/CartPole-v0/20220827-000433/models/", "n_states": 4, "n_actions": 2} \ No newline at end of file diff --git a/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/results/testing_curve.png b/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/results/testing_curve.png deleted file mode 100644 index e3c3489..0000000 Binary files a/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/results/testing_curve.png and /dev/null differ diff --git a/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/results/testing_results.csv b/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/results/testing_results.csv deleted file mode 100644 index fb73fd6..0000000 --- a/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/results/testing_results.csv +++ /dev/null @@ -1,21 +0,0 @@ -episodes,rewards -0,200.0 -1,200.0 -2,200.0 -3,200.0 -4,200.0 -5,200.0 -6,200.0 -7,200.0 -8,200.0 -9,200.0 -10,200.0 -11,200.0 -12,200.0 -13,200.0 -14,200.0 -15,200.0 -16,200.0 -17,200.0 -18,200.0 -19,200.0 diff --git a/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/results/training_curve.png b/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/results/training_curve.png deleted file mode 100644 index 1f954a1..0000000 Binary files a/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/results/training_curve.png and /dev/null differ diff --git a/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/results/training_results.csv b/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/results/training_results.csv deleted file mode 100644 index 715be6d..0000000 --- a/projects/codes/PolicyGradient/outputs/CartPole-v0/20220827-000433/results/training_results.csv +++ /dev/null @@ -1,201 +0,0 @@ -episodes,rewards -0,26.0 -1,53.0 -2,10.0 -3,37.0 -4,22.0 -5,21.0 -6,12.0 -7,34.0 -8,93.0 -9,36.0 -10,29.0 -11,18.0 -12,14.0 -13,62.0 -14,20.0 -15,40.0 -16,10.0 -17,10.0 -18,10.0 -19,11.0 -20,10.0 -21,14.0 -22,12.0 -23,8.0 -24,19.0 -25,33.0 -26,22.0 -27,32.0 -28,16.0 -29,24.0 -30,24.0 -31,24.0 -32,75.0 -33,33.0 -34,33.0 -35,72.0 -36,110.0 -37,48.0 -38,60.0 -39,43.0 -40,61.0 -41,34.0 -42,50.0 -43,61.0 -44,53.0 -45,58.0 -46,36.0 -47,44.0 -48,42.0 -49,64.0 -50,67.0 -51,52.0 -52,39.0 -53,42.0 -54,40.0 -55,33.0 -56,200.0 -57,199.0 -58,149.0 -59,185.0 -60,134.0 -61,174.0 -62,162.0 -63,200.0 -64,93.0 -65,72.0 -66,69.0 -67,51.0 -68,62.0 -69,98.0 -70,73.0 -71,73.0 -72,200.0 -73,200.0 -74,200.0 -75,200.0 -76,200.0 -77,200.0 -78,200.0 -79,133.0 -80,200.0 -81,200.0 -82,200.0 -83,200.0 -84,200.0 -85,200.0 -86,200.0 -87,200.0 -88,114.0 -89,151.0 -90,129.0 -91,156.0 -92,112.0 -93,172.0 -94,171.0 -95,141.0 -96,200.0 -97,200.0 -98,200.0 -99,200.0 -100,200.0 -101,200.0 -102,200.0 -103,200.0 -104,188.0 -105,199.0 -106,138.0 -107,200.0 -108,200.0 -109,181.0 -110,145.0 -111,200.0 -112,135.0 -113,119.0 -114,112.0 -115,122.0 -116,118.0 -117,119.0 -118,131.0 -119,119.0 -120,109.0 -121,96.0 -122,105.0 -123,29.0 -124,110.0 -125,113.0 -126,18.0 -127,90.0 -128,145.0 -129,152.0 -130,151.0 -131,109.0 -132,141.0 -133,109.0 -134,136.0 -135,143.0 -136,200.0 -137,200.0 -138,200.0 -139,200.0 -140,200.0 -141,200.0 -142,200.0 -143,200.0 -144,192.0 -145,173.0 -146,180.0 -147,182.0 -148,186.0 -149,175.0 -150,176.0 -151,191.0 -152,200.0 -153,200.0 -154,200.0 -155,200.0 -156,200.0 -157,200.0 -158,200.0 -159,200.0 -160,200.0 -161,200.0 -162,200.0 -163,200.0 -164,200.0 -165,200.0 -166,200.0 -167,200.0 -168,200.0 -169,200.0 -170,200.0 -171,200.0 -172,200.0 -173,200.0 -174,200.0 -175,200.0 -176,200.0 -177,200.0 -178,200.0 -179,200.0 -180,200.0 -181,200.0 -182,200.0 -183,200.0 -184,200.0 -185,200.0 -186,200.0 -187,200.0 -188,200.0 -189,200.0 -190,200.0 -191,200.0 -192,200.0 -193,200.0 -194,200.0 -195,200.0 -196,200.0 -197,200.0 -198,200.0 -199,200.0 diff --git a/projects/codes/PolicyGradient/pg.py b/projects/codes/PolicyGradient/pg.py deleted file mode 100644 index 7d84c6e..0000000 --- a/projects/codes/PolicyGradient/pg.py +++ /dev/null @@ -1,88 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: John -Email: johnjim0816@gmail.com -Date: 2020-11-22 23:27:44 -LastEditor: John -LastEditTime: 2022-10-09 21:28:18 -Discription: -Environment: -''' -import torch -import torch.nn as nn -import torch.nn.functional as F -from torch.distributions import Bernoulli -from torch.autograd import Variable -import numpy as np - - -class PolicyGradient: - - def __init__(self, model,memory,cfg): - self.gamma = cfg['gamma'] - self.device = torch.device(cfg['device']) - self.memory = memory - self.policy_net = model.to(self.device) - self.optimizer = torch.optim.RMSprop(self.policy_net.parameters(), lr=cfg['lr']) - - def sample_action(self,state): - - state = torch.from_numpy(state).float() - state = Variable(state) - probs = self.policy_net(state) - m = Bernoulli(probs) # 伯努利分布 - action = m.sample() - - action = action.data.numpy().astype(int)[0] # 转为标量 - return action - def predict_action(self,state): - - state = torch.from_numpy(state).float() - state = Variable(state) - probs = self.policy_net(state) - m = Bernoulli(probs) # 伯努利分布 - action = m.sample() - action = action.data.numpy().astype(int)[0] # 转为标量 - return action - - def update(self): - state_pool,action_pool,reward_pool= self.memory.sample() - state_pool,action_pool,reward_pool = list(state_pool),list(action_pool),list(reward_pool) - # Discount reward - running_add = 0 - for i in reversed(range(len(reward_pool))): - if reward_pool[i] == 0: - running_add = 0 - else: - running_add = running_add * self.gamma + reward_pool[i] - reward_pool[i] = running_add - - # Normalize reward - reward_mean = np.mean(reward_pool) - reward_std = np.std(reward_pool) - for i in range(len(reward_pool)): - reward_pool[i] = (reward_pool[i] - reward_mean) / reward_std - - # Gradient Desent - self.optimizer.zero_grad() - - for i in range(len(reward_pool)): - state = state_pool[i] - action = Variable(torch.FloatTensor([action_pool[i]])) - reward = reward_pool[i] - state = Variable(torch.from_numpy(state).float()) - probs = self.policy_net(state) - m = Bernoulli(probs) - loss = -m.log_prob(action) * reward # Negtive score function x reward - # print(loss) - loss.backward() - self.optimizer.step() - self.memory.clear() - def save_model(self,path): - from pathlib import Path - # create path - Path(path).mkdir(parents=True, exist_ok=True) - torch.save(self.policy_net.state_dict(), path+'checkpoint.pt') - def load_model(self,path): - self.policy_net.load_state_dict(torch.load(path+'checkpoint.pt')) \ No newline at end of file diff --git a/projects/codes/QLearning/Test_CliffWalking-v0_QLearning_20221030-014151/config.yaml b/projects/codes/QLearning/Test_CliffWalking-v0_QLearning_20221030-014151/config.yaml deleted file mode 100644 index d9b4258..0000000 --- a/projects/codes/QLearning/Test_CliffWalking-v0_QLearning_20221030-014151/config.yaml +++ /dev/null @@ -1,21 +0,0 @@ -general_cfg: - algo_name: QLearning - device: cpu - env_name: CliffWalking-v0 - load_checkpoint: true - load_path: Train_CliffWalking-v0_QLearning_20221030-013856 - max_steps: 200 - mode: test - save_fig: true - seed: 1 - show_fig: false - test_eps: 20 - train_eps: 400 -algo_cfg: - batch_size: 64 - buffer_size: 100000 - epsilon_decay: 300 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - lr: 0.1 diff --git a/projects/codes/QLearning/Test_CliffWalking-v0_QLearning_20221030-014151/logs/log.txt b/projects/codes/QLearning/Test_CliffWalking-v0_QLearning_20221030-014151/logs/log.txt deleted file mode 100644 index d89037e..0000000 --- a/projects/codes/QLearning/Test_CliffWalking-v0_QLearning_20221030-014151/logs/log.txt +++ /dev/null @@ -1,24 +0,0 @@ -2022-10-30 01:41:51 - r - INFO: - n_states: 48, n_actions: 4 -2022-10-30 01:41:51 - r - INFO: - Start testing! -2022-10-30 01:41:51 - r - INFO: - Env: CliffWalking-v0, Algorithm: QLearning, Device: cpu -2022-10-30 01:41:51 - r - INFO: - Episode: 1/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 2/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 3/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 4/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 5/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 6/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 7/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 8/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 9/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 10/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 11/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 12/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 13/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 14/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 15/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 16/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 17/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 18/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 19/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Episode: 20/20, Steps:13 Reward: -13.00 -2022-10-30 01:41:51 - r - INFO: - Finish testing! diff --git a/projects/codes/QLearning/Test_CliffWalking-v0_QLearning_20221030-014151/models/Qleaning_model.pkl b/projects/codes/QLearning/Test_CliffWalking-v0_QLearning_20221030-014151/models/Qleaning_model.pkl deleted file mode 100644 index 2022d46..0000000 Binary files a/projects/codes/QLearning/Test_CliffWalking-v0_QLearning_20221030-014151/models/Qleaning_model.pkl and /dev/null differ diff --git a/projects/codes/QLearning/Test_CliffWalking-v0_QLearning_20221030-014151/results/learning_curve.png b/projects/codes/QLearning/Test_CliffWalking-v0_QLearning_20221030-014151/results/learning_curve.png deleted file mode 100644 index 49a7daa..0000000 Binary files a/projects/codes/QLearning/Test_CliffWalking-v0_QLearning_20221030-014151/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/QLearning/Test_CliffWalking-v0_QLearning_20221030-014151/results/res.csv b/projects/codes/QLearning/Test_CliffWalking-v0_QLearning_20221030-014151/results/res.csv deleted file mode 100644 index c48c7ef..0000000 --- a/projects/codes/QLearning/Test_CliffWalking-v0_QLearning_20221030-014151/results/res.csv +++ /dev/null @@ -1,21 +0,0 @@ -episodes,rewards,steps -0,-13,13 -1,-13,13 -2,-13,13 -3,-13,13 -4,-13,13 -5,-13,13 -6,-13,13 -7,-13,13 -8,-13,13 -9,-13,13 -10,-13,13 -11,-13,13 -12,-13,13 -13,-13,13 -14,-13,13 -15,-13,13 -16,-13,13 -17,-13,13 -18,-13,13 -19,-13,13 diff --git a/projects/codes/QLearning/Test_FrozenLakeNoSlippery-v1_QLearning_20221030-014552/config.yaml b/projects/codes/QLearning/Test_FrozenLakeNoSlippery-v1_QLearning_20221030-014552/config.yaml deleted file mode 100644 index 537c003..0000000 --- a/projects/codes/QLearning/Test_FrozenLakeNoSlippery-v1_QLearning_20221030-014552/config.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: QLearning - device: cpu - env_name: FrozenLakeNoSlippery-v1 - load_checkpoint: true - load_path: Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504 - max_steps: 200 - mode: test - save_fig: true - seed: 10 - show_fig: false - test_eps: 20 - train_eps: 800 -algo_cfg: - epsilon_decay: 2000 - epsilon_end: 0.1 - epsilon_start: 0.7 - gamma: 0.95 - lr: 0.9 diff --git a/projects/codes/QLearning/Test_FrozenLakeNoSlippery-v1_QLearning_20221030-014552/logs/log.txt b/projects/codes/QLearning/Test_FrozenLakeNoSlippery-v1_QLearning_20221030-014552/logs/log.txt deleted file mode 100644 index d972a0c..0000000 --- a/projects/codes/QLearning/Test_FrozenLakeNoSlippery-v1_QLearning_20221030-014552/logs/log.txt +++ /dev/null @@ -1,24 +0,0 @@ -2022-10-30 01:45:52 - r - INFO: - n_states: 16, n_actions: 4 -2022-10-30 01:45:52 - r - INFO: - Start testing! -2022-10-30 01:45:52 - r - INFO: - Env: FrozenLakeNoSlippery-v1, Algorithm: QLearning, Device: cpu -2022-10-30 01:45:52 - r - INFO: - Episode: 1/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 2/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 3/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 4/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 5/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 6/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 7/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 8/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 9/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 10/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 11/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 12/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 13/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 14/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 15/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 16/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 17/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 18/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 19/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Episode: 20/20, Steps:6 Reward: 1.00 -2022-10-30 01:45:52 - r - INFO: - Finish testing! diff --git a/projects/codes/QLearning/Test_FrozenLakeNoSlippery-v1_QLearning_20221030-014552/models/Qleaning_model.pkl b/projects/codes/QLearning/Test_FrozenLakeNoSlippery-v1_QLearning_20221030-014552/models/Qleaning_model.pkl deleted file mode 100644 index 41a5a05..0000000 Binary files a/projects/codes/QLearning/Test_FrozenLakeNoSlippery-v1_QLearning_20221030-014552/models/Qleaning_model.pkl and /dev/null differ diff --git a/projects/codes/QLearning/Test_FrozenLakeNoSlippery-v1_QLearning_20221030-014552/results/learning_curve.png b/projects/codes/QLearning/Test_FrozenLakeNoSlippery-v1_QLearning_20221030-014552/results/learning_curve.png deleted file mode 100644 index 60eeac6..0000000 Binary files a/projects/codes/QLearning/Test_FrozenLakeNoSlippery-v1_QLearning_20221030-014552/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/QLearning/Test_FrozenLakeNoSlippery-v1_QLearning_20221030-014552/results/res.csv b/projects/codes/QLearning/Test_FrozenLakeNoSlippery-v1_QLearning_20221030-014552/results/res.csv deleted file mode 100644 index b871b84..0000000 --- a/projects/codes/QLearning/Test_FrozenLakeNoSlippery-v1_QLearning_20221030-014552/results/res.csv +++ /dev/null @@ -1,21 +0,0 @@ -episodes,rewards,steps -0,1.0,6 -1,1.0,6 -2,1.0,6 -3,1.0,6 -4,1.0,6 -5,1.0,6 -6,1.0,6 -7,1.0,6 -8,1.0,6 -9,1.0,6 -10,1.0,6 -11,1.0,6 -12,1.0,6 -13,1.0,6 -14,1.0,6 -15,1.0,6 -16,1.0,6 -17,1.0,6 -18,1.0,6 -19,1.0,6 diff --git a/projects/codes/QLearning/Test_Racetrack-v0_QLearning_20221030-014958/config.yaml b/projects/codes/QLearning/Test_Racetrack-v0_QLearning_20221030-014958/config.yaml deleted file mode 100644 index 42d7573..0000000 --- a/projects/codes/QLearning/Test_Racetrack-v0_QLearning_20221030-014958/config.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: QLearning - device: cpu - env_name: Racetrack-v0 - load_checkpoint: true - load_path: Train_Racetrack-v0_QLearning_20221030-014833 - max_steps: 200 - mode: test - save_fig: true - seed: 10 - show_fig: false - test_eps: 20 - train_eps: 400 -algo_cfg: - epsilon_decay: 300 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.9 - lr: 0.1 diff --git a/projects/codes/QLearning/Test_Racetrack-v0_QLearning_20221030-014958/logs/log.txt b/projects/codes/QLearning/Test_Racetrack-v0_QLearning_20221030-014958/logs/log.txt deleted file mode 100644 index f36fac9..0000000 --- a/projects/codes/QLearning/Test_Racetrack-v0_QLearning_20221030-014958/logs/log.txt +++ /dev/null @@ -1,24 +0,0 @@ -2022-10-30 01:49:58 - r - INFO: - n_states: 4, n_actions: 9 -2022-10-30 01:49:58 - r - INFO: - Start testing! -2022-10-30 01:49:58 - r - INFO: - Env: Racetrack-v0, Algorithm: QLearning, Device: cpu -2022-10-30 01:49:58 - r - INFO: - Episode: 1/20, Steps:14 Reward: -4.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 2/20, Steps:8 Reward: 2.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 3/20, Steps:6 Reward: 4.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 4/20, Steps:22 Reward: -12.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 5/20, Steps:15 Reward: -15.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 6/20, Steps:6 Reward: 4.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 7/20, Steps:5 Reward: 5.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 8/20, Steps:8 Reward: 2.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 9/20, Steps:15 Reward: -5.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 10/20, Steps:8 Reward: 2.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 11/20, Steps:5 Reward: 5.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 12/20, Steps:15 Reward: -5.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 13/20, Steps:6 Reward: 4.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 14/20, Steps:31 Reward: -51.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 15/20, Steps:13 Reward: -13.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 16/20, Steps:7 Reward: 3.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 17/20, Steps:6 Reward: 4.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 18/20, Steps:5 Reward: 5.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 19/20, Steps:17 Reward: -17.00 -2022-10-30 01:49:58 - r - INFO: - Episode: 20/20, Steps:15 Reward: -5.00 -2022-10-30 01:49:58 - r - INFO: - Finish testing! diff --git a/projects/codes/QLearning/Test_Racetrack-v0_QLearning_20221030-014958/models/Qleaning_model.pkl b/projects/codes/QLearning/Test_Racetrack-v0_QLearning_20221030-014958/models/Qleaning_model.pkl deleted file mode 100644 index 1f458e1..0000000 Binary files a/projects/codes/QLearning/Test_Racetrack-v0_QLearning_20221030-014958/models/Qleaning_model.pkl and /dev/null differ diff --git a/projects/codes/QLearning/Test_Racetrack-v0_QLearning_20221030-014958/results/learning_curve.png b/projects/codes/QLearning/Test_Racetrack-v0_QLearning_20221030-014958/results/learning_curve.png deleted file mode 100644 index 869b2c9..0000000 Binary files a/projects/codes/QLearning/Test_Racetrack-v0_QLearning_20221030-014958/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/QLearning/Test_Racetrack-v0_QLearning_20221030-014958/results/res.csv b/projects/codes/QLearning/Test_Racetrack-v0_QLearning_20221030-014958/results/res.csv deleted file mode 100644 index cf33c86..0000000 --- a/projects/codes/QLearning/Test_Racetrack-v0_QLearning_20221030-014958/results/res.csv +++ /dev/null @@ -1,21 +0,0 @@ -episodes,rewards,steps -0,-4,14 -1,2,8 -2,4,6 -3,-12,22 -4,-15,15 -5,4,6 -6,5,5 -7,2,8 -8,-5,15 -9,2,8 -10,5,5 -11,-5,15 -12,4,6 -13,-51,31 -14,-13,13 -15,3,7 -16,4,6 -17,5,5 -18,-17,17 -19,-5,15 diff --git a/projects/codes/QLearning/Train_CliffWalking-v0_QLearning_20221030-014916/config.yaml b/projects/codes/QLearning/Train_CliffWalking-v0_QLearning_20221030-014916/config.yaml deleted file mode 100644 index 7610f6c..0000000 --- a/projects/codes/QLearning/Train_CliffWalking-v0_QLearning_20221030-014916/config.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: QLearning - device: cpu - env_name: CliffWalking-v0 - load_checkpoint: false - load_path: Train_CartPole-v1_DQN_20221026-054757 - max_steps: 200 - mode: train - save_fig: true - seed: 1 - show_fig: false - test_eps: 20 - train_eps: 800 -algo_cfg: - epsilon_decay: 300 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - lr: 0.1 diff --git a/projects/codes/QLearning/Train_CliffWalking-v0_QLearning_20221030-014916/logs/log.txt b/projects/codes/QLearning/Train_CliffWalking-v0_QLearning_20221030-014916/logs/log.txt deleted file mode 100644 index d42935f..0000000 --- a/projects/codes/QLearning/Train_CliffWalking-v0_QLearning_20221030-014916/logs/log.txt +++ /dev/null @@ -1,804 +0,0 @@ -2022-10-30 01:49:16 - r - INFO: - n_states: 48, n_actions: 4 -2022-10-30 01:49:16 - r - INFO: - Start training! -2022-10-30 01:49:16 - r - INFO: - Env: CliffWalking-v0, Algorithm: QLearning, Device: cpu -2022-10-30 01:49:16 - r - INFO: - Episode: 1/800, Reward: -1586.00: Epislon: 0.493 -2022-10-30 01:49:16 - r - INFO: - Episode: 2/800, Reward: -1091.00: Epislon: 0.258 -2022-10-30 01:49:16 - r - INFO: - Episode: 3/800, Reward: -596.00: Epislon: 0.137 -2022-10-30 01:49:16 - r - INFO: - Episode: 4/800, Reward: -497.00: Epislon: 0.075 -2022-10-30 01:49:16 - r - INFO: - Episode: 5/800, Reward: -398.00: Epislon: 0.044 -2022-10-30 01:49:16 - r - INFO: - Episode: 6/800, Reward: -362.00: Epislon: 0.029 -2022-10-30 01:49:16 - r - INFO: - Episode: 7/800, Reward: -179.00: Epislon: 0.021 -2022-10-30 01:49:16 - r - INFO: - Episode: 8/800, Reward: -398.00: Epislon: 0.015 -2022-10-30 01:49:16 - r - INFO: - Episode: 9/800, Reward: -79.00: Epislon: 0.014 -2022-10-30 01:49:16 - r - INFO: - Episode: 10/800, Reward: -141.00: Epislon: 0.013 -2022-10-30 01:49:16 - r - INFO: - Episode: 11/800, Reward: -143.00: Epislon: 0.012 -2022-10-30 01:49:16 - r - INFO: - Episode: 12/800, Reward: -134.00: Epislon: 0.011 -2022-10-30 01:49:16 - r - INFO: - Episode: 13/800, Reward: -299.00: Epislon: 0.011 -2022-10-30 01:49:16 - r - INFO: - Episode: 14/800, Reward: -102.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 15/800, Reward: -61.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 16/800, Reward: -136.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 17/800, Reward: -176.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 18/800, Reward: -98.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 19/800, Reward: -92.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 20/800, Reward: -110.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 21/800, Reward: -67.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 22/800, Reward: -136.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 23/800, Reward: -98.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 24/800, Reward: -164.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 25/800, Reward: -65.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 26/800, Reward: -98.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 27/800, Reward: -33.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 28/800, Reward: -161.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 29/800, Reward: -72.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 30/800, Reward: -73.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 31/800, Reward: -116.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 32/800, Reward: -50.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 33/800, Reward: -66.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 34/800, Reward: -123.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 35/800, Reward: -40.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 36/800, Reward: -100.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 37/800, Reward: -56.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 38/800, Reward: -101.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 39/800, Reward: -55.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 40/800, Reward: -84.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 41/800, Reward: -68.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 42/800, Reward: -33.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 43/800, Reward: -113.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 44/800, Reward: -72.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 45/800, Reward: -36.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 46/800, Reward: -84.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 47/800, Reward: -45.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 48/800, Reward: -86.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 49/800, Reward: -57.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 50/800, Reward: -92.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 51/800, Reward: -39.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 52/800, Reward: -76.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 53/800, Reward: -39.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 54/800, Reward: -47.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 55/800, Reward: -88.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 56/800, Reward: -40.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 57/800, Reward: -55.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 58/800, Reward: -69.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 59/800, Reward: -51.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 60/800, Reward: -69.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 61/800, Reward: -36.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 62/800, Reward: -84.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 63/800, Reward: -38.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 64/800, Reward: -56.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 65/800, Reward: -58.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 66/800, Reward: -27.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 67/800, Reward: -64.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 68/800, Reward: -38.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 69/800, Reward: -53.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 70/800, Reward: -70.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 71/800, Reward: -40.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 72/800, Reward: -47.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 73/800, Reward: -71.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 74/800, Reward: -47.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 75/800, Reward: -32.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 76/800, Reward: -70.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 77/800, Reward: -36.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 78/800, Reward: -36.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 79/800, Reward: -73.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 80/800, Reward: -18.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 81/800, Reward: -67.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 82/800, Reward: -29.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 83/800, Reward: -25.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 84/800, Reward: -64.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 85/800, Reward: -28.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 86/800, Reward: -61.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 87/800, Reward: -33.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 88/800, Reward: -30.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 89/800, Reward: -25.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 90/800, Reward: -86.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 91/800, Reward: -19.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 92/800, Reward: -53.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 93/800, Reward: -39.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 94/800, Reward: -39.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 95/800, Reward: -36.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 96/800, Reward: -45.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 97/800, Reward: -43.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 98/800, Reward: -31.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 99/800, Reward: -37.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 100/800, Reward: -43.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 101/800, Reward: -46.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 102/800, Reward: -28.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 103/800, Reward: -32.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 104/800, Reward: -47.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 105/800, Reward: -37.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 106/800, Reward: -39.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 107/800, Reward: -38.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 108/800, Reward: -29.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 109/800, Reward: -44.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 110/800, Reward: -39.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 111/800, Reward: -26.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 112/800, Reward: -28.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 113/800, Reward: -63.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 114/800, Reward: -18.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 115/800, Reward: -52.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 116/800, Reward: -24.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 117/800, Reward: -26.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 118/800, Reward: -39.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 119/800, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 120/800, Reward: -44.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 121/800, Reward: -48.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 122/800, Reward: -24.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 123/800, Reward: -40.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 124/800, Reward: -31.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 125/800, Reward: -23.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 126/800, Reward: -35.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 127/800, Reward: -26.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 128/800, Reward: -31.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 129/800, Reward: -33.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 130/800, Reward: -32.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 131/800, Reward: -36.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 132/800, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 133/800, Reward: -43.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 134/800, Reward: -51.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 135/800, Reward: -17.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 136/800, Reward: -28.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 137/800, Reward: -24.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 138/800, Reward: -42.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 139/800, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 140/800, Reward: -24.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 141/800, Reward: -30.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 142/800, Reward: -24.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 143/800, Reward: -36.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 144/800, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 145/800, Reward: -42.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 146/800, Reward: -44.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 147/800, Reward: -25.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 148/800, Reward: -42.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 149/800, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 150/800, Reward: -30.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 151/800, Reward: -32.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 152/800, Reward: -20.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 153/800, Reward: -44.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 154/800, Reward: -20.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 155/800, Reward: -29.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 156/800, Reward: -20.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 157/800, Reward: -30.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 158/800, Reward: -18.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 159/800, Reward: -25.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 160/800, Reward: -38.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 161/800, Reward: -34.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 162/800, Reward: -25.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 163/800, Reward: -30.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 164/800, Reward: -24.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 165/800, Reward: -19.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 166/800, Reward: -34.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 167/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 168/800, Reward: -27.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 169/800, Reward: -22.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 170/800, Reward: -28.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 171/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 172/800, Reward: -20.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 173/800, Reward: -58.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 174/800, Reward: -18.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 175/800, Reward: -19.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 176/800, Reward: -29.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 177/800, Reward: -125.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 178/800, Reward: -156.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 179/800, Reward: -18.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 180/800, Reward: -25.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 181/800, Reward: -27.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 182/800, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 183/800, Reward: -22.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 184/800, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 185/800, Reward: -22.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 186/800, Reward: -25.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 187/800, Reward: -25.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 188/800, Reward: -27.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 189/800, Reward: -19.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 190/800, Reward: -26.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 191/800, Reward: -17.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 192/800, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 193/800, Reward: -30.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 194/800, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 195/800, Reward: -35.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 196/800, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 197/800, Reward: -134.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 198/800, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 199/800, Reward: -18.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 200/800, Reward: -27.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 201/800, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 202/800, Reward: -44.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 203/800, Reward: -23.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 204/800, Reward: -23.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 205/800, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 206/800, Reward: -18.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 207/800, Reward: -17.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 208/800, Reward: -18.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 209/800, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 210/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 211/800, Reward: -19.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 212/800, Reward: -26.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 213/800, Reward: -27.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 214/800, Reward: -133.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 215/800, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 216/800, Reward: -24.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 217/800, Reward: -29.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 218/800, Reward: -24.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 219/800, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 220/800, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 221/800, Reward: -24.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 222/800, Reward: -17.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 223/800, Reward: -143.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 224/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 225/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 226/800, Reward: -23.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 227/800, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 228/800, Reward: -26.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 229/800, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 230/800, Reward: -19.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 231/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 232/800, Reward: -19.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 233/800, Reward: -29.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 234/800, Reward: -26.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 235/800, Reward: -30.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 236/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 237/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 238/800, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 239/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 240/800, Reward: -18.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 241/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 242/800, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 243/800, Reward: -27.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 244/800, Reward: -17.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 245/800, Reward: -27.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 246/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 247/800, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 248/800, Reward: -28.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 249/800, Reward: -17.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 250/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 251/800, Reward: -17.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 252/800, Reward: -19.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 253/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 254/800, Reward: -23.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 255/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 256/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 257/800, Reward: -19.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 258/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 259/800, Reward: -22.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 260/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 261/800, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 262/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 263/800, Reward: -34.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 264/800, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 265/800, Reward: -26.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 266/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 267/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 268/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 269/800, Reward: -18.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 270/800, Reward: -18.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 271/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 272/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 273/800, Reward: -19.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 274/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 275/800, Reward: -27.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 276/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 277/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 278/800, Reward: -22.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 279/800, Reward: -17.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 280/800, Reward: -17.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 281/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 282/800, Reward: -26.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 283/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 284/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 285/800, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 286/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 287/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 288/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 289/800, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 290/800, Reward: -24.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 291/800, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 292/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 293/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 294/800, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 295/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 296/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 297/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 298/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 299/800, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 300/800, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 301/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 302/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 303/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 304/800, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 305/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 306/800, Reward: -22.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 307/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 308/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 309/800, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 310/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 311/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 312/800, Reward: -31.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 313/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 314/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 315/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 316/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 317/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 318/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 319/800, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 320/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 321/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 322/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 323/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 324/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 325/800, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 326/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 327/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 328/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 329/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 330/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 331/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 332/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 333/800, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 334/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 335/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 336/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 337/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 338/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 339/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 340/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 341/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 342/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 343/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 344/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 345/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 346/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 347/800, Reward: -22.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 348/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 349/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 350/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 351/800, Reward: -17.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 352/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 353/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 354/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 355/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 356/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 357/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 358/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 359/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 360/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 361/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 362/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 363/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 364/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 365/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 366/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 367/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 368/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 369/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 370/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 371/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 372/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 373/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 374/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 375/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 376/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 377/800, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 378/800, Reward: -123.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 379/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 380/800, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 381/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 382/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 383/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 384/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 385/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 386/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 387/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 388/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 389/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 390/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 391/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 392/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 393/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 394/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 395/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 396/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 397/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 398/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 399/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 400/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 401/800, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 402/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 403/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 404/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 405/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 406/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 407/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 408/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 409/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 410/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 411/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 412/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 413/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 414/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 415/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 416/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 417/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 418/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 419/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 420/800, Reward: -113.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 421/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 422/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 423/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 424/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 425/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 426/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 427/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 428/800, Reward: -115.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 429/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 430/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 431/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 432/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 433/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 434/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 435/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 436/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 437/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 438/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 439/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 440/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 441/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 442/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 443/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 444/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 445/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 446/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 447/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 448/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 449/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 450/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 451/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 452/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 453/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 454/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 455/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 456/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 457/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 458/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 459/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 460/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 461/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 462/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 463/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 464/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 465/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 466/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 467/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 468/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 469/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 470/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 471/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 472/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 473/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 474/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 475/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 476/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 477/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 478/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 479/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 480/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 481/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 482/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 483/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 484/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 485/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 486/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 487/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 488/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 489/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 490/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 491/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 492/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 493/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 494/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 495/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 496/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 497/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 498/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 499/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 500/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 501/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 502/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 503/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 504/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 505/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 506/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 507/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 508/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 509/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 510/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 511/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 512/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 513/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 514/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 515/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 516/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 517/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 518/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 519/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 520/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 521/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 522/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 523/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 524/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 525/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 526/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 527/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 528/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 529/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 530/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 531/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 532/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 533/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 534/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 535/800, Reward: -122.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 536/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 537/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 538/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 539/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 540/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 541/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 542/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 543/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 544/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 545/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 546/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 547/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 548/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 549/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 550/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 551/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 552/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 553/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 554/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 555/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 556/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 557/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 558/800, Reward: -115.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 559/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 560/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 561/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 562/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 563/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 564/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 565/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 566/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 567/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 568/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 569/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 570/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 571/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 572/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 573/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 574/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 575/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 576/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 577/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 578/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 579/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 580/800, Reward: -122.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 581/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 582/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 583/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 584/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 585/800, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 586/800, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 587/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 588/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 589/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 590/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 591/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 592/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 593/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 594/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 595/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 596/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 597/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 598/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 599/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 600/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 601/800, Reward: -122.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 602/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 603/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 604/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 605/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 606/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 607/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 608/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 609/800, Reward: -116.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 610/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 611/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 612/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 613/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 614/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 615/800, Reward: -115.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 616/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 617/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 618/800, Reward: -122.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 619/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 620/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 621/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 622/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 623/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 624/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 625/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 626/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 627/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 628/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 629/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 630/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 631/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 632/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 633/800, Reward: -116.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 634/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 635/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 636/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 637/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 638/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 639/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 640/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 641/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 642/800, Reward: -117.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 643/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 644/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 645/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 646/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 647/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 648/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 649/800, Reward: -223.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 650/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 651/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 652/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 653/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 654/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 655/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 656/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 657/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 658/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 659/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 660/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 661/800, Reward: -221.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 662/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 663/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 664/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 665/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 666/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 667/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 668/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 669/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 670/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 671/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 672/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 673/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 674/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 675/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 676/800, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 677/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 678/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 679/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 680/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 681/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 682/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 683/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 684/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 685/800, Reward: -113.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 686/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 687/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 688/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 689/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 690/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 691/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 692/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 693/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 694/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 695/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 696/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 697/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 698/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 699/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 700/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 701/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 702/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 703/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 704/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 705/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 706/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 707/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 708/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 709/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 710/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 711/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 712/800, Reward: -17.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 713/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 714/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 715/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 716/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 717/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 718/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 719/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 720/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 721/800, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 722/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 723/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 724/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 725/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 726/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 727/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 728/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 729/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 730/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 731/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 732/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 733/800, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 734/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 735/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 736/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 737/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 738/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 739/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 740/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 741/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 742/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 743/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 744/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 745/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 746/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 747/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 748/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 749/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 750/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 751/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 752/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 753/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 754/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 755/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 756/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 757/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 758/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 759/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 760/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 761/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 762/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 763/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 764/800, Reward: -122.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 765/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 766/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 767/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 768/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 769/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 770/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 771/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 772/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 773/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 774/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 775/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 776/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 777/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 778/800, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 779/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 780/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 781/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 782/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 783/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 784/800, Reward: -17.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 785/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 786/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 787/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 788/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 789/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 790/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 791/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 792/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 793/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 794/800, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 795/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 796/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 797/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 798/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 799/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Episode: 800/800, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:49:16 - r - INFO: - Finish training! diff --git a/projects/codes/QLearning/Train_CliffWalking-v0_QLearning_20221030-014916/models/Qleaning_model.pkl b/projects/codes/QLearning/Train_CliffWalking-v0_QLearning_20221030-014916/models/Qleaning_model.pkl deleted file mode 100644 index 3be0dc4..0000000 Binary files a/projects/codes/QLearning/Train_CliffWalking-v0_QLearning_20221030-014916/models/Qleaning_model.pkl and /dev/null differ diff --git a/projects/codes/QLearning/Train_CliffWalking-v0_QLearning_20221030-014916/results/learning_curve.png b/projects/codes/QLearning/Train_CliffWalking-v0_QLearning_20221030-014916/results/learning_curve.png deleted file mode 100644 index ee7abc9..0000000 Binary files a/projects/codes/QLearning/Train_CliffWalking-v0_QLearning_20221030-014916/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/QLearning/Train_CliffWalking-v0_QLearning_20221030-014916/results/res.csv b/projects/codes/QLearning/Train_CliffWalking-v0_QLearning_20221030-014916/results/res.csv deleted file mode 100644 index 3799662..0000000 --- a/projects/codes/QLearning/Train_CliffWalking-v0_QLearning_20221030-014916/results/res.csv +++ /dev/null @@ -1,801 +0,0 @@ -episodes,rewards,steps -0,-1586,200 -1,-1091,200 -2,-596,200 -3,-497,200 -4,-398,200 -5,-362,164 -6,-179,179 -7,-398,200 -8,-79,79 -9,-141,141 -10,-143,143 -11,-134,134 -12,-299,200 -13,-102,102 -14,-61,61 -15,-136,136 -16,-176,176 -17,-98,98 -18,-92,92 -19,-110,110 -20,-67,67 -21,-136,136 -22,-98,98 -23,-164,164 -24,-65,65 -25,-98,98 -26,-33,33 -27,-161,161 -28,-72,72 -29,-73,73 -30,-116,116 -31,-50,50 -32,-66,66 -33,-123,123 -34,-40,40 -35,-100,100 -36,-56,56 -37,-101,101 -38,-55,55 -39,-84,84 -40,-68,68 -41,-33,33 -42,-113,113 -43,-72,72 -44,-36,36 -45,-84,84 -46,-45,45 -47,-86,86 -48,-57,57 -49,-92,92 -50,-39,39 -51,-76,76 -52,-39,39 -53,-47,47 -54,-88,88 -55,-40,40 -56,-55,55 -57,-69,69 -58,-51,51 -59,-69,69 -60,-36,36 -61,-84,84 -62,-38,38 -63,-56,56 -64,-58,58 -65,-27,27 -66,-64,64 -67,-38,38 -68,-53,53 -69,-70,70 -70,-40,40 -71,-47,47 -72,-71,71 -73,-47,47 -74,-32,32 -75,-70,70 -76,-36,36 -77,-36,36 -78,-73,73 -79,-18,18 -80,-67,67 -81,-29,29 -82,-25,25 -83,-64,64 -84,-28,28 -85,-61,61 -86,-33,33 -87,-30,30 -88,-25,25 -89,-86,86 -90,-19,19 -91,-53,53 -92,-39,39 -93,-39,39 -94,-36,36 -95,-45,45 -96,-43,43 -97,-31,31 -98,-37,37 -99,-43,43 -100,-46,46 -101,-28,28 -102,-32,32 -103,-47,47 -104,-37,37 -105,-39,39 -106,-38,38 -107,-29,29 -108,-44,44 -109,-39,39 -110,-26,26 -111,-28,28 -112,-63,63 -113,-18,18 -114,-52,52 -115,-24,24 -116,-26,26 -117,-39,39 -118,-21,21 -119,-44,44 -120,-48,48 -121,-24,24 -122,-40,40 -123,-31,31 -124,-23,23 -125,-35,35 -126,-26,26 -127,-31,31 -128,-33,33 -129,-32,32 -130,-36,36 -131,-21,21 -132,-43,43 -133,-51,51 -134,-17,17 -135,-28,28 -136,-24,24 -137,-42,42 -138,-21,21 -139,-24,24 -140,-30,30 -141,-24,24 -142,-36,36 -143,-14,14 -144,-42,42 -145,-44,44 -146,-25,25 -147,-42,42 -148,-16,16 -149,-30,30 -150,-32,32 -151,-20,20 -152,-44,44 -153,-20,20 -154,-29,29 -155,-20,20 -156,-30,30 -157,-18,18 -158,-25,25 -159,-38,38 -160,-34,34 -161,-25,25 -162,-30,30 -163,-24,24 -164,-19,19 -165,-34,34 -166,-15,15 -167,-27,27 -168,-22,22 -169,-28,28 -170,-13,13 -171,-20,20 -172,-58,58 -173,-18,18 -174,-19,19 -175,-29,29 -176,-125,26 -177,-156,57 -178,-18,18 -179,-25,25 -180,-27,27 -181,-21,21 -182,-22,22 -183,-21,21 -184,-22,22 -185,-25,25 -186,-25,25 -187,-27,27 -188,-19,19 -189,-26,26 -190,-17,17 -191,-21,21 -192,-30,30 -193,-16,16 -194,-35,35 -195,-21,21 -196,-134,35 -197,-21,21 -198,-18,18 -199,-27,27 -200,-16,16 -201,-44,44 -202,-23,23 -203,-23,23 -204,-16,16 -205,-18,18 -206,-17,17 -207,-18,18 -208,-16,16 -209,-13,13 -210,-19,19 -211,-26,26 -212,-27,27 -213,-133,34 -214,-21,21 -215,-24,24 -216,-29,29 -217,-24,24 -218,-16,16 -219,-16,16 -220,-24,24 -221,-17,17 -222,-143,44 -223,-15,15 -224,-15,15 -225,-23,23 -226,-21,21 -227,-26,26 -228,-16,16 -229,-19,19 -230,-13,13 -231,-19,19 -232,-29,29 -233,-26,26 -234,-30,30 -235,-13,13 -236,-13,13 -237,-21,21 -238,-15,15 -239,-18,18 -240,-13,13 -241,-16,16 -242,-27,27 -243,-17,17 -244,-27,27 -245,-15,15 -246,-14,14 -247,-28,28 -248,-17,17 -249,-15,15 -250,-17,17 -251,-19,19 -252,-13,13 -253,-23,23 -254,-15,15 -255,-15,15 -256,-19,19 -257,-15,15 -258,-22,22 -259,-13,13 -260,-16,16 -261,-15,15 -262,-34,34 -263,-16,16 -264,-26,26 -265,-13,13 -266,-15,15 -267,-15,15 -268,-18,18 -269,-18,18 -270,-13,13 -271,-13,13 -272,-19,19 -273,-13,13 -274,-27,27 -275,-13,13 -276,-13,13 -277,-22,22 -278,-17,17 -279,-17,17 -280,-13,13 -281,-26,26 -282,-13,13 -283,-13,13 -284,-14,14 -285,-15,15 -286,-13,13 -287,-13,13 -288,-14,14 -289,-24,24 -290,-21,21 -291,-13,13 -292,-13,13 -293,-14,14 -294,-15,15 -295,-13,13 -296,-13,13 -297,-13,13 -298,-14,14 -299,-21,21 -300,-15,15 -301,-13,13 -302,-13,13 -303,-21,21 -304,-13,13 -305,-22,22 -306,-13,13 -307,-13,13 -308,-16,16 -309,-15,15 -310,-13,13 -311,-31,31 -312,-13,13 -313,-13,13 -314,-15,15 -315,-13,13 -316,-13,13 -317,-13,13 -318,-14,14 -319,-13,13 -320,-15,15 -321,-13,13 -322,-13,13 -323,-13,13 -324,-16,16 -325,-13,13 -326,-13,13 -327,-13,13 -328,-13,13 -329,-13,13 -330,-13,13 -331,-15,15 -332,-16,16 -333,-13,13 -334,-13,13 -335,-13,13 -336,-13,13 -337,-13,13 -338,-15,15 -339,-13,13 -340,-13,13 -341,-13,13 -342,-13,13 -343,-13,13 -344,-13,13 -345,-13,13 -346,-22,22 -347,-13,13 -348,-13,13 -349,-13,13 -350,-17,17 -351,-13,13 -352,-13,13 -353,-13,13 -354,-13,13 -355,-13,13 -356,-13,13 -357,-13,13 -358,-13,13 -359,-13,13 -360,-13,13 -361,-13,13 -362,-13,13 -363,-13,13 -364,-13,13 -365,-13,13 -366,-13,13 -367,-13,13 -368,-13,13 -369,-13,13 -370,-13,13 -371,-13,13 -372,-13,13 -373,-13,13 -374,-13,13 -375,-13,13 -376,-21,21 -377,-123,24 -378,-13,13 -379,-14,14 -380,-13,13 -381,-13,13 -382,-13,13 -383,-13,13 -384,-13,13 -385,-13,13 -386,-13,13 -387,-13,13 -388,-13,13 -389,-13,13 -390,-13,13 -391,-13,13 -392,-13,13 -393,-13,13 -394,-13,13 -395,-13,13 -396,-15,15 -397,-13,13 -398,-13,13 -399,-13,13 -400,-14,14 -401,-13,13 -402,-13,13 -403,-13,13 -404,-13,13 -405,-13,13 -406,-13,13 -407,-13,13 -408,-13,13 -409,-13,13 -410,-13,13 -411,-13,13 -412,-13,13 -413,-13,13 -414,-13,13 -415,-13,13 -416,-13,13 -417,-13,13 -418,-13,13 -419,-113,14 -420,-13,13 -421,-13,13 -422,-13,13 -423,-13,13 -424,-13,13 -425,-13,13 -426,-13,13 -427,-115,16 -428,-13,13 -429,-13,13 -430,-13,13 -431,-13,13 -432,-13,13 -433,-13,13 -434,-15,15 -435,-13,13 -436,-13,13 -437,-13,13 -438,-13,13 -439,-13,13 -440,-13,13 -441,-13,13 -442,-13,13 -443,-13,13 -444,-13,13 -445,-15,15 -446,-13,13 -447,-13,13 -448,-13,13 -449,-13,13 -450,-13,13 -451,-13,13 -452,-13,13 -453,-13,13 -454,-13,13 -455,-13,13 -456,-13,13 -457,-13,13 -458,-13,13 -459,-13,13 -460,-13,13 -461,-13,13 -462,-13,13 -463,-13,13 -464,-13,13 -465,-13,13 -466,-13,13 -467,-13,13 -468,-13,13 -469,-15,15 -470,-13,13 -471,-13,13 -472,-13,13 -473,-13,13 -474,-13,13 -475,-13,13 -476,-13,13 -477,-13,13 -478,-13,13 -479,-13,13 -480,-13,13 -481,-13,13 -482,-13,13 -483,-13,13 -484,-13,13 -485,-13,13 -486,-13,13 -487,-13,13 -488,-13,13 -489,-13,13 -490,-13,13 -491,-13,13 -492,-13,13 -493,-13,13 -494,-13,13 -495,-13,13 -496,-13,13 -497,-13,13 -498,-13,13 -499,-13,13 -500,-13,13 -501,-13,13 -502,-15,15 -503,-13,13 -504,-13,13 -505,-15,15 -506,-13,13 -507,-13,13 -508,-13,13 -509,-13,13 -510,-13,13 -511,-13,13 -512,-13,13 -513,-13,13 -514,-13,13 -515,-13,13 -516,-13,13 -517,-13,13 -518,-13,13 -519,-13,13 -520,-13,13 -521,-13,13 -522,-13,13 -523,-13,13 -524,-13,13 -525,-13,13 -526,-15,15 -527,-13,13 -528,-13,13 -529,-13,13 -530,-13,13 -531,-13,13 -532,-13,13 -533,-13,13 -534,-122,23 -535,-13,13 -536,-13,13 -537,-13,13 -538,-13,13 -539,-13,13 -540,-13,13 -541,-13,13 -542,-13,13 -543,-15,15 -544,-13,13 -545,-13,13 -546,-13,13 -547,-13,13 -548,-13,13 -549,-13,13 -550,-13,13 -551,-15,15 -552,-13,13 -553,-13,13 -554,-13,13 -555,-13,13 -556,-13,13 -557,-115,16 -558,-13,13 -559,-15,15 -560,-13,13 -561,-13,13 -562,-13,13 -563,-13,13 -564,-13,13 -565,-13,13 -566,-13,13 -567,-13,13 -568,-13,13 -569,-13,13 -570,-13,13 -571,-13,13 -572,-13,13 -573,-13,13 -574,-15,15 -575,-13,13 -576,-13,13 -577,-13,13 -578,-13,13 -579,-122,23 -580,-13,13 -581,-13,13 -582,-13,13 -583,-13,13 -584,-14,14 -585,-14,14 -586,-13,13 -587,-13,13 -588,-13,13 -589,-13,13 -590,-13,13 -591,-13,13 -592,-13,13 -593,-13,13 -594,-15,15 -595,-13,13 -596,-13,13 -597,-13,13 -598,-13,13 -599,-13,13 -600,-122,23 -601,-13,13 -602,-13,13 -603,-13,13 -604,-13,13 -605,-13,13 -606,-13,13 -607,-13,13 -608,-116,17 -609,-13,13 -610,-13,13 -611,-13,13 -612,-13,13 -613,-13,13 -614,-115,16 -615,-13,13 -616,-13,13 -617,-122,23 -618,-13,13 -619,-13,13 -620,-13,13 -621,-13,13 -622,-13,13 -623,-13,13 -624,-13,13 -625,-13,13 -626,-13,13 -627,-13,13 -628,-13,13 -629,-13,13 -630,-15,15 -631,-13,13 -632,-116,17 -633,-13,13 -634,-13,13 -635,-13,13 -636,-13,13 -637,-13,13 -638,-13,13 -639,-13,13 -640,-13,13 -641,-117,18 -642,-13,13 -643,-13,13 -644,-13,13 -645,-13,13 -646,-13,13 -647,-13,13 -648,-223,25 -649,-13,13 -650,-13,13 -651,-13,13 -652,-13,13 -653,-13,13 -654,-15,15 -655,-13,13 -656,-13,13 -657,-13,13 -658,-13,13 -659,-13,13 -660,-221,23 -661,-13,13 -662,-15,15 -663,-13,13 -664,-13,13 -665,-13,13 -666,-13,13 -667,-13,13 -668,-13,13 -669,-13,13 -670,-13,13 -671,-13,13 -672,-13,13 -673,-13,13 -674,-13,13 -675,-21,21 -676,-13,13 -677,-15,15 -678,-13,13 -679,-13,13 -680,-13,13 -681,-13,13 -682,-13,13 -683,-13,13 -684,-113,14 -685,-13,13 -686,-13,13 -687,-13,13 -688,-13,13 -689,-13,13 -690,-13,13 -691,-13,13 -692,-13,13 -693,-13,13 -694,-13,13 -695,-13,13 -696,-13,13 -697,-13,13 -698,-13,13 -699,-13,13 -700,-13,13 -701,-15,15 -702,-13,13 -703,-15,15 -704,-13,13 -705,-13,13 -706,-15,15 -707,-13,13 -708,-13,13 -709,-13,13 -710,-13,13 -711,-17,17 -712,-13,13 -713,-13,13 -714,-13,13 -715,-13,13 -716,-13,13 -717,-13,13 -718,-13,13 -719,-13,13 -720,-14,14 -721,-13,13 -722,-13,13 -723,-13,13 -724,-13,13 -725,-13,13 -726,-13,13 -727,-13,13 -728,-13,13 -729,-13,13 -730,-13,13 -731,-13,13 -732,-14,14 -733,-13,13 -734,-13,13 -735,-13,13 -736,-13,13 -737,-15,15 -738,-13,13 -739,-15,15 -740,-13,13 -741,-13,13 -742,-13,13 -743,-13,13 -744,-15,15 -745,-13,13 -746,-13,13 -747,-13,13 -748,-15,15 -749,-13,13 -750,-13,13 -751,-13,13 -752,-13,13 -753,-13,13 -754,-13,13 -755,-13,13 -756,-13,13 -757,-13,13 -758,-13,13 -759,-13,13 -760,-13,13 -761,-13,13 -762,-13,13 -763,-122,23 -764,-15,15 -765,-13,13 -766,-13,13 -767,-13,13 -768,-13,13 -769,-13,13 -770,-13,13 -771,-13,13 -772,-13,13 -773,-13,13 -774,-15,15 -775,-13,13 -776,-13,13 -777,-14,14 -778,-13,13 -779,-13,13 -780,-13,13 -781,-13,13 -782,-13,13 -783,-17,17 -784,-13,13 -785,-13,13 -786,-13,13 -787,-15,15 -788,-13,13 -789,-13,13 -790,-13,13 -791,-13,13 -792,-13,13 -793,-15,15 -794,-13,13 -795,-13,13 -796,-13,13 -797,-13,13 -798,-13,13 -799,-13,13 diff --git a/projects/codes/QLearning/Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504/config.yaml b/projects/codes/QLearning/Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504/config.yaml deleted file mode 100644 index a0bf456..0000000 --- a/projects/codes/QLearning/Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504/config.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: QLearning - device: cpu - env_name: FrozenLakeNoSlippery-v1 - load_checkpoint: false - load_path: Train_CartPole-v1_DQN_20221026-054757 - max_steps: 200 - mode: train - save_fig: true - seed: 10 - show_fig: false - test_eps: 20 - train_eps: 800 -algo_cfg: - epsilon_decay: 2000 - epsilon_end: 0.1 - epsilon_start: 0.7 - gamma: 0.95 - lr: 0.9 diff --git a/projects/codes/QLearning/Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504/logs/log.txt b/projects/codes/QLearning/Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504/logs/log.txt deleted file mode 100644 index f52cf7f..0000000 --- a/projects/codes/QLearning/Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504/logs/log.txt +++ /dev/null @@ -1,804 +0,0 @@ -2022-10-30 01:45:04 - r - INFO: - n_states: 16, n_actions: 4 -2022-10-30 01:45:04 - r - INFO: - Start training! -2022-10-30 01:45:04 - r - INFO: - Env: FrozenLakeNoSlippery-v1, Algorithm: QLearning, Device: cpu -2022-10-30 01:45:04 - r - INFO: - Episode: 1/800, Reward: 0.00: Epislon: 0.694 -2022-10-30 01:45:04 - r - INFO: - Episode: 2/800, Reward: 0.00: Epislon: 0.690 -2022-10-30 01:45:04 - r - INFO: - Episode: 3/800, Reward: 0.00: Epislon: 0.686 -2022-10-30 01:45:04 - r - INFO: - Episode: 4/800, Reward: 0.00: Epislon: 0.683 -2022-10-30 01:45:04 - r - INFO: - Episode: 5/800, Reward: 0.00: Epislon: 0.681 -2022-10-30 01:45:04 - r - INFO: - Episode: 6/800, Reward: 0.00: Epislon: 0.679 -2022-10-30 01:45:04 - r - INFO: - Episode: 7/800, Reward: 0.00: Epislon: 0.676 -2022-10-30 01:45:04 - r - INFO: - Episode: 8/800, Reward: 0.00: Epislon: 0.674 -2022-10-30 01:45:04 - r - INFO: - Episode: 9/800, Reward: 0.00: Epislon: 0.673 -2022-10-30 01:45:04 - r - INFO: - Episode: 10/800, Reward: 0.00: Epislon: 0.670 -2022-10-30 01:45:04 - r - INFO: - Episode: 11/800, Reward: 0.00: Epislon: 0.667 -2022-10-30 01:45:04 - r - INFO: - Episode: 12/800, Reward: 0.00: Epislon: 0.661 -2022-10-30 01:45:04 - r - INFO: - Episode: 13/800, Reward: 0.00: Epislon: 0.660 -2022-10-30 01:45:04 - r - INFO: - Episode: 14/800, Reward: 0.00: Epislon: 0.655 -2022-10-30 01:45:04 - r - INFO: - Episode: 15/800, Reward: 0.00: Epislon: 0.654 -2022-10-30 01:45:04 - r - INFO: - Episode: 16/800, Reward: 0.00: Epislon: 0.652 -2022-10-30 01:45:04 - r - INFO: - Episode: 17/800, Reward: 0.00: Epislon: 0.647 -2022-10-30 01:45:04 - r - INFO: - Episode: 18/800, Reward: 0.00: Epislon: 0.646 -2022-10-30 01:45:04 - r - INFO: - Episode: 19/800, Reward: 0.00: Epislon: 0.645 -2022-10-30 01:45:04 - r - INFO: - Episode: 20/800, Reward: 0.00: Epislon: 0.643 -2022-10-30 01:45:04 - r - INFO: - Episode: 21/800, Reward: 0.00: Epislon: 0.641 -2022-10-30 01:45:04 - r - INFO: - Episode: 22/800, Reward: 0.00: Epislon: 0.640 -2022-10-30 01:45:04 - r - INFO: - Episode: 23/800, Reward: 0.00: Epislon: 0.634 -2022-10-30 01:45:04 - r - INFO: - Episode: 24/800, Reward: 0.00: Epislon: 0.630 -2022-10-30 01:45:04 - r - INFO: - Episode: 25/800, Reward: 0.00: Epislon: 0.629 -2022-10-30 01:45:04 - r - INFO: - Episode: 26/800, Reward: 0.00: Epislon: 0.624 -2022-10-30 01:45:04 - r - INFO: - Episode: 27/800, Reward: 0.00: Epislon: 0.623 -2022-10-30 01:45:04 - r - INFO: - Episode: 28/800, Reward: 0.00: Epislon: 0.618 -2022-10-30 01:45:04 - r - INFO: - Episode: 29/800, Reward: 0.00: Epislon: 0.612 -2022-10-30 01:45:04 - r - INFO: - Episode: 30/800, Reward: 0.00: Epislon: 0.608 -2022-10-30 01:45:04 - r - INFO: - Episode: 31/800, Reward: 0.00: Epislon: 0.605 -2022-10-30 01:45:05 - r - INFO: - Episode: 32/800, Reward: 0.00: Epislon: 0.600 -2022-10-30 01:45:05 - r - INFO: - Episode: 33/800, Reward: 0.00: Epislon: 0.593 -2022-10-30 01:45:05 - r - INFO: - Episode: 34/800, Reward: 0.00: Epislon: 0.587 -2022-10-30 01:45:05 - r - INFO: - Episode: 35/800, Reward: 0.00: Epislon: 0.586 -2022-10-30 01:45:05 - r - INFO: - Episode: 36/800, Reward: 0.00: Epislon: 0.583 -2022-10-30 01:45:05 - r - INFO: - Episode: 37/800, Reward: 0.00: Epislon: 0.582 -2022-10-30 01:45:05 - r - INFO: - Episode: 38/800, Reward: 0.00: Epislon: 0.578 -2022-10-30 01:45:05 - r - INFO: - Episode: 39/800, Reward: 0.00: Epislon: 0.577 -2022-10-30 01:45:05 - r - INFO: - Episode: 40/800, Reward: 0.00: Epislon: 0.575 -2022-10-30 01:45:05 - r - INFO: - Episode: 41/800, Reward: 0.00: Epislon: 0.573 -2022-10-30 01:45:05 - r - INFO: - Episode: 42/800, Reward: 0.00: Epislon: 0.572 -2022-10-30 01:45:05 - r - INFO: - Episode: 43/800, Reward: 0.00: Epislon: 0.571 -2022-10-30 01:45:05 - r - INFO: - Episode: 44/800, Reward: 0.00: Epislon: 0.570 -2022-10-30 01:45:05 - r - INFO: - Episode: 45/800, Reward: 0.00: Epislon: 0.560 -2022-10-30 01:45:05 - r - INFO: - Episode: 46/800, Reward: 0.00: Epislon: 0.558 -2022-10-30 01:45:05 - r - INFO: - Episode: 47/800, Reward: 0.00: Epislon: 0.553 -2022-10-30 01:45:05 - r - INFO: - Episode: 48/800, Reward: 0.00: Epislon: 0.552 -2022-10-30 01:45:05 - r - INFO: - Episode: 49/800, Reward: 1.00: Epislon: 0.544 -2022-10-30 01:45:05 - r - INFO: - Episode: 50/800, Reward: 0.00: Epislon: 0.537 -2022-10-30 01:45:05 - r - INFO: - Episode: 51/800, Reward: 0.00: Epislon: 0.534 -2022-10-30 01:45:05 - r - INFO: - Episode: 52/800, Reward: 0.00: Epislon: 0.533 -2022-10-30 01:45:05 - r - INFO: - Episode: 53/800, Reward: 0.00: Epislon: 0.532 -2022-10-30 01:45:05 - r - INFO: - Episode: 54/800, Reward: 0.00: Epislon: 0.527 -2022-10-30 01:45:05 - r - INFO: - Episode: 55/800, Reward: 0.00: Epislon: 0.526 -2022-10-30 01:45:05 - r - INFO: - Episode: 56/800, Reward: 0.00: Epislon: 0.525 -2022-10-30 01:45:05 - r - INFO: - Episode: 57/800, Reward: 0.00: Epislon: 0.519 -2022-10-30 01:45:05 - r - INFO: - Episode: 58/800, Reward: 0.00: Epislon: 0.518 -2022-10-30 01:45:05 - r - INFO: - Episode: 59/800, Reward: 0.00: Epislon: 0.516 -2022-10-30 01:45:05 - r - INFO: - Episode: 60/800, Reward: 0.00: Epislon: 0.514 -2022-10-30 01:45:05 - r - INFO: - Episode: 61/800, Reward: 0.00: Epislon: 0.512 -2022-10-30 01:45:05 - r - INFO: - Episode: 62/800, Reward: 0.00: Epislon: 0.511 -2022-10-30 01:45:05 - r - INFO: - Episode: 63/800, Reward: 0.00: Epislon: 0.506 -2022-10-30 01:45:05 - r - INFO: - Episode: 64/800, Reward: 0.00: Epislon: 0.504 -2022-10-30 01:45:05 - r - INFO: - Episode: 65/800, Reward: 0.00: Epislon: 0.503 -2022-10-30 01:45:05 - r - INFO: - Episode: 66/800, Reward: 0.00: Epislon: 0.502 -2022-10-30 01:45:05 - r - INFO: - Episode: 67/800, Reward: 0.00: Epislon: 0.501 -2022-10-30 01:45:05 - r - INFO: - Episode: 68/800, Reward: 0.00: Epislon: 0.497 -2022-10-30 01:45:05 - r - INFO: - Episode: 69/800, Reward: 0.00: Epislon: 0.496 -2022-10-30 01:45:05 - r - INFO: - Episode: 70/800, Reward: 0.00: Epislon: 0.491 -2022-10-30 01:45:05 - r - INFO: - Episode: 71/800, Reward: 0.00: Epislon: 0.489 -2022-10-30 01:45:05 - r - INFO: - Episode: 72/800, Reward: 0.00: Epislon: 0.487 -2022-10-30 01:45:05 - r - INFO: - Episode: 73/800, Reward: 0.00: Epislon: 0.486 -2022-10-30 01:45:05 - r - INFO: - Episode: 74/800, Reward: 0.00: Epislon: 0.481 -2022-10-30 01:45:05 - r - INFO: - Episode: 75/800, Reward: 0.00: Epislon: 0.477 -2022-10-30 01:45:05 - r - INFO: - Episode: 76/800, Reward: 0.00: Epislon: 0.475 -2022-10-30 01:45:05 - r - INFO: - Episode: 77/800, Reward: 0.00: Epislon: 0.474 -2022-10-30 01:45:05 - r - INFO: - Episode: 78/800, Reward: 0.00: Epislon: 0.468 -2022-10-30 01:45:05 - r - INFO: - Episode: 79/800, Reward: 0.00: Epislon: 0.465 -2022-10-30 01:45:05 - r - INFO: - Episode: 80/800, Reward: 0.00: Epislon: 0.464 -2022-10-30 01:45:05 - r - INFO: - Episode: 81/800, Reward: 0.00: Epislon: 0.462 -2022-10-30 01:45:05 - r - INFO: - Episode: 82/800, Reward: 0.00: Epislon: 0.460 -2022-10-30 01:45:05 - r - INFO: - Episode: 83/800, Reward: 0.00: Epislon: 0.457 -2022-10-30 01:45:05 - r - INFO: - Episode: 84/800, Reward: 0.00: Epislon: 0.455 -2022-10-30 01:45:05 - r - INFO: - Episode: 85/800, Reward: 0.00: Epislon: 0.454 -2022-10-30 01:45:05 - r - INFO: - Episode: 86/800, Reward: 0.00: Epislon: 0.452 -2022-10-30 01:45:05 - r - INFO: - Episode: 87/800, Reward: 0.00: Epislon: 0.444 -2022-10-30 01:45:05 - r - INFO: - Episode: 88/800, Reward: 0.00: Epislon: 0.440 -2022-10-30 01:45:05 - r - INFO: - Episode: 89/800, Reward: 0.00: Epislon: 0.414 -2022-10-30 01:45:05 - r - INFO: - Episode: 90/800, Reward: 0.00: Epislon: 0.413 -2022-10-30 01:45:05 - r - INFO: - Episode: 91/800, Reward: 0.00: Epislon: 0.411 -2022-10-30 01:45:05 - r - INFO: - Episode: 92/800, Reward: 0.00: Epislon: 0.407 -2022-10-30 01:45:05 - r - INFO: - Episode: 93/800, Reward: 0.00: Epislon: 0.407 -2022-10-30 01:45:05 - r - INFO: - Episode: 94/800, Reward: 0.00: Epislon: 0.406 -2022-10-30 01:45:05 - r - INFO: - Episode: 95/800, Reward: 0.00: Epislon: 0.403 -2022-10-30 01:45:05 - r - INFO: - Episode: 96/800, Reward: 0.00: Epislon: 0.390 -2022-10-30 01:45:05 - r - INFO: - Episode: 97/800, Reward: 0.00: Epislon: 0.386 -2022-10-30 01:45:05 - r - INFO: - Episode: 98/800, Reward: 0.00: Epislon: 0.385 -2022-10-30 01:45:05 - r - INFO: - Episode: 99/800, Reward: 0.00: Epislon: 0.385 -2022-10-30 01:45:05 - r - INFO: - Episode: 100/800, Reward: 0.00: Epislon: 0.383 -2022-10-30 01:45:05 - r - INFO: - Episode: 101/800, Reward: 0.00: Epislon: 0.381 -2022-10-30 01:45:05 - r - INFO: - Episode: 102/800, Reward: 0.00: Epislon: 0.380 -2022-10-30 01:45:05 - r - INFO: - Episode: 103/800, Reward: 0.00: Epislon: 0.378 -2022-10-30 01:45:05 - r - INFO: - Episode: 104/800, Reward: 0.00: Epislon: 0.366 -2022-10-30 01:45:05 - r - INFO: - Episode: 105/800, Reward: 0.00: Epislon: 0.365 -2022-10-30 01:45:05 - r - INFO: - Episode: 106/800, Reward: 0.00: Epislon: 0.359 -2022-10-30 01:45:05 - r - INFO: - Episode: 107/800, Reward: 0.00: Epislon: 0.357 -2022-10-30 01:45:05 - r - INFO: - Episode: 108/800, Reward: 0.00: Epislon: 0.356 -2022-10-30 01:45:05 - r - INFO: - Episode: 109/800, Reward: 0.00: Epislon: 0.350 -2022-10-30 01:45:05 - r - INFO: - Episode: 110/800, Reward: 0.00: Epislon: 0.347 -2022-10-30 01:45:05 - r - INFO: - Episode: 111/800, Reward: 0.00: Epislon: 0.345 -2022-10-30 01:45:05 - r - INFO: - Episode: 112/800, Reward: 0.00: Epislon: 0.343 -2022-10-30 01:45:05 - r - INFO: - Episode: 113/800, Reward: 0.00: Epislon: 0.322 -2022-10-30 01:45:05 - r - INFO: - Episode: 114/800, Reward: 0.00: Epislon: 0.317 -2022-10-30 01:45:05 - r - INFO: - Episode: 115/800, Reward: 0.00: Epislon: 0.308 -2022-10-30 01:45:05 - r - INFO: - Episode: 116/800, Reward: 0.00: Epislon: 0.306 -2022-10-30 01:45:05 - r - INFO: - Episode: 117/800, Reward: 0.00: Epislon: 0.303 -2022-10-30 01:45:05 - r - INFO: - Episode: 118/800, Reward: 0.00: Epislon: 0.300 -2022-10-30 01:45:05 - r - INFO: - Episode: 119/800, Reward: 0.00: Epislon: 0.300 -2022-10-30 01:45:05 - r - INFO: - Episode: 120/800, Reward: 0.00: Epislon: 0.291 -2022-10-30 01:45:05 - r - INFO: - Episode: 121/800, Reward: 0.00: Epislon: 0.290 -2022-10-30 01:45:05 - r - INFO: - Episode: 122/800, Reward: 0.00: Epislon: 0.284 -2022-10-30 01:45:05 - r - INFO: - Episode: 123/800, Reward: 0.00: Epislon: 0.282 -2022-10-30 01:45:05 - r - INFO: - Episode: 124/800, Reward: 0.00: Epislon: 0.276 -2022-10-30 01:45:05 - r - INFO: - Episode: 125/800, Reward: 0.00: Epislon: 0.269 -2022-10-30 01:45:05 - r - INFO: - Episode: 126/800, Reward: 0.00: Epislon: 0.262 -2022-10-30 01:45:05 - r - INFO: - Episode: 127/800, Reward: 0.00: Epislon: 0.246 -2022-10-30 01:45:05 - r - INFO: - Episode: 128/800, Reward: 0.00: Epislon: 0.244 -2022-10-30 01:45:05 - r - INFO: - Episode: 129/800, Reward: 0.00: Epislon: 0.241 -2022-10-30 01:45:05 - r - INFO: - Episode: 130/800, Reward: 0.00: Epislon: 0.236 -2022-10-30 01:45:05 - r - INFO: - Episode: 131/800, Reward: 0.00: Epislon: 0.235 -2022-10-30 01:45:05 - r - INFO: - Episode: 132/800, Reward: 0.00: Epislon: 0.234 -2022-10-30 01:45:05 - r - INFO: - Episode: 133/800, Reward: 0.00: Epislon: 0.233 -2022-10-30 01:45:05 - r - INFO: - Episode: 134/800, Reward: 0.00: Epislon: 0.231 -2022-10-30 01:45:05 - r - INFO: - Episode: 135/800, Reward: 0.00: Epislon: 0.229 -2022-10-30 01:45:05 - r - INFO: - Episode: 136/800, Reward: 0.00: Epislon: 0.227 -2022-10-30 01:45:05 - r - INFO: - Episode: 137/800, Reward: 0.00: Epislon: 0.226 -2022-10-30 01:45:05 - r - INFO: - Episode: 138/800, Reward: 0.00: Epislon: 0.223 -2022-10-30 01:45:05 - r - INFO: - Episode: 139/800, Reward: 0.00: Epislon: 0.216 -2022-10-30 01:45:05 - r - INFO: - Episode: 140/800, Reward: 0.00: Epislon: 0.214 -2022-10-30 01:45:05 - r - INFO: - Episode: 141/800, Reward: 0.00: Epislon: 0.213 -2022-10-30 01:45:05 - r - INFO: - Episode: 142/800, Reward: 0.00: Epislon: 0.211 -2022-10-30 01:45:05 - r - INFO: - Episode: 143/800, Reward: 0.00: Epislon: 0.210 -2022-10-30 01:45:05 - r - INFO: - Episode: 144/800, Reward: 0.00: Epislon: 0.207 -2022-10-30 01:45:05 - r - INFO: - Episode: 145/800, Reward: 0.00: Epislon: 0.202 -2022-10-30 01:45:05 - r - INFO: - Episode: 146/800, Reward: 0.00: Epislon: 0.201 -2022-10-30 01:45:05 - r - INFO: - Episode: 147/800, Reward: 0.00: Epislon: 0.198 -2022-10-30 01:45:05 - r - INFO: - Episode: 148/800, Reward: 0.00: Epislon: 0.196 -2022-10-30 01:45:05 - r - INFO: - Episode: 149/800, Reward: 0.00: Epislon: 0.195 -2022-10-30 01:45:05 - r - INFO: - Episode: 150/800, Reward: 0.00: Epislon: 0.192 -2022-10-30 01:45:05 - r - INFO: - Episode: 151/800, Reward: 0.00: Epislon: 0.190 -2022-10-30 01:45:05 - r - INFO: - Episode: 152/800, Reward: 0.00: Epislon: 0.188 -2022-10-30 01:45:05 - r - INFO: - Episode: 153/800, Reward: 0.00: Epislon: 0.186 -2022-10-30 01:45:05 - r - INFO: - Episode: 154/800, Reward: 0.00: Epislon: 0.185 -2022-10-30 01:45:05 - r - INFO: - Episode: 155/800, Reward: 0.00: Epislon: 0.185 -2022-10-30 01:45:05 - r - INFO: - Episode: 156/800, Reward: 0.00: Epislon: 0.183 -2022-10-30 01:45:05 - r - INFO: - Episode: 157/800, Reward: 0.00: Epislon: 0.182 -2022-10-30 01:45:05 - r - INFO: - Episode: 158/800, Reward: 0.00: Epislon: 0.181 -2022-10-30 01:45:05 - r - INFO: - Episode: 159/800, Reward: 0.00: Epislon: 0.179 -2022-10-30 01:45:05 - r - INFO: - Episode: 160/800, Reward: 0.00: Epislon: 0.173 -2022-10-30 01:45:05 - r - INFO: - Episode: 161/800, Reward: 0.00: Epislon: 0.169 -2022-10-30 01:45:05 - r - INFO: - Episode: 162/800, Reward: 0.00: Epislon: 0.167 -2022-10-30 01:45:05 - r - INFO: - Episode: 163/800, Reward: 0.00: Epislon: 0.165 -2022-10-30 01:45:05 - r - INFO: - Episode: 164/800, Reward: 0.00: Epislon: 0.165 -2022-10-30 01:45:05 - r - INFO: - Episode: 165/800, Reward: 0.00: Epislon: 0.163 -2022-10-30 01:45:05 - r - INFO: - Episode: 166/800, Reward: 0.00: Epislon: 0.163 -2022-10-30 01:45:05 - r - INFO: - Episode: 167/800, Reward: 0.00: Epislon: 0.162 -2022-10-30 01:45:05 - r - INFO: - Episode: 168/800, Reward: 0.00: Epislon: 0.161 -2022-10-30 01:45:05 - r - INFO: - Episode: 169/800, Reward: 0.00: Epislon: 0.160 -2022-10-30 01:45:05 - r - INFO: - Episode: 170/800, Reward: 0.00: Epislon: 0.159 -2022-10-30 01:45:05 - r - INFO: - Episode: 171/800, Reward: 0.00: Epislon: 0.158 -2022-10-30 01:45:05 - r - INFO: - Episode: 172/800, Reward: 0.00: Epislon: 0.155 -2022-10-30 01:45:05 - r - INFO: - Episode: 173/800, Reward: 0.00: Epislon: 0.151 -2022-10-30 01:45:05 - r - INFO: - Episode: 174/800, Reward: 0.00: Epislon: 0.149 -2022-10-30 01:45:05 - r - INFO: - Episode: 175/800, Reward: 0.00: Epislon: 0.148 -2022-10-30 01:45:05 - r - INFO: - Episode: 176/800, Reward: 0.00: Epislon: 0.148 -2022-10-30 01:45:05 - r - INFO: - Episode: 177/800, Reward: 0.00: Epislon: 0.148 -2022-10-30 01:45:05 - r - INFO: - Episode: 178/800, Reward: 0.00: Epislon: 0.147 -2022-10-30 01:45:05 - r - INFO: - Episode: 179/800, Reward: 0.00: Epislon: 0.146 -2022-10-30 01:45:05 - r - INFO: - Episode: 180/800, Reward: 0.00: Epislon: 0.146 -2022-10-30 01:45:05 - r - INFO: - Episode: 181/800, Reward: 0.00: Epislon: 0.145 -2022-10-30 01:45:05 - r - INFO: - Episode: 182/800, Reward: 0.00: Epislon: 0.144 -2022-10-30 01:45:05 - r - INFO: - Episode: 183/800, Reward: 0.00: Epislon: 0.140 -2022-10-30 01:45:05 - r - INFO: - Episode: 184/800, Reward: 0.00: Epislon: 0.139 -2022-10-30 01:45:05 - r - INFO: - Episode: 185/800, Reward: 0.00: Epislon: 0.138 -2022-10-30 01:45:05 - r - INFO: - Episode: 186/800, Reward: 0.00: Epislon: 0.137 -2022-10-30 01:45:05 - r - INFO: - Episode: 187/800, Reward: 0.00: Epislon: 0.137 -2022-10-30 01:45:05 - r - INFO: - Episode: 188/800, Reward: 0.00: Epislon: 0.134 -2022-10-30 01:45:05 - r - INFO: - Episode: 189/800, Reward: 0.00: Epislon: 0.134 -2022-10-30 01:45:05 - r - INFO: - Episode: 190/800, Reward: 0.00: Epislon: 0.133 -2022-10-30 01:45:05 - r - INFO: - Episode: 191/800, Reward: 0.00: Epislon: 0.133 -2022-10-30 01:45:05 - r - INFO: - Episode: 192/800, Reward: 0.00: Epislon: 0.132 -2022-10-30 01:45:05 - r - INFO: - Episode: 193/800, Reward: 0.00: Epislon: 0.131 -2022-10-30 01:45:05 - r - INFO: - Episode: 194/800, Reward: 0.00: Epislon: 0.131 -2022-10-30 01:45:05 - r - INFO: - Episode: 195/800, Reward: 0.00: Epislon: 0.130 -2022-10-30 01:45:05 - r - INFO: - Episode: 196/800, Reward: 0.00: Epislon: 0.129 -2022-10-30 01:45:05 - r - INFO: - Episode: 197/800, Reward: 0.00: Epislon: 0.129 -2022-10-30 01:45:05 - r - INFO: - Episode: 198/800, Reward: 0.00: Epislon: 0.126 -2022-10-30 01:45:05 - r - INFO: - Episode: 199/800, Reward: 0.00: Epislon: 0.123 -2022-10-30 01:45:05 - r - INFO: - Episode: 200/800, Reward: 0.00: Epislon: 0.123 -2022-10-30 01:45:05 - r - INFO: - Episode: 201/800, Reward: 0.00: Epislon: 0.122 -2022-10-30 01:45:05 - r - INFO: - Episode: 202/800, Reward: 0.00: Epislon: 0.122 -2022-10-30 01:45:05 - r - INFO: - Episode: 203/800, Reward: 0.00: Epislon: 0.122 -2022-10-30 01:45:05 - r - INFO: - Episode: 204/800, Reward: 0.00: Epislon: 0.121 -2022-10-30 01:45:05 - r - INFO: - Episode: 205/800, Reward: 0.00: Epislon: 0.119 -2022-10-30 01:45:05 - r - INFO: - Episode: 206/800, Reward: 0.00: Epislon: 0.119 -2022-10-30 01:45:05 - r - INFO: - Episode: 207/800, Reward: 0.00: Epislon: 0.119 -2022-10-30 01:45:05 - r - INFO: - Episode: 208/800, Reward: 0.00: Epislon: 0.118 -2022-10-30 01:45:05 - r - INFO: - Episode: 209/800, Reward: 0.00: Epislon: 0.118 -2022-10-30 01:45:05 - r - INFO: - Episode: 210/800, Reward: 0.00: Epislon: 0.118 -2022-10-30 01:45:05 - r - INFO: - Episode: 211/800, Reward: 0.00: Epislon: 0.116 -2022-10-30 01:45:05 - r - INFO: - Episode: 212/800, Reward: 0.00: Epislon: 0.115 -2022-10-30 01:45:05 - r - INFO: - Episode: 213/800, Reward: 0.00: Epislon: 0.115 -2022-10-30 01:45:05 - r - INFO: - Episode: 214/800, Reward: 0.00: Epislon: 0.114 -2022-10-30 01:45:05 - r - INFO: - Episode: 215/800, Reward: 0.00: Epislon: 0.113 -2022-10-30 01:45:05 - r - INFO: - Episode: 216/800, Reward: 0.00: Epislon: 0.113 -2022-10-30 01:45:05 - r - INFO: - Episode: 217/800, Reward: 0.00: Epislon: 0.112 -2022-10-30 01:45:05 - r - INFO: - Episode: 218/800, Reward: 0.00: Epislon: 0.111 -2022-10-30 01:45:05 - r - INFO: - Episode: 219/800, Reward: 0.00: Epislon: 0.111 -2022-10-30 01:45:05 - r - INFO: - Episode: 220/800, Reward: 0.00: Epislon: 0.111 -2022-10-30 01:45:05 - r - INFO: - Episode: 221/800, Reward: 0.00: Epislon: 0.110 -2022-10-30 01:45:05 - r - INFO: - Episode: 222/800, Reward: 0.00: Epislon: 0.110 -2022-10-30 01:45:05 - r - INFO: - Episode: 223/800, Reward: 0.00: Epislon: 0.109 -2022-10-30 01:45:05 - r - INFO: - Episode: 224/800, Reward: 0.00: Epislon: 0.108 -2022-10-30 01:45:05 - r - INFO: - Episode: 225/800, Reward: 0.00: Epislon: 0.108 -2022-10-30 01:45:05 - r - INFO: - Episode: 226/800, Reward: 0.00: Epislon: 0.108 -2022-10-30 01:45:05 - r - INFO: - Episode: 227/800, Reward: 0.00: Epislon: 0.108 -2022-10-30 01:45:05 - r - INFO: - Episode: 228/800, Reward: 0.00: Epislon: 0.107 -2022-10-30 01:45:05 - r - INFO: - Episode: 229/800, Reward: 0.00: Epislon: 0.107 -2022-10-30 01:45:05 - r - INFO: - Episode: 230/800, Reward: 0.00: Epislon: 0.107 -2022-10-30 01:45:05 - r - INFO: - Episode: 231/800, Reward: 0.00: Epislon: 0.107 -2022-10-30 01:45:05 - r - INFO: - Episode: 232/800, Reward: 0.00: Epislon: 0.106 -2022-10-30 01:45:05 - r - INFO: - Episode: 233/800, Reward: 0.00: Epislon: 0.106 -2022-10-30 01:45:05 - r - INFO: - Episode: 234/800, Reward: 0.00: Epislon: 0.106 -2022-10-30 01:45:05 - r - INFO: - Episode: 235/800, Reward: 0.00: Epislon: 0.105 -2022-10-30 01:45:05 - r - INFO: - Episode: 236/800, Reward: 0.00: Epislon: 0.105 -2022-10-30 01:45:05 - r - INFO: - Episode: 237/800, Reward: 0.00: Epislon: 0.105 -2022-10-30 01:45:05 - r - INFO: - Episode: 238/800, Reward: 0.00: Epislon: 0.105 -2022-10-30 01:45:05 - r - INFO: - Episode: 239/800, Reward: 0.00: Epislon: 0.104 -2022-10-30 01:45:05 - r - INFO: - Episode: 240/800, Reward: 0.00: Epislon: 0.104 -2022-10-30 01:45:05 - r - INFO: - Episode: 241/800, Reward: 0.00: Epislon: 0.104 -2022-10-30 01:45:05 - r - INFO: - Episode: 242/800, Reward: 0.00: Epislon: 0.103 -2022-10-30 01:45:05 - r - INFO: - Episode: 243/800, Reward: 0.00: Epislon: 0.103 -2022-10-30 01:45:05 - r - INFO: - Episode: 244/800, Reward: 0.00: Epislon: 0.103 -2022-10-30 01:45:05 - r - INFO: - Episode: 245/800, Reward: 0.00: Epislon: 0.103 -2022-10-30 01:45:05 - r - INFO: - Episode: 246/800, Reward: 0.00: Epislon: 0.103 -2022-10-30 01:45:05 - r - INFO: - Episode: 247/800, Reward: 0.00: Epislon: 0.103 -2022-10-30 01:45:05 - r - INFO: - Episode: 248/800, Reward: 0.00: Epislon: 0.103 -2022-10-30 01:45:05 - r - INFO: - Episode: 249/800, Reward: 0.00: Epislon: 0.103 -2022-10-30 01:45:05 - r - INFO: - Episode: 250/800, Reward: 0.00: Epislon: 0.103 -2022-10-30 01:45:05 - r - INFO: - Episode: 251/800, Reward: 0.00: Epislon: 0.103 -2022-10-30 01:45:05 - r - INFO: - Episode: 252/800, Reward: 0.00: Epislon: 0.102 -2022-10-30 01:45:05 - r - INFO: - Episode: 253/800, Reward: 0.00: Epislon: 0.102 -2022-10-30 01:45:05 - r - INFO: - Episode: 254/800, Reward: 0.00: Epislon: 0.102 -2022-10-30 01:45:05 - r - INFO: - Episode: 255/800, Reward: 0.00: Epislon: 0.102 -2022-10-30 01:45:05 - r - INFO: - Episode: 256/800, Reward: 0.00: Epislon: 0.102 -2022-10-30 01:45:05 - r - INFO: - Episode: 257/800, Reward: 0.00: Epislon: 0.102 -2022-10-30 01:45:05 - r - INFO: - Episode: 258/800, Reward: 0.00: Epislon: 0.102 -2022-10-30 01:45:05 - r - INFO: - Episode: 259/800, Reward: 0.00: Epislon: 0.102 -2022-10-30 01:45:05 - r - INFO: - Episode: 260/800, Reward: 0.00: Epislon: 0.102 -2022-10-30 01:45:05 - r - INFO: - Episode: 261/800, Reward: 0.00: Epislon: 0.102 -2022-10-30 01:45:05 - r - INFO: - Episode: 262/800, Reward: 0.00: Epislon: 0.102 -2022-10-30 01:45:05 - r - INFO: - Episode: 263/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 264/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 265/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 266/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 267/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 268/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 269/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 270/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 271/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 272/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 273/800, Reward: 1.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 274/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 275/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 276/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 277/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 278/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 279/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 280/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 281/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 282/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 283/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 284/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 285/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 286/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 287/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 288/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 289/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 290/800, Reward: 0.00: Epislon: 0.101 -2022-10-30 01:45:05 - r - INFO: - Episode: 291/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 292/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 293/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 294/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 295/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 296/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 297/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 298/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 299/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 300/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 301/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 302/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 303/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 304/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 305/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 306/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 307/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 308/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 309/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 310/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 311/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 312/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 313/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 314/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 315/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 316/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 317/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 318/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 319/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 320/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 321/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 322/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 323/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 324/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 325/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 326/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 327/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 328/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 329/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 330/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 331/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 332/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 333/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 334/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 335/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 336/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 337/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 338/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 339/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 340/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 341/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 342/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 343/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 344/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 345/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 346/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 347/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 348/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 349/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 350/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 351/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 352/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 353/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 354/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 355/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 356/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 357/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 358/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 359/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 360/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 361/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 362/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 363/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 364/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 365/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 366/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 367/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 368/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 369/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 370/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 371/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 372/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 373/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 374/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 375/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 376/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 377/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 378/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 379/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 380/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 381/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 382/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 383/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 384/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 385/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 386/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 387/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 388/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 389/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 390/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 391/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 392/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 393/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 394/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 395/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 396/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 397/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 398/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 399/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 400/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 401/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 402/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 403/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 404/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 405/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 406/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 407/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 408/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 409/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 410/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 411/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 412/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 413/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 414/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 415/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 416/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 417/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 418/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 419/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 420/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 421/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 422/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 423/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 424/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 425/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 426/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 427/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 428/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 429/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 430/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 431/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 432/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 433/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 434/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 435/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 436/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 437/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 438/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 439/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 440/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 441/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 442/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 443/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 444/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 445/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 446/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 447/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 448/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 449/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 450/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 451/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 452/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 453/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 454/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 455/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 456/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 457/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 458/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 459/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 460/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 461/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 462/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 463/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 464/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 465/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 466/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 467/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 468/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 469/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 470/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 471/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 472/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 473/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 474/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 475/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 476/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 477/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 478/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 479/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 480/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 481/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 482/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 483/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 484/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 485/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 486/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 487/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 488/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 489/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 490/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 491/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 492/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 493/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 494/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 495/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 496/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 497/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 498/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 499/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 500/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 501/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 502/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 503/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 504/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 505/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 506/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 507/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 508/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 509/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 510/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 511/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 512/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 513/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 514/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 515/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 516/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 517/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 518/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 519/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 520/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 521/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 522/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 523/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 524/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 525/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 526/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 527/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 528/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 529/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 530/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 531/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 532/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 533/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 534/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 535/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 536/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 537/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 538/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 539/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 540/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 541/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 542/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 543/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 544/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 545/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 546/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 547/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 548/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 549/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 550/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 551/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 552/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 553/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 554/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 555/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 556/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 557/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 558/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 559/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 560/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 561/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 562/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 563/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 564/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 565/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 566/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 567/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 568/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 569/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 570/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 571/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 572/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 573/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 574/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 575/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 576/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 577/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 578/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 579/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 580/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 581/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 582/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 583/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 584/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 585/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 586/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 587/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 588/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 589/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 590/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 591/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 592/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 593/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 594/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 595/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 596/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 597/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 598/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 599/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 600/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 601/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 602/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 603/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 604/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 605/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 606/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 607/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 608/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 609/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 610/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 611/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 612/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 613/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 614/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 615/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 616/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 617/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 618/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 619/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 620/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 621/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 622/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 623/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 624/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 625/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 626/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 627/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 628/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 629/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 630/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 631/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 632/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 633/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 634/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 635/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 636/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 637/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 638/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 639/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 640/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 641/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 642/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 643/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 644/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 645/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 646/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 647/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 648/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 649/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 650/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 651/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 652/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 653/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 654/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 655/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 656/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 657/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 658/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 659/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 660/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 661/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 662/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 663/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 664/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 665/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 666/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 667/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 668/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 669/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 670/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 671/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 672/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 673/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 674/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 675/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 676/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 677/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 678/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 679/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 680/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 681/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 682/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 683/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 684/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 685/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 686/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 687/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 688/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 689/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 690/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 691/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 692/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 693/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 694/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 695/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 696/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 697/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 698/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 699/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 700/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 701/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 702/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 703/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 704/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 705/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 706/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 707/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 708/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 709/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 710/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 711/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 712/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 713/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 714/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 715/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 716/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 717/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 718/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 719/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 720/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 721/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 722/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 723/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 724/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 725/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 726/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 727/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 728/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 729/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 730/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 731/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 732/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 733/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 734/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 735/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 736/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 737/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 738/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 739/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 740/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 741/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 742/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 743/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 744/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 745/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 746/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 747/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 748/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 749/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 750/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 751/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 752/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 753/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 754/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 755/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 756/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 757/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 758/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 759/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 760/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 761/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 762/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 763/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 764/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 765/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 766/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 767/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 768/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 769/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 770/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 771/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 772/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 773/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 774/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 775/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 776/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 777/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 778/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 779/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 780/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 781/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 782/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 783/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 784/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 785/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 786/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 787/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 788/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 789/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 790/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 791/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 792/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 793/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 794/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 795/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 796/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 797/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 798/800, Reward: 0.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 799/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Episode: 800/800, Reward: 1.00: Epislon: 0.100 -2022-10-30 01:45:05 - r - INFO: - Finish training! diff --git a/projects/codes/QLearning/Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504/models/Qleaning_model.pkl b/projects/codes/QLearning/Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504/models/Qleaning_model.pkl deleted file mode 100644 index 41a5a05..0000000 Binary files a/projects/codes/QLearning/Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504/models/Qleaning_model.pkl and /dev/null differ diff --git a/projects/codes/QLearning/Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504/results/learning_curve.png b/projects/codes/QLearning/Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504/results/learning_curve.png deleted file mode 100644 index ad789b7..0000000 Binary files a/projects/codes/QLearning/Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/QLearning/Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504/results/res.csv b/projects/codes/QLearning/Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504/results/res.csv deleted file mode 100644 index 335c1d8..0000000 --- a/projects/codes/QLearning/Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504/results/res.csv +++ /dev/null @@ -1,801 +0,0 @@ -episodes,rewards,steps -0,0.0,20 -1,0.0,14 -2,0.0,13 -3,0.0,9 -4,0.0,10 -5,0.0,6 -6,0.0,11 -7,0.0,6 -8,0.0,3 -9,0.0,9 -10,0.0,11 -11,0.0,22 -12,0.0,5 -13,0.0,16 -14,0.0,4 -15,0.0,9 -16,0.0,18 -17,0.0,2 -18,0.0,4 -19,0.0,8 -20,0.0,7 -21,0.0,4 -22,0.0,22 -23,0.0,15 -24,0.0,5 -25,0.0,16 -26,0.0,7 -27,0.0,19 -28,0.0,22 -29,0.0,16 -30,0.0,11 -31,0.0,22 -32,0.0,28 -33,0.0,23 -34,0.0,4 -35,0.0,11 -36,0.0,8 -37,0.0,15 -38,0.0,5 -39,0.0,7 -40,0.0,9 -41,0.0,4 -42,0.0,3 -43,0.0,6 -44,0.0,41 -45,0.0,9 -46,0.0,23 -47,0.0,3 -48,1.0,38 -49,0.0,29 -50,0.0,17 -51,0.0,4 -52,0.0,2 -53,0.0,25 -54,0.0,6 -55,0.0,2 -56,0.0,30 -57,0.0,6 -58,0.0,7 -59,0.0,11 -60,0.0,9 -61,0.0,8 -62,0.0,23 -63,0.0,10 -64,0.0,3 -65,0.0,5 -66,0.0,7 -67,0.0,18 -68,0.0,8 -69,0.0,26 -70,0.0,6 -71,0.0,14 -72,0.0,4 -73,0.0,25 -74,0.0,21 -75,0.0,13 -76,0.0,4 -77,0.0,29 -78,0.0,21 -79,0.0,6 -80,0.0,6 -81,0.0,11 -82,0.0,21 -83,0.0,9 -84,0.0,9 -85,0.0,7 -86,0.0,48 -87,0.0,23 -88,0.0,160 -89,0.0,7 -90,0.0,10 -91,0.0,24 -92,0.0,4 -93,0.0,7 -94,0.0,17 -95,0.0,87 -96,0.0,28 -97,0.0,7 -98,0.0,5 -99,0.0,12 -100,0.0,14 -101,0.0,6 -102,0.0,13 -103,0.0,93 -104,0.0,4 -105,0.0,50 -106,0.0,8 -107,0.0,12 -108,0.0,43 -109,0.0,30 -110,0.0,15 -111,0.0,19 -112,0.0,182 -113,0.0,40 -114,0.0,88 -115,0.0,19 -116,0.0,30 -117,0.0,27 -118,0.0,5 -119,0.0,87 -120,0.0,9 -121,0.0,64 -122,0.0,27 -123,0.0,68 -124,0.0,81 -125,0.0,86 -126,0.0,200 -127,0.0,27 -128,0.0,41 -129,0.0,70 -130,0.0,27 -131,0.0,6 -132,0.0,18 -133,0.0,38 -134,0.0,26 -135,0.0,36 -136,0.0,3 -137,0.0,61 -138,0.0,105 -139,0.0,38 -140,0.0,18 -141,0.0,33 -142,0.0,29 -143,0.0,49 -144,0.0,88 -145,0.0,22 -146,0.0,65 -147,0.0,36 -148,0.0,30 -149,0.0,58 -150,0.0,43 -151,0.0,53 -152,0.0,43 -153,0.0,13 -154,0.0,8 -155,0.0,39 -156,0.0,29 -157,0.0,26 -158,0.0,60 -159,0.0,153 -160,0.0,116 -161,0.0,53 -162,0.0,54 -163,0.0,8 -164,0.0,58 -165,0.0,3 -166,0.0,47 -167,0.0,16 -168,0.0,21 -169,0.0,44 -170,0.0,29 -171,0.0,104 -172,0.0,158 -173,0.0,83 -174,0.0,26 -175,0.0,24 -176,0.0,10 -177,0.0,12 -178,0.0,40 -179,0.0,25 -180,0.0,18 -181,0.0,60 -182,0.0,200 -183,0.0,24 -184,0.0,56 -185,0.0,71 -186,0.0,19 -187,0.0,118 -188,0.0,26 -189,0.0,41 -190,0.0,41 -191,0.0,60 -192,0.0,31 -193,0.0,34 -194,0.0,35 -195,0.0,59 -196,0.0,51 -197,0.0,200 -198,0.0,200 -199,0.0,37 -200,0.0,68 -201,0.0,40 -202,0.0,17 -203,0.0,79 -204,0.0,126 -205,0.0,61 -206,0.0,25 -207,0.0,18 -208,0.0,27 -209,0.0,13 -210,0.0,187 -211,0.0,160 -212,0.0,32 -213,0.0,108 -214,0.0,164 -215,0.0,17 -216,0.0,82 -217,0.0,194 -218,0.0,7 -219,0.0,36 -220,0.0,156 -221,0.0,17 -222,0.0,183 -223,0.0,200 -224,0.0,43 -225,0.0,87 -226,0.0,42 -227,0.0,80 -228,0.0,54 -229,0.0,82 -230,0.0,97 -231,0.0,65 -232,0.0,83 -233,0.0,159 -234,0.0,178 -235,0.0,104 -236,0.0,21 -237,0.0,118 -238,0.0,80 -239,0.0,170 -240,0.0,94 -241,0.0,200 -242,0.0,37 -243,0.0,11 -244,0.0,31 -245,0.0,134 -246,0.0,32 -247,0.0,58 -248,0.0,38 -249,0.0,28 -250,0.0,159 -251,0.0,182 -252,0.0,51 -253,0.0,25 -254,0.0,73 -255,0.0,56 -256,0.0,55 -257,0.0,38 -258,0.0,200 -259,0.0,92 -260,0.0,200 -261,0.0,119 -262,0.0,100 -263,0.0,84 -264,0.0,24 -265,0.0,17 -266,0.0,159 -267,0.0,25 -268,0.0,73 -269,0.0,130 -270,0.0,111 -271,0.0,65 -272,1.0,58 -273,0.0,47 -274,0.0,48 -275,0.0,13 -276,0.0,100 -277,0.0,38 -278,0.0,111 -279,0.0,200 -280,0.0,26 -281,0.0,38 -282,0.0,83 -283,0.0,42 -284,0.0,199 -285,0.0,83 -286,0.0,28 -287,0.0,46 -288,0.0,200 -289,0.0,62 -290,0.0,123 -291,0.0,91 -292,0.0,53 -293,0.0,19 -294,0.0,26 -295,0.0,93 -296,0.0,38 -297,0.0,22 -298,0.0,43 -299,0.0,163 -300,0.0,25 -301,0.0,59 -302,0.0,71 -303,0.0,20 -304,0.0,115 -305,0.0,200 -306,0.0,48 -307,0.0,66 -308,0.0,58 -309,0.0,129 -310,0.0,122 -311,0.0,47 -312,0.0,60 -313,0.0,79 -314,1.0,137 -315,0.0,27 -316,1.0,93 -317,0.0,46 -318,1.0,83 -319,1.0,8 -320,1.0,6 -321,1.0,6 -322,0.0,4 -323,1.0,6 -324,0.0,2 -325,1.0,6 -326,1.0,6 -327,1.0,6 -328,1.0,6 -329,1.0,8 -330,0.0,5 -331,1.0,6 -332,1.0,7 -333,0.0,5 -334,1.0,6 -335,1.0,6 -336,1.0,8 -337,1.0,6 -338,1.0,6 -339,1.0,6 -340,1.0,7 -341,1.0,6 -342,1.0,6 -343,0.0,3 -344,1.0,7 -345,0.0,4 -346,1.0,6 -347,1.0,6 -348,1.0,7 -349,1.0,6 -350,1.0,6 -351,1.0,7 -352,1.0,7 -353,1.0,7 -354,1.0,6 -355,1.0,6 -356,1.0,6 -357,1.0,6 -358,1.0,6 -359,1.0,6 -360,1.0,6 -361,1.0,7 -362,0.0,4 -363,1.0,8 -364,1.0,8 -365,1.0,7 -366,1.0,6 -367,1.0,8 -368,1.0,6 -369,1.0,6 -370,1.0,7 -371,1.0,6 -372,1.0,6 -373,1.0,8 -374,1.0,7 -375,1.0,6 -376,1.0,6 -377,0.0,3 -378,1.0,11 -379,1.0,6 -380,1.0,8 -381,0.0,2 -382,1.0,6 -383,1.0,6 -384,1.0,6 -385,1.0,6 -386,1.0,8 -387,1.0,6 -388,1.0,7 -389,1.0,6 -390,1.0,7 -391,1.0,6 -392,1.0,8 -393,0.0,2 -394,1.0,6 -395,1.0,7 -396,1.0,6 -397,1.0,6 -398,1.0,10 -399,1.0,7 -400,1.0,6 -401,1.0,6 -402,1.0,6 -403,1.0,6 -404,1.0,6 -405,1.0,7 -406,0.0,4 -407,1.0,7 -408,1.0,6 -409,1.0,8 -410,0.0,3 -411,1.0,6 -412,1.0,6 -413,1.0,6 -414,1.0,6 -415,0.0,2 -416,1.0,6 -417,1.0,6 -418,1.0,6 -419,1.0,6 -420,1.0,6 -421,1.0,7 -422,1.0,6 -423,1.0,6 -424,1.0,7 -425,1.0,6 -426,1.0,6 -427,1.0,6 -428,1.0,6 -429,1.0,6 -430,1.0,6 -431,1.0,6 -432,1.0,8 -433,1.0,6 -434,1.0,8 -435,1.0,7 -436,1.0,6 -437,0.0,3 -438,1.0,6 -439,1.0,7 -440,1.0,6 -441,1.0,6 -442,1.0,6 -443,1.0,10 -444,1.0,6 -445,1.0,6 -446,1.0,6 -447,1.0,6 -448,1.0,10 -449,1.0,6 -450,1.0,8 -451,1.0,8 -452,1.0,7 -453,1.0,6 -454,0.0,5 -455,0.0,2 -456,1.0,8 -457,1.0,6 -458,1.0,10 -459,1.0,6 -460,1.0,8 -461,1.0,10 -462,1.0,6 -463,1.0,6 -464,1.0,6 -465,1.0,10 -466,1.0,6 -467,0.0,4 -468,1.0,6 -469,1.0,6 -470,1.0,6 -471,1.0,15 -472,1.0,6 -473,1.0,6 -474,1.0,6 -475,1.0,6 -476,1.0,6 -477,1.0,6 -478,1.0,8 -479,1.0,6 -480,1.0,7 -481,1.0,6 -482,1.0,6 -483,1.0,8 -484,1.0,6 -485,1.0,6 -486,1.0,8 -487,1.0,8 -488,1.0,6 -489,1.0,6 -490,1.0,6 -491,1.0,10 -492,1.0,6 -493,1.0,6 -494,1.0,6 -495,1.0,6 -496,1.0,6 -497,1.0,6 -498,1.0,6 -499,1.0,8 -500,1.0,8 -501,1.0,6 -502,1.0,6 -503,0.0,2 -504,1.0,6 -505,1.0,6 -506,1.0,6 -507,1.0,8 -508,1.0,6 -509,1.0,6 -510,1.0,6 -511,1.0,6 -512,1.0,6 -513,1.0,6 -514,1.0,6 -515,1.0,6 -516,1.0,6 -517,1.0,7 -518,0.0,3 -519,1.0,7 -520,1.0,6 -521,1.0,6 -522,1.0,6 -523,0.0,2 -524,1.0,6 -525,1.0,8 -526,1.0,6 -527,1.0,6 -528,1.0,6 -529,1.0,6 -530,1.0,9 -531,1.0,6 -532,1.0,6 -533,1.0,6 -534,1.0,6 -535,1.0,6 -536,1.0,6 -537,1.0,9 -538,1.0,7 -539,0.0,4 -540,1.0,6 -541,1.0,8 -542,1.0,11 -543,1.0,6 -544,1.0,6 -545,1.0,6 -546,1.0,6 -547,1.0,6 -548,1.0,8 -549,1.0,6 -550,1.0,6 -551,1.0,8 -552,1.0,7 -553,1.0,6 -554,1.0,8 -555,1.0,6 -556,0.0,5 -557,1.0,9 -558,1.0,8 -559,1.0,8 -560,1.0,6 -561,1.0,8 -562,1.0,8 -563,1.0,6 -564,0.0,5 -565,0.0,3 -566,0.0,2 -567,1.0,8 -568,1.0,6 -569,1.0,6 -570,1.0,6 -571,1.0,6 -572,1.0,6 -573,1.0,6 -574,1.0,6 -575,1.0,6 -576,1.0,6 -577,1.0,6 -578,1.0,6 -579,1.0,6 -580,1.0,6 -581,1.0,6 -582,0.0,2 -583,1.0,6 -584,0.0,4 -585,1.0,6 -586,1.0,6 -587,1.0,6 -588,1.0,6 -589,1.0,6 -590,1.0,8 -591,0.0,5 -592,1.0,6 -593,1.0,6 -594,1.0,6 -595,1.0,6 -596,1.0,6 -597,1.0,6 -598,0.0,3 -599,1.0,6 -600,1.0,6 -601,1.0,6 -602,0.0,2 -603,1.0,6 -604,0.0,4 -605,1.0,6 -606,1.0,6 -607,1.0,6 -608,1.0,6 -609,1.0,8 -610,1.0,6 -611,1.0,7 -612,1.0,6 -613,1.0,7 -614,1.0,6 -615,0.0,2 -616,1.0,6 -617,1.0,6 -618,0.0,5 -619,0.0,3 -620,0.0,3 -621,1.0,6 -622,0.0,5 -623,1.0,8 -624,1.0,8 -625,1.0,6 -626,1.0,6 -627,1.0,7 -628,1.0,6 -629,1.0,6 -630,1.0,6 -631,1.0,6 -632,1.0,6 -633,1.0,8 -634,0.0,2 -635,1.0,6 -636,1.0,6 -637,1.0,6 -638,1.0,6 -639,1.0,6 -640,1.0,6 -641,1.0,6 -642,1.0,8 -643,1.0,6 -644,1.0,8 -645,1.0,6 -646,1.0,6 -647,1.0,8 -648,1.0,8 -649,0.0,5 -650,0.0,4 -651,0.0,4 -652,1.0,6 -653,1.0,6 -654,1.0,6 -655,1.0,6 -656,1.0,8 -657,1.0,6 -658,0.0,4 -659,1.0,6 -660,1.0,8 -661,1.0,6 -662,1.0,6 -663,1.0,6 -664,1.0,6 -665,1.0,6 -666,1.0,6 -667,1.0,6 -668,1.0,8 -669,1.0,8 -670,1.0,6 -671,1.0,8 -672,1.0,9 -673,1.0,6 -674,1.0,6 -675,1.0,6 -676,1.0,6 -677,1.0,10 -678,1.0,6 -679,1.0,6 -680,1.0,6 -681,1.0,11 -682,1.0,10 -683,1.0,8 -684,1.0,6 -685,1.0,6 -686,1.0,6 -687,0.0,5 -688,1.0,6 -689,0.0,2 -690,1.0,9 -691,1.0,6 -692,1.0,8 -693,1.0,7 -694,1.0,6 -695,1.0,6 -696,1.0,7 -697,0.0,3 -698,1.0,7 -699,0.0,2 -700,1.0,6 -701,1.0,6 -702,1.0,8 -703,1.0,8 -704,1.0,6 -705,1.0,6 -706,0.0,2 -707,1.0,8 -708,1.0,6 -709,1.0,8 -710,1.0,6 -711,1.0,6 -712,1.0,9 -713,1.0,6 -714,1.0,8 -715,1.0,11 -716,1.0,6 -717,1.0,6 -718,1.0,6 -719,1.0,6 -720,1.0,8 -721,1.0,6 -722,1.0,6 -723,1.0,6 -724,0.0,5 -725,1.0,6 -726,1.0,6 -727,1.0,6 -728,1.0,6 -729,1.0,6 -730,1.0,7 -731,1.0,6 -732,1.0,6 -733,1.0,6 -734,1.0,6 -735,1.0,10 -736,1.0,6 -737,1.0,6 -738,1.0,6 -739,1.0,6 -740,1.0,6 -741,1.0,7 -742,1.0,6 -743,1.0,8 -744,1.0,7 -745,1.0,6 -746,1.0,6 -747,1.0,14 -748,1.0,6 -749,1.0,6 -750,1.0,12 -751,1.0,6 -752,1.0,6 -753,1.0,6 -754,1.0,6 -755,1.0,6 -756,1.0,6 -757,0.0,3 -758,1.0,6 -759,1.0,6 -760,1.0,6 -761,1.0,7 -762,1.0,6 -763,1.0,6 -764,1.0,6 -765,1.0,8 -766,0.0,2 -767,1.0,6 -768,1.0,6 -769,1.0,6 -770,1.0,6 -771,1.0,6 -772,1.0,6 -773,1.0,6 -774,1.0,6 -775,1.0,6 -776,0.0,4 -777,1.0,8 -778,1.0,6 -779,0.0,2 -780,1.0,10 -781,1.0,8 -782,1.0,6 -783,1.0,6 -784,1.0,6 -785,0.0,3 -786,1.0,6 -787,1.0,6 -788,0.0,6 -789,1.0,8 -790,1.0,6 -791,1.0,9 -792,1.0,6 -793,1.0,6 -794,1.0,8 -795,1.0,8 -796,1.0,6 -797,0.0,5 -798,1.0,6 -799,1.0,6 diff --git a/projects/codes/QLearning/Train_Racetrack-v0_QLearning_20221030-014833/config.yaml b/projects/codes/QLearning/Train_Racetrack-v0_QLearning_20221030-014833/config.yaml deleted file mode 100644 index d5b9c4c..0000000 --- a/projects/codes/QLearning/Train_Racetrack-v0_QLearning_20221030-014833/config.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: QLearning - device: cpu - env_name: Racetrack-v0 - load_checkpoint: false - load_path: Train_CartPole-v1_DQN_20221026-054757 - max_steps: 200 - mode: train - save_fig: true - seed: 10 - show_fig: false - test_eps: 20 - train_eps: 400 -algo_cfg: - epsilon_decay: 300 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.9 - lr: 0.1 diff --git a/projects/codes/QLearning/Train_Racetrack-v0_QLearning_20221030-014833/logs/log.txt b/projects/codes/QLearning/Train_Racetrack-v0_QLearning_20221030-014833/logs/log.txt deleted file mode 100644 index e737550..0000000 --- a/projects/codes/QLearning/Train_Racetrack-v0_QLearning_20221030-014833/logs/log.txt +++ /dev/null @@ -1,404 +0,0 @@ -2022-10-30 01:48:33 - r - INFO: - n_states: 4, n_actions: 9 -2022-10-30 01:48:33 - r - INFO: - Start training! -2022-10-30 01:48:33 - r - INFO: - Env: Racetrack-v0, Algorithm: QLearning, Device: cpu -2022-10-30 01:48:33 - r - INFO: - Episode: 1/400, Reward: -850.00: Epislon: 0.493 -2022-10-30 01:48:33 - r - INFO: - Episode: 2/400, Reward: -780.00: Epislon: 0.258 -2022-10-30 01:48:33 - r - INFO: - Episode: 3/400, Reward: -730.00: Epislon: 0.137 -2022-10-30 01:48:33 - r - INFO: - Episode: 4/400, Reward: -650.00: Epislon: 0.075 -2022-10-30 01:48:33 - r - INFO: - Episode: 5/400, Reward: -540.00: Epislon: 0.044 -2022-10-30 01:48:33 - r - INFO: - Episode: 6/400, Reward: -640.00: Epislon: 0.027 -2022-10-30 01:48:34 - r - INFO: - Episode: 7/400, Reward: -570.00: Epislon: 0.019 -2022-10-30 01:48:34 - r - INFO: - Episode: 8/400, Reward: -570.00: Epislon: 0.015 -2022-10-30 01:48:34 - r - INFO: - Episode: 9/400, Reward: -550.00: Epislon: 0.012 -2022-10-30 01:48:34 - r - INFO: - Episode: 10/400, Reward: -550.00: Epislon: 0.011 -2022-10-30 01:48:34 - r - INFO: - Episode: 11/400, Reward: -580.00: Epislon: 0.011 -2022-10-30 01:48:34 - r - INFO: - Episode: 12/400, Reward: -530.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 13/400, Reward: -580.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 14/400, Reward: -570.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 15/400, Reward: -550.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 16/400, Reward: -560.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 17/400, Reward: -550.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 18/400, Reward: -580.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 19/400, Reward: -520.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 20/400, Reward: -490.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 21/400, Reward: -480.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 22/400, Reward: -540.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 23/400, Reward: -550.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 24/400, Reward: -560.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 25/400, Reward: -510.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 26/400, Reward: -520.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 27/400, Reward: -480.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 28/400, Reward: -520.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 29/400, Reward: -480.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 30/400, Reward: -470.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 31/400, Reward: -540.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 32/400, Reward: -540.00: Epislon: 0.010 -2022-10-30 01:48:34 - r - INFO: - Episode: 33/400, Reward: -470.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 34/400, Reward: -540.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 35/400, Reward: -490.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 36/400, Reward: -530.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 37/400, Reward: -520.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 38/400, Reward: -510.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 39/400, Reward: -520.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 40/400, Reward: -510.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 41/400, Reward: -480.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 42/400, Reward: -510.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 43/400, Reward: -470.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 44/400, Reward: -490.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 45/400, Reward: -490.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 46/400, Reward: -490.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 47/400, Reward: -520.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 48/400, Reward: -530.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 49/400, Reward: -510.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 50/400, Reward: -460.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 51/400, Reward: -500.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 52/400, Reward: -470.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 53/400, Reward: -520.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 54/400, Reward: -490.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 55/400, Reward: -500.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 56/400, Reward: -460.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 57/400, Reward: -490.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 58/400, Reward: -510.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 59/400, Reward: -460.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 60/400, Reward: -530.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 61/400, Reward: -440.00: Epislon: 0.010 -2022-10-30 01:48:35 - r - INFO: - Episode: 62/400, Reward: -510.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 63/400, Reward: -520.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 64/400, Reward: -510.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 65/400, Reward: -460.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 66/400, Reward: -344.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 67/400, Reward: -500.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 68/400, Reward: -490.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 69/400, Reward: -490.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 70/400, Reward: -440.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 71/400, Reward: -77.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 72/400, Reward: -198.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 73/400, Reward: -440.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 74/400, Reward: -480.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 75/400, Reward: -354.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 76/400, Reward: -470.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 77/400, Reward: -480.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 78/400, Reward: -38.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 79/400, Reward: -460.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 80/400, Reward: -480.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 81/400, Reward: -490.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 82/400, Reward: -140.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 83/400, Reward: -102.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 84/400, Reward: -265.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 85/400, Reward: -145.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 86/400, Reward: -460.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 87/400, Reward: -500.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 88/400, Reward: -470.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 89/400, Reward: -325.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 90/400, Reward: -470.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 91/400, Reward: -376.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 92/400, Reward: -98.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 93/400, Reward: -130.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 94/400, Reward: -450.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 95/400, Reward: -146.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 96/400, Reward: 2.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 97/400, Reward: -18.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 98/400, Reward: -102.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 99/400, Reward: -163.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 100/400, Reward: -209.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 101/400, Reward: -460.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 102/400, Reward: -286.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 103/400, Reward: -189.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 104/400, Reward: -50.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 105/400, Reward: -398.00: Epislon: 0.010 -2022-10-30 01:48:36 - r - INFO: - Episode: 106/400, Reward: -72.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 107/400, Reward: -450.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 108/400, Reward: -125.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 109/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 110/400, Reward: -161.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 111/400, Reward: -408.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 112/400, Reward: -440.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 113/400, Reward: -188.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 114/400, Reward: -114.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 115/400, Reward: -415.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 116/400, Reward: -159.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 117/400, Reward: -234.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 118/400, Reward: -31.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 119/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 120/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 121/400, Reward: -63.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 122/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 123/400, Reward: -47.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 124/400, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 125/400, Reward: -49.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 126/400, Reward: -87.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 127/400, Reward: -2.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 128/400, Reward: -26.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 129/400, Reward: -238.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 130/400, Reward: -18.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 131/400, Reward: -235.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 132/400, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 133/400, Reward: -135.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 134/400, Reward: -20.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 135/400, Reward: -46.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 136/400, Reward: -66.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 137/400, Reward: -45.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 138/400, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 139/400, Reward: 1.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 140/400, Reward: -106.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 141/400, Reward: -112.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 142/400, Reward: -47.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 143/400, Reward: 1.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 144/400, Reward: -30.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 145/400, Reward: -147.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 146/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 147/400, Reward: -30.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 148/400, Reward: -167.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 149/400, Reward: 1.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 150/400, Reward: -72.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 151/400, Reward: -44.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 152/400, Reward: -76.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 153/400, Reward: -63.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 154/400, Reward: -34.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 155/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 156/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 157/400, Reward: -26.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 158/400, Reward: -80.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 159/400, Reward: -168.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 160/400, Reward: -164.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 161/400, Reward: 1.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 162/400, Reward: -19.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 163/400, Reward: -12.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 164/400, Reward: -44.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 165/400, Reward: -80.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 166/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 167/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 168/400, Reward: -29.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 169/400, Reward: -56.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 170/400, Reward: -47.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 171/400, Reward: -76.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 172/400, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 173/400, Reward: -145.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 174/400, Reward: -28.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 175/400, Reward: -63.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 176/400, Reward: -106.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 177/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 178/400, Reward: -28.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 179/400, Reward: -60.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 180/400, Reward: -49.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 181/400, Reward: -52.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 182/400, Reward: -84.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 183/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 184/400, Reward: -55.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 185/400, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 186/400, Reward: 1.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 187/400, Reward: -39.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 188/400, Reward: -47.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 189/400, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 190/400, Reward: -53.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 191/400, Reward: -50.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 192/400, Reward: -104.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 193/400, Reward: -253.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 194/400, Reward: -48.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 195/400, Reward: -190.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 196/400, Reward: -43.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 197/400, Reward: -35.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 198/400, Reward: 0.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 199/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 200/400, Reward: -11.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 201/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 202/400, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 203/400, Reward: -99.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 204/400, Reward: -22.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 205/400, Reward: -170.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 206/400, Reward: -109.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 207/400, Reward: -48.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 208/400, Reward: -275.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 209/400, Reward: -49.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 210/400, Reward: -147.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 211/400, Reward: -51.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 212/400, Reward: -67.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 213/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 214/400, Reward: -17.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 215/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 216/400, Reward: -69.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 217/400, Reward: -218.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 218/400, Reward: -63.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 219/400, Reward: -11.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 220/400, Reward: -34.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 221/400, Reward: -32.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 222/400, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 223/400, Reward: -26.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 224/400, Reward: -19.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 225/400, Reward: -148.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 226/400, Reward: -19.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 227/400, Reward: 1.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 228/400, Reward: -49.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 229/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 230/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 231/400, Reward: -223.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 232/400, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 233/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 234/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 235/400, Reward: 2.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 236/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 237/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 238/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 239/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 240/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 241/400, Reward: -44.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 242/400, Reward: -10.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 243/400, Reward: 2.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 244/400, Reward: -108.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 245/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 246/400, Reward: -27.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 247/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 248/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 249/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 250/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 251/400, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 252/400, Reward: -28.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 253/400, Reward: -112.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 254/400, Reward: -39.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 255/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 256/400, Reward: -48.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 257/400, Reward: -149.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 258/400, Reward: -27.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 259/400, Reward: -33.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 260/400, Reward: -30.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 261/400, Reward: -29.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 262/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 263/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 264/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 265/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 266/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 267/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 268/400, Reward: -52.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 269/400, Reward: -53.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 270/400, Reward: -62.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 271/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 272/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 273/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 274/400, Reward: -10.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 275/400, Reward: -8.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 276/400, Reward: -30.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 277/400, Reward: -25.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 278/400, Reward: -45.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 279/400, Reward: -48.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 280/400, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 281/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 282/400, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 283/400, Reward: -26.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 284/400, Reward: -116.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 285/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 286/400, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 287/400, Reward: -42.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 288/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 289/400, Reward: -31.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 290/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 291/400, Reward: -25.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 292/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 293/400, Reward: -43.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 294/400, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 295/400, Reward: -33.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 296/400, Reward: -12.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 297/400, Reward: -28.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 298/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 299/400, Reward: 0.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 300/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 301/400, Reward: -6.00: Epislon: 0.010 -2022-10-30 01:48:37 - r - INFO: - Episode: 302/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 303/400, Reward: 2.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 304/400, Reward: -12.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 305/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 306/400, Reward: -77.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 307/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 308/400, Reward: -32.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 309/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 310/400, Reward: -12.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 311/400, Reward: -36.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 312/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 313/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 314/400, Reward: -34.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 315/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 316/400, Reward: -21.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 317/400, Reward: -48.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 318/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 319/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 320/400, Reward: -25.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 321/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 322/400, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 323/400, Reward: -135.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 324/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 325/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 326/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 327/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 328/400, Reward: -11.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 329/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 330/400, Reward: -11.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 331/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 332/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 333/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 334/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 335/400, Reward: -12.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 336/400, Reward: -22.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 337/400, Reward: -16.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 338/400, Reward: -17.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 339/400, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 340/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 341/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 342/400, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 343/400, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 344/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 345/400, Reward: -90.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 346/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 347/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 348/400, Reward: -53.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 349/400, Reward: -87.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 350/400, Reward: -22.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 351/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 352/400, Reward: -12.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 353/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 354/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 355/400, Reward: -113.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 356/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 357/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 358/400, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 359/400, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 360/400, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 361/400, Reward: 2.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 362/400, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 363/400, Reward: -63.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 364/400, Reward: -14.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 365/400, Reward: -15.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 366/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 367/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 368/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 369/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 370/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 371/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 372/400, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 373/400, Reward: -12.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 374/400, Reward: -30.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 375/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 376/400, Reward: 2.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 377/400, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 378/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 379/400, Reward: -31.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 380/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 381/400, Reward: 2.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 382/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 383/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 384/400, Reward: -84.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 385/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 386/400, Reward: -27.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 387/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 388/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 389/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 390/400, Reward: 2.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 391/400, Reward: 2.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 392/400, Reward: 3.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 393/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 394/400, Reward: 4.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 395/400, Reward: -18.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 396/400, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 397/400, Reward: -41.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 398/400, Reward: 5.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 399/400, Reward: -41.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Episode: 400/400, Reward: -13.00: Epislon: 0.010 -2022-10-30 01:48:38 - r - INFO: - Finish training! diff --git a/projects/codes/QLearning/Train_Racetrack-v0_QLearning_20221030-014833/models/Qleaning_model.pkl b/projects/codes/QLearning/Train_Racetrack-v0_QLearning_20221030-014833/models/Qleaning_model.pkl deleted file mode 100644 index 1f458e1..0000000 Binary files a/projects/codes/QLearning/Train_Racetrack-v0_QLearning_20221030-014833/models/Qleaning_model.pkl and /dev/null differ diff --git a/projects/codes/QLearning/Train_Racetrack-v0_QLearning_20221030-014833/results/learning_curve.png b/projects/codes/QLearning/Train_Racetrack-v0_QLearning_20221030-014833/results/learning_curve.png deleted file mode 100644 index 8c1c331..0000000 Binary files a/projects/codes/QLearning/Train_Racetrack-v0_QLearning_20221030-014833/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/QLearning/Train_Racetrack-v0_QLearning_20221030-014833/results/res.csv b/projects/codes/QLearning/Train_Racetrack-v0_QLearning_20221030-014833/results/res.csv deleted file mode 100644 index 79373d8..0000000 --- a/projects/codes/QLearning/Train_Racetrack-v0_QLearning_20221030-014833/results/res.csv +++ /dev/null @@ -1,401 +0,0 @@ -episodes,rewards,steps -0,-850,200 -1,-780,200 -2,-730,200 -3,-650,200 -4,-540,200 -5,-640,200 -6,-570,200 -7,-570,200 -8,-550,200 -9,-550,200 -10,-580,200 -11,-530,200 -12,-580,200 -13,-570,200 -14,-550,200 -15,-560,200 -16,-550,200 -17,-580,200 -18,-520,200 -19,-490,200 -20,-480,200 -21,-540,200 -22,-550,200 -23,-560,200 -24,-510,200 -25,-520,200 -26,-480,200 -27,-520,200 -28,-480,200 -29,-470,200 -30,-540,200 -31,-540,200 -32,-470,200 -33,-540,200 -34,-490,200 -35,-530,200 -36,-520,200 -37,-510,200 -38,-520,200 -39,-510,200 -40,-480,200 -41,-510,200 -42,-470,200 -43,-490,200 -44,-490,200 -45,-490,200 -46,-520,200 -47,-530,200 -48,-510,200 -49,-460,200 -50,-500,200 -51,-470,200 -52,-520,200 -53,-490,200 -54,-500,200 -55,-460,200 -56,-490,200 -57,-510,200 -58,-460,200 -59,-530,200 -60,-440,200 -61,-510,200 -62,-520,200 -63,-510,200 -64,-460,200 -65,-344,154 -66,-500,200 -67,-490,200 -68,-490,200 -69,-440,200 -70,-77,47 -71,-198,88 -72,-440,200 -73,-480,200 -74,-354,154 -75,-470,200 -76,-480,200 -77,-38,28 -78,-460,200 -79,-480,200 -80,-490,200 -81,-140,70 -82,-102,52 -83,-265,125 -84,-145,75 -85,-460,200 -86,-500,200 -87,-470,200 -88,-325,155 -89,-470,200 -90,-376,156 -91,-98,58 -92,-130,70 -93,-450,200 -94,-146,66 -95,2,8 -96,-18,18 -97,-102,52 -98,-163,73 -99,-209,89 -100,-460,200 -101,-286,126 -102,-189,89 -103,-50,30 -104,-398,168 -105,-72,32 -106,-450,200 -107,-125,65 -108,4,6 -109,-161,71 -110,-408,178 -111,-440,200 -112,-188,78 -113,-114,64 -114,-415,185 -115,-159,69 -116,-234,104 -117,-31,21 -118,3,7 -119,4,6 -120,-63,33 -121,5,5 -122,-47,27 -123,-16,16 -124,-49,29 -125,-87,47 -126,-2,12 -127,-26,16 -128,-238,108 -129,-18,18 -130,-235,105 -131,-13,13 -132,-135,65 -133,-20,20 -134,-46,26 -135,-66,36 -136,-45,25 -137,-14,14 -138,1,9 -139,-106,56 -140,-112,62 -141,-47,27 -142,1,9 -143,-30,20 -144,-147,77 -145,5,5 -146,-30,20 -147,-167,77 -148,1,9 -149,-72,32 -150,-44,24 -151,-76,46 -152,-63,33 -153,-34,24 -154,5,5 -155,5,5 -156,-26,16 -157,-80,40 -158,-168,78 -159,-164,74 -160,1,9 -161,-19,19 -162,-12,12 -163,-44,24 -164,-80,40 -165,5,5 -166,4,6 -167,-29,19 -168,-56,26 -169,-47,27 -170,-76,46 -171,-13,13 -172,-145,65 -173,-28,18 -174,-63,33 -175,-106,56 -176,3,7 -177,-28,28 -178,-60,30 -179,-49,29 -180,-52,32 -181,-84,44 -182,5,5 -183,-55,35 -184,-14,14 -185,1,9 -186,-39,19 -187,-47,27 -188,-13,13 -189,-53,33 -190,-50,30 -191,-104,54 -192,-253,113 -193,-48,28 -194,-190,90 -195,-43,23 -196,-35,25 -197,0,10 -198,5,5 -199,-11,11 -200,5,5 -201,-16,16 -202,-99,49 -203,-22,22 -204,-170,80 -205,-109,59 -206,-48,28 -207,-275,115 -208,-49,29 -209,-147,77 -210,-51,31 -211,-67,37 -212,4,6 -213,-17,17 -214,3,7 -215,-69,39 -216,-218,88 -217,-63,33 -218,-11,11 -219,-34,24 -220,-32,22 -221,-15,15 -222,-26,16 -223,-19,19 -224,-148,78 -225,-19,19 -226,1,9 -227,-49,29 -228,5,5 -229,3,7 -230,-223,103 -231,-14,14 -232,4,6 -233,5,5 -234,2,8 -235,5,5 -236,4,6 -237,3,7 -238,3,7 -239,4,6 -240,-44,24 -241,-10,10 -242,2,8 -243,-108,58 -244,4,6 -245,-27,17 -246,3,7 -247,5,5 -248,5,5 -249,3,7 -250,-15,15 -251,-28,28 -252,-112,52 -253,-39,29 -254,4,6 -255,-48,28 -256,-149,69 -257,-27,17 -258,-33,23 -259,-30,20 -260,-29,19 -261,4,6 -262,4,6 -263,3,7 -264,3,7 -265,4,6 -266,5,5 -267,-52,42 -268,-53,33 -269,-62,42 -270,5,5 -271,4,6 -272,4,6 -273,-10,10 -274,-8,8 -275,-30,20 -276,-25,15 -277,-45,35 -278,-48,28 -279,-15,15 -280,4,6 -281,-14,14 -282,-26,16 -283,-116,56 -284,5,5 -285,-14,14 -286,-42,22 -287,3,7 -288,-31,21 -289,4,6 -290,-25,25 -291,5,5 -292,-43,23 -293,-21,21 -294,-33,23 -295,-12,12 -296,-28,18 -297,3,7 -298,0,10 -299,4,6 -300,-6,16 -301,4,6 -302,2,8 -303,-12,12 -304,4,6 -305,-77,47 -306,5,5 -307,-32,22 -308,5,5 -309,-12,12 -310,-36,26 -311,4,6 -312,4,6 -313,-34,24 -314,4,6 -315,-21,21 -316,-48,28 -317,4,6 -318,5,5 -319,-25,15 -320,4,6 -321,-14,14 -322,-135,65 -323,3,7 -324,5,5 -325,4,6 -326,3,7 -327,-11,11 -328,3,7 -329,-11,11 -330,3,7 -331,4,6 -332,3,7 -333,5,5 -334,-12,12 -335,-22,22 -336,-16,16 -337,-17,17 -338,-14,14 -339,3,7 -340,5,5 -341,-15,15 -342,-13,13 -343,4,6 -344,-90,40 -345,3,7 -346,3,7 -347,-53,33 -348,-87,47 -349,-22,22 -350,5,5 -351,-12,12 -352,3,7 -353,4,6 -354,-113,53 -355,3,7 -356,3,7 -357,-13,13 -358,-15,15 -359,-14,14 -360,2,8 -361,-15,15 -362,-63,33 -363,-14,14 -364,-15,15 -365,3,7 -366,3,7 -367,4,6 -368,4,6 -369,4,6 -370,3,7 -371,-13,13 -372,-12,12 -373,-30,20 -374,3,7 -375,2,8 -376,-13,13 -377,5,5 -378,-31,21 -379,3,7 -380,2,8 -381,4,6 -382,4,6 -383,-84,44 -384,3,7 -385,-27,17 -386,4,6 -387,4,6 -388,4,6 -389,2,8 -390,2,8 -391,3,7 -392,4,6 -393,4,6 -394,-18,18 -395,-13,13 -396,-41,31 -397,5,5 -398,-41,31 -399,-13,13 diff --git a/projects/codes/QLearning/config/CliffWalking-v0_QLearning_Test.yaml b/projects/codes/QLearning/config/CliffWalking-v0_QLearning_Test.yaml deleted file mode 100644 index d1a9903..0000000 --- a/projects/codes/QLearning/config/CliffWalking-v0_QLearning_Test.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: QLearning - device: cpu - env_name: CliffWalking-v0 - mode: test - load_checkpoint: true - load_path: Train_CliffWalking-v0_QLearning_20221030-013856 - max_steps: 200 - save_fig: true - seed: 1 - show_fig: false - test_eps: 20 - train_eps: 400 -algo_cfg: - epsilon_decay: 300 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - lr: 0.1 diff --git a/projects/codes/QLearning/config/CliffWalking-v0_QLearning_Train.yaml b/projects/codes/QLearning/config/CliffWalking-v0_QLearning_Train.yaml deleted file mode 100644 index 332f3ab..0000000 --- a/projects/codes/QLearning/config/CliffWalking-v0_QLearning_Train.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: QLearning - device: cpu - env_name: CliffWalking-v0 - mode: train - load_checkpoint: false - load_path: Train_CartPole-v1_DQN_20221026-054757 - max_steps: 200 - save_fig: true - seed: 1 - show_fig: false - test_eps: 20 - train_eps: 800 -algo_cfg: - epsilon_decay: 300 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - lr: 0.1 diff --git a/projects/codes/QLearning/config/FrozenLakeNoSlippery-v1_QLearning_Test.yaml b/projects/codes/QLearning/config/FrozenLakeNoSlippery-v1_QLearning_Test.yaml deleted file mode 100644 index 089e391..0000000 --- a/projects/codes/QLearning/config/FrozenLakeNoSlippery-v1_QLearning_Test.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: QLearning - device: cpu - env_name: FrozenLakeNoSlippery-v1 - mode: test - load_checkpoint: true - load_path: Train_FrozenLakeNoSlippery-v1_QLearning_20221030-014504 - max_steps: 200 - save_fig: true - seed: 10 - show_fig: false - test_eps: 20 - train_eps: 800 -algo_cfg: - epsilon_decay: 2000 - epsilon_end: 0.1 - epsilon_start: 0.7 - gamma: 0.95 - lr: 0.9 diff --git a/projects/codes/QLearning/config/FrozenLakeNoSlippery-v1_QLearning_Train.yaml b/projects/codes/QLearning/config/FrozenLakeNoSlippery-v1_QLearning_Train.yaml deleted file mode 100644 index 760750a..0000000 --- a/projects/codes/QLearning/config/FrozenLakeNoSlippery-v1_QLearning_Train.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: QLearning - device: cpu - env_name: FrozenLakeNoSlippery-v1 - mode: train - load_checkpoint: false - load_path: Train_CartPole-v1_DQN_20221026-054757 - max_steps: 200 - save_fig: true - seed: 10 - show_fig: false - test_eps: 20 - train_eps: 800 -algo_cfg: - epsilon_decay: 2000 - epsilon_end: 0.1 - epsilon_start: 0.7 - gamma: 0.95 - lr: 0.9 diff --git a/projects/codes/QLearning/config/Racetrack-v0_QLearning_Test.yaml b/projects/codes/QLearning/config/Racetrack-v0_QLearning_Test.yaml deleted file mode 100644 index 3aa9985..0000000 --- a/projects/codes/QLearning/config/Racetrack-v0_QLearning_Test.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: QLearning - device: cpu - env_name: Racetrack-v0 - mode: test - load_checkpoint: true - load_path: Train_Racetrack-v0_QLearning_20221030-014833 - max_steps: 200 - save_fig: true - seed: 10 - show_fig: false - test_eps: 20 - train_eps: 400 -algo_cfg: - epsilon_decay: 300 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.9 - lr: 0.1 diff --git a/projects/codes/QLearning/config/Racetrack-v0_QLearning_Train.yaml b/projects/codes/QLearning/config/Racetrack-v0_QLearning_Train.yaml deleted file mode 100644 index 63e51c3..0000000 --- a/projects/codes/QLearning/config/Racetrack-v0_QLearning_Train.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: QLearning - device: cpu - env_name: Racetrack-v0 - mode: train - load_checkpoint: false - load_path: Train_CartPole-v1_DQN_20221026-054757 - max_steps: 200 - save_fig: true - seed: 10 - show_fig: false - test_eps: 20 - train_eps: 400 -algo_cfg: - epsilon_decay: 300 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.9 - lr: 0.1 diff --git a/projects/codes/QLearning/config/config.py b/projects/codes/QLearning/config/config.py deleted file mode 100644 index e0ed62a..0000000 --- a/projects/codes/QLearning/config/config.py +++ /dev/null @@ -1,35 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-10-30 01:23:07 -LastEditor: JiangJi -LastEditTime: 2022-10-30 01:39:54 -Discription: default parameters of QLearning -''' -from common.config import GeneralConfig,AlgoConfig - -class GeneralConfigQLearning(GeneralConfig): - def __init__(self) -> None: - self.env_name = "CliffWalking-v0" # name of environment - self.algo_name = "QLearning" # name of algorithm - self.mode = "train" # train or test - self.seed = 1 # random seed - self.device = "cpu" # device to use - self.train_eps = 400 # number of episodes for training - self.test_eps = 20 # number of episodes for testing - self.max_steps = 200 # max steps for each episode - self.load_checkpoint = False - self.load_path = "tasks" # path to load model - self.show_fig = False # show figure or not - self.save_fig = True # save figure or not - -class AlgoConfigQLearning(AlgoConfig): - def __init__(self) -> None: - # set epsilon_start=epsilon_end can obtain fixed epsilon=epsilon_end - self.epsilon_start = 0.95 # epsilon start value - self.epsilon_end = 0.01 # epsilon end value - self.epsilon_decay = 300 # epsilon decay rate - self.gamma = 0.90 # discount factor - self.lr = 0.1 # learning rate \ No newline at end of file diff --git a/projects/codes/QLearning/qlearning.py b/projects/codes/QLearning/qlearning.py deleted file mode 100644 index 48dfa37..0000000 --- a/projects/codes/QLearning/qlearning.py +++ /dev/null @@ -1,66 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: John -Email: johnjim0816@gmail.com -Date: 2020-09-11 23:03:00 -LastEditor: John -LastEditTime: 2022-10-30 01:38:26 -Discription: use defaultdict to define Q table -Environment: -''' -import numpy as np -import math -import torch -from collections import defaultdict - -class QLearning(object): - def __init__(self,cfg): - self.n_actions = cfg.n_actions - self.lr = cfg.lr - self.gamma = cfg.gamma - self.epsilon = cfg.epsilon_start - self.sample_count = 0 - self.epsilon_start = cfg.epsilon_start - self.epsilon_end = cfg.epsilon_end - self.epsilon_decay = cfg.epsilon_decay - self.Q_table = defaultdict(lambda: np.zeros(self.n_actions)) # use nested dictionary to represent Q(s,a), here set all Q(s,a)=0 initially, not like pseudo code - def sample_action(self, state): - ''' sample action with e-greedy policy while training - ''' - self.sample_count += 1 - # epsilon must decay(linear,exponential and etc.) for balancing exploration and exploitation - self.epsilon = self.epsilon_end + (self.epsilon_start - self.epsilon_end) * \ - math.exp(-1. * self.sample_count / self.epsilon_decay) - if np.random.uniform(0, 1) > self.epsilon: - action = np.argmax(self.Q_table[str(state)]) # choose action corresponding to the maximum q value - else: - action = np.random.choice(self.n_actions) # choose action randomly - return action - def predict_action(self,state): - ''' predict action while testing - ''' - action = np.argmax(self.Q_table[str(state)]) - return action - def update(self, state, action, reward, next_state, done): - Q_predict = self.Q_table[str(state)][action] - if done: # terminal state - Q_target = reward - else: - Q_target = reward + self.gamma * np.max(self.Q_table[str(next_state)]) - self.Q_table[str(state)][action] += self.lr * (Q_target - Q_predict) - def save_model(self,path): - import dill - from pathlib import Path - # create path - Path(path).mkdir(parents=True, exist_ok=True) - torch.save( - obj=self.Q_table, - f=path+"Qleaning_model.pkl", - pickle_module=dill - ) - print("Model saved!") - def load_model(self, path): - import dill - self.Q_table =torch.load(f=path+'Qleaning_model.pkl',pickle_module=dill) - print("Mode loaded!") \ No newline at end of file diff --git a/projects/codes/QLearning/task0.py b/projects/codes/QLearning/task0.py deleted file mode 100644 index da52113..0000000 --- a/projects/codes/QLearning/task0.py +++ /dev/null @@ -1,106 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: John -Email: johnjim0816@gmail.com -Date: 2020-09-11 23:03:00 -LastEditor: John -LastEditTime: 2022-10-30 02:04:55 -Discription: -Environment: -''' -import sys,os -os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE" # avoid "OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized." -curr_path = os.path.dirname(os.path.abspath(__file__)) # current path -parent_path = os.path.dirname(curr_path) # parent path -sys.path.append(parent_path) # add path to system path - -import gym -import datetime -import argparse -from envs.gridworld_env import FrozenLakeWapper -from envs.wrappers import CliffWalkingWapper -from envs.register import register_env -from qlearning import QLearning -from common.utils import all_seed,merge_class_attrs -from common.launcher import Launcher -from config.config import GeneralConfigQLearning,AlgoConfigQLearning - -class Main(Launcher): - def __init__(self) -> None: - super().__init__() - self.cfgs['general_cfg'] = merge_class_attrs(self.cfgs['general_cfg'],GeneralConfigQLearning()) - self.cfgs['algo_cfg'] = merge_class_attrs(self.cfgs['algo_cfg'],AlgoConfigQLearning()) - def env_agent_config(self,cfg,logger): - ''' create env and agent - ''' - register_env(cfg.env_name) - env = gym.make(cfg.env_name,new_step_api=False) # create env - if cfg.env_name == 'CliffWalking-v0': - env = CliffWalkingWapper(env) - if cfg.seed !=0: # set random seed - all_seed(env,seed=cfg.seed) - try: # state dimension - n_states = env.observation_space.n # print(hasattr(env.observation_space, 'n')) - except AttributeError: - n_states = env.observation_space.shape[0] # print(hasattr(env.observation_space, 'shape')) - n_actions = env.action_space.n # action dimension - logger.info(f"n_states: {n_states}, n_actions: {n_actions}") # print info - # update to cfg paramters - setattr(cfg, 'n_states', n_states) - setattr(cfg, 'n_actions', n_actions) - agent = QLearning(cfg) - return env,agent - def train(self,cfg,env,agent,logger): - logger.info("Start training!") - logger.info(f"Env: {cfg.env_name}, Algorithm: {cfg.algo_name}, Device: {cfg.device}") - rewards = [] # record rewards for all episodes - steps = [] # record steps for all episodes - for i_ep in range(cfg.train_eps): - ep_reward = 0 # reward per episode - ep_step = 0 # step per episode - state = env.reset() # reset and obtain initial state - for _ in range(cfg.max_steps): - action = agent.sample_action(state) # sample action - next_state, reward, terminated, _ = env.step(action) # update env and return transitions - agent.update(state, action, reward, next_state, terminated) # update agent - state = next_state # update state - ep_reward += reward - ep_step += 1 - if terminated: - break - rewards.append(ep_reward) - steps.append(ep_step) - logger.info(f'Episode: {i_ep+1}/{cfg.train_eps}, Reward: {ep_reward:.2f}, Steps:{ep_step:d}, Epislon: {agent.epsilon:.3f}') - logger.info("Finish training!") - return {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - def test(self,cfg,env,agent,logger): - logger.info("Start testing!") - logger.info(f"Env: {cfg.env_name}, Algorithm: {cfg.algo_name}, Device: {cfg.device}") - rewards = [] # record rewards for all episodes - steps = [] # record steps for all episodes - for i_ep in range(cfg.test_eps): - ep_reward = 0 # reward per episode - ep_step = 0 - state = env.reset() # reset and obtain initial state - for _ in range(cfg.max_steps): - action = agent.predict_action(state) # predict action - next_state, reward, terminated, _ = env.step(action) - state = next_state - ep_reward += reward - ep_step += 1 - if terminated: - break - rewards.append(ep_reward) - steps.append(ep_step) - logger.info(f"Episode: {i_ep+1}/{cfg.test_eps}, Reward: {ep_reward:.2f}, Steps:{ep_step:d}") - logger.info("Finish testing!") - return {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - -if __name__ == "__main__": - main = Main() - main.run() - - - - diff --git a/projects/codes/RainbowDQN/rainbow_dqn.py b/projects/codes/RainbowDQN/rainbow_dqn.py deleted file mode 100644 index 0d7f783..0000000 --- a/projects/codes/RainbowDQN/rainbow_dqn.py +++ /dev/null @@ -1,215 +0,0 @@ -import math -import torch -import torch.nn as nn -import torch.nn.functional as F -import torch.optim as optim -from torch.autograd import Variable -import random -class ReplayBuffer: - def __init__(self, capacity): - self.capacity = capacity # 经验回放的容量 - self.buffer = [] # 缓冲区 - self.position = 0 - - def push(self, state, action, reward, next_state, done): - ''' 缓冲区是一个队列，容量超出时去掉开始存入的转移(transition) - ''' - if len(self.buffer) < self.capacity: - self.buffer.append(None) - self.buffer[self.position] = (state, action, reward, next_state, done) - self.position = (self.position + 1) % self.capacity - - def sample(self, batch_size): - batch = random.sample(self.buffer, batch_size) # 随机采出小批量转移 - state, action, reward, next_state, done = zip(*batch) # 解压成状态，动作等 - return state, action, reward, next_state, done - - def __len__(self): - ''' 返回当前存储的量 - ''' - return len(self.buffer) -class NoisyLinear(nn.Module): - def __init__(self, input_dim, output_dim, device, std_init=0.4): - super(NoisyLinear, self).__init__() - - self.device = device - self.input_dim = input_dim - self.output_dim = output_dim - self.std_init = std_init - - self.weight_mu = nn.Parameter(torch.FloatTensor(output_dim, input_dim)) - self.weight_sigma = nn.Parameter(torch.FloatTensor(output_dim, input_dim)) - self.register_buffer('weight_epsilon', torch.FloatTensor(output_dim, input_dim)) - - self.bias_mu = nn.Parameter(torch.FloatTensor(output_dim)) - self.bias_sigma = nn.Parameter(torch.FloatTensor(output_dim)) - self.register_buffer('bias_epsilon', torch.FloatTensor(output_dim)) - - self.reset_parameters() - self.reset_noise() - - def forward(self, x): - if self.device: - weight_epsilon = self.weight_epsilon.cuda() - bias_epsilon = self.bias_epsilon.cuda() - else: - weight_epsilon = self.weight_epsilon - bias_epsilon = self.bias_epsilon - - if self.training: - weight = self.weight_mu + self.weight_sigma.mul(Variable(weight_epsilon)) - bias = self.bias_mu + self.bias_sigma.mul(Variable(bias_epsilon)) - else: - weight = self.weight_mu - bias = self.bias_mu - - return F.linear(x, weight, bias) - - def reset_parameters(self): - mu_range = 1 / math.sqrt(self.weight_mu.size(1)) - - self.weight_mu.data.uniform_(-mu_range, mu_range) - self.weight_sigma.data.fill_(self.std_init / math.sqrt(self.weight_sigma.size(1))) - - self.bias_mu.data.uniform_(-mu_range, mu_range) - self.bias_sigma.data.fill_(self.std_init / math.sqrt(self.bias_sigma.size(0))) - - def reset_noise(self): - epsilon_in = self._scale_noise(self.input_dim) - epsilon_out = self._scale_noise(self.output_dim) - - self.weight_epsilon.copy_(epsilon_out.ger(epsilon_in)) - self.bias_epsilon.copy_(self._scale_noise(self.output_dim)) - - def _scale_noise(self, size): - x = torch.randn(size) - x = x.sign().mul(x.abs().sqrt()) - return x - -class RainbowModel(nn.Module): - def __init__(self, n_states, n_actions, n_atoms, Vmin, Vmax): - super(RainbowModel, self).__init__() - - self.n_states = n_states - self.n_actions = n_actions - self.n_atoms = n_atoms - self.Vmin = Vmin - self.Vmax = Vmax - - self.linear1 = nn.Linear(n_states, 32) - self.linear2 = nn.Linear(32, 64) - - self.noisy_value1 = NoisyLinear(64, 64, device=device) - self.noisy_value2 = NoisyLinear(64, self.n_atoms, device=device) - - self.noisy_advantage1 = NoisyLinear(64, 64, device=device) - self.noisy_advantage2 = NoisyLinear(64, self.n_atoms * self.n_actions, device=device) - - def forward(self, x): - batch_size = x.size(0) - - x = F.relu(self.linear1(x)) - x = F.relu(self.linear2(x)) - - value = F.relu(self.noisy_value1(x)) - value = self.noisy_value2(value) - - advantage = F.relu(self.noisy_advantage1(x)) - advantage = self.noisy_advantage2(advantage) - - value = value.view(batch_size, 1, self.n_atoms) - advantage = advantage.view(batch_size, self.n_actions, self.n_atoms) - - x = value + advantage - advantage.mean(1, keepdim=True) - x = F.softmax(x.view(-1, self.n_atoms)).view(-1, self.n_actions, self.n_atoms) - - return x - - def reset_noise(self): - self.noisy_value1.reset_noise() - self.noisy_value2.reset_noise() - self.noisy_advantage1.reset_noise() - self.noisy_advantage2.reset_noise() - - def act(self, state): - state = Variable(torch.FloatTensor(state).unsqueeze(0), volatile=True) - dist = self.forward(state).data.cpu() - dist = dist * torch.linspace(self.Vmin, self.Vmax, self.n_atoms) - action = dist.sum(2).max(1)[1].numpy()[0] - return action - -class RainbowDQN(nn.Module): - def __init__(self, n_states, n_actions, n_atoms, Vmin, Vmax,cfg): - super(RainbowDQN, self).__init__() - self.n_states = n_states - self.n_actions = n_actions - self.n_atoms = cfg.n_atoms - self.Vmin = cfg.Vmin - self.Vmax = cfg.Vmax - self.policy_model = RainbowModel(n_states, n_actions, n_atoms, Vmin, Vmax) - self.target_model = RainbowModel(n_states, n_actions, n_atoms, Vmin, Vmax) - self.batch_size = cfg.batch_size - self.memory = ReplayBuffer(cfg.memory_capacity) # 经验回放 - self.optimizer = optim.Adam(self.policy_model.parameters(), 0.001) - def choose_action(self,state): - state = Variable(torch.FloatTensor(state).unsqueeze(0), volatile=True) - dist = self.policy_model(state).data.cpu() - dist = dist * torch.linspace(self.Vmin, self.Vmax, self.n_atoms) - action = dist.sum(2).max(1)[1].numpy()[0] - return action - def projection_distribution(self,next_state, rewards, dones): - - - delta_z = float(self.Vmax - self.Vmin) / (self.n_atoms - 1) - support = torch.linspace(self.Vmin, self.Vmax, self.n_atoms) - - next_dist = self.target_model(next_state).data.cpu() * support - next_action = next_dist.sum(2).max(1)[1] - next_action = next_action.unsqueeze(1).unsqueeze(1).expand(next_dist.size(0), 1, next_dist.size(2)) - next_dist = next_dist.gather(1, next_action).squeeze(1) - - rewards = rewards.unsqueeze(1).expand_as(next_dist) - dones = dones.unsqueeze(1).expand_as(next_dist) - support = support.unsqueeze(0).expand_as(next_dist) - - Tz = rewards + (1 - dones) * 0.99 * support - Tz = Tz.clamp(min=self.Vmin, max=self.Vmax) - b = (Tz - self.Vmin) / delta_z - l = b.floor().long() - u = b.ceil().long() - - offset = torch.linspace(0, (self.batch_size - 1) * self.n_atoms, self.batch_size).long()\ - .unsqueeze(1).expand(self.batch_size, self.n_atoms) - - proj_dist = torch.zeros(next_dist.size()) - proj_dist.view(-1).index_add_(0, (l + offset).view(-1), (next_dist * (u.float() - b)).view(-1)) - proj_dist.view(-1).index_add_(0, (u + offset).view(-1), (next_dist * (b - l.float())).view(-1)) - - return proj_dist - def update(self): - if len(self.memory) < self.batch_size: # 当memory中不满足一个批量时，不更新策略 - return - state, action, reward, next_state, done = self.memory.sample(self.batch_size) - - state = Variable(torch.FloatTensor(np.float32(state))) - next_state = Variable(torch.FloatTensor(np.float32(next_state)), volatile=True) - action = Variable(torch.LongTensor(action)) - reward = torch.FloatTensor(reward) - done = torch.FloatTensor(np.float32(done)) - - proj_dist = self.projection_distribution(next_state, reward, done) - - dist = self.policy_model(state) - action = action.unsqueeze(1).unsqueeze(1).expand(self.batch_size, 1, self.n_atoms) - dist = dist.gather(1, action).squeeze(1) - dist.data.clamp_(0.01, 0.99) - loss = -(Variable(proj_dist) * dist.log()).sum(1) - loss = loss.mean() - - self.optimizer.zero_grad() - loss.backward() - self.optimizer.step() - - self.policy_model.reset_noise() - self.target_model.reset_noise() - \ No newline at end of file diff --git a/projects/codes/RainbowDQN/task0.py b/projects/codes/RainbowDQN/task0.py deleted file mode 100644 index 49a97a4..0000000 --- a/projects/codes/RainbowDQN/task0.py +++ /dev/null @@ -1,177 +0,0 @@ -import sys -import os -import torch.nn as nn -import torch.nn.functional as F -curr_path = os.path.dirname(os.path.abspath(__file__)) # 当前文件所在绝对路径 -parent_path = os.path.dirname(curr_path) # 父路径 -sys.path.append(parent_path) # 添加路径到系统路径 - -import gym -import torch -import datetime -import numpy as np -from common.utils import save_results_1, make_dir -from common.utils import plot_rewards -from dqn import DQN - -curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # 获取当前时间 - -class MLP(nn.Module): - def __init__(self, n_states,n_actions,hidden_dim=128): - """ 初始化q网络，为全连接网络 - n_states: 输入的特征数即环境的状态维度 - n_actions: 输出的动作维度 - """ - super(MLP, self).__init__() - self.fc1 = nn.Linear(n_states, hidden_dim) # 输入层 - self.fc2 = nn.Linear(hidden_dim,hidden_dim) # 隐藏层 - self.fc3 = nn.Linear(hidden_dim, n_actions) # 输出层 - - def forward(self, x): - # 各层对应的激活函数 - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - return self.fc3(x) - -class Config: - '''超参数 - ''' - - def __init__(self): - ############################### hyperparameters ################################ - self.algo_name = 'DQN' # algorithm name - self.env_name = 'CartPole-v0' # environment name - self.device = torch.device( - "cuda" if torch.cuda.is_available() else "cpu") # check GPU - self.seed = 10 # 随机种子，置0则不设置随机种子 - self.train_eps = 200 # 训练的回合数 - self.test_eps = 20 # 测试的回合数 - ################################################################################ - - ################################## 算法超参数 ################################### - self.gamma = 0.95 # 强化学习中的折扣因子 - self.epsilon_start = 0.90 # e-greedy策略中初始epsilon - self.epsilon_end = 0.01 # e-greedy策略中的终止epsilon - self.epsilon_decay = 500 # e-greedy策略中epsilon的衰减率 - self.lr = 0.0001 # 学习率 - self.memory_capacity = 100000 # 经验回放的容量 - self.batch_size = 64 # mini-batch SGD中的批量大小 - self.target_update = 4 # 目标网络的更新频率 - self.hidden_dim = 256 # 网络隐藏层 - ################################################################################ - - ################################# 保存结果相关参数 ################################ - self.result_path = curr_path + "/outputs/" + self.env_name + \ - '/' + curr_time + '/results/' # 保存结果的路径 - self.model_path = curr_path + "/outputs/" + self.env_name + \ - '/' + curr_time + '/models/' # 保存模型的路径 - self.save = True # 是否保存图片 - ################################################################################ - - -def env_agent_config(cfg): - ''' 创建环境和智能体 - ''' - env = gym.make(cfg.env_name) # 创建环境 - n_states = env.observation_space.shape[0] # 状态维度 - n_actions = env.action_space.n # 动作维度 - print(f"n states: {n_states}, n actions: {n_actions}") - model = MLP(n_states,n_actions) - agent = DQN(n_actions, model, cfg) # 创建智能体 - if cfg.seed !=0: # 设置随机种子 - torch.manual_seed(cfg.seed) - env.seed(cfg.seed) - np.random.seed(cfg.seed) - return env, agent - - -def train(cfg, env, agent): - ''' 训练 - ''' - print('开始训练!') - print(f'环境：{cfg.env_name}, 算法：{cfg.algo_name}, 设备：{cfg.device}') - rewards = [] # 记录所有回合的奖励 - ma_rewards = [] # 记录所有回合的滑动平均奖励 - steps = [] - for i_ep in range(cfg.train_eps): - ep_reward = 0 # 记录一回合内的奖励 - ep_step = 0 - state = env.reset() # 重置环境，返回初始状态 - while True: - ep_step += 1 - action = agent.choose_action(state) # 选择动作 - next_state, reward, done, _ = env.step(action) # 更新环境，返回transition - agent.memory.push(state, action, reward, - next_state, done) # 保存transition - state = next_state # 更新下一个状态 - agent.update() # 更新智能体 - ep_reward += reward # 累加奖励 - if done: - break - if (i_ep + 1) % cfg.target_update == 0: # 智能体目标网络更新 - agent.target_net.load_state_dict(agent.policy_net.state_dict()) - steps.append(ep_step) - rewards.append(ep_reward) - if ma_rewards: - ma_rewards.append(0.9 * ma_rewards[-1] + 0.1 * ep_reward) - else: - ma_rewards.append(ep_reward) - if (i_ep + 1) % 1 == 0: - print(f'Episode：{i_ep+1}/{cfg.test_eps}, Reward:{ep_reward:.2f}, Step:{ep_step:.2f} Epislon:{agent.epsilon(agent.frame_idx):.3f}') - print('Finish training!') - env.close() - res_dic = {'rewards':rewards,'ma_rewards':ma_rewards,'steps':steps} - return res_dic - - -def test(cfg, env, agent): - print('开始测试!') - print(f'环境：{cfg.env_name}, 算法：{cfg.algo_name}, 设备：{cfg.device}') - ############# 由于测试不需要使用epsilon-greedy策略，所以相应的值设置为0 ############### - cfg.epsilon_start = 0.0 # e-greedy策略中初始epsilon - cfg.epsilon_end = 0.0 # e-greedy策略中的终止epsilon - ################################################################################ - rewards = [] # 记录所有回合的奖励 - ma_rewards = [] # 记录所有回合的滑动平均奖励 - steps = [] - for i_ep in range(cfg.test_eps): - ep_reward = 0 # 记录一回合内的奖励 - ep_step = 0 - state = env.reset() # 重置环境，返回初始状态 - while True: - ep_step+=1 - action = agent.choose_action(state) # 选择动作 - next_state, reward, done, _ = env.step(action) # 更新环境，返回transition - state = next_state # 更新下一个状态 - ep_reward += reward # 累加奖励 - if done: - break - steps.append(ep_step) - rewards.append(ep_reward) - if ma_rewards: - ma_rewards.append(ma_rewards[-1] * 0.9 + ep_reward * 0.1) - else: - ma_rewards.append(ep_reward) - print(f'Episode：{i_ep+1}/{cfg.train_eps}, Reward:{ep_reward:.2f}, Step:{ep_step:.2f}') - print('完成测试！') - env.close() - return {'rewards':rewards,'ma_rewards':ma_rewards,'steps':steps} - - -if __name__ == "__main__": - cfg = Config() - # 训练 - env, agent = env_agent_config(cfg) - res_dic = train(cfg, env, agent) - make_dir(cfg.result_path, cfg.model_path) # 创建保存结果和模型路径的文件夹 - agent.save(path=cfg.model_path) # 保存模型 - save_results_1(res_dic, tag='train', - path=cfg.result_path) # 保存结果 - plot_rewards(res_dic['rewards'], res_dic['ma_rewards'], cfg, tag="train") # 画出结果 - # 测试 - env, agent = env_agent_config(cfg) - agent.load(path=cfg.model_path) # 导入模型 - res_dic = test(cfg, env, agent) - save_results_1(res_dic, tag='test', - path=cfg.result_path) # 保存结果 - plot_rewards(res_dic['rewards'], res_dic['ma_rewards'],cfg, tag="test") # 画出结果 diff --git a/projects/codes/SAC-S/sac.py b/projects/codes/SAC-S/sac.py deleted file mode 100644 index 6351c3d..0000000 --- a/projects/codes/SAC-S/sac.py +++ /dev/null @@ -1,27 +0,0 @@ -import torch -import torch.optim as optim -import torch.nn as nn -import numpy as np -class SAC: - def __init__(self,n_actions,models,memory,cfg): - self.device = cfg.device - self.value_net = models['ValueNet'].to(self.device) # $\psi$ - self.target_value_net = models['ValueNet'].to(self.device) # $\bar{\psi}$ - self.soft_q_net = models['SoftQNet'].to(self.device) # $\theta$ - self.policy_net = models['PolicyNet'].to(self.device) # $\phi$ - self.value_optimizer = optim.Adam(self.value_net.parameters(), lr=cfg.value_lr) - self.soft_q_optimizer = optim.Adam(self.soft_q_net.parameters(), lr=cfg.soft_q_lr) - self.policy_optimizer = optim.Adam(self.policy_net.parameters(), lr=cfg.policy_lr) - for target_param, param in zip(self.target_value_net.parameters(), self.value_net.parameters()): - target_param.data.copy_(param.data) - self.value_criterion = nn.MSELoss() - self.soft_q_criterion = nn.MSELoss() - def update(self): - # sample a batch of transitions from replay buffer - state_batch, action_batch, reward_batch, next_state_batch, done_batch = self.memory.sample( - self.batch_size) - state_batch = torch.tensor(np.array(state_batch), device=self.device, dtype=torch.float) # shape(batchsize,n_states) - action_batch = torch.tensor(action_batch, device=self.device).unsqueeze(1) # shape(batchsize,1) - reward_batch = torch.tensor(reward_batch, device=self.device, dtype=torch.float).unsqueeze(1) # shape(batchsize) - next_state_batch = torch.tensor(np.array(next_state_batch), device=self.device, dtype=torch.float) # shape(batchsize,n_states) - done_batch = torch.tensor(np.float32(done_batch), device=self.device).unsqueeze(1) # shape(batchsize,1) diff --git a/projects/codes/SAC/sacd_cnn.py b/projects/codes/SAC/sacd_cnn.py deleted file mode 100644 index e69de29..0000000 diff --git a/projects/codes/Sarsa/README.md b/projects/codes/Sarsa/README.md deleted file mode 100644 index 5258664..0000000 --- a/projects/codes/Sarsa/README.md +++ /dev/null @@ -1,19 +0,0 @@ -# Sarsa - -## 使用说明 - -运行```main.py```即可 - -## 环境说明 - -见[环境说明](https://github.com/JohnJim0816/reinforcement-learning-tutorials/blob/master/env_info.md)中的The Racetrack - -## 算法伪代码 - -![sarsa_algo](assets/sarsa_algo.png) - -## 其他说明 - -### 与Q-learning区别 - -算法上区别很小，只在更新公式上，但Q-learning是Off-policy，而Sarsa是On-policy，可参考[知乎：强化学习中sarsa算法是不是比q-learning算法收敛速度更慢？](https://www.zhihu.com/question/268461866) \ No newline at end of file diff --git a/projects/codes/Sarsa/Test_CliffWalking-v0_Sarsa_20221030-021206/config.yaml b/projects/codes/Sarsa/Test_CliffWalking-v0_Sarsa_20221030-021206/config.yaml deleted file mode 100644 index f1c252d..0000000 --- a/projects/codes/Sarsa/Test_CliffWalking-v0_Sarsa_20221030-021206/config.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: Sarsa - device: cpu - env_name: CliffWalking-v0 - load_checkpoint: true - load_path: Train_CliffWalking-v0_Sarsa_20221030-021146 - max_steps: 200 - mode: test - save_fig: true - seed: 1 - show_fig: false - test_eps: 20 - train_eps: 400 -algo_cfg: - epsilon_decay: 300 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - lr: 0.1 diff --git a/projects/codes/Sarsa/Test_CliffWalking-v0_Sarsa_20221030-021206/logs/log.txt b/projects/codes/Sarsa/Test_CliffWalking-v0_Sarsa_20221030-021206/logs/log.txt deleted file mode 100644 index 29ed4a8..0000000 --- a/projects/codes/Sarsa/Test_CliffWalking-v0_Sarsa_20221030-021206/logs/log.txt +++ /dev/null @@ -1,24 +0,0 @@ -2022-10-30 02:12:06 - r - INFO: - n_states: 48, n_actions: 4 -2022-10-30 02:12:06 - r - INFO: - Start testing! -2022-10-30 02:12:06 - r - INFO: - Env: CliffWalking-v0, Algorithm: Sarsa, Device: cpu -2022-10-30 02:12:06 - r - INFO: - Episode: 1/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 2/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 3/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 4/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 5/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 6/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 7/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 8/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 9/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 10/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 11/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 12/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 13/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 14/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 15/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 16/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 17/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 18/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 19/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Episode: 20/20, Reward: -15.00, Steps:15 -2022-10-30 02:12:06 - r - INFO: - Finish testing! diff --git a/projects/codes/Sarsa/Test_CliffWalking-v0_Sarsa_20221030-021206/models/checkpoint.pkl b/projects/codes/Sarsa/Test_CliffWalking-v0_Sarsa_20221030-021206/models/checkpoint.pkl deleted file mode 100644 index d226d4c..0000000 Binary files a/projects/codes/Sarsa/Test_CliffWalking-v0_Sarsa_20221030-021206/models/checkpoint.pkl and /dev/null differ diff --git a/projects/codes/Sarsa/Test_CliffWalking-v0_Sarsa_20221030-021206/results/learning_curve.png b/projects/codes/Sarsa/Test_CliffWalking-v0_Sarsa_20221030-021206/results/learning_curve.png deleted file mode 100644 index cf20c71..0000000 Binary files a/projects/codes/Sarsa/Test_CliffWalking-v0_Sarsa_20221030-021206/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/Sarsa/Test_CliffWalking-v0_Sarsa_20221030-021206/results/res.csv b/projects/codes/Sarsa/Test_CliffWalking-v0_Sarsa_20221030-021206/results/res.csv deleted file mode 100644 index 7f09e4b..0000000 --- a/projects/codes/Sarsa/Test_CliffWalking-v0_Sarsa_20221030-021206/results/res.csv +++ /dev/null @@ -1,21 +0,0 @@ -episodes,rewards,steps -0,-15,15 -1,-15,15 -2,-15,15 -3,-15,15 -4,-15,15 -5,-15,15 -6,-15,15 -7,-15,15 -8,-15,15 -9,-15,15 -10,-15,15 -11,-15,15 -12,-15,15 -13,-15,15 -14,-15,15 -15,-15,15 -16,-15,15 -17,-15,15 -18,-15,15 -19,-15,15 diff --git a/projects/codes/Sarsa/Test_Racetrack-v0_Sarsa_20221030-021347/config.yaml b/projects/codes/Sarsa/Test_Racetrack-v0_Sarsa_20221030-021347/config.yaml deleted file mode 100644 index 7c1b16f..0000000 --- a/projects/codes/Sarsa/Test_Racetrack-v0_Sarsa_20221030-021347/config.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: Sarsa - device: cpu - env_name: Racetrack-v0 - load_checkpoint: true - load_path: Train_Racetrack-v0_Sarsa_20221030-021315 - max_steps: 200 - mode: test - save_fig: true - seed: 10 - show_fig: false - test_eps: 20 - train_eps: 400 -algo_cfg: - epsilon_decay: 200 - epsilon_end: 0.01 - epsilon_start: 0.9 - gamma: 0.99 - lr: 0.1 diff --git a/projects/codes/Sarsa/Test_Racetrack-v0_Sarsa_20221030-021347/logs/log.txt b/projects/codes/Sarsa/Test_Racetrack-v0_Sarsa_20221030-021347/logs/log.txt deleted file mode 100644 index 7fd4614..0000000 --- a/projects/codes/Sarsa/Test_Racetrack-v0_Sarsa_20221030-021347/logs/log.txt +++ /dev/null @@ -1,24 +0,0 @@ -2022-10-30 02:13:47 - r - INFO: - n_states: 4, n_actions: 9 -2022-10-30 02:13:47 - r - INFO: - Start testing! -2022-10-30 02:13:47 - r - INFO: - Env: Racetrack-v0, Algorithm: Sarsa, Device: cpu -2022-10-30 02:13:47 - r - INFO: - Episode: 1/20, Reward: 3.00, Steps:7 -2022-10-30 02:13:47 - r - INFO: - Episode: 2/20, Reward: 3.00, Steps:7 -2022-10-30 02:13:47 - r - INFO: - Episode: 3/20, Reward: 2.00, Steps:8 -2022-10-30 02:13:47 - r - INFO: - Episode: 4/20, Reward: 3.00, Steps:7 -2022-10-30 02:13:47 - r - INFO: - Episode: 5/20, Reward: -12.00, Steps:12 -2022-10-30 02:13:47 - r - INFO: - Episode: 6/20, Reward: -49.00, Steps:29 -2022-10-30 02:13:47 - r - INFO: - Episode: 7/20, Reward: 3.00, Steps:7 -2022-10-30 02:13:47 - r - INFO: - Episode: 8/20, Reward: -17.00, Steps:17 -2022-10-30 02:13:47 - r - INFO: - Episode: 9/20, Reward: 4.00, Steps:6 -2022-10-30 02:13:47 - r - INFO: - Episode: 10/20, Reward: -17.00, Steps:17 -2022-10-30 02:13:47 - r - INFO: - Episode: 11/20, Reward: 2.00, Steps:8 -2022-10-30 02:13:47 - r - INFO: - Episode: 12/20, Reward: 3.00, Steps:7 -2022-10-30 02:13:47 - r - INFO: - Episode: 13/20, Reward: 3.00, Steps:7 -2022-10-30 02:13:47 - r - INFO: - Episode: 14/20, Reward: 2.00, Steps:8 -2022-10-30 02:13:47 - r - INFO: - Episode: 15/20, Reward: 3.00, Steps:7 -2022-10-30 02:13:47 - r - INFO: - Episode: 16/20, Reward: -34.00, Steps:24 -2022-10-30 02:13:47 - r - INFO: - Episode: 17/20, Reward: 3.00, Steps:7 -2022-10-30 02:13:47 - r - INFO: - Episode: 18/20, Reward: 5.00, Steps:5 -2022-10-30 02:13:47 - r - INFO: - Episode: 19/20, Reward: 5.00, Steps:5 -2022-10-30 02:13:47 - r - INFO: - Episode: 20/20, Reward: 3.00, Steps:7 -2022-10-30 02:13:47 - r - INFO: - Finish testing! diff --git a/projects/codes/Sarsa/Test_Racetrack-v0_Sarsa_20221030-021347/models/checkpoint.pkl b/projects/codes/Sarsa/Test_Racetrack-v0_Sarsa_20221030-021347/models/checkpoint.pkl deleted file mode 100644 index d950b3f..0000000 Binary files a/projects/codes/Sarsa/Test_Racetrack-v0_Sarsa_20221030-021347/models/checkpoint.pkl and /dev/null differ diff --git a/projects/codes/Sarsa/Test_Racetrack-v0_Sarsa_20221030-021347/results/learning_curve.png b/projects/codes/Sarsa/Test_Racetrack-v0_Sarsa_20221030-021347/results/learning_curve.png deleted file mode 100644 index fde014e..0000000 Binary files a/projects/codes/Sarsa/Test_Racetrack-v0_Sarsa_20221030-021347/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/Sarsa/Test_Racetrack-v0_Sarsa_20221030-021347/results/res.csv b/projects/codes/Sarsa/Test_Racetrack-v0_Sarsa_20221030-021347/results/res.csv deleted file mode 100644 index 5d08ed0..0000000 --- a/projects/codes/Sarsa/Test_Racetrack-v0_Sarsa_20221030-021347/results/res.csv +++ /dev/null @@ -1,21 +0,0 @@ -episodes,rewards,steps -0,3,7 -1,3,7 -2,2,8 -3,3,7 -4,-12,12 -5,-49,29 -6,3,7 -7,-17,17 -8,4,6 -9,-17,17 -10,2,8 -11,3,7 -12,3,7 -13,2,8 -14,3,7 -15,-34,24 -16,3,7 -17,5,5 -18,5,5 -19,3,7 diff --git a/projects/codes/Sarsa/Train_CliffWalking-v0_Sarsa_20221030-021146/config.yaml b/projects/codes/Sarsa/Train_CliffWalking-v0_Sarsa_20221030-021146/config.yaml deleted file mode 100644 index 4d61198..0000000 --- a/projects/codes/Sarsa/Train_CliffWalking-v0_Sarsa_20221030-021146/config.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: Sarsa - device: cpu - env_name: CliffWalking-v0 - load_checkpoint: false - load_path: Train_CartPole-v1_DQN_20221026-054757 - max_steps: 200 - mode: train - save_fig: true - seed: 1 - show_fig: false - test_eps: 20 - train_eps: 800 -algo_cfg: - epsilon_decay: 300 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - lr: 0.1 diff --git a/projects/codes/Sarsa/Train_CliffWalking-v0_Sarsa_20221030-021146/logs/log.txt b/projects/codes/Sarsa/Train_CliffWalking-v0_Sarsa_20221030-021146/logs/log.txt deleted file mode 100644 index 76df5b3..0000000 --- a/projects/codes/Sarsa/Train_CliffWalking-v0_Sarsa_20221030-021146/logs/log.txt +++ /dev/null @@ -1,804 +0,0 @@ -2022-10-30 02:11:46 - r - INFO: - n_states: 48, n_actions: 4 -2022-10-30 02:11:46 - r - INFO: - Start training! -2022-10-30 02:11:46 - r - INFO: - Env: CliffWalking-v0, Algorithm: Sarsa, Device: cpu -2022-10-30 02:11:46 - r - INFO: - Episode: 1/800, Reward: -1091.00, Steps:200, Epislon: 0.491 -2022-10-30 02:11:46 - r - INFO: - Episode: 2/800, Reward: -320.00, Steps:122, Epislon: 0.329 -2022-10-30 02:11:46 - r - INFO: - Episode: 3/800, Reward: -794.00, Steps:200, Epislon: 0.173 -2022-10-30 02:11:46 - r - INFO: - Episode: 4/800, Reward: -596.00, Steps:200, Epislon: 0.094 -2022-10-30 02:11:46 - r - INFO: - Episode: 5/800, Reward: -398.00, Steps:200, Epislon: 0.053 -2022-10-30 02:11:46 - r - INFO: - Episode: 6/800, Reward: -59.00, Steps:59, Epislon: 0.045 -2022-10-30 02:11:46 - r - INFO: - Episode: 7/800, Reward: -299.00, Steps:200, Epislon: 0.028 -2022-10-30 02:11:46 - r - INFO: - Episode: 8/800, Reward: -82.00, Steps:82, Epislon: 0.024 -2022-10-30 02:11:46 - r - INFO: - Episode: 9/800, Reward: -125.00, Steps:125, Epislon: 0.019 -2022-10-30 02:11:46 - r - INFO: - Episode: 10/800, Reward: -75.00, Steps:75, Epislon: 0.017 -2022-10-30 02:11:46 - r - INFO: - Episode: 11/800, Reward: -285.00, Steps:186, Epislon: 0.014 -2022-10-30 02:11:46 - r - INFO: - Episode: 12/800, Reward: -103.00, Steps:103, Epislon: 0.013 -2022-10-30 02:11:46 - r - INFO: - Episode: 13/800, Reward: -103.00, Steps:103, Epislon: 0.012 -2022-10-30 02:11:46 - r - INFO: - Episode: 14/800, Reward: -131.00, Steps:131, Epislon: 0.011 -2022-10-30 02:11:46 - r - INFO: - Episode: 15/800, Reward: -53.00, Steps:53, Epislon: 0.011 -2022-10-30 02:11:46 - r - INFO: - Episode: 16/800, Reward: -113.00, Steps:113, Epislon: 0.011 -2022-10-30 02:11:46 - r - INFO: - Episode: 17/800, Reward: -125.00, Steps:125, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 18/800, Reward: -95.00, Steps:95, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 19/800, Reward: -97.00, Steps:97, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 20/800, Reward: -145.00, Steps:145, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 21/800, Reward: -89.00, Steps:89, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 22/800, Reward: -97.00, Steps:97, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 23/800, Reward: -115.00, Steps:115, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 24/800, Reward: -121.00, Steps:121, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 25/800, Reward: -53.00, Steps:53, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 26/800, Reward: -111.00, Steps:111, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 27/800, Reward: -97.00, Steps:97, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 28/800, Reward: -206.00, Steps:107, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 29/800, Reward: -147.00, Steps:147, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 30/800, Reward: -36.00, Steps:36, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 31/800, Reward: -216.00, Steps:117, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 32/800, Reward: -103.00, Steps:103, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 33/800, Reward: -87.00, Steps:87, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 34/800, Reward: -80.00, Steps:80, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 35/800, Reward: -73.00, Steps:73, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 36/800, Reward: -83.00, Steps:83, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 37/800, Reward: -143.00, Steps:44, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 38/800, Reward: -241.00, Steps:142, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 39/800, Reward: -77.00, Steps:77, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 40/800, Reward: -49.00, Steps:49, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 41/800, Reward: -87.00, Steps:87, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 42/800, Reward: -47.00, Steps:47, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 43/800, Reward: -89.00, Steps:89, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 44/800, Reward: -31.00, Steps:31, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 45/800, Reward: -192.00, Steps:93, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 46/800, Reward: -85.00, Steps:85, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 47/800, Reward: -55.00, Steps:55, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 48/800, Reward: -59.00, Steps:59, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 49/800, Reward: -60.00, Steps:60, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 50/800, Reward: -50.00, Steps:50, Epislon: 0.010 -2022-10-30 02:11:46 - r - INFO: - Episode: 51/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 52/800, Reward: -101.00, Steps:101, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 53/800, Reward: -43.00, Steps:43, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 54/800, Reward: -70.00, Steps:70, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 55/800, Reward: -35.00, Steps:35, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 56/800, Reward: -47.00, Steps:47, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 57/800, Reward: -80.00, Steps:80, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 58/800, Reward: -61.00, Steps:61, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 59/800, Reward: -35.00, Steps:35, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 60/800, Reward: -73.00, Steps:73, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 61/800, Reward: -54.00, Steps:54, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 62/800, Reward: -37.00, Steps:37, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 63/800, Reward: -65.00, Steps:65, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 64/800, Reward: -41.00, Steps:41, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 65/800, Reward: -81.00, Steps:81, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 66/800, Reward: -39.00, Steps:39, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 67/800, Reward: -35.00, Steps:35, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 68/800, Reward: -61.00, Steps:61, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 69/800, Reward: -57.00, Steps:57, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 70/800, Reward: -43.00, Steps:43, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 71/800, Reward: -59.00, Steps:59, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 72/800, Reward: -43.00, Steps:43, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 73/800, Reward: -51.00, Steps:51, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 74/800, Reward: -43.00, Steps:43, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 75/800, Reward: -69.00, Steps:69, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 76/800, Reward: -41.00, Steps:41, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 77/800, Reward: -194.00, Steps:95, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 78/800, Reward: -35.00, Steps:35, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 79/800, Reward: -35.00, Steps:35, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 80/800, Reward: -81.00, Steps:81, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 81/800, Reward: -65.00, Steps:65, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 82/800, Reward: -35.00, Steps:35, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 83/800, Reward: -47.00, Steps:47, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 84/800, Reward: -53.00, Steps:53, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 85/800, Reward: -165.00, Steps:66, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 86/800, Reward: -69.00, Steps:69, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 87/800, Reward: -35.00, Steps:35, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 88/800, Reward: -56.00, Steps:56, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 89/800, Reward: -164.00, Steps:65, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 90/800, Reward: -45.00, Steps:45, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 91/800, Reward: -43.00, Steps:43, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 92/800, Reward: -43.00, Steps:43, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 93/800, Reward: -29.00, Steps:29, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 94/800, Reward: -69.00, Steps:69, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 95/800, Reward: -33.00, Steps:33, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 96/800, Reward: -57.00, Steps:57, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 97/800, Reward: -29.00, Steps:29, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 98/800, Reward: -55.00, Steps:55, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 99/800, Reward: -61.00, Steps:61, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 100/800, Reward: -162.00, Steps:63, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 101/800, Reward: -55.00, Steps:55, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 102/800, Reward: -31.00, Steps:31, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 103/800, Reward: -53.00, Steps:53, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 104/800, Reward: -39.00, Steps:39, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 105/800, Reward: -55.00, Steps:55, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 106/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 107/800, Reward: -33.00, Steps:33, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 108/800, Reward: -49.00, Steps:49, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 109/800, Reward: -65.00, Steps:65, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 110/800, Reward: -45.00, Steps:45, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 111/800, Reward: -29.00, Steps:29, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 112/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 113/800, Reward: -51.00, Steps:51, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 114/800, Reward: -43.00, Steps:43, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 115/800, Reward: -47.00, Steps:47, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 116/800, Reward: -41.00, Steps:41, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 117/800, Reward: -27.00, Steps:27, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 118/800, Reward: -31.00, Steps:31, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 119/800, Reward: -180.00, Steps:81, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 120/800, Reward: -43.00, Steps:43, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 121/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 122/800, Reward: -47.00, Steps:47, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 123/800, Reward: -65.00, Steps:65, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 124/800, Reward: -29.00, Steps:29, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 125/800, Reward: -31.00, Steps:31, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 126/800, Reward: -49.00, Steps:49, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 127/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 128/800, Reward: -45.00, Steps:45, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 129/800, Reward: -49.00, Steps:49, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 130/800, Reward: -37.00, Steps:37, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 131/800, Reward: -49.00, Steps:49, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 132/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 133/800, Reward: -29.00, Steps:29, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 134/800, Reward: -35.00, Steps:35, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 135/800, Reward: -37.00, Steps:37, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 136/800, Reward: -43.00, Steps:43, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 137/800, Reward: -31.00, Steps:31, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 138/800, Reward: -51.00, Steps:51, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 139/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 140/800, Reward: -51.00, Steps:51, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 141/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 142/800, Reward: -35.00, Steps:35, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 143/800, Reward: -31.00, Steps:31, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 144/800, Reward: -41.00, Steps:41, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 145/800, Reward: -29.00, Steps:29, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 146/800, Reward: -27.00, Steps:27, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 147/800, Reward: -47.00, Steps:47, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 148/800, Reward: -27.00, Steps:27, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 149/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 150/800, Reward: -45.00, Steps:45, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 151/800, Reward: -31.00, Steps:31, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 152/800, Reward: -33.00, Steps:33, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 153/800, Reward: -31.00, Steps:31, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 154/800, Reward: -148.00, Steps:49, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 155/800, Reward: -41.00, Steps:41, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 156/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 157/800, Reward: -29.00, Steps:29, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 158/800, Reward: -31.00, Steps:31, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 159/800, Reward: -33.00, Steps:33, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 160/800, Reward: -27.00, Steps:27, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 161/800, Reward: -27.00, Steps:27, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 162/800, Reward: -29.00, Steps:29, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 163/800, Reward: -27.00, Steps:27, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 164/800, Reward: -27.00, Steps:27, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 165/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 166/800, Reward: -35.00, Steps:35, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 167/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 168/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 169/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 170/800, Reward: -41.00, Steps:41, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 171/800, Reward: -39.00, Steps:39, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 172/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 173/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 174/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 175/800, Reward: -31.00, Steps:31, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 176/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 177/800, Reward: -155.00, Steps:56, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 178/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 179/800, Reward: -37.00, Steps:37, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 180/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 181/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 182/800, Reward: -39.00, Steps:39, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 183/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 184/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 185/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 186/800, Reward: -29.00, Steps:29, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 187/800, Reward: -29.00, Steps:29, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 188/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 189/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 190/800, Reward: -42.00, Steps:42, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 191/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 192/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 193/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 194/800, Reward: -29.00, Steps:29, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 195/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 196/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 197/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 198/800, Reward: -29.00, Steps:29, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 199/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 200/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 201/800, Reward: -33.00, Steps:33, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 202/800, Reward: -37.00, Steps:37, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 203/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 204/800, Reward: -29.00, Steps:29, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 205/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 206/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 207/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 208/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 209/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 210/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 211/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 212/800, Reward: -37.00, Steps:37, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 213/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 214/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 215/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 216/800, Reward: -31.00, Steps:31, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 217/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 218/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 219/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 220/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 221/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 222/800, Reward: -27.00, Steps:27, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 223/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 224/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 225/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 226/800, Reward: -43.00, Steps:43, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 227/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 228/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 229/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 230/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 231/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 232/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 233/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 234/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 235/800, Reward: -31.00, Steps:31, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 236/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 237/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 238/800, Reward: -29.00, Steps:29, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 239/800, Reward: -30.00, Steps:30, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 240/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 241/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 242/800, Reward: -24.00, Steps:24, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 243/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 244/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 245/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 246/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 247/800, Reward: -29.00, Steps:29, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 248/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 249/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 250/800, Reward: -29.00, Steps:29, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 251/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 252/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 253/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 254/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 255/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 256/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 257/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 258/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 259/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 260/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 261/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 262/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 263/800, Reward: -27.00, Steps:27, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 264/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 265/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 266/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 267/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 268/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 269/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 270/800, Reward: -27.00, Steps:27, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 271/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 272/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 273/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 274/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 275/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 276/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 277/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 278/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 279/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 280/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 281/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 282/800, Reward: -27.00, Steps:27, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 283/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 284/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 285/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 286/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 287/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 288/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 289/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 290/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 291/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 292/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 293/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 294/800, Reward: -120.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 295/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 296/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 297/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 298/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 299/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 300/800, Reward: -35.00, Steps:35, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 301/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 302/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 303/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 304/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 305/800, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 306/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 307/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 308/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 309/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 310/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 311/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 312/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 313/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 314/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 315/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 316/800, Reward: -32.00, Steps:32, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 317/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 318/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 319/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 320/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 321/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 322/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 323/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 324/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 325/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 326/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 327/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 328/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 329/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 330/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 331/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 332/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 333/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 334/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 335/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 336/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 337/800, Reward: -26.00, Steps:26, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 338/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 339/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 340/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 341/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 342/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 343/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 344/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 345/800, Reward: -27.00, Steps:27, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 346/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 347/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 348/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 349/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 350/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 351/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 352/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 353/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 354/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 355/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 356/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 357/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 358/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 359/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 360/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 361/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 362/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 363/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 364/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 365/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 366/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 367/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 368/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 369/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 370/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 371/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 372/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 373/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 374/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 375/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 376/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 377/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 378/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 379/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 380/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 381/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 382/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 383/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 384/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 385/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 386/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 387/800, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 388/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 389/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 390/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 391/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 392/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 393/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 394/800, Reward: -122.00, Steps:23, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 395/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 396/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 397/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 398/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 399/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 400/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 401/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 402/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 403/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 404/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 405/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 406/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 407/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 408/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 409/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 410/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 411/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 412/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 413/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 414/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 415/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 416/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 417/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 418/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 419/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 420/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 421/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 422/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 423/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 424/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 425/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 426/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 427/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 428/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 429/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 430/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 431/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 432/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 433/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 434/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 435/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 436/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 437/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 438/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 439/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 440/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 441/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 442/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 443/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 444/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 445/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 446/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 447/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 448/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 449/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 450/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 451/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 452/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 453/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 454/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 455/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 456/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 457/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 458/800, Reward: -22.00, Steps:22, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 459/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 460/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 461/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 462/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 463/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 464/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 465/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 466/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 467/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 468/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 469/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 470/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 471/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 472/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 473/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 474/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 475/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 476/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 477/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 478/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 479/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 480/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 481/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 482/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 483/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 484/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 485/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 486/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 487/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 488/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 489/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 490/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 491/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 492/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 493/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 494/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 495/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 496/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 497/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 498/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 499/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 500/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 501/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 502/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 503/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 504/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 505/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 506/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 507/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 508/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 509/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 510/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 511/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 512/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 513/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 514/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 515/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 516/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 517/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 518/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 519/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 520/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 521/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 522/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 523/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 524/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 525/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 526/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 527/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 528/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 529/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 530/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 531/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 532/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 533/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 534/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 535/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 536/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 537/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 538/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 539/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 540/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 541/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 542/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 543/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 544/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 545/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 546/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 547/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 548/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 549/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 550/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 551/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 552/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 553/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 554/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 555/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 556/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 557/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 558/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:47 - r - INFO: - Episode: 559/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 560/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 561/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 562/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 563/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 564/800, Reward: -20.00, Steps:20, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 565/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 566/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 567/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 568/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 569/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 570/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 571/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 572/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 573/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 574/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 575/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 576/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 577/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 578/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 579/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 580/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 581/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 582/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 583/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 584/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 585/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 586/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 587/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 588/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 589/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 590/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 591/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 592/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 593/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 594/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 595/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 596/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 597/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 598/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 599/800, Reward: -16.00, Steps:16, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 600/800, Reward: -16.00, Steps:16, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 601/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 602/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 603/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 604/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 605/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 606/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 607/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 608/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 609/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 610/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 611/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 612/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 613/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 614/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 615/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 616/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 617/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 618/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 619/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 620/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 621/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 622/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 623/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 624/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 625/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 626/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 627/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 628/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 629/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 630/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 631/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 632/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 633/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 634/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 635/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 636/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 637/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 638/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 639/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 640/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 641/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 642/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 643/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 644/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 645/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 646/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 647/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 648/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 649/800, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 650/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 651/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 652/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 653/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 654/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 655/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 656/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 657/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 658/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 659/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 660/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 661/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 662/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 663/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 664/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 665/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 666/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 667/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 668/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 669/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 670/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 671/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 672/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 673/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 674/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 675/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 676/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 677/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 678/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 679/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 680/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 681/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 682/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 683/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 684/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 685/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 686/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 687/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 688/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 689/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 690/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 691/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 692/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 693/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 694/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 695/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 696/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 697/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 698/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 699/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 700/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 701/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 702/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 703/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 704/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 705/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 706/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 707/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 708/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 709/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 710/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 711/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 712/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 713/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 714/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 715/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 716/800, Reward: -16.00, Steps:16, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 717/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 718/800, Reward: -117.00, Steps:18, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 719/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 720/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 721/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 722/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 723/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 724/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 725/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 726/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 727/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 728/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 729/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 730/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 731/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 732/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 733/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 734/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 735/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 736/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 737/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 738/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 739/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 740/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 741/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 742/800, Reward: -16.00, Steps:16, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 743/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 744/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 745/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 746/800, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 747/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 748/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 749/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 750/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 751/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 752/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 753/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 754/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 755/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 756/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 757/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 758/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 759/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 760/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 761/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 762/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 763/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 764/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 765/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 766/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 767/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 768/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 769/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 770/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 771/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 772/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 773/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 774/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 775/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 776/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 777/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 778/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 779/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 780/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 781/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 782/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 783/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 784/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 785/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 786/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 787/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 788/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 789/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 790/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 791/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 792/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 793/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 794/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 795/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 796/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 797/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 798/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 799/800, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Episode: 800/800, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:11:48 - r - INFO: - Finish training! diff --git a/projects/codes/Sarsa/Train_CliffWalking-v0_Sarsa_20221030-021146/models/checkpoint.pkl b/projects/codes/Sarsa/Train_CliffWalking-v0_Sarsa_20221030-021146/models/checkpoint.pkl deleted file mode 100644 index d226d4c..0000000 Binary files a/projects/codes/Sarsa/Train_CliffWalking-v0_Sarsa_20221030-021146/models/checkpoint.pkl and /dev/null differ diff --git a/projects/codes/Sarsa/Train_CliffWalking-v0_Sarsa_20221030-021146/results/learning_curve.png b/projects/codes/Sarsa/Train_CliffWalking-v0_Sarsa_20221030-021146/results/learning_curve.png deleted file mode 100644 index 3c4dd0f..0000000 Binary files a/projects/codes/Sarsa/Train_CliffWalking-v0_Sarsa_20221030-021146/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/Sarsa/Train_CliffWalking-v0_Sarsa_20221030-021146/results/res.csv b/projects/codes/Sarsa/Train_CliffWalking-v0_Sarsa_20221030-021146/results/res.csv deleted file mode 100644 index 16858a4..0000000 --- a/projects/codes/Sarsa/Train_CliffWalking-v0_Sarsa_20221030-021146/results/res.csv +++ /dev/null @@ -1,801 +0,0 @@ -episodes,rewards,steps -0,-1091,200 -1,-320,122 -2,-794,200 -3,-596,200 -4,-398,200 -5,-59,59 -6,-299,200 -7,-82,82 -8,-125,125 -9,-75,75 -10,-285,186 -11,-103,103 -12,-103,103 -13,-131,131 -14,-53,53 -15,-113,113 -16,-125,125 -17,-95,95 -18,-97,97 -19,-145,145 -20,-89,89 -21,-97,97 -22,-115,115 -23,-121,121 -24,-53,53 -25,-111,111 -26,-97,97 -27,-206,107 -28,-147,147 -29,-36,36 -30,-216,117 -31,-103,103 -32,-87,87 -33,-80,80 -34,-73,73 -35,-83,83 -36,-143,44 -37,-241,142 -38,-77,77 -39,-49,49 -40,-87,87 -41,-47,47 -42,-89,89 -43,-31,31 -44,-192,93 -45,-85,85 -46,-55,55 -47,-59,59 -48,-60,60 -49,-50,50 -50,-23,23 -51,-101,101 -52,-43,43 -53,-70,70 -54,-35,35 -55,-47,47 -56,-80,80 -57,-61,61 -58,-35,35 -59,-73,73 -60,-54,54 -61,-37,37 -62,-65,65 -63,-41,41 -64,-81,81 -65,-39,39 -66,-35,35 -67,-61,61 -68,-57,57 -69,-43,43 -70,-59,59 -71,-43,43 -72,-51,51 -73,-43,43 -74,-69,69 -75,-41,41 -76,-194,95 -77,-35,35 -78,-35,35 -79,-81,81 -80,-65,65 -81,-35,35 -82,-47,47 -83,-53,53 -84,-165,66 -85,-69,69 -86,-35,35 -87,-56,56 -88,-164,65 -89,-45,45 -90,-43,43 -91,-43,43 -92,-29,29 -93,-69,69 -94,-33,33 -95,-57,57 -96,-29,29 -97,-55,55 -98,-61,61 -99,-162,63 -100,-55,55 -101,-31,31 -102,-53,53 -103,-39,39 -104,-55,55 -105,-25,25 -106,-33,33 -107,-49,49 -108,-65,65 -109,-45,45 -110,-29,29 -111,-25,25 -112,-51,51 -113,-43,43 -114,-47,47 -115,-41,41 -116,-27,27 -117,-31,31 -118,-180,81 -119,-43,43 -120,-25,25 -121,-47,47 -122,-65,65 -123,-29,29 -124,-31,31 -125,-49,49 -126,-25,25 -127,-45,45 -128,-49,49 -129,-37,37 -130,-49,49 -131,-25,25 -132,-29,29 -133,-35,35 -134,-37,37 -135,-43,43 -136,-31,31 -137,-51,51 -138,-25,25 -139,-51,51 -140,-21,21 -141,-35,35 -142,-31,31 -143,-41,41 -144,-29,29 -145,-27,27 -146,-47,47 -147,-27,27 -148,-23,23 -149,-45,45 -150,-31,31 -151,-33,33 -152,-31,31 -153,-148,49 -154,-41,41 -155,-25,25 -156,-29,29 -157,-31,31 -158,-33,33 -159,-27,27 -160,-27,27 -161,-29,29 -162,-27,27 -163,-27,27 -164,-23,23 -165,-35,35 -166,-21,21 -167,-23,23 -168,-23,23 -169,-41,41 -170,-39,39 -171,-21,21 -172,-25,25 -173,-23,23 -174,-31,31 -175,-21,21 -176,-155,56 -177,-21,21 -178,-37,37 -179,-17,17 -180,-19,19 -181,-39,39 -182,-25,25 -183,-25,25 -184,-19,19 -185,-29,29 -186,-29,29 -187,-25,25 -188,-25,25 -189,-42,42 -190,-21,21 -191,-21,21 -192,-25,25 -193,-29,29 -194,-15,15 -195,-21,21 -196,-17,17 -197,-29,29 -198,-25,25 -199,-15,15 -200,-33,33 -201,-37,37 -202,-17,17 -203,-29,29 -204,-17,17 -205,-25,25 -206,-23,23 -207,-25,25 -208,-25,25 -209,-25,25 -210,-21,21 -211,-37,37 -212,-17,17 -213,-17,17 -214,-23,23 -215,-31,31 -216,-21,21 -217,-21,21 -218,-21,21 -219,-19,19 -220,-17,17 -221,-27,27 -222,-23,23 -223,-15,15 -224,-19,19 -225,-43,43 -226,-15,15 -227,-25,25 -228,-19,19 -229,-19,19 -230,-19,19 -231,-19,19 -232,-21,21 -233,-23,23 -234,-31,31 -235,-23,23 -236,-19,19 -237,-29,29 -238,-30,30 -239,-19,19 -240,-17,17 -241,-24,24 -242,-15,15 -243,-23,23 -244,-21,21 -245,-15,15 -246,-29,29 -247,-19,19 -248,-17,17 -249,-29,29 -250,-21,21 -251,-25,25 -252,-25,25 -253,-19,19 -254,-25,25 -255,-21,21 -256,-19,19 -257,-19,19 -258,-15,15 -259,-21,21 -260,-19,19 -261,-23,23 -262,-27,27 -263,-19,19 -264,-21,21 -265,-21,21 -266,-23,23 -267,-17,17 -268,-25,25 -269,-27,27 -270,-19,19 -271,-19,19 -272,-21,21 -273,-15,15 -274,-21,21 -275,-17,17 -276,-15,15 -277,-23,23 -278,-17,17 -279,-19,19 -280,-23,23 -281,-27,27 -282,-17,17 -283,-23,23 -284,-19,19 -285,-17,17 -286,-17,17 -287,-15,15 -288,-21,21 -289,-15,15 -290,-21,21 -291,-19,19 -292,-17,17 -293,-120,21 -294,-21,21 -295,-15,15 -296,-19,19 -297,-15,15 -298,-15,15 -299,-35,35 -300,-21,21 -301,-21,21 -302,-15,15 -303,-15,15 -304,-23,23 -305,-17,17 -306,-21,21 -307,-17,17 -308,-15,15 -309,-15,15 -310,-15,15 -311,-17,17 -312,-19,19 -313,-25,25 -314,-19,19 -315,-32,32 -316,-15,15 -317,-17,17 -318,-21,21 -319,-15,15 -320,-17,17 -321,-19,19 -322,-17,17 -323,-17,17 -324,-17,17 -325,-17,17 -326,-19,19 -327,-15,15 -328,-15,15 -329,-21,21 -330,-19,19 -331,-15,15 -332,-17,17 -333,-17,17 -334,-15,15 -335,-19,19 -336,-26,26 -337,-17,17 -338,-15,15 -339,-21,21 -340,-17,17 -341,-15,15 -342,-19,19 -343,-17,17 -344,-27,27 -345,-15,15 -346,-17,17 -347,-15,15 -348,-17,17 -349,-17,17 -350,-19,19 -351,-15,15 -352,-15,15 -353,-15,15 -354,-15,15 -355,-17,17 -356,-17,17 -357,-15,15 -358,-19,19 -359,-15,15 -360,-17,17 -361,-17,17 -362,-19,19 -363,-17,17 -364,-17,17 -365,-21,21 -366,-17,17 -367,-15,15 -368,-17,17 -369,-21,21 -370,-19,19 -371,-17,17 -372,-15,15 -373,-15,15 -374,-25,25 -375,-15,15 -376,-15,15 -377,-17,17 -378,-17,17 -379,-17,17 -380,-15,15 -381,-15,15 -382,-15,15 -383,-15,15 -384,-17,17 -385,-17,17 -386,-25,25 -387,-17,17 -388,-15,15 -389,-15,15 -390,-17,17 -391,-15,15 -392,-15,15 -393,-122,23 -394,-15,15 -395,-15,15 -396,-15,15 -397,-15,15 -398,-21,21 -399,-15,15 -400,-17,17 -401,-17,17 -402,-17,17 -403,-15,15 -404,-15,15 -405,-15,15 -406,-17,17 -407,-15,15 -408,-15,15 -409,-17,17 -410,-15,15 -411,-15,15 -412,-17,17 -413,-19,19 -414,-17,17 -415,-17,17 -416,-17,17 -417,-15,15 -418,-15,15 -419,-17,17 -420,-15,15 -421,-15,15 -422,-15,15 -423,-21,21 -424,-15,15 -425,-15,15 -426,-17,17 -427,-15,15 -428,-17,17 -429,-15,15 -430,-15,15 -431,-15,15 -432,-15,15 -433,-15,15 -434,-21,21 -435,-15,15 -436,-15,15 -437,-17,17 -438,-15,15 -439,-15,15 -440,-15,15 -441,-17,17 -442,-15,15 -443,-15,15 -444,-17,17 -445,-15,15 -446,-17,17 -447,-17,17 -448,-15,15 -449,-17,17 -450,-15,15 -451,-17,17 -452,-15,15 -453,-15,15 -454,-15,15 -455,-15,15 -456,-15,15 -457,-22,22 -458,-15,15 -459,-15,15 -460,-15,15 -461,-15,15 -462,-15,15 -463,-15,15 -464,-15,15 -465,-15,15 -466,-15,15 -467,-15,15 -468,-15,15 -469,-15,15 -470,-15,15 -471,-17,17 -472,-21,21 -473,-15,15 -474,-15,15 -475,-15,15 -476,-15,15 -477,-17,17 -478,-15,15 -479,-17,17 -480,-17,17 -481,-15,15 -482,-15,15 -483,-15,15 -484,-17,17 -485,-21,21 -486,-15,15 -487,-15,15 -488,-15,15 -489,-15,15 -490,-15,15 -491,-17,17 -492,-15,15 -493,-17,17 -494,-19,19 -495,-15,15 -496,-15,15 -497,-15,15 -498,-15,15 -499,-15,15 -500,-15,15 -501,-15,15 -502,-15,15 -503,-15,15 -504,-15,15 -505,-15,15 -506,-15,15 -507,-15,15 -508,-17,17 -509,-15,15 -510,-15,15 -511,-15,15 -512,-15,15 -513,-15,15 -514,-15,15 -515,-15,15 -516,-17,17 -517,-15,15 -518,-15,15 -519,-15,15 -520,-15,15 -521,-15,15 -522,-15,15 -523,-15,15 -524,-15,15 -525,-15,15 -526,-15,15 -527,-15,15 -528,-15,15 -529,-15,15 -530,-15,15 -531,-17,17 -532,-17,17 -533,-15,15 -534,-17,17 -535,-15,15 -536,-15,15 -537,-15,15 -538,-15,15 -539,-15,15 -540,-15,15 -541,-15,15 -542,-19,19 -543,-15,15 -544,-15,15 -545,-15,15 -546,-17,17 -547,-15,15 -548,-15,15 -549,-15,15 -550,-15,15 -551,-15,15 -552,-15,15 -553,-15,15 -554,-15,15 -555,-15,15 -556,-15,15 -557,-15,15 -558,-15,15 -559,-15,15 -560,-15,15 -561,-15,15 -562,-15,15 -563,-20,20 -564,-17,17 -565,-15,15 -566,-15,15 -567,-15,15 -568,-15,15 -569,-15,15 -570,-17,17 -571,-17,17 -572,-15,15 -573,-15,15 -574,-15,15 -575,-15,15 -576,-17,17 -577,-15,15 -578,-15,15 -579,-15,15 -580,-15,15 -581,-15,15 -582,-15,15 -583,-17,17 -584,-15,15 -585,-15,15 -586,-15,15 -587,-15,15 -588,-15,15 -589,-15,15 -590,-15,15 -591,-15,15 -592,-15,15 -593,-15,15 -594,-17,17 -595,-15,15 -596,-15,15 -597,-15,15 -598,-16,16 -599,-16,16 -600,-15,15 -601,-15,15 -602,-15,15 -603,-15,15 -604,-15,15 -605,-17,17 -606,-15,15 -607,-15,15 -608,-15,15 -609,-15,15 -610,-15,15 -611,-15,15 -612,-15,15 -613,-15,15 -614,-15,15 -615,-15,15 -616,-15,15 -617,-15,15 -618,-15,15 -619,-15,15 -620,-15,15 -621,-15,15 -622,-15,15 -623,-15,15 -624,-15,15 -625,-15,15 -626,-15,15 -627,-15,15 -628,-15,15 -629,-15,15 -630,-15,15 -631,-15,15 -632,-15,15 -633,-21,21 -634,-15,15 -635,-15,15 -636,-15,15 -637,-15,15 -638,-15,15 -639,-15,15 -640,-15,15 -641,-15,15 -642,-15,15 -643,-15,15 -644,-15,15 -645,-15,15 -646,-17,17 -647,-15,15 -648,-21,21 -649,-15,15 -650,-15,15 -651,-15,15 -652,-15,15 -653,-17,17 -654,-15,15 -655,-15,15 -656,-15,15 -657,-15,15 -658,-15,15 -659,-15,15 -660,-15,15 -661,-15,15 -662,-15,15 -663,-15,15 -664,-17,17 -665,-15,15 -666,-15,15 -667,-15,15 -668,-15,15 -669,-15,15 -670,-17,17 -671,-15,15 -672,-15,15 -673,-15,15 -674,-15,15 -675,-15,15 -676,-15,15 -677,-17,17 -678,-15,15 -679,-15,15 -680,-15,15 -681,-15,15 -682,-15,15 -683,-15,15 -684,-15,15 -685,-15,15 -686,-15,15 -687,-15,15 -688,-15,15 -689,-15,15 -690,-15,15 -691,-15,15 -692,-15,15 -693,-15,15 -694,-15,15 -695,-15,15 -696,-15,15 -697,-15,15 -698,-15,15 -699,-15,15 -700,-15,15 -701,-15,15 -702,-15,15 -703,-17,17 -704,-15,15 -705,-15,15 -706,-15,15 -707,-15,15 -708,-15,15 -709,-15,15 -710,-17,17 -711,-15,15 -712,-15,15 -713,-15,15 -714,-15,15 -715,-16,16 -716,-15,15 -717,-117,18 -718,-15,15 -719,-17,17 -720,-15,15 -721,-15,15 -722,-15,15 -723,-15,15 -724,-15,15 -725,-15,15 -726,-15,15 -727,-15,15 -728,-15,15 -729,-15,15 -730,-15,15 -731,-15,15 -732,-15,15 -733,-15,15 -734,-15,15 -735,-15,15 -736,-15,15 -737,-15,15 -738,-15,15 -739,-15,15 -740,-15,15 -741,-16,16 -742,-15,15 -743,-17,17 -744,-15,15 -745,-19,19 -746,-15,15 -747,-15,15 -748,-15,15 -749,-17,17 -750,-15,15 -751,-15,15 -752,-17,17 -753,-15,15 -754,-15,15 -755,-15,15 -756,-15,15 -757,-17,17 -758,-15,15 -759,-15,15 -760,-15,15 -761,-17,17 -762,-15,15 -763,-15,15 -764,-15,15 -765,-15,15 -766,-15,15 -767,-17,17 -768,-15,15 -769,-15,15 -770,-15,15 -771,-15,15 -772,-15,15 -773,-15,15 -774,-17,17 -775,-15,15 -776,-15,15 -777,-15,15 -778,-15,15 -779,-15,15 -780,-15,15 -781,-15,15 -782,-15,15 -783,-15,15 -784,-15,15 -785,-15,15 -786,-15,15 -787,-15,15 -788,-15,15 -789,-15,15 -790,-15,15 -791,-15,15 -792,-17,17 -793,-15,15 -794,-15,15 -795,-15,15 -796,-15,15 -797,-15,15 -798,-17,17 -799,-15,15 diff --git a/projects/codes/Sarsa/Train_Racetrack-v0_Sarsa_20221030-021315/config.yaml b/projects/codes/Sarsa/Train_Racetrack-v0_Sarsa_20221030-021315/config.yaml deleted file mode 100644 index 79e3694..0000000 --- a/projects/codes/Sarsa/Train_Racetrack-v0_Sarsa_20221030-021315/config.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: Sarsa - device: cpu - env_name: Racetrack-v0 - load_checkpoint: false - load_path: Train_CartPole-v1_DQN_20221026-054757 - max_steps: 200 - mode: train - save_fig: true - seed: 10 - show_fig: false - test_eps: 20 - train_eps: 400 -algo_cfg: - epsilon_decay: 200 - epsilon_end: 0.01 - epsilon_start: 0.9 - gamma: 0.99 - lr: 0.1 diff --git a/projects/codes/Sarsa/Train_Racetrack-v0_Sarsa_20221030-021315/logs/log.txt b/projects/codes/Sarsa/Train_Racetrack-v0_Sarsa_20221030-021315/logs/log.txt deleted file mode 100644 index ffa79ca..0000000 --- a/projects/codes/Sarsa/Train_Racetrack-v0_Sarsa_20221030-021315/logs/log.txt +++ /dev/null @@ -1,404 +0,0 @@ -2022-10-30 02:13:15 - r - INFO: - n_states: 4, n_actions: 9 -2022-10-30 02:13:15 - r - INFO: - Start training! -2022-10-30 02:13:15 - r - INFO: - Env: Racetrack-v0, Algorithm: Sarsa, Device: cpu -2022-10-30 02:13:15 - r - INFO: - Episode: 1/400, Reward: -870.00, Steps:200, Epislon: 0.336 -2022-10-30 02:13:15 - r - INFO: - Episode: 2/400, Reward: -740.00, Steps:200, Epislon: 0.129 -2022-10-30 02:13:15 - r - INFO: - Episode: 3/400, Reward: -710.00, Steps:200, Epislon: 0.054 -2022-10-30 02:13:15 - r - INFO: - Episode: 4/400, Reward: -600.00, Steps:200, Epislon: 0.026 -2022-10-30 02:13:15 - r - INFO: - Episode: 5/400, Reward: -580.00, Steps:200, Epislon: 0.016 -2022-10-30 02:13:15 - r - INFO: - Episode: 6/400, Reward: -620.00, Steps:200, Epislon: 0.012 -2022-10-30 02:13:16 - r - INFO: - Episode: 7/400, Reward: -590.00, Steps:200, Epislon: 0.011 -2022-10-30 02:13:16 - r - INFO: - Episode: 8/400, Reward: -590.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 9/400, Reward: -520.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 10/400, Reward: -570.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 11/400, Reward: -580.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 12/400, Reward: -580.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 13/400, Reward: -500.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 14/400, Reward: -540.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 15/400, Reward: -510.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 16/400, Reward: -570.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 17/400, Reward: -560.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 18/400, Reward: -540.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 19/400, Reward: -490.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 20/400, Reward: -490.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 21/400, Reward: -530.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 22/400, Reward: -520.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 23/400, Reward: -530.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 24/400, Reward: -520.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 25/400, Reward: -500.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 26/400, Reward: -510.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 27/400, Reward: -520.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:16 - r - INFO: - Episode: 28/400, Reward: -530.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 29/400, Reward: -560.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 30/400, Reward: -490.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 31/400, Reward: -530.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 32/400, Reward: -359.00, Steps:149, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 33/400, Reward: -470.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 34/400, Reward: -510.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 35/400, Reward: -520.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 36/400, Reward: -500.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 37/400, Reward: -540.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 38/400, Reward: -560.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 39/400, Reward: -500.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 40/400, Reward: -480.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 41/400, Reward: -490.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 42/400, Reward: -480.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 43/400, Reward: -540.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 44/400, Reward: -500.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 45/400, Reward: -500.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 46/400, Reward: -480.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 47/400, Reward: -550.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 48/400, Reward: -490.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 49/400, Reward: -540.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 50/400, Reward: -420.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 51/400, Reward: -530.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:17 - r - INFO: - Episode: 52/400, Reward: -510.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 53/400, Reward: -530.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 54/400, Reward: -460.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 55/400, Reward: -480.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 56/400, Reward: -480.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 57/400, Reward: -470.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 58/400, Reward: -490.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 59/400, Reward: -470.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 60/400, Reward: -500.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 61/400, Reward: -500.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 62/400, Reward: -480.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 63/400, Reward: -450.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 64/400, Reward: -490.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 65/400, Reward: -420.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 66/400, Reward: -480.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 67/400, Reward: -440.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 68/400, Reward: -490.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 69/400, Reward: -188.00, Steps:88, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 70/400, Reward: -327.00, Steps:167, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 71/400, Reward: -530.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 72/400, Reward: -48.00, Steps:28, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 73/400, Reward: -460.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 74/400, Reward: -460.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 75/400, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 76/400, Reward: -428.00, Steps:178, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 77/400, Reward: -460.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 78/400, Reward: -341.00, Steps:151, Epislon: 0.010 -2022-10-30 02:13:18 - r - INFO: - Episode: 79/400, Reward: -480.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 80/400, Reward: -346.00, Steps:156, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 81/400, Reward: -34.00, Steps:24, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 82/400, Reward: -480.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 83/400, Reward: -480.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 84/400, Reward: -222.00, Steps:112, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 85/400, Reward: -470.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 86/400, Reward: -409.00, Steps:169, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 87/400, Reward: -139.00, Steps:59, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 88/400, Reward: -520.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 89/400, Reward: -108.00, Steps:58, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 90/400, Reward: -3.00, Steps:13, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 91/400, Reward: -131.00, Steps:71, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 92/400, Reward: -355.00, Steps:145, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 93/400, Reward: -470.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 94/400, Reward: -450.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 95/400, Reward: -490.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 96/400, Reward: -425.00, Steps:185, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 97/400, Reward: -130.00, Steps:70, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 98/400, Reward: -246.00, Steps:116, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 99/400, Reward: -480.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 100/400, Reward: -500.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 101/400, Reward: -13.00, Steps:13, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 102/400, Reward: -63.00, Steps:33, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 103/400, Reward: -311.00, Steps:131, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 104/400, Reward: -450.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 105/400, Reward: -520.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 106/400, Reward: -430.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 107/400, Reward: -79.00, Steps:39, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 108/400, Reward: -94.00, Steps:44, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 109/400, Reward: -37.00, Steps:27, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 110/400, Reward: -235.00, Steps:115, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 111/400, Reward: -440.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 112/400, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 113/400, Reward: -424.00, Steps:194, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 114/400, Reward: -470.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:19 - r - INFO: - Episode: 115/400, Reward: -344.00, Steps:164, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 116/400, Reward: -307.00, Steps:147, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 117/400, Reward: -82.00, Steps:52, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 118/400, Reward: -387.00, Steps:177, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 119/400, Reward: -500.00, Steps:200, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 120/400, Reward: -315.00, Steps:145, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 121/400, Reward: -289.00, Steps:119, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 122/400, Reward: -139.00, Steps:79, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 123/400, Reward: -392.00, Steps:192, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 124/400, Reward: -13.00, Steps:13, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 125/400, Reward: -35.00, Steps:25, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 126/400, Reward: -82.00, Steps:42, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 127/400, Reward: -134.00, Steps:64, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 128/400, Reward: -93.00, Steps:53, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 129/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 130/400, Reward: -212.00, Steps:102, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 131/400, Reward: -87.00, Steps:47, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 132/400, Reward: -70.00, Steps:40, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 133/400, Reward: -109.00, Steps:49, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 134/400, Reward: -77.00, Steps:47, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 135/400, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 136/400, Reward: -118.00, Steps:58, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 137/400, Reward: -132.00, Steps:62, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 138/400, Reward: -76.00, Steps:36, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 139/400, Reward: -93.00, Steps:63, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 140/400, Reward: -357.00, Steps:157, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 141/400, Reward: -129.00, Steps:69, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 142/400, Reward: -46.00, Steps:26, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 143/400, Reward: -60.00, Steps:30, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 144/400, Reward: -339.00, Steps:159, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 145/400, Reward: -10.00, Steps:10, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 146/400, Reward: -164.00, Steps:84, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 147/400, Reward: -145.00, Steps:75, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 148/400, Reward: -53.00, Steps:33, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 149/400, Reward: -3.00, Steps:13, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 150/400, Reward: -55.00, Steps:35, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 151/400, Reward: -398.00, Steps:178, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 152/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 153/400, Reward: -20.00, Steps:20, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 154/400, Reward: -354.00, Steps:154, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 155/400, Reward: -439.00, Steps:189, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 156/400, Reward: -122.00, Steps:62, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 157/400, Reward: -80.00, Steps:40, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 158/400, Reward: -29.00, Steps:19, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 159/400, Reward: -185.00, Steps:85, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 160/400, Reward: -354.00, Steps:154, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 161/400, Reward: -35.00, Steps:25, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 162/400, Reward: -132.00, Steps:62, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 163/400, Reward: -155.00, Steps:75, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 164/400, Reward: -261.00, Steps:111, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 165/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 166/400, Reward: -135.00, Steps:65, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 167/400, Reward: -57.00, Steps:37, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 168/400, Reward: -432.00, Steps:182, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 169/400, Reward: -63.00, Steps:33, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 170/400, Reward: -119.00, Steps:59, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 171/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 172/400, Reward: -16.00, Steps:16, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 173/400, Reward: -112.00, Steps:62, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 174/400, Reward: 1.00, Steps:9, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 175/400, Reward: -354.00, Steps:164, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 176/400, Reward: -101.00, Steps:61, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 177/400, Reward: -86.00, Steps:46, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 178/400, Reward: -33.00, Steps:23, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 179/400, Reward: -339.00, Steps:139, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 180/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 181/400, Reward: -9.00, Steps:9, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 182/400, Reward: -224.00, Steps:104, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 183/400, Reward: -11.00, Steps:11, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 184/400, Reward: -52.00, Steps:32, Epislon: 0.010 -2022-10-30 02:13:20 - r - INFO: - Episode: 185/400, Reward: -98.00, Steps:48, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 186/400, Reward: -26.00, Steps:16, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 187/400, Reward: -89.00, Steps:39, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 188/400, Reward: 1.00, Steps:9, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 189/400, Reward: -66.00, Steps:36, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 190/400, Reward: -77.00, Steps:37, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 191/400, Reward: 5.00, Steps:5, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 192/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 193/400, Reward: -64.00, Steps:34, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 194/400, Reward: 5.00, Steps:5, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 195/400, Reward: -10.00, Steps:10, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 196/400, Reward: -79.00, Steps:39, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 197/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 198/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 199/400, Reward: 0.00, Steps:10, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 200/400, Reward: -33.00, Steps:23, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 201/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 202/400, Reward: 5.00, Steps:5, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 203/400, Reward: -110.00, Steps:50, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 204/400, Reward: -43.00, Steps:23, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 205/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 206/400, Reward: -13.00, Steps:13, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 207/400, Reward: 1.00, Steps:9, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 208/400, Reward: -32.00, Steps:22, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 209/400, Reward: -77.00, Steps:37, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 210/400, Reward: 5.00, Steps:5, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 211/400, Reward: -23.00, Steps:23, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 212/400, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 213/400, Reward: 4.00, Steps:6, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 214/400, Reward: 1.00, Steps:9, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 215/400, Reward: -42.00, Steps:22, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 216/400, Reward: -13.00, Steps:13, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 217/400, Reward: -64.00, Steps:34, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 218/400, Reward: -13.00, Steps:13, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 219/400, Reward: -2.00, Steps:12, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 220/400, Reward: 5.00, Steps:5, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 221/400, Reward: -129.00, Steps:69, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 222/400, Reward: -133.00, Steps:63, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 223/400, Reward: -47.00, Steps:37, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 224/400, Reward: -11.00, Steps:11, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 225/400, Reward: -25.00, Steps:25, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 226/400, Reward: -1.00, Steps:11, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 227/400, Reward: 5.00, Steps:5, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 228/400, Reward: -103.00, Steps:53, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 229/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 230/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 231/400, Reward: -67.00, Steps:37, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 232/400, Reward: -65.00, Steps:35, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 233/400, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 234/400, Reward: -30.00, Steps:20, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 235/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 236/400, Reward: 4.00, Steps:6, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 237/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 238/400, Reward: -13.00, Steps:13, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 239/400, Reward: 1.00, Steps:9, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 240/400, Reward: -16.00, Steps:16, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 241/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 242/400, Reward: -39.00, Steps:29, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 243/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 244/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 245/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 246/400, Reward: -13.00, Steps:13, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 247/400, Reward: 5.00, Steps:5, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 248/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 249/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 250/400, Reward: -12.00, Steps:12, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 251/400, Reward: -14.00, Steps:14, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 252/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 253/400, Reward: -57.00, Steps:37, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 254/400, Reward: -29.00, Steps:19, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 255/400, Reward: 4.00, Steps:6, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 256/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 257/400, Reward: -13.00, Steps:13, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 258/400, Reward: -40.00, Steps:30, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 259/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 260/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 261/400, Reward: -30.00, Steps:20, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 262/400, Reward: -34.00, Steps:24, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 263/400, Reward: -1.00, Steps:11, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 264/400, Reward: -13.00, Steps:13, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 265/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 266/400, Reward: 5.00, Steps:5, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 267/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 268/400, Reward: -42.00, Steps:32, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 269/400, Reward: -17.00, Steps:17, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 270/400, Reward: -12.00, Steps:12, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 271/400, Reward: -28.00, Steps:18, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 272/400, Reward: -13.00, Steps:13, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 273/400, Reward: -2.00, Steps:12, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 274/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 275/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 276/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 277/400, Reward: -14.00, Steps:14, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 278/400, Reward: -14.00, Steps:14, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 279/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 280/400, Reward: 4.00, Steps:6, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 281/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 282/400, Reward: 5.00, Steps:5, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 283/400, Reward: -13.00, Steps:13, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 284/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 285/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 286/400, Reward: 5.00, Steps:5, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 287/400, Reward: -1.00, Steps:11, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 288/400, Reward: -39.00, Steps:29, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 289/400, Reward: 5.00, Steps:5, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 290/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 291/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 292/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 293/400, Reward: -11.00, Steps:11, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 294/400, Reward: -30.00, Steps:20, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 295/400, Reward: -18.00, Steps:18, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 296/400, Reward: -13.00, Steps:13, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 297/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 298/400, Reward: 5.00, Steps:5, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 299/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 300/400, Reward: 4.00, Steps:6, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 301/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 302/400, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 303/400, Reward: -14.00, Steps:14, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 304/400, Reward: -13.00, Steps:13, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 305/400, Reward: -55.00, Steps:35, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 306/400, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 307/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 308/400, Reward: -12.00, Steps:12, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 309/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 310/400, Reward: -67.00, Steps:37, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 311/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 312/400, Reward: -20.00, Steps:20, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 313/400, Reward: 4.00, Steps:6, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 314/400, Reward: 5.00, Steps:5, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 315/400, Reward: -20.00, Steps:20, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 316/400, Reward: -36.00, Steps:26, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 317/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 318/400, Reward: 4.00, Steps:6, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 319/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 320/400, Reward: -12.00, Steps:12, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 321/400, Reward: 5.00, Steps:5, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 322/400, Reward: 4.00, Steps:6, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 323/400, Reward: -16.00, Steps:16, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 324/400, Reward: 4.00, Steps:6, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 325/400, Reward: -18.00, Steps:18, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 326/400, Reward: -36.00, Steps:26, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 327/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 328/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 329/400, Reward: -28.00, Steps:18, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 330/400, Reward: -31.00, Steps:21, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 331/400, Reward: -1.00, Steps:11, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 332/400, Reward: -109.00, Steps:59, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 333/400, Reward: -29.00, Steps:19, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 334/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 335/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 336/400, Reward: 0.00, Steps:10, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 337/400, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 338/400, Reward: -12.00, Steps:12, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 339/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 340/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 341/400, Reward: -14.00, Steps:14, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 342/400, Reward: -35.00, Steps:25, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 343/400, Reward: -16.00, Steps:16, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 344/400, Reward: -21.00, Steps:21, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 345/400, Reward: -28.00, Steps:18, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 346/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 347/400, Reward: -12.00, Steps:12, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 348/400, Reward: -28.00, Steps:18, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 349/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 350/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 351/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 352/400, Reward: -10.00, Steps:10, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 353/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 354/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 355/400, Reward: -62.00, Steps:32, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 356/400, Reward: 5.00, Steps:5, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 357/400, Reward: 4.00, Steps:6, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 358/400, Reward: -28.00, Steps:18, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 359/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 360/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 361/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 362/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 363/400, Reward: -16.00, Steps:16, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 364/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 365/400, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 366/400, Reward: -18.00, Steps:18, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 367/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 368/400, Reward: -18.00, Steps:18, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 369/400, Reward: -15.00, Steps:15, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 370/400, Reward: 5.00, Steps:5, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 371/400, Reward: -29.00, Steps:19, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 372/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 373/400, Reward: -14.00, Steps:14, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 374/400, Reward: 1.00, Steps:9, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 375/400, Reward: -19.00, Steps:19, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 376/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 377/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 378/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 379/400, Reward: -31.00, Steps:21, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 380/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 381/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 382/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 383/400, Reward: -14.00, Steps:14, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 384/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 385/400, Reward: 2.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 386/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 387/400, Reward: 4.00, Steps:6, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 388/400, Reward: -8.00, Steps:8, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 389/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 390/400, Reward: 4.00, Steps:6, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 391/400, Reward: -13.00, Steps:13, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 392/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 393/400, Reward: -12.00, Steps:12, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 394/400, Reward: -32.00, Steps:22, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 395/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 396/400, Reward: -27.00, Steps:17, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 397/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 398/400, Reward: 3.00, Steps:7, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 399/400, Reward: -37.00, Steps:27, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Episode: 400/400, Reward: -57.00, Steps:37, Epislon: 0.010 -2022-10-30 02:13:21 - r - INFO: - Finish training! diff --git a/projects/codes/Sarsa/Train_Racetrack-v0_Sarsa_20221030-021315/models/checkpoint.pkl b/projects/codes/Sarsa/Train_Racetrack-v0_Sarsa_20221030-021315/models/checkpoint.pkl deleted file mode 100644 index d950b3f..0000000 Binary files a/projects/codes/Sarsa/Train_Racetrack-v0_Sarsa_20221030-021315/models/checkpoint.pkl and /dev/null differ diff --git a/projects/codes/Sarsa/Train_Racetrack-v0_Sarsa_20221030-021315/results/learning_curve.png b/projects/codes/Sarsa/Train_Racetrack-v0_Sarsa_20221030-021315/results/learning_curve.png deleted file mode 100644 index 0626795..0000000 Binary files a/projects/codes/Sarsa/Train_Racetrack-v0_Sarsa_20221030-021315/results/learning_curve.png and /dev/null differ diff --git a/projects/codes/Sarsa/Train_Racetrack-v0_Sarsa_20221030-021315/results/res.csv b/projects/codes/Sarsa/Train_Racetrack-v0_Sarsa_20221030-021315/results/res.csv deleted file mode 100644 index d251623..0000000 --- a/projects/codes/Sarsa/Train_Racetrack-v0_Sarsa_20221030-021315/results/res.csv +++ /dev/null @@ -1,401 +0,0 @@ -episodes,rewards,steps -0,-870,200 -1,-740,200 -2,-710,200 -3,-600,200 -4,-580,200 -5,-620,200 -6,-590,200 -7,-590,200 -8,-520,200 -9,-570,200 -10,-580,200 -11,-580,200 -12,-500,200 -13,-540,200 -14,-510,200 -15,-570,200 -16,-560,200 -17,-540,200 -18,-490,200 -19,-490,200 -20,-530,200 -21,-520,200 -22,-530,200 -23,-520,200 -24,-500,200 -25,-510,200 -26,-520,200 -27,-530,200 -28,-560,200 -29,-490,200 -30,-530,200 -31,-359,149 -32,-470,200 -33,-510,200 -34,-520,200 -35,-500,200 -36,-540,200 -37,-560,200 -38,-500,200 -39,-480,200 -40,-490,200 -41,-480,200 -42,-540,200 -43,-500,200 -44,-500,200 -45,-480,200 -46,-550,200 -47,-490,200 -48,-540,200 -49,-420,200 -50,-530,200 -51,-510,200 -52,-530,200 -53,-460,200 -54,-480,200 -55,-480,200 -56,-470,200 -57,-490,200 -58,-470,200 -59,-500,200 -60,-500,200 -61,-480,200 -62,-450,200 -63,-490,200 -64,-420,200 -65,-480,200 -66,-440,200 -67,-490,200 -68,-188,88 -69,-327,167 -70,-530,200 -71,-48,28 -72,-460,200 -73,-460,200 -74,-25,25 -75,-428,178 -76,-460,200 -77,-341,151 -78,-480,200 -79,-346,156 -80,-34,24 -81,-480,200 -82,-480,200 -83,-222,112 -84,-470,200 -85,-409,169 -86,-139,59 -87,-520,200 -88,-108,58 -89,-3,13 -90,-131,71 -91,-355,145 -92,-470,200 -93,-450,200 -94,-490,200 -95,-425,185 -96,-130,70 -97,-246,116 -98,-480,200 -99,-500,200 -100,-13,13 -101,-63,33 -102,-311,131 -103,-450,200 -104,-520,200 -105,-430,200 -106,-79,39 -107,-94,44 -108,-37,27 -109,-235,115 -110,-440,200 -111,-19,19 -112,-424,194 -113,-470,200 -114,-344,164 -115,-307,147 -116,-82,52 -117,-387,177 -118,-500,200 -119,-315,145 -120,-289,119 -121,-139,79 -122,-392,192 -123,-13,13 -124,-35,25 -125,-82,42 -126,-134,64 -127,-93,53 -128,2,8 -129,-212,102 -130,-87,47 -131,-70,40 -132,-109,49 -133,-77,47 -134,-17,17 -135,-118,58 -136,-132,62 -137,-76,36 -138,-93,63 -139,-357,157 -140,-129,69 -141,-46,26 -142,-60,30 -143,-339,159 -144,-10,10 -145,-164,84 -146,-145,75 -147,-53,33 -148,-3,13 -149,-55,35 -150,-398,178 -151,3,7 -152,-20,20 -153,-354,154 -154,-439,189 -155,-122,62 -156,-80,40 -157,-29,19 -158,-185,85 -159,-354,154 -160,-35,25 -161,-132,62 -162,-155,75 -163,-261,111 -164,3,7 -165,-135,65 -166,-57,37 -167,-432,182 -168,-63,33 -169,-119,59 -170,3,7 -171,-16,16 -172,-112,62 -173,1,9 -174,-354,164 -175,-101,61 -176,-86,46 -177,-33,23 -178,-339,139 -179,3,7 -180,-9,9 -181,-224,104 -182,-11,11 -183,-52,32 -184,-98,48 -185,-26,16 -186,-89,39 -187,1,9 -188,-66,36 -189,-77,37 -190,5,5 -191,2,8 -192,-64,34 -193,5,5 -194,-10,10 -195,-79,39 -196,3,7 -197,3,7 -198,0,10 -199,-33,23 -200,2,8 -201,5,5 -202,-110,50 -203,-43,23 -204,3,7 -205,-13,13 -206,1,9 -207,-32,22 -208,-77,37 -209,5,5 -210,-23,23 -211,-15,15 -212,4,6 -213,1,9 -214,-42,22 -215,-13,13 -216,-64,34 -217,-13,13 -218,-2,12 -219,5,5 -220,-129,69 -221,-133,63 -222,-47,37 -223,-11,11 -224,-25,25 -225,-1,11 -226,5,5 -227,-103,53 -228,3,7 -229,2,8 -230,-67,37 -231,-65,35 -232,-15,15 -233,-30,20 -234,3,7 -235,4,6 -236,3,7 -237,-13,13 -238,1,9 -239,-16,16 -240,3,7 -241,-39,29 -242,3,7 -243,3,7 -244,3,7 -245,-13,13 -246,5,5 -247,3,7 -248,2,8 -249,-12,12 -250,-14,14 -251,2,8 -252,-57,37 -253,-29,19 -254,4,6 -255,2,8 -256,-13,13 -257,-40,30 -258,3,7 -259,3,7 -260,-30,20 -261,-34,24 -262,-1,11 -263,-13,13 -264,2,8 -265,5,5 -266,3,7 -267,-42,32 -268,-17,17 -269,-12,12 -270,-28,18 -271,-13,13 -272,-2,12 -273,3,7 -274,3,7 -275,3,7 -276,-14,14 -277,-14,14 -278,3,7 -279,4,6 -280,3,7 -281,5,5 -282,-13,13 -283,3,7 -284,2,8 -285,5,5 -286,-1,11 -287,-39,29 -288,5,5 -289,3,7 -290,3,7 -291,3,7 -292,-11,11 -293,-30,20 -294,-18,18 -295,-13,13 -296,2,8 -297,5,5 -298,3,7 -299,4,6 -300,2,8 -301,-15,15 -302,-14,14 -303,-13,13 -304,-55,35 -305,-19,19 -306,3,7 -307,-12,12 -308,3,7 -309,-67,37 -310,3,7 -311,-20,20 -312,4,6 -313,5,5 -314,-20,20 -315,-36,26 -316,3,7 -317,4,6 -318,2,8 -319,-12,12 -320,5,5 -321,4,6 -322,-16,16 -323,4,6 -324,-18,18 -325,-36,26 -326,3,7 -327,3,7 -328,-28,18 -329,-31,21 -330,-1,11 -331,-109,59 -332,-29,19 -333,3,7 -334,3,7 -335,0,10 -336,-15,15 -337,-12,12 -338,3,7 -339,3,7 -340,-14,14 -341,-35,25 -342,-16,16 -343,-21,21 -344,-28,18 -345,2,8 -346,-12,12 -347,-28,18 -348,3,7 -349,3,7 -350,3,7 -351,-10,10 -352,3,7 -353,3,7 -354,-62,32 -355,5,5 -356,4,6 -357,-28,18 -358,3,7 -359,3,7 -360,2,8 -361,2,8 -362,-16,16 -363,2,8 -364,-15,15 -365,-18,18 -366,3,7 -367,-18,18 -368,-15,15 -369,5,5 -370,-29,19 -371,3,7 -372,-14,14 -373,1,9 -374,-19,19 -375,3,7 -376,3,7 -377,3,7 -378,-31,21 -379,2,8 -380,3,7 -381,3,7 -382,-14,14 -383,3,7 -384,2,8 -385,3,7 -386,4,6 -387,-8,8 -388,3,7 -389,4,6 -390,-13,13 -391,3,7 -392,-12,12 -393,-32,22 -394,3,7 -395,-27,17 -396,3,7 -397,3,7 -398,-37,27 -399,-57,37 diff --git a/projects/codes/Sarsa/assets/sarsa_algo.png b/projects/codes/Sarsa/assets/sarsa_algo.png deleted file mode 100644 index 0abef7a..0000000 Binary files a/projects/codes/Sarsa/assets/sarsa_algo.png and /dev/null differ diff --git a/projects/codes/Sarsa/config/CliffWalking-v0_Sarsa_Test.yaml b/projects/codes/Sarsa/config/CliffWalking-v0_Sarsa_Test.yaml deleted file mode 100644 index f39b31b..0000000 --- a/projects/codes/Sarsa/config/CliffWalking-v0_Sarsa_Test.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: Sarsa - device: cpu - env_name: CliffWalking-v0 - mode: test - load_checkpoint: true - load_path: Train_CliffWalking-v0_Sarsa_20221030-021146 - max_steps: 200 - save_fig: true - seed: 1 - show_fig: false - test_eps: 20 - train_eps: 400 -algo_cfg: - epsilon_decay: 300 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - lr: 0.1 diff --git a/projects/codes/Sarsa/config/CliffWalking-v0_Sarsa_Train.yaml b/projects/codes/Sarsa/config/CliffWalking-v0_Sarsa_Train.yaml deleted file mode 100644 index 630ead8..0000000 --- a/projects/codes/Sarsa/config/CliffWalking-v0_Sarsa_Train.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: Sarsa - device: cpu - env_name: CliffWalking-v0 - mode: train - load_checkpoint: false - load_path: Train_CartPole-v1_DQN_20221026-054757 - max_steps: 200 - save_fig: true - seed: 1 - show_fig: false - test_eps: 20 - train_eps: 800 -algo_cfg: - epsilon_decay: 300 - epsilon_end: 0.01 - epsilon_start: 0.95 - gamma: 0.95 - lr: 0.1 diff --git a/projects/codes/Sarsa/config/Racetrack-v0_Sarsa_Test.yaml b/projects/codes/Sarsa/config/Racetrack-v0_Sarsa_Test.yaml deleted file mode 100644 index e07a7e2..0000000 --- a/projects/codes/Sarsa/config/Racetrack-v0_Sarsa_Test.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: Sarsa - device: cpu - env_name: Racetrack-v0 - mode: test - load_checkpoint: true - load_path: Train_Racetrack-v0_Sarsa_20221030-021315 - max_steps: 200 - save_fig: true - seed: 10 - show_fig: false - test_eps: 20 - train_eps: 400 -algo_cfg: - epsilon_decay: 200 - epsilon_end: 0.01 - epsilon_start: 0.9 - gamma: 0.99 - lr: 0.1 diff --git a/projects/codes/Sarsa/config/Racetrack-v0_Sarsa_Train.yaml b/projects/codes/Sarsa/config/Racetrack-v0_Sarsa_Train.yaml deleted file mode 100644 index da6299f..0000000 --- a/projects/codes/Sarsa/config/Racetrack-v0_Sarsa_Train.yaml +++ /dev/null @@ -1,19 +0,0 @@ -general_cfg: - algo_name: Sarsa - device: cpu - env_name: Racetrack-v0 - mode: train - load_checkpoint: false - load_path: Train_CartPole-v1_DQN_20221026-054757 - max_steps: 200 - save_fig: true - seed: 10 - show_fig: false - test_eps: 20 - train_eps: 400 -algo_cfg: - epsilon_decay: 200 - epsilon_end: 0.01 - epsilon_start: 0.9 - gamma: 0.99 - lr: 0.1 diff --git a/projects/codes/Sarsa/config/config.py b/projects/codes/Sarsa/config/config.py deleted file mode 100644 index 9980c04..0000000 --- a/projects/codes/Sarsa/config/config.py +++ /dev/null @@ -1,35 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-10-30 01:23:07 -LastEditor: JiangJi -LastEditTime: 2022-10-30 02:01:54 -Discription: default parameters of QLearning -''' -from common.config import GeneralConfig,AlgoConfig - -class GeneralConfigSarsa(GeneralConfig): - def __init__(self) -> None: - self.env_name = "CliffWalking-v0" # name of environment - self.algo_name = "Sarsa" # name of algorithm - self.mode = "train" # train or test - self.seed = 1 # random seed - self.device = "cpu" # device to use - self.train_eps = 400 # number of episodes for training - self.test_eps = 20 # number of episodes for testing - self.max_steps = 200 # max steps for each episode - self.load_checkpoint = False - self.load_path = "tasks" # path to load model - self.show_fig = False # show figure or not - self.save_fig = True # save figure or not - -class AlgoConfigSarsa(AlgoConfig): - def __init__(self) -> None: - # set epsilon_start=epsilon_end can obtain fixed epsilon=epsilon_end - self.epsilon_start = 0.95 # epsilon start value - self.epsilon_end = 0.01 # epsilon end value - self.epsilon_decay = 300 # epsilon decay rate - self.gamma = 0.90 # discount factor - self.lr = 0.1 # learning rate \ No newline at end of file diff --git a/projects/codes/Sarsa/sarsa.py b/projects/codes/Sarsa/sarsa.py deleted file mode 100644 index 753ee95..0000000 --- a/projects/codes/Sarsa/sarsa.py +++ /dev/null @@ -1,64 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: John -Email: johnjim0816@gmail.com -Date: 2021-03-12 16:58:16 -LastEditor: John -LastEditTime: 2022-10-30 02:00:51 -Discription: -Environment: -''' -import numpy as np -from collections import defaultdict -import torch -import math -class Sarsa(object): - def __init__(self,cfg): - self.n_actions = cfg.n_actions - self.lr = cfg.lr - self.gamma = cfg.gamma - self.epsilon = cfg.epsilon_start - self.sample_count = 0 - self.epsilon_start = cfg.epsilon_start - self.epsilon_end = cfg.epsilon_end - self.epsilon_decay = cfg.epsilon_decay - self.Q_table = defaultdict(lambda: np.zeros(self.n_actions)) # Q table - def sample_action(self, state): - ''' another way to represent e-greedy policy - ''' - self.sample_count += 1 - self.epsilon = self.epsilon_end + (self.epsilon_start - self.epsilon_end) * \ - math.exp(-1. * self.sample_count / self.epsilon_decay) # The probability to select a random action, is is log decayed - best_action = np.argmax(self.Q_table[str(state)]) # array cannot be hashtable, thus convert to str - action_probs = np.ones(self.n_actions, dtype=float) * self.epsilon / self.n_actions - action_probs[best_action] += (1.0 - self.epsilon) - action = np.random.choice(np.arange(len(action_probs)), p=action_probs) - return action - def predict_action(self,state): - ''' predict action while testing - ''' - action = np.argmax(self.Q_table[str(state)]) - return action - def update(self, state, action, reward, next_state, next_action,done): - Q_predict = self.Q_table[str(state)][action] - if done: - Q_target = reward # terminal state - else: - Q_target = reward + self.gamma * self.Q_table[str(next_state)][next_action] # the only difference from Q learning - self.Q_table[str(state)][action] += self.lr * (Q_target - Q_predict) - def save_model(self,path): - import dill - from pathlib import Path - # create path - Path(path).mkdir(parents=True, exist_ok=True) - torch.save( - obj=self.Q_table, - f=path+"checkpoint.pkl", - pickle_module=dill - ) - print("Model saved!") - def load_model(self, path): - import dill - self.Q_table=torch.load(f=path+'checkpoint.pkl',pickle_module=dill) - print("Mode loaded!") \ No newline at end of file diff --git a/projects/codes/Sarsa/task0.py b/projects/codes/Sarsa/task0.py deleted file mode 100644 index bdd5fda..0000000 --- a/projects/codes/Sarsa/task0.py +++ /dev/null @@ -1,106 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2022-09-19 14:48:16 -LastEditor: JiangJi -LastEditTime: 2022-10-30 02:11:31 -Discription: -''' -import sys,os -os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE" # avoid "OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized." -curr_path = os.path.dirname(os.path.abspath(__file__)) # current path -parent_path = os.path.dirname(curr_path) # parent path -sys.path.append(parent_path) # add path to system path -import gym -import datetime -import argparse -from envs.register import register_env -from envs.wrappers import CliffWalkingWapper -from Sarsa.sarsa import Sarsa -from common.utils import all_seed,merge_class_attrs -from common.launcher import Launcher -from config.config import GeneralConfigSarsa,AlgoConfigSarsa - -class Main(Launcher): - def __init__(self) -> None: - super().__init__() - self.cfgs['general_cfg'] = merge_class_attrs(self.cfgs['general_cfg'],GeneralConfigSarsa()) - self.cfgs['algo_cfg'] = merge_class_attrs(self.cfgs['algo_cfg'],AlgoConfigSarsa()) - - def env_agent_config(self,cfg,logger): - register_env(cfg.env_name) - env = gym.make(cfg.env_name,new_step_api=False) # create env - if cfg.env_name == 'CliffWalking-v0': - env = CliffWalkingWapper(env) - if cfg.seed !=0: # set random seed - all_seed(env,seed=cfg.seed) - try: # state dimension - n_states = env.observation_space.n # print(hasattr(env.observation_space, 'n')) - except AttributeError: - n_states = env.observation_space.shape[0] # print(hasattr(env.observation_space, 'shape')) - n_actions = env.action_space.n # action dimension - logger.info(f"n_states: {n_states}, n_actions: {n_actions}") # print info - # update to cfg paramters - setattr(cfg, 'n_states', n_states) - setattr(cfg, 'n_actions', n_actions) - agent = Sarsa(cfg) - return env,agent - - def train(self,cfg,env,agent,logger): - logger.info("Start training!") - logger.info(f"Env: {cfg.env_name}, Algorithm: {cfg.algo_name}, Device: {cfg.device}") - rewards = [] # record rewards for all episodes - steps = [] # record steps for all episodes - for i_ep in range(cfg.train_eps): - ep_reward = 0 # reward per episode - ep_step = 0 # step per episode - state = env.reset() # reset and obtain initial state - action = agent.sample_action(state) - # while True: - for _ in range(cfg.max_steps): - next_state, reward, done, _ = env.step(action) # update env and return transitions - next_action = agent.sample_action(next_state) - agent.update(state, action, reward, next_state, next_action,done) # update agent - state = next_state # update state - action = next_action - ep_reward += reward - ep_step += 1 - if done: - break - rewards.append(ep_reward) - steps.append(ep_step) - logger.info(f'Episode: {i_ep+1}/{cfg.train_eps}, Reward: {ep_reward:.2f}, Steps:{ep_step:d}, Epislon: {agent.epsilon:.3f}') - logger.info("Finish training!") - return {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - - def test(self,cfg,env,agent,logger): - logger.info("Start testing!") - logger.info(f"Env: {cfg.env_name}, Algorithm: {cfg.algo_name}, Device: {cfg.device}") - rewards = [] # record rewards for all episodes - steps = [] # record steps for all episodes - for i_ep in range(cfg.test_eps): - ep_reward = 0 # reward per episode - ep_step = 0 - state = env.reset() # reset and obtain initial state - for _ in range(cfg.max_steps): - action = agent.predict_action(state) - next_state, reward, done, _ = env.step(action) - state = next_state - ep_reward+=reward - ep_step+=1 - if done: - break - rewards.append(ep_reward) - steps.append(ep_step) - logger.info(f"Episode: {i_ep+1}/{cfg.test_eps}, Reward: {ep_reward:.2f}, Steps:{ep_step:d}") - logger.info("Finish testing!") - return {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - -if __name__ == "__main__": - main = Main() - main.run() - - - diff --git a/projects/codes/SoftActorCritic/env_wrapper.py b/projects/codes/SoftActorCritic/env_wrapper.py deleted file mode 100644 index dfe1c4d..0000000 --- a/projects/codes/SoftActorCritic/env_wrapper.py +++ /dev/null @@ -1,30 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2021-04-29 12:52:11 -LastEditor: JiangJi -LastEditTime: 2021-12-22 15:36:36 -Discription: -Environment: -''' -import gym -import numpy as np - -class NormalizedActions(gym.ActionWrapper): - def action(self, action): - low = self.action_space.low - high = self.action_space.high - - action = low + (action + 1.0) * 0.5 * (high - low) - action = np.clip(action, low, high) - - return action - - def reverse_action(self, action): - low = self.action_space.low - high = self.action_space.high - action = 2 * (action - low) / (high - low) - 1 - action = np.clip(action, low, high) - return action \ No newline at end of file diff --git a/projects/codes/SoftActorCritic/model.py b/projects/codes/SoftActorCritic/model.py deleted file mode 100644 index ba04737..0000000 --- a/projects/codes/SoftActorCritic/model.py +++ /dev/null @@ -1,108 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2021-04-29 12:53:58 -LastEditor: JiangJi -LastEditTime: 2021-11-19 18:04:19 -Discription: -Environment: -''' -import torch -import torch.nn as nn -import torch.nn.functional as F -from torch.distributions import Normal - -device=torch.device("cuda" if torch.cuda.is_available() else "cpu") - -class ValueNet(nn.Module): - def __init__(self, n_states, hidden_dim, init_w=3e-3): - super(ValueNet, self).__init__() - - self.linear1 = nn.Linear(n_states, hidden_dim) - self.linear2 = nn.Linear(hidden_dim, hidden_dim) - self.linear3 = nn.Linear(hidden_dim, 1) - - self.linear3.weight.data.uniform_(-init_w, init_w) - self.linear3.bias.data.uniform_(-init_w, init_w) - - def forward(self, state): - x = F.relu(self.linear1(state)) - x = F.relu(self.linear2(x)) - x = self.linear3(x) - return x - - -class SoftQNet(nn.Module): - def __init__(self, n_states, n_actions, hidden_dim, init_w=3e-3): - super(SoftQNet, self).__init__() - - self.linear1 = nn.Linear(n_states + n_actions, hidden_dim) - self.linear2 = nn.Linear(hidden_dim, hidden_dim) - self.linear3 = nn.Linear(hidden_dim, 1) - - self.linear3.weight.data.uniform_(-init_w, init_w) - self.linear3.bias.data.uniform_(-init_w, init_w) - - def forward(self, state, action): - x = torch.cat([state, action], 1) - x = F.relu(self.linear1(x)) - x = F.relu(self.linear2(x)) - x = self.linear3(x) - return x - - -class PolicyNet(nn.Module): - def __init__(self, n_states, n_actions, hidden_dim, init_w=3e-3, log_std_min=-20, log_std_max=2): - super(PolicyNet, self).__init__() - - self.log_std_min = log_std_min - self.log_std_max = log_std_max - - self.linear1 = nn.Linear(n_states, hidden_dim) - self.linear2 = nn.Linear(hidden_dim, hidden_dim) - - self.mean_linear = nn.Linear(hidden_dim, n_actions) - self.mean_linear.weight.data.uniform_(-init_w, init_w) - self.mean_linear.bias.data.uniform_(-init_w, init_w) - - self.log_std_linear = nn.Linear(hidden_dim, n_actions) - self.log_std_linear.weight.data.uniform_(-init_w, init_w) - self.log_std_linear.bias.data.uniform_(-init_w, init_w) - - def forward(self, state): - x = F.relu(self.linear1(state)) - x = F.relu(self.linear2(x)) - - mean = self.mean_linear(x) - log_std = self.log_std_linear(x) - log_std = torch.clamp(log_std, self.log_std_min, self.log_std_max) - - return mean, log_std - - def evaluate(self, state, epsilon=1e-6): - mean, log_std = self.forward(state) - std = log_std.exp() - - normal = Normal(mean, std) - z = normal.sample() - action = torch.tanh(z) - - log_prob = normal.log_prob(z) - torch.log(1 - action.pow(2) + epsilon) - log_prob = log_prob.sum(-1, keepdim=True) - - return action, log_prob, z, mean, log_std - - - def get_action(self, state): - state = torch.FloatTensor(state).unsqueeze(0).to(device) - mean, log_std = self.forward(state) - std = log_std.exp() - - normal = Normal(mean, std) - z = normal.sample() - action = torch.tanh(z) - - action = action.detach().cpu().numpy() - return action[0] \ No newline at end of file diff --git a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_policy b/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_policy deleted file mode 100644 index 9ae4e7b..0000000 Binary files a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_policy and /dev/null differ diff --git a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_policy_optimizer b/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_policy_optimizer deleted file mode 100644 index 49c0d2a..0000000 Binary files a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_policy_optimizer and /dev/null differ diff --git a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_soft_q b/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_soft_q deleted file mode 100644 index 3ff692f..0000000 Binary files a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_soft_q and /dev/null differ diff --git a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_soft_q_optimizer b/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_soft_q_optimizer deleted file mode 100644 index 73be931..0000000 Binary files a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_soft_q_optimizer and /dev/null differ diff --git a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_value b/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_value deleted file mode 100644 index 853ac6f..0000000 Binary files a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_value and /dev/null differ diff --git a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_value_optimizer b/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_value_optimizer deleted file mode 100644 index 79410e4..0000000 Binary files a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/models/sac_value_optimizer and /dev/null differ diff --git a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/test_ma_rewards.npy b/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/test_ma_rewards.npy deleted file mode 100644 index eca3369..0000000 Binary files a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/test_ma_rewards.npy and /dev/null differ diff --git a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/test_rewards.npy b/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/test_rewards.npy deleted file mode 100644 index 09edb0e..0000000 Binary files a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/test_rewards.npy and /dev/null differ diff --git a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/test_rewards_curve.png b/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/test_rewards_curve.png deleted file mode 100644 index 5cc6e1d..0000000 Binary files a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/test_rewards_curve.png and /dev/null differ diff --git a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/train_ma_rewards.npy b/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/train_ma_rewards.npy deleted file mode 100644 index 3e1feac..0000000 Binary files a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/train_ma_rewards.npy and /dev/null differ diff --git a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/train_rewards.npy b/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/train_rewards.npy deleted file mode 100644 index 1c77a83..0000000 Binary files a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/train_rewards.npy and /dev/null differ diff --git a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/train_rewards_curve.png b/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/train_rewards_curve.png deleted file mode 100644 index 3e4c8aa..0000000 Binary files a/projects/codes/SoftActorCritic/outputs/Pendulum-v1/20211222-162722/results/train_rewards_curve.png and /dev/null differ diff --git a/projects/codes/SoftActorCritic/sac.py b/projects/codes/SoftActorCritic/sac.py deleted file mode 100644 index c67257f..0000000 --- a/projects/codes/SoftActorCritic/sac.py +++ /dev/null @@ -1,222 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2021-04-29 12:53:54 -LastEditor: JiangJi -LastEditTime: 2021-12-22 15:41:19 -Discription: -Environment: -''' -import copy -import torch -import torch.nn as nn -import torch.optim as optim -import torch.nn.functional as F -from torch.distributions import Normal -import numpy as np -import random -device=torch.device("cuda" if torch.cuda.is_available() else "cpu") -class ReplayBuffer: - def __init__(self, capacity): - self.capacity = capacity # 经验回放的容量 - self.buffer = [] # 缓冲区 - self.position = 0 - - def push(self, state, action, reward, next_state, done): - ''' 缓冲区是一个队列，容量超出时去掉开始存入的转移(transition) - ''' - if len(self.buffer) < self.capacity: - self.buffer.append(None) - self.buffer[self.position] = (state, action, reward, next_state, done) - self.position = (self.position + 1) % self.capacity - - def sample(self, batch_size): - batch = random.sample(self.buffer, batch_size) # 随机采出小批量转移 - state, action, reward, next_state, done = zip(*batch) # 解压成状态，动作等 - return state, action, reward, next_state, done - - def __len__(self): - ''' 返回当前存储的量 - ''' - return len(self.buffer) - -class ValueNet(nn.Module): - def __init__(self, n_states, hidden_dim, init_w=3e-3): - super(ValueNet, self).__init__() - - self.linear1 = nn.Linear(n_states, hidden_dim) - self.linear2 = nn.Linear(hidden_dim, hidden_dim) - self.linear3 = nn.Linear(hidden_dim, 1) - - self.linear3.weight.data.uniform_(-init_w, init_w) - self.linear3.bias.data.uniform_(-init_w, init_w) - - def forward(self, state): - x = F.relu(self.linear1(state)) - x = F.relu(self.linear2(x)) - x = self.linear3(x) - return x - - -class SoftQNet(nn.Module): - def __init__(self, n_states, n_actions, hidden_dim, init_w=3e-3): - super(SoftQNet, self).__init__() - - self.linear1 = nn.Linear(n_states + n_actions, hidden_dim) - self.linear2 = nn.Linear(hidden_dim, hidden_dim) - self.linear3 = nn.Linear(hidden_dim, 1) - - self.linear3.weight.data.uniform_(-init_w, init_w) - self.linear3.bias.data.uniform_(-init_w, init_w) - - def forward(self, state, action): - x = torch.cat([state, action], 1) - x = F.relu(self.linear1(x)) - x = F.relu(self.linear2(x)) - x = self.linear3(x) - return x - - -class PolicyNet(nn.Module): - def __init__(self, n_states, n_actions, hidden_dim, init_w=3e-3, log_std_min=-20, log_std_max=2): - super(PolicyNet, self).__init__() - - self.log_std_min = log_std_min - self.log_std_max = log_std_max - - self.linear1 = nn.Linear(n_states, hidden_dim) - self.linear2 = nn.Linear(hidden_dim, hidden_dim) - - self.mean_linear = nn.Linear(hidden_dim, n_actions) - self.mean_linear.weight.data.uniform_(-init_w, init_w) - self.mean_linear.bias.data.uniform_(-init_w, init_w) - - self.log_std_linear = nn.Linear(hidden_dim, n_actions) - self.log_std_linear.weight.data.uniform_(-init_w, init_w) - self.log_std_linear.bias.data.uniform_(-init_w, init_w) - - def forward(self, state): - x = F.relu(self.linear1(state)) - x = F.relu(self.linear2(x)) - - mean = self.mean_linear(x) - log_std = self.log_std_linear(x) - log_std = torch.clamp(log_std, self.log_std_min, self.log_std_max) - - return mean, log_std - - def evaluate(self, state, epsilon=1e-6): - mean, log_std = self.forward(state) - std = log_std.exp() - - normal = Normal(mean, std) - z = normal.sample() - action = torch.tanh(z) - - log_prob = normal.log_prob(z) - torch.log(1 - action.pow(2) + epsilon) - log_prob = log_prob.sum(-1, keepdim=True) - - return action, log_prob, z, mean, log_std - - - def get_action(self, state): - state = torch.FloatTensor(state).unsqueeze(0).to(device) - mean, log_std = self.forward(state) - std = log_std.exp() - - normal = Normal(mean, std) - z = normal.sample() - action = torch.tanh(z) - - action = action.detach().cpu().numpy() - return action[0] - -class SAC: - def __init__(self,n_states,n_actions,cfg) -> None: - self.batch_size = cfg.batch_size - self.memory = ReplayBuffer(cfg.capacity) - self.device = cfg.device - self.value_net = ValueNet(n_states, cfg.hidden_dim).to(self.device) - self.target_value_net = ValueNet(n_states, cfg.hidden_dim).to(self.device) - self.soft_q_net = SoftQNet(n_states, n_actions, cfg.hidden_dim).to(self.device) - self.policy_net = PolicyNet(n_states, n_actions, cfg.hidden_dim).to(self.device) - self.value_optimizer = optim.Adam(self.value_net.parameters(), lr=cfg.value_lr) - self.soft_q_optimizer = optim.Adam(self.soft_q_net.parameters(), lr=cfg.soft_q_lr) - self.policy_optimizer = optim.Adam(self.policy_net.parameters(), lr=cfg.policy_lr) - for target_param, param in zip(self.target_value_net.parameters(), self.value_net.parameters()): - target_param.data.copy_(param.data) - self.value_criterion = nn.MSELoss() - self.soft_q_criterion = nn.MSELoss() - def update(self, gamma=0.99,mean_lambda=1e-3, - std_lambda=1e-3, - z_lambda=0.0, - soft_tau=1e-2, - ): - if len(self.memory) < self.batch_size: - return - state, action, reward, next_state, done = self.memory.sample(self.batch_size) - state = torch.FloatTensor(state).to(self.device) - next_state = torch.FloatTensor(next_state).to(self.device) - action = torch.FloatTensor(action).to(self.device) - reward = torch.FloatTensor(reward).unsqueeze(1).to(self.device) - done = torch.FloatTensor(np.float32(done)).unsqueeze(1).to(self.device) - expected_q_value = self.soft_q_net(state, action) - expected_value = self.value_net(state) - new_action, log_prob, z, mean, log_std = self.policy_net.evaluate(state) - - - target_value = self.target_value_net(next_state) - next_q_value = reward + (1 - done) * gamma * target_value - q_value_loss = self.soft_q_criterion(expected_q_value, next_q_value.detach()) - - expected_new_q_value = self.soft_q_net(state, new_action) - next_value = expected_new_q_value - log_prob - value_loss = self.value_criterion(expected_value, next_value.detach()) - - log_prob_target = expected_new_q_value - expected_value - policy_loss = (log_prob * (log_prob - log_prob_target).detach()).mean() - - - mean_loss = mean_lambda * mean.pow(2).mean() - std_loss = std_lambda * log_std.pow(2).mean() - z_loss = z_lambda * z.pow(2).sum(1).mean() - - policy_loss += mean_loss + std_loss + z_loss - - self.soft_q_optimizer.zero_grad() - q_value_loss.backward() - self.soft_q_optimizer.step() - - self.value_optimizer.zero_grad() - value_loss.backward() - self.value_optimizer.step() - - self.policy_optimizer.zero_grad() - policy_loss.backward() - self.policy_optimizer.step() - - for target_param, param in zip(self.target_value_net.parameters(), self.value_net.parameters()): - target_param.data.copy_( - target_param.data * (1.0 - soft_tau) + param.data * soft_tau - ) - def save(self, path): - torch.save(self.value_net.state_dict(), path + "sac_value") - torch.save(self.value_optimizer.state_dict(), path + "sac_value_optimizer") - torch.save(self.soft_q_net.state_dict(), path + "sac_soft_q") - torch.save(self.soft_q_optimizer.state_dict(), path + "sac_soft_q_optimizer") - - torch.save(self.policy_net.state_dict(), path + "sac_policy") - torch.save(self.policy_optimizer.state_dict(), path + "sac_policy_optimizer") - - def load(self, path): - self.value_net.load_state_dict(torch.load(path + "sac_value")) - self.value_optimizer.load_state_dict(torch.load(path + "sac_value_optimizer")) - self.target_value_net = copy.deepcopy(self.value_net) - - self.soft_q_net.load_state_dict(torch.load(path + "sac_soft_q")) - self.soft_q_optimizer.load_state_dict(torch.load(path + "sac_soft_q_optimizer")) - - self.policy_net.load_state_dict(torch.load(path + "sac_policy")) - self.policy_optimizer.load_state_dict(torch.load(path + "sac_policy_optimizer")) \ No newline at end of file diff --git a/projects/codes/SoftActorCritic/task0.py b/projects/codes/SoftActorCritic/task0.py deleted file mode 100644 index 668d289..0000000 --- a/projects/codes/SoftActorCritic/task0.py +++ /dev/null @@ -1,142 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2021-04-29 12:59:22 -LastEditor: JiangJi -LastEditTime: 2021-12-22 16:27:13 -Discription: -Environment: -''' -import sys,os -curr_path = os.path.dirname(os.path.abspath(__file__)) # 当前文件所在绝对路径 -parent_path = os.path.dirname(curr_path) # 父路径 -sys.path.append(parent_path) # 添加路径到系统路径 - -import gym -import torch -import datetime - -from SoftActorCritic.env_wrapper import NormalizedActions -from SoftActorCritic.sac import SAC -from common.utils import save_results, make_dir -from common.utils import plot_rewards - -curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # 获取当前时间 -algo_name = 'SAC' # 算法名称 -env_name = 'Pendulum-v1' # 环境名称 -device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # 检测GPU - -class SACConfig: - def __init__(self) -> None: - self.algo_name = algo_name - self.env_name = env_name # 环境名称 - self.device= device - self.train_eps = 300 - self.test_eps = 20 - self.max_steps = 500 # 每回合的最大步数 - self.gamma = 0.99 - self.mean_lambda=1e-3 - self.std_lambda=1e-3 - self.z_lambda=0.0 - self.soft_tau=1e-2 - self.value_lr = 3e-4 - self.soft_q_lr = 3e-4 - self.policy_lr = 3e-4 - self.capacity = 1000000 - self.hidden_dim = 256 - self.batch_size = 128 - - -class PlotConfig: - def __init__(self) -> None: - self.algo_name = algo_name # 算法名称 - self.env_name = env_name # 环境名称 - self.device= device - self.result_path = curr_path + "/outputs/" + self.env_name + \ - '/' + curr_time + '/results/' # 保存结果的路径 - self.model_path = curr_path + "/outputs/" + self.env_name + \ - '/' + curr_time + '/models/' # 保存模型的路径 - self.save = True # 是否保存图片 - -def env_agent_config(cfg,seed=1): - env = NormalizedActions(gym.make(cfg.env_name)) - env.seed(seed) - n_actions = env.action_space.shape[0] - n_states = env.observation_space.shape[0] - agent = SAC(n_states,n_actions,cfg) - return env,agent - -def train(cfg,env,agent): - print('开始训练!') - print(f'环境：{cfg.env_name}, 算法：{cfg.algo_name}, 设备：{cfg.device}') - rewards = [] # 记录所有回合的奖励 - ma_rewards = [] # 记录所有回合的滑动平均奖励 - for i_ep in range(cfg.train_eps): - ep_reward = 0 # 记录一回合内的奖励 - state = env.reset() # 重置环境，返回初始状态 - for i_step in range(cfg.max_steps): - action = agent.policy_net.get_action(state) - next_state, reward, done, _ = env.step(action) - agent.memory.push(state, action, reward, next_state, done) - agent.update() - state = next_state - ep_reward += reward - if done: - break - rewards.append(ep_reward) - if ma_rewards: - ma_rewards.append(0.9*ma_rewards[-1]+0.1*ep_reward) - else: - ma_rewards.append(ep_reward) - if (i_ep+1)%10 == 0: - print(f'回合：{i_ep+1}/{cfg.train_eps}, 奖励：{ep_reward:.3f}') - print('完成训练！') - return rewards, ma_rewards - -def test(cfg,env,agent): - print('开始测试!') - print(f'环境：{cfg.env_name}, 算法：{cfg.algo_name}, 设备：{cfg.device}') - rewards = [] # 记录所有回合的奖励 - ma_rewards = [] # 记录所有回合的滑动平均奖励 - for i_ep in range(cfg.test_eps): - state = env.reset() - ep_reward = 0 - for i_step in range(cfg.max_steps): - action = agent.policy_net.get_action(state) - next_state, reward, done, _ = env.step(action) - state = next_state - ep_reward += reward - if done: - break - rewards.append(ep_reward) - if ma_rewards: - ma_rewards.append(0.9*ma_rewards[-1]+0.1*ep_reward) - else: - ma_rewards.append(ep_reward) - print(f"回合：{i_ep+1}/{cfg.test_eps}，奖励：{ep_reward:.1f}") - print('完成测试！') - return rewards, ma_rewards - -if __name__ == "__main__": - cfg=SACConfig() - plot_cfg = PlotConfig() - # 训练 - env, agent = env_agent_config(cfg, seed=1) - rewards, ma_rewards = train(cfg, env, agent) - make_dir(plot_cfg.result_path, plot_cfg.model_path) # 创建保存结果和模型路径的文件夹 - agent.save(path=plot_cfg.model_path) # 保存模型 - save_results(rewards, ma_rewards, tag='train', - path=plot_cfg.result_path) # 保存结果 - plot_rewards(rewards, ma_rewards, plot_cfg, tag="train") # 画出结果 - # 测试 - env, agent = env_agent_config(cfg, seed=10) - agent.load(path=plot_cfg.model_path) # 导入模型 - rewards, ma_rewards = test(cfg, env, agent) - save_results(rewards, ma_rewards, tag='test', path=plot_cfg.result_path) # 保存结果 - plot_rewards(rewards, ma_rewards, plot_cfg, tag="test") # 画出结果 - - - - diff --git a/projects/codes/SoftActorCritic/task0_train.ipynb b/projects/codes/SoftActorCritic/task0_train.ipynb deleted file mode 100644 index 3be10c6..0000000 --- a/projects/codes/SoftActorCritic/task0_train.ipynb +++ /dev/null @@ -1,221 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "import sys\n", - "from pathlib import Path\n", - "curr_path = str(Path().absolute())\n", - "parent_path = str(Path().absolute().parent)\n", - "sys.path.append(parent_path) # add current terminal path to sys.path" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [], - "source": [ - "import gym\n", - "import torch\n", - "import datetime\n", - "\n", - "from SAC.env import NormalizedActions\n", - "from SAC.agent import SAC\n", - "from common.utils import save_results, make_dir\n", - "from common.plot import plot_rewards\n", - "\n", - "curr_time = datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\") # obtain current time" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "class SACConfig:\n", - " def __init__(self) -> None:\n", - " self.algo = 'SAC'\n", - " self.env = 'Pendulum-v0'\n", - " self.result_path = curr_path+\"/outputs/\" +self.env+'/'+curr_time+'/results/' # path to save results\n", - " self.model_path = curr_path+\"/outputs/\" +self.env+'/'+curr_time+'/models/' # path to save models\n", - " self.train_eps = 300\n", - " self.train_steps = 500\n", - " self.test_eps = 50\n", - " self.eval_steps = 500\n", - " self.gamma = 0.99\n", - " self.mean_lambda=1e-3\n", - " self.std_lambda=1e-3\n", - " self.z_lambda=0.0\n", - " self.soft_tau=1e-2\n", - " self.value_lr = 3e-4\n", - " self.soft_q_lr = 3e-4\n", - " self.policy_lr = 3e-4\n", - " self.capacity = 1000000\n", - " self.hidden_dim = 256\n", - " self.batch_size = 128\n", - " self.device=torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "def env_agent_config(cfg,seed=1):\n", - " env = NormalizedActions(gym.make(\"Pendulum-v0\"))\n", - " env.seed(seed)\n", - " n_actions = env.action_space.shape[0]\n", - " n_states = env.observation_space.shape[0]\n", - " agent = SAC(n_states,n_actions,cfg)\n", - " return env,agent" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "def train(cfg,env,agent):\n", - " print('Start to train !')\n", - " print(f'Env: {cfg.env}, Algorithm: {cfg.algo}, Device: {cfg.device}')\n", - " rewards = []\n", - " ma_rewards = [] # moveing average reward\n", - " for i_ep in range(cfg.train_eps):\n", - " state = env.reset()\n", - " ep_reward = 0\n", - " for i_step in range(cfg.train_steps):\n", - " action = agent.policy_net.get_action(state)\n", - " next_state, reward, done, _ = env.step(action)\n", - " agent.memory.push(state, action, reward, next_state, done)\n", - " agent.update()\n", - " state = next_state\n", - " ep_reward += reward\n", - " if done:\n", - " break\n", - " if (i_ep+1)%10==0:\n", - " print(f\"Episode:{i_ep+1}/{cfg.train_eps}, Reward:{ep_reward:.3f}\")\n", - " rewards.append(ep_reward)\n", - " if ma_rewards:\n", - " ma_rewards.append(0.9*ma_rewards[-1]+0.1*ep_reward)\n", - " else:\n", - " ma_rewards.append(ep_reward) \n", - " print('Complete training！')\n", - " return rewards, ma_rewards" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [], - "source": [ - "def eval(cfg,env,agent):\n", - " print('Start to eval !')\n", - " print(f'Env: {cfg.env}, Algorithm: {cfg.algo}, Device: {cfg.device}')\n", - " rewards = []\n", - " ma_rewards = [] # moveing average reward\n", - " for i_ep in range(cfg.test_eps):\n", - " state = env.reset()\n", - " ep_reward = 0\n", - " for i_step in range(cfg.eval_steps):\n", - " action = agent.policy_net.get_action(state)\n", - " next_state, reward, done, _ = env.step(action)\n", - " state = next_state\n", - " ep_reward += reward\n", - " if done:\n", - " break\n", - " if (i_ep+1)%10==0:\n", - " print(f\"Episode:{i_ep+1}/{cfg.train_eps}, Reward:{ep_reward:.3f}\")\n", - " rewards.append(ep_reward)\n", - " if ma_rewards:\n", - " ma_rewards.append(0.9*ma_rewards[-1]+0.1*ep_reward)\n", - " else:\n", - " ma_rewards.append(ep_reward) \n", - " print('Complete evaling！')\n", - " return rewards, ma_rewards\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [ - { - "ename": "DeprecatedEnv", - "evalue": "Env Pendulum-v0 not found (valid versions include ['Pendulum-v1'])", - "output_type": "error", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)", - "\u001b[0;32m~/anaconda3/envs/py37/lib/python3.7/site-packages/gym/envs/registration.py\u001b[0m in \u001b[0;36mspec\u001b[0;34m(self, path)\u001b[0m\n\u001b[1;32m 157\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 158\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv_specs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mid\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 159\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;31mKeyError\u001b[0m: 'Pendulum-v0'", - "\nDuring handling of the above exception, another exception occurred:\n", - "\u001b[0;31mDeprecatedEnv\u001b[0m Traceback (most recent call last)", - "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;31m# train\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0menv\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0magent\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menv_agent_config\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcfg\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mseed\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 6\u001b[0m \u001b[0mrewards\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mma_rewards\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcfg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0magent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 7\u001b[0m \u001b[0mmake_dir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcfg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresult_path\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcfg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel_path\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m\u001b[0m in \u001b[0;36menv_agent_config\u001b[0;34m(cfg, seed)\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0menv_agent_config\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcfg\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mseed\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0menv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mNormalizedActions\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgym\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmake\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Pendulum-v0\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mseed\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mseed\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mn_actions\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maction_space\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mn_states\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mobservation_space\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m~/anaconda3/envs/py37/lib/python3.7/site-packages/gym/envs/registration.py\u001b[0m in \u001b[0;36mmake\u001b[0;34m(id, **kwargs)\u001b[0m\n\u001b[1;32m 233\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 234\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mmake\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mid\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 235\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mregistry\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmake\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mid\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 236\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 237\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m~/anaconda3/envs/py37/lib/python3.7/site-packages/gym/envs/registration.py\u001b[0m in \u001b[0;36mmake\u001b[0;34m(self, path, **kwargs)\u001b[0m\n\u001b[1;32m 126\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 127\u001b[0m \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Making new env: %s\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpath\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 128\u001b[0;31m \u001b[0mspec\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mspec\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 129\u001b[0m \u001b[0menv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mspec\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmake\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 130\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m~/anaconda3/envs/py37/lib/python3.7/site-packages/gym/envs/registration.py\u001b[0m in \u001b[0;36mspec\u001b[0;34m(self, path)\u001b[0m\n\u001b[1;32m 185\u001b[0m raise error.DeprecatedEnv(\n\u001b[1;32m 186\u001b[0m \"Env {} not found (valid versions include {})\".format(\n\u001b[0;32m--> 187\u001b[0;31m \u001b[0mid\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmatching_envs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 188\u001b[0m )\n\u001b[1;32m 189\u001b[0m )\n", - "\u001b[0;31mDeprecatedEnv\u001b[0m: Env Pendulum-v0 not found (valid versions include ['Pendulum-v1'])" - ] - } - ], - "source": [ - "if __name__ == \"__main__\":\n", - " cfg=SACConfig()\n", - " \n", - " # train\n", - " env,agent = env_agent_config(cfg,seed=1)\n", - " rewards, ma_rewards = train(cfg, env, agent)\n", - " make_dir(cfg.result_path, cfg.model_path)\n", - " agent.save(path=cfg.model_path)\n", - " save_results(rewards, ma_rewards, tag='train', path=cfg.result_path)\n", - " plot_rewards(rewards, ma_rewards, tag=\"train\",\n", - " algo=cfg.algo, path=cfg.result_path)\n", - " # eval\n", - " env,agent = env_agent_config(cfg,seed=10)\n", - " agent.load(path=cfg.model_path)\n", - " rewards,ma_rewards = eval(cfg,env,agent)\n", - " save_results(rewards,ma_rewards,tag='eval',path=cfg.result_path)\n", - " plot_rewards(rewards,ma_rewards,tag=\"eval\",env=cfg.env,algo = cfg.algo,path=cfg.result_path)\n" - ] - } - ], - "metadata": { - "interpreter": { - "hash": "fe38df673a99c62a9fea33a7aceda74c9b65b12ee9d076c5851d98b692a4989a" - }, - "kernelspec": { - "display_name": "Python 3.7.10 64-bit ('mujoco': conda)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.10" - }, - "metadata": { - "interpreter": { - "hash": "fd81e6a9e450d5c245c1a0b5da0b03c89c450f614a13afa2acb1654375922756" - } - }, - "orig_nbformat": 2 - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/models/checkpoint.pth b/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/models/checkpoint.pth deleted file mode 100644 index fc80e6f..0000000 Binary files a/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/models/checkpoint.pth and /dev/null differ diff --git a/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/results/params.json b/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/results/params.json deleted file mode 100644 index 988c303..0000000 --- a/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/results/params.json +++ /dev/null @@ -1 +0,0 @@ -{"algo_name": "SoftQ", "env_name": "CartPole-v0", "train_eps": 200, "test_eps": 20, "max_steps": 200, "gamma": 0.99, "alpha": 4, "lr": 0.0001, "memory_capacity": 50000, "batch_size": 128, "target_update": 2, "device": "cpu", "seed": 10, "result_path": "/Users/jj/Desktop/rl-tutorials/codes/SoftQ/outputs/CartPole-v0/20220818-154333/results/", "model_path": "/Users/jj/Desktop/rl-tutorials/codes/SoftQ/outputs/CartPole-v0/20220818-154333/models/", "show_fig": false, "save_fig": true} \ No newline at end of file diff --git a/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/results/testing_curve.png b/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/results/testing_curve.png deleted file mode 100644 index 83750e7..0000000 Binary files a/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/results/testing_curve.png and /dev/null differ diff --git a/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/results/testing_results.csv b/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/results/testing_results.csv deleted file mode 100644 index b74878b..0000000 --- a/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/results/testing_results.csv +++ /dev/null @@ -1,21 +0,0 @@ -episodes,rewards -0,200.0 -1,200.0 -2,200.0 -3,200.0 -4,200.0 -5,200.0 -6,200.0 -7,200.0 -8,199.0 -9,200.0 -10,200.0 -11,200.0 -12,200.0 -13,200.0 -14,200.0 -15,200.0 -16,200.0 -17,200.0 -18,200.0 -19,200.0 diff --git a/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/results/training_curve.png b/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/results/training_curve.png deleted file mode 100644 index 9f3164b..0000000 Binary files a/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/results/training_curve.png and /dev/null differ diff --git a/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/results/training_results.csv b/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/results/training_results.csv deleted file mode 100644 index 0f52c1c..0000000 --- a/projects/codes/SoftQ/outputs/CartPole-v0/20220818-154333/results/training_results.csv +++ /dev/null @@ -1,201 +0,0 @@ -episodes,rewards -0,21.0 -1,23.0 -2,24.0 -3,27.0 -4,33.0 -5,18.0 -6,47.0 -7,18.0 -8,18.0 -9,21.0 -10,26.0 -11,31.0 -12,11.0 -13,17.0 -14,22.0 -15,16.0 -16,17.0 -17,34.0 -18,20.0 -19,11.0 -20,50.0 -21,15.0 -22,11.0 -23,39.0 -24,11.0 -25,28.0 -26,37.0 -27,26.0 -28,63.0 -29,18.0 -30,17.0 -31,13.0 -32,9.0 -33,15.0 -34,13.0 -35,21.0 -36,17.0 -37,22.0 -38,20.0 -39,31.0 -40,9.0 -41,10.0 -42,11.0 -43,15.0 -44,18.0 -45,10.0 -46,30.0 -47,14.0 -48,36.0 -49,26.0 -50,21.0 -51,15.0 -52,9.0 -53,14.0 -54,10.0 -55,27.0 -56,14.0 -57,15.0 -58,22.0 -59,12.0 -60,20.0 -61,10.0 -62,12.0 -63,29.0 -64,11.0 -65,13.0 -66,27.0 -67,50.0 -68,29.0 -69,40.0 -70,29.0 -71,18.0 -72,27.0 -73,11.0 -74,15.0 -75,10.0 -76,13.0 -77,11.0 -78,17.0 -79,13.0 -80,18.0 -81,24.0 -82,15.0 -83,34.0 -84,11.0 -85,35.0 -86,26.0 -87,9.0 -88,19.0 -89,19.0 -90,16.0 -91,25.0 -92,18.0 -93,37.0 -94,46.0 -95,88.0 -96,26.0 -97,55.0 -98,43.0 -99,141.0 -100,89.0 -101,151.0 -102,47.0 -103,56.0 -104,64.0 -105,56.0 -106,49.0 -107,87.0 -108,58.0 -109,55.0 -110,57.0 -111,165.0 -112,31.0 -113,200.0 -114,57.0 -115,107.0 -116,46.0 -117,45.0 -118,64.0 -119,69.0 -120,67.0 -121,65.0 -122,47.0 -123,63.0 -124,134.0 -125,60.0 -126,89.0 -127,99.0 -128,51.0 -129,109.0 -130,131.0 -131,156.0 -132,118.0 -133,185.0 -134,86.0 -135,149.0 -136,138.0 -137,143.0 -138,114.0 -139,130.0 -140,139.0 -141,106.0 -142,135.0 -143,164.0 -144,156.0 -145,155.0 -146,200.0 -147,186.0 -148,64.0 -149,200.0 -150,135.0 -151,135.0 -152,168.0 -153,200.0 -154,200.0 -155,200.0 -156,167.0 -157,198.0 -158,188.0 -159,200.0 -160,200.0 -161,200.0 -162,200.0 -163,200.0 -164,200.0 -165,200.0 -166,200.0 -167,200.0 -168,189.0 -169,200.0 -170,146.0 -171,200.0 -172,200.0 -173,200.0 -174,115.0 -175,170.0 -176,200.0 -177,200.0 -178,178.0 -179,200.0 -180,200.0 -181,200.0 -182,200.0 -183,200.0 -184,200.0 -185,200.0 -186,120.0 -187,200.0 -188,200.0 -189,200.0 -190,200.0 -191,200.0 -192,200.0 -193,200.0 -194,200.0 -195,200.0 -196,200.0 -197,200.0 -198,200.0 -199,200.0 diff --git a/projects/codes/SoftQ/softq.py b/projects/codes/SoftQ/softq.py deleted file mode 100644 index a9a38e1..0000000 --- a/projects/codes/SoftQ/softq.py +++ /dev/null @@ -1,71 +0,0 @@ -import torch -import torch.nn as nn -import torch.nn.functional as F -from collections import deque -import random -from torch.distributions import Categorical -import gym -import numpy as np - -class SoftQ: - def __init__(self,n_actions,model,memory,cfg): - self.memory = memory - self.alpha = cfg.alpha - self.gamma = cfg.gamma # discount factor - self.batch_size = cfg.batch_size - self.device = torch.device(cfg.device) - self.policy_net = model.to(self.device) - self.target_net = model.to(self.device) - self.target_net.load_state_dict(self.policy_net.state_dict()) # copy parameters - self.optimizer = torch.optim.Adam(self.policy_net.parameters(), lr=cfg.lr) - self.losses = [] # save losses - - def sample_action(self,state): - state = torch.FloatTensor(state).unsqueeze(0).to(self.device) - with torch.no_grad(): - q = self.policy_net(state) - v = self.alpha * torch.log(torch.sum(torch.exp(q/self.alpha), dim=1, keepdim=True)).squeeze() - dist = torch.exp((q-v)/self.alpha) - dist = dist / torch.sum(dist) - c = Categorical(dist) - a = c.sample() - return a.item() - def predict_action(self,state): - state = torch.tensor(np.array(state), device=self.device, dtype=torch.float).unsqueeze(0) - with torch.no_grad(): - q = self.policy_net(state) - v = self.alpha * torch.log(torch.sum(torch.exp(q/self.alpha), dim=1, keepdim=True)).squeeze() - dist = torch.exp((q-v)/self.alpha) - dist = dist / torch.sum(dist) - c = Categorical(dist) - a = c.sample() - return a.item() - def update(self): - if len(self.memory) < self.batch_size: # when the memory capacity does not meet a batch, the network will not update - return - state_batch, action_batch, reward_batch, next_state_batch, done_batch = self.memory.sample(self.batch_size) - state_batch = torch.tensor(np.array(state_batch), device=self.device, dtype=torch.float) # shape(batchsize,n_states) - action_batch = torch.tensor(np.array(action_batch), device=self.device, dtype=torch.float).unsqueeze(1) # shape(batchsize,1) - reward_batch = torch.tensor(np.array(reward_batch), device=self.device, dtype=torch.float).unsqueeze(1) # shape(batchsize,1) - next_state_batch = torch.tensor(np.array(next_state_batch), device=self.device, dtype=torch.float) # shape(batchsize,n_states) - done_batch = torch.tensor(np.array(done_batch), device=self.device, dtype=torch.float).unsqueeze(1) # shape(batchsize,1) - # print(state_batch.shape,action_batch.shape,reward_batch.shape,next_state_batch.shape,done_batch.shape) - with torch.no_grad(): - next_q = self.target_net(next_state_batch) - next_v = self.alpha * torch.log(torch.sum(torch.exp(next_q/self.alpha), dim=1, keepdim=True)) - y = reward_batch + (1 - done_batch ) * self.gamma * next_v - loss = F.mse_loss(self.policy_net(state_batch).gather(1, action_batch.long()), y) - self.losses.append(loss) - self.optimizer.zero_grad() - loss.backward() - self.optimizer.step() - def save_model(self, path): - from pathlib import Path - # create path - Path(path).mkdir(parents=True, exist_ok=True) - torch.save(self.target_net.state_dict(), path+'checkpoint.pth') - - def load_model(self, path): - self.target_net.load_state_dict(torch.load(path+'checkpoint.pth')) - for target_param, param in zip(self.target_net.parameters(), self.policy_net.parameters()): - param.data.copy_(target_param.data) \ No newline at end of file diff --git a/projects/codes/SoftQ/task0.py b/projects/codes/SoftQ/task0.py deleted file mode 100644 index fd67aa4..0000000 --- a/projects/codes/SoftQ/task0.py +++ /dev/null @@ -1,142 +0,0 @@ -import sys,os -curr_path = os.path.dirname(os.path.abspath(__file__)) # current path -parent_path = os.path.dirname(curr_path) # parent path -sys.path.append(parent_path) # add path to system path - -import argparse -import datetime -import gym -import torch -import random -import numpy as np -import torch.nn as nn -from common.memories import ReplayBufferQue -from common.models import MLP -from common.utils import save_results,all_seed,plot_rewards,save_args -from softq import SoftQ - -def get_args(): - """ hyperparameters - """ - curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # obtain current time - parser = argparse.ArgumentParser(description="hyperparameters") - parser.add_argument('--algo_name',default='SoftQ',type=str,help="name of algorithm") - parser.add_argument('--env_name',default='CartPole-v0',type=str,help="name of environment") - parser.add_argument('--train_eps',default=200,type=int,help="episodes of training") - parser.add_argument('--test_eps',default=20,type=int,help="episodes of testing") - parser.add_argument('--max_steps',default=200,type=int,help="maximum steps per episode") - parser.add_argument('--gamma',default=0.99,type=float,help="discounted factor") - parser.add_argument('--alpha',default=4,type=float,help="alpha") - parser.add_argument('--lr',default=0.0001,type=float,help="learning rate") - parser.add_argument('--memory_capacity',default=50000,type=int,help="memory capacity") - parser.add_argument('--batch_size',default=128,type=int) - parser.add_argument('--target_update',default=2,type=int) - parser.add_argument('--device',default='cpu',type=str,help="cpu or cuda") - parser.add_argument('--seed',default=10,type=int,help="seed") - parser.add_argument('--result_path',default=curr_path + "/outputs/" + parser.parse_args().env_name + \ - '/' + curr_time + '/results/' ) - parser.add_argument('--model_path',default=curr_path + "/outputs/" + parser.parse_args().env_name + \ - '/' + curr_time + '/models/' ) - parser.add_argument('--show_fig',default=False,type=bool,help="if show figure or not") - parser.add_argument('--save_fig',default=True,type=bool,help="if save figure or not") - args = parser.parse_args() - return args - -class SoftQNetwork(nn.Module): - '''Actually almost same to common.models.MLP - ''' - def __init__(self,input_dim,output_dim): - super(SoftQNetwork,self).__init__() - self.fc1 = nn.Linear(input_dim, 64) - self.relu = nn.ReLU() - self.fc2 = nn.Linear(64, 256) - self.fc3 = nn.Linear(256, output_dim) - - def forward(self, x): - x = self.relu(self.fc1(x)) - x = self.relu(self.fc2(x)) - x = self.fc3(x) - return x - -def env_agent_config(cfg): - ''' create env and agent - ''' - env = gym.make(cfg.env_name) # create env - if cfg.seed !=0: # set random seed - all_seed(env,seed=cfg.seed) - n_states = env.observation_space.shape[0] # state dimension - n_actions = env.action_space.n # action dimension - print(f"state dim: {n_states}, action dim: {n_actions}") - # model = MLP(n_states,n_actions) - model = SoftQNetwork(n_states,n_actions) - memory = ReplayBufferQue(cfg.memory_capacity) # replay buffer - agent = SoftQ(n_actions,model,memory,cfg) # create agent - return env, agent - -def train(cfg, env, agent): - ''' training - ''' - print("start training!") - print(f"Env: {cfg.env_name}, Algo: {cfg.algo_name}, Device: {cfg.device}") - rewards = [] # record rewards for all episodes - steps = [] # record steps for all episodes, sometimes need - for i_ep in range(cfg.train_eps): - ep_reward = 0 # reward per episode - ep_step = 0 - state = env.reset() # reset and obtain initial state - while True: - # for _ in range(cfg.max_steps): - ep_step += 1 - action = agent.sample_action(state) # sample action - next_state, reward, done, _ = env.step(action) # update env and return transitions - agent.memory.push((state, action, reward, next_state, done)) # save transitions - state = next_state # update next state for env - agent.update() # update agent - ep_reward += reward - if done: - break - if (i_ep + 1) % cfg.target_update == 0: # target net update, target_update means "C" in pseucodes - agent.target_net.load_state_dict(agent.policy_net.state_dict()) - steps.append(ep_step) - rewards.append(ep_reward) - if (i_ep + 1) % 10 == 0: - print(f'Episode: {i_ep+1}/{cfg.train_eps}, Reward: {ep_reward:.2f}') - print("finish training!") - res_dic = {'episodes':range(len(rewards)),'rewards':rewards} - return res_dic -def test(cfg, env, agent): - print("start testing!") - print(f"Env: {cfg.env_name}, Algo: {cfg.algo_name}, Device: {cfg.device}") - rewards = [] # record rewards for all episodes - for i_ep in range(cfg.test_eps): - ep_reward = 0 # reward per episode - state = env.reset() # reset and obtain initial state - while True: - action = agent.predict_action(state) # predict action - next_state, reward, done, _ = env.step(action) - state = next_state - ep_reward += reward - if done: - break - rewards.append(ep_reward) - print(f'Episode: {i_ep+1}/{cfg.test_eps}，Reward: {ep_reward:.2f}') - print("finish testing!") - env.close() - return {'episodes':range(len(rewards)),'rewards':rewards} - -if __name__ == "__main__": - cfg = get_args() - # 训练 - env, agent = env_agent_config(cfg) - res_dic = train(cfg, env, agent) - save_args(cfg,path = cfg.result_path) # 保存参数到模型路径上 - agent.save_model(path = cfg.model_path) # 保存模型 - save_results(res_dic, tag = 'train', path = cfg.result_path) - plot_rewards(res_dic['rewards'], cfg, path = cfg.result_path,tag = "train") - # 测试 - env, agent = env_agent_config(cfg) # 也可以不加，加这一行的是为了避免训练之后环境可能会出现问题，因此新建一个环境用于测试 - agent.load_model(path = cfg.model_path) # 导入模型 - res_dic = test(cfg, env, agent) - save_results(res_dic, tag='test', - path = cfg.result_path) # 保存结果 - plot_rewards(res_dic['rewards'], cfg, path = cfg.result_path,tag = "test") # 画出结果 \ No newline at end of file diff --git a/projects/codes/TD3/README.md b/projects/codes/TD3/README.md deleted file mode 100644 index 8001e9c..0000000 --- a/projects/codes/TD3/README.md +++ /dev/null @@ -1 +0,0 @@ -这是对[Implementation of Twin Delayed Deep Deterministic Policy Gradients (TD3)](https://arxiv.org/abs/1802.09477)的复现 \ No newline at end of file diff --git a/projects/codes/TD3/agent.py b/projects/codes/TD3/agent.py deleted file mode 100644 index f77a912..0000000 --- a/projects/codes/TD3/agent.py +++ /dev/null @@ -1,177 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2021-12-22 10:40:05 -LastEditor: JiangJi -LastEditTime: 2021-12-22 10:43:55 -Discription: -''' -import copy -import numpy as np -import torch -import torch.nn as nn -import torch.nn.functional as F -from TD3.memory import ReplayBuffer - -class Actor(nn.Module): - - def __init__(self, input_dim, output_dim, max_action): - '''[summary] - - Args: - input_dim (int): 输入维度，这里等于n_states - output_dim (int): 输出维度，这里等于n_actions - max_action (int): action的最大值 - ''' - super(Actor, self).__init__() - - self.l1 = nn.Linear(input_dim, 256) - self.l2 = nn.Linear(256, 256) - self.l3 = nn.Linear(256, output_dim) - self.max_action = max_action - - def forward(self, state): - - a = F.relu(self.l1(state)) - a = F.relu(self.l2(a)) - return self.max_action * torch.tanh(self.l3(a)) - - -class Critic(nn.Module): - def __init__(self, input_dim, output_dim): - super(Critic, self).__init__() - - # Q1 architecture - self.l1 = nn.Linear(input_dim + output_dim, 256) - self.l2 = nn.Linear(256, 256) - self.l3 = nn.Linear(256, 1) - - # Q2 architecture - self.l4 = nn.Linear(input_dim + output_dim, 256) - self.l5 = nn.Linear(256, 256) - self.l6 = nn.Linear(256, 1) - - - def forward(self, state, action): - sa = torch.cat([state, action], 1) - - q1 = F.relu(self.l1(sa)) - q1 = F.relu(self.l2(q1)) - q1 = self.l3(q1) - - q2 = F.relu(self.l4(sa)) - q2 = F.relu(self.l5(q2)) - q2 = self.l6(q2) - return q1, q2 - - - def Q1(self, state, action): - sa = torch.cat([state, action], 1) - - q1 = F.relu(self.l1(sa)) - q1 = F.relu(self.l2(q1)) - q1 = self.l3(q1) - return q1 - - -class TD3(object): - def __init__( - self, - input_dim, - output_dim, - max_action, - cfg, - ): - self.max_action = max_action - self.gamma = cfg.gamma - self.lr = cfg.lr - self.policy_noise = cfg.policy_noise - self.noise_clip = cfg.noise_clip - self.policy_freq = cfg.policy_freq - self.batch_size = cfg.batch_size - self.device = cfg.device - self.total_it = 0 - - self.actor = Actor(input_dim, output_dim, max_action).to(self.device) - self.actor_target = copy.deepcopy(self.actor) - self.actor_optimizer = torch.optim.Adam(self.actor.parameters(), lr=3e-4) - - self.critic = Critic(input_dim, output_dim).to(self.device) - self.critic_target = copy.deepcopy(self.critic) - self.critic_optimizer = torch.optim.Adam(self.critic.parameters(), lr=3e-4) - self.memory = ReplayBuffer(input_dim, output_dim) - - def choose_action(self, state): - state = torch.FloatTensor(state.reshape(1, -1)).to(self.device) - return self.actor(state).cpu().data.numpy().flatten() - - def update(self): - self.total_it += 1 - - # Sample replay buffer - state, action, next_state, reward, not_done = self.memory.sample(self.batch_size) - - with torch.no_grad(): - # Select action according to policy and add clipped noise - noise = ( - torch.randn_like(action) * self.policy_noise - ).clamp(-self.noise_clip, self.noise_clip) - - next_action = ( - self.actor_target(next_state) + noise - ).clamp(-self.max_action, self.max_action) - - # Compute the target Q value - target_Q1, target_Q2 = self.critic_target(next_state, next_action) - target_Q = torch.min(target_Q1, target_Q2) - target_Q = reward + not_done * self.gamma * target_Q - - # Get current Q estimates - current_Q1, current_Q2 = self.critic(state, action) - - # Compute critic loss - critic_loss = F.mse_loss(current_Q1, target_Q) + F.mse_loss(current_Q2, target_Q) - - # Optimize the critic - self.critic_optimizer.zero_grad() - critic_loss.backward() - self.critic_optimizer.step() - - # Delayed policy updates - if self.total_it % self.policy_freq == 0: - - # Compute actor losse - actor_loss = -self.critic.Q1(state, self.actor(state)).mean() - - # Optimize the actor - self.actor_optimizer.zero_grad() - actor_loss.backward() - self.actor_optimizer.step() - - # Update the frozen target models - for param, target_param in zip(self.critic.parameters(), self.critic_target.parameters()): - target_param.data.copy_(self.lr * param.data + (1 - self.lr) * target_param.data) - - for param, target_param in zip(self.actor.parameters(), self.actor_target.parameters()): - target_param.data.copy_(self.lr * param.data + (1 - self.lr) * target_param.data) - - - def save(self, path): - torch.save(self.critic.state_dict(), path + "td3_critic") - torch.save(self.critic_optimizer.state_dict(), path + "td3_critic_optimizer") - - torch.save(self.actor.state_dict(), path + "td3_actor") - torch.save(self.actor_optimizer.state_dict(), path + "td3_actor_optimizer") - - - def load(self, path): - self.critic.load_state_dict(torch.load(path + "td3_critic")) - self.critic_optimizer.load_state_dict(torch.load(path + "td3_critic_optimizer")) - self.critic_target = copy.deepcopy(self.critic) - - self.actor.load_state_dict(torch.load(path + "td3_actor")) - self.actor_optimizer.load_state_dict(torch.load(path + "td3_actor_optimizer")) - self.actor_target = copy.deepcopy(self.actor) - diff --git a/projects/codes/TD3/memory.py b/projects/codes/TD3/memory.py deleted file mode 100644 index bcf38bb..0000000 --- a/projects/codes/TD3/memory.py +++ /dev/null @@ -1,44 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: John -Email: johnjim0816@gmail.com -Date: 2021-04-13 11:00:13 -LastEditor: John -LastEditTime: 2021-04-15 01:25:14 -Discription: -Environment: -''' -import numpy as np -import torch - - -class ReplayBuffer(object): - def __init__(self, n_states, n_actions, max_size=int(1e6)): - self.max_size = max_size - self.ptr = 0 - self.size = 0 - self.state = np.zeros((max_size, n_states)) - self.action = np.zeros((max_size, n_actions)) - self.next_state = np.zeros((max_size, n_states)) - self.reward = np.zeros((max_size, 1)) - self.not_done = np.zeros((max_size, 1)) - self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") - def push(self, state, action, next_state, reward, done): - self.state[self.ptr] = state - self.action[self.ptr] = action - self.next_state[self.ptr] = next_state - self.reward[self.ptr] = reward - self.not_done[self.ptr] = 1. - done - self.ptr = (self.ptr + 1) % self.max_size - self.size = min(self.size + 1, self.max_size) - - def sample(self, batch_size): - ind = np.random.randint(0, self.size, size=batch_size) - return ( - torch.FloatTensor(self.state[ind]).to(self.device), - torch.FloatTensor(self.action[ind]).to(self.device), - torch.FloatTensor(self.next_state[ind]).to(self.device), - torch.FloatTensor(self.reward[ind]).to(self.device), - torch.FloatTensor(self.not_done[ind]).to(self.device) - ) \ No newline at end of file diff --git a/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/models/td3_actor b/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/models/td3_actor deleted file mode 100644 index 2b3b481..0000000 Binary files a/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/models/td3_actor and /dev/null differ diff --git a/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/models/td3_actor_optimizer b/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/models/td3_actor_optimizer deleted file mode 100644 index 9bb6195..0000000 Binary files a/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/models/td3_actor_optimizer and /dev/null differ diff --git a/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/models/td3_critic b/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/models/td3_critic deleted file mode 100644 index cccfb71..0000000 Binary files a/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/models/td3_critic and /dev/null differ diff --git a/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/models/td3_critic_optimizer b/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/models/td3_critic_optimizer deleted file mode 100644 index 1446c66..0000000 Binary files a/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/models/td3_critic_optimizer and /dev/null differ diff --git a/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/results/ma_rewards_train.npy b/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/results/ma_rewards_train.npy deleted file mode 100644 index 96d40db..0000000 Binary files a/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/results/ma_rewards_train.npy and /dev/null differ diff --git a/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/results/rewards_curve_train.png b/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/results/rewards_curve_train.png deleted file mode 100644 index e310371..0000000 Binary files a/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/results/rewards_curve_train.png and /dev/null differ diff --git a/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/results/rewards_train.npy b/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/results/rewards_train.npy deleted file mode 100644 index 718e407..0000000 Binary files a/projects/codes/TD3/outputs/HalfCheetah-v2/20210416-130341/results/rewards_train.npy and /dev/null differ diff --git a/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/models/td3_actor b/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/models/td3_actor deleted file mode 100644 index 40533d9..0000000 Binary files a/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/models/td3_actor and /dev/null differ diff --git a/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/models/td3_actor_optimizer b/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/models/td3_actor_optimizer deleted file mode 100644 index e91a68f..0000000 Binary files a/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/models/td3_actor_optimizer and /dev/null differ diff --git a/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/models/td3_critic b/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/models/td3_critic deleted file mode 100644 index ef6b3e5..0000000 Binary files a/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/models/td3_critic and /dev/null differ diff --git a/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/models/td3_critic_optimizer b/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/models/td3_critic_optimizer deleted file mode 100644 index 8094beb..0000000 Binary files a/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/models/td3_critic_optimizer and /dev/null differ diff --git a/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/results/train_ma_rewards.npy b/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/results/train_ma_rewards.npy deleted file mode 100644 index 288eb69..0000000 Binary files a/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/results/train_ma_rewards.npy and /dev/null differ diff --git a/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/results/train_rewards.npy b/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/results/train_rewards.npy deleted file mode 100644 index 5bdee4a..0000000 Binary files a/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/results/train_rewards.npy and /dev/null differ diff --git a/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/results/train_rewards_curve.png b/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/results/train_rewards_curve.png deleted file mode 100644 index 31e873c..0000000 Binary files a/projects/codes/TD3/outputs/Pendulum-v1/20211119-123814/results/train_rewards_curve.png and /dev/null differ diff --git a/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/ma_rewards_train.npy b/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/ma_rewards_train.npy deleted file mode 100644 index 017dbba..0000000 Binary files a/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/ma_rewards_train.npy and /dev/null differ diff --git a/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/rewards_curve_train.png b/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/rewards_curve_train.png deleted file mode 100644 index 098872d..0000000 Binary files a/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/rewards_curve_train.png and /dev/null differ diff --git a/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/rewards_train.npy b/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/rewards_train.npy deleted file mode 100644 index 3ef20c3..0000000 Binary files a/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/rewards_train.npy and /dev/null differ diff --git a/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/td3_actor b/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/td3_actor deleted file mode 100644 index 10e7154..0000000 Binary files a/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/td3_actor and /dev/null differ diff --git a/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/td3_actor_optimizer b/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/td3_actor_optimizer deleted file mode 100644 index ac8989e..0000000 Binary files a/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/td3_actor_optimizer and /dev/null differ diff --git a/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/td3_critic b/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/td3_critic deleted file mode 100644 index 5e16302..0000000 Binary files a/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/td3_critic and /dev/null differ diff --git a/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/td3_critic_optimizer b/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/td3_critic_optimizer deleted file mode 100644 index 3b7d759..0000000 Binary files a/projects/codes/TD3/outputs/Reacher-v2/20210415-021952/td3_critic_optimizer and /dev/null differ diff --git a/projects/codes/TD3/task0_eval.py b/projects/codes/TD3/task0_eval.py deleted file mode 100644 index cb977b4..0000000 --- a/projects/codes/TD3/task0_eval.py +++ /dev/null @@ -1,89 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2021-04-23 20:36:23 -LastEditor: JiangJi -LastEditTime: 2021-04-23 20:37:22 -Discription: -Environment: -''' -import sys,os -curr_path = os.path.dirname(__file__) -parent_path=os.path.dirname(curr_path) -sys.path.append(parent_path) # add current terminal path to sys.path - -import torch -import gym -import numpy as np -import datetime - - -from TD3.agent import TD3 -from common.plot import plot_rewards -from common.utils import save_results,make_dir - -curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # obtain current time - -class TD3Config: - def __init__(self) -> None: - self.algo = 'TD3 and Random' - self.env = 'HalfCheetah-v2' - self.seed = 0 - self.result_path = curr_path+"/results/" +self.env+'/'+curr_time+'/results/' # path to save results - self.model_path = curr_path+"/results/" +self.env+'/'+curr_time+'/models/' # path to save models - self.start_timestep = 25e3 # Time steps initial random policy is used - self.eval_freq = 5e3 # How often (time steps) we evaluate - self.max_timestep = 200000 # Max time steps to run environment - self.expl_noise = 0.1 # Std of Gaussian exploration noise - self.batch_size = 256 # Batch size for both actor and critic - self.gamma = 0.99 # gamma factor - self.lr = 0.0005 # Target network update rate - self.policy_noise = 0.2 # Noise added to target policy during critic update - self.noise_clip = 0.5 # Range to clip target policy noise - self.policy_freq = 2 # Frequency of delayed policy updates - self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") - -# Runs policy for X episodes and returns average reward -# A fixed seed is used for the eval environment -def eval(env_name,agent, seed, eval_episodes=50): - eval_env = gym.make(env_name) - eval_env.seed(seed + 100) - rewards,ma_rewards =[],[] - for i_episode in range(eval_episodes): - ep_reward = 0 - state, done = eval_env.reset(), False - while not done: - eval_env.render() - action = agent.choose_action(np.array(state)) - state, reward, done, _ = eval_env.step(action) - ep_reward += reward - print(f"Episode:{i_episode+1}, Reward:{ep_reward:.3f}") - rewards.append(ep_reward) - # 计算滑动窗口的reward - if ma_rewards: - ma_rewards.append(0.9*ma_rewards[-1]+0.1*ep_reward) - else: - ma_rewards.append(ep_reward) - return rewards,ma_rewards - -if __name__ == "__main__": - cfg = TD3Config() - env = gym.make(cfg.env) - env.seed(cfg.seed) # Set seeds - torch.manual_seed(cfg.seed) - np.random.seed(cfg.seed) - n_states = env.observation_space.shape[0] - n_actions = env.action_space.shape[0] - max_action = float(env.action_space.high[0]) - td3= TD3(n_states,n_actions,max_action,cfg) - cfg.model_path = './TD3/results/HalfCheetah-v2/20210416-130341/models/' - td3.load(cfg.model_path) - td3_rewards,td3_ma_rewards = eval(cfg.env,td3,cfg.seed) - make_dir(cfg.result_path,cfg.model_path) - save_results(td3_rewards,td3_ma_rewards,tag='eval',path=cfg.result_path) - plot_rewards({'td3_rewards':td3_rewards,'td3_ma_rewards':td3_ma_rewards,},tag="eval",env=cfg.env,algo = cfg.algo,path=cfg.result_path) - # cfg.result_path = './TD3/results/HalfCheetah-v2/20210416-130341/' - # agent.load(cfg.result_path) - # eval(cfg.env,agent, cfg.seed) \ No newline at end of file diff --git a/projects/codes/TD3/task0_train.py b/projects/codes/TD3/task0_train.py deleted file mode 100644 index 58e4af9..0000000 --- a/projects/codes/TD3/task0_train.py +++ /dev/null @@ -1,173 +0,0 @@ -import sys,os -curr_path = os.path.dirname(__file__) -parent_path=os.path.dirname(curr_path) -sys.path.append(parent_path) # add current terminal path to sys.path - -import torch -import gym -import numpy as np -import datetime - - -from TD3.agent import TD3 -from common.plot import plot_rewards -from common.utils import save_results,make_dir - -curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # obtain current time - - -class TD3Config: - def __init__(self) -> None: - self.algo = 'TD3' - self.env = 'HalfCheetah-v2' - self.seed = 0 - self.result_path = curr_path+"/results/" +self.env+'/'+curr_time+'/results/' # path to save results - self.model_path = curr_path+"/results/" +self.env+'/'+curr_time+'/models/' # path to save models - self.start_timestep = 25e3 # Time steps initial random policy is used - self.eval_freq = 5e3 # How often (time steps) we evaluate - # self.train_eps = 800 - self.max_timestep = 4000000 # Max time steps to run environment - self.expl_noise = 0.1 # Std of Gaussian exploration noise - self.batch_size = 256 # Batch size for both actor and critic - self.gamma = 0.99 # gamma factor - self.lr = 0.0005 # Target network update rate - self.policy_noise = 0.2 # Noise added to target policy during critic update - self.noise_clip = 0.5 # Range to clip target policy noise - self.policy_freq = 2 # Frequency of delayed policy updates - self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") - -# Runs policy for X episodes and returns average reward -# A fixed seed is used for the eval environment -def eval(env,agent, seed, eval_episodes=10): - eval_env = gym.make(env) - eval_env.seed(seed + 100) - avg_reward = 0. - for _ in range(eval_episodes): - state, done = eval_env.reset(), False - while not done: - # eval_env.render() - action = agent.choose_action(np.array(state)) - state, reward, done, _ = eval_env.step(action) - avg_reward += reward - avg_reward /= eval_episodes - print("---------------------------------------") - print(f"Evaluation over {eval_episodes} episodes: {avg_reward:.3f}") - print("---------------------------------------") - return avg_reward - -def train(cfg,env,agent): - # Evaluate untrained policy - evaluations = [eval(cfg.env,agent, cfg.seed)] - state, done = env.reset(), False - ep_reward = 0 - ep_timesteps = 0 - episode_num = 0 - rewards = [] - ma_rewards = [] # moveing average reward - for t in range(int(cfg.max_timestep)): - ep_timesteps += 1 - # Select action randomly or according to policy - if t < cfg.start_timestep: - action = env.action_space.sample() - else: - action = ( - agent.choose_action(np.array(state)) - + np.random.normal(0, max_action * cfg.expl_noise, size=n_actions) - ).clip(-max_action, max_action) - # Perform action - next_state, reward, done, _ = env.step(action) - done_bool = float(done) if ep_timesteps < env._max_episode_steps else 0 - # Store data in replay buffer - agent.memory.push(state, action, next_state, reward, done_bool) - state = next_state - ep_reward += reward - # Train agent after collecting sufficient data - if t >= cfg.start_timestep: - agent.update() - if done: - # +1 to account for 0 indexing. +0 on ep_timesteps since it will increment +1 even if done=True - print(f"Episode:{episode_num+1}, Episode T:{ep_timesteps}, Reward:{ep_reward:.3f}") - # Reset environment - state, done = env.reset(), False - rewards.append(ep_reward) - # 计算滑动窗口的reward - if ma_rewards: - ma_rewards.append(0.9*ma_rewards[-1]+0.1*ep_reward) - else: - ma_rewards.append(ep_reward) - ep_reward = 0 - ep_timesteps = 0 - episode_num += 1 - # Evaluate episode - if (t + 1) % cfg.eval_freq == 0: - evaluations.append(eval(cfg.env,agent, cfg.seed)) - return rewards, ma_rewards -# def train(cfg,env,agent): -# evaluations = [eval(cfg.env,agent,cfg.seed)] -# ep_reward = 0 -# tot_timestep = 0 -# rewards = [] -# ma_rewards = [] # moveing average reward -# for i_ep in range(int(cfg.train_eps)): -# state, done = env.reset(), False -# ep_reward = 0 -# ep_timestep = 0 -# while not done: -# ep_timestep += 1 -# tot_timestep +=1 -# # Select action randomly or according to policy -# if tot_timestep < cfg.start_timestep: -# action = env.action_space.sample() -# else: -# action = ( -# agent.choose_action(np.array(state)) -# + np.random.normal(0, max_action * cfg.expl_noise, size=n_actions) -# ).clip(-max_action, max_action) -# # action = ( -# # agent.choose_action(np.array(state)) -# # + np.random.normal(0, max_action * cfg.expl_noise, size=n_actions) -# # ).clip(-max_action, max_action) -# # Perform action -# next_state, reward, done, _ = env.step(action) -# done_bool = float(done) if ep_timestep < env._max_episode_steps else 0 - -# # Store data in replay buffer -# agent.memory.push(state, action, next_state, reward, done_bool) -# state = next_state -# ep_reward += reward -# # Train agent after collecting sufficient data -# if tot_timestep >= cfg.start_timestep: -# agent.update() -# print(f"Episode:{i_ep}/{cfg.train_eps}, Episode Timestep:{ep_timestep}, Reward:{ep_reward:.3f}") -# rewards.append(ep_reward) -# # 计算滑动窗口的reward -# if ma_rewards: -# ma_rewards.append(0.9*ma_rewards[-1]+0.1*ep_reward) -# else: -# ma_rewards.append(ep_reward) -# # Evaluate episode -# if (i_ep+1) % cfg.eval_freq == 0: -# evaluations.append(eval(cfg.env,agent, cfg.seed)) -# return rewards,ma_rewards - - -if __name__ == "__main__": - cfg = TD3Config() - env = gym.make(cfg.env) - env.seed(cfg.seed) # Set seeds - torch.manual_seed(cfg.seed) - np.random.seed(cfg.seed) - n_states = env.observation_space.shape[0] - n_actions = env.action_space.shape[0] - max_action = float(env.action_space.high[0]) - agent = TD3(n_states,n_actions,max_action,cfg) - rewards,ma_rewards = train(cfg,env,agent) - make_dir(cfg.result_path,cfg.model_path) - agent.save(path=cfg.model_path) - save_results(rewards,ma_rewards,tag='train',path=cfg.result_path) - plot_rewards(rewards,ma_rewards,tag="train",env=cfg.env,algo = cfg.algo,path=cfg.result_path) - # cfg.result_path = './TD3/results/HalfCheetah-v2/20210416-130341/' - # agent.load(cfg.result_path) - # eval(cfg.env,agent, cfg.seed) - - diff --git a/projects/codes/TD3/task1_eval.py b/projects/codes/TD3/task1_eval.py deleted file mode 100644 index 0d28c48..0000000 --- a/projects/codes/TD3/task1_eval.py +++ /dev/null @@ -1,83 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: JiangJi -Email: johnjim0816@gmail.com -Date: 2021-04-23 20:36:23 -LastEditor: JiangJi -LastEditTime: 2021-04-28 10:14:33 -Discription: -Environment: -''' -import sys,os -curr_path = os.path.dirname(__file__) -parent_path=os.path.dirname(curr_path) -sys.path.append(parent_path) # add current terminal path to sys.path - -import torch -import gym -import numpy as np -import datetime - - -from TD3.agent import TD3 -from common.plot import plot_rewards -from common.utils import save_results,make_dir - -curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # obtain current time - -class TD3Config: - def __init__(self) -> None: - self.algo = 'TD3' - self.env = 'Pendulum-v0' - self.seed = 0 - self.result_path = curr_path+"/results/" +self.env+'/'+curr_time+'/results/' # path to save results - self.model_path = curr_path+"/results/" +self.env+'/'+curr_time+'/models/' # path to save models - self.batch_size = 256 # Batch size for both actor and critic - self.gamma = 0.99 # gamma factor - self.lr = 0.0005 # Target network update rate - self.policy_noise = 0.2 # Noise added to target policy during critic update - self.noise_clip = 0.5 # Range to clip target policy noise - self.policy_freq = 2 # Frequency of delayed policy updates - self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") - -# Runs policy for X episodes and returns average reward -# A fixed seed is used for the eval environment -def eval(env_name,agent, seed, eval_episodes=50): - eval_env = gym.make(env_name) - eval_env.seed(seed + 100) - rewards,ma_rewards =[],[] - for i_episode in range(eval_episodes): - ep_reward = 0 - state, done = eval_env.reset(), False - while not done: - # eval_env.render() - action = agent.choose_action(np.array(state)) - state, reward, done, _ = eval_env.step(action) - ep_reward += reward - print(f"Episode:{i_episode+1}, Reward:{ep_reward:.3f}") - rewards.append(ep_reward) - # 计算滑动窗口的reward - if ma_rewards: - ma_rewards.append(0.9*ma_rewards[-1]+0.1*ep_reward) - else: - ma_rewards.append(ep_reward) - return rewards,ma_rewards - -if __name__ == "__main__": - cfg = TD3Config() - env = gym.make(cfg.env) - env.seed(cfg.seed) # Set seeds - torch.manual_seed(cfg.seed) - np.random.seed(cfg.seed) - n_states = env.observation_space.shape[0] - n_actions = env.action_space.shape[0] - max_action = float(env.action_space.high[0]) - td3= TD3(n_states,n_actions,max_action,cfg) - cfg.model_path = './TD3/results/Pendulum-v0/20210428-092059/models/' - cfg.result_path = './TD3/results/Pendulum-v0/20210428-092059/results/' - td3.load(cfg.model_path) - rewards,ma_rewards = eval(cfg.env,td3,cfg.seed) - make_dir(cfg.result_path,cfg.model_path) - save_results(rewards,ma_rewards,tag='eval',path=cfg.result_path) - plot_rewards(rewards,ma_rewards,tag="train",env=cfg.env,algo = cfg.algo,path=cfg.result_path) \ No newline at end of file diff --git a/projects/codes/TD3/task1_train.py b/projects/codes/TD3/task1_train.py deleted file mode 100644 index 868f686..0000000 --- a/projects/codes/TD3/task1_train.py +++ /dev/null @@ -1,122 +0,0 @@ -import sys,os -curr_path = os.path.dirname(os.path.abspath(__file__)) # 当前文件所在绝对路径 -parent_path = os.path.dirname(curr_path) # 父路径 -sys.path.append(parent_path) # 添加路径到系统路径 - -import torch -import gym -import numpy as np -import datetime - -from TD3.agent import TD3 -from common.plot import plot_rewards -from common.utils import save_results,make_dir - -curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # 获取当前时间 - - -class TD3Config: - def __init__(self) -> None: - self.algo = 'TD3' # 算法名称 - self.env_name = 'Pendulum-v1' # 环境名称 - self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # 检测GPU - self.train_eps = 600 # 训练的回合数 - self.start_timestep = 25e3 # Time steps initial random policy is used - self.epsilon_start = 50 # Episodes initial random policy is used - self.eval_freq = 10 # How often (episodes) we evaluate - self.max_timestep = 100000 # Max time steps to run environment - self.expl_noise = 0.1 # Std of Gaussian exploration noise - self.batch_size = 256 # Batch size for both actor and critic - self.gamma = 0.9 # gamma factor - self.lr = 0.0005 # 学习率 - self.policy_noise = 0.2 # Noise added to target policy during critic update - self.noise_clip = 0.3 # Range to clip target policy noise - self.policy_freq = 2 # Frequency of delayed policy updates -class PlotConfig(TD3Config): - def __init__(self) -> None: - super().__init__() - self.result_path = curr_path+"/outputs/" + self.env_name + \ - '/'+curr_time+'/results/' # 保存结果的路径 - self.model_path = curr_path+"/outputs/" + self.env_name + \ - '/'+curr_time+'/models/' # 保存模型的路径 - self.save = True # 是否保存图片 - - - -# Runs policy for X episodes and returns average reward -# A fixed seed is used for the eval environment -def eval(env,agent, seed, eval_episodes=10): - eval_env = gym.make(env) - eval_env.seed(seed + 100) - avg_reward = 0. - for _ in range(eval_episodes): - state, done = eval_env.reset(), False - while not done: - # eval_env.render() - action = agent.choose_action(np.array(state)) - state, reward, done, _ = eval_env.step(action) - avg_reward += reward - avg_reward /= eval_episodes - print("---------------------------------------") - print(f"Evaluation over {eval_episodes} episodes: {avg_reward:.3f}") - print("---------------------------------------") - return avg_reward - -def train(cfg,env,agent): - print('开始训练!') - print(f'环境：{cfg.env_name}, 算法：{cfg.algo}, 设备：{cfg.device}') - rewards = [] # 记录所有回合的奖励 - ma_rewards = [] # 记录所有回合的滑动平均奖励 - for i_ep in range(int(cfg.train_eps)): - ep_reward = 0 - ep_timesteps = 0 - state, done = env.reset(), False - while not done: - ep_timesteps += 1 - # Select action randomly or according to policy - if i_ep < cfg.epsilon_start: - action = env.action_space.sample() - else: - action = ( - agent.choose_action(np.array(state)) - + np.random.normal(0, max_action * cfg.expl_noise, size=n_actions) - ).clip(-max_action, max_action) - # Perform action - next_state, reward, done, _ = env.step(action) - done_bool = float(done) if ep_timesteps < env._max_episode_steps else 0 - # Store data in replay buffer - agent.memory.push(state, action, next_state, reward, done_bool) - state = next_state - ep_reward += reward - # Train agent after collecting sufficient data - if i_ep+1 >= cfg.epsilon_start: - agent.update() - if (i_ep+1)%10 == 0: - print('回合：{}/{}, 奖励：{:.2f}'.format(i_ep+1, cfg.train_eps, ep_reward)) - rewards.append(ep_reward) - if ma_rewards: - ma_rewards.append(0.9*ma_rewards[-1]+0.1*ep_reward) - else: - ma_rewards.append(ep_reward) - print('完成训练！') - return rewards, ma_rewards - - -if __name__ == "__main__": - cfg = TD3Config() - plot_cfg = PlotConfig() - env = gym.make(cfg.env_name) - env.seed(1) # 随机种子 - torch.manual_seed(1) - np.random.seed(1) - n_states = env.observation_space.shape[0] - n_actions = env.action_space.shape[0] - max_action = float(env.action_space.high[0]) - agent = TD3(n_states,n_actions,max_action,cfg) - rewards,ma_rewards = train(cfg,env,agent) - make_dir(plot_cfg.result_path,plot_cfg.model_path) - agent.save(path=plot_cfg.model_path) - save_results(rewards,ma_rewards,tag='train',path=plot_cfg.result_path) - plot_rewards(rewards,ma_rewards,plot_cfg,tag="train") - - diff --git a/projects/codes/assets/image-20200820174307301.png b/projects/codes/assets/image-20200820174307301.png deleted file mode 100644 index 1197da0..0000000 Binary files a/projects/codes/assets/image-20200820174307301.png and /dev/null differ diff --git a/projects/codes/assets/image-20200820174814084.png b/projects/codes/assets/image-20200820174814084.png deleted file mode 100644 index 4c9e3dc..0000000 Binary files a/projects/codes/assets/image-20200820174814084.png and /dev/null differ diff --git a/projects/codes/common/atari_wrappers.py b/projects/codes/common/atari_wrappers.py deleted file mode 100644 index 48dab94..0000000 --- a/projects/codes/common/atari_wrappers.py +++ /dev/null @@ -1,284 +0,0 @@ -import numpy as np -import os -os.environ.setdefault('PATH', '') -from collections import deque -import gym -from gym import spaces -import cv2 -cv2.ocl.setUseOpenCL(False) -from .wrappers import TimeLimit - - -class NoopResetEnv(gym.Wrapper): - def __init__(self, env, noop_max=30): - """Sample initial states by taking random number of no-ops on reset. - No-op is assumed to be action 0. - """ - gym.Wrapper.__init__(self, env) - self.noop_max = noop_max - self.override_num_noops = None - self.noop_action = 0 - assert env.unwrapped.get_action_meanings()[0] == 'NOOP' - - def reset(self, **kwargs): - """ Do no-op action for a number of steps in [1, noop_max].""" - self.env.reset(**kwargs) - if self.override_num_noops is not None: - noops = self.override_num_noops - else: - noops = self.unwrapped.np_random.randint(1, self.noop_max + 1) #pylint: disable=E1101 - assert noops > 0 - obs = None - for _ in range(noops): - obs, _, done, _ = self.env.step(self.noop_action) - if done: - obs = self.env.reset(**kwargs) - return obs - - def step(self, ac): - return self.env.step(ac) - -class FireResetEnv(gym.Wrapper): - def __init__(self, env): - """Take action on reset for environments that are fixed until firing.""" - gym.Wrapper.__init__(self, env) - assert env.unwrapped.get_action_meanings()[1] == 'FIRE' - assert len(env.unwrapped.get_action_meanings()) >= 3 - - def reset(self, **kwargs): - self.env.reset(**kwargs) - obs, _, done, _ = self.env.step(1) - if done: - self.env.reset(**kwargs) - obs, _, done, _ = self.env.step(2) - if done: - self.env.reset(**kwargs) - return obs - - def step(self, ac): - return self.env.step(ac) - -class EpisodicLifeEnv(gym.Wrapper): - def __init__(self, env): - """Make end-of-life == end-of-episode, but only reset on true game over. - Done by DeepMind for the DQN and co. since it helps value estimation. - """ - gym.Wrapper.__init__(self, env) - self.lives = 0 - self.was_real_done = True - - def step(self, action): - obs, reward, done, info = self.env.step(action) - self.was_real_done = done - # check current lives, make loss of life terminal, - # then update lives to handle bonus lives - lives = self.env.unwrapped.ale.lives() - if lives < self.lives and lives > 0: - # for Qbert sometimes we stay in lives == 0 condition for a few frames - # so it's important to keep lives > 0, so that we only reset once - # the environment advertises done. - done = True - self.lives = lives - return obs, reward, done, info - - def reset(self, **kwargs): - """Reset only when lives are exhausted. - This way all states are still reachable even though lives are episodic, - and the learner need not know about any of this behind-the-scenes. - """ - if self.was_real_done: - obs = self.env.reset(**kwargs) - else: - # no-op step to advance from terminal/lost life state - obs, _, _, _ = self.env.step(0) - self.lives = self.env.unwrapped.ale.lives() - return obs - -class MaxAndSkipEnv(gym.Wrapper): - def __init__(self, env, skip=4): - """Return only every `skip`-th frame""" - gym.Wrapper.__init__(self, env) - # most recent raw observations (for max pooling across time steps) - self._obs_buffer = np.zeros((2,)+env.observation_space.shape, dtype=np.uint8) - self._skip = skip - - def step(self, action): - """Repeat action, sum reward, and max over last observations.""" - total_reward = 0.0 - done = None - for i in range(self._skip): - obs, reward, done, info = self.env.step(action) - if i == self._skip - 2: self._obs_buffer[0] = obs - if i == self._skip - 1: self._obs_buffer[1] = obs - total_reward += reward - if done: - break - # Note that the observation on the done=True frame - # doesn't matter - max_frame = self._obs_buffer.max(axis=0) - - return max_frame, total_reward, done, info - - def reset(self, **kwargs): - return self.env.reset(**kwargs) - -class ClipRewardEnv(gym.RewardWrapper): - def __init__(self, env): - gym.RewardWrapper.__init__(self, env) - - def reward(self, reward): - """Bin reward to {+1, 0, -1} by its sign.""" - return np.sign(reward) - - -class WarpFrame(gym.ObservationWrapper): - def __init__(self, env, width=84, height=84, grayscale=True, dict_space_key=None): - """ - Warp frames to 84x84 as done in the Nature paper and later work. - If the environment uses dictionary observations, `dict_space_key` can be specified which indicates which - observation should be warped. - """ - super().__init__(env) - self._width = width - self._height = height - self._grayscale = grayscale - self._key = dict_space_key - if self._grayscale: - num_colors = 1 - else: - num_colors = 3 - - new_space = gym.spaces.Box( - low=0, - high=255, - shape=(self._height, self._width, num_colors), - dtype=np.uint8, - ) - if self._key is None: - original_space = self.observation_space - self.observation_space = new_space - else: - original_space = self.observation_space.spaces[self._key] - self.observation_space.spaces[self._key] = new_space - assert original_space.dtype == np.uint8 and len(original_space.shape) == 3 - - def observation(self, obs): - if self._key is None: - frame = obs - else: - frame = obs[self._key] - - if self._grayscale: - frame = cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY) - frame = cv2.resize( - frame, (self._width, self._height), interpolation=cv2.INTER_AREA - ) - if self._grayscale: - frame = np.expand_dims(frame, -1) - - if self._key is None: - obs = frame - else: - obs = obs.copy() - obs[self._key] = frame - return obs - - -class FrameStack(gym.Wrapper): - def __init__(self, env, k): - """Stack k last frames. - Returns lazy array, which is much more memory efficient. - See Also - -------- - baselines.common.atari_wrappers.LazyFrames - """ - gym.Wrapper.__init__(self, env) - self.k = k - self.frames = deque([], maxlen=k) - shp = env.observation_space.shape - self.observation_space = spaces.Box(low=0, high=255, shape=(shp[:-1] + (shp[-1] * k,)), dtype=env.observation_space.dtype) - - def reset(self): - ob = self.env.reset() - for _ in range(self.k): - self.frames.append(ob) - return self._get_ob() - - def step(self, action): - ob, reward, done, info = self.env.step(action) - self.frames.append(ob) - return self._get_ob(), reward, done, info - - def _get_ob(self): - assert len(self.frames) == self.k - return LazyFrames(list(self.frames)) - -class ScaledFloatFrame(gym.ObservationWrapper): - def __init__(self, env): - gym.ObservationWrapper.__init__(self, env) - self.observation_space = gym.spaces.Box(low=0, high=1, shape=env.observation_space.shape, dtype=np.float32) - - def observation(self, observation): - # careful! This undoes the memory optimization, use - # with smaller replay buffers only. - return np.array(observation).astype(np.float32) / 255.0 - -class LazyFrames(object): - def __init__(self, frames): - """This object ensures that common frames between the observations are only stored once. - It exists purely to optimize memory usage which can be huge for DQN's 1M frames replay - buffers. - This object should only be converted to numpy array before being passed to the model. - You'd not believe how complex the previous solution was.""" - self._frames = frames - self._out = None - - def _force(self): - if self._out is None: - self._out = np.concatenate(self._frames, axis=-1) - self._frames = None - return self._out - - def __array__(self, dtype=None): - out = self._force() - if dtype is not None: - out = out.astype(dtype) - return out - - def __len__(self): - return len(self._force()) - - def __getitem__(self, i): - return self._force()[i] - - def count(self): - frames = self._force() - return frames.shape[frames.ndim - 1] - - def frame(self, i): - return self._force()[..., i] - -def make_atari(env_id, max_episode_steps=None): - env = gym.make(env_id) - assert 'NoFrameskip' in env.spec.id - env = NoopResetEnv(env, noop_max=30) - env = MaxAndSkipEnv(env, skip=4) - if max_episode_steps is not None: - env = TimeLimit(env, max_episode_steps=max_episode_steps) - return env - -def wrap_deepmind(env, episode_life=True, clip_rewards=True, frame_stack=False, scale=False): - """Configure environment for DeepMind-style Atari. - """ - if episode_life: - env = EpisodicLifeEnv(env) - if 'FIRE' in env.unwrapped.get_action_meanings(): - env = FireResetEnv(env) - env = WarpFrame(env) - if scale: - env = ScaledFloatFrame(env) - if clip_rewards: - env = ClipRewardEnv(env) - if frame_stack: - env = FrameStack(env, 4) - return env \ No newline at end of file diff --git a/projects/codes/common/config.py b/projects/codes/common/config.py deleted file mode 100644 index da0beb9..0000000 --- a/projects/codes/common/config.py +++ /dev/null @@ -1,38 +0,0 @@ - -class DefaultConfig: - def __init__(self) -> None: - pass - def print_cfg(self): - print(self.__dict__) -class GeneralConfig(DefaultConfig): - def __init__(self) -> None: - self.env_name = "CartPole-v1" # name of environment - self.algo_name = "DQN" # name of algorithm - self.mode = "train" # train or test - self.seed = 0 # random seed - self.device = "cuda" # device to use - self.train_eps = 200 # number of episodes for training - self.test_eps = 20 # number of episodes for testing - self.eval_eps = 10 # number of episodes for evaluation - self.eval_per_episode = 5 # evaluation per episode - self.max_steps = 200 # max steps for each episode - self.load_checkpoint = False - self.load_path = None # path to load model - self.show_fig = False # show figure or not - self.save_fig = True # save figure or not - -class AlgoConfig(DefaultConfig): - def __init__(self) -> None: - # set epsilon_start=epsilon_end can obtain fixed epsilon=epsilon_end - # self.epsilon_start = 0.95 # epsilon start value - # self.epsilon_end = 0.01 # epsilon end value - # self.epsilon_decay = 500 # epsilon decay rate - self.gamma = 0.95 # discount factor - # self.lr = 0.0001 # learning rate - # self.buffer_size = 100000 # size of replay buffer - # self.batch_size = 64 # batch size - # self.target_update = 4 # target network update frequency -class MergedConfig: - def __init__(self) -> None: - pass - \ No newline at end of file diff --git a/projects/codes/common/launcher.py b/projects/codes/common/launcher.py deleted file mode 100644 index 2c0793c..0000000 --- a/projects/codes/common/launcher.py +++ /dev/null @@ -1,124 +0,0 @@ -from common.utils import get_logger,save_results,save_cfgs,plot_rewards,merge_class_attrs,load_cfgs -from common.config import GeneralConfig,AlgoConfig,MergedConfig -import time -from pathlib import Path -import datetime -import argparse - -class Launcher: - def __init__(self) -> None: - self.get_cfg() - def get_cfg(self): - self.cfgs = {'general_cfg':GeneralConfig(),'algo_cfg':AlgoConfig()} # create config - def process_yaml_cfg(self): - ''' load yaml config - ''' - parser = argparse.ArgumentParser(description="hyperparameters") - parser.add_argument('--yaml', default = None, type=str,help='the path of config file') - args = parser.parse_args() - if args.yaml is not None: - load_cfgs(self.cfgs, args.yaml) - def print_cfg(self,cfg): - ''' print parameters - ''' - cfg_dict = vars(cfg) - print("Hyperparameters:") - print(''.join(['=']*80)) - tplt = "{:^20}\t{:^20}\t{:^20}" - print(tplt.format("Name", "Value", "Type")) - for k,v in cfg_dict.items(): - print(tplt.format(k,v,str(type(v)))) - print(''.join(['=']*80)) - def env_agent_config(self,cfg,logger): - env,agent = None,None - return env,agent - def train_one_episode(self,env, agent, cfg): - ep_reward = 0 - ep_step = 0 - return agent,ep_reward,ep_step - def test_one_episode(self, env, agent, cfg): - ep_reward = 0 - ep_step = 0 - return agent,ep_reward,ep_step - def evaluate(self, env, agent, cfg): - sum_eval_reward = 0 - for _ in range(cfg.eval_eps): - _,eval_ep_reward,_ = self.test_one_episode(env, agent, cfg) - sum_eval_reward += eval_ep_reward - mean_eval_reward = sum_eval_reward/cfg.eval_eps - return mean_eval_reward - # def train(self,cfg, env, agent,logger): - # res_dic = {} - # return res_dic - # def test(self,cfg, env, agent,logger): - # res_dic = {} - # return res_dic - def create_path(self,cfg): - curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # obtain current time - self.task_dir = f"{cfg.mode.capitalize()}_{cfg.env_name}_{cfg.algo_name}_{curr_time}" - Path(self.task_dir).mkdir(parents=True, exist_ok=True) - self.model_dir = f"{self.task_dir}/models/" - self.res_dir = f"{self.task_dir}/results/" - self.log_dir = f"{self.task_dir}/logs/" - def run(self): - self.process_yaml_cfg() # load yaml config - cfg = MergedConfig() # merge config - cfg = merge_class_attrs(cfg,self.cfgs['general_cfg']) - cfg = merge_class_attrs(cfg,self.cfgs['algo_cfg']) - self.print_cfg(cfg) # print the configuration - self.create_path(cfg) # create the path to save the results - logger = get_logger(self.log_dir) # create the logger - env, agent = self.env_agent_config(cfg,logger) - if cfg.load_checkpoint: - agent.load_model(f"{cfg.load_path}/models/") - logger.info(f"Start {cfg.mode}ing!") - logger.info(f"Env: {cfg.env_name}, Algorithm: {cfg.algo_name}, Device: {cfg.device}") - rewards = [] # record rewards for all episodes - steps = [] # record steps for all episodes - if cfg.mode.lower() == 'train': - best_ep_reward = -float('inf') - for i_ep in range(cfg.train_eps): - agent,ep_reward,ep_step = self.train_one_episode(env, agent, cfg) - logger.info(f"Episode: {i_ep+1}/{cfg.train_eps}, Reward: {ep_reward:.3f}, Step: {ep_step}") - rewards.append(ep_reward) - steps.append(ep_step) - # for _ in range - if (i_ep+1)%cfg.eval_per_episode == 0: - mean_eval_reward = self.evaluate(env, agent, cfg) - if mean_eval_reward >= best_ep_reward: # update best reward - logger.info(f"Current episode {i_ep+1} has the best eval reward: {mean_eval_reward:.3f}") - best_ep_reward = mean_eval_reward - agent.save_model(self.model_dir) # save models with best reward - # env.close() - elif cfg.mode.lower() == 'test': - for i_ep in range(cfg.test_eps): - agent,ep_reward,ep_step = self.test_one_episode(env, agent, cfg) - logger.info(f"Episode: {i_ep+1}/{cfg.test_eps}, Reward: {ep_reward:.3f}, Step: {ep_step}") - rewards.append(ep_reward) - steps.append(ep_step) - agent.save_model(self.model_dir) # save models - # env.close() - logger.info(f"Finish {cfg.mode}ing!") - res_dic = {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps} - save_results(res_dic, self.res_dir) # save results - save_cfgs(self.cfgs, self.task_dir) # save config - plot_rewards(rewards, title=f"{cfg.mode.lower()}ing curve on {cfg.device} of {cfg.algo_name} for {cfg.env_name}" ,fpath= self.res_dir) - # def run(self): - # self.process_yaml_cfg() # load yaml config - # cfg = MergedConfig() # merge config - # cfg = merge_class_attrs(cfg,self.cfgs['general_cfg']) - # cfg = merge_class_attrs(cfg,self.cfgs['algo_cfg']) - # self.print_cfg(cfg) # print the configuration - # self.create_path(cfg) # create the path to save the results - # logger = get_logger(self.log_dir) # create the logger - # env, agent = self.env_agent_config(cfg,logger) - # if cfg.load_checkpoint: - # agent.load_model(f"{cfg.load_path}/models/") - # if cfg.mode.lower() == 'train': - # res_dic = self.train(cfg, env, agent,logger) - # elif cfg.mode.lower() == 'test': - # res_dic = self.test(cfg, env, agent,logger) - # save_results(res_dic, self.res_dir) # save results - # save_cfgs(self.cfgs, self.task_dir) # save config - # agent.save_model(self.model_dir) # save models - # plot_rewards(res_dic['rewards'], title=f"{cfg.mode.lower()}ing curve on {cfg.device} of {cfg.algo_name} for {cfg.env_name}" ,fpath= self.res_dir) \ No newline at end of file diff --git a/projects/codes/common/memories.py b/projects/codes/common/memories.py deleted file mode 100644 index fd50ab9..0000000 --- a/projects/codes/common/memories.py +++ /dev/null @@ -1,207 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -@Author: John -@Email: johnjim0816@gmail.com -@Date: 2020-06-10 15:27:16 -@LastEditor: John -LastEditTime: 2022-08-28 23:44:06 -@Discription: -@Environment: python 3.7.7 -''' -import random -import numpy as np -from collections import deque -class ReplayBuffer: - def __init__(self, capacity): - self.capacity = capacity # 经验回放的容量 - self.buffer = [] # 缓冲区 - self.position = 0 - - def push(self, state, action, reward, next_state, done): - ''' 缓冲区是一个队列，容量超出时去掉开始存入的转移(transition) - ''' - if len(self.buffer) < self.capacity: - self.buffer.append(None) - self.buffer[self.position] = (state, action, reward, next_state, done) - self.position = (self.position + 1) % self.capacity - - def sample(self, batch_size): - batch = random.sample(self.buffer, batch_size) # 随机采出小批量转移 - state, action, reward, next_state, done = zip(*batch) # 解压成状态，动作等 - return state, action, reward, next_state, done - - def __len__(self): - ''' 返回当前存储的量 - ''' - return len(self.buffer) - -class ReplayBufferQue: - def __init__(self, capacity: int) -> None: - self.capacity = capacity - self.buffer = deque(maxlen=self.capacity) - def push(self,transitions): - '''_summary_ - Args: - trainsitions (tuple): _description_ - ''' - self.buffer.append(transitions) - def sample(self, batch_size: int, sequential: bool = False): - if batch_size > len(self.buffer): - batch_size = len(self.buffer) - if sequential: # sequential sampling - rand = random.randint(0, len(self.buffer) - batch_size) - batch = [self.buffer[i] for i in range(rand, rand + batch_size)] - return zip(*batch) - else: - batch = random.sample(self.buffer, batch_size) - return zip(*batch) - def clear(self): - self.buffer.clear() - def __len__(self): - return len(self.buffer) - -class PGReplay(ReplayBufferQue): - '''replay buffer for policy gradient based methods, each time these methods will sample all transitions - Args: - ReplayBufferQue (_type_): _description_ - ''' - def __init__(self): - self.buffer = deque() - def sample(self): - ''' sample all the transitions - ''' - batch = list(self.buffer) - return zip(*batch) - -class SumTree: - '''SumTree for the per(Prioritized Experience Replay) DQN. - This SumTree code is a modified version and the original code is from: - https://github.com/MorvanZhou/Reinforcement-learning-with-tensorflow/blob/master/contents/5.2_Prioritized_Replay_DQN/RL_brain.py - ''' - def __init__(self, capacity: int): - self.capacity = capacity - self.data_pointer = 0 - self.n_entries = 0 - self.tree = np.zeros(2 * capacity - 1) - self.data = np.zeros(capacity, dtype = object) - - def update(self, tree_idx, p): - '''Update the sampling weight - ''' - change = p - self.tree[tree_idx] - self.tree[tree_idx] = p - - while tree_idx != 0: - tree_idx = (tree_idx - 1) // 2 - self.tree[tree_idx] += change - - def add(self, p, data): - '''Adding new data to the sumTree - ''' - tree_idx = self.data_pointer + self.capacity - 1 - self.data[self.data_pointer] = data - # print ("tree_idx=", tree_idx) - # print ("nonzero = ", np.count_nonzero(self.tree)) - self.update(tree_idx, p) - - self.data_pointer += 1 - if self.data_pointer >= self.capacity: - self.data_pointer = 0 - - if self.n_entries < self.capacity: - self.n_entries += 1 - - def get_leaf(self, v): - '''Sampling the data - ''' - parent_idx = 0 - while True: - cl_idx = 2 * parent_idx + 1 - cr_idx = cl_idx + 1 - if cl_idx >= len(self.tree): - leaf_idx = parent_idx - break - else: - if v <= self.tree[cl_idx] : - parent_idx = cl_idx - else: - v -= self.tree[cl_idx] - parent_idx = cr_idx - - data_idx = leaf_idx - self.capacity + 1 - return leaf_idx, self.tree[leaf_idx], self.data[data_idx] - - def total(self): - return int(self.tree[0]) - -class ReplayTree: - '''ReplayTree for the per(Prioritized Experience Replay) DQN. - ''' - def __init__(self, capacity): - self.capacity = capacity # the capacity for memory replay - self.tree = SumTree(capacity) - self.abs_err_upper = 1. - - ## hyper parameter for calculating the importance sampling weight - self.beta_increment_per_sampling = 0.001 - self.alpha = 0.6 - self.beta = 0.4 - self.epsilon = 0.01 - self.abs_err_upper = 1. - - def __len__(self): - ''' return the num of storage - ''' - return self.tree.total() - - def push(self, error, sample): - '''Push the sample into the replay according to the importance sampling weight - ''' - p = (np.abs(error) + self.epsilon) ** self.alpha - self.tree.add(p, sample) - - - def sample(self, batch_size): - '''This is for sampling a batch data and the original code is from: - https://github.com/rlcode/per/blob/master/prioritized_memory.py - ''' - pri_segment = self.tree.total() / batch_size - - priorities = [] - batch = [] - idxs = [] - - is_weights = [] - - self.beta = np.min([1., self.beta + self.beta_increment_per_sampling]) - min_prob = np.min(self.tree.tree[-self.tree.capacity:]) / self.tree.total() - - for i in range(batch_size): - a = pri_segment * i - b = pri_segment * (i+1) - - s = random.uniform(a, b) - idx, p, data = self.tree.get_leaf(s) - - priorities.append(p) - batch.append(data) - idxs.append(idx) - prob = p / self.tree.total() - - sampling_probabilities = np.array(priorities) / self.tree.total() - is_weights = np.power(self.tree.n_entries * sampling_probabilities, -self.beta) - is_weights /= is_weights.max() - - return zip(*batch), idxs, is_weights - - def batch_update(self, tree_idx, abs_errors): - '''Update the importance sampling weight - ''' - abs_errors += self.epsilon - - clipped_errors = np.minimum(abs_errors, self.abs_err_upper) - ps = np.power(clipped_errors, self.alpha) - - for ti, p in zip(tree_idx, ps): - self.tree.update(ti, p) diff --git a/projects/codes/common/models.py b/projects/codes/common/models.py deleted file mode 100644 index 41d1b17..0000000 --- a/projects/codes/common/models.py +++ /dev/null @@ -1,139 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: John -Email: johnjim0816@gmail.com -Date: 2021-03-12 21:14:12 -LastEditor: John -LastEditTime: 2022-10-31 23:53:06 -Discription: -Environment: -''' -import torch -import torch.nn as nn -import torch.nn.functional as F -from torch.distributions import Categorical - -class MLP(nn.Module): - def __init__(self, input_dim,output_dim,hidden_dim=128): - """ 初始化q网络，为全连接网络 - input_dim: 输入的特征数即环境的状态维度 - output_dim: 输出的动作维度 - """ - super(MLP, self).__init__() - self.fc1 = nn.Linear(input_dim, hidden_dim) # 输入层 - self.fc2 = nn.Linear(hidden_dim,hidden_dim) # 隐藏层 - self.fc3 = nn.Linear(hidden_dim, output_dim) # 输出层 - - def forward(self, x): - # 各层对应的激活函数 - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - return self.fc3(x) - -class ActorSoftmax(nn.Module): - def __init__(self, input_dim, output_dim, hidden_dim=256): - super(ActorSoftmax, self).__init__() - self.fc1 = nn.Linear(input_dim, hidden_dim) - self.fc2 = nn.Linear(hidden_dim, hidden_dim) - self.fc3 = nn.Linear(hidden_dim, output_dim) - def forward(self,x): - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - probs = F.softmax(self.fc3(x),dim=1) - return probs - -class ActorSoftmaxTanh(nn.Module): - def __init__(self, input_dim, output_dim, hidden_dim=256): - super(ActorSoftmaxTanh, self).__init__() - self.fc1 = nn.Linear(input_dim, hidden_dim) - self.fc2 = nn.Linear(hidden_dim, hidden_dim) - self.fc3 = nn.Linear(hidden_dim, output_dim) - def forward(self,x): - x = F.tanh(self.fc1(x)) - x = F.tanh(self.fc2(x)) - probs = F.softmax(self.fc3(x),dim=1) - return probs -class ActorNormal(nn.Module): - def __init__(self, n_states,n_actions, hidden_dim=256): - super(ActorNormal, self).__init__() - self.fc1 = nn.Linear(n_states, hidden_dim) - self.fc2 = nn.Linear(hidden_dim, hidden_dim) - self.fc3 = nn.Linear(hidden_dim, n_actions) - self.fc4 = nn.Linear(hidden_dim, n_actions) - def forward(self,x): - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - mu = torch.tanh(self.fc3(x)) - sigma = F.softplus(self.fc4(x)) + 0.001 # avoid 0 - return mu,sigma -# class ActorSoftmax(nn.Module): -# def __init__(self,input_dim, output_dim, -# hidden_dim=256): -# super(ActorSoftmax, self).__init__() -# self.actor = nn.Sequential( -# nn.Linear(input_dim, hidden_dim), -# nn.ReLU(), -# nn.Linear(hidden_dim, hidden_dim), -# nn.ReLU(), -# nn.Linear(hidden_dim, output_dim), -# nn.Softmax(dim=-1) -# ) -# def forward(self, state): -# probs = self.actor(state) -# dist = Categorical(probs) -# return dist -class Critic(nn.Module): - def __init__(self,input_dim,output_dim,hidden_dim=256): - super(Critic,self).__init__() - assert output_dim == 1 # critic must output a single value - self.fc1 = nn.Linear(input_dim, hidden_dim) - self.fc2 = nn.Linear(hidden_dim, hidden_dim) - self.fc3 = nn.Linear(hidden_dim, output_dim) - def forward(self,x): - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - value = self.fc3(x) - return value - -class ActorCriticSoftmax(nn.Module): - def __init__(self, input_dim, output_dim, actor_hidden_dim=256,critic_hidden_dim=256): - super(ActorCriticSoftmax, self).__init__() - - self.critic_fc1 = nn.Linear(input_dim, critic_hidden_dim) - self.critic_fc2 = nn.Linear(critic_hidden_dim, 1) - - self.actor_fc1 = nn.Linear(input_dim, actor_hidden_dim) - self.actor_fc2 = nn.Linear(actor_hidden_dim, output_dim) - - def forward(self, state): - # state = Variable(torch.from_numpy(state).float().unsqueeze(0)) - value = F.relu(self.critic_fc1(state)) - value = self.critic_fc2(value) - - policy_dist = F.relu(self.actor_fc1(state)) - policy_dist = F.softmax(self.actor_fc2(policy_dist), dim=1) - - return value, policy_dist - -class ActorCritic(nn.Module): - def __init__(self, input_dim, output_dim, hidden_dim=256): - super(ActorCritic, self).__init__() - self.critic = nn.Sequential( - nn.Linear(input_dim, hidden_dim), - nn.ReLU(), - nn.Linear(hidden_dim, 1) - ) - - self.actor = nn.Sequential( - nn.Linear(input_dim, hidden_dim), - nn.ReLU(), - nn.Linear(hidden_dim, output_dim), - nn.Softmax(dim=1), - ) - - def forward(self, x): - value = self.critic(x) - probs = self.actor(x) - dist = Categorical(probs) - return dist, value \ No newline at end of file diff --git a/projects/codes/common/multiprocessing_env.py b/projects/codes/common/multiprocessing_env.py deleted file mode 100644 index 28c8aba..0000000 --- a/projects/codes/common/multiprocessing_env.py +++ /dev/null @@ -1,153 +0,0 @@ -# 该代码来自 openai baseline，用于多线程环境 -# https://github.com/openai/baselines/tree/master/baselines/common/vec_env - -import numpy as np -from multiprocessing import Process, Pipe - -def worker(remote, parent_remote, env_fn_wrapper): - parent_remote.close() - env = env_fn_wrapper.x() - while True: - cmd, data = remote.recv() - if cmd == 'step': - ob, reward, done, info = env.step(data) - if done: - ob = env.reset() - remote.send((ob, reward, done, info)) - elif cmd == 'reset': - ob = env.reset() - remote.send(ob) - elif cmd == 'reset_task': - ob = env.reset_task() - remote.send(ob) - elif cmd == 'close': - remote.close() - break - elif cmd == 'get_spaces': - remote.send((env.observation_space, env.action_space)) - else: - raise NotImplementedError - -class VecEnv(object): - """ - An abstract asynchronous, vectorized environment. - """ - def __init__(self, num_envs, observation_space, action_space): - self.num_envs = num_envs - self.observation_space = observation_space - self.action_space = action_space - - def reset(self): - """ - Reset all the environments and return an array of - observations, or a tuple of observation arrays. - If step_async is still doing work, that work will - be cancelled and step_wait() should not be called - until step_async() is invoked again. - """ - pass - - def step_async(self, actions): - """ - Tell all the environments to start taking a step - with the given actions. - Call step_wait() to get the results of the step. - You should not call this if a step_async run is - already pending. - """ - pass - - def step_wait(self): - """ - Wait for the step taken with step_async(). - Returns (obs, rews, dones, infos): - - obs: an array of observations, or a tuple of - arrays of observations. - - rews: an array of rewards - - dones: an array of "episode done" booleans - - infos: a sequence of info objects - """ - pass - - def close(self): - """ - Clean up the environments' resources. - """ - pass - - def step(self, actions): - self.step_async(actions) - return self.step_wait() - - -class CloudpickleWrapper(object): - """ - Uses cloudpickle to serialize contents (otherwise multiprocessing tries to use pickle) - """ - def __init__(self, x): - self.x = x - def __getstate__(self): - import cloudpickle - return cloudpickle.dumps(self.x) - def __setstate__(self, ob): - import pickle - self.x = pickle.loads(ob) - - -class SubprocVecEnv(VecEnv): - def __init__(self, env_fns, spaces=None): - """ - envs: list of gym environments to run in subprocesses - """ - self.waiting = False - self.closed = False - nenvs = len(env_fns) - self.nenvs = nenvs - self.remotes, self.work_remotes = zip(*[Pipe() for _ in range(nenvs)]) - self.ps = [Process(target=worker, args=(work_remote, remote, CloudpickleWrapper(env_fn))) - for (work_remote, remote, env_fn) in zip(self.work_remotes, self.remotes, env_fns)] - for p in self.ps: - p.daemon = True # if the main process crashes, we should not cause things to hang - p.start() - for remote in self.work_remotes: - remote.close() - - self.remotes[0].send(('get_spaces', None)) - observation_space, action_space = self.remotes[0].recv() - VecEnv.__init__(self, len(env_fns), observation_space, action_space) - - def step_async(self, actions): - for remote, action in zip(self.remotes, actions): - remote.send(('step', action)) - self.waiting = True - - def step_wait(self): - results = [remote.recv() for remote in self.remotes] - self.waiting = False - obs, rews, dones, infos = zip(*results) - return np.stack(obs), np.stack(rews), np.stack(dones), infos - - def reset(self): - for remote in self.remotes: - remote.send(('reset', None)) - return np.stack([remote.recv() for remote in self.remotes]) - - def reset_task(self): - for remote in self.remotes: - remote.send(('reset_task', None)) - return np.stack([remote.recv() for remote in self.remotes]) - - def close(self): - if self.closed: - return - if self.waiting: - for remote in self.remotes: - remote.recv() - for remote in self.remotes: - remote.send(('close', None)) - for p in self.ps: - p.join() - self.closed = True - - def __len__(self): - return self.nenvs \ No newline at end of file diff --git a/projects/codes/common/utils.py b/projects/codes/common/utils.py deleted file mode 100644 index 212ec5f..0000000 --- a/projects/codes/common/utils.py +++ /dev/null @@ -1,195 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: John -Email: johnjim0816@gmail.com -Date: 2021-03-12 16:02:24 -LastEditor: John -LastEditTime: 2022-11-14 10:27:43 -Discription: -Environment: -''' -import os -import numpy as np -from pathlib import Path -import matplotlib.pyplot as plt -import seaborn as sns -import yaml -import pandas as pd -from functools import wraps -from time import time -import logging -from pathlib import Path - - -from matplotlib.font_manager import FontProperties # 导入字体模块 - -def chinese_font(): - ''' 设置中文字体，注意需要根据自己电脑情况更改字体路径，否则还是默认的字体 - ''' - try: - font = FontProperties( - fname='/System/Library/Fonts/STHeiti Light.ttc', size=15) # fname系统字体路径，此处是mac的 - except: - font = None - return font - -def plot_rewards_cn(rewards, ma_rewards, cfg, tag='train'): - ''' 中文画图 - ''' - sns.set() - plt.figure() - plt.title(u"{}环境下{}算法的学习曲线".format(cfg.env_name, - cfg.algo_name), fontproperties=chinese_font()) - plt.xlabel(u'回合数', fontproperties=chinese_font()) - plt.plot(rewards) - plt.plot(ma_rewards) - plt.legend((u'奖励', u'滑动平均奖励',), loc="best", prop=chinese_font()) - if cfg.save: - plt.savefig(cfg.result_path+f"{tag}_rewards_curve_cn") - # plt.show() -def smooth(data, weight=0.9): - '''用于平滑曲线，类似于Tensorboard中的smooth - - Args: - data (List):输入数据 - weight (Float): 平滑权重，处于0-1之间，数值越高说明越平滑，一般取0.9 - - Returns: - smoothed (List): 平滑后的数据 - ''' - last = data[0] # First value in the plot (first timestep) - smoothed = list() - for point in data: - smoothed_val = last * weight + (1 - weight) * point # 计算平滑值 - smoothed.append(smoothed_val) - last = smoothed_val - return smoothed - -def plot_rewards(rewards,title="learning curve",fpath=None,save_fig=True,show_fig=False): - sns.set() - plt.figure() # 创建一个图形实例，方便同时多画几个图 - plt.title(f"{title}") - plt.xlabel('epsiodes') - plt.plot(rewards, label='rewards') - plt.plot(smooth(rewards), label='smoothed') - plt.legend() - if save_fig: - plt.savefig(f"{fpath}/learning_curve.png") - if show_fig: - plt.show() - -def plot_losses(losses, algo="DQN", save=True, path='./'): - sns.set() - plt.figure() - plt.title("loss curve of {}".format(algo)) - plt.xlabel('epsiodes') - plt.plot(losses, label='rewards') - plt.legend() - if save: - plt.savefig(path+"losses_curve") - plt.show() - -def save_results(res_dic,fpath = None): - ''' save results - ''' - Path(fpath).mkdir(parents=True, exist_ok=True) - df = pd.DataFrame(res_dic) - df.to_csv(f"{fpath}/res.csv",index=None) -def merge_class_attrs(ob1, ob2): - ob1.__dict__.update(ob2.__dict__) - return ob1 -def get_logger(fpath): - Path(fpath).mkdir(parents=True, exist_ok=True) - logger = logging.getLogger(name='r') # set root logger if not set name - logger.setLevel(logging.DEBUG) - formatter = logging.Formatter( - '%(asctime)s - %(name)s - %(levelname)s: - %(message)s', - datefmt='%Y-%m-%d %H:%M:%S') - # output to file by using FileHandler - fh = logging.FileHandler(fpath+"log.txt") - fh.setLevel(logging.DEBUG) - fh.setFormatter(formatter) - # output to screen by using StreamHandler - ch = logging.StreamHandler() - ch.setLevel(logging.DEBUG) - ch.setFormatter(formatter) - # add Handler - logger.addHandler(ch) - logger.addHandler(fh) - return logger -def save_cfgs(cfgs, fpath): - ''' save config - ''' - Path(fpath).mkdir(parents=True, exist_ok=True) - - with open(f"{fpath}/config.yaml", 'w') as f: - for cfg_type in cfgs: - yaml.dump({cfg_type: cfgs[cfg_type].__dict__}, f, default_flow_style=False) -def load_cfgs(cfgs, fpath): - with open(fpath) as f: - load_cfg = yaml.load(f,Loader=yaml.FullLoader) - for cfg_type in cfgs: - for k, v in load_cfg[cfg_type].items(): - setattr(cfgs[cfg_type], k, v) -# def del_empty_dir(*paths): -# ''' 删除目录下所有空文件夹 -# ''' -# for path in paths: -# dirs = os.listdir(path) -# for dir in dirs: -# if not os.listdir(os.path.join(path, dir)): -# os.removedirs(os.path.join(path, dir)) - -# class NpEncoder(json.JSONEncoder): -# def default(self, obj): -# if isinstance(obj, np.integer): -# return int(obj) -# if isinstance(obj, np.floating): -# return float(obj) -# if isinstance(obj, np.ndarray): -# return obj.tolist() -# return json.JSONEncoder.default(self, obj) - -# def save_args(args,path=None): -# # save parameters -# Path(path).mkdir(parents=True, exist_ok=True) -# with open(f"{path}/params.json", 'w') as fp: -# json.dump(args, fp,cls=NpEncoder) -# print("Parameters saved!") - - -def timing(func): - ''' a decorator to print the running time of a function - ''' - @wraps(func) - def wrap(*args, **kw): - ts = time() - result = func(*args, **kw) - te = time() - print(f"func: {func.__name__}, took: {te-ts:2.4f} seconds") - return result - return wrap -def all_seed(env,seed = 1): - ''' omnipotent seed for RL, attention the position of seed function, you'd better put it just following the env create function - Args: - env (_type_): - seed (int, optional): _description_. Defaults to 1. - ''' - import torch - import numpy as np - import random - if seed == 0: - return - # print(f"seed = {seed}") - env.seed(seed) # env config - np.random.seed(seed) - random.seed(seed) - torch.manual_seed(seed) # config for CPU - torch.cuda.manual_seed(seed) # config for GPU - os.environ['PYTHONHASHSEED'] = str(seed) # config for python scripts - # config for cudnn - torch.backends.cudnn.deterministic = True - torch.backends.cudnn.benchmark = False - torch.backends.cudnn.enabled = False - \ No newline at end of file diff --git a/projects/codes/common/wrappers.py b/projects/codes/common/wrappers.py deleted file mode 100644 index 4793b36..0000000 --- a/projects/codes/common/wrappers.py +++ /dev/null @@ -1,29 +0,0 @@ -import gym - -class TimeLimit(gym.Wrapper): - def __init__(self, env, max_episode_steps=None): - super(TimeLimit, self).__init__(env) - self._max_episode_steps = max_episode_steps - self._elapsed_steps = 0 - - def step(self, ac): - observation, reward, done, info = self.env.step(ac) - self._elapsed_steps += 1 - if self._elapsed_steps >= self._max_episode_steps: - done = True - info['TimeLimit.truncated'] = True - return observation, reward, done, info - - def reset(self, **kwargs): - self._elapsed_steps = 0 - return self.env.reset(**kwargs) - -class ClipActionsWrapper(gym.Wrapper): - def step(self, action): - import numpy as np - action = np.nan_to_num(action) - action = np.clip(action, self.action_space.low, self.action_space.high) - return self.env.step(action) - - def reset(self, **kwargs): - return self.env.reset(**kwargs) \ No newline at end of file diff --git a/projects/codes/envs/README.md b/projects/codes/envs/README.md deleted file mode 100644 index d30725b..0000000 --- a/projects/codes/envs/README.md +++ /dev/null @@ -1,18 +0,0 @@ -# 环境说明汇总 - -## 算法SAR一览 - -说明：SAR分别指状态(S)、动作(A)以及奖励(R)，下表的Reward Range表示每回合能获得的奖励范围，Steps表示环境中每回合的最大步数 - -| Environment ID | Observation Space | Action Space | Reward Range | Steps | -| :--------------------------------: | :---------------: | :----------: | :----------: | :------: | -| CartPole-v0 | Box(4,) | Discrete(2) | [0,200] | 200 | -| CartPole-v1 | Box(4,) | Discrete(2) | [0,500] | 500 | -| CliffWalking-v0 | Discrete(48) | Discrete(4) | [-inf,-13] | [13,inf] | -| FrozenLake-v1(*is_slippery*=False) | Discrete(16) | Discrete(4) | 0 or 1 | [6,info] | - -## 环境描述 - -[OpenAI Gym](./gym_info.md) -[MuJoCo](./mujoco_info.md) - diff --git a/projects/codes/envs/assets/action_grid.png b/projects/codes/envs/assets/action_grid.png deleted file mode 100644 index 7759f8b..0000000 Binary files a/projects/codes/envs/assets/action_grid.png and /dev/null differ diff --git a/projects/codes/envs/assets/gym_info_20211130180023.png b/projects/codes/envs/assets/gym_info_20211130180023.png deleted file mode 100644 index 723b67f..0000000 Binary files a/projects/codes/envs/assets/gym_info_20211130180023.png and /dev/null differ diff --git a/projects/codes/envs/assets/image-20200820174307301.png b/projects/codes/envs/assets/image-20200820174307301.png deleted file mode 100644 index 1197da0..0000000 Binary files a/projects/codes/envs/assets/image-20200820174307301.png and /dev/null differ diff --git a/projects/codes/envs/assets/image-20200820174814084.png b/projects/codes/envs/assets/image-20200820174814084.png deleted file mode 100644 index 4c9e3dc..0000000 Binary files a/projects/codes/envs/assets/image-20200820174814084.png and /dev/null differ diff --git a/projects/codes/envs/assets/image-20201007211441036.png b/projects/codes/envs/assets/image-20201007211441036.png deleted file mode 100644 index ae5b0f8..0000000 Binary files a/projects/codes/envs/assets/image-20201007211441036.png and /dev/null differ diff --git a/projects/codes/envs/assets/image-20201007211858925.png b/projects/codes/envs/assets/image-20201007211858925.png deleted file mode 100644 index 0bbb5b2..0000000 Binary files a/projects/codes/envs/assets/image-20201007211858925.png and /dev/null differ diff --git a/projects/codes/envs/assets/image-20210429150622353.png b/projects/codes/envs/assets/image-20210429150622353.png deleted file mode 100644 index 1216b4c..0000000 Binary files a/projects/codes/envs/assets/image-20210429150622353.png and /dev/null differ diff --git a/projects/codes/envs/assets/image-20210429150630806.png b/projects/codes/envs/assets/image-20210429150630806.png deleted file mode 100644 index 45107d5..0000000 Binary files a/projects/codes/envs/assets/image-20210429150630806.png and /dev/null differ diff --git a/projects/codes/envs/assets/track_big.png b/projects/codes/envs/assets/track_big.png deleted file mode 100644 index f7b3dc1..0000000 Binary files a/projects/codes/envs/assets/track_big.png and /dev/null differ diff --git a/projects/codes/envs/blackjack.py b/projects/codes/envs/blackjack.py deleted file mode 100644 index 87f02d2..0000000 --- a/projects/codes/envs/blackjack.py +++ /dev/null @@ -1,122 +0,0 @@ -import gym -from gym import spaces -from gym.utils import seeding - -def cmp(a, b): - return int((a > b)) - int((a < b)) - -# 1 = Ace, 2-10 = Number cards, Jack/Queen/King = 10 -deck = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 10, 10, 10] - - -def draw_card(np_random): - return np_random.choice(deck) - - -def draw_hand(np_random): - return [draw_card(np_random), draw_card(np_random)] - - -def usable_ace(hand): # Does this hand have a usable ace? - return 1 in hand and sum(hand) + 10 <= 21 - - -def sum_hand(hand): # Return current hand total - if usable_ace(hand): - return sum(hand) + 10 - return sum(hand) - - -def is_bust(hand): # Is this hand a bust? - return sum_hand(hand) > 21 - - -def score(hand): # What is the score of this hand (0 if bust) - return 0 if is_bust(hand) else sum_hand(hand) - - -def is_natural(hand): # Is this hand a natural blackjack? - return sorted(hand) == [1, 10] - - -class BlackjackEnv(gym.Env): - """Simple blackjack environment - Blackjack is a card game where the goal is to obtain cards that sum to as - near as possible to 21 without going over. They're playing against a fixed - dealer. - Face cards (Jack, Queen, King) have point value 10. - Aces can either count as 11 or 1, and it's called 'usable' at 11. - This game is placed with an infinite deck (or with replacement). - The game starts with each (player and dealer) having one face up and one - face down card. - The player can request additional cards (hit=1) until they decide to stop - (stick=0) or exceed 21 (bust). - After the player sticks, the dealer reveals their facedown card, and draws - until their sum is 17 or greater. If the dealer goes bust the player wins. - If neither player nor dealer busts, the outcome (win, lose, draw) is - decided by whose sum is closer to 21. The reward for winning is +1, - drawing is 0, and losing is -1. - The observation of a 3-tuple of: the players current sum, - the dealer's one showing card (1-10 where 1 is ace), - and whether or not the player holds a usable ace (0 or 1). - This environment corresponds to the version of the blackjack problem - described in Example 5.1 in Reinforcement Learning: An Introduction - by Sutton and Barto (1998). - https://webdocs.cs.ualberta.ca/~sutton/book/the-book.html - """ - def __init__(self, natural=False): - self.action_space = spaces.Discrete(2) - self.observation_space = spaces.Tuple(( - spaces.Discrete(32), - spaces.Discrete(11), - spaces.Discrete(2))) - self._seed() - - # Flag to payout 1.5 on a "natural" blackjack win, like casino rules - # Ref: http://www.bicyclecards.com/how-to-play/blackjack/ - self.natural = natural - # Start the first game - self._reset() # Number of - self.n_actions = 2 - - def reset(self): - return self._reset() - - def step(self, action): - return self._step(action) - - def _seed(self, seed=None): - self.np_random, seed = seeding.np_random(seed) - return [seed] - - def _step(self, action): - assert self.action_space.contains(action) - if action: # hit: add a card to players hand and return - self.player.append(draw_card(self.np_random)) - if is_bust(self.player): - done = True - reward = -1 - else: - done = False - reward = 0 - else: # stick: play out the dealers hand, and score - done = True - while sum_hand(self.dealer) < 17: - self.dealer.append(draw_card(self.np_random)) - reward = cmp(score(self.player), score(self.dealer)) - if self.natural and is_natural(self.player) and reward == 1: - reward = 1.5 - return self._get_obs(), reward, done, {} - - def _get_obs(self): - return (sum_hand(self.player), self.dealer[0], usable_ace(self.player)) - - def _reset(self): - self.dealer = draw_hand(self.np_random) - self.player = draw_hand(self.np_random) - - # Auto-draw another card if the score is less than 12 - while sum_hand(self.player) < 12: - self.player.append(draw_card(self.np_random)) - - return self._get_obs() diff --git a/projects/codes/envs/cliff_walking.py b/projects/codes/envs/cliff_walking.py deleted file mode 100644 index 05b9b2e..0000000 --- a/projects/codes/envs/cliff_walking.py +++ /dev/null @@ -1,84 +0,0 @@ -import numpy as np -import sys -from gym.envs.toy_text import discrete - - -UP = 0 -RIGHT = 1 -DOWN = 2 -LEFT = 3 - -class CliffWalkingEnv(discrete.DiscreteEnv): - - metadata = {'render.modes': ['human', 'ansi']} - - def _limit_coordinates(self, coord): - coord[0] = min(coord[0], self.shape[0] - 1) - coord[0] = max(coord[0], 0) - coord[1] = min(coord[1], self.shape[1] - 1) - coord[1] = max(coord[1], 0) - return coord - - def _calculate_transition_prob(self, current, delta): - new_position = np.array(current) + np.array(delta) - new_position = self._limit_coordinates(new_position).astype(int) - new_state = np.ravel_multi_index(tuple(new_position), self.shape) - reward = -100.0 if self._cliff[tuple(new_position)] else -1.0 - is_done = self._cliff[tuple(new_position)] or (tuple(new_position) == (3,11)) - return [(1.0, new_state, reward, is_done)] - - def __init__(self): - self.shape = (4, 12) - - nS = np.prod(self.shape) - n_actions = 4 - - # Cliff Location - self._cliff = np.zeros(self.shape, dtype=np.bool) - self._cliff[3, 1:-1] = True - - # Calculate transition probabilities - P = {} - for s in range(nS): - position = np.unravel_index(s, self.shape) - P[s] = { a : [] for a in range(n_actions) } - P[s][UP] = self._calculate_transition_prob(position, [-1, 0]) - P[s][RIGHT] = self._calculate_transition_prob(position, [0, 1]) - P[s][DOWN] = self._calculate_transition_prob(position, [1, 0]) - P[s][LEFT] = self._calculate_transition_prob(position, [0, -1]) - - # We always start in state (3, 0) - isd = np.zeros(nS) - isd[np.ravel_multi_index((3,0), self.shape)] = 1.0 - - super(CliffWalkingEnv, self).__init__(nS, n_actions, P, isd) - - def render(self, mode='human', close=False): - self._render(mode, close) - - def _render(self, mode='human', close=False): - if close: - return - - outfile = StringIO() if mode == 'ansi' else sys.stdout - - for s in range(self.nS): - position = np.unravel_index(s, self.shape) - # print(self.s) - if self.s == s: - output = " x " - elif position == (3,11): - output = " T " - elif self._cliff[position]: - output = " C " - else: - output = " o " - - if position[1] == 0: - output = output.lstrip() - if position[1] == self.shape[1] - 1: - output = output.rstrip() - output += "\n" - - outfile.write(output) - outfile.write("\n") diff --git a/projects/codes/envs/gridworld.py b/projects/codes/envs/gridworld.py deleted file mode 100644 index cf3aec2..0000000 --- a/projects/codes/envs/gridworld.py +++ /dev/null @@ -1,125 +0,0 @@ -import io -import numpy as np -import sys -from gym.envs.toy_text import discrete - -UP = 0 -RIGHT = 1 -DOWN = 2 -LEFT = 3 - -class GridworldEnv(discrete.DiscreteEnv): - """ - Grid World environment from Sutton's Reinforcement Learning book chapter 4. - You are an agent on an MxN grid and your goal is to reach the terminal - state at the top left or the bottom right corner. - - For example, a 4x4 grid looks as follows: - - T o o o - o x o o - o o o o - o o o T - - x is your position and T are the two terminal states. - - You can take actions in each direction (UP=0, RIGHT=1, DOWN=2, LEFT=3). - Actions going off the edge leave you in your current state. - You receive a reward of -1 at each step until you reach a terminal state. - """ - - metadata = {'render.modes': ['human', 'ansi']} - - def __init__(self, shape=[4,4]): - if not isinstance(shape, (list, tuple)) or not len(shape) == 2: - raise ValueError('shape argument must be a list/tuple of length 2') - - self.shape = shape - - nS = np.prod(shape) - n_actions = 4 - - MAX_Y = shape[0] - MAX_X = shape[1] - - P = {} - grid = np.arange(nS).reshape(shape) - it = np.nditer(grid, flags=['multi_index']) - - while not it.finished: - s = it.iterindex - y, x = it.multi_index - - # P[s][a] = (prob, next_state, reward, is_done) - P[s] = {a : [] for a in range(n_actions)} - - is_done = lambda s: s == 0 or s == (nS - 1) - reward = 0.0 if is_done(s) else -1.0 - - # We're stuck in a terminal state - if is_done(s): - P[s][UP] = [(1.0, s, reward, True)] - P[s][RIGHT] = [(1.0, s, reward, True)] - P[s][DOWN] = [(1.0, s, reward, True)] - P[s][LEFT] = [(1.0, s, reward, True)] - # Not a terminal state - else: - ns_up = s if y == 0 else s - MAX_X - ns_right = s if x == (MAX_X - 1) else s + 1 - ns_down = s if y == (MAX_Y - 1) else s + MAX_X - ns_left = s if x == 0 else s - 1 - P[s][UP] = [(1.0, ns_up, reward, is_done(ns_up))] - P[s][RIGHT] = [(1.0, ns_right, reward, is_done(ns_right))] - P[s][DOWN] = [(1.0, ns_down, reward, is_done(ns_down))] - P[s][LEFT] = [(1.0, ns_left, reward, is_done(ns_left))] - - it.iternext() - - # Initial state distribution is uniform - isd = np.ones(nS) / nS - - # We expose the model of the environment for educational purposes - # This should not be used in any model-free learning algorithm - self.P = P - - super(GridworldEnv, self).__init__(nS, n_actions, P, isd) - - def _render(self, mode='human', close=False): - """ Renders the current gridworld layout - - For example, a 4x4 grid with the mode="human" looks like: - T o o o - o x o o - o o o o - o o o T - where x is your position and T are the two terminal states. - """ - if close: - return - - outfile = io.StringIO() if mode == 'ansi' else sys.stdout - - grid = np.arange(self.nS).reshape(self.shape) - it = np.nditer(grid, flags=['multi_index']) - while not it.finished: - s = it.iterindex - y, x = it.multi_index - - if self.s == s: - output = " x " - elif s == 0 or s == self.nS - 1: - output = " T " - else: - output = " o " - - if x == 0: - output = output.lstrip() - if x == self.shape[1] - 1: - output = output.rstrip() - - outfile.write(output) - - if x == self.shape[1] - 1: - outfile.write("\n") - - it.iternext() diff --git a/projects/codes/envs/gridworld_env.py b/projects/codes/envs/gridworld_env.py deleted file mode 100644 index 9d0724a..0000000 --- a/projects/codes/envs/gridworld_env.py +++ /dev/null @@ -1,100 +0,0 @@ -import gym -import turtle -import numpy as np - -# turtle tutorial : https://docs.python.org/3.3/library/turtle.html - -def GridWorld(gridmap=None, is_slippery=False): - if gridmap is None: - gridmap = ['SFFF', 'FHFH', 'FFFH', 'HFFG'] - env = gym.make("FrozenLake-v0", desc=gridmap, is_slippery=False) - env = FrozenLakeWapper(env) - return env - - -class FrozenLakeWapper(gym.Wrapper): - def __init__(self, env): - gym.Wrapper.__init__(self, env) - self.max_y = env.desc.shape[0] - self.max_x = env.desc.shape[1] - self.t = None - self.unit = 50 - - def draw_box(self, x, y, fillcolor='', line_color='gray'): - self.t.up() - self.t.goto(x * self.unit, y * self.unit) - self.t.color(line_color) - self.t.fillcolor(fillcolor) - self.t.setheading(90) - self.t.down() - self.t.begin_fill() - for _ in range(4): - self.t.forward(self.unit) - self.t.right(90) - self.t.end_fill() - - def move_player(self, x, y): - self.t.up() - self.t.setheading(90) - self.t.fillcolor('red') - self.t.goto((x + 0.5) * self.unit, (y + 0.5) * self.unit) - - def render(self): - if self.t == None: - self.t = turtle.Turtle() - self.wn = turtle.Screen() - self.wn.setup(self.unit * self.max_x + 100, - self.unit * self.max_y + 100) - self.wn.setworldcoordinates(0, 0, self.unit * self.max_x, - self.unit * self.max_y) - self.t.shape('circle') - self.t.width(2) - self.t.speed(0) - self.t.color('gray') - for i in range(self.desc.shape[0]): - for j in range(self.desc.shape[1]): - x = j - y = self.max_y - 1 - i - if self.desc[i][j] == b'S': # Start - self.draw_box(x, y, 'white') - elif self.desc[i][j] == b'F': # Frozen ice - self.draw_box(x, y, 'white') - elif self.desc[i][j] == b'G': # Goal - self.draw_box(x, y, 'yellow') - elif self.desc[i][j] == b'H': # Hole - self.draw_box(x, y, 'black') - else: - self.draw_box(x, y, 'white') - self.t.shape('turtle') - - x_pos = self.s % self.max_x - y_pos = self.max_y - 1 - int(self.s / self.max_x) - self.move_player(x_pos, y_pos) - - - -if __name__ == '__main__': - # 环境1：FrozenLake, 可以配置冰面是否是滑的 - # 0 left, 1 down, 2 right, 3 up - env = gym.make("FrozenLake-v0", is_slippery=False) - env = FrozenLakeWapper(env) - - # 环境2：CliffWalking, 悬崖环境 - # env = gym.make("CliffWalking-v0") # 0 up, 1 right, 2 down, 3 left - # env = CliffWalkingWapper(env) - - # 环境3：自定义格子世界，可以配置地图, S为出发点Start, F为平地Floor, H为洞Hole, G为出口目标Goal - # gridmap = [ - # 'SFFF', - # 'FHFF', - # 'FFFF', - # 'HFGF' ] - # env = GridWorld(gridmap) - - env.reset() - for step in range(10): - action = np.random.randint(0, 4) - obs, reward, done, info = env.step(action) - print('step {}: action {}, obs {}, reward {}, done {}, info {}'.format(\ - step, action, obs, reward, done, info)) - # env.render() # 渲染一帧图像 \ No newline at end of file diff --git a/projects/codes/envs/gym_info.md b/projects/codes/envs/gym_info.md deleted file mode 100644 index 49da18f..0000000 --- a/projects/codes/envs/gym_info.md +++ /dev/null @@ -1,50 +0,0 @@ -# OpenAi Gym 环境说明 -## 基础控制 - -### [CartPole v0](https://github.com/openai/gym/wiki/CartPole-v0) - -image-20200820174307301 - -通过向左或向右推车能够实现平衡，所以动作空间由两个动作组成。每进行一个step就会给一个reward，如果无法保持平衡那么done等于true，本次episode失败。理想状态下，每个episode至少能进行200个step，也就是说每个episode的reward总和至少为200，step数目至少为200 - -### CartPole-v1 - -```CartPole v1```环境其实跟```CartPole v0```是一模一样的，区别在于每回合最大步数（max_episode_steps）以及奖励阈值（reward_threshold），如下是相关源码： - -![](assets/gym_info_20211130180023.png) - -这里先解释一下奖励阈值（reward_threshold），即Gym设置的一个合格标准，比如对于```CartPole v0```如果算法能够将奖励收敛到195以上，说明该算法合格。但实际上```CartPole v0```的每回合最大步数（max_episode_steps）是200，每步的奖励最大是1，也就是每回合最大奖励是200，比Gym设置的奖励阈值高。笔者猜测这是Gym可能是给算法学习者们设置的一个参考线，而实际中在写算法时并不会用到这个算法阈值，所以可以忽略。 - -再看每回合最大步数，可以看到```CartPole v1```的步数更长，相应的奖励要求更高，可以理解为```v1```是```v0```的难度升级版。 - - -### [Pendulum-v0](https://github.com/openai/gym/wiki/Pendulum-v0) - -注：gym 0.18.0之后版本中Pendulum-v0已经改为Pendulum-v1 -image-20200820174814084 - -钟摆以随机位置开始，目标是将其摆动，使其保持向上直立。动作空间是连续的，值的区间为[-2,2]。每个step给的reward最低为-16.27，最高为0。目前最好的成绩是100个episode的reward之和为-123.11 ± 6.86。 - -### - -悬崖寻路问题（CliffWalking）是指在一个4 x 12的网格中，智能体以网格的左下角位置为起点，以网格的下角位置为终点，目标是移动智能体到达终点位置，智能体每次可以在上、下、左、右这4个方向中移动一步，每移动一步会得到-1单位的奖励。 - -image-20201007211441036 - -如图，红色部分表示悬崖，数字代表智能体能够观测到的位置信息，即observation，总共会有0-47等48个不同的值，智能体再移动中会有以下限制： - -* 智能体不能移出网格，如果智能体想执行某个动作移出网格，那么这一步智能体不会移动，但是这个操作依然会得到-1单位的奖励 - -* 如果智能体“掉入悬崖” ，会立即回到起点位置，并得到-100单位的奖励 - -* 当智能体移动到终点时，该回合结束，该回合总奖励为各步奖励之和 - -实际的仿真界面如下： - -image-20201007211858925 - -由于从起点到终点最少需要13步，每步得到-1的reward，因此最佳训练算法下，每个episode下reward总和应该为-13。 - -## 参考 - -[Gym环境相关源码](https://github.com/openai/gym/tree/master/gym/envs) \ No newline at end of file diff --git a/projects/codes/envs/mujoco_info.md b/projects/codes/envs/mujoco_info.md deleted file mode 100644 index aaa8cbb..0000000 --- a/projects/codes/envs/mujoco_info.md +++ /dev/null @@ -1,42 +0,0 @@ -# MuJoCo - -MuJoCo（Multi-Joint dynamics with Contact）是一个物理模拟器，可以用于机器人控制优化等研究。安装见[Mac安装MuJoCo以及mujoco_py](https://blog.csdn.net/JohnJim0/article/details/115656392?spm=1001.2014.3001.5501) - - - -## HalfCheetah-v2 - - - -该环境基于mujoco仿真引擎，该环境的目的是使一只两只脚的“猎豹”跑得越快越好(下面图谷歌HalfCheetah-v2的，https://gym.openai.com/envs/HalfCheetah-v2/)。 - -image-20210429150630806 - -动作空间：Box(6,)，一只脚需要控制三个关节一共6个关节，每个关节的运动范围为[-1, 1]。 - -状态空间：Box(17, )，包含各种状态，每个值的范围为![img](assets/9cd6ae68c9aad008ede4139da358ec26.svg)，主要描述“猎豹”本身的姿态等信息。 - -回报定义：每一步的回报与这一步的中猎豹的速度和猎豹行动的消耗有关，定义回报的代码如下。 - -```python -def step(self, action): - xposbefore = self.sim.data.qpos[0] - self.do_simulation(action, self.frame_skip) - xposafter = self.sim.data.qpos[0] - ob = self._get_obs() - reward_ctrl = - 0.1 * np.square(action).sum() - reward_run = (xposafter - xposbefore)/self.dt - # =========== reward =========== - reward = reward_ctrl + reward_run - # =========== reward =========== - done = False - return ob, reward, done, dict(reward_run=reward_run, reward_ctrl=reward_ctrl) -``` - -当猎豹无法控制平衡而倒下时，一个回合(episode)结束。 - -但是这个环境有一些问题，目前经过搜索并不知道一个回合的reward上限，实验中训练好的episode能跑出平台之外： - -image-20210429150622353 - -加上时间有限，所以训练中reward一直处于一个平缓上升的状态，本人猜测这可能是gym的一个bug。 \ No newline at end of file diff --git a/projects/codes/envs/racetrack.py b/projects/codes/envs/racetrack.py deleted file mode 100644 index 69836d5..0000000 --- a/projects/codes/envs/racetrack.py +++ /dev/null @@ -1,242 +0,0 @@ -import time -import random -import numpy as np -import os -import matplotlib.pyplot as plt -import matplotlib.patheffects as pe -from IPython.display import clear_output -from gym.spaces import Discrete,Box -from matplotlib import colors -import gym - -class RacetrackEnv(gym.Env) : - """ - Class representing a race-track environment inspired by exercise 5.12 in Sutton & Barto 2018 (p.111). - Please do not make changes to this class - it will be overwritten with a clean version when it comes to marking. - - The dynamics of this environment are detailed in this coursework exercise's jupyter notebook, although I have - included rather verbose comments here for those of you who are interested in how the environment has been - implemented (though this should not impact your solution code).ss - """ - - ACTIONS_DICT = { - 0 : (1, -1), # Acc Vert., Brake Horiz. - 1 : (1, 0), # Acc Vert., Hold Horiz. - 2 : (1, 1), # Acc Vert., Acc Horiz. - 3 : (0, -1), # Hold Vert., Brake Horiz. - 4 : (0, 0), # Hold Vert., Hold Horiz. - 5 : (0, 1), # Hold Vert., Acc Horiz. - 6 : (-1, -1), # Brake Vert., Brake Horiz. - 7 : (-1, 0), # Brake Vert., Hold Horiz. - 8 : (-1, 1) # Brake Vert., Acc Horiz. - } - - - CELL_TYPES_DICT = { - 0 : "track", - 1 : "wall", - 2 : "start", - 3 : "goal" - } - - - def __init__(self) : - # Load racetrack map from file. - self.track = np.flip(np.loadtxt(os.path.dirname(__file__)+"/track.txt", dtype = int), axis = 0) - - - # Discover start grid squares. - self.initial_states = [] - for y in range(self.track.shape[0]) : - for x in range(self.track.shape[1]) : - if (self.CELL_TYPES_DICT[self.track[y, x]] == "start") : - self.initial_states.append((y, x)) - high= np.array([np.finfo(np.float32).max, np.finfo(np.float32).max, np.finfo(np.float32).max, np.finfo(np.float32).max]) - self.observation_space = Box(low=-high, high=high, shape=(4,), dtype=np.float32) - self.action_space = Discrete(9) - self.is_reset = False - - def step(self, action : int) : - """ - Takes a given action in the environment's current state, and returns a next state, - reward, and whether the next state is done or not. - - Arguments: - action {int} -- The action to take in the environment's current state. Should be an integer in the range [0-8]. - - Raises: - RuntimeError: Raised when the environment needs resetting.\n - TypeError: Raised when an action of an invalid type is given.\n - ValueError: Raised when an action outside the range [0-8] is given.\n - - Returns: - A tuple of:\n - {(int, int, int, int)} -- The next state, a tuple of (y_pos, x_pos, y_velocity, x_velocity).\n - {int} -- The reward earned by taking the given action in the current environment state.\n - {bool} -- Whether the environment's next state is done or not.\n - - """ - - # Check whether a reset is needed. - if (not self.is_reset) : - raise RuntimeError(".step() has been called when .reset() is needed.\n" + - "You need to call .reset() before using .step() for the first time, and after an episode ends.\n" + - ".reset() initialises the environment at the start of an episode, then returns an initial state.") - - # Check that action is the correct type (either a python integer or a numpy integer). - if (not (isinstance(action, int) or isinstance(action, np.integer))) : - raise TypeError("action should be an integer.\n" + - "action value {} of type {} was supplied.".format(action, type(action))) - - # Check that action is an allowed value. - if (action < 0 or action > 8) : - raise ValueError("action must be an integer in the range [0-8] corresponding to one of the legal actions.\n" + - "action value {} was supplied.".format(action)) - - - # Update Velocity. - # With probability, 0.85 update velocity components as intended. - if (np.random.uniform() < 0.8) : - (d_y, d_x) = self.ACTIONS_DICT[action] - # With probability, 0.15 Do not change velocity components. - else : - (d_y, d_x) = (0, 0) - - self.velocity = (self.velocity[0] + d_y, self.velocity[1] + d_x) - - # Keep velocity within bounds (-10, 10). - if (self.velocity[0] > 10) : - self.velocity[0] = 10 - elif (self.velocity[0] < -10) : - self.velocity[0] = -10 - if (self.velocity[1] > 10) : - self.velocity[1] = 10 - elif (self.velocity[1] < -10) : - self.velocity[1] = -10 - - # Update Position. - new_position = (self.position[0] + self.velocity[0], self.position[1] + self.velocity[1]) - - reward = 0 - done = False - - # If position is out-of-bounds, return to start and set velocity components to zero. - if (new_position[0] < 0 or new_position[1] < 0 or new_position[0] >= self.track.shape[0] or new_position[1] >= self.track.shape[1]) : - self.position = random.choice(self.initial_states) - self.velocity = (0, 0) - reward -= 10 - # If position is in a wall grid-square, return to start and set velocity components to zero. - elif (self.CELL_TYPES_DICT[self.track[new_position]] == "wall") : - self.position = random.choice(self.initial_states) - self.velocity = (0, 0) - reward -= 10 - # If position is in a track grid-squre or a start-square, update position. - elif (self.CELL_TYPES_DICT[self.track[new_position]] in ["track", "start"]) : - self.position = new_position - # If position is in a goal grid-square, end episode. - elif (self.CELL_TYPES_DICT[self.track[new_position]] == "goal") : - self.position = new_position - reward += 10 - done = True - # If this gets reached, then the student has touched something they shouldn't have. Naughty! - else : - raise RuntimeError("You've met with a terrible fate, haven't you?\nDon't modify things you shouldn't!") - - # Penalise every timestep. - reward -= 1 - - # Require a reset if the current state is done. - if (done) : - self.is_reset = False - - # Return next state, reward, and whether the episode has ended. - return np.array([self.position[0], self.position[1], self.velocity[0], self.velocity[1]]), reward, done,{} - - - def reset(self) : - """ - Resets the environment, ready for a new episode to begin, then returns an initial state. - The initial state will be a starting grid square randomly chosen using a uniform distribution, - with both components of the velocity being zero. - - Returns: - {(int, int, int, int)} -- an initial state, a tuple of (y_pos, x_pos, y_velocity, x_velocity). - """ - - # Pick random starting grid-square. - self.position = random.choice(self.initial_states) - - # Set both velocity components to zero. - self.velocity = (0, 0) - - self.is_reset = True - - return np.array([self.position[0], self.position[1], self.velocity[0], self.velocity[1]]) - - - def render(self, mode = 'human') : - """ - Renders a pretty matplotlib plot representing the current state of the environment. - Calling this method on subsequent timesteps will update the plot. - This is VERY VERY SLOW and wil slow down training a lot. Only use for debugging/testing. - - Arguments: - sleep_time {float} -- How many seconds (or partial seconds) you want to wait on this rendered frame. - - """ - # Turn interactive mode on. - plt.ion() - fig = plt.figure(num = "env_render") - ax = plt.gca() - ax.clear() - clear_output(wait = True) - - # Prepare the environment plot and mark the car's position. - env_plot = np.copy(self.track) - env_plot[self.position] = 4 - env_plot = np.flip(env_plot, axis = 0) - - # Plot the gridworld. - cmap = colors.ListedColormap(["white", "black", "green", "red", "yellow"]) - bounds = list(range(6)) - norm = colors.BoundaryNorm(bounds, cmap.N) - ax.imshow(env_plot, cmap = cmap, norm = norm, zorder = 0) - - # Plot the velocity. - if (not self.velocity == (0, 0)) : - ax.arrow(self.position[1], self.track.shape[0] - 1 - self.position[0], self.velocity[1], -self.velocity[0], - path_effects=[pe.Stroke(linewidth=1, foreground='black')], color = "yellow", width = 0.1, length_includes_head = True, zorder = 2) - - # Set up axes. - ax.grid(which = 'major', axis = 'both', linestyle = '-', color = 'k', linewidth = 2, zorder = 1) - ax.set_xticks(np.arange(-0.5, self.track.shape[1] , 1)); - ax.set_xticklabels([]) - ax.set_yticks(np.arange(-0.5, self.track.shape[0], 1)); - ax.set_yticklabels([]) - - # Draw everything. - #fig.canvas.draw() - #fig.canvas.flush_events() - plt.show() - # time sleep - time.sleep(0.1) - - def get_actions(self) : - """ - Returns the available actions in the current state - will always be a list - of integers in the range [0-8]. - """ - return [*self.ACTIONS_DICT] -if __name__ == "__main__": - num_steps = 1000000 - env = RacetrackEnv() - state = env.reset() - print(state) - for _ in range(num_steps) : - - next_state, reward, done,_ = env.step(random.choice(env.get_actions())) - print(next_state) - env.render() - - if (done) : - _ = env.reset() diff --git a/projects/codes/envs/racetrack_env.md b/projects/codes/envs/racetrack_env.md deleted file mode 100644 index c5e2d7f..0000000 --- a/projects/codes/envs/racetrack_env.md +++ /dev/null @@ -1,37 +0,0 @@ -## The Racetrack Environment -We have implemented a custom environment called "Racetrack" for you to use during this piece of coursework. It is inspired by the environment described in the course textbook (Reinforcement Learning, Sutton & Barto, 2018, Exercise 5.12), but is not exactly the same. - -### Environment Description - -Consider driving a race car around a turn on a racetrack. In order to complete the race as quickly as possible, you would want to drive as fast as you can but, to avoid running off the track, you must slow down while turning. - -In our simplified racetrack environment, the agent is at one of a discrete set of grid positions. The agent also has a discrete speed in two directions, $x$ and $y$. So the state is represented as follows: -$$(\text{position}_y, \text{position}_x, \text{velocity}_y, \text{velocity}_x)$$ - -The agent collects a reward of -1 at each time step, an additional -10 for leaving the track (i.e., ending up on a black grid square in the figure below), and an additional +10 for reaching the finish line (any of the red grid squares). The agent starts each episode in a randomly selected grid-square on the starting line (green grid squares) with a speed of zero in both directions. At each time step, the agent can change its speed in both directions. Each speed can be changed by +1, -1 or 0, giving a total of nine actions. For example, the agent may increase its speed in the $x$ direction by -1 and its speed in the $y$ direction by +1. The agent's speed cannot be greater than +10 or less than -10 in either direction. - - - - -The agent's next state is determined by its current grid square, its current speed in two directions, and the changes it makes to its speed in the two directions. This environment is stochastic. When the agent tries to change its speed, no change occurs (in either direction) with probability 0.2. In other words, 20% of the time, the agent's action is ignored and the car's speed remains the same in both directions. - -If the agent leaves the track, it is returned to a random start grid-square and has its speed set to zero in both directions; the episode continues. An episode ends only when the agent transitions to a goal grid-square. - - - -### Environment Implementation -We have implemented the above environment in the `racetrack_env.py` file, for you to use in this coursework. Please use this implementation instead of writing your own, and please do not modify the environment. - -We provide a `RacetrackEnv` class for your agents to interact with. The class has the following methods: -- **`reset()`** - this method initialises the environment, chooses a random starting state, and returns it. This method should be called before the start of every episode. -- **`step(action)`** - this method takes an integer action (more on this later), and executes one time-step in the environment. It returns a tuple containing the next state, the reward collected, and whether the next state is a terminal state. -- **`render(sleep_time)`** - this method renders a matplotlib graph representing the environment. It takes an optional float parameter giving the number of seconds to display each time-step. This method is useful for testing and debugging, but should not be used during training since it is *very* slow. **Do not use this method in your final submission**. -- **`get_actions()`** - a simple method that returns the available actions in the current state. Always returns a list containing integers in the range [0-8] (more on this later). - -In our code, states are represented as Python tuples - specifically a tuple of four integers. For example, if the agent is in a grid square with coordinates ($Y = 2$, $X = 3$), and is moving zero cells vertically and one cell horizontally per time-step, the state is represented as `(2, 3, 0, 1)`. Tuples of this kind will be returned by the `reset()` and `step(action)` methods. - -There are nine actions available to the agent in each state, as described above. However, to simplify your code, we have represented each of the nine actions as an integer in the range [0-8]. The table below shows the index of each action, along with the corresponding changes it will cause to the agent's speed in each direction. - - - -For example, taking action 8 will increase the agent's speed in the $x$ direction, but decrease its speed in the $y$ direction. \ No newline at end of file diff --git a/projects/codes/envs/register.py b/projects/codes/envs/register.py deleted file mode 100644 index 38074cf..0000000 --- a/projects/codes/envs/register.py +++ /dev/null @@ -1,34 +0,0 @@ - -from gym.envs.registration import register - -def register_env(env_name): - if env_name == 'Racetrack-v0': - register( - id='Racetrack-v0', - entry_point='envs.racetrack:RacetrackEnv', - max_episode_steps=1000, - kwargs={} - ) - elif env_name == 'FrozenLakeNoSlippery-v1': - register( - id='FrozenLakeNoSlippery-v1', - entry_point='gym.envs.toy_text.frozen_lake:FrozenLakeEnv', - kwargs={'map_name':"4x4",'is_slippery':False}, - ) - else: - print("The env name must be wrong or the environment donot need to register!") - -# if __name__ == "__main__": -# import random -# import gym -# env = gym.make('FrozenLakeNoSlippery-v1') -# num_steps = 1000000 -# state = env.reset() -# n_actions = env.action_space.n -# print(state) -# for _ in range(num_steps) : -# next_state, reward, done,_ = env.step(random.choice(range(n_actions))) -# print(next_state) -# if (done) : -# _ = env.reset() - \ No newline at end of file diff --git a/projects/codes/envs/snake/README.md b/projects/codes/envs/snake/README.md deleted file mode 100644 index b49b4e8..0000000 --- a/projects/codes/envs/snake/README.md +++ /dev/null @@ -1,38 +0,0 @@ -# 贪吃蛇 - -贪吃蛇是一个起源于1976年的街机游戏 Blockade，玩家控制蛇上下左右吃到食物并将身体增长，吃到食物后移动速度逐渐加快，直到碰到墙体或者蛇的身体算游戏结束。 - -![image-20200901202636603](img/image-20200901202636603.png) - -如图，本次任务整个游戏版面大小为560X560，绿色部分就是我们的智能体贪吃蛇，红色方块就是食物，墙位于四周，一旦食物被吃掉，会在下一个随机位置刷出新的食物。蛇的每一节以及食物的大小为40X40，除开墙体(厚度也为40)，蛇可以活动的范围为480X480，也就是12X12的栅格。环境的状态等信息如下： - -* state：为一个元组，包含(adjoining_wall_x, adjoining_wall_y, food_dir_x, food_dir_y, adjoining_body_top, adjoining_body_bottom, adjoining_body_left, adjoining_body_right). - - * [adjoining_wall_x, adjoining_wall_y]：提供蛇头是否与墙体相邻的信息，具体包含9个状态 - - adjoining_wall_x：0表示x轴方向蛇头无墙体相邻，1表示有墙在蛇头左边，2表示有墙在右边adjoining_wall_y：0表示y轴方向蛇头无墙体相邻，1表示有墙在蛇头上边，2表示有墙在下边 - - 注意[0,0]也包括蛇跑出480X480范围的情况 - - * [food_dir_x, food_dir_y]：表示食物与蛇头的位置关系 - - food_dir_x：0表示食物与蛇头同在x轴上，1表示食物在蛇头左侧(不一定相邻)，2表示在右边 - - food_dir_y：0表示食物与蛇头同在y轴上，1表示食物在蛇头上面，2表示在下面 - - * [adjoining_body_top, adjoining_body_bottom, adjoining_body_left, adjoining_body_right]：用以检查蛇的身体是否在蛇头的附近 - - adjoining_body_top：1表示蛇头上边有蛇的身体，0表示没有 - - adjoining_body_bottom：1表示蛇头下边有蛇的身体，0表示没有 - - adjoining_body_left：1表示蛇头左边有蛇的身体，0表示没有 - - adjoining_body_right：1表示蛇头右边有蛇的身体，0表示没有 - -* action：即上下左右 - -* reward：如果吃到食物给一个+1的reward，如果蛇没了就-1，其他情况给-0.1的reward - - - diff --git a/projects/codes/envs/snake/agent.py b/projects/codes/envs/snake/agent.py deleted file mode 100644 index b32de9d..0000000 --- a/projects/codes/envs/snake/agent.py +++ /dev/null @@ -1,106 +0,0 @@ -import numpy as np -import utils -import random -import math - - -class Agent: - - def __init__(self, actions, Ne, C, gamma): - self.actions = actions - self.Ne = Ne # used in exploration function - self.C = C - self.gamma = gamma - - # Create the Q and N Table to work with - self.Q = utils.create_q_table() - self.N = utils.create_q_table() - self.reset() - - def train(self): - self._train = True - - def eval(self): - self._train = False - - # At the end of training save the trained model - def save_model(self, model_path): - utils.save(model_path, self.Q) - - # Load the trained model for evaluation - def load_model(self, model_path): - self.Q = utils.load(model_path) - - def reset(self): - self.points = 0 - self.s = None - self.a = None - - def f(self,u,n): - if n < self.Ne: - return 1 - return u - - def R(self,points,dead): - if dead: - return -1 - elif points > self.points: - return 1 - return -0.1 - - def get_state(self, state): - # [adjoining_wall_x, adjoining_wall_y] - adjoining_wall_x = int(state[0] == utils.WALL_SIZE) + 2 * int(state[0] == utils.DISPLAY_SIZE - utils.WALL_SIZE) - adjoining_wall_y = int(state[1] == utils.WALL_SIZE) + 2 * int(state[1] == utils.DISPLAY_SIZE - utils.WALL_SIZE) - # [food_dir_x, food_dir_y] - food_dir_x = 1 + int(state[0] < state[3]) - int(state[0] == state[3]) - food_dir_y = 1 + int(state[1] < state[4]) - int(state[1] == state[4]) - # [adjoining_body_top, adjoining_body_bottom, adjoining_body_left, adjoining_body_right] - adjoining_body = [(state[0] - body_state[0], state[1] - body_state[1]) for body_state in state[2]] - adjoining_body_top = int([0, utils.GRID_SIZE] in adjoining_body) - adjoining_body_bottom = int([0, -utils.GRID_SIZE] in adjoining_body) - adjoining_body_left = int([utils.GRID_SIZE, 0] in adjoining_body) - adjoining_body_right = int([-utils.GRID_SIZE, 0] in adjoining_body) - return adjoining_wall_x, adjoining_wall_y, food_dir_x, food_dir_y, adjoining_body_top, adjoining_body_bottom, adjoining_body_left, adjoining_body_right - - - def update(self, _state, points, dead): - if self.s: - maxq = max(self.Q[_state]) - reward = self.R(points,dead) - alpha = self.C / (self.C + self.N[self.s][self.a]) - self.Q[self.s][self.a] += alpha * (reward + self.gamma * maxq - self.Q[self.s][self.a]) - self.N[self.s][self.a] += 1.0 - - def choose_action(self, state, points, dead): - ''' - :param state: a list of [snake_head_x, snake_head_y, snake_body, food_x, food_y] from environment. - :param points: float, the current points from environment - :param dead: boolean, if the snake is dead - :return: the index of action. 0,1,2,3 indicates up,down,left,right separately - Return the index of action the snake needs to take, according to the state and points known from environment. - Tips: you need to discretize the state to the state space defined on the webpage first. - (Note that [adjoining_wall_x=0, adjoining_wall_y=0] is also the case when snake runs out of the 480x480 board) - ''' - - _state = self.get_state(state) - Qs = self.Q[_state][:] - - if self._train: - self.update(_state, points, dead) - if dead: - self.reset() - return - Ns = self.N[_state] - Fs = [self.f(Qs[a], Ns[a]) for a in self.actions] - action = np.argmax(Fs) - self.s = _state - self.a = action - else: - if dead: - self.reset() - return - action = np.argmax(Qs) - - self.points = points - return action diff --git a/projects/codes/envs/snake/example_assignment_and_report2.pdf b/projects/codes/envs/snake/example_assignment_and_report2.pdf deleted file mode 100644 index 84008c0..0000000 Binary files a/projects/codes/envs/snake/example_assignment_and_report2.pdf and /dev/null differ diff --git a/projects/codes/envs/snake/main.py b/projects/codes/envs/snake/main.py deleted file mode 100644 index 16776ad..0000000 --- a/projects/codes/envs/snake/main.py +++ /dev/null @@ -1,185 +0,0 @@ -import pygame -from pygame.locals import * -import argparse - -from agent import Agent -from snake_env import SnakeEnv -import utils -import time - -def get_args(): - parser = argparse.ArgumentParser(description='CS440 MP4 Snake') - - parser.add_argument('--human', default = False, action="store_true", - help='making the game human playable - default False') - - parser.add_argument('--model_name', dest="model_name", type=str, default="checkpoint3.npy", - help='name of model to save if training or to load if evaluating - default q_agent') - - parser.add_argument('--train_episodes', dest="train_eps", type=int, default=10000, - help='number of training episodes - default 10000') - - parser.add_argument('--test_episodes', dest="test_eps", type=int, default=1000, - help='number of testing episodes - default 1000') - - parser.add_argument('--show_episodes', dest="show_eps", type=int, default=10, - help='number of displayed episodes - default 10') - - parser.add_argument('--window', dest="window", type=int, default=100, - help='number of episodes to keep running stats for during training - default 100') - - parser.add_argument('--Ne', dest="Ne", type=int, default=40, - help='the Ne parameter used in exploration function - default 40') - - parser.add_argument('--C', dest="C", type=int, default=40, - help='the C parameter used in learning rate - default 40') - - parser.add_argument('--gamma', dest="gamma", type=float, default=0.2, - help='the gamma paramter used in learning rate - default 0.7') - - parser.add_argument('--snake_head_x', dest="snake_head_x", type=int, default=200, - help='initialized x position of snake head - default 200') - - parser.add_argument('--snake_head_y', dest="snake_head_y", type=int, default=200, - help='initialized y position of snake head - default 200') - - parser.add_argument('--food_x', dest="food_x", type=int, default=80, - help='initialized x position of food - default 80') - - parser.add_argument('--food_y', dest="food_y", type=int, default=80, - help='initialized y position of food - default 80') - cfg = parser.parse_args() - return cfg - -class Application: - def __init__(self, args): - self.args = args - self.env = SnakeEnv(args.snake_head_x, args.snake_head_y, args.food_x, args.food_y) - self.agent = Agent(self.env.get_actions(), args.Ne, args.C, args.gamma) - - def execute(self): - if not self.args.human: - if self.args.train_eps != 0: - self.train() - self.eval() - self.show_games() - - def train(self): - print("Train Phase:") - self.agent.train() - window = self.args.window - self.points_results = [] - first_eat = True - start = time.time() - - for game in range(1, self.args.train_eps + 1): - state = self.env.get_state() - dead = False - action = self.agent.choose_action(state, 0, dead) - while not dead: - state, points, dead = self.env.step(action) - - # For debug convenience, you can check if your Q-table mathches ours for given setting of parameters - # (see Debug Convenience part on homework 4 web page) - if first_eat and points == 1: - self.agent.save_model(utils.CHECKPOINT) - first_eat = False - - action = self.agent.choose_action(state, points, dead) - - - points = self.env.get_points() - self.points_results.append(points) - if game % self.args.window == 0: - print( - "Games:", len(self.points_results) - window, "-", len(self.points_results), - "Points (Average:", sum(self.points_results[-window:])/window, - "Max:", max(self.points_results[-window:]), - "Min:", min(self.points_results[-window:]),")", - ) - self.env.reset() - print("Training takes", time.time() - start, "seconds") - self.agent.save_model(self.args.model_name) - - def eval(self): - print("Evaling Phase:") - self.agent.eval() - self.agent.load_model(self.args.model_name) - points_results = [] - start = time.time() - - for game in range(1, self.args.test_eps + 1): - state = self.env.get_state() - dead = False - action = self.agent.choose_action(state, 0, dead) - while not dead: - state, points, dead = self.env.step(action) - action = self.agent.choose_action(state, points, dead) - points = self.env.get_points() - points_results.append(points) - self.env.reset() - - print("Testing takes", time.time() - start, "seconds") - print("Number of Games:", len(points_results)) - print("Average Points:", sum(points_results)/len(points_results)) - print("Max Points:", max(points_results)) - print("Min Points:", min(points_results)) - - def show_games(self): - print("Display Games") - self.env.display() - pygame.event.pump() - self.agent.eval() - points_results = [] - end = False - for game in range(1, self.args.show_eps + 1): - state = self.env.get_state() - dead = False - action = self.agent.choose_action(state, 0, dead) - count = 0 - while not dead: - count +=1 - pygame.event.pump() - keys = pygame.key.get_pressed() - if keys[K_ESCAPE] or self.check_quit(): - end = True - break - state, points, dead = self.env.step(action) - # Qlearning agent - if not self.args.human: - action = self.agent.choose_action(state, points, dead) - # for human player - else: - for event in pygame.event.get(): - if event.type == pygame.KEYDOWN: - if event.key == pygame.K_UP: - action = 2 - elif event.key == pygame.K_DOWN: - action = 3 - elif event.key == pygame.K_LEFT: - action = 1 - elif event.key == pygame.K_RIGHT: - action = 0 - if end: - break - self.env.reset() - points_results.append(points) - print("Game:", str(game)+"/"+str(self.args.show_eps), "Points:", points) - if len(points_results) == 0: - return - print("Average Points:", sum(points_results)/len(points_results)) - - def check_quit(self): - for event in pygame.event.get(): - if event.type == pygame.QUIT: - return True - return False - - -def main(): - cfg = get_args() - app = Application(cfg) - app.execute() - -if __name__ == "__main__": - main() diff --git a/projects/codes/envs/snake/snake_env.py b/projects/codes/envs/snake/snake_env.py deleted file mode 100644 index a4afe0a..0000000 --- a/projects/codes/envs/snake/snake_env.py +++ /dev/null @@ -1,202 +0,0 @@ -import random -import pygame -import utils - -class SnakeEnv: - def __init__(self, snake_head_x, snake_head_y, food_x, food_y): - self.game = Snake(snake_head_x, snake_head_y, food_x, food_y) - self.render = False - - def get_actions(self): - return self.game.get_actions() - - def reset(self): - return self.game.reset() - - def get_points(self): - return self.game.get_points() - - def get_state(self): - return self.game.get_state() - - def step(self, action): - state, points, dead = self.game.step(action) - if self.render: - self.draw(state, points, dead) - # return state, reward, done - return state, points, dead - - def draw(self, state, points, dead): - snake_head_x, snake_head_y, snake_body, food_x, food_y = state - self.display.fill(utils.BLUE) - pygame.draw.rect( self.display, utils.BLACK, - [ - utils.GRID_SIZE, - utils.GRID_SIZE, - utils.DISPLAY_SIZE - utils.GRID_SIZE * 2, - utils.DISPLAY_SIZE - utils.GRID_SIZE * 2 - ]) - - # draw snake head - pygame.draw.rect( - self.display, - utils.GREEN, - [ - snake_head_x, - snake_head_y, - utils.GRID_SIZE, - utils.GRID_SIZE - ], - 3 - ) - # draw snake body - for seg in snake_body: - pygame.draw.rect( - self.display, - utils.GREEN, - [ - seg[0], - seg[1], - utils.GRID_SIZE, - utils.GRID_SIZE, - ], - 1 - ) - # draw food - pygame.draw.rect( - self.display, - utils.RED, - [ - food_x, - food_y, - utils.GRID_SIZE, - utils.GRID_SIZE - ] - ) - - text_surface = self.font.render("Points: " + str(points), True, utils.BLACK) - text_rect = text_surface.get_rect() - text_rect.center = ((280),(25)) - self.display.blit(text_surface, text_rect) - pygame.display.flip() - if dead: - # slow clock if dead - self.clock.tick(1) - else: - self.clock.tick(5) - - return - - - def display(self): - pygame.init() - pygame.display.set_caption('MP4: Snake') - self.clock = pygame.time.Clock() - pygame.font.init() - - self.font = pygame.font.Font(pygame.font.get_default_font(), 15) - self.display = pygame.display.set_mode((utils.DISPLAY_SIZE, utils.DISPLAY_SIZE), pygame.HWSURFACE) - self.draw(self.game.get_state(), self.game.get_points(), False) - self.render = True - -class Snake: - def __init__(self, snake_head_x, snake_head_y, food_x, food_y): - self.init_snake_head_x,self.init_snake_head_y = snake_head_x,snake_head_y # 蛇头初始位置 - self.init_food_x, self.init_food_y = food_x, food_y # 食物初始位置 - self.reset() - - def reset(self): - self.points = 0 - self.snake_head_x, self.snake_head_y = self.init_snake_head_x, self.init_snake_head_y - self.food_x, self.food_y = self.init_food_x, self.init_food_y - self.snake_body = [] # 蛇身的位置集合 - - def get_points(self): - return self.points - - def get_actions(self): - return [0, 1, 2, 3] - - def get_state(self): - return [ - self.snake_head_x, - self.snake_head_y, - self.snake_body, - self.food_x, - self.food_y - ] - - def move(self, action): - '''根据action指令移动蛇头，并返回是否撞死 - ''' - delta_x = delta_y = 0 - if action == 0: # 上 - delta_x = utils.GRID_SIZE - elif action == 1: - delta_x = - utils.GRID_SIZE - elif action == 2: - delta_y = - utils.GRID_SIZE - elif action == 3: - delta_y = utils.GRID_SIZE - old_body_head = None - if len(self.snake_body) == 1: - old_body_head = self.snake_body[0] - - self.snake_body.append((self.snake_head_x, self.snake_head_y)) - self.snake_head_x += delta_x - self.snake_head_y += delta_y - - if len(self.snake_body) > self.points: # 说明没有吃到食物 - del(self.snake_body[0]) - - self.handle_eatfood() - - # 蛇长大于1时，蛇头与蛇身任一位置重叠则看作蛇与自身相撞 - if len(self.snake_body) >= 1: - for seg in self.snake_body: - if self.snake_head_x == seg[0] and self.snake_head_y == seg[1]: - return True - - # 蛇长为1时，如果蛇头与之前的位置重复则看作蛇与自身相撞 - if len(self.snake_body) == 1: - if old_body_head == (self.snake_head_x, self.snake_head_y): - return True - - # 蛇头是否撞墙 - if (self.snake_head_x < utils.GRID_SIZE or self.snake_head_y < utils.GRID_SIZE or - self.snake_head_x + utils.GRID_SIZE > utils.DISPLAY_SIZE-utils.GRID_SIZE or self.snake_head_y + utils.GRID_SIZE > utils.DISPLAY_SIZE-utils.GRID_SIZE): - return True - - return False - - def step(self, action): - is_dead = self.move(action) - return self.get_state(), self.get_points(), is_dead - - def handle_eatfood(self): - if (self.snake_head_x == self.food_x) and (self.snake_head_y == self.food_y): - self.random_food() - self.points += 1 - - def random_food(self): - '''生成随机位置的食物 - ''' - max_x = (utils.DISPLAY_SIZE - utils.WALL_SIZE - utils.GRID_SIZE) - max_y = (utils.DISPLAY_SIZE - utils.WALL_SIZE - utils.GRID_SIZE) - - self.food_x = random.randint(utils.WALL_SIZE, max_x)//utils.GRID_SIZE * utils.GRID_SIZE - self.food_y = random.randint(utils.WALL_SIZE, max_y)//utils.GRID_SIZE * utils.GRID_SIZE - - while self.check_food_on_snake(): # 食物不能生成在蛇身上 - self.food_x = random.randint(utils.WALL_SIZE, max_x)//utils.GRID_SIZE * utils.GRID_SIZE - self.food_y = random.randint(utils.WALL_SIZE, max_y)//utils.GRID_SIZE * utils.GRID_SIZE - - def check_food_on_snake(self): - if self.food_x == self.snake_head_x and self.food_y == self.snake_head_y: - return True - for seg in self.snake_body: - if self.food_x == seg[0] and self.food_y == seg[1]: - return True - return False - - diff --git a/projects/codes/envs/snake/utils.py b/projects/codes/envs/snake/utils.py deleted file mode 100644 index 01c9b00..0000000 --- a/projects/codes/envs/snake/utils.py +++ /dev/null @@ -1,55 +0,0 @@ -import numpy as np -DISPLAY_SIZE = 560 -GRID_SIZE = 40 -WALL_SIZE = 40 -WHITE = (255, 255, 255) -RED = (255, 0, 0) -BLUE = (72, 61, 139) -BLACK = (0, 0, 0) -GREEN = (0, 255, 0) - -NUM_ADJOINING_WALL_X_STATES=3 -NUM_ADJOINING_WALL_Y_STATES=3 -NUM_FOOD_DIR_X=3 -NUM_FOOD_DIR_Y=3 -NUM_ADJOINING_BODY_TOP_STATES=2 -NUM_ADJOINING_BODY_BOTTOM_STATES=2 -NUM_ADJOINING_BODY_LEFT_STATES=2 -NUM_ADJOINING_BODY_RIGHT_STATES=2 -NUM_ACTIONS = 4 - -CHECKPOINT = 'checkpoint.npy' - -def create_q_table(): - return np.zeros((NUM_ADJOINING_WALL_X_STATES, NUM_ADJOINING_WALL_Y_STATES, NUM_FOOD_DIR_X, NUM_FOOD_DIR_Y, - NUM_ADJOINING_BODY_TOP_STATES, NUM_ADJOINING_BODY_BOTTOM_STATES, NUM_ADJOINING_BODY_LEFT_STATES, - NUM_ADJOINING_BODY_RIGHT_STATES, NUM_ACTIONS)) - -def sanity_check(arr): - if (type(arr) is np.ndarray and - arr.shape==(NUM_ADJOINING_WALL_X_STATES, NUM_ADJOINING_WALL_Y_STATES, NUM_FOOD_DIR_X, NUM_FOOD_DIR_Y, - NUM_ADJOINING_BODY_TOP_STATES, NUM_ADJOINING_BODY_BOTTOM_STATES, NUM_ADJOINING_BODY_LEFT_STATES, - NUM_ADJOINING_BODY_RIGHT_STATES,NUM_ACTIONS)): - return True - else: - return False - -def save(filename, arr): - if sanity_check(arr): - np.save(filename,arr) - return True - else: - print("Failed to save model") - return False - -def load(filename): - try: - arr = np.load(filename) - if sanity_check(arr): - print("Loaded model successfully") - return arr - print("Model loaded is not in the required format") - return None - except: - print("Filename doesnt exist") - return None \ No newline at end of file diff --git a/projects/codes/envs/stochastic_mdp.py b/projects/codes/envs/stochastic_mdp.py deleted file mode 100644 index 3c1ad4d..0000000 --- a/projects/codes/envs/stochastic_mdp.py +++ /dev/null @@ -1,53 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: John -Email: johnjim0816@gmail.com -Date: 2021-03-24 22:12:19 -LastEditor: John -LastEditTime: 2021-03-26 17:12:43 -Discription: -Environment: -''' -import numpy as np -import random - - -class StochasticMDP: - def __init__(self): - self.end = False - self.curr_state = 2 - self.n_actions = 2 - self.n_states = 6 - self.p_right = 0.5 - - def reset(self): - self.end = False - self.curr_state = 2 - state = np.zeros(self.n_states) - state[self.curr_state - 1] = 1. - return state - - def step(self, action): - if self.curr_state != 1: - if action == 1: - if random.random() < self.p_right and self.curr_state < self.n_states: - self.curr_state += 1 - else: - self.curr_state -= 1 - - if action == 0: - self.curr_state -= 1 - if self.curr_state == self.n_states: - self.end = True - - state = np.zeros(self.n_states) - state[self.curr_state - 1] = 1. - - if self.curr_state == 1: - if self.end: - return state, 1.00, True, {} - else: - return state, 1.00/100.00, True, {} - else: - return state, 0.0, False, {} diff --git a/projects/codes/envs/track.txt b/projects/codes/envs/track.txt deleted file mode 100644 index 4bbe230..0000000 --- a/projects/codes/envs/track.txt +++ /dev/null @@ -1,15 +0,0 @@ -1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 -1 1 1 1 1 1 1 1 0 0 0 0 0 3 3 3 3 3 1 -1 1 1 1 1 1 0 0 0 0 0 0 0 3 3 3 3 3 1 -1 1 1 1 1 0 0 0 0 0 0 0 0 3 3 3 3 3 1 -1 1 1 1 0 0 0 0 0 0 0 0 0 3 3 3 3 3 1 -1 1 1 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 -1 1 1 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 -1 1 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 -1 1 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 -1 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 -1 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 -1 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 -1 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 -1 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 -1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 \ No newline at end of file diff --git a/projects/codes/envs/windy_gridworld.py b/projects/codes/envs/windy_gridworld.py deleted file mode 100644 index 2a9d4a4..0000000 --- a/projects/codes/envs/windy_gridworld.py +++ /dev/null @@ -1,82 +0,0 @@ -import gym -import numpy as np -import sys -from gym.envs.toy_text import discrete - -UP = 0 -RIGHT = 1 -DOWN = 2 -LEFT = 3 - -class WindyGridworldEnv(discrete.DiscreteEnv): - - metadata = {'render.modes': ['human', 'ansi']} - - def _limit_coordinates(self, coord): - coord[0] = min(coord[0], self.shape[0] - 1) - coord[0] = max(coord[0], 0) - coord[1] = min(coord[1], self.shape[1] - 1) - coord[1] = max(coord[1], 0) - return coord - - def _calculate_transition_prob(self, current, delta, winds): - new_position = np.array(current) + np.array(delta) + np.array([-1, 0]) * winds[tuple(current)] - new_position = self._limit_coordinates(new_position).astype(int) - new_state = np.ravel_multi_index(tuple(new_position), self.shape) - is_done = tuple(new_position) == (3, 7) - return [(1.0, new_state, -1.0, is_done)] - - def __init__(self): - self.shape = (7, 10) - - nS = np.prod(self.shape) - n_actions = 4 - - # Wind strength - winds = np.zeros(self.shape) - winds[:,[3,4,5,8]] = 1 - winds[:,[6,7]] = 2 - - # Calculate transition probabilities - P = {} - for s in range(nS): - position = np.unravel_index(s, self.shape) - P[s] = { a : [] for a in range(n_actions) } - P[s][UP] = self._calculate_transition_prob(position, [-1, 0], winds) - P[s][RIGHT] = self._calculate_transition_prob(position, [0, 1], winds) - P[s][DOWN] = self._calculate_transition_prob(position, [1, 0], winds) - P[s][LEFT] = self._calculate_transition_prob(position, [0, -1], winds) - - # We always start in state (3, 0) - isd = np.zeros(nS) - isd[np.ravel_multi_index((3,0), self.shape)] = 1.0 - - super(WindyGridworldEnv, self).__init__(nS, n_actions, P, isd) - - def render(self, mode='human', close=False): - self._render(mode, close) - - def _render(self, mode='human', close=False): - if close: - return - - outfile = StringIO() if mode == 'ansi' else sys.stdout - - for s in range(self.nS): - position = np.unravel_index(s, self.shape) - # print(self.s) - if self.s == s: - output = " x " - elif position == (3,7): - output = " T " - else: - output = " o " - - if position[1] == 0: - output = output.lstrip() - if position[1] == self.shape[1] - 1: - output = output.rstrip() - output += "\n" - - outfile.write(output) - outfile.write("\n") diff --git a/projects/codes/envs/wrappers.py b/projects/codes/envs/wrappers.py deleted file mode 100644 index 0baa03b..0000000 --- a/projects/codes/envs/wrappers.py +++ /dev/null @@ -1,78 +0,0 @@ -import gym -class CliffWalkingWapper(gym.Wrapper): - def __init__(self, env): - gym.Wrapper.__init__(self, env) - self.t = None - self.unit = 50 - self.max_x = 12 - self.max_y = 4 - - def draw_x_line(self, y, x0, x1, color='gray'): - assert x1 > x0 - self.t.color(color) - self.t.setheading(0) - self.t.up() - self.t.goto(x0, y) - self.t.down() - self.t.forward(x1 - x0) - - def draw_y_line(self, x, y0, y1, color='gray'): - assert y1 > y0 - self.t.color(color) - self.t.setheading(90) - self.t.up() - self.t.goto(x, y0) - self.t.down() - self.t.forward(y1 - y0) - - def draw_box(self, x, y, fillcolor='', line_color='gray'): - self.t.up() - self.t.goto(x * self.unit, y * self.unit) - self.t.color(line_color) - self.t.fillcolor(fillcolor) - self.t.setheading(90) - self.t.down() - self.t.begin_fill() - for i in range(4): - self.t.forward(self.unit) - self.t.right(90) - self.t.end_fill() - - def move_player(self, x, y): - self.t.up() - self.t.setheading(90) - self.t.fillcolor('red') - self.t.goto((x + 0.5) * self.unit, (y + 0.5) * self.unit) - - def render(self): - if self.t == None: - self.t = turtle.Turtle() - self.wn = turtle.Screen() - self.wn.setup(self.unit * self.max_x + 100, - self.unit * self.max_y + 100) - self.wn.setworldcoordinates(0, 0, self.unit * self.max_x, - self.unit * self.max_y) - self.t.shape('circle') - self.t.width(2) - self.t.speed(0) - self.t.color('gray') - for _ in range(2): - self.t.forward(self.max_x * self.unit) - self.t.left(90) - self.t.forward(self.max_y * self.unit) - self.t.left(90) - for i in range(1, self.max_y): - self.draw_x_line( - y=i * self.unit, x0=0, x1=self.max_x * self.unit) - for i in range(1, self.max_x): - self.draw_y_line( - x=i * self.unit, y0=0, y1=self.max_y * self.unit) - - for i in range(1, self.max_x - 1): - self.draw_box(i, 0, 'black') - self.draw_box(self.max_x - 1, 0, 'yellow') - self.t.shape('turtle') - - x_pos = self.s % self.max_x - y_pos = self.max_y - 1 - int(self.s / self.max_x) - self.move_player(x_pos, y_pos) \ No newline at end of file diff --git a/projects/codes/scripts/DQN_Acrobot-v1.sh b/projects/codes/scripts/DQN_Acrobot-v1.sh deleted file mode 100644 index 623a0cc..0000000 --- a/projects/codes/scripts/DQN_Acrobot-v1.sh +++ /dev/null @@ -1,3 +0,0 @@ -# run DQN on Acrobot-v1, not the best tuned parameters -codes_dir=$(dirname $(dirname $(readlink -f "$0"))) # "codes" path -python $codes_dir/DQN/main.py --env_name Acrobot-v1 --train_eps 100 --epsilon_decay 1500 --lr 0.002 --memory_capacity 200000 --batch_size 128 --device cuda \ No newline at end of file diff --git a/projects/codes/scripts/DQN_CartPole-v1.sh b/projects/codes/scripts/DQN_CartPole-v1.sh deleted file mode 100644 index e4fe811..0000000 --- a/projects/codes/scripts/DQN_CartPole-v1.sh +++ /dev/null @@ -1,3 +0,0 @@ -# run DQN on CartPole-v1, not finished yet -codes_dir=$(dirname $(dirname $(readlink -f "$0"))) # "codes" path -python $codes_dir/DQN/main.py --env_name CartPole-v1 --train_eps 2000 --gamma 0.99 --epsilon_decay 6000 --lr 0.00001 --memory_capacity 200000 --batch_size 64 --device cuda \ No newline at end of file diff --git a/projects/codes/scripts/DoubleDQN_CartPole-v0.sh b/projects/codes/scripts/DoubleDQN_CartPole-v0.sh deleted file mode 100644 index 0154227..0000000 --- a/projects/codes/scripts/DoubleDQN_CartPole-v0.sh +++ /dev/null @@ -1,3 +0,0 @@ -# run Double DQN on CartPole-v0 -codes_dir=$(dirname $(dirname $(readlink -f "$0"))) # "codes" path -python $codes_dir/DoubleDQN/main.py --device cuda \ No newline at end of file diff --git a/projects/codes/scripts/PolicyGradient_CartPole-v0.sh b/projects/codes/scripts/PolicyGradient_CartPole-v0.sh deleted file mode 100644 index d7e0a69..0000000 --- a/projects/codes/scripts/PolicyGradient_CartPole-v0.sh +++ /dev/null @@ -1,13 +0,0 @@ -# source conda, if you are already in proper conda environment, then comment the codes util "conda activate easyrl" -if [ -f "$HOME/anaconda3/etc/profile.d/conda.sh" ]; then - echo "source file at ~/anaconda3/etc/profile.d/conda.sh" - source ~/anaconda3/etc/profile.d/conda.sh -elif [ -f "$HOME/opt/anaconda3/etc/profile.d/conda.sh" ]; then - echo "source file at ~/opt/anaconda3/etc/profile.d/conda.sh" - source ~/opt/anaconda3/etc/profile.d/conda.sh -else - echo 'please manually config the conda source path' -fi -conda activate easyrl # easyrl here can be changed to another name of conda env that you have created -codes_dir=$(dirname $(dirname $(readlink -f "$0"))) # "codes" path -python $codes_dir/PolicyGradient/main.py \ No newline at end of file diff --git a/projects/codes/scripts/Qlearning_CliffWalking-v0.sh b/projects/codes/scripts/Qlearning_CliffWalking-v0.sh deleted file mode 100644 index 6ba8b53..0000000 --- a/projects/codes/scripts/Qlearning_CliffWalking-v0.sh +++ /dev/null @@ -1,2 +0,0 @@ -codes_dir=$(dirname $(dirname $(readlink -f "$0"))) # "codes" path -python $codes_dir/QLearning/main.py --env_name CliffWalking-v0 --train_eps 400 --gamma 0.90 --epsilon_start 0.95 --epsilon_end 0.01 --epsilon_decay 300 --lr 0.1 --device cpu \ No newline at end of file diff --git a/projects/codes/scripts/Qlearning_FrozenLakeNoSlippery-v1.sh b/projects/codes/scripts/Qlearning_FrozenLakeNoSlippery-v1.sh deleted file mode 100644 index c4638fe..0000000 --- a/projects/codes/scripts/Qlearning_FrozenLakeNoSlippery-v1.sh +++ /dev/null @@ -1,2 +0,0 @@ -codes_dir=$(dirname $(dirname $(readlink -f "$0"))) # "codes" path -python $codes_dir/QLearning/main.py --env_name FrozenLakeNoSlippery-v1 --train_eps 800 --epsilon_start 0.70 --epsilon_end 0.1 --epsilon_decay 2000 --gamma 0.9 --lr 0.9 --device cpu \ No newline at end of file diff --git a/projects/codes/scripts/Qlearning_Racetrack-v0.sh b/projects/codes/scripts/Qlearning_Racetrack-v0.sh deleted file mode 100644 index aba42b2..0000000 --- a/projects/codes/scripts/Qlearning_Racetrack-v0.sh +++ /dev/null @@ -1,2 +0,0 @@ -codes_dir=$(dirname $(dirname $(readlink -f "$0"))) # "codes" path -python $codes_dir/QLearning/main.py --env_name Racetrack-v0 --device cpu \ No newline at end of file diff --git a/projects/codes/scripts/Sarsa_CliffWalking-v0.sh b/projects/codes/scripts/Sarsa_CliffWalking-v0.sh deleted file mode 100644 index 9207c9d..0000000 --- a/projects/codes/scripts/Sarsa_CliffWalking-v0.sh +++ /dev/null @@ -1,2 +0,0 @@ -codes_dir=$(dirname $(dirname $(readlink -f "$0"))) # "codes" path -python $codes_dir/Sarsa/main.py --env_name CliffWalking-v0 --train_eps 400 --gamma 0.90 --epsilon_start 0.95 --epsilon_end 0.01 --epsilon_decay 300 --lr 0.1 --device cpu \ No newline at end of file diff --git a/projects/codes/scripts/Sarsa_FrozenLakeNoSlippery-v1.sh b/projects/codes/scripts/Sarsa_FrozenLakeNoSlippery-v1.sh deleted file mode 100644 index 9c77e75..0000000 --- a/projects/codes/scripts/Sarsa_FrozenLakeNoSlippery-v1.sh +++ /dev/null @@ -1,2 +0,0 @@ -codes_dir=$(dirname $(dirname $(readlink -f "$0"))) # "codes" path -python $codes_dir/Sarsa/main.py --env_name FrozenLakeNoSlippery-v1 --train_eps 800 --ep_max_steps 10 --epsilon_start 0.50 --epsilon_end 0.01 --epsilon_decay 2000 --gamma 0.9 --lr 0.1 --device cpu \ No newline at end of file diff --git a/projects/codes/scripts/Sarsa_Racetrack-v0.sh b/projects/codes/scripts/Sarsa_Racetrack-v0.sh deleted file mode 100644 index ff8317e..0000000 --- a/projects/codes/scripts/Sarsa_Racetrack-v0.sh +++ /dev/null @@ -1,2 +0,0 @@ -codes_dir=$(dirname $(dirname $(readlink -f "$0"))) # "codes" path -python $codes_dir/Sarsa/main.py --env_name Racetrack-v0 \ No newline at end of file diff --git a/projects/notebooks/2.Sarsa.ipynb b/projects/notebooks/2.Sarsa.ipynb deleted file mode 100644 index 493cb59..0000000 --- a/projects/notebooks/2.Sarsa.ipynb +++ /dev/null @@ -1,896 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1、定义算法\n", - "\n", - "在阅读该教程之前，请先阅读Q learning教程。Sarsa算法跟Q learning算法基本模式相同，但是根本的区别在于，Sarsa是先做出动作然后拿这个做的动作去更新，而Q learning是假定下一步最大奖励对应的动作拿去更新，然后再使用$\\varepsilon$-greedy策略，也就是说Sarsa是on-policy的，而Q learning是off-policy的。如下方代码所示，只有在更新的地方Sarsa与Q learning有着细微的区别。" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np\n", - "from collections import defaultdict\n", - "import torch\n", - "import math\n", - "class Sarsa(object):\n", - " def __init__(self,\n", - " n_actions,cfg):\n", - " self.n_actions = n_actions \n", - " self.lr = cfg.lr \n", - " self.gamma = cfg.gamma \n", - " self.sample_count = 0 \n", - " self.epsilon_start = cfg.epsilon_start\n", - " self.epsilon_end = cfg.epsilon_end\n", - " self.epsilon_decay = cfg.epsilon_decay \n", - " self.Q = defaultdict(lambda: np.zeros(n_actions)) # Q table\n", - " def sample(self, state):\n", - " self.sample_count += 1\n", - " self.epsilon = self.epsilon_end + (self.epsilon_start - self.epsilon_end) * \\\n", - " math.exp(-1. * self.sample_count / self.epsilon_decay) # The probability to select a random action, is is log decayed\n", - " best_action = np.argmax(self.Q[state])\n", - " action_probs = np.ones(self.n_actions, dtype=float) * self.epsilon / self.n_actions\n", - " action_probs[best_action] += (1.0 - self.epsilon)\n", - " action = np.random.choice(np.arange(len(action_probs)), p=action_probs) \n", - " return action\n", - " def predict(self,state):\n", - " return np.argmax(self.Q[state])\n", - " def update(self, state, action, reward, next_state, next_action,done):\n", - " Q_predict = self.Q[state][action]\n", - " if done:\n", - " Q_target = reward # 终止状态\n", - " else:\n", - " Q_target = reward + self.gamma * self.Q[next_state][next_action] # 与Q learning不同，Sarsa是拿下一步动作对应的Q值去更新\n", - " self.Q[state][action] += self.lr * (Q_target - Q_predict) \n", - " def save(self,path):\n", - " '''把 Q表格 的数据保存到文件中\n", - " '''\n", - " import dill\n", - " torch.save(\n", - " obj=self.Q,\n", - " f=path+\"sarsa_model.pkl\",\n", - " pickle_module=dill\n", - " )\n", - " def load(self, path):\n", - " '''从文件中读取数据到 Q表格\n", - " '''\n", - " import dill\n", - " self.Q =torch.load(f=path+'sarsa_model.pkl',pickle_module=dill)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2、定义训练\n", - "\n", - "同样地，跟Q learning差别也不大" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [], - "source": [ - "def train(cfg,env,agent):\n", - " print('开始训练！')\n", - " print(f'环境:{cfg.env_name}, 算法:{cfg.algo_name}, 设备:{cfg.device}')\n", - " rewards = [] # 记录奖励\n", - " for i_ep in range(cfg.train_eps):\n", - " ep_reward = 0 # 记录每个回合的奖励\n", - " state = env.reset() # 重置环境,即开始新的回合\n", - " action = agent.sample(state)\n", - " while True:\n", - " action = agent.sample(state) # 根据算法采样一个动作\n", - " next_state, reward, done, _ = env.step(action) # 与环境进行一次动作交互\n", - " next_action = agent.sample(next_state)\n", - " agent.update(state, action, reward, next_state, next_action,done) # 算法更新\n", - " state = next_state # 更新状态\n", - " action = next_action\n", - " ep_reward += reward\n", - " if done:\n", - " break\n", - " rewards.append(ep_reward)\n", - " print(f\"回合：{i_ep+1}/{cfg.train_eps}，奖励：{ep_reward:.1f}，Epsilon：{agent.epsilon}\")\n", - " print('完成训练！')\n", - " return {\"rewards\":rewards}\n", - " \n", - "def test(cfg,env,agent):\n", - " print('开始测试！')\n", - " print(f'环境：{cfg.env_name}, 算法：{cfg.algo_name}, 设备：{cfg.device}')\n", - " rewards = [] # 记录所有回合的奖励\n", - " for i_ep in range(cfg.test_eps):\n", - " ep_reward = 0 # 记录每个episode的reward\n", - " state = env.reset() # 重置环境, 重新开一局（即开始新的一个回合）\n", - " while True:\n", - " action = agent.predict(state) # 根据算法选择一个动作\n", - " next_state, reward, done, _ = env.step(action) # 与环境进行一个交互\n", - " state = next_state # 更新状态\n", - " ep_reward += reward\n", - " if done:\n", - " break\n", - " rewards.append(ep_reward)\n", - " print(f\"回合数：{i_ep+1}/{cfg.test_eps}, 奖励：{ep_reward:.1f}\")\n", - " print('完成测试！')\n", - " return {\"rewards\":rewards}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 3、定义环境\n", - "为了具体看看Q learning和Sarsa的不同，笔者决定跟Q learning使用相同的环境\n" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "import gym\n", - "import turtle\n", - "import numpy as np\n", - "\n", - "# turtle tutorial : https://docs.python.org/3.3/library/turtle.html\n", - "\n", - "def GridWorld(gridmap=None, is_slippery=False):\n", - " if gridmap is None:\n", - " gridmap = ['SFFF', 'FHFH', 'FFFH', 'HFFG']\n", - " env = gym.make(\"FrozenLake-v0\", desc=gridmap, is_slippery=False)\n", - " env = FrozenLakeWapper(env)\n", - " return env\n", - "\n", - "\n", - "class FrozenLakeWapper(gym.Wrapper):\n", - " def __init__(self, env):\n", - " gym.Wrapper.__init__(self, env)\n", - " self.max_y = env.desc.shape[0]\n", - " self.max_x = env.desc.shape[1]\n", - " self.t = None\n", - " self.unit = 50\n", - "\n", - " def draw_box(self, x, y, fillcolor='', line_color='gray'):\n", - " self.t.up()\n", - " self.t.goto(x * self.unit, y * self.unit)\n", - " self.t.color(line_color)\n", - " self.t.fillcolor(fillcolor)\n", - " self.t.setheading(90)\n", - " self.t.down()\n", - " self.t.begin_fill()\n", - " for _ in range(4):\n", - " self.t.forward(self.unit)\n", - " self.t.right(90)\n", - " self.t.end_fill()\n", - "\n", - " def move_player(self, x, y):\n", - " self.t.up()\n", - " self.t.setheading(90)\n", - " self.t.fillcolor('red')\n", - " self.t.goto((x + 0.5) * self.unit, (y + 0.5) * self.unit)\n", - "\n", - " def render(self):\n", - " if self.t == None:\n", - " self.t = turtle.Turtle()\n", - " self.wn = turtle.Screen()\n", - " self.wn.setup(self.unit * self.max_x + 100,\n", - " self.unit * self.max_y + 100)\n", - " self.wn.setworldcoordinates(0, 0, self.unit * self.max_x,\n", - " self.unit * self.max_y)\n", - " self.t.shape('circle')\n", - " self.t.width(2)\n", - " self.t.speed(0)\n", - " self.t.color('gray')\n", - " for i in range(self.desc.shape[0]):\n", - " for j in range(self.desc.shape[1]):\n", - " x = j\n", - " y = self.max_y - 1 - i\n", - " if self.desc[i][j] == b'S': # Start\n", - " self.draw_box(x, y, 'white')\n", - " elif self.desc[i][j] == b'F': # Frozen ice\n", - " self.draw_box(x, y, 'white')\n", - " elif self.desc[i][j] == b'G': # Goal\n", - " self.draw_box(x, y, 'yellow')\n", - " elif self.desc[i][j] == b'H': # Hole\n", - " self.draw_box(x, y, 'black')\n", - " else:\n", - " self.draw_box(x, y, 'white')\n", - " self.t.shape('turtle')\n", - "\n", - " x_pos = self.s % self.max_x\n", - " y_pos = self.max_y - 1 - int(self.s / self.max_x)\n", - " self.move_player(x_pos, y_pos)\n", - "\n", - "\n", - "class CliffWalkingWapper(gym.Wrapper):\n", - " def __init__(self, env):\n", - " gym.Wrapper.__init__(self, env)\n", - " self.t = None\n", - " self.unit = 50\n", - " self.max_x = 12\n", - " self.max_y = 4\n", - "\n", - " def draw_x_line(self, y, x0, x1, color='gray'):\n", - " assert x1 > x0\n", - " self.t.color(color)\n", - " self.t.setheading(0)\n", - " self.t.up()\n", - " self.t.goto(x0, y)\n", - " self.t.down()\n", - " self.t.forward(x1 - x0)\n", - "\n", - " def draw_y_line(self, x, y0, y1, color='gray'):\n", - " assert y1 > y0\n", - " self.t.color(color)\n", - " self.t.setheading(90)\n", - " self.t.up()\n", - " self.t.goto(x, y0)\n", - " self.t.down()\n", - " self.t.forward(y1 - y0)\n", - "\n", - " def draw_box(self, x, y, fillcolor='', line_color='gray'):\n", - " self.t.up()\n", - " self.t.goto(x * self.unit, y * self.unit)\n", - " self.t.color(line_color)\n", - " self.t.fillcolor(fillcolor)\n", - " self.t.setheading(90)\n", - " self.t.down()\n", - " self.t.begin_fill()\n", - " for i in range(4):\n", - " self.t.forward(self.unit)\n", - " self.t.right(90)\n", - " self.t.end_fill()\n", - "\n", - " def move_player(self, x, y):\n", - " self.t.up()\n", - " self.t.setheading(90)\n", - " self.t.fillcolor('red')\n", - " self.t.goto((x + 0.5) * self.unit, (y + 0.5) * self.unit)\n", - "\n", - " def render(self):\n", - " if self.t == None:\n", - " self.t = turtle.Turtle()\n", - " self.wn = turtle.Screen()\n", - " self.wn.setup(self.unit * self.max_x + 100,\n", - " self.unit * self.max_y + 100)\n", - " self.wn.setworldcoordinates(0, 0, self.unit * self.max_x,\n", - " self.unit * self.max_y)\n", - " self.t.shape('circle')\n", - " self.t.width(2)\n", - " self.t.speed(0)\n", - " self.t.color('gray')\n", - " for _ in range(2):\n", - " self.t.forward(self.max_x * self.unit)\n", - " self.t.left(90)\n", - " self.t.forward(self.max_y * self.unit)\n", - " self.t.left(90)\n", - " for i in range(1, self.max_y):\n", - " self.draw_x_line(\n", - " y=i * self.unit, x0=0, x1=self.max_x * self.unit)\n", - " for i in range(1, self.max_x):\n", - " self.draw_y_line(\n", - " x=i * self.unit, y0=0, y1=self.max_y * self.unit)\n", - "\n", - " for i in range(1, self.max_x - 1):\n", - " self.draw_box(i, 0, 'black')\n", - " self.draw_box(self.max_x - 1, 0, 'yellow')\n", - " self.t.shape('turtle')\n", - "\n", - " x_pos = self.s % self.max_x\n", - " y_pos = self.max_y - 1 - int(self.s / self.max_x)\n", - " self.move_player(x_pos, y_pos)" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "def env_agent_config(cfg,seed=1):\n", - " '''创建环境和智能体\n", - " Args:\n", - " cfg ([type]): [description]\n", - " seed (int, optional): 随机种子. Defaults to 1.\n", - " Returns:\n", - " env [type]: 环境\n", - " agent : 智能体\n", - " ''' \n", - " env = gym.make(cfg.env_name) \n", - " env = CliffWalkingWapper(env)\n", - " env.seed(seed) # 设置随机种子\n", - " n_states = env.observation_space.n # 状态维度\n", - " n_actions = env.action_space.n # 动作维度\n", - " print(f\"状态数：{n_states}，动作数：{n_actions}\")\n", - " agent = Sarsa(n_actions,cfg)\n", - " return env,agent" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 4、设置参数\n", - "同样的参数也是一样" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "import datetime\n", - "import argparse\n", - "import matplotlib.pyplot as plt\n", - "import seaborn as sns\n", - "def get_args():\n", - " \"\"\" \n", - " \"\"\"\n", - " curr_time = datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\") # 获取当前时间\n", - " parser = argparse.ArgumentParser(description=\"hyperparameters\") \n", - " parser.add_argument('--algo_name',default='Sarsa',type=str,help=\"name of algorithm\")\n", - " parser.add_argument('--env_name',default='CliffWalking-v0',type=str,help=\"name of environment\")\n", - " parser.add_argument('--train_eps',default=400,type=int,help=\"episodes of training\") # 训练的回合数\n", - " parser.add_argument('--test_eps',default=20,type=int,help=\"episodes of testing\") # 测试的回合数\n", - " parser.add_argument('--gamma',default=0.90,type=float,help=\"discounted factor\") # 折扣因子\n", - " parser.add_argument('--epsilon_start',default=0.95,type=float,help=\"initial value of epsilon\") # e-greedy策略中初始epsilon\n", - " parser.add_argument('--epsilon_end',default=0.01,type=float,help=\"final value of epsilon\") # e-greedy策略中的终止epsilon\n", - " parser.add_argument('--epsilon_decay',default=300,type=int,help=\"decay rate of epsilon\") # e-greedy策略中epsilon的衰减率\n", - " parser.add_argument('--lr',default=0.1,type=float,help=\"learning rate\")\n", - " parser.add_argument('--device',default='cpu',type=str,help=\"cpu or cuda\") \n", - " args = parser.parse_args([]) \n", - " return args\n", - "\n", - "def smooth(data, weight=0.9): \n", - " '''用于平滑曲线，类似于Tensorboard中的smooth\n", - "\n", - " Args:\n", - " data (List):输入数据\n", - " weight (Float): 平滑权重，处于0-1之间，数值越高说明越平滑，一般取0.9\n", - "\n", - " Returns:\n", - " smoothed (List): 平滑后的数据\n", - " '''\n", - " last = data[0] # First value in the plot (first timestep)\n", - " smoothed = list()\n", - " for point in data:\n", - " smoothed_val = last * weight + (1 - weight) * point # 计算平滑值\n", - " smoothed.append(smoothed_val) \n", - " last = smoothed_val \n", - " return smoothed\n", - "\n", - "def plot_rewards(rewards,cfg, tag='train'):\n", - " sns.set()\n", - " plt.figure() # 创建一个图形实例，方便同时多画几个图\n", - " plt.title(f\"{tag}ing curve on {cfg.device} of {cfg.algo_name} for {cfg.env_name}\")\n", - " plt.xlabel('epsiodes')\n", - " plt.plot(rewards, label='rewards')\n", - " plt.plot(smooth(rewards), label='smoothed')\n", - " plt.legend()\n", - " plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 5、我准备好了！\n", - "仔细看，会发现Sarsa收敛得快一些，但是收敛之会低些，Q learning会相反，至于为什么请同学们自行思考哟~" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "状态数：48，动作数：4\n", - "开始训练！\n", - "环境:CliffWalking-v0, 算法:Sarsa, 设备:cpu\n", - "回合：1/400，奖励：-1524.0，Epsilon：0.2029722781251147\n", - "回合：2/400，奖励：-1294.0，Epsilon：0.011808588201828951\n", - "回合：3/400，奖励：-192.0，Epsilon：0.01050118158853445\n", - "回合：4/400，奖励：-346.0，Epsilon：0.010049747911736582\n", - "回合：5/400，奖励：-252.0，Epsilon：0.010009240861841986\n", - "回合：6/400，奖励：-168.0，Epsilon：0.010003005072880926\n", - "回合：7/400，奖励：-393.0，Epsilon：0.01000042188120369\n", - "回合：8/400，奖励：-169.0，Epsilon：0.010000136281659052\n", - "回合：9/400，奖励：-97.0，Epsilon：0.010000071145264558\n", - "回合：10/400，奖励：-134.0，Epsilon：0.010000029022085234\n", - "回合：11/400，奖励：-124.0，Epsilon：0.010000012655059554\n", - "回合：12/400，奖励：-74.0，Epsilon：0.010000007701309915\n", - "回合：13/400，奖励：-135.0，Epsilon：0.010000003120699265\n", - "回合：14/400，奖励：-84.0，Epsilon：0.010000001776639691\n", - "回合：15/400，奖励：-101.0，Epsilon：0.010000000903081117\n", - "回合：16/400，奖励：-111.0，Epsilon：0.010000000429438717\n", - "回合：17/400，奖励：-114.0，Epsilon：0.010000000200165738\n", - "回合：18/400，奖励：-114.0，Epsilon：0.010000000093299278\n", - "回合：19/400，奖励：-82.0，Epsilon：0.010000000053829002\n", - "回合：20/400，奖励：-85.0，Epsilon：0.01000000003044167\n", - "回合：21/400，奖励：-108.0，Epsilon：0.010000000014768242\n", - "回合：22/400，奖励：-66.0，Epsilon：0.010000000009479634\n", - "回合：23/400，奖励：-74.0，Epsilon：0.010000000005768887\n", - "回合：24/400，奖励：-114.0，Epsilon：0.010000000002688936\n", - "回合：25/400，奖励：-98.0，Epsilon：0.010000000001394421\n", - "回合：26/400，奖励：-94.0，Epsilon：0.010000000000742658\n", - "回合：27/400，奖励：-58.0，Epsilon：0.010000000000502822\n", - "回合：28/400，奖励：-100.0，Epsilon：0.010000000000257298\n", - "回合：29/400，奖励：-208.0，Epsilon：0.010000000000123995\n", - "回合：30/400，奖励：-184.0，Epsilon：0.010000000000070121\n", - "回合：31/400，奖励：-62.0，Epsilon：0.010000000000046227\n", - "回合：32/400，奖励：-117.0，Epsilon：0.01000000000002112\n", - "回合：33/400，奖励：-47.0，Epsilon：0.010000000000015387\n", - "回合：34/400，奖励：-54.0，Epsilon：0.0100000000000107\n", - "回合：35/400，奖励：-120.0，Epsilon：0.010000000000004792\n", - "回合：36/400，奖励：-75.0，Epsilon：0.010000000000002897\n", - "回合：37/400，奖励：-62.0，Epsilon：0.01000000000000191\n", - "回合：38/400，奖励：-70.0，Epsilon：0.010000000000001194\n", - "回合：39/400，奖励：-67.0，Epsilon：0.010000000000000762\n", - "回合：40/400，奖励：-87.0，Epsilon：0.010000000000000425\n", - "回合：41/400，奖励：-92.0，Epsilon：0.01000000000000023\n", - "回合：42/400，奖励：-79.0，Epsilon：0.010000000000000136\n", - "回合：43/400，奖励：-49.0，Epsilon：0.010000000000000097\n", - "回合：44/400，奖励：-103.0，Epsilon：0.010000000000000049\n", - "回合：45/400，奖励：-40.0，Epsilon：0.010000000000000037\n", - "回合：46/400，奖励：-214.0，Epsilon：0.010000000000000018\n", - "回合：47/400，奖励：-83.0，Epsilon：0.01000000000000001\n", - "回合：48/400，奖励：-62.0，Epsilon：0.010000000000000007\n", - "回合：49/400，奖励：-37.0，Epsilon：0.010000000000000005\n", - "回合：50/400，奖励：-73.0，Epsilon：0.010000000000000004\n", - "回合：51/400，奖励：-66.0，Epsilon：0.010000000000000002\n", - "回合：52/400，奖励：-48.0，Epsilon：0.010000000000000002\n", - "回合：53/400，奖励：-96.0，Epsilon：0.01\n", - "回合：54/400，奖励：-189.0，Epsilon：0.01\n", - "回合：55/400，奖励：-42.0，Epsilon：0.01\n", - "回合：56/400，奖励：-46.0，Epsilon：0.01\n", - "回合：57/400，奖励：-85.0，Epsilon：0.01\n", - "回合：58/400，奖励：-52.0，Epsilon：0.01\n", - "回合：59/400，奖励：-86.0，Epsilon：0.01\n", - "回合：60/400，奖励：-41.0，Epsilon：0.01\n", - "回合：61/400，奖励：-51.0，Epsilon：0.01\n", - "回合：62/400，奖励：-59.0，Epsilon：0.01\n", - "回合：63/400，奖励：-145.0，Epsilon：0.01\n", - "回合：64/400，奖励：-76.0，Epsilon：0.01\n", - "回合：65/400，奖励：-43.0，Epsilon：0.01\n", - "回合：66/400，奖励：-49.0，Epsilon：0.01\n", - "回合：67/400，奖励：-36.0，Epsilon：0.01\n", - "回合：68/400，奖励：-41.0，Epsilon：0.01\n", - "回合：69/400，奖励：-69.0，Epsilon：0.01\n", - "回合：70/400，奖励：-38.0，Epsilon：0.01\n", - "回合：71/400，奖励：-63.0，Epsilon：0.01\n", - "回合：72/400，奖励：-46.0，Epsilon：0.01\n", - "回合：73/400，奖励：-30.0，Epsilon：0.01\n", - "回合：74/400，奖励：-45.0，Epsilon：0.01\n", - "回合：75/400，奖励：-38.0，Epsilon：0.01\n", - "回合：76/400，奖励：-88.0，Epsilon：0.01\n", - "回合：77/400，奖励：-19.0，Epsilon：0.01\n", - "回合：78/400，奖励：-40.0，Epsilon：0.01\n", - "回合：79/400，奖励：-62.0，Epsilon：0.01\n", - "回合：80/400，奖励：-25.0，Epsilon：0.01\n", - "回合：81/400，奖励：-54.0，Epsilon：0.01\n", - "回合：82/400，奖励：-41.0，Epsilon：0.01\n", - "回合：83/400，奖励：-57.0，Epsilon：0.01\n", - "回合：84/400，奖励：-52.0，Epsilon：0.01\n", - "回合：85/400，奖励：-42.0，Epsilon：0.01\n", - "回合：86/400，奖励：-51.0，Epsilon：0.01\n", - "回合：87/400，奖励：-53.0，Epsilon：0.01\n", - "回合：88/400，奖励：-42.0，Epsilon：0.01\n", - "回合：89/400，奖励：-53.0，Epsilon：0.01\n", - "回合：90/400，奖励：-31.0，Epsilon：0.01\n", - "回合：91/400，奖励：-75.0，Epsilon：0.01\n", - "回合：92/400，奖励：-148.0，Epsilon：0.01\n", - "回合：93/400，奖励：-41.0，Epsilon：0.01\n", - "回合：94/400，奖励：-47.0，Epsilon：0.01\n", - "回合：95/400，奖励：-184.0，Epsilon：0.01\n", - "回合：96/400，奖励：-34.0，Epsilon：0.01\n", - "回合：97/400，奖励：-45.0，Epsilon：0.01\n", - "回合：98/400，奖励：-52.0，Epsilon：0.01\n", - "回合：99/400，奖励：-44.0，Epsilon：0.01\n", - "回合：100/400，奖励：-49.0，Epsilon：0.01\n", - "回合：101/400，奖励：-30.0，Epsilon：0.01\n", - "回合：102/400，奖励：-49.0，Epsilon：0.01\n", - "回合：103/400，奖励：-23.0，Epsilon：0.01\n", - "回合：104/400，奖励：-37.0，Epsilon：0.01\n", - "回合：105/400，奖励：-37.0，Epsilon：0.01\n", - "回合：106/400，奖励：-44.0，Epsilon：0.01\n", - "回合：107/400，奖励：-40.0，Epsilon：0.01\n", - "回合：108/400，奖励：-28.0，Epsilon：0.01\n", - "回合：109/400，奖励：-50.0，Epsilon：0.01\n", - "回合：110/400，奖励：-46.0，Epsilon：0.01\n", - "回合：111/400，奖励：-28.0，Epsilon：0.01\n", - "回合：112/400，奖励：-35.0，Epsilon：0.01\n", - "回合：113/400，奖励：-35.0，Epsilon：0.01\n", - "回合：114/400，奖励：-45.0，Epsilon：0.01\n", - "回合：115/400，奖励：-38.0，Epsilon：0.01\n", - "回合：116/400，奖励：-39.0，Epsilon：0.01\n", - "回合：117/400，奖励：-27.0，Epsilon：0.01\n", - "回合：118/400，奖励：-49.0，Epsilon：0.01\n", - "回合：119/400，奖励：-27.0，Epsilon：0.01\n", - "回合：120/400，奖励：-25.0，Epsilon：0.01\n", - "回合：121/400，奖励：-50.0，Epsilon：0.01\n", - "回合：122/400，奖励：-41.0，Epsilon：0.01\n", - "回合：123/400，奖励：-22.0，Epsilon：0.01\n", - "回合：124/400，奖励：-38.0，Epsilon：0.01\n", - "回合：125/400，奖励：-125.0，Epsilon：0.01\n", - "回合：126/400，奖励：-25.0，Epsilon：0.01\n", - "回合：127/400，奖励：-40.0，Epsilon：0.01\n", - "回合：128/400，奖励：-33.0，Epsilon：0.01\n", - "回合：129/400，奖励：-56.0，Epsilon：0.01\n", - "回合：130/400，奖励：-32.0，Epsilon：0.01\n", - "回合：131/400，奖励：-21.0，Epsilon：0.01\n", - "回合：132/400，奖励：-33.0，Epsilon：0.01\n", - "回合：133/400，奖励：-23.0，Epsilon：0.01\n", - "回合：134/400，奖励：-33.0，Epsilon：0.01\n", - "回合：135/400，奖励：-34.0，Epsilon：0.01\n", - "回合：136/400，奖励：-33.0，Epsilon：0.01\n", - "回合：137/400，奖励：-21.0，Epsilon：0.01\n", - "回合：138/400，奖励：-40.0，Epsilon：0.01\n", - "回合：139/400，奖励：-23.0，Epsilon：0.01\n", - "回合：140/400，奖励：-31.0，Epsilon：0.01\n", - "回合：141/400，奖励：-31.0，Epsilon：0.01\n", - "回合：142/400，奖励：-26.0，Epsilon：0.01\n", - "回合：143/400，奖励：-26.0，Epsilon：0.01\n", - "回合：144/400，奖励：-32.0，Epsilon：0.01\n", - "回合：145/400，奖励：-27.0，Epsilon：0.01\n", - "回合：146/400，奖励：-33.0，Epsilon：0.01\n", - "回合：147/400，奖励：-35.0，Epsilon：0.01\n", - "回合：148/400，奖励：-21.0，Epsilon：0.01\n", - "回合：149/400，奖励：-23.0，Epsilon：0.01\n", - "回合：150/400，奖励：-33.0，Epsilon：0.01\n", - "回合：151/400，奖励：-25.0，Epsilon：0.01\n", - "回合：152/400，奖励：-41.0，Epsilon：0.01\n", - "回合：153/400，奖励：-31.0，Epsilon：0.01\n", - "回合：154/400，奖励：-28.0，Epsilon：0.01\n", - "回合：155/400，奖励：-133.0，Epsilon：0.01\n", - "回合：156/400，奖励：-22.0，Epsilon：0.01\n", - "回合：157/400，奖励：-21.0，Epsilon：0.01\n", - "回合：158/400，奖励：-33.0，Epsilon：0.01\n", - "回合：159/400，奖励：-33.0，Epsilon：0.01\n", - "回合：160/400，奖励：-24.0，Epsilon：0.01\n", - "回合：161/400，奖励：-34.0，Epsilon：0.01\n", - "回合：162/400，奖励：-20.0，Epsilon：0.01\n", - "回合：163/400，奖励：-21.0，Epsilon：0.01\n", - "回合：164/400，奖励：-126.0，Epsilon：0.01\n", - "回合：165/400，奖励：-36.0，Epsilon：0.01\n", - "回合：166/400，奖励：-18.0，Epsilon：0.01\n", - "回合：167/400，奖励：-35.0，Epsilon：0.01\n", - "回合：168/400，奖励：-26.0，Epsilon：0.01\n", - "回合：169/400，奖励：-24.0，Epsilon：0.01\n", - "回合：170/400，奖励：-33.0，Epsilon：0.01\n", - "回合：171/400，奖励：-17.0，Epsilon：0.01\n", - "回合：172/400，奖励：-23.0，Epsilon：0.01\n", - "回合：173/400，奖励：-26.0，Epsilon：0.01\n", - "回合：174/400，奖励：-23.0，Epsilon：0.01\n", - "回合：175/400，奖励：-21.0，Epsilon：0.01\n", - "回合：176/400，奖励：-35.0，Epsilon：0.01\n", - "回合：177/400，奖励：-26.0，Epsilon：0.01\n", - "回合：178/400，奖励：-17.0，Epsilon：0.01\n", - "回合：179/400，奖励：-20.0，Epsilon：0.01\n", - "回合：180/400，奖励：-28.0，Epsilon：0.01\n", - "回合：181/400，奖励：-34.0，Epsilon：0.01\n", - "回合：182/400，奖励：-27.0，Epsilon：0.01\n", - "回合：183/400，奖励：-22.0，Epsilon：0.01\n", - "回合：184/400，奖励：-24.0，Epsilon：0.01\n", - "回合：185/400，奖励：-26.0，Epsilon：0.01\n", - "回合：186/400，奖励：-20.0，Epsilon：0.01\n", - "回合：187/400，奖励：-30.0，Epsilon：0.01\n", - "回合：188/400，奖励：-28.0，Epsilon：0.01\n", - "回合：189/400，奖励：-15.0，Epsilon：0.01\n", - "回合：190/400，奖励：-30.0，Epsilon：0.01\n", - "回合：191/400，奖励：-29.0，Epsilon：0.01\n", - "回合：192/400，奖励：-22.0，Epsilon：0.01\n", - "回合：193/400，奖励：-25.0，Epsilon：0.01\n", - "回合：194/400，奖励：-21.0，Epsilon：0.01\n", - "回合：195/400，奖励：-19.0，Epsilon：0.01\n", - "回合：196/400，奖励：-23.0，Epsilon：0.01\n", - "回合：197/400，奖励：-21.0，Epsilon：0.01\n", - "回合：198/400，奖励：-32.0，Epsilon：0.01\n", - "回合：199/400，奖励：-30.0，Epsilon：0.01\n", - "回合：200/400，奖励：-22.0，Epsilon：0.01\n", - "回合：201/400，奖励：-20.0，Epsilon：0.01\n", - "回合：202/400，奖励：-27.0，Epsilon：0.01\n", - "回合：203/400，奖励：-21.0，Epsilon：0.01\n", - "回合：204/400，奖励：-26.0，Epsilon：0.01\n", - "回合：205/400，奖励：-19.0，Epsilon：0.01\n", - "回合：206/400，奖励：-17.0，Epsilon：0.01\n", - "回合：207/400，奖励：-31.0，Epsilon：0.01\n", - "回合：208/400，奖励：-18.0，Epsilon：0.01\n", - "回合：209/400，奖励：-24.0，Epsilon：0.01\n", - "回合：210/400，奖励：-17.0，Epsilon：0.01\n", - "回合：211/400，奖励：-26.0，Epsilon：0.01\n", - "回合：212/400，奖励：-27.0，Epsilon：0.01\n", - "回合：213/400，奖励：-33.0，Epsilon：0.01\n", - "回合：214/400，奖励：-16.0，Epsilon：0.01\n", - "回合：215/400，奖励：-32.0，Epsilon：0.01\n", - "回合：216/400，奖励：-19.0，Epsilon：0.01\n", - "回合：217/400，奖励：-20.0，Epsilon：0.01\n", - "回合：218/400，奖励：-15.0，Epsilon：0.01\n", - "回合：219/400，奖励：-119.0，Epsilon：0.01\n", - "回合：220/400，奖励：-26.0，Epsilon：0.01\n", - "回合：221/400，奖励：-26.0，Epsilon：0.01\n", - "回合：222/400，奖励：-22.0，Epsilon：0.01\n", - "回合：223/400，奖励：-22.0，Epsilon：0.01\n", - "回合：224/400，奖励：-15.0，Epsilon：0.01\n", - "回合：225/400，奖励：-24.0，Epsilon：0.01\n", - "回合：226/400，奖励：-15.0，Epsilon：0.01\n", - "回合：227/400，奖励：-31.0，Epsilon：0.01\n", - "回合：228/400，奖励：-24.0，Epsilon：0.01\n", - "回合：229/400，奖励：-20.0，Epsilon：0.01\n", - "回合：230/400，奖励：-20.0，Epsilon：0.01\n", - "回合：231/400，奖励：-22.0，Epsilon：0.01\n", - "回合：232/400，奖励：-15.0，Epsilon：0.01\n", - "回合：233/400，奖励：-19.0，Epsilon：0.01\n", - "回合：234/400，奖励：-21.0，Epsilon：0.01\n", - "回合：235/400，奖励：-27.0，Epsilon：0.01\n", - "回合：236/400，奖励：-15.0，Epsilon：0.01\n", - "回合：237/400，奖励：-25.0，Epsilon：0.01\n", - "回合：238/400，奖励：-22.0，Epsilon：0.01\n", - "回合：239/400，奖励：-16.0，Epsilon：0.01\n", - "回合：240/400，奖励：-18.0，Epsilon：0.01\n", - "回合：241/400，奖励：-13.0，Epsilon：0.01\n", - "回合：242/400，奖励：-13.0，Epsilon：0.01\n", - "回合：243/400，奖励：-13.0，Epsilon：0.01\n", - "回合：244/400，奖励：-23.0，Epsilon：0.01\n", - "回合：245/400，奖励：-29.0，Epsilon：0.01\n", - "回合：246/400，奖励：-26.0，Epsilon：0.01\n", - "回合：247/400，奖励：-19.0，Epsilon：0.01\n", - "回合：248/400，奖励：-21.0，Epsilon：0.01\n", - "回合：249/400，奖励：-17.0，Epsilon：0.01\n", - "回合：250/400，奖励：-17.0，Epsilon：0.01\n", - "回合：251/400，奖励：-15.0，Epsilon：0.01\n", - "回合：252/400，奖励：-20.0，Epsilon：0.01\n", - "回合：253/400，奖励：-23.0，Epsilon：0.01\n", - "回合：254/400，奖励：-19.0，Epsilon：0.01\n", - "回合：255/400，奖励：-21.0，Epsilon：0.01\n", - "回合：256/400，奖励：-19.0，Epsilon：0.01\n", - "回合：257/400，奖励：-17.0，Epsilon：0.01\n", - "回合：258/400，奖励：-17.0，Epsilon：0.01\n", - "回合：259/400，奖励：-15.0，Epsilon：0.01\n", - "回合：260/400，奖励：-21.0，Epsilon：0.01\n", - "回合：261/400，奖励：-17.0，Epsilon：0.01\n", - "回合：262/400，奖励：-19.0，Epsilon：0.01\n", - "回合：263/400，奖励：-19.0，Epsilon：0.01\n", - "回合：264/400，奖励：-15.0，Epsilon：0.01\n", - "回合：265/400，奖励：-19.0，Epsilon：0.01\n", - "回合：266/400，奖励：-17.0，Epsilon：0.01\n", - "回合：267/400，奖励：-15.0，Epsilon：0.01\n", - "回合：268/400，奖励：-19.0，Epsilon：0.01\n", - "回合：269/400，奖励：-27.0，Epsilon：0.01\n", - "回合：270/400，奖励：-15.0，Epsilon：0.01\n", - "回合：271/400，奖励：-17.0，Epsilon：0.01\n", - "回合：272/400，奖励：-17.0，Epsilon：0.01\n", - "回合：273/400，奖励：-25.0，Epsilon：0.01\n", - "回合：274/400，奖励：-19.0，Epsilon：0.01\n", - "回合：275/400，奖励：-22.0，Epsilon：0.01\n", - "回合：276/400，奖励：-23.0，Epsilon：0.01\n", - "回合：277/400，奖励：-18.0，Epsilon：0.01\n", - "回合：278/400，奖励：-23.0，Epsilon：0.01\n", - "回合：279/400，奖励：-21.0，Epsilon：0.01\n", - "回合：280/400，奖励：-21.0，Epsilon：0.01\n", - "回合：281/400，奖励：-21.0，Epsilon：0.01\n", - "回合：282/400，奖励：-19.0，Epsilon：0.01\n", - "回合：283/400，奖励：-18.0，Epsilon：0.01\n", - "回合：284/400，奖励：-15.0，Epsilon：0.01\n", - "回合：285/400，奖励：-19.0，Epsilon：0.01\n", - "回合：286/400，奖励：-19.0，Epsilon：0.01\n", - "回合：287/400，奖励：-21.0，Epsilon：0.01\n", - "回合：288/400，奖励：-15.0，Epsilon：0.01\n", - "回合：289/400，奖励：-32.0，Epsilon：0.01\n", - "回合：290/400，奖励：-18.0，Epsilon：0.01\n", - "回合：291/400，奖励：-17.0，Epsilon：0.01\n", - "回合：292/400，奖励：-15.0，Epsilon：0.01\n", - "回合：293/400，奖励：-24.0，Epsilon：0.01\n", - "回合：294/400，奖励：-22.0，Epsilon：0.01\n", - "回合：295/400，奖励：-31.0，Epsilon：0.01\n", - "回合：296/400，奖励：-17.0，Epsilon：0.01\n", - "回合：297/400，奖励：-19.0，Epsilon：0.01\n", - "回合：298/400，奖励：-19.0，Epsilon：0.01\n", - "回合：299/400，奖励：-20.0，Epsilon：0.01\n", - "回合：300/400，奖励：-21.0，Epsilon：0.01\n", - "回合：301/400，奖励：-26.0，Epsilon：0.01\n", - "回合：302/400，奖励：-20.0，Epsilon：0.01\n", - "回合：303/400，奖励：-16.0，Epsilon：0.01\n", - "回合：304/400，奖励：-20.0，Epsilon：0.01\n", - "回合：305/400，奖励：-21.0，Epsilon：0.01\n", - "回合：306/400，奖励：-16.0，Epsilon：0.01\n", - "回合：307/400，奖励：-19.0，Epsilon：0.01\n", - "回合：308/400，奖励：-24.0，Epsilon：0.01\n", - "回合：309/400，奖励：-20.0，Epsilon：0.01\n", - "回合：310/400，奖励：-17.0，Epsilon：0.01\n", - "回合：311/400，奖励：-16.0，Epsilon：0.01\n", - "回合：312/400，奖励：-25.0，Epsilon：0.01\n", - "回合：313/400，奖励：-16.0，Epsilon：0.01\n", - "回合：314/400，奖励：-19.0，Epsilon：0.01\n", - "回合：315/400，奖励：-19.0，Epsilon：0.01\n", - "回合：316/400，奖励：-27.0，Epsilon：0.01\n", - "回合：317/400，奖励：-15.0，Epsilon：0.01\n", - "回合：318/400，奖励：-15.0，Epsilon：0.01\n", - "回合：319/400，奖励：-15.0，Epsilon：0.01\n", - "回合：320/400，奖励：-19.0，Epsilon：0.01\n", - "回合：321/400，奖励：-23.0，Epsilon：0.01\n", - "回合：322/400，奖励：-24.0，Epsilon：0.01\n", - "回合：323/400，奖励：-15.0，Epsilon：0.01\n", - "回合：324/400，奖励：-20.0，Epsilon：0.01\n", - "回合：325/400，奖励：-18.0，Epsilon：0.01\n", - "回合：326/400，奖励：-19.0，Epsilon：0.01\n", - "回合：327/400，奖励：-19.0，Epsilon：0.01\n", - "回合：328/400，奖励：-26.0，Epsilon：0.01\n", - "回合：329/400，奖励：-16.0，Epsilon：0.01\n", - "回合：330/400，奖励：-18.0，Epsilon：0.01\n", - "回合：331/400，奖励：-15.0，Epsilon：0.01\n", - "回合：332/400，奖励：-15.0，Epsilon：0.01\n", - "回合：333/400，奖励：-17.0，Epsilon：0.01\n", - "回合：334/400，奖励：-17.0，Epsilon：0.01\n", - "回合：335/400，奖励：-16.0，Epsilon：0.01\n", - "回合：336/400，奖励：-24.0，Epsilon：0.01\n", - "回合：337/400，奖励：-15.0，Epsilon：0.01\n", - "回合：338/400，奖励：-18.0，Epsilon：0.01\n", - "回合：339/400，奖励：-16.0，Epsilon：0.01\n", - "回合：340/400，奖励：-15.0，Epsilon：0.01\n", - "回合：341/400，奖励：-18.0，Epsilon：0.01\n", - "回合：342/400，奖励：-15.0，Epsilon：0.01\n", - "回合：343/400，奖励：-20.0，Epsilon：0.01\n", - "回合：344/400，奖励：-18.0，Epsilon：0.01\n", - "回合：345/400，奖励：-17.0，Epsilon：0.01\n", - "回合：346/400，奖励：-19.0，Epsilon：0.01\n", - "回合：347/400，奖励：-15.0，Epsilon：0.01\n", - "回合：348/400，奖励：-15.0，Epsilon：0.01\n", - "回合：349/400，奖励：-15.0，Epsilon：0.01\n", - "回合：350/400，奖励：-18.0，Epsilon：0.01\n", - "回合：351/400，奖励：-16.0，Epsilon：0.01\n", - "回合：352/400，奖励：-16.0，Epsilon：0.01\n", - "回合：353/400，奖励：-15.0，Epsilon：0.01\n", - "回合：354/400，奖励：-20.0，Epsilon：0.01\n", - "回合：355/400，奖励：-15.0，Epsilon：0.01\n", - "回合：356/400，奖励：-17.0，Epsilon：0.01\n", - "回合：357/400，奖励：-15.0，Epsilon：0.01\n", - "回合：358/400，奖励：-17.0，Epsilon：0.01\n", - "回合：359/400，奖励：-15.0，Epsilon：0.01\n", - "回合：360/400，奖励：-16.0，Epsilon：0.01\n", - "回合：361/400，奖励：-15.0，Epsilon：0.01\n", - "回合：362/400，奖励：-18.0，Epsilon：0.01\n", - "回合：363/400，奖励：-17.0，Epsilon：0.01\n", - "回合：364/400，奖励：-22.0，Epsilon：0.01\n", - "回合：365/400，奖励：-15.0，Epsilon：0.01\n", - "回合：366/400，奖励：-15.0，Epsilon：0.01\n", - "回合：367/400，奖励：-15.0，Epsilon：0.01\n", - "回合：368/400，奖励：-16.0，Epsilon：0.01\n", - "回合：369/400，奖励：-16.0，Epsilon：0.01\n", - "回合：370/400，奖励：-15.0，Epsilon：0.01\n", - "回合：371/400，奖励：-20.0，Epsilon：0.01\n", - "回合：372/400，奖励：-15.0，Epsilon：0.01\n", - "回合：373/400，奖励：-15.0，Epsilon：0.01\n", - "回合：374/400，奖励：-15.0，Epsilon：0.01\n", - "回合：375/400，奖励：-16.0，Epsilon：0.01\n", - "回合：376/400，奖励：-15.0，Epsilon：0.01\n", - "回合：377/400，奖励：-15.0，Epsilon：0.01\n", - "回合：378/400，奖励：-17.0，Epsilon：0.01\n", - "回合：379/400，奖励：-20.0，Epsilon：0.01\n", - "回合：380/400，奖励：-17.0，Epsilon：0.01\n", - "回合：381/400，奖励：-15.0，Epsilon：0.01\n", - "回合：382/400，奖励：-15.0，Epsilon：0.01\n", - "回合：383/400，奖励：-15.0，Epsilon：0.01\n", - "回合：384/400，奖励：-15.0，Epsilon：0.01\n", - "回合：385/400，奖励：-16.0，Epsilon：0.01\n", - "回合：386/400，奖励：-15.0，Epsilon：0.01\n", - "回合：387/400，奖励：-18.0，Epsilon：0.01\n", - "回合：388/400，奖励：-15.0，Epsilon：0.01\n", - "回合：389/400，奖励：-15.0，Epsilon：0.01\n", - "回合：390/400，奖励：-15.0，Epsilon：0.01\n", - "回合：391/400，奖励：-16.0，Epsilon：0.01\n", - "回合：392/400，奖励：-18.0，Epsilon：0.01\n", - "回合：393/400，奖励：-15.0，Epsilon：0.01\n", - "回合：394/400，奖励：-15.0，Epsilon：0.01\n", - "回合：395/400，奖励：-15.0，Epsilon：0.01\n", - "回合：396/400，奖励：-20.0，Epsilon：0.01\n", - "回合：397/400，奖励：-15.0，Epsilon：0.01\n", - "回合：398/400，奖励：-15.0，Epsilon：0.01\n", - "回合：399/400，奖励：-15.0，Epsilon：0.01\n", - "回合：400/400，奖励：-15.0，Epsilon：0.01\n", - "完成训练！\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEXCAYAAABCjVgAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABUmElEQVR4nO3dd3hUZdrA4d+Zmk4KSSCUANIFZGlSFAQREkJVWFGK6C64rgXd/VgiKrIoYlhXxe5aV0Sxg7hEUBQUKQKiqBQxktBSJr1n2vv9MWRIyASSmAbz3NflZfLmnDPPHGbOc956NKWUQgghhAB0TR2AEEKI5kOSghBCCDdJCkIIIdwkKQghhHCTpCCEEMJNkoIQQgg3SQoN6JZbbiE7O7tW+/z444/cdddd591u0qRJ5Ofn1zU0UU+2bdvGyJEjue666ygtLa30t6SkJObNm8eECROYMGECM2fOZM+ePY0eY2FhIdOnTycuLo6NGzf+rmN99NFHXH/99UyaNIlx48bxwAMPuD+HTz/9NEuXLgVg7ty5/PrrrwA8+OCDjBo1iieeeILnn3+eq666ilmzZjFo0CCcTqf72H//+9/p1asXhYWF7rJ//vOfrFixotp4du3axfjx4wGIj4/nlVdeqbLNypUrWbt27e963zVRUlLC3//+d2JjYxk7diyff/55g79mg1CiwXTt2lVlZWU1dRiiAcXHx6tnn33W49/GjRunNm3a5P7922+/Vf369VM5OTmNFN2Z1x09evTvPs7zzz+vbrjhBmWxWJRSSlmtVrVkyRJ1ww03KKWUeuqpp9Q///nPKvt169ZNpaamKqWUGjVqlNq9e7dyOBxq0KBB6sCBA0oppWw2mxo6dKi65ZZb1IYNG9z7jh07Vu3atavamHbu3Kni4uKUUkotXLhQvfzyy7/7fdZVQkKCuv/++5VSSp08eVINGzbM/b4vJIamTkoXq3vvvReAm266if/85z/MmDGDPn36cPjwYf72t79hMBh48cUXsVqtZGdnM3nyZO6++2527drFQw89xCeffEJ8fDwBAQEcPnyYtLQ0OnXqxOOPP46/vz/dunVjx44dbNmyhc8++wydTkdKSgpGo5GEhAS6du1KSkoKixYtIi8vj/DwcJRSTJw4kWuvvbZSrBaLhQcffJDffvsNnU7H9OnTmT17NrNmzWLGjBnExMQAVPq9V69eXH311Rw6dIipU6eyZ88eXnzxRcB1hzxnzhy2bNlCcnIyy5YtIzc3F4fDwaxZs5g6dWqV83XkyBGWLl1Kbm4umqZxyy23MHnyZHbt2sUTTzxBu3btOHLkCFarlcWLFzN48OAqx3j//fd57bXX0Ol0hISEkJCQwLFjx1ixYgWRkZEcP34cHx8fHn30US655BLi4+Pp0qULf/rTnwCq/F7OZrPx6KOPsmPHDvR6PX369OHee+9lzZo1bN68GbPZTEFBAQsXLqxyXouLi92/Dxw4kCeffBK9Xg/ACy+8wOeff05ZWRklJSUsXLiQa665hqeffprvv/+ejIwMunXrxm233cZ9992H1WpFKcXUqVOZMWMGmZmZLF68mKysLCwWC23atOHJJ58kLCzM/Zq//fYbixYtIj09nUmTJvHOO++wbds2nnnmGRwOBwEBAdx777306dOnyus+9thj7uMUFxfz4osv8tFHH9GyZUsAjEYj//jHP/jss8+wWq2V3vuoUaNYuXIly5cvRynF3LlzCQ0NJT09nfvuu4/58+dzxRVXsGvXLnr06MHevXvp1q0bMTExfPHFF8TGxpKenk5WVhb9+vXjyy+/9Ph9qc7y5cs5dOgQzz33HA899JD737V3797MmzePb775hoyMDGbPns2cOXNwOBysWLGCL774gsDAQPr06UNSUhKrVq2qdNyjR48yffp0vv76a0wmEw6Hg5EjR/Lqq6/y+eefu89ZVFQUV1xxBYmJidx8883VxtksNXFSuqhVrCmMHDlSPfPMM0oppZxOp5o5c6Y6evSoUkqptLQ01aNHD5WVlVXlzuf6669XZWVlymq1qsmTJ6v333+/0rE/+OAD1b9/f/cdydKlS9U//vEPpZRSf/zjH9Xq1auVUkr9+uuv6rLLLlMffPBBlThvv/12lZCQoJRSKj8/X8XFxank5GQ1c+ZMlZiY6N6u4u9du3ZVH330kVJKqYKCAtW/f3+VkZGhlFJqxYoV6vHHH1c2m02NGzdO/fTTT+5jx8bGqn379lV6fZvNpq6++mq1ceNG9/m48sor1Xfffad27typevTo4b6jfOWVV9SMGTOqvIeDBw+qyy+/XJ06dUoppdRrr72mHnjgAbVz507VvXt3tXv3bqWUUm+99ZaaMmWK+/xWvLOs7k5z5cqV6o477lBWq1U5HA4VHx+vHnjggXPuo5RS69evVwMGDFDDhg1Td911l1q1apW7lnDixAk1a9YsVVJSopRS6pNPPlHjx49XSrnuuMeOHatsNptSSql7771Xvfjii0oppTIyMtTdd9+tHA6Hev31193lTqdT/fnPf1avvPJKlTgqfqZ+/fVXNXToUHXs2DGllFLbt29Xw4YNUwUFBVVet6Iff/xRDR482OP7LFexpjBy5Ei1f/9+pVTV70F5+dq1a9Vtt92mlFJq+fLl6r///a9KT09Xl19+ubLb7eqjjz5S8+fPr9X35aWXXlJLlixRt99+uyorK6vyb9S1a1e1atUq93vq1auXKi0tVW+//baaMWOGKi0tVWVlZeqWW25RM2fO9Pg+Z8yY4f4ebNmyRU2fPl0ppVSvXr3c3wGllHr88cfVI488cs5z1hxJn0IjGjBgAACapvHCCy/w888/88wzz/Doo4+ilKKkpKTKPldeeSUmkwmj0UjXrl3Jy8urss2ll15Kq1atAOjZsyd5eXnk5eWxf/9+pk2bBsAll1zi8e4aYPv27Vx//fUABAYG8sknnxAdHV3j9xMQEMDYsWP5+OOPcTgcfPzxx0ydOpXk5GSOHTvGokWLmDRpEjNnzqS0tJQDBw5UOk5ycjJlZWWMGTMGgMjISMaMGcPXX38NuO66evToUen9nW3Hjh1cccUVtG7dGoA5c+a427e7d+/ujvW6667j4MGD5OTknPf9lfvqq6+YPn06RqMRnU7HrFmz3LGdy/jx49m2bRsrVqygU6dOfPDBB8TFxXHixAnatGlDQkIC69ev57HHHmPNmjUUFRW59+3bty8Gg6sif8011/Dyyy9zxx13sGnTJu6//350Oh033XQT/fr147XXXmPJkiUcOXKkUs3Ek507dzJ48GDatWsHwJAhQwgNDeWnn36q8roV6XS6Su3/9eHKK69k7969OJ1OvvzyS0aOHElERARRUVH89NNP7Ny5k6uuuqpW35fXX3+dNWvWcOedd2IymTy+7tVXXw24vjdWq5Xi4mK2bt3KpEmTMJvNmEwm9/fBk2nTpvHRRx8B8OGHH7q/Y8rDikE63YV3ib3wIr6A+fn5Aa6q+JQpU/j555/p2bMn//jHPzAYDB4/VD4+Pu6fNU2r8TblTRQVty8vO5vBYEDTNPfvx48fd3f2VdzfZrN5fD/g+qKsXbuWr7/+ms6dO9OuXTscDgdBQUGsW7fO/d+7777LddddV+k4ni42SinsdnuNz4Fer6/0HkpLS0lKSvL4vsvPz9nHOvv9VRef0+msdttySUlJPPbYY5jNZoYOHcr8+fP56KOP6Nq1Kxs3buTnn39m+vTpFBYWMmzYMP785z9X2r/iuR05ciQbN24kNjaWgwcPMmHCBI4dO8a//vUvVq5cSUhICNdffz3Dhg3zeG7Ofu+eysrPdcXXrahz587Y7XZSUlIqlZeVlTF37lzS09PP+bqehIaG0q5dOzZt2oRer3cnqquuuoq9e/fy7bffMnz48Fp9XwYOHMiiRYu49957q/03MpvNAO7Pi1KqSiIsv5iXN7uV/5eenk5MTAw//PADSUlJ7N69m9jYWABat26NxWJxHyMjI8N9s3YhkaTQgPR6vfvLVlFKSgqFhYXcfffdjBo1im+//Rar1Vqvd2IBAQH069ePDz/8EHBd6Hfs2FHpwlluyJAhfPDBBwAUFBRw0003kZycXOkO8tixYxw+fLja1+vbty8Azz77rPvOqWPHjpjNZtatWwdAamoq48ePdx+zXMeOHTEajWzatAlwfRE3btzI0KFDa/x+L7/8cnbs2EFGRgYAa9as4V//+hcAhw4d4tChQwC888479OvXj6CgIEJCQtyxZGdnVzsy6Morr2TNmjXYbDacTierV69m2LBh54ynZcuWvPvuu3z66afustzcXDIzM+nZsye7d++mV69e3HzzzQwaNIjNmzfjcDg8Huvvf/87GzZsIC4ujgcffJCAgABSU1PZtm0bN910E5MnTyYsLIzt27dXe4xygwcP5ptvvuH48eOAq4aVmprKZZddds79TCYTc+fOZdGiRWRmZgJgtVp55JFHKCkpITIy8pz7V2f48OE899xzXHXVVe6yq666inXr1tGyZUtCQ0Nr9X3p1asXM2fOJDAwkGeeeabGcYwYMYKPP/4Yq9WK3W531wQiIyMr3dRERkZiNpuJi4sjPj6eMWPG4OvrC7hqIO+88w4AaWlpfP3114wcObJO56UpSUdzA7rmmmu48cYbee655yqVd+vWjauuuorY2FiCgoJo3749nTt3JiUlpdoqb10kJCRw33338dZbbxEZGUnbtm0r3XWXW7x4MUuWLGHChAkopbj11lvp1asXt912G/Hx8WzdupVOnTq5m2CqM23aNJ577jlGjx4NuC4kzz33HMuWLePll1/Gbrczf/58+vfvX2k/o9HIc889x8MPP8zTTz+Nw+Hg9ttvZ/DgwezatatG77Vbt24sWLDAfccdHh7OI488QnJyMi1btuTJJ5/k5MmThIaGuoc4zpo1i//7v/9j7NixtG3blkGDBnk89m233UZCQgKTJ0/GbrfTp08fHnjggXPG06JFC/773//y73//mxUrVuDr64vJZOJPf/oTQ4YMoUuXLmzatIlx48ZhNBoZMmQIeXl5lYZjlvvrX//KfffdxzvvvINer2f06NEMGjSI22+/nRUrVvDcc8+h1+vp168fx44dO2dcnTt35sEHH+SOO+7A4XDg4+PDCy+8QGBg4HnP8V/+8hd8fX3dHfFlZWUMGjSoyue7NoYPH86zzz5b6Xz27t2bzMxMbrzxRqD23xdN03jkkUeYPHkyI0aMqFEc1157LUePHmXy5Mn4+fnRtm1b98Xek2nTpvHmm2+yZMkSd9mdd97JkiVLiIuLw+FwsGDBAtq3b1/DM9F8aOp89U1xwXr++ecZM2YMl1xyCQUFBUycOJGXXnqJzp07N3VojabiaC4hqrNt2zaysrKYNGkSAA8//DBms5kFCxY0cWSNT2oKF7EOHTpwzz33oNPpcDgczJ0716sSghA11aVLF1555RVeeeUVHA4H3bt3r1QL8CZSUxBCCOEmHc1CCCHcJCkIIYRwk6QghBDCTZKCEEIIt4ti9FFOThFOZ+37y8PCAsjKqjouvKlJXLUjcdVec41N4qqdusal02mEhPh7/NtFkRScTlWnpFC+b3MkcdWOxFV7zTU2iat26jsuaT4SQgjh1mySwvr16xk3bhzXXHMNq1evbupwhBDCKzWL5qP09HSeeOIJPvzwQ0wmE9OnT+fyyy+X2bdCCNHImkVNYfv27QwePJjg4GD8/PwYO3ZspdUlhRBCNI5mkRQyMjIIDw93/x4REVGn9dmFEEL8Ps2i+cjT8kue1v2vTlhYQJ1fOzz8/EsGNwWJq3YkrjOUUpR/pXS66r9Hcs5qx1viahZJITIystIDTjIyMoiIiKjx/llZhXUalhUeHojFUlDr/RpafcZVWGLDqNdhNnl+6lptXKjny2Z3oNfp0Ok0UtIKCA/2wc/HiFMp9hzKoGeHUAJ8je7tD6bk4HA6ubRDaKWbE6UUTqU4kVHE5u9OMHtsNwx6V2U7p6AMP7Oh0nkOCvYjP9f1eMzDx3LILbTSNtwfnU7jWHohg3pEkF9s48vvTjCibxt+O5XHnsMWZo/thq/Z9dX8fM9xCktspOeUMHXEJQT5Gym1OtiflMVvqflMGtaR4xmFHDqWw6h+bdmwM4VDx3LIyswhwmxl1rhedIqOJKPASW6xncPJmeRk5xIzqA1+ZgNOp8YJSwEn0vNJzyqgQ4Q/vkYNX5PGT0kWOkT4kZZVSFSoD74mjZRTuUS08KFrxwiUwYy/vy/JllI6tQ1DZzRSZnVw9FQePydl0LqFno6RfvyUlElUdFsKrRq/HMuhW7sgLr80iiMZNnILSggPNBDRwoSf2UBGfim7fzzFwO4t2fVzKpd2CCEq1IfdB9IoKLESFmhCOZ10ah3I4ZRsfj2Ri8mgcUWvSEICTHy59zjtI/0J8NET4KOntMxG8qk88orKwOkkItgHq81BoJ8RS24JZoNGcUkZAT569CjCg82UllrJLyihpNSKSQ8BPjpQTmxWG74+RspsTux2B0a95nqokXIS7G/C16wnI6cE2+m/6XU6lFKYjDqKSqzoAL0OHA4nLfyN+Pn7kpZTht1uR4cTo04DnPgYdZSWWtFQ+Jj0KKUosznR6zR8THqKSl0P7nIEtCLu9rvq9J3U6bRqb6abxSqp6enp3HDDDbz//vv4+voyffp0HnroIfr06VOj/SUpVG/Ja9/SNjyA60ZcgiW3hK7tgut8rJYtA3j+ve8Z1COS6FauuxOllPuxlmfX7vKLreg0jZ9+y6L3JWF8sfcEBr2OK/q05lhGIZd2CAXg3S9+JSW9gHYRAYzq1wZfs4GEt/ZxWecwdJrG5Cs7cuhYLkF+JkICzfiZDbz75a/8+FsWU67sROvIQNqEuB6IsvNAGu9+8SuDe7biij6tOZCczdqvj6KAf9zwB/75+m6iWwUyvE9rjqYWsO3HVIL8TfTrGs7+pEzGDmrP258fARSTh7Thisva8d4Xh2mht5Kfk4UBRU6RFX1pPkP7RnP0VA6O0iJKCwvw09kIMkO/Ppew+5dscnLyMWl2939mzYZNGfDTyvDRbISHBVJkhfR8G3alx0ezYdAc+F55EwVOH77Ye4JTmYUEaKVE6PMJCzJhKy3FZC8iUFdCkK6EAK0UDYW/zgpAoFZCiL4IX63qoygdSkOvNfnX3c2mdBi1+n3uc0NyKA0N0GmumpgTDaVpOJWGQsN1ZjV0Gq7fFTgr1NgUGk40QMPhVBhwoNcUStO5/+ZQGg6n66mNStOwOVzfMaNBh92hTj86VIcGFPu24op7Hrk4kwK4hqS++OKL2Gw2pk6dyty5c2u8rySFytZsPkLnNi0Y0D2Cvz6+Fb1Ow2zSk51fxn8WXOW+u63os93H6dy2BR1bBwGuCTGWvBIiQ1zP7N2flMVX+1P57rDrcZevxo/iaGo+j6zay70z+/PmpsO0DvNjxjVd8fMxsmn3cdZsPkLLFj5k5pUSO7g9iTtdTwXr1zWc736xMCe2O5EhviS8tc8dR/vIAMJb+LL3Fwt6HARoZbT0sWG0uy6OgbpSWgVAUVEJEfp8DJrDdaH19yW/2MYObSC/5Jc/XU7hr5XRRp9NuL4AZ6eh7DiY5f6bCTvtDNlcYs7GXxURpBUTpCsmWF9CC10JRqo+SvV8nEpDd9aF16E07JqREqceA06KlZkyZUCPE6PmwIADo+bArhkJpIij9nBynf5EmoqJ1OdhcpZ6fi2DD/kOM1a7kyKnD0aDRlhkBAFhERiCwjleoOfzb4/io9kI9VV0bu1Pu6iWYPThwMkiCgpKCPLVExbsR+uWgTjQOHS8gIiwAOxOCAv2p6jMSXiIPx9uS+GXkwXMiulBZl4ZX+w+SvtQI3ochAcaKC4uIdRPj04HEaEBhIcGkF0MWUV2fj6aQ1luBv5mHWMHRfP5dycpKS4hUCulc4dw/P39yC60cSAll1YtA+jRIYz0nBIiQgM4ciKP3CIbg3q25teTeZzILGFonyg+33uSolIHN8f1xGg0sO3HNH5LLaBnxzAiQwMwmwzs+SWTg8fyuPGabrRvFYRer8ep4FhGIZ/uOs6EYR2ICPXHYDBgc8LJrFL0ej1+/maMBgMhQX7YlYbNqdG+XRiHkzJJzymm5+mbmTP/5ors/FK++uEU3dqFcGnHM3+35JaQlVdK9+iQSvtk5JaQllXEpR1D0evOfB+VUpRaHe6aot3hxGZ34ms2kFNQxsnMQnp1DHNvX9drxQWRFH6PCzUpOJ2Kbw+mM6hnJAXFNp5f+xN/jutBjy4RWCwFWG0O7A4nfj6upo307GJMRj0hga4Hj9sdTnb8nMb/tqdwU0w3tv+cxhW9W7svspf3jGTXgcod9nNiu3N5j9PP09XAoNc4kVHEP1/fja/ZwLK5l1NUYiMlvYCXPznIXVP70LdzSxJWf8fh47nu43Rp24JSq4PjGYVEtwokJa0ADSchRiu9o4NIOppKuD4fM3aiDRaKlZkAXRlBWgmflfbiN3skJmzYMBClz6anbyadzZmYrPkE6EoJ0Erx03l+8LpTgRMdNr9w0gscGHFg1OxE6As4bg+lVPMhXMslQCvFUOFO9EdrWyyOICL1eXTyycHXWXzmoAYzJboAThQZaNkqkrBWrfj6cCH5hcVccVlbQsJbUqb58V1SDgbNSUBoOEdPZNK/Ryvat40Ekx+a2Y9Pvz3Bzn2/MqZ/G665sjuHUvIJbuFHwOlEmVdUxvWjumCzO/jmxzRWf/YLHVoFotdpzBzbje1v/YcxPj9SoAUQ2rot+uBIdCFtSbMFsONQFrFDOhEQ2hLNLwjNYMbucHLSUkS7yADXPehZtbV9Ryxk55cx/LLWGA1nmrZq+9m3O5ykZRfTNjzA/dk9V39FRXmFZTzy5l7mxHSnx+kL6tc/nOLn5GxunXipO+b8Iisd2oeSfY5lG5xKodM0yqwOcgvLiAz1q3ZbpRRFpfZKTYN11dTXiupIUqhGc0wKOQVl7DmUwaj+bSrdCVT8+2e7j/Ppt8eYO6EnqVnFfLI9mZjL2zNyYHsMSpG4K4XP95xwX5hvefQLAP6z4CoA7nl6m7t9saaM2BnQoxUpqbmYnCX07hDK9p9OEanPI09rQct2HfghKYs/dGnJviOuB7QH+RnJLz5zgR7WM5STvxwmXJdPZ2M6GooQXRGXmLPQO6teyEsxY1RWrBgw4sCgOSl0upJExSYNzT8Eu38EDqM/Vr0/244UMmpoD4LCwtB8gtD5BqL5BpFRoMgrstK1fQh2h5M3Nv3Cjh9TGWw8xB/9d5GjBRPaqSf4tkD5BFFkDOPglkQGmI/iQI8jIALfVh3RWrRCH9YWY6uuaD4B2B1ODh/LpWeHEDRNw+5wkl9kJTSo6nOta6Imny+7w+muuWXnl/J/z20HFFf1bcPsmO51et36iq0pSFy10xBJoVl0NF8M7A5XR1D5Xc9XP5xi3bajWHJLuPGarlW2//uz37h/ttoc5OS7mggKS2wsfGYb7SMD8DtdhVy96TCfbE92b79p93H6XBLmTgjld+rlLmkTRJiWT376SXw0Gz2NJwnRFdFOn4WvzgbpuAYj64ATMDrYtZ9SsL5gHIGaP/uPpAOuO8v8YhtmrHQ1pjEq8Fc6ZaRCkAMAm96XfKsOvW8APl1GkGYPIrvYSe9ubdh4sJQfk/MZOaIf//nkINNGdOKHPd9zrf5LMg2RBES25WhyGn4Rbbl6YiyaX3ClO91pMZ7PdWQYRIa5FvMy6HX87cb+pKXncddK2J/TnuiObfjb1X3d2xvK7Lz9vzy+Ku1Ou86XcMuUAR6Pa9DrKlX9DXpdnRNCTVVsyitvMgANXx/5aoqmIZ+8OnKermDpNA2H08m8f20h9vL2TBvZmW9+TCU5NR+A745YGD+0A9t/SuPq/m0xGnRYbY5Kx/I1G/j1lGv7bftTAUjPKaFzmxYAZOWXkZVf5t7+o69+o/zSuWB6X0qtDp7+cD+t9bm00ufyRy0Jv+JTcHqkWjG+lDj0HNF3pkQfSGa+lehWgYS0bMmxzGJCg3z5NqmIq/W7idM+ZWKIk91lnVhf3I8/93PQsjQF36xDaNZi8AnC2HUMhtZd0YIicfhHkJtZTHTrQDRNowPQ4XRsY9s7ucrmwM/HQGgLX7q0bcHXP6bxaPYkLokKYvaQ7jx58FvmD+yDzr9ym2ttGA06fEwGfM0GCop98TFV/libTXoc6ElxhNPRt+7DlxtaxZFLfmb5aoqmIZ+8Onry3R8oLLFx/00DyMgpASBx1zGmjezMK/876N4uv8jGmi+OsPPndN798leuHd6pSkdVdn4Z6dnFlcqC/U1VkgdA7OXtSdx1jO9+saDXaVzSwkrJ7g9Z3OIAYfrTbbG6lpiG3Mh/PjuOj2ajX+wUNuw6zj1/vIwAXyMZuSWEt/BB0zTKGyjKfkxl9acmxvj9hAkrA82/MdD8GxwFzbcF+vZ9CL88hgJzazSD2R2PHugUFeTxHBkNOowG151w+agn/9N3wL5mA+0iAnjh7yMwGX//cFkAX5OBgmIbvmcNv9Vpro72UqvD3T/THOkq1JKac5zi4iZJoRpf/XCK1xMPebxoKaX46Wg2ADt/TnPfmep1rlpDRXaHk50/n+ns/WzPcfzP6vj6/ogFgP5dw9n7i4WQQDM5BWVV5hZE6nIZnP8zvYN+4530wUwNyaDsw9XoND1l/u3Iancp7S7tg65lRzSdjn3/c/VBzOwYRv/uke7jRAT7Vnm/4cG+nHCE8WrBCIK0Yu4M2sivzjaMvnEWuuAoNE3DNzyQwt/Zrlp+sStvKqmvhADgY3Yd6+yaArj+beBMUmrupKYgmop88qrxwdYkAN787Bcu7xHJpR1DKSmz8/AbeypdVI+mFtDC3wSAXu8aFeHJ/Kl9yCuy8nriIRJ3plT62y8n8ggOMPGn8T2IyWxPwYlf2PBVCql5bWjnV0obx3H6m47SyZCBPtNAsMHK31okggJDh8GYB19Pt3M0v/jUYOJacOCZu/985ceyvCm08DcxNqTNefetjYo1hfpWngw8vd/ycQgXSlu93wUSp7j4yCevGg6H6yqybX8q2/an8mr8KFLSCkjNKiY160xTT3pOMUUlrhE3VpuT3EKr+2+BfkYKTo/aCQ/2pcPpCV+ZeaV0btuCX0/kubdtHeaPMe84UQc/xZ60izuD4LuyaPqYT2LATh6B7Ld3YuD1f+Wf//mKvqYUAjv0ZPzV11T7HsprHDVZMiTQw7A9k7H+l8bycyeF+qshlCsfIekp4ZQPsvO/QJplpKYgmop88s7y9Af7KSyxYT+rGSgjt4TjGZXHT7cND+Cn37JPjw93jd45YTmzTWSoHwXFrgt/yxY+mIx67p52GV9+d4LYwdGseedT4ny/5+2iIfRyplH80SYw+qB6xZDy/R76mVPINkXRJmYOARGdaXt62nymCuKz0t6MD40+53tZ+qdBlJTVbMiqp7vr+mzaKVfeJObroYnn9ypPfp5rCq6kcKHcgV8ocYqLj3zyzlI+Nl9/1sSc/21PRlH57r9Xp1BOWAppFebHjdd05d9rvietQi0ioMJdafkFts8lYfRu50dR4pPcE3QYgAeDP4Ii0Lfthe/o29H8Ann16wD6m3/Dp9M1RLc6M6TVtaSE6+dAP9M534u/j7HGd8aa5prCrziT4EyGhribP31ea77eYY2VH9rHQy2k/JxdKHfgF0qc4uIjn7xqOCpMhusRHcLXp4eK9uwQQr+u4RxIznH3JfTuFEbw6Z9zC88MHT27GUM5ndiPfEPZno9QxbnYu45m349H6W/6jeTAy+gz5q9oBhP+vgbylB9flPZiotlMdYLOkxRqy2DQYbM7aeFvIrfQirkBmo/Kk0JDTpn01NFcPrnxgulovkDiFBcf+eTVwJ/H9+SbH1P59WQeV/VtQ98uLRnVry0FxVbSs4uZOKwjpVZXM03FPoVBPSLY8XMaAKqsiOLEf+PM+A1deCd8rv4LhlZdWfv9Vt7PHcjwzpdwmcF1ka+YTM7VhBPoV7/t4wa9hs0OLQLM5BZaPa6R9HuV3803xEPQyysfBg/LL5xpPmrefQrtIwM4ll5YaUkKIRqTJIUKyjzMCwAIDjAxfmiHKuWBfqYqSxHknK4pPPTny2nT0p//m96XsEADJZuewpl5DJ+R8zB0HuJu/9Yb9BSWmjBXuPhX7Bg2Gaq/MNd7TUGvAxwE+5tIgQZp4ukRHcLH3yRXWSCsXpTXQjz8qV8X13Dfhujgrk8LbvgDORUmKgrR2CQpVFBQfOYuv3zkDtTsgT/lnZvlzUflbcI9O4RS+vV/saUexmfUrRg7D/G4/9lzElxzHlQj1xRcCahFwOkmqwZo4unWPoTn/zaiXp7vcLa+nVvy89Fsj/Mw5k3sSV6h1eM6VM1JbfqBhGgIkhQqKO9AnjK8E6P7tyWvyFrj0Tu608tT551uPjIbXQ/HsP30GbaDX2LqG+cxIajTzSjmsy7+er0rKZxdXlFAAzQfwZlk01DN/g2REABG9WvDoB4RHjvgjQY9LT0kCyFEZc37tqmRlSeFHu1D8DUbaBXq536+QE1UXF7Bx6SnbMdblO14C337vpgGXOdxn/Km9bMv/obTd7Se5gosvPEPxA5uX+93vVf2iQIgrMXpReAusAV0NU0774gsIcS5SU2hgsKS00+vquMduGvUi9XVD1BowfbzZgxdr8Bn+By0ai7g5R2uVZqPTt+1e2o+6tY+hG7t679NPm5INKMHtOWX4665FRdWShBC1AdJCrhmu36w9Td3f0Bdk0J5J6bZpMe67xPQ6TAPvA5NV/1pVpQ3H1VOGuXt++ZGHIWiaRo+JoN7jsYFVlEQQtQDSQpAqdXBhp0p7sE2dV2Xp3x8fLQxG9sv32DsOfK8S0KXT5yu0qegK68pNEELXwOMOhJCXBjq/Yqzd+9errvuOiZNmsRNN93EyZMnAcjPz2fevHnExsYyY8YMLBbXyqBWq5UFCxYQGxvLlClTSEpKqu+Qzquo1NWXoHB1ttZktJEnvmYDBhxM129C8w/G1G/iefcpHz9/dvNReU2hpo88rE/lr3gRPJRPCFFL9Z4UFixYwLJly1i3bh0TJkzg4YcfBuDJJ59kwIABJCYmMm3aNJYtWwbAqlWr8PX1JTExkUWLFhEfH1/fIVWrtMxO4q4UikrOjDCq64Qt5XTQxXGEiX57CaQInxF/Qud7/k5qZzWjjyJCXCNlzl5uozFcEtWC6MhA/jiqc6O/thCiadVrUrBarcyfP5/u3V0Turp160Zqqmt5iC1btjBhwgQAxo8fz1dffYXNZmPLli1MnOi6ox44cCA5OTmcOnWqPsOq1tubDvPel0l88d0Jd1ldk4Lt580MzfmYET6HyPdtiz6qR432c9cUzkoKcyf0ZO74nrQ+/djJxmQ26Xnw5oF0aFXzkVdCiItDvfYpmEwmJk2aBIDT6eSZZ55h9OjRAGRkZBAeHu56UYOBgIAAsrOzK5UDhIeHk5aWRlRUVI1ft7oHUJ9P+ZPN8io8lN5k1BMeHlir4zhtZRz7/mMshPFW/gDmTRtPRETNLqjlLTRtWrcgoMJwyg7tQunQLrSavZpObc9NY5G4aq+5xiZx1U59x1XnpJCYmMjy5csrlXXq1InXX38dq9VKfHw8drudW2+9tdpj6KoZplldeXWysgrrtJZO+XDPjAqPwtRpYKnl08Vsv3yDs6SQgktvIHmblZBAv1ofoyC/hJIi1+in8PDAWu/fGCSu2mmucUHzjU3iqp26xqXTadXeTNc5KcTGxhIbG1ulvKioiNtuu43g4GCef/55jEbX8M6IiAgyMzNp1aoVdrudwsJCgoODiYiIwGKxEB3tejaAxWIhIiKirmHVivH0yJ68ojNrzdSl+ch2+Cu0oEj6DB3Ky8Nq1wcQP6Mfuw6kN8jic0IIUVsN0tEcHR3NypUrMZnONIeMGDGCtWvXArBhwwYGDBiA0WhkxIgRrFu3DoA9e/ZgNptr1XT0e5Q/L6Ck7MxCeLW9ODvz0nCkHsbY7co6jVrq2i6YWWO71Xo/IYRoCPXap3DgwAE2b95M586dmTx5MuCqIbz00kvMnz+f+Ph44uLiCAwM5LHHHgNg1qxZLF68mLi4OEwmEytWrKjPkM7J02zh8vV/asp2eBtoGsauw+orLCGEaDL1mhR69uzJ4cOHPf4tODiYF154oUq52WwmISGhPsOoMU8TwwznWKr6bMrpwPbLNvTt+px3kpoQQlwIvLoh29OzCoy1aD5yHP8RVZyLsfvw+gxLCCGajFcnBU/9B/paNB/ZDn+F5huEof1l9RmWEEI0Ga9OCp4Gsda0pqDKirCn/IChy9BzLngnhBAXEu9OCh6yQk1HH9lTD4FyYIj+Qz1HJYQQTcerk4KnukJNRx85Th4EvQl9xCX1HZQQQjQZr04K5ctWV1TTmoLj1EH0rbui6aXpSAhx8fDqpFCxplA+76wmQ1KdxXk4c07WeNE7IYS4UHh1UqjYpxB0ejE6Qw3WXXKkHnJtK0lBCHGR8eqkUHENvSD/00nBcP4+BcfJg2D0RdcyuqFCE0KIJuHVSaFi81F5UqjJkFT7qYPoW3dD0zXe85OFEKIxeHVSqNh81OJ0UtCfJyk4C7NQ+ekY2kjTkRDi4uPlSeFMVmhRw5qCI9W1tpO+dfeGC0wIIZqIdyeFCj8Hlnc0n2eegsOSDHoTutC2DReYEEI0Ee9OCqezQkig2b1i6vnmKTizUtCFtZP+BCHERcnLk4IrKzx480B3MjhXUlDKiSMzBb2MOhJCXKS8PCm4/q/TNHdfwrmGpKr8DLCVom/ZoRGiE0KIxuflSeFMr0L5ktnnmrzmsCQDyPwEIcRFq8GSwoEDB+jVq5f7d6vVyoIFC4iNjWXKlCkkJSUBrgtzQkICMTExjBs3jr179zZUSFWUpwRNo0JN4RxJITMFdAZ0IW0aITohhGh8DbKaW0lJCUuXLsVms7nLVq1aha+vL4mJiezevZv4+Hjee+89Nm7cSFJSEhs2bCAlJYV58+aRmJiIwdDwC82VVxQ0NHcyONeQVHcnsyyCJ4S4SDVITeHRRx9lzpw5lcq2bNnCxIkTARg4cCA5OTmcOnWKrVu3Mm7cOHQ6HR07diQqKop9+/Y1RFhVlDcfaRoEB5gBaBFgqnZbR2YK+jBpOhJCXLzq/ZZ38+bNlJaWEhMTU6k8IyOD8PBw9+/h4eGkpaWRkZFBRERElfLaCAsLqFOsSqUD0LJlAO3bhvDGkrGEBPp43NaWm05hWREtOnYjKDywTq9XG+GN8Bp1IXHVTnONC5pvbBJX7dR3XHVOComJiSxfvrxSWadOnSgsLOT111+v0TF0Ol2lzt6K5bWRlVWI0+np4Zrn49onO6uIIpNr3oGl1OZxS9tvPwNQbI6kzFJQh9equfDwQCwN/Bp1IXHVTnONC5pvbBJX7dQ1Lp1Oq/Zmus5JITY2ltjY2Epl7733Hi+++CIzZsxwl02aNInVq1cTERGBxWIhOtrV/GKxWIiIiCAyMhKLxeLevry8MbjzSA0etubMTAFNLzOZhRAXtXrtU5g2bRqff/4569atY926dQCsW7eOgIAARowY4S7bs2cPZrOZqKgohg8fzvr163E4HKSkpJCcnEzv3r3rM6xqufsUarCtI/s4uuDWaAbPfQ5CCHExaLRhNLNmzWLx4sXExcVhMplYsWIFADExMezfv9/dCb1s2TJ8fDy36zcUTTt/WnDmpsqkNSHERa9Bk8Lhw4fdP5vNZhISEqpso2kaCxcuZOHChQ0ZikfOCqOPzkXZrah8C7rOQxohKiGEaDpePaOZGvZNO/PSACWT1oQQFz2vTgrOCmsfnXO7nFOu7UJaN3RIQgjRpLw6KVSY0nxOztxToGnoWrRq+JiEEKIJeXVSqOmIVGfOKbSgCDS9saFDEkKIJuXVSeFMR/N5mo9yU9EHRzVGSEII0aS8Oimgzl9LUE4Hzrw0dMHSnyCEuPh5dVJQnL+WoAos4HSgC5GaghDi4ufdSUGp885RcBZkAqAFhp97QyGEuAh4eVI4/zblSUEXENbA0QghRNPz8qSgzt98VJgFmg7NP6SRohJCiKbj5Unh/EtcOAsy0fxD0HT6xglKCCGakHcnBWow+qgwC11gy8YIRwghmpx3J4UaNB85C7PQpD9BCOElvDwpcM6qgnI6UEU50skshPAa3p0UUOjOlRSKckA50aT5SAjhJbw7KSg4V1VBhqMKIbyNlycFdc6OZlWYBYAuQGoKQgjvUO9JISMjg3nz5jF58mSmT5/OiRMnAMjPz2fevHnExsYyY8YMLBYLAFarlQULFhAbG8uUKVNISkqq75Cqd54hqc7C07OZA0IbKSAhhGha9Z4U/vGPfzBy5EjWrl3LpEmTeOyxxwB48sknGTBgAImJiUybNo1ly5YBsGrVKnx9fUlMTGTRokXEx8fXd0jVcp5n9JEqyELzDUIzmBotJiGEaEr1mhSys7M5dOgQ06dPB+C6667j7rvvBmDLli1MmDABgPHjx/PVV19hs9nYsmULEydOBGDgwIHk5ORw6tSp+gyrWudb5cI1HFWajoQQ3qNek8Lx48eJiorikUceYeLEidx1110Yja4H02RkZBAe7lpUzmAwEBAQQHZ2dqVygPDwcNLS0uozrOopzjn6yFmYiS5QOpmFEN7DUNcdExMTWb58eaWy6OhoDhw4wJ133sl9993He++9R3x8PKtWrfJ4DJ3Oc06qrrw6YWEBtdq+nFMpdHod4eGBVf6mlJPCwmz8e1xOmIe/NzRPMTUHElftNNe4oPnGJnHVTn3HVeekEBsbS2xsbKWyY8eOMWXKFEaOHAm4mokefvhhACIiIsjMzKRVq1bY7XYKCwsJDg4mIiICi8VCdHQ0ABaLhYiIiFrFkpVViNNZgyVPPVBOhcVSUKXcWZyLctgo1QV5/HtDCg8PbPTXrAmJq3aaa1zQfGOTuGqnrnHpdFq1N9P12nzUvn17IiMj2bp1KwBffvkll156KQAjRoxg7dq1AGzYsIEBAwZgNBoZMWIE69atA2DPnj2YzWaiohrngTbnWhDPPRxVmo+EEF6kzjWF6jzzzDM8+OCD/Otf/yIgIIBHH30UgPnz5xMfH09cXByBgYHuUUmzZs1i8eLFxMXFYTKZWLFiRX2HVK1zrX3kLHAlBeloFkJ4k3pPCp06dfLYhxAcHMwLL7xQpdxsNpOQkFDfYdTIuR6yc2bimsxREEJ4D6+e0exU1a995CzJA70JTH6NG5QQQjQhr04KLp6zgirOQ/Nrcd6ltYUQ4mLi1UnB1adQzd9K8tH8WjRuQEII0cS8PCmcY/RRcR4636DGDUgIIZqYJIXqmo9K8tB8paYghPAu3p0U8Nx8pJx2VGmBNB8JIbyOdycFhcf2I1XimiEoNQUhhLfx8qTg+SE7qiQPAM1P+hSEEN7Fu5MCnjuaVXE+ADqpKQghvIx3J4VqlrlQpa6koMnoIyGEl/HypOB56poqkaQghPBOXp0UwHPzkbMk37XEhcHc+AEJIUQT8uqk4HoGg6fRR/lovoGyxIUQwut4dVIAz4/jdCUF6WQWQngfr04Kzmo6FVRJAZpv83z0nhBCNCSvTgrVLXOhSmTdIyGEd/LqpABVO5qVUqdrCpIUhBDep96TwokTJ5gxYwaTJk1i1qxZnDx5EgCr1cqCBQuIjY1lypQpJCUlAa6LcEJCAjExMYwbN469e/fWd0jVcnpaOttaDMohSUEI4ZXqPSmsXLmSuLg41q1bx5gxY3jiiScAWLVqFb6+viQmJrJo0SLi4+MB2LhxI0lJSWzYsIFnn32W+Ph47HZ7fYflmYKzOxWc5UtcSFIQQnihek8KTqeTwsJCAEpKSvDx8QFgy5YtTJw4EYCBAweSk5PDqVOn2Lp1K+PGjUOn09GxY0eioqLYt29ffYflkfLwOE73Yng+khSEEN7HUN8HnD9/PtOnT2fVqlXYbDbeeecdADIyMggPD3dvFx4eTlpaGhkZGURERFQpbwzKwzQFmc0shPBmdU4KiYmJLF++vFJZp06dKCsrY+nSpYwePZqNGzdyxx138PHHH3s8hk6nQynlsbw2wsICarV9OYXCZDQQHn5m+GleipVSoGXbKAwBTTcstWJMzYnEVTvNNS5ovrFJXLVT33HVOSnExsYSGxtbqSw7O5vY2FhGjx4NwNixY3nwwQfJyckhIiICi8VCdHQ0ABaLhYiICCIjI7FYLO5jlJfXRlZW4enZybWjFNjtDiyWAndZmSUD0MguAq2koPqdG1B4eGClmJoLiat2mmtc0Hxjk7hqp65x6XRatTfT9dqnEBISgtlsZs+ePQDs3bsXf39/QkNDGTFiBOvWrQNgz549mM1moqKiGD58OOvXr8fhcJCSkkJycjK9e/euz7Cq5WnumirJR/MJQNPpGyUGIYRoTuq1T0HTNJ555hkeeughSktL8ff35+mnnwZg1qxZLF68mLi4OEwmEytWrAAgJiaG/fv3uzuhly1b5u6cbmgKVWWiQvm6R0II4Y3qvaO5T58+vPfee1XKzWYzCQkJVco1TWPhwoUsXLiwvkM5L6Wqrn2kSgvQfCQpCCG8k1fPaPbUya3KitDMdeu4FkKIC52XJwWqLI+tyorQfPybKCIhhGhaXp4UKi9zoZRClRVKTUEI4bW8Oylw1ugjhxUcdjBLTUEI4Z28OilwVvORKi0CQJOkIITwUl6dFM5+yI4qc63ZJElBCOGtvDopuJqPKtQUyk7XFHykT0EI4Z28OilwdkdzaXlNQZKCEMI7eXVScJ61zIW7piDNR0IIL+XVSaFKR7O7+UiSghDCO3l1UlCc9TjOsiLQG0BvarKYhBCiKXl3UjhrlYvyiWtnz3IWQghv4eVJQaE7a56C9CcIIbyZVycF51lTml01BUkKQgjv5dVJAVSV0UeSFIQQ3syrk8LZq6S6VkiVOQpCCO/l5Unh7MlrRbIYnhDCq/3upLBy5Ur3IzcB8vPzmTdvHrGxscyYMQOLxQKA1WplwYIFxMbGMmXKFJKSkgDXhTkhIYGYmBjGjRvH3r17f29INVZx9JGyW8FhldnMQgivVuekUFBQwKJFi3j11VcrlT/55JMMGDCAxMREpk2bxrJlywBYtWoVvr6+JCYmsmjRIuLj4wHYuHEjSUlJbNiwgWeffZb4+HjsdvvveEs1p8A9+khmMwshxO9ICps3b6ZDhw7cfPPNlcq3bNnChAkTABg/fjxfffUVNpuNLVu2MHHiRAAGDhxITk4Op06dYuvWrYwbNw6dTkfHjh2Jiopi3759v+Mt1ZyqsEqqzGYWQojfkRQmT57MvHnz0Ov1lcozMjIIDw8HwGAwEBAQQHZ2dqVygPDwcNLS0sjIyCAiIqJKeWOouHK2LIYnhBBgON8GiYmJLF++vFJZp06deP3112v8Ijqd59yj0+lcd+s13L46YWF1u5ArwNfXRHh4IIWZdkqA0NaRmMMD63S8+hTeDGLwROKqneYaFzTf2CSu2qnvuM6bFGJjY4mNja3xASMiIsjMzKRVq1bY7XYKCwsJDg4mIiICi8VCdHQ0ABaLhYiICCIjI92d0RXLayMrqxCns2pyOR+lFGWlNiyWAqzp6QDklurRWQpqfaz6FB4eiKWJY/BE4qqd5hoXNN/YJK7aqWtcOp1W7c10vQ9JHTFiBGvXrgVgw4YNDBgwAKPRyIgRI1i3bh0Ae/bswWw2ExUVxfDhw1m/fj0Oh4OUlBSSk5Pp3bt3fYflkaowo1mV5AMamk/zvBsQQojGcN6aQm3Nnz+f+Ph44uLiCAwM5LHHHgNg1qxZLF68mLi4OEwmEytWrAAgJiaG/fv3uzuhly1bho+PT32H5ZlS7slrqrQAzScATac/z05CCHHx+t1J4c4776z0e3BwMC+88EKV7cxmMwkJCVXKNU1j4cKFLFy48PeGUmsVH7KjSvLRfKWWIITwbl49oxkq1BRK8tF8gpo4HiGEaFpenRScCvcyF86SfDRfSQpCCO/m1UkBaT4SQohKvDopqNPNR8ppB2uxNB8JIbyeVycFpxPQQFlLANDMfk0bkBBCNDGvTgruh+yUJwWTb5NGI4QQTc2rk0L5Q3bKawpIUhBCeDkvTwqumoKyFgOgmaT5SAjh3bw7KUDlPgWj1BSEEN7Nu5OCOv2QHelTEEIIwOuTgmtlVelTEEIIF+9OCrhmNJ/pU5CkIITwbt6dFJzqzOgjvQFNb2zqkIQQokl5d1Lg9DIXthIZeSSEEHh7Uqg4T0GajoQQwtuTgjrdpyA1BSGEAG9PCuX/txZLJ7MQQlAPSWHlypU8/fTT7t+TkpK48cYbmTRpEtdffz0HDx4EwGq1smDBAmJjY5kyZQpJSUmA6249ISGBmJgYxo0bx969e39vSDVX/jhOa6lMXBNCCH5HUigoKGDRokW8+uqrlcrvv/9+5s6dy7p167j77rvdj9lctWoVvr6+JCYmsmjRIuLj4wHYuHEjSUlJbNiwgWeffZb4+HjsdvvveEs1V/44TmUtlj4FIYTgdySFzZs306FDB26++eZK5dOmTWP48OEAdOvWjdTUVAC2bNnCxIkTARg4cCA5OTmcOnWKrVu3Mm7cOHQ6HR07diQqKop9+/bVNaxaO9OnIElBCCHqnBQmT57MvHnz0Ov1lcqvvfZad9lTTz3F6NGjAcjIyCA8PNy9XXh4OGlpaWRkZBAREVGlvKGVz2Z2DUktlY5mIYQADOfbIDExkeXLl1cq69SpE6+//nq1+yilWLFiBT/88ANvvPFGtdvpdDr3xfns8toICwuo1fYADqfrdYN8ARQBIcEEhzefx3GGN6NYKpK4aqe5xgXNNzaJq3bqO67zJoXY2FhiY2NrfEC73c7ChQtJT0/njTfeIDDQFXBERAQWi4Xo6GgALBYLERERREZGYrFY3PuXl9dGVlYhTmfV5HIuDqcTAGthPgBFNh02S0GtjtFQwsMDsTSTWCqSuGqnucYFzTc2iat26hqXTqdVezNd70NSExISKCws5NVXX3UnBIARI0awbt06APbs2YPZbCYqKorhw4ezfv16HA4HKSkpJCcn07t37/oOq4ryCorRWQbIukdCCAE1qCnURnZ2NqtXr6Zt27ZMmzbNXb5u3TpmzZrF4sWLiYuLw2QysWLFCgBiYmLYv3+/uxN62bJl+Pj41GdYHpUnBYOjPClIn4IQQvzupHDnnXe6fw4NDeXAgQMetzObzSQkJFQp1zSNhQsXuoeuNh5XVjCo00nB2PCJSAghmjuvndHsPKv5CKkpCCGE9yYF3M1HpYD0KQghBHhxUlDlzUdO6VMQQohy3psUTtcU9I4y0HRgMDVtQEII0Qx4fVIwOMvA5OtaGE8IIbyc1yYF9+gjR6n0JwghxGlemxTKRx/pnWWSFIQQ4jSvTQrlay4ZHGXyLAUhhDitXmc0X0jsjvLRR6VgCmriaITwXg6HnZwcC3a7lYwMHc7T65I1JxdqXAaDiZCQcPT6ml/qvTgpuE6k3lEmw1GFaEI5ORZ8fPzw92+F0ajHbm9+F1+DQXfBxaWUoqgon5wcCy1btq7xMb22+ahyUpDmIyGait1uxd8/SEYA1jNN0/D3D8Jut9ZqPy9OCgpQ6GX0kRBNThJCw6jLefXipODEiANNOeT5zEKIZmHq1Amkpp5q0hi8Oin4aDZAlrgQQohyXtzRrCokBakpCCFcvvtuD88//xQOh5PWrVvj6+vH0aNJOBxOZsyYzahR1zBpUgzvvrsWPz9/brvtFoYNG87MmXP4/PONfP/9Pm677Q6WL38IiyWDzEwLffv+gfvvX8q+fXvdx+7U6RLuuutvLF36ABkZ6XTo0Amr1dX+/+uvR1ixYhkOhwOTycSiRQ/Srl37Rnn/XpwUnPhqrn8AmacgRPOwbf8ptu5rmOaTK/q0Zljvmo3COX78GO+//wmrVr1Gy5bhLFnyEHl5+fzlL7fQs2cv+vcfwL593/GHP/QnNTWV77//jpkz57Bz53auvvoatm/fRpcuXXn44QRsNhszZ07j8OFDlY4dEBDA448n0LVrdx577Cm+//47vvjiMwDeffctpk+fyahRo9m8eRM///yjJIWG5mo+Ot0rLzUFIUQF7dpFExAQwJ4931JWVsqGDR+jFJSWlnL06G8MGXIFe/d+i06nMWZMLJs3b8Jut/PDD9+zYMEizGYzBw78xLvvvkVy8lHy8vIoKSmudGyAffv2smTJIwD07duPqKg2AAwZMozHH1/Brl3bGTr0Sq666upGe+9enBSk+UiI5uaKPlEM7tmqqcPAbDYD4HQ6eOCBh7j00p7Y7U6ys7MICmpBQUEBa9asRq830L//QI4dS+aTT9bSqVMnzGYz77+/hi1bvmDixClMnTqIo0eT3KsolB8bXKODKk4+0+v1AIwcOZpevfrwzTdf8957b7Nz5zcsXHh/o7z3393RvHLlSp5++ukq5WlpaQwaNIgTJ04ArokUCQkJxMTEMG7cOPbu3eve9tVXXyUmJoaxY8eyadOm3xtSjdgdTkyaHZBHcQohPOvXbyBr174PQGZmJjfddAPp6WmEhIRgNpv55puv6NOnL/36DeT1119h6NArAdi9excTJ17LmDGxgMaRI794nHk8YMAgNm1KBODgwZ85edJ1vVy8+F4OHPiZyZOv489//ou76akx1DkpFBQUsGjRIl599dUqf3M6ndx3333YbDZ32caNG0lKSmLDhg08++yzxMfHY7fb2b9/Px9//DHr1q3jrbfeYsWKFeTm5tY1rBqz252YTycFeZaCEMKTW26ZS1lZGTfeOI358//CX/96F23atAVcTTwBAYH4+fnRv/9AMjMtDB16BQB//OONvPbaf7jllhk8/ngCvXr18TjU9E9/upWTJ08wc+YfefPN193NR7Nm3cyqVa9xyy0zePbZJ7nzznsa7T3Xuflo8+bNdOjQgZtvvrnK315++WWGDh3K0aNH3WVbt25l3Lhx6HQ6OnbsSFRUFPv27WPXrl1cc801mM1mzGYzgwYNYsuWLUyePLmuodWI3anO1BQM5vNsLYTwFv36DaBfvwEA+PsHsHjxQx6Xk5gz58/MmfNnAC65pDPbtu1x/61//4G8/faH1R6/nL9/AI888i+P27388hu/633UVZ2TQvlF++ymo59++oldu3bx0ksvsXr1and5RkYGERER7t/Dw8NJS0sjIyOD3r17VymvjbCwgFrH7+NjxIwrKYS3DkPT6Wt9jIYUHh7Y1CF4JHHVTnONC5pPbBkZOgyGM40WFX9uTi7UuHQ6Xa3+rc+bFBITE1m+fHmlsk6dOvH6669X2bakpISlS5fy5JNPotNVDrS8k+XsYKsrr42srEKczqrHOZfc/BJMmg30JjKzimu1b0MLDw/EYilo6jCqkLhqp7nGBc0rNqfT6b4LvxAXnmtKNYnL6XRW+bfW6bRqb6bPmxRiY2OJjY2tUYB79uwhMzOT2267DXDVDubNm8czzzxDZGQkFovFva3FYiEiIsJjeceOHWv0er+Hu0/BKE1HQghRrl7rQ1deeSVffPEF69atY926dURERPCf//yHTp06MXz4cNavX4/D4SAlJYXk5GR69+7N8OHD2bRpEyUlJWRnZ7Nz506GDBlSn2F5ZHe4+hSkP0EIIc5otHkKMTEx7N+/n4kTJwKwbNkyfHx86NOnDxMnTmTq1KnY7XbuuusuIiMjGzweu8OJj86BJjUFIYRw05SnRv0LTF36FN7+/Ahdf3md7lFm/CcvbqDI6qY5tfdWJHHVTnONC5pXbGlpKbRqFQ1c2G33TaEmcVU8v+XO1afQPLvTG4Hd4cSsk+YjIYSoyKuTgkmzy2xmIUSjO3XqJMuXLwVcq7Lecce8Oh9rw4b1LFu2pJ4i8/KkYNZsIDUFIUQjS0tLdS9p0dx49YJ4JuxoRlniQgjhkpGRztKlD1BSUoJOpzF//gKWLFnE6NFj2Lbta/R6Pbfeejtr1rzJiRPHuf32u7n66mvIzs7i0UcfIj09Db1ez7x5tzN48FBKS0tJSHiYX3/9BZ1Ox/TpM4mNHc/KlY9x6tRJ/v3vBEaOvJrc3Fz+7//u4uTJE7RvH81DDyVgMplITPyE9957G6dT0a1bd/72t4WYzWY+/fR//Pe/rxAQEEBkZCt8fevvQWFenBScGLGBQZqPhGguyg5to+zg1gY5trHbcIxdh51zm08+WcfQoVdw442z+e67Pezf/z3gWmnhzTff5ZFH/smbb77OU0+9wI8//sBTT/2bq6++hiee+Bf9+g1g+vSZnDx5gr/+9c+89tpq1qx5kxYtWrBq1bvk5uYyd+5NdOnSjfnz/49XX/0Pf//7Qr77bg/p6WmsWPEErVq15tZb57Bnz7e0atWa9evX8vzzr2I2m3nhhWd4++1VjB8/ieeff4rXXnuL0NAQ/va3u+o1KXhv85HdiRG7DEkVQrgNGDCIt99+kyVL7iMz08J11/0RcC1+BxAZ2Yq+ffthMBho1ao1BQWuEVzffbeb8eMnA9CmTVt69uzFgQM/sXfvHuLiJgEQHBzMlVcOZ9++vVVet3PnLkRFtUGn0xEd3ZG8vFz27dvDiRPHufXWm5kz50a2bdvKsWMp/PjjD/Tq1YfQ0DAMBsPplVjrj9fWFJTDhg4lfQpCNCPm7leg7zy0yV6/T5++vPnmu2zfvo3NmzexYcN6AAwGo3ub8mceVFR1SLzC4XCgVOXhokqBw2Gvsn/FY2qahlIKh8PJqFGjufvuBQAUFxfjcDjYu/fbSq/nKZ7fw2trCia7K8NrZv8mjkQI0Vw899xKNm7cQGzseO65ZyG//HK4Rvv17z+ATz5ZC8DJkyf48ccfuPTSPvTrN5D//W8dALm5uXz99Rb+8IcB6PUGHA7HOY/5hz/056uvtpCTk41Sin//eznvvvsWffr05cCBH7FYMnA6ne5HeNYXr60ptLafBEDfqnMTRyKEaC6uu+56/vnP+9mw4RN0Oh1//3s8zz//1Hn3u/vuBaxYsYwNG9ajaRoLF95Py5YtufnmP/Pvfycwe/b1OJ1OZs++hW7dupOXl0thYQEPPfSAu3npbF26dOXmm+dy111/QSlFly7dmDlzDmazmbvvXsDdd/8VX19foqPrd604r53RvPOl5VyiHaPln55F05pXhak5zTatSOKqneYaFzSv2GRGc93JjOZ6FKgrJSuwW7NLCEII0ZS8tvmoy8xFhLcMJK/A2tShCCFEs+G1t8kmsxmTj4w8EkKIirw2KQghmo+LoGuzWarLeZWkIIRoUgaDiaKifEkM9UwpRVFRPgZD7Zby8do+BSFE8xASEk5OjoXCwlx0Oh1OZ/Mb5XOhxmUwmAgJCa/VMSUpCCGalF5voGXL1kDzGipbkTfFJc1HQggh3CQpCCGEcLsomo90Oq1J9m1IElftSFy111xjk7hqpy5xnWufi2KZCyGEEPVDmo+EEEK4SVIQQgjhJklBCCGEmyQFIYQQbpIUhBBCuElSEEII4SZJQQghhJskBSGEEG6SFIQQQrh5ZVJYv34948aN45prrmH16tVNGsvs2bOJi4tj0qRJTJo0iR9++KFJ4yssLGT8+PGcOHECgO3btzNhwgTGjBnDE0884d7u4MGDXHfddYwdO5b77rsPu93eqHHde++9jBkzxn3ePvvss3PG21CeeeYZ4uLiiIuLY8WKFeeMoTHPmae4msM5W7lyJePGjSMuLo7XXnvtnK/fmOfLU1zN4XyVS0hIID4+Hqj+vJw6dYoZM2YQExPDbbfdRlFRUd1eTHmZtLQ0NXLkSJWTk6OKiorUhAkT1JEjR5okFqfTqYYNG6ZsNluziO/7779X48ePV5deeqk6fvy4KikpUSNGjFDHjh1TNptN3XLLLWrLli1KKaXi4uLUvn37lFJK3XvvvWr16tWNFpdSSo0fP16lp6dX2u5c8TaEb775Rl1//fWqrKxMWa1WNXv2bLV+/fomP2ee4tq0aVOTn7Ndu3ap6dOnK5vNpkpKStTIkSPVwYMHm/x8eYorKSmpyc9Xue3bt6vLL79cLVy4UClV/XmZN2+e+uSTT5RSSj3zzDNqxYoVdXo9r6spbN++ncGDBxMcHIyfnx9jx47l008/bZJYfvvtNzRNY+7cuUycOJE333yzSeN79913efDBB4mIiABg//79REdH065dOwwGAxMmTODTTz/l5MmTlJaW0rdvXwCuvfbaBo3x7LiKi4s5deoUDzzwABMmTOCpp57C6XRWG29DCQ8PJz4+HpPJhNFo5JJLLiE5ObnJz5mnuE6dOtXk52zQoEG88cYbGAwGsrKycDgc5OfnN/n58hSX2Wxu8vMFkJubyxNPPMFf/vIXgGrPi81mY/fu3YwdO7ZSeV1cFKuk1kZGRgbh4WeeRBQREcH+/fubJJb8/HyGDBnCkiVLKC0tZfbs2cTGxjZZfMuWLav0u6dzlZ6eXqU8PDyc9PT0RosrKyuLwYMHs3TpUvz8/Lj11lt5//338fPz8xhvQ+nSpYv75+TkZDZs2MCsWbOa/Jx5iuutt97i22+/bfJzZjQaeeqpp3j11VeJiYlpNp+xs+NyOBzN4jO2ePFi7rnnHlJTU4Gq38ny85KTk0NAQAAGg6FSeV14XU1BeVgUVtOaZkncP/zhD6xYsQI/Pz9CQ0OZOnUqTz31VJXtmiq+6s5VU5/Ddu3a8eyzzxIWFoavry+zZs1i69atTRbXkSNHuOWWW1i4cCHt27f3GENTxFYxrk6dOjWbc3bXXXexY8cOUlNTSU5O9vj6TR3Xjh07mvx8vffee7Ru3ZohQ4a4yxrjO+l1NYXIyEj27Nnj/j0jI8PdLNHY9uzZg81mc/+jK6Vo06YNmZmZzSK+yMhIj7GcXW6xWBo1xsOHD5OcnOyuKiulMBgM1cbbkPbu3ctdd93FokWLiIuL49tvv20W5+zsuJrDOUtKSsJqtdKjRw98fX0ZM2YMn376KXq9vsrrN+b58hTXhg0bCA4ObtLztWHDBiwWC5MmTSIvL4/i4mI0TfN4XkJDQyksLMThcKDX63/X+fK6msLQoUPZsWMH2dnZlJSUsGnTJoYPH94ksRQUFLBixQrKysooLCzko48+4l//+lezie+yyy7j6NGjpKSk4HA4+OSTTxg+fDht2rTBbDazd+9eANauXduoMSqleOSRR8jLy8Nms/HOO+9wzTXXVBtvQ0lNTeX222/nscceIy4uDmge58xTXM3hnJ04cYL7778fq9WK1Wpl8+bNTJ8+vcnPl6e4Bg4c2OTn67XXXuOTTz5h3bp13HXXXYwaNYrly5d7PC9Go5EBAwawYcOGSuV14ZU1hXvuuYfZs2djs9mYOnUqffr0aZJYRo4cyQ8//MDkyZNxOp3ceOON9O/fv9nEZzabefTRR7nzzjspKytjxIgRxMTEAPDYY49x//33U1RURM+ePZk9e3ajxdW9e3fmzZvHDTfcgN1uZ8yYMYwfPx6g2ngbwiuvvEJZWRmPPvqou2z69OlNfs6qi6upz9mIESPcn3e9Xs+YMWOIi4sjNDS0Sc+Xp7juuOMOQkJCmvwz5kl15+XBBx8kPj6e559/ntatW/P444/X6fjy5DUhhBBuXtd8JIQQonqSFIQQQrhJUhBCCOEmSUEIIYSbJAUhhBBukhSEqCdz587l119/rdU+t956Kx9++GEDRSRE7XndPAUhGspLL73U1CEI8btJUhBe74svvuD555/HZrPh4+PDwoUL2bZtG0eOHCEzM5OsrCy6d+/OsmXLCAgI4K233mLNmjUYjUbMZjNLly6lc+fOjBo1ipUrV9K7d2/eeecdVq1ahU6no2XLljzwwAN07NiR9PR04uPjycjIICoqiqysLHccSUlJLFu2jNzcXBwOB7NmzWLq1KkUFRVx7733kpKSgk6n49JLL2Xp0qXodFLRFw2gTgtuC3GROHr0qBo/frzKzs5WSin1yy+/qGHDhqlHH31UDR8+XFksFuVwONTf/vY39eijjyq73a4uvfRS9zr7H330kVqzZo1SSqmRI0eq/fv3q+3bt6vRo0errKwspZRSH3zwgYqNjVVOp1P99a9/VU888YRSSqnk5GTVt29f9cEHHyibzabGjRunfvrpJ6WUUvn5+So2Nlbt27dPffTRR+qWW25RSillt9vVfffdp5KTkxvzNAkvIjUF4dW++eYbMjIymDNnjrtM0zSOHTtGTEwMLVu2BGDq1Kk88sgjLFy4kJiYGKZPn85VV13FsGHDmDBhQqVjfv3114wbN47Q0FDAtbb9smXLOHHiBNu3b2fhwoUAREdHc/nllwOuJa6PHTvGokWL3McpLS3lwIEDXHnllTzxxBPMmjWLoUOHctNNNxEdHd2Qp0V4MUkKwqs5nU6GDBnCk08+6S5LTU3lnXfewWq1VtquvLnmscce45dffmH79u289NJLvP/++zz//PPubZWHlWOUUtjt9irLHJevf+9wOAgKCmLdunXuv2VmZhIYGIjZbOazzz5j165d7Ny5k5tvvpn777+/0dfcEd5BGiWFVxs8eDDffPMNSUlJAGzdupWJEydSVlbG5s2bKSgowOl08u677zJy5Eiys7MZMWIEwcHBzJkzh7vvvpvDhw9XOuYVV1zBhg0byM7OBuCDDz4gODiY6OhorrzySt555x3A9UzdXbt2AdCxY0fMZrM7KaSmpjJ+/Hh++ukn3nrrLe69916uuOIKFixYwBVXXMGRI0ca6xQJLyML4gmvl5iYyAsvvOBeM3/RokXs2LGDnTt34nA4yMnJYeDAgdx///34+PiwZs0a3njjDXx8fNDr9dxzzz0MHTq0Ukfz6tWrWbNmDU6nk9DQUBYvXkyXLl3Izs7m3nvv5dixY7Rq1Qq73c6UKVO49tprOXTokLuj2W63M3v2bG644QaKi4tZtGgRhw8fxtfXl6ioKJYtW0aLFi2a+tSJi5AkBSE8ePrpp8nJyWHx4sVNHYoQjUqaj4QQQrhJTUEIIYSb1BSEEEK4SVIQQgjhJklBCCGEmyQFIYQQbpIUhBBCuElSEEII4fb/sUajzpoUYJcAAAAASUVORK5CYII=", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "开始测试！\n", - "环境：CliffWalking-v0, 算法：Sarsa, 设备：cpu\n", - "回合数：1/20, 奖励：-15.0\n", - "回合数：2/20, 奖励：-15.0\n", - "回合数：3/20, 奖励：-15.0\n", - "回合数：4/20, 奖励：-15.0\n", - "回合数：5/20, 奖励：-15.0\n", - "回合数：6/20, 奖励：-15.0\n", - "回合数：7/20, 奖励：-15.0\n", - "回合数：8/20, 奖励：-15.0\n", - "回合数：9/20, 奖励：-15.0\n", - "回合数：10/20, 奖励：-15.0\n", - "回合数：11/20, 奖励：-15.0\n", - "回合数：12/20, 奖励：-15.0\n", - "回合数：13/20, 奖励：-15.0\n", - "回合数：14/20, 奖励：-15.0\n", - "回合数：15/20, 奖励：-15.0\n", - "回合数：16/20, 奖励：-15.0\n", - "回合数：17/20, 奖励：-15.0\n", - "回合数：18/20, 奖励：-15.0\n", - "回合数：19/20, 奖励：-15.0\n", - "回合数：20/20, 奖励：-15.0\n", - "完成测试！\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEXCAYAAACgUUN5AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA4dUlEQVR4nO3deVxU9f7H8dcMKKKQpeKSt1CuW2qp1x0hwgUdFsXSm5WokKJmlnotxSUVJZIsJbRyuUrumguYiku4homYpdii5m6iCLhggMDM9/eHP+c6ggozCtR8no+Hj4dzzvme85nvzPA+53tmztEopRRCCCGslra0CxBCCFG6JAiEEMLKSRAIIYSVkyAQQggrJ0EghBBWToJACCGsnATBYxIUFERGRobZ7SdMmMDRo0cBGD9+PPv27XtUpQkzpaSk4OvrS/fu3fnxxx9N5t28eZMJEybg5+dH9+7d8ff35+uvvy6VOidNmkTHjh2ZOXOmRes5dOgQb775Jj169MDX15fg4GCOHz8OQGJiIr6+vgBERkYSExMDwLp163jppZd48803+e677/D09MTf358mTZrw66+/Gte9atUqGjZsyN69e43T4uLi6NWr1wNratiwIRkZGaxbt47BgwcXmB8fH8+0adMset5F9eWXX9KtWze6dOlCVFQUf+lv4ivxWDRo0EClp6eb3d7T01MdOXLkEVYkLLV+/XrVv3//QudNnjxZhYWFKYPBoJRS6tKlS8rDw0Pt3bu3BCu8rWHDhiolJcWidRw4cEB5eHio5ORk47TY2FjVpk0blZ6ervbv3698fHwKtAsICFAxMTFKKaXGjh2r5syZo5RSatiwYWrRokXG5QYPHqwGDx6sJk+ebJw2ceJEFRUV9cC67nyu1q5dq4KDgy15ihbZtWuX6tGjh/rzzz9VTk6OeuONN9SmTZtKrR5L2ZZ2EP0dhYSEANC/f3/mzZuHVqslNDSUlJQU8vLy8PHxYciQIeTn5zN16lQOHTpEuXLl+Mc//kF4eDjz5s0jNTWV0aNHExERwYwZM3jjjTdo2rQpAwYMwMPDg8OHD3P9+nVGjhyJt7c32dnZTJo0icOHD+Po6Ei9evUA+Oijj0xqy8/P5+OPP2bXrl3Y2NjQokULJk2axNy5c7l69SoffPABAFFRUcbHAQEBVK5cmVOnTvHqq6/y+eefs3fvXsqXL49er8fT05OFCxdSo0YNwsLCOH78OHl5ebRv3573338fW1vTt1lmZiZTpkzht99+Q6PR4O7uzqhRo7C1teX5558nODiYhIQEUlNT6devHwMGDCjQx4cPH2batGlkZ2dTrlw53n//fdq3b0/jxo3p378/iYmJZGVlMWrUKLy8vFi3bh1bt25l7ty5AAUe323VqlUsWbIErVZLtWrVmDhxIpcvX2bWrFlkZmYSEBDAkiVLTNpcuXKFqlWrkpeXR/ny5alRowZRUVE8+eSTAOzcuZO5c+eSm5tLRkYG/v7+jBgxgsTERMLCwqhYsSJZWVksW7aM8ePHc/bsWbRaLU2aNCE0NBSADz/8kMOHD/Pnn3+ilGLatGm0bNnSpI7XX38dpRSDBg1i0qRJVK5cmdDQUK5du4ZGoyEoKAh/f/8C212zZg3ly5c3ruezzz7jrbfeomnTpsZp3bt3x87ODr1eb7LNsWPHUr9+fS5fvkxycjIXLlzgypUrxMfHY2dnR2ZmJi+++CK7du1iwIAB5OTkcOTIERYvXszAgQOZNGkSAPv37+eTTz7h9OnThIaGkpWVRWpqKo0aNWLWrFnY2dkVeK0AtmzZwowZM5g3bx4//fST8XUNCAigefPmHDp0iJSUFFq2bMn06dPRarWsW7eOefPmUaFCBdq1a8fixYv55ZdfTNZrMBjw9PRk9uzZPP/88wCMHDmS1q1b88svv+Dr60vFihUBePnll9mwYQPe3t6F1ljmlXYS/V3dfUQQEBCg4uPjlVJK5eTkqICAALVp0yaVlJSkunXrZtyLjIiIUD/88INSyvSIoG/fviouLk6dP39eNWjQQO3YsUMppdSWLVvUSy+9pJRSasaMGWrUqFFKr9erzMxM5efnp8aMGVOgrq+++kq98cYbKjs7W+n1evXuu++q9evXq88++0xNmTLFuNzdj/v27atCQkKM89544w0VFxenlLq9Z9SnTx+l1O09wMWLFyullMrPz1ejR49W8+bNK1DD+++/r6ZOnaoMBoO6deuWCgoKUnPnzjX225IlS5RSSiUnJ6umTZuqnJwck/a5ubmqQ4cOaufOncblfH19lV6vVw0aNFBffPGFUkqpX3/9VbVs2bLQPcj77VHu27dPde7c2fjarV27Vul0OmUwGB64F/rrr78qLy8v1aJFCxUUFKRmz56tTp06pZRSymAwqL59+6rTp08rpW4fLTz33HPGPetGjRqpCxcuKKVuH3UEBQUZ+3D8+PHqzJkz6tChQ2r48OFKr9crpZSaO3euGjx4cKG13Hnv5eXlqU6dOqmtW7cat+vu7q4OHTpUYLv3at68uTpx4kSh85RSJkcEY8aMUQsWLFBK/e+9eu/0lJQU1aZNG6XX61V8fLwaNmyYUkqprl27qp9//ln98ccfytXVVRkMBvXRRx8Zjypyc3OVr6+v2rJli8lzu/NabNiwQfn4+KiLFy8aX687r1Hfvn3VO++8Y/xMuLm5qe+//16dOHFCtW/f3njUFBUVpRo0aFDo84yMjDR+Dq5du6batGmjbty4oYKCgtTGjRuNyyUkJCh/f//79ldZJ0cEj1lWVhZJSUlcv36dyMhI47TffvsNNzc3bGxs6N27N25ubnTt2pUXXnjhgesrV64cHh4eADRu3Jhr164BsHv3bkJCQtBqtTg4ONCzZ0+OHTtWoP2+ffvo0aMHFSpUAGDWrFnA7SOAB2nVqpXx/71792b9+vV069aNdevW0bt3bwB27dpFcnIya9asASAnJ6fQde3Zs4cVK1ag0WgoX748ffr04auvviI4OBiATp06AdCkSRNyc3PJysoy2Rs8fvw4Wq2Wl156CYCmTZvyzTffGOf37dsXgEaNGtGgQQOSkpIe+NzutnfvXry9valSpQpwe08vLCyMCxcuPLBdo0aN2LJlCz///DNJSUkkJCTw5ZdfEhkZSceOHfnyyy/ZtWsXGzdu5OTJkyilyM7OBqBWrVrUrl0bgJYtWzJz5kwCAgJwdXWlf//+ODs74+zsTOXKlVm5ciXnz58nMTGRSpUqPbCmM2fOcOvWLby8vACoUaMGXl5e7N27l7Zt25ps915arRaDwVDkfnuYmjVrUr16dY4dO8bOnTuNr52npyffffcd1apV48UXX0Sj0fDee++RkJDA/PnzOXPmDKmpqWRlZRVYZ3JyMnv37mXcuHHUqlWr0O16enoaPxPOzs5cv36d3377jQ4dOlCzZk3g9vvlfu//V155hV69ejF27Fg2btyIp6cnjo6OhZ4P0Gr/uqdcJQgeM4PBgFKKlStXYm9vD0BGRgZ2dnZUqlSJ2NhYDh06xP79+xkxYsR9h0LuKFeunPENp9FojNNtbW1N3pz3e1PeO0yTlpaGwWBAo9GYtM/LyzNZ7s4hMEC3bt0IDw/n5MmTJCUlGYefDAYDkZGR/POf/wTgxo0bJjXe3Sf3Ps7Pzzc+vvNH/07bez90NjY2BdZ7/PhxXFxcjPPvXved5R/0/O4o7AOulDKp7175+flMmTKF//znPzRt2pSmTZsSGBjI559/zqpVq2jXrh09e/akc+fOtGrVildeeYVvv/3WuK27+/aZZ55h+/btJCYmsn//fgIDA5kwYQIVKlQgLCyMwMBAOnXqhIuLCxs2bLhvTXee+4Oey93bvVfz5s05fPgwDRo0MJk+ZcoUunTpYtLHReXu7s6BAwfYvXs37777LgAeHh5ER0fzxBNPGHcARo0ahV6vR6fT8dJLL5GSklLo6+Lo6Mgnn3zCiBEjeOmll/jHP/5RYJk7OzyA8T1gY2Njsr67n8ugQYNITU0F4J133qFTp040btyYXbt2sW7dOsaNGwfcDu8rV64Y212+fNkYLH9Ff90IK+NsbGzIz8/HwcGB5s2bs2jRIuD2H8fXXnuN+Ph4du7cyYABA2jRogXDhw/H39+f3377zaR9UXl4eLB27VoMBgPZ2dls3Lix0D/C7du3Z+PGjeTm5mIwGJg8eTKbNm3iqaee4ueff0YpRVZWFt999919t2VnZ4ePjw9jx47Fy8vLGHBubm5ER0ejlCI3N5ehQ4eydOnSAu3d3NxYtmyZcbnVq1fj6upa5Ofq4uKCRqMhISEBgJ9//pn+/fsb//Dd+QbLzz//zOnTp2ndujVVqlThxIkT3Lp1i/z8fHbu3Fnout3c3Ni8ebPxG19r167lySefxNnZ+b712NracubMGT7//HNjwOTn53P+/HkaN27M2bNnuXnzJiNGjKBjx44cOHDA2P/3Wr58OSEhIbi5ufHee+/h5ubGiRMnSEhIwNPTk9dff53nn3+eb7/9tsBY/b3q1q1LuXLl2LZtG3D7j9XWrVuL1NdDhw5l9uzZxm+uwf/Oq9wbDkX14osvsnbtWqpXr061atWA20eax48f58cff6RDhw4AfPfddwwbNgxvb280Gg2HDx8u9LnWqVOH9u3bExAQwJgxY4p8BOPm5sb333/P5cuXAUy+3TV//nxiY2OJjY01BtO///1v5s+fT05OjvGcTKdOndiwYQNZWVnk5uaybt06OnfubFa/lAVyRPCYdOnShddff53PP/+cGTNmMHXqVPz8/MjNzTV+BVGv17Nnzx7jSafKlSszdepUADp37szIkSOL/FW4wYMHExoaip+fH46OjlStWtVkb+iOPn368Mcff/Dyyy+jlKJNmzYEBASQnZ3N3r178fLyokaNGrRo0eKBX4fr3bs3S5cuZfLkycZp48ePJywsDD8/P/Ly8nB1dWXgwIEF2k6YMIFp06YZl3N3d2fIkCFFep4A5cuXJyoqig8//JCIiAjKlStHVFSU8WTnoUOHWL16NQaDgZkzZ1K5cmU6dOhA69at0el0ODk50bZt20KHzjp06MCAAQOMwVKlShXmzp370MP+yMhIPv74Y7p27Yq9vT1KKTp37sywYcOMw1g6nY4nnniCZ599lnr16nH27FmTE7QA/v7+HDhwAG9vb+zt7Xn66afp168faWlpjB49Gj8/P2xsbGjVqhXbtm3DYDDct7Zy5crx+eefM23aNKKiotDr9QwbNox27dqRmJj4wOfTqlUrpk2bRlhYGFlZWeTl5fHss8+yePFiqlWrxsmTJx/YvjAtW7bkwoULBAUFGafd+YLAtWvXcHBwAG6fkB02bBiVK1fG3t6e1q1bc+7cufuud8iQIezYsYMFCxYYA+ZB6tatS0hICG+++Sbly5fnueeeM+7MFKZjx45MmTKFQYMGmUw7fvw4vXv3Ji8vj06dOuHv71+EXiibNOpBn3bxl7Fp0yYcHBzw8PDAYDAwfPhwOnTowOuvv17apZWohg0b8v333xvH+IW41/nz54mNjeWtt95Cq9Wybds25s+fX2q/+ygL5Ijgb6J+/fp88MEHfPrpp+Tl5dG2bVvjSVwhxP/UrFmT1NRU49GVo6MjH374YWmXVarkiEAIIaycnCwWQggrJ0EghBBWToJACCGsnMVBEBkZWeiv8i5dukSbNm0K/UWmXq9n0qRJ+Pr64uPjQ3R0tKVlCCGEMJPZ3xrKzMwkPDycTZs2FfiuuMFgYPz48ff99ea6deu4du0aGzZsICcnh169etG6dWuaNGlS5O1fvfonBkPxz3NXrepAevrNYrcrKVKfZaQ+y0h9linL9Wm1Gp56qvDLkpgdBPHx8dSpU4fAwMAC8xYsWICrqyunT58utG39+vVp3rw5Wq2WihUr8swzz5CSklKsIDAYlFlBcKdtWSb1WUbqs4zUZ5myXl9hzA6CO7+iu3dY6OjRoyQmJjJ//nyWLVtWaNvmzZsb/3/o0CGOHDlCREREsbZftapDsZa/m5OTo9ltS4LUZxmpzzJSn2XKen2FeWgQxMXFER4ebjLNxcWl0HH97OxsQkNDmTVrVpGuxHfgwAFGjRrFjBkzqFy5ctGrBtLTb5qVvE5Ojly5klnsdiVF6rOM1GcZqc8yZbk+rVZz3x3ohwaBTqdDp9MVaUMHDx4kLS2NoUOHApCamkpwcDCzZ882Xhnyjm3btjF58mRmzpxJ27Zti7R+IcTfg1KKq1evkJubAxR9hy419dFeHvtRK/36NJQvX4GnnnIq9KKT9/NILzHh7u7Ojh07jI87duzIvHnzClwe9siRI0yePJmFCxfSqFGjR1mCEOIv4ObN62g0GmrU+AcaTdG/vGhrqyU/v+wGQWnXp5SBa9fSuHnzOo6OTxa5XYn9jiA5Odl49b4vvvgCvV7PmDFj6NGjBz169CA+Pr6kShFClLLs7Js4Oj5ZrBAQD6fRaHF0fIrs7OJ9c+kve60hOUdQOqQ+y0h9t126dJYaNZ4t1vAFlP4e98OUhfqUUly+fI6aNU3vofGgcwQSx0KIUlHcEBBFY06/ShAIIUQp6tXLj5SUi6VagwSBEEJYObkxjRDC6h06dJAvvvgMvd5ArVq1sLevyKlTJzEYDLzxRj86duxCjx7dWL06hooVKzF0aBAdOrxI374D+Pbbrfz0048MHfo206dP4/Lly6SlXaF58xZMmBDKjz/+YFy3i8s/eeedUYSGTiQ19TJ16riQm5sLwO+/nyAiIgy9Xk/58uUZN24SzzzzbIk8fwkCIUSpS0hO4bsjKQ9dTqOB4n69xe2FWnR4vtZDlzt//hxr1mxkyZJFVKvmxIQJU/jzz5sMGRJE48ZNadmyFT/+eIgWLVqSkpLCTz8dom/fAezfv49Onbqwb9931K/fgNDQj8jLy6Nv394cO/abybodHBz49NPpNGjQiBkzPuOnnw6xY8d2AFavXk6fPn3p2LEz8fHb+PnnZAkCIYQoSc8844yDgwMHDx7g1q0cNm3aAEBOTg6nT5+ifXs3fvjhAFqtBi8vHfHx28jPz+fw4Z94771x2NnZcezYL6xevZwzZ05z/fp1srOzTNYN8OOPPzB58u1bYzZv/i+efro2AO3bd+DTTyNITNyHq6s7L73UqcSeuwSBEKLUdXi+aHvtj/PrmXZ2dgAYDHomTpxKw4a3f+yakZHOE09UJjMzk5Url2FjY0vLlq05d+4MGzfG4OLigp2dHWvWrGT37h34+fWkV682nD59kjvfzr+zbrj9rZ67f31sY2MDgKdnZ5o2fYGEhL18/fUK9u9PYMyYCY/lud5LThYLIcRd/vWv1sTErAEgLS2N/v1f4/LlSzz11FPY2dmRkLCHF15ozr/+1Zro6P/i6uoOQFJSIv7+r+DlpQM0nDhxvNDLTbRq1YZt2+IA+PXXn/njj9v3bPnggxB++eVn/P1fYeDAIcZhpZIgQSCEEHcJChrErVu3CAj4N+++O4S33nqH2rVvXyanffsOODg4UrFiRVq2bE1a2hVcXd0A+Pe/X+e//51HUNAbfPrpdJo2faHQr4W++eZg/vjjAn37/pulS6ONQ0MBAYEsWbKIoKA3mDNnFsOHjyyx5yy/LC5jpD7LSH2WKclfFt/7y9eiKAu/3H2QslJfYf0rvywWQghxXxIEQghh5SQIhBDCykkQCCGElZMgEEIIKydBIIQQVk6CQAghrJzFQRAZGUlUVFSB6ZcuXaJNmzZcuHDhge3feeedQtsLIcTfzcWLfxAeHgrcvuLp228Hm72uzZu/ISxs8iOpy+wgyMzMZNy4cSxcuLDAPIPBwPjx48nLy3vgOtasWUNiYqK5JQghxF/KpUspxktKlCVmX3QuPj6eOnXqEBgYWGDeggULcHV15fTp0/dtf/bsWdavX0+fPn3MLUEI8TeRdzyBvGN7HrqcRqOhuBdDKNfwRco16PDAZVJTLxMaOpHs7Gy0Wg3vvvsekyePo2PH25eXtrGxYfDgYaxcuZQLF84zbNgIOnXqQkZGOh99NJXLly9hY2PD0KFv07p1e3Jycpg+fRq//34crVZLnz590el8iYycwcWLf/DJJ9Px9OzEtWvXGD36Hf744wLPPuvM1KnTKV++PHFxG/n66xUYDIqGDRsxatQY7Ozs2LJlE1999V8qVXKgZs2a2NtXLFZf3I/ZRwT+/v4EBwcbr5x3x9GjR0lMTCw0IO7Iz89nwoQJTJkyBVtbuQCqEKJ0bdwYi6urG//97xKGDn2HI0d+AqBaNSeWLl1Nw4aNWLo0mk8/nc3EiaEsXboIgJkzP+Zf/2rFV1+tZOrU6YSFTSEjI52FC+dSuXJllixZTWTklyxcOJ/ffz/Bu++OpmHD5/jPf8YAcPnyJUaNGsOyZWvIyEjn4MEDnDp1km++ieGLLxYSHb2cp56qwooVS0hLu8IXX3zGnDnz+fLLhWRlZT2y5//Qv8JxcXGEh4ebTHNxcSE6OrrAstnZ2YSGhjJr1iy02vtnTFRUFF26dKFevXrFr/j/3e+aGUXh5ORodtuSIPVZRuqzTEnUl5qqxdb2f38jbBu7Y9/Y/bFv937atm3H2LGj+f3347i6uvHqq31Yt241bm5u2NpqqVWrFtWrV6dChfLUrl2bzMxMbG21HDp0kPHjJ2Jrq8XZ+VmaNHme33775f+nT8LWVku1alXw8PDg8OFD1KtXH41Gg62tFhsbLfXrN+DZZ58BoG5dFzIzr3P58kUuXDjPkCG3d6bz8vJo2LARv/ySzPPPN6N6dScAdDpvkpKSTPrxDq1WW6zX8aFBoNPp0Ol0RVrZwYMHSUtLY+jQoQCkpqYSHBzM7NmzcXFxMS63detWypcvz9q1a0lLSwPA3t6egQMHFrlwuehc6ZD6LCP13WYwGMy6ONvjuqhbkyYvsHTpavbt+47t27eycePtm9JoNDbk5xswGBQaze1t6/W3t397uoH8fGWsSSlFXl6e8fndma7X356u1xtQShnXo9Vq72oLer2BvDw9HTt2ZsSI9wDIyspCr9fzww8H0Ovv7jetcV33MhgMBV7HErvonLu7Ozt27CA2NpbY2FiqV6/OvHnzTEIAYMuWLWzYsIHY2Fj69OlDnz59ihUCQgjxKH3+eSRbt25Gp/Nl5MgxHD9+rEjtWrZsxcaNMQD88ccFjhz5iSZNXuBf/2rNpk2xAFy7do29e3fRokUrbGxs0ev1D1xnixYt2bNnF1evZqCU4pNPwlm9ejkvvNCcX35J5sqVVAwGg/EWl49CiQ3QJycn89lnnzF//vyS2qQQQhTJK6+8ypQpE9i8eSNarZb//GcsX3zx2UPbjRjxHhERYWze/A0ajYZx4z6gWrVqBAYO5JNPptOv36sYDAb69QuiYcNGXL9+jZs3M5k6dSI+Pj0KXWf9+g0IDBzEO+8MQSlF/foN6dt3AHZ2dowY8R4jRrxFhQr21KlT95E9f7kfQRkj9VlG6rOM3I/AMmWlPrkfgRBCiGKRIBBCCCsnQSCEKBV/0VHpMs+cfpUgEEKUOK3WBr0+v7TL+FvS6/PRam0evuBdJAiEECXO3t6BzMxrKFX6J1b/TpQykJl5FXv74v3gVq7vIIQocQ4Olbl69QqXL18Aij6UodVqMRjKbniUfn0aypevgIND5WK1kiAQQpQ4jUZDlSrVi91Ovn77eMjQkBBCWDkJAiGEsHISBEIIYeUkCIQQwspJEAghhJWTIBBCCCsnQSCEEFZOgkAIIaycBIEQQlg5CQIhhLByFgdBZGQkUVFRBaZfunSJNm3acOHChULbrV69mp49e9K1a1e5faUQQpQis4MgMzOTcePGsXDhwgLzDAYD48ePJy8vr9C2Bw8eZOHChSxevJh169bx9ddf8/vvv5tbihBCCAuYfdG5+Ph46tSpQ2BgYIF5CxYswNXVldOnTxfaNi4ujtdffx1HR0cAFi5cyJNPPmluKUIIISxg9hGBv78/wcHB2NiY3gDh6NGjJCYmFhoQd5w9e5aMjAz69u1Ljx492LlzJw4Oxbt+thBCiEfjoUcEcXFxhIeHm0xzcXEhOjq6wLLZ2dmEhoYya9YstNr7Z4xer+fQoUPMnTuX/Px8+vbtS/369WnXrl2RC69a1fzgcHJyNLttSZD6LCP1WUbqs0xZr68wDw0CnU6HTqcr0soOHjxIWloaQ4cOBSA1NZXg4GBmz56Ni4uLcblq1arRpEkTKlWqBIC7uzvJycnFCoL09JsYDMW/N2dZv1641GcZqc8yUp9lynJ9Wq3mvjvQj/Tro+7u7uzYsYPY2FhiY2OpXr068+bNMwkBAE9PT7Zv305ubi45OTns37+fpk2bPspShBBCFFGJ/Y4gOTmZQYMGAeDt7Y27uzs9e/akR48edO7cmfbt25dUKUIIIe6iUUoVf3ylDJChodIh9VlG6rOM1Ge+EhsaEkII8dcjQSCEEFZOgkAIIaycBIEQQlg5CQIhhLByEgRCCGHlJAiEEMLKSRAIIYSVkyAQQggrJ0EghBBWToJACCGsnASBEEJYOQkCIYSwchIEQghh5SQIhBDCykkQCCGElZMgEEIIKydBIIQQVs7iIIiMjCQqKqrA9EuXLtGmTRsuXLhQaLsFCxbg7e2Nt7c3ixYtsrQMIYQQZjI7CDIzMxk3bhwLFy4sMM9gMDB+/Hjy8vIKbXv27FmWL1/OunXrWLNmDYsXL+bs2bPmliKEEMICZgdBfHw8derUITAwsMC8BQsW4OrqylNPPVVoW4PBQF5eHrdu3SI3NxelFLa2tuaWIoQQwgJm//X19/cHKDAsdPToURITE5k/fz7Lli0rtG3dunXx9fXF09MTpRS9e/emdu3axdp+1aoOZtUN4OTkaHbbkiD1WUbqs4zUZ5myXl9hHhoEcXFxhIeHm0xzcXEhOjq6wLLZ2dmEhoYya9YstNr7H2zs2bOHo0ePsnfvXpRSDBo0iM2bN+Pt7V3kwtPTb2IwqCIvf4eTkyNXrmQWu11JkfosI/VZRuqzTFmuT6vV3HcH+qFBoNPp0Ol0RdrQwYMHSUtLY+jQoQCkpqYSHBzM7NmzcXFxMS63c+dOunbtSqVKlQDw9fUlKSmpWEEghBDi0XikA/Pu7u7s2LHD+Lhjx47MmzePf/zjHybLNWrUiG3btvHaa69hMBjYs2cPXbt2fZSlCCGEKKIS+x1BcnIygwYNAqB3797Uq1cPHx8fevbsSb169ejZs2dJlSKEEOIuGqVU8QfaywA5R1A6pD7LSH2WkfrM96BzBPLLYiGEsHISBEIIYeUkCIQQwspJEAghhJWTIBBCCCsnQSCEEFZOgkAIIaycBIEQQlg5CQIhhLByEgRCCGHlJAiEEMLKSRAIIYSVkyAQQggrJ0EghBBWToJACCGsnASBEEJYOQkCIYSwchbfszgyMhKtVsvw4cMBSEpK4u2336ZmzZoANG7cmPDwcJM2ubm5jB8/nqNHj1KhQgVmzJjBP//5T0tLEUIIYQazgyAzM5Pw8HA2bdrEwIEDjdOTk5MJCgpi8ODB9227ZMkS7O3tiYuLIykpibFjx/L111+bW4oQQggLmD00FB8fT506dQgMDDSZnpycTEJCAv7+/gwZMoSUlJQCbXft2kX37t0BaN26NVevXuXixYvmliKEEMICZh8R+Pv7AxAVFWUy3dHRER8fHzp37syKFSsYOXIkK1euNFkmNTUVJycn42MnJycuXbrE008/bW45RfLzjk3YnvkeZcZN70vKMa1G6rOA1GcZqc8yj7s+vYsrTTr6PPL1PjQI4uLiCozxu7i4EB0dXejyoaGhxv+/9tprfPLJJ2RmZuLo6PjA7Wi1xTs4qVrVoVjLA9hXKEceoNFqit22JEl9lpH6LCP1WeZx1mdfoRxOTg/+W2qOhwaBTqdDp9MVaWUGg4G5c+cSHByMjY3N/zZia7qZ6tWrc+XKFZydnQG4cuUK1atXL07dpKffxFDM5HVx9cKpxytcuZJZrHYlycnJUeqzgNRnGanPMiVRn7nr12o1992BfqRfH9VqtWzfvp2tW7cCEBMTQ7NmzbC3tzdZzsPDg9jYWAAOHjyInZ3dYx8WEkIIUbhH/juC6dOns3jxYnx8fFi7di3Tpk0DYMWKFURGRgIQEBBAbm4uPj4+hIWFERER8ajLEEIIUUQapVTZPfPyAOYMDYEcWlpK6rOM1GcZqc98JTY0JIQQ4q9HgkAIIaycBIEQQlg5CQIhhLByEgRCCGHlJAiEEMLKSRAIIYSVkyAQQggrJ0EghBBWToJACCGsnASBEEJYOQkCIYSwchIEQghh5SQIhBDCykkQCCGElZMgEEIIKydBIIQQVu6hN69/mMjISLRaLcOHDwcgKSmJt99+m5o1awLQuHFjwsPDTdqkpqYSEhJCWloaWq2W999/n/bt21taihBCCDOYHQSZmZmEh4ezadMmBg4caJyenJxMUFAQgwcPvm/biIgIPD096du3L6dOnSIgIIA9e/ZgY2NjbjlCCCHMZHYQxMfHU6dOHQIDA02mJycnk56eTlxcHDVr1mTSpEnUqlXLZBkvLy/atm0LgLOzM7du3SIrKwtHR0dzyxFCCGEms88R+Pv7ExwcXGAv3tHRkX79+hETE4OHhwcjR44s0NbLy4vKlSsD8N///pfnnntOQkAIIUqJRimlHrRAXFxcgTF+FxcXoqOjAYiKigIwniO4V6tWrdi5c2ehf+ijo6NZsmQJS5cuLXDUIIQQomQ8dGhIp9Oh0+mKtDKDwcDcuXMLHCnY2hbcTEREBLt372bZsmXGE8vFkZ5+E4PhgRlWKCcnR65cySx2u5Ii9VlG6rOM1GeZslyfVquhalWHwuc92g1p2b59O1u3bgUgJiaGZs2aYW9vb7JcdHQ0iYmJrFixwqwQEEII8ehY/PXRe02fPp2JEycyZ84cqlSpQkREBAArVqwgNTWVd955hzlz5uDg4EBAQICx3bx586hRo8ajLkcIIcRDPPQcQVklQ0OlQ+qzjNRnGanPfCU2NCSEEOKvR4JACCGsnASBEEJYOQkCIYSwchIEQghh5SQIhBDCykkQCCGElZMgEEIIKydBIIQQVk6CQAghrJwEgRBCWDkJAiGEsHISBEIIYeUkCIQQwspJEAghhJWTIBBCCCsnQSCEEFZOgkAIIaycxUEQGRlJVFSU8XFSUhJt27alR48e9OjRg5CQkPu2vXnzJp07dyYxMdHSMoQQQpjJ7JvXZ2ZmEh4ezqZNmxg4cKBxenJyMkFBQQwePPih65g6dSo3btwwtwQhhBCPgNlHBPHx8dSpU4fAwECT6cnJySQkJODv78+QIUNISUkptP3mzZupVKkSDRs2NLcEIYQQj4DZRwT+/v4AJsNCAI6Ojvj4+NC5c2dWrFjByJEjWblypckyFy9e5KuvvuKrr75i0KBBZm2/alUHs9oBODk5mt22JEh9lpH6LCP1Waas11eYhwZBXFwc4eHhJtNcXFyIjo4udPnQ0FDj/1977TU++eQTMjMzcXS83TkGg4Hx48czceJEKlSoYHbh6ek3MRhUsds5OTly5Uqm2dt93KQ+y0h9lpH6LFOW69NqNffdgX5oEOh0OnQ6XZE2ZDAYmDt3LsHBwdjY2PxvI7b/28ypU6c4deoU48ePB+DcuXNMmDCBqVOn0q5duyJtRwghxKNj9tBQYbRaLdu3b8fZ2Rlvb29iYmJo1qwZ9vb2xmXq1avH7t27jY8DAgJ4++23adu27aMsRQghRBE98t8RTJ8+ncWLF+Pj48PatWuZNm0aACtWrCAyMvJRb04IIYSFNEqp4g+0lwFyjqB0SH2WkfosI/WZ70HnCOSXxUIIYeUkCIQQwspJEAghhJWTIBBCCCsnQSCEEFZOgkAIIaycBIEQQlg5CQIhhLByEgRCCGHlJAiEEMLKSRAIIYSVkyAQQggrJ0EghBBWToJACCGsnASBEEJYOQkCIYSwchIEQghh5Sy+Z3FkZCRarZbhw4cDkJSUxNtvv03NmjUBaNy4MeHh4SZtcnNziYiI4ODBg+Tl5RESEoKbm5ulpQghhDCD2UGQmZlJeHg4mzZtYuDAgcbpycnJBAUFMXjw4Pu2XbBgAVevXmX9+vX8/vvvBAUFsWfPHjQajbnlCCGEMJPZQ0Px8fHUqVOHwMBAk+nJyckkJCTg7+/PkCFDSElJKdA2Li6OQYMGodFoqF+/PosWLeIveutkIYT4yzM7CPz9/QkODsbGxsZkuqOjI/369SMmJgYPDw9GjhxZoO3Zs2dJSkri5Zdf5tVXXyUtLQ2tVk5XCCFEadCoh+yKx8XFFRjjd3FxITo6GoCoqCgA4zmCe7Vq1YqdO3fi6OhonNakSRMGDBjA6NGjOXbsGAMHDiQuLs5kGSGEECXjoecIdDodOp2uSCszGAzMnTu3wJGCra3pZqpVq4aPjw8ajYZGjRpRs2ZNTp8+zQsvvFDkwtPTb2IwFH84ycnJkStXMovdrqRIfZaR+iwj9VmmLNen1WqoWtWh8HmPdkNatm/fztatWwGIiYmhWbNm2Nvbmyzn6enJ5s2bATh//jwpKSnUrVv3UZYihBCiiB75wPz06dNZvHgxPj4+rF27lmnTpgGwYsUKIiMjARg9ejSpqan4+PgwZMgQpk2bJsNCQghRSh56jqCskqGh0iH1WUbqs4zUZ74SGxoSQgjx1yNBIIQQVk6CQAghrJwEgRBCWDkJAiGEsHISBEIIYeUkCIQQwspJEAghhJWTIBBCCCsnQSCEEFZOgkAIIaycBIEQQlg5CQIhhLByEgRCCGHlJAiEEMLKSRAIIYSVkyAQQggr99Cb1z9MZGQkWq2W4cOHA5CUlMTbb79NzZo1AWjcuDHh4eEmbXJzcwkJCeH48eNotVrGjBmDq6urpaUIIYQwg9lBkJmZSXh4OJs2bWLgwIHG6cnJyQQFBTF48OD7to2NjcVgMPDNN99w7NgxBg0axJ49e8wtRQghhAXMHhqKj4+nTp06BAYGmkxPTk4mISEBf39/hgwZQkpKSoG2BoOB7Oxs9Ho92dnZVKhQwdwyhBBCWMjsIPD39yc4OBgbGxuT6Y6OjvTr14+YmBg8PDwYOXJkgbY9e/bk2rVruLu707dvX0aPHm1uGUIIISykUUqpBy0QFxdXYIzfxcWF6OhoAKKiogCM5wju1apVK3bu3Imjo6Nx2qeffkpubi5jxozhzJkzDBgwgOXLl1O7dm1LnosQQggzPPQcgU6nQ6fTFWllBoOBuXPnFjhSsLU13Ux8fDwzZ85Eo9FQt25dmjVrxpEjR4oVBOnpNzEYHphhhXJycuTKlcxityspUp9lpD7LSH2WKcv1abUaqlZ1KHSexd8aMt2Qlu3bt+Ps7Iy3tzcxMTE0a9YMe3t7k+UaNWrEt99+S4MGDcjIyODo0aOMGjWqmNvSWFCn+W1LgtRnGanPMlKfZcpqfQ+q66FDQw9z79DQiRMnmDhxIpmZmVSpUoWIiAhq1arFihUrSE1N5d133yUtLY2JEydy7tw5tFotgwcPxtfX15IyhBBCmMniIBBCCPHXJr8sFkIIKydBIIQQVk6CQAghrJwEgRBCWDkJAiGEsHISBEIIYeUkCIQQwspJEAghhJX72wbBN998g7e3N126dGHZsmUF5v/666+88sordO3alfHjx5Ofn1+i9c2ePRsfHx98fHyIiIgodL6npyc9evSgR48ehT6Hx6lfv374+PgYt3/48GGT+fv27cPPzw8vLy9mzpxZorV9/fXXxrp69OhBy5YtCQ0NNVmmtPrv5s2b+Pr6cuHCBaBo/XTx4kXeeOMNunXrxtChQ/nzzz9LrL5Vq1bh6+uLn58fISEh5ObmFmgTExODm5ubsS8f5+t9b30hISF4eXkZt719+/YCbUrys3x3fbt37zZ5H7Zr167Q+7CUZP+ZTf0NXbp0SXl6eqqrV6+qP//8U/n5+akTJ06YLOPj46N+/PFHpZRSISEhatmyZSVWX0JCgnr11VfVrVu3VG5ururXr5/atm2byTKDBw9Whw4dKrGa7mYwGFSHDh1UXl5eofOzs7OVh4eHOnfunMrLy1NBQUFq165dJVzlbcePH1ddunRR6enpJtNLo/9++ukn5evrq5o0aaLOnz9f5H4KDg5WGzduVEopNXv2bBUREVEi9Z06dUp16dJFZWZmKoPBoN5//321aNGiAu1CQ0PVN99881hqelB9Sinl6+urLl++/MB2JfVZLqy+O1JTU1WnTp3U6dOnC7Qrqf6zxN/yiGDfvn20a9eOJ598kooVK9K1a1e2bNlinP/HH3+Qk5ND8+bNAXj55ZdN5j9uTk5OjB07lvLly1OuXDn++c9/cvHiRZNljh49yvz58/Hz8yM0NJRbt26VWH2nTp1Co9EwaNAgunfvztKlS03mHzlyBGdnZ5555hlsbW3x8/Mr0f672+TJkxk5ciRVqlQxmV4a/bd69WomTZpE9erVgaL1U15eHklJSXTt2hV4vO/Fe+srX748kydPxsHBAY1GQ4MGDQq8D+H2zaZiYmLo3r07o0eP5vr16yVSX1ZWFhcvXmTixIn4+fnx2WefYTAYTNqU5Gf53vruFhERQZ8+fahTp06BeSXVf5b4WwZBamoqTk5OxsfVq1fn8uXL953v5ORkMv9xq1+/vvGNe+bMGTZv3oyHh4dx/p9//slzzz3HmDFjWL9+PTdu3ODzzz8vsfpu3LhB+/btmTNnDtHR0axcuZKEhATj/If1b0nZt28fOTk5BS6TXlr9FxYWRqtWrYyPi9JPV69excHBwXip9sf5Xry3vtq1axvvFZ6RkcGyZcvo1KlTgXZOTk4MHz6c2NhYatWqVWAY7nHVl56eTrt27fjwww9ZvXo1Bw8eZM2aNSZtSvKzfG99d5w5c4YDBw7Qr1+/QtuVVP9Z4m8ZBKqQ6+hpNJoizy8pJ06cICgoiDFjxpjsSVSqVIn58+fj7OyMra0tQUFB7N69u8TqatGiBREREVSsWJEqVarQq1cvk+2Xlf5buXJlgVulQun33x1F6aey0JeXL1+mf//+vPLKK7Rt27bA/Dlz5tCsWTM0Gg0DBw4ssfuLP/PMM8yZM4eqVatib29PQEBAgdexLPTfqlWreP311ylfvnyh80ur/4rjbxkENWrUIC0tzfg4NTXV5HDu3vlXrlwp9HDvcfrhhx8YMGAA//nPf+jZs6fJvIsXL5rs+SilCtzc53E6ePAg33///X23/7D+LQm5ubkkJSXRsWPHAvNKu//uKEo/ValShZs3b6LX64GSfy+ePHmS1157jZ49ezJs2LAC8zMzM413I4SS7ctjx46xdevWB267LHyW4+Pj8fb2LnReafZfcfwtg8DV1ZXvv/+ejIwMsrOz2bZtGy+++KJxfu3atbGzs+OHH34Abp/Vv3v+45aSksKwYcOYMWMGPj4+BeZXqFCBjz/+mPPnz6OUYtmyZXTp0qXE6svMzCQiIoJbt25x8+ZN1q9fb7L9Zs2acfr0ac6ePYter2fjxo0l2n9w+49EnTp1qFixYoF5pd1/dxSln8qVK0erVq3YvHkzULLvxZs3b/Lmm2/y7rvvEhQUVOgyFStWZMGCBcZvjS1durTE+lIpxYcffsj169fJy8tj1apVBbZd2p/ljIwMcnJyeOaZZwqdX5r9VyylcIK6RGzYsEH5+PgoLy8vNW/ePKWUUgMHDlRHjhxRSin166+/qldeeUV169ZNjRo1St26davEaps6dapq3ry56t69u/Hf8uXLTerbsmWLsf6xY8eWaH1KKTVz5kzVrVs35eXlpaKjo5VSSnXv3l1dunRJKaXUvn37lJ+fn/Ly8lJhYWHKYDCUaH2bNm1SI0aMMJlWVvrP09PT+K2S+/XTuHHj1LfffquUUurChQuqb9++SqfTqaCgIHXt2rUSqW/RokWqSZMmJu/DWbNmFagvKSlJ+fv7q27duqkhQ4aoGzdulEh9Sim1dOlSpdPpVJcuXdTHH39sXKY0P8t313f48GHVu3fvAsuUZv+ZQ25MI4QQVu5vOTQkhBCi6CQIhBDCykkQCCGElZMgEEIIKydBIIQQVk6CQAgLDBo0iN9//71YbQYPHsy6deseU0VCFF/Z+4mbEH8h8+fPL+0ShLCYBIGwSjt27OCLL74gLy+PChUqMGbMGL777jtOnDhBWloa6enpNGrUiLCwMBwcHFi+fDkrV66kXLly2NnZERoaSr169ejYsSORkZE8//zzrFq1iiVLlqDVaqlWrRoTJ06kbt26XL58mbFjx5KamsrTTz9Nenq6sY6TJ08SFhbGtWvX0Ov1BAQE0KtXL/78809CQkI4e/YsWq2WJk2aEBoailYrB/HiMSjtX7QJUdJOnz6tfH19VUZGhlLq9j0NOnTooD766CP14osvqitXrii9Xq9GjRqlPvroI5Wfn6+aNGlivC7++vXr1cqVK5VSt39leuTIEbVv3z7VuXNn430R1q5dq3Q6nTIYDOqtt95SM2fOVEopdebMGdW8eXO1du1alZeXp7y9vdXRo0eVUkrduHFD6XQ69eOPP6r169eroKAgpZRS+fn5avz48erMmTMl2U3CisgRgbA6CQkJpKamMmDAAOM0jUbDuXPn6NatG9WqVQOgV69efPjhh4wZM4Zu3brRp08fXnrpJTp06ICfn5/JOvfu3Yu3t7fxvggvv/wyYWFhXLhwgX379jFmzBgAnJ2djVf4PHPmDOfOnWPcuHHG9eTk5PDLL7/g7u7OzJkzCQgIwNXVlf79++Ps7Pw4u0VYMQkCYXUMBgPt27dn1qxZxmkpKSmsWrXK5FaNBoPBOBQzY8YMjh8/zr59+5g/fz5r1qzhiy++MC6rCrlSi1KK/Px8NBqNyfw7V5/U6/U88cQTxMbGGuelpaXh6OiInZ0d27dvJzExkf379xMYGMiECRPo1q3bI+sHIe6QAUdhddq1a0dCQgInT54EYPfu3XTv3p1bt24RHx9PZmYmBoOB1atX4+npSUZGBh4eHjz55JMMGDCAESNGcOzYMZN1urm5sXnzZjIyMgBYu3YtTz75JM7Ozri7u7Nq1Srg9iWyExMTAahbty52dnbGIEhJScHX15ejR4+yfPlyQkJCcHNz47333sPNzY0TJ06UVBcJKyMXnRNWKS4uji+//NJ4ffhx48bx/fffs3//fvR6PVevXqV169ZMmDCBChUqsHLlShYvXkyFChWwsbFh5MiRuLq6mpwsXrZsGStXrsRgMFClShU++OAD6tevT0ZGBiEhIZw7d46aNWuSn59Pz549efnll/ntt9+MJ4vz8/Pp168fr732GllZWYwbN45jx45hb2/P008/TVhYGJUrVy7trhN/QxIEQvy/qKgorl69ygcffFDapQhRomRoSAghrJwcEQghhJWTIwIhhLByEgRCCGHlJAiEEMLKSRAIIYSVkyAQQggrJ0EghBBW7v8Ab4ItqHBRUq8AAAAASUVORK5CYII=", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "# 获取参数\n", - "cfg = get_args() \n", - "# 训练\n", - "env, agent = env_agent_config(cfg)\n", - "res_dic = train(cfg, env, agent)\n", - " \n", - "plot_rewards(res_dic['rewards'], cfg, tag=\"train\") \n", - "# 测试\n", - "res_dic = test(cfg, env, agent)\n", - "plot_rewards(res_dic['rewards'], cfg, tag=\"test\") # 画出结果" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.12 ('rl_tutorials')", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.12" - }, - "orig_nbformat": 4, - "vscode": { - "interpreter": { - "hash": "4f613f1ab80ec98dc1b91d6e720de51301598a187317378e53e49b773c1123dd" - } - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/projects/notebooks/3.DQN.ipynb b/projects/notebooks/3.DQN.ipynb deleted file mode 100644 index 6b73846..0000000 --- a/projects/notebooks/3.DQN.ipynb +++ /dev/null @@ -1,541 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1、定义算法\n", - "\n", - "教程中提到相比于Q learning，DQN本质上是为了适应更为复杂的环境，并且经过不断的改良迭代，到了Nature DQN（即Volodymyr Mnih发表的Nature论文）这里才算是基本完善。DQN主要改动的点有三个：\n", - "* 使用深度神经网络替代原来的Q表：这个很容易理解原因\n", - "* 使用了经验回放（Replay Buffer）：这个好处有很多，一个是使用一堆历史数据去训练，比之前用一次就扔掉好多了，大大提高样本效率，另外一个是面试常提到的，减少样本之间的相关性，原则上获取经验跟学习阶段是分开的，原来时序的训练数据有可能是不稳定的，打乱之后再学习有助于提高训练的稳定性，跟深度学习中划分训练测试集时打乱样本是一个道理。\n", - "* 使用了两个网络：即策略网络和目标网络，每隔若干步才把每步更新的策略网络参数复制给目标网络，这样做也是为了训练的稳定，避免Q值的估计发散。想象一下，如果当前有个transition（这个Q learning中提过的，一定要记住！！！）样本导致对Q值进行了较差的过估计，如果接下来从经验回放中提取到的样本正好连续几个都这样的，很有可能导致Q值的发散（它的青春小鸟一去不回来了）。再打个比方，我们玩RPG或者闯关类游戏，有些人为了破纪录经常Save和Load，只要我出了错，我不满意我就加载之前的存档，假设不允许加载呢，就像DQN算法一样训练过程中会退不了，这时候是不是搞两个档，一个档每帧都存一下，另外一个档打了不错的结果再存，也就是若干个间隔再存一下，到最后用间隔若干步数再存的档一般都比每帧都存的档好些呢。当然你也可以再搞更多个档，也就是DQN增加多个目标网络，但是对于DQN则没有多大必要，多几个网络效果不见得会好很多。" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 1.1、定义模型\n", - "\n", - "前面说了DQN的模型不再是Q表，而是一个深度神经网络，这里我只用了一个三层的全连接网络（FCN），这种网络也叫多层感知机（MLP），至于怎么用Torch写网络这里就不多说明了，以下仅供参考。" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [], - "source": [ - "import torch.nn as nn\n", - "import torch.nn.functional as F\n", - "class MLP(nn.Module):\n", - " def __init__(self, n_states,n_actions,hidden_dim=128):\n", - " \"\"\" 初始化q网络，为全连接网络\n", - " \"\"\"\n", - " super(MLP, self).__init__()\n", - " self.fc1 = nn.Linear(n_states, hidden_dim) # 输入层\n", - " self.fc2 = nn.Linear(hidden_dim,hidden_dim) # 隐藏层\n", - " self.fc3 = nn.Linear(hidden_dim, n_actions) # 输出层\n", - " \n", - " def forward(self, x):\n", - " # 各层对应的激活函数\n", - " x = F.relu(self.fc1(x)) \n", - " x = F.relu(self.fc2(x))\n", - " return self.fc3(x)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 1.2、定义经验回放\n", - "\n", - "经验回放首先是具有一定容量的，只有存储一定的transition网络才会更新，否则就退回到了之前的逐步更新了。另外写经验回放的时候一般需要包涵两个功能或方法，一个是push，即将一个transition样本按顺序放到经验回放中，如果满了就把最开始放进去的样本挤掉，因此如果大家学过数据结构的话推荐用队列来写，虽然这里不是。另外一个是sample，很简单就是随机采样出一个或者若干个（具体多少就是batch_size了）样本供DQN网络更新。功能讲清楚了，大家可以按照自己的想法用代码来实现，参考如下。" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": {}, - "outputs": [], - "source": [ - "from collections import deque\n", - "import random\n", - "class ReplayBuffer(object):\n", - " def __init__(self, capacity: int) -> None:\n", - " self.capacity = capacity\n", - " self.buffer = deque(maxlen=self.capacity)\n", - " def push(self,transitions):\n", - " ''' 存储transition到经验回放中\n", - " '''\n", - " self.buffer.append(transitions)\n", - " def sample(self, batch_size: int, sequential: bool = False):\n", - " if batch_size > len(self.buffer): # 如果批量大小大于经验回放的容量，则取经验回放的容量\n", - " batch_size = len(self.buffer)\n", - " if sequential: # 顺序采样\n", - " rand = random.randint(0, len(self.buffer) - batch_size)\n", - " batch = [self.buffer[i] for i in range(rand, rand + batch_size)]\n", - " return zip(*batch)\n", - " else: # 随机采样\n", - " batch = random.sample(self.buffer, batch_size)\n", - " return zip(*batch)\n", - " def clear(self):\n", - " ''' 清空经验回放\n", - " '''\n", - " self.buffer.clear()\n", - " def __len__(self):\n", - " ''' 返回当前存储的量\n", - " '''\n", - " return len(self.buffer)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 1.3、真定义算法\n", - "\n", - "到了高级一点的算法，定义算法就比较麻烦，要先定义一些子模块，再定义好子模块之后我们就可以实现我们的算法核心部分。如下，可以看到，其实去掉子模块的话，DQN跟Q learning的算法结构没啥区别，当然因为神经网络一般需要Torch或者Tensorflow来写，因此推荐大家先去学一学这些工具，比如\"eat_pytorch_in_20_days\"。\n", - "\n", - "这里我们主要分析一下DQN的更新过程，也就是update函数。首先我们知道目前所有基于深度神经网络的更新方式都是梯度下降，如下：\n", - "$$\n", - "\\theta_i \\leftarrow \\theta_i - \\lambda \\nabla_{\\theta_{i}} L_{i}\\left(\\theta_{i}\\right)\n", - "$$\n", - "那么这个$\\theta$又是什么呢，注意到前面我们讲的DQN跟Q learning算法的一个主要区别就是使用神经网络替代了Q表，而这个$\\theta$实际上就是神经网络的参数，通常用$Q\\left(s_{i}, a_{i} ; \\theta\\right)$表示。根据强化学习的原理我们需要优化的是对应状态下不同动作的长期价值，然后每次选择价值最大对应的动作就能完成一条最优策略，使用神经网络表示Q表时也是如此，我们将输入的状态数作为神经网络的输入层，动作数作为输出层，这样的神经网络表达的功能就跟在Q learning中的Q表是一样的，只不过具有更强的鲁棒性。\n", - "\n", - "讲完了为什么要优化的是这个参数$\\theta$，接下来我们从代码层面进一步剖析，稍微了解一点Torch知识的同学都知道，上面的公式其实只需要定义一个优化器，然后计算损失之后用优化器迭代即可，如下：\n", - "```python\n", - "optimizer = optim.Adam(Q_net.parameters(), lr=0.01) # 定义优化器，对应的网络是Q_net，学习率为0.01\n", - "loss = ... # 计算损失，这里掠过\n", - "# 然后优化器先zero_grad()，loss再反向传播，然后优化器step() ，这是一个固定的套路\n", - "optimizer.zero_grad() \n", - "loss.backward()\n", - "optimizer.step() \n", - "```\n", - "当然强烈建议同学们了解一下深度学习中的梯度下降，并且使用numpy实现，这样就会更加清楚整个梯度下降过程到底是怎么回事，上述只是在同学们了解了梯度下降的具体实现方式的前提下为了方便学习更多其他的知识形成的套路。这就好比我们玩一个竞技游戏，如果我们之前从来没有接触过该类游戏，那么肯定是从普通攻击，每个技能一步一步地学起打好基础，然后再学习技能连招等等也就是形成固定的套路，但是如果不先打基础，直接学习套路可能会是一脸懵逼的状态，尤其是很多高端玩家会对这些连招套路简化名称比如光速qa和1233321等等，一开始我们是很难听懂的。等当我们先打好基础，然后再学习了很多套路之后会发现这些基础并不能用得上，甚至有的时候可能会忽然忘记了这些基础，但其实我们并没有忘记，再回顾一遍也能很快拣起来。在这点上我想强调的是基础固然重要，但是不要死磕基础，除非是学术研究需要。再比如我们小学学完简单加减乘除之后很快就去背九九乘法表，而不会去过多纠结一加一等于几的问题，上大学后也是如此，只是很多时候我们很可能看起来这个问题值得研究，但意识不到自己就是在纠结一加一等于几的问题，这也是我在和众多读者们学习讨论的过程中在他们身上发现的问题。\n", - "\n", - "回归正题，细心的同学会发现数学公式和代码的对应是有一定的壁垒的，只要通过多加练习跨越了这个壁垒，那么对于往后我们想要复现论文也会轻松许多。我们目前讲了参数的更新过程，但是最关键的是损失是如何计算的，在DQN中损失的计算相对来说比较简单，如下：\n", - "$$\n", - "L(\\theta)=\\left(y_{i}-Q\\left(s_{i}, a_{i} ; \\theta\\right)\\right)^{2}\n", - "$$\n", - "这里的$y_{i}$通常称为期望值，$Q\\left(s_{i}, a_{i} ; \\theta\\right)$称为实际值，这个损失在深度学习中通常称作均方差损失，也就是mseloss，使用这个损失函数通常追溯到数学上的最小二乘法，感兴趣的同学可以了解一下深度学习中的各种损失函数以及各自的使用场景。\n", - "$y_{i}$在DQN中一般表示如下：\n", - "$$\n", - "y_{i}= \\begin{cases}r_{i} & \\text {对于终止状态} s_{i+1} \\\\ r_{i}+\\gamma \\max _{a^{\\prime}} Q\\left(s_{i+1}, a^{\\prime} ; \\theta\\right) & \\text {对于非终止状态} s_{i+1}\\end{cases}\n", - "$$\n", - "该公式的意思就是将下一个状态对应的最大Q值作为实际值（因为实际值通常不能直接求得，只能近似），这种做法实际上只是一种近似，可能会导致过估计等问题，也有一些改善的方法具体可以在后面各种改进的DQN算法比如Double DQN中看到，在这里我们暂时不要深究为什么要用这个来近似实际值。然后注意到这里其实有一个终止状态的判断，因为如果当前状态是终止状态，那么实际上是没有下一个状态的，所以DQN干脆直接使用对应的奖励表示Q的实际值。" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [], - "source": [ - "import torch\n", - "import torch.optim as optim\n", - "import math\n", - "import numpy as np\n", - "class DQN:\n", - " def __init__(self,model,memory,cfg):\n", - "\n", - " self.n_actions = cfg['n_actions'] \n", - " self.device = torch.device(cfg['device']) \n", - " self.gamma = cfg['gamma'] # 奖励的折扣因子\n", - " # e-greedy策略相关参数\n", - " self.sample_count = 0 # 用于epsilon的衰减计数\n", - " self.epsilon = cfg['epsilon_start']\n", - " self.sample_count = 0 \n", - " self.epsilon_start = cfg['epsilon_start']\n", - " self.epsilon_end = cfg['epsilon_end']\n", - " self.epsilon_decay = cfg['epsilon_decay']\n", - " self.batch_size = cfg['batch_size']\n", - " self.policy_net = model.to(self.device)\n", - " self.target_net = model.to(self.device)\n", - " # 复制参数到目标网络\n", - " for target_param, param in zip(self.target_net.parameters(),self.policy_net.parameters()): \n", - " target_param.data.copy_(param.data)\n", - " self.optimizer = optim.Adam(self.policy_net.parameters(), lr=cfg['lr']) # 优化器\n", - " self.memory = memory # 经验回放\n", - " def sample_action(self, state):\n", - " ''' 采样动作\n", - " '''\n", - " self.sample_count += 1\n", - " # epsilon指数衰减\n", - " self.epsilon = self.epsilon_end + (self.epsilon_start - self.epsilon_end) * \\\n", - " math.exp(-1. * self.sample_count / self.epsilon_decay) \n", - " if random.random() > self.epsilon:\n", - " with torch.no_grad():\n", - " state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(dim=0)\n", - " q_values = self.policy_net(state)\n", - " action = q_values.max(1)[1].item() # choose action corresponding to the maximum q value\n", - " else:\n", - " action = random.randrange(self.n_actions)\n", - " return action\n", - " @torch.no_grad() # 不计算梯度，该装饰器效果等同于with torch.no_grad()：\n", - " def predict_action(self, state):\n", - " ''' 预测动作\n", - " '''\n", - " state = torch.tensor(state, device=self.device, dtype=torch.float32).unsqueeze(dim=0)\n", - " q_values = self.policy_net(state)\n", - " action = q_values.max(1)[1].item() # choose action corresponding to the maximum q value\n", - " return action\n", - " def update(self):\n", - " if len(self.memory) < self.batch_size: # 当经验回放中不满足一个批量时，不更新策略\n", - " return\n", - " # 从经验回放中随机采样一个批量的转移(transition)\n", - " state_batch, action_batch, reward_batch, next_state_batch, done_batch = self.memory.sample(\n", - " self.batch_size)\n", - " # 将数据转换为tensor\n", - " state_batch = torch.tensor(np.array(state_batch), device=self.device, dtype=torch.float)\n", - " action_batch = torch.tensor(action_batch, device=self.device).unsqueeze(1) \n", - " reward_batch = torch.tensor(reward_batch, device=self.device, dtype=torch.float) \n", - " next_state_batch = torch.tensor(np.array(next_state_batch), device=self.device, dtype=torch.float)\n", - " done_batch = torch.tensor(np.float32(done_batch), device=self.device)\n", - " q_values = self.policy_net(state_batch).gather(dim=1, index=action_batch) # 计算当前状态(s_t,a)对应的Q(s_t, a)\n", - " next_q_values = self.target_net(next_state_batch).max(1)[0].detach() # 计算下一时刻的状态(s_t_,a)对应的Q值\n", - " # 计算期望的Q值，对于终止状态，此时done_batch[0]=1, 对应的expected_q_value等于reward\n", - " expected_q_values = reward_batch + self.gamma * next_q_values * (1-done_batch)\n", - " loss = nn.MSELoss()(q_values, expected_q_values.unsqueeze(1)) # 计算均方根损失\n", - " # 优化更新模型\n", - " self.optimizer.zero_grad() \n", - " loss.backward()\n", - " # clip防止梯度爆炸\n", - " for param in self.policy_net.parameters(): \n", - " param.grad.data.clamp_(-1, 1)\n", - " self.optimizer.step() " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2、定义训练" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [], - "source": [ - "def train(cfg, env, agent):\n", - " ''' 训练\n", - " '''\n", - " print(\"开始训练！\")\n", - " rewards = [] # 记录所有回合的奖励\n", - " steps = []\n", - " for i_ep in range(cfg['train_eps']):\n", - " ep_reward = 0 # 记录一回合内的奖励\n", - " ep_step = 0\n", - " state = env.reset() # 重置环境，返回初始状态\n", - " for _ in range(cfg['ep_max_steps']):\n", - " ep_step += 1\n", - " action = agent.sample_action(state) # 选择动作\n", - " next_state, reward, done, _ = env.step(action) # 更新环境，返回transition\n", - " agent.memory.push((state, action, reward,next_state, done)) # 保存transition\n", - " state = next_state # 更新下一个状态\n", - " agent.update() # 更新智能体\n", - " ep_reward += reward # 累加奖励\n", - " if done:\n", - " break\n", - " if (i_ep + 1) % cfg['target_update'] == 0: # 智能体目标网络更新\n", - " agent.target_net.load_state_dict(agent.policy_net.state_dict())\n", - " steps.append(ep_step)\n", - " rewards.append(ep_reward)\n", - " if (i_ep + 1) % 10 == 0:\n", - " print(f\"回合：{i_ep+1}/{cfg['train_eps']}，奖励：{ep_reward:.2f}，Epislon：{agent.epsilon:.3f}\")\n", - " print(\"完成训练！\")\n", - " env.close()\n", - " return {'rewards':rewards}\n", - "\n", - "def test(cfg, env, agent):\n", - " print(\"开始测试！\")\n", - " rewards = [] # 记录所有回合的奖励\n", - " steps = []\n", - " for i_ep in range(cfg['test_eps']):\n", - " ep_reward = 0 # 记录一回合内的奖励\n", - " ep_step = 0\n", - " state = env.reset() # 重置环境，返回初始状态\n", - " for _ in range(cfg['ep_max_steps']):\n", - " ep_step+=1\n", - " action = agent.predict_action(state) # 选择动作\n", - " next_state, reward, done, _ = env.step(action) # 更新环境，返回transition\n", - " state = next_state # 更新下一个状态\n", - " ep_reward += reward # 累加奖励\n", - " if done:\n", - " break\n", - " steps.append(ep_step)\n", - " rewards.append(ep_reward)\n", - " print(f\"回合：{i_ep+1}/{cfg['test_eps']}，奖励：{ep_reward:.2f}\")\n", - " print(\"完成测试\")\n", - " env.close()\n", - " return {'rewards':rewards}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 3. 定义环境" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": {}, - "outputs": [], - "source": [ - "import gym\n", - "import os\n", - "def all_seed(env,seed = 1):\n", - " ''' 万能的seed函数\n", - " '''\n", - " env.seed(seed) # env config\n", - " np.random.seed(seed)\n", - " random.seed(seed)\n", - " torch.manual_seed(seed) # config for CPU\n", - " torch.cuda.manual_seed(seed) # config for GPU\n", - " os.environ['PYTHONHASHSEED'] = str(seed) # config for python scripts\n", - " # config for cudnn\n", - " torch.backends.cudnn.deterministic = True\n", - " torch.backends.cudnn.benchmark = False\n", - " torch.backends.cudnn.enabled = False\n", - "def env_agent_config(cfg):\n", - " env = gym.make(cfg['env_name']) # 创建环境\n", - " if cfg['seed'] !=0:\n", - " all_seed(env,seed=cfg['seed'])\n", - " n_states = env.observation_space.shape[0]\n", - " n_actions = env.action_space.n\n", - " print(f\"状态空间维度：{n_states}，动作空间维度：{n_actions}\")\n", - " cfg.update({\"n_states\":n_states,\"n_actions\":n_actions}) # 更新n_states和n_actions到cfg参数中\n", - " model = MLP(n_states, n_actions, hidden_dim = cfg['hidden_dim']) # 创建模型\n", - " memory = ReplayBuffer(cfg['memory_capacity'])\n", - " agent = DQN(model,memory,cfg)\n", - " return env,agent" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 4、设置参数" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "metadata": {}, - "outputs": [], - "source": [ - "import argparse\n", - "import matplotlib.pyplot as plt\n", - "import seaborn as sns\n", - "def get_args():\n", - " \"\"\" 超参数\n", - " \"\"\"\n", - " parser = argparse.ArgumentParser(description=\"hyperparameters\") \n", - " parser.add_argument('--algo_name',default='DQN',type=str,help=\"name of algorithm\")\n", - " parser.add_argument('--env_name',default='CartPole-v0',type=str,help=\"name of environment\")\n", - " parser.add_argument('--train_eps',default=200,type=int,help=\"episodes of training\")\n", - " parser.add_argument('--test_eps',default=20,type=int,help=\"episodes of testing\")\n", - " parser.add_argument('--ep_max_steps',default = 100000,type=int,help=\"steps per episode, much larger value can simulate infinite steps\")\n", - " parser.add_argument('--gamma',default=0.95,type=float,help=\"discounted factor\")\n", - " parser.add_argument('--epsilon_start',default=0.95,type=float,help=\"initial value of epsilon\")\n", - " parser.add_argument('--epsilon_end',default=0.01,type=float,help=\"final value of epsilon\")\n", - " parser.add_argument('--epsilon_decay',default=500,type=int,help=\"decay rate of epsilon, the higher value, the slower decay\")\n", - " parser.add_argument('--lr',default=0.0001,type=float,help=\"learning rate\")\n", - " parser.add_argument('--memory_capacity',default=100000,type=int,help=\"memory capacity\")\n", - " parser.add_argument('--batch_size',default=64,type=int)\n", - " parser.add_argument('--target_update',default=4,type=int)\n", - " parser.add_argument('--hidden_dim',default=256,type=int)\n", - " parser.add_argument('--device',default='cpu',type=str,help=\"cpu or cuda\") \n", - " parser.add_argument('--seed',default=10,type=int,help=\"seed\") \n", - " args = parser.parse_args([])\n", - " args = {**vars(args)} # 转换成字典类型 \n", - " ## 打印超参数\n", - " print(\"超参数\")\n", - " print(''.join(['=']*80))\n", - " tplt = \"{:^20}\\t{:^20}\\t{:^20}\"\n", - " print(tplt.format(\"Name\", \"Value\", \"Type\"))\n", - " for k,v in args.items():\n", - " print(tplt.format(k,v,str(type(v)))) \n", - " print(''.join(['=']*80)) \n", - " return args\n", - "def smooth(data, weight=0.9): \n", - " '''用于平滑曲线，类似于Tensorboard中的smooth曲线\n", - " '''\n", - " last = data[0] \n", - " smoothed = []\n", - " for point in data:\n", - " smoothed_val = last * weight + (1 - weight) * point # 计算平滑值\n", - " smoothed.append(smoothed_val) \n", - " last = smoothed_val \n", - " return smoothed\n", - "\n", - "def plot_rewards(rewards,cfg, tag='train'):\n", - " ''' 画图\n", - " '''\n", - " sns.set()\n", - " plt.figure() # 创建一个图形实例，方便同时多画几个图\n", - " plt.title(f\"{tag}ing curve on {cfg['device']} of {cfg['algo_name']} for {cfg['env_name']}\")\n", - " plt.xlabel('epsiodes')\n", - " plt.plot(rewards, label='rewards')\n", - " plt.plot(smooth(rewards), label='smoothed')\n", - " plt.legend()\n", - " plt.show()\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 5、我准备好了！" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "超参数\n", - "================================================================================\n", - " Name \t Value \t Type \n", - " algo_name \t DQN \t \n", - " env_name \t CartPole-v0 \t \n", - " train_eps \t 200 \t \n", - " test_eps \t 20 \t \n", - " ep_max_steps \t 100000 \t \n", - " gamma \t 0.95 \t \n", - " epsilon_start \t 0.95 \t \n", - " epsilon_end \t 0.01 \t \n", - " epsilon_decay \t 500 \t \n", - " lr \t 0.0001 \t \n", - " memory_capacity \t 100000 \t \n", - " batch_size \t 64 \t \n", - " target_update \t 4 \t \n", - " hidden_dim \t 256 \t \n", - " device \t cpu \t \n", - " seed \t 10 \t \n", - "================================================================================\n", - "状态空间维度：4，动作空间维度：2\n", - "开始训练！\n", - "回合：10/200，奖励：14.00，Epislon：0.611\n", - "回合：20/200，奖励：10.00，Epislon：0.470\n", - "回合：30/200，奖励：11.00，Epislon：0.372\n", - "回合：40/200，奖励：18.00，Epislon：0.302\n", - "回合：50/200，奖励：15.00，Epislon：0.228\n", - "回合：60/200，奖励：62.00，Epislon：0.121\n", - "回合：70/200，奖励：128.00，Epislon：0.039\n", - "回合：80/200，奖励：200.00，Epislon：0.011\n", - "回合：90/200，奖励：200.00，Epislon：0.010\n", - "回合：100/200，奖励：200.00，Epislon：0.010\n", - "回合：110/200，奖励：200.00，Epislon：0.010\n", - "回合：120/200，奖励：200.00，Epislon：0.010\n", - "回合：130/200，奖励：200.00，Epislon：0.010\n", - "回合：140/200，奖励：200.00，Epislon：0.010\n", - "回合：150/200，奖励：200.00，Epislon：0.010\n", - "回合：160/200，奖励：200.00，Epislon：0.010\n", - "回合：170/200，奖励：200.00，Epislon：0.010\n", - "回合：180/200，奖励：200.00，Epislon：0.010\n", - "回合：190/200，奖励：200.00，Epislon：0.010\n", - "回合：200/200，奖励：200.00，Epislon：0.010\n", - "完成训练！\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAHJCAYAAACrCBICAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAACVtklEQVR4nOzdd3gU1frA8e9sSzaNdELvHUIHQUBEBVQUEa/3XgEbNrzKT6ygWBErIiIioCgWuFbsXrGjWGii0kIvoSQB0tu2md8fm51k0xM2ZZf38zw8JDOzO+dkdrNv3vOeM4qmaRpCCCGEEI2QoaEbIIQQQghREQlUhBBCCNFoSaAihBBCiEZLAhUhhBBCNFoSqAghhBCi0ZJARQghhBCNlgQqQgghhGi0JFARQgghRKMlgYooQ9YAFELUhvzuEHVBAhXh5bvvvuO+++7zyXOtXr2aLl26cOTIkTp9jDizrFixgrPPPpvExEQWL15c7jFdunTx+te9e3cGDx7M9ddfzw8//FDuYzRN49NPP+Xqq6/mrLPOom/fvowbN44XX3yRjIyMcs8xcuRIcnNzy+w7cuQIXbp0YfXq1ZX25dlnn2XQoEH06dOHjz/+uOrO14LNZmPFihVMnDiR/v37M2jQIP71r3/x8ccf+yywSElJ4aabbuLo0aP6tilTppS5Dj179mTkyJE8+uijZGVl1egcM2fOZNSoUT5pb3Vs3bqVKVOm0LdvX4YNG8b8+fOx2+31dn5RzNTQDRCNy4oVK3z2XCNHjuTdd98lPj6+Th8jzhy5ubk8/fTTjBw5kuuvv56WLVtWeOwVV1zBP/7xDwAcDgcnTpzgww8/5JZbbuGBBx7g6quv1o+12+3ccccdrF27lokTJ3LdddcRHBzM1q1befPNN1m9ejVLliyhS5cuXuc4fvw4Tz31FI8//niN+7J7925effVVrrzySsaPH0/79u1r/BxVOXnyJDfccAPHjx9nypQpJCYmoqoqP/zwAzNnzmTTpk3MmTMHRVFO6zy//vora9euLbO9e/fuPPzww/r3DoeD7du3M3/+fHbu3Ml///vf0z53XUhOTua6666jT58+LFiwgH379vH888+TmZnJY4891tDNO+NIoCLqTHR0NNHR0XX+GHHmyMrKQlVVzj//fAYOHFjpsQkJCfTp08dr20UXXcTtt9/OM888w6hRo/RAZ+HChaxdu5alS5cybNgw/fghQ4YwYcIEJk+ezPTp0/nkk08IDg7W90dERPD+++9z4YUXcvbZZ9eoL5mZmQBcfPHFDBgwoEaPra777ruPlJQU3n33Xdq2batvHzlyJM2bN2f+/Pmce+65nHfeeXVy/rCwsDLXYODAgeTl5bFw4UL++uuvMvsbg1deeYXQ0FAWL16MxWLhnHPOITg4mDlz5nDLLbfQvHnzhm7iGUWGfoRuypQpbNiwgQ0bNtClSxfWr1/P+vXr6dKlC++88w7nnnsu/fr145dffgHg/fff5/LLL6dPnz4kJiYyfvx4/ve//+nPV3oYZ+bMmVx77bV8+OGHjBkzhp49ezJ+/Hh++umn03oMwJYtW5g0aRJ9+vRh5MiRvPHGG1x77bXMnDmz0j7/+eefXH/99fTr14+zzjqLO++8k9TU1HLb4jFq1Civ5+3SpQuLFi3i8ssvJzExkUWLFtGtWzfefvttr8elp6fTo0cPPWulqirLli3jggsuoGfPnowZM4a33nqryuuUk5PDk08+yfnnn0+vXr0YN24cH3zwQZk2Lly4kKeffpqhQ4eSmJjI1KlTOXjwYKXPbbfbWbBgAeeddx6JiYmMGzeOjz76SN8/ZcoUZs6cyZIlSxg6dCj9+/fn1ltv9Ur5l5eir+5QyC+//MJVV11F//79GTx4MHfddRfHjx8H3NfD87z3339/mexGdc2YMQOHw6H/zHJzc3nzzTeZOHGiV5DiERcXxwMPPMDBgwf5/PPPvfb985//pF27dsyePbvcIaCKvPjii0yZMgWAa665Ru+Xy+Vi5cqVXHLJJSQmJjJy5EjmzZuHzWbTHztz5kyuueYaHn74Yfr168dFF12Ey+Uqc46dO3eybt06pk6d6hWkeFx77bVMmjSJkJAQfdvGjRuZOnUqAwcOpGfPnowaNYoXX3wRVVWB4uv4+uuvM3bsWHr37s2HH37IrFmzADjvvPOqfM8B9OzZE4Bjx47p27788ksuv/xy+vbty9lnn81DDz1U5fDQ+++/z8UXX6wPKb344ovl/ixKGjNmDNOnTy+zffz48UybNg2AdevWcc4552CxWPT9Y8eORVVV1q1bV2X/hG9JoCJ0Dz/8MN27d6d79+68++679OjRQ9+3aNEi7rvvPh566CH69u3LypUreeihhzj//PNZunQp8+bNw2KxcPfdd5OSklLhObZt28by5cuZPn06L730Ekajkdtvv73SX0hVPWbfvn1ce+21AMyfP5/bb7+dZcuWsXnz5kr7u2PHDiZPnozNZuOZZ57h0UcfZdu2bUydOhWn01mDnxwsWbKESy65hIULFzJmzBgGDRrEF1984XXMV199haZpXHzxxQA88sgjLFy4kEsvvZQlS5YwduxYnnjiCV566aUKz1NYWMhVV13FZ599xg033MDixYvp378/DzzwAEuWLPE69s0332T//v08+eSTPP7442zbtq3K+qO7776b119/nX/84x96dmHmzJleH9Dfffcdq1evZvbs2Tz66KPs3LmTKVOmUFBQUKOfWWkff/wx119/Pc2aNWP+/PnMmjWLLVu28M9//pNTp04xcuRIFi1aBMC0adN49913a3We9u3b07x5c/318csvv2Cz2Tj//PMrfMywYcOIjIzk22+/9doeFBTEk08+SUpKCs8880y12/CPf/yDhx56CICHHnpI79dDDz2kB6Evv/wykyZN4u233+bWW2/1qifZtGkTx48f56WXXuKuu+7CaDSWOcfPP/8MUGFdR1BQEA899BBDhgwBICkpiWuvvZbIyEief/55Xn75ZQYMGMCiRYu8/gABd6B144038swzzzB06FD9A37RokXceuutVfb/wIEDALRq1QqAxYsXc+edd9KnTx8WLlzIf/7zH9asWcOUKVMoLCws9zmWLl3Kgw8+yJAhQ1iyZAmTJk3ilVde4cEHH6z03Jdeeilr1671Ciz37dtHUlIS48ePp7CwkKNHj9KuXTuvx0VHRxMWFqa3XdQfGfoRuo4dOxIWFgZQJh171VVXMXbsWP375ORkpk6d6vVLqUWLFlx++eVs3rxZ/zAuLScnh9WrV9O6dWsAQkJCmDx5Mr///jtjxoyp1WOWLl1KeHg4r776KlarFXB/GP3rX/+qtL9LliwhMjKS1157jaCgIADi4+O566672LNnT6WPLW3AgAFcd911+vfjx4/n/vvv59ixY3qa+IsvvmDo0KHExcVx4MAB3nvvPe68805uuukmwP1hqCgKS5cu5aqrriIqKqrMeVavXs3u3bt555136Nu3LwDDhw/H6XSyePFi/vWvfxEZGQm4hyUWL16sf4gdPnxYLwwt77l3797NmjVruP/++7nmmmsA99DH0aNHWb9+PePGjQOgoKCA1atX6x8y7du3Z8KECXz88cf8+9//rtHPzUNVVebNm8ewYcN47rnn9O2ejMHy5cu599576datGwCtW7c+rSGD2NhYTp48CRT/Vd+iRYsKjzcYDLRo0cIrc+TRt29frrnmGj3LMHTo0CrPn5CQQMeOHQH3+6579+7s3buXDz74gLvuukt/TZx99tnEx8dz77338tNPP3HOOecA4HQ6eeyxx0hISKjwHJ5MVGV1PCUlJSUxdOhQnn32WQwGg37+77//nvXr13u9py+88EImTpyof+95b3br1s3rfJqmeQX9WVlZbNiwgZdffpm+ffvSs2dPsrKyePnll7nyyiv14A2gc+fOTJo0iQ8//JBJkyZ5tTUnJ4fFixfzz3/+k9mzZwPFweTs2bO57rrr6NSpU7n9vPTSS3nxxRf59ttvueyyywD4/PPPiYiIYNSoUfofQJ7fhSWFhobWKHMmfEMyKqJaPB8QHjNnzuTuu+8mOzubP//8k08++YSVK1cCVFoZHx0drf9SA/RftJX9NV7VY37//XdGjBihByng/vCo7IMHYPPmzYwYMUIPUjyP+/7778v0tyqljx89ejRBQUF8+eWXgPtDY/PmzYwfP15vs6ZpjBo1CqfTqf8bNWoUNputwmzQhg0baNGihR6keFx66aXYbDb++usvfVuvXr28/tKu6mftOefo0aO9tr/44ovMmTNH/75fv356kALugslWrVqxcePGcp+3Og4cOMCJEyf0YMijdevW9O3blw0bNtT6ucujaVqNizgNBoM+BFLaHXfcQdu2bWs8BFSSp4+lg/yLL74Yo9HI+vXr9W2RkZGVBimAfu2rGgrxuOyyy3jllVdwOBwkJSWxZs0aFi5ciMvlwuFweB1b3ffHxo0b6dGjh/5v6NCh3HnnnfTs2ZPnnnsORVH4888/sdvtZa79gAEDaNGiRbnXfsuWLRQWFpb7/gF3lswTJJX8p2karVq1ol+/fvp7E9x/RIwdOxaLxVLhNfZojMW/gU4yKqJaSo5jg/uv84ceeojffvsNs9lM+/bt6dq1K1D5WgolgwkoftNX9suhqsekp6cTExNT5nGxsbEVPie4ixnLe1xtlP75hIWFcf755/PFF19www038OWXX2K1WvXhhZKFlOXx1MmUlpWVRVxcXJntnr5mZ2fr20r/3Dx/JVf0s/a0qaqfSdOmTctsi4mJqfF00/LOXd41i42NZceOHbV+7vKkpKTQuXNnAD3jdeTIETp06FDu8ZqmceTIEXr16lXu/uDgYJ544gkmT57MM888o2dEasLz8yt9fU0mE1FRUeTk5OjbQkNDq3w+T6B+7NgxPXtTWmpqKvHx8SiKQmFhIXPmzOGTTz7B6XTSsmVL+vbti8lkKvOeLv16r0iPHj149NFHAff7NigoiGbNmnllKzz9rujal+y3h+f1UtHPOS0tjQ0bNnjN7AL3cOjgwYMZP348c+bMISMjgyNHjnDo0CGeeOIJoDiTkpeXV+Z5c3NzCQ8Pr6rbwsckUBE1pqoqN910E2azmQ8++IBu3bphMpnYu3cvn3zySb23JyEhQU/jl3Tq1KlKp3yGh4eTnp5eZvvatWvp1q1bhUFUeb/AynPppZdy0003cejQIb744gvGjBmjBw8REREAvPHGG+V+6FQ0q6BJkyYcOnSozPYTJ04AlDukU12eNqWnp3v9tb5v3z4yMzPp378/QLlripw8eVLPeimKUuav+Pz8/ErP7RmuKu86njhx4rT6VdrevXs5ceKEPpxw9tlnExwczNdff60PrYA7GI+MjCQiIoKNGzeSkZHBiBEjKnze/v37M2XKFN58880KA5rKNGnSBHD3t2Q20OFwVDhcVxlPYfDatWvLDVScTifjx4+nX79+LF68mLlz57JmzRoWLFjA0KFD9WDEU8NSG6GhoVX+LDz9PnnyZJn364kTJ7yydx6e1+q8efPKLRSOjY0lPDy8TJG5p+7kwgsv5PHHH+fbb79l//79tGjRQn99h4aG0rRp0zLvs1OnTpGXl1dhMCvqjgz9CC+ev7ork5GRwYEDB7jiiivo1asXJpM73vXMxKkqdeprAwcO5Oeff/aaGbFjx44qF40bMGAAv/zyi9dQ1Y4dO7jpppvYvn27/pdVyeJgz4d2dQwbNozY2FjefPNNtm/frg/7eM4N7p9lr1699H/p6em88MILFZ5j4MCBHD16lC1btnht//TTTzGbzSQmJlarbeXx/KL+/vvvvbbPmzePuXPn6t9v3rzZK1jZtm0bR44c0T/QQkNDycjI8LoeVRU2t2vXjri4uDKzapKTk/nzzz/p169f7TpVjoULFxIcHMyECRMA91/Q1157LR999JHXbLLly5czfPhwlixZwiOPPEJCQgJXXHFFpc9955130rp1a55++ukat2vQoEEAZYqwv/jiC1wul359qqtTp06MGDGCV155heTk5DL7ly5dSkZGBpdeeingvkaDBw/m/PPP14OUbdu2kZ6eXuV7ujq/NyrSu3dvLBZLmWu/adMmjh07Vu617927N2azmdTUVK/3j8lkYv78+Rw5coSwsDCvfb169dLf0xEREZx77rl89913rFmzhksvvdRrSOfss8/mxx9/9PrdsGbNGoxGI2eddVat+ypqRzIqwktERARbtmzht99+o3v37uUeExMTQ4sWLVi5ciUJCQlERETw888/8+abbwKV15vUhVtuuYUvv/ySG264geuvv57s7GxeeOEFDAZDpePJt956K//85z+5+eabufrqqyksLGTBggUkJiZy9tlnU1hYSHBwME899RT/93//p6/94PnrvypGo5GLL76Yt99+m6ZNmzJ48GB9X5cuXbj00kt58MEHOXr0KD179uTAgQM8//zztGzZsty/EgEuv/xyVq1axX/+8x+mT59Oy5Yt+f777/nwww+57bbb9L80a6Nr166MHTuWZ599lsLCQrp168ZPP/3EDz/8oM9KAff1veGGG5g2bRp5eXk8//zzdO7cWa8xOPfcc3nrrbd44IEHuOKKK9i9ezevv/56uTNTPAwGA3feeSezZs3irrvu4tJLLyUjI4NFixbRpEkTr0Ll6kpJSeHPP/8E3NmD1NRUPvroI9atW1emEPW2227j4MGDTJs2jSuuuIJRo0Zx0UUXcfjwYZ5//nkAnn766TLDaaWVHAKqqY4dOzJhwgQWLlxIQUEBAwcOZOfOnSxatIjBgwczfPjwGj/no48+yjXXXMOVV17J1VdfTe/evcnLy+Orr77iiy++4F//+pdeJJ+YmMj//vc//vvf/9KhQweSkpJ4+eWXURSlyve053X3zTffMGLEiBplHSIjI7npppt46aWXMJvNnHvuuRw5coQXXnhB/5mUFhUVxQ033MALL7xAbm4ugwcPJjU1lRdeeAFFUfRh6MpceumlTJ8+HZfL5fVHBMANN9ygD9ted911HDx4kPnz53PllVfKGioNQAIV4WXSpEls27aNG2+8kSeffLLCFWI9qeKZM2disVjo2LEjL7/8Mk888QSbNm3S14ioD23atGH58uU888wzTJ8+nZiYGG6++WZefvnlSsfyu3fvzltvvcVzzz3HHXfcQVhYGOeccw533303FosFi8XCiy++yHPPPcd//vMfWrRowW233Vajpc7Hjx/PG2+8wbhx48r81fnkk0+ydOlS3nnnHVJSUoiJieGiiy7ijjvuqPBD3Wq16m32/JJu3749c+fOrfKv/ep49tlnWbRoEW+88QYZGRl06NCBhQsXek3dHTBgAGeddRYPPPAA4J7+eu+99+prTpx99tncd999vPXWW6xZs4YePXqwaNGiKmdhXX755YSGhrJ06VL+85//EBYWxvDhw7nzzjvLrcupygcffKCn/g0GA5GRkfTu3ZvXX3+9zHCG2WzmhRde4PPPP+e9997j3nvvxWaz0axZM6ZOnUp2djYPPPAA69evZ86cOXoWsTwDBgxg8uTJ1VoTp7S5c+fSpk0bPvzwQ1555RXi4+O5+uqrufXWW2uVtWjevDnvvvsub7zxBp9//jnLli3DYrHQvn17nnvuOS666CL92JkzZ+JwOFiwYAF2u52WLVsybdo09u7dy/fff19pUe7gwYMZOnQozz33HL/99hvLli2rUTtvv/12YmNjefvtt3n33XeJjIxk7Nix3HHHHRXWw9xxxx3ExcWxatUqXn31VZo0acKQIUO48847q1VHcs455xAeHk6rVq3KTEXu0KEDr732mv47JSoqimuvvbbc9VdE3VM0uYuU8HOegt6Sq3tmZ2czdOhQ7r333jIFdaL2PAFobT6E/d1ff/3Fzz//zG233dbQTRHijCIZFeH3tm/fzsKFC7nzzjvp0aMHmZmZvP7664SHh5eZ8ihEbfXu3ZvevXs3dDOEOONIoCL83vXXX4/dbue///0vx48fJyQkhEGDBvHkk0/KfYOEEMLPydCPEEIIIRotmZ4shBBCiEZLAhUhhBBCNFoSqAghhBCi0ZJARQghhBCNVkDM+tE0DVX1fU2wwaDUyfM2JtJH/xfo/QPpYyAI9P6B9LE2z1Wdu1EHRKCiqhrp6dW7UVx1mUwGoqJCyc7Ox+ms33vX1Bfpo/8L9P6B9DEQBHr/QPpYG9HRoRiNVQcqMvQjhBBCiEZLAhUhhBBCNFoSqAghhBCi0ZJARQghhBCNVkAU01aHe2aQiqpWfKvyklRVobDQiN1uw+UKzCruQOij0WjCYJB4WwghAlXAByqaplFQkEtubla1gxSPkycNqGpgVm97BEIfrdYwIiKiqzXNTQghhH8J+EAlOzudgoJcgoNDCQ4OwWAwVvsDzWhU/DbTUF3+3EdN07DbbeTmZgDQpElMA7dICCGErwV0oKKqLgoK8ggLiyQsrEmNH28yGQJ2PryHv/fRYgkCIDc3g/DwKBkGEkKIABPQv9VdLhegERQU3NBNEXXIE6y4XM4GbokQQghfC+hApZjULgQyqU0RQojAVeNAJTMzk4ceeogRI0bQr18//v3vf7Np0yZ9/2+//cbll19O7969GTt2LF988YXX4202G48++ihDhgyhb9++3HXXXaSnp59+T4QQQggRcGocqNx5551s2bKF+fPn8+GHH9KtWzemTp3K/v372bdvHzfffDPDhw9n9erV/OMf/+Dee+/lt99+0x//yCOPsG7dOl588UXeeOMN9u/fz/Tp033aKSGEEEIEhhoV0x46dIhffvmFVatW0b9/fwAefPBBfv75Zz777DNOnTpFly5dmDFjBgAdOnRgx44dvPrqqwwZMoTU1FQ+/vhjlixZwoABAwCYP38+Y8eOZcuWLfTt29fH3RMN4bbbbqJZs+Y88MAjDd0UIYQQfq5GgUpUVBTLli2jV69e+jZFcd+mOTs7m02bNnH++ed7Peass85i7ty5aJrG5s2b9W0e7dq1o2nTpmzcuPG0AhWTqWxySFVrX7vgKXtQFND8c/ZulQKtj0ajUuZ1YDQavP73Z6qq8cVvhziVVaBvUxQFi8WE3e5EC4SLWA7pYxFNQ0HFoLmK/+Eq871SdJznf9BQNBUFzf2v1H7PNkp87b3d/biiRqBo7v/1tpfY57Wt6DjFvQGTQcHlUr2OVbQKHkuJx57GNVeov9eLAhiMBlSXilaD89b+U6oe+xYSSc9LJmE0uiem1Pfv0xoFKhEREZxzzjle29asWcOhQ4e4//77+eijj0hISPDaHx8fT0FBARkZGaSmphIVFUVQUFCZY1JSUmrZBTAYFKKiQstsLyw0cvKkodwPsOoKhA+4qvi6j57gtbY/85pSVQWDwUCTJiEEB5c/wysiwlovbalL2/ef4v0f9jZ0M0SVNCw4CTXYsCp2QhQ7IYqNEMVOkOLAojgJUpxYFCcWiv4vZ5tZcWHChVFRMaFiUvx3GQHh53Ig8+gQOvdzj6TU9+/T01pH5Y8//mDWrFmMHj2akSNHUlhYiMVi8TrG873dbqegoKDMfoCgoCBsNlut26GqGtnZ+WW22+02VFXF5dLKrBWiaRp2R8VvfEVxf4C7XKrPsw0Ws6HGM1WGDRvAddfdyJdffobT6WDRoldISGjGK6+8zNdf/4+8vFzatevADTfcwqBBZ7Fv316uueZfLF/+Nl26dAVg1qy7+eOPjXz55fcYjUY0TeWSS0YzffqdjB59EZ999jEffPAOycnJGAwKnTt3Zfr0O+natTsAV1xxCSNHnsfvv/9CRkY6jz/+DD169GLJkhf5+uuvcDjsjB8/EVVV0bTin/mqVW/x8ccfcOJEGrGxcVx88aVcc81Un83Wcbnct0fIysqnoMB79WGj0UBEhJXs7IISf835pxOncgGIDLMwql9LABSDQlCQGZvNgaYGaLahkfTR6Cog2JGNxZGDxZmDxZmLxZFDUNHXZmc+JlchJlcBBur+taYBqmJCU4yoirHE/wZAQVMM7lyI/r0CGNAUBa3o/+LvFShxvPt793Hux0Lx3/6ePIVSIh1QYhsUPdb7e/fvU80rd+LuR4nfA0rpbUXfn9bvinqaFagoGE1GXE5X/WWo62nGozEsmh6tOpGdXeDT36cREdZq/aFc60Dl22+/5e6776Zfv37MmzcPcAccdrvd6zjP91arleDg4DL7wT0TyGo9vQitvEXLKlpxVdM0nnz7D/YezTqtc9ZWx5ZNmDWpX40/qD/66H3mzVuI0+miVavWPPLIAxw6dICHHppDXFw8v/zyE/feewdPPDGPoUOH0axZczZu/J0uXbricrnYsmUT+fn57N6dRLduPdi+fTs5ObmcddYw1q79geeff4b77ptN7959OXnyJAsWPMtTTz3OihWr9DasXv0eTz/9POHh4bRv35EFC57ll19+5oEHHqZp02a8+eZr/PXXFpo3bwHAunU/8dZbr/PYY0/QqlVbtm//m8cff5hmzZozZsxFPv25lheQFu9T/XphOwBHUWAdFR7EuKFtAfeQZ1RUKBkZeX7fv4rUVx81TUMryELNPI6anYaWfQI1Ow01x/0/tryaPaHBiBIUihIUBkEh7q/NVhSzBUxBKKYgMLv/NwYFExYZQZ4NVIMFxWQBkwWMJhSDCQymoq+NYDSDwYTiR4sbyuvU/2mgByf1/fu0VoHK22+/zdy5cxk7dixPP/20niVp1qwZaWlpXsempaUREhJCeHg4CQkJZGZmYrfbvTIraWlpNG3a9DS6UQt+uPTGmDEX6dmNI0eS+fbbNbz++ko6deoCwL/+NZm9e/ewatWbDB06jLPPHs7GjeuZPPladu7cjslkpmfPXvzxxya6devBb7+to0+fPkRERNCkSRNmznyQ0aMvBCAhoRnjxl3K/PnPeLXhrLPOZuDAwQDk5+fxv/99zl133ceQIcMAmDXrIf74o3i6+rFjR7BYzCQkNCchIYGEhARiY+Np2tR7iFBUTS36M81g8MMXbyOjqSpq5lFcaftR04/o/7TCnEofpwSFoYREooRGooQ0wRAS6f4+JBIlOMy9PyjEHZyYLNX+Y8RkMhAWFYojQD/khDgdNQ5UVq1axZw5c5gyZQoPPPCA1xtxwIABbNiwwev433//nX79+mEwGOjfvz+qqrJ582aGDBkCwIEDB0hNTWXgwIGn2ZXqUxSFWZP6VTr0A3W3vHxthn4AWrZsrX+9e/cuAG699QavY5xOJ2Fh4QCcffZwPv30I2y2QjZuXE///gNISGjO5s2bmDTpGn77bR0XX3wJAH369OPgwQOsWPEqhw4d5MiRw+zbt7fMDQtbtmylf3348CEcDgddu/bQtwUFBdG5cxf9+9GjL+KLLz7l3/++nLZt2zNw4GBGjjyvTC2TqJpaNOxhkAXuakxzFOI6noQrZQ+utP24ThwAR2HZAxUFJTwOQ0R80b84FM/X4fEo5qCyjxFC1KkaBSoHDhzgiSee4IILLuDmm2/m5MmT+r7g4GCmTJnChAkTmDdvHhMmTGDt2rV89dVXvPrqqwA0bdqUiy++mNmzZ/PEE09gtVp5+OGHGTRoEH369PFpx6qiKApBFmOlx5hMBoyN6K/XkkXImuYOIF566RVCQrwLiT33u+nbdwBms5ktW/5g06YNjBlzEc2aNWP16vdISTnOnj27GT58JABff/0Vc+c+zOjRF9KzZyLjx1/O/v37mD//6QrboI8fa97BjMlU/LKKjIzk9ddXsW3b32zcuJ7163/j/ff/y9SpN3PddTee1s/jTKNnVCRQqZKmqagnDuA8sg3Xke24UveBVuru6eZgjHHtMMS0xhjdEkN0KwxRzd3DLkKIRqNGgcqaNWtwOBx88803fPPNN177JkyYwFNPPcXixYt59tlneeONN2jZsiXPPvusnj0BmDNnDk888QS33XYbACNGjGD27Nk+6MqZpV27DgCcOnWSzp276tuXLn0Jo9HIDTfcgslkYtCgIaxbt5YdO7Zx//0PExsbi8vlYvnypbRv35HmzZvjdKqsXLmCSy65jLvvnqU/188/rwXcY/flZYBat26DxRLE33//pQ8/OZ1O9uzZTb9+7nVyvv76f+Tk5DBx4pUkJvZh6tSbefrpx/nuu68lUKkhPaPSiILnxkRTVVype3Du34DzwGa0/Eyv/Up4LKbm3TA07YgxvgOGyOZ+VechxJmqRoHKLbfcwi233FLpMSNGjGDEiBEV7g8JCeHxxx/n8ccfr8mpRSnt23dg6NDhPPvsk9x55320a9eeH3/8jrffXsH99z+sHzds2AiefvpxYmPjaNHCPVOkZ89E1qz5kquvvl4/Lj6+KVu3/sWuXUmEhYWxbt1aVq9+D3AXRJeeUg7ua3nFFVfy2mtLiY2NpW3b9vz3v29x8uQJ/Ri73cZLL71AaGgovXv3JS0tjS1b/qBPH1ncr6akRqV8roxjOHb+iHPferSCEgXyZiumFt0xtuyBqWVPlPA4uS+UEH7otKYni4b12GNPsmzZSzz77BPk5GTTvHlLZs58kAsvHKcfM2TI2bhcLj3DATBgwCD++GMTw4YVr4kzY8a9PPPMXG677SYsFjMdO3Zm9uxHefjh+0lK2kHv3uUHFjfffBsWSxDz5z9Nfn4+o0ZdwNlnFweq48ZdRlZWFitWvEpaWirh4eGMHHke06bJbRNqyiU1KjrN5cR5cDOOHT/gOp5UvMNixdSmH+b2AzG27IFiNDdcI4UQPqFoAbDUo8ulkp5eduqgw2Hn1KnjxMQ0w2yu+bhzXRXTNiaB0MfKrnMgTRn86a9jrPhfEn06xjL9ikQgsPpXkZJ9dBTkY9/+LY6tX6MVZLsPUBRMrftg7nqO3wYngX4dA71/IH2sjejo0LpdR0UIUb88NSpnYkJFtRdSuOULCrd8qU8hVkIiMXcdgbnrORjCYhq4hUKIuiKBihB+wnUGFtNqLieF278n+88vcOW560+UiHiC+o3H1HGwezE0IURAk3e5EH7CU0zbmKbM1yXnkW3Yfl2JmnkcAENEPJa+l2DqNNS9QqsQ4owggYoQfkI7Q4pptcJcCn9diXPvbwAo1nBizp2Es9UgXJpMJxbiTCOBihB+wnM/vkCeYus8tIXCn153F8oqCuYe5xMy+HIiEuLJyMiDAC1SFEJUTAIVIfyEq+h2BoE49KM57djWv4dj+7cAGKKaE3zOVPfCbCbJoghxJpNARQg/4cmoBNpiqmruKQq+Xoh68hAA5l5jCBp0hV9OMxZC+J4EKkL4iUCsUXEe30XhN4vQCnNQgsMJHnkDpta9G7pZQohGRAIVIfyEZ3qyEiBDP449v1L443LQXBhiWmMdPR1DeGxDN0sI0chIoCKEn9CnJwdARsX+91fYfn8HAFP7gQSPvAHFVPZ+UkIIEWCj3aI+OZ1O3n13pf798uVLueKKS3x+nrp6Xn8TKDcltG36SA9SzD1HE3zeNAlShBAVkkBF1No333zFiy8+39DNOGOoAVCjYtv0EfY/PgHAMugKgob8G0WRX0NCiIrJ0I+otQC4n6VfKZqd7LcZFdsfn+pBStBZ/8SSeGEDt0gI4Q/O2EBF0zRw2qs4xoBWFwtMmSy1WrTrt99+4dVXl3Dw4H6s1hCGDDmb22+/k717dzNjxn947LGnWLLkRVJTU+nZsxcPPPAI//3vW3z11ReYTGb+8Y9/cc01U/Xn+9//Pufdd1dy+PBhoqOjGTduPFOmXIfR6F6ePDU1haVLX2LTpg3k5+eRmNiHW2/9Pzp27MSXX37GE088CsCwYQNYuHCJ/rxvv72CDz98j6ysLHr06Mm99z5Aq1atAcjNzeWll17g559/wOFw0KVLN269dTpdu3bXH//JJ6tZtepNTpw4wcCBg2jWrHmtfsyBpnjop4EbUgv2nT9i37QakCBFCFEzZ2Sgomka+Z/ORU3d2yDnNzbthPXS+2sUrGRmZvLAA/dw220zGDp0GGlpqcyZ8zCLF7/A6NEX4nK5ePPN13j44cdxOp3cc88dXHvtVYwbN55ly97g66//xyuvvMywYefQoUNH3ntvFUuWLGL69Dvp338QO3ZsY/78p8nKyuL//u8u8vPzmDZtKs2bt+Cpp57DbLbw2mvLuO22G1mx4r+cd94F5ObmsnDhc3zyyVdERDRhy5bNpKQcZ+vWv3j22RdwOOzMmfMQTz01h5deegVN07jnnulYLME8/fQCwsLC+OqrL5g2bSpLl75O585d+eabr5g//2n+7//uZsCAQfz00w8sW7aY+PimdXhF/IO/Dv04Dv6Bbd0bAFj6XiJBihCiRvzwbzPfUPCvX/YnTqRit9tp2jSBhIRmJCb24emn5zNx4j/1Y2644Ra6du1Oz56J9O8/EKvVyq23Tqd16zZMmXItAPv370XTNN5++w0uv/xKrrjiSlq1as2YMRcxdeotfPTR++Tm5rJmzf/Iyspkzpyn6d69J506deaRRx4nKCiY1avfIygomLCwMABiYmIxm92Lc5lMJh56aA4dO3aiW7cejB9/OUlJOwDYvHkj27ZtZc6cJ+nRoydt2rTl5pv/Q48evXj/fXdx5QcfvMv554/m8sv/QevWbZg8+VrOPnt4Pf6kGy9/LKZ1nTxE4XdLQNMwdxmBZcDlDd0kIYSfOSMzKoqiYL30/iqHfkwmA85GMvTTqVMXzj9/DPfdN4OYmFgGDhzM0KHDGTFiJH///ScALVu20o+3Wq00a9ZcP09QUDAADoeDzMwM0tNPkZjYx+scffv2w+l0cujQQfbt20urVm2IiorS9wcFBdO9ew/27dtXYTujo2MIDQ3Tvw8Pj8BmswGwe3cSmqYxceI4r8fY7Xb9mP3793L++WO89vfsmciePbur82MKaP6WUVELsin4eiG47Bhb9iRo+DUBfZ8iIUTdOCMDFSi6sZu58imRismAojSem6A98shcrr/+Rn7//Vc2blzPnDkPkpjYR687MZm8L2dFHwoVFcF6Pgjdz1PRMSomk7HCNhoqKaBQVZXQ0FCWL3+7zD5PRgYUNM37Z166X2cqPVDxg4yKproo/HYxWu4plIimWM+bhmKo+HUjhBAVOWOHfvzN9u3bWLjwOVq3bsuVV17Fs8++wKxZD7F580YyMjJq9FzR0TFER8fomRiPv/7agtlspkWLlnTo0Ink5ENkZKTr+202G0lJO2nbtj1Q87v4tm/fkby8PBwOBy1bttL/rVz5BuvWrQWgU6fO/P33X16PS0raWaPzBCp96McPshL2zR/jOp4E5mCsY6ajBIU2dJOEEH5KAhU/ERoayurV77N48UKOHElm//69fPfd17Rs2ZrIyMgaP9+//z2F1avf48MP3+fIkWS+/vorXnttGZdeOoGwsDAuuGAsTZpE8uCDM9m5czt79+7hscdmU1BQwPjx7joDq9UKuAMJm62wynMOHjyETp068/DDs/jjj00cOZLMiy/O58svP9ODn8mTr+Wnn35g1ao3SU4+zAcfvMOPP35X4/4FouKbEjbuQMV5bCf2LZ8DEDziOoxRLRq4RUIIfyY5dT/Rtm075s59ltdff4WPPnofg8FAv34Dee65haSmptT4+f7978lYLGbeeWclzz//LPHxTZk06RquumoKAGFhYbz44lIWLVrA//3frQAkJvbm5ZeX07y5+4OnX7+BdO/ek2nTrufBB+dUeU6j0cjzzy9m8eIXeOihmRQUFNC2bXvmzn2W/v0HAjB06DAefvhxXnttGa++uoQePXrxr39N5ptvvqpxHwNNcY1KAzekEmphDoXfLwXcxbPmDoMbuklCCD+naAGwapfLpZKenldmu8Nh59Sp48TENMNsttT4eeusmLYRCYQ+VnadTSYDUVGhZGTk+X0/X1q9lc27TzBldGfO7dcSaHz9K/h2Mc79GzBENiNkwiMoVdSBVUdj62NdCPQ+Bnr/QPpYG9HRoRiNVQ/syNCPEH7CU6PSWO+e7DiwCef+DaAYCB51s0+CFCGEkEBFCD/RmKcna4W52Na9CYCl90UYY9s2bIOEEAFDAhUh/ISrEc/6Kfz9XbSCbAyRzbH0u7ShmyOECCASqAjhJ7SijIqxkQ39uFL34tz9MwDB51yPYqp5PZgQQlTkDAlU/L5eWFQiAOrBq8UzPVlpRO9aTVUp/OUtAEydh2Ns2rGBWySECDSN6Fee77nvAqxUa40P4b/sdvfy+0ZjYM+2dzXCGhXHrp9QTx4Cs5WgQVc0dHOEEAEooH+zGwxGrNZQcnMzcTodBAeHYDAYq72iqqoquFyB/de6P/dR0zTsdhu5uRlYrWGVLt8fCDyzfhrL0I9mz8e+4QMAggZMwBDSpIFbJIQIRKcVqCxdupR169bx1lvu1O+UKVPYsGFDucc+/fTTXHbZZbhcLvr27avfhM7jtttu4/bbbz+d5pQrIiIaszmI3NxMCgvLrrVSGYPBgKoG5nx4j0Doo9UaRkREdEM3o85pjSyjYv/7KzRbLoYmCZh7jGro5gghAlStA5WVK1eyYMECBgwYoG978cUXcTgc+veapjFjxgyysrK44IILADh48CA2m41PPvmEmJgY/diQkJDaNqVSiqIQEhKG1RqKqqqoqqtajzMaFZo0CSErK99vMw5VCYQ+Go2mgM+keHiGfhrDOipqfhb2v9cAYBl0BYohoJOzQogGVOPfLqmpqTz88MOsX7+etm3beu0rfc+Zt99+m7///ptPPvmE0FD3Tcl27dpFWFgYXbt2rXWja0NRFIxGY1HdStVMJgPBwcEUFLgCepXBQO9jIGlMQz/2LZ+B04Yhrh2mtv0bujlCiABW4z9Ft2/fjtls5tNPP6V3794VHpeens6CBQuYNm0a7du317fv2rWLDh061K61QpzBPCN0DT30o+acxLHzBwCCBv2jxnfRFkKImqhxRmXUqFGMGlX1ePQrr7xCcHAwU6dO9dq+e/dunE4nU6dOJSkpiaZNm3LNNdcwfvz4mjbFi8nk2/S/5/4D1bkPgb+SPvoXzzRss8mgv94bon/5f38JqgtTy+4Et+lZ5+cLpGtYkUDvY6D3D6SPdalOBpZzc3N57733uO222wgK8r7fx549e1BVlenTp5OQkMDatWuZNWsWDoeDK66o3fRGg0EhKirUF00vIyLCWifP25hIH/1EUeaiSRNrmdd7ffXPmZNBZpJ7cbe4kf/CWkfvu/IExDWsQqD3MdD7B9LHulAngcq3336L3W5n4sSJZfZ9/vnnuFwuvWala9euHDt2jOXLl9c6UFFVjezs/NNqc2lGo4GICCvZ2QW4XIFZvyF99C9Op7sQPC/PRkaGewZbffcv/9cP0VwOjAmdKAhvQ2FGzWbS1UYgXcOKBHofA71/IH2sjYgIa7WyM3UWqJxzzjlERESU2RccHFxmW+fOnfn0009P65x1VQzqcqkBX2gqffQPnlk/mqqV6Ut99E8rzMW27XsALH3GFc0Uq7/ZYoFwDasS6H0M9P6B9LEu1MlA06ZNmxgyZEiZ7dnZ2QwaNIjVq1d7bd+6dSudOnWqi6YIETA8s34MDTTrx77jO/dMn5jWGFslNkgbhBBnHp9nVI4fP05GRka5048jIiI466yzeP7554mJiaFNmzZ8/fXXfPrppyxdutTXTREioKgNuOCb5nLi2F6UTel9kcz0EULUG58HKidOnADKrqni8cQTT/Diiy/y8MMPc+rUKTp06MDChQsZPny4r5siREDx3JSwITIqzv0b0AqyUEIiMbUfUPUDhBDCR04rUHnqqafKbEtMTGTXrl0VPiYsLIxZs2Yxa9as0zm1EGec4oxK/Z/bvu1bAMzdR8kqtEKIehW4E76FCDB6oFLPkYorbR/qif1gMGHuNrJezy2EEBKoCOEnGqqY1r7tGwBMHQdjsJadySeEEHVJAhUh/ERDFNNqhbk4928CwNLj/Ho7rxBCeEigIoSfaIibEjr2/gaq0z0lOa5dvZ1XCCE8JFARwk94bkpYX1ODNU3DkfQTAOauI+rlnEIIUZoEKkL4AU3T6r1GRT15EDU9GYwmzB3LLuAohBD1QQIVIfyAVmKl+voa+nEkrQXA1G4ASlD93XxQCCFKkkBFCD+glohU6iNO0Zx2HHvXA2DuIsM+QoiGI4GKEH7Ac0NCqJ8aFefhP8FRgBIWg7F52dthCCFEfZFARQg/oJYIVOpj6MfpyaZ0PAtFkV8TQoiGI7+BhPADWsmhnzoOVDR7Ps7kvwAwdTirTs8lhBBVkUBFCD9Qcuinrhd8cx7YDC4nhqjmGKJb1um5hBCiKhKoCOEHSsQp1HWJimPv74A7m1Jfa7YIIURFJFARwg+UXD6/LoMHNT8L17EdAJg7DK6z8wghRHVJoCKEHyi+c3Ldnsd5cDNoGoa4dhiaNK3bkwkhRDVIoCKEH9BXpa3r+pSDfwBgajewTs8jhBDVJYGKEH6gPpbP12x5uI7uBMDctl+dnUcIIWpCAhUh/EDJGpW64kz+GzSXe7ZPZEKdnUcIIWpCAhUh/EBxjUodBiqeYZ82kk0RQjQeEqgI4Qc805PrKlDRnHacyVsBMMmwjxCiEZFARQg/UDz0UzfP7zq2AxyFKKFRGOLa1c1JhBCiFiRQEcIP1HUxrfPgFgBMbfrKIm9CiEZFAhUh/EBdFtNqmlY87NOmj8+fXwghTocEKkL4gbrMqKgZx9Dy0sFoxtisq8+fXwghTocEKkL4gbrMqLiO/A2AsXlXFJPF588vhBCnQwIVIfxAXU5PdiZvA8DUspfPn1sIIU6XBCpC+AF9erKPMyqaoxDX8V0AmFpJoCKEaHwkUBHCD7jq6KaErmNJoDpRwuNQmshqtEKIxkcCFSH8QF3dlFCf7dOql0xLFkI0ShKoCOEHtKKMitHHNSrOI+76FGPLnj59XiGE8BUJVITwA56hH8WHgYqaewotOxUUA6bm3Xz2vEII4UsSqAjhB+pi6Md1LMn9nLFtUSxWnz2vEEL40mkFKkuXLmXKlCle22bPnk2XLl28/o0aNUrfr6oqCxcuZPjw4fTp04cbb7yR5OTk02mGEAHPE6j4cujHeWwHAKbmssibEKLxqnWgsnLlShYsWFBm+65du7jllltYt26d/u+DDz7Q9y9evJhVq1YxZ84c3nnnHVRV5YYbbsBut9e2KUIEPE11/++rOEXTND2jYpRhHyFEI1bjQCU1NZVbbrmFefPm0bZtW699mqaxd+9eevbsSVxcnP4vOjoaALvdzmuvvcb06dMZOXIkXbt25fnnnyclJYWvv/7aJx0SIhD5ukZFyzmBlnsKFCPGhE4+eU4hhKgLNQ5Utm/fjtls5tNPP6V3795e+w4fPkx+fj7t27cv97FJSUnk5eUxZMgQfVtERATdu3dn48aNNW2KEGcMfejHRzUqzmM7ATDEt0MxB/vkOYUQoi6YavqAUaNGedWclLR7924A3nrrLX766ScMBgMjRoxgxowZhIeHk5KSAkCzZs28HhcfH6/vqy2Tybd1wUajwev/QCR99B+e+MRoNHi91mvbP1vRarSWlt19/t7xtUC5hpUJ9D4Gev9A+liXahyoVGb37t0YDAbi4+NZsmQJhw8f5plnnmHPnj288cYbFBQUAGCxeN/4LCgoiKysrFqf12BQiIoKPa22VyQiIvBnQ0gfG7+gYPd7JijIVO5rvSb90zSN7OPu+pSoLn2x1tF7x9f8/RpWR6D3MdD7B9LHuuDTQGXatGlcddVVREVFAdC5c2fi4uK48sor2bp1K8HB7hSz3W7Xvwaw2WxYrbXvuKpqZGfnn17jSzEaDUREWMnOLsDlUn363I2F9NF/5OYVAuByqWRk5Onba9M/V2YKrtx0MJgoCGtFYYnna4wC5RpWJtD7GOj9A+ljbUREWKuVnfFpoGIwGPQgxaNTJ3ehXkpKij7kk5aWRuvWrfVj0tLS6NKly2md2+msmxeGy6XW2XM3FtLHxs/pLCqmpfzXek365zjqHvYxxLXFhQn85Ofi79ewOgK9j4HeP5A+1gWfDjTde++9XHvttV7btm5130ukY8eOdO3albCwMNavX6/vz87OZseOHQwcONCXTREioKiemxL6oJbWlboXAGNTme0jhGj8fBqojBkzht9++41FixZx+PBh1q5dy/3338+4cePo0KEDFouFyZMnM2/ePL777juSkpKYMWMGCQkJjB492pdNESKg+HJlWlfqHgCMCR1P+7mEEKKu+XTo57zzzmPBggUsW7aMV155hfDwcC655BLuuOMO/Zjp06fjdDqZPXs2hYWFDBw4kOXLl2M2m33ZFCECip5ROc2UimbLQ804BkhGRQjhH04rUHnqqafKbLvwwgu58MILK3yM0Wjknnvu4Z577jmdUwtxRvFVoOJK3QeA0qQpBmvEabdLCCHqWuBO+BYigPhq6Ecf9mkqwz5CCP8ggYoQfkAPVE47oyKFtEII/yKBihB+wKWefkZFU5240txDPxKoCCH8hQQqQvgB/e7Jp/GOVU8dAacdLCEYoppV/QAhhGgEJFARwg/4YuinZH2KoshbXwjhH+S3lRB+wBdDP64TBwAwxpd/d3MhhGiMJFARwg/4YtaPeuIgAMa4tj5okRBC1A8JVITwA1pRRsVYy6EfzV6AmnkcAEOcZFSEEP5DAhUh/IAno6LUMlBxnTwIaChhMbLQmxDCr0igIoQfcJ3mTQn1YZ/Ytr5pkBBC1BMJVITwA2rR9GRjLecnewppDfHtfNUkIYSoFxKoCOEHNO30Mir6jB+pTxFC+BkJVITwA56hn9rUqGiFuWg5JwAwxrbxabuEEKKuSaAihB84nenJnmyK0qQpSlCoT9slhBB1TQIVIfyAehrTk4uHfaQ+RQjhfyRQEcIPeAKV2iyhr548CIAxVgIVIYT/kUBFCD9QFKdQm4VpXacOA2CIbe3DFgkhRP2QQEUIP+CpUanp0I9my0PLOel+bIwEKkII/yOBihB+QK3lTQld6UcAUMJipJBWCOGXJFARwg/UtkZFLRr2kWyKEMJfSaAihB+o7fRkT6BiiGnl8zYJIUR9kEBFCD9Q24yK61Sy+3HREqgIIfyTBCpC+IHaZFQ01YWa4a5RkRVphRD+SgIVIfyA56aENcmoqJkp4HKCORglPLaOWiaEEHVLAhUh/IBLH/qp/mPU9KJC2uhWKIq81YUQ/kl+ewnhB7RaDP24TkohrRDC/0mgIoQfqE2NippeVEgrU5OFEH5MAhUh/EBtZv0Ur6EiGRUhhP+SQEUIP+CqYaCiFuagFWS7HxPVss7aJYQQdU0CFSH8QE1rVNSMYwAo4bEo5qA6a5cQQtQ1CVSEaAQcTpXPfz3IoZSccvd77p5c3ZsSqhlHATBEtfBJ+4QQoqFIoCJEI7DtwClW/7SfD3/aV+5+z9BPdWtp1XR3oGKUQEUI4edOK1BZunQpU6ZM8dr2/fffM3HiRPr27cuoUaN4+umnKSws1Pdv3ryZLl26lPm3fv3602mKEH4tr8AJQIHNWe7+mhbTqpnuoR9DVHMftE4IIRqOqbYPXLlyJQsWLGDAgAH6tk2bNnHbbbcxffp0xo4dy6FDh3jooYfIzMzkySefBGDXrl20bt2aVatWeT1fkyZNatsUIfyew+kCwOnSvLY7XSomo0GvUZGhHyHEmabGGZXU1FRuueUW5s2bR9u2bb32vfPOOwwePJhbbrmFtm3bcs455zBjxgw+++wz7HY7ALt376Zjx47ExcV5/bNYLD7pkBD+yO50r5Hvcqn6tpVf7+b/Fv7MqazC4lk/1Rj78ZrxEykZFSGEf6txoLJ9+3bMZjOffvopvXv39tp3/fXXc99993mfwGDA4XCQm5sLuDMqHTp0OI0mCxF49EBFLc6o7ErOpMDm4lBqjr7gm1KNjIqnPkVm/AghAkGNh35GjRrFqFGjyt3XvXt3r+8dDgcrVqygZ8+eREdHA7Bnzx6ioqK4/PLLSU1NpXPnzsyYMYPExMRaNL+YyeTbumCj0eD1fyCSPjYenkyKy6Xpr2VX0Z0IHS4VrSjRYjEbvV7r5fXPlX0cAFN0C5+/LxqCv1zD0xHofQz0/oH0sS7VukalKk6nk3vvvZc9e/awcuVKAI4fP05OTg75+fnMnj0bo9HI22+/zeTJk1m9ejUdO3as1bkMBoWoqFBfNl8XEWGtk+dtTKSPDc9gMgLupfI9r2WtxD5XUUYlOiqEqCZl+1KyfyfzUgEIad6uzt4XDaGxX0NfCPQ+Bnr/QPpYF+okUMnNzeWOO+5gw4YNLFq0SM+WNGvWjI0bN2K1WjGbzQD06tWLHTt28NZbb/Hoo4/W6nyqqpGdne+z9oM7YoyIsJKdXeBVNxBIpI+NR06uDXAPAWVk5Lm/trsLbNMz8vVZP9nZBRjU4n6U17/844cAcFjj9OfyZ/5yDU9HoPcx0PsH0sfaiIiwVis74/NAJS0tjRtvvJGjR4+yfPlyBg4cWKphEV7fGwwGOnToQGpq6mmd1+msmxeGy6XW2XM3FtLHhmcrCkqczuJ2Oot+EeQVOvTjNFUrtx8l++cqmvFDk+aNus811divoS8Eeh8DvX8gfawLPh1oysrK4pprriE9PZ2VK1eWCVJ++ukn+vbtS3Jysr7N6XSSlJRU62EfIQKBvWh6csliWs9U5fzC4rVVqlpHRWb8CCECjU8zKk8++STJycm8+uqrREdHc+LECX1fdHQ0/fr1Iyoqivvuu4/7778fs9nMsmXLyMzM5Nprr/VlU4TwK45SWRQAZ9EQT36JReCqmp6s3+MnLEZm/AghAoLPAhWXy8WXX36Jw+HgmmuuKbP/u+++o2XLlqxYsYJ58+YxdepUbDYb/fv35+233yY2NtZXTRHC75ScnqxpGoqi4Covo1JVoJKV4j4uslkdtVQIIerXaQUqTz31lP610Wjk77//rvIxrVu3ZuHChadzWiECjsPh0r92qRoGQ/EwkFdGpaqhn8yiQKVJQh20Uggh6l+dTU8WQlSfvURhmtOlYtSKy8cKvGpUKn8eTc+oSKAihAgMEqgI0Qg4vAIVDU0r/r5GNSpZklERQgQWCVSEaAQ8s37Ae+YPFE9PVhRQKglUNNWFmp0GSI2KECJwSKAiRCNQcujH5VLRtOKAxO5w76sqm6LlnATVBUYLSmhU3TRUCCHqmQQqQjQCDod3jYpBKxuUVFlIqw/7NEVRAvd+I0KIM4sEKkI0AvZSNSpGrewx1Z/x09SnbRNCiIYkgYoQDUzVNO+F3koN/XhIIa0Q4kwkgYoQDcxR6p4Z7kXfyh5XRUJFFnsTQgQkCVSEaGBlAhWXhmooG6kYa1CjIoQQgUICFSEamL3EqrRQcTGtUkmgojkK0fIyABn6EUIEFglUhGhgpTMqTrWCWT+V1Ki4igppleBwlOAw3zZQCCEakAQqQjQwe+lAxaWVW49S2dCP3ONHCBGoJFARooGVXJUWihZ8Kyd7UmlGpag+RZH6FCFEgJFARYgGVnKxNyjKqJSzXltlNSpqVtHS+RHxPm2bEEI0NAlUhGhgpYd+XKqKWm6NSsXPoWafcB8jgYoQIsBIoCJEA3M4S8/60VCUmk1PdmVLRkUIEZgkUBGigZXJqLjUcu+SXFGNiuqwFU9NlkBFCBFgJFARooGVmZ5cQUalohoVZ6Y7m4LZCkGhPm+fEEI0JAlUhGhg5a2jolA2KKlo6MeRmQq4synlZWKEEMKfSaAiRAMrPT3ZvY6KO6OiKOj3/alo6MeZUbSGSkRc3TVSCCEaSDmTIIUQ9an09GSXS8XpckcnocFmfXtFtbQlMypCCBFoJFARooGVnZ6s4XS5t4VaSwQqFdWoZLgDFSVcMipCiMAjgYoQDazs0I+KqyijEmYtHp2tKFCRjIoQIpBJjYoQDazcWT8UBSpeQz9lAxVNU/WMitSoCCECkWRUhGhg9qIalSCLESiqUVE9GZXKh360vEw0lwMUA0pYTD20Vggh6pcEKkI0MM/KtCFB7gSn01VBjUo5GRV9RdrwWBSDsa6bKoQQ9U4CFSEamKeY1loUqLhUFVc1i2nlZoRCiEAnNSpCNDBPjYq1aOjHMzXZs82gKKiaVu70ZFW/x4/UpwghApNkVIRoYJ5ZP1Z96EfVh35MJgPBRQFMeRkVz9CPsYlkVIQQgUkCFSEamGfBt+KhHw1XUTGt2WggOKjiQEXNPuHeJ0M/QogAJUM/QjSw0jUqnmwKgNGoEGQuClTKKaZVc06594XLjB8hRGCSQEWIBlberB8Pk8FAsMW9vXRGRXM50PIz3fvCY/FejUUIIQLDaQ39LF26lClTpnht27lzJ5MnT6ZPnz6MGjWKN99802u/qqosXLiQ4cOH06dPH2688UaSk5NPpxlC+LXijErROiolZv2YjCVqVEplVLRcdzZFMVlQgsPrq7lCCFGvah2orFy5kgULFnhty8jI4LrrrqN169Z8+OGH/Oc//2HevHl8+OGH+jGLFy9m1apVzJkzh3feeQdVVbnhhhuw2+217oQQ/swTqASXyKg49EBFqbCY1jPsY2oSh1LBnZWFEMLf1XjoJzU1lYcffpj169fTtm1br33vvfceZrOZxx57DJPJRIcOHTh06BDLli1j4sSJ2O12XnvtNe6++25GjhwJwPPPP8/w4cP5+uuvGTdunC/6JIRf8RTTeoZ+XC4VrWj0x2gsHvoxlgpG1NyTAJhkxo8QIoDVOFDZvn07ZrOZTz/9lJdeeomjR4/q+zZt2sSgQYMwmYqf9qyzzmLp0qWcPHmSY8eOkZeXx5AhQ/T9ERERdO/enY0bN55WoGIy+XYCk9Fo8Po/EEkfG57TpaIWRSVhIe7F3VyqRtGkH4IsRqzBRYGKSfF6nTvy0wF3RqWx9s8XGvs19IVA72Og9w+kj3WpxoHKqFGjGDVqVLn7UlJS6Ny5s9e2+Hj3X3vHjx8nJSUFgGbNmpU5xrOvNgwGhaio0Fo/vjIREdY6ed7GRPrYcPILHfrX8bFhAKgaaEXBS1RkCM3j3dtjo0K9XucOWxbgDlQaa/98Sfro/wK9fyB9rAs+nfVTWFiIxWLx2hYUFASAzWajoKAAoNxjsrKyan1eVdXIzs6v9ePLYzQaiIiwkp1doBc2BhrpY8PLyrXpXztsTvf/DpeeZSnIt3F2j6ZYDAoDu8aTkZGnH194yh3cm5vENdr++UJjv4a+EOh9DPT+gfSxNiIirNXKzvg0UAkODi5TFGuzuX8Rh4SEEBwcDIDdbte/9hxjtZ5ehOZ01s0Lw+VS6+y5GwvpY8MpKHQHJ2aTQa9sdw8HFX2jQZDJyLBe7ixkyT64cjw1KnEUNNL++VJjvYa+FOh9DPT+gfSxLvh0oCkhIYG0tDSvbZ7vmzZtqg/5lHdM06ZNfdkUIfyCZ8aPxWTAaHQXy5a8e7LJWP5sHk11oeUW16gIIUSg8mmgMnDgQDZv3ozL5dK3/f7777Rr146YmBi6du1KWFgY69ev1/dnZ2ezY8cOBg4c6MumCOEXPDckNJsMmIpSoE5V1Rd9M1WQFtXyM0FTwWDEGBZZH00VQogG4dNAZeLEieTm5vLAAw+wd+9eVq9ezYoVK7j55psBd23K5MmTmTdvHt999x1JSUnMmDGDhIQERo8e7cumCOEXHHpGxVgcqDg1ffy3ovFbtWjYxxAWg2Iw1kNLhRCiYfi0RiUmJoZXX32VuXPnMmHCBOLi4rj33nuZMGGCfsz06dNxOp3Mnj2bwsJCBg4cyPLlyzGbzb5sihB+wXPnZLPZgLFoQTdPIS1UMvSTK/f4EUKcGU4rUHnqqafKbEtMTOTdd9+t8DFGo5F77rmHe+6553ROLURAKFmjUl5QYjJUkVEJj627xgkhRCMQuCvTCOEHimtUjOUO8xiryqiESUZFCBHYJFARogHZHe6hn4oyKkZD+YGKqg/9SEZFCBHYJFARogGVnPVjUBRKhiUmo1LhzQZl6EcIcaaQQEWIBqTXqJiNKIriNfxT0YwfTdOkmFYIccaQQEWIBuTwzPoxeW72VZxBMVUw7KPZcsHlvkeQISy6jlsohBANSwIVIRqQ3VE89APewUmFi70VrUirWCNQjDKtXwgR2CRQEaIBOUpMTwbv4KTCNVTyMgBQQqPquHVCCNHwJFARogHpC76Z3KvLlgxOKlyVNs+dUTGEyrCPECLwSaAiRAOyl8qoGL0yKhUM/UhGRQhxBpFARYgG5Cgx6we8102pqJjWk1FRJKMihDgDSKAiRAPSF3wzl61RqXB6clFGxSAZFSHEGUACFSEaUOmhn5I1KhXfkLAooyJTk4UQZwAJVIRoQMVL6BcN/VRRo6JpGqqeUZFARQgR+CRQEaIBFa9MW3YdlXJvSGjPB6cNkGJaIcSZQQIVIRpQpRkVQ9m3pyebogSFoZgs9dBCIYRoWBKoCNGA9JsSlpNRKa9GRfPM+AmTbIoQ4swggYoQDah0RsVURY2KnlGR+hQhxBlCAhUhGlDpGhWjsfIaFc+MH5maLIQ4U0igIkQD8tyUsLoZFU0yKkKIM4wEKkI0EKdLRdU0oOSCbyVXpi1v6EcyKkKIM4sEKkI0EE8hLZS410+J4MRkKq+YVjIqQogziwQqQjQQTyGtQvEwj9FrZdqKMyoy60cIcaaQQEWIBmIvMTVZUdwBite9fkrdlFCzF4C9AABDiAQqQogzgwQqQjSQ0lOTofS9frzfnmq+e9gHixXFYq37BgohRCMggYoQDaT01GTwLqAtHahouXLXZCHEmUcCFSEaSHkZlcrWUdHyMwFQZNhHCHEGkUBFiAbimfXjmfEDpWb9lBn6yQRACWlS940TQohGQgIVIRqIzbPYm7mCGpXSxbRFgYohJLLO2yaEEI2FBCpCNBCH0z30YzaVn0UpU6OSnwWAIoGKEOIMIoGKEA3EU0wbZK5pjUpknbdNCCEaCwlUhGggnmJar4xKpTUqnoyK1KgIIc4cJl8+2fr167n66qvL3deyZUu+++47Xn75ZRYsWFBm/65du3zZFCEavfKmJ3uvTFv8taZpaEXrqEiNihDiTOLTQKVv376sW7fOa9uff/7J7bffzq233gq4A5Lx48dzzz33+PLUQvid8hd8Kxm0lMioOArBaQdk6EcIcWbxaaBisViIi4vTv8/Pz+fJJ59kwoQJTJw4EYDdu3dz5ZVXeh0nxJlIX0Lfq5i2/Fk/nvoUzFYUc1C9tE8IIRqDOq1RWbJkCQUFBdx3330A2O12Dh48SPv27evytEL4BUc505ONFcz6UfWpyVKfIoQ4s/g0o1JSeno6K1as4K677iIyMhKAvXv34nK5WLNmDXPnzsVmszFw4EDuuece4uPjT+t8JpNvYy6jfjfbwK03lj42LKfqDlSCLUb99VtyBlBQie2qLRtwL59f8rXemPvnK9JH/xfo/QPpY12qs0Bl1apVhIeH889//lPftnv3bgCsVisvvPACp06dYv78+Vx99dV8/PHHBAcH1+pcBoNCVFSoT9pdWkRE4N/8TfrYMLSiOyZHNrHqr9+oLJu+PzoqVN+eqeaTBwRFxZT7Wm+M/fM16aP/C/T+gfSxLtRZoPLxxx9z2WWXeQUfl112GSNGjCA6Olrf1qlTJ0aMGMH333/PRRddVKtzqapGdnb+abe5JKPRQESElezsAlwu1afP3VhIHxtWXr67ONZpd5KRkQdAfn5xoJKXV0hGhjvDkn8yzX2sKVw/Fhp3/3xF+uj/Ar1/IH2sjYgIa7WyM3USqCQlJZGcnMwll1xSZl/JIAUgPj6eyMhIUlJSTuucTmfdvDBcLrXOnruxkD42jEK7e9aPyajobVO04v2KVvy6duW5pyZjjSi3H42xf74mffR/gd4/kD7WhToZaNq0aRMxMTF07drVa/vzzz/PmDFj0LTi38ZHjhwhIyODjh071kVThGi0HFXePbn47SnL5wshzlR1Eqjs2LGDLl26lNl+wQUXcPToUR555BEOHDjAxo0buf322+nXrx/Dhw+vi6YI0WjZylnwzfteP2WnJ0ugIoQ409RJoHLixAl9pk9JPXv25JVXXmHXrl1cfvnl3HbbbXTr1o0lS5agKErZJxIigDn0dVRKZFQMJVemLTE9OS8TkOXzhRBnnjqpUXnllVcq3DdkyBCGDBlSF6cVwq/oK9NWkVHRnDZwFABgCImqxxYKIUTDq7NZP0KIyun3+imRUQkJNhEeYsZiMug1Kp76FEwWMNduCr8QQvgrCVSEaCAVZVQev2EwBoOCoWg4VC1RnyJDpEKIM40EKkI0EEc5GRWA8BCL1/eavnx+ZH00SwghGpXAXetXiEbM6VJxqe5p+iUzKuXRpJBWCHEGk0BFiAbgKLFYkqWK+1SpWe7FEGVqshDiTCSBihANwFNIq+A906c0TXXi3L8RAFPLnvXRNCGEaFSkRkWIeqJqGq9+toP4KCtn92oGgNlsqLRA1pW8Fa0wB8UagVECFSHEGUgCFSHqyYmMAn7fkYpBURjQJR4oW0hbmmP3LwCYOg5BMVR+rBBCBCIZ+hGinuTbnIA7s5KW6V7ArbJCWq0wF+ehPwEwdz67ztsnhBCNkQQqQvhYRo6NYyfzymwvLApUAI6fcu83V5JRcezfAKoTQ0wrjDGtfd9QIYTwAxKoCOFDqqrx1MrNPLpiIxk5Nq99BXaX/nXKqXwAgiqZ8ePctx4AcyfJpgghzlwSqAjhQ3uOZHIisxCHU2Xf0SyvfQUlMiop6e5AxVzB0I+mqrhOHADA2CqxjlorhBCNnwQqQvjQ5l0n9K+T03K99hWWyKgcL8qoVFRMq2angNMOJguGJgl10FIhhPAPEqgI4SOqprF5d2WBSnFGxVNYW9Fib+rJwwAYoluhGORtKoQ4c8lvQCFqwKWqqEVL35d28HiOV11KclqO1/4Cm6v0Q7CYy8+ouE4eAsAY26a2TRVCiIAggYoQNfDUyj+4f9nv+p2PS9q8Kw2Anu2jATiVbSOv0KHvLyiRUfGoMKNyqiijIrN9hBBnOAlUhKgmp0tl39Fs0jIL2HPEu1BW0zS9PmV4YnNiIoIBOFJi+Kfk9GSP8jIqmqahSkZFCCEACVSEqDZbiSzKzkMZXvtSMwpIyyzAZDTQq300rZuGAXA4tThQKW/ox1xORkXLS0ez5YJiwBDVwlfNF0IIvySBihDVVGirOFDJynXXpsQ0CSbYYqJVvDtQKVlQW1je0E85GRW9kDayOYrJcvoNF0IIPyaBihDVVFgio3IwJZv8wuLAw7OYm9XiDjzKC1Q8x5S8W3J5NSouT31KrNSnCCGEBCpCVJOtxDoomga7kouzKp7F3KxB7vt8tmoaDsDRk7k4XSpQXKPSPCZEf1y5GZVTRfUpMVKfIoQQEqgIUU22UkM3JYd/CksFKrFNggm2GHG6NH0VWk9GpUVcqP64cjMqRYW0klERQggJVISotpIrywIklQhU8vVAxZ0hMSgKLUsN/3iCmRZxYfrjShfTOvatR8s9BShyI0IhhEACFSGqzVOj0rooADlyIo/sPLt7n16jYtKPbxplBSA9uxCXqmJ3uoeAWsQWZ1SCSgz9OI9so/CHZQCYe41GCSo+TgghzlQSqAhRTZ4aldhIKy2LsiJ7jmQCxRmV4KDiQCUi1D1jJzvP4ZWNKRmoeDIqas4JCr5ZBKoLU/tBBJ31z7rriBBC+BEJVISoJk+wEWQ26tmSLE9GpShQCSkZqIS4A5WcfLtebGs2GYhuEozRoADFxbTO5K3gKMQQ24bgc29EUeStKYQQIIGKENXmWQcl2GIk1OoOSPKKpih7FnMLDioeyvEEKtn5dn0NFqvFiEFRSIgOKTrGDICacRQAU4seKEZzXXdFCCH8hqnqQ4QQULwybZDFCLgDkrwC9718CsrJqISHugOO7DyHfp+f4KIalpsu7cGxk3l6Ya2acQwAQ1TzOu6FEEL4FwlUhKgmT41KsMWoD914Fn0rHYiA99CPZ9jIk3FpFR+mLwoHJQKVSAlUhBCiJAlUhKgmPdgwG/XaEs/dkcvNqOiBikMPaErOCvLQCnPRCtw3OTRENquj1gshhH+SGhUhSsktcLDvWFaZ7XoxrcVISHDVNSrhnvoTTeNUdiFQvCBcSa7M4wAoYTEoFquvuiGEEAFBAhUhSnnti53MfXMzB1OyvbaXrFEJtbqDEE9GxVNoWzJjYjIaCC0KaNIy3KvTBlvKWTK/qJBWsilCCFGWzwOV1NRUunTpUubf6tWrAdi5cyeTJ0+mT58+jBo1ijfffNPXTRDitKRlFgBwMrPQa7s+9GMx6QFIfqETh9OF06UBZTMmnuGftAz3cwaXk1EpLqRt4asuCCFEwPB5jUpSUhJBQUF8++23KIqibw8PDycjI4PrrruOUaNG8eijj/Lnn3/y6KOPEhoaysSJE33dFCFqxZMdsTtd5W4PNhsJDS7KqBQ49GEfKJsxiQi1kJKeT2pRoGItL6OSKTN+hBCiIj4PVHbv3k3btm2Jj48vs++NN97AbDbz2GOPYTKZ6NChA4cOHWLZsmUSqIhGwzO7x7Pkvb695NBPUUbF7lTJzncv+hZsMWIwKF6P8ayTkpFjcx9TSUbFKDN+hBCiDJ8P/ezatYsOHTqUu2/Tpk0MGjQIk6n4l/VZZ53FwYMHOXnypK+bIkSNaZqmD/HYHaUClRLTk4ODTHgShiezKi6UDS9aRt+jdEZFsxeg5aUDklERQojy1ElGJSoqikmTJnHgwAHatGnDtGnTGDFiBCkpKXTu3NnreE/m5fjx48TGxtb6vCaTb2Muo9Hg9X8gkj6WZXe6cKnuehOnS9VfVyUDmFCrGYvZSEiwmbwCB5m57myJNchU5nUYGRbk9X2o1ex1jPNUCgBKSCTm0PCadk+uYYAI9D4Gev9A+liXfBqoOJ1O9u/fT8eOHZk5cyZhYWF88cUX3HTTTbz++usUFhZisXj/hRkU5P5FbrPZan1eg0EhKqpu7jQbERH400Wlj8WycotfhwaTUX9dOUoEMAnxEYRazUSEWMgrcJBT6A5gwkMtZV6HCXFhXt/HRod6HZNz+CQ5QHB8q9N6Dcs1DAyB3sdA7x9IH+uCTwMVk8nE+vXrMRqNBAcHA9CzZ0/27NnD8uXLCQ4Oxm63ez3GE6CEhITU+ryqqpGdnV/7hpfDaDQQEWElO7sAl0ut+gF+SPpY1omioleA7JxCMjLyAPfqsh4F+YXYC+1Yi9ZMOZLqnsZsNir68R4mNK/vXQ6n1zH5R/YDoIY3LfPY6pBrGBgCvY+B3j+QPtZGRIS1WtkZnw/9hIaW/auwU6dOrFu3joSEBNLS0rz2eb5v2rTpaZ3X6aybF4bLpdbZczcW0sdiuUX37gH3dGTPYzz39LGYDGgqOFVVr0nxBDfBFlOZc3hmB3lYTEavYxxHkwBQotuc1jWQaxgYAr2Pgd4/kD7WBZ8ONO3Zs4d+/fqxfv16r+3btm2jY8eODBw4kM2bN+NyFU/n/P3332nXrh0xMTG+bIoQteKZggxgd7hKbC95Q0I3z8wfvZi2nKnHntVpPawlVq5VC3NQTxwEwNiy52m2XAghApNPA5UOHTrQvn17HnvsMTZt2sS+fft48skn+fPPP5k2bRoTJ04kNzeXBx54gL1797J69WpWrFjBzTff7MtmCFFrnoAEvAMVz4yfIHPJQMUdhHiyMOXN+okoNeun5E0LXUe2AxqG6JYYQqNOv/FCCBGAfDr0YzAYWLJkCc899xx33HEH2dnZdO/enddff12f7fPqq68yd+5cJkyYQFxcHPfeey8TJkzwZTOEqDVbyUClRGqz0FE8Ndkj1Or99ikvUAkJMmE0KHohbsnHO49sAySbIoQQlfF5jUpsbCxPPvlkhfsTExN59913fX1aIXyioKKhH1vx8vkeIUGlhnXKGfpRFIWIUAsZOTYUioeONE3DVRSomCRQEUKICgXuhG8haqGwgoyKzeEOYIJqmFGB4jqV4CAjhqJV4tSMI2j5mWC0YEzoXO7jhBBCSKAihJeqalSCy6lR8agoUIkoujGhd31K0bBP8y4oJku5jxNCCCGBihBealSjEuwdmAQHlR36geI7KHseq6kqjgObARn2EUKIqkigIkQJFU5PtpU3PbmaGZVQs75fU1UK1y5HTd0LihFTm74+a7sQQgQiCVSEKMF76KdkjUrZQCWkVEbFaql86MdqNmBbtwLnnl9AMRB83i0YIsreZVwIIUQxCVSEKMG7mLbsgm9eNSrW6mVUWjd132wwMTQNR9JPoCgEj7oZc/uBPmu3EEIEKglUhCjBVmLox+nSUIvWP/EMCZUsiLWYDJiMiv69tYIalR7tonnq5rMYyh8AmHucj7nDYJ+3XQghApEEKkKUUDKjAsVZFVs5S+griqLXqSiK96q1pUUXJqOm7AaDCUvihb5uthBCBCwJVIQooUygUlSnYitn1g8U16lYLSYURaEi9i2fAWDuMgxDWLTP2iuEEIHO5yvTCuHPSs76geKZP4Xl3OsHimf+lDfsozltOHb/gutYknvdFMWApffFddFsIYQIWBKoCFFC6YyKrWgtFb2Y1lI6UCnKqJQqpNU0jYI1C3Ed3a5vM3cfhSEizudtFkKIQCaBihAleAISRQFNA4enRsVR9l4/ACHBnuXxvbc7d69zBylGM5Y+4zA264KxWZe6br4QQgQcCVQqYbc7GroJoh45nKp+l+PwEAvZeXa9RqWwnGJaKL7fT0iJQEXNz6Twt/+6jx8wAUvvi+q87UIIEaikmLYCf36ykgPPTOLwlt8buiminpSsT4koupGg3eFC07Tie/2UGfoxl9lu+3UV2PMxxLbF3GtMXTdbCCECmgQqFcjIKsCsuDBv/i+a0w5Aclou6dmFDdwyUVc8WROLyaBnTmwOFYdTRdXcmZbSxbQ920cTFR5En06xALgyj+HcvwFQCB5xHYqh4inLQgghqiaBSgWSoweSqYYQbM/A/vdXZOfbeWzFRp5958+GbpqoIyWzJhaTO8BwOF36DQmhbKDSoXkTnvvP2ZzVPcF9/LZvATC16YMxtk19NFsIIQKa1KhUICgkhI/z+3Nt2M/Yt3xOriuEIebdJGfHoWmDK10zQzReuw5nsGFnGhpgtRgZM7i1fi+e4pk9Jj0gsTtVPYCxmA0YDBVfd82Wh2P3OgDMvUbXYS+EEOLMIYFKBUKCTGyxt+XioIPE2ZIJ3/I2/wgFm2aiMPMcrFFyMzl/9PqXSaRlFujfB1uMXHJ2O6C4RiXIYsRidicbbQ6XVwBTGUfSWnDaMUS3wtisa100Xwghzjgy9FMB97oYCr+GnoehaUfyI9py0hVGkOLE/tuqhm6eqIVCu1MPUrq0igTgVImao5JrpZhN7reG3eEqHhKqZIl8TXVhLxr2sfQaLRk3IYTwEQlUKuCZbprijCR0/Gx2d7uJV3PPxaUpGI78ifPgFq/jnS61IZopauD4qXwAwkPMDOnprinJzLXr+0tmTixmT42KSr6t6IaEFdx0EMB5cDNaXjpKcDgmueGgEEL4jAQqFbAWrTiab3MU/e/kuCuKHwq7A1D4y1uo+VkAHDmRy23P/8SHa/c1TGNFtRw/lQdAs5hQIsPcdSmZuTZ9f/Edko0EFRXT2h0qOfnuYCa8qJalPPatXwPu1WcVU8XHCSGEqBkJVCrgyagUFLo/vPKL/v+qoDcOazRaXjr5nz6Bmn2C3cmZ2J0qOw6mN1h7RdU8GZXmsaE0CQ0Cys+oeNWoOF3k5LuD1XCrudzndaXtR03dCwYj5u7n1ln7hRDiTCTFtBXw3LvFk/b3/O/AxKEeU+m063W07FTyPnqE7lowcyLz+LugC5rWH0WR+K8xOnbSk1EJITLcHajk5NlxqSpGg8GrRsUz9GN3uMgpcAczYSHlByr2be5siqnDYAwhkXXZBSGEOOPIJ2oF9IxKUYDi+R8gyxBJyKUPYIhqCbY8Qu2niDAUMszwF4U/v4mmSb1KY3SsREYlPMSMQVHQgOw8d8bEVqJGxVNM63Cq5HoyKuUM/ag5J3Du2wiApadMSRZCCF+TjEoFPDUqTpeGw+nSh37AHbQYQqMIuexBXMd28u5Phyk8eZQrQtbjTPoRm6IQPPyahmq6KIfDqZKWURSoxIRiUBSahFnIyLGRmWsjKjxIr1GxWozF66g4VP1+P+ElMiqaoxD7X//DvnUNaC6MCZ0xxrWt304JIcQZQAKVClhLrJmRb3PpQz9QXK+imIMwtenD9oJCUmxhFGpmpoT9gmPnDxhbdMPcflC9t1uULzU9H00Da5BRL6RtElocqECpGhVT8Toq9qI7KJesUSn84RWcBzcDYIhtQ9CIa+urK0IIcUaRoZ8KGAyKXqdSYHOWyah4aJpGRo77g26zvT057S8AwLbuLdSC7HpssajM0ZPFM348a5xEhrnrVLKKCmpLzvrRa1ScrjJDP670o0VBikLwebcSMuERjJHN660vQghxJpFApRKhwcWBSsngpPTXthL3gjmScA6G6JZohTnYfnm7/horKlWykNaj9BRlr3VU9AXf1OJZP0VDP46tXwFgatcfc4dBsribEELUIQlUKhFSlOrPtzm9hn4KbMWBiSeb4pFVoBJ8zg2gGHDu3+CuYRANzhOoNI8N1bd5MiqeKcqemw+WzKgUlLj2YVYzan4mjj2/AWBJHFs/jRdCiDOYBCqVCA0uClQKnRSWrFEp8XXpQCU7z44xri2WgRMBsP32Xxy7fq6H1orKHCsx9OPRpHRGxVZ2HRXP9VUUCLWacWz/DlQnhqYdMTbtWG/tF0KIM5UEKpUIKRr6ycguRCuxveTQT3rpjEqe+3tL74sw9xoDQOFPr+E4sKluGysq5HKppJSYmuxRcY2KCUvRyrQu1X3lw6xmFNWJfcf3AFgSL6yfxgshxBlOApVKeDIqJ0vcuA68A5XMokDFs+5GVp77Q09RFILO+hfmLsNB0yj8bgnOI9vqo9milNT0fBwuFbPJQGxEsL69eOjHu0bFWiKj4hFmNeNK3ga2PJSQSExt+tZT64UQ4szm80AlMzOThx56iBEjRtCvXz/+/e9/s2lTcTbhuuuuo0uXLl7/pkyZ4utm+ERoUY3KqayKAxVPRqVVfBjgHvrxUBSFoOHXYWo3AFQnBV8vxJW6t66bLUpJTs0BoFl0CAZDceGrp5g2O9+OzeHSsyfBFqOeUfEID7Hg2Pc74F6BVjFIjC+EEPXB5+uo3HnnnZw4cYL58+cTExPDW2+9xdSpU/noo49o3749u3bt4pFHHuH888/XH2M2l780eUPzDP2cKsqoBFuMFNpdFNhcaJqGoij6X+NtEsLZfyxbz6h4KAYDwaNupmBNIa4j28j/33xCLpmFMaZV/XbmDHbYE6iUGPYBd/BhUBRUTWPHAfd9moLMRoItJn3tFI8oKzgPue+YbZa7IwshRL3x6Z+Fhw4d4pdffuGRRx5hwIABtGvXjgcffJD4+Hg+++wzTp06xalTp+jduzdxcXH6v8jISF82w2c8GZX0bHcwEtPEPWygapo+Jdmzr21COAA5eQ5UTfN6HsVoxnrB7RiadgR7PgVfPoualVIvfRBwJC0XgOYlpiaDe62ciFD3Nf7ujyMA9OoQg8GglMmodNQOgtOOEhGPIa5d3TdaCCEE4ONAJSoqimXLltGrVy99m6IoKIpCdnY2u3btQlEU2rXzj1/0IUU1KrkF7nU0IsOCMBStmeGZopyR4862tGkajoI7iPEcX5JiDiJk7AwMMa3QCrLJ++gxCn96HefxXfXQkzObPvQTE1pmn6dOZcfBDAAGdIkD3EGMyVj89mhrSwLc2RRZN0UIIeqPT4d+IiIiOOecc7y2rVmzhkOHDnH//feze/duwsPDeeyxx/jll18ICQlh7Nix3HrrrVgsZW/4VhMmk29rBoxGg77gm0eY1Yw1yEheoRO704WqaeQVrVjbNCaEsBAzOfkO8gqdRJco2ixuZDjhl9xLzqdPo6YfwZG0FkfSWoJ6XYD17KtQDMayj6lDxqIPYqMxcOstDAaFI2nuQKVV07Ayr5Oo8CAOprj3m40G+naO04+xmA04XSoRSj6xefsACO4yBKOPX2un40y4htJH/xfo/QPpY12q03v9/PHHH8yaNYvRo0czcuRI7r//fmw2G4mJiVx33XXs3LmTZ555hmPHjvHMM8/U+jwGg0JUVNm/lk9XiNW7diaqiZWwEAt5hU5MFjOuooLKIIuRFglNiI4IJiffgYtK2hMVSvTN8yk8tJ2cbT+T+/f32LZ+gyH/JE0n3IkhKKT8x9WhiAhrvZ+zvpzIKKDA5sJoUOjSPk6fneXRNDYM9pwEoF/XeJonNKHw6B4K9m2hrcVBml3llvDvMGguLE3bEduhS0N0o0qBfA09pI/+L9D7B9LHulBngcq3337L3XffTb9+/Zg3bx4Ajz32GPfddx9NmjQBoHPnzpjNZmbMmMG9995LbGxsrc6lqhrZ2fk+azt4MiregYpR0QgqmraaejKH9Az311FhQWRm5hNWFNgcTc2mbXwVgVNkB8zDOhDarAd53y2lYN8WklfMJmzc3RhCmvi0LxUxGg1ERFjJzi7A5VLr5Zz1bVdRkWzT6BBycwrK7LeWmIbcu0M06adyyP7gWdTcU9xsAmcTAyZFxWmNJuK8W8jIyKu3tlfHmXANpY/+L9D7B9LH2oiIsFYrO1Mngcrbb7/N3LlzGTt2LE8//bQ+rGMymfQgxaNTp04ApKSk1DpQAXA6ff/CCCk19BNsNup3Vc7Nd+jTWaPCg3A6Vf1eMBnZtmq3x9CmPyGXzKLgq+dxnTxE9uo5hFx4F4YmTX3Yk8q5XGqd/Pzq0qmsQlZ8lcS4IW3o0jqqwuOSPYW0sSHl9jGi6JoZDQq92sdgO/gnau4pMJpxulyYFJXDzhgiRtxJVFjTRvtz8sdrWFPSR/8X6P0D6WNd8PlA06pVq5gzZw6TJk1i/vz5XrUnU6ZMYdasWV7Hb926FbPZTNu2bX3dlNNWOqMSEmTS76icX+jUl1f3FGQ2CXX31bM6bXUZ49oRcun9KOGxaNlp5L13PwXfL8V18tDpdiFg/fz3MbYfSOfzXw9Wepx+j59yCmkB2jdvggIM6BpPaLAZ+44fATB3H8VrQdexLOdcFmaPISy69kG0EEKI2vNpRuXAgQM88cQTXHDBBdx8882cPHlS3xccHMyYMWN44oknSExMZNiwYWzdupVnnnmGqVOnEhYW5sum+ETpGhVrcHGgUmBzklE0NTk6whOouP/PLrWWSnUYmiQQcukDFP6wDNexnTj3/oZz3+9YBv4DS+8LZaZJKakZ7mGc/cdzUDVNn41Vmh6oxJX/+moVH8aztw4lPMSMmnsKV/JfAFi6jcR14DhJDvd6N2HWxrnWjxBCBDqfBipr1qzB4XDwzTff8M0333jtmzBhAk899RSKovDWW2/xxBNPEBcXx7XXXstNN93ky2b4jCco8QgJMhPiyajYnBw75f4QjIt0FxZ51uQovehbRfILHZzILKRN0RoshtAoQsbdh+vEAexbPsd5cDP2De+hpu0jeOQNKJbAL9KqrtR0d01Sgc1Janp+uVOPNU3jqH7X5IqLlD0ztGxJP4GmYWzWFUNkMyzmNMC90F/pIlwhhBD1w6eByi233MItt9xS6TGTJk1i0qRJvjxtnTEaFH01Wiga+gl2TyEusDn1+gfP8vmejEp1A5U31+xiw8407ruqr1edhTGuHcEX3IZj54/Yfl2J8+Bm8j9LwzpmBoawaJ/1z19pmqZnVAD2H8suN1DJyXeQV+BAUcpfQ6Xk87kO/4mj6IaD5m4jAbCY3dfaU3skhBCi/smfiVUIKZFVKTn0c/xUPrkFDgyKQouipdk9NSrVHfrZlZwJQNLhzDL7FEXB0v1cQi6dhWKNQD2VTP7Hj+FM3opWauXbM01OgcPrfkv7jmWXe9zxooxXfFQIQWbvNWo0TcOVtg/bls/J/3gOBWteQCvMQWnSFFO7/gAEFWVRwqynt8aPEEKI2qvTdVQCgTXYBEVFsyWLafcdzQIgISZE/8u7SdFN7nLyHdgdLn17efILHWTlugMaT2amPMb4DoRc9hAFXz2PmnGUgv89hyGmNUEDJpyxd/BNS/eeZrz/WFa5xx1KLcp4NQ0vs8+2/l0cf39VvMFowdLrAiy9L0IxujMoZsmoCCFEg5OMShVKZlRCgk369/aiqVmeYR9wF1x6VrNNSa98XZdjp4r3JxetnFoRQ3gsIeMfwNzzAjBZUE8dpmDNCxT+9Dqas2YzjAJBaob7Z9c0yl2zcyQtT7/3UklJh9zL4vdsH+O13Xn4Tz1IMbXtT9DQSYT++xmCBv0DJah4iMhSlFGRQEUIIRqOBCpV8KylYjQoWEyGMgW2rUsEKoqi6HfoPX6q8kDl+MnihcNOZBZ6DWWUR7GEEDx0EmFXzcecOBZQcCStJX/1I7jS9tWkS42Gpmm1Gsby1Kd0bRNFkzALqqZxKMU72HOpKruS3YFKYqfiqcVqfiaFPy4HwNzzAqyjb8fS8wIMIZFlztO86Fq2ii+bkRFCCFE/JFCpgicwsQaZUBSlTKBSMqMCxXfoPXay8hVMPTOGPCob/ilJCQ4j+Kx/Yb34HpSQSNTM4+R//DiFv67EmbwVNTPFL2pYTmQWcNuCn3j3+701fmyanlEJoX2zCMBdUFvSoZRcCmwuQoJNtG8RCbgDo8K1r6EV5mCIbkXQoH9Uep7hic148qazuGBAyxq3UQghhG9IoFIFz1CPJ7NSJlApVf/gWVjs+KnKA5XSGZfSgUp6dmGlz2Fq0Z3QKx7H1GkooOHY9g0F/3uOvPdmkv/JHFzpRys9f0Pbtv8UBTYXP/99DJdasxUOU4tqVJpGWenQwr3Scek6lZ2H3Evnd2sThdHgXmPFuedXXMl/g9FE8Hm3oJgqL5JVFIWm0SGyho0QQjQgCVSqYC0KUPSApUSgEhFq0Wf6eHiGfo5VMfTjybh0bun+oC1Zp6KqGk++/QePrtior35bHiU4DOu5N2EdeyemNn0xRLcEowk1bT/5qx+mcN2bOPb+hprV+LIsx0561kFxcSiletkk8ExNdj82Pro4o7LzUAa5BQ79OE99Sre27mnfan42hb+tAsDS7zKMUS1OvxNCCCHqnMz6qUJIkHcmxRpUPJOn9LAPFGdUUtPzcbpU8goc/LDlKCP7ttCX2rfZXZzKKgRgcPem7D6S5ZVROZyWw6ls9/4dB9M5u1ezSttoap2IqXUiLlXlx1+20S/ja0wp23Ds+F5fG0QJjcLYvDvmbiMxJXSq1c/Cl0oOfe08lE775hHVelxOvoNCuwsFiI8MpmmUlWYxIRw/lc+73+9h6sXdcThV9hxxZ1i6t41G0zTy170FtjwMMa2x9B5bF10SQghRBySjUoXo8OCi/91BRpDZqC/X3rqcQCU6IoggsxGXqnEis4AP1+7n018O8vLH21CLshop6flouGcJdW/rXsDtyIk8fQhkZ1E2oPTXVfl1aworfznJq7mjsI65A3OP8zDEdwCDCS0vA+eeXyj4dC75nz2J88i2Bs2ylAxUkmrQR082JToiCLPJiMlo4LoLu6EAv2xNYfuBdPYfy8LuVIkItdAiNpTMn9/HsXc9KArBI65HMUh8LoQQ/kJ+Y1dhYPd4bA4XPdu5Awp3Qa2RvEJnuRkVRVFoFhPCwZQcktNy2bLnBAB7jmTx45ajjOrXUv+Qbh4TQlyUlSCzEZvDRWp6Ac1jQ9l50DtQ0TStWnUSe4rWdjmUmoux9XBMbfoAoDntuFL34tz3O47dv+A6vouC47uwx7fHPPgiXOGt0ELjUJT6iVtLriED7p+Nw6nyy7bjbN51gpsv7VHhvXU89SnxUcVL4nds2YTz+rfk281HeOmjrfr6Nd3aRGHb8jkFv78PQNCQqzDGta2jXgkhhKgLEqhUwWIyMqJ3c69tLeLC2H8si04tI8t9TLOYUA6m5PDDH0fJK3SiABrw/o/76N0hVq9PaRYbikFRaBkfyr6j2RxOyyE+ysruI5n6c2Xk2EjLKKBpdMX3qvE4UDTzJd/m5FR2IbFNrGiahhMT5hbdMbXojqXfZdj//h+OnWtxpe3nxGeL3A82WzHGtsEQ2wbFYkXFgCkyAWPzrhis1RuWqS5P/U5kmAVVc6/k+8OWo7z3/V5UTWP9jlTO6+8900bVNHLyHRw54R4iK/3zuPyc9vy17yQnMgvQ7AV0M59gfMEvFPzunrptPetKTD0v8Gk/hBBC1D0JVGphxj96k1fo0G9mV5rnBnieJfKHJTbj+Kl89h7Ncv/FX7SQmKeepVV8OPuOZrP3SBbR4cHYHSrhIWaaRYew+0gWOw5l6B/MuQUOHnl9A81jQrnzn330cxbYnF5TopPTcoltYuXVz3fyx+4TTLqgM8MSm2EIiyZ46CQsfS/BtfM7tONJ2FIPgKMA1/EkXMeT9OfwrOxiiGmFueMQTB2HYAgtvidRbXnWkGkRG0pYiIX1O1J597s9eAaikg5leAUqmqYx983NHDhePAW5aZQVNTcd55GtuI7uQM0+wezwLDRTNopaVFSbBRiMRI34F3Qfg9NZs9lFQgghGp4EKrUQZDESZKl4efzmpW6AN7BrPDFNgpn75mYOlliYzLOgWGKHGH7ccpS1fx4jO9/9Idu1dRQtYkPZfSSLnYcyOLeve5bKH7tPkJ5tIz3bxvFTefrN9g4cz6ZkxUlyWi7d20azMSkVp0vjtS93svdoJpMu6ILZZMBgjcAyaCJRUaGkn8rGfvII6omDOE8l88fOYxTkF9DSmE4LUwbqqWRsp5KxbXjfXZDbaSimdv1RzOUHalXxDH01iwmlZXwY63ekouFeVM+laiQdzkDVNL0W6PipfD1IUYCIEBMDCn8jb9VXgHedjT5AFhSKufMwQvqMJap1azIyKp8uLoQQonGSQKUOeKYog3vWUNc2UZiMBh66biCLV2/lcNEMn2ZFi8P17hBD306xbNlzkk1JaYB7Wm2L2FA+XneApEPFH9ybdqXpz71p1wkuGeo+l2fBM0UBTYPk1Fz2Hs3C6dKwmAw4nCo//XWciFALl4/o4NVexWDEGN0KY3QrftiYzDsn9uj7zuvRhH90zMG5+xdcqXtwHd2O6+h2WPcmpnb9MbXogRIWjRIcgWI0gTkIxdqk0poazxoyzWNC6NYihAgln6bGLCb0MLBv3xGS7U04tqcpLTp1QlEMekFx99ZNuHNcK2y/v4Mz6Q8ADPHtMbXshSG2DQZrhPvc1ggUs7v42WCSenEhhPBnEqjUgbjIYExGBadLo0+nWExG94dlfKSV+6f059NfDmI0KPrQkaIoTB7dhaTDmfpS+t3aRBETEUyQ2UhugYMjabnENgn2KrTdvCuNS4a2BYoDlcT2Mfy17xTJabn6bJoBXePp1iaK5V/s5Ic/jjJuSFuvGybuO5rFT38dQ1M1ft2eAriHq9b9fZyfduVw6XlDCe82EjU7Dcee33Ds+QUtOw3nnl9x7vm17A/AEoIxuiVKaDSKNRxDaDRKk3gMEQkYIuJIP3mKC61/0vevDzBsymeOZzQpGVpYAAvw46/k/haKqVkXEo5lMzPiBE3zcsl7p+iePgYTwSOuxdx5mA+umBBCiMZKApU6YDQYaBUfzoHj2QzqFu+1z2I2csXIDmUeExUexD9HdWTF/5KIiwwmPtKKoih0aR3J3/tO8fXGZLq3jcKlasREBJORY+Nwai5pmQXENQnWV2Y9p08L/tp3irTMArbsOQm4g54hPRL4ZN0BTmYV8vuOVL1A2KVqvLR6KyeL1nUB6No6kmsv7EpyWi6HUnL46a9jXDykLYaIeIL6j8fS71LUtH049v6OmnkMNTcdR24WTqcDCy4M9nxcKbsr+OkozDAomKwqeNZnUwwoYdEY49qRnG0k9/gB2pjTsdjycB78g9bgfqVqgMGEIbolwWdPxti0Y+0vkhBCCL8ggUoduWFcN46eyCOxQ2zVBxcZntgMa5CJhBLLtl8ytC1b953i120p7C1axGxozwT2HnXXrvyx6wQDusSRne/AaFDo3jaKqPAgMnJsenFttzZRGAwKo/q15L0f9vLtpmSGJ7oXkduwPYWTWYWEBpu4YEArTCYDZ/dqhkFROL9/S5Z/sZPv/zjK6IGtMRcNoyiKgrFpRz1QWPnNbr47cAQAIy6aGrOYMjicplYHO5MOEUEurUPyITsNHAWYFI3jajTtRv8bU6teYArS+2tNy+Wp1zZgNSs8f1VLMvZt58uNx8g0RHLbtedjCo9FMchwjhBCnCkkUKkjzWJC9ULX6lIUhYFdvTMwHVo04bwBLfl20xHSMt1riPTvEkeTMAs7D2WwMSlNvw9Rq/gwLGYjreLD9KX3m0ZZ9SGm4b2b8fG6/Rw5kUfS4Ux6dYjhs5/3AzCybwsuHdbO69yDujXlg7X7yMix8fmvB5kwon2ZNv+55yTfbT6CosAV53SgwO7k818PsfhPMwqQnd8NgPgoKzeO68axo6l8+eN2mjRrzcz2A8o8X4u4UMKsZnILHBxyxbHP3I+fbRH07hCDuUl8meOFEEIENvnT1A9cPqI9sU3cwUZ8pJVW8WH07RQHuGf7rPife0qxZxn6kgvRdWtTPJ04NNjM2T3dmZR3v9/D3/tOsnXfSQyKos8qKslsMjDp/M4AfPn7oTI3TswvdPLW17sAGDuoNRee1YZLhrajWUwIOfkOsvMdtIwLJSYiiLSMAua+9Qevf3+UVDWS5nFlF8sDMCgKXYvavGZDMn/vKx6+EkIIceaRQMUPBFtMTL24G1HhQVx4VmsURSEqPIhhic1QcE/JDQkycVb3BMA7UOla6gN+zKBWWINMHE7NZd5//wRgQNe4CteE6d8ljn6d43CpGq9/udPrTscf/LiXjBwb8VFWxhdlY8wmAzeM605cZDAjejfngasH8PB1g+jbKRZFcbfVGmSkf9eKsyMjEpuhKO6p2EmHM8vthxBCiDODojW22+rWgsulkp7u23UyTCYDUVGhZGTk+d1CYSnp+dy/7HcAFkwfRkSI9x2eUzPyeWn1Nn2V19nXDNDvQlyejBwbs19dT4HNSc/20Uy9qBtf/H6Ibze561Luu6ovXVr7NpDYeSiDpZ9sIzvfQZjVzILpw/R1VWrCn69jdQR6/0D6GAgCvX8gfayN6OhQjMaq8yVSoxKAEqJDmDC8HSHB5jJBCkDTqBAeuLo//1t/mCbhwXRq2QSXq+J4NSo8iBsu7saST7ezbX86dy/+FZfqPn7C8HY+D1LAPdTz8HWDWP3TPnq0ja5VkCKEEML/SaASoC45u12l+4OKpkl7ouPSK7yW1rdzHLOvHsBLH20lLaMAa5CJG8Z102tl6kJUeBBTL+5eZ88vhBCi8ZNARVRbq/gwHrpmIBt2ptKzXTSxkdaGbpIQQogAJ4GKqJGQYBMjy5khJIQQQtQFmfUjhBBCiEZLAhUhhBBCNFoSqAghhBCi0ZJARQghhBCNlgQqQgghhGi0JFARQgghRKMlgYoQQgghGq0GCVRUVWXhwoUMHz6cPn36cOONN5KcnNwQTRFCCCFEI9YggcrixYtZtWoVc+bM4Z133kFVVW644QbsdntDNEcIIYQQjVS9Byp2u53XXnuN6dOnM3LkSLp27crzzz9PSkoKX3/9dX03RwghhBCNWL0voZ+UlEReXh5DhgzRt0VERNC9e3c2btzIuHHjavW8JpNvYy7PraercwtqfyV99H+B3j+QPgaCQO8fSB/rUr0HKikpKQA0a9bMa3t8fLy+r6YMBoWoqNDTblt5IiIC/8Z70kf/F+j9A+ljIAj0/oH0sS7Ue6BSUFAAgMVi8doeFBREVlZWrZ5TVTWys/NPu20lGY0GIiKsZGcX4HKpPn3uxkL66P8CvX8gfQwEgd4/kD7WRkSEtVrZmXoPVIKDgwF3rYrnawCbzYbVWrsozWBQ6izCCw0NqpPnbUykj/4v0PsH0sdAEOj9A+ljTRgMSrWOq/dAxTPkk5aWRuvWrfXtaWlpdOnSpVbPqSgKRmP1OlxTgTze6CF99H+B3j+QPgaCQO8fSB/rQr3/RLt27UpYWBjr16/Xt2VnZ7Njxw4GDhxY380RQgghRCNW7xkVi8XC5MmTmTdvHtHR0bRo0YJnn32WhIQERo8eXd/NEUIIIUQjVu+BCsD06dNxOp3Mnj2bwsJCBg4cyPLlyzGbzQ3RHCGEEEI0UoqmaVpDN0IIIYQQojyBX/UjhBBCCL8lgYoQQgghGi0JVIQQQgjRaEmgIoQQQohGSwIVIYQQQjRaEqgIIYQQotGSQEUIIYQQjZYEKkIIIYRotCRQEUIIIUSjJYFKOVRVZeHChQwfPpw+ffpw4403kpyc3NDNqrXMzEweeughRowYQb9+/fj3v//Npk2b9P3XXXcdXbp08fo3ZcqUBmxxzaWmppbpQ5cuXVi9ejUAO3fuZPLkyfTp04dRo0bx5ptvNnCLq2/9+vXl9q1Lly6cd955ALz88svl7vcXS5cuLfOaq+qa+dv7tLw+fv/990ycOJG+ffsyatQonn76aQoLC/X9mzdvLve6lrypa2NRXv9mz55dpu2jRo3S9/v7NZwyZUqF782PP/4YAJfLRWJiYpn9L774YgP1wltVnw+//fYbl19+Ob1792bs2LF88cUXXo+32Ww8+uijDBkyhL59+3LXXXeRnp7u20ZqoowXX3xRGzx4sPbDDz9oO3fu1K6//npt9OjRms1ma+im1cp1112njRs3Ttu4caO2f/9+7dFHH9USExO1ffv2aZqmaUOGDNFWrVqlpaWl6f8yMjIattE19OOPP2q9evXSUlNTvfpRUFCgpaena4MHD9ZmzZql7d27V/vggw+0Xr16aR988EFDN7tabDabV5/S0tK0r7/+WuvSpYveh//7v//T7rnnnjLH+YO3335b69q1qzZ58mR9W3WumT+9T8vr48aNG7Vu3bppL7/8snbgwAHtxx9/1EaMGKHNnDlTP2blypXa+eefX+a6NrY+ltc/TdO0K664Qps/f75X20+dOqXv9/drmJGR4dW31NRU7aqrrtIuvvhiLTc3V9M0Tdu7d6/WuXNnbefOnV7HevY3tMo+H/bu3av16tVLmz9/vrZ3717t1Vdf1bp37679+uuv+uNnzpypnX/++drGjRu1v/76S7vsssu0SZMm+bSNEqiUYrPZtL59+2orV67Ut2VlZWmJiYnaZ5991oAtq52DBw9qnTt31jZt2qRvU1VVO//887UFCxZoJ0+e1Dp37qxt3769AVt5+pYtW6Zdcskl5e5bsmSJNmzYMM3hcOjbnnvuOW306NH11TyfysvL084991yvD7QLL7xQe/311xuuUbWQkpKi3XzzzVqfPn20sWPHen0AVHXN/OV9Wlkf77rrLu3aa6/1Ov6jjz7SevTooX9QP/zww9ott9xSr22uicr6p6qq1qdPH+3rr78u97GBcA1Le+utt7SePXvqfwRqmqZ98cUXWr9+/eqjqTVW1efDgw8+qF1xxRVej7nzzju166+/XtM098+ma9eu2o8//qjv379/v9a5c2ftjz/+8Fk7ZeinlKSkJPLy8hgyZIi+LSIigu7du7Nx48YGbFntREVFsWzZMnr16qVvUxQFRVHIzs5m165dKIpCu3btGrCVp2/Xrl106NCh3H2bNm1i0KBBmEzFNws/66yzOHjwICdPnqyvJvrMkiVLKCgo4L777gPAbrdz8OBB2rdv38Atq5nt27djNpv59NNP6d27t9e+qq6Zv7xPK+vj9ddfr19DD4PBgMPhIDc3F6j8dd0YVNa/w4cPk5+fX+HrMhCuYUnp6eksWLCAadOmefW5MV/Dqj4fNm3a5HV9wP0+3Lx5M5qmsXnzZn2bR7t27WjatKlPr6Gp6kPOLCkpKQA0a9bMa3t8fLy+z59ERERwzjnneG1bs2YNhw4d4v7772f37t2Eh4fz2GOP8csvvxASEsLYsWO59dZbsVgsDdTqmtu9ezdRUVFMmjSJAwcO0KZNG6ZNm8aIESNISUmhc+fOXsfHx8cDcPz4cWJjYxuiybWSnp7OihUruOuuu4iMjARg7969uFwu1qxZw9y5c7HZbAwcOJB77rlH72djNGrUKK96hZKqumb+8j6trI/du3f3+t7hcLBixQp69uxJdHQ0AHv27CEqKorLL7+c1NRUOnfuzIwZM0hMTKzztldHZf3bvXs3AG+99RY//fQTBoOBESNGMGPGDMLDwwPiGpb0yiuvEBwczNSpU7227969G6fTydSpU0lKSqJp06Zcc801jB8/vq6aXG1VfT589NFHJCQkeO2Pj4+noKCAjIwMUlNTiYqKIigoqMwxvryGklEppaCgAKDMh3RQUBA2m60hmuRTf/zxB7NmzWL06NGMHDmS3bt3Y7PZSExM5NVXX2XatGm8//77zJ49u6GbWm1Op5P9+/eTlZXF7bffzrJly+jTpw833XQTv/32G4WFheVeT8DvrumqVasIDw/nn//8p77N84FgtVp54YUXmDt3Lvv37+fqq6/2Ksz0J1Vds0B7nzqdTu6991727NnDww8/DLgDspycHPLz85k9ezaLFy8mNjaWyZMns3fv3gZucdV2796NwWAgPj6eJUuWMHPmTNatW8ett96KqqoBdQ1zc3N57733mDp1apkP7T179pCZmcmUKVNYvnw5Y8aMYdasWXzwwQcN1NqKlf58KO996PnebrdTUFBQ7h+0vr6GklEpJTg4GHBfBM/X4P7laLVaG6pZPvHtt99y9913069fP+bNmwfAY489xn333UeTJk0A6Ny5M2azmRkzZnDvvff6RbbBZDKxfv16jEajfs169uzJnj17WL58OcHBwdjtdq/HeN5EISEh9d7e0/Hxxx9z2WWXeb02L7vsMkaMGKH/FQ7QqVMnRowYwffff89FF13UEE09LVVds0B6n+bm5nLHHXewYcMGFi1apGdLmjVrxsaNG7FarZjNZgB69erFjh07eOutt3j00UcbstlVmjZtGldddRVRUVGA+3dLXFwcV155JVu3bg2oa/jtt99it9uZOHFimX2ff/45LpeL0NBQALp27cqxY8dYvnw5V1xxRX03tULlfT4EBQWVeR96vrdareW+T8H311AyKqV40pBpaWle29PS0mjatGlDNMkn3n77bW6//XbOPfdclixZokf9JpNJD1I8OnXqBNCo0q9VCQ0N9fplB+5+pKamkpCQUO71BPzqmiYlJZGcnMwll1xSZl/JIAXcqdfIyEi/uoYlVXXNAuV9mpaWxqRJk/jzzz9Zvnx5mTR8RESEHqSAu4alQ4cOpKam1ndTa8xgMOhBikfJ3y2Bcg3B/SF/zjnnEBERUWZfcHCwHqR4dO7cuVG9Nyv6fGjWrFm51yckJITw8HASEhLIzMwsE6z4+hpKoFJK165dCQsL81qnIDs7mx07djBw4MAGbFntrVq1ijlz5jBp0iTmz5/vlaqbMmUKs2bN8jp+69atmM1m2rZtW88trZ09e/bQr1+/MmtLbNu2jY4dOzJw4EA2b96My+XS9/3++++0a9eOmJiY+m5urW3atImYmBi6du3qtf35559nzJgxaJqmbzty5AgZGRl07NixvpvpE1Vds0B4n2ZlZXHNNdeQnp7OypUry7T7p59+om/fvl7rijidTpKSkvziut57771ce+21Xtu2bt0KQMeOHQPiGnqUV3QK7v4MGjRIX8/JY+vWrXrQ1tAq+3wYMGAAGzZs8Dr+999/p1+/fhgMBvr374+qqnpRLcCBAwdITU316TWUQKUUi8XC5MmTmTdvHt999x1JSUnMmDGDhIQERo8e3dDNq7EDBw7wxBNPcMEFF3DzzTdz8uRJTpw4wYkTJ8jJyWHMmDF88skn/Pe//yU5OZkvv/ySZ555hqlTpxIWFtbQza+WDh060L59ex577DE2bdrEvn37ePLJJ/nzzz+ZNm0aEydOJDc3lwceeIC9e/eyevVqVqxYwc0339zQTa+RHTt2lLuI2wUXXMDRo0d55JFHOHDgABs3buT222+nX79+DB8+vAFaevqqumaB8D598sknSU5O5tlnnyU6Olp/X544cQKXy0W/fv2IiorivvvuY9u2bezatYv77ruPzMzMMgFAYzRmzBh+++03Fi1axOHDh1m7di33338/48aNo0OHDgFxDcFdS5SRkVHmDwhwZ8TOOussnn/+/9u7/5iq6z2O488jHmYkAZ0DNqPGao2WB5OKQ4QSHNpsnlqDtmoBYQtZLiRE59F1Vjrs1OQgEkQkRyiyWmVro8FqSg6Wf0RuLvphW2i20DwhGUSJ8ePcP5jfyT1wu3mZnO5ej+38wfl8z/m++ZydfV/7/Djfajo7Ozlx4gS7d++mtbWVdevWzUG1U/3V9aGgoICenh68Xi/Hjh2jqamJjz76iKKiImBydNPpdOJ2u/nss8/o6emhvLwcu93OsmXLZq1OrVGZRmlpKWNjY7jdbkZGRkhJSWHPnj1ThmD/KT7++GNGR0fZv38/+/fvn9KWk5PDiy++iMlk4o033sDj8RAbG8vq1aspLi6eo4r/vnnz5tHQ0EBVVRVlZWUMDQ1xyy230NzcbOwc8fl8PP/88+Tk5BAbG8umTZvIycmZ48r/nv7+fmOnz8VsNhuNjY3U1NSQm5tLeHg42dnZuFwuTCbT5S90Flgslr/8zP7J39Px8XHa29sZHR2lsLAwqL2jo4P4+Hhee+01vF4vTzzxBOfPn+f2229n7969/4i1Y9nZ2ezatYvdu3fT2NhIZGQk999/P2VlZcYx/+TP8IL+/n6Aab+bAB6Ph9raWp577jkGBga48cYbjV/jnWv/zfWhvr6eyspKXn/9deLj46msrJwyelRRUYHH46GkpASAjIyMWd+MYQpcPF4sIiIiEkI09SMiIiIhS0FFREREQpaCioiIiIQsBRUREREJWQoqIiIiErIUVERERCRkKaiIiIhIyFJQERERkZCloCIic66vr4/ExMSge6Jcis2bN+NwOGahKhEJBfoJfRGZc3Fxcbzzzjtcf/31c12KiIQYBRURmXPh4eGzehMzEfn/oakfEflL7733Hk6nE5vNRmZmJrW1tYyPjwOTUy0FBQXs27ePrKwskpOTKSws5NtvvzVePzExQXV1NQ6HA5vNhsPhoKqqitHRUWD6qZ8TJ05QWlpKeno6y5Yto6CgYMrt5AEGBwfZsmULdrudlJQUKisrmZiYCKr/wIED5ObmkpSURHp6Otu3b+ePP/4w2kdGRti6dSsZGRnYbDbuvfde9uzZM6t9KCKXRiMqIvIfvfrqq1RXV5Ofn8+WLVs4evQotbW1/PTTT3g8HgCOHj3K8ePHKS8vJyoqipdeeon8/Hza29uJi4ujsbGRt99+G5fLxXXXXccXX3xBdXU1ZrOZ0tLSoHP29vby0EMPkZCQgNvtxmw209LSQmFhIU1NTdjtdiYmJigqKuLkyZO4XC6io6Px+Xx8+eWXxMXFGe/14YcfsnHjRuPOvSdPnqS6upre3l6am5sxmUx4PB4+/fRTXC4XVquVrq4uduzYQXR0NA8++OBl62sRCaagIiIz+u2336ivr+fhhx82bt2+fPlyoqOjcbvdPP7448ZxDQ0N3HHHHQAsXbqUe+65h5aWFjZu3Eh3dzc2m8246Nvtdq644goiIyOnPW9dXR3h4eG0tLSwcOFCADIzM7nvvvvYsWMH+/bto6uri56eHhobG8nIyAAgLS1tykLaQCCA1+tlxYoVeL1e4/mEhARWr15NZ2cnmZmZdHd3k56ejtPpBCA1NZWIiAgsFstsdqeIXAJN/YjIjI4cOcLIyAgOh4OxsTHjcSEMHDp0CID4+HgjpMDk4tjk5GQ+//xzYPLCf+jQIR599FF8Ph+9vb3k5+fzwAMPTHve7u5usrKyjJACMH/+fJxOJ1999RW///47hw8fxmw2s2LFCuOYiIgI7r77buPv48ePc/r06aD6U1JSWLhwoVF/amoq7777LmvWrGHv3r38+OOPPPXUU2RmZs5OR4rIJdOIiojM6NdffwWguLh42vaff/4ZgEWLFgW1WSwWvv76awCKioq48soref/99/F6vVRWVnLTTTfhdru58847g147ODiI1WoNet5qtRIIBBgeHmZwcJDo6GhMJtOUY2JjY4Pq37ZtG9u2bZux/meeeYZrrrmG1tZWKioqqKioIDk5ma1bt3LzzTdP+7+LyOWhoCIiM7rqqqsA8Hq9JCQkBLVbrVZqamo4e/ZsUNuZM2eMqZN58+aRl5dHXl4eAwMDdHZ20tDQwLp164xRjYtFRUVx5syZoOf7+/sBiImJISYmhrNnzzI+Pk5YWJhxzIVwcnH9mzZtwm63T3semNx1tHbtWtauXcupU6c4ePAg9fX1bNiwgba2tpm6R0QuA039iMiMbr31VsxmM36/n6SkJOMxf/58du7cSV9fHzC5Q+fYsWPG6/x+P0eOHCEtLQ2ARx55hO3btwOTIy25ubnk5eUxNDTE8PBw0HlTUlI4ePDglLbx8XHa2tpISkoiPDyctLQ0xsbGOHDggHHMn3/+OSX43HDDDVgsFvr6+qbUv2jRIqqqqvjmm28YGRlh5cqVNDU1AbB48WLy8vJwOp2cOnVqFntTRC6FRlREZEYxMTEUFRVRU1PD8PAwqamp+P1+ampqMJlMxrRIIBDgySefZP369YSFhVFXV0dUVBQFBQXAZPBoamrCarWSnJyM3++nubkZu93O1VdfPWWrMEBJSQldXV089thjFBcXYzabjbUjPp8PmFw4u3z5ctxuNwMDA1x77bW0tLTwyy+/GCM5YWFhrF+/nmeffZawsDCysrIYGhqivr4ev9/PkiVLWLBgAUuWLKGurg6z2UxiYiLff/89H3zwAStXrryMvS0i0zEFAoHAXBchIqHtzTff5K233uKHH34gKiqKtLQ0ysvLWbx4MZs3b6a7u5s1a9bw8ssvc+7cOe666y5cLhfx8fEAjI2N8corr9Da2srp06eJjIzE4XCwYcMGYmJi6OvrIzs7mxdeeIHc3Fxgcsvzzp07OXz4MCaTiaVLl1JSUjJl0e65c+fwer20tbVx/vx5Vq1aRUREBB0dHXzyySfGce3t7fh8Pr777jsiIiK47bbbKCsrIzExEYDh4WF27dpFR0cH/f39WCwWVq1axdNPP82CBQsuY0+LyL9TUBGR/8mFoHJxMBARmS1aoyIiIiIhS0FFREREQpamfkRERCRkaURFREREQpaCioiIiIQsBRUREREJWQoqIiIiErIUVERERCRkKaiIiIhIyFJQERERkZCloCIiIiIh618aLFYcOQzydAAAAABJRU5ErkJggg==", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "开始测试！\n", - "回合：1/20，奖励：200.00\n", - "回合：2/20，奖励：200.00\n", - "回合：3/20，奖励：200.00\n", - "回合：4/20，奖励：200.00\n", - "回合：5/20，奖励：200.00\n", - "回合：6/20，奖励：200.00\n", - "回合：7/20，奖励：200.00\n", - "回合：8/20，奖励：200.00\n", - "回合：9/20，奖励：200.00\n", - "回合：10/20，奖励：200.00\n", - "回合：11/20，奖励：200.00\n", - "回合：12/20，奖励：200.00\n", - "回合：13/20，奖励：200.00\n", - "回合：14/20，奖励：200.00\n", - "回合：15/20，奖励：200.00\n", - "回合：16/20，奖励：200.00\n", - "回合：17/20，奖励：200.00\n", - "回合：18/20，奖励：200.00\n", - "回合：19/20，奖励：200.00\n", - "回合：20/20，奖励：200.00\n", - "完成测试\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHJCAYAAAB5WBhaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAABrmElEQVR4nO3deVxUZf//8RfDooig4AKalTvmgoKCkuRCpuWSZt32zSVNct+1VMw1U8uFzD1SM7eyRb1NLUrvXPJ2Q6009yUVF0RFERVZZn5/+GNuJ0CBcDvzfj4ePJRzrnPO9ZkZmDfnus4ZB4vFYkFERETEIEwPuwMiIiIieUnhRkRERAxF4UZEREQMReFGREREDEXhRkRERAxF4UZEREQMReFGREREDEXhRkRERAxF4Uby3P26L6TuNyliDPpZlvtN4Uby1Pr16xkyZEie73fXrl107drV+n1MTAy+vr4sX748z48lxrBt2zaaNGlC1apVefvttzNt06FDB3x9fa1flSpVwt/fn9atW7Nw4UJSU1Mz3W7Xrl3079+f+vXr4+fnx/PPP8+IESM4duxYpseoXLkye/fuzXRfoaGhDB069K61rFmzhoYNG1K1alVGjhx5j8pzLyoqirCwMJ599llq1KhB8+bNmTVrFomJiXmy/+TkZMaPH8/3339vXTZ9+nSb58DX15fKlStTu3ZtevXqxZEjR3J0jOXLl+Pr60tMTEye9PleLl68yKBBg6hduzY1a9Zk4MCBXLhw4YEcW7Lm9LA7IMayYMGC+7Lfb775xuaNo3jx4ixbtoynnnrqvhxPHn8TJ07EbDYTGRlJkSJFsmxXuXJlRo0aBUBaWhpXr15l06ZNTJgwgejoaKZOnYrJ9L+/Az/99FM+/vhjQkJCGDx4MMWKFePkyZN8+eWXvPLKK0yYMIFmzZrZHCMtLY3w8HCWL1+Oi4tLjmt5//33KV26NB9++CHe3t453v5ezGYz7777Lj/++COvvvoqb7zxBm5ubvz222/MmzePdevWsWDBAjw8PP7RcS5cuMAXX3zBhAkTMqxbtmyZ9f9paWmcPXuWjz/+mHbt2rFmzRqKFSv2j459P6SmptKlSxcSExMZPXo0qampTJkyhbCwMJYvX46zs/PD7qLdUriRx5KLiws1atR42N2QR9iVK1cIDAzk2WefvWu7ggULZngthYaGUrZsWcaNG8fq1at5+eWXAdi4cSMRERH07NmTfv36WdsHBQXRqlUrBg0axNChQ6lYsSIVKlSwrnd3d+fIkSPMnDmTAQMG5KqWunXrUrt27Rxvmx1z585l9erVzJgxgxdeeMG6PDg4mKCgINq1a8fMmTMJDw+/L8cHMjwHNWvWpESJErRr144VK1bYnLl9VPz444/s37+fNWvWUL58eQCeeeYZmjdvzg8//GB93ciDp2EpyTMdOnRgx44d7NixA19fX7Zv3w7c/sU8cuRInn32WapVq0abNm3YunWrzbZbtmyhTZs2+Pv7ExgYSI8ePaxnaoYOHcqKFSs4c+aMdSjq78NSy5cvp3Llyvz++++8/vrrVKtWjYYNGzJv3jyb41y4cIEBAwYQFBREYGAgI0eO5OOPPyY0NPSutV24cIEhQ4YQHByMv78/7du3Z8+ePUDWQ2RDhw612W+HDh1455136Nu3LzVq1OCtt96iSZMm9O3bN8PxWrZsSY8ePazfr1u3jtatW1OtWjXq1q3LBx98wI0bN+7a57S0NJYsWUKLFi3w8/OjQYMGTJ48mVu3btn0sVOnTnz33XfWIZyWLVuyadOmu+4bYOXKlbzyyitUr16dBg0aMGXKFJKTk4HbQw2hoaH88ssvvPjii1SvXp02bdpYXxOQ9fBBdoZp/vrrL/r27UvdunWpUaMGHTp0YNeuXcD/no8zZ86wcuVKm9diTrRv3x5vb2+++uor67LZs2dTtmzZTJ8zZ2dn3n//fRwdHfnss89s1j3zzDO0atWKuXPnsm/fvmz3Yfv27fj6+gIwc+ZMm8dry5YttG3blpo1a1K7dm0GDRrEuXPnrNum/0x888031K1bl6CgII4ePZrhGCkpKcyfP5969erZBJt0NWvWpG/fvtY3b7j9GA8ePJiQkBCqVKlCcHAwgwcPJj4+3tomNDSU8ePH07FjR/z8/OjUqRPPP/88AOHh4ff8mQOoWrUqAGfOnLEu27t3L2FhYdSuXZuAgAC6d+9+z6Gr6Oho2rdvT/Xq1QkKCmLIkCFcvnz5rtuMGDGCunXrkpaWZrN83Lhx1K5dm5SUFH799VfKlClj89iUL1+ecuXKsXHjxnvWJ/ePwo3kmVGjRlG5cmUqV67MsmXLqFKlCrdu3aJjx46sX7+eAQMGMGPGDHx8fHj77betAef06dP07NmTqlWrMnv2bMaNG8eJEyfo2rUrZrOZnj17Ur9+fYoVK8ayZcto0KBBpsc3m83079+fpk2bEhkZSUBAABMnTmTz5s3A7fH+jh07snv3boYNG8aECRM4ePAg8+fPv2td169f54033mD79u28++67zJgxg3z58tG5c2f++uuvHD1GP/zwA25ubsyePZu3336bl19+mY0bN9rMaTh27BgHDx6kZcuWAHz//ff06tWLsmXLMnPmTHr37s2qVavo2bPnXSdmjhw5kgkTJtCoUSNmz55Nu3btWLx4cYbt9u3bx7x58+jbty8zZ87E0dGRPn36cPXq1Sz3vWTJEoYMGUKVKlWYMWMGXbt2ZdGiRXzwwQfWNpcvX2bIkCG0bduWTz75hPz58xMWFsaBAwdy9Jj93dGjR2ndujUxMTEMHz6cyZMn4+DgQMeOHdmxY4d1yLJYsWLUr1/f+lrMKZPJRHBwMH/88QepqanEx8ezZ88enn/+eRwcHDLdpnDhwjz77LOsX78+w7phw4bh6elJeHi4NQTeS5UqVazDNa+99hrLli2jePHirFy5ks6dO1OiRAkiIiIIDw9nz549vP7661y6dMm6fVpaGvPnz2fcuHGEh4dTrly5DMf4888/iY+Pp2HDhln2o2fPnvzrX/8C4ObNm7z55pscO3aMUaNGMW/ePN58803WrFnDxx9/bLPdkiVLqFatGrNmzaJnz57MmDEDgB49elj/fzcnTpwAsA4/b9u2jTfeeAOA8ePH88EHH3Du3Dn+7//+L9P5TgA7d+6kU6dO5M+fn6lTpzJs2DB27NjBm2++SVJSUpbHbtmyJRcvXrQJxmazmR9++IFmzZrh7OzMsWPHKF26dIZtn3rqKWvf5eHQsJTkmfLly1OwYEHgf6eYv/76aw4ePMjXX39N9erVAahXrx4dOnRg8uTJfPfdd/zxxx8kJSXRrVs363wCHx8f1q9fz40bN3jqqafw8vKyGYrK7KyFxWKx+SVcs2ZNfv75ZzZs2MBzzz3HqlWrOH78ON999531L8I6derQqFGju9aVftZoxYoVPPPMMwAEBATQqlUrdu7cSXBwcLYfI2dnZ8aMGWOdd/HUU08xffp01q1bR6tWrQBYvXo1Hh4ehIaGYrFYmDx5Ms899xyTJ0+27qd06dJ06tSJjRs3Zhr2jh49yrfffsugQYOsp/Pr1q1L8eLFGTx4MJs2baJ+/foAXLt2jeXLl1vfQAoUKED79u2tE3L/zmw2M3PmTBo1amQTZm7evMmaNWtISUmxfj969GhrXemPdWRkZIY3wZyYMWMGLi4uLFy40Pp6a9CgAc2bN2fixIl8++231KhRAxcXF7y8vP7R8GXRokVJSUnhypUrxMbGAvDEE0/cdZunn36a9evXc+XKFQoXLmxdXqhQId5//3169OiR7eGpO4fMfHx8qFGjBmazmcmTJxMSEsKUKVOsbQMCAmjatCnz5s1j8ODB1uXdu3fP8g8CwHq2p1SpUvfsD9w+a+bj48NHH33Ek08+Cdx+bn///Xd27Nhh07ZkyZK888471u/Tzzo99dRTVK5c2abtnZO3k5KSOHjwIOPHj8fd3d06vDNlyhSefvppIiMjcXR0BCAkJIQXXniBadOm8cknn2To75QpUyhTpgyffvqpdZvq1avTrFkzvvvuO9q1a5dpnTVr1uSJJ55g9erV1qHN7du3ExcXZ/3D49q1azz99NMZtnVzc+P69etZPYTyAOjMjdxXW7dupVixYlSpUoXU1FRSU1NJS0ujYcOG7Nu3j6tXr1K9enXy5cvHa6+9xrhx49i8eTOVKlViwIAB1jev7PL397f+P/3NLT0Ibdu2jSeffNIabOD2m8fd/mKF21fGlCpVyhpsAFxdXYmKirIGqewqW7aszYTSJ598koCAANauXWtdtmbNGl588UVcXFw4fvw458+fJzQ01Pr4paamEhgYSMGCBdmyZUumx0l/k/n7xNZmzZrh6Oho89eol5eXzcRsHx8f4HY4ycyJEye4dOlShiGMv0+idHJyonnz5tb1+fPnp169euzcuTPrBygbduzYQcOGDW1eG05OTjRr1ox9+/bl6ZtK+hmurM7UZCa9rdlszrAuNDSUl19+mblz5/Lnn3/mqk8nTpwgLi7O5rGF24HB398/Q8C483WbGScnpyz7m5lnnnmGpUuX8sQTT/DXX3+xceNG5s2bx/HjxzOckbrXse9UpUoV61fNmjVp164dycnJzJgxg2LFinHjxg327t3LSy+9ZA0pAB4eHjRs2DBD3XD7Nfz7779Tv359LBaL9efnySefpFy5ctafnzt/tlJTUzGbzTg4OPDyyy+zbt06a11r1qyhdOnS1j/U7nbmNCevGcl7OnMj99WVK1eIi4vLclggLi6O8uXLs3jxYiIjI/n2229ZuHAhHh4etG3blv79++fol0T+/PltvjeZTNZfQPHx8ZleNXO3K2nSa7hXm+xyc3PLsKxly5aMHTuW+Ph4YmJiOHnyJOPHj7ceG2DMmDGMGTMmw7ZZXXKaPqT09ytMnJyc8PT05Nq1a9Zlrq6uNm3u9uZ8Z5/u9ZgULVrU+saZrkiRItbtc+vq1asULVo00+NZLBYSExMzfZxzIzY2lvz581O4cGHr43KvS4xjYmIoUKCAzVmbOw0fPpytW7cSHh7Od999l+M+pT9+WT0G+/fvt1lWoECBu+6vZMmSgO28lr+7fPkyBQsWtAbzzz//nDlz5nDlyhWKFi1K1apVcXV1tXldZefYd/r222+t/3d2dqZYsWI2r7Fr165hsViyrPvvxwZISEjAbDbz2WefZZgHBZAvXz6ADL+fevfuTZ8+fWjZsiWzZ89m8+bNPPfcc/z000907NjR2q5gwYKZhunExETc3d2zUbXcLwo3cl+5u7tTunRpmyGVO6WfCvfz82PGjBkkJyeza9culi1bxpw5c6hUqRIvvfRSnvTF29s70zkyd85RyIy7u3umb2i7d++mUKFC1kD194mH95rwm+6ll17igw8+YN26dRw/fpwnnniCmjVrAlgvvR08eDBBQUEZti1UqFCm+0xfHhcXZzOMkpKSQnx8PJ6entnqW2bS+/T3CZnx8fHs37/fevYssxBz8eJF6xtWViHqXmdeChUqxMWLFzMsj4uLA/hHtd0pNTWV7du3ExAQgKOjI15eXvj7+7Nu3ToGDRpkvTz86tWrJCQk8OSTT3Lt2jX++9//EhISYnP5+N/7P3r0aHr16sWsWbNy3K/00JTVY5DT+p955hmKFi3Kpk2bshyiGT58OL/99hsbNmwgKiqKDz/8kHfffZfWrVvj5eUFQL9+/bK8l092VKtW7a7r3d3dcXBwyLLuzMKkm5sbDg4OdOrUKcNZTPhfsL8zWMHtW00AlClTBj8/P3744QdMJhMJCQk2V0CVKVMm0zlkp06dws/P7671yP2lYSnJU3//hR4UFMS5c+coUqQI1apVs35t2bKFuXPn4ujoyIIFC2jYsCHJycm4uLgQHBzM2LFjATh79mym+82NoKAgYmJibH4ZJSUlWSccZ6VWrVqcPn3a5oqMW7du0adPH7799lvr8Ej6nAy4HSL++OOPbPUr/bT6+vXriYqK4uWXX7a+8ZctW5YiRYoQExNj8/h5e3szZcqUDH+l31kr3D6Nfqc1a9aQlpZmDU+5UbZsWTw9Pfnll19slv/73/+ma9eu1jk3f39sk5KS2LRpk3WOUvrjdv78eWubY8eO3fPMTmBgIL/88ovNJOy0tDTWrFlDtWrVcnUfmcwsW7aMuLg46wRWuP0X/cmTJ23mdvz66680btyYoUOHMnLkSG7evEn37t3vuu9GjRrRvHlzIiMj73nVzt+VKVOGYsWKsXr1apvlp0+f5rfffiMgICBH+zOZTHTq1IkNGzbwn//8J8P6bdu2sXHjRutQ6a5du/Dw8ODtt9+2Bpvr16+za9euew5t3TmclFMFChSgatWq/PDDDzZ/SFy7do0NGzZk+pouWLAglStX5vjx4zY/PxUqVGD69OnW4dk716X/fKVr2bIlmzdvZs2aNQQEBFjnGcHt+T7Hjh2zuQrt6NGjHDt2jLp16+a6VvnndOZG8pSHhwd79uxh69atVK5cmdatW7N48WLeeustunfvTokSJfjvf//LZ599Rvv27XF2dqZOnTpMnjyZXr160b59exwdHfnqq69wcXGxzofx8PDg4sWLbNy4MUfj+HdKfzPp1asX/fr1w8PDg88//5xLly5ZT81npnXr1ixatIgePXrQt29fPD09WbhwISkpKbRt25ZChQrh7+/PokWLePrppylUqBALFy4kKSkp26flX375Zfr27UtaWpp1siLcfjMYMGAAI0eOxNHRkYYNG5KQkMCsWbOIjY3NcrivfPnyvPLKK0ybNo2bN28SGBjIgQMHmDFjBrVr1+a5557L2YN3h/Srqd5//32KFClCaGgoJ06cYNq0abRr187mbFJ4eDj9+/enSJEizJs3jxs3blgvca9duzb58+fnww8/pF+/fly/fp1p06ZlOZyTrnfv3mzatIk333yTrl274uzszOLFizl9+jRz587NcT2JiYn89ttvwO2zSPHx8fz6668sW7aMl19+mcaNG1vbhoSEMGTIECZOnMj+/ft55ZVX8Pb25s0337TewLJ169bZujprxIgRbNu2LdMzEXdjMpkYOHAg4eHhDBo0iJdffpn4+HhmzJhBoUKFeOutt3K0P4BOnTqxc+dO+vTpQ5s2bahfvz4mk4mdO3eyaNEinnnmGQYNGgTcPsv65Zdf8uGHH9KwYUMuXLjAvHnzuHjxYpZnEtOlD9Vs3bqVcuXKWeeuZNegQYMICwuja9eutG3blpSUFCIjI0lOTqZXr16ZbjNw4EC6du1qfazSryD7/fff6dmz5z2P2bRpUz788EPWrl1rvdnjnevmzJlDly5drI/PlClTqFixYp6dcZbcUbiRPNWuXTv27dtHly5dmDBhAi1atGDJkiVMmTKFSZMmce3aNZ544gkGDRpE586dAahUqRJz5sxh5syZDBw4kLS0NKpWrcr8+fMpW7YscPsNY+PGjfTq1Yu+ffvStGnTHPfNycmJefPmMW7cOEaPHo2TkxMvv/wyhQsXvutlmwULFmTx4sVMnDiRsWPHYjabqVGjBgsXLrT+Fffhhx8yduxYhg8fTsGCBXnttdeoWbMm33zzTbb6Vr9+fdzd3XnyyScpU6aMzbp//etfuLm5MXfuXJYtW0aBAgUICAhg8uTJNn9F/t24ceN4+umn+e677/jss88oXrw4b775Jj179vzHZ8LatWtHgQIFmDdvHsuWLcPHx4cuXbrQpUsXm3ajR49m/PjxXL58mYCAAL788kvr1SUeHh5Mnz6dKVOm0KtXL5544gl69+7NypUr73rsChUqsHTpUusl0A4ODvj5+bFw4UJq1aqV41r279/P66+/DtweKnNzc6NixYqMHj060wnjb731FjVq1OCLL77gww8/JD4+nqJFi9KqVSuefPJJ5s6dy9mzZxk3btxdr0AqXLgwo0ePpnfv3jnuc+vWrXFzc+PTTz+lV69eFCxYkOeee46BAwfm6k6+zs7OzJo1i2XLlvHvf/+btWvXkpyczJNPPknPnj1p3769Nai/8sorxMTE8N1337F06VK8vb2pX78+bdu2tX4ERWaXnMPtn6W33nqLZcuWsXHjxiwnxGclODiYzz//nGnTpjFw4EBcXFyoVasWH330kc1NE+8UEhLCvHnzmDFjBn379sXZ2ZkqVarw+eefZ+tKOi8vL0JCQtiyZQsvvviizToXFxc+//xzxo0bx4gRI3B2dqZu3bqEh4dnmG8mD5aDRZ9gJnbiyJEjHD9+nMaNG9tMUn7ttdfw8fHJ1n03JHumT5/OjBkzOHTo0MPuygN39uxZFi1aRO/evfNsYrOI5IyipdiNGzdu0K9fP9q2bcsLL7xAWloaa9euZd++fTb34hD5J0qWLHlfPjxWRLJP4UbsRvXq1Zk6dSrz5s1j5cqVWCwWKleuzNy5c6lTp87D7p6IiOQRDUuJiIiIoehScBERETEUhRsRERExFIUbERERMRSFGxERETEUu7xaymKxYDbfn3nUJpPDfdv3o8aeagX7qle1Gpc91atajcVkcsj2BynbZbgxmy1cvnz3D+fLDScnE56ebiQk3CA19e6fsfK4s6dawb7qVa3GZU/1qlbj8fJyw9Exe+FGw1IiIiJiKAo3IiIiYigKNyIiImIoCjciIiJiKHY5oVhERB5fZrOZtLTUe7RxICnJkeTkW6SlGfsqIqPU6ujohMmUN+dcFG5EROSxYLFYSEi4zM2bidlqf/GiCbPZuFcP3ckotbq6FsTDwyvbl3xnReFGREQeC+nBpmBBT1xc8t3zDdDR0eGxPpORE497rRaLheTkWyQmxgNQqFCRf7Q/hRsREXnkmc1p1mBTsKBHtrZxcjIZ+r4vdzJCrS4u+QBITIzH3d3zHw1RaUKxiIg88tLS0oD/vQGKMaU/v/eaU3UvCjciIvLY+KdzMeTRllfPb47DzZUrVxg5ciT16tUjICCAN954g+jo6AztTp48SY0aNYiJibFZfuvWLcaMGUNwcDD+/v4MGjSIy5cv3/WYMTExdOvWjYCAAEJCQpg6dao1xYuIiIjcKcfhZuDAgezZs4eIiAi+++47nnnmGcLCwjh+/Li1zbFjx+jcuTM3b97MsP3o0aP59ddfmT59Ol988QXHjx+nb9++WR4vJSWFsLAwAL766itGjx7Nl19+ycyZM3PadREREbEDOQo3J0+eZMuWLYwePZpatWpRpkwZRowYQfHixfn+++8B+PTTT3nttdcoVKhQhu1jY2NZuXIlw4cPp1atWvj5+REREcHOnTvZs2dPpseMiori7NmzTJw4kYoVK9KoUSMGDhzIF198QXJyci5KFhERkezo3bsr48aNftjdyLEcXS3l6elJZGQk1apVsy5zcLj9EeQJCQkArFu3jgkTJuDp6cmbb75ps/2uXbsAqFOnjnVZmTJl8Pb2ZufOnfj7+2c4ZnR0NFWqVLEJS3Xq1CExMZEDBw5QvXr1nJRg5eSU99ONHB1NNv8amT3VCvZVr2o1rse5XrM5Z3Mx0qduODiA5fG9QjpbjFiro6PDP3qfzlG48fDwoH79+jbLoqKiOHnyJMOGDQPgm2++AWD79u0Zto+NjcXT05N8+WxnuxcvXpzz589neszz58/j4+OToT3AuXPnchVuTCYHPD3dcrxddnl4uN63fT9q7KlWsK96VatxPY71JiU5cvGiKcdveo9jkMut+1Fr+gmM+3FCIDNmswMmk4lChQqQP3/+XO/nH93nZvfu3YSHh9O4cWMaNGhwz/Y3b97ExcUlw/J8+fJx69atTLdJSkrCw8MjQ3sgy23uxWy2kJBwI1fb3o2jowkPD1cSEm6SlvZ432/gXuypVrCvelWrcT3O9SYn3/r/H7tgsbmfi8ViITklYy0ODrfrTUsz37ezGS7Oplxd3RMSUou33urC2rXfk5qawowZn+HjU4LPPpvNTz/9wPXriZQpU4633+5OUFAdjh07SseO/8e8eYvx9a0EQHj4O+zevZO1a/+Dk5MjDg7QtGkj+vQZSJMmTfn++5V8++1XnD59GpPJgYoVK9G370AqVaoMwGuvtaBBg+fZtm0L8fGX+eCDiVSpUo05c6bz008/kpKSTMuWr2I2m7FY/veYL126iJUrvyUu7gJFixajWbOX6dgxLM+uckpLs2A2m7l69QY3b9peOOTh4ZrtAJfrcLNu3TreeecdAgICmDx5cra2yZ8/f6bzZG7duoWra+Z/SWS2TXqoKVCgQA57/T/382ZHaWnmx/5mStllT7WCfdWrWo3rcaw3s7vvWiwWJizezdEzVx9Cj6B8qUKEtwvI1Rv7ihXfMHnyNFJT03jyyacYPfo9Tp48wciRYylWrDhbtmxi8OD+jB8/mWefDaFEiZLs3LkNX99KpKWlsWdPNDdu3ODw4YM880wVDhzYx7Vr1wgODmHjxl/4+OOJDBkynOrV/bl48SJTp07iww8/YMGCpdY+LF/+NR999DHu7u6ULVueqVMnsWXLZt57bxTe3iVYuHA+v/++h5IlnwDg1183sWjR57z//niefLI0f/75Bx98MIoSJUrSpEnTPHtcgQwhNqdyFW4WL17MuHHjePHFF/noo48yPRuTGR8fH65cuUJycrLNNhcuXMDb2zvLbQ4fPmyz7MKFCwBZbiMiInbiMb3tTZMmTa1nUWJiTrNuXRSff76EChV8Afi//2vP0aNHWLp0Ic8+G0Ldus+xc+d22rfvxIEDf+Lk5EzVqtXYvTuaZ56pwpYtv1K9uj8eHh4UKlSIoUNH0LjxSwD4+JSgefOXiYiYaNOHOnXqEhhYG4AbN67zww+rGTRoCMHBIQCEh49k9+7/3erl7NkYXFyc8fEpiY+PDz4+PhQtWhxvb9upI4+CHIebpUuXMnbsWDp06MB7772Xo8Ras2ZNzGYzu3btIjg4GIATJ04QGxtLYGBgptsEBgaycuVKEhMTKViwIADbtm3Dzc2NSpUq5bT7IiJiEA4ODoS3C8h0WAru/0cS5HZYCqBUqaes/z98+BAAPXu+bdMmNTWVggXdAahb9zlWrVrBrVtJ7Ny5nZo1a+HjU5Jdu6Jp164jW7b8yosvNgOgRo0A/vrrBAsWzOXkyb+IiTnFsWNHM3ywZqlST1r/f+rUSVJSUqhUqYp1Wb58+ahY0df6fePGTVmzZhVvvNGa0qXLEhhYmwYNns8wL/ZRkKNwc+LECcaPH88LL7xAt27duHjxonVd/vz5cXd3v+v23t7eNGvWjOHDhzN+/HhcXV0ZNWoUQUFB1KhRA4Dk5GSuXr1KoUKFcHFxoVGjRkydOpX+/fvzzjvvEBMTQ0REBJ07d872GSMRETEmBwcH8rk4ZrrOycmEo+nRPLVz54U1Fsvt0DFz5mcUKGB7sUv65yv5+9fC2dmZPXt2Ex29gyZNmlKiRAmWL/+a8+fPceTIIcaNu31m5qeffmTcuFE0bvwSVav60bJla44fP0ZExEdZ9iH9FFh6X9I5Of0vJhQuXJjPP1/Kvn1/sHPndrZv38o333xJWFg33nqryz97QPJYjqY/R0VFkZKSws8//0xISIjN17hx47K1j7FjxxIcHEzv3r0JCwujbNmyTJs2zbp+z549hISEWO97ky9fPubOnYvZbKZNmzaMGTOGtm3b0rNnz5x0XURE5JFUpkw5AC5dukipUk9av9asWcXatbfvIefk5ERQUDC//rqR/fv3UbNmIH5+NUhLS2PevE8pV648JUqUBGDJkgW0aNGK994bzauvtqFGjQDOnLn9aQGWLGZXP/XU07i45OOPP363LktNTeXIkf9NC/nppx9YseJb/PxqEBbWjcjI28dZv/6n+/K4/BM5OnPTvXt3unfvnq22tWvX5tChQxmWFyhQgA8++IAPPvgg29s9/fTTzJ8/PyddFREReSyULVuOZ599jkmTJjBw4BDKlCnLhg3rWbx4AcOGjbK2Cwmpx0cffUDRosV44olSAFSt6kdU1Fo6dQqztite3Ju9e3/n0KGDFCxYkF9/3cjy5V8Dt0dH/n47Frj93vzaa22YP/9TihYtSunSZfnyy0VcvBhnbZOcfIuZMz/Bzc2N6tX9uXDhAnv27KZGjYz3qHvY/tGl4CIiIvLPvf/+BCIjZzJp0niuXUugZMlSDB06gpdeam5tExxcl7S0NAICalmX1aoVxO7d0dSr97970A0YMJiJE8fRu3dXXFycKV++IsOHj2HUqGEcPLif6tUzDyPduvXGxSUfEREfcePGDUJDX6Bu3XrW9c2bt+Lq1assWDCXCxdicXd3p0GD5+nRI+uPUHpYHCxZnaMysLQ0M5cvX8/z/To5mfD0dCM+/vpjd5llTtlTrWBf9apW43qc601JSebSpXMUKVICZ+fszbe83xOKHyVGqfVuz7OXl1u273NjP7duFBEREbugcCMiIiKGonAjIiIihqJwIyIiIoaicCMiIiKGonAjIiIihqJwIyIiIoaicCMiIiKGonAjIiIihqJwIyIiIoaicCMiImLHUlNTWbZsifX7efM+5bXXWuT5ce7XfjOjcCMiImLHfv75R6ZP//hhdyNPKdyIiIjYMSN+frbTw+6AiIhIblksFkhNzmKdCcv9/KRsJxccHBxyvNnWrVuYO3cOf/11HFfXAgQH16VPn4EcPXqYAQN68f77HzJnznRiY2OpWrUa7703mi+/XMSPP67BycmZf/3r/+jYMcy6vx9+WM2yZUs4deoUXl5eNG/ekg4d3sLR0RGA2NjzfPrpTKKjd3DjxnX8/GrQs2c/ypevwNq13zN+/BgAQkJqMW3aHOt+Fy9ewHfffc3Vq1epUqUqgwe/x5NPPgVAYmIiM2d+wubNv5CSkoKv7zP07NmXSpUqW7f/97+Xs3TpQuLi4ggMDKJEiZK5ephzQ+FGREQeSxaLhRurxmGOPfpQju/oXQHXl4flKOBcuXKF9957l969B/DssyFcuBDL2LGjmDXrExo3fom0tDQWLpzPqFEfkJqayrvv9qdTp7Y0b96SyMgv+OmnH/jss9mEhNSnXLnyfP31UubMmUHfvgOpWTOI/fv3ERHxEVevXqVfv0HcuHGdHj3CKFnyCT78cArOzi7Mnx9J795dWLDgS55//gUSExOZNm0K//73j3h4FGLPnl2cP3+OvXt/Z9KkT0hJSWbs2JF8+OFYZs78DIvFwrvv9sXFJT8ffTSVggUL8uOPa+jRI4xPP/2cihUr8fPPPxIR8RH9+r1DrVpBbNr0C5GRsyhe3Ps+PiP/o2EpERF5bDmQ8zMnD1NcXCzJycl4e/vg41MCP78afPRRBK+++rq1zdtvd6dSpcpUrepHzZqBuLq60rNnX5566mk6dOgEwPHjR7FYLCxe/AWtW7fhtdfa8OSTT9GkSVPCwrqzYsU3JCYmEhX1A1evXmHs2I+oXLkqFSpUZPToD8iXLz/Ll39Nvnz5KViwIABFihTF2dkZACcnJ0aOHEv58hV45pkqtGzZmoMH9wOwa9dO9u3by9ixE6hSpSpPP12abt16UaVKNb755isAvv12GY0aNaZ163/x1FNP0759J+rWfe6BPc46cyMiIo8lBwcHXF8eluWwlJOTidRHbFiqQgVfGjVqwpAhAyhSpCiBgbV59tnnqFevAX/88RsApUo9aW3v6upKiRIlrcfJly8/ACkpKVy5Es/ly5fw86thcwx//wBSU1M5efIvjh07ypNPPo2np6d1fb58+alcuQrHjh3Lsp9eXkVwcyto/d7d3YNbt24BcPjwQSwWC6++2txmm+TkZGub48eP0qhRE5v1Vav6ceTI4ew8TP+Ywo2IiDy2HBwcwDlf5uucTDg43Mdwk0ujR4+jc+cubNv2X3bu3M7YsSPw86thnUfj5GT71pxVgMpqIrDZbLljP1m1MePk5JhlH02mrAd2zGYzbm5uzJu3OMO69DM/4IDFYvvY/72u+0nDUiIiIg/In3/uY9q0KTz1VGnatGnLpEmfEB4+kl27dhIfH5+jfXl5FcHLq4j1jE+633/fg7OzM088UYpy5Spw+vRJ4uMvW9ffunWLgwcPULp0WSDr8JSVsmXLc/36dVJSUihV6knr15IlX/DrrxsBqFChIn/88bvNdgcPHsjRcf4JhRsREZEHxM3NjeXLv2HWrGnExJzm+PGjrF//E6VKPUXhwoVzvL833ujA8uVf89133xATc5qffvqR+fMjefnlVyhYsCAvvPAihQoVZsSIoRw48CdHjx7h/feHc/PmTVq2bA3cHvqC2+Hj1q2kex6zdu1gKlSoyKhR4ezeHU1MzGmmT49g7drvrYGpfftObNr0C0uXLuT06VN8++1XbNiwPsf15ZaGpURERB6Q0qXLMG7cJD7//DNWrPgGk8lEQEAgU6ZMIzb2fI7398Yb7XFxcearr5bw8ceTKF7cm3btOtK2bQcAChYsyPTpnzJjxlT69esJgJ9fdWbPnkfJkk8AEBAQSOXKVenRozMjRoy95zEdHR35+ONZzJr1CSNHDuXmzZuULl2WceMmUbNmIADPPhvCqFEfMH9+JHPnzqFKlWr83/+15+eff8xxjbnhYDHi3XvuIS3NzOXL1/N8v05OJjw93YiPv35/J7E9AuypVrCvelWrcT3O9aakJHPp0jmKFCmBs7NLtra57xOKHyFGqfVuz7OXlxuOjtkbcNKwlIiIiBiKwo2IiIgYisKNiIiIGIrCjYiIiBiKwo2IiDw27PAaGLuSV8+vwo2IiDzy0j/hOjn51kPuidxP6c+vo+M/u1NNjra+cuUKERERbNiwgcTERHx9fRk0aBC1atUCYOvWrUyaNIljx45RokQJ+vTpQ7NmzQBYvnw54eHhme63du3aLFy4MNN1q1at4t13382wfP369ZQqVSon3RcRkceUyeSIq2tBEhNv38XXxSXfPe+sazY7kJZmH2d6HvdaLRYLycm3SEyMx9W14F0//iE7chRuBg4cSFxcHBERERQpUoRFixYRFhbGihUrsFgsdOvWjbfeeotJkyaxYcMGBg8ejJeXF8HBwTRt2pTnnrP9RNAff/yRCRMm0L179yyPeejQIYKCgoiIiLBZ7uXllZOui4jIY87D4/bv/fSAcy8mkwmz+fG/90t2GKVWV9eC1uf5n8h2uDl58iRbtmxh6dKl1KxZE4ARI0awefNmvv/+ey5duoSvry8DBgwAoFy5cuzfv5+5c+cSHBxM/vz5yZ8/v3V/58+f55NPPqFnz548++yzWR738OHD+Pr6UqxYsdzWKCIiBuDg4EChQkVwd/ckLS31rm0dHR0oVKgAV6/eeKzPaGSHUWp1dHT6x2ds0mU73Hh6ehIZGUm1atWsyxwcHHBwcCAhIYHo6GgaNWpks02dOnUYN24cFoslw+nDSZMmUbx4cbp27XrX4x46dIjQ0NDsdjPbnJzyfrpR+p0Ts3sHxceZPdUK9lWvajUu49Rr4l5vX46OJvLnz09ysoW0tMf/jMbd2FOt2ZXtcOPh4UH9+vVtlkVFRXHy5EmGDRvGihUr8PHxsVlfvHhxbt68SXx8vM0w0qFDh1i9ejUzZ87ExSXr22hfvXqV2NhYoqOjWbp0KfHx8fj5+fHuu+9SpkyZ7HY9A5PJAU9Pt1xvfy8eHq73bd+PGnuqFeyrXtVqXPZUr2q1T7mejrx7927Cw8Np3LgxDRo0ICkpKUNQSf8+OTnZZvmCBQvw9fXl+eefv+sxjhw5AtyeaDRhwgSSkpKYPXs2bdu25fvvv6do0aK56rvZbCEh4Uautr0bR0cTHh6uJCTcNHx6tqdawb7qVa3GZU/1qlbj8fBwzfZZx1yFm3Xr1vHOO+8QEBDA5MmTAciXL1+GEJP+ffrHqQMkJSXx448/8u67795zpnutWrXYunUrnp6e1rYzZsygQYMGLF++/J5DWndzPz9gLC3NbIgPMMsOe6oV7Kte1Wpc9lSvarVPOR54Xbx4MX369KFhw4bMmTOHfPnyAVCiRAkuXLhg0/bChQsUKFAAd3d367ItW7aQkpLCSy+9lK3jeXl52YQgV1dXSpUqRWxsbE67LiIiInYgR+Fm6dKljB07lnbt2hEREWEzDFWrVi127Nhh037btm0EBATYzH6Ojo6mUqVKeHp63vN4y5Yto3bt2ty48b8hpMTERP766y/Kly+fk66LiIiInch2uDlx4gTjx4/nhRdeoFu3bly8eJG4uDji4uK4du0aHTp04I8//mDy5MkcO3aM+fPn8+OPP/L222/b7Gf//v1UqlQp02OkpaURFxdHUlISAPXq1cNsNjN48GCOHDnC3r176dOnD15eXrRu3foflC0iIiJGle1wExUVRUpKCj///DMhISE2X+PGjaNChQrMmjWLjRs30qpVK7755hsmTZpEcHCwzX7i4uIoXLhwpsc4d+4cISEhrF27Frg91LVgwQJu3LjBG2+8QadOnXB3d2fhwoXW4TARERGROzlY7PBTyNLSzFy+fD3P9+vkZMLT0434+OuGn9RlT7WCfdWrWo3LnupVrcbj5eWW7aulHvc7OYmIiIjYULgRERERQ1G4EREREUNRuBERERFDUbgRERERQ1G4EREREUNRuBERERFDUbgRERERQ1G4EREREUNRuBERERFDUbgRERERQ1G4EREREUNRuBERERFDUbgRERERQ1G4EREREUNRuBERERFDUbgRERERQ1G4EREREUNRuBERERFDUbgRERERQ1G4EREREUNRuBERERFDUbgRERERQ1G4EREREUNRuBERERFDUbgRERERQ1G4EREREUNRuBERERFDUbgRERERQ1G4EREREUPJcbi5cuUKI0eOpF69egQEBPDGG28QHR1tXb9161Zat25N9erVefHFF1mzZo3N9rt27cLX1zfD1/bt27M8ZkxMDN26dSMgIICQkBCmTp1KWlpaTrsuIiIidsAppxsMHDiQuLg4IiIiKFKkCIsWLSIsLIwVK1ZgsVjo1q0bb731FpMmTWLDhg0MHjwYLy8vgoODATh06BBPPfUUS5cutdlvoUKFMj1eSkoKYWFhlC5dmq+++opTp07x3nvvYTKZ6Nu3by5KFhERESPLUbg5efIkW7ZsYenSpdSsWROAESNGsHnzZr7//nsuXbqEr68vAwYMAKBcuXLs37+fuXPnWsPN4cOHKV++PMWKFcvWMaOiojh79ixff/01hQoVomLFily6dImJEyfSvXt3XFxcclKCiIiIGFyOwo2npyeRkZFUq1bNuszBwQEHBwcSEhKIjo6mUaNGNtvUqVOHcePGYbFYcHBw4NChQ9ZglB3R0dFUqVLF5sxOnTp1SExM5MCBA1SvXj0nJVg5OeX9dCNHR5PNv0ZmT7WCfdWrWo3LnupVrfYtR+HGw8OD+vXr2yyLiori5MmTDBs2jBUrVuDj42Ozvnjx4ty8eZP4+Hi8vLw4cuQInp6etG7dmtjYWCpWrMiAAQPw8/PL9Jjnz5/PdJ8A586dy1W4MZkc8PR0y/F22eXh4Xrf9v2osadawb7qVa3GZU/1qlb7lOM5N3favXs34eHhNG7cmAYNGpCUlJRhmCj9++TkZM6dO8e1a9e4ceMGw4cPx9HRkcWLF9O+fXuWL19O+fLlMxwjKSkJDw8Pm2X58uUD4NatW7nqt9lsISHhRq62vRtHRxMeHq4kJNwkLc2c5/t/lNhTrWBf9apW47KnelWr8Xh4uGb77FSuw826det45513CAgIYPLkycDt0JGcnGzTLv17V1dXChUqxM6dO3F1dcXZ2RmAatWqsX//fhYtWsSYMWMyHCd//vwZ9pkeagoUKJDb7pOaev9eAGlp5vu6/0eJPdUK9lWvajUue6pXtdqnXA3QLV68mD59+tCwYUPmzJljPZNSokQJLly4YNP2woULFChQAHd3d+D20FZ6sAEwmUyUK1eO2NjYTI/l4+OT6T4BvL29c9N9ERERMbAch5ulS5cyduxY2rVrR0REhM0wVK1atdixY4dN+23bthEQEIDJZGLTpk34+/tz+vRp6/rU1FQOHjyY6ZAUQGBgIPv37ycxMdFmn25ublSqVCmn3RcRERGDy1G4OXHiBOPHj+eFF16gW7duXLx4kbi4OOLi4rh27RodOnTgjz/+YPLkyRw7doz58+fz448/8vbbbwMQEBCAp6cnQ4YMYd++fRw6dIghQ4Zw5coVOnXqBNwexoqLi7MORTVq1IhixYrRv39/Dh48yLp164iIiKBz5866DFxEREQyyFG4iYqKIiUlhZ9//pmQkBCbr3HjxlGhQgVmzZrFxo0badWqFd988w2TJk2y3uOmYMGCLFiwgKJFixIWFsbrr7/OlStXWLx4MUWLFgVgz549hISEsGfPHuD2PJ65c+diNptp06YNY8aMoW3btvTs2TOPHwoRERExAgeLxWJ52J140NLSzFy+fD3P9+vkZMLT0434+OuGn9RlT7WCfdWrWo3LnupVrcbj5eWW7auldMcfERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExlByHmytXrjBy5Ejq1atHQEAAb7zxBtHR0db1W7dupXXr1lSvXp0XX3yRNWvW2Gx/7tw5Bg4cSN26dQkMDCQsLIwjR47c9ZjDhw/H19fX5is0NDSnXRcRERE74JTTDQYOHEhcXBwREREUKVKERYsWERYWxooVK7BYLHTr1o233nqLSZMmsWHDBgYPHoyXlxfBwcEkJyfTtWtXChcuzJw5c8ifPz/Tp0+nY8eOrF69Gi8vr0yPeejQIbp370779u2tyxwdHXNftYiIiBhWjsLNyZMn2bJlC0uXLqVmzZoAjBgxgs2bN/P9999z6dIlfH19GTBgAADlypVj//79zJ07l+DgYKKjozl8+DCbNm3C29sbgEmTJlG7dm3+85//8Nprr2U4psVi4ejRo3Tt2pVixYr903pFRETE4HIUbjw9PYmMjKRatWrWZQ4ODjg4OJCQkEB0dDSNGjWy2aZOnTqMGzcOi8VChQoViIyMtAYbAJPp9shYQkJCpsc8deoUN27coGzZsjnp6j05OeX9dCNHR5PNv0ZmT7WCfdWrWo3LnupVrfYtR+HGw8OD+vXr2yyLiori5MmTDBs2jBUrVuDj42Ozvnjx4ty8eZP4+HiKFSuWYftFixaRlJRE3bp1Mz3m4cOHre02bdqEyWSiXr16DBgwAHd395x038pkcsDT0y1X22aHh4frfdv3o8aeagX7qle1Gpc91ata7VOO59zcaffu3YSHh9O4cWMaNGhAUlISLi4uNm3Sv09OTs6w/c8//8yUKVPo1KkTvr6+mR7j8OHDmEwmihcvzpw5czh16hQTJ07kyJEjfPHFF9YzPzlhNltISLiR4+3uxdHRhIeHKwkJN0lLM+f5/h8l9lQr2Fe9qtW47Kle1Wo8Hh6u2T47letws27dOt555x0CAgKYPHkyAPny5csQYtK/d3W1TZRffvklY8eO5eWXX2bw4MFZHqdHjx60bdsWT09PACpWrEixYsVo06YNe/fupXr16rnqf2rq/XsBpKWZ7+v+HyX2VCvYV72q1bjsqV7Vap9yNUC3ePFi+vTpQ8OGDZkzZw758uUDoESJEly4cMGm7YULFyhQoIDNENKkSZMYPXo0b775JhMmTLjr2ReTyWQNNukqVKgAwPnz53PTfRERETGwHIebpUuXMnbsWNq1a0dERITNMFStWrXYsWOHTftt27YREBBgDTCTJk1i7ty5DBkyhKFDh+Lg4HDX4w0ePJhOnTrZLNu7dy8A5cuXz2n3RURExOByFG5OnDjB+PHjeeGFF+jWrRsXL14kLi6OuLg4rl27RocOHfjjjz+YPHkyx44dY/78+fz444+8/fbbAGzfvp25c+fSoUMHWrRoYd02Li6O69evA5CUlERcXBxpaWkANGnShK1btzJjxgxOnTrFxo0bGTZsGM2bN6dcuXJ5/HCIiIjI4y5Hc26ioqJISUnh559/5ueff7ZZ98orr/Dhhx8ya9YsJk2axBdffEGpUqWYNGkSwcHBAKxevRq4feXTokWLbLbv3bs3ffr0Ye3atYSHh7N+/XpKlSrF888/z9SpU4mMjOSzzz7D3d2dFi1a0L9//39QtoiIiBiVg8VisTzsTjxoaWlmLl++nuf7dXIy4enpRnz8dcNP6rKnWsG+6lWtxmVP9apW4/Hycsv21VK644+IiIgYisKNiIiIGIrCjYiIiBiKwo2IiIgYisKNiIiIGIrCjYiIiBiKwo2IiIgYisKNiIiIGIrCjYiIiBiKwo2IiIgYisKNiIiIGIrCjYiIiBiKwo2IiIgYisKNiIiIGIrCjYiIiBiKwo2IiIgYisKNiIiIGIrCjYiIiBiKwo2IiIgYisKNiIiIGIrCjYiIiBiKwo2IiIgYisKNiIiIGIrCjYiIiBiKwo2IiIgYisKNiIiIGIrCjYiIiBiKwo2IiIgYisKNiIiIGIrCjYiIiBhKjsPNlStXGDlyJPXq1SMgIIA33niD6Oho6/qtW7fSunVrqlevzosvvsiaNWtstr916xZjxowhODgYf39/Bg0axOXLl+96zJiYGLp160ZAQAAhISFMnTqVtLS0nHZdRERE7ECOw83AgQPZs2cPERERfPfddzzzzDOEhYVx/Phxjh07Rrdu3XjuuedYvnw5//rXvxg8eDBbt261bj969Gh+/fVXpk+fzhdffMHx48fp27dvlsdLSUkhLCwMgK+++orRo0fz5ZdfMnPmzFyUKyIiIkbnYLFYLNltfPLkSRo3bszSpUupWbMmABaLhcaNG9O8eXMuXbrEgQMH+Oabb6zbDBo0iCtXrjBv3jxiY2Np0KABc+bMoX79+gCcOHGCF198ka+++gp/f/8Mx1y9ejXh4eH8+uuvFCpUCIBly5YxceJEtm7diouLS46LTkszc/ny9Rxvdy8mE7i6OHD16k1SU815vv9HiZOTiUKFXO2iVrCvelWrcdlTvar14XJxzY/JlLczX7y83HB0zN4+nXKyY09PTyIjI6lWrZp1mYODAw4ODiQkJBAdHU2jRo1stqlTpw7jxo3DYrGwa9cu67J0ZcqUwdvbm507d2YabqKjo6lSpYo12KRvn5iYyIEDB6hevXpOSrBycsrbB91sNnMwMpwS5nN5ut9HVQpw82F34gGyp3pVq3HZU72q9eE6aSrBM10n5HnAya4chRsPDw/rGZd0UVFRnDx5kmHDhrFixQp8fHxs1hcvXpybN28SHx9PbGwsnp6e5MuXL0Ob8+fPZ3rM8+fPZ7pPgHPnzuUq3JhMDnh6uuV4u7sxm83g4JCn+xQREXkcOTjcfp99LMLN3+3evZvw8HAaN25MgwYNSEpKyjBMlP59cnIyN2/ezHQYKV++fNy6dSvTYyQlJeHh4ZGhPZDlNvdiNltISLiRq23vpkr3D3FxtHAtMQlzWrZH+x5LJkcH3Avmt4tawb7qVa3GZU/1qtaHq5Jrfq5ezdvzSR4ervdnWOpO69at45133iEgIIDJkycDt0NHcnKyTbv0711dXcmfP3+G9XA7pLi6umZ6nMy2SQ81BQoUyG3378u4pJOTiQLuBbmV6vDIjHveL7drdbOLWsG+6lWtxmVP9arWh8ts/v8jGg9Jrs4XLV68mD59+tCwYUPmzJljPZNSokQJLly4YNP2woULFChQAHd3d3x8fLhy5UqGsHLhwgW8vb0zPZaPj0+m+wSy3EZERETsV47DzdKlSxk7dizt2rUjIiLCZpipVq1a7Nixw6b9tm3bCAgIwGQyUbNmTcxms3ViMdy+Wio2NpbAwMBMjxcYGMj+/ftJTEy02aebmxuVKlXKafdFRETE4HIUbk6cOMH48eN54YUX6NatGxcvXiQuLo64uDiuXbtGhw4d+OOPP5g8eTLHjh1j/vz5/Pjjj7z99tvA7TMtzZo1Y/jw4Wzfvp0//viDgQMHEhQURI0aNYDbw1hxcXHWszuNGjWiWLFi9O/fn4MHD7Ju3ToiIiLo3Llzri4DFxEREWPLUbiJiooiJSWFn3/+mZCQEJuvcePGUaFCBWbNmsXGjRtp1aoV33zzDZMmTSI4ONi6j7FjxxIcHEzv3r0JCwujbNmyTJs2zbp+z549hISEsGfPHuD2PJ65c+diNptp06YNY8aMoW3btvTs2TOPHgIRERExkhzdxM8o7tdN/JycTHh6uhEff/2RmdR1v9hTrWBf9apW47KnelWr8eTkJn764EwRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETEUhRsRERExFIUbERERMRSFGxERETGUfxRuPv30Uzp06GCzbPPmzbz66qv4+/vTokULVq9ebV23fPlyfH19M/168803szzOqlWrMt0mJibmn3RfREREDMgptxsuWbKEqVOnUqtWLeuyXbt20aVLF9q1a8fEiRM5ePAgI0aMIDU1lVatWtG0aVOee+45m/38+OOPTJgwge7du2d5rEOHDhEUFERERITNci8vr9x2X0RERAwqx+EmNjaWUaNGsX37dkqXLm2zbt68efj5+TFixAgAypUrx6lTp5g2bRqtWrUif/785M+f39r+/PnzfPLJJ/Ts2ZNnn302y2MePnwYX19fihUrltPuioiIiJ3Jcbj5888/cXZ2ZtWqVcycOZMzZ85Y1508eZJ69erZtK9cuTJnzpzh7NmzlCxZ0mbdpEmTKF68OF27dr3rMQ8dOkRoaGhOu3pXTk55P93I0dFk86+R2VOtYF/1qlbjsqd6Vat9y3G4CQ0NzTJoFC9enHPnztksS58Xc+nSJZtwc+jQIVavXs3MmTNxcXHJ8nhXr14lNjaW6Oholi5dSnx8PH5+frz77ruUKVMmp90HwGRywNPTLVfbZoeHh+t92/ejxp5qBfuqV7Ualz3Vq1rtU67n3GSmZcuWDBs2jFWrVtG0aVOOHDnC/PnzAUhJSbFpu2DBAnx9fXn++efvus8jR44AYLFYmDBhAklJScyePZu2bdvy/fffU7Ro0Rz302y2kJBwI8fb3YujowkPD1cSEm6SlmbO8/0/SuypVrCvelWrcdlTvarVeDw8XLN9dipPw02rVq04c+YMI0aMYMiQIZQoUYIuXbowevRo3N3dre2SkpL48ccfeffdd3FwcLjrPmvVqsXWrVvx9PS0tp0xYwYNGjRg+fLl9xzSykpq6v17AaSlme/r/h8l9lQr2Fe9qtW47Kle1Wqf8jTcAPTq1Yvu3btz8eJFihUrxubNm3F0dLQZktqyZQspKSm89NJL2drn36+KcnV1pVSpUsTGxuZp30VEROTxl6ezjxYvXszYsWNxdHTE29sbk8lEVFQU/v7+uLn9b45LdHQ0lSpVwtPT8577XLZsGbVr1+bGjf8NIyUmJvLXX39Rvnz5vOy+iIiIGECehpty5crx1VdfsXLlSmJiYoiMjGTVqlX069fPpt3+/fupVKlSpvtIS0sjLi6OpKQkAOrVq4fZbGbw4MEcOXKEvXv30qdPH7y8vGjdunVedl9EREQMIE/DTXBwMGPGjGHWrFk0a9aMqKgoZs+eTVBQkE27uLg4ChcunOk+zp07R0hICGvXrgWgRIkSLFiwgBs3bvDGG2/QqVMn3N3dWbhwIfny5cvL7ouIiIgBOFgsFsvD7sSDlpZm5vLl63m+XycnE56ebsTHXzf8pC57qhXsq17Valz2VK9qNR4vL7dsXy2lO/6IiIiIoSjciIiIiKEo3IiIiIihKNyIiIiIoSjciIiIiKEo3IiIiIihKNyIiIiIoSjciIiIiKEo3IiIiIihKNyIiIiIoSjciIiIiKEo3IiIiIihKNyIiIiIoSjciIiIiKEo3IiIiIihKNyIiIiIoSjciIiIiKEo3IiIiIihKNyIiIiIoSjciIiIiKEo3IiIiIihKNyIiIiIoSjciIiIiKEo3IiIiIihKNyIiIiIoSjciIiIiKEo3IiIiIihKNyIiIiIoSjciIiIiKEo3IiIiIih/KNw8+mnn9KhQwebZZs3b+bVV1/F39+fFi1asHr1apv1u3btwtfXN8PX9u3bszxOTEwM3bp1IyAggJCQEKZOnUpaWto/6bqIiIgYlFNuN1yyZAlTp06lVq1a1mW7du2iS5cutGvXjokTJ3Lw4EFGjBhBamoqrVq1AuDQoUM89dRTLF261GZ/hQoVyvQ4KSkphIWFUbp0ab766itOnTrFe++9h8lkom/fvrntvoiIiBhUjsNNbGwso0aNYvv27ZQuXdpm3bx58/Dz82PEiBEAlCtXjlOnTjFt2jRruDl8+DDly5enWLFi2TpeVFQUZ8+e5euvv6ZQoUJUrFiRS5cuMXHiRLp3746Li0tOSxAREREDy3G4+fPPP3F2dmbVqlXMnDmTM2fOWNedPHmSevXq2bSvXLkyZ86c4ezZs5QsWZJDhw5Rs2bNbB8vOjqaKlWq2JzZqVOnDomJiRw4cIDq1avntAQAnJzyfrqRo6PJ5l8js6dawb7qVa3GZU/1qlb7luNwExoaSmhoaKbrihcvzrlz52yWxcTEAHDp0iVKlizJkSNH8PT0pHXr1sTGxlKxYkUGDBiAn59fpvs8f/48Pj4+GY4DcO7cuVyFG5PJAU9Ptxxvl10eHq73bd+PGnuqFeyrXtVqXPZUr2q1T7mec5OZli1bMmzYMFatWkXTpk05cuQI8+fPB27PnTl37hzXrl3jxo0bDB8+HEdHRxYvXkz79u1Zvnw55cuXz7DPpKQkPDw8bJbly5cPgFu3buWqn2azhYSEG7na9m4cHU14eLiSkHCTtDRznu//UWJPtYJ91atajcue6lWtxuPh4Zrts1N5Gm5atWrFmTNnGDFiBEOGDKFEiRJ06dKF0aNH4+7uTokSJdi5cyeurq44OzsDUK1aNfbv38+iRYsYM2ZMhn3mz5+f5ORkm2XpoaZAgQK57mtq6v17AaSlme/r/h8l9lQr2Fe9qtW47Kle1Wqf8nyArlevXuzevZsNGzawbt06SpYsiaOjIyVLlgTAw8PDGmwATCYT5cqVIzY2NtP9+fj4cOHCBZtl6d97e3vndfdFRETkMZen4Wbx4sWMHTsWR0dHvL29MZlMREVF4e/vj5ubG5s2bcLf35/Tp09bt0lNTeXgwYOZDkkBBAYGsn//fhITE63Ltm3bhpubG5UqVcrL7ouIiIgB5Gm4KVeuHF999RUrV64kJiaGyMhIVq1aRb9+/QAICAjA09OTIUOGsG/fPg4dOsSQIUO4cuUKnTp1AiA5OZm4uDjrUFSjRo0oVqwY/fv35+DBg6xbt46IiAg6d+6sy8BFREQkgzwNN8HBwYwZM4ZZs2bRrFkzoqKimD17NkFBQQAULFiQBQsWULRoUcLCwnj99de5cuUKixcvpmjRogDs2bOHkJAQ9uzZA9yePDx37lzMZjNt2rRhzJgxtG3blp49e+Zl10VERMQgHCwWi+Vhd+JBS0szc/ny9Tzfr5OTCU9PN+Ljrxt+Upc91Qr2Va9qNS57qle1Go+Xl1u2r5bSHX9ERETEUBRuRERExFAUbkRERMRQFG5ERETEUBRuRERExFAUbkRERMRQFG5ERETEUBRuRERExFAUbkRERMRQFG5ERETEUBRuRERExFAUbkRERMRQFG5ERETEUBRuRERExFAUbkRERMRQFG5ERETEUBRuRERExFAUbkRERMRQFG5ERETEUBRuRERExFAUbkRERMRQFG5ERETEUBRuRERExFAUbkRERMRQFG5ERETEUBRuRERExFAUbkRERMRQFG5ERETEUBRuRERExFAUbkRERMRQ/lG4+fTTT+nQoYPNss2bN/Pqq6/i7+9PixYtWL16tc36c+fOMXDgQOrWrUtgYCBhYWEcOXLkrscZPnw4vr6+Nl+hoaH/pOsiIiJiUE653XDJkiVMnTqVWrVqWZft2rWLLl260K5dOyZOnMjBgwcZMWIEqamptGrViuTkZLp27UrhwoWZM2cO+fPnZ/r06XTs2JHVq1fj5eWV6bEOHTpE9+7dad++vXWZo6NjbrsuIiIiBpbjcBMbG8uoUaPYvn07pUuXtlk3b948/Pz8GDFiBADlypXj1KlTTJs2jVatWhEdHc3hw4fZtGkT3t7eAEyaNInatWvzn//8h9deey3D8SwWC0ePHqVr164UK1YsFyWKiIiIPclxuPnzzz9xdnZm1apVzJw5kzNnzljXnTx5knr16tm0r1y5MmfOnOHs2bNUqFCByMhIa7ABMJluj4wlJCRkerxTp05x48YNypYtm9Ou3pWTU95PN3J0NNn8a2T2VCvYV72q1bjsqV7Vat9yHG5CQ0OznO9SvHhxzp07Z7MsJiYGgEuXLlGtWjXq169vs37RokUkJSVRt27dTPd5+PBha7tNmzZhMpmoV68eAwYMwN3dPafdB8BkcsDT0y1X22aHh4frfdv3o8aeagX7qle1Gpc91ata7VOu59xkpmXLlgwbNoxVq1bRtGlTjhw5wvz58wFISUnJ0P7nn39mypQpdOrUCV9f30z3efjwYUwmE8WLF2fOnDmcOnWKiRMncuTIEb744gvrmZ+cMJstJCTcyPF29+LoaMLDw5WEhJukpZnzfP+PEnuqFeyrXtVqXPZUr2o1Hg8P12yfncrTcNOqVSvOnDnDiBEjGDJkCCVKlKBLly6MHj06w1mWL7/8krFjx/Lyyy8zePDgLPfZo0cP2rZti6enJwAVK1akWLFitGnThr1791K9evVc9TU19f69ANLSzPd1/48Se6oV7Kte1Wpc9lSvarVPeT5A16tXL3bv3s2GDRtYt24dJUuWxNHRkZIlS1rbTJo0idGjR/Pmm28yYcKEu559MZlM1mCTrkKFCgCcP38+r7svIiIij7k8DTeLFy9m7NixODo64u3tjclkIioqCn9/f9zcbs9xmTRpEnPnzmXIkCEMHToUBweHu+5z8ODBdOrUyWbZ3r17AShfvnxedl9EREQMIE/DTbly5fjqq69YuXIlMTExREZGsmrVKvr16wfA9u3bmTt3Lh06dKBFixbExcVZv65fvw5AUlIScXFxpKWlAdCkSRO2bt3KjBkzOHXqFBs3bmTYsGE0b96ccuXK5WX3RURExADydM5NcHAwY8aMYdasWcTGxlK+fHlmz55NUFAQgPVuxYsWLWLRokU22/bu3Zs+ffqwdu1awsPDWb9+PaVKleL5559n6tSpREZG8tlnn+Hu7k6LFi3o379/XnZdREREDMLBYrFYHnYnHrS0NDOXL1/P8/06OZnw9HQjPv664Sd12VOtYF/1qlbjsqd6VavxeHm5ZftqKd3xR0RERAxF4UZEREQMReFGREREDEXhRkRERAxF4UZEREQMReFGREREDEXhRkRERAxF4UZEREQMReFGREREDEXhRkRERAxF4UZEREQMReFGREREDEXhRkRERAxF4UZEREQMReFGREREDEXhRkRERAxF4UZEREQMReFGREREDEXhRkRERAxF4UZEREQMReFGREREDEXhRkRERAxF4UZEREQMReFGREREDEXhRkRERAxF4UZEREQMReFGREREDEXhRkRERAxF4UZEREQMReFGREREDOUfhZtPP/2UDh062CzbvHkzr776Kv7+/rRo0YLVq1fbrL916xZjxowhODgYf39/Bg0axOXLl+96nJiYGLp160ZAQAAhISFMnTqVtLS0f9J1ERERMahch5slS5YwdepUm2W7du2iS5cu1KhRg2+//Zbu3bszcuRIVq5caW0zevRofv31V6ZPn84XX3zB8ePH6du3b5bHSUlJISwsDICvvvqK0aNH8+WXXzJz5szcdl1EREQMzCmnG8TGxjJq1Ci2b99O6dKlbdbNmzcPPz8/RowYAUC5cuU4deoU06ZNo1WrVsTGxrJy5UrmzJlDrVq1AIiIiODFF19kz549+Pv7ZzheVFQUZ8+e5euvv6ZQoUJUrFiRS5cuMXHiRLp3746Li0suyhYRERGjynG4+fPPP3F2dmbVqlXMnDmTM2fOWNedPHmSevXq2bSvXLkyZ86c4ezZs/z2228A1KlTx7q+TJkyeHt7s3PnzkzDTXR0NFWqVKFQoULWZXXq1CExMZEDBw5QvXr1nJYAgJNT3k83cnQ02fxrZPZUK9hXvarVuOypXtVq33IcbkJDQwkNDc10XfHixTl37pzNspiYGAAuXbpEbGwsnp6e5MuXL8N258+fz3Sf58+fx8fHJ0N7gHPnzuUq3JhMDnh6uuV4u+zy8HC9b/t+1NhTrWBf9apW47KnelWrfcpxuLmbli1bMmzYMFatWkXTpk05cuQI8+fPB27Pnbl582amw0j58uXj1q1bme4zKSkJDw+PDO2BLLe5F7PZQkLCjVxtezeOjiY8PFxJSLhJWpo5z/f/KLGnWsG+6lWtxmVP9apW4/HwcM322ak8DTetWrXizJkzjBgxgiFDhlCiRAm6dOnC6NGjcXd3J3/+/CQnJ2fY7tatW7i6Zp44M9smPdQUKFAg131NTb1/L4C0NPN93f+jxJ5qBfuqV7Ualz3Vq1rtU54P0PXq1Yvdu3ezYcMG1q1bR8mSJXF0dKRkyZL4+Phw5cqVDGHlwoULeHt7Z7o/Hx8fLly4kKE9kOU2IiIiYr/yNNwsXryYsWPH4ujoiLe3NyaTiaioKPz9/XFzc6NmzZqYzWZ27dpl3ebEiRPExsYSGBiY6T4DAwPZv38/iYmJ1mXbtm3Dzc2NSpUq5WX3RURExADyNNyUK1eOr776ipUrVxITE0NkZCSrVq2iX79+wO0zLc2aNWP48OFs376dP/74g4EDBxIUFESNGjUASE5OJi4uznp2p1GjRhQrVoz+/ftz8OBB1q1bR0REBJ07d9Zl4CIiIpJBnoab4OBgxowZw6xZs2jWrBlRUVHMnj2boKAga5uxY8cSHBxM7969CQsLo2zZskybNs26fs+ePYSEhLBnzx7g9uThuXPnYjabadOmDWPGjKFt27b07NkzL7suIiIiBuFgsVgsD7sTD1pampnLl6/n+X6dnEx4eroRH3/d8JO67KlWsK96Vatx2VO9qtV4vLzcsn21lO74IyIiIoaicCMiIiKGonAjIiIihqJwIyIiIoaicCMiIiKGonAjIiIihqJwIyIiIoaicCMiIiKGYpc38bNYLJjN96dsR0eToT9y/k72VCvYV72q1bjsqV7VaiwmkwMODg7ZamuX4UZERESMS8NSIiIiYigKNyIiImIoCjciIiJiKAo3IiIiYigKNyIiImIoCjciIiJiKAo3IiIiYigKNyIiImIoCjciIiJiKAo3IiIiYigKNyIiImIoCjciIiJiKAo3IiIiYigKNzlgNpuZNm0azz33HDVq1KBLly6cPn06y/bx8fEMGjSIwMBAgoKCGDNmDDdv3nyAPc69K1euMHLkSOrVq0dAQABvvPEG0dHRWbafPXs2vr6+Gb4eF7GxsZn2f/ny5Zm2f1yf2+3bt2dap6+vL88//3ym2+zatSvT9tu3b3/Avc+ZTz/9lA4dOtgsO3DgAO3bt6dGjRqEhoaycOHCe+7nhx9+oGnTpvj5+dGqVSu2bt16v7r8j2RW73/+8x9effVV/P39CQ0N5aOPPiIpKSnLfaSlpeHn55fhuZ4+ffr97n6OZFbr8OHDM/Q7NDT0rvt5HJ7bv9faoUOHLH+GV65cmeV+3nrrrQzt//4YGopFsm369OmW2rVrW3755RfLgQMHLJ07d7Y0btzYcuvWrUzbt2/f3vLqq69a9u3bZ/nvf/9radiwoWXw4MEPuNe589Zbb1maN29u2blzp+X48eOWMWPGWPz8/CzHjh3LtH2/fv0s7777ruXChQs2X4+LDRs2WKpVq2aJjY216f/Nmzczbf+4Pre3bt3K8Bz99NNPFl9fX8u3336b6TZLliyxNGrUKMN2Wb3uHwWLFy+2VKpUydK+fXvrssuXL1tq165tCQ8Ptxw9etTy7bffWqpVq5Zl3RaLxbJ161ZLlSpVLF988YXl6NGjlg8//NBStWpVy9GjRx9EGdmWWb07d+60PPPMM5bZs2dbTpw4YdmwYYOlXr16lqFDh2a5n6NHj1oqVqxoOXDggM1znZiY+CDKyJbMarVYLJbXXnvNEhERYdPvS5cuZbmfx+G5zazW+Ph4mxpjY2Mtbdu2tTRr1uyuz1NwcLBl6dKlNtvGx8c/gCoeDoWbbLp165bF39/fsmTJEuuyq1evWvz8/Czff/99hva7d++2VKxY0eYHZfPmzRZfX1/L+fPnH0ifc+uvv/6yVKxY0RIdHW1dZjabLY0aNbJMnTo1021eeukly+eff/6Aepj3IiMjLS1atMhW28f5uf2769evWxo2bHjXN7xRo0ZZunfv/gB7lXvnz5+3dOvWzVKjRg3Liy++aPOmMGfOHEtISIglJSXFumzKlCmWxo0bZ7m/zp07W/r162ez7PXXX7eMGDEiz/ueG3erd9CgQZZOnTrZtF+xYoWlSpUqWQbTNWvWWAICAu5rn3PrbrWazWZLjRo1LD/99FO29/coP7d3q/XvFi1aZKlatWqWf3haLBbLxYsXLRUrVrT8+eef96O7jyQNS2XTwYMHuX79OsHBwdZlHh4eVK5cmZ07d2ZoHx0dTbFixShXrpx1WVBQEA4ODuzateuB9Dm3PD09iYyMpFq1atZlDg4OODg4kJCQkKF9cnIyf/31F2XLln2Q3cxThw4dsnmu7uZxfm7/bs6cOdy8eZMhQ4Zk2SYnj83D9ueff+Ls7MyqVauoXr26zbro6GiCgoJwcnKyLqtTpw5//fUXFy9ezLAvs9nM7t27bX7mAWrXrp3pz/zDcLd6O3funOF5NZlMpKSkkJiYmOn+HuXn+m61njp1ihs3bmT7d9Cj/tzerdY7Xb58malTp9KjR4+71n7o0CEcHBwoU6bM/ejuI8np3k0E4Pz58wCUKFHCZnnx4sWt6+4UGxuboa2LiwuFCxfm3Llz96+jecDDw4P69evbLIuKiuLkyZMMGzYsQ/ujR4+SlpZGVFQU48aN49atWwQGBvLuu+9SvHjxB9Xtf+Tw4cN4enrSrl07Tpw4wdNPP02PHj2oV69ehraP83N7p8uXL7NgwQIGDRpE4cKFs2x35MgRPD09ad26NbGxsVSsWJEBAwbg5+f34DqbTaGhoVnOszh//jwVK1a0WZb++jx37hxFixa1WZeQkMCNGzfw8fHJsE1mP/MPw93qrVy5ss33KSkpLFiwgKpVq+Ll5ZXpNocPHyY1NZWwsDAOHjyIt7c3HTt2pGXLlnne95y6W62HDx8GYNGiRWzatAmTyUS9evUYMGAA7u7uGdo/6s/t3Wq902effUb+/PkJCwu7a7vDhw/j7u7O+++/z5YtWyhQoAAvvvgiPXv2xMXFJa+6/UjRmZtsSp8s+vcXQr58+bh161am7TN70WTV/lG2e/duwsPDady4MQ0aNMiwPv0Xi6urK5988gnjxo3j+PHjvPnmm3edvPioSE1N5fjx41y9epU+ffoQGRlJjRo16Nq1a6YTDI3y3C5duhR3d3def/31LNucO3eOa9eucePGDYYPH86sWbMoWrQo7du35+jRow+wt/9cUlJSpj+/QKbPW/prN7s/84+y1NRUBg8ezJEjRxg1alSW7Y4cOcKVK1fo0KED8+bNo0mTJoSHh/Ptt98+wN7m3OHDhzGZTBQvXpw5c+YwdOhQfv31V3r27InZbM7Q3gjPbWJiIl9//TVhYWHW13FWDh8+zK1bt/Dz82Pu3Ln06NGDb775huHDhz+g3j54OnOTTfnz5wduD8Gk/x9u/1J0dXXNtH1ycnKG5bdu3aJAgQL3r6N5bN26dbzzzjsEBAQwefLkTNu0atWKevXq2fw1WKFCBerVq8d//vMfmjZt+qC6mytOTk5s374dR0dH63NbtWpVjhw5wrx58zKcujbKc7ty5UpatWpl83r+uxIlSrBz505cXV1xdnYGoFq1auzfv59FixYxZsyYB9Xdfyyz5y39jSyz5y39DSOzbTL7mX9UJSYm0r9/f3bs2MGMGTPuesZt9erVpKWl4ebmBkClSpU4e/Ys8+bN47XXXntQXc6xHj160LZtWzw9PQGoWLEixYoVo02bNuzduzfD0I4Rntt169aRnJzMq6++es+277//PkOGDKFQoULA7cfH2dmZAQMGMHjw4AxnLY1AZ26yKX0Y4sKFCzbLL1y4gLe3d4b2Pj4+GdomJydz5cqVx2aoZvHixfTp04eGDRsyZ86cu/518PfT3MWLF6dw4cKPxCne7HBzc8vwJl+hQgViY2MztDXCc3vw4EFOnz5NixYt7tnWw8PDGmzg9ryNcuXKZfrYPMoye97Sv8/sZ7hw4cIUKFAg2z/zj6ILFy7Qrl07fvvtN+bNm5dhuPnv8ufPbw026SpWrPjI/xybTCZrsElXoUIFgEz7boTndt26ddSvXx8PD497tnVycrIGm3R3e3yMQOEmmypVqkTBggVt7u2RkJDA/v37CQwMzNA+MDCQ8+fPc/LkSeuyHTt2AFCzZs373+F/aOnSpYwdO5Z27doRERFx13HZjz/+mCZNmmCxWKzLYmJiiI+Pp3z58g+iu//IkSNHCAgIyHDfln379mXa/8f9uYXbk2uLFClCpUqV7tpu06ZN+Pv729zPKTU1lYMHDz4Wz+2dAgMD2bVrF2lpadZl27Zto0yZMhQpUiRDewcHBwICAqzPbbrt27dTq1at+97ff+rq1at07NiRy5cvs2TJkkx/T90pISGBoKCgDPd22rt3r/WN8FE1ePBgOnXqZLNs7969AJm+Th/35xZu/wz//axyVjp06EB4eLjNsr179+Ls7Ezp0qXvQ+8ePoWbbHJxcaF9+/ZMnjyZ9evXc/DgQQYMGICPjw+NGzcmLS2NuLg461hu9erVCQgIYMCAAfzxxx9s27aNkSNH0qpVq0f+L4MTJ04wfvx4XnjhBbp168bFixeJi4sjLi6Oa9eukZycTFxcnPWU7gsvvMCZM2cYPXo0J06cYOfOnfTp04eAgACee+65h1zNvZUrV46yZcvy/vvvEx0dzbFjx5gwYQK//fYbPXr0MNRzm27//v1Z3mQxLi6O69evAxAQEICnpydDhgxh3759HDp0iCFDhnDlypUMbyaPuldffZXExETee+89jh49yvLly1mwYAHdunWztrl27RqXL1+2fv/WW2+xZs0aPv/8c44dO8bEiRM5cOAAHTt2fBgl5MiECRM4ffo0kyZNwsvLy/ozHBcXZw14V65c4cqVK8DtM3R16tTh448/ZuPGjfz1119ERkayatUq+vTp8xArubcmTZqwdetWZsyYwalTp9i4cSPDhg2jefPm1qu/jPTcnjt3jvj4+Cz/OLl+/TpxcXHW75s0acK///1vvvzyS06fPs3atWuZOHEiYWFhFCxY8EF1+8F62NeiP05SU1MtEydOtNSpU8dSo0YNS5cuXSynT5+2WCwWy+nTpy0VK1a0fPfdd9b2Fy9etPTp08dSo0YNS+3atS2jRo2yJCUlPazuZ9vs2bMtFStWzPRryJAhlm3btlkqVqxo2bZtm3Wb//73v5bXX3/dUqNGDUtQUJAlPDzccuXKlYdYRc7ExcVZhg4daqlbt66lWrVqltdff92yc+dOi8VirOc23dtvv23p379/pusqVqxomTZtmvX7kydPWvr06WMJCgqyVK9e3dK5c2fLoUOHHlRXc23IkCEZ7g/y+++/W9q0aWOpWrWqpWHDhpZFixZl2KZhw4Y2y1asWGF54YUXLNWqVbO88sorlv/+97/3ve+5cWe9qamplmrVqmX5c5z+e6t9+/Y2j9G1a9cs48ePt9SvX99StWpVS8uWLS0///zzQ6nnbjJ7bteuXWtp1aqVxc/Pz1K3bl3Lhx9+aPMz+bg+t1m9jv9+r607TZs2zVKxYkWbZYsXL7a89NJL1tf+7NmzLWlpafet3w+bg8Vyx1iCiIiIyGNOw1IiIiJiKAo3IiIiYigKNyIiImIoCjciIiJiKAo3IiIiYigKNyIiImIoCjciIiJiKAo3IiIiYigKNyLyWIqJicHX1zfDZyHlxtChQwkNDc2DXonIo8DpYXdARCQ3ihcvzrJly3jqqacedldE5BGjcCMijyUXFxdq1KjxsLshIo8gDUuJyH3xzTff0KxZM6pWrUqDBg2YPn269dOohw4dSocOHfj2229p2LAh/v7+dOzYkYMHD1q3N5vNfPzxx4SGhlK1alVCQ0OZMmUKKSkpQObDUn/99Rd9+/albt261KhRgw4dOrBr1y6bfl29epXw8HCCgoIIDAxk0qRJmM3mDP1ft24drVu3plq1atStW5cPPviAGzduWNcnJSUxevRo6tWrR9WqVXnxxReZN29enj6GIpI7OnMjInnu008/5eOPP6Z9+/aEh4dz4MABpk+fzrlz5xg/fjwABw4c4Pjx4wwcOJBChQoxbdo02rdvz9q1aylevDifffYZX375JUOGDOHJJ5/k999/5+OPP8bZ2Zm+fftmOObRo0dp06YNpUuXZvjw4Tg7O7Nw4UI6duzI/PnzCQoKwmw28/bbb3PmzBmGDBlC4cKFmTt3Lnv37qV48eLWfX3//fe88847tGjRgv79+3PmzBk+/vhjjh49yueff46DgwPjx4/n119/ZciQIRQtWpRNmzYxceJEChcuzKuvvvrAHmsRyUjhRkTy1LVr15g1axavv/46w4cPByAkJITChQszfPhw3nrrLWu7OXPmUKtWLQD8/Pxo1KgRCxcu5J133mHHjh1UrVrVGhSCgoJwdXXF3d090+POmDEDFxcXFi5cSMGCBQFo0KABzZs3Z+LEiXz77bds2rSJP/74g88++4x69eoBEBwcbDOZ2GKxMHnyZJ577jkmT55sXV66dGk6derExo0badCgATt27KBu3bo0a9YMgNq1a1OgQAGKFCmSlw+niOSChqVEJE/t2bOHpKQkQkNDSU1NtX6lB4gtW7YAUKpUKWuwgdsThP39/dm5cydwOyxs2bKFtm3bMnfuXI4ePUr79u1p2bJlpsfdsWMHDRs2tAYbACcnJ5o1a8a+ffu4fv060dHRODs789xzz1nbFChQgPr161u/P378OOfPn8/Q/8DAQAoWLGjtf+3atfn666/p0qULixcv5vTp0/Tq1YsGDRrkzQMpIrmmMzcikqeuXLkCQNeuXTNdf+HCBQC8vb0zrCtSpAh//vknAG+//TZubm589913TJ48mUmTJlGhQgWGDx9OnTp1Mmx79epVihYtmmF50aJFsVgsJCYmcvXqVQoXLoyDg4NNm2LFimXo/5gxYxgzZkyW/X/vvffw8fFh1apVjB07lrFjx+Lv78/o0aOpVKlSprWLyIOhcCMiecrDwwOAyZMnU7p06QzrixYtyieffEJ8fHyGdRcvXrQO65hMJtq1a0e7du24dOkSGzduZM6cOfTp08d69uROhQoV4uLFixmWx8XFAeDp6Ymnpyfx8fGkpaXh6OhobZMeaO7s/+DBgwkKCsr0OHD7aq0ePXrQo0cPzp49yy+//MKsWbMYNGgQa9asyerhEZEHQMNSIpKnqlevjrOzM7GxsVSrVs365eTkREREBDExMcDtK5uOHTtm3S42NpY9e/YQHBwMwP/93//xwQcfALfP6LRu3Zp27dqRkJBAYmJihuMGBgbyyy+/2KxLS0tjzZo1VKtWDRcXF4KDg0lNTWXdunXWNsnJyTZhqWzZshQpUoSYmBib/nt7ezNlyhT2799PUlISTZo0Yf78+QCULFmSdu3a0axZM86ePZuHj6aI5IbO3IhInvL09OTtt9/mk08+ITExkdq1axMbG8snn3yCg4ODdcjGYrHQvXt3BgwYgKOjIzNmzKBQoUJ06NABuB1W5s+fT9GiRfH39yc2NpbPP/+coKAgvLy8bC7LBujduzebNm3izTffpGvXrjg7O1vnwsydOxe4PXk4JCSE4cOHc+nSJZ544gkWLlzI5cuXrWeMHB0dGTBgACNHjsTR0ZGGDRuSkJDArFmziI2NpUqVKuTPn58qVaowY8YMnJ2d8fX15cSJE6xYsYImTZo8wEdbRDLjYLFYLA+7EyJiPEuWLGHp0qWcPHmSQoUKERwczMCBAylZsiRDhw5lx44ddOnShZkzZ3Lz5k2effZZhgwZQqlSpQBITU1l9uzZrFq1ivPnz+Pu7k5oaCiDBg3C09OTmJgYnn/+eSZMmEDr1q2B25eXR0REEB0djYODA35+fvTu3dtm4vLNmzeZPHkya9as4datWzRt2pQCBQqwfv16/vOf/1jbrV27lrlz53LkyBEKFChAQEAA/fv3x9fXF4DExESmTp3K+vXriYuLo0iRIjRt2pR+/fqRP3/+B/hIi8jfKdyIyAOXHm7uDBMiInlFc25ERETEUBRuRERExFA0LCUiIiKGojM3IiIiYigKNyIiImIoCjciIiJiKAo3IiIiYigKNyIiImIoCjciIiJiKAo3IiIiYigKNyIiImIo/w+ztV4eBAvaNAAAAABJRU5ErkJggg==", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "# 获取参数\n", - "cfg = get_args() \n", - "# 训练\n", - "env, agent = env_agent_config(cfg)\n", - "res_dic = train(cfg, env, agent)\n", - " \n", - "plot_rewards(res_dic['rewards'], cfg, tag=\"train\") \n", - "# 测试\n", - "res_dic = test(cfg, env, agent)\n", - "plot_rewards(res_dic['rewards'], cfg, tag=\"test\") # 画出结果" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.12 ('easyrl')", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.12" - }, - "orig_nbformat": 4, - "vscode": { - "interpreter": { - "hash": "f5a9629e9f3b9957bf68a43815f911e93447d47b3d065b6a8a04975e44c504d9" - } - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/projects/notebooks/A2C.ipynb b/projects/notebooks/A2C.ipynb deleted file mode 100644 index 8966eac..0000000 --- a/projects/notebooks/A2C.ipynb +++ /dev/null @@ -1,370 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "import torch\n", - "import torch.optim as optim\n", - "import torch.nn as nn\n", - "import torch.nn.functional as F\n", - "from torch.distributions import Categorical\n", - "import numpy as np\n", - "from multiprocessing import Process, Pipe\n", - "import argparse\n", - "import gym" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 建立Actor和Critic网络" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [], - "source": [ - "class ActorCritic(nn.Module):\n", - " ''' A2C网络模型，包含一个Actor和Critic\n", - " '''\n", - " def __init__(self, input_dim, output_dim, hidden_dim):\n", - " super(ActorCritic, self).__init__()\n", - " self.critic = nn.Sequential(\n", - " nn.Linear(input_dim, hidden_dim),\n", - " nn.ReLU(),\n", - " nn.Linear(hidden_dim, 1)\n", - " )\n", - " \n", - " self.actor = nn.Sequential(\n", - " nn.Linear(input_dim, hidden_dim),\n", - " nn.ReLU(),\n", - " nn.Linear(hidden_dim, output_dim),\n", - " nn.Softmax(dim=1),\n", - " )\n", - " \n", - " def forward(self, x):\n", - " value = self.critic(x)\n", - " probs = self.actor(x)\n", - " dist = Categorical(probs)\n", - " return dist, value" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "class A2C:\n", - " ''' A2C算法\n", - " '''\n", - " def __init__(self,n_states,n_actions,cfg) -> None:\n", - " self.gamma = cfg.gamma\n", - " self.device = cfg.device\n", - " self.model = ActorCritic(n_states, n_actions, cfg.hidden_size).to(self.device)\n", - " self.optimizer = optim.Adam(self.model.parameters())\n", - "\n", - " def compute_returns(self,next_value, rewards, masks):\n", - " R = next_value\n", - " returns = []\n", - " for step in reversed(range(len(rewards))):\n", - " R = rewards[step] + self.gamma * R * masks[step]\n", - " returns.insert(0, R)\n", - " return returns" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "def make_envs(env_name):\n", - " def _thunk():\n", - " env = gym.make(env_name)\n", - " env.seed(2)\n", - " return env\n", - " return _thunk\n", - "def test_env(env,model,vis=False):\n", - " state = env.reset()\n", - " if vis: env.render()\n", - " done = False\n", - " total_reward = 0\n", - " while not done:\n", - " state = torch.FloatTensor(state).unsqueeze(0).to(cfg.device)\n", - " dist, _ = model(state)\n", - " next_state, reward, done, _ = env.step(dist.sample().cpu().numpy()[0])\n", - " state = next_state\n", - " if vis: env.render()\n", - " total_reward += reward\n", - " return total_reward\n", - "\n", - "def compute_returns(next_value, rewards, masks, gamma=0.99):\n", - " R = next_value\n", - " returns = []\n", - " for step in reversed(range(len(rewards))):\n", - " R = rewards[step] + gamma * R * masks[step]\n", - " returns.insert(0, R)\n", - " return returns\n", - "\n", - "\n", - "def train(cfg,envs):\n", - " print('Start training!')\n", - " print(f'Env:{cfg.env_name}, Algorithm:{cfg.algo_name}, Device:{cfg.device}')\n", - " env = gym.make(cfg.env_name) # a single env\n", - " env.seed(10)\n", - " n_states = envs.observation_space.shape[0]\n", - " n_actions = envs.action_space.n\n", - " model = ActorCritic(n_states, n_actions, cfg.hidden_dim).to(cfg.device)\n", - " optimizer = optim.Adam(model.parameters())\n", - " step_idx = 0\n", - " test_rewards = []\n", - " test_ma_rewards = []\n", - " state = envs.reset()\n", - " while step_idx < cfg.max_steps:\n", - " log_probs = []\n", - " values = []\n", - " rewards = []\n", - " masks = []\n", - " entropy = 0\n", - " # rollout trajectory\n", - " for _ in range(cfg.n_steps):\n", - " state = torch.FloatTensor(state).to(cfg.device)\n", - " dist, value = model(state)\n", - " action = dist.sample()\n", - " next_state, reward, done, _ = envs.step(action.cpu().numpy())\n", - " log_prob = dist.log_prob(action)\n", - " entropy += dist.entropy().mean()\n", - " log_probs.append(log_prob)\n", - " values.append(value)\n", - " rewards.append(torch.FloatTensor(reward).unsqueeze(1).to(cfg.device))\n", - " masks.append(torch.FloatTensor(1 - done).unsqueeze(1).to(cfg.device))\n", - " state = next_state\n", - " step_idx += 1\n", - " if step_idx % 200 == 0:\n", - " test_reward = np.mean([test_env(env,model) for _ in range(10)])\n", - " print(f\"step_idx:{step_idx}, test_reward:{test_reward}\")\n", - " test_rewards.append(test_reward)\n", - " if test_ma_rewards:\n", - " test_ma_rewards.append(0.9*test_ma_rewards[-1]+0.1*test_reward)\n", - " else:\n", - " test_ma_rewards.append(test_reward) \n", - " # plot(step_idx, test_rewards) \n", - " next_state = torch.FloatTensor(next_state).to(cfg.device)\n", - " _, next_value = model(next_state)\n", - " returns = compute_returns(next_value, rewards, masks)\n", - " log_probs = torch.cat(log_probs)\n", - " returns = torch.cat(returns).detach()\n", - " values = torch.cat(values)\n", - " advantage = returns - values\n", - " actor_loss = -(log_probs * advantage.detach()).mean()\n", - " critic_loss = advantage.pow(2).mean()\n", - " loss = actor_loss + 0.5 * critic_loss - 0.001 * entropy\n", - " optimizer.zero_grad()\n", - " loss.backward()\n", - " optimizer.step()\n", - " print('Finish training！')\n", - " return test_rewards, test_ma_rewards" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "import matplotlib.pyplot as plt\n", - "import seaborn as sns \n", - "def plot_rewards(rewards, ma_rewards, cfg, tag='train'):\n", - " sns.set()\n", - " plt.figure() # 创建一个图形实例，方便同时多画几个图\n", - " plt.title(\"learning curve on {} of {} for {}\".format(\n", - " cfg.device, cfg.algo_name, cfg.env_name))\n", - " plt.xlabel('epsiodes')\n", - " plt.plot(rewards, label='rewards')\n", - " plt.plot(ma_rewards, label='ma rewards')\n", - " plt.legend()\n", - " plt.show()" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Start training!\n", - "Env:CartPole-v0, Algorithm:A2C, Device:cuda\n", - "step_idx:200, test_reward:18.6\n", - "step_idx:400, test_reward:19.7\n", - "step_idx:600, test_reward:24.2\n", - "step_idx:800, test_reward:19.5\n", - "step_idx:1000, test_reward:33.9\n", - "step_idx:1200, test_reward:36.1\n", - "step_idx:1400, test_reward:32.6\n", - "step_idx:1600, test_reward:36.3\n", - "step_idx:1800, test_reward:38.9\n", - "step_idx:2000, test_reward:60.8\n", - "step_idx:2200, test_reward:41.9\n", - "step_idx:2400, test_reward:42.2\n", - "step_idx:2600, test_reward:71.6\n", - "step_idx:2800, test_reward:123.6\n", - "step_idx:3000, test_reward:57.5\n", - "step_idx:3200, test_reward:155.4\n", - "step_idx:3400, test_reward:111.4\n", - "step_idx:3600, test_reward:133.8\n", - "step_idx:3800, test_reward:133.8\n", - "step_idx:4000, test_reward:114.3\n", - "step_idx:4200, test_reward:165.5\n", - "step_idx:4400, test_reward:119.4\n", - "step_idx:4600, test_reward:173.4\n", - "step_idx:4800, test_reward:115.4\n", - "step_idx:5000, test_reward:159.7\n", - "step_idx:5200, test_reward:178.1\n", - "step_idx:5400, test_reward:137.8\n", - "step_idx:5600, test_reward:146.0\n", - "step_idx:5800, test_reward:187.4\n", - "step_idx:6000, test_reward:200.0\n", - "step_idx:6200, test_reward:169.2\n", - "step_idx:6400, test_reward:167.8\n", - "step_idx:6600, test_reward:184.3\n", - "step_idx:6800, test_reward:162.3\n", - "step_idx:7000, test_reward:125.4\n", - "step_idx:7200, test_reward:150.6\n", - "step_idx:7400, test_reward:152.6\n", - "step_idx:7600, test_reward:122.5\n", - "step_idx:7800, test_reward:136.3\n", - "step_idx:8000, test_reward:131.4\n", - "step_idx:8200, test_reward:174.6\n", - "step_idx:8400, test_reward:91.7\n", - "step_idx:8600, test_reward:170.1\n", - "step_idx:8800, test_reward:166.0\n", - "step_idx:9000, test_reward:150.2\n", - "step_idx:9200, test_reward:104.6\n", - "step_idx:9400, test_reward:147.2\n", - "step_idx:9600, test_reward:111.8\n", - "step_idx:9800, test_reward:118.7\n", - "step_idx:10000, test_reward:102.6\n", - "step_idx:10200, test_reward:99.0\n", - "step_idx:10400, test_reward:64.6\n", - "step_idx:10600, test_reward:133.7\n", - "step_idx:10800, test_reward:119.7\n", - "step_idx:11000, test_reward:112.6\n", - "step_idx:11200, test_reward:116.1\n", - "step_idx:11400, test_reward:116.3\n", - "step_idx:11600, test_reward:116.2\n", - "step_idx:11800, test_reward:115.3\n", - "step_idx:12000, test_reward:109.7\n", - "step_idx:12200, test_reward:110.3\n", - "step_idx:12400, test_reward:131.4\n", - "step_idx:12600, test_reward:128.3\n", - "step_idx:12800, test_reward:128.8\n", - "step_idx:13000, test_reward:119.8\n", - "step_idx:13200, test_reward:108.6\n", - "step_idx:13400, test_reward:128.4\n", - "step_idx:13600, test_reward:138.2\n", - "step_idx:13800, test_reward:119.1\n", - "step_idx:14000, test_reward:140.7\n", - "step_idx:14200, test_reward:145.3\n", - "step_idx:14400, test_reward:154.1\n", - "step_idx:14600, test_reward:165.2\n", - "step_idx:14800, test_reward:138.2\n", - "step_idx:15000, test_reward:143.5\n", - "step_idx:15200, test_reward:125.4\n", - "step_idx:15400, test_reward:137.1\n", - "step_idx:15600, test_reward:150.1\n", - "step_idx:15800, test_reward:132.9\n", - "step_idx:16000, test_reward:140.4\n", - "step_idx:16200, test_reward:141.3\n", - "step_idx:16400, test_reward:135.5\n", - "step_idx:16600, test_reward:135.5\n", - "step_idx:16800, test_reward:125.6\n", - "step_idx:17000, test_reward:126.8\n", - "step_idx:17200, test_reward:124.7\n", - "step_idx:17400, test_reward:129.6\n", - "step_idx:17600, test_reward:114.3\n", - "step_idx:17800, test_reward:57.3\n", - "step_idx:18000, test_reward:164.7\n", - "step_idx:18200, test_reward:165.8\n", - "step_idx:18400, test_reward:196.7\n", - "step_idx:18600, test_reward:198.8\n", - "step_idx:18800, test_reward:200.0\n", - "step_idx:19000, test_reward:199.6\n", - "step_idx:19200, test_reward:189.5\n", - "step_idx:19400, test_reward:177.9\n", - "step_idx:19600, test_reward:159.3\n", - "step_idx:19800, test_reward:127.7\n", - "step_idx:20000, test_reward:143.6\n", - "Finish training！\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAEXCAYAAABI/TQXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAB+YElEQVR4nO2dd3hUVfrHP3f6THpCElpAei8KSFMQlCYg2EUF26rrWlb3t67oYt3FxbK6rmV1XdfdVRQQBV3XDqJSBEWq9BJCSEjv0+ee3x+Tmcwkk2TSyeR8nsdHMnPn3nPu3Hnve7/nLYoQQiCRSCSSiETT1gOQSCQSScshjbxEIpFEMNLISyQSSQQjjbxEIpFEMNLISyQSSQQjjbxEIpFEMNLIN5GtW7cyZ86cVjnWCy+8wNq1a1vlWB2FPXv2MHXq1Gbb38aNG5kyZQqXX345dru9xvtvvfUWAwYMYOfOnUGvFxYWctdddzF37lwuvvhinnrqKVRV9b//9ddfs3DhQubPn8/s2bO59957yc7ODjmGDz74gAsuuIBbbrmlSXPJyclh8eLFzJ07l0suuYQrr7ySr776qsH7KSsrY9GiRf6/p06dyowZM5g3b55/Pk8++WTQfEPx4osv8sQTTzT4+LWxa9cuLrvsMmbNmsUNN9xAbm5us+37jEJImsT3338vZs+e3dbDkDSS3bt3iylTpjTb/hYvXixefvnlWt+/+OKLxf/93/+Je++9N+j1//u//xPPPfecEEIIu90urr32WvHee+8JIYT46KOPxKxZs0R6eroQQghVVcWrr74qLrroIuFwOGocY+HChWLt2rVNmkdBQYG44IILxJo1a4SqqkIIIfbv3y/GjRsnNm7c2KB9nTx5UowcOdL/95QpU8Tu3bv9fzscDnHVVVeJt956q879/PWvfxWPP/54g45dGw6HQ0yaNEn8+OOPQgghli9fLn7xi180y77PNHRtfZOJJJxOJ88++yw//PADHo+HwYMHs2TJEqKjo/n666957bXXcDqdFBYWMn/+fO699162bt3K0qVLsVgsWK1W7r//fl5++WXS0tI4fPgwTqeTRx55hHHjxrF48WL69evHLbfcwrBhw7jtttvYtGkTubm5LFq0iBtvvBGPx8PTTz/N+vXriYmJYfjw4Rw9epS33nqrxnhfe+011qxZg06no2fPnixbtowvv/ySzz//nNdeew3weoW+vxcvXkxxcTEnT55k4sSJrF69ms8//5zk5GQArrrqKu68807Gjx9f63kIxOVysWzZMrZs2YJWq2X48OE8+OCDREdHM3XqVC699FK2bNlCdnY2s2bN4ne/+12NORw/fpxHHnmEwsJCNBoNd9xxBxdffDFTp07lhRdeYNiwYQBBf7/zzjv8+9//Jjo6mv79+/v3lZ+fzyOPPEJBQQF5eXl069aNv/zlLyQlJYU17hUrVrBu3TqMRiNlZWU88MADQZ/bunUrJSUl3H///UybNo3s7Gy6dOkCwLRp0zjnnHMAMBqN9OvXj6ysLACef/55/vCHP9CzZ08AFEXhtttuo2vXrjidTgwGg/8YTz75JHv27CEzM5OioiIuv/xyHn/8cQ4cOICiKJx//vn85je/QafTMXToUC688EIOHDjAs88+6z9XAO+88w7nnHMO8+fP9782cOBAXnzxRWJiYgBYvXo1K1euxOVyUVJSwq233sq1117LBx98wOrVq7HZbP7v3G63M2/ePD744IMa36HBYGDUqFEcO3YMgK+++oqXXnoJj8dDdHQ0Dz74IMOHDw/6TE5ODk888QTZ2dm4XC5mz57NL3/5yxr7fu655ygvL+eRRx4B4Ntvv+XFF19k8eLFREdHM2rUKACuuOIKnnzySYqKikhISKixn3ZNW99l2juBnvyLL74oli1b5vd8/vznP4tHH31UqKoqrr/+enH8+HEhhBCnT58WgwYNEgUFBeL7778XAwcOFJmZmf79DRo0SOzbt08IIcQbb7whrrvuOiGEEA888ID4xz/+IYQQon///n7PZ8+ePWLo0KHCbreLd999V1x33XXCbrcLh8Mhbr75ZnH99dfXGPdXX30lpk+fLoqLi4UQQjz55JPilVdeEe+//7647bbb/NsF/v3AAw+IG264wf/e7373O/94jhw5Ii644ALh8XhqPQ/VeeGFF8Rdd90lnE6n8Hg8YvHixeLhhx8WQni9vWXLlvnP17Bhw0RGRkaNfcyfP1+8/fbbQgghsrKyxIUXXijKyspqeIu+v/ft2yfGjx8vcnNzhRBCPPzww35P/l//+pd47bXXhBBeb/kXv/iFeOONNxo07sDvqDq//vWv/XO69dZbxdNPPx1yu59//lmMGjVK7Nu3TxQWFor+/fsLq9UacttQXH/99eLTTz8VQni/oz/84Q9CVVX/9eCbY//+/cWaNWtC7uP222/3n9dQlJeXi6uuukoUFhYKIYTYsWOH31t///33xZgxY0RZWZkQon5P/vTp02LmzJnis88+E0eOHBETJkzwf9ebN28WEydOFGVlZUGe/MKFC8W6deuEEN4nn4ULF4r//e9/NcaZkZEhxo4d63/i+fWvfy1WrVolPv74Y3HzzTcHbXv++eeL/fv31zrn9or05JuRDRs2UFZWxubNmwGvx5eUlISiKLz66qts2LCBjz/+mKNHjyKEwGazAdClSxe6devm30/Xrl0ZNGgQAIMHD2bNmjUhj3fhhRcCMGTIEJxOJ1arlW+++YZ58+ZhNBoBuPrqq0N68Vu2bGHmzJnExcUB8OCDDwKE9LQC8Xk+AFdeeSWPP/44t9xyC++//z6XXXYZGo2m1vNQnW+//Zb77rsPvV4PwMKFC7nzzjtrzC81NZWkpCRKSkpIS0vzv19cXMyBAwe48sorAe95rE8z3rJlCxMnTvQ/fVx99dVs3LgRgBtuuIEff/yRN998k/T0dA4fPsyIESMaPO5Q5OXl8dVXX/H+++8DMH/+fB577DHuvPNOLBaLf7vvvvuO+++/nyVLljBo0CBKSkoA6tWra+Pbb7/l3XffRVEUDAYD11xzDf/+97+57bbbABg9enTIzymKgqij4klUVBSvvvoq33zzDenp6Rw4cACr1ep/f8CAATWe3AL57W9/i8lkQlVV9Ho9V155JTNmzGD58uWMGzfO/z2PHz+exMRE9u7d6/+s1Wrlhx9+oKSkhBdeeMH/2oEDB7j44ouDjpOWlsbAgQNZv34948ePZ8uWLSxdupT169eHHJdWq611zO0VaeSbEVVVeeihh5g8eTIAFRUVOBwOrFYrl156KRdddBGjR4/m8ssv56uvvvL/iAJ/5AAmk8n/77p+bD5DrigKAEIIdLrgr1SjCb22rtVq/Z8DKC0tpbS0tMbxXC5X0OcCxzp69Gjcbje7d+/m448/ZsWKFXWeh+pUN1yqqgYdzzc/3xyrnwffXAPncezYMbp27QoQtL3T6Qy5n8Af9TPPPMPu3bu5/PLLGTt2LG63O+S5r2/coXjvvfcAuOOOO/yfKS8vZ82aNVx33XUAvPnmm/z973/nueeeY8KECQDExcVx1llnsWvXLv9rPn79619zxx13MHDgwFqPG2qsbrfb/3f1a8/HyJEj2blzJ9dff33Q6ytWrMBmszFr1iyuvvpqrrrqKkaNGsXMmTP5+uuv692vj+rykI9Q51sIETRmVVURQrBixQrMZjPgXbg2Go28++67/utw6NChLF26lCuvvJK1a9dSUFDAtGnTiIqKokuXLuTl5fn36XK5KCoqIjU1tc5xt0dkdE0zct5557F8+XKcTieqqvLwww/z3HPPceLECcrLy7n33nuZOnUq27Zt82/T3EyePJmPPvoIp9OJ2+2u9SlgwoQJfPnll5SXlwPeyIV//etfJCYmcvjwYRwOB263O+iHG4orr7ySP/zhDwwYMMBvXGs7D9U5//zzWbFiBS6XC1VVWb58ORMnTgx7rtHR0QwZMsQfcZSdnc2CBQsoKysL8v527tzp/0FPmDCBTZs2cfr0aYCg87Nx40ZuuOEG5s+fT1JSEps3b8bj8TR53B6Ph1WrVvH444+zfv161q9fz4YNG7j99tv5z3/+gxCCN998k+XLl7Nq1aoaxvyuu+5i6dKlnDhxwr+/V155hQMHDtC7d+86z5HvuxBC4HQ6Q+4/FFdffTXbtm3jo48+8hvevXv38te//pX+/fuzd+9eEhMT+dWvfsX555/vv05CnS+dTofH46nzycDHuHHj2LRpEydPngTwr8kEPlFFR0czcuRI3nzzTcDroCxYsIB169axYMECPvzwQz788EOWLl0KeNc7fv75Z1atWsVVV10FwIgRIyguLuann34C4P3332fkyJHExsbWO8b2hvTkm5Ff/epXPPXUU1x66aV4PB4GDRrE4sWLsVgsXHDBBcyaNYvY2Fh69OhB3759OXHiRNCiWXNw2WWXcfz4cebPn4/FYqF79+5+byeQyZMnc+TIERYsWABA3759+cMf/oDJZGLMmDHMmjWL5ORkxo4dy8GDB2s93vz583nuueeCjHht56E6d9xxB0899RTz58/H7XYzfPhwHn744QbN989//jOPP/44b731FoqisHTpUpKTk/ntb3/LY489xsqVKxkyZAhDhgwBvDLC/fffzw033EBUVFTQgt6dd97J008/zSuvvIJWq+Wcc84hIyOjyeP++uuvUVWVuXPnBr1+44038p///Id169bxwgsvEBMTw1133eV/f+bMmdxxxx3MnTsXIQS/+c1vcLvdOBwOhgwZwr///e96r58lS5bwxz/+kblz5+JyuTj//PNDLlBWJz4+nrfeeotnnnmG1157DY1Gg9lsZunSpUycOBGbzcbq1auZOXMmZrOZ4cOHk5iY6L8RBZKcnMzgwYOZNWsW7777bp3H7du3L48++ih33XUXHo8Hk8nEq6++6l/s9fHss8/yhz/8gblz5+J0OpkzZw6XXHJJyH0aDAYuvvhiNm/e7P++9Xo9L730Ek888QQ2m434+Hieeuqpes9Le0QR4dxeJe2GjRs3UlBQwLx58wD44x//iNFo5P7772/jkUkkkrZAGvkIw5fAUlBQgMfjYeDAgTz22GM1PCGJRNIxkEZeIpFIIhi58CqRSCQRjDTyEolEEsFIIy+RSCQRjDTyEolEEsGccXHyRUUVqGrD14KTkqIpKChvgRGd2XTEeXfEOUPHnHdHnDM0bN4ajUJCQlSt759xRl5VRaOMvO+zHZGOOO+OOGfomPPuiHOG5pu3lGskEokkgpFGXiKRSCIYaeQlEokkggnLyL/00kvMnj2b2bNn8/TTTwOwefNm5s6dy/Tp03n++ef92+7fv5/LL7+cGTNm8Pvf/z6oRKhEIpFIWpd6jfzmzZvZuHEja9asYe3atfz88898/PHHPPTQQ7zyyit88skn7N27l2+++QaA+++/n4cffpjPP/8cIQSrVq1q8UlIJBKJJDT1Gvnk5GQWL16MwWBAr9fTp08f0tPT6dmzJ2lpaeh0OubOnctnn33GqVOnsNvtjBw5EvCWvf3ss89aeg4SiUQiqYV6Qyj79evn/3d6ejqffPIJCxcu9LdPA0hJSSEnJ4fc3Nyg15OTk8nJyWnmIUvagoycMl5YvZsli0aTEGOs/wMSSYTi9qj8eDCX9dtPcSKnjE5xJlITLPTrHsfMsT2COpWdCYQdJ3/48GFuv/12HnjgAXQ6HcePHw96v7Y2dQ2dcFJS7X0h6yM5uWOW022NeX+1I4uiMgcVLpX+Z8B5lt91x+FMmbMQgs+2pPPuFwcpKnPQpVMUM8b1pKDETnpWKTuP5DN1bE+6N9N4m2veYRn57du3c8899/DQQw8xe/Zstm3bRn5+vv/93NxcUlJSSE1NDXo9Ly+PlJSUBg2ooKC8UUkAyckx5OWVNfhz7Z3Wmvfuw7kAZGQV0z2xZqep1kR+1x2HM2XOVruLf316gB8P5jEgLZ4bZg5kaO9ENJVObF6xjQde3cK3P55k2pi0evZWPw2Zt0aj1Okc16vJZ2dnc+edd/Lss88ye/ZswNsf8fjx45w4cQKPx8PHH3/MpEmT6NatG0ajke3btwOwdu1aJk2aFNZAJWcuqhAcO1UKQGlF3Q2rJZJI42RuOY+9+QM/Hcrnyil9uP/asxneJ8lv4AGS482kJlrYe7ywDUcamno9+TfeeAOHw8GyZcv8r11zzTUsW7aMu+++G4fDweTJk5k5cybg7b24ZMkSKioqGDx4MIsWLWq50UtahewCK1aHNxS2pMLRxqORSFqXNd8ew+70sPj6c+jbLa7W7Yb2SuS7XVm43B70Om0rjrBu6jXyS5YsYcmSJSHf++ijj2q8NnDgQFavXt30kUnOGI6eKgFAp1UorXC28WgkktalqMxB766xdRp4gGG9E1m3PZNDJ0sY0iuxlUZXPzLjVVIvR0+VEGXS0SM1Rhp5SYej1Ook1mKod7sBaQnotAp7jhW0wqjC54yrQik58zhyqoQ+3eLQahTyim1tPRyJpNVQhaC0wklcdP1G3mjQ0j8tnp/PMF1eevKSOqmwu8gusNKnayxxUQbpyUs6FFa7G48qwvLkAYb2SuJUfgWFpfYWHln4SCMvqZNjWd6omr7d4oiNMlBmdeFR1TYelUTSOpRUOjWxUWEa+d5eLf5MirKRRl5SJ0cyS1AU6FXpyQug3CrDKCUdg9JybzRZXJhGvlunKBJijOw9g3R5aeQldXI0q4TuydGYDDq/N1MiJRtJB6HE2jBPXlEUhvRKZF960RnzxCuNvKRWVFVwLKvUHzoWF+WtWSN1eUlHwZf8F66RBxhyViJWh5uMnDOjN6008pJaycqvwO700KdbLACxUXpAevKSjkNJhQOtRiHKFH4g4oAe8QAczChumUE1EGnkJbVyIsdbO6NXF5+R93oz0pOXdBRKK5zERhkaVGgxPtpIaoKZQyeLW25gDUAaeUmtlFUusPpkGpNBh1GvlZ68pMNQUuEMe9E1kP5p8Rw6WYwaojJvayONvKRWrA4XGkXBbKyqwxEbpZeevKTD4PPkG0r/tHisDjen8ipaYFQNQxp5Sa1U2NxYTLqgR9W4KGOtnrzL7eHpd346Yx5TJZKm0lgjPyAtHuCM+C1IIy+plQq7q8aCU2wdWa+n8is4kFF8RiWCSCSNxVvSwNUouaZTvJmkWCMHM4paYGQNQxp5Sa1Y7W4sJn3Qa3FRhlo9ed+jaX6JrG8jaf9U2FyoQjTKk4cqXT5Ux7zWRBp5Sa3U5smX21y4PTUTPfxGvvjMqdshkTQWnzPTGE8evEa+1OridKG1OYfVYKSRl9RKhd1NlLmmJw9VkTeBZOZ7kz/ypCcviQBKm8HIAxxsY11eGnlJrVTYXFhCePIQOlY+K9/ryZeUO3G5PS0/QImkBSltYHGy6nROtBAbZWjzxVdp5DsoHlWt0xCrQmB1uImqpslX1a8JbgNotbspLHXQrVMUAPklUrKRtG+aKtcoikL/tHgOZrStLh+2kS8vL2fOnDlkZmbyzTffMG/ePP9/48aN4/bbbwfgpZdeYsqUKf73li9f3mKDlzSetd8d54//2V7r+3aHGyGoocnH1VKkzOfFj+jbCZBGXtL+Ka1wotMqmI2N7600IC2eojIHBW34ewhr9Lt27WLJkiWkp6cDMHnyZCZPngxAXl4eCxYs4MEHHwRg7969PPfcc5x99tktM2JJs3D0VEmdC0IVdm/j7to8+epyjU+PH9E3iU++P0G+7CAlaeeUNKKkQXV6psYAkJlfQad4c3MNrUGE5cmvWrWKRx99lJSUlBrvPf3001xzzTWcddZZgNfIv/7668ydO5cnnngCh8NR4zOStie7wIrLreJ0hZZsKuzehdXqnrxRr8VkqFnaICuvAqNBS5+ucei0CnnSk5e0c0obWdIgkNREr2E/XdB2ETZhGfmlS5cyevToGq+np6ezbds2Fi1aBEBFRQWDBg3igQceYM2aNZSWlvLKK68074glTcbmcPuNtM9jr47fk68WXQOhE6JO5VfQrVMUGo1CUqypxeQah9PDidOlLbJviSSQ0orwGnjXRYzFQJRJR05R2xn5JjXyXrlyJddeey0Gg/dEREVF8frrr/vfv/nmm3nooYe47777wt5nUlJ0o8eTnBzT6M+2Zxo670MBWXgGsyHk57WnvIa0e5e4Gu93ijdjc6pBr2cXWBkzOJXk5Bi6JkdTXOFske9jzYYjvP3ZAVYuvRidtuPFDXTEa7yt5lxmczHgrMQmHz8tNYaCMkeD99Nc826SkV+3bh1vvPGG/++srCw2b97MFVdcAYAQAp2uYYcoKChHVRu+Ep2cHENeXlmDP9feCTXvv3/0M53izVw2qXfIzxw4mu//d2ZWMVG6mprj6VzvPh02Z439mw1asvIr/K+XWp0UlztIijaQl1dGnEXP4ZPFLfJ9nMwuxenykJVd0qQFsfZIR7zG22rOqhCUlDsx6pQmHz8p1sjeY4UN2k9D5q3RKHU6x412hQoLC7Hb7aSlpflfM5lMPPPMM5w8eRIhBMuXL2fatGmNPYSkkRw5VUJ6du2SRnZhVWW8clttck1oTR5qyjVZlZmu3ZK9F1qneDPlNhc2R+h9NwWrwzsup/vMaK0miUzKfSUNmijXgDdevqTC2SK/h3BotJHPzMykc+fOQa8lJibyxBNPcMcddzBz5kyEENx0001NHqSkYdidHuy1LKiCdxHIoPd+9T5jXp0KuxudVoNBr63xXlyUgQq7G1eloT1VGT7ZtTJGvlOcCWiZMEpr5VqBq475SSRNpbS8aYlQgXRO9P4u2qq8QYOed9evX+//9/Dhw1m1alWNbWbMmMGMGTOaPjJJo7E7PdgddRj5Qiu9Osdy8GRxrUbeancRZQ59ecT6Sxs4SYw1cSqvnCiTjvho7+ud4rwRBfklNtJSGr/GEgrfgnB1T97p8vDz8UJG9OuEpgkhbxIJVDXwbmp0DUDngAgbX5e11qTjrVxFOG6Pituj4nCFfjRUVUFOkY2enWPQahQqapNrbDWzXX34Lnyfp55ZGVnjiyfuFF/pybdAoTJr5U3JVc3I7zlWwIsf7OHLH042+zElHY/m9ORTEiwoStt58tLIRxh2pyfo/9UpKLXjcqt0SbIQZdbXIdfUrEDpo3tyNDqthudW7mTFusOcyqvw6/EAMWY9Rr22RQqVWSt1zepG3vf6+98c41ReebMfV9KxqCppYGzyvvQ6DZ3iTNLIS5oHe6Wxc9Ri5H0XWudEC1EmHRW22jX52jz55Hgzf7jlXMYMTOGrHzOxOdx+PR68NTs6xZtaxJOvkmuC5+cz+lqtwusf7wtZClkiCZdSqxOdVhPU+rIpdE6MkkZe0jzYKo27062GDEX1Zd51SYoiyqSvNRnKWocnD5CaaOGWOYNZettYLj2/F+OGpAa9nxxnbvbmIW6P6r95VffknS7v39dP609GTjkfbUpv1mNLOhYl5U7iovRNKmkQSOdECzmF1jZp7C2NfDvm6KkSSsqDy0bYne6Af9f05k8XWrEYdcRY9F5Pvha5pjxEV6hQpCZYmDuxVw2vv1OcN+u1OavvBYag1TDylZ792MGpTBzamf9tSW/zZg2RTlsYrJZGCMHRUyWkny4lthmkGh+dkyw43SpFpa1f5kUa+XaKEIJnVuzgw2+PBr1uC4iqcYQIM8wuqKBzkgVFUbyafIiFV5/HXJcnXx+d4kzYnZ5anxQagzVgX6HkGq1GQafVMHvCWQgBhzOLm+3YkmBKrU7u+ct37DqSX//G7YQfDuTyyD+3sfSt7RSU2ms8nTaFzgmVETZtUN6gY6UMRhBOt4rTpVJSHlxDJtiTdwPB3sjpQiuDz0oEqJRranry1jrq1oSLr+JeXrGN6CbsJ5DAG0YouUav8/osKfFmDDoNmbkVSFqGAyeKsDrcnMgp85eXbu+s/e4YTpeHG2YO4NxBqc2aUd05qTJWvsDKkMrfX2shPfl2im+BtbqRDpRoqss1Noeb4nInXZIsAESZddidnhqLlL59Vu8K1RBaIiHKGjDX6nHyLrcHQ6WR12gUuiVHkSmjbFqMAye89Y+KyyKnyqzTpTKgRwKTR3Zr9pIZ8dEGjAZtm0iI0si3U3wLrNWjY+yO2jV5XyW8zomVRr5SR7dWk1RqqyXfEJIrPfnmrCtvrVOTV9HrqiIhuiVHczK3vE078kQy+yuNfFEEGXm3qrZY0TtFUeicYJFGXhI+vkXIGkY+wLBXD6PMLqhm5CszWqs/DVjrqFsTLmajDoNeQ6m1Zi/YxhIo11Svg+90efylGgDSkqMpt7lC9qKVNI2iMgc5Rd6bd1F5BBl5t4q+BSubdk6ytEldeanJt1N8Hru1Trkm2EM/XWBFUbwZeADRlZ569cXXumrJN4QYs55ya+joncbgm6uigMtT05M3BHjy3SvLKZzMKycuuvmiJCRVUk3PzjGR5cl7BFpt85bE8BRn4dr3NQjBRFcFMU4HDusIjJao+j/cTEgj306pkmvc1V4PkGtcNeWaTnEm/wKlz4iXV7tR+J4OmqLJ+/ZfVkuyVWOoKpqmweWqrsmr6AM8+e7J3h9RZm4FQ3slNdsYJLA/owiLUcfw3kl8vDkdt6flZI7WxO2pWrxvKkIIXD+vw7F1pfcFnYE0l5OeFhcVX9gxXPJ/KJrmSbSqD2nk2yk+uaY8hFwTa9FTanXVKFJWbnMF1eLwyTHVJR9/dE0TjXyMWV9rRm1jsNrdRJl0KBqlZoEytwdjQMXMGIuBuGiDXHxtAQ6cKGJAj3gSY40IvIlDSZUL7e0VVQg8qmiWm5VqL8O+/jU8mXvRpg3DNPkWNJZ4jmaVsH7lu1yT+z2OjW9hPP+GZku2qov2f/vtoPhkGbdHxRUQM253uP1JHNXj5K12NxZjlQTj8+Srx7KX212YDFq0mqZdHs3tyVvtLiwmHQa9NmQIZaBcA15dPjNXGvnmJL/ERn6JnYE9EkiI8V5nkaDLeyrlP10T5RrhcWP/4kU82QcwTlyIeeZv0FjiAYizGNji6M/pLpNxHdiAc9f/mjrssJBGvp0SmP0ZGB1jq0xiMug0NRZerXZ3kARjNupQCO3JN9WLB4gxG5rVk6+oHL9Rrwm6sUGlJq8Pvpy7p0STVVCBR5V1bJqLAyeKARjYM4H4yrWOSAijdLm9UVhN9eQdW97Bc/oQpsm3YBhyYZCn7nuKPpg4GV2fsTi3rcb29et4ik416Zj1IY18G+L2qPx8vLBRnw3U3q2O4AQok0GLyaCtsfBqdQQbeY2iYAlR2qDC5mpS+KSPKLOOCru72Yys1eEtmqbXaUPGyVfXU9OSo3F7BKcLm78aZkflQEYR0WY93ZKjiI8gT96t+jz5xptE5/4NuPatxzDiYvR9x9d436D3/i5LrW5MF/wC/bAZuI//gPW932P7/AWEo2WS96SRb0N2HSngzyt3ktOI2NnA8gVBRt7hwWTUYTRogxZehRCVck2whx6qSFlFtZtBY4mpbJ3WXKUNfHKNMUy5ppt/8VVKNs2BEIIDGUUM7BGPRlGIMevRaZWI8OTdlddTYxdePTlHcGx6C23aMAxjrqh1u9goA6VWJ4pWj2n8AqKu/TOGc+bhKc5CLc5u1LHrQxr5NsQnuTQmljww6clmD/bkzQYtJoMuaOHV7vSgClHDeHu97RCefDOUIvCVM2iuMErfTcqg19Qw8i53zciILklRaDWKXHxtJnKLbBSWOhjYMwHwJvjERxsjw5NvgiYvVBX7xv+gmOMwT/0lSh1rWdX7I2tMMRhHX0r01U+hTe3b8IGHgTTybYgv1rsxnq7N4fY/WlqrZbmaDF5PPnDh1XdDqS7DRJlqFilrLk3eb+Tr0OVzCq0UltZf+kD1PYmY9Bj02qACZUKIGslQ4PXKOidZpCffTOw47C1GNqx3VUhqfLQxIjx5l6fxmrz78CbUggyMY69CMdYd/x5nMVDajLkj4RD2jMrLy5kzZw6ZmZkAPPjgg0yfPp158+Yxb948vvzySwA2b97M3LlzmT59Os8//3zLjDpC8D0i2hpj5J0eEmO9mqjPyHtUFadbxWTUYtIHa/K+xdkack217lBCiMquUM3oyYcw8qoq+O+m4/z+9a385/OD9e7L7vAgoHJROViucXsEAmrINeDtYiU9+eZh+6FceqRG+0tWAMTHGCkqb/9ZxVXRNQ0z8sJlx7FtNZqUPuj6jK13+5hqnnxrEJa7tmvXLpYsWUJ6err/tb179/L222+TkpLif81ut/PQQw/x1ltv0aVLF26//Xa++eYbJk+e3OwDjwSqPPmG39ntDjeJMUZyi2z+m4QvrNJk0GEyaCkI8JB9NwJzdbmmWncop1vF7akp6zSG2ox8frGN1z/ex+HMEgx6DQVhFDGzBhRNqy7X+CJtDCH01O7JUWzdl1Op5zdPNcyOSFGZg6OnSrn0/F5BrydEG9lztAAhRKvEfLcUrkYaeefO/yFsJZin3x3W/GMtesptrlZNIAvrKKtWreLRRx/1G3Sr1UpWVhYPP/wwc+fO5a9//SuqqrJ792569uxJWloaOp2OuXPn8tlnn7XoBNozPk8+UG4JF5vTTXy0EY1G8X/ep8GbDFrvwmtACKW/smSIhVer3e1vAOEz+M2iyVtCG/lX1u7lZG45t84dzIShXfz9NOuiwv8k4pNrqoy87996fU1PPq2yvEFmniw73BR2HM4D4JwBKUGvJ8QYcbg8QYEA4bDrSD4/HcprtvE1Fd9vsSGavFpegHP3Z+j6jAtbT4+rDKMsa0XJJix3benSpUF/FxQUMG7cOJ544gksFgu33347q1evxmKxkJyc7N8uJSWFnJycBg0oKSm6/o1qITk5ptGfbQv0lQZXKJoGj93hUkmINxNl0iMUheTkGKyVsb6pydHklzlwugv8+9VV1htJ6xpPckA/1tTkaK8MEm0i2mKgonIfXVJimuV8GvRaPCj+fQkhyMqv4OKJvbjkgn68+/kBynecIj4hqs7IhqzKfrHdOseSkV+B26P69+mpXOhKSrDUGHP/ynuBO2AM7Zm2msOe44V0S45mxMDUII81rWscABqDrkFj++DNH1BVwYyJvevdtjXmnFkZZtspKTrs4+V+/x8UoOusG9HFhfeZ7l2850sbxvlqrnk36pk8LS2Nl19+2f/3woULWbt2LTNnzqyxbUMf4QoKykP2Jq2P5OQY8vLKGvy5tqSkUk4pKLI2eOxWuwtUlSizjsJiG3l5ZWTllALgtLsQHhWb3U1ubimKonC6Upe2Wx3k5VV5waJS6jiRWURKgoXMrGIA3E53s5zPKJOOvIIK/77KbS7vuoFOQ15eGbrKy+PYiQISY2tPjc/2z82JQafB6fL495ntm5vNWWPMDqt3UTAnr6zdXR/VaatrvNzmYs+RAmaN60F+fvD6hk54r6WjGYWYwlQfrHY3mTllKIpC9umSOmWL1ppzQaH3Sa+i3B7W8dSS01Ts2YB+6DSKnCYId4yeyt/bqWJi62gS3pB5azRKnc5xo0ShgwcP8vnnn/v/FkKg0+lITU0lP7+qHVhubm6QZi8Jxhe21VC5xlWpm5sNOqLM+iq5pnKh1ZcMpQpRdYxKuaN69/nqpQ18/49uJv06xqwPkmt8kTSJlYk0vsfX+iQbv1xTGSfv9gi/M+DT5w0h5Bpf84dQ53jXkfw6I38kXnYezkcVglEDkmu850uIakiEzfHTpQi8EVONyRFpCdwN1OQdP30EGh2GERc36Di+rNfWXHxtlJEXQvDkk09SUlKCy+Vi5cqVTJs2jREjRnD8+HFOnDiBx+Ph448/ZtKkSc095ojBZ5yqlwuuD184pNmo82rq1TR5s0HnL9bl0+WtdnfIejRV5Ya9Y2iOrlAAwl6O44f36afPocxWdUEXVjYy9nntsdHhGfmqoml6vzH3nT9fbflQC696nQadVqlh5G0ON39dvZtvdrZsSnkksP1gLkmxRnqm1pQPfKUNGlJy+FhWqf/fWW1QXz0UrgbEyavF2biPbEE/ZCoaS1yDjhNbmSDYnH0W6qNRv+SBAwdy2223sWDBAtxuN9OnT2fOnDkALFu2jLvvvhuHw8HkyZNDSjgSL9W97HDxlTQwG7VEmfUUVHZfsgV58t6v1u70EGMBq8MV0nD7Gof4yg37usk3ti+rEAL30e9xbHkXYStlLrCRUQj1HBSNhsIyOwoqCc4sHD9toFPGXi63KJSV9QFq7xVqdbjQKAomg9ZfUtjp9mA0VIVTBoZQCrcTtTATT0EG86J2UFYRG7S/CpvLW0FRNhWpE5vDzc/phUw5u3tI6dWo12Ix6ihuQELUsVMldIozUVBiJyv/zFgQ9zQgTt7x04eg1TfYiwfvb1Ov07SqJ98gI79+/Xr/v6+77jquu+66GtuMHz+ejz76qOkj6wD4EjAamgwV6LGH8uRNRm8IJVR1hwpV0gCqkqN8CVE/Hcqjd9fYRvW4FG4nti9fwnNyN5rk3pin3c2BdR9xfsV2bP97Cm2XAfQ+sIc/JWSi+8KFE9DGd2WSKYvSvX9H7f9bNFEJIfftK07m2r+B3ns3YuBcv3F3uIJT0p0/r8Px/bvg8c5pkg5O5ZUjxDl+Q2WtpVSzJJh12zNxewRjBtUuuybEGMP25IUQHM0qZWTfThw6WXzGGPlwQyg9RVm4j2zFMGIWGnNsnduGQlEUYi2tGysvM17bkMaGUPrkGpOxdk3eaKgp14Qy8j7vvsLuIruggozccs4dlNqI2YBz58d4Tu7GOG4BlnlL0Hbux+Eel7K8fCKevOM4d/wXjbOcn0VfTFN/SdTCvxJ91ZO8ZZ+KxZ6Ldc3juE/uQXhqng+r3c0IYyaOjf/BUnyUSaYDfiPvj5PXCuwb/+OtIdJ1MKZpdxN1zTNs0J5PN+cxXAe+8e/Pd2NtzVC29kZesY3/bk5nVP9k+narXZaIjzEGefL70wtrNfp5JXbKbS56d42la6cosgrODCMfbgilc8dHoDOgH954haJ6aQMhBN/tzmoxCUca+TbE5z3YHO4GRRQFyTUmHQ6nB4+qYnd60Gk16LQavydvd1W2CXS4QyYD+ba12t1s25+LAowZ2PDFck/hSZw7/oeu3wQMw2f463dEm/Vsc/aBy54h+oZXeNe4gC3RF6HvO87vCZ0y9+fT+AWg0WL79M+U//tXWP/3DM7dn6HavPqtqfwUl2rWoUk+C3vyIC40/YzL6o30cLpVTIoTy6aXce1bj374LMwz7kXfaxSa2GQORZ1DhqY7ji3vopbmes9HpTzVnO0JIwkhBMu/PIRGUVhwUb86t02IrvLkT+WV8+zKnfxvS3rIbY+dKgGgd9dYunTy9jw9E0pBu8OQa9SyPNxHt6EfdEGjvHgfcVEGSiqqrrvsAitvfnLA31axuZFGvg1xByT02Jzhe/NBck2ldm5zeLA5Pf7oGf/CqyPAk69lMTXKpKPc5mLb/hwG9Ij3N4MIF6Gq2L99E8VowTh+QdB7/qxXjw7FYKaozOGPrPERF2XghCOOqCv+iGna3egHTkbYSnB8v4KK5fdh++oVppd/iF0xY57xa+zDLsWkONEf/AIAj9PObdHrUfKPYLrgF5jGXR1UJMps0vORegFoNNi+/juuYz8Qd+hjbon+mlT70QbNtaPw06F8dh8tYN55veoMbQWIjzFQUuFEVQUffHsMIahVhjmWVYpBr6FbchRdk6LwqILcotpLQQvR8HDqxuBbH6srV8O5+zNQFAzDZjTpWLFResoCvHZf2Y3OiZYm7bc2ZPu/NiSwGbW3KFh4i53+BVZjlZG3Otz+WvK+96CqO5TVEVquAa8uf+hkMfkldqaNTmv4PPatQ809hmnq7WhMwREYgVmvQggKyxycUy0ULzbKQPrpMhSDGX2vUeh7jQLAU3gK14ENuA5tQo+LLxKu5VpLPNpOJrY7ezP6+DeoZTPpfWQF8bpcdOffjr5/zTreFqOOww4TpikLsX/9d+w5R0hWtETpdAxSTuHOGoGu66AGzztSsTvdvPPVIbonR3PR6O71bp8QbUQIb1bsjsP5GHQasmsJjTyaVUqvzrFoNRq6ViblZeVb6ZLk/bcQgvTTZfx0KI/tB/Nwq4LHbxrTqDWihuAz8lpNaLlGtZXiOvAdur7j0UQnNulYsVEGyqwuVCHQKN4qqRpF8Z+D5kZ68m2I263iu6QaEmHjD6E0aP0SjM3u9taSr4yqMQWEUKqqwFZHjfgos578EjsaRQkZC10XnsJMHNtWo00bjq7PuBrvB5YbLrN6a3bU9OSNIaNctIndME24jujr/8Jz9qtwRXcBvE8pn9pGgFCxfvAYiWWHeM86DlO/mscH77qD1eFG13c85lm/wTL/Edb3/R1LS+aR74nB9tlf8OQea9C8I5ldRwooKnOw4MK+YUWb+GLl3/7yELFRBqaf24OScmdQ9zLwrp1k5JTRu5tX6uiS5PVcA3X5Dzce5w///pFPv88gxqInv9jGhh0tH+bq8qjotEqtyZuun9eBx9moiJrqxFoMqEL4F/0zcyvonGRptibi1ZFGvg1xeVRiKpMjGhIrb3d60GoU9DqN34j6PHmzz5P3L7y6/Z5/bQW6fGWFB/dK8Df6CAe1NA/bJ8+iGMyYamlKHFikzKfbJsQEP/7HRRtwOD01Oln50erJs+v8Nym9TkOBGkNxl3EIRzmHki5gm3sAmlq8MItRV5lApqJLG442pTflTrAKE6+UTUMYo7F++mc8+SfCnnskk1sZktu7jsXWQHzyXkm5k7kTzuKszt6nudPVvPmMnHI8qqB3ZWq/yaAjKdZEdqW04/aorP/pFEN7J/KXe87jwetHcXb/ZD7fllGjX3Fz43bX3sRbuBw4f/4KXc+z0SZ0bfKxqidEZeaV0z25Zbx4kEa+TXG5VX9yREPCKG0OryyjKEqVXGN3Y3N6/DKNXqdBUbw3hNrKDPvw7WNsA6JqVGsJ1k+eQXhcmC/+LZropJDbxZi98yu3uaqyXWNravJQexag063iUYVfzvIlQ2WfNRPz3Ac5GDMefYgywz58NwdrQBEtX9JXqbBQPO5OFK0e64d/wLn3q1bTgduaU3nlfLTxeI355hXbiIsy+Nd16sOXENUpzsTkkV39Hnp2tciZo5VJUL27Vi1adu0U5dfvdx0poNzm4qJRaX7n4OppAyi1uvh2Z1YjZhg+brX2qpCug9+Co6JZvHgISIiq8D7t5JfY6Z7c+Jpd9SGNfBvi9qjEVWZ8NiSM0uZw+zXKKgPmwu6o0uQVRcFk8EbeWANKAoQiMdaEUa/l7H7hSTXCacP26bMIazGWmfehTaxdtzXovdE+5TYXhWXB2a4+6ittUP0m5TM+To8WXZcBOD0iZLarD9/nAp+WrHY3vgePUiUOy2WPo+06GMfmt7F9/hd/VE8k89nWDNZuPF7j5ppfbAuqGV8fsVEGRg1IZuGMAei0GpLjzWg1CtnVslmPZZWQGGsMWtjv2slCdqEVVRVs2pNNXLSBIb2qciWG9E5iQFo8n249UaN5e3PidqshwyeFquLc8zna1H5oO9cdZRQugZ78qcrqqC1p5OXCaxvicqt+A9cwTd7jN/LRAdE1vq5QPkyVfV59xq22bk/Tx6QxfnBqWKUMhFCxb3gdtfAU5pn31VtiVVEUYix6yir1R63G+3cgvou+JKD5xJFTJeQWWZkwtEuNUgtVGa9VcfJ16Zm+c2UL8uTdJMWayC+xU2Z1obEkYZ55H66fv8KxdSW2T57FcsnvUfQNizRqL6hCsOdYAeCVVeKiq+aZX2Knb/fw0/U1isKdlw7z/+0z9KerGfmjp0ro3TV4v12TonC5VY5llbL7aAEzzk1Do3hLB3jy0yk6VM613UpZl3uag+tyGdCzE2h1KAYL2m6DUXThy4t1UVt9d3fGDkRZPvqxVzfLcSDYyPvyWLqntJxcI418G+L2CKLNejSKgtXREE2+Sns3m3xyjavSyFc9Ypsqa8pbA2rdhMKo12IM03Nz7vgYd/pPGMcvQJc2rP4P4Gsx6MLtVkmIMaKppt37DEygJ//RpuP8fKyQbp2i/XqsT64x1qhdo4YsTuYj8GnHh9XuIjXR4jXylTcgRVEwDJ2GJjYV2+fPY9/wOqaLfoWiRN4D74nTZf42dDlFNgb08HrPbo9KQamd8XGdm7T/LkmWoAib/BIbBaUOZo6NRwgV99GtqEVZ9HHoOduQx8/rsphvzmRi0SbK/3UKXN51ATsQD1weBZwAe+Cyid6MvvcYdP0nok3tV2dv1fpwe0RIR8G150uU6CR0Z53T6H1XJ8qkQ6tRKLW6sDltmI1akuoJU20K0si3Ib7m0xaTrsGevE/m0Wq89Vwq7G4crmAjb9Rrw5JrwsWdsRvnj2vQ9R2Hfuj0sD/n8+QrbK6QMdcxZj2KUmXkhRCkZ5chgHfXHWbmuT2Cxl9VoMxr/J1uNUy5puocV9jdDIwzoVGUoJhlAF2P4RjHXoXj+5U4f/ovxlHzwp5re2HPsQIUvGVqAxdIC8scCAGd4htndIRQUXOP0SvGwe6j3kQnrUbDoZPFAAyKKsa69p+oeccAhRgEN0YDDnCa9Bi0PdH2G4+201loknuR0qcP+XmlHDtZyMcbD4PwoLo9OIpyWNi/iNijW3Ed/BbFFIO2+1B0PUag6zECxRC+3AShPXlPwUk82QcwnHsViia89YlwUBTFn/WaW2SlW3J0i3bVkka+jVBVgSoEeq0GizHYyJdZnfx5xU6/B67Tarh17mB6dfEuWNmcbjobqxInLCadP3Il0Fv3evJu/6JuU/q2quUF2Na/iiapO6ZJNzXooowy68nMLcftUUOmx2s0vnoe3jkUBKS+HzpZ7DfSPiOv02rQKEqQXFOnkTdVRSCB9ybiy0uItuhD1q/RD5uJpyAT5/Y1aBK6ou89ptb97z1WwJFTJcw/v/4GGGcKe44WcFaXWBwuT1C537zKyJrkuPCNpFBV1MKTuI9tw3V4C6KikMlAv+h4SjbmEh0fi2XvYe6Oyybm22yEOQ7TBbei6zsO4bTy9BvfUFThZuZFo5hyTnCehkZvRNGb6NO7K7/u7Y1scXtU7n7hO9YZx7Bw4a24T+zEnbELT+Ze3Ee2gFaHLm04ut5j0CT1RBPTqV5ZxxdCGfTa3i9Aa8AwsPkr6cZaDJRanWTmVXDu4MaVEQkXaeTbCH9BJJ0Gc2Uct4/002Vk5JYzvE8SUSYdW37OYe/xQr+Rtzuq5Brweqq+yJVgucb7utXhXWQ0Ghrvjbj2fQ0uG+aL7kLRNUynjjF7M/zsTg8JsaE/Gxdl8Gvyx097myVce1F//vXpAXYe8fYoCLxJ6XVVfV6dbrXOZBnfTSKwF64qBFEmHTEWfcjSBoqiYDr/Bqwlp7FveB1NXCrapB4h97/tQC6b9mQza2zPJp3j1qLM6uRYVimXnNeLjJwycgIyTvN9Rr4e+U543Di3r8WdfQC1IAPcTlA0aLsPRX/uFeTl5FGxazNdDnyGA0FXjFQY4jAMm4Nh5Gy/p62YYtAndaPIWsK5g8OTiHRaDYN7JrDnaAFM74++7zj0fcchVBVP7lHcx7bhPvYD7vSf/J9RLPHoB0/FMPJiFE3Na8W78FrlKKi2UlxHtqDvfx6KqfkXRX0JgFaHu0XDJ0Ea+TbDZ6D0Wo23mXZA5Ed+ZWPrRTMGkBhrYv+JInIDvK3AUEnweu++H2qgsfP1ebVVFierroWHi1A9uA5tRJs2HE1cw72OaLPe/zSRGBNaBoiNNvjlmvTsUnRahbSUaK69qB9Pv7sDCG54EmTkXR7/AnYoDHoN2oBeuIHyle8GFApFZ8A8/W6sHzyG7fMXsFz6aMiaJU6XByHgZG55gxYsG0tBiZ1os77RN5S9xwsRwLDeSThcHvYcK0BVBRqNQl6xHa1Gqbe0hWPrSlx7v0Sb2g/9wMloO52FtvtQf3312O4ulmyM4ZrzuzJ2UCqL/76DK6f0ode5PWvsa86Es5hY5mjQk+aw3knsOJxPdoHVnzmraDToOvdD17kfYvwC1PwTqCWnUUvz8OQexfnjB7iP/4hp8i1oOwWPw616n6p9uPZvAI8b/dBpYY+pIcRa9P6oppaMrAEZQtlmuAM8+epyTX6JDa1G8WcSpiZY/Ebc7VFxudUanrzvgqmx8OryUFFLLflw8WTuRViL0Q84v1GfD6xNXz3b1Ye3aFOlJ59dSvfkaPQ6DQN7JjB6QDJx0YaghicGvQZngCZfV3SNoiiYjVVPSxX+aCM90eaqyJ9QaCzxmKffg7CVYP/q5ZAVMp2VpY6Pn26dsMulb/3Isnd+qpFRGi57jhUQY9FzVpcYOidacHsEBZVPgvklNpLiTLUmlgG4jnyPa++X6IdOxzLv95gmXIe+/8SgBhrRZj2xFj2ZJYLDud7z2797fMj9DeiRwLghDVvoHdY7yT+XUCiKBm1yL/R9x2M85xIsM+/DNO1uhLUY65oncB3/MWj7QE9eLcnBuesTtGnD0CZ0a9C4wiU2wClpaU9eGvk2whVQ2tRi0gcZ+YISu/eHVul5pyaaySnyevK+kKtATz7QgAeGUBr1Xk3eW2a48Xq86+B3KKYYdD1GNurz0QEhk7UVu4qLMlJaWeTqRE6ZX5oCuHXuYB5eNDpoe71WE1BquO7oGvDeCH1yTZAnbzHUW25Ym9Ib06Sb8WQfxLHtvRrv+6J/Tpxu+V6kFXYXxeVOTpwu46UP9vjPQbioqmDvsUKG9kpCoyikJnhlE58un1dsIzmu9kVXT2Em9m//ibZzf4zjrqrzWJ2TojhdYOXQyWIMeg09OzdfQ+6kOBNdO0XVauSrcyq/AtJGEnXlk2jiUnHu/F/Q++5KTV54XNjWvQIaLabzbmi28VbHZ+STYo21ZqI3F9LItxGBVe8s1TT5/BI7nQJ+aKkJFsqsLqx2d1U4pCFYrvFR3ZN3ulQqbI335FVbKe4TO9D1m4Cibdw+Aj35ujR5jyo4ll2KzeHxp8YD6HXaGjcHvU4bJNfUtfAKBK17BC5Ee6UkV72lnvX9JqAfcL63GFu1RClf+8H0VjDyvoXR0QOS2X+iiDf+tw+XWyUzr5yt+3K8xqwOjmeXUm5zMayPt8iWr/Kh70kxr9heqx4v7OXYvnwRRW/2hpaG0LYD6ZJk4XSh18j36RoXdv/UcBnWO5FDJ4trL4dRyfHsUh75x1Y27MhCMUWjH3QBat5xPAUn/du4KkMoHVtXoeafwDT5FjQxtXcqayo+I9+thaUakEa+zQjU5H21VXwhgfnFtiAjn5Lg+yFasYeIeQ/y5I01vfqicketJQ3qw31kC6ieRks1UGXkdVoNMbW0FfSFhO6qXGQ9q0vd9bq9ck2AJ19HWQMgSBKzBiRXxVj0CBFexrF++CzwuL16bQA+Tz47v6Jeg9NU8oq9ssqcCWdx1ZS+bNufyy//vIFH3tjGax/9zGsf/lzn538+XoiiwNBeXrkjNsqAyaDldKEVm8NNuc1FpxBGXrWVYv34KUR5AaZpd6KxxNc71i6JFsptLjJyy+mfVv/2DWVY7yTcHsGBE8W1biOE4J0vDyGoktN0/caDRovr4Hf+7TwelZ7Oo5Uy1DT0zRgXHwqfkU9LOYOMfHl5OXPmzCEzMxOAlStXMmfOHObOncuDDz6I0+nVU1966SWmTJnCvHnzmDdvHsuXL2+ZkbdzAtuN+TJRrZWx7qVWF50CQthSEysfqYusAU28AzX5KsNZ3ZMHKC5zNsqTF0LgOvgdmuRedZYuqA+fYU+MMdYaeulbON11xFuqtmunumtrG3QaXC4PQoh6NXkg6GmpypPX+aWk2hZfA9EmdEXbfSiufesRapUxd7pULEYdAm8RrpYkLyD6ZebYHtw0ayAzx/bgtrmDmTm2B5l55f4CY6E4lV9BcrzZf+NVFIXURAs5hVb/gn91T161lWL7+GnUktOYZ9yLrnP/sMbaOaB0bksY+X7d4zHqtX7JRhWC3GIbakAtnu/35XA0qxSzUcvJyu9GY4pB1/Ns3Ic3+9dYTJ4yxpR8iqbTWRjH1i1DNQc+Seyszo1vPhIuYRn5Xbt2sWDBAtLT0wE4fvw4b7zxBitWrOCjjz5CVVXeeecdAPbu3ctzzz3Hhx9+yIcffhiyD6ykqmGIvjKEErzeZEHlDy3Ik6/80eUW2rBVavLmoOiaKsMeKOP4oi9UIRpl5NX8dNTCzCZ58VBVAK16YbJAfJ5NZl4FPVJjghZZQ6HTaXBVLkKD17OvC4tR579BWh0uFLxPPb4CauG2ATQMnYawFuM+VrVw53B76FcZVdPSkk1esY1os97//Z8/oitXXtCXcUM6c8HZ3kXCHYfyav18TpGV1ITgG2hqgpnThVb/DSTw2lPLC7B9/BRqaS7mmfeh6z407LH6CpVpNUpQUbLmQq/TMKhnAruPFvDFDyd56O/fs/jVLfx19W7KrE4cTg+rNxylZ+cYpp7TnewCq/9pWT9gEsJRjvvEDoQQXKL9Dq1wY556O4q2ZTVy8D6d//EXYzmnf8tJQj7CMvKrVq3i0UcfJSXF2xbOYDDw2GOPER3tzdTq378/WVneKnF79+7l9ddfZ+7cuTzxxBM4HOF3ce9IBLYb8zfTtrv93lSgJ2/Qa0mMNQbJNYEeu2/hxld+2IdJHxyB01Bc+78GrQF9n7EN/mwgJoMWnbbusLy4qKr3zgpjgc6g0+J0q37Jpj65xhwg1/iagmuUqjo64Rp5bdowlLhUnHu/9L/mdHlITjCTEGMkvYUjbPKKbaQkhNbMU+LNdE+OrtXIC+HtwlT9850TLRSU2v1VI32evPvETirefwS1vADzzHvRdRvcoLEmxZrQaTWc1SUm7IqWDWVYnyQKSu2sWHeY2CgDF4/ryb70Qh578wfe/HQ/RWUOrr2oHz1SY1CFICvfu8Cs7T4UJSoB16GNuA58wwBtJgeSpqKJ79Ii4wxF105RLZrp6iOsX/7SpUuD/u7WrRvdunm9hsLCQpYvX86f/vQnKioqGDRoEA888ADdunVj8eLFvPLKK9x3331hDygpqfEaVXJy863etzSWXO8PKiU5Gk/lop/eqMde2fuxf+8kkgIMffeUGArLnOgqPfXuXeP9i5GdU7zztph0QecgtVK/BUjtFN2g8+OxlZNx5Htihk0iuXvT6pgAzJvUh6F9OtU6BiG8lSSdbpVh/VPqHWtMlJHsAiuxlecoMcFS52eSk6JwuDwkJEahCoWYKAPJyTEoeu/5VHSasM9Pydg5FHzxBjHO0yiKwsW67xmSr9InPo7jOTHEKd0xdGq8vFUXBaUOBvRMqHWs543sxqqvDqI3GfwhuD6KyuzYnR76pAV/vl/PRMSmdI5klWIx6ejZNZqib1Zg+/5DDKm9SL3sN+gTG1dH/fIpfTmra2yTfpt1fXbu5L4IjcLogan0qQzRnD7+LJ5660e27c9l0shuTDg7jVN55cBeim0uRlfur3DEFIq3rEXNPsghVxeKu088o2xIc42lSclQOTk5/OIXv+Dyyy9n7Fivt/f666/737/55pt56KGHGmTkCwrKG9TU2kdycgx5eS0f3dAYjmeXsm1/DldN6eu/cxcUeo18eakdbWU6dXZuKRk55ei0GtwOF3l5VbpvYrSBHw7kklfpbVnL7HgcLpKTY3BVLiQa9dqgc2AP0Jk9bk+Dzo9zz+cItxNP7/Ob5bzOHuvNFq1rX7FRBm9kUbS+zu2Sk2NQPR5sDhfZOV7P2WF31vkZUfmYnpFZRGGJzX+ufI/v2bllYc9TdB0N+uVkvf0IuJ1MMGhwOBIZ6DzMYKGS+do6NKl9MQyagq73mGarlJiQGEVekY0xA5NrHevA7rGoAtZtTWfSiGDDfDizGACLXhP0eUul1PXzsQJGJlaQ8frvUAtPejNEx11DsccAjbwGZlS2D2zsNRTO73pq5Tx928UatSxZOIrvdmUxbmhn8vLK0KkCg17Dz0fzGdHLG1mkpo2FzR8gFIXlFROY5GjYb6QlaYg902iUOp3jRhv5o0ePcuutt3L99ddz8803A5CVlcXmzZu54oorAK93ptPJpNqfDuXx+baTXDapt7+5hT9OXqfx66vWSrkmMEbeR0qCxS/nKEqwBu3T203VMiBNhsbJNUKoOPetR5Pat0ZmYEsSF2WgzOatDlkfvoxXXyJSvdE1AeseVrvLv9it12kxGrRhyzUAisGMcdSluDN2ovYYzZJPncybMoTOCUbeXbOJO8cpJGRvxb7hddj0FtrUvmi7DEDbuT/aTmc1unxxfuWiYl11ZdJSokmKNbHjUF4NI59T6NXcUxPNCCFwp/+E+8ROko0xjDcWkqwpY4pnH8Ieh3n6r9GddXajxnkmYDbqmH5uVRkKjUYhLTnav/gKoIlLxTDmckR8GsUrC1us/V5b0ygLXF5ezi233MJ9993HvHlVFfpMJhPPPPMMY8eOpXv37ixfvpxp01omLbg9Ya+sY+5wqVVG3hMcQgleI19QEhw+6cMXYZN+ugyzQRek5fluEqZqhjxYtw//q/ac2o8oycF4TutWXxzaO4kenWPCKr9gqIyT92W91r/w6qu77y3YlhAQd+8tbRC+kQcwDJ+BYfgMisoc2MQmDAYtPbsmkKPG87OpL9Ovmo8n+wDuo9vwnD6E84f3vR9UFDQJ3dEm90LTqSfaTj3RJKaFZfhPVz7FhdLkhdOG+9Q+hL2M2V2L2X68HFtRF8wJKf5tcoqsaBSFBKUM2+ev4cnYBQYLuOxcE+W9Hk/GjGDQZbehGFs2C7MtSEuJZtv+XIQQ/t+P8ey5lQvy3zZ7HP+ZQqOM/OrVq8nPz+ef//wn//znPwGYOnUqv/71r3niiSe44447cLlcnHPOOdx0003NOuD2iK/HqsPp8YeuBZY10Os0GHQavyd/TmpNLc4XEXEyt5y4qODVf4sxuK+rj1CLs+Hg2rfOm+FaR+XFlmDeeb3C3lZfqd/7PPm62v9BgCdvD/bkwVcKuf4QylD4EqGMOi1xUQYSY42kny5DURR0XQeh6zoI8CYSeXIO48k7jif3GK707XDwW+9OFAVNfFdvqGpKb3Q9z0YTlVDjWL5OS76FUbX4NO6MHbgzduPJPgTCO5ZRwKgocL23DnqPxjB8JprE7uhP72Vh/AEc778LioJx3DX+2iwvvf0tJ0+XMGPUaAZHoIEHr5HfsDOLwlIHSQGOVGA4cyTSICO/fv16AG688UZuvPHGkNvMmDGDGTNmNHlgkYQvdC+wGXFVMpTXozCbdBSWebsUhfLkk+PNKIr35lDdY9frtOh1mqCSBhBcdTJcuUYtL8B9YgeGEbNbJZSssfgerX3ntt6M14CnJV90jY9osyGsOPlQ+L5TX1mFnqkxIcMoFVM0up5no+vplUCEEIiKQjz5J1Dz0/HkpePJ2IX70EYcm95CmzYcff/zvLXRKzX9nIIKEnVWLPv/S3n6dkTJaQA0Cd0xDJ+BtscINDHJeBwVvPDOVs5PzGNw5h7cx34AjZYpqgenYkDXezTGMZcH9eWNSupMfpYguZF15NsDaZXO08nc8iAj7w4oMRKJSMG8FfDVmwk08oFlDcBrhE/mevXCTiE0V71O429XZzbU/Nriow01KjFqNRq/dh1Waz+PC/u3bwIK+kGTw5tcG+Ez6hWVxcXqTYYyVmX/BjYFB68nnxVQDuBYVinx0YZa6+wE4nuSMFbKRf26x7PjcD6nC63+kgGhUBQFJTrJa2grsyuFEIiS07gOb8Z1aCP2r14GvcnfCCPt8BaWxP6Ma7dA220IuqEXoesxskb6vSY6kbizSll+LJW/3HEj7kMbUcsLeHmroMvAkVw7ZVCN8fjkwFDXXqTQPTkKBTiZW8bIflXnzK1WhTNHItLItwI+b9MZwpPXVl5YUSY9R7NKAEJ68oC/XV2o2un/d/XIkJKMUa/1hyfWhVDd2Nf9DU/mXkyTbkYTE15T77ZCX+k5+7JX6y1QVnmTy68MKw325Ksah1jtLp5+9yfO6ZfMbZcMqXccDnewJz9+SCrvf3OUb3dlcdWUuvvfVkdRFJT4LhjHXI5h1KV4svb566K7j26lOzr2G4Yz9rLr0MTW/f0M7JnAlp9zyCrx0H3oNEoqnPy8YSPDa4nCGD+kMy63Suek+he92ysmg47kBDMZucFZyYGJiZFIZM7qDKPKk6+qGOjrRONbZLSYdPiysWs18pULboEZrj5SEixBhcB8mAxaLEZdnUkXQlWxf/0Pb+/WCdejb4FOOM2N35OvDB+t7yZmNGhRFG8pXaCGJ+9weXC6PGzeexqnS/U/VdWHs/K79SX7xEUbGdG3E5v2ZPuf1hqDotGg6z4U06SbiLr+L1jmP8Iy29Uc7TqrXgMPMLCyZ+vBjGIAciurmKbWkkiVGGti/vm9G91zoL2QlhJd47v1fU/1ZVm3V6Qn3wr4Fl4DPXm3WwQ9Hlr8IX2aoFrTgfgWX6tr73VhMmj9TwuhEG4n9g3/wH1sG4Zzr8Qw9KKw992W6P1GPjxPXqMoWIw6f0axJWjh1Xu+y20uvt5xCoDThdaQfT+r4wgR3TN5ZFd+OpTHjsP5jBnojW4pLLXz6dYMSiqcVNhc2J1utFrvgrvZoGNgzwTO6Z8cMitY0WixxaSRb0uvt2OTj+R4M0mxJg5kFHHhqO5V4ZMJkeuph0OPlGh+OpiH3en2/478kW66yLzBReat6wyjKoQyQK6pZkB8mnGnOFOtXrdPNw3lydeGsdKTD4VqLcb63z/hPvYDxrFXYxw5O+z9tjU+I18epiYP3sXXKk8+WK4B+PFgHtkFVoaclYBHFf5olrqo0uSrvpMhZyWSFGvim53eG4bD5eGF1bv5ZmcWp/LKcblVosx69FoNDqeHjNwyln95iP97eRNL3/rR3zsgkLww2/IFMrBHPAcziisLd3nDJ5PqqBXfEUhLiUHgrZHkw+ORmrykCaiq8Bv3oIXXapUTfZ5lXT9CnxcWauG1NmaMCd2X1FOYie3TPyMcVszT72l3iS++5KdwF16Bap58sFwD8OnWE1iMOi6d1Ief038kM6+83lKw1aNrwJt4c/6ILqz97ji5RVY++PYYmbnl3HvVCH9Ho+pk5Vfw06E8Pvn+BO99fZS7LhsW9H6jjHzPBDbtPU1WXgU5hd78i0g1ZOHi+z5P5pT5m8rLEEpJkwisL149ukYf5Ml7DU1d0Q2d4k0M75PEgB7xYR9/9MCUkK87Ni8H1YNl3u9rbVB9JqMP0OR1Wk1YWnKgRBPKky8pd3LR6O70SI1Gq1E4FeDt1YY/Tr5aMtb5w7vy4cbj/OW93ZwutHLFBX1qNfDgLVbVtVMUbo/KR5vSycgpo0dAvkSoCpH14btODmQUhSxM1hFJjDWi0yrkl1bVdaoKoYxMIx+ZszqDsDmqDLvDGRxdowvhydfVek2r0XDvlSMY0KNmokxDUEtO48naj37otHZp4KHKyFvt7hoGtjZ8UUkKwaWafZo8wJSzu6HTauiSZCEzr/7FV4dLRaGmgUiIMTKiTydOF1oZMzCFWWPDO8/Tx6RhNur4cOPxoNfzim3ERxtDRlbVRqc4M53iTBzIKA5ZYrgjoigKJoPOHwwBAYmJERonL418C2NzBjeX8OGq5slHhSHXNBeuA9+Comlynfi2xBCgyYcb+ua7kZqNuiDP31d2eGCPeLpUNrronhxdWbmwbpwuDwaDNuQ6ymWTejPlnG7cfPGgsEvKWkx6po9JY8fh/KCesXnFdlIbEd44sEcCe48VYHd6pCdficmgDXK4fGW/ZQilpFHYAz15V3VPvuqH37VTFNFmfb1t75qK8LhxHfwOXc+RYbVwO1PxxcnbnZ56i5P58Eli1RPDNIrC9TP6s+Ciqo5H3ZKjKCh1BDVYD4XT5cFYi3HonhLNwukDgjKPw2Ha6DQsAd680+Uhp8hK58SGlxsY2DPeX3Pft3Df0TEZtCE9+bqi0NozUpNvYWxhavJdkqL4669b3rN2n/gJYS9DP/CCFj9WSxJ47vRhyjU+4x4VImnsgpHdgv72NVg+lV9Ov8o65aFwuNR6wzcbisWkY8a5aaz57jiP/XMbp/Ir8KiCtNSG91oYGCDtpUi5BvBGnAWulQUWC4xEpJFvYXweg6LUNPJGQ+vXhnHt/wYlOgltA9q4nYkExqXXlwjlw6dnh1PioXuy12vOzKuo08g7XZ4W6Xp00eg0dh0twKjXMnNsD3p3iWXK2J4UhwivrIvEWBMp8WZvnf4OHj7pw2TQ+TusQWAIZWRq8tLItzC+kgaxFkMNuaa1PQe1NBfPqZ8xjLoUpZ1n9wUa9vDlGp8nX/9lnxRrwmzU1rv46nB76i1z3BjMRh1LFo0Oeq2+Spu1cc6AZA6dLI7Y6JGGYjJoKS6rakvqivDoGmnkWxifxxAXbfCnwAO4PKLVF3pcB74BRWnXC64+dE2Qa8Ipu6woCt06RXOqnvIGTmfLePLNyRUX9GnrIZxRmPTBco07wuPkI3NWZxC2SsMeG2XA4a6KrnG7Pa16UQmnDee+r9H1PAdNdGKrHbelUJSqpuUt4cmDV7I5lV+BELW3o3S4m1+Tb240ihLxNWkaggyhlDQrNocbo16L2aALrkLZyp68c986cFoxnD2n1Y7Z0hj8Rr6hnnx4Rr5bcjQVdjfF5bXXmne6PGe8kZcEYzJWj64R6LRK2GGu7Q1p5FsYu9ONyajFqNfWKGvQWp68cDtw7f4cbfehaJPD7750puP35MOUa3yZrYHJT3VRtfhau2RTVwil5MzEqNfiUYVfi3d71IgNnwRp5Fscm8OD2aDzGnlncIGy1lp4de3/BmEvw3DOJa1yvNbCZ+TDXZBMjDVx9+XDGDsoNaztfWGUdRl5h0vF0MA4eEnb4muL6dPlW/O32BaENbPy8nLmzJlDZmYmAJs3b2bu3LlMnz6d559/3r/d/v37ufzyy5kxYwa///3vcbvrTiTpCNicbsxGLQaDxl9PXgjh9eRbwQMUHhfO3Z+i7TIAXef+9X+gHeHT4sOVawDO7pccdnJStFlPfLSBwydLat3G68lLI9+e8JUY9jld3qfqyJRqIAwjv2vXLhYsWEB6ejoAdrudhx56iFdeeYVPPvmEvXv38s033wBw//338/DDD/P5558jhGDVqlUtOvj2gN3hwVTpybs9Kqoq8KgCQVV/15bEdfA7REURhrPntvixWpsqT77lbpbnDe/KziP5bNydXeM9VQicbrVFQiglLUeVJ19p5D0iYiNrIAwjv2rVKh599FFSUrzVDHfv3k3Pnj1JS0tDp9Mxd+5cPvvsM06dOoXdbmfkyJEAXHbZZXz22WctOvj2gNeT1/m9TofLUxWX24LGSa0owrbhHzg2voUmtS/abvW3smtvVGnyLedJzzvvLAb1TOA/nx8MqiUD4ApRS15y5lPTyLfe+lhbUO/Mli5dyujRVUkZubm5JCdXtR9LSUkhJyenxuvJycnk5OQ083DbH3aHB7NB65cIHC5PVRPvFrqwnPvWU7HyAdxHvkc/fAaWmfdFZORAQ6NrGoNWo+H2eUOIseh5ec0ef5MSqNnfVdI+8Mk1dpdXTo50I9/gZKhQMcOKotT6ekNJqqXRcDgkJ8fUv1Er43B5SIgz06myuFRUjMnv1SfEW5plzIH7UJ12TmxdialLH5Ln3oU+oXOT938mkpwcQ3SUt1VeYkJUi373ycCSm8fywEsbWbPxOL+5dhQAaqG3xECnxOb5HsMayxl4jbc0zT3n8sonMIPJQHJyDBqtBrNJd8ad2+YaT4ONfGpqKvn5+f6/c3NzSUlJqfF6Xl6eX+JpCAUF5ahq7ckntZGcHENeXln9G7YiQgisdjdCVXHavbHW2adLqx4Xbc4mj7n6vF1Hvke4HCgj5lHsjoIz7Jw0B745C9X7Y3XYm34e6yPBrGNor0QOZxT5j5WdX1F5fFerXHtn4jXe0rTEnG0V3pIGuXnl5OWVYbW5EEKcUee2IfPWaJQ6neMGP6OMGDGC48ePc+LECTweDx9//DGTJk2iW7duGI1Gtm/fDsDatWuZNGlSQ3cfUTjdKqoQmI06v27rdKm4WrB+tfvoVhRLPNoIi6QJhU/uakm5JpDYKD1l1iq5xhmi9Z/kzMcUIJ1C5IdQNtiTNxqNLFu2jLvvvhuHw8HkyZOZOXMmAM8++yxLliyhoqKCwYMHs2jRomYfcHvCV7fGbND6DYHD5fEb9+bWAYWjAvfJPegHT233BcjCwVdTvrGFuxpKjMVAeaXXpyhKVes/mQzVrqgeJ+92qw3qm9zeCHtm69ev9/97/PjxfPTRRzW2GThwIKtXr26ekUUAvro1pgBP3uHy+Bdhm9uTd6f/BKobfd+xzbrfMxWfBx9u+7+mEmPW41EFVoebKJO+qom3TIZqV+i0GrQapVoIZeQFJviQLkgLYvN78rqg6JqWKm3qOroVJSYZTXLvZt3vmUpDM16biq8cgk+y8bVzlMlQ7Qtvn1etDKGUNB2/XFNZuwa8Om5LhFCqtlI8p/ah7zM2IsMlQ6FvhRDKQGIs3to3ZVbvIrr05Nsvgd2hpJGXNBq/XGPQ+SUFh0vFXenJN6dc4z7+IwgVXZ+OIdVAVVmDcOvJN5WanrzU5NsrgeWGXR4VvS5yHSN5dbYgPk/BbAxeeHU1c/1qIVRchzahie+KJrF7s+yzPeCTwEytFN1S05OvjLeW0TXtDpOhqmCgxyMiugpl5C4pnwHYHFULr77FHmeAJt8cnrwQAseWd1Fzj2I874YOI9UAjB2UgtmoJS7a2CrHqzLywZ68rF3T/jDqtcGefAQb+cid2RmA35Ov9Dh95Yabs91Y8eY1uPZ+iX7YDPSDLmjy/toTFpOecYNbL6NXr/OWp/AZeYfbg06roO0A4aqRhilQk2/F3g5tgfTkWxCbw4NWo/gvIKNB26wFylwHvqXs2+Xo+o7HOO7qDuXFtxUxZj1lNq9c43SqsjhZO8WnyQvhrQobySGU0si3IL4KlD7ja9BpKguUVWa8NsF7EE4r9k3/wdxrBNoLbkFRItcTOZOIsRiCPHmpx7dPfCGUvt9iJHvykTuzMwC7w+3PrgOvXON0qbgqqxc2RZN3n9gJHjcJk69B0ch7dWsRY9H7F15lf9f2S5WRb5mclTOJyJ3ZGYDN4cFsrDLABp9cU+k9aDWNf0R0H/sBJSoRY9e+TR6nJHy8Rr4qGUqGT7ZPTAZvEx/f4mtLNp5payJ3ZmcAdqfbv+gK+Jt5uz0qep2m0Rq6cNpwZ+5B12u0lGlaGZ9cI4TA4fLIRKh2irGyVk2F3XvD1kawJi8tRAtic3gwBXjyPiPvauJqvjtjF3jc6HqPaY5hShpAjEXv9wC9/V3lT6g94pNRKyqbwMgQSkmjsFcuvPow6jX+EMom6fHHfvCWE07t0xzDlDSAGHNl1qvNhcOlSk2+neIz8j7pTWrykkZhc3pqyDW+ZKjGNvEWLjvuk7vR9RolpZo2IDDr1enyyBDKdorPyPvaOUojL2kUdoc7SK4x6LU43GqTCiK5T+4GjwtdLynVtAWB9WtkCGX7xdfntcrIS01e0kDcHhWnW60ZQums9OQbKde4j/2AYo7tEJ2fzkSqe/KypEH7pIYnH8FrK5E7szbGF5oV2HHGaNAi8NaZb4wnr1YU4c7Y5Y2qkan0bYLPyJdbXd4QSunJt0uM1Yy8XHiVNBhfLXmTMdiTByi3uRvsOQhVxf713wEwDJ3WTKOUNBSjXotep6Go3IFHFVKuaadUl2siOYSy0amS7733Hm+//bb/78zMTObNm4fNZmP79u2YzWYA7rrrLqZN63hGyRbCk/c92lfYXX6PMFycO/+LJ2s/psm3oInv0nwDlTQIRVGIsegpKLEDspZ8e6UjhVA22shfeeWVXHnllQAcPnyYO++8k7vuuosbbriBt99+m5SUlGYbZHvE3/qvWpw8eC+shmjy7uyDOLevRdd3PLr+5zXvQCUNJsZs8Bt5mQzVPjHoNCiKNxQWZHRNvTz22GPcd999mEwmsrKyePjhh5k7dy5//etfUVW1OQ7R7vCVMQ2Ua3yP9k53+PWrhcuOff1rKDEpmM5bJCtNngHEWPQUlPo8eWnk2yO+Pq8VcuG1fjZv3ozdbmfWrFkUFBQwbtw4nnzySVatWsWPP/7I6tWrm2Oc7Q5fw5CghdcA/Tbci8p1cCOiohDTBbegGMzNO0hJo4ix6Kmwe2/iUpNvv5gMOv/3qGtCHakznSaXL1yxYgU33XQTAGlpabz88sv+9xYuXMjatWu56qqrwt5fUlJ0o8eSnBzT6M82N3pjAQDdu8aRFOc1zkU2t//96ChDveMVQuXk/q8wdutPl2Gjat3uTJp3a9GWc05JigZyvP/uFN2qY5HfdfNhMekpKnMA0Dk1ttU6jIVLc827SUbe6XTyww8/sGzZMgAOHjxIeno6M2bMALyt6XS6hh2ioKAcVRUNHktycgx5eWUN/lxLkZtfAUBFmR21Urqxltv973vcar3jdZ/YgbvoNLpzLq112zNt3q1BW89Zp1Rdnzaro9XG0tbzbgtacs6BWeclxVaclc1gzgQaMm+NRqnTOW6SXHPw4EHOOussLBYL4DXqTz75JCUlJbhcLlauXNkhI2vAq8krVMXjQjW5JoyQLeeeL1CiEtH1Gt0SQ5Q0El/WKyDj5NsxgYmKkZzx2iRP/uTJk3TuXNVjc+DAgdx2220sWLAAt9vN9OnTmTNnTpMH2R6xOtwYDVo0AQulgZEY9S28egoy8GTtx3DuVSgaaUjOJGLMVeGvMuO1/WIKWC/TRnB0TZOM/MUXX8zFF18c9Np1113Hdddd16RBRQKlFU5iAzw+CPb66guhdO75EnQGDAMntcj4JI1HevKRgc+T12qUIGcs0ojc21cbU1rhJC462MgbdBp8l1JdcbmqrRT3kS3o+5+HYmr8QrSkZQhMZJPRNe0Xn5GP5PBJkEa+xSgud9ZYrVcUxW8U6vLkXQc3gupGP+SiFh2jpHEEGnmjlGvaLb71skgOnwRp5FuMkgoncVGGGq/7jEJtnrwQAtfBb9B27o82oWuLjlHSOMxGHVqNgkJkZ0pGOj5NXnrykgbjdHmwOdzER9c08vV58p7sA4iSHPQDJ7foGCWNR1EUoi16DAatzEBux/jkmkiuWwPNkAwlqUlxhTfeNjaUJ1/PheU68A0YzOh6y7DJM5kYswGhOtp6GJIm4JdrItzIR/bs2ojScq+Rjw+RQeeLxgj1iCjs5biP/4i+7wQU3ZmVfScJJsail4uu7Rz/wmsEx8iD9ORbhOJyr4cXSpM3VBr3UJ6868gW8LjRD5JSzZlOn26xWEzy59Oe8WvyEe7Jy6u0BSiplGtC1cKo8uSDvQchBK7936BJ7oU2qUfLD1LSJC6b1KethyBpIqY6nqojicieXRtRUuFAUYIzI33Upsl7Th9CLcqUC64SSSvhKwMuQyglNSgstVNSXvuiW3G5k9goA5oQF4+hFu/BuX0tijkWfd/xzTtYiUQSEhlCKamVv324l399eqDW90srnMRHhV449ck1gZ68+9Q+b52as+ei6OWCq0TSGoT6LUYiUpNvBNn5VszG2iMrissdISNrIECTr7ywhBA4fngfJSoR/aALmn2sEokkNCYZQikJhdXuwupwU1DqwOnyhNymtmxXqMp49SVDeTJ2oeYexXDOJSjahjX3lkgkjcfYQUIopZFvIHnFVY0/cotsNd5XVRGyOJmPQE9eCBXHjx+gxKagHyAbdEskrYlGUTDqtdKTlwSTX1Jl2E8XWmu8X2ZzIQTE1aLJD+mdxPnDuxAXZcBzcg9qQQbGUfNRNFI5k0hamwE94unZObJbKkrL0kACPflQRr6kjkQogG6dorjp4kEAOE/uAZ0BXe9zW2CkEomkPu69ckRbD6HFkUa+geSX2LAYdRj0GnJCGPniOkoaVMeTtR9t5/4oWvk1SCSSlkHKNQ0kv8ROp3gTnRMtnC4K4clXeD352Fo0eR+qtQS16BTaroNbZJwSiUQC0sg3mLxiG8lxZjonWsgprLnwWuLz5GuRa3x4svYDoOs2qPkHKZFIJJU0SSdYtGgRBQUF6HTe3TzxxBNkZGTwt7/9DZfLxY033hhR/V6FEBSU2BnWO4mEGCPlNhflNhfRAeULSiqcmI26eisUerL2g8GMJqlnSw9bIpF0YBpt5IUQHDt2jA0bNviNfE5ODvfddx8ffPABBoOBa665hrFjx9K3b99mG3BbUlrhxOlWSY43kxRnAryLr327xfm3KSl31LroGog7az+6LgNRNPJhSiKRtByNtjDHjh1DURRuvfVWLrnkEt5++202b97MuHHjiI+Px2KxMGPGDD777LPmHG+bklfijazpFGeiS6IFoMbia0mFM2RHqEDUsnxEaS7arlKqkUgkLUujPfnS0lLGjx/PY489ht1uZ9GiRcyaNYvk5GT/NikpKezevbtB+01Kim7skEhObtl4159PlgDQv1cSXZOj0WoUSu3uoOOW29z0S4uvcyxlWT9QASQPHY2hGcbc0vM+E+mIc4aOOe+OOGdovnk32sifffbZnH322QBYLBauuOIK/vSnP/HLX/4yaLuG9sAsKChHVUWDx5OcHENeXlmDP9cQjp0sAkCjqhQVVtAp3szxzGL/cYUQFJTaGKZPrHMstoM7UEwxFBOP0sQxt8a8zzQ64pyhY867I84ZGjZvjUap0zlutFzz448/smXLFv/fQgi6detGfn6+/7Xc3FxSUlIae4gzjvxiG7FRBn9pgs4J5qCEKLvTg9Ol1qnJCyG88fFdB6EoUo+XSCQtS6OtTFlZGU8//TQOh4Py8nLWrFnDM888w5YtWygsLMRms/HFF18wadKk5hxvm5JfYie5csEVoHOShZwiG6rwPnlUdYSqw8iX5CAqiqQeL5FIWoVGyzVTpkxh165dzJ8/H1VVufbaaxk1ahT33XcfixYtwuVyccUVVzB8+PDmHG+bkldso09AJE1qogWXW6Wo1EFSnCmgpEHNbFehunEf3YZz1yeAjI+XSCStQ5Pi5O+9917uvffeoNfmzp3L3Llzm7LbMxKPqlJY6mDs4ABPPsEbYXO6yOo18rV48p7cY9i+fAlRUYgmviumqb9EE9e59QYvkTQDHo+boqI83G5nqx0zN1eDqqqtdrwzhVDz1ukMJCQko21gGRRZNCVMikodqEKQHG/2v5ZaGUZ5usDKkLMS/dmu1TV5587/gceFeea9aNOGSy1e0i4pKsrDZLIQFdW5wQEVjUWn0+B2dzwjX33eQggqKkopKsqjU6cuDdqXtDZhEhgj7yM+2oDRoPXHyhdXONBqlKAMWOFy4D65B12fc9H1GCkNvKTd4nY7iYqKbTUDL6lCURSiomIb9RQlPfkwyS/21qnpFODJK4pC5wQLm/aeZvuhPErKnSTEGIN+BO7MPeBxous1utXHLJE0N9LAtx2NPffSrQyTvBI7igKJMcGLqpPP7kqvLjEM7pnArHE9uHl28IKq+/iPKKYYtJ37t+ZwJRJJC3LFFXPJzs5q62GEhfTkwyS/xEZijKlGq7ALRnbjgpHdQn5GeFy4T+xC33sMiqbugmUSiUTSEkgjHyb5xXaS4031bxiA59R+cNnQ9RrVQqOSSDouP/30I3/721/xeFS6dOmC2Wzh2LGjqKrKddctYurUacybN5NVq9ZisURxxx03M3HiJK6//ka++upzdu7cwR133MWf/vQH8vJyyc/PY+TIs1my5Al27Nju33fv3n24557f8MQTD5Obm8NZZ/XG6fRq40eOHObpp5fi8XgwGAw89NCjpKX1aOMzE4w08mHw2dYMjpwqYebYhn157uM/gt6MtptsDCKJPDbtyWbj7uwW2fd5w7swcVj9USQnT2awevXHvPXWm3TqlMySJY9TUVHOL395M4MHD2XUqNHs2PETZ589iuzsbHbu/Inrr7+R77/fzIUXTmPz5o3069efP/7xKVwuF9dffyUHDx4I2nd0dDTPPfcU/fsP5Nln/8rOnT+xfv2XAKxa9Q7XXHM9U6dexLp1X/Dzz3ukkW9PCCF4/5tjfPL9CUYPTOHS83uH/1nVg/vEDnQ9R6Bo9fV/QCKRNJi0tJ5ER0fz44/bcDjs/O9/HwFgt9s5fvwY48efx/bt29BoFKZPn8W6dV/gdrvZtWsn99//EEajkX379rJq1Tukpx+npKQEm80atG+AHTu289hjTwIwcuQ5dO3qlWjHj5/Ic889zdatm5kw4XwuuODCNjgLddPhjbwqBHuOFtA/LR6zsep0CCF464tDbNhxigtGduX66QPQaMJf3facPoSwl6E7S0o1kshk4rDwvO2WxGj0BkKoqoeHH/4DAwYMBKCwsIDY2DjKyspYsWI5Wq2OUaPGkJGRzscfr6V3794YjUZWr17Bhg3rueSSS7niinM5fvwoorJMiW/f4I1sCUxO0mq9a2xTplzE0KHD2bTpO957712+/34TDzywpLWmHxYdOrrGanfx0vt7eGH1bl76YA+egC9x/U+n2LDjFDPH9mDhjIYZeCEErr1fgdaALi1yyjpIJGcq55wzhrVrVwOQn5/PDTcsICfnNAkJCRiNRjZt+pbhw0dyzjlj+Ne/3mDChPMB+OGHrVxyyWVMnz4LUDh8+FDIDNvRo8/liy8+BWD//p85dSoTgEceeZB9+35m/vzL+cUvfumXes4kOqyRz8gp44l//cieYwWMH5LK/hNFrFp/FIDj2aWsWHeY4X2SuOKCPg2OT3Uf2og7fTuGUfNQ9DXr2Egkkubl5ptvxeFwsHDhVfz617/kV7+6h27dugNeSSU6OgaLxcKoUWPIz89jwoTzALjqqmt5882/c/PN1/Hcc08xdOjwkKGRt9xyO6dOZXL99Vfx9tv/8ss1CxfexFtvvcnNN1/Hyy//hbvvvq/1Jh0mivA9m5whtEY9+ZJyB4tf+x6zUcuv5g+jb/c43vnyEF9tz2TBhf344oeTgODRm84Nyl4NB7Ukh4r3H0Gb3Avz7N+1eHu/jlhvuyPOGdp+3qdPn6Bz59btSSzLGgQT6juor558h9TkD2eW4HB5+L9rRvr7s141tS+ZeeW8u+4wWo3Cg9eParCBF6ob2/rXQKPFNOVW2b9VIpG0OR3SCp3IKUOrUeiZWnX302k13DF/KP3T4lk0YwC9u8aGvT8hBO7Th7B98RJq3jFMk25EE53UEkOXSCSSBtEhPfkTOWV07RSFXhechRpjMbD4unPC3o9wOXAd2ohr33rUolOgN2MYfRn63uc295AlEomkUXQ4Iy+E4MTpMkb06dTofagVRbj2folz/wZwWtF0OgvjpJvQ9xknF1olEskZRYcz8sXlTsqsLnp2blwndNVajHXN4whbCbpeozEMnY4mta+szieRSM5IOpyRP3HaG53QM7XhRl54XNi+eBHhtGK59FG0nc5q5tFJJBJJ89IkI//SSy/x6afeBIHJkyfzu9/9jgcffJDt27djNnvrrt91111Mmzat6SNtJtJPl6IAaSm1hxyFQgiB/bv/oOYexXTRndLASySSdkGjjfzmzZvZuHEja9asQVEUfvGLX/Dll1+yd+9e3n77bVJSUppznM1GRk45nZMsGA3hl/4VQsW56xPch77DcM4l6HuPacERSiSSjsIbb7wGeJOtWopGG/nk5GQWL16MweDtZ9qnTx+ysrLIysri4YcfJisri2nTpnHXXXehOYPixU/klDGgR3xY2wqh4j6+Hef2D1GLMtGdNQrDqPktOj6JRCJpThpt5Pv16+f/d3p6Op988gnvvPMO27Zt44knnsBisXD77bezevVqrrrqqrD3W1fmVn0kJwfr7E6Xh52H8xg1MBWtRqGozE5RmYMhfZJrbBuI8Lgp37eJku8/xJl7An1SVzrNv4+oQePPyOYfdc0lUumIc4a2nXdurgadrvUdtlDH3L79R/71rzcQQnDqVCZTp15IVFQM3377NUIInnvuRZKSknjvvRV8+ukn2O02FEXhj39cRq9ewdVk58+fzZAhQzl8+BCvvvoG33+/mRUr3kEIlYEDB/Hb3y7mxRf/Qq9evbn88itZu/YD3n33bVau/AC328Vll13CBx98xJo174c8VvX9f/LJf1m79gPi4+OJiYll8OAhgIc//vFxjh3zlla57LIrmT//shrz1mg0Db4GmrzwevjwYW6//XYeeOABevfuzcsvv+x/b+HChaxdu7ZBRr45yxr4ShVce1E/Lhqdxp5jBQB0itaHTA9XbaW4D2/CuedLREUhmoSumKbchq7POGwaDbYCa4PH1dK0dap7W9AR5wxtP29VVYNS7V2HNuE6+G2LHEs/YBL6/hNrTe/3eFR+/nkvb721kri4eObOncadd97LP/7xFk8++Tiff/4Zs2fPZcOGDbz44qsYjSb+8Y9XWb16Fffd97sa+xs7dgKPP/4njh07ytq1H/C3v72B0Wjk1Vdf4q23/s24cRP5+OO1zJt3OT/8sJXS0lJyc/NITz/OkCHDqKiw1Xks3/4PHNjHf//7If/853IUReGXv7yJgQMHs2PHTkpKSvjnP5dTUlLMyy+/wJw582uMU1XVGtdAi5Y12L59O/fccw8PPfQQs2fP5uDBg6SnpzNjxgzAu1ip07VNAM/x7FLWbc9Er9Ow5rtjjBmU6o+sSUupuhP6JBnXoY14Tu4BoaLt3B/DeYvQ9hiOopw5UpNEIqmid+8+pKZ2BiAuLp7Ro71JiKmpnSkrKyUqKprHHvsjX331BSdPZrB162b69RsQcl+DBw8FYMeOH8nMPMntt98EgNvton//gSxYsNDfAerEiRNceOF0du7cwYEDPzNhwnn1Hsu3/59+2s64cROxWCyAt1Sxx+Ohd+8+ZGSc4De/uYtx4yZy5533NNt5arQFzs7O5s477+T5559n/PjxgNeoP/nkk4wbNw6LxcLKlSu59NJLm22w4eJRVf792QFiowzcc8VwnnxrO+99fQSHy0NKghmLyTttd+ZeHFtXohacRIlKwDB8Jrp+E9Emhu7ZKpFIqtD3n4i+/8Q2O351B9JX491HTs5p7r77di6//CrGjZtAYmIShw8fDLkvX+14j0dl6tSLuPfe+wGwWq14PB6MRiN9+/bniy8+pWfPnpx99ii2b9/G7t27uPbaG+o9lm//iqIgRHBdeo/HQ1xcPG+9tYofftjKli2buPHGa/nPf1YRE9N0ea7Rbuobb7yBw+Fg2bJlzJs3j3nz5rFjxw5uu+02FixYwOzZsxk0aBBz5sxp8iAbylc/ZpKRU8610/rTq0ssM8f2YPPe0+w9XkjP1Bg8ucew/u8ZbJ88i3DaME29nagFf8Y49ipp4CWSCOHAgX10757G1Vdfx+DBQ/n++82oqqfOz5x99ii+/XYDRUWFCCH485//xKpV7wAwYcJE/vWvf3D22aM4++xRbNz4LWazifj4+LCPNXr0GDZv3kh5eTkOh4Nvv/0agI0bv+GJJx5mwoTzuPfe32I2W8jNzWmW89BoT37JkiUsWRK6A8p1113X6AE1BVUI9p8oYs13xxjeJ4nRA5IRLgezz0lh58/pKBW5zLRvwbr2AIoxGuP4BegHT5Xt+SSSCGTMmHGsWbOa66+/Er1ez+DBQ/0Lm7XRr19/brrpVu6555cIIejXbwDXX38jAOPHn8ezzy7j7LNHExsbS3x8AuPHn9egY/XrN4Arr1zAL36xiJiYGFJTvZ21xo2byNdfr2PhwqswGAxccMFU+vTp2yznISLqyTsdTvZt/ob9BzOx22wkGV1M6uFBV5KJqCgM2lbVmTCNnIVh6HQUg7k5h94mtPViXFvQEecMbT9vWU++9ZD15Ktx/Ifv6HN4OX00QBSAgsbaGU2X/mgSuqPo9KBosHk0xA4Yi2JqfJimRCKRtCciwsj3nzgV0/gx2Gwu0BlQdEYUnaHGdjVfkUgkksgmIoy8oijEdu6GowM+wkskEkldyCBwiUQSNmfYEl6HorHnXhp5iUQSFjqdgYqKUmno2wAhBBUVpehCyND1ERFyjUQiaXkSEpIpKsqjvLy41Y6p0WhQ1Y4XXRNq3jqdgYSE5AbvSxp5iUQSFlqtjk6durTqMds6bLStaM55S7lGIpFIIhhp5CUSiSSCOePkGo2m8Q2xm/LZ9kxHnHdHnDN0zHl3xDlD+POub7szrqyBRCKRSJoPKddIJBJJBCONvEQikUQw0shLJBJJBCONvEQikUQw0shLJBJJBCONvEQikUQw0shLJBJJBCONvEQikUQw0shLJBJJBBMRRv6///0vF198MdOmTWP58uVtPZwW46WXXmL27NnMnj2bp59+GoDNmzczd+5cpk+fzvPPP9/GI2w5nnrqKRYvXgzA/v37ufzyy5kxYwa///3vcbvdbTy65mf9+vVcdtllzJw5kz/+8Y9Ax/iuP/zwQ/81/tRTTwGR+32Xl5czZ84cMjMzgdq/3ybPX7RzTp8+LaZMmSKKiopERUWFmDt3rjh8+HBbD6vZ2bRpk7j66quFw+EQTqdTLFq0SPz3v/8VkydPFhkZGcLlcombb75ZbNiwoa2H2uxs3rxZjB07VjzwwANCCCFmz54tduzYIYQQ4sEHHxTLly9vw9E1PxkZGeK8884T2dnZwul0igULFogNGzZE/HdttVrFmDFjREFBgXC5XOKKK64QmzZtisjve+fOnWLOnDliyJAh4uTJk8Jms9X6/TZ1/u3ek9+8eTPjxo0jPj4ei8XCjBkz+Oyzz9p6WM1OcnIyixcvxmAwoNfr6dOnD+np6fTs2ZO0tDR0Oh1z586NuLkXFxfz/PPP88tf/hKAU6dOYbfbGTlyJACXXXZZxM35yy+/5OKLL6Zz587o9Xqef/55zGZzxH/XHo8HVVWx2Wy43W7cbjc6nS4iv+9Vq1bx6KOPkpKSAsDu3btDfr/Ncb2fcVUoG0pubi7JyVXdUlJSUti9e3cbjqhl6Nevn//f6enpfPLJJyxcuLDG3HNyctpieC3GI488wn333Ud2djZQ8/tOTk6OuDmfOHECvV7PLbfcQl5eHlOmTKFfv34R/11HR0fz61//mlmzZmEymTj33HPR6/UR+X0vXbo06O9QdiwnJ6dZrvd278mLEEU0FSVyS5MePnyYm2++mQceeIAePXrUeD+S5v7ee+/RpUsXxo8f73+tI3zfHo+HLVu28Mwzz7Bq1Sr27Nnj120DibR5HzhwgPfff5+vv/6ajRs3otFo2LRpU43tIm3eUPt13RzXe7v35FNTU/nxxx/9f+fm5vofgSKN7du3c8899/DQQw8xe/Zstm3bRn5+vv/9SJv7J598Ql5eHvPmzaOkpASr1YqiKEFzzsvLi6g5A3Tq1Inx48eTmJgIwIUXXshnn32GVqv1bxNp3zXAxo0bGT9+PElJSYBXmnjjjTci/vsGrx0L9Vuu/npj5t/uPfkJEyawZcsWCgsLsdlsfPHFF0yaNKmth9XsZGdnc+edd/Lss88ye/ZsAEaMGMHx48c5ceIEHo+Hjz/+OKLm/uabb/Lxxx/z4Ycfcs899zB16lT+9Kc/YTQa2b59OwBr166NqDkDTJkyhY0bN1JaWorH4+G7775j5syZEf1dAwwcOJDNmzdjtVoRQrB+/XrOPffciP++ofbfcrdu3Zo8/4jw5O+77z4WLVqEy+XiiiuuYPjw4W09rGbnjTfewOFwsGzZMv9r11xzDcuWLePuu+/G4XAwefJkZs6c2YajbB2effZZlixZQkVFBYMHD2bRokVtPaRmZcSIEfziF7/g2muvxeVyMXHiRBYsWEDv3r0j+rs+77zz2LdvH5dddhl6vZ5hw4Zx2223MW3atIj+vgGMRmOtv+WmXu+yM5REIpFEMO1erpFIJBJJ7UgjL5FIJBGMNPISiUQSwUgjL5FIJBGMNPISiUQSwUgjL5GE4NZbb+XIkSMN+sztt9/OBx980EIjkkgaR7uPk5dIWoLXX3+9rYcgkTQL0shLIor169fzt7/9DZfLhclk4oEHHmDjxo0cPnyY/Px8CgoKGDhwIEuXLiU6Opp33nmHFStWoNfrMRqNPPHEE/Tt25epU6fywgsvMGzYMFauXMlbb72FRqOhU6dOPPzww/Tq1YucnBwWL15Mbm4uXbt2paCgwD+Oo0ePsnTpUoqLi/F4PCxcuJArrriCiooKHnzwQU6cOIFGo2HIkCE88cQTaDTyoVrSQjRbgWSJpI05fvy4mDNnjigsLBRCCHHo0CExceJEsWzZMjFp0iSRl5cnPB6P+M1vfiOWLVsm3G63GDJkiMjJyRFCCLFmzRqxYsUKIYQQU6ZMEbt37xabN28WF110kSgoKBBCCPH++++LWbNmCVVVxa9+9Svx/PPPCyGESE9PFyNHjhTvv/++cLlc4uKLLxZ79+4VQghRWloqZs2aJXbs2CHWrFkjbr75ZiGEEG63W/z+978X6enprXmaJB0M6clLIoZNmzaRm5vLjTfe6H9NURQyMjKYOXMmnTp1AuCKK67gySef5IEHHmDmzJlcc801XHDBBUycOJG5c+cG7fO7777j4osv9hcLu+yyy1i6dCmZmZls3ryZBx54AICePXsyduxYwFsKOiMjg4ceesi/H7vdzr59+zj//PN5/vnnWbhwIRMmTOCGG26gZ8+eLXlaJB0caeQlEYOqqowfP56//OUv/teys7NZuXIlTqczaDufPPLss89y6NAhNm/ezOuvv87q1av529/+5t9WhKj6IYTA7XbXKAWr03l/Th6Ph9jYWD788EP/e/n5+cTExGA0Gvnyyy/ZunUr33//PTfddBNLliyJuDo0kjMHKQRKIoZx48axadMmjh49CsA333zDJZdcgsPhYN26dZSVlaGqKqtWrWLKlCkUFhYyefJk4uPjufHGG7n33ns5ePBg0D7PO+88PvnkEwoLCwF4//33iY+Pp2fPnpx//vmsXLkSgKysLLZu3QpAr169MBqNfiOfnZ3NnDlz2Lt3L++88w4PPvgg5513Hvfffz/nnXcehw8fbq1TJOmAyAJlkoji008/5dVXX0UIgU6n46GHHmLLli18//33eDweioqKGDNmDEuWLMFkMrFixQr+85//YDKZ0Gq13HfffUyYMCFo4XX58uWsWLECVVVJTEzkkUceoV+/fhQWFvLggw+SkZFB586dcbvdXHrppVx22WUcOHDAv/DqdrtZtGgRCxYswGq18tBDD3Hw4EHMZjNdu3Zl6dKlxMXFtfWpk0Qo0shLIp4XX3yRoqIiHnnkkbYeikTS6ki5RiKRSCIY6clLJBJJBCM9eYlEIolgpJGXSCSSCEYaeYlEIolgpJGXSCSSCEYaeYlEIolgpJGXSCSSCOb/AW9dunIlhGDfAAAAAElFTkSuQmCC", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "import easydict\n", - "from common.multiprocessing_env import SubprocVecEnv\n", - "cfg = easydict.EasyDict({\n", - " \"algo_name\": 'A2C',\n", - " \"env_name\": 'CartPole-v0',\n", - " \"n_envs\": 8,\n", - " \"max_steps\": 20000,\n", - " \"n_steps\":5,\n", - " \"gamma\":0.99,\n", - " \"lr\": 1e-3,\n", - " \"hidden_dim\": 256,\n", - " \"device\":torch.device(\n", - " \"cuda\" if torch.cuda.is_available() else \"cpu\")\n", - "})\n", - "envs = [make_envs(cfg.env_name) for i in range(cfg.n_envs)]\n", - "envs = SubprocVecEnv(envs) \n", - "rewards,ma_rewards = train(cfg,envs)\n", - "plot_rewards(rewards, ma_rewards, cfg, tag=\"train\") # 画出结果" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.12 ('rl_tutorials')", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.12" - }, - "orig_nbformat": 4, - "vscode": { - "interpreter": { - "hash": "4f613f1ab80ec98dc1b91d6e720de51301598a187317378e53e49b773c1123dd" - } - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/projects/notebooks/DDPG.ipynb b/projects/notebooks/DDPG.ipynb deleted file mode 100644 index 5194644..0000000 --- a/projects/notebooks/DDPG.ipynb +++ /dev/null @@ -1,559 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1. 定义算法" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 1.1. 定义模型\n", - "\n", - "注意DDGP中critic网络的输入是state加上action。" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": {}, - "outputs": [], - "source": [ - "import torch\n", - "import torch.nn as nn\n", - "import torch.nn.functional as F\n", - "class Actor(nn.Module):\n", - " def __init__(self, n_states, n_actions, hidden_dim = 256, init_w=3e-3):\n", - " super(Actor, self).__init__() \n", - " self.linear1 = nn.Linear(n_states, hidden_dim)\n", - " self.linear2 = nn.Linear(hidden_dim, hidden_dim)\n", - " self.linear3 = nn.Linear(hidden_dim, n_actions)\n", - " \n", - " self.linear3.weight.data.uniform_(-init_w, init_w)\n", - " self.linear3.bias.data.uniform_(-init_w, init_w)\n", - " \n", - " def forward(self, x):\n", - " x = F.relu(self.linear1(x))\n", - " x = F.relu(self.linear2(x))\n", - " x = torch.tanh(self.linear3(x))\n", - " return x\n", - " \n", - "class Critic(nn.Module):\n", - " def __init__(self, n_states, n_actions, hidden_dim=256, init_w=3e-3):\n", - " super(Critic, self).__init__()\n", - " \n", - " self.linear1 = nn.Linear(n_states + n_actions, hidden_dim)\n", - " self.linear2 = nn.Linear(hidden_dim, hidden_dim)\n", - " self.linear3 = nn.Linear(hidden_dim, 1)\n", - " # 随机初始化为较小的值\n", - " self.linear3.weight.data.uniform_(-init_w, init_w)\n", - " self.linear3.bias.data.uniform_(-init_w, init_w)\n", - " \n", - " def forward(self, state, action):\n", - " # 按维数1拼接\n", - " x = torch.cat([state, action], 1)\n", - " x = F.relu(self.linear1(x))\n", - " x = F.relu(self.linear2(x))\n", - " x = self.linear3(x)\n", - " return x" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 1.2 定义经验回放" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": {}, - "outputs": [], - "source": [ - "from collections import deque\n", - "import random\n", - "class ReplayBuffer:\n", - " def __init__(self, capacity: int) -> None:\n", - " self.capacity = capacity\n", - " self.buffer = deque(maxlen=self.capacity)\n", - " def push(self,transitions):\n", - " '''_summary_\n", - " Args:\n", - " trainsitions (tuple): _description_\n", - " '''\n", - " self.buffer.append(transitions)\n", - " def sample(self, batch_size: int, sequential: bool = False):\n", - " if batch_size > len(self.buffer):\n", - " batch_size = len(self.buffer)\n", - " if sequential: # sequential sampling\n", - " rand = random.randint(0, len(self.buffer) - batch_size)\n", - " batch = [self.buffer[i] for i in range(rand, rand + batch_size)]\n", - " return zip(*batch)\n", - " else:\n", - " batch = random.sample(self.buffer, batch_size)\n", - " return zip(*batch)\n", - " def clear(self):\n", - " self.buffer.clear()\n", - " def __len__(self):\n", - " return len(self.buffer)" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": {}, - "outputs": [], - "source": [ - "import torch.optim as optim\n", - "import numpy as np\n", - "class DDPG:\n", - " def __init__(self, models,memories,cfg):\n", - " self.device = torch.device(cfg['device'])\n", - " self.critic = models['critic'].to(self.device)\n", - " self.target_critic = models['critic'].to(self.device)\n", - " self.actor = models['actor'].to(self.device)\n", - " self.target_actor = models['actor'].to(self.device)\n", - " \n", - " # 复制参数到目标网络\n", - " for target_param, param in zip(self.target_critic.parameters(), self.critic.parameters()):\n", - " target_param.data.copy_(param.data)\n", - " for target_param, param in zip(self.target_actor.parameters(), self.actor.parameters()):\n", - " target_param.data.copy_(param.data)\n", - " self.critic_optimizer = optim.Adam(\n", - " self.critic.parameters(), lr=cfg['critic_lr'])\n", - " self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=cfg['actor_lr'])\n", - " self.memory = memories['memory']\n", - " self.batch_size = cfg['batch_size']\n", - " self.gamma = cfg['gamma']\n", - " self.tau = cfg['tau'] # 软更新参数\n", - " def sample_action(self, state):\n", - " state = torch.FloatTensor(state).unsqueeze(0).to(self.device)\n", - " action = self.actor(state)\n", - " return action.detach().cpu().numpy()[0, 0]\n", - " @torch.no_grad()\n", - " def predict_action(self, state):\n", - " ''' 用于预测，不需要计算梯度\n", - " '''\n", - " state = torch.FloatTensor(state).unsqueeze(0).to(self.device)\n", - " action = self.actor(state)\n", - " return action.cpu().numpy()[0, 0]\n", - " def update(self):\n", - " if len(self.memory) < self.batch_size: # 当memory中不满足一个批量时，不更新策略\n", - " return\n", - " # 从经验回放中中随机采样一个批量的transition\n", - " state, action, reward, next_state, done = self.memory.sample(self.batch_size)\n", - " # 转变为张量\n", - " state = torch.FloatTensor(np.array(state)).to(self.device)\n", - " next_state = torch.FloatTensor(np.array(next_state)).to(self.device)\n", - " action = torch.FloatTensor(np.array(action)).to(self.device)\n", - " reward = torch.FloatTensor(reward).unsqueeze(1).to(self.device)\n", - " done = torch.FloatTensor(np.float32(done)).unsqueeze(1).to(self.device)\n", - " # 注意看伪代码，这里的actor损失就是对应策略即actor输出的action下对应critic值的负均值\n", - " actor_loss = self.critic(state, self.actor(state))\n", - " actor_loss = - actor_loss.mean()\n", - "\n", - " next_action = self.target_actor(next_state)\n", - " target_value = self.target_critic(next_state, next_action.detach())\n", - " # 这里的expected_value就是伪代码中间的y_i \n", - " expected_value = reward + (1.0 - done) * self.gamma * target_value\n", - " expected_value = torch.clamp(expected_value, -np.inf, np.inf)\n", - "\n", - " actual_value = self.critic(state, action)\n", - " critic_loss = nn.MSELoss()(actual_value, expected_value.detach())\n", - " \n", - " self.actor_optimizer.zero_grad()\n", - " actor_loss.backward()\n", - " self.actor_optimizer.step()\n", - " self.critic_optimizer.zero_grad()\n", - " critic_loss.backward()\n", - " self.critic_optimizer.step()\n", - " # 各自目标网络的参数软更新\n", - " for target_param, param in zip(self.target_critic.parameters(), self.critic.parameters()):\n", - " target_param.data.copy_(\n", - " target_param.data * (1.0 - self.tau) +\n", - " param.data * self.tau\n", - " )\n", - " for target_param, param in zip(self.target_actor.parameters(), self.actor.parameters()):\n", - " target_param.data.copy_(\n", - " target_param.data * (1.0 - self.tau) +\n", - " param.data * self.tau\n", - " )" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2. 定义训练\n", - "\n", - "注意测试函数中不需要动作噪声" - ] - }, - { - "cell_type": "code", - "execution_count": 25, - "metadata": {}, - "outputs": [], - "source": [ - "class OUNoise(object):\n", - " '''Ornstein–Uhlenbeck噪声\n", - " '''\n", - " def __init__(self, action_space, mu=0.0, theta=0.15, max_sigma=0.3, min_sigma=0.3, decay_period=100000):\n", - " self.mu = mu # OU噪声的参数\n", - " self.theta = theta # OU噪声的参数\n", - " self.sigma = max_sigma # OU噪声的参数\n", - " self.max_sigma = max_sigma\n", - " self.min_sigma = min_sigma\n", - " self.decay_period = decay_period\n", - " self.n_actions = action_space.shape[0]\n", - " self.low = action_space.low\n", - " self.high = action_space.high\n", - " self.reset()\n", - " def reset(self):\n", - " self.obs = np.ones(self.n_actions) * self.mu\n", - " def evolve_obs(self):\n", - " x = self.obs\n", - " dx = self.theta * (self.mu - x) + self.sigma * np.random.randn(self.n_actions)\n", - " self.obs = x + dx\n", - " return self.obs\n", - " def get_action(self, action, t=0):\n", - " ou_obs = self.evolve_obs()\n", - " self.sigma = self.max_sigma - (self.max_sigma - self.min_sigma) * min(1.0, t / self.decay_period) # sigma会逐渐衰减\n", - " return np.clip(action + ou_obs, self.low, self.high) # 动作加上噪声后进行剪切\n", - "\n", - "def train(cfg, env, agent):\n", - " print(\"开始训练！\")\n", - " ou_noise = OUNoise(env.action_space) # 动作噪声\n", - " rewards = [] # 记录所有回合的奖励\n", - " for i_ep in range(cfg['train_eps']):\n", - " state = env.reset()\n", - " ou_noise.reset()\n", - " ep_reward = 0\n", - " for i_step in range(cfg['max_steps']):\n", - " action = agent.sample_action(state)\n", - " action = ou_noise.get_action(action, i_step+1) \n", - " next_state, reward, done, _ = env.step(action)\n", - " ep_reward += reward\n", - " agent.memory.push((state, action, reward, next_state, done))\n", - " agent.update()\n", - " state = next_state\n", - " if done:\n", - " break\n", - " if (i_ep+1)%10 == 0:\n", - " print(f\"回合：{i_ep+1}/{cfg['train_eps']}，奖励：{ep_reward:.2f}\")\n", - " rewards.append(ep_reward)\n", - " print(\"完成训练！\")\n", - " return {'rewards':rewards}\n", - "def test(cfg, env, agent):\n", - " print(\"开始测试！\")\n", - " rewards = [] # 记录所有回合的奖励\n", - " for i_ep in range(cfg['test_eps']):\n", - " state = env.reset() \n", - " ep_reward = 0\n", - " for i_step in range(cfg['max_steps']):\n", - " action = agent.predict_action(state)\n", - " next_state, reward, done, _ = env.step(action)\n", - " ep_reward += reward\n", - " state = next_state\n", - " if done:\n", - " break\n", - " rewards.append(ep_reward)\n", - " print(f\"回合：{i_ep+1}/{cfg['test_eps']}，奖励：{ep_reward:.2f}\")\n", - " print(\"完成测试！\")\n", - " return {'rewards':rewards}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 3. 定义环境" - ] - }, - { - "cell_type": "code", - "execution_count": 26, - "metadata": {}, - "outputs": [], - "source": [ - "import gym\n", - "import os\n", - "import torch\n", - "import numpy as np\n", - "import random\n", - "class NormalizedActions(gym.ActionWrapper):\n", - " ''' 将action范围重定在[0.1]之间\n", - " '''\n", - " def action(self, action):\n", - " low_bound = self.action_space.low\n", - " upper_bound = self.action_space.high\n", - " action = low_bound + (action + 1.0) * 0.5 * (upper_bound - low_bound)\n", - " action = np.clip(action, low_bound, upper_bound)\n", - " return action\n", - "\n", - " def reverse_action(self, action):\n", - " low_bound = self.action_space.low\n", - " upper_bound = self.action_space.high\n", - " action = 2 * (action - low_bound) / (upper_bound - low_bound) - 1\n", - " action = np.clip(action, low_bound, upper_bound)\n", - " return action\n", - "def all_seed(env,seed = 1):\n", - " ''' 万能的seed函数\n", - " '''\n", - " env.seed(seed) # env config\n", - " np.random.seed(seed)\n", - " random.seed(seed)\n", - " torch.manual_seed(seed) # config for CPU\n", - " torch.cuda.manual_seed(seed) # config for GPU\n", - " os.environ['PYTHONHASHSEED'] = str(seed) # config for python scripts\n", - " # config for cudnn\n", - " torch.backends.cudnn.deterministic = True\n", - " torch.backends.cudnn.benchmark = False\n", - " torch.backends.cudnn.enabled = False\n", - "def env_agent_config(cfg):\n", - " env = NormalizedActions(gym.make(cfg['env_name'])) # 装饰action噪声\n", - " if cfg['seed'] !=0:\n", - " all_seed(env,seed=cfg['seed'])\n", - " n_states = env.observation_space.shape[0]\n", - " n_actions = env.action_space.shape[0]\n", - " cfg.update({\"n_states\":n_states,\"n_actions\":n_actions}) # 更新n_states和n_actions到cfg参数中\n", - " models = {\"actor\":Actor(n_states,n_actions,hidden_dim=cfg['actor_hidden_dim']),\"critic\":Critic(n_states,n_actions,hidden_dim=cfg['critic_hidden_dim'])}\n", - " memories = {\"memory\":ReplayBuffer(cfg['memory_capacity'])}\n", - " agent = DDPG(models,memories,cfg)\n", - " return env,agent" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 4. 设置参数" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": {}, - "outputs": [], - "source": [ - "import argparse\n", - "import matplotlib.pyplot as plt\n", - "import seaborn as sns\n", - "def get_args():\n", - " \"\"\" 超参数\n", - " \"\"\"\n", - " parser = argparse.ArgumentParser(description=\"hyperparameters\") \n", - " parser.add_argument('--algo_name',default='DDPG',type=str,help=\"name of algorithm\")\n", - " parser.add_argument('--env_name',default='Pendulum-v1',type=str,help=\"name of environment\")\n", - " parser.add_argument('--train_eps',default=300,type=int,help=\"episodes of training\")\n", - " parser.add_argument('--test_eps',default=20,type=int,help=\"episodes of testing\")\n", - " parser.add_argument('--max_steps',default=100000,type=int,help=\"steps per episode, much larger value can simulate infinite steps\")\n", - " parser.add_argument('--gamma',default=0.99,type=float,help=\"discounted factor\")\n", - " parser.add_argument('--critic_lr',default=1e-3,type=float,help=\"learning rate of critic\")\n", - " parser.add_argument('--actor_lr',default=1e-4,type=float,help=\"learning rate of actor\")\n", - " parser.add_argument('--memory_capacity',default=8000,type=int,help=\"memory capacity\")\n", - " parser.add_argument('--batch_size',default=128,type=int)\n", - " parser.add_argument('--target_update',default=2,type=int)\n", - " parser.add_argument('--tau',default=1e-2,type=float)\n", - " parser.add_argument('--critic_hidden_dim',default=256,type=int)\n", - " parser.add_argument('--actor_hidden_dim',default=256,type=int)\n", - " parser.add_argument('--device',default='cpu',type=str,help=\"cpu or cuda\") \n", - " parser.add_argument('--seed',default=1,type=int,help=\"random seed\")\n", - " args = parser.parse_args([]) \n", - " args = {**vars(args)} # 将args转换为字典 \n", - " # 打印参数\n", - " print(\"训练参数如下：\")\n", - " print(''.join(['=']*80))\n", - " tplt = \"{:^20}\\t{:^20}\\t{:^20}\"\n", - " print(tplt.format(\"参数名\",\"参数值\",\"参数类型\"))\n", - " for k,v in args.items():\n", - " print(tplt.format(k,v,str(type(v)))) \n", - " print(''.join(['=']*80)) \n", - " return args\n", - "def smooth(data, weight=0.9): \n", - " '''用于平滑曲线，类似于Tensorboard中的smooth\n", - "\n", - " Args:\n", - " data (List):输入数据\n", - " weight (Float): 平滑权重，处于0-1之间，数值越高说明越平滑，一般取0.9\n", - "\n", - " Returns:\n", - " smoothed (List): 平滑后的数据\n", - " '''\n", - " last = data[0] # First value in the plot (first timestep)\n", - " smoothed = list()\n", - " for point in data:\n", - " smoothed_val = last * weight + (1 - weight) * point # 计算平滑值\n", - " smoothed.append(smoothed_val) \n", - " last = smoothed_val \n", - " return smoothed\n", - "\n", - "def plot_rewards(rewards,cfg,path=None,tag='train'):\n", - " sns.set()\n", - " plt.figure() # 创建一个图形实例，方便同时多画几个图\n", - " plt.title(f\"{tag}ing curve on {cfg['device']} of {cfg['algo_name']} for {cfg['env_name']}\")\n", - " plt.xlabel('epsiodes')\n", - " plt.plot(rewards, label='rewards')\n", - " plt.plot(smooth(rewards), label='smoothed')\n", - " plt.legend()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 5. 我准备好了！" - ] - }, - { - "cell_type": "code", - "execution_count": 28, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "训练参数如下：\n", - "================================================================================\n", - " 参数名 \t 参数值 \t 参数类型 \n", - " algo_name \t DDPG \t \n", - " env_name \t Pendulum-v1 \t \n", - " train_eps \t 300 \t \n", - " test_eps \t 20 \t \n", - " max_steps \t 100000 \t \n", - " gamma \t 0.99 \t \n", - " critic_lr \t 0.001 \t \n", - " actor_lr \t 0.0001 \t \n", - " memory_capacity \t 8000 \t \n", - " batch_size \t 128 \t \n", - " target_update \t 2 \t \n", - " tau \t 0.01 \t \n", - " critic_hidden_dim \t 256 \t \n", - " actor_hidden_dim \t 256 \t \n", - " device \t cpu \t \n", - " seed \t 1 \t \n", - "================================================================================\n", - "开始训练！\n", - "回合：10/300，奖励：-1549.57\n", - "回合：20/300，奖励：-1515.84\n", - "回合：30/300，奖励：-1413.30\n", - "回合：40/300，奖励：-972.99\n", - "回合：50/300，奖励：-829.94\n", - "回合：60/300，奖励：-727.91\n", - "回合：70/300，奖励：-954.71\n", - "回合：80/300，奖励：-1318.39\n", - "回合：90/300，奖励：-981.19\n", - "回合：100/300，奖励：-1262.05\n", - "回合：110/300，奖励：-640.49\n", - "回合：120/300，奖励：-1100.00\n", - "回合：130/300，奖励：-764.66\n", - "回合：140/300，奖励：-352.27\n", - "回合：150/300，奖励：-891.03\n", - "回合：160/300，奖励：-1318.07\n", - "回合：170/300，奖励：-124.30\n", - "回合：180/300，奖励：-240.08\n", - "回合：190/300，奖励：-491.77\n", - "回合：200/300，奖励：-1000.77\n", - "回合：210/300，奖励：-128.87\n", - "回合：220/300，奖励：-950.32\n", - "回合：230/300，奖励：-122.48\n", - "回合：240/300，奖励：-246.52\n", - "回合：250/300，奖励：-374.37\n", - "回合：260/300，奖励：-368.25\n", - "回合：270/300，奖励：-364.17\n", - "回合：280/300，奖励：-725.39\n", - "回合：290/300，奖励：-131.21\n", - "回合：300/300，奖励：-610.10\n", - "完成训练！\n", - "开始测试！\n", - "回合：1/20，奖励：-116.05\n", - "回合：2/20，奖励：-126.18\n", - "回合：3/20，奖励：-231.46\n", - "回合：4/20，奖励：-246.40\n", - "回合：5/20，奖励：-304.69\n", - "回合：6/20，奖励：-124.40\n", - "回合：7/20，奖励：-1.06\n", - "回合：8/20，奖励：-114.20\n", - "回合：9/20，奖励：-348.97\n", - "回合：10/20，奖励：-116.11\n", - "回合：11/20，奖励：-117.20\n", - "回合：12/20，奖励：-118.66\n", - "回合：13/20，奖励：-235.18\n", - "回合：14/20，奖励：-356.14\n", - "回合：15/20，奖励：-118.39\n", - "回合：16/20，奖励：-351.94\n", - "回合：17/20，奖励：-114.51\n", - "回合：18/20，奖励：-124.78\n", - "回合：19/20，奖励：-226.47\n", - "回合：20/20，奖励：-121.49\n", - "完成测试！\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAAHJCAYAAACBuOOtAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOxdd5gdVfl+p9y2vSTZTe89hCQkgdCbgHRQQelN9KeIQhARkCIgUlQMGEF6k141gEiVkk5CCKGlZ5Ns77u3Tfn9MffMnHNm5ra9d3ezzPs8ebJ3ypkzZ8p55/ve7/sEXdd1ePDgwYMHDx48DGCIfd0BDx48ePDgwYOHfMMjPB48ePDgwYOHAQ+P8Hjw4MGDBw8eBjw8wuPBgwcPHjx4GPDwCI8HDx48ePDgYcDDIzwePHjw4MGDhwEPj/B48ODBgwcPHgY8PMLjwYMHDx48eBjw8AiPh4zg5an04MGDBw97IjzC4yFtvP322/jNb36Tk7ZefPFFTJ48GTU1NXndx8O3C4888ggOOOAAzJw5E4sXL3bcZvLkycy/adOmYd9998UFF1yAd999N+fbT548GTNnzsRxxx2H+++/H5qm2fZZt24dfvvb3+KII47AzJkzMX/+fJx77rn497//ndZ533HHHZg/fz5mzZqFl19+Oa190sXy5ctt5zNlyhTMmTMHP/zhD/HOO+/k9Hjp9GX58uV53aev8cc//hFnn312X3djwEHu6w542HPwyCOP5KytQw89FM888wyGDBmS1308fHvQ2dmJ2267DYceeiguuOACjBgxwnXb73//+/jBD34AAIjH42hoaMALL7yAn/70p7jmmmtwzjnn5Gx7AAiHw3jzzTdx5513or29HQsXLjTXPfLII7j99tsxd+5cXHLJJRgxYgTa2trw5ptv4oorrsDatWtx7bXXup7L119/jQceeACnnXYaTjrpJIwbNy79QcsA1113HaZPnw7AsPS2tbXhoYcews9+9jPcd999OOSQQ/Jy3G8bHnroITz88MOYP39+X3dlwMEjPB76BBUVFaioqMj7Ph6+PWhra4OmaTjyyCMxb968pNtWV1dj1qxZzLJjjz0Wv/jFL3D77bfj8MMPZwhTLrZfsGABNm/ejCeffBKXXnopfD4fli9fjj/+8Y8466yzbKTmyCOPxJQpU3Dbbbfh+OOPt7VH0NraCgA47rjjMHfu3KTn3RNMmDDB1oe5c+fi0EMPxWOPPeYRnh5ix44duO222/DOO++guLi4r7szIOG5tDykhbPPPhsrVqzAihUrTPMwMRU//fTTOOywwzBnzhx89NFHAIDnnnsOp556KmbNmoWZM2fipJNOwuuvv262x7unrrrqKpx33nl44YUXcPTRR2PGjBk46aST8L///a9H+wDAmjVrcOaZZ2LWrFk49NBD8eijj+K8887DVVddlfSc165diwsuuABz5szBfvvth8svvxx1dXWOfSE4/PDDmXYnT56Me+65B6eeeipmzpyJe+65B1OnTsUTTzzB7Nfc3Izp06ebVjRN0/CPf/wD3/nOdzBjxgwcffTRePzxx1Nep46ODtx666048sgjsddee+H444/H888/b+vjokWLcNttt2H//ffHzJkzceGFF2Lr1q1J247FYrjrrrtMt8vxxx+Pl156yVx/9tln46qrrsK9996L/fffH/vssw9+9rOfYefOneY2V111FQ4//HCm3ZqaGkyePBkvvvhi0uN/9NFHOOOMM7DPPvtg3333xcKFC7F7924AxvUg7V599dWYPHlyyrFywmWXXYZ4PG4bs1xtP2PGDHR1daGtrQ0A8Le//Q1Dhw7Fr3/9a8ftzznnHBxxxBEIh8OO6++++27T9XHuueeaY6CqKp588kmccMIJmDlzJg499FDceeediEaj5r5XXXUVzj33XFx//fWYM2cOjj32WKiqmtZ5EBQVFWHs2LHYtWuXuay1tRXXXXcd9t9/f+y111447bTTsHTpUma/yZMn48knn8Q111yD+fPnY/bs2fjlL3+JxsZGZrunn34aRx99NGbOnImzzjqLOQ45f6drPXnyZNx9992OfU7nHiTvtqVLl+Lss882x/C5555DfX09LrnkEsyePRuHHHJISsv3vffeixkzZpjXnOCRRx7B9OnT0dTUBAC49dZbsW3bNjz66KOYOnVq0jY9ZAeP8HhIC9dffz2mTZuGadOm4ZlnnjFN2wBwzz334De/+Q2uu+46zJ49G08++SSuu+46HHnkkbjvvvtw5513wu/344orrkBtba3rMdavX48HH3wQl156Kf72t79BkiT84he/sL0oMtln06ZNOO+88wAAf/7zn/GLX/wC//jHP7B69eqk57thwwacddZZiEajuP3223HjjTdi/fr1uPDCC6EoSgYjZ7zwTjjhBCxatAhHH3005s+fjyVLljDbvPHGG9B1HccddxwA4IYbbsCiRYtw4okn4t5778UxxxyDP/zhD/jb3/7mepxIJIIzzjgD//rXv3DRRRdh8eLF2GeffXDNNdfg3nvvZbZ97LHHsHnzZtx66624+eabsX79+pT6rCuuuAIPP/wwfvCDH+C+++7DgQceiKuuuorRmbz99tt48cUXce211+LGG2/EF198gbPPPtt1wk4XL7/8Mi644AIMHToUf/7zn/Hb3/4Wa9aswemnn46mpiYceuihuOeeewAA//d//4dnnnkmq+OMGzcOw4YNS3l/ZLv9li1bUFhYiMrKSrS1tWHlypU44ogjEAgEHLeXZRmLFy/GggULHNf/4Ac/wHXXXQfAcDmRMbjuuutM4vv3v/8dZ555Jp544gn87Gc/YwIPVq1ahd27d+Nvf/sbFi5cCEmS0joPglgshpqaGowaNQoAEI1Gce655+Ltt9/GZZddhnvuuQfV1dW46KKLbKTnL3/5CzRNw5///GdceeWVePfdd/GHP/zBXP/EE0/g+uuvxyGHHILFixdj7733xu9+97uM+tdTXH755Tj88MNx3333YezYsbj++utxzjnnYOLEiVi8eDFmzpyJW2+9FevWrXNt44QTToCiKHjzzTeZ5UuWLMGBBx6IyspKAMCvfvUrvPrqqymtkx6yh+fS8pAWJkyYgKKiIgCwmbXPOOMMHHPMMebvHTt24MILL8TPfvYzc9nw4cNx6qmnYvXq1eakzqOjowMvvvii+fIsKCjAWWedhWXLluHoo4/Oap/77rsPxcXFeOCBBxAKhQAYk9QPf/jDpOd77733oqysDA899JA5GQ0ZMgQLFy7EN998k3RfHnPnzsX5559v/j7ppJNw9dVXY9euXRg2bBgA4+W3//77Y/DgwdiyZQueffZZXH755bj44osBAAceeCAEQcB9992HM844A+Xl5bbjvPjii/j666/x9NNPY/bs2QCAgw46CIqiYPHixfjhD3+IsrIyAEBJSQkWL15sTnDbt2/H3XffjZaWFse2v/76a/znP//B1VdfjXPPPReA4aLZuXMnli9fjuOPPx6AoVV58cUXMXLkSADGWJ9yyil4+eWX8aMf/SijcSPQNA133nknDjzwQPzpT38ylxOrxIMPPogrr7zS/CoeNWqUq/snHQwaNMhmach0e03TTGKs6zoaGxvxr3/9C++88w4uuugiCIKAnTt3QtM0jB07ltlX13WbpUUQBEcyUl1djQkTJgAwntFp06Zh48aNeP7557Fw4ULz/jnggAMwZMgQXHnllfjf//5nup8URcHvf/97VFdXpzxP+pwURcHOnTuxePFiNDc348wzzwQAvPLKK/jyyy/x7LPPYu+99wYAHHzwwTj77LNx55134oUXXjDbmzRpEm699Vbz97p16/DGG2+YY7B48WIce+yxuPrqqwEYz0BnZyeefvrplH3NFb73ve+Zz25BQQFOO+00zJw5E7/85S8BAFOmTMGbb76JTz75BDNnznRsY/jw4Zg3bx7+/e9/m7qu7du3Y926dfjLX/5ibjdp0qQ8n40Hz8Ljocfgza9XXXUVrrjiCrS3t2Pt2rV45ZVX8OSTTwIwvgjdUFFRYRIXAOZLOJl1INU+y5Ytw8EHH2ySHQCYPXs2hg8fnvScVq9ejYMPPpj58p49ezbeeeedjM3N/PZHHXUUAoEAXnvtNQDA7t27sXr1apx00klmn3Vdx+GHHw5FUcx/hx9+OKLRqKs1YcWKFRg+fLhJdghOPPFERKNRfPrpp+ayvfbai5lAU401OeZRRx3FLL/77rtx0003mb/nzJljkh0AmDZtGkaOHImVK1c6tpsOtmzZgoaGBpNUEYwaNQqzZ8/GihUrsm7bCbquQxCEHm2/ePFiTJ8+HdOnT8eMGTNw6KGH4m9/+xtOP/10/OIXvwAAx2gtAFi6dKm5L/lHrJTpgIwH/2Fx3HHHQZIkJlqprKwsLbIDAOedd57Zn7333hvHHnssli5dimuvvRYHH3yw2ffBgwdj+vTp5n2rqioOO+wwrF+/nrHW8qS0urravP82b96MpqYmHHbYYcw23/3ud9MbhByBfpaIJYYQOQDmx0FHRwcAw5VIP7PkGp944olYuXIlGhoaABgfOEVFRTbXmof8wrPweOgxCgoKmN/bt2/Hddddh6VLl8Ln82HcuHGYMmUKgOR5fGhSAsCcRNwmhnT2aW5uNl9UNAYNGuTaJmDoEJz2ywb8+BQVFeHII4/EkiVLcNFFF+G1115DKBTCkUceaR4bsE9YBERHxKOtrQ2DBw+2LSfn2t7ebi7jx00UjW8ft7EmfUo1JlVVVbZlxH2TLcixna7ZoEGDsGHDhqzbdkJtbW1GX9tO25922mk47bTTABj3ZGFhIUaMGAGfz2duQ6x7vA5s5syZjCbo+uuvz6j/ZKz5e0GWZZSXl5uTMwAUFham3e6NN95ourIlSUJpaSmGDRvGkL3W1lY0NDQwLm8aDQ0NKC0tBeB8D5L3AzkH3trodH/nE8SqTYPvN43vfOc7jGbtlFNOwR//+Eccc8wxuOmmm/D666/jnHPOwZIlS3D00UcjGAzmpd8enOERHg85haZpuPjii+Hz+fD8889j6tSpkGUZGzduxCuvvNLr/amurnZ0TzQ1NSUN3y0uLkZzc7Nt+fvvv4+pU6e6krGurq60+nXiiSfi4osvxrZt28yXH3mRlpSUAAAeffRRxwmJTJQ8SktLsW3bNtty8lXp5KpKF6RPzc3NjEVg06ZNaG1txT777AMAaGlpse3b2NhoWuEEQbC5a7q7u5Mem7jhnK5jQ0NDj86Lx8aNG9HQ0GC6aLLdfsiQIdhrr72S7ltRUYHZs2fjrbfewhVXXGFa3IqKiph9MyElAExC0dDQwFgy4/G4q8syHYwdOzblORUXF2PMmDG48847HdcnSxVAg/SRCHoJCPklIM+hqqrm+KV6BrO5B9PF3//+d8aKTc6juLgYhx9+OF5//XXst99++Oabb3pdj+TBc2l5yADECpAMLS0t2LJlC77//e9jr732giwbnJpETiWz1uQD8+bNwwcffMBEp2zYsCFl8sK5c+fio48+Yl5eGzZswMUXX4zPP//c/PKjRdhk8k8HBx54IAYNGoTHHnsMn3/+uenOIscGjLHca6+9zH/Nzc3461//6nqMefPmYefOnVizZg2z/NVXX4XP53PVGKQDQmj4JHN33nknbrnlFvP36tWrGdKzfv161NTUmKLbwsJCtLS0MNcjleB37NixGDx4sC0J344dO7B27VrMmTMnu5NywKJFixAMBnHKKafkZXseP//5z7Fjxw7cfvvtjtbPtrY21NfXZ9Qmyd/CC+OXLFkCVVXNa5kPzJ8/H7t370ZlZSVz73700Ud44IEH0hZFjxkzBkOHDjU1PQR8oken5zDV/ZTNPZguJk+ezJw3TfBOOukkrF27Fk899RSGDRvm5dnpA3gWHg9po6SkBGvWrMHSpUsxbdo0x20qKysxfPhwPPnkk6iurkZJSQk++OADPPbYYwCS63HygZ/+9Kd47bXXcNFFF+GCCy5Ae3s7/vrXv0IUxaQ6jZ/97Gc4/fTT8ZOf/ATnnHMOIpEI7rrrLsycORMHHHAAIpEIgsEg/vjHP+KXv/wlurq6sGjRItMakQqSJOG4447DE088gaqqKuy7777musmTJ+PEE0/E7373O+zcuRMzZszAli1b8Je//AUjRozAmDFjHNs89dRT8c9//hM///nPcemll2LEiBF455138MILL+CSSy4xrTTZYMqUKTjmmGNwxx13IBKJYOrUqfjf//6Hd99914wMAozre9FFF+H//u//0NXVhb/85S+YNGmSqb857LDD8Pjjj+Oaa67B97//fXz99dd4+OGHk06Eoiji8ssvx29/+1ssXLgQJ554IlpaWnDPPfegtLSUEYSni9raWqxduxaAIcCtq6vDSy+9hA8//NBRxJvp9unioIMOwu9+9zvceuutWLt2LU455RSMHTsW3d3dWLFiBV544QVEo1FbYsNkmDBhAk455RQsWrQI4XAY8+bNwxdffIF77rkH++67Lw466KCs+poOTj31VDzxxBM4//zz8dOf/hRDhw7Fxx9/jPvvvx9nnXUW49JLBkEQcMUVV2DhwoW49tprccwxx5hkgcYhhxyCW2+9Fddddx0uvPBCM+IsmVUsm3swFzjooINQVlaGZ555xhSue+hdeITHQ9o488wzsX79evz4xz/Grbfe6prxePHixbjllltw1VVXwe/3Y8KECfj73/+OP/zhD1i1alWvpkwfPXo0HnzwQdx+++249NJLUVlZiZ/85Cf4+9//nvSlOG3aNDz++OP405/+hF/96lcoKirCIYccgiuuuAJ+vx9+vx933303/vSnP+HnP/85hg8fjksuuSSjtP4nnXQSHn30URx//PE269mtt96K++67D08//TRqa2tRWVmJY489Fr/61a9cX8yhUMjs81//+ld0dnZi3LhxuOWWW/D9738/7X654Y477sA999yDRx99FC0tLRg/fjwWLVpkao8Awzq133774ZprrgFg5Py58sor4ff7ARjRQr/5zW/w+OOP4z//+Q+mT5+Oe+65J2XU3KmnnorCwkLcd999+PnPf46ioiIcdNBBuPzyy7PSdTz//POmTkYURZSVlWHvvffGww8/7BgCnun2meDMM8/E/Pnz8dRTT+Hhhx9GbW0tJEnC2LFjcdZZZ+H000931EYlwy233ILRo0fjhRdewP33348hQ4bgnHPOwc9+9rO0LLXZoqCgAE8++ST+9Kc/4Y477kBHRweGDx+OhQsX4oILLsioLfJcLF68GK+88gomTZqE3//+97j88svNbcaOHYvbbrsNf//733HxxRdj/PjxuOmmmxghPY9s78GeQpZlHHfccXj88cdx4okn5vVYHpwh6F41SA8DGEQ4TWegbW9vx/77748rr7wyoy9nD8lBiGw6CRI9ePDgobfhWXg8DGh8/vnnWLRoES6//HJMnz4dra2tePjhh1FcXGwLc/bgwYMHDwMXHuHxMKBxwQUXIBaL4amnnsLu3btRUFCA+fPn49Zbb/Xqcnnw4MHDtwieS8uDBw8ePHjwMODhhaV78ODBgwcPHgY8PMLjwYMHDx48eBjw8AiPBw8ePHjw4GHAwyM8Hjx48ODBg4cBDy9KKwFd16Fp+dFvi6KQt7YHGryxygzeeKUPb6zShzdWmcEbr/SR67ESRSHtrNUe4UlA03Q0N6dX+DETyLKI8vJCtLd3Q1F6t47UngZvrDKDN17pwxur9OGNVWbwxit95GOsKioKIUnpER7PpeXBgwcPHjx4GPDwCI8HDx48ePDgYcDDIzwePHjw4MGDhwEPj/B48ODBgwcPHgY8PMLjwYMHDx48eBjw8AiPBw8ePHjw4GHAwyM8Hjx48ODBg4cBD4/wePDgwYMHDx4GPDzC48GDBw8ePHgY8PAIjwcPHjx48OBhwMMjPB48ePDgwYOHAQ+P8Hjw4MGDBw8eBjw8wuPBgwcPHjx4GPDwCI8HDx48ePCQB2yv68CSpVsR96qo9wt4hMeDBw8ecoBttR24+h/LsPqrhr7uSq9i3aYm3PHUGuxs6OzrrvQ7PPfeJrzw/mZ8vrW5r7viAR7h8eDBg4ec4J4X16G2uRt/e+kzKKqGdZsa0R1RHLfd2dCJdZuaenQ8Re0bq0FbZxSRmHFe9S3duPeV9fhiWwseXPIFNE3v9f786+OtePi1L6DrvX/sVOjsjgMAojG1j3vijnc/qcFbq3b0dTd6BR7h8eDBg4ccoLUzZv792rJtuOu5dfjzs2sdt/37K5/jruc+RWNbOKtj1bd049K/foBn392Y8b6aruNfH20xrQ67m7rQHYk7bhuLq9ha226Sie5IHL+5byn++OQn0HUd9736OSKJyXxrbQfeW7szq/PpCV7632Z8sG43Nu1qz9sxOsNx/OnpNVj6eW1G+0Xixtjkm4xtr+tAfUt3xvvFFQ1P/vcbPPXWNyaJHcjwCI8HDx485AAqZd1Y+nkdAGCzyyTc2W2Qo65wdpPMjvpORGIqvtjakvG+W3d34KUPtuCf//0aTW0RXPvAcix6fp3jts+9twm/f2QV1m5sBAA0d0QRi2vY1diF+tYwtuzugCyJOHa/0QAMa0tfoSvsTNpygS+3teDzrS14d81OtHZGcdW9S7Fk6daU+xESkU++E44quPmx1bj9qTUZ76uoGjRdhw4glkJnVNvcjbue+xQba9pQ09CJ3z+yssdWyt6GR3g8ePDgIccI+qSk6wk30rKcCcn+4Sy+yok1p6M7jtqWbug60NAWcdy2rtmwGjS2GutV1TiwoupoS1i0Sgp9OGKfEQAsF05vgbac5FMYTNqOKxo27WxDfWs4La0WcWVle53TQWc4DkXV0NwezdiSRJN0JcX4rfiiDus2NeHDz3bhs01N2FrbgeUbMrN49TU8wuPBgwcPOUbAn5zwkIlJzVLzQrQykWjmhId8yXdHFFNjpLrogYi7iuiF4tR2LR1RAEAoIEMUBaMdTe9VLQ1NJOJ51DSR81dUzTxOKg2Vruu9QnjoeyjT+4khPCnOJxwl94oOJbFfLL5nRZ95hMeDBw8ecoxgCsJDJsBsRb5k/3AWYthY3JqECWmJq879IBYkc5JXHAiPX4aUIDx033oDqto7Fh6T8CmaZe1xGTOCWFwD2SKfQ0KT1UzHgN5XSXE+4ahx36i6Dp0Qnj0s3N4jPB48ePCQQ0iiAH8ql5ZG/u+ZhSeuaBlHa9GTFBFNu1p4El/1poVDs7Zr7jDcXMGAxBKeXozUoi0UeXVpma48zSQGqVxARLAM5Fe03BMLj5aBhYfokTRNN48TV/pv9JkTPMLjwYOHPQKxuNqr1oNs4fdJKTU8ZALMXsNj7RfO0K0VpSbipoR2x80dRFxahEwoinVcNwtPKktBLkGPQyyPk69KWbgI0UlFEKKUviqfHLAnpI/ZN6VLK2EZ1HRz3KOeS8uDBw8ecotwVMEViz92jSbqTwj4REbD4zQx9tSlRfOkTN1aMYrwNCYIj67b+6LrujnJEaKjOGp4JFPDA/ShSyuDyTfTPsZVy42lpKnhicR638KTqbWPsQ6ldGkp5j6aZ+Hx4MGDh/ygviWMznDcNcy7PyHgkxCgLDwRB0JiurRyYOHJVLhMC00bqegsfrKMK5p5HCehbnO7sW8owFp4Uk2cuQQ9YdMupGS4/18b8Ju/f5yRZcxZtJz8PFnCk/ahMgarw8mjhYcKsVc9DY8HDx485AfEXdEXmXzTAf0F7/dJEKz533Fi7WmUlk7tl6lLi3b90Pvyk6XTOnpSbOsywtJDfhmCIJikJ9tzygYqpSlyIpZO+HxLE5rao6htTj9Rn2nhokTLKV1aFAHrrSitzF1a1vZuOi6CCG3hIUTYIzwePHjwkFuQF2t/1fDQk1vALyXV2OiJRG+AZenJFDSnyNil5TJJ8RYLmkBYk7y1DTnFYEAGACo0vfcmQZoAp5spmJC2TMgzITc6rGtNT/xOoMtJ6HkkgaxLK/uw9LiSfF9Tw6NbLq1Ymla1/gKP8Hjw4KHfg0zS/dXCQ5MDnyQyLgze8kCvy9qlRU/0Gbu0nCepdCw8TlaNUEKv1DcWHnoc0pt8iUsvk37S500fJ1mkVrgPRMsZu7TU9PbVdd08H0O0bCz3XFoePHjwkGOQiaU3J9NMQJMDnfoC5tcBLMnpaR4ep/ZTwS1ZXDLCkyzZXihh4SGEp6/C0qNpWBvokOpULhwatCuPJjLJSAJj4UEeLTxpkhYnpBuWHotrJlE3RMtWXqL+WLTVDX1CeHbv3o3LL78cBxxwAObNm4cLL7wQ33zzDbPN66+/jmOPPRYzZ87EySefjKVLlzLrW1pasHDhQsybNw/z58/HjTfeiHA4u0J8Hjx46N8wNTz99OVKW3E0nY+i4giP1nPCk6soLRp8Ij36nMxQbIcvep7w9KZoOVOXFk1cMiHP9DlFGCKYxKUV7yXRMuVC7IloOdm+rLVK75FuqC/R64QnFovh4osvRkNDA+69917885//RGFhIc4991w0NxvVe5ctW4Zf//rX+OEPf4iXXnoJCxYswMUXX4xNmzaZ7Vx66aXYtm0bHnnkEfz1r3/F+++/jxtuuKG3T8eDBw+9AGI61/X+SXr4EGSdiaLKg0urJxYelwmKt3g4W3js/Q0GDJeW2NcurTSIHz05Kxn0kyZK3Wm6tBgS3EsanlQ6nGT7JtP/0PeCpumM9mxPcmv1OuFZtWoVvv76a9x5553Ya6+9MHHiRNxxxx3o7u7GO++8AwC4//77ceSRR+Kcc87B+PHj8Zvf/AbTp0/Ho48+CgBYs2YNVqxYgdtuuw3Tp0/HggUL8Pvf/x6vvPIK6urqevuUPHjwkGfQE1V/1PHQ1gU9lYWHWtnTWlpA+toVAncLDztxsRYeex4egpCfWHiM6WRPITyZWKJoYhNJ06XVa3l4euDSStc6xJM3JuHjHiRc7nXCM3HiRPzjH/9AVVWV1YnEg9Le3g5N0/DJJ59gwYIFzH777rsvVq5cCcAgTYMHD8b48ePN9fPnz4cgCFi9enUvnIUHDx56E/RE1R81AzTp0HSds8CwEwKj4cmFhSfDiunuFh62L6nC0glMl5bU+1FatFUqPcJjbZNJP2lrULeD5csJ0V7Kw0PfC/kSLdstPHumS0vu7QMOHjwYhxxyCLPs8ccfRyQSwQEHHID29nZ0d3ejurqa2WbIkCGorTVK0dfV1WHo0KHMer/fj7KyMuzevTvrvsly7vmfJInM/x7c4Y1VZvg2jRf9JS+IQsbPar7HKsZPFlQenlhcZfpLZyUGev7eiXLtp4Ib4dETfSFjRJcNUFQNsiw6RhsVFfiM/RLnJQiZX59sIVAJjyIxBZIkMMt48Geebj9VlygtwP3eYkpdCPmZXwAwcmhN1zM7DjVUmu7eR/qe0cGKsNUMjtnX76ycE56amhocccQRruuXLl2KiooK8/d///tf/OlPf8J5552HyZMnm6TG7/cz+wUCAUSjRirzcDhsW89vkylEUUB5eWFW+6aDkpJQ3toeaPDGKjN8G8ZLpF6QxSUFKAr5smonX2MliCyh8futV6sKMO8WQbbWBYP+rN47gYB1/nFVz6gNty/5YIjtCz2tq5pxDH6iEgRgaFUJBEFAIHHOoYLszikbhOq7zL91HSgoCiLod5/WGjtj5t/8+SaDTpEoWowcKgiY9xR/b9FymkDAl7cx8Qes8/X5MztOMGTNo7JPct1XkKjacAIgUb8DGYwjQV+9s3JOeKqqqvDaa6+5ri8tLTX/fuqpp3DTTTfhxBNPxJVXXgnAIC2AIW6mEY1GEQoZgxQMBm3ryTYFBQVZ9VvTdLS3p595M11IkoiSkhDa28MZhUF+G+GNVWb4No1XR5f1IdPc3Il4gf2DJxnyPVbNbVaEaFzREInEzd9tHRG0tFgTc2undS6dXey6dNEdtt5/7V3RjNpwy9vT2taNlpYuc6zaOqyyE9G4ipaWLnR1s+/dkF9Ga6vx3iSuxrb2cFbnlA3a2tjI3Nq6dpQWBVy3b6KyK7d3pD/2UZcxa27pQntFyPHe6uyyxqo7HMvbmHRS91Mm5wQAHdQ17kxyHzVRy2NxjbmHmpq70FLiPuY08vEclpSE0rYY5Zzw+Hw+RlvjhjvuuAMPPPAAzj//fPzmN78xzZBlZWUoKChAfX09s319fb2p+6mursZbb73FrI/FYmhtbcWQIUOy7nsyxX1PoapaXtsfSPDGKjN8G8aL1kPE4tmfLxmrzzY34Yk3v8KFx03DpJFlPe5fN0VwVKrApLFOYfpLF7lUFD2rc6HbD3Ptp4JbhetoTGXaCUdYDY+iaDZ3WDAgmfuIiXd4LNZ792OcE8x2heMoDLpb/2jBcTyupt1PN51KNKaaEzf/HNKkQFWzu87pgBYNxzI4J9u+Sa5bV9i6vzVN41x8md1/QN+9s/rEkUbIzm9+8xtcddVVjM9VEATMmTMHK1asYPZZvnw55s6dCwCYN28eamtrsW3bNnM92X6fffbphTPw4MFDb4JOlpeLKK21GxvR0BrBZ5ubetwWwOo6dOhslBYvWqYrVGcp8NWp3TLJw6NpuqtLK2niQaqWFI0Q5T7q60zLQGrhci7C0tNZDrDFTPtrtXRGfJw0D0+yKK0952Or1wnP8uXL8cADD+Dss8/GCSecgIaGBvNfV5dhNjv//POxZMkSPPzww9i0aRNuv/12fPHFFzj33HMBAHvvvTfmzJmDyy67DOvWrcOyZctw3XXX4eSTT2aivzx48DAwQL+Mc0F4SBu5mofsYenuCfF0Jkoru+NlWy2dnvB9nNCUz8NCT3KKamTU5SdUkoMHoAlPL0Zp9YDwZBKW7uZ+SZa7hk1GmT/Cky5pcQJN+pK5mOh7TNPZY9LibE3XsbGmrd+Gqvc64fn3v/8NwIjMOvDAA5l/Dz30EADgwAMPxB/+8Ac89dRTOOWUU7Bs2TLce++9pqtMEATcc889GDFiBM4991z86le/wsEHH+wlHvTgYYCCdl2oOZg8yGSXqy9vPucKU9wzH6UlmAlHS/vLPkpNTqWFrA6Kb4MnUopqJzyMhUfqi9ISfO6g5OQv27B0t4zKydwyvRWWzlh4Mkw8mI2FR9XYTMu0m/OtVTX4wxOr8dh/vsqoH72FXg9Lv+mmm3DTTTel3O7kk0/GySef7Lq+srISixYtymHPPHjw0F+RawsPeWHn6subT8ymM3ly+Dw8YLbNBny/IzEVRaHU36/ky9sniygM+tDYZolW+cmbJ2qKqtksGiEqQqivMy0D1nXojsRR4KDlYSw8mRQPdSE2biRB03UmmiufFp6eJR6kLTwZZFqmzofWpD3ztlEi6uP1tbjo+GkZ9aU3MPATeHjw4GGPB/0VmRvCY5WqyAV4lxbdxWhMZfrMurR6XksLsFtj3vmkBjc+vBLtXTGs3diI+179HOGoYuot/LKIogKDEMgJywyvaeHdQ3FFs03wIcqlJfdBpmX+XojEVKzf0oRL7voAz7+3ybZ91pmWXaxBbgQjyo1dPi08dN8ydWnRbqykeqQkiQdpl1b/SwnKwiM8Hjx46HeIxlVs2d1ukoNcl5bItYUnzGVa5l1lbnWVnM6luT2Cu577FJ983eB6PL7f3Rzh+Xh9LbbVdeCbmlb8Z/l2LN9Qh8+3NJuTk98n4fgFo3Hw3kMxZ9JgAKwVQ9d1WwZnRdVsOg86502+LDxxRcO6TY02EuF0rEhMxTPvbAQAvLZsm2377C08Li4tlb2Wry3bho0722yV2/Nq4emBaDn94qGsS5mppRUnddashdnmyco3PMLjwYOHfodn3v4GNz26Cuu3GAWFs52o3GCKlnOkr7VZeGwTMbuewOlcbnl8NdZtasL9/97gejwnywYNQl5UKiqLsfD4JEweVY7zvjsVJYmcRrSlIBpTzX6ScPO4qplaloKEK4t2aVnV0nMrWv5g3S7c9dw6vL7cTmDshEdBcZLJlifOze0RfFPTymzT0hG1kVI3wkJP8l/vaMXz723CU29906sWHtalldmBsikeqmvO1dJrm6wcR6VFmeXJ6i14hMfDHotVX9bjkde/zPirxkP/R2O7oS1pSmhMYjnWQ5gWnhwZ4fmIHL6LbnoO/lzqW8No6TASySUTVPP7haMK1nzTgAeXbEA0rpruCVXTTVdVJK6a4xigIrTkRNI2MuFt3tWGvzz9CQAjk3JB0CA1imLlThlcbiSBLS+2Es4RwpNr0TJJ1EjGhQbvlorEVBQlSUpJu20UTcM9L36GW5/4BItf+gyxuIpNu9qw8G8f4Yn/fu24Dw/aKtaeSMrYFY7bCGgm4vhM7296vDPNbaOlaeGJcKJlJiw9YTXcVtdhLeunUVq9Llr24CFXePWjLahp6ML+M6pzkjzOQ/8BmcgIMcmXaFnPQVuKqjFfvLqDS4v+mSxK67WllhVjxOAi12PykpJwTMGbK3Zga20H5k0ZYo4frbeIxlRTC+XzUYRHTmh4EuuWfLwNK7+sN/tNwtfjqmZagb53yDi0dsQwf6qV6DVfeXgIEXOyQPDkIBpTUVxgWXgUVTMJHcDmjFE1HVtrjUl61VcNqCjZjJFDjDGvaei0tktCBOj7kuRiisTVpKkIkuG/K3fg5Q8348ofzcHo6uK09km34rnzvnbCo+k6djZ0YfigQoiiYLg3mbB0TsOTGFOa8Di5H/sDPAuPhz0WxLyeqVDPQ/8HeaGahCeeW5eWar7ce9yUTa+h63bxJj3hseSH3e7zLVYixGSTl00jFFXRlcj2HI1rjIWHjFeUsvD4ZbvYWKG2I5gwotQSNVNZoStLgjhw5lD4fVQeHik/hEc1CY99PJw0PAWUm629iy2FQb8rVFVHYdDadmdDpzVW1ITtFpLO94mQnGhctWt40nxFPfX2NwhHVTywxN2dySN3Gh7j7w/X7cb1D63AGyu2AzBcViyhZ0kWsfBsr7NIols2776GR3g87LnQra9YDwMLpsvJwcKTi9w5VuLBnrdl12votnuSITmMPoSdGLop8XOyKCKbSyummMJpWlzMEB7KwhOgiAqpdM27Q35w2AQsPG0WfAlyFKfC0p1qF4l5itIik6sj4eGWqZrGjG8bT3i4PDy01YgeK9pCk8xNRIuZidsnFlN75NICMrOQ0PdJxlFaDu6w+hajPhnR5Dhl8qbHjWh4djXS9bbUvGaXzhYe4fGwx4I8Tx7hGXhQTQuPkeE316LlXEZp8ZObptsnODfdDs13dF1nwn+TTV5kCEhVnnBUMd0OimKJi1WK/NCWBz/t0hJJWLrl0gCA8mI/An7JsvCoVoJDnwPhyVemZXKtkrm0aHcafX/QhVoBu/hdYTQ9rPvPWp6eS4tEtekAOrrjzHaZ3rK8hSgZepJ4UGXO3/ibjBE5H3JPUhWgmHGMKcYz2k3VXtMBW921/gCP8HjoN/j4s9148NX1aX8Z6PAsPAMVZNLkC3ECOdbw5MWlpdsmOPo4blFa0bjKuMKSaUeI9oi4b9q74hQxsEgOHWHk5tIi1hryhc9HZxFyE4tbrg1Cgmj4EYcENQ8anmQWHuNYRGfE54hp6+QtPLTehSVHqqqb40aT2KQWHgcND2C3LOkZiuMzsvD0wKXlJHgmbRCiQ4gP7SqkjxOLG5ZDm56qHwqXPcLjod/g2Xc34uX3N2FnQ1fqjUFZePqh6dRDz0BrePgvxVxGaeXC7E4mRzNKycHCQ8MtCSFfZDRZmDAZAxJB1dxuZUyOqzqr4VGJm0Y1Baa0aJmQBXI80idS1JmIfmk3j8xZeLRIBw7d/ndcVvI61Bx/2dNEzm2dX7bcaelaePhIIlWzCF1MsVxjya6Dk4YHANq72ONmStIzsY6oPUk86KDhiXOkj9yXhSFWDE731bQCwbo3Yv1QuOwRHg/9BuRllKoAIAGZK3ozs6uH3oFKEx5OAJnLPDw5ES0n7teg37CaOFl4aJJGkxz6XPjInmRf66T/hYnyCXTItqGfsNo3J/G4aj5jAdle9JMcj+xLXBiEENHPJU94lK8+QFDtxEi5GUWROtd+Z4N0RMtEZ6RyEXOtSSw8vBVFo0L4Aet8k10HmgzR40MsS8RK1lt5eDLNgeRkHTLzNiXOh5AZWuDNa3jItsGAbD4HnoXHg4ck0JJ8yTnBdGl5Fp4BB1rDw3+15sKlpeTSwhM3JgSShM8p07JbZBZT9ZxPHpjUwmP8Tyah5g7LwkNnSGaFuKpZPNTvaOGx3GCANVkTckOHJtMuLV3XEPviPfP34K5vXPudDZJqeIiFx0dbeKz7pY238FD3Ej8hG9Yw+/rkhIe28FjtkZw8pPRGPgW8TBLAHiQeJGOjmB+e7i4tGjEqDD/ol0xBfH+M1PIIj4d+A/LyT9ec64mWBy7oPDxxW5r+XLSfw7D0xERHCI+u27/o3epn0fcuIRRkYkknLJ0UyKTdYbYkcU5h6XQ4OReWTvonJCw/JIqLTHySKJjuLgBQaz6H3l5v/q4Kb3TtdzZIquExLTwW4aHHtJXT0tDWQhvhUXWOgBrnm8xNRFuMaAsd0fDQ90S+wOTh6UHiQZIh3BQtR1mXViggm9ZAGjFFY7YJeBYeDx5Sgzy3dOhoMpCXfo6DQjz0A5CJR9V024STiygg0n4uEg+ahIdxablbeHRXwmO0Q4p68hltafAaHhp0pJehS7GitOjioQQ+LvEgOSKZ24homYhyZZmdNuJffQAAaCyZCgCoiO2CFm537Hc2MHMxOUzm5NyICJvX8GRm4dEY95Dl0kpTw0ORznaO8KRrhRYFO6FIhZ5US+cj0OJUJF4kphiRg8R6E5AZomvuo6jmPWdYeIz7wyM8HjwkAXkppFsPhmzlubQGHsw8Mqpdw5OTKC2VtWb0BJG4pV8AiIWHbdctFF11sCjQmYLdcvGQMXAkPFyhUiYPj5OFR0ouWiaEiFh46JB0PR6Fsn0tAGBn9cGoUcohAIitfhm6yoZmZws6pxAPjbfwcJFXbV0xLhLJGhtew6NwZMkkPGlGadGuRDKWFgl2bYIBsY5kAvoeylS0zD9LClUvTdcNi5hpvfG7WHjimjlWIcql1R/LS3iEx0O/gZlkLm0LD7ufh4EDOvGgXcOTu/ZzEpaeeNkTV5QRCs5u4+bS0h0m2OKQVQvK7Yud7FYUtBfKpLU2imLV9WLy8NAWHonX8BjLyce8zFl4JEq/o+z4FFBiEIoHI1w4HB9FJwMA4hveQfj1P+dEu5LcwqMz58O7tHQd6Oi23Fq0u9zJpUUTzCgnWqYnezImcYfEgzQytfAEKG1VumNH95nPgpzJvoBB1HgSR4hcKCBBdGAMqqajM5HlOxiQTTLdH8tLeITHQ78BeSnE002eRbk9PAwsMKJl3sKTy+Kh+bDwILmrzC1KixCVIpfwXxpJXVpMWQQqn4yqoz2REI8OMZYkPkqLFS0T6wnpH23hUTatMJaNmwdZEvFxdBLeKz4BEGWou76A1rLTsf+ZgL4X3Nb5fJK5Df8+oHPi8Hl4aGia7mhxM601lGiXiJHpMYtEHQhPMDMNT8BvHSNdLSM/LpkkH+THii4QCxj3Uth0V8muLjdCKlnRskd4PHhwhE5VmKZf0slAnlXPpTXwwISlK7zWoufXW8uDhSdEuSNUm0sr9d/Wl7RMhYo7d5AQqkInCw9FePhJk1Sfp91mZi0tqngkYM/DQ/pHfhvurHXGsvHzISb6vMU3EdLwaUabWz9x7H8mIP1y+hDSHCw8/P3RFbZca05WIjKJ81FaES5Ki5AcwHDv0OucEu/R26X7jqLJJG2pSwYbacnEwuOwL5M9OqqYRM6w8DgTHlOz5Jc9wuPBQyrQ74NM06N7Lq09D92ROH5731I8+65zRA+deJCfpHKTaZm1ZvQE0Rhr4QHsfWSLhzqLlumJhbe68CC7ybJoWmCsdqjyFJx1jExCtBVJtiUeNJbbRMuJ8zRLTWz/FFANd5ZYOdqM9lI1HfKYOcY229Y69j8T0IkH+etlurQSkyyfaRkAOhKER3MgQwAQ8JN+s9Yhcj0IAQhS1pcgZ+EJR5zJySA04fDgekBL001PZWROm/DwbqkskxaSfeMuFp5QQHYlPCTvUDDgER4PHlKC/gLKOErLs/DscdhW24G6ljA++brBtk7XrYlJVfNFeHLv0qLzlPCTkGvxUDrTcsxyHfBWFx6a6XZiLUsA69Jyc4sUF1g6IZkjVzYLD+fSIhYeZXPCnTV+PgRBMEmapmmQR88y/m7YDK2rxbEP6YJJrKe5EB7awsONGbHwOFl3AIss8YJnMmGrnADZ+JtYeIx1buRk77pXcVLBJxgd3+R6fjTo+yTdBKz8mGQiXLaLlnUu8kxh7ktetEzyQDUmLIchvwR/gkBGY/0vfNYjPB76BegHz+3FZNvHEy3vsYgqVhQWD43TuNgITw9Jika5T3Pp0grSLi3uy9lNtOxkUQgFJJvVhQdxaYmCwFiWAPbL2unjwS+LTLV0ny1Ky1gucBYeEq0jSyL0eMSw8ACQx80HwBbwFAvKIA4ZZ7S7bY3jOaQLJs8Mn6KAFDM1o7Q0051IrFjEwuNGBMhY6ACnX2Hz8AQZDY/MrHMiPKVCF4ojtcbfWmvScyRwysuUCja3VAbJB+37shXkwzHOpcVpeMqLgwCAukSF9WBARtCz8HjwkByMhSftBzbxle4Rnj0OJGTVSW9AkyBN02wanp5eb7b9HFh4uMSDgH0ioX8yOXmYKC3LdcBbXXjQyQFDfucMuICzhYfW7wB0WLqW0NKxomWZy8QsS4LhqlLjEEqqIFaOMtoxq64b+8tj5hq/Ny5z7V86oCdgJ6ExwFt4jGWlRYYVqzOFhSdIkT8miosLSw8yFh4w65zIyXS/Jdgu0jqcT45DstpqbnByS6ULJ8Jjc2mR+9Jvd2mVFwfM/YxtvLB0Dx5Sgn5m03VpkWfVi9La80By6zhZeJgK1g4Wnp5eb7findkiSkpLUBNiMg0PvU51mOCCfsl0G7nm4TF1NgIjpuXB5zACgCIq7B0AfFSYuUqF1FuiZXaS80saYp++Zuw7bp65HZkMzdw4E/YDIECt/Rpah911mS6SVQMn4+ejEg+S45cW8oTH+b1C576hJ2m+lhYhtEPENny/7m78svh1VKIVmq47Ex5fjfl3kZZeIkbGxZmhhodcpUxcWinD0qOKdV9yFh5BsEglQdAvw+9lWvbgITnoBz1tkyxxaXl8Z48DsdokCzUGDGtBLl1adOZhAMiFysCM0gq6W3h0l78ZFwat4SF5XtwsPKZLi7Us8XCa5ItcLDyAMQHqVNsAbKLo+dFl0Jq2QwgUwTfjSKsdkSVpYlEFpGFTjH58s9S1j6lAa3J46wU5FlNLS3cmPG56poCLhYfPtEwsaXMDmyHrcYzzNeCK0iWINe20kRMfFEzy7TZ/F+vpWnisv8Ox1ISH1rsR4pZJtmXyLBAiE1NU5t7tDMcZwkdreCRRMMeYIBTwwtI9eEiJHomWPcazx4FYHpIVhCR/8xNVttf7nVU7cPHt72HtxkZzWS5KSxDRMu1aIpMG+SB2s/AwUVqUa0wmehiXyYvOhhzsoUuLtuDEVc0mWqZDpSvEDsyIrAIABA65AGJBmbmO1vA89+5G/GfFdvgm7g8AUDb2gPBwBJiGY6Zl4tIqNNwtnd3JXVos4aEtPCQPT6J8hU+EAGCaz3BVdWt+BAQF0c/eNglPtdSKAiGK/QNfwy+o0ETj2hTrHWlZE5laXmlYeOjtTcKTgUtLM8kSG4lH0NJhlebgXVqiIKCsKMBsH/TLKFQ7IELzCI8HD26gJ550NTxkKy9Ka8+DaeFJ5dJyiNLK1qX12cZGxBUNX+9oNZf1lO9omlX6Iuig4SEkgK2W7ixgJhNcyG+Jll0tPERnIyZ3aTlN8nRIOmBMXISYqapmhaUnZgeZIjzTfDshAJCGToYvEXpOQM61qT2C15dvx/PvbYI0Zh9AEKG17obW2eTaz2RgNDwu9wKppaXpVrX0EptLy1jOl2+QZdG0cMQdiovGTe2SiApfGCPlZugQ8Hy3IdbWNi1FpDuMab4a/KbkVVxb+hKOLzCE2s3jjzX6hzgQ6055rqzFLzVhoLcnxI0nhclgWocS+/IkqzVBeAI+IwcP7dISHSw8hXWfYOyyW3BN6cuYGPkcupaeW6634BEeD/0C9DPqWXgGPghJ0HR73hTa5WRoePhq6dldb/LFTn/F9lTDQ3/FOml4RJPw0Loh2LZTVM20xgQZC4+LhicxRIaGJ5mFx/4s0SHpWrgdXf9ciF8Wv4FhUgviVK4bgcu0DABTfLsAANKIvWztkrB0Yu1QNR2qFIRYOdL4XfuNaz+TgYnS0twIj9VHQmzcRMtBnvCIVkh9VLFreIiVrSRWjwUBI29UV+EIrFPHoUktBGLdKNi9GqcWrIQoAIViDH5BxTZhBDpGHowOzYhk0joakQqMSysNCw9NBon4OhvRMiE8PMlqThAekneIt/CUUIRHhAb/5/8CAAySOvFd/V10PXs14l9/CD3NPET5hkd4PPQLsAX+0rTwEA2PR3j2ONDiUF7Hw4qWragRotPI9nqTCYwW8vbUOkgIjyAYxMC0lCSz8DB5eNi+AcaELHH1rXhYFh77BE6DTzwIAMV06Ypta6B3NWOsXI+FJUugtewyLad8LS0JKib6jDBreeQMW7tOSeniigqpaiIAQK1zTjKZDHQKAcD+biD3jo+qQUWuL5mMIzGViT7iXYCyLJrXib43yDWJqzrGyXWY/fVifMdvWG46K6ZAkiQsixrnNm7LixgsdaBLKMR/wnvh89hwvB/6DkRRQItWaJxLGhauTEXL9LNCLFcZiZZ5wsMdk7i0iLuWrqUligIjWp7n3wShsxFaoBivdM9Bpx6E3l6PyHsPoPvlm6ArbOX6voBHeDz0C2Sn4TH+59P4e+j/oC0PbqHGABulFfRZkTjZgLzMI5QYtKe3Dp2DRxAEW54S8tvNjUXOhbgSfLIIWbKyJ6d0aQkCk/CQh9PpMdXYaz43/5YFDdi9gRJEC9DjERQ0fYEZvu2YF9iMoKAgKhaYoeg0ZIfKktG4BqmaEJ7MLTy8hSuuamhuj9isu8SlBVi6peKQzyRtXeE4lTGZt/DQhMchSkvRMNO/3Vquy+ismg1ZEvFhdDLUyvHmuvVlh+G18Gz8o/MIqMFyCKKA5gTh0dMgPJkmHqSfBZJAMRPRsjl+iTHhj0naJ25TycWl5YOCYwqMMiPKlKPwTmQG/tj1fQT2PQ3whaA1boVC3Wt9BY/weOgXYPPwpPfAkjTsyYSnS9fXYtHz69IO8fTQO6DFtLbsuVyeHEs0mkgQl2VoFSE6UeorvqcuLTJBkC9kvraik4XHydoT4epxSSlcWk6JB2nxbTIQDY+ua1B3fQEA2KkNMlaG2xgyFf34nyhdcR9+XPweflRoCI8bC8dBEOxTh5OFJ6aokKomGOfatAN6PJJWHwn4yXvp+lpcsfhjvLHcICBm8VDKpUX28cmiWWusIxw3P6SCPl7DI5gWNSbTckyFrutQNA2TZMOy9bJ+OK5uOR0oGgyfLKJbD6DzkMvw1rAfY1H7UWgbMtvcP+CXIAJoUYuM80/HwpNG4sFwVMG6TU1QVM10t0miYEsgmQpMhJeLhocgaFp42CitgAwU+XR8J/QZKsQuCIUVkCcfCgDojInwzfwufBMXAIB5r/UlPMLjoV8gm0zLpoVH09HSEcUX21ps29z/7w1Yu7ERb67ckZN+esgNaNeBLbcKZ+FRzNDjhIUnWw1PIp8IrbvpqTeUtEWqXNssPITwULYWxtqT6IAZkp4gLz45lUvL+F8QgJKEJqeiJOC4LQ+i4dGatkOPdAC+IL7GWGNluNXKtAwdyva1AIDdajmiutG3utKZju3yZQcA4zqLRZUQCisAXYNavzmtPhLwZHhrrZHPpqahk1nPh84DxtgTcrdpZxveXm3kxeE1T8VqKy7y/Qv7+Nm+abphXfTFOjFcNt4tO6RRUCElLHGJRIuKjnq9HJuUakbEG/AZVj/i0tIpDU9LRxQffbYbm3a1sWkS0nBpvfLhFtz13KdYvqHOcp1KgtmfdN+fjDvMRcNDQMaMJjyVQge6nrsWNxU9gSOD64129j8D/lDION9EX6RhU43j7fQIjwcPAHjRcuoHlk/V/+dn1+KOp9Y41mYCLNfDnor/fboL735Sk3rDPQSMhof7ImWjtDRz0g+YGp7sTDymhSeHomVimQmaFh520ne28NjJD0nuRrQSkpj8a93MhiwKmDK6DN87ZBzOOmpyWn0meXiIi0EaOgUdYrHRXrjNarujDnq4HZB8WDHmQvy25XRc3XIa2ssmObbrSHgSVhVi5VFr1qfVRwKe8JAyEWTcCWGURMF2fIkiPI++8RW27O5A0C/h0NnDGUvc5MZ3MFqsw9GhdbbjR+IqquPGx1KkcCjmzRqPCcNLMW5YqVVTTNVMclIU8pn9MNycMF1atIXn+fc24sElX+CWx1bj7hc+M5ezomXnd1ZDq1HGobUzypw/nRG7O6Jg2YbapO89NsKLrZfGWyqJ5ZEQnkqxAxdIL0Nvq4Uo6JAEHZsxCvKYfRhLYzSumrmYtJYaaN3pJWDMF9ydvx489CJot1Q6Jll6C03TsbOhCwCwZOlWzJk02La937fncntV0/D4f76CpulYMKM6ad6VPQU04eEjb2waHuLSIqHH2bq0osSllQ8Lj5tLy7jvUuXhiZhJB4120i0tIQoCJFHEcQvGpKXdEAAUJdw8ytZPjGONmI5wTSegAWLUcmlJDV8Z/1dPwoGzR+GdT+vRpUuMBoo9V2cLDwDI4+dD2bwCsS/fh3/OiRB8wZR9Bex5iLo4wqNwhIcmSJIoMiH4oYCMmy6cj4qSICRRhKJqGCE1YUiHYXmoktoxRm7AUcF1EKGjSw+g5c0NmBHeBMhApGIivjN3JL4z14g6M5NDKhbhCfpl+H0SwlEFQb8Ryl2nlgIAtOYd0JUYBNmP1kR1cQD4bHMTwlEFoYDMWHicIuzoc1dVnTp/0ezPtroOvPS3DxGLazjjyIk4MtFf29gygmc50bZF3DoS+YsAYFRVMbSOBhzV9SrapSmY5d+KQiECsWIk/qUdgl07dkIdMgl7J1Ic+GQRccXIxVNcWgyxYiS05h1Qdn0BDB/q2J/ewJ47C3gYUMhYtEzrIKi/t+x2zmiarr6hP0LTDF+7DvdssXsaorSGx2bhYfU9NpdWFixF13XTXE8Tnp4mHuSJCm/hEZ2itJi/WQ0PIU58/SoeZIgETlORCoUhH0RRgNqwFVr9JkCUII+bh4hkWCHESLvZV6k+QXiGTcWY6hKzDVogTIPO2ExAiK08eg6E0iog2oX4l/9L2U8Ct8KY5BrS4f+S5G7hAYC5kwejoiQIrb0BY32GJfjY0FpmnwuK3sN0/05M9e/C3MAWVNSvwoiEOys2eAqzLfmIiikqRXgk01oSSAjZ67USdOoFgKpArd9kOy9dBzbvbk/87UyMaZgJETVWwzNnovGht+zzOpNo1ieKejpBdbTwGONKpy4YWlmAw+YMR+yzNzE6vhnHFazBRF+dMQZ7fxdq2Sisj4+EFAiZ+5Bxf/69TXj23Y3YhmFGn3d96dqf3kCfE55Vq1Zh6tSpWL58ObN86dKlOPXUU7H33nvjmGOOwZIlS5j10WgUN954IxYsWIDZs2dj4cKFaG5u7s2ue8ghMi0eyke60MnESCglPXH692jCY/3tJmLd00C7LVNreDiXlq7jvyt3MBmTU0FRrXw/9LF7HJbOiZZ5zkFICH0cnbt3jf4lhLYJ0iCncGlZFh5rmSAItrpXBCRPTVkijDj2+X+N44ybB7GgDBHZcGlJagSyHoMAHUL918Y2CZfEjRfMx2FzhuPIuSMcj+Eclm6clyCK8M/8rnHsz/6TtivRjfDZXVqiaU2j+xOkkjLuO60Kuq6j+/U7cUnBEhwY+BLT/TuhQ8BngnGOpaJBEF7v3hsvd++Df3fPxnvqHLyBg1E9fR7TPrG0RqIU4QlIprUk6JMSFj8BW3RjwifCXTPDceK++SaRDJN+1t0SCNIlL2gNz6yJg/DDIyYy2/qSWLbp54xE+pGcRQUBGUPKQ/D7RFxy6l6QJRHqboMAT5DrMEIy5lpp6BSUJO4pOtP4Dw4dD0kUsOKLeryxfDue3jIIuuSHWNZ31h2gj11aHR0duPLKK20++U2bNuEnP/kJzj//fNxxxx147733cOWVV6KiogILFhiK7xtuuAGrVq3C3XffDb/fj+uvvx6XXnopnnjiib44FQ89RDbFQwl0TUdBQDYnn/VbmnDQzGFMiOWe7NJyErnuKegMx3H7Pz/BvtOqcNyCMeZyNg+Pu4bHKUqrvqUbyzfUoaTQj7t+cWBa/XBzw/Q4LD3OEp50NDws+TEIEEkWRyw7KV1aVJQWDVkSoaj252fCiFJMH1OB8cNLoUU6oGwyPjD9041aWLocRESXERQUFAthVMldEGLdgC8IcbAhaB45pAhnJ9EJOVmYaGuab+L+iH70OPTOJuhdzRCKKl3bInCz5kVjChNlJImCjXBJomBO4AAwZVQ5tLbd0NsM68QPClcAAFrLp+HTjunYSzGsD9uVSvhmn4iWtghOPnAsqioKHPtArHrhmMK4tGgLD7k+W/Wh2AsbDeHuXOtjbPKoMqzb1ISNO9sAcGTYhezSRU35fE9HzRuJYZUF+NfHW/FNTZt5n+i6jnte/AwBn4SLT5wOgNU/EVEycRnKkoDrz5sHRdVQXOCHHu2C1mRomWTB6HsrSlBcVIG5kwNYt7ER+8+oNvu43/RqlBT68ey7G7G9rhM1aiUajr4dk8dUOJ5Tb6FPZ4EbbrgBI0fa/YuPPvooJk+ejMsuuwzjx4/HhRdeiGOOOQYPPPAAAKCurg4vv/wyrr32WsydOxczZ87En//8Z6xcuRJr1qzp7dPwkAOwLq3MRMuqpjO+/s+3GF8fEUr0x08MexLYc92zXFpbdrejpqELyzfUMcvTFi1rFhkghIdoCzq6Y2lbCtzEmz218JC+EoLCz/mCSXhoEmfvg1W+gFQoT+7SMiOpHAiPE2RJxHf3G41JI8uMSVdVIFaMgDhkvLm+TTMm9mK9C4cEDUuEb/x8CGJ61lFn0bLVf0H2Qyw1vvC15vQE+MkmffraSZKzaPnoeaPgl0X88PAJhivPIRdM0/CD0CwPRqtmuGTeiUzDQbOG4ScnTnclO4Bl0QhHFVMfFvRLqCwx9EmDSkPme2ezlrDw1G+GHo+a5zV5VBkAYNPOdiiq5lpYloaZyZqyWoqUdWvGuEpMGllmbgMYHx5rvmnEsg115vuVvDNFivCQI8qyiFBANl1bRg4ltj87ReOchlYW4ppz5mLvCYOY9dPGVOCG8+djaKUxhpmUvMgX+ozwvPLKK1izZg2uvvpq27pVq1aZlhyC/fbbD6tXr4au61i9erW5jGDs2LGoqqrCypUr89txD3mBxvm0U03svA6CniSb2oxcH3S14T253hZ9rtkm3esrkBcu3+9oMpcWN8mRvDnky5mQJV1PvyJzxGW7nkZpkfuWEBtXC0+SY2qa5boyXVqmhSeFS4ub5J3CswGW8JMEgNLQKWZ/RQFoT0z4Y+U67OUz8tz49jrasT0nOBGeODfuYsVwow/pEp4kbh36PjHE23b91OjqYvx94SE4ar6RKFHduQEA0KYbmqWtyiDESsdAEkU81HEonu7aD2tiY2zuMScQC097V8x8RoN+Ced+dwoW/nAWJo4oNUXszXqRYdHSVai1X5upFUYNKUYoICMaV7G9rpNp38gyzZ6/rusmeafz8MgOZA+wUjjEmbxXGrNOEu3JK30ccVZ3G+7NTqnMXLZLHO46NjRMcXcGCRHzhT5xadXU1OCWW27B4sWLUVhYaFtfW1uL6upqZtmQIUMQDofR0tKCuro6lJeXIxAI2Lapra3Nul+yy8uiJyBCPidBnwcL/ItbR/LrwYv+mN+JfekHTBCEvFzf3gAtxqTPY4+4txJd1zTd7Leu64yFh7/WvDGObEs0E3TiwJiioagg9fm7h3fn5rn3SaJRhNJl4hHp+487P1EUrIy3PqNwKMl8q+m6rX80eff7RGY9P1GZy2VrOy1R4sE/bJK5TJQE08JDhLzC8L0QGOwc4eMIByOqorH9lytHGu601p1pjbsbHVU1ncnHFAhINusWEQ2bbamKESUE4CXhKAS7duPL+DCcHZAhyyK+VgdjmzrY3DdV/wqCxv3Y2mVEXAkACkI+FAkCKhJWHuueF+AbOhmxbz6G3rQVmmYdZ+zQEmzY2oydjZ22YwiiwJxXJKYwRZP1xPlJEvt+85nHNfpAj6P5DiH7ioKZqoCA3IcE3bWGfuerkn0xpeldBIU4dsoj07qGpC+a3vfvrJwTnpqaGhxxxBGu6z/66CP8+te/xumnn465c+eipsbO9CORCPx+tgor+R2LxRAOh23rASAQCCAaza5ehygKKC+3k69coaQklHqjbzEKGtlKwoVFIaYwHY/uiOWblySRsRAJiWsp13eZy4Ihf16vb15BRcUUFAZs59Gf761gyLiGumA9X3FFYzQtoQL22gRC7HUnm5YmJhE6XNcX8GFXSwQlhX6MrCp27cf2RpdK1QJ6dF/4EiSsIHEO/Is8GDAmEvr+8/vZyaWkNAQp4a4rSlzfkiLjXEVRtPWPtohVlBeiiIqoCbhUTg8GfSgvL4QWi6ClcRsAoHLyTMilhWY/2zT2Piqce2xGY+NkLRMktv/+UeMRWQGgbXdabRc0uVcYl6lxHFRRBB8VmCCKAioqipjtu75eCcQjEEPFaIwPxY6ocb+UlYbM60RQWcGOqxPKywyCSFyswYCESu6Y7ZHEvSoARaMnovmbjyG21QAYYrZRnHjPyQ7pJopLQkwaiqY2K+pKlCSEEn0M+GVmPIuKCOGSUF5eaPUDQFFxCMWFfrSGSTkTCVWD2WenkHomlc5WtNRvAQC0lU7CPVv9CAoxFFSUp3UNQ4k0CIGgz3xX9dU7K+eEp6qqCq+99prr+qeffhrhcBi/+MUvXLcJBAKIxWLMMvI7FAohGAza1gNG5FYolN1AapqO9nb3hytbSJKIkpIQ2tvDtpwSHiy0t7Phkw2NHVBj7rk66Cyk0ZjCfMHH4ypaWrrQ0GR9MXV1RdHS0oU9Ea0dFolvbetGS6HxAtkT7i1yXck1AYAuiqwCQGtbmLk2HR3OobSaarmyCNZ9VY/7XjV0GY9de6RrP5pbnZ9tVdV7dF90h433UCymGO3Y3FXGdaHvv3CYfXc1NXehs8u4xopijFMs4Y7tjsRs/aMJX3t7GPGoNZ5uSjU10W5855eArkEoLEe7GoKQaFtRVLRrll6lSS2Cv3JixmPD58Jp72SfOzVgWDZijTVobmpPqQ9qbXUPq95VZyWxa2vrhkDZMSRRMI+raxoin/wLkZUvAQDk0bMgbLLaiYRjtsCZjg52XJ2gJ+7HxkQfAz7ZNl7kXlZVHbFCQ/MS2bUJsfjeAICurgjURDsdHfaP9aamLtOSBAB1TVb73eEY2ggB0tn7OJboe3fYuH8am611Tc2duPbeT7Gt1kjhIQiAEmPPVVM1s73ImjcBXYNUNR4dKMIutRwAMI3aJhmExDPR2hZGe3s45++skpJQ2hajnBMen8+H8ePHu65/8cUXUV9fj3333ReA9VXw4x//GCeffDJ+//vfY+jQoaivr2f2q6+vR0FBAYqLi1FdXY3W1lbEYjHG0lNfX4+qqqqs+67kMceJqmp5bX9PBy9UjsTUpONFawMUVbOFqSuKZkYcAMa13VPHn3b/xGL28+jP95YpkKTEx91hNmIqFmevdcyhwjfgXJxy/WYre22yMeiOOEdpaVS/sgGp3i24HJ94VBTqGvGapVhcNa+xJBjtiOY6+7WNx6iwepXtv5toWUi0G9tlaDGkqgkJDYzV/zbd+lhcFp2AU3Qh47EROcITjSpMG3pBBSD7ASWGeHNtyjDlWBKNVke3QRwlUYCq6oxOSRSNvqv1mxFd+pSpW5LHzYd/vx9B2GyJlwXYgxr0NO4LEupP0mAE/ZJtHzpKCmVGOL/W0Qif1g1ABJQ4hka3Yp6/FvH4GMfz91Nuo04qGWBc0czxEQXna0XuO7pGVnN7xCQ7ZF8+t5KUaE/XdUQ2vAcAkCcfDGE7615P5/4gbt1oTDVJTl+9s3pdw/P4449DUazBr6urw9lnn42bb74ZBxxwAABg7ty5WLFiBbPfsmXLMGfOHIiiiH322QeapmH16tWmuHnLli2oq6vDvHlsrgQPewb4iIRUkVr01vwEwhdkdGp/TwJtNNjTorTI5Ed/zfEZZHmRstu1ckot0NSeXjFKtygt3VUlkh5M0TIR//IaHiF58VDSRjxBnKyw9EQxS4evYJrc83onN8JDRLiktINUxeZrEQQBXZplUV0eHY9Ts4hslEQBtK0gypFXQRAhlg+H1rAFanNNSsKTLOs6IbFkzGnRsiQIiLz/EOJfJZIc+oIIHnA25In7GxoWaltZFh0ivNIRLRvTJ3lXBR3ciYRIaboOIVAIoWQI9PZ6VKEJJbKC0v++hCOjHUARsL6lGgDrIuKvP01cVFVjEi+y/U+IlknuKaodfkwlSTCroRMQ3Y1a85kRxu8Lwjd+Xwg1VtX4dBJd0m2lW+Mrn+h1wjN8OKvsliRjoKuqqlBZaeRlOPvss3HKKafgzjvvxCmnnIL3338fb7zxhhmWXlVVheOOOw7XXnst/vCHPyAUCuH666/H/PnzMWvWrF49Hw+5AR9FlYr905vz2YfJfEm/HPj2axo6sWFrCw6fM9x1kugv4K1XexLIC5tJmc9NgskSD9JwyvDb2JYm4Ym7WHh6OJx0pAuQLNMyFZZuc3tZuYbIvWhlWrZ3kG7LFqXlknhQEgWodRuh7v4SECTIo2cz60UB2KQMQWPxZCyvC6FNL7SRqXTAT4JOObXE8hHQGrYYoenjkn+gJiP4XQnCIzkQnvFyrUF2BAHyxP0R2OdkiMVWyRl6W1lkCY8A5ySKPEJ+ifttn07JGJL7TBo0Gkp7PU6SP0RZcSfEqHUtC8K1AMab+/HBGAD7EccnHqRhEh7VHqXFRzYaZTlEBHySuU6WBcS//giRDx4BAPgm7AfBF2TGKd1UH6myhvcm+uWbfuLEiVi8eDHef/99nHzyyXjuuedwxx13MKHqN910ExYsWIBLLrkEF154IcaNG4dFixb1Ya899AT8ey1VCCP90ue/HHQnCw83yTz37iY8/fY32LC1JZvu9iqcMvPuKSC5N+h+8xYePj+H0znKkj0CCkjfwhNxs/D0cDz5L2y3TMtM4kHNnfBYmZbdEw/SuzslHnSCJAqIrn7ZOMakAyCWsPXmREGACgmfjjgdb0ZmOradDnjC41QKRRqUCBFv3JqyvWT3OwlcMAkPde5H+NYAAHxTDkXo0B8zZIfflrfw8OTBDUEulDvotxNyQWAJrzhoNACj+KYk6FBG74vPio3kmcGY9S4i9wF//nSqDUWzEg/y7l5yP5L3HusythMeAKaVR4SGKfVvIvLe/YAahzRqbwT2Pd1Yx7kN00GqnFK9iT6vQjhixAh89dVXtuUHH3wwDj74YNf9CgoKcPPNN+Pmm2/OZ/c89BL4CI+ULi1qc35bswJ1EpcWeVm6ZeDtT2BcWntYaQnNdGm5W3h4s73TV71PtudZAdLPlOyeeDC9/d1gZTw2frtaeCjXmc2lxSQeZMN2nSw8TEQi79JyDBPWMantI6jN6wFBgn/28bYtBJNgubedDvhJ0EmDIyUyN2sNW6Drum3MaCS737s4lxb5f7xch7HibkCUHc8V4Cw8EluHK92JnCc4TkV9Rc6lKQ0aY657IzwTh+97HtrffhMAEIq3AjAsTJIkAoqW2sJD1dKiQQgQWR9PopETTcIjo7UzhkOCX2Bks5Hrzj/nRPj3ORmCIDLbGudmO11H9CeXVr+08Hj49sHm0kpl4aH+diM8ESbxILs/ebHvCdqePdqlpdnHmZ8E+Und6ZpIorOFJ124JSjsaeJBK7V/YkKwWXisfCgE/L1OC7pl2WjAl6S0BNlfEOwEyykPz3GhtZjc/D4AwD/7eIglQ2zbkH7TY5/NePPaF17DAwBi5ShAkKCH26F3Ja9/qCRxadEWHrVuI85u/wfm+zdiH/9mAEYpC9GlfAVNEHwSW4crHf0OYCc4ThoegRtXadg0+PY6Gk92HYDXw7MgySLCchkAoCBuWHhCQhw/DLyPQwIb7BYeTsOjuLi0ZC2C2f4tgGqMEW0xt7u0jPMlyQfn+40QtsB+P0Rg7qkm2TG2zZwYknuyPyQe9AiPh36BjEXLSVxapmiZKi3Buy7Ii3RPIBBspuW+f2lkAvKFqcO6Lrybgz8np696nyyaAuBs4ObS6mkGbtNVkSrTskvxUKMP9kzLUhI3gFsdLcCyEM30bceZhR+iQIjigIBhQQ8sOAOBuac4ngdpiyYYWWl4uInXScMjyH4r43LD1qTtpWPhkUQBsc/eREjvxjGhdZjuN3K7yePmJumnyPzNuLRyaOEROAuPIIrw7/dDrIiON48V8Rth3gEtjEIhgv8rfhOz5E04tXAV1DibwoC38DiJlvVYGGPW/wPnFX2AGbG1AHgLj5tLS0aV2Iphcis0QYJv8kG286HvuXTHyXRpKX3/ru1zl5YHD0AWFp6kLi3j/0iS0hLkRbonlJygydqeYJGiwdfFEkXB9sLlJzWnmjuyZC8OySOZe8Q1SqunomWbS4tdT4wFbCkUdhtNs7u05CQuLbc6WgBxaek4tWAFyqVuFApRFIoxxMUgiqa7J4QlLi36egmCkHEUm03D45JiQBo8BlrTdmgNW4Cx+7i2R/rjk0Xbc06itAKiBmX7pwCASsnIvRWDjKKhU9Lqp49zaaWr4QnYRMtOUVrG/2wtNetvSRSgSUF0aEEUixGcW/Q/jJKtVAtCaw0wtNz8zRIee/FQXYkh/OYiBDt3AQCGqTsBcISHG0dJFKDs+Awndz2DtkJjXWfpBJQG7EkFWZdWhlFanoXHgwcDNtFyBhYep6gXgNPw2FxaCcvDHsAfBoJLy/jbGHP+hcu7LZxInZtomUaya+lm4TH2y35MNW7CoUkI7XJym/DIbzNKK41q6VYdLXt/ZEnAUKkV5ZKRaHG635jwmosnJk3yRyYvWk+VjT2NJzxRBwsPAIiDxxnHa9iStD1yzwR89r4TwjNerAEUNmnfVmEkBNk9UzJLcNh7i69L5QZREJh+Obq0qFpq5B6gCb4kihBEoEkzMjRP9rGlkaSmrcxvJixdozU8opFg8d1/QN31BfSEG6pKq4Wu64yljf/gqEYdwv+9G4OV3ZjgM4r8tg/Z2/mcs3Bpmfeyp+Hx4MEAT1p6InDTTQ2PaltG4KQt6a9g8/D0//7SUB2sU6k0PE5uO1my50rhkcxal0yc3hMrH1/Ek4liEQSTNLB5eHiXlqXh8dksPEkIj4tLa6pvp215S8nkpOdBmqItVsnExG7gJ8F4EgsPYERq6UnctIQcOEVAkYzdUwWDNDX5rPqLW8SxSfuZTMOTiXaJJjkBn7toGbDuAfqZEEVDjN+oWqUdIroPH8KwesktW5n2XC08koDoimehbFkFiDIaZl2IuC4ihCj09jpXDY8IDUd1LwGUGNoC1YjqMlq1ECKDZziebzZRWr5+FKXlER4P/QI2wpPi4Ug2SZmiZToPDx/67JAfpr9iT7bw0OOuuBAe3qXlGpaeYgJOdimTVVXvyS1gEQR7WLogCM4WHgeXlmseHoexcNPw6EoURWobpiUIT5duFFdWdQHtpROSnodp4eESKWYKQhzI7nwKAvN4FSMAfwEQ7YK6c71re2RcaPcRISuGhUfHeH0bAGBd2eHYrlSiQwtim5yK8Fj9FEU2EWG6omWA1e3wyftI+wTkOaYJvSQJEAXBtPAAwFfqCNQIRhkKXytrAaOJO52HZ1j4G8TXvWH06bAfIzZkGmoUQ7Ct1m92jdIaLLajSO8E5ADWjTsfv2v5AW5rOxFiwCozQiMr0bIXpeXBAwubqDjVw5FkkiLvk2R5ePakKC02LL3vXxqZgCYzbqJl3qXlTHjS0/C4wU3Dk2q/VOAtPAJj4bEmvGR5eFRNRzwxTsT8b+bhcXgOyO78eETevR8Hbb4bExNuiX9rhyCmS/gsPhK6P3mRR9IWuV7ZEx5jv6KQUe+NFtbSEEQZvklGZv34F++5tkfuhSDlOiotMlxV3VEFFWIXgogCooS24HAsaj8aN7WegrjkPGGb/ZRINJzI/Ob/TgXa8uQoWqYcg+QesLJzG0RT5Cw8X2qjUS8OgaYDcrgZWnebuY5206uqBlXV4UccsxqWGOcz4yj4xu8LSRSwTR1kbFe/yVW0PExuAQCIlSMRKChEFD506wGTpPDIRsMje1FaHjyw4N+JKS08DsvoNO6apjNf9bzVfE9yae3JFh76a5ZMpry1hSdxedHwJI7p1EJPAt/sGh5rHWPhYfLwsB3VaZcWX1pC0+3bc0JpwIjMUbauMX/XqaXYJI7Fda0/wGOdB6V0B5pRWolrkW0GAHKcYqrSuJuVxzflUOOY29ZC62px3Ibc77SFp7QwYP49TEpM2OXDIEo+xCEjCl/Ke4VE/Jl5j5iw9PRPPsQQHgfRMjXDmhoe7p4xLDwG4VF1Ad9oI6BKQdSpZcay+o1mG06ZlofLLfBrYQihEgT2Pc1se6tCCM9m10zLw6RWY/uKEYyFyi2BpchYwtIkPF6mZQ8eWPQkLJ2ATBaapttEqm5RYHsCf0gmeO3voF0ypAwDubbkBWtzabmEpad6vyZzTxILD115Op39UsGeaZk2+btYeLjDqU4uLWrCseUpMvPwWMdSdn0B6Coi/jK80DUP/4weClEUENb9UCGlrCbN54vJNucRsY4UUePsGqlVMdyo6aVrUDYuddyG3Au0OLi00CJTQ03CM4KdjFOcL+knsahlE5YOsFYd5zw8Vltm1mM+d5MoYIsyGF9JE/Hv8GzEBKOEw9eKoUmKf/k/s40wVy5HUTWL9A0aA0GSzfPbphjZpbXG7fDFWnFA4CvM9W9mciOZ+1aMRIjKHO1q4aGGJuM8PJ5Ly4MHA5mGpTu5tMjLS9N1m0iVz4NiWnj2AA0P3UUnTUd/Bk3QiCWHmNTJCzad0hKS6JxpmQZPgtu7YuiOKNB13SQ8xNWSbL9MwOtpGAsPBMbq6HY8p0zLMuVW4Z8Fsjs94ZCioO1lk/G/6FQ0CoNYIW4K90OuNDykT36/ZFb5TlbxXBptRAOpTTsc1xN3J23hKSuyCA89YTNuqVQWHpHVSrGWi0w0PFa/nGppMVY4zqVFlyPRIGKJdBTeicyAIBj9/yAyBToEqNs/hdps6LL4D7loXDXHQKoYQR1XQLNWhC1aNaCrOLjpBZxWuBxnFn4IKWZVSjfHr3KkmXgQSM/C49XS8uAhS2QapeVEVMhXia6zvm5je+tvp8ih/gxtD+svDecoLWLhMV6w6ZWWSO3SooemOxLHr+7+EJf+9QPEFM3kx4UOhKcnQ0qsVk4aHsFNw2MrlKub63mXFmAngFaUFmlbh7LjMwBAV/nkxP4sQUylSzE1PA7uskxAShoEfBL8CauMUz0t87iJrM9aR4PjejNKi9HwUC4tuRUAIFWOyKiwJdmW9DcTskQjmMKlRd8PVlg6Ww7C0k9p5m9RFNCglaCjchoAILbudegOH3I04REpwkMsXG/F5wAAytVGYxsBGBIzKp6HhBgqpC5j+4oRjIVHdrlfepRpuR8kHvQIj4d+AX4i31bbgTXfOL8E3UCbYWnTL98+MwnvERYeuu99/5WUCWgyo2o6ttd1oLbFyBFDvijTKS2RXpSWtd+2WuMrVqOsOwBQGMyThYf6WicQRcEUrSZzS9KaCjLRiKJlHXLLJG7qg9rqoHc0AKKESMX4RDtiRpM4WU2+wntq4Qn4JPN5dNPwABbh0dvrHderDhYeIlqWoWKw2G60UzEyI4JnurRkouHJUrQckB3/JhAZl5bxP1/h3E42BZOw1A8z6kkqm5Yh2tkJXTcqwV9U9A4qxA6EI3EMTehwxMqR1jkk2tykVEMaPp05/nDFyERN3IEtWhEEf0GaLi1WlJ8OvMSDHjxw4Oe4TbvacfcLn6E+MTmm2h5gv4pJUjJre7trBdgzLCaMdWoPKx5Kk8udDV248eGV2NlgfFVWlAQBpBullToPD81baJ2CpY8R4PfZX3k9uQVI103RMtVHQ7Rs7xvPr2IM4aGqeEvsl7/VX5ZkxT5/y+jDsKmQ/CEAdotYqrEjBKfHGh6K8JgWHhcND0ARnnA79FjYtt4ULSfaEqCjSqnFaKkBk3y7IQk6IkIQQkFZRjocy8LDWnqMdZm7tEQBpguPhnNYul20DFjElri0AKAzNAJi+QhAVRDdtBwiNJxR+DH28tfgu6FPIUVaEBLj0AQJYulQ2/mpmobgIRdgpTwXz3bvBwAYpe/AWYUf4tyiDwAANUoZALCEJ5eiZbO0hEd4PHgAYA9LJ+gIx112cHdpAfZEc3T7tEVhz7Pw9P/+0qD7W98ahg5DR3P+sVMwc1wiT0haeXgyC0unrQpx01UgOmoTemLhsefh4V1axMJj7cPfc1Eqgoyt4u2ci4fWDWndrYh/+R4AwL/3sUzUEZ1bJtXYkfUKdz6ZgvTf7xMRSEPDI/gLIASMHDRObi3FTDxoTMYHB77AiE/uxuWlr+Mnxe8AAFrlQRAoqwjdD/d+shqe7F1aRr9CAdnRKsa6tIz/bYTHycJDlgGQJ+5v7L9pKfbxb8GgRPmM2f6tqIoZ2qdYaLApWKbb1jQdYlElPhD2xSfRMVB1ARVCB+YFNqNMND4mNysG6SwIyCgMygj4JIb80Mgm8WCyrOG9DY/weOgXIJPA4PIQs9zNouE0R9GEh99P03W8uWI7rrl/GZraI9byvn8GUyJfYen/XbkD9/9rQ15JH91fMvENLgvhoJnDzOtl1/C4uLS4FywvQKZ3o91E5MtZlgQXwpPOmTjDloeHWicyFh53kk00LrIsMhOkW0p+q5YWEPv0dUBVIFZNgDRsKsYPL8XwQYXYd9qQjIS4VmkJ4i5LurkraAuPz0dcWskfMoHoeBzcWuTeCPhESFBxeOhzAEC7FoSqG8faHRjLnAOQejKWuHxH2UdpGRYeJ3cWAX8PWKkMRKbfKuVONAmPqsM3cQEAAVLTJpxQ8ImxXBfgEzQcpK8EAMSKLOsOfQ7kesZVDVH4sV21qse/ruyHx9TjMPuEHxn9EAVcddY++O1Zc0zrHA8v8aAHDzkAmQTmT6vGVWfNQUWJIUx0czk5LaXNsLy/WNOBlV/WY3dTN77a3mo7bn+Gm+C6p1iybBuWfl6L3Y1dOWuTB01mCAkxQ4HJhO5iwWDS/8t2DQ+5RwgYCw/lRiEvWkkUHLUJPXFrJqulJbpYeNxcWjwZMyumaxq+2NqMVz/cAk3TGdGysnkFACAw6zgIgoCikA83XbQvjlswJqPcMmQ10cxkq+GprjQS/g0bVAi/bEyaa75pwDc1re7HTqLjsfQuIk4e0YgyMQyhoAyvD/sZft1yBm5oPRUrhNmJbdIneKYryylKK4vEg24WEcAiNKaFx0W0TKxZRLQMJAruFpZDGm6Il0vFMLr0IN5Q5gMAykTj2VVKLf2OcQ6JAA4kogATFs9PY6MBAO9HpqBrzCH4+f/9ANPHDzb3Gz6oEKOqiuEGZpzSLR7aj0pLeNXSPfQLmFoIScC0MRUoCPjQjKjrBO/khpCpyYx/uDTNCkVnqqhnMNntqO/EW6t2YPzwUhy897C09+spWMFr7l4aZKJ1qsidK2ianYTwuWbIV2hjWxhfbGsx++X3iQhHjb+dwtJPPXgcdtR34oX3NwPgNTx2C48kio7RJ7kMS6cnBEEQTCKhQbftw/fVx/WNzl/y1NvfoKahC9PGVpiEp1zohN7VAoiSOSHSyMRqwVdLz1bDc+x+o7Hv1CoMKgth2edGxuePPqvFss/rcN+vD3V0lYkliXwxLoSnUuzA+K8fR4myDQDgm/EdnDl1Gj7e0IgWrQjTig0tmJzB+ZJ7z0e5ANPdl8agUsMiPaTCPbOzQR6tjNN2DY+xnaLRGh4rpxgABA86F7uWv4n/rm9Da9F41CGEyfEtCApxrImNwdyxBzLHpMdZVXXzGXgvMhVfxIejVi3FoSlyFTmBub8zTDwYV7UePWu5gEd4PPQL8AURJe4FzMPRpZUkWZuuWy+cZCUn3PD26ho8+d+vAQDrtzT3KuGhOU4uRcvkJdhbLi0ysfPuBPKif/adjVj1laXj8MmSSXicwtKHVRZi5vhB+NfHWxGLsy9TOjIrToX7Olp4sj89G0FgMy0nt/CIggBN1y0iyPWNWA8iMRXt3YaWLRJVTIvfCBiVtcVBYyDIrLULyMz9wLu0sg1LFwQBg8oMEkCTS1XTEYurWPFFPXyyiAXTrUKfIufS2rC1GaqmY69xlVBVDQcHv0RJyxdG+6ES+KceCiEg4/afLsBry7fj8NnDbeeY6nz3Gl+JWRMG4dDEvrRVJxOyN3ZoMa48YzamTxwCqC51wziXFn/P8GU9jCgtS3QMGGPUOPYYfLB6HcZWFENT4ri77RjzGPsHg8wxJWbsNfNZ1yGiNpHBORNiZ55LD0TLut73GkSP8HjoF+BdA/wDz0N3cGpJkgjjW8rBwqNTRUVpwpPmA/jZ5ibz7/auWFr75AqMaDlH5ETrpeSLCmPhSbhuzNwnxNRtbNPSEWX2paNenDQ8hCCYEUbUOvoamxYeSWCicQjcBPPpgIyds0uLrpZu1/DIkoCYolOuPrZvJCt0VySO7kRl8LiqmccYoe8yjl09ybFvmbhpcuXSolHbzEZdNbdH8ejrX0KSBMyfOsQq4EkRHkXVsOiFdVBVHYt+eRAUTcdY2SBC/vnfh3/aERASkWiDykI452irCnwmVpqyogAu/f5M83e2xUMFQcCMcZUoLwmipcXZNczfn7awdC7po8hpeAhMy6csQZZYchXgNDf0+Wua7hgS7pZrJxlE7v5OB74klvfehqfh8dAvwIs/yf+uGh6HxTLl++aFnq4urTTnOvpBdRP05QtsSH1uyAmtrdHz+A5iXFqUeBigSG2iL91c7iQfT3gEgREFy5IIXdexwPclpvlqGFJB52GiNTy8FQXITWkJ8u6n51k2LN1OeAjhi5kuLZ7wGKLsts6YSQpVVTcJ2nB9NwBAdiE8mUQe2VxaOSA8h84exlyvxrZI4mNEZyywYmkVAEDvbEK4swOxuAZV09HcHgGUOEZIzQAA37h9TbLjhGwjrYztKbKUBRFIBtK0q2iZ6ysdls6K/o372O+TbOSY6KWsY1ptKpruKBjOhNhZ+2QhWqa1lX0sXPYIj4d+ATLpkhet7PDAM9s7LBZFq1gjn9tFc3Nppcl4aALV27l7mGrpOTo2/eLJZzJDJ5eWqeERRWabLi53Ev0SpxPyEfhkEco3H+GUwFL8pPgd5rqEYzThsXRA+YrS4nOqAKldWkSzE+W0TQSGhUdHY6sVVRhXNWg6UChEUKG3GMeunujYN1pUmmmUVhZzoQ2HzxmBRb86yBSXt3RY50F/QAihUiPXjK5DWfeGubylM4rBah0kQYfiL4ZQPCjp8bJNHshvn42rJxms5JPGb/JuMq3Z3PF40TIBSbXg99m1aHx+KTpCMB7XHO/xrFxaQubjlCyJZm/DIzwe+gXcLDyuhMfJpSUK5ota4dKY0y6cbDQ8ceqLtLf90PkoHhpnchHlpElH0JakGKfhMaO0iIWHJzw+1sIDsIRHFoDoqpfM33rcmlCdLDyiKMIn21/SuSweShtG2LB0+z68hUdO9E3XVMTW/QffrXsAvy97HmqzVWdKUQyt0hg5USqgfBiEYJFj3zLRpZB+9rSWFo/CoM+sMUW7LOkPCEEQ4J97itHnr9/G6QVLcWxoDdqa21CtGVasWNmYlH3KpLREsn2d3J49AV+YlZBKJ5JMtpc50TJAWXhkBwuPg9WZkFw+JxlBVi6tLMeYWGv72qXlaXg89AvwhIc8rJnk4ZGo8gO2gouabhIH+gWQrn5D6cPszPSEzFuuskVcyZz0ZYOkFh4qSiuuqLZrxmt4AOr+EHQo65ZA77S0VQi3AzDyjDgRnnzk4bGLlmkLj0CFJDtreAA6SisxKWxeieiyp1AEACJwWPvLmFxUggIxipboCGj+UgyR2ozjVrDhyDREauKWUxAeS0eSOw0PARFfN7dThId7ruUxcyAOHgutYQv2D34DAAiv24523Zii4uVjUh6HdUtlRloyETxnCoG7B+wkmbPwUKJl+nmnoxd5suIkxpdEAYoKRFwSP2Y6RqRNs58ZjJMsCYjG2Q+tvoBn4fHQL8CH95qZQt1mIyfCQ00wVk2gRPtUhABt4UlXBMwQHl3v1fBKxqWVoxcGPeH0RLSbCk4aBFO0TEVp8dYdwIjSIjAJjyDABwWXlryB2KoXme2FcJv5d3fUHqUliYJjyvwehaUTl5ZDtXQ6Dw89xLpJeNjEfCYBrN8EAGgum4YGtRhl6MA0/06MkRtR1vgpNE3HEFJDqtSKduKRSVi6PVIo6eYZgSTlc3NpAcY4BQ86D+HKKXgvMhVNahFC8TZUwSC0asXYlMfJJoLIaftcu7RIv9wyLdtcWtR7jHVpWRYemqz4HXJU0e3yFdb59ZmAub8zITxm8kH3rNu9AY/weOgXsFt4WEGr2/Y0JKr8AJnkyCSiaT3T8NgKOPYi4clHtXRGk9TbFp6E64a28PD6HYB1aRFXlCQKmOXfhjFSA+ALIbDgDGzXjCgfRNrN7SO0hSdO5eHJk2jZKh7KWnhMlxaThwdmfwArhN68Vxu3AQC6B03H/R2HYXN8MHYoFQCAstYvoek6BkuE8FS59i2bsHTSy1xaOUwLD+XScooakgaNxvbpF+Cl7nm4re0EfBLcD2E9gDq1BHrF6JTHYVx4mbq06LIUORYtWx9dCZcW59LkPWiCYPWBdWlRFh7q+rgFUZBrGHUhPE7WzlTI1m1oWi/7uGK6R3g89AuQScC08Ej2L5xUkEQr0Rv5UiWmX13Xza/sKKPhSa9t/ou0N4t40kfKmWiZq2KeLzhnWmbrF6ma7mLhYV1asc/exDR5OxYEDJeHf9ax8O91FDpQCAAQKMLDuLToPDy96tJyES2DvTeJGFWWRei6BjVBePSK0ajTyvDXju/i0U6janZp5xYgFsEQKUMLT4rJjZ+7cunSctLwuD0/5GMkCh/eVebg1thZuLXtJEg+e54hHkxoeYakJZOkhZnClmnZpXioub3oEpZuipZZDY9TQVzAGgM3DU9v5eEBPA2PBw8MdM7Ck1K07DBLiaJghteyFh6VcWnRWXjTdefwFp7eFC7no5ZWnIk6y0mTjqD7Tv7kK1UDQEe3PbcRHaUV7KpFdOk/caaPtCvAN8nILtsFI1RZiBokQFE1pn4Tk4cnh6UldF23kghyWXPJMoHalj8embToTMtaWy2gRAHJD1/lMABGDpoGrQS1aimqpTaUNn2GUtHIcZPMwpPJ5GSbdHPp0qISKBI4WXgA9tls6YxC03TosGfZdkLuXFp5Ei2bFh7n0hLW9oKlYXRwCQdkkSF0fEg6AXm+cunSYrVO6e/Hu2/7Cp6Fx0O/gOXSMn7T1X6dt7cvk6jwR4XTRdBh6U7HTQVeZNmbLi2alOUqhJyecPKpR3L6krcsPNbrp92B8NAWnkB3HbNukzAKYmE5AKBTN9L6ixFDw8O/4Jk8PI6lJVKfhxPoe0A0NTy8hcd+DHI5TQsPFZauNWw12hs0CgVBP3O89bERAIDBuz8AAISFEIRAoWv/stHwWH3PoUvLoc4UnyeLgLZGdHTHbdm5kyHb8hBG+9nvmwq8hYdPsmonm5SFR7NbSHkLD5900GwnHy6tLKqlA/3HwuMRHg/9AuZLgBMt84UlTThqeKgoLfMr2iJOTtaRtPPwpFHRO1/IR/HQvtDwEFhRWtYLk5ROoMFoeLqtOksxXcJqabb5u1MwCA9xafEJDGNmHh6RecnTZDgb0PeOcx4ewRahQ//NTziyLJruLGnQaDPxIAEp/BiKGOU32qXypP3LRsPj9rsnCPntE7JbtCFPVhUuhDsZso0gArj8MnnS8JilJfiwdIfEg855eIwx88lslJarSytBAMNuFp5eDEuXTQ2PR3g8eDAndT4s3dXC47CMzcNjt/A4WTLSidKic/iYy/rIpZUz0XIvhNnrDuMG2FPqA0CHQ7kOJkqryyA8b+v74sqWH6HON8Jc153Q8IjRDgBAmNMDWXl4LA3PVN9O/K74WYyT69KycMUVDZ9ubGT1X9RN6JaHRzQnO1D7ORMenyRCa9wKAJAGjTFLSxBsVyvR5rMqW3fIKQhPBlYLfu7KqWjZ0cLjouGJZm+N6EkunbwmHjQj9YxzVkwLj3OmZTosnX424wkLT8DHRWm5WHikFBaeXtXwSKzUoK/gER4P/QK2aBcHky6DVC4tjY/ScrY2pKPhcfoq6U3Ck4+w9HgvWHjc2iXXRBAsF1NH2MHCQ7m0pE6D8LQI5dDBWmq6BEPDIyY0POGoM+GhS0vs69+IMrEb8/2b0hKuP/XW1/jr8+vwzLsbzWX0/eTk0mLD0h20TNwXtiwKpoVHHDwGfpnPtyJgU8He5q8OuSJpnzPJiutkZcgVgk4WHpeJz0lgK8DdbUMj2wKgQL41PM4uLafIPrLcqbRElISl+0SG0PkddGkAHZbuIlrOplo6Z8FMF+Tjxcu07MEDsqmW7mA5EOlaWqxLy404pTPZ0S/nlK62PIAmV/nQ8OTLPedGzthoGuMV5FSQ1XqR6xAShKdVLANgfTECQLdgWXh0TWPKSgCsgJ1YeIbKrQCAYXJLWhae99YahTrfW7PTXEaTGMs9Ye3D1tKCbT/ealGstQDxCCD5IJYNgyAIKOCsIxv9U6AJxrLOlBYea0JNNTnZs/3m0qXlYOFxIzwOSfIO2GsoQg5WIh49yaWT1zw8vEuLiJYl+z0DcLW0HDKVG5mWrT6m1PAQHRR3XqmSUSZr06m9ZCD99TQ8HjwgWR4eF8LjsIyx8JBkc2ZFbhfCk8ZkT2cHJf7yXrXwgCY8uXJp2S0OuYZbXxkdTeI6O4mWiam+VOiGoEQBQUSHUGprIywUQNMBATr0SIfNwqNwFh4JKgYnEvcNlVqhaeknQyN1oQD2HiD8wD0snXJLEsLDTRjlsVoAgFg5EoJonDuv4wnrAWwZdgw+i41AbWBs0r66aUScwG+SUw1PwD4hu7k2iIaHJno/OtK5VhiPnkRpCZRQOOfFQ7nkkynD0lNEafltLq1UYenGmPKWtmyIXdaZls3Eg15YugcPpmtJ5F467tXSnUXLAkd4zIRXLsQpHXcOrQcyXkRqr4qW8108NF/kza2vjF4icX06HETLJLKjKpFzRigZDIRlZh0AQBTRpQZRLESgd7ciHGVf7KaGJ1FaYojUDkkw+uYXVEhdTQAGww10/piq8gLb+dEWFFstrcTf9EiQ68mHyJdEDMIjDRpjLuN1PIqqYXflPDz3WQUOkJLnpslkAreVN8jhp3DQ0cLjouFJWOdOOGAMauo78Z15I9Oy7gCsKyobDZIkClA1PQ8aHuN/W+LBJKJlx2rpLsVD3cLSSQAITXjoBJ89dWllk3iwrwlPn1h4HnzwQRxxxBGYOXMmTj31VCxbtoxZ/8UXX+Css87CrFmzcPjhh+Oxxx5j1muahkWLFuGggw7CrFmz8OMf/xg7duyAhz0X5Lnmv3oyqZbOiJa5TMs9sfCY5Em2vgJ7VbScj0zLav41PGlZeBIvbmeXlvEiN+tGlVZbOXwosiBAQLtm6Hj0cFtSDY9fFlEttTLrfR07kQxba9sdl/O5owC7xqG65m2cV/Q+o3A2RcscqyiKGIUyxUGjzWW8SyuuaOb1ElJMzFbdsTQsPHkNS3fQ8LiGpRuT84ghRbjw+GkYVVWc9nFYIp29uyZ/Gh6XKK0kGh7n4qGshs1VtJzYhpBInnj2NEorI5fWtzUsffHixbjnnnuwcOFCvPrqq5g1axb+7//+zyQsLS0tOP/88zFq1Ci88MIL+PnPf44777wTL7zwAtPGP//5T9x00014+umnoWkaLrroIsRi9pemhz0DvJCPPKwZEx7TwsNqeHpi4aE1IGIfEB7ampUr7VCvWHhcXm5ykogY2v1BrDhmVuGyoeYkT7/wRQFo14MAAL3bgfCYtbREFBf4MLeaXe/r2J30PLbVdph/04nx+C91gCUKElRU7XoXs/3bUK42mMvN+lvMhKOjsNvQCaWy8PB159xg1ixLY2LLZ1i6o4XHRYsWdXG/pANGh5NF/zNxAWYCtzw8fIAGgQC6eKie2FdnMy1T+7i5tPg8PPyYZlMVntGoZRSllbDwfJsIT3d3N+6//35cccUVOPbYYzFmzBhcc801GDlyJFavXg0AePbZZ+Hz+fD73/8e48ePx/e+9z2cd955+Mc//gEAiMVieOihh3DppZfi0EMPxZQpU/CXv/wFtbW1ePPNN3vzdDzkEO6iZecHRHdQ8Yi0aDltC0/qvtFtpRJT5wNMHp6cFQ/NP+Fxa1diLDzsK6isyHLTkLEeLrcAMAgPmchol5YgCGjXDFeT1tViIzyxuGXhEQQBswYZLqo2oQQA4O9MTni2UoQnFqfD0snEZW1LzwHlWgsE3Th2oWpZiUyXFnXulWInZDUCSDLEiuHmcl7Do6iaLYWDG9wmVCcIDsLZXMExD08KC48TSUqFTEppOO4vpU8QM4EtD49mEXDAISxdpKJUE8+7olpZvfnioW6iZb54KJ8eICsLTwaRfzR8/UTD06uEZ/Xq1QiHwzjuuOPMZZIk4dVXX8XJJ58MAFi1ahXmz58PWbYuzn777YetW7eisbERX375Jbq6urBgwQJzfUlJCaZNm4aVK1f22rl4yC3cRMvuGh77Mkm0Eg9aVpkEAXJ50NLT8Bjb+PqBhSdXx2XD0nPSpA2uLq0kIcDlxRbhEUUBMlSMlhqNbasnmuNP18QSBKBZMyK19M4mW/r6uMJGqagthgvrK2kyAMDflYLw7LbICm3h4Um60Rfr73Kt0fy7UOtg9gFYS9co2dhWrBgJQbTefYVB3qVlZQxPRUqcSni4wcmtkiv4HKp5x1NoeJxIUirkQsMDZBe9lAxmaoLEbWkVDyUuLXZ7WrRM7pcYVWXcpuFJl/DkQLQsUJXcMxIt95PEg70qWt6yZQtKS0vx1Vdf4a677sLWrVsxYcIEXHbZZZgzZw4AoLa2FpMmTWL2GzLEqIa8e/du1NYawr6hQ4fatiHrsoVTnZ2ewvpq8ALikoGuSSRJovlFoOnO18XpYfP7KQtM4oVKXgZuc7qu6ymvu5bY2yeL1oQl5Od+cQQ1WWi6DkkyLBU9ubcYMpKvc3F5Hwb8snk8UjmdoLIkSG0nYbTcCJ+gQgiVwl85HJJkhKf7fZLZhigKaFGLABiEhwchv0XxRnT++yXobUaZis2+SZivrIQ/0gxJUCFIPtu+iqoxWaBjimYeV6Be/GQZ/dVcqVl9KdI6jEzKlEmRnqgmysa7yzd0InMtikJsn1RNM28HWXKu/k7gS7g63KrEM9ty60Xq/soFQgFWMOv03CmqZrqeC0O+jO9J+p3g96U+Zx5kDOj7Mx2keg7NpJQiKRBrHU+WRRthkSTRdFOpmp64byz3aTAgM/uEApJjf8kyQtJ592jA77xfKkiSAE3RmWcwFUh/+UrxvY2cEp6amhocccQRrut/+ctfIhKJ4LrrrsPChQsxbNgwPPPMMzj33HPx8ssvY/z48YhEIvD72RoygYDx1ReNRhEOG0XznLZpa2vLuu+iKKC83L0uTU9RUhLKW9sDAXQSrpKSEIqLjYlPlETH61JQaI9QKS0JwednH6yCAr9tOxqCKKCoOITG1jCGDnK+/sG6TgBAICCbupSCgkBe7xcaAc4UXVxSwExQ2dxbIm0SD/jyci6tYeeEZ+XlBebxaNeFLAkYXGH1Y/CgIkxIEIGCsTNQUVGEYMAgACVF1vj7fBKaNIPwCN3NkIv4KC3jXhjX+CHiresAAKFxs9DRPBhatwBR0FES1CEX2ccgrrAh67G4Zh63pds4P59s3aOFBdZ9WUERnmJ0ory8kGmvmHLfTfYZVqbyKXNRQF2LQYnxEATjo0DVAX9iDEKh5NetrLTAHJ9U15e/VoKQ23dWQcjHRgjJ9j7RwvWhVSVZTYpknMrLCjO+p39wxCR8+k0DZk+rzqrOlNt4kcm+sNC4Z8mzV1IcNPoosfdrKOhDWZkVDRgqDMAfSxQO9RvjVkodq8LlXEOcO7SSahMAKiuKUF6W+TU+5dAJqG/uxoTRFWmL20sTHzJ6Yvu+mg9zSniqqqrw2muvua5/++23EYlEcPXVV+OQQw4BAEyfPh1r1qzBE088geuvvx7BYNAmPo5GDZ97QUEBgkFj4GKxmPk32SYUyn4QNU1He3t31vu7QZJElJSE0N4edhVxerC0EaIooL09jEgi824kGkdLS5dt+87OiG1ZuDsGja95pSTPsRKLa/jrU6vx/tpduO68eZgwotS2TUurcV/Qj3Zre9ixX/lAd5h9HpqaO8308tneW11U3puu7qjruXy9oxWrv6pHQUDG/nsNxeAMXpBk3HiEu6zj0e66UECGTllAiv0iFgxuA7oAfdBEtLR0QVWN6xmPq2YbmqqZLq14WwMiMjteRK9U0r0dAFD4nZ/BN2FfqE98gm7djyIhita6BkhxOzmOcYnwIjEFzc2dEAQBrW3W+ZG+RCKWNagkZgmVC9V2tLR0MS6xiiLjeJViBwZJndAFEZGS0YhS16IkaEyGwyoLsbOxC/G4at4P8ZiS9B7s7iLh9HrKe7Wzg32eBEHI6TsrwFkCOrvs91xjq/Ex65NFtLeHszqOJApQVB1dnRG0tGQ2ve07ZTD2nTIYHRkeO9VzSN5JHR0RtLR0IRI17pFoxHi38RGKsZhiXruOrhjO//2b5nPik0XjPopY+8Rjzu9I/t0ncnbuzo4wJD3z63v8fqMAAK0uz7cT4glXZTjxfOTy3iopCaVNjnNKeHw+H8aPH++6/vPPPwcATJ482VwmCALGjx+PmpoaAEB1dTXq6+uZ/cjvqqoqKIpiLhs1ahSzDd1uNsinf1FVtT73X/ZnmKZOQYCqaia5UBTdcdwcx1LXbV6UVNEamqZhd5Px4O5q7MSYansYLIlykEXB9MPH42qvXU+VqzsUjarMeWVzb9ETuaK47//Ia1+gpsF4mW6t7cDPT9kr/WO41PAhxwSAcsrKUVVeYGpzfFAQ+WopyiOJkPHqSVAUDSSzjSSw90CrVggdAgQ1Dp9if/kXC2GEYi0ABAjDZ0BVdQgCENb9KEIU8e4O6CX2MeAJj64b9Z58smiRdEGw+pKYmAKIoUiztD9FegcURTPrIQHAxOGlmD1xEELbvwYAxMvGQhUDAHVeE4aX4ppz9gEA3PLYasTpa6Unf2eNHFKE6ooC7DNpcMr7g9dbiaKQ03dWgNOPxBLnoWoadtR3YtSQYnQmPnKCfinr44qiAKhG3bzeft+mGi9y7fjr56TLI3nJ+EK4PklkngPAeMc5HZe3vvAZr/UU908uMbqqCH5ZxLihRqBAX82HvarhmTt3LgRBwNq1a3HMMccAML7wNm7caIqQ582bh6effhqqqkJKmPqWLVuGsWPHorKyEsXFxSgqKsLy5ctNwtPe3o4NGzbgrLPO6s3T8ZBD2MLSTXFw+g+FJIq2UMlUpmmjxpZxDLfQdTNKSxbNcNq+Kh7q9Dsb0OearD36hdvlUO8qGdzz8FjX6IzvTMLM8ZUQBAFTR5ejO6rg2Xc34kelqxB5xyACQmE5xFJDs0c0PrSlSRAEaBChBErhi7aiUGkF4EeF2AFFl9CuF2CMbFhbxIrhEPwF5n5degCD0QFEnC0gTkMTjRuEh79nSZsAMFQ23Oua6IOoxVGod0HXFJMwi9Cg12/E2SO2YHfdV8bCoVNtxxIEAeOHlaK+xSDlcVWzCfzdUBTy4Q8X75d0G4J8ZloG7JMteaaeeWcj3lpVg+/uOwqzJxnJH9Opm+WGicNLsaupGxWUFqyvwZcX4UXLvHg4mX6KjE0momUCPqeTnONotGQYU12Cey472LGQbG+iV48+bNgwfO9738PNN9+MUCiEUaNG4fHHH0dNTQ3OOOMMAMD3vvc9PPDAA7jmmmtw0UUXYd26dXjkkUdw4403AjC0O2eddRbuvPNOVFRUYPjw4bjjjjtQXV2No446qjdPx0MOYauWbtbASj9/jiQJthd1qoda03VT4OwWMknIgU8SEU+REDEf4E81F6ZgWkuSjFPSYfCZkrx0Eg8WhXzYb3q1+buk0I/bf7oAgf+8BbQB8vh94d/raJNIfP/QcdhvehVjiSPv9XigHL5oK0JqO8rEEK4q/Rc6tQBuajsFYxOER662yhQIAtCtGW4lPdrp2FenjN6xuAqEfOZ4MOn2E38OlYxQ+kjpGPiaN8EnaNC7WqDJZQCAHxe9i9iSnZAAjEi8hcsmzXHsAz1miqKBeCFyyUnyWTwUsEcIka/7t1YZlv3Xl2/H1DHliW2zn5YuO30WVFW3ibD7EmZkp66jOxJnMnTT/xPQmZZ5EDFzWqUleMLDiZZznWAxFbLRReW8D719wBtuuAH33HMPrr32WrS1tWHatGl46KGHMG7cOABAZWUlHnjgAdxyyy045ZRTMHjwYFx55ZU45ZRTzDYuvfRSKIqCa6+9FpFIBPPmzcODDz4In88eZeFhz4Du8hLINvEgQcoILF03j+GWq4cQIVkSmJdXb4E/Vi7IVjxNCw99rEwP65ZDKdWLr7LEj84OI5IqMP8HEIsHmet8soSxCbM4ASFDsUAZCgAUqW3YP7ADAUFBQFIwTGoxLTxy9QRzP1EQ0K0bLjU96mzhoc854JcQjammDscpASDpS6VoEKh4wRB0NdZisNQBrbMZemkpBOiYlBApy6NnQywbCrFyJKQh41zHhNzHqqZDdQiH7ynymXgQsHLAFARkdEcVR2tqJEryxWRv4REFAaKcY7bWQ5B74um3v8GD//7CzNvkVjyUzrTMg2QfT8fCw5NYG+HpRQtPf0GvEx6fz4fLLrsMl112mes2M2fOxDPPPOO6XpIk/PrXv8avf/3rfHTRQx/AMtMbv1NZeJzizOnEgwSpsolqWmrCQ7u0UhU1zQd4K0MuCE+6iQdZwpOhhcdljFLl/9DbGwBNBWQ/hKKKlMchzcUChoWgRG3BlMBmc/1k324zz41cPcG8dQzCQyw8boTHOodggvCQRIaqg2uJ8IRy0WhPCZWjRSvEYKkDemcTtJKxKBIikAUNEAQEv/NzJu+OG+i8Q8Q6l0tSwjeV62zDY6qL8eG63Rg/vBSfbW6yPWuiIJhEMpssy/0ZZGxJvThSq9Yt8WByCw8hPFSUpctHHf/uC/pliIIATdcNYphrM94eAK94qId+Addq6Zm4tETBpkVIaeHRdNNF5O7SoktLJC95kQ/wh+rpsXVd5xIPJiM8GvV37l1aTtBaEzWlSodC4FMAO4C28ADAlPgGJqXqkcH18Aka4v4SiKXVJhETBKS08FBplxD0SWgDKAuPsc5Jw0MIjxo0CA8Aw8Kj6+Y6oaA8LbIDsGNGCFcmqf1TwcmtkkscPmcE5k0Zgi+2tTgSHp9P7FGW5f4MN2KRvFp6CpcWk3MoPQtPwCeaOXS+jdYdwCM8HvoJePcA+frJxJIiUZmQCVJlTdV0UBYe52MRcuCjSkv0pkvLZuHpgYbn/n99jq21HeakCQDJIlPp8dczJDzupSWSXxO11agpJZYNTbodgUl4/OXM8k36CIwXalAoGiG8bUPnYYgggJgHRUGgNDwuFh4zq7FgTiyxJC4tcv9VJFxaasgiPHpnIzRNN9elY70ioF0YVnRY2runhK28QR6+/osL/OZzzT9rPkm0ilz2QLTcH5GK8AiCEXNFRoROKsrDTxJ2ZqHhCfhlSKKAuMO6bwv6XkXkwQMcRMspiIWjhcfBTJvSmpCGhscqRNo3tbRy6dL65JtG7G7qRlO7lXdFdRljnRqbZNu5gUS00VfESWfFw7TwlKdLeIz/IwGLQKyOjsErwlGI6sY3naYDHcP3te2X2sJjWYNIhEw0QRbNIqCcS0uEhlLRyOWihsrRoBoCa7W5Bpquo1wyjiUWWdqk1OcomKSHlM7IpdvJbuHJz4ToS+hr+CKSfsrCw4ew7+lwG0r6+jF/C+4kyc9FaUmi4Co+thEen/X+6g8C4r7At/OsPfQ7uFUQdrVmOImWEyUXaKRyaemUhsetkq+l4aFEy73p0uK6lS3h0XTdMTdOuvXKMo7SItFt1BdoOqZ0LUMLD5kcooFyBA46D2/4jsJjXQdB9wWxVTFIxVfKUOiFLMEQ0hAt0yVPyJd0LK6iO6I4hqWLgoBSsRuioEOFBC1QjM2KURpHq98CPR4zXVpiBhYewJqkiIVHcKvdkQXyreEhcCvm65NEc1l/irDKBdzII/0s8OTH7TkhSytLg5g0ohQH7FXtuB3fPmCQJbMMxrfUwuO5tDz0C+hc5Ekq0bLTUkkUbBEPqVxaKqXhcUuERYiQr59YeLIlW/G45jhublY0Psoq0+OS7f2yZLrQUonIdV23LDxlw9I6Dl2N2j/1UHy5YiWADvgkEf+LTEWZ2I3Xu2fhVO7YoiigK03RMm3h+d+nu3D/vzdgwvBSox3q+IFIs+myisglEAURzVoR2lGIEr0LaNpsaXiK07fwAIQsqKaFKacWnjyHpRNYhIdzackSZUkdWJOx21jSzwIf6edGSIjwWRJFXHXWPkmPS1t+/D6R0QZ5Gh4PHvoQvHsgFbFwyo+Sr7B0RaFFy31g4cmRSysad8587KbN4SeljKO0Eu3SX+ypJjO9vR6IhQFBgFhaldZxyBc06R7habIkYn18JNa3jQRg/6oVBCCcQsNjubQsDc9XO1oBABt3tjHtKtvWYNqnizCk0KjrFZZLEn0TsANDMR0bIdR/YxKizC08CXeQkgcNT57D0gnofEJsIVW6QObAsvC4jSVr1aG3d7fAtHfHHJenap/ooqyK8ANrjNPFt/OsPfQ78BEv5KWXrruF7GMTLafwVeuUaDmeKtMyRXjccszkA7lKPBhxITzuFp6eWZbI/nQUSbKaN2rdRnS/egsAQKwc7Vi93Am0hQewzocnu/xXLZ2HB9Fupo4XgenSEgQEXMShJHIvtubfAIBBkkFowr5Ss287YLjnpMaNloWnqDKt8yMwXVq9ouHJWdMMCGlTVM3U7ACsS+vbYuGROFco/bfgEjZenKIYslv7RBf1bbfweITHQ7+Axrm0xFQWHodl2WRaBiwLjrtLy7JU9IVLy2bhyTJCLOpS28o19J8nPBknHiQurdQWHl3XEfngUejhdogVIxA6/KdpH4dYeEj/yHj5OHLlZOEheXgAHYjZiyFa92WS8N9Es2IhGyUWkUvNvm3XDcIjN3xtRo1lIloGLEsZXcMrV+gLDQ9JNEhANF/fFguPm4ZH4N6BAHDcgtGYNWEQfniElTgzFRjCQyw8nobHg4e+ByMA1XVTe5OJS0t0cGml8/IkLbmKlqlMy1KfuLTY39kmPXQjPG6nwo995nl4jHFjCY/z9dCatkFr3gFIMgqOvwpCsCjt45B3NyEnmoMrDbDfC4IgQIUERfRD1mLQo12245JTFgTBtcYTmZj0SAezPOIrQXGibw16GYSCEuhho6BoWPej2J9+5XkgzxaeXghLByyrm6LqCMesOm2qppv3y0CzPrhaeDirjvV3Yr0kgFSAmTVhEL53iHthbsf2k1p4BhapTBffzrP20O/AJx5M5Tri+Y6QCOXk89RJYvqxLKlEy6xLa88LS4/EFcflbhoe3nWWdZQWRTzcCGj8qw8AAPLoORmRHcCaLCwNj5VGgNnOZVJXJIN4OOl49HQsPIQEdzUzy6P+Msb65J97qrlOhnsleTeQ8yF5oXJJC3otLJ228FAEXNU0835JFWiwp8HdwkOJlmkLj6mzoTQ4WYTqi04WHoe2v03wCI+HPoeu64xWAkBKSwpPAsw07TYLj73chBtSiZZpl1bvhqXnJkrL3cLTexoeJ5eWrsQQ37gMAOCbfFBGxwDcNTw+rqYSf2xyW8RFo7K2E+GhEw+6pfCXRAG6bhQHpRH2lzNf974ph0ANGpFdNfqQVKdlg4/vfw4nLZtLK0/zIR192R2xCLii6ubzN9CsD27kkU9nwP/NiI6zyD5NW8oslxb7fv22wXNpeehz0POtKApQQUVpubhv+KVWIT7uS1UkuXlST9Yp8/D0mYWH60+WgumIG+Fxi9KyaXhyEaXFTma6riPy4eNAtAtCYQWk4dMzOgaQPEqLhpNLCwDiSS08ZFvA7/KVLYoC9HAHoKnQIeAfHYehQIhiqL+Csj7pEAQBLQf9Gp//6wl87ZuKmRmeJy/Czq2GR4Ag0OebXwsPAHSGrYgjlcqHNdAm47REy1yUFsA+96EsCqrS97vl0rJXW/82wSM8Hvoc9ERqEp7EA6kn1vMvd37uJf5wZwsPkI4HwbW0BEV4+kbDkxsLT8w1SsvluD0mPE4aHvb6xNe9AeXrD4xCmgefByELwaprlFaaLi2L8HTa2tZJGQoxiYZHEEzrjuovxob4CADASaJg9o0Mpeovxovd8zG4LJj2+RHw55NGmbGMIAqCY0HUXII+h85EThnAcJ/SHxYDCa6JBxnCY50zcWnR9e6yyT5Nt8+HpQ80nVS6GFh3loc9EvTEaka8UC8JRysP79IyLTzsZm5p2sfLtTilYCV8oMzqrsVDSdSPJYruSwtPtqJl17B0F4tRTxMPkn66haUru79CdMVzAIDAgjMgj8zU5mHA0snwLq3kYenktkju0kpsCwF+2d3CQ/Q7SqCUaZ8oyEwy5lB/K13wUWe5FhbTE3O+wtLpa9ARpgjPALbwuJFHNiyd2l6wE55sItcYDY+fdWl5eXg8eOgDKKqGLsqXLzp8gThNtPwi18rDLrWbvle4EocGv8C8wGZzWcoorT4KS8+VaDnjKC1OdJypJ80pWoqIJfVYGJG3/w7oGuSJ+8M3/cjMGqfAuwDM46YMSzd+R6VEcc+uVlvbTC0tfxINT4LwEI0OaV/g+qb3wILilEcol2DdKvkhHXS2306O8NA16wYSXF1armHpxv89LVBMC5OJdVI2XVoDi1SmC8+l5aFP8aen15qZawF7pmWAWBrYr2ud0+S4ER6nZIQBxDFUNI45Ua7Fx9FJANxFy7RLq6pjA0ZJXdC09Oo85QK5yrScqYZHpYhDXNGg6bqpRUkHVh4eSrScID/q7i+hd7dCKKpE8MBze6QZsTQ8rBXFlnjQVlrC+L9bLjP2a6+3tc3U0nKz8FAuLSVQRi0Ho+EB2DD3TMETuFzrbPjyBvmCLItQY6rNpTVQw9JFl3g6erzdQtR7gqRh6QPMipYuPMLjoU+xvZ7VTThFKDhO8DYLjzEZCNyDLAr2L6yRchNEwWhggq8WPigYJHWgVXVOBEeIULBtO6Zufw4jiwN4TZ2a/MRyCFNICuO0c11agidU7d0xtHXGWNFx1OpLuu/jZHl4SL0sach4CL5A2ufgBJtoWWePReBm4en0lRv7ORAeq5ZW8jw8Wqdh4dEoC49o+LSYPtFh7pnCJlrOsSGEJjn59HjIooAogI5uTrQ8QMPS3Z4XxoXoErHVEziGpZtRWgPLipYuPMLjoU/BW1VEylJDokacJnh+kjY1PNy7gndpCQDGyg3m7xIxgoUlr2Go3IqHug5P2kd/3ToAQJEYxeDoNgB7pT7BHIC2WMS5GkSZIN2w9LtfWIfNO9tx7nenAGBdUqqmp+2OMScwWTTJGpnMtNZaAEi7Inoy8C4AK9Oy3b3ptF+XVAYA0DuboGsKBNF6LbJ5eJK5tBKi5VA51b5gs5TQBCpT2ETYObfw5K9tGoS40RoeRdPNqMCBFkGUzvPCvKNydPr0ONry8AwwK1q6GFh3loc9CrquM0JhAexEQL5Ckol0yeZ8wkICkcvD4/dJGE0RHgAYKrcCAI4MfOoo4I0riYl792fmshHhr137lGuQSde0jmRo4WnpiGJXY1faLq3Gtgh0AA2tYQAs4clEV0AifiRRsL4syTm05Y7wiKYw2PhNEinyBMGNMITFIkDyAboGvZNNHkiGRkxm4REEaAnCowXLzOVGMsxEn0x3G3vsTGATLefYEuJU3iAfIDoSRsOj6maiy4Hm0kpnKPOhn3IsLfEtD0v/dp61h34BVWOVOPwL3BQIO0yyZCLifdJOVZ/pZgM+AWPkRgDAl+pIZttRchPiOzcwy3TdSIhWIXZAbN9tLh8R/cax2GQ+QI5CLBaZRmnd9s9PcMPDK9DUHnFunzsNQkKJC4whPBmQLas2kkU6yZclcWmJZdVpt+cG3sKjump4nF1aOgSIJYONNji3lsaIlt1raRHRsmYTLSfO1yZaTvPkKMiy/d7OJZzKG+QDpoWH1vBomnXdBpi7JR3ymEzDk+11ZsLS/V7iQcAjPB76EDZ3lkMOHcClOrhpPRCZ/20vC11BkWBYKg4MfImFgWdRLEagQMT7yt4AgAa1GB9HjKJ88bWvMfuTl/AMXw0AoLNoNLo0P0J6GGrtVxmcbfbQuQk8Ew1PZziO+pYwFFVHTYM9zwxgt9qQaBniAsvawkOFGZPrI0si9EinWXdKLO054TFrWZkaHpcoLZdMy5quQyypMv5uq2O20SkXlJtoOah3A2ocEARooTKmfbccQblwaZUVpV85Ox3QXcpXHh7A7moEjGtHwrC/nRYe2rrGrvO5uFIzadPvubQAeITHQx+CT/TH+67NGkUOE7zuZuHhXtTqx4/iUvGfGC414ajQZygTjEl/O4ZjJ6rxp7ZjcVf7MfhvZCZUXYC+ewPUxq3m/uQlPM1vEJ72yun4Km64YbSGregNkNMnE3gmhKe22aoATsZsUCmb9M5OeDgLDzXRZnJsjdJk0AnPiDtLKKyA4Ms8AR8PmlTQZUr4PDxuiQd1XYdQYpR64C08dMkTWRIwYnAhyosDjJ6nQGk1+lFQDlH2Uf0S7ILqHLm0CoMyhg4qzLiNZOgtl5abO4Ukxhxo1od0rrWQxMLjdylpkgpOFp7CoHF/FgR9jvsMdHiEx0OfIa6ksPBQdXd4kCWjqorg94mYMLw00Qa7nVqzHrKg4djQpygVw1AFGa8Vn4bOuRdCFAVsVwehUw+hWSvCmtgYAECMsvIoqgYfFIyXjYmwu2IymjWjuCVfLDJfMDMHmxae9F1pdRThITjvu1NwykFjcc4xk432qfHVdCsBXCxuheOTYXUrNOoEUgLDsPAkvixFkXJn5Sa0nyYVNHdLFZZOZ0EmLi2dd2mZtbSM4/zu3Hm49eL9EApYwuZCpQ0AIBYP4iYu6wVLLDx0Xp9MQVt4Jo4oy6tLK498x2Z5IyBV4AdeHp40XFo02eReYjxxTxdOYelHzB2BHx05EYfNHp5Vm3s6Btad5aFfYcnSrXjsjS+h6zo6w3E0tbEaErcILYJkSf7IxDF0UCHu/uXBOO1wwyVFvywKhQiQcJ3MSFhoWkMjcfqPjsUBc8bYJoy3I0YdJ2XLSvNLX1F1jJPr4RdUCIXliBdWoVVLJKrr7B3Cw4uWM9Hw1DoQntKiAE44YCyGVRrnQQ8v7T6MUl/c2dQQI/2kheOyJFiEJwfuLIDV8NDWqnQTDxouLWLhYQXtfG0pnyzC75OYr+5QvNXYpngQQxQEUbCFpZP+ZUNWaAI3aWRZxvunglMBy3wglTtl4Fl4rL+HVhYAACpLWMsmW0iU3b8oS2sMU0sr4dIqKfDjO3NHoijkWXg8eMgpXv1oK95buwtN7RHc9uQnuOb+ZeiOWELFlBaepITH+F8A+wVEtzFUbrft11I41tqWu/t3qRVQqqYBuo7oqheNPqoapvh2AQDkETMgSSJaNeOlpXHVsfMFqximNUGnCyfCQ+rqkLGiI9NIRBpgJSqkCYvbsddtasJ1D67Af1fuoPpNa3gI4RFzKlimz0PXWWtVqjw8lvaHJjz1TGZrN5Gxj9LzBE0Lz2AbabB+EwsPe+xMQPd/4sjSJFtmh97S8KSy4AxkC8/oqmLc/tMFuPmifZltnMjmRcdPxeCyIC46YVpWx3XKw/Ntx8C6szz0G2i6bhKaWFxDXUs3YoqG5o6ouU0qCw8pqOeo4YGz+JN+cVTLbbb92mjC4/AV2z3leKNvG5dBrdsIJRo1CY80YgZEUTAJj95LLi1yruSLLZOodCfCQ8zbRDNFe8joa2JqKiTRIkcOx357dQ3ueu5T1DR04n+f7rK1JYkio7VSG7YAAMSKkfbGsgCt4aHJcbq1tHQdEIoGGQOixqB3t5rbuImM6bZDMWN7sXiQLeeTPSli9i6txraw+ffoquLMG0iBXgtLpwiNT7bnIR5oomX6tSZLIgaVhWwRf05jv/+Mobjtp/tjxOCirI4b9EsQBGOMPcJjwCM8HvICOr9OXNGsyB8q2y9fu4r/qCRJ6pyitGgLD9MGdUdXSwbhicIw34Y1HzoLh1Hb2l+skeLhkCcsAAB0v3Izil/5FYbJrdB0QBo+DZIoWC6t7jbommJrI9cg52paeNJkPJquo645bFvOJyGjrTaKg0tLpi08Dsd+55Maq6+J/zfWtGHzLsPCNqg0aO5foLYahEKUIA0Zi1xAoPLw0NYZ3nVi0/DAOn9BkiEUVRq/KR2PaZHhjkm7tAIJwmNzaQkC424DelY8dPKocgCGOyQfVpDeDksHgKmjy23EdCCHpfO6MgI26WNujhsKyPjx8dPw0xOn59VitydhYN1ZHvoNaDITjlqkgM72y1cndyr8CTjn4eG1FU5tVIkG4VktzcFOpRzvRqZBlCyxqdNLQFE0BOZ/H0KIdRnUBCdCDBZDFAV06kGoEAHo0LvtVqRcw8y0nGHiwea2iM2KJomCSQQsq43VHn3diEtLEq1cRk7HZoq/CoZl6IElG6ADOGBGNUZXF5tko7TbcHmJg0ZDkHtWUoLAJBXQGQsUreER4JyUEqBITcKtRQuXTc2Ni5BUgG4SHrGIEy2LdgtPT1xa00aX46oz5+D68+dlvG86YKul946GZ+8Jg2wWnYFm4aGH0k2/lC/r2n7TqzF70uCctbenwyst4SEvoPU5DOGhLDx8WLqraNlBpGu5tNjldBtDEgVCa6QReKbdiEg6LYUwU1F1iEWVKDzzL7jt8eXYtrsNZxw5AfvPGWf0SRCgQ0AHClGGDkO4nLAM5AvEauFkkUmG2hbDnVVeHEBLwpUY8EnmC9XUvlAsgSahlkvL0uDwhEfXdUaXpWo6vqlpQ31LGMUFPvzoyIlM34s7txu/qyamdQ7pwE3DQ4c/OxEMPkeOWDIY6k7WwuPm0iI5eYqFMERdAQQBQlE5hKglzKctPOQ4ZnsZn6XRXj7EygS9VVoiErXeAXuPr8RL/9sMwFo28ETL1vm4RagxouUBdv79CZ6Fx0NeQBOebhfCY3dpOX/pJcvDYyM8iQUBxFAmdgEAWqUKq80URfpInwRRREdMQBR+DBlSaUvJ3o7eC00nk6RZliFNwkPcWWOqi003Fq0dEEwCZe1Dk1ASJiyKIrUte+xYXGP20TTdJEpDykJmvg9yLQs7thm/qyeldQ7pgCYutEaGthQ4TaKWaJsQnkTyQQeXli0ZXMLCUyEZ95hQWAFBlG2h3fRvXXe3GPUHMAUs8zgzfLHNEvtXlASZayOJ9vpjezoYl5Yb4emlCLlvOzzC4yEvcCU8SVxafP4Jkm49WbV0gftWJi+XKsnQjwihUsTFkLleTPFSp/vUnXDV0DlXyP7tekLH0wuEx9TwJNHROCESM/pfGPKZWXmDFOEhQ0GPL+8CA9goK/5adFHWHbKebEOPdUFQRkiIIdBtZDKWqiakdQ7pQKCIC62RkVJ8NVt1rhLtOCQftIqHOru0KkQjkaVYPCjRF7p9dh+NSorYHyd1lqzlr39HzTPE6gfONPIwMcR0gLmzAE607KrhYYmyh/zAc2l5yAvcXVpGtW9F0R1KS7BtSEkyC7tFuxASM0I2iIhYMQJiG70+lUvLTtQKgtZjIpmEpwAQAK0XcvGYLi0zLD29/cg18EkiyooCqGsJM9EaqUTL9HZOeh+A1e+Q9RrnggOA0w+fiN2fNUP4RodQUgWxIHdh1bQWhw6Fp6+vk4VH4M5JdCA8JILN7tIybrTKBOERigzCYw9Lt/bRdSCesH5lmz03n6A/APJJeI7ffwwmjSzD5FFlAMBZePrfuPQU9Fh6Lq2+hUd4POQF7qJlBXc+tRY7G7twwgFjmH14T42V7M49SosXQ5AJZ3SiQKg0eCyEdueJj7co0f2OK6pJGAoCVpIu0n6baeHJfy4eQnAyFS2Tc5ElEWXFhkCYtfDYNTy8mxEwiBbv/iHo5i08qs4UDSUYPqgQg0o6EAMgDRmXVv/ThaNLS0zDwmMTLSfEndEu6NEuCIFC18zIJA/PkEQkoFhaZR6X7hdLHHRECOHph2HCdF+lPBIenyxi+ljazWyRgIFY4ylz0XK+e/TtxcCj0x76BZJZeDbtakNnOI7aJjZHTExRmd9JEw/C7mrQY2FUrX8Mc/xbMFpKEJ4h411fJk4vdaJH6U4IKwUAwYDdKtKikuSDvajhydClpSSSCMqygPIig/DQFh6nZIKKYm+bzrTMH7szTNx+RrusS4t9vagNm432ck14zPByixyKgsBcd0cLT+J/kyT5ghBCJcayRMZlcrZuLq0hCdcpKZNBnzEvWtZ0S8MWdKm83pdg3Cq9ODOk0lrt6aDvw/TC0gfeGPQX9Anh6erqwo033ogDDzwQc+fOxY9//GNs2rSJ2Wbp0qU49dRTsffee+OYY47BkiVLmPXRaBQ33ngjFixYgNmzZ2PhwoVobu6dRHAeUiPuoIUBjOrdhFTw+g9Su4kgnUzLNJStn6Cwfh2+X7AcVVIrAEAcMtbVteEWlm702ehbMMAKUck+LST5YG+4tPiw9DRFy8Q95ZNEVCYKhhYVsMUtAffEgwSSKLoSHjJOxQV+c72TS0vXdWiJhIPS4LFp9T9dmFoc0Bqe1Nead2kBtHDZ0BrRtbRoGC4pHUMS/lJCePgikPRvXdcRixnj2x8tPPQQ9abGyHNpeaLl3kKf3F033XQTli9fjkWLFuGZZ56BJEm46KKLEI0aobObNm3CT37yExx00EF48cUX8YMf/ABXXnklli5darZxww034MMPP8Tdd9+NRx99FJs3b8all17aF6fjwQFuFp7WTivTMq//iMWdLTyOFg0H8ScpWVAoxiAKRiSVWFDGfj0lqVkDWBO+qd8JsF5f8jVqZlsOt0HPoJhnNiCnn2lYuqnhkUXsP6Mapxw8DsfvP8ZcT1t4iOsmtYaHXUeuYXGCSKm6biaKZAhPZxP0cDsgSBArR6XV/3RB57qho6AEqrSDUzI7sogeTl647Cpa9okoFiIoEOPQIZj6HzbxIGwankjcGK+Ar/9N7KmejXzh2+TScisE6rm0egd98tS99dZb+NGPfoQ5c+Zg/Pjx+NWvfoVdu3Zh48aNAIBHH30UkydPxmWXXYbx48fjwgsvxDHHHIMHHngAAFBXV4eXX34Z1157LebOnYuZM2fiz3/+M1auXIk1a9b0xSl54BBXLfLSTeXdaKVKS/D6jxifiDBJHh4N1pe8uaytltlmt1DFtAOwL9dkGp5wxC5YBiw3WLsaACAAug49Yq/ZlUtYomWj7+lWLFcoDU8oIOOE/cegqrzAXM9PxkASDQ8pQ2ETLRvXsISy8DhFaRF3llg5AoLsT6v/6YLR8HDH5v9n9xPM/QhItJXe0QTAIni2CtaSaOp3lFCFeU58AjnewhNNWDGD/v4nn+ytxIM8ZCZKq/8RwZ6CJsuuYemeaLlX0Cd3V2VlJV577TU0NTUhFovh+eefR1lZGUaNMr78Vq1ahQULFjD77Lfffli9ejV0Xcfq1avNZQRjx45FVVUVVq5c2Xsn4sEVbmHpjIUnnLwsAyEnycLSaWitLOGpE42vbreU+cmitNwsPORlpGiCqfegay/lA5Zo2d3F5wRatOwE2gKjmRYeJw2P6Ope7OYsPHRoON2+Wk/cWbnV7wC0a8pejZyu4WXfD2afzWVFhphW625JtGkn1oDhkiKpD+KFQ2x9cdpH0y0rpr8/WngcXLe9AfrayANwsmdFy+mEpQ+8Megv6JPPjFtuuQVXXnkl9t9/f0iShIKCAjz00EMoLjYK4tXW1qK6mq2kPGTIEITDYbS0tKCurg7l5eUIBAK2bWpr2UkvE7gJynoC8sUyEL9ckoGeGCMU4Wnvtqw6XVFnwkPGSpYTX+BwuDaJd4IsiZBlEbqmmbqL3UoZhkht2CGPhiyLEKkvSJ9PMttyyvmhasaxSFmFwpCPObbfb4lzxaJyqOE2CJG2vNw7BMQCQSKDyHikurfINQj6Jcf++TgBsyyLju4ynyyaAmRBYK8FIYalCVG0qukmF5Vl49pEPn0D8Q1vG21Vj8/5WMlmmQdrspAS52MSHon8TxeulGxt6CWViAJAVwtkWbSyUifaIwj6JdPCoxVVmetol4Usi4xWR5IEU7RcEPTl9Z7JBvTzIApCr72zZG7M+tu4pEKq55AmOW7PIr3MtweOQbro6/kw54SnpqYGRxxxhOv6pUuX4quvvsLIkSNx8803o6CgAPfffz8uueQSPPPMMxg6dCgikQj8ftbsTX7HYjGEw2HbegAIBAKmDihTiKKA8vLCrPZNByUlodQbDSD4KJN92IXYhDmXFgEZq4KQcY0DAdl2bfyJ9kMFfpSXFyLeWodWNQ5dlPGX9mNQJEYxpLQa5eWFCFJh5aUlIbMtejmBJIsoLy+Enpjgy0qCzLFFn3FcXQf8JRUIN2xFCGGU5PHeIQSiJBFaLooi0yf3e8uYwEpLQ473dihmXZeS0gKEArI5rjSKi4NmhuZQQYBpi7ghqwdZFZ2lxBiFgj6EumrQ8tE/AQDBUdMxZN7hEH25qaFFUFhotCf7JBQVGeJsOXEdyYuVkDt6rIoT20qyZJ5TLD4MnQD07haUlxciFDLukWDQZ26jRrowou49jPIbgRa+QcPNdZLfuqdKSkKoqLDGqrS0wCShgyoK8/q+yQYB6toLQu+9s+jnMOC3P+t7CtzGi9yTAFBeXuB4foUF1nxWVua8zUBCX82HOSc8VVVVeO2111zXb9myBTfddBPeeecdDBtmVK6+66678N3vfhcPPfQQrrnmGgQCAcRiMWY/8jsUCiEYDNrWA0bkViiU3UBqmo729u7UG2YISRJRUhJCe3vYser3QEVbu1VTiNfmELh5ZshYKYmv4c6uKFpauphtIgmyFI3E0dLShfh2QyOiFQ5GtNGPqObHIE1DS0uX2Q4AdFNtqVQYvF8WEVM0dHXF0NLSheZEHSpZBHNsOrJM8RkWyc6GOqhc/3IJ4nKJRo1jx+IqWlq6Ut5bhFDGEmPEg3Y7Njd3oSAoo70jYtsuGolBS7Tf0RFh2iKaLJkylpE2VEVFy/+3d97xUdT5/3/NzPb0BFIg1NCkBiRApKioyInd0/MO0LOfjVP0BBV7ORXE3tGze5ZTf3p4nl+9Ew8PpB5YqKGXFEhPNltmPr8/Zmd2ZrZkN9lG8n4+HjzI7vTPzs7nve/36/1+794qLy8eAdus29DQ7AUQ2w7zzlb5WeByeVHfILfTYEz+3NQIic9zpR2rllb53N1ur3pNkig/PyRnE2pr6tDcIq/jcYvqOi0rXkfG9n+pggCnJc+/THN/tDS3ob7e/0ypq2tBi1Ne7nEF/0ySiVfzfeA4LmHPLJ3on7GUG5f2aPd72OrS/O0Oen3KdxuQv2PWrungict8mJlpj9hjFHODx2w2o6SkJOTyV199FXl5eaqxo2wzfPhw7N27FwBQVFSE6upq3XbV1dVwOBzIyMhAYWEh6uvr4Xa7dZ6e6upqFBQUdPjcja0OYokoSnHdf6qhbSERLcpYKaFsjzdw7BQjQJIYvF4JnqOH5G3TNXoKBH6mjLGgn4PVIsDtleD2iPB6JTT7JiabWdCtrxUMizZZwyM218X1s1WFuEq9GUk/HqHuLcXQ5Ljg97akmWjcHhEWE6+KarVoQ0Ue3/goKBO4VtytfPYcOHjr5DAjl1PsE59HWCY6CpSQnzwO8rF53zWromWlTYl2rAz3EAAwwQaYLIDXDU9jrUYwL6/DvG64d/izRb2Mhye9UN1e0migGIN6HzMm38faDvSp9jzQBngFjkvYM4szaIdSbVwiJdR4aaPEPIJ/F7Vjz6Tgz6iuRLLmw4TbkYWFhairq9MZNJIkYefOnejfvz8AYPz48VizZo1uu9WrV2PcuHHgeR7HH388JElSxcuA7DmqqqpCWVlZQq6DCE+wbJ9oiaQOjyo89WVoSRl+g1dQM3QC9ym/7/9bKcjnMYqWbfqwl65OiC1bPmZLfSSX02GUaxWiFC1r6/AEQyuUVIyqYGnpvLbwYDtZWoB/DHmeUz8XpRJxPFCztICADDFeo8EJ3C4w1Z7jOHBpOfL7zbUBaenePesBtxOiPReLG2bh6cbTwdvSNdsHnpe2m7s7lQsP6qqQJ+64WqFyVyw8qBMtUy+tpJJwg+fkk09Gnz59MG/ePGzatAkVFRW46667cPjwYVxyySUAgLlz52Lz5s1YsmQJKioq8Nprr+HLL7/ElVdeCUAOm82aNQuLFi3CDz/8gM2bN2P+/PmYMGECSktLE31JRBA8MbDe1SytIJlDijRWnbR8NXiYxuBRhLahsk+07ysTkFppuS1EHR5tZpNN7gfFnA2RX1QHUCZdf2uJyLZTDZ4QD1ltJWA1SyvI52bieb+HRGMdSIwFZGkB/n5RgsCB+erZKHVq4oG2z1foLK3AMTBeu7q/NDlTi7XUagoP+jxc2/4DAHD3nYgDYh72ij1DZtgof2vT5hXRsjUlCw8mJ1NIl6XVBZM79GnpwcfVWLCSiA8Jv7scDgfefPNN9O7dG9dffz0uvvhiHD58GO+99x6Ki4sBAIMHD8bzzz+PFStW4Nxzz8WHH36IxYsX61LVH3jgAZSXl+OGG27AFVdcgYEDB+Lpp59O9OUQIeiswePZ+h3GHP4Yw80HVP2IlgAPj1IoLsM/sSrP0VBGjvZvRZSrnHerL6ZuNxg82mcR84W04tlPizF/1pMyMbAgmVTBUK4l3CRi7JHlDeI90nZL1x7a6fKq55ZmM6v7UkJpAsfUFg3x9PBoPSiKMWj07ATzHPAhxlP18LTU+/tscYC3cjvEg78A4CANKA/Yj/a4gPZe8RlLoqQa1KlYaVl7bydy0u3qrSV0hQcjKBFB9k78SEpaekFBAR5//PGw60ybNg3Tpk0LudzhcODBBx/Egw8+GOvTI2JANAZPToYVdU0uZKfLYRHGGNpWv4dCtxPXZPyMHbXNAIbqtlENHnBgkhdM6WmV0QOAb5I1hDW07xn/tpkVD4/SWiJ44UGOkyd/UWIQrVkwwV9tmYtDWXztXKx4KSKutNxOHR5AHgNR0w4imIfH4mnAWQ3vwGYdCFHyfw5KlWWLmfelrnOQROZvuuptAiQvwJvApeVFdM4dIVjzUOWj8Ic1Q9fhCWhaq/XwqM1DGVzfvwMAMA+bBlNWTwA7dcfQ7hPQGF2+97S6tpQMaSXJy9DVKy1rf0NEUmmZCg/Gj9Qr90l0CaLR8Fx4cgna3CLGDpa7VbPmWsDtVJcXOncEbOMPafnWZwwQLOB9YSbA/+DQNQwN8WCxhNLwWAO/IqqRYEmHokhlbY3gHNkRX3OkaI0bZTKIvHmoz+AJU9MjwMMT5HPLPLwW2WIVTrW1Yrc0S31fqZSd5tM5CTwHryh3mgeAdFH2fPGZPeNiDCpoW0soovLICg8G1yVxadny/lrqIPkyivu2bIF0dC9gscNSdgEsUmDX+VB/K8fRCpZTMXSj6/SeyOahXbyXljYbKXThQf/fHMjgiRdd7+4iUoJoFPgOqwknlfZWG1yKtQcAAC6TnPad4a0DcxlSOVUPjz+cxWf00HXoFtrz8GieK6qGxxu+tYR2HyI4cEpYK07VlpnO4AlTeToISvgk1K9KwO8JUXYZzFC1124HAPQUmiC0+fVKSqXsNN8YFQr1sHFu1cOT5pG9blwc9TuA3sOjjA1n8OwEDWlpDCXd/nweHqmlTh3/fKdcc8cy4lTw9kzdmIbqg6T8rfyv1KNKxXAWkEQPj8ar0xU9PNrva0jRsu6HWNxPqdtCQ0vEBY838rR0468exeBpSu+PGlE2ekRfp20Ff6iBg9R0RP47o0dQ13BkGh550vaKEjxeSdWhBDN4lH5aosT83oA4GTxa20aZDKLV8ITSDQBBPDwGQ9UKNywNe9XXGc3+z2HXYbm1QnaGFZ49GzDf8QnmpK1UjSaHz+Dhs/RV02ONtrGp2qnd4OEJFibgNYaS7n1NSEtZlOM6KO+vaBgAvRGpD2n5f59zITw8qRjOAgxehgTaHYLuR0rXm5K0Bk8oQ5K6pSeGrnd3ESlBNBqeAIPnqGzwOB0F2OeVtR9i9S7dOuocxQGsyafZyegZtFeW9gESKqSlTEKtLq+u95c9SOVhQRNaUsJYsUxNlxhDdb1T1wxTPndFwxPBPjS6nHC/mo3p5saQ1iBzFTjmfy+jabe6/+/+J9c+Kh+WB9fqvwIAhpoPw+PrCG53+QyeBHp4tN3Stf+HDWlJIUJazgYwyYt0zok0bz0ADkL+AABKQ1YBHAI9Nmo7CkU07/u/zZ3aHh4uxA+DeKMLaXVxD08ojE1nifhAGh4iLkSj4TGGXNSQlqMA+7z1ON66B5LBw6M8QniO82cCZfbUP7QVDY9m96G8Pf0KMmASeNQ2urBtn6w9sVuF4J4BTX0g3pENEbH18Hy36RDe/HIbzp7cHzPK+qrvC1FoeDwR6AaAYBoe/b6HmuV0/xYhE2liI7Ja9gAAftpdi6ONbUizmTBG+hmiL6xo4UTkeI+glZeQ3eQT9fbo1+75dga9hke5Lvn/8B6eECEteyYgmADRC6unEf1MsgeRz+kFzuLvNv+Hc0aipc2DdLu+VhPHAWCBHh6nS0lJT83fmUlrHip0cYMngmehfuzjeTbdGxpaIi501MPDJBFinew5cKUVYq/YA4Ds4dGGHrR/Sz4PD5fRM6hrOBIPT7rDjNLB8rE++34PAL8Y14ig8YooHp5YGjxfrdmvnofWcBEM3phwaD014TU8+knf6OFRDJ6tmZMhMcDhOgqptR7/2SR/RtOOy4K46XMAgJv5tDysEmfZN4BnIoTeI8Dnh668HguCeXgi0fCEqsPDcbzqlXK4a9HfZ/AIhusYNTAPk4YHhuuM9XeU/xUPjy1FPTzJqsOj/f6buuBsTx6e1KHr3V1EShCdweP/gnvqqgDRAwgWiPZcHPDmQgInhxeU1HNAdfHwHMB8Gh4+o0dQ4ybUg1z7bDXxHE4YIU9eh47IAulJI4JrT7RF+OIR0upb4K/cu2F7jf8c1cKDEXh4lLYSCF3bxL11BX4l/Bc8JPWhrDWw0jknCn0dwSvThmC/KIcXPb/8CwdqmgEAk7mNgKsFfE4x1mAUAOB47heMs+4FAwfrpN/E/QGurZgsGbK0OlKHBwD4TLlukN1di/4mnwexIDLDTbmvVA+j731Fw2NJVQ2P5vuQWA1PF/fwRGnwkIYnfpDBQ0REpEJZhaDZPkFSvAG9B8JTsw8AwOf2hslkggcmHOV9Xp5Kf3q6qtWQ3GBtTfLfmT11D+1209J1xhGPkQNz1fBEcc80nHVC/6Dnq7a8EBl4jd4jVmiNxX+tPxBw3Ig8PJqU9GAGB2MSXP99FxP5n1Fq2avR8Pj3PdAkh6mkzF4QTWn42jkSAODe/E9keapxku0XpO9bCQCwTroIB7kiAEARL4cE63oeDyHPH5KLF8pHKjEGUREtK4ZO2NYSynaB+1Qyy9LcteirengGRnQ+uRk2mAQeGb57yShaTsUqy0Dyqv129UrL/Qoy2l3Hr/ciYyeekIaHaJfNFUew7O9bcPms41A6qEdE2wTz8KTbTXC6vLBbBVXPAOiziNxH5Amez+6len4O8L3RU6qBeOgXmAdN0u3T6vZVObamgbM4wGs6owfroxTql5QgyLVRzp06AP/eeBBXnTWi3SJhWtFyLKstt2kK1B30eZt4jlPDNJG0lmiv6CBrrgW8chfnE6zbDVlaDAIklJh91at7DgLHgM2evmiwFiHLdRjXWz4BLAAkQOhbCqF4FKr4OsB3bi5mQnW/GYivekfGr8VhAXV4woa0ENqAVCpDFzp3wMZ54eUt4LN7R3Q+t/1uLJwur9qH7VgMaSVWwxM8462rMLx/Dq49dyR690gLuU6wH2dE7Ol65jQRc7burUez04OteyOf1IMbPPIEkJ1u1b2vrU3hUQyenCL1QbgP8kTjPbRVXU+Zoxxt/ho8gP5BHbQOjza8FcT4mT6uGA9cMRF98v1hJSOqh0er4fFVW44FysSoheP0noz2UGvwhAgRSD6dFAAMNldBaJKbfHolCVem/xv3Z3+EMea98goFQ3zXzGFrj1MAjofIODmDbuJc2E+9DhzHwSPYUSnKhR+/cY4As2cZDxsXtKJlxVvDGQwdPog2xKhf0i1TPDzeegBAk613xMUTs9OtKMrzT25GDw+FtPR0dQ8Px3EoG5aPXuEMnjCeSCJ2dL27i4g5ygQbaYVfIJTBY/H9bw75kHMfleudaD08+1ghwMmNKKXmowD8Iba8mo3yPnqPkLfTGTTy/8GMIHld/7lFox1Q08MlJmf0aKotxwKth0dBbmkRvYYnlJdKqj+ke51W8RUYY8gU6zDKcgDpvAs5QisAgC8YrI7rEVs/8BcuwYK63+LxxlmwjTgZnEn+XAWew/stk/BF6xh80zYyYb/WdaJl1cMjvxeJaDm4hkefSt/oiMy7Eww1Ld1X7iBVQ1rJay0R/PvZnSAPT2Igg4doF2USESPU8UgSCyrUS7fLEVS71aSrRaKEtBiT4PEZPEJ2LzVjo5VZwPcYIJ+Dz8vDAOTyzchokHU95uNOAhBcrxPKVS8EqcocCdq0dI4XYl5tOZjBw/MaD08EBo+3nZCWYvDsQy9IDEg7vAGuHz7ASK5Ct16NmAEhI1cXxnPxDnhggkngAioO7/IW4J9tY+BF8JT+eKD38Ojr8ITvpeW/poBl6XkA579Hm9L6dPr81MKDKWrwJE3D08XT0iMhWN8/IvaQwUO0S7QeHq1gWfv9zc2UW0dkpVlg8dUi4TnOH1porgXzuABeAJfZU52svaIEUy+5wq330BZ5XQaUW3eAA5NTn32/yDmOU49pbN4IhPPwRP5VEHj9RBnrasvBQ1p+DQ9D+yJyVcMTwsOjpP7/KIzEB62yLsqz+R840bwZALDaMwSNkg3/dQ2GwHP+zDTG4FQrBuslgILhYZ2oqrnaUJ9q8ESSpaV6eAL3yfECuEy/Xq05rbjD5+fX8PhCWilq8OgrLSdu4tWmonfFtPRIMPZdI+JD97y7iKhQ7JyIDR5NOEtbqXjK6CJcdPIgnHVCf9WtbzL5v+DKJMxnFYLjBfXXnihKakl/sUouZscYwyizXK/GPOxE3fGNk52uKaL2oa4Nq3XAw6MaPDFMTWeMqROjNl2f5zjdr79QOp5mpweffb8blUdbffsI/IozxiDVy/V16vgcrHINwZFBZwOQCwe6mYCv2STcVX8h/tU2EgLPqwYDk5gamjG2SDB6URIX0tJ4eCS9h0eppZQWrCeash2Cj6WSmn5ETIfXHFrT1e75QfHw+EJaKavh0XpCE3fcrp6WHgnGqtxEfKAsLaJdlEkkWoNH4DlYLYLaqiHDbsbMiXKassUkP/S1GVpinTwJCzlF6vaALMBVUoJZQyVYWzMYgCzepzHJ1f/6VrqZG0NaPMeFdNtHMzmr5+UTKfurLTdEvI9QuD2S6nHIcFhQ1+Tynav+fEPpo//7UyU+/c9uVSAeTMPDnA2AqwUAhwY+B0Ar6ounoiiLh2f9p1jvGgDe5gDQql6vYhyKEtP0hDJ4eAzGVeJCWvL/eg2P/OavJvVDfo4dU8f0CrldqLHkM/MhAtjr7dGpUIPRw5OqIa1kFR7UhbS6qYdHuW4SLceX7nl3EVGheBMi1fAojUNNJl5n0GgnX6uFV9dRUDw8Qo48OWm7g3O2dHC+JpRi9S7wkgcO3g0A4B36bCBjhWWjgFVdT/fLMvKvQigPD2vtfGq60+cF4ABdywJZtNy+h6epVR6TZqcHQPDGoYp3h8voAUkwq/vjRp2F++vPw4etE2H3eSE4yNerrQGkpldb9RO30WhMiodHKUjpO3ZWmgXTxxUHrQGlTWcPhmXEqThgLcH/tY3qlAGgbKs0pE3VkFby6vBoQlrd1MOjJFhQSCu+kMFDtAvroIfHLPA6I0cbXgnm4VGEtLxq8CgeHp/HyFfaX6yugE2SvQ8SJwCa/kZAkEq3IXQcobQ97SFovB1AbENaanE6iwCLVhDM6cMMoT4Lp0uv/wkW0pKO7JH3mdNL12ncI0o4KmVAhKAaCMqvb3/PLX9PqICQVoCGJ7EeHilIllb47fzXHgw+uxArss/DYTGnU9kzxm0VYz/V0Gt4EndcCmkF1xsSsSc1v3lEStFR0bLZxKseHJOgDyepGh5tSKu+CgAgZBfqlik1ZZSwllhdAbskF+QTLekBv4qMGh7/a/15hipI2B5G0XIsqy2r3hOLoDMWOUM4LpSHR1vQEQj+i9m7e4O8rPcI9QErSpJqWHIcVGPL6GrXeniMneQDDMoEPb15nYdHH9IKR7i0dAXVY9QJC8C4rc2cmkqCYI13E4HWyOmuouWCHAfS7WYM7JWY2lXdldT85hEphTIfRNITBtDXgFE8OEYtiZKlpRg1TJJUg4FPz4UEf5hJFCUwxiAUKB6eXXDwcg1frzmwbDsXENLS/6/QUQ1PyJBWDKott7n8+hizye9B4XlONwmFMj6NGV7GcZda6iBWyan8pgHjIfy8Vz53SdOOQuDVz8Wvg/If16/hSRXRsu/ctB6eCI6tNeJCwaLwGLV3fgqWY6BbekI1PNryEN3Uw+OwmfD49ZO7bUgvUaTmN49IKaL28GgNHt+Ea9SSKDoGsy9Li7U1+SwrDpxNNmKULz/znQOfWwwIFsDdikKvHP4SLYEGT2ANFuheG9cLtiwcxpCW3IaAA2uth+TsXPFBxZiwW40eHoNoOcRHYQxpGcfdu3udfM4Fg8Cn5/rbVTAGr2+nJsHvmVO9ZBojz6l6ocJ7eBJl8Gi1OMY6POHQan9CoXZfj4GGRyFVNTzaIUtkaEU7yXfFSsuRYg7R946IHd337iIiRs3SirDQslbDozzAjPVgjCEtxbsjpGWC4+Vl2gnTKzJwvAl8npyRVSzKKeneYAaPQaSs1rgI0PD4DaNoHjQBHh6LQ9UdKWnzHaVNY0xoNTxKarNfSxMipGUoWmgcd++utQAA88CygP151c+NU9P0VQ2PLqTlN8q0BBg8CZq89Gnp8nvRhLSA0F4e5e3OhHiMW6bqr/hgrVYSAVVaJhIFGTxEu/jr8ETWKyoyD48hpOXzjAg+PYx2GSCHtQCAz5JT1nswuYO1aAmsjxKo3QkhWg5TlC4cSoE9bYhPCbdJnTZ4/OEik66KsfJ/eIOnLYxomXndqkFm6jdO3h/nN2S0xQpV7ZVhLEWJ6cJuWsJlwcUTXVq66uFpfzvtpB5Kx6N6eDp1fvqtg2XOpQKh+szFG13z0BQdG6JrQHcX0S5qa4kOiJZVg8fo4VGytHzvKzVshPRsdR2jhweQM2cA/wQkWjIDjq/13ACBmh7/er7jRPmLW9taQj3X/EHye9UVQbeJFK3BYzGIluVjy69DipYNHh5dFtyRvQCTwNkzwRmarUpM345CEY8K6v+BouV209IT5CXQZltFI1rWnm4oWz4mHh7DpqGqXycbfS+txB1X11ePPDxEHEnNbx6RUnRYwyPwqvveGJtXNTy+9yXF4NF4eLS1Z/x6mULdfoJqeNQQlvJa/j+0hye6r4ExSwuQNTEAIFbvBpMCW0NEijakpetTZTDaQhk8AR4ebZ2jmt3yPnoO1BhQwUJavFoB2+gti0q0nKDQjb9FBNOIjCPX8ADhQlqKhqfj52f08KSqTkVXhTxZIa0UDfcRXYPU/OYRKYVq8ESr4THxaqaR0cMzoCgDPMdhQC9f401noMEDQNdPCwD47CLdctEaGNLiDIaMMbSlYPQERYqyX62Hh88uBKxpgOiGdHR/VPvTojUmjKJl7TkHMz69oqQWt1MwayYQxeAR8gf4z9u3WJIYPEr6v8D5s7SMGh6J6YwyLckLafk1PGIHsrTkbdsJaXUqLV3/OmVDWsnS8GhDWt00LZ1IDHR3Ee3Cog1pRZCWPrRvDp67eRrOmCSnlwcLaQGBxQflJqGaX+bWYCEt/f+KARRqQo52Yg6WzsxxvKYw4q6Q2zLGUNfkCjnB6uvw6NPSmceFE8xb4OBcQY3PYF3W9R4e+byEnhqDR3MtwcTmxm7jksTUWj92g4fHWEMl0WnpDNFlaUUV0opRlpZgKC+QSujT0hN3XF1Iizw8RByhOjxEu0TbPFTp/+SwmtXQSDA3vraJouLhMaXlwKNZx1+Lx/dL22QBl9EDrKlGfj9MWrpRlGzUlHTcwxOo4QEAPqc3xP2bITVUhtz2q7X78f6/dmLujCE4eVxgB25tnyrt/jlwcG/4fzjLsgqZUh0kaWrgtq7AUJoqCne1gDX4Cjv2HOg/Z98YvP+vnbptlInH6CWTGCL28CS6tYQUdZZWokJa/r9TNZwF+D8/DkkMaZGHh4gjdHcR7eJPS4/M4Nm2Ty7AN6g4K6SHx4g/pKWvNKp6eDQ/wRXhspsJgMkWsC+jITOgKBN9C9IxaUShYT35/2gzQ0JlSsn1eACpsTrktoph8dZX24Mu14mWzVoND+DdtwkAMNBUHdT4NAqWAX/4RAlncRk9wdn8YcBghkGryxskpCUv0zUPtaZG4UFllBhjqoESyaEjCmlFoQkKhXbLVPZgKGeW6FowVouA4p5p6N0jLeCeIohYQh4eol1YFKLlZqcH+6ubAQDD+mbjSIMTQPu6BSlESEvbMV2BzyqEuP9HNEn2oOGB/Gw79lc3o0eWHYDchPPeyyYErNerRxpsFgElvQLDYuFQs7REg8GTmQ8AqiclGJkOMxpbPSGXK14am9UEaDw2GVwLpLqD8nkLdajzugO2NRYdBPyGpnhoKwB/ew7/tQSeQ3HPdL/BYzAetaLlVGktoW8e6vPIRFJ4UPN3qFtbMbNjVXgwVTO0AK1nNMHH5Tjcc5m+LhRBxAMyeIh2USaDSDQ82/bVgwEoynMgK92KYX1zkOkwY9TAvJDbMNEDuOTeWEJaDuDyL1M7potaD48sXG5itqChhqvPHo66Zjfys+1hzzUr3Yonb5zSrvfJiBrSYsENHqmpBkySwAWZOYrz0/HLHtkD1tLmQZrNrFuu9fBor3kA2+c/PsfANxwA0NOwbfCQFmMMnoof5Nf9xurP2XCOfzhnBIb2ycamiqPysQzNQ7VGVXtZWomavNTWEhLzi5YjDGlxnFKwMHxIK1atJVJVsAxoshuTYHRQKItIBHSXEe0STVr6Vl84a1i/HABASe8sPHHjFJSPLAy5jVJ0ELwA3p6mW6Y8CL2aY5v6j8MhFOD7tiFBH85mk9CusaNgMQtRP+D93g690pVLywV4EyCJYC21QbfV6l72VDYFLHeGEC33l/SZX6a6vYHbuoKIlgUe0pE9suZJsMDUr1S3XDuRZ6dbMOG4AmSlW5GbYQUA5KTL/ysek5Y2j287LsBQNOm0GNFVr+4M+krLTD1+JGjbUgRD+Yhj5eGJ1rhOJIrNQU4WoqtCHh6iXdrT8GzYXoN126pR29CG/TVyOOu4vjnq8vYmCyVDi7NnguMMk6ighI80Hh5HNt4WzsdBdwumRXktsSCUaJnjefCZPSHVH4bUWA3eV9xPi1dzHXsON2JE/1zdcq1oWTFgODD08Rk8FeiDEuyHqT6IwRPEw2M2cRrvzhhwZr3mSesJyXRY1L+HD8jFbb8di74Fst5HCW21tvk6pVsDDcWOdp/vLMqxZA1PdMfneQ6ixNoVLXfGAaE9lVQWLYcq0EkQXQUyeIh2USaDYCEtSWJ46bOf1ZRmQG4boXh4IkHx8PCOrIBl/jo8+mMr81My3O/h2jtwmflA/WFIDVVA7+EBy7WG2+7DgR6eYHV40rg22Hxxvv9xI1DC9sNcvy9wW5+BlJVmQUOLrPEx85zaMNRUMjHktQBARprf4OE5TvcZqkaF77UxnGXcVyJ7IvnT0rV1c6LbNpTzMhb3mU7Dk8IGj7G4JUF0NcjgIdpFcesHm+CbnR7V2PnDOSNgNvHolZeGdLs5YN2Q+/elmAc3eBTRsj58pKYLR3yU2BGstYS6LDMfIgCpMbhwWWu47T6s76wuSpI6ltrWEpm8LPzmbBk47C4EGGBurYHU1gzAHwJU9DV5WTbV4LF66sGajgCcAFPxqMDz1UxuGY7Qn5nRY2JMSQeS1wRSew1qz7UIJ201pBXC4onWgAqGLqSVwllaat2q1D1FgugUcf25cffdd2PhwoUB769atQrnn38+xowZg5kzZ2L58uW65S6XC/fddx/Ky8sxduxY3HLLLaitrY1qH0TsULO0grj9m1rliTXNZsKE4wowdnBPFOQ6otq/Z8cqAICp17CAZcY6PP5zkv9PxsM5VEgLAPhMOTWdhUhN125T1+TSGZGNLX59jN3qby2hGjyObLh4B6pEOavMe1if2q4aPJn+sJWtYY+8z579wJmtgeerMUy0Ia2A9QwDHSx9OFkhLe2pRVNpGWi/VYdftNwZD4//72MhSysZXlOCSARx+fZJkoSlS5fi/fffD1hWUVGBa665BlOnTsXHH3+MCy+8ELfddhtWrVqlrnPvvfdi5cqVeOaZZ/DGG29g165dmDdvXlT7IGJHONFyky/FOiPMZBkO8eh+SDW7AE6AZeiUgOX+tHSDh8f3f6qFtNRMrRAGj/E6tK/3V8shrqI8B0yCv/FqFtcKAODSssHxHHZ6ZAG499AW3b4UDU9elt/gsdT5qisXDglxLf6/w3l4jB6boB4eLjkeHk7j51ObzEbq4dE0Tw1GTEJamr9TOaSlXCPZO0RXJeYhrYqKCtx5553Yu3cvevXqFbD8jTfewNChQ3HzzTcDAEpKSvDLL79g2bJlKC8vR1VVFT799FO8+OKLGD9+PABg6dKlmDlzJjZu3IixY8e2uw8itoTrlt7kVAyeyENYWjxbvwUAmPqPDa/hkYwens6HGjqKMrEH9fBk+Q0exljARGnUInlFBotv6PZVyYLvPj6hcICHx54Ngeeww1uAydgO78Gtun0pGh4lwwoAzLXtGDwhRMtGjONstwYxeIQkGTw6D48U1fFV/U87Ia3OpaUfI1laIXrOEURXIeYGz+rVq1FSUoLnnnsON910U8DydevW4dRTT9W9N2nSJDz00ENgjGH9+vXqewoDBgxAQUEB1q5di7Fjx7a7j47+GouHu1lQK9am7oOuPZSpQJJYwBgpacqZaZaox48xBu+uNQAA24iTgo6VMkEwZji276RMAp/wMIHS6Z2xwHuGZfkys7xuCN4W8HZ9UUPR2LSJ8+/jgC/DrX9hJkwmXjUqFIPHlC4bPBUeOWwmHtkHsa1FHa82j2zwZKRZUNI7C831deAbDwMArL2Hgg8yTlqPQ3amNeRYWgwi5TSbKWBdbRq9wCfuc7FI/uMqRqjJpD9+qO+hGsbhuaDnq3h4TCahw9ejNQQtndhPvDGb9W1EjuVnViLpCs/4RJHssYq5wTN79uywyysrK1FYqK/Jkp+fD6fTibq6OlRVVSEnJwdWqzVgncrKyoj2kZurT/WNBJ7nkJOT1v6KHSQzM7K6MCmJpleRcYyU5KweOY6ox89TX416ZxPAm9BzxHhwJtnVoR0rh0/8bLGadftX6sJkZtnj+rkFIyNDDhnxQrB7Jg1N6TkQm+uQjhZYc/Td3Y1+hLR0G3J8NYMO1MjFF4eX9EBOThqsdtnjkuUzeNJ65MNiMaGROeBNy4eppRpt+35B5hC5Sq3HFx7rmZuGJX+chuata1D7KWDuUYy8XsHrIDk0mVnFhVkhx7LZrTfUcoKMe6ZGO2Q2Cwn7XFweTf0h372amWELenzj91Ax+NJDrK/cZ1mZHb/PLJrwX5rDkvD7NVLynHJIVOlxd0w/s5IAjVfkJGusojJ4Dhw4gFNOOSXk8lWrVrVrbLS1tcFi0bvOlddutxtOpzNgOQBYrVa4XK6I9tERJImhsbG1Q9uGQxB4ZGba0djo1KUkH0t4vfKEIkoS6upadMuqa+XXVoELWNYe7opfAABCXjHqm9wQBG/AWEm+/5ua23T7V5Y3N7VFfdzO0uaU7zGXyxv82Gm5QHMd6g8dgMWuN3jchn5XR442Q2AS2txeHD4i7ys3zYy6uhZV35PJy/dlG+dQx6M5cwCyW6rh3LMZYuFIiKKEJl9mlujxoqnRidY9PwEAuPzBIcfI1aap3eMVQ67X0tyme80zFrCus1X73QtcHi+0JRGU8W1tdeuOH+p7qIRGGxpaUZcWGJb1+vbd3Nzx+0z5/gCAJIYe42STZRNw1uT+KOkth5aP5WdWIukKz/hEEY+xysy0R+wxisrgKSgowBdffBFyeVZWoAbDiNVqDTBKlNd2ux02my2o0eJyuWC32yPaR0fxeuN3s4qiFNf9xxNFw8MY4PaIOt1HQ7OSpWWO+vo8VbK+hM/rp9tWO1bKsdwe/fhp+3slelyVMIc3xLG5tDwAFfA21IA3LDdqkVxuEV6vhD2Hm8AgVzt2WE3weiX1GjM52cPDbFkAZGFzY9ZQZB/+AS1bVoE//iKIkj9Ly2IS4PVK8BzaBgDgCwaHHCOXplihctxgGAXaVrMQuK5mFZ7jEva5aB+cipHIWPDPxvg9VO4vjyf499N/73fiPtOOC5+4cekI500dqIbcjuVnVjKg8YqcZI1VVAaP2WxGSUlJpw5YVFSE6mp9Bkt1dTUcDgcyMjJQWFiI+vp6uN1unRenuroaBQUFEe2DiC3auU6SGHiNJkFJS++IaFms2QMA4Hv0D7mOEKIOTwRdLuKGmpYuBj8JpcKy1Hw0YJnxV41yXfurZEOmb4H//pW1aEwNaXGOLAi8rPNpyBoMzpoGsbkO3kNbwBUep3ZLt1sFMK8LUo1cjVkoHBzyWlqcfoPHGqSYoHpNEaSlJ0u0rKvDE+u0dMSiDo//71QWLRNEVyfh377x48djzZo1uvdWr16NcePGged5HH/88ZAkSRUvA8Du3btRVVWFsrKyiPZBxBbtr3tjzyF/llZ0aemMMYhH9gAAhJ4DQq5n4oPX4VFIRlXYcGnpAMCly41SWdORgGXBsrQA4Kfdcp0ppZWDgoNzwcTJRhHnyPYXPeQEmEvkDvDu7f+V//dpWcwmAWL1LoCJ4NJywaUHtrhQaG4L3bldi9GAcATJ0kp2pWXAb0BGel8oGp0Q9o76GXOdKHGpLzxIzyeCSBYJ//bNnTsXmzdvxpIlS1BRUYHXXnsNX375Ja688koAcths1qxZWLRoEX744Qds3rwZ8+fPx4QJE1BaWhrRPojYojVyjKnYSh2eaCorAz5jwNUiNwzN7R1yvdAenuS5eMIVHgQAPkM2eIJ5eJTrUKooe0UJtY1t+N9O2TiaOFwvLla8O63MCk4wa5pdApYhcgkG9651EL1uf0aRwEGslIsSCoWDw2YtNjs7ZvAETUtPVh0ezm+OiGodnsi2VdYL3S3dt14nrkdXeJAMHoJIGgn/9g0ePBjPP/88VqxYgXPPPRcffvghFi9erKuf88ADD6C8vBw33HADrrjiCgwcOBBPP/10VPsgYofWuNBODBJjaG7tWB0esboCAMDnFoMTQm8bqg6PootIZqXlUEaX4lExGjyMMdVIUsJHXlHCt/87BMaAYX2z0buHIfPJp99phly9mtNM0KaiIeAdmYDbCW/1Xs358RAPKwZP8Po7CtqaPeEwGjDBDJ5kVVoG/F6UeFVa7lRIS+MdIoOHIJJHXHtpvfXWW0HfnzZtGqZNC93n2uFw4MEHH8SDDz4Ycp329kHEDm3pGK3d0drmVSeKaEJazOuGa90nAABT7xFh1w3WLR3QTkTJC2mF9PD4QlpwtYC5neAs9oD1bRYBTa0euL0Svtt0CAAwfVyxbj+e7StxnkNu/NnE7LpjS4yB43jYeg9B64518FZVALDAAg/Y98sgHfwZACAUDQ17LRecVAIGYNrowCKhumuKpPBgkkJagM8gYf46RxGHtNoxeCSmX68jaIeCNDwEkTzo20e0ixQipKUIlrWdvSPBvf5TsMYqcI5sWMaeGXZdgQ/RLd33f1I9PMYigj44ix2wyp4arZdHq0OymmWDoaHZhUZfOnnpYL/WRmqoQtu3y1BkqgcANEoO3bFVT1EvWZAs1cges9PtmyFVrAbAwVJ6JoRcvRFlJNNhweVnHIdBxeEzLANCWkEEzjrRcoI9GYpBoraWiNTD4zvNUBFSfy+tzp8b4DfgCYJIPGTwEO3CQoS0mjoQzmKSF+6fvwYAWKfMBWcJ32hU9fBIRg+P/H+q9dJS11GEy81+4bJXcw02n8Hg9LWDMAm8Ltzh2anvC3dQzJH3a+jube0tGzzsyG6Y4UW5dYe8/+nXwDrh19FeWujrMYxz0JCWZp1Ei8mNBknUHp6QrSWi21/wY/j/Jg8PQSSPuIa0iK6B1tbQGh7NHcjQko4eALxuwOKAqd/YdtdXNTwB3dKV7JnEozY0bcfgkY7ug6TJ1PLqPDzydSm1c7QTIWNM7SD/dvNkHBBzUYNc/BbaEIy8rq1oEAAOXPMRnGjbgjTeDS6jB0wDJ3T6OnXXY7Aogk3cyQ1p6Y8XKw2PFAsNj87DQwYPQSQL+vYR7aITLWvmBbUGTxQZWmKV7IEQCgaB49q//UJlafk9PBEfOmZE4uHhsuRsK6nukPqeokMyCZw68QUzeKSa3WCNVYDJgk3uvjgs5vizj3yrqU0tbWngc2T9zVmOjQAAy/Dp4GJcnsFoQATzrOnS0hMcujGeTsRZWhGHtGLk4SGDhyCSBn37iLAYf/n6O6dL+NlXOyYaD49YJWtNhILIClj66/AYDB7f/8kIaQkRGDxCXh95naP71fcUo03QhK/81ZE14ayKHwAApn7j4IZsTCpH4oOEYEwFA9W/m5kN5qGxF/NrJ3yLOfhjQ0iBLK1oj99uSEsKvv+OnluqNg4liO4AffuIsBgnAkUs++ryLVi3rQYcgPHDeka8P7F6JwBAKAhd/VeL38MTIqSVRNFyqCwtAODz+srr1O5Xz1W5BpOmM7dSHVnr4fHu+5+83sDxIY+tNUQtgyaCCWZscPXDC65zwdnSA7aLJVZz8IrMgsarJBwjGp6EpKVTHR6CSAno20eExVhZmUkMXlHC6p+rAADXnjsSo0tCV/LVIrXWywUHOS5sdWUt/jo8ITw8Ee0ltkQkWs4uBHgT4GlTKy4rBpJJ4GHy7UMNafmuU6qvBGuoAnghaMo+F+TY5r6jUXfGErzRciJahPi3Vgll8CQ3pGUIuUWs4ZH/D23w+NbrhMeK19ylJFomiORB3z4iLMbMa1Fiuu7Uo0vyIt6XWCV7d/jcYrU2TXv46/CE8vAkI6TlC7OF0/DwJlVbI9buA6ANaWk8PAYNj+LdEYqGBR2jUB4J5SNJhFg4tIdHG9JK7KPFeNmRepgUQyZka4kY3Gek4SGI1IC+fURYAjQ8mmrBQHQuenHfZgCAUBC++q8WIWSWlvx/qnp4AIA36HhENaTFq9qkQINnk7xO3zEAgBEDcgHIVZgBrYZHfyzlM0lE/ZtQTUZ1Hp4EG6Lx0PBovZuxy9KiOjwEkSwoLZ0Ii9Hg0Xp4eI6LeGJhkgjvng0AANOA4yM+vknVy6ROHR7Fk8Hg6x4fYgyE3D7wwm/w6D08vpCWT8NjMvGQ6ivVlhCmfqUAgGvOHoE1W6pQNiwfQGCWloKyb1OKeHiOmSytMBoe7VuxytIi0TJBJA8yeIiwGH/5Sj4NDwB10o4E8fA2MFczOGt6u+0OtPjr8IRqLRHxrmKGdvITwxg8qnD5qC+kJSlp6TzSpUZkcq1wuuTrM/Mc2v7zOsBECH1Ggc+UDZx0u1nXciKUd8nv4UmewZMKvbTU11FqeIKFtLRGUOcqLfv/ppAWQSQP+vYRYTFOBDqDJwqdhne33BPK1H8cOD74hBmMUFlaCsn08ADtpKb36AdwAlhTDcSj+9VryOUbccLuF/GnrL+D87oAAMM9myEe3gqYLLBNviTkPoOlpQP+cJmQAO1MJGnpiS482OEsrTCNYPUhrdhcD2VpEUTyoG8fEZaAkBZj/vTqKNzzajgrSKp1OELV4QmVVZMItF6UsMJlaxpM/eVq0p4t/1aNkmniKpiYB5l8G4637sYw80GMb/oXAMA6/gLwmaHT/EOKljVFDeONLRINT5I9PJEeX9mOBfkcpRiFtLT3CGVpEUTyoG8fERajJ4FpPTwRTq6SsxGstR4AF1U4Cwjj4VF7HEW1u5igndjbM7zMw6cDADw7/gvJ48QgUyUGiRXq8tNsP+Gy9BXgwWAafALMo2ZEdGzj/KyGtBIwIJZQIS2OU8M3iQ9pBZ5LJPjT0gOXae/9zjh4OiryJwgittC3jwhLMNGy3+CJ7PaRag8AALjMnuBM1qiOrxzD6ElRXyWjeSjHqdlh4Tw8ACD0GgYuswDwtCGjcgNOs/8IADiSOxpuJiBPaIaN86LG3h+2aZe3GzpRJ+gADY+/inO8CaXhAfwGV7I9PJFG9sKHtELvPxq0JRUoS4sgkgcZPERYjPO5JDF4fVlakQowpbqDAAAhp3fUxw8mWo5VunBniDQ1neN4WEbIXp6e+/+FYebDYAAO9zkNa11ye40dngL8r/fF4IT2cwhCTdDaKs7xZli/nJDLeNXgSeyjJUC0HGWl5WAhLeY3qyM2oIKhzTBMhuaMIAgZytIiwmKcCESJqV3CI80Ikmplg4fvgMGjHEOUGBhj4DhO/8s76j3GBoHnIEosQFsUDPOwE+Ha8BksrkYAwEHzADBHD3zaejy2enphi6cXzrBE5vlqP0srfobGI9dMwqGjrRjRPzfkOrKhIyU8pBUgWo5YwyP/335IKzYeHoIgkgd5eIiwBCs82FEPD5/bAQ+P5qe1Mqlrf3kn6xezMqGKEYinObMNlpGnqa8rHGMgCBzcMGOzpy88MOmah4Y9bhJFy/k5DpQOCt9GJGVCWjHopRUrw9rbjheQIIjEQAYPEZZgdXg8UWh4GGMQFYMnp7idtQPRTuDKpK7XVkS9y5gQScd0LZYRp8JlzsQhbzaqHIMCjMVIM97aT0tPbsiET5LBoz1cNMdW6vUYe8Zp3+O4znp42vcCEgQRf8jgIcISLBtIbZEQgTeBtdYD7laA4+WGmlGiDZs9+/GP+HHXUcMv7+RM8JF0TNfC2dKxZvCNWNI4C4JJCAg9RZquHDpLyydaTrB2xogyLsksPBiNcRJKBA74x7gzKelA5PcIQRDxhQweIizGX74Si87Do4azsgrACeaoj6/NiPplTx2e+GDTMSVa1uJmAkTIxo7RwxNpeFBtdhnCw5PsLCDFOEh8SMv/dzTH9oe0ApfFqpo3eXgIIjUgg4cIS4CGR9daov3bR6zZDaBjgmVA/rVu9IYw3fIO7bbTROvh0a4r8FyAYRKxh4fT70tBFZIn28MjJF/D47BFnosRLqSl3Pud9fCEqhJOEERiIYOHCIuxK7ds8CjehPZvH++utQAAofeIDp9DYOPQFBItR2HwaOsXGccucoMnuMjWX4cnuR6eZIW0tIdLs0XuSQwnWlY+2s7eYxTSIojUgAweIizhCw+GnwjEuoOQju4DeAHmgWUdPgfjXOT2aOqadHivnYP3eVKiCWl5NWGnDhs8IerwqKLlZIe0UqAOT7o9Cg+PquEJXBazkFawnRMEkXDI4CHCEpClxRBxWrp352oAgFA8CpwtPWbnVN/sUv9OloenMyEtE88HhrQi1fCEKJSXyOah4RBSQMOTZo+Nh0d5q9OiZQppEURKQAYPEZYA0bLE4I2wjYGnYg0AwDxoUkzPqb7Zrf6dbA1PdB4ef9gp0MMTWQf5UFlaymeSbNFyXpYNAJCbGV0Lkc6i9/BEYfCE0/BIsfLwkMFDEKkAVVomwhKsZ5PXK78XzivB3E6wxioAgKnPqE6dw8WnDMa2fXWoqW/DgZpmNOg8PJ3adYfpiIZHjIWGJ8RxU6UOzxWzhuNIgxN9CzISelzt6HVIwxM2pNW5Me1bkIGDR1o6tQ+CIDoPGTxEWIL20hLbF8hKDbKxw9kzwVnTOnUOM8r6YEZZHzzzt804UNOM+haNhyfJdXg6puEJEtKKMksrpGg5ySEth82EvrbEGjtAxz08XIjxBDQhrU4akb89dTCy0y04YWT0dagIgogdZPAQYQlsLeEPzYTz8EgNlQAAPrMgZueSlWYBgGPWw6Mainxgqn1nNTzeFKnDkyx0Gp4o0tLV8QyTlt7ZeyzdbsaFJw/q3E4Igug0pOEhwhKsSaUngjo8ki+cxWXFzuDJ9Bk8eg1Pkjw8nGLwRJ6B42/wyQUWHjRH9lXkQmVpJaB5aCrTWQ1P8MKDvnWowzlBdAm659ORiBjjD19J21oijKtfCWnxMTR4stJlIazi4UnmNNSRSstaDY8xHBiph8cfStO/r/UedUe0YadosrT8aenhCg927twIgkgNyOAhwhKsW3pEHp54GDyqh8cX0kriRNSRtHSthkfg9eqjaAsPBoiWJQppKXQkpBW88GBsRMsEQaQGZPAQYQnWLV2pwxOu0jJTDJ44aHiUkFYyQw0dEi0rqeM8p2uZwXGRe2aU1YyaE1FMDdFyKtDRtPRdhxqxp7JRXaYMMdk7BNE1oKcjEZaglZbb8SYwVwuYqxlAfDw8qVDXRBUtB/EMhMIr6nU2ZhPn+5+P2IsQqtJye59JV6fNLap/d6TwoMcr4cE31+H+19ehtc0LwG9UkoaHILoGcTV47r77bixcuDDg/b/97W8466yzUFpaihkzZuDll1+GKPofWHV1dbjllltQVlaGCRMm4L777oPT6dTt4x//+AfOOOMMjB49Gueeey5WrVoVz0vptgRtHtqOh0dNSXdkgzPbYnYuimhZIZmhho54eERDSw7FGxOpfgfwX3OAmDxFKi0nC6fPSAEi6/GmoNxCWoNJ8fL4Cw+SwUMQXYG4PB0lScLSpUvx/vvvByz77LPPcM8992DOnDn47LPPcNNNN+Gll17CCy+8oK4zb9487N27F6+//jqeeuoprFixAvfee6+6fPXq1fjTn/6Eiy++GJ988gnKy8tx9dVXo6KiIh6X061hxuahjLWblq6mpMfQuwMAFrMAu9VfkTiZ81CHCg9quqUDft1OpPod7baKIfqfjQfxzfoDKdM8NFm0tHk6tJ3Ww6Owv1r2TlJIiyC6FjE3eCoqKvC73/0OH374IXr16hWw/L333sO5556L3/zmN+jbty/OOOMMXH755fjoo48AABs3bsSaNWvw6KOPYsSIESgvL8f999+P//f//h+qqmTPwSuvvIJTTz0Vl1xyCUpKSrBgwQKMGDECb7zxRqwvp9sTvHlo+BRoqfYAAIDPKor5+WQ4/F6eZE5Eqmg5ij5J2m7p2n1EY/Bos8O8ooTH3l6HN/6xFUcb2uR9d9OUolaXt/2VgqCMp/LZAMDeqiYAmpBWNx1TguhqxLzw4OrVq1FSUoLnnnsON910U8DyW2+9Fbm5ubr3eJ5HQ0MDAGDdunXo2bMnSkpK1OUTJkwAx3FYv349Zs6ciQ0bNgSEyiZOnIivvvqqU+ceLuuooyhGwTFbH8XwrGfML761WYSgY9ZatQMAYO41OKoxjWSsbBaNhwdcXD6zSFDDJlzk941iHFl946YYOhZT8HEMhrKNxIA6TT0iRXdiCfGZdHXcHr/BEuz6Q91byrpaT93eyiaYTLxa84jnknefJYNj/pmVYGi8IifZYxVzg2f27Nlhlx9//PG6101NTXjvvfcwdepUAEBVVRWKivSeAYvFguzsbBw+fBiNjY1obW1FYaG+THt+fj4qKys7fN48zyEnp3MtEMKRmWmP277jid2u182YzYLq6s/JcQSMmeRxoa56NwAg77ixMGdHP6bhxipNcz7x/szCoYyLxWqK+ByUOTU3Jw05OWmwWuSvn80W+T7qnX5PRqtGd6JM19nZgZ9JdyPc9RvvrbQ0ubaTNjJ5+GgrrHYL0tJk/ZnZJHTLMT1Wn1nJgsYrcpI1VlEZPAcOHMApp5wScvmqVasCvDfhaGlpwXXXXQeXy4XbbrsNAOB0OmGxWALWtVqtcLlcaGuTXffGdZTlHUWSGBobWzu8fSgEgUdmph2NjU5VtHos0dysH1Nnmwcu30TrbHWhrk7fFNFz8BdA8oJLy0WTlAauLvKmiZGMlVaiwgEBx08UXq88Bi0tbvUcqmpbsfrnSpxa1idoA0u3uk0b6uoENcWcR+TX0dwk3/9eUcKegw0By50tgZ9JdyPY9Ye6t9qcspeszRAS+9+WSrh9uh6JSd1qTI/1Z1aiofGKnHiMVWamPWKPUVQGT0FBAb744ouQy7OysiLeV01NDa655hocOHAAr776KoqLiwEANpsNbrc7YH2XywWHwwGrVf5FZlzH5XLBbu+c1ej1xu9mFUUprvuPF17DTekVJbXwIAcu4JrcB7YCAITCIb4QTvQp5OHGyqh3SdaYKnaXx+s/18+/343vNh2Gw2rCyeOKA7ZRxpJj8nkr2hCTwEd8HYquRJIYqmuDGOgseWOSCljM4cfSeG8p4+kxbFNxsBGFeQ4A8mfdHcf0WH1mJQsar8hJ1lhFZfCYzWadtqajVFRU4Morr4QkSXjnnXcwePBgdVlhYSG+/vpr3fputxv19fXIz89HdnY2HA4HqqurdetUV1ejoCC2WUFE8MKDxvRqLeLhbQAAoWhIXM7HakmNLK1gaemKjiaUgFY0iL1NHREtayoD19Q7A5Z31yyt4p7pOFDTjPIR0XUkV7O0DIZ9bVMbCnLlH1CUlk4QXYOEK4f279+PSy+9FHa7HX/96191xg4AlJWVobKyEnv37lXfW7NmDQBZ/8NxHMaNG6e+p/DDDz9g/Pjx8b+AboYx61rSNg81uBGZ6IVYJZcGEIqGxuV8LCatwZO8iShYLy1F+KoV0CowxnTNQwG/YDYag4fT9NKq8WVmaemuwslbfjMGl84ciotPGdz+yhqCpaUrr6l5KEF0LRL+dLzjjjvgdruxdOlSmEwm1NTUqP8AYMyYMRg3bhxuvvlmbN68GatXr8bdd9+Nc889V/XgXHbZZVi+fDn+8pe/oKKiAo899hi2bNmCSy+9NNGX0+UJKHDHGLxef08o3bpH9gCiG5wtA3x2YEmCWGA1p5aHx6vp4qkYNEZvgXYZAJh4Xvd/dB4e+X+JMRwJ4uHprmnpWelWnFjaW3d/RALnG3pj6NbtkTSFB2NyigRBJJmYZ2mFo6qqSvXMnHPOOQHLt23bBo7j8Oyzz+K+++7DpZdeCqvVipkzZ+L2229X15syZQoefvhhPP/883jiiScwaNAgvPjiizEJtxF6FI2DwHMQfZ3Sldo8xpCW9/B2ed3CwXHzvlgtfuMgqd3Sg1Q8ViZNo7dAuwzwj5vq4YnCK6P1SDR4A7Vu3dXD01GU8QzQonlFVX1GIS2C6BrE1eB56623dK8LCgqwbdu2drfLy8vD008/HXadc889F+eee25nTi/pVNa2Yvv+ekwZVZSyxc38xg0PURLh9vhToY0eHrHSp98pjE84C0idkFYwDY+i0Qlu8Gg8PIqGR4hew9Nek9FIm5ASMorBY5TWyyEtpZdWgk+KIIi4kFAPD6Hn3a+346ddtcjNsGLkwLxkn05QJE1TSpcHcGkMHu1EzSQJYqXPwxMn/Q6gD2kl08Wj6HDEIBqeYAZPVZ2cUcUBuuwsoGManlB01+ahHSWU0ez2iNRLiyC6GOT/TiJNLXL/n7qmjtcPijWMMTz+1414+qPNYIypomVlctYKcrXeBKl2P+B2AmYb+Lw+cTs/bZZWMsWkwUTLakjLoAepqmvFMx9tBgCMGOivU+XvpRW57qS9a+6uzUM7Sqjh0ouWE3c+BEHED/LwJBG1EF1bx/oARcqhIy2wWQTkZrbfubzZ6cHPe+rk89MINxWDx+U7Z5PA6X75qt6dwsHg+OiEo9FgSZES/4Lv2kUWxMOj8YIBwNdrD6Cx1YO++en4w9kj1PdPGFmImjonJh6XH/FxjQZPut2MZqe/cWZ3TUvvKKEMSLdXUsO57XnVCII4NkiN2aObooQ+Otr4MBIaW9xYtOwH3Pr8fyNaX+vBcXlFMOgFysryAP2Oko5eMKjT5xwObUgrmfNQuLR0o4enyVfNd/KoIjg0FZhLemVh/m9K0btnehTH1b8+payvfxnHUQp1lBjDVcpLrcFDY0oQXQMyeJKIYvA44+jhqdakLotS+5UtFa8TALjdIpRNlIwiRbQcYPBU+wye/Phmyll0acfJFy1ru6WHytJSjERtOK6jaCff7HQLhg/wh8jIuxM9RgPS5utv5vGKakiLRpUgugZk8CQRpVdPi8vTzpodRxsCao3AsNJ7eCRdlhbg92JoxbGSsxGsqQYAByF/YCxOOySpUodH8fCIEWRpKULvWITjtNl8E44rgN3qj0pThlb0GL03dqt8f7k9miwtGleC6BKQwZNEPD5vSiSGSEfRSEwiOo42C0ubqWKsFaP18EjVuwAAfHYROIujM6fbLhazpg5PChg8kmaAlSKExpCW4hWLtiheuOMCwOTRRbDb/AaP0etGtE+AwePz8Li9oirYpywtgugakGg5SUgSU2uzxFPDo60EHMlxdCEtjxiyyKB2clXCWXycw1mA0cOTvIlIqZIc1MPjMXp45NeWWBg8HId5vx4Nt0fEgKJMNLb5Py/y8ESP8Ray+cKOHqq0TBBdDjJ4koQ27BFPDY9WY9LS1n7oTBfS8khgioYnjIdH9Hl44h3OAvRGQ1IrLUchWlY8PFrvVGcoHdRD/dtm1Xp4aGaOFmO4ShlPBv93lETLBNE1IB94ktBOipEYIh1F29IgMg1PKA+P0eDxVahlkt/gKUish0c0djZNIEJQDU9w0bKSyh+LkJYRm0YITTV4osdozGjHU/kukL1DEF0DekImCa1hEc+QlnZCjqTej9ur9fD4DR5jBpCStcUaqwGPExDM4HN6x+KUw6L1kgSraJwo/KLlIM1DA7K0FA9P7A0eh0bDQ1la0WMMi1rNguo5bPN9buThIYiuARk8SUI7Kbo9UkC35lih9/C070nSiZa9UkjRsvJaPLIXAMDn9YlrwUEFradJqzdKNMaQFmMsjMHjS0uPg8FjEnjV20QenugxDpnAczD7jGqX6uEhg4cgugL0hEwSbsOkGC8vj1bDE21Iy+UOHdJSe0nV7JFf9+jfyTONnmR6eIwhLa0nTWJM9fx4RUldFisNjxaO49T6PuThiR6j90YQeLVBrduteHgSfloEQcQBMniShNE7Ea/UdK9OKxRdHR63pviayRTcwyMd9Xl4evTr7KlGjbYDeaIxenhEw7ko46g1IC1R9MyKBkV3YqKZOWoCDB6OUw3TNvLwEESXggyeJOE1enjiZvBoPTwRZGlpDDGthsc4mZoEXg7j+EJaQhIMnmRiMnh4vIYq1oooXUlJ5zkubllUSnVggerwRI3RlhEETm3mqoS0KFJIEF0D+ionicCQVnwytbQTcdQeHm3zUFNglhZrPgK4WgBeSIhgOZVoz8OjGLTalPR4eQoUDw/V4YkeY1o6z3NqRWyXmzw8BNGVIIMnSbg9ifHw6DQ8kRQe9Bg9PPLfwerwqILlnGJwghndCWNrCWOKvKIvcsWwynIo1JAWeXiiJiCkRQYPQXRZ6AmZJDwJ0vCI0WZp6bLHRLAgvbMA2eMjqYLl7hXOAgJFy8YsO4/q4VGqLMfvq2YlD0+H4fhAg8ds0mdp0bASRNeADJ4kkagsLa8UPEurzR38ePrCg/7moca09Ay7GWLVDgAAn4CCg1pSoS6Kcg5SCA+P8vnGs+iggs2saHiSPy7HGsYHoMBzar0kSksniK4FGTxJwphSnYgsrdY2LyTG8O8NB3D9E9/hfzuOBKwfENKSAtPSTQKHk8cU+issFw6Oy7mHwhxHb0mkKAJhZXzFAA+PPI7xLDqoQCGtjmPU8AgCr3p4lB8h5DkjiK4BPSGTRKDBEyfRskbDwwC0uUTsOtwIxoA9lY0B62s9T9qO0VrR8nnTBiLdeQgQPeCs6eCziuJy7qGwmJJ/2zp8PZecLhGMsYAUeX+Wls/gieM526wU0uooRu8Nz/k1PEo4MsPRvfRpBNFVoeahSSKgDk/cCg8GGlaqviRI4T594UEJNouvaJ6Jx8yJfeF0eXF6WV94f/oSgOzdSbTLX65nE7/+Y5GgtHSQGEObWwwpWo5nlWUFJS2dPDzRY7QRtWnpChkOSwLPiCCIeEEGT5JQJkSH1YRWlzchdXgAOTVdObbHE8zgMXh4fBM5z3O46ORB6jKxUtHvJDacBcRXABzxOZjklg6ixOB0eXU9tYDALK14hrQmHJePLXtqUT6iIG7H6KoEEy0b768MO3l4CKIrQAZPklAMi6x0i2zwxMvDIwXx8IiKhyewF5XLqxUty+EaQC8UZoxBrNoJADAlWL8DQNVYJBOO45BmM6Gx1YOWNm9gSMtQhyeeHp7ePdNx2+/GxW3/XZngaekGD08aeXgIoiuQ/Jmjm6KIWrN8D1NnvLK0DBNxq8vrL4oXNKSl7ZYuqRoerbiTNdWAORvlgoNJSEmPV4uGaLHb5F/+rW2eMB6e+KelEx0n0ODhA/RWpOEhiK4BPYWThGJsOHyTZmcaYba5vfhx19GASRcI1PBoQ1pavY56Xh69h0dSPTyaffqys/i8vuBMif/1mwoeHgBI8+l4Wl3J9fAQHSegtYSmW7pCJml4CKJLkBozRxfH4xVRU+80vKcYPCbd62iRJIaFL63GEx9swobtgWnmXoOYtlWr4TEcU9vZG5BryyjraH8JizW7AQBC/sAOnXNnKch1JOW4RpRMrdY2b0BrCX/YUPHwkMGTigSmpetDWiaBU9P+CYI4tiGDJwE8/u4G3PLs99i+v159TzEk0jpp8Hyz/gAaW9wAgAPVzQHLjRWAXR4xYDI2npOWNqW8vmZiEKsrAABCz+QYPBecOBBlw/Lxx1+PTsrxFRRjtbUtjGjZ7e+lRaQexpAWz3E6D2KGw0KFBwmii0BP4QTw/aZDAIC3vtqmvqcIhtWQlhi9wdPS5sHfvqtQXwerw6J4HpRfqW6vGDKkpbzm4Hf1KxWZlYmBSV5ISof0/MRWWFZIs5lx7bkjMWZQj6QcX0H57FraPIEeHlUnRSGtVMbYCV328GgNHtLvEERXgQyeBHKwpkXNenIH8fAoyyKlus5pEBkHanIUD48aOvNIIUNaLk34RZmgFQ+PMjFItQcB0QNYHOCy8qM6366GGtJyeXVd6QG/oeMvPEgGTypi9N4IPK+rw0M1eAii60AGTwLQPjT3VclhJ6UGTprN/wvSKHxtjzZDZpexAzvg7/Fk903OOg+PN7iHx2Lm/f2EFIPHNzF49/8IQNbvcFz3vn3SdCEt/WfnNRYeJB1IShI0LV0TfswkDw9BdBm694yVILRu8407agD4Q1jKpAkEdlBvD6dbv344D4/f4JFCanjUzt4mAVaD5oTjOLh/+hrutX8DAJiKR0V1rl0Ruy2MaNlYeDBFMsuIQLRGj1yHR6/hIQiia0BP4QSg1cr8vKdW957NaoLyuG1PuLyvqgkffrtTrdlj7HgerJCg4jVSwi8utxiy0nIwD48MQ9bOL+D679sAGMzDp8M88rSw59odULxzrS5vYPNQUa+TIg1P6qJ18vCabukAaXgIoisRV4Pn7rvvxsKFC0MuZ4zhiiuuwNy5c3Xvu1wu3HfffSgvL8fYsWNxyy23oLa2VrfOqlWrcP7552PMmDGYOXMmli9fHpdriAXaUFOzUzZSFKPDYvJ3Z27P4Pns+z34x+p92LBd9hI5XQYPjzvQ4FEmYn+zS7+RFBDS8vrbIGgn6AmWCmRU/J+8bPx5sE6eC86o9uyG+NPSPe320qK09NRFm5ou99IiDw9BdEXiMmtJkoSlS5fi/fffD7veG2+8gZUrVwa8f++992LlypV45pln8MYbb2DXrl2YN2+euryiogLXXHMNpk6dio8//hgXXnghbrvtNqxatSrm19JZRElf20YxOJRwkllr8LSTqdXilBtmKn23FA+Psn2wysleg4anRdOzyysytVcWoNGbmPTVZkvM1fJxRp4G67hzKE3Xh0NXeND/eQLBemmRgZiqaENaJkOlZfLwEETXIea9tCoqKnDnnXdi79696NWrV8j1tm3bhueeew6lpaW696uqqvDpp5/ixRdfxPjx4wEAS5cuxcyZM7Fx40aMHTsWb7zxBoYOHYqbb74ZAFBSUoJffvkFy5YtQ3l5eawvqVMYvTaK0FjR61hMAkwReniUyVP5X8mgykqz4EhDW0QantY2fZdxj1dSBbVaD4/2V28O3wIAEHr0D3t+3Q3F4GnRiJZtFgEer6Qan5SWnvponZU8z1GWFkF0UWL+s3P16tUoKSnB3//+dxQXFwddx+Vy4dZbb8W8efMwYMAA3bL169cDACZNmqS+N2DAABQUFGDt2rUAgHXr1gUYNpMmTcL69eujTu2ON0Yjxu2V08IVbY3Z7P9FGcxDoyXA4PGFtDJ9/biCtYoQVYNHfog3t4XW/WjDL9oJOttn8HDpuWHPr7uh1UUpY6fUO/J4JUiMUUjrGICDQbRMWVoE0SWJuYdn9uzZ7a6zePFi5OfnY86cObj99tt1y6qqqpCTkwOr1ap7Pz8/H5WVlQCAyspKFBYWBix3Op2oq6tDbm7HJmZTHDJpRJf8PwdAMcWcGrGx3WpSa7RIjIU9B0Wj4xElmEw82nwGTna6PFZujxSwveJ5SPf9UjXqfETmv26llozNImiKGDLVw2PO6gkhjtlGgsDr/k91MtP9v/5bfJ4zxZPmFSVoTe80uynm99exNl7JJNxYab2ZFouANLsZFhMPxoC8LHtcngupDN1X0UHjFTnJHquoDJ4DBw7glFNOCbl81apV7Rob3333HT7//HN89tlnQbUgTqcTFkugG9lqtcLlkq2Htra2gHWU1263u93rCAbPc8jJSevQtuFw+jw5dpsJXpHB7RHBeP+v/YKeGbD5QiNWmyXsOageII5HTk4aFH9Qfl4agBp4RClge0UW1DM3+H7tDv8xeZ/hlZFuhcl3Q6ZxLlg42UjKKy4GZ4r/L97MTHvcjxEr7FYBTpcIp8+Tk5FmBdAMiTHYHX6jPb9nZtBK2LHgmt9BZQAANSdJREFUWBqvZBNsrEyah29uThp65jpw1xUTwRhQWJCZyNNLKei+ig4ar8hJ1lhFZfAUFBTgiy++CLk8Kysr7Pa1tbW44447cO+996KgoCDoOjabLajR4nK5YLfLg2S1WgPWUV4r60SLJDE0NrZ2aNtw1NbJ+zQJPMyCHHbaf7gBgOw+b2x0qnHFuvpW1NW1hNyXInhubG5DXV0LGptlA9BmkifSNrc3YHtFK8TE4DV+jhxtQZrPhd/Q2CavK0lqx2jFu8NsmahvcgPomEEZCYLAIzPTjsZGZ0Cad6pit5rgdIk46msOaxZ8n4VLRHVNk+89Ho0Nsb+3jsXxShbhx8rvi2tucsLCMfTrKf8ICPd97KrQfRUdNF6RE4+xysy0R+wxisrgMZvNKCnpeP+kFStWoKamBnfccQfuuOMOALKhIkkSxo4di+XLl6OwsBD19fVwu906L051dbVqJBUVFaG6ulq37+rqajgcDmRkZHT4/LztaGg6gkvJpBJ4mAQODS1Anc+wMJt4eL2S+guzze0NeQ7aTuZtLhFer6Rma2Wo4SoJHo+o85wpWqFQhe+cLv8xFU2QWeCRl2kD4Dd4JHtuXMYnGKIoJexYncVhNaEWLrWBq6J9cntFNPuy6qwWIa7XcyyNV7IJNlba7wtj8XkOHIsca/eVJEkQRW/7K8YYQeBgsXBwOp0BBUgJPR0ZK0EwgY9RGZSYa3jCcdppp2HcuHG695YsWYLKykosWbIE+fn5OP744yFJEtavX68Kk3fv3o2qqiqUlZUBAMaPH481a9bo9rN69WqMGzcuZgMTK9R6O5pifo2t8uSoGCFqKnOQ1hAK2gwsf5aW/OXO8omWJcYgSgwmn5eBMaYRLQf/qLVCZzVLy8SjZ7bsKVMEy0jLaf9iuyFKA1HlM7WrGW+SmkWnCMaJ1EQbaYxX2JGIH4wxNDbWwulsTto5HDnCQ5KOHeMwmXRkrOz2dGRm5na6JEpCDZ709HSkp6fr3ktLS4PNZkO/fv0AyGGzWbNmYdGiRXj44Ydht9txzz33YMKECWoK+9y5c3HeeedhyZIlOO+887BixQp8+eWXWLZsWSIvJyKULB2ziVezehp83gCz0eAJ4+LTio3dQdLS1fU8ouoxkhhTnfWOUAaP5hecv9KyoBo8akjLQRlawVDGVTFs0+yyAeR2i2oI0mZJ6NeMiBLO0FqCOLZQjJ309BxYLNak1AkTBI68OxESzVgxxuB2u9DcXAcAyMrK69SxU/JJ/MADD+Dhhx/GDTfcAACYNm0aFi1apC4fPHgwnn/+eSxevBhvvPEGiouLsXjx4pSrwQP4jRiziVdTlhtVg0dQlwHh6/CE8/Ck280QeA6iJKdBp8nRKN1NFcrD49EZPP4U6qx0C8wmXjV4QAZPUBw2/bgq7SYYgHpFY0WNQ1MaXaVlMniOKSRJVI2d9PTkCcxNPnkC0T7RjpXFIid/NDfXISMjp1NRnLgaPG+99Va76zzyyCMB7zkcDjz44IN48MEHQ243bdo0TJs2rVPnlwgUg8Is8KrRUd8kT4RKwT+zEL3BwxhTNTc2iwCLmYfTJerW03ZfN5t4mAReLUSooA1puTQhLZ7j0CPLhhy3LLalGjzBMRo8dqugGp91TYrBk5K/KwgfOg+PQAbPsYToS8ZQJkWia6J8vqLoBc93vBhoagleuiAeTfViu2/iq6yVjQglFKXU4Qlr8Li1Bo8kGz2+1zarSdUHaQ0YryZOqu8CzWD1ZVsF8/BYzQLEukM41fYjBpjlvl1cGhk8wTCGCk1BDFvy8KQ2WqcOT21Tjkmo3U3XJlafL/30jDNaDY/dpoiW5eydbF/huo6EtJTGoRwne2SsPqNJu54S0lLc9OlmL1pdwKm2nzDLvhFfOsfA7fZn3SnGUsG+f6L1P//CeE3Vai69c7HTrooSwlIQeA42i4Bmp0f18JBoObVRQloCz9HESRBdGPLwxBlVwyPwqodHIStNdtOF6qXlFSW1VUabVrTsFlX9jt1iAsdxGg+PpNsekN307vWf4A7zmzjF9hNOs/8IngPOcGxC0aF/+ffrlVDA1yN7zzcAY6hPH4CD3hyscQ0EZ+++BdjCYQxpCRoPD4W0jg2U1hI86XcIoktDT+I4o2p4zDxshvBHgIdHUxywqdWNO15ejVED83D12SN0nhsGoMnnJbL5vAdWX6FAt0eEV5TwyDsb1Ad4X1Mt3Bs/B8cBZzs2AACaJSvSeRf616yA9+BEmHoPh9sjYrTlAABAKB6J6pJL8NwnPwEAylMs3T9VMBo8Jp5TU9PrSLR8TKDc2iRYJoj2ueGGq1FU1At33nlvsk8lamgWizNKmMhsEtSJUCHL1wMrmGh5f3UzWtq8+Gl3LYDAHlhKarviNVI8PC6PiIM1Ldh1qBE7DzSAg4RfW1cCjMGjsW/faynHyrYhAIC27/4C5nHB7REx0iwbPKZ+pcjLsqnrk70THKOGRxA41cOjGqXk4UlpFN0OGTwE0bWhaSzOaNPSjanhWT4Pj9KdWVsTR6mi3OL0QJQknYcHCEx59lf4lVRjCAAGmGpQxNcCZjs+tP8Wuzw9sdHdDz95+uCz1nFoFTLBmmrg/t/fIXhbMcAki5RNfUtRpOm/ZaLGeEFxGDU8QuDnTB6e1Ear4SEIoutCPz3jjFppOchEmBPGw9PqK1qnhK+MBo9Sy8dmEeDZ/j3Oa/gYguU4uNyD0eAzhgBguPkgAMDUbwxaj/TAU02/ks/HzMPl4bAxazom134K94//xHhhBHiOgWUXg8/oASuAR/8g1zYigyc4aUFCWsbQpY1EyymNolOmbtddB8aYTs8Yb0SJqbVlLGY+avH7lCnjcdllV+GLLz6H1+vBs8++gsLCIrzyygv46qt/oKWlGQMGlODKK/+ACRMmoaJiJy699GK8+urbGDp0GADg9ttvxYYNa/HFF/+CIAiQJAlnnz0DN944H6effgY+//xTfPTRX7F//37wPIchQ4Zh3rz5GDZsOADg178+CyeddApWr/4edXW1ePDBxzBixCi8+OIz+OqrL+HxuHHOOReoulKFd999C59++hFqaqrRo0dPzJp1Ni699IqUTAAggyfOKO0izGY+4Jd+pi8tPaPtMBxcm97gafP3hGlscetEy4Dfw5Nl9qDt+7eRLTpxSfpKHNrbhp2FpyONa4ObmTBCCVH1GQ1Lvf+BnmYzw+1xYY+pBFN7DoRUswtn2jcCAPi+Y9X1lIrLRHCMRqzA8wGhSwpppTZKSItS0rsGjDH8+e0N2HmwISnHH1Schdtnj4t6wv/kkw+xZMnT8HpF9OnTF/feeyf27t2Nu+9+AD175uP777/DbbfdhIcfXoITTpiCoqJeWLt2NYYOHQZRFLFx4zq0trZi+/atOO64Efjll5/R1NSE8vIpWLHi33jiicewYMEijBkzFkeOHMGTTy7GI488iNdff1c9h48//gCPPvoEMjIyMHDgIDz55GJ8//1/cOed96CgoAhvvvkaNm3aiF69egMAVq78Dm+99Rfcf//D6NOnP37+eTMefPAeFBX1wumnnxHTcY0F9CSOM0qYyhjSSreb5UKAh7ehZNMzuDUzDZ96f6Mub3V51L8bW9y6+jqAX8MzxrUW8Djh4h0wi63oVbsOuQ1bMSW7GQ2SAzlCKyRwMPUZDfPWver2aTYT6ppccHsZrOW/hnP5YwCATe6+mDg29W7UVMVmEcBzHCTfrx6thkfBaAARqYWq4aGig12HY/CjPP30M1Rvy4ED+/H11//EX/7yDgYPHgoAuPjiOdi5cwfeffdNnHDCFEyePBVr1/6AOXN+jy1bfobJZMbIkaOwYcM6HHfcCKxatRJjxoxFZmYmsrKysHDhXZgxQ/bwFxYW4cwzz8bSpY/pzmHSpMkoK5sIAGhtbcE//vF33HLLApSXTwEA3H773diwYZ26/qFDB2CxmFFY2AuFhYUoLCxEjx75KCgojPt4dQQyeOJMKA2PkqHl/t/fAQB5Qgtmtf0dzDMRnNmKFq2HpzXQw9PQ7EYW14rBLbJX5sfCs7F2Ww2uyPovbGIzwAE5glzg8DBXgCxbulrgEPBrT9xeEabeoyFOmIsP/r0T6zyDMcVKXp1I4TgODptJ7Ywu8IEGD3l4UhvS8HQtOI7D7bPHJTSkpW2X0JGQFgAUF/dV/96+fRsA4LrrrtSt4/V6kZ6eAQCYPHkqPvvsE7hcbVi79gccf/x4FBb2wvr16zB79qVYtWolZs48EwBQWjoOe/bsxuuvL8PevXtw4MA+VFTsDGjiWVzcR/1737698Hg8GDZshPqe1WrFkCFD1dczZpyB5cs/w29/ez769x+IsrKJOOmkU1BYSAZPt0SttGwSdCGtrHQrxKP7Ie7/EQwcWiUzCvlqOL98AvaZN8OpC2l54PZIsHFuDDRVY6unFxpa3Jhp3wSBeSEUDkFT1hBs9VjwdeHlsBzZih+PWHBJ2nfIE1qwRxiAYYCm0rJfe6J4oFx9y7HaxVGRvA6gM3iEwNAliZZTG1XDQwZPl4HjOLV1TyIwmfhO3z9Wq789BmPyc/m5516Bw5GmW0/pJTV27HiYzWZs3LgB69atwemnn4GioiJ8/PEHqKw8jB07tuOhh04EAHz11Zd46KF7MGPGrzBy5Gicc8752LWrAkuXPhryHBQ3mXIu/mvV/HDPzsZf/vIufvppM9au/QE//LAKH374Hq644hpcdtlVnRqPeEAqvTjj0VRaNgm8anRkp1ng/vFLAICrqBQvNZ0CFzNDPLwVzn8+CU9rMyZadqLEVClreFwe/CH9G1yT8S9cn/EV+rorMMm6EwBgmXAhLD4vQiNzYHVbCfZ4e+Kpxl/hg5aJ+Mkia3LMZr2GR3t+iiha6wUiIkObmm4KEtIyipiJ1MKflk6PQyI1GDBAroB/9OgRFBf3Uf8tX/4ZvvjicwCy4TFhQjlWrlyBX375CccfX4bRo0shiiJeffUlDBw4CEVFvQAA77zzOs4661zceee9uOCCi1BaOg4HD8r6TqMIWaFv336wWKzYvHmT+p7X68WOHdvV11999Q988slHGD26FFdccQ1eflk+zjfffBWXceks9CSOMx6NhgeQJz+3142sNAvEfZsBAO4BU7H353q86f0VrnJ8BfHQFvwWO2BOl708h/bvRLa3h9rXapC5GoPM1QCA+qyhyCgcDOsh+eZ1eSQ0NMv6ngbmwPeuoRhhko0bq8aYSbMrHh7R97/fHUtEh7b4oKApPKhAHp7URglpUaVlIlUYOLAEJ5wwFYsX/xnz5y/AgAED8e233+Dtt1/HHXfco643Zco0PProg+jRoyd69y4GAIwcORr//OcXuOSSy9X18vML8OOPm7Bt21akp6dj5coV+PjjDwAAbrfb4NmRcTgc+PWvL8Jrr72EHj16oH//gXjvvbdw5EiNuo7b7cJzzz2FtLQ0jBkzFtXV1di4cQNKS8cG7C8VIIMnzmg1PIAsYG1sAQrMjWBtTYBgAtdzIIAN2O3pAccZt6L1iyUwe9rQIllg4zzo5dqFXtgFAFgjDUe2WIs8vhmtzAIMOweAv/BgQ4tLV88H8Kfbaj08ioZHMcgUUbSyHyJytLV4BJ6HXWMAKb3OiNRFMXNItEykEvff/2e8/PJzWLz4YTQ1NaJXr2IsXHgXfvWrM9V1yssnQxRFjBs3Xn1v/PgJ2LBhHaZMOVF97+abb8Njjz2EG264GhaLGYMGDcGiRffhnnvuwNatv2DMmOAGyjXX3ACLxYqlSx9Fa2srpk8/DZMnT1OXn3nmuWhoaMDrry9DdXUVMjIycNJJp+Daa+fFYUQ6Dxk8ccZtCBXJk6MThR65Po6QXwKLz7r2iBKEgkFwnHMnPvnr5/i2qT/SODcuytmMIWwXDnuzsNo+DRVVrer+HyruB8BfeLCmzhlwDkoNHW24Kl3R8PjOTxH4UUgregJCWhqRss3X64xIXRTPjok8PESSWLlyXcB7NpsN8+bdgnnzbgm5XWZmFlas+EH33iWXXK7z7gBAr1698eSTzwdsf8opM9S/P/ro84DlgiDgyiv/gCuv/EPIc5g9+1LMnn1pyOWpBBk8ccYY0jqzvB/WbatGL2klJABC4RB/Ly2P3CxUyO2D/2sdDg+T0MpseLttOjLZ8ahxChjS0w74DB4OQI8sOaNKCUUpndi1KGI6rafBn6Xl8/D4QltWCmlFTZoupKXvmUYi8NRHrcNDBg9BdGnI4IkzRoNn7JCeGDukJ5rffQ0AIBQNVZcxyBU7GZN0RQibWj1o4R3wMoZMh0V9PzfTqm5rNYSiTAIHr8jUvwGDaFnR8HgkSIz5RcsU0ooarYbHJHAwCf4xpJT01IfjSbRMEN0B+obHGUXDo/WuSE1HwJqPAhwPoWCQarQAsoGkVFlWfm9KjKnGS0aaXy+irYJsNFQKNX2wFA2PVrScm2GDwMsF8+qbXJqQFt0S0aLT8AgczJoUVRIspz48paUTRLeAZrc44/boPTyMMbhW/xWArN/hzDZdnyqPV1L7aDlspoBeTVkaD09+jt/gMa7Xu6ff4FFc9loPj80iqAbT4dpWNaRFHp7o0Wp4eI4Dx/lT08ngSX2oWzpBdA/I4Ikz/pCWAOZuhXv9J/DuXgdwAqwn/A6AXCRL1fF4JbXKst1qUvttAfKDOc0e3MOTn+PASaW91Ne9evgNHrc3sMaO2cSjKM8BAKg82ur38JDBEzVKSMskcKpAWTF0KKSV+nDUWoIgugX0NI4joiSpPZZMzqNo+ftDcio6AEvZ+RB6DlDXNQs8PF4Jbq+ohrTSbGY4bCYcPiqLlCXGdFqd/ByH7nhzTx+Kgb2ysH5bNcqHF+CT7+RU9jaX4r3x27dmE4/CXL/BoyyjkFb0KAaPVgOieHioj1bqo3xsJFomiK4NGTxxRNfLZcvXYG1N4DJ6wlI6C+ZhJ+rWNZt4wKVoeORMK4fNhFOOL8aWvXUAAKtF0Hlg8g2dzDmOw5TRRZgyukj3fpvb6zuG3sNT6PPwHK5tQZFP80MenujJSbeCg9wQVsEf0qKvWKrDUUiLILoF9DSOI2rRQXjh3fFfAIBtyiUw9RkVsK4a0hL1Gp5xQ3rirkvH471vdmBE/1xdGXCthiccSuNRq+8YPMdB4HnVyKmsbUVupk1eh9LSoyY304Ybfz1abQgL+D07NkpLT3lIw0MQ3QMyeOKI0j13vG0v4G4Fl9ETQvGIoOsqBo9Xk6WliGEHFGXijjnHAwC27atTtzH2bAqFYvDkZtkwpiQPuVmycaN4eGobXWhqkdtRUOHBjlE6qIfutRLmivQzIpIHT2npBNEtoG94HFEEy+XWHQAA83Eng+OCD7li8Cz56//w6X92A9DXd1EY0icbZ0/ujxvOD/QShULxBPEchz9eOAZzZwwFIIdglDDMvupmANRLK1acPLYY44b0RNmw/GSfCtEO1C2dIALxer14//131NevvvoSfv3rs2J+nHjtNxg0u8URt1dCFteKfoLc6NM8uDzkuqKvzo4oMVXorK3vosBxHM6dOhDjhvRs9/h3XToe5SMKccnpQ0Ouo2Rq1TW5AJCGJ1YMKs7CDeeP0mXSEakJVVomiED+7/++xDPPPJHs04gpZPDEEY9XwmjLPgCAUDgYfFpOyHWL89MB6FsRmIXOfTwDijJx1VnDVX1OMBSDR4FCWkR3QykhQOFHgvCj1Yt2FegbHkc8XhFjLHsBAJaB48Oue+nMoZhR1geFuQ5c/8R3ACIXJXeG4p7putckWia6GyeP7Q2B53Cipo4VcWzDGAO87gQejwdT2gGZLB1qGLxq1fdYtuxF7NmzC3a7A+Xlk3HjjfOxc+d23Hzz9bj//kfw4ovPoKqqCiNHjsKdd96L9957C19+uRwmkxkXXngxLr30CnV///jH3/HXv76D/fv3ITc3F2eeeQ7mzr0Mgq/1TVVVJV566TmsW7cGra0tGD26FNdd90cMGjQYX3zxOR5++D4AwJQp4/H00y+q+3377dfxt799gIaGBowYMRK33XYn+vTpCwBobm7Gc889hf/859/weDwYOvQ4XHfdPAwbNlzd/tNP/4a33noDNTU1KCubgKKixH3vyOCJI2JrI0pMvnDWwPEIZy/bLCYMKMoEADxyzST8tLsWYwblxf0c+xZk6F5TSIvobuRm2nDu1IHJPg0iRjDG0PrZQ5Cqdibl+ELBYNjPviMqo6e+vh533vkn3HDDzTjhhCmorq7CAw/cg+effwozZvwKoijizTdfwz33PAiv14s//ekm/P73v8OZZ56Dl19+A1999Q+88soLmDLlRJSUDMIHH7yLF198FjfccDPKyibil19+wtKlj6KhoQF//OMtaG1twbXXXoFevXrjkUceh9lswWuvvYwbbrgKr7/+Hk455TQ0Nzfj6acfx//7f18iMzMLGzeuR2XlYfz44yYsXvwUPB43HnjgbjzyyAN47rlXwBjDn/40DxaLDY8++iTS09Px5ZfLce21V+Cll/6CIUOG4f/+70ssXvwo/vjHWzB+/AR8992/8fLLzyM/vyCOn4gf+jkfRxxiA3iOocZUBCGzfc2NQn6OA9PHFScka6RPvt7DQ6JlgiCOdTgcW3qsmpoquN1uFBQUorCwCKNHl+LRR5figgt+o65z5ZV/wLBhwzFy5Ggcf3wZ7HY7rrtuHvr27Ye5c38PANi1aycYY3j77Tdw/vkX4fzzL0SfPn1x+uln4Ior/oBPPvkQzc3N+Oc//4GGhno88MCjGD58JAYPHoJ7730QVqsNH3/8AaxWG9LT5bkhL68HzGZZT2oymXD33Q9g0KDBOO64ETjnnPOxdesvAID169fip59+xAMP/BkjRoxEv379cc0112PEiFH48EO5ndJHH72P006bgfPPvxB9+/bDnDm/x+TJUxM2zuThiSN9jhuJ3S1XYOCo0ck+lZDYrSbk59hRXecEQBoegiCObTiOg/3sOxIa0jKZeLUMSUdCWoMHD8Wpp56OBQtuRl5eD5SVTcQJJ0zFtGknYfPm/wEAiov7qOvb7XYUFfVSj2O1yjpNj8eD+vo61NYexejRpbpjjB07Dl6vF3v37kFFxU706dMPOTl+XanVasPw4SNQUVER8jxzc/OQlub/kZyRkQmXS0542b59KxhjuOCCM3XbuN1udZ1du3ZixoyZuuUjR47Gjh3bIxmmTkMGTxwReB7DTjgROTlpqKtrSfbphKRvQYbf4CEPD0EQxzgcxwFma+KOZ+LBcVL7K4bh3nsfwuWXX4XVq/+LtWt/wAMP3IXRo0tVXY7JpJ+uQxlVocTGksQ0+wm1jgRTmB+9fJiogyRJSEtLw6uvvh2wTPEQARwY04+T8briCc1uBPpqwlqk4SEIgkgsP//8E55++nH07dsfF130Oyxe/BRuv/1urF+/FnV1de3vQENubh5yc/NUz5DCpk0bYTab0bt3MUpKBmP//r2oq6tVl7tcLmzdugX9+8t6tmi9VAMHDkJLSws8Hg+Ki/uo/9555w2sXLkCADB48BBs3rxJt93WrVuiOk5nIIOH0KWmWymkRRAEkVDS0tLw8ccf4vnnn8aBA/uxa9dOfPPNVygu7ovs7Oyo9/fb387Fxx9/gE8++QgHDuzHV199iddeexlnn30e0tPTcdppM5GVlY277lqILVt+xs6dO3D//YvgdDpxzjnnA5DDZoBskLhcbe0ec+LEcgwePAT33HM7NmxYhwMH9uOZZ5biiy8+V42oOXN+j2+//RfeffdN7N+/Dx999Fd8++03UV9fR4mrwXP33Xdj4cKFAe/v3r0bV199NcaOHYvJkyfj/vvvh9PpVJdLkoSnn34aU6dORWlpKa666irs379ft48tW7Zgzpw5KC0txfTp0/Hmm2/G81K6NEp2GACYKaRFEASRUPr3H4CHHlqMDRvW4bLLfodrr70CPC/g8cef7lCK+29/OwfXX/9HvP/+u5gz50IsW/YCZs++FPPm3QIASE9PxzPPvISMjEz88Y/X4brrroTL5cILL7yKXr16AwDGjSvD8OEjce21l+P771e2e0xBEPDEE89j2LDhuPvuhbj00ovxv/9txEMPLcbxx5cBAE44YQruu+8hLF/+GS699GKsWPFvXHzxnKivr6NwLA7VhSRJwpNPPomXXnoJ5513Hh555BF1WV1dHc4880yMGTMG8+fPR3V1NRYsWIBTTjkF9957LwDg2Wefxdtvv41HHnkEhYWFWLx4MQ4cOIDPP/8cFosFdXV1+NWvfoXp06fjiiuuwP/+9z/cd999uOeee3DBBRd06JxFUUJtbex1NiYTr2p4VFFbCvLTrqMwCTyG9QtdHDHeHCtjlSrQeEUOjVXkHEtj5fG4cfToYeTlFcFstrS/QZzQiZaJsHRkrMJ9zrm5aRAiLNIbc7VQRUUF7rzzTuzduxe9egUWFHr77bdhMpnwxBNPwGq1YtCgQZg3bx7ee+89MMbg8Xjw2muv4dZbb8VJJ50EAHjiiScwdepUfPXVVzjzzDPxwQcfwGw24/7774fJZEJJSQn27t2Ll19+ucMGT3dn5MD41/whCIIgiGQR8/jF6tWrUVJSgr///e8oLi4OWL5y5UqcdtppsFr9CvoLL7wQH3/8MTiOw9atW9HS0oLycn/fqczMTAwfPhxr164FAKxbtw4TJkzQqbsnTZqEPXv24MiRI7G+JIIgCIIgjnFi7uGZPXt22OW7d+/GKaecgj//+c/45z//CbPZjNNOOw1//OMfYbVaUVlZCQAoKirSbZefn68uq6ysxJAhQwKWA8Dhw4fRo0ePDp27yRR7/YriaovU5dadobGKDhqvyKGxipxjaawkKfkFBhWJDccBXbD9VEzp7FgJAtepeToqg+fAgQM45ZRTQi5ftWoVcnNzw+6jubkZr7zyCmbNmoVnn30Whw4dwgMPPICamhosXrxYFS9bLPo4ndVqRUNDAwCgra0t6HIAaoGjaOF5Djk5aR3aNhIyM6lrdqTQWEUHjVfk0FhFzrEwVm1tAo4c4Ts9EcaCY8FATBWiHStJ4sDzPLKyHLDZQjfDbo+oDJ6CggJ88cUXIZdnZWW1f0CTCQMGDFAFyiNHjoQoirjpppuwcOFC9WLcbrfuwlwul5omZ7PZ4Hbrq2gqho7Doe/+HSmSxNDY2NqhbcMhCDwyM+1obHRCFEnUFg4aq+ig8YocGqvIOZbGyu12QZIkiCJLmmiY4+QxE0WJPDzt0NGxEkUGSZLQ0NAKp1PULcvMtMdHtGw2m1FSUhLNJgEUFhZi8ODBuveU1wcPHlRDWdXV1ejbt6+6TnV1NYYOHaruo7q6WrcP5XVBQcebkMXzCyOKEqn4I4TGKjpovCKHxipyjoWxEkV51oxDsnHEKIcmY6d9OjpWyufbWcM24T64srIybN68WXeDbt++HYIgoLi4GMOGDUN6ejp++OEHdXljYyN++eUXlJWVqftYv349RNFv6a1evRoDBgxAXh5lGxEEQXQHBEEulOp2d0zKQBwbKJ+vIHROdpzwXlpXXHEFzj//fNxzzz247LLLcODAATz66KM455xzVP3PnDlzsGTJEuTm5qJ3795YvHgxCgsLMWPGDADABRdcgGXLluHOO+/ElVdeic2bN+P111/Hfffdl+jLIQiCIJIEzwuw29PR3Cy3X7BYrB0q1NdZJIlTvU1EeKIZK8YY3G4XmpvrYLenh+3lFQkJN3gGDhyIN998E4899hjOOeccZGRk4Oyzz8bNN9+srjNv3jx4vV4sWrQIbW1tKCsrw6uvvqo2IMvLy8OyZcvw0EMP4bzzzkPPnj1x22234bzzzkv05RAEQRBJJDNT/qGsGD3JgOd5SFJqh/9ShY6Mld2ern7OnSEulZaPRbp7peVUgMYqOmi8IofGKnKO1bGSxcvehB9XEDhkZTnQ0NBKXp526MhYCYIprGcnqZWWCYIgCCLR8DwPnk98ewmTiYfNZoPTKR5TBmIySPZYUeEAgiAIgiC6PGTwEARBEATR5SGDhyAIgiCILg+Jln0wxiBJ8RkKpbIk0T40VtFB4xU5NFaRQ2MVHTRekRPrseJ5LuJSBGTwEARBEATR5aGQFkEQBEEQXR4yeAiCIAiC6PKQwUMQBEEQRJeHDB6CIAiCILo8ZPAQBEEQBNHlIYOHIAiCIIguDxk8BEEQBEF0ecjgIQiCIAiiy0MGD0EQBEEQXR4yeAiCIAiC6PKQwUMQBEEQRJeHDB6CIAiCILo8ZPAQBEEQBNHlIYMnTkiShKeffhpTp05FaWkprrrqKuzfvz/Zp5USVFVVYejQoQH/Pv74YwDAli1bMGfOHJSWlmL69Ol48803k3zGyeGll17C3Llzde+1Nzbd+b4LNl6LFi0KuM+mT5+uLu9O41VfX4+7774b06ZNw7hx4/Db3/4W69atU5evWrUK559/PsaMGYOZM2di+fLluu1dLhfuu+8+lJeXY+zYsbjllltQW1ub6MtICO2N1WWXXRZwX2nvve40VgBw9OhR/OlPf8KkSZMwduxYXH311aioqFCXp8xzixFx4ZlnnmETJ05k//73v9mWLVvY5ZdfzmbMmMFcLleyTy3pfPvtt2zUqFGsqqqKVVdXq/+cTierra1lEydOZLfffjvbuXMn++ijj9ioUaPYRx99lOzTTihvv/02GzZsGJszZ476XiRj013vu2DjxRhjv/71r9nSpUt199nRo0fV5d1pvC677DJ25plnsrVr17Jdu3ax++67j40ePZpVVFSwnTt3slGjRrGlS5eynTt3smXLlrHhw4ez//73v+r2CxcuZKeeeipbu3Yt27RpEzv33HPZ7Nmzk3hF8SPcWDHGWHl5OXv33Xd191VdXZ26fXcaK8YY+81vfsMuvPBCtmnTJrZz50524403silTprDW1taUem6RwRMHXC4XGzt2LHvnnXfU9xoaGtjo0aPZ559/nsQzSw1efvlldtZZZwVd9uKLL7IpU6Ywj8ejvvf444+zGTNmJOr0kkplZSW75pprWGlpKZs5c6ZuAm9vbLrjfRduvCRJYqWlpeyrr74Kum13Gq89e/awIUOGsHXr1qnvSZLETj31VPbkk0+yu+66i/3617/WbTN//nx2+eWXM8bkcR42bBj79ttv1eW7du1iQ4YMYRs2bEjMRSSI9sbqyJEjbMiQIeznn38Oun13GivGGKuvr2fz589n27ZtU9/bsmULGzJkCNu0aVNKPbcopBUHtm7dipaWFpSXl6vvZWZmYvjw4Vi7dm0Szyw12LZtG0pKSoIuW7duHSZMmACTyaS+N2nSJOzZswdHjhxJ1CkmjZ9//hlmsxmfffYZxowZo1vW3th0x/su3Hjt27cPra2tGDhwYNBtu9N45eTk4OWXX8aoUaPU9ziOA8dxaGxsxLp163TjAMj31vr168EYw/r169X3FAYMGICCgoJuN1bbtm0Dx3EYMGBA0O2701gBQFZWFh5//HEMGTIEAFBbW4vXX38dhYWFGDRoUEo9t8jgiQOVlZUAgKKiIt37+fn56rLuzPbt21FbW4vZs2fjhBNOwG9/+1t89913AOSxKyws1K2fn58PADh8+HDCzzXRTJ8+Hc888wz69OkTsKy9semO91248dq+fTsA4K233sL06dNx6qmn4v7770dTUxOA7vU9zczMxIknngiLxaK+989//hN79+7F1KlTQ95bTqcTdXV1qKqqQk5ODqxWa8A63W2stm/fjoyMDNx///2YNm0aZs6ciSeffBJutxsAutVYGbnrrrtQXl6O5cuX46GHHoLD4Uip5xYZPHHA6XQCgO4LAwBWqxUulysZp5QyeL1e7Nq1Cw0NDbjxxhvx8ssvo7S0FFdffTVWrVqFtra2oOMGoNuPXXtjQ/ednu3bt4PneeTn5+PFF1/EwoULsXLlSlx33XWQJKlbj9eGDRtw++23Y8aMGTjppJOC3lvKa7fbDafTGbAc6J5jtX37drhcLowePRrLli3Dtddeiw8//BCLFi0CgG49Vpdeein+9re/4cwzz8T111+Pn3/+OaWeW6b2VyGixWazAZAfFMrfgPzh2u32ZJ1WSmAymfDDDz9AEAR1bEaOHIkdO3bg1Vdfhc1mU38pKSg3vcPhSPj5phLtjQ3dd3quvfZa/O53v0NOTg4AYMiQIejZsycuuugi/Pjjj912vL7++mvceuutGDduHJYsWQJAnlyM95by2m63B733gO45Vvfffz8WLFiArKwsAPJ9ZTabcfPNN+O2227rtmMFAIMGDQIAPPTQQ9i0aRPefvvtlHpukYcnDiiuuerqat371dXVKCgoSMYppRRpaWm6GxsABg8ejKqqKhQWFgYdNwDdfuzaGxu67/TwPK8aOwqDBw8GIIezuuN4vf3227jxxhtx8skn48UXX1R/aRcVFQUdB4fDgYyMDBQWFqK+vj5g4uqOY2UymVRjR0F7X3W3saqtrcXy5cvh9XrV93iex6BBg1BdXZ1Szy0yeOLAsGHDkJ6ejh9++EF9r7GxEb/88gvKysqSeGbJZ8eOHRg3bpxubADgp59+wqBBg1BWVob169dDFEV12erVqzFgwADk5eUl+nRTivbGhu47Pbfddht+//vf69778ccfAci/RLvbeL377rt44IEHMHv2bCxdulQXQhg/fjzWrFmjW3/16tUYN24ceJ7H8ccfD0mSVEEuAOzevRtVVVXdbqzmzp2L22+/Xbf+jz/+CLPZjP79+3e7sTpy5Ajmz5+PVatWqe95PB788ssvKCkpSa3nVkxzvgiVpUuXsgkTJrCvv/5aV1fA7XYn+9SSiiiK7IILLmBnnHEGW7t2Ldu5cyd7+OGH2ciRI9m2bdvYkSNHWFlZGVuwYAHbsWMH+9vf/sZGjRrFPv7442SfesJZsGCBLs06krHpzvedcby+/vprNmTIEPbMM8+wvXv3sm+//ZZNnz6dzZ8/X12nu4zXrl272IgRI9j111+vqx1TXV3NGhsb2fbt29mIESPY4sWL2c6dO9mrr74aUIdn/vz5bPr06Wz16tVqbRlj3aOuQHtj9dZbb7HjjjuOvfvuu2zfvn1s+fLlbOLEiWzp0qXqPrrLWClceeWVbMaMGWzNmjVs27ZtbP78+aysrIwdPHgwpZ5bZPDECa/Xyx577DE2adIkVlpayq666iq2f//+ZJ9WSlBTU8MWLlzIJk+ezEaNGsV+85vfsLVr16rLN23axC666CI2cuRIdvLJJ7O33noriWebPIwTOGPtj013vu+CjdcXX3zBzj33XDZ69Gg2efJk9sgjj7C2tjZ1eXcZrxdeeIENGTIk6L8FCxYwxhhbsWIFO/PMM9nIkSPZzJkz2fLly3X7aGlpYXfeeScbP348Gz9+PJs/fz6rra1NxuXElUjG6u2332a/+tWv1O/hCy+8wERRVPfRXcZKobGxkd1zzz1s8uTJbPTo0ezyyy9n27dvV5enynOLY4yx2PqMCIIgCIIgUgvS8BAEQRAE0eUhg4cgCIIgiC4PGTwEQRAEQXR5yOAhCIIgCKLLQwYPQRAEQRBdHjJ4CIIgCILo8pDBQxAEQRBEl4cMHoIgCIIgujxk8BAE0WU4cOAAhg4dio8//rjT+1q4cCGmT58eg7MiCCIVMCX7BAiCIGJFfn4+3n//ffTt2zfZp0IQRIpBBg9BEF0Gi8WC0tLSZJ8GQRApCIW0CIJIGB9++CFmzZqFkSNH4qSTTsIzzzwDURQByCGkuXPn4qOPPsLJJ5+MsWPH4tJLL8XWrVvV7SVJwhNPPIHp06dj5MiRmD59Oh5//HF4PB4AwUNae/bswbx58zB58mSUlpZi7ty5WL9+ve68GhoacPvtt2PChAkoKyvD4sWLIUlSwPl//fXXOP/88zFq1ChMnjwZDz74IFpbW9XlbW1tuPfeezFt2jSMHDkSM2fOxKuvvhrTMSQIomOQh4cgiITw0ksv4YknnsCcOXNw++23Y8uWLXjmmWdw+PBhPPzwwwCALVu2YNeuXZg/fz6ysrLw9NNPY86cOfjiiy+Qn5+PV155Be+99x4WLFiAPn36YNOmTXjiiSdgNpsxb968gGPu3LkTF110Efr3749FixbBbDbjzTffxKWXXorXXnsNEyZMgCRJuPLKK3Hw4EEsWLAA2dnZWLZsGX788Ufk5+er+/r8889x66234qyzzsJNN92EgwcP4oknnsDOnTvxl7/8BRzH4eGHH8bKlSuxYMEC9OjRA9999x0ee+wxZGdn44ILLkjYWBMEEQgZPARBxJ2mpiY8//zz+M1vfoNFixYBAKZMmYLs7GwsWrQIl112mbreiy++iPHjxwMARo8ejVNPPRVvvvkmbr31VqxZswYjR45UjYcJEybAbrcjIyMj6HGfffZZWCwWvPnmm0hPTwcAnHTSSTjzzDPx2GOP4aOPPsJ3332HzZs345VXXsG0adMAAOXl5TrBMmMMS5YswdSpU7FkyRL1/f79++P3v/89VqxYgZNOOglr1qzB5MmTMWvWLADAxIkT4XA4kJeXF8vhJAiiA1BIiyCIuLNx40a0tbVh+vTp8Hq96j/FqPj+++8BAMXFxaqxA8gi5LFjx2Lt2rUAZAPi+++/x+9+9zssW7YMO3fuxJw5c3DOOecEPe6aNWtw8sknq8YOAJhMJsyaNQs//fQTWlpasG7dOpjNZkydOlVdx+Fw4MQTT1Rf79q1C5WVlQHnX1ZWhvT0dPX8J06ciA8++ABXXXUV3n77bezfvx/XX389TjrppNgMJEEQHYY8PARBxJ36+noAwNVXXx10eXV1NQCgoKAgYFleXh5+/vlnAMCVV16JtLQ0/O1vf8OSJUuwePFiDB48GIsWLcKkSZMCtm1oaECPHj0C3u/RowcYY2hubkZDQwOys7PBcZxunZ49ewac/3333Yf77rsv5PnfeeedKCwsxGeffYYHHngADzzwAMaOHYt7770Xw4YNC3rtBEEkBjJ4CIKIO5mZmQCAJUuWoH///gHLe/Togaeeegp1dXUBy44cOaKGhHiex+zZszF79mwcPXoUK1aswIsvvogbb7xR9bJoycrKwpEjRwLer6mpAQDk5OQgJycHdXV1EEURgiCo6yhGjvb8b7vtNkyYMCHocQA5S+zaa6/Ftddei0OHDuHf//43nn/+edxyyy1Yvnx5qOEhCCIBUEiLIIi4M2bMGJjNZlRVVWHUqFHqP5PJhKVLl+LAgQMA5IyqiooKdbuqqips3LgR5eXlAICLL74YDz74IADZ83P++edj9uzZaGxsRHNzc8Bxy8rK8O9//1u3TBRFLF++HKNGjYLFYkF5eTm8Xi++/vprdR23260zoAYOHIi8vDwcOHBAd/4FBQV4/PHH8csvv6CtrQ2nn346XnvtNQBAr169MHv2bMyaNQuHDh2K4WgSBNERyMNDEETcycnJwZVXXomnnnoKzc3NmDhxIqqqqvDUU0+B4zg13MMYwx/+8AfcfPPNEAQBzz77LLKysjB37lwAsgHz2muvoUePHhg7diyqqqrwl7/8BRMmTEBubq4uRRwAbrjhBnz33Xe45JJLcPXVV8NsNqvammXLlgGQBcpTpkzBokWLcPToUfTu3RtvvvkmamtrVc+SIAi4+eabcffdd0MQBJx88slobGzE888/j6qqKowYMQI2mw0jRozAs88+C7PZjKFDh2L37t345JNPcPrppydwtAmCCAbHGGPJPgmCILoH77zzDt59913s3bsXWVlZKC8vx/z589GrVy8sXLgQa9aswVVXXYXnnnsOTqcTJ5xwAhYsWIDi4mIAgNfrxQsvvIDPPvsMlZWVyMjIwPTp03HLLbcgJycHBw4cwCmnnII///nPOP/88wHIqe5Lly7FunXrwHEcRo8ejRtuuEEnjnY6nViyZAmWL18Ol8uFM844Aw6HA9988w3+9a9/qet98cUXWLZsGXbs2AGHw4Fx48bhpptuwtChQwEAzc3NePLJJ/HNN9+gpqYGeXl5OOOMM/DHP/4RNpstgSNNEIQRMngIgkgJFINHa2AQBEHECtLwEARBEATR5SGDhyAIgiCILg+FtAiCIAiC6PKQh4cgCIIgiC4PGTwEQRAEQXR5yOAhCIIgCKLLQwYPQRAEQRBdHjJ4CIIgCILo8pDBQxAEQRBEl4cMHoIgCIIgujxk8BAEQRAE0eX5/18k8gTGKm4WAAAAAElFTkSuQmCC", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjMAAAHJCAYAAABws7ggAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAAC7AklEQVR4nOydd3wb5f3H36ctD3lvO8tJHLI3hIQQdliBphQokAYKpS0jbeBXAoWywk6AAmGPAmE07BbChrKzEwgh286wHe8lD1nzfn9IJ9uxHS9tPe/Xyy/b0o3n0Z3uPvedkizLMgKBQCAQCARhiirYAxAIBAKBQCAYCELMCAQCgUAgCGuEmBEIBAKBQBDWCDEjEAgEAoEgrBFiRiAQCAQCQVgjxIxAIBAIBIKwRogZgUAgEAgEYY0QMwKBQCAQCMIaIWYEXeKvWoqiRqNAIBAIfI0QM4JOfPHFFyxZssTn2920aRNXXnml9/+SkhIKCgp45513fL4vQWSwdu1aTjvtNMaOHcsVV1zR5TILFiygoKDA+zNq1CgmTZrE/Pnzefnll3E4HD5dvqCggLFjxzJnzhzuuOMOGhoaOo2pvLyc5cuXc9ZZZzFp0iQmTZrEr371K5555hksFkuP8169ejUnnHACY8eO5dZbb+3DJ9Y7Dp9PQUEB48eP58wzz+TZZ5/F5XL5fJ9HGstjjz3m93WCyZdffklBQUGwhxHRaII9AEHo8eKLL/plu2+++SaFhYXe/9PT01m1ahWDBg3yy/4E4c8DDzyAy+XimWeeISUlpdvlRo8ezW233QaA0+mkoaGBb775hnvvvZeNGzfyz3/+E5VK5ZPlAex2O7/88gsPPfQQO3bs4PXXX0eSJADWrVvHokWLSEhI4KKLLqKgoACXy8W6det48skn+fTTT3n11VfR6/XdzufOO+9kyJAh3HfffWRkZPTvw+uB8847j9/85jfe/y0WC59++inLly/HbDZz/fXX+2W/0ca6devEZxkAhJgRBA2dTsfEiRODPQxBCFNfX8+0adM49thjj7hcXFxcp3PpxBNPZNiwYdx999188MEHzJs3z6fLT5s2jebmZh599FF++uknJk6cSG1tLYsXL2bIkCH861//IiYmxrv8zJkzOemkk/jtb3/LSy+91MFK2dW8Z86cydFHH33EeQ+EzMzMTnOaMWMGRUVFvPrqqyxatAitVuu3/Uc6TU1NPPvsszz77LPEx8fT0tIS7CFFNMLNJOjAggULWL9+PevXr6egoIB169YB7ovrrbfeyrHHHsu4ceM4//zzWbNmTYd1v//+e84//3wmTZrEtGnT+POf/+y1xNx44428++67lJaWel1Lh7uZ3nnnHUaPHs1PP/3EBRdcwLhx4zjhhBN4/vnnO+ynsrKSxYsXM336dKZNm8att97Kww8/zIknnnjEuVVWVrJkyRJmzJjBpEmTuOSSS9iyZQvQvcvrxhtv7LDdBQsW8H//938sWrSIiRMnctlll3HaaaexaNGiTvs755xz+POf/+z9//PPP2f+/PmMGzeOmTNnctddd/V4gXM6nbz66qucffbZjB8/njlz5rB8+XKsVmuHMV566aW8/fbbXpfMOeecwzfffHPEbQO89957/OpXv2LChAnMmTOHBx98EJvNBsBjjz3GiSeeyP/+9z/mzp3LhAkTOP/8873nBLiPWUFBASUlJR22e+KJJ3LjjTcecd/79+9n0aJFzJw5k4kTJ7JgwQI2bdoEtB2P0tJS3nvvvQ7nYl+45JJLyMjI4N///rdflh87diwAhw4dAuC1116jpqaGu+66q4OQUZgwYQILFy7s8j1wP8Ur7ojHH3+8w2f7/fffc9FFFzFlyhSOPvporr/+esrKyrzrKt+fN998k5kzZzJ9+nT27t3bq3kcPqfm5mav+0yxjJ1yyimMHTuW0047jZUrV3ZYZ8GCBdx8880888wzzJkzh3HjxnHhhReydevWDsutX7+eCy64gAkTJnDaaafxww8/dDn/w4/1ggULWLBgQZfj7e05WFBQwOuvv86NN97IlClTmD59OnfddRetra3cf//9HHPMMRx99NHcfPPNHb5fh7N582YKCgr43//+1+H1HTt2UFBQwGeffQbAW2+9xRtvvMGtt97KJZdc0u32BL5BiBlBB2677TZGjx7N6NGjWbVqFWPGjMFqtbJw4UK++OILFi9ezIoVK8jMzOSKK67wCpri4mKuuuoqxo4dy5NPPsndd9/Nvn37uPLKK3G5XFx11VUcf/zxpKWlsWrVKubMmdPl/l0uF3/9618544wzeOaZZ5g8eTIPPPAA3377LQA2m42FCxeyefNm/v73v3Pvvfeyc+dOXnjhhSPOq7m5md/+9resW7eOv/3tb6xYsQK9Xs/vf/979u/f36fP6KOPPiI2NpYnn3ySK664gnnz5vH111/T1NTkXaawsJCdO3dyzjnnAPD+++9z9dVXM2zYMB5//HGuueYa/vvf/3LVVVcdMSj61ltv5d577+Xkk0/mySef5OKLL+aVV17ptN62bdt4/vnnWbRoEY8//jhqtZprr722y3gOhVdffZUlS5YwZswYVqxYwZVXXsnKlSu56667vMvU1tayZMkSLrroIh555BEMBgOXX345O3bs6NNndjh79+5l/vz5lJSUcMstt7B8+XIkSWLhwoWsX7/e64JMS0vj+OOP956LfUWlUjFjxgy2bt3aKRbGF8vv27cPgLy8PMAdb1ZQUMCIESO6XWfJkiXd3tzGjBnDqlWrALcbaNWqVaSnp/Pee+/x+9//nqysLB566CFuuukmtmzZwgUXXEBNTY13fafTyQsvvMDdd9/NTTfdRH5+fo9z6GpOsbGxXrfe7bffzqOPPsq8efN46qmnmDt3Lvfccw+PP/54h/U++eQTvvjiC2655RYeeughqqurufbaa3E6nQD88ssv/P73vyc+Pp5HH32U3/3ud1x33XV9Ht9AWLZsGTqdjhUrVnDuueeycuVKzj33XMrKyli+fDkLFizgrbfe6iTW2jN58mQGDRrE6tWrO7z+wQcfkJiYyPHHHw+4xdSXX37JhRde6Nc5CdwIN5OgA8OHDycuLg7Aa4J+44032LlzJ2+88QYTJkwAYPbs2SxYsIDly5fz9ttvs3XrVlpbW/njH//o9fFnZmbyxRdf0NLSwqBBg0hOTu7gWurKKiHLMldddZXXlz9lyhQ+++wzvvrqK4477jj++9//UlRUxNtvv+19Kj7mmGM4+eSTjzgvxSr07rvvctRRRwHui9K5557Lhg0bmDFjRq8/I61Wyx133IFOpwNg0KBBPPbYY3z++eece+65gPvCZjKZOPHEE5FlmeXLl3PcccexfPly73aGDBnCpZdeytdff92luNu7dy9vvfUW119/vdclMXPmTNLT07nhhhv45ptvvBfOxsZG3nnnHW/8UUxMDJdccok3gPZwXC4Xjz/+OCeffHIH8WKxWFi9ejV2u937/+233+6dl/JZP/PMMzz88MO9/swOZ8WKFeh0Ol5++WXv+TZnzhzOOussHnjgAd566y0mTpyITqcjOTl5QO7I1NRU7HY79fX1pKam9mt5WZY7iJuGhgbWr1/Pk08+yaRJk7zn4sGDB5k5c2anbXYljDSazpff9u4sxQ3kcrlYvnw5s2bN4sEHH/QuO3nyZM444wyef/55brjhBu/rf/rTn7p9WGiPy+XyjkuWZaqrq3n//ff58ssvueKKK5AkiX379vHGG29w3XXXec/BWbNmIUkSTz/9NBdddBFJSUneOT7//PPe49nc3MySJUvYsWMHY8eO5emnnyYlJYUnn3zS675KSkpi8eLFPY7VVwwfPpw777wTgOnTp/Pmm29it9tZvnw5Go2GWbNm8cknn7B58+YjbmfevHm88MILtLa2YjAYkGWZDz/8kLlz53a4LggCh7DMCHpkzZo1pKWlMWbMGBwOBw6HA6fTyQknnMC2bdtoaGhgwoQJ6PV6zjvvPO6++26+/fZbRo0axeLFi70Xt94yadIk79/KzUwRPmvXriUvL8978wD3DeCEE0444jY3bdpEbm6uV8gAGI1GPvnkkw5BkL1h2LBh3gsWuJ/KJ0+ezIcffuh9bfXq1d4LW1FREeXl5Zx44onez8/hcDBt2jTi4uL4/vvvu9zP+vXrATjzzDM7vH7mmWeiVqs7mOKTk5M7XDwzMzMBus2c2bdvHzU1NZxyyikdXr/88st55513vDcbjUbDWWed5X3fYDAwe/ZsNmzY0P0H1AvWr1/PCSec0OHc0Gg0nHnmmWzbto3m5uYBbb89igVLCdDtz/IbNmxgzJgx3p9jjz2W6667jrFjx/Lggw96l+0qC8jhcHRYV/npLfv27aOqqqrDcQD3zXLSpEne80Sh/Tl+JJ544gnvWJTsrMcff5wLLriAa6+9FnB/32RZ7nTunnjiiVitVq9bEDo+CAHehxrlHNy0aRPHHXdchzicU089FbVa3evPYqC0v7ao1WqSkpIYM2ZMB2GZmJhIY2Mj0Cb42l/3wC1mWlpavK6mzZs3c+jQIa8lVhB4hGVG0CP19fVUVVV1ewGuqqpi+PDhvPLKKzzzzDO89dZbvPzyy5hMJi666CL++te/9vpGAu4bZntUKpX3BlNXV9dlVsuRMl2UOfS0TG+JjY3t9No555zD0qVLqauro6SkhAMHDnDPPfd49w1wxx13cMcdd3Rat7Kyssv9KC6itLS0Dq9rNBqSkpK8F1xwC7P2HOnm2n5MPX0mqampnSwIKSkp3vX7S0NDQ5dWktTUVGRZpqmpqcvPuT9UVFRgMBhITEzs9/JjxozxHjtJktDr9WRlZXUS6jk5OZSWlnZ4TaPR8NZbb3n/f+ONN3jjjTd6PX7ls+7u89q+fXuH17qLxzmc888/n/PPPx9wzyk2Npbc3NwOYkPZ9+GCWqGiosL79+HnoJINppyDDQ0NXiuOgnIuB4quHqyO9Hn9/e9/59133/X+n5OTw5dffsngwYOZNGkSq1ev5vTTT2f16tUMGjSIyZMn+2Xcgp4RYkbQI/Hx8QwZMqSDi6Q9ubm5AIwfP54VK1Zgs9nYtGkTq1at4qmnnmLUqFGcfvrpPhlLRkZGlzEu7eMGuiI+Pr5TgCC4n6gSEhK8Akp58lLobQbC6aefzl133cXnn39OUVEROTk5TJkyBQCTyQTADTfcwPTp0zutm5CQ0OU2lderqqrIycnxvm6326mrqxvQTUAZU21tbYfX6+rq2L59u/cJtivRUl1d7RVB3YmmniwrCQkJVFdXd3q9qqoKwGc3OIfDwbp165g8eXKvLADdLR8bG8u4ceN6XP/EE0/kmWeeobi42BtHA3RY96uvvurTHBRR1d3n1d/PKj09vcc5KefJSy+91KW4zM7O7vX+EhMTO81BluUOcV1HOp+6E7f9PQd7wzXXXMPFF1/s/b+9RXbevHnce++9NDY28vHHH/Pb3/52wPsT9B/hZhJ0on19DXD7lsvKykhJSWHcuHHen++//57nnnsOtVrNiy++yAknnIDNZkOn0zFjxgyWLl0KtGV6HL7d/jB9+nRKSko6BKC2trZ6A4S7Y+rUqRQXF7Nnzx7va1arlWuvvZa33nrL+8TW/knTbrd3ysboDpPJxAknnMAXX3zBJ598wrx587wX2WHDhpGSkkJJSUmHzy8jI4MHH3yw05N1+7kCnQINV69ejdPp9Iql/jBs2DCSkpI6ZWT85z//4corr/TGzBz+2ba2tvLNN994Y4yUz628vNy7TGFhYY+Wm2nTpvG///2vQ9C00+lk9erVjBs3rsNNYyCsWrWKqqqqXt9o+rr84Vx88cUkJiZy4403dpibgtPppKioqE/bHDp0KGlpaXzwwQcdXi8uLubHH3/0qzVg6tSpgFvktj93a2treeSRR/pkoZsxYwbffPNNB9fnt99+6z3XoOvzqaGhoUN9qsPp7znYG3JzczvMu33huzPOOANZlnnkkUeoqanpkMovCDzCMiPohMlkYsuWLaxZs4bRo0czf/58XnnlFS677DL+9Kc/kZWVxQ8//MCzzz7LJZdcglar5ZhjjmH58uVcffXVXHLJJajVav7973+j0+m88Swmk4nq6mq+/vrrXvv1D+ess87imWee4eqrr+Yvf/kLJpOJf/3rX9TU1BzxKXH+/PmsXLmSP//5zyxatIikpCRefvll7HY7F110EQkJCUyaNImVK1cyePBgEhISePnll2ltbe212X7evHksWrQIp9PZwXeuVqtZvHgxt956K2q1mhNOOAGz2cwTTzxBRUVFt+674cOH86tf/YpHH30Ui8XCtGnT2LFjBytWrODoo4/muOOO69uH1w4l2+nOO+8kJSWFE088kX379vHoo49y8cUXd7AW3XTTTfz1r38lJSWF559/npaWFm/K+dFHH43BYOC+++7jL3/5i7fuSk8unWuuuYZvvvmG3/3ud1x55ZVotVpeeeUViouLee655/o8n6amJn788UfA/YReV1fHd999x6pVq5g3bx6nnnrqgJbvLRkZGaxYsYK//OUvzJs3jwsuuIAxY8agUqnYtm0bb7/9Nvv37+/TjU+lUnHddddx0003cf311zNv3jzq6upYsWIFCQkJXHbZZf0aa28oKChg3rx5/OMf/6C0tJSxY8eyb98+Hn74YXJzcxkyZEivt3X11Vfz+eefc/nll3PFFVdQW1vLP//5zw5urYKCArKysnj88ceJi4vzBhof7sJqT3/PwYGiZC699tprTJo0icGDB/t1f4IjI8SMoBMXX3wx27Zt4w9/+AP33nsvZ599Nq+++ioPPvggy5Yto7GxkZycHK6//np+//vfAzBq1CieeuopHn/8ca677jqcTidjx47lhRdeYNiwYYBbUHz99ddcffXVLFq0iDPOOKPPY9NoNDz//PPcfffd3H777Wg0GubNm0diYqI3TbYr4uLieOWVV3jggQdYunQpLpeLiRMn8vLLL3vdAffddx9Lly7llltuIS4ujvPOO48pU6bw5ptv9mpsxx9/PPHx8eTl5TF06NAO7/3mN78hNjaW5557jlWrVhETE8PkyZNZvnx5B3fE4dx9990MHjyYt99+m2effZb09HR+97vfcdVVVw3Y0nXxxRcTExPD888/z6pVq8jMzOQPf/gDf/jDHzosd/vtt3PPPfdQW1vL5MmTef31170XbpPJxGOPPcaDDz7I1VdfTU5ODtdccw3vvffeEfc9YsQIXnvtNW+asSRJjB8/npdfftlrDegL27dv54ILLgDa4j9GjhzJ7bff3mWAd1+X7wtTp07l/fff5/XXX+fjjz/m2WefxWazkZWVxTHHHMPDDz/M6NGj+7TN+fPnExsby9NPP83VV19NXFwcxx13HNddd12nmCpfc++99/L000/z73//m/LyclJSUjjjjDP461//2qfg3SFDhvDKK69w3333sXjxYlJSUliyZAn33Xefdxm1Ws2jjz7KPffcw3XXXUdqaioLFy6kqKio2+93f89BX3DOOefw+eefc/bZZ/t9X4IjI8mi858gjNizZw9FRUWceuqpHYKKzzvvPDIzM1mxYkUQRxdZPPbYY6xYsYJdu3YFeygCgUBwRIRlRhBWtLS08Je//IWLLrqIU045BafTyYcffsi2bdv4v//7v2APTyAQCARBQIgZQVgxYcIE/vnPf/L888/z3nvvIcsyo0eP5rnnnuOYY44J9vAEAoFAEASEm0kgEAgEAkFYI1KzBQKBQCAQhDVCzAgEAoFAIAhrhJgRCAQCgUAQ1ggxIxAIBAKBIKyJmmwmWZZxuXwf66xSSX7ZbigSTXOF6JqvmGvkEk3zFXONPFQqqVeNiqNGzLhcMrW1A2881h6NRkVSUixmcwsOR9fdiSOFaJorRNd8xVwjl2iar5hrZJKcHIta3bOYEW4mgUAgEAgEYY0QMwKBQCAQCMIaIWYEAoFAIBCENULMCAQCgUAgCGuiJgBYIBAIBOGJy+XC6XT0sIxEa6sam82K0xnZWT6RMle1WoNK5RubihAzAoFAIAhJZFnGbK7FYmnq1fLV1SpcrsjO7lGIlLkajXGYTMm9Sr8+EkLMCAQCgSAkUYRMXFwSOp2+xxueWi2FtaWiL4T7XGVZxmaz0tRUB0BCQsqAtifEjEAgEAhCDpfL6RUycXGmXq2j0agivu6KQiTMVafTA9DUVEd8fNKAXE4iAFggEAgEIYfT6QTabniCyEQ5vj3FRPVESIoZl8vFo48+ynHHHcfEiRP5wx/+QHFxcbCHJRAIBIIAM9BYCkFo46vjG5Ji5oknnuC1115j6dKl/Pvf/8blcnHFFVdgs9mCPTSBQCAQCAQhRsiJGZvNxgsvvMCiRYuYM2cOo0aN4uGHH6a8vJxPP/002MMTCAQCgUAQYoRcAPDOnTtpbm5mxowZ3tdMJhOjR49mw4YNnHXWWf3etkbjW+2mVqs6/I5kommuEF3zFXONXMJ5vi5X39wPirdCkkAO3ySfXuHPuV5zzZVkZWVz8823+3bDPaBWSwO6R4ecmCkvLwcgKyurw+vp6ene9/qDSiWRlBQ7oLF1h8lk9Mt2Q5Fomiv4Z76NLTY+/H4fc2cMISEudIIbo+nYRtNcITzn29qqprpa1eebXDgKt/7ij7lKkoQkDUxY9AWXS0KlUpGQEIPBYOj3dkJOzFgsFgB0Ol2H1/V6PQ0NDf3ersslYza3DGhsh6NWqzCZjJjNFpzO8E6R64lomiv4d76vfbabj9cdpKKmmQWnFfh02/0hmo5tNM0Vwnu+NpvVU/lX7lUKsiS55+t0uqLCMuOvucqyjCz37jP3BU6njMvloqGhBYvF2el9k8nYK9EWcmJGUWY2m62DSrNarRiNA3u68NfBcTpdYZ/v31uiaa7gn/nuLXGL8l0H60Lqs4ymYxtNc4XwnG93BeFkWcZm73ouGo3/b8I6rarPGTizZk3lssv+wIcfvo/DYWfFimfJzMzi2Wef5NNPP6K5uYmhQ/O54oo/MX36MRQW7mXhwgt5/vlXKCgYBcBNN/0fmzdv4MMPv0StVuNyuTjzzJO59trrOO20M3j//fd4661/U1xcjEolMXLkKBYtuo5Ro0YDcN55ZzNnzkmsXfs9dXW13HXXA4wZM46nnnqMTz/9GLvdxjnn/Br5MHX02msree+9t6iqqiQ1NY0zz5zHwoWX+zzLrLeitTtCTswo7qXKykoGDRrkfb2yspKCguA/xQoEA8HpcnGwohGAkspmrDYnep06yKMSCMIDWZa595XN7C3tv5V+oAzPTeCmiyf3+Wb+7rtvsnz5ozgcTvLyBnH77Tdz4MA+br11KWlp6Xz//TfccMNfueee5Rx77CyysrLZsGEtBQWjcDqdbNmykZaWFnbv3slRR41h+/ZtNDY2MmPGLL7++n88/PADLFlyCxMmTKK6upp//nMZ9913Fy+++Jp3DO+88wb33/8w8fHxDBs2nH/+cxnff/8tN998GxkZWbz88gv89NMWsrNzAPjuu29YufJf3HnnPeTlDeGXX7Zy1123kZWVzWmnneHTz3WghJyYGTVqFHFxcaxbt84rZsxmM9u3b+eSSy4J8ugEgoFRVt2CzfP04ZJl9pWZGTU4KcijEgjCiDAtO3PaaWd4rSQlJcV8/vkn/OtfrzJihPsh/cILL2Hv3j289trLHHvsLGbOPI4NG9ZxySWXsmPHL2g0WsaOHcfmzRs56qgxfP/9d0yYMAmTyURCQgI33vgPTj31dAAyM7M466x5PPTQAx3GcMwxM5k27WgAWlqa+eijD7j++iXMmDELgJtuupXNmzd6lz90qASdTktmZjaZmZlkZmaSmppORkam3z+vvhJyYkan03HJJZewfPlykpOTycnJYdmyZWRmZnLqqacGe3gCwYDYV27u8H/hoQYhZgSCXiJJEjddPPkIbib/l/jvj5sJIDe3zdOwe/cuAK666ooOyzgcDuLi4gGYOfM4/vvfd7FaW9mwYR1TpkwlMzObTZs2cvHFC/n++++YO/dMACZOnMz+/ft48cXnOHBgPyUlByks3NupEWVubp7374MHD2C32xk1aoz3Nb1ez8iRbR6QU089g9Wr/8tvfzufIUOGMW3a0cyZcxKZmULM9IpFixbhcDi45ZZbaG1tZdq0aTz//PNotdpgD00gGBD7y90uJqNejcXqpLDU3MMaAoGgPZIkdeua1WhUqFWhabrR69syF2XZLTIef/xZYmI6Ztkq/YkmTZqKVqtly5bNbNy4ntNOO4OsrCzeeecNysvL2LNnF3ff7ba8fPrpx9x9922ceurpjB07nnPOmU9RUSEPPXR/t2NQTFzKWBQ0mjZZkJiYyL/+9Rrbtm1lw4Z1rFu3hjfffJ3LL/8jl132h4F9ID4mJHPY1Go1f/vb31izZg1btmzhmWeeITc3N9jDEggGzP4yt5iZNS4bcFtmDg+4EwgEkc3QofkA1NRUk5ub5/1Zvfq/fPjh+4BbVEyfPoPvvvua7du3MWXKNMaPn4jT6eT5558mP384WVnu68irr77I2Wefy803386vf30+EydOprS0BKDb68ugQYPR6fRs3fqT9zWHw8GePbu9/3/66Ue8++5bjB8/kcsv/yPPPOPezxdfhF4B25AUMwJBJOJwuiiubALg+InZaNQSjS12quotQR6ZQCAIJMOG5XPsscexbNm9fPfdN5SWlvDqqy/xyisvkpPT9uA+a9ZsPvzwfVJT08jJyUWvNzB27Hg++eRDZs+e410uPT2Dn3/+iV27dlJaWsKqVa/yzjtvAHTbBigmJobzzjufF154mq+//pIDB/azfPm9VFdXeZex2aw8/vgjfPzxasrKDvHTTz+yZctmxo4d758PZgCEpJtJIIhESquacThdxBo0ZKXEMDgjnsJDZgpLzaQnxQR7eAKBIIDceee9PPPM4yxbdg+NjWays3O58cZ/cPrpbVXuZ8yYidPpZPLkqd7Xpk6dzubNG5k9+3jva4sX38ADD9zNNddciU6nZfjwkdxyyx3cdtvf2blzOxMmTOpyDH/84zXodHoeeuh+WlpaOPHEU5g5c7b3/bPOOpeGhgZefPE5KisriI+PZ86ck/jznxf54RMZGJIcJTZup9NFbW2zT7ep0ahISoqlrq457Go49JVomiv4Z75f/VjKyx/vYsyQJK6/cBKvf76HzzYWc+LkHC45NXhlB6Lp2EbTXCG852u326ipKSMlJQutVtfzCgQmADhUiJS59nSck5Nje1U0T7iZBIIAocTLDMkyAZCf4/4tgoAFAoFgYAgxIxAEiP2etOwhme7Uy+E5CQAUVzZhtXUu4y0QCASC3iHEjEAQAOwOJ6VVbjfnkEy3RSbZZCApXo9Llr1CRyAQCAR9R4gZgSAAFFc243TJxMdoSTa11XoYlu1xNR0SYkYgEAj6ixAzAkEAaHMxmTpUD83PdruaCoPYa0YgEAjCHSFmBIIA4A3+9cTLKChxM4WlonieQCAQ9BchZgSCAOC1zGR1FDODM+NQqyTMLXaqGlqDMTSBQCAIe4SYEQj8jNXupLS6Y/CvglajZlCGW+AUCVeTQCAQ9AshZgQCP1Nc0YQsQ0KcjqR4faf3Rb0ZgUAgGBhCzAgEfmafx8U09DCrjIISN7P3kLDMCAQCQX8QYkYg8DPdBf8qKBlNJZVNWO2ieJ5AIPAvDoeDVate9f7//PNPc955Z/t8P/7ablcIMSMQ+Jnugn8Vkk16EuJ0OF0yB8obAzk0gUAQhXz22cc89tjDwR6GTxFiRiDwIxarg/KaFqBz8K+CJEkMF/VmBAJBgIjEMhCaYA9AIIhkDlY0IgMpJj2m2O47/+bnJLBpdxV7hZgRCI6ILMvgsHXzngrZ352kNboOhS97w5o13/Pcc0+xf38RRmMMM2bM5Nprr2Pv3t0sXnw1d955H0899RgVFRWMHTuOm2++nddfX8nHH69Go9Hym99cyMKFl3u399FHH7Bq1ascPHiQ5ORkzjrrHBYsuAy1Wg1ARUU5Tz/9OBs3rqelpZnx4ydy1VV/YfjwEXz44fvcc88dAMyaNZVHH33Ku91XXnmRt99+g4aGBsaMGcsNN9xMXt4gAJqamnj88Uf49tv/YbfbKSg4iquuWsSoUaO96//nP+/w2msvU1VVxbRp08nKyu73x9xXhJgRCPzI/nIlXqZrq4yCN6PpkBlZlvt8sRQIogFZlmn57924KvYGbQzqjBEY5/2919/R+vp6br75b1xzzWKOPXYWlZUVLF16G0888Qinnno6TqeTl19+gdtuuwuHw8Hf/vZXLr30Is466xyeeeYlPv30I5599klmzTqe/PzhvPHGazz11AoWLbqOKVOms337Nh566H4aGhr4y1+up6WlmT//+XKys3O4774H0Wp1vPDCM1xzzR948cXXOemkU2hqauLRRx/kP//5GJMpgS1bNlFeXsbPP//EsmWPYLfbWLr0Vu67bymPP/4ssizzt78tQqczcP/9/yQuLo6PP17Nn/98OU8//S9GjhzFZ599zEMP3c9f/vJ/TJ06nW+++R/PPPME6ekZfj4iboSbSSDwI/vKjhwvozA4I95dPK/ZRo0onicQdItEeAn9qqoKbDYbGRmZZGZmMX78RO6//yF+/esLvMtcccWfGDVqNGPHjmfKlGkYjUauumoRgwYNZsGCSwEoKtqLLMu88spLzJ9/Pueddz55eYM47bQzuPzyP/Huu2/S1NTEJ598RENDPUuX3s/o0WMZMWIkt99+F3q9gXfeeQO93kBcXBwAKSmpaLVaADQaDbfeupThw0dw1FFjOOec+ezcuR2ATZs2sG3bzyxdei9jxoxl8OAh/PGPVzNmzDjefPPfALz11ipOPvlU5s//DYMGDeaSSy5l5szjAvY5C8uMQOBHemuZ0WnVDMqIY19ZI3sPNZCaaAzE8ASCsEKSJIzz/t6tm0mjUeEIMTfTiBEFnHzyaSxZspiUlFSmTTuaY489jtmz57B1648A5ObmeZc3Go1kZWV796HXGwCw2+3U19dRW1vD+PETO+xj0qTJOBwODhzYT2HhXvLyBpOUlOR9X683MHr0GAoLC7sdZ3JyCrGxcd7/4+NNWK1WAHbv3oksy/z612d1WMdms3mXKSray8knn9bh/bFjx7Nnz+7efEwDRogZgcBPNLfaqayzADC4m7Ts9uRnJ7CvrJHCUjPHjM709/AEgrBEkiTQdi4+CSBpVEiSn8VMP7j99rv5/e//wNq1P7BhwzqWLv0H48dP9MbBaDQdb8XdiaXuAnddLrnddrpbxoVGo+52jCpV944al8tFbGwszz//Sqf3FMsOSMhyx8/+8Hn5E+FmEgj8hJJmnZZoIM6o7WFpdxAwiIwmgSCS+OWXbTz66IMMGjSE88+/iGXLHuGmm25l06YN1NXV9WlbyckpJCeneC06Cj/9tAWtVktOTi75+SMoLj5AXV2t932r1crOnTsYMmQY0L1Y6o5hw4bT3NyM3W4nNzfP+/Pqqy/x3XdfAzBixEi2bv2pw3o7d+7o034GghAzAoGf6K2LSSE/271ccWUTNlE8TyCICGJjY3nnnTd54olHKSkppqhoL1988Sm5uYNITEzs8/Z++9sFvPPOG7z99puUlBTz6acf88ILzzBv3q+Ii4vjlFPmkpCQyD/+cSM7dvzC3r17uPPOW7BYLJxzznzA7coCt9iwWnuO0Tv66BmMGDGS2267ic2bN1JSUsxjjz3Ehx++7xVIl1xyKd988z9ee+1liosP8tZb/+arr77o8/z6ixAzAoGf2N/L4F+FlAQDCbHu4nn7RfE8gSAiGDJkKHffvYzNmzdy2WUX8ec/X45KpebBBx/tV9bib397CVdf/Rf+/e9XueSS3/Dcc09y8cULWbToegDi4uJ47LGniY838Ze/XMVVV12B1WrlySefJzs7B4DJk6cxevRY/vzn3/P999/1uE+1Ws3DDz/BqFGjufXWG1m48EJ+/HELd9+9jClTpgFw7LGzuO22u1i9+r8sXHghX3/9Py688JI+z6+/SHIkVs/pAqfTRW1ts0+3qdGoSEqKpa6u2f9BZ0EmmuYKvpnvDU/+QHVDK3/77SSOGpzU8wrAind+ZvPuKn5zQj6nHz24X/vtK9F0bKNprhDe87XbbdTUlJGSkoVW232NpvYEJAA4RIiUufZ0nJOTY1Gre7a7CMuMQOAHGltsVHtSrAdn9M4yA6KDtkAgEPQHIWYEAj+gBP9mJMcQY+h9RH9+u7YGUWI0FQgEggEjxIxA4Af2ecTM0F6kZLdnSKa7eF5Ds40asyieJxAIBL1BiBmBwA94g3/7KGZ0WjV56e7CVcLVJBAIBL1DiBmBwA9407KzepeW3R5Rb0YgaEO4WyMbXx1fIWYEAh/T0GSlrtGKBAzKiOtx+cNR6s0UHhJiRhC9KB2gbTZrkEci8CfK8VWrB1YtWLQzEAh8jGKVyUqNxaDr+1dMscwcrHAXz9Npuy9BLhBEKiqVGqMxjqYmd5VcnU7fY10Wl0vC6YwOS064z1WWZWw2K01NdRiNcUdsp9AbhJgRCHxMW+XfvsXLKKQmGDDF6jA32zhQ0ciI3EQfjk4gCB9MpmQAr6DpCZVKhcsV/rVXekOkzNVojPMe54EgxIxA4GOU4N+h/YiXAXfflPxsE1v2VFNYahZiRhC1SJJEQkIK8fFJOJ2OIy6rVkskJMTQ0NAS1haL3hApc1WrNQO2yCgIMSMQ+BBZlr1p2f21zIDb1eQWMyJuRiBQqVSoVEeuAqzRqDAYDFgszoiojHskommuvSXgAcCbNm2ioKCg08+6deu8y6xZs4b58+czYcIE5s6dy+rVqwM9TIGgX9Q1WjE321BJkjfFuj8oQcB7D4nieQKBQNATAbfM7Nq1i0GDBvHaa691eD0hwZOOWljIH//4Ry677DKWLVvGV199xQ033EBycjIzZswI9HAFgj6hxMvkpMUOKHB3SJbJXTyvyUat2UpKgsFXQxQIBIKII+BiZvfu3QwfPpy0tLQu33/ppZcoKChg8eLFAOTn57N9+3aee+45IWYEIc/+8v4VyzscvVZNbnocB8obKTzUIMSMQCAQHIGAu5l27dpFfn5+t+9v3Lixk2g55phj2LRpkzC3C0Ke/WX9L5Z3OF5Xk4ibEQgEgiMScMvMnj17SEpKYv78+VRUVDBy5EgWL17M+PHjASgvLyczM7PDOunp6VgsFurq6khO7n8Kl0bjW+2mtCXvTXvycCea5gr9m68sy143U35OwoDPt5F5iXy5uZR9ZWafn7vtiaZjG01zheiar5hrdONTMVNSUsJJJ53U7ftfffUVjY2NtLS0cMstt6BWq3nllVe45JJLeOeddxg+fDitra3odB2j1pX/bTZbv8emUkkkJcX2e/0jYTIZ/bLdUCSa5gp9m29FbQtNFjsatcT4gnS0moEVu5syJgv+8wsHyhuJjTP4vXheNB3baJorRNd8xVyjE5+KmYyMDD788MNu309PT2fDhg0YjUa0Wi0A48aNY/v27axcuZI77rgDvV7fSbQo/xuN/T9wLpeM2dzS7/W7Qq1WYTIZMZstOJ2RnR4XTXOF/s33x50VAOSmx9HUOPCO1zpJ9hbP+3FHOSPyEge8za6IpmMbTXOF6JqvmGtkYjIZe2WB8qmY0Wq1R4yHATCZOsYSqFQq8vPzqahw3wiysrKorKzssExlZSUxMTHExw8sqNJf+fhOpytqcv2jaa7Qt/kqNWGGZMT77DMalmXix73V7DpY3+8ifL0lmo5tNM0Vomu+Yq7RSUAdbt988w2TJk2iuLjY+5rD4WDnzp0MHz4cgKlTp7J+/foO661du5bJkyf7rFKgQOAPfBn8q5Cf495WkWg6KRAIBN0SUHUwefJkkpKSWLJkCdu2bWPXrl0sWbKE+vp6Lr30UgAWLFjA1q1bWb58OYWFhbzwwgt8/PHHXHHFFYEcqkDQJ9oH/w40Lbs9wz1NJwsPmX22TYFAIIg0Aipm4uLiePHFF0lNTeXyyy/nggsuoL6+nldeeYXU1FQARowYwRNPPMHXX3/Nueeey5tvvsmyZctEjRlBSFNZb8FidaBRq8hO9V2g+ZBMEypJoq7RSq154HE4AoFAEIkEPDV70KBBPProo0dcZvbs2cyePTtAIxIIBo7iYhqUEYfGh+mSep2a3PRYDlY0sbe0gekmUTxPIBAIDkcEoQgEPsBXlX+7It/jaioSriaBQCDoEiFmBAIf4A3+zfR9xtHwbE/cjKgELBAIBF0ixIxAMEBcLpn9FUomkz8sM26BdKCiEbtIwxQIBIJOCDEjEAyQ8toWrDYnOq2K7BTfV5lOSzQSZ9TicMoc8IgmgUAgELQhxIxAMECUeJnBGfGoVJLPty9JkjdFu0i4mgQCgaATQswIBAPEn/EyCoqraa8IAhYIBIJOCDEjEAwQb7E8P8TLKOSLIGCBQCDoFiFmBIIB4HS5OFjh+8q/hzM0y4QkIYrnCQQCQRcIMSMQDICy6hZsDhcGnZqM5Bi/7UevU5OXFgeIejMCgUBwOELMCAQDYF+7YnkqyffBv+1RiuftFa4mgUAg6IAQMwLBAGhrLum/4F8FJQi4UHTQFggEgg4IMSMQDABvJpMfg38VFMvMgXJRPE8gEAjaI8SMQNBPHE4XxZVNgH+DfxXS2xXPO1gpiucJBAKBghAzAkE/Ka1qxuF0EaPXkJZo9Pv+JEkiP9vjaioVQcACgUCgIMSMQNBPvJ2ys+KR/Bz8q6C4mkS9GYFAIGhDiBmBoJ8EMvhXwStmRBCwQCAQeBFiRiDoJ21tDPwfL6MwNCseSYJas5W6RmvA9isQCAShjBAzAkE/sDuclFR5gn8DkMmkYNBpyPUUzxOuJoFAIHAjxIxA0A+KK5txumTijFpSTIaA7lu4mgQCgaAjQswIBP0gGMG/CiKjSSAQCDoixIxA0A+UeJmhAQz+VVAsM/vLG3E4RfE8gUAgEGJGIOgH7S0zgSYjSSme5+JgRVPA9y8QCAShhhAzAkEfsdqdlFY3A4FNy1aQJIlhXleTiJsRCAQCIWYEgj5SXNGELENCnI6keH1QxiCCgAUCgaANIWYEgj6yz+NiCka8jIIIAhYIBII2hJgRCPpIMIrlHc7QLBOSBDXmVuqbRPE8gUAQ3QgxIxD0kWAG/yoY9RpyUkXxPIFAIAAhZgSCPmGxOiivaQFgcBDdTADDc4SrSSAQCECIGYGgTxysaEQGkk16EmJ1QR3LsGwRBCwQCAQgxIxA0CeC0Sm7O/I9lhlRPE8gEEQ7QswIBH2gTcwEL15GITM5hliDBrvDRXGlKJ4nEAiiFyFmBII+sL8s+MG/CpIkeevN7BVBwAKBIIoRYkYg6CUtrXYq6ixAaLiZAG8l4KJDIghYIBBEL0LMCAS95IDHxZSaYCDOqA3yaNx4KwELy4xAIIhihJgRCHrJPiVeJis0rDIAw7JMSEB1QysNonieQCCIUvwqZm699VZuvPHGTq+vWbOG+fPnM2HCBObOncvq1as7vG+1WrnjjjuYMWMGkyZN4vrrr6e2ttafQxUIekSJlxkaAsG/Cka9hpy0WAD2inozAoEgSvGLmHG5XDz00EOsWrWq03uFhYX88Y9/5LjjjuOdd97hN7/5DTfccANr1qzxLnP77bfz3Xff8dhjj/HSSy9RVFTEokWL/DFUgaDXhFImU3uUejNFot6MQCCIUjS+3mBhYSE333wzBw4cIDs7u9P7L730EgUFBSxevBiA/Px8tm/fznPPPceMGTOoqKjgvffe46mnnmLq1KkAPPTQQ8ydO5ctW7YwadIkXw9ZIOiRxhYb1Q2tQPAr/x5Ofo6Jb346JOJmBAJB1OJzy8zatWvJz8/ngw8+IDc3t9P7GzduZMaMGR1eO+aYY9i0aROyLLNp0ybvawpDhw4lIyODDRs2+Hq4AkGvUIJ/M5JjiDH4/BlgQAz3BAGL4nkCgSBa8flV+eKLLz7i++Xl5WRmZnZ4LT09HYvFQl1dHRUVFSQlJaHX6zstU15ePqCxaTS+1W5qtarD70gmmuYKned70FOUbliWyefn0UDJSY8j1qChudXBoZoWb7p2b4mmYxtNc4Xomq+Ya3TTJzFTUlLCSSed1O37a9asITk5+YjbaG1tRafr2NNG+d9ms2GxWDq9D6DX67Fa+5+toVJJJCXF9nv9I2EyGf2y3VAkmuYKbfMtqW4GYHR+qt/Oo4FQMCSZzTsrKauzMGVMVr+2EU3HNprmCtE1XzHX6KRPYiYjI4MPP/yw2/cTEhJ63IZer8dms3V4TfnfaDRiMBg6vQ/uDCejsf8HzuWSMZtb+r1+V6jVKkwmI2azBWeEm/ejaa7Qeb67D9QBkJmop66uOcij68yQ9Dg276xk6+4qZo7J6NO60XRso2muEF3zFXONTEwmY68sUH0SM1qtlvz8/H4PCiArK4vKysoOr1VWVhITE0N8fDyZmZnU19djs9k6WGgqKyvJyOjbRfpwHA7/HHSn0+W3bYca0TRXcM+3pt5CbaMVCchJjQ3J+Q/11L7ZU1Lf7/FF07GNprlCdM1XzDU6CbjDberUqaxfv77Da2vXrmXy5MmoVCqmTJmCy+XyBgID7Nu3j4qKCqZNmxbo4QoE3pTsrNRYDLrQCv5VGNq+eF5zZ8umQCAQRDIBFzMLFixg69atLF++nMLCQl544QU+/vhjrrjiCsDtyjrzzDO55ZZbWLduHVu3buW6665j+vTpTJw4MdDDFQhCtr5Me2IMGrJT3bE8IkVbIBBEGwEXMyNGjOCJJ57g66+/5txzz+XNN99k2bJlHdK1ly5dyowZM7jmmmu4/PLLGTZsGI8++mighyoQAO06ZYewmAF3vRmAQlE8TyAQRBl+tZmvXLmyy9dnz57N7Nmzu10vJiaGu+66i7vuustfQxMIeoUsy22WmRDqydQV+dkJfPNTGYWirYFAIIgyRJK6QHAE6hqtNDTbUEkSeelxwR7OEVE6aO8vM4vieQKBIKoQYkYgOAL7PC6m7NRY9Fp1kEdzZDJTYojRa7A5XJRUNQV7OAKBQBAwhJgRCI7AvjLFxRTa8TIAKknyVv8VriaBQBBNCDEjEByBfYfcomBoiAf/KiiuJhEELBAIogkhZgSCbpBl2etmCvXgXwVvRpNIzxYIBFGEEDMCQTdU1llosthRqyRy00I7+FdhmEd0VdW3YhbF8wQCn2FuttHcag/2MATdEJrlTAWCEGBvcT0AuWlxaEOsU3Z3xBi0ZKXEUFbTwoGKRsYNSwn2kASCsKeu0cpNT6/BJctMHZXOCZNyGJ6TgCRJwR6awIMQMwNgy/uriKv4kZjYGOLjY5G0eiSNDjTtfmuV//Wg0R3hfZ37fSk8bprRwJ5id3PJcAj+bU9aopGymhbqGvvfZT5YWKwOXv1sN80WO3qdGr3W86NTo/P8bdCp0WlVbe+1W0b5W6tVoRI3GoGPOFDRiM3TA2ntLxWs/aWC3LRYTpiUwzFjMjHqxa002IgjMABiStaTKleBDZx1PtqoWttB+Kgy8tGNm4s6Jc9HOxD0lr0l9UBbE8dwISleDxCWYubHPdX8sK3cJ9vqIHh0bUIoPlaPWgKDTo1RryFGr8HY7idGr8GgV3tf1+vUQhhFOfWe79KQzHhy0+NYv72CkqpmVn66mze+KmTG6AzmTMphUEZ4PfhEEkLMDID0827ilw0b+GV3GS57Kzqc6FQO8pJ0DEnTkxIrgcMGDhuyw3rYbxvYre7fznaxDU47stMOVpABV0M5jt3fo84di27CGaizjxKmzQAgy7LXzRTqbQwOJylOETOtQR5J36n1jHl4bgJTC9Kx2p3Y7E6sNidWu+fH+7fL/V67123tOgjb7C5sdheNDCzOQQIMeg0xenUn0WM8TPgY9RpyUmPFTa0Lymtb2LSrkpOn5KHXhXbNpsNRHgyGZJn43WkFXHDicH74uZyvfiylrKaFr348xFc/HiI/x8QJk3KYNiodrSa85hjuCDEzAJLSUjn74vM4uqqR9dsr+PrHUnYerAcLcAhSTAZmT8zmuPFZJHpuMF0hyy5w2D1CxyNwHDbk1kbsu77DsW8DzpJtWEq2oUoZjG7CXDTDpiGpxOHzF5V1FppbHWjVKm8Dx3ChzTITfgHAtZ6bxqhBSZw6re/WSJcsewSOyy2EbE5aPWLHZnNid8moNWqq65ppttixWJ1YrA7vT4v3b/frTpeMDN73oWdrlyTB3X84hszkmD6PP5J555siNu6sJD5Gx+wJ2cEeTp+oa3If96Q4HQCxBi2nTMvj5Km57DpYz/+2lLJ5dxWFpWYKS828/vkeZo3PYs6kHDKSxHkQCMTd0AdoNSqOHp3B0aMzKKtp5usfD/H9z2XUmFt595si/vvdPiaOSGXOxByOGpLUyWQtSSrQ6pG0nQWPJm88LnMltp8/xb7rG1w1B2j98mmk9W+hG3cq2oLZSDpjoKYaNSgp2YMy49CowyuOKZzdTHVmz00jvnvxfyRUkoRBp8Gg6/p9jUZFUlIsdXXNOBxHbvkgyzI2h6ud2HF2IXo6CqC9pQ2Ym23sPFgnxMxhVNVbOvwOJxQ3U+Jh56UkSYwanMSowUk0NFn5ZmsZ3/xYSo3Zyifri/lkfTFjhiQxZ1IuE0ekoFaF17UknBBixsdkpcRy4UkjmD97GBt3VfLVj4fYW9LApl1VbNpVRXqikeMnZjNzfBammG6uuIehMqVjmHkJ+innYtv+JfZfPkduqsG65nWsm/6DbvQJaMeegiom0b+TiyIUMRNu8TLQdsENRzeT9wm4n2LGl0iS5I25OZJltT1vf13I6jUHKDpkZs7EHD+PMLxQBEFYiuxenJcJcXrOPnYIZx4zmK1FNXy1pZSfC2v4ZX8dv+yvIzHObZE6fmJOSJzfvmTNtnKsDmdQz3khZvyETqvm2LFZHDs2i5KqJr7ecogffimjst7Cm18V8u63RUwemcYJk3IYmZfYqzgYyRCHfvI8dOPnYt/zA7atHyM3lGP7cTW2rZ+gHTED7fi5qJPERXSgeCv/hqGYSfZcKJtbHdjsTnQh3lOqPcqNLqmX4iHUUNpJFB0S7STa43C6vHWPwlHM1PfhvFSpJCYOT2Xi8FSq6i1889Mhvv3pEPVNNv77/X4++OEAE0ekcsKkri314UZji43nVm9HQmL2hOygzUeImQCQmxbHxaeO5Lw5+azfUcFXP5ayr6yR9TsqWb+jkqyUGI6fmMOxYzOJM2p73J6k0aE7ag7aUbNxHPgR+08f4azYg33Xt9h3fYt60AR0409HnVUggoX7gUuW2V/u7skUjmLGqNeg06qw2V3UNVnDxmfvcLpo9NzwkkzhKmbc7STKqpuxWB0iZdeDudmG7Pk73MSMze6kudUB9N1imJZo5NfH53POrKFs2lXF/7aUsru4ns27q9i8u4r0JCNzJuYwa3xWr679ociekgZkGTJTjEEVZuKbFkD0OjXHTcjmuAnZHChv5KsfS1n7SwVlNS38+4s9vP11IdNGpTNnYg75OaYehYgkqdAOmYx2yGScFXux/fQRjv2bcR78CcvBn1ClDUU34XQ0Q6YgqcLn6TzYVNS20GpzotepyUqNQT5yaEXIIUkSSXF6Kuos1DeGj5ipb7IiAxq1RHyYXtgTYnWkmAzUmFvZV2Zm9JDkYA8pJGgvYOoarciyHDYPWoqLSadV9VucatRtcZWlVU18pVjq6yy88b+9vPNNEdNGpfPr44eRbDL4cvh+Z7cn63NkXmJQxyHETJAYnBnPwrmjOP+E4azdXsHXW0o5WNnED9vK+WFbOblpsV5rTW++QOqM4RhPvRZXfTm2nz/Bvvs7XFX7aP38CaT4NHTjTkNbcFyXQcaCjpRWNQMwJNOEWqXC4QozNYP7CbKizhJWT8H1nuyrxDh92NzoumJYtokacytFh4SYUWh/HlrtTixWJzGG8Lj9tHcx+eK8zGlnqV+3o4L/bS7lQEUja34pR62S+P2ZRw14H4Fkj6ce18jcxKCOQ4RWBxmjXsMJk3K47bJp3PK7qcwal4VOo6KkqplXP9vNQ2/82KftqRIzMRy3kNiLHkQ3+RwkfRxyYxXWH16h6bXrsG58B5dF+POPREVdCwBZaeGVkt2ecMxoUmrMJId5cKSIm+mMYt3o7v9QxhvH5ePzUq9TM3tCNrdeOpU/nDUagB0HfFV9NTC02hwcKG8ChGVG4EGSJIZlmxiWbeLCk4bz/bZyXv98D4WlZhpbbMT3MvNJQWU0oZ/6K3QTz8C+6zt3sHBjFbbN/8X200doR85EN24uqsRMP80ofFFSR7NSwlnMuE3V4SRm6rpJfw03vGKmzBxW7hR/Un/YeVjX2EpOmNRvUoSXv85LSZKYNDIVlSRRY26lusFCakJ4lNsoPGTGJcukmPSkJATXPSYsMyFIjEHLKVPzyEpxxzrsKWno97YkjR7dmJOIveB+DCdfjSptGDjt2Hd8RfMbN2H59DGclUW+GnpEUFnnETNhcrHtCq9lJgyfgJPjwytm4HAGZ8SjVkmYm23UmMMvPd4fdLLMhKHI9meGnUGn8faA23Ww3m/78TV7PPEyI4JslQEhZkIaxWynBFgNBEmlQjtsGjHn/gPj2TehHjQRkHHs30TLe3fS8sH9OEp+QZblnjYV8VTWh7+YSYwLPzdTpFhmdFo1uWlxgHA1KSiWGb2nTEA4nZfdFczzNQWe6/0uH1zvA4U3+DfI8TIgxExIo5wgSoCVL5AkCU1WATFz/0rMb+5GM3ImSGqch3Zg+XAZLe/egb1oA3IYBr36ApvdSa2nCm04u5mSTWEoZpoUy0x4ixmAYTkibqY9de0aNUJnt1Mo09bKwM9iZlAiALvDxDLjcLoo9JzfwjIjOCIj8tw1Kw6UN9Fqc/h8++qkHIxz/kDshfejHXsKqHW4qvfT+vnjNL/5d2w7v3Y3vYwiqhrcbgGjXo0ptm9xSqGEYplpaLLhcoWHtU1pZRDulhmAYVltcTPRjizLXkGgxBOFk8iu91MA8OGMyE1EktyW4XD4fPaXN2J3uIgzaslOCX75ByFmQpjUBCPJJj0uWfbrE54qPhXDsRcTe9FydJPngT4WuaEc6zf/ovnfN7iDh23h10+lP1R54mXSk2LCOnAzIVaHSpJwyTINzaHfcNIly9RHkmXGc9M+UN6IwxmdVk4Fi9WBze7+DJQilOFwswblvPQUcvTzeWnUaxicocTNhH5WkzdeJjchJK6VQsyEOIqryRdxMz3hzoCaT9xvl6M/5gKkmETk5jqsa/9Nw8rrqP369YhP6670pGVnJIVHNkF3qFQSCZ4Ov/VhEATc2GLH6ZKRJMLaIqaQkRxDjF6D3eGipKop2MMJKopwiTVoSPd8r8IlML2x2eY+LwnMeam4mnaGgaspVIrlKQgxE+IovsiBZDT1FUlnRDf+dGJ/uwz97MuQEjKQrc3Uf/cWDSuvo/WHV3E11QRsPIFECf4Nl6q5XSHLLmSXK6xqzShNMU2xurDrUt4VKkliqKg3A3RMbVaq2za22LH30LU8FFDGHqjzsiAvCQj9IGCXLLO31H1PChUxI+rMhDgjc91xM4WHGnA4XQG90EtqLbpRx6MdeRxy8WbsP32ErbwQ+7bPsP/yJZoRx6CbcEZENbas9LqZQs8yIzvtyC0NyC31uCwNnr89/7c0IFvcf8stZiR9DOO1J1BEYpiImchxMSkMyzLxy75aig6ZOXFysEcTPNqnNscaNGjUKhxOF/VNVtISQ+971p5AZ9iNzEtAwt1Spb7J2utu7YHmUFUzza0O9Fo1gzLigj0cQIiZkCcrNZZYg4bmVgcHKhrJ9zSyCySSSoU2fzrpU+ZQ9fN6LJvex3loB47d3+PY/T2awZPQTTwTdcbwgI/N1yhiJiM5MJYZWZbB1uIWIy31XkHiaidUZEsDrpYGsDb3frutjcxp/S8NhinUmQf5cQa+wXvTCNGLd38QlYDdtE9tliSJ5Hi9N8g11MVMX7pl+4IYg5a89DgOVjaxu7ie6UdlBGS/fUWxHOXnuFu+hAJCzIQ4KkliRG4iP+6tZk9xQ1DEjIIkSWjzxiJljcZZWYTtx9U49m/GcWALjgNbUGeNcoua3LEhERDWVxxOl7fImb8sM7Is4yjagP2Xz3E11SBbGsDZh0w1lQYpJgEpJgGVMQEpJtHzf6Lnf/ePbcsH2Hf8j3NjNlFYYkd2LUJShe7XPVIK5rVHcTOV17bQ3Gon1hCezTMHSp0SQOsRBIntxEyo403LDqDFcOSgRA5WNrHrYOiKmVDpx9Se0L26CbyMzHOLmd3F9cw9OjSestXpwzCeei3OukPubt17fsBZthNL2U5UKYPRTTwTzdCpSCGi2ntDrbkVp0tGq1H5xazsqi+j9ftXcJb+0vlNXQwqrzBJQDImtPs/0fN/AuhjeyUU9bN+x8HWWDKKPiDfshXLRw9hPPlqJH1o1s5pM+eHf/CvgilGR1qigap6dwftsUNTgj2koHB4anNyWMVyBb5cwKhBSXy+sSRk42ZkWQ654F8QYiYsUOrN7CmpxyXLqELI6qFOysY453JcU8/FtvUT7Du/wlVzgNYvnkBKyMAwcwGa3LHBHmav8MbLJBp9+hnLDiu2LR9g++lDcDlBrUE34Uw0eeO8wkXS+PYmLkkScsFJPLe1mUvjv0VXup2W/9yFce5iVKZ0n+7LF0SiZQZgWHYCVfXuDtrRKmYOFwSJYSRm6oMQy6UIhEPVzZibbSGX3VfV0Ep9kw21SvK6UkOB8HlsjmIGZ8Sj06pobnVQVt37uIlAoopLwXDsRcRd9BC6Ked6atVUYPlwOa1fP4/ch3iPYKFkMvnSj+84sIXmN2/GtuV9cDlR540n9jf3oJ/6K9QZw1HFp/lcyCgkxuv5xZ7H481nIMUm4aovo+XdO3GU7/bL/gaCvzoTBxsRN9O5gm5bll3o961SXGSBtMzEGbXkprktqIEoydFXlPoyQ7Li0XnaU4QCQsyEARq1yhsrszuAKdr9QTLEoZ9yLnG/Xe6uKoyEfde3NL95M479W4I9vCPiy0wmV2MVlk8ewfLJI8iN1UixyRhOuTaglhHl5rHfmoh0+t9RpQ5BtjZh+eAB7Ht+CMgYeoMsy1EhZqKx75nD6aKxuWPROeW8DIdaM4FoMtkV3hTtEKw3E0r9mNojxEyYMMKTor0nBJV6V0g6I4ZjL8Y47yakhEzklnosnz6C5YuncLU2Bnt4XeILMSM77Vi3vE/zGzfjOLAFJDW6CWcQe/69aIdOCWhgtE6rJtbg9iTXO43EnH0TmiFTwOWg9X/PYN34bkjcYC1WJ1a7E4iMVgbtGZQej0Yt0WSxU1UfHVW029PQZEMG1CqJuBh3AHRSmPQNs9qcWKzu4PxAi2yleN6u4tCrBLw7hDplt0eImTDB20Hbh00nA4EmcySxv74T3YQzQJJwFK6l5Y2/Yy9cHxI30vYobqb+ihlHyS80v/UPbBveBqcNddYoYs67E/3R5yNpg3OT9pr0m6xIWj2GU652HwvAtvk/tH75FLIjuO0OFHdDrEHj7aocKWg1KvLS3SXqo9HV5C2YF6f3xqElhUnfMGXsep0aoz6w4aXK9b6kqpkmS+j0x2toslJRZ0Gi7QE7VBBiJkzIz05ArZKoNVupbgivJzxJo0N/9PnEnPMPVEm5yK2NtH7xBK2frcDVUh/s4QHuipaV7foy9Wnd5josXzyJ5cNlyA3lSEYThhOuxHjWkqAXFPQGW3qaOEqSCv3R56OffRlIahyF62j8z304m4PnvgxG+qu/kF1O5MNS7aM5bqarJo0JcTokCZwuGXNL6PYNC5aLCdwVh7M8zRtDKW5GqUSfkxYXcqUG/Co3b731Vmw2G/fdd1+H1y+77DJ++KGjz3769OmsXLkSAKvVyn333cfHH39Ma2srJ554IjfffDPJycn+HG5Io9epGZQRz74yM3uKG0hNCO1iU12hTh9GzPzbsW15H9uWD3Ds34SjbCeGGRehGXFsUGvT1DdacThdqFUSKabeXbxklxP7L59j3fgu2FtBktCOPgn91F+FTAp0cnzX8Qm6Ucejik/D8tkKnBV7Kf3XjcSc/lcwZQd8jOHULVt22pGbanE11eBqrEJuqsHVWN32u7kOVGo0gyeiGX4MmrzxDMs28cWm6Oyg3VVqs1qlIiFWR32TjbrG0K1yG6hu2d1RMCiJspoWdh2sZ/LItKCM4XDaUrJDyyoDfhIzLpeLf/7zn6xatYpf/epXnd7ftWsXt99+OyeffLL3Na22TeXdfvvtbNy4kcceewydTsdtt93GokWLeOWVV/wx3LBhZF4C+8rM7C6pZ8bYzGAPp19Iag36qb9CM3QqrV8/h6v6AK1fPYu6cC2G4y5FFRec9FXFKpOSYOhVRUtn+R5av3sZV20xAKr0YRhmLUSdOtiv4+wryo2iq/gETc5oYs/9B5aPH8bRUIn5naUYT7464Kn0dSHULVt2WHE11iA3VXf47WqqRm6sRm5pAHpwjThdOIo24CjaALoYRuVMYrgmlgMVYHe40GqixyB+eCaTQlK8wStmhmYFY2Q9095FFgxGDUrkqy2lIRU3o4Q5hFJ9GQWfi5nCwkJuvvlmDhw4QHZ256e8mpoaampqmDBhAmlpndVmRUUF7733Hk899RRTp04F4KGHHmLu3Lls2bKFSZMm+XrIYcPI3EQ+WV8cUmbH/qJOySPm3Fuxbf0I26b3cBb/TPObN6M/+gK0Rx2PJAX2gt/beBmXxYxt/ZvYd33rfkEfi376b9COmh3wMfeGnppNqhKziP/1bVg/X0Fr8Q4sHz2EfuYl6EafGLAxtmUy+b/GjGy3YqusxlZajKO+yitSXE01brHSm+B0tQ5VfApSfCqqOOV329+ypQH73rU49q5FbqlHs+97rjVBvSuG2q9KSJ14PKqUQWFZJbuvdGfdSIrXs68stIOAg51hV+ARDMUVTSFRQdpidVBc6e4APyLEMpnAD2Jm7dq15Ofn8/jjj/PXv/610/u7du1CkiSGDh3a5fqbNm0C4JhjjvG+NnToUDIyMtiwYcOAxIzGx09Eak/TR3WAmj+OGuJO1yuracFicxAfE7hiSv6Zqwrt1HkY8qfS/OVzOCv2Yv3uJZz71hMz5/eoEwJXyru6wR2Empkcg0aj6jRfWXZh2/41lrVveGvm6I46HuMx56MyxgdsnH0l1VMzp77J2u35r9YnkHzRbRx6bwXWXd9h/e5lMFdgPPa3AangXO95Ak5JMPjkOyrbrTgbKnB5fpztfsvNddT3tAGtAXV8Gqr4FFQmj1BRfsenIhnjjyxEElLQZw5DPvZCHGU7se1eQ+POtSSqWqDoS1qKvkSVlINuxAx0I2egNvnPhRDoa9ThdHdsUxLcwrWh2eaz67Kv59rQ7Nvzsq+kJBrJTI6hvLaFokNmJrVzNQXjuO47YEaW3UVF00KwEa/PxczFF198xPd3795NfHw8d955J99//z0xMTHMnTuXq666Cp1OR0VFBUlJSej1HdVweno65eXl/R6XSiWRlOSfOAaTKTAHNikplryMeIorGimtbWVGTlJA9tsev8w1aQQpQ+7BvPEjar96DUfpDhpX3UzyCRdjmno6ksr/GS5KcawhOYkdzhOTyYi1rIjqj5/BemgPALr0IaSe/gcMuaP8Pq6BMjjHHYxa32Tr8fzP/vVfqf8+j7qvX8e69RPULdWk/2oxKp1/z29ziztbY1B2Qq+/oy67FXttGfa6Mhy15Z6/3b+dTbVHXFdliEWTkI4mMR1NQhrahDQ0CWnu1xLSUBl61zKiV6RMg7HT+PdHJ/Dz118xN6OMXGsRrrpSWte/Rev6t9DnFhA35jjijjoWdax/YhECdY06nIZujm22J8Or2erw+XXZV3NtbHF/d/KyTH67d/TEhJFplK89wL6KJk48ekin9wN5XA9UHgBg7PDUoH0eR6JPYqakpISTTjqp2/fXrFnTY5Du7t27sVqtjB8/nssuu4wdO3bwwAMPcOjQIR544AEsFgs6XWeLg16vx2rtv0nS5ZIxm1v6vX5XqNUqTCYjZrMFp9Pl0213x/AcE8UVjWzeUc6o3MCVkg7IXEecQHz6GFq+eh5H6Q5qPvsX9Vu/JfbEK1An+TcwtaTS7V6I16upq2tGrVYRq3NR8dlKWn/+HGQZtAaM03+NftzJWFRqLHWhX9VY44nvMDfbqKxq7DJeQzm2jY2tMOZ0YvXJNH/xDC17N1H8r78Td8Z1qOL8F3xf5YlX0kpQ1+4z7cnCciQkfSyqhAzUCZmoEtJRJWaiTshAm5xFYnpah/PYBdg8P7QCrb69TgBkpSXwqn0QFc0FPHDFYmxFG7Ht+QFHyQ6sJbuwluyi5tMX0OSNQzdyBrqhk5G0A3e7BeMapSDLMjUe960GucOxNWrcYrG8urnD6wPB13OtqnOfB4efl4FkaEYcAD/truowhmAc1592V7nHlBkX0M/DZDL2ygLVJzGTkZHBhx9+2O37CQk9P1XceeedLFmyxLvsyJEj0Wq1LF68mBtuuAGDwYDN1jldz2q1YjQOTIU6HP456E6ny2/bPpzhOQn8b3Mpuw7WBWyf7fH7XGNTMZxxA/adX2Nd+2+cFXsxv3ELusnnopvgHyuNLMtU1LovXCkGF9byIqjZT93G93A21wOgyT8G/TEXoIpNwukCXIH/7PuDXqNCq1Fhd7iorrccsVWDcmxVQ6YRc1YSlk8fxVl9EPNbt2M87a+o04b4fHw2uwOp1cwITT3xB7+laXsFrrpDuBrKkXtK29fHojJluMWKKQNVgufHlIFkiOtyFdkj5gL5nQUY7LkpVdRZaLCpiRsxC+OIWbha6nHsXYe9cC2uqn04Dv6E4+BPtGh0aIZMRjv8GHcX+gF2PA/0fAGaW+3YPPuMN2o77N/kcZHXNlp9Pi5fzNXlkqn3WGtNMbqgXGvBfb0H2F9uprHZ1qneTaCOq93hpOiQOy17eHZC0D6PI9Gnb4hWqyU/P39gO9RoOomeESNGAFBeXk5mZib19fXYbLYOFprKykoyMkKzHXogUUpIHyhvotXmwKCLvF6hkiShO2oOmrxxtH77Es7irdg2vIVj3wYMx1+OOqX/ncPd2SrVyI1VuMzVuBqrsNVXcq1hPykxTRg/eJn2z+WqxCz0MxegyRk98IkFAUmSSIrTU1lvoa7R2uu+U+qM4cSc+w8sH/8TV10pLe/fg+GEP6IdOqVf45BlGbm51i1U6g7hqnf/dtSWcleS+xOX10On8mD9ECyhSKxBS0ZyDBWe+Ifx+e6sPVVMIrrxp6Ebfxqu+jLse9di37sW2VyBwxNELOnj0ORPRzN8BuqM4WETOKwE0MYaNJ16+LRVAW5FluWQm5O5xYZLlpEkMMUGL/A22WTwdl7fU9LgPW8Czb6yRhxOGVOsziftXvxBwO+ECxYsIDc3l3vvvdf72s8//4xWq2XIkCGkpaXhcrnYtGkTM2bMAGDfvn1UVFQwbdq0QA835EhJMJBi0lNjtlJ4yMyYIZFbe0cVl4Jx7mIce36gdc1ruKoP0PLOHegmnYVu0tlI6s6nr+xyuOuANLqFimyuavu7sQrZ0rnWhwTkttuUZDShMqWRMOZYXCNOwCmHXpZSX0iKbxMzfUEVn0bMOTdj+fwJnCXbaP1sBfLRv0E7/vRubz6y7EJurMFVX4qr7hDOduIFe+fGghLgkqEeE+lD8lElZaNKzEaVmBl2gqUnhmWZPGKm65uSKjEL/dRfoZtyLq6qfdj3rsFRuA7ZYsa+/Uvs279Eik9FN+YUtONOCcnsufYcqU6Lkqpts7uwWB3EhFgBNuW7khCr61WpBn9SMCiJqvoydhXXBU3MtPVjSgg54akQcDFz2mmncc899zB+/HhmzZrFzz//zAMPPMDll19OXFwccXFxnHnmmdxyyy3cc889GI1GbrvtNqZPn87EiRMDPdyQZEReIjW/VLCnuD6ixQy4LQvakTNR547B+t1KHPs3Ydv8Hxz7NqIddypySz0us8fS0ljljqWQezCBao3u7JT4NKT4NA406fhgazOm9CyuuHA2klaPRqMiMSnW7RsOQZNqX+gpPftISLoYjHMXY/3hVezbv8S67g1cDeXoj70EuakWp0e0tFlbysDZTVVXSeW2qiRmu0VLUg7banU89WU1+YNSueG0yQOZZsgzLNvEml/Ke6wELEkS6vRhqNOHIR9zIc5DO9zCZt8m5MZqrGtfx1m+G8OcK5D8HJw9ELoqmKeg9A1rbnVQ22gNWTETClWpC/IS+W5rWVCbTir1ZUKtH1N7Ai5mLrnkEiRJYuXKldxzzz2kpaVx6aWXcuWVV3qXWbp0Kffccw/XXHMNALNnz+aWW24J9FBDlpG5iaz9pSIi6s30FlVMIoZTrsGxbwPW71biqivF+s2/ul5YrUEVl4pkSkMVn+ZOp41v+xt9x2yVHd8W8Yt9P7NTs4LWQ8mfJA5AzABIKjX6mQtQJWZhXfMa9p3fYN/5Ld0Wj1Np3JaVRLdgcQuXbLel5TBrWvnaA9ipD4mbhr9R2hrsKzP32rUiqdRocseiyR2LPGsh9l3fYF3zbxz7N9HynzKMpy5ClRCaBTS7K5inkBSvp7nVQX2jldy00LLAeYVYCFQnVppO7i9rDEpogcsls9fTxiDUOmW3x6+fitKe4HAuvvjiI6Zwx8TEcNddd3HXXXf5a2hhjaKOiw6ZcThdaIJUQyLQSJKEdth01NlHYdv4Lq76MiSlBojHyqKKT0WKSeiTCb6tYF7fejKFC0ndtDToC5IkoRt7CipTGpYvnnK7jNQ6VIlZbWIlKRt1Yg6SKa3XgdqBLJgXbPLS49CoVTS3Oqios5CZ3LfzTdLo0I05GXXqECyfrcBVd4jmd+/AeOKf0Aya4KdR95+e2gEkxRsoqWqmNgQL59WHUL+w1AQjKSYDNeZW9pY2MHZoYF1NxZVNtNqcGPVq8tJDS3S2J/KiR6OA7JQY4oxamix2DpQ3kp8Ten0y/InKEI9h1u98tj0lNTi9l8Gx4YbyZFzvg5uGZtBE4i55BNliRopPGXDcRiiZ8/2NRq1icGYchaVmig419FnMKKgzhhMz/3ZaP3scZ8UeLB//E920+egmnhVS8QxHcjMBJMW7Ezx8cV76mlA7LwsGJfLDtnJ2HawPuJhRPADDcxJRqULn/Dqc6HikjzAkSfK2X1d8mYL+U1HXu1YG4UpbzEznANz+IGn1qExpPglAjaSO2b1hWJb7ezvQDtqqmESMZy1Be9QJgIxtw9vuAG2bxQej9A09u5nc1rhQtMyEkpsJ2lob7ApCaEFbP6bQfmgWYiZMUXpj7CluCO5AwpyWVgdNFndCcG/TlsMNRSjUN7nTTUOJUHsC9jdK3MxAxQy4m7YajluIfvZloNJ44miW4mqoGPC2fUHPbiblvAw9MRNKbiZoi5vZd8iM1e4M2H5lWWaPR0CFYj+m9ggxE6YoXUv3lNSH3A0qnKjyxMuYYrSdClJFCglxOiQJnC6ZxpZOlVyChtPlosFTmCxUbhr+Jt8jZoorm7A7fHNT0o06npizb0SKSfTG0TgObvXJtvuLw+nytqno3s3kfr3WHHpiJtREdlqikaR4PU6XTFFp4B5gK+osmFvsaNQqhmYFruJ8fxBiJkwZlBGHTusOJjxUHfpl9UOVCk/J8kgN/gVQq1SYYt3xCb5yNfkCc7MdlyyjVkneirCRTkqCAVOMFqdL5kBFk8+2q8TRqDKGg60Fy8cPY93yAXKQHnQUy4ZGLRFv7Drt2hvLFWKWGYvVQavNLTRDxc0kSZLXOrMzgCnaSrzMsKz4LluhhBKhPTpBt2jUKvKz3T7MPVGUou1rquojO15GIXmA6dn+oNYjrBLidCEdWOhLJEliWLZv4mYORxWTSMxZN6I9ag7uOJq3aP50Ba4gxNHUN7otbolx+m6DkpUqwE0WO7YAuk56QhFXBp06pKy1wYib8RbL8wipUEaImTBGcTXtLhFxM/2lIsIzmRQSfZjR5Ct6iqmIVIZ642Z8/711x9Fciv64S0Glxl64gdIX/44zwHE0SvBvdy4mgBi9Bp3naT+UrDOh5mJSKBiUBLhFsM1HLsqeaKv8mxiQ/Q0EIWbCmJFKRlNxfdDMyeFOZYRnMil44xNCSMwoY+ku2yVS8WUQcHfojppDzNk3IcUkYq86SONbt+EoDlwcTV0vjq0kSQOqTu0vQlXMZCQZSYjV4XC6KCr137mjUNdopbqhFUkiLMp/CDETxgzLSUCtkqhrtFLTEDqxEOGE4mZKixIxE5qWmcgvmNeeoZkmJKC6oRVzczetH3yAOmM4pt/cgT6nANnaguWjh7H+GJg4mt5a3UJRzNT3kFIeLDrEzRyo8/v+FKvMoPT4kHK3dYcQM2GMXqtmcGY8IOrN9Aeb3em9iGZEcAAwhKZlJlSfgP1NjEFDZor7fPOndQZAFZtE9iV3oBvtqUez/i1av3gCuYumn77E62Y6TBAcLqRCUcz0VOwvmChxMzsPBkDMePsxhb5VBoSYCXsUX+ZuUW+mzyhWmRi9hlhD6D95DIRQzByJVjED7VxNZf7/3koaLbFzLvPG0TiKNtDyn7twmSv9ts+6RisqXGRRhe3nT7F8/gRNr15H04t/xrrlfWSXu3nrQPuG+YNQPi9HeuJm9pY0+Cy1vzv2hFG8DIh2BmHPiLwEPl7vrjcj6BtKvExakjGkysD7gyST25UjbhqhwbDsBL7/uecO2r5Ed9Qc1Ek57r5OtSVtfZ3yxvlk+3JrE86KvTgr9nJW0yaykirR/eTk8DPOtuFtnKXbMZxwJcnxoXdehqqbCdytbOJjtDS22Nl9sJ7sJP+4aJssdkqq3CU/RoZwp+z2CDET5ihVGctqWjC32KKmXocvUDKZMiI8XgbaLsytNicWqyPoPnBZltsCgKNRzGS1ddB2yTKqAIlpdeYIYubf7hY0lYVYPn4I3bTz0E04o0+CXpZduOrLcVbsweURMK76Mu/7gz2bknUxaDKGo84cgTpjOC5zJdYfXsV5aActb/2DnBG/AQbWBNXXhLKbSZIkCvIS2birim1F1WRPyfXLfpQu2ZnJMd4aVaGOEDNhTpxRS05qLKXVzewpbmBKQVqwhxQ2REuNGQC9p2aGxeqgrtEadDHT3OrA4fS4GkLwCdjf5KbHotOosFidlNe0kJ0aG7B9q2KTiDn7Rqzfr8S+8xts69/EVX0Aw/GXI2m7Phay3YqzqshreXFW7AVr52KdqsQsXCnDeP0nF/scadyx+Gx02nZF87KPQpM5EssXT+GqOUDOtn8xP2YU3zbO8Nd0+4TT5aKhObSrUhcMSnKLmcIaTvWTmAmXfkztEWImAhiRl+gWMyX1Qsz0gUpP9d9I7cl0OMnxekqtDuqarAG9eXZFrdkdgBofow35yqL+QK1SMSQznt0lDRQdMgf8eEhqLYbZv0eVOhTrD6/gKFpPS/0hjKcuQmVKx9VU4xYt5XvcVpeagyC7DpuEDnX6UNQZbquLOmM4kiGOksom1m5YT5xR21HIeFAlZhFz7i1Y17+F/edPON6wk+GOCuy1w9Em5wToE+gac7MdWQaVFPiq1LIs98o65g0C3l/rfSDwNb3pxyTLMrK5Emf5bpwVe1ElZqEbP9cv4+kNQsxEACNzE/hqS6k3lU7QOyrrFTdTZGcyKSTG6ymtbqYuBHrhhFojv2AwLDvBLWbKzMwanxWUMehGn4AqOZfWzx5zx9G8czuS1oDcXNtpWSk22S1aPC4jVUoekqrzLaS7TKYO21JrMcz4Larso6j+6ClyNHVY3r0DeebFaAtmBy2GTXEx+bsqtSzLyA3lHitXIc7KvbjqDiHFp6FOG4I6bSiqtKGoUwcjaTvGxWSnxRJr1NJssbO/rJEhnoxWX2G1O9lf3gh0jJeRXQ5cNcVu8VK+B2f5bmRLu5gvjQ7tuFORpOA8nAgxEwEoJ9zBiiZabQ4MOnFYe8LhdFHtqc0TLZYZbxpsCMQnRGvBvPZ4M5oC2DiwKzSZI4iZfweWzx7DVVmEbGsBSYUqdbDX4qLOGI4qLqVX2+tLYLdu8ESedv6as51fUEA51m/+hbPkFwzHLUTSB9566K+gdNlmwVm1r81NV1nYpZtONlfgMFfgKFznfkGSUCVmo/IIHHXaUFTJeYwalMimXVXsPFjnczFTdMiM0yWTES+R2FiIda9buDgrC8FxWF0klcY9rswRaPKPDpqQASFmIoJkk4EUk4EacyuFpWbGDE0O9pBCnpqGVmQZdBoViXHhEeA2UJJCqKWBt6iaKboK5rVHETMlVc1Y7U70WnXQxuKOo7kJR/HPSDoj6rRh3cbP9ERbwbzefa+08Uk8eegUbp5STdr+T3EUrae5shDjiX9CnTmiX2PoL77IZPJaXSoLvZYXV10JHF6sUK1BnToUVUa+WzCmDMJlrsRZtQ9X1T6cVfuRm2tx1ZXiqivFsft793qSml8bM8iPicW1+xDOYcehSs7p0krWF1zNdTjL9yBv2cj/mfaSo6mj9aPDxqyP9VjnRrp/UgcjaULj+inETIQwMi+BNb+0sru4XoiZXlDZLvg30tOyFUKpQFmbZSY0LoTBICleT0KcjoYmGwfKG4OeAiuptWiHTB7wdnrjZmpPUryeIiSKko5l8IRpWL54Ermxipb370U35Vx0E89CUgXmib8/mUy9tbpIcSluIZCe73HTDUJSd7wFq0zpaHLHev93tdR7hY0icuTWRmJbDjHTALTsoeWdz0GtRZUyqM16kzYUVWJmt5YSdzZamddd5Czfg9xYBUAeeJWBFJ/WQbyokrKCan05EkLMRAgj8hJZ80uFqDfTS7w1ZqLExQShJWbqorSVQXskSWJYlokte6opOmQOupjxFX111ShWkLpGK+r04cT++k5av30JR+FabBvfwXloB4YTrkQVm+S3MSv0NHa31aUCZ+Xevlld0vP7NX5VTCKqwZPQDJ7Utv+mGuSa/Xy0+huyqGS4sR6VoxVXZSGuykLsyspaA+rUwe7Ym7ShSMYEt7XIE7DbSXBJElJyHt+WxbLHlsavzzuVnEH+yZbyB0LMRAhKlcbCQ2YcThcadWiq51Chsi66gn8htGJm2txM0RszA25Xk1vMRE4F7752Q1fOAeW8lHRGDCf+EUfeWFq/W+mtSWOYc7n3pu4v6hrdcXTt3UyuphocB3/CWfyz24Jhbeq0nhSX4rW4dGd18QWSJCHFp6JJSqckL4Z3tpdz/pRhnDrKgLN6H87Kfbiq9+Os3g/2Vpxlu3CW7WoTOO3R6NxjzhzpDupOz2dftZ03X95IrEFDVl5wM8v6ihAzEUJWSgxxRi1NFjv7yxsZHgZdToOJNy07CmrMKCimc3OzLeiCVwQAuxmW7f6eFpUFrhKwv+mzm6mLWC5JktCOnIU6fTiWL5/EVX0AyyePoB1zMvqjz/dbnEZdkw0VLtIdJVjXfYOjeCuu2pKOC/nI6jJQxuansH57ObuKG5h7zBBUiZloh7vr9cguJ676Q7g87iln1T5ki9nthsoYgTpzBKrUQZ3ibHYXu11NI3ITA1bI0VcIMRMhSJLEiNwEtuypZk9xvRAzPVAZRQXzFOKNWjRqCYdTpr7JSmpCcObeanNgsTqA6E7NBhiSGY8E1Jqt1DdZw76AoN3horHFbQfotWXmCE1QVYmZxJzTVpPG/svnOMt2YTjpz6iTsn02bldrI46DWznD/jkjEkuJXW/Dm7cjSajS89EMmoAmZzSqlMF+sbr0lbH57uyy3SUNuFxyh1RySaVGnZyHOjkPbcFxvd7mnjBrLtme4B8Rgc8YmZfIlj3V7C6u5/RjBgd7OCGLyyV7q/9mRFHMjCRJJMbpqW5opb7RFjQxo8QlGDxViaMZo15DdlospVXNFB0yM3lkeBe9bPBYZTRqFXHGzgXzukIRM/WN1i4Lxyk1aTS5Y2j96jlctcW0vHM7+gHUpJFlGVfNQRwHf3JbXyoKAZmJypD1sWjyxrkFTO44JENcn/fhb4ZlJ2DQqbFYHRRXNjF4gCnaLllmj6eNQTjGb0X3lSTCUE7AvaUNAe33Em7UNVpxOGXUKonkKEsNTop3i5naxlYgOE9ffY2piHTys00RI2baXEy6XosM5TywOVw0tzq6FUGavPHE/PpOWv/3LM7SXzw1abZhOO7SXtWkke2tOEp/wXnwJxwHtyK31Hd435mQw5flSexlMH+7Yj6SKnip8r1BrVYxMi+RrYU17CquH7CYKatupsliR6dVMTjDt7VrAoEQMxHEoIw49Fo1za0ODlU1k5seek8ToYASL5OaaPRrlc9QpP1TcLCI5gaTXTEsO4FvfiqLiCDg/hSd02rU3ni/+kbrES06qphEjGdcj33rx1jXv42jaAPNlUXu7t+5BZ2WdzVUuK0vB3/CWbYLXI62NzU61Nmj3daXQePZUSXxwaofyU6NDXkhozBqcJJbzBys49RpeQPa1m6PVSY/OyEsE0iEmIkg1CoV+Tkmtu+vY3dJvRAz3eCNl4kiF5NCKGQ0+avKarji7aBd3tgp9iHc6K/VLSleT5PFTm2jtcfrliSp0E04A3XWKCxfPoVsrqTl/XtxTZtPwpz52Et+wbrvRxwHf0JuKO+4bnyaR7xMQJ1V0CGQuK7I3fU7nGofHTXYHXi8u7h+wNb4tn5M4RcvA0LMRBwjcxPdYqa4nhMnh0+NgECipGVHU/CvQvuaHsGirknUmGlPdmosep0aq83Joerwtqj2NZNJISleT3Flk7cCb29Qpw8jdv4dtH73Mo69a2hd/xb7N74LLmfbQpIaddZINIPGox40AVVCVrfuL+/Yw0hkD86M91rjS6uayRvAudPWKTvRN4MLMELMRBiKqt5T0tDrLqzRRjRmMiko7QOCKmbMwjLTHpVKYmhmPDsP1lNUZg5vMTMAywy0dVPvLZLOiPHEP2LPHUvr9yvB3opkTEDtDd4dg6TrXS2pcIzl0qhVDM9N4Jd9tew6WNdvMVPdYKHWbEWtksjPFpYZQQgwLCcBtUqirtFKdUNrVFW47S1ey0wUfjahZZkJn5uGvxmabXKLmUMNzJ7gu5TjQNNvN5NSa6af7k/tyJnoh00iXm2lSZOE09nzOodTF6a1jwryEt1ipriek6f2L25mT7E7XmZQRjx6XXjECx1O+EX5CI6IXqv2RrXv9vhABW3IshzVbqZET/O/+iZ3GmwwCNebhj8ZluUpnncovIvnDcTNBF3XmuktKkMcuvRB/e4dFI5uJoCCQYmA+3rf3+/0Ls+9oiBMXUwgxExEorQ2EH2aOmNusWO1O5EgaHVWgklinB4JcDhlGi1dFjn3Kw6ni8ZmdzmyaG9l0B6lg3ZpdTOtNkcPS4cmsixT1+g5tv10MwUzy07Zd3KYxXINzTKh06hobLFzqKalX9sI52J5CkLMRCDKCbm7OPxTPX2NkpadbDKg1UTf6a9Rq4iPdVtnlNiVQFLfZEUGNGqJ+F4WVYsGkuL1JMXrkWXYX9YY7OH0i+ZWBw6nC+i/ZSZY7k+H04XZI7LDzTKjUavI91R833Wwrs/rm1tslHlE0AjPg3A4En1X8yhAOSHLa1u8X1CBm2h2MSl442aCkJ6t3KwS4/QiOP0wFOtMuPZpUo5tnFHb5wcFRcw0tzqw2vsR8DJAzM02ZECtkoiPCT+Rrbiadh2s7/O6SrxMTmpsr6s2hyJCzEQgcUYtOanuipjC1dQRIWaCa9Kv85ryw+vpNxAoWSThGjczkPpBRr0GvdYdeBrM8zIxTheWldOVWJdd/YibaXMxJfp2UAHG52KmrKyM6667jpkzZzJt2jQuv/xy9uzZ02GZjz76iDPOOIPx48dz7rnnsmbNmg7v19XVcf311zNt2jSmT5/OHXfcgcVi8fVQIxrlxBSupo5Ec1q2gi+CLfuL96YhxEwnvJaZMK0EXD+ALDVJkrznRDBcTeF+Xg7LNqFRqzA32yiv7VvcjJIoMjJMi+Up+FTM2Gw2rrzySqqqqnjqqad47bXXiI2NZeHChdTW1gKwdu1a/va3v3HhhRfy7rvvMmPGDK688koKCwu921m0aBEHDhzgxRdf5JFHHuHrr7/m9ttv9+VQIx7lxNwtLDMdaEvL7l3tiUgkMSQsM+EVZBkIBmfGo5Ik6ptsfa63Egq0dyH2h+Rgipmm8M6w02rU5HvE8K4+ZLFarA4OVjQB4VssT8GnYmbjxo3s3r2b5cuXM27cOEaMGMGyZctoaWnhyy+/BODZZ5/l5JNP5ne/+x35+fksWbKEMWPG8NJLLwGwZcsW1q9fz/3338+YMWOYMWMGd955J//5z3+oqKjw5XAjGuXEPFjRiMUantkR/kAJAI5my0xyEFsahPsTsD/Ra9Xkprndw+Hoahpom4rEIMZy1UfAeelN0e5D3EzhIXdT4tQEQ9g33fWpmBkxYgTPPPMMGRkZbTtQuXdhNptxuVxs3ryZGTNmdFjv6KOPZsOGDYBbEKWlpZGfn+99f/r06UiSxKZNm3w53Igm2WQgxWRAlt0nrACaW+00t7qFXTQWzFMIqjm/ScTMHIk2V1P4iZmBuJkAkj2p+sHIsouEQo79iZtRwhDCOYtJwacVgNPS0jj++OM7vLZy5UpaW1uZOXMmZrOZlpYWMjMzOyyTnp5Oebm7IVhFRQVZWVkd3tfpdCQmJlJWVjag8Wl8nIqr9nQWVYdoh9GCQYn8sK2cvaVmJo5IG9C2Qn2uvaHGc5FMiNMR20PGQiTMtzuUqtD1jVY0GlVA56o8AackGnz+fewNoX5ch+cm8tWPh9hXZvbJ5xPQY+sRBCkJ/Tu2KQluy0B9s7Vf6w9krg1NNs8YjEE5L/tKV3MdOTjJW/29ttFKRnLPrvS9njCEUYOTwmLeR6JPYqakpISTTjqp2/fXrFlDcnKy9//PPvuMBx98kEsvvZSCggKvYNHpOnYl1ev1WK3uL4LFYun0/uHL9AeVSiIpKbbf6x8Jkyk0n/Injcrgh23lFJWZfTb3UJ1rb2je767BkJMW1+vPI5zn2x16o/v71WJ1YIjRY9S7LwP+nqvLJXutQUNzk0kKoqsvVI/rpKMy4IPt7K9oxGQy+kyEBGK+9R5BMDgnsV/XmzxPFWRzi31A16v+zLXBU8JiUHaC3+4T/uDwuY4clMSO/bUcrG5hVP6RH2DtDqfXAjhtbFZYzbsr+iRmMjIy+PDDD7t9PyGhLRr69ddfZ+nSpcybN48bbrgBcAsScAcKt8dqtWI0ug+KwWDo9L6yTExM/4M2XS4Zs7l/1RG7Q61WYTIZMZstOD3FokKJ3FT357XrQB2VVY0DKhIX6nPtDfs8TyEp8Xrq6pqPuGwkzPdIGHRqWm1O9hXXkpseH5C5NjRZcbpkJAlwOno8Bv4g1I9rrE6FUa/GYnXy8+5Kb2uS/hKo+dodbUXn1LKrX8dW68mIrq639Gv9/s5VlmWqG9yJAVrkoJyXfaW7uY7IMbFjfy2bd5QzbWTqEbexp7gem8NFfIyWWK0UsvPurajvk5jRarUdYlm6Y9myZTz33HNcdtllLFmyxFscKzExkZiYGCorKzssX1lZ6Y2zyczM5PPPP+/wvs1mo76+nvT09L4MtxMOh3++zE6ny2/bHgjpCQbijFqaLHYKSxoY7oPUu1Cda28oq3F/WVMTDL2eQzjP90gkxespq2mhus5CVor7iczfc63ypMWbYnUg++/72BtC+bgOyTSx40Ade4rrvfWiBoq/51vtObYatQqDVt2vfSV4XL/1TVasNgdqVf8evvo615ZWOza7e/l4ozZkz4uuOHyuSuzLzgN1Pc5jxwG3pXpkbiJOpwwEp1ebr/C5k0wRMkuWLOHGG2/sUOVTkiQmT57M+vXrO6yzbt06pk6dCsC0adMoLy/nwIED3veV5adMmeLr4UY0kiQxQqRoe6nyFsyL3rRshaQgZDSJgnm9IxyDgNsymXT9ruwcH6tDrZKQ5bYYlkCg1FuKNWjQacOzY7TC8JwE1CqJGrPVKzC7Q6kvE+7F8hR8KmbWrVvHc889x4IFCzj77LOpqqry/jQ3u5+KL7vsMlavXs2//vUvCgsLeeCBB9ixYwcLFy4EYMKECUyePJnFixezdetW1q5dy6233sq5557bIUtK0DtGeovn1Qd1HKFAhSiY58Xb0iCAGU0DrUMSLXgrAYdRW4N6H9RpUUkSiXGevmEBPC8jIS1bQa9TM8TjmjxSvRmXS2ZPiTuTaWQYN5dsj0/FzAcffAC4M5hmzZrV4eeFF14AYNasWdxzzz28/vrr/OpXv2Lt2rU89dRTXveVJEmsWLGC3NxcFi5cyF//+ldmz54tiub1E0XM7C1x1xOIVqw2p/dpT4iZto7VwRAzomDekVEsM2XVzWFTI8pX9YOCUTbAa1WKEJE9shd9mkqqmrBYHeh1avLS4wIzMD/j09TspUuXsnTp0h6XO/fcczn33HO7fT8lJYVHH33UhyOLXgZlxKHXqmmxOiitao6YE7evKPEasQYNsYbwbabmK4JqmYnvnK0oaMMUqyM1wUB1Qyv7ysyMHpLc80pBZqAF8xSS4g2AObDnZVPkWGYACvKS+GjtQXYeoYO2YpVxu6XCOyVbITJmIegWtUpFfo77SS+aXU0VosFkB4L5BCwsMz0TbnEzvnAztV8/kLFc9RFmmRmRm4AkQXVDa7dtMSKlH1N7hJiJAkZ6ItyjuYN2Vb0I/m2PIiiCEQAczlVWA8WwrPASM75yMyUF080UIeelUa9pi5vpwtUky7I3ISTc+zG1R4iZKGBEuyDgvraHjxSUnkxpUdzGoD3KTcfcZMMRgHorsixH3E3DnwxTgoAPNYTFd9Z3bqYgiJkIczOB29UEsKu4s6upqt5CQ5MNjVryWgAjASFmooBh2SbUKnc33qqG8OvG6wsUN1OGcDMBEB+jdafBEpg0WIvVidXuBCLrpuEvBmXEoVZJmFvs1IT4d1aWZW/13wG7mbxiJnBzjjQ3Exw5CFjJchqSZUKrCe9U9PYIMRMF6LVt6Xp7ojRupkqkZXfAnQYbuKdg5eYUa9CgD/NaHoFAp1WT6wnWD/UU7SaL3Wvd852byRYQi5TD6cLcYu+w70hgZG4CEu6HuPrDXMl7PM0lR0ZAc8n2CDETJYyI4nozDqeLGk8gXDR3yz4c5eJdG4Cn4EjoShxowiUIWBHD8TFaNAPsJaUIbIfTRZPFPuCx9YRyo1erJOJ6aD4bTsQYtORluMXw4daZtniZyAn+BSFmogZFhe/2pORFE9UNrciy20JlihVpwQqBzGiqM0deXIK/CZcgYF9lMgFoNSriPaIiEOdlfaPbPZYYp0fVz8rFoYo3bqZdinZDk5XKOgsS7rTsSEKImShB6ctUUdvi7RAbLbQP/u1vqfVIJDmQYqZJtDLoK/mem82BisaABGn3F19lMikEMgg4ki2GBUrcTDtrvPIwm5ceR0yE1dsSYiZKiDNqyUlzN62LtrgZEfzbNYpJv7taFL5EtDLoOxlJRmINGuwOFyVVTcEeTrf4OkstkLVmfC3EQomReYlIQFlN2wNspPVjao8QM1FEm6upPqjjCDRVomBelwT0CVgpmGcSBfN6iyRJDA0DV5Mv3UwASZ5zRHFN+pP6xsi1GLofYN1xM4qI8RbLE2JGEM6M8AR8KdHs0UKlJ5MpTYiZDgRDzESiOd+fhEMQcJ0Sd+Izy4yn2WQgLDNNkW0x9LqaDtbR0mqnpNJt4Yukyr8KQsxEEYpl5mBlY9g0sPMFlYqbSWQydcArZsxWv6fBRlozv0ChiJnCkBYzPnYzKdWphcgeMAUeC8yu4nr2ljYg47ZQJ0Tg91CImSgi2WQgNcGALENhaXRYZ1wuWbQy6AbladTudNHY4r80WLvD6U2zVbp1C3qH4maqqG2hudX/qcr9weduJo+wqA9INlNkixmleF5pVTObd1e5X4uw+jIKQsxEGSOiLG6m1tyK0yWjUUsRe8HqL+3TYGsaLH7bj/L0q9OoiNFr/LafSCQ+RuetjbQvBK0z7YWqr7OZav0sZmRZjshWBu0xxejITnUnfnz/czkQmfEyIMRM1DEyyuJmvPEyiUZUKpGWfTjK07Q/S+a3N+WL1Pi+E8pxM3WeNgZajYpYg2+EqiJmLFYHrTb/ucObWx3YHe6UdyVOJxJRXE1Ol9uVHGnF8hSEmIkyFFVeVGb2fpEjGSVeRjSY7BrliTQQlhlhGesfQxUxE4JtDdr3NfKVUDXqNeh17pYX/oybUcYeZ9RGVI+iw1GCgAES4nQRey0UYibKyEyOIT5Gi93h4kB5Y7CH43cqRU+mI5IcH1jLjKDvtLfMhFoHbX/VaUkOQNxMpGcyKRS0cyuNzE2MWOuoEDNRhiRJURU3481kEsG/XaLchKrrA2GZETVm+sOg9Hg0aokmi90bzB4q+Euoegs6+lPMRInITojTk5nsvv5FarwMCDETlSg1BqKh6WT7VgaCzigX8ho/VgGOlpuGv9BqVAzKcHe9D7W4GV9nMil4LTN+rDXTlskUufEyCr85IZ+jR2dw7NjMYA/FbwgxE4Uopaz3ljTgCjGztS+RZdnrZhKtDLrGmzniTzdTBPe/CRSh2nTSX26mxABkNNVGUYuNSSPS+OO8MRgjOJtQiJkoZFBGHHqdmharg9Kq5mAPx280NNuw2V1IEqQkCBdHVyhP1IFxM0X+TcNfDAvRIGB/CdVAxMzUC5EdUQgxE4WoVSqGey6O2/fXBnk0/kOJl0kxGdCoxaneFUocS5PFjtXu9Pn2nS4XDZ70XXHT6D+KmDlY0RhSWYj1fqrsHAjLjBDZkYW4wkcpowYnAfDGl3t56eOd3sJXkUSl6JbdI0a9Gr3Wf2mw5mY7LllGJUmYYiI/NsFfpCUaiTNqcThlDlaGRhaiLMte60aij+NOkj0i26/ZTFHkZooGhJiJUk6ZmseMMRnIwNc/HuKmp9fwvy2luFyRE0NTWe8J/hWZTN0iSVK7Hk2+j5upbXRvMzFeJ4oWDgBJkkKueF6jxY7D6b5e+FoQKJYZc7MNh9P3lii7w9XWYkNYZiICIWaiFJ1WzR/OHsONF08mNy2O5lYHKz/ZxdKXNrI3Qvo2KZaZdJHJdESSTf7rnh3pvW8CiRIEHCptDZRja4rR+tyNGx+jRa2SkMHrpvQlikVJo1YRZ9T6fPuCwCPETJQzMi+R2y6bykUnj8Co13CgopF7Vm7ihdU7MDf7/iISSLxiRriZjog/e+HU+immIhoJNcuMvzKZAFSS5LX21PkhPbvNxaSL2CJy0YYQMwLUKhUnT83j3iuPYdb4LAC++7mMm55Zy+cbi3G6QifgsC8IMdM7lCBg/1pmRDbZQFHaGlTWW0Iixq3OTzVmFJL8aTEUmUwRhxAzAi+mWB2/P+Mobl4whcEZ8VisDl77fA93/Gsjuw7WBXt4faLJYqfF6m5SJwrmHZk2N5PvY2ZExojviDVovcJ8fwikaPvbhaiIJH+IGXFeRh5CzAg6kZ+TwD8WTuV3pxUQa9BQUtXE3S9v4sFXN/k1u8CXKFaZxDidN1tH0DVeN5NZ3DRCnaFK3EwIiBl/upmg7Zzxp8gWmUyRgxAzgi5RqSTmTMrh3j/OYM7EbCTgq80l3PDkD3y87qBfMgx8iZLJlC4ymXqk7aYhxEyoMzTT3dZgX1nw07P97mby43kp3EyRhxAzgiMSZ9Tyu7mjuO330ykYlESrzckb/9vLbS+sZ0cIF9wTmUy9R6np0dBk82l8lCzLbQHA4qbhE4Yolpny4Ftm/O5mEiJb0AeEmBH0imHZJh649jguP+so4oxaympaWPbvH3nyvW3U+rFJYX8Rwb+9xxTrrgHjkmXMzb4LLG1udXgteMKc7xsGZ8QjSW7h6Y+bfF8InJvJn9lM4ryMFISYEfQalUri+Ik53PvHYzhpci6SBBt2VvL3Z9eyes3+kCqzrjSYFGKmZ1QqydsLx5c3DkXkxsdo0WrEpcYX6HVqclJjgeDGzdjsTppb3QH2/rbM1DdZkX3YENdduVi02Ig0xBVG0GdiDVouPnUkt106jRG5CdjsLt7+uohbn1/Hz0U1wR4eICwzfSXF447zpZgRcQn+YUgIBAErx1anURHjp07MitXE4ZRp9GEqepPFLiyGEYgQM4J+MygjnhsvnswfzhpNQqyOijoLD7/xE4+9vdWvXZh7otXm8Bb8EzEzvUPpKu7LzBFRMM8/KBlNwUzPbu9i8lfROY1ahSnW3fOpzoeZdsrYhcUwsvD5kSwrK+O6665j5syZTJs2jcsvv5w9e/Z0WObUU0+loKCgw8+NN97ofb+uro7rr7+eadOmMX36dO644w4sluDdHAXdI0kSM8Zmcs+Vx3DqtDxUksSWPdXc/Nw63v9+Hy4fmod7i2KViTNqiTGIUuW9ISXBY5nxYbVVb4CoSRTM8yVKW4P95Y0+db/0BX9nMikk+aEKcH2Axi4ILD61D9psNq688koSExN56qmnMBgMPPbYYyxcuJAPPviA5ORkWlpaKC4u5umnn2bMmDHedQ2GtgveokWLsFgsvPjii5jNZm6++WZaWlq4//77fTlcgQ8x6jVceNIIjhufxauf7WbnwXre/XYfCXF6Zk/IDuhYhIup76Qm+L5LcZtlRnTL9iU5abFo1CqaWx1U1lvICEL5gfrGwMScJMXrOVDR6FP3p78DlwXBwaeWmY0bN7J7926WL1/OuHHjGDFiBMuWLaOlpYUvv/wSgL179+JyuZg0aRJpaWnen/h4d/2ELVu2sH79eu6//37GjBnDjBkzuPPOO/nPf/5DRUWFL4cr8AM5aXH87beTOPvYIQB8tqE44E+PVSL4t88kJ/g+ZqZOtDLwCxq1ikEZcUDw4mYCJQj8kdEk0rIjE5+KmREjRvDMM8+QkZHRtgOVexdms/tLt2vXLlJTU0lISOhyGxs3biQtLY38/Hzva9OnT0eSJDZt2uTL4Qr8hCRJnDY9D71WTWl1MzsPBLYVQoWoMdNn2mJm/OFmEjcNXzM0U+mgHZzieQFzM/mhCrBwM0UmPnUzpaWlcfzxx3d4beXKlbS2tjJz5kzALWZiYmJYtGgRmzdvJikpiV//+tf87ne/Q6VSUVFRQVZWVodt6HQ6EhMTKSsrG9D4ND4O9lJ72t4rvyOZvs7VFKdn1vgsvthUwhebSxk3PNWfw+tAdYNbzGSlxPb7mEfbsU1tFzOjVks+CepUhFFqotHn373+EinHNT/XxBebYX+5+Yifrb/m2+ARBCmJBr8eW0VkNzTZetxPb+eqpGWnJPh37P4kUs5jX9InMVNSUsJJJ53U7ftr1qwhOTnZ+/9nn33Ggw8+yKWXXkpBQQEAe/bswWw2c9ppp3H11VezadMmli1bRkNDA3/5y1+wWCzodJ197Hq9Hqu1/0+NKpVEUlJsv9c/EiZT9FgA+jLXX580ki82lbBlTxVWF2Sm+OfzP5yqBvdT3PDByQM+5tFybPV2JwA2uwudQUdczMDiXCxWh7fR57C8pJALxA734zpxVCb8dzsHKpowmYw93tR8Pd8GT7bg4OxEv11Xle0DNLTYer2fnuZqbnGneedlJ/h17IEg3M9jX9InMZORkcGHH37Y7fvtXUevv/46S5cuZd68edxwww3e15999lmsVqs3RqagoICmpiaefPJJrr32WgwGAzabrdO2rVYrMTH9D3RzuWTM5pZ+r98VarUKk8mI2WzBGeK9igZKf+Yap1Mxdlgy24pqeefL3fz25JF+HiXYHS6qPW4mo0airq65X9uJxmMbZ9TSZLGzr7iO3PS4AW2zrMb9uRt0aqwWG1ZL5+90MIiU4xqjlTDo1LTanGzbU8mgjPgul/PHfF2yTI3ngUGN3O/vWG/QSO54u+p6S4/76e1clbIRWgm/jt2fRMp53Bt6I9ahj2JGq9V2iGXpjmXLlvHcc89x2WWXsWTJkg4ma51O18nyMnLkSFpaWmhoaCAzM5PPP/+8w/s2m436+nrS09P7MtxOOPxUodbpdPlt26FGX+d68pRcthXV8tWWQ5x97BAMOv8U2FIor2lGxn0TNerUAz4u0XRsk+L1NFnsVNVbyEweWIaMIiiT4vUh+flFwnEdkhnPzoP17C1pILsHq6cv52tutuF0yUhAnEHj188x3ui26FmsThqbbRh7UaDvSHO12Z00eQrwxRu1YX8ORMJ57Ct87nBThMySJUu48cYbOwgZWZY5+eSTWbFiRYd1fv75Z9LS0khKSmLatGmUl5dz4MAB7/vr168HYMqUKb4ersDPjB2WQnqSEYvVwZpt5X7fX/vgX38V84pUfJk5IhpM+p8hQSqe5y06F6tD4+eYDaNeg1GvBtoCdweCsg2tRkWswb8PVoLA4tMzcd26dTz33HMsWLCAs88+m6qqKu9Pc3MzkiRxyimn8Pzzz/Phhx9y8OBBVq1axXPPPceiRYsAmDBhApMnT2bx4sVs3bqVtWvXcuutt3Luued2yJIShAcqSeKkKbkAfL6pxO9p2lWixky/STb5rtaMSH/1P0O9bQ0Cm9EUqEwmBSW1v9aX52Wc/yoXC4KDT6XpBx98ALgzmFauXNnhvWuuuYZrr72W66+/nri4OB566CHKy8vJzc3l5ptv5vzzzwfcab0rVqzgjjvuYOHChej1eubOnctNN93ky6EKAsiscVm8800RZTUtbN9fx5ihyT2v1E/aCuYFvpBYuKMID5/cNLx9mUSNGX8xNNMdJ1NS1YTd4USrUQdkv/UBFqpJcToOVTf7RmQ3iYJ5kYpPxczSpUtZunTpkXeo0XD11Vdz9dVXd7tMSkoKjz76qC+HJggiRr2GWePcadqfbyz2q5ipqHcHeQvLTN9p36V4oCi9dIRlxn+kJBiIj9HS2GLnYGUT+dld1+7yNYGuoOtLy0ygKhcLAo9IUhcEBMXVtLWwhoo632aVtadKFMzrN8km38XM1ImO2X5HkqR2TScD52pqczMFpk2FIpp86v4UBfMiDiFmBAEhMzmGccNSkIEvN5X6ZR9Ol4tqT8qosMz0HeUJ2CdiRtw0AsIQj6spkG0N6gNsmUn2YWC6cDNFLkLMCALGKVPd1pnvfj6ExVNQzZfUmq04XTIatUpcrPqBYkVpstixO5z93o7D6aLRU1RNtDLwL21BwIETM4G2uiX6UMwEOt5HEDiEmBEEjNFDk8lMjsFidfKDH9K023fLVolMhT4Ta9Cg85R3r2vqf5G7+iYrMqBRS946IQL/oIiZ8poWvzwgdEV9gK1uyT7szyQshpGLEDOCgHF4mrbLx2nalZ5YHBEv0z8kSWp7Cjb3/8bhDRAV6a9+xxSrI8WkRwb2l/s/bsZmd9Lc6hZNgbbMmFvsOAZQ7dYly97g9sT4wMT7CAKHEDOCgHLs2EyMejUVtS38sq/Wp9uurBc1ZgaK8sRaN4CMJkXMJAtTfkAIZPE85bzQaVW9qsbrC+KNWjRqtygeSBBwU4sdp8v9AJUoLDMRhxAzgoDiTtPOBuDzjSU+3bbiZkoTlpl+o8S4KCms/SHQqbvRTiDjZuqDUHROkiSv+PCFyDbFaP1euVgQeMQRFQScE6fkIAE/F9VQXuu7NG3FMpMhLDP9RrHM1A4gPqHNMiMK5gWCod6MJv+7mYJV2dkXGU0ikymyEWJGEHAykmIYn58CwBc+ss64ZFm0MvABvqjpISwzgWVwptsyU2Nuxdzi3+7kwRIEvshoCnTgsiCwCDEjCAonT8sD4LttZbS0DjwLo6HJhs3hQiVJ3h5Dgr7jfQIeiDm/ScTMBJIYg8bb5dzfcTPBygZK9kENJNEvLLIRYkYQFEYPTiIrJQarzcn3P5cNeHtKJlNqgkH4wweAL56AlVYGwjITOIZmBcbVFOiCeQo+OS+FmymiEVd9QVCQJImTp7qtM1/4IE3bG/wrXEwDQnnibmiy9euYtE9/FZaZwBGoIOBAd8xW8EXMjCiYF9kIMSMIGseOycSo11BZb+HnwpoBbUukZfuGhDgdkgROl+yt4tsXGj3prxLuGiiCwDC0XXq27OP6Te0JliDwpWVGiJnIRIgZQdDQ69TMnpAFuIvoDQTFMpMh0rIHhFqlIsEjQvrTpVip0mqK0wl3XwAZlBGHWiVhbrFTax542f+ucFvdgtN1WrHM1DdZ+23FFQHAkY242giCyomTc5GAX/bVcqi6ud/bEW4m35E0gIwmUTAvOGg1anLSYgH/uZqCaXUzxeqQ8FgMW+x9Xj8YlYsFgUWIGUFQSUs0MnFEKuCOnekPsiy3czPF+Gxs0Yq3e3Y/MpratzIQBBZ/x80o4tYUG3irm0atwhTnFlD96dEUjMrFgsAixIwg6CiBwN9vK6Olte9PXU0WOxarAwlITxRp2QPF29JgQJYZcRwCjb/FTLDrBw3kvAxG5WJBYBFiRhB0Rg1KJCctFpvdxbdb+56mrVhlEuP1aDVqXw8v6lBaGgxEzIhGfoFniKcS8IGKRp83cYXgZTIpJA0gCFjUmIl8hJgRBB1JkjjZ0037i00luFx9uxB7g39FvIxPEJaZ8CQnLRadRoXF6qTCh21CFIItCHwhZkSNmchFiBlBSHDMmExiDRqqG1r5qbC6T+uKBpO+ZSBpsOKmETzUKhWDMpTieb53NQWrYJ6CTywzIpYrYhFiRhAS6LVqZk/oXzftStGTyae0L1DWl5olsiyLbKYgM8SPlYDD2s0kqv9GPELMCEKGEybnIEmw40AdpVVNvV6vst5tUs8QmUw+QbngW+1OLFZnr9ezWJ1Y7c4O2xAElvbF83xNsCvo+ioAWBCZCDEjCBlSE4xMHpEG9K2InnAz+Ra9Vk2MJ321L+nZSspsrEGDXisCsYOBImYOVjbhcLp8uu1guxCTTAMoGSCq/0Y8QswIQoqTp7oDgddsK6fJ0nOatsXq8BbREm4m36FkNPWlcJ64YQSf9CQjMXoNdoeL0qr+F6E8HKvdSYvVU3QuWG4mz36tNicWz1h6g0uWaQhS5WJB4BBiRhBSjMxLJC89DpvDxbdbD/W4vGKViY/RimJYPkS5cdT2oUCZ6JYdfFSS1BY3U+47V5MiavVaNUZ9cKxuel2bxbAvrTYam22iX1gUIMSMIKRon6b95aYSnK4jm8qrRINJv5DYj5YGIvg3NPBH3Ex7F1Mwi871p9WGYjEMRuViQeAQR1YQchw9OoM4o5Yas5Uf9xy5m3ZFnTv4Nz1RBP/6Em9GU1PvO2d7M0ZEkGVQGZKpVAL2XUZTWyZTcC0bipjpk8VQlAuICoSYEYQcOq2a4ye607S/2FR8xGVFWrZ/8NaaMff9ppFsEgXzgslQj5uptKrZm102UIKdyaTQH4uhyGSKDoSYEYQkJ0zKQSVJ7DxYT3Fl92naws3kH7xpsH3KZgqNG160kxSvJyFWh0uWOVDuG+tMqFg3kvtRa0YEpkcHQswIQpJkk4HJBZ407Y3dW2cqFMuMSMv2Kf2KTRBPwCGBJEneuJmiQ76Jmwl2wTyF/lSnDhUhJvAvQswIQpZTPGnaa7dX0NjSOXbDZnd6L1TCMuNbFDFjbrFjd/Rcr8TucHpT6ZW0bkHw8GY0+UjMhIqbqT+WGeFmig6EmBGELMNzEhicEY/d4eKbnzqnaVc1uOM5jHoNcUZtoIcX0cQZtd7Mj4ZeuJqUm4tOo/KmzwqCh9cy46OMplBpB5DYH/enqDETFQgxIwhZJEnyFtH7cnNppzTtSm8mkzGo6aKRiCRJJMW7M1d6c+NoHy8jjkXwGZLptsxU1Lb0qvjkkehQdC7I1g0luLyxlxZDEG6maEGIGUFIM/2odOJjtNQ1Wtmyu2M37SqRyeRX+tILRwT/hhbxMTpSE9w3/r3FdQPalrfonAQJQU7NjjVovBbD+l6I7PbVgoMtxAT+RYgZQUij1ag5fmIO0DkQuEJkMvmVvgRbCjETegzLdrua9hTXD2g77YvOqVXBvWVIktSnuBll7MGsXCwIDELMCEKeEybloFZJ7C5p6JBqWiUymfxKcrynsV+fxIyoMRMqKMXzBixmQiyAtj8iO9iViwX+x+di5uDBg/z5z39m6tSpTJ06leuuu46KiooOy6xZs4b58+czYcIE5s6dy+rVqzu8b7VaueOOO5gxYwaTJk3i+uuvp7a21tdDFYQJSfF6pihp2u2K6ImCef5FWGbCG6V43p6DA3MzhUomk0JfLDPK2EWLjcjHp2LGZrNx6aWX4nK5eO2111i5ciWVlZX86U9/QpZlAAoLC/njH//IcccdxzvvvMNvfvMbbrjhBtasWePdzu233853333HY489xksvvURRURGLFi3y5VAFYcYpU/MAWLe9AnOzDYfTRbUnmyk9SbQy8AfKzatXAcCiMFnIMTgzHkmC6obWXsWXdEeoZDIp9ElkixYbUYNPxUxZWRnjxo3j3nvvZeTIkRx11FFceumlbN++nbo699PBSy+9REFBAYsXLyY/P5/LL7+cuXPn8txzzwFQUVHBe++9xy233MLUqVMZP348Dz30EBs2bGDLli2+HK4gjBiWbWJoVjwOp8zXPx2i1tyKS5bRaVRBD0qMVLxixiwsM+GIQachOzUWGFi9mVBzM3nPy170ZxLnZfTg04IQgwcP5pFHHvH+f+jQIV5//XXGjBlDUlISABs3buTkk0/usN4xxxzD3XffjSzLbNq0yfuawtChQ8nIyGDDhg1MmjSp3+PTaHzrVVN7ourVUdCJNRTmeur0QTz9n1/4akuptyhYepIRndb3gX2hMN9A0d1cUz2xSPVNVlRqCVU3MQdOl8v75J+WaPT598yXRNNxBcjPSaC0qpn95Y1MGpnWr20oadkpCYaQOLZKllZ9s63DeLo6tg3N7vMyVMbuK6LtPO4Nfqtu9fvf/57vv/+ehIQEXnrpJW/wVXl5OZmZmR2WTU9Px2KxUFdXR0VFBUlJSej1+k7LlJeX93s8KpVEUlJsv9c/EiZT9MRsBHOupx07lFVf7qWu0cqnG0oAyEmP99txheg+tvEmI5IETpeMWqvt1s1Q02BBlt3fsUG5SahV/9/enUc1daZ/AP8mkAARAgEEtbZ1GcFWQKCCMioKtmq1PXW0p85UqLi2tG5oFWn51bW2RdytWtfW9UyrrbWjM+3YRafWBW3rjoigokJAthCEBJL390fIlZiFgAFyyfM5h3Pg5k143rw33Id3u/Y/0dJR2vXZbr44/uc93JIrm/w5KX+g26fmqU6ezfpZs9bTT3gBAMqUapPx1G/bige6ZdlPdpTaRey25ijnsTUalczcuXMHQ4YMMfv4yZMn4e3tDQCYO3cuZs6ciU8//RQJCQk4ePAgOnbsiOrqaojFhsMC+p/VajWqqqqMHgcAFxcXqFRNH/fVahkUigdNfr4pTk5CSKVuUCiqoNFYt4ETX9lLXQeHdsLB/+Xick4xAMDbQ4zS0kqb/x57qW9LsFRXaTsxypVq5OaVoEvdrrKPyr1bDgDwchdDUW7bz5itOVK7AkAnb93FLut2KUpKlE1a0VNctwWCswDN8llrLGfo5l+WKqpRXKyEsC55NtW2+o01RXYSu6040nkslbpZ1QPVqGTG398fR44cMfu4p6cn9/0zzzwDAFi9ejViYmJw4MABTJs2DS4uLlCrDe+zo//Zzc0Nrq6uRo8DuhVObm6Pl4XWWrljZGNpNNpme21709p1je7dCd+duAmNVvcHzVfq2qzxtHZ9W5Kpunq5u6Bcqcb9smp0bu9u8nn36y52Mg8X3rxXjtKuT/i2g7OTAMqqGuQXP2j0NgYqtQYP6jadk7qJ7OI9a+fqzPUYliiqjSb36ttWq324c7FUIraL2G3NUc5jazQqmRGJROjevbvZx/Pz83H+/HkMHz6cOyaRSNC5c2cUFhYCADp27Mh9r1dYWAiJRAIPDw906NABZWVlUKvVBj00hYWF8Pf3b0y4pA3ycndBxDN+OHVZt9yfVjI1L28PF9wqqLC4oqnEziaIkodEzkJ06eSJ7Lwy3MxXNDqZ4TadEzvBzU7uueUkFMKznRhlSjVKK1RmVyqVV6qhZbqdi6Xt6N5tbZ1NZw9lZmZi5syZyMnJ4Y4pFArk5uZySVCfPn1w5swZg+edOnUK4eHhEAqFeO6556DVarmJwACQm5sLuVyOiIgIW4ZLeEq/TBugPWaam5cVK0fKaMM8u9bjSS8AQG4TbjppbyuZ9GRWbOion5TuaQc7F5PmZ9MW7t+/P3r27Ink5GRcunQJly9fxowZMyCTyTBmzBgAQHx8PC5cuID09HTcuHED27dvx3/+8x9MnjwZgG4oa+TIkUhNTcXp06dx4cIFzJ49G5GRkQgNDbVluISnunaU4qW/dsHQiCe5lQ2keVhzfyZa/mrfArhkpsJyQRPsbcM8PZkVe83QeelYbJrMiMVibNmyBU899RQmT56M+Ph4SKVS7N69G+7uuvH2Hj16YMOGDTh27BhGjRqFr776CsuXL0dUVBT3OkuWLEFUVBSmTZuGSZMmoVu3bli7dq0tQyU8Nzq6G/4+pAdtUd7M9BeCMgsXjRK6aNi1Hk/ptsW4VVABbd1cM2vZ66ZzjUlm7C120jxsPgjq5+eHFStWWCwTHR2N6Ohos49LJBIsXboUS5cutXV4hJBG0F80Six151MyY9c6+3nAReQEVY0G+cWVeMLMRG5T7LV3w5pkpox2pXYoNJBICDGL65kxMwGYMUY9M3bOSSjgNpls7FCTvSaq1uwCbK+JGGkelMwQQszSd9FXqTSoqluiW19ldS1q6/a5oO58+9Wtk26PoNyCxk0CttthJv1cLqXxNh56NMzkWCiZIYSY5ebiDDcX3e0iTPXOlCh0/xl7SEQQtaHt4tuarnUbHt5s5Iome+3dkEkf9szob2L8KBpmciz014cQYpGXhRVNdMHgB33PTF6hEjVWbrJWf9M5e2tffc+MukZrsscQsN9EjDQPSmYIIRZ5W5hsSRvm8UN7Lze4u4lQq2G4U6S06jmKB/a76ZxY5IR2rrr1K6Ymp1epalGt1gCgYSZHQckMIcQiLwvJTKmiLpmR0n4/9kwgEKBLB/0kYOuGmvTtba+bzlnaNkDfY+hqRzsXk+Zlf2coIcSucCtHTMyZ0R+TuRvfHJbYF/2NQq1NZux1JZOefhdgUz0zNMTkeCiZIYRYpL9omPoP+OFFg3pm7F3XuuXZN61cnm2vK5n0ZB66BNrSeWmvsRPbo2SGEGKRfj6Mqf+A7f2/d/KQfkXTveJKVKtNT5qtz957Nyz1zNDEdMdDyQwhxCJLcxNowzz+8HJ3gczDBYzpbm3QEHtPVC1t6GjviRixPUpmCCEW6S8Iiko1t0EeAFSra7llsXTR4Ieu3LyZhpMZ+x9mqusxVNAwE6FkhhDSAHeJCE5CARjA7TsCPLxg0IoR/uDmzVixE7C9927ohz9N9czQMJPjoWSGEGKRUCB4uHFevQuHvV/siLHGrGiy9/bV7wKsrKqBukZj8Ji9x05sj5IZQkiDHm4fT8kMn3Wt22umqKwayqoas+X4sOmcxMUZ4rpbaNTvndFotSivtM+di0nzoWSGENIgmYlbGlAywz8SVxH8ZW4ALN+niQ+bzgkEgnp3z354XpYr1WBM16MoldD+R46CkhlCSINMrWgqpXkJvNTViqEmviSqppIZbudidzGEQkGrxEVaHiUzhJAGcStHKqq5Y9ytDGjDPF7pYsWKJr6sBjKVzNB2AY6JkhlCSIOoZ6bt0K9oys1XgDFmsgxfVgOZum9YaV3CTTc/dSyUzBBCGmTq/kzcUARdNHjlKX8PCAUClFeqTd48FODPMJN3Xa+gqWEmLzuPndgWJTOEkAY9nACsBmMMtRotKvQrRqR00eATF5ETOvm2A2B+qIkvw0wmtwxQ8CMRI7ZFyQwhpEH6/3JrNVooq2pQplSBAXB2EsDDTdS6wZFGa2jzPL4MM1maAEw9ho6FkhlCSIOcnYSQSnRJS2mFyuA/d4GAVozwTUMrmvgyzKSPr1yphlarm/9Dw0yOiZIZQohV6k+25MvFjpimT2Zu5lcYTQKuv+mcvQ8zebYTQygQQMsYyit1Q6B0bjomSmYIIVaR1ZufQBcMfnuifTuInIV4oKpFYWmVwWOKyhpu0znPdva96ZxQKICnuy7GEkU1HlTXQlV3awMaZnIslMwQQqwik+pWjpTV65nxpj1meMnZSYin/NwBGA818W3TufrzZorLdYmZm4szXMROrRkWaWGUzBBCrCLT/wdcf84M9czwlrnN8/iykkmv/oaOxeXVBseI47DPm24QQuyOV72N86rruvK96aLBW9zmeY+saOLLSia9+vcN45IZd/seHiO2R8kMIcQq3AZlShWqVXV3VObJBY8Y008Cvl1QAY1WCyehrqOeb0ubuTu6K1QoVuiGmei8dDw0zEQIsYr+AlGiUHH/vVPPDH/5e0vg5uIEda0Wd4squeMPhxD50bthsmeGzkuHQ8kMIcQq+otGlaoWGi2DAIDUzle7EPOEAgGe9tdvnvdw3gzvhpm4JLsaJeV0XyZHRckMIcQqbi5OBitEpO5iODvRnxA+M7V5Hu+GmUysZqJhJsdDf4kIIVYRCAQGFzgaYuI/k8mMkl8r1fTJjLpWi7xCpcEx4jgomSGEWK3+RYIvS3eJeV3qVjTdLapETa0GVapaqNR1m87xJCEQOTvBve7+YFzsdG46HEpmCCFWq3+Bow3z+M9H6gqpRASNluG2XMkNMbm5OMFVzJ/FrvXPSyehAB40l8vhUDJDCLGaQc8MT1a7EPMEAkG9zfMUD4eYeNazUf+89HTX3a+JOBabJzO3b99GYmIi+vTpgz59+mD27NmQy+UGZSZMmIDAwECDr/j4eO5xlUqFRYsWISoqCmFhYZgzZw5KSkpsHSohpJG83Klnpq3pWm8n4DKe3nOrfrwyOi8dkk2TGbVajYSEBGi1Wuzduxe7du1CYWEh3nrrLYM7s167dg0LFy7Er7/+yn2tW7eOe1z/2Lp16/DFF18gJycHM2bMsGWohJAm8DbomeHXBY+Ypt8J+GaBgncrmfRoYjqx6aBofn4+goODsWDBAnh7ewMAEhIS8M4776C0tBTe3t4oLi5GcXExevfujfbt2xu9hlwux8GDB7Fp0yb06dMHALBy5UoMHz4cf/zxB8LCwmwZMiGkEbw86KLR1uiHmQqKHyC/WLd5Ht8SVYOeGSm/Yie2YdOemaeffhpr1qzhEpl79+5h37596NWrF2QyGQBdr4xAIEDXrl1Nvsa5c+cAAP369eOOde3aFf7+/sjIyLBluISQRvKuu3O2APy74BHTpBIxfKSuYAAu3CgGwPdhJn7FTmyj2aarT5w4ESdOnICnpye++OILCOomZGVlZcHDwwOLFy/GiRMnIJFIMHz4cLz99tsQi8WQy+WQyWRwcTE8If38/FBQUPBYMTk723aKkFPdhmFODrBxmCPVFXCs+jamrj6ernh1cHe4iJ3Qrm45LJ84UrsC1te32xNSFCuqUVldC0DXzrb+e9mcfGVu3Pc+nm68ir0pHO08tkajkpk7d+5gyJAhZh8/efIk1yszd+5czJw5E59++ikSEhJw8OBBdOzYEVlZWVCpVAgJCcGECRNw9epVpKWl4d69e0hLS0NVVRXEYuNVEi4uLlCpVI2s3kNCoQAyWbsmP98SqdSt4UJthCPVFXCs+lpb1/EvBzVzJM3PkdoVaLi+vbr5IuNqIffz0094Ndvfy+Ygcn14zejcQcqr2B+Ho53HljQqmfH398eRI0fMPu7p6cl9/8wzzwAAVq9ejZiYGBw4cADTpk3D4sWLkZyczJUNCAiASCRCUlIS5s2bB1dXV6jVaqPXVqlUcHNresNptQwKxYMmP98UJychpFI3KBRV0Gi0Nn1te+NIdQUcq75U17bL2vp2kBmuAHJmDKWllWZK2x/GGDwkIlQ8qIG72IlXsTeFI53HUqmbVT1QjUpmRCIRunfvbvbx/Px8nD9/HsOHD+eOSSQSdO7cGYWFuqzf2dnZIOkBgB49egAACgoK0KFDB5SVlUGtVhv00BQWFsLf378x4RqprW2eRtdotM322vbGkeoKOFZ9qa5tV0P1fbK9OwQAGHQ3oJS4OPPu/ZnxaghqIYCXu5h3sTeVo53Hlth0wC0zMxMzZ85ETk4Od0yhUCA3N5dLguLj45GSkmLwvIsXL0IkEqFLly547rnnoNVquYnAAJCbmwu5XI6IiAhbhksIIQSAm4szOvhIANRtOifk36ZzgU/JMKD3E60dBmklNk1m+vfvj549eyI5ORmXLl3C5cuXMWPGDMhkMowZMwYAMGzYMHz77bfYt28f8vLycOTIEaSlpWHSpElwd3eHv78/Ro4cidTUVJw+fRoXLlzA7NmzERkZidDQUFuGSwghpE6XDrol2rQaiPCRTVczicVibNmyBZ988gkmT54MtVqNAQMGYPfu3XB3dwcAxMXFQSAQYNeuXVi2bBnat2+PhIQETJ06lXudJUuWYNmyZZg2bRoAIDo6GqmpqbYMlRBCSD09nvTEycsF8JfRpFLCPwJWf2veNkyj0aKkxLaTwpydhZDJ2qG0tLLNj1s6Ul0Bx6ov1bXtakx9a2q1OHEpHyHdfLj9hPjEkdrWkerq7d3O9hOACSGEtE0iZyEGh9KcE8JPtOMOIYQQQniNkhlCCCGE8BolM4QQQgjhNUpmCCGEEMJrlMwQQgghhNcomSGEEEIIr1EyQwghhBBeo2SGEEIIIbxGyQwhhBBCeI2SGUIIIYTwGiUzhBBCCOE1SmYIIYQQwmuUzBBCCCGE1wSMMdbaQbQExhi0WttX1clJCI2mbd+CXc+R6go4Vn2prm2XI9WX6tr2CIUCCASCBss5TDJDCCGEkLaJhpkIIYQQwmuUzBBCCCGE1yiZIYQQQgivUTJDCCGEEF6jZIYQQgghvEbJDCGEEEJ4jZIZQgghhPAaJTOEEEII4TVKZgghhBDCa5TMEEIIIYTXKJkhhBBCCK9RMkMIIYQQXqNkhhBCCCG8RsmMBVqtFmvXrsXAgQMRGhqKKVOmIC8vz2z50tJSzJkzBxEREYiMjMSiRYtQVVXVghE3XVlZGT744ANER0cjPDwc//jHP3D27Fmz5Tdu3IjAwECjL76Qy+Um4//6669Nludr254+fdpkPQMDAzFkyBCTzzl37pzJ8qdPn27h6Bvns88+Q3x8vMGxq1evIi4uDqGhoYiNjcXOnTsbfJ1///vfGDFiBEJCQjBq1CicPHmyuUJ+LKbq+9NPP2HMmDEICwtDbGwsPvnkE1RXV5t9DY1Gg5CQEKO2XrduXXOH3yim6pqammoUd2xsrMXX4UPbPlrX+Ph4s5/hgwcPmn2dCRMmGJV/9D1sUxgxa926daxv377s559/ZlevXmUTJ05kQ4cOZSqVymT5uLg4NmbMGHbp0iX222+/sZiYGDZv3rwWjrppJkyYwF566SWWkZHBcnJy2KJFi1hISAi7ceOGyfIzZ85kc+fOZYWFhQZffPHLL7+w4OBgJpfLDeKvqqoyWZ6vbatSqYza6IcffmCBgYFs//79Jp+zZ88e9vzzzxs9z9x5bw92797NevbsyeLi4rhjJSUlrG/fviwlJYVlZ2ez/fv3s+DgYLP1ZoyxkydPsl69erEvvviCZWdns48//pgFBQWx7OzslqiG1UzVNyMjgz3zzDNs48aNLDc3l/3yyy8sOjqazZ8/3+zrZGdns4CAAHb16lWDtlYqlS1RDauYqitjjL366qts5cqVBnEXFxebfR0+tK2pupaWlhrUUS6Xs9dff52NHDnSYjtFRUWxvXv3Gjy3tLS0BWrROiiZMUOlUrGwsDC2Z88e7lh5eTkLCQlh3333nVH533//nQUEBBh8MP73v/+xwMBAVlBQ0CIxN9XNmzdZQEAAO3v2LHdMq9Wy559/nq1evdrkc1588UW2Y8eOForQ9jZv3sxefvllq8ryuW0fVVlZyWJiYixe4BYsWMDeeuutFoyq6QoKCtibb77JQkND2fDhww0uAps2bWIDBgxgNTU13LEVK1awoUOHmn29iRMnspkzZxocGzt2LPu///s/m8feFJbqO2fOHJaQkGBQ/ptvvmG9evUym4gePnyYhYeHN2vMTWWprlqtloWGhrIffvjB6tez57a1VNdH7dq1iwUFBZn9R5Mxxu7fv88CAgLY5cuXmyNcu0TDTGZkZmaisrISUVFR3DGpVIpnn30WGRkZRuXPnj2L9u3bo3v37tyxyMhICAQCnDt3rkVibiqZTIbNmzcjODiYOyYQCCAQCKBQKIzKq9Vq3Lx5E926dWvJMG3q2rVrBm1lCZ/b9lGbNm1CVVUVkpOTzZZpzHvT2i5fvgyRSIRDhw6hd+/eBo+dPXsWkZGRcHZ25o7169cPN2/exP37941eS6vV4vfffzf4zANA3759TX7mW4Ol+k6cONGoXYVCIWpqaqBUKk2+nj23taW63r59Gw8ePLD6b5C9t62lutZXUlKC1atXIzEx0WLdr127BoFAgK5duzZHuHbJueEijqmgoAAA0LFjR4Pjfn5+3GP1yeVyo7JisRheXl7Iz89vvkBtQCqVYtCgQQbHvv/+e9y6dQvvvfeeUfns7GxoNBp8//33+PDDD6FSqRAREYG5c+fCz8+vpcJ+LFlZWZDJZBg3bhxyc3Px9NNPIzExEdHR0UZl+dy29ZWUlODzzz/HnDlz4OXlZbbc9evXIZPJMHr0aMjlcgQEBCApKQkhISEtF6yVYmNjzc6TKCgoQEBAgMEx/fmZn58PX19fg8cUCgUePHiADh06GD3H1Ge+NViq77PPPmvwc01NDT7//HMEBQXB29vb5HOysrJQW1uLSZMmITMzE/7+/hg/fjxeeeUVm8feWJbqmpWVBQDYtWsXjh8/DqFQiOjoaCQlJcHDw8OovL23raW61rdlyxa4urpi0qRJFstlZWXBw8MDixcvxokTJyCRSDB8+HC8/fbbEIvFtgrbrlDPjBn6yZ2PNryLiwtUKpXJ8qZOEnPl7dnvv/+OlJQUDB06FIMHDzZ6XP+HxM3NDWvWrMGHH36InJwcvPHGGxYnG9qL2tpa5OTkoLy8HNOnT8fmzZsRGhqKqVOnmpwQ2Fbadu/evfDw8MDYsWPNlsnPz0dFRQUePHiA1NRUbNiwAb6+voiLi0N2dnYLRvv4qqurTX5+AZhsN/25a+1n3p7V1tZi3rx5uH79OhYsWGC23PXr11FWVob4+Hhs27YNw4YNQ0pKCvbv39+C0TZeVlYWhEIh/Pz8sGnTJsyfPx+//vor3n77bWi1WqPybaFtlUolvvzyS0yaNIk7j83JysqCSqVCSEgItm7disTERHz11VdITU1toWhbHvXMmOHq6gpAN6Si/x7Q/RF0c3MzWV6tVhsdV6lUkEgkzReojR09ehTvvvsuwsPDkZ6ebrLMqFGjEB0dbfDfXo8ePRAdHY2ffvoJI0aMaKlwm8TZ2RmnT5+Gk5MT17ZBQUG4fv06tm3bZtQV3Vba9uDBgxg1apTB+fyojh07IiMjA25ubhCJRACA4OBgXLlyBbt27cKiRYtaKtzHZqrd9BcuU+2mv0CYeo6pz7y9UiqVmDVrFs6cOYP169db7FH717/+BY1Gg3bt2gEAevbsiXv37mHbtm149dVXWyrkRktMTMTrr78OmUwGAAgICED79u3x2muv4eLFi0ZDNW2hbY8ePQq1Wo0xY8Y0WHbx4sVITk6Gp6cnAN37IxKJkJSUhHnz5hn1SrYF1DNjhn5YobCw0OB4YWEh/P39jcp36NDBqKxarUZZWRlvhl52796N6dOnIyYmBps2bbKY/T/abe3n5wcvLy+76LK1Rrt27Ywu6j169IBcLjcq2xbaNjMzE3l5eXj55ZcbLCuVSrlEBtDNu+jevbvJ98aemWo3/c+mPsNeXl6QSCRWf+btUWFhIcaNG4c///wT27ZtMxo+fpSrqyuXyOgFBATY/edYKBRyiYxejx49AMBk7G2hbY8ePYpBgwZBKpU2WNbZ2ZlLZPQsvT9tASUzZvTs2RPu7u4Ge2soFApcuXIFERERRuUjIiJQUFCAW7duccfOnDkDAHjuueeaP+DHtHfvXixZsgTjxo3DypUrLY6rrlq1CsOGDQNjjDt2584dlJaW4i9/+UtLhPtYrl+/jvDwcKN9Uy5dumQyfr63LaCbDOvj44OePXtaLHf8+HGEhYUZ7KdUW1uLzMxMXrRtfRERETh37hw0Gg137NSpU+jatSt8fHyMygsEAoSHh3Ntq3f69Gn06dOn2eN9XOXl5Rg/fjxKSkqwZ88ek3+n6lMoFIiMjDTaW+nixYvchc9ezZs3DwkJCQbHLl68CAAmz1O+ty2g+ww/2mtsTnx8PFJSUgyOXbx4ESKRCF26dGmG6FofJTNmiMVixMXFIT09HT/++CMyMzORlJSEDh06YOjQodBoNCgqKuLGYnv37o3w8HAkJSXhwoULOHXqFD744AOMGjXK7jP/3NxcLFu2DC+88ALefPNN3L9/H0VFRSgqKkJFRQXUajWKioq4LtoXXngBd+/excKFC5Gbm4uMjAxMnz4d4eHhGDhwYCvXpmHdu3dHt27dsHjxYpw9exY3btzARx99hD///BOJiYltqm31rly5YnZTw6KiIlRWVgIAwsPDIZPJkJycjEuXLuHatWtITk5GWVmZ0cXD3o0ZMwZKpRLvv/8+srOz8fXXX+Pzzz/Hm2++yZWpqKhASUkJ9/OECRNw+PBh7NixAzdu3EBaWhquXr2K8ePHt0YVGuWjjz5CXl4eli9fDm9vb+4zXFRUxCV0ZWVlKCsrA6DrgevXrx9WrVqFY8eO4ebNm9i8eTMOHTqE6dOnt2JNGjZs2DCcPHkS69evx+3bt3Hs2DG89957eOmll7jVWW2pbfPz81FaWmr2n5HKykoUFRVxPw8bNgzffvst9u3bh7y8PBw5cgRpaWmYNGkS3N3dWyrsltXaa8PtWW1tLUtLS2P9+vVjoaGhbMqUKSwvL48xxlheXh4LCAhgBw4c4Mrfv3+fTZ8+nYWGhrK+ffuyBQsWsOrq6tYK32obN25kAQEBJr+Sk5PZqVOnWEBAADt16hT3nN9++42NHTuWhYaGssjISJaSksLKyspasRaNU1RUxObPn8/69+/PgoOD2dixY1lGRgZjrG21rd7kyZPZrFmzTD4WEBDA1q5dy/1869YtNn36dBYZGcl69+7NJk6cyK5du9ZSoTZZcnKy0f4c58+fZ6+99hoLCgpiMTExbNeuXUbPiYmJMTj2zTffsBdeeIEFBwezv/3tb+y3335r9tibon59a2trWXBwsNnPsf7vVlxcnMF7VFFRwZYtW8YGDRrEgoKC2CuvvML++9//tkp9LDHVtkeOHGGjRo1iISEhrH///uzjjz82+EzytW3NnceP7nVV39q1a1lAQIDBsd27d7MXX3yRO/c3btzINBpNs8Xd2gSM1RsrIIQQQgjhGRpmIoQQQgivUTJDCCGEEF6jZIYQQgghvEbJDCGEEEJ4jZIZQgghhPAaJTOEEEII4TVKZgghhBDCa5TMEEIIIYTXKJkhhPDCnTt3EBgYaHQvoaaYP38+YmNjbRAVIcQeOLd2AIQQYg0/Pz/885//xFNPPdXaoRBC7AwlM4QQXhCLxQgNDW3tMAghdoiGmQghNvHVV19h5MiRCAoKwuDBg7Fu3Trubs3z589HfHw89u/fj5iYGISFhWH8+PHIzMzknq/VarFq1SrExsYiKCgIsbGxWLFiBWpqagCYHma6efMmZsyYgf79+yM0NBTx8fE4d+6cQVzl5eVISUlBZGQkIiIisHz5cmi1WqP4jx49itGjRyM4OBj9+/fH0qVL8eDBA+7x6upqLFy4ENHR0QgKCsLw4cOxbds2m76HhJCmoZ4ZQshj++yzz7Bq1SrExcUhJSUFV69exbp165Cfn49ly5YBAK5evYqcnBzMnj0bnp6eWLt2LeLi4nDkyBH4+flhy5Yt2LdvH5KTk/Hkk0/i/PnzWLVqFUQiEWbMmGH0O7Ozs/Haa6+hS5cuSE1NhUgkws6dOzF+/Hhs374dkZGR0Gq1mDx5Mu7evYvk5GR4eXlh69atuHjxIvz8/LjX+u677/Duu+/i5ZdfxqxZs3D37l2sWrUK2dnZ2LFjBwQCAZYtW4Zff/0VycnJ8PX1xfHjx5GWlgYvLy+MGTOmxd5rQogxSmYIIY+loqICGzZswNixY5GamgoAGDBgALy8vJCamooJEyZw5TZt2oQ+ffoAAEJCQvD8889j586dePfdd3HmzBkEBQVxiUFkZCTc3Nzg4eFh8veuX78eYrEYO3fuhLu7OwBg8ODBeOmll5CWlob9+/fj+PHjuHDhArZs2YLo6GgAQFRUlMHkX8YY0tPTMXDgQKSnp3PHu3TpgoSEBBw7dgyDBw/GmTNn0L9/f4wcORIA0LdvX0gkEvj4+Njy7SSENAENMxFCHssff/yB6upqxMbGora2lvvSJwwnTpwAAHTu3JlLZADdhN6wsDBkZGQA0CUHJ06cwOuvv46tW7ciOzsbcXFxeOWVV0z+3jNnziAmJoZLZADA2dkZI0eOxKVLl1BZWYmzZ89CJBJh4MCBXBmJRIJBgwZxP+fk5KCgoMAo/oiICLi7u3Px9+3bF19++SWmTJmC3bt3Iy8vD++88w4GDx5smzeSENJk1DNDCHksZWVlAICpU6eafLywsBAA4O/vb/SYj48PLl++DACYPHky2rVrhwMHDiA9PR3Lly9Hjx49kJqain79+hk9t7y8HL6+vkbHfX19wRiDUqlEeXk5vLy8IBAIDMq0b9/eKP5FixZh0aJFZuN///330aFDBxw6dAhLlizBkiVLEBYWhoULF6Jnz54m604IaRmUzBBCHotUKgUApKeno0uXLkaP+/r6Ys2aNSgtLTV67P79+9wwjVAoxLhx4zBu3DgUFxfj2LFj2LRpE6ZPn871jtTn6emJ+/fvGx0vKioCAMhkMshkMpSWlkKj0cDJyYkro09g6sc/b948REZGmvw9gG41VWJiIhITE3Hv3j38/PPP2LBhA+bMmYPDhw+be3sIIS2AhpkIIY+ld+/eEIlEkMvlCA4O5r6cnZ2xcuVK3LlzB4Bu5dGNGze458nlcvzxxx+IiooCAPz973/H0qVLAeh6bEaPHo1x48ZBoVBAqVQa/d6IiAj8/PPPBo9pNBocPnwYwcHBEIvFiIqKQm1tLY4ePcqVUavVBslRt27d4OPjgzt37hjE7+/vjxUrVuDKlSuorq7GsGHDsH37dgBAp06dMG7cOIwcORL37t2z4btJCGkK6pkhhDwWmUyGyZMnY82aNVAqlejbty/kcjnWrFkDgUDADcEwxvDWW28hKSkJTk5OWL9+PTw9PREfHw9Al5xs374dvr6+CAsLg1wux44dOxAZGQlvb2+DZdIAMG3aNBw/fhxvvPEGpk6dCpFIxM1l2bp1KwDdZN8BAwYgNTUVxcXFeOKJJ7Bz506UlJRwPUJOTk5ISkrCBx98ACcnJ8TExEChUGDDhg2Qy+Xo1asXXF1d0atXL6xfvx4ikQiBgYHIzc3FN998g2HDhrXgu00IMUXAGGOtHQQhhP/27NmDvXv34tatW/D09ERUVBRmz56NTp06Yf78+Thz5gymTJmCTz/9FFVVVfjrX/+K5ORkdO7cGQBQW1uLjRs34tChQygoKICHhwdiY2MxZ84cyGQy3LlzB0OGDMFHH32E0aNHA9At9165ciXOnj0LgUCAkJAQTJs2zWCicVVVFdLT03H48GGoVCqMGDECEokEP/74I3766Seu3JEjR7B161Zcv34dEokE4eHhmDVrFgIDAwEASqUSq1evxo8//oiioiL4+PhgxIgRmDlzJlxdXVvwnSaEPIqSGUJIs9MnM/WTB0IIsRWaM0MIIYQQXqNkhhBCCCG8RsNMhBBCCOE16pkhhBBCCK9RMkMIIYQQXqNkhhBCCCG8RskMIYQQQniNkhlCCCGE8BolM4QQQgjhNUpmCCGEEMJrlMwQQgghhNf+H1nDPLI2LtJCAAAAAElFTkSuQmCC", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "# 获取参数\n", - "cfg = get_args() \n", - "# 训练\n", - "env, agent = env_agent_config(cfg)\n", - "res_dic = train(cfg, env, agent)\n", - " \n", - "plot_rewards(res_dic['rewards'], cfg, tag=\"train\") \n", - "# 测试\n", - "res_dic = test(cfg, env, agent)\n", - "plot_rewards(res_dic['rewards'], cfg, tag=\"test\") # 画出结果" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.13 ('easyrl')", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.13" - }, - "orig_nbformat": 4, - "vscode": { - "interpreter": { - "hash": "8994a120d39b6e6a2ecc94b4007f5314b68aa69fc88a7f00edf21be39b41f49c" - } - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/projects/notebooks/MonteCarlo.ipynb b/projects/notebooks/MonteCarlo.ipynb deleted file mode 100644 index c0613ce..0000000 --- a/projects/notebooks/MonteCarlo.ipynb +++ /dev/null @@ -1,480 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1、定义算法" - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "metadata": {}, - "outputs": [], - "source": [ - "from collections import defaultdict\n", - "import numpy as np\n", - "class FisrtVisitMC:\n", - " ''' On-Policy First-Visit MC Control\n", - " '''\n", - " def __init__(self,cfg):\n", - " self.n_actions = cfg.n_actions\n", - " self.epsilon = cfg.epsilon\n", - " self.gamma = cfg.gamma \n", - " self.Q_table = defaultdict(lambda: np.zeros(cfg.n_actions))\n", - " self.returns_sum = defaultdict(float) # 保存return之和\n", - " self.returns_count = defaultdict(float)\n", - " \n", - " def sample_action(self,state):\n", - " state = str(state)\n", - " if np.random.uniform(0, 1) > self.epsilon:\n", - " action = np.argmax(self.Q_table[str(state)]) # 选择Q(s,a)最大对应的动作\n", - " else:\n", - " action = np.random.choice(self.n_actions) # 随机选择动作\n", - " return action\n", - " # if state in self.Q_table.keys():\n", - " # best_action = np.argmax(self.Q_table[state])\n", - " # action_probs = np.ones(self.n_actions, dtype=float) * self.epsilon / self.n_actions\n", - " # action_probs[best_action] += (1.0 - self.epsilon)\n", - " # action = np.random.choice(np.arange(len(action_probs)), p=action_probs)\n", - " # else:\n", - " # action = np.random.randint(0,self.n_actions)\n", - " # return action\n", - " def predict_action(self,state):\n", - " state = str(state)\n", - " state = str(state)\n", - " if np.random.uniform(0, 1) > self.epsilon:\n", - " action = np.argmax(self.Q_table[str(state)]) # 选择Q(s,a)最大对应的动作\n", - " else:\n", - " action = np.random.choice(self.n_actions) # 随机选择动作\n", - " return action\n", - " # if state in self.Q_table.keys():\n", - " # best_action = np.argmax(self.Q_table[state])\n", - " # action_probs = np.ones(self.n_actions, dtype=float) * self.epsilon / self.n_actions\n", - " # action_probs[best_action] += (1.0 - self.epsilon)\n", - " # action = np.argmax(self.Q_table[state])\n", - " # else:\n", - " # action = np.random.randint(0,self.n_actions)\n", - " # return action\n", - " def update(self,one_ep_transition):\n", - " # Find all (state, action) pairs we've visited in this one_ep_transition\n", - " # We convert each state to a tuple so that we can use it as a dict key\n", - " sa_in_episode = set([(str(x[0]), x[1]) for x in one_ep_transition])\n", - " for state, action in sa_in_episode:\n", - " sa_pair = (state, action)\n", - " # Find the first occurence of the (state, action) pair in the one_ep_transition\n", - "\n", - " first_occurence_idx = next(i for i,x in enumerate(one_ep_transition)\n", - " if str(x[0]) == state and x[1] == action)\n", - " # Sum up all rewards since the first occurance\n", - " G = sum([x[2]*(self.gamma**i) for i,x in enumerate(one_ep_transition[first_occurence_idx:])])\n", - " # Calculate average return for this state over all sampled episodes\n", - " self.returns_sum[sa_pair] += G\n", - " self.returns_count[sa_pair] += 1.0\n", - " self.Q_table[state][action] = self.returns_sum[sa_pair] / self.returns_count[sa_pair]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2、定义训练" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": {}, - "outputs": [], - "source": [ - "def train(cfg,env,agent):\n", - " print('开始训练！')\n", - " print(f'环境:{cfg.env_name}, 算法:{cfg.algo_name}, 设备:{cfg.device}')\n", - " rewards = [] # 记录奖励\n", - " for i_ep in range(cfg.train_eps):\n", - " ep_reward = 0 # 记录每个回合的奖励\n", - " one_ep_transition = []\n", - " state = env.reset(seed=cfg.seed) # 重置环境,即开始新的回合\n", - " for _ in range(cfg.max_steps):\n", - " action = agent.sample_action(state) # 根据算法采样一个动作\n", - " next_state, reward, terminated, info = env.step(action) # 与环境进行一次动作交互\n", - " one_ep_transition.append((state, action, reward)) # 保存transitions\n", - " agent.update(one_ep_transition) # 更新智能体\n", - " state = next_state # 更新状态\n", - " ep_reward += reward \n", - " if terminated:\n", - " break\n", - " rewards.append(ep_reward)\n", - " if (i_ep+1)%10==0:\n", - " print(f\"回合：{i_ep+1}/{cfg.train_eps}，奖励：{ep_reward:.1f}\")\n", - " print('完成训练！')\n", - " return {\"rewards\":rewards}\n", - "def test(cfg,env,agent):\n", - " print('开始测试！')\n", - " print(f'环境：{cfg.env_name}, 算法：{cfg.algo_name}, 设备：{cfg.device}')\n", - " rewards = [] # 记录所有回合的奖励\n", - " for i_ep in range(cfg.test_eps):\n", - " ep_reward = 0 # 记录每个episode的reward\n", - " state = env.reset(seed=cfg.seed) # 重置环境, 重新开一局（即开始新的一个回合）\n", - " for _ in range(cfg.max_steps):\n", - " action = agent.predict_action(state) # 根据算法选择一个动作\n", - " next_state, reward, terminated, info = env.step(action) # 与环境进行一个交互\n", - " state = next_state # 更新状态\n", - " ep_reward += reward\n", - " if terminated:\n", - " break\n", - " rewards.append(ep_reward)\n", - " print(f\"回合数：{i_ep+1}/{cfg.test_eps}, 奖励：{ep_reward:.1f}\")\n", - " print('完成测试！')\n", - " return {\"rewards\":rewards}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 3、定义环境" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [], - "source": [ - "import gym\n", - "import turtle\n", - "import numpy as np\n", - "\n", - "# turtle tutorial : https://docs.python.org/3.3/library/turtle.html\n", - "\n", - "class CliffWalkingWapper(gym.Wrapper):\n", - " def __init__(self, env):\n", - " gym.Wrapper.__init__(self, env)\n", - " self.t = None\n", - " self.unit = 50\n", - " self.max_x = 12\n", - " self.max_y = 4\n", - "\n", - " def draw_x_line(self, y, x0, x1, color='gray'):\n", - " assert x1 > x0\n", - " self.t.color(color)\n", - " self.t.setheading(0)\n", - " self.t.up()\n", - " self.t.goto(x0, y)\n", - " self.t.down()\n", - " self.t.forward(x1 - x0)\n", - "\n", - " def draw_y_line(self, x, y0, y1, color='gray'):\n", - " assert y1 > y0\n", - " self.t.color(color)\n", - " self.t.setheading(90)\n", - " self.t.up()\n", - " self.t.goto(x, y0)\n", - " self.t.down()\n", - " self.t.forward(y1 - y0)\n", - "\n", - " def draw_box(self, x, y, fillcolor='', line_color='gray'):\n", - " self.t.up()\n", - " self.t.goto(x * self.unit, y * self.unit)\n", - " self.t.color(line_color)\n", - " self.t.fillcolor(fillcolor)\n", - " self.t.setheading(90)\n", - " self.t.down()\n", - " self.t.begin_fill()\n", - " for i in range(4):\n", - " self.t.forward(self.unit)\n", - " self.t.right(90)\n", - " self.t.end_fill()\n", - "\n", - " def move_player(self, x, y):\n", - " self.t.up()\n", - " self.t.setheading(90)\n", - " self.t.fillcolor('red')\n", - " self.t.goto((x + 0.5) * self.unit, (y + 0.5) * self.unit)\n", - "\n", - " def render(self):\n", - " if self.t == None:\n", - " self.t = turtle.Turtle()\n", - " self.wn = turtle.Screen()\n", - " self.wn.setup(self.unit * self.max_x + 100,\n", - " self.unit * self.max_y + 100)\n", - " self.wn.setworldcoordinates(0, 0, self.unit * self.max_x,\n", - " self.unit * self.max_y)\n", - " self.t.shape('circle')\n", - " self.t.width(2)\n", - " self.t.speed(0)\n", - " self.t.color('gray')\n", - " for _ in range(2):\n", - " self.t.forward(self.max_x * self.unit)\n", - " self.t.left(90)\n", - " self.t.forward(self.max_y * self.unit)\n", - " self.t.left(90)\n", - " for i in range(1, self.max_y):\n", - " self.draw_x_line(\n", - " y=i * self.unit, x0=0, x1=self.max_x * self.unit)\n", - " for i in range(1, self.max_x):\n", - " self.draw_y_line(\n", - " x=i * self.unit, y0=0, y1=self.max_y * self.unit)\n", - "\n", - " for i in range(1, self.max_x - 1):\n", - " self.draw_box(i, 0, 'black')\n", - " self.draw_box(self.max_x - 1, 0, 'yellow')\n", - " self.t.shape('turtle')\n", - "\n", - " x_pos = self.s % self.max_x\n", - " y_pos = self.max_y - 1 - int(self.s / self.max_x)\n", - " self.move_player(x_pos, y_pos)" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": {}, - "outputs": [], - "source": [ - "import gym\n", - "import os\n", - "def all_seed(env,seed = 1):\n", - " ''' omnipotent seed for RL, attention the position of seed function, you'd better put it just following the env create function\n", - " Args:\n", - " env (_type_): \n", - " seed (int, optional): _description_. Defaults to 1.\n", - " '''\n", - " import torch\n", - " import numpy as np\n", - " import random\n", - " # print(f\"seed = {seed}\")\n", - " env.seed(seed) # env config\n", - " np.random.seed(seed)\n", - " random.seed(seed)\n", - " torch.manual_seed(seed) # config for CPU\n", - " torch.cuda.manual_seed(seed) # config for GPU\n", - " os.environ['PYTHONHASHSEED'] = str(seed) # config for python scripts\n", - " # config for cudnn\n", - " torch.backends.cudnn.deterministic = True\n", - " torch.backends.cudnn.benchmark = False\n", - " torch.backends.cudnn.enabled = False\n", - " \n", - "def env_agent_config(cfg):\n", - " '''创建环境和智能体\n", - " ''' \n", - " env = gym.make(cfg.env_name,new_step_api=True) # 创建环境\n", - " env = CliffWalkingWapper(env)\n", - " if cfg.seed !=0: # set random seed\n", - " all_seed(env,seed=cfg.seed) \n", - " try: # 状态维度\n", - " n_states = env.observation_space.n # print(hasattr(env.observation_space, 'n'))\n", - " except AttributeError:\n", - " n_states = env.observation_space.shape[0]\n", - " n_actions = env.action_space.n # 动作维度\n", - " setattr(cfg, 'n_states', n_states) # 将状态维度添加到配置参数中\n", - " setattr(cfg, 'n_actions', n_actions) # 将动作维度添加到配置参数中\n", - " agent = FisrtVisitMC(cfg)\n", - " return env,agent" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 4、设置参数" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": {}, - "outputs": [], - "source": [ - "import torch\n", - "import matplotlib.pyplot as plt\n", - "import seaborn as sns\n", - "class Config:\n", - " '''配置参数\n", - " '''\n", - " def __init__(self):\n", - " self.env_name = 'CliffWalking-v0' # 环境名称\n", - " self.algo_name = \"FirstVisitMC\" # 算法名称\n", - " self.train_eps = 400 # 训练回合数\n", - " self.test_eps = 20 # 测试回合数\n", - " self.max_steps = 200 # 每个回合最大步数\n", - " self.epsilon = 0.1 # 贪婪度\n", - " self.gamma = 0.9 # 折扣因子\n", - " self.lr = 0.5 # 学习率\n", - " self.seed = 1 # 随机种子\n", - " # if torch.cuda.is_available(): # 是否使用GPUs\n", - " # self.device = torch.device('cuda')\n", - " # else:\n", - " # self.device = torch.device('cpu')\n", - " self.device = torch.device('cpu')\n", - "def smooth(data, weight=0.9): \n", - " '''用于平滑曲线\n", - " '''\n", - " last = data[0] # First value in the plot (first timestep)\n", - " smoothed = list()\n", - " for point in data:\n", - " smoothed_val = last * weight + (1 - weight) * point # 计算平滑值\n", - " smoothed.append(smoothed_val) \n", - " last = smoothed_val \n", - " return smoothed\n", - "\n", - "def plot_rewards(rewards,title=\"learning curve\"):\n", - " sns.set()\n", - " plt.figure() # 创建一个图形实例，方便同时多画几个图\n", - " plt.title(f\"{title}\")\n", - " plt.xlim(0, len(rewards), 10) # 设置x轴的范围\n", - " plt.xlabel('epsiodes')\n", - " plt.plot(rewards, label='rewards')\n", - " plt.plot(smooth(rewards), label='smoothed')\n", - " plt.legend()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 5、我准备好了！" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "开始训练！\n", - "环境:CliffWalking-v0, 算法:FirstVisitMC, 设备:cpu\n", - "回合：10/400，奖励：-200.0\n", - "回合：20/400，奖励：-200.0\n", - "回合：30/400，奖励：-200.0\n", - "回合：40/400，奖励：-200.0\n", - "回合：50/400，奖励：-200.0\n", - "回合：60/400，奖励：-200.0\n", - "回合：70/400，奖励：-200.0\n", - "回合：80/400，奖励：-200.0\n", - "回合：90/400，奖励：-200.0\n", - "回合：100/400，奖励：-200.0\n", - "回合：110/400，奖励：-200.0\n", - "回合：120/400，奖励：-200.0\n", - "回合：130/400，奖励：-200.0\n", - "回合：140/400，奖励：-200.0\n", - "回合：150/400，奖励：-200.0\n", - "回合：160/400，奖励：-200.0\n", - "回合：170/400，奖励：-200.0\n", - "回合：180/400，奖励：-200.0\n", - "回合：190/400，奖励：-200.0\n", - "回合：200/400，奖励：-200.0\n", - "回合：210/400，奖励：-200.0\n", - "回合：220/400，奖励：-200.0\n", - "回合：230/400，奖励：-200.0\n", - "回合：240/400，奖励：-200.0\n", - "回合：250/400，奖励：-200.0\n", - "回合：260/400，奖励：-200.0\n", - "回合：270/400，奖励：-299.0\n", - "回合：280/400，奖励：-200.0\n", - "回合：290/400，奖励：-200.0\n", - "回合：300/400，奖励：-200.0\n", - "回合：310/400，奖励：-200.0\n", - "回合：320/400，奖励：-200.0\n", - "回合：330/400，奖励：-200.0\n", - "回合：340/400，奖励：-200.0\n", - "回合：350/400，奖励：-200.0\n", - "回合：360/400，奖励：-200.0\n", - "回合：370/400，奖励：-200.0\n", - "回合：380/400，奖励：-200.0\n", - "回合：390/400，奖励：-200.0\n", - "回合：400/400，奖励：-200.0\n", - "完成训练！\n", - "开始测试！\n", - "环境：CliffWalking-v0, 算法：FirstVisitMC, 设备：cpu\n", - "回合数：1/20, 奖励：-200.0\n", - "回合数：2/20, 奖励：-200.0\n", - "回合数：3/20, 奖励：-200.0\n", - "回合数：4/20, 奖励：-200.0\n", - "回合数：5/20, 奖励：-200.0\n", - "回合数：6/20, 奖励：-200.0\n", - "回合数：7/20, 奖励：-200.0\n", - "回合数：8/20, 奖励：-200.0\n", - "回合数：9/20, 奖励：-200.0\n", - "回合数：10/20, 奖励：-299.0\n", - "回合数：11/20, 奖励：-200.0\n", - "回合数：12/20, 奖励：-200.0\n", - "回合数：13/20, 奖励：-200.0\n", - "回合数：14/20, 奖励：-200.0\n", - "回合数：15/20, 奖励：-200.0\n", - "回合数：16/20, 奖励：-200.0\n", - "回合数：17/20, 奖励：-200.0\n", - "回合数：18/20, 奖励：-200.0\n", - "回合数：19/20, 奖励：-200.0\n", - "回合数：20/20, 奖励：-200.0\n", - "完成测试！\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAHJCAYAAABtzYa7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAACPp0lEQVR4nO3dd3wU1drA8d+29EogCVV6kBISSIDQREBFQEXQq1dBUbGgiAIKqFwBAb0KghQREVAQuaKCXBUs14r6UiOC9I4ESAIkIUDKZnfn/WPZYTfZNNI2mef7+aDZqedMfeacM2d0iqIoCCGEEEJoiL6qEyCEEEIIUdkkABJCCCGE5kgAJIQQQgjNkQBICCGEEJojAZAQQgghNEcCICGEEEJojgRAQgghhNAcCYCEEEIIoTkSAFUD0lelEKVXleeNnLNCeD4JgDzcDz/8wIQJE8plWWvXriUqKoqkpKQKnUdoywcffEC3bt2Ijo5m4cKFbqfp3bs3UVFRhf5LS0sDICoqivnz55cpPZmZmYwfP57t27cDMGnSJFq3bs3Zs2cLneeJJ56gd+/e2Gw2evfuzcSJE0u8vvzTL1y4kKVLl6q/HedQVFQUx44dc7uMjRs3qtPkl5KSwhtvvEG/fv1o37493bt354knnlDzV5QDBw4waNAg2rZtS//+/Uucp9L666+/eP755+nVqxfR0dH07duXf/3rX5w8edJlOuf9u2XLFqKiotiyZYs6fubMmXTq1ImYmBjWrVvH5s2bueWWW2jbtq26fX788ccC6x84cCBRUVFs2LChwLjbbruNBx98sET5yJ+m+fPnu90nDklJSURFRbF27doSLb+yfPXVVwwYMIDo6GhuvfVWPv/886pOkkcyVnUCRNE++OCDcltWr169WL16NeHh4RU6j9COS5cu8frrr9OrVy8efvhhGjRoUOi0N9xwA08++aTbcUFBQQCsXr2ayMjIMqVp3759/Pe//2XIkCEADBkyhE8//ZT169czfPjwAtOfP3+eX3/9lZEjR6LX61mwYAEBAQElXl/+6efOncuoUaMKTKfX6/nmm28YOXJkgXHubtwAiYmJPPXUU4SGhvLAAw/QpEkTMjIyWL16NcOGDeO1115j0KBBhabt7bff5vTp07z99tvUqlWrxHkqjY8++ohXX32Vzp07M27cOMLDwzlx4gRLly7lu+++Y/ny5bRq1arAfG3atGH16tU0b94cgIMHD7JkyRL+8Y9/cMcdd9C0aVNGjBiBzWZj8eLFhIWFMWTIEHbs2EHv3r3V5SQnJ3Po0CFCQkL49ddfXQK9jIwMDh06xNixYysk7+Hh4axevZpGjRpVyPKvxbfffstzzz3HAw88QI8ePfj++++ZOHEiXl5eDBgwoKqT51EkANKQWrVqlfoieC3zCO24cOECNpuNvn37Eh8fX+S0tWrVIiYmpshpiht/LWJjY2nWrBlffvml2wDoyy+/xGazMXjwYABat25dquWXdPoOHTrw9ddfFwiAzGYz33//Pddffz379u1Th2dkZPDss8/SuHFj3n//fXx9fdVxt9xyC4899hgvv/wy3bt3p3bt2m7XmZ6eTsuWLbnhhhtKlaeSSkxMZMaMGdx///289NJL6vDOnTvTt29fBg0axIsvvui2hCQgIMBlf2dkZAAwYMAA4uLi1GHx8fF07doVgHbt2vHHH3+4LOfXX3/Fz8+Pe++9lzVr1riM2759O4qi0K1bt/LIbgFeXl4VcsyWxezZs+nXrx8vvvgiAD169ODChQvMnTtXAqB8pArMgw0bNoytW7eydetWtVjWUUT78ccfc+ONN9KhQwd+//13AD799FMGDx5MTEwM0dHR3HHHHXz99dfq8vJXZ02cOJHhw4ezZs0atZj5jjvuYOPGjWWaB2DHjh3cf//9xMTE0KtXL5YvX87w4cOLrVr4888/efjhh+nQoQNdunRh7NixpKSkuE2LQ/4qiKioKBYsWMDgwYOJjo5mwYIFXH/99axcudJlvrS0NNq0aaOWsjmeNG+66Sbatm3LLbfcwocffljsfrp48SKvvfYaffv2pV27dgwcOJDPPvusQBrnzZvH66+/TteuXYmOjuaRRx7h+PHjRS7bbDbz1ltv0adPH6Kjoxk4cKBLcfawYcOYOHEiixYtomvXrnTs2JEnn3ySU6dOqdNMnDjR5YkZSl50//vvv3PffffRsWNH9Qn/zJkzgH1/OJb74osvFllVUFLuqkjyH+tpaWmMGzeObt260a5dO+644w7WrVunzvPAAw8A8MADDzBs2DDAXgq0e/dut1VQn3/+OV27dqVevXpAwePpq6++4vbbbyc6OpouXbrw3HPPqcdk/ukd22DBggUFtkf//v05cOBAgTRs3LgRnU5Hz549XYavW7eO1NRUXnzxRZfgB+ylSc899xz3338/ly5dKnRbbt26lW3btrns6+PHjzN69Gi6detGTEwMw4YNIzExUZ3PcWy8//77arVb/sDCYenSpQQGBrotYalVqxYTJ06kT58+ZGVlFRjvXN00f/58dV89+OCDapXpqVOnWLdunTpd165d+euvvzCbzepyfv31Vzp37kyPHj04e/Ys+/fvV8clJiYSGhqqBqnbtm3jkUceIT4+nrZt29K7d2/mz5+PzWZzm7/8Tp8+Ta9evRg8eDCZmZkFzqO1a9fSunVrdu7cyT333EO7du248cYbXapEAVJTUxkzZgydOnUiPj6el19+mTlz5hQ4T50lJycXex1LSkri+PHj3HTTTS7T3HLLLZw4caLY643WSADkwSZPnkzr1q1p3bo1q1evpk2bNuq4BQsWMGHCBF5++WViY2P56KOPePnll+nbty/vvvsus2bNwsvLi+eee47k5ORC17F7926WLl3K6NGjefvttzEYDDz99NNcuHDhmuc5cuSI+qQ9e/Zsnn76aRYvXuxykXVn7969DB06lNzcXN544w2mTp3K7t27eeSRR7BYLKXYcrBo0SJuu+025s2bxy233EKnTp1Yv369yzTffPMNiqKoT0VTpkxh3rx53H777SxatIh+/frx6quv8vbbbxe6npycHO677z6+/PJLRowYwcKFC+nYsSMvvfQSixYtcpl2xYoVHD16lNdee43p06eze/fuYtt3Pffcc7z//vvcfffdvPvuu3Tv3p2JEyfy1VdfqdP88MMPrF27lkmTJjF16lT27dvHsGHDyM7OLtU2y2/dunU8/PDD1K1bl9mzZ/PCCy+wY8cO7rnnHs6fP0+vXr1YsGABACNHjmT16tVFLk9RFCwWS4F/xcl/rD///PMcOXKEqVOn8t5779G6dWsmTJjA5s2badOmDS+//DIAL7/8MpMnTwbgjjvuwGg08uWXX7ose//+/ezfv5+7777b7boTExMZP348N998M++99x4vvPACmzdvZty4cW6nd2yDu+66q8D26NatG8HBwXzzzTcuwzds2MBNN92EyWRyGf7rr79Su3ZtoqOj3a6rVatWTJgwgcaNGxeaFufrR69evTh8+DCDBw8mKSmJSZMmMWvWLHQ6HQ8++CBbt251mX/+/Pk8+uijvPHGG25LUBRF4bfffiMhIaFAgObQv39/nnrqKfz8/NyOd7j77rtd9tucOXNYvXo1derU4YYbblCvfwkJCeTm5rJ3714ArFYrmzZtonv37sTExBAQEMCvv/6qLnfbtm107doVnU7H/v37GT58OCEhIcyZM4d33nmHuLg4FixY4PKgWJizZ8+q87///vtqtW1+NpuNZ599lv79+7N48WI6dOjAG2+8oabLbDbz4IMP8scff/Diiy/y2muvsX//fpYtW1bk+iMjI4u9jh05cgSgwDFx3XXXARTaBk2rpArMgzVv3lxtW5C/mPW+++6jX79+6u+TJ0/yyCOPuLSxqF+/PoMHDyYxMbHQos+LFy+ydu1atQ7bz8+PoUOHqo0Pr2Wed999l8DAQJYsWaJeGJs2bcq9995bZH4XLVpESEgIy5Ytw9vbG7DXsY8bN45Dhw4VOW9+cXFxPPTQQ+rvO+64gxdffJHTp0+rT/rr16+na9eu1KlTh2PHjvHJJ58wduxYHnvsMQC6d++OTqfj3Xff5b777iM0NLTAetauXcvBgwf5+OOPiY2NBexFzhaLhYULF3LvvfcSEhIC2Nu5LFy4EIPBAMDff//N/PnzSU9Pd7vsgwcP8u233/Liiy+qjTgTEhI4deoUW7ZsYeDAgQBkZ2ezdu1aGjZsCNi39Z133sm6dev45z//Wart5mCz2Zg1axbdu3fnzTffVId36NCB/v37s3TpUsaPH8/1118PQKNGjYqtCli3bp1aUuNs9erVRc6b/1jfunUrTz31FH379gWgU6dOhISE4OXlRUBAgNqmpHnz5urftWvXplevXnz11VeMHj3aJU2hoaGFPnknJibi4+PDY489hpeXFwAhISH89ddfKIqCTqdzmd6Rj8jIyAJ5MhqN9O3b16UaLDs7m59++om33367wANCcnIy9evXL3S7FMcREDin65VXXsHLy4sVK1ao43r16sXAgQN54403XEoub731VrUdlTvp6enk5uYW2e6rpCIjI132W/v27QF7FZNz1Wn79u3x8/Pjjz/+ICYmhp07d5KZmUn37t0xGo107tyZX3/9lUcffZTLly+zb98+9bqzf/9+unbtysyZM9Hr7c/+3bp148cff2TLli1FVg+lp6fz0EMP4ePjw/vvv09wcHCh0yqKwpNPPqkG1R07duR///sfP//8Mz169OCLL77g6NGjrFmzhrZt2wLQpUsX9XguSnHXMUdpYP42bP7+/gCFlhZqlZQAVVOOG4/DxIkTee6558jMzOTPP//kv//9Lx999BGAS3FxfrVq1XJpwOdogFpU6UFx82zevJmePXu6PBXGxsYWezFPTEykZ8+eavDjmO/HH38skN/i5J/+5ptvxtvbW21seubMGRITE7njjjvUNCuKQu/evV1KJ3r37k1ubm6hpVdbt26lfv36avDjcPvtt5Obm8vOnTvVYe3atVODHyh+WzvWefPNN7sMnz9/PtOmTVN/d+jQQQ1+wN4mpWHDhmzbts3tckvi2LFjnD17Vg2yHBo1akRsbGyB0oKSuPHGG/nss88K/GvZsmWR8+Xfl507d2b+/PmMHj2aTz/9lHPnzjFhwgQ6dOhQ5HKGDBnCiRMn1H1itVr58ssvueOOO9TgJr/4+Hiys7MZOHAgb775Jtu3b6d79+6MGjWqQPBTEvmrwX766Sf8/Pzo3LlzgWkNBgNWq7XU6yjK1q1bufHGG11ukEajkQEDBrB7924uX76sDi/unHMcy+WdxqKYTCbi4uLUdkC//fYbDRo0UEs8unXrxh9//EFOTg5//vknFotFLb0aNGgQ7733Hnl5eezfv59vv/2WefPmYbVaycvLK3K9I0aM4NChQ7z44otuH1byc74eOII4RzXg5s2badiwoRr8gD1gufHGG9XfNpvNbUlpcdex4qryHIGfsJOtUU3lL1L++++/GT58OPHx8QwdOpSlS5eqJ01RfZLkL7p2XNSLOpGKmyctLY2wsLAC8xXWUNMhIyPD7XzXIv/2CQgIoG/fvmrx8YYNG/D19VWfupwbYLZp00b953iKc27z4ezChQvUqVOnwHBHXjMzM9Vh7tpxQOHb2pGm4rZJREREgWFhYWFFVmMWx7Fud/usdu3aXLx4sdTLDAkJoV27dgX+FVc9kn/8nDlzGD58OLt372bSpEnccMMNPPLIIy7tntzp2bMnderUUavBfvvtN86dO1do9RfYb2SLFy+mYcOGvP/++9x///307NmzRG3D3OnSpQuhoaFqNdiGDRvo16+fS2DsUK9ePbW9VWGKG5/fhQsXCt2niqK4lBAUt1+Cg4Px9/fn9OnThU6TlZVVpuPQnYSEBDUA+vXXX+nevbs6rnv37uTl5bFjxw62b99O06ZNqVu3LmCvrn7ppZfo2LEjgwYNYubMmZw6dQqj0Vhsv03Z2dk0aNCAN998s0TthXx8fFx+6/V6dR3p6eluz2nnYW+//bbLdcjR/KG461hgYCCASyALFFoypHUSANUANpuNxx57jPPnz/PZZ5/x559/8sUXX6hVOZUtMjKSc+fOFRh+/vz5IucLDAxU+4Nx9ssvv5CamlpocJb/ZC/M7bffzt69ezlx4gTr16/nlltuUYMSR33+8uXL3ZZSOFfBOAsODnbbv4xjWEmeFgvjSFP+bXLkyBGXEqn09PQC8547d059e0+n0xV4SnfXKNWZo9rO3X48e/ZsmfJVVoGBgTz//PP8+OOPfP3114wdO5Y//viDqVOnFjmf0Whk0KBBbNiwAavVyrp164iJiVGrXgrTo0cPli5dyrZt21i0aBEtW7Zk+vTp7Nq1q9RpNxqN3HzzzXzzzTdcunSJjRs3Flr10qNHD86fP89ff/3ldvy+ffvo1atXqbrKCA4OLnSfQumP1+7du7NlyxZyc3Pdjv/kk0/o0qULe/bsKdVyi5KQkMD58+fZt28fu3fvpkePHuq46667jgYNGrB9+3a2bdvm0nZpxowZfPvtt7z11lv88ccffP/998ycOROjsfiWIMuXL2fy5Mns2rWLFStWlCn9ERERxV4f//GPfxS4BjkUdR1r0qQJACdOnHBZtuN3s2bNypT2mkYCIA9XkiLL9PR0jh07xl133UW7du3UE9rxZlZJ33AoL/Hx8fz6668uF8W9e/cW25liXFwcv//+u0uV3d69e3nsscfYs2eP+vTi3Kj7yJEjamlFcRyvC69YsYI9e/aoxcaOdYN9WzqXTqSlpTF37txC1xEfH8+pU6fYsWOHy/AvvvgCk8lUaAPWkujYsSNAgY7fZs2axYwZM9TfiYmJLkHQ7t27SUpKIiEhAbDX/zvaazjPU5QmTZpQp04dl8bWYG9r9ueffxZb3VRRTp06xQ033KCWoDRt2pRHH32Url27qiUR7kpTHIYMGcL58+f57bff+Pnnn7nrrruKXN/rr7/OkCFDUBQFX19fbrzxRrXhemElH8Wds/3792f//v28//771K5du0D1qcPtt99OnTp1eO2118jJyXEZZ7VamTVrFiaTiVtvvbXI9TmLj4/np59+cinpsVqtrF+/nnbt2hVaFViYhx9+mIyMDN56660C486ePcuyZcto3ry5ywscZdWqVStCQ0P54IMP0Ov1dOnSxWV8t27d2L17N3/99Zf6+jzYj3nH6/mO0q3du3eTlpZW7DWyTp069OzZk1tvvZW5c+eWqWPYTp06kZSU5NLlQU5Ojkvj7YiIiAIlpQ5FXcccAeC3337rss7vvvuOxo0bl0t7rZpEGkF7uKCgIHbs2MGmTZsK7W8kLCyM+vXr89FHHxEZGUlQUBC//vqr+qRS1reBSuuJJ55gw4YNjBgxgocffpjMzEzmzp2LXq8vst3Ek08+yT333MPjjz/OAw88QE5ODm+99RbR0dF069aNnJwcfHx8+Pe//80zzzzD5cuXmTdvnlpaURyDwcCAAQNYuXIlERERLu0uoqKiuP322/nXv/7FqVOnaNu2LceOHWPOnDkubQzyGzx4MKtWreKpp55i9OjRNGjQgB9//JE1a9YwatSoQt8UKYlWrVrRr18/Zs6cSU5ODtdffz0bN27kp59+Ut++Avv+HTFiBCNHjuTy5cvMmTOHli1bqu13brzxRj788ENeeukl7rrrLg4ePMj7779fZKCg1+sZO3YsL7zwAuPGjeP2228nPT2dBQsWEBwc7NLAvDLVr1+fyMhIpk+fzqVLl2jUqBG7d+/ml19+4fHHHweuVgP8/PPPBAcHu3TC16RJEzp06MCrr74KUGzvyF26dOH9999n4sSJ3H777eTl5bFkyRJCQkIK3HgdgoKC+OOPP9i2bZsaWDvr1KkTderU4d1332X48OGFnhOBgYH8+9//ZtSoUdx9990MHTqUxo0bk5yczEcffcSuXbt488033VaBFmbUqFFs3LiRBx54gMceewyTycTKlSs5efIkS5YsKfFyHGJiYnjmmWd46623OHLkCIMGDSI0NJRDhw6xdOlScnNz3QZHZaHT6ejSpQvr1693aejt0L17d/W1fOdzPDo6mq+//pr//Oc/NGvWjP379/POO++g0+lKfI188cUX+fXXX5k8eXKBV9tLauDAgSxevJinnnqKZ555hqCgIN5//33Onz+vNmwuSlHXMYCnnnqKF154gZCQEHr37s0PP/zA119/zZw5c64pvTWZBEAe7v7772f37t08+uijvPbaa4X2yLxw4UJmzJih9vjZvHlz3nnnHV599VW2b9+u9rFRGa677jqWLl3KG2+8wejRowkLC+Pxxx/nnXfeUd9GcKd169Z8+OGHvPnmmzz77LMEBARwww038Nxzz+Hl5YWXlxfz58/nzTff5KmnnqJ+/fqMGjXK7ZtFhbnjjjtYvnw5AwcOLPCk/tprr/Huu+/y8ccfk5ycTFhYGP379+fZZ58tNFjw9fVV0zx37lwuXbpE06ZNmTFjRrGlCyUxc+ZMFixYwPLly0lPT6dZs2bMmzfP5Y2RuLg4unTponZE17t3b8aPH68+zXfr1o0JEybw4Ycf8u2339KmTRsWLFhQ7Ft5gwcPxt/fn3fffZennnqKgIAAevTowdixY922e6osCxYsYPbs2cydO5f09HTq1q3LqFGj1CrfFi1aMHDgQD766CN+/fXXAqVYd911Fy+++CJDhgwp8ngEe+/Vs2bNYtmyZWrD544dO7JixYpCA+8nnniChQsX8uijj7rt4Vmv13PLLbewcuXKYjum6969O59++inLli3j3Xff5dy5c4SEhNC2bVtWr16tvi1VUi1atGDVqlVqtwY6nY7o6GhWrFjhNlgriZEjR9K6dWu1R+gLFy5Qt25devXqxRNPPKG2wSlPCQkJfP311y7tf5zHKYpCTEyMy/6dOHEieXl5vPXWW5jNZho0aMDIkSM5fPgwP/74Y4kac4eHhzN27FheeeUV1q1bd03bzGg0snTpUmbMmMGUKVMwGo3cfvvthISElPg19aKuY4MHD8ZsNrNs2TLWrFlDw4YNef311yv0UyjVlU6Rr/aJcrZp0yb1bQ2HzMxMunbtyvjx49WO6kTZOQLba22UK4SoXIcOHeLo0aPcfPPNLqV/d911F5GRkS6lu6JiSQmQKHd79uxh3rx5jB07ljZt2pCRkcH7779PYGBggdeqhRBCS7KysnjmmWe47777uOmmm7BarWzYsIHdu3fz3HPPVXXyNEUCIFHuHn74YcxmM//5z384c+YMfn5+dOrUiddee02+KyaE0LT27dvz1ltvsXTpUtatW4eiKLRu3ZolS5YU2q5MVAypAhNCCCGE5shr8EIIIYTQHAmAhBBCCKE5EgAJIYQQQnMkABJCCCGE5mjmLTBFUbDZtNfeW6/XSb41RPKtLZJvbdFivvV6XZFfECgLzQRAOp2OzMwsLJbK/S5WVTIa9YSG+ku+NULyLfnWAsm3tvJdq5Y/BkPFBEBSBSaEEEIIzZEASAghhBCaIwGQEEIIITRHAiAhhBBCaI5mGkELIYSonmw2G1ar5crfOnJyDJjNuVit2nkjqibm22AwotdXXTmMBEBCCCE8kqIoZGamkZ19yWX4uXN6bDbtvAnlUBPz7esbQFBQrQp71b0oEgAJIYTwSI7gJyAgFC8vb/UmaTDoakwpSGnUpHwrioLZnMulS+kABAeHVXoaJAASQgjhcWw2qxr8BAQEuYwzGvWa6gvHoabl28vLG4BLl9IJDAyt9OqwCl3byy+/zMSJEwsM37RpE4MHD6Z9+/b069eP9evXu4zPzc1l6tSpJCQkEBsby7hx40hLS6vIpAohhPAgVqsVuHqTFDWTY/862nhVpgoJgGw2G7Nnz2b16tUFxh05coTHH3+cHj16sHbtWu6++27Gjx/Ppk2b1GmmTJnCb7/9xvz581m+fDlHjx5l9OjRFZFUIYQQHqwq2oaIylOV+7fcq8COHDnCSy+9xIkTJ6hXr16B8cuXLycqKooxY8YA0KxZM/bu3cuSJUtISEggJSWFdevWsWjRIuLi4gCYPXs2/fr1Y8eOHcTGxpZ3koUQQgihMeVeArR582aaNWvGV199RYMGDQqM3759OwkJCS7DunTpQmJiIoqikJiYqA5zaNKkCREREWzbtq28kyuEEEIIDSr3EqD777+/yPHJyclERka6DAsPDyc7O5v09HRSUlIIDQ3F29u7wDTJycllSpvBoK1+Hx35lXxrg+Rb8l2T2Gzuq0YcNSY6HSg144WoEqmsfI8a9Rh169bjpZemVNxK3DAYdBiNBY/liqwhK1UAlJSURJ8+fQodv2nTJmrVqlXkMnJycvDy8nIZ5vhtNpvJzs4uMB7A29ub3Nzc0iTXRZ7ZTFCQ7zXPX51JvrVF8q0tNTXfOTkGzp3TF3pjrKmBX3EqOt86nQ6dzv02rwg2mw69Xk9wsB8+Pj6Vsk6HUgVAERERbNiwodDxwcHBxS7D29sbs9nsMszx29fXFx8fnwLjwf5mmK/vtZ/olqyLZOf4YbXWnFcIi2Mw6AkK8iUzM1vyrQGSb8l3TWI2517pAVpxefVbp7Pn3Wq1aa4EqDLyrSgKiqJU2uv2VquCzWbjwoUssrOtBcYHB/tW2OvxpQqATCYTzZo1K9MK69atS2pqqsuw1NRU/Pz8CAwMJDIykoyMDMxms0tJUGpqKhEREde8Xh0KVqutRvWhUFKSb22RfGtLTc13YR3+2WwKFqu1yvLsZdKX+s2l7t3jeOihR9mw4UssljwWLHiPyMi6vPfeO3z33ddcvnyJJk2aMWLEE3Tq1IUjRw7z4IP3snTpSqKiWgEwceJz/PHHdjZs+AGDwYDNZuP222/m6afHcsst/fnyy3V89tnHnDx5Er1eR8uWrRg9eiytWrUG4K67bqNXrz5s3vw76elpTJ/+Bm3atGPRovl899035OWZueOOISj5oqtVqz5k3brPOHs2ldq16zBgwO08+OAj5f72Vv5A16Eig71K7wgxLi6OrVu3ugzbvHkzHTp0QK/X07FjR2w2G4mJiWpj6WPHjpGSkkJ8fHwZ1qyhRwUhhKiBFEXhtZV/cPjUhSpLQ/MGwbxwf4dSBwCff/4ps2bNw2Kx0rBhI6ZMeYkTJ47x8svTqFMnnN9/38j48c/y6quz6Nq1O3Xr1mPbts1ERbXCarWyY8d2srIuc/Dgfq6/vg179+7h4sWLJCR055dffmLOnDeYMGES7dvHcu7cOd56ayb//vd0PvhglZqGtWs/4fXX5xAYGEjTps15662Z/P77r7z00mQiIuqyYsUydu7cQb169QH47beNfPjh+7zyyqs0bNiYPXt2MX36ZOrWrcctt/Qv1+1aFSo9ABo2bBh33nkns2bN4s477+SXX37hm2++YcmSJYC9mm3AgAFMmjSJV199FV9fXyZPnkynTp2IiYm55vXqJP4RQojqr5p2C3TLLf3V0pikpJN8//23vP/+R7RoEQXAvfcO5fDhQ6xatYKuXbvTrVsPtm3bwtChw9m3bw9Go4m2baP544/tXH99GzZt+o327WMJCgoiODiYiRP/xc033wpAZGRdBg68ndmz33BJQ5cu3YiP7wxAVtZlvv76K8aNm0BCQncAXnjhZf74Y7s6/enTSXh5mYiMrEdkZCSRkZHUrh1ORITri0zVVaUHQC1atGDhwoXMnDmT5cuX06BBA2bOnOnyavy0adN49dVXGTVqFAA9e/Zk0qRJZVyzREBCCFGd6XQ6Xri/AzaFalUFBtCgQSP174MHDwDw5JMjXKaxWCwEBAQC0K1bD7744nNyc3PYtm0LHTvGUa9efRITt3P//Q+yadNv9Os3EICYmA4cP36MDz5YwokTx0lK+psjRw4X+HBqgwYN1b///vsEeXl5tGrVRh3m7e1Ny5ZR6u+bb+7P+vVf8M9/DqZx46bEx3emV68+Bd7krq4qNAD68MMP3Q7v2bMnPXv2LHQ+Pz8/pk+fzvTp08stLdX0oUEIIYQTnU6Ht0mPQV+9rurOXbsoij0wefvt9/Dz83eZztHgNzY2DpPJxI4df7B9+1ZuuaU/9evX57PPVpOcfIZDhw4yY8YNAHz33TfMmDGZm2++lbZto7njjsEcPXqE2bNfLzQNjruiIy0ORuPVsCAkJIT331/F7t272LZtC1u2bOLTT//DI488zkMPPVq2DeIBNPQeoZQACSGEqHpNmthfJjp//hwNGjRU/61f/wUbNnwJ2AORTp0S+O23X9i7dzcdO8bTvn0MVquVpUvfpWnT5tSta//awkcffcBttw3ipZemMGTIP4iJ6cCpU0kABRo1OzRqdB1eXt7s2rVTHWaxWDh06KD6+7vvvubzzz8jOjqGRx55nMWL7ev54YfvKmS7VDbNBEA6Cj8QhBBCiMrStGkzunbtwcyZr/Hbbxs5dSqJjz5azsqVH1C//tUvKHTv3pMNG76kdu061K/fAB8fH9q2jebbbzfQo8cN6nTh4RH89ddODhzYz6lTSaxe/RFr134C4LZbGbDXtNx11z9YtuxdfvnlR06cOM6sWa9x7txZdRqzOZe3357LN9+s58yZ0+zc+Sc7dvxB27bRFbRlKleltwGqUtY8tJZlIYQQnueVV15j8eK3mTnzVS5ezKRevQZMnPgvbr11oDpNQkI3rFYrHTrEqcPi4jrxxx/b6d79agA0Zsx43nhjBqNGPYaXl4nmzVsyadJUJk9+kf3799K+vftvaD7++Ci8vLyZPft1srKy6N37Jrp1u9o8ZeDAQVy4cIEPPlhCamoKgYGB9OrVh5Eja8bHyXWKRopF8tJTyMwBm9GvqpNSaYxGPaGh/qSnX66R/YQURvIt+daCmp7vvDwz58+fISysLiaT69cBjEZ9jcxzcWpivovazwC1avlXWO/XmqkCA8DivihQCCGEENqiqQBIkQBICCGEEEgAJIQQQggN0lQAJFVgQgghhACNBUCKVQIgIYQQQmgsAMKSV9UpEEIIIYQH0FQApFhyqzoJQgghhPAAGguApARICCGEEBoLgJASICGEEEKgsQBIXoMXQgghBEgAJIQQQmiKxWJh9eqP1N9Ll77LXXfdVu7rqajllhdNBUDSD5AQQgit+9//vmH+/DlVnYwqp6kASEqAhBBCaJ1GvoFeLGNVJ6BSSQAkhBDVmqIoKHm5KFX1VXSjFzqdrlSzbNr0O0uWLOL48aP4+vqRkNCNp58ey+HDBxkz5ileeeXfLFo0n5SUFNq2bcdLL03hP//5kG++WY/RaOLuu+/lwQcfUZf39ddf8fHHH3Hy5N/UqlWLgQPvYNiwhzAYDACkpCTz7rtvs337VrKyLhMdHcOTTz5D8+Yt2LDhS159dSoA3bvHMW/eInW5K1d+wJo1n3DhwgXatGnL+PEv0bBhIwAuXbrE22/P5ddffyIvL4+oqOt58snRtGrVWp3/v/9dy6pVKzh79izx8Z2oW7feNW/myqCpAEhKgIQQovpSFIWsL2ZgSzlcZWkwRLTA9/YXSxwEZWRk8NJLzzNq1Bi6du1OamoK06ZNZuHCudx8861YrVZWrFjG5MnTsVgsPP/8swwffh8DB97B4sXL+e67r3nvvXfo3v0GoqJa8sknq1i0aAGjRo0hPr4ze/fuZvbs17lw4QLPPDOOrKzLjBz5CPXq1eff/34Tk8mLZcsWM2rUo3zwwX/o0+cmLl26xLx5b/Lf/35DUFAwO3Ykkpx8hr/+2snMmXPJyzMzbdrL/Pvf03j77fdQFIXnnx+Nl5cPr7/+FgEBAXzzzXpGjnyEd999n5YtW/G//33D7Nmv88wzzxEX14mNG39i8eKFhIdHVPAeuXZSBSaEEKLa0FG60peqdvZsCmazmYiISCIj6xIdHcPrr89myJB71GlGjHiCVq1a07ZtNB07xuPr68uTT46mUaPrGDZsOABHjx5GURRWrlzO4MH/YPDgu2nYsBG33NKfRx55gs8//5RLly7x7bdfc+FCBtOmvU7r1m1p0aIlU6ZMx9vbh7VrP8Hb24eAgAAAwsJqYzKZADAajbz88jSaN2/B9de34Y47BrN//14AEhO3sXv3X0yb9hpt2rTluusa8/jjT9GmTTs+/fRjAD77bDV9+97M4MF306jRdQwdOpxu3XpU4pYuPU2VAEkVmBBCVF86nQ7f21/EiAVLNakCa9Eiir59b2HChDGEhdUmPr4zXbv2oGfPXuza9ScADRo0VKf39fWlbt166jq8vX0AyMvLIz09nbS080RHx7isIza2AxaLhRMnjnPkyGEaNryO0NBQdby3tw+tW7fhyJEjhaazVq0w/P0D1N+BgUHk5tr7zjt4cD+KojBkyECXecxmszrN0aOH6dv3FpfxbdtGc+jQwZJspiqhqQBIPoYqhBDVm06nQ2f0RqerogDoGkyZMoOHH36UzZv/j23btjBt2r+Ijo5R2/UYja634sICrMIaL9tsitNyCpvGhtFoKDSNen3hFUI2mw1/f3+WLl1ZYJyjBAl0KIrrPsmfL0+jsSow+RSGEEKIyrNnz27mzXuTRo0a849/3MfMmXN54YWXSUzcRnp6eqmWFRYWRq1aYWrJkcPOnTswmUzUr9+AZs1acPLkCdLT09Txubm57N+/j8aNmwKFB1iFadq0OZcvXyYvL48GDRqq/z76aDm//fYLAC1atGTXrp0u8+3fv69U66lsmgqA5FMYQgghKpO/vz9r137KwoXzSEo6ydGjh/nhh+9o0KARISEhpV7eP/85jLVrP+Hzzz8jKekk3333DcuWLeb22+8kICCAm27qR3BwCP/610T27dvD4cOHeOWVSWRnZ3PHHYMBezUb2AOU3NycYtfZuXMCLVq0ZPLkF/jjj+0kJZ1k/vzZbNjwpRpUDR06nI0bf2LVqhWcPPk3n332MT///EOp81eZPLt8qpxJCZAQQojK1LhxE2bMmMn777/H559/il6vp0OHeN58cx4pKcmlXt4//zkULy8Tq1evYu7cWYSHR3D//Q9y333DAAgICGD+/HdZsOAtnnnmSQCio9vzzjtLqVevPgAdOsTTunVbRo58mH/9a1qx6zQYDMyZs5CFC+fy8ssTyc7OpnHjpsyYMZOOHeMB6Nq1O5MnT2fZssUsWbKINm3ace+9Q/nf/74pdR4ri07RSI9IeekpJK2YRMB9s6s6KZXGaNQTGupPevrlqmswWAUk35JvLajp+c7LM3P+/BnCwupiMnm5jDMa9TUyz8Wpifkuaj8D1Krlj8FQMZVVGqsCkxIgIYQQQmgsAFKkDZAQQggh0FgAhMWMLetCVadCCCGEEFVMMwGQ9UpWbeeOV21ChBBCCFHlNBcAWSUAEkKIakMj7+loVlXuXw0FQPYeMG1nj1dtQoQQQhTL8WVzs1nabtZkjv1rMFR+rzya6QfIEQBJCZAQQng+vd6Ar28Aly7Ze0v28vJWezC22XRYrdorGapJ+VYUBbM5l0uX0vH1DSjyUxwVRTMBkAUDoEO5nI7l9D6M9a6v6iQJIYQoQlBQLQA1CHLQ6/XYbDWrP5ySqIn59vUNUPdzZdNMAKQAxnpRWE7vJ/ur1zE2jce72zD0vkFVnTQhhBBu6HQ6goPDCAwMxWq1AGAw6AgO9uPChawaUxpSEjUx3waDsUpKfhw0EwAB+N8yiqxNn5J3YCOWo9tAp8e3z8iqTpYQQogi6PV69Hp7L8FGox4fHx+ys601rlfkomg13xVJM42gAfS+Qfj0fAjfW54FwHpqr7xhIIQQQmhQuQdAZ86cYezYsXTr1o34+HgeeeQRDh065DLN119/Tf/+/YmOjmbQoEFs2rTJZXx6ejrjxo0jPj6eTp06MXXqVLKzs8stjYb6bcBgQsm5iHKh9B+jE0IIIUT1Vq4BkNls5rHHHuPs2bMsWrSIVatW4e/vz4MPPkhaWhoAmzdv5vnnn+fee+/l888/JyEhgccee4wjR46oyxk9ejQnTpzggw8+YO7cufzyyy9MmTKl3NKpMxgx1GkCgDXlcLktVwghhBDVQ7kGQNu3b+fgwYPMmjWLdu3a0aJFC2bOnElWVhY//vgjAO+99x59+/blgQceoFmzZkyYMIE2bdqwfPlyAHbs2MHWrVt5/fXXadOmDQkJCbzyyiv897//JSUlpdzSaohoDoA1+VAxUwohhBCipinXRtAtWrRg8eLFREREqMMcLbwzMzOx2Wz88ccfTJw40WW+zp0789133wH2IKpOnTo0a9ZMHd+pUyd0Oh2JiYn079+/TGm8nJPH9v2pBGXVojlgTXENgPYcS+PM+csABPl70TGqDnuOpZOangVASIA3HaLqsPPwOc5fyHGZV6/XEdO8NrWCfMjOtbBtfyrmPGuR6alb25+ohiFs35/Kpeyiv1bvWPdfR85Tv44/ep2OPw+fw2Zz345Jb9Dh5+tNVnYutnxvDVzfuBb1a/tjsdpKtO7yVre2P20a21993HM8jbAgH3y9jfxx8CxWa8kb+Dm2uc2msPPIeRRFKTLf5aFWkA8xLWqz4+A50i/mFD9DRVBs6G0W9Eoeelue/W+9gq+3kdzsHBSrFZ1iQ4cNFMX+95XfOsVqH8aV7azY/6O7+uPKeKffTn/r1HZzruN0ivMwx/zu0l74Pgls0JS618ewfX8q2bmWkmyJ8tnfioJOsVzdlooFnWK9ut3UbWnLty2dhqEAivvt6ViHy7D829N1XHHbU6cDLy8jZrPl6iYtpzaNQde14Lo2MSiKwh9VeZy7odcr+HsbyL18CSx56GxX9hWOfaMU3Ffujn0FfAJDiOrchV1Hzhe4nlc0nWJFb7OnX2/Lu3LMKfZ0U/A40+sUfLwM5ObmwpXz23G8OY49k18AUZ27czApU72PVWV+/AICaR/dzOUeWhq+3kYG3tC8AlJrV64BUJ06dbjhhhtchn344Yfk5OTQrVs3MjMzycrKIjIy0mWa8PBwkpPtbXFSUlKoW7euy3gvLy9CQkI4c+bMtSdOAYNRz1ebTvDtlr8J0uUwLRRsF1Iw6BV0egPnMrJ5c/WfLrPd06c5q384XOwwh93H0hh7Twzf/vo3X/x+vERJ+0fv5nzyY8mq4hzrvv66UExGPbuOnC/RfPmFh/gya1Q3tu5LYfGXe69pGSWnEKjLIVR/mSB9FkH6HP60+dFw5L3YbApvfvwnDcMDaBgewP/tLn2brL+OppFjtnDg74zyT3o+Juw35cF9WhV6DJSUHhuBumxC9FkE67MI1mfjp8vFT2/GV2fGT5d75f9mvHQWTDorJix46awYdTXzLRDrSR2/ZIzl4/87W+p5dSj463II1mcRos8mSJ9FgC5H3YbO29VbZ8Gks2DCiklnxUtX9IOK1uSd1GNtNpfjaTbe/vyvcl++HhtB+myCdVmE6LMI0mfhrzPjWwXH/vasbN7dVmi4XiQv8gjWZ1855rII1GdfPd4cedCb8dHlqel35MWgq5gXcP66lMmCPwOuaV4TFqfrURaB+hyXfeGnM+Orz8W3FPnZmv4w720p2cOMOzclNMHoWzHva5UqAEpKSqJPnz6Fjt+0aRO1al3t0Oh///sfb775JsOHDycqKkoNcry8vFzm8/b2tke1QHZ2doHx+ae5VkFBvmTn2i90FxVfrBgwKFYCDdmYQiJIzbQv39vLQICvifMXcjiTZm987edjxMtkIONirjos0M9ETMtwAM5fyGbvsTSyci2EhvqTnWc/SRvXDaJhRKDb9Gzdm0yu2Upyun154bX8iGoU6nbav46cc1n3xew8TEb7QdG2WRihgT4l2gY5Zgvb9qaQcdlMaKg/VzZHkesuKYPNTHDeWYLzUgk2p+JvycDfcgE/6wWMSsETwHbhBmzB9QC4cNlMrStP/FHXhRIe6lfs+tIyc9hz9DxZuRaycuzzRjevTXCAd5nyAeBjuUioOZngvLP4W9IJsKTjb8nAz3oRs2Lg5+RHAAgO8CK6eZ0il+VlzSLUnExQ3jkCLecJzEsjwJKGj/VS4aUkpWBDj1VnxKbTo6BH0emx4fhbh4LeaZxOncZepuBcJuH4W+f4M1+5g66Q6QpZhtPv/BQ3g4MvHSdIn4Ph7EEglPp1AmhaP7jAdAZbHiF5KQSbUwm0nCcgL41ASxp+lkz0lP3maEOHVWdy2Z4KV7ZpCban2+1QYHvquHq7KPv2dOZu27oqeoLQS0cJ0OeiP3+UPOxtJUtynLtjP/bPEJR3vkKOfSsGbDojNqf9YHPaH6776sqwK78NOenU0l3EO3UP0Nbleu5CUQiwpBFiTlHzEJh3Hn/LBbyU8ik1suiM2DDkS7/9+HL5nW+Y/Tiy/zPkZFBbl4HPuX1APN5eBjq1jiy4MkXB35JBqDn5Sn7OE2BJI8CSgZetfPJj1RkxXLnm+yTvBNrg52OkY6sIt9N7WbPUe0eQ+RyBljT8LekYFQvehn7lkiZ3ShUARUREsGHDhkLHBwdfvVj95z//Ydq0adx+++2MHz8esAcxYG8s7Sw3NxdfX18AfHx8Cox3TOPnV/xNsSiZmdlk5diXraAjUx9EqC2d9JN/Y1ICSM+wF9GFBnjTKDKQ8xdyyLxkD4rqhPgSEuBNxsVcdVhkLT8eHWjvUfrPQ+fYeyyNvDwb6emXuZxlX09Cmwhu7XKd2/QcOJFGqjlbXd71jUJ4ZKD7Hqrf/PhPl3WbzVa16uv2ro1pdV3B4MVg0BMU5EtmZrZarZR+MZdte1OwWOzpvHgpp9h1F8aWlYnl9H4sp/eRd2o/tvTTuFaLONOh8w9F7x9CRkoygbosso/s4HLTEADyLFZyrgQxN8U1oNP17k8UZzsPn2PP0fPk5VmxWOyR3O3dGnN941oF8l0URVGwpZ8i7+RuLEl7sZw9hpJ1odDpvXRWwjJ2A81pWCdAPQYcrJlnyft7J5bTB7GmHsWWmVr4yvUG9H4hV7ZNKDqfAHTefui9/dGp//zQmbzB6IXO4IXOaLL/bfQCgxe6K9XM7vZ3dbLurTe4wWs3IRePAh2JbVGbf/Ruji3nEpaTu8lL2oM19SjWtFOgFJY/HTrfIPQBV7anbxA6Lz/7dvS5sl29/NB5+V7Znle3pX17mtBVwTeJrkVF7e8v58ygu/cBLh3+kwvh9huou+PcHeuFVPL+3oXlzAGsqccq5Ng3ePsQFBrExSwbhdT+l8hnKz6lz6UvCcs6DrRVr+eKzYo15Qh5J//CknwE69mjKLlFVN+YfNBfyYPeLxidj3P6r+TBy+/qMeZ87hpN9mNOV3w4WNz+Xvvpt9x49iNqZx9HRxyhAd5X8mPDevaYfb8kH8aaehQlt4jqMaP31fPHLxi9t32/uObJF53R+2p+DM7XJHt+/rvqM3pmfHFl+7ahToivegxZL6RgOX0Ay5kDWE4fKPw40RtcHhXKW6nOdJPJ5NI2pzAzZ85kyZIlPPTQQ0yYMEHduSEhIfj5+ZGa6prZ1NRUtd1QZGQk33//vct4s9lMRkYG4eFuovNSsFptmPOuHjiZBBJKOnkZKegiW5Fjtt9EDQYdhivHY/aVm7JBr8Og1xUY5uiQyhGM2GwKFosN85Xhep2u0E6r8i9Pry/5tHlWG4rTgVFUx1hWqw2LxYai2ODMfu7028ZxS23M5l5qG6Wi1u3MdiGZvKPbsRzbjs3Nd9V0vkHowxqhr9UQfUgk+sA66ANro/Ovpd5Yvlv8Ln3YhCH1ADkNetrzY1HUbaYrJj+qvGyuN53CZm3IJZu91FCxKerFwZFvdxSbDevpfViObMZy8i+UrIx8GdGhD6mPPqwh+uBI9EF10AeF89O3P9M55zcick8AzdHrdeTlWbGdPUreka1YT+7CllGwqlYXHIGhVkP0IXXtywupiy6glv0GrSt58e6VViZX2YB8XeMXlW9Pdkypzw3sJjLvJLX0LWl6/lcurFmFLfVIgbYtOt9g9HUaow+pi6lWXYIbNCFLF4jVOxCdvuSXtQKtmRSgmm278t7fR2z16M4BbGf2kxtyE1D49UFRFGypR64e+266FdEFR2JwnEdXjn9dQBg638BrOvb1Rj16L19sly+XKd+pXg2xKRBiTSNMf5EoayoXv92KJWk35A8QDEb0YY0whNZHF1zXfm0LikAfUMseTJciD87/B8Ba4KwuUmH7O8OnHrmKEV+yaWA4T3N9Che/24k1aTdKzkXXifVG+7UttL49L8F10QdHoPcPBS+/EgVkxeXnnHcjbAqEWs8TqMumLjlc/v1jLMf/cH+cBNbBUKsB+loN7MdJUDj64EgwFKwRKi/l/qjjCH4mTJjAww8/7DJOp9PRoUMHtm7dyt13360O37JlC3FxcQDEx8cza9YsTpw4wXXX2UtOtm7dCkDHjh3LnD6LU+ScQSDXAcrFc/ZxVw4qo0GP0WA/MbPNFqdhOtdhxqsnr6M3b0fHileXVfiBZMq3Dsdvd/KvO89iw/ESX1HzAdiyLpD71w/k7d8Il9Po5QMWRY8555/kXdkeRS1DseZhOZZI3r6fsJ454DJOX6shhnqtMNRthSGiOXq/glUW+SUZGoJ1E15pR7BcKe2zWG3qvikuP9ZzJ8jb8z2NDvzKE4Fw2HodHyk329OjL/rEtV1OJ2/PD+Qd+BUl26mUx+CFoV4Uxvpt7PkIa4jOWLAq7aT3KTrn/Eak7QxBuixico+T9cl/XE9onR5DRHMMDdpiCG+GoU5jdN7+xW4XrTutr4tV0RFEJpNDPoezqBVa+tD6GBq2wxDRAkOdJuj8Q9WLtNGoxzfUn5z0y9iqWfDiif5W6mJTQJ95BnIygYLnpJJ7GfO+X8g7sNG1LzWdHkNkCwz123j8sW8z+XHKWouGxjReCl6HIVfB4uiNxdsfY4O29utaeFP0teqXKrCuCgajiaN54VzvdZoxQV9jsClYHM0UvXzz5adhhZd02kz+6vZ9Pvgrgs3ZmHdeGak32I+PyJYY6rbEENEcnVfZaniuRblugS1btrBkyRKGDRvGbbfdxtmzVxsy+vn54e/vz0MPPcRjjz1G69at6dmzJ2vWrGHfvn3MmDEDgPbt29OhQwfGjBnDlClTyMrK4uWXX2bQoEEub5ddK4vTWyIZir1tju3iWZdxJqcAyFEqZDTo1IuAY5jzRUH9SvGVxVtsV4OpwhgKrKPwab304Kszq9NabTYc7TYLC7JsF89x9vflXPzrF7BdaYPj5UtOrgUfXR7W0/uwWoMKXbdiMZN3YCPmPzegXE5zZBRDvdYYm8ZjvC62RAFPfumGOlzM8yGQHEzpJ9ThuXlFbwdL8kHM29ZiPbPfZXgkZ9USOH0hTy62zFRyE9dhObwFlCsbztsfU9NOGJvEYYhsYS++LUa2Vy3SrP7UMlxmcshajFk2+03a4IWxSQeMjTtirN/aYy/6nsxm8OKIJYKWpmRsCmQGNiUitifGhu3QB4RVdfI0I89wNTAIyDgM+KkPe7asDMw7vyZv/y+Qd6W9iNHLftw37oCxQZsquZFdC5NBz/68ejQ0pmHQKVzUBxEW3RNjo/bow5ui0xuqOomlYjLo2W+px/VepzHoFC7hT2h0D4zXxWKIaFbpAZzRoOdgXiQNjWkE67OxYMCnaUeMTeIwNmxXqpKzCktjeS7sq6++Auxvfn344Ycu40aNGsXTTz9N9+7defXVV1m4cCFz5syhefPmLFq0SK1a0+l0LFiwgKlTp/Lggw/i7e1Nv379eOGFF8qUNkfY41wCdF4NgM65jDMadG4CIH2BgMXgdKN2/FWwBKjwoMZ0JXBxDrLcsfz9J7eeXc6dIRd4L+82Ugkmz6LgeD7Ovw4l9zK5O74kb88PYLW/3q4Pb4ZX25swNO7A7wtn0c3nINZTe8izdi6wbkVRsBzbTu6m/6iBj84vBFOrGzC16lnmm5HJaD8xOnofxyftANAg33ZwzY814zS5mz7GenKXfYDOgLFpHKnB7aj1xxICyFIb3OUvAVJyLpGbuI68fT+B7cp+i2yJqe1NGK+LLfVTkNGo50BeXRIMhzHqbFww1ia8622YmnbyiBO6OjMa9KzK7Eq0/1n+vBTOgI4xNLy+YVUnS3OMBp0aGARfOAjE4KWzkrvjK8x/fqUGPvrQBpja3VRtj32DQc//stthMflxOCuEkGbteLJTu6pO1jUzGHT8mhOFyWTi72x/9HVb8XyXuCpLj9Gg58ecNgR7WTmWE0xOvY482bdzlaXHnXINgKZNm8a0adOKnW7QoEEMGjSo0PFhYWHMmzevHFN2lXMAlGYNAINTFZjVqQrMeCU4yb1aPaWWAKnDrt5sr5YAXQmArpRIFBUAOZ6qHMvLP60tK4OcjR9g/ftPAgB00FY5yH7ir6RVX2A+y987yfn1A5TL6QD4XNcGU4fBUOdq262D1vp04yCc3o01JN5lGbbMs+T8thxr0m57vvxD8YoZgCmqZ4lKSErCaNSzN68+Hb2PE3B+H2oA5NgOV7a9YrNg3vk15sT/2kuwdAZMrXrgFXs7+oBa5CVlkGUz4afPI1jJ5DwBOMc/5qPbyfrlA5RsezG+oUFbvOMGYwhveu1pN+jZkB2DWefNfnMEdVrH82Cr0jUeF+4ZDXrSbQFsyg7CrNhcqphF5TEa9OzNqs9NvrsJuXiIpsa69Dv7BeYzGQDo6zTBO+5ODA3albitiCcyGfTkYuLHrNaYLTYSqvnxZjLosWLgl+wozBYbbY1VW2VnNOq4pPiy+nICZouNjibPKxn07ErNCuBcBXbW6m8PgLIyUCxm1wDoSqMeR8Ncg0GH4UrAc3WYUxWYLY9u3gfIUOztlkrSBsgRdJjdlBZZ/t5Fzs/v2Ruv6Qyk+jYmPOsI1xtPAnGAzqXESrGYyf19JXkHNtrTExyBf49h1InuQkZGlkujuaNKPayKDsPFVLx90tR15x1LJOeXJWDOBr0Rr5gBeMUMKLfAxznf+/Lqo6DDN+sMN/rs4VBeJEkWe8mSUa/HdvEs2f97W21obWgYjU/X+9EHX60GNRj0nLcF4qdPI4RMIAC9XoeSl0Pqf5dwebd9W+hD6+HddSjG+q3LJe2Zih9rL9vbo/UxVK9ick9mzHd+Oc5BUbmMBj3HLXWwGX3wsmTzTNC3YAGdfy28O92FsXmXUjVe9lRFXc+ro/z5KerhuzK4u4d6Gu0FQE6BwEWLCfx9IC8H24UULNYrjSoNugJPnyaDXu13x3kYgO1SGhG/vso//DM4oZwE+l8NTop4qsh/gBoNOhRFwbx9LeYdXwL2RsY+vZ/gr52Z9Dw4k9qGS8yt9SEbc6JYk2UvTjSYM8n630JsZ48BOkztbsY7fjAmH1+3T2g2g4/a1iIy+xA6GtD0zDfk7Pw/+zojmuPba4S9BX4FMBr0XFZ8uOzfgIDLJxnkl0imzYfJGXdhQ49X2iGytixFyb0E3v74dL0fY/OEAnnR63ScswXSkDRCdZlAPQzZaVxcuxjr+ZOg0+PVvj9eHe+wv6ZZLml3TYOjtEqUXYHzQbZtlTAa7H3RZNdqiX+qvdo5KbAdrYaMrDbte0qisOt5dZU//UU9fFcGd/dQT6O9AMjplWGLDQyRLbCe/AvL3zuxWKMB+47Lf/AY1IbRCo6OxIwGPbYLyWStn4khJwOAethf8XeUNBmLeCupuXkftX1O811OO0CH0aCQ88syLAd/BcDUpg/ene+x9x9hMnPUEkGUyf6KdU+fA3yVFUttwyWUDTPsr3F7++Pb96liSzqMBh27zQ1paUqmUc4BHgw4Rt1Ue2NkU3Q/vDvdVaEN5hzbNj2oJQGXTwIQpM+hiTGV2vpL+Py6yf5Ji9qN8b356ULbHOl0cN5q7/G0Fhe5znAWnx/WYs29hME/BL+bR0Gd8u1GveBFxvNO6uoq/7b0xAumFjgCz7QGvcjKvMhXZxtSP+oGrq9BwQ+4ewCt3sdb/oCjqquQ3d1DPY2GAiDXxsmOv42NO9oDoOOJWCLbAvaiO28lh9amJPbm1Qd0mPRwffJXxIfsYV7mLaTaggm1pJL1xXyU7EwUnV79NouiKMWWAJl3fUu3zA3gBycstTlsiaD5oVVYMvaDTod3j+F4tbr6WRGjQcf/5bSguTFZ7XL8Jt/ddPU+CFlm9KH18L3lWfRBxfeVZDTo+SurIYPZRrjlDOFeYNMZ8OvzOKamna5l45aK40KTFNYJm9WC6fRO6hkzGOK3jfrGdFDA2KIrPj2GF1n9ZtDbS4AAWhhPExd0GF1uHoba11H/ny9y0epb7v3h5D+Jq/tF05Pk35aeeMHUAkfVxSX/+mwPv4c/T5/muhq4LwqU5npgFU1pFAjoqrgKuTo80HheiiqYcxsgq03B0CgG0GE7e0wtxfHXZdPuwGIeD/yROK9jgEJs2tfUTUskUJ9DV5+DhOsvEJ+0EiU7E31YI9J7/wsAk84KeTlF9mlj3v09uZv/o/6O8TrOMP/fCM7YDwYTvjeNdgl+wH4w/ZnXmOfT7+N/2fZA7Sbf3fjrzejDm+J3x6QSBT+OZaXZAsgLtH+GIlcx8neb4ZUS/DjWD5CLN0dq92JDdgyAPfgBaNUHn16PFtv2SK/XqSVAEYYL+OjyUMJbEnjnJIxBtSsk7aYadtH0JPm3pSdeMLXA8dBmtSpXS7Jr4HFe40qAClTPV21+CpSWe2CVdvXe49fAkq8LcZt3EIbIFgCEZezFhIWuqZ/gk2v/yGgn7yPc5LObhhd3qvPEeh3n8cAf8LJlo6/TBL+BE1D8w8hV7AVqSvYF9cKR/yk27/Bmcv9vJQApvva3kbr6HCbW+wSKzoDvLc9ibBxbIN2Ok9OKgZ3mRurwk9Yw/G4dV6q6eceJktG0H0n6+rydeRO5taJKPH9ZOdZvsdg7P9yfV4+cK9vup+zWGDvfW6K3SxxtgBwO54Wj9Bpl7zq/gtS0i6YncdcmTlQ+R7V9nlPnpDXxOK9px1vBEqCqzU/+Rs+eeAx5XooqypWCn7x8AZDFasPY1P4qeGTmX/zDfzMh5qs9m7YwJjPQbwcAJxv1J0cxEaLPprbhEjleofj2G4PO2x+9Tkemzd4Xhi3rglMPy1cPAuv5v8n5ZZl9eLtbSKx7D5dsV2/Wqa3vxdigjdvkO5+cJ61h7Mi9jn3meizL6VfqTvccB+LFsNZ8ZhrECWudSn1acKzfYrNfYPMwsvhiH1Zc6s667I6YjCV7s0qn15FuC2BX3nXsNDdi8cU+6L1K9lHYayUBUMWRbesZHNcCS40PgDyrxKSsCr5E4GElQB54DHleiiqQzabk/6TQlQCoE+j0hOSeoZP3URR0/B39GH9bwtBfaW9zKqwT6fV78JfZ3jHbZZsX+5sPQ+9r70lZp9Nx0Wa/+SrZmer3qBw7Xcm5RPZ388FqtvdH0/keDEYjm3ObY1Ng7eV4cusV/qkP14NHxweXb2DRpb5YDKW/4bte4K72fl1Z1ADIomCx2Nd/xBJBorkpOq5+c604Bp39K8jvX+rFsku9yMVU4nmvVYE3G6r5RdOTmIye/8SoBY5rgcWpCqwmHuc1LeD2tPxUhxI2zTSCVihY+gP2k1wfGIyhfmu1878jEX3RhbXkj9xmNDKeZ39eXdIa3EqIQc8X2dEYsPJzTmt6+l/tk0ang4vK1RIgi/VqJ4WKopD902KUi2fRBdbBt/cT6PT2N82+zO7AjzltuKz40KUEfQbldy0NRd0XcVfewalWgVltWPN9ztlg0Je4czVHr882p6i2ojtmy1+sXNEBl5YUbGAu27YqGJzOT8c1syYe554WMJSVpzXqrg7bVzMBELi2/zHodVhtV9/W8mrdh+yk3WzLbUpueDcaGnT8nhtFkrUWJy1h3Gs0YTToOWsLYvllewNl5wPMuQTIlpUB1FKnsRz6P/tnHAxGfG9+Gp1PwJVxehR0XFZ81N+FKWzctZTcODdyzKuCTrPUEiCrTf2Gl0P+UoCiuLsmF/cx1LKSEqCKU7DRpGzbqqCWAFlsakl2TTzOq0MJRWl42ltXBQMyzzuGPC9FFcj5lWgfL3s7E7UEpHEsa2o/wcrL3TAYDWpwctwSjhXDle+DFb5D9TrUNkDWy5nqcIP5EjmbVgHg1WEQhrBGbud399t1nPuT81p613S8HpnnVAJTVQFQ/lI5Qyle3XQX7FR4FVj+V7Wlt+JyU6DRpGzbKuEoibPYnEuAat6+qA436NLwtG4kqkOAWb33eCnlWa/e7K/ehK+WQFzGH9BhctMTtP37YIVH2HqdTq0CU7Iy1OHWrR9D7mX0YY3wat/Pdf4C6yiiCqyQJ7DyKwGqvIPTZLy67a3W/CVAZQuACvsafHkp8Kq2B77aWV1JCZBnuFoC5NwGqOYd59Whp+LSKNiNhGf1BO2JAaamqsCsTu1dnNuhOOQ5NVx293RQVKt2nQ4yHY2gc+wlQNebTmM9ugV0Onx6Plygd+UCT7xFXPALOzmv5SbhODHyLDastuI/2VHeDEW0ASpNIOYu2KnobzNWhzcbqquCRfg176ZbHTj6a6npb4HVtHO54AN1VZcAeX4Jm6YCINcA50o1kFO1mNXNeAejQe+mX4P8bYDsJUBkZwKK+vq8qU1fDHUaF0hPgROwiGLmwqq6rqWvB0fRqNVmI8/i+GRH5R2czm+Z2GyuVWClOUnylwDpdboKbwQtPUFXHE8rwtcqx7XA+S3RmnicF3U9r4487dokjaA9jKMNkMl4NcCxupQAXTnZje4CIF2xJUAXrzRmJieTGK8TNDCcB5MPXrG3uU1P1ZUAXQ3+SvLR1vJWdAlQKQKg/B9HrYQs1LSnRk8iPUF7Bse1IM9qU6+ZNfE4r2nnsqdVIUsbIA+j9s6s110tAXJqg6IGA3o3DZ6N+iIjbL1OR4bNnyybFzqbhX/627+s7tXuFrWvoPxK8/Xewp6Gr+WkdQQguXnWEq27vKkBmFMR+7WkI3/AU9Htf6DmPTV6kvwPFBX9Rp9wz1GqbP8URuW3EawsnlZiUlYFXyKo6tfgPb+jSc9LUQWyOL3S6ajndi4Bci4NcdeteIHvQBldq8Bs6NmYez0APjoL2Yo3XtG3FJqe0pyAhbWHuJaT1hGA5JidA6DKLAG6WvpWMAC69hIgXSWc8DXtqdGTOG9LKf2pOi4lQDW6DVDh1/PqyONLgDzwgabmHdVFsDi90un8KvjV8QXfEnNwXy3m+ho8wMbcVtgM9s9b/B8xRX6jK/8BW9RFv7AL0LU8mRndBECVecO5WgWnuLyF55y2ktDpdC6Nng2VUAJU4M0GD3yqqa6cb0A18YZbXVxtHuD0MdQaeJxXhzYqpeFp+fG0T3O443kpqkDq97mMV19zt7grAXLT3sdUSMNoB0fj28s2b9La3c8P2W34Ux9dZHpcG1EXXeRfeAB0Da/BX1lvTq6lROsub471W21lKwEC11KgyshD/qcYT3yqqa6cG+LXxCqX6sKx7V1KgGrgce5pAUNZeVr1fGke8KuK56WoAjn3euw4oS3u2gC5eePL4DYAcg1gHC6GteaL7I5gNBWZntIU+ZdrAGR0LQGq7APT+Q28srQBAtegpzKu0VICVHGct6Vs16rjfH6qHaXWwP1RHRrploZe5/odxaoO6NzdQz2N56WoArkEQO5KgJzGG/Q6nHefqZieoJ1fvzbnOb4EX/KgpriDo7CT85qqwK48aWebLSVad3lz7gnauXdu53El5RIAVUYJUCm6LhCl43wsy3atOleryC1Xh9XA/VEd+qkpLec8VHV+3N1DPU313+Ol4BzgOPdFo45X2wDZ+5Mx5DuYiurZ0rkqxhFolSaoKe7gKOwJrHxKgCr3wLzaCWXZ2gBBFVSBSU/QFcb5gaEmljhUF27bCNbA47w69FRcWi4PEVUccLi7h3oaz0tRBcpzauPj3BeNg/qW2JUdZcrXKDN/EaPJpQSo4HpKE9QUd8HPv241Dddwo1DbAF25wFX2zca59C1/FVhpL7TOm6QyXoOXzvoqjuvTa8274VYXpnzXB6iZx3lR1/Pqyvla7gn5yX8P9TSel6IKlOe2BKjgW2COk91dcaJzvabz33qXKjBrgfndcXniLUERs7veoK/l45/5i7gru3jbufQtfwlQaS+0hkquAnNen0Gvq5SgSys8qfheywz5rg81+Tgv7HpeXTlfyz0haPX0c9rzUlSBnAMgQ74AyGZTsCmOD/+5C4DsJ4fj5q3D9WboUgJUwt5TXYoHS1AK4y6iv5YSIMdyHG2VKrsEyHnbO7Z5/rSVlE7vPgitKDrd1U40PfGErs5ci+9l21YVxzVFvT7U4H1R2PW8unIpAfKAamRPL9Wt+i1UiZy/fO78xWNw7Q/IcSK4uyAbnP7v3PDZpRG0pWSBhcml0WfxB4e7iN5wDaU3Vd1jaFFVg6V9CqvsNkBw9bjwxBO6OvP0i6VW5A8EavK+KOx6Xl057ytPCOg8/aHG81JUgZx7es7fBsi5R+j8JUB6nU69uTpu3vnbqjjHIWbLlSqwYg7A0pcAlU8boKruMbSootnSlgC5BECVdL5LCVDF8PTicq3wtK+KV6TCrufVlUvXKh5UAuR8D/UkVb+FKpGjBMjkpg2Q8zfBrpYAXbnRuWnIlf+ioHPzFljxJUClu+C7m6YsPUGXZt3l6Vp6vC5MZbcBAvdVpKLsPK0Bp1ZV9fWhMtW0hxnn88YzSoAK3kM9Sc3Y6yXkqOYyGJx7grYHPlanN8QcwYxa1aEvGKgUCICc1+MIgIqpntLrr37KoSSBjPsA6FreAst/gavcg9M53/mVNj+V3QYI3FeRirIz1rAGqdVVVV8fKlNNC4Ccq+c9oUrP3T3Uk3hmqiqIcwnQ1Z6gHSVABRv8XY1e3QVArgeXTne106erjYuLPwBNpTgByy0Aypeuqjj5C3vCL3VP0M6vwVd2CZAHFDHXJPIxVM9QHb7iXV4Ku55XV54W0Lm7h3oSz0xVBbE4dVDo/MVjcP0QqoPjb5fGymqEXXDTOSLuPEvJXoN3nqZk0xY8ScvSE3T+NFSm8vq0R1VUgTkannvqU0115VoCJNu2qmipt/OirufVkacGQJ7YCzRoLADKc+ro0PmLx+DcS3Thb345D3N3gDmuE+YSvgbvvL5KrQIr0Mix8g/Owj/tUZZG0JVVAnRln3lovXZ1ZdDr1apRKQGqOgW/4l1zj3NPCxjKytPeUHV3D/UknpmqCuL8GrzzF4/B9UOoDvn7/oGrwUPRJUAFg6nCOJZXkgu+u2LEaylaLPA5h6ooASrs0x6lvNi6tgEqU5JK7OpTjaZOn0qhVgnX4Juup/OE60NlKep6Xh15Wn7c3UM9iWemqoI4BzlXS4CKCoAc0WvB/nrcBTeOp9eSdoRoX17JI2THul2++FuGnqAdqiI6d+mx1CU/ZSgBqrR+gDz7qaY6c9cLu6hcunyfiKjJx3lR1/PqyHH99JTzx9091JN4xlaqJC5fgzeUpg1Q6UqAzKVpA3QNJUA+XoYCw0ojf7qqugSoLPmpijZAUgJUcUw1rE1GdaWVLgk8rcSkrDwtP55+rfTMVFUQ1yowe9YdPUFbrO7aAF0pcSlpG6ACJUAlqAJzlOqUog2Qj5fRaf6yV4FVRXTuXHLlkp+yvAVWSW2Arh4XnvlUU51dLQGSbVuVjC4lQDV3X9S4NkAeVqLl7h7qSTwzVRUkz6UK7Mpr8LZ8VWBuvtDu/i2wggeY4wZcmm/oqOsoQcmHY50+3uVcAlQFryi6lAA55afUPUFXQT9Anv5UU52VplsIUXE0UwLkYY2Gy8roYV10uLuHepJy30p///03I0eOJC4ujri4OMaOHUtKSorLNJs2bWLw4MG0b9+efv36sX79epfxubm5TJ06lYSEBGJjYxk3bhxpaWllTptzD81XS4Bc+wFyPtndXYwdw9wFDWojaMeyShTUlPyCb1JLgJwDhtIfWAa9zqXjxqp+Dd6lCqwMAVBl9Xxa054aPYmnFeFrVWl7qa+uirqeV0ee9nDm6Q805Zoqs9nM8OHDsdlsrFq1ig8//JDU1FSeeOIJlCtf/T5y5AiPP/44PXr0YO3atdx9992MHz+eTZs2qcuZMmUKv/32G/Pnz2f58uUcPXqU0aNHlzl9zq+6Ow74qz1B2//vXNxrcNMeQf14npuqp6uNoK2FTpNfaZ5AHOv2dakyKv0u1Ol0+ar1qvY1eJ8y5Me51EdXaQFQzXpq9CSeVoSvVVV9fagsRV3PqyNPq553dw/1JMbiJym5M2fO0K5dOyZPnkytWrUAGD58OE899RTp6enUqlWL5cuXExUVxZgxYwBo1qwZe/fuZcmSJSQkJJCSksK6detYtGgRcXFxAMyePZt+/fqxY8cOYmNjrzl9zqU8BqeeoC1WG38dPa+Oc7j6Sq67EiB3b4G5VoGV5AN7ZS0ButYDy2TUcSVOq/ISIC+jfX9YbUrp2wBVxWvwUkpRYWTbegbXav+auy+Kup5XRyZPLQHy0BK2cg2ArrvuOubOnav+Pn36NP/5z39o06YNoaGhAGzfvp2+ffu6zNelSxdmzJiBoigkJiaqwxyaNGlCREQE27ZtK1sAdKUEyMvLgI+3PesWm8KajUfZui/VPs5kUHeW95WSCS+T3mmYocB0Do4bsNWmFDpNfo7leXuVfFp/X5M6zNfHWOh86tONm5PBy2ggO9da4nWXN2+nIM7LZMBk1GM1W/HxLjw/7hjyXaiNRn2R+S4PPqXYZ5WpovNdGbxN9m1bmuOgJuT7WlRkvr1MV8/PmnycF3U99zQlybe3t2flx909tLQqsmlnuQZAzh5++GF+//13goODWb58uVo6kpycTGRkpMu04eHhZGdnk56eTkpKCqGhoXh7exeYJjk5uUxpuhKXEBLsR5C/F2DvByjjklmd5o4bmhMa6g/ATV0a83fqJW7rcXXYLV2bcCYtmwHdm6nDHPI/KYWE+BWYJr87bmiOXq/nxvjrCA3xLXJax7oH9WpBRO0ALBYb9SKDi813UFDB5d5zUxQ/bv+bIH/vEq27vN1xQ3MuZVvUv1s3rc3pc5eIalq7VB/x8/G+Ggz6+Jhctre7fJeHW7s15XymmX7dmha7f6tCReW7Mtx5YwtCEk/SPbYBAX5epZq3Oue7LCoi33f1bsm6jYfxNhlr9HFe1PXcUxWV7xvjr+NQUiZ39PSM/Li7h3qSUgVASUlJ9OnTp9DxmzZtUqu+nn/+eZ555hnefvtthg8fzrp166hbty45OTl4eble2By/zWYz2dnZBcYDeHt7k5ubW5rkFmC78sbXxYvZGJQrjZ8tNrKy8wB47PbWNAjzJT39MgA+BnjqzrYA6rAALz2jBrsOc7jSzEl1+VJugWnya1Tbz74OxVbstM7rHtilkds0ODMY9AQF+ZKZma12+OjQvW0E3dtGXEl48esub41q+/HC0A7q7wZh9pM6IyOrVMuxOOrxAEuelfT0y0XmuzyE+BoLPQaqUkXnuzK0ahBEqwZtyMvNIz03r0Tz1IR8X4uKzHfbxiG0bRyn/q6px3lR13NPU5J8myh4z6pK7u6hpRUc7Iu+gtpolSoAioiIYMOGDYWODw6+Whpx/fXXA/DWW29x4403smbNGkaNGoW3tzdms9llPsdvX19ffHx8CowH+5thvr5li/gdAYrNpqhvQVltCuY8+01Uh05tKH0t8hdcKIpSpuWVF6vV5hHpqAj5y4qc81mT810Uybe2SL61RWv5zl+wUJ5KFQCZTCaaNWtW6PgzZ86wc+dO+vXrpw7z8/OjQYMGpKba29jUrVtX/dshNTUVPz8/AgMDiYyMJCMjA7PZ7FISlJqaSkRERGmSW4DtypbUoXOprsrJK3nPzUXJHwBVVr80WlYVr8ELIYSo/sq1XGn//v0888wzHD16VB2WmZnJsWPH1MApLi6OrVu3usy3efNmOnTogF6vp2PHjthsNrUxNMCxY8dISUkhPj6+TOlzjiRdAiCzIwAq2w00f9sVuSFXvKr4GrwQQojqr1wDoG7dutGqVSsmTJjA7t272bNnD6NHjyY0NJQhQ4YAMGzYMHbt2sWsWbM4cuQIy5Yt45tvvmHEiBGAvZptwIABTJo0iS1btrBr1y7Gjh1Lp06diImJKZd06nSubw/lmO2Nccvaaj5/ACT344rn2g9QFSZECCFEtVKutwwvLy/ee+89GjVqxIgRIxg2bBhBQUGsXLmSgIAAAFq0aMHChQv55ZdfGDRoEJ9++ikzZ84kISFBXc60adNISEhg1KhRPPLIIzRt2pR58+aVWzp1Oh16py8e51x5HfxavqzuLP/slfVxTi2TKjAhhBDXotxfgw8PD+fNN98scpqePXvSs2fPQsf7+fkxffp0pk+fXt7JA642nDVe6Xsm19EGqIwlQPmrYOSGXPGq4ltgQgghqj9NVho47pP5S3zK2numNIKufC5fg5eAUwghRAlpMgByRCr5S3zK2rNogTZAckOucFICJIQQ4lpoMgBSq8Dyda50LV9Wd1luvtkNckOucC5vgUnAKYQQooS0GQA5qsDylQCVtR+g/CUQckOueFXxMVQhhBDVn0YDIPudMn+JT9k7QswXAMkNucJJCZAQQohroc0A6Mr/87f5KXsJUL71yA25wrmUAMn2FkIIUULaDICu3Cfzv/UlPUFXPy5vgUmbKyGEECWkyQBIfQus3KvAXH/LDbniyVtgQgghroUmAyD1LTCngEev05W5CqVAI2i5IVc4g1SBCSGEuAbaDIAcb4E5BUBGY9lvngVKgOSGXOGkEbQQQohroc0AiIJVYPn7BLqm5RZ4Db7MixTFcN7mEv8IIYQoKU3eot31A1TW74CBm4+hShVYhZMqMCGEENdCkwGQg3OpT1l7gQbX0gidrmCJkCh/0ghaCCHEtdBkAKRz8y2wsn4HDORmXBXkY6hCCCGuhUYDIPv/ndsAlfVL8HD17TKQm3Fl0UnQKYQQ4hpoMwC68n/nt8AM5VwFJgFQ5XD+4Kx0PCmEEKKktBkAqR0hOrcBKo+3wK7+LaURlcM50JQ2V0IIIUpKkwEQbqrAytoLNOTrk0buxZXCtR+gKkyIEEKIakWTtwzHLdO51Kes3wGDfO1RJAKqFM7bWarAhBBClJQ2AyA3VWDlUwLk9LfcjCuFvHknhBDiWmg0ALL/v7yrwFx7JZabcWVwjjN1EnQKIYQoIW0GQFf+7/otMGkEXR3pJegUQghxDbQZALmrAiuH0gMd0h6lskkbICGEENdCkwGQQ7l/C8xpEVIdUzmkDZAQQohrockASG0DpK/INkBlXpwoAeegR6fJo1kIIcS10OQtw923wMrjNXjnoEeqYyqHVIEJIYS4FtoMgK78v7xfg5e3wCqfXhqeCyGEuAbaDIAq4TV4aQNUOaQNkBBCiGuh0QDIXUeI5VsFJjfjyqGX3reFEEJcA00GQA7l/zFUaY9S2Vy/BSbbXAghRMloMgByVwVmKPevwZd5caIE5AO0QgghroU2A6ArzaCdS31M5dEPkJRGVDppAySEEOJaaDMAunKfdC71KY8qK5cSIAmAKoW0ARJCCHEtNBoAXSkBMlZgCZCURlQKvQSdQgghroEmAyAHQ0X2BC0340ohVWBCCCGuheYCIOd7pKmce4KWr8FXPoMEnUIIIa5BhQZA27dv5/rrr2fLli0uwzdt2sTgwYNp3749/fr1Y/369S7jc3NzmTp1KgkJCcTGxjJu3DjS0tLKJU2FfbFdSoCqJykBEkIIcS0qLAC6ePEi48ePx2azuQw/cuQIjz/+OD169GDt2rXcfffdjB8/nk2bNqnTTJkyhd9++4358+ezfPlyjh49yujRo8slXc73SJ1Op5b8lEcA5NoRYpkXJ0rANeiswoQIIYSoVowVteApU6bQsGFDTp065TJ8+fLlREVFMWbMGACaNWvG3r17WbJkCQkJCaSkpLBu3ToWLVpEXFwcALNnz6Zfv37s2LGD2NjYMqUrfyGB0aDHYrWWUwAkJUCVzSAfQxVCCHENKuSZ+b///S87duzgxRdfLDBu+/btJCQkuAzr0qULiYmJKIpCYmKiOsyhSZMmREREsG3btnJInetN0hH4lHsbILkZVwrn7ayTKjAhhBAlVO4lQElJScyYMYOFCxfi7+9fYHxycjKRkZEuw8LDw8nOziY9PZ2UlBRCQ0Px9vYuME1ycnKZ06fTgdGp8XOP9vU4nJRBo8jAMpcCufYrpHdZT1VwpKc8ern2VMEBXnRoWQcfLwM+3vbDWQv5dkfyLfnWAsm3tvJdkc+1pQqAkpKS6NOnT6Hjf//9d55//nnuuece4uLiSEpKKjBNTk4OXl5eLsMcv81mM9nZ2QXGA3h7e5Obm1ua5Lql1+sIDb0amD15d0yZl+ng53c13b4+Jpf1VKWgIN+qTkKFmvp4V7fDa3q+CyP51hbJt7ZoNd8VoVQBUEREBBs2bCh0/Mcff0x2djZPP/10odN4e3tjNptdhjl++/r64uPjU2A82N8M8/Uthx2vQHr65bIvx43cnDz1b4vFWmHrKSmDQU9QkC+ZmdlYrbbiZ6ghJN+Sby2QfEu+tSA42Bd9Bb3hUqoAyGQy0axZs0LHr127ltTUVDp37gyAoigAPProowwaNIhXXnmFunXrkpqa6jJfamoqfn5+BAYGEhkZSUZGBmaz2aUkKDU1lYiIiNIk1z0dWCwVc/Bcya6qotZTWlarzWPSUpkk39oi+dYWybc25L+vlqdybQP04YcfYrFY1N8pKSkMGzaM6dOn061bNwDi4uLYunWry3ybN2+mQ4cO6PV6OnbsiM1mIzExUW0sfezYMVJSUoiPjy9zGiuybbK8Bi+EEEJUD+UaANWvX9/lt8FgAOxVZ2FhYQAMGzaMO++8k1mzZnHnnXfyyy+/8M0337BkyRJ12gEDBjBp0iReffVVfH19mTx5Mp06dSImJqYcUllxkYl0hCiEEEJUD5XenLxFixYsXLiQX375hUGDBvHpp58yc+ZMl1fjp02bRkJCAqNGjeKRRx6hadOmzJs3r1zWX5FhiXwKQwghhKgeKqwjRIAGDRpw4MCBAsN79uxJz549C53Pz8+P6dOnM3369HJPU0XGJVICJIQQQlQP2upQgIrtLE8vvRILIYQQ1YLmAqCKlP87Y0IIIYTwTJoLgCr2LTDnL5NX3HqEEEIIUTaaC4AqshGQ85KlDZAQQgjhuTQXAFXsW2DSBkgIIYSoDjQXAFVkBOTcW7e8Bi+EEEJ4Ls0FQBUZmDiXAEkjaCGEEMJzaS4AqkjOMY9UgQkhhBCeS3MBUEUWzOilI0QhhBCiWtBeAFRZ3wKT+EcIIYTwWNoLgCryUxhOf0sJkBBCCOG5NBcAVSTnoEfeAhNCCCE8l+YCoIp9C8xpPVICJIQQQngszQVAFdkPkGsbIAmAhBBCCE+luQCoIsMSvZQACSGEENWC9gKgSuoIUQIgIYQQwnNpMACquGXL1+CFEEKI6kFzAVBFkkbQQgghRPWguQCosr4FJo2ghRBCCM+luQCoQr8GLyVAQgghRLWguQCoIsMSaQQthBBCVA/aC4AqsGpKL1VgQgghRLWgvQCoIpftXAUm8Y8QQgjhsTQXAFVaT9ASAQkhhBAeS3MBkK4CIyB5DV4IIYSoHrQXAFVaR4gSAAkhhBCeSgKgClq2BEBCCCGE59JcAFSRjYCcgx6DVIEJIYQQHktzAVBFxiXOhT46CYCEEEIIj6W5AKjS3gKT+EcIIYTwWJoLgCrrLTCpAhNCCCE8l/YCoIp8C0wvb4EJIYQQ1YH2AqCKXLZ0hCiEEEJUC5oLgCqyCMh5Y0oAJIQQQnguzQVAFRqXSD9AQgghRLWguQCoIjk3sJYSICGEEMJzaS4A0lVgyYyCov4t8Y8QQgjhuco9AEpMTCQqKqrAvy1btqjTbNq0icGDB9O+fXv69evH+vXrXZaRm5vL1KlTSUhIIDY2lnHjxpGWllYu6avIminlavwjJUBCCCGEBzOW9wIPHDhAo0aNWLVqlcvw4OBgAI4cOcLjjz/OQw89xMyZM/n5558ZP348tWrVIiEhAYApU6awfft25s+fj5eXF5MnT2b06NGsXLmyzOmryLDEKf6RNkBCCCGEByv3AOjgwYM0b96cOnXquB2/fPlyoqKiGDNmDADNmjVj7969LFmyhISEBFJSUli3bh2LFi0iLi4OgNmzZ9OvXz927NhBbGxs2RJYSUVAEv8IIYQQnqtCSoA6duxY6Pjt27fTt29fl2FdunRhxowZKIpCYmKiOsyhSZMmREREsG3btjIHQAa9DqOxYpo+RYb5U6+2Pz5eBny8jRXa3qgkDAa9y/+1QvIt+dYCybfkWwsq8jZa7gHQoUOHCA0NZfDgwaSkpNCyZUvGjBlDdHQ0AMnJyURGRrrMEx4eTnZ2Nunp6aSkpBAaGoq3t3eBaZKTk8ucPpPJQGiof5mXU5iFE/qgw7PaAAUF+VZ1EqqE5FtbJN/aIvkWZVWqACgpKYk+ffoUOv7nn3/m4sWLZGVlMWnSJAwGAytXrmTo0KGsXbuW5s2bk5OTg5eXl8t8jt9ms5ns7OwC4wG8vb3Jzc0tTXLdslispKdfLvNyqgODQU9QkC+ZmdlYrbaqTk6lkXxLvrVA8i351oLgYF/0+oop9SpVABQREcGGDRsKHR8eHs62bdvw9fXFZDIB0K5dO/bu3cuHH37I1KlT8fb2xmw2u8zn+O3r64uPj0+B8WB/M8zXtxwiXwUsFu0cPABWq01zeQbJt9ZIvrVF8q0Nzm9Xl7dSBUAmk4lmzZoVOU1QUJDLb71eT7NmzUhJSQGgbt26pKamukyTmpqKn58fgYGBREZGkpGRgdlsdikJSk1NJSIiojTJdauq2+UIIYQQouqVa7nSxo0biY2N5eTJk+owi8XC/v37ad68OQBxcXFs3brVZb7NmzfToUMH9Ho9HTt2xGazqY2hAY4dO0ZKSgrx8fHlmVwhhBBCaFS5BkAdOnQgNDSUCRMmsHv3bg4cOMCECRPIyMhg+PDhAAwbNoxdu3Yxa9Ysjhw5wrJly/jmm28YMWIEYK9mGzBgAJMmTWLLli3s2rWLsWPH0qlTJ2JiYsqcRikAEkIIIUS5BkABAQF88MEH1K5dm0ceeYR77rmHjIwMVq5cSe3atQFo0aIFCxcu5JdffmHQoEF8+umnzJw5U+0EEWDatGkkJCQwatQoHnnkEZo2bcq8efPKJY1SBSaEEEIInaJUZBMjz5F8/jKPvvo9HVvW4anB7ao6OZXCaNQTGupPevplTTWak3xLvrVA8i351oJatfwrrO8jbfWoBBX7LQwhhBBCVAuaC4CkCkwIIYQQ2guAqjoBQgghhKhy2guAJAISQgghNE+DAZBEQEIIIYTWaS8AquoECCGEEKLKaS4AkghICCGEEJoLgHQSAQkhhBCap70ASOIfIYQQQvO0FwBVdQKEEEIIUeW0FwBJEZAQQgiheZoLgKQISAghhBCaC4Ak/hFCCCGE9gIgqQITQgghNE+DAVBVp0AIIYQQVU17AVBVJ0AIIYQQVU5zAZAUAQkhhBBCcwGQxD9CCCGE0F4AVNUJEEIIIUSV02AAJCGQEEIIoXXaC4Ak/hFCCCE0T3MBkBQACSGEEEJzAZBUgQkhhBBCewGQxD9CCCGE5kkAJIQQQgjN0V4AJFVgQgghhOZpLwCS+EcIIYTQPM0FQFIAJIQQQgjNBUBSBSaEEEII7QVAEv8IIYQQmicBkBBCCCE0R3MBkDQCEkIIIYTmAiC9xD9CCCGE5mkuABJCCCGE0FwApJNGQEIIIYTmaTAAquoUCCGEEKKqVUgAtHTpUvr06UN0dDSDBw9m8+bNLuP37dvH0KFDiYmJoXfv3qxYscJlvM1mY968efTo0YOYmBgeffRRTp48WS5pk/hHCCGEEOUeAC1cuJAFCxYwbtw4vvjiC2JiYhg5cqQawKSnp/PQQw/RqFEj1qxZw1NPPcWsWbNYs2aNyzJWrVrFtGnT+Pjjj7HZbIwYMQKz2Vz2BEoRkBBCCKF55RoAZWVl8d577/Hcc8/Rv39/GjduzEsvvUTDhg1JTEwE4JNPPsFkMvHKK6/QrFkzhgwZwvDhw1m8eDEAZrOZZcuWMXr0aHr16kWrVq2YM2cOycnJfPfdd2VOo4Q/QgghhCjXACgxMZHs7GwGDBigDjMYDHzxxRcMGjQIgO3bt9OpUyeMRqM6TZcuXTh+/Djnzp1j//79XL58mYSEBHV8UFAQrVu3Ztu2bWVOoxQACSGEEMJY/CQld+zYMYKDgzlw4ABvvfUWx48fp3nz5owZM4YOHToAkJycTMuWLV3mCw8PB+DMmTMkJycDULdu3QLTOMaVhcGgx2jURttvg0Hv8n+tkHxLvrVA8i351oKKLLQoVQCUlJREnz59Ch3/zDPPkJOTw8svv8y4ceOoV68eq1ev5sEHH2TdunU0a9aMnJwcvLy8XObz9vYGIDc3l+zsbAC301y4cKE0yXXLz9eL0FD/Mi+nOgkK8q3qJFQJybe2SL61RfItyqpUAVBERAQbNmwodPwPP/xATk4OL774IjfccAMAbdq0YceOHaxcuZLJkyfj4+NToDFzbm4uAH5+fvj4+AD2tkCOvx3T+PqWfcfn5OSRnn65zMupDgwGPUFBvmRmZmO12qo6OZVG8i351gLJt+RbC4KDfdHrK6bUq1QBkMlkolmzZoWO37NnDwBRUVHqMJ1OR7NmzUhKSgIgMjKS1NRUl/kcvyMiIrBYLOqwRo0auUzjvNxrZbMpWCzaOXgArFab5vIMkm+tkXxri+RbGxSl4pZdrmFVXFwcOp2OP//8Ux2mKAqHDx/muuuuAyA+Pp7ExESsVqs6zebNm2nSpAlhYWG0atWKgIAAtmzZoo7PzMxk7969xMfHlzmN0ghaCCGEEOXaCLpevXoMGTKE6dOn4+vrS6NGjfjwww9JSkrivvvuA2DIkCEsWbKEl156iREjRrBr1y4++OADpk6dCtjb/gwdOpRZs2ZRq1Yt6tevz8yZM4mMjOTmm28ucxrlUxhCCCGEKNcACGDKlCksWLCASZMmceHCBVq3bs2yZcto2rQpAGFhYSxZsoQZM2Zw5513UqdOHcaPH8+dd96pLmP06NFYLBYmTZpETk4O8fHxLF26FJPJVOb0SfgjhBBCCJ2iVGQNm+dIPn+ZR1/9nn/2bcFNcQ2rOjmVwmjUExrqT3r6ZU3VGUu+Jd9aIPmWfGtBrVr+Ffbqv7Y6FAD0UgUmhBBCaJ7mAiAhhBBCCM0FQFIAJIQQQggNBkASAQkhhBBap70AqKoTIIQQQogqp7kASCIgIYQQQmguAJK3wIQQQgihuQBICCGEEEJzAZCU/wghhBBCcwGQREBCCCGE0FwAJG2AhBBCCKG5AEgIIYQQQnMBkBQACSGEEEKDAZBEQEIIIYTWaS8AquoECCGEEKLKaS4AkghICCGEEJoLgOQtMCGEEEJoLgASQgghhNBcACSNoIUQQgihvQCoqhMghBBCiCqnvQBIIiAhhBBC8zQXAEkZkBBCCCE0FwDpJf4RQgghNE9zAZAUAAkhhBBCcwGQTiIgIYQQQvO0FwBJ/COEEEJongRAQgghhNAczQVA0ghICCGEEJoLgOQtMCGEEEJoLgCSAiAhhBBCaC4AkrfAhBBCCKG5AEjiHyGEEEJoLgDSXIaFEEIIUYD24gF5D14IIYTQPM0FQBL+CCGEEEJ7AZBEQEIIIYTmaTAAkghICCGE0LpyDYDWrl1LVFSU238PPPCAOt2+ffsYOnQoMTEx9O7dmxUrVrgsx2azMW/ePHr06EFMTAyPPvooJ0+eLM+kCiGEEELDyjUA6t+/P7/99pvLv0mTJmEwGHjiiScASE9P56GHHqJRo0asWbOGp556ilmzZrFmzRp1OQsXLmTVqlVMmzaNjz/+GJvNxogRIzCbzWVOo15KgIQQQgjNM5bnwnx8fPDx8VF/JycnM3fuXJ588km6du0KwCeffILJZOKVV17BaDTSrFkzTpw4weLFixkyZAhms5lly5bx3HPP0atXLwDmzJlDjx49+O677xg4cGDZEinxjxBCCKF5FdoGaObMmYSHh/PYY4+pw7Zv306nTp0wGq/GXl26dOH48eOcO3eO/fv3c/nyZRISEtTxQUFBtG7dmm3btpU5TRL/CCGEEKJcS4CcHThwgK+++oq3334bLy8vdXhycjItW7Z0mTY8PByAM2fOkJycDEDdunULTOMYVxZGox6jURttvw0Gvcv/tULyLfnWAsm35FsLKrLVSqkCoKSkJPr06VPo+E2bNlGrVi0APvjgA6KiogpMn5OT4xIQAXh7ewOQm5tLdnY2gNtpLly4UJrkuhUU5EtoqH+Zl1OdBAX5VnUSqoTkW1sk39oi+RZlVaoAKCIigg0bNhQ6Pjg4GLAHOd988w3PP/98gdfOfXx8CjRmzs3NBcDPz09tQ2Q2m13aE+Xm5uLrW/Ydf/FiDunpl8u8nOrAYNATFORLZmY2VqutqpNTaSTfkm8tkHxLvrUgONgXvb5iSr1KFQCZTCaaNWtW7HS///47eXl53HrrrQXGRUZGkpqa6jLM8TsiIgKLxaIOa9Sokcs0UVFRpUmuW4pNwWLRzsEDYLXaNJdnkHxrjeRbWyTf2qAoFbfsCgmrtm/fTqtWrQgNDS0wLj4+nsTERKxWqzps8+bNNGnShLCwMFq1akVAQABbtmxRx2dmZrJ3717i4+MrIrlCCCGE0JgKCYD27t1Lq1at3I4bMmQIly5d4qWXXuLw4cOsXbuWDz74gMcffxywt/0ZOnQos2bN4ocffmD//v2MGTOGyMhIbr755jKnTboBEkIIIUSFvAV29uxZ2rVr53ZcWFgYS5YsYcaMGdx5553UqVOH8ePHc+edd6rTjB49GovFwqRJk8jJySE+Pp6lS5diMpnKnDb5FIYQQgghKiQAKqqhNEB0dDSrV68udLzBYOD555/n+eefL++kST9AQgghhNDex1AlAhJCCCGE5gIgiX+EEEIIob0ASNoACSGEEJqnwQCoqlMghBBCiKqmuQBICCGEEEJzAZBeioCEEEIIzdNcACStoIUQQgihuQBI4h8hhBBCaC8AkiowIYQQQvO0FwBVdQKEEEIIUeU0FwBJBCSEEEIIzQVA8haYEEIIITQXAAkhhBBCaC4AkkbQQgghhNBgAFTVKRBCCCFEVdNeAFTVCRBCCCFEldNcACRFQEIIIYTQXAAk8Y8QQgghtBcAVXUChBBCCFHltBcASRGQEEIIoXkaDICqOgVCCCGEqGraC4CqOgFCCCGEqHKaC4AkBBJCCCGE5gIgqQITQgghhARAQgghhNAc7QVAUgUmhBBCaJ7mAiCJf4QQQgihuQBILwGQEEIIoXmaC4CkCEgIIYQQmguApBG0EEIIISQAEkIIIYTmaC8AkiowIYQQQvM0FwBJ/COEEEIIzQVAEv8IIYQQQnsBkDQCEkIIITRPUwGQhD5CCCGEAK0FQFL6I4QQQggqIAC6fPkyU6dOpXv37sTFxfHoo49y5MgRl2k2bdrE4MGDad++Pf369WP9+vUu43Nzc5k6dSoJCQnExsYybtw40tLSypw2iX+EEEIIARUQAE2bNo0tW7Ywb948Vq9ejcFgYMSIEeTm5gJw5MgRHn/8cXr06MHatWu5++67GT9+PJs2bVKXMWXKFH777Tfmz5/P8uXLOXr0KKNHjy5z2vx8jGVehhBCCCGqv3IPgL7//nv++c9/0qFDB5o1a8azzz7L6dOnOXz4MADLly8nKiqKMWPG0KxZMx555BH69evHkiVLAEhJSWHdunVMmjSJuLg4oqOjmT17Ntu2bWPHjh3XnK7QQG+evbt9ueRRCCGEENVbuReJhIWFsWHDBvr3709gYCCfffYZISEhNGrUCIDt27fTt29fl3m6dOnCjBkzUBSFxMREdZhDkyZNiIiIYNu2bcTGxl5Tury9jLRqXAur1XaNOat+DAa9y/+1QvIt+dYCybfkWwsqsulKuQdAM2bMYPz48XTt2hWDwYCfnx/Lli0jMDAQgOTkZCIjI13mCQ8PJzs7m/T0dFJSUggNDcXb27vANMnJyWVKW1CQb5nmr64k39oi+dYWybe2aDXfFaFUAVBSUhJ9+vQpdPymTZs4cOAADRs2ZPr06fj5+fHee+8xatQoVq9eTd26dcnJycHLy8tlPsdvs9lMdnZ2gfEA3t7eajuia5WZma25EqCgIF/Jt0ZIviXfWiD51la+g4N90esrptSrVAFQREQEGzZsKHT8sWPHmDZtGj/++CP16tUD4K233uLWW29l2bJlvPTSS3h7e2M2m13mc/z29fXFx8enwHiwvxnm61u2yNdqtWGxaOfAcZB8a4vkW1sk39qitXwrSsUtu1QBkMlkolmzZoWOX7p0KWFhYWrw45indevWnDhxAoC6deuSmprqMl9qaip+fn4EBgYSGRlJRkYGZrPZpSQoNTWViIiI0iRXCCGEEMKtci1XioyMJD093SXAsdlsHD58mMaNGwMQFxfH1q1bXebbvHkzHTp0QK/X07FjR2w2m9oYGuwlSykpKcTHx5dncoUQQgihUeUaAN144400bNiQ0aNHs3PnTo4cOcK//vUvzpw5wwMPPADAsGHD2LVrF7NmzeLIkSMsW7aMb775hhEjRgD2arYBAwYwadIktmzZwq5duxg7diydOnUiJiamPJMrhBBCCI3SKUr51rClpKTwxhtvsGXLFnJzc2nXrh3jx4+nVatW6jQbN25k5syZHD9+nAYNGvD000/Tv39/dXxWVhavvvoq3377LQA9e/Zk0qRJhIaGlilt6emXNVV3ajTqCQ31l3xrhORb8q0Fkm9t5btWLf8Ke/W/3AMgT6a1A0erJ4zkW/KtBZJvybcWVGQApK0elYQQQgghkABICCGEEBokAZAQQgghNEcCICGEEEJojgRAQgghhNAcTb0FpqXvpzgYDHrJt4ZIvrVF8q0tWsy3Xq9DV0GfhNdUACSEEEIIAVIFJoQQQggNkgBICCGEEJojAZAQQgghNEcCICGEEEJojgRAQgghhNAcCYCEEEIIoTkSAAkhhBBCcyQAEkIIIYTmSAAkhBBCCM2RAEgIIYQQmiMBkBBCCCE0RwIgIYQQQmiOBEBCCCGE0JwaHQDZbDbmzZtHjx49iImJ4dFHH+XkyZNVnaxyl5KSQlRUVIF/a9euBWDfvn0MHTqUmJgYevfuzYoVK6o4xWX37rvvMmzYMJdhxeWzJhwP7vI9adKkAvu+d+/e6vjqmO+MjAxefvllevbsSYcOHfjnP//J9u3b1fGbNm1i8ODBtG/fnn79+rF+/XqX+XNzc5k6dSoJCQnExsYybtw40tLSKjsbpVZcvh966KEC+9r5eKiu+QY4f/48zz//PF26dCE2NpbHHnuMI0eOqONr6vldXL5r4vnt7NixY8TGxqr3K6jEfa3UYPPnz1c6d+6s/PTTT8q+ffuUhx9+WLn55puV3Nzcqk5aufr555+Vdu3aKSkpKUpqaqr6Lzs7W0lLS1M6d+6svPDCC8rhw4eVzz77TGnXrp3y2WefVXWyr9nKlSuVVq1aKUOHDlWHlSSf1f14cJdvRVGUu+66S5k9e7bLvj9//rw6vjrm+6GHHlIGDhyobNu2TTl69KgydepUJTo6Wjly5Ihy+PBhpV27dsrs2bOVw4cPK0uWLFFat26t/N///Z86/8SJE5W+ffsq27ZtU3bu3KkMGjRIuf/++6swRyVTVL4VRVESEhKUVatWuezr9PR0df7qmm9FUZR77rlHufvuu5WdO3cqhw8fVp5++mmle/fuSlZWVo0+v4vKt6LUzPPbwWw2K4MHD1ZatmyprFmzRlGUyr2W19gAKDc3V4mNjVU++ugjddiFCxeU6Oho5csvv6zClJW/xYsXK7fddpvbcYsWLVK6d++u5OXlqcPefPNN5eabb66s5JWb5ORk5fHHH1diYmKUfv36uQQCxeWzOh8PReXbZrMpMTExynfffed23uqY7+PHjystW7ZUtm/frg6z2WxK3759lbfeekv517/+pdx1110u84wdO1Z5+OGHFUWxb69WrVopP//8szr+6NGjSsuWLZU//vijcjJxDYrL97lz55SWLVsqe/bscTt/dc23oihKRkaGMnbsWOXAgQPqsH379iktW7ZUdu7cWWPP7+LyXRPPb2dvvvmm8sADD7gEQJW5r2tsFdj+/fu5fPkyCQkJ6rCgoCBat27Ntm3bqjBl5e/AgQM0a9bM7bjt27fTqVMnjEajOqxLly4cP36cc+fOVVYSy8WePXswmUx88cUXtG/f3mVccfmszsdDUfn++++/ycrKomnTpm7nrY75Dg0NZfHixbRr104dptPp0Ol0ZGZmsn37dpf8gH1fJyYmoigKiYmJ6jCHJk2aEBER4bF5huLzfeDAAXQ6HU2aNHE7f3XNN0BwcDBvvvkmLVu2BCAtLY0PPviAyMhImjdvXmPP7+LyXRPPb4dt27axevVq/v3vf7sMr8x9bSx+kuopOTkZgLp167oMDw8PV8fVFAcPHiQ0NJT777+fY8eOcd111zFy5Eh69uxJcnKyenI5hIeHA3DmzBlq165dFUm+Jr1793ap+3ZWXD6r8/FQVL4PHjwIwIcffsjGjRvR6/X07NmTMWPGEBgYWC3zHRQUxA033OAy7Ntvv+XEiRO8+OKLfP7550RGRrqMDw8PJzs7m/T0dFJSUggNDcXb27vANJ6aZyg+3wcPHiQwMJBXXnmF33//HT8/P/r168eTTz6Jl5dXtc13fv/617/45JNP8PLy4p133sHPz69Gn98O7vJdE89vgMzMTMaPH8+kSZMKpL0y93WNLQHKzs4GwMvLy2W4t7c3ubm5VZGkCmGxWDh69CgXLlzg6aefZvHixcTExPDYY4+xadMmcnJy3G4DoEZth+LyWVOPh4MHD6LX6wkPD2fRokVMnDiR3377jSeffBKbzVYj8v3HH3/wwgsvcPPNN9OrVy+3+9rx22w2k52dXWA8VK88Q8F8Hzx4kNzcXKKjo1myZAkjR47k008/ZdKkSQA1Jt8PPvgga9asYeDAgTz11FPs2bNHE+e3u3zX1PN7ypQpxMbGcttttxUYV5n7usaWAPn4+AD2C6Ljb7BvQF9f36pKVrkzGo1s2bIFg8Gg5rNt27YcOnSIpUuX4uPjg9lsdpnHcZD4+flVenorSnH5rKnHw8iRI7nvvvsIDQ0FoGXLltSpU4d//OMf/PXXX9U+399//z3PPfccHTp0YNasWYD9Qpd/Xzt++/r6uj0WoPrkGdzn+5VXXmHChAkEBwcD9n1tMpkYM2YM48ePrxH5BmjevDkAM2bMYOfOnaxcuVIT57e7fM+YMaPGnd/r1q1j+/btfPnll27HV+a+rrElQI7isdTUVJfhqampREREVEWSKoy/v7/LgQDQokULUlJSiIyMdLsNgBq1HYrLZ009HvR6vXpxdGjRogVgL0quzvleuXIlTz/9NDfeeCOLFi1SnwLr1q3rNj9+fn4EBgYSGRlJRkZGgYtodcgzFJ5vo9GoBj8Ozvu6Ouc7LS2N9evXY7FY1GF6vZ7mzZuTmppaY8/v4vJdE8/vNWvWcP78eXr16kVsbCyxsbEATJ48mREjRlTqvq6xAVCrVq0ICAhgy5Yt6rDMzEz27t1LfHx8FaasfB06dIgOHTq45BNg9+7dNG/enPj4eBITE7Fareq4zZs306RJE8LCwio7uRWmuHzW1ONh/PjxDB8+3GXYX3/9BdifKKtrvletWsW0adO4//77mT17tktxd1xcHFu3bnWZfvPmzXTo0AG9Xk/Hjh2x2Wxqo2Cw9zWSkpLi0XmGovM9bNgwXnjhBZfp//rrL0wmE40bN67W+T537hxjx45l06ZN6rC8vDz27t1Ls2bNauz5XVy+a+L5PWvWLDZs2MC6devUfwCjR49mxowZlbuvy/4im+eaPXu20qlTJ+X777936SvAbDZXddLKjdVqVYYMGaL0799f2bZtm3L48GHl1VdfVdq2bascOHBAOXfunBIfH69MmDBBOXTokLJmzRqlXbt2ytq1a6s66WUyYcIEl9fBS5LPmnA85M/3999/r7Rs2VKZP3++cuLECeXnn39WevfurYwdO1adprrl++jRo0qbNm2Up556yqXvk9TUVCUzM1M5ePCg0qZNG2XmzJnK4cOHlaVLlxboB2js2LFK7969lc2bN6v94eTvP8nTFJfvDz/8ULn++uuVVatWKX///beyfv16pXPnzsrs2bPVZVTHfDuMGDFCufnmm5WtW7cqBw4cUMaOHavEx8crp06dqtHnd1H5ronntzvOr8FX5r6u0QGQxWJR3njjDaVLly5KTEyM8uijjyonT56s6mSVu7NnzyoTJ05UunXrprRr10655557lG3btqnjd+7cqfzjH/9Q2rZtq9x4443Khx9+WIWpLR/5AwFFKT6fNeF4cJfvDRs2KIMGDVKio6OVbt26Kf/+97+VnJwcdXx1y/c777yjtGzZ0u2/CRMmKIqiKL/88osycOBApW3btkq/fv2U9evXuyzj8uXLyksvvaTExcUpcXFxytixY5W0tLSqyE6JlSTfK1euVG699Vb1GH/nnXcUq9WqLqM65tshMzNTmTx5stKtWzclOjpaefjhh5WDBw+q42vq+V1cvmva+e2OcwCkKJW3r3WKoijlUawlhBBCCFFd1Ng2QEIIIYQQhZEASAghhBCaIwGQEEIIITRHAiAhhBBCaI4EQEIIIYTQHAmAhBBCCKE5EgAJIYQQQnMkABJCCCGE5kgAJISoFpKSkoiKimLt2rVlXtbEiRPp3bt3OaRKCFFdGas6AUIIURLh4eGsXr2aRo0aVXVShBA1gARAQohqwcvLi5iYmKpOhhCihpAqMCFEufj0008ZMGAAbdu2pVevXsyfPx+r1QrYq5yGDRvGZ599xo033khsbCwPPvgg+/fvV+e32WzMmTOH3r1707ZtW3r37s2bb75JXl4e4L4K7Pjx44wePZpu3boRExPDsGHDSExMdEnXhQsXeOGFF+jUqRPx8fHMnDkTm81WIP3ff/89gwcPpl27dnTr1o3p06eTlZWljs/JyWHKlCn07NmTtm3b0q9fP5YuXVqu21AIUXmkBEgIUWbvvvsuc+bMYejQobzwwgvs27eP+fPnc+bMGV599VUA9u3bx9GjRxk7dizBwcHMmzePoUOHsmHDBsLDw3nvvff4z3/+w4QJE2jYsCE7d+5kzpw5mEwmRo8eXWCdhw8f5h//+AeNGzdm0qRJmEwmVqxYwYMPPsiyZcvo1KkTNpuNESNGcOrUKSZMmEBISAhLlizhr7/+Ijw8XF3Wl19+yXPPPcdtt93Gs88+y6lTp5gzZw6HDx/m/fffR6fT8eqrr/Lbb78xYcIEateuzcaNG3njjTcICQlhyJAhlbathRDlQwIgIUSZXLx4kYULF3LPPfcwadIkALp3705ISAiTJk3ioYceUqdbtGgRcXFxAERHR9O3b19WrFjBc889x9atW2nbtq0aTHTq1AlfX18CAwPdrnfBggV4eXmxYsUKAgICAOjVqxcDBw7kjTfe4LPPPmPjxo3s2rWL9957j549ewKQkJDg0gBaURRmzZpFjx49mDVrljq8cePGDB8+nF9++YVevXqxdetWunXrxoABAwDo3Lkzfn5+hIWFlefmFEJUEqkCE0KUyY4dO8jJyaF3795YLBb1nyPI+P333wFo0KCBGvyAvVFzbGws27ZtA+wBxe+//859993HkiVLOHz4MEOHDuWOO+5wu96tW7dy4403qsEPgNFoZMCAAezevZvLly+zfft2TCYTPXr0UKfx8/PjhhtuUH8fPXqU5OTkAumPj48nICBATX/nzp355JNPePTRR1m5ciUnT57kqaeeolevXuWzIYUQlUpKgIQQZZKRkQHAY4895nZ8amoqABEREQXGhYWFsWfPHgBGjBiBv78/a9asYdasWcycOZMWLVowadIkunTpUmDeCxcuULt27QLDa9eujaIoXLp0iQsXLhASEoJOp3OZpk6dOgXSP3XqVKZOnVpo+l966SUiIyP54osvmDZtGtOmTSM2NpYpU6bQqlUrt3kXQnguCYCEEGUSFBQEwKxZs2jcuHGB8bVr12bu3Lmkp6cXGHfu3Dm1Ckmv13P//fdz//33c/78eX755RcWLVrE008/rZbCOAsODubcuXMFhp89exaA0NBQQkNDSU9Px2q1YjAY1GkcQY9z+sePH0+nTp3crgfsb6GNHDmSkSNHcvr0aX766ScWLlzIuHHjWL9+fWGbRwjhoaQKTAhRJu3bt8dkMpGSkkK7du3Uf0ajkdmzZ5OUlATY39g6cuSIOl9KSgo7duwgISEBgHvvvZfp06cD9pKhwYMHc//995OZmcmlS5cKrDc+Pp6ffvrJZZzVamX9+vW0a9cOLy8vEhISsFgsfP/99+o0ZrPZJaBq2rQpYWFhJCUluaQ/IiKCN998k71795KTk8Mtt9zCsmXLAKhXrx73338/AwYM4PTp0+W4NYUQlUVKgIQQZRIaGsqIESOYO3culy5donPnzqSkpDB37lx0Op1aPaQoCk888QRjxozBYDCwYMECgoODGTZsGGAPaJYtW0bt2rWJjY0lJSWF999/n06dOlGrVi2XV9IBRo0axcaNG3nggQd47LHHMJlMatucJUuWAPYGz927d2fSpEmcP3+e+vXrs2LFCtLS0tSSJ4PBwJgxY3j55ZcxGAzceOONZGZmsnDhQlJSUmjTpg0+Pj60adOGBQsWYDKZiIqK4tixY3z++efccsstlbi1hRDlRacoilLViRBCVH8fffQRq1at4sSJEwQHB5OQkMDYsWOpV68eEydOZOvWrTz66KO8/fbbZGdn07VrVyZMmECDBg0AsFgsvPPOO3zxxRckJycTGBhI7969GTduHKGhoSQlJdGnTx9ee+01Bg8eDNhfrZ89ezbbt29Hp9MRHR3NqFGjXBpbZ2dnM2vWLNavX09ubi79+/fHz8+PH374gR9//FGdbsOGDSxZsoRDhw7h5+dHhw4dePbZZ4mKigLg0qVLvPXWW/zwww+cPXuWsLAw+vfvzzPPPIOPj08lbmkhRHmQAEgIUeEcAZBzwCGEEFVJ2gAJIYQQQnMkABJCCCGE5kgVmBBCCCE0R0qAhBBCCKE5EgAJIYQQQnMkABJCCCGE5kgAJIQQQgjNkQBICCGEEJojAZAQQgghNEcCICGEEEJojgRAQgghhNCc/wfvtSb4uTBH9gAAAABJRU5ErkJggg==", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkIAAAHJCAYAAABpOFaGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB5j0lEQVR4nO3deXzL9+MH8FeSJk3vS7XOGasabVFaurptmGuGbb6oumaYY9iUzVwdcxRzX3XMOZthB7PZZmx+rpYxw9ymQ5W2qmea5PP7I/LRSHonbSqv5+Phof3kk0/e73ya5JX39ZEIgiCAiIiIyAZJy7sAREREROWFQYiIiIhsFoMQERER2SwGISIiIrJZDEJERERksxiEiIiIyGYxCBEREZHNYhAiIiIim8UgVIFYau1LrqlJz4Ly/Dvma4io4mIQqiB++eUXREVFmf248fHxGDZsmPh7QkIC/P39sWvXLrM/Fj0bjh07ho4dOyIgIABDhw41uc+kSZPg7++f77/9+/cDACIiIhAREVHqMq1YsQLr1q0DAHz77bfw9/fHDz/8kO/+69evR7169fDvv/9i0qRJaNeuXZEf6+n9n35t6l9D/v7+2LFjh8ljPHr0CIGBgfD398fx48cNbsvJycHGjRvRq1cvNGnSBKGhoejTpw/27NlTaOBKT0/H8OHD0bBhQ4SEhODGjRtFrldxJCYmYt68eejUqRMaNmyIFi1aYPjw4YiLizPY7+nz6+/vj6VLl4q/7927F23btkVAQACmTp2Ku3fvol+/fggMDERoaCjq1auHOXPmGD3+zJkz4e/vj2nTphndFh0djUaNGkGlUhWpLnnLdPz4cZPnJK927dph0qRJRTp2Wfnrr78QERGBxo0bo0WLFli4cGGR60+AXXkXgIpm48aNFjnuV199hatXr4q/V65cGTt27EDNmjUt8nhU8c2bNw9arRZr1qyBl5dXvvt5e3tj2bJlJm+rVasWAJj8ICuJxYsXY9SoUQCADh06YObMmfjuu+/w6quvmtx/9+7dCA0NRc2aNTFy5EgMGDCgyI/19P75vTalUin279+Pt956y+i2AwcOmPygun//PoYOHYo7d+4gIiICQUFB0Gq1OHjwICZNmoS4uDhER0dDIpGYfMw9e/bg4MGDmDp1Kvz8/FC9evUi16uo4uPj8e6778LDwwMDBgzA888/j9TUVOzYsQMRERH49NNP0aNHD5P33bFjB3x9fcXfZ86ciVq1amHOnDnw8fHB559/jj///BPz58+Hj48P5s2bh9OnTxsd5/fff4e7uzv++OMPo9tOnjyJpk2bQqFQmK3OeS1btgzOzs4WOXZJ3Lp1C4MGDUKjRo3w2Wef4erVq1i0aBFSU1Mxc+bM8i5ehcAgRAYUCgUaNWpU3sUgK5aamoqQkBC89NJLBe5XlL+lF154wYwl01EqlejatSt27tyJhw8fws3NzeD28+fP49KlS5g/fz4AFDv0F3X/4OBgHD9+HMnJyfD09DS4be/evXjxxRdx4cIFg+1RUVG4e/cuduzYIYZFAGjTpg2qVq2KhQsXom3btmjfvr3Jx0xNTQUA9O3bN9+wVBqpqal47733UKtWLWzYsAEODg7ibR07dsSwYcMwdepUtGjRApUqVTK6/9N/D6mpqQgPD0ezZs3E3ytXrozOnTsDAJo3b45169YhJycH9vb2AIB///0X//77LyZMmIAFCxbg2rVrqF27NgAgLS0Nly9fxuuvv272uuvVr1/fYscuibVr18LJyQkrVqyAQqFA69atoVQqER0djeHDh6Nq1arlXUSrx66xCiAiIgInTpzAiRMnDJptU1NTMXXqVLz00ksIDAzEm2++iaNHjxrc98iRI3jzzTfRuHFjhISEYMSIEWIL0KRJk7B79278999/YnfY011ju3btQv369XHmzBm89dZbCAwMRNu2bcVuCL179+5h3LhxCA0NRUhICKZOnYpFixYV2uVw7949REVFISwsDI0bN0b//v3Fb4D5ddM93TURERGB999/H2PGjEGjRo0waNAgdOzYEWPGjDF6vNdeew0jRowQf//555/Rs2dPBAYGIjw8HJ988gkyMzMLLLNGo8HWrVvRrVs3BAUFoU2bNoiJiUFOTo5BGQcOHIivv/5a7EZ67bXXcPjw4QKPDei+1b/++uto2LAh2rRpgwULFoitB0uXLkW7du1w8OBBsVvizTffNGjK37VrF/z9/ZGQkGBw3KI06d+4cQNjxoxBeHg4GjVqhIiICMTHxwN4cj7+++8/7Nmzp9AuhKIw1XWybNky9OzZE0FBQVi2bBm0Wq34txQQEIB27dphwYIFyM3NFe8D6L6p63/u1asXcnNzxS64vHbv3g1XV1d07NgRgPHf07lz5xAZGYkmTZqgcePGGDhwIP7880/x9rz75/faBIBXXnkFUqkUBw4cMHj8lJQUHDt2DF26dDHYfuHCBfzxxx8YMmSIQQjSGzhwIPr16wdHR8d8n0t9F0+9evXEc/3o0SN8+umnePnllxEYGCiGxLzatWuH2bNnIzIyEkFBQfjoo49MPsaePXtw7949fPjhhwYhCNC1gL3//vvo168f0tPTTd5f3w2l74ICgOXLl8Pf3x/t2rXDrl27cPv2bXG/l156Cbm5ufjrr7/EY/z+++9wdXVF//794eDgYNAqFB8fD61WK4b0hIQETJw4ES1atECDBg0QFhaGiRMnIiUlxWT5nqZSqTB48GA0a9ZMDK15X0f618QPP/yAMWPGoHHjxggNDcWUKVMM3kdyc3MRExODVq1aISgoCEOGDBFfQ0+/TvMqyvvYH3/8gdatWxu0gHXq1AlardZkixkZYxCqAKZNm4b69eujfv362LFjBxo0aICcnBxERkbil19+wbhx47Bs2TL4+vpi6NChYhi6desWRo4ciYCAAKxcuRKzZs3C9evXMWzYMGi1WowcORKtW7eGt7c3duzYgTZt2ph8fK1Wi/feew+dO3fGmjVrEBwcjHnz5uH3338HoHuziIyMxKlTp/Dhhx/i008/xcWLF7F+/foC65WRkYH//e9/OH78OD744AMsW7YM9vb2GDx4cLHHNvzwww9wcnLCypUrMXToUHTv3h2HDh0yeEO+evUqLl68iNdeew0A8N133+Hdd99F7dq1sXz5cowaNQrffvstRo4cWeBYjKlTp4ofLCtXrkS/fv2wZcsWo/udO3cO69atw5gxY7B8+XLIZDKMHj0aDx8+zPfYW7duRVRUFBo0aIBly5Zh2LBh2Lx5Mz755BNxn+TkZERFRaFv375YvHgxlEolhgwZYtS6UFxXrlxBz549kZCQgClTpiAmJgYSiQSRkZE4ceKE2G3q7e2N1q1bi3+LBVGr1Ub/ChvnsmrVKnTr1g1LlixBx44dsXbtWmzfvh3vvvsu1q9fj//9739Yt24dVq5cCQDiOJzevXuLP+vH33z33XdG5dm7dy+6desmtjDklZ6ejqFDh8LDwwNLly7FokWLkJWVhSFDhuDRo0dG+5t6beq5uroiPDzcKIz9+OOPqFq1KoKCggy2619P+X15sLe3x9SpUxEWFmby9mnTpqF3797iczJy5EhkZ2ejb9+++O677zB06FCsWLECTZo0wUcffYRVq1YZ3H/r1q0IDAzEihUrxOM87ffff0elSpWMyq5Xr149REVFmQxyeTVo0MDovC1atMjg/eiNN95A48aNoVQqcerUKYMyhIWFwdHRESEhIeLzBui6xby9veHv74+srCwMGDAAV69exbRp07Bu3ToMGDAAe/fuxaJFiwosH6D7Wxk3bhzOnTuH9evX48UXX8x332nTpqFatWpYsWIFhgwZgp07d4p/n4DuPePzzz9H//79sXz5clSqVAkff/xxoWUo7H0sOzsb//33H55//nmD+3l6esLZ2RnXr18v9DGIXWMVwgsvvCD2Seublr/88ktcvHgRX375JRo2bAgAaNWqFSIiIhATE4Ovv/4aZ8+eRXZ2Nt555x34+PgAAHx9ffHLL78gMzMTNWvWhKenp0EXhqnWEEEQMHLkSLzxxhsAgCZNmuDAgQP47bff0LJlS3z77be4du0avv76awQEBADQNWm//PLLBdZL3xq1e/du8U0mODgYPXr0wMmTJ/N9wzdFLpdjxowZ4reimjVrYunSpfj555/F8Qrff/89XF1d0a5dOwiCgJiYGLRs2RIxMTHicWrVqoWBAwfi0KFDJoPhlStXsHPnTkyYMEEcZB4eHo7KlStj4sSJOHz4MFq3bg1A9018165dYleKo6Mj+vfvLw42fppWq8Xy5cvx8ssvGwSfrKws7N27V2wBycrKwvTp08V66Z/rNWvWFOkNPj/Lli2DQqHApk2bxL+3Nm3aoGvXrpg3bx527tyJRo0aQaFQwNPTs9Bur//++89kUMr73JnStGlTDBo0SPx9zpw5CAgIQK9evQAAoaGhcHBwgIuLC4AnrwlfX1+DMvXq1Quffvopbt++LXYPHD58GA8ePMj3g/7KlStISUnBgAEDEBwcDACoXbs2duzYgYyMDPEx9Uy9NvXdUwDw6quv4sMPPzToHtu7d6/Y9ZPXnTt3AKDE43peeOEFcfyNvizbtm3DpUuX8MUXX6Bx48YAgJYtW0KtVmPFihXo06cP3N3dAQBVq1bF+++/X+Bj3L17F9WqVStR+fJydnY2ed6efj8CdO83+iCkUqlw/PhxTJ48GYDutffZZ59BpVJBoVAgLi5ObA26ceMGfH19MXfuXNSoUQOA7rVy5swZnDhxosDyabVaTJo0CcePH8eGDRsKDfytW7cWB8yHhYXhyJEj+O233zBhwgT8+++/2L17N6KiosS/65YtW+L+/fuFtth07969wPcx/ZcqU2OWnJyc8m2ZI0NsEaqgjh49Cm9vbzRo0ED8pq3RaNC2bVucO3cODx8+RMOGDWFvb4/evXtj1qxZ+P3331GvXj2MGzeu2IP99G+iAMQPQn1oOnbsGGrUqCGGIED3wmzbtm2Bx4yPj0f16tUNvmk5ODjgxx9/FENXUdWuXdugabhGjRoIDg7Gvn37xG179+5Fp06doFAocO3aNdy9exft2rUzaK0ICQmBs7Mzjhw5YvJx9G+gT3drdOnSBTKZzKBrxNPT02A8if5DKisry+Sxr1+/jgcPHuCVV14x2D5kyBDs2rULcrkcAGBnZ4euXbuKtyuVSrRq1QonT57M/wkqghMnTqBt27YGfxt2dnbo0qULzp07h4yMjGIdz9vbGzt37jT6V9j4jae/eTdr1gxHjhxB3759ERsbiytXrqB///5iy15+unfvDjs7O3z//ffitj179ogtOKb4+fnB09MTw4cPx9SpU3HgwAFUqlQJH3zwgcEg36J6+eWXIZPJxO6xe/fuIS4uzuD86clkMgC6rldzOXHiBKpVq2bw+gV0z01OTg7OnDkjbiuoxSNvGc1ZvqIICwvD6dOnIQgCTp06hczMTLRo0QIA0KJFC2RlZSE+Ph5ZWVk4f/68GIRefPFFbNu2DdWqVcONGzdw6NAhrFu3DteuXSt0RlVMTAy+++47DBgwAIGBgYWW8ekvBb6+vuL74/HjxyEIAjp16mSwT96/AUEQTLacFvY+ptVqCyyXJcaJPYsYhCqo1NRUJCUloUGDBgb/5s2bBwBISkpC9erVsWXLFjRs2BA7d+7E0KFDER4ejkWLFhV73ROlUmnwu1QqFY+RkpJicvZQQTOK9HUobJ+icnJyMtr22muv4f/+7/+QkpKCv/76Czdv3hQ/PPXf2mfMmGH0HKanp+PevXsmH0f/Dczb29tgu52dHTw8PAy6T54eQ6F/U8rvzUtfpsKek0qVKsHOzrAx18vLy6AloiQePnxocoBrpUqVIAhCsb9dKhQKBAYGGv17+rl72tNjYIYOHYqpU6ciOzsbMTEx6NKlC7p27Ypjx44VeBwPDw+0a9dO7B57+PAhDh48mG9rEKD7O9q6dStat26NH374AaNGjUJYWBimTp1aounIzs7OaNWqldg9tn//frzwwgvw8/Mz2lff0nL79u18j5eYmFis1+7Dhw9NPt/685yWliZuy2/sUV5Vq1YVW67yU9jtxRUWFobU1FRcu3YNv//+O2rXri228OlbwY4fP44///wTubm5CA8PF++7YcMGhIWFoWPHjvjwww9x4sQJo9elKdevX0dISAg+//xzJCYmFrq/qfFS+vOUnJwMwPh1nff3EydOGL0P6b90FfQ+pv/SYupLSnp6ulELJpnGrrEKysXFBbVq1TLo1slL37yuH3CqUqkQHx+PHTt2YNWqVahXr16+U4uLy8fHx+SYngcPHhR4PxcXF5MDBU+dOgU3NzcxfD39DbSwwcx6r776Kj755BP8/PPPuHbtGqpVq4YmTZoA0I3fAICJEyciNDTU6L5PzzR6entSUpJBF0Fubi5SUlLg4eFRpLKZoi+T/o1TLyUlBefPnxe/1ZsKPPfv3xffWPMLXIW16Li5ueH+/ftG25OSkgCgVHUrDalUin79+qFfv3548OABDh06hFWrVmH06NE4cuRIgdOke/fujbfffhuXLl1CfHw8JBIJunXrVuDj1a5dG/Pnz4dGo8HZs2fxzTffYPv27ahZs2a+6yYVpHPnzvjggw+QnJyMffv2GbUm6ulbOQ4dOmRyNp1arcZrr72G4OBgrFixokiP7ebmhps3bxptL+k5bdmyJQ4ePIi//vrLZEvJhQsX0KNHD0yePBkDBw4s1rHzU79+fbi7u+PPP//E0aNHxedJ76WXXkJ8fDykUinq1q0rBr/vvvsOc+bMwQcffICePXuKXZNjx441GHxtSnR0NEJDQ/Hqq69ixowZRX6+TdEPS7h//77BDK68r/MGDRoYDWDXj/sp6H3MyckJPj4+Ruf4wYMHyMjIQJ06dUpcblvCFqEKQio1PFWhoaG4c+cOvLy8DL5tHzlyBLGxsZDJZNi4cSPatm0r9p+HhYUhOjoawJNvnU8ftyRCQ0ORkJBgMFg3OzvbYBCjKU2bNsWtW7dw+fJlcVtOTg5Gjx6NnTt3it928n4jy83NxdmzZ4tULldXV7Rt2xa//PILfvzxR3Tv3l0MCbVr14aXlxcSEhIMnj8fHx8sWLAA58+fz7eugK55Oq+9e/dCo9GIb1AlUbt2bXh4eODgwYMG27/55hsMGzZMHCP09HObnZ2Nw4cPi2Oq9M/b3bt3xX2uXr1aaItRSEgIDh48aNDyo9FosHfvXgQGBlpsXZbC9OnTRxwz5eXlhZ49e6Jfv35IS0sTy5rf33GLFi3g6+uLn376CT/88AM6dOggBk5T9u/fj+bNmyMpKQkymQyNGzfG9OnT4erqmm9LTWGvobZt20KhUGDLli34888/8w1Cfn5+aNWqFdauXYtbt24Z3b569WqkpKSge/fuBT5eXiEhIfjvv/+M1uL59ttvIZfL8x30nJ/u3bvD29sbn376KbKzsw1u02g0iImJgVwuN9uXLED3/DZr1gxHjx7FxYsXjYJQixYtcPHiRZw6dcqgNSg+Ph6urq4YOnSoGIIyMjLEmWUFqVSpEry9vTF+/Hj88ssvBS7OWZgmTZoYdI/q/fTTT+LPzs7ORi2n+tdxQe9jgG6c1G+//WbQYvnjjz9CJpOhefPmJS63LWGLUAXh6uqK06dP4+jRo6hfvz569uyJLVu2YNCgQRg+fDiqVKmC//u//8PatWvRv39/yOVyNG/eHDExMXj33XfRv39/yGQyfPHFF1AoFOL4HVdXV9y/fx+HDh0q0hgBU7p27Yo1a9bg3XffxdixY+Hq6ooNGzbgwYMHBa5h0bNnT2zevBkjRozAmDFj4OHhgU2bNiE3Nxd9+/aFm5sbGjdujM2bN+O5556Dm5sbNm3ahOzs7CI14wO6N+4xY8ZAo9EYjCmRyWQYN24cpk6dCplMhrZt2yItLQ0rVqxAYmJivoMjX3jhBbz++utYsmQJsrKyEBISggsXLmDZsmVo1qwZWrZsWbwnLw/9rLKZM2fCy8sL7dq1w/Xr17FkyRL069fPoJVq8uTJeO+99+Dl5YV169YhMzNTnE7brFkzKJVKzJkzB2PHjkVGRgaWLFkiDorNz6hRo3D48GEMGDAAw4YNg1wux5YtW3Dr1i3ExsaWuF6lFRISgvXr16NSpUpo3LgxEhMTsWHDBoSGhoofcK6urjh16pS4mJ7+g0IqlaJHjx7Ys2cPbt++jQ0bNhT4WMHBwdBqtXj33XcxbNgwODk54YcffsCjR4/QoUMHk/d5+rX5NEdHR7Ru3Rpr1qxBUFCQOHDXlBkzZiAyMhJvvvkmBgwYgIYNGyIjIwP79+/H3r170adPH6OxJgXp2bMntm3bhnfffRdjxoxB9erV8euvv+Lrr7/GqFGjCgyFpri4uGDOnDkYNWoU3njjDfTv3x+1atXC3bt3sXXrVpw9exYLFiwQW0HMpXnz5pg9ezZkMplRC25YWBjS09MRFxeHIUOGiNuDgoKwfft2zJkzB23btsW9e/ewbt063L9/P98W36fpV/SeNWsWXnrppSLfL68aNWqgV69eWLhwIXJzc1GvXj0cOHBA/MJTlC+j+b2PAbqu471792Lo0KEYNGgQbty4gYULF+LNN9/kGkJFxCBUQfTr1w/nzp3D22+/jU8//RTdunXD1q1bsWDBAsyfPx+PHj1CtWrVMGHCBAwePBiAbirrqlWrsHz5cowfPx4ajQYBAQFYv369uABZz549cejQIfGN0tRslsLY2dlh3bp1mDVrFqZPnw47Ozt0794d7u7uBU7fdHZ2xpYtWzBv3jxER0dDq9WiUaNG2LRpk/hhMWfOHERHR2PKlClwdnZG79690aRJE3z11VdFKlvr1q3h4uKCGjVqGE0xfeONN+Dk5ITY2Fjs2LEDjo6OCA4ORkxMTIEfVrNmzcJzzz2Hr7/+GmvXrkXlypUxYMAAjBw5stQtbPp1YtatWyeuwvv222/j7bffNthv+vTpmD17NpKTkxEcHIzt27fjueeeA6D7YF66dCkWLFiAd999F9WqVcOoUaOwZ8+eAh/bz88P27Ztw8KFCzF58mRIJBIEBQVh06ZNaNq0aanqVRpjx46FQqHA119/jeXLl8PFxQXt2rXDhAkTxH2GDx+OFStW4O2338a+ffsMPgB69+6N1atXo0aNGia7QfOqXLkyYmNjsXjxYnz00UfIysqCn58fli5dmu+366dfm08PTAZ03WP79+8v9PVVtWpV7NixA59//jm+//57rFmzBgqFArVr18aCBQuK/fp0cHDA5s2bsWDBAixevBjp6emoXbs2Zs2aVeBYqYK0aNECX331FdavX4/Vq1fj/v37cHd3R0BAAHbs2CHOYjWnsLAw5Obm4qWXXjIaj+Pp6Yn69evj0qVLCAkJEbe//vrrSEhIwNdff41t27bBx8cHrVu3Rt++ffHxxx/j6tWrhXYdSaVSzJw5E7169cLcuXMxe/bsEpX/448/hqOjI9avX4/09HSEhYVhxIgRWL58eZG+1BX0PlanTh2sX78e8+bNE79QDhw40OT6Q2SaRODVAqmULl++jGvXrqFDhw4GTba9e/eGr69vvpdZoOJbunQpli1bhn/++ae8i0JERZCamorDhw+jZcuWBmOy5s6di127dpV6UVIqPbYIUallZmZi7Nix6Nu3L1555RVoNBrs27cP586dK3RdEiKiZ5mDgwNmzZqFF198EZGRkXB0dMSff/6JLVu24J133inv4hHYIkRmsn//fqxbtw5Xr16FIAioX78+RowYYTSwkUqHLUJEFc+FCxfw2Wef4c8//0RWVhZq1qyJPn36oF+/flzrxwowCBEREZHN4vR5IiIislkMQkRERGSzGISIiIjIZjEIERERkc2ymenzgiBAq7W9ceFSqYT1tiGst21hvW2LLdZbKpVYfGadzQQhiUSCtLRMqNUFX2PmWWJnJ4WHhxPrbSNYb9bbFrDetlVvT08nyGSWDULsGiMiIiKbxSBERERENotBiIiIiGwWgxARERHZLAYhIiIislkMQkRERGSzGISIiIjIZjEIERERkc1iECIiIiKbxSBERERENsvsQejOnTsYP348wsPDERISgiFDhuDy5csG+/zwww/o3LkzgoKC0KNHDxw9etTg9pSUFEyYMAEhISEIDQ3FjBkzkJWVZe6iEhERkY0zaxBSqVQYNmwYkpKSsGrVKmzbtg1OTk6IjIxEcnIyAODYsWP44IMP0KdPH+zevRthYWEYNmwYrl69Kh5nzJgxuHnzJjZu3IjFixfj0KFDmD59ujmLSkRERGTeIBQXF4dLly4hJiYGgYGB8PPzw/z585GZmYlff/0VALB27Vq8/PLLGDBgAOrUqYOoqCg0aNAAn3/+OQDg9OnTOHHiBObOnYsGDRogLCwMM2fOxDfffIPExERzFpeIiIhsnFmvPu/n54c1a9bAx8dH3CaV6rJWWloatFotTp06hUmTJhncr1mzZvjpp58A6MKUt7c36tSpI94eGhoKiUSC+Ph4dO7cuURly3qYgtObl0Mo0b0rJgl0z79Wq2W9bUBJ6u1QvS5ebNvFksWyqEeZKuz//V+otYBKpYYg2M4Zl0gkUCjsWG8bYY31lkgkCA+sghequ1nwMSx2aJFZg5C3tzdat25tsG3z5s3Izs5GeHg40tLSkJmZCV9fX4N9KleujLt37wIAEhMTUaVKFYPbFQoF3N3dcefOnRKXzU6rQq1Hp0t8f6Jn0oXTkNV/Dq71mpV3SUrk4J938N2RG+VdDCKb9TAzF9MCm5d3MUqlWEEoISEB7du3z/f2o0ePwtPTU/z9wIEDWLBgAQYOHAh/f38x7CgUCoP72dvbIycnBwCQlZVldPvT+5SE2k6JGz5tAStJ0mVCIoGdTAq1Rst624Ji1vthwjU0VPyL+98tR65jFUhdvMqgkOZ1O+kRAKBBbS/Uq+kOQWs751silcDeXo6cnFzW2wZYY70lEiDkRR+kpGRY7DHc3BzEniVLKVYQ8vHxwb59+/K93c3tSfPY9u3bER0dje7du2PixIkAdGEG0A2qzisnJwcODg4AAKVSaXS7fh9HR8fiFNeAg4sbGvcaBLVaW+JjVDR2dlJ4eDghJSWD9bYBxa33e58dhIf0e9TEA6T/tAIO3SZBIpWVQUnN51GG7r0i5EUftGtclefbBrDe1ldvS5anLL7LFisIyeVyg7E7+Zk/fz5iY2MxaNAgREVFQfK4k8/d3R2Ojo64d++ewf737t0TxxX5+vri559/NrhdpVIhNTUVlStXLk5xiagASgclNj5shSle+4DEy1DF74F9SK/yLlaxpGflAgBcnIxbkYmIisLs7U36EBQVFYVJkyaJIQjQDawKDg7GiRMnDO5z/PhxNG3aFAAQEhKCu3fv4ubNm+Lt+v2bNGli7uIS2SxnBzkeaF2Q6N8bAKA6/T3UCX+Xc6mKJ0MfhBwZhIioZMwahI4fP47Y2FhERESgW7duSEpKEv9lZOj6EAcNGoS9e/diw4YNuHr1KubNm4cLFy4gMjISANCwYUMEBwdj3LhxOHv2LI4dO4apU6eiR48eBrPRiKh0nB3kAIC7Lg0gr9cagIDsg6uhzXxYvgUrBn2LkCtbhIiohMwahL7//nsAupliLVq0MPi3fv16AECLFi0we/ZsbN++Ha+//jqOHTuGVatWiV1uEokEy5YtQ/Xq1REZGYn33nsPrVq14oKKRGbmpNQFoYysXNi/1BdSj2oQstKQ/dtaCIJ1jUHIj9g15igv55IQUUVl1unz0dHRiI6OLnS/Hj16oEePHvne7uXlhSVLlpixZET0NH2LUHpWLiR29lC+PBKZu2ZAk3AOqjP7YN+oazmXsGBarYDMbDWAx2OE1JpyLhERVUS86CqRjXJy0H0PysjWtarIPKrBPrwfAEB1chc0dy/ne19rkJmjFheOdHZg1xgRlQyDEJGNetIipBa3yf1bwa5Oc0DQIuvXVRCy08ureIXSD5RWKmSQ2/GtjIhKhu8eRDYqb9eYnkQigbJlJCSulSGkP0D24fVWs5z/0/Tl1teDiKgkzDpGiIgqDieHJ4Ol85IoHODQfiQyv4mG+sYp5P79CxQBL5dHEQskBiEOlCYqEUGTC0GVBeRmQ8jNfvKzKgtCbjaQmwVBlW3i52wIuVkQ1CooAjpA0SD/K05UBAxCRDbKWWncIqQn864F+2ZvIefoNuQc+wIyXz/IKj1X1kUsEFuEyFaJAUaVqQstqiwIqkzg8f/6bXgcaITcLOBxiHnycxagLf0EA83dSwCDEBFVRHm7xgRBMFj8FADkAa9Ac/sC1DdPI+uXFXB6fTokCofyKKpJGQxCVAEJGjWEnIw8ISbzqSDzZJsuyGSJoSdNnQ1NdiagMf7yUip2CkjkSkDhAIlcCYncAZArIVHk/fnxbYrHvz/+Wer9vHnLUg4YhIhslD5AaLQCslUaONgbvh1IJBIoWw9BxtdTITxMRPYfm6BsO8woMJWX9GwGISp7giA87j7KhJCTASEnE4IqA8jJFH8Wch7fpsrUbRe3ZQIa42tplphcCYnCURdOFA6Pw4oDJArHJ78/DjB5wwzkDo9DzuPtFewag+bGIERkoxRyqXi1+oysXKMgBAASpTOU7Ycj67s5UF85CnW1+pD7tyyH0hrTz3ZjEKKSENSqx0EmHUJ2uu7nx/8jJ0MMMrpwow89upac0l8JVAIonoQYw+Di+Lh1xvFJkFE4QubgBLdKnniUI4FWaq8LMxa+KrutYBAislESiQTODnZITVchPTsXlWC628vOty4UTXpAFbcL2Uc2Q1q5DmQeVcu4tMY4WJoAfVdTOoTsx6EmJx0Qf87Isz3DIPCUumVGageJvSMk9k660GLvpAsx9o6P/3cC8vys2677vyQhxs5OCnsPJ2SmZECwsqvPV3QMQkQ2zNlBrgtCJgZM56Vo1BWa2xeguX0B2b+sgGOPqZDYle8ihuIYISWD0LNC0Kp1YSX7ke7/rEdPfs5O0wWYnEfIyM2COiMN2pwMIDe75A8okepCitIZsHeCxN4ZEuXj/58KMVA4GYYcmdxquompdBiEiGzYk+uNqQvcTyKVQtnuHWTu/Bja5ATkHN0OZcvIsihivjLYImTVxLE02Y+e/Mt6lCfoPII265GuteZx4IEqs2QPJpFAonAClM6Pg4su3Bj8b+/8+Gf9diddywzDjM1jECKyYaYWVcyP1NEdyrbDkPXDAuReOAhZtRchrx1q6SLmi4Oly56gyYWQlaYLLlkPIWSlQZv1MM/vj3QtN48DD7QFB2yTJJLHocUFEgcX3c8OLrrflS6QObnCtVIlZOTKoJU7Pe6acoBEwvEyVDIMQkQ2LL9FFfNjVyMQioadoTqzD9mHN0BW6XlIXb0tWcR8cR0h8xBysx+Hm8ehJjNN/F0fdnS3pZWsxcZOIYaYJ8HGVdc6k3e70hlSpatuvE0B42fs7KRw9HBCTkoG1BwrQ2bAIERkw4rTIqSnCOkJ9d1L0CZeQdavK+HY/UNIpGX7VpKr1kCVq/sQ5AVXjQlaNYTMhxAyU6HNTBV/FjKNW3GgLuagYanscZBxhcTRFRIHV0gd3HRhxsEtT7DRhRuJnb1lKklkJgxCRDZMDELZRQ9CEqkdHNoNR8bXU6G9dw05J76GsvlbliqiSfqp8zKpBA72trMGiqDJFQONNvshHgqZyLp/D+r0lCdBJzNVN96mOGSKp0KNa55/T36XOrjqBhVzXA09QxiEiGyYk4PuLaA4LUIAIHWpBGXrwcg+sAy5Z3+AXdUXYVczyBJFNElfXiel3TPxoSyoVU9abzJSH4caw9YcbWYqkJNhcL8M04fTkcp0IcbRHVInd0gc3Y1CjcTx8e9ypSWrR2TVGISIbJhzMccI5SV/vik09dsj9/wvyP5tLRx7zYTUycPcRTRJDEIVYHyQkJsDISMZ2owU3f/pyRAyUqDNSIaQkQwhPUW39k1Ryex04cbRHUqPSlDLnSEo3SB1fBx2HN11AUfpzAHEREXAIERkw0oyRigv++ZvQZN4GdoH/yL74Bo4dP6gTFa7tZbrjAmqrMeBJgVCerIYbrQZKRDSdWGnyAOMZQpInNzzBBo3MfDoA47U0U3smrKzk8LDwwkpHDRMVCoMQkQ27EkQKsE0ZwASOwUc2o9Exq5p0Ny+ANXp72Df5DVzFtGkspgxJqhzoE1/AOHR/cetOMnQputadfQtOkVezE/uAKmzByROnpA66f6XOHlA6uwpboPC8Zno5iOqaBiEiGyYvmspK0cNjVYLWQlac6TuvlC2GIDs39ZCdWoPZFX8YVe1nrmLaiAjWz9GqORBSMjJeBx0HkCbfj9P6NH9X+QBx/ZOYriROnlC4uyh+98pT/BRmL58CRGVPwYhIhvmpHzyFpCRrYarY8mmosvrhkN9+zzUl44g+9dVcOwdDanSxVzFNFJYi5AgCLrVix/lDTj3oX30AEK6LvhAlVX4A8kdIHXxgsTZC1JnL10rjr41Rx925JweTlSRMQgR2TCZVAoHeztk5aiRkZVb4iAEAMrwCGQmXoX24V1k/xYLh47vWayrJz1TBWdJFnyFO1Bduo8UdRoy7t2BJi1J7MoqykU1JUoXXchxqWT0v9TZS7dqMRE90xiEiGycs4MuCJV0wLSeRK6E8uWRyNwzE5p/zyD3rx+hCOpU4uMJWrVuAHLaPfGfkJYE7aN7eP3BXbzpkQtcBTKu5jeNXKIbcOxSCVLnSk9adlwqQeL8OOiwNYfI5jEIEdk4Zwc5klKzSx2EAEDmVRP2zf+HnCObkXPiK8h860JWuXa++wuqrDxBJwnCI93/2rR7ENIfAILp2VByAFoB0Cjd4OBVBcpKvlAr3AFHT13wcamkG5As41scERWM7xJENs6plFPonyav3w6a2xegvh6HrF9WwvHVCbpLOuQJPNpHutadQgcky+SQunpD4uINqWvlx/+88dn+O7iUYocJfZuiQW0vTiMnohJjECKycU8WVSzZFPqnSSQSKFsNQkbSdQiPkpDx5aSC91e6QOL6OOg8DjySx6FH4uhmclHAf7MeQY3cCrGgIhFZNwYhIhvnrDRvixAASOyd4PDySGR+Pw/Q5OrG5jxuzZG46P7Xt/AUd2q5VhDE6fPlvaAiEVV8DEJENq60q0vnR1a5DpwHLAWkUrNenT4rRw1B0P1cmnWEiIgABiEim+dUiuuNFUZiV/Lp+PnRBzZ7uQxyO15Li4hKh+8iRDZOfwV6fXeTtdOPZXJ24Pc4Iio9BiEiG2eprjFLqUhXnici68cgRGTjKloQspYrzxPRs4FBiMjGPZk1poagH4VsxcriyvNEZDsYhIhsnL6LSa3RQpVr/QsSsmuMiMyJQYjIxikVMsikuoujVoTusXT9GkKcOk9EZsAgRGTjJBJJhRonxDFCRGRODEJE9CQIVYAp9BwjRETmxCBERBZdVNHcOEaIiMyJQYiIKmTXmBMXVCQiM2AQIiI4KR+vLl0BglB6tn5labYIEVHpMQgRUZ4WIXU5l6Rgao0WOSoNAAYhIjIPBiEiqjBdY/rySSSAgz27xoio9BiEiOjJYGkrnzUmDpRWyiGVSMq5NET0LGAQIqIK0yLENYSIyNwYhIiowgQhriFERObGIEREFWYdIQYhIjI3BiEiEoNFZrYaWq31XoE+nWsIEZGZMQgRkbiOkADrHjCd8Xh6vxMvuEpEZsIgRESwk0nhYC8DAGRkW+9aQuKV59k1RkRmwiBERACetLJY84BpzhojInNjECIiAE8GTFtzEOJgaSIyNwYhIgLwJFxY88wxXnmeiMyNQYiIAFSMtYTYNUZE5sYgREQAAGcrHyMkCII4kJtBiIjMhUGIiAA8WZvHWrvGsnI00Dxe48iZ6wgRkZkwCBERAOvvGtNPnVfIpZDbycq5NET0rDB7ELpz5w7Gjx+P8PBwhISEYMiQIbh8+bJ4u1arRWxsLDp27IhGjRqhS5cu+OqrrwyOkZCQgHfeeQfBwcFo0aIFPvvsM2g0GnMXlYjysPYglJHnyvNEROZi1iCkUqkwbNgwJCUlYdWqVdi2bRucnJwQGRmJ5ORkAMDq1auxevVqjB07Ft9++y0GDBiA6dOnY8+ePQCA3NxcDBkyBADwxRdfYPr06di+fTuWL19uzqIS0VOeBCHrXFCRU+eJyBLM2tEeFxeHS5cu4fDhw/Dx8QEAzJ8/H82aNcOvv/6K3r17Y/v27Rg8eDA6d+4MAKhZsybOnDmDr776Cj169MCPP/6I27dv48svv4Sbmxvq1q2LBw8eYN68eRg+fDgUCoU5i0xEj4kXXrXSS2xwxhgRWYJZg5Cfnx/WrFkjhiAAkEp1jU5paWnQarWYO3cunn/+eYP7SaVSpKWlAdCFqQYNGsDNzU28vXnz5khPT8eFCxfQsGHDEpdPJrOtIVH6+rLetqG09XZz1n3JyMjKhZ2d9T13mSpdS5WLo9ygfDzfrLctsNV6SySWfwyzBiFvb2+0bt3aYNvmzZuRnZ2N8PBwSKVShIWFGdx++/Zt7N27F3369AEA3L17F76+vgb7VK5cGYBu/FFpgpCrq0OJ71uRsd62paT1Vih1QUil1sLRWQl7uXUNSNYIundEL3dHeHg4Gd3O821bWG8yl2IFoYSEBLRv3z7f248ePQpPT0/x9wMHDmDBggUYOHAg/P39jfa/f/8+3n77bXh5eWHEiBEAgOzsbLi6uhrsZ29vDwDIyckpTnGNpKVlQaPRluoYFYlMJoWrqwPrbSNKW29BECCTSqDRCki4nQpPV6UFSllySckZAAC5FEhJyRC383yz3rbAVuvt5uYg9ixZSrGCkI+PD/bt25fv7Xm7s7Zv347o6Gh0794dEydONNr32rVrGDZsGDQaDTZt2iSGH6VSCZVKZbCvPgA5OjoWp7hGNBot1Grb+QPSY71tS2nq7aS0Q1pmLlIf5cDV0brG4z3K1I0RcrS3M1k/nm/bwnrbBkGw/GMUKwjJ5XLUqVOn0P3mz5+P2NhYDBo0CFFRUZA81ckXHx+PESNGwMfHB7GxsQZjinx9fXHp0iWD/e/duwcABvsRkfk5OciRlplrlYsq8jpjRGQJZm9v0oegqKgoTJo0ySgEnT17FkOHDoWfnx+2bt1qFG5CQkJw/vx5pKeni9uOHTsGJycn1KtXz9zFJaI8xCn02dY3hZ7T54nIEswahI4fP47Y2FhERESgW7duSEpKEv9lZGRArVbj/fffh5eXF+bMmYOcnBzxdv06Qy+//DK8vb3x3nvv4eLFi/j555+xcOFCDB48mFPniSzMmhdV5PR5IrIEs84a+/777wHoZopt3rzZ4LZRo0YhPDwcN2/eBKALPHlVq1YNv/76K+zt7REbG4sZM2bgzTffhJubG/r27YuRI0eas6hEZIKTFQchdo0RkSWYNQhFR0cjOjq6wH3++eefQo/z3HPPYf369eYqFhEVkb61xdrGCKk1WmSrdJfZYYsQEZmTba3MREQFclJa5xXoMx6PWZJAN2uMiMhcGISISGStY4T05XFU2kEqLYOlZonIZjAIEZHoyawx6wpCHChNRJbCIEREImu9Aj2nzhORpTAIEZHIyUoHS3PGGBFZCoMQEYnEWWPZudCWxdr2RcSuMSKyFAYhIhLpg4YgAJlWtLo0u8aIyFIYhIhIZCeTwl4hA2Bd3WNi15iSU+eJyLwYhIjIgLPS+qbQs0WIiCyFQYiIDOQdJ2Qt9AsqcrA0EZkbgxARGXBy0HU/WVOLEAdLE5GlMAgRkQFrXEuIXWNEZCkMQkRkwNquQC8IAoMQEVkMgxARGdAPlraWWWPZKg00Wt2aRhwjRETmxiBERAas7cKr+kAmt5PCXi4r59IQ0bOGQYiIDFhbENJfAJbdYkRkCQxCRGTA2q439mQxRQYhIjI/BiEiMiC2CFnJOkJPBkpzVWkiMj8GISIy4Gxl6whlZHExRSKyHAYhIjKgDxyqXC1y1ZpyLg0XUyQiy2IQIiIDDvZ2kEh0P1vDoopcQ4iILIlBiIgMSCUScWCyNQyY1o9V4mBpIrIEBiEiMmJNU+jZIkRElsQgRERGrCkIcYwQEVkSgxARGbGmKfRsESIiS2IQIiIjTo+n0FvFGCFx+jzXESIi82MQIiIj1tI1ptFqkZXDdYSIyHIYhIjIiLUEoYw80/edlGwRIiLzYxAiIiNPrjdWvusIZTweo+RobweZlG9XRGR+fGchIiPOSusYLM2B0kRkaQxCRGTEWq5AL155nkGIiCyEQYiIjFjLGCG2CBGRpTEIEZER5zxjhARBKLdy6McoOXPqPBFZCIMQERnRBw+tIIjT18sDu8aIyNIYhIjIiNxOBoVc9/ZQnt1j7BojIktjECIik56MEyq/FiH9YG1eeZ6ILIVBiIhMEqfQs0WIiJ5hDEJEZJI1TKHXr2PEIERElsIgREQmWcMV6DPYIkREFsYgREQmlXeLkCAIvPI8EVkcgxARmaSfQl9eY4RUuVqoNdrHZWGLEBFZBoMQEZlU3oOl9Y9rJ5PAXi4rlzIQ0bOPQYiITCrvrrG8iylKJJJyKQMRPfsYhIjIpPJeR4gzxoioLDAIEZFJ5X3hVXHGGBdTJCILYhAiIpPKe/o8rzNGRGWBQYiITNIHkByVRpy9VZaerCrNqfNEZDkMQkRkkqPSDvohyuXRPcYWISIqCwxCRGSSVCKBo1LXGlMeM8cyHg/S5mBpIrIkBiEiyld5DpjOyOZgaSKyPAYhIspXeU6h55XniagsMAgRUb7ERRXLYeYYxwgRUVlgECKifJVr1xhbhIioDDAIEVG+yisIabUCMrM5WJqILI9BiIjy5VROQSgjOxfC45/1M9eIiCzB7EHozp07GD9+PMLDwxESEoIhQ4bg8uXLJvdVqVTo1q0bJk2aZLA9JSUFEyZMQEhICEJDQzFjxgxkZWWZu6hEVAjncrrwqj54OdjLYCfj9zUishyzvsOoVCoMGzYMSUlJWLVqFbZt2wYnJydERkYiOTnZaP958+bh0qVLRtvHjBmDmzdvYuPGjVi8eDEOHTqE6dOnm7OoRFQE5dU1pl9DyIlT54nIwswahOLi4nDp0iXExMQgMDAQfn5+mD9/PjIzM/Hrr78a7Pv777/jhx9+gJ+fn8H206dP48SJE5g7dy4aNGiAsLAwzJw5E9988w0SExPNWVwiKoSzfkHF7LKdPs8rzxNRWTFrEPLz88OaNWvg4+Pz5AGkuodIS0sTtyUnJ2Py5MmIjo6Gh4eHwTHi4uLg7e2NOnXqiNtCQ0MhkUgQHx9vzuISUSHKbYwQZ4wRURkx6yhEb29vtG7d2mDb5s2bkZ2djfDwcHHbRx99hLZt26Jdu3bYsGGDwf6JiYmoUqWKwTaFQgF3d3fcuXOnVOWT2dhYA319WW/bYIl6uznbA9AFE5lMAolEUsg9zCMzR9cC5eKogJ1dwfXh+Wa9bYGt1rss3nKKFYQSEhLQvn37fG8/evQoPD09xd8PHDiABQsWYODAgfD39wcAfPHFF7h69SoWLFhg8hhZWVlQKBRG2+3t7ZGTk1Oc4hpxdXUo1f0rKtbbtpiz3g5OuiCk0QpQOtrDsYzG7KgfTxnz8nCAh4dTke7D821bWG8yl2IFIR8fH+zbty/f293c3MSft2/fjujoaHTv3h0TJ04EAFy7dg3z58/HunXr4OjoaPIYSqUSKpXKaHtOTk6+9ymqtLQsaDTaUh2jIpHJpHB1dWC9bYSl6i23kyJXrUXC7Yfw9iibN+H7KZm6x5ZKkJKSUeC+PN+sty2w1Xq7uTmIQ2wspVhBSC6XG4zdyc/8+fMRGxuLQYMGISoqSmxO37dvHzIyMjBo0CBx3+zsbJw6dQo//vgjTp8+DV9fX/z8888Gx1OpVEhNTUXlypWLU1wjGo0WarXt/AHpsd62xdz1dnaQI+VRDlLTc+DhYm+24xbkUYbuy5CjvV2R68LzbVtYb9sgCIXvU1pmX6lMH4KioqIwePBgg9v69++Pbt26GWx7//334evri/fffx8AEBISgpiYGNy8eRPPPfccAODEiRMAgCZNmpi7uERUCCelLgiV5VpCT64zxsUUiciyzPouc/z4ccTGxiIiIgLdunVDUlKSeJujoyPc3d3h7u5ucB+lUgknJycx9DRs2BDBwcEYN24cpk+fjszMTEydOhU9evQwmI1GRGXD+XEYKcuZY/qr3TtzHSEisjCzBqHvv/8egG6m2ObNmw1uGzVqFEaPHl3oMSQSCZYtW4YZM2YgMjIS9vb26NSpEyZPnmzOohJREZXHoor6q93zyvNEZGlmDULR0dGIjo4u1n2eDkwA4OXlhSVLlpirWERUCuURhNK5jhARlRHbWpCAiIpN3ypTVqtLq3I1yH08GJRBiIgsjUGIiAqkv95XWQ2W1rcGyaQSKBWyMnlMIrJdDEJEVKCy7hp7MmNMXmYrWROR7WIQIqIClXUQ4nXGiKgsMQgRUYHKvEUoWz91nmsIEZHlMQgRUYH0ixrqp7RbWt6uMSIiS2MQIqIC6VuEsnI0UJfBNY4YhIioLDEIEVGBnJRy6Icsl8UUeo4RIqKyxCBERAWSSiVwVJbdZTa4mCIRlSUGISIqlLioYhkEIbYIEVFZYhAiokKV5aKK6frrjPGCq0RUBhiEiKhQZTmFXrzyvAOnzxOR5TEIEVGh9KEkvQym0LNrjIjKEoMQERXKqYxahLSCIK5XxCBERGWBQYiICuVcRoOlM7PVEATdz1xHiIjKAoMQERXqyRghy64jpA9a9goZ7GR8eyIiy+M7DREVqqwGS4trCHHGGBGVEQYhIipUWa0jxMUUiaisMQgRUaH0LTRl1iLEqfNEVEYYhIioUOJg6excCPrRzBagv5YZB0oTUVlhECKiQjk9bqFRawTk5Gos9jjsGiOissYgRESFspfLYCfTXYPekt1jXEyRiMoagxARFUoikeQZMG25KfT6kMWuMSIqKwxCRFQkZTGFnl1jRFTWGISIqEjKYuYYu8aIqKwxCBFRkZRJi9Dj64w5cUFFIiojDEJEVCRlsagi1xEiorLGIERERWLpFqFctQaqXK3BYxERWRqDEBEViRiEsi0ThPQXdJVKJHCwZ4sQEZUNBiEiKhL9ooqWmj6fIU6dt4NEIrHIYxARPY1BiIiKxNKzxjh1nojKA4MQERWJpQdLczFFIioPDEJEVCSWHiytH3vkzKnzRFSGGISIqEj0QSgzRw2NVmv243MxRSIqDwxCRFQkTnnW9snINv+A6fQ8g6WJiMoKgxARFYlMKhWntVtinBAHSxNReWAQIqIi06/4bIlxQvpp+RwsTURliUGIiIrMkgOmxRYhDpYmojLEIERERfZkCr35xwhlZLNrjIjKHoMQERWZJRdV5BghIioPDEJEVGRii5CZrzcmCALHCBFRuWAQIqIis9QYoawcNbSCYPAYRERlgUGIiIrMUkFIfzx7uQxyO74tEVHZ4TsOERXZkyvQmzsI6brFnLmYIhGVMQYhIioyS7cIOXHqPBGVMQYhIioySwWhDF55nojKCYMQERXZk+nzagiPBzebA6fOE1F5YRAioiLTt9ioNVqocs13BXoGISIqLwxCRFRkSoUMMqkEgHnXEtIfi11jRFTWGISIqMgkEokYVsw5TogtQkRUXhiEiKhYLDFgOkMMQpw+T0Rli0GIiIrFWakLK+ZtEdKvI8QWISIqWwxCRFQsT65Ab/6uMY4RIqKyxiBERMViia6x9MeDpZ25oCIRlTEGISIqlidBSG2W46k1WuSoNADYIkREZY9BiIiKxdwtQvrjSCSAo5KDpYmobJk9CN25cwfjx49HeHg4QkJCMGTIEFy+fNlgn7Nnz6Jfv34ICgpC69atsWTJEmi1TxZny8nJwYwZMxAWFobGjRtjwoQJSE5ONndRiagExDFCZlpHKO91xqQSiVmOSURUVGYNQiqVCsOGDUNSUhJWrVqFbdu2wcnJCZGRkWKQuX79OgYMGIA6derg22+/xYcffoiNGzdi3bp14nGmT5+OP/74A0uXLsXnn3+Oa9euYcyYMeYsKhGVkLOZB0vzOmNEVJ7M2g4dFxeHS5cu4fDhw/Dx8QEAzJ8/H82aNcOvv/6K3r17Y/Xq1XjhhRcwY8YMSCQS1KpVC//88w9OnToFAEhMTMSePXuwatUqNG3aFACwcOFCdOrUCadPn0bjxo3NWWQiKibzd43pp86zW4yIyp5Z33n8/PywZs0aMQQBgFSqa3RKS0sDAPzxxx8YOnQoJHmawPO29sTHxwMAmjdvLm57/vnn4ePjg5MnTzIIEZUzJzOvI5TBGWNEVI7MGoS8vb3RunVrg22bN29GdnY2wsPDkZ6ejqSkJLi4uODDDz/E4cOH4erqih49emDIkCGQyWRITEyEh4cH7O3tDY5TuXJl3L17t1Tlk8lsa2y4vr6st20oq3q7uehem5nZakilEkilpRvXk5mjaxFycVLAzq74Zef5Zr1tga3WuyyGDRYrCCUkJKB9+/b53n706FF4enqKvx84cAALFizAwIED4e/vLwaZuXPnYsCAAVi7di0uXLiAWbNmITMzE++99x6ysrKgUCiMjm1vb4+cnJziFNeIq6tDqe5fUbHetsXS9XZ20R1fACBXKuDqZPx6LQ7143kSlTwc4eHhVOLj8HzbFtabzKVYQcjHxwf79u3L93Y3Nzfx5+3btyM6Ohrdu3fHxIkTdQ9mp3u4l156CaNGjQIAvPjii0hOTsby5csxduxYKJVKqFQqo2Pn5OTAwaF0fwBpaVnQaLSF7/iMkMmkcHV1YL1tRFnWW6mQIVulQcKdVFTxKnl4AYD7KZkAADsJkJKSUez783yz3rbAVuvt5uYgDrGxlGIFIblcjjp16hS63/z58xEbG4tBgwYhKipKHA+k7/KqW7euwf5+fn7IzMxEcnIyfH19kZqaCpVKZdAydO/ePYOxRyWh0WihVtvOH5Ae621byqLezg5yZKs0eJiugrdb6b6gPMrUffFxsLcrVbl5vm0L620bBMHyj2H2mKUPQVFRUZg0aZLBoGiZTIbg4GCcOXPG4D7//PMPXF1d4e7ujiZNmkCr1YqDpgHdlPvExESEhISYu7hEVAJOZpw5li5eeZ6DpYmo7Jk1CB0/fhyxsbGIiIhAt27dkJSUJP7LyNA1eY8YMQK///47li5din///Rf79u3DmjVrEBkZCZlMBh8fH3Tp0gVTpkzB8ePHcfbsWYwfPx6hoaFo1KiROYtLRCVkzrWExCDEVaWJqByY9Z3n+++/B6CbKbZ582aD20aNGoXRo0ejWbNmWL16NRYtWoTVq1fD29sbw4YNw9ChQ8V9o6OjMXv2bHEcUatWrTBlyhRzFpWISsGcawlxQUUiKk9mDULR0dGIjo4udL+WLVuiZcuW+d7u6OiITz75BJ988ok5i0dEZqJf86e0l9kQBAEZ2foFFRmEiKjs2daCBERkFk4O+kUVS3cF+myVBhqtbjQkgxARlQcGISIqNnMNltbfX2EnhUIuK3W5iIiKi0GIiIrNXIOl0zk+iIjKGYMQERWbuQZLZ3DqPBGVMwYhIio2cwUhsUWIU+eJqJwwCBFRsTmZuWuMLUJEVF4YhIio2PTT51VqLVS5mhIfh0GIiMobgxARFZuDvQwyqe7yOaXpHst4PP2eg6WJqLwwCBFRsUkkEnFcT2mCUHo2W4SIqHwxCBFRiYjjhLJLvqgiZ40RUXljECKiEjHHgGmuI0RE5Y1BiIhKRD9gulRdY2wRIqJyxiBERCVijrWEMjhGiIjKGYMQEZVIaYOQWqNFVo5u6j0XVCSi8sIgREQlor8CfUnHCOkHWUsAOCnZIkRE5YNBiIhKpLQtQvr7OSrtIH28JhERUVljECKiEhGDUHYJW4Q4Y4yIrACDEBGVyJMWoZKtI8QZY0RkDRiEiKhESruOEBdTJCJrwCBERCXiLK4snQutIBT7/vouNQ6UJqLyxCBERCWiDzCCAGTlFL97jF1jRGQNGISIqETkdlLYy2UASjZz7EnXGNcQIqLywyBERCWmDzElCUL6QdZsESKi8sQgREQlVpoB07zgKhFZAwYhIiqx0iyqyHWEiMgaMAgRUYmVZi0hcbA0Z40RUTliECKiEnMqYYuQIAicNUZEVoFBiIhKTN+aU9wxQtkqDTRa3dpDDEJEVJ4YhIioxPIuqlgc+v3tZFIo5HwbIqLyw3cgIioxpxJOn88Qp87bQSLhleeJqPwwCBFRiZV01hjHBxGRtWAQIqISK+k6QgxCRGQtGISIqMRKOn1eXEyRU+eJqJwxCBFRiemDUE6uBrlqbZHvx8UUichaMAgRUYk52NtBP9a5OOOE2DVGRNaCQYiISkwqkYjdW8UZJ5SezSBERNaBQYiISqUkM8eeXHDVziJlIiIqKgYhIiqVkiyq+GQdIbYIEVH5YhAiolIpSYtQBscIEZGVYBAiolJxUhZ/dWkOliYia8EgRESl8mRRxaKtJaTRapGZoza4LxFReWEQIqJSKW7XWEb2k8Ckb00iIiovDEJEVCrFDkKP93Owt4NMyrcgIipffBciolIRg1ARZ409GR/E1iAiKn8MQkRUKsW98CoHShORNWEQIqJSKW7XWDqvM0ZEVoRBiIhKxTnPrDFBEArdn4spEpE1YRAiolLRj/XRCgKycjSF7q9fgdpZySBEROWPQYiISkVuJ4NCrnsrKcqAaY4RIiJrwiBERKVWnCvQc4wQEVkTBiEiKrXiDJjmdcaIyJowCBFRqRUnCD1pEeI6QkRU/hiEiKjUnEoQhNgiRETWgEGIiErNuYiLKgqCgHT99HnOGiMiK8AgRESlpp9CX1iLkCpXC7VGC4CDpYnIOjAIEVGp6Vt3CgtC+ttlUgmUCpnFy0VEVBizB6E7d+5g/PjxCA8PR0hICIYMGYLLly8b7LN371507doVDRs2ROfOnbFnzx6D21NSUjBhwgSEhIQgNDQUM2bMQFZWlrmLSkRmUtTrjeUdHySRSCxeLiKiwpg1CKlUKgwbNgxJSUlYtWoVtm3bBicnJ0RGRiI5ORkAcOzYMUycOBH9+/fH999/j379+mHy5Mk4dOiQeJwxY8bg5s2b2LhxIxYvXoxDhw5h+vTp5iwqEZnRkyvQqwvcT1xVmt1iRGQlzBqE4uLicOnSJcTExCAwMBB+fn6YP38+MjMz8euvvwIAfvnlF/j7+6NPnz6oUaMG+vXrh3r16uH3338HAJw+fRonTpzA3Llz0aBBA4SFhWHmzJn45ptvkJiYaM7iEpGZFLdFiOODiMhamDUI+fn5Yc2aNfDx8XnyAFLdQ6SlpQEAvLy8cPnyZRw7dgyCIOD48eO4evUqgoKCAOjClLe3N+rUqSMeIzQ0FBKJBPHx8eYsLhGZSVHXEeJiikRkbcy6opm3tzdat25tsG3z5s3Izs5GeHg4ACAiIgJnz55FZGQkZDIZNBoNhg8fju7duwMAEhMTUaVKFYNjKBQKuLu7486dO6Uqn0xmW2PD9fVlvW1DedbbzVkBAMhWaQAJYJdPGTJzdF1nLo5y2NmZp5w836y3LbDVepfFUMJiBaGEhAS0b98+39uPHj0KT09P8fcDBw5gwYIFGDhwIPz9/QHoBlOnpKRg6tSpCA4OxrFjx7Bo0SLUqFEDvXv3RlZWFhQKhdGx7e3tkZOTU5ziGnF1dSjV/Ssq1tu2lEe9Xd0cIZEAggDYKeTwcFWa3C9X0P1fycMRHh5O5i0Dz7dNYb3JXIoVhHx8fLBv3758b3dzcxN/3r59O6Kjo9G9e3dMnDhR3D569Gh07doV/fr1AwC8+OKLePjwIebPn4+ePXtCqVRCpVIZHTsnJweOjo7FKa6RtLQsaB6vYWILZDIpXF0dWG8bUd71dlTKkZGVi4S7DwGNxuQ+D1IyAQB2EiAlJcMsj1ve9S4vrDfrbQvc3BzEITaWUqwgJJfLDcbu5Gf+/PmIjY3FoEGDEBUVJU6TTU5OxrVr1xAYGGiwf6NGjbBy5UqkpqbC19cXP//8s8HtKpUKqampqFy5cnGKa0Sj0UKttp0/ID3W27aUV72dlXbIyMrFw0c58PUw/aXlUaZujJCDvZ3Zy8jzbVtYb9sgCJZ/DLPHLH0IioqKwqRJkwzWCnFzc4ODgwP++ecfg/v8888/cHV1haenJ0JCQnD37l3cvHlTvP3EiRMAgCZNmpi7uERkJk8GTOc/hZ7XGSMia2PWwdLHjx9HbGwsIiIi0K1bNyQlJYm3OTo6wsnJCQMGDMDKlSvh7e2NJk2aID4+HqtXr8a7774LAGjYsCGCg4Mxbtw4TJ8+HZmZmZg6dSp69OhhMBuNiKyLOIU+O/+ZYwxCRGRtzBqEvv/+ewC6mWKbN282uG3UqFEYPXo0xo4dCw8PD6xevRp37txB9erV8cEHH6BPnz4AAIlEgmXLlmHGjBmIjIyEvb09OnXqhMmTJ5uzqERkZkW58GoG1xEiIitj1iAUHR2N6OjoAveRyWQYNGgQBg0alO8+Xl5eWLJkiTmLRkQWVthaQlqtgMzHK0+zRYiIrIVtLUhARBbjpCz4CvSZOWoIT+1LRFTeGISIyCwKaxHSb1cqZPkuuEhEVNb4bkREZlHY9cY4UJqIrBGDEBGZRWFXoOcFV4nIGjEIEZFZFNY1xguuEpE1YhAiIrPIO31eMLEcLLvGiMgaMQgRkVnou7w0WkF3FfqniEFIySBERNaDQYiIzMJeLoPcTveWYqp77Mliipw6T0TWg0GIiMzGuYDLbKRzMUUiskIMQkRkNk7K/AdMc7A0EVkjBiEiMhtnh/xXl+ZgaSKyRgxCRGQ2TxZVNF5LiOsIEZE1YhAiIrMpaC0hXnmeiKwRgxARmU1+QUiVq4FKrdXtw+nzRGRFGISIyGz0g6Wfvt6YPhjJpBI42MvKvFxERPlhECIis8mvRUgcH6S0g0QiKfNyERHlh0GIiMwmvyDE8UFEZK0YhIjIbPJbUDGDiykSkZXiWvd5aLVaaDTG034rKq1WguxsGVSqHGg0xhfBfFY9XW+ZzA5SKTN/WXAS1xEyfB1xDSEislYMQgAEQUBaWjKystLLuyhmd/++FFqttryLUeaerreDgzNcXT05PsXC9F1fWTlqaLRayKSG1x5j1xgRWRsGIUAMQc7OHlAo7J+pD0uZTGJTrUF6+noLggCVKgfp6SkAADc3r3Iu2bPNSfnkLSUjSw1XJwUAXnmeiKyXzQchrVYjhiBnZ9fyLo7Z2dlJoVbbXotQ3norFPYAgPT0FLi4eLCbzIJkUikc7e2QmaNGelauGIR45XkislY2/4mg0WgAPPmwpGeT/vw+S2PArJWpmWMcI0RE1srmg5Des9QdRsZ4fsvOk+uN5QlC2QxCRGSdGISIyKxMtwhx+jwRWScGISIyK2f9FPo8awlxQUUislYMQlQuRo0ahlmzppd3McgCnJ5qEdIKgrjAIluEiMjaMAgRkVmJq0s/7g7LylFDEAxvIyKyFgxCRGRWzk8Nlta3DNkrZLCT8S2HiKwLF/XIhyAIUOWW3/o7Crm02DOdWrRoikGD3sa+fd9Brc7FsmVrUb16NaxcuRw//fQDMjLS8fzzdTB06HCEhjbH1atXEBnZB+vWbYG/fz0AwOTJ7+PUqZPYt+9XyGQyaLVadO/eAaNHj0fHjp3x3Xd7sHPnF7h16xakUgnq1q2HMWPGo169+gCA3r27oU2b9jh27AhSUpLxySfz0KBBIFatWoqfftqP3FwVXnutFwTBcJHHbds2Y8+enUhKuodKlbzRpUt3REYO4WyvCshJadg1xsUUiciaMQiZIAgCPt1yClf+e1huZXihuhsm9wsudhDYvfsrxMQsgVqtQY0aNTFz5hRcv34NU6dGw9u7Mo4cOYyJE9/D7NkxeOmlFqhSpSpOnjwGf/960Gg0OH06DpmZmbh06SJefLEBzp//G48ePUJYWAscOnQQixbNQ1TUFDRs2Bj379/HZ5/Nx5w5n2Djxm1iGXbt+hJz5y6Ci4sLatd+AZ99Nh9HjvyOjz6aBh+fKti0aT3OnDmNqlWrAQD++OMwNm/egJkzZ6NGjVr4+++z+OSTaahSpSo6duxs1ueVLE+cNfZ4XBAXUyQia8Z3pvxU0IaIjh07i60zCQm38NNP+7Fhw1b4+fkDAPr06Y8rVy5j27ZNeOmlFggPb4mTJ4+jf/+BuHDhb9jZyREQEIhTp+Lw4osNcPToH2jYsDFcXV3h5uaGSZM+RocOrwIAfH2roGvX7li4cJ5BGZo3D0dISDMAQGZmBn744XtMmBCFsLAWAIDJk6fi1Kk4cf/btxOgUMjh61sVvr6+8PX1RaVKleHj42vx54vM7+np81xMkYisGYOQCRKJBJP7BVe4rjEAqF69pvjzpUv/AABGjhxqsI9arYazswsAIDy8Jb79djdycrJx8uRxNGnSFL6+VREfH4d+/SJx9Ogf6NSpKwCgUaNg3LhxHRs3xuLmzRtISPgXV69eMbqoa/XqNcSf//33JnJzc1GvXgNxm729PerW9Rd/79ChM/bu/Rb/+19P1KpVGyEhzdCmTXv4+jIIVUT6lp+MrFwIgsA1hIjIqjEI5UMikcBeISvvYhSbvf2TS4UIgi6gLF++Fo6OTgb76a+31bhxU8jlcpw+fQpxcSfQsWNnVKlSBbt2fYm7d+/g8uVLmDWrNQDgp5/2Y9asaejQ4VUEBAThtdd64tq1q1i4cG6+ZdA3renLomdn9+RPz93dHRs2bMO5c2dx8uRxHD9+FF99tR1DhryDQYPeLt0TQmVOH3jUGgE5uRpeeZ6IrBqncDzDnn++DgDgwYP7qF69hvhv795vsW/fdwB0gSQ0NAx//HEI58+fQ5MmIQgKagSNRoN161ajdu0XUKVKVQDA1q0b0a1bD3z00XT06vUmGjUKxn//JQCA0eBnvZo1n4NCYY+zZ8+I29RqNS5fviT+/tNPP2D37p0ICmqEIUPewZo1usf55ZefLPK8kGXZy2Wwk+kCcHpWrjhGiIOlicgaMQg9w2rXroPw8JaYP/9T/PHHYfz3XwK2bv0cW7ZsRLVq1cX9WrRohX37vkOlSt6oVq067O2VCAgIwo8/7kPLlq3F/SpX9sFff53BP/9cxH//JWDHjq3YtetLAIBKpTJZBkdHR/Tu/SbWr1+NQ4d+xc2bNxAT8ynu308S91GpcrB8+WLs378Xd+7cxpkzf+L06VMICAiy0DNDliSRSPJcb0zNMUJEZNXYNfaMmzVrDlasWIb582fj0aM0VK1aHZMmfYxXX+0q7hMWFg6NRoPg4KbitqZNQ3HqVBxatHgShMaNm4h582Zh1KhhUCjkeOGFupgyZQamTfsQFy+eR8OGjU2W4Z13RkGhsMfChXORmZmJdu1eQXh4K/H2rl174OHDh9i4MRb37iXCxcUFbdq0x4gRYyzwjFBZcHaQ42G6CulZuQxCRGTVJEJ+fRrPoJSUDKjVhmNVcnNVePDgDry8qkAuV5RTySzHzk5qVGdb8HS9n/XzDOjq7OHhZPLvvKzN3XoK/9xKxfDXGmDfsZv4NzEd773REEF1vMz+WNZU77LEerPetsDT0wkyCy/Eyq4xIjK7vFPoM9giRERWjEGIiMxOP4Ve1zWmNthGRGRNGISIyOz0g6VT01XIydUAYIsQEVknBiEiMjt96ElMzgQASCSAgz1bhIjI+jAIEZHZ6dcMSkzRBSEnpRxSXkCXiKwQgxARmZ2+RSg5LcfgdyIia8MgRERm9/TlNBiEiMhaMQgRkdk9HXwYhIjIWjEIEZHZPR18OHWeiKwVgxBZlFqtxo4dW8Xf161bjd69u5n9cSx1XCqZp4MPW4SIyFoxCJFFHTiwH0uXLirvYlAZk0mlcLCXib878crzRGSlGITIomzoUnb0lLzhhy1CRGSt2HGfD0EQALWq/Apgp4CkBOuuHD16BLGxq3DjxjU4ODjipZdaYNSocbhy5RLGjXsXM2fOwapVS5GYmIiAgEB89NF0bN++Gfv374WdnRxvvNEHkZFDxOP98MP3+OKLrbh16194enqia9fXEBExCDKZ7tt+YuJdrF69HHFxJ5CZmYGgoEYYOXIsXnjBD/v2fYfZs2cAAFq0aIolS1aJx92yZSO+/vpLPHz4EA0aBGDixI9Qo0ZNAEB6ejqWL1+M338/iNzcXPj7v4iRI8egXr364v2/+WYXtm3bhKSkJISEhKJKlaoleprJcpwd5Lj/MFv8mYjIGjEImSAIAjK/nQVt4pVyK4PMxw8O3T8sVhhKTU3FRx99gFGjxuGll1rg3r1EfPLJNKxYsRgdOrwKjUaDTZvWY9q0T6BWq/HBB+9h4MC+6Nr1NaxZ8zl++ukHrF27Ei1atEadOi/gyy+3YdWqZRg1ahxCQprh/PlzWLhwLh4+fIixYycgMzMDI0YMQdWq1TBnzgLI5QqsX78Go0a9jY0bt6N9+1eQnp6OJUsW4Jtv9sPV1Q2nT8fj7t07+OuvM5g/fzFyc1WIjp6KOXOisXz5WgiCgA8+GAOFQom5cz+Ds7Mz9u/fixEjhmD16g2oW7ceDhzYj4UL52Ls2PfRtGkoDh8+iDVrVqByZR8LnhEqrrzh5+np9ERE1oJdY/mQoOKtgpuUlAiVSgUfH1/4+lZBUFAjxMR8hl693hL3GTp0OOrVq4+AgCA0aRICBwcHjBw5BjVrPoeIiIEAgGvXrkAQBGzZ8jl69nwTPXu+gRo1aqJjx84YMmQ4du/+Cunp6fjxxx/w8GEqoqPnon79APj51cX06Z/A3l6JXbu+hL29Es7OzgAAL69KkMt1H4Z2dnaYOjUaL7zghxdfbIDXXuuJixfPAwDi40/i3Lm/EB39KRo0CMBzz9XCO++8iwYNAvHVV18AAHbu3IGXX+6Anj3fQM2az6F//4EID29Zhs80FUXeIMQWISKyVmwRMkEikcCh+4cVrmvMz88fL7/cEVFR4+DlVQkhIc3QsmUrhIe3xtmzfwIAqlevIe7v4OCAKlWqio9jb68EAOTm5iI1NQXJyQ8QFNTI4DEaNw6GWq3GzZs3cPXqFdSo8Rw8PDzE2+3tlahfvwGuXr2abzk9Pb3g5OQs/u7i4oqcHN0KxJcuXYQgCOjVq6vBfVQqlbjPtWtX8PLLHQ1uDwgIwuXLl4ryNFEZcWIQIqIKgEEoHxKJBJDbl3cxim369FkYPPhtHDv2fzh58jimT5+CoKBG4rgfOzvDU55f2MpvkLNWK+Q5Tn77aGFnJzN5GwBIpfk3RGq1Wjg5OWHdui1Gt+lblAAJBEFrcNvT9aLyZ9gixPNDRNaJXWPPkL//PoclSxagZs1aePPNvpg/fzE++mga4uNPIiUlpVjH8vT0gqenl9iSpHfmzGnI5XJUq1Ydder44datm0hJSRZvz8nJwcWLF1CrVm0A+Qet/NSu/QIyMjKQm5uL6tVriP+2bv0cf/xxCADg51cXZ8+eMbjfxYsXivU4ZHn6IKSQSyEvIBgTEZUnswehf//9FyNGjEDTpk3RtGlTjB8/HomJiQb7HD16FD179kTDhg3RqVMn7N271+D2nJwczJgxA2FhYWjcuDEmTJiA5ORkUMGcnJywa9dXWLFiCRISbuHatSv4+eefUL16Tbi7uxf7eP/7XwR27foSu3fvRELCLfz0036sX78G3bu/DmdnZ7zySie4ubnj448n4cKFv3HlymXMnDkFWVlZeO21ngB03W+ALqjk5GQX+pjNmoXBz68upk2bjFOn4pCQcAtLly7Evn3fieGqf/+BOHz4ILZt24Rbt/7Fzp1f4Lfffil2/ciy9IsqsluMiKyZWYOQSqXCwIEDodVqsW3bNmzevBn37t3D8OHDxa6Wq1ev4p133kHLli2xa9cuvPHGG5g4cSKOHj0qHmf69On4448/sHTpUnz++ee4du0axowZY86iPpNq1Xoes2bNx6lTcRg0qC9GjBgCmUyGBQuWlGgq/v/+1x/vvjsWO3ZsQ//+byA2diX69YvEmDETAADOzs5YunQ1XFxcMXbsSIwcORQ5OTlYuXIdqlatBgAIDg5B/foBGDFiMI4c+aPQx5TJZFi0aAXq1auPqVMnITKyD/788zRmzZqPJk1CAAAvvdQC06Z9gr17v0VkZB8cOnQQffr0L3b9yLK8XHVjzjwf/09EZI0kghlXvLt58yYWLlyIadOmwdPTEwDw888/491338XRo0fh6emJqVOn4sKFC/jqq6/E+02YMAGpqalYt24dEhMT0aZNG6xatQqtW7cGAFy/fh2dOnXCF198gcaNG5e4fCkpGVCrDceW5Oaq8ODBHXh5VYFcrijxsa2VnZ3UqM624Ol6P+vnGdDV2cPDyeTfeXkQBAFH/76LWr6uqFrJyWKPY231LiusN+ttCzw9nSCTWXYUj1mP/txzz2Hx4sViCLp9+za2b9+OBg0aiDOL4uLiEBYWZnC/5s2bIz4+HoIgID4+Xtym9/zzz8PHxwcnT540Z3GJyIIkEgleCqhi0RBERFRaFpvKMXjwYBw5cgRubm74/PPPxa6Zu3fvwtfX12DfypUrIysrCykpKUhMTISHhwfs7e2N9rl7926pymQqVWq1FW+9oKLS94ZJJIAtXemioHrLZBLY2T2bcwT0f9+W/vZkbVhv1tsW2Gq9SzCqo9iKFYQSEhLQvn37fG/Xd38BwAcffICxY8di+fLlGDhwIPbs2YMqVaogOzsbCoVh14T+d5VKhaysLKPbAcDe3l5cR6akXF0djLZlZ8tw/77UJj4gbU3eemu1EkilUri5OUKpfLbHrJj6O7cFrLdtYb3JXIoVhHx8fLBv3758b3dzcxN/fvHFFwEAn332Gdq2bYuvv/4ao0aNgr29PVQqw4UK9b87ODhAqVQa3Q7oZpLpZyCVVFpaFjQaw75VlSoHWq0WGo3wzPW7SiS6MKDRaG2uRejpems0ArRaLR4+zERWlqZ8C2ghMpkUrq4OJv/On2WsN+ttC2y13m5uDgWuPWcOxQpCcrkcderUyff2O3fu4MyZM+jUqZO4zdHREdWrV8e9e/cAAFWqVBF/1rt37x4cHR3h4uICX19fpKamQqVSGbQM3bt3Dz4+pbuWlEajNQo7Gs2zmxD0IcCWQhBQcL2fxcD7NFN/57aA9bYtrLdtKIvPL7PGrIsXL2Ls2LG4du2auC0tLQ3Xr18XA1TTpk1x4sQJg/sdO3YMwcHBkEqlaNKkCbRarThoGtDNGktMTERISIg5i2vAjJPnyArx/BIRkSlmDULh4eGoV68eoqKicO7cOfz9998YM2YMPDw80KtXLwBAREQEzp49i5iYGFy9ehXr16/H/v37MXToUAC67rcuXbpgypQpOH78OM6ePYvx48cjNDQUjRo1MmdxAejWrQF0XWT07NKfX5mMl3ogIqInzLqOEKDrwpo7dy6OHDkClUqFFi1aYPLkyahSpYq4z+HDhzF//nzcuHED1atXx+jRo9G5c2fx9szMTMyePRs//vgjAKBVq1aYMmWKwcU9SyK/9RcePnyArKx0ODt7QKGwL9Hig9ZKJpM8091/+dHXWxAEqFQ5SE9PgYODM9zcvMq7aBZjq+uMsN6sty2w1XqXxTpCZg9C1iy/PyBBEJCWloysrPRyKJVlSaVSaLW286LRe7reDg7OcHX1fKZC7tNs9Y2S9Wa9bYGt1rssghD7CaBb+M3NzQsuLh7QaNTlXRyzkckkcHNzxMOHmTbVKvR0vWUyO4vPOiAiooqJQSgPqVQKqfTZufyCnZ0USqUSWVkam/oGYav1JiKi4uPXZCIiIrJZDEJERERksxiEiIiIyGbZ1KwxW1qWXE9/qQlbw3rbFtbbtrDetkMqlVh8tq9NBSEiIiKivNg1RkRERDaLQYiIiIhsFoMQERER2SwGISIiIrJZDEJERERksxiEiIiIyGYxCBEREZHNYhAiIiIim8UgRERERDaLQYiIiIhsFoMQERER2SwGISIiIrJZDEJERERks56JIKTVarFkyRK0bNkSjRo1wttvv41bt27lu39KSgomTJiAkJAQhIaGYsaMGcjKyirDEpdeamoqpk6dilatWiE4OBj/+9//EBcXl+/+K1euhL+/v9G/iigxMdFkXXbt2mVy/2fhfB8/ftxknf39/dG+fXuT94mPjze5//Hjx8u49CWzevVqREREGGy7cOEC+vfvj0aNGqFdu3bYtGlTocf54Ycf0LlzZwQFBaFHjx44evSopYpsFqbq/euvv6JXr15o3Lgx2rVrh7lz5yI7OzvfY2g0GgQFBRmd+6VLl1q6+CVmqt5TpkwxqkO7du0KPE5FP98RERH5vtb37NmT73EGDRpktP/Tz2d5K+xz6+jRo+jZsycaNmyITp06Ye/evYUec+vWrWjfvj2CgoLQt29fnD9/vvgFE54BS5cuFZo1ayYcPHhQuHDhgjB48GChQ4cOQk5Ojsn9+/fvL/Tq1Us4d+6c8H//939C27ZthYkTJ5ZxqUtn0KBBQteuXYWTJ08K165dE2bMmCEEBQUJV69eNbn/2LFjhQ8++EC4d++ewb+K6LfffhMCAwOFxMREg7pkZWWZ3P9ZON85OTlG5+6nn34S/P39hZ07d5q8z9atW4WXX37Z6H75vS6syZYtW4R69eoJ/fv3F7clJycLzZo1EyZPnixcuXJF2LlzpxAYGJhv/QVBEI4ePSo0aNBA+Pzzz4UrV64Ic+bMEQICAoQrV66URTWKzVS9T548Kbz44ovCypUrhevXrwu//fab0KpVK2HSpEn5HufKlStC3bp1hQsXLhic+/T09LKoRrGZqrcgCELv3r2FhQsXGtThwYMH+R7nWTjfKSkpBvVNTEwU+vbtK3Tp0qXA8xcWFiZs27bN4L4pKSllUIuiK+hz68qVK0JgYKCwcOFC4cqVK0JsbKxQv3594f/+7//yPd6uXbuEoKAg4ZtvvhEuX74sfPDBB0JoaGiBfyOmVPgglJOTIzRu3FjYunWruO3hw4dCUFCQ8N133xntf+rUKaFu3boGL4zff/9d8Pf3F+7evVsmZS6tGzduCHXr1hXi4uLEbVqtVnj55ZeFzz77zOR9Xn31VWHDhg1lVELLWrNmjdCtW7ci7fssnG9TMjIyhLZt2xb4YTht2jRh+PDhZViq0rt7967wzjvvCI0aNRI6depk8AGxatUqoUWLFkJubq64bcGCBUKHDh3yPd7gwYOFsWPHGmx76623hI8//tjsZS+Nguo9YcIEYeDAgQb77969W2jQoEG+oXbv3r1CcHCwRctsDgXVW6vVCo0aNRJ++umnIh/vWTjfT9u8ebMQEBCQ75dcQRCE+/fvC3Xr1hX+/vtvSxTXLAr73Pr444+F3r17G9xn/PjxwuDBg/M9ZocOHYR58+aJv+fm5gqtW7cWVq1aVayyVfiusYsXLyIjIwNhYWHiNldXV9SvXx8nT5402j8uLg7e3t6oU6eOuC00NBQSiQTx8fFlUubS8vDwwJo1axAYGChuk0gkkEgkSEtLM9pfpVLhxo0bqF27dlkW02L++ecfg/NXkGfhfJuyatUqZGVlISoqKt99ivM8WYu///4bcrkc3377LRo2bGhwW1xcHEJDQ2FnZydua968OW7cuIH79+8bHUur1eLUqVMG7w0A0KxZM5PvDeWpoHoPHjzY6DxLpVLk5uYiPT3d5PEqyrkvqN7//vsvMjMzi/y+9ayc77ySk5Px2WefYcSIEQU+D//88w8kEgmef/55SxTXLAr73IqLizM6d82bN0d8fDwEQTA63oMHD3Djxg2D+9jZ2aFp06bFPt92he9i3e7evQsAqFKlisH2ypUri7fllZiYaLSvQqGAu7s77ty5Y7mCmpGrqytat25tsO3HH3/EzZs38eGHHxrtf+XKFWg0Gvz444+YNWsWcnJyEBISgg8++ACVK1cuq2KbzaVLl+Dh4YF+/frh+vXreO655zBixAi0atXKaN9n4Xw/LTk5GRs3bsSECRPg7u6e736XL1+Gh4cHevbsicTERNStWxfjxo1DUFBQ2RW2mNq1a5fvGJC7d++ibt26Btv0f7937txBpUqVDG5LS0tDZmYmfH19je5j6r2hPBVU7/r16xv8npubi40bNyIgIACenp4m73Pp0iWo1WoMGTIEFy9ehI+PDyIjI/Haa6+ZveylUVC9L126BADYvHkzDh8+DKlUilatWmHcuHFwcXEx2v9ZOd95rV27FkqlEkOGDClwv0uXLsHFxQUzZ87EkSNH4OjoiE6dOmHkyJFQKBTmKnapFPa5tXv3bpPnLisrCykpKUZ/6wV99l+8eLFYZavwLUL6Qa9Pn2x7e3vk5OSY3N/UH0Z++1cEp06dwuTJk9GhQwe0adPG6Hb9G4qDgwMWL16MWbNm4dq1axgwYECBAy6tkVqtxrVr1/Dw4UOMHj0aa9asQaNGjTBs2DCTgyKfxfO9bds2uLi44K233sp3nzt37uDRo0fIzMzElClTsGLFClSqVAn9+/fHlStXyrC05pOdnW3ydQ7A5LnU/20X9b2hIlCr1Zg4cSIuX76MadOm5bvf5cuXkZqaioiICKxbtw4dO3bE5MmTsXPnzjIsbelcunQJUqkUlStXxqpVqzBp0iT88ccfGDlyJLRardH+z9r5Tk9Px5dffokhQ4aIf+f5uXTpEnJychAUFITY2FiMGDECX331FaZMmVJGpS2+pz+3TL2+9b+rVCqj+xf3s78gFb5FSKlUAtA9UfqfAd0bo4ODg8n9TT2pOTk5cHR0tFxBLeTnn3/G+++/j+DgYMTExJjcp0ePHmjVqpVBovbz80OrVq3w66+/onPnzmVV3FKzs7PD8ePHIZPJxPMdEBCAy5cvY926dUZNq8/a+QaAPXv2oEePHgZ/70+rUqUKTp48CQcHB8jlcgBAYGAgzp8/j82bN2PGjBllVVyzMXUu9W94ps6l/sPD1H1MvTdYu/T0dLz33ns4ceIEli1bVmDL3vfffw+NRgMnJycAQL169XD79m2sW7cOvXv3Lqsil8qIESPQt29feHh4AADq1q0Lb29vvPnmm/jrr7+MupSetfP9888/Q6VSoVevXoXuO3PmTERFRcHNzQ2A7rmSy+UYN24cJk6caNRaWt5MfW7Z29sbnTv97/l9lufdR68k57vCtwjpm8Xu3btnsP3evXvw8fEx2t/X19doX5VKhdTU1ArXTbRlyxaMHj0abdu2xapVqwr81vB0s2LlypXh7u5udU3GReHk5GQUAvz8/JCYmGi077N0vgHdmLhbt26hW7duhe7r6uoqhiBAN66kTp06Jp+nisDUudT/buq17u7uDkdHxyK/N1ize/fuoV+/fvjzzz+xbt06oy6GpymVSjEE6dWtW7dCvd6lUqkYgvT8/PwAwGQ9nqXzDejCQuvWreHq6lrovnZ2dmII0ivouSpP+X1uValSxeS5c3R0NNkVWtzP/oJU+CBUr149ODs7G6yNkpaWhvPnzyMkJMRo/5CQENy9exc3b94Ut504cQIA0KRJE8sX2Ey2bduG6Oho9OvXDwsXLiywH3jRokXo2LGjwYCzhIQEpKSk4IUXXiiL4prN5cuXERwcbLQWzrlz50zW5Vk533pxcXHw8vJCvXr1Ctzv8OHDaNy4scF6Wmq1GhcvXqxw51wvJCQE8fHx0Gg04rZjx47h+eefh5eXl9H+EokEwcHB4vnWO378OJo2bWrx8prLw4cPERkZieTkZGzdutXk+1peaWlpCA0NNVpX66+//hI/HCuCiRMnYuDAgQbb/vrrLwAw+Tf8rJxvPVODh/MTERGByZMnG2z766+/IJfLUatWLQuUrmQK+txq2rSp0bk7duwYgoODIZUaRxUvLy88//zzBp8FarUacXFxhb5Gnlbhg5BCoUD//v0RExODX375BRcvXsS4cePg6+uLDh06QKPRICkpSew/btiwIYKDgzFu3DicPXsWx44dw9SpU9GjR48K863h+vXrmD17Nl555RW88847uH//PpKSkpCUlIRHjx5BpVIhKSlJbDJ85ZVX8N9//2H69Om4fv06Tp48idGjRyM4OBgtW7Ys59oUT506dVC7dm3MnDkTcXFxuHr1Kj799FP8+eefGDFixDN5vvM6f/58vgthJiUlISMjAwAQHBwMDw8PREVF4dy5c/jnn38QFRWF1NRUow+XiqJXr15IT0/HRx99hCtXrmDXrl3YuHEj3nnnHXGfR48eITk5Wfx90KBB2Lt3LzZs2ICrV69i3rx5uHDhAiIjI8ujCiXy6aef4tatW5g/fz48PT3F13pSUpIYClNTU5GamgpA1xLYvHlzLFq0CIcOHcKNGzewZs0afPvttxg9enQ51qR4OnbsiKNHj2LZsmX4999/cejQIXz44Yfo2rWrOCPuWTzfgG6MX0pKSr5feDIyMpCUlCT+3rFjR3zzzTfYvn07bt26hX379mHevHkYMmQInJ2dy6rYBSrscysiIgJnz55FTEwMrl69ivXr12P//v0YOnSoeIy8f+eAbkblhg0bsHv3bly5cgUffvghsrOzi9/9W6zJ9lZKrVYL8+bNE5o3by40atRIePvtt4Vbt24JgiAIt27dEurWrSt8/fXX4v73798XRo8eLTRq1Eho1qyZMG3aNCE7O7u8il9sK1euFOrWrWvyX1RUlHDs2DGhbt26wrFjx8T7/N///Z/w1ltvCY0aNRJCQ0OFyZMnC6mpqeVYi5JLSkoSJk2aJISHhwuBgYHCW2+9JZw8eVIQhGfzfOc1dOhQ4b333jN5W926dYUlS5aIv9+8eVMYPXq0EBoaKjRs2FAYPHiw8M8//5RVUUstKirKaH2VM2fOCG+++aYQEBAgtG3bVti8ebPRfdq2bWuwbffu3cIrr7wiBAYGCq+//nqBC7RZg7z1VqvVQmBgYL6vd/37XP/+/Q2eq0ePHgmzZ88WWrduLQQEBAivvfaacODAgXKpT1GZOt/79u0TevToIQQFBQnh4eHCnDlzDF67z9r51jtz5ozR+md5LVmyRKhbt67Bti1btgivvvqq+NpYuXKloNFoLFbu4irsc0sQBOHQoUNC165dhYCAAKFTp07C3r17DY7x9N+5IAhCbGys0KpVKyEoKEjo27evcP78+WKXTSIIJiboExEREdmACt81RkRERFRSDEJERERksxiEiIiIyGYxCBEREZHNYhAiIiIim8UgRERERDaLQYiIiIhsFoMQERER2SwGISKqEBISEuDv7290Da2SmDRpEtq1a2eGUhFRRWdX3gUgIiqKypUrY8eOHahZs2Z5F4WIniEMQkRUISgUCjRq1Ki8i0FEzxh2jRGRWXz11Vfo0qULAgIC0KZNGyxdulS8OvqkSZMQERGBnTt3om3btmjcuDEiIyNx8eJF8f5arRaLFi1Cu3btEBAQgHbt2mHBggXIzc0FYLpr7MaNGxgzZgzCw8PRqFEjREREID4+3qBcDx8+xOTJkxEaGoqQkBDMnz8fWq3WqPw///wzevbsicDAQISHh+OTTz5BZmameHt2djamT5+OVq1aISAgAJ06dcK6devM+hwSUdljixARldrq1auxaNEi9O/fH5MnT8aFCxewdOlS3LlzB7NnzwYAXLhwAdeuXcP48ePh5uaGJUuWoH///ti3bx8qV66MtWvXYvv27YiKikKNGjVw5swZLFq0CHK5HGPGjDF6zCtXruDNN99ErVq1MGXKFMjlcmzatAmRkZFYv349QkNDodVqMXToUPz333+IioqCu7s7YmNj8ddff6Fy5crisb777ju8//776NatG9577z38999/WLRoEa5cuYINGzZAIpFg9uzZ+OOPPxAVFYVKlSrh8OHDmDdvHtzd3dGrV68ye66JyLwYhIioVB49eoQVK1bgrbfewpQpUwAALVq0gLu7O6ZMmYJBgwaJ+61atQpNmzYFAAQFBeHll1/Gpk2b8P777+PEiRMICAgQQ0VoaCgcHBzg4uJi8nGXLVsGhUKBTZs2wdnZGQDQpk0bdO3aFfPmzcPOnTtx+PBhnD17FmvXrkWrVq0AAGFhYQYDpQVBQExMDFq2bImYmBhxe61atTBw4EAcOnQIbdq0wYkTJxAeHo4uXboAAJo1awZHR0d4eXmZ8+kkojLGrjEiKpXTp08jOzsb7dq1g1qtFv/pw8aRI0cAANWrVxdDEKAb/Ny4cWOcPHkSgC5YHDlyBH379kVsbCyuXLmC/v3747XXXjP5uCdOnEDbtm3FEAQAdnZ26NKlC86dO4eMjAzExcVBLpejZcuW4j6Ojo5o3bq1+Pu1a9dw9+5do/KHhITA2dlZLH+zZs3w5Zdf4u2338aWLVtw69YtvPvuu2jTpo15nkgiKhdsESKiUklNTQUADBs2zOTt9+7dAwD4+PgY3ebl5YW///4bADB06FA4OTnh66+/RkxMDObPnw8/Pz9MmTIFzZs3N7rvw4cPUalSJaPtlSpVgiAISE9Px8OHD+Hu7g6JRGKwj7e3t1H5Z8yYgRkzZuRb/o8++gi+vr749ttvER0djejoaDRu3BjTp09HvXr1TNadiKwfgxARlYqrqysAICYmBrVq1TK6vVKlSli8eDFSUlKMbrt//77YtSSVStGvXz/069cPDx48wKFDh7Bq1SqMHj1abJXJy83NDffv3zfanpSUBADw8PCAh4cHUlJSoNFoIJPJxH304Sdv+SdOnIjQ0FCTjwPoZq2NGDECI0aMwO3bt3Hw4EGsWLECEyZMwN69e/N7eojIyrFrjIhKpWHDhpDL5UhMTERgYKD4z87ODgsXLkRCQgIA3Qyvq1evivdLTEzE6dOnERYWBgDo06cPPvnkEwC6lqKePXuiX79+SEtLQ3p6utHjhoSE4ODBgwa3aTQa7N27F4GBgVAoFAgLC4NarcbPP/8s7qNSqQyCVe3ateHl5YWEhASD8vv4+GDBggU4f/48srOz0bFjR6xfvx4AULVqVfTr1w9dunTB7du3zfhsElFZY4sQEZWKh4cHhg4disWLFyM9PR3NmjVDYmIiFi9eDIlEInYbCYKA4cOHY9y4cZDJZFi2bBnc3NwQEREBQBds1q9fj0qVKqFx48ZITEzEhg0bEBoaCk9PT4Op7AAwatQoHD58GAMGDMCwYcMgl8vFsTuxsbEAdAOjW7RogSlTpuDBgweoVq0aNm3ahOTkZLElSiaTYdy4cZg6dSpkMhnatm2LtLQ0rFixAomJiWjQoAGUSiUaNGiAZcuWQS6Xw9/fH9evX8fu3bvRsWPHMny2icjcJIIgCOVdCCKq+LZu3Ypt27bh5s2bcHNzQ1hYGMaPH4+qVati0qRJOHHiBN5++20sX74cWVlZeOmllxAVFYXq1asDANRqNVauXIlvv/0Wd+/ehYuLC9q1a4cJEybAw8MDCQkJaN++PT799FP07NkTgG5K/sKFCxEXFweJRIKgoCCMGjXKYFB2VlYWYmJisHfvXuTk5KBz585wdHTEL7/8gl9//VXcb9++fYiNjcXly5fh6OiI4OBgvPfee/D39wcApKen47PPPsMvv/yCpKQkeHl5oXPnzhg7diyUSmUZPtNEZE4MQkRkcfoglDd4EBFZA44RIiIiIpvFIEREREQ2i11jREREZLPYIkREREQ2i0GIiIiIbBaDEBEREdksBiEiIiKyWQxCREREZLMYhIiIiMhmMQgRERGRzWIQIiIiIpv1/2Po3UtxEkgJAAAAAElFTkSuQmCC", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "# 获取参数\n", - "cfg = Config() \n", - "# 训练\n", - "env, agent = env_agent_config(cfg)\n", - "res_dic = train(cfg, env, agent)\n", - " \n", - "plot_rewards(res_dic['rewards'], title=f\"training curve on {cfg.device} of {cfg.algo_name} for {cfg.env_name}\") \n", - "# 测试\n", - "res_dic = test(cfg, env, agent)\n", - "plot_rewards(res_dic['rewards'], title=f\"testing curve on {cfg.device} of {cfg.algo_name} for {cfg.env_name}\") # 画出结果" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.12 ('easyrl')", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.12" - }, - "orig_nbformat": 4, - "vscode": { - "interpreter": { - "hash": "f5a9629e9f3b9957bf68a43815f911e93447d47b3d065b6a8a04975e44c504d9" - } - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/projects/notebooks/PolicyGradient.ipynb b/projects/notebooks/PolicyGradient.ipynb deleted file mode 100644 index b6326da..0000000 --- a/projects/notebooks/PolicyGradient.ipynb +++ /dev/null @@ -1,202 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1. 定义算法\n", - "\n", - "最基础的策略梯度算法就是REINFORCE算法，又称作Monte-Carlo Policy Gradient算法。我们策略优化的目标如下：\n", - "\n", - "$$\n", - "J_{\\theta}= \\Psi_{\\pi} \\nabla_\\theta \\log \\pi_\\theta\\left(a_t \\mid s_t\\right)\n", - "$$\n", - "\n", - "其中$\\Psi_{\\pi}$在REINFORCE算法中表示衰减的回报(具体公式见伪代码)，也可以用优势来估计，也就是我们熟知的A3C算法，这个在后面包括GAE算法中都会讲到。\n", - "\n", - "### 1.1. 策略函数设计\n", - "\n", - "既然策略梯度是直接对策略函数进行梯度计算，那么策略函数如何设计呢？一般来讲有两种设计方式，一个是softmax函数，另外一个是高斯分布$\\mathbb{N}\\left(\\phi(\\mathbb{s})^{\\mathbb{\\pi}} \\theta, \\sigma^2\\right)$，前者用于离散动作空间，后者多用于连续动作空间。\n", - "\n", - "softmax函数可以表示为：\n", - "$$\n", - "\\pi_\\theta(s, a)=\\frac{e^{\\phi(s, a)^{T_\\theta}}}{\\sum_b e^{\\phi(s, b)^{T^T}}}\n", - "$$\n", - "对应的梯度为：\n", - "$$\n", - "\\nabla_\\theta \\log \\pi_\\theta(s, a)=\\phi(s, a)-\\mathbb{E}_{\\pi_\\theta}[\\phi(s,)\n", - "$$\n", - "高斯分布对应的梯度为：\n", - "$$\n", - "\\nabla_\\theta \\log \\pi_\\theta(s, a)=\\frac{\\left(a-\\phi(s)^T \\theta\\right) \\phi(s)}{\\sigma^2}\n", - "$$\n", - "但是对于一些特殊的情况，例如在本次演示中动作维度=2且为离散空间，这个时候可以用伯努利分布来实现，这种方式其实是不推荐的，这里给大家做演示也是为了展现一些特殊情况，启发大家一些思考，例如Bernoulli，Binomial，Gaussian分布之间的关系。简单说来，Binomial分布，$n = 1$时就是Bernoulli分布，$n \\rightarrow \\infty$时就是Gaussian分布。\n", - "\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 1.2. 模型设计\n", - "\n", - "前面讲到，尽管本次演示是离散空间，但是由于动作维度等于2，此时就可以用特殊的高斯分布来表示策略函数，即伯努利分布。伯努利的分布实际上是用一个概率作为输入，然后从中采样动作，伯努利采样出来的动作只可能是0或1，就像投掷出硬币的正反面。在这种情况下，我们的策略模型就需要在MLP的基础上，将状态作为输入，将动作作为倒数第二层输出，并在最后一层增加激活函数来输出对应动作的概率。不清楚激活函数作用的同学可以再看一遍深度学习相关的知识，简单来说其作用就是增加神经网络的非线性。既然需要输出对应动作的概率，那么输出的值需要处于0-1之间，此时sigmoid函数刚好满足我们的需求，实现代码参考如下。" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import torch\n", - "import torch.nn as nn\n", - "import torch.nn.functional as F\n", - "class PGNet(nn.Module):\n", - " def __init__(self, input_dim,output_dim,hidden_dim=128):\n", - " \"\"\" 初始化q网络，为全连接网络\n", - " input_dim: 输入的特征数即环境的状态维度\n", - " output_dim: 输出的动作维度\n", - " \"\"\"\n", - " super(PGNet, self).__init__()\n", - " self.fc1 = nn.Linear(input_dim, hidden_dim) # 输入层\n", - " self.fc2 = nn.Linear(hidden_dim,hidden_dim) # 隐藏层\n", - " self.fc3 = nn.Linear(hidden_dim, output_dim) # 输出层\n", - " def forward(self, x):\n", - " x = F.relu(self.fc1(x))\n", - " x = F.relu(self.fc2(x))\n", - " x = torch.sigmoid(self.fc3(x))\n", - " return x" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 1.3. 更新函数设计\n", - "\n", - "前面提到我们的优化目标也就是策略梯度算法的损失函数如下：\n", - "$$\n", - "J_{\\theta}= \\Psi_{\\pi} \\nabla_\\theta \\log \\pi_\\theta\\left(a_t \\mid s_t\\right)\n", - "$$\n", - "\n", - "我们需要拆开成两个部分$\\Psi_{\\pi}$和$\\nabla_\\theta \\log \\pi_\\theta\\left(a_t \\mid s_t\\right)$分开计算，首先看值函数部分$\\Psi_{\\pi}$，在REINFORCE算法中值函数是从当前时刻开始的衰减回报，如下：\n", - "$$\n", - "G \\leftarrow \\sum_{k=t+1}^{T} \\gamma^{k-1} r_{k}\n", - "$$\n", - "\n", - "这个实际用代码来实现的时候可能有点绕，我们可以倒过来看，在同一回合下，我们的终止时刻是$T$，那么对应的回报$G_T=\\gamma^{T-1}r_T$，而对应的$G_{T-1}=\\gamma^{T-2}r_{T-1}+\\gamma^{T-1}r_T$，在这里代码中我们使用了一个动态规划的技巧，如下：\n", - "```python\n", - "running_add = running_add * self.gamma + reward_pool[i] # running_add初始值为0\n", - "```\n", - "这个公式也是倒过来循环的，第一次的值等于：\n", - "$$\n", - "running\\_add = r_T\n", - "$$\n", - "第二次的值则等于：\n", - "$$\n", - "running\\_add = r_T*\\gamma+r_{T-1}\n", - "$$\n", - "第三次的值等于：\n", - "$$\n", - "running\\_add = (r_T*\\gamma+r_{T-1})*\\gamma+r_{T-2} = r_T*\\gamma^2+r_{T-1}*\\gamma+r_{T-2}\n", - "$$\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import torch\n", - "from torch.distributions import Bernoulli\n", - "from torch.autograd import Variable\n", - "import numpy as np\n", - "\n", - "class PolicyGradient:\n", - " \n", - " def __init__(self, model,memory,cfg):\n", - " self.gamma = cfg['gamma']\n", - " self.device = torch.device(cfg['device']) \n", - " self.memory = memory\n", - " self.policy_net = model.to(self.device)\n", - " self.optimizer = torch.optim.RMSprop(self.policy_net.parameters(), lr=cfg['lr'])\n", - "\n", - " def sample_action(self,state):\n", - "\n", - " state = torch.from_numpy(state).float()\n", - " state = Variable(state)\n", - " probs = self.policy_net(state)\n", - " m = Bernoulli(probs) # 伯努利分布\n", - " action = m.sample()\n", - " \n", - " action = action.data.numpy().astype(int)[0] # 转为标量\n", - " return action\n", - " def predict_action(self,state):\n", - "\n", - " state = torch.from_numpy(state).float()\n", - " state = Variable(state)\n", - " probs = self.policy_net(state)\n", - " m = Bernoulli(probs) # 伯努利分布\n", - " action = m.sample()\n", - " action = action.data.numpy().astype(int)[0] # 转为标量\n", - " return action\n", - " \n", - " def update(self):\n", - " state_pool,action_pool,reward_pool= self.memory.sample()\n", - " state_pool,action_pool,reward_pool = list(state_pool),list(action_pool),list(reward_pool)\n", - " # Discount reward\n", - " running_add = 0\n", - " for i in reversed(range(len(reward_pool))):\n", - " if reward_pool[i] == 0:\n", - " running_add = 0\n", - " else:\n", - " running_add = running_add * self.gamma + reward_pool[i]\n", - " reward_pool[i] = running_add\n", - "\n", - " # Normalize reward\n", - " reward_mean = np.mean(reward_pool)\n", - " reward_std = np.std(reward_pool)\n", - " for i in range(len(reward_pool)):\n", - " reward_pool[i] = (reward_pool[i] - reward_mean) / reward_std\n", - "\n", - " # Gradient Desent\n", - " self.optimizer.zero_grad()\n", - "\n", - " for i in range(len(reward_pool)):\n", - " state = state_pool[i]\n", - " action = Variable(torch.FloatTensor([action_pool[i]]))\n", - " reward = reward_pool[i]\n", - " state = Variable(torch.from_numpy(state).float())\n", - " probs = self.policy_net(state)\n", - " m = Bernoulli(probs)\n", - " loss = -m.log_prob(action) * reward # Negtive score function x reward\n", - " # print(loss)\n", - " loss.backward()\n", - " self.optimizer.step()\n", - " self.memory.clear()" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.13 ('easyrl')", - "language": "python", - "name": "python3" - }, - "language_info": { - "name": "python", - "version": "3.7.13" - }, - "orig_nbformat": 4, - "vscode": { - "interpreter": { - "hash": "8994a120d39b6e6a2ecc94b4007f5314b68aa69fc88a7f00edf21be39b41f49c" - } - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/projects/notebooks/Q-learning/Q-learning探索策略研究.ipynb b/projects/notebooks/Q-learning/Q-learning探索策略研究.ipynb deleted file mode 100644 index 40583fd..0000000 --- a/projects/notebooks/Q-learning/Q-learning探索策略研究.ipynb +++ /dev/null @@ -1,32 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Q learning with different exploration strategies\n", - "\n", - "Authors: [johnjim0816](https://github.com/johnjim0816)\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.13 ('easyrl')", - "language": "python", - "name": "python3" - }, - "language_info": { - "name": "python", - "version": "3.7.13" - }, - "orig_nbformat": 4, - "vscode": { - "interpreter": { - "hash": "8994a120d39b6e6a2ecc94b4007f5314b68aa69fc88a7f00edf21be39b41f49c" - } - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/projects/notebooks/Q-learning/QLearning.ipynb b/projects/notebooks/Q-learning/QLearning.ipynb deleted file mode 100644 index debb47e..0000000 --- a/projects/notebooks/Q-learning/QLearning.ipynb +++ /dev/null @@ -1,459 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1、定义算法\n", - "强化学习算法的模式都比较固定，一般包括sample（即训练时采样动作），predict（测试时预测动作），update（算法更新）以及保存模型和加载模型等几个方法，其中对于每种算法samle和update的方式是不相同，而其他方法就大同小异。" - ] - }, - { - "cell_type": "code", - "execution_count": 49, - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np\n", - "import math\n", - "import torch\n", - "from collections import defaultdict\n", - "\n", - "class QLearning(object):\n", - " def __init__(self,n_states,\n", - " n_actions,cfg):\n", - " self.n_actions = n_actions \n", - " self.lr = cfg.lr # 学习率\n", - " self.gamma = cfg.gamma \n", - " self.epsilon = cfg.epsilon_start\n", - " self.sample_count = 0 \n", - " self.epsilon_start = cfg.epsilon_start\n", - " self.epsilon_end = cfg.epsilon_end\n", - " self.epsilon_decay = cfg.epsilon_decay\n", - " self.Q_table = defaultdict(lambda: np.zeros(n_actions)) # 用嵌套字典存放状态->动作->状态-动作值（Q值）的映射，即Q表\n", - " def sample_action(self, state):\n", - " ''' 采样动作，训练时用\n", - " '''\n", - " self.sample_count += 1\n", - " self.epsilon = self.epsilon_end + (self.epsilon_start - self.epsilon_end) * \\\n", - " math.exp(-1. * self.sample_count / self.epsilon_decay) # epsilon是会递减的，这里选择指数递减\n", - " # e-greedy 策略\n", - " if np.random.uniform(0, 1) > self.epsilon:\n", - " action = np.argmax(self.Q_table[str(state)]) # 选择Q(s,a)最大对应的动作\n", - " else:\n", - " action = np.random.choice(self.n_actions) # 随机选择动作\n", - " return action\n", - " def predict_action(self,state):\n", - " ''' 预测或选择动作，测试时用\n", - " '''\n", - " action = np.argmax(self.Q_table[str(state)])\n", - " return action\n", - " def update(self, state, action, reward, next_state, terminated):\n", - " Q_predict = self.Q_table[str(state)][action] \n", - " if terminated: # 终止状态\n", - " Q_target = reward \n", - " else:\n", - " Q_target = reward + self.gamma * np.max(self.Q_table[str(next_state)]) \n", - " self.Q_table[str(state)][action] += self.lr * (Q_target - Q_predict)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2、定义训练\n", - "强化学习算法的训练方式也比较固定，如下：\n", - "```python\n", - "for i_ep in range(train_eps): # 遍历每个回合\n", - " state = env.reset() # 重置环境,即开始新的回合\n", - " while True: # 对于一些比较复杂的游戏可以设置每回合最大的步长，例如while ep_step<100，就是每回合最大步长为100。\n", - " action = agent.sample(state) # 根据算法采样一个动作\n", - " next_state, reward, terminated, _ = env.step(action) # 与环境进行一次动作交互\n", - " agent.memory.push(state, action, reward, next_state, terminated) # 记录memory\n", - " agent.update(state, action, reward, next_state, terminated) # 算法更新\n", - " state = next_state # 更新状态\n", - " if terminated:\n", - " break\n", - "```\n", - "首先对于每个回合，回合开始时环境需要重置，好比我们每次开一把游戏需要从头再来一样。我们可以设置智能体在每回合数的最大步长，尤其是对于比较复杂的游戏，这样做的好处之一就是帮助智能体在训练中快速收敛，比如我们先验地知道最优解的大概步数，那么理论上智能体收敛时也应该是这个步数附近，设置最大步数可以方便智能体接近这个最优解。在每个回合中，智能体首先需要采样（sample），或者说采用探索策略例如常见的$\\varepsilon$-greedy策略或者UCB探索策略等等。采样的过程是将当前的状态state作为输入，智能体采样输出动作action。然后环境根据采样出来的动作反馈出下一个状态以及相应的reward等信息。接下来对于具有memory的智能体例如包含replay memory的DQN来说，需要将相应的transition（记住这个词，中文不好翻译，通常是状态、动作、奖励等信息）。紧接着就是智能体更新，对于深度强化学习此时一般从memory中随机采样一些transition进行更新，对于Q learning一般是采样上一次的transition。更新公式是比较关键的部分，但是也很通用，一般基于值的算法更新公式都是一个套路如下：\n", - "$$\n", - "y_{j}= \\begin{cases}r_{j} & \\text { for terminal } s_{t+1} \\\\ r_{j}+\\gamma \\max _{a^{\\prime}} Q\\left(s_{t+1}, a^{\\prime} ; \\theta\\right) & \\text { for non-terminal } s_{t+1}\\end{cases}\n", - "$$\n", - "智能体更新完之后，通常需要更新状态，即```state = next_state```，然后会检查是否完成了这一回合的游戏，即```terminated==True```，注意完成并不代表这回合成功，也有可能是失败的太离谱，等同学们有了自定义强化学习环境的经验就知道了（等你长大就知道了XD）。\n", - "如果需要记录奖励、损失等等的话可以再加上，如下方代码，实际项目中更多地使用tensorboard来记录相应的数据，甚至于笔者就在这些教学代码中使用过，但是看起来有些繁琐，容易给大家增加不必要的学习难度，因此学有余力以及需要在项目研究中做强化学习的可以去看看，也很简单。\n", - "此外稍微复杂一些的强化学习不是一次性写完代码就能收敛的，这时需要我们做一个调参侠。为了检查我们参数调得好不好，可以在终端print出奖励、损失以及epsilon等随着回合数的变化，这点说明一下强化学习的训练过程一般都是先探索然后收敛的，官方的话就是权衡exploration and exploitation。e-greedy策略的做法就是前期探索，然后逐渐减小探索率至慢慢收敛，也就是这个epsilon。这个值越大比如0.9就说明智能体90%的概率在随机探索，通常情况下会设置三个值，epsilon_start、epsilon_end以及epsilon_decay，即初始值、终止值和衰减率，其中初始值一般是0.95不变，终止值是0.01，也就是说即使在收敛阶段也让智能体保持很小概率的探索，这样做的原因就是智能体已经学出了一个不错的策略，但是保不齐还有更好的策略，好比我们知道要出人头地学历高比较重要，但是“人还是要有梦想的，万一实现了呢”，总是存在意外的可能，对吧。回归正题，比较关键的是epsilon_decay这个衰减率，这个epsilon衰减太快了学来的策略往往过拟合，好比一条只能选择一朵花的花道上，你早早选择了一朵看起来还可以的花，却错过了后面更多的好花。但是衰减的太慢会影响收敛的速度，好比你走过了花道的尽头也还没选出一朵花来，相比前者不如更甚。当然强化学习的调参相比于深度学习只能说是有过之无不及，比较复杂，不止epsilon这一个，这就需要同学们的耐心学习了。\n", - "强化学习测试的代码跟训练基本上是一样的，因此我放到同一个代码段里。相比于训练代码，测试代码主要有以下几点不同：1、测试模型的过程是不需要更新的，这个是不言而喻的；2、测试代码不需要采样（sample）动作，相比之代替的是预测（sample）动作，其区别就是采样动作时可能会使用各种策略例如$\\varepsilon$-greedy策略，而预测动作不需要，只需要根据训练时学习好的Q表或者网络模型代入状态得到动作即可；3、测试过程终端一般只需要看奖励，不需要看epislon等，反正它在测试中也是无意义的。" - ] - }, - { - "cell_type": "code", - "execution_count": 50, - "metadata": {}, - "outputs": [], - "source": [ - "def train(cfg,env,agent):\n", - " print('开始训练！')\n", - " print(f'环境:{cfg.env_name}, 算法:{cfg.algo_name}, 设备:{cfg.device}')\n", - " rewards = [] # 记录奖励\n", - " for i_ep in range(cfg.train_eps):\n", - " ep_reward = 0 # 记录每个回合的奖励\n", - " state = env.reset(seed=cfg.seed) # 重置环境,即开始新的回合\n", - " while True:\n", - " action = agent.sample_action(state) # 根据算法采样一个动作\n", - " next_state, reward, terminated, info = env.step(action) # 与环境进行一次动作交互\n", - " agent.update(state, action, reward, next_state, terminated) # Q学习算法更新\n", - " state = next_state # 更新状态\n", - " ep_reward += reward\n", - " if terminated:\n", - " break\n", - " rewards.append(ep_reward)\n", - " if (i_ep+1)%20==0:\n", - " print(f\"回合：{i_ep+1}/{cfg.train_eps}，奖励：{ep_reward:.1f}，Epsilon：{agent.epsilon:.3f}\")\n", - " print('完成训练！')\n", - " return {\"rewards\":rewards}\n", - "def test(cfg,env,agent):\n", - " print('开始测试！')\n", - " print(f'环境：{cfg.env_name}, 算法：{cfg.algo_name}, 设备：{cfg.device}')\n", - " rewards = [] # 记录所有回合的奖励\n", - " for i_ep in range(cfg.test_eps):\n", - " ep_reward = 0 # 记录每个episode的reward\n", - " state = env.reset(seed=cfg.seed) # 重置环境, 重新开一局（即开始新的一个回合）\n", - " while True:\n", - " action = agent.predict_action(state) # 根据算法选择一个动作\n", - " next_state, reward, terminated, info = env.step(action) # 与环境进行一个交互\n", - " state = next_state # 更新状态\n", - " ep_reward += reward\n", - " if terminated:\n", - " break\n", - " rewards.append(ep_reward)\n", - " print(f\"回合数：{i_ep+1}/{cfg.test_eps}, 奖励：{ep_reward:.1f}\")\n", - " print('完成测试！')\n", - " return {\"rewards\":rewards}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 3、定义环境\n", - "\n", - "OpenAI Gym中其实集成了很多强化学习环境，足够大家学习了，但是在做强化学习的应用中免不了要自己创建环境，比如在本项目中其实不太好找到Qlearning能学出来的环境，Qlearning实在是太弱了，需要足够简单的环境才行，因此本项目写了一个环境，大家感兴趣的话可以看一下，一般环境接口最关键的部分即使reset和step。" - ] - }, - { - "cell_type": "code", - "execution_count": 51, - "metadata": {}, - "outputs": [], - "source": [ - "import gym\n", - "import turtle\n", - "import numpy as np\n", - "\n", - "# turtle tutorial : https://docs.python.org/3.3/library/turtle.html\n", - "\n", - "class CliffWalkingWapper(gym.Wrapper):\n", - " def __init__(self, env):\n", - " gym.Wrapper.__init__(self, env)\n", - " self.t = None\n", - " self.unit = 50\n", - " self.max_x = 12\n", - " self.max_y = 4\n", - "\n", - " def draw_x_line(self, y, x0, x1, color='gray'):\n", - " assert x1 > x0\n", - " self.t.color(color)\n", - " self.t.setheading(0)\n", - " self.t.up()\n", - " self.t.goto(x0, y)\n", - " self.t.down()\n", - " self.t.forward(x1 - x0)\n", - "\n", - " def draw_y_line(self, x, y0, y1, color='gray'):\n", - " assert y1 > y0\n", - " self.t.color(color)\n", - " self.t.setheading(90)\n", - " self.t.up()\n", - " self.t.goto(x, y0)\n", - " self.t.down()\n", - " self.t.forward(y1 - y0)\n", - "\n", - " def draw_box(self, x, y, fillcolor='', line_color='gray'):\n", - " self.t.up()\n", - " self.t.goto(x * self.unit, y * self.unit)\n", - " self.t.color(line_color)\n", - " self.t.fillcolor(fillcolor)\n", - " self.t.setheading(90)\n", - " self.t.down()\n", - " self.t.begin_fill()\n", - " for i in range(4):\n", - " self.t.forward(self.unit)\n", - " self.t.right(90)\n", - " self.t.end_fill()\n", - "\n", - " def move_player(self, x, y):\n", - " self.t.up()\n", - " self.t.setheading(90)\n", - " self.t.fillcolor('red')\n", - " self.t.goto((x + 0.5) * self.unit, (y + 0.5) * self.unit)\n", - "\n", - " def render(self):\n", - " if self.t == None:\n", - " self.t = turtle.Turtle()\n", - " self.wn = turtle.Screen()\n", - " self.wn.setup(self.unit * self.max_x + 100,\n", - " self.unit * self.max_y + 100)\n", - " self.wn.setworldcoordinates(0, 0, self.unit * self.max_x,\n", - " self.unit * self.max_y)\n", - " self.t.shape('circle')\n", - " self.t.width(2)\n", - " self.t.speed(0)\n", - " self.t.color('gray')\n", - " for _ in range(2):\n", - " self.t.forward(self.max_x * self.unit)\n", - " self.t.left(90)\n", - " self.t.forward(self.max_y * self.unit)\n", - " self.t.left(90)\n", - " for i in range(1, self.max_y):\n", - " self.draw_x_line(\n", - " y=i * self.unit, x0=0, x1=self.max_x * self.unit)\n", - " for i in range(1, self.max_x):\n", - " self.draw_y_line(\n", - " x=i * self.unit, y0=0, y1=self.max_y * self.unit)\n", - "\n", - " for i in range(1, self.max_x - 1):\n", - " self.draw_box(i, 0, 'black')\n", - " self.draw_box(self.max_x - 1, 0, 'yellow')\n", - " self.t.shape('turtle')\n", - "\n", - " x_pos = self.s % self.max_x\n", - " y_pos = self.max_y - 1 - int(self.s / self.max_x)\n", - " self.move_player(x_pos, y_pos)" - ] - }, - { - "cell_type": "code", - "execution_count": 52, - "metadata": {}, - "outputs": [], - "source": [ - "import gym\n", - "def env_agent_config(cfg,seed=1):\n", - " '''创建环境和智能体\n", - " ''' \n", - " env = gym.make(cfg.env_name,new_step_api=True) \n", - " env = CliffWalkingWapper(env)\n", - " n_states = env.observation_space.n # 状态维度\n", - " n_actions = env.action_space.n # 动作维度\n", - " agent = QLearning(n_states,n_actions,cfg)\n", - " return env,agent" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 4、设置参数\n", - "\n", - "到这里所有qlearning模块就算完成了，下面需要设置一些参数，方便大家“炼丹”，其中默认的是笔者已经调好的～。另外为了定义了一个画图函数，用来描述奖励的变化。" - ] - }, - { - "cell_type": "code", - "execution_count": 53, - "metadata": {}, - "outputs": [], - "source": [ - "import datetime\n", - "import argparse\n", - "import matplotlib.pyplot as plt\n", - "import seaborn as sns\n", - "class Config:\n", - " '''配置参数\n", - " '''\n", - " def __init__(self):\n", - " self.env_name = 'CliffWalking-v0' # 环境名称\n", - " self.algo_name = 'Q-Learning' # 算法名称\n", - " self.train_eps = 400 # 训练回合数\n", - " self.test_eps = 20 # 测试回合数\n", - " self.max_steps = 200 # 每个回合最大步数\n", - " self.epsilon_start = 0.95 # e-greedy策略中epsilon的初始值\n", - " self.epsilon_end = 0.01 # e-greedy策略中epsilon的最终值\n", - " self.epsilon_decay = 300 # e-greedy策略中epsilon的衰减率\n", - " self.gamma = 0.9 # 折扣因子\n", - " self.lr = 0.1 # 学习率\n", - " self.seed = 1 # 随机种子\n", - " if torch.cuda.is_available(): # 是否使用GPUs\n", - " self.device = torch.device('cuda')\n", - " else:\n", - " self.device = torch.device('cpu')\n", - "\n", - "def smooth(data, weight=0.9): \n", - " '''用于平滑曲线\n", - " '''\n", - " last = data[0] # First value in the plot (first timestep)\n", - " smoothed = list()\n", - " for point in data:\n", - " smoothed_val = last * weight + (1 - weight) * point # 计算平滑值\n", - " smoothed.append(smoothed_val) \n", - " last = smoothed_val \n", - " return smoothed\n", - "\n", - "def plot_rewards(rewards,title=\"learning curve\"):\n", - " sns.set()\n", - " plt.figure() # 创建一个图形实例，方便同时多画几个图\n", - " plt.title(f\"{title}\")\n", - " plt.xlim(0, len(rewards), 10) # 设置x轴的范围\n", - " plt.xlabel('epsiodes')\n", - " plt.plot(rewards, label='rewards')\n", - " plt.plot(smooth(rewards), label='smoothed')\n", - " plt.legend()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 5、我准备好了！\n", - "\n", - "到现在我们真的可以像海绵宝宝那样大声说出来“我准备好了！“，跟着注释来看下效果吧～。" - ] - }, - { - "cell_type": "code", - "execution_count": 54, - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "c:\\Users\\24438\\anaconda3\\envs\\easyrl\\lib\\site-packages\\gym\\core.py:318: DeprecationWarning: \u001b[33mWARN: Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n", - " \"Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\"\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "开始训练！\n", - "环境:CliffWalking-v0, 算法:Q-Learning, 设备:cuda\n", - "回合：20/400，奖励：-126.0，Epsilon：0.010\n", - "回合：40/400，奖励：-43.0，Epsilon：0.010\n", - "回合：60/400，奖励：-37.0，Epsilon：0.010\n", - "回合：80/400，奖励：-52.0，Epsilon：0.010\n", - "回合：100/400，奖励：-49.0，Epsilon：0.010\n", - "回合：120/400，奖励：-38.0，Epsilon：0.010\n", - "回合：140/400，奖励：-26.0，Epsilon：0.010\n", - "回合：160/400，奖励：-23.0，Epsilon：0.010\n", - "回合：180/400，奖励：-17.0，Epsilon：0.010\n", - "回合：200/400，奖励：-36.0，Epsilon：0.010\n", - "回合：220/400，奖励：-18.0，Epsilon：0.010\n", - "回合：240/400，奖励：-29.0，Epsilon：0.010\n", - "回合：260/400，奖励：-13.0，Epsilon：0.010\n", - "回合：280/400，奖励：-16.0，Epsilon：0.010\n", - "回合：300/400，奖励：-13.0，Epsilon：0.010\n", - "回合：320/400，奖励：-14.0，Epsilon：0.010\n", - "回合：340/400，奖励：-13.0，Epsilon：0.010\n", - "回合：360/400，奖励：-13.0，Epsilon：0.010\n", - "回合：380/400，奖励：-13.0，Epsilon：0.010\n", - "回合：400/400，奖励：-13.0，Epsilon：0.010\n", - "完成训练！\n", - "开始测试！\n", - "环境：CliffWalking-v0, 算法：Q-Learning, 设备：cuda\n", - "回合数：1/20, 奖励：-13.0\n", - "回合数：2/20, 奖励：-13.0\n", - "回合数：3/20, 奖励：-13.0\n", - "回合数：4/20, 奖励：-13.0\n", - "回合数：5/20, 奖励：-13.0\n", - "回合数：6/20, 奖励：-13.0\n", - "回合数：7/20, 奖励：-13.0\n", - "回合数：8/20, 奖励：-13.0\n", - "回合数：9/20, 奖励：-13.0\n", - "回合数：10/20, 奖励：-13.0\n", - "回合数：11/20, 奖励：-13.0\n", - "回合数：12/20, 奖励：-13.0\n", - "回合数：13/20, 奖励：-13.0\n", - "回合数：14/20, 奖励：-13.0\n", - "回合数：15/20, 奖励：-13.0\n", - "回合数：16/20, 奖励：-13.0\n", - "回合数：17/20, 奖励：-13.0\n", - "回合数：18/20, 奖励：-13.0\n", - "回合数：19/20, 奖励：-13.0\n", - "回合数：20/20, 奖励：-13.0\n", - "完成测试！\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "c:\\Users\\24438\\anaconda3\\envs\\easyrl\\lib\\site-packages\\seaborn\\rcmod.py:400: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.\n", - " if LooseVersion(mpl.__version__) >= \"3.0\":\n", - "c:\\Users\\24438\\anaconda3\\envs\\easyrl\\lib\\site-packages\\setuptools\\_distutils\\version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.\n", - " other = LooseVersion(other)\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkgAAAHJCAYAAAB+GsZPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAACFiElEQVR4nO3dd3hUZcLG4d+Zkt6BJFTpofcWQVB0BRXRRV1dERXFrqhYwBU72CgWFLF3XVfFCvYC6kcHAUF6bwkhjfTMzPn+mMwwkwIBEkIyz31dXpJzzpx536nPvO0YpmmaiIiIiIiXpaYLICIiInKyUUASERERKUUBSURERKQUBSQRERGRUhSQREREREpRQBIREREpRQFJREREpBQFJBEREZFSFJBqMa3xKSIiUj0UkGqpn376ifHjx1fJuWbPnk1SUhK7du2q1ttI7ZCUlMSMGTNOyH2tX7+eCy+8kE6dOnHuuece8dj//Oc/DB48mC5dunD66aczbtw4Vq5cWan72rVrF0lJScyePbsqil6tJkyYwODBg0/IfeXk5HDjjTfStWtXevfuzbZt26rlflJSUnj66acZOnQoXbt2ZcCAAdx4440sXbrU77hRo0YxatQo79+lX49z5szhjDPOoFOnTjz44IPs27ePkSNH0rlzZ/r06UO7du148skny9z/o48+SlJSEg899FCZfY899hjdunWjqKioUnXxLdOiRYtISkpi0aJFFR4/ePBgJkyYUKlznyirV69m1KhRdO/enQEDBjB9+vRK1z9Q2Gq6AHJs3nrrrSo71+mnn85HH31EfHx8td5GpLQXX3yRPXv28OKLLxIXF1fhcZ9//jkPPPAAHTp04LbbbqNx48bs27ePTz75hH//+9/cc889jB49+gSWvHrdfPPNXHnllSfkvj7//HN++eUXHnzwQdq0aUOTJk2q/D6WLVvGLbfcQmxsLFdeeSUtWrQgMzOTjz76iFGjRvHEE09w4YUXlnvbjz76iMTERO/fjz76KM2bN+fJJ58kISGBt99+mz///JMpU6aQkJDA008/zYoVK8qc57fffiMmJobff/+9zL4lS5bQq1cvgoKCqqzOvl544QUiIiKq5dzHYufOnYwePZpu3brx7LPPsnnzZp555hkyMzN59NFHa7p4Jw0FJCEuLu6wX05VdRuR0jIyMmjbti2DBg2q8Jh169YxceJEhg0bxuOPP47Fcqjhe/jw4UyePJmnnnqKpKQkTj311BNR7GrXrFmzE3ZfmZmZAFx++eUYhlEt57/jjjto3rw5b775JqGhod59Q4YM4frrr+fBBx9kwIAB1K9fv8ztu3XrVuZ8/fv3p2/fvt6/4+PjvS2Q/fr14/XXX6ewsJDg4GAAduzYwY4dO7jrrruYNm0aW7ZsoWXLlgBkZ2ezceNG/vnPf1Z53T06dOhQbec+Fq+++irh4eHMnDmToKAgBg0aREhICI899hg33ngjjRo1qukinhTUxVYLjRo1isWLF7N48WJv066nmfe///0vZ5xxBj169OCPP/4A4OOPP2bEiBF069aNLl26cMEFF/DNN994z1e6u2zChAlcffXVfPrppwwZMoROnTpxwQUXMH/+/OO6DcCKFSsYOXIk3bp14/TTT+ftt9/m6quvPmLz859//sk111xDjx496NevH+PGjSMlJaXcsniUbtZOSkrihRdeYMSIEXTp0oUXXniB9u3b89577/ndLj09nY4dO3pb6VwuF6+88gr/+Mc/6NSpE0OGDOHdd9894vN08OBBnnjiCc466yw6d+7MsGHD+OSTT8qU8fnnn+epp57i1FNPpUuXLlx77bVH7OYoKiri2Wef5cwzz6RLly4MGzaMzz77rMK6V/Q4LV68mEsvvZSuXbsyZMgQ/u///q/Mfe3atYt7772XAQMG0LFjR5KTk7n33nvJyMg4bBlTU1O57777GDRoEF26dOHiiy/mp59+8u5PSkpi8eLFLFmy5LBdX6+++iphYWE8+OCDfuHI45577qFhw4a8+OKLhy1PZVXm+XY6nbzyyisMGzaMLl260K1bNy677DIWLlzoPWbGjBn84x//4IUXXqBPnz4MGDCArKysSj3npbvYKvs6+eyzzzj33HPp3Lkzw4cPZ8GCBXTo0KHCx3bUqFHerqJ27dp5XzOVfe0+/vjjXHXVVXTp0oX777+/3Pv4/PPPSU1N5T//+Y9fOAKwWCzcfffdjBw5kpycnHJv7+nO8nzGgbvlMSkpicGDBzN79mz27NnjPe7UU0+luLiY1atXe8/x22+/ERUVxRVXXEFoaKhfK9KyZctwuVzecH2sr3ePoqIirrnmGvr27cvff//tfaw8j62nq/ebb75h7NixdO/enT59+jBx4kTy8vK85ykuLmbq1KkMHDjQ+3x//vnnRxzaMGTIEMaOHVtm+wUXXMBNN90EwO+//86gQYP8WsyGDh2Ky+Uqt4UtUKkFqRZ66KGHuOeee7z/bt26NWvWrAHcTbkTJ06koKCA7t278/777zNp0iRuu+02evbsSVZWFq+++ip333033bt392u69vXXX3+RmprK2LFjiYiI4LnnnuO2225j/vz5REdHH9NtNm/ezNVXX02nTp2YPn06GRkZTJ8+nezsbM4777wK67t27VquuOIKunbtytNPP43T6WTatGneD4yjMWvWLO666y5atGhB48aNWbJkCXPmzOGKK67wHvPtt99imqa3TA8//DCzZ8/mhhtuoHv37ixZsoTHH3+c7OxsbrnllnLvp6CggMsvv5wDBw4wduxYGjduzI8//sj9999PWloaN954o/fYd955h549e/LEE0+QlZXF5MmTGT9+PB999FGF9bj77ruZN28eN910E127dmXevHlMmDABu93OsGHDKvVYrFmzhmuuuYZ+/frx/PPPs2vXLsaNG+d3TH5+PldeeSWxsbE89NBDREZGsmLFCl544QVCQkIqbI5PS0vj4osvJjg4mDvvvJPY2Fhmz57NLbfcwtNPP83w4cP56KOPeOSRRwD367iiVpNffvmFQYMGERYWVu7+oKAgzjrrLN59910yMjKIjY2tVP0rUpnne+rUqXz44YfcddddJCUlkZKSwosvvsjtt9/Or7/+6g0Ce/bsYd68ed7uC89751ie8yPd5vPPP2fChAlccskl3HfffaxatYqbb74Zp9NZ4Tkfeugh3nzzTT755BM++ugj4uLijuq1+/777zN69Giuu+46wsPDy72P3377jfr169OlS5dy97dr14527dod5hlx69ixIx999BGXXnopF198MZdccgmGYfDiiy+ydu1aXnjhBRITE4mLiyMkJITly5fTq1cvbxmSk5MJCwujd+/e/Pbbb94uzCVLltCgQQOSkpKO+fXu4XA4uPPOO/nrr7948803ad++fYXHPvTQQ1x00UXMnDmTVatW8cwzzxAbG8tdd90FwIMPPsjXX3/NbbfdRvv27fn666954IEHjvg4DR8+nFdeeYWcnBxvt97mzZtZt24dN910EwUFBezevZsWLVr43S4uLo6IiAi2bt16xPsIFApItVDr1q29L/zSzc+XX345Q4cO9f69c+dOrr32Wm6++WbvtsaNGzNixAiWLVtWYTA5ePAgs2fP9n5phYWFccUVV7Bw4UKGDBlyTLd5+eWXiYyM5LXXXvN+gbRs2ZLLLrvssPWdNWsWMTExvPHGG94m8/j4eO666y42btx42NuW1qtXL7+xKhdccAH/+c9/2LNnj7dZec6cOZx66qk0aNCArVu38r///Y9x48Zx/fXXAzBgwAAMw+Dll1/m8ssvL/cLefbs2WzYsIH//ve/dO/eHYDTTjsNh8PBzJkzueyyy4iJiQEgKiqKmTNnYrVaAXd3wIwZMyr8st+wYQPfffcd//nPf7jqqqsASE5OZvfu3SxatKjSAenll1+mXr16vPTSS9jtdgBiY2O58847vcds27aNxMREnnrqKZo2bQq4uzBWrlzJ4sWLKzz3m2++SXp6Ot999x2NGzcGYNCgQVx99dU8/fTTDBs2jG7dulX4OvbIzMwkNzfXe46KnHLKKZimyZ49e44rIFX2+U5NTeXOO+/0G0wcHBzMbbfdxvr16731cTgcjB8/3vtF7XG0z3llbvPcc89xxhlnMGnSJMD9erPb7UybNq3C+rZu3dr7I8lT5g8++KDSr91GjRpx9913H/Yx3bdv3xGfv8qIiIjwljExMdH777i4OIKCgvxeQz179mT58uWAu0Vn0aJF3HfffQD079+fZ599lqKiIoKCgli6dKm39ehYX+/gbnmcMGECixYt4s0336Rjx46HPX7QoEHeiTbJycn88ccf/Prrr9x1113s2LGDzz77jPHjx3s/r0477TTS0tKO2MIzfPhwZsyYwY8//ugd1/X1118TFRXF4MGDycrK8j6epYWHh1fYkheI1MVWx5T+xTJhwgTuvvtusrOz+fPPP/niiy94//33AQ47YyEuLs7vF73nQzQ/P/+Yb7Nw4UIGDhzo18zevXv3I354Llu2jIEDB3rDked2P//882F/oZWn9PFnn302wcHBzJ07F4C9e/eybNkyLrjgAm+ZTdNk8ODBOBwO73+DBw+msLCQZcuWlXs/ixcvpnHjxt4vGI/hw4dTWFjoN/Oqc+fO3i89OPJj7bnPs88+22/7jBkzeOyxx474GPiex/Ml6nH22Wf7laV9+/Z88MEHNG7cmG3btjFv3jxef/11tmzZctjXz+LFi8t9bocPH87+/fvZsmVLpctZGZ6xM06nE9M0/Z4rh8NR6fNU9vmeNm0aV111Fenp6SxdupRPP/2UL7/8Eij7virvNXq0z/mRbrN9+3b27Nnj9+MIOGzLbEWO5rVbmfef1Wo9bCtWdUhOTmbFihWYpsny5cvJy8tjwIABgDvw5ufns2zZMvLz81m7dq03IB3r6x3crYpfffUVV155JZ07dz5iGUv/KEhMTPR2sS1atAjTNMs8n74/fsp7nZumSdOmTenRo4f3Mw3cP/qGDh1KUFAQLpfrsOWqjnFotZVakOqY0t0QO3bs4MEHH2TBggXY7XZatmzpbc4+3DpKpccKeN40h3tzHek26enp1KtXr8ztyhuY6SszM7Pc2x2L0o9PREQEZ511FnPmzGHMmDHMnTuX0NBQzjrrLO99Q8VfNJ5xUKVlZWXRoEGDMts9dc3OzvZuK29cBlT8WHvKdLyPSVZWVpnWCpvNVmbbm2++yaxZs8jMzKR+/fp06tSJ0NBQDh48eNhze36B+yqv/ocTExNDeHj4EZeT8Oxv3Lgxn332mbe1wMN37NPhVPb5Xr16NY888girV68mNDSU1q1be1sgS7+vyut6Otrn/Ei3SU9PB8q+Jo703irP0bx2K+r29NWoUSNWrVp12GP27t1Lw4YNj7KkFUtOTmbq1Kls2bKF3377jZYtW3qfH0+rmWdafnFxMf379/fe9lhe7+Bufezduzdvv/02l156KQkJCYc9vrzn0/Paqej59P178eLFZWY6vvPOO/Tt25cLLriAxx57jIyMDHbt2sX27dt5/PHHgUMtR7m5uWXKlJOTQ2Rk5GHLHUgUkOowl8vF9ddfj91u55NPPqF9+/bYbDY2bdrEF198ccLLk5iYSFpaWpntBw4c8M4oKU9kZKT3A8PXvHnzaN++fYXhrbwPgPIMHz6c66+/nu3btzNnzhyGDBni/fCKiooC4O233y73i66i2R7R0dFs3769zPb9+/cDHFc3kKdM6enpfmPINm/eTGZmJj179gQo86vddwAouMNH6efDNE1vEzzAV199xZNPPsk999zDiBEjvDMXb7/9dr9BsKVFR0d76+rrWOp/xhlnMH/+fHJzc73PQVFREVu3biUpKQmn08mPP/5Ix44dqVevHmeccUaZAcXx8fGkpqYe8b4q83zn5OQwZswYkpKSmDNnDi1btsRisTBv3jy+++67SterKnleBwcOHPDbXvrvyqjq1+5pp53GL7/8wurVq8ttWfn777+58MILue+++7j66quPurzl6dChAzExMfz5558sWLDA23rkceqpp7Js2TIsFgtt27b1BsJjfb2Dey2lPn36cM455/DII48wc+bMYy6/J1ylpaX5fcb4fg527NixzOvcM67onHPOYdKkSfz4449s2bKFxo0bez8XwsPDSUhIKPMcHzhwgNzcXFq1anXM5a5r1MVWS5U3m6e0jIwMtm7dysUXX0znzp2x2dx52DOz7EhNrVXNMziysLDQu23t2rVHbB3o1asXf/zxh18T99q1a7n++utZs2aN9xfRvn37vPs9YaEyPNOL33nnHdasWePtXvPcN7gfy86dO3v/S09P57nnnqvwPnr37s3u3bvLrMfy5ZdfYrfbKxywWhmeD7qff/7Zb/vUqVOZPHky4P6V6Pt4AGW6A5OTk5k/f75ft85vv/1GcXGx322ioqIYM2aM98siNzfXO/OnIr1792bFihXs3r3bb/uXX35JgwYNOOWUUypbXW644QYKCgp45JFHvPf5119/ceGFF3LDDTfwxBNPsGPHDu8A6tjYWL/nqnPnzpVe36Yyz/eWLVvIzMzkyiuvpHXr1t73Yk29r8AdkJo1a8YPP/zgt/37778/6nNV9Wt3+PDhNGjQgCeeeIKCggK/fU6nk6lTp2K32znnnHOOuqwVsVgs9O3blwULFrBu3boyAWnAgAGsW7eO5cuX+7UeHevrHdwtbA0aNGDcuHH89NNPfjOFj1bPnj2xWq2HfT4jIiLKvM49n4VRUVGcccYZ/PTTT3z33XcMHz7cr+usf//+/Prrr36fqd999x1Wq5V+/fodc7nrGrUg1VJRUVGsWLHCO423PPXq1aNx48a8//77JCYmEhUVxW+//cY777wDHH68Q3W48cYbmTt3LmPGjOGaa64hOzub5557DovFcth+75tvvplLL72UG264gSuvvJKCggKeffZZunTpQv/+/SkoKCAkJIQnn3yS22+/ndzcXJ5//nnvQNIjsVqtnHfeebz33nskJCR411cB9xTj4cOH88ADD7B79246derE1q1beeaZZ2jSpAnNmzcv95wjRozggw8+4JZbbmHs2LE0adKEn3/+mU8//ZRbb73V21JxLNq1a8fQoUOZMmUKBQUFtG/fnvnz5/PLL7/wwgsvAO5Wl5dffpmXX36Zrl278vPPP/tNQQe45ZZb+PHHH7n22msZM2YM6enpPPvss35jkrp06cKHH37Ik08+yRlnnEFqaiqvv/46aWlpFc5mBBg9ejRffvklV199NbfeeisxMTF8/vnnLFy4sMxaRkfStm1bnnzySe677z527NjBZZddRpMmTbjjjjt47rnncDqdJCcnV3rl6T/++KPcLr5zzjmnUs93Xl4eERERzJo1C5vNhs1m47vvvvP+mj/R7ytwd2ePHTuWu+++m4ceeoh//OMfrFu3zrv0wdE83lX92o2MjOTJJ5/k1ltv5ZJLLuGKK66gefPm7Nu3j/fff59Vq1Yxbdq0I3ZJHa1+/frx+OOPY7Va6dOnj9++5ORkcnJyWLp0Kddee613+7G+3n1ddtllfP7550yePJlTTz210rfz1bRpUy666CKmT59OcXEx7dq144cffuCXX34BKvd8Dh8+nLFjx+J0Ov1+9AGMGTPGO6xg9OjRbNu2jenTp/Ovf/1LayD5UECqpUaOHMlff/3FddddxxNPPFHhitYzZ85k8uTJTJgwgaCgIFq3bs1LL73E448/ztKlS/1m4VS3U045hddff52nn36asWPHUq9ePW644QZeeumlCqcIg7u5/N1332XatGnccccdREREMGjQIO6++26CgoIICgpixowZTJs2jVtuuYXGjRtz6623HtUSABdccAFvv/02w4YNK/Ph88QTT/Dyyy/z3//+l3379lGvXj3OPfdc7rjjDr9Bs75CQ0O9ZX7uuefIycmhZcuWTJ48mYsvvrjS5arIlClTeOGFF3j77bfJyMigVatWPP/8896xUzfccAPp6em8/vrrFBcXc/rppzN58mTvOigAzZs357333uPJJ5/kzjvvpF69eowfP97vMg3//Oc/2bVrF59++ikffPABCQkJDBo0iMsvv5wHHniAzZs3l9sk36BBAz788EOmTZvGpEmTvB/yM2fO5Mwzzzzq+p533nm0bduWt956i+eff579+/cTExPjXafntdde46KLLmLSpElHXJTv66+/5uuvvy6zvVOnTiQkJBzx+Y6MjGTmzJk8/fTT3H777YSHh3vX07ruuutYunTpCbtMiK/zzz+fvLw8Xn/9dT799FPatGnD/fffz/3331+psUIe1fHaHTBgAB9//DFvvPEGL7/8MmlpacTExNCpUyc++ugjunbtekznPZzk5GSKi4s59dRTy4z3iYuLo0OHDmzYsIHevXt7tx/r692XxWLh0Ucf5aKLLuKpp57yjv05Wg888ABhYWG88cYb5OTkkJyczE033cSLL75Yqedz0KBBREZG0rRp0zJT+lu1asUbb7zh/SyOjY3l6quvLnf9pEBmmLriqZwgnoHivlOes7OzOfXUU7n33ntP2KUVpO5JT0/n3Xff5aKLLqqWS2XUBl9//TUdOnTwG8/366+/csMNN/DFF19Uaq0hOTlkZmYyf/58TjvtNL8xX0899RSzZ88+7HXfpOqoBUlOmDVr1vD8888zbtw4OnbsSGZmJm+++SaRkZGVXrtHpDxxcXHcfvvtNV2MGvXll1/yzDPPcMcdd9CwYUO2b9/O888/772Aq9QeoaGhTJ48mfbt23PVVVcRFhbGn3/+yXvvvccNN9xQ08ULGGpBkhPG5XIxa9YsvvjiC/bu3UtYWBh9+vThrrvuOqpBuyJSVkZGBtOmTWP+/Pmkp6dTv35972UnDteFLSenv//+m2effZY///yT/Px8mjVrxmWXXcbIkSO1VtEJooAkIiIiUoqm+YuIiIiUooAkIiIiUooCkoiIiEgpCkgiIiIipWiafwnTNHG5Am+8usViqN4BRPUOLKp3YAnEelssRrXN6lNAKmEYBtnZeTgcJ/46SjXFZrMQGxuuegcI1Vv1DgSqd2DVOy4uHKu1egKSuthERERESlFAEhERESlFAUlERESkFAUkERERkVIUkERERERKUUASERERKUUBSURERKQUBSQRERGRUmptQHK5XDz//POcdtppdOvWjeuuu46dO3fWdLFERESkDqi1AWnmzJl88MEHPPbYY/z3v//F5XIxZswYioqKarpoIiIiUsvVyoBUVFTEG2+8wdixYzn99NNp164dzzzzDPv27eP777+v6eKJiIhILVcrr8W2bt06cnNzSU5O9m6LioqiQ4cOLFmyhGHDhh3Tea3WWpkXj5mnvqp3YFC9Ve9AoHoHVr2r6Tq1QC0NSPv27QOgYcOGftvj4+O9+45FVFTocZWrtlK9A4vqHVhU78ASqPWuDrUyIOXn5wMQFBTktz04OJisrKxjPm92dj5OZ+BcBdlqtRAVFap6H0ZegYOMnEIa1w8/QaUr367UHOrHhBASdGxv2b0HcomJDKFRQlS1Pd9FDidBNisAxQ4Xew/k0iwhssrO7zJN9h3Io2G9MIySn40u0yQlPY/EuEPbPPfdND4CwzDIKSgmNDSYIAt+9d64K5O1W9M579Tm2Ep+de9IOciBrAK6tanvPV95DuYVkVfgICEuzG97scPlPu+2DAZ1bUSD2ENfVi6XyTcLt9OiURQhQVZ+X7WXYLuVs3o1pV50iPe8O1JyAGjbNAa77VBrgGma/LRsFyFBVk7t3BAD2Lr3IDn5xbRsFIXT6WLDzky6tK7PL8t30bBeOL07NWLHnkwSfcrpdLlYszWdVZsO4HSZnNo5EYCVmw5wSkIEwSWvsbBgG5k5hWzbm02H5nG0ax5LQaGTtKx8GtUPxzBg9/5c6kWHcCCrgIVrUggJspKTX0x0RBC9kuIJDrIy9/+2UVxczCnx4TRPCGNPajZ70w7SKC6UjqdEsyslm7+3pmGzWRjYvwuxMRGs2pzGpp2ZxEcHsTftIPUj7XRtGUt+QRELV+/G5XLRt3srUrOKWL8lBavpwGI6sJoObDgItpq4igqxmMVYcNGyZRMaN23I6k37SUnPpUF0CAdzC8nNKyTYbqFJ/VCsFtiXloPVYhAWbCEtIw/T5cLAhQEYpguLYRIeYqOwqBinw0lUmJ1m8eFs3JmBw+HAMM2S400MIDTIgtPhdO/DBEwM07PfxBoeQ+dzLuKbRbvIzCks97XWqnE0CXFhLFqb4v++NU2sZjE2VxE2iunerhH14uP48f82YRYXeh8P//87MXBiMV1YLRAWZKGgsBjT6TxUbk8dTJd3Gybush+6c4CSv03vVsM0MQyIDAuiqNhBQZGzpN6lbm/63KZkvzWyPu3OuYS5C3eSnVvJscSGwU0XdSUkuHqiTK0MSCEh7g+ToqIi778BCgsLCQ099vTsdLpwOAInKHgEcr2LipzMXbid1o2jaXdKrN9+h9PF4+8sZUdqDtcP70C/Donlnic7r4jQICv2knBQEdM0mffnHgqKnAzp0/SwX8C+vvx9K5//vpVOLeIYd2m3MvtTM/P54rctbNyVxS3/7MwpiYdCictl8sfqvbz1zTqaxEdw+dB2/LBwO1cOTWLTrizqR4fQqH44BUVOIkLtZcqbmpnPM/9bSZ/2CYwY2JLdabnMW7Gbc/qdQmxksPfY75fs5JNfN3FG9yZcdmZrZny6mj83pXHd+R1I7njocVu3PYPX56zlokGtWLp+P/mFDq49r733XB/8uBGXy6RpfARzF27nzJ5NaNEwipjIYH5etovvl+ykV1IDrju/A3ablVe/WsuCNfs4vXtjhvZthukyefnLNezYl0XfNjFcdk4XHnt7KXmFDu6/she/Lt9N88RIwkNtzPp8FYajEKsBjeOC+X3FdnZs30OIUcymtons3JdBw2g7HZpFEZnYhGYtWwCQW1DMw28sIeNgIf8+qw1rtqbTrU19Vm5KY8XGNG9dN+zI4PZLuvK/nzexcVcmvdrF8/lvWwEIthuEOHKJtuSxa91fjB45hJ+W7+Hr/9vm/TppFh/B+JE9CC358P903mbmLNgOwE9LdxEXFcKSdakARITasdssZBwsJCYiiMycIsAkJmgpruIiRl3Qi8TYEL5fsJktW/cQ7Mwh2pJHhFHAr5tjyHEF48o/yAFLAZFGASGWIkIoJsQopqFRTP6qYtYFWSh2uDBNF1mYWA0XFlwUGSYWXPTBhQUTKy4shgtjuQunYXKOp0buohMNtAfYAs6l0BD3fwDmdjiAhcami6Ylb4+Ongd0OYQBgz1/74RIoFWZd0Q5UqFgIbTB/V8ZO9z/8+2TaF2Z85bctlNljy0tC/78NIvPt7QkypJPnCWHWEsuMZY8Ii35RBoFsKGYDMNBJ8NBiFFMcMnzEmw4sBiHgga73P8761jLUpWOpZ3iIPw5O58vNjcj2uexcL9OCwm3FBJuFBBuFBJmKSLcKCTYKMaZ/zIExx75/MfAME2fKFdLrFq1iksuuYQffviBZs2aebf/+9//JikpiYcffviYzpuRkVvng0Kxw8m8P/fQMymeBrGhxMaGk5GRS2GRg49+3kSrRtH07ZBwxPP8uTGN4CAr7U+JJbegmM9/20rfDgm0bhx9xNtu3ZtNdHgQcVEhLFufytvfrmfkP9qWuV+ny8Xb36wnOMjK5We5P9Z8Q4XLZWKxlA0ZLtNk7dZ0WjaKIizEXma/1WoQExNOVlYePyzZybvfrQfgjQmDcThd3taEz3/bwpd/bAMgyG7hgSt70bhBhPc8+zPzef7TVezen0uz+AgeGt3br3wpGXkUFDo5JTESh9PFe99vYP7KPQAMO7U5vyzfxb/PasPmPdlkZBeS3CkRA+jYIs77pTh/5R7e+mad95zD+zen2OEiyG7lH72asCMlhxmzV+MszCfUKOLsQV04p98pAHyzcDufzNvs+2MNi8XA5TJp0TCKrXuzMQyIDg/iYF4x15zXni17smndOJq129LZsDOTuKgQtmxPJchwcPGglsxfuZv0rDyaJ0Ry4wXtsVis/LRsJ/+3dBNhlkLCjUI6Nwlh395UgnBQaNoJDbYQYjU4pVEMG1MKcORk4sBGqFFImFGE3eL+hR4WE0daejZBOLEYLkKNYkKNIoIMB06sFJsWoo08rIYL0x5KaFQc21PdX/RW3F/SFsMkCAehlmIACswgDFy4TIN0orC4nAQZxdgNJ+FGIVajch9/TtOgeMCN1OvQh1lfrGFpSTAJoph61hxCjGJiLbk0sGQTb80m1pJLkWnDNAzsOLAbTuw4CTIcJfddgN049FmTZm/IG0XnsC+jkARrFs2Cs7E5Cwhq3J6Rl5zO1/+3jc9KwlWQzUKRw0WoUUSCNZvmYblYig4SY8kjzpJDqFFMPctBIi0F2IyT9/PMiQWnaeA0LVisVkyXkxCj+LC3cZkGLsPANA3shtO73cTAadhwWey4LDZMaxBOrDgNO9n5TsKcBwk1CjENCzarFacJGBYsFgtOE4qd4MLAZrXiwsBpgt1mw7BY3G09hgFYcOE+1mK1YhgGGTnFFDpMwkPthIcFY2KBkuNdGBQ5TCwWC1abtWR7yX4MDqRn0dXY4H4sTKPSr8XSTKAYO1bTgdUwMQGX57EwbCWPiw2XYcdlWDENK6ZxqN5Wmw3DYgXDwMSCaXjq7P6/e5CP4ddm5P6n4dt25P2/w2WSnVuE3WYlPMyO4dfO5Pt5bWCW/HkgNY2elnXush/lY9Fk7GsERVZPQKqVLUjt2rUjIiKCRYsWeQNSdnY2a9eu5Yorrqjh0tU8V8k3oqWcFor3vt/Ab6v2snzDfv5zZS/v9vU7Mvlx6S5+ZBd5hQ7O6N64wvOnZeYzY/Yq7DYLz489jVlfrGHN1nT+3LifKTf39zvWNE2+XbyDjIOFXHZmG/bsz2XSO0tpXD+c2y7qwouf/QW4w0jpgPTtoh38vnovAHFRwXz+21auG9aBXu3imfn5X6zYsJ82TaIZfW57DCDIbiUqPIjP5m9hzoLtJHdMIDoimJT0PG75Z2d3ODBNnnxnOdl5RfxnVE9WbTr0i//1OWv5Y/U+oiOCGHlWW+YudP/kjY8JJTUzn49/3QxAsN3Kded3YP7KPezenwvAjtQctu49SMtGUbhKukL+9/MmXC6Tmy7sxI+LtrI/JYWO9kxCjUL2LN1KH0se635ah+EqprGlgLQ9hdgMJ8vt0PKUhsQkNmTXgt2cF3qQWEsusZZcwlcXEFLyBZ+6LgJ7cTEPhBYQHu5ukl6TUgicQmGRk68XbC8JRybRRh7hlkLiLDnE2XMpPGCjfWgOkUY+oRQTGl5E6Lyv6WsUwyY4y3BwLg6cWRYi4wrcD9Aq6GYAMUAhFPzPvTkZSPbNxZlAeQ25e6ARFewDyN1O6+AK9pXmLICMDNqUzb9+QoySpnoDGnPgiPN2TXsoOU47LkcxhtUOVjtFhYXUt+bAordZtfx3OmdlcXZ0FvUsB48vgBgWHEFROAtyqF+8lxv5kPDYAr8vB0fGUpb+bLJgeSad7Af5RwsHzaz7Kd6/g2BX3qFzHeaT3GUafi0NpjUIS3gMlrAYtqSbWPPSCDYcmCFRNGqciBESiREcBvYQjKBQDHsIOQ4ri9alER5io3f7RPKLTbBYCQsNZsHa/RwscHJW7+YEBdsxLFZyC51M+/gvMnMdnNGzGcNPawUWKxgWsNi8I2v/2ppOkM1CUrNYZnyyko2bd2HFxT9Pb82ALo3dX9wWG07guyW72Z9VxKWDW7P3QB7T319EvUg7947qR1hYsPfHic1m8f7wczhc/L0tnUf/+ycAo89tx2ldGh37c1ZKwyInew7k0jwxstItwh4zP1vNwZ1fMSBkA1bDxIUFa2Q9LBH1MCLqYQmLxgiJpIBg8l1W6sVGQVAIht39HyXPDbYgCopcPPe/FRQUFjPu3z2JCa/sG+nk8Mz/VuJIKaZv8Oayj0V4LEZoFEZIBEZwhPv/nn8Hh2ENO/KP8mNVK1uQAJ555hn++9//8vjjj9O4cWOmTJnCrl27+Prrr7Hbj/CpWYG60oI049NVbN6TzaQxfct0m1zz5M/ef78z8SzvB8mvy3fzxty/AbBaDKbf2p/IsCBM0+TDnza6WxMiQ7ju/A4s/juFt791t7rcOqIzL8xe7T3n9Fv7k5VT5O3m+WbRdj7+xR0s/nNFTzbuyvQGjRYNI9m69yAANqvBzHGDsFktOF0uvl+yk8/mb8VRzliZe//dnac/XOH9u1lCBHvS8oiJCOKmCzvx2NtLCaaYImwlv/nggdH9iIsKYWfKQaaUfFh2aB7Htr3Z5BU6ytxHeIiN3AIHLWJNru9j5dd5K3BiIdwoJMhw0KBBLHsOQmRhKo2t6RgG5JnB2C0mNouJ6XAQaSkg2CjGheWIv4yrisOwEd5/JOv2FbJs5WaSQg7QOTID8jKO+9wu0/2r2DAMHKb716Ol5HehGRROSGQ0aflWDhRacdnDCQsPZ+eeA0SEBRMXE8bOPekEG8UQGk1eXgHWkDBO692GnCKTxWtTKMzOwIGNXp2bUVDkpFWLhliCwzhYZOH9b//CVVzEPwZ14UCOk9+WbiLSUkBkiJV/Du1BWFgohtXdjmS1BWGEhPP6d5vYtnErDtOK3XASa8khJiaSq4Z1wbAFYwSHsWJHAf/7bjVnJ7dicJ9WGBZ3N9L6nRkkNY3FbrPwzH+Xcnb6hzSzHSj/gQkKw2ELwwiLJqReI4zoBNIcEXw9fwPBQVbO6d+K39YcYPO+fBIbRHPFuZ0xgsMxIuLYk17AW299yc2RP3jDlhkUhjWuKdt37aepLf2wz4kRGo0lthFGeBxGaBSWqAYYQWFYIuphj2lAbMME3vr6b77/fR1tmsVx08U9sNsPjd1cuGYfr3y1FoCbL+xEr3bxx/068di9P4c1W9M5o0cTv7FUFdm1P4epH66gT/sE/n1WmyMGjoyDhQTbrYSF+KfD0gEJ4LvFOygqdjLs1OZHHWSqyzvfrmPen7tpaM0k1xVM/z5JXHxGuR2AlWK1GkTHhJOdlVfrvsde+WoNi9bspZE1gxxXCKef2oELTqtU5ylxceHVNnOvVrYgAYwdOxaHw8HEiRMpKCigd+/evP7668ccjmqaaZqHfePu3p+DYRg0Kmew8Jqt6Xzxx1auHJJEvagQ/tyYhgls3JlJ97YNvMelZxf43a6o+FATdVbuoQGCTpfJxl1Z9GjbgLSsAn5c6u7c3pGSw+K/U/h7+6Ev21dLPlw9nnxvOamZ+Vw4oAWndk7kk5JwBLBhVybrdhy6rScchRqFOJ0Wtu7Jok3TWH5YsouPf9mMgUlT6wEsmBw0Q2lkzcBmOPn9y+2cHZJF2/BsIhzpuPINIiILcJhW8r8I4eHoPGKteRSZVmy4yDWDWb66HrOXZvqVde22dMKNAvoG7cKBBQsmp9jSiLdmkekKJy4yh1ZGKpYlJuf6j8WFg5AEUNEPNb93lvtxNg0r1ugEtmRacBQXU2iLwCwuxGlY6dKxBcER0TgNGz/9uY/inCxiLbkEG8XEN25M0xancJBwjNAoYuJi+XT+VrZt24XFFsQ1I3rz565iope/SVv7Pgp/f5sWQAvPSyUPMKwU28Jw2MNJc0WRk52DKyyO7t3aYgSFkeO0sWpnHu1bNyI7twjDFkyOw8KPi7Zw3j960b5NY9KzC8jJL6ZZQiQp6Xms3Z6B0+miVeNoWjSMAiACaF5yty7TxLIjk5aNogi2Wzmwcg9L1+9n9LntyMwpJC4yhLDwIMKAXm3ymPHpKnolxdNqYEu/h7IecGF0S/YeyKND+3hMICckAYthMKhbI293ZGnDB7Vj0rYcWjeJwQRWbUpjdK922BIOtSD07AA92jfye+/ZbRY6tajn/TssLJSZ2//BwMjtuIrySUyox2n9u2CJbYwRHIYRVPrFAY2BoU16EhMRRHREMMlNc9nx80YG9G+BtcGhX7yRYXY2OxKYnn0u0ZY89jhiefruc7FZrbw541f+bX5DG3sKea4gLJH1iWjcEmtCa6z1m2OJTsAIqnjMpcVmwWIL4oLTWtG6cQytm8SUCSpdW9cnItSOxWLQuWW9Cs50bBo3iPDrkj6SJg0ieHbsaZU+3ncM3JEM6dPsyAedYOGhdkwM9jjd3UMRYcfX6mMYBtZyhhzUBhEhdkws7Ha6X4PH+1hUlVobkKxWK/fccw/33HNPTRfluH31x1a+X7KTuy/r7jfA1iPjYCEPvL6Y0GAbz97Wv8xg4F9X7GbTriz+b/U+urau5+0X3rk/xy8grdzs/wt41/4clm5I49uF26gXGeK3b8POTHq0bcC2fQf9tq/YmMbWvdnevwu9IcsEDFIz87HgYu7vG9hzIBcTqGc56P7w32Kwf18e/YJ3YcdBhKWQrqF7ach+AJzffERGZAJxBXHcHHmA5iHZBDvzqJAJlBkXneP9V1DJGIVIo4BmWz+jR1Bzwo1CWtpSSYrOw5WXTbhRwJE+Uyz1TyE7OJE1W9IIi4omOiaKrdv3EWEUYEY1pN9pfXlu9hpCjSJaNomleeNYmsRHERUXx5Y0B8vX7eX05CTi490zo6JTDrJq8wFO796YT37dRFLTWKI6HRrI3K1pDo++tQSny2Rg14b0OTsJm9Xil8WGDGnIhz9t5NROicQ1rU9Qxl7ezjmNC6L+IsKZTZDhAHswrbv1ILRJEtb4lthD3GPONmxJ478/beS85FMILpllFgec3tV9bt+Ozm49O3v/HRcVQlyU+3WSEBdWZgZXmcfNMGjvM/D9tK6NOK2rO5zERPh/ACbGhTH5un4VnqtZQqR3RpwBnFsyzupwEmLDmH5rf4KDbRg2G8vX7qVLOSHgSC0KESF28s0gvst2/7o/O74ptmZH/qXv+15uXD+ccf/qVu65DWC3M47dzjjCQ2zYrO4XdWhYKC/uP9s9lgs7957TnQanHP1YC4vFoH3zuHL3hQbbeOSaPhgGBAcdfpKBVK3SrfvhobX26/i4hZd+LEJOjsfi5ChFgPMMvpz0zlJevfeMMvt/Xu5uwckvdLAvPZ8mDcLZsDOT7LximidGsi/DHSJ27c8hKvxQ8/mu1By/86wuFZDm/bmHX5bvBmBbSWtOq8ZRbN6dzcZdme7t+7L9tq8qOUcjazrRlnxCjCK6Be+ik30HGc4wXBg0sBzEYphs3hXPmdGFJFpLpjTk4G5iqIAVFxzcS3v2gh1wgssaTF4xhBlFpBlxBIdHUuyC4Kg46rdIojA8nt9X7qV+YgO+/79NhBhFdG7XjLimzfnftyuJtuRzc+QPtGAnLSJ8rtVXgHc8SkF4I3ZlOgCD8Mat2VkYyYG9e8kyQ/nXZecR2bAp4YA9K5+YiGCsFoMl365n9so93PSPToS2jOf0cxLZkZLD0IEt/X6lt6kPbdr5NxX7ftlffU77Mo9D0/gIHrq6Ny7TrHCafHREMDdecGjuTHiIjRwzlPezersfS4vBzHEDy51Z1yA2lJsuPOZ5N7WK3WbFYhjERoXQMyn+mLoeSn9xRYZVXSu1xWIQHmonJ9/dBRsRduj9GxkWhIlBIe77iwwPKvccx+toWmKk6pQOSKX/DiQn62OhgFRNvvq/bWze7Z52XZn+d3B3be3an0MTn2bpomL3rDOPPWm5ZOcVMa1kHE1UmJ28QndLye60XL/uhl37c9mdlsvu/Tk0T4z0tvy0aRLNxl1Z3nDkq3e7BDbvzmbrXvd6MNv2HiSYIi5I2MmBnLVYnYXEWnLLHY/RwOrf2tTK7p7pY2LhgDOMCEshIUYxqdYE9haEUOiy0qlvX6yndGfyB6uwFOfRypZCgjWbLEs0oy45nfzwhkx8cSEWXJze4xRGnt3W7z6CgHNadQdgXU4U2/Ye5NTBXbFZDX5Ylc7GXVl8lteLwSFryHCFExkTS9OktsS17UquKwRXcBSYIcx4/ncAHhvQl8j9Oby/dQ2N64cT3bCp977qRx/qzrhqaBL/HNiS6JIvrX4dE+nXkSrTJL7yXRNAmXEY7jV01CJQFUp/WEeFVW1QiQoP8gakSJ/7Kh3EoqspIEnNKN1qcrKEgppQusWo9GNTUxSQqsln87cAsGRdCqd2OrS6RnZeERt3ZtGj7aGF6ILtVm9X1W8r99KiUSQ5ecUM6taYNVvTvR+e4A5IDpfL53yH9mUcLGTN1kODOvel5/HAa4sA96QR03R3e/Tv3JCNu8ouVBFMEZ3yFnFdzFpMRxGb3/uB/qaV0bH7CN7m4BQrh7q0LFZ2FUXhwEpks3bsCmvHohVbcGHQoVtnflm2nfb2PWS6whg2/EzmrjjA6i0HiAyxMGFUb6Iy88nNd5BQ0rU09fbB/LRsN//9aSMAHZvHYk9sjR2IiQwh42AhLRtFHfYxH3V2kt/f913Rk9fnrOX31fB7YTsAruyfRLteTQmLDaewZBBnJHDZ4NYUOlw0qhdGw3phXJZTRNumFc+OMAzjpPrCKj0OpypbOQJd6S+uyCoOSL6hKMIvIB26H6vFKBOCpXY7WVtNakLZ7saT47HQO64a5BUcCi1pWf4Do9/7bj1L1+/n2vPa079zQwqLnT7jeGDT7ix+Xr4Lp8u9qGD75v5jDvYcyCUnr+IZUeXNyIJDC5c2bhBO55b1CLJZiI+yc0WnQpYvXkWuGcw5oSuJXF9AJwvuphkfRnQiRvM+pOTbaJhYj+Am7QnLsbM7LYdOHRPJ2JTGmsXuKdX/6tiKn/7O4Y8cd/fQDU0SuLxeHCs2xnJa14aEh9hpWM9/sLnVYqFv+3hvQKofc6i15oIBLVi5KY1ubepXWO+KlA4OYRUM6D3bZxCnAZzdu2m5x52swkut9xQZevKEt9ouolTYjAyv2g9v3zDre1+lt5e3bIfUXidrKKgJZVrTTpIfAydHKWqxbfuysdusfpeiOJB9aEZYSrr7siiL1qYQExHEXyUtPH9uSiO5UyIH8/yXVPcdAL07LZe9B9zji7q0qseqzQfYlZpDRsmS9D2TGrBs/f4yZWrdJJq0zPyS1XShd7t474q7SfE2wtZ9zZQ26+HANviriPN8xtoaUQkEtR8EweH839pUtm1PxajXjCv/NQzDMPCdX3RKxKGBqJ76B9utNG4QTouGUazYmEZMRBARoXYiQu0M7Xv4mSTREcF0b1OfPzem+a1VMrBrIwZ2Pba1S0JLXZqjrv4KL10vtSBVnWpvQfIbd3Tovny78qKr+D6l5pUJSHX0s6kyfAOSxTAqnJl6op0cpail8gqKeeK95RQ7XNxxSVe6tHLPkDng02q0MzWHnak5vPzlGr/bLlu/n7te+MP7xRYSZKWgyElpnkUf+3ZIYNXmA6RkuANXaLCV/p0alhuQ/jmgBXmFDr5esJ2rh7YjNTOfJetSOcW6n7MPfEXRnkNT7a0RcazMCKeJNY0Vzracf8ltGFZ3mQYmmURvTKNpQsQRZ/rEx4Zx3fkdiAoLwma10LKROyA1jT+6a3Fdf35HMnMKjzhDqrJKv9FKt7TUFcFBVm83KiggVaWIkNJjkKqvBcm35c9v+0nUnStVIyzYhoF7Mm5YsA2rpXrW8qkNfFuMwkJsJ81aVQpIx2F/ZgHFJbNiXpi9mqduTCY2MpgDPusN7T2Qy+bd5V+YJiu3iKySi/IlxIaRlVvobfXxzBrz6NKqnl+IatEwilaND43J6dM+nsV/p9K2aYx3Sm+PNvVx7lhJvS2/cF/0dvdsskIwIhsQ3P18ghonUa9FKx5/8if2pOXSuEE4w62HXhIWw/BbJuBIfK+5NbhHEzJzijitS8PD3KKs4CBrlYUjKNuyUlEXW21nMQzCgt2LW0LVt3IEMt9ft0E2C8H2qh387vtcVTQGqaoHhkvNs5SMK8stcAT0+CNw/5C1GO4rHZxMj0Xd/LaoZnvScvl28Q6a+sw2czhdbNmTRc+keL8FGZ0u09u9dTiRYXbCQmzegHRal0bsTM2hqNhFvagQwkPstG4c7e2iS2oWS2RYEE3jI9iZmsN5yc35R6+mNImPwJWdSvH63yjetBDzoLuFKdHqvl6RvW1/QvpdhhESgdVmwTAMEuJC2ZOWS0wV/koNDbYx8h9tj3xgNSvdghRah5uxPR+2oBakqhQSZMVqMXC6TCLD7FX+67YyY5BOpgkBUnUiQu3kFjgCevwRuCe9hIXYyMkvPqnWgzp5SlILbN2bjcUweOmLv0gt6erylZrp3nag1IrVvitPg/uCo56LoHpEhgURFmLzHts8MZI2jaNZsy2DZgnuIHbThZ1Yuy2DwmIHPZPclwS4dURn0rIKaFoyLdyZsoncb6ZBUUn5gsOxJw3E1rQzlqh4LJFlBzonxLpbbKIj6t56KKHB/r/262oLEkBYsB33Ak/+6+nI8TEMg4hQO1m5RdXSMuc3BqmCFqSqHhguJ4eIUDspGfknVatJTYkoWQ+sdJd2Taq73xZVYNHaFBatTWHMsA64TJOn3l+O1Wohv4KZYvsz3V9OnoCUGBfGvvRDK0FfflYbYiKC6dUunv6dG/LOt+tYs80diKLC7cSXzNyyGAYN64VzaqeGrNmWQfc27m6u0GAbPZP8u7waxITSICYUV/Z+itb+RPGaH8HpwNKgBUGd/oGtRU8M2+GDz4AuDdmyO+uou8NqA98WpGC7FVs1XbPnZODbnagWpKpVvQGp/BaksBCbt+VKXWx1k6flKOIkajWpKZ6Wo5OpNU3PymF4BlZ/On8zbZvEUORwQTkr8TaqH86etFz2l6xo7Rmkff6pzXn160PXKhvUrZF38b4GMaEkxIV5A1JkWBCnJLrHFDVLiMBus5DcKZGuresdcUR/4fIvKFr6mfdva7OuhJ55M4a9ci1CzRIimXBFz0odW9v4thjV1RlsHp76GZQdWCzHx/OhXdUDtN3nLL8FyeLTchWlLrY6ydNydDKFgpri+cw6mVrT6vY3RhXZtvcgxcUVX6IgqWkMe9JySc3Mx+F0kVUyjqhjC//rH5Ve2Tg+9tBg5MgwOy0bRTH2oi40qn9oe9hhvuhc+dkU/TmH4tXfAWBt3IGgzkOxNu180swCqGmhgRSQgg/9ArPU0otWnqw8waW6WpD6dnBffLf0+71Lq3qs3JRG83Ku0Si1X7OESP7vr300O8rZvnWRZ6bmydT6Xbe/MarIgewCMg4WVLg/qVkMv6zYzYGsQlIz8jFxXxE8MszOlUOTeOfb9ZzZo0mZ2yXEHloM0fMrsrKLITrTtpM352kozAUgqPfFBHcfdhS1Cgx+AakOjz+CQ0sYnEwfMHVFYj33j5aG9atuhqWHYRjcMLz869SMPrc9LpepwFtH/aNXE3q0rU+9qJAjH1zHnd2rKVaL4XfliZpWt78xqkh2btFh97dsFIXNasHhdLFio3vWWJMG4RiGwendGnNKQiSNSq0cDfhNZz+aX6auzL3kz50KhblYYpsQ1PMC7C17V/r2gSTIZvGO46jrAckzQy/yJGqiriuG929O11b1j3i5m+qgcFR3GYbhd43HQNYkPoKrhrar6WL4qdvfGCeAYUBMRDANYkLYeyCPJX+7p/Q3Tzz0QdqiYfkfqvWjD/1qqOwqqq7sVPLmTMEsOIil/imEDRuPEVT1v2rrCqNkVdac/OLDdlfWBTER7pAdF61fo1XNbrPSuknF1+YTkbpHAekwQoOt5BceWt26fnRImWurRYe7V45uEBPK3gN57EjNAQ5dguNwbFYLV5/TjqxKrhzt2LGS/F9ecbccxTQi9Jy7FI4qITTYWhKQ6vbLvU/7BAqKnHRvffTXrBMREX91+xvjOLlM/78HdG7I579v9dvm6Ttu0TCKVZsPeLdXdlBlZa8xVrj8S4qWzgbA0qAloWffhiX0xDf310aecUh1vYst2G7lH71q10V2RUROVnV3UZhjlJNfzGNvL+Hr/9uGw2dKf2xkMB18ZqV1a12f5I4JnN+/BQCnd/MPOo3qlx1zdKwc2//0hiN7h8GEDb8PS3hslZ2/rvMEo7regiQiIlVH3xilfPTzRrbuPcjWvQf9tndqEUdc5KF1hRLjwvjX4Nbev6MjgmnbNIYNOzOxWowqW5DQLMqn4Pd3ALB3HkJI8r+r5LyBJCEujHU7Mqv0Gm8iIlK3KSCVsmz9/jLbzujemOH9mxMeavdefTk6ouyssxsv6Mj7P2yodLdZZRQu/hgzN919gdneI6rsvIHkssFtGNC5YY3MQBIRkdpJAclHTn4xBUXOMtsvO7O1d5HHqPAgsnKLyg1IMRHB3PLPzlVWHseevyle+zMAIQNHH/GSIVK+4CArrRprBpKIiFSexiD5WLkxrdztVp/usiYlF4VtXD+iWsviTN9F/vczALAnnYatcYdqvT8RERE5RC1IPlIz88tss1kNLD6X7bhheEcOZBXQNL76ApLpdFDwwwtQlIcloTXBp15RbfclIiIiZSkg+TBNs8y20oOtI0Lt1X4xvaJV3+LK2ocRGkXYkDsqfdFZERERqRrqYvNRTj6qstloleXKy6RoxZcABPe7DCOkervyREREpCwFJB/ltSDZbSf2ISpa/hU4irDEt8TWOvmE3reIiIi4KSD5KKcBCZv1xF0o0pW1j+J1vwIQ3OcSDEMXqRQREakJCki+arCLzXS5KPj1dXA5sTbtjK1R+xNyvyIiIlKWApIPVzkJyX6CAlLx+vk4UzaCPYSQAVedkPsUERGR8ikg+SqvBekEjEEyTZPi1d8BENzzQiyRuhq7iIhITVJA8uGqoS425+41uDL3gj0Ee7tB1X5/IiIicngKSH7K62Kr/oHSnsuJ2Nv2xwgKrfb7ExERkcNTQPJRE+sgmYW5OHasBMDe/vRqvS8RERGpHAUkH+WupF3NY5CKty4FlxNLXBOscU2r9b5ERESkchSQfJS3DlJ1z2JzbFoIgK11v2q9HxEREak8BSQfJ7qLzZm6BeeevwEDe6u+1XY/IiIicnQUkHycyEuNmKZJ4eKPAbC1ORVLZINquR8RERE5egpIR1BdLUiu1M3u1iOLjeBe/6yW+xAREZFjo4Dko9x1kGzVM82/2DP2qGVvLQwpIiJyklFA8lVeF1s1tCCZLheOLYvd59fgbBERkZOOApKPEzVI27l3HWZ+NgSHY23cscrPLyIiIsdHAcmHJx8F2Q89LNURkByb3d1r9ha9MKy2Kj+/iIiIHB8FJB+eWWxBNqt3W1XPYjOdDoq3LgO09pGIiMjJSgHJh6eLLdivBalqB2k7d/0FhbkYYTFYE5Oq9NwiIiJSNRSQfJglnWxB9kMtSFXdxVZcMjjb1rI3hkUPv4iIyMlI39A+PC1I1dXFZpoudwsSYGveo8rOKyIiIlVLAcmHNyBV0yBt14Gd7tlrtmCsCW2q7LwiIiJStRSQ/FRvF5tz9xoArI3aafaaiIjISUwBycehLrZDD4u9CgdpO3a5A5KtSacqO6eIiIhUPQUkH4dmsfm0IFXRGCTTUYRz33oALQ4pIiJyklNA8uFdB8lnDFJVDdJ27tsATgdGeByWmIZVck4RERGpHgpIPrwraduqfgySo2T2mrVxRwyjei6AKyIiIlVDAcnHoUuN+Ezzr6KA5PSOP1L3moiIyMlOAclHeV1sVdGC5MrLwpW+EzCwKiCJiIic9BSQfJWzUGRVDNJ27tsAgCWuCZaQyOM+n4iIiFQvLcbjw+XTghQXFUxRsYvwkON/iDwByZrY9rjPJSIiItVPAakcFsPg4dF9cLrMKulic+7bCIA1Uatni4iI1AYKSD486yAZBkSE2qvmnMUFuA7sANSCJCIiUltoDJIPzyDtqpyG70zdAqYLI6Ieloi4KjuviIiIVB8FJB/mkQ85ahp/JCIiUvsoIPk41IJUdefU+CMREZHaRwHJh3cMElWTkEyXE2fKJkABSUREpDZRQCpHVbUguQ7sBEchBIViiW1cNScVERGRaqeA5MNVxV1szpSS7rWENhiGHmoREZHaQt/avqq4i03jj0RERGonBSQfLp91kKqCc/9WAKwNWlbNCUVEROSEUEDy421COv4zFeZiHtwPgLX+Kcd/QhERETlhTnhAWrZsGUlJSWX+W7RokfeYBQsWMGLECLp27crQoUOZM2eO3zkKCwt55JFHSE5Opnv37tx1112kp6cfd9k8s9gsVdCE5EzbDoAR2QAjJOK4zyciIiInzgm/1Mj69etp1qwZH3zwgd/26OhoADZv3swNN9zA6NGjmTJlCr/++iv33nsvcXFxJCcnA/Dwww+zdOlSZsyYQVBQEA899BBjx47lvffeO66yedZBqgqHuteaV9k5RURE5MQ44QFpw4YNtG7dmgYNGpS7/+233yYpKYk777wTgFatWrF27Vpee+01kpOTSUlJ4fPPP2fWrFn06tULgOnTpzN06FBWrFhB9+7dj7lsnnhUFS1IrrRt7nPVb37c5xIREZETq0ZakHr27Fnh/qVLl3LWWWf5bevXrx+TJ0/GNE2WLVvm3ebRokULEhISWLJkyXEFJA+r1cBmO77eR1dJF1tQQovjPld1sVotfv8PFKq36h0IVG/VOxBU5ZUvSjvhAWnjxo3ExsYyYsQIUlJSaNu2LXfeeSddunQBYN++fSQmJvrdJj4+nvz8fDIyMkhJSSE2Npbg4OAyx+zbt++4ymaxuF9YEREhxMaGH/N5XEUFZGSnAlCvVTus4cd+rhMhKiq0potQI1TvwKJ6BxbVW45XlQakXbt2ceaZZ1a4/9dff+XgwYPk5eUxceJErFYr7733HldccQWzZ8+mdevWFBQUEBQU5Hc7z99FRUXk5+eX2Q8QHBxMYWHhcZXf4XQBkJdXSEZG7rGfJ9U9/sgIjSS7yAZFx36u6mS1WoiKCiU7Ox9nSd0DgeqtegcC1Vv1DgTR0aHexo2qVqUBKSEhgblz51a4Pz4+niVLlhAaGordbgegc+fOrF27lnfffZdHHnmE4OBgioqK/G7n+Ts0NJSQkJAy+8E9sy009PiSs6tkISSX08ThOPYXWHHaLgAsMY2O6zwnitPpqhXlrGqqd2BRvQOL6h0YqnBuVRlVGpDsdjutWrU67DFRUVF+f1ssFlq1akVKSgoADRs2JDU11e+Y1NRUwsLCiIyMJDExkczMTIqKivxaklJTU0lISDjOGnguNXJ8nZqujN0Auv6aiIhILXVCR3PNnz+f7t27s3PnTu82h8PBunXraN26NQC9evVi8eLFfrdbuHAhPXr0wGKx0LNnT1wul3ewNsDWrVtJSUmhd+/ex1U+bxI9zkFfTm9AanR8JxIREZEacUIDUo8ePYiNjWX8+PH89ddfrF+/nvHjx5OZmcnVV18NwKhRo1i1ahVTp05l8+bNvPHGG3z77beMGTMGcHfjnXfeeUycOJFFixaxatUqxo0bR58+fejWrdtxlc+sooW0XRl7ALUgiYiI1FYnNCBFRETw1ltvUb9+fa699louvfRSMjMzee+996hfvz4Abdq0YebMmcybN48LL7yQjz/+mClTpngXiQR47LHHSE5O5tZbb+Xaa6+lZcuWPP/888ddPs9CkcfTxWYWF2IeTAMUkERERGqrEz7Nv1mzZkcMMwMHDmTgwIEV7g8LC2PSpElMmjSpSsvm7WE7jiYkV+ZewMQIicQSGnXE40VEROTkE1grSh3BoRakYz+HS+OPREREaj0FJB+HxiAde0LSDDYREZHaTwGpPMfRgqQZbCIiIrWfApIPV0kT0vE8KJrBJiIiUvspIPk6zlHapkMz2EREROoCBSQfnnxkOcYuNs1gExERqRsUkHyYx3lRF1e6xh+JiIjUBQpIPryz2I6xi00z2EREROoGBSQfx7sOktM7QFstSCIiIrWZApKPQ9eqVQuSiIhIIFNA8uXtYjuGm2oGm4iISJ2hgOTDdWgp7aO/rWawiYiI1BkKSOU4li42zWATERGpOxSQfLiOY5C2K1MraIuIiNQVCki+jmMMkruLDSwxDauwQCIiIlITFJB8HM8sNldWCgCW6IQqLJGIiIjUBAUkH8e6DpJpunBlpwJgiU6s6mKJiIjICaaA5ONYLzRi5maAsxgMK0ZEvSotk4iIiJx4Ckg+PLP8LUfZhOTpXjOiGmBYrFVdLBERETnBFJB8mMe4DpIrax+g8UciIiJ1hQJSOY72YrXeAdpRCkgiIiJ1gQKSD9cxLqStGWwiIiJ1iwKSr2OdxeadwaaAJCIiUhcoIPnwDkE6ioRkmiaunJKL1EY2qI5iiYiIyAmmgOTDLJnofzQNSGZhDjiK3LcLj62GUomIiMiJpoDkwzyGS42YOQfctwmNwrAFVUOpRERE5ERTQPJheq81UvmE5PIEpIj61VAiERERqQkKSD6OqYstJx0AS0RcNZRIREREaoICko9j6WI71IKkS4yIiIjUFQpIvrzrIB3FLLaSgGRRQBIREakzFJB8eLvY1IIkIiIS0BSQfBzTOkgHS1qQIhWQRERE6goFJB/eWWyVPd5ZjJmfBagFSUREpC5RQPJhliQkSyUbkMzcDPc/rHaM4IhqKpWIiIicaApIPrwNSJXsYnPlZboPD489qm45ERERObkpIJUwffrXKpt1PC1IlrCYaiiRiIiI1BQFpHJUti3IE5B0DTYREZG6RQGpHJXtLnMpIImIiNRJCkjlqHQXW8kYJHWxiYiI1C0KSOVQF5uIiEhgU0Aqh7rYREREApsC0jEyTdOni00BSUREpC5RQCrhu4h2pRqQCnPBWew+Piy6WsokIiIiNUMBqRyV6WJz5ZV0rwVHYNiCqrtIIiIicgIpIHn4NCFVpgHJzM10H6vxRyIiInWOAlI5KtOCdGgGW0w1l0ZERERONAWkclRmDJKni00DtEVEROoeBaRjpDWQRERE6i4FpHJYKjNI2zMGSatoi4iI1DkKSOWpRBeb6eliUwuSiIhInaOAVI7KzWJTF5uIiEhdpYBUwvSZ53+kWWymy4GZf9B9rLrYRERE6hwFpFIq1XqUlwWYYLFihEZWd5FERETkBFNAKuWo1kAKi8Ew9BCKiIjUNfp2L6VyayBluo/V+CMREZE6SQHJo2QIUmUCkqcFyaLxRyIiInWSAlIZR3OZEbUgiYiI1EUKSKVUqostP8t9bGh0NZdGREREaoICUgnPJP9KdbGVTPG3hEZVX4FERESkxigglWJUpostP9t9bJgCkoiISF2kgFRK5VqQSgJSiAKSiIhIXaSAVMqRApJpmocCkrrYRERE6iQFpDKOkJCK88HlcB+pgCQiIlInKSB5lIzSthypBamk9Qh7CIYtqHrLJCIiIjVCAekoudS9JiIiUucpIJVypGuxafyRiIhI3VetAenBBx9kwoQJZbYvWLCAESNG0LVrV4YOHcqcOXP89hcWFvLII4+QnJxM9+7dueuuu0hPTz+qcxytyq6D5AlIWgNJRESk7qqWgORyuZg+fTofffRRmX2bN2/mhhtu4LTTTmP27Nlccskl3HvvvSxYsMB7zMMPP8zvv//OjBkzePvtt9myZQtjx449qnMcq0q3IGmKv4iISJ1lq+oTbt68mfvvv5/t27fTqFGjMvvffvttkpKSuPPOOwFo1aoVa9eu5bXXXiM5OZmUlBQ+//xzZs2aRa9evQCYPn06Q4cOZcWKFXTv3v2I5zgeR1oG6VAXW+Rx3Y+IiIicvKq8BWnhwoW0atWKr7/+miZNmpTZv3Tp0jIhpl+/fixbtgzTNFm2bJl3m0eLFi1ISEhgyZIllTrHcalkF5vGIImIiNRdVd6CNHLkyMPu37dvH4mJiX7b4uPjyc/PJyMjg5SUFGJjYwkODi5zzL59+yp1jri4uGMouTtYWQwDm+0wubHAfR02W0TM4Y+rBaxWi9//A4XqrXoHAtVb9Q4Elbn6xbE6qoC0a9cuzjzzzAr3L1iw4IjhpKCggKAg//WDPH8XFRWRn59fZj9AcHAwhYWFlTrH8bBaDGJjwyvcn1PkDkhR8fGEHua42iQqKrSmi1AjVO/AonoHFtVbjtdRBaSEhATmzp1b4f7o6OgjniM4OLhMiPH8HRoaSkhISLkhp7CwkNDQ0Eqd43iYJmRk5Fa435GTBUCuI4iCwxxXG1itFqKiQsnOzsfpdNV0cU4Y1Vv1DgSqt+odCKKjQ7FYqqfV7KgCkt1up1WrVsd1hw0bNiQ1NdVvW2pqKmFhYURGRpKYmEhmZiZFRUV+rUSpqakkJCRU6hzHwnfkksNR/ovLdDowC92hyBUUiVnBcbWN0+mqsM51meodWFTvwKJ6B4bjHXZ8OCe8s7JXr14sXrzYb9vChQvp0aMHFouFnj174nK5vIO1AbZu3UpKSgq9e/eu1DmOScmDfLj+TLNk/BGGBYLDju1+RERE5KR3wgPSqFGjWLVqFVOnTmXz5s288cYbfPvtt4wZMwZwd+Odd955TJw4kUWLFrFq1SrGjRtHnz596NatW6XOcTwOG5C8ayBFYhiBNRBOREQkkJzwb/k2bdowc+ZM5s2bx4UXXsjHH3/MlClT/KbtP/bYYyQnJ3Prrbdy7bXX0rJlS55//vmjOsexMg4zz19T/EVERAJDlU/z9/Xuu++Wu33gwIEMHDiwwtuFhYUxadIkJk2aVOExRzrHsapUC5ICkoiISJ2mfqLSDpOQFJBEREQCgwJSKYdbc8qlgCQiIhIQFJBKURebiIiIKCCV8CylYByui63AHZAsCkgiIiJ1mgJSKZVrQTq2xShFRESkdlBA8vAsFHm4Q/LdC0UaIWpBEhERqcsUkMooPyKZpnmoBSnsyNecExERkdpLAakUS0VNSEV54HIA7pW0RUREpO5SQPI6fB+b5yK12IIwbEHlHyQiIiJ1ggJSKRXNYjMLctz7gyNOZHFERESkBigglfBO869of0kLkhESfkLKIyIiIjVHAamUiqb5ewOSWpBERETqPAWkUo7cxaYWJBERkbpOAamUI3axqQVJRESkzlNAKqXCFqTCkhYkjUESERGp8xSQPI4wSluz2ERERAKHAlIpR+5iUwuSiIhIXaeAVMqRutgIUQuSiIhIXaeAVKLS6yCpBUlERKTOU0AqpcJ1kDQGSUREJGAoIJVSXhebabqgMM+9X7PYRERE6jwFpMooysfTCacWJBERkbpPAakUSzldbJ7uNewhGFbbiS2QiIiInHAKSKWV18WmAdoiIiIBRQGplPIGaXtX0VZAEhERCQgKSCXMknn+RjkT/U3PAG0FJBERkYCggORVMgi73Bakki62oLATWSARERGpIQpIpZS3DJJZ5GlBUkASEREJBApIpZS7DlJJFxvqYhMREQkICkillLuSdpG62ERERAKJAlIleAdpKyCJiIgEBAWkUsrtYivKd+/TGCQREZGAoIBUymFnsSkgiYiIBAQFpFION4uNIA3SFhERCQQKSCVK1okst4uNQk3zFxERCSQKSB6elbRL5SPTNA+tg6RB2iIiIgFBAelIigvAdAG61IiIiEigUEAqxVKqCck7/shiA6u9BkokIiIiJ5oCUmmlu9h8xh+VOz5JRERE6hwFpFIMym9B0vgjERGRwKGAVEqZQdolayChGWwiIiIBQwGplDKdaGpBEhERCTgKSCXMknn+pccZHRqDpBlsIiIigUIBqbQKutjUgiQiIhI4FJA8ShaKtJQOSEVqQRIREQk0CkhlVLAOklqQREREAoYCUillljrSddhEREQCjgJSKWUGaRcpIImIiAQaBaRSSjcgaZC2iIhI4FFAKlEyRruchSK1DpKIiEigUUAqpcJLjWgWm4iISMBQQCrNJx+ZLicUF7j/0BgkERGRgKGAVIpfF1tR/qHt6mITEREJGApIpfh2sXnXQLKHYFisNVQiEREROdEUkDxKRmn7tiBpBpuIiEhgUkAqxT8gaQ0kERGRQKSAVEp5XWxqQRIREQksCkglylsHydPFpuuwiYiIBBYFpNL8ZrFpDSQREZFApIBUisWnCUljkERERAKTAtJhaAySiIhIYFJA8nKPQtIsNhEREVFAKsV/FpvWQRIREQlECkillNeCpOuwiYiIBBYFpBKHpvn7JKSSC9Ua9tATXyARERGpMQpIHp5LjfhuKrlYrRGkgCQiIhJIFJBK8etiKy4JSPaQGiqNiIiI1IRqDUgPPvggEyZMKLN99OjRJCUl+f03atQo7/7CwkIeeeQRkpOT6d69O3fddRfp6el+51iwYAEjRoyga9euDB06lDlz5lRNoUsSkmma3i421IIkIiISUKolILlcLqZPn85HH31U7v7169fz8MMP8/vvv3v/mzFjhne/Z9+MGTN4++232bJlC2PHjvXu37x5MzfccAOnnXYas2fP5pJLLuHee+9lwYIFx112bwOSowjMkqn/GoMkIiISUGxVfcLNmzdz//33s337dho1alRm/4EDBzhw4ABdu3alQYMGZfanpKTw+eefM2vWLHr16gXA9OnTGTp0KCtWrKB79+68/fbbJCUlceeddwLQqlUr1q5dy2uvvUZycvJxld/TxebpXsMwwBZ0XOcUERGR2qXKW5AWLlxIq1at+Prrr2nSpEmZ/evXr8cwDFq0aFHu7ZctWwZAv379vNtatGhBQkICS5YsAWDp0qVlglC/fv1YtmyZu2vsOHhnsRWVdK/ZQ/xntomIiEidV+UtSCNHjjzs/g0bNhAZGcmjjz7KH3/8QVhYGEOHDuXmm28mKCiIlJQUYmNjCQ4O9rtdfHw8+/btA2Dfvn0kJiaW2Z+fn09GRgZxcXHHXH6r1cBms+BwHZrib7PVzbHsVqvF7/+BQvVWvQOB6q16B4LqbL84qoC0a9cuzjzzzAr3L1iw4IjhZMOGDRQWFtKlSxdGjx7N33//zdNPP82ePXt4+umnyc/PJyiobJdWcHAwhYWFABQUFJQ5xvN3UVHR0VTJy9PuFBYaTGxsOPlZcBCwhYYRGxt+TOesLaKiAnOMleodWFTvwKJ6y/E6qoCUkJDA3LlzK9wfHR19xHM8+uijjB8/3nts27Ztsdvt3Hnnndx7772EhISUG3IKCwsJDXU/8cHBwWWO8fztOeZYFRQUkZGRS1F6BgAuazAZGbnHdc6TldVqISoqlOzsfJxOV00X54RRvVXvQKB6q96BIDo6FIulelrNjiog2e12WrVqdXx3aLOVCVJt2rQBDnWdZWZmUlRU5NdKlJqaSkJCAgANGzYkNTXV7xypqamEhYURGRl5bAUraUIyXSYOhwtnQckgbVsIDkfdfrE5na46X8fyqN6BRfUOLKp3YDjOYceHdcI7K0eNGsV9993nt2316tXY7XaaN29Oz549cblc3sHaAFu3biUlJYXevXsD0KtXLxYvXux3joULF9KjR4/jT5KedZC0iraIiEjAOuEBaciQIXzxxRd8+OGH7Ny5k7lz5/L0009z7bXXEhERQUJCAueddx4TJ05k0aJFrFq1inHjxtGnTx+6desGuEPWqlWrmDp1Kps3b+aNN97g22+/ZcyYMcddvjLT/LWKtoiISMCp8llsR3LFFVdgGAbvvvsujz/+OA0aNODqq6/m+uuv9x7z2GOP8fjjj3PrrbcCMHDgQCZOnOjd36ZNG2bOnMmUKVN4++23adKkCVOmTDnONZBKFoX0/FkyzV8tSCIiIoGnWgPSu+++W+72kSNHHnY5gLCwMCZNmsSkSZMqPGbgwIEMHDjwuMtYmmfNI12HTUREJHAF1oIJleDtYvMuFKkWJBERkUCjgFTCMxDe28VW7OliUwuSiIhIoFFAKkVdbCIiIqKAVFqpLjYN0hYREQk8CkilHOpi0zR/ERGRQKWA5FEyCMnQQpEiIiIBTwGpFE8LkukZpK0WJBERkYCjgFSKYRiYpss7i01dbCIiIoFHAamEd5q/waFwBBhBYTVSHhEREak5Ckjl8Iw/wmIFq71mCyMiIiInnAJSKRbDwCzKA9ytR55B2yIiIhI4FJBKM3xakNS9JiIiEpAUkEoxDMDbgqQp/iIiIoFIAamEWTJK22qxaA0kERGRAKeA5OVOSO4xSJ6ApC42ERGRQKSAVIrVYmAWurvYUAuSiIhIQFJAKsViMXzGIKkFSUREJBApIJU4NAbJ0BgkERGRAKeAVIrFojFIIiIigU4BqYTnUiPuFiRN8xcREQlkCkge5qFZbHgXilRAEhERCUQKSKWoi01EREQUkEqU28UWrIAkIiISiBSQSrFoDJKIiEjAU0DyKGlCshgmFBe4/1AXm4iISEBSQCrh7WJzFnq3qQVJREQkMCkgebkjktVZ0npktWFY7TVYHhEREakpCkgeni42h2awiYiIBDoFpBKeLjaLo6SLTQFJREQkYCkglWIpGYNk2INruCQiIiJSUxSQSrG4igEwbApIIiIigUoBqRTDM4tNLUgiIiIBSwGpFIuzCFALkoiISCBTQCrFcLgDEgpIIiIiAUsBqRRDg7RFREQCngJSaZ5p/ragmi2HiIiI1BgFJB8WwwCHxiCJiIgEOgUkH1aLgelQF5uIiEigU0DyYbEYUOyZ5h9Ss4URERGRGqOA5MOvBUldbCIiIgFLAcmHxXJoDJIGaYuIiAQuBSQfVouBWawxSCIiIoFOAcmHxaeLTQtFioiIBC4FJB/uaf5qQRIREQl0Ckg+/LrY1IIkIiISsBSQfLgHaauLTUREJNApIPmwGy5wOQF1sYmIiAQyBSQfwRbnoT/UgiQiIhKwFJB8hFgc7n8YVgyrrWYLIyIiIjVGAclHkFESkOxaJFJERCSQKSD5CC5pQdIMNhERkcCmgOQjGE8LkgKSiIhIIFNA8uHpYlMLkoiISGBTQPKhgCQiIiKggOQnSF1sIiIiAmguu4/KtCC5XC6cTseJKlK1crkMCgqsFBUV4nSaNV2cE6a8elutNiwW/V4QERE3BSQfdord/7CVneZvmibZ2enk5+ec4FJVr7Q0Cy6Xq6aLccKVV+/Q0AiiouIwDKOGSiUiIicLBSQf9pIutvIuM+IJRxERsQQFBdeZL1Gr1Qio1iMP33qbpklRUSE5ORkAREfXq8miiYjISUAByUeQtwXJPyC5XE5vOIqIiKqBklUfm82CwxF4LUil6x0U5H7Oc3IyiIyMVXebiEiA07eAD5tZfguS0+m+RpvnS1TqJs/zW1fGmImIyLFTQPJhr6AFyaOudKtJ+fT8ioiIhwKSD5vpDkhaB0lERCSwKSD58LQglTdIW0RERAKHApIPTwuSFoo8Odx66/VMnvxwTRdDREQCkAKSD3WxiYiICCgg+bG6Dj9IW0RERAKD1kHy4W1BqmQXm2maFBXXzBpCQXbLMc26GjCgF6NHX8fcuV/hcBTz0kuv06BBAq+++hLff/8Nubk5tGjRijFjbqRPn35s3ryJq666jNdff4+kpHYA3Hff3SxfvoS5c3/GarXicrkYPvxsbrttHEOGnMtXX33OJ5/8l507d2KxGLRt246xY8fRrl0HAC6++HxOP/1MFi78g4yMdCZNepqOHTsza9YMvv/+W4qLi7jggoswTf8FLD/44F0+//wT9u9PpX79Bpx33nCuuupazT4TEZEqV+UBae/evUyZMoVFixZRVFREly5dmDBhAm3atPEe88033zBjxgx27dpFy5YtGT9+PMnJyd79GRkZTJo0ifnz52MYBueddx733nsvoaGhlT7HsbC6itz/qEQLkmmaPPHecjbtzjqu+zxWrZtEc9/IHscUDj777GOmTn0eh8NJs2bNmDjxPrZv38qDDz5Ggwbx/PHHfO699w4ef3wqp546gIYNG7FkyUKSktrhdDpZsWIpeXl5bNiwjvbtO7J27RoOHjxIcvIA5s37hWeeeZrx4yfStWt30tLSePbZKTz55CTeeusDbxlmz/4fTz31DJGRkbRs2Zpnn53CH3/8xv33P0RCQkPeeecNVq5cQaNGjQH4/ff5vPvumzz66OM0bdqcNWtWMWnSQzRs2IghQ86tssdVREQEqriLraioiOuvv579+/cza9YsPvjgA8LDw7nqqqtIT08HYOHChdxzzz1cdtllfPbZZyQnJ3P99dezefNm73nGjh3L9u3beeutt3juueeYN28eDz/8sHd/Zc5xLKxH2YJELW24GDLkXNq160CnTp3ZuXMHP/74Hf/5z0P06NGLpk2bcdllV3DWWUP44IN3AOjf/zSWLFkEwN9/r8Fms9OpU2eWL18KwIIFv9O1a3eioqKIjo5mwoQHGDLkXBITG9KpU2eGDRvOli2b/MrQr19/evfuS7t2HXA4ivnmm6+57robSU4eQMuWrbjvvgeJizt0yY89e3YRFGQnMbERiYmJnHnm2Tz77Et07drjBD1qIiISSKq0BWnp0qVs2LCB+fPnk5CQAMCUKVPo27cvP//8MxdffDGvvvoqZ511FldeeSUA48ePZ8WKFbz99ts8+uijrFixgsWLFzN37lxatWoFwKOPPsqYMWMYN24cCQkJRzzHsbKY7hWzKzNI2zAM7hvZo9Z1sQE0adLM++8NG9YDcPPNY/yOcTgcREREAu6A9OWXn1FYWMCSJYvo2bMXiYmNWLZsKSNHXsWCBb8zdOgwALp168G2bVt5663X2L59G7t27WDz5k1lLgzbpElT77937NhOcXEx7dp19G4LDg6mbdsk799nn30uc+Z8yb//PYLmzVvSu3dfTj/9TBITE4/pMRARETmcKg1Ibdq04ZVXXvGGI8B7Tavs7GxcLhfLly9nwoQJfrfr27cv33//PeAOWQ0aNPCGI4A+ffpgGAbLli1j6NChRzzHsTDwGe9iC6rcbQyD4CDrMd9nTQkOPhQAPcHlxRdfJSws3O84z3PXvXsv7HY7K1YsZ+nSxQwZci4NGzZk9uz/sW/fXjZu3MDkyYMA+P77b5k8+SHOPvscOnXqwgUXjGDLls1Mn/5UhWXwNMWZpn+IstkOvTxjYmJ4880P+OuvVSxZsohFixbw8ccfcu21NzB69HXH94CIiIiUUqUBqUGDBgwaNMhv27vvvktBQQH9+/cnOzubvLy8Mr/64+Pj2bdvHwApKSk0bNjQb39QUBAxMTHs3bu3Uuc4Ft62GMPAFhzs1zrjctXSvrQjMAxo1ao1AAcOpNG2bTvvvpdffhGr1cqYMTdis9no0yeZ33+fx9q1f/Gf/zxE/fr1cTqdvP76y7Rs2ZqGDRsB8P77b3H++Rdy9933ec/122/zAPe4rfJavZo1O4WgoGBWrVpJmzbuViOHw8HGjRvo0aMXAN9//w0HDx7koov+RZcu3bj22ht46qlJ/PTT90cdkDxFMAwoNQ4cAKvVwGarexM8rVaL3/8DheqtegeCQK13dc7ROaqAtGvXLs4888wK9y9YsIC4uDjv3z/88APTpk3j6quvJikpyRtggoL8W2iCg4MpLCwEID8/v8x+32MKCgqOeI5j4WlBMuwhxMVF+O0rKLCSlmapM1+cFsuherRs2Yr+/U9j6tQnufvu8bRs2Yqff/6R9957i4kTH/YeN3DgIJ544jHq12/AKae4u+g6d+7Cd9/N5eqrr/Uel5CQyOrVK9m0aT0RERH89ts8Zs/+HwAul8PbcuRbhqioCC655FLeeOMV4uMb0KJFS95//13S0vZjGO7jHI5iZs58jqioCLp27UFqagp//rmcbt16HPNzUvqDxOUysFgsREeHERISckznrA2iokKPfFAdpHoHFtVbjtdRBaSEhATmzp1b4f7o6Gjvvz/88EMee+wxhg8fzr333gsc6lYpKiryu11hYaF3hlpISEiZ/Z5jwsLCKnWO42ILIiMj129TUVEhLpcLp9PE4aiZMUdVyeVy18Mw3CHh0Uef4OWXX+TJJydz8GA2jRo1KRlofZ63vn37norT6aRHj17ebT179mHZsqWceupA77Y77riHp5+ezE03XUdQkJ3WrdsyceIjPPTQf/jrr7/o2rW7Xxk8rr/+Fuz2IKZOfZK8vDwGD/4H/fsPxDTdx5177nAyMjJ4/fVXSU1NITIyktNPP5Obbhp71M+Jp95Op8uvBcnpNHG5XGRl5ZGf7zyeh/ikZLVaiIoKJTs7H6ez9r+OK0v1Vr0DQaDWOzo61DscpKodVUCy2+1+Y4MqMmXKFF577TVGjx7N+PHjvd0qMTExhIWFkZqa6nd8amqqd9xSYmIiP/74o9/+oqIiMjMziY+Pr9Q5joV3DJItuMwXrtNZTj9MLfX770u9//aEg+DgEMaOvYuxY++q8HZRUdHMm7fIb9uVV17DlVde47etUaPGPPvszDK3P/PMs73//uSTr8rs93TnjRlzY4VlGDnyKkaOvKrC/ZXlqXd53WtAnQnCFXE6XXW6fhVRvQOL6h0YKvocrwpVHrs84Wj8+PFMmDDBb8yJYRj06NGDxYsX+91m0aJF9OrlHmvSu3dv9u3bx/bt2737Pcf37NmzUuc4Fp5S6jIjIiIiUqWDtBctWsRrr73GqFGjOP/889m/f793X1hYGOHh4YwePZrrr7+eDh06MHDgQD799FP+/vtvJk+eDEDXrl3p0aMHd955Jw8//DB5eXk8+OCDXHjhhd4WoiOd41h4W5B0oVoREZGAV6UtSF9//TXgnrk2YMAAv//eeOMNAAYMGMDjjz/Ohx9+yD//+U8WLlzIrFmzvF13hmHwwgsv0KRJE6666iruuOMOBg4c6LdQ5JHOcSy8M5vUgiQiIhLwDLP0Ba8CVGZ6JukvXYvtlO6EDrndb19xcREHDuylXr2G2O2VWyOptnDPEAuc/mqP8updl59ncNc5NjacjIzcgHrOVW/VOxAEar3j4sKrbWmD2j9nvcqoi01ERETcFJBKaJC2iIiIeCgglfCd5i8iIiKBTQGphLcFSV1sIiIiAU8BqYRakERERMRDAamEWpBqhsPh4KOP3vf+/frrL3PxxedX+f1U13lFRKRuUkDy8rQg1b3p3SezH374lhkznqnpYoiIiPhRQCphMdwByQiOqOGSBBYtwyUiIiejKr3USG1mxb2wliW68he8NU0THEXVVaTDswX5XeeushYs+IPXXpvFtm1bCA0N49RTB3DrrXeyadMG7rzzFh599ElmzZpBSkoKnTp15v77H+bDD9/l22/nYLPZueSSy7jqqmu95/vmm6/573/fZ+fOHcTFxTFs2AWMGjUaq9UKQErKPl5++UWWLl1MXl4uXbp04+abb6d16zbMnfsVjz/+CAADBvTi+ednec/73ntv8emn/yMrK4uOHTtx773307RpMwBycnJ48cXn+O23XyguLiYpqT033zyWdu06eG//xRez+eCDd9i/fz+9e/ehYcNGx/Qwi4hIYFJAKmEAJgaWqPhKHW+aJnlfTsaVsql6C1YBa0IbQof/56hCUmZmJvfffw+33nonp546gNTUFCZNeoiZM5/j7LPPwel08s47b/DQQ5NwOBzcc88dXH315QwbdgGvvPI233//Da+++hIDBgyiVavW/O9/HzBr1gvceuud9O7dl7Vr/2L69KfIysri9tvvIi8vl5tuupZGjRrz5JPTsNuDeOONV7j11ut4660POfPMf5CTk8Pzz0/jiy++JSoqmhUrlrFv315Wr17JlCnPUVxcxGOPPciTTz7Giy++imma3HPPWIKCQnjqqWeJiIjg22/ncNNN1/Lyy2/Stm07fvjhW6ZPf4rbb7+bXr36MH/+L7zyykzi4ysffkVEJLCpi82HGRaHcRRjkAyOvgWnJu3fn0JRUREJCYkkJjakS5duTJ36LBdddKn3mDFjbqRduw506tSFnj17Exoays03j6VZs1MYNepqALZs2YRpmrz33tuMGPEvRoy4hKZNmzFkyLlce+2NfPbZx+Tk5PDdd9+QlZXJY489RYcOnWjTpi0PPzyJ4OAQZs/+H8HBIUREuLs069Wrj91uB8Bms/Hgg4/RunUb2rfvyAUXjGDdurUALFu2hL/+Ws1jjz1Bx46dOOWU5txwwy107NiZjz/+LwCffPIRZ511NiNGXEKzZqdwxRVX07//aSfwkRYRkdpOLUg+XJGVb2EwDIPQ4f+pVV1sbdokcdZZQxg//k7q1atP7959Oe20gfTvP4hVq/4EoEmTpt7jQ0NDadiwkfd+goNDACguLiYzM4P09AN06dLN7z66d++Bw+Fg+/ZtbN68iaZNTyE2Nta7Pzg4hA4dOrJ58+YKyxkXV4/w8ENjwSIjoygsLARgw4Z1mKbJRRcN87tNUVGR95gtWzZx1llD/PZ36tSFjRs3VOZhEhERUUDyE3V0XTCGYdS6a7c9/PBkrrnmOhYu/D+WLFnEww9PpEuXbt5xRTab/0uiohBW0eBql8v0OU9Fx7iw2awVltFiqbhh0+VyER4ezuuvv1dmn6cFCgxM0/9ijaXrJSIicjjqYvN1lAGptlmz5i+ef34azZo151//upwpU57j/vsfYtmyJWRkZBzVueLi6hEXV8/b8uSxcuUK7HY7jRs3oVWrNuzcuZ2MjHTv/sLCQtat+5vmzVsCFQewirRs2Zrc3FyKi4tp0qSp97/333+b33+fB0CbNm1ZtWql3+3Wrfv7qO5HREQCmwKSDyM6saaLUK3Cw8OZPftjZs58nl27drJlyyZ+/PF7mjRpRkxMzFGf79//HsXs2f/js88+YdeunXz//be88cYrDB/+TyIiIvjHP4YSHR3DAw9M4O+/17Bp00YefXQi+fn5XHDBCMDdjQfuAFNYWHDE++zbN5k2bdry0EP3sXz5Unbt2smMGdOZO/crb+i64oqrmT//Fz744B127tzBJ5/8l19//emo6yciIoFL/Q4+Qho0PfJBtVjz5i2YPHkKb775Kp999jEWi4VevfowbdrzpKTsO+rz/fvfVxAUZOejjz7gueemEh+fwMiRV3H55aMAiIiIYMaMl3nhhWe5/fabAejSpSsvvfQ6jRo1BqBHj9506NCJm266hgceeOyI92m1WnnmmZnMnPkcDz44gfz8fJo3b8nkyVPo2bM3AKeeOoCHHprEG2+8wmuvzaJjx85cdtkV/PDDt0ddRxERCUyGqZX6ACjMzyevwIXD4Sqzr7i4iAMH9lKvXkPs9rq10rbNZim3znVdefWuy88zuOscGxtORkZuQD3nqrfqHQgCtd5xceFYrdXTGaYuthLBJV09IiIiIgpIIiIiIqUoIImIiIiUooAkIiIiUooCkoiIiEgpCkhHQRP+6jY9vyIi4qGAVAlWq/uyGEVFhTVcEqlOnufXatXyYCIigU7fBJVgsVgJDY0gJ8d9OY6goOCjvkTGycrlMnA6A6/lxLfepmlSVFRITk4GoaERh70WnIiIBAYFpEqKiooD8IakusJiseByBc6iYh7l1Ts0NML7PIuISGBTQKokwzCIjq5HZGQsTqejpotTJaxWg+joMLKy8gKqFam8elutNrUciYiIlwLSUbJYLFgsdeMyFDabhZCQEPLznQG1NH2g1ltERCpPP5lFRERESlFAEhERESlFAUlERESkFMPU6nheTmfgjUexWi2qdwBRvQOL6h1YArHeFotRbcvuKCCJiIiIlKIuNhEREZFSFJBERERESlFAEhERESlFAUlERESkFAUkERERkVIUkERERERKUUASERERKUUBSURERKQUBSQRERGRUhSQREREREpRQBIREREpRQFJREREpBQFJBEREZFSAjoguVwunn/+eU477TS6devGddddx86dO2u6WFUuJSWFpKSkMv/Nnj0bgL///psrrriCbt26MXjwYN55550aLvHxe/nllxk1apTftiPVsy68Hsqr98SJE8s894MHD/bur631zszM5MEHH2TgwIH06NGDf//73yxdutS7f8GCBYwYMYKuXbsydOhQ5syZ43f7wsJCHnnkEZKTk+nevTt33XUX6enpJ7oaR+1I9R49enSZ59v3NVFb633gwAHuuece+vXrR/fu3bn++uvZvHmzd39dfX8fqd519f3ta+vWrXTv3t37nQUn6Pk2A9iMGTPMvn37mr/88ov5999/m9dcc4159tlnm4WFhTVdtCr166+/mp07dzZTUlLM1NRU73/5+flmenq62bdvX/O+++4zN23aZH7yySdm586dzU8++aSmi33M3nvvPbNdu3bmFVdc4d1WmXrW9tdDefU2TdO8+OKLzenTp/s99wcOHPDur631Hj16tDls2DBzyZIl5pYtW8xHHnnE7NKli7l582Zz06ZNZufOnc3p06ebmzZtMl977TWzQ4cO5v/93/95bz9hwgTzrLPOMpcsWWKuXLnSvPDCC82RI0fWYI0q53D1Nk3TTE5ONj/44AO/5zsjI8N7+9pa70svvdS85JJLzJUrV5qbNm0yb7vtNnPAgAFmXl5enX5/H67epll3398eRUVF5ogRI8y2bduan376qWmaJ+7zPGADUmFhodm9e3fz/fff927Lysoyu3TpYn711Vc1WLKq98orr5jnn39+uftmzZplDhgwwCwuLvZumzZtmnn22WefqOJVmX379pk33HCD2a1bN3Po0KF+QeFI9azNr4fD1dvlcpndunUzv//++3JvW1vrvW3bNrNt27bm0qVLvdtcLpd51llnmc8++6z5wAMPmBdffLHfbcaNG2dec801pmm6H7N27dqZv/76q3f/li1bzLZt25rLly8/MZU4Bkeqd1pamtm2bVtzzZo15d6+ttY7MzPTHDdunLl+/Xrvtr///tts27atuXLlyjr7/j5Svevq+9vXtGnTzCuvvNIvIJ2o5ztgu9jWrVtHbm4uycnJ3m1RUVF06NCBJUuW1GDJqt769etp1apVufuWLl1Knz59sNls3m39+vVj27ZtpKWlnagiVok1a9Zgt9v58ssv6dq1q9++I9WzNr8eDlfvHTt2kJeXR8uWLcu9bW2td2xsLK+88gqdO3f2bjMMA8MwyM7OZunSpX51AvfzvWzZMkzTZNmyZd5tHi1atCAhIaFW13v9+vUYhkGLFi3KvX1trXd0dDTTpk2jbdu2AKSnp/PWW2+RmJhI69at6+z7+0j1rqvvb48lS5bw0Ucf8eSTT/ptP1HPt+3Ih9RN+/btA6Bhw4Z+2+Pj47376ooNGzYQGxvLyJEj2bp1K6eccgo33XQTAwcOZN++fd43n0d8fDwAe/fupX79+jVR5GMyePBgv753X0eqZ21+PRyu3hs2bADg3XffZf78+VgsFgYOHMidd95JZGRkra13VFQUgwYN8tv23XffsX37dv7zn//w2WefkZiY6Lc/Pj6e/Px8MjIySElJITY2luDg4DLH1OZ6b9iwgcjISB599FH++OMPwsLCGDp0KDfffDNBQUG1tt6+HnjgAf73v/8RFBTESy+9RFhYWJ1+f3uUV++6+v4GyM7O5t5772XixIllyn+inu+AbUHKz88HICgoyG97cHAwhYWFNVGkauFwONiyZQtZWVncdtttvPLKK3Tr1o3rr7+eBQsWUFBQUO5jANSpx+FI9ayrr4cNGzZgsViIj49n1qxZTJgwgd9//52bb74Zl8tVZ+q9fPly7rvvPs4++2xOP/30cp9vz99FRUXk5+eX2Q+1v94bNmygsLCQLl268Nprr3HTTTfx8ccfM3HiRIA6Ue+rrrqKTz/9lGHDhnHLLbewZs2agHh/l1fvuvz+fvjhh+nevTvnn39+mX0n6vkO2BakkJAQwP1h6fk3uB/c0NDQmipWlbPZbCxatAir1eqtZ6dOndi4cSOvv/46ISEhFBUV+d3G8wIKCws74eWtLkeqZ119Pdx0001cfvnlxMbGAtC2bVsaNGjAv/71L1avXl0n6v3jjz9y991306NHD6ZOnQq4PwhLP9+ev0NDQ8t9PUDtr/ejjz7K+PHjiY6OBtzPt91u58477+Tee++tE/Vu3bo1AJMnT2blypW89957AfH+Lq/ekydPrpPv788//5ylS5fy1Vdflbv/RD3fAduC5Gl6S01N9duemppKQkJCTRSp2oSHh/u9SADatGlDSkoKiYmJ5T4GQJ16HI5Uz7r6erBYLN4PT482bdoA7mbq2l7v9957j9tuu40zzjiDWbNmeX9FNmzYsNw6hYWFERkZSWJiIpmZmWU+ZGt7vW02mzccefg+37W13unp6cyZMweHw+HdZrFYaN26NampqXX2/X2ketfV9/enn37KgQMHOP300+nevTvdu3cH4KGHHmLMmDEn7PkO2IDUrl07IiIiWLRokXdbdnY2a9eupXfv3jVYsqq1ceNGevTo4VdPgL/++ovWrVvTu3dvli1bhtPp9O5buHAhLVq0oF69eie6uNXmSPWsq6+He++9l6uvvtpv2+rVqwH3L9LaXO8PPviAxx57jJEjRzJ9+nS/5vRevXqxePFiv+MXLlxIjx49sFgs9OzZE5fL5R20DO61VlJSUmp1vUeNGsV9993nd/zq1aux2+00b9681tY7LS2NcePGsWDBAu+24uJi1q5dS6tWrers+/tI9a6r7++pU6cyd+5cPv/8c+9/AGPHjmXy5Mkn7vmumol4tdP06dPNPn36mD/++KPfOglFRUU1XbQq43Q6zYsuusg899xzzSVLlpibNm0yH3/8cbNTp07m+vXrzbS0NLN3797m+PHjzY0bN5qffvqp2blzZ3P27Nk1XfTjMn78eL/p7pWpZ114PZSu948//mi2bdvWnDFjhrl9+3bz119/NQcPHmyOGzfOe0xtrPeWLVvMjh07mrfccovf+i+pqalmdna2uWHDBrNjx47mlClTzE2bNpmvv/56mXWQxo0bZw4ePNhcuHChdz2g0mtInWyOVO93333XbN++vfnBBx+YO3bsMOfMmWP27dvXnD59uvcctbHepmmaY8aMMc8++2xz8eLF5vr1681x48aZvXv3Nnfv3l2n39+Hq3ddfX+Xx3ea/4l6vgM6IDkcDvPpp582+/XrZ3br1s287rrrzJ07d9Z0sarc/v37zQkTJpj9+/c3O3fubF566aXmkiVLvPtXrlxp/utf/zI7depknnHGGea7775bg6WtGqWDgmkeuZ514fVQXr3nzp1rXnjhhWaXLl3M/v37m08++aRZUFDg3V8b6/3SSy+Zbdu2Lfe/8ePHm6ZpmvPmzTOHDRtmdurUyRw6dKg5Z84cv3Pk5uaa999/v9mrVy+zV69e5rhx48z09PSaqE6lVabe7733nnnOOed4X+cvvfSS6XQ6veeojfU2TdPMzs42H3roIbN///5mly5dzGuuucbcsGGDd39dfX8fqd518f1dHt+AZJon5vk2TNM0q6pZTERERKQuCNgxSCIiIiIVUUASERERKUUBSURERKQUBSQRERGRUhSQREREREpRQBIREREpRQFJREREpBQFJBEREZFSFJBEpE7YtWsXSUlJzJ49+7jPNWHCBAYPHlwFpRKR2spW0wUQEakK8fHxfPTRRzRr1qymiyIidYACkojUCUFBQXTr1q2miyEidYS62ETkhPj4448577zz6NSpE6effjozZszA6XQC7i6tUaNG8cknn3DGGWfQvXt3rrrqKtatW+e9vcvl4plnnmHw4MF06tSJwYMHM23aNIqLi4Hyu9i2bdvG2LFj6d+/P926dWPUqFEsW7bMr1xZWVncd9999OnTh969ezNlyhRcLleZ8v/444+MGDGCzp07079/fyZNmkReXp53f0FBAQ8//DADBw6kU6dODB06lNdff71KH0MROXHUgiQi1e7ll1/mmWee4YorruC+++7j77//ZsaMGezdu5fHH38cgL///pstW7Ywbtw4oqOjef7557niiiuYO3cu8fHxvPrqq3z44YeMHz+epk2bsnLlSp555hnsdjtjx44tc5+bNm3iX//6F82bN2fixInY7XbeeecdrrrqKt544w369OmDy+VizJgx7N69m/HjxxMTE8Nrr73G6tWriY+P957rq6++4u677+b888/njjvuYPfu3TzzzDNs2rSJN998E8MwePzxx/n9998ZP3489evXZ/78+Tz99NPExMRw0UUXnbDHWkSqhgKSiFSrgwcPMnPmTC699FImTpwIwIABA4iJiWHixImMHj3ae9ysWbPo1asXAF26dOGss87inXfe4e6772bx4sV06tTJGzb69OlDaGgokZGR5d7vCy+8QFBQEO+88w4REREAnH766QwbNoynn36aTz75hPnz57Nq1SpeffVVBg4cCEBycrLfAG3TNJk6dSqnnXYaU6dO9W5v3rw5V199NfPmzeP0009n8eLF9O/fn/POOw+Avn37EhYWRr169ary4RSRE0RdbCJSrVasWEFBQQGDBw/G4XB4//OEkD/++AOAJk2aeMMRuAddd+/enSVLlgDuwPHHH39w+eWX89prr7Fp0yauuOIKLrjggnLvd/HixZxxxhnecARgs9k477zz+Ouvv8jNzWXp0qXY7XZOO+007zFhYWEMGjTI+/eWLVvYt29fmfL37t2biIgIb/n79u3L//73P6677jree+89du7cyS233MLpp59eNQ+kiJxQakESkWqVmZkJwPXXX1/u/tTUVAASEhLK7KtXrx5r1qwBYMyYMYSHh/Ppp58ydepUpkyZQps2bZg4cSL9+vUrc9usrCzq169fZnv9+vUxTZOcnByysrKIiYnBMAy/Yxo0aFCm/I888giPPPJIheW///77SUxM5Msvv+Sxxx7jscceo3v37jz88MO0a9eu3LqLyMlLAUlEqlVUVBQAU6dOpXnz5mX2169fn+eee46MjIwy+9LS0rxdVBaLhZEjRzJy5EgOHDjAvHnzmDVrFrfddpu3FcdXdHQ0aWlpZbbv378fgNjYWGJjY8nIyMDpdGK1Wr3HeEKRb/nvvfde+vTpU+79gHsW3U033cRNN93Enj17+OWXX5g5cyZ33XUXc+bMqejhEZGTlLrYRKRade3aFbvdTkpKCp07d/b+Z7PZmD59Ort27QLcM842b97svV1KSgorVqwgOTkZgMsuu4xJkyYB7palESNGMHLkSLKzs8nJySlzv7179+aXX37x2+d0OpkzZw6dO3cmKCiI5ORkHA4HP/74o/eYoqIiv8DVsmVL6tWrx65du/zKn5CQwLRp01i7di0FBQUMGTKEN954A4BGjRoxcuRIzjvvPPbs2VOFj6aInChqQRKRahUbG8uYMWN47rnnyMnJoW/fvqSkpPDcc89hGIa3+8k0TW688UbuvPNOrFYrL7zwAtHR0YwaNQpwB5433niD+vXr0717d1JSUnjzzTfp06cPcXFxflPuAW699Vbmz5/PlVdeyfXXX4/dbveODXrttdcA94DsAQMGMHHiRA4cOEDjxo155513SE9P97ZcWa1W7rzzTh588EGsVitnnHEG2dnZzJw5k5SUFDp27EhISAgdO3bkhRdewG63k5SUxNatW/nss88YMmTICXy0RaSqGKZpmjVdCBGp+95//30++OADtm/fTnR0NMnJyYwbN45GjRoxYcIEFi9ezHXXXceLL75Ifn4+p556KuPHj6dJkyYAOBwOXnrpJb788kv27dtHZGQkgwcP5q677iI2NpZdu3Zx5pln8sQTTzBixAjAvXTA9OnTWbp0KYZh0KVLF2699Va/weD5+flMnTqVOXPmUFhYyLnnnktYWBg//fQTP//8s/e4uXPn8tprr7Fx40bCwsLo0aMHd9xxB0lJSQDk5OTw7LPP8tNPP7F//37q1avHueeey+23305ISMgJfKRFpCooIIlIjfMEJN9AIiJSkzQGSURERKQUBSQRERGRUtTFJiIiIlKKWpBERERESlFAEhERESlFAUlERESkFAUkERERkVIUkERERERKUUASERERKUUBSURERKQUBSQRERGRUv4fwQ4H0Ypddl0AAAAASUVORK5CYII=", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkYAAAHJCAYAAABg0/b8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAABeGUlEQVR4nO3dd1QU198G8GeXooh0BcQuCihFQMHYa9BgjRpNBMSCGonYIxgVNHZF7F3UWEMsGCPEGI2xxYb6MxorWAJGEaUIipTdef9g2Nd1AUEpwj6fczzHnblz9353tjx7Z3aQCIIggIiIiIggLesBEBEREX0sGIyIiIiIRAxGRERERCIGIyIiIiIRgxERERGRiMGIiIiISMRgRERERCRiMCIiIiISMRiVUyV1XU5e75OIiNQZg1E5dOzYMfj7+xd7v5cuXcLIkSMVt+Pi4mBtbY39+/cX+31R2fLy8oKXl1ep3FdaWhq+/vprNG3aFC4uLnjw4EG+bZ8/f46lS5eie/fucHR0RMuWLeHt7Y3IyMhC31+nTp0QEBBQDCMvWfv374e1tTXi4uJK5f4WL14MV1dXODo64sCBAyVyHxkZGdi6dSv69euHZs2awdXVFV9++SUOHDig9KVr5cqVsLa2Vtx++/l4+/Zt9OnTB3Z2dnB3d0d2djYCAgLg5OQEZ2dn9OvXD3379lW5/8OHD8Pa2hqfffaZyrrffvsN1tbWiIqKKlQtb4/J2toaK1euzLd9QEAAOnXqVKi+S8uzZ88wadIktGjRAs2aNcPEiRPx9OnTsh7WR0+zrAdARbd169YS6XfPnj2IiYlR3DY1NUVYWBjq1KlTIvdH6uHAgQM4fvw4AgMD0ahRI9SqVSvPdjdu3MCoUaOgoaEBb29vNGnSBKmpqTh27BgmTZqE3377DcHBwdDS0irlCkpGhw4dEBYWBlNT0xK/rzt37mDTpk0YMGAAevfujQYNGhT7fTx79gw+Pj54/PgxvLy84ODgALlcjuPHjyMgIABRUVGYPXs2JBKJyrZBQUFKt1evXo3//vsPq1evhrGxMU6dOoXw8HD4+vqiVatW+P3337Fz5068evUKVapUUWx36tQpGBoa4t69e3j06BFq1qypWHfx4kXo6urC0dGx2GsHAF9fXwwePLhE+n4f2dnZGDFiBNLS0jBz5kxkZ2djyZIlGD58OPbv319hXkclgcGI8qWtrV1ibyKkPpKTkwEAgwYNyvNDEQBevXqFcePGwdjYGNu2bYOBgYFiXZcuXdCxY0f4+fmhfv36GD9+fCmMuuQZGxvD2Ni4VO4rdx90794dzZs3L5H78Pf3x5MnTxAWFoZ69eoplnfo0AEWFhYICQlBx44d0blzZ5VtGzZsqHQ7KSkJVlZWaN++PQAgPDwcANC3b1/Url0bWVlZ+OGHH3D16lW0bNlSsd3p06fh6emJDRs24NSpU/jyyy8V66KiouDq6gpNzZL52PvYvkAePnwYN27cQEREhOLxbdy4MXr06IFff/0VvXr1KuMRfrx4KK2c8fLywoULF3DhwgVYW1vj/PnzAHLe+AIDA9GqVSvY29tjwIABOHv2rNK2Z86cwYABA+Dk5AQXFxeMHj1aMUMUEBCA8PBwPHr0SHH47O1Dafv370eTJk1w9epVDBw4EPb29ujYsSNCQ0OV7ufp06eYMGECXF1d4eLigsDAQCxduvSd08xPnz6Fv78/WrZsCScnJ3h6euLKlSsA8j+s9/b0tZeXFyZPnoyxY8fC0dERQ4cORdeuXTF27FiV++vduzdGjx6tuH306FH07dsX9vb2aN26NebMmYNXr14VOGaZTIadO3eiZ8+ecHBwQIcOHRAcHIyMjAylMQ4ZMgT79u1D165dYWdnh969e+PkyZMF9g3kzLZ8/vnnaNq0KTp06IAlS5YgMzMzz9rze5z+++8/jBkzBs2aNUPr1q2xZcsWlft5/fo1lixZAjc3N9jZ2cHZ2RlDhw7FzZs3CxxfRkYGVq9ejW7dusHe3h5ubm7YsGED5HI5gJz9kXv4wcbGJt9DXIcOHcK///6LmTNnKoWiXG5ubnB3d8fWrVvx8uXLAsdUWIXZ30ePHsWgQYPg5OQEOzs7dOvWDTt37lSsP3/+PKytrfHjjz+iY8eOcHZ2xpkzZwq1z98+lFbY58mVK1fg4eEBR0dHdOjQAT/88AOGDBmS72O7cuVKxSEhb29vxXOmsM9db29vBAUFwdnZGe7u7pDJZCr3cfPmTZw+fRrDhw9XCkW5hgwZAg8PD6XZnTe9edjK2toaFy5cwMWLF2Ftba10aLRLly7w8vJCs2bNUKlSJVy+fFnRx927d/HkyRPFfjh9+rRiXWpqKm7fvo3WrVsDeP/n+5tWrFiBxo0bK0Lb26/HTp06YcWKFVi4cCFatWoFBwcHDB8+XOVQcnh4ONzd3WFvb49evXrh7NmzaNKkSYGnMMyYMQOtW7dW2Rdz585FixYtkJWVhdOnT6N+/fpKobNhw4awtLTEiRMnCl2nOmIwKmeCgoLQpEkTNGnSBGFhYbC1tUVGRga8vb1x7NgxTJgwAatWrYK5uTl8fHwU4Sg2Nha+vr6ws7PD2rVrMXfuXNy/fx8jR46EXC6Hr68v2rdvj+rVqyMsLAwdOnTI8/7lcjnGjx8Pd3d3bNiwAc7Ozli0aBFOnToFAMjMzIS3tzcuX76M7777DvPnz8etW7ewefPmAut6+fIlvvrqK5w/fx7ffvstVq1ahUqVKmHYsGEFnpOSl19//RW6urpYu3YtfHx80KtXL5w4cQJpaWmKNjExMbh16xZ69+4NAPjll1/wzTffoEGDBli9ejXGjBmDgwcPwtfXt8AT0gMDAzF//nx06dIFa9euhYeHB3bs2KGy3fXr1xEaGoqxY8di9erV0NDQgJ+fH1JSUvLte+fOnfD394etrS1WrVqFkSNHYvv27ZgzZ06hH4tXr17B09MTd+7cwezZszFjxgzs2bNHEThzTZkyBfv27cPIkSOxefNmTJ06FXfv3sWkSZPyrV8QBHz99dfYtGkTvvjiC6xbtw7dunXDsmXLFIdGgoKC0L9/fwBAWFgYfH198+zrjz/+QLVq1eDk5JRvLd27d0d6ejrOnDlT6PrzU5j9/eeff+Kbb76Bra0t1qxZg5UrV6J27dr4/vvvcfXqVaX+Vq1aBX9/fwQGBipqeJ99/q5tYmJiMGTIEABASEgI/Pz8sGHDBly6dCnfPr/44gsEBgYCyHm+rlq1SvH/wjx3o6Ki8PjxY6xevRqTJk2ChoaGyn3kvv7z+/JTqVIlBAYGKs3u5CcsLEzpPW7p0qWKLzCrVq1CUFAQKlWqBCcnJ6VgdPr0aRgbG8PW1hZt2rTB2bNnkZ2dDQC4fPky5HK5Ihi9z/P9TaGhoVizZg1mz56Nzz//PN9227Ztw7179zB//nzMmTMH169fVzo/9MCBAwgICICzszPWrFmDrl27wtfXN8/w+abevXvj2bNnii/GQM5786+//oru3btDS0sLMTExeYbUOnXq4P79+++sUZ3xUFo507BhQ1StWhUAFIe5fvrpJ9y6dQs//fQTmjZtCgBo164dvLy8EBwcjH379uHvv//G69evMWrUKJiZmQEAzM3NcezYMbx69Qp16tSBsbGx0uGzvGZLBEGAr68vvvjiCwBAs2bN8Pvvv+PPP/9E27ZtcfDgQdy7dw/79u2DnZ0dAOCTTz5Bly5dCqwrd7YqPDwcjRs3BgA4OzujT58+uHjxYqHeUHNpaWlh1qxZ0NbWBpDzRrBy5UocPXoUffr0AZAzQ6Gvr49OnTpBEAQEBwejbdu2CA4OVvRTr149DBkyBCdOnMgzKEZHR2Pv3r2YNGmS4qT11q1bw9TUFFOmTMHJkycVhwJSU1Oxf/9+xXR7lSpV4OnpiXPnzqFr164qfcvlcqxevRpdunRRCkLp6emIiIhAVlZWoR6L8PBw/Pfffzh06JDim2PTpk3x6aefKtpkZmbi5cuXmD59Otzd3QEArq6uSEtLw4IFC/Ds2TNUr15dpe+TJ0/ir7/+QkhICLp3766ov3Llyli+fDkGDx6MRo0awdzcHAAKPCz79vkgecl97B49elSo2vNT2P0dHR2Nzz//HNOmTVO0cXJyQosWLXD+/HnFaw3IOUzYrVs3pfsp6j4vzDbr16+Hnp4eNm3aBB0dHQBAgwYNlA4Zvc3c3Fyx7xs2bIgmTZoU6bmbnZ2N77//XrEf8/L48WMAyPf8saJwdHRUeY/Lndlu3Lix4j5atmyJTZs2QS6XQyqV4tSpU2jVqhUkEgnatGmD4OBgXLlyBS4uLrh48SJq1KiBBg0avPfzPdfu3buxePFifP/994rQnx99fX2sWbNGESb//fdfrFy5EklJSTAyMsLy5cvRsWNHxWu8bdu20NLSwpIlSwrst1mzZqhZsyYOHTqEVq1aAciZvUxISFB82UtNTUXdunVVttXV1S22WdeKijNGFcDZs2dRvXp12NraIjs7G9nZ2ZDJZOjYsSOuX7+OlJQUNG3aFJUqVUL//v0xd+5cnDp1CjY2NpgwYYLiTaiw3vxWr62tDWNjY0WIOnfuHGrXrq0IRQBQtWpVdOzYscA+L126hFq1ailCEQDo6Ojgt99+U4SwwmrQoIEiFAFA7dq14ezsrPTLpoiICHTr1g3a2tq4d+8enjx5gk6dOikev+zsbLi4uKBq1ar5zlBcuHABABShIFf37t2hoaGh9G3O2NhY6RyE3A+Z9PT0PPu+f/8+nj9/rhRgABT5xMmoqCjUqVNHaTq9Ro0aSiFFW1sboaGhcHd3R3x8PM6dO4cff/wRx48fBwDFobu3XbhwAZqamiqBIPfchdzHp7hIpTlvV7nfpt/cV9nZ2YW+1ERh97ePjw8WLFiAly9f4vr164iMjMT69esBqD4mbz5vcxV1nxdmm3PnzqFdu3aKUATkvB7fFSrfVpTnrqGhYYGhCIDig/9dMx3FqVWrVkhNTcWdO3fw+vVrREVFoW3btgByDttWq1YNf/31F4Cc10FugHjf5zsAHD9+HLNmzULz5s0xYMCAd47R3t5eaYbtzf358OFD/Pfffyqvn7f3ydvPc7lcDolEgl69euHo0aOK8UZERKBevXqKwF7Q6yG/c/0oB2eMKoDk5GQkJCTA1tY2z/UJCQlo2LAhduzYgQ0bNmDv3r3Ytm0b9PX1MWjQIIwfP75IL5TKlSsr3ZZKpYoXYVJSEkxMTFS2yWvZ2zW8q01h6erqqizr3bs3Zs+ejaSkJMTFxeHhw4eYN2+e4r4BYNasWZg1a5bKtvn9vDX38Mbb3y41NTVhZGSE1NRUxbI3P8iA/39jyj0X5225Y/rQxyQlJQVGRkYqy6tXr45nz54pbp86dQrz5s3DvXv3oKurCxsbG8X5IPm9web2/fahldzH483636VmzZr4+++/C2wTGxsLALCwsEBcXJzKSbzz58/P8yfcbyvs/k5MTERQUBCOHj0KiUSCunXrKk5cfvsxyevcmaLu88Jsk5iYmOdzolq1avn2mZeiPHfzej29LTeY/ffffyonUueKj4+HqalpsX0o29raQk9PD5cvX0Z8fDwyMzMVh8okEglatmyJ8+fP4/Xr17h+/brSL8be5/kOAP/88w86dOiAP//8E3/88cc7z5t8e3/mhnu5XI7ExEQAqq/xt/fl2+/rY8aMgZ+fH3r37o21a9fi1KlTaNu2LY4cOQJvb29Fu6pVq+Y5M5SWlgY9Pb0Cx63uGIwqAD09PdSrV0/psMCbcqeeHRwcsGrVKmRmZuLSpUsICwvDunXrYGNjk+d1P96HmZlZnucEPX/+vMDt9PT08ryey+XLl2FgYKAIY29/I33XydG5PvvsM8yZMwdHjx7FvXv3ULNmTTRr1gxAznQ3kHPegaurq8q2eZ0M/ObyhIQEpW/sWVlZiqny95U7ptw3z1xJSUm4ceMGnJycIJFI3vl4GBkZ4eHDhyr954YDIGd6/5tvvkGXLl2wfv161K5dGxKJBDt37lScO5IXAwMDJCUlQSaTKYWj3GBRlPo7deqE48ePIyoqSulXUzdu3ICNjQ2kUimOHDkCLS0tfPLJJ6hatSr27t2r1EdhD+MUdn9PnjwZ9+7dw9atW+Hk5ARtbW2kp6fjp59+KnRdxc3c3Fwp0OZ6/vx5kX6CX9zP3TZt2gAATpw4kWcwys7ORu/evRXn0hQHDQ0NuLq64urVq4iNjYW1tbVS0GvTpg2mT5+OixcvIjs7G5988gmA93++A8DAgQMxa9YsfPXVV5g1axZcXV2LPOOeK3f26O33xrdvv/08z728Q/369eHg4IBff/0VUqkUL168UPqlWf369fM8mfzff/+Fg4PDe41ZXfBQWjmU+60jl6urKx4/fgwTExPY29sr/p05cwabNm2ChoYGtm7dio4dOyIzMxPa2tpo2bIlZs+eDSDnW15e/b4PV1dXxMXFKb0gX79+/c43nObNmyM2NhZ3795VLMvIyICfnx/27t2rePOJj49XrM/KynrnLEMufX19dOzYEceOHcNvv/2GXr16Kb65NmjQACYmJoiLi1N6/MzMzLBkyRLcuHEj31qBnCnsN0VEREAmkymC1/to0KABjIyMFNP7uX7++WeMHDkSWVlZ0NXVRVJSktKviN4+CfeTTz5BXFwcrl27pliWmJiI//3vf4rb169fR0ZGBkaOHIk6deooHpfcfZbfN2hXV1dkZ2fj8OHDSssPHjwIAEWqv2fPnqhXrx5mzZqlCG25Pyro0aMHNm/ejJ9//hmenp6Kc+He3Ff29vaF/jAv7P6+dOkS3Nzc0KJFC8Wh2dxfiBU061OSXFxccOrUKaV9fuPGjSJfJLK4n7uNGjVCu3btsHHjRsXM3pvWr1+PpKSkYv+JeKtWrfD333/j4sWLinCWq3Xr1sjKysKePXvQpEkTxaUR3vf5DuTMsEkkEsycORPPnj1757lABTE3N0edOnXw+++/Ky0/cuSI0u23n+e554gCOTPhp06dQkREBJydnVG7dm3FujZt2iAmJgbR0dGKZdHR0YiJiVHMrFHeOGNUDunr6+PKlSuKn3X27dsXO3bswNChQ/H111+jRo0a+Ouvv7Bx40Z4enoqvmUHBwfjm2++gaenJzQ0NPDjjz9CW1tbcf6Pvr4+nj17hhMnTuR5zkRh9OjRAxs2bMA333yDcePGQV9fH1u2bMHz589hYWGR73Z9+/bF9u3bMXr0aIwdOxZGRkbYtm0bsrKyMGjQIBgYGMDJyQnbt29H3bp1YWBggG3btuH169f5/gT4bb169cLYsWMhk8kUJygCOd88J0yYgMDAQGhoaKBjx4548eIF1qxZg/j4+HwPUTZs2BCff/45VqxYgfT0dLi4uODmzZtYtWoVWrRooTjf4X3k/hrp+++/h4mJCTp16oT79+9jxYoV8PDwgIGBATp27Ijt27dj2rRp6N+/P+7cuYMtW7Yozd707t0b27Ztw5gxYxTnk61du1bpg93W1haamppYvHgxhg0bhszMTOzfvx9//vkngPxn5dq1a4cWLVpg+vTpiI+Ph42NDS5cuICNGzfi888/z/eQSl50dHSwYsUKjBo1Cn369MGQIUPQpEkTBAUFYfbs2Vi4cCGMjIzw9ddfF6q/6OjoPC+E6uzsDAcHh0LtbwcHB/zyyy+wtbWFubk5Ll++jA0bNkAikRR4nlBJ+vrrrxEZGQkfHx8MGzYML168wPLlyyGVSot0iKoknruzZs2Ct7c3BgwYgMGDB6Np06Z4+fIlDh8+jIiICHz55Zcq59N8qJYtW2LOnDnQ0NDApEmTlNZVr14dVlZW+OOPPzB06FDF8vd9vr/JxsYG3t7e2Lx5M3r27AlnZ+cij10ikWDs2LGYPHkygoKC8Omnn+LWrVtYvXo1gMJ9UXV3d8eCBQsQGRmpcpFMd3d3rFu3DiNGjFA8NkuWLIGVlVWxHSGoqBiMyiEPDw9cv34dI0aMwPz589GzZ0/s3LkTS5YsweLFi5GamoqaNWti0qRJGDZsGICcF/K6deuwevVqTJw4ETKZDHZ2dti8ebNiCr5v3744ceIEvvnmG4wdO1bxi42i0NTURGhoKObOnYuZM2dCU1MTvXr1gqGhYYE/Ea1atSp27NiBRYsWYfbs2ZDL5XB0dMS2bdsU34IWLFiA2bNnY/r06ahatSr69++PZs2aYc+ePYUaW/v27aGnp4fatWujfv36Suu++OIL6OrqYtOmTQgLC0OVKlXg7OyM4OBgpW9hb5s7dy7q1q2Lffv2YePGjTA1NcXgwYPh6+v7wTNwudd9CQ0NRVhYGMzNzTFixAiMGDECQM43Yn9/f2zfvh2//fab4mf9b/5CSVtbGz/88APmzZuHuXPnQiKRYMCAAahdu7Ziyr5u3bpYsmQJVq1ahdGjR8PAwACOjo7Yvn07vLy8EBUVpfTnG3JJJBKsX78eK1aswNatW5GYmIhatWph4sSJSh9EhZV7/aVt27Zhz549ePToESpVqgQrKyt06tQJ+/fvR8+ePREQEKBygurbrl27pjRLlmvcuHFwcHAo1P7Ofb7lzqzmzmgdPHiw0H9WorjVrVsXoaGhWLRoEcaOHQsTExOMGjUKa9euLdS5QG8q7ueuhYUFwsLC8MMPP+DQoUPYsGEDtLW10aBBAyxZsuS93k/exdLSEtWrV0daWlqes1xt2rTBnTt3FCdeA+//fH+bn58fDh8+jOnTp7/3n1jp2bMnXr16hdDQUOzbtw+NGjXCtGnTMG3atEJ94TM2NkabNm1w5swZldCpra2NLVu2YO7cuZgxYwa0tLTQunVrTJ06tcQucllRSAT+1VAqRnfv3sW9e/fg5uam9A22f//+MDc3V1xDhaioMjMzceDAAZiamuZ7na2K7uzZs9DS0lI6D+vFixdo1aoVpkyZ8lH9SQp6t0OHDqFJkyZK54f9+eefGDVqFH7++WfY2NiU4ejUF2MjFavcP+0waNAgfPrpp5DJZIiMjMT169cxefLksh4elWPa2tqF+ol0RfbPP/9gxYoVmDhxImxtbZGcnIwtW7ZAT08PPXr0KOvhUREdPHgQS5cuxfjx41GjRg08fPgQK1asgKurK0NRGeKMERW7w4cPIzQ0FDExMRAEAU2aNMHo0aNVTo4koqKRy+VYt24dfv75Zzx+/BhVqlSBq6srJk2alOfF/OjjlpSUhCVLluDkyZNITExEtWrVFH/CqKiHRqn4MBgRERERifhzfSIiIiIRgxERERGRiMGIiIiISMRgRERERCRS25/rC4IAuVz9zjuXSiWsW42wbvXCutWLOtYtlUqK7Q8R50dtg5FEIsGLF6+QnV02f/OoLGhqSmFkpMu61QTrZt3qgHWrV93GxrrQ0CjZYMRDaUREREQiBiMiIiIiEYMRERERkYjBiIiIiEiktidfExFR+SSXyyGTZYv/l+D1aw1kZmZAJlOfX2hVxLo1NDQhlZb9fA2DERERlQuCIODFi0Skp6cpLX/2TAq5XH1+mZWrItato1MV+vrGJf6T/IIwGBERUbmQG4qqVjWCtnYlxYenhoakwsyaFEVFqlsQBGRmZiAtLQkAYGBgUmZjYTAiIqKPnlwuU4SiqlX1ldZpakrV6lo+uSpa3dralQAAaWlJ0NMzKrPDamV/MI+IiOgdZDIZgP//8KSKKXf/5p5DVhYYjIiIqNwoy3NPqOR9DPuXwYiIiIhIxGBEREREJGIwIiIiojyNGTMSc+fOLOthlCoGIyIiIiIRgxERERGRiNcxIiKicksQBGRkysrsej7aWtIi/5KqTZvmGDp0BCIjf0F2dhZWrdoIc/Ma2LhxLY4c+RUvX6ahfn1L+Ph8DVfXTxATEw1v7y8RGroD1tY2AICpUyfj8uUoREYeg4aGBuRyOXr1coOf30R07eqOX345gL17f0RsbCykUgmsrGwwduxE2Ng0AQD0798THTp0xrlzZ5CUlIg5cxbB1tYe69atxJEjh5GVlYnevftBEJQvILlr13YcOLAXCQlPUa1adXTv3gve3sM/il+TFRcGIyIiKpcEQcD8HZcR/SilzMbQsJYBpno4FzkYhIfvQXDwCmRny1C7dh3MnDkNDx/eR2DgbFSvboozZ05iypTxmDcvGK1atUGNGha4ePEcrK1tIJPJcOVKFF69eok7d26hcWNb3LjxD1JTU9GyZRucOHEcS5cugr//dDRt6oRnz55h2bLFWLBgDrZu3aUYw/79P2HhwqXQ09NDgwYNsWzZYpw5cwrTpgXBzKwGtm3bjKtXr8DCoiYA4PTpk9i+fQu+/34eateuh3/++Rtz5gShRg0LdO3qXqyPa1liMCIiovKrnE5UdO3qrpi9iYuLxdGjv2HLlp1o1MgaAPDll56Ijr6LXbu2oVWrNmjdui0uXjwPT88huHnzH2hqasHOzgGXL0ehcWNbnD17Gk2bOkFfXx8GBgYICJgBN7fPAADm5jXQo0cvhIQsUhrDJ5+0hotLCwDAq1cv8euvhzBpkj9atmwDAJg6NRCXL0cp2v/3Xxy0tbVgbm4Bc3NzmJubo1o1U5iZmZf441WaGIyIiKhckkgkmOrhDLmAcnUoDQBq1aqj+P+dO7cBAL6+PkptsrOzUbWqHgCgdeu2OHgwHBkZr3Hx4nk0a9YcFhY1celSFDw8vHH27Gl069YDAODo6IwHD+5j69ZNePjwAeLi/kVMTLTKH5ytVau24v///vsQWVlZsLGxVSyrVKkSrKysFbfd3NwREXEQX33VF/XqNYCLSwt06NAZ5uYMRkRERB8FiUSCSlpSaEjL19RRpUr//6dNBCEnsKxevRFVqugqtcv9e2FOTs2hpaWFK1cuIyrqArp2dUfNmjWxd28Ynjx5jLt372Du3PYAgCNHDmPu3CC4uX0GOzsH9O7dF/fuxSAkZGG+Y8idessdSy5Nzf+PCYaGhtiyZReuX/8bFy+ex/nzZ7Fnz24MHz4KQ4eO+LAH5CPCX6URERGVofr1LQEAz58/Q61atRX/IiIOIjLyFwA5AcXVtSVOnz6BGzeuo1kzFzRt6giZTIbQ0PVo0KAhatSwAADs3LkVPXv2wbRpM9Gv3wA4Ojrj0aM4AFA5mTpXnTp1oa1dCX//fVWxLDs7G3fv3lHcPnLkV4SH74WDgyOGDx+FDRty7ufYsSMl8riUFQYjIiKiMtSggSVatWqLxYvn4/Tpk3j0KA47d/6AHTu2ombNWop2bdq0Q2TkL6hWrTpq1qyFypUrw87OAb/9Fom2bdsr2pmamuHatau4ffsWHj2KQ1jYTuzf/xMAIDMzM88xVKlSBf37D8Dmzetx4sQfePjwAYKD5+PZswRFm8zMDKxevRyHD0fg8eP/cPXq/3DlymXY2TmU0CNTNngojYiIqIx9//18bNiwGosXz0Nq6gtYWNRCQMAMfPZZD0Wbli1bQyaTwdm5uWJZ8+auuHw5Cm3a/H8wmjBhChYtmosxY0ZCW1sLDRtaYfr0WQgK+g63bt1A06ZOeY5h1Kgx0NauhJCQhXj16hU6dfoUrVu3U6zv0aMPUlJSsHXrJjx9Gg89PT106NAZo0ePLYFHpOxIhPzm1YpBYGAgMjMzsWDBAqXl+/btw9atWxEbGwtTU1P0798fw4cPh4aGxjv7vH//Pvr27YsZM2agb9++HzS+pKSXZXbCXlnQ1JTCyEiXdasJ1s26K5KsrEw8f/4YJiY1oKWlrbROU1NaIWt+l4pYd0H7GQCMjXWhoVGyB7tKpHe5XI6QkBCEhYWprDt48CCCgoLg6emJgwcPYvz48Vi/fj3Wrl37zn6zsrIwefJkvHr1qiSGTURERGqu2A+lxcTEYNq0aXj48CEsLCxU1u/evRt9+vTBwIEDAQB16tTB/fv3sWfPHowZM6bAvleuXImqVasW95CJiIiIAJRAMDp37hwsLS2xevVqjB8/XmX95MmTYWxsrLRMKpUiJaXgK5devHgRYWFhOHDgADp06FAsYy3p6biPTW69rFs9sG7WXZHI5Xn/HD/3EkISCVByJ4Z8fCp63RoaEmhqqj6XS+MvjxR7MPLw8ChwfbNmzZRup6amYvfu3Wjbtm2+27x48QJTpkzB9OnTUaNGjWIZJwDo6+sUW1/lCetWL6xbvVTUul+/1sCzZ9J8PzAraiB8l4pWt1wugVQqhYFBFVSuXLlMxlCkYBQXF4fOnTvnu/7s2bMqs0EFefnyJXx9fZGRkYEpU6bk227mzJlwcnJCz549izLcd3rxIh0yWcU6ca0gGhpS6OvrsG41wbpZd0WSmZkBuVwOmUxQOuFYIsmpXSaTV8iZk/xU1LplMgFyuRwpKa+Qni5TWW9goKO46GVJKVIwMjMzQ2RkZL7rDQwMCt1XQkICRo0ahbi4OISGhqJWrVp5tjtw4ACioqLwyy+/FGWohSKTySvcGf2FwbrVC+tWLxW1bpks70//3FBQkcJBYVT0ut8OwLlKo94iBSMtLS1YWlp+8J3GxMTAx8cHcrkcO3fuRKNGjfJtu2/fPjx//lzlvKKgoCBERkZi06ZNHzweIiIiIqAMLvAYGxsLb29v6OvrIzQ09J3nDAUHB+P169dKy9zc3DB27Fj06tWrJIdKREREaqbUg9F3332HzMxMhISEQFNTEwkJ/3+58erVqwMAEhMToaWlBT09PZiZmeXZj4mJSb7riIiIiN5HqQaj+Ph4XLhwAQDQu3dvlfW3b98GAPTv3x+urq4qV8wmIiIiKkklGoy2b9+udNvMzEwRfgryxx9/FLi+MH0QERFR4WRnZ2PfvjAMHJhzyZ3Q0PX49ddD2Lu3eH/4VFL9FqeKdQEEIiIiKrLffz+MlSuXlvUwPgoMRkRERGquBP+efLlT6idfExERFRdBECBkZUAoq2s3aWpD8h5/p+Ls2TPYtGkdHjy4Bx2dKmjZsjX8/CYiOvoOJkz4Bt9/vwDr1q1EfHw87OzsMW3aTOzevR2HD0dAU1MLX3zxJYYPH6Ho79dfD+HHH3ciNvZfGBsbo0eP3vDyGgoNDQ0AQHz8E6xfvxpRURfw6tVLODg4wtd3HBo2bITIyF8wb94sAECbNs2xYsU6Rb87dmzFvn0/ISUlBba2dpgyZRpq164DAEhLS8Pq1ctx6tRxZGVlwdq6MXx9x8LGpoli+59/3o9du7YhISEBLi6uqFFD9W+ofmwYjIiIqFwSBAGvDs6FPD66zMagYdYIOr2+K1I4Sk5OxrRp32LMmAlo1aoNnj6Nx+zZQVizZjnc3D6DTCbDtm2bERQ0B9nZ2fj22/EYMmQQevTojQ0bfsCRI79i48a1aN++A+rVs8RPP+3CunWrMGbMBLi4tMCNG9cRErIQKSkpGDduEl69eonRo4fDwqImFixYAi0tbWzevAFjxozA1q270bnzp0hLS8OKFUvw88+Hoa9vgCtXLuHJk8e4du0qFi9ejqysTMyeHYgFC2Zj9eqNEAQB3347FtralbFw4TJUrVoVhw9HYPTo4Vi/fgusrGzw+++HERKyEOPGTUbz5q44efI4NmxYA1PTj/sX5TyURkRE5ZYEpfBXRYtZQkI8MjMzYWZmDnPzGnBwcMTChSHo12+goo2Pz9ewsWkCOzsHNGvmAh0dHfj6jkWdOnXh5TUEAHDvXgwEQcCOHT+gb98B6Nv3C9SuXQddu7pj+PCvER6+B2lpafjtt1+RkpKM2bMXokkTOzRqZIWZM+egUqXK2L//J1SqVBlVq1YFAJiYVIOWlhYAQFNTE4GBs9GwYSM0bmyL3r374tatGwCAS5cu4vr1a5g9ez5sbe1Qt249jBr1DWxt7bFnz48AgL17w9Clixv69v0CderUhafnELRunf/fRf1YcMaIiIjKJYlEAp1e30ET2WX3Z1De41Bao0bW6NKlK/z9J8DEpBpcXFqgVau2aNeuA/7++38AgFq1aiva6+jooEYNC8X9VKqU88dVMzMzkZychMTE53BwcFS6DycnZ2RnZ+PhwweIiYlG7dp1YWRkpFhfqVJlNGlii5iYmHzHaWxsAl3dqorbenr6yMjIAADcuXMLgiCgX78eSttkZmYq2ty7F40uXboqrbezc8Ddu3cK8zCVGQYjIiIqtyQSCSSalSCRlK+/Dzdz5lwMGzYC5879hYsXz2P27BlwcHCEt/dwADmzNW/KL3zld9K0XC680U9+beTQ1NTId4wF/bFWuVwOXV1dhIbuUFmXO+MESCAIyvvl7bo+RjyURkREVIr++ec6VqxYgjp16mHAgEFYvHg5pk4NxKVLF5GUlFSkvoyNTWBsbKKYacp19eoVaGlpoWbNWrC0bITY2IdISkpUrM/IyMCtWzdRr14DAPkHr/w0aNAQL1++RFZWFmrVqq34t3PnDzh9+gQAoFEjK/z991Wl7W7dulmk+ykLDEZERESlSFdXF/v378GaNSsQFxeLe/eicezYEdSqVQeGhoZF7u+rr7ywf/9PCA/fi7i4WBw5chibN29Ar16fo2rVqvj0024wMDDEjBkBuHnzH0RH38X3309Heno6evfuCyDncB2QE1wyMl4XdHcAgBYtWqJRIysEBU3F5ctRiIuLxcqVIYiM/EURtjw9h+DkyePYtWsbYmP/xd69P+LPP48Vub7S9vHPaREREVUg9erVx9y5i7Fly0aEh++BVCqFs7MLlixZgfj4J0Xu76uvPKGtrYWwsF1YvjwYpqZm8PDwxqBBXgCAqlWrYuXK9Vi1ahnGjfMFADg4NMXataGwsKgJAHB2dkGTJnYYPXoYZsyY/c771NDQwNKla7BmzXIEBgYgPT0d9eo1wNy5i9GsmQsAoFWrNggKmoPNmzdg06Z1sLW1x5dfeuL33w8XucbSJBHU+KpOSUkvy+6EvTKgqSmFkZEu61YTrJt1VyRZWZl4/vwxTExqQEtLW2mdpqa0Qtb8LhWx7oL2MwAYG+tCQ6NkD3bxUBoRERGRiMGIiIiISMRgRERERCRiMCIiIiISMRgREVG5oca/F1ILH8P+ZTAiIqKPXu5fic/MzCjjkVBJyt2/GhpldzUhXseIiIg+elKpBnR0qiItLefK0NralRRXa5bLJZDJyn6mobRVpLoFQUBmZgbS0pKgo1O1wD9HUtIYjIiIqFzQ1zcGAEU4yiWVSiGXV6zr+RRGRaxbR6eqYj+XFQYjIiIqFyQSCQwMTKCnZwSZLBsAoKEhgYFBFaSkvKowsyeFURHr1tDQLNOZolwMRkREVK5IpVJIpTlXRdbUlKJy5cpIT5dVuKtAF0Rd6y4NZR/NiIiIiD4SDEZEREREIgYjIiIiIhGDEREREZGIwYiIiIhIxGBEREREJGIwIiIiIhIxGBERERGJGIyIiIiIRAxGRERERCIGIyIiIiIRgxERERGRiMGIiIiISMRgRERERCRiMCIiIiISMRgRERERiRiMiIiIiEQMRkRERESiEg1GgYGBCAgIUFm+b98+9OzZE46OjnBzc8OGDRsgk8kK7OvAgQNwd3eHvb09unfvjl9//bWkhk1ERERqqkSCkVwuR0hICMLCwlTWHTx4EEFBQfD09MTBgwcxfvx4rF+/HmvXrs23v59//hnTpk2Dh4cHIiIi0KNHD0ycOBFXrlwpieETERGRmtIs7g5jYmIwbdo0PHz4EBYWFirrd+/ejT59+mDgwIEAgDp16uD+/fvYs2cPxowZo9JeEAQsX74cgwcPhoeHBwBg9OjRiIqKwoULF+Dk5FTcJRAREZGaKvZgdO7cOVhaWmL16tUYP368yvrJkyfD2NhYaZlUKkVKSkqe/d2/fx+PHj1Cz549lZaHhoYW25iJiIiIgBIIRrmzOvlp1qyZ0u3U1FTs3r0bbdu2zbP9/fv3AQCvXr3C8OHDcePGDdSqVQujR49Gp06dPmisGhrqde55br2sWz2wbtatDli3etUtkZT8fRQpGMXFxaFz5875rj979qzKbFBBXr58CV9fX2RkZGDKlCl5tklLSwMA+Pv7Y8yYMZg8eTJ+++03+Pr6YsuWLWjZsmVRSlCir6/z3tuWZ6xbvbBu9cK61Yu61l2SihSMzMzMEBkZme96AwODQveVkJCAUaNGIS4uDqGhoahVq1ae7bS0tAAAw4cPx+effw4AaNy4MW7cuPHBwejFi3TIZPL33r680dCQQl9fh3WrCdbNutUB61avug0MdCCVluwsWZGCkZaWFiwtLT/4TmNiYuDj4wO5XI6dO3eiUaNG+bY1MzMDAFhZWSktb9iwIf78888PGodMJkd2tvo8oXKxbvXCutUL61Yv6la3IJT8fZT6wcnY2Fh4e3tDR0cHP/74Y4GhCABsbW2hq6uLq1evKi2/c+cO6tSpU5JDJSIiIjVT7Cdfv8t3332HzMxMhISEQFNTEwkJCYp11atXBwAkJiZCS0sLenp6qFy5Mnx8fLB69WqYmZnBwcEBEREROHPmDLZu3VrawyciIqIKrFSDUXx8PC5cuAAA6N27t8r627dvAwD69+8PV1dXLFiwAADg6+sLHR0dLF26FPHx8bC0tMTKlSvRokWL0hs8ERERVXgSQSiNI3Yfp6Skl2p1bFZTUwojI13WrSZYN+tWB6xbveo2NtYt8UsUqNcFEIiIiIgKwGBEREREJGIwIiIiIhIxGBERERGJGIyIiIiIRAxGRERERCIGIyIiIiIRgxERERGRiMGIiIiISMRgRERERCRiMCIiIiISMRgRERERiRiMiIiIiEQMRkREREQiBiMiIiIiEYMRERERkYjBiIiIiEjEYEREREQkYjAiIiIiEjEYEREREYkYjIiIiIhEDEZEREREIgYjIiIiIhGDEREREZGIwYiIiIhIxGBEREREJGIwIiIiIhIxGBERERGJGIyIiIiIRAxGRERERCIGIyIiIiIRgxERERGRiMGIiIiISMRgRERERCRiMCIiIiISMRgRERERiRiMiIiIiEQMRkREREQiBiMiIiIiEYMRERERkahEg1FgYCACAgJUlu/btw89e/aEo6Mj3NzcsGHDBshksnz7yc7OxvLly9GxY0c4OTnBw8MD//vf/0pw5ERERKSOSiQYyeVyhISEICwsTGXdwYMHERQUBE9PTxw8eBDjx4/H+vXrsXbt2nz7W7t2Lfbs2YPZs2fjwIEDqF+/Pnx8fPD06dOSGD4RERGpqWIPRjExMRg0aBD27NkDCwsLlfW7d+9Gnz59MHDgQNSpUwfu7u4YNmwY9u7dm2+fR48eRY8ePdCmTRvUrVsXAQEBSE1N5awRERERFatiD0bnzp2DpaUlDh06hFq1aqmsnzx5MoYPH648CKkUKSkp+fZpYmKC48ePIy4uDjKZDGFhYdDW1oaNjU1xD5+IiIjUmGZxd+jh4VHg+mbNmindTk1Nxe7du9G2bdt8t5k2bRrGjRuHzp07Q0NDA1KpFCtXrkSdOnU+aKwaGup17nluvaxbPbBu1q0OWLd61S2RlPx9FCkYxcXFoXPnzvmuP3v2LIyNjQvd38uXL+Hr64uMjAxMmTIl33bR0dHQ09PD6tWrYWZmhj179mDy5MnYsWMHGjduXJQSlOjr67z3tuUZ61YvrFu9sG71oq51l6QiBSMzMzNERkbmu97AwKDQfSUkJGDUqFGIi4tDaGhonofdAODx48eYNGkStm7diubNmwMA7O3tER0djZUrV2LNmjVFKUHJixfpkMnk7719eaOhIYW+vg7rVhOsm3WrA9atXnUbGOhAKi3ZWbIiBSMtLS1YWlp+8J3GxMTAx8cHcrkcO3fuRKNGjfJte/XqVWRlZcHe3l5pedOmTXHy5MkPGodMJkd2tvo8oXKxbvXCutUL61Yv6la3IJT8fZT6wcnY2Fh4e3tDR0cHP/74Y4GhCADMzc0BALdv31ZafufOHdSrV6+khklERERqqNSD0XfffYfMzEyEhIRAU1MTCQkJin+5EhMTkZqaCgBwcHBAs2bN4O/vj3PnzuHBgwdYtmwZzp49i5EjR5b28ImIiKgCK/ZfpRUkPj4eFy5cAAD07t1bZX3urFD//v3h6uqKBQsWQCqVYu3atVi2bBmmTp2KlJQUWFlZYevWrWjatGlpDp+IiIgqOIkglMYRu49TUtJLtTo2q6kphZGRLutWE6ybdasD1q1edRsb65b4JQrU6wIIRERERAVgMCIiIiISMRgRERERiRiMiIiIiEQMRkREREQiBiMiIiIiEYMRERERkYjBiIiIiEjEYEREREQkYjAiIiIiEjEYEREREYkYjIiIiIhEDEZEREREIgYjIiIiIhGDEREREZGIwYiIiIhIxGBEREREJGIwIiIiIhIxGBERERGJGIyIiIiIRAxGRERERCIGIyIiIiIRgxERERGRiMGIiIiISMRgRERERCRiMCIiIiISMRgRERERiRiMiIiIiEQMRkREREQiBiMiIiIiEYMRERERkYjBiIiIiEjEYEREREQkYjAiIiIiEjEYEREREYkYjIiIiIhEDEZEREREIgYjIiIiIhGDEREREZGIwYiIiIhIVKLBKDAwEAEBASrLt2/fDjc3N9jb26N79+7Yt2/fO/vauXMnOnfuDAcHBwwaNAg3btwoiSETERGRGiuRYCSXyxESEoKwsDCVdWFhYQgODoafnx8iIyPh7e2NGTNm4OjRo/n2Fx4ejkWLFmHcuHHYv38/atWqhaFDhyIxMbEkhk9ERERqSrO4O4yJicG0adPw8OFDWFhYqKxPTU3FpEmT0LNnTwBA7dq1sWvXLpw5cwZdunTJs89169bB09MTvXr1AgDMmzcPXbp0wZ49ezBq1Kj3GqdcLsfrl6+QnS1/r+3LI01NKV5pCnj9Mp11qwHWzbrVAev+uOrW1qkMqbR8n6VT7MHo3LlzsLS0xOrVqzF+/HiV9T4+Por/Z2Vl4ffff0dMTAzGjBmTZ3/Pnz/HgwcP0LJly/8ftKYmmjdvjosXL753MJKlJCD9B9/32ra8ygKQXtaDKAOsW72wbvXCuj8uD6U10Hjk/BILRxJJiXSrpNiDkYeHR6HaRUVFwcvLC3K5HP369UPnzp3zbPfkyRMAQI0aNZSWm5qa4tatWx82WCIiIio2EokERka65XrWqEjBKC4uLt8AAwBnz56FsbFxofqqX78+wsPDce3aNcybNw9GRkb49ttvVdqlp+dkYm1tbaXllSpVQkZGRhFGr0zDoDp0h62HXCa8dx/ljVRDAr2qlZGa9pp1qwHWzbrVAev+uOq20amMlJSSm8syMNAp8dBVpGBkZmaGyMjIfNcbGBgUui8TExOYmJjAxsYGiYmJWLVqFcaNG6cSgCpXrgwAyMzMVFqekZEBHR2dIoxemVQqhXZlnY/q2GxJ09SUooqeLjKyJaxbDbBu1q0OWPfHVbdcnnMOb0kRSiEDFikYaWlpwdLS8oPu8OTJk7CwsEDDhg0Vy6ytrZGZmYnk5GSYmpoqtc89hPb06VOl+3769CnMzMw+aCxEREREbyr1g4DLli3DmjVrlJZdvXoVhoaGqFatmkp7ExMT1K9fH+fPn1csy87ORlRUFFxcXEp8vERERKQ+iv3k63fx8fHBxIkT4ezsjLZt2+L8+fMIDQ3FlClTFMcNk5OTAQCGhoYAgGHDhmHu3LmoW7cu7O3tsWHDBrx+/Rr9+/cv7eETERFRBVbqwcjd3R1ZWVnYuHEjFi5cCAsLC8yYMQNffPGFoo2fnx+AnCtkA8CAAQOQmpqKZcuWITk5GXZ2dtiyZUuhT/QmIiIiKgyJIJTGqUwfp6Sklx/VSWslTVNTCiMjXdatJlg361YHrFu96jY21oWGRsmeBVR+LzRAREREVMwYjIiIiIhEDEZEREREIgYjIiIiIhGDEREREZGIwYiIiIhIxGBEREREJGIwIiIiIhIxGBERERGJGIyIiIiIRAxGRERERCIGIyIiIiIRgxERERGRiMGIiIiISMRgRERERCRiMCIiIiISMRgRERERiRiMiIiIiEQMRkREREQiBiMiIiIiEYMRERERkYjBiIiIiEjEYEREREQkYjAiIiIiEjEYEREREYkYjIiIiIhEDEZEREREIgYjIiIiIhGDEREREZGIwYiIiIhIxGBEREREJGIwIiIiIhIxGBERERGJGIyIiIiIRAxGRERERCIGIyIiIiIRgxERERGRiMGIiIiISMRgRERERCQq0WAUGBiIgIAAleXbt2+Hm5sb7O3t0b17d+zbt6/Afl6/fo0lS5agU6dOcHJyQt++fXHs2LGSGjYRERGpqRIJRnK5HCEhIQgLC1NZFxYWhuDgYPj5+SEyMhLe3t6YMWMGjh49mm9/c+bMwS+//IKgoCAcOHAAXbp0wZgxY3D+/PmSGD4RERGpqWIPRjExMRg0aBD27NkDCwsLlfWpqamYNGkSevbsidq1a2PAgAGwsrLCmTNn8uwvPT0dBw4cwMSJE9G+fXvUrVsXvr6+cHV1fedMExEREVFRFHswOnfuHCwtLXHo0CHUqlVLZb2Pjw8GDx4MAMjKykJkZCRiYmLQunXrPPuTSCRYt24d2rVrpzxwqRQvXrwo7uETERGRGtMs7g49PDwK1S4qKgpeXl6Qy+Xo168fOnfunGe7ypUro02bNkrL/v77b5w7dw7Tp0//oLFqaKjXuee59bJu9cC6Wbc6YN3qVbdEUvL3UaRgFBcXl2+AAYCzZ8/C2Ni4UH3Vr18f4eHhuHbtGubNmwcjIyN8++2379zu3r17+Oabb+Dg4IABAwYUeux50dfX+aDtyyvWrV5Yt3ph3epFXesuSUUKRmZmZoiMjMx3vYGBQaH7MjExgYmJCWxsbJCYmIhVq1Zh3Lhx0NbWzneby5cvw9fXF+bm5li3bh20tLSKMnwVL16kQyaTf1Af5YmGhhT6+jqsW02wbtatDli3etVtYKADqbRkZ8mKFIy0tLRgaWn5QXd48uRJWFhYoGHDhopl1tbWyMzMRHJyMkxNTfPc7siRI5g8eTKaNm2KNWvWQE9P74PGAQAymRzZ2erzhMrFutUL61YvrFu9qFvdglDy91HqByeXLVuGNWvWKC27evUqDA0NUa1atTy3+eOPPzBhwgR06NABoaGhxRKKiIiIiN5W7Cdfv4uPjw8mTpwIZ2dntG3bFufPn0doaCimTJmimB5LTk4GABgaGiIlJQX+/v6wtbXFtGnTkJKSouhLS0sLhoaGpV0CERERVVClHozc3d2RlZWFjRs3YuHChbCwsMCMGTPwxRdfKNr4+fkByLlC9smTJ/HixQtcvXpV5Sf7rq6u2L59e6mOn4iIiCouiSCUxhG7j1NS0ku1OjarqSmFkZEu61YTrJt1qwPWrV51GxvrlvglCtTrAghEREREBWAwIiIiIhIxGBERERGJGIyIiIiIRAxGRERERCIGIyIiIiIRgxERERGRiMGIiIiISMRgRERERCRiMCIiIiISMRgRERERiRiMiIiIiEQMRkREREQiBiMiIiIiEYMRERERkYjBiIiIiEjEYEREREQkYjAiIiIiEjEYEREREYkYjIiIiIhEDEZEREREIgYjIiIiIhGDEREREZGIwYiIiIhIxGBEREREJGIwIiIiIhIxGBERERGJGIyIiIiIRAxGRERERCIGIyIiIiIRgxERERGRiMGIiIiISMRgRERERCRiMCIiIiISMRgRERERiRiMiIiIiEQMRkREREQiBiMiIiIiEYMRERERkYjBiIiIiEhUosEoMDAQAQEBKsu3b98ONzc32Nvbo3v37ti3b1+h+0xMTESbNm2wcuXK4hwqERERUckEI7lcjpCQEISFhamsCwsLQ3BwMPz8/BAZGQlvb2/MmDEDR48eLVTf06dPR0JCQnEPmYiIiKj4g1FMTAwGDRqEPXv2wMLCQmV9amoqJk2ahJ49e6J27doYMGAArKyscObMmXf2HRYWhgcPHqB69erFPWwiIiKi4g9G586dg6WlJQ4dOoRatWqprPfx8cHgwYMBAFlZWYiMjERMTAxat25dYL/3799HcHAwFi9eDG1t7eIeNhERERE0i7tDDw+PQrWLioqCl5cX5HI5+vXrh86dO+fbNisrC5MmTcLw4cNha2tbXEOFhoZ6nXueWy/rVg+sm3WrA9atXnVLJCV/H0UKRnFxcQUGmLNnz8LY2LhQfdWvXx/h4eG4du0a5s2bByMjI3z77bd5tl2xYgUqVaqEESNGFGW476Svr1Os/ZUXrFu9sG71wrrVi7rWXZKKFIzMzMwQGRmZ73oDA4NC92ViYgITExPY2NggMTERq1atwrhx41QOk124cAG7d+9GeHg4NDQ0ijLcd3rxIh0ymbxY+/yYaWhIoa+vw7rVBOtm3eqAdatX3QYGOpBKS3aWrEjBSEtLC5aWlh90hydPnoSFhQUaNmyoWGZtbY3MzEwkJyfD1NRUqX14eDhevXqFXr16KZalp6dj/fr1OHz4MCIiIt57LDKZHNnZ6vOEysW61QvrVi+sW72oW92CUPL3UeznGL3LsmXLUK9ePYSEhCiWXb16FYaGhqhWrZpK+8mTJ+Prr79WWubl5QU3NzcMHTq0xMdLRERE6qPUg5GPjw8mTpwIZ2dntG3bFufPn0doaCimTJmimB5LTk4GABgaGioOuSkNWlMTBgYGqFmzZmkPn4iIiCqwUg9G7u7uyMrKwsaNG7Fw4UJYWFhgxowZ+OKLLxRt/Pz8AORcIZuIiIiotEgEoTSO2H2ckpJeqtWxWU1NKYyMdFm3mmDdrFsdsG71qtvYWLfEL1GgXhdAICIiIioAgxERERGRiMGIiIiISMRgRERERCRiMCIiIiISMRgRERERiRiMiIiIiEQMRkREREQiBiMiIiIiEYMRERERkYjBiIiIiEjEYEREREQkYjAiIiIiEjEYEREREYkYjIiIiIhEDEZEREREIgYjIiIiIhGDEREREZGIwYiIiIhIxGBEREREJGIwIiIiIhIxGBERERGJGIyIiIiIRAxGRERERCIGIyIiIiIRgxERERGRiMGIiIiISMRgRERERCRiMCIiIiISMRgRERERiRiMiIiIiEQMRkREREQiBiMiIiIiEYMRERERkYjBiIiIiEjEYEREREQkYjAiIiIiEjEYEREREYkYjIiIiIhEDEZEREREohINRoGBgQgICFBZvn37dri5ucHe3h7du3fHvn373tnXiRMn0LdvX9jb26NLly7YuXNnSQyZiIiI1FiJBCO5XI6QkBCEhYWprAsLC0NwcDD8/PwQGRkJb29vzJgxA0ePHs23vwsXLmD06NHo0KEDIiIiMGrUKMydOxeRkZElMXwiIiJSU5rF3WFMTAymTZuGhw8fwsLCQmV9amoqJk2ahJ49ewIAateujV27duHMmTPo0qVLnn2uXLkSXbp0wdixYwEAderUwZUrVxAVFQV3d/fiLoGIiIjUVLEHo3PnzsHS0hKrV6/G+PHjVdb7+Pgo/p+VlYXff/8dMTExGDNmTJ79paenIyoqCitWrFBaPm/evGIdNxEREVGxByMPD49CtYuKioKXlxfkcjn69euHzp0759nu4cOHkMvl0NDQwNixY3Hx4kWYmprC09MTX3zxxQeNVUNDvc49z62XdasH1s261QHrVq+6JZKSv48iBaO4uLh8AwwAnD17FsbGxoXqq379+ggPD8e1a9cwb948GBkZ4dtvv1Vpl5aWBiDnRO6RI0di9OjROH/+PGbNmgUAHxSO9PV13nvb8ox1qxfWrV5Yt3pR17pLUpGCkZmZWYEnPBsYGBS6LxMTE5iYmMDGxgaJiYlYtWoVxo0bB21tbaV2WlpaAIDevXtj8ODBAIDGjRvj4cOH2Lp16wcFoxcv0iGTyd97+/JGQ0MKfX0d1q0mWDfrVgesW73qNjDQgVRasrNkRQpGWlpasLS0/KA7PHnyJCwsLNCwYUPFMmtra2RmZiI5ORmmpqZK7c3NzQEAVlZWSssbNmyI/fv3f9BYZDI5srPV5wmVi3WrF9atXli3elG3ugWh5O+j1A9OLlu2DGvWrFFadvXqVRgaGqJatWoq7c3MzFCnTh1cvXpVafmdO3dQp06dEh0rERERqZdSD0Y+Pj6IjIzEjh078PDhQ/z0008IDQ2Fn5+fYnosOTkZycnJim3GjBmDsLAw7Ny5E7Gxsfjxxx+xb98+DB8+vLSHT0RERBVYsf8q7V3c3d2RlZWFjRs3YuHChbCwsMCMGTOUzhXy8/MDkHOFbCDn/CIAWL9+PebPn4+aNWsiKCgIffr0Ke3hExERUQUmEYTSOGL3cUpKeqlWx2Y1NaUwMtJl3WqCdbNudcC61atuY2PdEr9EgXpdAIGIiIioAAxGRERERCIGIyIiIiIRgxERERGRiMGIiIiISMRgRERERCRiMCIiIiISMRgRERERiRiMiIiIiEQMRkREREQiBiMiIiIiEYMRERERkYjBiIiIiEjEYEREREQkYjAiIiIiEjEYEREREYkYjIiIiIhEDEZEREREIgYjIiIiIhGDEREREZGIwYiIiIhIJBEEQSjrQZQVmUxe1kModRoaUtatRli3emHd6kUd65ZKJZBIJCV6H2odjIiIiIjexENpRERERCIGIyIiIiIRgxERERGRiMGIiIiISMRgRERERCRiMCIiIiISMRgRERERiRiMiIiIiEQMRkREREQiBiMiIiIiEYMRERERkYjBiIiIiEjEYEREREQkqpDBSC6XY8WKFWjbti0cHR0xYsQIxMbG5ts+KSkJkyZNgouLC1xdXTFr1iykp6eX4oiLR3JyMgIDA9GuXTs4Ozvjq6++QlRUVL7t165dC2tra5V/5U18fHyedezfvz/P9hVhf58/fz7Pmq2trdG5c+c8t7l06VKe7c+fP1/Ko39/69evh5eXl9KymzdvwtPTE46OjujUqRO2bdv2zn5+/fVXuLu7w8HBAX369MHZs2dLasjFIq+6//jjD/Tr1w9OTk7o1KkTFi5ciNevX+fbh0wmg4ODg8r+X7lyZUkP/73lVff06dNVaujUqVOB/ZT3/e3l5ZXv6/3AgQP59jN06FCV9m8/nmXtXZ9bZ8+eRd++fdG0aVN069YNERER7+xz586d6Ny5MxwcHDBo0CDcuHGjaIMSKqCVK1cKLVq0EI4fPy7cvHlTGDZsmODm5iZkZGTk2d7T01Po16+fcP36deGvv/4SOnbsKEyZMqWUR/3hhg4dKvTo0UO4ePGicO/ePWHWrFmCg4ODEBMTk2f7cePGCd9++63w9OlTpX/lzZ9//inY29sL8fHxSnWkp6fn2b4i7O+MjAyV/XbkyBHB2tpa2Lt3b57b7Ny5U+jSpYvKdvm9Lj42O3bsEGxsbARPT0/FssTERKFFixbC1KlThejoaGHv3r2Cvb19vo+BIAjC2bNnBVtbW+GHH34QoqOjhQULFgh2dnZCdHR0aZRRZHnVffHiRaFx48bC2rVrhfv37wt//vmn0K5dOyEgICDffqKjowUrKyvh5s2bSvs/LS2tNMoosrzqFgRB6N+/vxASEqJUw/Pnz/PtpyLs76SkJKV64+PjhUGDBgndu3cvcP+1bNlS2LVrl9K2SUlJpVBF4RX0uRUdHS3Y29sLISEhQnR0tLBp0yahSZMmwl9//ZVvf/v37xccHByEn3/+Wbh7967w7bffCq6urgU+R95W4YJRRkaG4OTkJOzcuVOxLCUlRXBwcBB++eUXlfaXL18WrKyslF4kp06dEqytrYUnT56UypiLw4MHDwQrKyshKipKsUwulwtdunQRli1bluc2n332mbBly5ZSGmHJ2bBhg9CzZ89Cta0o+/ttL1++FDp27FjgB2NQUJDw9ddfl+KoiseTJ0+EUaNGCY6OjkK3bt2UPjDWrVsntGnTRsjKylIsW7JkieDm5pZvf8OGDRPGjRuntGzgwIHCjBkzin3sH6KguidNmiQMGTJEqX14eLhga2ubb9CNiIgQnJ2dS3TMxaGguuVyueDo6CgcOXKk0P1VhP39tu3btwt2dnb5fukVBEF49uyZYGVlJfzzzz8lMdxi8a7PrRkzZgj9+/dX2mbixInCsGHD8u3Tzc1NWLRokeJ2VlaW0L59e2HdunWFHleFO5R269YtvHz5Ei1btlQs09fXR5MmTXDx4kWV9lFRUahevTosLS0Vy1xdXSGRSHDp0qVSGXNxMDIywoYNG2Bvb69YJpFIIJFI8OLFC5X2mZmZePDgARo0aFCawywRt2/fVtp/Bako+/tt69atQ3p6Ovz9/fNtU5TH6WPyzz//QEtLCwcPHkTTpk2V1kVFRcHV1RWampqKZZ988gkePHiAZ8+eqfQll8tx+fJlpfcHAGjRokWe7w9lqaC6hw0bprKvpVIpsrKykJaWlmd/5WX/F1T3v//+i1evXhX6faui7O83JSYmYtmyZRg9enSBj8Pt27chkUhQv379khhusXjX51ZUVJTKvvvkk09w6dIlCIKg0t/z58/x4MEDpW00NTXRvHnzIu1vzXc3KV+ePHkCAKhRo4bSclNTU8W6N8XHx6u01dbWhqGhIR4/flxyAy1m+vr6aN++vdKy3377DQ8fPsR3332n0j46OhoymQy//fYb5s6di4yMDLi4uODbb7+FqalpaQ27WNy5cwdGRkbw8PDA/fv3UbduXYwePRrt2rVTaVtR9vebEhMTsXXrVkyaNAmGhob5trt79y6MjIzQt29fxMfHw8rKChMmTICDg0PpDfY9dOrUKd9zSJ48eQIrKyulZbnP38ePH6NatWpK6168eIFXr17B3NxcZZu83h/KUkF1N2nSROl2VlYWtm7dCjs7OxgbG+e5zZ07d5CdnY3hw4fj1q1bMDMzg7e3N3r37l3sY/8QBdV9584dAMD27dtx8uRJSKVStGvXDhMmTICenp5K+4qyv9+0ceNGVK5cGcOHDy+w3Z07d6Cnp4fvv/8eZ86cQZUqVdCtWzf4+vpCW1u7uIb9Qd71uRUeHp7nvktPT0dSUpLKc72gz/9bt24VelwVbsYo9yTat3d8pUqVkJGRkWf7vJ4k+bUvLy5fvoypU6fCzc0NHTp0UFmf+wajo6OD5cuXY+7cubh37x4GDx5c4AmcH5vs7Gzcu3cPKSkp8PPzw4YNG+Do6IiRI0fmeYJlRdzfu3btgp6eHgYOHJhvm8ePHyM1NRWvXr3C9OnTsWbNGlSrVg2enp6Ijo4uxdEWr9evX+f5WgeQ5/7MfW4X9v2hPMjOzsaUKVNw9+5dBAUF5dvu7t27SE5OhpeXF0JDQ9G1a1dMnToVe/fuLcXRfpg7d+5AKpXC1NQU69atQ0BAAE6fPg1fX1/I5XKV9hVtf6elpeGnn37C8OHDFc/z/Ny5cwcZGRlwcHDApk2bMHr0aOzZswfTp08vpdEW3dufW3m9vnNvZ2Zmqmxf1M///FS4GaPKlSsDyHnQcv8P5LxJ6ujo5Nk+rwc4IyMDVapUKbmBlqCjR49i8uTJcHZ2RnBwcJ5t+vTpg3bt2ikl7kaNGqFdu3b4448/4O7uXlrD/SCampo4f/48NDQ0FPvbzs4Od+/eRWhoqMo0bEXc3wcOHECfPn2Unu9vq1GjBi5evAgdHR1oaWkBAOzt7XHjxg1s374ds2bNKq3hFqu89mfuG2Be+zP3wySvbfJ6f/jYpaWlYfz48bhw4QJWrVpV4OzfoUOHIJPJoKurCwCwsbHBf//9h9DQUPTv37+0hvxBRo8ejUGDBsHIyAgAYGVlherVq2PAgAG4du2ayiGoira/jx49iszMTPTr1++dbb///nv4+/vDwMAAQM5jpaWlhQkTJmDKlCkqs6llLa/PrUqVKqnsu9zb+X2ev9kmV1H3d4WbMcqdQnv69KnS8qdPn8LMzEylvbm5uUrbzMxMJCcnl7tDSgCwY8cO+Pn5oWPHjli3bl2B3yrenoY0NTWFoaHhRzfF/C66uroqoaBRo0aIj49XaVvR9vetW7cQGxuLnj17vrOtvr6+IhQBOeekWFpa5vk4lRd57c/c23m93g0NDVGlSpVCvz98zJ4+fQoPDw/873//Q2hoqMohibdVrlxZEYpyWVlZlavXu1QqVYSiXI0aNQKAPOuoSPsbyAkP7du3h76+/jvbampqKkJRroIeq7KU3+dWjRo18tx3VapUyfPQaVE///NT4YKRjY0NqlatqnRtlhcvXuDGjRtwcXFRae/i4oInT57g4cOHimUXLlwAADRr1qzkB1yMdu3ahdmzZ8PDwwMhISEFHkdeunQpunbtqnQCW1xcHJKSktCwYcPSGG6xuHv3LpydnVWuxXP9+vU866hI+xvIOfnYxMQENjY2BbY7efIknJyclK7nlZ2djVu3bpWr/f02FxcXXLp0CTKZTLHs3LlzqF+/PkxMTFTaSyQSODs7K/Z5rvPnz6N58+YlPt7ikpKSAm9vbyQmJmLnzp15vre96cWLF3B1dVW5tte1a9cUH5blwZQpUzBkyBClZdeuXQOAPJ/HFWV/58rrZOT8eHl5YerUqUrLrl27Bi0tLdSrV68ERvd+Cvrcat68ucq+O3fuHJydnSGVqsYXExMT1K9fX+nzIDs7G1FRUe98jbypwgUjbW1teHp6Ijg4GMeOHcOtW7cwYcIEmJubw83NDTKZDAkJCYpjz02bNoWzszMmTJiAv//+G+fOnUNgYCD69OlTrr5R3L9/H/PmzcOnn36KUaNG4dmzZ0hISEBCQgJSU1ORmZmJhIQExRTjp59+ikePHmHmzJm4f/8+Ll68CD8/Pzg7O6Nt27ZlXE3hWVpaokGDBvj+++8RFRWFmJgYzJ8/H//73/8wevToCru/c924cSPfi3ImJCTg5cuXAABnZ2cYGRnB398f169fx+3bt+Hv74/k5GSVD5rypF+/fkhLS8O0adMQHR2N/fv3Y+vWrRg1apSiTWpqKhITExW3hw4dioiICGzZsgUxMTFYtGgRbt68CW9v77Io4b3Mnz8fsbGxWLx4MYyNjRWv9YSEBEVITE5ORnJyMoCc2cJPPvkES5cuxYkTJ/DgwQNs2LABBw8ehJ+fXxlWUjRdu3bF2bNnsWrVKvz77784ceIEvvvuO/To0UPxi7uKuL+BnPMEk5KS8v0S9PLlSyQkJChud+3aFT///DN2796N2NhYREZGYtGiRRg+fDiqVq1aWsMu0Ls+t7y8vPD3338jODgYMTEx2Lx5Mw4fPgwfHx9FH28+z4GcX2xu2bIF4eHhiI6OxnfffYfXr18X7XBxoX/YX45kZ2cLixYtEj755BPB0dFRGDFihBAbGysIgiDExsYKVlZWwr59+xTtnz17Jvj5+QmOjo5CixYthKCgIOH169dlNfz3snbtWsHKyirPf/7+/sK5c+cEKysr4dy5c4pt/vrrL2HgwIGCo6Oj4OrqKkydOlVITk4uwyreT0JCghAQECC0bt1asLe3FwYOHChcvHhREISKu79z+fj4COPHj89znZWVlbBixQrF7YcPHwp+fn6Cq6ur0LRpU2HYsGHC7du3S2uoxcLf31/l+i5Xr14VBgwYINjZ2QkdO3YUtm/frrJNx44dlZaFh4cLn376qWBvby98/vnnBV4w7mPwZt3Z2dmCvb19vq/33Pc6T09PpccqNTVVmDdvntC+fXvBzs5O6N27t/D777+XST2Fldf+joyMFPr06SM4ODgIrVu3FhYsWKD0+q1o+zvX1atXVa7B9qYVK1YIVlZWSst27NghfPbZZ4rXxtq1awWZTFZi4y6qd31uCYIgnDhxQujRo4dgZ2cndOvWTYiIiFDq4+3nuSAIwqZNm4R27doJDg4OwqBBg4QbN24UaVwSQcjjYgBEREREaqjCHUojIiIiel8MRkREREQiBiMiIiIiEYMRERERkYjBiIiIiEjEYEREREQkYjAiIiIiEjEYEREREYkYjIioXIqLi4O1tbXK3/96HwEBAejUqVMxjIqIyjvNsh4AEdH7MDU1RVhYGOrUqVPWQyGiCoTBiIjKJW1tbTg6Opb1MIioguGhNCIqEXv27EH37t1hZ2eHDh06YOXKlYq//B4QEAAvLy/s3bsXHTt2hJOTE7y9vXHr1i3F9nK5HEuXLkWnTp1gZ2eHTp06YcmSJcjKygKQ96G0Bw8eYOzYsWjdujUcHR3h5eWFS5cuKY0rJSUFU6dOhaurK1xcXLB48WLI5XKV8R89ehR9+/aFvb09WrdujTlz5uDVq1eK9a9fv8bMmTPRrl072NnZoVu3bggNDS3Wx5CISh9njIio2K1fvx5Lly6Fp6cnpk6dips3b2LlypV4/Pgx5s2bBwC4efMm7t27h4kTJ8LAwAArVqyAp6cnIiMjYWpqio0bN2L37t3w9/dH7dq1cfXqVSxduhRaWloYO3asyn1GR0djwIABqFevHqZPnw4tLS1s27YN3t7e2Lx5M1xdXSGXy+Hj44NHjx7B398fhoaG2LRpE65duwZTU1NFX7/88gsmT56Mnj17Yvz48Xj06BGWLl2K6OhobNmyBRKJBPPmzcPp06fh7++PatWq4eTJk1i0aBEMDQ3Rr1+/Unusiah4MRgRUbFKTU3FmjVrMHDgQEyfPh0A0KZNGxgaGmL69OkYOnSoot26devQvHlzAICDgwO6dOmCbdu2YfLkybhw4QLs7OwUIcPV1RU6OjrQ09PL835XrVoFbW1tbNu2DVWrVgUAdOjQAT169MCiRYuwd+9enDx5En///Tc2btyIdu3aAQBatmypdOK1IAgIDg5G27ZtERwcrFher149DBkyBCdOnECHDh1w4cIFtG7dGt27dwcAtGjRAlWqVIGJiUlxPpxEVMp4KI2IitWVK1fw+vVrdOrUCdnZ2Yp/ueHjzJkzAIBatWopQhGQczK1k5MTLl68CCAnaJw5cwaDBg3Cpk2bEB0dDU9PT/Tu3TvP+71w4QI6duyoCEUAoKmpie7du+P69et4+fIloqKioKWlhbZt2yraVKlSBe3bt1fcvnfvHp48eaIyfhcXF1StWlUx/hYtWuCnn37CiBEjsGPHDsTGxuKbb75Bhw4diueBJKIywRkjIipWycnJAICRI0fmuf7p06cAADMzM5V1JiYm+OeffwAAPj4+0NXVxb59+xAcHIzFixejUaNGmD59Oj755BOVbVNSUlCtWjWV5dWqVYMgCEhLS0NKSgoMDQ0hkUiU2lSvXl1l/LNmzcKsWbPyHf+0adNgbm6OgwcPYvbs2Zg9ezacnJwwc+ZM2NjY5Fk7EX38GIyIqFjp6+sDAIKDg1GvXj2V9dWqVcPy5cuRlJSksu7Zs2eKQ1FSqRQeHh7w8PDA8+fPceLECaxbtw5+fn6KWZs3GRgY4NmzZyrLExISAABGRkYwMjJCUlISZDIZNDQ0FG1yw9Cb458yZQpcXV3zvB8g51dxo0ePxujRo/Hff//h+PHjWLNmDSZNmoSIiIj8Hh4i+sjxUBoRFaumTZtCS0sL8fHxsLe3V/zT1NRESEgI4uLiAOT8giwmJkaxXXx8PK5cuYKWLVsCAL788kvMmTMHQM5MUt++feHh4YEXL14gLS1N5X5dXFxw/PhxpXUymQwRERGwt7eHtrY2WrZsiezsbBw9elTRJjMzUyloNWjQACYmJoiLi1Mav5mZGZYsWYIbN27g9evX6Nq1KzZv3gwAsLCwgIeHB7p3747//vuvGB9NIiptnDEiomJlZGQEHx8fLF++HGlpaWjRogXi4+OxfPlySCQSxWEmQRDw9ddfY8KECdDQ0MCqVatgYGAALy8vADlBZ/PmzahWrRqcnJwQHx+PLVu2wNXVFcbGxko/nQeAMWPG4OTJkxg8eDBGjhwJLS0txbk/mzZtApBzonWbNm0wffp0PH/+HDVr1sS2bduQmJiomKnS0NDAhAkTEBgYCA0NDXTs2BEvXrzAmjVrEB8fD1tbW1SuXBm2trZYtWoVtLS0YG1tjfv37yM8PBxdu3YtxUebiIqbRBAEoawHQUQVz86dO7Fr1y48fPgQBgYGaNmyJSZOnAgLCwsEBATgwoULGDFiBFavXo309HS0atUK/v7+qFWrFgAgOzsba9euxcGDB/HkyRPo6emhU6dOmDRpEoyMjBAXF4fOnTtj/vz56Nu3L4CcSwCEhIQgKioKEokEDg4OGDNmjNJJ3unp6QgODkZERAQyMjLg7u6OKlWq4NixY/jjjz8U7SIjI7Fp0ybcvXsXVapUgbOzM8aPHw9ra2sAQFpaGpYtW4Zjx44hISEBJiYmcHd3x7hx41C5cuVSfKSJqDgxGBFRqcsNRm8GESKijwHPMSIiIiISMRgRERERiXgojYiIiEjEGSMiIiIiEYMRERERkYjBiIiIiEjEYEREREQkYjAiIiIiEjEYEREREYkYjIiIiIhEDEZEREREov8D+ezTcT/6oFEAAAAASUVORK5CYII=", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "# 获取参数\n", - "cfg = Config() \n", - "# 训练\n", - "env, agent = env_agent_config(cfg)\n", - "res_dic = train(cfg, env, agent)\n", - " \n", - "plot_rewards(res_dic['rewards'], title=f\"training curve on {cfg.device} of {cfg.algo_name} for {cfg.env_name}\") \n", - "# 测试\n", - "res_dic = test(cfg, env, agent)\n", - "plot_rewards(res_dic['rewards'], title=f\"testing curve on {cfg.device} of {cfg.algo_name} for {cfg.env_name}\") # 画出结果" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.12 ('easyrl')", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.12" - }, - "orig_nbformat": 4, - "vscode": { - "interpreter": { - "hash": "f5a9629e9f3b9957bf68a43815f911e93447d47b3d065b6a8a04975e44c504d9" - } - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/projects/notebooks/Value Iteration/README.md b/projects/notebooks/Value Iteration/README.md deleted file mode 100644 index e69de29..0000000 diff --git a/projects/notebooks/common/multiprocessing_env.py b/projects/notebooks/common/multiprocessing_env.py deleted file mode 100644 index 28c8aba..0000000 --- a/projects/notebooks/common/multiprocessing_env.py +++ /dev/null @@ -1,153 +0,0 @@ -# 该代码来自 openai baseline，用于多线程环境 -# https://github.com/openai/baselines/tree/master/baselines/common/vec_env - -import numpy as np -from multiprocessing import Process, Pipe - -def worker(remote, parent_remote, env_fn_wrapper): - parent_remote.close() - env = env_fn_wrapper.x() - while True: - cmd, data = remote.recv() - if cmd == 'step': - ob, reward, done, info = env.step(data) - if done: - ob = env.reset() - remote.send((ob, reward, done, info)) - elif cmd == 'reset': - ob = env.reset() - remote.send(ob) - elif cmd == 'reset_task': - ob = env.reset_task() - remote.send(ob) - elif cmd == 'close': - remote.close() - break - elif cmd == 'get_spaces': - remote.send((env.observation_space, env.action_space)) - else: - raise NotImplementedError - -class VecEnv(object): - """ - An abstract asynchronous, vectorized environment. - """ - def __init__(self, num_envs, observation_space, action_space): - self.num_envs = num_envs - self.observation_space = observation_space - self.action_space = action_space - - def reset(self): - """ - Reset all the environments and return an array of - observations, or a tuple of observation arrays. - If step_async is still doing work, that work will - be cancelled and step_wait() should not be called - until step_async() is invoked again. - """ - pass - - def step_async(self, actions): - """ - Tell all the environments to start taking a step - with the given actions. - Call step_wait() to get the results of the step. - You should not call this if a step_async run is - already pending. - """ - pass - - def step_wait(self): - """ - Wait for the step taken with step_async(). - Returns (obs, rews, dones, infos): - - obs: an array of observations, or a tuple of - arrays of observations. - - rews: an array of rewards - - dones: an array of "episode done" booleans - - infos: a sequence of info objects - """ - pass - - def close(self): - """ - Clean up the environments' resources. - """ - pass - - def step(self, actions): - self.step_async(actions) - return self.step_wait() - - -class CloudpickleWrapper(object): - """ - Uses cloudpickle to serialize contents (otherwise multiprocessing tries to use pickle) - """ - def __init__(self, x): - self.x = x - def __getstate__(self): - import cloudpickle - return cloudpickle.dumps(self.x) - def __setstate__(self, ob): - import pickle - self.x = pickle.loads(ob) - - -class SubprocVecEnv(VecEnv): - def __init__(self, env_fns, spaces=None): - """ - envs: list of gym environments to run in subprocesses - """ - self.waiting = False - self.closed = False - nenvs = len(env_fns) - self.nenvs = nenvs - self.remotes, self.work_remotes = zip(*[Pipe() for _ in range(nenvs)]) - self.ps = [Process(target=worker, args=(work_remote, remote, CloudpickleWrapper(env_fn))) - for (work_remote, remote, env_fn) in zip(self.work_remotes, self.remotes, env_fns)] - for p in self.ps: - p.daemon = True # if the main process crashes, we should not cause things to hang - p.start() - for remote in self.work_remotes: - remote.close() - - self.remotes[0].send(('get_spaces', None)) - observation_space, action_space = self.remotes[0].recv() - VecEnv.__init__(self, len(env_fns), observation_space, action_space) - - def step_async(self, actions): - for remote, action in zip(self.remotes, actions): - remote.send(('step', action)) - self.waiting = True - - def step_wait(self): - results = [remote.recv() for remote in self.remotes] - self.waiting = False - obs, rews, dones, infos = zip(*results) - return np.stack(obs), np.stack(rews), np.stack(dones), infos - - def reset(self): - for remote in self.remotes: - remote.send(('reset', None)) - return np.stack([remote.recv() for remote in self.remotes]) - - def reset_task(self): - for remote in self.remotes: - remote.send(('reset_task', None)) - return np.stack([remote.recv() for remote in self.remotes]) - - def close(self): - if self.closed: - return - if self.waiting: - for remote in self.remotes: - remote.recv() - for remote in self.remotes: - remote.send(('close', None)) - for p in self.ps: - p.join() - self.closed = True - - def __len__(self): - return self.nenvs \ No newline at end of file diff --git a/projects/parl_tutorials/DDPG.ipynb b/projects/parl_tutorials/DDPG.ipynb deleted file mode 100644 index a0db09f..0000000 --- a/projects/parl_tutorials/DDPG.ipynb +++ /dev/null @@ -1,465 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1. 定义算法" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[32m[09-28 00:38:01 MainThread @utils.py:73]\u001b[0m paddlepaddle version: 2.3.2.\n" - ] - } - ], - "source": [ - "import parl\n", - "import paddle\n", - "import paddle.nn as nn\n", - "import paddle.nn.functional as F\n", - "class Actor(parl.Model):\n", - " def __init__(self, n_states, n_actions):\n", - " super(Actor, self).__init__()\n", - "\n", - " self.l1 = nn.Linear(n_states, 400)\n", - " self.l2 = nn.Linear(400, 300)\n", - " self.l3 = nn.Linear(300, n_actions)\n", - "\n", - " def forward(self, state):\n", - " x = F.relu(self.l1(state))\n", - " x = F.relu(self.l2(x))\n", - " return paddle.tanh(self.l3(x))\n", - "\n", - "class Critic(parl.Model):\n", - " def __init__(self, n_states, n_actions):\n", - " super(Critic, self).__init__()\n", - "\n", - " self.l1 = nn.Linear(n_states, 400)\n", - " self.l2 = nn.Linear(400 + n_actions, 300)\n", - " self.l3 = nn.Linear(300, 1)\n", - "\n", - " def forward(self, state, action):\n", - " x = F.relu(self.l1(state))\n", - " x = F.relu(self.l2(paddle.concat([x, action], 1)))\n", - " return self.l3(x)\n", - "class ActorCritic(parl.Model):\n", - " def __init__(self, n_states, n_actions):\n", - " super(ActorCritic, self).__init__()\n", - " self.actor_model = Actor(n_states, n_actions)\n", - " self.critic_model = Critic(n_states, n_actions)\n", - "\n", - " def policy(self, state):\n", - " return self.actor_model(state)\n", - "\n", - " def value(self, state, action):\n", - " return self.critic_model(state, action)\n", - "\n", - " def get_actor_params(self):\n", - " return self.actor_model.parameters()\n", - "\n", - " def get_critic_params(self):\n", - " return self.critic_model.parameters()" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [], - "source": [ - "from collections import deque\n", - "import random\n", - "class ReplayBuffer:\n", - " def __init__(self, capacity: int) -> None:\n", - " self.capacity = capacity\n", - " self.buffer = deque(maxlen=self.capacity)\n", - " def push(self,transitions):\n", - " '''_summary_\n", - " Args:\n", - " trainsitions (tuple): _description_\n", - " '''\n", - " self.buffer.append(transitions)\n", - " def sample(self, batch_size: int, sequential: bool = False):\n", - " if batch_size > len(self.buffer):\n", - " batch_size = len(self.buffer)\n", - " if sequential: # sequential sampling\n", - " rand = random.randint(0, len(self.buffer) - batch_size)\n", - " batch = [self.buffer[i] for i in range(rand, rand + batch_size)]\n", - " return zip(*batch)\n", - " else:\n", - " batch = random.sample(self.buffer, batch_size)\n", - " return zip(*batch)\n", - " def clear(self):\n", - " self.buffer.clear()\n", - " def __len__(self):\n", - " return len(self.buffer)" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "import parl\n", - "import paddle\n", - "import numpy as np\n", - "\n", - "\n", - "class DDPGAgent(parl.Agent):\n", - " def __init__(self, algorithm,memory,cfg):\n", - " super(DDPGAgent, self).__init__(algorithm)\n", - " self.n_actions = cfg['n_actions']\n", - " self.expl_noise = cfg['expl_noise']\n", - " self.batch_size = cfg['batch_size'] \n", - " self.memory = memory\n", - " self.alg.sync_target(decay=0)\n", - "\n", - " def sample_action(self, state):\n", - " action_numpy = self.predict_action(state)\n", - " action_noise = np.random.normal(0, self.expl_noise, size=self.n_actions)\n", - " action = (action_numpy + action_noise).clip(-1, 1)\n", - " return action\n", - "\n", - " def predict_action(self, state):\n", - " state = paddle.to_tensor(state.reshape(1, -1), dtype='float32')\n", - " action = self.alg.predict(state)\n", - " action_numpy = action.cpu().numpy()[0]\n", - " return action_numpy\n", - "\n", - " def update(self):\n", - " if len(self.memory) < self.batch_size: \n", - " return\n", - " state_batch, action_batch, reward_batch, next_state_batch, done_batch = self.memory.sample(\n", - " self.batch_size)\n", - " done_batch = np.expand_dims(done_batch , -1)\n", - " reward_batch = np.expand_dims(reward_batch, -1)\n", - " state_batch = paddle.to_tensor(state_batch, dtype='float32')\n", - " action_batch = paddle.to_tensor(action_batch, dtype='float32')\n", - " reward_batch = paddle.to_tensor(reward_batch, dtype='float32')\n", - " next_state_batch = paddle.to_tensor(next_state_batch, dtype='float32')\n", - " done_batch = paddle.to_tensor(done_batch, dtype='float32')\n", - " critic_loss, actor_loss = self.alg.learn(state_batch, action_batch, reward_batch, next_state_batch,\n", - " done_batch)" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "def train(cfg, env, agent):\n", - " ''' 训练\n", - " '''\n", - " print(f\"开始训练！\")\n", - " rewards = [] # 记录所有回合的奖励\n", - " for i_ep in range(cfg[\"train_eps\"]):\n", - " ep_reward = 0 \n", - " state = env.reset() \n", - " for i_step in range(cfg['max_steps']):\n", - " action = agent.sample_action(state) # 采样动作\n", - " next_state, reward, done, _ = env.step(action) \n", - " agent.memory.push((state, action, reward,next_state, done)) \n", - " state = next_state \n", - " agent.update() \n", - " ep_reward += reward \n", - " if done:\n", - " break\n", - " rewards.append(ep_reward)\n", - " if (i_ep + 1) % 10 == 0:\n", - " print(f\"回合：{i_ep+1}/{cfg['train_eps']}，奖励：{ep_reward:.2f}\")\n", - " print(\"完成训练！\")\n", - " env.close()\n", - " res_dic = {'episodes':range(len(rewards)),'rewards':rewards}\n", - " return res_dic\n", - "\n", - "def test(cfg, env, agent):\n", - " print(\"开始测试！\")\n", - " rewards = [] # 记录所有回合的奖励\n", - " for i_ep in range(cfg['test_eps']):\n", - " ep_reward = 0 \n", - " state = env.reset() \n", - " for i_step in range(cfg['max_steps']):\n", - " action = agent.predict_action(state) \n", - " next_state, reward, done, _ = env.step(action) \n", - " state = next_state \n", - " ep_reward += reward \n", - " if done:\n", - " break\n", - " rewards.append(ep_reward)\n", - " print(f\"回合：{i_ep+1}/{cfg['test_eps']}，奖励：{ep_reward:.2f}\")\n", - " print(\"完成测试！\")\n", - " env.close()\n", - " return {'episodes':range(len(rewards)),'rewards':rewards}\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "import gym\n", - "import os\n", - "import paddle\n", - "import numpy as np\n", - "import random\n", - "from parl.algorithms import DDPG\n", - "class NormalizedActions(gym.ActionWrapper):\n", - " ''' 将action范围重定在[0.1]之间\n", - " '''\n", - " def action(self, action):\n", - " low_bound = self.action_space.low\n", - " upper_bound = self.action_space.high\n", - " action = low_bound + (action + 1.0) * 0.5 * (upper_bound - low_bound)\n", - " action = np.clip(action, low_bound, upper_bound)\n", - " return action\n", - "\n", - " def reverse_action(self, action):\n", - " low_bound = self.action_space.low\n", - " upper_bound = self.action_space.high\n", - " action = 2 * (action - low_bound) / (upper_bound - low_bound) - 1\n", - " action = np.clip(action, low_bound, upper_bound)\n", - " return action\n", - "def all_seed(env,seed = 1):\n", - " ''' 万能的seed函数\n", - " '''\n", - " env.seed(seed) # env config\n", - " np.random.seed(seed)\n", - " random.seed(seed)\n", - " paddle.seed(seed)\n", - "def env_agent_config(cfg):\n", - " env = NormalizedActions(gym.make(cfg['env_name'])) # 装饰action噪声\n", - " if cfg['seed'] !=0:\n", - " all_seed(env,seed=cfg['seed'])\n", - " n_states = env.observation_space.shape[0]\n", - " n_actions = env.action_space.shape[0]\n", - " print(f\"状态维度：{n_states}，动作维度：{n_actions}\")\n", - " cfg.update({\"n_states\":n_states,\"n_actions\":n_actions}) # 更新n_states和n_actions到cfg参数中\n", - " memory = ReplayBuffer(cfg['memory_capacity'])\n", - " model = ActorCritic(n_states, n_actions)\n", - " algorithm = DDPG(model, gamma=cfg['gamma'], tau=cfg['tau'], actor_lr=cfg['actor_lr'], critic_lr=cfg['critic_lr'])\n", - " agent = DDPGAgent(algorithm,memory,cfg)\n", - " return env,agent" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [], - "source": [ - "\n", - "import argparse\n", - "import matplotlib.pyplot as plt\n", - "import seaborn as sns\n", - "def get_args():\n", - " \"\"\" 超参数\n", - " \"\"\"\n", - " parser = argparse.ArgumentParser(description=\"hyperparameters\") \n", - " parser.add_argument('--algo_name',default='DDPG',type=str,help=\"name of algorithm\")\n", - " parser.add_argument('--env_name',default='Pendulum-v0',type=str,help=\"name of environment\")\n", - " parser.add_argument('--train_eps',default=200,type=int,help=\"episodes of training\")\n", - " parser.add_argument('--test_eps',default=20,type=int,help=\"episodes of testing\")\n", - " parser.add_argument('--max_steps',default=100000,type=int,help=\"steps per episode, much larger value can simulate infinite steps\")\n", - " parser.add_argument('--gamma',default=0.99,type=float,help=\"discounted factor\")\n", - " parser.add_argument('--critic_lr',default=1e-3,type=float,help=\"learning rate of critic\")\n", - " parser.add_argument('--actor_lr',default=1e-4,type=float,help=\"learning rate of actor\")\n", - " parser.add_argument('--memory_capacity',default=80000,type=int,help=\"memory capacity\")\n", - " parser.add_argument('--expl_noise',default=0.1,type=float)\n", - " parser.add_argument('--batch_size',default=128,type=int)\n", - " parser.add_argument('--target_update',default=2,type=int)\n", - " parser.add_argument('--tau',default=1e-2,type=float)\n", - " parser.add_argument('--critic_hidden_dim',default=256,type=int)\n", - " parser.add_argument('--actor_hidden_dim',default=256,type=int)\n", - " parser.add_argument('--device',default='cpu',type=str,help=\"cpu or cuda\") \n", - " parser.add_argument('--seed',default=1,type=int,help=\"random seed\")\n", - " args = parser.parse_args([]) \n", - " args = {**vars(args)} # 将args转换为字典 \n", - " # 打印参数\n", - " print(\"训练参数如下：\")\n", - " print(''.join(['=']*80))\n", - " tplt = \"{:^20}\\t{:^20}\\t{:^20}\"\n", - " print(tplt.format(\"参数名\",\"参数值\",\"参数类型\"))\n", - " for k,v in args.items():\n", - " print(tplt.format(k,v,str(type(v)))) \n", - " print(''.join(['=']*80)) \n", - " return args\n", - "def smooth(data, weight=0.9): \n", - " '''用于平滑曲线，类似于Tensorboard中的smooth\n", - "\n", - " Args:\n", - " data (List):输入数据\n", - " weight (Float): 平滑权重，处于0-1之间，数值越高说明越平滑，一般取0.9\n", - "\n", - " Returns:\n", - " smoothed (List): 平滑后的数据\n", - " '''\n", - " last = data[0] # First value in the plot (first timestep)\n", - " smoothed = list()\n", - " for point in data:\n", - " smoothed_val = last * weight + (1 - weight) * point # 计算平滑值\n", - " smoothed.append(smoothed_val) \n", - " last = smoothed_val \n", - " return smoothed\n", - "\n", - "def plot_rewards(rewards,cfg,path=None,tag='train'):\n", - " sns.set()\n", - " plt.figure() # 创建一个图形实例，方便同时多画几个图\n", - " plt.title(f\"{tag}ing curve on {cfg['device']} of {cfg['algo_name']} for {cfg['env_name']}\")\n", - " plt.xlabel('epsiodes')\n", - " plt.plot(rewards, label='rewards')\n", - " plt.plot(smooth(rewards), label='smoothed')\n", - " plt.legend()\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "训练参数如下：\n", - "================================================================================\n", - " 参数名 \t 参数值 \t 参数类型 \n", - " algo_name \t DDPG \t \n", - " env_name \t Pendulum-v0 \t \n", - " train_eps \t 200 \t \n", - " test_eps \t 20 \t \n", - " max_steps \t 100000 \t \n", - " gamma \t 0.99 \t \n", - " critic_lr \t 0.001 \t \n", - " actor_lr \t 0.0001 \t \n", - " memory_capacity \t 80000 \t \n", - " expl_noise \t 0.1 \t \n", - " batch_size \t 128 \t \n", - " target_update \t 2 \t \n", - " tau \t 0.01 \t \n", - " critic_hidden_dim \t 256 \t \n", - " actor_hidden_dim \t 256 \t \n", - " device \t cpu \t \n", - " seed \t 1 \t \n", - "================================================================================\n", - "状态维度：3，动作维度：1\n", - "开始训练！\n", - "回合：10/200，奖励：-945.22\n", - "回合：20/200，奖励：-700.56\n", - "回合：30/200，奖励：-128.48\n", - "回合：40/200，奖励：-266.74\n", - "回合：50/200，奖励：-387.26\n", - "回合：60/200，奖励：-133.07\n", - "回合：70/200，奖励：-243.47\n", - "回合：80/200，奖励：-383.76\n", - "回合：90/200，奖励：-130.47\n", - "回合：100/200，奖励：-385.78\n", - "回合：110/200，奖励：-128.11\n", - "回合：120/200，奖励：-245.72\n", - "回合：130/200，奖励：-3.26\n", - "回合：140/200，奖励：-231.93\n", - "回合：150/200，奖励：-122.84\n", - "回合：160/200，奖励：-370.19\n", - "回合：170/200，奖励：-126.60\n", - "回合：180/200，奖励：-118.99\n", - "回合：190/200，奖励：-115.58\n", - "回合：200/200，奖励：-246.70\n", - "完成训练！\n", - "开始测试！\n", - "回合：1/20，奖励：-122.76\n", - "回合：2/20，奖励：-1.78\n", - "回合：3/20，奖励：-128.77\n", - "回合：4/20，奖励：-124.03\n", - "回合：5/20，奖励：-125.87\n", - "回合：6/20，奖励：-130.87\n", - "回合：7/20，奖励：-127.97\n", - "回合：8/20，奖励：-134.63\n", - "回合：9/20，奖励：-126.38\n", - "回合：10/20，奖励：-1.42\n", - "回合：11/20，奖励：-126.13\n", - "回合：12/20，奖励：-1.88\n", - "回合：13/20，奖励：-133.22\n", - "回合：14/20，奖励：-132.14\n", - "回合：15/20，奖励：-245.42\n", - "回合：16/20，奖励：-123.41\n", - "回合：17/20，奖励：-127.20\n", - "回合：18/20，奖励：-130.53\n", - "回合：19/20，奖励：-129.29\n", - "回合：20/20，奖励：-288.72\n", - "完成测试！\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAAHJCAYAAACBuOOtAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAADg6klEQVR4nOydd3gcxfnHv1uun069We5N7rhjbGNsDJhqOuRHDxBCaAmYlkBICElIgEACxvSEDgklEHrvGDds3HCvsq1itZOu7W35/bE3e7t7e6eTdGrn+TwPj9HelpnZ3Zl338ooiqKAQqFQKBQKJYthe7oBFAqFQqFQKF0NFXgoFAqFQqFkPVTgoVAoFAqFkvVQgYdCoVAoFErWQwUeCoVCoVAoWQ8VeCgUCoVCoWQ9VOChUCgUCoWS9VCBh0KhUCgUStZDBR5Ku6B5KikUCoXSF6ECDyVtPvnkE9xyyy0ZOdfrr7+OyspKVFVVdekxlEOLp59+GrNmzcKECROwZMkSy30qKysN/40ZMwaHH344Lr30Unz22WcZ37+yshITJkzASSedhCeeeAKyLCccs3btWvz617/G/PnzMWHCBEyfPh0XX3wx3n777bT6fe+992L69OmYOHEi3njjjbSOSZdly5Yl9GfUqFGYPHkyfvKTn+DTTz/N6PXSacuyZcu69JieIBAI4M4778SsWbMwadIk/OxnP8OOHTt6ullZBd/TDaD0HZ5++umMnWvu3Ln497//jZKSki49hnLo0Nrair/+9a+YO3cuLr30UvTv3z/pvmeddRbOPvtsAEA0GkVdXR1ee+01XHnllbjttttw0UUXZWx/AAiFQvjwww9x3333we/3Y9GiRdpvTz/9NO655x5MnToV11xzDfr374/m5mZ8+OGHuPHGG7FmzRrcfvvtSfuyZcsWPPnkkzjnnHNw6qmnYujQoekPWju44447MHbsWACqpre5uRn//Oc/cdVVV+Gxxx7DUUcd1SXXPVRYtGgRfvjhB9x0003wer1YvHgxLrroIrzzzjvIzc3t6eZlBVTgofQIBQUFKCgo6PJjKIcOzc3NkGUZxxxzDKZNm5Zy37KyMkycONGw7cQTT8S1116Le+65B0cffbRBYMrE/kcccQR27NiBF154Addddx1sNhuWLVuGv/zlL7jgggsShJpjjjkGo0aNwl//+lecfPLJCecjNDU1AQBOOukkTJ06NWW/O8Pw4cMT2jB16lTMnTsXzz77LBV4OsHq1avx2Wef4fHHH9fGcerUqZg/fz5efPFF/OIXv+jhFmYH1KRFSYsLL7wQy5cvx/LlyzX1MFEVv/zyy5g3bx4mT56Mb775BgDwyiuv4IwzzsDEiRMxYcIEnHrqqXjvvfe085nNU7feeisuueQSvPbaa1iwYAHGjRuHU089FV9++WWnjgHUyeT888/HxIkTMXfuXDzzzDO45JJLcOutt6bs85o1a3DppZdi8uTJmDFjBm644QbU1NRYtoVw9NFHG85bWVmJxYsX44wzzsCECROwePFijB49Gs8//7zhuIaGBowdO1bTosmyjMcffxzHHnssxo0bhwULFuC5555r8z61tLTg7rvvxjHHHIPx48fj5JNPxquvvprQxgcffBB//etfMXPmTEyYMAGXXXYZdu3alfLcgiDg73//u2Z2Ofnkk/Hf//5X+/3CCy/ErbfeikcffRQzZ87ElClTcNVVV2Hfvn3aPrfeeiuOPvpow3mrqqpQWVmJ119/PeX1v/nmG5x33nmYMmUKDj/8cCxatAgHDhwAoN4Pct7f/OY3qKysbHOsrLj++usRjUYTxixT+48bNw6BQADNzc0AgIcffhjl5eW46aabLPe/6KKLMH/+fIRCIcvfH3roIVx44YUAgIsvvlgbA0mS8MILL+CUU07BhAkTMHfuXNx3332IRCLasbfeeisuvvhi/O53v8PkyZNx4oknQpKktPpB8Hq9GDJkCPbv369ta2pqwh133IGZM2di/PjxOOecc7B06VLDcZWVlXjhhRdw2223Yfr06Zg0aRJ++ctf4uDBg4b9Xn75ZSxYsAATJkzABRdcYLgO6b/Vva6srMRDDz1k2eZ0nkEyty1duhQXXnihNoavvPIKamtrcc0112DSpEk46qij2tR8P/rooxg3bpx2zwlPP/00xo4di/r6enz99ddwu92YPXu29ntBQQGmTZuGL774IuX5KelDBR5KWvzud7/DmDFjMGbMGPz73//WVNsAsHjxYtxyyy244447MGnSJLzwwgu44447cMwxx+Cxxx7DfffdB7vdjhtvvBHV1dVJr7F+/Xo89dRTuO666/Dwww+D4zhce+21CRNFe47Zvn07LrnkEgDA/fffj2uvvRaPP/44Vq1albK/GzduxAUXXIBIJIJ77rkHd955J9avX4/LLrsMoii2Y+TUCe+UU07Bgw8+iAULFmD69Ol45513DPu8//77UBQFJ510EgDg97//PR588EEsXLgQjz76KI4//nj8+c9/xsMPP5z0OuFwGOeddx7eeustXH755ViyZAmmTJmC2267DY8++qhh32effRY7duzA3XffjT/+8Y9Yv359m/5ZN954I/71r3/h7LPPxmOPPYbZs2fj1ltvNfiZfPLJJ3j99ddx++23484778SPP/6ICy+8MOmCnS5vvPEGLr30UpSXl+P+++/Hr3/9a6xevRrnnnsu6uvrMXfuXCxevBgA8Itf/AL//ve/O3SdoUOHol+/fm0+Hx3df+fOnfB4PCgsLERzczNWrFiB+fPnw+FwWO7P8zyWLFmCI444wvL3s88+G3fccQcA1eRExuCOO+7QBN9HHnkE559/Pp5//nlcddVVhsCDlStX4sCBA3j44YexaNEicByXVj8IgiCgqqoKAwcOBABEIhFcfPHF+OSTT3D99ddj8eLFKCsrw+WXX54g9DzwwAOQZRn3338/br75Znz22Wf485//rP3+/PPP43e/+x2OOuooLFmyBIcddhh++9vftqt9neWGG27A0UcfjcceewxDhgzB7373O1x00UUYMWIElixZggkTJuDuu+/G2rVrk57jlFNOgSiK+PDDDw3b33nnHcyePRuFhYXYvn07+vfvnzD+AwcOxM6dO7ukb4ci1KRFSYvhw4fD6/UCQIJa+7zzzsPxxx+v/b13715cdtlluOqqq7RtFRUVOOOMM7Bq1SptUTfT0tKC119/XZs83W43LrjgAnz33XdYsGBBh4557LHHkJOTgyeffBIulwuAukj95Cc/SdnfRx99FHl5efjnP/+pLUYlJSVYtGgRtm7dmvJYM1OnTsVPf/pT7e9TTz0Vv/nNb7B//37069cPgDr5zZw5E8XFxdi5cyf+85//4IYbbsAVV1wBAJg9ezYYhsFjjz2G8847D/n5+QnXef3117Flyxa8/PLLmDRpEgDgyCOPhCiKWLJkCX7yk58gLy8PAODz+bBkyRJtgt2zZw8eeughNDY2Wp57y5Yt+OCDD/Cb3/wGF198MQDVRLNv3z4sW7YMJ598MgDVV+X111/HgAEDAKhjffrpp+ONN97A//3f/7Vr3AiyLOO+++7D7Nmz8be//U3bTrQSTz31FG6++WaMHj0agLpIJDP/pENRUVGCpqG9+8uyrAnGiqLg4MGDeOutt/Dpp5/i8ssvB8Mw2LdvH2RZxpAhQwzHKoqSoGlhGMZSGCkrK8Pw4cMBqO/omDFjsG3bNrz66qtYtGiR9vzMmjULJSUluPnmm/Hll19qZhNRFPGHP/wBZWVlbfZT3ydRFLFv3z4sWbIEDQ0NOP/88wEAb775JjZt2oT//Oc/OOywwwAAc+bMwYUXXoj77rsPr732mna+kSNH4u6779b+Xrt2Ld5//31tDJYsWYITTzwRv/nNbwCo70BraytefvnlNtuaKc4880zt3XW73TjnnHMwYcIE/PKXvwQAjBo1Ch9++CG+//57TJgwwfIcFRUVmDZtGt5++23Nr2vPnj1Yu3YtHnjgAQDqPEbmVz0ejweBQKArunZIQjU8lE5DFhrCrbfeihtvvBF+vx9r1qzBm2++iRdeeAGA+kWYjIKCAk1wAaBNwqm0A20d891332HOnDmasAMAkyZNQkVFRco+rVq1CnPmzDF8eU+aNAmffvppQn/bwrz/cccdB4fDgXfffRcAcODAAaxatQqnnnqq1mZFUXD00UdDFEXtv6OPPhqRSCSpNmH58uWoqKjQhB3CwoULEYlE8MMPP2jbxo8fb1hA2xprcs3jjjvOsP2hhx7CXXfdpf09efJkTdgBgDFjxmDAgAFYsWKF5XnTYefOnairq9OEKsLAgQMxadIkLF++vMPntkJRFDAM06n9lyxZgrFjx2Ls2LEYN24c5s6di4cffhjnnnsurr32WgCwjNYCgKVLl2rHkv+IljIdyHiYPyxOOukkcBxniFbKy8tLS9gBgEsuuURrz2GHHYYTTzwRS5cuxe233445c+ZobS8uLsbYsWO151aSJMybNw/r1683aGvNQmlZWZn2/O3YsQP19fWYN2+eYZ8TTjghvUHIEPp3qbCwEAA0QQ6A9nHQ0tICQDUl6t9Zco8XLlyIFStWoK6uDoD6geP1ejXTWqp0H+15FimpoRoeSqdxu92Gv/fs2YM77rgDS5cuhc1mw9ChQzFq1CgAqV9svVACxF/0ZAtDOsc0NDRoE5WeoqKipOcEVD8Eq+M6gnl8vF4vjjnmGLzzzju4/PLL8e6778LlcuGYY47Rrg0kLlgE4kdkprm5GcXFxQnbSV/9fr+2zTxuLKt++yQba9KmtsaktLQ0YRsx33QUcm2re1ZUVISNGzd2+NxWVFdXY+TIkZ3a/5xzzsE555wDQH0mPR4P+vfvD5vNpu1DtHtmP7AJEyYYfIJ+97vftav9ZKzNzwLP88jPz9cWZ0DVIKTLnXfeqZmyOY5Dbm4u+vXrZ1iQm5qaUFdXZzB566mrq9MijqyeQTI/kD6YtY1Wz3dXYqV1Mbdbz7HHHmvwWTv99NPxl7/8BccffzzuuusuvPfee1rk1YIFC+B0OrXrWGkVA4EAcnJyMtATCkAFHkqGkWUZV1xxBWw2G1599VWMHj0aPM9j27ZtePPNN7u9PWVlZZYTSX19fcrw3ZycHDQ0NCRs/+KLLzB69Oikwli66ueFCxfiiiuuwO7du7XJj0ykPp8PAPDMM89YLkhkoTSTm5uL3bt3J2wnX5VWpqp0IW1qaGgwaAS2b9+OpqYmTJkyBQDQ2NiYcOzBgwc1LRzDMAnmmmAwmPLaxAxndR/r6uo61S8z27ZtQ11dnWai6ej+JSUlGD9+fMpjCwoKMGnSJHz88ce48cYbNY2b1+s1HNseoQSAJlDU1dUZNJnRaDSpyTIdhgwZ0mafcnJyMHjwYNx3332Wv6dKFaCHtLG+vt6wnQi/BPIeSpKkjV9b72BHnsF0eeSRRwxabNKPnJwcHH300XjvvfcwY8YMbN261eCPNGTIEHz99deQZVn7+ACA3bt3Y9iwYRlpG4WatCjtQP8iJqOxsRE7d+7EWWedhfHjx4PnVZmaRE6l0tZ0BdOmTcNXX31liE7ZuHFjm8kLp06dim+++cYweW3cuBFXXHEFNmzYoH356Z2wyeKfDrNnz0ZRURGeffZZbNiwQTNnkWsD6liOHz9e+6+hoQH/+Mc/kl5j2rRp2LdvH1avXm3Y/r///Q82my2pj0E6EIHGnGTuvvvuw5/+9Cft71WrVhmEnvXr16OqqkpzuvV4PGhsbDTcj7YcfocMGYLi4uKEJHx79+7FmjVrMHny5I51yoIHH3wQTqcTp59+epfsb+bqq6/G3r17cc8991hqP5ubm1FbW9uuc06fPh0AEhzj33nnHUiSpN3LrmD69Ok4cOAACgsLDc/uN998gyeffDJtp+jBgwejvLxc8+khmBM9Wr2HbT1PHXkG06WystLQb72Ad+qpp2LNmjV46aWX0K9fP+0+Aep8EAgE8NVXX2nbGhoasHLlSsyaNSsjbaNQDQ+lHfh8PqxevRpLly7FmDFjLPcpLCxERUUFXnjhBZSVlcHn8+Grr77Cs88+CyC1P05XcOWVV+Ldd9/F5ZdfjksvvRR+vx//+Mc/wLJsStv4VVddhXPPPRc///nPcdFFFyEcDuPvf/87JkyYgFmzZiEcDsPpdOIvf/kLfvnLXyIQCODBBx/UtBFtwXEcTjrpJDz//PMoLS3F4Ycfrv1WWVmJhQsX4re//S327duHcePGYefOnXjggQfQv39/DB482PKcZ5xxBl588UVcffXVuO6669C/f398+umneO2113DNNddoWpqOMGrUKBx//PG49957EQ6HMXr0aHz55Zf47LPPtMggQL2/l19+OX7xi18gEAjggQcewMiRIzX/m3nz5uG5557DbbfdhrPOOgtbtmzBv/71r5QLIcuyuOGGG/DrX/8aixYtwsKFC9HY2IjFixcjNzfX4BCeLtXV1VizZg0A1QG3pqYG//3vf/H1119bOvG2d/90OfLII/Hb3/4Wd999N9asWYPTTz8dQ4YMQTAYxPLly/Haa68hEokkJDZMxfDhw3H66afjwQcfRCgUwrRp0/Djjz9i8eLFOPzww3HkkUd2qK3pcMYZZ+D555/HT3/6U1x55ZUoLy/Ht99+iyeeeAIXXHCBwaSXCoZhcOONN2LRokW4/fbbcfzxx2vCgp6jjjoKd999N+644w5cdtllWsRZKq1YR57BTHDkkUciLy8P//73vzXHdcK0adMwffp03HTTTbjpppuQl5eHhx56CDk5OR129qckQgUeStqcf/75WL9+PX72s5/h7rvvTprxeMmSJfjTn/6EW2+9FXa7HcOHD8cjjzyCP//5z1i5cqWWN6Q7GDRoEJ566incc889uO6661BYWIif//zneOSRR1JOimPGjMFzzz2Hv/3tb/jVr34Fr9eLo446CjfeeCPsdjvsdjseeugh/O1vf8PVV1+NiooKXHPNNe1K63/qqafimWeewcknn5ygPbv77rvx2GOP4eWXX0Z1dTUKCwtx4okn4le/+lXSidnlcmlt/sc//oHW1lYMHToUf/rTn3DWWWel3a5k3HvvvVi8eDGeeeYZNDY2YtiwYXjwwQc13yNA1U7NmDEDt912GwA158/NN98Mu90OQI0WuuWWW/Dcc8/hgw8+wNixY7F48eI2o+bOOOMMeDwePPbYY7j66qvh9Xpx5JFH4oYbbuiQX8err76q+cmwLIu8vDwcdthh+Ne//mUZAt7e/dvD+eefj+nTp+Oll17Cv/71L1RXV4PjOAwZMgQXXHABzj33XEvfqFT86U9/wqBBg/Daa6/hiSeeQElJCS666CJcddVVaWlqO4rb7cYLL7yAv/3tb7j33nvR0tKCiooKLFq0CJdeemm7zkXeiyVLluDNN9/EyJEj8Yc//AE33HCDts+QIUPw17/+FY888giuuOIKDBs2DHfddZfBkd5MR5/BzsLzPE466SQ899xzWLhwYcLvixcvxl/+8hfcc889kGUZkydPxt///neaZTmDMAqtBknJYojjtD4Drd/vx8yZM3HzzTe368uZkhoiyKaTIJFCoVC6G6rhoWQ1GzZswIMPPogbbrgBY8eORVNTE/71r38hJycnIcyZQqFQKNkLFXgoWc2ll14KQRDw0ksv4cCBA3C73Zg+fTruvvtuWpeLQqFQDiGoSYtCoVAoFErWQ8PSKRQKhUKhZD1U4KFQKBQKhZL1UIGHQqFQKBRK1kMFHgqFQqFQKFkPjdKKoSgKZLlr/LdZlumyc/cGsr1/AO1jNpDt/QOyv4/Z3j+A9rEj50q3ojwVeGLIsoKGhvQKP7YHnmeRn++B3x+EKHZvHanuINv7B9A+ZgPZ3j8g+/uY7f0DaB87QkGBBxyXnsBDTVoUCoVCoVCyHirwUCgUCoVCyXqowEOhUCgUCiXroQIPhUKhUCiUrIcKPBQKhUKhULIeKvBQKBQKhULJeqjAQ6FQKBQKJeuhAg+FQqFQKJSshwo8FAqFQqFQsh4q8FAoFAqFQsl6+qzAI8syHnzwQRx55JGYOHEifvazn2Hv3r093SwKhUKhUCi9kD4r8CxZsgQvvvgi7rrrLrz88suQZRmXX345BEHo6aZRKBQKhULpZfRJgUcQBPzzn//Eddddh7lz52LUqFF44IEHUF1djQ8//LCnm0ehUCgUCqWX0SerpW/atAmBQABHHHGEts3n82HMmDFYsWIFTj755A6dl+czL/9xHGv4N9vobP/8AQGBcBTlhZ5MNittIlEJNp4Fy6jVdhVFQTAswuOyafuQvkWiEhpbIigrcPdIW7uSjtzHDTsbsK+uFcdOGwCGSa9acSYIRUS4HO2bulL172BTCABQlOcCAIiSjE27G+Fy8Cgv9MDt7Pw0eaA+AIeNQ4HPmXSfUEQEwwBOe9vXS/WcWvUxFBERiohgWQYsw4BlGTjtHPg+NC+l6p+sKAiEoshx2zN+XVGStTHravrCehERJLAsA1uK9XLnAT8YAIPLfQm/9WQf+6TAU11dDQAoLy83bC8pKdF+ay8syyA/v+sWXZ/P1WXn7moURUF9c1hbEKzoaP/ueGo5qmpb8M/fHof8nOSLQVfQEhTwi799gXFDC3H7pYcDAF7+aDNe/GAT/njlTEwYXmzY/++vrMWPO+vx5G3HoTi//f39fNVehAUJxx8xOBPN7xLacx+ffWQpDtQHcPiECsuJrStYs6UWv3t8KS44YTTOnj+y3ceb+ydKMq6+/0sAwDO/WwAbz+Kdr3fg0f+u0/Y5dc4wXH7quA63uTUo4LdPLkeBz4EnfnOs5T6SJOOGP30EjmPx+K+PaXNxffLN9Xjrq+2475dzMGJAvuE3cx83727ArQ9/DVFSDNvzcxx4+Oaju0RI6EqsntFHXvsB7y/dhQeun4uhFbkZu1ZUlPGLv36CHI8d9/9yTrcJ9r11vQgLIq77+8coynPigevnWu4TiUq4+7nPwDAMXvjD8bDxnOV+PdHHPinwhELqF5ndbnxRHQ4HmpubO3ROWVbg9wc73TYzHMfC53PB7w9BkuSMn787+GD5Hrzw4RZceepYzBxvFDI70z9ZVrC72g9FAbbsrMfIAXkZbHXbbK1qQiAUxabdDWhsDAAAftxRD0UBNmw7iAGFqiaH9HHnvmbICrBjbwN4tK+toiTj7y+vhiwrGDMwt9ctMh25j/6g6i+3q6oRuU7rSS3TrP6xBrIC/LClFsdMrkj7uGT9awkKaCH92NuAojwXNu9qAADwHANRUvDV6iqcOWdIh9u884AfQlRCTX0QDQ2tloumPyDgYHMYAHCgxt+mVmnTznrICrBmUw2KvPaUfVyzuUYTdhgAROxpbIlg/ZZajBqUj75Aqmd08+4GyAqwcVsd8t2ZW9YO1AdQ0xBETUMQB2r87dYstpfevl7sqWlBU2sEza2RpM9yTUMQYUECAOza25jwoZzpPvp8rrS1RX1S4HE6VU2AIAja/wNAJBKBy9VxqVEUu+4BkyS5S8/fleypbgEAVNW1Ju1DR/rXHBCgxGbfRn+k28cnFBYBANFovO2CqL6oQlQytCcqyghG1P0FQWp3Wxv8YUiy2tn6pjBcaZgteoL23EeyX3feu8bWCACgqUXo0DXN/QuEotr/1zeHked1oLFFvcaC6QPxztLd8AcFRKNSh7/uG2KCjALVHGBlRmoNxoMtAqEo7G2Y10OC+ixajb25j+Q5nzW+DJedNAaKouAPT6/E7poWBMPRPjcvWT2j4Yj63gYjYkb70+iPaP9f3xzuNnN2b10vyHgoAIJhEQ5b4odOgz+s/T95p6zoiT72XkNhCogpq7a21rC9trYWpaWlPdGkrCYSVScTs0q8s/gD8Um+Jdj90XVCVH3ZRN1XBnkBzS+iPxCf+ES5/S9pSzC+sPp7oK9dARm37rx3LbFnJlNjGIl9iQJAU6sQ+1e91wNKvADU554Iux2hWfecR5NM8GFdO8j7lgrS7nTGgZzPaVOFbIZR/XfU33rfotoRyPjp72cm0M9Rza2RFHseGjTr5kEhyXPa3Kobs0Dvmuv6pMAzatQoeL1eLFu2TNvm9/uxceNGTJs2rQdblp0QwUDKsMCjf3l6Qggg2pyoKEOJqZqisUU8alK16l/ijoyD/sXPBoFHlhVNY9Wdk5o/Jjj6AwJkpfPPo37BJ88j6U9xngsuB6ddr6Pox0dIIvDohZx0Fm2ywKfTroigXtNuj0/3jpjAExY6Lsj1Jsj4JVuEO4r+3jW19v33trPoxyOZYN7ci4XE3qlXbwO73Y4LLrgA9913HwoKClBRUYF7770XZWVlOO6443q6eVkHebClDmg2UqEXIloC0RR7dg2kXwoASVZUnw0xJviYFqYm3YtLFvr2oF+Y/D3Q10yjFwi7U4Aj4yjJmYnK0U/aTa2qEEWukeuxw+dxIBQJorlV6HAkob9Vr+GxXiT0Gp50hJBwtP0aHr35gfx/pjUiPQXpYzrasfZg0PD0Mm1FT+A3CDzW60FvFhL7pMADANdddx1EUcTtt9+OcDiMadOm4amnnoLNZmv7YEq7IJqQjJu0gj2r9RB0L6woyeA5NoWGJ2LYt73o+9cT5rtMox+Dlm5cCPRj5w8InRd4dAt+c2sEraEoJFkBA8DnsSPXbUNNQ+eeT70mM5lJK2IwaaV+vmRFgdAeDY9m0rIQeDIsIPQEkixr45rp/vRmbUVPYNBWJhlrvflf/+z3BvqswMNxHG666SbcdNNNPd2UrIeoxLtUw9ODJi1AXYic9uQ+PAaTVqc1PFkg8Ih6U1D3aKxESUYgHNd+NAcEVBSnOCAN9Atkc0DQ7rPXbQPPsfB57NpvHSUdH56QTqvT1qIdjcpapJU/2PbYE2HKbtcJPPbsEXjI/AR0rYant2kregL9PJiOD09vG7M+6cND6V6IYNCRhT4VBiEgjYk70xg1PEYfHrMWR/911xEfHqOGJwtMWrqFu7uEVfO4ZUJwNJq0Itp9zvU4DP9myocnHQ1PWyatsMnfp61FPqWGR+j7TssG/6cMO2Hr39vepq3oCfTjkZYPTy/7uKMCD6VNNB+ejDst96zWQ/+FopmyROO/BINJqwOaLqNw17smgY4QNeSyiWbEgbgtzM9IJiZTo0lL0L5I82K5bXweW6ev1V6nZaGNRdssELX17lj68Gganr7vtKwfj0w7LVMfHiNGDU/bPjy9zQxIBR5Km1iFb2cC/WQSCEUzbjJrC/0LSwQcUdPwGBfwzkZpZZ1JSzcGJK1/V2MWFDMi8OgWSH9QQGOLmkMkVxN41H87es/CgmgQqtIJS29Lw2N2NG5T4Int79CZtIi2J5wFTsvtjXBLF0XnwA4Y54BDEVGS0ap7z600PLJpzPyBKOQMWwY6AxV4KG0iRLvGpKVfsBQAraHu/dqM6Hx4RJNmJ5XTcod8eLIsD4954e4OIc58jUybtBQF2FunZtwmydI6K/CYj0snSqstE5VZSGnreUql4WlLm9QXiLQzh1G6hAXJoJFrDUUz/tHXlzCblK3GOhgWtfmRgSoAtXTDx1C6UIGHkhJRkrUHOJMCj/5rgYvVDerOaB/AqP4WJRmyEs8tk2DSCugFnvaX0ND7uQhRuc+HA5sn/u7wwSILO3leMm3SAuJZxXNjgo7mw9NBIdXcxrSitNrwqzEvNB0yadmyJw9Pe5M2pgu553ae1Z65bNDOdhRz362EZZK+w+uyIccdMwf3IrMWFXgoKdELBZms7UK+FliGQWksXXt3az7MJi19/1I5Lbc3PL81FIWiqF88pKxAX9fymDVg3bEQkFxNJL1/JkwM5gWyNlY5Pa7hUSdtf0DQklO2B3Mbk/nwhA1RWm04LWfApBUPS+/7Gguj/1MGBZ7YuPo8ds3E2duijroTs9O2lXBJBPxcjx25sXeoN/k+UYGHkhL9hChmUMNDXh6fx6Z9TXe7wCMaNTz6r2/9/wtRCaFIxwU/0i+PS9fXXjQJdISE0hvdcO/IxNk/VvIhE9dMphEgCxy5Xx0tL5GuhkcfedWWX02i03Jy7ZosK5qQZe203Lc1jYBZw5M5AU6fgJI8B71JW9HdmIV3q2eHJNk0Com9Z8yowENJiVHDk0GBp5VMJg5N9dnd2ZbNGp6orn+iIZOwsV3tNe0ZM/fGNAZ9XcPTAz48xCzYv9ij/d1Zh0ii/WBNhUHJAmfjOa1Cdkf62BGTVlt+NQlOyymeJf2iZKnh6eOmVaDrfHgMGp6YabOpj3+odAbzc2alTdM0PF69kNh7xowKPJSU6CeQTEZR6ScTn7uHNDymsHRzdXSCeaFrr+BHjs9x27TMwH09F4/Z5NcduXjI81FR5AXDqE7Gnb0ueb6L8pyG7bm6Cs+dcVz2m8wAmSgtYfbJSdUu8owzgKECe1ZpePTvsShnLCpIb54haQqohifuQ2dt0iJ5rOyaWZgKPJQ+g/5rM6MaHt1kktNDZp6IaPTZ0fulGPPMGNvV3jw8BuEuS0xaiT483eC0HBuzvBy7Jjh21j+AmEBK893aNpeDM5h/ct0dz8VDJnsibJjHTWtHB6K0ivNcANLT8NjtHBidFiurwtJNfRCSCJXthWh29e9tb/JH6W5I3wtz1Y8DK/NhfF53aBqepl6UsJEKPJSUGEK3M+rDE1d9+ohJq5u1HoLpy1Cv4RFTaXjaa9LST5zu7BB4yPho0StdrOFRFEV7PnzuzI0jEQhK8l3aNmK+IHRmsTMvEsnMVUan5bYSDxKBRz1nqjEg++oFOCCu4VGjMPu247JZ4MmUH4/+Q6U3aiu6GzIeRNC2NGm16rVi1GmZ0scQdJNJJqO0tMnE3TtMWqKkJNXwmNvVbqdlQ1+zxIcnpu3Lz+l86YV0COjye+S44w6RndbwCIkCDzFfEDpTXqLZtEgk0/AYy0Wkl3iQtDkQFpPmhyEClsNmnOr1AlBfLy8RNkW1ZcpMp39ve6MDbndjfpatBB5tzLy6d7QXjRkVeCgp0Wt4MpmHR6/h6SmTlmDy2YkaNDzxvhLNAjEItNuHJxj/UiR9zRYfngJfTMvQxQIcMSu6HDxsPJtxDY/epKX33wGMoentQZ91lmh4rJyWZVkxaH7ajNKKtbnQ59ScrZM9T0QYcNiMdaJ5jtGO7et+PAkmrQyZ6ayclnuTtqK7IX0viQk8bYal65yWO5LSoSugAg8lJUYfnsx9CepfjJ4wacmKYgxDl2TDV7KahFD9Wz/xAe0X/Jp1x2s+PH1cw0NMWoU+dSEQonKXJrGLf22rz0rGNDyWJi2jhqejJi21XIr6rBT5kgs85oWjbR8edZxdDh45bQhjRHvjsBuneoZhssZxOdzO8UuX5mCi07I/IHRL3bjeRlSUEIqlZSCmVLN5Vp9MNs/r0D4cBFE2pPXoSajAQ0mJMUorcy+6XoggDqiRqNRtYbJR88tq0vCo29T+xp1l1Re4venlW4J64S47fHiIacbjtMFuI8kUu05g1ftBAchIPiNZjgu9HicPj1PVguQlaHg6di0iIHldNjhjoe1WAo9ZoyNKSspnLKLzy8ltw3mbJDF0mnx4hA2f4GznN3BA6POh6Yk+PJ3vj34u0n+oSLJiqCdFUBSl12gxOoMQlSyj3MjzxXOM9n4ky/jNsQzcTh4OGweXg4sd3zvMWlTgoaTE4OeSIYFHTeSnTsS5Hjucdg62WBRLd4Q3A0ZTHZCo4SHb1DapExzxV2mP4KcvQJjjtmkaitZg7yqq117Iws23YV5qDUWxp6al09fT+1MAnXMkJhhy1Ng4bSLPTebDk8azqb+nei2mLZZh2yqCSIuk0oWNC1EJ+w8G8PB/1+Hb9QcMWgWyEDvtnM5EmkzgUe+TXSfwCGvfR+Sb5zCF24yLvF8jIrRfUP3fNzvx70+3Wv4mSnK3Lv6ZEHj2HQzgh20Htb/J82bjWTjtHHiOhdelvrurNtUiGI6PmaIouPuF73HXMysz9k7LioLGloil4HugPoDVW+s6rGmKipKlNnbp+mr86qGv8fdXf0j4Tf8sx7N0G8e5sYUkk7Vr5lLNFNhLnL35tnehHMpEusCkdaA+CEAts+By8GAYBj63DfX+CPzBKIryXG2cofOYHe4sNTyS0aTVEYEnFBG1UhS5HjtYlgEDUiw1qi3cfQ0yNjaOhc9jx8HmcEIttAP1Adzz4mo0BwT87OQxOGJcGURJxrKNNRhUloP+xd42r9MaiiIqyglmRS3ktTUCRVEMIdepEEQJm3c1YsSAPG3CZqAubNNGlyD0w35UDsgzHGMuL2F1rX0HA3jyrY1oCQn44+WHw2nnDVlniRbMnKEaiJuoPC4bpIAASVYQFiR8vfYAVm2uw6rNdfhg+V5ctKASwypyNROOw84ZhE0hKiEYNgrSeuEIAKJbvkHku5cBADIYjLNXoenHd1DvORWLX12JUUNKcNYxY8Cxyb+Fo6KEN77aCQA4btpA7b0A1Pv1m8e/w5jB+bjy1HEJxyqKgv31QazfUY/BZTmoHJif9DqELXubsHZ7PSaOKMLhue6E38l48BwDUVIQiX1Q/fn5VRg/pBDnHD28zWs8+sZ67DsYwN0/n4HSfLdBwCb3uzjPhdZQFM99uAUvfrwV15wxHocNL0JrKIptVc0A1OeR+LW1l0hUwrod9diweyuWbTiA5lYBLMOgMNeBicOLcczU/vh+Sx1e/Xw7JFnBlMpiXHbSaDjtxmX8YFMIG3c3IirKmDe5QhM+REnGp9/vw1vf7ISiAL88ewJG9M9DWBDx0sdb8dXaAwCAzXuaEp7z+LPsgN1UeHZ3dQue/WAzdpvq0JH/r24I9hoTPhV4DlHSXSQykWm5vjmM1VvrkJ/jRCQq4oWP1C/DwWU5Whty3PaYwJP4YkiyWmzT7bQZzpnrtWu1qd78eie+XnsAt100JcEkYd2vROHGHEETFeVYOLRR4EllbpAVBaGICI/TmLvF5eBg49WJwuOyoTUUhT8odJnAs+9gALur/ThibFnawoCe7fua0RKKYuLwIsvfrTQ8zbp7V9MQxD0vrdb6/+wHmzGw1Is3vtqJVVvqwHMsLj1xFGaMLVO/VBWAZePtbGqN4J2lu/HFmn0QJUUzN5Gs3CV5LjBQhecn396Is+cNx7aqZuypbUWux46iXCfGDC7QNIeEd7/djde/3IHzjhmB8cMKAcRz1CycNQQLZw0x7K8oMnx2CT4miIDkQDB2b0MRER+t3AtZViArwIfL92hO8DUNIQwqy9FKUXicvKbhsfTh0QklQpRDICwiEpUM2qu9ta147oPN+P2l0zUTmNPOa4vL/77dhVc/3w7yhg4s9eK2C6cYkhSK+39E+It/AgBs4xfg3c0KjhU+RN7uT4Ddn+AGBsAuoPVJBrb+4+A6/nowMcEnIkiaz4/e184fEAwCz+6aFrSGoti0pymhnz/ubsQLH23B/oNqRXqf24a/X3ekcSyiEv73zU5MHlmMYf1yAQAvfrwFe2pa8e53u1FW+COuOm2cQVjWm54a/BEIURm7Dvixry6AYFhMEHjqm8N49oPNOG7aAIwdUgAgrp042BQ2Cjy69/Oyk0bjs9X78P2WOjS2RLB+RwMOG15kHI+ggAKfE7urW/D8h5txxlHDMHqQUajbvKcRH6+sQlmhGwNKvKhpDGFbVTM27WlMeD5kRUFdUxgfrdyLj1buNfy2anMdahqCWPSTScj12KEoCu7/zw/YsLNB26es0I2xgwsQCEfxp2dXobohqP32t3+vwamzh+DTVftQ7w9rH2JRUUZYkLQM4wC0iuc+t03T8AhRCYqi4Is1+7DzgB8AUOBz4LjpA7TjeiogJRlU4DlEefTNDdi4qwEzx5Vj3uQKrSCjGb3Ao0BV2esXpnT496dbsXJznWHbsAofrjx1rPY3mVisKqY//Pp6bNjVgL/8/Ajk5ziwu7oFdz69AjPHleHyk8cAAFZsqkW9P4ytVc2YNqqkzTaZTQtRUU4oCipKMiJRSVvI8mOCVCrB78m3N2LFj7X4488OR2m+W5sMiZ8S6WtrKKpOAsVtNjWBt7/dhc17GnHdWRM0IcrMv979ETv2+1GS58bw/rnadkVR8OTbG7Fjvx/DKnIxbkgBpo8pTTh+8X/XwR8QcP81sxOceAGjhoeo+gOxSVGSZdz/nzVobhVQUeyBx8FjS1Uz7nx6hTbGoiTj8bc24uNVVThQH4SiKPj9T6ehJN+NxpYIbn/yO4OjYyAcN4ECQFGeC+cePRz/+Ww7lm6owdINNQltnHNYP1xywijDtn2xxfZAQxAjY5ocs38LQWrYh9AHD0BpOYi78oEW2YmW3WXwVI7H1+sOaFoOM0RjE9FpYmw2YtJK7sPjsHFw2OMCDxG0j5nSHx+vqkK9P6yeV1cMtKzQbdhG2FPTin0HA1obcuFH6KNnAEUCP2wGHDPOxe696/BeUw2Od68Dg3i7WCiQqtZB3LEctuEz8N3Gajzxv4249KTRmDW+PGGB19MUExz05g5ZUfDOt7vwxteqZoFjGUiyAn8wikhUMoTIr9tej/e+24NdB1pw0/9NAqAKKID6rFXXB7F0QzXOnhsXYsi1ctyqwBMRJG2Btop4W7W5Fut21MNuYzWBh5yjJSQY+kVM0ADQr8iD848diZI8F176ZKu2j96cSBb25ZtqsH2/H9+uP5Ag8Lzz3W6s39EAKwp9ThwxvhxjBuVhWD8fWkMidle34ONVe7FxVyPsPIufzB+B/sVePPzfdaiqC+CL1fuwcPYQtASjmrBj51kIoqxpZnZXt6C6IQiHncO584bj+y11WL+zAa98th0AUJTrxE9PGIUHX1+HiCDBHxAMAk8w9v65nTbN9KpAfZ6Jf90584bj+MMHGvqTo6Xh6B1RqVTgOUTZuKsBgbD6lfrxqr1YdO5EjBlckLCf2U4ryTJY1nqBSEZdkzphFeU6IUoyjhhXhtOPHKppZwD9i5Eo8OzY34yoKKOqrlUVeGI+IXrfEDLptMb+DYSjuOfF1ZhaWYxTTF/tQKKGJyolmrSioqxN7jaehTu2sKcyaW2raoYkK9hd3YLSfLchQovgc9uwH/HJMRKVcN/LqzFmUAFOnzM06bkJH6/cC38wip0HWrRF20xto1r1+6A/hOGICzyhiKQJBzWNIXy7vhpelw0TRxolr9agWuG9NShYCjwkDw/xcQDii0ujP4K6pjA4lsFNP5kESVbwu38uR2soChvP4urTx2PL3ia8+91u7Njv1865ZW8zSvLd2FvbglBEgs9jxxWnjEFRngsfr9yLqtpWTBgW1zgdN30gBpXl4NH/bUBzq4DSAjdG9s/F/voAtu/zo7YxCDMkJ0hLQEgo0aBHEYIIffQglJa4X0cOG4b81UOI2n8OoboVE2y7MTRXRLkzhFJ7CELTQRwU7BDClQDyNcHHaePhChxAEetHVLQwyeg0PJFovMYVeT6G9vMBq6AJQqTdThuHmePK4HbwcDl4DOnnQ3mpD7+6/3PsrW2FPxBVNTOIYnrNK0AkALZ4CJxHXQqGYeGwsXg/fBiKZ56Cg80RvL/qAIaWuFHZshQLXOsQ+f5/4IdOx459fihQTUuzxpcb3lHzlzvJUyMIkqZFfu+73fhvTDicPb4cP5k/Atcv/lozVRbrTNhEUCHCXVSUNWH36Kn98cGyPQZBWFEUTdgjz2kkKmmOxRFdO7RnINZ+Mu5q8kX1eW6Nve9kgc6xePbNTuz6xZz4qjS1xH6zyEBOfhs9SH1GinJdGFaRi1ED8zCknw8FBV40NgYgijLyczjk5zgwcUQRDtQH4LBxmsns6Cn98d8vd2hjRdrjddkwckAevt9Sp5n7yBj1L/Jg7qQKzBpfjife3ojvN9fhmKn9cfqRQ+Gwq07wtUIIzQH1fSIEI2o/3A7emMNJN9YFPqNmXQ40Ypi4Da32/WgJlCWMQ09ABZ5DFOKAnOuxozkgYE9Nq6XAk2j6UWBr51NDvpp+cdo4DCn3We7jS1JjSlEUtIbUCU+bYGL/kslRVuKRE+T4rXubsbe2FWFBTCLwtB0ZE5VkMLGJwu3kwXNxW3jSvsau30QmvtgCkOcxanj0/dix34/t+/xoaom0KfDIiqL1O5nDrj48tNU0nuRe2HkWpQVu7K1tRV1TKOEaZAFIlrWW+KLwHANnLBIjHFuIwroxI3295ozxeO+73VgwfSBGDcrHhGGFmDCsEDUNQazcXId1O+q1SA6yaAwqzdGeyfOOGWnZjsqB+fjzz2YgGBa1XDdrth7Eg6+tTZknhGgXAKNDL6CascKfPQGluQaMpwDu0+/AQ//bihkH/4sx9v0If/QQ5gCYkwNABhCM/Qeg1A7U7voaGHmuNg7l0l4Uf/s8bvDZsEQ8N6FNmgBj57VjIlFJEyxKC9yw8Syiooy6xvi9csScaadUFkMJNoGp34RgVS0WMitxwC2jxT8MEUHEuZ6l8EbqwLjz4DruOjC8XTseAEKyDU3hCBSwmDi6H975ciyOdGyGu2k/xJ0rEBbUd1Z77/QCj+kDhdw7Yhqx2zhsjfm3nHTEIJx51DAA6rxzsDmMZpPAE46ZAYlvFrkmzzEojC30IV3VekGUNTOeTxftSfyYZEV9r/WaUPKORnRjTYjPI0YneT3m5KF6oY9sI++9lSmH1Fc79+jhGFiaY/gtlfm5vNBj+FvLcxO7RrMu35e5OGw4GheqAfVD5arTxiEqSoax8XnsqG0KJbSbaHhcTh4sy2jPoxCVNM0u0fQqYgSh9/8Oaf+PGA9gvBfYWXMQilwJpp0fy5mGCjyHKGTB6l/sQXNASJpePlHD0z4/Hn1JgByXLel+OUmyLYciohaN0KJ9fRFtThSKoiAYFkECFogwoE1GSVSp5oXcnHgQUMeInNdp5zVHzmRjEBHiX99k8dYqpev8isxJzMi+6UTBBUJRrU3J7OL67eYQWlKR3uexY2g/H/bWtiYITnqTnVU2VSAewcZzrOY0STQaIaLZ0FXnHjkgL0EbRbZVNwZjAg8Zj3hSynRwxTQcBJJV2LLWT2t80SY5avTtBIDo2g8g7l4NsDxcx14D1p0Hb44HT+w6GjcP24jy5rUIyHbUiW4UllegoLwfWG8hln2/BRPDy1Gw93Mowimq3xkTxoQD74GBAg8rYLa8CsB8w/XIIu+wcwZtGXneSRK3g81hzQejiGuBsvwlBBv2QmqoAiKB+LgCGOkEmn58GmFlMKY4dkEBC/cxV4P1xM0rDl09LfKe5HkdkHknvgiPxgnuHxBZ/iomCSWo9PqxqWUqgMMSfHj06DMRR6IS7DZOW3QHlMT9boije8LCGhsLNa+TpNVhyvXY4XYYnzPAaMojOYkiUUkTvkn/9Is6+QiImLQfQHyOadVM0YlzlvmDRS8AkmdXE3hM85ksxz9YzAku20s+H8YI/gBaW92G9vjcNt1zJMb+JWZQ45JvNombhSgCuS/kHthjAk8kKmv9IQJPZOlLkPb/CDAMwu4y8K01GCJsRviTR+CcfyWAngvUoALPIYj+C568AMn8UswLXnsjtcKCpAkSORZfSwQSCWP24WnRLdhxmznxFVEdhPWLuvkLjQghZrOF2YdHTBKWToQQNTSV0a5rhX5yazZreHSLtzlpHtk3nZBWg/o8SW4L/WRl1piRccnRlfQw76Mfh2QhvkRgtjJp6Z1q08FcukEfAtseFFmCfHA3cvf+gAs9a7ExOgbA4drvQlTSJm4S1QQYyy5I9XsQWfGqun3meeBKVI1bfo4DMlgsy12AC866Br9fslT1NTptKpz9VA3Ixm0lKAtsRhmaIaz9AIIwDOd5voVDbIHi8IGJ+DGd+xFyczXY3LiKX//1HYo9pw3+iK6Uhg15OQ4cbA6jpjEIJyPg2pwPEd0QF3LAMGBzy+AqG4y1tSwq6pchr3UnpkM1JVUPPA4jy0YYxsthjzuf6p10nXYeXwRHY4FvC9iWOgxHHWAHxkT3I7prAAKtbhxm240oOPhNpoom3bMXiUrIQaJ2AUi+sOoFlabWSLw2k9eRoEnUj53Dxmm+WEJUMvjuRAQJOTpLol83N5B2EswfTKkEHlLSw0oA1AvWepOa+jfAMKk/AK1QxAikA5shVm2AtG8DBjdU4RofsC2yEUp0ojZ3ljnDGBHcCs5ZjahQbuirlfmWIDXuw/ToMsh2Bs2BwYbf4j48PMT9P2KwvQEbwj5EBKOGJ7prFaI/fg4AcJ1wI/bL/fD2K2/hUu/nwM6ViG78DLaJC9rV70xCBZ5DEL3QQl6AZNoF81ey2bG3LcgEYrex2gRrRbyelnHx1ZtkyAvtNy3ohn00NbPxOIcp1N0syFlreBQoCtECxDU8yUxafosvPSsfHnPSPLJPOgJPq4VQZUa/vcWs4QnFv1yTZX3W98/KyVa/j41jk35NulLcbz15Ws0dsljEv+pTocgi5LpdEA9shnRgM6TqLUA0jBwAUx3AcLkWinQ6GE6d5ohDLaAuVkT4ISYtRRQQ/vRxQJbAD5oE2+h52v7EYb2pJQJFYbS26iOUHA4b3g1OxKU5X0BY+x5Ol3i47EHIDAdp7rXY+tbTGGPfh8jyV+E69hrtuIggwQYRR9b9G5IoooqdhoPNqumqv7MV8vr3cGZ0BTa4clBbX4Qz3CuQxwbA5BTDMeU0sAX9weaVw+Z0Ij/fg9p3N+J/u3JwTe6nsCkCfhT6gR00N2H8nAYNT9yE47Rz8AfsaDrsApT4N+DrnRI8wX0Yb69C+KPFmAsethx1/w319VDkkWDYxDHWBAqLxVbLo2SqsxTSaW+aWiKGZ8EVE6BDFhoeh53T5YeRDR9BZsdlMkcQYUlvtm81fVRZfaR5XDawDKOal4PRBJ+miE6wFiX1o8xtitrMcdvTCv6QW+oQ3b4CUtU6SNVbATnedwUMJIXBcHYfgm//FaViOW7xbUS/g00AgLFuYMPBHACj4/5kjvg9UBQFctMBiLu/h7hzFeS6nRgFYJQX2LW3CYp4DRhefb5Jf4oC2xFa9hQu5xn8y3YUmgPjIckK3EwErl1fIbz6DQCAbcIJ4PuPha8+gA3R/ng6fCx+PrwKXMmwNvvclVCB5xAkqqsTRRarZCYtsyakvZWVNY2CK/XilcykZdTwGLU35Hf9360W+zQHhYTcPmSSczk4hCKS6rRsoeEhwqFDr+FJIvTpBTEyUZOFMc9g0rLW8KRj0kplTiA06TQ/rebx1H25JssgrBdqk2Xh1YelO0FMDTENT4RMrulqeEzjYWEG1CPV7UR081eIbl9mMOUAAOxuSEXDEazagjw2AHHbUtgq1dDnxpawYVciVDjtHKSDuxBZ+V/IjVVgXD445vzU4E9Bsmw3tkTQHCsvwDKMQShz2nl8FR0Iv6MUvkgNXBAQkO1oHn0G+hUPwv9CkzHKth/izpUIffoYHDPOBevOQ1iQcILrBxSHVG3MIl8t1lfX49qcTRhuq4GwHOgHoJ8L8B/YDZ+jBQoA17wrwJu0NoCqLd0lluBN91koD23D24Eh+LkjUZsQN2mJBhMOmROa80Zh0JRZ+GrHMuxvHYGfeJbicMd22CCgUXIjlw1irPADQu/cC+e8K8B4CtAcEOBhwhAUXvtY0kerETSh3/SBo/fPaWoVDM+ClYZHC+m3cVp+mIggGT6CzKUnWkwmLb2JLB0fHpZhkOO2oTkgwB8QDFrp5oCQIMQ1B4QEgSeVMC+LAiKbv0Fkw2eqEK+D8RSA7z8WXP9xUEpH4c+PfICf53wCb91ODMVOgFcFoZCzGO5wLUb6v4McOiuudbVx6vuzfRnE3auhNOuiGxkOLd6B8Ph3YXBgHYJv/hHuk28F4/AgFBHhZASUb/2fOgZQcLH3S+zf6sQlnh8x3r4X4nfq/WaLBsMx7UwA8Xl9XagU9uPPBcf1bK5jKvAcgogWGp5ki3hnfXiIz4iValgPWXxbg1FtMSF/E6yiIlqCgkEoMquk9cfpIYKc22FDKCKpJi0LHx6Skdlp58FpJi1roa/FYG5Krq0wZwkmpql0NDxW/gJmUmp4dF+uviQho1IaJi29Dw8X+1JNNGmlp+FJHI/ki0J081cIf/FUfIPDA768Elx5JbjyUWALBqAlJOLTJx7Fqe7vEVnzDvgRswCwWq4VwsGmMBjImN38FoKvb1I3Mgyccy4F6zI61xNNTlNrRDNT5nqNX+kOGwcFDFYUnYoTivfh5XUMvmrIw3X9J8HGszgg5ePd0GE42f0DxG1LIe5eA8e0M+BpZTDPuREAEODz4RUbMSP8BWBTkwPa+o/FtmgJCg98Cx+rRiauYCdjvoWwo46n2tad4TxskSYhqIQsTRlEAGloiZvPiEkLMGrsZLB4KTAT4+fMwzurG/BNtQdjbFW4yPs1nAc2I/Cf34AZexx+6vweY+37sFMsQkSYAiCJhsfNgYGSIBzotTHqWMc+GJJoeLTiqAYNj2R47vVCu5plOKbZESTIimLQYrfE/AJbUvjwkHFqDqgCmd80RzWZNK/+gKA5HGsmOotnW26tR2jT52je9AXkIIleZMD1GwV+8BTw/ceCyTXm1TrIl+Ef/uNxy8jN2NWo4NvGYkyaMxdRzoGipX/HQL4ewvf/Q1SYiKn2HZhZ9QmCW/fFL8ry4CrGgB80CfyQKdi6N4IP/vchfur7Gp76vQh99jhcC36JYFjEqa6V4CNNYHwl2BrwYbi0DQOr3sfA2HcJWzgAtpGzYauco2lV3U5eS0PQEowaNKI9ARV4DkGIwMOxjG4RT+bDY1zc25t8UO8zkgoysUiy6oRMHOBaTT48si4RoHr+qGEfMmEZTFoWjstkkvO4eNT7SR6eRA0PqbzsdHDgNZOW9RiYfWeEqM7x1ELD0xIUIMtKu0xafguhKlU7EqK0dBoeck/MflNRg0kriQ8PMWnxrCacpnJaTgXx4QlFRAi6hHvmRUEO+RFe+hIAgB88GbYx88H1G60lxyM4bBy+CVfiWOd6uJurIe5aBdvIww3mFgCoawphlmML+gc3AQwLfth02CecAK5oUEIbiYauOSBoeWHMkzfp70ElD47pM7Fp9VLICBlKp3wUnoCTzzwR7MqXINftROTbFzAfDFhGQX3BYVhdeAI8615Fpf0AVkSGorl8Bi47cRZa1h3Ak5sKsdD9PTjI+D5/hsn1OY6WGToogKyNlgJPbBuJ0nM7+JgTurVPlgIGDXljsSW8CQpC2BAdgPv9J+H2kRsg12yF8sNbGBu7ZUP4g6ipWgVl0LGGKDQ52Izo+o8wed3HGJDrwJctxwCYoLUpqNPwNLbqTFpeu6YxtNLw6AUeQRcqrW8/YJwLFKg19fTm7dZQFMGIaPCf0iM17oew+i1cK67Cl66RaG0dATnYjIWuH8AC+Cg8DvXNYXiYMEbZ9uPHaD/Th1DiB5BUvRXC2vch7v4eJCKB8RTANnoubCNng/UmRs8SfB47qhtyUTPp5/jfp9uwW2jB7Lw8hAQR/wtOxjW+jxDd+BkWMN/C5Q0CIQAsD37IFPW//uPB2PWFc5uxVSzHC+IJuMLxFqQ9PyDy7Ys4CbswzbkNChi4jroMn3/ZiubatzDadRDLgwOwxzMeV595YkL7WIaB1xXXhlGBh9LtkMWK5+Nf50kjj8zh2+01aekydKaC51i4HTyCEREtQUETeEgYNaAu1q26KCVAnaD0i7ooybEIl9SaEDLJkYzIYpI8PPrJui3h0Cw47DsYgIJEB8Uctx0Mo85tLUFB++qTZKXNDNjmJGdW++u/mtXx0jtNkoi5eEHEYEQ09F1qh0nLxrHaYt5Rp2U1C7Ua9VHbGDJkztUTWfoSIATBFg6C85irk4a42mwsIrDhy8goHO9ai8iKV2Ev7IdGv4Ip9h0Ya6vCN5FKtPgFnORaDQBwzDwf9rHJRAjVtEH8NnbFUujne60FHqtQYI5ltS/daO5A5J/6W0Q3fY7I8lfBCkG0yE7UDlsIe5jDy8GZWpj7XJ+aDTrP60Cr4sKLgVkAgAllyRcOIkC2BAXNP8nKf45sa9aVwND3I34/40JIc8CYDb1G8kE+9kY4di9Fy+oP8N1BHxQwOMq5Cfk73kd02lxIsoIi1g/bqhcR2P4NIIngAJRxYZwVeR2RZSHYp58FhmE1cyigmrSaNOHXAScnYyB3EHulQoiSDJ5jDaYaBx9/xvXPMymgSn7TEzE5OEuyouWwUgVVdSzk1gZElr8Ccdt3ABQ4ABzrWo/g6gO4wemHi1XPO92xDU3rNuL3eT/Czkholl04cMANxJKhxn36eER3roKw9j3INdu06/MVY1A442REikdDktv28fHpSjdoflgeOxQF2CqWYxczEIOVPXApQTTJbgQHzsKIeackaDD15wOALUEfHPMvQuSLpxDd8DGmxaYvZtzx4MsrYbetw7OBORiel4ttB5sxujh5iZAct6oN6646iamgAs8hiJY0jmO15H9W0VeyEq8m7bBxiESlLtPwAGqSr2BENKiAjcJMfDLSnz8h9DooGCY2q+zNxBnXHStZICbNtBwPXea1sPS2nZYBaAkSfR6j6YNlGeS47fAHBNT7I4b2y4oCLqXAE99XkhUEdNowgl6lTiLZiA+BPvrEqG4WQPJT64Xa5E7L6ljxHGNYIBVFabdJi4n5whxsDmNvbSsA1cldf7y4azXEbUsBMHAeeXHKfB4sw8Bh4/BleBQW5O0Ammvg/89vMY7Pw+FeNRPtJPsu1Mq5cLNRtLj6watzULY8J8sg12tHY0tES6Nv/lolAoQ52RvZbuNZSILqL8awLOxjjgY/ZCo+eulFfF1fiDM8Ptglo58ReW/yTCH6qcaWaCUUJXV0jnkb+Shx6cK/1ejF+HtR3xzW7m88Y7KIitFzsVmqxGtv/wg7ophk3wVfuAGRFa/iEs9mHGbfA2WLeh62ZBhCQ+fhhy8+xXTHDgg/vAs4PHBMPAkhnfDRHNPwsJBRUr8C4vKPsCi3AV+FKxEW5sDrYrUPklnCV+j/+XrMcUzAN/7RKGGbMd+1ATuixYgI8RxO5kU3rEvkSDhQHzCMo7D2fURWvg6I6rH84ClYEyjGoJrPkSPUAyywVyyAjVVQxjbC07IeYICwwiOXDSF369MIO/bBMfV0tLSGMc2+HUfteQfhrfXqBVkethEzYRu/AI6SAfDkeyA0BoA0Pi7jNeUEQ/0vct/fUebgV+Nq8PY2Hu/vL8QVR41PKuwAcYEnKsoQB82GbcI+RNe+j7XCAHwUGo/bDz8LQNzRnyQ9TOWy4PPYgLr0iu92NVTgOQTRJ41LpbXQq3rdTl4VeNrpw+NP04cHUCfcmgbjom4WZvbVtRr+Npu0AKC6IWSoJGz1opk1PGqmZXUbKUJISksA6qLFtcNpGVDT+wPW9vpcjyrw7K01VhKXZQWp/PrME3ZzQEgQePymcPWWUFQTeMjYkorGOW6bwTkUMDktt+HDo4alx1IbyEpMw0ZMWulPL0TgIUJinscBhmEgNVRBWPGamhcHgG3M0VqoeCocNhb+qBMtR92Cgi1vQNy1Cp5oA4KyHbukEoyxVaGMa4asAHsGLUS/FMUyCfk5jtQCTywjJ8nuq3eoBdSx0qdpAADW5cPXyiTskwJqHh6zEBJ7dvISrpVc4OE5Fh4nr2UoTra/WeuTY6HhMUc4kdIcHMugOM+F6oag+kFR5NEEbQE2fBCagLM9y4GNH2JSrOncgAmwTzwJXNlIcFEJL7wrYJdYjHM8yyCseA182Uj0F/fgSO967JMKsK5lIoojdfhF7nfIWdusJRg80rkZkd1r4R01BRFBwkDuIMaEVgEAzvSswBRxJ/pzDeAZGdPt27G2aTyA/gAS54KIICVEbO4/qKrXctx2iFXrtWKrXOkIOGaeD654MJqW7cZLW3KwsGArNrd4sc1WiXynHf2bV2GE4yC+CQ7DfqYMJ9iWYbZzC6LrP4K4fRmOj/Ao8DYAEQAOD+xjjoZt7Hyw7ryk9zMV5Pmorg/o/LBsWj8Pii44Z56PjVtWQEZLyucGiIX32zktcq9sxk/gH34CnnpyFRw2TvtAJs8OMRN7UoTYJ0t/0RNQgecQRDQ4nCZPpqf333E5eDS2RFJmGbaCmKS8aQk8iZFaZqfbfXXGqJzWUKLAQ4oTEiydljWBR30FoqKiab5cDl5Ti0d05hm9+c/KlEReaLuNhRCVsZcs3hbRRrkeO/YC2BPTaNggQgTbZiZr86ThDwioKIpnYFWUuE8QEdxag1GU5lsngfS57YavQ8BY0TtZ4sG40GzUxISEeNI3fQhsW5CJm2h48j0sIstfhfDDe4AiAQwDW+WRcBx+TlrnU79AowjZfHAddy2U/evxwUfL8FZ1P5SUFOGbhg04ybUGq4QhGJiX6LNjBTFhESEgmQ9PWBAhROMZgIngR0x/ZtOpNl52PlEIib03xL+GvH9tCZM+j90o8NgTBbpkwlXcaVkymJgAoKpWfbe8sSg/fSVsfdLBpZEROKFwFzyRg1gZHoylymG47YRT49e2qyUKvomMxBmjJPB7VyL47r34uUc91xjsxzxlI/ic2Fg5c+Caeiq++GIVpnM/gl/+LJTBlQhHRJzuXgEAkPMGQG7ch8G8Wg7ELzvhY8MYtvdNKPJ0MCzfpkkLiGt4cl28JuzYRs+DY/ZF2jvvc9vRqrjw78bDIMkK+ufakeN14uvaUfg6NgzDKnx4Zd8MBIvH4Xj2W8jN1SgAEJTtiI5agIpZJ4Oxdc6nRXtvYh+CLgcPG88lmFcj0fS1rj6PHWFBzbZcVuBGSFSfHaINB2CopwWkzinkTVE2qLvp2RgxSo8gWkTYWJm0tNT7PAu+DV+fZGgahTRNWoApU3DseCJakC9Msni0BAVtH/L1kiDwWHxZJJi0dD48RKUf1Wl4nA7OUPvLahyIoEEqOZPFW++LIrc2IPTxElzQ8gQWOH9AfXUNTnStxt35L+My7+cGzZQVRMNDBDVz8kE1GZp6DlIQltyDsCBp956YSqxC0/UmLatsxfrElTzPgmUZ2G1xPx6i4XGlqeGRGqpwmLIRbiaCPTUtGMZX4wLhZQhr3laLXQ6eDM/Zf4ZzzqVpLxBEcCD3zzZwAr4UxiKkONC/xIP10YH4q38hPg6PT9v0ZhZckws8kmbWYgCtcCjxBzELPHotYqKZSb0/DMMYzFqpcloBRq2i/sNGT7Jraf2IiEmFAb0PGHl2iIaHZRhI4LBs4KU4eNyf8HxgNpptiVVy1TYyOFh5FhhfCSAKkBQG34ZHYGe0GDwjQ1aAZeIoeM/9C5wTjsOX3CzUSD6w4WYE3/krhlW/j6G2OoiMDcrcq/GA/wQsiwzDEy3z8JfmhWiVHciJ1ED44X0A6rvgZsI40vEj8tiAITs6gcwfY7EJckMVYHfDMe1MwwcO6XvcudmeoMkdWKKWjdgsVsB91h/hmHkBPhIm4a7m08GOO77Twk58DIGq2IcgaZc+i7bRzNz2O2mOmgyG43W0COZnJy0Nj0Vdse6GangOQQwhxWmYtOw2Tsuf0JU+PMSHwMqkVZzvQm1jSDNplRe4sae2FS3BqPYlW1boxu7qFk0oIk7QKTU8uhdVW6iJwJPEaRlQx0GflV2SZS3j6IASL3bs92tCFVmohE1fqI630TBcAE50/wCEfwBiQRKjbfshCVHAaT15yLq6YhXFXmzZ26RVQyYQh2WPk0dejgNVdQFNy6YlgeTjSSC1/EdJTFpWGh69BsgWey6cdh5CVEA4Imp+GG0JEoosQ/jhXQir/otJsoQxeTx2REsw2r0fkADGnQfHrAthGzIl5Xms0CJ2SISRomhh6UQgJZhraSUjL8du+tvah0fVGsSSGto5LYqNjJU58i1eZJRLGDN98co8r5ptGWh7bPXvmz6TtFV7CeT9S2XS0kcvmWtKEQ1PUa4TtU0hhEQOYcURa0Nie0ndpmaBxbATFqH5h09x7zIHGtgC2GQGhcEaRMGBKxiAYxyqFtPmcuH56tm4vuBjoH4vhmAvAGBv0UyMyC1ClVSoOXYDwH+DU3Gh9xsIK1+HEmyE72AObs39ALlsCLPErWgJT014xmubQnAyAsb7v1bbPvkUME7jM2P+gNPntSIMLFWP8QcEMBwPZtQ8vP22ei/SLZvSFuSaWgFV7R7qzcxxgactQRlITIxKIudcOg2P+Txms7pVG6mGh9IjiCKpdM2kjNIiX/cOG9tmDhor0slnocecfFCWFU2I6BdzYibamn7FsbwWAUFLVtavUNVo7I99hfaP7RMIRRParYWl64QLch6SIViUFG0ydNo54OBOTLdvA6AknK81GFUjsgCDiQlQI0ykhipEvnwaiIbBlg7Hjv6noEZSnQcPSl5EFB48I0Nq2p90fILheF2xCtJ/0yRColryvA4t2aM5mZr+XuhDmAl6bZ+lwKP7nWjarBbJVIuy7K9F8K0/Q1jxKiBLEGw5cDAiRtvV/u/PnwLP2X/qkLADwJCThbRLqxhtEniSCQRmzBqdhCgt3Ve12X8HgKYF02t49E7BTgeXVAgBjI7Lbfli6BffZPchQcOTYNISNWGMNZlvfR69hidWzDYm8JTkqxJ8JColOG7r0SecZHNL0VJ5MqqlPLjsPPJynNgjFeGAlG/ot8vBY49UhO3TboF92plo5XKxVyxAbfmchP4wAFYKQ7HdNQ5QZEQ3fIIjGt5ALqsGPpTzTcjd9SnESAgXeb7ELb7/YYZjKyrYetzgexdOqRVMTjFsY49JOb6AKgAlaHhihUHJu6U3NbvTTMrZFuZrahoenQkzLIhplZYwnyOu4THW0QLiJi1C6jqJiR+yPQXV8ByC6E1aWuSRhUlLr+EhJq32lJaIROMOmumYtMiLRqKqAuGoZiMuL3RjTTx6UxMqyPkZBijNVwUe8nKXF3mwdV8zFEUVSPS5cLTEg7qvllDMl4JoeESdhsfFRBF692843xtEk9+dkBWZCGJety1hYcz12CGs+g8ABfygSXAedy1CP9biobW5GMDVY59UgCtzPsYIWw3QsBcYONxyfMgXl9vBoyB2jWQanlyvPWGiadHamJgEUa/h0efhsXJa1i/YRGA2CjzJnZYVRUF085eIfPsiIEYAmxPOWRdgG0biw/99iAn2PfheGIIjDz8KlQ5PwvHpYhZ4yOTttHMoznMa9k3XuVov4HicfIJmyKnTDJJFQi9sEA2PMWQ6Pr4Om9GkxTKMQQOpf37b0vDoBaVkGiwbz2rpEYD4Bwc5d0gnvBKtDSHHZczUrSiKZtIqzXdj/c4GVeCJJl9ofaaacpqg7OCR53Vo/nq5JoEHAFoVFxyTTsEr2wdhzbaDuMTlAssyBj+nfJ8DDf4IvnAei3FHn4jIdy9Brt+LL8OV2CcV4v8836K46lPMZ1ejwKH6/fwfv1S7lmDPRd4xV4HhEhdz8wdcjk4ABFSBgMxH4ZhjtD6/VKrUE+0hmcDDsayW6qE1FNU+lNIx38Y1POpcYi4cCrTPpKXl+6IaHkpPIKZp0oroTVps+zU8mhMvn7qOFsGc+ZdoJlwOPsF8UJLv1toEqJoa81dXrseufXmYc/EIuomYlIwg/XUbfHjU/np3fwUIavTGZPuuBNOePgeGuSRCkXwQ4s6VABjYp50JhmGR61GLUe6WiiGCwz4pllyscW/S8dFraMwV1wn6SVVL3hhMoeFxJxZxNFZLT7zfepMombgNWoEkTsuKLCPyzXOIfPkvQIyAK6+E56y7YBs5G7leJzaL/fBKcAa2i6UJ97K92E0V0/W5ZszmVXuaGh79M5if40z4Xb8IkOdB/9zbLDQ8ZKz4WIoI/Tly3DaDZkWv6WhT4ElDw8PEwvfNx+iFV5JEsjjPCf0SrZq04hrZkC6Xk6bh0fkyWbUh12ROjS+sHPIMtefi404+UMw5gkg/9Nq6Ip96jyJRCXzFGLjPuBN/ky/Ga8HDsc05FhuECrCKhALpIFpkJ5bz09Asq23fKFSgesYN4IqHWI4diYTTxs5k0sr12uHS+f35A4L2ceLzdN53h2B+lvVtIGOiz7zeHg0P0dwR4b3DJi3qtEzpSfQhxURosKrjJOhy8HTEhydVxWErzF8C+ogis4bI57YZIr9y3LaEl06vdjerU8lCbudZgzMyADhj6mBRlBERRNgRhX3rJ9rvE+x7IEUTo6XUdiWqtvN2fgAA4IdOA1eghsea99kvxhJ3NVUhGYayEEmqTeurS5Nxj5u04kkHCZZOy21oeIiWz6ZTa2tagYgY14rpNCeKGEH4o4cQ3fgpAAb26WfDdfItYHNUZ1bzeLS3UrqZRA1PPMOtPvOxft+20DstW2WM5bm4iZhoO/QmLSsNj1kg0AsG5sVMf/22PiCsFj4r9OeJOy3rTVpEE2p63zxGp+XG1rj2kTx3bZm0iIaHPHtaDTa78QPHSsND9jXX6dJfpzDXFetHzIGcYVEXUu9BcZ4brwQPR5j3oY4pwgP+E7C9eC7uajod9zafhMdbj4Y3L3mGY8A4xub3Ps+rplWIm4yjadXRai82njVoXqwEXVJbz25j0ypYmmv6CLLS8Nj59AUe8hwLUTlpItPuggo8hyBkweJYJh6WbiHIkIfTbms7I7MVViaUVJCXVY00ijsBe3UOkvp99Qu312VLEKx8brulUy4QN2k5bMbF71jnWpy0+16MtVVpGp5Zzi1gIq2qn4HsgocVoFT/aOxrQK99ibdrsn0nuH0/AAwD+9TTtO1mp8UqSRV4uKYqKEkitfSVzs2OhQTiOJrnscMbGx/NadkkgCqyrPkzdMRp2aZz4tYKTurOQ7bJ4RYE375HzaXD8XAecxUcE08Cw8TH3Uo71xn0ZQYAYxFXhmEMz1M62kdAXWxJn6wEHoaJJ2EkpkV9AVXynAmiGhH4464G7bkh7bUbNC7G51kv8DhT5S6A0YScSuAhAhnPMXDFNHJakU6TP5b+HuW47IYvd70pVT/2Vr5MBPPCSpzdXTGTllW/40kRTVm9NQ1P/DrEdEmEoqgoa2brolwnGmUvPh90NZ5mzka9nIOiXCei4FElFUIB0+aHmv4ZNZu0iJbXp5t/4oVQMyfwAMZ3J9edKPCQZz9d021cEFXvKfFt1Jv/9Zo0jmVSahz1Hxg9reWhAs8hiLZg8amdkfVCAdHwtCcPT6qKw1a4nbymwm8JRrUF3uuyGaJVAPWrIceg4bEnCFYGwcD0omkanqgfbi6msmUEHONaD06R8H+eb8EJrXCILTjauQEA4Jx8CjbIgwEA7J5VhvMR5+F8F8DWbsYgZwtmOzbhQs9XAGIJ8/L6xfvq4DVTGgBUS3mQFAZMNAgl0GA5PvqINzJp+mP1uAiapimFD4/PY4cSCSD09l+Q/+HtuC33v5gnfwvBr17XqOFJYdIyaHjUyVAfmmzjWdU5+c0/Qq7dDjg8cJ10M2xDpyWc08abTASdFXhMeUj0CzJgjmJKP18QEXSS1QRymAQ//bntWli6hLe/3YV7X16D+//zA4C4kKGatuK5XvTkdtCklUqgI+1Ty50YzZMRXR4ep503FcCNm3CEqIyaWAb0PK8jXrU8GjdppdbwqPcmrAsa0Jvv9NclCTSJKcus4dELjIW5zti+6j7k/eFYRrt/ITGuyS70Gc2U6RY8BlSNs8cZz9VF2q+PUNJrgTOJVWFiIPFZtBI6Lc+nCaJqWRpLp2XdubwuW0qfJIZhEuainqJHBJ4DBw7ghhtuwKxZszBt2jRcdtll2Lp1q2Gf4447DpWVlYb/br31Vu33xsZGLFq0CNOmTcP06dNx5513IhQKmS9FscAyD08qHx6e61AenvZEaAHQMv+qx8ZLRphNWhzLwO3kDee11PB4rDU8oiRDkhWUsM3g3/4truDfhANRHOHYAiejvtw5bBjzQh/gF+734GPDYPPKYB85ExtlNcsvf+AHKJKuRk8gigK2FUfvewKhd+7BDe7/4mzPcrAMYBszH46Z5xvapqq7433yuJ2okXIBAHL9Hsvx0Veez3HbwCBWj0uXeDFeXVpn0jIJPHl8BMG374FUvQUAUMK1YK5zI+o+eU4bH4IQlRI0Tvo6WgRNfR4L/XbaOSjBJgTf+guU5how3kK4F94GvmykZd+A+Fex12VLMDO2F3uCSctYLyru3Mm061pkUSzwJRF4NL8JdRwMPjy6xIMkkjCeSJBLOEenTFq6Y1OF3ZPzJPP50Tt7GzQ8brWqOvF/+mKNWoE7P8dhyAGTKjrIvLBqRWcTNDyJJi2iDYoIRpOgQyeEF5lMWprG2WUzCHXkGSH7k3Gx8anHWH9/iMBIxii/hzQ8hvuY4llM53yipGrENJOWLqJVfz/TSSprjsDtKbpd4BEEAVdccQXq6urw6KOP4sUXX4TH48HFF1+Mhgb1CzMYDGLv3r147LHH8PXXX2v/3Xbbbdp5rrvuOuzevRtPP/00/vGPf+CLL77A73//++7uTp8kapl4MHmmZUNZhXYJPO3z4VH3jb8YrbrIJ6/LpjlNemPOnHqTVk7sC0v/naH68CQ6zJF+zXX+CEhRFDFNONuzDEc5VTNVfcVsSAqDQfIeFHMtOCh5UXTObWA4HvvYMjTLLrDREMKfPY7olq8h1e0E27IfV+d8CGe0GXB4EIEdgsJhhW06HLMuMJhvCOTLjET8VMUcl6VkAk8oruHhWDaewVQnzGm+Kt6403IwopoIW4ICPEwYlZuegly/G4zLB9fC2/C/yAwAQGTbSiiSaHgWFCQmyhMtNTzqBNhINCkOCaH374cSaACbWwb3ab8Fl98PqSDjkYkFwZlE4CEOsOSZbI92BwBOnT0Ex0zpj6mVJdbXJZoui69qvcBDzAyzJ5Rj1MA8HD2pv+4cRAgxvjceJ49CnyNB+LBCn8Qw1Zc92ccgIMWit4C4idTp4IwmHLfNcNyemlbYeBbzJlVo1xN0WYwtNTyGhVVEKGxt0tI7LSf14SEmQd11imIaHlLqQ68hdVhooQp8Dm3+aKvYsb79+mzj5mdY7+dkVSk9EyQTeBwmrWu6CTbtNk4zbza1RuJOy7ogBP04e5PkDTO0UUs+2LMCT7eHpa9cuRJbtmzBl19+idLSUgDAvffei8MPPxyffvopzjrrLGzbtg2yLGPSpEnIzc1NOMfq1auxfPlyvPvuuxg2bBgA4A9/+AMuv/xy3HDDDdp5KdaIuuKhmjOyhUlLn2mZmE3aZ9KKO9mmCyk01xKIxstSuGxgWQZetw0twaj28pg1PBzLwh2rIcSxaq4Lc8QBoJrq3EwE0xzbtW3THDsAAE2yCy2Vp+C7ra04yb0GNZIPS/zH4dmicjQ1BcGyPFZGhmC+ayPEHcsh7lgOADgTADgg6ipE3hm348WP92HFj9WYOroMRydR96oTeQtyvQ5IsoL9EdWPRz6YRODRslbbYsfb0RKMojkQwQB4sf9gQPNRyGWC4Kv3oIxtQq3sQyAURSQYwJU5n8AerAfjyYf7pJvB5pVjo6MO8+Q1yBGCEPdvQlQyai8iUcmgJYjqykoAaqi5l1Un86bWCDhI+An/BeT6KlWoOmFRWrWCtMUiAwtCokmL+PDEFqLYM5TuVy9hWEUuhlUkzkmEBL8Jh7XAQwSJORP6YXh/4/nsSTQ8DMPgtxdPgyjJaUbb2FDXJKU2aVkIVwzDwGVXk3Y2aho7HkxMHCDvlnqcWgPNxrO47qwJGFaRq4WvR6K6xJ0W7VUXVh6hiKjm0xLiC2uBz4EZY0rhchrLbeijtISoFM9hFFvcybi4HJwm8MuKEhP44xpSTSAWRC05pcvBa/NHOnNWrqYtjJt05k2uwNdrD2Dc0EL1N4Ofk1HozhSkHeZM3Zq2kZhX2/Gsl+S5sbumBdUNQYQiJNOyTsOj+9hJ5bBM6C2RWt0u8IwYMQKPP/64QShhY46zfr9alG/z5s0oKiqyFHYAVWgqLi7WhB0AmD59OhiGwapVq3DiiSd2qG08n3mFFxEouE6q6DMJyclgt7Pa5CrJSkL/iXDjcvCaZkdRjOOUqn/EJJXndaQ9tmQyaA3HMyj7POrxvtgCn+uxg+dZ5JoiOcg+gbAIn8cOLtyEYrYFDghoCUS0NkiygiMcW2FnJHBFA/FdUwmmiSsBAF+FR+FwtxMfhsdjq1iOfWIeWLsTDMOAi4XxvxOahMmzZqBMPgCpZgek5moowWbsF/Pgnn0dSnILUZTXABksivJcSftOIlHyvHY0twrYJ6oaHrlhr+Ux2njmqOOR61UzKe+qbgHDMHjkjfUAgHFlLOT3/oxIoBG/zgNEhYXyzue4hGlFGdcExeGFb+EtmsbF53FiXX1/zHRug7jre8jKEWAgg4eMKHhIpntORF57zAcs+NlTOHzTVwi7R+HNwFT8xLMUg1EF8A54T1oEviC9D5D4eKT/vCSDaAIEUQbPs9pEmxsbO3Itp53L6HtPBJx4SgWbdn6y4Iiyoi1+hbnOhOvneR04UB9ESX7is1OQmxgOTzC/iz6PA3VNYbgcfNI+EkHBPOZOB4dgRNTa6XHympCT47bBFps3po8uRWNLBJefPAbjhxVq+wIxp2WS6sFps2xDfo5DE3jIvh6nev6rzhif0D9NwxOV4A/FS8rkxIQO0p8cl92QG0aUFQRiC3eux66FWLeGRS3Xl8dlQ447Pn+09VwU56kmsEJf/B7Om9wf8ybHtXV5MRPoqs112hxalJd4z/V91P+bDsQfKdfUZrdWfiYeQZfus96/xIvdNS04UB/QTFo5nvg9dOvGNp2xyouZgIWo3KNrYrcLPMXFxTjqqKMM25577jmEw2HMmqWmBN+8eTPcbjeuu+46fP/998jPz8eZZ56Jiy66CCzLoqamBuXl5YZz2O125OXl4cCBAx1qF8syyM/veKKztvD5XG3v1E1wMdu01+NAQSw5lqIgof9K7Ksl1+eEEjN92ey85ThZ9Y+8KOWlOWmPbXEsW7IgKZqdvrzEi/x8DwpzXdhXF0BRvhv5+R6U6TLmlpf4kJ/vQV6OEwfqg5ji3ofm559ChSLjngIgItgQeO092Ioq4MgfgSMdmwAABTNOwbpveHhqDiCfbcUKaTROyPcAYLBTVEOm88jXrM8Fh52HBA7s0GmoiJk1FEXBube+gZDI4MnhQ5Gf78aZ8ythd9hw8uwhyI+NsZnSWPLE4nwPghEJ+2KRWrK/FrkugHUax4z46lSU5SI/34PSQg827GzA61/s0PYZN9CLK93vQaptBOv0IhwOw86IQHMVyjggrNhQcfZvkDtohHZMYb4L66oHYqZzG6I7v4erYhJu9r0NLxvGQ/4FcLrshvvniH3puZw2sJs+grBJdcw+yrkJo2z7Ucr5IYNBvzMXwT18XFr3HQCOOKwCX/6wHzMPq+j0u1gYG3NJUZCf79EEkH6x56SsWM2C63bZMvre53qNAklBnks7f26O+o4EIqJmVh48ID/Bx+bqsydi/Y56zJo0IK0wYjPkXSwpcGP7vmYU5LmT9nHK6FIs21iDqWPLDft4XHY0+CPaO1xU4NECB8oKPdq+5584BuedMNrgtOp0q4ubgngyz8IC6zb0K1Y1kwFB1or3FhV4krbXFatkLogyojFJpSjPhYICdS7wEaHZ50BhoRd2GwchKsHhcoD43xcVuFFUoJ5fnwyvtMSHfJ8T1Q1BFOcnHzPCrMluNAQEjBtWlHTf/qVqNnVJVsBzLM6cNxzDBxe2mXiwPevFqKFFAIBB5T5DO4iwRaJdfV5n2s/6yEH5+GbdAVQ3hTWzZL/SXO1DwadzbShKcm/1LDxqOEKCjONnDdH61hNrYsYFnqqqKsyfPz/p70uXLkVBQTy/wUcffYS//e1vuOSSS1BZWQkA2Lp1K/x+PxYsWICrr74aq1atwr333ovm5mb88pe/RCgUgt2eqHJ0OByIRCIJ29NBlhX4/cEOHZsKjmPh87ng94cssxn3BK0xW7IYlRBoVWvzREUZjY3GopvE3ipLEsTY11cgGDHsl6p/jX71OqyceO5kOGK+QrUNATT51bYxseNJyQeHjUVjYwCszgzHyBIaGwNwOzgM4A7ihOiHAGQoLA9GFuFgohBqdkKo2Qnga+RzQEBxIq/fJCjYgMda1WfW4+QRDhnVriSNut8fAqkP3NQcQmNjAJIs440vd2oVhRVRRGNjADyA02cPBqAk7fvEoQVYOygfM8eWYHd1MwKKE1F3MWzBOtR+/wUco+MfBrKixLMqS2pf544rhFBXBaGpFvZIE8aWAuPcdZBqdoJxeuE98/d46L+7cXB/Fc6d7Mam9ZuwXeqHu3IqDG1y2Tgsi5ZDZB1AawMm7/gncvgmAMBPvV+grmYOvLpU9c1+1WQxMLINDZ+9BQBoLZ8K5/7vUcqpWtrlOcfihMJRiKR53wFgSIkHSxYdBZZh0n5ekhGNmUcCoSiqa/2aaYs8J/3ynbDbWAzv5+v0tfQwMPq4yaKknV+Mqm3aH6sH53byCLSGYb66z8lh5pgSNDe3bz4yv4vzJvaDJEoYMzA3aR+njizC4zfPBc+xhn3svHFBFqMiynJz8LNTxmBoG2OmL4DbEHtWREG0PCY/ZmLctb9Ji9bSj5m5f0RrEQxFsXtfEwBVQ6rtH1uIXXYOjY0BOGwshKiE2roW1MYcxR0cg6hAymGo75SNZ+FvDmo5uOx8es/gURPUD+9k+5bmOjB2SAEKchw4bc5QFOe50NSU/L52ZL0o8dlx20VT0K/IY2iHYnJTYFLMRWYKYvdl4456LRO3EBbQKMYDNWwci6gkg2eS95/g4hhccry6vvv9oYyuiT6fK21tUcYFntLSUrz77rtJf9ebqV566SXcddddWLhwIW6++WZt+xNPPIFIJIKcHPUrrLKyEq2trXjkkUdw7bXXwul0QhASbYGRSARut/XXdDqIYtcJJJIkd+n52wPJTcIyIOs3JDmxfcQxkGdZzZlPiFr3Q9+/7fua8dHKvWiICSwuB59234mz4qpNdXGTml09fmCJF8s21mBAsQeiKBvCJN0te9H61ds4InQQpTkbwEMEN2A8+PnX4tr7P0c+G8DJ492YUhRAcPsacM178S07FWeAN2Rs5jkW5m8vYguXJBlc7MtMiEoIhaO456XV2L5PXeSPmzYAHMOk3dfSfDdu+r9JAOK1iprLpqFox7sIr/8M3IgjtX331bXGFhIF/Lo30bTpC+SGW3A2AHAA3ABaALkFAMvDedx1UDxF8LoPYIucg28aC7E2zCE/xxFzSo4vSh6nqrWq8wxHecsG5ETrEVF4CAqPfnwTgqteQrTsau2rNCxIYCFjdvBjAApsY46Gf+Ap+OfmQpzmXolVwhBEKyZ2+HmXTUJDRyBRhRFBQnPMD4VjGdh5FqIoo9DnxEO/nANb7O9MYa4xZOPi5yfPWV3MxyXXY++SOYG8i8MrcjE85m/U1nXMv5t9hGwcC0lScMTYsrTOZ7exEKKy5jvHc9bjXBjTGNQ0hBCKmbDtKe6JFqUVEXGwSZ1f8rzxcSTj73HaIIqqr1MLomgNRbUIQo/TFi+pExOQHDYOoiijLKYZLM1zZ+TecCyDRedO1P5O95ztXS+G9Uu8z3aTEJBqXM2UF6jjQHy47Lw6LxrOb1MFHnc75nc9PbEmZlzgsdlsBt+aZNx777148skn8dOf/hS33HKLQcVnt9sTNDgjR45EMBhEc3MzysrK8PHHHxt+FwQBTU1NKCmxjp6gxDE4LaeokaUvv5Bu4sH9BwP483OrtCXrsGGFWrREOkwZWYxP+/mwfb9f20aikY4/fCCmjirRzkdU7PlsKxxfLUZUCGIEALBAi6MU5fOvAmO3Y9bEgfh8zX788wfgg+IKHDPlcDzz/iYMLvPhDBgzBuvzoBD0zn76cfhhWz227/PDaedw0fGVmDGmLO1+miGmi+bSqSja9QHkuh2QDu5Gq7MMT769ERt2NQIA5uXshLjm6/iBNidYbxGYnEKw3kIw3kLwAyaAKxygjl3M1r52ez0AY2izuX9VrpEob1FzDr3QOgutigNX53wE9/6VENa8A8ekkwGok95YWxVcShCMywfHEefBWRfCdrEUf/OfBACYn2aSs65Cn2lZn39InWcSM0VnCnNyN0MtLZJ4MGZbsboXvYVU/UgHh42DEJXjNZySOFkTP5iDTSFD4sFk6H2zSOX4Al2ZjymVxdi0uxFHxjQv+vDzhtjiXeBzJDjwkudl4ewhGDe0ECMHJHdM7ys4HR2/h/k5DrgcXLy+oDPxnjjsHAJhMWUdrd5Gj8xKRNi55ZZbcOmllxp+UxQFxx57LE477TRcc8012vZ169ahuLgY+fn5mDZtGu677z7s3r0bgwYNAgAsX65Gy0yZMqX7OtJHIVI1z7MpS0aQpHN2W7z8AlFBfvZ9FaKSghNmDDQcc6A+CAVqltOrTx+vVQxOF7uNw3VnTcCfn1uFmsYQGMSdIBmG0SZIQI20mT+pH2bVvAiEg2AL+gMVE3DAL6PfjAVg7Oq+Fy6oxMiBeXjp463YVxfAq59vB8BovhP6fDI2nk3Iv6H/2tUnYCTRSpUD8jol7ABxjYTAe8APngJxx3JEf/wM33uOwYZdjWAAzB4g4tTQMkAG7FNOg33csWDaKLA5ZnA+vvphP3weO0YMyMOCaQMS9iEJ8Xbbh2PWmLn4cq8NPzSoC8abwSk4w7MSwopXwXA22CcsgCjJONyhVnK1jZwNhuMTJtP2LpCZRp/8jvjvdDaZYTqYNSOGPDymL+68DOdjySSJ97N9SwXRrGh/J3keyPtc1xTS3idXimdHn+33QMxEpc+J1L/Yi1vOn6xrdzwnENE4F/qcCQKYVprCxmH0oPw2etc3SHgW25GCgWEYVBR5sW1fMwBYVncvzXej0R9Bv8KOW1W6m24XeJYtW4Ynn3wSF154IU455RTU1dVpv7ndbng8Hhx77LF46qmnMHToUIwbNw5Lly7Fk08+qeXhOeywwzB58mRcf/31+P3vf49gMIg77rgDp512Gg1JTwOrxIOyokBRFIOmzZhpOa4JiggSnv9oCxRFnVDOPKZSO4Y4AfYv9rZb2CHkuO244dyJuP/fa1Be6NHKX+hRomHITdU407ca0d17AJsTrmOvBZtbCnOtcYZhMGNMGfoXe3Hnv1Zo0V8kaRpv0vDYzBoeg8ATz1ukL67aWVid5sg2Zp4q8Gz7DoWFHM5278XQXBEVUh0UOQpuwATYJy+0zO1jZvroUkwaUWQo9GmGaB7CIuCZeyk2/XsNALV69BeRMZhZmYuyqk8Q+e4lgOPAhEowxqYmmuMrZwOwWiB7VuCJlzeQdYVdu16jkkpQMBcpNReZ7U109n6mu9gWxco/kHcSSNRM6LHxarFfUVKw/6Aq8FgVctWuq2UbjmjXKPA5DUVZ1fb1nijaTGEWMtsrtFYUe1IKPFefPh5NrRGUJAnK6I10u8Dz9ttvA1Ajs5577jnDb9dccw2uvfZaLFq0CF6vF/fffz+qq6vRv39/3HbbbTjnnHMAqAvY4sWLceedd+Liiy+Gw+HA8ccfj1//+tfd3Z0+CRF4bLpq6QCJJNAJPLpMy1rNLVlGWBA1R7bnP9yMw0aVotCU4K89uXesKM5z4U9XzEiYmABAqtmG4Hv3a9XLAcA56wKwuamF3f7FXhw3bQDeW6bmuXHw1hoec+Zd/WTN68ZBMCU96wxE4JFlBVz5KDC5ZVCaqzGk+hMMcQKIqIYYJqcYrnlXpCXsENrKGGvXTC2xmkMmR8KqsnkYWOSCsOZtRL55HmNcA8ExChoc/ZETK5dhnkxTmSW6A/0CRr7su0PDYxYMUmp4uqE9HcV8P9ubr8j8EZDseKedh89tg19XcqAt4cpp59EaiuoEmOSCI9HkHIhFd7kdPFwOHrKiQOfCmJF3uLdh1mK1V2itKIprj61MWm4nb9C49QW6vbV33XUX7rrrrpT78DyPq6++GldffXXSfQoLC/Hggw9munmHBCT8k9dVSwdUrYV+bRR0Ji2970pE52gmSgr++uwK/OGy6eBZVit/YM4S2xGshB1FFBD6/ElACIJxeMHmlYMfMhX8iFlpnXPhrCFY/mMN6v0R2Cw0PDaOScgpYfDh0Wm69OPTWfROlAzDwDnrQgjrPsDuRgXr6xj0HzQA06aOBlc6Aowts5oBs28JMW+yDANZURCJyrBPPxOKFEV03QfID6kC477ciRgUO4fdpmbnJYJwT2t49AtufSxasFtMWuavan2mZdMClJekHldvwFzqwupdTIVe4GSQ6MytpzjPpQk8evN5MlwOXjNTAkBBinEk92PfQTUyriAWqs0yDOx2LmXpi75Ogp9SewUeXdoPKw1PXyQ7ekFpF5pJS1ctHSDZluMvhd6kxetMOfrMpE47h+r6IDbsbMBhw4riGh5X1ywuwqo3oDRXg3HnwXP2n9r0YTHjsHO49MTR+Nd7mzB5hJpnRz/B8jwLlmHAsUw8giOJ03JXmbQAgO8/Fnz/sVjx/iZ8vmc/TisaAr7/kE5fxwqiASL3mzwfbqe6sAiiBIZh4JjxE0AWEd3wCcIKj7rcsdo5SKVw4uTYXvV5pmEZNSJLEGXUN3ejhifFV7VZw5PpEgOZRN/ujgiv+vvvsHMp884U57m0IAVXGs+NQRizcym1ieTdJeYvfQCFw6YTeHpYQO8KOmuWrCiOz63uNMpH9AWyz3BJaROj03J8IhJNEVhxDUbcpCXKMiIiyYjKa2Gv9TGzgVavJgMaHj2KLCK6bSmEte8BAJyzL263sEMYPbgA9/xiJqaOUiP6bAYND5uwzeDDozdpxcYx0yYtPZkUqpJhNmmRiD3iLE7awDAMHDPPx6r84/FEy9FgHcbEYfpFrqc1PEB8ETvYrIaBd4vAo1t8Gcb0bFlkVO6tdPZe6rWebQkTRbpAhFT+O1btKchxpBSmyL5Eg6Q3f+mF0658v3qKBLNkO/voc9u1d4ZqeCh9Fn3xUJZhNFOEPlKLVBQH1Imas9DwOGycltac5Gvwa/WeMre4RLctReS7f0MJNqntHjYD/OBJGTu/zaThAYjWJ1HdzZuct4FMmbTI+Br9Z0ikXFc6VRLTHrlWXMNjAxCCIOgSPDIstjonYJtYjckmjYV+IeppHx4gHinU0J0mLZtRM2JIt2ESeDJdNTuT6AtFdkRbZxiHNhba4ry41iVVhFa8bfH2pDJnWV2bVLsHjIJYW23si5gFnI4IrhVFHvgDguF56Mv0/KxE6Xb0TsuAqrVQBZz4wqavkO1I4sNjt3GaHwJJ6kUqd2dK4JH9tQh//hQgi2CcObCNmgP7pIUZOTdB76hNvsINGh5dlmFOF56vN/l1Fs2kpfSEhodENCWatABoGj1CPMrP+GXdWTNIptHn4gG6P0rLLCgYnymux81+qeishsdQxLKN40t0Gp50BGW9FijflzrHl8M0xgU+o0kr3Tb2RfTzFtAxwXXW+DLUNoYwbkhhpprVo/TeN47SZYia03Ks+jHHQJSMSQXJ4seAJOPTLfRadBJr0PBIsqzVbcnJ0Nd05LuXAVkEVzEWruOvB8Nl/pE1Jx4EjFofpy1+Tb3gp5n82oiCSgeuDZNWVzpVmk1aksmkRbYTiDBsNtEYF8men1rMQmJ3R2klZCvWPSe92ZwFdF54bY/2pCi3fQKPy2TSSoW57YW51hqebDRpcSyr+bEBHRPqZo4rx8xx5W3v2EegPjyHIPo8PIDenBJfbPVaHCbmxAuofj56rUO+Ny7wtIbUysMMAK+r8wueuG8jxF3fAwwLx8zzukTYAYxOy0TQ4XlrHwTeIg9PJr4OOYt7AOiyXXfhF6jN7MMj601acV8ugvn5IfQ+DY+xfd0h8NhTaA30AmJvDkkHTPeyA+ZJvbBnb+NZyM9xaM9/OiYtfXsK2tLwpDBpOdthduurkGeQYVJHyh0q0BE4BEk0aSWWjdBy8MQWDYMPj86ZOd8XF3hIsVGPy2aZLLA9SA17EflGzdNkG3M0uPyKTp0vFVaOpXpzjZXTsqjLw5OJicQcpUWI+/B0vcAjSgpkWdGc2s1OywTt+UnQ8PQuE4F5zLpD4NFrQ82LKM8xWp223uy/A2TWpNWWMMGyjBY9lY5wpReK8tvQ8JgjLPWRcamE02yB3AezP9mhSs/rnSndTlSMmbSIwEPMVTofnri5hmiBLBLu8aymmg8LEmqbOh8NI7fWI/LdyxB3rAAAMM4cOKac1uHzpYPBadkySkvnw2MRlp4JYYQIUrKSRMPTDT48gGquIiZPEpmRYNJKquFR93fY25+3pSswJP3jWTjsHELtK0DeIZx2Dq0hOUFQYBgGtpiJodebtByd09Y52in8Fue5UNMYSss5Vi+MtWXSMgtHrC7vmNNg0srOb3/Sx2zMM9QRqMBziKEoSoLTqZU5JSoanWUtSyrEcmC4HDxCERF7a9XkXj53+0PSFUWBuPVbhL99HhBUwYkfOh2OqWeAcXrbOLpzGEtLqP3UC0HWJi3ZoOnqLMlMWpmMBEuGTXduQZQ0kxYpCmjW8CT34eEM//Y0+vuS47J12xeuKvBELRd6IvD0fg1P56K09M+r3gcuGUP7+bB+ZwPKC9pONaEXilKVlQCM726hyfxl9DPKzqXQob2T2dm/9kJH4RBDbzIhC72VSSticsg15J8hZpbYb4W5TlTVtmJPTQuAjpWVEFa/BWHl6wAAtmQonEf+VKv43dWYS0sAZh8endOyrthqROe83VmSm7S6XsPDMoxWn0iIyhBFs4bH7MNj1BASnL1sctWPmbcDQniHr5tC8CPPV2/X8HAsCxvPIiomaqrSob0RUKfMGoxpo0rQr6htgYc8X04712ZpA307zP4+hjZmrYYnrnWlUIHnkEMfbp5g0pL0Ji2TD4/eaVk0/lbgIwIP0fC0T+BRwq0Q1ryjnnPyQtgnnwqG7b4X1FhaIjFKy+jDo4vSEjMXMm4VpaXPhdTVE5ad5yBKIiJRSTOruZP58IhGHzACmVzTcTztDvT3rbO13dqD05Zc8CNt6u0CD6AKFB0VeJztFHg4ljWUMkgFeS7NGhvLdujuQWGucczb41jdV9GeRWrSAkAFnkMOvQYhpdOyaTG3Ki1BJgwS6kmyLbc3y7Kw7gNAjIAtHAT7lNO73bnOUsPDWfvwkO1iV5m05EShE8hM6HsqbDYWiAChSLxqtceZxKSVzGnZ0btMWvr7luPqPg1PKr+Jk2cOxqbdjRjRP7fb2tNRnHYOLcFox0xaXZjUb/SgfMwYW4pJsdIwqdDfA7OA5OzCNvYWUmkbD0WowHOIQTQ8LMNoZhQte7AhSsvotKx3bNaHrANAYa6xxEB7NDxKJABh/cfq+Saf0iORBLxVHp4kYelEMAkJcSGgq0xaxKxITE5dCRH6guG4wON2xk1aiqJo94Y8Q+Y2jR1cgGH9fDhyQr8ubWu69JxJK25yMTNrfDlmje8beU08ThvqENai9dpDVyb1s9s4XHHK2LZ3hDH5XoIPj8GklZ0CARl7atJSoQLPIYbmsMzHFyvNP0eyCksnPjz6sPREk5ae9pgPhA2fANEQ2PwK8IMnt6svmcKmW7h5k4aHYYwaIDIOesEgI4kHuUSTVjzPD9vlgiC5z3oNDxF4ZEWBKCmw8aSsRrwWm54CnxO3XTS1S9vZHvSTfHeatEYNzMP6HfUYVtH7tTipOO3IoVi3vR6VA/PbfWxvESY4Vo3OiwiSIekgYEo8mKUCQW8LJOhpqMBziGHOwQMkMaeIJg2PIcOwtUmL4EvTpKUoMqIbYtqdSaeAYXrGcVCfAdfsw2POX0EEoWBMMLDxrCHUtaNwjIWGR8icj1BbkPtM+sWxjGGhEkRJ03pFk/jw9DaMPjzdp+E5ZuoAzJ1UkeDU3deYMKwQE4YVdujY3lS24Zx5w3GwOYSyArdhuyF0Pks1PKMG5uOz7/dhzOCCnm5Kr4AKPIcYZLHi9AIPl44PD6vtEzYtxAVmgSfNr2n54B4oIT9gc4If2nOaAd5Cw0MWd7OwQcYqFFZLaGQqe6mVSSuTtbragvSXaHg4jgHPqTXUJFktlOpx2iAritZGs4ant2EwaXWjDw+QGMF2qNGexINdzbxJ1klLSRsZZG8W4vFDC7H4+jm9Ii9WbyA77zIlKSSkWG/GsTZpET8dY6ZlIL4oOmKTRKHP6MOTrvlA3LsWAMBXjAHD9pzsbZlpOWa+MQsbmkkrktmSD0QAtTRpdYeGJ3YNYqojiSbJdqLx00fy9XYNj72HTFoU1QmezBg9reFJBhHE7FmehZgKO3GohucQw6oOUqoIIYeWhydR4CELSr7PAQaAEtsvnWypACBVrVfP3X98B3qSOaxqaZF/EwQeXZQWkLnoKctcSELXl5Ug2E0aHqK9cdhYhCJx8xrJ0g30fi1GT5m0KOoi67BzCAtSWgVBe4KyQjdG9M/FwJKcnm4KpZvonU8ipcuwcji1MmlFTCHX+sUtqGl44r/5PHY0BwT4PPa0vpYUIQipZpt6/IBxHe5PJuBYRhPYtIKqvLXAw5v8dTIljLAWPjxCBhMbtgXRbAWIhoczarjCgogNOxuQp8sQ3NWRY52lJ01aFOC02UNQ3RBEab6r7Z17AJ5j8esLpvR0MyjdCBV4DjFSangsSkvYTE7L6jnU/fQmg/wcB5oDQtpf0uK+jYAig80tA5vTdj6NrkRf48ich8esjudMAk+mSj7Eo7TiWrZItPuclonjtqbhMZm0XvhoK6rqWuO11biujxzrLHpBsTvD0ikqx00f2NNNoFAM9G6dNCXjEJOEMUor7pBM0KK0bCQ8m0mwBTt0WqK8WBG/dB2Wpb3r1GsP6FlzFoEsiCTnCMmEm2/KiMuZzDiZEkasy3tk1k8oFeQ+J5q01GtX1alZtAWtjlbvFnYAtc5Sfo4DQ8pzujxxI4VC6f1QDc8hhrlwKKA3aVn48OjLKnAMZJ0Ph36xz48JPOk4hyqKAjEm8PA97L9DuHLhODS0hLWcQpNGFOGq08Zh5MA8w34c130mLU3D0w2LtTksnZju9Bqss+YOg9dlw5tf7+wTOWZsPIu//HxGRtIGUCiUvg8VeLKYH7YdxOer9+GSE0YhN6apsDJp8RYmLXPxUPUYBtF4XjrDYjik3IfPsA8DS9uuhyM3VkEJNAAcD65fZQd6lnmG988FEF/EeY7F1FElCfsRUw8h8yatHorSIiatMAlLV/tFBNlZ48pwwuEDwTAMjpzQNzIFA8YcSxQK5dCGCjxZzKff78O6HfVYu6NeS/cftfThiUUeWebhMe8nxY5ntOMA4KiJ/TCify5K8tp2UBS3faeeo/94MHzvL6Kox+zDkylhRF+clSCQKC17Nzgt20wanpgAdtbc4ThsWBEmjijSfHZ6u+8OhUKhWEEFniwmLKiLl6jT3JBK19ZRWhbV0nVfyPrF3rzQMwyD0nxjJlMrFEVGdOtStQ0jZqbXkV6E2aSVKXMTa1EtvTs1PObEg0QgzvXYLTVdFAqF0tegTstZDFkw9cnirBMPJpq0zE7LgHGx76izrnRgs2rOsrvADzysQ+foScxOy5nSvhBtmZXA0z2lJdRraFmUe3mOHQqFQmkvdFbLYki2ZL0jrGVYulVpCVMeHsDov9LRRVjc+i0AwDZ0Ghi+72W/NefhyXTiQdEyD0/3OS0TenuOHQqFQmkvVODJYoiGQDRoeCxMWlZh6ZpJy1rD4+hA7RlFFBDdsVK9/vC+Z84CLDQ8mYrSYpPn4ekWk1aSjNIUCoWSLdBZLYshQotekImmqpYe+02UZO0YvSZH78Nj70BuGHHPGiAaAuMtBFc+st3H9wa6LPGgpQ9P95eWINiohodCoWQZVODJYuI+PHqnZfX/Ocs8POpvpKI6YMxWq4/K6oiGR9y+HABgG3Y4GKZvPnrdGqXVjaUlzIIb1fBQKJRsg85qWYoky5qDspUPT6pMy2ShZWDK19MJp2UlGoa45wf1PMOmt+vY3oTZmTfTmZYtnZa7IdOyufI5dVqmUCjZBp3VshTidAwYfXis8/AYTVpa+QCbsV5SqrD0thD3/ABIUTC+ErCFg9p1bG8iISw9Q9oX1jL5o7FifVdiFtyo0zKFQsk2qMCTpZDFEjBqeKQ0orSscvCo++mjtNr36Ig7VgCIRWf14cR1LMNA3/xMm7QkpYc0PAlRWnRqoFAo2QWd1bKUZAJPlOTh4S00PETgEYmzrNmvo2MmLYM5a+i0tI/rreiFgcyFpRvz8MiKomnpusVpmWp4KBRKlkMFniwlmUlLy7SsW9BIfh3NpJUk4Z0+D097FuFsMWcRDKa9DGlfzCataNTacbyrSMjDw9KpgUKhZBd0VstSDBoeqX2JB0k4tNnMYQhLb4/AkyXmLIJhHDoQrZbqnLKiQFEUw/3rjkzLCSatDPWLQqFQegt0VstSjCYti8SDFk7LJKormYanI4kHFSkKsWq9es3BU9Juf29GP3aZ0vDox1bWCTx2ngXbDUJiojav7wumFAqFoocKPFmKICTz4UleLZ0IRiQPj1mo0efhSdeRVjqwGYiGwbhywRYPbkcPei8GX6ZMFQ/VCTWSpCQVOruKBG0edVqmUChZBp3VspSImMSkJRKnZYvEg7H9yLGpNTzpLcTi7jUAAH7gYX022aAZohFjGSZjzr16M5kkK92aZRkgfbHOuUShUCjZQLevQKtWrUJlZWXCf8uWLdP2Wbp0Kc444wwcdthhOP744/HOO+8YzhGJRHDnnXfiiCOOwKRJk7Bo0SI0NDR0d1d6NQanZTnx/y3z8Ghh6YmFQwGjmSOdsHRFUbToLG7QxPY0v1dDxs5uylPUGZKZtDJlMksHvXM0DUunUCjZBt/dF9y8eTMGDhyIF1980bA9NzcXALB9+3b8/Oc/x09/+lPce++9+Pzzz3HzzTejoKAARxxxBADg97//PVauXImHHnoIdrsdv/vd73Ddddfh+eef7+7u9FoiQjINTwofnoQ8PMlNWuloHuTG/VBa6gCOB18xtr1d6LWQ8cqk9sVs0op0Y1kJgt3GIRAWAVCBh0KhZB/dLvBs2bIFw4cPR3FxseXvzzzzDCorK3H99dcDAIYNG4aNGzfiySefxBFHHIGamhq88cYbePTRRzF16lQAwP3334/jjz8eq1evxqRJk7qtL70ZY5RWotOyIQ8PZwpLF5MlHmxflJa4Z416XL8xYGyO9jS/V0MEv0xlWQYAhmHAMgxkRYEkK0mTP3Yl+ntKTVoUCiXb6BENz5QpyaN1Vq5ciWOOOcawbcaMGfjTn/4ERVGwatUqbRthyJAhKC0txYoVKzol8HRFKC4RJrrbCVSfe0dS4n0jkVgOO6dtIwudLCvgeVbbx+ngDGNi0y3wbicPnmdT9i8YE3jsQyb26TBncx9JXxw2LqP94jgGsqiAYRnNudx8D7oKjmMNAo/d3j3X7S566j3sTrK9j9neP4D2savpdoFn69atyM/PxxlnnIGamhqMHDkS119/PSZMmAAAqK6uRllZmeGYkpIShEIhNDY2oqamBvn5+XA4HAn7VFdXd7hdLMsgP9/T4ePbwudzddm5LdE9TAwDrW/EbFVY4NG2FYRUM4YMdT8mpsHIzXEaxsTrjo95UaHX8Ju5f9GmGjRWbwMAFE+YCT6368a2uyB9dDrU18bjsmf0meE5BlER8Hqd4Hj1Gl6Po0ufSz16jVV+rrvbrtuddPt72ANkex+zvX8A7WNXkVGBp6qqCvPnz0/6++eff46WlhYEg0Hcfvvt4DgOzz//PC644AK8/vrrGD58OMLhMOx2u+E48rcgCAiFQgm/A4DD4UAkEulw22VZgd8f7PDxyeA4Fj6fC35/yGBa6mr8LfGxiAgSGhsDAIBozFQSbI2gMbbABVrD6m+ijMbGAPyt6rGSFD9OPVaMnzMkoLExkLR/wW/fBqCAHzAOLbIb0J2nr2HuoxJz/GYZGMansxAH6MamABqbQ+o1FCWj10gGx7EG81k4dn+zhZ56D7uTbO9jtvcPoH3sCD6fK21tUUYFntLSUrz77rtJfy8pKcGKFSvgcrlgs9kAAOPHj8fGjRvx3HPP4c4774TD4YAgCIbjyN8ulwtOpzPhd0CN3HK5OicxEoferkCS5C49v5mwEBdORN21oxZlJqAY2xiJHcuzrKHNeq8OjmUMv+n7p4gCIj9+AQCwjZ7frf3uSkgfSRkIO89mtG/EGVoQJIQi6j2wZfgaqTA7YWfLfdPT3e9hT5Dtfcz2/gG0j11FRgUem82GYcOGpdzH5/MZ/mZZFsOGDUNNTQ0AoLy8HLW1tYZ9amtr4Xa7kZOTg7KyMjQ1NUEQBIOmp7a2FqWlpRnqSd/HKkpLlhWQYtxGp2Xr4qGpo7SSS9Ti9mVAJADGWwhu4GGd6EXvhGeNvk+ZgtWlBxC0KK2ecVq2ZbEPAYVCOTTp1lntyy+/xKRJk7B3715tmyiK2LRpE4YPHw4AmDp1KpYvX2447rvvvsPkyZPBsiymTJkCWZY152UA2LlzJ2pqajBtWt+vxJ0pBIvSEnrtjj4KR8u0nGZpCQbJw5YVRYGw4WMAgG3M0Zo/UDbRFWHp+vOqiQd7QuBJFIIpFAolW+jW1Wjy5MnIz8/HLbfcgvXr12Pz5s245ZZb0NTUhEsuuQQAcOGFF2Lt2rW47777sH37dvzzn//E+++/j8svvxyAajY76aSTcPvtt2PZsmVYu3YtbrjhBkyfPh0TJ07szu70aiKGaulK7F+9wJO4uJHClZqGx6TFiSfc45Im3JPrdkI+uBvgeNhGzclAT3ofZLwyGZYO6AqIyt1fWsJ8LZqHh0KhZBvdOqt5vV48/fTTKCoqwmWXXYZzzz0XTU1NeP7551FUVAQAGDFiBJYsWYIvvvgCp512Gl555RXce++9WtJBALjrrrtwxBFH4JprrsFll12GoUOH4sEHH+zOrvR6jBqemMATE2QYGEsZ8KayBkk1PJpmI/ljE930pXrOIdPAOnM60YPeC9GIZVr7wmo1zXSlJbox07JB4KHFQykUSpbR7WHpAwcObFM4mTNnDubMSa4dcLvd+OMf/4g//vGPmW5e1mBVLV0rHMobSyLofXPUwpWkeKi1wJNM66BEI4hu/w4Asla7A8TNgZnWvlibtLoz07KutEQW5eChUCgUoAcEHkr3oBd44iYt9V9zFl29v4Yky7rioeYK2ql9V8SdK9TK6DnF4MorO9mD3sv00aXYW9uKSSOKMnpevUkr0gOZlh3UpEWhULIYKvBkKXofHuKMbFVHCzCat0Q5ruGxmb7y+TZKKkQ3f6UeV3lk1lRGt2LskAKMHVKQ8fPGo7RkCEL3Fw/Va6w4atKiUChZRvauSoc4eh8e4owctaijBcTrOAHEpGUdITSsIhfFeU5MqSxJuJ7UVA3pwGaAYWAbOTujfTlU6PEoLZ5qeCgUSvZCNTxZiCjJmqMyQZIVLUrLajEjdZyiumPNPir5OQ789cqZltcMr3lPPU//8WC9mdd+HApYmbS6U+DR+wvR4qEUCiXboJ9xWYheu0PQZ1u2SipHFttQOJ6h2Zx4MBnhvZsgbPxMPWbiSe1uL0XFqOGxTg3QldhiwhXHMknTDlAoFEpfhQo8WYjef4cgyQqimtNyCoEnEhd4zKYvKxRJRN27j6j7Vx4JPoudlbua3pJpmZqzKBRKNkJntiyELJZOncOrJCmIEqdlPvHrnRRfaw1FtWPT+coPr3kP0YNVYJw5cBx+bqfbfihD0gMIUUkzK3an0zIxaVFzFoVCyUaowJOF6P0/iOZGlGTNhyeVSasxVik9x21r8zqKLCGy9gMAgGvmT8A4vZ1v/CFMXMsWN0lSDQ+FQqFkBjqzZSEGgUdXGFTUJR40QxbbJk3gsSfsY0ba/yOUkB+sKwf2EUe0uT8lNeQeBGNmRY5lulX4iAs8VMNDoVCyDyrwZCERXWkITleuIJpKwxPb1tQiAAB8aQg80W1LAQDe0TPBcDTgr7OwJj+q7qyjBQD9ijzgWAblhZ5uvS6FQqF0B3SVykK00hA21mjSSpJ4EIjXTiIaHm8bJi1FFCDuVCvWe8cdiVBmmn5IY3Yc784ILQAozHXh79fN7tbszhQKhdJdUA1PFqLX8BDzhCTFNTyporSa0vThEff8AETDYHOK4OhPI7MygWbSiqUG6E7/HUKu15FWdB6FQqH0NejMloUYnZbjJi0tD4+VDw9nFHjaMmmJ29QiofbhM7K6jER3YjZp9YTAQ6FQKNkKXamyEH0dprjTsqzl4bGO0lK3kQihVBoeRQipGh4A9pHUWTlTmE1aVOChUCiUzEEFniwkEtPk2HlWM1+Jkj5KyyIPj6lYZKooLbFqHSCLYHLLwBUOyFSzD3mI0BnUBB76elIoFEqmoDNqFiJY5OGR5NROy5wpFDmVSUvcvUY9z6CJGWgthdDTUVoUCoWSzVCBJwuJ6E1abKLTsqUPD2vclsykpciSZs7iB03KWJspiXl4ujPLMoVCoWQ7VODJQgRRl4eHI2HpSlrFQwnJBB6pZhsQCQAOD7jS4Zls9iEPuVdKrNA99eGhUCiUzEEFniyEFA918Cx4LUpLTh2WrjNpOe0cbElysYi7V6vnGDABDEsX5EzCmmqXUYGHQqFQMgcVeLIQYtKy282lJWLV0lOUlgBSR2hJmv8ONWdlGrOWjfrwUCgUSuagAk8WYpmHRx+lZVErSe/DkyxCSzq4C3JzNcBy4AeMy3SzD3nMjuM0SotCoVAyB51RsxDiw+PQZVoWZRnRNBIPAtYRWtLB3Qi9+zcAMXOW3Z3xdh/qsFTDQ6FQKF0GraWVhUSEWB4eXS0tQ5RWilpaQGIdLalhL4Jv/xUQgmCLBsFx1KVd1fRDGo768FAoFEqXQQWeLMSQh4dLLC1hXUtLb9IyCjzRdR+pwk7pcLhPuIFqd7oIznRfqMBDoVAomYOatLKQiFXiQUnWZVpun0lLatoPALCPO44KO12I2aRFBR4KhULJHFTgyUL0eXjiPjxK3IenjTw8eg2PoiiQmw4AANi8si5rMyUxSos6LVMoFErmoDNqliErCsKCVZRWXMPTltOyPkpLCbeoiQbBgM0t7cKWUxLC0mmmZQqFQskYVODJMsIRScvU63HyulpailYtvT0+PES7w+QUguEdXdVsCqhJi0KhULoSKvD0UVZsqsVDr61FMCwatgfCUQCqFkc1aeny8Iip8vBY+/DIzdUAADaXmrO6mkSTFhV4KBQKJVNQgaeP8uGKPVi99SB+3N1g2E4EILdTDcDjdHl4xFRh6VxqDQ+bV57B1lOsoAIPhUKhdB1U4OmjED8d8i+BaHi8TlVoMeThSZV4MLafw1RHiwo83Qc1aVEoFErXQQWePkokqcBj0vDEFlFBlBBz7UkZlu4z5eCRm2ImLSrwdDlmDY+NRmlRKBRKxqAzah9FiGlrSM4dAtHweGIaHmKqiugEI2unZXWxNURoSVEoLbUAqMDTHegdx+08m1A9nUKhUCgdhwo8fRQi6Jg1PMSHx2PS8Oj3s/LhyfWqEVil+fHEgrK/FlAUwOYC48rNYOspVuhNWrSOFoVCoWQWWlqiD6IoCoSYABNJ4sPjJj48MeGGCDwswyT4igDAhKGF+NXZEzCk3Kdt0yccZKi2ocvRm7So/w6FQqFkFirw9EGioqz540SiprD0kEnDwxk1PFYOy4CqXZgwrMiwjTosdy8GgYcmHaRQKJSMQk1afRDivwNYmbRiPjwuY5QWEYyscvAkgzosdy+sQcNDX00KhULJJHRW7YPozViJJi1jlBbPGk1aVhFayZBjRUNp0sHugZq0KBQKpeugAk8fRB+ZlTxKy2jSIoKRlcOyFYokQm7Yq56jcEDnGkxJC446LVMoFEqX0a0+PK+//jp+/etfW/52+OGH49lnnwUA3H777XjllVcMv1dUVODTTz8FAMiyjMWLF+OVV15BS0sLpk2bhjvuuAMDBhwaCzOphg4kj9LSnJZjGh4hRdJBK+T63YAkgnHmgPHRoqHdAUs1PBQKhdJldKvAc+KJJ+LII480bHv//fdx991348orr9S2bd68GVdeeSUuuOACbRvHxReAJUuW4MUXX8Rf/vIXlJWV4d5778Xll1+Ot956C3a7HdmOwaSVoOGxdlomWOXgsUKq3gYAYEuG0QitboLT3Rsq8FAoFEpm6VaTltPpRHFxsfafJEn4xz/+gauuugozZ84EoIZcb9u2DePGjTPsW1BQAAAQBAH//Oc/cd1112Hu3LkYNWoUHnjgAVRXV+PDDz/szu50K02tEUgySTZo7bQsywpCESLwxBIPsh0UeGpVgYcrHd7xRlPaBcfoTVrU2kyhUCiZpEfD0u+9916UlJTgiiuu0Lbt2bMHwWAQQ4cOtTxm06ZNCAQCOOKII7RtPp8PY8aMwYoVK3DyySd3uD3tcehNF/LVzqUpaFhRVdeK2x77DkeMK8OVp42DKMcFnkhU0trdEhS07T6vHTzHwm4Kb7bzbFr9lGpUgcfeb2TK/TPRv95Od/VRf69cDr5LnsdkZPt9zPb+Adnfx2zvH0D72NX0mMCzefNmvP3223j44YcNZqgtW7YAAJ577jl8+eWXYFkWc+bMwfXXX4+cnBxUV6uh0uXlxlDpkpIS7beOwLIM8vM9HT6+LXw+V4eP3binGQqA/fVB5Od7YLPHb1tEkJCX5wbDMAhJanYel4NDcVEOACCvMWw4l8tpa7Ofov8gGgONAMOiaORYsHZnm23sTP/6Cl3dR9YWv6+5PleXPo/JyPb7mO39A7K/j9neP4D2savIqMBTVVWF+fPnJ/196dKlmmnq6aefRmVlZcL+W7ZsAcuyKCkpwaOPPoo9e/bgnnvuwdatW/HMM88gFAoBQIKvjsPhQHNzc4fbLssK/P5gh49PBsex8Plc8PtDkCS57QMsaGxS29UaFNDYGEBDU0j7TZIV1B1shY1ncaDGD0DVDjQ2BgAAoWDEdDZF+y0ZwtYf1LYXDURzQAICyffPRP96O93Vx0Aoqv2/Iklt3qdMku33Mdv7B2R/H7O9fwDtY0fw+Vxpa4syKvCUlpbi3XffTfp7bq5ajykcDuP999/HTTfdlOAQ+4tf/ALnnXce8vPzAQAjR45EcXExzjnnHKxbtw5Op6ptEARB+38AiEQicLk6JzGKYtc9YJIkd/j8YUH1ywmGRYiijFDYnF05Cq/LBn+ratJyO2zxaymGXcGxTJvtEA5sBaA6LKfb5s70r6/Q1X2U5fjN4jm2R8Yz2+9jtvcPyP4+Znv/ANrHriKjAo/NZsOwYcPa3O+bb75BNBrFCSeckPAby7KasEMYMWIEAKC6ulozZdXW1mLgwIHaPrW1taisrOxM83stJKQ8JIhQFCUhMissiPC6bFqEltcVv63mKK108vAQ/x3qsNy96LNg00zLFAqFkll6ZFZduXIlRo0alSDYAMDNN9+MSy65xLBt3bp1AIDhw4dj1KhR8Hq9WLZsmfa73+/Hxo0bMW3atC5td08RjQk8iqI6KQsmgYeEqQdNhUMBYzI7oO0oLSUahnxwj3osFXi6FZqHh0KhULqOHhF4Nm7ciFGjRln+tmDBAixduhSLFy/Gnj178MUXX+A3v/kNTj75ZAwbNgx2ux0XXHAB7rvvPnzyySfYtGkTrr/+epSVleG4447r5p50D1Gd2i8UkRI1PLG/W01lJYBET/i2Eg9Gf/wCUCQwvlIw3qKU+1IyC8tQgYdCoVC6ih6J0qqrq8P48eMtf5s/fz7+/ve/4/HHH8cTTzyBnJwcnHLKKfjVr36l7XPddddBFEXcfvvtCIfDmDZtGp566inYbDbLc/Z1jJmVxQSBx6zh8eo0PO3Jw6OIAoS17wEA7BNPpAkHuxmGYcCxDCRZoaUlKBQKJcP0iMCTyrEZAE444QRL/x4Cx3G46aabcNNNN2W6ab2SqC7RYDAiQogaHb2IwGMuHAokanh4PrkQE938FZRgExhPAWwjZnW63ZT2w8YEHqrhoVAolMxCPSP7AFFd6F44hUkraCorAST68CRzWlYkEcKadwDEtDtcj+akPGQh98fpoAIPhUKhZBK6qvUB9BqdUERMdFqO/U3yuLTXaVmRZUS+fgZKoAGMOw+2yjkZazulfZw5dxhqG4Moycv+xGMUCoXSnVCBpw8Q1fnwhCJxHx6Xg1f/Npm0PLqwdLOAY3ZaVmQR4U8fh7hjOcAwcMw8Dwyf/QVYeyvzJlX0dBMoFAolK6EmrT6AIUpLkDSNj8+jCiaa03JE1fB49BqeNqqlC9+/pQo7LAfn/KtgGzo98x2gUCgUCqWHoQJPH0AQ9T48cQ1PrlsVbMKaSSvRaZllGOiDrcwaHunAZgCAY8ZPYBuanXmMKBQKhUKhAk8fQK/hCeoEnhydhkeUZG27XsMDGLU6vEnjIzerBVe5krYzZFMoFAqF0lehAk8fQC/w6PPwEJNWWJC0CC0AcDuMrll6x2UbH4/+UYQQlGATAIDNLc14uykUCoVC6S1QgacPoE88GIzEfXhy3TENT1RCIJZ00OXgDSUKAKPAo9fwyP4aAADjzAHj8HRN4ykUCoVC6QVQgacPoNfw+AOC9v9xp2VR89/R5+Ah/H97dx4fRXn4cfyzZw5ykHAkYFTuIEc45DAKiGCBilZE21qBogW1+EMrHhxKRQGxAoKKInIJIlqPosVCvRWL5ZYqyh0OQSVcIRCS7GZ35/fHZpcsCZBrk836fb9eeUlmZmefZ4d1vjzXFF18sOg6PJ4T3u4sc+0GlVtgERGREKPAUwMUDTwnchz+P8cWGbR8/FQ+ALVjI4q9vmirTtHxPJ5sbwuPurNERCTcKfDUAEVnaWXneFt4bFYzUYVjdRxON0ezvYGnXnxksdcHdGlZiwYebwuPKT658gstIiISQhR4aoCiCw/6BixH2CxE2L0DkPOLBJ468cVX6LWYz9GlVRh4zAo8IiIS5hR4QpzHMHC5jWLbI2xmIgsfMOkocHP0RB4AdUto4Qno0ips4TEM48wYHgUeEREJcwo8Ia7o+J2i7EVaeC7cpVW0hccbfoy8k1CQB5gwx9Wr5FKLiIiEFgWeEHe+wBNp947hcbo8Z7q0SnjopKWEQcv+8TuxdfTsLBERCXsKPCHOF3gsZlPA4OMIm4UI25lFBF1uDyYTJJYwS6ukQcsavyMiIr8kCjwhzrfooM1qJtJ+JuBE2CxYLYEhKDE2otjDQSFwKrpv0LKhKekiIvILosAT4goKV1UuOg0dwG4zYzKZAlp5SpqhBWc/WqKwhefEz4BaeERE5JdBgSfEFbi9gcd+VuDxBZ2IIq0+JQ1YhsDA4/uzf9FBrbIsIiK/AAo8Ic5Z4OvSshB1VpcWENDNVeccgcfXpWW1eFuFDMOD59RhAMxx9YNSbhERkVCiwBPifIOWz+7S8rfwFOnSqnuuLq3CWVq+7izj9Alwu8BkxhRTJxjFFhERCSkKPCHOF3jO7tKy27yXrmgLT73a5+/S8q3B4zl1BABTTB1MZkuJrxEREQknCjwhzlmkhSfyAi085+rS8i086F9l+aS6s0RE5JdFgSfEnZmWbiEq4ky4sZ81aNlsMpFQwho8cObREv5FBwtbeMyxWmFZRER+GRR4QpyraJeW/dwtPIlxEQGPkCjKt923Bo/nZGGXlh4pISIivxAKPCHO36VlK3kMj6+Fp6SHhvr4Bi37V1n2tfAo8IiIyC+EAk+IcwYMWi4+LT26MATVLeEZWj7+WVqWs8bwxGoMj4iI/DJYL3yIVCf/tHSLpcQurfQ2yRw6nsu1l6ec8xz+QcsWE0aBw/ukdNTCIyIivxwKPCGuwDdo+awuLV9XVlJCNH++sc15z2E1n+nS8nVnEVELU0StIJRYREQk9KhLK8Q5z7UOj7X0l65ol5ZxUjO0RETkl0eBJ8QVBKzDU3wMT2n4Z2lZzUUeKaHAIyIivxwKPCHuTOCxBLbw2EsfeC67NIG68ZG0b1YXj3/AsgKPiIj8cmgMT4jzPTzUbjUTHWGlVqQVwzgzO6s0Lk2OZeqIKwHI/bdvDR7N0BIRkV8OBZ4QV+A+06VltZh59I+dMAzDv2pyWRlaZVlERH6BFHhCXEHBmcADkJwYXe5zGYZHiw6KiMgvksbwhLgzs7Qq/lRzIzcb3C4wmTHF1Knw+URERGoKBZ4QV3SWVkV5ju4DwByfhMlc8QAlIiJSUyjwhDj/woOVEHhcP+8AwJLcvMLnEhERqUkUeEKcv0vLVvFL5T60EwBLcmqFzyUiIlKTBDXwPPbYY4wdO7bY9jVr1jBw4EDatWtHv379WLFiRcB+h8PBE088QXp6Oh06dODBBx/k+PHjZTpHuDjzLK2KXSqjwIHnyH4ALMktKlwuERGRmiQogcfj8TBjxgzefPPNYvsyMjK4++676d69O8uWLeO3v/0to0ePZs2aNf5jHn/8cVavXs2sWbNYvHgxe/bs4b777ivTOcKFP/CUYWXlkrgPZ4DhxlQrAVNs3coomoiISI1R6dPSMzIyePTRR9m/fz8NGzYstn/x4sWkpqYyatQoAJo2bcrWrVuZP38+6enpZGZm8t577zFnzhw6deoEwIwZM+jXrx+bN2+mQ4cOFzxHuDAMA6frzMKDFXGmO6sFJpOpwmUTERGpSSo98Kxdu5amTZvy4osvcv/99xfbv3HjRq699tqAbVdccQVPPvkkhmGwadMm/zafxo0bk5SUxIYNG+jQocMFz1HeG7q1EgYGn81S2BVlKUeXlMvtwTC8f46KsFaofHmZ3sBjuyi1UutZkfrVFKpjzRfu9YPwr2O41w9Ux2Cr9MAzaNCg8+4/dOgQycnJAdvq169PXl4eWVlZZGZmkpCQQERERLFjDh06VKpzJCYmlrncZrOJhIRaZX5dacXFRZX5Nbn5Bf4/168Xi72c3VqG28WJzAwA6qS2xx6EepanfjWN6ljzhXv9IPzrGO71A9UxWMoUeA4ePEjv3r3PuX/NmjUXDBv5+fnY7faAbb7fnU4neXl5xfYDRERE4HA4SnWO8vB4DE6ezC3Xa8/HYjETFxfFyZN5uAsfE1Fa2TkO/59zTuWVu+XKlZmBUeDAFFGLHGsipqzT5TpPSSpSv5pCdaz5wr1+EP51DPf6gepYHnFxUaVuLSpT4ElKSmLlypXn3B8fH3/Bc0RERBQLJb7fo6KiiIyMLDG0OBwOoqKiSnWO8nK5gvcXzO32lPn8eQ4X4F2Dx+02AKPs73tkL3mfzQG843fcboDKr2d56lfTqI41X7jXD8K/juFeP1Adg6VMgcdms9G0adMKvWGDBg04fPhwwLbDhw8THR1NbGwsycnJnDhxAqfTGdCKc/jwYZKSkkp1jnBR4H+sRPn6Ogv2biL/k9mFs7MSsXceWJnFExERqTGqfNRQp06dWL9+fcC2tWvX0rFjR8xmM5dffjkej8c/eBlg7969ZGZm0rlz51KdI1w4Cyr2WImCbz8Aw4310g7UumUSlsSLK7N4IiIiNUaVp4MhQ4bw7bffMn36dDIyMli4cCEffPABw4cPB7zdZv3792f8+PGsW7eOb7/9lgceeIAuXbrQvn37Up0jXFTkOVqGYeA+8RMA9k43YYoI3oBsERGRUFflgad58+bMnj2bVatWMWDAAN5++22mTZsWsH7OpEmTSE9PZ+TIkQwbNowmTZrw/PPPl+kc4aDAvwZP2WdnGXknwXEaMGGOT77g8SIiIuGs0qelF7VkyZISt/fo0YMePXqc83XR0dFMnjyZyZMnn/OYC50jHDgr0MLjKWzdMcXWxWQtPutNRETklyR8BryEoYp0aXmyvIHHnFB8tWsREZFfGgWeEFaRWVq+Fh5zbQUeERERBZ4Q5nuOlq0cY3g8J34GwKIWHhEREQWeUFahMTzq0hIREfFT4AlhrnJ2aRmO0xi5JwB1aYmIiIACT0jzt/CU8aGhvu4sU60ETPbwfwidiIjIhSjwhDD/GJ5SPhjNx9+dpdYdERERQIEnpJV3WrpvhWWN3xEREfFS4Alh5R3Dc6aFp0Gll0lERKQmUuAJYf4WHlsZA0/hGB5zwkWVXiYREZGaSIEnhBW4CwNPGcbwGPk5GKeOAGBR4BEREQEUeEJaecbwuA/tArzdWabImKCUS0REpKZR4Alh5Qk8rkM7ALAktwhKmURERGoiBZ4Qdmal5dKvw+M+tBNQ4BERESlKgSeE+Vt4SjmGx3A58BzZD4ClgQKPiIiIjwJPCCtrl5b78B4w3N4VlmPqBrNoIiIiNYoCTwjzz9IqbeD5+Ux3lslkClq5REREahoFnhDm8j1aorSBR+N3RERESqTAE8LKMobH8LhxZ+4GFHhERETOpsATwvxdWqVYadlzdB+4HGCPxpyoBQdFRESKUuAJYWVp4XHt3QSANaU1JpMuq4iISFG6M4Yoj2HgchvAhcfwGIZBwZ4NAFibdA562URERGoaBZ4Q5XtSOlw48HiO/eB9fpbFjvXitGAXTUREpMZR4AlRvvE7cOHA4/K17lzcFpMtMqjlEhERqYkUeEKUs8AbeMwmExbzuS+TYRgU7FV3loiIyPko8ISo0i466Dl+ECM7EyxWrJe0q4qiiYiI1DgKPCGqtI+VcGWsA8Ca0haTPSro5RIREamJFHhClKsUgcfwuCjY8R8ArM2vrJJyiYiI1EQKPCGqNGvwuPb/DyMvG1NUHNZGHaqqaCIiIjWOAk+IKijFc7QKtn3hPSa1OyaztSqKJSIiUiMp8ISoCw1a9pw8gvvgd95jWl5dZeUSERGpiRR4QtSFBi0XbF8FgOWi1pjj6ldZuURERGoiBZ4QdaHA49r/P+/+lj2qqkgiIiI1lgJPiDrfoGXD5cRz4icALMktqrRcIiIiNZECT4g63xgez/GDYHgwRcVhiq5dxSUTERGpeRR4QpTv0RIlBR730X0AmOtcgslkqspiiYiI1EgKPCHqvC08R/cDYKnbqCqLJCIiUmMp8ISoM2N4LMX2uQsDj7nupVVaJhERkZpKgSdEnevREobH5R3DA1gUeEREREolqIHnscceY+zYscW2/+Mf/+CGG26gffv29OnTh7lz5+J2u/37ly9fTmpqarGfgwcP+o/597//zXXXXUdaWhoDBgxgzZo1waxKlfO18FjPCjyerJ/A4wJ7FKbYetVRNBERkRonKIHH4/EwY8YM3nzzzWL7li9fzoQJExg8eDDLly/n/vvv5+WXX+all17yH7Njxw66dOnC6tWrA34aNGgAwNq1a3n44Ye59dZbeffdd0lPT+euu+4iIyMjGNWpFgXukh8t4R+/U+dSDVgWEREppUp/AFNGRgaPPvoo+/fvp2HDhsX2v/HGGwwYMIDf//73AFxyySXs3buXt99+m5EjRwKwc+dOUlNTqVev5BaMefPmce211/LHP/4RgDFjxrB582YWL17MxIkTK7tK1cLXwmM/K/Bo/I6IiEjZVXoLz9q1a2natCn/+te/SElJKbb/oYceYtiwYYGFMJvJzs72/75jxw6aNm1a4vk9Hg9ff/016enpAdu7du3Khg0bKqEGoeFcKy2fmaGlwCMiIlJald7CM2jQoPPuv/zyywN+P3XqFG+88Qbdu3cHIDs7m8zMTDZu3Mjrr79OVlYWaWlpPPzwwzRu3JiTJ0+Sm5tLcnJywHnq16/PoUOHKlT2s8fLVAZL4UrJlhJWTD4fl8cAIMJu8ZfL8LhxH/sBAHtSYyxBKG9Zlbd+NYnqWPOFe/0g/OsY7vUD1THYyhR4Dh48SO/evc+5f82aNSQmJpb6fKdPn+aee+7B4XAwevRoAHbt2gWAYRg89dRT5Ofn89JLL3Hbbbfx/vvv43K5ALDb7QHnioiIwOFwlKU6AcxmEwkJtcr9+guJi4sq4yu843Nqx0f7y5W3bwsnXA7MUTHUadwUk7n4lPXqUvb61TyqY80X7vWD8K9juNcPVMdgKVPgSUpKYuXKlefcHx8fX+pzHTlyhLvvvpuDBw+yYMECf/dXp06dWLNmDQkJCf5BuS+88AI9e/Zk2bJl/Pa3vwXA6XQGnM/hcBAVVf4P0OMxOHkyt9yvPxeLxUxcXBQnT+bhLlxMsDRy8wsAcDoKyMo67d32zWoArJd24ER2fqWXtTzKW7+aRHWs+cK9fhD+dQz3+oHqWB5xcVGlbi0qU+Cx2WznHFtTFhkZGQwfPhyPx8PSpUtp3rx5wP6zW4mioqJISUkhMzOT2rVrEx0dzeHDhwOOOXz4MElJSRUql2/tm2Bwuz1lOr+jwDtLy2LylsswPDj3bPRua3R5UMtaHmWtX02kOtZ84V4/CP86hnv9QHUMlirvRDtw4ABDhw4lKiqKv//978XCzptvvknXrl3JzT3T2pKTk8O+ffto1qwZJpOJjh07sn79+oDXrVu3jk6dOlVJHaqC66ynpbszMzByT4AtCstFrauxZCIiIjVPlQeeRx55BKfTyYwZM7BarRw5csT/A9CjRw88Hg+jR49m165dbNmyhXvvvZfExEQGDhwIwB133MGKFSt45ZVXyMjIYOrUqWzbto2hQ4dWdXWC5swsLe84Hddeb+uO9dL2mCy2aiuXiIhITVTps7TOJzMz098yc+ONNxbbv2PHDho0aMCiRYt45pln+MMf/oBhGFx11VW8+uqrREREANCtWzemTJnC7NmzmTlzJs2aNWPOnDmV0t0WKoo+PNQwjDOBp3H4tGKJiIhUlaAGniVLlgT8npSUxI4dOy74utatW7Nw4cLzHjNgwAAGDBhQkeJVi237jvPTsVx6X158jaKiij5awnN0P0bOMbBGYL24bVUUU0REJKxUaQuPwKIPtnPkRD6XXZpAw7qB0+A/3XSQWlFWrmiVHLDwoPvH3QBYGqRistqLnVNERETOT4GniuXkuQr/WxCw/WSuk6Uf78RqMdPlsqSAR0t4jh0AtLqyiIhIeYXvco4hylk43Tzf6Q7Ynu/wBiGX28Op0048hnelZZvV7F9d2Vzn4iosqYiISPhQ4AmiApeL7as/I/t4FuANM+7CR0b41tk5c+yZ9QiOnTyzYrTVZOA5fhAAS+IlwS6yiIhIWFLgCaJd//2CpG8XsXXJdCAw1OQ7XQHHFhRZcfL4yTOrKFtOHwF3AVgjMMXVD3KJRUREwpMCTxCdtNUFoF7ODtynjvq7s6B4l1bRMOQLPFaLCeO4d/yOOTEFk1mXS0REpDx0Bw2muGR2FDTAjIHzu89wFAk1jvMEHl+Xls1qxlMYeCx11J0lIiJSXgo8QWS3WfhPfioAjm1f4Mw/01V1vjE8vhYem0UDlkVERCqDpqUHUYTNwvcFKZwklrj8U9j/9zb3x+7CanLzfd6QgGMDAs+pwsBTdEq6WnhERETKTS08QRRhs+DBzNdcBkDUD/+lse0IF1uP0/zoFwHHFh207OvSirM6vQ8MxYQ5US08IiIi5aXAE0R2m/fjXV+Qijm2Lq6I2qzKbwlAk9xvcB/d7z+2aAvPydNOABqajwNgiq+PyRZRVcUWEREJOwo8QRRh8z7p/LjTRvyQGWR0fYRluV3Y5GiECXD8dylG4QKDRQOPT7LpGKDuLBERkYpS4Akie2HgcThdGIbhH6i8PO9yCrDiPrQT175NQMmBp4nhHbBsqde4ikosIiISnhR4gsjXwuMxvGN0nIWh5oSnFl+b0gAo2PaF97/uwMCTYM4hxXMQMGFt2rXKyiwiIhKOFHiCyDeGB8BZ4AlYeHCjpwUA7oPf48k5XqyFp5N9LwCWhi0xx9SpgtKKiIiELwWeILJazFjMJsC70GDRwHPIGYOlQSpgULDrv7gCAo9B54gMAGwtrqrCEouIiIQnrcMTZBF2C7n5LhwFbn+XFngDkK1FN9w/78C1czXEpXBbra/wGCZ2u5JJspzEZbJhbXR5NZZeREQkPCjwBFmE7UzgKbq6sqPAjblRZ/hqCZ7sQ1xzai6REbkApLMbgJ+iWpBgj6qWcouIiIQTdWkFmW/gsrPAjbMgcJxOgcmOtXFnACI9uRxxx7LFmeLffzC+XdUVVEREJIyphSfIfAOXHQVuClyBz89yON3EtPkVrr0b2G++lJcyLycfO8l5WdQyOWjeokl1FFlERCTsqIUnyCL8a/EUb+HJd7qx1GtEzO1z+KTW9eQZdhJiI/jZncBuVzI2iy6PiIhIZdAdNcgi7IWBp8CN46wWnnyn93eT2eyflp4YG+nfb7Pq8oiIiFQG3VGD7MwYHk+xFp6ig5h9Cw8mxp15ZpbNaqmCEoqIiIQ/BZ4g83dpFQSuwwNnWnjgzKMlEuPOtPDY1cIjIiJSKXRHDbKAMTyFocZUuK9oC49v4cE6cerSEhERqWy6owaZ3V68hadWlA2AfIfLf9yZMTxFu7R0eURERCqD7qhBFmE9My3dF3jiatkByC9hDE/t2AhMhU1AmqUlIiJSOXRHDbLAWVreUBMX7W3hcZQwhsdusxBT2AKkFh4REZHKoTtqkJ0ZtOwp1sITMEurMPDYrGbqJ0QFHCciIiIVo5WWg8wXeE7nFWAY3m2x0YVdWiW08NgsZu68vhU/Hc3lkqTYqi2siIhImFLgCTJ7YeA5lev0b4st7NLKd3oHLbs9HjyFachmNZMQG0H9hOgqLqmIiEj4UpdWkPnG8JzKLQDAYjZRKzJwDI+vdQc0bkdERCQYdHcNsoizWnjsNguRhSHIN0srIPBoZpaIiEil0901yHyBx+X2dlnZbeaAxQjhTOCxmE2YzaYSziIiIiIVoTE8QWa3BWbKCOuZFh5/4HGfmaElIiJl4/F4cLtdFz4wxHk8JvLzLTidDtyF/0gON2Wpo8VixWyuvPuiAk+Q+VpzfOw2s39cT/5ZLTwKPCIipWcYBidPHicvL6e6i1Jpjh414/F4LnxgDVaWOkZFxRAXl4jJVPHeDwWeIPOFGx/vGB7vx372GB4FHhGR0vOFnZiYBOz2iEq5KVY3i8UUtq07PqWpo2EYOJ0OcnKyAIiPr1Ph91XgCbJiLTzWMy08Z4/hsVkDjxURkZJ5PG5/2ImJiavu4lQaq9Xsf5h0uCptHe1277Mlc3KyiI1NqHD3lpoUgqx4l5aFSNuZx014DCNg0UEREbkwt9v7D0bfTVHCk+/6VsYYraDeYR977DHGjh1bbPsdd9xBampqwM+QIUP8+x0OB0888QTp6el06NCBBx98kOPHjwecY82aNQwcOJB27drRr18/VqxYEcyqlJvZbAroqirawgPgLHCrS0tEpJzCoRtLzq0yr29Q7rAej4cZM2bw5ptvlrh/x44dPP7446xevdr/M2vWLP9+375Zs2axePFi9uzZw3333effn5GRwd1330337t1ZtmwZv/3tbxk9ejRr1qwJRnUqLLJIwLHbLNitZv8T0fOdbs3SEhERCbJKH8OTkZHBo48+yv79+2nYsGGx/ceOHePYsWO0a9eOevXqFdufmZnJe++9x5w5c+jUqRMAM2bMoF+/fmzevJkOHTqwePFiUlNTGTVqFABNmzZl69atzJ8/n/T09MquUoVF2K3+lZbtNgsmk4lIu4U8hxuH002By9s0q8AjIiISHJV+h127di1NmzblX//6FykpKcX279ixA5PJROPGjUt8/aZNmwC44oor/NsaN25MUlISGzZsAGDjxo3Fgs0VV1zBpk2bMIzQG91edByPvTDU+LblO90awyMiIjXCyJF38eSTj1d3Mcql0lt4Bg0adN79O3fuJDY2lokTJ/LVV18RHR1Nv379uOeee7Db7WRmZpKQkEBEROBAtPr163Po0CEADh06RHJycrH9eXl5ZGVlkZiYWK6yW4PQwmKxmImMOBN4IiOsWK3mwqnpTlweD+7CkGa3W4JShmCyFIY0SxiHNdWx5gv3+kH41/Hs+nk84Td2xzfUwWSCEPy3e6Uobx0tFlOF749lCjwHDx6kd+/e59y/Zs2aC4aNnTt34nA4SEtL44477mDbtm1MnTqVn376ialTp5KXl4fdbi/2uoiICBwOBwD5+fnFjvH97nQ6i722NMxmEwkJtcr12gvxrbsDUDsukoSEWtSKtsFxsNpt2Gze/THR9qCVIdji4qKquwhBpzrWfOFePwj/Ovrql59v4ehRc6XcCENNKIdWk8mEyVTxz7y0dfR4TJjNZuLjo4mMjKzQe5Yp8CQlJbFy5cpz7o+Pj7/gOSZOnMiYMWP8x7Zo0QKbzcaoUaMYPXo0kZGRJYYWh8NBVJT3L3pERESxY3y/+44pK4/H4OTJ3HK99nwsFnNAl5bb5SYr6zS2wmdmHT1+muxT+QAYbg9ZWacrvQzBZLGYiYuL4uTJPNzu8Fw7QnWs+cK9fhD+dTy7fk6no/CREkaxNV0Mw8BZUD2fgd1mLvPMom7dOnHHHXeycuX7uFwuXnxxLklJDZg37yU++ujfnD6dQ+PGTRk+/M906XIFGRm7GTr0VhYseI3U1JYAjBv3EF9/vYGVKz/DYrHg8Xj4zW/6cO+9D9C373W8//57vPPO3zlw4ABms4kWLVpy330P0LJlKwBuueUGevbszdq1X5GVdZzJk6fSunVb5syZxUcffUBBgZMbb7wZj8eDYZz5zF9/fQnvvfcOR44cpm7devTv/xuGDh12zs/AZPJeS7fbU6oWHrfbwOPxkJ2dS16eu9j+uLioUoenMgUem81G06ZNy/KS4m9otRYLRs2bNwfOdFWdOHECp9MZ0Ipz+PBhkpKSAGjQoAGHDx8OOMfhw4eJjo4mNja23GUL1mJPRaehW80mXC4P9sIQlJtX4F+A0FK4ryZyuz01tuylpTrWfOFePwj/Ovrqd66Veg3D4KnXvmb3j9lVXDKvZinxjBvUscyh591332b69OcBDykpl/D444+yf/9eHntsEvXq1eerr75k9Oj7mTJlOlde2Y0GDRqyYcNaUlNb4na72bx5I7m5uezcuZ3LLmvN1q3fc+rUKdLTu7Fq1efMnDmVMWPG065dB44ePcqzz07jb3+bzKJFr/vLsGzZWzz99ExiY2Np0qQZzz47ja+++g+PPjqBpKQGvPrqQr75ZjMNG14EwOrVX7JkyStMnDiFiy9uxPfff8vkyRNo0KAhffteV2I9fSGnrF12JQXbsqrylZaHDBlCSkoKTz31lH/bli1bsNlsNGrUiHr16uHxeNi0aZN/YPLevXvJzMykc+fOAHTq1In169cHnHft2rV07NixUh80VlnOnpZedFvAoOUwa5YVEakWNXB4T9++19GyZSusVjP79u3nk08+5JVXltK8eSoAt946mN27d/H6669y5ZXduOqq7mzYsI7Bg29n27bvsVpttGnTlq+/3shll7VmzZrVtGvXgbi4OOLj4xk79q/06fNrAJKTG3D99b9hxoypAWW44oqr6Ny5KwC5uaf597//xYMPjiE9vRsA48Y9xtdfb/Qf/9NPB7HbbSQnNyQ5OZnk5GTq1q1PUlLgGNtQUeWBp2/fvkyZMoW0tDS6devGli1bmDp1KsOGDSMmJoaYmBj69+/P+PHjmTJlClFRUUyYMIEuXbrQvn17wBuabrrpJqZPn85NN93EqlWr+OCDD5g/f35VV6dUio7hsRc+PiI6wrvtdH6B1uEREakkJpOJcYM61qguLYCUlEv8f965cwcA99wzPOAYl8tFTIy3F+Oqq7qzfPm7OBz5bNiwjssv70RyckM2bdrIoEFDWbNmNf36XQ9A+/Yd2bdvL4sWzWf//n0cPPgDGRm7iz3AMyXlYv+ff/hhPwUFBbRs2dq/LSIighYtUv2/9+lzHStWLOcPfxhIo0ZN6Ny5Kz179i42qShUVHngGTx4MCaTiSVLljBlyhTq1avH7bffzl133eU/ZtKkSUyZMoWRI0cC0KNHD8aPH+/f37x5c2bPns20adNYvHgxKSkpTJs2LSTX4IHALq0ImzfUxNXydtedzC3ApcAjIlJpTCZTsQc3h7qiM5MNw3tPePHFeURHB05k8fVidOjQCZvNxubNX7Nx43r69r2OBg0asGzZWxw69DO7du3kySevBuCjjz7gyScn0KfPr2nTJo0bbxzInj0ZzJjx9DnL4Gsm85XFx2otMgmndm1eeeV1vvvuWzZsWMe6dWt4++03GDbsbu64486KfSBBENTAs2TJkhK3Dxo06LzT16Ojo5k8eTKTJ08+5zE9evSgR48eFS5jVYgooUvLH3hOO/1r82gdHhERadzYO1b22LGjtGjR0r/95ZdfxGKxMHz4n7FarXTpks7q1avYuvU7HnlkAnXr1sXtdrNgwcs0adKMBg28i/8uXbqIG24YwEMPjfOf6z//WQV4xzyV1CJ1ySWXYrdH8O233/i71VwuF7t27aRjR++iwB999G9OnTrFzTf/jrS09gwbdjdPPz2ZTz/9KCQDj+6wVSCgS8vXwhPta+FxagyPiIj4NWnSlCuv7M60aU+xevWX/PjjQZYuXcxrry3ioovOLOjbrVsPVq58n7p163HRRSlERETSpk0aH364ku7dr/YfV79+Elu2fMOOHdv58ceDvPnmUpYtews491Iu0dHR3HLL71i48GVWrfqM/fv3MX36Uxw9esR/jNPp4MUXn+ODD1bw888/8c03/2Pz5q9p0yYtSJ9MxVR5l9YvUcCgZWvxFh7feJ5wW0tCRETKZ+LEp5g790WmTZvCqVMnadgwhbFj/8qvf329/5j09Ktwu93+FheATp268PXXG+nW7UzgGTVqNFOnPsnIkXdht9to1qwF48c/wYQJj7B9+1batetQYhnuvnskdnsEM2Y8TW5uLr16/YqrrjrTs3L99QPIzs5m0aL5HD6cSWxsLD179mbEiPtKPF91Mxmh+CyGauB2ezh+vPLXwLFazWzceZTn3/ofAFNHpFM3PopDx3N5ZO5aIu0WGjeIY9v+LO76TSuuaBWag73OxWo1k5BQi6ys02E7FVZ1rPnCvX4Q/nU8u34FBU6OHfuZOnUaYLMVX6y2prJazWF5/YoqSx0vdJ0TE2uVeh0eNSlUgcAurcIWnsIurXynm9P5hQ8WtdasQXYiIiI1hQJPFYgo8iytiMJQExVhwVqYSo+f9D4yQ2N4REREgkN32CpQdAyPrXDQsslkIr6WDYCcPG8Lj2ZpiYiIBIfusFXA9ywtm9WMucj0v9jowP5ItfCIiIgEh+6wVaBWlLclJ+qshbB8M7V8FHhERESCQ9PSq8BF9WK4sVtjkhIDn+SuwCMiIlI1FHiqgMlk4uaeTYtNw4s/O/BoDI+IiEhQ6A5bjeI0hkdERKRK6A5bjWILZ2n5KPCIiIgEh+6w1SheLTwiIhKCXC4Xb7651P/7ggUvc8stN1T6+wTrvCXRHbYanT1o2aoxPCIiEgI+/vgDZs2aWd3FqFS6w1ajooHHajFjKrJGj4iISHUJx8dsapZWNaoVZcNsMuExDHVniYhUEsMwwOWsnje32sv1j9c1a75i/vw57Nu3l6ioKNLTr+Leex9g9+6djBr1f0yc+DfmzJlFZmYmbdq05dFHH+eNN5bwwQcrsFpt/Pa3tzJ06DD/+f7973/x978v5cCBH0hMTOT6629kyJA7sFi868FlZh7i5ZdfZOPG9eTmniYtrT333PMXmjVrzsqV7zNlyhMAdOvWieefn+M/72uvLeIf/3iL7OxsWrduw+jRj3LxxZcAkJOTw4svPsd//vM5BQUFpKZexj333EfLlq38r//nP5fxxhuvcvjwETp37kKDBg3L9TGXhwJPNTKbTMRG28g+7VTgERGpBIZhkLv8STyZu6vl/S1JzYn6zSNlCj0nTpzg0UcfZuTIUXTv3p2ffz7EpEkTmD37Ofr0+TVut5tXX13IhAmTcblcPPzw/dx++21cf/2NzJ27mI8++jfz5r1Et25X07RpM95663XmzHmBkSNH0blzV7Zu/Y4ZM54mOzubv/zlQXJzTzNixDAaNryIv/3tGWw2OwsXzmXkyDtZtOgNevf+FTk5OTz//DP8858fEBcXz+bNmzh06Ge2bPmGadOeo6DAyaRJj/G3v03ixRfnYRgGDz98H3Z7JE8//SwxMTF88MEKRowYxssvv0KLFi35+OMPmDHjaUaNepiOHTvz5ZefM3fubOrXTwriFTlDd9lq5uvW0ho8IiKVw0TNGh5w5EgmTqeTpKRkGjRoSFpae55+egY33/x7/zHDh/+Zli1b0aZNGpdf3pmoqCjuuec+LrnkUoYMuR2APXt2YxgGr722mIEDf8fAgb/l4osvoW/f6xg27M+8++7b5OTk8OGH/yY7+wSTJj1Nq1ZtaN68BY8/PpmIiEiWLXuLiIhIYmJiAKhTpy42m3dGsdVq5bHHJtGsWXMuu6w1N944kO3btwKwadMGvvtuC5MmPUXr1m249NJG3H33/9G6dVvefvvvALzzzptce20fbrnld1xyyaUMHnw7V13Vvco+Z7XwVDN/4FELj4hIhZlMJqJ+80iN6tJq3jyVa6/ty5gxo6hbty6dOnXlyiu706NHT7799n8ApKRc7D8+KiqKBg0a+t8nIiISgIKCAk6cyOL48WOkpbUPeI8OHTricrnYv38fGRm7ufjiS0lISPDvj4iIpFWr1mRkZJyznImJdahVK8b/e2xsHA6HA4CdO7djGAY333x9wGucTqf/mD17dnPttX0D9rdpk8auXTtL8zFVmAJPNfMtPqjAIyJSOUwmE9giqrsYZfL440/ypz/dyfr1a1i3bi2TJv2VtLT2/nE5Vmvg7fpcoepcg409HqPIec51jAer1VLiPgCz+dz3KY/HQ61atViw4LVi+3wtRGDCMAKfOHB2vYJJd9lqFq8WHhGRX7Tvv/+O559/hksuacSttw5i2rTnGDfuMTZt2kBWVlaZzpWYWIfExDr+liGfb77ZjM1m46KLUmjatDkHDuwnK+u4f7/D4WD79m00atQEOHegOpcmTZpx+vRpCgoKSEm52P+zdOliVq9eBUDz5i349ttvAl63ffu2Mr1PReguW818qy3bFXhERH6RatWqxbJlbzN79vMcOPADe/bs5tNPPyIl5RJq165d5vP94Q9DWLbsLd599x0OHjzARx99wMKFc/nNb24iJiaGX/2qH/HxtfnrX8eybdv37N69i4kTx5OXl8eNNw4EvN1m4A0kDkf+Bd+za9d0mjdvwYQJ4/j6640cPHiAWbNmsHLl+/4QNXjw7Xz55ee89tpiDhz4gXfe+TtffPFpmetXXurSqmb14r1/qWLOWnVZRER+GRo1asyTT07jlVfm8e67b2M2m+nYsTPPPPM8mZmHyny+P/xhMHa7jTfffJ3nnptO/fpJDBo0lNtuGwJATEwMs2a9zAsvPMtf/nIPAGlp7XjppQU0bHgRAB07dqZVqzaMGPEn/vrXSRd8T4vFwsyZs5k9+zkee2wseXl5NGrUhCefnMbll3cG4MoruzFhwmQWLpzL3Lkv0bp1W269dTAff/xBmetYHiYjHFcXKge328Px46cr/bxWq5mEhFpkZZ0u9rR0AJfbw5rvDnHZpQnUrR1V6e8fbBeqXzhQHWu+cK8fhH8dz65fQYGTY8d+pk6dBths4fMPRqvVHJbXr6iy1PFC1zkxsRaWUs5yVgtPNbNazHRvV3ULL4mIiPwSaeCIiIiIhD0FHhEREQl7CjwiIiIS9hR4REREJOwp8IiISI2licbhrTKvrwKPiIjUOBaL9xEITqejmksiweS7vhZLxSeVa1q6iIjUOGazhaioGHJyvI9esNsjyvw4hFDk8Zhwu8O71ao0dTQMA6fTQU5OFlFRMed9jldpKfCIiEiNFBeXCOAPPeHAbDbj8YT3woNlqWNUVIz/OleUAo+IiNRIJpOJ+Pg6xMYm4Ha7qrs4FWaxmIiPjyY7OzdsW3nKUkeLxVopLTs+CjwiIlKjmc1mzOaa/3gJq9VMZGQkeXnusH28RHXWUYOWRUREJOwp8IiIiEjYU+ARERGRsGcytGoT4J0C5/EE56OwWMy43eHZHwvhXz9QHcNBuNcPwr+O4V4/UB3Lymw2lXo5AgUeERERCXvq0hIREZGwp8AjIiIiYU+BR0RERMKeAo+IiIiEPQUeERERCXsKPCIiIhL2FHhEREQk7CnwiIiISNhT4BEREZGwp8AjIiIiYU+BR0RERMKeAo+IiIiEPQUeERERCXsKPEHi8Xh4/vnn6d69O+3bt+fOO+/kwIED1V2scjtx4gSPPfYYPXr0oGPHjvzhD39g48aN/v133HEHqampAT9DhgypxhKXXWZmZrE6pKamsmzZMgC2bdvG4MGDad++Pb169eLVV1+t5hKXzbp160qsX2pqKr179wbgpZdeKnF/TfDyyy8X+zt3oWtW076nJdXxs88+4+abb6ZDhw706tWLp59+mvz8fP/+TZs2lXhN161bV9XFv6CS6jd+/PhiZe/Vq5d/f02/hkOGDDnn9/K9994DwO12k5aWVmz/rFmzqqkWxV3oHrFmzRoGDhxIu3bt6NevHytWrAh4vcPh4IknniA9PZ0OHTrw4IMPcvz48cotpCFBMWvWLKNr167G559/bmzbts3405/+ZPTp08dwOBzVXbRyueOOO4zrr7/e2LBhg7Fnzx7jiSeeMNLS0oyMjAzDMAwjPT3deP31143Dhw/7f7Kysqq30GX0xRdfGG3btjUyMzMD6pGXl2ccP37c6Nq1qzFu3Dhj9+7dxjvvvGO0bdvWeOedd6q72KXmcDgC6nX48GHjo48+MlJTU/31+Mtf/mI8/PDDxY4Lda+99prRsmVLY/Dgwf5tpblmNel7WlIdN2zYYFx22WXGSy+9ZOzdu9f44osvjB49ehhjx471H7N06VLj2muvLXZNQ62OJdXPMAzjlltuMWbMmBFQ9mPHjvn31/RrmJWVFVC3zMxM47bbbjP69+9v5OTkGIZhGLt37zZatGhhbNu2LeBY3/5QcL57xO7du422bdsaM2bMMHbv3m3Mnz/faNWqlfHf//7X//qxY8ca1157rbFhwwbjm2++MQYMGGAMGjSoUsuowBMEDofD6NChg7F06VL/tuzsbCMtLc14//33q7Fk5bNv3z6jRYsWxsaNG/3bPB6Pce211xrPPvuscfToUaNFixbG999/X42lrLi5c+caN9xwQ4n75syZY3Tr1s0oKCjwb3vmmWeMPn36VFXxKt3p06eNa665JuDm+Otf/9p45ZVXqq9QZXTo0CHj7rvvNtq3b2/069cv4EZyoWtWU76n56vjgw8+aNx+++0Bx7/77rtG69at/Tf8CRMmGH/+85+rtMxlcb76eTweo3379sZHH31U4mvD4RqebcmSJUabNm38/5g0DMNYsWKF0bFjx6ooarlc6B7x17/+1bjlllsCXvPAAw8Yf/rTnwzD8H4+LVu2NL744gv//j179hgtWrQwvv7660orp7q0gmD79u2cPn2a9PR0/7a4uDhatWrFhg0bqrFk5ZOQkMDcuXNp27atf5vJZMJkMnHy5El27NiByWSicePG1VjKituxYwdNmzYtcd/GjRvp0qULVqvVv+2KK65g3759HD16tKqKWKnmzJlDXl4eY8aMAcDpdLJv3z6aNGlSzSUrve+//x6bzcby5ctp165dwL4LXbOa8j09Xx3/9Kc/+a+fj9lspqCggJycHOD8f69Dwfnq98MPP5Cbm3vOv5PhcA2LOn78OM8++ywjRowIqHOoX8ML3SM2btwYcI3A+13ctGkThmGwadMm/zafxo0bk5SUVKnX0XrhQ6SsDh06BECDBg0CttevX9+/ryaJi4vj6quvDtj24Ycfsn//fh555BF27txJbGwsEydO5KuvviI6Opp+/fpxzz33YLfbq6nUZbdz504SEhIYNGgQe/fu5dJLL2XEiBH06NGDQ4cO0aJFi4Dj69evD8DPP/9M3bp1q6PI5Xb8+HEWLVrEgw8+SO3atQHYvXs3brebDz/8kCeffBKHw0Hnzp15+OGH/XUNNb169QoYz1HUha5ZTfmenq+OrVq1Cvi9oKCARYsW0aZNGxITEwHYtWsXCQkJDBw4kMzMTFq0aMGoUaNIS0sLetlL43z127lzJwBLlizhyy+/xGw206NHD0aNGkVsbGxYXMOi5s2bR2RkJMOGDQvYvnPnTlwuF8OGDWP79u0kJSUxdOhQbrzxxmAVuUwudI949913SU5ODthfv3598vLyyMrKIjMzk4SEBCIiIoodU5nXUS08QZCXlwdQ7GYfERGBw+GojiJVqq+//ppx48bRp08fevbsyc6dO3E4HKSlpTF//nxGjBjB22+/zfjx46u7qKXmcrnYs2cP2dnZ3HvvvcydO5f27dtz1113sWbNGvLz80u8nkCNvKavv/46sbGx/P73v/dv891coqKieO6553jyySfZs2cPf/zjHwMGwdYUF7pm4fY9dblcjB49ml27djFhwgTAG+xOnTpFbm4u48ePZ/bs2dStW5fBgweze/fuai7xhe3cuROz2Uz9+vWZM2cOY8eOZfXq1dxzzz14PJ6wuoY5OTm89dZbDBs2rNiNf9euXZw4cYIhQ4awYMEC+vbty7hx43jnnXeqqbTnd/Y9oqTvou93p9NJXl5eif84ruzrqBaeIIiMjAS8F9L3Z/D+TzYqKqq6ilUpPvnkEx566CE6duzI9OnTAZg4cSJjxowhPj4egBYtWmCz2Rg1ahSjR4+uEa0fVquVdevWYbFY/NesTZs27Nq1iwULFhAZGYnT6Qx4je+LGB0dXeXlraj33nuPAQMGBPz9HDBgAD169PC3DAA0b96cHj168Nlnn3HddddVR1HL7ULXLJy+pzk5Odx///2sX7+eF154wd9606BBAzZs2EBUVBQ2mw2Atm3bsnXrVpYsWcITTzxRncW+oBEjRnDbbbeRkJAAeP/fUq9ePX73u9+xZcuWsLqGn3zyCU6nk5tvvrnYvn/961+43W5q1aoFQMuWLfnpp59YsGABt9xyS1UX9bxKukdEREQU+y76fo+KiirxuwqVfx3VwhMEvubVw4cPB2w/fPgwSUlJ1VGkSvHaa69x7733cs011zBnzhz/v0KsVqs/7Pg0b94cIKSalS+kVq1aAf/TBG89MjMzSU5OLvF6AjXumm7fvp0DBw5www03FNtXNOyAt0m5du3aNeo6+lzomoXL9/Tw4cMMGjSI//3vfyxYsKBY10JcXJw/7IB3jE/Tpk3JzMys6qKWmdls9ocdn6L/bwmXawjeoHD11VcTFxdXbF9kZKQ/7Pi0aNEi5L6X57pHNGjQoMRrFB0dTWxsLMnJyZw4caJY6Kns66jAEwQtW7YkJiYmYJ2LkydPsnXrVjp37lyNJSu/119/nUmTJjFo0CBmzJgR0Pw4ZMgQxo0bF3D8li1bsNlsNGrUqIpLWj67du2iY8eOxdYm+e6772jWrBmdO3dm06ZNuN1u/761a9fSuHFj6tSpU9XFrZCNGzdSp04dWrZsGbB95syZ9O3bF8Mw/NsOHjxIVlYWzZo1q+piVtiFrlk4fE+zs7MZOnQox48fZ+nSpcXK/eWXX9KhQ4eAdWlcLhfbt2+vEdd09OjR3H777QHbtmzZAkCzZs3C4hr6lDSwF7z16dKli389MJ8tW7b4w18oON89olOnTqxfvz7g+LVr19KxY0fMZjOXX345Ho/HP3gZYO/evWRmZlbqdVTgCQK73c7gwYOZPn06n376Kdu3b2fUqFEkJyfTp0+f6i5eme3du5cpU6bwq1/9irvvvpujR49y5MgRjhw5wqlTp+jbty///Oc/eeONNzhw4AArV65k6tSpDBs2jJiYmOoufqk0bdqUJk2aMHHiRDZu3EhGRgZPPfUU//vf/xgxYgQ333wzOTk5PProo+zevZtly5axaNEi7r777uoueplt3bq1xMUEf/WrX/Hjjz/y+OOPs3fvXjZs2MC9995Lx44d6d69ezWUtGIudM3C4Xv61FNPceDAAaZNm0ZiYqL/e3nkyBHcbjcdO3YkISGBMWPG8N1337Fjxw7GjBnDiRMnigWJUNS3b1/WrFnDCy+8wA8//MCqVat45JFHuP7662natGlYXEPwjrXKysoq9o8Q8LbQXXHFFcycOZNVq1axb98+5s6dy/Lly7n33nurobTFXegeMWTIEL799lumT59ORkYGCxcu5IMPPmD48OGAt8W1f//+jB8/nnXr1vHtt9/ywAMP0KVLF9q3b19p5dQYniC57777cLlcjB8/nvz8fDp37syCBQsCmpZrig8//JCCggI+/vhjPv7444B9N910E3/7298wmUwsWbKEKVOmUK9ePW6//Xbuuuuuaipx2ZnNZubMmcMzzzzD/fffz8mTJ2nVqhWvvPKKf6bP/PnzefLJJ7npppuoV68eo0eP5qabbqrmkpfdkSNH/DOzimrTpg3z5s3jueeeY+DAgdjtdnr37s2YMWMwmUxVX9AKqlOnzgWvWU3+nrrdblauXElBQQFDhw4ttv/TTz8lJSWFRYsWMX36dIYNG4bD4eDyyy/ntddeqxFj63r37s2zzz7L3LlzmTdvHrGxsdxwww3cf//9/mNq8jX0OXLkCECJ30uAKVOmMGvWLCZMmMCxY8do2rSpf3XpUFCae8Ts2bOZNm0aixcvJiUlhWnTpgW0aE2aNIkpU6YwcuRIAHr06FHpE19MRtH2axEREZEwpC4tERERCXsKPCIiIhL2FHhEREQk7CnwiIiISNhT4BEREZGwp8AjIiIiYU+BR0RERMKeAo+IiIiEPQUeEQkLBw8eJDU1tdgzh8pj7Nix9OrVqxJKJSKhQo+WEJGwUL9+fd58800uueSS6i6KiIQgBR4RCQt2u71SHzQoIuFFXVoiUiXefvtt+vfvT5s2bejZsyezZs3C7XYD3i6kIUOG8M4773DNNdfQoUMHhg4dyvbt2/2v93g8zJw5k169etGmTRt69erFM888Q0FBAVByl9a+ffu47777uOqqq2jfvj1Dhgxh06ZNAeXKzs5m3LhxdOnShc6dOzNt2jQ8Hk+x8n/yyScMHDiQtm3bctVVVzF58mRyc3P9+/Pz83n88cfp0aMHbdq0oV+/fixYsKBSP0MRKT+18IhI0L388svMnDmTwYMHM27cOLZt28asWbP4+eefmTJlCgDbtm1jz549PPDAA8THx/P8888zePBgVq5cSf369Zk3bx5vvPEGY8aM4eKLL+abb75h5syZ2Gw27rvvvmLvuXv3bn73u9/RqFEjxo8fj81m49VXX2Xo0KEsXLiQLl264PF4GD58OD/++CNjxoyhdu3azJ8/ny1btlC/fn3/ud5//30eeugh/5O6f/zxR2bOnMnu3bt55ZVXMJlMTJkyhdWrVzNmzBjq1q3Ll19+ydSpU6lduzY333xzlX3WIlIyBR4RCapTp04xe/Zsfv/73zN+/HgAunXrRu3atRk/fjx33HGH/7g5c+bQqVMnANLS0rj22mt59dVXeeihh1i/fj1t2rTxh4cuXboQFRVFbGxsie/7wgsvYLfbefXVV4mJiQGgZ8+eXH/99UydOpV33nmHL7/8km+//ZZ58+bRo0cPANLT0wMGLBuGwfTp0+nevTvTp0/3b2/UqBG33347q1atomfPnqxfv56rrrqK/v37A9C1a1eio6OpU6dOZX6cIlJO6tISkaDavHkz+fn59OrVC5fL5f/xhYqvvvoKgJSUFH/YAe8g5A4dOrBhwwbAGyC++uorbrvtNubPn8/u3bsZPHgwN954Y4nvu379eq655hp/2AGwWq3079+f7777jtOnT7Nx40ZsNhvdu3f3HxMdHc3VV1/t/33Pnj0cOnSoWPk7d+5MTEyMv/xdu3blrbfe4s477+S1117jwIED/N///R89e/asnA9SRCpELTwiElQnTpwA4K677ipx/+HDhwFISkoqtq9OnTp8//33AAwfPpxatWrxj3/8g+nTpzNt2jSaN2/O+PHjueKKK4q9Njs7m7p16xbbXrduXQzDICcnh+zsbGrXro3JZAo4pl69esXK/8QTT/DEE0+cs/yPPvooycnJLF++nEmTJjFp0iQ6dOjA448/TsuWLUusu4hUHQUeEQmquLg4AKZPn06jRo2K7a9bty7PPfccWVlZxfYdPXrU3yVkNpsZNGgQgwYN4tixY6xatYo5c+Zw7733+ltZioqPj+fo0aPFth85cgSAhIQEEhISyMrKwu12Y7FY/Mf4Qk7R8o8ePZouXbqU+D7gnSU2YsQIRowYwU8//cTnn3/O7NmzefDBB1mxYsW5Ph4RqSLq0hKRoGrXrh02m43MzEzatm3r/7FarcyYMYODBw8C3hlVGRkZ/tdlZmayefNm0tPTAbj11luZPHky4G35GThwIIMGDeLkyZPk5OQUe9/OnTvz+eefB+xzu92sWLGCtm3bYrfbSU9Px+Vy8cknn/iPcTqdAQGqSZMm1KlTh4MHDwaUPykpiWeeeYatW7eSn59P3759WbhwIQANGzZk0KBB9O/fn59++qkSP00RKS+18IhIUCUkJDB8+HCee+45cnJy6Nq1K5mZmTz33HOYTCZ/d49hGPz5z39m1KhRWCwWXnjhBeLj4xkyZAjgDTALFy6kbt26dOjQgczMTF555RW6dOlCYmJiwBRxgJEjR/Lll1/yxz/+kbvuugubzeYfWzN//nzAO0C5W7dujB8/nmPHjnHRRRfx6quvcvz4cX/LksViYdSoUTz22GNYLBauueYaTp48yezZs8nMzKR169ZERkbSunVrXnjhBWw2G6mpqezdu5d3332Xvn37VuGnLSLnYjIMw6juQohI+Fu6dCmvv/46+/fvJz4+nvT0dB544AEaNmzI2LFjWb9+PXfeeScvvvgieXl5XHnllYwZM4aUlBQAXC4XL730EsuXL+fQoUPExsbSq1cvHnzwQRISEjh48CC9e/fmqaeeYuDAgYB3qvuMGTPYuHEjJpOJtLQ0Ro4cGTA4Oi8vj+nTp7NixQocDgfXXXcd0dHRfPrpp3z22Wf+41auXMn8+fPZtWsX0dHRdOzYkfvvv5/U1FQAcnJyePbZZ/n00085cuQIderU4brrruMvf/kLkZGRVfhJi0hJFHhEpNr5Ak/RgCEiUpk0hkdERETCngKPiIiIhD11aYmIiEjYUwuPiIiIhD0FHhEREQl7CjwiIiIS9hR4REREJOwp8IiIiEjYU+ARERGRsKfAIyIiImFPgUdERETC3v8DqXHNDupu+GEAAAAASUVORK5CYII=", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjMAAAHJCAYAAABws7ggAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAACdY0lEQVR4nOzdd3xT5f4H8M/JHm3adC/2KEIZLVsEBRW4qIi4rgLi9rpQ4CfjykUQcYE4AEUEFUG8KKL3KrgXV2SDIrJ3927TNjvn/P5Iz2nSmaQZJ+33/XrxKk1OkudJTtJvnuf7fR6G4zgOhBBCCCFhShLqBhBCCCGEtAYFM4QQQggJaxTMEEIIISSsUTBDCCGEkLBGwQwhhBBCwhoFM4QQQggJaxTMEEIIISSsUTBDCCGEkLBGwQxpVKDWUqQ1GgkhhPgbBTOkgR9++AFz5871+/0ePHgQDz74oPB7Tk4O0tPTsW3bNr8/Fmkb9uzZg3HjxiEjIwP3339/o8dMmzYN6enpwr9evXohMzMTkydPxgcffAC73e7X49PT05GRkYGrrroKixcvRmVlZYM2FRQUYPny5bj++uuRmZmJzMxM3HTTTVi7di1MJlOL/d6+fTtGjx6NjIwMLFy40ItnzDP1+5Oeno5+/frhuuuuwzvvvAOWZf3+mM21ZeXKlQG/TShs2LAB1157Lfr164ebbroJv/zyS6ib1GbJQt0AIj7vv/9+QO73k08+wdmzZ4XfExISsGXLFnTs2DEgj0fC38svvwyWZbF27VrExsY2eVzv3r3xzDPPAAAcDgcqKyuxc+dOvPDCCzhw4ABee+01SCQSvxwPADabDX/99RdWrFiB48eP46OPPgLDMACAvXv3YsaMGYiKisKdd96J9PR0sCyLvXv34q233sK3336LDz/8EEqlssn+PPvss+jcuTNefPFFJCYm+vbkteCWW27BrbfeKvxuMpnw7bffYvny5TAYDJg9e3ZAHre9eO+997Bs2TI8+uijyMjIwKeffoqHH34YH3zwAQYNGhTq5rU5FMyQkFEoFBgwYECom0FErKKiAoMHD8bll1/e7HERERENzqUxY8aga9euWLp0Kb788ktMnDjRr8cPHjwYNTU1eOONN/DHH39gwIABKCsrw8yZM9G5c2e899570Gg0wvEjRozA1VdfjTvuuAMbNmxwG6VsrN8jRozA0KFDm+13ayQlJTXo0/Dhw3Hu3Dl8+OGHmDFjBuRyecAevy0zm8148803cc899+DRRx8FAIwaNQp///vfsXr1arz33nshbmHbQ9NMxM20adOwb98+7Nu3D+np6di7dy8A54frwoULcfnll6Nv37647bbbsHv3brfb7tq1C7fddhsyMzMxePBgPPzww8JIzLx58/DZZ58hNzdXmFqqP820bds29O7dG3/88Qduv/129O3bF6NHj8b69evdHqeoqAgzZ87EkCFDMHjwYCxcuBCvvvoqxowZ02zfioqKMHfuXAwfPhyZmZmYOnUqDh8+DKDpKa958+a53e+0adPwf//3f5gxYwYGDBiAe+65B+PGjcOMGTMaPN6NN96Ihx9+WPj9+++/x+TJk9G3b1+MGDECzz33HIxGY7Ntdjgc+PDDD3HDDTegX79+uOqqq7B8+XJYLBa3Nt5999349NNPhSmZG2+8ETt37mz2vgHg888/x0033YT+/fvjqquuwiuvvAKr1QoAWLlyJcaMGYOffvoJ48ePR//+/XHbbbcJ5wTgfM3S09ORk5Pjdr9jxozBvHnzmn3sCxcuYMaMGRgxYgQGDBiAadOm4eDBgwDqXo/c3Fx8/vnnbueiN6ZOnYrExET8+9//DsjxGRkZAIC8vDwAwObNm1FaWornnnvOLZDh9e/fH9OnT2/0OsA5qpOeng4AWL16tdtzu2vXLtx5550YOHAghg4ditmzZyM/P1+4Lf/++eSTTzBixAgMGTIEZ86c8agf9ftUU1MjTJ/xI2PXXnstMjIyMG7cOGzcuNHtNtOmTcPTTz+NtWvX4qqrrkLfvn3x97//HUeOHHE7bt++fbj99tvRv39/jBs3Dr/99luj/a//Wk+bNg3Tpk1rtL2enoPp6en46KOPMG/ePAwcOBBDhgzBc889B7PZjJdeegnDhg3D0KFD8fTTT7u9v+o7dOgQ0tPT8dNPP7ldfvz4caSnp+O7777DH3/8AYPBgGuvvVa4nmEYXHvttdi7dy/MZnOT9098Q8EMcfPMM8+gd+/e6N27N7Zs2YI+ffrAYrFg+vTp+OGHHzBz5kysWrUKSUlJuP/++4WAJjs7G4888ggyMjLw1ltvYenSpTh//jwefPBBsCyLRx55BFdeeSXi4+OxZcsWXHXVVY0+PsuyePLJJzFhwgSsXbsWWVlZePnll/G///0PAGC1WjF9+nQcOnQI//znP/HCCy/gxIkTePfdd5vtV01NDe644w7s3bsXTz31FFatWgWlUol7770XFy5c8Oo5+uqrr6DVavHWW2/h/vvvx8SJE/HLL7+gurpaOObs2bM4ceIEbrzxRgDAF198gUcffRRdu3bF6tWr8dhjj+G///0vHnnkkWaTohcuXIgXXngB11xzDd566y1MmTIFmzZtanC7o0ePYv369ZgxYwZWr14NqVSKxx9/vNF8Dt6HH36IuXPnok+fPli1ahUefPBBbNy4Ec8995xwTFlZGebOnYs777wTr7/+OlQqFe677z4cP37cq+esvjNnzmDy5MnIycnBggULsHz5cjAMg+nTp2Pfvn3CFGR8fDyuvPJK4Vz0lkQiwfDhw3HkyJEGuTD+OP78+fMAgA4dOgBw5pulp6ejR48eTd5m7ty5mDp1aqPX9enTB1u2bAHgnAbasmULEhIS8Pnnn+Pee+9FcnIyVqxYgfnz5+Pw4cO4/fbbUVpaKtze4XDg3XffxdKlSzF//nx069atxT401ietVitM6y1atAhvvPEGJk6ciDVr1mD8+PF4/vnnsXr1arfbffPNN/jhhx+wYMECrFixAiUlJXj88cfhcDgAAH/99RfuvfdeREZG4o033sBdd92FWbNmed2+1li2bBkUCgVWrVqFSZMmYePGjZg0aRLy8/OxfPlyTJs2DVu3bm0QrLnKyspCx44dsX37drfLv/zyS0RHR+PKK68UvsR17tzZ7ZhOnTrB4XDg0qVLfu9be0fTTMRN9+7dERERAQDCEPTHH3+MEydO4OOPP0b//v0BOIdMp02bhuXLl+PTTz/FkSNHYDab8dBDDwlz/ElJSfjhhx9gNBrRsWNHxMTEuE0tNTYqwXEcHnnkEWEuf+DAgfjuu+/w888/Y+TIkfjvf/+Lc+fO4dNPPxW+FQ8bNgzXXHNNs/3iR4U+++wzXHbZZQCcH0qTJk3C/v37MXz4cI+fI7lcjsWLF0OhUAAAOnbsiJUrV+L777/HpEmTADg/2HQ6HcaMGQOO47B8+XKMHDkSy5cvF+6nc+fOuPvuu/HLL780GtydOXMGW7duxezZs4UpiREjRiAhIQFz5szBzp07ceWVVwIAqqqqsG3bNiH/SKPRYOrUqUICbX0sy2L16tW45ppr3IIXk8mE7du3w2azCb8vWrRI6Bf/XK9duxavvvqqx89ZfatWrYJCocAHH3wgnG9XXXUVrr/+erz88svYunUrBgwYAIVCgZiYmFZNR8bFxcFms6GiogJxcXE+Hc9xnFtwU1lZiX379uGtt95CZmamcC5eunQJI0aMaHCfjQVGMlnDj1/X6Sx+GohlWSxfvhxXXHEFXnnlFeHYrKwsTJgwAevXr8ecOXOEy//xj380+WXBFcuyQrs4jkNJSQm++OIL/Pjjj7j//vvBMAzOnz+Pjz/+GLNmzRLOwSuuuAIMw+Dtt9/GnXfeCb1eL/Rx/fr1wutZU1ODuXPn4vjx48jIyMDbb7+N2NhYvPXWW8L0lV6vx8yZM1tsq790794dzz77LABgyJAh+OSTT2Cz2bB8+XLIZDJcccUV+Oabb3Do0KFm72fixIl49913YTaboVKpwHEcduzYgfHjx0OhUAhfbPjngqfVagHA7YsP8Q8amSEt2r17N+Lj49GnTx/Y7XbY7XY4HA6MHj0aR48eRWVlJfr37w+lUolbbrkFS5cuxf/+9z/06tULM2fObPCGbklmZqbwf/6PGR/47NmzBx06dBD+eADOD4zRo0c3e58HDx5EWlqaEMgAgFqtxjfffOOWBOmJrl27CoEM4PxWnpWVhR07dgiXbd++XfhgO3fuHAoKCjBmzBjh+bPb7Rg8eDAiIiKwa9euRh9n3759AIDrrrvO7fLrrrsOUqnUbSg+JibGLZE6KSkJAJqsnDl//jxKS0vdhsEB4L777sO2bduEPzYymQzXX3+9cL1KpcKoUaOwf//+pp8gD+zbtw+jR492OzdkMhmuu+46HD16FDU1Na26f1f8CBafoOvL8fv370efPn2Ef5dffjlmzZqFjIwMvPLKK8KxjVUB2e12t9vy/zx1/vx5FBcXu70OgDOIzszMFM4Tnus53pw333xTaAtfnbV69WrcfvvtePzxxwE4328cxzU4d8eMGQOLxSJMCwLuX4QACF9q+HPw4MGDGDlypFseztixYyGVSj1+LlrL9bNFKpVCr9ejT58+boFldHQ0qqqqANQFfK6fe4AzmDEajcJU06FDh5CXlyeMxLZUDeaaXE78g0ZmSIsqKipQXFzc5AdwcXExunfvjk2bNmHt2rXYunUrPvjgA+h0Otx555148sknPf5DAjj/YLqSSCTCH5jy8vJGq1qaq3Th+9DSMZ7iv125uvHGG7FkyRKUl5cjJycHFy9exPPPPy88NgAsXrwYixcvbnDboqKiRh+HnyKKj493u1wmk0Gv1wsfuIAzMHPV3B9X1za19JzExcU1GEGIjY0Vbu+rysrKRkdJ4uLiwHEcqqurG32efVFYWAiVSoXo6Gifj+/Tp4/w2jEMA6VSieTk5AaBempqKnJzc90uk8lk2Lp1q/D7xx9/jI8//tjj9vPPdVPP17Fjx9wuayofp77bbrsNt912GwBnn7RaLdLS0tyCDf6x6wfUvMLCQuH/9c9B/g82fw5WVlYKozg8/lwOlsa+WDX3fP3zn//EZ599JvyempqKH3/8EZ06dUJmZia2b9+Ov/3tb9i+fTs6duyIrKwsAEBkZCQA5+hUVFSUcHt+RIa/nvgPBTOkRZGRkejcubPbFImrtLQ0AEC/fv2watUqWK1WHDx4EFu2bMGaNWvQq1cv/O1vf/NLWxITExvNcXHNG2hMZGRkgwRBwPmNKioqSgig+G9evJYSdHl/+9vf8Nxzz+H777/HuXPnkJqaioEDBwIAdDodAGDOnDkYMmRIg9u6ftg1dnlxcTFSU1OFy202G8rLy1v1R4BvU1lZmdvl5eXlOHbsmPANtrGgpaSkRAiCmgqaWhpZiYqKQklJSYPLi4uLAcBvf+Dsdjv27t2LrKwsj0YAmjpeq9Wib9++Ld5+zJgxWLt2LbKzs4U8GgBut/3555+96gMfVDX1fPn6XCUkJLTYJ/482bBhQ6PBZUpKisePFx0d3aAPHMe55XU1dz41Fdz6eg564rHHHsOUKVOE311HZCdOnIgXXngBVVVV+Prrr3HHHXcI13Xp0gUAcPHiRfTr10+4/OLFi5DL5W7nBvEPGusiDdQfAh0yZAjy8/MRGxuLvn37Cv927dqFdevWQSqV4v3338fo0aNhtVqhUCgwfPhwLFmyBEBdpYc/hlaHDBmCnJwctwRUs9ksJAg3ZdCgQcjOzsbp06eFyywWCx5//HFs3bpV+Mbm+k3TZrM1qMZoik6nw+jRo/HDDz/gm2++wcSJE4UP2a5duyI2NhY5OTluz19iYiJeeeWVBt+sXfsKoEGi4fbt2+FwOIRgyRddu3aFXq9vUJHxn//8Bw8++KCQM1P/uTWbzdi5c6eQY8Q/bwUFBcIxZ8+ebXHkZvDgwfjpp5/ccgccDge2b9+Ovn37uv3RaI0tW7aguLjY7Q+NP4+vb8qUKYiOjsa8efMazYtwOBw4d+6cV/fZpUsXxMfH48svv3S7PDs7G7///rswGhAI/Hoo5eXlbuduWVkZXn/9da9G6IYPH46dO3e6TX3+73//E841oPHzqbKy0m19qvp8PQc9kZaW5tZvvtIMACZMmACO4/D666+jtLTUrZQ/MzMTGo0G33zzjXAZx3H47rvvMGTIEL+d36QOjcyQBnQ6HQ4fPozdu3ejd+/emDx5MjZt2oR77rkH//jHP5CcnIzffvsN77zzDqZOnQq5XI5hw4Zh+fLlePTRRzF16lRIpVL8+9//hkKhEPJZdDodSkpK8Msvv3g8r1/f9ddfj7Vr1+LRRx/FE088AZ1Oh/feew+lpaXNfkucPHkyNm7ciIcffhgzZsyAXq/HBx98AJvNhjvvvBNRUVHIzMzExo0b0alTJ0RFReGDDz6A2Wz2eNh+4sSJmDFjBhwOhzB3Djjn5mfOnImFCxdCKpVi9OjRMBgMePPNN1FYWNjk9F337t1x00034Y033oDJZMLgwYNx/PhxrFq1CkOHDsXIkSO9e/Jc8NVOzz77LGJjYzFmzBicP38eb7zxBqZMmeI2WjR//nw8+eSTiI2Nxfr162E0GoWS86FDh0KlUuHFF1/EE088Iay70tKUzmOPPYadO3firrvuwoMPPgi5XI5NmzYhOzsb69at87o/1dXV+P333wE4v6GXl5fj119/xZYtWzBx4kSMHTu2Vcd7KjExEatWrcITTzyBiRMn4vbbb0efPn0gkUhw9OhRfPrpp7hw4YLbH76WSCQSzJo1C/Pnz8fs2bMxceJElJeXY9WqVYiKisI999zjU1s9kZ6ejokTJ+Jf//oXcnNzkZGRgfPnz+PVV19FWlpag2qd5jz66KP4/vvvcd999+H+++9HWVkZXnvtNbdprfT0dCQnJ2P16tWIiIgQEo3rT2G58vUcbC2+cmnz5s3IzMxEp06dhOvUajXuvfderF69GnK5HJmZmfj000/x119/4YMPPghou9orCmZIA1OmTMHRo0fxwAMP4IUXXsANN9yADz/8EK+88gqWLVuGqqoqpKamYvbs2bj33nsBAL169cKaNWuwevVqzJo1Cw6HAxkZGXj33XfRtWtXAM6A4pdffsGjjz6KGTNmYMKECV63TSaTYf369Vi6dCkWLVoEmUyGiRMnIjo6WiiTbUxERAQ2bdqEl19+GUuWLAHLshgwYAA++OADYcj3xRdfxJIlS7BgwQJERETglltuwcCBA/HJJ5941LYrr7wSkZGR6NChgzDMzLv11luh1Wqxbt06bNmyBRqNBllZWVi+fHmzQ85Lly5Fp06d8Omnn+Kdd95BQkIC7rrrLjzyyCOtHumaMmUKNBoN1q9fjy1btiApKQkPPPAAHnjgAbfjFi1ahOeffx5lZWXIysrCRx99JHxw63Q6rFy5Eq+88goeffRRpKam4rHHHsPnn3/e7GP36NEDmzdvFsqMGYZBv379fF4d9dixY7j99tsB1OV/9OzZE4sWLWo0wdvb470xaNAgfPHFF/joo4/w9ddf45133oHVakVycjKGDRuGV199Fb179/bqPidPngytVou3334bjz76KCIiIjBy5EjMmjWrQU6Vv73wwgt4++238e9//xsFBQWIjY3FhAkT8OSTT3qVvNu5c2ds2rQJL774ImbOnInY2FjMnTsXL774onCMVCrFG2+8geeffx6zZs1CXFwcpk+fjnPnzjX5/vb1HPSHG2+8Ed9//z1uuOGGBtc9+uijkEql+Pjjj/Huu++ie/fuePPNN1s1okqaxnC08x8JI6dPn8a5c+cwduxYt6TiW265BUlJSVi1alUIW9e2rFy5EqtWrcLJkydD3RRCCGkWjcyQsGI0GvHEE0/gzjvvxLXXXguHw4EdO3bg6NGj+L//+79QN48QQkgIUDBDwkr//v3x2muvYf369fj888/BcRx69+6NdevWYdiwYaFuHiGEkBCgaSZCCCGEhDUqzSaEEEJIWKNghhBCCCFhjYIZQgghhIQ1CmYIIYQQEtbaTTUTx3FgWf/nOkskTEDuV4zaU1+B9tVf6mvb1Z76S31teyQSxqONittNMMOyHMrKWr/xmCuZTAK9XguDwQi7vfkt38Nde+or0L76S31tu9pTf6mvbVNMjBZSacvBDE0zEUIIISSsUTBDCCGEkLBGwQwhhBBCwhoFM4QQQggJa+0mAZgQQkh4YlkWDoe9hWMYmM1SWK0WOBxtu8qnrfRVKpVBIvHPmAoFM4QQQkSJ4zgYDGUwmao9Or6kRAKWbdvVPby20le1OgI6XYxH5dfNoWCGEEKIKPGBTESEHgqFssU/eFIpE9YjFd4I975yHAer1YLq6nIAQFRUbKvuj4IZQgghosOyDiGQiYjQeXQbmUzS5tdd4bWFvioUSgBAdXU5IiP1rZpyEmUCMMuyeOONNzBy5EgMGDAADzzwALKzs0PdLEIIIUHicDgA1P3BI20T//q2lBPVElEGM2+++SY2b96MJUuW4N///jdYlsX9998Pq9Ua6qYRQggJotbmUhBx89frK7pgxmq14t1338WMGTNw1VVXoVevXnj11VdRUFCAb7/9NtTNI4QQQojIiC6YOXHiBGpqajB8+HDhMp1Oh969e2P//v0hbBkhhBBCxEh0CcAFBQUAgOTkZLfLExIShOt8JZP5N3aTSiVuP9uy9tRXoH31l/radoVzf1nWu+kHfraCYQAufIt8PBLIvj722INITk7B008v8u8dt0AqZVr1N1p0wYzJZAIAKBQKt8uVSiUqKyt9vl+JhIFer21V25qi06kDcr9VRitUChnkfg7CWiNQfRWr9tRfMfTVZndg209nMOiyRHRLiw7Y4wSyrw6Wg1QirjwPMby23jKbpSgpkXj9Ry4cAzdfBaKvDMOAYVoXWHiDZRlIJBJERWmgUql8vh/RBTN8Z6xWq1vHLBYL1Grf35Asy8FgMLa6fa6kUgl0OjUMBhMcDv+WyFVUW/B/q3ehW0oU5k8b6Nf79kUg+ypG7am/Yurr7qMF2PT1CRw4VoB/3jXI7/cf6L6u+fwo/rpQhucfHIZIjaLlGwSYmF5bb1mtltqVfzmPSpAZxtlfh4NtFyMzgeorx3HgOM+ec39wODiwLIvKSiNMJkeD63U6tUdBm+iCGX56qaioCB07dhQuLyoqQnp6eqvuO1AvjsPB+v2+T14sh9XG4uSlCpgtdshE8m0jEH0Vs/bUXzH09VJhFQAgu6gaNpsjYJUsgegry3LYf6IINjuLkxcrMKBHnF/vvzXE8Np6q6kF4TiOg9XWeF9kssD/EVbIJV6fl1dcMQj33PMAduz4Ana7DatWvYOkpGS8885b+Pbbr1BTU40uXbrh/vv/gSFDhuHs2TOYPv3vWL9+E9LTewEA5s//Pxw6tB87dvwIqVQKlmVx3XXX4PHHZ2HcuAn44ovPsXXrv5GdnQ2JhEHPnr0wY8Ys9OrVGwBwyy034KqrrsaePbtQXl6G5557GX369MWaNSvx7bdfw2az4sYbbwZXLzravHkjPv98K4qLixAXF4/rrpuI6dPv8/t709OgtSmiC2Z69eqFiIgI7N27VwhmDAYDjh07hqlTp4a4dcGTW1IDAGA5DoXlJqTGBWaKjBAxyas972vMdhiMNkRpQz+64amSShNstR/GBWX+HQUmThzH4YVNh3Am1/eUg9bqnhaF+VOyvP5j/tlnn2D58jdgtzvQoUNHLFr0NC5ePI+FC5cgPj4Bu3btxJw5T+L555fj8suvQHJyCvbv34P09F5wOBw4fPgAjEYjTp06gcsu64Njx46iqqoKw4dfgV9++Qmvvvoy5s5dgP79M1FSUoLXXluGF198Du+/v1low7ZtH+Oll15FZGQkunbtjtdeW4Zdu/6Hp59+BomJyfjgg3fxxx+HkZKSCgD49ded2LjxPTz77PPo0KEz/vrrCJ577hkkJ6dg3LgJfn1eW0t0wYxCocDUqVOxfPlyxMTEIDU1FcuWLUNSUhLGjh0b6uYFDf+hDgD5JTUUzJB2wfW8zyupCatgJq+kLoDJL61p5kjSKuJKR/LYuHEThFGSnJxsfP/9N3jvvQ/Ro4dzxuHvf5+KM2dOY/PmD3D55VdgxIiR2L9/L6ZOvRvHj/8FmUyOjIy+OHToAC67rA927foV/ftnQqfTISoqCvPm/Qtjx/4NAJCUlIzrr5+IFStedmvDsGEjMHjwUACA0ViDr776ErNnz8Xw4VcAAObPX4hDhw4Ix+fl5UChkCMpKQVJSUlISkpCXFwCEhOTAv58eUt0wQwAzJgxA3a7HQsWLIDZbMbgwYOxfv16yOXyUDctaHJdgxn6lkfaAZudRVGFSfg9v7QGl3XSh7BF3slzCWBoZCYwGIbB/ClZzUwzBX6Jf1+mmQAgLa0ubeLUqZMAgEceud/tGLvdjoiISADAiBEj8d//fgaLxYz9+/di4MBBSEpKwcGDBzBlynTs2vUrxo+/DgAwYEAWLlw4j/ffX4eLFy8gJ+cSzp4902AjyrS0DsL/L126CJvNhl69+giXKZVK9OxZl84xduwEbN/+X9xxx2R07twVgwcPxVVXXY2kJApmPCKVSvHUU0/hqaeeCnVTQsLBsigopW95pH0pLDO6JTO6jtKEA7fR1FIKZgKFYRgoFdJGr5PJJKKrJOMplXXbMnCcM8hYvfodaDTuo+78/kSZmYMgl8tx+PAhHDiwD+PGTUBycjK2bfsYBQX5OH36JJYudY68fPvt11i69BmMHfs3ZGT0w403Tsa5c2exYsVLTbaBH+Li28KTyerCgujoaLz33mYcPXoE+/fvxd69u/HJJx/hvvsewj33PNC6J8TPxJFVStwUlZvgYOs+1fNL6IORtH159YL2cAtmXL90VJtsqDbZQtgaImZdunQDAJSWliAtrYPwb/v2/2LHji8AOIOKIUOG49dff8GxY0cxcOBg9Os3AA6HA+vXv41u3bojOTkFAPDhh+/jhhsm4emnF+Hmm2/DgAFZyM3NAYAGCb28jh07QaFQ4siRP4TL7HY7Tp8+Jfz+7bdf4bPPtqJfvwG4776HsHat83F++EF8q/FTMCNCucXOD8UItXNaLb+sBmxbrzUk7R4fvKTFRwAIr9ENjuOEnBl+ZKAgjNpPgqtr1264/PKRWLbsBfz6607k5ubgww83YNOm95GamiYcd8UVo7BjxxeIi4tHamoalEoVMjL64ZtvdmDUqKuE4xISEvHnn3/g5MkTyM3NwZYtH2Lbto8BoMk9DTUaDW655Ta8++7b+OWXH3Hx4gUsX/4CSkqKhWOsVgtWr34dX3+9Hfn5efjjj99x+PAhZGT0C8wT0wqinGZq7/gP9b5dY7DveBGsNhZlBjPiosJv4StCPMUHLwPT45FTXI3KGiuqTTYhqBezMoMFFpsDUgmDHmlROHGpAvllNeieFhXqphGRevbZF7B27WosW/Y8qqoMSElJw7x5/8Lf/na9cMzw4SPgcDiQlVW35tKgQUNw6NABjBp1pXDZzJlz8PLLS/HYYw9CoZCje/eeWLBgMZ555p84ceIY+vfPbLQNDz30GBQKJVaseAlGoxFjxlyLESNGCddff/0kVFZW4v3316GoqBCRkZG46qqr8fDDMwLwjLQOwzU1BtXGOBwsysr8O2wtk0mg12tRXl7j16Sztz4/iv0ninDb6O749c985JXUYOZt/dG3a6zfHsNbgeqrWLWn/oqlr/9avxe5xTV48tb++OCbEygzWDB/ahZ6+HEl4ED19c9zpXj14z+QEqfFZZ30+OFgDv42tCNuHd3db4/hC7G8tr6w2awoLc1HbGwy5HLPqtqCkQAsFm2lry29zjExWo8WzaNpJhHiR2ZS4rRIjtUAcJZnE9JWuSa9p8RqkBLrTIoMl6km4T0bq0FSTO17NkzaTkhbQMGMyNgdrFDWmRqnRXLth3oefTCSNqy4wgwHy0EhlyAmSlV33odJEM8n/7p+AaHybEKCh4IZkSmsrWRSKaSI0SnrRmaoPJu0YXzQkhyjhYRhkBKncbtc7Pjk35Q4rTAyU1xhgj3M9kMiJFxRMCMyrlNMDMOE3XA7Ib6oG9nQ1P7Uul0uZs5KptpgLFYLfaQSSrkUDpZDscsigISQwKFgRmRyi6sB1H2Y89/yqk02VBkbL7EjJNy5BgOuP0sNFpgs9pC1yxOVNVYYLXYwDJAUowbDMML7lsqzCQkOCmZEhv9Q5/diUiqkiNWpANDoDGm7+JwwPoiPUMuhq92XSey5J/x7NiFaDbnMuTIt5c0QElwUzIhMbr1gBgCS+fyBMBhyJ8RbLMe5JdDyUmLDI28mv14gBgBJsVTRREgwUTAjInYHi6Jy5xy7+4d6bf4AbWtA2qCySjOsNhYyKYP4aJVweXIcX8kn7mDGNc+NJ0wz0cgMIUFBwYyIFJQZ4WA5qJVS6CPrNgSjiibSlvFTTIkxGkgldR9J4RLE1+X7aITLkmPrEpjbybqkhIQUBTMiUr+Sief6wUhIW1M/+ZcnTDOJ/LzPa2SKLFGvBgOgxmxHFW04SUjAUTAjIvwGkyn1P9Tj6io7zFZxV3YQ4i0hX8ZlZAOoO++LK0yw2hxBb5cnDEYrqozOYCU5pu59q5BLERvlnDKjiiYiNna7HVu2fCj8vn7927jllhv8/jiBut/GUDAjIvUrmXgRajkiNc7N9mgOnrQ1jY1sAIBOq4BWJQPHife857cZiYtSQamQul1HeTNErL777musXPlqqJvhVxTMiAhfyZQSr21wXXKY5A8Q4g2O44Rzuv6IJMMwLlOs4jzvG6tk4vEVTTQyQ8SmLeZxyULdAOJks9dVMqXGRTS4PjlWg1PZFaLPHyDEG64LziXGaBpcnxKnwZncStGWZzeW/MtLjqHE/UDgOA6wN76AKMdJwAV6J2mZwi2n0RO7d+/CunVrcOHCOajVGgwfPgKPPz4LZ86cwsyZj+LZZ1/EmjUrUVhYiIyMvnj66UX46KON+Prr7ZDJ5Lj11r9j+vT7hPv76qsvsWXLh7h06RJiYmJw/fU3Ytq0eyCVOkcHCwsL8Pbbq3HgwD4YjTXo128AHnnkCXTv3gM7dnyB559fDAC44opBeOONNcL9btr0Pj799GNUVlaiT58MzJnzNDp06AgAqK6uxurVr+N///sJNpsN6emX4ZFHZqBXr97C7f/zn23YvPkDFBcXY/DgIUhOTvH5afYWBTMiUVhmBMtxUCtliI5ouA262L+hEuIL9wXnGg4Up8SKuzxbmCKLbWxkxnkZTTP5D8dxMP53KdjCMyFrgzSxB9QT/+lxQFNRUYGnn34Kjz02E5dffgWKigqxZMkzePPN1zF27N/gcDjwwQfv4plnnoPdbsdTTz2Ju+++E9dffyPWrt2Ab7/9Cu+88xauuOJKdOvWHR9/vBlr1qzCjBmzMHDgEBw7dhQrVryEyspKPPHEbBiNNXj44fuQkpKKF198BXK5Au++uxaPPfYA3n//I1x99bWorq7GG2+8gv/852vodFE4fPggCgry8eeff2DZstdhs1mxZMlCvPjiEqxe/Q44jsNTT82AQqHCSy+9hoiICHz99XY8/PB9ePvt99CzZy98993XWLHiJTzxxP9h0KAh2LnzJ6xd+yYSEhID/Io40TSTSLgultfYmySFyrNJG9TcNA1Qt9aMWIP4xtaY4fGjNcUVZtpw0o8YeDcqEmrFxYWwWq1ITExCUlIy+vUbgJdeWoGbb75dOOb++/+BXr16IyOjHwYOHAy1Wo1HHpmBjh07Ydq0uwEA586dAcdx2LRpAyZPvg233HIbOnToiHHjJuC++/6Bzz77BNXV1fjmm69QWVmBJUteQu/eGejRoycWLXoOSqUK27Z9DKVShYgI5+h/bGwc5HJnPqZMJsPChUvQvXsPXHZZH9x442ScOHEMAHDw4H4cPfonlix5AX36ZKBTp8546KFH0adPX3zyyb8BAFu3bsE114zF5Mm3omPHTpg69W6MGDEyaM8zjcyIRG4zH4pA3chMUblzJ16ZlOJQEv6aKsvm8SMehWVG0Z33RrMdFdXO6Y7G2h+lVUClkMJsdaCo3NTke5t4jmEYqCf+s8lpJplMArvIppl69EjHNdeMw9y5MxEbG4fBg4fi8stHYtSoq3DkyO8AgLS0DsLxarUayckpwmMolc6qOJvNhoqKcpSVlaJfvwFuj5GZmQW73Y6LFy/g7Nkz6NChE/R6vXC9UqlC7959cPbs2SbbGRMTC622LsUhMlIHi8UCADh16gQ4jsPNN1/vdhur1Socc+7cGVxzzTi36zMy+uH06VOePE2tRsGMSDRVycSL0Tl34rXYHCiuMDX54U9IOKm/W3Z9MTollAopLCIMCPi26yOV0KgafpQ6E5g1OJ9fhfxSo6jaHs4YhgHkysavk0nAMOIbBVu0aCnuvfcB7NnzG/bv34slS/6Ffv0GCHkwMpn7+dNUsNRU4i7Lci7309QxLGQyaaPXAYBE0vQXBZZlodVqsX79pgbX8SM7AAOOc3/u6/crkMTzNaeda66SCXCe3EnCXjXiHHInxFstjcwwDCPaRNrmkn95deXZ4mo7CZ6//jqKN954BR07dsZtt92JZctex/z5C3Hw4H6Ul5d7dV8xMbGIiYkVRnR4f/xxGHK5HKmpaejWrQeysy+ivLxMuN5iseDEiePo3LkrgKaDpaZ07dodNTU1sNlsSEvrIPz78MMN+PXXXwAAPXr0xJEjf7jd7sSJ4149TmtQMCMCNrsDReXOAKWpkRmA8mZI21JtssHALzjXTEDAj2iIraKpueRfnpAELNKcHxJ4Wq0W27Z9gjfffAM5Odk4d+4MfvjhW6SldUR0dLTX93fHHdOwbdvH+PTTT5CTk41vv/0a7767FhMn3oSIiAhce+14REVF41//mofjx//CmTOn8eyzC2AymXDjjZMBOKeyAGewYbGYW3zMoUOHo0ePnnjmmfk4dOgAcnKysXLlCuzY8YUQIE2dejd27vwJmzd/gOzsS9i69d/4+ecfvO6fr2iaSQTyS43gOECrkiFK27CSiUfbGpC2hA9OYnUqqBRNfxQJwYzIAgJ+hLS56SNhVIkqmtqtzp27YOnSZXjvvXfw2WefQCKRICtrMF555Q0UFhZ4fX933DEVCoUc//73h3j11WVISEjElCnTceed0wAAERERWLnybaxa9RqeeOIRAEC/fv3x1lvrkZKSCgDIyhqM3r0z8PDD9+Jf/1rS4mNKpVK8+uqbePPN17Fw4TyYTCZ07twVS5cuw8CBgwEAl19+BZ555jm8++5arFu3Bn369MXf/z4V3333tdd99AUFMyLQ1J5M9SXHivNDnRBf8CMbyU3ky/CEjVbFNjLTQtI+4L5wHsdxXg/vk7ZhxIiRjVb2pKam4ddfD7hd9vTTixocV/+Ym2++HbfffkeTyc6pqWl44YXlTbZHp9Nh7dr3XS65Bvfd95DbMRMm3IAJE+q2IoiOjsY///lMk/cJAFdfPRZXXz3W7bKHHnq02dv4C00ziUBuC8m/PD5JsqDUuSYNIeGsqZV/6+ODhfwyo5DoGGoWqwOlBufwfHPBDL/hpNFiF6bUCCH+R8GMCHjyDQ8A4qPVkEoYWGwOlBsswWgaIQHT1J5M9cVHqSGTSmCzsygxtDy/Hwz5tQm9kRo5ItTyJo+Ty6SIi+Y3nBTXyBIhbQkFMyLg6ciMTCpBgt6ZuEV5MyTc5XuQQAsAEgkjVAWJJQlY+ALiwRIJSTF1I0uEkMCgYCbErDYHimv3ZEqJb7gnU320rQFpC0wWO8pqRxdbypkB6qZYxZI340nyLy+ZNpwkJOAomAmx/FIjODgrmXSapoereclUnk3aAD4Yj9IqoFW1fN7XVTSJ47z3dGoYcEkCppEZn7TFHZ5JHX+9vhTMhFheC3sy1ZdCFU2kDcj3MF+GJ5z3Ilkwsm6KrOVRJbEu+id2/A7QVivlB7Zl/OsrlbauuJpKs0OsbuXflqeYgLohefpgJOHMk9VzXdVtOFkT8hJnm92BooraqWGPRmacx5RUmmGzOyBvZkl5UkcikUKtjkB1tXOVXIVC2eLrzrIMHI72MZIT7n3lOA5WqwXV1eVQqyOa3U7BExTMhFhLezLVl1ybTFhltKHaZGu2koIQsWppt+z6EvXOSj6z1YHyKgtidKpANq9ZBWUmcBygUcqga2aRS55OI4daKYPJYkdhuQlpHn5xIYBOFwMAQkDTEolEApYV395MgdBW+qpWRwivc2tQMBNi3sy9A4BSIUWsTolSgwV5JTXo2SE6gK0jJDBa2pOpPr6SL7/UiLzSmpAGM54ucsnjN5w8l2dAQamRghkvMAyDqKhYREbq4XDYmz1WKmUQFaVBZaUxrEcsPNFW+iqVylo9IsOjYCaE+B2wAc9HZgDnH4BSgwX5pRTMkPBjtTlQXOn5NA0vJVbrDGZKjMjoEhuo5rWoLpjxbIoMcG44eS7PQOXZPpJIJJBImh8Fk8kkUKlUMJkcTa6M21a0p756ihKAQ6igtpIpQi33aLiaR+XZJJwVlNXtReZJBR/PNW8mlDzZYLI+Ks8mJLAomAmh3JJqAN6NygB1ScBiKVMlxBt1ezJ5Nk3D4yuHQr1wnrf5PgCERf8Kyug9S0ggUDATQnWVTN4FM/w3wnyRlKkS4g1P92SqT1hrpqQmZGuP2B0sCmunijzN9wHqKpqco1Lhm+NAiFhRMBNCecXeVTLx+EW4Sg1mWGwOv7eLkEDK82KNFldJMRowAGrMdlSFaNPGonITHCwHpUKKGJ3S49slRKshYRiYLA5U1lgD2EJC2icKZkLI0z2Z6tNpFEJJNs3Bk3DjyzQNACjkdZs2hmqqqW5PJo1XU2RymURoO+W6EeJ/FMyEiMXqQEmlcwdgbz/UAdrWgIQnX6dpeHUrYIcomPEh+ZeXHEPbGhASKBTMhAj/oajTyBGp8bySiZdM2xqQMFRc4ds0DU+oaApRvpivo0pA3fQwfQEhxP8omAkRbxfLqy+FPhhJGBIWy4vxbpqGF/KRGS8X+3OV7JIETAjxLwpmQqQuX8a31UDr1tygD0YSPlodxLtUNAUby3IuIzPeJS8DLuXZ9J4lxO8omAmRPB/Lsnl8zkxhmRGONrA/B2kfWjNNA9Sd95U1VtSYg1vRVFxpgt3BOpN5o9Re316oQqw0w0pViIT4FQUzIZLrY1k2L0angkIugYPlUFRu8mfTCAkYb3fLrk+tlEEf6cy1CXbejOsUmUTi/RRZpFoOrUoGDkAhvWcJ8augBzMHDx5Eenp6g3979+4Vjtm9ezcmT56M/v37Y/z48di+fXuwmxlQZqsdpQbfK5kAQMIwwg7aNNVEwgHLcsLeRL6e9663DXbeTGtHlRiGEUZnKG+GEP8K+kaTJ0+eRMeOHbF582a3y6OiogAAZ8+exUMPPYR77rkHy5Ytw88//4w5c+YgJiYGw4cPD3ZzAyKv9hulTlu3XowvkuM0uFhYVZsEHO+n1hESGCUGM2x2FjKpBPE+TNPwkmM1+Ot8WdDzZlo7qgQAyTFanM01UOI+IX4W9GDm1KlT6N69O+LjG//ju2HDBqSnp2PmzJkAgG7duuHYsWNYt25dmwlmfN2TqT6hPJu2NSBhgA8GknycpuGFamSmtcnLQF3eDCUBE+JfIRmZGThwYJPXHzhwANdcc43bZcOGDcPSpUvBcZxP5Zw8mcy/s2pSqcTtp6f4Iea0hIhWtSmtNnm4sNzo977V52tfw1V76m+w+lpY7jzvU+O1rTpfOyREAnDmzHh7P772leXqKpk6JEb63P7U2vdsQRDeswCdx21Ve+qrp4IezJw+fRp6vR6TJ09GYWEhevbsiZkzZ6Jfv34AgIKCAiQlJbndJiEhASaTCeXl5YiJifHpcSUSBnp960ZCmqLTeTdkXljhzJfp0SmmVW3q1TUOgHMuPzrat3U7vOVtX8Nde+pvoPtaYrAAALp10LfqvO+tcE7NlhrMUGmUUCu9/xjztq9FZUZYbA7IpAzSu8ZB5uMfkfQuzvdsYVnw3rMAncdtVXvqa0v8Gszk5OTg6quvbvL6n3/+GVVVVTAajViwYAGkUik2bdqEqVOnYtu2bejevTvMZjMUCvcVcfnfrVbfN2hjWQ4Gg3+HdqVSCXQ6NQwGExwOz8ujL+YZAAB6jRzl5b4PlatlTO3mdXacu1SGGJ3K5/tqia99DVftqb/B6uuFvEoAQExE6857wJlvZqix4tiZYnRN0Xl8O1/7evxcCQAgMUaDKoPvlUgqKSCVODecDPR7FqDzuK1qT33V6dQejUD5NZhJTEzEjh07mrw+ISEB+/fvh1qthlzu/HbVt29fHDt2DBs3bsTixYuhVCobBC3872p166JQuz0wL7rDwXp83yZLXSVTUoy61W1K0KtRUGZEdmE1dD5si+Atb/raFrSn/gayrxzHCcsRJEa3/rxPidXAUGNFdmEVOiZ4v/Ckt33NLnTmuSXHaFrd9rhoNQrLjMgpCs57FqDzuK1qT31tiV+DGblcjm7dujV7jE7n/i1KIpGgW7duKCwsBAAkJyejqKjI7ZiioiJoNBpERkb6s7khwSctRkUooFX5XsnES47VoKDMiLzSGvTp4tsUHCGBVlFthdnqgIRhkBjjezUQLzlWixOXKoK2LIE/kn95yTEaFJYZUVBmRO/O9J4lxB+Cmj20c+dOZGZmIjs7W7jMbrfjxIkT6N69OwBg0KBB2Ldvn9vt9uzZg6ysLEgk4Z/slNfKxfLqS6FtDUgY4IOBBL3a53wTV8He1kDYLdsP79u6DSfpPUuIvwQ1OsjKyoJer8fcuXNx9OhRnDx5EnPnzkVFRQXuvvtuAMC0adNw5MgRLF++HGfPnsW7776Lr7/+Gvfff38wmxowuX78hgfUrXmRH4K9agjxlD9HNoC6jVaDUZ7NcZyw/EGKDxtM1pccQwvnEeJvQQ1mIiIi8P777yMuLg733Xcfbr/9dlRUVGDTpk2Ii3Nm+ffo0QNvvvkmfvnlF0yaNAmffPIJli1b1mbWmMkr8e/IDL/WDC3CRcSMPz9bs+CcK36j1eIKE2z2wO5zVFFthcliB8PAL1NkdWvN0HuWEH8Jeml2x44d8cYbbzR7zKhRozBq1KggtSi4Wrtbdn38HweD0YZqk61VKwoTEij+HpmJ0iqgUcpgtNhRUGZCBx+SgD3FB2IJeg3kflgbhv8CUmqwwGJ1QKmQtvo+CWnvwj8JJYwYzXaUVznX2kiJ8883VJVChhhd7cZ79E2PiFReqf+maQDnPkfBypsRAjE/jSpFqOXClw5+IUFCSOtQMBNE/Py+PlIJjR8qmXh1U030wUjEx2C0otpkA4O6KRZ/EPLFAhzE57Vyg8nGUBIwIf5FwUwQ+XuonccnFNLIDBEjPjk9NkoFpdx/UyrBH5nx3/uWkoAJ8S8KZoIo189l2bxkKs8mIhaIkQ3X+wv0eR+ILyFJQRpVIqS9oGAmiPJqd8v2+4c6X6ZK5dlEhAIxsgHUTTMVlBlhD9CS7gGbIoup3XCSRmYI8QsKZoLI32vM8ITqiEozrLbAlqkS4i1/l2XzYnTOaSsHy6G4wvf9kpoTqCmyJJdAjOU4v90vIe0VBTNBYjTbUFHt3GPK399QIzVyaFUycKBvekR8ApUrJmEYIUDiF7Xzt0BNkcVFqSCVMLDaWFTUVjgSQnxHwUyQ8KMyzkom/y7vwzCMkDcTjBVRCfGU0WwXgvhkPwfxrvcZqPM+UFNkMqkECXrnxrmU60ZI61EwEyS5fl75t74UYVsD+mAk4sFPMUVHKPwexAN16zUFajsPPphJ9tO6UK6SqAqREL+hYCZI+A0m/T1czaNtDYgY+XODxsakBHhkJj+A7effszQ1TEjrUTATJIEemaGF84gY8SOFgZhiAtzLs1nWv4m0rnlufPWRP9WNzNB7lpDWomAmSIRvqPGBnWYqKDPCwQamTJUQbwV6ZCYuWgWZVAKbnUWJwezX++aTfwOR5wa4l5YTQlqHgpkgqDHbUBmgSiZeTJQKCpkEDpZDSYV/P9QJ8ZW/9zWqTyqRICmmNpHWz3kzgW47X55dXmWB2WoPyGMQ0l5QMBME/Mq/sTol1MrAbFQuYRhh2JoqmogYWGwOlFY6A+vkAI3MAC7bGvj5vK9L/g1M27UqOXSa2g0nywKzTg4h7QUFM0FQt85GREAfh7Y1IGJSUGoEB+cu0TqNImCPIyQB+3lkJj9Aa8y4ooomQvyDgpkgCHTyL0/YRZi2NSAiIOTLBGiahheoID5Qa8y4SqKKJkL8goKZIAjUCqj11ZWp0gcjCb3gnfd1e5NxftoawGy1o7Q2oTg4IzP0niWkNSiYCQJhZCZAlUy8ZJedeP31oU6Ir/g/0IEqy+YlxmggYRiYrQ6U+2lrAL7tOo0cEWq5X+6zMXXvWQpmCGkNCmYCrNpkg6GGX849sMPtrh/q/PoYhIRKsEZmArE1QLDazlc0FZbThpOEtAYFMwGWW1wNwLmxnEoRmEomnkwqQXzthzpVNJFQsjtYFJU7K3QCHcQDLhVNfsoX498/gazCApyfCzIpA5udRVklLalAiK8omAmwYH3D46VQEjARgcIy50iDSiGFPlIZ8Mfj92jyVxDPr1wcyORfwLlOTqKeFs8jpLUomAmw3CAHM7StARED13wZhmEC/njCee/nkZlAV2IBlARMiD9QMBNgeUEqy+a5JgETEip1I5KBDwaAuhGUXD9UNFltDhRXOKfIgvElJIm2NSCk1SiYCbBgj8yk0MJ5RAQCvSdTfUmxGjAAasx2VBltrbqvgjIjOA7QqmTQaQO32B+PFs4jpPUomAkgg9EqfLAGeu6dx38wVtZYYTS37kOdEF/lBXi37PqUcilio1QAWh8UuCb/BnOKjEZmCPEdBTMBlFe7J1NclApKhTQoj6lWyoSES1o8j4QCy3LCH+Zgjcy4PlZrK5qClfzL47+AVFRbYbLQhpOE+IKCmQAK1jYG9dG2BiSUiitNsDtYyGUSxOlUQXvcumCmdUF8MJN/AUCjkiGqdjqLRmcI8Q0FMwEkJEEGeOXf+qiiiYSSsNt0jAYSSeCnaXh8EN/a8uxgL6cA1I3OFNB7lhCfUDATQMGuZOKl+OlDnRBfCGXZwT7v+ZGZVpz3rov9BTOYEUZTy+g9S4gvKJgJoLpppoigPm7dyAx9MJLgq9ttOjjTNDw+x6Wy2vfk98JyExwsB2WQFvvjCbtn08gMIT6hYCZADDVWVJtsYFC3jkSw8N+ISyrMsNocQX1sQkIxTQP4J/k9XwjEglPJxBPKsylnhhCfUDATIPyoTHy0Gkp5cCqZeDqNHFqVDBwooZAEF8dxQdstuzHCFKuPye/BTv7l8dNMhWUmsCxtOEmItyiYCZBQfTsFAIZhKAmYhESZwQKLzQGphBF2sg6m5FaWZ4fqfRurU0EmlcDuYFFioA0nCfEWBTMBIuTLBLmSiUfbGpBQ4M+3BL0aMmnwP15SWhnEC4v9BTmYkUgYJMU4gz/KmyHEexTMBEhecTWA0IzMAFSeTUIjlCOSro/ry8hMqBb749WVZ9MXEEK8RcFMAHAcF7IF83g0MkNCIS+E+TJAXRBSajDDbPVuNV1+sT9FkBf74yXRtgaE+IyCmQAw1FhRY7aDYeq+bQUbP0xeQAmFJIjqNpgMzXkfoZYjUiMH4H1QwI/mJAV5sT9esrDhJAUzhHiLgpkAcK1kUgS5kokXp1NBLnMmFBZXmkLSBtK+cBznVtocKvxjezvVFOopsqRYKs8mxFcUzARAqKeYAD6hkN+jiT4cSeAZjDbniCRCNyIJ1I1KejvCEarkXx7/nBlox3tCvEbBTACE+hsej/JmSDDliWBEEvB9rZm6NWZC875VK2WIjnBuOEmjM4R4h4KZABDDyAzgMtxOwQwJAj5oTg7ygnP1+VLRxHKc0P5Q5fsAdYnTVJ5NiHcomPEzjuOQVyySkRkfh9sJ8YV4RiSdj19UYYLNznp0mzKDGVYbG7LF/nhCeTaNzBDiFQpm/Kyi2gqjxVnJFOpvqK7TTBxHFU0ksPhgJlRl2bzoCAXUShk4Dij0MCjg82WSYjSQSkL3scgnAdPIDCHeoWDGz/gP9AS9BnJZ6PIGACBRrwHDACaLA5U11pC2hbR9/AhgqEdmGIYRpoo8nWIVArFQjyrRhpOE+ISCGT8TS74MAMhlEiREO4fM833cq4YQT9SYbULAHOoRSWcbvMubCdUGk/XxIzNF5UY4WM+myAghFMz4XV5JaLcxqE/4UKdhaxJAfPm/PlIJtVIW4ta4Jr97dt7niyTfJ0angkImgd3BoaSSNpwkxFMBDWYWLlyIefPmNbh89+7dmDx5Mvr374/x48dj+/btbtdbLBYsXrwYw4cPR2ZmJmbPno2ysrJANtVv+Ll3MYzMAFSeTYKjbuVfcZz3fDs8GZHkOE4IekK52B8ASBgGibQSMCFeC0gww7IsVqxYgS1btjS47uzZs3jooYcwcuRIbNu2DbfeeivmzJmD3bt3C8csWrQIv/76K1auXIkNGzbg3LlzmDFjRiCa6ldi2JOpPtpwkgRDXfJv6KeYgLrpooKylqdrKqqtMNUm7SeGcLE/Xt2Gk/SeJcRTfh8PPnv2LJ5++mlcvHgRKSkpDa7fsGED0tPTMXPmTABAt27dcOzYMaxbtw7Dhw9HYWEhPv/8c6xZswaDBg0CAKxYsQLjx4/H4cOHkZmZ6e8m+w3/oej67SrUkr1MhCTEF2IbmYmJUkEhl8BqY1FUbmq2wopvuzNpv3Xf71hjBdiKAkiik8Coo8Aw3u/xlCwEYvSeJcRTfg9m9uzZg27dumH16tV48sknG1x/4MABXHPNNW6XDRs2DEuXLgXHcTh48KBwGa9Lly5ITEzE/v37RR3M5NbmyyTGqFv9oegvyTHOD/HKaiuMZjs0qtDnM5C2Rwx7MrmSMAySY7W4WFCFvBJj88FMSeuTf1lzFayHv4Ttrx8Atna3bqUWUn0qJPoUSPSpwj9GrWs2yKHybEK85/e/bFOmTGn2+oKCAiQlJbldlpCQAJPJhPLychQWFkKv10OpVDY4pqCgoFVtk/k5wJBKJW4/+YWu0uIj/P5YvtJFKKCPVKK8yoKiChO6p0X5dD/1+9rWtaf+travZqsdpQYLAKBDonjO/dQ4ZzBTWG4U2tRYX4X3bYL3bedsZph//xrm33cANmfCLqPVgzNWAJYaOApOwVFwyu02jCoC0phUZ6ATk+b8f0wqJGqdsx3xEQCc5dmtfS7pPG6b2lNfPeVVMJOTk4Orr766yet3796NmJiYZu/DbDZDoVC4Xcb/brVaYTKZGlwPAEqlEhaLxZvmupFIGOj1gfnWqNM5y5+LK53t69ZBH7DH8kXHpEiUV1lQabK1ul18X9uL9tRfX/t6OrscABAdoUTHVL0/m9Qq3Tvq8dvRApQYLA3Oe9e+FlU4g5AenWI8fn9wDhsMh75Dxa6tcNRUAgAUSV0RM3oq1F36gXPYYCvJhbUkG7bibFiLs2EtyYa9vBCcuRr2vJOw5510u0+JRgdFfAck69MwQlmBAms0JKwZUbGxrXkaGvS3raO+tk9eBTOJiYnYsWNHk9dHRbX8rV+pVMJqdV/Ajf9drVZDpVI1uB5wVjip1b6/cCzLwWDw77CtVCqBTqeGwWCCw8HiXK7zQy02UoHycvHMd8dHqQAApy+VY2CPOJ/uo35f27r21N/W9vXkuVIAQFKMWlTnvV4rBwCcz6sU2tVYXy8VGAAA0WpZi+3nWBbW07th3vcp2KoSAIAkKhHqobdA3m0wLIwElorazxllApCaACZ1IJQAlAA4mwWOinw4ynLhKMsBW5YLR3kuWEMxWKMB5ot/ARf/wm21MVXpmm9Qpo4SRm+kMamQRKdAGp0ERtNyTg6dx21Te+qrTqf2aATKq2BGLpejW7duPjcKAJKTk1FUVOR2WVFRETQaDSIjI5GUlISKigpYrVa3EZqioiIkJia26rHtHu7T4i2Hg4XN5hDWmEnUqwP2WL7gqyPyimta3S6HgxVV3wKtPfXX175mFznP++RYraieq0R9bYlzSQ2sNgckLn/4+b4ajFZUGW1g4Nztu6n2cxwHx6U/YNm/FWxZDgCA0URDkXUj5L1GgpHI4HAAQAv9Z+SAviOk+o6QunyUcjYL2Io8sOW5cJTl4syx44iwFCNWWgPOVAl7biXsucfc70uugkSXAElUEiRRibX/ksBEJYJRRrgFOnQet03tqa8tCXo26KBBg7Bv3z63y/bs2YOsrCxIJBIMHDgQLMvi4MGDGD58OADg/PnzKCwsxODBg4PdXI+VV1lgsjgglTBC8CAWybR7NgkgseyWXV98tAoyKQOrnUVppRnx0Q1HdvnE5dgoFZTyxrcfsRecgnXf1rrcF4UGigHXQZFxDRiZstHbeIuRKyGN7wJpfBfIAfxuOIkfD+Xi+iFJuDFD5QxyynPBlueCrSgAV10C2MxgSy+BLb3U8A4VGkiikiCLTgKSOsCqigEXkQBJVCIYhbheJ0L8IejBzLRp03DTTTdh+fLluOmmm/DLL7/g66+/xrp16wA4p7Kuu+46LFiwAM8//zzUajWeeeYZDBkyBAMGDAh2cz2WK+zJpIZMZElZfJVGcYUJNrsj5HtGkbZFLLtl1yeVSJAUo0FOcQ3ySmoaDWaaa7ujLBuWfVvhuPRH7R3Koci4FooB14FRBrav/BeQ3HI7pAldIU3oCrnL9ZzDBraqGFxlIdjKArCVhcI/rqYMsBrBFp+DtfgcrKfd75tRRQojOPxojiQqERJdIhi5f4IzQoIt6MFMjx498Oabb2LZsmXYsGED0tLSsGzZMmEUBgCWLFmC559/Ho899hgAYNSoUViwYEGwm+qV3GJxLZbnSqdVQKOUwWixo7DMhLSEiFA3ibQRNrsDRRUmAKHfLbsxybFa5BTXIL/UiP7dG16f18jmmKyhGJaDn8F+ejcADmAkkKePgmLgjZBog5PgLCyc18SGk4xUDml0ChDdcC0vzm4BaygCW1kIpqoQMlMpTEW5cFTkgzMZwJmr4DBXAYWnG96vNgby3qOh6D8BjIS+9JDwEdBgZuPGjY1ePmrUKIwaNarJ22k0Gjz33HN47rnnAtU0vxPrt1PAuYtwcqwGZ/MMyCutoWCG+E1hmQkcB6iVMkRHNKxCDDX+/djUhpOuKxezJgOsh7+A7diPAOsAAMi6DoFy0GRIopMavX2gJAsbTppgd7BejfYyMiWkMR0gjekAmUwCvV6L8nJnvhxnNYE1FIKtKHD+dBnZgaUGXE0ZrPs/hf3i71CPfgCSqOD2mxBf0QpqfiJsYxAvzkAhOVaLs3kG2taA+JXrbtO+rHYbaEIw00S+WF5pDZSwokfpL6g58DNgdy6vIE3tA+WQWyCN7xKsprqJjlQKKxiXVJr9lofHKNSQxnWGNK5zg+s4czXsFw/DvHsz2KKzqPl0IZRDb4e89xhRvraEuKJgxg+cG9WJd2QGqNvWgDacJP4kjGz46by3F5yC7Y+vwHEsJGqdc7Vctc65NYDr76oIMEzLoxWuG61yHOd2XXWNEQNsv2Ns9J/QnnYGMZL4LlAOuRWy1N5+6Y+vJIyzkOBSYTXyS2uCUlTAqCIgTx8JaWpvmH9ZD0fuMVh2bYT94mGorrwvaFNshPiCghk/KK00w2J1VjIl6sW5iJFQ0VRCIzPEf/y12zRbXQbL3o9hP7tHuMzR3A0YBowqsmGQo46CRF17uUaHBEUE5AwLk8W5d1q8Xg2OdcBy4n+w/7YVk7XOBf8kUUlQDL4Zsi6DRDMKkRyrxaXC6ibzZgJFEhEL9YT/g+2vH2DZ+zEcOUdRs3UBVCOmQd59WMt3QEgIUDDjB/wUU1KMRnSVTDzXXYRZloNEIo4PbBLe8oURSd9GDji7FdY/v4H18BeA3QqAgbzXKEgSuoIzVjoTVuv/s1QDHCf83pLleqCGVYD7z9eoitaj2lIFW2kepAAqWA3+0F6O6269Q3QJr/xoTCimhhlGAkXGtZCm9YH5p3fAFp+H+cc1sF84BNUVd4FRiXM6nbRfFMz4QU6xc9Ewf04xOQrPwHLwc3AWIxi1rm7IXRNV71uoDlBqW/w2GRflLBm3O1iUVJqQoKe1JkjrOFhW2AzR25EZjuNgv3gYlt0fgasqBgBIErtDNWJqo/kcbrdl7eBMVQ2CHNbUePADjoVWYgWMRbAbnQt2MkotjmmHYv2ZJFzZvbPoAhnAdffs0I2mSqNToLnxaVgPfwnrof/Cfm4fagpOQXXlvZB16BeydhFSHwUzfuDPsmzWZIB13yewnfyf2+XND7lLwbgMrTMqZ7Aj0bjnGnTTA6dKWOSVGimYIa1WVG6Cg+WgkEsQU7tlhicc5Xmw7N4MR85RAM6VdJXDboes2zCPpngYiQyMVg94kMPBcSy++Okv7D5wGiN7ajG+vx4alRTWhN749dOTsKFMtHluQnl2iJP2GYkMyoGTIOvYH+af1oKtyIfpqxWQXzYaymF/p7VpiChQMOMH/ijL5lgHbMd+guXANsDq/PCS9bwCss5ZLt8yK+t9EzUAlhqAc4AzVjh36i1t+jEeAcDqAfvOz1BzWO8MgJRaQK4Go6j7B+H/GjBylXM1UbUGrFbSIImStF/89EdyjNZtq4CmcFYjLAf/A9vR7wHOAUhkUPQbD0Xm9c7zLAAYRoL4hHgUssU4Uh2FiT0GI7K2VJlf/be1+T6BklgbzFSbbKgyWhGpCW3puzS+CzSTF8Oy7xPYjn4H2/GfYM89BvXoByBNbGQRH0KCiIKZVmJZrm5kJt63D0V7wSlYdm0EW5oNAJDEdoLqimkefUBwDnuDYIetH/wYnf9nzVWQMIDCXgO2vAYo966dlQDAMPWCH40z+JHXC4b435UaQKFxBkau13tQiULEra6SqflRPo5jYTv5P1j3bQVnrgIAyDplQjn8Dkh0CQFvZ115dt0Ih8liR6nB4na92CjlUsTqlCg1WFBQZgx5MAMAjEwB1eVTIOuUCfPP68AZCmH871Io+l8HxcBJYKT0J4WEBp15rVRcYYLF5oBMyiDBy0om1lgBy54tsJ/Z7bxAqYVy8M2Q97oKjMSzP/aMVAYmIgaIiGnx2H3H8vHhF4dwWZIU91+d5gx0rCZwVhNgNYKzmcFZjbW/Oy/nbHXXg+Oc/6xG53Fe9dat1YBcBUapcQ+I+KCn9nK3IEjpfgykctFUnbRXQvJvMyMbjsIzMP/2Idji8wAASXQylJdPgSwtIyhtBICkWA0YOEc4DDVW6PVaoe06rQIRannzdxBCSbFalBosyC81okdadKibI5Cl9ob2liUw/7YZ9tO7YP39S9izj0A1+kFIY9JC3TzSDlEw00qXCpzVFEkxGkg9DEA41g7b0R9gOfgZYDODr+BQDLkFElVkwNqaEheJKk6No2UySFP7eBUMSKUMoiNkKC8qgc1UUxfsWE3grEbA6hII2dyvc15vAmcxAqwdAOc8xmbyPSCSyJxBjyqy9l9E3U9+HRK3yyPByEL/zbYt4cv8GxvZYGvKYdn3Ceynf3NeIFdDOXAS5BlXg5EE92NHKZciNkqFkkoz8kpq0ClNXzc1LLLNMetLitHgr/NlIU0Cbgqj1EI9+gHYOg2A5X8bwJZegnHbIiiH3Ax5xjiPv5AR4g8UzLTSpQLnsLmnQ9X2vOOw7NoEtjwXACCJ7+qs4EjoGrA28hJj1GAYwGixw1BjRVSE54l7DMNAolBBotVDqozyuQ2c3Vo3EuQS7HBWE2Axuv3OWYy1gZERnKVuxAjgANbucWmuQKZoGOA0FvSoIiGJ0IHTUfDTFJbjkF/WcLdszmGD9c9vnaXWfKCePhKKwTdDovH9vGmtlDitEMwAdcsp+Guxv0ARKppEvHK3vOtgSJN6wLzzPTgu/eEcbb74O1RX3Q9JZHyom0faCQpmWulSoTOYaamSia0ph2XPv2E/uxcAwCgjoBh6K+TpI4OWPyKXSREfpUZRhQl5pUavghl/YWQK5wiJj3/YOI4FbJbaAKfGWaJrrgJnrq730/3/YB2A3QquuhRcdTNZ0i4qAed0lkLdMEm6QdK0M1m6bsrM9VhVm8sRKqs0w2pjIZXUTa/aL/0O828fgTMUAgAkCd2gunxKUAL1lqTEanHkbKkQxOQVizv5l5fMrzUjwpEZVxJNNNTjnoTt5E5Ydn8ER/5J1Gz9F1TD74QsfSRNCZOAo2CmlfhppqZGZjiH3flN9dB/nPu+MIyzpHHQ5JAsPJUcq0FRhQn5pTW4rFP4LU/OMBIhWEBErEe34TgOsJldAhxDw+DHVC8gstQA4ACHDZzJBpgMrcgRgjOgcQ18FGowytopMXXtaJBaV7uqrfOnMwgS5x8BfvuOpBgNGEMhjLs/giP7CACAUUdBOfQ2yHoMF00Qx49w8Mn6uSLeGNZVUm2wVezDhpPBxjAMFL2uhCzlMph/XgdHwSmYd74L2cXDUI68O6Qjc6Tto2CmFViOQ3ZR0wvm2XOOOqeUKgsA8IuCTYM0rlNQ2+kqOU6LP86WIr8dbWvAMExdAORh9YxUAkRpJSgvKoHdVOMyNcbnBplrk6ZNTSRM1/5k7c47tJmdCdY1XpSQSeV1wU1twMMHOvUDH0atC+p6H3klRihhxXWqY6jZutY58iWRQtF3HBSZNzifaxHh35/5pTWw2BwoLje5XS5W0REKKBVSWKwOFFeYhG1JxEyiS4D6+nmw/fk1LPu3wX7xMByFZ6AcdTfknQeGunmkjaJgphVKKkywWBtWMrHVpbDs/gj28wcAAIxaV/tN9fKQf1Plv6E2tYswcWIkEkhVWkgjAU7dcqVYUzi7FZzN7B4I2Wrzg8xVYE1VzpEifrrM5Pw/HFbnqFBNGbiaMs8eTKqoXTxRB0apdU7nSRVgZHJAqgBkcjBSuTN3iP9dpgCrUKImKhI2MwsHIxeOr/8TEikYhgHHsZBf2oMF0T9CZzQ7H7pjf6iG3wFJVJLPz1Ug8UFAeZUFpy6VgwOgVcmg04i3kglwBuJJMRpcLKhCfqkxLIIZwPn+UfSfAGmHvs6F9kqzYf52Jexdh0AS28G54rLwT+b8nZEIlzESmcv10kaOl9S7XgZOIQdnV9BaWO0UBTOtwA9ZJ8dqIZVInMmPf3wF6+EvnX+MGAnkfa6GcuAk5+J0IsDnCNDu2cEh5AipdV7djrNZnIENH+A0CHwMdUv6mw2Aww44vMsJcuXR2cAwtUGNBEOsJkACWNRxiL5yGmQd+3v9mMGkUckQHaFARbUVvx3JA+AcpRTrNJ6r5Fg+mKkBEF4JtdKYDtBMWgjrwf/A+sd22M/tA87tC9jjVQAAmNrAXQHInP/qAvPa3xsE9UqX29T+zh8jUzhHSWVKQK5wrqElVwIyFVVsiQgFM62Q47JYnv3SHzD/tllIfpQmp0N5+VRIYzuEsokN8N/sKqqtMFnsUCvpFBAjRq4EI48HdC3/8XLLCeIDIEsNOLsVsNvAOazO5GeHDbDbnEGP8NMKhrVByjlgs5hrb8Mf6xwdcnkgZ94XADMnx9emfhg9cRpkSdEBehb8KyVOi4pqK3b/me/8PUxGOYRtDUSeBNwURiqHcsgtkHUaANuZ3c7zi2WdU5OsHWAd4FhH7e+Out+5er+7XW93+90d53wMuxWwCJcEhlQBRqECZEowChUYmQpQqGoDH3Xt+1jlzH+Tq5xBkPD/usvBH0frZ/mM/pK1Qm5xNWIkVRhbsxemr08C8H6fmWDTqGSIilCgstqK/FIjuqZ4N2JAxMeXnCBXMpkE+tol/u121u06jnMmQcNhc/5xcNhQWVmNhR+egpVR4O9x4XP+pMRqcexCOUornVNjYs+X4fFfQMRcnu0JaWL3gG17wLEsZBIW0ZEKlJdWwG621J6vVuG85ewWZzBvt9YG+Db3Y+y1x7ic6/xthC8ENotzyQGu9n3isIIzWZ1t8EtPmLrRINefUvffJXIlHBFaWOwMOIncGUzJ+JEkZcP7kCog0USJZoYgECiYaQV94X7Mj9oJhcEBMFLI+14LZdaNokt+rC8lVlsbzNRQMEOaxTBM3VB97QdhfpkMFiiQGK2GXBY+w+z115RJaWEbBrFwHZnhOE6UX5JCjZFIwMhkkKi0kGgAiYJt+UY+4gN8zm5xLhZqN9f+tDiT/u0WZ36c2+VmwO4sAoDNUlsoYBaOhcPK37vzstoR0OYCJGsz1zWKkUI59FbI+45rk+cQBTOtkOn4HQrGAS6xF7SjpkGqTw11kzySHKvB8YvlAU0Cdu59Y0aEWo5Ijdzj1ZGJ+Al7MoXJNA2v/mq/4TLNlKhXgwFQY7ajymiDTkuLOTZm4zcncfR8GZJjNEiL16JjYiQ6JkQgNsq/SxzwAT4jUwB+WrGdY1lnsMNP87r+dFgbXM6wNqjkHExVNWBtFpfrXUaW7BbndDJ/uaUGlj3/hqMsG6orpre5FdEpmGmFqOtmwmatQnz3DDgc4ZNBz/8R8md5dnmVBadzKnA6pxKncyqQXVQNvqiAARChkUOnVUCnUSBKq3D+n/89wvlTp1UgUiMX9VoapG637HCZpuG5tlelkEIfGfxFI32hcNmOoaDMSMFMIyw2B77fnw0OQH5JDQ6dKhauUytl6JgQgQ6JEeiQEIGOCZFIidOKalSRkUiEveg80dzUcGM4joPtr+9h2b0Z9lO7YKwogHrs45BoolvZcvGgYKYVEjt1Fk6oAKaY+R1fnu1rRRPLccgvqRECl9M5lSipzUNwpVHKYLLawXFAldGGKqMNuR7UzUSo+cBHjqgIZW2g47wsql4QRCM+wVc3MhMe0zS8SI0zWK4y2pASJpVMvKRYDUoqzcgvrUHPDtGhbo7oFJQ6N76NUMtx48guuJBvQHZhNXJLamCy2HEyuwInsyuE46USBsmxWnRMjKgNdCLRISFC1JuOtgbDMFBkXAtJdDJM378JtugsjJ8thnrsDEjju4S6eX5BwUw7xI/MFFWYYLOzLX5DsdkdOJtnQPahXPxxqhhncipQY7a7HcMwQMeESHRPi0KPtCj0SIuGPlIJluVQVbtbMf+v0vWnse7yKqMNLMeh2mRDtcmGvBb6oZBL0D01Cukd9UjvEI0uyTpRfdtqi2x2VpieDLeRGcDZ5pOXKlrcfkRskmO0OHpOnBtOikFuiXPx0k7JOowb0lEYrbA7WOSXGnGpsArZRdXCzxqzHTnF1cgprsZvLvcTo1OiY4IzsOmY6Axy4qJUkIRR4NscWVoGtDcthOmbN8BW5MH43+ehuvI+yLsPC3XTWo2CmXYoOkIBtVIKk8WBwnIj0uLdt1WoNtlwxmXU5UKBAfZ602gKuQTdUuoCl64pukbLvCUSBlG1Iyot4QOZ+kGP8Hu9wMdqY3HsQjmOXXCuqiuXSdAtRYf0jnr06uhsk1wmbcUzRXhF5Ub88nse/nckH9UmGxgm/EZmAKB7ahROXqpAt9TwWlo/SRhNpWCmMfwO7h2T3HNYZFIJOiQ4p5d4HMehzGDBpSJnYJNdWI1LRVUorjCjzGBBmcGC38+UCMerFFJheio1QYuUWC1S4rRhO4ojiUqCZtICmH58G45Lf8D84xqwZTlQDJ4c8kVdW4OCmXaIYZxDrOfyDMgvNUIhk9ROGTkDmMY+MKMiFMjoGodOiRHolqJDh4QIv+e2SBjGOaWkUbS4NhjLccgrrnEOH18qx8nsClQZbThxqQInLlXgP3B+kHVN0SG9QzTSO0ajW2oUlPLQBDf8EvqF5UYUlZtQWWNFj7QoZHSNDVmbWuJgWfx+uhQ//56Lv87XrUKsj1Ti+uGdoFKE38fHTaO6YsSANCTrlUJ1bTjgN5wM9/LsQOGnPjsmtpyQyzAMYqNUiI1SIbNH3QeNsXa0RhjFKapGbnENzFaH8PnoSqdVIDWOD240SInTIjlO6/z8EjlGoYF67BOw7t8K6x87YP39SzjKcqAe85Doq3GbEn6fRsQvkmM1OJdnwDtf/NVg1IW/vkdadO3ISxSS47SIiYnwOOEs0CQMg7SECKQlRODqgWngOA75pUYhsDl5qQKVNVacyq7AqewKfPGbc568i0tw0z01yq9/kOsCFhOKyo1C4FJYbkJ5laXB8d/uz4ZCLkG/bnEYlB6Pft1iRREglBnM2PlHHnb+kYeKamcBKAOgT9cYjB6Qin7dY8M2V0khl6Jv9zjnecyG/jz2FD8yU1zp2dRweyMEM0m+VxdpVDL07BDtlpNkd7AoKDMKozd5JUbkldSg1GAWRomPX3Tfby1CLUdKnHP0JiVWI/w/SqsIWZ6Wo/Zcd33fMhIJlENvgyQmDead78Fx6XcY/7ME6rFPQBKVGJJ2tkboPzlJSHRN1mHXnwWwOzjnH/lkHXqkRaF7WhS6p0Yhst63C7EnSzIMI3xojM5yBjeF5Sa34Ka8yoIzOZU4k1OJ7bsvQiph0CkpEukdo5HeQY8eaVEtrohstTlQVGFCYRkfsNT9bCxgcaVRypAYo0aiXgO1UoY/z5WipNKMAyeKcOBEEeQyCfp2jcWg9Hj07x4X1NWZWZbDkbMl+OFADn4/UyJUokVq5LiiXzKuHJCKhOjw/MbWFkRp66aGi8qNSK03NdyeWWzOTTgBoENiJGCvvyKw72RSCdLiI5AWH4HhqNt3zGSxo6DMGdgI/0prUFJhRrXJJnyJcqVRymo/ozTCVFVKnBb6SGWLn692Bwuj2Q6jxQ6TxQ6LzQFGVoGikhrUmGwwWmx119f+NFrsMJqdx5utDqiVMiycPgiJMe7Tw/Iel0MSlQTTt2+ALc9DzefPQn3No5Cl9vbb8xgMFMy0UyP7p0CrliM6QonOSZFQiHSqw1f8Bn1JMRpcOSAVHMehuMKEk5cqhOCm1GDGuTwDzuUZ8NWeS2AYoFOiM7i5rHMMNBoDzmaXIb/E6FPAkqB3+RmjaTDHznEcLhZW4cCJYhw4WYSichMOnSrGoVPFkEkZZHSJxcD0eGT2iINGFZj5+coaK347WoCdf+Sh0CW5NL1DNK7KTEVWz3gaBRAB/nw+n1+FgjIKZly5VjJFRyhRURH4qTi1UoYuyTp0SXZfdNRic6Cg1Ii8Utcgx/n5YbTYcSa3Emdy3aerVAopUuK0SNSr4WA594CkNhix+mE03GSx49jF8gbBDABIE7pCM3mRM6ApOgfTjuVQDr8D8j7XiP6LLI+CmXZKJpVgyGXhN5ToK4ZhkKDXIEGvwcj+KQCAksra4OZSBU5ml6O4wowLBVW4UFCFb/ZlN3lffMCSoNcgsV7AolXJPH7zMwyDzkk6dE7S4eYruyK7qBoHThbjwIkiFJQZ8fuZEvx+pgRSCYPenWMwKD0emT3jW514yHEcTl6qwM+/5+LgyWI4WOcwjEYlw+UZSbhqQGpYViq1dUkxWpzPr6Ik4Hr4KaZUEZTbK+VSdEqKRKd60102uwMFZSa3UZy8khoUlZtgtjqEL1UtUSul0Chl0Kjk0EUooZAyUClk0ChlUKtktdfV+10pw1d7L2LnH/koqTQ1ed8STTQ018+D+X8bYD+9C5bfPgRblg3liLvASMUfKoi/hYQESFyUGnF91RjRNxmAM1eEH7U5m1cJtUqOOJ0SCdFqt1GWCLX/N4NjGMa5YmliJG4a2QV5JTXOwOZkEXKLa/DnuVL8ea4UG74+ics6RWNgrwRk9Yj3agG1GrMNu/4swM+Hc91KfLul6nD9Fd2Q0Tka0jD5FtYe8XkznpZn2x0sTFY7rByDgqIq1JhsMFkcMFud0w5mix0mq/N3/nIHy2H8kI5hVe2VW1K34a9YyWXSBlVVgPM1Kiw3Ib+kBoXlRihk0rpgxCUw0ahkUClkkEic709vF83jV7suqWi4HpgrRqaA6qr7YYvtAMveLbCd2Am2PB+qsY9Dohb31jcUzBBSK0anwvA+SRjeJ8nrDwt/YhgGqfERSI2PwI1XdEF+qTOwOXiiCJeKqvHXhXL8daEcG785ifQO0RjUKwFZPeMRHdFwRVuO43Au34CfD+di3/Ei2Gr7opRLMbxPIq4ckIpuaVEh6yvxHF/RdPJSOT7+8QxMtUEJnxPhDE74yxywO3x7LS02B2bdNsCPLQ8sYWQmDKfeZFIJUuO0AV/3KK42362xxU3rYxgGin7jIdGnwPTDW3AUnoZx2yKoxz0BaVyngLazNSiYIUTkkmO1uOFyLW64vDMKy404WDsVdaGgSihF//DbU+iRFoWBvRIwsGc81EoZ9h4rxM+Hc3GpqFq4r7R4LUZnpmJYn6SgJhiT1uNHHkoNFny975LHt1PIJFApZVDJpVAppVApZFArpFAp636qFFKYLQ58ve8Ssl3Ol3DgOs1EGhcXpQKAZqeZ6pN16AftpIUwfvM6uMoCGP+7FKqr7oe865BANbNV6NOMkDCSqNdgwrBOmDCsE0oqTM4Rm5NFOJtnwKmcSpzKqcRH35+GQiYRkgad+VEJuCozFd1SdCHPKyC+SY7V4u9juiO/zAiVQgq1whmE8MGImv+pkAlBS4RGjvi4SI9G3cxWO77edwmV1VZUGa0NKhrFyOpSySTmaaZQ44OZKqMNZqvd4yUgJNHJ0E76l3OEJucozN+/CTYrB4qBk0S3wB4FM4SEqbhoNcYP7YjxQzuizGB2jticLMKZnEpY7SwSYzQYPSAFl/dNDtvVSom7sUM6enW8NwtbqhQyJESrUVRhQk5xDS7rJP5gJr+2kkmrktEGnM3QqOTQKGUwWuworTR7NSXHKLVQj58Fy76PYTvyNayH/gu2LAeq0Q+CkasC2GrvUDBDSBsQo1Ph2sEdcO3gDqiotsBQY0WHhAgahSFeSUuIcAYzRdW4rJM+1M1pkZgqmcQuLlqFS4XVKPYymAGcC+yphv0d0pgOMO98D/YLh2D8z3POBfZ0LSzXHiTiGicihLRadIQSHRMj6cOdeC2tdqomuzg88mbCedPTYIuPciYBl3qQBNwUec8R0NwwD4w6CmxZDoyfLYY977i/mtgqFMwQQggBAGHT2ZwwSQLOLaZgxlOxtXkzfI6Rr6SJ3aGZvAiS+C7gLNUwbV8O67Ef/dDC1qFghhBCCAAI66DkltSAZRvu2SY2/MgMVTK1LN6L8uyWSLR6aG6YD1n3YQDngOXXD2De8+9W32+r2hTSRyeEECIa8dFqKOQS2OwsCsvFvdKwtXZjV4BGZjwR60N5dnMYmQKq0Q9BMeRWAAxsx34CF8LNWykBmBBCCABAImGQGheB8/kG5BbXIDlWvEFCQRlVMnkjng9mWlgF2BsMw0A54DrI0jIAjgUjCd34CI3MEEIIEXRIqE0CFnneDL+NQQpVMnkkrjYB2LmBpc2v9y2N6wRpfBe/3qe3KJghhBAiEJKARV7RRCv/ekepkCJS41xvyh95M2JDwQwhhBABH8yIfWSGD2aSKZjxGD86Q8EMIYSQNi2ttqKppNIMk8Ue4tY0LZdGZrwm7NHUyvJsMaJghhBCiCBCLYc+0rkDO7+Oi9i47clEwYzH4qJr15qhkRlCCCFtnTDVJNK8mYIyIziOKpm8FeeHVYDFKqDBzMKFCzFv3rwGl99zzz1IT093+zdt2jTheovFgsWLF2P48OHIzMzE7NmzUVZWFsimEkIIqZVWW9Ek1iTgPKpk8glfnl3sp7VmxCQg68ywLIvXXnsNW7ZswU033dTg+pMnT2LRokW45pprhMvk8rpdfRctWoQDBw5g5cqVUCgUeOaZZzBjxgxs2rQpEM0lhBDiooPItzVwLcsmnotzWQWY47g2FQj6PZg5e/Ysnn76aVy8eBEpKSkNri8tLUVpaSn69++P+PiGu20WFhbi888/x5o1azBo0CAAwIoVKzB+/HgcPnwYmZmZ/m4yIYQQF3wScE5xtSj/6OVRMOOTWJ0zF8pidaDaZEOkpu1M0fl9mmnPnj3o1q0bvvzyS6SlpTW4/uTJk2AYBl26NL7AzsGDBwEAw4YNEy7r0qULEhMTsX//fn83lxBCSD1JMRpIJQxMFgdKDeLLr6BgxjdymRTREc4Apq2VZ/t9ZGbKlCnNXn/q1ClERkbi2Wefxa5du6DRaDB+/Hg88sgjUCgUKCwshF6vh1KpdLtdQkICCgoKWtU2mcy/sZtUKnH72Za1p74C7au/1Ne2y9f+ymQSpMRpkV1UjfxSI5JEtK2B1e5AUW0lU6fESOFzvT29tq3pa3y0GhXVVpRXWdCjQ9t5rrwKZnJycnD11Vc3ef3u3bsRExPT7H2cOnUKFosF/fr1wz333IPjx4/j5ZdfRl5eHl5++WWYTCYoFA2HvpRKJSwWizfNdSORMNDrA/OG1OnUAblfMWpPfQXaV3+pr22XL/3t1iEa2UXVKK6yBOyz0xfn8yrBcc4S8s4d9A2mwNrTa+tLX1MTInE6pxLVFoeoXtfW8iqYSUxMxI4dO5q8PioqqsX7ePbZZzF37lzh2J49e0Iul2PmzJmYM2cOVCoVrFZrg9tZLBao1b6fpCzLwWDw7y6wUqkEOp0aBoMJDkfodgsNhvbUV6B99Zf62na1pr+JtWuSnLpYjvJy8aw3c+xsCQAgOVaDioq6z/T29Nq2pq+62i0NLhUYRPW6NkWnU3s0AuVVMCOXy9GtWzefGwUAMpmsQdDTo0cPAEBBQQGSkpJQUVEBq9XqNkJTVFSExMTEVj223R6YE9zhYAN232LTnvoKtK/+Ul/bLl/6m1o7tZRdWCWq5yq7sAqAM1+msXa1p9fWl77G1C6IWFRubFPPU9AnzKZNm4b58+e7Xfbnn39CLpejc+fOGDhwIFiWFRKBAeD8+fMoLCzE4MGDg91cQghpl/iKpoIyI2x2R4hbU0dI/hVRHk844deaaWsL5wU9mBk3bhz+85//4KOPPkJ2djZ27NiBl19+Gffddx8iIiKQmJiI6667DgsWLMDevXtx5MgRzJo1C0OGDMGAAQOC3VxCCGmXorQKRKjl4Dggr8S/U/StIQQz8RTM+CK23lozbUVAFs1rztSpU8EwDDZu3Ijnn38e8fHxuPvuu/Hggw8KxyxZsgTPP/88HnvsMQDAqFGjsGDBgmA3lRBC2i2GYdAhIQLHL5Yju6ganZIiQ90k2FwqmWhPJt/ERCrBMIDNzqKyxoroCGXLNwoDAQ1mNm7c2OjlU6ZMabaEW6PR4LnnnsNzzz0XqKYRQghpQWq8FscvlotmW4P8UueeTBqlDFG0J5NPZFIJYiKVKDVYUFJpbjPBTNspMieEEOJX/LYG2SLZ1iCvtG6KSWyrEocTfsPJkoq2s0cTBTOEEEIa5bqtgRhQ8q9/xNWW3belVYApmCGEENIo567UQJXRhsqahut/BVtusTOYoXyZ1hFGZtrQ7tkUzBBCCGmUUi5Fol4DQBw7aOeVOquqaE+m1omrLc8urqCRGUIIIe0AP9UU6rwZm92BonIKZvwhrg2uNUPBDCGEkCZ1qF3PJdR5MwVlJqGSid/5mfgmvnatmVKDGSzbNtaaoWCGEEJIk9JqK5pCPc2UW+J8fGceD1UytUZ0hBJSCQMHy6Gi2vcNnMWEghlCCCFN4qeZ8kprYA/hBo78KsQ0xdR6EgmDWB2fN9M2koApmCGEENKk2CgVVAop7A4OhWWh29ZAKMumYMYv2lp5NgUzhBBCmiRhmLqpptrS6FDILaGybH/ik4ApmCGEENIuhHrxPJudpUomP2trqwBTMEMIIaRZfEVTqMqzC8qcezKpqZLJb2hkhhBCSLsS6pGZPJcpJqpk8o+46La1CjAFM4QQQpqVGucMZsoMFtSYbUF//Fwh+VcT9Mduq+JrR2bKqiwhrVLzFwpmCCGENEujkgmlvKFYbyZfCGYigv7YbZVOq4BcJgHHOQOacEfBDCGEkBZ1SAhdRRONzPgfwzB1eTNtIAmYghlCCCEtSksIzbYGzkom5x/bVBqZ8avYNpQETMEMIYSQFoVqW4PCMiNYjoNaKaVKJj+Lj2o7ScAUzBBCCGmR6zQTywVvc8Jcl5V/qZLJv9rSKsAUzBBCCGlRgl4NuUwCi80R1ByLPFr5N2DqFs6jYIYQQkg7IJVIkBLLL54XvCRgYU+mWApm/I1PAC6maSZCCCHtRSiSgPNKa4OZeApm/I0PZiqrrbDZHSFuTetQMEMIIcQjHYKcBGyzsygsc44a0MiM/0Wo5VAqpADCP2+GghlCCCEeCfa2Bq6VTPpIZVAesz1hGEZYCbiUghlCCCHtAR/MFJWbYLEGflpCmGKiSqaA4ZOAiymYIYQQ0h7oNApEaRXgUFcyHUi5xZT8G2h1u2eHdxIwBTOEEEI8FsypJn5khsqyA6duSwMamSGEENJOpMXz5dlBCGZcFswjgREX3TZWAaZghhBCiMeCta2B3eFSyUTBTMDEtZH9mSiYIYQQ4rEOLtNMXAC3NSigSqag4BOAq4w2mK32ELfGdxTMEEII8VhyrBYShkGN2Y6KamvAHsd15V+qZAocjUoGrUoGILzLsymYIYQQ4jG5TILkWA2AwObNUL5M8MQK2xpQMEMIIaSdCEZFUy4FM0ETL2w4Gb5JwBTMEEII8Qpf0RTIJGDaLTt4YttAEjAFM4QQQrzCVzRlB2hkxu5gUVROlUzBEi+UZ1MwQwghpJ3gK5oKSo2w2Vm/339hmREOloNKQZVMwdAWVgGmYIYQQohX9JFKaJQyOFgO+aX+39bANV+GKpkCry2sAkzBDCGEEK8wDCMkAfP7J/kTVTIFF7/WjNFih9FsC3FrfEPBDCGEEK91CGDeDCX/BpdSIUWkRg4gfPNmKJghhBDitbSEwFU0UVl28PGjM8VhOtVEwQwhhBCv8dNM/h6Zca1kopGZ4OHzZkrDNAmYghlCCCFe4wONymorDEb/bWtAlUyhERcd3qsAUzBDCCHEayqFDAm165Pk+nGqKa/UCIAqmYKNXwU4XPdnomCGEEKIT+qmmvxX0ZRbO22VEktTTMEUJ+zPRNNMhBBC2hFhWwM/5s1QWXZoxPGrAFeYwXFciFvjPQpmCCGE+IRfCdifFU38NFNqPAUzwRSrc+YnWWwOVJvCb60ZCmYIIYT4RFg4r6QGLNv6b/N2B4vCstqcGZpmCiq5TIroCAWA8Fxrxu/BTH5+PmbNmoURI0Zg8ODBuO+++3D69Gm3Y7766itMmDAB/fr1w6RJk7B7926368vLyzF79mwMHjwYQ4YMweLFi2Eyhec8HiGEtFXx0Woo5BLY7CwKy42tvr/CchMcLAelQooYHVUyBVtcGG846ddgxmq14sEHH0RxcTHWrFmDzZs3Q6vVYvr06SgrKwMA7NmzB0899RT+/ve/47PPPsPw4cPx4IMP4uzZs8L9zJgxAxcvXsT777+P119/Hb/88gsWLVrkz6YSQghpJQnDCDto5/ghCVjIl4mlSqZQqNujKfwGD/wazBw4cACnTp3C8uXL0bdvX/To0QPLli2D0WjEjz/+CAB45513cM011+Cuu+5Ct27dMHfuXPTp0wcbNmwAABw+fBj79u3DSy+9hD59+mD48OF49tln8Z///AeFhYX+bC4hhJBW4pOAs/2QN8NXMtFieaEhrALc3kdmevTogbVr1yIxMbHuASTOhzAYDGBZFocOHcLw4cPdbjd06FDs378fgDMgio+PR7du3YTrhwwZAoZhcPDgQX82lxBCSCsJIzN+CGZc15ghwSeMzIRhebbMn3cWHx+PK6+80u2yjRs3wmw2Y8SIETAYDDAajUhKSnI7JiEhAQUFBQCAwsJCJCcnu12vUCgQHR2N/Pz8VrVPJvNvipBUKnH72Za1p74C7au/1Ne2Kxj97ZysA+BMAm7tZ2x+7TRTh8QIr++rPb22geprUowGgHPhPH//vQw0r4KZnJwcXH311U1ev3v3bsTExAi/f/fdd3jllVdw9913Iz09XQhYFAqF2+2USiUsFgsAwGQyNbi+/jG+kEgY6PWBifZ1OnVA7leM2lNfgfbVX+pr2xXI/mYonbstF1eYoFQroFHJfbofu4NFQW0lU+/u8dDrNT7dT3t6bf3d126s82dppRnR0ZqwylvyKphJTEzEjh07mrw+KipK+P9HH32EJUuWYOLEiZgzZw4AZ0ACOBOFXVksFqjVzhdFpVI1uJ4/RqPx7eQGAJblYDC0PtvelVQqgU6nhsFggsPB+vW+xaY99RVoX/2lvrZdwepvTKQSZVUWHD1VhB4don26j9ziamFPJhnHorzcu4Ti9vTaBqqvUo4FwwBWO4sL2eWIFsHeWDqd2qMRKK+CGblc7pbL0pRly5Zh3bp1uOeeezB37lwhuouOjoZGo0FRUZHb8UVFRUKeTVJSEr7//nu3661WKyoqKpCQkOBNcxuw2wNzgjscbMDuW2zaU1+B9tVf6mvbFej+piVEoKzKggsFVehSO+3krexCZ85NcqwWDgcHwLd1a9rTaxuIvsZEqlBqMKOgzIgItW+jbKHg90kxPpCZO3cu5s2b5zZMxTAMsrKysG/fPrfb7N27F4MGDQIADB48GAUFBbh48aJwPX/8wIED/d1cQgghreSPJOBcYRsD30fgSeuFa3m2X4OZvXv3Yt26dZg2bRpuuOEGFBcXC/9qapwn6j333IPt27fjvffew9mzZ/Hyyy/j+PHjmD59OgCgf//+yMrKwsyZM3HkyBHs2bMHCxcuxKRJk9yqpAghhIiDUJ7dij2a+GAmNS7CL20ivomL5jecDK/ybL9WM3355ZcAnBVMGzdudLvusccew+OPP44rrrgCzz//PN588028+uqr6N69O9asWSNMXzEMg1WrVmHx4sWYPn06lEolxo8fj/nz5/uzqYQQQvxE2NaguBocx/mUOJpPG0yKAr/WTGmYlWf7NZhZsmQJlixZ0uJxkyZNwqRJk5q8PjY2Fm+88YYfW0YIISRQkmI0kEoYmCwOlBrMwh9ET7lWMtE0U2jx00zFFeE1MhNeheSEEEJERyaVCCMqOUXeb2tQ5LInU6xO5e/mES/ER/MjMxTMEEIIaWf4JGBf8mbq9mQKr7VN2iJ+ZKbUYPbLTujBQsEMIYSQVuuQ4HtFUx7ly4hGdIQSUgkDB8uhvMr3hWqDjYIZQgghrZaWUDvN5MPITC4FM6IhkTDCVF847dFEwQwhhJBW46eZCsqMsNocXt02TyjLpmBGDPjy7JIwypuhYIYQQkirRWkViFDLwXFAXqnnScDulUwUzIhB3e7ZFMwQQghpRxiGccmb8TyYESqZ5FLEUCWTKPCl9eG0CjAFM4QQQvxC2NbAi7yZPJdtDCRUySQK4bgKMAUzhBBC/IJPAs72oqKpriybppjEIhxXAaZghhBCiF/w00zZRc5tDTzB59ekxFMwIxbxtTkzZVUW2B3hsQM5BTOEEEL8IiVWC4YBqk02GGqsHt0ml0ZmREenVUAuk4DjnAFNOKBghhBCiF8o5FIk6p17K3myErDdwaKg1FnJRGXZ4sEwTF1FU5gkAVMwQwghxG/SvKhoKq5wqWSKokomMREqmsIkCZiCGUIIIX7TId7zlYBzi50BT3IsVTKJTd1aMzQyQwghpJ1J82KPJj75l6aYxEdYBbiCRmYIIYS0Mx1q15rJK61psRKGNpgUL5pmIoQQ0m7FRqmgUkhhd3AorN2moCkUzIgXP81UTNNMhBBC2huGYYSppuYqmhws7ckkZnwwU1lthc3u3cahoUDBDCGEEL8StjVopqKpqNwEu4ODQi5BLFUyiU6EWg6lQgogPKaaKJghhBDiV55UNLluY0CVTOLDMIywEjAFM4QQQtodoaKpmWAml/JlRC+ckoApmCGEEOJXqXHOYKbMYEGN2dboMfzIDJVli1c4rQJMwQwhhBC/0qhkwh/Cptab4YOZZApmRCuOppkIIYS0Z0IScHHDJGDXSiYamRGvuGh+molGZgghhLRDQnl2IyMzVMkUHmhkhhBCSLuW1kxFkzDFRJVMosYnAFcZbTBb7SFuTfMomCGEEOJ3HWpHZnKLa8BynNt1lPwbHjQqGbQqGQDxj85QMEMIIcTvEvRqyGUSWGyOBtUwVJYdPmLDZKqJghlCCCF+J5VIhGAlu95KwHkltI1BuIjn15oReXk2BTOEEEICokN8w8XznJVMNDITLmhkhhBCSLsmrATsUtFUXGEWKpniqJJJ9OKjw2MVYApmCCGEBAS/R5Pr7tm5xVTJFE7CZRVgCmYIIYQERGrtyExxuUko7c0rcQY2KbE0xRQOwmWtGQpmCCGEBIROo0CUVgEOdRVMeaW1K//GUzATDvi1ZowWO4xN7LMlBhTMEEIICZj6eTP8NBONzIQHpUKKSI0cgLhHZyiYIYQQEjAdXPZoct2TKYVGZsIGPzpTXEHBDCGEkHYoLaF2W4Oi6tpKJhYKGVUyhZP4aOdrVSriDScpmCGEEBIwaS5rzVAlU3ji15oppmkmQggh7VFyrBZSCYMasx1/XSgDQIvlhZtwWAWYghlCCCEBI5dJkBSrAQAcOFEEAEiJ04SyScRLQnm2gUZmCCGEtFP8VFO1yVnamxoXEcrmEC/F8asAV5jB1dsBXSwomCGEEBJQafUql2hkJrzE6pQAAIvNIQSkYkPBDCGEkIDqkFA3EuOsZFKHsDXEW3KZFNERCgDiXWuGghlCCCEBxU8zAUBSrAYSCVUyhRt+qqlYpEnAFMwQQggJKH2kElqVDACQSpVMYYlPAi6lkRlCCCHtEcMwwugMlWWHJ2EVYJEGMzJ/32F+fj6WLVuGvXv3wmq1ol+/fpg3bx569OghHDN27FhcvHjR7XY33XQTXnzxRQBAeXk5nnvuOezcuRMMw+C6667DnDlzoFbTPCshhISjvw3rBJlMgsszkkPdFOKDut2zxTnN5Ndgxmq14sEHH0R0dDTWrFkDlUqFlStXYvr06fjyyy8RExMDo9GI7OxsvP322+jTp49wW5WqbmnrGTNmwGQy4f3334fBYMDTTz8No9GIl156yZ/NJYQQEiT9usWiX7fYUDeD+CieD2ZEuj+TX4OZAwcO4NSpU9i5cycSExMBAMuWLcPQoUPx448/4pZbbsGZM2fAsiwyMzMRFRXV4D4OHz6Mffv2YceOHejWrRsA4Nlnn8X999+PWbNmCfdLCCGEkOAQ1pqpNIPlONFtR+HXnJkePXpg7dq1bgGHROJ8CIPBAAA4efIk4uLiGg1kAGdAFB8fLwQyADBkyBAwDIODBw/6s7mEEEII8YA+UgmGAewOFoYaa6ib04BfR2bi4+Nx5ZVXul22ceNGmM1mjBgxAoAzmNFoNJgxYwYOHToEvV6Pm2++GXfddRckEgkKCwuRnOw+p6pQKBAdHY38/PxWtU8m82++s1QqcfvZlrWnvgLtq7/U17arPfWX+hpYMpkEsToVSirNKK+yCCM1YuFVMJOTk4Orr766yet3796NmJgY4ffvvvsOr7zyCu6++26kp6cDAE6fPg2DwYBx48bh0UcfxcGDB7Fs2TJUVlbiiSeegMlkgkKhaHDfSqUSFovFm+a6kUgY6PWByaLX6cT1ogZSe+or0L76S31tu9pTf6mvgZMUp0VJpRlGOxewv6e+8iqYSUxMxI4dO5q83nXq6KOPPsKSJUswceJEzJkzR7j8nXfegcViQWRkJAAgPT0d1dXVeOutt/D4449DpVLBam04hGWxWKDR+L4ENstyMBiMPt++MVKpBDqdGgaDCQ4H69f7Fpv21FegffWX+tp2taf+Ul8DT691DjRczK1AeRd9UB5Tp1N7NALlVTAjl8vdclmasmzZMqxbtw733HMP5s6dC8YlUUihUDQYeenZsyeMRiMqKyuRlJSE77//3u16q9WKiooKJCQkeNPcBuz2wLzoDgcbsPsWm/bUV6B99Zf62na1p/5SXwMnRuesaCoqN4ruOfb7hBsfyMydOxfz5s1zC2Q4jsM111yDVatWud3mzz//RHx8PPR6PQYPHoyCggK3dWj27dsHABg4cKC/m0sIIYQQD/BrzRSLsDzbrwnAe/fuxbp16zBt2jTccMMNKC4uFq7TaDTQarW49tprsX79enTt2hUZGRnYvXs31q1bh6effhoA0L9/f2RlZWHmzJlYtGgRjEYjFi5ciEmTJlFZNiGEEBIi8UJ5tvgWzvNrMPPll18CcFYwbdy40e26xx57DI8//jhmz56NiIgIrFixAgUFBUhLS8PTTz+N2267DYBz2etVq1Zh8eLFmD59OpRKJcaPH4/58+f7s6mEEEII8QI/MlNmsIBlOVFtGMpwHMeFuhHB4HCwKCur8et9ymQS6PValJfXiG7+0N/aU1+B9tVf6mvb1Z76S30NPJbl8NDyn+FgOSx7+HLERqlavlErxcRoPUoAbvsF+YQQQghpNYmEQaxOnHs0UTBDCCGEEI/ERfPBjLiSgCmYIYQQQohH4qKcScDFFTQyQwghhJAwxCcBl9LIDCGEEELCET/NVEzBDCGEEELCET/NVEoJwIQQQggJR/H8WjNVFthFtAcWBTOEEEII8YhOq4BcJgHHOQMasaBghhBCCCEeYRhGSAIuEVFFEwUzhBBCCPEYnzcjprVmKJghhBBCiMeEkRkRJQFTMEMIIYQQjwmrAFfQyAwhhBBCwhBNMxFCCCEkrPHTTMU0zUQIIYSQcBQf7RyZqay2wmpzhLg1ThTMEEIIIcRjWpUMSoUUAFBqEMdUEwUzhBBCCPEYwzDCSsBiyZuhYIYQQgghXhFbEjAFM4QQQgjxithWAaZghhBCCCFeqatoopEZQgghhIShuNqKplKRlGdTMEMIIYQQrwgjMyJZBZiCGUIIIYR4hU8ArjbZYLbaQ9waCmYIIYQQ4iWNSgatSgZAHBVNFMwQQgghxGuxIlprhoIZQgghhHgtnl9rRgTl2RTMEEIIIcRrcdE0MkMIIYSQMCamVYApmCGEEEKI18S0CjAFM4QQQgjxWhwlABNCCCEknPHTTEaLHUazLaRtoWCGEEIIIV5TKqSI1MgBhH4lYApmCCGEEOITsSQBUzBDCCGEEJ/EC+XZoU0CpmCGEEIIIT4RyyrAFMwQQgghxCdiWQWYghlCCCGE+EQozzbQyAwhhBBCwlBcND8yYwbHcSFrBwUzhBBCCPFJrE4FhVwCB8vCwYYumJGF7JEJIYQQEtbkMglm3tofNgcLmTR04yMUzBBCCCHEZ+kd9aFuAk0zEUIIISS8UTBDCCGEkLBGwQwhhBBCwhoFM4QQQggJaxTMEEIIISSsUTBDCCGEkLDm92Dm0qVLePjhhzFo0CAMGjQIs2bNQmFhodsxu3fvxuTJk9G/f3+MHz8e27dvd7veYrFg8eLFGD58ODIzMzF79myUlZX5u6mEEEIIaQP8GsxYrVbcfffdYFkWmzdvxsaNG1FUVIR//OMfwjLHZ8+exUMPPYSRI0di27ZtuPXWWzFnzhzs3r1buJ9Fixbh119/xcqVK7FhwwacO3cOM2bM8GdTCSGEENJG+HXRvPz8fPTt2xfPPPMMYmJiAAB33303Hn30UZSXlyMmJgYbNmxAeno6Zs6cCQDo1q0bjh07hnXr1mH48OEoLCzE559/jjVr1mDQoEEAgBUrVmD8+PE4fPgwMjMz/dlkQgghhIQ5v47MdOrUCa+//roQyOTl5eGjjz5Cnz59oNc7Vwg8cOAAhg8f7na7YcOG4eDBg+A4DgcPHhQu43Xp0gWJiYnYv3+/P5tLCCGEkDYgYNsZ3Hvvvdi1axeioqKwYcMGMAwDACgoKEBSUpLbsQkJCTCZTCgvL0dhYSH0ej2USmWDYwoKClrVJpnMvylC0tp9KKQh3I8iWNpTX4H21V/qa9vVnvpLfW3fvApmcnJycPXVVzd5/e7du4VRmaeeegpPPPEEVq9ejbvvvhuff/45kpOTYTaboVAo3G7H/261WmEymRpcDwBKpRIWi8Wb5rqRSBjo9Vqfb98cnU4dkPsVo/bUV6B99Zf62na1p/5SX9snr4KZxMRE7Nixo8nro6KihP9fdtllAIDXXnsNo0ePxqefforHHnsMSqUSVqvV7Xb872q1GiqVqsH1gLPCSa32/YVjWQ4Gg9Hn2zdGKpVAp1PDYDDB4WD9et9i0576CrSv/lJf26721F/qa9uk06k9GoHyKpiRy+Xo1q1bk9fn5+fjjz/+wPjx44XLNBoN0tLSUFRUBABITk4W/s8rKiqCRqNBZGQkkpKSUFFRAavV6jZCU1RUhMTERG+a60YiYQIWxWq1ypYPaiPaU1+B9tVf6mvb1Z76S31tWyQSxqPj/Jozc+LECTzxxBP46quv0LVrVwCAwWDA+fPnMXHiRADAoEGDsG/fPrfb7dmzB1lZWZBIJBg4cCBYlsXBgweFROHz58+jsLAQgwcP9rltDMNAKvXsSfFWe5q3bE99BdpXf6mvbVd76i/1tX3y6zMxYsQI9OrVC3PnzsXRo0fx119/YcaMGdDr9bj55psBANOmTcORI0ewfPlynD17Fu+++y6+/vpr3H///QCcU1nXXXcdFixYgL179+LIkSOYNWsWhgwZggEDBvizuYQQQghpAxiOX83OT4qKivDSSy9h165dsFqtuOKKKzB//nwkJycLx+zcuRPLli3DhQsXkJaWhscffxwTJkwQrjcajXj++efxzTffAABGjRqFBQsWCOXdhBBCCCE8vwczhBBCCCHBRBNuhBBCCAlrFMwQQgghJKxRMEMIIYSQsEbBDCGEEELCGgUzhBBCCAlrFMwQQgghJKxRMEMIIYSQsEbBDCGEEELCGgUzhBBCCAlrFMwQQgghJKxRMNMMlmXxxhtvYOTIkRgwYAAeeOABZGdnN3l8eXk5Zs+ejcGDB2PIkCFYvHgxTCZTEFvsu4qKCixcuBCjRo1CVlYW7rjjDhw4cKDJ49966y2kp6c3+BcuCgsLG23/tm3bGj0+XF/bvXv3NtrP9PR0XH311Y3e5uDBg40ev3fv3iC33jtvv/02pk2b5nbZ8ePHMXXqVAwYMABjxozBBx980OL9fPXVV5gwYQL69euHSZMmYffu3YFqcqs01t8ff/wRN998MzIzMzFmzBi89NJLMJvNTd6Hw+FAv379GrzWK1euDHTzvdJYXxcsWNCg3WPGjGn2fsLhta3f12nTpjX5Hv7888+bvJ977rmnwfH1n8M2hSNNWrlyJTd06FDup59+4o4fP87de++93NixYzmLxdLo8VOnTuVuvvlm7ujRo9xvv/3GjR49mpszZ06QW+2be+65h7v++uu5/fv3c+fOneMWL17M9evXjzt79myjxz/xxBPcU089xRUVFbn9Cxc///wz17dvX66wsNCt/SaTqdHjw/W1tVgsDV6jb7/9lktPT+e2bt3a6G0+/PBD7pprrmlwu6bOezHYtGkT16tXL27q1KnCZWVlZdzQoUO5+fPnc2fOnOG2bt3K9e3bt8l+cxzH7d69m+vTpw+3YcMG7syZM9yLL77IZWRkcGfOnAlGNzzWWH/379/PXXbZZdxbb73FnT9/nvv555+5UaNGcfPmzWvyfs6cOcP17NmTO378uNtrXV1dHYxueKSxvnIcx91yyy3cihUr3NpdWlra5P2Ew2vbWF/Ly8vd+lhYWMjdeeed3HXXXdfs6zR8+HBu8+bNbrctLy8PQi9Cg4KZJlgsFi4zM5P78MMPhcsqKyu5fv36cV988UWD4w8dOsT17NnT7Y3xv//9j0tPT+cKCgqC0mZfXbhwgevZsyd34MAB4TKWZblrrrmGe+211xq9zd/+9jfuvffeC1IL/W/t2rXcDTfc4NGx4fza1ldTU8ONHj262T9wzzzzDPePf/wjiK3yXUFBAffQQw9xAwYM4MaPH+/2R2DNmjXcFVdcwdlsNuGyV155hRs7dmyT93fvvfdyTzzxhNtlt99+O/evf/3L7233RXP9nT17Nnf33Xe7Hf/ZZ59xffr0aTIQ3b59O5eVlRXQNvuqub6yLMsNGDCA+/bbbz2+PzG/ts31tb6NGzdyGRkZTX7R5DiOKykp4Xr27Mn99ddfgWiuKNE0UxNOnDiBmpoaDB8+XLhMp9Ohd+/e2L9/f4PjDxw4gPj4eHTr1k24bMiQIWAYBgcPHgxKm32l1+uxdu1a9O3bV7iMYRgwDAODwdDgeKvVigsXLqBr167BbKZfnTx50u21ak44v7b1rVmzBiaTCXPnzm3yGG+em1D766+/IJfL8d///hf9+/d3u+7AgQMYMmQIZDKZcNmwYcNw4cIFlJSUNLgvlmVx6NAht/c8AAwdOrTR93woNNffe++9t8HrKpFIYLPZUF1d3ej9ifm1bq6vly5dgtFo9PgzSOyvbXN9dVVWVobXXnsNDz/8cLN9P3nyJBiGQZcuXQLRXFGStXxI+1RQUAAASE5Odrs8ISFBuM5VYWFhg2MVCgWio6ORn58fuIb6gU6nw5VXXul22TfffIOLFy/in//8Z4Pjz5w5A4fDgW+++QZLly6FxWLB4MGD8dRTTyEhISFYzW6VU6dOQa/XY8qUKTh//jw6deqEhx9+GKNGjWpwbDi/tq7Kysrw/vvvY/bs2YiOjm7yuNOnT0Ov12Py5MkoLCxEz549MXPmTPTr1y94jfXQmDFjmsyTKCgoQM+ePd0u48/P/Px8xMXFuV1nMBhgNBqRlJTU4DaNvedDobn+9u7d2+13m82G999/HxkZGYiJiWn0NqdOnYLdbsd9992HEydOIDExEdOnT8eNN97o97Z7q7m+njp1CgCwceNG7Ny5ExKJBKNGjcLMmTMRGRnZ4Hixv7bN9dXVO++8A5VKhfvuu6/Z406dOoXIyEg8++yz2LVrFzQaDcaPH49HHnkECoXCX80WFRqZaQKf3Fn/hVcqlbBYLI0e39hJ0tTxYnbo0CHMnz8fY8eOxVVXXdXgev6DRK1W4/XXX8fSpUtx7tw53HXXXc0mG4qF3W7HuXPnUFlZiccffxxr167FgAED8OCDDzaaENhWXtvNmzcjMjISt99+e5PH5Ofno6qqCkajEQsWLMCbb76JuLg4TJ06FWfOnAlia1vPbDY3+v4F0Ojrxp+7nr7nxcxut2POnDk4ffo0nnnmmSaPO336NCoqKjBt2jSsX78e48aNw/z587F169YgttZ7p06dgkQiQUJCAtasWYN58+bh119/xSOPPAKWZRsc3xZe2+rqanz88ce47777hPO4KadOnYLFYkG/fv2wbt06PPzww/jkk0+wYMGCILU2+GhkpgkqlQqAc0qF/z/g/BBUq9WNHm+1WhtcbrFYoNFoAtdQP/v+++/xf//3f8jKysLy5csbPWbSpEkYNWqU27e9Hj16YNSoUfjxxx8xYcKEYDXXJzKZDHv37oVUKhVe24yMDJw+fRrr169vMBTdVl7bzz//HJMmTXI7n+tLTk7G/v37oVarIZfLAQB9+/bFsWPHsHHjRixevDhYzW21xl43/g9XY68b/weisds09p4Xq+rqajz55JPYt28fVq1a1eyI2pdffgmHwwGtVgsA6NWrF/Ly8rB+/XrccsstwWqy1x5++GHceeed0Ov1AICePXsiPj4et912G/78888GUzVt4bX9/vvvYbVacfPNN7d47LPPPou5c+ciKioKgPP5kcvlmDlzJubMmdNgVLItoJGZJvDTCkVFRW6XFxUVITExscHxSUlJDY61Wq2oqKgIm6mXTZs24fHHH8fo0aOxZs2aZqP/+sPWCQkJiI6OFsWQrSe0Wm2DP+o9evRAYWFhg2Pbwmt74sQJZGdn44YbbmjxWJ1OJwQygDPvolu3bo0+N2LW2OvG/97Yezg6Ohoajcbj97wYFRUVYcqUKfj999+xfv36BtPH9alUKiGQ4fXs2VP072OJRCIEMrwePXoAQKNtbwuv7ffff48rr7wSOp2uxWNlMpkQyPCae37aAgpmmtCrVy9ERES4ra1hMBhw7NgxDB48uMHxgwcPRkFBAS5evChctm/fPgDAwIEDA9/gVtq8eTOWLFmCKVOmYMWKFc3Oq7766qsYN24cOI4TLsvJyUF5eTm6d+8ejOa2yunTp5GVldVg3ZSjR4822v5wf20BZzJsbGwsevXq1exxO3fuRGZmptt6Sna7HSdOnAiL19bV4MGDcfDgQTgcDuGyPXv2oEuXLoiNjW1wPMMwyMrKEl5b3t69ezFo0KCAt7e1KisrMX36dJSVleHDDz9s9HPKlcFgwJAhQxqsrfTnn38Kf/jEas6cObj77rvdLvvzzz8BoNHzNNxfW8D5Hq4/atyUadOmYf78+W6X/fnnn5DL5ejcuXMAWhd6FMw0QaFQYOrUqVi+fDl++OEHnDhxAjNnzkRSUhLGjh0Lh8OB4uJiYS62f//+yMrKwsyZM3HkyBHs2bMHCxcuxKRJk0Qf+Z8/fx7PP/88rr32Wjz00EMoKSlBcXExiouLUVVVBavViuLiYmGI9tprr0Vubi4WLVqE8+fPY//+/Xj88ceRlZWFkSNHhrg3LevWrRu6du2KZ599FgcOHMDZs2fxwgsv4Pfff8fDDz/cpl5b3rFjx5pc1LC4uBg1NTXA/7d3/zFVlv8fx5+EhxlhgBA2s8bagjbPQXBxgPAHUI0Sm4WbtIDIAsslhj/WgWSl02EDlEACS4JGlqs0Fw3+SXOwNIc2Z1nkQtPAHwQKCCYVcH3+YJ7JB/x+Px818Xz2emznj3Pf1839vq/7HHhxXfd9DjB9+nR8fX1xOBwcOXKEo0eP4nA46OrqGvHH41Y3f/58ent7WbVqFc3NzXz++ed88MEHvPTSS842PT09nD9/3vl84cKF1NbWUlVVxbFjx8jPz6epqYm0tLSxOIT/yvr162lpaaGgoICJEyc638Pt7e3OQNfV1UVXVxcwNAIXGRlJUVER9fX1nDhxgvfee4+amhoyMzPH8Ej+f/Hx8Xz77beUlpby22+/UV9fz+uvv87cuXOdd2f9L53bM2fO0NnZedV/Ri5evEh7e7vzeXx8PF988QXbtm2jpaWFuro68vPzefHFF/Hy8rpZZd9cY31v+K2sv7/f5Ofnm8jISBMaGmoyMjJMS0uLMcaYlpYWExQUZHbs2OFs39HRYTIzM01oaKiJiIgwb775punr6xur8v9j5eXlJigoaNSHw+Ew+/fvN0FBQWb//v3Obfbt22eSkpJMaGiosdvtJicnx3R1dY3hUfx32tvbTXZ2tomOjjY2m80kJSWZAwcOGGP+t87tZenp6SYrK2vUdUFBQaakpMT5/OTJkyYzM9PY7XYzbdo088ILL5ijR4/erFKvmcPhGPH5HIcPHzYLFiwwVqvVxMbGmg8//HDENrGxscOW7dy50zz22GPGZrOZp59+2uzbt+8fr/1aXHm8/f39xmazXfV9fPn3VkpKyrA+6unpMXl5eWb27NnGarWaefPmma+++mpMjuf/Mtq5raurM0899ZQJCQkx0dHR5q233hr2nnTVc3u11/G/f9bVlUpKSkxQUNCwZVu3bjVPPPGE87VfXl5uBgYG/rG6x5qbMVfMFYiIiIi4GE0ziYiIiEtTmBERERGXpjAjIiIiLk1hRkRERFyawoyIiIi4NIUZERERcWkKMyIiIuLSFGZERETEpSnMiIhLaG1tJTg4eMR3CV2L7Oxs4uLibkBVInIrGDfWBYiI/CcCAgL45JNPuO+++8a6FBG5xSjMiIhL8PDwIDQ0dKzLEJFbkKaZROSG+Oyzz0hISMBqtRITE8OmTZuc39acnZ1Namoq27dvJzY2lrCwMNLS0vj555+d2w8ODlJUVERcXBxWq5W4uDg2bNjA33//DYw+zXTixAmWLl1KdHQ0oaGhpKam8t133w2rq7u7m5ycHOx2O+Hh4RQUFDA4ODii/l27dpGYmIjNZiM6Opp169bxxx9/ONf39fWxevVqZs2ahdVq5fHHH+f999+/oX0oItdGIzMict3effddioqKSElJIScnh6amJjZt2sSZM2fIy8sDoKmpiePHj7N8+XK8vb0pKSkhJSWFuro6AgIC2LJlC9u2bcPhcHDvvfdy+PBhioqKsFgsLF26dMQ+m5ubWbBgAYGBgeTm5mKxWKiuriYtLY3KykrsdjuDg4Okp6dz6tQpHA4HPj4+VFRU8MMPPxAQEOD8WV9++SUrV67kySefJCsri1OnTlFUVERzczNVVVW4ubmRl5fHN998g8PhwN/fn4aGBvLz8/Hx8WH+/Pk3ra9FZCSFGRG5Lj09PZSVlZGUlERubi4AM2bMwMfHh9zcXBYuXOhst3nzZh566CEAQkJCePTRR6murmblypU0NjZitVqdwcBut3P77bczYcKEUfdbWlqKh4cH1dXVeHl5ARATE8PcuXPJz89n+/btNDQ08P3337NlyxZmzZoFQFRU1LCLf40xFBYWMnPmTAoLC53LAwMDef7556mvrycmJobGxkaio6NJSEgAICIiAk9PT/z8/G5kd4rINdA0k4hcl0OHDtHX10dcXBz9/f3Ox+XAsHfvXgCmTJniDDIwdEFvWFgYBw4cAIbCwd69e3n22WepqKigubmZlJQU5s2bN+p+GxsbiY2NdQYZgHHjxpGQkMCRI0e4ePEiBw8exGKxMHPmTGcbT09PZs+e7Xx+/Phxzp49O6L+8PBwvLy8nPVHRETw6aefkpGRwdatW2lpaeGVV14hJibmxnSkiFwzjcyIyHXp6uoCYNGiRaOu//333wGYNGnSiHV+fn78+OOPAKSnp3PHHXewY8cOCgsLKSgo4IEHHiA3N5fIyMgR23Z3d+Pv7z9iub+/P8YYent76e7uxsfHBzc3t2Ft7rrrrhH1r1mzhjVr1ly1/lWrVnH33XdTU1PD2rVrWbt2LWFhYaxevZoHH3xw1GMXkZtDYUZErsudd94JQGFhIYGBgSPW+/v7U1xcTGdn54h1HR0dzmma2267jeTkZJKTkzl37hz19fVs3ryZzMxM5+jIlby9veno6BixvL29HQBfX198fX3p7OxkYGAAd3d3Z5vLAebK+l977TXsdvuo+4Ghu6kWL17M4sWLOX36NHv27KGsrIwVK1ZQW1t7te4RkZtA00wicl2mTZuGxWKhra0Nm83mfIwbN46NGzfS2toKDN15dOzYMed2bW1tHDp0iKioKACeeeYZ1q1bBwyN2CQmJpKcnMyFCxfo7e0dsd/w8HD27NkzbN3AwAC1tbXYbDY8PDyIioqiv7+fXbt2Odv89ddfw8LR/fffj5+fH62trcPqnzRpEhs2bOCnn36ir6+P+Ph4KisrAZg8eTLJyckkJCRw+vTpG9ibInItNDIjItfF19eX9PR0iouL6e3tJSIigra2NoqLi3Fzc3NOwRhjePnll1m2bBnu7u6Ulpbi7e1NamoqMBROKisr8ff3JywsjLa2NqqqqrDb7UycOHHYbdIAS5YsoaGhgeeee45FixZhsVic17JUVFQAQxf7zpgxg9zcXM6dO8c999xDdXU158+fd44Iubu7s2zZMt544w3c3d2JjY3lwoULlJWV0dbWxtSpUxk/fjxTp06ltLQUi8VCcHAwv/76Kzt37iQ+Pv4m9raIjMbNGGPGuggRcX0fffQRH3/8MSdPnsTb25uoqCiWL1/O5MmTyc7OprGxkYyMDN555x0uXbrEww8/jMPhYMqUKQD09/dTXl5OTU0NZ8+eZcKECcTFxbFixQp8fX1pbW3lkUceYf369SQmJgJDt3tv3LiRgwcP4ubmRkhICEuWLBl2ofGlS5coLCyktraWP//8kzlz5uDp6cnu3bv5+uuvne3q6uqoqKjgl19+wdPTk+nTp5OVlUVwcDAAvb29vP322+zevZv29nb8/PyYM2cOr776KuPHj7+JPS0i/05hRkT+cZfDzJXhQUTkRtE1MyIiIuLSFGZERETEpWmaSURERFyaRmZERETEpSnMiIiIiEtTmBERERGXpjAjIiIiLk1hRkRERFyawoyIiIi4NIUZERERcWkKMyIiIuLS/gVm3uHj07uzGwAAAABJRU5ErkJggg==", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "# 获取参数\n", - "cfg = get_args() \n", - "# 训练\n", - "env, agent = env_agent_config(cfg)\n", - "res_dic = train(cfg, env, agent)\n", - " \n", - "plot_rewards(res_dic['rewards'], cfg, tag=\"train\") \n", - "# 测试\n", - "res_dic = test(cfg, env, agent)\n", - "plot_rewards(res_dic['rewards'], cfg, tag=\"test\") # 画出结果" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.13 ('parl')", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.13" - }, - "orig_nbformat": 4, - "vscode": { - "interpreter": { - "hash": "29c8e495d55843cb894bac6655c13e4a65f834e86169d4dce1750654c48fe628" - } - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/projects/parl_tutorials/DQN.ipynb b/projects/parl_tutorials/DQN.ipynb deleted file mode 100644 index 4b02022..0000000 --- a/projects/parl_tutorials/DQN.ipynb +++ /dev/null @@ -1,538 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1、定义算法\n", - "相比于Q learning，DQN本质上是为了适应更为复杂的环境，并且经过不断的改良迭代，到了Nature DQN（即Volodymyr Mnih发表的Nature论文）这里才算是基本完善。DQN主要改动的点有三个：\n", - "* 使用深度神经网络替代原来的Q表：这个很容易理解原因\n", - "* 使用了经验回放（Replay Buffer）：这个好处有很多，一个是使用一堆历史数据去训练，比之前用一次就扔掉好多了，大大提高样本效率，另外一个是面试常提到的，减少样本之间的相关性，原则上获取经验跟学习阶段是分开的，原来时序的训练数据有可能是不稳定的，打乱之后再学习有助于提高训练的稳定性，跟深度学习中划分训练测试集时打乱样本是一个道理。\n", - "* 使用了两个网络：即策略网络和目标网络，每隔若干步才把每步更新的策略网络参数复制给目标网络，这样做也是为了训练的稳定，避免Q值的估计发散。想象一下，如果当前有个transition（这个Q learning中提过的，一定要记住！！！）样本导致对Q值进行了较差的过估计，如果接下来从经验回放中提取到的样本正好连续几个都这样的，很有可能导致Q值的发散（它的青春小鸟一去不回来了）。再打个比方，我们玩RPG或者闯关类游戏，有些人为了破纪录经常Save和Load，只要我出了错，我不满意我就加载之前的存档，假设不允许加载呢，就像DQN算法一样训练过程中会退不了，这时候是不是搞两个档，一个档每帧都存一下，另外一个档打了不错的结果再存，也就是若干个间隔再存一下，到最后用间隔若干步数再存的档一般都比每帧都存的档好些呢。当然你也可以再搞更多个档，也就是DQN增加多个目标网络，但是对于DQN则没有多大必要，多几个网络效果不见得会好很多。" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 1.1 定义模型" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[32m[09-26 17:18:11 MainThread @utils.py:73]\u001b[0m paddlepaddle version: 2.3.2.\n" - ] - } - ], - "source": [ - "\n", - "import paddle\n", - "import paddle.nn as nn\n", - "import paddle.nn.functional as F\n", - "import parl\n", - "\n", - "class MLP(parl.Model):\n", - " \"\"\" Linear network to solve Cartpole problem.\n", - " Args:\n", - " input_dim (int): Dimension of observation space.\n", - " output_dim (int): Dimension of action space.\n", - " \"\"\"\n", - "\n", - " def __init__(self, input_dim, output_dim):\n", - " super(MLP, self).__init__()\n", - " hidden_dim1 = 256\n", - " hidden_dim2 = 256\n", - " self.fc1 = nn.Linear(input_dim, hidden_dim1)\n", - " self.fc2 = nn.Linear(hidden_dim1, hidden_dim2)\n", - " self.fc3 = nn.Linear(hidden_dim2, output_dim)\n", - "\n", - " def forward(self, state):\n", - " x = F.relu(self.fc1(state))\n", - " x = F.relu(self.fc2(x))\n", - " x = self.fc3(x)\n", - " return x" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1.2 定义经验回放" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [], - "source": [ - "from collections import deque\n", - "class ReplayBuffer:\n", - " def __init__(self, capacity: int) -> None:\n", - " self.capacity = capacity\n", - " self.buffer = deque(maxlen=self.capacity)\n", - " def push(self,transitions):\n", - " '''_summary_\n", - " Args:\n", - " trainsitions (tuple): _description_\n", - " '''\n", - " self.buffer.append(transitions)\n", - " def sample(self, batch_size: int, sequential: bool = False):\n", - " if batch_size > len(self.buffer):\n", - " batch_size = len(self.buffer)\n", - " if sequential: # sequential sampling\n", - " rand = random.randint(0, len(self.buffer) - batch_size)\n", - " batch = [self.buffer[i] for i in range(rand, rand + batch_size)]\n", - " return zip(*batch)\n", - " else:\n", - " batch = random.sample(self.buffer, batch_size)\n", - " return zip(*batch)\n", - " def clear(self):\n", - " self.buffer.clear()\n", - " def __len__(self):\n", - " return len(self.buffer)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1.3 定义智能体" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "from random import random\n", - "import parl\n", - "import paddle\n", - "import math\n", - "import numpy as np\n", - "\n", - "\n", - "class DQNAgent(parl.Agent):\n", - " \"\"\"Agent of DQN.\n", - " \"\"\"\n", - "\n", - " def __init__(self, algorithm, memory,cfg):\n", - " super(DQNAgent, self).__init__(algorithm)\n", - " self.n_actions = cfg['n_actions']\n", - " self.epsilon = cfg['epsilon_start']\n", - " self.sample_count = 0 \n", - " self.epsilon_start = cfg['epsilon_start']\n", - " self.epsilon_end = cfg['epsilon_end']\n", - " self.epsilon_decay = cfg['epsilon_decay']\n", - " self.batch_size = cfg['batch_size']\n", - " self.global_step = 0\n", - " self.update_target_steps = 600\n", - " self.memory = memory # replay buffer\n", - "\n", - " def sample_action(self, state):\n", - " self.sample_count += 1\n", - " # epsilon must decay(linear,exponential and etc.) for balancing exploration and exploitation\n", - " self.epsilon = self.epsilon_end + (self.epsilon_start - self.epsilon_end) * \\\n", - " math.exp(-1. * self.sample_count / self.epsilon_decay) \n", - " if random.random() < self.epsilon:\n", - " action = np.random.randint(self.n_actions)\n", - " else:\n", - " action = self.predict_action(state)\n", - " return action\n", - "\n", - " def predict_action(self, state):\n", - " state = paddle.to_tensor(state , dtype='float32')\n", - " q_values = self.alg.predict(state) # self.alg 是自带的算法\n", - " action = q_values.argmax().numpy()[0]\n", - " return action\n", - "\n", - " def update(self):\n", - " \"\"\"Update model with an episode data\n", - " Args:\n", - " obs(np.float32): shape of (batch_size, obs_dim)\n", - " act(np.int32): shape of (batch_size)\n", - " reward(np.float32): shape of (batch_size)\n", - " next_obs(np.float32): shape of (batch_size, obs_dim)\n", - " terminal(np.float32): shape of (batch_size)\n", - " Returns:\n", - " loss(float)\n", - " \"\"\"\n", - " if len(self.memory) < self.batch_size: # when transitions in memory donot meet a batch, not update\n", - " return\n", - " \n", - " if self.global_step % self.update_target_steps == 0:\n", - " self.alg.sync_target()\n", - " self.global_step += 1\n", - " state_batch, action_batch, reward_batch, next_state_batch, done_batch = self.memory.sample(\n", - " self.batch_size)\n", - " action_batch = np.expand_dims(action_batch, axis=-1)\n", - " reward_batch = np.expand_dims(reward_batch, axis=-1)\n", - " done_batch = np.expand_dims(done_batch, axis=-1)\n", - "\n", - " state_batch = paddle.to_tensor(state_batch, dtype='float32')\n", - " action_batch = paddle.to_tensor(action_batch, dtype='int32')\n", - " reward_batch = paddle.to_tensor(reward_batch, dtype='float32')\n", - " next_state_batch = paddle.to_tensor(next_state_batch, dtype='float32')\n", - " done_batch = paddle.to_tensor(done_batch, dtype='float32')\n", - " loss = self.alg.learn(state_batch, action_batch, reward_batch, next_state_batch, done_batch) " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2. 定义训练" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "def train(cfg, env, agent):\n", - " ''' 训练\n", - " '''\n", - " print(f\"开始训练！\")\n", - " print(f\"环境：{cfg['env_name']}，算法：{cfg['algo_name']}，设备：{cfg['device']}\")\n", - " rewards = [] # record rewards for all episodes\n", - " steps = []\n", - " for i_ep in range(cfg[\"train_eps\"]):\n", - " ep_reward = 0 # reward per episode\n", - " ep_step = 0\n", - " state = env.reset() # reset and obtain initial state\n", - " for _ in range(cfg['ep_max_steps']):\n", - " ep_step += 1\n", - " action = agent.sample_action(state) # sample action\n", - " next_state, reward, done, _ = env.step(action) # update env and return transitions\n", - " agent.memory.push((state, action, reward,next_state, done)) # save transitions\n", - " state = next_state # update next state for env\n", - " agent.update() # update agent\n", - " ep_reward += reward #\n", - " if done:\n", - " break\n", - " steps.append(ep_step)\n", - " rewards.append(ep_reward)\n", - " if (i_ep + 1) % 10 == 0:\n", - " print(f\"回合：{i_ep+1}/{cfg['train_eps']}，奖励：{ep_reward:.2f}，Epislon: {agent.epsilon:.3f}\")\n", - " print(\"完成训练！\")\n", - " env.close()\n", - " res_dic = {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps}\n", - " return res_dic\n", - "\n", - "def test(cfg, env, agent):\n", - " print(\"开始测试！\")\n", - " print(f\"环境：{cfg['env_name']}，算法：{cfg['algo_name']}，设备：{cfg['device']}\")\n", - " rewards = [] # record rewards for all episodes\n", - " steps = []\n", - " for i_ep in range(cfg['test_eps']):\n", - " ep_reward = 0 # reward per episode\n", - " ep_step = 0\n", - " state = env.reset() # reset and obtain initial state\n", - " for _ in range(cfg['ep_max_steps']):\n", - " ep_step+=1\n", - " action = agent.predict_action(state) # predict action\n", - " next_state, reward, done, _ = env.step(action) \n", - " state = next_state \n", - " ep_reward += reward \n", - " if done:\n", - " break\n", - " steps.append(ep_step)\n", - " rewards.append(ep_reward)\n", - " print(f\"回合：{i_ep+1}/{cfg['test_eps']}，奖励：{ep_reward:.2f}\")\n", - " print(\"完成测试！\")\n", - " env.close()\n", - " return {'episodes':range(len(rewards)),'rewards':rewards,'steps':steps}\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 3. 定义环境" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/Users/jj/opt/anaconda3/envs/easyrl/lib/python3.7/site-packages/gym/envs/registration.py:250: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.\n", - " for plugin in metadata.entry_points().get(entry_point, []):\n" - ] - } - ], - "source": [ - "import gym\n", - "import paddle\n", - "import numpy as np\n", - "import random\n", - "import os\n", - "from parl.algorithms import DQN\n", - "def all_seed(env,seed = 1):\n", - " ''' omnipotent seed for RL, attention the position of seed function, you'd better put it just following the env create function\n", - " Args:\n", - " env (_type_): \n", - " seed (int, optional): _description_. Defaults to 1.\n", - " '''\n", - " print(f\"seed = {seed}\")\n", - " env.seed(seed) # env config\n", - " np.random.seed(seed)\n", - " random.seed(seed)\n", - " paddle.seed(seed)\n", - " \n", - "def env_agent_config(cfg):\n", - " ''' create env and agent\n", - " '''\n", - " env = gym.make(cfg['env_name']) \n", - " if cfg['seed'] !=0: # set random seed\n", - " all_seed(env,seed=cfg[\"seed\"]) \n", - " n_states = env.observation_space.shape[0] # print(hasattr(env.observation_space, 'n'))\n", - " n_actions = env.action_space.n # action dimension\n", - " print(f\"n_states: {n_states}, n_actions: {n_actions}\")\n", - " cfg.update({\"n_states\":n_states,\"n_actions\":n_actions}) # update to cfg paramters\n", - " model = MLP(n_states,n_actions)\n", - " algo = DQN(model, gamma=cfg['gamma'], lr=cfg['lr'])\n", - " memory = ReplayBuffer(cfg[\"memory_capacity\"]) # replay buffer\n", - " agent = DQNAgent(algo,memory,cfg) # create agent\n", - " return env, agent" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 4. 设置参数" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/Users/jj/opt/anaconda3/envs/easyrl/lib/python3.7/site-packages/seaborn/rcmod.py:82: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.\n", - " if LooseVersion(mpl.__version__) >= \"3.0\":\n", - "/Users/jj/opt/anaconda3/envs/easyrl/lib/python3.7/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.\n", - " other = LooseVersion(other)\n" - ] - } - ], - "source": [ - "import argparse\n", - "import seaborn as sns\n", - "import matplotlib.pyplot as plt\n", - "def get_args():\n", - " \"\"\" \n", - " \"\"\"\n", - " parser = argparse.ArgumentParser(description=\"hyperparameters\") \n", - " parser.add_argument('--algo_name',default='DQN',type=str,help=\"name of algorithm\")\n", - " parser.add_argument('--env_name',default='CartPole-v0',type=str,help=\"name of environment\")\n", - " parser.add_argument('--train_eps',default=200,type=int,help=\"episodes of training\") # 训练的回合数\n", - " parser.add_argument('--test_eps',default=20,type=int,help=\"episodes of testing\") # 测试的回合数\n", - " parser.add_argument('--ep_max_steps',default = 100000,type=int,help=\"steps per episode, much larger value can simulate infinite steps\")\n", - " parser.add_argument('--gamma',default=0.99,type=float,help=\"discounted factor\") # 折扣因子\n", - " parser.add_argument('--epsilon_start',default=0.95,type=float,help=\"initial value of epsilon\") # e-greedy策略中初始epsilon\n", - " parser.add_argument('--epsilon_end',default=0.01,type=float,help=\"final value of epsilon\") # e-greedy策略中的终止epsilon\n", - " parser.add_argument('--epsilon_decay',default=200,type=int,help=\"decay rate of epsilon\") # e-greedy策略中epsilon的衰减率\n", - " parser.add_argument('--memory_capacity',default=200000,type=int) # replay memory的容量\n", - " parser.add_argument('--memory_warmup_size',default=200,type=int) # replay memory的预热容量\n", - " parser.add_argument('--batch_size',default=64,type=int,help=\"batch size of training\") # 训练时每次使用的样本数\n", - " parser.add_argument('--targe_update_fre',default=200,type=int,help=\"frequency of target network update\") # target network更新频率\n", - " parser.add_argument('--seed',default=10,type=int,help=\"seed\") \n", - " parser.add_argument('--lr',default=0.0001,type=float,help=\"learning rate\")\n", - " parser.add_argument('--device',default='cpu',type=str,help=\"cpu or gpu\") \n", - " args = parser.parse_args([]) \n", - " args = {**vars(args)} # type(dict) \n", - " return args\n", - "def smooth(data, weight=0.9): \n", - " '''用于平滑曲线，类似于Tensorboard中的smooth\n", - "\n", - " Args:\n", - " data (List):输入数据\n", - " weight (Float): 平滑权重，处于0-1之间，数值越高说明越平滑，一般取0.9\n", - "\n", - " Returns:\n", - " smoothed (List): 平滑后的数据\n", - " '''\n", - " last = data[0] # First value in the plot (first timestep)\n", - " smoothed = list()\n", - " for point in data:\n", - " smoothed_val = last * weight + (1 - weight) * point # 计算平滑值\n", - " smoothed.append(smoothed_val) \n", - " last = smoothed_val \n", - " return smoothed\n", - "\n", - "def plot_rewards(rewards,cfg,path=None,tag='train'):\n", - " sns.set()\n", - " plt.figure() # 创建一个图形实例，方便同时多画几个图\n", - " plt.title(f\"{tag}ing curve on {cfg['device']} of {cfg['algo_name']} for {cfg['env_name']}\")\n", - " plt.xlabel('epsiodes')\n", - " plt.plot(rewards, label='rewards')\n", - " plt.plot(smooth(rewards), label='smoothed')\n", - " plt.legend()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 5. 收获成果！" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "seed = 10\n", - "n_states: 4, n_actions: 2\n", - "开始训练！\n", - "环境：CartPole-v0，算法：DQN，设备：cpu\n", - "回合：10/200，奖励：10.00，Epislon: 0.062\n", - "回合：20/200，奖励：85.00，Epislon: 0.014\n", - "回合：30/200，奖励：41.00，Epislon: 0.011\n", - "回合：40/200，奖励：31.00，Epislon: 0.010\n", - "回合：50/200，奖励：22.00，Epislon: 0.010\n", - "回合：60/200，奖励：10.00，Epislon: 0.010\n", - "回合：70/200，奖励：10.00，Epislon: 0.010\n", - "回合：80/200，奖励：22.00，Epislon: 0.010\n", - "回合：90/200，奖励：30.00，Epislon: 0.010\n", - "回合：100/200，奖励：20.00，Epislon: 0.010\n", - "回合：110/200，奖励：15.00，Epislon: 0.010\n", - "回合：120/200，奖励：45.00，Epislon: 0.010\n", - "回合：130/200，奖励：73.00，Epislon: 0.010\n", - "回合：140/200，奖励：180.00，Epislon: 0.010\n", - "回合：150/200，奖励：163.00，Epislon: 0.010\n", - "回合：160/200，奖励：191.00，Epislon: 0.010\n", - "回合：170/200，奖励：200.00，Epislon: 0.010\n", - "回合：180/200，奖励：200.00，Epislon: 0.010\n", - "回合：190/200，奖励：200.00，Epislon: 0.010\n", - "回合：200/200，奖励：200.00，Epislon: 0.010\n", - "完成训练！\n", - "开始测试！\n", - "环境：CartPole-v0，算法：DQN，设备：cpu\n", - "回合：1/20，奖励：200.00\n", - "回合：2/20，奖励：200.00\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/Users/jj/opt/anaconda3/envs/easyrl/lib/python3.7/site-packages/seaborn/rcmod.py:400: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.\n", - " if LooseVersion(mpl.__version__) >= \"3.0\":\n", - "/Users/jj/opt/anaconda3/envs/easyrl/lib/python3.7/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.\n", - " other = LooseVersion(other)\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "回合：3/20，奖励：200.00\n", - "回合：4/20，奖励：200.00\n", - "回合：5/20，奖励：200.00\n", - "回合：6/20，奖励：200.00\n", - "回合：7/20，奖励：200.00\n", - "回合：8/20，奖励：193.00\n", - "回合：9/20，奖励：200.00\n", - "回合：10/20，奖励：200.00\n", - "回合：11/20，奖励：200.00\n", - "回合：12/20，奖励：200.00\n", - "回合：13/20，奖励：200.00\n", - "回合：14/20，奖励：194.00\n", - "回合：15/20，奖励：200.00\n", - "回合：16/20，奖励：200.00\n", - "回合：17/20，奖励：200.00\n", - "回合：18/20，奖励：200.00\n", - "回合：19/20，奖励：199.00\n", - "回合：20/20，奖励：200.00\n", - "完成测试！\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAEXCAYAAABI/TQXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAACBiElEQVR4nO2dd7gkVZn/P1XV8eY7996JTGSYM6SBIUeJBhQVf5hlUVcUXFx1FVfXLKY1oK55F0VcERVhRaJEkZyZGWYGDgxMjjfHThV+f1To6r7dN6fpOZ/nmWduV1WfOhX6rbe+5z3vqzmOg0KhUCgqE326O6BQKBSKyUMZeYVCoahglJFXKBSKCkYZeYVCoahglJFXKBSKCkYZeYVCoahgItPdgUpDCHE38F4pZdsovnMc8Dkp5duH2W4NcKaUsmtcnVSMCyHE64CrgX3Aa6SUqdA6B1gPWIDmLf6dlPL7oW2agSuBc4A0YAPXAz+UUpqhdr4ppfxi6HtvBz4mpTyzRJ++AFwK3Cel/OA4ju1Q4BvAIYADdAFfkFI+PMp2lgLfl1Je6H0OnxcHiAK/l1J+e5h2rgXWh8/feBBCvAn4NhAH1gEfklL2TETbMxXlyU88rx3tF6SUTw9n4L3tjlYGfkbwbuBqKeXxYQMf4izvWh2Fa8jfLoT4NIAQog54FNgGHO5tczZwPPD7onY+JYR4zQj79CFc52I8Bl4A9wH/I6Vc5fXtSuA2IcTho2xuMSCKlvnnZTVwCvBhIcT5Y+3vaBFCtAC/AS6UUgrgVeA/p2r/04Xy5CcQIcRvvD//LoR4I/AQ8ASwCvg8kPP+jwGzgd9KKb8khDgT+KmU8gjPc+kBjgQWAi8C75ZS9nneUAtwPvA2XA/wECALXCylXC+EWA5cA8wCduN6k9dJKa8t6utc4JfASq+dX0opfyyEeMDry43edsFnIUQG+CtwFPBrXC/2fG+7lbgGYhGwAvgvoAkwgB9LKa8pcb4OB37qbecAV0kp/9c7H9/E/REeget1XS6l/HuJNv4Z+DSuh9gGvB84GPgesBNYBqSAD0gpXyj2DMt5ikKIKPADXCNt4V7HfwMuAy4AUkKIeinlZ4r7FEZK2SaE+BRwkxDiB7je9otSyv8MbdMphLgY2CqEOEFK+aS36gvAdUKIo6SUneX2IYT4E3AQ8GshxJeBR4BfAEtwr/9vpZTfE0Iswb0nX/DWnSGl3B1q6nPAb6SUd4X6dp8Q4j3eOUQI8Xnv+BNANXCFlPIvQoivAicD84ANuA+tBUKIu6SUry9xXnqEEE/j3n+3CSE+Anwc91zvxX1jeanoOA9lZPfV9cCzoWt8GXAWcAvwlJTyZW/TXwBrhRCXSykrdlao8uQnkJAXdZaUcrv393op5aHAzbjG6P1SyuOAk4D/8F7dizkWeANwKDAfeEeJbc4A/lVKeQTuj9o3Nr8D/uAt/zjuD68UPwdeklKu9Lb5iPeAGIoYcKvnBf0COM17WAB8ENdL0oAbceWnY71+XiGEOCnckBAigvuj+4mUchVwHvAtIYTf3xNxjf5q3AfKV4s7I4Q4CvgO8AavjVtwDSPAMd73V3n9+t0wx1bMF3HP/VHePx34npTye95+fjicgQ+xFpgLNAOnAf8o3kBKmcY1wKeGFv/OW/Y/QzUupXwXsAt4n5TyT7hvBH+XUh7ptXeREOLd3uYHAV+XUq4oMvAAx+HeS8Xt3ymlfFUIsRg4F/fhsAr3XF8Z2nQxcIyU8j3AJcArpQw8BG8NZwD/EEKcDfw77u/mKFzp6mYhhBbaPsII7iuPq3Ef9j4f9JYtBLaHlu8A6oDaUn2sFJSRn3weAvA8hTcDxwohvoLrJWq43lAxf5NSZqSUOeB5XK+8mGeklDu8v58FZgkhGoETgF95+3wB17suxbl4xkNK2S2lPEJKuWkUx9OL+6O7SAhhABfhGuMVuJ70Nd4Ywj+AJLC6qJ0VQEJK+X9ee7uAm3AfbgBbpZRrwsdXoi/nAHf5D1Qp5Y+klJd569ZKKR/y/r4GWC2EaBrB8fmch/t2k5NS2sBPvGVjwfcSS0k7xRhFnz+Ke89cMpIdCSGqcQ37z8C9tsC15PtuAo+V+brNEDZBSrkV13i+Twjxn7hvNTWhTR73xxTK8HchxBohxPPAb3HfAp7CveZ/klK2evu5FliA+7bhM9L7CuABICGEOE4IcRju2+99QxybNUSf93uUXDP59EHw43sO+AuuobwG97VXK/GdsDFwRrGNf7OGty93A5vkjQ9CiGW4ckfx/mJF3+sL/f0r3AfFC8BGKeVmIcSRQJeU8uhQ23OA7qJ2Sv3gdNwBORjZOSg+hiSuN+mv89HIn5/hjq9c/8J9Gy3HA5s9ye0R4EzgKq/Ps4B+b7tTgB+Gv+jJGu8D7gS+O4J96Qw+V+G+Z4YwxI/jvmHeFl7oSUCv4F7nv3p9vBvX0P4itGn43ijFWWUCEkrdCxqF59ugzH0lhLgSeIu3+BYp5ZeFEL8GLgYywK+llI4QYhvuG6LPAqBTStlPBaM8+YnHorQxOAT31fCLUspbcV834wz23MaMFyXwCO7rqR/hcA4hQxji3tB29bieziFAK+5rO0KIg3HHE8rt73HcH+OXcV+HASSQFkJc5LWxEDeq4tjirwNZIcT/87abD1wI3DOKQ/47cK4QYp73+VLyhvBoIYTf948Aj3iD1uHjawZOL9P2XcBlQoioEEIHLh9l3/D2MR9XUvI1/18CK4QQn/PegM7BfVu7FdcTLiWXPIb7UPjycPvz3rAe9/rrX9uLR9j37+EOhr4u1P83AJ/AlZxeAzwtpfwBroG/gPL3r8nIH4p3Ae/yBkYRQnwQaAfCb5Zl7ysp5Ze9Ad2jpZT+OboW1/C/A1euA/fBdJIQ4hDv82W4D62KRhn5ief/gIeFEEcULV+H6yG9KIR4FvcG3AgMp4OPlouBdwoh1uK+sm8GBkps9zHgUCHEOtwHw7ellM/ghs+9TgixHtc4PTjM/q7GHdy8GUBKmQXeClzitX038KVi4+VJURcAn/C2uxe4stTgajmklM/jjkX8zTveN+D+cAH2AN/0pIELgH/ylv8EmCeEkLja9QNlmv+G18YaXA82imvsRoIvSzyDq9//r5Ty516fe3A99kW41/8ruI5BF9AihDi1dJN8E3fwdyS8DzjHO/YncWWwa4f7kifXnY+rda8TQmwAPgu8WUq5HvgD0CyE2Ag8g+u5zxJClNK0NwCWEOLJsLZeZr/34L4d3O/t8/3A+Z5M5m8zovsqtP0eXJlvnScFIqXch+vY3CiEeAE3uOHTw52X/R1NpRquLIQbL32TlPJFz4tbB5wnpdw4zV2bMsLRStPdl9HgSWYNUspnp7svispBafKVx0vAn4QQNu71/c8DycDvz0gpX53uPigqD+XJKxQKRQWjNHmFQqGoYJSRVygUigpmJmnycdx44t1U+OQEhUKhmEAM3HQST+HOCyhgJhn54/FmUyoUCoVi1JwODMoWOpOM/G6Azs5+bHv0g8FNTTW0tw834W56mKl9U/0aHapfo2em9q2S+qXrGo2N1eDZ0GJmkpG3AGzbGZOR9787U5mpfVP9Gh2qX6NnpvatAvtVUuZWA68KhUJRwSgjr1AoFBXMTJJryuI4Dp2drWSzaUrn2oJ9+3Rs2y65brqZqX3r7IyTSNSSTJbKdqxQKCqBERl5L//5O72Pt0sp/10IcS5uTvQkbi7oL3rbHo2btKoeN7nVZcPkmB6Wvr5uNE1jzpyD0LTSLx+RiI5pzjxDCjOzb47jYNs52tv3AShDr1BUKMPKNZ4xfx1ucv6jcQsYvAc3H/pbcasXHS+E8IsSXIdbsWgFbhraD4+3k6lUH7W1DWUNvGL0aJpGPJ6goaGFvr6u6e6OQqGYJEbiye8GPu2l+sRL0bkCeFlKudlbdh3wDi8FadLLMw5uetOvUVhYYNTYtoVh7BfK0n5HNBrDssb1oqVQTDiO4wTCrK5pg5aNF9t2sIfJ26XhOkPAsNtOBP5xTjTDWk4p5Qb/by/Z/ruAH1MYk7kbt3bk/DLLx402SSfgQEedV0WYl7Z38Zs7XuArHzyeRGz6HKvf/u1FHly7m1hE54vvP46Gmjj/8d+P0Z+eOodELGzgs+87hrue3Maf7h9JZcyxoWOTNBw+8d4TaGmZ+HKzI76KQojDgduBK4AcIIo2sSldom1UYnRTU82gZfv26UQiw0s1I9lmupiovl1wwZv4+c+vZv78+RPSXiSio+v6pNxc42Gm9cen0vv1zKZ29namcAxjwtocSzsvbOuiqT5Be3eatAVEDPrTJq85egEHzZn8a7BpexdPbtxDJBFlzSvtzGuu5qxjF46qDd3KEs92Es32EMv2EMt1u/9ne4iafUTMAfeflcZBozG5DDhowu+xkQ68nopbXeaTUso/CiHOwK0+7zMPt1r8zjLLR0x7e9+gyQC2bQ87cDkTBzd9JrpvljX8+RgJfr9s26a1tXcCejYxtLTUzqj++BwI/erpcUvrbt/VTdIY/1teqb79/Ob1HLlsFqevKu2opDImrZ0pTj58Lo9176Gtow/Ncuf5HHNIM6sOHk099pH3K8zSOdU8uXEPf39iKy9v6+K8kxZx7urS/XXMLHb7Nqy2Ldhdu7G79mB37cbp7xi0rZaoRatuRKtrRIsvREvUoMWr0aoaMOvch8hor6WuayWdY59hjbxXS/Fm4F1Syvu9xU+4q8Ry3PJy7wWukVJuFUKkhRCnemW5LsYtQFxRPPvs0/ziFz/GsmzmzZtHMlnFq6++gm3bvO99F3P22a/lrW99AzfccDNVVdV8+MMf5NRTT+eiiz7AvffexZo1z/HRj36Mb3/767S27qOtrZWjj17NF794Jc8990zQ9rJlB/Pxj3+KK6/8Evv27WXJkmVks1kANm16me9+95tYlkUsFuPzn/8KCxcumuYzo9jf8f2r3oHcpO1j4+YO4hG9rJHf0+FWq1wyr5bHNuwhm7PJ5lwjH49Ozdv64jm1xGMGdz6xDdtxEIsagnX2QBfWjg1Yezdhtb6K3b4DHG+yaTSB3jAPY55Ab5iHXteCVj0LvboRraoBLVKubvzkMRJP/gogAfxAiECh+SXwAVzvPgHcAdzorXsfcLVX9/E5XP1+wnjk+d08vG5wigZNg/GOjZy2ah6nHjlv+A2B7du3ceONt/G73/2G5uYWvvjFr9Hf38dll/0zhx12BMceexzPPfcsq1cfy549u1iz5lkuuugDPP74o5xzzmt59NGHOeSQFXzjG98hl8tx0UXvQMoXC9quqanhBz/4DitWrOT73/8xa9Y8y/33u/WYb7jhet797os4++xzue++u9mw4Xll5BXjxi8i1J+ePCNvOc6Q2vrO1n4Als6tAyBn2mS9N9doZMLq3g9JxNA5ZEE96zd3oGuwLN5J5on7MLevw+7Y4W4UTWC0LCV21Hnos5diNC91vfQZNs41koHXT1C+gPFRJbZfC5wwzn7NeBYuXExNTQ1PP/0kmUya22+/BYB0Os3mza9y8smn8cwzT6LrGq9//Ru55567ME2TtWvX8JnPfJ54PM7Gjeu54Ybr2bJlM93d3aRSAwVtAzz33DN89avfAuDoo49h/vwFAJx88qn84Aff5YknHuWUU07nzDPPmYazoKg0fEepLzWJRt5yGMiUN/K72vuJGDrzm925G1nTIucZ+dgUjLs52RS5TY/xJmMDh1b1cmRyD9ZtPVi6gTF3BbET3kHkoCPQmxbuF2Hd+11c4qlHlva2p1qTj8fjgBve+aUvfR0hVgLQ0dFOXV09vb29/PGPv8cwIpxwwgls2bKZ2267mWXLlhGPx7nxxj/ywAP385a3vI23v/0ENm9+JfCi/LbBjX4Jz5Y1DNeTOeusczniiFU88shD/PnPf+Dxxx/hs5/94lQdvqJC8UMFJ9PI27bDwBBvCrva+pk7q4pEzL3XszmbjCfXxCZRrrG6dpFbfy+5lx+FXJq50SpmxbP0JReROO7tRJYeixbf/yYNzvzH0AznmGOO5+abXaWqra2N97//Pezdu4fGxkbi8TiPPPIgRx21mmOOOZ5rr/01p5xyOgBPPfUEb3nL/+N1rzsP0Hj55ZdKpj447rgTuPtud1jjhRc2sHOn+6r45S//Bxs3buCCCy7kkksuC6QehWI8TLYn7zhufPqQnnxbP/Obq9B1jYihFXjykyHX2N17SN3/3wz8+Qvk5INElh5L1QVfpubin/KA+BzV53+G6MrX7JcGHvZDT36m8c///GGuuuo7/NM/vRPbtvmXf/k4Cxa4UwNOPvlUHn30Yaqqqjj22OP58Y+v4pRTTgPgne98L9///rf54x9/R1VVNUccsYrdu3cF3/X50Icu5Zvf/BoXXfROFi9eHMg1//RPH+Q73/kGv/3trzAMg3/913+b2gNXVCSBJj9JRt6yfc2/tJHPZC3autOctsp9W49GDHK5vCY/kZ68k+kn8+SN5F78B+gRoke+wdXXk3XBNu84c/mE7W+6UEZ+DBxzzHEcc8xxAFRX1/DlL3+95HYf+MAlfOADlwBw8MHLefjhp4N1xx57PH/4w/+Vbd+nurqGb33reyW3+9Wv/ndM/VcoyjHZco0fHp3JWpiWzf3P7GD2rCqOXt4MwO4Od9B1gafHxyI6WdMmZ1rB5/HiOA59Gx+h/2+/wkn3ET3sbGKr34xeVT/utmciysgrFAcgW/f0svaVNt5y6tKC5fYkyzVWaA5MKmNyx+NbEYsaAyPf1eeGCDfWJgDXc8+aFpmcjYYb9TIenEw/6X/8mr4tz6I3LyF53qcxmhePq82ZjjLyCsUByDMv7eO2R7fy5lOWFIT8TZVcA+6DpHcgF+jtQD4e3ht0jXlyTc60iEb1cYUnWu3bSN39E5y+DmadczHZpWeg6VMTkjmdKCOvUByA+GP8juPOMfHxB1770yaO40x4zHfYyO/tSOHghkj6ZLKFk56inlyTNW1i4xh0zb38KOkHf4MWr6bqLf9BwxGrZ+Ts5clAGXmF4gDE194t20HX84bc18wt2yGVsahKTKyJCKcs2d3u6u/ZkCefCWa2ep581CCbs8jmrDEPumbX30Pm0d9jzFtJ4pyPVqz2Xg5l5BWKAxBflilOoRtO5tuXzk24kbdCYcK7PCOfyw1h5CM6/WmTnGmPKXwys+Z2sk/+mciSY0mccxmaER1P9/dLVJy8QnEA4tva4mSAYZs/Gbp8WK7Z0+7O8C6Qa3IWmubKNOD+nzMtsjl71JE1mWdudg38wSeROPejB6SBB2XkFYoDEruMJ28XDYwOxdY9vbywtXN0+y2Qa1wjHx54zWRt4lEjGAtw5RqbrGmNyshn199D9pmbiaw4ncRZH0HTD1zRQhn5GcquXTv59revBNyslx/72EfG3NYdd9zKN7/51QnqmaIScEKafOHy/N/DGflbHtnM7+95aVT7taz8DvxZr8WavC/VgB8nb7kDr9GRyTW5Lc+SefR6IkuOIfGaD6LpB7aZO7CPfgazZ8/uIIWBQjHR+LbdKa7d4Izck8+aNr0D2VHtt/ihAgQTncANoSw08gY50yaXswMJZ8j2O3eS/vv/oLcsIXH2ZQe8gYf9cOA199Ij5OSDg5ZrmhZ4J2MlKl5DdMWpQ26zb99errzyS6RSKXRd4xOf+Axf/ernOftsN32wYRhceunl/PGP17Fjx3Yuv/yTvP71r6ejo53//M+vs3fvHgzD4CMfuZyTTjqFdDrNd77zDTZtegld13n3uy/ivPPO57/+6/vs2rWTq676DmeddQ5dXV1cccXH2blzB4sWLebrX/8OsViMO++8jT//+Q/YtoMQK/nUpz5LPB7nb3+7nd/+9tdUV9cwd+5cksmqcZ0bRWURjqIJ4zjuoGc2Zw2ryZumTV8qh+04QX3SNZvaEAsbSMZLm5ZSRj5bNPAa9tijUT+Ecni5xsmlSd39E7RIjORr/3VacrfPRNRjbpTcdttfOeWU0/j1r3/HRz/6cdatWwNAc3ML1113A0Ks5LrrruUHP/gpX/rSlVx33W8A+OEPv8cxxxzHb3/7R77+9e/w7W9fSUdHO9dc89/U19fzu9/dwH/91y+55pqr2bTpZT7xiSsQ4lA+/enPArB37x4+9anP8vvf30hHRztPP/0kr776CrfeejO/+MU1XHvt9TQ2zuIPf/gdbW2t/OIXP+ZnP7uaX/7yGgYGBqbrdClmKIEmX8KTN3SNqkRkWE/etG0cBwa8PDR7Owf48Y3reHzDnvL7LWHkLdsJlqezVpB9Ely5JmfaXgjl0HJN+pHf43TvdcMka2YNue2BxH7nyUdXnFrS256qVMPHHXcCX/jCv/PSS5JTTjmNCy98J//3fzdw0kmnADBnzlyam1uIRCLMnTuP3l53wsWzzz4VpAJesOAgDjvsCDZuXM8zzzzN5z73JQAaGho4/fTX8Nxzz3DwwYWJkZYvPyRITrZ48VK6u7vYvXsnO3Zs59JLPwiAaeZYsWIlzz+/liOOWMWsWW6ZtNe97jyeeeapST83iqnBsm1e3t7NysWNY26jbAil48bNj8jIm+53ewey1CSjbNvbB+RTE5TrO0DE0DBD+nzWtEjEImRzRUbeM+z9aXNITz736pOYLz1EbPWbicw/dMh+H2jsd0Z+ulm16miuu+4GHn30Ye67727uuONWAKLRfHiWn/M9zGAPxsGyLByn8MHkOGBZgzP0hdv0pSnLsjn77HP55Cc/A8DAwACWZfHMM08W7K9UfxT7L8+/0sGPb1rHf156ErMbxybD+SGUpeQaTYPqRCTw0MthWm4j/sNg217Xoekd4uHg76+2KkZnb4b66hjd/Vmypk0i5so1ddV5mcXX4dNZq2ycvJ3uJfPw79BblhI79oIh+3wgMmK5RghRJ4RYL4RYIoR4oxBiTehfqxDiNm+7LwshtobWXT553Z96fv7z/+Kuu+7gvPPO59/+7bO89JIc0feOPfY4brvtZgB27tzB88+v5fDDV3HMMcdz++1/BaCrq4uHHnqA1auPwzAiWJZVtj2A1auP5cEHH6CzswPHcbjqqm9zww3Xs2rV0Wzc+DytrfuwbTsoGaioDNJZLyolN/Y317wnP3i5pmlUJ6JDlugDyPlG3qsHu32f68kPNRjrOx+1Sdcpam5wE5H5E6IyOSvIWwOFWSfLzXjNPHo9TnaAxBkfOiBy0YyWEXnyQogTgauBFQBSyjtw67oihJgLPAL4Cc2PB94tpXxswns7A7jwwnfxta99kTvuuA1d1/n0pz/HL34xfBnbT37yM3z3u9/kjjtuRdM0PvvZL9Lc3MwHP3gJV131HS6++F3Yts3FF/8zQqyku7uLvr5evv71L/GmN721ZJuHHLKCD37ww3z845fhOA6HHCK46KIPEI/H+eQnP8MnP/kvJBJJlixZWvL7iv2TcjHuY2qjhCavaVCViLCvKzVkG74n73vueSNf3pM3A0/eNfIt9Ule2dkTTIjK5OzC6JpoaYMftLdzI+amx4gd81aMWQcNWq8YuVzzYeBy4Hcl1n0P+KWU8mXv83HAZ4UQy4AHgSuklOlx93SGMGfOXH7+818VLDvnnNcGf3/oQ5cGf8+bN58bb3TlnObmFr773R8Naq9cPvr6+gZ+97sbgs/hHPNf+MJXg7/f/OYLePObLxj0/bPOOpezzjp32ONR7H/4ksd4gsl82z7YyIPuefLDyzX53PO9A1k6ezPACD35KleSaar3PHlvPC2THRwn71Ms1zi2RebR69Fqm4kd/aYh+3ogMyIjL6W8BEAIUbBcCHEIcCbgr68BngOuALYA1wJfAr4w0g41NdUMWrZvn05kBDGyI9lmupipfYtEdHRdp6Wldrq7UsBM64/PTOhXdbVrGOsbkkF/RtuvqGdI6+qTBd+NxyNEIjrNs6oYyJg0N9cUZKL09feaZDR42Fho9GZcIz2vuZr+VK6gzfDf1bt6AFh6UANPy1ZWLGni9se2Ul2ToLm5hqxp0Rg6rua2fGTYrMaqgra6n76Tvs4dzLnw36me1zSq4y/u10xiovs13oHXjwA/l1JmAKSUfcAb/ZVCiKuAaxiFkW9v7xvsXdj2sJEzU13IezTM1L75/bJte0alXW1pqZ1R/fGZKf3q6XFllI6OAVoTkTH1K+0V0m7v6Kc1mTcDAwM5HNtB88Iat+/sKoh5//GN6wD4+NtXkfOSie1r72OdN/F15cIGHnhuJ3v39qDr2qC+dXa5RvuwRQ3856Un0epJQvtae6mJ6W7gQc4KvpPqzwTfzaSzwXInl6H/wRsw5gn6Zx3KwCiPf6Zcy2LG0i9d10o6x8H6cfbpAuCP/gchxCIhxD+H1mvAhGQ5Gu9EJ0Vp1Hnd/8jLNWO/dk4ZucbBCaJrYHCSsr2dA3R7hjc88Lp9Xx8NNTHmNVXh4GawLIW/v6ihMasuEUgwWdMelIESCjX58PLs+rtxUj3ET3jHhOe8rzTG7MkLIZqBpJRyc2hxCviuEOLvuHLN5cBfxtVDQNcNLMskEjkws8hNJrlcFsPI3wY500bXwVDTwWcskznw6niafFXC/a31p02aQ+t7+rMYehzLmwgFroTT3pNm0ZzaQGvv7c9SVzV4xqmfu8bPYe9r7jnTJusVDAlH0RRq8u7fTqaf7No7MBYdjTFn/y+0PdmM55e8DChIriKlbAUuBW4FJK4nf9U49gFAMllDb2/XoJhyxdhxHIdMJk1XVys1NQ3B8u/94Tn+78FXp69jimGxJ2Lg1S4/GUrTNGo8CWcg5JGblh3kdg9PZOrsy7C7fYCFs2uCqJlyETaWtz/fiYh6Bt2t4zrYk48WGHzP6994P2RTxI9722gP+4BkVJ68lHJJ6O8ngZNKbHMTcNO4exaipqaezs5W9u7dAZS+s3Vdx7Zn5kNgpvYtHo9TW9tIMlkdLOvoTdPZm5jGXimGwzfM45FryoZQ2n4IZd6T9/EHXbOmHYRPakBHjyvfuEbe8+TLTIjyPXkj8OTzck3aM/KFaQ3C0o2OY2bIPX83xsIjK74A90SxX8x41TSNWbNmD7nNTB1IgZnbt1L9smxnXB6iYvLJe+Fjb8O/xqVmvOq6ltfkQ558T78bGpkz84EQtdWxYPnC2TXBIG25MEq/74bhGnlfgsnl8nJNoSZf6Mnn5EM46V5iR58/2kM+YFHCq6IAJ5QsSjEzKSe1TEQbtuOgoQVl/8Kx8j2e4c5ZdjDo2lgTB1xjPKexiprkMHKNt18/a6Wvubtyje21VSZO3oDs8/egzz4YY+6KUR/zgYoy8ooCXE9eGfmZTJALftIGXl1v2tC1Armmt9813GZIk2+sdY38wpYadF0jYuhUJyKBJ//AM9v56jVPhoqUuIbcl2uioYHXUpp8xNDxY2cSbS/i9OwldsRrVUTNKFBGXlGA7YxvQE8x+UzEjNcghLKoDdsbeHXz10QK5JpuT5axbIeMJ60ERn52Pk67pioWePJb9/SybV9f8FAolms0TSMa0cuGUGqaFgy+Rl75B1qynsjS/OxvxfAoI68owLadcckAisnHmcCBV8senAXVj56tKkpSFtbZ/SRps+o8Iz8nP0uztioabOsP0PoGvFiuAa/EX84KHhzhBGXueoMmvRdt13qih56JZuwXQ4kzBmXkFQXYjhp4nen4hnI8AVuBJj9IrnECKaQ6GSkIofQHWCFfn3XRnFoufoPg5MPnBOtqk9HAk/eNfDZk5A1dK5BbhvLk/fUnxjeBphFdecbYD/oARRl5RQHKk5/55OPkJ0KTH7zct7/ViSj9qfDAa97gpzwjHzV0zjx6AYlY3ruurYoFnrwfMpnO5o28PxHKx6/jmslZ6JpGxChcH49qnBB/BeOgI1XFpzGgjLyiAEsZ+RnPRIZQDp4MlZdSqoo0+Z6QXJPKuEa7VOK9RMwg44VYFss1tufJh4lFPbkmZxGP6YGX7zgO5p6XWK2/RKM+QFScPubjPZBR4pYiID/JZpo7ohiSyZwMVSDXxAvTDff0Z4nHDDJZK5BrosZgI2/oWuDB+9JSsVwTJup58sV1XK2dG0jd8X3eAPQ7cWoWrx7z8R7IKE9eEVBOp1XMLOwyXvio2rALjXC4bd8GVycjpDImthdW2zuQpanOnQ3tyzXF0gqAYehYngfvT5oKD7wOlmtcTT5dlEs+t+kxiCW5L/5a/mS/QQ24jhFl5BUBE6H1KiYfP0XGxIRQlqoM5cs1URzcQdZUxsK0nBJGfrAJiRgaDu79ZNq+kbeDvg/y5KM6OdMim7NJeEbeMbOYm58lsuQ4tlQdwb7IvLEf7AGOejQqAiYi/lox+fiDpZMl1wSefCKfpMzfzK/kNJSR9424admBbOOHR1pWCU0+YtBpZsjkLGJe+KS5/XnIpYguP5EzD5o7ZLUpxdAoI68ImIgUtorJZ0LK/5UNoSTw5P1EYemsFUTHNHlx8b4mX2rg1Tf8puUE6Q8CucZxBqWxjkV0cjk3usbPfWO+8gRasg5j/qEcrYpzjwsl1ygCJiKFrWLycSbgYVyuDTcLZVHKAcsOPPe6ajfLZCrtD7yW0OQ9T92y7UCbD4y8NViTd+PkLdq60zTUxFypZttaIkuOQVMGftwoT14RoDT5/YNgMtS45JrCtnz83DUQ8shDqYWrvRTEKc+zN0pq8nlP3iySa2zbCVIa+MQiBj39OWzHYeHsWqxdG8HMEFly7JiPT5FHefKKgImI2lBMPhMR6jqSEMqwJ5/zomR8OSU1VAilMYQnbzsY2uCBV78/i2bXYG55FqIJjPkrx36AigBl5BUBfh4TFUE5s5mINy6nbKphghmveU/eCYx8lWfkBzImuqYNkl4AIp7mbllOybQGpUIofQ5qqcLcuobIwlVohir3ORGMWK4RQtQBjwLnSym3CCGuAU4H+r1Nvial/IsQ4lzgB0AS+JOU8osT3WnF5BCksFVWfkYzETNey8s1TjDjtcCT94y1n2c+k7UKCnqE8T150w7JNcGMV3uQXOPvp6kuTqJnK6lUD5Elx4z94BQFjMjICyFOBK4Gwpn6jwdeI6XcHdouCVwDnAFsB24XQpwnpbxz4rqsmCwmwngoJp+JyCcfZLIslbtmUK53K5jU5Bt5KC3VQL5+q2XZwdthOHdNsVzjl/hbOLsW85UnwIgSWXTUmI9NUchIPfkPA5cDvwMQQlQDi4CrhRCLgL8AXwNOAF6WUm72trsOeAegjPx+QBCaV6aOrmJmMBGTofKphssPvEZLhELGIjoRQ8O0nJKDrpCfBWtaDqY5OK1BcWy9ny9+UUsV5qtPEVl0FFosOfaDUxQwIiMvpbwEQAjhL5oD3A9cCvQBtwEf8v7eHfrqbuCg0XSoqalm+I3K0NJSO/xG08RM7Vu4X/3eD1LXtWnv73TvvxwzoV+G5/lWVceC/oy2X/4DIp6IFHxX0zUSiSgtLbXEkjFvmyhp796YN7eeWNTAtEziMaPkfps6UgDU1iWCGa827j2lGzqJeOE+mxrdQvKrGztxZA9Nx5xF9RSc55lwLUsx0f0aUwillPJV4G3+ZyHET4CLgT+X2HxUWa/b2/vGlDtlphbLhpnbt+J+tbX1AW5R5ens7/5yvqaLtBfZ0tuTprW1d0z98n9j/f3Zgu+apkU2a9La2htE0HR2pegdyBIxdNra+oh4rr4OJffb15sGoK29P4iu6Rtw95POmMQjesH36uIGzfVxGnc9AdEE/fWHMDDJ53mmXMtixtIvXdeGdI7HFF0jhDhSCHFhaJEG5ICdwNzQ8nnArrHsQzH1qBmv+wf56zQRbZRPNVw88Op/jnpvEqVmu0I+Ksey7RIDr4VpDaz27cxrfYyvLvgH2ran3cpPkdjYD0wxiLFOhtKAHwkh7seVaD4C/BZ4AhBCiOXAZuC9uAOxiv0ANRlq/8CZgLETP/9Nqegaf1zU0DXXezNtTNMOZrf6xr5U3hoIRdeEQihLZaF0bJvUHd/DSfWgJetInPEhIitOG/MxKUozVrlmnRDi28AjQBS4SUr5BwAhxAeAm4AEcAdw48R0VTHZqARl+wcTkrumzGQoO5S7RtM0IhEd05sMlffkfSM/OEYeQmkNLHvQjNdwPnl73ys4qR4SZ36YyCGnFJQEVEwcozLyUsolob9/Dvy8xDb3ASr+aT9kInKiKCafcgZ6pITf1EqlGg4X2Y4aOjnTlWsinkzjT14qF0KZl2ucghmvjuN+DrJUbn0ONIPI4qOVgZ9E1IxXRYDy5PcPxiurhQ17qSyU4Qmp0Yhn5E07MOojlWtypu2FTGo4jpt62A5loTS3rcGYL9Di1WM6DsXIUEZeEaAGXvcP8pOhxvh9O/x3+dw14BpyMxh4dZdHhjHyflqDbDCByk1PkMnZgSZvd+/F7txFZNHRYzsIxYhRRl4REHiIasrrjGa8WSjDbwCDyv/Z+YFXyHvyZtiTN4bR5L3lvg7v57vJZC23aIihkXv1KbcNVbd10lFGXhHge3jKxs9s8snFxvb94eWacp68ay78YtvDhVD6s1z9VAjpnOWGUAK5Fx/AmLcSva5lbAehGDHKyCsCLBVCuV8w3utUINcUNeFQKNcUaPLewGveky+Xu8bz5IuMfDZnYdkOczKbcXrbiB521pj6rxgdysgrAiZiko1i8hlvPvlCT75wQrptU1KuyZl2IM9Eh42ucbfL5goLjWSyrpFf3PMcWqJWFQWZIpSRVwSoyVD7B3kjPxEhlIPXhfO9Rw1tiDj5obNQBp68r8nnLHTbpGVgE5HlJ6MZqjDdVKCMvCLAHueAnmJqGO91Chv2QQOvTrEnb7gDr9bIJ0PpuoamDZZrMjmL+UYbumNhzBMlv6uYeJSRVwRMRFk5xeQz3oLr4cHWwblrCidDRSJ6UP4vania/DCevL+u2MgPZEyWRloBMOYcPLbOK0aNMvKKADXwun8w3vkMzhDRNbZTFEJpaKEZr0W5a8pE1ziZfuZFuoPompqYBjik0q6RT8ca0asaxtR3xehRRl4REMgAo0oOrZhq/OszEQOvpYuGFEXXWDam6QwacC038Jp+8Dd8LHkrTjbFmYmNHP3Ul/le4/U07XmMJZFWeqsXjq3jijGhRj4UAeMd0FNMDeOdDBW26+GJb/51L57xms1Z2I4TGPUgTr6EJm937cbc/AxxzWFZTnJifCPZmvls67QQ++5C1x021ywaU78VY0N58oqAfPk/ZehnMs44x07Cht1ywkbe/b84hDKVsby/h4+Tz669E4wInU4tZ9iP02gM0LP89fwxdw4ZzS3pl1JGfkpRRl4RUOjVTWNHFEMyWQnK/OXFco1PcZx8sZF30n3kXn6EqDidZzmMpJal206Sm3MEWrKWu+Ov47nMYtI1c1FMHcrIKwIsZ/APXjGzsJ18qZCxG3n3f00rNPJ5uSa/bdiQDwqhjBTKNeaujWBbRA85hRd0Qc4xeCS9gkg0SnUiysb0XK7tPwPDiI6p34qxoTR5RUCpH7xiZlHgeY9xgNx/Y4sYesHAq/9nOU++2MgXD7xaOzZANInespS00cHXOy6g10lykq5RnYywr3MAoKD8n2LyGbGRF0LUAY8C50sptwghPgJ8HFfCfRq4VEqZFUJ8GfgQ0Ol99Wop5c8muN+KScAu8YNXzCwm4kHsv6VFDL1kOGXxwKtPNCgaYgxa5zgO5s4NROavRNMNIoZGt+PmiTcMjZpElP60GXxWTB0jMvJCiBOBq4EV3ucVwGeAY4Fe4FrgcuCHwPHAu6WUj01CfxWTiKU8+RlPgZ4+1oFX73sRQyu65u7/xUVDfHxNfsm8Wt562lJWLmrMf7e3Fae3DWPVG4B8agP3e3qQv8ZtXxn5qWSknvyHcY3477zPGeCjUsoeACHE84A/ZH4c8FkhxDLgQeAKKWV64rqsmCzCRkPFys9MCjz5MRbyDnvyVon2tKLyf8HfoQHXt562tKBNc8d6d92Cw71t8m0YnlwT/qyYOkZk5KWUlwAIIfzPW4Gt3rIW4GPAB4QQNcBzwBXAFlwP/0vAFya224rJYCIMiGJyKXwQjzXVsG/k3dmsPuVCKIO/h0hjYO3YgFY9C63ejZwxQtu6Rj7vySu5ZmoZ18CrEGIBcCfwaynlA97iN4bWXwVcwyiMfFNTzZj709JSW3Zdd1+GX92yno/+v1VBObKpZKi+TSfhfiVC56WxsZr6mvh0dAnYP87XdBDtzQR/x2KRoD+j6VdbXw6AeCzCQMYKvuu3XVeXDJY17e0Lvje7pbbkfhwrx5ZdG6g5/DW0zK4DIBm6l2bPrmVeV/5lvrGxetrPI0z/tSzHRPdrzEZeCLES+BvwEynlVd6yRcC5UsprvM00IDeadtvb+8bkobS01NLa2lt2/ZqX23jgmR0cf0gzhy6ZNer2x8NwfZsuivvV1583IG1tfWRT2eno1n5zvqaDzpCRT6dztLb2jrpfHZ397h9ecW3/u919btv9/ZlgWSp0T/T2pGhtHWwyzJ0bcbJpzJZDg+/ZVv4NobtrADtnBp/7etPTfh5nwrUsxVj6pevakM7xmIy8EKIWuBv4vJTyutCqFPBdIcTfceWay4G/jGUfE03WdGft+SP8isGogdeZj1NihupoCcs1pSKqhouTL8bcvg70CMaCQ4NlYd3d0LWCgVelyU8tY/XkLwHmAFcIIa7wlt0ipfyyEOJS4FYgBjwMXDX+bo4fX3scyCgjX46JiNxQTC5WiRmqoyUYeI3oJUMyh4uTH9SnbWsx5gm0aCJYVjjwqhdq8srITymjMvJSyiXenz/0/pXa5ibgpvF1a+LxjXx/elTq0QGFmgw187EnwJMPQij1Yk9+uBmvxuD+9OzD7tpN/NAzC5YXDLwaGtWJvKnRlZGfUg6YtAZZ38inlCdfjqGKSShmBhNxjQomQ4U+5+Pky3jyJaJizO3r3LYWHVWwPOIZcl3X0DUtKBwCypOfag4YI5/zNPkB5cmXRc14nfnYJSYvjbUN30svTnhWNoSyhFxjbluHVjcHvb4w6ZjvyfvG3tB1kl6tV2Xkp5YDyMj7co3y5MtRKAUoKz8TKYiTH3NlKPd/XzfP14x1l5dLazAo66SZwdr1ApFFqwbtwzfkYdmmxpsQpeSaqeWAMfJZpckPS8GgnnLlZyQTnbsm/HmogdeIoRcYfwBr14tg5YgsHGzkIyVyzvsRNsYQk6oUE88Bc7aVJz88EyEFKCaXiRh49a+zUdaTz28blPqLlNDjt60DI4YxTwxa578lhKNs/AgbQ+WumVIOICOvNPnhKCgLp6z8jGQi5jL4D4po4MkXthf25IPi3SW8b2vfJoy5y9EisUHrSsk1foSNSmswtVSUkd+0o7tgRmAYX64ZUJ58WdTA68zHngBJLa/Juz//oGZsiVTDhq5j6NqgQVfHMrE7dmA0Lym5D7/t8MPB9+SVJj+1VJSR/+lfnueWRzaXXJcLGXkVHlgaNfA68wkMMTDWRKGDNPkgusZdX2yDI4ZOpChG3u7cCbaF3ry45D7ynnxIrvE1eWXkp5SKMvLZnMXutv6S63wj7wApNeu1JKVyiytmFr6BNooKfoyqjTKafKlUw+AOvhbHyFttW9w2yhn5EgOvB7VUU1cVJR4dPKlKMXlUVPk/23bY05kquS6bs4K/+9NmQS4NhYuaDDXzyXvh2vhnvPpyjePLNe7y4nHRaEQfJNfYbdsgmkCrm11yH6UGXo9fOZvjV84e9BBRTC4V5clbtkNPf7ak7p4LZcXrT6nB11IoIz/zCbzwopQEo2oj9KCAfM3XYOC1SE6JGNrgeq7tWzGaFqFppU2IXxkqPPCqaZoy8NNARRl5/+bd6xUMDpPL2cHovhp8LY0V1uRVZagZie9tu/VZx9hGkSZvFWnygz15o8CTd2wbu21bWT3ebdv35CvKxOyXVMwVsG0nuEn3dgw28lnTDopgqAlRpXFsJ/iBK09+ZuIbZFeuGWN0TXFaA1+ucUpr8o01MRpq8wVk7LYtYGXL6vGQ9+DVIOv0UzGafNgo7Slh5HOmzZxZSXa19asJUWWwbIeIoZMzbRVdM0Pxr4uh62MOc7UDTb5o4LVEnDzAv7ztyGCZY9ukH70OLV6DUZSULIyfsyZSJj2xYuqomCsQjgzZW2LwNWdaNHqevJoQVRrbcfI/fGXjZyRWKDJmotIaBHHyZUIok/EI8ZgbEZNbfw/2vleJn3oReqJ8mbp8grKKMTH7LRVzBazQwGpJT96yqUpEiRi68uTLYNtOMGCmPPmZSdhAjzlBWXEIZVHumqEGR3MvPYQxdwWRg08cch9+22p26/RTMUY+74Vo7O0YGGSksjmbaESnOhlRnnwZLNsJfpTKxs9M7AJNfoxteN+LFk2GKlU0JIxjZrE7d7lVoIaJkvHlmlLpEBRTy4g0eSFEHfAocL6UcosQ4lzgB0AS+JOU8ovedkcDVwP1wIPAZVLKKXGbfU++uSHBvs4UvakcdVVuTg3bdrBsh1hEpzoRHXfhENt2QBusXe7v2E7+9VoNvM5M8iGUOo4ztvs40PXLzngtfV/bHTvAsdHLpDIIEwy8Kk9+2hn2MSuEOBG3VusK73MSuAZ4K3AocLwQ4jxv8+uAf5VSrsCdef3hyeh0KfwbtcorTGCaefnGn+0ajehUJSLjrvP6ld88yZ2Pbx1XGzMR27aDgTIl18xMwjHu468MVXrgtZyXPtws1zAqhHLmMJIr8GHgcmCX9/kE4GUp5WbPS78OeIcQYjGQlFI+7m13LfCOCe5vWYLQskjhYBJA1stA6U7P1jGtsQeB247DrrZ+drSWTp+wP2Pb4R/+NHdGUZJ8SoJxxMn7v5XgrY2C/8u9oNptWyFejVbTNOw+IiUmQymmh2GvgJTyEinlQ6FF84Hdoc+7gYOGWD4l+Ddusc4IeU8+FjXQRzlT8OF1u/nSr54IPqczJo4DPf3Ziej2jMJ2nODHqTz5mYl/6xr6OOLk/RDKIofIH5AtJ9dYbVsxmhePaNaqUSKtgWJ6GEucfKmrZg+xfFQ0NdWMukNAkJis2tPh6xuqaGlxQ7xyXtdmNVaRTETJ5Kxg3XB0p7axs62fhsZqohGdPe3ufgYy5ojbAEa17VQS7pcDJDy5q7YuMa193h/O13SQ9O7v6qoYmqYF/RlNvxJJt43mpmoAamvda127pxeApqbqQe05Vo7ezh3UH/8mmkawL0vPJyib7nNWjgOlX2Mx8juBcNXeebhSTrnlo6K9vW9MOTksv/CBpzO0tfWR8N5T9uzrAyA9kMXMWWSyFq2tvSNqt8fLT79jVxc1ySjbdvcA0NGTHnEbLS21I952Kinul2nawfnr6kpNW5/3l/M1HfT0pAEwcxY5y6a1tXfU/errT6Np0NPtzifp7BqgtbWXri73c1fnAK3Rolw1bVvBMslUzRvRvrq9fhq6Nu3nrBQz4VqWYiz90nVtSOd4LILZE4AQQiwXQhjAe4E7pZRbgbQQ4lRvu4uBO8fQ/pjwXzmjJTT5vFyjYxijk2ssz+ilvcFaP7lZ30Cu4uqg2raTj7hQcs2MxAkNmo5dk3clGV+Wyacadimlxlitbp0Go2XpiPYRTIZSM16nnVFfASllGvgAcBOwEXgRuNFb/T7gh0KIF4Bq4McT083hyccPDzZSfum/qOFWubFGYZxN7xUh7aUq7vNi7B2gr8KyWbozXpWRn8lYBQOvY9XkHXRdC7JNBrlrfE2+RL4Za+/LaInasqmFi1HRNTOHEcs1Usolob/vAwYlrpBSrsWNvplyBnnyVv4H4Jf+i45h4DXw5LOukQ/H2Pf0Z6mrHlzfcn/FtvNpDZSNn5mEZ7yOJwulpuWNuTWCEEpr7yaMOctHnCo4EtLkFdNLxVyB4ugayy4RJ2/oGJpWsG44/B9AOuvJNaHZsj0DlRVhYzt5uUZF18xMghBKbRz55H25Ri+Ok3fXF9txO9WD070Xfc4hI95HPGZw8esFr1m9YEx9VEwcFWPkfc/d9+TtEnHysaiOPka5JlPKk68wI2+FPHkVJz8z8b1wTc+X6xstjuOgaxpGkSbvvyUUh1Da+14BwJhz8Kj2c+bqBcxurBpTHxUTR+UYeTvvrUNhAYxcLj/jdbQVdfx0CYFck86R8DLy9fRXjibvOG4+fhUnP7OxbAdDdwdNx55quFCTD8r/lcldY+3dBJox4kFXxcyiYoy8f4OW8uT90n/RiIGh66Py5PNyje/J52hpSGLoGr0V5MkPKgk3nZ1RlMUJRcaMPdUwBZq8UyTXhD15xzYxt69Db16EFqmc8acDiYox8r5cUzKtgefJxyKuXDOayJG8J+9r8iY1ySi1VVG6K2jWa3i6fPizYmbhe+GaNnZJzbYd70Hhfh5q4DXz+A3Y7duJHfn6cfVbMX1UjpH3btTYkJ78OEIoQ3JNdTJKXVWM3ooy8u7/+egaZeRnIpZnoLVxePJ+CKURhFBS8L9v/M3dktz6u4ke8Vqiy08ab9cV00TllP8ripMvnAxloWnu7Dtd1wrCK4ejlFxTk4iQro7RM1A5mrxVdP6UIz8zyevpYw9zDQZviydDFXny1q4XAI348ReOu9+K6aPiPPmS0TU5m1jEQPPCxsYSJ5/JWjiOQ3/apDoZpbYqVlFJynwJyyiaIKOYWdi2J9cwDk3e0/WNIE7evceLQyjttq3oDXPRoolx91sxfVScJ18yrYFlB8sNXcPB84hGMLEjL9eYpLMWlu1QnYhiWja9A1kcxxnxBJGZTPGbkLLxMxNXT3cNscPYZDU/hFIvlmvsIk++bSvGvBUT0m/F9FExRj4IoSylyecKjby/Xh9BGtSwXOPnralORnBwyJo26axFMr7/n8ZBco3Sa2YkdiiEEsb2MLYdB00vNRkqHydvp3pw+jtGVCBEMbOpPLmmRJx81rSCAVmjaCr3cPgFRtI5KygAXpOIUpOMAvmEZfs7+ZJwfgilMvIzEdt7c9TGIavZjju4qmsaGuHJUO56TQO7fRvAiEr9KWY2+78L6hHIDaU8eTPvyQcTQCwHosO36w/SpjNWkJysOhkNflzjLSU4Uwg8eX3sHqJi8rEdL4DAewkdy3Vy7LxUaRgapq/Jk/fkg1J/TYvG3WfF9FIxRn6QJz/IyLuzVIsz7w2Haefj5AO5JhEJPPxUhRj5YOBVyTUzGssfeNXG48nnx5GiET3I7ZTPQglm21a02ha0ePUE9VwxXVSekS+Zu8Yes1zje/KZkFxTnYwGmS0rxZMvTvCm4uRnJo6dnwwFYx14zcfCRyNGYOTz0TUaVvs25cVXCBWjydvWEAOvZumB15EQHnjt7sugATXJKFUJ9/lYKZ58Pk95YcSFYmbhT4aaiIFXcB/qeSPvNeZYOD1t6A3zJqTPiumlYoz88HJNkSY/wjnhflqDnGmztzNFY12ciKEHETWpjDUxBzDNBGkNVIKyGY0f+qsFRn4Mck1Ik49F9UGePP2d4FhodS0T0mfF9FIxRn6oBGWWbQda81g8eV/q2b6vj9kNSQCqPCNfMXKNk9djNU158jMVu0iuGct1ctMauH+HPflA3+9tBUAfYRUoxcxmzJq8EOIS4GOhRUuB3wFVwOlAv7f8a1LKv4y5hyPE184NQ0ej0JO3rHye9OJqOEPhOA6W7VBXHSPbm2FP+wAHz3drlUcMnWhEJ5WuECPvvdj4MdjKk5+Z2J6B1sc18Jqf8BSN6kF5TD87pd2zD1BGvlIYs5GXUv4K+BWAEOJw4Gbgq8DfgddIKXdPQP9GTKApe5M8wje/adtBnnRfjhiJkfe3qU5E6ezNYDsOLZ4nD5CMRyrHkw/V99S00WXqVEwdtu1gaOMLoQzLNVFDD4II/Jmwds8+0CNoVY0T1W3FNDJR0TW/AD4PpIBFwNVCiEXAX3A9+UmvM+QbZD8nR9iIm2FPXhu5XOOHSdYk86dpdmOhka+cgVf3WP0UtMrGz0zycs3YNXk/CyVALGoEOZj80EqnZx96bTOaXjFq7gHNuI28EOJcICml/LMQYhlwP3Ap0AfcBnwIuHqk7TU11YypH77Rnj27FsPQicUjtLTUButqquO0tNQyq9VVkerqk8H6cvR5RUEa65NAFwCHLGkKvldfE8NyGLYdGNk204Hfr93daQBmzarGMDQSiei09nmmn6/pwogYxKMGdXVu0rDGRjeOfTT9MiIGsahOS0st1VUxuvqztLTUkkzE3AyXqQ6izfMn7Fin+5yV40Dp10R48pcCPwCQUr4KvM1fIYT4CXAxozDy7e19Y5qIY9k2hq7R1taHrkF/X5bW1l7AjYzJZk1aW3vp6017++mnITH04fseTjTk0ERxgnajukZ3bzr4XI6Wltpht5kOwv3q6BgAoKc7BUB/f3ba+rw/nK/pIp0xMTTo68sA0NbWR3NDsmS/9nWlaKlPDEqgl8ma6FqE1tZeHNsmnXZ/G/0DGTQcsh27iTYvm5BjnQnnrBSV1C9d14Z0jsf1PiaEiAFnALd4n48UQoSTT2vAlCR38V9jwT3ocO4ay7aDxFujGXj15ZrqhJv/IBmPUB16MFSUJh9KNayhNPmZSpBPfhi5pq0rxX/892M8/2r74DYGafKW1xbUaBnIpdWgawUxXk9+FfCSlNKPpNGAHwkh7seVaz4C/Hac+xgRVpGRt+38YFJYk/cr1I9q4NVLRja7IVngFVWSkbdCA6+6rqJrZiq+gQ5CKMtst68rheNAR09m0LrCGa+FaQ2aDdeLVEa+chjvyMoyYIf/QUq5Dvg28AiwEVgjpfzDOPcxIsLeSSQ08BqOugEGpVcdirwn7z4LWxoKiydU0sCrExq41ipg4NVxnIq5NmEGefJl7uMuT84pdQ7CuWtiRWkNmnTXyGu1yshXCuPy5KWUNwA3FC37OfDz8bQ7FiwvzzZQUP0pKPBtjD53jb9NMh4hHjOY31yYrKkqESGbszGtvBy0vxJ+GFZCCOWaTW389y0b+MHlpwUpKCqBwZOhSl+n7j53PCmVLW3kfWcn4nnyjuNg41Cvuy/les2sSei9YjqomLu/UK7RQ56866UEM16NkXvy/gMiauh88Z+OZVbdYE8e3Lw2Ncn928j7xkLT/RDK/dvIt3enyeZsBtK5yjPyWv6NtNxl6vKNfHpw2o2wXBOL6Di4vx/HgXptAKIJtFhy0PcU+yf7t2UKYVl24KUbIU/eDDz5wjj5EQ28Bg8IjQUtNYMqQFVSagN7kCc/zR0aJzk/55A16VM0phT/jXW4OHlfril1b9p2YaphcOsg27ZDnT6AXq0mQVUSlWPkPQ8HvKIHgZH3DHXoAeBuP/yPP0iVUGZSSJCkrAJSG+Qnk7nnr5zWu7/g68z+NawUsl5tBH/4v9xl6vaMfHoYucbPy5SzXMmmjgE0ZeQrioox8uEbt8CTL6pdOpqBVz8DZaRMLdhkJXnyQYIyrSISlPlG3hxhttH9hUzOIhEzhvfkvTkeJQdeQw6RX0ktl7NwHKjVBtCqGia+44ppo3KMvOWg63lD7sfJ+4ba1+LHMvBazpOvildOTnk7FF1TCQnKAiNfQZ684zhksxaxqBFkkSx1mRzHCUXXlNPk89E14Hrytm1Ti5JrKo2KMfLh6BqjVHSNPnpP3gwyW5bx5CuocIgZikLS9P0/uiYw8mblePI508YB4lF9yPJ/6axFNle+PGVx+T9wNfmY1Y+h2cqTrzAqyMjbeU2+QK7xJZeiLJQjMGJBZI5e2shX0sCraeXPUyUkKKtEuSaTc73yeNQI7vVSRt734hMxo/TAayiffFiTT1h9AEqTrzAqxsjbNgWavFUUXVMs14zGky8XA5+Iua+6lTDwagblEysjTt6PqqkkuSaTzRv5vCY/eDs/fHJeU1XJgVenIK2BuyyXs0hY3mxX5clXFBVj5P0EZVA8GcrzUItmvI4k6sKyC/X8YiKGTiyqV4Qn73u+RoV58lYFhVAGnnzMGDJ3jR9ZM3dWNablBEVBfPyiIU6mn9kPfZvzkms8T96dCKU8+cqigoy8U9qTDwpUj37Ga/AWUEauAVeyqRRN3q8KpVXQwGslxclnPJ3d9eTdZcN58gADRYOvfnGQzNN/ITLQxusS69DbXiVp9eIAWlX9pB2DYuqpGCMfzl2jayU8+eIQyhFp8kPLNeAmL+uvELnGD6cLn7/9Fd97raQ4+bAmHwy8lrhOXX0ZYhGdJm+GdrrICbEdhzpzH7mN95NbfBKddjVz5Q3UWN0MkETTK2eGsKKCjHxxdI0/sFrsjY8qhNIaeuAVoDYZDYqL7M/kLJuo9zDTNNjfTWNek68kTz4s17jLSso1/Vnqa2Jl53HYDhzUt9G90Me8nT/1n0w83cayzIv0UT2oPcX+TcUY+eJ88oEnH3jjY8lCObwnX5OM0puakpT5k4pp2sE50irCk6+8gdesZ+RjYU++xHY9/VnqqmMk4+6oarEn79gOSasHraqBSHUd0pxPZ/1KdGz6tLFVZlPMXCrGyFu2XSa6pkiu8VLpjiitwTAhlI7jsCDSgZ3qGXf/p5twJk1d3/8TlFXiwGvai65JRIceeO0dyFJXFfbkiwdeHRJmH1p1YxBC+eq812Oh06uMfMVRMeKb5VWxh6I4eWtwhExxoe+ybQ7hyVttW0nd81PO7m3lqGgNZs9qInUt4z6O6SJnOcHEmIpIUFaRA695uSZINVzi8HoHciybX0eizIxs24a41YtetTS45j1GA7fWvJN92SSnT94hKKaByvHkraLyf0UzXsOpCcIPgaEwbQeNvMQTJvPUTZBLs2XBeSS0LKlbvo2T6R/cyH6CadrBOaqItAZW5SUoywYDr3rZ3DW249A7kKO2KpZPu1EUK+84DnHPkzd0HUPXyJk2+/Q5pA2lyVcaFWPkixOUDZZrxuLJ2yVj5K3OXVjb1xE94lz6lpzBr/vOhIEOzM3PTMCRTA+mZRON+NFJFZCgLFeZM141zZuVXCZ3zUDaxHYcapPR/GS9kCfvOA5RckTtTJC+wC8c4sfPKyqLcck1Xi3XOeSLdV8KHAx8EYgBP5RS/mxcPRwhllUmd02JMEhD10eWhdJ2gvj6MLnn7wIjSvTQs6jdneMVcw5mVRO5V58kuvI1E3E4U05Yk9c0DWc/N46VOePVDsInNUp78r1epFdtVcydrBfRC4y87TjU6ykgP7M1FqoOVTFenyJgzNdUCKEBK4GjpJRHSymPxq33+k3gNOAo4CNCiMMmoqPDURxdM5QnP1K5xrKcYKasj2PlyG16jOjyk9GTddRURQGN3uZVWDs3Yqd7J+iIpha32Lkv14xsHsFMxXGcikxQlsmZxD3vvFz5v94B19+qrXKLz7t1iPMDr44DdfqA24Y3szUa0cmabqrhUtKkYv9mPA9ugRtOfacQYq0Q4mPAucD9UsoOKWU/cCPw9gno57C4CcrKZ6EMa/KGrgUzYYfCtO1Bnry1dxOYWSJLVgNuCCXAvrrDwbH3W8kmZ9kFA6/7sY0v8N5Hcp1nIg+v282ejoGCZZmc68kDoeiawu/ljXwMGFxs3rbznrwv10S9Yt7h7JSKymE8ck0jcB/wUSAJPAD8Cdgd2mY3cMJoGm1qGlsIl207VFfFaGmppaY6ju04tLTUEktE0TWYM6cu2DYa0YnFIrS01A7ZZjQaIRbRC7br2LCJlKYz58jj0ONV1Na5tTAHahYQnTUPbdtTtLzmzYPaGm5f00W4X/75i8cjpE17Wvs8nn33h+YtRKPGhB7HVJwTx3G49s4XOO3oBXzmouPyKzSN6mSUlpZaUt6DrKY2UdivTe0ALFnYSHNDktrqGFZofWdvmnrPk5+96CD0RDXJeARN14lEDOITfL4K+jbDOFD6NWYjL6V8DHjM+9gvhPg18ANcuSbMqN6X29v7xjQRx7IdslmT1tZeMpkcluXQ2tpLb28aw9BpbQ3JKA4MDGQLl5WgbyCDplGwXf/Lz6HPXkZ7jwX0ugNZEZ097f1oy04i/fRf2Pvqq+i1+XDKlpbaYfdVjoF0jrWb2jn5iLlj+v5QhPuVzphYpkVray9mziKXtcbc54ns11jo7s/PQO7rz0zYcYy3XyMlZ1rYDjy9cS979nYHb6E9fRkMTaO1tZeuLtdQd3e7Xrnfr1373P+zqSytOZOoodHdmw7Wv7qrh3otha1Haeux0Hp70YD+gSzZrIledL+Pl6k6Z6Olkvql69qQzvF4NPnThBDnhBZpwBYgbI3mAbvGuo/REE5Qpnupch3H8bTm/Ctodv29nKA/P+I4+bDM42T6sVs3E1lweLBM0zRqklH6BnJEDzkVgNzLj07UYfHUi/u4+raNdPSkJ6zNUpgFaQ327xDKcNbF3H448OonIhvImGza0R0sz+Ys4tH8NYLSA6+JmBFIb8lYhL6UGWzX1p2iTh/ASTYEbcSC6Jp8/idF5TAeTb4B+J4QIiGEqAXeD1wEnCOEaBFCVAEXAn8bfzeHxwolKAtyxjuOq6t7htpxHLLP3cqZPEHUHCjblo9ZFEJp7noRHAdjQeFYcm0ySl8qh17bjDH/UHLy4Qkzkv4sx8nOdJmznHyCsjFUhlrzclvJ3OXTQS402DodM1637ulld/vY50yE+7/2lfbg70zOLf0HQw+8+oOuAAvn1LC3Y4Af/nktA+kc7T1p6vUURk0+nbA78OqHUI6524oZypiNvJTyNuB24DngGeAaKeUjwBeAvwNrgOullE9OQD+HxQ4lKAvnpwnHujvde3FS3USxWJZ6ftg2LdsJygYCmK8+BbEqjNkHF2xXWxWlN+VKBNEVp+H0tmLtfnFCjsufAOMb+8nCzV0TSlA2Chvf1Zfhxzet48kX9k1S70ZH2EhORwjltX97kRsfeGXM3896byIasHZTW7A8nbWC2PfyA6/ZYNAV4PxTlvDecw9h/asdPLh2N23daRqMFJEiIx+EUCorX3GMK05eSvkl4EtFy64Hrh9Pu2OhMJ+8HixzwyDdz+YeCUAXdaxMr8WxbbQyRbr97wcPiEw/5paniYoz0IzC01ZTFaNtt5u/JrLsOHjsenIb/05k/qHBNo7j8PyrHaxc1BB4YyMh6xmsSTfyRXLNaDx5/y1jYIakXA6nMpiOLJSptMnAKK5xMf5ErkVza9m6p5dUxiQZj3hyzfCevJ9iGEBzLM5apvFK3R727q6hJ2tQrw8UFAZxjbzlJT4bc7cVM5SKyV1jF1WGcpc5mHZek7d2vYiWrOMR8xTelPsb5rbniC45tmyblmUHcfK5TY+BZZac7FSTjAaha1okTlScTu75e7AHuoIJJ3s7U/zoz2s55Yi5XHL+yKcO+CXfJlsKyVk2kTHOePWLRvtvHdONHxuvMT1yTca0gjwzY/0+QGNNnK30ks5aJOMRMjk7cBCG8uQXz63FyaZIP/ArzB3Pg5nlvRGw9t1LlihRTIzmJcF3YhHDu/668uQrkIqZ4FZKk7dsp2Amp7XnJYx5gi2Rg+nVashtuG/INk1vxqvjOORefAi9aRFG8+JB29UmowxkzMBrjB12NjgWuRceCLbpT7sPgUfX7+GhtSMfi/Zf3VOZyTOglm3jOIWZOkczpuAbtNEaNtt22Lx74jN4+m8/ibgxLQOv2dz4jLzvyfvaejZnueNJBZ784OI3TpC3JkruxQcwtzxDdMVpJM6+jKcXXcxD6ZW8mJvPY/PeR+TgE4PvRSM62Zx7DygbX3lUhJF3I2komPEKvibvavV2bytOXzvGXIFuGKyPHIm1cyNW586y7fp6vrX7Rez2rUQPPbPkdjXej9GvEKXXzSayeDXZtXdid7nTBrKeR15fE+M3d77I9fe+NCIpwfeSJ9OTN83C1A+aNrJ8+z7+g2i0hm3tK218/bdPs68rNarvDYevySfjkSn35B3HIZO1x/VW4z+kfG09nbXImjYObgZKq20Lxm1fYpbeV+DJpzIWlu1QlzDIPn8PxjxB4rSLiS4/iZolR/CXgeO4tvd0tLkrCyY9RSM6OcvV5NVkqMqjMoy8Z5DCCcrA8+S9WavWvs3uurnLMXSN543DwIiQW39v2XZ9PT/73K1oyTqiK04ruZ0/67UvNAknftrFaJEYqXt/jm1mg7C4y95yOGcds4B7n97BYxv2DHtsmSkYeM2VyLk/moFX/0E0WiPvS1y9E1xZKzDysciUz3i1bAfbccZ1vfwHhH9fZUJvBvGogbXvVbTefZyd2FDwxuUP/h808CJOfwexVecF6xbNzsdRN9XnNXvwjHzOLkgNoqgcKsrIG6E4eX+55cXJ2127AA29YT66rtNvJ4gecio5+RB2996S7Zq2wxxrD9bOjcRWnYcWiZXczo94yIR+2Hp1I4kzP4TdsZ2ep/9GOud64nXVMd537goMXWNf5/Ae7FQMvPrebjRUGcoZRQHAvFwzOq/ZP6bMBB9b2JOf6oHXsZ6LMDmzUK7J5KzgHMWjBs5AFwAnxjdhZPuC763d1E5SyzB/x93oDfMxFq0K1rU0JgOpp7nIyMciuhtubDlKrqlAKsLI+xObBmnyTl6Ttzt3odU2o0ViQW6b2HFvAyNC+rE/lG7Xsjk4uxEicaKHnVV2//6PJ1MkqUQWHY2x4DC6H/8ruXSaODkSe58n++QNfLjuHyT3DR/GmQ+hnDy5ZpAnr49SrvH6OFqJwj9fpYx8Z29mVG2F8Y8nETem3Mj7bzWmZQ95DrM5KxinGbTOk798uSaTtQoKhjgDXWDEiGCxeNut5Dr3sHX7Hh548Fk+OvspIpkeEmd+CE0L1VDQNA6a7eaKLzbyiZgbf9GXzgXZLRWVQ0UYeX/wqZQmb1puGKTdtQu9cX6w3nIc9KoGYqvfgrVtDeb2dYPaNS2HFnM3xuxlaNHEoPU+/o+klLcdO+atWP1dHPzCNXyz8U9EHv4lufX3sEjfx+mdN5N+dOho07wmP3mefFDLNqKTefavLBzYOCa5ZtRG3j+2ou9t29PDp3/2CK/s6i71tWHxPeGqeGTK4+TDktVQ8tVN/3iV7/3huZLrsqU8+VDBEHugG+rmcG/6SGZ1vcD2n1/OrDs/x3/U/oXFuVeJn/jOQXM5AA6eX099dYyqRLRg+dJ5bl6nTNZiiIhixX5KRVzSYrkmrMlblk1UA7t7D3pD3sj734kd+Vq0ujlkHr0exyr0lnU7S0OuteQPJowv1xQbK4DIPEFyyZFUD+zkyczBRM/7d2o++EtunXMZz9iHklt/N+auF8q27Xt1k2rkTV+u0ck+fzerO+4i6Qw/I9hnrNE1vgdf/L1dbe5s0Y6esXnzflqDRGzqB16zIzTy7T3pssfnt1HrafLprBUM3PtyjVZVz+2p1Tx96L9Rd84HuDl1HOvmvJmqt32V2KrXl2z3gtOX8sWLjxu0fMm82qDWqxp4rTwqysiXjK6xHeq1HrBMDM+Tj4TyzWtGlMTJ78Hu3kNuQ+Eg7BynFR0bY87yIfcfL6HJF7Tzzv/ggYM/xY2pU4gfdCiaEaGhvoobeo5Bq24k88QNwQCan0PEZ0rlGs2GTD9RJ8u5kZGnTM5H14xSk8+Vlmu6+1zjN9ZUDr4nn4gZM9aTT2XMstfU739NKITSP7euXNMdpAnOxOrY2XQSf08dRu0RZ2C0LCm7z0QsMmjQFVyZbvlB9YA7R0JRWVSEkbeKB16L4uRn2Z3u8oZ5wfpw7U9j0VEYC1eReeomzN0yWL5Ac6fp67OXDbl/X5Mv523r0Tj9dpR4LF+bc1ZtgrRtYB35FuzWzZivPonjOHzuvx/j78/mwzozUyLXeAbEcrPfpYxajjVewmrfPqLvB9E1o+xjpszAa5enx6fHauS9cZhoRJ+2gVcY+nyksyam5ZTsX9Z0c/vHogYa7rX3H4jxiIaT6kFLukbZth3Wv9KOpsEhBzWMud9ioftd5clXHhVh5H1P3r8/IwVG3qHR6gAo0OTD3rKmaSTOvAS9ponUXT/C3O4OiC7U9tEfbURP5nPRl8L35IfytjPZ/EQWgFl1cQA6mlejNy0i8/ifyAwM0NmbYW+oWER+MtRkxsl7Rj7nGvkXms4mRYzM438Y0aSo7BjlGl/eKpa5ujxPfqwPtpxnJP1avlNZ5SobepsZ6nyks+VluGzOIubNPo1FDTI5K2g3bqfAsQNP3nFg/attLJ5TSzI+9gnsYpGb5kDNeK08KsPIO6U9eduLk6+z2tGqGtBiVcF2xamG9WQdyTd9Bi1ZT+rOqxi45Vssj+yiO3HQsPvXNY2492MsRyZXZOS9Yg/tPVkSp/4TTn8HmYeu5Y3J54j2uROo3FmOk+/J+7NCY56R74u1cE/2aHey2NY1w34/M+bomtKefHefG++dGqNEZXpG3k+3a02hZDMauSb8fxjfkwfXgcjkrCAvUMJyQyb16gZvWwu5tROxqGFc/V46r45YNH/OFJVDRVxRa4jJUJbl0GC2BV68v12p8Da9ponqt3+d+InvwjEzaDi0160YUR/iMWPI1/NsqHQbhDz53gzG3EOIitPRtz3F65PPc2LHLTiWWZBNcSrkmkjWNfK5SC2PZQV6wzzSj16Hkx06nj+IrikaTxiOTBlvNpBrxnjMWdNNtuYnqptKyaZg4DVbfr9DefI5M5+jJh7VyWTdcEtd04h6D2LfyG/e3UvOtFnhyS1jJRrR+cy7V/P6ExaNqx3FzKMijHw+ukbHcWyMXH+wPG4PUJ/dgzHnkGB7QxvsyftoRpTYUecRf+tX+Gzne2lvOGJEfUhEjSGNUiZnEYvljXxNMkosogfFQOKnvZ+2Mz7P1b1n0mi1k117RxBKVxNzSGfzhR9+ddvGcaWyLSZv5HtAM8hFqzBtncQZH3LfMB4bOswzEyrSMRpvPl1G5umagIHXaETPJ6abwlmv4cHncuciPCO2lMTnyzUA8WiETM6iP21SnYzAgBtW6ie+8++flobkuPt+8IJ6Gmvj425HMbOoCCMfngxlbnqc2fd/maWRfVi2w1JtFxoQWXhksL1hlPbkw/jeacQY2SmKx4Y38mFPXtM0GusSdHheq2ZE6Is2sT63iBf15WSfvYXsvi2cm3ieK6t/z5nxDYExfHlHFy/v6BpRv0aC/8ZgZHrQqurRdTcpmzFnObGj3kROPoS5tXRMNxTq0NlRRNgEoZdFhq57gjR5/9q9uquH//jvxwrSToS5+aFXueaO8mGsoyFs2EuF1EKhPFVSkzdtohHPk4/ppLMWA+kcVYkotjfbVa+qRyOfSqO6KPZdofCpCCOfnwwF1t5NaI7NRdUP42RTrDB2kNMT6C35CBld17DsoY3RS54RXTRnZEV1E7FhNPmsRaIox/is2jidobJ+vuf6N+sUtEQNxoM/403JNaT0JBdUPUP24d/iOA59KbOgjul48T15Pe2G5mla/pzGjn0r+qyFpB/8DXa6dO3JkerQg74XxMmHKjnZdpDLZjzRNWEjv2lnF3s7U+zpKB37/8LWTl7c2jmmfRUzkuia9DBGPhcq85eIGt7sWJPqRARnoBvi1WiRGLqu0Z9yz1FVomKyhismmHEZeSHEV4QQG7x/3/WWXSOEeFkIscb797aJ6Wp5fHtt6BpW+zbsRAOz9H5aNl6PiO6is2ZZQXGQUgOvxazb1EY8ZoxY6xyJJ19cLGRWXTzw5MGt6QnQnomSOPtStFQ3rXYtt8z6Z+5NHY626SEyz9/FcvtVYgNjr8JkOw5X/WkNj65zUx77seRauhu9uqEgQZlmREmc9RGcTD+Zf1yD45QI+ctZwWT4kRp5N1vjYF26byAX7Ds1Hk/eyMs1/kBuOU++uz9bdt1oyeZsYp6BLifXhGWoUg+yQk8+Qjrne/IRnIEu9Co3fNJ/GEcjeiDvKBTFjKeQ97nA64DVwNHAsZ5BPx54jZTyaO/fXyakp0MQTIbSHOyOHZjzV3HTwPHUtm+kQU/RWVs4mck3YuUGCR3HYe0r7RyxZNaIow1cTX6IEMqcFYRa+jTUxOnpzwZae1BhKWMRmX8oHSd9jJ/1vI6q2hpuS60mO/tQco//kUtqH+ATVbcwIMdWMHxXaz8bNnfwvFdazpdrtJTvyRcWiTaaFhI/6d2YW58j+9RNg9rL5myqQxkTh2Lz7h6+e/2z9KfNIAVaJpc/bz1eZspYRB/zBLBiucZ/6+kvY8h7+rOks9aEDNBmchaJWISIoZc9F8N68qYd0uS9gdeUSU0iip3KT4Tyr1NNMqri2xVlGc/jfzfwaSllVkqZA14AFnn/rhZCrBNCfE0IMekuhi+9xNKdkEtjNxzEw5mVbFl6IdvNWfQ0rCzY3giFWJZi+74+OnszHLW8ecR9iA8n14RewX1qk1Es2wk8eN/Im5ZNzrTpr11Kt1NFfVUMB532I/+JzNHv4Gc9r2W72YT1j/9h4JZvYe7cOOJ+AsjtXUB+gNOybQwsyPajVTUEsx7DD8Ho4ecSPfRMsmtuJycfGnRsddVuMq3sMN73i9s6eXFbFzv25bMnhmWNHs8gtzQmx1woJYiTH4Enn8lagaEt9xAYDVnvOsej5Y18ODS01IPMfevzQyj9gdccVXEDp3tvULrPt+s1VUqPV5RnPIW8N0gpHwcQQhwCvAv4G3A/8M/AScDpwIcmoJ9D4hvruBdfTqMb2767/ii+33M+TqJwMpM+jJFfu6kNDTjy4KYR9yERi5TVYG3bGRRCCfksg35e9XCN1FTWDF73fQM64MToXnAaL5nz+Fnva+ldeQF2fwepO64i9+pTI+7ri9tc/dk3qDnTpl53wyT1Ak8+/x1N04ifehHGgsNJP3Qt5q58ofKsaVGfcG+l4VIb9Pa7x9rqFQqpSUYLBih7PD1+dkNy7J68p8n7NWu7+t2HWSkj3x3KZT8Rko0/wJ4YIqQ2nRnekw/kmqg38JoxmefsxUn3EjnIDSLQA0++dApshQImoMarEOJw4HbgCimlBN4WWvcT4GLg6pG219RUM/xGRezocA1GndkKmk7T0hXAHqJx18Opr0/S0pIfQK2vc8PNGmdVD8rIB/DK7l6WLqhn+ZKRG/lZDUnSWYvm5ppBr86+7trUWFXQj4Pmuf02YhFaWmqxQ9+rqk4QT6a87VwNNhaPEvGOycKg/5BzOPL8C9n9x2+QvvdnDOgR+mqXsurDX0GPlw6pcxyHl3e4YXjdfRlaWmqJxaPMirj7apg3j5qsG0bX1FQzaBzBevdn2XXtf5D+2w9ofuOl1BxxBrPsDj6Uup4nq5YQjx9dcIzFZL0Ha7/3MGiqT7B1Ty+zmmrcFNCam9t/yYIGnnu5jYbGqsDgjRTbcaipjtPU5KbW9R+iFtqgvrX15Q17JB4dsu/AsOsdTaM6GUPTNdD1kttHt4QGeUtsY1o29XUJWlpqmdVQFchpS61NYESYe8yp6PGq4I20pmr4fk8nM7VvB0q/xmXkhRCnAjcBn5RS/lEIcSSwQkrpC7caMCr3qL29b1S5zAE6Ot2oCat1K7G62XT3u95RZ7e7PDWQobU1HxmS8ry3vft6g+o7PjnT5oUtHZy1ekHBd4bDyrml13bv6Rmk40e8B0kuaxa0aXta9PZd3bTUxOjqzkfa7NzdTVtHv9e4ezz72vuIhwze9t3dHDKvlui5n8TZeB9PPfsyR3StY/t1V5J8w7+hxQYb+p2tffT0Z4nHDLr7s7S29tLTm6becPfdY8ZJeRWG9u3rHTSOABA779+x7/sFrbf8hI6tm3lH1ZPojsWpiZfoeuJq9i34FJpRWkLY1+4e01YvjXC1FxWyc1cXyXiEXft6iRgaCa+o+PadXcEbz0hJZ0xs06Kv1z0m/35q6+gfdE237uwK/t6xu4c5dXFMy+Z/75KsXNTAyYfPDR7aLS21w94TfQNZdyKWptHTlym5/b42V6pKxAw6e1KDtklnLcyce6+YwXiFQ13beoz5h9HeYwH579Qko6O6V6eSkZyz6aCS+qXr2pDO8XgGXhcCNwPvlVL+0VusAT8SQjQKIaLAR4ApG3iN9O5Eb1oUyDHlYt2Hkms27+4hZ9pBwqaREmSiDEkPqYzJr27bSKtXAWqwXOMaQl+iGMiYgc46kBks16QzVoGk4GvNWixJ/Ojz+Wv6RH6fOgNr7yYGbr6yZP1aX48/6uAmevqz2I5DzrJZGOkETUOvaQoKR5QbmNar6km+6TNEV74Ga+1tLI/uQ857I3/uP4GGzhdI3fNTHKvMIKd3rL5cU+8dm3/eevtz1NfEqYoX5uhf83Ibdzy+tWSbYbI5i76BHLMSNtU7HnfHGnCo1wZKyjE9/fnoJr+Ix96OAR5et5tf3fYC/33LhtEVUPFyFA2V5sI/psbaeIF0A/nMqf7D3H/YHWR0EM90Ell6bLBtMPA6yoeg4sBiPIOiVwAJ4Ad+uCRwCvBt4BFgI7BGSlm67NIEYtkOUUyMgQ70xgXBa6xvJIuNfDjtQTEvbutEAw4ZpZFPlEhS9vKOLh5dv4fH17tjBcNp8qmMSUONK5WkM2Yw47Wuys8rbtKfzqHhem/hWHnbdujqy/BMehHaaz+Fk+ln4MYvk37wWrfIhMfOtn6q4hGWzavDth0G0iamaXOYsRVjrkCLVwcPwaGSk2m6Qfz0D2Ad+gaeyiyjb97xPJxZycvz34S1bS0Dt34bu6d10Pf8Y/VLH9ZXu8fr69c9A1kaauMkIoURRw+u3cWtj2wZNmHano4BHODw9LPUrvsj/1J7D5fX3sOVjTdyQv/fcczC+QXhc+g/BPyw1uNEC0++sI/bH9sy5D7DZEw3hHKogfhU1iRiaO54RNG4g5+QLhrVyW64l9VrvskbEmv5UM0D2JEEkSXHBNv6A+S1STXwqijPmOUaKeUngE+UWf3zsbY7FhzHodnwcnrUzwkGpHwjaRQlyR7Kk5fbujhods0gGWc4SlWHavPkl2173L4V69vRiE4iZgSTf1IZk6a6BJ29mQJP3vcM01kLy3KoSkRorHXDLx/fsAc0EAsbg4dWd81S5l14JdlnbyX34j8wNz9N/PT3E112PO3daRbUaSzreAgRcegdyJLMtDFH7ySy9I1APmpjOAdW03RSh72Z6x55go8kY2jAlrpjOfKwJaQfvIb+m75M4owPEl12AuBeJ3+wty+VY7bezXFtN7PRWBQYxI6eNMfV7uWQh3/O26uWk04dAdTS0ZMmk7Po6ssOOfXeLzjS0LsJO1HPEqeNnGPwbGYxx8afZ+CvXyd5zr8Eaae7+7PUVkVJhd6S/FQB7zx7ORFD5+aHN3Po4lmBVrqzrZ9H1+/mwjMOHpS1MevNh7Cd8nHy6awbZpmIRQYVMffv2SonTebJm0DTOa9qLb12gv7TPkF9Iq/X+p58tYquUQxBRUyTs2yHFr0HAL1hbglPvvCHGK4BG8a0bF7Z2c1rjprPaMnXeS1h5Pf2etsMfnGqrYrSF4quWTa/Dna6Bt+fWKNpGglvslU2Z1GdiFJfHaOrL8Mf73uZmqoYLefl9ffuvgwLmmeROO2fiB5+DukHriZ9788wDz6RwzszHKW9RHLbAP9SB6nHOlnS754P30vUteE9eZ9M6EEUi7mzM6PLjsdoXkLq/l+QvvfnmAc/S/ykd5EyarBsmxgmOSK8p/pRWrpa+be6DQys7cc+8z2Y3a2cxh3YsWpOdyTZh7+HlbyUZN8OFhome9v7hzTyO9v6qdazRLq2kjv0PL7zaIyUEyUXreE5axuXxJ+g//++Qvzk9xJdeQbdvRnOSr5IPNpLW787wN3ek0HTXDnlotcdwpYtO7n7yc2cvNqN2np8wx7ufHwbRyyZxaFLZhWej6xFo9NDDpNXykxKTmdMEjE3Aqe1q/BB4N+zi/bcD2aGPSddwc13PUurXccX5xXWNQhCKFV0jWIIKsLI27bDbMMz8nVz8pq878mXk2uKJr9s29tHdowZ/UqVAPSN/B5vsNH39sPUVsXoHXAnRKWyricP7mzPjGkR87TZ+uoYnb0ZHByqkxHqqmNs2NKB40B/2gw0bsjHvwMYjfOpeusXyD53G9lnb+U4x6ErsYjccW/j0Xvu4/W713C4Y7GHFg6pcaOJ9BF68pAf94gV6dB6XQtVb/k82WdvJbvmdsxta8iseC2X1jyLiO5ikzmXZdFWOsUFbFi7kdO2/IO+65/iM9VpNGKkzvgU1974CJfGnmTgr9/gXxJAAvoeeZ6ccwGRJasLClX77Grr5/iGdnActPmH02q70TqLGpOs27uAyFu/iv3INWQeupbcCw9wVrfNEmcbGJDe/RLZje/E7IhyQd160n99GLt9K/+RsLBaNbZdczuxE95FV0cXc/Qunly7mZWLGwuiqWwzw2l7biBh9XGwvoDMmi6M5sXozYvRPS88lbE4LbqelT17uSl7dEH/c6ZNvdZP076niB52JkbjfF4y9wCDUxfkNXnlySvKUxFG3rIdWowenEQdWiyJ4VmnwJMfoVyzebf7oFg2f+giIaVIlCgB2O4Zed8hjpXy5JNROnszpLMWjuMa/Yihe558fgLV/OZqXt7RTV11LPDk/XYt2+GFUO4Vf0DWR9MjxI+9gOyyM/jsfz/N249ZyXEHtXBnqp2Fx59F9Yu3sYHl+Hk6i2e8DkWhpFQ4AUjTI8SPexvRQ04h/dj1xDfcysHRCJu0pSyPbOaV3GyaVp7Nnx+rY9EJZzO79UkefjXDcW98M9Wz5vKyOY+NR/wrK7Pr+cPj7VRrGd4UeYn0PT9Bb5hP/MR30GbM4aXdKV5zvFuHd1f7AKcmdoOdxJi9DHCN/JzGKrbt7WNAr6HxjVeQe+EBcvJhFljbWVd7Oi9aB3FK5kESD/8v5wG2oYF+MLEjX0+nVcWjT73ImR3bMf/6Dd4J0OA23feHFqJLjyF22Fk4tbM5PvIyCauP7fWraep8meyTf/bPBpElxxBbfT51qR2cYT+Glna4LLqZ9EPtxFadh14/h2zO5tTES4DjZkMdcO+riKENSl3gv3HVKk9eMQQVYeRtx6FF78WpnQPkX2P9iTllPfkiI//qrh7qq2NjSrdaqjpUe3dhHvbigVeA2uoYW/f2BgOMybhBVdxwB15z+bzi85qreXzjXkzLZnZjYxCVUpOM0pfKsX5zh9u+Bl19pXWCtrSBiUFzfSIY9G11Gngw8aaCc+Gfv5Gkhs8EnrzuevIlJvfo9XOoesO/seGJJ/j1P/axcuUy/vDiK1hGki94cf+dVYvpWbqEWzds5LyDV+J4oYN9Voy981/Dc9k1aEB/7SlctjpN5pmbSd31X1QDqxyNns4TiC06ksMHnmOp/hKRxatwInkPd84st2BMXyrHrLoEscPOJnroWfzb9x/g3BULSXeluC51Pl99XZxb73qKnuZVfOCtJ7vfdRzWvNDC5ojDx45q45aHN2MlGsn1dXJWQx/1G+4j9/zdaAuP4tzkS3RXL2LTgjdz8+Yt/M/Hj4POHVg7N5LdeD/mlmd4M1EGtBrWLn0/bLybU+XD5F78B5Glx8GcUzk1/hKplsOoq20hbrphwFWJwakL1IxXxUioiKxGgVxTNxtwPVFD18iZpTX5wJN3HGzb4Z6nt9OfzvHq7h6Wza8bUx6QRJEmn8lZ9AzkAmMMlIw5r62K0juQC1IbJOMRkvFIMPDqyzXzvYk93f1ZqhMR6mrcds8+ZgEa0NmbYVZdnIaaeIFcE8Z/s2iuTxCN6CTjEXoHckFN1OD8aEPPCA7je/KxqEHMy5hYjj2RBXTZ1SxoqabTroFYsiD0dF9XCg3XIIcfmv5A6MI5NezuTBNdfhLV7/gGibMv4/7ImTycETjb15B78Fe8teoZMrULiR//9oLrPqfRHbMIh1GmMm6d1bqqGDXJKP1pk8ji1dzVK0g25lNaaJrGCYfOYcP2AboXn82dfYdTf8TpvFh9PP/TezbJ93yf2DFvxt73Mg36AHsXnEPck+bWbEvzfy/FuDNzDMaF3yF2wjvpopbHGs7DqW3hhoGTMC78T2KrzsPcvp5Zj/2IGj1DaukZQN4xqC6RZTI/41UZeUV5KsLIk0tRq6fR6uYEiwxdC6SDiF7ek1+/uYM/3Psy19/zMns7Blg6b/RSDYQ8+VCUCFBQlq2kJ5+MYdkOHT2uYa5KREjEI6QyFlkzn9FwfnNV8J3qRJSl8+qYO6uKU4+cR3ODq+PPqkvQUB0L8rHbjsODa3cFERxtISMPUF8TozeVwzSdgglco5Fr/IIhQWy4WT6tQe9AFo38AysRNYKHYzprsq8zRUNtnFjUcEsqeoPNnd65WbmokdauFJZto+kRjGUncmfnEv5v4AQeWP5pNh17Bd/qegu5s/4NvWFuwYMr7Mn7+OGT9TV5I987kCNn2kF5Rp/lC9xB2SdecLN/zm5Mcv7JS9jR2s/aHVnix/0/0m/6Jt/tfhPpphXBtf6fWzdyz1PbueWRLTy5qYf40W/kp5n/R0/NsmCMJhOpJX7iO6l531W0L3sjf08fijNbBOcVSueLV568YiRUhJGPpdxsinr93GCZrmuBbFFd5On4ZeFs22HtK+53H9vgDm4tHYMeD64nq5HPS+IbVL9AsqFrJQuQ+BOi9nmzdpPxCFXxSJC7xpdrZjcmA8+0JhllTmMV3/rISbQ0JAOjOas2Tn1NLDju515q5do7XwwmEbV3p105yDMY9dVxegeymHaRJ+/9OVQpP9txeOT53fR6htKXa4ZKUNYzkKOmKkqDJ4fFYwbRiI6muZ58a3eqoMJRImaQypi096Spq45xUEsNlu0E53ZfVyoY+H1l7wBr90XoMpqY6xl0XdfQvbc6/8EWTkK2o9UdEG+pT1KdcJPF+SGYs+oKjfziubVoGjy50dX4WxqSnHDYbOY0JrnFi9/P2BF2Wk3uA8976JuWzef/6Via6xOs9bJ+prJudE2ySOLTYlXsW3AGNw8cHyr/5/5fKl+8rmteXdbRpX1QHFhUlpEv8uRzps1BLTWDNHY9iK5xWLepjcVza4NX36Vzx2bkdU0jFpoA4xsiP1KnlBcP+QlRe73JQVWeXJPKmGRy+ZSzhq4Hxqs6WfiDn9/sGfm6hCvX9GewHYdbHtkCwGMb9mLZNm3dKZrq8ka0riZG70AO07QLpI1SCcqKWbepnV/f/gJ3P7UdgFjEIDZE5kVwE6LVVcWCB1s8ZhSEh7Z2ppgdMvLJWIR01qKjN0NTXTw4ft8Q+6Gpi+fUsnlXD0+9uJdjRUvBW0nE0KhORAJJI+zJr9vURnUiwtL5tcF6v02/Bm/Ql3iEhXNq2entu6UhiaHrnH/KErbv62PNy235cNKYEVzvVQc3sXReHUctb2bj1k7SWXesJRmPlJxbEY5WAs+QR/QynrymKkIphqUijHxtXQ0d1BNrKvTkAY5aPjjJmC/XbNnTS3tPhrNWL+Cs1QtYsbBhXBV2wnVe27pTGLrGvFlV1FZFS+rxEPbkXSOfjEdIxryBV7OwZKBvzIt/2IGRr3U1+WzO5rH1e9i+r49jRQs9/Vk2bO6grScdeLTg5bMfyGJadpCxEcKTocpb+Ue9Wbz9aTcVQ8TQhpzKD+5s1tqqKHXegy0R8lZ7B3J092dpaQwZ+bhBytPkZ9UmmN9cTTIe4de3vcCTL+xl+74+DF3j1CPn0p82SWUsTj1ibsE+DUOnOhklYrizUPu8Skq27bDu1XaOXNaEoet5I++lQC725CH/VlZbFSXppV046fA5zG5I8tdHNgfHHovozJlVRVU8wltPWwq492HOtHnuZdchSXpx8lBo5P1xpHAkzaK5tSyeOzhplaapilCK4amIO0ScciYtb31zQWKfvJEfnBN+Vm0cXdO46R9uMewjlzVx+qp54+5HPGawfV8f/1izk5e2dTGrLo6ua7Q0VpXNVe4bvO373L7nB14tYlGnIOzSl2WK5aflC+qJRXQWz817mtff+xJzZlVxyfmHIbc9yp2Pb6OtK81Kz1CBmxOnbyBHLKoTCRkVP9XAM7KV1x6XYP3mDlZ5aZfXvdLO0nm1rNnURlNdnPaeDPGo65G7RVBytHWlSMQjPPdSa8GDorUzhVjUQMxLxes/+OKxCJu8zJgtDXnjmoi5bzQdvRkOXzqLqkSEL7//OH51+0b++68baKpPMK+pKnhbaqqLIxbnjw98T949XzWJKJt39/D4hj001MTpHcixynMCfCP/0vYuIoYePHzDHLKokXue3FYgKRm6zptOWcxv7niR+57e4R2PwYLman7yydODtyKxsJF41OC+Z9xtEvEIibh7/Dta+2j1Bp39N8CwBPP5i/L5asJoaFTFlVSjGJqKMPKlMHQ3N8iyEgOpzQ1JPve+Y/j1HS8wqzY+YRXqZzckWb+5I4i3P1a0AK4HuHNf6cxyddUxquIRuvqy1NfEiHlRL6mMSSpDgUFZsbCBaESnpb7Qy5wzq4pffPoMNE2j38tJn83ZfOTNhxGPGpy2ah5/e2IbAIvm5LPVLZxTi2U7pDJWgee6clEDJxw6m5sf2syDa3fR1p1mgfe2sLOtn3jMwLQcLjn/MH54w9pAWjjj6Pnc+cRWbnlkCzvb+oPzEMZ/UC2cXRPILy0NCda/2oGmwaLZeY91zqwqHnjOTbLmTxKbM6uKT7/raK689mn2dAxw8uFzWdBSTUNNjDNXLxiUZiBi6EFkytymKjZs7mDTzm4iho6uaRy5zDXyzQ0JDF2jrTvNkpB8F8b35MOSEsDJh8/ljse3sWZTG4auBfmHwlFa0YjO6hXNPL7B0/TrE4Fc84d7Xx60r1JzKopprk9M2L2rqFwq1sg31sZZNKc28OiLWX5QPd/68IlDShKj5eNvXxUk4AKoq3a9wcv+3ypa20ob+WhE5/uXn0IqY1GViKBpGnOb3Fzhbz1tKW84cVGw7crFjfz8U68JBo7D+AZl7iz3u+88e3kQKfSOMw/mtcctRNe1gpDOc45fxOKWamzboaEmVtDW+9+wkq17+7Asm/ecewh/e2IbjuPwnnMP4Z6ntnNQczUrFjZw8hFz2bY3L3Gcvmo+f/cM84fPP4yVRZ61v5/PvveYQBb6+IXueYtFC7Xnd529nFhE556ntrM4VFA9EYvw0QuO4Fu/e4ZDFtZj6Drf/egpg3IUgZuvfp73gPqEd3227u3ld3dJFs6uCfbXUBPnvz5+OpmcRU2y9M9i8dxa6mtiLClyHCKGztc+eDz9aZN41CgroVzypsN4x5nLiRgatVUxUhmTZNzg4AX1vOvsQ8iZFtfd/RJdfZmSg/TF/OuFRzJxd6+iUtFGEiY3RSwBNo8lnzwMzsNsWjaaRkmDONWMNke047jpZkfyQy9FOmuWTKEw2n5lcxa6FxXka8XRiIFl29i2E/ztOPlMn+3dab706yc47ch5vPe1K8bU/+J+5UyrZARJOmsGUlE5yt0H7jEw4hq+fr927uoiEtFLevpjIecNeofDVkd77WdqbnSYuX2rpH6F8skvBbYUr69YT36sBnImoGnaoAlco2EkBn4khLNmho2soev4p7fYeDbVJ7jq8lODgcmJoFyI4EiOs9x9ED6G0VCcSXS8FD9kxnvtFYpi9l9LqJixTKSBVygU40MZeYVCoahglJFXKBSKCmZS3quFEO8FvgjEgB9KKX82GftRKBQKxdBMuCcvhFgAfBM4DTgK+IgQ4rCJ3o9CoVAohmcy5JpzgfullB1Syn7gRuDtk7AfhUKhUAzDZBj5+cDu0OfdwEGTsB+FQqFQDMNkaPKlgnzLJxnPYwB+UP+YaGkZnMRppjBT+6b6NTpUv0bPTO1bBfar5CSOyfDkdwLhVIDzgF0j+N74M4QpFArFgUtJGzoZnvy9wFeFEC1AP3Ah8JERfO8p4HRcead8vlqFQqFQhDFwDfxTpVZOSu4aL4Ty87ghlL+SUn53wneiUCgUimGZSQnKFAqFQjHBqBmvCoVCUcEoI69QKBQVjDLyCoVCUcEoI69QKBQVjDLyCoVCUcEoI69QKBQVTEWU8JlJqY2FEF8B3ul9vF1K+e9CiGtwJ3r1e8u/JqX8yxT3635gDuBXGr8UOJhpPG9CiEuAj4UWLQV+B1QxTedLCFEHPAqcL6XcIoQ4F/gBkAT+JKX8orfd0cDVQD3wIHCZlNKc4r59BPg44ABPA5dKKbNCiC8DHwI6va9ePZnXtkS/St7v5c7lVPQLOAz4Vmj1AuAJKeX5U3m+ytiHSb3H9vs4eS+18cPAsUAG96K+R0q5cRr6ci7wNeAs3B/e34CfAlcCr5NS7h7i65PZLw033cQi/yaZSefN68/hwM3AycDfmYbzJYQ4EfdHtRJYAewFJHAGsB24HfiRlPJOIcR64BIp5eNCiF8DT0spfzGFfYt5/TkW6AWuBdZIKX8ohLgV+JaU8rHJ6k+5fnlG/nmKrp8QIkmZczlV/Qqtmws8ArxBSvnyVJ2vMvbhV8B3mMR7rBLkmpmU2ng38GkpZVZKmQNeABZ5/64WQqwTQnxNCDHV513g3lR3CiHWCiE+xsw6bwC/wJ0lnWL6zteHgcvJ51o6AXhZSrnZezheB7xDCLEYSEopH/e2uxZ4xxT3LQN8VErZI6V0gOdxzxvAccBnvfP3UyFEYqr6JYSopvT1K3kup6pfRXwP+KWU8mXv81Sdr1L2YQWTfI9VgpGfMamNpZQb/IsihDgEeBfu0/p+4J+Bk3BfYz80xV1rBO4DLgDOAS7D/SHOiPPmeThJKeWfcSWlaTlfUspLpJQPhRaVu7em/J4r7puUcquU8l4AL0/Ux4C/CiFqgOeAK4BjgAbgS1PVL8pfvyk9ZyX6BQS/yzOBH3ufp+x8lbEPNpN8j1WCJj/W1MaThic93A5cIaWUwNtC634CXIz7KjkleK+h/qtov/fq9wPcCl5hpuu8XYrbH6SUrzLN5ytEuXtrxtxznux2J/BrKeUD3uI3htZfBVwDfGEq+jPE9ftzic2n45x9BPi5lDIDIKXsY4rPV9g+4I6RiaJNJvQeqwRPfqypjScFIcSpuF7z56SUvxVCHCmEuDC0iUZ+8HOq+nSaEOKcoj5sYQacNyFEDFePvMX7PO3nK0S5e2tG3HNCiJW42vJvpZRf95YtEkL8c2izKT1/Q1y/GXHOcN9m/+h/mOrzVWwfmIJ7rBI8+bGmNp5whBALcQcP3yWlvN9brAE/8qJb+ry+/XaKu9YAXCmEOAWIAu8HLgKumwHnbRXwkjcuADPjfPk8AQghxHJgM/Be4Bop5VYhRFoIcaqU8hFcT3VSBhDLIYSoBe4GPi+lvC60KgV8Vwjxd9wH+eXAVEZylbt+Jc/lFPYLIUQzriy4ObR4ys5XGfsw6ffYfu/JSyl34r5a/R1YA1wvpXxymrpzBZAAfiCEWCOEWAOcAnwb1+PaiBsB8Yep7JSU8jbc18PngGdwb6JHmBnnbRmww/8gpVzHNJ+vUF/SwAeAm7y+vIg7QA3wPuCHQogXgGo8jXcKuQRX/77Cv9eEEFdKKVtx5a9bcaNZNOCqqepUues3zLmcKgruNYApPl+l7MMHmOR7bL8PoVQoFApFefZ7T16hUCgU5VFGXqFQKCoYZeQVCoWiglFGXqFQKCoYZeQVCoWiglFGXqEogRDiDiHEYaP8zm1CiA9MUpcUijFRCZOhFIoJR0r5xuG3UihmPsrIKyoKIcSbyefIH8CdgPJ64HDcaeJzcCd/XSKl7BFCfBQ3YVsWSOPmZN8ohNgCvF1K+XQob7uFm374Y1LKl4QQ83Fnc84HtgKzQ/04FPgvoAkwgB9LKa/xEmL9BjgENxfJM94+pzXfkqJyUXKNomLwMvt9C3ijlHI17pT6/8OdLXgSbirllYAJfFkIYQA/ws0rfjzwP8BpRW2eDfw7cJaU8ijgeuBmL0f/z4DHpZSH4z4EVnrfieDOWvyclPJY3Nw8VwghTsJN3lUrpTwaON7bzbKJPxsKhYsy8opK4rW4iZzu86aM/x7XW14O/FlKudfzmH8NvF5KaeFmR3xUCPFToNtbF+YNuNV6WgGklNfiVhVagpuT/1pv+SbcFLvg5gg/GLjG68c/cKv+rMYt1HK4EOIB4HO4BSI2TeA5UCgKUEZeUUkYwH1SyqP9f7ge/Hpc791Hx5VekFJeBLwZ2AR8Ftfzp2jbYjTcRG8OhSlh/X0YQFeJfvzGS461HDe/Sx1wrxBiOou1KCocZeQVlcT9wOu8FLwIId4IrMNNCvVWIUS9V6Xow8CtQohmIcR2oF1K+SNcLf+oojbvAt7lZetECPFBoB33ofA3vMydQohFuGXdwE10lRZCXOStW4j7oDnWGwP4DXC3lPKzXvtHTPiZUCg8lJFXVAxSyg24RvePQoi1wNeBt+CmUt4L3IFbcq0bt6ZnG/ANXHnnGeA/cbM7htu8B/ghcL8QYgNumubzPdnncuAwL0vgr3EHdJFSZoG3ApcIIdbhpgT+kpf5839xPf2NQoincb35/5qcM6JQqCyUigMAIcRXgWYp5cemuy8KxVSjPHmFQqGoYJQnr1AoFBWM8uQVCoWiglFGXqFQKCoYZeQVCoWiglFGXqFQKCoYZeQVCoWiglFGXqFQKCqY/w98IwcRdZl1qQAAAABJRU5ErkJggg==", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAEXCAYAAABI/TQXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABSk0lEQVR4nO2dd3zT1frHP9lNmnRByyx7XdFaFQWZllnahl0BZclPRWSIomyECxQLci+IAy94FW9BS2WUISACggIOZgFHQaBAodCdZjXre35/hISWrrRN0iQ879eLF02+3+85T06++eTJc57zHB5jjIEgCILwSfh1bQBBEAThOkjkCYIgfBgSeYIgCB+GRJ4gCMKHIZEnCILwYUjkCYIgfBgSeRcxceJE5Ofn1/j6BQsW4OLFiwCA+fPn48SJE84yjaghWVlZiIuLw6BBg3D27NlSx8aOHYvevXtj8ODBGDx4MGJiYrBo0SJoNBr7OWazGevXr4dSqYRSqURsbCz++c9/oqCgoFQ7Y8aMAcdx9ufy8/PRvn37cm06duwYoqKiMHz4cBQXF9f4tRUVFWHZsmVQKpUYPHgwhgwZgm+++aZGbZW890uOy5AhQxAbG4vZs2dDr9dX2sb27dsxadKkGvVfHhkZGXjhhRcQExODESNG4MqVK05r2+NhhEto164dy8vLq/H1UVFR7Pz58060iKgtO3bsYOPHjy/32JgxY9i+ffvsj41GI3v33XfZpEmT7M/NmDGDTZ8+nRUWFtrPWb9+Pevfvz9Tq9X2dh599FH28ccf26/Ly8tj7dq1K7ffOXPmlDq3JhQXFzOlUsnWr1/PTCYTY4yxzMxM1rdvX5aSklLt9kre+w+OC8dxbNq0aSwxMbHSNrZt28ZeffXVavddEcOHD2e7du1ijDF25MgRFhMTwziOc1r7ngx58i5g7ty5AIDx48cjKysLd+/exZQpUzBs2DAolUp8+umnAKye3aJFi6BUKjFs2DBMnz4dWq0Wq1evRnZ2Nt5++22kpaVh7Nix2L9/PzIzM9G3b18sXboUI0aMQL9+/bB3714AgF6vx6xZszBgwACMGDECc+bMwZw5c8rYZjab8d5772HAgAGIiYnB/PnzYTQa8eGHH2LJkiX280o+Hjt2LKZOnYqYmBh8+eWX6Ny5M4xGIwDAYrGgZ8+e+Pvvv6FWqzFnzhz761y+fDnMZnMZG9RqNd5++23ExcVBqVRi5cqV9vMee+wxfPjhhxg1ahR69+6NjRs3ljvGaWlpiI+PR1xcHIYOHYqff/4ZAPDII49gxYoVGDZsGKKjo3HgwAEAZT3DyjzFLVu22D32iRMn4tq1a/jll1+wZs0aXLhwAWPHjq3gnb+PSCTC3LlzcfLkSVy5cgXnz5/HyZMnkZiYiMDAQPs5r7zyClq1aoWvv/7afu3rr7+Ozz//HOfOnau0j88++wyHDh3C119/jRUrVsBkMmHp0qWIiYmBUqnE/Pnz7b8kevfujRkzZmDgwIH4/vvvS7Wzd+9eyGQyvPLKKxAKhQCAJk2aYM2aNWjbti0A4IcffsCoUaMwbNgwPPfcc1izZg0A4Ndff8WgQYMwatQoDBo0qMy9/yA8Hg+dO3fG1atXAQCnTp3C888/b/8M/Pjjj2WucfS+2rJlS6n39MqVK+jRowfu3r2Lq1evIjY2FgDQq1cv6PV6/PHHH5WOr69AIu8C3nvvPQDAl19+iUaNGuGdd97B8OHDsX37dmzduhUnTpzA3r17ce7cOfz222/YtWsXtm/fjvDwcKSnp+PNN99EWFgYVq1ahccff7xU2zdv3kT37t2xdetWvP3223j//fcBAJ988gksFgv27duHjRs3VngDf/XVV/j999+xc+dO7NmzB1qt1v5FURkBAQHYu3cvxo8fj7Zt2+Lw4cMArOGCJk2aoE2bNli+fDk6duyI7du3IzU1FQUFBfjiiy/KtLVs2TIEBQVh9+7d2LZtG9LT0/H5558DAIxGI4KDg5GcnIy1a9fiX//6FwwGQ6nrTSYTpkyZgilTpmDPnj1YunQpli9fDo7jYLFYEBgYiO3bt2PNmjWYN29etcJmP//8Mz777DP873//w65duxAXF4cpU6agc+fOmD59Ojp16oSkpCSH2vLz80OLFi1w6dIlnDlzBo8++iikUmmZ87p164YzZ87YH7ds2RKzZs3C22+/XSrc8yAvv/wyevfujQkTJmD27NlYt24dsrOzsXPnTuzcuRMcx2HlypX289u2bYt9+/ahX79+pdq5ePEinnzyyTLtd+zYEZGRkWCM4fPPP0diYiK2b9+OLVu2YP369fZxvXz5Mv71r39h165dZe79B1GpVNi3bx86d+6MgoICTJ8+HfPnz8fu3buxYsUKvPPOO7h582apaxy9r2JjY3H69Gnk5OQAsH6RDxs2DFlZWQgLCwOff1/uGjRogDt37lQ4tr6EsK4N8HV0Oh1OnjwJlUqFDz74wP7cX3/9he7du0MgECA+Ph7du3fHgAEDEBERUWl7IpEIvXr1AmD1WgsLCwEAR48exdy5c8Hn8yGXyzF06FCkp6eXuf7EiRMYPHgw/Pz8AMDukX344YeV9tupUyf73/Hx8dixYweio6Oxfft2xMfHAwCOHDmCCxcuYOvWrQBQYYz4xx9/xNdffw0ejwexWIxRo0bhyy+/xKuvvgoA6NOnDwCryBiNRuh0OkgkEvv1ly5dAp/Px3PPPQcAePTRR7F792778TFjxgAAOnTogHbt2uHkyZOVvraS/PTTT4iJiUFISAgAYNiwYUhISEBmZqbDbZSEx+OVK+wPUjIGDwDPP/88jh07hsWLF2PevHkO9fXjjz/izTffhEgkAmD9BTZlyhT78ZLv4YM2skqqm/B4PHz66ac4cuQI9uzZgytXroAxZo+rN2rUCE2aNKnw+pUrV2LdunX2PqKiojBu3DgcP34czZo1szsybdu2xZNPPonffvsNPB7Pfr2j95VcLseAAQOwa9cuTJgwAbt27cJXX31lF/0HEQgEFdrsS5DIuxiO48AYQ3Jysv3Dnp+fD4lEAn9/f+zcuRNnzpzBL7/8ghkzZmDcuHGYMGFChe2JRCK7R1LygyAUCkt9UEt6LSWx/Ry3kZubC47jynzQTSZTqfNkMpn97+joaLz33nu4cuWKPQRhe60ffPABWrduDcA6mVfSxpJj8uDjkj+/bYJuu/ZBARIIBGXavXTpElq1amU/XrJt2/mVvT4b5YkdY6zc8EBV6PV6XLlyBW3btkVISAg2bNgAvV4PqVQKo9EIrVaL4OBg/PLLL4iMjCxz/dKlSzFo0CDs2rXLof7KG9eSr7Pke1iSyMhIbN68uczzhw4dwqlTpzBt2jQMHToUffv2RadOnTB8+HAcPHjQPlYVtWtj1qxZiI6OrtJe4P5Y276obOeVd18dOnQIa9euBQCEhYVhw4YNiI+Px8KFC9G6dWu0adMG4eHhEIlEyM3NBWPMft/cvXsXDRs2rNRuX4HCNS5CIBDAbDZDLpcjMjLS/vOyqKgIo0ePxqFDh/DDDz9gwoQJeOKJJzBt2jQMGTIEf/31V6nrHaVXr17Ytm0bOI6DXq/Hnj17yhXYZ599Fnv27IHRaATHcVi8eDG+/fZbBAcH4/fffwdjDDqdDseOHauwL4lEgtjYWMyZMwf9+/e3f3l1794dGzduBGMMRqMRkydPxqZNm8pc3717d2zevNl+XkpKCrp27erwa23VqhV4PB6OHz8OAPj9998xfvx4u2ikpqban7927RqefvpphISE4PLlyzAYDDCbzfjhhx/Kbbt79+7Yu3evPRSxbds2BAUFoXnz5g7bB1i9zeXLl6Nnz55o0qQJIiIi0LlzZ8yZMwcqlQo3b97Eiy++iGnTpiE9PR0vvvhimTYCAwPx/vvvY/Xq1Q712aNHDyQnJ8NkMoHjOGzevBndunWr8rr+/ftDo9Fgw4YNsFgsAKxhwcTERLRu3RrXr1+HRqPBjBkz0Lt3b/z222/2+6c8HL13H3/8cVy7dg3nz58HYA37nDx5Es8880yp8yq6r/r06WMPTW3YsAEA7F+WH3/8sf0XZsOGDdGsWTN7WPKnn34Cn89Hu3btqrTRFyBP3kX069cPL7zwAj755BOsWrUKS5cuhVKphNFotE/qWSwW/Pjjj4iLi4NMJkNgYCCWLl0KAOjbty/efPNNLFu2zKH+Jk2ahCVLlkCpVEKhUKBevXr2kExJRo0ahVu3bmHYsGFgjOGZZ57B2LFjodfr8dNPP6F///5o0KABnnjiiUp/wsfHx2PTpk1YvHix/bn58+cjISEBSqUSJpMJXbt2xcsvv1zm2gULFtjT9UwmE3r06IHXXnvNodcJAGKxGB9++CGWL1+OlStXQiQS4cMPP4RYLAYAnDlzBikpKeA4DqtXr0ZgYCC6deuGp59+GgMHDkRoaCg6d+5cbjirW7dumDBhgv1LIyQkBP/5z38q/GVUEltYgs/nw2w2o2vXrpg/f36p459//jnGjBkDxhhMJhMEAgH8/f1x6NAhDB06tEybzzzzDCZMmGCfrK+MyZMnY8WKFRgyZAjMZjMiIiKwcOHCKq8Ti8X44osv8P7770OpVEIgEEAgEGDy5MkYNmwYOI7Dc889h4EDByIgIADNmjVDmzZtcP36dfuYl6TkvV8ZISEh+OCDD7B06VIUFxeDx+PhvffeQ8uWLUulqDp6X9mIj4/HJ598gr59+9qf+/e//42FCxdi3bp1EIvF+OCDDxx6T30BHqvsk0x4Dd9++y3kcjl69eoFjuMwbdo0dOvWDS+88EJdm+ZW2rdvj59//tkeU/cGioqKcPHixWr9miEIRyGR9xEuXbqEd999F3q9HiaTCZ07d8a8efNKxTYfBrxR5AnClZDIEwRB+DAPR1CKIAjiIYVEniAIwochkScIgvBhSOQJgiB8GI/Lky8o0ILjajYXXK+eHHl5Fdf6qGvIvtpB9tUOsq92eKp9fD4PwcH+FR73OJHnOFZjkbdd78mQfbWD7KsdZF/t8HT7yoPCNQRBED4MiTxBEIQPQyJPEAThwzgk8h999BFiY2MRGxtr34TgxIkTUCqV6N+/f6kqeX/++SeGDx+OAQMGYP78+TUq0UoQBEE4hypF/sSJEzh27Bh27NiB1NRU/P7779izZw/mzZuHTz75BHv37sXFixdx9OhRAMA777yDhQsX4rvvvgNjDCkpKS5/EQRBEET5VJldExoaijlz5thLirZu3RoZGRlo3rw5wsPDAQBKpRL79+9HmzZtUFxcbK/pPGzYMKxdu9YtlRBtW79ZOIvL+6opvm6fgF93O+0wxlDXeQ8cx8DVYSkofjn7B7gLZ4z/wz5+5e3/4AyqFHnbRr4AkJGRgb1792Ls2LEIDQ21Px8WFoa7d+8iOzu71POhoaG4e/euk00ui7qwENots6HlGao+uQ65XtcGVEFt7DMzPlT/GIoWPZVOs8fhvi0c5vznZ+QXefb770p4AMYP7ICejzeuk/6XfnkKGXfUddK3s4jr2hzDerZ2e78cx/DvlHOIaFUP/Z9p5vT2Hc6Tv3z5MiZNmoTZs2dDKBTi2rVrpY5XtE9kdb+d6tWTV+t8AAgO8sNv7WNRoFFV+1qi9jAGFN+4iDZ/7YD8qccgbfGYy/oKDVWUeS47X4f8IgOefawRWjYOdFnfnsyOI38jq0Bf7viUpKrjNcFk5pBxR43ItqF4pFU9p7fvDtKv52Pvz9cR3bUVmjcKqPA8V4zfgV+v44+MAsT1aO2S9h0S+dOnT2P69OmYN28eYmNj8dtvvyE3N9d+PDs7G2FhYWjQoEGp53NychAWFlYtg/LyNDVacND2uUEIDVUgJ8dzvQlftu+Nfx3AbNkB3Nm6CrKhi8APCK36ompSkX0ZWUUAgE7t6uOJts7v11Hq8v398Uwm7uZqK+3fVfblF1k31n68dQh6Rdb8l0Rdjl+XDqH4KyMf67aew1sjI8t1Tl1hn95gxpff/oE2TQLRvnHN2ufzeZU6x1VOvGZlZWHKlClYtWoVYmNjAdzfm/H69euwWCzYs2ePfS9LiUSC06dPA7DutdmzZ89qG014H2KpHEeDhoAxDvoDa8FM7gudaPTWzaoVsrJb0T0sBMrFUGmNddK3rd9Af0md9O8M5FIRBnVvid8zCnD+Sp7b+t37y3UUaY0Y3bdt3cXk//vf/8JgMCAxMdH+3KhRo5CYmIhp06bBYDCgV69e9t3YV61ahQULFkCr1eKRRx7BuHHjXGI44VkoZCJkmUSQ9pkM/f5/o/jIBvj1neKyG7ckap3RbsPDSqC/GHfytHXSt13k5d79JRv1RBP8cOYWthz+Gx1bhkAocO0yotxCPb777Sae7dgALSsJEdWWKkV+wYIFWLBgQbnHdu3aVea5Dh06YOvWrbW3jPAq5DIR1DoThOGRkDwTD8OvKTCe3Q3Jk4Nc3rdad8+Tlz7cIq/SGl2apVERRXZP3rtFXijg4/nebbB263kcOXsLfTuFu7S/rUevgM8Dhvdy7WQvrXglnIJCKobmnkctihgIYZsuMJ7aAfP1cy7vW60zQcDnQSrxuHp7biNQLoHZwqAtdv/iw0KNNTQX4OUiDwCPt66HR1oEY+exa/YwoCv4O1OF3/7MRnTnZggJ8HNZPwCJPOEkFDIR1Pc+FDweD349J4Jfvxn0hz+FpeC2S/vW6I2Qy0Ru92A9CZsXXRdxeZXWCLlU5PLwhjvg8XgY1bstdAYzdh2/VvUFNYBjDF8fuoQguRgDOzd3SR8l8f53hfAIFDIRjCYOBpN1MRVPKIa0/3TwhGLoD3wAZnBdvFitMz3UoRrgvsgXady/VqBIY/T6UE1JmobJ0evxxvjhzC1kuWCe49ff7+JalhrDe7WGROz6BYQk8oRTsGW22CZBAYAvrwe/vlPAinKhP/wfMI5zSd9qnemhzqwB7k961pUn7wuhmpIM6dEKYhEf3/xwxantGkwWbD16BS0aKvDsow2d2nZFkMgTTsHmST8YxxQ2ag9JtzGw3DwP40nXTMirdcaHOrMGqOtwjcHrM2seJMBfjLhnW+Dc37n4PSPfae1+9+sNFKgNGNWnrdvKKJDIE07hvidfdrJK/EgURP94Dsa0vTD9/YvT+9boTVBIfUtkqoIxDqxYA0vhbZiz0iG6cxEKoQkqjXtFnjEGlcaIIC/Oka+Ivp3CUT/QD1sOXXbKjlAFagP2/nodnTqEoV14UO0NdJCHNx2BcCo2T7pkuKYkkq5jwBXcRvHRz8EPagRBfedMOJktHLTFZsi93JNnFjNYsdr6T6++/3dFjw0aaz2JEryhCMLxotEA2rjN7mKjBUYz53PhGgAQCfl4PqoNPkm9iB/P30Z8v9rlsm87egUcxxD/nHvr45DIE05Bbhf58tPOeAIh/PpOgW7HP6E/sNZa+kBa+wUgWvtqV88TeWY2gumL7v1TgdMXgelUYHrV/ed11udh1FXQCg88P/m9fwrwAxuB17AdeH4K6z+p9X9m1CPw4Hr0yk0Gp20Hvn+wW16jryyEqoin2oeiXdNApP54FbE9ai7O17KKcOLiHQzs0gyhQVInWlg1JPKEU5BJhBDweZXmFvNlgZD2nw7drgQUH/wY0th3wOPX7ha0L4Ry08SrLUzCdCowXYFVpLUFYLpC3OV0KC7MB6dXgemKAJO+/EbEUvCkgeBLA8APaQqBNBA8WUAJ4bb9LQdPIgeP71hU9WBALvoWbYduTyJksbPBl4c48ZWXj+peNo8vZdeUhMfjYVTftli68RRSDl5CbOfqV4lkjCH50GUEyESIe7aF842sAhJ5winweDzIpaIKwzU2BKEt4NfzJRT/sB6GE1/Br3vtyl7YcvNrm0JZRry1heB0hWC6QquQ33uO6VVAeTX3xTJAEQyI5RDUaw5e0wCrWMusYs6TBlofSwPAE7pGEA3BrfB5zgC8pjsI3e73IFPOAV/u2qqQKh9Z7VoZLRoGoOujDbHzx6t4un0owqrpiZ9Kz8HlTBXGRbevkwV7JPKE01DcK21QFaK2XWHJuwHT+f3g128OcYdeNe7T9qVSWUyeMc4ax9bmg9Pmg2nywbQF9r85bQGYtgDgylktKvEHXxYEniwI/MaNwJcFgucfDJ4sEDxZsPWxLAg8objOq4wG+YtxRBsCyYiZMOz/t1Xo42aDr3BdZU7bRG+g3PcmXksyrFdrnLqUg60//I3XhzpeSttktuCbH/5G01A5ekbUTa1/EnnCaVg9eceWgkueeR5cfiYMx/4HQVBjCBq2rfqiclBrjfDnFUOhz4Ipo8hxAecLwfMPBl8eAkGDNuD7B98T73uCLguyCrmLvG5XEHAvLq7xD0dg7DvQ7V0F3e5Eq9AHVK/kt6OotEYI+Dz4+/m2lAQrJBge1RZfffcXLt0sdDg75vtTmchVFePtUZHgwQJOXQBOkw+myQOnzrXeo5pcMG0BxI8PhKhdd6fb7tvvDOFWFDIxbmRrHDqXx+dD2mcytDuWQP/9h5ANXVxuDJkxZp2g1ORCk6OF4VYmmCb33gckF08W5uCZYBOwHyi2XVSegMtDwPMPAd8/xPq3nxw8nm9lEJfMlQ9p1AqyuFnQffu+Xejhgg0pbDnynl5SgjFmzUxSW+8d2/3DqXOtz2lyAQbwJDLwJDJALANPLANP4n/vfxn6BQbiTsB1nPo+Gy37Pwa+nz94Yn/r+SIpYDaA0+RZ29Xkozg/G/K0S5gXVoyGx/dA831BmYwonp8CPEV98AMbgueiL2ISecJpKGQie5EyR+BJ/K0TsTuXQv/9hxA/1h+cOs/+obP+nwdYrL8O7PknEn/w5fXBD2yE65Ym+DOXjxFxz4Avrweev28KuCPY6rnb4uSC+i0gi50F/bfvQ7cnEaZxSwA4V+hVWs8oaVCliKtzAcsD96bEH3xFfWtKb/hjAI8HGHVgBh2YUQemLwJXmAVm1AFGHYyMYbAQgBko3ru/apvARzhfhuDgxhAEtwRfXh88eQj48nr2v93xS5FEnnAacqkI2mIzzBbO4WJVgpAm8It6FcUH1qL48H8AlPBuQppC0DzSKuiK+qjXrBkKTVLwxPcnvn5OvYgbYg1Gt3jKJa/JmwiylTYoUb9GUL85pMrZ0O9ZidtJ78IvZhb4QY2c1qdKY0Q9F1dRtME4DkybB64oB5zqLriibLCibHD3/sH8QN2eB0Scr7DeRzxFfavIih2fQGWMoX6gEHdv3cWGradg1Knx6oCWEHLFYAYtmEEHnkgCnrw++PIQZBX7YXFyOvo81Qyj+9YsFOksSOQJp2FLY9TqTdWaiBO1eBKCUSvBLCbrh09U/rXiUAV4D0xsUkmD+9jG/8HSBoKQcEjj5sCwbyV0u9+DNG42BMFNnNKnSmtEq8bO2/CCWUxWz7vobikx54qywdQ5pTObBELwFWHgBYRB1Pgf4AeE1ljEq4LH44EvkUGoqI9+/Z7F8k2n8V1WCIb0aFX2NTCGr5PPQSoRYVD3Fk6zoaaQyBNOw77qtZoiD6DGE4NqvanaKW2+ikjIh7+fsNz6NYKQJmg0Zglu/e9d6PesgDR2FgQhTWvVH8cxqHXVD9cwxlknxwuz7v27A06VhRuaHJhVuQBKxK1FfuAHNIAgpCn4LZ8CLyAM/Hv/eP7BdRKWa9M0EM/8Iwz7f72Bno83LlMPPu3vPPx5vQAv9G0Lf7+6d0BI5AmnUVn9Gleh0ZnQunGg2/rzdALlEhRVUL9GXL8pZMo50O1ZcV/o69V89yO1zgjGKs6RZyYDONWdEmKede/xndLxcZEU/KCGkIb/A8bWwVYRD2wAXkCYNXTngZO6I55rjTOXcrH16BW8quxof95s4bDl8GU0qifDc08459dSbSGRJ5yGbUFSVQuinAVjzFqcjMI1dmzbAFYEP6gRZMq50O1ZcW9l7Kwa1xGy9sMQLDLCfOsPcAW3wanueeaFWWDaktUbedYwSlAja2glsCH4QY3AD2poXSjG49X5OoPqUD9QigHPhOPbn6+j71Ph9pDV4TO3cLdAjxnxER6ziYpDIq/RaDBq1Ch8+umnaNq0KbZv347PPvsMAoEAnTt3xpw5cyAUCnH+/Hn885//hNFoROPGjbFs2TKEhrpuIQbhWSiqqF/jbHQGMywce+g3DClJoFyMvzNVlZ7DD2xg9+h1366ELOYdCEJbVNk2M2hhKbgFLj8TXP4t+N3OQEJQJuQ/G2Av4CDys050Nmp/T8Tv/QsI86o1B44Q06U5jp3PQvKhy5g75kloi83YdewaOrYMwWOtXLvSuDpUKfJpaWlYsGABMjIyAABXr17FmjVrsHXrVoSFhWHx4sVISkrChAkTMH36dCQmJqJLly7Yu3cvFi5ciE8//dTVr4HwEPwrqCnvKtxdt8YbCPQXo8iBDb35AWElhH6FVejDrJOIzGQAV3ALXMEtWPIz7X8zbcH9BkR+4CRhOG9shme6RELRpCX4wY3tXvnDgFQixNCerbBx3184+Vc2LmeqoDeaMbJ3G48agypFPiUlBYsWLcKsWbMAAOnp6YiMjERYmHWiLCoqCuvXr8fgwYNRXFyMLl262J+fNWsWjEYjxGL6ED4MCAXWiT93hWs0Os+tQFlXBPpLYDRzKDZaqqyTwleEQhZnE/r3IWjU3irm6pz7JwlE4Ac3hqDxIxCENAE/uCn4IU3A8w/Bj79cx7YbVxH1RC8IRa7fxs4T6f5YIxw+nYmvD16GWmdCr8gmaBoqr2uzSlGlyCckJJR63KFDByQmJiIrKwthYWHYv38/cnNzERwcDJlMhmPHjqF79+749ttvYTKZUFBQgAYNGrjsBRCeRXVKG9QWR+rWPGyUXPXqSDEsvqI+ZMo50B/8BEydA0FoS/Db9wA/pAkEwU3AU4RVWAVTpbH2IX5IBR4A+HweRvZpi/e/PgupRIAhPVrWtUllqPbEa8uWLTFz5kxMnjwZfn5+iI6OxoULF8Dj8bB27VqsWLECq1atwuDBgxEUFASRqHofwHr1avctGOqCpdvOxNftCwmUwmDmXPY6S7bLrlgn9po3DUZosMwl/VWXun5/mzWxRsd5QkG5tpRrX6gCeGVltfsqNnMICfBz6muu6/GriorG9K6qGE3DFGjd3HNi8TaqLfIGgwERERFITU0FABw4cADh4dY0LKFQiKSkJABAYWEhPvnkEwQFBVWr/bw8TY232vL02fmHwT4/ER85hXqXvM4H7cvKLgIAmPRG5JjLKf/rZjzi/b03DtdvFaJBQOm1Cs62LztPC7mf0GltesT4VUJl9j0XYV1FXBf28/m8Sp3jauf46HQ6jB8/HhqNBkajEUlJSYiJiQEAzJs3D+fPnwcAfP7554iOjgbfwQ0PCN/A0XLDzkCtM0EiEjzU4YIHsS1Cc8deryqt0Wd3hPIlqu3JBwcHY+rUqRg5ciTMZjPi4uKgVCoBAIsXL8aiRYug1+vRvn37MvF8wveRS8XQ6E1VZnc4A7XOBDmlT5ZC5mfdoauyXHlnYS1O5tt15H0Bh0X+8OHD9r/j4+MRHx9f5pyIiAjs2LHDOZYRXolCJoKFY9AZzC5f0q3WU92aB+HzeAjwF0OlNVR9ci0wGC0oNlrIk/cCKJZCOBWb6GrcELJR60yUI18OVa16dQYqne9v++crkMgTTsWd9Ws0OippUB5BconLY/K+voG3L0EiTzgVuRvr16j1RorJl0OAOzz5h2RvV1+ARJ5wKiXLDbsSg8kCo4kjT74cAv3FUOuMNU5FdgTblwh58p4PiTzhVO6Ha1zrSdrap5h8WQLlYjDm2vdApTWCz+PRLykvgESecCrWvHW+y2PytiJo5MmXpWRpA1dRpDVA4S8Cn+85hbiI8iGRJ5yOwg31a+wVKKXkyT+ILU5e6MLJ10KNZ2zgTVQNiTzhdOQyMdR6d4VryJN/kPuevOty5VVaI4Jo0tUrIJEnnI5CJnJ5njyVGa6YgHsiX+TScI3R3g/h2ZDIE05HIRW7PlyjN0HA5zlUTvdhQyISQCoRuCxXnmMMRVoK13gLJPKE01HIRG4J18ilIo/agceTCPCXuGziVas3wcIxEnkvgUSecDoKmQhGEweDyXXlf9W02rVSgvzF9lWpzoYWQnkXJPKE07HlrrsyLq/WU92aygiUu27VKy2E8i5I5Amno7CVNnBhyIbKDFeOK0sb2LJ2qAKld0AiTzgd256rrpx81eiozHBlBPqLUWy0wGB0fsiMPHnvgkSecDquDteYLRy0xWYK11SCLYdd5YLSBiqNERKRAH5iymzyBkjkCadjL1Lmotop2mJzqX6IstgXRLlg8lVF6ZNeBYk84XRkEiH4PJ7LKlHavjwoJl8xAXaRd4Unb6B4vBdBIk84HR6PB7lM5DJP3l63hsI1FWLf0NsFk6/kyXsXJPKES1DIXFekjOrWVI1CKgKP5xqRL6INvL0Kh0Reo9EgLi4OmZmZAIDt27cjJiYGSqUSy5Ytg9lsjZFmZmbixRdfxODBgzF27FjcunXLdZYTHo1CKnJZuOZ+mWHyJiuCz7du6F3k5CJlJrN10juAwjVeQ5Uin5aWhtGjRyMjIwMAcPXqVaxZswYbN27E7t27YTabkZSUBAD44IMPEBsbi507d6J///5YvXq1S40nPBeFzHX1a2zt+vtRdkdlBPqLnV5u2J4jT+Ear6FKkU9JScGiRYsQFhYGAEhPT0dkZKT9cVRUFA4ePAgA4DgOGo0GAKDX6+Hn5+cquwkPRy4TQeOymLwR/n5CCAUUbayMQBfUr7G1F0SevNdQpSuUkJBQ6nGHDh2QmJiIrKwshIWFYf/+/cjNzQUAvPHGGxg1ahSSkpJgMpmwZcuWahtUr5682teUJDRUUavrXc3DYl/D+nJoi28hJMQfAieKcWioAkYLQ5BC4pFj6Uk2Najnj9t52lI21da+K3etTlzzpsEuea2eNH7l4en2lUe1f++2bNkSM2fOxOTJk+Hn54fo6GhcuHABADB79mwsWbIEffv2xXfffYepU6di165d1aoUmJenqfEGxKGhCuTkqGt0rTt4mOzjM+t7eO1mgdN+2tvsyyvUQyoRetxYetr7KxHyUKg24G52Efg8nlPsu3lbBQDgjGanv1ZPG78H8VT7+Hxepc5xtV0sg8GAiIgIpKamIjk5GY0bN0Z4eDjy8/Nx9epV9O3bFwAwYMAA5OTkoKCgoObWE16LKxdEqXVGe30comIC/cWwcMw+Ue0MCjUG8ECZTd5EtUVep9Nh/Pjx0Gg0MBqNSEpKQkxMDIKDgyGRSHDq1CkAwOnTp+Hv74+QkBCnG014PvYiZS6YfKUyw45hy5UvcuLka5HWCLlMRPMhXkS1wzXBwcGYOnUqRo4cCbPZjLi4OCiVSgDARx99hKVLl6K4uBj+/v748MMPnW4w4R3Y69c4OY2SMatnSumTVXN/r1cjmjqpTRXlyHsdDov84cOH7X/Hx8cjPj6+zDkRERH45ptvnGMZ4dW4KlyjN5hh4RiFaxzAFRt6q7RGKmngZdBvLsIl+LsoXGNrT07hmioJKOHJOwuVhkoaeBsk8oRLEAr4kEmETvfkqW6N40glQkhEztvQmzEGldZAIu9lkMgTLkMhEzk9Jk91a6pHoBN3iNIZzDBbaANvb4NEnnAZrihtYKuHo5CS0DhCgNx5G3rTBt7eCYk84TIULig3bK8lT568QzjTk6dt/7wTEnnCZcilzi83rNaZIBbxIREJnNqurxLoL0aR00SeNvD2RkjkCZehkImh0ZvAWM3KVJSHWmeiUE01CJRLoC02w2Su/Ybe9nANefJeBYk84TIUMhEsHIPeYHZam9aFUBSqcZRAJ6ZRqrRGCAV8SCVU4tmbIJEnXMb9BVHOC9modUaKx1cDp4q8xoggubhaBQeJuodEnnAZ8nthFeeKPIVrqoMtfu6M+jVFlCPvlZDIEy7D7snrnZdho9YbKVxTDWx1ZpwVrgkgkfc6SOQJl+HscE2x0QyjiSORrwYKmQg8WEsE15ZCjZFy5L0QEnnCZdhKDzgrV96WCkglDRxHKOBDLhPVOo3SbOGg0ZsoXOOFkMgTLkMiEkAs5DvNk7fFlakCZfVwxoIo23tIOfLeB4k84VKcWb/GthiHPPnq4QyRty+EIk/e6yCRJ1yK3In1a2whB0qhrB6BckmtK1HeXwhFMXlvg0SecCnOrF9jExqaeK0eVk/eUKuVx1S3xnshkSdcisKJ9WuKtAYI+DzIaMVltQj0F8NsYdDWImxmq2RJKZTeB4k84VJs9WucgUpjhFwqohWX1STg3mRpgbrmaZQqrRH+fkKIhCQZ3obDLpFGo8GoUaPw6aefomnTpti+fTs+++wzCAQCdO7cGXPmzIFKpcLEiRPt16jVahQUFODs2bMuMZ7wfBQyEQwmC4wmC8S1rBxZpDVQPL4G2OLoBepiNAr0q1Eb1r1dKR7vjTj0tZyWlobRo0cjIyMDAHD16lWsWbMGGzduxO7du2E2m5GUlIR69eph586d2LlzJ3bs2IEmTZpgyZIlrrSf8HDu58rX3ptXaYyUPlkDgmyefFHtPHmKx3snDol8SkoKFi1ahLCwMABAeno6IiMj7Y+joqJw8ODBUtds27YNUqkUSqXSySYT3oT8nig7I2RTpDVQ+mQNsIlzgbq4xm2oNFS3xltxKFyTkJBQ6nGHDh2QmJiIrKwshIWFYf/+/cjNzbUft1gsWLduHdatW1dtg+rVk1f7mpKEhipqdb2redjsC7+XEcMXCWvdtkpjRGS7MI8eQ0+0jTEGkZCPgiJDjexjjKFIZ0LDULnLX58njl9JPN2+8qhRmkLLli0xc+ZMTJ48GX5+foiOjsaFCxfsx3/66Se0bNkS7du3r3bbeXkacFzNUr1CQxXIyVHX6Fp38DDaZzFaa8lnZqkQXk9a83Y467J6AZjHjqEnv78BMjEK1MU1sk9vMMNgtEDM57n09Xny+AGeax+fz6vUOa6RyBsMBkRERCA1NRUAcODAAYSHh9uPHzx4EDExMTVpmvAx7hcpq12uvEZvvtcehQxqQqBcXOPsmiLKkfdqapQPpdPpMH78eGg0GhiNRiQlJZUS9XPnzqFTp05OM5LwXqQSIfg8HtS1jMnbviRoIVTNCPQXo7CGIm9bCBVAdWu8khp58sHBwZg6dSpGjhwJs9mMuLi4UhOsN2/eRMOGDZ1mJOG98Hk8yGW1XxBlu548+ZoRKJfgyu2iGl1rK1NMnrx3Ui2RP3z4sP3v+Ph4xMfHl3teWlpa7awifApnlDawZedQCmXNCPQXo0hrhNnCQSio3g94Kmng3dDyNcLlKKQiCtfUMTaBrskvqiKtEQI+D/70BeuVkMgTLkcuE0PjpHANCU3NuL+hd/Xj8iqNdds/PpWT8EpI5AmX44xwjVpnrVtT3VADYcVWkqAmJYdptat3Q58YwuUopCJoi82wcFyN29DoTVQBsRbc9+RrIPK02tWrIZEnXI4tI8aW614T1DoTFciqBbYvSFUNNvS2FicjkfdWSOQJl2ObLNXUImSj1hnJk68FIiEfcqmo2p48xzEU6Yy0I5QXQyJPuBxb2mNtcuXJk689wQGSaou8Wm8CY7SBtzdDIk+4HHu54RqmUTLGKCbvBIIVftWeeFXRQiivh0SecDm1rV+jN5hh4Rh5k7UkWOFX7RTK+wuh6FeUt0IiT7gcW257TXPlbWGeABKaWmEL11RnQ2+b5091a7wXEnnC5QgFfMgkwhrH5G3XkSdfO4IVEhhNHIqNFoevsXn+FK7xXkjkCbegkImg1tcsXGO7jmLytSNIYd3ftagak68qrRFSiQCSWu7PS9QdJPKEW1DIxLX35ClcUytCAqzjV1iNXHlrSQMad2+GRJ5wC3JpzcsN2yZsKS5cO4LvefLVSaOkkgbeD4k84RZqFa7RmSAW8eEnrtH2B8Q9ghT36teQyD9UkMgTbkFxrxJldTI7bGj0Jqoj7wQUMjEEfF61YvJFWgNNeHs5JPKEW1DIRLBwDHpD9evXqHUmyGlHqFrD5/MQ4C92eEGUwWSB3mAhT97LIZEn3ILcVtqgBqte1TojbRbiJAL9xSh0cEEULYTyDUjkCbdgL21Qg8lXtc4EhZS8SWcQ6C9GkYOevO08Ctd4Nw6LvEajQVxcHDIzMwEA27dvR0xMDJRKJZYtWwaz2fozPDs7G6+++iqGDBmCUaNG2c8nHm5qU9pAozeRJ+8kAuVihydeaSGUb+CQyKelpWH06NHIyMgAAFy9ehVr1qzBxo0bsXv3bpjNZiQlJQEAZs2ahaioKKSmpmLw4MFYtWqVy4wnvIf7Il89T95ossBgspDIO4kAfwmKdEZwXNUT4PZwDVX/9GocEvmUlBQsWrQIYWFhAID09HRERkbaH0dFReHgwYPIz8/HX3/9hVGjRgEAhg8fjhkzZrjGcsKrsIVbNNWMydu+FBQ08eoUguRiMObY3IhKYwSPB8ps8nIcEvmEhAR06tTJ/rhDhw5IS0tDVlYWLBYL9u/fj9zcXNy8eRONGzfG8uXLMWjQIEyfPh0iEd0gBCARCyAW8qsdrrHl1stJaJxCYDV2iFJpDQiQicHn0wbe3kyNVpe0bNkSM2fOxOTJk+Hn54fo6GhcuHABZrMZf/zxB6ZNm4b58+fjm2++wZw5c+yhHEeoV09eE5PshIYqanW9q3mY7QtUSGDiqtfHzTw9ACC8cSCAh3v8nEHzJsHWP4SCKm3VmzjUC5S69TV5+vh5un3lUSORNxgMiIiIQGpqKgDgwIEDCA8PR2hoKPz9/REVFQUAiIuLw7Jly6rVdl6exqF4YXmEhiqQk6Ou0bXu4GG3TyYRIidfV60+MrNUAACL0Tqx/zCPX20JDVWAu5cgceOWCs3qySo9PydfB7lM5LbX5A3j54n28fm8Sp3jGqVQ6nQ6jB8/HhqNBkajEUlJSYiJiUGzZs3QoEEDHD16FADwww8/oGPHjjWznPA5FFIRNNUsbWAL79DEq3Owh2scyJVXaY0Iohx5r6dGnnxwcDCmTp2KkSNHwmw2Iy4uDkqlEgDw0UcfYdGiRXj//fchl8uRmJjoVIMJ70UhE+FOvq5a16j1JvB5PEglVLfGGUhEAkglgirTKDnGUKQ1Uo68D1CtT87hw4ftf8fHxyM+Pr7MOa1atapWDJ54eKhJuWFrSQMR+Dya/HMWAf6SKksbaPUmWDhGNfx9AFrxSrgNhUwEg8kCo8nxnYmopIHzCfSvekHU/ZIGJPLeDok84TZsaZDVyZVXUwVKp0Mi/3BBIk+4jZrUr6EKlM4n0F+MoiomXm11a4JotavXQyJPuI2a1K/RULjG6QTKxdAbrOUiKsLmyVNM3vshkSfcht2TdzBcY+E4aIvNFK5xMrbSwZWFbAo1hnu7cdEG3t4OiTzhNuw15R0M12j01oU7VLfGudjSIisrOVx0b9s/HmU1eT0k8oTbkPkJwefxHA7X0EIo1+DIgiiV1kjVJ30EEnnCbfB5PMhlIsc9eVsFSgrXOJX7Il/xly1t4O07kMgTbkUhEzmcQmmL3VO4xrkoZGLweKh0QZRKYyCR9xFI5Am3opCKKFxTx/D5PATIxBWGa0xm64Q3ibxvQCJPuBV5NUob2M7zp3CN0wn0F1foyRfRjlA+BYk84VYUMsc9eY3OBJlECKGAblNnE1DJXq+02tW3oE8P4VYUUhF0xWZYOK7Kc9V6WgjlKiorbWDfwJsqUPoEJPKEW1HIxGAAtPdy4CtDrTPRpKuLCJJLUKQ1gmNlN+i578lTuMYXIJEn3Ep1ShuodUba29VFBPiLYeEYtOVkOtli9fQryjcgkSfciqIaq17VehMJjYuoLFdepbV+udJciG9A7yLhVmwVJavKlWeMQUPhGpdRqchrDAiieLzPQCJPuBVHwzV6gxkWjpEn7yJsJYTLq19TRKtdfQoSecKtOFqkzHacYvKuIaCKcE0ATbr6DA6LvEajQVxcHDIzMwEA27dvR0xMDJRKJZYtWwaz2ZotkZqaiu7du2Pw4MEYPHgwVq9e7RrLCa9EKOBDJhFWLfJU0sCl+IkFEIv4KNSUXvXKGEOhhjbw9iUc2sg7LS0NCxYsQEZGBgDg6tWrWLNmDbZu3YqwsDAsXrwYSUlJeOmll3DhwgXMmTMHcXFxrrSb8GLkMhHU+srDNVTSwLXweLx7O0SVfh/0BjPMFo7CNT6EQ558SkoKFi1ahLCwMABAeno6IiMj7Y+joqJw8OBBAMCFCxeQmpqKQYMG4e2334ZKpXKR6YS3onCgEqXtOIm86wj0l5QJ19hz5MmT9xkcEvmEhAR06tTJ/rhDhw5IS0tDVlYWLBYL9u/fj9zcXABAaGgopk2bhp07d6JRo0ZYsmSJaywnvBaFtOr6NXZPXkpi4yoCyyltYMuRp4VQvoND4ZoHadmyJWbOnInJkyfDz88P0dHRuHDhAgDg448/tp/38ssvo2/fvtVqu149eU1MshMaqqjV9a6G7ANCQ2S4ka2utC8LeBCLBGjaJKj0tTR+taKkfQ3ry5F+o7DUc3/ctP7ybhkeXCevxZvGz1uokcgbDAZEREQgNTUVAHDgwAGEh4dDrVZj27ZtmDBhAgDrJI5QWL0u8vI04LiyS60dITRUgZwcdY2udQdknxUh3+oxZmcXVbi9XHaeFgqpsJQ9NH6140H7xHzreoXbWSqIhNYf9ZlZVpG3GE1ufy3eNn6eAp/Pq9Q5rlEKpU6nw/jx46HRaGA0GpGUlISYmBjIZDJ89tlnSEtLAwBs2rQJ/fr1q5nlhM+ikFqX1OsNlgrPUetM9oVThGuwlRIuOfmq0hrtGVCEb1CjdzI4OBhTp07FyJEjYTabERcXB6VSCQBYs2YNFi9ejOLiYrRo0QIrV650qsGE92NfEKU3QuZX/i2o1hlp2z8XUzJXvl6gn/1v2sDbt6iWyB8+fNj+d3x8POLj48uc06lTJ+zYsaP2lhE+iy33Xa0zoUFw+edo9CY0qidzo1UPH7bSBSV3iLJu4E2/oHwJWvFKuB2bJ6+pJMOGygy7HlsGTckdomhvV9+DRJ5wO/crUZa/IMpossBgslBJAxdj+7JVPRCTJ5H3LUjkCbdjD9dUUImSFkK5B6GAD7lUZBd5s4WDRmeyx+oJ34BEnnA7ErEAYiG/Qk9eQ3Vr3EagXAzVvfo1ap0JDPcrVBK+AYk8USfIZaIKY/JUt8Z9BJWoX1NEG3j7JCTyRJ2gkIqrDNdQTN71BPhLUHhv4tVWkTKAsmt8ChJ5ok6wFikrP1xz35MnsXE1tvo1jLESG3jTuPsSJPJEnVBZJUq13gQ+j1fhQinCeQT6i2G2cNAbzCTyPgqJPFEnyKsI18hlIvBp1aXLKbnXa5HGCH8/IURCQR1bRTgTEnmiTlDIRDAYLTCZy9avoZIG7sNWv0alMUKlNVD6pA9CIk/UCfc39C7rzav1JsqscRM2T75Qa0AhLYTySUjkiTqhZP2aB9FQBUq3YatTU6SxhmsCKUfe5yCRJ+oEW3pkeXu9qnVG8uTdhEwihFDAg0prpJIGPgqJPFEnVBSusXActMVmism7CduG3tkFehhMFqpA6YOQyBN1QkXhGo3eXOo44XoC5RLcyLbueESevO9BIk/UCTI/Ifg8HjQPhGs0VNLA7QT6i5FTWHzvb4rJ+xok8kSdwOfxIJcKy3jy9gqUFK5xGyW9d/LkfQ8SeaLOUMjEZUX+3gIpyq5xHyVz46luje9BIk/UGeXVr6EKlO7HVlpYwOdRUTgfhESeqDPkMrG9drwNDVWgdDu2EE2Av5hKSfggDom8RqNBXFwcMjMzAQDbt29HTEwMlEolli1bBrPZXOr8P/74A48++qjzrSV8CoW0bJEytc50L3eb/A93YQvRUEkD36TKT1JaWhpGjx6NjIwMAMDVq1exZs0abNy4Ebt374bZbEZSUpL9fL1ejyVLlsBkqniTZoIArCEZrd4EjmP259R6I+QUqnErNk+eJl19kypruaakpGDRokWYNWsWACA9PR2RkZEICwsDAERFRWH9+vV46aWXAACJiYmYMGECzp496zQjLRYzCgpyYDaXX3/cRnY2HxzHOa1fZ+OJ9gmFYgQHh0IgcH9ZX4VMDAbrdn82L1Kto7o17oZE3rep8pOdkJBQ6nGHDh2QmJiIrKwshIWFYf/+/cjNzQUAHDp0CMXFxYiOjq6xQfXqycs8d/XqVfj7+0MubwwexQydBmMMarUKOl0BWrVqBQAIDVW4rf8mDQIAACI/kb1fvdGCBiGyCu1wp301wVvt69yxIZ6NaFLn9td1/1Xh6faVR7Xdt5YtW2LmzJmYPHky/Pz8EB0djQsXLiAnJwfr1q3Dxo0ba2VQXp6m1M93ANBqdWjQoD4sFgaAlX8hAKGQD7PZszzlkniifVKpAnfvFiAnR43QUAVyctRu65u7N5dz41YhpALrl3eBuhjhof7l2uFu+6qLN9s3SfkIANSp/d48fnUJn88r1zm2UW2RNxgMiIiIQGpqKgDgwIEDCA8Px5EjR1BYWIgXX3zRfu7gwYOxefNmyOUVG+Ao5MG7hroc1wdLGzDG7lWgpHANQTiLaqcw6HQ6jB8/HhqNBkajEUlJSYiJiUF8fDwOHjyInTt3YufOnQCAnTt3OkXgifuMGKFEVtbtujbDKdwvUmada9EbzLBwDAopxYYJwllU25MPDg7G1KlTMXLkSJjNZsTFxUGpVLrCNsLHuV9u2FTqf5p4JQjn4bDIHz582P53fHw84uPjKz0/PT295lZ5OGfOnMK6dWthsXBo1KgRpFIZrl69Ao7j8OKL49C7dz8MHhyNlJRUyGT+mDx5Irp164kJEybi4MHvcO7cWUyePBXvvbcUOTnZyM3NQWTkE1iwYAnOnj1tb7tVq9aYPv0tLFmyENnZd9GiRSsYjVav9++/L2PlygRYLBaIxWLMm7cI4eHN6nhkqodQwIdUcr9+jb1uDZU0IAin4f68uVpy/EIWjp3PKvcYjwewiudlq6R7RCN0e6yRQ+fevHkDW7fuQVLSF6hfPxQLFvwTWq0Gr702EY888iieeqoTzp49gyeeeApZWVk4d+4MgIn45ZcT6NOnH06cOIa2bdth2bIVMJlMGDMmHunpf5VqWy6X49//XoF27Tpg1aq1OHfuDA4f/h4AkJLyFUaNGoPevfvi0KED+P33C14n8kDp0gZU0oAgnI/XibynEB7eHHK5HKdO/QaDoRjffrsLAFBcXIxr167i2We74/Tp38Dn89C//0AcOnQAZrMJaWnn8M478yCRSPDHHxeRkvIVMjKuQaVSQa/XlWobAM6ePY3Fi5cDACIjn0Tjxk0AAM8+2w3//vdK/PrrCXTt2gPPPdenDkah9lhF/gFPnkoaEITT8DqR7/ZYxd62O1MUJRJrUSeOs2DhwqVo374DACA/Pw8BAYFQq9VITt4MgUCIp556GjduZGDXrp1o1aoVJBIJtm5NxpEjhzFo0FCMGPEMrl27AnbvZ4itbcCa/VJyAZVAIAAAREX1xaOPRuD48Z/wzTdf45dfjmP27AVuee3ORCEVI6/IWstco6dwDUE4GyoQUkuefPJppKZuBQDk5uZi/PjRuHv3DoKDgyGRSHD8+I+IiIjEk08+jS++2ICuXXsAAE6e/BWDBg1D//4DAfBw+fKlclfDdur0DA4c2AcA+PPP33HrlrV+0LvvzsUff/yOIUOG4+WXX7OHerwN+QPhGrGQD4lYUMdWEYTvQCJfSyZOfAUGgwFjxz6PN954Da+/Ph1NmjQFYA2pyOUKyGQyPPXU08jJyUHXrt0BAM8//wK++GI9Jk58Ef/+9wo8+mhEuamR//d/k3DrVibGjHkemzZttIdrxo59CUlJX2DixBfx8cdrMG3am+570U7EFq5hjEFNOfIE4XR4jNVmqtL5lLfi9c6d62jYsHmV13riitKSeKp9tvGtixV9+3+9gZQf/sZHM3riP7t+R5HWiEUvPV3uuZ664tAG2Vc7yL6aUdWKV/LkiTrFlkmj0Ruh0Rsps4YgnAyJPFGn3F/1aqJwDUG4ABJ5ok4pWb9GrTNRSQOCcDIk8kSdYsuJz1cXw2CyULiGIJwMiTxRp9g8+axc3b3HJPIE4UxI5Ik6RSziQyTk43aeFgAgp3ANQTgVEnmiTuHxeFDIRHaRJ0+eIJwLibyHcvv2Lbz33hIA1qqXU6e+WuO29u7djYSExU6yzPkopGKoNFScjCBcAYm8h3LnTpa9hIGvU1LYqW4NQTgXrytQVtdkZ9/FkiULodfrwefz8MYb72Dx4nno3dtaPlggEGDSpClITt6EzMybmDJlBvr06Yf8/DysWLEUd+7cgUAgwKuvTkGXLl1RXFyMFSuW4e+/L4HP52PUqDEYODAOH3ywCrdv38K//rUCUVF9UFhYiLffno5btzLRrFlzLF26AmKxGPv27cE333wNjmNo374D3nprNiQSCfbv/xZffvlf+PvL0bBhQ0ilsroeugqx5cbzeTzI/OiWJAhn4nWfKNOl4zCl/1juMR6Ph9pUaRC17wlRu26VnrNnz0507dodL7wwDmfOnML58+cAAPXrh2LTphQsX/5PbNq0EWvXfooLF9Kwdu2/0KdPP6xe/T6eeuoZPP/8C7h1KxOvv/4yvvhiM5KTNyEwMBBJSSkoLCzEK6+MR9u27fHGG2/j88/XY+bM2Thz5hTu3r2DlStXo2HDRpg0aQJOnfoNDRs2wu7dqVi37nNIJBJ8+ulH+PrrJMTFDca6dWvxxRdfISAgELNmzfBokbflxsulQvBpL1+CcCoUrqkmnTo9g6+/3oTFi+cjNzcHw4c/DwDo0qUrAKBBg4aIjHwSQqEQDRs2glptrXVx5sxJDBo0GADQpElTPPLIo/jjj4s4ffoUYmOtzwcFBaFHj544e/Z0mX7btGmLxo2bgM/no3nzllCpCnH27ClkZt7EpEkvYcKEF3Ds2FHcuHEdFy6k4dFHIxASUg9CofBepUvPxRauoVANQTgfr/PkRe26Vehtu6MAWEREJDZtSsGJE8dw6NAB7N2722qX6H5c2VbzvSQPFl0DGCwWCxgrbS9jgMViLnN9yTZtv1gsFg69e/fFjBnvALBusm6xWHD69G+l+ivPHk/ivsjTpCtBOBuHPXmNRoO4uDhkZlonA7dv346YmBgolUosW7YMZrNVmE6dOoVhw4ZBqVTitddeg0qlco3ldcQnn3yA777bi4ED4/Dmm7Nx6ZJje9k+9VQn7Nq1EwBw61YmLlxIQ8eOEXjyyafx7bfW5wsLC/HTT0fwxBOdIBAIYbFYKm3ziSeewo8/HkFBQT4YY/jXv95DSspXiIiIxB9/XEBOTjY4jrNvGeipyO3hGhJ5gnA2Dol8WloaRo8ejYyMDADA1atXsWbNGmzcuBG7d++G2WxGUlISAGDu3LlYuXIldu/ejTZt2uC///2vy4yvC4YPH4kjRw5jwoQXMG/e25g5c45D182Y8Q5On/4N48aNxLx5b2P27AWoX78+XnrpZRQVFWHcuJGYOvUVjBs3Ee3bd0CLFi2g0aixdOnCCtts27YdXnrpFUyf/hrGjn0eHMcwZswEhITUw4wZ72DGjNfxyivjIZP5O+vluwQK1xCE63Convz8+fMxdOhQzJo1C//73/9w4cIF7Nu3D2vXrgUAHD16FOvXr8fmzZthMpkgEolgMpkwb948tG/fHi+//LLDBlE9efdTl/XkASArT4v5G37FoG4tMKRHqwrP89R63jbIvtpB9tWMqurJOxSTT0hIKPW4Q4cOSExMRFZWFsLCwrB//37k5uYCsMam09PT8dJLL0EoFOKtt96qlsHlGZudzYdQ6FhkydHz6gpPtI/P5yM0VAEA9v/diTxACqlEiHYt6lXZf13YVx3IvtpB9jmfGk28tmzZEjNnzsTkyZPh5+eH6OhoXLhwwX68ffv2OHHiBJKTk/Hmm28iOTnZ4bbL8+Q5jnPIA/ZUT9mGp9rHcRxyctR16qmser0rJGJBpf17qidlg+yrHWRfzXDJzlAGgwERERFITU1FcnIyGjdujPDwcBgMBhw8eNB+3qBBg5Ce7tjEJPFwI5VQjjxBuIIaibxOp8P48eOh0WhgNBqRlJSEmJgYCIVC/POf/8TFixcBAPv27cOTTz7pFEM9bCtan4HGlSB8mxqFa4KDgzF16lSMHDkSZrMZcXFxUCqVAIDVq1fj3XffhcViQYMGDcrE82tkpFAMrbYI/v4B4JG35zQYY9BqiyAUUlYLQfgqDmXXuJPyYvIWixkFBTkwm42VXsvn88FxnhfztuGJ9gmFYgQHh0IgEHpszNEG2Vc7yL7a4an2OSW7pq4RCISoX79Rled56ptgw9PtIwjC9/C8fD6CIAjCaZDIEwRB+DAeF67h82s3sVrb610N2Vc7yL7aQfbVDk+0ryqbPG7ilSAIgnAeFK4hCILwYUjkCYIgfBgSeYIgCB+GRJ4gCMKHIZEnCILwYUjkCYIgfBgSeYIgCB+GRJ4gCMKHIZEnCILwYbxO5Hfv3o2YmBj069cPmzdvLnP8zz//xPDhwzFgwADMnz8fZrPZrfZ99NFHiI2NRWxsLFauXFnu8aioKAwePBiDBw8u9zW4knHjxiE2Ntbef1paWqnjJ06cgFKpRP/+/bF69Wq32gYA33zzjd22wYMH46mnnsKSJUtKnVMXY6jRaBAXF4fMzEwAjo3T7du38eKLLyI6OhqTJ0+GVqt1m31btmyx7/Mwd+5cGI1ly3Snpqaie/fu9nF05fv9oH1z585F//797X1///33Za5x52e5pH1Hjx4tdQ926dIFkyZNKnONO8evVjAv4s6dOywqKooVFBQwrVbLlEolu3z5cqlzYmNj2dmzZxljjM2dO5dt3rzZbfYdP36cjRw5khkMBmY0Gtm4cePYgQMHSp0zadIkdubMGbfZVBKO41i3bt2YyWQq97her2e9evViN27cYCaTiU2cOJEdOXLEzVbe59KlS6xfv34sLy+v1PPuHsNz586xuLg41rFjR3bz5k2Hx+nVV19le/bsYYwx9tFHH7GVK1e6xb6rV6+yfv36MbVazTiOY7NmzWJffPFFmeuWLFnCdu/e7RKbKrOPMcbi4uLY3bt3K73OXZ/l8uyzkZ2dzfr06cOuXbtW5jp3jV9t8SpP/sSJE+jSpQuCgoIgk8kwYMAA7N+/33781q1bKC4uRmRkJABg2LBhpY67mtDQUMyZMwdisRgikQitW7fG7du3S51z8eJFbNiwAUqlEkuWLIHBYHCbfVevXgWPx8Mrr7yCQYMGYdOmTaWOnz9/Hs2bN0d4eDiEQiGUSqVbx+9BFi9ejDfffBMhISGlnnf3GKakpGDRokUICwsD4Ng4mUwmnDx5EgMGDADg2nvxQfvEYjEWL14MuVwOHo+Hdu3albkPAeDChQtITU3FoEGD8Pbbb0OlUrnFPp1Oh9u3b2PhwoVQKpVYu3Ztmc103PlZftC+kqxcuRKjRo1CixYtyhxz1/jVFq8S+ezsbISGhtofh4WF4e7duxUeDw0NLXXc1bRt29Z+U2ZkZGDv3r3o1auX/bhWq8U//vEPzJ49Gzt27EBRURE++eQTt9lXVFSEZ599Fh9//DE2btyI5ORkHD9+3H68qvF1JydOnEBxcTEGDhxY6vm6GMOEhAR06tTJ/tiRcSooKIBcLodQaC306sp78UH7mjRpgq5duwIA8vPzsXnzZvTp06fMdaGhoZg2bRp27tyJRo0alQmLucq+vLw8dOnSBcuXL0dKSgpOnTqFrVu3lrrGnZ/lB+2zkZGRgd9++w3jxo0r9zp3jV9t8SqRZ+UUzCy552tVx93F5cuXMXHiRMyePbuUB+Dv748NGzagefPmEAqFmDhxIo4ePeo2u5544gmsXLkSMpkMISEhGDFiRKn+PWX8ACA5ORkvvfRSmefregwBx8bJE8by7t27GD9+PIYPH47OnTuXOf7xxx/j8ccfB4/Hw8svv4wff/zRLXaFh4fj448/Rr169SCVSjF27Ngy76EnjN+WLVvwwgsvQCwufw/kuhq/6uJVIt+gQQPk5ubaH2dnZ5f6ifXg8ZycnHJ/grmS06dPY8KECZg5cyaGDh1a6tjt27dLeSyMMbun5w5OnTqFn3/+ucL+qxpfd2E0GnHy5En07t27zLG6HkPAsXEKCQmBRqOBxWIB4P578cqVKxg9ejSGDh2KKVOmlDmuVquxceNG+2N3jmN6ejq+++67Svv2hM/yoUOHEBMTU+6xuhy/6uJVIt+1a1f8/PPPyM/Ph16vx4EDB9CzZ0/78SZNmkAikeD06dMArLPfJY+7mqysLEyZMgWrVq1CbGxsmeN+fn54//33cfPmTTDGsHnzZvTr189t9qnVaqxcuRIGgwEajQY7duwo1f/jjz+Oa9eu4fr167BYLNizZ49bx89Geno6WrRoAZlMVuZYXY8h4Ng4iUQidOrUCXv37gXg3ntRo9Hg//7v//DGG29g4sSJ5Z4jk8nw2Wef2bOrNm3a5LZxZIxh+fLlUKlUMJlM2LJlS5m+6/qznJ+fj+LiYoSHh5d7vC7Hr9rUwWRvrdi1axeLjY1l/fv3Z+vXr2eMMfbyyy+z8+fPM8YY+/PPP9nw4cNZdHQ0e+utt5jBYHCbbUuXLmWRkZFs0KBB9n9fffVVKfv2799vt3/OnDlutY8xxlavXs2io6NZ//792caNGxljjA0aNIjduXOHMcbYiRMnmFKpZP3792cJCQmM4zi32scYY99++y2bMWNGqec8YQyjoqLs2RcVjdO8efPYwYMHGWOMZWZmsjFjxrCBAweyiRMnssLCQrfY98UXX7COHTuWug/XrFlTxr6TJ0+yIUOGsOjoaPbaa6+xoqIit9jHGGObNm1iAwcOZP369WPvv/++/Zy6/CyXtC8tLY3Fx8eXOacux6+m0M5QBEEQPoxXhWsIgiCI6kEiTxAE4cOQyBMEQfgwJPIEQRA+DIk8QRCED0MiTxDl8Morr+Dvv/+u1jWTJk3C9u3bXWQRQdQMz1yiRRB1zIYNG+raBIJwCiTyhE9x+PBhrFu3DiaTCX5+fpg9ezaOHTuGy5cvIzc3F3l5eejQoQMSEhIgl8vx1VdfITk5GSKRCBKJBEuWLEGbNm3Qu3dvfPDBB3jsscewZcsWJCUlgc/no379+li4cCFatmyJu3fvYs6cOcjOzkbjxo2Rl5dnt+PKlStISEhAYWEhLBYLxo4dixEjRkCr1WLu3Lm4fv06+Hw+OnbsiCVLloDPpx/VhIuo69VYBOEsrl27xuLi4lh+fj5jzFqPvlu3biwxMZH17NmT5eTkMIvFwt566y2WmJjIzGYz69ixo72u+Y4dO1hycjJjzLr68fz58+zEiROsb9++9pr227ZtYwMHDmQcx7HXX3+drV69mjHGWEZGBouMjGTbtm1jJpOJxcTEsIsXLzLGGCsqKmIDBw5kZ8+eZTt27GATJ05kjDFmNpvZ/PnzWUZGhjuHiXjIIE+e8BmOHz+O7OxsTJgwwf4cj8fDjRs3EB0djfr16wMARowYgeXLl2P27NmIjo7GqFGj8Nxzz6Fbt25QKpWl2vzpp58QExNjr2k/bNgwJCQkIDMzEydOnMDs2bMBAM2bN7dXeszIyMCNGzcwb948ezvFxcX4448/0KNHD6xevRpjx45F165dMX78eDRv3tyVw0I85JDIEz4Dx3F49tlnsWbNGvtzWVlZ2LJlS6nt7ziOs4dHVq1ahUuXLuHEiRPYsGEDtm7dinXr1tnPZeVU/WCMwWw2g8fjlTpuq0JosVgQEBCAnTt32o/l5uZCoVBAIpHg+++/x6+//opffvkFL730EhYsWIDo6GinjQNBlIQCgYTP0KVLFxw/fhxXrlwBABw9ehSDBg2CwWDAoUOHoFarwXEcUlJSEBUVhfz8fPTq1QtBQUGYMGECZsyYgfT09FJtdu/eHXv37kV+fj4AYNu2bQgKCkLz5s3Ro0cPbNmyBYC1BPKvv/4KAGjZsiUkEold5LOyshAXF4eLFy/iq6++wty5c9G9e3e888476N69Oy5fvuyuISIeQqhAGeFT7Nu3D59++qm9vve8efPw888/45dffoHFYkFBQQGefvppLFiwAH5+fkhOTsb//vc/+Pn5QSAQ4M0330TXrl1LTbxu3rwZycnJ4DgOISEhePfdd9G2bVvk5+dj7ty5uHHjBho2bAiz2YyhQ4di2LBh+Ouvv+wTr2azGePGjcPo0aOh0+kwb948pKenQyqVonHjxkhISEBgYGBdDx3ho5DIEz7Phx9+iIKCArz77rt1bQpBuB0K1xAEQfgw5MkTBEH4MOTJEwRB+DAk8gRBED4MiTxBEIQPQyJPEAThw5DIEwRB+DAk8gRBED7M/wOOGE0T8YP6OQAAAABJRU5ErkJggg==", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "# 获取参数\n", - "cfg = get_args() \n", - "# 训练\n", - "env, agent = env_agent_config(cfg)\n", - "res_dic = train(cfg, env, agent)\n", - " \n", - "plot_rewards(res_dic['rewards'], cfg, tag=\"train\") \n", - "# 测试\n", - "res_dic = test(cfg, env, agent)\n", - "plot_rewards(res_dic['rewards'], cfg, tag=\"test\") # 画出结果" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3.7.13 ('easyrl')", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.13" - }, - "orig_nbformat": 4, - "vscode": { - "interpreter": { - "hash": "8994a120d39b6e6a2ecc94b4007f5314b68aa69fc88a7f00edf21be39b41f49c" - } - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/projects/parl_tutorials/README.md b/projects/parl_tutorials/README.md deleted file mode 100644 index fd03c9a..0000000 --- a/projects/parl_tutorials/README.md +++ /dev/null @@ -1,19 +0,0 @@ -## 运行环境 - -由于```parl```和```paddle```容易与notebook相关模块发生版本冲突，因此推荐新建一个Conda环境： -```bash -conda create -n parl python=3.7 -``` - -然后安装```parl```和```paddle```： -```bash -pip install parl==2.0.5 - -pip install paddlepaddle-gpu==2.3.2 -i https://pypi.tuna.tsinghua.edu.cn/simple - -pip install paddlepaddle==2.3.2 -i https://pypi.tuna.tsinghua.edu.cn/simple -``` -安装其他依赖： -```bash -pip install -r parl_requirements.txt -``` \ No newline at end of file diff --git a/projects/parl_tutorials/parl_requirements.txt b/projects/parl_tutorials/parl_requirements.txt deleted file mode 100644 index cc8624d..0000000 --- a/projects/parl_tutorials/parl_requirements.txt +++ /dev/null @@ -1,7 +0,0 @@ -gym==0.19.0 -ipykernel==6.0.0 -jupyter==1.0.0 -pyzmq==18.1.1 -jupyter-client==7.0.0 -matplotlib==3.5.3 -seaborn==0.12.0 \ No newline at end of file diff --git a/projects/requirements.txt b/projects/requirements.txt deleted file mode 100644 index 5cda89e..0000000 --- a/projects/requirements.txt +++ /dev/null @@ -1,11 +0,0 @@ -pyyaml==6.0 -ipykernel==6.15.1 -jupyter==1.0.0 -matplotlib==3.5.3 -seaborn==0.12.1 -dill==0.3.5.1 -argparse==1.4.0 -pandas==1.3.5 -pyglet==1.5.26 -importlib-metadata<5.0 -setuptools==65.2.0 \ No newline at end of file