diff --git a/LICENSE b/LICENSE deleted file mode 100644 index cbe5ad1..0000000 --- a/LICENSE +++ /dev/null @@ -1,437 +0,0 @@ -Attribution-NonCommercial-ShareAlike 4.0 International - -======================================================================= - -Creative Commons Corporation ("Creative Commons") is not a law firm and -does not provide legal services or legal advice. Distribution of -Creative Commons public licenses does not create a lawyer-client or -other relationship. Creative Commons makes its licenses and related -information available on an "as-is" basis. Creative Commons gives no -warranties regarding its licenses, any material licensed under their -terms and conditions, or any related information. Creative Commons -disclaims all liability for damages resulting from their use to the -fullest extent possible. - -Using Creative Commons Public Licenses - -Creative Commons public licenses provide a standard set of terms and -conditions that creators and other rights holders may use to share -original works of authorship and other material subject to copyright -and certain other rights specified in the public license below. The -following considerations are for informational purposes only, are not -exhaustive, and do not form part of our licenses. - - Considerations for licensors: Our public licenses are - intended for use by those authorized to give the public - permission to use material in ways otherwise restricted by - copyright and certain other rights. Our licenses are - irrevocable. Licensors should read and understand the terms - and conditions of the license they choose before applying it. - Licensors should also secure all rights necessary before - applying our licenses so that the public can reuse the - material as expected. Licensors should clearly mark any - material not subject to the license. This includes other CC- - licensed material, or material used under an exception or - limitation to copyright. More considerations for licensors: - wiki.creativecommons.org/Considerations_for_licensors - - Considerations for the public: By using one of our public - licenses, a licensor grants the public permission to use the - licensed material under specified terms and conditions. If - the licensor's permission is not necessary for any reason--for - example, because of any applicable exception or limitation to - copyright--then that use is not regulated by the license. Our - licenses grant only permissions under copyright and certain - other rights that a licensor has authority to grant. Use of - the licensed material may still be restricted for other - reasons, including because others have copyright or other - rights in the material. A licensor may make special requests, - such as asking that all changes be marked or described. - Although not required by our licenses, you are encouraged to - respect those requests where reasonable. More considerations - for the public: - wiki.creativecommons.org/Considerations_for_licensees - -======================================================================= - -Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International -Public License - -By exercising the Licensed Rights (defined below), You accept and agree -to be bound by the terms and conditions of this Creative Commons -Attribution-NonCommercial-ShareAlike 4.0 International Public License -("Public License"). To the extent this Public License may be -interpreted as a contract, You are granted the Licensed Rights in -consideration of Your acceptance of these terms and conditions, and the -Licensor grants You such rights in consideration of benefits the -Licensor receives from making the Licensed Material available under -these terms and conditions. - - -Section 1 -- Definitions. - - a. Adapted Material means material subject to Copyright and Similar - Rights that is derived from or based upon the Licensed Material - and in which the Licensed Material is translated, altered, - arranged, transformed, or otherwise modified in a manner requiring - permission under the Copyright and Similar Rights held by the - Licensor. For purposes of this Public License, where the Licensed - Material is a musical work, performance, or sound recording, - Adapted Material is always produced where the Licensed Material is - synched in timed relation with a moving image. - - b. Adapter's License means the license You apply to Your Copyright - and Similar Rights in Your contributions to Adapted Material in - accordance with the terms and conditions of this Public License. - - c. BY-NC-SA Compatible License means a license listed at - creativecommons.org/compatiblelicenses, approved by Creative - Commons as essentially the equivalent of this Public License. - - d. Copyright and Similar Rights means copyright and/or similar rights - closely related to copyright including, without limitation, - performance, broadcast, sound recording, and Sui Generis Database - Rights, without regard to how the rights are labeled or - categorized. For purposes of this Public License, the rights - specified in Section 2(b)(1)-(2) are not Copyright and Similar - Rights. - - e. Effective Technological Measures means those measures that, in the - absence of proper authority, may not be circumvented under laws - fulfilling obligations under Article 11 of the WIPO Copyright - Treaty adopted on December 20, 1996, and/or similar international - agreements. - - f. Exceptions and Limitations means fair use, fair dealing, and/or - any other exception or limitation to Copyright and Similar Rights - that applies to Your use of the Licensed Material. - - g. License Elements means the license attributes listed in the name - of a Creative Commons Public License. The License Elements of this - Public License are Attribution, NonCommercial, and ShareAlike. - - h. Licensed Material means the artistic or literary work, database, - or other material to which the Licensor applied this Public - License. - - i. Licensed Rights means the rights granted to You subject to the - terms and conditions of this Public License, which are limited to - all Copyright and Similar Rights that apply to Your use of the - Licensed Material and that the Licensor has authority to license. - - j. Licensor means the individual(s) or entity(ies) granting rights - under this Public License. - - k. NonCommercial means not primarily intended for or directed towards - commercial advantage or monetary compensation. For purposes of - this Public License, the exchange of the Licensed Material for - other material subject to Copyright and Similar Rights by digital - file-sharing or similar means is NonCommercial provided there is - no payment of monetary compensation in connection with the - exchange. - - l. Share means to provide material to the public by any means or - process that requires permission under the Licensed Rights, such - as reproduction, public display, public performance, distribution, - dissemination, communication, or importation, and to make material - available to the public including in ways that members of the - public may access the material from a place and at a time - individually chosen by them. - - m. Sui Generis Database Rights means rights other than copyright - resulting from Directive 96/9/EC of the European Parliament and of - the Council of 11 March 1996 on the legal protection of databases, - as amended and/or succeeded, as well as other essentially - equivalent rights anywhere in the world. - - n. You means the individual or entity exercising the Licensed Rights - under this Public License. Your has a corresponding meaning. - - -Section 2 -- Scope. - - a. License grant. - - 1. Subject to the terms and conditions of this Public License, - the Licensor hereby grants You a worldwide, royalty-free, - non-sublicensable, non-exclusive, irrevocable license to - exercise the Licensed Rights in the Licensed Material to: - - a. reproduce and Share the Licensed Material, in whole or - in part, for NonCommercial purposes only; and - - b. produce, reproduce, and Share Adapted Material for - NonCommercial purposes only. - - 2. Exceptions and Limitations. For the avoidance of doubt, where - Exceptions and Limitations apply to Your use, this Public - License does not apply, and You do not need to comply with - its terms and conditions. - - 3. Term. The term of this Public License is specified in Section - 6(a). - - 4. Media and formats; technical modifications allowed. The - Licensor authorizes You to exercise the Licensed Rights in - all media and formats whether now known or hereafter created, - and to make technical modifications necessary to do so. The - Licensor waives and/or agrees not to assert any right or - authority to forbid You from making technical modifications - necessary to exercise the Licensed Rights, including - technical modifications necessary to circumvent Effective - Technological Measures. For purposes of this Public License, - simply making modifications authorized by this Section 2(a) - (4) never produces Adapted Material. - - 5. Downstream recipients. - - a. Offer from the Licensor -- Licensed Material. Every - recipient of the Licensed Material automatically - receives an offer from the Licensor to exercise the - Licensed Rights under the terms and conditions of this - Public License. - - b. Additional offer from the Licensor -- Adapted Material. - Every recipient of Adapted Material from You - automatically receives an offer from the Licensor to - exercise the Licensed Rights in the Adapted Material - under the conditions of the Adapter's License You apply. - - c. No downstream restrictions. You may not offer or impose - any additional or different terms or conditions on, or - apply any Effective Technological Measures to, the - Licensed Material if doing so restricts exercise of the - Licensed Rights by any recipient of the Licensed - Material. - - 6. No endorsement. Nothing in this Public License constitutes or - may be construed as permission to assert or imply that You - are, or that Your use of the Licensed Material is, connected - with, or sponsored, endorsed, or granted official status by, - the Licensor or others designated to receive attribution as - provided in Section 3(a)(1)(A)(i). - - b. Other rights. - - 1. Moral rights, such as the right of integrity, are not - licensed under this Public License, nor are publicity, - privacy, and/or other similar personality rights; however, to - the extent possible, the Licensor waives and/or agrees not to - assert any such rights held by the Licensor to the limited - extent necessary to allow You to exercise the Licensed - Rights, but not otherwise. - - 2. Patent and trademark rights are not licensed under this - Public License. - - 3. To the extent possible, the Licensor waives any right to - collect royalties from You for the exercise of the Licensed - Rights, whether directly or through a collecting society - under any voluntary or waivable statutory or compulsory - licensing scheme. In all other cases the Licensor expressly - reserves any right to collect such royalties, including when - the Licensed Material is used other than for NonCommercial - purposes. - - -Section 3 -- License Conditions. - -Your exercise of the Licensed Rights is expressly made subject to the -following conditions. - - a. Attribution. - - 1. If You Share the Licensed Material (including in modified - form), You must: - - a. retain the following if it is supplied by the Licensor - with the Licensed Material: - - i. identification of the creator(s) of the Licensed - Material and any others designated to receive - attribution, in any reasonable manner requested by - the Licensor (including by pseudonym if - designated); - - ii. a copyright notice; - - iii. a notice that refers to this Public License; - - iv. a notice that refers to the disclaimer of - warranties; - - v. a URI or hyperlink to the Licensed Material to the - extent reasonably practicable; - - b. indicate if You modified the Licensed Material and - retain an indication of any previous modifications; and - - c. indicate the Licensed Material is licensed under this - Public License, and include the text of, or the URI or - hyperlink to, this Public License. - - 2. You may satisfy the conditions in Section 3(a)(1) in any - reasonable manner based on the medium, means, and context in - which You Share the Licensed Material. For example, it may be - reasonable to satisfy the conditions by providing a URI or - hyperlink to a resource that includes the required - information. - 3. If requested by the Licensor, You must remove any of the - information required by Section 3(a)(1)(A) to the extent - reasonably practicable. - - b. ShareAlike. - - In addition to the conditions in Section 3(a), if You Share - Adapted Material You produce, the following conditions also apply. - - 1. The Adapter's License You apply must be a Creative Commons - license with the same License Elements, this version or - later, or a BY-NC-SA Compatible License. - - 2. You must include the text of, or the URI or hyperlink to, the - Adapter's License You apply. You may satisfy this condition - in any reasonable manner based on the medium, means, and - context in which You Share Adapted Material. - - 3. You may not offer or impose any additional or different terms - or conditions on, or apply any Effective Technological - Measures to, Adapted Material that restrict exercise of the - rights granted under the Adapter's License You apply. - - -Section 4 -- Sui Generis Database Rights. - -Where the Licensed Rights include Sui Generis Database Rights that -apply to Your use of the Licensed Material: - - a. for the avoidance of doubt, Section 2(a)(1) grants You the right - to extract, reuse, reproduce, and Share all or a substantial - portion of the contents of the database for NonCommercial purposes - only; - - b. if You include all or a substantial portion of the database - contents in a database in which You have Sui Generis Database - Rights, then the database in which You have Sui Generis Database - Rights (but not its individual contents) is Adapted Material, - including for purposes of Section 3(b); and - - c. You must comply with the conditions in Section 3(a) if You Share - all or a substantial portion of the contents of the database. - -For the avoidance of doubt, this Section 4 supplements and does not -replace Your obligations under this Public License where the Licensed -Rights include other Copyright and Similar Rights. - - -Section 5 -- Disclaimer of Warranties and Limitation of Liability. - - a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE - EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS - AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF - ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, - IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, - WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR - PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, - ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT - KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT - ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. - - b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE - TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, - NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, - INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, - COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR - USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN - ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR - DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR - IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. - - c. The disclaimer of warranties and limitation of liability provided - above shall be interpreted in a manner that, to the extent - possible, most closely approximates an absolute disclaimer and - waiver of all liability. - - -Section 6 -- Term and Termination. - - a. This Public License applies for the term of the Copyright and - Similar Rights licensed here. However, if You fail to comply with - this Public License, then Your rights under this Public License - terminate automatically. - - b. Where Your right to use the Licensed Material has terminated under - Section 6(a), it reinstates: - - 1. automatically as of the date the violation is cured, provided - it is cured within 30 days of Your discovery of the - violation; or - - 2. upon express reinstatement by the Licensor. - - For the avoidance of doubt, this Section 6(b) does not affect any - right the Licensor may have to seek remedies for Your violations - of this Public License. - - c. For the avoidance of doubt, the Licensor may also offer the - Licensed Material under separate terms or conditions or stop - distributing the Licensed Material at any time; however, doing so - will not terminate this Public License. - - d. Sections 1, 5, 6, 7, and 8 survive termination of this Public - License. - - -Section 7 -- Other Terms and Conditions. - - a. The Licensor shall not be bound by any additional or different - terms or conditions communicated by You unless expressly agreed. - - b. Any arrangements, understandings, or agreements regarding the - Licensed Material not stated herein are separate from and - independent of the terms and conditions of this Public License. - - -Section 8 -- Interpretation. - - a. For the avoidance of doubt, this Public License does not, and - shall not be interpreted to, reduce, limit, restrict, or impose - conditions on any use of the Licensed Material that could lawfully - be made without permission under this Public License. - - b. To the extent possible, if any provision of this Public License is - deemed unenforceable, it shall be automatically reformed to the - minimum extent necessary to make it enforceable. If the provision - cannot be reformed, it shall be severed from this Public License - without affecting the enforceability of the remaining terms and - conditions. - - c. No term or condition of this Public License will be waived and no - failure to comply consented to unless expressly agreed to by the - Licensor. - - d. Nothing in this Public License constitutes or may be interpreted - as a limitation upon, or waiver of, any privileges and immunities - that apply to the Licensor or You, including from the legal - processes of any jurisdiction or authority. - -======================================================================= - -Creative Commons is not a party to its public -licenses. Notwithstanding, Creative Commons may elect to apply one of -its public licenses to material it publishes and in those instances -will be considered the “Licensor.” The text of the Creative Commons -public licenses is dedicated to the public domain under the CC0 Public -Domain Dedication. Except for the limited purpose of indicating that -material is shared under a Creative Commons public license or as -otherwise permitted by the Creative Commons policies published at -creativecommons.org/policies, Creative Commons does not authorize the -use of the trademark "Creative Commons" or any other trademark or logo -of Creative Commons without its prior written consent including, -without limitation, in connection with any unauthorized modifications -to any of its public licenses or any other arrangements, -understandings, or agreements concerning use of licensed material. For -the avoidance of doubt, this paragraph does not form part of the -public licenses. - -Creative Commons may be contacted at creativecommons.org. diff --git a/README.md b/README.md deleted file mode 100644 index 16b2c92..0000000 --- a/README.md +++ /dev/null @@ -1,69 +0,0 @@ -# Easy-RL - -李宏毅老师的《深度强化学习》是强化学习领域经典的中文视频之一。李老师幽默风趣的上课风格让晦涩难懂的强化学习理论变得轻松易懂，他会通过很多有趣的例子来讲解强化学习理论。比如老师经常会用玩 Atari 游戏的例子来讲解强化学习算法。此外，为了教程的完整性，我们整理了周博磊老师的《强化学习纲要》、李科浇老师的《百度强化学习》以及多个强化学习的经典资料作为补充。对于想入门强化学习又想看中文讲解的人来说绝对是非常推荐的。 - -## 使用说明 - -* 第 4 章到第 11 章为[李宏毅《深度强化学习》](http://speech.ee.ntu.edu.tw/~tlkagk/courses_MLDS18.html)的部分； -* 第 1 章和第 2 章根据[《强化学习纲要》](https://github.com/zhoubolei/introRL)整理而来； -* 第 3 章和第 12 章根据[《百度强化学习》](https://aistudio.baidu.com/aistudio/education/group/info/1335) 整理而来。 - - -## 在线阅读(内容实时更新) -地址：https://datawhalechina.github.io/easy-rl/ - -## 内容导航 -| 章节 | 习题 | 相关项目 | -| ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | -| [第一章 强化学习概述](https://datawhalechina.github.io/easy-rl/#/chapter1/chapter1) | [第一章 习题](https://datawhalechina.github.io/easy-rl/#/chapter1/chapter1_questions&keywords) | | -| [第二章 马尔可夫决策过程 (MDP)](https://datawhalechina.github.io/easy-rl/#/chapter2/chapter2) | [第二章 习题](https://datawhalechina.github.io/easy-rl/#/chapter2/chapter2_questions&keywords) | | -| [第三章 表格型方法](https://datawhalechina.github.io/easy-rl/#/chapter3/chapter3) | [第三章 习题](https://datawhalechina.github.io/easy-rl/#/chapter3/chapter3_questions&keywords) | [Q-learning算法实战](https://datawhalechina.github.io/easy-rl/#/chapter3/project1) | -| [第四章 策略梯度](https://datawhalechina.github.io/easy-rl/#/chapter4/chapter4) | [第四章 习题](https://datawhalechina.github.io/easy-rl/#/chapter4/chapter4_questions&keywords) | | -| [第五章 近端策略优化 (PPO) 算法](https://datawhalechina.github.io/easy-rl/#/chapter5/chapter5) | [第五章 习题](https://datawhalechina.github.io/easy-rl/#/chapter5/chapter5_questions&keywords) | | -| [第六章 DQN (基本概念)](https://datawhalechina.github.io/easy-rl/#/chapter6/chapter6) | [第六章 习题](https://datawhalechina.github.io/easy-rl/#/chapter6/chapter6_questions&keywords) | | -| [第七章 DQN (进阶技巧)](https://datawhalechina.github.io/easy-rl/#/chapter7/chapter7) | [第七章 习题](https://datawhalechina.github.io/easy-rl/#/chapter7/chapter7_questions&keywords) | [DQN算法实战](https://datawhalechina.github.io/easy-rl/#/chapter7/project2) | -| [第八章 DQN (连续动作)](https://datawhalechina.github.io/easy-rl/#/chapter8/chapter8) | [第八章 习题](https://datawhalechina.github.io/easy-rl/#/chapter8/chapter8_questions&keywords) | | -| [第九章 演员-评论家算法](https://datawhalechina.github.io/easy-rl/#/chapter9/chapter9) | [第九章 习题](https://datawhalechina.github.io/easy-rl/#/chapter9/chapter9_questions&keywords) | | -| [第十章 稀疏奖励](https://datawhalechina.github.io/easy-rl/#/chapter10/chapter10) | [第十章 习题](https://datawhalechina.github.io/easy-rl/#/chapter10/chapter10_questions&keywords) | | -| [第十一章 模仿学习](https://datawhalechina.github.io/easy-rl/#/chapter11/chapter11) | [第十一章 习题](https://datawhalechina.github.io/easy-rl/#/chapter11/chapter11_questions&keywords) | | -| [第十二章 深度确定性策略梯度 (DDPG) 算法](https://datawhalechina.github.io/easy-rl/#/chapter12/chapter12) | [第十二章 习题](https://datawhalechina.github.io/easy-rl/#/chapter12/chapter12_questions&keywords) | [DDPG算法实战](https://datawhalechina.github.io/easy-rl/#/chapter12/project3) | -| [第十三章 AlphaStar 论文解读](https://datawhalechina.github.io/easy-rl/#/chapter13/chapter13) | | | -## 算法实战 - -[点击](https://github.com/datawhalechina/easy-rl/tree/master/codes)或者跳转```codes```文件夹下进入算法实战 - -## 贡献者 - - - - - - - - - -
- pic
- Qi Wang -

教程设计(第1~12章)
中国科学院大学

- 		- pic
- David Young -

习题设计&第13章
清华大学

- 		- pic
- John Jim -

算法实战
北京大学

-
- - -## 致谢 - -特别感谢 [@Sm1les](https://github.com/Sm1les)、[@LSGOMYP](https://github.com/LSGOMYP) 对本项目的帮助与支持。 - -## 关注我们 -
Datawhale是一个专注AI领域的开源组织，以“for the learner，和学习者一起成长”为愿景，构建对学习者最有价值的开源学习社区。关注我们，一起学习成长。
- -## LICENSE -知识共享许可协议
本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可。 - diff --git a/codes/A2C/README.md b/codes/A2C/README.md index e69de29..5856b80 100644 --- a/codes/A2C/README.md +++ b/codes/A2C/README.md @@ -0,0 +1,5 @@ +## A2C + + + +https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f \ No newline at end of file diff --git a/codes/A2C/agent.py b/codes/A2C/agent.py index aafe7c1..9de9aab 100644 --- a/codes/A2C/agent.py +++ b/codes/A2C/agent.py @@ -1,32 +1,27 @@ #!/usr/bin/env python # coding=utf-8 ''' -Author: John +Author: JiangJi Email: johnjim0816@gmail.com -Date: 2020-11-03 20:47:09 -LastEditor: John -LastEditTime: 2021-03-20 17:41:21 +Date: 2021-05-03 22:16:08 +LastEditor: JiangJi +LastEditTime: 2021-05-03 22:23:48 Discription: Environment: ''' -from A2C.model import ActorCritic import torch.optim as optim - +from A2C.model import ActorCritic class A2C: - def __init__(self,state_dim, action_dim, cfg): - self.gamma = 0.99 - self.model = ActorCritic(state_dim, action_dim, hidden_dim=cfg.hidden_dim).to(cfg.device) - self.optimizer = optim.Adam(self.model.parameters(),lr=cfg.lr) - def choose_action(self, state): - dist, value = self.model(state) - action = dist.sample() - return action + def __init__(self,state_dim,action_dim,cfg) -> None: + self.gamma = cfg.gamma + self.device = cfg.device + self.model = ActorCritic(state_dim, action_dim, cfg.hidden_size).to(self.device) + self.optimizer = optim.Adam(self.model.parameters()) + def compute_returns(self,next_value, rewards, masks): R = next_value returns = [] for step in reversed(range(len(rewards))): R = rewards[step] + self.gamma * R * masks[step] returns.insert(0, R) - return returns - def update(self): - pass \ No newline at end of file + return returns \ No newline at end of file diff --git a/codes/A2C/env.py b/codes/A2C/env.py deleted file mode 100644 index 652824b..0000000 --- a/codes/A2C/env.py +++ /dev/null @@ -1,48 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: John -Email: johnjim0816@gmail.com -Date: 2020-10-30 15:39:37 -LastEditor: John -LastEditTime: 2021-03-17 20:19:14 -Discription: -Environment: -''' - -import gym -from A2C.multiprocessing_env import SubprocVecEnv - -# num_envs = 16 -# env = "Pendulum-v0" - -def make_envs(num_envs=16,env="Pendulum-v0"): - ''' 创建多个子环境 - ''' - num_envs = 16 - env = "CartPole-v0" - def make_env(): - def _thunk(): - env = gym.make(env) - return env - - return _thunk - - envs = [make_env() for i in range(num_envs)] - envs = SubprocVecEnv(envs) - return envs -# if __name__ == "__main__": - -# num_envs = 16 -# env = "CartPole-v0" -# def make_env(): -# def _thunk(): -# env = gym.make(env) -# return env - -# return _thunk - -# envs = [make_env() for i in range(num_envs)] -# envs = SubprocVecEnv(envs) -if __name__ == "__main__": - envs = make_envs(num_envs=16,env="CartPole-v0") \ No newline at end of file diff --git a/codes/A2C/main.py b/codes/A2C/main.py deleted file mode 100644 index 9237f48..0000000 --- a/codes/A2C/main.py +++ /dev/null @@ -1,106 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -@Author: John -@Email: johnjim0816@gmail.com -@Date: 2020-06-11 20:58:21 -@LastEditor: John -LastEditTime: 2021-04-05 11:14:39 -@Discription: -@Environment: python 3.7.9 -''' -import sys,os -curr_path = os.path.dirname(__file__) -parent_path=os.path.dirname(curr_path) -sys.path.append(parent_path) # add current terminal path to sys.path - -import torch -import gym -import datetime -from A2C.agent import A2C -from common.utils import save_results,make_dir,del_empty_dir - - - -SEQUENCE = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # 获取当前时间 -SAVED_MODEL_PATH = os.path.split(os.path.abspath(__file__))[0]+"/saved_model/"+SEQUENCE+'/' # 生成保存的模型路径 -if not os.path.exists(os.path.split(os.path.abspath(__file__))[0]+"/saved_model/"): - os.mkdir(os.path.split(os.path.abspath(__file__))[0]+"/saved_model/") -if not os.path.exists(SAVED_MODEL_PATH): - os.mkdir(SAVED_MODEL_PATH) -RESULT_PATH = os.path.split(os.path.abspath(__file__))[0]+"/results/"+SEQUENCE+'/' # 存储reward的路径 -if not os.path.exists(os.path.split(os.path.abspath(__file__))[0]+"/results/"): - os.mkdir(os.path.split(os.path.abspath(__file__))[0]+"/results/") -if not os.path.exists(RESULT_PATH): - os.mkdir(RESULT_PATH) - -class A2CConfig: - def __init__(self): - self.gamma = 0.99 - self.lr = 3e-4 # learnning rate - self.actor_lr = 1e-4 # learnning rate of actor network - self.memory_capacity = 10000 # capacity of replay memory - self.batch_size = 128 - self.train_eps = 200 - self.train_steps = 200 - self.eval_eps = 200 - self.eval_steps = 200 - self.target_update = 4 - self.hidden_dim=256 - self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") - -def train(cfg,env,agent): - print('Start to train ! ') - for i_episode in range(cfg.train_eps): - state = env.reset() - log_probs = [] - values = [] - rewards = [] - masks = [] - entropy = 0 - ep_reward = 0 - for i_step in range(cfg.train_steps): - state = torch.FloatTensor(state).to(cfg.device) - dist, value = agent.model(state) - action = dist.sample() - next_state, reward, done, _ = env.step(action.cpu().numpy()) - ep_reward+=reward - state = next_state - log_prob = dist.log_prob(action) - entropy += dist.entropy().mean() - log_probs.append(log_prob) - values.append(value) - rewards.append(torch.FloatTensor(reward).unsqueeze(1).to(cfg.device)) - masks.append(torch.FloatTensor(1 - done).unsqueeze(1).to(cfg.device)) - if done: - break - print('Episode:{}/{}, Reward:{}, Steps:{}, Done:{}'.format(i_episode+1,cfg.train_eps,ep_reward,i_step+1,done)) - next_state = torch.FloatTensor(next_state).to(cfg.device) - _, next_value =agent.model(next_state) - returns = agent.compute_returns(next_value, rewards, masks) - - log_probs = torch.cat(log_probs) - returns = torch.cat(returns).detach() - values = torch.cat(values) - advantage = returns - values - actor_loss = -(log_probs * advantage.detach()).mean() - critic_loss = advantage.pow(2).mean() - loss = actor_loss + 0.5 * critic_loss - 0.001 * entropy - - agent.optimizer.zero_grad() - loss.backward() - agent.optimizer.step() - - print('Complete training！') - - - -if __name__ == "__main__": - cfg = A2CConfig() - env = gym.make('CartPole-v0') - env.seed(1) # set random seed for env - state_dim = env.observation_space.shape[0] - action_dim = env.action_space.n - agent = A2C(state_dim, action_dim, cfg) - train(cfg,env,agent) - diff --git a/codes/A2C/model.py b/codes/A2C/model.py index 46b59de..5e77d4d 100644 --- a/codes/A2C/model.py +++ b/codes/A2C/model.py @@ -1,36 +1,36 @@ #!/usr/bin/env python # coding=utf-8 ''' -Author: John +Author: JiangJi Email: johnjim0816@gmail.com -Date: 2020-11-03 20:45:25 -LastEditor: John -LastEditTime: 2021-03-20 17:41:33 +Date: 2021-05-03 21:38:54 +LastEditor: JiangJi +LastEditTime: 2021-05-03 21:40:06 Discription: Environment: ''' import torch.nn as nn +import torch.nn.functional as F from torch.distributions import Categorical - class ActorCritic(nn.Module): - def __init__(self, state_dim, action_dim, hidden_dim=256): + def __init__(self, num_inputs, num_outputs, hidden_size, std=0.0): super(ActorCritic, self).__init__() + self.critic = nn.Sequential( - nn.Linear(state_dim, hidden_dim), + nn.Linear(num_inputs, hidden_size), nn.ReLU(), - nn.Linear(hidden_dim, 1) + nn.Linear(hidden_size, 1) ) self.actor = nn.Sequential( - nn.Linear(state_dim, hidden_dim), + nn.Linear(num_inputs, hidden_size), nn.ReLU(), - nn.Linear(hidden_dim, action_dim), + nn.Linear(hidden_size, num_outputs), nn.Softmax(dim=1), ) def forward(self, x): value = self.critic(x) - print(x) probs = self.actor(x) dist = Categorical(probs) return dist, value \ No newline at end of file diff --git a/codes/A2C/multiprocessing_env.py b/codes/A2C/multiprocessing_env.py deleted file mode 100644 index 46bbc08..0000000 --- a/codes/A2C/multiprocessing_env.py +++ /dev/null @@ -1,153 +0,0 @@ -#This code is from openai baseline -#https://github.com/openai/baselines/tree/master/baselines/common/vec_env - -import numpy as np -from multiprocessing import Process, Pipe - -def worker(remote, parent_remote, env_fn_wrapper): - parent_remote.close() - env = env_fn_wrapper.x() - while True: - cmd, data = remote.recv() - if cmd == 'step': - ob, reward, done, info = env.step(data) - if done: - ob = env.reset() - remote.send((ob, reward, done, info)) - elif cmd == 'reset': - ob = env.reset() - remote.send(ob) - elif cmd == 'reset_task': - ob = env.reset_task() - remote.send(ob) - elif cmd == 'close': - remote.close() - break - elif cmd == 'get_spaces': - remote.send((env.observation_space, env.action_space)) - else: - raise NotImplementedError - -class VecEnv(object): - """ - An abstract asynchronous, vectorized environment. - """ - def __init__(self, num_envs, observation_space, action_space): - self.num_envs = num_envs - self.observation_space = observation_space - self.action_space = action_space - - def reset(self): - """ - Reset all the environments and return an array of - observations, or a tuple of observation arrays. - If step_async is still doing work, that work will - be cancelled and step_wait() should not be called - until step_async() is invoked again. - """ - pass - - def step_async(self, actions): - """ - Tell all the environments to start taking a step - with the given actions. - Call step_wait() to get the results of the step. - You should not call this if a step_async run is - already pending. - """ - pass - - def step_wait(self): - """ - Wait for the step taken with step_async(). - Returns (obs, rews, dones, infos): - - obs: an array of observations, or a tuple of - arrays of observations. - - rews: an array of rewards - - dones: an array of "episode done" booleans - - infos: a sequence of info objects - """ - pass - - def close(self): - """ - Clean up the environments' resources. - """ - pass - - def step(self, actions): - self.step_async(actions) - return self.step_wait() - - -class CloudpickleWrapper(object): - """ - Uses cloudpickle to serialize contents (otherwise multiprocessing tries to use pickle) - """ - def __init__(self, x): - self.x = x - def __getstate__(self): - import cloudpickle - return cloudpickle.dumps(self.x) - def __setstate__(self, ob): - import pickle - self.x = pickle.loads(ob) - - -class SubprocVecEnv(VecEnv): - def __init__(self, env_fns, spaces=None): - """ - envs: list of gym environments to run in subprocesses - """ - self.waiting = False - self.closed = False - nenvs = len(env_fns) - self.nenvs = nenvs - self.remotes, self.work_remotes = zip(*[Pipe() for _ in range(nenvs)]) - self.ps = [Process(target=worker, args=(work_remote, remote, CloudpickleWrapper(env_fn))) - for (work_remote, remote, env_fn) in zip(self.work_remotes, self.remotes, env_fns)] - for p in self.ps: - p.daemon = True # if the main process crashes, we should not cause things to hang - p.start() - for remote in self.work_remotes: - remote.close() - - self.remotes[0].send(('get_spaces', None)) - observation_space, action_space = self.remotes[0].recv() - VecEnv.__init__(self, len(env_fns), observation_space, action_space) - - def step_async(self, actions): - for remote, action in zip(self.remotes, actions): - remote.send(('step', action)) - self.waiting = True - - def step_wait(self): - results = [remote.recv() for remote in self.remotes] - self.waiting = False - obs, rews, dones, infos = zip(*results) - return np.stack(obs), np.stack(rews), np.stack(dones), infos - - def reset(self): - for remote in self.remotes: - remote.send(('reset', None)) - return np.stack([remote.recv() for remote in self.remotes]) - - def reset_task(self): - for remote in self.remotes: - remote.send(('reset_task', None)) - return np.stack([remote.recv() for remote in self.remotes]) - - def close(self): - if self.closed: - return - if self.waiting: - for remote in self.remotes: - remote.recv() - for remote in self.remotes: - remote.send(('close', None)) - for p in self.ps: - p.join() - self.closed = True - - def __len__(self): - return self.nenvs diff --git a/codes/A2C/outputs/CartPole-v0/20210503-224814/results/train_ma_rewards.npy b/codes/A2C/outputs/CartPole-v0/20210503-224814/results/train_ma_rewards.npy new file mode 100644 index 0000000..57f4174 Binary files /dev/null and b/codes/A2C/outputs/CartPole-v0/20210503-224814/results/train_ma_rewards.npy differ diff --git a/codes/A2C/outputs/CartPole-v0/20210503-224814/results/train_rewards.npy b/codes/A2C/outputs/CartPole-v0/20210503-224814/results/train_rewards.npy new file mode 100644 index 0000000..bdb3fce Binary files /dev/null and b/codes/A2C/outputs/CartPole-v0/20210503-224814/results/train_rewards.npy differ diff --git a/codes/A2C/outputs/CartPole-v0/20210503-224814/results/train_rewards_curve.png b/codes/A2C/outputs/CartPole-v0/20210503-224814/results/train_rewards_curve.png new file mode 100644 index 0000000..5f1cf9a Binary files /dev/null and b/codes/A2C/outputs/CartPole-v0/20210503-224814/results/train_rewards_curve.png differ diff --git a/codes/A2C/task0_train.py b/codes/A2C/task0_train.py new file mode 100644 index 0000000..69f6976 --- /dev/null +++ b/codes/A2C/task0_train.py @@ -0,0 +1,120 @@ +import sys,os +curr_path = os.path.dirname(__file__) +parent_path = os.path.dirname(curr_path) +sys.path.append(parent_path) # add current terminal path to sys.path + + +import gym +import numpy as np +import torch +import torch.optim as optim +import datetime +from common.multiprocessing_env import SubprocVecEnv +from A2C.model import ActorCritic +from common.utils import save_results, make_dir +from common.plot import plot_rewards + +curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # obtain current time +class A2CConfig: + def __init__(self) -> None: + self.algo='A2C' + self.env= 'CartPole-v0' + self.result_path = curr_path+"/outputs/" +self.env+'/'+curr_time+'/results/' # path to save results + self.model_path = curr_path+"/outputs/" +self.env+'/'+curr_time+'/models/' # path to save models + self.n_envs = 8 + self.gamma = 0.99 + self.hidden_size = 256 + self.lr = 1e-3 # learning rate + self.max_frames = 30000 + self.n_steps = 5 + self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") +def make_envs(env_name): + def _thunk(): + env = gym.make(env_name) + env.seed(2) + return env + return _thunk +def test_env(env,model,vis=False): + state = env.reset() + if vis: env.render() + done = False + total_reward = 0 + while not done: + state = torch.FloatTensor(state).unsqueeze(0).to(cfg.device) + dist, _ = model(state) + next_state, reward, done, _ = env.step(dist.sample().cpu().numpy()[0]) + state = next_state + if vis: env.render() + total_reward += reward + return total_reward +def compute_returns(next_value, rewards, masks, gamma=0.99): + R = next_value + returns = [] + for step in reversed(range(len(rewards))): + R = rewards[step] + gamma * R * masks[step] + returns.insert(0, R) + return returns + + +def train(cfg,envs): + env = gym.make(cfg.env) # a single env + env.seed(10) + state_dim = envs.observation_space.shape[0] + action_dim = envs.action_space.n + model = ActorCritic(state_dim, action_dim, cfg.hidden_size).to(cfg.device) + optimizer = optim.Adam(model.parameters()) + frame_idx = 0 + test_rewards = [] + test_ma_rewards = [] + state = envs.reset() + while frame_idx < cfg.max_frames: + log_probs = [] + values = [] + rewards = [] + masks = [] + entropy = 0 + # rollout trajectory + for _ in range(cfg.n_steps): + state = torch.FloatTensor(state).to(cfg.device) + dist, value = model(state) + action = dist.sample() + next_state, reward, done, _ = envs.step(action.cpu().numpy()) + log_prob = dist.log_prob(action) + entropy += dist.entropy().mean() + log_probs.append(log_prob) + values.append(value) + rewards.append(torch.FloatTensor(reward).unsqueeze(1).to(cfg.device)) + masks.append(torch.FloatTensor(1 - done).unsqueeze(1).to(cfg.device)) + state = next_state + frame_idx += 1 + if frame_idx % 100 == 0: + test_reward = np.mean([test_env(env,model) for _ in range(10)]) + print(f"frame_idx:{frame_idx}, test_reward:{test_reward}") + test_rewards.append(test_reward) + if test_ma_rewards: + test_ma_rewards.append(0.9*test_ma_rewards[-1]+0.1*test_reward) + else: + test_ma_rewards.append(test_reward) + # plot(frame_idx, test_rewards) + next_state = torch.FloatTensor(next_state).to(cfg.device) + _, next_value = model(next_state) + returns = compute_returns(next_value, rewards, masks) + log_probs = torch.cat(log_probs) + returns = torch.cat(returns).detach() + values = torch.cat(values) + advantage = returns - values + actor_loss = -(log_probs * advantage.detach()).mean() + critic_loss = advantage.pow(2).mean() + loss = actor_loss + 0.5 * critic_loss - 0.001 * entropy + optimizer.zero_grad() + loss.backward() + optimizer.step() + return test_rewards, test_ma_rewards +if __name__ == "__main__": + cfg = A2CConfig() + envs = [make_envs(cfg.env) for i in range(cfg.n_envs)] + envs = SubprocVecEnv(envs) # 8 env + rewards,ma_rewards = train(cfg,envs) + make_dir(cfg.result_path,cfg.model_path) + save_results(rewards,ma_rewards,tag='train',path=cfg.result_path) + plot_rewards(rewards,ma_rewards,tag="train",env=cfg.env,algo = cfg.algo,path=cfg.result_path) diff --git a/codes/A2C/utils.py b/codes/A2C/utils.py deleted file mode 100644 index b6d66c6..0000000 --- a/codes/A2C/utils.py +++ /dev/null @@ -1,32 +0,0 @@ -#!/usr/bin/env python -# coding=utf-8 -''' -Author: John -Email: johnjim0816@gmail.com -Date: 2020-10-15 21:31:19 -LastEditor: John -LastEditTime: 2020-11-03 17:05:48 -Discription: -Environment: -''' -import os -import numpy as np -import datetime - -SEQUENCE = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") -SAVED_MODEL_PATH = os.path.split(os.path.abspath(__file__))[0]+"/saved_model/"+SEQUENCE+'/' -RESULT_PATH = os.path.split(os.path.abspath(__file__))[0]+"/results/"+SEQUENCE+'/' - - -def save_results(rewards,moving_average_rewards,ep_steps,path=RESULT_PATH): - if not os.path.exists(path): # 检测是否存在文件夹 - os.mkdir(path) - np.save(RESULT_PATH+'rewards_train.npy', rewards) - np.save(RESULT_PATH+'moving_average_rewards_train.npy', moving_average_rewards) - np.save(RESULT_PATH+'steps_train.npy',ep_steps ) - -def save_model(agent,model_path='./saved_model'): - if not os.path.exists(model_path): # 检测是否存在文件夹 - os.mkdir(model_path) - agent.save_model(model_path+'checkpoint.pth') - print('model saved！') \ No newline at end of file diff --git a/codes/DDPG/README.md b/codes/DDPG/README.md index 351615b..bbcedcc 100644 --- a/codes/DDPG/README.md +++ b/codes/DDPG/README.md @@ -1,5 +1,7 @@ # DDPG +#TODO + ## 伪代码 ![image-20210320151900695](assets/image-20210320151900695.png) \ No newline at end of file diff --git a/codes/DQN/README.md b/codes/DQN/README.md index 530d666..45612be 100644 --- a/codes/DQN/README.md +++ b/codes/DQN/README.md @@ -1,5 +1,5 @@ # DQN - +#TODO ## 原理简介 DQN是Q-leanning算法的优化和延伸，Q-leaning中使用有限的Q表存储值的信息，而DQN中则用神经网络替代Q表存储信息，这样更适用于高维的情况，相关知识基础可参考[datawhale李宏毅笔记-Q学习](https://datawhalechina.github.io/easy-rl/#/chapter6/chapter6)。 diff --git a/codes/DQN/agent.py b/codes/DQN/agent.py index 7890270..669295f 100644 --- a/codes/DQN/agent.py +++ b/codes/DQN/agent.py @@ -5,7 +5,7 @@ @Email: johnjim0816@gmail.com @Date: 2020-06-12 00:50:49 @LastEditor: John -LastEditTime: 2021-03-30 17:01:26 +LastEditTime: 2021-04-29 22:19:18 @Discription: @Environment: python 3.7.7 ''' @@ -39,6 +39,8 @@ class DQN: hidden_dim=cfg.hidden_dim).to(self.device) self.target_net = MLP(state_dim, action_dim, hidden_dim=cfg.hidden_dim).to(self.device) + for target_param, param in zip(self.target_net.parameters(), self.policy_net.parameters()): + target_param.data.copy_(param.data) self.optimizer = optim.Adam(self.policy_net.parameters(), lr=cfg.lr) self.loss = 0 self.memory = ReplayBuffer(cfg.memory_capacity) @@ -48,21 +50,16 @@ class DQN: ''' self.frame_idx += 1 if random.random() > self.epsilon(self.frame_idx): - with torch.no_grad(): - # 先转为张量便于丢给神经网络,state元素数据原本为float64 - # 注意state=torch.tensor(state).unsqueeze(0)跟state=torch.tensor([state])等价 - state = torch.tensor( - [state], device=self.device, dtype=torch.float32) - # 如tensor([[-0.0798, -0.0079]], grad_fn=) - q_value = self.policy_net(state) - # tensor.max(1)返回每行的最大值以及对应的下标， - # 如torch.return_types.max(values=tensor([10.3587]),indices=tensor([0])) - # 所以tensor.max(1)[1]返回最大值对应的下标，即action - action = q_value.max(1)[1].item() + action = self.predict(state) else: action = random.randrange(self.action_dim) return action - + def predict(self,state): + with torch.no_grad(): + state = torch.tensor([state], device=self.device, dtype=torch.float32) + q_values = self.policy_net(state) + action = q_values.max(1)[1].item() + return action def update(self): if len(self.memory) < self.batch_size: @@ -109,3 +106,5 @@ class DQN: def load(self, path): self.target_net.load_state_dict(torch.load(path+'dqn_checkpoint.pth')) + for target_param, param in zip(self.target_net.parameters(), self.policy_net.parameters()): + param.data.copy_(target_param.data) diff --git a/codes/DQN/outputs/CartPole-v0/20210418-143542/models/dqn_checkpoint.pth b/codes/DQN/outputs/CartPole-v0/20210418-143542/models/dqn_checkpoint.pth deleted file mode 100644 index 3bc041d..0000000 Binary files a/codes/DQN/outputs/CartPole-v0/20210418-143542/models/dqn_checkpoint.pth and /dev/null differ diff --git a/codes/DQN/outputs/CartPole-v0/20210418-143542/results/ma_rewards_train.npy b/codes/DQN/outputs/CartPole-v0/20210418-143542/results/ma_rewards_train.npy deleted file mode 100644 index 152ad7a..0000000 Binary files a/codes/DQN/outputs/CartPole-v0/20210418-143542/results/ma_rewards_train.npy and /dev/null differ diff --git a/codes/DQN/outputs/CartPole-v0/20210418-143542/results/rewards_curve_train.png b/codes/DQN/outputs/CartPole-v0/20210418-143542/results/rewards_curve_train.png deleted file mode 100644 index ad42573..0000000 Binary files a/codes/DQN/outputs/CartPole-v0/20210418-143542/results/rewards_curve_train.png and /dev/null differ diff --git a/codes/DQN/outputs/CartPole-v0/20210429-222132/models/dqn_checkpoint.pth b/codes/DQN/outputs/CartPole-v0/20210429-222132/models/dqn_checkpoint.pth new file mode 100644 index 0000000..2b2200e Binary files /dev/null and b/codes/DQN/outputs/CartPole-v0/20210429-222132/models/dqn_checkpoint.pth differ diff --git a/codes/DQN/outputs/CartPole-v0/20210429-222132/results/eval_ma_rewards.npy b/codes/DQN/outputs/CartPole-v0/20210429-222132/results/eval_ma_rewards.npy new file mode 100644 index 0000000..e25eb51 Binary files /dev/null and b/codes/DQN/outputs/CartPole-v0/20210429-222132/results/eval_ma_rewards.npy differ diff --git a/codes/DQN/outputs/CartPole-v0/20210429-222132/results/eval_rewards.npy b/codes/DQN/outputs/CartPole-v0/20210429-222132/results/eval_rewards.npy new file mode 100644 index 0000000..2fc0e4e Binary files /dev/null and b/codes/DQN/outputs/CartPole-v0/20210429-222132/results/eval_rewards.npy differ diff --git a/codes/DQN/outputs/CartPole-v0/20210429-222132/results/eval_rewards_curve.png b/codes/DQN/outputs/CartPole-v0/20210429-222132/results/eval_rewards_curve.png new file mode 100644 index 0000000..295fdac Binary files /dev/null and b/codes/DQN/outputs/CartPole-v0/20210429-222132/results/eval_rewards_curve.png differ diff --git a/codes/DQN/outputs/CartPole-v0/20210429-222132/results/train_ma_rewards.npy b/codes/DQN/outputs/CartPole-v0/20210429-222132/results/train_ma_rewards.npy new file mode 100644 index 0000000..f71e613 Binary files /dev/null and b/codes/DQN/outputs/CartPole-v0/20210429-222132/results/train_ma_rewards.npy differ diff --git a/codes/DQN/outputs/CartPole-v0/20210429-222132/results/train_rewards.npy b/codes/DQN/outputs/CartPole-v0/20210429-222132/results/train_rewards.npy new file mode 100644 index 0000000..fa9ffc3 Binary files /dev/null and b/codes/DQN/outputs/CartPole-v0/20210429-222132/results/train_rewards.npy differ diff --git a/codes/DQN/outputs/CartPole-v0/20210429-222132/results/train_rewards_curve.png b/codes/DQN/outputs/CartPole-v0/20210429-222132/results/train_rewards_curve.png new file mode 100644 index 0000000..a6857d3 Binary files /dev/null and b/codes/DQN/outputs/CartPole-v0/20210429-222132/results/train_rewards_curve.png differ diff --git a/codes/DQN/outputs/CartPole-v0/20210429-222344/models/dqn_checkpoint.pth b/codes/DQN/outputs/CartPole-v0/20210429-222344/models/dqn_checkpoint.pth new file mode 100644 index 0000000..0b192d4 Binary files /dev/null and b/codes/DQN/outputs/CartPole-v0/20210429-222344/models/dqn_checkpoint.pth differ diff --git a/codes/DQN/outputs/CartPole-v0/20210429-222344/results/eval_ma_rewards.npy b/codes/DQN/outputs/CartPole-v0/20210429-222344/results/eval_ma_rewards.npy new file mode 100644 index 0000000..f91ed3c Binary files /dev/null and b/codes/DQN/outputs/CartPole-v0/20210429-222344/results/eval_ma_rewards.npy differ diff --git a/codes/DQN/outputs/CartPole-v0/20210429-222344/results/eval_rewards.npy b/codes/DQN/outputs/CartPole-v0/20210429-222344/results/eval_rewards.npy new file mode 100644 index 0000000..51e06c2 Binary files /dev/null and b/codes/DQN/outputs/CartPole-v0/20210429-222344/results/eval_rewards.npy differ diff --git a/codes/DQN/outputs/CartPole-v0/20210429-222344/results/eval_rewards_curve.png b/codes/DQN/outputs/CartPole-v0/20210429-222344/results/eval_rewards_curve.png new file mode 100644 index 0000000..0327b47 Binary files /dev/null and b/codes/DQN/outputs/CartPole-v0/20210429-222344/results/eval_rewards_curve.png differ diff --git a/codes/DQN/outputs/CartPole-v0/20210429-222344/results/train_ma_rewards.npy b/codes/DQN/outputs/CartPole-v0/20210429-222344/results/train_ma_rewards.npy new file mode 100644 index 0000000..fadc523 Binary files /dev/null and b/codes/DQN/outputs/CartPole-v0/20210429-222344/results/train_ma_rewards.npy differ diff --git a/codes/DQN/outputs/CartPole-v0/20210418-143542/results/rewards_train.npy b/codes/DQN/outputs/CartPole-v0/20210429-222344/results/train_rewards.npy similarity index 60% rename from codes/DQN/outputs/CartPole-v0/20210418-143542/results/rewards_train.npy rename to codes/DQN/outputs/CartPole-v0/20210429-222344/results/train_rewards.npy index 58fb2b8..61cf9fc 100644 Binary files a/codes/DQN/outputs/CartPole-v0/20210418-143542/results/rewards_train.npy and b/codes/DQN/outputs/CartPole-v0/20210429-222344/results/train_rewards.npy differ diff --git a/codes/DQN/outputs/CartPole-v0/20210429-222344/results/train_rewards_curve.png b/codes/DQN/outputs/CartPole-v0/20210429-222344/results/train_rewards_curve.png new file mode 100644 index 0000000..b9667f1 Binary files /dev/null and b/codes/DQN/outputs/CartPole-v0/20210429-222344/results/train_rewards_curve.png differ diff --git a/codes/DQN/main.py b/codes/DQN/task0_train.py similarity index 62% rename from codes/DQN/main.py rename to codes/DQN/task0_train.py index cd22aad..fc13983 100644 --- a/codes/DQN/main.py +++ b/codes/DQN/task0_train.py @@ -5,7 +5,7 @@ @Email: johnjim0816@gmail.com @Date: 2020-06-12 00:48:57 @LastEditor: John -LastEditTime: 2021-04-29 02:02:12 +LastEditTime: 2021-04-29 22:23:38 @Discription: @Environment: python 3.7.7 ''' @@ -36,21 +36,28 @@ class DQNConfig: '/'+curr_time+'/results/' # path to save results self.model_path = curr_path+"/outputs/" + self.env + \ '/'+curr_time+'/models/' # path to save results + self.train_eps = 300 # 训练的episode数目 + self.eval_eps = 50 # number of episodes for evaluating self.gamma = 0.95 - self.epsilon_start = 1 # e-greedy策略的初始epsilon + self.epsilon_start = 0.90 # e-greedy策略的初始epsilon self.epsilon_end = 0.01 self.epsilon_decay = 500 self.lr = 0.0001 # learning rate - self.memory_capacity = 10000 # Replay Memory容量 - self.batch_size = 32 - self.train_eps = 300 # 训练的episode数目 + self.memory_capacity = 100000 # Replay Memory容量 + self.batch_size = 64 self.target_update = 2 # target net的更新频率 - self.eval_eps = 20 # 测试的episode数目 self.device = torch.device( "cuda" if torch.cuda.is_available() else "cpu") # 检测gpu self.hidden_dim = 256 # 神经网络隐藏层维度 - +def env_agent_config(cfg,seed=1): + env = gym.make(cfg.env) + env.seed(seed) + state_dim = env.observation_space.shape[0] + action_dim = env.action_space.n + agent = DQN(state_dim,action_dim,cfg) + return env,agent + def train(cfg, env, agent): print('Start to train !') print(f'Env:{cfg.env}, Algorithm:{cfg.algo}, Device:{cfg.device}') @@ -60,13 +67,15 @@ def train(cfg, env, agent): state = env.reset() done = False ep_reward = 0 - while not done: + while True: action = agent.choose_action(state) next_state, reward, done, _ = env.step(action) ep_reward += reward agent.memory.push(state, action, reward, next_state, done) state = next_state agent.update() + if done: + break if i_episode % cfg.target_update == 0: agent.target_net.load_state_dict(agent.policy_net.state_dict()) print('Episode:{}/{}, Reward:{}'.format(i_episode+1, cfg.train_eps, ep_reward)) @@ -79,17 +88,39 @@ def train(cfg, env, agent): print('Complete training！') return rewards, ma_rewards +def eval(cfg,env,agent): + rewards = [] # 记录所有episode的reward + ma_rewards = [] # 滑动平均的reward + for i_ep in range(cfg.eval_eps): + ep_reward = 0 # 记录每个episode的reward + state = env.reset() # 重置环境, 重新开一局（即开始新的一个episode） + while True: + action = agent.predict(state) # 根据算法选择一个动作 + next_state, reward, done, _ = env.step(action) # 与环境进行一个交互 + state = next_state # 存储上一个观察值 + ep_reward += reward + if done: + break + rewards.append(ep_reward) + if ma_rewards: + ma_rewards.append(ma_rewards[-1]*0.9+ep_reward*0.1) + else: + ma_rewards.append(ep_reward) + print(f"Episode:{i_ep+1}/{cfg.eval_eps}, reward:{ep_reward:.1f}") + return rewards,ma_rewards if __name__ == "__main__": cfg = DQNConfig() - env = gym.make(cfg.env) - env.seed(1) - state_dim = env.observation_space.shape[0] - action_dim = env.action_space.n - agent = DQN(state_dim, action_dim, cfg) + env,agent = env_agent_config(cfg,seed=1) rewards, ma_rewards = train(cfg, env, agent) make_dir(cfg.result_path, cfg.model_path) agent.save(path=cfg.model_path) save_results(rewards, ma_rewards, tag='train', path=cfg.result_path) plot_rewards(rewards, ma_rewards, tag="train", algo=cfg.algo, path=cfg.result_path) + + env,agent = env_agent_config(cfg,seed=10) + agent.load(path=cfg.model_path) + rewards,ma_rewards = eval(cfg,env,agent) + save_results(rewards,ma_rewards,tag='eval',path=cfg.result_path) + plot_rewards(rewards,ma_rewards,tag="eval",env=cfg.env,algo = cfg.algo,path=cfg.result_path) diff --git a/codes/QLearning/README.md b/codes/QLearning/README.md new file mode 100644 index 0000000..cc8ef5e --- /dev/null +++ b/codes/QLearning/README.md @@ -0,0 +1,3 @@ +# Q-learning + +#TODO diff --git a/codes/QLearning/agent.py b/codes/QLearning/agent.py index 2d2cb97..3a4eadb 100644 --- a/codes/QLearning/agent.py +++ b/codes/QLearning/agent.py @@ -5,8 +5,8 @@ Author: John Email: johnjim0816@gmail.com Date: 2020-09-11 23:03:00 LastEditor: John -LastEditTime: 2021-03-26 16:51:01 -Discription: +LastEditTime: 2021-04-29 16:59:41 +Discription: use defaultdict to define Q table Environment: ''' import numpy as np @@ -15,7 +15,7 @@ import torch from collections import defaultdict class QLearning(object): - def __init__(self, + def __init__(self,state_dim, action_dim,cfg): self.action_dim = action_dim # dimension of acgtion self.lr = cfg.lr # learning rate @@ -26,17 +26,20 @@ class QLearning(object): self.epsilon_end = cfg.epsilon_end self.epsilon_decay = cfg.epsilon_decay self.Q_table = defaultdict(lambda: np.zeros(action_dim)) # A nested dictionary that maps state -> (action -> action-value) + def choose_action(self, state): self.sample_count += 1 self.epsilon = self.epsilon_end + (self.epsilon_start - self.epsilon_end) * \ math.exp(-1. * self.sample_count / self.epsilon_decay) # e-greedy policy if np.random.uniform(0, 1) > self.epsilon: - action = np.argmax(self.Q_table[str(state)]) + action = self.predict(state) else: action = np.random.choice(self.action_dim) return action - + def predict(self,state): + action = np.argmax(self.Q_table[str(state)]) + return action def update(self, state, action, reward, next_state, done): Q_predict = self.Q_table[str(state)][action] if done: diff --git a/codes/QLearning/agent1.py b/codes/QLearning/agent1.py new file mode 100644 index 0000000..4f025ac --- /dev/null +++ b/codes/QLearning/agent1.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python +# coding=utf-8 +''' +Author: John +Email: johnjim0816@gmail.com +Date: 2020-09-11 23:03:00 +LastEditor: John +LastEditTime: 2021-04-29 17:02:00 +Discription: +Environment: +''' +# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import numpy as np +import math +#!/usr/bin/env python +# coding=utf-8 +''' +Author: John +Email: johnjim0816@gmail.com +Date: 2020-09-11 23:03:00 +LastEditor: John +LastEditTime: 2021-04-29 16:45:33 +Discription: use np array to define Q table +Environment: +''' +import numpy as np +import math + +class QLearning(object): + def __init__(self, + state_dim,action_dim,cfg): + self.action_dim = action_dim # dimension of acgtion + self.lr = cfg.lr # learning rate + self.gamma = cfg.gamma + self.epsilon = 0 + self.sample_count = 0 + self.epsilon_start = cfg.epsilon_start + self.epsilon_end = cfg.epsilon_end + self.epsilon_decay = cfg.epsilon_decay + self.Q_table = np.zeros((state_dim, action_dim)) # Q表 + + def choose_action(self, state): + self.sample_count += 1 + self.epsilon = self.epsilon_end + (self.epsilon_start - self.epsilon_end) * \ + math.exp(-1. * self.sample_count / self.epsilon_decay) + if np.random.uniform(0, 1) > self.epsilon: # 随机选取0-1之间的值，如果大于epsilon就按照贪心策略选取action，否则随机选取 + action = self.predict(state) + else: + action = np.random.choice(self.action_dim) #有一定概率随机探索选取一个动作 + return action + + def predict(self, state): + '''根据输入观测值，采样输出的动作值，带探索，测试模型时使用 + ''' + Q_list = self.Q_table[state, :] + Q_max = np.max(Q_list) + action_list = np.where(Q_list == Q_max)[0] + action = np.random.choice(action_list) # Q_max可能对应多个 action ，可以随机抽取一个 + return action + + def update(self, state, action, reward, next_state, done): + Q_predict = self.Q_table[state, action] + if done: + Q_target = reward # 没有下一个状态了 + else: + Q_target = reward + self.gamma * np.max( + self.Q_table[next_state, :]) # Q_table-learning + self.Q_table[state, action] += self.lr * (Q_target - Q_predict) # 修正q + def save(self,path): + np.save(path+"Q_table.npy", self.Q_table) + def load(self, path): + self.Q_table = np.load(path+"Q_table.npy") + + diff --git a/codes/QLearning/main.ipynb b/codes/QLearning/main.ipynb deleted file mode 100644 index 91d2a6b..0000000 --- a/codes/QLearning/main.ipynb +++ /dev/null @@ -1,152 +0,0 @@ -{ - "metadata": { - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.10-final" - }, - "orig_nbformat": 2, - "kernelspec": { - "name": "python3", - "display_name": "Python 3.7.10 64-bit ('py37': conda)", - "metadata": { - "interpreter": { - "hash": "fbea1422c2cf61ed9c0cfc03f38f71cc9083cc288606edc4170b5309b352ce27" - } - } - } - }, - "nbformat": 4, - "nbformat_minor": 2, - "cells": [ - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "import sys\n", - "from pathlib import Path\n", - "curr_path = str(Path().absolute())\n", - "parent_path = str(Path().absolute().parent)\n", - "sys.path.append(parent_path) # add current terminal path to sys.path\n", - "\n", - "import gym\n", - "\n", - "from envs.gridworld_env import CliffWalkingWapper, FrozenLakeWapper\n", - "from QLearning.agent import QLearning\n", - "from common.plot import plot_rewards\n", - "from common.utils import save_results" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [], - "source": [ - "class QlearningConfig:\n", - " '''训练相关参数'''\n", - " def __init__(self):\n", - " self.train_eps = 200 # 训练的episode数目\n", - " self.gamma = 0.9 # reward的衰减率\n", - " self.epsilon_start = 0.99 # e-greedy策略中初始epsilon\n", - " self.epsilon_end = 0.01 # e-greedy策略中的终止epsilon\n", - " self.epsilon_decay = 200 # e-greedy策略中epsilon的衰减率\n", - " self.lr = 0.1 # learning rate" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "def train(cfg,env,agent):\n", - " rewards = [] \n", - " ma_rewards = [] # moving average reward\n", - " steps = [] # 记录所有episode的steps\n", - " for i_episode in range(cfg.train_eps):\n", - " ep_reward = 0 # 记录每个episode的reward\n", - " ep_steps = 0 # 记录每个episode走了多少step\n", - " state = env.reset() # 重置环境, 重新开一局（即开始新的一个episode）\n", - " while True:\n", - " action = agent.choose_action(state) # 根据算法选择一个动作\n", - " next_state, reward, done, _ = env.step(action) # 与环境进行一次动作交互\n", - " agent.update(state, action, reward, next_state, done) # Q-learning算法更新\n", - " state = next_state # 存储上一个观察值\n", - " ep_reward += reward\n", - " ep_steps += 1 # 计算step数\n", - " if done:\n", - " break\n", - " steps.append(ep_steps)\n", - " rewards.append(ep_reward)\n", - " if ma_rewards:\n", - " ma_rewards.append(ma_rewards[-1]*0.9+ep_reward*0.1)\n", - " else:\n", - " ma_rewards.append(ep_reward)\n", - " if (i_episode+1)%10==0:\n", - " print(\"Episode:{}/{}: reward:{:.1f}\".format(i_episode+1, cfg.train_eps,ep_reward))\n", - " return rewards,ma_rewards" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "Episode:10/200: reward:-82.0\n", - "Episode:20/200: reward:-59.0\n", - "Episode:30/200: reward:-50.0\n", - "Episode:40/200: reward:-32.0\n", - "Episode:50/200: reward:-102.0\n", - "Episode:60/200: reward:-151.0\n", - "Episode:70/200: reward:-34.0\n", - "Episode:80/200: reward:-71.0\n", - "Episode:90/200: reward:-34.0\n", - "Episode:100/200: reward:-26.0\n", - "Episode:110/200: reward:-32.0\n", - "Episode:120/200: reward:-48.0\n", - "Episode:130/200: reward:-25.0\n", - "Episode:140/200: reward:-31.0\n", - "Episode:150/200: reward:-38.0\n", - "Episode:160/200: reward:-47.0\n", - "Episode:170/200: reward:-29.0\n", - "Episode:180/200: reward:-36.0\n", - "Episode:190/200: reward:-21.0\n", - "Episode:200/200: reward:-34.0\n" - ] - }, - { - "output_type": "display_data", - "data": { - "text/plain": "
", - "image/svg+xml": "\n\n\n\n \n \n \n \n 2021-03-31T18:50:18.442345\n image/svg+xml\n \n \n Matplotlib v3.3.4, https://matplotlib.org/\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n", - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEXCAYAAABCjVgAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABc4klEQVR4nO3dd3wUZf7A8c9sT7JJliSbRgmE0DuGKhipEiB0PBBB4RS7iCcSEDlBEUFUwAInd8rvsAAiRZRYwEOQ3ov0mgRCOqRny8zvj5CVkIQU0mSf9+vlSzI7O/PdZ2fnO0+ZZyRFURQEQRAEAVBVdwCCIAhCzSGSgiAIguAgkoIgCILgIJKCIAiC4CCSgiAIguAgkoIgCILgIJLCX8yHH37I7Nmzq2RfTz75JOfOnauSfVW3JUuW8OCDDzJt2rRCr9lsNpYsWUL//v0ZMGAAAwYMYNasWVy/fr3M+9mzZw+tW7dm8ODBDBkyhMGDBzNs2DB+/fXXEt/bs2dPjh07xrFjx3jxxRfLvO+iNGnShIiICAYPHuz477XXXgNg8ODBpKWllXpb6enpjBs3rsjXPvroI/7+978XWn78+HG6dOnCkSNHSvxMixYtYv369Y7tbd68udA6sbGxNGnShDFjxhR6bdq0aTRp0oSUlBQA7HY7n3/+OcOGDWPw4MH079+fd999F4vFUuT+y7p+adypzO5k7dq1PPXUU+Xe7x0pwl/K4sWLlVmzZlV3GPecnj17Kvv27SvytRdeeEF5+eWXldTUVEVRFMVisSiffvqp0rdvXyU9Pb1M+9m9e7cyYMCAAstOnjyptG3bVklOTr7je3v06KEcPXq0TPsrSePGjUvcb2nFxMQobdu2LfK1+Ph4pWXLlsrVq1cLLH/99deV9957r8z7evTRR5WoqKgiY2jVqpXStWtXJTY21rE8MzNT6dOnT4HPO2PGDOWFF15Q0tLSHOs888wzyiuvvFLkPsu6fmncqczu5Ntvv1UmTpxY7v3eiaZyUk3NJssyb7/9NkeOHCEzMxNFUXjrrbdo3LgxYWFh/PTTT5jNZgAefvhhnnvuObp06cKCBQvYt28fdrud5s2bM2PGDIxGIz179qR169acPn2al19+GY1Gw7/+9S8sFgspKSkMGTKEl156CYBPP/2UNWvW4ObmRmhoKFu2bOHXX3/FYrEUu/3ixMfHM3v2bOLi4rBarQwYMICnn34agKVLl7J582Zyc3PJzs5m6tSp9OnThw8//JDDhw+TkJBAkyZNCAoK4sqVKyQmJnLlyhW8vLz44IMP8PPzo2fPnixatIisrCw++OAD6taty9mzZ7FYLMycOZPOnTuTkpLCtGnTiI6OxmQyYTabadSoES+88EKBWDMzM3nrrbc4ePAgarWa3r17M3nyZKZNm0ajRo0cV5GRkZGOv28t1xdeeIElS5awceNGANLS0ujVqxebN28mJyen2HK41bVr13jjjTe4cuUKiqIwZMgQnnjiCV566SXi4+N57bXXmDRpEv3793e859ChQxw+fJhff/0VjSbv56LVannyySc5ePAgK1eu5IknnqBVq1ZMnDiRHTt2kJCQwLhx43j88cdLdTw2bdoUg8HAlStXcHd355133mHXrl2o1Wpat27NtGnTChwHe/bs4c033+T7778vslyffvppwsLCWL16NQ0aNABg/PjxjBkzht69e5cqJsirRezatYutW7eyZs0asrOzMRqNvP/++0ydOpXU1FQAwsLCeOmll5g2bRo5OTkMHjyYtWvXolarHdvy9fWlZ8+erF27lueeew7IOyaioqLYsGFDgc+0f/9+3nnnHWRZBuCpp57ioYcechwbBoOB48ePM3/+fNRqNX369CkQt1qtJjw8nI0bNzqOg59//plevXrx2WefARATE8PGjRv5/fffHWXr6urKrFmzOHToUKGyKM366enpzJo1i1OnTiFJEt27d3ecD4o7Pm4vszZt2tCrVy9OnTrFggULyM3NZf78+WRnZ6PVannppZd44IEHSv0dlodTNh8dOXKEhIQEVq1axaZNmxg6dCjLli3D3d2dPn368N133wFw/vx5EhMT6d69O59++ilqtZq1a9fy3Xff4evry4IFCxzbbNSoEVFRUfTu3ZvPPvuMd955h7Vr17Jq1So+/fRTUlJS2L59O2vXrmXNmjWsXbuWzMxMx/tL2n5RpkyZwvDhwx3b3LlzJ5s2beLKlSvs3LmTL774go0bNzJ58mQWL17seN+VK1dYt26dY/v79+9n0aJF/Pjjj3h4eLBq1apC+zp69CgTJkxg/fr1jBgxgo8++giAt956i5CQEKKioli0aBEHDx4sMtbFixeTm5vLpk2bWL9+PQcPHmTv3r0lflf55RoeHk5mZibHjh0D4PvvvycsLAxPT89iy+F2r7zyCp06dWLjxo18/fXXfPfdd/zwww8sXLjQUd63JgTISwpt27Z1JIRbde3alQMHDgBgsVioVasWK1euZPHixbz33nvk5uaW+Pkg74SlUqkICQlhyZIlJCQksGHDBjZs2IAsy8yfP7/Y9xZVrseOHWPIkCF88803AERHR3Px4kV69OhR5DYee+yxAs1HycnJhdY5d+4cK1asYMWKFaxevZo6deqwbt06vvzySy5fvkx6ejpz587FYDCwYcOGAgkh35gxY1i7di3KzUkUfvjhBzp27EhgYGCB9T788EPGjx/P2rVrefvtt9m9e3eh7bRs2ZJXX321UELIN2TIEMfvGGD9+vUMHTrU8feJEycICQkpdNFlNpvp27dvoe2VZv233noLk8nExo0b+fbbbzl9+rQjCRV3fNxeZlarlR49evDTTz9Rp04dXnzxRV577TU2btzIvHnzmDJlCjExMUV+5orilDWFdu3a4enpycqVK4mJiWHPnj24ubkBMHLkSGbNmsXf//53vv32W4YNG4ZKpWLr1q2kp6ezc+dOAKxWK97e3o5thoaGAiBJEkuXLmXr1q18//33nD9/HkVRyM7O5rfffqNfv354eHgAeQd3/gFf0vZvl5WVxb59+7hx4waLFi1yLDt16hT9+/dn3rx5bNy4kcuXLztqRPluP8l17NjRcbA3b96cGzduFNpfYGAgzZo1c6yzbt06AH777TfHv319fenXr1+R8e7cuZNp06ahVqtRq9V88cUXAI73FufWch0xYgTr1q2jVatWrF27lilTppRYDreW18GDBx0/Und3d4YNG8a2bdsYMGDAHWOw2WxFLrdYLEiS5Pi7V69eALRo0QKLxUJWVhZ6vb7Q+6Kjoxk8eLBj2/7+/nzyySe4uLiwbds2Jk+ejFarBWDs2LGOK+uiFFeuvr6+PProo0yePJlVq1YxYsSIIk/UAP/3f/+Hl5fXHcugSZMmjmOke/fuTJw4kbi4OLp27co//vEP3N3dizxubtWxY0dcXFzYvXs3Xbp0YdWqVfzjH/8otF54eDizZ8/m119/pWvXrrz88st33G5RWrZsiUql4vjx43h7e5OZmUnjxo0dr6tUKkdNpDRKs/62bdv4+uuvkSQJnU7HqFGj+L//+z8mTpwIFH18FCX/mD969Cj16tWjTZs2QN4FUvv27dm7d2+B466iOWVS2Lp1K3PmzGH8+PH06tWL4OBgx1VFaGgoNpuNo0eP8v3337Ny5Uogr8lp+vTphIWFAXlV31uvBF1dXYG8k8/QoUPp3bs3oaGhDB8+nM2bN6MoChqNxnGVBBT4kZa0/dvJsoyiKKxcuRIXFxcAUlJS0Ov1/PHHHzz77LM8/vjj3H///XTo0IFZs2YVijWfwWBw/FuSpAIxlrTO7Z9JpSq68qnRaAocyHFxcRgMhkL7s1qtBd53a6zDhw9nyJAhjBw5kvT0dDp16kRGRkax5VBUed2+rLgTfr727dvz2WefkZ2d7dh+vj179tChQwfH3/n7zP+ciqLw2muvcfz4cQBGjRpFcHAw9erVY8OGDUXu7/YTjyzLhcrkVsWVa4MGDWjSpAlbtmxh48aNjlpDed36PbRu3ZotW7awa9cudu/ezciRI/n444/x9fV1rLNlyxZH7dTX15dly5YBMHr0aNasWYPJZCIrK4uuXbsW2teoUaPo0aMHO3bsYPv27Xz00UcFrvpLa9CgQXz33Xd4eXk5kvCtn+HChQtkZGQUuPqPj4/n9ddfZ/HixQWO+dKsX9R3d+vxVdTxUZT8si4qCSmKgs1mc1w0VAanbD7asWMHPXr04JFHHqFVq1Zs3rwZu93ueH3kyJG8+eabNGnSxFG17datG19++SUWiwVZlnn99dd5//33C2378uXLZGRk8NJLL9GzZ0/27t3reE9YWBg///wz6enpAKxZs8bxvtJuP5/RaKRt27Z8/vnnQF4b++jRo9myZQv79u2jZcuWjB8/no4dO7Jly5YCn68ihYWFOT5HamoqmzdvLvIqpkuXLqxbtw5ZlrFYLLz44ovs27ePWrVqOU6aKSkp7N+/v9h9+fn50aZNG2bOnMmIESOAO5fDrYxGI23atOHLL78E8tp/169fX+RJ6VZt27alY8eOREZGOq6E7XY7S5Ys4dKlS4waNeqO758zZ46jKWj06NF3XBfyrsJXrlyJ1WpFlmW+/PJL7r///mLXL65cAR555BHmz59PmzZt8PPzK3HfpbVgwQI++eQTevfuzWuvvUZISAiXLl1Co9Fgt9tRFIVevXo5Pnd+QoC8EU179uzhq6++4pFHHily+6NGjeLkyZMMGzaMN998k7S0tEK1ELVaXWJCHzx4MD/++CObNm1i4MCBBV7z8/MjIiKC6dOnk5GRAUBGRgZvvPEGJpOpQEIo7fr5v2FFUbBYLKxevbrE4+vWMrtdmzZtuHjxIkePHgXg7Nmz7Nu3j44dO95xm3fLKZPCqFGj2LdvHxEREfztb3+jbt26xMbGOjLzkCFDOHnyJCNHjnS859lnn6V27doMHTqU/v37oygKkZGRhbbdpEkTHnzwQcLDwxk6dCi//vorISEhXL58mS5duvDwww/zt7/9jWHDhpGenu64+izt9m+1YMECjhw5QkREBCNHjmTgwIEMGjSIgQMHkpqaSv/+/Rk2bBiurq7cuHHDcTBXpGnTpnHhwgUiIiJ48cUXCQwMLPSDAnj++efRarWOoZhhYWH07duXsWPHkpiYyEMPPcSUKVNKPOBHjhzJyZMnC7QPF1cOt1uwYAG7du0iIiKCESNG0LdvX4YNG1biZ3z33Xdp1aoVjz76KBEREfTv35+4uDhWrlyJu7t7KUqp9J555hl8fHwYMmQI4eHh2Gw2xxDRohRXrgA9evQgKyurxMRVVo899hinTp1i4MCBDB8+nDp16jBw4EDMZjPNmzcnPDzc0Ql9O6PRSJ8+fRx9eUV55ZVXWLx4MUOGDGHcuHE8//zz1KlTp8A6PXr0YN68eXdsfvTz86Nhw4bUr18fk8lU6PV//vOfhISEMGrUKAYPHszIkSMJCQnhrbfeKnJ7Ja0/Y8YMUlJSiIiIICIiggYNGhQ54OFWdyozLy8vFi1axJtvvklERAT/+Mc/mDt3rmPwQGWRlOLqMEKFO3bsGIcOHXKMS/788885cuQICxcurN7A7sKXX35J8+bNadeuHRaLhUceeYQXXnjB0QwmVJ+DBw/y+uuv8/3331dqG7Rwb3HKPoXq0qBBA5YtW8bq1auRJImAgADefPPN6g7rroSEhPDmm2862r779esnEkINMHXqVPbu3cu8efNEQhDKRNQUBEEQBAen7FMQBEEQiiaSgiAIguAgkoIgCILgIJKCIAiC4HBPjD5KTc1ElsveX+7tbSQ5ueLH7t8tEVfZ1dTYRFxlI+Iqu7LGplJJ1KrlVuzr90RSkGWlXEkh/701kYir7GpqbCKushFxlV1FxiaajwRBEAQHkRQEQRAEhxqRFDZu3Ej//v3p06ePY8IyQRAEoepVe59CfHw8H3zwAWvXrnXMQd6pUydCQkKqOzRBEASnU+01hZ07d9K5c2dMJhOurq489NBD/Pjjj9UdliAIglOq9qSQkJDgeB4y5D2QIz4+vhojEgRBcF7V3nxU1Hx8ZZ3V0du7+Ifbl8Rsrtj58CtKdcd1NSkDPy831KqC30VFxqUoCqu3nCGsXR38vYsfN11ad4pNUZQSj6uonRcJCvCgeYPiH4Oav60z0amE1K2FWiUhywp7/ogj1yrzYPs6BfanKAoWJLQaFWaTS7ExpGdZOHEhGS9PA43q1nJs43R0KiF1TKhVEjuPxZGQkkVtXyMdm/s73mu1yWw/fIX4lCz+1rsxl6+l8cveaMaFN8Ogz/uJp2VasNtlann8+awLL2+j4/vNyLKQa7Xj5qLFoCt4WkhIzeJaciatQ8zYZYWMLAuexj+fbJeQkoVepy6w7HZ2Wcl7+qBaRVaOlfOxN1CrJRrXq4VGXfDa9GTsDS7FpeFjciG8S30kSSI9y8KilYdw0WsIbebH/pPxtG/qS4/76haI8/j5JPy83GgR7M3VpAzOxVwnwMeNQB8jFqud4xeSCa7tiVol8d32CwQHenJfM182br9AWqaFWu4GwtrXxs2g5fj5ZHYcu0pwoCcjexkdx1diajb/jTqBVq3i2RFtCsSvKAox8ekcPpuILEOu1caFKzc4F3sD2S7z3Mi2hDbzw2aXOXg6AZtNxuiq5fTlVNQqiQAfN2Lj0wjwdqVr60AklRpZVkhIzeLQqTi8PV3p1Kp2ofKtyN9ltScFPz+/Ak/bSkhIKPBYv9JITs4o1zhds9mdxMT0Mr+vIqzffgEvDwMPtAks9Fp548rKsXE29jptQnwKLE/LsrDz2DV6tK+NXlv0c3rz5VhsrNxylm1H4hj2QDADu9a/Y1y5VjuxiRk0DPR0LJNlBQUFdTGP5syXdD2bL6JOcTU+nUf7NiE+NYtdx6+RmWNjdO9G5FrsHL+YQuuG3oXizsi2cikujSPnkjl75TovjW6PyVD04ZxrsTPt01082LY2g7oV/YCSQ2cS+WTtMTxctcyZ2JnYhAzOxFwn22Knvr87LRt443pz+7v/uManG0/wwvBWtGzgxTtfHuJiXBoAWhQOn01i/+kEJo1oTdS+GHYejQOgRf1aPDOkJQa9hr0n4/njYgqjezXi3JU0Pvz2KPabx3D7xmZG9Qxh+5Gr/LL7LO0b+2L21LF7/xncVTnoJBtJ7YNo3zqYbEXDv7/dR256GlrJhmvKGU6fv0ZORjqrYvbTp1MDLidm8/Ou86jlXOp56wkOMHI9I4ezl5MJqKVHIykkpWaiRkarkvFy01DLw4Cbq4HMXDuXr6Yg2W1cr+3N9RtZZGTmEBToiauLgSvJWSSlZqKVZHw8dBgNeY9nzc61kWOxofetT9sBw1kWdY7L19J5bmgrPl57lLTUVNykXAK9DPTpUAcfkyvederj5uHKu18cQJJAUSD1ejatG3rz4bdHSb6eiafGwvkjh/FUZfNbXB1a1jMBsP3oVZZvOoUCSMCD7Wqz63gcki0bNykXNykXF5UFF8mCq2TBVWXFgIVYyUri9zaMkg2zWkYlWzmzx4YeG3rJRrhkh8sKJ/erUVBhkSVsdpm+WFEhc3qeFkmjxe7mi3fHfuy9kMXuQxdwlSy4Sbm4qnIJMdjpbLBCbibXV63jd60Esh1kO2pJJgeZEMmOFjs6yYaflPewr0ub/jw+ZQWaSpCmuBKbOJRsgw831F4EBQeV+XyhUkl3vJCu9qmz4+PjHc9tdXFxYdSoUbz55pu0bt261Nv4qyUFWVF47oNtaFQS7z13P7rbTnhmszvRsam46P88yVmsdnKsdjxcdcVud9nGE+z64xpvjO9APb+8Kwe7LLPg68OcjrnOA20CeTy8KQAX49LIzLHS8rar4v/78RTbDl/F5K5HURTmP9PVcXUXk5KNJcfKgdOJxCSk071NIL8dvsrla+m8/HAbWgZ7k51rY+4XB6jlbuClka2RJIkci42vfjlLaFNfWjf8c3+nLqcy/+tDeHvoef2xDkz91y5yLXmPDf3HqLYcPpvElgOxuLtq6R1al7C2geRY7Hy/8xI7jsahAFqNCr1WjVajolMzX7YfucI/Rrenvr+HYz/nr95gzn8PABDaxExqRi5J13OQJOjc3J/aZje++d85DDoNiTeyqetr5Er8DTxU2ZjUOejJxcNVw5CwJnh6uPH5d4cgJ4PGXR6gYX1/3vy//Qy6vz47jl0jx2IjJycXkzoXjWTHSCY9GhkwKulcio7HoAGNZEexWdEg4+OuITcnB71KJsCoIOdkIFmzkJHQIKORSv9w+buhIKGoNNhRYZUlFFlGhYJKUpAlDbKkRrJbkFGBWoNit6NGRiUpqDQaZNTk2hTsct628mogCiYpA/RuZObYUWPDrqhwkayopMK/11yDN6oGHThw+Cxt/CA3PRWrJW9drWTHXcpB4s/3yYqEtkE7kq0unLwQT4CbTG0PyLqRitqahavKgorizwuKSo1dpcOiaDG4uqDRu2BXabmeDbJKi6vRFXd3N87HZRAbn4ZWBR4uaowuGgL8vLh2w8Kl2BS0ko3G2nhMqszCO5EkJJ0bkosHit6N5HQbOTYFRVLj6eGCVqvFKksYja6g1pFlk3B1cyU2KYvzV27g7qLB002Hq05Feq6CKv4kIdoEAK7gR9OJ8yo8KdSImsLkyZMZN24cVquVESNGlCkhVIXsXFuBE3R5HT6bhNlkQKdVk2uxkwvsORlP99aByIrCiYspNG/gxeVraUxatJ3nhrWi7c2r/v/+dJozMdeZ93QXJEkiNiGDXw/GotOqCanticmoZ9cf1wDYfSLekRTWbrvA6ZjrNK5rYtuRq7Rs4MV9Tcx8+t0fxKdmM6BLEA91rIfRRYusKBw8k0iHZr50aeHPojVHOXgmkY7N/Fi77QK/HrwC5J2IvT0M/PfH0+h1ajxctaz//SLN63uxbOMJYhMziU3M5MSlVFo08OJU9HV+PxbH78fieKBNIOP6NUElSSSn5QCQnJbLV5vPkGuxM2NcKAu/OcKPuy9z7koaLYJMuKuy2b9jLyd35V3xuassPBusobYhBzf7deT0ZCzZWejO2ujrCknbj2DvMQRJ74Zk9OZKYt6PtXVDb06dv0ZTX4nWdexI2TdIPXaYRLIZosmhtZ+GbH0y6tw03Lwshb/A33/CBozVAEbIPHmczMzujHA9Q+f433nAeJ2srGw8XLMp0Eh0s4ssyE2HVVYhS2rUrlpsiprrWXZAjb+PCRd3DySDOznoOHs5FVRq2rSsz+noVDJy7HRoF4La1YRd0rJx+1muXrmGAStNG9ela2gI6bkS/7fxEMF1zQwIa8Zvh2JISknH00VNt3ZBuLi5Iqt1XLyWSe1AEy4aNYqkQlGpUasKXpikZVpIupGDi16Nn5crsqzw095omgbVIjjAgyPnklGrJZrWM6HV/Pleq82OJElo1Cq+23GRY7t283zrGxw+lYLJ00hmdg7BQQHUruOPZDBisUvEp2bz+4HzhGnP4nXyR5ppXTBoAnGp04CT0WnodRrq1/FG7+mF5GpCcjVx4HIucYd/p1f8efTZObTSq3F390JlcMfgGUKqRYPWxxuNizuSwYhkcEPSuYHeDUnviqRzBbW2yOY8j9v+bgm0d9GRk5mL6pbmVBPgfT2bw+eSmLX5NLP6u7Nyy1laNK1H3/ubIundQOeCJP1ZYy6poTu/EajZzf9udTo6lXlf1WNquJlfdp3D4FObpiVsrzyqPSkAjmea1kTR8enMWr6PN8Z3pK5v6fsuLl9LZ++peEaENXQceP/54QSN6pjo3iYAAL1OzZYDsXRrFcDxC8ks/OYoTw9uQVqOHbus8Mu+GNqG+JCWaWHPiXjsskLijRzORF/ns00n0WlVKAr8vC8GCfBw1RLo48aeE/GMeLAhVqvMlgOxdG7hx4T+zXjrv/tZ/b9z+NZyIT41m9o+bvyw6zI/7LrMfU3M9OtYj/QsK21DfGgV7I3ZZGDzgVjaNfJhz4l4OrXwJ6x1ALU9FAw5SVy8kIGXJpuUhASiL8VyYNn33G/L4RG/LGRLDrm/rCenTTdyk13oazhPc38V8ed2cnSliZa9BpB8XcFPdR21JJN0Np6/BWQTcOos/6gVQ0Kynf4uOdTOykSyW8DztgK+oUKye6FyN6Op1xqVxkB6Lpw8HcN9qUfJWnsIAMUzALPdh+me0fil54BHLuSQ9x+ACygqDZKLJyrJhD6wLmmyKzpfXyQ3EyoXTyS9G9czrfx+6DLpmdkE+pq4GJtMf3k7vhc24qHXolUHoanTBFuugtrbF627F5JaiykwgHS7K5LRG0mtLfARZFlh46aT+Hu50uKWZjoD0PmW9dq2LfjRNcCwh0P47veLXLqWTpd+LdFo1NQCnn0qGK1GhSRJ9Ozhz+1UQKOGJsfVZXG9LB5uOjzc/qyVqtQSA7r8GWPbRj5FvIsCCaKur5H1Nj92enTlm6zzvDSgDZ0bFqyZaoFg4PtYb/6V0pK2jbzYfuQai4d2R5Ikintit0lK5dNd2QTcN5ZPvzvB8LDgAvG5FvO+8vI06rFkF75QMJtcaB3szdeo+CnWgz9yAwlr0BSVR9mawEujjq8RkDia4srB6yZGti38/VaEGpEUarJL19JRFLiSmIGPp4F/f3+CZkG1CGsbiFaj5vejcZyJuc6EAQXz+v8OxbLtSBydm/tT19dIjsVG5s02/wYBedcDg+6vzzf/O09MQgbnruS1Se/+I57MXBsScPJyKnHJmRw+m+Robz4fe4NtR69S28eNqWPa46JXc/xCCtuOXKVrywCsdjuffneCM9HXyci2YrHKdG8VgEaton/nIJZu+IMVP51GAl4Z3Y5ryZnsOZnA1kNXSEnLRSVJtKhnRI45zKNBVzl8Oo7/fXOaMCmOnoDu9ysomSnkAAE3P2uApMZNZyBL0eHh5Y6bfzBxaTIJMTF4HomiOQrNXYFcNwJcVajSLpK9fj+dVUZ6mG6ZyCsX7HG18HT3IS0jFZveE12zDqg8/VC5ed284jOCwZhXJb+lzyL/JLfi2kFO2DoxuBls2XGCPoareOWeJ07jS52mjZFcTahcPR1XnCpXz7yrx1uuGIvq8vYGBjds7vj7g9VH+E9GHZoHathyOouPh+Y9gtTltve5mt3JLKZqr1JJPDGweZGvlUQlSQzpHlxo+e1NkdUp/yJq580a7J0uqhoEenD4XBJ/XFJRx2wscVBAoDnvW9p+JK+/JqT27VcNVce3lgvurlp23/ycwYG31zUqhptBi7eHnh3H8/bTIKBy9iOSQhGuJGbw0dpjTB3TnsTr2QCkZuRyIS6NQ2eTOHQ2icPnknhlVDt2Ho/jVPR1hoUFY7pl9MWFq3kn+aPnk6jrayQ1PReAzBwb+04l4ONpoEsLf77533lOXErl4tUbABy7kIxdVuh9Xx3+d+gKq349x5XETELqeBKTkMGxi8lcuJJGeOe8Jh+ANiE+js7lXKsdvU7Nz/ti0GlVGF20NL7ZGde+sRlPo47zV9NoUteEp5sOD51CfbUCl66SkXyB/j4J8M1Ksm251AfquwLpIBsk9NY6KAFNUHvXQ1WrNpKxFpKLJ5LBiMYi46uWHFeKgRYbcxduJ6KpiXPnruDmG8hTw9rjYpeZ/P5mxte/jCo9nkuqujQMDuRkTBrDhvZEY8wbeXPieBxB/h4YfMo2KinAx419JzM4odRne67CVTqRnJtD8/petO1avhNwUbw99FyMS8Pd3RUfz5o7UVp18vYw4KrXcCUxEzeDBpOx+P6w4JsnuJj4DHrdHMF1Jx6uOjxctZy8nIpKkqhfSSfI0pAkiZDanhw6m4TJqMPrlhFeFa2O2ciR88lIQJB/5YxQFEmhCOeu3CA+NZtzsTdISL2ZFNJyMRryTsL3NTFz8Ewi2bk2LsfnXQX+cTGF+1vlXTvnWGxcScprxz5yLpkBXeo7kgJAbGImbUN8MBn1BHi7cuJyChfi0qnv786la3nb69Y6AIvNzrabV0KjejXif4di2XsiAVlRaBVc9LBJvVbtqIFIEnRvHegYBaRGpl8LI3sOnGKo21ky16xDTo0FRWGoBBjBiivaRl3QBHdE5VWHuOsWPvxqN11DG/P3EfcV26Hloi840sig0xBS25Nd53OIv+7K0NZeAGjUKszmWmyxmUm25FA/wJ0hD7ak1W3b69oygPII8HYlM8fG4XNJQF5yVoDa5rsf8norLw8DGdlWriZnUtun/EOi72WSJFHX18jpmOvU9b3z1X9+7Rmgtm/pvqtAHzfSoq9Tz89Y4qi6ytaojolDZ5MKjMKrDHX98pJCoI9bhfRzFkUkhSLkn8CvJGWScEtNwdWgQQK6tvTnwOlE9pyMJzs3b7TMHxdTaN/YzPWMXG5kWFCUvAP9/NUbZGRbSUnL26ZGLWGzK46TVNOgWmw9dAVFyRtGF7X7MnZZoa6vkcf6NeXhHo1QqyT0OjXR8emcuJSKQae+YxX1oQ71OHoumdMx1wltYsYefw7LsZ+xxRylqzWHrh5AkgYpoAm6+hGozcFg9Obk5VQaNW+KwfDnFV1tF4h8qo+jVlIWLRp4sXbbBQDq3dJ0UM/PnQOnE8ix2Lmvibm4t5dL4M37Hc7EXMfH00DSjbzOg4o+cXt75l0NJl7PoU3DotvXBRxJobb5zuXvatDi7+XKtZQs6pawbr7aZiOnoq8TUqf6mo7y5cdQWU1H+er65iXPymo6ApEUAPhlfwynLqei16kZH97MkRRiEzP+rCmk52LQqTG562lc1wTArwfyRuPU9nHjj0spvL/qMJeupXN/q7wOoCHdg/lg9RGOXUgmNT3v5NSygTeHzyVR5+aB36xeLf53c1RPcKAHEwe1wN39z5ucXG8Ze59/4DULKnzDz60kCZ56KIjTBxNpcPIzsq78gaQ3om3YGZVvAyS9G5rAZnmjI27Ryrtukdsz3eGmpDspkBT8/rwSDPIzsu3IVQB8PCu2qh3g/WcXY4/2tdm44xI5Fju1y9gMVRLvW5oIKvoz3Evy+xFKM0ijQYAH11KyCCzld5X/nVZnf0K+4EAPRvdqRJeWldP5my+/yagyE6FICsAPuy6TlWPDZpfpE1qX1Iy8pHA29gbZuTYgLynotWq8PPS4GbQEeLsSm5iBRi3Rp0NdlkedIj3LikYtse1IHL61XGjRwAs3g4bT0XntnkYXLa2CvTh8Lol6fnk/kiY32/v1OjWB3m6oVFKx444bBnriZtDQqblfkZ/DnnIFy9Eo7DFHUWen0RxQXE3oOo5E16IXkrZqT15B/u6OGsat7cm3JgjvCj6h1nLXY9CpybHYaVTbRLOgWpyKvo6XR/kSW3Fu3V5Ff4Z7SfP6XtT1NdI8qFaJ6/brVI+2TX1L3SzSJsSHDpdTC91rUx1UUt55oLL5mlz45+MdqFPKJrbyEEkBsNtlQmp7cCr6Ogmp2Y6aQlpm3hC0OmY3riZloVZJjuphw0BP4pKzqGM20qahNxq1RPc2gZjcdKzbfpHgQA9UkkSQvzuX4zPwdNPh5a6ne5tAArzdCLjZzOHuqqNBgDtuBm2BMdBFcTVoWDype6G2WcWSjeVoFJbDP4Bah6ZeG9S+wahM/qhrN0dSVc/XrJIket1Xh1yrvUDMdXyNjjtWvSu4U06SJAK83YiOT6een5FRvRqRmp5b5qlTSlLLXe/4DD6et485EvJ5exqYNaG4gaUF1fU10r5FQKlvxKrlrueZIS3vJry/pMrqYM4nkgJgsyv4e7vlJYXr2VxPz8XdVUt6lhWAJnVrEZuYSdKNHDo0zRt/HFLHk9+PxVHf3x1Po553nuqCyV2P1SZzOuY6nZrlXc3X83Nn8/4YrDZXzJ4GNGoVTW+7anpxeGukEhJCvltPbooiYz25Fcv+dSg56WhCuqDv+ggqQ82Zz2lwEdNK6LVq/L1ciUvOqpSr7PaNffDzckGnVWM2uWA2VfxJW61SUctdT0parqgpCPcUkRTImwrCVZ83ZC42IYPMHBudW/ix+4+8W1Eb1fVky8FYAMdws0Y32/Qa3mzPzF+u16p5ZVQ7x7aD/Nyx2RWuJmU6+iJud6eJxIojpyeS879l2K+dQR3QFH2nh1H7Fh63XlM1CPAgK8dWaPK1inDrTUyVydvDQGaODbdi5lwShL8ipz+aFUXBZldQqyR8TS6cibkOQNN6tdh3MgEPNx1+tf7svMxv7gjwduOfj3cosQMtv+8A8qq7FcF6YS852z4HBQwPPoGm0f0V3jxS2UY82JA+oZXfBluZGgZ6ort597Ag3CucPinIN+cDVKslfGu5ciY27yYys6eBAG83jC4aTO5FdyqWpm3Pz8sVvVZNrtWO110mBTk9EcvB77Ce3o7KNxiXns+g8qjYIZ1VxWTUl3tUU03xcE/xdEDh3uP0ScFuz0sKGrUKc60/255N7nomRjRHrZZwd9WiVknYZQXvMo5iUUkSdf2MnIu9Ue6agiLL5O7+GuvxzSBJ6Nr0R9dhWLV1IAuCcO9y+rOK7WZSUKukAuPN84Y2/lk8JqOerFwrroay38QV5Ote7qSg2Czk/LoU26WDaJv1QNcuApXRq8zbEQRBKA2nTwp2OW++erVKcoxScdFrCnWA1vLQY8gt36309zUxczkhvcxDF5WcDLJ+Wogcfx591zHoWvYp1/4FQRBKSyQF+c/mI9+bzUdFtf0P7dYAq718E581DarF9KD7yvQeOS2RrKj3UDKSMPR+Fm1wh3LtWxAEoSycPinY7H/WFNwMWowu2gIdy/ma1a+6Jhtr6jWyNryJIttxGfAqGv/GVbZvQRCcm9MnhVtrCgB9O9TFx1R9NyMp1hyurZuHIttxHTQdda3CD+kWBEGoLE6fFBwdzeq8sea3Pqi+qimynZz/fYotKRaX8JdFQhAEocoVP9Wmk7Df0nxUnRRZzksIlw7i3edxNHWcb04XQRCqn0gKcn5NoXqLwnJkE7bze9B1HIlnhwHVGosgCM5LJIX8m9eqsaZgT47GcmAdmuCO6Nr0r7Y4BEEQRFKQq7f5SM5JJ3vzJ0h6I4Zu48Q8OoIgVCunTwq2amw+Uqy5ZEd9gJKRjKHPc0gG8axfQRCql9MnBUdHs7rqr9BzD6xDTryAodcz4l4EQRBqBJEUHH0KVVsU9pQYrMd+Rtv0AbT121fpvgVBEIrj9Enhz+ajqq0p5O78Cknnir7jw1W6X0EQhDtx+qRQHfcp2OJOY796El37QaIfQRCEGkUkhdumuagKlsM/IBnc0TYLq7J9CoIglIbTJwVbFdcU7EmXsMccRduqL5Lmr/3kMUEQ7j0VnhTWr19Pt27dGDx4MIMHD+aDDz4A4OrVq4wZM4Z+/frxzDPPkJmZCUBaWhoTJ04kPDycMWPGkJiYWNEh3VFV3tGsKAq5u75G0hvRNe9Z6fsTBEEoqwo/Ex47dozIyEg2bNjAhg0bmDx5MgCzZs3ikUce4ccff6Rly5Z88sknACxcuJDQ0FCioqIYOXIkc+bMqeiQ7ujPx3FWfk3BdmEv9rjT6DoMR9K7Vfr+BEEQyqpSksL69esZNGgQr7zyCjdu3MBqtbJv3z4eeughAIYNG8aPP/4IwNatW4mIiABg4MCBbNu2DavVWtFhFctWRXc0K7Kd3D2rUXkHoW0q+hIEQaiZKnzqbLPZzMSJE2ndujXvv/8+s2fPZurUqRiNRjQajWOd+Ph4ABISEjCbzXnBaDQYjUZSUlLw8/Mr9T69vcs/gsdg0AHg7+eJVlN5TUiZp3aTkZGMb78ncPPzLHF9s9m90mK5GzU1Lqi5sYm4ykbEVXYVGVu5k0JUVBRz584tsCw4OJjly5c7/n7iiSfo3bs3r776aqH332mOH1UZbyRLTs5Alsv+qEyz2Z0badkApKZkVOq8Q1k7NyIZvck0NSErMb3EuBJLWKc61NS4oObGJuIqGxFX2ZU1NpVKuuOFdLmTQnh4OOHh4QWWpaens3z5ch5//HEgr2NVo9Hg5eVFRkYGdrsdtVpNYmIivr6+APj6+pKUlIS/vz82m42MjAxMJlN5wyozu6ygkqRKTQj2lFjscafQdXwYqYrvnBYEQSiLCj1Dubq68u9//5sjR44A8MUXX9CnTx+0Wi2hoaFs2rQJyBuh9MADDwAQFhbG+vXrAdi0aROhoaFotdqKDOuO7LJSqZ3MiqKQu2cVaA3omj5QafsRBEGoCBXap6BWq1m4cCFvvPEGOTk51K9fn/nz5wPwz3/+k8jISJYsWUJAQADvv/8+AJMmTSIyMpIBAwbg7u7OggULKjKkEtnscqVOcWG7sA97zDH0XR4Rdy8LglDjVXhHc2hoKOvWrSu0vHbt2qxYsaLQcpPJxNKlSys6jFKzywrqSmrSUew2cnd9hcqnPtoWvStlH4IgCBXJ6Ru47Xal0moKtgt7UbKuo+8wXPQlCILwl+D0Zyq7Xa60R3Fa/tiM5OmPuk6LStm+IAhCRRNJQVYqZYoLe8IF5IQL6Fr0QpKcvpgFQfiLcPqzlc0uV8rdzNbT20CjQ9u4W4VvWxAEobI4fVKojI5mRbZhu7AfTVA7JJ1LhW5bEAShMomkUAn3KdivnEDJzUDTsFOFblcQBKGyOX1SqIz7FKzn94DOBU3dVhW6XUEQhMrm9EnBbq/Y5iPFbsV28SCa+vchqavuzmxBEISKIJJCBTcf2WKOgTUbrWg6EgThL8jpk0Le6KOKKwbb+T1IBnfUtZtV2DYFQRCqitMnhYqsKSjWXGyXD6FpEIqkqvAZRARBECqd0yeFirxPwRZ9GGwWMepIEIS/LKdPChV5R7Pt4gEkF0/U/o0rZHuCIAhVTSQFu1Ihcx8psh1b7DE09VqLye8EQfjLcvqzl02umPsU7PHnwJKNum7rCohKEAShejh9Uqio+xTs0UdAUqOp07ICohIEQageIinIFfM8BVvMUdQBjcVcR4Ig/KWJpGCX0dxlTUHOSEZOiUVTTzQdCYLw1yaSQgXUFGzRRwFQ121TESEJgiBUG6dOCoqi3Jw6+26TwhEkdzMqU0AFRSYIglA9nDop2OwKwF3dp6DYLNivnkBTtzWSVDmP9RQEQagqTp0U7HYZ4K6mubDHnc67i7meaDoSBOGvz6mTgu1mUribIam2mKOg1qEObFpRYQmCIFQbJ08KN5uP7qJPwX7tDGq/hkgaXUWFJQiCUG2cOinY5btrPlIs2cjJ0WKuI0EQ7hlOnRSstrtrPrInXABFQe3fqCLDEgRBqDZOnRTscv7oo/LVFOzXzoAkofZtWJFhCYIgVBunTgo2W37zUTlrCvHnUHnVFVNbCIJwz3DupOAYfVT2moIi27HHn0PtJ5qOBEG4d9x1Uli0aBEffvih4++0tDQmTpxIeHg4Y8aMITExEQCLxcKUKVMIDw9n6NChnD9/Hsi7q3jevHn069eP/v37c+DAgbsNqdTym4/K09Esp8SALVf0JwiCcE8pd1JIT09n+vTpfPbZZwWWL1y4kNDQUKKiohg5ciRz5swBYMWKFbi4uBAVFcX06dOJjIwE4KeffuL8+fNs2rSJjz/+mMjISGw22118pNK7m45m+7Wzee8VSUEQhHtIuZPCli1bqF+/PuPHjy+wfOvWrURERAAwcOBAtm3bhtVqZevWrQwaNAiADh06kJqaytWrV/ntt9/o378/KpWKBg0aEBgYyKFDh+7iI5Ve/pDU8jQf2a+dRXLzQmX0ruiwBEEQqo2mvG8cMmQIQIGmI4CEhATMZnPexjUajEYjKSkpBZYDmM1mrl27RkJCAr6+voWWl4W3t7FcnyEmJRsAH28jZrN7qd+nKArRCedwDWpWpveVRWVt927V1Lig5sYm4iobEVfZVWRsJSaFqKgo5s6dW2BZcHAwy5cvL/VOVMU0z6hUKhRFKfX6xUlOzkCWC2+nJPkdzenp2SQmppf6fXJ6IvaMFGymBmV6X2mZze6Vst27VVPjgpobm4irbERcZVfW2FQq6Y4X0iUmhfDwcMLDw0u9Q19fX5KSkvD398dms5GRkYHJZMLX15fExESCgoIASExMxNfXFz8/P0dn9K3Lq4K9nKOPRH+CIAj3qgofkhoWFsb69esB2LRpE6GhoWi1WsLCwtiwYQMA+/fvR6/XExgYyAMPPMDGjRux2+1cvnyZS5cu0apVq4oOq0g2W/mmzrbHnwOtAZVX3coISxAEodqUu0+hOJMmTSIyMpIBAwbg7u7OggULABg7diwzZ85kwIAB6HQ65s+fD0C/fv04evSooxN6zpw5GAyGig6rSLb8uY/KWlNIuoTaJwjpLh/jKQiCUNPcdVJ44YUXCvxtMplYunRpofX0ej3z5s0rtFySJKZOncrUqVPvNpQyczQfleE+BUWRkVNi0TYNq6ywBEEQqo1TX+paHVNnl74YlLREsFlQedWprLAEQRCqjVMnhfLUFOwpMXnvEf0JgiDcg5w7Kchlf8iOnBwDSKi8aldSVIIgCNXHuZPCzeYjlVSGpJASi+Tph6TRV1ZYgiAI1capk4J888Y5VRlqCvaUWNSiP0EQhHuUcyeFMjYfKdYclLQEcX+CIAj3LKdOCvl9CqVtPpJTrwAKKm9RUxAE4d7k1Ekhv6ZQ2i4Fe7IYeSQIwr3NuZOCoqCSJKTS1hRSYkBrQHL3qeTIBEEQqodTJwW7XaYsM1XIKbGovOogSU5dbIIg3MOc+uwmK6UfeaQoCvbkGDHySBCEe5pTJwW7LJe6k1nJTAFLlhh5JAjCPc2pk4IsK6UfeXRzeguVt0gKgiDcu0RSKGXzkT0lFkA0HwmCcE9z6qRgL0NSkJNjkYzeSDrXSo5KEASh+jh1UpBlpdR3M8spMWK6bEEQ7nlOnRTsskJpcoJityJfjxM3rQmCcM9z6qQgK0qpblyTU6+CIotOZkEQ7nnOnRRK2Xwk3+xkFsNRBUG41zl1UihtR7M9JQbUGlSeflUQlSAIQvVx6qRQ2vsU5OQYVLVqI6nUVRCVIAhC9RFJoVTNRzGi6UgQBKfg1EnBXoqagpydhpKdJkYeCYLgFJw6KchKyTWFPzuZa1dFSIIgCNXKuZOCXSlx6mw5LQEAlad/FUQkCIJQvZw7KSglNx8paQmgUiO5eVVRVIIgCNXHqZOCvRT3KchpCUjuZqSyPI1HEAThL8qpz3SyXPIdzXJaPCoP3yqKSBAEoXrddVJYtGgRH374oePvffv20alTJwYPHszgwYOZNm0aAGlpaUycOJHw8HDGjBlDYmIiABaLhSlTphAeHs7QoUM5f/783YZUaiXd0awoCnJaokgKgiA4jXInhfT0dKZPn85nn31WYPmxY8eYMGECGzZsYMOGDcydOxeAhQsXEhoaSlRUFCNHjmTOnDkArFixAhcXF6Kiopg+fTqRkZF38XHKxi7Ldxx9pOSkgzVHJAVBEJxGuZPCli1bqF+/PuPHjy+w/NixY+zYsYMhQ4bw9NNPExcXB8DWrVuJiIgAYODAgWzbtg2r1crWrVsZNGgQAB06dCA1NZWrV6+WN6wykWXu2NGsOEYeiaQgCIJzKHdSGDJkCBMnTkStLjj1g7u7O+PGjWP9+vWEhYUxefJkABISEjCbzQBoNBqMRiMpKSkFlgOYzWauXbtW3rDKpKT7FOQb8QBIoqYgCIKT0JS0QlRUlKMJKF9wcDDLly8vcv3Zs2c7/j169Gjee+890tPTi1xXVcyInuKWF8fb21im9fPZZRkXgxaz2b3I11NO3iAHCb8GwUgabbn2UV7FxVTdampcUHNjE3GVjYir7CoythKTQnh4OOHh4aXamCzL/Otf/ypUg9BoNPj6+pKUlIS/vz82m42MjAxMJhO+vr4kJiYSFBQEQGJiIr6+ZbsyT07OQJaVMr0nL14Fq9VGYmLRSSs7LgbJ6EVSag6QU+btl5fZ7F5sTNWppsYFNTc2EVfZiLjKrqyxqVTSHS+kK3RIqkql4pdffuGnn34CYP369bRp0wYXFxfCwsJYv349AJs2bSI0NBStVktYWBgbNmwAYP/+/ej1egIDAysyrGKVNHW2nJYgOpkFQXAqJdYUymrevHm8/vrrfPzxx3h5eTF//nwAJk2aRGRkJAMGDMDd3Z0FCxYAMHbsWGbOnMmAAQPQ6XSO9atCSVNnK2kJqOu3q7J4BEEQqttdJ4UXXnihwN+NGjVi5cqVhdYzmUwsXbq00HK9Xs+8efPuNoxyudPU2YolGyUnXXQyC4LgVJz6juY7TZ3tmAjPQzxtTRAE5+HUSUFWir+j+c+kIGoKgiA4D6dOCnZ7aWoKIikIguA8nDopyIqCVEwJKGkJSAZ3JJ1L1QYlCIJQjZw7KdxhQjw5LUF0MguC4HScOimU1NEsmo4EQXA2Tp0UihuSqtitKBkpIikIguB0nDopFFdTUNKTAEUkBUEQnI7TJgVZyZsrqaiaghh5JAiCs3LepCDfISlkJAMguftUaUyCIAjVTSSFIvqZlYwUkNRILp5VHJUgCEL1ct6kcKfmo4xkJDcTUhmf6yAIgvBX57RnvfyagrqojubMFFRG76oOSRAEodo5b1K4+UweqbiagtGriiMSBEGofs6bFPJrCrclBUWWUTJTRU1BEASn5LRJwe7oaL4tKWTfANmO5CZqCoIgOB+nTQpKMR3NSmZK3nJRUxAEwQk5bVKQi6kpyBl5SUH0KQiC4IycNinYHTWFgsuVmzeuiZqCIAjOyGmTQnF3NMsZKaDRg861OsISBEGoViIp3N7RnJGMyuiNVMyU2oIgCPcy500KN+9TKNSnkJki+hMEQXBazpsUirtPISMZlUgKgiA4KedNCjc7mm+9o1mxW1Gy05DcRCezIAjOyXmTQhE1BSUzFUDUFARBcFpOmxSKuqPZ8RwFMRxVEAQn5bRJwXFH8y1dCkpG/t3MoqYgCIJzctqkUNR9Co6agpj3SBAEJ+W0ScFexNxHSkYKksEdSaOrrrAEQRCqVbmTwoEDBxg+fDiDBw/mscce48qVKwCkpaUxceJEwsPDGTNmDImJiQBYLBamTJlCeHg4Q4cO5fz580BeM868efPo168f/fv358CBAxXwsUomy3n/L9CnIO5REATByZU7KUyZMoU5c+awYcMGIiIieOuttwBYuHAhoaGhREVFMXLkSObMmQPAihUrcHFxISoqiunTpxMZGQnATz/9xPnz59m0aRMff/wxkZGR2Gy2Cvhod1ZU85GSkYxKNB0JguDEypUULBYLkyZNomnTpgA0adKEuLg4ALZu3UpERAQAAwcOZNu2bVitVrZu3cqgQYMA6NChA6mpqVy9epXffvuN/v37o1KpaNCgAYGBgRw6dKgiPtsdOZ7RXGD0UYoYeSQIglMrV1LQ6XQMHjwYAFmW+eijj+jduzcACQkJmM1mADQaDUajkZSUlALLAcxmM9euXSMhIQFfX99Cyyvb7fcpKJYssGaLkUeCIDg1TUkrREVFMXfu3ALLgoODWb58ORaLxdHc89RTTxW7DdXt81Pfsjx/aGhp1i+Ot7exTOsDGGPTbr7XDbPZHUtCKhmAZ2AdjGb3Mm+voplrQAxFqalxQc2NTcRVNiKusqvI2EpMCuHh4YSHhxdanpmZyTPPPIPJZGLJkiVotVoAfH19SUpKwt/fH5vNRkZGBiaTCV9fXxITEwkKCgIgMTERX19f/Pz8HJ3Rty4vi+TkDMeVf2ldv5GV9//rWbioJWwx0QBk2F3ITkwv07YqmtnsTmI1x1CUmhoX1NzYRFxlI+Iqu7LGplJJd7yQvquO5qCgIBYtWoRO9+cQzrCwMNavXw/Apk2bCA0NRavVEhYWxoYNGwDYv38/er2ewMBAHnjgATZu3Ijdbufy5ctcunSJVq1alTesUrPf1nwk35ziQow+EgTBmZVYUyjKiRMn2LJlCyEhIQwZMgTIqyEsW7aMSZMmERkZyYABA3B3d2fBggUAjB07lpkzZzJgwAB0Oh3z588HoF+/fhw9etTRCT1nzhwMBkMFfLQ7u72jWcm6AYDk4lnp+xYEQaipypUUmjdvzunTp4t8zWQysXTp0kLL9Xo98+bNK7RckiSmTp3K1KlTyxNKueV3ZeQPSVWyriPpjUjqchWJIAjCPcF572i+7T4FJfsGkqupGiMSBEGofk6bFG5/HKecdR3JVTQdCYLg3Jw3Kdw295GSdUMkBUEQnJ7zJoVbagqKoqBk3UAlmo8EQXByzpsUlFuGpOZmgmwTNQVBEJye8yYFR0czyNliOKogCAKIpIAkSX/eoyCajwRBcHLOmxRuuU9Bybqe92/RfCQIgpNz2qRglxUk6WZHs6gpCIIgAE6cFBRF+XPeo6zroNGBtvKn1xAEQajJnDYp2GXlz3mPsm8guXgi3fLAHUEQBGfktElBlpUCN66JexQEQRCcOSnc0nykiCkuBEEQgHLOknovuLWmIGenoQ5sXs0RCfcqu91GamoiNpulukNxSEhQIctydYdRiIir7O4Um0ajo1YtM+oyzP7svElBuTkc1W6D3EwkV4/qDkm4R6WmJmIwuOLm5l9j+q00GhU2W807yYm4yq642BRFITMzjdTURHx8Akq9PedtPpJl1CoJJSfvMXaSQSQFoXLYbBbc3DxqTEIQnIMkSbi5eZS5hurESeHmPQrZaQBILiIpCJVHJAShOpTnuHPepKAoqNQqkRQEoYqMGBFBXNzV6g5DKIHzJgVZQX1LTUElkoIgCIIzdzQrqFSg5IiaguA8Dh7cz5Ili5FlGX//AFxcXLlw4TyyLDNmzDh69uzD4MH9WL16Pa6ubjzzzATuv/8BHn30cTZv/onDhw/xzDPPM3fumyQmJpCUlEjbtu2YMWM2hw4dYMmSxdjtMsHBDXnxxZeZPft1EhLiqV8/GIslr2373LmzzJ8/B7vdjk6nY/r0f1K3br1qLhkhn9MmBbusoFKpULLTQa0RU1wIVWLHsTh+PxpXKdvu1jqA+1uVPMokJiaa9et/4PPPP8PHx8yMGbPIzMzg6acn0Lx5S+67L5RDhw7Srt19xMXFcfjwQR599HF2795Jr1592Lnzdxo1asxbb83DarXy6KMjOX36lGPba9Z8j9Fo5P3359G4cVMWLFjM4cMH+fXXXwBYvforRo16lJ49e7Nly8/88ccxkRRqEKdNCrKcd/OanJ2GZBAjQwTnUbduEEajO/v37yU3N4cffvgOgJycHC5evECXLt04cGAvKpVE377hbNnyMzabjSNHDjNlynT0ej0nThxn9eqvuHTpIjdu3CA7O+uWbRsBOHToAG+88TYAbdu2JzCwNgBdutzP++/PZ8+enXTt2p0HH+xVDaUgFMdpk4KSf59CdppoOhKqzP2tSnc1X5n0ej0Asmzn9dffpEmTpgCkpCTj4eFJeno6K1d+iVqt4b77OhAdfYnvv19PcHAwer2eNWtWsnXrrwwaNJQRIzpy8eJ5lJtPMszfNuSNfLn1piq1Wg1Ajx69admyNTt2bOebb75m9+4dTJ06o6o+vlACp+1ott+8o1nJSRdJQXBK7dt3YP36NQAkJSXx2GOjiY+/Rq1atdDr9ezYsY3WrdvSvn0Hli//D127dgdg3749DBo0jL59wwGJs2fPFHlHbWhoR37+OQqAkyf/4MqVWABmzpzGiRN/MGTIcJ544mlH05NQMzhtUpCVm6OPsm4gubhXdziCUOUmTHiS3Nxcxo59mEmTnubZZ1+kdu06QF4Tj9HojqurK/fd14GkpES6du0GwMMPP8Lnn3/KhAljeP/9ebRs2brIoaZ///tTXLkSy6OPPswXXyx3NB+NHTueFSs+Z8KEMXz88UJeeGFy1X1ooUSSkl/v+wtLTs5wPF6ztN79+hAAT2d9jLZFHwyd/1YZoZWL2exOYmJ6dYdRSE2NC2pubGazO8eOHcffP6i6Qymgpk7bIOIqu5Jiu3btcoHjT6WS8PY2Fru+89YUZAW9ygp2m7hHQRAE4SbnTQqKghvZgLhHQRAEIZ/zJgVZJAVBEITblXtI6oEDB3j77bex2WyYTCbefvttateuzb59+3j++efx9/cHoHnz5sydO5e0tDReeeUVYmJi8PLyYuHChZjNZiwWC6+99hrHjx/HYDCwYMECGjZsWGEfsDiyouAikoIgCEIB5a4pTJkyhTlz5rBhwwYiIiJ46623ADh27BgTJkxgw4YNbNiwgblz5wKwcOFCQkNDiYqKYuTIkcyZMweAFStW4OLiQlRUFNOnTycyMrICPlbJZBnclLwbbiSDGH0kCIIA5UwKFouFSZMm0bRp3k0vTZo0IS4u79b9Y8eOsWPHDoYMGcLTTz/tWL5161YiIiIAGDhwINu2bcNqtbJ161YGDRoEQIcOHUhNTeXq1cqfSdEuK7jZ00FSi0dxCoIg3FSu5iOdTsfgwYOBvIfVfPTRR/Tu3RsAd3d3BgwYQO/evfn666+ZPHkyK1euJCEhAbPZnLdTjQaj0UhKSkqB5QBms5lr164RGBhY6njuNLyqOCq1hLt8A43JjK+fqczvr2xmc82svdTUuKDmxqZSqdBoal73XU2MCURc5XGn2FQqVZl+GyUmhaioKEcTUL7g4GCWL1+OxWIhMjISm83GU089BcDs2bMd640ePZr33nuP9PSix4+rVEV/kOKWF6c89ylYLHaMuusort41bnx7TR5zXxPjgpobm9nsjizLNW6Me0WOu//99984deokTzzx9F1vq6beD1CT4tq0aSOHDh3gtdfeAEqOTZblAr+Nku5TKDEphIeHEx4eXmh5ZmYmzzzzDCaTiSVLlqDVapFlmX/9619MnDjRMc9JXtAafH19SUpKwt/fH5vNRkZGBiaTCV9fXxITEwkKyru5IjExEV9f35LCumuyomC0XUfl3qTS9yUI97Ju3cLo1i2susMQKki5Rx9NmTKFoKAgZs+e7ZhhVKVS8csvvxAUFET//v1Zv349bdq0wcXFhbCwMNavX8/TTz/Npk2bCA0NRavVEhYWxoYNGwgNDWX//v3o9foyNR2Vl0bORS9lI3mYS15ZEO4RBw/u57///QyAK1diefDBXri5ubF9+28oisKCBYvw8vJmx47tLFu2BEWRCQyszZQp0zlx4jjffbeO+fMXAvDtt6uIiYmmceOmjivXESMieOih/uzdu4vs7BxmzJhF06bNuHDhHHPmzMJut9OmTVt2797JqlXrC8R24cI5Fi5cQFZWFqmpKYwa9ShDh45g+PCBfP75l3h5eZOWdoOxY//Gt99+z/79e/nPf5Zis9kICKjN1Kmv4elpYsSICJo3b8nZs6f55JN/s3r11xw4sI+0tDRMJhNz5szH29uHLVt+4T//WYrBYKBx46bY7XZee+0NTp78g8WL3yc3NwdPTxNTpkynXr26BWJ9/vmJeHh4cvHieWbPnktycnKhWDZt+p7U1BSeffZF9u3bzfTprxIV9SsajYZHHx3J4sVLOXz4ECtXfkFubi65ublERs6gbdv2hbZ//vw5/u///oObmxF/f39cXFwB+OijhezfvweVSkW3bmFMmDDxro+RciWFEydOsGXLFkJCQhgyZAgAvr6+LFu2jHnz5vH666/z8ccf4+Xlxfz58wGYNGkSkZGRDBgwAHd3dxYsWADA2LFjmTlzJgMGDECn0znWr2weys0nromkIFQh65kdWE9vq5Rta5s8gLbx/SWud+LEH3z99Te4uXkQEdGH5557if/8ZwVvvz2LzZt/pk+fh3j33bdZsuQ/BAQE8tVX/+X99+fzxhtzePfdvOHlHh4ebN78Ey+88DKXLl0ssH1PT0+WLfsva9asZMWKz5gz513eeusNnnzyabp06caqVV9it9sLxbVx4wbGj/877dp14MqVWB5//BFGjhxFjx69+d//NjN8+N/YuvVXund/kPT0dJYu/YjFi5fi4eHB+vXfsmTJh0RGvg5A585dmT17LrGxMURHX2Lp0s9QqVS8+eZMfv75R/r1G8Dixe/x73//F29vH2bMmIqbmxtWq5V33nmLefM+wN/fnz17djFv3hw+/nhpoXgbNgzh7bffJTU1lTlzZhWKZfToscyenRfP/v37MBgMnDlzCpOpFi4urphMtdiw4Vvmz1+IyWTi++838NVXK2jbtn2B7SclJbJkyWI+//wrPDw8efXVl3BxceXatTh2797JypVryMzMZt68t8jNzS0wU215lCspNG/enNOnTxf5WqNGjVi5cmWh5SaTiaVLCxesXq9n3rx55Qnjrpjyk4J75TdVCUJNEhzcED8/f2w2GU9PE6GhHQHw8/MnPT2NEyf+oFmzFgQE5NXYBw0axooVy9FoNISF9eC3336lQ4dO3Lhxg+bNWxZKCp06db25nxB+++1/pKXd4Nq1OLp0yZtQb8CAwXzzTeFzxPPPv8T+/btZseJzzp0763hGQ79+/Vm06D2GD/8bmzf/xJNPPsOJE8eJj7/Giy/m9WPIsh0Pjz9HETZv3hKAOnXq8vzzk9m4cT3R0Zf5449j1K5dh6NHD9GyZSvM5rzff3j4ALZt20pMzGWuXo0lMvJlx7YyMzOLLMf8fRQXS1BQfTIzM0hLS+Po0UMMH/4whw8fxGBwoWvXbqhUKt5++1127NhOdPRlDh06UKA/NX/7x44doWXL1nh5eQPQt284Bw7sw8fHjF6v58knx9O1azeefPKZu04I4MTPUzAhagpC1dM2vr9UV/OVSaMp+LO/tf8PQFHk2/5WHFf2ffv259//XkJ6ehp9+vQrcvs6na7Ae1UqNaWZd3PmzEg8PDzp2rUbvXr1ZcuWnwFo2rQ56elpnDz5BwkJCbRq1Ybt27fSunUb5s37AIDc3FyysrIc28o/OZ46dZI33niNUaMeoUePXqjVqpsxqYocnGK35zWXLV/+1c2/7aSmphQZ763PpSgulk6durBt2/8Aia5du/Hvfy8FJP7+96fIysriiSfG8dBD/WnTph0NG4bw7berC20/77kUf8aa/31pNBo+/XQ5x44d4vfff+fpp8fz4YefUq/e3U2+WHPHWFWyWlI6VpUBSe9W3aEIQo3SvHlLTpw45pgO+7vv1tK+/X0AtGzZiqSkJH76adPN5ymUzGg0UqdOHXbt2gHAL7/8WOSTDvft28vEic/QvfuDHD58EMCRjPr06ce7775N7959HTH+8ccxoqMvA7B8+b/55JNFhbZ5+PAB2rW7jyFDRlC/fjB79+5BlmVatmzDqVMnSEpKQlEUNm/+GUmSCAqqT1paGkeO5M2i/MMP3/HGG6+VWF7FxdKlSzdWrPic1q3b0qhREy5evEhMzGWaNGlKTEw0KpWKceMmcN99Hdi9e2eRz6Vo3botJ04cIzExAVmWHY81PXPmFM8/P/FmH8RL1K8f7IjhbjhtTaEW6WTralV3GIJQ43h5eTNlymtMn/4KVqsNf39/IiNnOl7v1asPe/bscjx7oTRee20Wc+fOZtmyT2jYsFGRzRwTJjzJU09NwGg0UrduEAEBgcTFXaVOnbo89FB//v3vpY7He3p7+xAZOZOZM6chy3bMZj9mzpxdaJu9evVl+vQpPPbYKNRqDQ0bhhAXd5VatWrx0kuvMHnys+h0egICAtDpPNDpdLz55jssWrQAi8WCq6sbM2bMuuNnu1Ms7drdR3JyEu3a3YckSTRu3BgPDxMAISGNCAlpzCOPjMBgMNC2bXuuXSv8/G4vL29eemkKL730LAaDC/XrNwCgceOmtGzZmjFjHkav19OoURM6d+5a6u+kOE77PIXzSyeBqTYNR71aSVGVX00ec18T44KaG5t4nkKezz9fRkTEUHx8fPjtt1/5+eco5sx5t9riunHjOmvWrGL8+CdRqVQsXPguderUZcSIUUWuX5PuU7hdRT9PwWlrCmoJLF4164cqCPcqPz9/Jk9+Fo1Gg7u7h2OUUHXJfxb1uHF/Q61W07hxUyIihlZrTDWF09YUcrKyCAj0IvV6TiVFVX41+aq3JsYFNTc2UVMoGxFX2Yknr1UQg6srGq22usMQBEGoUZw2KQhCVboHKuTCX1B5jjuRFAShkmk0OjIz00RiEKqUoihkZqah0ehKXvkWTtvRLAhVpVYtM6mpiWRkXK/uUBzybt6qeW3kIq6yu1NsGo2OWrXKdoOuSAqCUMnUag0+PgHVHUYBNbljXsRVNhUdm2g+EgRBEBxEUhAEQRAc7onmI5Wq8DwqVfHeyiTiKruaGpuIq2xEXGVXlthKWveeuHlNEARBqBii+UgQBEFwEElBEARBcBBJQRAEQXAQSUEQBEFwEElBEARBcBBJQRAEQXAQSUEQBEFwEElBEARBcBBJQRAEQXC4J6a5KKuNGzeyZMkSrFYrjz/+OGPGjKm2WD766COioqIACAsL49VXX2XatGkcOHAAFxcXAJ5//nn69OlT5bGNGzeO5ORkNJq8w2T27NlER0dXa9l98803fPHFF46/Y2NjGTx4MNnZ2dVWZhkZGYwaNYqlS5dSp04ddu7cydy5c8nNzSU8PJzJkycDcPLkSWbMmEFGRgahoaHMmjXLUbZVEdeqVatYsWIFkiTRsmVLZs2ahU6n46OPPuLbb7/Fw8MDgIcffrhSv9fb4yrueC+uHCvTrbGdP3+e999/3/FafHw8bdq04V//+leVlllR54hKPcYUJ3Pt2jWlR48eSmpqqpKZmalEREQoZ8+erZZYduzYofztb39TcnNzFYvFoowbN075+eeflYEDByrx8fHVElM+WZaV+++/X7FarY5lNansFEVRzpw5o/Tp00dJTk6utjI7fPiwMnDgQKVFixZKTEyMkp2drYSFhSnR0dGK1WpVJkyYoGzdulVRFEUZMGCAcujQIUVRFGXatGnKl19+WWVxXbhwQenTp4+Snp6uyLKsvPrqq8rnn3+uKIqiPPXUU8rBgwcrLZY7xaUoSpHf3Z3KsSpjy5eQkKD06tVLuXjxoqIoVVdmRZ0jNm7cWKnHmNM1H+3cuZPOnTtjMplwdXXloYce4scff6yWWMxmM5GRkeh0OrRaLQ0bNuTq1atcvXqV119/nYiICBYvXlwtD/e4cOECkiTx5JNPMmjQIL744osaVXYAb7zxBpMnT8ZgMFRbma1evZp//vOf+Pr6AnD06FGCgoKoW7cuGo2GiIgIfvzxR65cuUJOTg5t27YFYNiwYZVadrfHpdPpeOONNzAajUiSROPGjbl69SoAx48fZ9myZURERDB79mxyc3OrLK6srKwiv7viyrEy3R7brebPn8+oUaOoX78+UHVlVtQ54tKlS5V6jDldUkhISMBs/vNJRL6+vsTHx1dLLI0aNXJ8gZcuXWLTpk10796dzp078/bbb7N69Wr279/PmjVrqjy2tLQ0unTpwscff8zy5ctZuXIlV69erTFlt3PnTnJycggPDyc5ObnaymzOnDmEhoY6/i7u+Lp9udlsrtSyuz2u2rVr07VrVwBSUlL48ssv6dWrF5mZmTRr1oypU6eybt060tLS+OSTT6osruK+u+r4nd4eW75Lly6xd+9exo0bB1ClZVbUOUKSpEo9xpwuKShFTAorSdU7Je7Zs2eZMGECU6dOJTg4mI8//hhvb29cXFwYO3Ysv/32W5XH1K5dO+bPn4+rqyteXl6MGDGCxYsXF1qvuspu5cqVjB8/HoC6devWiDKD4o+vmnLcxcfH89hjjzF8+HA6deqEm5sby5YtIygoCI1Gw4QJE6q07Ir77mpKeQGsWrWKRx55BJ0u71nH1VFmt54j6tWrV+j1ijzGnC4p+Pn5kZSU5Pg7ISGhyOpiVTlw4ACPP/44//jHPxg6dCinT5/mp59+cryuKEqldkYWZ//+/ezatatAHLVr164RZWexWNi3bx89e/YEqDFlBsUfX7cvT0xMrPKyO3/+PKNHj2bo0KE899xzAFy9erVAraqqy664764m/U63bNlC//79HX9XdZndfo6o7GPM6ZJC165d2bVrFykpKWRnZ/Pzzz/zwAMPVEsscXFxPPfccyxYsIABAwYAeQfY22+/zY0bN7BaraxatapaRh6lp6czf/58cnNzycjIYN26dbz77rs1ouxOnz5N/fr1cXV1BWpOmQG0adOGixcvcvnyZex2O99//z0PPPAAtWvXRq/Xc+DAAQDWr19fpWWXkZHB3//+dyZNmsSECRMcyw0GA++++y4xMTEoisKXX35ZpWVX3HdXXDlWtZSUFHJycqhbt65jWVWWWVHniMo+xpxuSKqfnx+TJ09m3LhxWK1WRowYQevWrasllv/85z/k5ubyzjvvOJaNGjWKiRMnMnr0aGw2G3379mXgwIFVHluPHj04cuQIQ4YMQZZlHnnkEe67774aUXYxMTH4+/s7/m7atGmNKDMAvV7PO++8wwsvvEBubi5hYWH069cPgAULFjBjxgwyMzNp3ry5o426KqxZs4akpCQ+++wzPvvsMwB69uzJpEmTmD17Ns888wxWq5X27ds7muWqwp2+u+LKsSrFxsYWONYAvLy8qqzMijtHVOYxJp68JgiCIDg4XfORIAiCUDyRFARBEAQHkRQEQRAEB5EUBEEQBAeRFARBEAQHkRQEoQI8+eSTnDt3rkzveeqpp1i7dm0lRSQI5eN09ykIQmVYtmxZdYcgCBVCJAXBqf3666+O50MYDAamTp3K77//ztmzZ0lKSiI5OZmmTZsyZ84cjEYjX331FStXrkSr1aLX65k9ezYhISH07NmTRYsW0apVK8dzC1QqFT4+Prz++us0aNCA+Ph4IiMjSUhIIDAwkOTkZEcc58+fZ86cOVy/fh273c7YsWMZMWIEmZmZTJs2jcuXL6NSqWjRogWzZ89GpRKVfKGSlH+mb0H4a7t48aIycOBAJSUlRVGUvOcz3H///co777yjPPDAA0piYqJit9uVl19+WXnnnXcUm82mtGjRwjH3/7p165SVK1cqiqIoPXr0UI4ePars3LlT6d27t5KcnKwoiqJ8++23Snh4uCLLsvLss88qH3zwgaIoinLp0iWlbdu2yrfffqtYrValf//+yvHjxxVFUZS0tDQlPDxcOXTokLJu3TplwoQJiqIois1mU1577TXl0qVLVVlMgpMRNQXBae3YsYOEhAQef/xxxzJJkoiOjqZfv374+PgAMGLECN5++22mTp1Kv379GDVqFA8++CD3338/ERERBba5fft2+vfvj5eXF5A3p/2cOXOIjY1l586dTJ06FYCgoCA6deoE5E2JHB0dzfTp0x3bycnJ4cSJE3Tv3p0PPviAsWPH0rVrVx577DGCgoIqs1gEJyeSguC0ZFmmS5cuLFy40LEsLi6OVatWYbFYCqyX31yzYMECzpw5w86dO1m2bBlr1qxhyZIljnWVImaNURQFm81WaHrj/Jk17XY7Hh4ebNiwwfFaUlIS7u7u6PV6fvnlF/bs2cPu3bsZP348M2bMqJZ5gATnIBomBafVuXNnduzYwfnz5wH47bffGDRoELm5uWzZsoX09HRkWWb16tX06NGDlJQUwsLCMJlMPP7447z00kucPn26wDa7devGpk2bSElJAeDbb7/FZDIRFBRE9+7dWbVqFZA3/fKePXsAaNCgAXq93pEU4uLiGDhwIMePH+err75i2rRpdOvWjSlTptCtWzfOnj1bVUUkOCExIZ7g1KKioli6dKljTvzp06eza9cudu/ejd1uJzU1lQ4dOjBjxgwMBgMrV67kv//9LwaDAbVazeTJk+natWuBjuYvv/ySlStXIssyXl5ezJw5k0aNGpGSksK0adOIjo7G398fm83G0KFDGTZsGKdOnXJ0NNtsNsaNG8fo0aPJyspi+vTpnD59GhcXFwIDA5kzZw6enp7VXXTCPUokBUG4zYcffkhqaiozZ86s7lAEocqJ5iNBEATBQdQUBEEQBAdRUxAEQRAcRFIQBEEQHERSEARBEBxEUhAEQRAcRFIQBEEQHERSEARBEBz+H0nZIJBEFGUBAAAAAElFTkSuQmCC\n" - }, - "metadata": {} - } - ], - "source": [ - "cfg = QlearningConfig()\n", - "env = gym.make(\"CliffWalking-v0\") # 0 up, 1 right, 2 down, 3 left\n", - "env = CliffWalkingWapper(env)\n", - "action_dim = env.action_space.n\n", - "agent = QLearning(action_dim,cfg)\n", - "rewards,ma_rewards = train(cfg,env,agent)\n", - "plot_rewards(rewards,ma_rewards,tag=\"train\",algo = \"On-Policy First-Visit MC Control\",save=False)" - ] - } - ] -} \ No newline at end of file diff --git a/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/models/Qleaning_model.pkl b/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/models/Qleaning_model.pkl new file mode 100644 index 0000000..cd4f55b Binary files /dev/null and b/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/models/Qleaning_model.pkl differ diff --git a/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/eval_ma_rewards.npy b/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/eval_ma_rewards.npy new file mode 100644 index 0000000..a67d064 Binary files /dev/null and b/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/eval_ma_rewards.npy differ diff --git a/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/eval_rewards.npy b/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/eval_rewards.npy new file mode 100644 index 0000000..6de67e1 Binary files /dev/null and b/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/eval_rewards.npy differ diff --git a/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/eval_rewards_curve.png b/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/eval_rewards_curve.png new file mode 100644 index 0000000..776685a Binary files /dev/null and b/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/eval_rewards_curve.png differ diff --git a/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/train_ma_rewards.npy b/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/train_ma_rewards.npy new file mode 100644 index 0000000..c18ceaf Binary files /dev/null and b/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/train_ma_rewards.npy differ diff --git a/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/train_rewards.npy b/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/train_rewards.npy new file mode 100644 index 0000000..fb83837 Binary files /dev/null and b/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/train_rewards.npy differ diff --git a/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/train_rewards_curve.png b/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/train_rewards_curve.png new file mode 100644 index 0000000..d737d7f Binary files /dev/null and b/codes/QLearning/outputs/CliffWalking-v0/20210429-165825/results/train_rewards_curve.png differ diff --git a/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/models/Qleaning_model.pkl b/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/models/Qleaning_model.pkl new file mode 100644 index 0000000..5d963e7 Binary files /dev/null and b/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/models/Qleaning_model.pkl differ diff --git a/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/eval_ma_rewards.npy b/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/eval_ma_rewards.npy new file mode 100644 index 0000000..a67d064 Binary files /dev/null and b/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/eval_ma_rewards.npy differ diff --git a/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/eval_rewards.npy b/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/eval_rewards.npy new file mode 100644 index 0000000..6de67e1 Binary files /dev/null and b/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/eval_rewards.npy differ diff --git a/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/eval_rewards_curve.png b/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/eval_rewards_curve.png new file mode 100644 index 0000000..2992927 Binary files /dev/null and b/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/eval_rewards_curve.png differ diff --git a/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/train_ma_rewards.npy b/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/train_ma_rewards.npy new file mode 100644 index 0000000..13d68d5 Binary files /dev/null and b/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/train_ma_rewards.npy differ diff --git a/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/train_rewards.npy b/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/train_rewards.npy new file mode 100644 index 0000000..c504109 Binary files /dev/null and b/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/train_rewards.npy differ diff --git a/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/train_rewards_curve.png b/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/train_rewards_curve.png new file mode 100644 index 0000000..9f0ed62 Binary files /dev/null and b/codes/QLearning/outputs/CliffWalking-v0/20210429-170453/results/train_rewards_curve.png differ diff --git a/codes/QLearning/results/20210326-171621/ma_rewards_train.npy b/codes/QLearning/results/20210326-171621/ma_rewards_train.npy deleted file mode 100644 index 0f842f2..0000000 Binary files a/codes/QLearning/results/20210326-171621/ma_rewards_train.npy and /dev/null differ diff --git a/codes/QLearning/results/20210326-171621/rewards_curve_train.png b/codes/QLearning/results/20210326-171621/rewards_curve_train.png deleted file mode 100644 index 985b8c7..0000000 Binary files a/codes/QLearning/results/20210326-171621/rewards_curve_train.png and /dev/null differ diff --git a/codes/QLearning/results/20210326-171621/rewards_train.npy b/codes/QLearning/results/20210326-171621/rewards_train.npy deleted file mode 100644 index ed8f524..0000000 Binary files a/codes/QLearning/results/20210326-171621/rewards_train.npy and /dev/null differ diff --git a/codes/QLearning/saved_model/20210326-171621/Qleaning_model.pkl b/codes/QLearning/saved_model/20210326-171621/Qleaning_model.pkl deleted file mode 100644 index 47e7279..0000000 Binary files a/codes/QLearning/saved_model/20210326-171621/Qleaning_model.pkl and /dev/null differ diff --git a/codes/QLearning/task0_eval.py b/codes/QLearning/task0_eval.py new file mode 100644 index 0000000..851cfe6 --- /dev/null +++ b/codes/QLearning/task0_eval.py @@ -0,0 +1,84 @@ +#!/usr/bin/env python +# coding=utf-8 +''' +Author: John +Email: johnjim0816@gmail.com +Date: 2020-09-11 23:03:00 +LastEditor: John +LastEditTime: 2021-04-29 17:01:43 +Discription: +Environment: +''' +import sys,os +curr_path = os.path.dirname(__file__) +parent_path=os.path.dirname(curr_path) +sys.path.append(parent_path) # add current terminal path to sys.path + +import gym +import datetime + +from envs.gridworld_env import CliffWalkingWapper +from QLearning.agent import QLearning +from common.plot import plot_rewards +from common.utils import save_results + +curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # obtain current time + +class QlearningConfig: + '''训练相关参数''' + def __init__(self): + self.algo = 'Qlearning' + self.env = 'CliffWalking-v0' # 0 up, 1 right, 2 down, 3 left + self.result_path = curr_path+"/outputs/" +self.env+'/'+curr_time+'/results/' # path to save results + self.model_path = curr_path+"/outputs/" +self.env+'/'+curr_time+'/models/' # path to save models + self.train_eps = 300 # 训练的episode数目 + self.eval_eps = 30 + self.gamma = 0.9 # reward的衰减率 + self.epsilon_start = 0.95 # e-greedy策略中初始epsilon + self.epsilon_end = 0.01 # e-greedy策略中的终止epsilon + self.epsilon_decay = 200 # e-greedy策略中epsilon的衰减率 + self.lr = 0.1 # learning rate + +def env_agent_config(cfg,seed=1): + env = gym.make(cfg.env) + env = CliffWalkingWapper(env) + env.seed(seed) + state_dim = env.observation_space.n + action_dim = env.action_space.n + agent = QLearning(state_dim,action_dim,cfg) + return env,agent + +def eval(cfg,env,agent): + # env = gym.make("FrozenLake-v0", is_slippery=False) # 0 left, 1 down, 2 right, 3 up + # env = FrozenLakeWapper(env) + rewards = [] # 记录所有episode的reward + ma_rewards = [] # 滑动平均的reward + for i_ep in range(cfg.eval_eps): + ep_reward = 0 # 记录每个episode的reward + state = env.reset() # 重置环境, 重新开一局（即开始新的一个episode） + while True: + action = agent.predict(state) # 根据算法选择一个动作 + next_state, reward, done, _ = env.step(action) # 与环境进行一个交互 + state = next_state # 存储上一个观察值 + ep_reward += reward + if done: + break + rewards.append(ep_reward) + if ma_rewards: + ma_rewards.append(ma_rewards[-1]*0.9+ep_reward*0.1) + else: + ma_rewards.append(ep_reward) + print(f"Episode:{i_ep+1}/{cfg.eval_eps}, reward:{ep_reward:.1f}") + return rewards,ma_rewards + +if __name__ == "__main__": + cfg = QlearningConfig() + env,agent = env_agent_config(cfg,seed=15) + cfg.model_path = './'+'QLearning/outputs/CliffWalking-v0/20210429-165825/models'+'/' + cfg.result_path = './'+'QLearning/outputs/CliffWalking-v0/20210429-165825/results'+'/' + agent.load(path=cfg.model_path) + rewards,ma_rewards = eval(cfg,env,agent) + save_results(rewards,ma_rewards,tag='eval',path=cfg.result_path) + plot_rewards(rewards,ma_rewards,tag="eval",env=cfg.env,algo = cfg.algo,path=cfg.result_path) + + diff --git a/codes/QLearning/task0_train.ipynb b/codes/QLearning/task0_train.ipynb new file mode 100644 index 0000000..3862439 --- /dev/null +++ b/codes/QLearning/task0_train.ipynb @@ -0,0 +1,230 @@ +{ + "metadata": { + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.10" + }, + "orig_nbformat": 2, + "kernelspec": { + "name": "python3710jvsc74a57bd0fbea1422c2cf61ed9c0cfc03f38f71cc9083cc288606edc4170b5309b352ce27", + "display_name": "Python 3.7.10 64-bit ('py37': conda)" + } + }, + "nbformat": 4, + "nbformat_minor": 2, + "cells": [ + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import sys\n", + "from pathlib import Path\n", + "curr_path = str(Path().absolute())\n", + "parent_path = str(Path().absolute().parent)\n", + "sys.path.append(parent_path) # add current terminal path to sys.path\n", + "\n", + "import gym\n", + "import datetime\n", + "\n", + "from envs.gridworld_env import CliffWalkingWapper\n", + "from QLearning.agent import QLearning\n", + "from common.plot import plot_rewards\n", + "from common.utils import save_results,make_dir\n", + "curr_time = datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\") # obtain current time" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "class QlearningConfig:\n", + " '''训练相关参数'''\n", + " def __init__(self):\n", + " self.algo = 'Qlearning'\n", + " self.env = 'CliffWalking-v0' # 0 up, 1 right, 2 down, 3 left\n", + " self.result_path = curr_path+\"/outputs/\" +self.env+'/'+curr_time+'/results/' # path to save results\n", + " self.model_path = curr_path+\"/outputs/\" +self.env+'/'+curr_time+'/models/' # path to save models\n", + " self.train_eps = 300 # 训练的episode数目\n", + " self.eval_eps = 30\n", + " self.gamma = 0.9 # reward的衰减率\n", + " self.epsilon_start = 0.95 # e-greedy策略中初始epsilon\n", + " self.epsilon_end = 0.01 # e-greedy策略中的终止epsilon\n", + " self.epsilon_decay = 200 # e-greedy策略中epsilon的衰减率\n", + " self.lr = 0.1 # learning rate" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "def env_agent_config(cfg,seed=1):\n", + " env = gym.make(cfg.env) \n", + " env = CliffWalkingWapper(env)\n", + " env.seed(seed)\n", + " state_dim = env.observation_space.n\n", + " action_dim = env.action_space.n\n", + " agent = QLearning(state_dim,action_dim,cfg)\n", + " return env,agent" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "def train(cfg,env,agent):\n", + " rewards = [] \n", + " ma_rewards = [] # moving average reward\n", + " for i_ep in range(cfg.train_eps):\n", + " ep_reward = 0 # 记录每个episode的reward\n", + " state = env.reset() # 重置环境, 重新开一局（即开始新的一个episode）\n", + " while True:\n", + " action = agent.choose_action(state) # 根据算法选择一个动作\n", + " next_state, reward, done, _ = env.step(action) # 与环境进行一次动作交互\n", + " agent.update(state, action, reward, next_state, done) # Q-learning算法更新\n", + " state = next_state # 存储上一个观察值\n", + " ep_reward += reward\n", + " if done:\n", + " break\n", + " rewards.append(ep_reward)\n", + " if ma_rewards:\n", + " ma_rewards.append(ma_rewards[-1]*0.9+ep_reward*0.1)\n", + " else:\n", + " ma_rewards.append(ep_reward)\n", + " if (i_ep+1)%10==0:\n", + " print(\"Episode:{}/{}: reward:{:.1f}\".format(i_ep+1, cfg.train_eps,ep_reward))\n", + " return rewards,ma_rewards" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "def eval(cfg,env,agent):\n", + " # env = gym.make(\"FrozenLake-v0\", is_slippery=False) # 0 left, 1 down, 2 right, 3 up\n", + " # env = FrozenLakeWapper(env)\n", + " rewards = [] # 记录所有episode的reward\n", + " ma_rewards = [] # 滑动平均的reward\n", + " for i_ep in range(cfg.eval_eps):\n", + " ep_reward = 0 # 记录每个episode的reward\n", + " state = env.reset() # 重置环境, 重新开一局（即开始新的一个episode）\n", + " while True:\n", + " action = agent.predict(state) # 根据算法选择一个动作\n", + " next_state, reward, done, _ = env.step(action) # 与环境进行一个交互\n", + " state = next_state # 存储上一个观察值\n", + " ep_reward += reward\n", + " if done:\n", + " break\n", + " rewards.append(ep_reward)\n", + " if ma_rewards:\n", + " ma_rewards.append(ma_rewards[-1]*0.9+ep_reward*0.1)\n", + " else:\n", + " ma_rewards.append(ep_reward)\n", + " if (i_ep+1)%10==0:\n", + " print(f\"Episode:{i_ep+1}/{cfg.eval_eps}, reward:{ep_reward:.1f}\")\n", + " return rewards,ma_rewards" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Episode:10/300: reward:-158.0\n", + "Episode:20/300: reward:-131.0\n", + "Episode:30/300: reward:-37.0\n", + "Episode:40/300: reward:-93.0\n", + "Episode:50/300: reward:-47.0\n", + "Episode:60/300: reward:-67.0\n", + "Episode:70/300: reward:-56.0\n", + "Episode:80/300: reward:-44.0\n", + "Episode:90/300: reward:-41.0\n", + "Episode:100/300: reward:-61.0\n", + "Episode:110/300: reward:-52.0\n", + "Episode:120/300: reward:-14.0\n", + "Episode:130/300: reward:-44.0\n", + "Episode:140/300: reward:-31.0\n", + "Episode:150/300: reward:-17.0\n", + "Episode:160/300: reward:-35.0\n", + "Episode:170/300: reward:-34.0\n", + "Episode:180/300: reward:-16.0\n", + "Episode:190/300: reward:-20.0\n", + "Episode:200/300: reward:-25.0\n", + "Episode:210/300: reward:-13.0\n", + "Episode:220/300: reward:-16.0\n", + "Episode:230/300: reward:-20.0\n", + "Episode:240/300: reward:-27.0\n", + "Episode:250/300: reward:-17.0\n", + "Episode:260/300: reward:-14.0\n", + "Episode:270/300: reward:-15.0\n", + "Episode:280/300: reward:-20.0\n", + "Episode:290/300: reward:-13.0\n", + "Episode:300/300: reward:-13.0\n", + "results saved!\n" + ] + }, + { + "output_type": "display_data", + "data": { + "text/plain": "
", + "image/svg+xml": "\n\n\n\n \n \n \n \n 2021-04-29T17:04:54.671110\n image/svg+xml\n \n \n Matplotlib v3.3.4, https://matplotlib.org/\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEXCAYAAABCjVgAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABK2klEQVR4nO3dd3wU1f7/8dfMtpRNhRRCCb13gpQAAaSFEDoKIorY9Vq4P7kELFcpKlwUEBXLtXzFAooQBGmCF5AqvUgRkRpCOunJtvn9EbIQktBDAvt5Ph482J3MzJ4zk+x7zjlTFE3TNIQQQghALe8CCCGEqDgkFIQQQjhJKAghhHCSUBBCCOEkoSCEEMJJQkEIIYSThMJdaM6cOUyaNOm2fNbjjz/OX3/9dVs+q7zNnTuXrl27MmHChGI/y83N5Z133qFv375ER0cTHR3NzJkzyc3Ndc7ToEEDUlNTy7yca9euZcqUKbdsfVeq9/U4duwYzz33HNHR0fTv358HH3yQHTt2AHDmzBlatWoFwHfffccnn3wCwMaNG+nWrRtDhgxh9+7d9OjRg0GDBtG8eXPWrFnjXPfGjRtp0KAB8+fPd07bt28f4eHhXOms++7du7N//362bdtGv379iv18//79PP/88zdV72u1cOFCIiMj6dWrF//+97+xWq235XOL0cRd57333tPeeOON8i7GXad79+7a9u3bi023Wq3a8OHDtcmTJ2u5ubmapmlaTk6ONnnyZG348OGa1WrVNE3T6tevr6WkpNzWMt8KpdX7ehw7dkwLDw/XNmzY4Jy2efNmrU2bNtqff/6pnT59WmvZsmWx5WJiYrQPPvhA0zRNmzNnjjZx4kRN0zTtzTff1KZMmeKcb9KkSdqTTz6pPf74485pH3/8sRYTE3PFcnXr1k3bt2+ftnXrVi0qKuqm6ngzjhw5onXp0kVLSUnR7Ha7NnbsWO2TTz4pl7LoyyeKKj6Hw8Gbb77J3r17yc7ORtM0pkyZQv369YmIiGDVqlUEBAQAcN999/Hss8/SoUMHZsyYwfbt27Hb7TRu3JhXXnkFs9lM9+7dad68OUeOHOGf//wner2ejz/+GIvFQmpqKgMHDuTFF18E4JNPPmHhwoV4enoSFhbG2rVr+fXXX7FYLKWuvzQJCQlMmjSJ+Ph4rFYrUVFRPPXUUwB89NFHrFmzhvz8fHJzcxk/fjw9e/Zkzpw57Nmzh8TERBo0aEBoaChxcXEkJSURFxeHv78/M2fOJCgoiO7duzN79mxycnKYOXMm1atX5+jRo1gsFl577TXat29PamoqEyZM4NSpU/j6+hIQEEC9evV47rnnipQ1OzubKVOmsGvXLnQ6HT169GDs2LFMmDCBevXq8eijjwIQExPjfH/pdn3uueeYO3cuS5cuBSAjI4N7772XNWvWkJeXV+p2uNS5c+d4/fXXiYuLQ9M0Bg4cyGOPPcaLL75IQkICL7/8Mi+88AJ9+/Z1LvPLL7+Qm5vLxIkTUdWCxre7uzsvv/wyAwcOZPXq1UXmB/jhhx/47rvvcDgc+Pr68uqrr1KnTh2OHz/OpEmTyMnJITExkYYNGzJr1ixMJhNNmzbl3nvv5fDhw8yYMYMHHniAJ554gk2bNpGYmMhDDz3E6NGjWbRoEatWreLjjz9m1KhRtGzZkl27dhEfH0+bNm2YNm0aqqqyaNEiPvnkE9zc3Gjfvj1fffUVBw8eLFLOy+vdunXrErfPmTNnGDlyJHXq1CEuLo558+YRGBjoXM+nn37KkCFD6Ny5s3Nahw4deOedd3BzcyvymXPmzCEtLY2QkBDWrl2LyWQiNjaW7Oxs7HY7eXl5DB48mOnTpzuX+d///sdnn33GfffdR05ODh4eHmzZsoX777+f5ORkXnvtNVJSUkhKSqJq1arMmjWLSpUqlfg3s2PHDsaNG8c777yD1Wpl8uTJLFu2jJiYGMxmM0eOHOHcuXPUrl2bd999F09PT9avX8+MGTNQVZVGjRqxefNmvv32W6pVq1Zk3cOHD2f06NH06dMHgBkzZqBpmvM7wt/fH4D777+fKVOm8Pjjj5dYxrIk3Uel2Lt3L4mJiSxYsIDly5czaNAgPv30U7y8vOjZsyc//fQTUNAkTkpKonPnznzyySfodDoWLVrETz/9RGBgIDNmzHCus169eqxYsYIePXrw+eef8/bbb7No0SIWLFjAJ598QmpqKr/99huLFi1i4cKFLFq0iOzsbOfyV1t/ScaNG8eQIUOc69y8eTPLly8nLi6OzZs38/XXX7N06VLGjh3Le++951wuLi6OxYsXO9e/Y8cOZs+ezcqVK/H29mbBggXFPmvfvn2MGTOG2NhYhg4dyvvvvw/AlClTqFu3LitWrGD27Nns2rWrxLK+99575Ofns3z5cmJjY9m1axe///77VfdV4XaNjIwkOzub/fv3A7Bs2TIiIiLw8fEpdTtc7qWXXqJdu3YsXbqU7777jp9++omff/6ZWbNmObf35V/wO3fupE2bNs5AKKQoCh07dixW399//53Y2Fi++eYbYmNjeeyxx5wB+f333zNw4EAWLFjA6tWrOXPmDOvWrQPAarXSrVs3Vq1aRbNmzbBYLPj5+TF//nzee+893nnnHfLz84vV6dSpU8ybN4+ffvqJrVu38vvvv/PXX38xY8YMvvzyS2JjYzGbzdjt9mLLXl7v0rYPFATqM888w6pVq4oEAsCBAwdo3bp1sfVHRERQvXr1YtMBHnvsMbp3787o0aNZvXo1w4cPp2/fvrzzzju0bduWU6dOcf78eY4cOYKPjw+1atWiefPmbNq0CYvFwr59++jUqRM///wzLVu2ZMGCBaxduxY3NzeWLFlS4mdu3bqVCRMmMHfu3BLLe+DAAT777DOWL19OYmIiK1euJC0tjX/961/85z//YcmSJbRr146EhIQS1z9s2DAWL14MgN1u56effmLYsGHEx8dTpUoV53zBwcGlrqOsSUuhFK1atcLHx4f58+dz+vRptm3bhqenJ1CwY9944w0effRRfvzxRwYPHoyqqqxbt47MzEw2b94MFPwRX3o0EhYWBhR8WXz00UesW7eOZcuWcezYMTRNIzc3l/Xr19OnTx+8vb0BGDlyJFu3bgW46vovl5OTw/bt20lPT2f27NnOaYcPH6Zv375MmzaNpUuXcvLkSWeLqFDLli3R6y/+etxzzz3OFknjxo1JT08v9nkhISE0atTIOU/hL//69eudrwMDA51HSZfbvHkzEyZMQKfTodPp+PrrrwGcy5bm0u06dOhQFi9eTLNmzVi0aBHjxo276na4dHvt2rWLzz//HAAvLy8GDx7Mhg0biIqKumIZrsThcBR5v27dOk6ePMnw4cOd09LT0zl//jzjxo1j06ZNfPrpp5w4cYLExERycnKK1bXQvffeC0CTJk2wWCxF5i3UrVs3VFXFbDYTGhpKeno6hw8fJjw8nODgYAAefPBB5syZc8V6XGn7tGjRAr1eT8uWLUtcVlGUYtvhZhiNRtq1a8eOHTv466+/6Nq1q7OuGzduxNvbmyZNmmA2m3n44YfZsWMHX3zxBSdOnODo0aO0aNGi2DrPnTvHU089xYgRI2jYsGGJn9u5c2eMRiMA9evXJz09nR07dlCnTh3nMoMGDSp1TCcyMpLp06eTlJTEwYMHCQ0NpWbNmiWOe1x+kHG7SCiUYt26dUydOpVHHnmEe++9l9q1aztbB2FhYdhsNvbt28eyZcucg1sOh4OJEycSEREBFHSHXHrk5uHhART8cQ0aNIgePXoQFhbGkCFDWLNmDZqmodfri/yC6HQ65+urrf9yDocDTdOYP38+7u7uAKSmpmIymfjjjz945plnGD16NOHh4bRt25Y33nijWFkLXdrEVxSlxF/i0ua5vE6l/bLr9XoURXG+j4+Px83NrdjnXT4Ad2lZhwwZwsCBAxk2bBiZmZm0a9eOrKysUrdDSdvr8mk2m63E8hZq3bo1H3/8MQ6HA1VVycrKQlEUPD092bZtG6NHjy62zgEDBjBu3Djn+8TERHx8fBg7dix2u53IyEi6du1KfHx8kTJdvl8K61C43a51v+h0ulJ/z0pzte1jNBqLHEhcqmXLluzZs4du3boVmf7+++9To0aNEo/Kr6Zz585s376dvXv3MnHiRKCg5bFgwQL8/f2dQfGf//yHffv2MWTIENq1a4fNZitxO+l0Oj755BOeeeYZIiMjad68ebF5rmVbwsXf8ZdffpkDBw4ABV1HI0aMoHfv3ixbtozdu3czbNgwAKpUqUJiYqJz+YSEBGdg327SfVSKTZs20a1bNx544AGaNWvGmjVrijSvhw0bxuTJk2nQoAEhISEAdOrUiW+++QaLxYLD4eDVV1/l3XffLbbukydPkpWVxYsvvkj37t35/fffnctERESwevVqMjMzgYIzEgpd6/oLmc1mWrZsyRdffAEU9LGPGDGCtWvXsn37dpo2bcojjzzCPffcw9q1a0vsPrgVIiIinPVIS0tjzZo1Rb78C3Xo0IHFixfjcDiwWCw8//zzbN++HT8/P+cfVmpqqvOMlZIEBQXRokULXnvtNYYOHQpceTtcymw206JFC7755hsAMjMziY2NpWPHjlesX69evfD09OTNN98kLy+PQ4cOMXjwYJ566il0Ol2x7qbw8HB+/vln55fAd999x8MPPwwUnEXz7LPP0rdvXxRFYe/evWWyXzp16sSWLVucXRQ//PDDVZe50e0D8Oijj/LDDz+wceNG57QNGzYwb968Uo/Kr6ZLly5s2rSJs2fP0qxZMwBnV9SaNWucB08bN27k4YcfZuDAgVSqVInNmzeXuE0DAgJo3bo148ePZ9y4cUXOHLuS1q1bc+LECQ4fPgzAqlWryMjIQFEUpk6dypIlS1iyZAkjRowACsYgFy1axO7du+nduzdQcBbUr7/+SkpKCpqmsWDBAnr06HFD2+VmSUuhFMOHD+ell14iOjoanU5HWFgYq1evdh4NDhw4kHfffbfIl/IzzzzDtGnTGDRoEHa7nUaNGhETE1Ns3Q0aNKBr165ERkbi7e1NjRo1qFu3LidPnqRz587cd9993H///bi5uVGvXj3n0e21rv9SM2bMYPLkyURHR2OxWOjXrx/9+/cnOTnZOQBqMBjo0KED6enpZGVl3doNCUyYMIFXXnmF6OhofH19CQkJKTa4CPCPf/yDqVOnMmDAAOx2O3379qVXr140a9aMl156id69e1OtWjXuueeeK37esGHDeOGFF5g7d+5Vt8PlZsyYwaRJk1i0aBEWi4Xo6GgGDx58xc/T6/V89tlnfPjhhwwaNMh51O3h4UFSUhK//fZbkSPkzp078/jjjzNmzBgURcFsNvP++++jKApjx47l2WefxcfHB3d3d2ff+a1Wq1YtJkyYwKOPPorRaKRRo0bO37MrKW37xMXFXXG50NBQPvroI2bNmsW0adNwOBz4+/szd+5c6tevz5kzZ667DtWrV8dmsxEeHl7kIKNz586sXr2aOnXqAPDss88yffp0PvzwQ3Q6Ha1bt77iNh00aBCrVq3i7bffLhboJfH19eXdd99l/PjxqKpK06ZN0ev1pW7Pwp/37t3b2dJr2LAhzz77LA8//DBWq5UWLVqUyyAzIKekVjT79u3T/u///s/5/vPPP9deeOGF8ivQLfD1119ru3bt0jRN0/Lz87UhQ4Zo69atK+dS3R6JiYk3fTpnWTh16pQ2Z84czW63a5qmaatWrdKGDh1azqW6M2VmZmrTpk3TcnJyNE3TtAMHDmjh4eGaw+Eo55LdGGkpVDC1atXi008/5fvvv0dRFKpUqcLkyZPLu1g3pW7dukyePBmHw4HVaqVPnz7Opv3dLiAgwHnqckUSHBxMYmKisyXs5eXFm2++Wd7FuiOZzWYMBgNDhw5Fr9ej1+uZNWtWiV2kdwJF0+QhO0IIIQrIQLMQQggnCQUhhBBOEgpCCCGcJBSEEEI43RVnH6WlZeNwXP94eaVKZlJSbv15+eVB6lLx3C31AKlLRXUjdVFVBT8/z1J/fleEgsOh3VAoFC57t5C6VDx3Sz1A6lJR3eq6SPeREEIIJwkFIYQQThUiFJYuXUrfvn3p2bOn82ZbQgghbr9yH1NISEhg5syZLFq0CKPRyPDhw2nXrh1169Yt76IJIYTLKfeWwubNm2nfvj2+vr54eHjQu3dvVq5cWd7FEkIIl1TuoZCYmFjkhmGBgYHl9hg6IcrS3XSbseupy62ud3lsR03TcFzhc6+3TKWtr3B64focl70vbdqtVO7dRyVV7HrvLlipUukPrr+agACvG162ornZupxJzMTh0KgR7H2LSnSR3aFx9FQaDUL9nPs332pnxeYTdA+rjrdnwSMOC0+vu7wu51KyCfB1R6e7eBxjtTnIt9jwdDeU+Dvz/g978DGbGBXZqNjPks/nkmexUTXAjEMDnaqQl28jPiWbaoFmDPqiTyJLzcjj77h02jQMvPBoSQ1VLfqZKem5uJv0WKwOYtf/xeBu9Zz1OJ2QyeTPfueJQc0IaxRU4jbadTiRaoFmvll1GLOHgcY1K7Fm+yk6NqtCz3ahJKXl4uGmJ99qx2jQEbv+L84kZNGlVVUa1fTHYNCxassJsvOsjOjVAINex96jSWzed5b0bAsBvu50aBpEVo6V0BBf5v64l+TULN56ugM7jyTz+8FzjHuoHX+dOc+6nWeoGmhmy754RvVtxPGD58jIstC1TTUOHk/hvQV7GBhRhx/WHmVIt7qcjM9gx8F4Jo5pzzcrD1Ovhh8976nB4ROpfLrkAE3rVKJ7m+ocOZXGqXOZJJ/PoVaAEXtOFrlZ6XRsEkTHFtWJT8lhycbjpKVnE+pvJCXlPDqHhY6tQvHxNfPzb8dwN6lU9jZhtdjIybOg2W3Uq+ZDemYuefkW4hIyqFrZk87Ng0lIzuTQ8WTyLVb8zUbc9KCqkJGZS2VfD3RGE4ePJ1MtwAO9qpGTnY+vWU9yahZGvUJOroUqwf6cij+P6rBTyawjPy+fyn4eaChk5Njw9DCRkJZHlUAvMrLysVhs+HgasNls5OVZsVqt+JqNKJoDVVFwM6qkZuRjsdpwN+qp7OvO+cw8LDY7iuYAuxUPTw8cqoHzGTnoFTtGVUMfWIvEWn35evWfuCsWWjaqysRHw2/5d1i53yV18eLF7Nixg6lTpwLwwQcfoGka//jHP655HSkpWTd0rm5AgBdJSZnXvVxZy7fYMRmv/HjEA8dTqFfNF5OhYL5rqcunS/8gNNibXm1LflD61K92kJlrZcpj7QqOWBxgtTswuxtYtvkEf54+zz/vb+mc32pz8PfZdExGHTUvBMm+Y8ms+v00zw5qioebwTnvxn3xfL78EE8PbErbhgUPdV+/J47/W3mEetV8GH5vPczuBib/3w6iOtXCZrHRvnEwG/fHExrsxQeL9tOzbXU6Ng0myM+DtKx8pn2zi/OZuUR1qMXRM+l4u6tkpJ4nPyudtq3qsnL7ady0PLq0qIKvp5HzaZkYdBphjUP4fNkByMumsrsdd6NKRNvarN3yF5bcLHzcVBrUDiTfrlI9xJ/9x9M4HpeG3WKhUXUvHDYr8UnpVPE14uWmkpSSiZ+HSlp6NnoVTO4e2HIyMLsbyLfY0KsKGEyQm4FOBQcqql5PZT9PfHy88G7Tl/8dh2/XHMWoOvAhi0r6XLzJxF/NxqjaqBPsydnEDHSKA8VhR684cCMfP30+es2CnoJpKg4UNFBUVDczKXkqeocFd9WKEQtuig2HBjb06LCjUy7+3WQ7jNjaj+G7DacxWLNxVyy4q5aC/y/8C/bWkZNvR2/NQkPBTbHiqeTjpeahoPGHXzd+OWmisi4THzUXbyWHYA8bOms2buTjoVgw6yy4kY9K+Xz1ODQFTVFA05z1d2gKDhQcqNg0FUVVsV8ooUGzoik60OnJs6ug6rBYC7adXlXQ7DZ0KgVf6IqCqqpY7YCqFjxwSVHJsdhRFB0aBQdIBr0ON6OOPKudPMuFsDDpQVHRVD252dkYFHvB42j1erLzbIRoCeQoHhgc+RgUO3m+tWj87Izr/g5TVeWKB9LlHgoJCQmMGDGChQsX4u7uzvDhw5k8eXKJz0ctzZ0SCgeOp/Djur+Z8GBrjIaSv/T/OpPOtG938e9H2lItoOQddzw+g8n/t4PwpsGM7FUfk0GHVVFRbHYMepVN++OJT8mhQ9Ngqlb2ZNnmE/iaTXy+/BDuJh1vPdkBT7eCI9r1e84S7O9B09r+PPPuBmx2ByGVPTEZdPh4Gjl4MpUR99Zj8W/Hyci28J+nO5KSkceuP5M4cuo8JxMyMGKjXR0zUW2D+Prnvejz0mlWL5A2jaqy90Q6OTaFpJRMcpLicDeoeLrpad+kCtuPpqBmnMNdy8Go2FDcfbDnZOCmWNErdtxUB4pmx66peKm5qIqGTVMxGI3k2xXcHLl4qzlYLjwWxE2xlri9ypJdU7Cjw6qpqAYjNruGzmEhT3XHageTXsWugc6Rj83ghcWm4WZQcNjt2G1WvJVcjAaVk/k+BBhy8dSyin1ZWtFjdSig6kDVg6oreG/0wC8ggHOZGlZNJT3Hhs5gIMjPk1Pn0rHnZuCGlerVAvDz8yE+08H5PIWEtBwsubmENa5Kep6Dw6fSybPYuMd0jEBdyX8PmsGdfIyk5YKqaChu3mTmWAgMqoRN747ZrxLHDhygoSG+yHIOVY/OwxfczORoJtzM3hg9vVBMns5/DoMHa3fHc/hECr6eegZ3roXZ0x1Nb0QxuJHr0PHf2L2YdA4e7N2QjBwbdhR8vdxxoHLiXBapWRZa1A0k16ZRNdCbxPP57P87FXd3Ex2ahqDqCr7MNRT8/M2kn8/hWNx5Dh5Ppne7mhgNemfXTVxSNtUDzSiKQkaOhXW74ujWuipeHkZnvdKz8vF0N6DXqeTkWTEadGzcH0/Tmv5U9i3+xDWHpqFw8bnOl7ZsT57LxNNNX2S5FdtOYrdr9OtYE4DFG/7m3I61tDYn4HDzoU3LeuiqNia4YZO7LxSg4JTUjz/+GKvVytChQ6/7MXQVORR2HkkiduPf9G0fyumELFb+fop+HWuyaX88kx69B7tdY/2eOPp2CEWnqsxfe5TV208zsmd97m1TjbikLKZ9u5vnhzanblUfABb8epRVv592fkZUh1BWbjvFgE61qFrZkzmL9qMAZg8DY+9rweQvC55pXLiFDIqd5tU9yM3KJCM9E2+jneiIRixduxd/NQsvNQ8TNhRFI9CQjc5hwV2x4qHko3j6k56dj7uWj6eaj1lnQdVu/BnCDkWH4u5Nao4Ds5ZNruKBu7c3uVaFhAwrBqMRm8WC0cuPlEwrJp2G3WbFoNipGVoFm8mX3YfOYDLoaNm0Fm5e3pxItbN/3584UOgY1gCjQU9Wng2d0UR8Wj5Z2Tm0qh9EYFAAipuZb1b/SXxSGg/1a0FgYCVOJ+dxOj6NbQfOkJScwZjI+tStUQl0BrItYNNUfH08ybcrJKTlUTXAk7PJ2dQI8iIxLYdftp9hYJdaaDodHnqF85n5/Lz1JNEda+JrLnj8oqZpHD2Tzsff/caYgN3k52QTWLUqAdWqo3pVRvGqjGquhOLph6IzcDoxi0rebni4XVuPr9Vm5z/z92C3a7zyUJsiX0KZORZSM/IJDb7Y7XDoZBofzt9CW58ksuxGnry/Q8GXttEDDO4EBvmQlJRJ8vlcMnKs1A7xJivXitn9Ymvw2XfW0lx/glybjjH3d8ErIAiMHtfcHfzXmXQCfN3wubCNLpWbb0OnKqUeTF2PitpDcCXrdsfx1aojKAp0bVmVUb0bADdWl6uFQrmPKQBER0cTHR1d3sW4oiUbj2OzOxgSUee6ltt2KIG4pGz+u/QgDWr4ArBy2ylsdgdHTp3n8Kk01uw4Q+0QHzzc9Bw4ngoUtAYAVv1+mqxcK7G//Y3JoGNIRB22H06kSS1/qgV4sudoMj9vOYmXksv+P8+ydU8e4ZVT6d9AY9++o+yP3c1Aj1zcFQt1jMn4KVnosEPh7VJ8Lvz/+0qevPAd4UDBphU0dfU+gZxN18h3GEnHB/fMLHQ6lcAaNTF6eqFz9wKTmTSLjhMpVkKC/KhcJYSN++JISc0iolkAe/+M58DxNAZFdaR2jUqcz8xn6cZjpKZmMHJwOIGVvNh14Zd+aNc6PBzdlITEDM4dSaJeTT/Ssy0E+XlgtRc0sz9ffojW9QMIahiIpmn8mL6fFnUrEdCyKgA1cyzM2m4ipLInD3ZqV2R/1CthH40cGnyhSV8wXhFa3ZvQ6oE0bhjKmaQs6tep7Jz30t5bNz2EBhd8KdYIKvhJoJ8HI3vVBy7+wfp7uzGqV4Min6koCvWr+6J5+PFxRjeycq282i4MU5WSx3OqB17fuJlBryNmZGvsdq3Yl7KXh7HIUS+Ar9lItubGuvPVqRnsha5yzRLXW9nX3XlEe2kgAJg93dl2vhaqomAOqYWiXt/YYN1qPqX+zN1UIb6qyo2vV+HBRMG+KkuuvaWvwaINx3Az6tn7VzLZedYiofDLjtPEp+Tw0IXUjk/J5sjp84Q3DXYOVB4/m17Qd2ix8+fpdABsdgcAe/9KZseRRADmrTpC4vlcABTg6JnzLN96kq0Hz2HUqxw8kQZAQlouuZkZRNdPpLo9jp6mo1h8c/FULWChYI86gEMKzU0eGO3ZWE168jFh8QzCvV44GN3ZfzoHNw8zDeuGsOFgKgcOnSQPAy88HoVm8GD/0TTSs/OJ6lATy/lcMnMs7Nlzlm2HEogZ2Rrfywajgy/8K9SrSi3n60oN7AQfT6VOvcooioK/Bzw8pOhga3izKmTlWoloGQKAqijOsYfCsQmTWrBNn+zfxLmcoig8P7RoV6OXh5FebasT5OdxxX1bSFWVYoPGAP7ebvh7u13TOm5UJW83jp0tOADw9yp+hHwzVEVB1V/bF7OP58XP9rvBcvh4mkg6n4eXp6HE7Slu3KW/G763+PfkchIKV7HlQAK+Xkaycq2kZORhtTmcR5S/7ooj+XwuI+6th0GvsnTzCbb+kcDm/ecY0LkWv2w/TUpGPj3DqvPLjtNFTkFTgN/2FfS/ensYSDyfi5tRh8mgo3FNf7b+Ec+aDfvoWjmHrgHJnDsdR7DuPDYrePvmovtbw+5VGbfqjTgen4dHQFX2HIrD3cOdqMhw9EF1+PtUNnMX7qRKgA8vj76n4MvvwlFjWKuLdexczcHPx7cQGuSFydsfgE7Nqzh/HujrTqCvOyGVPIkOr0lln+J9pldiMuhoVf/Kzyk26FVn/+mtcH/3ktoEFY//hVDQqQpenmV7BHgl7iYdBr2K1ebA3+vGgtDnQvl9Pcv2S8sVXRoEfiV0r91KEgqX+Ozng9QI8qJnWMHZOVabg9SMPPQ6hew8K5oGiedzqVrZk5T0PBJScwA4lZhJnRAf0rMsAPwVl86CtUc5k5QNQNuGgfy27yx5Fjs+ZiPpWRZa1qvM7qPJRLQMwexu4OctJxnS2kwn7zhy9n/PEL8sDIoDbECqJ97BgeSYmnDo7xSqVK9Gg8490FWqAUAbwNfPk/nv/I/IdjUwVC84Zm9c04TR3YOGtSqh15V+SYpBrzLtqQ5crevX3aR3+Wb8rebvXfAH7udlcgZ2eVAUBR9PI8npefh539iXjveFbg2fMu7ecEVe7gb0OgWbXZOWwu20688kEtJynaGQnJ6LBpzPtpBvKRhMXbT+GA1D/XC7ZMDr+NkM6oT4XAgQFZvd4ewKAggNNlM1wJNjcRnc17UuOp1Cg+q+1K3qTfcqGWQdWE3rSsep/GcaFsBUvTkZhsp4VqmGzi8EXXBdFFWPGagdn0FokFex5rlBr/LGmHuKTZv8WDvcr3J6K3DF0BBlp7B7qlIZd1NdC1+zqSAUbrj7qCAMvMuxxXO3UhTlpvfPtZJQuCDPYiM3305cUpbzlLGEtIIv9sJAANh9NJk/jqdSr5oP3p5GVAX+PpuBpmmkZubTpKYfe4+lYLE66NIihKgOoRj0OqpWLgiF0GAvgg1ZWHZ/R+dT+7DsS8Xk7kNQSDX0oX3RVW2Czi+E0nrDa5UyEFkaH/kDrdAKw8D/Bo/Ob6XC35UbHdsoDAP5nSsbvl4m0rMteJRxa11C4YLCrp/cfDspGXlU9nEnMS23xHktNgd/nEgjsl0NEs/nsvVgAonnc7HaHDSo4ceB46nYHRrVA80EXDhTo3FNf06eiMP3yBKyj6wHNPTVmqEPG4S+bnsUnaHEzxJ3t8IwKOsB7WtR2O3jd4NlcY4plHGft6sK8nMnL9923Xd8uF4SChecz8p3vj6TmH0hFHJKnLeStxvns/LpEVYdu8OBoijsOFxwFlGArzsBvu6cS82hSqWC433NZqGV8QSNfVdhP5SKPrQVpg4jUM2Vyr5iokIL9HXHZNQ5T2ktT8H+HrgZdfjd4JiAs9VTxt0brmr4vfWK9FqUFQmFC85faCkAvPfjPnqGVXcOJBd6sn8TqgWaSUzLISvH6uzb69GmmjMU/L1NVKnkcSEUPNEsOeSseBdHwl8o7t549J+ILvD6rnUQdy8PNwMz/xHuvF1JeeraqiptGgQWu+/TtaoeaGbsfS1oUtP/FpdMAHi6GfB0K/seBQmFCwpbCl4eBjJzrKzZWXDFcKNQPw6dLLhGoHaINwG+7lStXPSh17WqeDsHmCt5u1G3mg9nkrLw0eeT8/NMHCmncOv+JPrabVFU2eSiKDdjxfid0OvUmxrEVBSFZrWl9Xunk1NOLkjPsmDQq7z+yD28+UR7jAYd3h5GHuhZ3znP5VdwFjLoVWpX8UKvU/HyMND7nhpM7utFzsJXcKSexr3XcxjqdpBAEEJUePItdcH5rHx8zUbnkdJL97fE3aR3DhTrVAW3K5zaeW9YdWqdTUfLTMKybxXWQ+tQfavgHvn/0FUOvS11EEKIm+XyoWC7cD+d81n5RW7EVafqxfuwuBl1GPXqFUf92zYMpE0VBzlLpqJZctDXbotb54dRjNd39a8QQpQnlw+FJ/6zjhZ1KpGWZSn1pmNmd4Pz1hal0fKzyVk5EzQHHoNfR+dXtQxKK4QQZcvlQwFg77EUdKpC6/qVS/y5l4cRva70VoLlj7Xk71gE1jzc+8VIIAgh7lgSChfYHRqhpZwrPqxrnRLv+qhpGpbdS7HsWISuahNMYYPQBdUt66IKIUSZkVC4RGndRw1D/Uqcbtn7M5Ydi9DXC8ctYgyKWv7nmgshxM2QULjAqFev+f77AI7z8Vh2LC4YUO76KIoiZ/cKIe588k12QbVA8zU/GERz2Mnb8AXojZg6PiiBIIS4a7j8t5nRULAJ6lf3veZlLLt+wn7uT9zCR6F6lP4IQSGEuNO4fChoGnRrVZXBXWpf0/y2s4ex7P4Jfb1wDPU6lnHphBDi9pJQ0MDNpLumh8xodit5Gz5H8QrELfzB21A6IYS4vSQUNA2FaxtLsB74BS0jEbfwB+VKZSHEXcnlQwG46rOJARw56eTv+gldjRboqzcr+0IJIUQ5cPlQ0LRrCwXLzliwWXFrP7zMyySEEOVFQkHT4CrdR5olF+vRTRgahKP6Vrk9BRNCiHIgoQBc7fIE69+/g82CoUGX21ImIYQoLy4dCgWthKvMY7Ng3b8a1bcKqjxGUwhxl3PxUCj4X73CoEL+jkU40uIwtR9+xecpCCHE3cDFQ+FCKpTyXa9Z87Ae/B/6eh3R12hx+womhBDlxLVD4cL/pR3/247vAFs+hkZdb1OJhBCifLl2KBQ2FErpFrL+uQnFOxBdUL3bWCohhCg/Lh4KBalQUiY4MpOxnz2EoV64jCUIIVyGa4fChf9L+tK3Ht0MgKG+3PROCOE6XDsUHBdaCpdP1zSsRzehq9IA1Svg9hdMCCHKyS0PhdjYWDp16sSAAQMYMGAAM2fOBODs2bOMHDmSPn368PTTT5OdnQ1ARkYGTzzxBJGRkYwcOZKkpKRbXaRSldZScCT8hZaegKF+p9tWFiGEqAhueSjs37+fmJgYlixZwpIlSxg7diwAb7zxBg888AArV66kadOmfPjhhwDMmjWLsLAwVqxYwbBhw5g6deqtLlKpSrt4zfrnJtAb0dcKu21lEUKIiqBMQiE2Npb+/fvz0ksvkZ6ejtVqZfv27fTu3RuAwYMHs3LlSgDWrVtHdHQ0AP369WPDhg1YrdZbXawSXbx47ZJpDjvW49vR12wjt8cWQrgc/a1eYUBAAE888QTNmzfn3XffZdKkSYwfPx6z2Yxer3fOk5CQAEBiYiIBAQX99nq9HrPZTGpqKkFBQdf8mZUqmW+orFk5FgDMXm4EBHgBkHvyAFn52fi3CMd8YdqdIuAOK++V3C11uVvqAVKXiupW1+WGQ2HFihW89dZbRabVrl2bL7/80vn+scceo0ePHvzrX/8qtvyVTvNU1etrwKSkZOFwXP0+Rpdz8zQBkJ2VT1JSJgB5ezaBqifHuy65F6bdCQICvJx1uNPdLXW5W+oBUpeK6kbqoqrKFQ+kbzgUIiMjiYyMLDItMzOTL7/8ktGjRwMFffZ6vR5/f3+ysrKw2+3odDqSkpIIDAwEIDAwkOTkZIKDg7HZbGRlZeHr63ujxbouFy9eK3yvYTu5G13VxigGt9tSBiGEqEhu6ZiCh4cH//3vf9m7dy8AX3/9NT179sRgMBAWFsby5cuBgjOUunQpuA11REQEsbGxACxfvpywsDAMBsOtLFapLl68VpAKjrQzaJlJ6ENb3ZbPF0KIiuaWjinodDpmzZrF66+/Tl5eHjVr1mT69OkA/Pvf/yYmJoa5c+dSpUoV3n33XQBeeOEFYmJiiIqKwsvLixkzZtzKIl3R5S0F24ndAOhDW962MgghREVyyweaw8LCWLx4cbHpVatWZd68ecWm+/r68tFHH93qYlwTZ0vhwnvbyT2oAbVRPf3KpTxCCFHeXPaK5rzcHKxWG1DQfaRZ83AkH0dfrUk5l0wIIcrPLW8p3CnOfTWBtCrtgGBQwJ74N2gaumC5I6oQwnW5bEvBoFnQZRZcK6EA9oSjgIJOHrkphHBhLhsKOZoJgyMXKOg+sp87iupXFcXkWc4lE0KI8uOyoZCLCYM9DwDFYcee8Jd0HQkhXJ7rhoJmwugoCAVz9hmw5qGTQWYhhItz3VDAiPFC95FX+lFQFPQhjcq5VEIIUb5cOBRMGLWCloJX+lHUgNoyniCEcHkuGwp5mgmDZsWs5OKedQZ9tablXSQhhCh3LhsKuRTc8K6F8RQKGjoJBSGEcN1QyFOMALQynsCuM6ELrFXOJRJCiPLnsqGQf6GlUM+QQLZPHRTVZS/uFkIIJ5cNhTxMzte5vrXLsSRCCFFxuHAoGJ2vc71Dy7EkQghRcbhuKCjuztcWc0g5lkQIISoOlw2FfOViSwHd7XnSmxBCVHQuGwooOgCOWoOcj+MUQghX57Kn3CjAHPURjmdaeKa8CyOEEBWEy7YUFEUhGzfs6KSlIIQQF7hwKIDdoV14U75lEUKIisJlQ0FVFBwXQkEyQQghCrhsKKBwMRSk+0gIIQAXDgUFBYdWGArlXBghhKggXDYU1CIthXIujBBCVBAuGwqKojgHmhUZVRBCCMCFQwFpKQghRDEuGwqqwsUxhXIuixBCVBQuGwrKJaekSlNBCCEKuHAoXOw+UiUThBACcOVQuOSUVCGEEAVcNxQU0Jy9R9JUEEIIcOlQUC55XY4FEUKICsRlQ+HScQRpKQghRIGbDoXZs2czZ84c5/uMjAyeeOIJIiMjGTlyJElJSQBYLBbGjRtHZGQkgwYN4tixYwBomsa0adPo06cPffv2ZefOnTdbpOsmkSCEEAVuOBQyMzOZOHEin3/+eZHps2bNIiwsjBUrVjBs2DCmTp0KwLx583B3d2fFihVMnDiRmJgYAFatWsWxY8dYvnw5H3zwATExMdhstpuo0rUp2n0ksSCEEHATobB27Vpq1qzJI488UmT6unXriI6OBqBfv35s2LABq9XKunXr6N+/PwBt27YlLS2Ns2fPsn79evr27YuqqtSqVYuQkBB27959E1W6NkW7j8r844QQ4o5ww6EwcOBAnnjiCXQ6XZHpiYmJBAQEAKDX6zGbzaSmphaZDhAQEMC5c+dITEwkMDCw2PSyJq0DIYQo7qrPaF6xYgVvvfVWkWm1a9fmyy+/vOYPUdWSs0dVVbQSrhUobf7SVKpkvq75AYymi1X39/ckIMDrutdR0dwNdSh0t9TlbqkHSF0qqltdl6uGQmRkJJGRkde8wsDAQJKTkwkODsZms5GVlYWvry+BgYEkJSURGhoKQFJSEoGBgQQFBTkHoy+dfj1SUrIu3rLiGtmsdufr82k5JJl0V5i74gsI8CIpKbO8i3FL3C11uVvqAVKXiupG6qKqyhUPpG/5KakRERHExsYCsHz5csLCwjAYDERERLBkyRIAduzYgclkIiQkhC5durB06VLsdjsnT57kxIkTNGvW7FYXq5hLO4+kJ0kIIQpctaVwvV544QViYmKIiorCy8uLGTNmADBq1Chee+01oqKiMBqNTJ8+HYA+ffqwb98+5yD01KlTcXNzu9XFKkbOPhJCiOIUraRO/TvMjXQfzflxH7uPJgMw6dF7qBZw/eMSFYmrN4krorulHiB1qajuiO6jO0WRlkI5lkMIISoSFw6F0t4IIYTrcuFQuBgE8jwFIYQo4LqhUN4FEEKICsh1Q+GSVFCl+0gIIQAXDoUiQSCZIIQQgAuHgiLPUxBCiGJcNhQubR5IJAghRAGXDQW5dbYQQhTnsqFQ9OI1SQUhhACXDoWSXwshhCuTUEAGmoUQopALh4IEgRBCXE5CAbnNhRBCFHLhUCjtjRBCuC7XDYVLX0smCCEE4MqhIM9TEEKIYlw4FC59LbEghBDgwqGgFnlGczkWRAghKhCXDYVLSSYIIUQBlw2Foi0FiQUhhAAXDgXJASGEKM6FQ+HSi9ckIYQQAlw6FC59U27FEEKICkVCAbnNhRBCFHLdUECaCkIIcTnXDQV5noIQQhTjwqEgF68JIcTlXDgULnkt3UdCCAG4dChIS0EIIS7nsqGgyg3xhBCiGJcNhUISB0IIcZHLhoLzKmZJBSGEcHLZUCjsMpJBZiGEuEh/syuYPXs2qqry3HPPAbB9+3b+8Y9/EBwcDEDjxo156623yMjI4KWXXuL06dP4+/sza9YsAgICsFgsvPzyyxw4cAA3NzdmzJhBnTp1brZYV+VsKEgmCCGE0w23FDIzM5k4cSKff/55ken79+9nzJgxLFmyhCVLlvDWW28BMGvWLMLCwlixYgXDhg1j6tSpAMybNw93d3dWrFjBxIkTiYmJuYnqXLvCLJBQEEKIi244FNauXUvNmjV55JFHikzfv38/mzZtYuDAgTz11FPEx8cDsG7dOqKjowHo168fGzZswGq1sm7dOvr37w9A27ZtSUtL4+zZszdarGvm7D6SVBBCCKcb7j4aOHAgAHPmzCky3cvLi6ioKHr06MF3333H2LFjmT9/PomJiQQEBBR8qF6P2WwmNTW1yHSAgIAAzp07R0hIyDWXpVIl83WX38vLDSgIhYAAr+teviK6W+oBd09d7pZ6gNSlorrVdblqKKxYscLZBVSodu3afPnllyXOP2nSJOfrESNG8M4775CZmVnivKpackOltOmlSUnJwuHQrmuZ7Ox8oKAbKSmp5PLdSQICvO6KesDdU5e7pR4gdamobqQuqqpc8UD6qqEQGRlJZGTkNX2Yw+Hg448/5oknnkCn0138EL2ewMBAkpOTCQ4OxmazkZWVha+vL4GBgSQlJREaGgpAUlISgYGB1/R5N8N58Zr0HgkhhNMtPSVVVVV++eUXVq1aBUBsbCwtWrTA3d2diIgIYmNjAVi+fDlhYWEYDAYiIiJYsmQJADt27MBkMl1X19ENuzCWIM9SEEKIi276lNTLTZs2jVdffZUPPvgAf39/pk+fDsALL7xATEwMUVFReHl5MWPGDABGjRrFa6+9RlRUFEaj0Tl/Wbs4viypIIQQhRRN066vM74CupExhQ17z/LlisN4mPS8P7ZLGZXs9nH1ftKK6G6pB0hdKqqyGFNw3SuaC/+XhoIQQji5bCjgvKJZUkEIIQq5bCiozovXyrkgQghRgbhsKMhNUoUQojgXDgW5zYUQQlzOdUOh2AshhBCuGwrO5ykIIYQo5MKhUPi/xIIQQhRy2VCQs4+EEKI4lw2FQvI4TiGEuMhlQ0GRloIQQhTjsqGgOscUyrccQghRkbhsKFw8+0hSQQghCrlsKCAtBSGEKMZlQ0GV26QKIUQxLhsKcvGaEEIU58KhUPR/IYQQrhwKyA3xhBDicq4bCtJSEEKIYlw4FGRMQQghLueyoaDKDfGEEKIYlw2FQhIJQghxkcuGgiLP4xRCiGJcNhRUeRynEEIU47KhIA0FIYQozmVDARloFkKIYlw2FOTJa0IIUZzLhoJcvCaEEMW5biggz1MQQojLuW4oSEtBCCGKceFQkNtcCCHE5Vw4FC5/IYQQwoVDoSAMVMkEIYRwuuFQ2LlzJ0OGDGHAgAE8/PDDxMXFAZCRkcETTzxBZGQkI0eOJCkpCQCLxcK4ceOIjIxk0KBBHDt2DABN05g2bRp9+vShb9++7Ny58xZU6+okC4QQorgbDoVx48YxdepUlixZQnR0NFOmTAFg1qxZhIWFsWLFCoYNG8bUqVMBmDdvHu7u7qxYsYKJEycSExMDwKpVqzh27BjLly/ngw8+ICYmBpvNdguqdmWKXLwmhBDF3FAoWCwWXnjhBRo2bAhAgwYNiI+PB2DdunVER0cD0K9fPzZs2IDVamXdunX0798fgLZt25KWlsbZs2dZv349ffv2RVVVatWqRUhICLt3774VdbsiuXhNCCGK09/IQkajkQEDBgDgcDh4//336dGjBwCJiYkEBAQUrFyvx2w2k5qaWmQ6QEBAAOfOnSMxMZHAwMBi08uatBSEKHt2u420tCRsNkt5F8UpMVHF4XCUdzFuiavVRa834ucXgE537V/1V51zxYoVvPXWW0Wm1a5dmy+//BKLxeLs7nnyySdLXYeqltwgUVUVTdOuef7SVKpkvq75AewXPsNk1BMQ4HXdy1dEd0s94O6py91SD7ixuvz99994enpiNofIAdhtpmkamZnp5OSkUbt27Wte7qqhEBkZSWRkZLHp2dnZPP300/j6+jJ37lwMBgMAgYGBJCcnExwcjM1mIysrC19fXwIDA0lKSiI0NBSApKQkAgMDCQoKcg5GXzr9eqSkZOFwFA+XK0k7nwuA1WonKSnzupatiAICvO6KesDdU5e7pR5w43XJzs4hKKgydrsGXN/faFnR61VstrujpXC1uri7e5GQkFZk36mqcsUD6ZsaaA4NDWX27NkYjUbn9IiICGJjYwFYvnw5YWFhGAwGIiIiWLJkCQA7duzAZDIREhJCly5dWLp0KXa7nZMnT3LixAmaNWt2o8W6ZoqMKQhxW0gLofzcyLa/oTGFgwcPsnbtWurWrcvAgQOBghbCp59+ygsvvEBMTAxRUVF4eXkxY8YMAEaNGsVrr71GVFQURqOR6dOnA9CnTx/27dvnHISeOnUqbm5uN1Ks6yK/p0KI22no0GjmzPmYKlVCyrsoV3RDodC4cWOOHDlS4s98fX356KOPik03mUxMmzat2HRFURg/fjzjx4+/kaLcsIsXr0k6CCFEoRsKhbuBZIEQrmfXrh189NEcbDY7VapUwd3dg7//PobD4WDkyIfo3r0nAwb04fvvY/Hw8OTpp8cQHt6FBx8czZo1q9izZzdPP/0P3nprMklJiSQnJ9GyZSteeWUSu3fvZO7c97DbHdSuXYfnn/8nkya9SmJiAjVr1sZiKTgD66+/jjJ9+lTsdjtGo5GJE/9N9eo1ynnLXOTCoSBjCkLcTpv2x7NxX3yZrLtT8yqEN6tyTfOeOnWShQuXMW/eF1SuHMArr7xBdnYWTz01hsaNm9KmTRi7d++iVas2xMfHs2fPLh58cDRbt27m3nt7snnzRurVq8+UKdOwWq08+OAwjhw5DMDp06dYuHAZZrOZd9+dRv36DZkx4z327NnFr7/+AsD333/L8OEP0r17D9auXc0ff+yXUKgI5DoFIVxTjRo1MZvN7NjxO/n5efz8808A5OXlcfz433To0ImdO39HVRV69Ypk7drV2Gw29u7dw7hxEzGZTBw8eIDvv/+WEyeOk56eTm5uDgDVq4diNhec2bN7905ef/1NAFq2bE1ISFUAOnQI5913p7Nt22Y6duxM1673lsNWKJ3rhkLh/5IJQtwW4c2u/Wi+LJlMJgAcDjuvvjqZBg0K7syQmpqCt7cPmZmZzJ//DTqdnjZt2nLq1AmWLYuldu3amEwmFi6cz7p1v9K//yCGDr2H48ePOa+3Klw3FBxwXnphmU6nA6Bbtx40bdqcTZt+44cfvmPr1k2MH//K7ar+Vbn8XVIlE4RwTa1btyU2diEAycnJPPzwCBISzuHn54fJZGLTpg00b96S1q3b8uWXn9GxY2cAtm/fRv/+g+nVKxJQOHr0zxKvKg4Lu4fVq1cAcOjQH8TFnQHgtdcmcPDgHwwcOITHHnvK2fVUUbhsKKjSVBDCpY0Z8zj5+fmMGnUfL7zwFM888zxVq1YDCrp4zGYvPDw8aNOmLcnJSXTs2AmA++57gC+++IQxY0by7rvTaNq0OfHxZ4ut/9FHnyQu7gwPPngfX3/9pbP7aNSoR5g37wvGjBnJBx/M4rnnxt6+Sl8DRSvpPhN3mBu5ojk338azMzdwT6NAnhrQtIxKdvvI1bMVz91SD7jxupw7d5Lg4NAyKNGNc6UrmqH4PiizK5qFEELcfVw2FFS5eE0IIYpx2VC4+Izmci2GEEJUKC4cCoVnH0kqCCFEIRcOhaL/CyGEkFCQUBBCiEu4cChI95EQQlzOdUOh2AshhKj4PvvsYz777OMyW7/rhoLzlNRyLogQQlQgLntDPCgMBEkFIVzBrl07+OqrzwGIiztD16734unpyW+/rUfTNGbMmI2/fyV+/HEBK1cuJy8vF1VVeeONt6hZs1aRdQ0dGk3jxk05evQIH374X7Zu3cwPP3yHw6HRoEFD/vnP8Xz44Wxq1qzNoEFD+emnxSxY8A3ffLMQm83GffcN4Pvvl7BkyY8lftbl61+xYhk//bQYHx9fvLy8aNSoCTabjSlTJnHs2F8ADBo0jP79B930dnLpUFAURVoKQtwm1j83YT2yoUzWbWjQBUP98KvOd/DgH3z33Q94enoTHd2TZ599kc8+m8ebb77BmjWriYqKZsOG9bz//seYTG78978fsXjxD4wd+69i62rfviOTJr3F338fY+nSWObO/RyTycRHH73Pd9/No0OHTixbFsugQUPZufN3MjIySE1N4cSJ4zRp0oz8/Lwrflbh+g8fPsjPP//E559/g6IoPPXUIzRq1IT9+/eSkZHBF198S3r6ed5/f5aEws1SFEVOPxLChdSuXYegoGBsNgc+Pr6Ehd0DQFBQMJmZGXh6mnn99SmsWbOa06dPsW3bZurVa1Diuho3Lrhn2u7dOzhz5jRPPvkIADablfr1GzJixCjnE9ZOnjzJvff2Ys+e3Rw+/AcdO3a66mcVrn/Xrp20bx+Oh4cHUHDrbbvdTu3adTh16gT//Oc/aN8+nKeffu6WbCMXDwXJBCFuF0P98Gs6mi9Len3Rr7zCZxwUSkg4x3PPPcmQIffRvn1H/P0rcfRoyc+jL3x2gt3uoHv3Hrz44jgAcnJysNvtmEwm6tatz+rVKwgNDaVVqzbs3Pk7+/bt5YEHHr7qZxWuX1EUNK3ocxnsdjs+Pr58++1CtmzZwpYtmxgz5kHmzfseLy+vm9pGLjvQDAUbWzJBCFHo8OGDVKtWnfvvH0njxk3ZunUzDof9isu0atWGDRvWkZaWiqZpvPPOW3z//bcAdOwYzpdf/pdWrdrQqlUbNm7cgLu7G76+vtf8WWFhbdm8eSNZWVnk5+ezYcP/ANi4cT2vv/4KHTt24sUXX8Ld3Z3ExISb3gYu3VJQFblOQQhxUdu27Vm8eCEPPjgMg8FA48ZN+fvvY1dcpl69+jzyyOM8//xTaJpGvXoNePDB0QB06NCJGTPeplWrMLy9vfH19aNDh07X9Vn16jVg2LARPPbYQ3h5eREUVPD0uvbtw1m//ldGjboPo9FIRER36tSpe9PbwGWfpwDw7MwNhDcN5oGe9cugVLeX3Lu/4rlb6gHyPIWKSp6ncIv5e7vh62W6+oxCCOEiXLr7aNY/Izifll3exRBCiArDpVsKbkY9OtWlN4EQQhQh34hCiDJ1Fwxb3rFuZNtLKAghyoxebyQ7O0OCoRxomkZ2dgZ6vfG6lnPpMQUhRNny8wsgLS2JrKzz5V0UJ1VVcTjujrOPrlYXvd6In1/Ada1TQkEIUWZ0Oj2VK1cp72IUIacKX5l0HwkhhHCSUBBCCOF0V3QfqTdx/+ubWbaikbpUPHdLPUDqUlFdb12uNv9dcZsLIYQQt4Z0HwkhhHCSUBBCCOEkoSCEEMJJQkEIIYSThIIQQggnCQUhhBBOEgpCCCGcJBSEEEI4SSgIIYRwcslQWLp0KX379qVnz55888035V2c6/bQQw8RFRXFgAEDGDBgAHv37r3j6pSVlUW/fv04c+YMAJs3byY6OppevXoxc+ZM53yHDh1iyJAh9O7dm5dffhmbzVZeRS7V5XWZMGECvXr1cu6fX375BSi9jhXF+++/T1RUFFFRUUyfPh24M/dLSfW4U/fJ7Nmz6du3L1FRUXzxxRfAbdgnmos5d+6c1q1bNy0tLU3Lzs7WoqOjtaNHj5Z3sa6Zw+HQwsPDNavV6px2p9Vpz549Wr9+/bQmTZpop0+f1nJzc7WIiAjt1KlTmtVq1caMGaOtW7dO0zRNi4qK0nbv3q1pmqZNmDBB++abb8qx5MVdXhdN07R+/fppCQkJRea7Uh0rgk2bNmn333+/lp+fr1ksFu2hhx7Sli5desftl5LqsXr16jtyn2zbtk0bPny4ZrVatdzcXK1bt27aoUOHynyfuFxLYfPmzbRv3x5fX188PDzo3bs3K1euLO9iXbO///4bRVF4/PHH6d+/P19//fUdV6fvv/+ef//73wQGBgKwb98+QkNDqV69Onq9nujoaFauXElcXBx5eXm0bNkSgMGDB1e4el1el5ycHM6ePcurr75KdHQ07733Hg6Ho9Q6VhQBAQHExMRgNBoxGAzUqVOHEydO3HH7paR6nD179o7cJ/fccw9fffUVer2elJQU7HY7GRkZZb5P7oq7pF6PxMREAgIuPokoMDCQffv2lWOJrk9GRgYdOnTg9ddfJy8vj4ceeojIyMg7qk5Tp04t8r6kfZKQkFBsekBAAAkJCbetnNfi8rqkpKTQvn17Jk2ahIeHB08++SQLFy7Ew8OjxDpWFPXq1XO+PnHiBMuXL2fUqFF33H4pqR7ffvstv//++x23TwAMBgPvvfcen3/+OX369Lktfysu11LQSrgprKLcObfRbdWqFdOnT8fDwwN/f3+GDh3Ke++9V2y+O6lOpe2TO3FfVa9enQ8++IBKlSrh7u7OqFGjWL9+/R1Tl6NHjzJmzBjGjx9PjRo1iv38Ttkvl9ajdu3ad/Q+ef7559myZQvx8fGcOHGi2M9v9T5xuVAICgoiOTnZ+T4xMdHZ9L8T7Nixgy1btjjfa5pG1apV7+g6lbZPLp+elJRU4et15MgRVq1a5XyvaRp6vf6O+L3buXMno0eP5v/9v//HoEGD7tj9cnk97tR9cuzYMQ4dOgSAu7s7vXr1Ytu2bWW+T1wuFDp27MiWLVtITU0lNzeX1atX06VLl/Iu1jXLzMxk+vTp5Ofnk5WVxeLFi/nPf/5zR9epRYsWHD9+nJMnT2K321m2bBldunShatWqmEwmdu7cCUBsbGyFr5emabz55pukp6djtVpZsGABPXv2LLWOFUV8fDzPPvssM2bMICoqCrgz90tJ9bhT98mZM2d45ZVXsFgsWCwW1q5dy/Dhw8t8n7jcmEJQUBBjx47loYcewmq1MnToUJo3b17exbpm3bp1Y+/evQwcOBCHw8EDDzxAmzZt7ug6mUwm3n77bZ577jny8/OJiIigT58+AMyYMYNXXnmF7OxsGjduzEMPPVTOpb2yhg0b8sQTTzBixAhsNhu9evWiX79+AKXWsSL47LPPyM/P5+2333ZOGz58+B23X0qrx524TyIiIpx/6zqdjl69ehEVFYW/v3+Z7hN58poQQggnl+s+EkIIUToJBSGEEE4SCkIIIZwkFIQQQjhJKAghhHCSUBDiFnj88cf566+/rmuZJ598kkWLFpVRiYS4MS53nYIQZeHTTz8t7yIIcUtIKAiX9uuvvzJ37lysVitubm6MHz+ejRs3cvToUZKTk0lJSaFhw4ZMnToVs9nMt99+y/z58zEYDJhMJiZNmkTdunXp3r07s2fPplmzZixYsIB58+ahqiqVK1fm1VdfpVatWiQkJBATE0NiYiIhISGkpKQ4y3Hs2DGmTp3K+fPnsdvtjBo1iqFDh5Kdnc2ECRM4efIkqqrSpEkTJk2ahKpKI1+UkRu/27cQd7bjx49r/fr101JTUzVN07Q///xTCw8P195++22tS5cuWlJSkma327V//vOf2ttvv63ZbDatSZMmzvvyL168WJs/f76maZrWrVs3bd++fdrmzZu1Hj16aCkpKZqmadqPP/6oRUZGag6HQ3vmmWe0mTNnapqmaSdOnNBatmyp/fjjj5rVatX69u2rHThwQNM0TcvIyNAiIyO13bt3a4sXL9bGjBmjaZqm2Ww27eWXX9ZOnDhxOzeTcDHSUhAua9OmTSQmJjJ69GjnNEVROHXqFH369KFy5coADB06lDfffJPx48fTp08fhg8fTteuXQkPDyc6OrrIOn/77Tf69u2Lv78/UHBf+6lTp3LmzBk2b97M+PHjAQgNDaVdu3ZAwS2eT506xcSJE53rycvL4+DBg3Tu3JmZM2cyatQoOnbsyMMPP0xoaGhZbhbh4iQUhMtyOBx06NCBWbNmOafFx8ezYMECLBZLkfkKu2tmzJjBn3/+yebNm/n0009ZuHAhc+fOdc6rlXDXGE3TsNlsxW5xrNcX/PnZ7Xa8vb1ZsmSJ82fJycl4eXlhMpn45Zdf2LZtG1u3buWRRx7hlVdeqVD36BF3F+mYFC6rffv2bNq0iWPHjgGwfv16+vfvT35+PmvXriUzMxOHw8H3339Pt27dSE1NJSIiAl9fX0aPHs2LL77IkSNHiqyzU6dOLF++nNTUVAB+/PFHfH19CQ0NpXPnzixYsACAs2fPsm3bNgBq1aqFyWRyhkJ8fDz9+vXjwIEDfPvtt0yYMIFOnToxbtw4OnXqxNGjR2/XJhIuSG6IJ1zaihUr+Oijj5z32J84cSJbtmxh69at2O120tLSaNu2La+88gpubm7Mnz+fr776Cjc3N3Q6HWPHjqVjx45FBpq/+eYb5s+fj8PhwN/fn9dee4169eqRmprKhAkTOHXqFMHBwdhsNgYNGsTgwYM5fPiwc6DZZrPx0EMPMWLECHJycpg4cSJHjhzB3d2dkJAQpk6dio+PT3lvOnGXklAQ4jJz5swhLS2N1157rbyLIsRtJ91HQgghnKSlIIQQwklaCkIIIZwkFIQQQjhJKAghhHCSUBBCCOEkoSCEEMJJQkEIIYTT/wfyEWgyRb2AewAAAABJRU5ErkJggg==\n" + }, + "metadata": {} + }, + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Episode:10/30, reward:-13.0\nEpisode:20/30, reward:-13.0\nEpisode:30/30, reward:-13.0\nresults saved!\n" + ] + }, + { + "output_type": "display_data", + "data": { + "text/plain": "
", + "image/svg+xml": "\n\n\n\n \n \n \n \n 2021-04-29T17:04:55.053953\n image/svg+xml\n \n \n Matplotlib v3.3.4, https://matplotlib.org/\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEXCAYAAACtTzM+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA0B0lEQVR4nO3deWBN577/8fdOtjkJQii3DtIaWzVUFTEFRUKIIRVtja1W9WjprR9SemoeTqokVa1ekWuooWqIIaXioBFRU6U9WsUpikgiURJk2nv9/hD7isSQQSO7n9df9trredbzXSvy2etZK2ubDMMwEBGRvzyHoh6AiIg8GhQIIiICKBBERCSLAkFERAAFgoiIZFEgiIgIoECwG8HBwUyZMuVP2dbw4cM5efLkn7KtorZw4UI6dOjAhAkTcrx348YNPvroI7y9vfHx8cHHx4ePP/6YGzdu2NapV68eSUlJD32cERERTJs2rdD6u1fdeXHq1ClGjRqFj48PPXv25JVXXuHgwYMAnDt3jqZNmwKwcuVKFi1aBEBkZCSenp707duXI0eO0LlzZ3r37s0zzzzDjh07bH1HRkZSr149Vq1aZVsWExODh4cH97qbvmPHjvz444/s37+fHj165Hj/xx9/5O233y5Q3Q9q7dq1eHl50aVLF/7xj3+QkZHxp2z3rgyxC0FBQcbkyZOLehh2p2PHjsaBAwdyLM/IyDD8/f2NqVOnGjdu3DAMwzCuX79uTJ061fD39zcyMjIMwzCMunXrGomJiX/qmAvD3erOi1OnThkeHh7Gnj17bMuioqKMZ5991vj111+N33//3WjSpEmOduPHjzcWLFhgGIZhBAcHGwEBAYZhGMaMGTOMadOm2dabMmWK8cYbbxjDhw+3Lfv888+N8ePH33Ncnp6eRkxMjBEdHW107969QDUWxPHjx4127doZiYmJhsViMcaMGWMsWrSoyMZjGIZhLto4erRYrVZmzJjB0aNHuXbtGoZhMG3aNOrWrUv79u3Ztm0bbm5uALz44ou89dZbtGrVisDAQA4cOIDFYqFhw4ZMnDgRJycnOnbsyDPPPMPx48d59913MZvNfP7556Snp5OUlISvry+jR48GYNGiRaxdu5Zy5crRvHlzIiIi2LlzJ+np6Xft/27i4uKYMmUKsbGxZGRk0L17d0aMGAHAZ599xo4dO0hLS+PGjRuMGzeOF154geDgYH744Qfi4+OpV68eNWvW5Pz58yQkJHD+/HlcXV35+OOPqVq1Kh07dmT+/Plcv36djz/+mBo1anDixAnS09P54IMPaNmyJUlJSUyYMIGzZ89SoUIF3NzcqFOnDqNGjco21mvXrjFt2jQOHz6Mo6MjnTt3ZsyYMUyYMIE6derw6quvAjB+/Hjb69v366hRo1i4cCGbNm0C4OrVq3Tq1IkdO3aQmpp61/1wu4sXL/Lhhx9y/vx5DMPA19eX1157jdGjRxMXF8f777/PO++8g7e3t63Nt99+y40bNwgICMDB4eaJdpkyZXj//ffx9fVl+/bt2dYH+Oqrr1i5ciVWq5UKFSowadIknnjiCX777TemTJnC9evXiY+Pp379+sybN49SpUrx9NNP06lTJ3755RcCAwN56aWXeP3119m7dy/x8fEMGjSIIUOGsG7dOrZt28bnn3/OwIEDadKkCYcPHyY2NpZnn32W2bNn4+DgwLp161i0aBGlS5emZcuWLF26lGPHjmUb5511N2vWLNf9c+7cOV5++WWeeOIJzp8/z7Jly6hSpYqtny+++IK+ffvStm1b27JWrVrx0UcfUbp06WzbDA4O5vLly1SvXp2IiAhKlSrFhg0buHbtGhaLhdTUVPr06cOcOXNsbf71r3+xePFiXnzxRa5fv07ZsmXZt28f/fv359KlS3zwwQckJiaSkJDAf/3XfzFv3jwqVaqU6/+ZgwcPMnbsWD766CMyMjKYOnUqmzdvZvz48Tg5OXH8+HEuXryIu7s7c+fOpVy5cuzevZvAwEAcHBxo0KABUVFRfPnllzz++OPZ+vb392fIkCF069YNgMDAQAzDsP2OcHV1BaB///5MmzaN4cOH5zrGP4OmjG5z9OhR4uPjWb16NVu3bqV379588cUXODs788ILLxAWFgbcPA1OSEigbdu2LFq0CEdHR9atW0dYWBhVqlQhMDDQ1medOnUIDw+nc+fOhISEMGvWLNatW8fq1atZtGgRSUlJfPfdd6xbt461a9eybt06rl27Zmt/v/5zM3bsWPr27WvrMyoqiq1bt3L+/HmioqJYvnw5mzZtYsyYMQQFBdnanT9/nvXr19v6P3jwIPPnz+ebb77BxcWF1atX59hWTEwMw4YNY8OGDfTr149PPvkEgGnTpvHkk08SHh7O/PnzOXz4cK5jDQoKIi0tja1bt7JhwwYOHz7M999/f99jdWu/enl5ce3aNX788UcANm/eTPv27Slfvvxd98Od3nvvPZ5//nk2bdrEypUrCQsLY8uWLcybN8+2v+/85X7o0CGeffZZWxjcYjKZaN26dY56v//+ezZs2MCKFSvYsGEDr732mi0c16xZg6+vL6tXr2b79u2cO3eOXbt2AZCRkYGnpyfbtm2jUaNGpKenU7FiRVatWkVQUBAfffQRaWlpOWo6e/Ysy5YtIywsjOjoaL7//ntOnjxJYGAgoaGhbNiwAScnJywWS462d9Z9t/0DN8N05MiRbNu2LVsYAPz00080a9YsR//t27enRo0aOZYDvPbaa3Ts2JEhQ4awfft2/P398fb25qOPPuK5557j7Nmz/PHHHxw/fpzy5ctTu3ZtnnnmGfbu3Ut6ejoxMTG0adOGLVu20KRJE1avXk1ERASlS5dm48aNuW4zOjqaCRMmsHDhwlzH+9NPP7F48WK2bt1KfHw833zzDZcvX+b//b//xz//+U82btzI888/T1xcXK79+/n5sX79egAsFgthYWH4+fkRGxtLtWrVbOs99thjd+3jz6IzhNs0bdqU8uXLs2rVKn7//Xf2799PuXLlgJsHdfLkybz66qt8/fXX9OnTBwcHB3bt2kVycjJRUVHAzf/At38Kad68OXDzF8Vnn33Grl272Lx5M6dOncIwDG7cuMHu3bvp1q0bLi4uALz88stER0cD3Lf/O12/fp0DBw5w5coV5s+fb1v2yy+/4O3tzezZs9m0aRNnzpyxnQnd0qRJE8zm//uRaNGihe1MpGHDhly5ciXH9qpXr06DBg1s69z6wd+9e7ft31WqVLF9OrpTVFQUEyZMwNHREUdHR5YvXw5ga3s3t+/Xfv36sX79eho1asS6desYO3bsfffD7fvr8OHDhISEAODs7EyfPn3Ys2cP3bt3v+cY7sVqtWZ7vWvXLs6cOYO/v79t2ZUrV/jjjz8YO3Yse/fu5YsvvuD06dPEx8dz/fr1HLXe0qlTJwCeeuop0tPTs617i6enJw4ODjg5OVGzZk2uXLnCL7/8goeHB4899hgAr7zyCsHBwfes4177p3HjxpjNZpo0aZJrW5PJlGM/FETJkiV5/vnnOXjwICdPnqRDhw62WiMjI3FxceGpp57CycmJwYMHc/DgQZYsWcLp06c5ceIEjRs3ztHnxYsXGTFiBAMGDKB+/fq5brdt27aULFkSgLp163LlyhUOHjzIE088YWvTu3fvu17D8fLyYs6cOSQkJHDs2DFq1qxJrVq1cr3OcecHjD+bAuE2u3btYvr06QwdOpROnTrh7u5uOyto3rw5mZmZxMTEsHnzZtuFLKvVSkBAAO3btwduToHc/omtbNmywM3/WL1796Zz5840b96cvn37smPHDgzDwGw2Z/vhcHR0tP37fv3fyWq1YhgGq1atokyZMgAkJSVRqlQp/v3vfzNy5EiGDBmCh4cHzz33HJMnT84x1ltuP603mUy5/gDfbZ07a7rbD7rZbMZkMtlex8bGUrp06Rzbu/Ni2+1j7du3L76+vvj5+ZGcnMzzzz9PSkrKXfdDbvvrzmWZmZm5jveWZs2a8fnnn2O1WnFwcCAlJQWTyUS5cuXYv38/Q4YMydFnr169GDt2rO11fHw85cuXZ8yYMVgsFry8vOjQoQOxsbHZxnTncblVw6399qDHxdHR8a4/Z3dzv/1TsmTJbB8ibtekSRN++OEHPD09sy3/5JNP+Nvf/pbrp/H7adu2LQcOHODo0aMEBAQAN884Vq9ejaurqy0k/vnPfxITE0Pfvn15/vnnyczMzHU/OTo6smjRIkaOHImXlxfPPPNMjnUeZF/C//2Mv//++/z000/AzemiAQMG0LVrVzZv3syRI0fw8/MDoFq1asTHx9vax8XF2cK6qGjK6DZ79+7F09OTl156iUaNGrFjx45sp9R+fn5MnTqVevXqUb16dQDatGnDihUrSE9Px2q1MmnSJObOnZuj7zNnzpCSksLo0aPp2LEj33//va1N+/bt2b59O8nJycDNOw9uedD+b3FycqJJkyYsWbIEuDmnPmDAACIiIjhw4ABPP/00Q4cOpUWLFkREROQ6ZVAY2rdvb6vj8uXL7NixI9sv/ltatWrF+vXrsVqtpKen8/bbb3PgwAEqVqxo+0+VlJRkuzMlN1WrVqVx48Z88MEH9OvXD7j3fridk5MTjRs3ZsWKFQAkJyezYcMGWrdufc/6unTpQrly5ZgxYwapqan8/PPP9OnThxEjRuDo6JhjisnDw4MtW7bYfgGsXLmSwYMHAzfvlnnrrbfw9vbGZDJx9OjRh3Jc2rRpw759+2zTEl999dV92+R3/wC8+uqrfPXVV0RGRtqW7dmzh2XLlt310/j9tGvXjr1793LhwgUaNWoEYJt+2rFjh+2DU2RkJIMHD8bX15dKlSoRFRWV6z51c3OjWbNmjBs3jrFjx2a7Q+xemjVrxunTp/nll18A2LZtG1evXsVkMjF9+nQ2btzIxo0bGTBgAHDzmuO6des4cuQIXbt2BW7e7bRz504SExMxDIPVq1fTuXPnfO2XwqIzhNv4+/vz3nvv4ePjg6OjI82bN2f79u22T4G+vr7MnTs32y/kkSNHMnv2bHr37o3FYqFBgwaMHz8+R9/16tWjQ4cOeHl54eLiwt/+9jeefPJJzpw5Q9u2bXnxxRfp378/pUuXpk6dOrZPtQ/a/+0CAwOZOnUqPj4+pKen06NHD3r27MmlS5dsFztLlChBq1atuHLlCikpKYW7I4EJEyYwceJEfHx8qFChAtWrV89xIRHg73//O9OnT6dXr15YLBa8vb3p0qULjRo14r333qNr1648/vjjtGjR4p7b8/Pz45133mHhwoX33Q93CgwMZMqUKaxbt4709HR8fHzo06fPPbdnNptZvHgxn376Kb1797Z92i5btiwJCQl899132T4Zt23bluHDhzNs2DBMJhNOTk588sknmEwmxowZw1tvvUX58uUpU6aMba68sNWuXZsJEybw6quvUrJkSRo0aGD7ObuXu+2f8+fP37NdzZo1+eyzz5g3bx6zZ8/GarXi6urKwoULqVu3LufOnctzDTVq1CAzMxMPD49sHzDatm3L9u3beeKJJwB46623mDNnDp9++imOjo40a9bsnvu0d+/ebNu2jVmzZuUI89xUqFCBuXPnMm7cOBwcHHj66acxm8133Z+33u/atavtDK9+/fq89dZbDB48mIyMDBo3blykF5QB3Xb6KIiJiTH+93//1/Y6JCTEeOedd4puQIVg+fLlxuHDhw3DMIy0tDSjb9++xq5du4p4VH+O+Pj4At+y+TCcPXvWCA4ONiwWi2EYhrFt2zajX79+RTyq4ik5OdmYPXu2cf36dcMwDOOnn34yPDw8DKvVWsQjKxidITwCateuzRdffMGaNWswmUxUq1aNqVOnFvWwCuTJJ59k6tSpWK1WMjIy6Natm+103t65ubnZbk9+lDz22GPEx8fbzoCdnZ2ZMWNGUQ+rWHJycqJEiRL069cPs9mM2Wxm3rx5uU6LFicmw9AX5IiIiC4qi4hIFgWCiIgACgQREcmiQBAREaAY/x3C5cvXsFrzfj28UiUnEhML/777omJv9YD91WRv9YD91WRv9UDOmhwcTFSsWO6ebYptIFitRr4C4VZbe2Jv9YD91WRv9YD91WRv9UDea9KUkYiIAAoEERHJUmynjETk0WQYBpcvJ5CengoUj2mY+HiHQn1Ud9ExUbJkaSpWzN9fyisQRKRQpaRcwWQyUbXq45hMxWMSwmx2IDOz+AeCYVj5449LpKRcoUoVlzy3Lx5HS0SKjRs3UnB2rlBswsCemEwOODtX5MaN/N0xpSMmIoXKarXg6KjJh6Li6GjGas3f92koEESk0BX3p34WZwXZ9woEEZGHpF8/H2JjLxT1MB6YAkFERADdZSQidu7w4YMsXBiExWKlWrVqlClTlv/85xRWq5WXXx5Ex44v4OPzAqtXb6Bs2XK8+eYwPDza8corQ9ixYxs//HCEN9/8OzNnTiUhIZ5LlxJo0qQpEydO4ciRQ7a+3d2f4O2332XKlEnEx8dRq5Y76enpAJw8eYI5c6ZjsVgoWbIkAQH/oEaNvxXxnslJgSAiD83eH2OJjIl9KH23eaYaHo2qPdC6v/9+lrVrN7Ns2RIqV3Zj4sTJXLuWwogRw2jY8GmeffY5jhw5TNOmzxIbG8sPPxzmlVeGEB0dRadOLxAVFUmdOnWZNm02GRkZvPKKH8eP/5KtbycnJ+bOnU3duvUJDAzihx8Os3PntwCsWfMl/v6v0LFjZyIitvPvf/9on4Ewf/58HBwcGDVqFACnTp1i0qRJXLt2jdKlS/Phhx/SoEGDXNtmZmby8ssv079///t+qbmISH7VqFETJycnDh78nrS0VLZsCQMgNTWV3377Dx4ebTh06HscHEx06eJFRMR2MjMzOXr0B8aODaBUqVIcO/YTa9Z8yenTv3HlyhVu3LierW+AI0cO8eGHN7+WtEmTZlSv/l8AtGrlwdy5c9i/P4rWrdvSoUOnItgL95fvQEhOTmbmzJls2bKF1157zbZ84sSJvP7663h6erJv3z7GjRtHWFhYrn0sWLCA06dP53cIIvKI82j04J/iH6ZSpUoBN2+JnTRpKvXq1QcgKSkRF5fy3LhxjRUrluPoaObZZ5/j7NnTbN68AXd3d0qVKsXatavYtWsnPXv2pl+/Fvz22yluffvwrb7h5h0+t//Fs6OjIwCenp15+uln2Lv3O776aiXR0XsZN27in1X+A8v3ReWIiAhq1arF0KFDsy338/OjXbt2ANSrV4/Y2NxPFw8dOsTx48fx9PTM7xBERPKkWbPn2LBhLQCXLl1i8OABxMVdpGLFipQqVYq9e/fwzDNNaNbsOUJDF9O6dVsADhzYT8+efejSxQswceLEr7k+6qJ58xZs3x4OwM8//5vz588B8MEHEzh27N/4+vbltddG2KabHjX5PkPw9fUFIDg4ONvy26d+goKC6Ny5c462KSkpzJo1i4ULFxIYGJiv7Veq5JSvdgBubs75bvsosrd6wP5qsrd64O41xcc7YDY/OjcwOjo6YDKZMJsdeP31N5gzZyaDBvXHYrHw97+/Q82aN+fyPTzasHfvd7i4ONGiRQuCgj6ibdt2mM0ODBjwMnPmzGTVquWUK1eWRo0aExcXy+OP17D1DfDGG28yZco/GDjwRWrWrEX16v+Fo6MDQ4e+yowZU/jf/12M2ezI6NH//VD3kYPDzb7z+nNnMm6d99xFeHg4M2fOzLbM3d2d0NBQ4P8C4dY1BLj5cKs5c+YQHR3N0qVLcXbOPqixY8fSrVs3OnXqxPjx42nRokWeryEkJqbk6/nlbm7OJCQk57ndo8re6gH7q8ne6oF713Tx4hkee6zmnzyigrGXZxndcvHiGRo1ejrbMXJwMN33g/R9zxC8vLzw8vJ64IFkZmYybtw44uLicg2DlJQU9u3bx6+//kpQUBCxsbFER0djNpvp2bPnA29HREQKV6Hfdjp79mxSUlIICQmhZMmSOd53cnIiMjLS9vrWGYLCQESkaBVqICQlJbFixQoef/xx/Pz8bMs3btxIREQEO3fuZPr06YW5SRERKST3vYbwqNI1hJvsrR6wv5rsrR7QNYRHXX6vITw6twKIiEiRUiCIiAigQBARkSwKBBGRR9zixZ+zePHnD307CgQREQH0+GsReYgyft1LxvE9D6XvEvXaUaKuxz3XOXz4IEuXhmAYcOHCOTp06ES5cuX47rvdGIZBYOB8XF0r8dVXq9i6dQupqTdwcHBg8uSZ1KpVO1tf/fr50LDh05w4cZxPP/0foqOj+OqrlVitBvXq1efdd8fx6afzqVXLnd69+xEWtp7Vq1ewYsVaMjMzefHFXqxZs5GNG7/mm2+25tjWnf2Hh28mLGw95ctXwNnZmQYNniIzM5OZMyfzn/+cAqB3bz969uxdaPtUZwgiYteOHfs3AQEfsGzZGjZsWEuFChVZvHgZTz5Zhx07tnPtWgq7d+/ik08+Z9myNbRt24H167/Kta+WLVuzcuU6Ll++zKZNG1i4MITQ0C+pWNGVlSuX0arVzcdoAxw69D1Xr14lKSmRmJgfeOqpRqSlpbJnz+67butW/3FxF9myJYyQkBXMm/cpCQnxAPz441GuXr3KkiVfMm/ep/z449FC3Vc6QxCRh6ZEXY/7fop/2Nzdn6Bq1ccAKF++As2btwCgatXHSE6+SrlyTkyZMoMdO7bz++9n2b8/ijp16uXaV8OGTwNw5MhBzp37nTfeuPm058zMDOrWrc+AAQNt34x25swZOnXqwg8/HOGXX/5N69ZtKFfOiQ8/nHbXbd3q//DhQ7Rs6UHZsmWBm4/PtlgsuLs/wdmzZ3j33b/TsqUHb745isKkQBARu2Y2Z/81d+s7Cm6Ji7vI22+PoE8fP1q2bI2rayVOnDiea1+3vvvAYrHSsWNnRo8eC8D169exWCyUKlWKJ5+sy/bt4dSsWZOmTZ/l0KHviYk5yksvDSYu7iKjRr1B374v5rqtW/2bTCYMI/v3KlgsFsqXr8CyZWs4cGA/+/btZdiwV1i2bE2OZ8bll6aMROQv7ZdfjvH444/Tv//LNGz4NNHRUVitlnu2adr0Wfbs2cXly0kYhsFHH81kzZovAWjd2oPQ0P+hadNnadr0WSIj91CmTGkqVKiQta0a991W8+bPERUVSUpKCmlpaezZ8y8AIiN3M2XKJFq3bsPo0e9RpkwZ4uPjCm1f6AxBRP7SnnuuJRs2fM0rr/hRokQJGjZ82nbR9m7q1KnL0KHDefvtERiGQZ069XjllSEAtGrVhsDAWTRt2hwXFxcqVKhIq1ZtbNtav37tfbdVp049/PwG8Nprg3B2dqZq1ZvfOteypQf/+lcEAwe+SMmSJWnfviNPPPFkoe0LPcuomLO3esD+arK3ekDPMnrU6VlGIiJSIAoEEREBFAgi8hAU05lou1CQfa9AEJFC5eDgiMWSWdTD+MuyWDJxcHC8/4q5UCCISKEqU8aJ5OQ/st1HL38Ow7CSnHyZMmXuffH4bnTbqYgUKien8ly+nEBc3DmgeEwdOTg4YLXaQ4CZKFmyNE5O5fPVWoEgIoXKZDLh6lqlqIeRJ/Z4a3B+aMpIREQABYKIiGQpcCDMnz+f4OBg2+tTp07x0ksv0atXL/r378/PP/+co41hGCxYsABfX1+6du3Khg0bCjoMEREpoHwHQnJyMgEBAYSEhGRbPnHiRIYPH87GjRsZPXo048aNy9E2LCyMqKgo1qxZw/Lly5kzZw5Xr17N71BERKQQ5DsQIiIiqFWrFkOHDs223M/Pj3bt2gFQr149YmNjc7QNDw9n2LBhlCxZEjc3N7788ktKly6d36GIiEghyHcg+Pr68vrrr+d4tnifPn1sy4KCgujcuXOOtmfOnOHUqVP079+f3r17c+zYMUqWLJnfoYiISCG4722n4eHhzJw5M9syd3d3QkND79rGMAzmzJnD0aNHWbp0aY73LRYLx48fZ/ny5Vy6dIkBAwbQsGFDatWq9cADv99T++7Fza1wvkziUWFv9YD91WRv9YD91WRv9UDea7pvIHh5eeHl5fXAHWZmZjJu3Dji4uJYunRprt/kU7lyZbp160aJEiWoVq0ajRs35tixY3kKBD3++iZ7qwfsryZ7qwfsryZ7qwdy1lQkj7+ePXs2KSkphISE3PVr3Tw9PQkPD8cwDC5fvkxMTAwNGjQo7KGIiEgeFOpfKiclJbFixQoef/xx/Pz8bMs3btxIREQEO3fuZPr06QwZMoR//vOf9OjRA4vFwsiRI6ldu3ZhDkVERPJI35hWzNlbPWB/NdlbPWB/NdlbPfCITBmJiEjxpEAQERFAgSAiIlkUCCIiAigQREQkiwJBREQABYKIiGRRIIiICKBAEBGRLAoEEREBFAgiIpJFgSAiIoACQUREsigQREQEUCCIiEgWBYKIiAAKBBERyaJAEBERQIEgIiJZFAgiIgIoEEREJIsCQUREgEIIhPnz5xMcHGx7ferUKV566SV69epF//79+fnnn3NtN2PGDLp3706PHj3YvHlzQYchIiIFlO9ASE5OJiAggJCQkGzLJ06cyPDhw9m4cSOjR49m3LhxOdru27ePmJgYwsLCCA0NZfLkydy4cSO/QxERkUKQ70CIiIigVq1aDB06NNtyPz8/2rVrB0C9evWIjY3N0dZisZCWlkZmZiY3btygZMmS+R2GiIgUEpNhGEZBOrg1XTRq1Kgc73344YekpaUxc+bMHO+9/fbb7N+/n+vXr/Pee+8xePDgggxDREQKyHy/FcLDw3P8Qnd3dyc0NPSubQzDYM6cORw9epSlS5fmeH/16tU4OjoSGRnJH3/8waBBg2jcuDFNmjR54IEnJqZgteY9y9zcnElISM5zu0eVvdUD9leTvdUD9leTvdUDOWtycDBRqZLTPdvcNxC8vLzw8vJ64EFkZmYybtw44uLiWLp0Kc7OzjnWiYiIYMCAAZQoUQI3Nzc6dOjAwYMH8xQIIiJSuAr9ttPZs2eTkpJCSEhIrmEAUL9+fXbs2AHA9evXiY6O5umnny7soYiISB4UaiAkJSWxYsUKfvvtN/z8/OjVqxe9evUCbp4VvP/++wCMGDGCzMxMvLy8ePHFF+nVqxctW7YszKGIiEgeFfiiclHRNYSb7K0esL+a7K0esL+a7K0eyN81BP2lsoiIAAoEERHJokAQERFAgSAiIlkUCCIiAigQREQkiwJBREQABYKIiGRRIIiICKBAEBGRLAoEEREBFAgiIpJFgSAiIoACQUREsigQREQEUCCIiEgWBYKIiAAKBBERyaJAEBERQIEgIiJZFAgiIgIUQiDMnz+f4OBg2+uTJ0/i7+9Pz549GThwIOfPn8/RxjAMZs+eTbdu3fD29ubQoUMFHYaIiBRQvgMhOTmZgIAAQkJCsi2fPHkyI0eOJCwsDG9vb+bOnZuj7bZt2zh16hRbt25lwYIFjB8/nszMzPwORURECoE5vw0jIiKoVasWQ4cOzbZ8yZIlmM1mrFYrFy5cwMXFJUfb3bt34+3tjYODA7Vr16Z69eocOXKE5557Lr/DERGRAsp3IPj6+gJkmy4CMJvNXL16FW9vb1JTU1m2bFmOtvHx8VSpUsX22s3NjYsXL+Z3KA/s3zu3YD69D8NqPPRt/VmOO5jsqh6wv5rsrR6wv5qKSz0W99Y81bH7Q+v/voEQHh7OzJkzsy1zd3cnNDT0rm1cXFyIjIxkz549vPnmm0RERODo6Gh73zBy7ngHh7zNXlWq5JSn9QHKlC5BBmByMOW57aPM3uoB+6vJ3uoB+6upONRTpnQJ3NycH3j9vKwLDxAIXl5eeHl5PXCHW7duxcvLC5PJRLt27UhNTeXKlSu4urra1qlatSoJCQm21wkJCdnOGB5EYmIK1jwmunvrLrj16ktCQnKe2j3K3Nyc7aoesL+a7K0esL+ailM9DzrOO2tycDDd94N0od92GhISwrfffgtAdHQ0FStWzBYGAO3atWPTpk1YLBbOnDnD6dOnadSoUWEPRURE8iDf1xDuZtasWUyaNIkFCxbg7OxMUFAQcPMi9M6dO5k+fTrdunUjJiaGnj17AjB9+nRKly5d2EMREZE8MBm5TegXA/mZMoLidWr4IOytHrC/muytHrC/muytHnhEpoxERKR4UiCIiAigQBARkSwKBBERARQIIiKSRYEgIiKAAkFERLIoEEREBFAgiIhIFgWCiIgACgQREcmiQBAREUCBICIiWRQIIiICKBBERCSLAkFERAAFgoiIZFEgiIgIoEAQEZEsCgQREQEUCCIikqXAgTB//nyCg4Ntr0+ePIm/vz89e/Zk4MCBnD9/Pkeba9eu8c477+Dj44OPjw9btmwp6DBERKSA8h0IycnJBAQEEBISkm355MmTGTlyJGFhYXh7ezN37twcbRctWkT16tXZtGkToaGhzJw5k0uXLuV3KCIiUgjM+W0YERFBrVq1GDp0aLblS5YswWw2Y7VauXDhAi4uLjnatmjRgtq1awNQqVIlKlSowKVLl6hcuXJ+hyMiIgWU70Dw9fUFyDZdBGA2m7l69Sre3t6kpqaybNmyHG09PDxs/966dSvp6ek8+eST+R2KiIgUgvsGQnh4ODNnzsy2zN3dndDQ0Lu2cXFxITIykj179vDmm28SERGBo6Njrn3PmDGD//mf/8Fszls2VarklKf1b+fm5pzvto8ie6sH7K8me6sH7K8me6sH8l7TfX8Le3l54eXl9cAdbt26FS8vL0wmE+3atSM1NZUrV67g6uqabb1ly5axePFiFi9eTL169fI0aIDExBSsViPP7dzcnElISM5zu0eVvdUD9leTvdUD9leTvdUDOWtycDDd94N0vqeM7iYkJASz2UyXLl2Ijo6mYsWKOcJgx44dhIaGsnLlSqpVq1bYQxARkXwo9ECYNWsWkyZNYsGCBTg7OxMUFATcvAi9c+dOpk+fTlBQEGlpaYwYMcLWbtq0aTRq1KiwhyMiIg/IZBhG3uddHgGaMrrJ3uoB+6vJ3uoB+6vJ3uqB/E0Z6S+VRUQEUCCIiEgWBYKIiAAKBBERyaJAEBERQIEgIiJZFAgiIgIoEEREJIsCQUREAAWCiIhkUSCIiAigQBARkSwKBBERARQIIiKSRYEgIiKAAkFERLIoEEREBFAgiIhIFgWCiIgACgQREcmiQBAREUCBICIiWQocCPPnzyc4ONj2+uTJk/j7+9OzZ08GDhzI+fPn79o2MzOT/v37s27duoIOQ0RECijfgZCcnExAQAAhISHZlk+ePJmRI0cSFhaGt7c3c+fOvWsfCxYs4PTp0/kdgoiIFCJzfhtGRERQq1Ythg4dmm35kiVLMJvNWK1WLly4gIuLS67tDx06xPHjx/H09MzvEEREpBDlOxB8fX0Bsk0XAZjNZq5evYq3tzepqaksW7YsR9uUlBRmzZrFwoULCQwMzNf2K1Vyylc7ADc353y3fRTZWz1gfzXZWz1gfzXZWz2Q95ruGwjh4eHMnDkz2zJ3d3dCQ0Pv2sbFxYXIyEj27NnDm2++SUREBI6Ojrb3J0+ezIgRI6hcuXKeBnu7xMQUrFYjz+3c3JxJSEjO93YfNfZWD9hfTfZWD9hfTfZWD+SsycHBdN8P0vcNBC8vL7y8vB54EFu3bsXLywuTyUS7du1ITU3lypUruLq6AjfPDvbt28evv/5KUFAQsbGxREdHYzab6dmz5wNvR0RECle+p4zuJiQkBLPZTJcuXYiOjqZixYq2MABwcnIiMjLS9nr8+PG0aNFCYSAiUsQK/e8QZs2axZIlS+jVqxeffPIJQUFBwM2L0O+//35hb05ERAqJyTCMvE/EPwJ0DeEme6sH7K8me6sH7K8me6sH8ncNQX+pLCIigAJBRESyKBBERARQIIiISBYFgoiIAAoEERHJokAQERFAgSAiIlkUCCIiAigQREQkiwJBREQABYKIiGRRIIiICKBAEBGRLAoEEREBFAgiIpJFgSAiIoACQUREsigQREQEUCCIiEgWBYKIiACFEAjz588nODjY9vrkyZP4+/vTs2dPBg4cyPnz53O0MQyDBQsW4OvrS9euXdmwYUNBhyEiIgWU70BITk4mICCAkJCQbMsnT57MyJEjCQsLw9vbm7lz5+ZoGxYWRlRUFGvWrGH58uXMmTOHq1ev5ncoIiJSCMz5bRgREUGtWrUYOnRotuVLlizBbDZjtVq5cOECLi4uOdqGh4czbNgwSpYsiZubG19++SWlS5fO71BERKQQ5DsQfH19AbJNFwGYzWauXr2Kt7c3qampLFu2LEfbM2fOcOrUKRYtWkR6ejrDhw+nVq1a+R2KiIgUgvsGQnh4ODNnzsy2zN3dndDQ0Lu2cXFxITIykj179vDmm28SERGBo6Oj7X2LxcLx48dZvnw5ly5dYsCAATRs2DBPoVCpktMDr3snNzfnfLd9FNlbPWB/NdlbPWB/NdlbPZD3mu4bCF5eXnh5eT1wh1u3bsXLywuTyUS7du1ITU3lypUruLq62tapXLky3bp1o0SJElSrVo3GjRtz7NixPAVCYmIKVqvxwOvf4ubmTEJCcp7bParsrR6wv5rsrR6wv5rsrR7IWZODg+m+H6QL/bbTkJAQvv32WwCio6OpWLFitjAA8PT0JDw8HMMwuHz5MjExMTRo0KCwhyIiInlQ6IEwa9YslixZQq9evfjkk08ICgoCbl6Efv/99wEYMmQIlStXpkePHgwYMICRI0dSu3btwh6KiIjkgckwjLzPuzwCNGV0k73VA/ZXk73VA/ZXk73VA4/IlJGIiBRPCgQREQEUCCIikkWBICIigAJBRESyKBBERARQIIiISBYFgoiIAAoEERHJokAQERFAgSAiIlny/QU5Rc3BwVQkbR9F9lYP2F9N9lYP2F9N9lYPZK/pQeortg+3ExGRwqUpIxERARQIIiKSRYEgIiKAAkFERLIoEEREBFAgiIhIFgWCiIgACgQREcmiQBAREeAvFAibNm3C29ubF154gRUrVhT1cArFoEGD6N69O7169aJXr14cPXq0qIeULykpKfTo0YNz584BEBUVhY+PD126dOHjjz8u4tHl3Z31TJgwgS5dutiO07ffflvEI8ybTz75hO7du9O9e3fmzJkDFP9jlFtNxfk4zZ8/H29vb7p3786SJUuAfB4j4y/g4sWLhqenp3H58mXj2rVrho+Pj3HixImiHlaBWK1Ww8PDw8jIyCjqoRTIDz/8YPTo0cN46qmnjN9//924ceOG0b59e+Ps2bNGRkaGMWzYMGPXrl1FPcwHdmc9hmEYPXr0MOLi4op4ZPmzd+9eo3///kZaWpqRnp5uDBo0yNi0aVOxPka51bR9+/Zie5z2799v+Pv7GxkZGcaNGzcMT09P4+eff87XMfpLnCFERUXRsmVLKlSoQNmyZenatSvffPNNUQ+rQP7zn/9gMpkYPnw4PXv2ZPny5UU9pHxZs2YN//jHP6hSpQoAMTEx1KxZkxo1amA2m/Hx8SlWx+rOeq5fv86FCxeYNGkSPj4+BAUFYbVai3iUD87NzY3x48dTsmRJSpQowRNPPMHp06eL9THKraYLFy4U2+PUokULli5ditlsJjExEYvFwtWrV/N1jP4SgRAfH4+bm5vtdZUqVYiLiyvCERXc1atXadWqFQsWLCA0NJRVq1axd+/eoh5Wnk2fPp3mzZvbXhf3Y3VnPYmJibRs2ZIZM2awZs0aDh48yNq1a4twhHlTp04dmjRpAsDp06fZunUrJpOpWB+j3Gpq27ZtsT5OJUqUICgoiO7du9OqVat8/z/6SwSCkcsDXU2m4v2o26ZNmzJnzhzKli2Lq6sr/fr1Y/fu3UU9rAKzt2NVo0YNFixYQKVKlShTpgwDBw4slsfpxIkTDBs2jHHjxvG3v/0tx/vF8RjdXpO7u3uxP05vv/02+/btIzY2ltOnT+d4/0GO0V8iEKpWrcqlS5dsr+Pj422n9MXVwYMH2bdvn+21YRiYzcX26y1s7O1YHT9+nG3bttleF8fjdOjQIYYMGcJ///d/07t3b7s4RnfWVJyP06lTp/j5558BKFOmDF26dGH//v35OkZ/iUBo3bo1+/btIykpiRs3brB9+3batWtX1MMqkOTkZObMmUNaWhopKSmsX7+eF154oaiHVWCNGzfmt99+48yZM1gsFjZv3lysj5VhGMyYMYMrV66QkZHB6tWri9Vxio2N5a233iIwMJDu3bsDxf8Y5VZTcT5O586dY+LEiaSnp5Oenk5ERAT+/v75OkbFIwILqGrVqowZM4ZBgwaRkZFBv379eOaZZ4p6WAXi6enJ0aNH8fX1xWq18tJLL9G0adOiHlaBlSpVilmzZjFq1CjS0tJo37493bp1K+ph5Vv9+vV5/fXXGTBgAJmZmXTp0oUePXoU9bAe2OLFi0lLS2PWrFm2Zf7+/sX6GN2tpuJ6nNq3b2/7XeDo6EiXLl3o3r07rq6ueT5G+sY0EREB/iJTRiIicn8KBBERARQIIiKSRYEgIiKAAkFERLIoEEQKYPjw4Zw8eTJPbd544w3WrVv3kEYkkn9/ib9DEHlYvvjii6IegkihUSDIX9LOnTtZuHAhGRkZlC5dmnHjxhEZGcmJEye4dOkSiYmJ1K9fn+nTp+Pk5MSXX37JqlWrKFGiBKVKlWLKlCk8+eSTdOzYkfnz59OoUSNWr17NsmXLcHBwoHLlykyaNInatWsTFxfH+PHjiY+Pp3r16iQmJtrGcerUKaZPn84ff/yBxWJh4MCB9OvXj2vXrjFhwgTOnDmDg4MDTz31FFOmTMHBQSf18hA9vKd0izyafvvtN6NHjx5GUlKSYRiG8euvvxoeHh7GrFmzjHbt2hkJCQmGxWIx3n33XWPWrFlGZmam8dRTT9melb9+/Xpj1apVhmEYhqenpxETE2NERUUZnTt3NhITEw3DMIyvv/7a8PLyMqxWqzFy5Ejj448/NgzDME6fPm00adLE+Prrr42MjAzD29vb+OmnnwzDMIyrV68aXl5expEjR4z169cbw4YNMwzDMDIzM43333/fOH369J+5m+QvSGcI8pezd+9e4uPjGTJkiG2ZyWTi7NmzdOvWjcqVKwPQr18/ZsyYwbhx4+jWrRv+/v506NABDw8PfHx8svX53Xff4e3tjaurKwB9+vRh+vTpnDt3jqioKMaNGwdAzZo1ef7554Gbj14+e/YsAQEBtn5SU1M5duwYbdu25eOPP2bgwIG0bt2awYMHU7NmzYe5W0Q0ZSR/PVarlVatWjFv3jzbstjYWFavXk16enq29W5N0QQGBvLrr78SFRXFF198wdq1a1m4cKFtXSOXJ8AYhkFmZiYmkynb+7eeommxWHBxcWHjxo229y5duoSzszOlSpXi22+/Zf/+/URHRzN06FAmTpxYrJ4ZJMWPJiTlL6dly5bs3buXU6dOAbB792569uxJWloaERERJCcnY7VaWbNmDZ6eniQlJdG+fXsqVKjAkCFDGD16NMePH8/WZ5s2bdi6dStJSUkAfP3111SoUIGaNWvStm1bVq9eDcCFCxfYv38/ALVr16ZUqVK2QIiNjaVHjx789NNPfPnll0yYMIE2bdowduxY2rRpw4kTJ/6sXSR/UXq4nfwlhYeH89lnn9meex8QEMC+ffuIjo7GYrFw+fJlnnvuOSZOnEjp0qVZtWoVS5cupXTp0jg6OjJmzBhat26d7aLyihUrWLVqFVarFVdXVz744APq1KlDUlISEyZM4OzZszz22GNkZmbSu3dv+vTpwy+//GK7qJyZmcmgQYMYMGAA169fJyAggOPHj1OmTBmqV6/O9OnTKV++fFHvOrFjCgSRLMHBwVy+fJkPPvigqIciUiQ0ZSQiIoDOEEREJIvOEEREBFAgiIhIFgWCiIgACgQREcmiQBAREUCBICIiWf4/HLka4/aihHoAAAAASUVORK5CYII=\n" + }, + "metadata": {} + } + ], + "source": [ + "cfg = QlearningConfig()\n", + "env,agent = env_agent_config(cfg,seed=1)\n", + "rewards,ma_rewards = train(cfg,env,agent)\n", + "make_dir(cfg.result_path,cfg.model_path)\n", + "agent.save(path=cfg.model_path)\n", + "save_results(rewards,ma_rewards,tag='train',path=cfg.result_path)\n", + "plot_rewards(rewards,ma_rewards,tag=\"train\",env=cfg.env,algo = cfg.algo,path=cfg.result_path)\n", + "\n", + "env,agent = env_agent_config(cfg,seed=10)\n", + "agent.load(path=cfg.model_path)\n", + "rewards,ma_rewards = eval(cfg,env,agent)\n", + "save_results(rewards,ma_rewards,tag='eval',path=cfg.result_path)\n", + "plot_rewards(rewards,ma_rewards,tag=\"eval\",env=cfg.env,algo = cfg.algo,path=cfg.result_path)" + ] + } + ] +} \ No newline at end of file diff --git a/codes/QLearning/main.py b/codes/QLearning/task0_train.py similarity index 56% rename from codes/QLearning/main.py rename to codes/QLearning/task0_train.py index 0892bee..73fedae 100644 --- a/codes/QLearning/main.py +++ b/codes/QLearning/task0_train.py @@ -5,11 +5,10 @@ Author: John Email: johnjim0816@gmail.com Date: 2020-09-11 23:03:00 LastEditor: John -LastEditTime: 2021-03-31 18:14:59 +LastEditTime: 2021-04-29 17:01:08 Discription: Environment: ''' - import sys,os curr_path = os.path.dirname(__file__) parent_path=os.path.dirname(curr_path) @@ -18,40 +17,41 @@ sys.path.append(parent_path) # add current terminal path to sys.path import gym import datetime -from envs.gridworld_env import CliffWalkingWapper, FrozenLakeWapper +from envs.gridworld_env import CliffWalkingWapper from QLearning.agent import QLearning from common.plot import plot_rewards -from common.utils import save_results - -SEQUENCE = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # obtain current time -SAVED_MODEL_PATH = curr_path+"/saved_model/"+SEQUENCE+'/' # path to save model -if not os.path.exists(curr_path+"/saved_model/"): - os.mkdir(curr_path+"/saved_model/") -if not os.path.exists(SAVED_MODEL_PATH): - os.mkdir(SAVED_MODEL_PATH) -RESULT_PATH = curr_path+"/results/"+SEQUENCE+'/' # path to save rewards -if not os.path.exists(curr_path+"/results/"): - os.mkdir(curr_path+"/results/") -if not os.path.exists(RESULT_PATH): - os.mkdir(RESULT_PATH) +from common.utils import save_results,make_dir +curr_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") # obtain current time class QlearningConfig: '''训练相关参数''' def __init__(self): - self.train_eps = 200 # 训练的episode数目 + self.algo = 'Qlearning' + self.env = 'CliffWalking-v0' # 0 up, 1 right, 2 down, 3 left + self.result_path = curr_path+"/outputs/" +self.env+'/'+curr_time+'/results/' # path to save results + self.model_path = curr_path+"/outputs/" +self.env+'/'+curr_time+'/models/' # path to save models + self.train_eps = 300 # 训练的episode数目 + self.eval_eps = 30 self.gamma = 0.9 # reward的衰减率 - self.epsilon_start = 0.99 # e-greedy策略中初始epsilon + self.epsilon_start = 0.95 # e-greedy策略中初始epsilon self.epsilon_end = 0.01 # e-greedy策略中的终止epsilon self.epsilon_decay = 200 # e-greedy策略中epsilon的衰减率 self.lr = 0.1 # learning rate + +def env_agent_config(cfg,seed=1): + env = gym.make(cfg.env) + env = CliffWalkingWapper(env) + env.seed(seed) + state_dim = env.observation_space.n + action_dim = env.action_space.n + agent = QLearning(state_dim,action_dim,cfg) + return env,agent def train(cfg,env,agent): rewards = [] ma_rewards = [] # moving average reward - steps = [] # 记录所有episode的steps - for i_episode in range(cfg.train_eps): + for i_ep in range(cfg.train_eps): ep_reward = 0 # 记录每个episode的reward - ep_steps = 0 # 记录每个episode走了多少step state = env.reset() # 重置环境, 重新开一局（即开始新的一个episode） while True: action = agent.choose_action(state) # 根据算法选择一个动作 @@ -59,55 +59,52 @@ def train(cfg,env,agent): agent.update(state, action, reward, next_state, done) # Q-learning算法更新 state = next_state # 存储上一个观察值 ep_reward += reward - ep_steps += 1 # 计算step数 if done: break - steps.append(ep_steps) rewards.append(ep_reward) if ma_rewards: ma_rewards.append(ma_rewards[-1]*0.9+ep_reward*0.1) else: ma_rewards.append(ep_reward) - print("Episode:{}/{}: reward:{:.1f}".format(i_episode+1, cfg.train_eps,ep_reward)) + print("Episode:{}/{}: reward:{:.1f}".format(i_ep+1, cfg.train_eps,ep_reward)) return rewards,ma_rewards - + def eval(cfg,env,agent): # env = gym.make("FrozenLake-v0", is_slippery=False) # 0 left, 1 down, 2 right, 3 up # env = FrozenLakeWapper(env) rewards = [] # 记录所有episode的reward ma_rewards = [] # 滑动平均的reward - steps = [] # 记录所有episode的steps - for i_episode in range(cfg.train_eps): + for i_ep in range(cfg.eval_eps): ep_reward = 0 # 记录每个episode的reward - ep_steps = 0 # 记录每个episode走了多少step state = env.reset() # 重置环境, 重新开一局（即开始新的一个episode） while True: - action = agent.choose_action(state) # 根据算法选择一个动作 + action = agent.predict(state) # 根据算法选择一个动作 next_state, reward, done, _ = env.step(action) # 与环境进行一个交互 state = next_state # 存储上一个观察值 ep_reward += reward - ep_steps += 1 # 计算step数 if done: break - steps.append(ep_steps) rewards.append(ep_reward) - # 计算滑动平均的reward if ma_rewards: - ma_rewards.append(rewards[-1]*0.9+ep_reward*0.1) + ma_rewards.append(ma_rewards[-1]*0.9+ep_reward*0.1) else: ma_rewards.append(ep_reward) - print("Episode:{}/{}: reward:{:.1f}".format(i_episode+1, cfg.train_eps,ep_reward)) + print(f"Episode:{i_ep+1}/{cfg.eval_eps}, reward:{ep_reward:.1f}") return rewards,ma_rewards if __name__ == "__main__": cfg = QlearningConfig() - env = gym.make("CliffWalking-v0") # 0 up, 1 right, 2 down, 3 left - env = CliffWalkingWapper(env) - action_dim = env.action_space.n - agent = QLearning(action_dim,cfg) + env,agent = env_agent_config(cfg,seed=1) rewards,ma_rewards = train(cfg,env,agent) - agent.save(path=SAVED_MODEL_PATH) - save_results(rewards,ma_rewards,tag='train',path=RESULT_PATH) - plot_rewards(rewards,ma_rewards,tag="train",algo = "On-Policy First-Visit MC Control",path=RESULT_PATH) + make_dir(cfg.result_path,cfg.model_path) + agent.save(path=cfg.model_path) + save_results(rewards,ma_rewards,tag='train',path=cfg.result_path) + plot_rewards(rewards,ma_rewards,tag="train",env=cfg.env,algo = cfg.algo,path=cfg.result_path) + + env,agent = env_agent_config(cfg,seed=10) + agent.load(path=cfg.model_path) + rewards,ma_rewards = eval(cfg,env,agent) + save_results(rewards,ma_rewards,tag='eval',path=cfg.result_path) + plot_rewards(rewards,ma_rewards,tag="eval",env=cfg.env,algo = cfg.algo,path=cfg.result_path) diff --git a/codes/README.md b/codes/README.md index 2f51e2a..c392ed3 100644 --- a/codes/README.md +++ b/codes/README.md @@ -27,26 +27,25 @@ python 3.7、pytorch 1.6.0-1.7.1、gym 0.17.0-0.18.0 ## 算法进度 -| 算法名称 | 相关论文材料 | 环境 | 备注 | -| :--------------------------------------: | :----------------------------------------------------------: | ------------------------------------- | :--------------------------------: | -| [On-Policy First-Visit MC](./MonteCarlo) | | [Racetrack](./envs/racetrack_env.md) | | -| [Q-Learning](./QLearning) | | [CliffWalking-v0](./envs/gym_info.md) | | -| [Sarsa](./Sarsa) | | [Racetrack](./envs/racetrack_env.md) | | -| [DQN](./DQN) | [DQN Paper](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf),[Nature DQN Paper](https://www.nature.com/articles/nature14236) | [CartPole-v0](./envs/gym_info.md) | | -| [DQN-cnn](./DQN_cnn) | [DQN Paper](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf) | [CartPole-v0](./envs/gym_info.md) | 与DQN相比使用了CNN而不是全链接网络 | -| [DoubleDQN](./DoubleDQN) | | [CartPole-v0](./envs/gym_info.md) | | -| [Hierarchical DQN](HierarchicalDQN) | [H-DQN Paper](https://arxiv.org/abs/1604.06057) | [CartPole-v0](./envs/gym_info.md) | | -| [PolicyGradient](./PolicyGradient) | | [CartPole-v0](./envs/gym_info.md) | | -| [A2C](./A2C) | [A3C Paper](https://arxiv.org/abs/1602.01783) | [CartPole-v0](./envs/gym_info.md) | | -| [SAC](./SAC) | [SAC Paper](https://arxiv.org/abs/1801.01290) | [Pendulum-v0](./envs/gym_info.md) | | -| [PPO](./PPO) | [PPO paper](https://arxiv.org/abs/1707.06347) | [CartPole-v0](./envs/gym_info.md) | | -| [DDPG](./DDPG) | [DDPG Paper](https://arxiv.org/abs/1509.02971) | [Pendulum-v0](./envs/gym_info.md) | | -| [TD3](./TD3) | [TD3 Paper](https://arxiv.org/abs/1802.09477) | HalfCheetah-v2 | | - +| 算法名称 | 相关论文材料 | 环境 | 备注 | +| :--------------------------------------: | :----------------------------------------------------------: | ----------------------------------------- | :--------------------------------: | +| [On-Policy First-Visit MC](./MonteCarlo) | [medium blog](https://medium.com/analytics-vidhya/monte-carlo-methods-in-reinforcement-learning-part-1-on-policy-methods-1f004d59686a) | [Racetrack](./envs/racetrack_env.md) | | +| [Q-Learning](./QLearning) | [towardsdatascience blog](https://towardsdatascience.com/simple-reinforcement-learning-q-learning-fcddc4b6fe56),[q learning paper](https://ieeexplore.ieee.org/document/8836506) | [CliffWalking-v0](./envs/gym_info.md) | | +| [Sarsa](./Sarsa) | [geeksforgeeks blog](https://www.geeksforgeeks.org/sarsa-reinforcement-learning/) | [Racetrack](./envs/racetrack_env.md) | | +| [DQN](./DQN) | [DQN Paper](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf),[Nature DQN Paper](https://www.nature.com/articles/nature14236) | [CartPole-v0](./envs/gym_info.md) | | +| [DQN-cnn](./DQN_cnn) | [DQN Paper](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf) | [CartPole-v0](./envs/gym_info.md) | 与DQN相比使用了CNN而不是全链接网络 | +| [DoubleDQN](./DoubleDQN) | [DoubleDQN Paper](https://arxiv.org/abs/1509.06461) | [CartPole-v0](./envs/gym_info.md) | | +| [Hierarchical DQN](HierarchicalDQN) | [H-DQN Paper](https://arxiv.org/abs/1604.06057) | [CartPole-v0](./envs/gym_info.md) | | +| [PolicyGradient](./PolicyGradient) | [Lil'log](https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html) | [CartPole-v0](./envs/gym_info.md) | | +| [A2C](./A2C) | [A3C Paper](https://arxiv.org/abs/1602.01783) | [CartPole-v0](./envs/gym_info.md) | | +| [SAC](./SAC) | [SAC Paper](https://arxiv.org/abs/1801.01290) | [Pendulum-v0](./envs/gym_info.md) | | +| [PPO](./PPO) | [PPO paper](https://arxiv.org/abs/1707.06347) | [CartPole-v0](./envs/gym_info.md) | | +| [DDPG](./DDPG) | [DDPG Paper](https://arxiv.org/abs/1509.02971) | [Pendulum-v0](./envs/gym_info.md) | | +| [TD3](./TD3) | [TD3 Paper](https://arxiv.org/abs/1802.09477) | [HalfCheetah-v2]((./envs/mujoco_info.md)) | | ## Refs [RL-Adventure-2](https://github.com/higgsfield/RL-Adventure-2) -[RL-Adventure](https://github.com/higgsfield/RL-Adventure) +[RL-Adventure](https://github.com/higgsfield/RL-Adventure) \ No newline at end of file diff --git a/codes/README_en.md b/codes/README_en.md index 95a6455..5e9a30c 100644 --- a/codes/README_en.md +++ b/codes/README_en.md @@ -30,25 +30,26 @@ similar to file with ```eval```, which means to evaluate the agent. ## Schedule -| Name | Related materials | Used Envs | Notes | -| :--------------------------------------: | :----------------------------------------------------------: | ------------------------------------- | :---: | -| [On-Policy First-Visit MC](./MonteCarlo) | | [Racetrack](./envs/racetrack_env.md) | | -| [Q-Learning](./QLearning) | | [CliffWalking-v0](./envs/gym_info.md) | | -| [Sarsa](./Sarsa) | | [Racetrack](./envs/racetrack_env.md) | | -| [DQN](./DQN) | [DQN-paper](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf),[Nature DQN Paper](https://www.nature.com/articles/nature14236) | [CartPole-v0](./envs/gym_info.md) | | -| [DQN-cnn](./DQN_cnn) | [DQN-paper](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf) | [CartPole-v0](./envs/gym_info.md) | | -| [DoubleDQN](./DoubleDQN) | | [CartPole-v0](./envs/gym_info.md) | | -| [Hierarchical DQN](HierarchicalDQN) | [Hierarchical DQN](https://arxiv.org/abs/1604.06057) | [CartPole-v0](./envs/gym_info.md) | | -| [PolicyGradient](./PolicyGradient) | | [CartPole-v0](./envs/gym_info.md) | | -| [A2C](./A2C) | [A3C Paper](https://arxiv.org/abs/1602.01783) | [CartPole-v0](./envs/gym_info.md) | | -| [SAC](./SAC) | [SAC Paper](https://arxiv.org/abs/1801.01290) | | | -| [PPO](./PPO) | [PPO paper](https://arxiv.org/abs/1707.06347) | [CartPole-v0](./envs/gym_info.md) | | -| [DDPG](./DDPG) | [DDPG Paper](https://arxiv.org/abs/1509.02971) | [Pendulum-v0](./envs/gym_info.md) | | -| [TD3](./TD3) | [TD3 Paper](https://arxiv.org/abs/1802.09477) | HalfCheetah-v2 | | +| Name | Related materials | Used Envs | Notes | +| :--------------------------------------: | :----------------------------------------------------------: | ----------------------------------------- | :---: | +| [On-Policy First-Visit MC](./MonteCarlo) | [medium blog](https://medium.com/analytics-vidhya/monte-carlo-methods-in-reinforcement-learning-part-1-on-policy-methods-1f004d59686a) | [Racetrack](./envs/racetrack_env.md) | | +| [Q-Learning](./QLearning) | [towardsdatascience blog](https://towardsdatascience.com/simple-reinforcement-learning-q-learning-fcddc4b6fe56),[q learning paper](https://ieeexplore.ieee.org/document/8836506) | [CliffWalking-v0](./envs/gym_info.md) | | +| [Sarsa](./Sarsa) | [geeksforgeeks blog](https://www.geeksforgeeks.org/sarsa-reinforcement-learning/) | [Racetrack](./envs/racetrack_env.md) | | +| [DQN](./DQN) | [DQN Paper](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf),[Nature DQN Paper](https://www.nature.com/articles/nature14236) | [CartPole-v0](./envs/gym_info.md) | | +| [DQN-cnn](./DQN_cnn) | [DQN Paper](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf) | [CartPole-v0](./envs/gym_info.md) | 与DQN相比使用了CNN而不是全链接网络 | +| [DoubleDQN](./DoubleDQN) | [DoubleDQN Paper](https://arxiv.org/abs/1509.06461) | [CartPole-v0](./envs/gym_info.md) | | +| [Hierarchical DQN](HierarchicalDQN) | [H-DQN Paper](https://arxiv.org/abs/1604.06057) | [CartPole-v0](./envs/gym_info.md) | | +| [PolicyGradient](./PolicyGradient) | [Lil'log](https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html) | [CartPole-v0](./envs/gym_info.md) | | +| [A2C](./A2C) | [A3C Paper](https://arxiv.org/abs/1602.01783) | [CartPole-v0](./envs/gym_info.md) | | +| [SAC](./SAC) | [SAC Paper](https://arxiv.org/abs/1801.01290) | [Pendulum-v0](./envs/gym_info.md) | | +| [PPO](./PPO) | [PPO paper](https://arxiv.org/abs/1707.06347) | [CartPole-v0](./envs/gym_info.md) | | +| [DDPG](./DDPG) | [DDPG Paper](https://arxiv.org/abs/1509.02971) | [Pendulum-v0](./envs/gym_info.md) | | +| [TD3](./TD3) | [TD3 Paper](https://arxiv.org/abs/1802.09477) | [HalfCheetah-v2]((./envs/mujoco_info.md)) | | + ## Refs [RL-Adventure-2](https://github.com/higgsfield/RL-Adventure-2) -[RL-Adventure](https://github.com/higgsfield/RL-Adventure) +[RL-Adventure](https://github.com/higgsfield/RL-Adventure) \ No newline at end of file diff --git a/codes/common/plot.py b/codes/common/plot.py index 8bf1689..92e4d96 100644 --- a/codes/common/plot.py +++ b/codes/common/plot.py @@ -5,7 +5,7 @@ Author: John Email: johnjim0816@gmail.com Date: 2020-10-07 20:57:11 LastEditor: John -LastEditTime: 2021-04-28 10:13:21 +LastEditTime: 2021-04-29 15:41:48 Discription: Environment: ''' @@ -19,7 +19,7 @@ def plot_rewards(rewards,ma_rewards,tag="train",env='CartPole-v0',algo = "DQN",s plt.plot(ma_rewards,label='ma rewards') plt.legend() if save: - plt.savefig(path+"rewards_curve_{}".format(tag)) + plt.savefig(path+"{}_rewards_curve".format(tag)) plt.show() # def plot_rewards(dic,tag="train",env='CartPole-v0',algo = "DQN",save=True,path='./'): # sns.set() diff --git a/codes/common/utils.py b/codes/common/utils.py index d397c89..5d51eea 100644 --- a/codes/common/utils.py +++ b/codes/common/utils.py @@ -5,7 +5,7 @@ Author: John Email: johnjim0816@gmail.com Date: 2021-03-12 16:02:24 LastEditor: John -LastEditTime: 2021-04-13 18:34:20 +LastEditTime: 2021-04-29 15:32:38 Discription: Environment: ''' @@ -18,8 +18,8 @@ from pathlib import Path def save_results(rewards,ma_rewards,tag='train',path='./results'): '''保存reward等结果 ''' - np.save(path+'rewards_'+tag+'.npy', rewards) - np.save(path+'ma_rewards_'+tag+'.npy', ma_rewards) + np.save(path+'{}_rewards.npy'.format(tag), rewards) + np.save(path+'{}_ma_rewards.npy'.format(tag), ma_rewards) print('results saved!') def make_dir(*paths): diff --git a/codes/envs/assets/image-20210429150622353.png b/codes/envs/assets/image-20210429150622353.png new file mode 100644 index 0000000..1216b4c Binary files /dev/null and b/codes/envs/assets/image-20210429150622353.png differ diff --git a/codes/envs/assets/image-20210429150630806.png b/codes/envs/assets/image-20210429150630806.png new file mode 100644 index 0000000..45107d5 Binary files /dev/null and b/codes/envs/assets/image-20210429150630806.png differ diff --git a/codes/envs/mujoco_info.md b/codes/envs/mujoco_info.md new file mode 100644 index 0000000..aaa8cbb --- /dev/null +++ b/codes/envs/mujoco_info.md @@ -0,0 +1,42 @@ +# MuJoCo + +MuJoCo（Multi-Joint dynamics with Contact）是一个物理模拟器，可以用于机器人控制优化等研究。安装见[Mac安装MuJoCo以及mujoco_py](https://blog.csdn.net/JohnJim0/article/details/115656392?spm=1001.2014.3001.5501) + + + +## HalfCheetah-v2 + + + +该环境基于mujoco仿真引擎，该环境的目的是使一只两只脚的“猎豹”跑得越快越好(下面图谷歌HalfCheetah-v2的，https://gym.openai.com/envs/HalfCheetah-v2/)。 + +image-20210429150630806 + +动作空间：Box(6,)，一只脚需要控制三个关节一共6个关节，每个关节的运动范围为[-1, 1]。 + +状态空间：Box(17, )，包含各种状态，每个值的范围为![img](assets/9cd6ae68c9aad008ede4139da358ec26.svg)，主要描述“猎豹”本身的姿态等信息。 + +回报定义：每一步的回报与这一步的中猎豹的速度和猎豹行动的消耗有关，定义回报的代码如下。 + +```python +def step(self, action): + xposbefore = self.sim.data.qpos[0] + self.do_simulation(action, self.frame_skip) + xposafter = self.sim.data.qpos[0] + ob = self._get_obs() + reward_ctrl = - 0.1 * np.square(action).sum() + reward_run = (xposafter - xposbefore)/self.dt + # =========== reward =========== + reward = reward_ctrl + reward_run + # =========== reward =========== + done = False + return ob, reward, done, dict(reward_run=reward_run, reward_ctrl=reward_ctrl) +``` + +当猎豹无法控制平衡而倒下时，一个回合(episode)结束。 + +但是这个环境有一些问题，目前经过搜索并不知道一个回合的reward上限，实验中训练好的episode能跑出平台之外： + +image-20210429150622353 + +加上时间有限，所以训练中reward一直处于一个平缓上升的状态，本人猜测这可能是gym的一个bug。 \ No newline at end of file diff --git a/codes/envs/snake/agent.py b/codes/envs/snake/agent.py index e514dc3..b32de9d 100644 --- a/codes/envs/snake/agent.py +++ b/codes/envs/snake/agent.py @@ -78,7 +78,6 @@ class Agent: :param points: float, the current points from environment :param dead: boolean, if the snake is dead :return: the index of action. 0,1,2,3 indicates up,down,left,right separately - TODO: write your function here. Return the index of action the snake needs to take, according to the state and points known from environment. Tips: you need to discretize the state to the state space defined on the webpage first. (Note that [adjoining_wall_x=0, adjoining_wall_y=0] is also the case when snake runs out of the 480x480 board)