Scalable hierarchical evolution strategies
| dc.contributor.advisor | Nitschke, Geoff | |
| dc.contributor.author | Abramowitz, Sasha | |
| dc.date.accessioned | 2023-02-23T08:43:16Z | |
| dc.date.available | 2023-02-23T08:43:16Z | |
| dc.date.issued | 2022 | |
| dc.date.updated | 2023-02-20T12:09:06Z | |
| dc.description.abstract | Hierarchical reinforcement learning (HRL) has been steadily growing in popularity for solving the hardest reinforcement learning problems. However, current HRL algorithms are relatively slow and brittle to hyperparameter changes. This paper offers a solution to these slow and brittle HRL algorithms, by investigating a novel method combining Scalable Evolution Strategies (SES) and HRL. S-ES, named for its excellent scalability, was popularised by Open AI when they showed its performance to be comparable to state-of-the art policy gradient methods. However, S-ES has not been tested in conjunction with HRL methods, which empower temporal abstraction thus allowing agents to tackle more challenging problems. We introduce a novel method merging S-ES and HRL, which creates a highly scalable and fast (wall-clock time) algorithm. We demonstrate that S-ES needs no hyper-parameter tuning for the HRL tasks tested and is indifferent to delayed rewards. This results in a method that is significantly faster than gradient-based HRL methods while having competitive task performance. We extend this method using transfer learning with the aim of increasing task performance and novelty search with the goal of improving its exploration characteristics. The paper's main contribution is thus a novel evolutionary HRL method, namely Scalable Hierarchical Evolution Strategies, which yields greater learning speed and competitive task-performance compared to state-of-the-art gradient-based methods, across a range of tasks. | |
| dc.identifier.apacitation | Abramowitz, S. (2022). <i>Scalable hierarchical evolution strategies</i>. (). ,Faculty of Science ,Department of Computer Science. Retrieved from http://hdl.handle.net/11427/36999 | en_ZA |
| dc.identifier.chicagocitation | Abramowitz, Sasha. <i>"Scalable hierarchical evolution strategies."</i> ., ,Faculty of Science ,Department of Computer Science, 2022. http://hdl.handle.net/11427/36999 | en_ZA |
| dc.identifier.citation | Abramowitz, S. 2022. Scalable hierarchical evolution strategies. . ,Faculty of Science ,Department of Computer Science. http://hdl.handle.net/11427/36999 | en_ZA |
| dc.identifier.ris | TY - Master Thesis AU - Abramowitz, Sasha AB - Hierarchical reinforcement learning (HRL) has been steadily growing in popularity for solving the hardest reinforcement learning problems. However, current HRL algorithms are relatively slow and brittle to hyperparameter changes. This paper offers a solution to these slow and brittle HRL algorithms, by investigating a novel method combining Scalable Evolution Strategies (SES) and HRL. S-ES, named for its excellent scalability, was popularised by Open AI when they showed its performance to be comparable to state-of-the art policy gradient methods. However, S-ES has not been tested in conjunction with HRL methods, which empower temporal abstraction thus allowing agents to tackle more challenging problems. We introduce a novel method merging S-ES and HRL, which creates a highly scalable and fast (wall-clock time) algorithm. We demonstrate that S-ES needs no hyper-parameter tuning for the HRL tasks tested and is indifferent to delayed rewards. This results in a method that is significantly faster than gradient-based HRL methods while having competitive task performance. We extend this method using transfer learning with the aim of increasing task performance and novelty search with the goal of improving its exploration characteristics. The paper's main contribution is thus a novel evolutionary HRL method, namely Scalable Hierarchical Evolution Strategies, which yields greater learning speed and competitive task-performance compared to state-of-the-art gradient-based methods, across a range of tasks. DA - 2022_ DB - OpenUCT DP - University of Cape Town KW - Computer Science LK - https://open.uct.ac.za PY - 2022 T1 - Scalable hierarchical evolution strategies TI - Scalable hierarchical evolution strategies UR - http://hdl.handle.net/11427/36999 ER - | en_ZA |
| dc.identifier.uri | http://hdl.handle.net/11427/36999 | |
| dc.identifier.vancouvercitation | Abramowitz S. Scalable hierarchical evolution strategies. []. ,Faculty of Science ,Department of Computer Science, 2022 [cited yyyy month dd]. Available from: http://hdl.handle.net/11427/36999 | en_ZA |
| dc.language.rfc3066 | eng | |
| dc.publisher.department | Department of Computer Science | |
| dc.publisher.faculty | Faculty of Science | |
| dc.subject | Computer Science | |
| dc.title | Scalable hierarchical evolution strategies | |
| dc.type | Master Thesis | |
| dc.type.qualificationlevel | Masters | |
| dc.type.qualificationlevel | MSc |