Meta-learning adaptive intrinsic reward weighting for curiosity-driven reinforcement learning

Ziki, Batsirayi Mupamhi

Meta-learning adaptive intrinsic reward weighting for curiosity-driven reinforcement learning

Thesis / Dissertation

2026

Publisher

University of Cape Town

Department

Department of Mathematics and Applied Mathematics

Faculty

Faculty of Science

Abstract

For both organisms and artificial agents, exploration is essential to continue learning and avoid becoming trapped in suboptimal behaviours. Reinforcement learning (RL) agents can also face exploration challenges in environments with sparse feedback. Curiosity-driven exploration algorithms can help address these challenges by providing intrinsic rewards based on the novelty of situations an agent encounters. These intrinsic rewards are typically combined with extrinsic rewards using a weighted sum with the parameter λ. However, fine-tuning λ for each task across multiple environments can become computationally expensive. We propose a meta-learning approach for automatic tuning of λ using a recurrent neural network (RNN) that dynamically outputs λ values. We call this RNN the reward combiner. The reward combiner was trained using evolutionary strategies on XLand-MiniGrid environments, where feedback is sparse. The fitness function was the total extrinsic reward obtained during the training phase of an agent. We used BYOL-Explore, a curiosity-driven exploration algorithm, for intrinsic reward generation. The reward combiner takes normalised extrinsic and intrinsic rewards as input, along with actions that provide task-specific context for λ selection. Trained on Unlock and Empty-16x16 environments, the reward combiner generalises across different grid sizes of the same task, outperforming baselines when tested on DoorKey environments. It also generalises across different tasks when tested on UnlockPickUp, where the objective differs from the training environments. Our approach achieves higher extrinsic returns at the end of training than curiosity-driven baselines across all test environments. Despite being tested only within XLand-MiniGrid environments, our results indicate this approach has potential to eliminate costly hyperparameter sweeps when switching to new tasks with similar mechanics.

Keywords

reinforcement learning

exploration algorithms

Reference:

Collections

Masters

Full item page