Project
Automatic Hyperparameter Optimization and Reward-Shaping
The efficacy of Deep Reinforcement Learning (RL) is critically dependent on algorithmic design choices, primarily hyperparameter configurations and reward formulations. In practical applications beyond standard benchmarks, defining an optimal configuration is non-trivial; hyperparameter settings and reward shaping exhibit deep interdependencies, and the optimisation landscapes are characteristically rugged and non-stationary. Automated Reinforcement Learning (AutoRL) addresses these challenges through data-driven methodologies, aiming to automate the derivation of these configurations. Advancing AutoRL necessitates High-Performance Computing (HPC) infrastructure. RL evaluation inherently requires substantial computational resources due to sample inefficiency and the necessity of running multiple random seeds to account for high performance variance. AutoRL scales this cost exponentially. Tasks such as executing black-box optimisation over joint search spaces, generating massive meta-datasets for surrogate modelling, and empirically validating dynamic adaptation frameworks require immense parallelisation and hardware acceleration. HPC enables the systematic exploration of these complex task spaces, making rigorous AutoRL research computationally tractable.
Project Details
Project term
February 5, 2025–May 4, 2026
Affiliations
RWTH Aachen University
Institute
Faculty of Computer Science
Principal Investigator
Methods
1. Joint Optimisation of Hyperparameters and Reward Shape: We formulated a combined search space encompassing both algorithmic hyperparameters and reward shaping coefficients. This was optimised using the state-of-the-art black-box optimiser DEHB. To address policy instability, we integrated a variance penalty into a multi-objective optimisation framework. The methodology was evaluated using Proximal Policy Optimisation (PPO) and Soft Actor-Critic (SAC) across environments, including LunarLander, Google Brax, and Robosuite Wipe.
2. Hardware-Accelerated Benchmarking (ARLBench): To mitigate the extreme costs of AutoRL evaluation, we developed ARLBench. This framework utilises JAX-based implementations of PPO, DQN, and SAC to maximise hardware utilisation. Furthermore, we employed predictive performance modelling to isolate a minimal representative subset of environments that accurately predicts global algorithm performance, drastically reducing the required evaluation budget.
3. Surrogate Modelling and Landscape Analysis: We evaluated the efficacy of surrogate models for predicting RL performance across diverse hyperparameter configurations using a large-scale meta-dataset. To counteract the inherent noise in RL landscapes, we developed a configuration space reduction technique based on HyperSHAP, isolating the subset of hyperparameters that can be reliably modelled.
4. Dynamic Online Adaptation (DOODAT): We developed the Dynamic Optimisation Of Discount-factor And Trace-decay (DOODAT) framework to adjust the discount factor (γ) and trace-decay (λ) during a single training run. DOODAT augments the value and policy functions with auxiliary heads perturbed by varying γ and λ. It evaluates these policies under a shared value function, moving the primary hyperparameters toward the most performant gradient, thereby avoiding the myopic instability of standard meta-gradients.
Results
1. Joint Optimisation (2024): Empirical results demonstrated that joint optimisation matches the performance of isolated optimisation under identical compute budgets, while significantly improving performance in complex continuous control tasks (e.g., Humanoid and Robosuite Wipe). The inclusion of the variance penalty successfully yielded configurations that maintained high expected returns while substantially reducing policy variance.
2. Benchmarking Efficiency (2025): ARLBench achieved massive reductions in computational overhead. Evaluating a hyperparameter optimisation budget of 32 configurations (10 seeds each) required only 937 GPU hours, compared to 8,163 GPU hours for standard implementations. This resulted in speedups of 9.6x for PPO, 7.14x for DQN, and 11.61x for SAC. This phase also produced a public meta-dataset containing over 100,000 RL runs.
3. Performance Prediction Constraints (2025): The analysis revealed that standard offline surrogate models fail to generalise accurately across RL hyperparameter landscapes due to extreme ruggedness and noise. However, the proposed HyperSHAP-based reduction method successfully identified constrained subspaces where surrogate modelling remains viable, providing a mechanism to extract reliable local hyperparameter importance.
4. Dynamic Adaptation Efficacy (2026): DOODAT consistently outperformed static hyperparameter tuning and Bootstrapped Meta-Gradients (BMG) across the Arcade Learning Environment (ALE). Notably, DOODAT demonstrated extreme robustness; when initialised with highly suboptimal hyperparameters that caused baseline methods to fail, DOODAT recovered and achieved an Interquartile Mean (IQM) of 1.0.
Discussion
Our findings underscore that isolated tuning of RL components is suboptimal. The mutual dependence of hyperparameters and reward shaping dictates that optimal performance can only be achieved through unified AutoRL approaches. Furthermore, the inherent noise of RL landscapes limits the utility of offline surrogate models for global performance prediction, highlighting the necessity of dynamic, single-run adaptation mechanisms like DOODAT. The integration of JAX-based parallelisation and intelligent environment subset selection (ARLBench) provides a critical foundation for managing the computational demands of this research.
Future work will focus on the cost of RL in hyperparameter optimisation. Particularly, how cost-aware optimisation methodologies can improve performance efficiently.
Additional Project Information
DFG classification: 409-05 Interactive and Intelligent Systems, Image and Language Processing, Computer Graphics and Visualisation
Software: Python3, JAX
Cluster: CLAIX
Publications
Julian Dierkes, Emma Cramer, Holger H. Hoos, Sebastian Trimpe,
Combining Automated Optimisation of Hyperparameters and Reward Shape,
Peer-reviewed publication, 2024
Julian Dierkes, Theresa Eimer, Marius Lindauer, Holger Hoos,
Performance Prediction In Reinforcement Learning: The Bad And The Ugly,
Peer-reviewed publication, October 2025
Jannis Becktepe, Julian Dierkes, Carolin Benjamins, Aditya Mohan, David Salinas, Raghu Rajan, Frank Hutter, Holger Hoos, Marius Lindauer, Theresa Eimer,
ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning,
Peer-reviewed publication, March 2026