Support for Hierarchical RL

[muhd-umer]

Hi! 👋🏻

I'm really impressed by how quickly the rl/ repo has grown. I was wondering if hierarchical reinforcement learning (HRL) is currently supported or can be implemented using existing modules in version 0.5.0.

I did note that HRL is on the roadmap:

...
Our roadmap to get there comprises:
#
# - Distributed solutions
# - Offline RL
# - Greater support for meta-RL
# - Multi-task and hierarchical RL
...

If HRL is under development, I'd be happy to contribute if I have the time.

Thanks,
Umer 💓

[vmoens]

Hey! Would love to see more HRL in the lib indeed!
Can you list the features you'd like to see?

[muhd-umer]

Hi there!

I've been digging into some recent HRL papers, and there are a few key features I think would be awesome to see:

1. Support for temporal abstraction → This could allow defining higher-level policies that operate over extended time scales while low-level policies handle immediate actions

2. Multi-level critics → Support for critics that operate at different levels of temporal abstraction, like in h-DQN impl.

3. Options → A way to define and learn reusable skills/subtasks that can be executed by the agent. Maybe it can also be extended into flexible goal spaces and goal-conditioned policies?

4. Policy architectures → Something like a meta-controller that selects subgoals for lower-level controllers

5. Intrinsic motivation → These could help with exploration at different levels of the hierarchy, i.e., the curiosity-driven approach in Kulkarni's paper

And of course, since torchrl has both low- and high-level abstractions, having reference implementations of algorithms like HIRO, HAC, or the Option-Critic architecture would be super helpful!

I know that's a lot and a bit vague, but even starting with good abstractions for options and goal-conditioned policies would be an awesome foundation to build on.

I'm pretty new to contributing to open-source projects, but I'm eager to help out however I can. Thanks again for being so open to suggestions.