[Question] Auto-regressive manner policy network?

[wadmes]

❓ Question

Hi there,

I am trying to implement a autoregressive policy network, for example,
in each action, I need to sample target object, and then use the target object embedding to sample the action type, and finally output the action parameters.

But it looks it is impossible to achive this since what _build_mlp_extractor in MaskableActorCriticPolicy asks it to return a action embedding and value embedding.

Is there a good way to achieve the feature?

Example figure:

Thanks!

Checklist

• I have checked that there is no similar issue in the repo
• I have read the documentation
• If code there is, it is minimal and working
• If code there is, it is formatted using the markdown code blocks for both code and stack traces.

[araffin]

Is there a good way to achieve the feature?

For something custom like that, it's probably better to fork SB3 / use CleanRL as a starting point.