Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: independent sac #1013

Merged
merged 111 commits into from
Mar 12, 2024
Merged

feat: independent sac #1013

merged 111 commits into from
Mar 12, 2024

Conversation

sash-a
Copy link
Contributor

@sash-a sash-a commented Feb 12, 2024

What?

Feed Forward Independent Soft Actor Critic implementation

Extra

Will make a smaller PR for the changes around SAC so that the only thing I'm changing in this PR is adding SAC. Coming soon...

sash-a and others added 30 commits November 21, 2023 17:11
@sash-a
feat: isac acting
5df6e06
@sash-a
chore: add flashbax to requirements
557b213
@sash-a
feat: add actor and uncomment components of loss
9aada83
@sash-a
feat: more progress on learner code
01fd55a
@sash-a
feat: isac piping through
0c2c5e6
@sash-a
feat: evaluation and discrete SAC
03e5bfd
@OmaymaMahjoub
feat: get loss and metrics as output from the learner + fix the nan i…
8f6fa27 
…ssue
@OmaymaMahjoub
Merge branch 'develop' into feat/anakin-sac
4b6315a
@OmaymaMahjoub
feat: add logger to the system
14a0fc7
@OmaymaMahjoub
feat: make config file for sac
3276599
@OmaymaMahjoub
feat: remove eval function from isac file
ffef471
@OmaymaMahjoub
fix: fix the alpha loss fn
105248a
@OmaymaMahjoub
fix: minor fix to the alpha loss
44618dc
@sash-a @OmaymaMahjoub
feat: low rollout length
dfa195e 
Co-authored-by: Omayma Mahjoub <[email protected]>
@sash-a
feat: sac working on mamujoco
5a8fef9
@sash-a
chore: merge develop
3e6bef2
@sash-a
feat: starting to convert pytorch to jax
0336653
@sash-a
Merge branch 'develop' into feat/anakin-sac
cf110dc
@sash-a
feat: sac mostly working
38d04db
@sash-a
feat: remove the multiple policy updates for now
ff5eab5
@sash-a
feat: alpha per agent
7ebbaac
@sash-a
fix: alpha
b75b6a3
@sash-a
feat: mabrax + gym wrapper
74f2010
@sash-a
feat: mabrax->jumanji->gym working
ab8e7e3
@sash-a
fix: only pass agents view out for obs
47c02bc
@sash-a
feat: vmap wrapper
e237bae
@sash-a
Merge branch 'develop' into feat/jax-sac
67222a8
@sash-a
Merge branch 'feat/order-agnostic-wrappers' into feat/jax-sac
e792f31
@sash-a
chore: starting to jit jax sac
548865b
@sash-a
feat: jitted sac running
8a3d609
OmaymaMahjoub
OmaymaMahjoub requested changes Mar 11, 2024
View reviewed changes
Copy link
Contributor

@OmaymaMahjoub OmaymaMahjoub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sash-a keen to see isac part of Mava 🔥 The system training fine however it would be great if we could also work on aligning it closely with PPO systems for better consistency.

mava/configs/system/sac/ff_isac.yaml Show resolved Hide resolved
mava/systems/sac/types.py Outdated Show resolved Hide resolved
mava/systems/sac/ff_isac.py Show resolved Hide resolved
mava/configs/system/sac/ff_isac.yaml Outdated Show resolved Hide resolved
mava/configs/system/sac/ff_isac.yaml Outdated Show resolved Hide resolved
mava/systems/sac/ff_isac.py Outdated Show resolved Hide resolved
mava/systems/sac/ff_isac.py Show resolved Hide resolved
RuanJohn
RuanJohn reviewed Mar 11, 2024
View reviewed changes
Copy link
Collaborator

@RuanJohn RuanJohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really super work here @sash-a!

A first pass through and just a few minor suggestions and comments from me. Will do another pass a bit later today.

mava/configs/system/sac/ff_isac.yaml Outdated Show resolved Hide resolved
mava/configs/system/sac/ff_isac.yaml Outdated Show resolved Hide resolved
mava/networks.py Show resolved Hide resolved
mava/networks.py Show resolved Hide resolved
mava/systems/sac/types.py Show resolved Hide resolved
mava/systems/sac/ff_isac.py Show resolved Hide resolved
mava/systems/sac/ff_isac.py Show resolved Hide resolved
mava/systems/sac/ff_isac.py Outdated Show resolved Hide resolved
mava/systems/sac/ff_isac.py Outdated Show resolved Hide resolved
mava/systems/sac/ff_isac.py Outdated Show resolved Hide resolved
RuanJohn
RuanJohn reviewed Mar 11, 2024
View reviewed changes
Copy link
Collaborator

@RuanJohn RuanJohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few other minor comments. Thanks @sash-a 🔥

mava/systems/sac/types.py Outdated Show resolved Hide resolved
mava/systems/sac/types.py Outdated Show resolved Hide resolved
mava/systems/sac/ff_isac.py Outdated Show resolved Hide resolved
mava/systems/sac/ff_isac.py Outdated Show resolved Hide resolved
mava/systems/sac/ff_isac.py Outdated Show resolved Hide resolved
mava/systems/sac/ff_isac.py Show resolved Hide resolved
mava/systems/sac/ff_isac.py Outdated Show resolved Hide resolved
mava/systems/sac/ff_isac.py Show resolved Hide resolved
mava/systems/sac/ff_isac.py Show resolved Hide resolved
RuanJohn
RuanJohn reviewed Mar 12, 2024
View reviewed changes
Copy link
Collaborator

@RuanJohn RuanJohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @sash-a! Vert close! Just some very small final comments from me.

mava/configs/logger/ff_isac.yaml Outdated Show resolved Hide resolved
mava/configs/system/sac/ff_isac.yaml Show resolved Hide resolved
mava/networks.py Show resolved Hide resolved
mava/systems/sac/ff_isac.py Show resolved Hide resolved
mava/systems/sac/ff_isac.py Outdated Show resolved Hide resolved
mava/systems/sac/ff_isac.py Show resolved Hide resolved
WiemKhlifi
WiemKhlifi reviewed Mar 12, 2024
View reviewed changes
mava/configs/system/sac/ff_isac.yaml Outdated Show resolved Hide resolved
mava/networks.py Outdated Show resolved Hide resolved
mava/networks.py Show resolved Hide resolved
OmaymaMahjoub
OmaymaMahjoub approved these changes Mar 12, 2024
View reviewed changes
Copy link
Contributor

@OmaymaMahjoub OmaymaMahjoub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sash-a 🔥

RuanJohn
RuanJohn approved these changes Mar 12, 2024
View reviewed changes
Copy link
Collaborator

@RuanJohn RuanJohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚢 🔥

@sash-a sash-a merged commit 481a076 into develop Mar 12, 2024
3 checks passed
@sash-a sash-a deleted the feat/jitting-jax-sac branch March 12, 2024 10:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WiemKhlifi WiemKhlifi WiemKhlifi left review comments

@lbeyers lbeyers lbeyers left review comments

@OmaymaMahjoub OmaymaMahjoub OmaymaMahjoub approved these changes

@RuanJohn RuanJohn RuanJohn approved these changes

@arnupretorius arnupretorius Awaiting requested review from arnupretorius arnupretorius is a code owner

@DriesSmit DriesSmit Awaiting requested review from DriesSmit DriesSmit is a code owner

@jcformanek jcformanek Awaiting requested review from jcformanek jcformanek is a code owner

@siddarthsingh1 siddarthsingh1 Awaiting requested review from siddarthsingh1 siddarthsingh1 is a code owner

@ulricharmel ulricharmel Awaiting requested review from ulricharmel ulricharmel is a code owner

@callumtilbury callumtilbury Awaiting requested review from callumtilbury

Assignees

@sash-a sash-a

Labels
size/XL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@sash-a @lbeyers @RuanJohn @OmaymaMahjoub @WiemKhlifi