Skip to content

Conversation

@wantbook-book
Copy link

Description

Summary:

  • Add a logging compatibility layer to support SwanLab or W&B backends and disable Artifact/Histogram logging when using SwanLab.
  • Add a minimal SwanLab logging test entry point and pytest guard to validate a tiny StandardSAE run.

Motivation:

  • SwanLab does not support W&B Artifacts/Histogram objects, which caused logging failures. The change routes logging through a backend selector and conditionally disables unsupported features while preserving scalar logs and training flow.

Dependencies:

  • swanlab (when SAE_LENS_LOGGING_BACKEND=swanlab)

Fixes #604

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist:

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

You have tested formatting, typing and tests

  • I have run make check-ci to check format and linting. (you can run make format to format code if needed.)

Performance Check.

If you have implemented a training change, please indicate precisely how performance changes with respect to the following metrics:

  • L0
  • CE Loss
  • MSE Loss
  • Feature Dashboard Interpretability

Please links to wandb dashboards with a control and test group.

@wantbook-book wantbook-book changed the title Feature log backend featFeature log backend Dec 25, 2025
@wantbook-book wantbook-book changed the title featFeature log backend feat: Add Swanlab logging backend Dec 25, 2025
@wantbook-book
Copy link
Author

  1. CI is failing because the workflow attempts to request an OIDC token, but permissions: id-token: write isn’t enabled for the job/workflow. Since this PR is from a fork, token/permission restrictions may also apply.

  2. For linting, I ran poetry run ruff check --fix tests/_comparison/sae_lens_old locally and got All checks passed!.

Copy link
Collaborator

@chanind chanind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for contributing this! However, I'm inclined to hold off on merging this until the next major release of SAELens. Ideally we should have a generic "Logger" interface, and then the user should specify the loggers they want in the config. That should make it possible for users to register their own loggers if they want as well. I would envision something like the following:

cfg = LanguageModelSAERunnerConfig(
  ...
  logger=[
      WandBLoggingConfig(...),
      SwanLabLoggingConfig(...),
  ],
)

This would be a breaking change though so we'd need to de a major release, and requires a bit more refactoring than is ideal in just this PR.

if backend not in {"auto", "wandb", "swanlab"}:
backend = "auto"

if backend in {"auto", "swanlab"}:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather the default be wandb, not swanlab. If a user just happens to have SAELens and swanlab deps in the same project their logging will seem to break and this will be very hard to debug. IMO if users want to use swanlab they should need to opt-in specifically with a ENV var.

)
for key, value in self.sae.log_histograms().items():
eval_metrics[key] = wandb.Histogram(value) # type: ignore
if BACKEND != "swanlab":
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather not need to have if-statements like this in code. Can you provide an interface so swanlab works identical to wandb, and we don't need any of these checks? Or maybe make a generic Logger interface and have both Swanlab and WandB versions of that same interface? It shouldn't be the responsibility of the SAETrainer to keep track of what logger is being used.

run_minimal_standard_sae_training(output_dir)


if __name__ == "__main__":
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: don't use main functions in tests. Just run the test using pytest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Proposal] Support SwanLab as an alternative logging backend to wandb

2 participants