Organize project repo

[deanwampler]

Our main repo: https://github.com/The-AI-Alliance/trust-safety-evals

We need to have everything nicely organized in the above. Currently things are all over the place:

BenchBench forked here: https://github.com/The-AI-Alliance/tse-ibm-benchbench

HF repos:
SafetyBAT: https://huggingface.co/spaces/aialliance/safetybat
SafetyArena (leaderboard): https://huggingface.co/spaces/aialliance/safetyarena
SafetyArena (backend): https://huggingface.co/spaces/aialliance/safetyarena-backend

[deanwampler]

For the HF repos, we can set up forks to copies in the Alliance.

For these repos and for BenchBench, it feels to me like these are standalone projects for deploying tools that might dependencies on https://github.com/The-AI-Alliance/trust-safety-evals, but don't need to live within the repo. True?

When it's better to have the content of one repo nested within another, we should consider using submodules first, then consider if that's sufficient or we should just move the contents into the other repo.

[deanwampler]

@bnayahu Thoughts on this? I take it https://github.com/The-AI-Alliance/tse-ibm-benchbench is repo for what's running on HF. What else is deployed to HF that isn't in this repo?

I see a few things we need to do offhand to better engineer all this:

• Document all the repos where code is kept, including what's used to deploy to HF.
• Document operational procedures, e.g., what do to when something has stopped running. (How can we prevent that or get alerted immediately?)

[bnayahu]

We also need to decide on how we govern our code, our dependencies and the lifecycle.

Lets consider SafetyBAT as an example: the "production" environment is a HF space, which is essentially a repo they host (https://huggingface.co/spaces/aialliance/safetybat/tree/main). This space is a clone of the IBM BenchBench space (https://huggingface.co/spaces/ibm/benchbench/tree/main), with some small modifications. The BenchBench space uses the BenchBench library (https://github.com/ibm/benchbench).

The way it SHOULD work (I think): Both IBM-owned repos should be forked into the AIA org (currently the BB library is forked into tse-ibm-benchbench), and serve as our baseline. Any changes or enhancements we need should be done in these forks, potentially (where it makes sense) contributed back to the origins. The HF repo should be configured as an additional remote for our repo, so there's a single source for the code. If this makes sense conceptually, and if it can be done technically, lets do it.

[deanwampler]

That makes sense to me. I see from the settings for safetybat that it is associated with the benchbench repo and it appears there are updates to the latter that could be merged. What's not clear to me is how to automate things like this, but we could at least document the process for manual steps. I also don't see a way to move the upstream repo from benchbench to tse-ibm-benchmark, but it might be doable.

Something else to manage, I see in the README editing view that the streamlit dependency can be upgraded. https://huggingface.co/spaces/aialliance/safetybat/edit/main/README.md