Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Leaderboard: Add more explainers #1986

Open
Muennighoff opened this issue Feb 5, 2025 · 5 comments · May be fixed by #2039
Open

Leaderboard: Add more explainers #1986

Muennighoff opened this issue Feb 5, 2025 · 5 comments · May be fixed by #2039
Assignees
Labels
leaderboard issues related to the leaderboard

Comments

@Muennighoff
Copy link
Contributor

Similar to https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/ could be via hovering; Things to explain imo:

  • Max Tokens
  • Embedding Dimension
  • Number of Parameters

Let's make the LB maximally accessible

@isaac-chung isaac-chung added the leaderboard issues related to the leaderboard label Feb 6, 2025
@KennethEnevoldsen
Copy link
Contributor

I completely agree. We have already looked into this but haven't found it within gradio. We could add an FAQ box at the bottom, but I don't think it is needed for these features (we e.g. have it for zero-shot)

@KennethEnevoldsen
Copy link
Contributor

will close this, but if there is potential solutions please do re-open.

@Muennighoff
Copy link
Contributor Author

How about an accordion sth like the below next to zero-shot, borda?

More information

Number of Parameters: This is the total number of parameters in the model including embedding parameters. A higher value means the model requires more CPU/GPU memory to run; thus, less is generally desirable.
Embedding Dimension: This is the vector dimension of the embeddings that the model produces. When saving embeddings to disk, a higher dimension will require more space, thus less is usually desirable.
Max tokens: This refers to how many tokens (=word pieces) the model can process. Generally, a larger value is desirable.

@KennethEnevoldsen
Copy link
Contributor

Would be happy with that - potentially rephrased slightly?

What do the columns mean?

  • Number of Parameters: This is the total number of parameters in the model including embedding parameters. A higher value means the model requires more CPU/GPU memory to run; thus, less is generally desirable.
  • Embedding Dimension: This is the vector dimension of the embeddings that the model produces. When saving embeddings to disk, a higher dimension will require more space, thus less is usually desirable.
  • Max tokens: This refers to how many tokens (=word pieces) the model can process. Generally, a larger value is desirable.
  • Zero-shot: ...

@Muennighoff
Copy link
Contributor Author

Yeah that looks great!!

@KennethEnevoldsen KennethEnevoldsen self-assigned this Feb 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
leaderboard issues related to the leaderboard
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants