Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add SFR-Embedding-Code-2B_R #2019

Closed
Muennighoff opened this issue Feb 7, 2025 · 5 comments
Closed

Add SFR-Embedding-Code-2B_R #2019

Muennighoff opened this issue Feb 7, 2025 · 5 comments
Labels
good first issue Good for newcomers new model Questions related to adding a new model to the benchmark

Comments

@Muennighoff
Copy link
Contributor

No description provided.

@isaac-chung isaac-chung added good first issue Good for newcomers new model Questions related to adding a new model to the benchmark labels Feb 10, 2025
@ayush1298
Copy link
Contributor

ayush1298 commented Feb 11, 2025 •
edited
Loading

@Muennighoff, I have tried adding meta information for SFR-Embedding-Code-2B_R and tested it for the STS12 task which works perfectly. However, I am not able to evaluate it on all tasks because it takes a very long time even for single tasks on my local. I tried running it on Google Colab but got a Cuda out-of-memory error there though running tasks in batches of 3. So, what is the best way to run the model on all tasks and get the results?

@KennethEnevoldsen
Copy link
Contributor

Doing a PR with just the model meta and working implementation is perfectly fine. Then you can also get a review on potential implementation issues.

ayush1298 added a commit to ayush1298/mteb that referenced this issue Feb 11, 2025
@ayush1298
Add SFR-Embedding-Code-2B_R(embeddings-benchmark#2019)
9fcf1a2
@ayush1298 ayush1298 mentioned this issue Feb 11, 2025
Merged
7 tasks
@ayush1298
Copy link
Contributor

ayush1298 commented Feb 12, 2025 •
edited
Loading

@KennethEnevoldsen ,#2032 is the PR for the Addition of the SFR-Embedding-Code-2B_R model, It was approved, so I am merging it. Can you suggest a solution for the issue related to the generation of results that I mentioned above?

Samoed pushed a commit that referenced this issue Feb 12, 2025
@ayush1298
Add SFR-Embedding-Code-2B_R (#2032)
0caea67 
* Add SFR-Embedding-Code-2B_R(#2019)

* make lint

* Added HF repo name for adapted_from parameter

* replace instruct_wrapper
@Samoed Samoed closed this as completed Feb 12, 2025
@ayush1298
Copy link
Contributor

ayush1298 commented Feb 12, 2025 •
edited
Loading

@Samoed, Isn't evaluation of model on all tasks and adding results to the leaderboard also part of this issue? or will that issue be in the https://github.com/embeddings-benchmark/results repo?

@Samoed
Copy link
Collaborator

Samoed commented Feb 12, 2025

You can just submit them later

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
good first issue Good for newcomers new model Questions related to adding a new model to the benchmark
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@KennethEnevoldsen @Samoed @isaac-chung @Muennighoff @ayush1298