Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v2] Merge MIEB into v2 #1973

Open
wants to merge 322 commits into
base: v2.0.0
Choose a base branch
from
Open

[v2] Merge MIEB into v2 #1973

wants to merge 322 commits into from
+24,033 −1,366

Conversation

Samoed
Copy link
Collaborator

@Samoed Samoed commented Feb 5, 2025 •
edited
Loading

Code Quality

  • Code Formatted: Format the code using make lint to maintain consistent style.

Documentation

  • Updated Documentation: Add or update documentation to reflect the changes introduced in this PR.

Testing

  • New Tests Added: Write tests to cover new functionality. Validate with make test-with-coverage.
  • Tests Passed: Run tests locally using make test or make test-with-coverage to ensure no existing functionality is broken.

github-actions bot and others added 30 commits January 30, 2025 21:34
@github-actions
Update tasks table
933f4af
@github-actions
Update tasks table
67f8a79
@github-actions
Update tasks table
1b76261
@github-actions
Update tasks table
51faf65
@github-actions
Update tasks table
f7438b8
@github-actions
Update tasks table
3123d1c
@github-actions
Update tasks table
0bbc4c7
@github-actions
Update tasks table
c46cb8b
@github-actions
Update tasks table
21b60f5
@github-actions
Update tasks table
d9ab239
@github-actions
Update tasks table
9a6275e
@github-actions
Update tasks table
e35c8dd
@github-actions
Update tasks table
d510ddb
@github-actions
Update tasks table
4bb4ec6
@github-actions
Update tasks table
9076213
@github-actions
Update tasks table
12ad5bd
1.31.6
6df0b8c 
Automatically generated by python-semantic-release
@github-actions
Update tasks table
490b59c
@github-actions
Update tasks table
93d631f
@github-actions
Update tasks table
257578c
@github-actions
Update tasks table
1275f93
@github-actions
Update tasks table
47c63c3
@github-actions
Update tasks table
e29348c
@github-actions
Update tasks table
253a499
@github-actions
Update tasks table
4d30059
@github-actions
Update tasks table
0a073df
@github-actions
Update tasks table
58d5248
@github-actions
Update tasks table
1faa897
@github-actions
Update tasks table
f539d9a
@github-actions
Update tasks table
803e973
@Samoed Samoed force-pushed the merge_mieb_into_v2 branch from 5e780fd to ce8f057 Compare February 5, 2025 15:44
Samoed and others added 7 commits February 5, 2025 18:44
@Samoed
add descriptive stats
ce8f057
@Samoed
remove is_multilingual
9047eb2
@KennethEnevoldsen
fix: leaderboard and benchmark.py inconstiencies (#1956)
64c17b6 
* fix: fix task types in MIEB

* fix: French leaderboard inconsistencies

fixes #1919

* Fixes  #1917
@Samoed
remove multisubset loader
65fc0cc
1.34.2
adc06ab 
Automatically generated by python-semantic-release
@Ressnn @Samoed
Add mini-gte model (#1906)
1520979 
* add mini-gte model

* fix formatting

* Update mteb/models/qtack_models.py

add adapted from metadata

Co-authored-by: Roman Solomatin <[email protected]>

* add datasets to mini-gte

* linting for qtack_models.py

---------

Co-authored-by: Roman Solomatin <[email protected]>
@SamuelYang1
Fix HagridRetrieval load_dataset "trust_remote_code" (#1989)
9618505
sergeyz-zh and others added 14 commits February 6, 2025 11:19
@sergeyz-zh
fix: Meta information ru_sentence_models (#1991)
370b26c 
cointegrated/rubert-tiny
sergeyzh/rubert-tiny-turbo
sergeyzh/LaBSE-ru-turbo
1.34.3
56f90a7 
Automatically generated by python-semantic-release
@KennethEnevoldsen
fix: Added description and resolved bug in rangeslider (#1990)
8583383 
* fix: Added description and resolved bug in rangeslider

addresses #1987
fixes #1954

* format

* remove certificate
1.34.4
4a37967 
Automatically generated by python-semantic-release
@KennethEnevoldsen
CI clean up: Remove MMTEB points check (#1994)
fb6bade 
Fixes #1992
@afalf @Samoed
Add gte-modernbert-base (#1845)
e2c44ed 
* Update gte_models.py

* Update gte_models train data

* Update gte_models.py

* Update gte_models.py

* Update gte_models.py

* Update gte_models.py

* Update mteb/models/gte_models.py

* Update gte_models.py

* Update gte_models.py

* Update gte_models.py

---------

Co-authored-by: Roman Solomatin <[email protected]>
@Muennighoff @KennethEnevoldsen
fix: training data for gritlm (#1932)
d810e4e 
* Fix training data

* Fix data

* Fix data

---------

Co-authored-by: Kenneth Enevoldsen <[email protected]>
@KennethEnevoldsen
Sort benchmarks and add temp. reference to old leaderboard (#1993)
3887d83 
* fix: Added description and resolved bug in rangeslider

addresses #1987
fixes #1954

* format

* remove certificate

* Sorted benchmarks #1984
1.34.5
d424037 
Automatically generated by python-semantic-release
@Muennighoff
fix: Update faq of on leaderboard (#2004)
4fe4c99 
Simplify
1.34.6
7ac4650 
Automatically generated by python-semantic-release
@nikolay-banar
fix: BEIR-NL metadata mistake (#2010)
b1ac052 
Update SCIDOCSNLRetrieval.py
@github-actions
Update tasks table
64f256e
1.34.7
eacde2f 
Automatically generated by python-semantic-release
@Samoed Samoed changed the title Merge MIEB into v2 [v2] Merge MIEB into v2 Feb 7, 2025
Samoed and others added 5 commits February 7, 2025 16:26
@Samoed
add model memory usage (#1934)
e46539a 
* add model memory usage

* lint

* update

* calculate memory usage based on file size

* calculate memory usage

* add memory usage for MIEB models

* add last model usage

* add memory_usage_mb to overview

* fix rerank

* update memory usage

* update memory usage
@Samoed
Merge branch 'refs/heads/main' into merge_mieb_into_v2
612626d 
# Conflicts:
#	mteb/model_meta.py
#	mteb/models/gritlm_models.py
#	mteb/models/overview.py
#	mteb/models/promptriever_models.py
#	mteb/tasks/Retrieval/eng/HagridRetrieval.py
@Samoed
Merge branch 'refs/heads/v2.0.0' into merge_mieb_into_v2
09b3c5a 
# Conflicts:
#	tests/test_tasks/test_all_abstasks.py
@Samoed
Merge branch 'refs/heads/v2.0.0' into merge_mieb_into_v2
f1d0268 
# Conflicts:
#	mteb/benchmarks/benchmarks.py
#	mteb/tasks/__init__.py
#	mteb/tasks/aggregated_tasks/__init__.py
@Samoed
fix metas
4187c57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@isaac-chung isaac-chung Awaiting requested review from isaac-chung

@gowitheflow-1998 gowitheflow-1998 Awaiting requested review from gowitheflow-1998

Assignees
No one assigned
Labels
v2 Issues and PRs related to `v2` branch
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.