Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Should Precision/Recall be added to the leaderboard? #46

Open
patel-zeel opened this issue Oct 7, 2024 · 3 comments
Open

Should Precision/Recall be added to the leaderboard? #46

patel-zeel opened this issue Oct 7, 2024 · 3 comments
Labels
enhancement New feature or request

Comments

@patel-zeel
Copy link

patel-zeel commented Oct 7, 2024

Hi,

In most papers, we only see mAP, which is a good aggregate metric for the comparison over the horizon, but it fails to provide further insights. The inclusion of F1 in the leaderboard is a great step! If we go one step further, between precision and recall, specific applications may prefer one over the other. If we are searching for an important object from satellite imagery, we'd mostly not mind losing the precision to gain high recall. For example, searching for objects in the vicinity of the national borders which are of national security concern.

If we include Precision/Recall in the leaderboard, we can potentially answer the following and similar questions:

  • For the same recall, which model provides better precision?
  • Which model provides the highest recall at a specified precision threshold? For example, we may like to bind the precision to be > 0.5.
@onuralpszr
Copy link
Collaborator

@patel-zeel hello, Precision already there and I was writing on mAR and hopefully I will run them all again to add mAR as well :)

@onuralpszr onuralpszr added the enhancement New feature or request label Oct 7, 2024
@patel-zeel
Copy link
Author

patel-zeel commented Oct 7, 2024

Interesting, @onuralpszr! Could you please guide me to the supervision code which computes mAR?

@onuralpszr
Copy link
Collaborator

Interesting, @onuralpszr! Could you please guide me to the supervision code which computes mAR?

I didn't open PR yet, I can cc you when I did ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants