[Question] How to reproduce the topic-wise evaluation results (Tech, Sports, etc.) reported in the paper?

[Suzukitakeko]

Hi, thanks for the impressive work on WorldSense!

I am currently trying to reproduce the evaluation results reported in the paper. Specifically, I am looking to get the breakdown by Topics (e.g., Tech & Science, Culture & Politics, Games, Sports, etc.) as shown in the table below:

However, when I run the evaluation using VLMEvalKit, the output provides metrics based on Task Categories (e.g., Spatial Relation, Event Recognition) and Task Domains (Understanding, Reasoning, Recognition). The log output looks like this:

I have two specific questions regarding this:

1. Metric Aggregation: Is there a specific script or argument provided in this repository (or VLMEvalKit) that maps these "Task Categories" into the "Topic Categories" shown in the paper? Or do I need to calculate them manually based on specific video IDs?
2. Dataset Source Discrepancy: I noticed a difference in the dataset sources:

• VLMEvalKit seems to load from: https://huggingface.co/datasets/lmms-lab/WorldSense
• The README in this repository points to: https://huggingface.co/datasets/honglyhly/WorldSense
  Does the honglyhly/WorldSense dataset contain specific metadata (topic tags) that might be missing or handled differently in the lmms-lab version?

Any guidance on how to obtain the topic-wise results would be greatly appreciated. Thanks!

[Suzukitakeko]

I managed to solve the issue.

The dataset provided in lmms-lab/WorldSense stores the topic information (e.g., "Tech & Science", "Sports") in the domain column. However, the current code in tasks/worldsense/utils.py was reading from task_domain, which contains different categories (Understanding/Reasoning/etc.).

Solution:
Modify worldsense_process_results in lmms_eval/tasks/worldsense/utils.py:

def worldsense_process_results(doc, results):
    # ...
    category = doc["task_type"]
    
    # OLD: domain = doc["task_domain"]
    # NEW: Try to get 'domain' (Topic) first, fallback to 'task_domain'
    domain = doc.get("domain", doc.get("task_domain", "Unknown"))
    
    # ...

[JaaackHongggg]

Sounds good.

The worldsense evaluation in llms_eval was implemented by the llms_eval team, and we appreciate their contributions. The worldsense evaluation code in VLMEvalKit was submitted by our team; we implemented fine-grained classification statistics for different video analogies, which can be accessed via the link provided here https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/utils/worldsense.py.

By the way, the data used in VLMEvalKit is sourced from https://huggingface.co/datasets/honglyhly/WorldSense instead of https://huggingface.co/datasets/lmms-lab/WorldSense. The dataset at https://huggingface.co/datasets/honglyhly/WorldSense is the official raw data provided by our team.