Clarification needed on the calculation method for the "Avg." column in Table 1 and Table 2

[wtma1999]

Thank you for your excellent work and for sharing this insightful paper.

We are currently analyzing the results in Table 1 (and potentially Table 2) and had a quick question regarding the "Avg." (Average) column.

We've observed that the reported average values do not always seem to correspond to a simple arithmetic mean of all the individual task scores. We also noted that the evaluate_agent_results.py script calculates metrics for individual tasks but doesn't seem to provide a final, overall "Avg." for the entire dataset as shown in the tables.

This leads us to believe there might be a more specific calculation method being used (perhaps a weighted average, or an average of the main category scores like AR, TTL, LRU).

Could you please clarify the exact methodology used to compute the "Avg." column for these tables?

Thank you for your time and clarification!

[wangyu-ustc]

Hi thank you for pointing this out! "Avg." should be the average across all datasets. And you are right, there are some miscalculations in the paper (sorry about this!) The following are the corrected tables:

Method	Metric	SQuAD	AR HotpotQA	PerLTQA	TREC-C	TTL NLU	Pubmed	LRU BookSum	Avg.
Long-Context	Perf.	0.742	0.852	0.605	0.623	0.708	0.533	0.052	0.588
	Mem.	10.6K	9.7K	13.1K	3.9K	6.1K	16.7K	15.4K	10.8K
RAG-Top2	Perf.	0.762	0.849	0.623	0.612	0.508	0.570	0.042	0.567
	Mem.	10.6K	9.7K	16.7K	3.9K	6.1K	16.7K	15.6K	11.3K
MemAgent	Perf.	0.091	0.140	0.052	0.562	0.290	0.343	0.103	0.226
	Mem.	0.79K	0.76K	0.29K	1.24K	0.99K	0.94K	0.59K	0.84K
MEM1	Perf.	0.039	0.083	0.068	0.269	0.056	0.175	0.085	0.111
	Mem.	0.16K	0.22K	0.14K	0.23K	0.22K	0.08K	0.16K	0.17K
Mem-α	Perf.	0.786	0.832	0.659	0.666	0.658	0.545	0.187	0.619
	Mem.	10.1K	8.7K	11.2K	4.0K	6.5K	12.3K	2.2K	7.9K

Table 1: Performance and total number of tokens in memory across validation datasets.
Perf.: task-specific metrics (F1/Accuracy), Mem.: memory in thousands of tokens.
AR: Accurate Retrieval, TTL: Time Time Learning, LRU: Long Range Understanding.

Method	Metric	Single-Doc	Multi-Doc	LME(S)	TREC-C	NLU	TREC-F	Clinic	Banking77	LRU InfBench	Avg.
Long-Context	Perf.	0.280	0.270	0.292	0.640	0.740	0.340	0.860	0.770	0.125	0.480
	Mem.	33K	33K	33K	33K	33K	33K	33K	33K	33K	33K
RAG-Top2	Perf.	0.690	0.450	0.581	0.690	0.650	0.210	0.700	0.750	0.065	0.532
	Mem.	217K	474K	348K	124K	134K	126K	131K	128K	181K	207K
MemAgent	Perf.	0.070	0.160	0.050	0.370	0.260	0.210	0.250	0.370	0.043	0.198
	Mem.	1.02K	1.02K	0.56K	1.02K	1.02K	0.77K	1.02K	1.02K	0.73K	0.92K
MEM1	Perf.	0.070	0.180	0.090	0.180	0.000	0.000	0.090	0.000	0.029	0.071
	Mem.	0.30K	0.38K	0.22K	0.16K	0.11K	0.13K	0.28K	0.11K	0.19K	0.21K
Mem-α-4B	Perf.	0.740	0.680	0.520	0.710	0.710	0.410	0.730	0.700	0.129	0.592
	Mem.	160K	323K	127K	120K	142K	123K	18K	133K	19K	129K

Table 2: Performance and the total number of tokens in the memory on MemoryAgentBench.
Perf.: task-specific metrics (F1/Accuracy), Mem.: memory in thousands of tokens.

We will update these numbers in our paper in the next version, and thank you again for pointing this out!

[wtma1999]

Thank you for your quick response and for clarifying the point regarding the "Avg." column. I really appreciate your detailed explanation. I have been following this work closely, and it's an excellent contribution to the field. Thanks again for your attention to this, and I look forward to any further updates! Best regards, wtma Yu Wang ***@***.***> 于2025年10月29日周三 22:39写道：

…

*wangyu-ustc* left a comment (wangyu-ustc/Mem-alpha#4) <#4 (comment)> Hi thank you for pointing this out! "Avg." should be the average across all datasets. And you are right, there are some miscalculations in the paper (sorry about this!) The following are the corrected tables: *Method* *Metric* *SQuAD* *AR HotpotQA* *PerLTQA* *TREC-C* *TTL NLU* *Pubmed* *LRU BookSum* *Avg.* *Long-Context* Perf. 0.742 *0.852* 0.605 0.623 *0.708* 0.533 0.052 0.588 Mem. 10.6K 9.7K 13.1K 3.9K 6.1K 16.7K 15.4K 10.8K *RAG-Top2* Perf. 0.762 0.849 0.623 0.612 0.508 *0.570* 0.042 0.567 Mem. 10.6K 9.7K 16.7K 3.9K 6.1K 16.7K 15.6K 11.3K *MemAgent* Perf. 0.091 0.140 0.052 0.562 0.290 0.343 0.103 0.226 Mem. 0.79K 0.76K 0.29K 1.24K 0.99K 0.94K 0.59K 0.84K *MEM1* Perf. 0.039 0.083 0.068 0.269 0.056 0.175 0.085 0.111 Mem. 0.16K 0.22K 0.14K 0.23K 0.22K 0.08K 0.16K 0.17K *Mem-α* Perf. *0.786* 0.832 *0.659* *0.666* 0.658 0.545 *0.187* *0.619* Mem. 10.1K 8.7K 11.2K 4.0K 6.5K 12.3K 2.2K 7.9K *Table 1:* Performance and total number of tokens in memory across validation datasets. *Perf.*: task-specific metrics (F1/Accuracy), *Mem.*: memory in thousands of tokens. *AR*: Accurate Retrieval, *TTL*: Time Time Learning, *LRU*: Long Range Understanding. Method Metric Single-Doc Multi-Doc LME(S) TREC-C NLU TREC-F Clinic Banking77 LRU InfBench Avg. *Long-Context* Perf. 0.280 0.270 0.292 0.640 *0.740* 0.340 *0.860* *0.770* 0.125 0.480 Mem. 33K 33K 33K 33K 33K 33K 33K 33K 33K 33K *RAG-Top2* Perf. 0.690 0.450 *0.581* 0.690 0.650 0.210 0.700 0.750 0.065 0.532 Mem. 217K 474K 348K 124K 134K 126K 131K 128K 181K 207K *MemAgent* Perf. 0.070 0.160 0.050 0.370 0.260 0.210 0.250 0.370 0.043 0.198 Mem. 1.02K 1.02K 0.56K 1.02K 1.02K 0.77K 1.02K 1.02K 0.73K 0.92K *MEM1* Perf. 0.070 0.180 0.090 0.180 0.000 0.000 0.090 0.000 0.029 0.071 Mem. 0.30K 0.38K 0.22K 0.16K 0.11K 0.13K 0.28K 0.11K 0.19K 0.21K *Mem-α-4B* Perf. *0.740* *0.680* 0.520 *0.710* 0.710 *0.410* 0.730 0.700 *0.129* *0.592* Mem. 160K 323K 127K 120K 142K 123K 18K 133K 19K 129K *Table 2:* Performance and the total number of tokens in the memory on MemoryAgentBench. *Perf.*: task-specific metrics (F1/Accuracy), *Mem.*: memory in thousands of tokens. We will update these numbers in our paper in the next version, and thank you again for pointing this out! — Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BBPZ4HQ2SJYKKCGUGDVL5WL32DGSDAVCNFSM6AAAAACKQTRMZSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTINRRHEYTGMBQHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[wtma1999]

Hello,

We are currently trying to reproduce the evaluation results for the mem-alpha-4b model using the provided model and dataset.

We observed that the evaluation code requires an external agent model. To meet this dependency, we have deployed qwen3-32b using vllm as the agent model.

Here is the vllm command we used to launch the server:

vllm serve $MODEL_PATH/Qwen3-32B \
    --served-model-name qwen3-32b \
    --api_key EMPTY \
    --trust_remote_code \
    --gpu_memory_utilization 0.9 \
    --tensor-parallel-size 4 \
    --max_model_len 32000 \
    --disable_log_requests  \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --host 127.0.0.1 \
    --port 8001

After running the evaluation, we found that our reproduced metrics do not align with those reported in the paper. We are seeing significant discrepancies in several key metrics.

Here is a comparison of the paper's results versus our own (we have bolded the metrics with the largest differences):

Memalpha Dataset	SQuAD	HotpotQA	PerLTQA	TREC-C	NLU	Pubmed	BookSum	Avg.
Mem-alpha(Repr)	0.740	0.806	0.631	0.256	0.22	0.296	0.202	0.450
Mem-alpha(papers)	0.786	0.832	0.659	0.666	0.658	0.545	0.187	0.619

MemoryAgentBench	Single-Doc	Multi-Doc	LME(S)	TREC-C	NLU	TREC-F	Clinic	Banking777	InfBench	Avg.
Mem-alpha(Repr)	0.7	0.63	0.393	0.84	0.81	0.46	0.81	0.88	0.131	0.628
Mem-alpha(papers)	0.74	0.68	0.52	0.71	0.71	0.41	0.73	0.7	0.129	0.592

After that, we conducted a "bad case" analysis on the memalpha dataset results.
We have identified a potential issue in the evaluate_agent_results.py script. https://github.com/wangyu-ustc/Mem-alpha/blob/e95873d5128bcd4132f7e98642ccf08f6f77dea8/evaluate_agent_results.py#L19
Specifically, when processing the TTL-type datasets, the _extract_answer_from_response function does not seem to properly handle cases where a response contains both the "<answers></answer>" tags and the \box{} or \boxed{} marker simultaneously.

To address this, we modified the _extract_answer_from_response function.

def _search_box_patterns(text: str):

    boxed_pattern = r'\\boxed\{([^}]*)\}'
    boxed_match = re.search(boxed_pattern, text)
    if boxed_match:
        return boxed_match.group(1).strip()

    box_pattern = r'\\box\{([^}]*)\}'
    box_match = re.search(box_pattern, text)
    if box_match:
        return box_match.group(1).strip()

    return None

def _extract_answer_from_response(predicted_answer: str):
    answer_pattern = r'<answer>(.*?)</answer>'
    answer_match = re.search(answer_pattern, predicted_answer, re.DOTALL | re.IGNORECASE)

    if answer_match:
        inner_content = answer_match.group(1).strip()
        
        box_result = _search_box_patterns(inner_content)
        
        if box_result is not None:
            return box_result
        else:
            return inner_content
    else:
        box_result = _search_box_patterns(predicted_answer)
        
        if box_result is not None:
            return box_result

    return predicted_answer.strip()

After applying this patch, the metrics for these specific cases were matched much more effectively, and our overall reproduced scores improved significantly.

Memalpha Dataset	SQuAD	HotpotQA	PerLTQA	TREC-C	NLU	Pubmed	BookSum	Avg.
Mem-alpha(Repr)	0.740	0.806	0.631	0.709	0.686	0.614	0.202	0.627
Mem-alpha(papers)	0.786	0.832	0.659	0.666	0.658	0.545	0.187	0.619

This leads us to two main questions:

Is our modification to _extract_answer_from_response appropriate? Does it reflect the intended logic for answer extraction in these ambiguous cases?

We are also concerned if this is a symptom of a different problem. Is it possible that our reproduction process (specifically, using the vllm-served qwen3-32b as an agent) is generating responses in this "dual-format" (e.g., ... \boxed{...} ...), whereas the agent setup used for the paper produced cleaner responses that the original script was designed to handle?

In short, we are unsure if we've fixed a bug in the evaluation script or if we've simply created a patch to handle an unexpected output format from our specific agent setup.

Any clarification on this would be greatly appreciated. Thank you!

[wtma1999]

Additionally, we also have tried to analyze the longmemeval results in MemoryAgentBench, but we have not yet found any obvious reasons for the discrepancy.

Our setup relies on the same vllm-served qwen3-32b model, which, for this benchmark, is used as the judge.

We are wondering if this might be the source of the problem.

1. Is it possible that the performance of qwen3-32b as a judge is insufficient or not calibrated for this task, leading to the scoring discrepancies?

2. Was a different model or setup used as the judge to obtain the results in the paper?

We would be very grateful if you could provide any insights into potential issues.

Thank you very much!

[wangyu-ustc]

Hi, thanks for the information! There are a few points here:

• Potential issues from my implementation: There does exist some ambiguities in my code. In the file process_data.py I am processing the dataset to ask the agent to put the answer in <answer></answer>, however, in the config file config/prompts_wrt_datasource.yaml I was asking the agent to put the answer in \boxed{}. I will update the dataset to make sure the label has to be put in \boxed{} and remove everything about <answer></answer>
• Despite the ambiguity, I don't understand why this is happening. In the file https://github.com/wangyu-ustc/Mem-alpha/blob/main/verl/workers/reward_manager/naive.py#L180 you can see if the model outputs something like "\boxed{5}" it will get zero reward, thus this is never encouraged in the training. This also didn't happen in my training process. Also I checked the code and couldn't find the causes. Can I also ask what does the training curve look like? Are the training rewards on the TTL datasets increasing all the way to around 0.7? I don't quite understand why it is outputting such kind of responses. Perhaps you can try rerun the experiments with the new datasets once I have finished processing them?
• Performance on LME(S): After more careful check, I guess the performance on LME(S) can be a little unstable (if you look at Table 9, when $\beta=0.2$ it is 0.367 for no reason, worse than $\beta=0.4$ and $\beta=0.05$, I didn't quite pay attention to this earlier but now it stands out to me), would you mind checking the performance of some other checkpoints on this dataset or rerun the evaluation on LME(S) to see what the performance would look like?

[wtma1999]

Hi, thanks for your detailed reply and clarifications!

The issue I mentioned actually occurred during test inference using your released model, rather than in our own training. We encountered the problem in that context and modified the _extract_answer_from_response function to work around it.

Regarding the training process, we’re currently working on reproducing it. If we run into any related issues, we’ll definitely report them.

For the performance on LME(S), we agree with your observation — since the judging is based on a large model, some fluctuations are expected. In our repeated experiments, we also observed scores around 0.49–0.50, which is consistent with your findings.

One remaining question we have is about the training time. You mentioned that training Qwen3-4B on 562 samples takes about 3 days on 32 GPUs. This seems quite long to us — though we haven’t started the full training yet, perhaps the QA inference step contributes significantly to the overall runtime?

Thanks again for your thorough explanation and for maintaining this project!

[wangyu-ustc]

I have uploaded the datasets here: https://huggingface.co/datasets/YuWangX/Memalpha
Now there shouldn't be any ambiguities anymore! I will also update the codebase to remove everything about <answer></answer>. Currently the code will still parse <answer></answer> first but it doesn't matter because it should never appear in the responses.

[wangyu-ustc]

• Oh I see, that is kind of wired, let me run my model to see what is happening.
• sounds good!
• Yes QA inference does take a long time, there are ways to improve the training efficiency, please refer to this issue here: Training slowdown caused by frequent API calls #5

[wtma1999]

Hi, many thanks for your prompt and insightful response.

We will proceed to reproduce the training process based on your implementation, and we hope to reach out for further discussion if we encounter any issues during replication.

We truly appreciate your clarification and the rigor of your work. This is a very solid and well-executed study.