When reproducing the results, it is possible to reproduce the NLG results but not the CE metric results.

[mengweiwang]

I used the data format (only for the findings section) as R2Gen and R2GenCMN (Chen et al.) followed in this article, but I was unable to obtain the CE metric results mentioned in the paper.

I used the provided epoch=8-val_chen_cider=0.425092.ckpt model for cvt_21_to_distilgpt2 task and also tested epoch=0-val_chen_cider=0.410965.ckpt model for cvt_21_to_distilgpt2_scst task, but neither of them achieved the CE metric results mentioned in the paper.

In terms of CE metric, precision_macro can reach the result mentioned in the paper, but recall_macro and f1_macro cannot achieve it and there is a significant difference between them.

When calculating CE metrics here, only text related to findings is considered; do I need to perform any other processing?

The results obtained from performing cvt_21_to_distilgpt2 task are as follows:
{'test_ce_f1_example': 0.36598095297813416,
'test_ce_f1_macro': 0.2593880891799927,
'test_ce_f1_micro': 0.4408090114593506,
'test_ce_num_examples': 3858.0,
'test_ce_precision_example': 0.4171517491340637,
'test_ce_precision_macro': 0.3600466549396515,
'test_ce_precision_micro': 0.4919118881225586,
'test_ce_recall_example': 0.3665845990180969,
'test_ce_recall_macro': 0.25423887372016907,
'test_ce_recall_micro': 0.3993246555328369,
'test_chen_bleu_1': 0.39292487502098083,
'test_chen_bleu_2': 0.24805393815040588,
'test_chen_bleu_3': 0.17164887487888336,
'test_chen_bleu_4': 0.1269991397857666,
'test_chen_cider': 0.3902686834335327,
'test_chen_meteor': 0.15456412732601166,
'test_chen_num_examples': 3858.0,
'test_chen_rouge': 0.286588191986084}

The results in the paper are as follows:
precision_macro: 0.3597
recall_macro: 0.4122
f1_macro: 0.3842

The results obtained from performing cvt_21_to_distilgpt2_scst task are as follows:
{'test_ce_f1_example': 0.36484676599502563,
'test_ce_f1_macro': 0.26361414790153503,
'test_ce_f1_micro': 0.4410783648490906,
'test_ce_num_examples': 3858.0,
'test_ce_precision_example': 0.4175392985343933,
'test_ce_precision_macro': 0.3873042166233063,
'test_ce_precision_micro': 0.49624764919281006,
'test_ce_recall_example': 0.3643813729286194,
'test_ce_recall_macro': 0.2558453679084778,
'test_ce_recall_micro': 0.3969484865665436,
'test_chen_bleu_1': 0.39466917514801025,
'test_chen_bleu_2': 0.248764768242836,
'test_chen_bleu_3': 0.1718045324087143,
'test_chen_bleu_4': 0.1269892156124115,
'test_chen_cider': 0.37993040680885315,
'test_chen_meteor': 0.15499255061149597,
'test_chen_num_examples': 3858.0,
'test_chen_rouge': 0.28760746121406555}

Reproduced the above content, only modifying the task parameters in task/mimic_cxr_jpg_chen/jobs.yaml.

[anicolson]

Hi,

Please see the updated README.md for the labels from Chen et al.
https://github.com/aehrc/cvt2distilgpt2

I will look into the discrepency with the results.

[mengweiwang]

Hi,

Please see the updated README.md for the labels from Chen et al. aehrc/cvt2distilgpt2

I will look into the discrepency with the results.

@anicolson

Yes, I am using this dataset and the precision has reached the level reported in the paper. However, the recall rate is low and cannot reach the level reported in the paper.

Also, I have checked and tested the updated source code. The CE metric results did not change much, and there is a bug in the latest source code when running it. The bug is as follows:

The bug that occurred while I was executing the cvt_21_to_distilgpt2 task.

On line 281 of transmodal.model.py, the content is if not getattr(self, metric).compute_on_step:.

It indicates that the compute_on_step attribute does not exist.

[anicolson]

Hi, there are some errors in the preprint, the correct results are reported in the updated repository.