json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 831)

[hoangNguyen210]

Hi,
Thank you for your quick reply regarding the data generation file. However, I noticed that it still has some conflicts when generating the data.

When training with the generated data, I encountered the bug shown in the attached image. I tried to fix it, but solving one bug led to another, so I decided to revert to the original issue (shown in the screenshot). I believe this problem is related to the format of the .jsonl file.

Could you please help me fix this issue?

Overall, I hope it would be possible for you to rerun the experiments and make them consistent so I can reproduce the results when training. In particular, the training command in the architectures/Exgra-Med/bashscript seems essential for reproducibility.

Thank you very much for your reading. I'm looking forward to hearing from you.

Best regards,

[leduckhai]

Hi,

Since the model was not developed by me, I do not know which format of the dataset should be used. Maybe @nghiemkythu and @duyhominhnguyen who ran architectures/Exgra-Med/bashscript will know what the format input looks like.

My assumption is, if we know how the format input looks like, then we can easily convert JSONL into such format and the model would be run fine without conflicts.

Also, I'm independently reproducing all config files to ensure that every format is consistent. But it might take time. Not sure when finished.

[leduckhai]

We just uploaded an example of our JSON file which we used to run experiments last time (see English/llava_med_mri_bbox_test_CoT.json. )

Note that this is just an example. It looks like this:

[
    {
        "image": "11590.jpg",
        "id": "11590",
        "conversations": [
            {
                "from": "human",
                "value": "How would you determine the pathological impact?\n<image>"
            },
            {
                "from": "gpt",
                "value": "\n\n+ Reasoning:\n- Step 1: Recognize the disease area\n- Answer: ```json\n[{\"bbox_2d\": [37, 242, 122, 405], \"label\": \"disease area\"}, {\"bbox_2d\": [113, 244, 199, 400], \"label\": \"disease area\"}]\n```\n- Step 2: How would you specify the visible traits of this lesion?\n- Answer: Substantial sulcal widening of posterior cingulate and parieto-occipital sulci, substantial gyral atrophy.\n- Step 3: What is the stage of the lesion?\n- Answer: Koedam = 2\n\n+ Final Answer: Moderate-Dementia"
            }
        ],
        "conversations_long": [
            {
                "from": "human",
                "value": "How would you determine the pathological impact?\n<image>"
            },
            {
                "from": "gpt",
                "value": "\n\n+ Reasoning:\n- Step 1: Recognize the disease area\n- Answer: ```json\n[{\"bbox_2d\": [37, 242, 122, 405], \"label\": \"disease area\"}, {\"bbox_2d\": [113, 244, 199, 400], \"label\": \"disease area\"}]\n```\n- Step 2: How would you specify the visible traits of this lesion?\n- Answer: Substantial sulcal widening of posterior cingulate and parieto-occipital sulci, substantial gyral atrophy.\n- Step 3: What is the stage of the lesion?\n- Answer: Koedam = 2\n\n+ Final Answer: Moderate-Dementia"
            }
        ]
    },
]

It might require the image path for every image inside the JSON file, e.g. "image": "11590.jpg" in JSON above should be replaced with "image": "link_to_image_file"

Feel free to tell me if you have further problem 👍🏻

[nghiemkythu]

Hi hoangNguyen210,

I have just added an instruction about preparing dataset for training and testing in architectures/Exgra-Med/README.md, you can follow to download .json files as well as ./images folder for training and testing in CoT settings.

Feel free to tell me if you have further problem.

[hoangNguyen210]

Hi Nghiem,

I have processed the data according to your format, as shown in the attached image. However, I now face a new issue: the image processing function is not working correctly.

This is the bug I am facing:

WARNING:root:Loading data...
WARNING:root:Formatting inputs...Skip in lazy mode
/mnt/vast-nhr/projects/nim00014/Verified_SCOT_RAG/S-Chain/exgra-med-env/lib/python3.11/site-packages/torch/distributed/fsdp/_init_utils.py:295: UserWarning: FSDP is switching to use NO_SHARD instead of ShardingStrategy.FULL_SHARD since the world size is 1.
warnings.warn(

0%| | 0/2022 [00:00<?, ?it/s]Traceback (most recent call last):
File "/mnt/vast-nhr/projects/nim00014/Verified_SCOT_RAG/S-Chain/exgra-med-env/lib/python3.11/site-packages/transformers/feature_extraction_utils.py", line 165, in convert_to_tensors
tensor = as_tensor(value)
^^^^^^^^^^^^^^^^
File "/mnt/vast-nhr/projects/nim00014/Verified_SCOT_RAG/S-Chain/exgra-med-env/lib/python3.11/site-packages/transformers/feature_extraction_utils.py", line 147, in as_tensor
return torch.tensor(value)
^^^^^^^^^^^^^^^^^^^
RuntimeError: Could not infer dtype of numpy.float32

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/user/nguyen50/u12045/.project/dir.project/Verified_SCOT_RAG/S-Chain/architectures/Exgra-Med/llava/train/train_mem_CoT.py", line 13, in
train()
File "/mnt/vast-nhr/projects/nim00014/Verified_SCOT_RAG/S-Chain/architectures/Exgra-Med/llava/train/train_CoT.py", line 931, in train
trainer.train()
File "/mnt/vast-nhr/projects/nim00014/Verified_SCOT_RAG/S-Chain/exgra-med-env/lib/python3.11/site-packages/transformers/trainer.py", line 1644, in train
return inner_training_loop(
^^^^^^^^^^^^^^^^^^^^
File "/mnt/vast-nhr/projects/nim00014/Verified_SCOT_RAG/S-Chain/exgra-med-env/lib/python3.11/site-packages/transformers/trainer.py", line 1881, in _inner_training_loop
for step, inputs in enumerate(epoch_iterator):
File "/mnt/vast-nhr/projects/nim00014/Verified_SCOT_RAG/S-Chain/exgra-med-env/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 634, in next
data = self._next_data()
^^^^^^^^^^^^^^^^^
File "/mnt/vast-nhr/projects/nim00014/Verified_SCOT_RAG/S-Chain/exgra-med-env/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 678, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/vast-nhr/projects/nim00014/Verified_SCOT_RAG/S-Chain/exgra-med-env/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/vast-nhr/projects/nim00014/Verified_SCOT_RAG/S-Chain/exgra-med-env/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 51, in
data = [self.dataset[idx] for idx in possibly_batched_index]
~~~~~~~~~~~~^^^^^
File "/mnt/vast-nhr/projects/nim00014/Verified_SCOT_RAG/S-Chain/architectures/Exgra-Med/llava/train/train_CoT.py", line 477, in getitem
image = processor.preprocess(image, return_tensors="pt")[
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/vast-nhr/projects/nim00014/Verified_SCOT_RAG/S-Chain/exgra-med-env/lib/python3.11/site-packages/transformers/models/clip/image_processing_clip.py", line 337, in preprocess
return BatchFeature(data=data, tensor_type=return_tensors)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/vast-nhr/projects/nim00014/Verified_SCOT_RAG/S-Chain/exgra-med-env/lib/python3.11/site-packages/transformers/feature_extraction_utils.py", line 76, in init
self.convert_to_tensors(tensor_type=tensor_type)
File "/mnt/vast-nhr/projects/nim00014/Verified_SCOT_RAG/S-Chain/exgra-med-env/lib/python3.11/site-packages/transformers/feature_extraction_utils.py", line 171, in convert_to_tensors
raise ValueError(
ValueError: Unable to create tensor, you should probably activate padding with 'padding=True' to have batched tensors with the same length.

I have bolded the error. Can you help me fix this? Thanks

[leduckhai]

Hi!

Some of our authors are being stuck with ICML, so it might take some time to fix these bugs!

To my knowledge, the CLIP encoder in the Exgra-Med returns 2 different tensor length. As suggested by the ValueError, could you please try activate 'padding=True'.

Or maybe could you please "force" the 2 tensors to have the same length? Not sure if this action reproduces the same experimental results, but I think it at least "works".