can't load LISA++

[victoriazinkovich]

Dear authors,
I am currently working with the "Senqiao/LISA_Plus_7b" model and have encountered an issue while loading it using the following code:

model = LISAForCausalLM.from_pretrained(
    pretrained_model_name_or_path="Senqiao/LISA_Plus_7b",
    low_cpu_mem_usage=True,
    seg_token_idx=32000
)

When I run this code, I receive a warning indicating that some weights of LISAForCausalLM were not initialized from the model checkpoint and are newly initialized: ['model.mm_projector.bias', 'model.mm_projector.weight'].

Could you please provide guidance on the correct way to load this model to avoid such discrepancies? Your assistance would be greatly appreciated. Thank you for your time!

[victoriazinkovich]

Dear Authors,
If I understand correctly, the error is caused by the fact that the LLaVA package has been updated, and the new version of LISA++ is missing the multimodal_projector folder. May I kindly ask you to update your code?
Thank you!

[Yangsenqiao]

Oh! The new version LISA++ is based on LLaVA-1.5, whereas the current version of LISA code is based on LLaVA. I am sorry to confuse you. Let me see how to update it.

[SHYuanBest]

@Yangsenqiao Hi, any update?

[ktthn]

@Yangsenqiao Hello, any updates on LISA++ code?

[Yangsenqiao]

Thanks for your interest in LISA++! I’ve published a new branch named lisa_plus.