Can I train OpenFlamingo without LIAON dataset?

[ElegantLin]

Thanks for your great job. I wonder whether we can train open flamingo with MMC4 dataset only and I wonder why the loss from MMC4 dataset could be nan.

Thanks for your explanation.

[i-gao]

Hi, thanks for your question! The code is not currently configured like this, but it wouldn't be hard to implement (similar to #145). If you'd like to contribute a PR, this would make a great first issue!

Regarding nan losses: great question. This bit of code originally sought to catch cases where the mmc4 sequence looks like "text text ". In this case, all labels are masked to -100, since there are no text tokens after image tokens. We later updated data.py upstream to prevent these sequences from being sampled, so that issue is resolved. There may still be nan cases from training at larger scales than 9B. We have not worked with those scales yet to observe them.

[ElegantLin]

Thanks for your kind reply. I am planning to contribute a PR to make this project more complete :). I will close this issue after I finish the PR.

[YerongLi]

Good point, we need to train on smaller dataset. Wish we can get an example workflow.