Clarification question regarding your training stages for LLaVA

[vishaal27]

Hey,

Thanks for releasing your code and the new dataset, and for your awesome work!

I had a clarification question regarding your main experiments: as far as I understand, the original LLaVA was trained in two stages: an adapter-training stage (where everything except the connector is kept frozen, and where mostly captioning-only data is used for training) and the instruction-tuning stage (where the connector and LLM are tuned, and where the actual instruction tuning data is used). For all your experiments, if I understand correctly, you follow the same two stages, and your data selection method only operates on the second stage, correct? You don't do any changes to the data mix of the first stage of training (the adapter-training stage)?

Please correct me if I misunderstand any parts of your work. Thanks!

[XindiWu]

Hi,

Thank you for your interest in our work! :)

That’s correct! We focus on the visual instruction-tuning stage, so the data selection is specifically for the 665K LLaVA visual instruction-tuning data. Let me know if you have any other questions. Thanks!

Xindi