Resize token embedding.

[hungphongtrn]

Hi,

Are there any methods to resize a pretrained models after I add in new vocabulary?

Thanks in advance!

[ebsmothers]

Hi @hungphongtrn thanks for creating the issue. We don't currently have any such methods in the library, but I'm curious to know more about what you have in mind here. Is the idea to append additional indices to the embedding for an existing vocabulary? If so, how would the new indices be initialized -- randomly, or from some other set of weights that can be passed in? Similar question for the final projection back to the vocab dimension since this would need to be updated as well.