Tokens to be integers

[Tommy-Hsu]

Hello, I would like to ask about the meaning of tokens being integers. I noticed that the final forward pass to the tokenizer involves the cls_logits_softmax tensor, and it directly performs a matrix multiplication with the codebook. However, these operations are all in floating-point. So, what does it mean for tokens to be integers in classifier stage?

[ndsl555]

I'd like to know too

[KunmingS]

Seems like the integer token only happens when stage I training. I think is the variable encoding_indices is this line: https://github.com/Gengzigang/PCT/blob/main/models/pct_tokenizer.py#L142

[Tommy-Hsu]

Seems like the integer token only happens when stage I training. I think is the variable encoding_indices is this line: https://github.com/Gengzigang/PCT/blob/main/models/pct_tokenizer.py#L142

That's true. In stage 1, the encoding_indices would be integers but not in stage 2.

[Tommy-Hsu]

Figure 1 is quite confusing to me. In the reference stage, the class head output should be logits, and the codebook context is composed of floating-point data. However, this image shows that both are composed of integers, which is what I find puzzling.