Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[QA] Internevo是否支持tied_embedding? #267

Open
Cerberous opened this issue Jul 1, 2024 · 3 comments
Open

[QA] Internevo是否支持tied_embedding? #267

Cerberous opened this issue Jul 1, 2024 · 3 comments
Assignees
@zigzagcai @yhcc
Labels
question Further information is requested

Comments

@Cerberous
Copy link

Describe the question.

请问 Internevo是否支持tied_embedding?有的话怎么使用呢？
@Cerberous Cerberous added the question Further information is requested label Jul 1, 2024
@sunpengsdu
Copy link
Contributor

@zigzagcai 帮忙看一下

@zigzagcai
Copy link
Collaborator

zigzagcai commented Jul 11, 2024 •
edited
Loading

目前InternEvo内置的模型（InternLM/InternLM2）没有使用tied word embedding
并且，我查看了下，类似的LLaMA也没有使用tied word embedding，meta-llama/llama#138

@divisionblur
Copy link

lm_head不和embedding一样这样的好处是什么呢？

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zigzagcai zigzagcai

@yhcc yhcc

Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@sunpengsdu @zigzagcai @yhcc @Cerberous @divisionblur