Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEA：add split token and generate related resource #59

Open
wants to merge 39 commits into
base: main
Choose a base branch
from
Open

FEA：add split token and generate related resource #59

Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
502463d
txy
txy77 Sep 28, 2022
ee1d80a
txy77
txy77 Sep 29, 2022
e25f2d8
Merge branch 'main' of https://github.com/txy77/CRSLab
txy77 Sep 29, 2022
10c36fe
txy77
txy77 Sep 29, 2022
b8d82ed
Merge branch 'main' of https://github.com/txy77/CRSLab
txy77 Sep 29, 2022
c8f1385
txy77
txy77 Sep 29, 2022
ad22cde
Merge branch 'main' of https://github.com/txy77/CRSLab
txy77 Sep 29, 2022
661824a
txy77
txy77 Sep 30, 2022
b588195
Merge branch 'main' of https://github.com/txy77/CRSLab
txy77 Sep 30, 2022
3226e5e
txy77
txy77 Oct 6, 2022
99796d0
txy77
txy77 Oct 7, 2022
1eb81e4
txy77
txy77 Oct 7, 2022
0b0436e
Delete settings.json
txy77 Oct 7, 2022
735c3aa
txy77
txy77 Oct 11, 2022
d775117
txy77
txy77 Oct 11, 2022
7f43abe
txy77
txy77 Oct 11, 2022
36e120d
txy77
txy77 Oct 11, 2022
d245105
txy77
txy77 Oct 11, 2022
0fcd3d3
txy77
txy77 Oct 11, 2022
53427ea
txy77
txy77 Oct 11, 2022
8ac3b83
txy77
txy77 Oct 12, 2022
2a529d4
txy77
txy77 Oct 12, 2022
4117166
txy77
txy77 Oct 12, 2022
2707552
txy77
txy77 Oct 12, 2022
f92d485
txy77
txy77 Oct 12, 2022
4763e57
txy77
txy77 Oct 12, 2022
b6d4c34
txy77
txy77 Oct 14, 2022
b6445df
txy77
txy77 Oct 14, 2022
d7b9d5b
txy77
txy77 Oct 14, 2022
2cab622
txy77
txy77 Oct 14, 2022
97ea317
txy77
txy77 Oct 14, 2022
af93741
txy77
txy77 Oct 14, 2022
699a537
txy77
txy77 Oct 15, 2022
b58aefa
txy77
txy77 Oct 15, 2022
de8e3f3
txy77
txy77 Oct 18, 2022
86a23d4
txy77
txy77 Nov 14, 2022
f4c073e
txy77
txy77 Nov 14, 2022
c7acee8
txy77
txy77 Nov 14, 2022
4edd74a
txy77
txy77 Jun 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions config/conversation/gpt2/durecdial.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,17 @@
dataset: DuRecDial
tokenize:
conv: gpt2
# tokenize path
conv_tokenize_path: 'GPT2-chitchat'
# dataloader
context_truncate: 256
response_truncate: 30
item_truncate: 100
scale: 0.01
# model
conv_model: GPT2
# pretrained path
conv_pretrained_path: 'GPT2-chitchat'
# optim
conv:
epoch: 1
Expand Down
4 changes: 4 additions & 0 deletions config/conversation/gpt2/gorecdial.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,17 @@
dataset: GoRecDial
tokenize:
conv: gpt2
# tokenize path
conv_tokenize_path: 'gpt2'
# dataloader
context_truncate: 256
response_truncate: 30
item_truncate: 100
scale: 0.01
# model
conv_model: GPT2
# pretrained path
conv_pretrained_path: 'gpt2'
# optim
conv:
epoch: 1
Expand Down
4 changes: 4 additions & 0 deletions config/conversation/gpt2/inspired.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,17 @@
dataset: Inspired
tokenize:
conv: gpt2
# tokenize path
conv_tokenize_path: 'gpt2'
# dataloader
context_truncate: 256
response_truncate: 30
item_truncate: 100
scale: 1
# model
conv_model: GPT2
# pretrained path
conv_pretrained_path: 'gpt2'
# optim
conv:
epoch: 1
Expand Down
4 changes: 4 additions & 0 deletions config/conversation/gpt2/opendialkg.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,17 @@
dataset: OpenDialKG
tokenize:
conv: gpt2
# tokenize path
conv_tokenize_path: 'gpt2'
# dataloader
context_truncate: 256
response_truncate: 30
item_truncate: 100
scale: 0.01
# model
conv_model: GPT2
# pretrained path
conv_pretrained_path: 'gpt2'
# optim
conv:
epoch: 1
Expand Down
4 changes: 4 additions & 0 deletions config/conversation/gpt2/redial.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,17 @@
dataset: ReDial
tokenize:
conv: gpt2
# tokenize path
conv_tokenize_path: 'gpt2'
# dataloader
context_truncate: 256
response_truncate: 30
item_truncate: 100
scale: 0.01
# model
conv_model: GPT2
# pretrained path
conv_pretrained_path: 'gpt2'
# optim
conv:
epoch: 1
Expand Down
4 changes: 4 additions & 0 deletions config/conversation/gpt2/tgredial.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,17 @@
dataset: TGReDial
tokenize:
conv: gpt2
# tokenize path
conv_tokenize_path: 'GPT2-chitchat'
# dataloader
context_truncate: 256
response_truncate: 30
item_truncate: 100
scale: 1
# model
conv_model: GPT2
# pretrained path
conv_pretrained_path: 'GPT2-chitchat'
# optim
conv:
epoch: 50
Expand Down
7 changes: 7 additions & 0 deletions config/crs/inspired/durecdial.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ dataset: DuRecDial
tokenize:
rec: bert
conv: gpt2
# tokenize path
rec_tokenize_path: 'bert-base-chinese'
conv_tokenize_path: 'GPT2-chitchat'
# dataloader
context_truncate: 256
response_truncate: 30
Expand All @@ -11,8 +14,12 @@ scale: 1
# model
# rec
rec_model: InspiredRec
# pretrained path
rec_pretrained_path: 'bert-base-chinese'
# conv
conv_model: InspiredConv
# pretrained path
conv_pretrained_path: 'GPT2-chitchat'
# embedding: word2vec
embedding_dim: 300
use_dropout: False
Expand Down
7 changes: 7 additions & 0 deletions config/crs/inspired/gorecdial.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ dataset: GoRecDial
tokenize:
rec: bert
conv: gpt2
# tokenize path
rec_tokenize_path: 'bert-base-uncased'
conv_tokenize_path: 'gpt2'
# dataloader
context_truncate: 256
response_truncate: 30
Expand All @@ -11,8 +14,12 @@ scale: 1
# model
# rec
rec_model: InspiredRec
# pretrained path
rec_pretrained_path: 'bert-base-uncased'
# conv
conv_model: InspiredConv
# pretrained path
conv_pretrained_path: 'gpt2'
# embedding: word2vec
embedding_dim: 300
use_dropout: False
Expand Down
7 changes: 7 additions & 0 deletions config/crs/inspired/inspired.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ dataset: Inspired
tokenize:
rec: bert
conv: gpt2
# tokenize path
rec_tokenize_path: 'bert-base-uncased'
conv_tokenize_path: 'gpt2'
# dataloader
context_truncate: 256
response_truncate: 30
Expand All @@ -11,8 +14,12 @@ scale: 1
# model
# rec
rec_model: InspiredRec
# pretrained path
rec_pretrained_path: 'bert-base-uncased'
# conv
conv_model: InspiredConv
# pretrained path
conv_pretrained_path: 'gpt2'
# optim
rec:
epoch: 1
Expand Down
7 changes: 7 additions & 0 deletions config/crs/inspired/opendialkg.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ dataset: OpenDialKG
tokenize:
rec: bert
conv: gpt2
# tokenize path
rec_tokenize_path: 'bert-base-uncased'
conv_tokenize_path: 'gpt2'
# dataloader
context_truncate: 256
response_truncate: 30
Expand All @@ -11,8 +14,12 @@ scale: 1
# model
# rec
rec_model: InspiredRec
# pretrained path
conv_pretrained_path: 'bert-base-uncased'
# conv
conv_model: InspiredConv
# pretrained path
conv_pretrained_path: 'gpt2'
# embedding: word2vec
embedding_dim: 300
use_dropout: False
Expand Down
7 changes: 7 additions & 0 deletions config/crs/inspired/redial.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ dataset: ReDial
tokenize:
rec: bert
conv: gpt2
# tokenize path
rec_tokenize_path: 'bert-base-uncased'
conv_tokenize_path: 'gpt2'
# dataloader
context_truncate: 256
response_truncate: 30
Expand All @@ -11,8 +14,12 @@ scale: 1
# model
# rec
rec_model: InspiredRec
# pretrained path
conv_pretrained_path: 'bert-base-uncased'
# conv
conv_model: InspiredConv
# pretrained path
conv_pretrained_path: 'gpt2'
# embedding: word2vec
embedding_dim: 300
use_dropout: False
Expand Down
7 changes: 7 additions & 0 deletions config/crs/inspired/tgredial.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ dataset: TGReDial
tokenize:
rec: bert
conv: gpt2
# tokenize path
rec_tokenize_path: 'bert-base-chinese'
conv_tokenize_path: 'GPT2-chitchat'
# dataloader
context_truncate: 256
response_truncate: 30
Expand All @@ -11,8 +14,12 @@ scale: 1
# model
# rec
rec_model: InspiredRec
# pretrained path
rec_pretrained_path: 'bert-base-chinese'
# conv
conv_model: InspiredConv
# pretrained path
conv_pretrained_path: 'GPT2-chitchat'
# embedding: word2vec
embedding_dim: 300
use_dropout: False
Expand Down
3 changes: 2 additions & 1 deletion config/crs/kgsf/durecdial.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# dataset
dataset: DuRecDial
tokenize: jieba
embedding: word2vec.npy
embedding: True
# dataloader
context_truncate: 256
response_truncate: 30
Expand All @@ -21,6 +21,7 @@ learn_positional_embeddings: false
embeddings_scale: true
reduction: false
n_positions: 1024
copy: true
# optim
pretrain:
epoch: 1
Expand Down
3 changes: 2 additions & 1 deletion config/crs/kgsf/gorecdial.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# dataset
dataset: GoRecDial
tokenize: nltk
embedding: word2vec.npy
embedding: True
# dataloader
context_truncate: 256
response_truncate: 30
Expand All @@ -21,6 +21,7 @@ learn_positional_embeddings: false
embeddings_scale: true
reduction: false
n_positions: 1024
copy: true
# optim
pretrain:
epoch: 1
Expand Down
3 changes: 2 additions & 1 deletion config/crs/kgsf/inspired.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# dataset
dataset: Inspired
tokenize: nltk
embedding: word2vec.npy
embedding: True
# dataloader
context_truncate: 256
response_truncate: 30
Expand All @@ -21,6 +21,7 @@ learn_positional_embeddings: false
embeddings_scale: true
reduction: false
n_positions: 1024
copy: true
# optim
pretrain:
epoch: 1
Expand Down
3 changes: 2 additions & 1 deletion config/crs/kgsf/opendialkg.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# dataset
dataset: OpenDialKG
tokenize: nltk
embedding: word2vec.npy
embedding: True
# dataloader
context_truncate: 256
response_truncate: 30
Expand All @@ -21,6 +21,7 @@ learn_positional_embeddings: false
embeddings_scale: true
reduction: false
n_positions: 1024
copy: true
# optim
pretrain:
epoch: 1
Expand Down
3 changes: 2 additions & 1 deletion config/crs/kgsf/redial.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# dataset
dataset: ReDial
tokenize: nltk
embedding: word2vec.npy
embedding: True
# dataloader
context_truncate: 256
response_truncate: 30
Expand All @@ -21,6 +21,7 @@ learn_positional_embeddings: false
embeddings_scale: true
reduction: false
n_positions: 1024
copy: true
# optim
pretrain:
epoch: 3
Expand Down
3 changes: 2 additions & 1 deletion config/crs/kgsf/tgredial.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# dataset
dataset: TGReDial
tokenize: pkuseg
embedding: word2vec.npy
embedding: True
# dataloader
context_truncate: 256
response_truncate: 30
Expand All @@ -21,6 +21,7 @@ learn_positional_embeddings: false
embeddings_scale: true
reduction: false
n_positions: 1024
copy: true
# optim
pretrain:
epoch: 50
Expand Down
3 changes: 2 additions & 1 deletion config/crs/ntrd/tgredial.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# dataset
dataset: TGReDial
tokenize: pkuseg
embedding: word2vec.npy
embedding: True
# dataloader
context_truncate: 256
response_truncate: 30
Expand All @@ -24,6 +24,7 @@ n_positions: 1024
gen_loss_weight: 5
n_movies: 62287
replace_token: '[ITEM]'
copy: true
# optim
pretrain:
epoch: 50
Expand Down
6 changes: 6 additions & 0 deletions config/crs/tgredial/durecdial.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ dataset: DuRecDial
tokenize:
rec: bert
conv: gpt2
# tokenize path
rec_tokenize_path: 'bert-base-chinese'
conv_tokenize_path: 'GPT2-chitchat'
# dataloader
context_truncate: 256
response_truncate: 30
Expand All @@ -11,6 +14,9 @@ scale: 0.01
# model
rec_model: TGRec
conv_model: TGConv
# pretrained path
rec_pretrained_path: 'bert-base-chinese'
conv_pretrained_path: 'GPT2-chitchat'
hidden_dropout_prob: 0.2
initializer_range: 0.02
hidden_size: 50
Expand Down
Loading