Questions on running time

[XingyuXiong]

I wonder in each step how much time it will take? In preprocessing, trainning and running the model.

[XingyuXiong]

Oh unfortunately I get this when run preprocess.py
Traceback (most recent call last):
File "preprocess.py", line 52, in
train['y'], train['c'], train['r'] = get_values('ubuntu_data/train.txt', get_c_d=False, w2id=w2id)
File "preprocess.py", line 28, in get_values
c = [' EOS '.join(a[1:-1]).split() for a in data]
File "preprocess.py", line 28, in
c = [' EOS '.join(a[1:-1]).split() for a in data]
MemoryError
What should I do?

[DeepInEvil]

for me the training took 22 mins/epoch and evaluation around a minute
your error suggests that you don't have enough ram on the machine you are running on

[XingyuXiong]

I find it is possibly because I am running on a windows 10 system, and there is a memory limitation for the maximum of 3g on python 32bit. Thank you!

[XingyuXiong]

Hi, I solve some environment problems and start to run preprocessing in Ubuntu 18.04 in VMware. Still, it takes me lots of time, maybe more than several hours. Is there any possible ways to accelerate the process?

[DeepInEvil]

yes, we have a custom preprocessor in place, can you try changing it to multi-threaded? in preprocess,py I can try to do it too when I have some time.