Slow NER training with GPU - SpaCy v3.2

[faouinti]

Hi,

I am using GPU to train a NER model from Scratch in SpaCy v3.2 (with the --gpu-id option) and SLURM job scheduler:

sbatch -p gpu --gres = gpu: v100: 1 my_script.sh

Here is the "my_script.sh" submission script:

#! / bin / bash
python -m spacy train config.cfg --output ./output --paths.train ./train.spacy --paths.dev ./train.spacy --gpu-id 0

When I use nvidia-smi, I can clearly see GPU usage at 7% with memory usage at 0% (slow). That's why, I think that adjustments on my side are to be made at the level of the SpaCy to optimize its use.

Could you please tell me where this slow training with GPU comes from?

Thanks in advance,
FA

[polm]

What makes you say your training is slow? Is it faster in another context? You say that memory usage is low and therefore it's slow but I don't understand the connection.

Also, how big is your training set, your GPU, your RAM...? It's not clear if SLURM is relevant here or not either.

[faouinti]

Thank you for your reply.

My training dataset contains 574 files (short texts), I was able to retrieve an accounting info on the submitted job (on GPU v100) which ran ~ 11 minutes for 7 epochs (~ 24 minutes on CPU). Is this normal?

I think that if we have a ratio of less than 4, we can talk about slowness.

[polm]

How long is a short text?

GPU speedup for non-Transformer models is a thing, but it's not dramatic. Are you using a Transformer?

Your training time doesn't sound particularly weird to me. Maybe try the example projects and see how they compare?

[faouinti]

The mean length of the contents is around 2000 characters (space included).

I just selected GPU (transformer) for Hardware when creating the Config file for training the NER model from scratch. To get better use of GPU, is it necessary to use Transformer? Also, how to specify the number of epochs in SpaCy v3.2?

Thank you.

[polm]

Transformers have higher accuracy than other models, but they are very slow unless you use a GPU.

Based on your description here, it sounds like things are working as expected. I'm still not sure why you think this is slow.

Epochs and patience settings are are specified in the training config.

[faouinti]

Thank you for the clarification.

[lashmore]

@polm, are there more statistics like the one you gave us above? I have a 7,000 document training dataset and a 3,000 document dev dataset, and it takes roughly 45 minutes for every 200 steps (STEPS not epochs). My GPU utilization is about 2000/24000 as well, so low. It is very hard to debug when training time is soooo slow.