Not able to train model on long spans

[belalsalih]

Hi,
We are trying to create a Resume parser using spaCy and are able to get the model to extract the personal details such as: Name, email, mobile etc... However, the requirement is that the model should extract Work History as a whole section then extract the sub details for this section [Company name, designation, start & end dates and role summary].

We are using SpanCat for this section but when we try to train the model to extract Work History as a single span without the sub details for this section, but we are getting zeroes as shown in the attached training output file:
config and data debug files, attached as well.

training output: training_output.txt
config file used for training: config.txt
debug data output: debug_data.txt

Note: we are using "5346" CVs for training and "1768" Cvs for validation.

Environment:

• Operating System: Ubuntu 20.04.6 LTS
• Python Version Used: 3.8.10
• spaCy Version Used: 3.6.0
• Environment Information:

Please advise us if we are doing something wrong or is this not possible using spaCy?

Thanks for your help.

[shadeMe]

SpanCat can work with long spans, but the span suggester will also have to be configured accordingly. The currently used ngram suggester with [1,2,3] ngrams is not going to be good enough for the span lengths you're hoping to predict. If your spans are distinct, you could experiment with span_finder (you might need to train this component separately before using it in conjunction with SpanCat). That being said, it might be better to reframe the problem in such a way that you can work with longer chunks (by employing rule-based chunking) instead of token spans, i.e., sentences or paragraphs.

[belalsalih]

Thanks for your reply,
I tried to change the ngram suggester from [1,2,3] to a ngram range, but ran into "Segmentation fault (core dumped)" as reported in issue 13026
I will try span_finder, whoever there are very few examples on how to use/train span_finder. Do you know any good example on using span_finder in conjunction with SpanCat.
As for "rule-based chunking" I doubt this will work since each CV is written differently and it is impossible to write a code to predict the start and end of each CV section.

Regards.

[belalsalih]

BTW, I found the following tutorial on using SpanCat with sapn_finder:
https://github.com/explosion/spacy-experimental/blob/master/projects/span_finder/project.yml
I managed to run training using the attached config file, unfortunately I got the same "Segmentation fault (core dumped)" as shown below:

[2023-10-03 09:07:20,463] [DEBUG] Config overrides from CLI: ['paths.train', 'paths.dev']
ℹ Saving to output directory: output_v3_lg_1.3
ℹ Using CPU

=========================== Initializing pipeline ===========================
[2023-10-03 09:07:20,622] [INFO] Set up nlp object from config
[2023-10-03 09:07:20,630] [DEBUG] Loading corpus from path: spacy_models_v3/test_data.spacy
[2023-10-03 09:07:20,630] [DEBUG] Loading corpus from path: spacy_models_v3/train_data.spacy
[2023-10-03 09:07:20,630] [INFO] Pipeline: ['tok2vec', 'span_finder', 'spancat']
[2023-10-03 09:07:20,634] [INFO] Created vocabulary
[2023-10-03 09:07:20,634] [INFO] Finished initializing nlp object
[2023-10-03 09:07:26,572] [INFO] Initialized pipeline components: ['tok2vec', 'span_finder', 'spancat']
✔ Initialized pipeline

============================= Training pipeline =============================
[2023-10-03 09:07:26,580] [DEBUG] Loading corpus from path: spacy_models_v3/test_data.spacy
[2023-10-03 09:07:26,580] [DEBUG] Loading corpus from path: spacy_models_v3/train_data.spacy
ℹ Pipeline: ['tok2vec', 'span_finder', 'spancat']
ℹ Set annotations on update for: ['span_finder']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPAN_...  LOSS SPANCAT  SPAN_FINDE...  SPAN_FINDE...  SPAN_FINDE...  SPANS_SC_F  SPANS_SC_P  SPANS_SC_R  SCORE 
---  ------  ------------  -------------  ------------  -------------  -------------  -------------  ----------  ----------  ----------  ------
Segmentation fault (core dumped)

config file: config_cfg.txt

VM RAM did not exceed 30%.

[adrianeboyd]

Can you try first training only the span_finder and only adding spancat once the span_finder's performance is reasonably good?

If it helps, here's an example project with two training steps due to the fact that spancat doesn't train well until the suggestions are reasonable: https://github.com/adrianeboyd/workshop-dh2023/blob/6092cb4abb6840d7664117f4dd4b04d6a4fde933/litbank/project.yml#L130-L154

There is one config just with span_finder, and then a second config with span_finder+spancat that sources the previously trained span_finder.

A randomly initialized span_finder may suggest every single possible span in a document before it is trained further, and then in step 0 spancat crashes when it tries to handle way too many suggested spans. Usually this would look like a memory error instead of a segmentation fault, but it's possible it's related.

It's also possible you've found a bug (a segfault is unusual) but it's pretty hard for us to track it down without a full working example where we can reproduce it.

[belalsalih]

Thanks for your reply, I will try your suggestion and let you know how it goes.

[belalsalih]

Hi,
Sorry for the late reply, I did manage to train sapn_finder model separately but unfortunately it was not giving any suggestion. It could be the number of samples not big enough.

Regards.

[vignesh-spericorn]

Hello @belalsalih

Can you please share how did you annotate the data to train a span_finder model ?
I cannot find any resource which shares on how to annotate for training span_finder model and how the annotations.json(Training data) file would look like. Also how to convert it to .spacy format ; Is it same as in the case of converting annotation json files for training an NER model ?

Regards,