Entity Linking with Wikipedia and Wikidata on spaCy v3.7

[RistovaIvona]

I'm planning to perform custom training with spaCy for entity linking using Wikidata. Could you provide guidance on creating a KnowledgeBase, especially since the latest spaCy version needs InMemoryLookupKB? I'm also considering refactoring the code in the current GitHub repository (https://github.com/explosion/projects/tree/master/nel-wikipedia) to match the latest version. Do you think this is a good approach, or is there a simpler way? Any helpful tips or tricks would be welcome.

[rmitsch]

Hi @RistovaIvona! InMemoryLookupKB is identical to the previous KnowledgeBase class - the renaming was done as part of a refactoring that will make knowledge bases more customizable and capable in spaCy v4.

Having said that, some resources that might be useful for you:

• wikid is a spaCy project for pulling Wikidata dumps, preprocessing and storing them in a SQLite database.
• The EL benchmark project leverages wikid to use Wikidata as knowledge base. The dataset used for training and evaluation is Mewsli-9, but it should be easy enough to adjust the scaffolding to train with any other dataset.

In general I'd recommend adjusting the linked EL benchmark project - I think you'll have less work updating that one to your requirements than the (considerably older) nel-wikipedia project.

If you have any specific questions, feel free to drop them here.

[rmitsch]

To be clear, did you run all of the following steps?

• wikid_download_assets
• wikid_parse
• wikid_create_kb

[RistovaIvona]

Yes, for both questions.

[rmitsch]

What's the content of wikid/assets and how large are the files in there, respectively?

[RistovaIvona]

wikidata_entity_dump.json.bz2 - 13.3GB
wikidata_entity_dump_filtered.json.bz2 - 98.3MB
wikipedia_dump_filtered.xml.bz2 - 112 MB
en-wikipedia_dump.xml.bz2 - 258MB

[RistovaIvona]

Should I set 'False' in the filter field?

[rmitsch]

wikidata_entity_dump.json.bz2 - 13.3GB
wikidata_entity_dump_filtered.json.bz2 - 98.3MB
wikipedia_dump_filtered.xml.bz2 - 112 MB
en-wikipedia_dump.xml.bz2 - 258MB

Both the Wikipedia dump as well as the Wikidata dump are considerably too small for them to be the full dumps (they should be around 21 and 80 GB, respectively). Try downloading them directly:

• English Wikipedia dump
• Wikidata dump

Should I set 'False' in the filter field?

Yes, that's important for the processing steps. It doesn't explain why you ended up with only a subset of the data though. So as next steps: download the dumps directly and set filter to "False".

[RistovaIvona]

I went through the entire process again, setting the filter to 'False' and using the --extraoption, but without success. (I have again this as output: start: 3, end: 4, surface_form: McCarthy, label: PERSON, id: NIL)
Currently, I am downloading both the Wikipedia dump and the Wikidata dump datasets, and I will upload them manually. I should then be able to continue the process from the wikid_parse stage?

[RistovaIvona]

Hello, I am facing an issue because the database is extremely large, and I'm running into out-of-memory problems, even when using powerful machines. Now, I want to create and use a filtered version of the database. By default, the existing filtered version contains only articles and entities mentioning "New York" or "Boston." I would like to use entities wherever people are mentioned (entity_id = "Q5", property_id = "P31"). How can I achieve this (either through code or by creating a new database on my own)?

[rmitsch]

You can have a look at scripts/wiki/demo_dump.py - this goes through the dump, keeps only entities/articles containing the terms in configs/filter_terms.txt, and stores two new dumps (one Wikidata and one Wikipedia dump). There is currently no support to filter by exact entity or property IDs though.

Wheree are you running into OOM problems? We had a similar complaint very recently, so we might look into fixing some of these memory issues.

[RistovaIvona]

I was at a point in the project where I need to create a kb ('spacy project run wikid_create_kb'). I was using a virtual machine with the following specifications: Machine type: e2-highmem-16, vCPUs: 16, Memory (GB): 128, Max number of persistent disks (PDs): 128, Max total PD size (TiB): 257, and Maximum egress bandwidth (Gbps): 16, along with a Boot Disk of 500GB.

[rmitsch]

Hm, 128 GB memory should be enough - we didn't use more either. Anyway, we'll mitigate the memory bottlenecks in the code. It should be possible to use generators at least partially to reduce memory requirements. I can't tell you when we'll get around to doing this (I hope in the next two weeks, but I can't promise that).

If you want to tackle this yourself meanwhile, I recommend looking into this thread which links to the relevant places in the code.

[lashmore]

Hello @rmitsch and @RistovaIvona , I'm following this thread and I'm currently on the wikid_parse step, and it's been running for hours now. I'm currently on iteration ~30,000,000. Wondering how long this parse takes? They didn't add tqdm so I'm not sure how far along I am. Here's what I see currently:

(nel) $ weasel run wikid_parse

================================ wikid_parse ================================
Running command: spacy project run parse wikid --vars.language en --vars.filter False

=================================== parse ===================================
Running command: env PYTHONPATH=scripts python ./scripts/parse.py en False
Parsing entity data: 30864146it [6:01:14, 677.90it/s] 

[lashmore]

So to update any interested parties:

For the wikidata parse, it looks like the total # entities was ~107,000,000.

I didn't set gpu_id to 0 before I ran the parse code, but looking at the code, I don't see any code that would benefit from a gpu to begin with - I think it's only for the train step. What a waste that it's running on a GPU VM though! Wish someone would have warned me to run these initial steps on a CPU machine!

[geekyayush]

I feel you man

[lashmore]

Well I ran through training but I'm not really happy with the results. It looks like training didn't really learn much, and it only performed 1000 steps.

E    #       LOSS ENTIT...  SENTS_F  SENTS_P  SENTS_R  ENTS_F  ENTS_P  ENTS_R  NEL_MICRO_F  NEL_MICRO_R  NEL_MICRO_P  SCORE 
---  ------  -------------  -------  -------  -------  ------  ------  ------  -----------  -----------  -----------  ------
  0       0           1.08    90.69    92.83    88.65    0.00    0.00    0.00        83.73        81.75        85.81    0.58
  0     200          62.05    90.69    92.83    88.65    0.00    0.00    0.00        81.27        79.34        83.28    0.57                                                                              
  0     400          50.64    90.69    92.83    88.65    0.00    0.00    0.00        81.17        79.25        83.19    0.57                                                                              
  0     600          51.68    90.69    92.83    88.65    0.00    0.00    0.00        81.36        79.43        83.38    0.57                                                                              
  0     800          49.23    90.69    92.83    88.65    0.00    0.00    0.00        81.07        79.15        83.08    0.57                                                                              
  0    1000          50.56    90.69    92.83    88.65    0.00    0.00    0.00        81.33        79.41        83.35    0.57   

I set max_training_steps to 0 and max_epochs is also 0 (which means unlimited for both), so the patience parameter now controls early stopping, but we'll see if that helps or is just a continued trend of the above for even longer training times...

[svlandeg]

@lashmore : hm, that certainly doesn't look like the model is really training/learning. I doubt increasing patience will help anything.

Can you share your config just for my information? And how many examples are there in training and in dev?

[lashmore]

I agree @svlandeg! It's been running for longer and the lack of learning continues!

E    #       LOSS ENTIT...  SENTS_F  SENTS_P  SENTS_R  ENTS_F  ENTS_P  ENTS_R  NEL_MICRO_F  NEL_MICRO_R  NEL_MICRO_P  SCORE 
---  ------  -------------  -------  -------  -------  ------  ------  ------  -----------  -----------  -----------  ------
  0       0           1.08    90.69    92.83    88.65    0.00    0.00    0.00        83.73        81.75        85.81    0.58
  0     200          62.05    90.69    92.83    88.65    0.00    0.00    0.00        81.27        79.34        83.28    0.57                                                                              
  0     400          50.64    90.69    92.83    88.65    0.00    0.00    0.00        81.17        79.25        83.19    0.57                                                                              
  0     600          51.67    90.69    92.83    88.65    0.00    0.00    0.00        81.34        79.42        83.36    0.57                                                                              
  0     800          49.24    90.69    92.83    88.65    0.00    0.00    0.00        81.06        79.14        83.07    0.57                                                                              
  0    1000          50.56    90.69    92.83    88.65    0.00    0.00    0.00        81.33        79.41        83.35    0.57                                                                              
  0    1200          48.11    90.69    92.83    88.65    0.00    0.00    0.00        81.54        79.61        83.56    0.57                                                                              
  0    1400          53.94    90.69    92.83    88.65    0.00    0.00    0.00        81.03        79.11        83.04    0.57                                                                              
  0    1600          48.79    90.69    92.83    88.65    0.00    0.00    0.00        81.95        80.02        83.99    0.57                                                                              
  0    1800          47.96    90.69    92.83    88.65    0.00    0.00    0.00        81.21        79.29        83.23    0.57                                                                              
  0    2000          51.66    90.69    92.83    88.65    0.00    0.00    0.00        80.98        79.06        82.99    0.57                                                                              
  0    2200          49.63    90.69    92.83    88.65    0.00    0.00    0.00        80.94        79.03        82.95    0.57                                                                              
  0    2400          51.35    90.69    92.83    88.65    0.00    0.00    0.00        81.28        79.36        83.30    0.57                                                                              
  0    2600          48.78    90.69    92.83    88.65    0.00    0.00    0.00        81.27        79.34        83.28    0.57                                                                              
  0    2800          48.35    90.69    92.83    88.65    0.00    0.00    0.00        81.89        79.95        83.92    0.57                                                                              
  0    3000          50.02    90.69    92.83    88.65    0.00    0.00    0.00        81.76        79.83        83.79    0.57                                                                              
  0    3200          46.28    90.69    92.83    88.65    0.00    0.00    0.00        81.54        79.61        83.56    0.57   

Here's how training is called:

#!/bin/bash

gpu_id="${6:--1}"

# Runs training. Expects as arguments:
#   (1) dataset ID,
#   (2) run name,
#   (3) language,
#   (4) config file name,
#   (5) max. steps.
#   (6) GPU information if GPU is to be used.
PYTHONPATH=scripts python -m spacy train configs/$4 \
          --paths.dataset_name $1 \
          --output training/$1/$2 \
          --paths.train corpora/$1/train.spacy \
          --paths.dev corpora/$1/dev.spacy \
          --paths.kb wikid/output/$3/kb \
          --paths.base_nlp wikid/output/$3/nlp \
          --paths.language $3 \
          --training.max_steps $5 \
          -c scripts/custom_functions.py \
          --gpu-id $gpu_id

And the bash script is called like:

bash scripts/train.sh "mewsli_9" "cg-default" "en" "nel.cfg" 0 0

Here is my complete config:

[paths]                                                                                                                                                                                                   
dataset_name = null                                                                                                                                                                                       
language = null                                                                                                                                                                                           
train = ""                                                                                                                                                                                                
dev = ""                                                                                                                                                                                                  
raw = null                                                                                                                                                                                                
init_tok2vec = null                                                                                                                                                                                       
kb = ""                                                                                                                                                                                                   
base_nlp = ""                                                                                                                                                                                             
vectors = "${paths.base_nlp}"                                                                                                                                                                             
                                                                                                                                                                                                          
                                                                                                                                                                                                          
[system]                                                                                                                                                                                                  
seed = 342                                                                                                                                                                                                
gpu_allocator = null                                                                                                                                                                                      
                                                                                                                                                                                                          
                                                                                                                                                                                                          
[nlp]                                                                                                                                                                                                     
lang = "en"                                                                                                                                                                                               
pipeline = ["senter","ner","entity_linker"]                                                                                                                                                               
disabled = []                                                                                                                                                                                             
before_creation = null                                                                                                                                                                                    
after_creation = null                                                                                                                                                                                     
after_pipeline_creation = null                                                                                                                                                                            
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}                                                                                                                                                          
                                                                                                                                                                                                          
                                                                                                                                                                                                          
[components]                                                                                                                                                                                              
                                                                                                                                                                                                          
[components.senter]                                                                                                                                                                                       
source = "${paths.base_nlp}" 

[components.ner]                                                                                                                                                                                          
source = "${paths.base_nlp}"                                                                                                                                                                              
component = "ner"

[components.entity_linker]
factory = "entity_linker"
entity_vector_length = 64
incl_context = true
incl_prior = true
labels_discard = []
get_candidates = {"@misc":"spacy.CandidateGenerator.v1"}

[components.entity_linker.model]
@architectures = "spacy.EntityLinker.v1"
nO = null

[components.entity_linker.model.tok2vec]
@architectures = "spacy.HashEmbedCNN.v1"
pretrained_vectors = null
width = 96
depth = 2
embed_size = 2000
window_size = 1
maxout_pieces = 3
subword_features = true


[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null

[initialize.components]

[initialize.components.entity_linker]

[initialize.components.entity_linker.kb_loader]
@misc = "spacy.KBFromFile.v1"
kb_path = ${paths.kb}

[initialize.tokenizer]

[corpora]

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}


[training]
train_corpus = corpora.train
dev_corpus = corpora.dev
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.2
patience = 10000
eval_frequency = 200
accumulate_gradient = 2
max_epochs = 0
max_steps = 500
annotating_components = ["senter"]
frozen_components = ["senter","ner"]
before_to_disk = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = true

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
nel_micro_p = 0.0
nel_micro_r = 0.0
nel_micro_f = 1.0


[pretraining]


[optimizer]
@optimizers = "Adam.v1"
learn_rate = 0.001
beta1 = 0.9
beta2 = 0.999
L2 = 0.0
eps = 0.00000001
grad_clip = 1.0
L2_is_weight_decay = true
use_averages = true

And here is the info on my corpora:

import spacy
nlp = spacy.load("en_core_web_lg")
from spacy.training import Corpus
corpus = Corpus("corpora/mewsli_9/train.spacy")
train_data = corpus(nlp)
len([el for el in train_data])    # Output: 6336

corpus = Corpus("corpora/mewsli_9/dev.spacy")
dev_data = corpus(nlp)
len([el for el in dev_data])    # Output: 2536

corpus = Corpus("corpora/mewsli_9/dev.spacy")
test_data = corpus(nlp)
len([el for el in test_data])    # Output: 3804

I used the pipeline as specified here, but for the download_mewsli pipe I had to do some sleuthing about how to map the SVN revision number to a Git commit in the repo it was downloading files from. I believe I did it correctly / that the corpora were created as desired. And according to Table 1 here, the doc counts match up if you sum all 3 data partitions, you'll get the # docs reported for English, which this pipeline seems only to extract English data (I could tweak the download_mewsli9 pipe to be multilingual data instead but I didn't). What do you see?

[svlandeg]

What confuses me, is that you have nel_micro_f = 1.0 in your weights config, but yet the score from the original output log doesn't correspond 1-1 with the NEL_MICRO_F column. That's a detail though, and not related to the EL training (or absence of it).

That said, the F-score is actually pretty high (>80%) and the loss really low, making me wonder whether the EL ML model actually has a job left to do. What I mean by that is this: the EL component has two parts: 1) candidate generation and 2) ML-based reranker. Part 1 is a dictionary-based approach with the KB, it's only Part 2 that is actually learned. Imagine you having a dataset where each Named Entity would only have 1 candidate ID, then in fact the reranker has no job left to do, and whatever similarity-based ranking it attempts to learn, wouldn't make a difference.

This requires looking more into the data: what kind of entities are in your .spacy files? What IDs are in the gold data? How many candidates would each entity get from the KB? (just a quick manual inspection of a handful of examples from your training data will give you some idea).

[lashmore]

@svlandeg - so thankful to be talking to someone "in the know"! I admit, I'm still learning a lot of the "guts" of some spacy classes, like Corpora and InMemoryLookupKB, etc, but here's what I attempted to do to answer your questions about the Examples in the training data and how many candidates are produced from the KB

Here's the first Example from the train.spacy Corpus, which I believe is a generator object housing Examples.

import spacy
nlp = spacy.load("en_core_web_lg")  # the base model NEL was trained with

from spacy.kb import InMemoryLookupKB
from spacy.training import Corpus

corpus = Corpus("corpora/mewsli_9/train.spacy")
train_data = corpus(nlp)
ex = next(iter(train_data))
print(ex.reference)   # Outputs: "Two men fined over 2006 German train crash that killed 23. Two men have been fined for their failure to prevent the 2006 Lathen maglev train accident in Germany..." 
print(ex.reference.ents)                                                                                # Outputs:   (Transrapid, Osnabrück)
print([cat for cat in ex.get_aligned_ner() if cat!="O"])                              # Outputs:   ['U-NIL', 'U-NIL']
print([tok.ent_kb_id_ for tok in ex.reference if tok.ent_kb_id_ != ""])     # Outputs:   ['Q157018', 'Q2916']
print([tok.ent_kb_id_ for tok in ex.predicted if tok.ent_kb_id_ != ""])     # Outputs:   []   <- Sanity check, as training hasn't started

kb = InMemoryLookupKB(vocab=nlp.vocab, entity_vector_length=64)
kb.from_disk("wikid/output/en/kb/")

# The first entity generates only 1 candidate from the KB, as you guessed 
cand = kb.get_candidates(ex.reference[89:90])  # Corresponds to "Transrapid" entity Q157018 in reference doc
print(len(cand))            # Outputs: 1
print(cand[0].entity)    # Outputs: 15928567685345592367
print(cand[0].entity_)  # Outputs: `Q157018`

# The second entity generates 11 candidates from the KB
cand = kb.get_candidates(ex.reference[115:116])  # Corresponds to "Osnabrück" entity Q2916 in reference doc
print(len(cand))            # Outputs: 11
print(cand[0].entity)    # Outputs: 14312696856104742181
print(cand[0].entity_)  # Outputs: `Q2916`
print(cand[1].entity_)  # Outputs: 'Q5940'

Is it possible that the InMemoryLookupKB entity vectors are just THAT good already? How does the KB compute the vectors behind the scenes as it was parsing Wikipedia/Wikidata?

[svlandeg]

That's a great start to dig more into the data! Just as a small comment, you don't have to loop through your tokens in ex.reference, you could just run through ex.reference.ents and for each entity in it (which will be of type Span) print the kb_id - that's slightly cleaner than printing tok.ent_kb_id_ especially as one span can consist of multiple tokens.

Now that you have this code, you could probably have it run on your entire dataset, fetch for each entity the list of candidates like you're doing here, and count the len of those list, to obtain some sort of frequency list: how many entities have one candidate? how many have 11? etc. Would be interesting to better understand the challenge at hand.

Is it possible that the InMemoryLookupKB entity vectors are just THAT good already?

Here's another experiment I would run if I were you to get to the bottom of this. Go back to your config file which reads

[components.entity_linker]
...
incl_context = true
incl_prior = true

and run two more training steps.
Once with

incl_context = false
incl_prior = true

This won't learn anything, this setting basically disables the ML component (step 2 I wrote about before) - it will just take the prior probabilities of the candidates. Then we'll see whether this in fact already obtains 80% F-score by itself.

And then:

incl_context = true
incl_prior = false

This ONLY uses the similarity ML component to rerank in step 2 -> it won't use the prior probability. I'd be interested to see whether the learning is still "flat" when setting this. If it is, it might still be because there's usually only 1 candidate, or something else is going on still.

How does the KB compute the vectors behind the scenes as it was parsing Wikipedia/Wikidata?

If I remember correctly, the vectors represent the description we get from Wikidata, e.g. "district in Lower Saxony, Germany" for https://www.wikidata.org/wiki/Q5940 and "German city" for https://www.wikidata.org/wiki/Q2916. These descriptions are encoded using the base vector model, and the similarity component then compares that vector to the sentence in which the entity is used. Usually, the Wikidata descriptions are pretty concise and informative, and if you use e.g. en_core_web_lg as vector model, that should already be pretty capable of encoding the signal of these descriptions.

[lashmore]

Great ideas for debugging this @svlandeg!

Here's the distribution of candidate counts across the training Corpus's entities:

import spacy
from spacy.kb import InMemoryLookupKB
from spacy.training import Corpus
from tqdm import tqdm 

nlp = spacy.load("en_core_web_lg")
corpus = Corpus("corpora/mewsli_9/train.spacy")
train_data = corpus(nlp)
kb = InMemoryLookupKB(vocab=nlp.vocab, entity_vector_length=64)
kb.from_disk("wikid/output/en/kb/")

it = iter(train_data)
cands_lens = []
for ex in tqdm(it):
    for ent in ex.reference.ents:
        cand = kb.get_candidates(ent)
        cands_lens.append(len(cand))

from collections import Counter
dist_of_lens = Counter(cands_lens)
del dist_of_lens[0]    # delete counts associated with 0 candidates
print(dist_of_lens)

Results in:

Counter({1: 8991,
         2: 2299,
         3: 1227,
         4: 762,
         5: 544,
         6: 399,
         8: 318,
         7: 315,
         9: 306,
         10: 253,
         12: 191,
         13: 168,
         14: 168,
         16: 144,
         19: 134,
         11: 132,
         21: 130,
         15: 116,
         26: 115,
         17: 115,
         20: 113,
         22: 107,
         18: 95,
         1664: 88,
         56: 88,
         28: 85,
         24: 85,
         23: 79, .... })

So 8991 (39.3% of entities) of them only have 1 candidate, but a total of 13858 (60.6%) entities have >1 candidate. This doesn't account for possible duplicate entities in the training corpus, but I would have had to do NEL to write that code to begin with, so this is the best snapshot I can think of ha ha.

Ran with incl_context = false and incl_prior = true and got:

E    #       LOSS ENTIT...  SENTS_F  SENTS_P  SENTS_R  ENTS_F  ENTS_P  ENTS_R  NEL_MICRO_F  NEL_MICRO_R  NEL_MICRO_P  SCORE 
---  ------  -------------  -------  -------  -------  ------  ------  ------  -----------  -----------  -----------  ------
  0       0           1.08    90.69    92.83    88.65    0.00    0.00    0.00        86.67        84.62        88.82    0.59
  0     200          62.05    90.69    92.83    88.65    0.00    0.00    0.00        86.62        84.57        88.77    0.59
  0     400          50.64    90.69    92.83    88.65    0.00    0.00    0.00        86.67        84.62        88.82    0.59
  0     600          51.68    90.69    92.83    88.65    0.00    0.00    0.00        86.67        84.62        88.82    0.59

Ran with incl_context = true and incl_prior = false and got the below. I'm surprised it's not learning much either!

E    #       LOSS ENTIT...  SENTS_F  SENTS_P  SENTS_R  ENTS_F  ENTS_P  ENTS_R  NEL_MICRO_F  NEL_MICRO_R  NEL_MICRO_P  SCORE 
---  ------  -------------  -------  -------  -------  ------  ------  ------  -----------  -----------  -----------  ------
  0       0           1.08    90.69    92.83    88.65    0.00    0.00    0.00        46.81        45.70        47.97    0.45
  0     200          62.05    90.69    92.83    88.65    0.00    0.00    0.00        48.18        47.05        49.38    0.46
  0     400          50.64    90.69    92.83    88.65    0.00    0.00    0.00        47.99        46.86        49.18    0.46
  0     600          51.68    90.69    92.83    88.65    0.00    0.00    0.00        48.16        47.02        49.35    0.46
  0     800          49.23    90.69    92.83    88.65    0.00    0.00    0.00        47.89        46.75        49.07    0.46
  0    1000          50.54    90.69    92.83    88.65    0.00    0.00    0.00        48.28        47.13        49.47    0.46
  0    1200          48.09    90.69    92.83    88.65    0.00    0.00    0.00        48.06        46.92        49.25    0.46
  0    1400          53.94    90.69    92.83    88.65    0.00    0.00    0.00        48.38        47.24        49.58    0.46
  0    1600          48.81    90.69    92.83    88.65    0.00    0.00    0.00        49.26        48.10        50.49    0.46
  0    1800          47.86    90.69    92.83    88.65    0.00    0.00    0.00        49.19        48.02        50.41    0.46
  0    2000          51.51    90.69    92.83    88.65    0.00    0.00    0.00        48.65        47.50        49.86    0.46

Another thing I noticed is that this benchmarking NEL code was written 2 years ago (commit was dated July 4, 2022), so at best they were developing in Spacy v3.3.1, and I also see in the requirements for the entire repo that spacy should be spacy>=3.0.0,<3.6.0. I'd previous been running training with Spacy v. 3.6.1 in a Conda environment with Python v. 3.10. I wondered if something happened to the NEL training code after 3.6.0 that's causing the above behavior, so I tested Spacy v 3.5.4 and got the same training behavior for the first three evals. i.e., it doesn't appear to be a package version issue:

E    #       LOSS ENTIT...  SENTS_F  SENTS_P  SENTS_R  ENTS_F  ENTS_P  ENTS_R  NEL_MICRO_F  NEL_MICRO_R  NEL_MICRO_P  SCORE 
---  ------  -------------  -------  -------  -------  ------  ------  ------  -----------  -----------  -----------  ------
0       0           1.08    90.69    92.83    88.65    0.00    0.00    0.00        83.73        81.75        85.81    0.58
0     200          62.05    90.69    92.83    88.65    0.00    0.00    0.00        81.27        79.34        83.28    0.57
0     400          50.64    90.69    92.83    88.65    0.00    0.00    0.00        81.17        79.25        83.19    0.57

[lashmore]

OKAY. Sorry for my 6 month hiatus. Work priorities shift and you have to flow with it sometimes.

You were on to something with the NEL hypothesis. What I did was hack the scripts/compile_corpora.py pipeline (specifically in scripts/datasets/utils.py in the create_spans_from_doc_annotations() by accepting the entity label as given by en_core_web_lg. Previously, it was just setting label to NIL for all annotations, because Mewsli-9 didn't explicitly label this (I guess). But Spacy's entity linker ignores all entities tagged as NIL, from what I think I understand. When I did this hack, the NEL portion is actually learning!

    ent_labels = []
    for i, annot_data in enumerate(
        sorted([{"annot": annot, "count": entities_info[annot.entity_id].count if annot.entity_id in entities_info
                    else -1} for annot in annotations
            ], key=lambda a: a["count"],  reverse=True)): 
        ....
        if not overlaps:
            ent_idx = [idx for idx, el in enumerate(sorted(doc_ents_idx)) if (annot.start_pos, annot.end_pos) in [el]]
            if len(ent_idx)>0:
                ent = doc.ents[ent_idx[0]]
                ent_labels.append(ent.label_)
            else:
                ent_labels.append("NIL")
            doc_annots.append(annot)

This at least enables the ENT F1's and the NEL F1's are officially improving, at least on the small subset of the training data I've run on so far. Will test with full dataset to verify it's solved! The ENTS_F aren't improving, but I think that's because I'm setting use_gold_ents = true. Does that sound right to you @svlandeg or does this still smell "off" to you?

[components.entity_linker]
...
use_gold_ents = true

E    #       LOSS ENTIT...  SENTS_F  SENTS_P  SENTS_R  ENTS_F  ENTS_P  ENTS_R  NEL_MICRO_F  NEL_MICRO_R  NEL_MICRO_P  SCORE 
---  ------  -------------  -------  -------  -------  ------  ------  ------  -----------  -----------  -----------  ------
  0       0           0.00    97.56    96.77    98.36   15.20    8.27   94.12        80.00        75.00        85.71    0.64
  6     200          56.77    97.56    96.77    98.36   15.20    8.27   94.12        86.67        81.25        92.86    0.66
 13     400          32.85    97.56    96.77    98.36   15.20    8.27   94.12        90.00        84.38        96.43    0.67

[svlandeg]

Yep that sounds right - if your config has "ner" in its frozen_components, then effectively the weights of that model won't be updated.

Happy to hear we got this sorted!

[lashmore]

@svlandeg , so I ran on the full Mewsli-9 training data and I am getting the same behavior as 6 months ago - actually more like training is hurting performance than staying the same - but given the success of the small portion of the training data, I am now suspicious the learning rate is not optimal for the full training data OR that the task is just TOO HARD to learn anything and the candidate ranking is going to be the strongest signal. I think Mewsli covers like 30k+ entities. Can a neural model really do very good with that? Maybe for 100 entities but for 30k entities? What's your take? We actually have 2 use cases: full Mewsli-9 model and subsetting by project on a list of entities of interest (which will require augmentation of Mewsli-9 if not found). So this isn't necessarily a bad conclusion.

E    #       LOSS ENTIT...  SENTS_F  SENTS_P  SENTS_R  ENTS_F  ENTS_P  ENTS_R  NEL_MICRO_F  NEL_MICRO_R  NEL_MICRO_P  SCORE 
---  ------  -------------  -------  -------  -------  ------  ------  ------  -----------  -----------  -----------  ------
0       0           0.00    98.29    97.71    98.88   12.57    6.78   85.13        82.40        80.13        84.81    0.64
0     200          41.52    98.29    97.71    98.88   12.57    6.78   85.13        80.81        78.58        83.17    0.63
0     400          40.46    98.29    97.71    98.88   12.57    6.78   85.13        80.30        78.09        82.65    0.63
0     600          42.52    98.29    97.71    98.88   12.57    6.78   85.13        79.83        77.63        82.16    0.63        

[svlandeg]

@lashmore: From recent projects I've done with Entity Linking, I've also started coming to the conclusion that often a good candidate generation algorithm is more important for overall performance than anything else.

In terms of EL algorithms, the other thing you could try (instead of training a supervised ML model) is relying on transformers and measuring sentence/context similarity between the entity descriptions and your text (e.g using https://github.com/UKPLab/sentence-transformers). Depending on your application you can tune a bit how those descriptions look like and which text exactly you'd take (the sentence, all sentences of the same mention, the paragraph, the full article....). I've come to the conclusion recently that it's probably too ambitious to expect a supervised EL model to be able to outperform a more generally trained transformer for this kind of text similarity task.

[lashmore]

Thanks so much for weighing in! I think we have some great data / ideas to move forwards with. Major ups for all of your excellent help and support!

[Luisgarchi]

Hello maintainers and contributors,

I am following this forum to implement the nel benchmark and got the following error on the wikid_parse step.

Looking at the error and examining the code in __init__.py, wikidata.py and wikipedia.py within the wikid/scripts/wiki directory it appears to me, to have successfully finished executing all the steps (given that the function read_texts is the last step, which finishes executing as per the print statement "Processed 23644888 articles" shown in the terminal before the error).

The size of my sqlite DB stands at 20.2 GB which seems to correspond to another person in this forum but not the 12GB mentioned by rmitsch. I also ran some sanity check queries on the main tables of the db and everything seems to be in order.

Machine specs: RAM 32GB, CPUs 16, Storage 500 GB (wikid_parse took ~ 24 hours)

• Is it safe to proceed ignoring this error?
• What can we attribute this error to? (My guess is python / weasel is trying to check the integrity of some file using md5 but it is to big to load into memory, since I dont know the path I assume it is something "big" like either the db or the dumps).
• Can I expect to run into memory issues in the create_kb step if using only 32GB.

Persisting alias-entity prior probabilities: 100%|█| 28351143/28351143 [
Parsing article texts: 6717273it [4:11:21, 445.39it/s] 
Processed 23644888 articles.
  Of which viable (with article ID and text): 6717273 (28.41%)
    Of which processed (title in entity table): 6717273 (100.00%)
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/spacy/cli/_util.py", line 87, in setup_cli
    command(prog_name=COMMAND)
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/typer/core.py", line 783, in main
    return _main(
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/typer/core.py", line 225, in _main
    rv = self.invoke(ctx)
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/typer/main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/weasel/cli/run.py", line 42, in project_run_cli
    project_run(
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/weasel/cli/run.py", line 113, in project_run
    update_lockfile(current_dir, cmd)
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/weasel/cli/run.py", line 270, in update_lockfile
    data[command["name"]] = get_lock_entry(project_dir, command)
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/weasel/cli/run.py", line 287, in get_lock_entry
    outs = get_fileinfo(project_dir, command.get("outputs", []))
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/weasel/cli/run.py", line 308, in get_fileinfo
    md5 = get_checksum(file_path) if file_path.exists() else None
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/weasel/util/hashing.py", line 33, in get_checksum
    return hashlib.md5(Path(path).read_bytes()).hexdigest()
  File "/usr/lib/python3.10/pathlib.py", line 1127, in read_bytes
    return f.read()
MemoryError```