Training fails without any error message for most hyperparameter combinations on Windows

[mayankshrivastava]

I've built the package on Windows using GnuWin.

My first issue was with the pre-trained models that were on the GitHub page, when I tried to use them to encode sentences, I got an assertion failed error : Assertion failed: (counts.size() == osz_). I never got around this.

Later, I was trying to train a model on a subset of Wikipedia restricted to a certain domain, so that the embedding is a domain specific one. The training fails without any error shown with most of the hyperparameter combinations, and no bin file is outputted. Only with very select hyperparameter combinations does the training complete - and then also sometimes it gets stuck after getting to 100% and nothing is outputted. The attached image below shows a training cycle which reached a 100% and then hasnt outputted anything for 4 days.

My input dataset is about 221 MB in size, with about 120k words having a word count > 5.

[mayankshrivastava]

For more context, the assertion fails in the following function:

void Model::setTargetCounts(const std::vector<int64_t>& counts) {
assert(counts.size() == osz_);
if (args_->loss == loss_name::ns) {
initTableNegatives(counts);
}
if (args_->loss == loss_name::hs) {
buildTree(counts);
}
}

osz_ has a count of 0, whereas counts.size() is 54455 as per the screenshot above.

[mayankshrivastava]

Also, I built latest version of fastText separately, and trained a skip-gram model using it, and that completed fine, with no such assertion errors.

[tqx94]

Hi, I encountered the same issue as shown in your screenshot as well. May I understand how did you resolve?

[mayankshrivastava]

Hi, I unfortunately was not able to find any solution to this.