During import, continent worker shuts down unexpectedly

[jpan127]

Importing ALL

I am trying to run an import for geonames using npm start. It runs fine for ~10 minutes then always eventually crashes.

error: [wof-pip-service:master] continent worker exited unexpectedly with code null, signal SIGKILL
info: [admin-lookup:worker] locality worker process exiting, stats: {"calls":1502669,"hits":0,"misses":1502669}
info: [admin-lookup:worker] county worker process exiting, stats: {"calls":1514365,"hits":0,"misses":1514365}
info: [admin-lookup:worker] region worker process exiting, stats: {"calls":1514988,"hits":0,"misses":1514988}
info: [admin-lookup:worker] neighbourhood worker process exiting, stats: {"calls":997623,"hits":0,"misses":997623}
info: [admin-lookup:worker] country worker process exiting, stats: {"calls":1515020,"hits":387,"misses":1514633}

/home/jpan127/p/geonames/node_modules/pelias-wof-admin-lookup/src/pip/index.js:159
        throw `${layer} worker shutdown unexpectedly`;
        ^
continent worker shutdown unexpectedly
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! pelias-geonames@0.0.0-semantic-release start: `node --max_old_space_size=4096 import.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the pelias-geonames@0.0.0-semantic-release start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/jpan127/.npm/_logs/2018-08-20T14_21_07_183Z-debug.log

There are these errors when it starts running but I ignored it because the documentation said there is no dependency on wof or any other imporeter:

info: [geonames] streaming datafile from: http://download.geonames.org/export/dump/allCountries.zip
error: [wof-pip-service:master] unable to locate /data/whosonfirst/meta/wof-borough-latest.csv
error: [wof-pip-service:master] unable to locate /data/whosonfirst/meta/wof-localadmin-latest.csv
error: [wof-pip-service:master] unable to locate /data/whosonfirst/meta/wof-macrocounty-latest.csv
error: [wof-pip-service:master] unable to locate /data/whosonfirst/meta/wof-macroregion-latest.csv
error: [wof-pip-service:master] unable to locate /data/whosonfirst/meta/wof-dependency-latest.csv
error: [wof-pip-service:master] unable to locate /data/whosonfirst/meta/wof-empire-latest.csv
error: [wof-pip-service:master] unable to locate /data/whosonfirst/meta/wof-marinearea-latest.csv
error: [wof-pip-service:master] unable to locate /data/whosonfirst/meta/wof-ocean-latest.csv

Importing US

Running the geonames importer for US actually throws a different error which made me go with ALL.

> pelias-geonames@0.0.0-semantic-release start /home/jpan127/p/geonames
> node --max_old_space_size=4096 import.js

info: [geonames] streaming datafile from: http://download.geonames.org/export/dump/US.zip
error: [wof-pip-service:master] unable to locate /data/whosonfirst/meta/wof-borough-latest.csv
error: [wof-pip-service:master] unable to locate /data/whosonfirst/meta/wof-localadmin-latest.csv
error: [wof-pip-service:master] unable to locate /data/whosonfirst/meta/wof-macrocounty-latest.csv
error: [wof-pip-service:master] unable to locate /data/whosonfirst/meta/wof-macroregion-latest.csv
error: [wof-pip-service:master] unable to locate /data/whosonfirst/meta/wof-dependency-latest.csv
error: [wof-pip-service:master] unable to locate /data/whosonfirst/meta/wof-empire-latest.csv
error: [wof-pip-service:master] unable to locate /data/whosonfirst/meta/wof-marinearea-latest.csv
error: [wof-pip-service:master] unable to locate /data/whosonfirst/meta/wof-ocean-latest.csv
info: [wof-pip-service:master] starting with layers neighbourhood,locality,county,region,country,continent
info: [wof-pip-service:master] county worker loaded 1 features in 1.03 seconds
info: [wof-pip-service:master] region worker loaded 1 features in 1.006 seconds
info: [wof-pip-service:master] country worker loaded 1 features in 1.067 seconds
info: [wof-pip-service:master] locality worker loaded 141 features in 1.047 seconds
info: [wof-pip-service:master] neighbourhood worker loaded 357 features in 1.1 seconds
info: [wof-pip-service:master] continent worker loaded 1 features in 1 seconds
info: [wof-pip-service:master] PIP Service Loading Completed!!!
info: [dbclient]  paused=false, transient=1, current_length=238
info: [dbclient]  paused=false, transient=1, current_length=238
info: [dbclient]  paused=false, transient=0, current_length=370, indexed=27500, batch_ok=55, batch_retries=0, failed_records=0, venue=23915, locality=2918, neighbourhood=613, county=44, localadmin=10, persec=2750
{ Error: unexpected end of file
    at Zlib.zlibOnError [as onerror] (zlib.js:153:17) errno: -5, code: 'Z_BUF_ERROR' }
internal/streams/legacy.js:57
      throw er; // Unhandled stream error in pipe.
      ^

Error: unexpected end of file
    at Zlib.zlibOnError [as onerror] (zlib.js:153:17)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! pelias-geonames@0.0.0-semantic-release start: `node --max_old_space_size=4096 import.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the pelias-geonames@0.0.0-semantic-release start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/jpan127/.npm/_logs/2018-08-20T15_23_26_615Z-debug.log

Extra details

Node : 10.9.0
NPM : 6.2.0
Elasticsearch : 2.4.6
Ubuntu 16.04
My config:

{
  "logger": {
    "level": "info",
    "timestamp": false
  },
  "esclient": {
    "hosts": [{
        "host": "localhost",
        "port": 9200
    }]
  },
  "interpolation": {
    "client": {
      "adapter": "http",
      "host": "http://localhost:9999"
    }
  },
  "imports": {
    "adminLookup": {
      "enabled": true
    },
    "geonames": {
      "datapath": "/data/geonames",
      "countryCode": "ALL"
    },
    "openstreetmap": {
      "datapath": "/data/openstreetmap",
      "import": [{
        "filename": "california.osm.pbf"
      }]
    },
    "openaddresses": {
      "datapath": "/data/openaddresses",
      "files": [
        "us/ca/alameda.csv",
        ...
      ]
    },
    "polyline": {
      "datapath": "/data/polylines",
      "files": [ "edges_2017_01_08-15_25_01.0sv" ]
    },
    "whosonfirst": {
      "datapath": "/data/whosonfirst"
    }
  }
}

[orangejulius]

Hi @jpan127

Thanks for the report. The second issue is #26. There's a workaround listed there but it's also an intermittent issue and may go away if you try again.

For the first one, the only thing I currently know of that causes this issue is running out of memory. The continent worker is probably the smallest in terms of memory footprint, but it could still be killed if your system is running out of memory. Do you see any indication that this is the case? You should be able to see lines in the output of dmesg indicating that Linux is looking for processes to terminate.

Other than that, what version of the code are you using? Those errors were changed to warnings in a recent update, to better indicate the nature of the message. Let me know if using the latest code from the master branch helps.

[jpan127]

Workaround you posted worked. I downloaded the file, unzipped, rezipped, then ran npm start and it imported without error.

I currently only have 8GB allocated for my VM, maybe that is too little? Other processes maybe take up 2GB out of it. I will reattempt failure later with dmesg.

I am using the most up to date in production. 44bd672