Memory and recursion issues in tilelive-copy with --parts

[jflemer-ndp]

Using tilelive-copy with --parts on a large copy seems to have a few issues. In both cases, I was using:
tilelive-copy --concurrency=24 --withoutprogress --timeout=900000 --minzoom=0 --maxzoom=12 --retry=1000 --parts=$PARTS --part=$n mapnik://./project.xml .../foo.mbtiles

Initially, I tried PARTS=1000, and got a Maximum call stack size exceeded. The stack (now gone, sorry) looked to involve a massive recursion on skip()/get(), right about here:

tilelive/lib/stream-scanline.js

Line 120 in efdfe7d

get(push);

I figured even with the recursion, that I'd avoid blowing the stack with PARTS=20, and I did. But each part ended up getting killed or dying due to out of memory. The same command without --parts at all runs just fine (no out of memory errors).

I did not try the pyramid or list scheme, so this may or may not be limited to scanline.

Here are some of the versions in use:

| +-- [email protected]
| | +-- [email protected]
| | `-- [email protected]
| |   +-- [email protected]
| |   +-- [email protected]
| |   `-- [email protected]
| |     +-- [email protected]
| |     | `-- [email protected]
| |     `-- [email protected]
| |       `-- [email protected]
| +-- [email protected]
| | +-- [email protected]
| | `-- [email protected]
| |   `-- [email protected]
| `-- [email protected]

[rclark]

@jflemer-ndp thanks for the report - @jakepruitt also ran into this recently and we'll see if we can pick it apart this week.