Illuminati tile pyramids

[zerebubuth]

👁️△

Follows on from #271.

The idea here is that the output of a tilequeue run on a z10 tile will, in addition to outputting all the internal tiles which have changed, also output a z10 tile containing a simplified version of all the features for which min_zoom for any layer was less than 10. These tiles would be output to S3 and a secondary process would gather up all the tiles under a z5 pyramid and merge them together, triggering a re-render of the z5 pyramid.

The output of that re-render would include a z5 tile containing simplified versions of all features with min_zoom less than 5. These would be gathered into a z0 tile and re-rendered.

Note that the z5 and z0 tiles are specific to the style, as the style assigns the min_zoom. So these shouldn't be written to the same S3 bucket as the original RAWR tiles, although probably worth writing them in the same format.

The hard parts here could be:

1. When gathering z10 output tiles containing features at z < 10 to form a z5 tile, ideally it would be worth waiting until all the z10 tiles have been finished before starting this process. However, that could be a complex problem to solve properly. Some possible ideas:
   • The gather process runs in a queue that's lower priority than all z10 tiles, guaranteeing that they have all been done before the gather runs. The gather would produce jobs at an even lower priority, guaranteeing all the gathers had been run before any z5 tile re-render.
   • The gather process could be run on a cron-like timer. This offers no guarantee that some z5 tiles wouldn't be formed from partially stale data, but does guarantee that updates happen on a certain frequency.
2. Appropriate simplification (and merging) of features at the z10 and z5 levels to avoid file size bloat.
3. Deduplication of features across multiple tiles in the gather, as features may have been included in several tiles. This could have some nasty interactions with any merge processes.

Additionally, updates to tilequeue to fetch the z0 or z5 tiles from S3 rather than going to the database. This should be covered by #270, but worth mentioning here.