Architecture
This is a description of the Tilezen architecture as it was running at Mapzen in production. Of note is that this is meant to capture the various systems as they were running before rawr tiles and global builds.
The system that emerged reflects the choices and trade-offs that were made given our goals. Typically, the variables that affect most of the decisions are acceptable latencies, both typical and worst case, data freshness, and cost. We wanted to serve tiles that were usually just a few hours out of date, with most of the service being reasonably fast, while it being acceptable for a small percentage of tiles to be slow to render.
The way we chose to balance the goals was to create a concept called the toi, or tiles of interest. This represented a set of tiles that should be fast, and is the list of tiles that were most requested in a particular time window. This is the set of tiles that was pre-generated. Requests for tiles outside of this set would be served live.
Requests are satisfied by first trying a cache, then if the tile is pre-generated, and finally it is rendered on-demand if necessary. fastly would run this logic, and the tapalactl service would return a 404 if the tile didn't exist, which would signal to fastly that the request should be directed to tileserver to generate the tile on-demand.
osm2pgql applies diffs to the database, and it would generate a tile expiry list. tilequeue (tilequeue intersect command) would read this list, and the tiles that were in the toi would get enqueued onto sqs for processing. A number of instances running tilequeue process would read tiles from sqs, generate them and store them onto s3.
To manage the toi, a gardener process (tilequeue prune-tiles-of-interest command), would run periodically to add and remove tiles from the toi based on a rolling window of requests. New tiles would get enqueued for processing, and tiles to be removed would get deleted from s3.