Development Seed

Blog

Generating Custom Map Tiles Rapidly in the Cloud

Using Amazon Web Services to Render Map Tiles with Specialized Data

Jul 1, 2009 by Will White

When you’re keeping tabs on an event like the upcoming election in Afghanistan, a basic street map that plots news stories is quite useful. But what could you do with a map that plots those news stories over voting regions that are shaded by poverty rate, literacy rate, or another human development indicator? The effectiveness of a map increases drastically when you add specialized data to the base layer. In this case, not only would you see the hot spots of activity, you could identify possible explanations for the activity.

The maps we’re familiar with are powered by tile sets – collections containing hundreds of thousands of individually rendered images that stitch together to form a larger map view. Tile sets are useful because they allow users to pan and zoom around a map with a web browser, but creating and maintaining a tile set is challenging. Tile generation demands a considerable amount of computing power and can take days depending on the size of the region being rendered. Finished tile sets occupy many gigabytes of disk space, making storage and distribution difficult.

With the help of Amazon Web Services, we’re building an infrastructure capable of generating beautiful interactive maps quickly. We’re using four Amazon services in this workflow: SQS (job queuing), EC2 (tile generation), S3 (storage), and CloudFront (distribution). The figure below illustrates the design.

The steps in the workflow include:

  1. GIS data is collected and managed in a PostgreSQL database.
  2. Designers use Quantum GIS to create beautiful maps with the data.
  3. Quantum GIS generates a configuration file for the map, which is submitted to the SQS job queue.
  4. Render nodes running on EC2 periodically check the queue for unclaimed jobs. When a job is found, they claim it and get to work rendering tiles. More EC2 instances can be created on the fly during times of high volume or to increase speed.
  5. Finished tiles are saved to S3, and when all tiles are complete the render node marks the job completed and becomes ready for another job.
  6. After the tiles are approved, the tile set is bundled up and pushed to Amazon Cloudfront for Internet delivery.
  7. An OpenLayers client embedded in any webpage can access the finished product.

We’re working on firm benchmark numbers for tile generation time and size. An important component to this is establishing a standard base layer, because these figures depend entirely on the complexity and area of the map being rendered. We expect to have more figures to share in mid July. We’re also planning to share the AMI for our service once it is stable. It will include PostgreSQL, Mapnik, TileCache, and all of the custom glue necessary to allow anyone to start rendering tiles with custom data quickly.

For more technical information about creating queued services with AWS, refer to this excellent guide by Mitch Garnaat.

Will, It is not strictly

Jul 11, 2009 by Milver

Will,

It is not strictly free as I assume you need to have a local computer with certain specs to efficiently produce maps. Thinking about corporations, desktops are usually a waste of money since they depreciate really fast. Still, it may not be economically feasible now…but those Amazon hourly costs could go down. I am also wondering if EC2 and EC3 could help in the advent of a true web-based open-source GIS application. Ultimately this could force proprietary GIS packages to be virtualized. That would reduce the need for expensive desktops and corporate licenses…you pay for what you use. Overhead goes down.

Great use of SQS

Jul 2, 2009 by Mitch Garnaat

Nice article. SQS is under-appreciated, I think. It’s nice to see people really take advantage of it.

Mitch

Will, excellent article. A

Jul 2, 2009 by Milver

Will, excellent article. A couple of questions ((I’m a complete noob on all the concepts and implementations you are describing here, so please bear with me): Would it be possible to have the entire system within the Amazon solutions? For instance, could you have the PostGRE database virtualized in EC2, as well as the QGIS client app? If so, what would you think are the advantages and disadvantages of that type of system?

Thanks,

M.

Hey Milver, This is a

Jul 9, 2009 by Will

Hey Milver, This is an interesting idea. PostgreSQL can run on an EC2 instance for sure. That’s what we’re doing here. But since Qgis is a desktop tool, it’s not so straightforward. My feeling is that EC2 is great for running services and completing computation tasks, but it’s not so useful or cost effective for desktop computing.

It may be possible with a remote desktop connection to an EC2 instance, but you’d end up paying by the hour, when you can simply run Qgis locally for free.

We’re running Qgis on our local machines and connecting to a PostgreSQL server that’s running on an EC2 instance.