DASNET

For some time, I've wanted to be able to build my own maps showing cache locations. One of the biggest obstacles was finding a good source of map information. There are plenty of online map sources, but many of them are very old (like the USGS Topographic maps), or really aren't very pleasing to look at (Mapquest, etc.)

Recently, however, Google opened up their beta mapping service (at maps.google.com. It's fast, it's easy, it has terrific-looking maps, and it's actually very easy to grab individual map images directly from Google. Of course, these images are copyrighted -- the data comes from NAVTEQ and TeleAtlas, and the renderings are (presumably) created and copyrighted by Google. So you shouldn't abuse these maps, lest they become harder to grab. And you absolutely shouldn't use them commercially, in any way.

Update 4/29 - The first part of this hack, the map grabber, is now available. Check out the amazingly-underutilized downloads page.

So I began digging into how the maps worked, and it turns out that it's actually fairly simple. The helpful details are buried deep within a javascript file that's downloaded when you load a map. Essentially, Google set a position as the "Center" of the US Map (it may actually be close to the "geographical center" of the continental US, but I'm not sure). This point is defined as the origin on a cartesian plane. Locations east of the center have a positive X value, points west, negative. North is also negative, while south is positive. (a little bit of a flip from traditional geometry).

Google maps are available in several different zoom levels -- from 0 (very very close to the street) to 14 (showing almost all of North America). These zooms are exponentially related -- zoom level 2, for a given size image, shows twice as much area as zoom 1, which shows twice as much as zoom 0. For each level, there are a certain, predictable number of "pixels per degree." Technically, this varies slightly for longitudinal measurements, as the distance between lines of longitude decreases as you approach the poles, but Google uses an approximation that works well for the continental US (and, now that I think of it, is probably a result of the map projection used).

So now you've got 14 different map images, from a fairly small one about 300 pixels across for zoom level 14, to a massive image, several million pixels across, for zoom level 0. Obviously, you can't send the entire image to the browser and expect it to scroll around, so they break the image up into 128x128 pixel tiles. When you drag the map around in your browser, new tiles are added to the edge of the image as necessary, giving the impression of a single, continuous image.

So how does the browser know which tiles to request? That's the magic. Let's walk through it for a map centered on 38 57.000 N by 77 23.000 W. Of course, we should convert that to a decimal measure, which works out to about 38.95 N, 77.383 W.

First, we figure out the relative location for that position (the distance between the desired point and the map center). To make things simple, I'm dropping the sign from all longitude components -- of course, this'll break any tools we build once Google expands beyond the Western Hemisphere, but I'm willing to live with that. Given the pre-defined map center of 98.35 degrees west longitude and 39.5 degrees north latitude, we find that our target location is 20.967 degrees east of center (98.35 - 77.383), and 0.55 degrees south of center (39.5 - 38.95). So the Google coordinates are (20.967, 0.55) degrees.

Now we must convert that measurement into pixels. To do that, we have to know what zoom level we want for our map. Neighborhood streets show up on the map at about zoom level 5, but for this demonstration I want something a little closer, so we'll go with zoom 4 (which shows townhome / apartment complex streets). At zoom level 4, there are 8192 pixels per degree. How'd I get that? There's a nifty little formula buried in Google's javascript. The number of pixels per degree, at zoom Z, is given by 2^(17-Z). In this case, zoom 4 means we're at 2^(17-4) == 2^13 == 8192 pixels per degree. Multiply this number by the Google Coordinate and you get (20.967, 0.55) * (8192) == (171761, 4505) (I dropped the fractions).

But that's not entirely correct, because the map projection that Google uses isn't exactly square. Degrees longitude (X pixels) are slightly closer together than degrees latitude, and Google defines a 14-digit fraction for that ratio. I'm just going to call it 0.771 for now, 'cause I don't feel like typing out the whole thing over and over again. So we multiply the X coordinate by 0.771, and end up with (132427, 4505) as the final location for our target coordinate, on the massive zoom 4 map image.

Finally, we have to figure out what tile that position is on. To get that, we divide each coordinate by 128 (because each tile is 128 pixels on a side). We end up, after all this work, with (1034, 35). So if you try to fetch that tile from Google, like this, you should find a small map image with your target coordinate somewhere inside it (it won't necessarily be in the exact center of the tile, btw).

Now, it's just a question of grabbing tiles around your target tile, and assembling them into a single image. Some perl magic (along with ImageMagick's montage tool) will eventually get you something that looks like this:

Now for the fun part! Geocaching.com offers something called "Pocket Queries," which are pre-defined queries that run automatically and send you, in email, a big horkin' GPX file. Contained within that GPX file are all the caches that match that particular query's parameters. Since it's an XML-formatted file, it's easy to parse with any of a million XML tools. However, since I'm lazy (and not releasing this ugly hack anyway), I just parse the files by reading line-by-line.

A waypoint entry in the GPX file starts with the tag "wpt", which has attributes for the latitude and longitude for that waypoint. Subsequent tags include "time" (the date the cache was placed), "name" (the waypoint ID), and a "groundspeak:cache" tag (that includes an "available" attribute). Also contained within the top-level wpt tag are multiple "groundspeak:log" entries, including "groundspeak:finder", with an "id" attribute identifying the finder for that entry.

By parsing all these, I end up with a simple comma-separated file that looks like this:

GCJD40, 38.997733, -77.0618, 285, True
GC6781, 38.89325, -77.17225, 971, Found
GCGPEK, 38.885717, -77.1735, 548, False


These lines show a waypoint, its coordinates, the age of the cache in days, and a status field. In this case, "True" is for any normal, available cache, "False" means that the cache is for some reason unavailable (maybe it got muggled or washed away), and "Found" means that I personally have found that cache.

Now, finally, I need to plot these points on my maps. To make things easy, my "Grab Map" script embeds the coordinates and zoom level inside the map's comment field. My "Map Caches" script opens up each map in turn, reads the parameters for that map, and figures out which of the caches from my "Parse GPX" script actually appear on that particular map. Then, it uses the GD graphic library module for perl to open the map into memory and add little colored circles wherever I need 'em. The coordinates for each cache is figured along the same lines as determining Google map tiles -- find the distance, in degrees, from the map edge to a point, convert that to pixels, and then draw a circle at that pixel location.

I've also color coded the map. Right now, I'm using green for caches I've found, red for those I haven't, and yellow for caches that are currently unavailable. I also thought it'd be good to see cache age displayed somehow, so for caches I haven't found, blue is brand new (less than 15 days old), cyan is relatively recent (placed within the last 90 days), and purple is for caches placed in the last year. I think I'll also try to display locationless finds as a triangle or something, because I've already confused myself trying to remember what some of the marks represent. Also, I need a better color for unavailable caches (it matches the large roads too closely).

UPDATE: I now use triangles (the international "Lookey Here" symbol) for unavailable caches, with the same red / blue / green color scheme. Not sure what I'll use for locationless, then...

The final result looks something like this:

This was a fun hack, and could actually be relatively useful for me. I currently have a big map of Fairfax County as my computer background, and already I'm planning which caches to visit for my upcoming "March Madness" cache-a-thon (trying to do a cache a day for the month of March). I'd like to get these automated, so every week when new PocketQueries arrive in my email, it automatically generates new maps and updates my desktop background. I'm also thinking about wrapping an HTML file around the map image, so I can hover the mouse over each cache icon and get the cache's name and other useful information. It might also be fun to build an animated map showing caches found over time -- as new caches appear and as I knock them down. If I end up building anything else cool, I'll try to post it here...

UPDATE: I just found, on another site, that you can specify latitude and longitude directly in a URL: http://maps.google.com/?sll=38.950,-77.383. You can't specify the zoom right off (it seems to jump straight to level 4), and there's no marker to show where your requested location actually is, but it's an easy way to get a map based on lat/lon. Even more simply, you can just enter a decimal coordinate in the search field, like so: 38.950, -77.383. Still no marker, but it gets you in the ballpark.