"What the fuck are you talking about?": The story of NodeTiles
I recently lost access to my 10-year old Code for America email address when I changed jobs. I tried to preserve some of the good stuff and this is one story.
During our Fellowship, Jessica Lord gave a tech talk on building custom map tiles and first introduced me to TileMill. I couldn’t find a record in my email of when that happened, but know she planted this seed 🌱
On April 25, 2012, I sent this email to Tom Carden:
This week I came across a gist of yours and eventually made my way to nodemap while researching how to generate server-side map tiles. Thank you so much for putting those pieces together; they’re fantastic! I matched your code with node-canvas-heroku to get it running in the cloud and it’s pretty fantastic:
Anyways, just wanted to send you a big thanks for the code and inspiration. I’m working on building a node-based utfgrid renderer to bootstrap an entire wax interaction layer… all without Mapnik (though maybe I should just work on packaging that for Heroku).
Tom gave some helpful advice and caveat emptor about JS performance and data caching. He continued to follow up with node-canvas and cairo suggestions for getting stuff running on Heroku. Tom ended up being a Code for America mentor for a different project and he was the source of us using the Toner map style in demos and also a nice leaflet clustering plugin.
On May 2, 2012, I sent this email to “Fuzz”, Code for America’s random-nerdsnipe list:
Last week Rob and I built a heroku-compatible map-tile renderer (and UTFGrid generator) that can quickly (and with low bandwidth/client-processing) serve interactive maps and markers comprising tens of thousands of points and polygons—and unlike TileMill, the final map can be connected to a live database and updated on-the-fly. It’s kind’ve badass.
Here are 2 examples:
Static map of San Francisco streets and Parks: http://morning-spring-2292.herokuapp.com/
Map of all ~1.5k Occupy sites worldwide (I did this in about 15min, so it’s a little clunky): http://occupy-map.herokuapp.com/
In terms of why it’s badass (from less to more geeky):
- It’s fast. Nothing is rendered in the browser, instead all of the tiles are rendered and streamed to the browser as PNGs.
- It’s small. No shapefiles are sent to the browser. Each Map (the entire initial pageload) is less than 500kb.
- It can do dynamic data. TileMill makes beautiful maps, but once you export them to MapBox (or your own mbtile server), you can’t change them. This can be hooked up to a live database (right now it parses geojson).
- It runs on Heroku. It uses node-canvas (not Mapnik) so it is fast, lightweight and has few dependencies.
- It renders UTFGrids, which can provide mouse-hovers and other straightforward UI stuff that most maps don’t (way less clicky)
- It’s not a tileserver, it’s a tile renderer. Seriously, all those maptiles are being generated on on-the-fly (we should probably cache them, but hey, this is bleeding edge)
- It’s nodejs and all the code is on Github: https://github.com/bensheldon/nodetiles
I’m still working on it, so if you have a big dataset that you’ve had trouble mapping before, let me know since I’m looking for more/better use cases.
Prashant Singh, another Code for America fellow, was excited about the project and gave me lots of feedback. Prashant was using my project to generate large static maps from the commandline, and he gave some feedback on his experience doing similar work with Mapnik. Prashant and two other Fellows, Alicia Rouault and Matt Hampel, would subsequently build a business on top of it.
On June 5, 2012 I sent this email to Eric Gundersen at Mapbox:
I haven’t done a big writeup yet. I’m hoping to submit to NACIS for their October conference, but I’m still trying to clean up the code, modularize it, and write tests and documentation.
General backstory: I had been playing around a lot with TileMill and Mapnik and trying to programmatically generate map tiles; at CfA we have a lot of big, dynamic data and I was trying to set up an automated tile renderer. Then I came across a Tom Carden project where he was use Node-Canvas to render tiles… and then I discovered a node-canvas module that would run on Heroku… and by that point I was in love with the idea of a low-dependency (compared to mapnik) on-the-fly tile rendering implementation.
I think the next step (in addition to cleanup and documentation) is loading up some OSM metro data and actually seeing how the performance compares to Mapnik. Also, while I can’t foresee implementing something like Cascadenik, node-canvas gives you direct access to the graphics renderer and it might be useful for creating boutique maps with graphical flourishes that Mapnik hasn’t implemented… and of course for creating layers with dynamic data.
Let me know if you have any specific questions or ideas you’d like me to try to implement or use as demos. Thanks for the interest!
Alex Yule, another Code for America Fellow, and an actual experienced cartographer, was also interested. Alex would build a proper database adapter layer and the actual spatial indexing functionality that all of my previous emails yolo. Alex joined Rob and my presentation proposal at NACIS. This was our abstract:
NodeTiles: Joyful Map Tile Rendering with Node.js
Afterwards, I remember riding BART back from the airport when we arrived home in San Francisco. I was talking with a maybe-a-big-deal person from the map-tech community, and they asked me something pointed like “But how much did you really contribute to NodeTiles?” I said something like “It was a team effort” and then it got really quiet. In hindsight that was probably a job interview. And also fuck that, it was a team effort.
During Fall of 2012 Code for America Fellowship was ending and interviewing as an inexperienced engineer is freaking hard (and also incredibly dumb, but those are different stories). So I mostly put down NodeTiles. And also:
- Going from something that’s a tech demo to something understandable (“What the fuck are you talking about”) and usable by outsiders is hard.
- The sheer difficulty of spatial work. That dumb NodeTiles logo took me hours just to get the tooling together to just like, draw straight lines on a globe to say “NODETILES”.
- Maps are a commodity, and also if you need to dynamically map a billion+ polyons (I’ve been informed that today there is Postgres MVT) and you need a pipeline for auth and anti-scrape and billing and raster and…. well, god bless you.
- Given that this is an email-based retrospective, I will admit I found some emails from kinda big deals like Stamen that I didn’t respond to. I dunno why I didn’t respond; probably burnout. If that was you, it wasn’t personal, and I’m happy with the path I’ve taken, and I hope you are too 🤗
NodeTiles is still up on GitHub. If you can get node-canvas linking to cairo (memories of this seems like a fever dream), NodeTiles is really cool. Maybe you’ll do something neat with Nodetiles ❤️