“The Heriarchy of not-for-profit rigor” from White Courtesy Telephone;

At a certain level of experience in some fields, one realizes that “the problem” is not that people are incompetent or bad at their jobs, but rather that they are spectacularly excellent at what they are doing but they happen to be doing the wrong thing. Which is what makes the above chart so enjoyable for me, having reached the level of nonprofit experience in which I can juxtapose the superb competence of those inside the sector with those voices outside the sector braying of its incompetence, while recognizing just how closely aligned both of them are.

From  Why’s (Poignant) Guide to Ruby:

Let’s start with some deep breathing. Give me a good deep breath and count to four with me.

Here we go. 1. 2. 3. 4. Now exhale. You can feel your eyes. Good, that’s exactly it.

Now let’s take a deep breath and, in your mind, draw a hippopotamus as fast as you can. Quick quick. His legs, his folds, his marshmallow teeth. Okay, done. Now exhale.

Take another deep breath and hold it tight. As you hold it tightly in your chest, imagine the tightness is shrinking you down into a bug. You’ve held your breath so hard that you’re an insect. And all the other bugs saw you shrink and they loved the stunt. They’re clapping and rubbing their feelers together madly. But you had an apple in your hand when you were big and it just caught up with you, crushed the whole crowd. You’re dead, too. Now exhale.

Give me a solid deep breath and imagine you live in a town where everything is made of telephone cords. The houses are all telephone cords, the shingles, the rafters. The doorways are a thick mass of telephone cords which you simply thrust yourself through. When you go to bed, the bedspread is telephone cords. And the mattress and box springs are telephone cords, too. Like I said, everything is made out of telephone cords. The telephone itself is made of telephone cords. But the telephone cord going to the telephone is made out of bread and a couple sticks. Now exhale.

Breathe in. 1. 2. 3. 4. Breathe out.

Breath in. 1. 2. Another short breath in. 3. 4. Imagine both of your hands snapping off at the wrists and flying into your computer screen and programming it from the inside. Exhale.

Big, big deep breath. Deep down inside you there is a submarine. It has a tongue. Exhale.

Breathe through your nostrils. Deep breath. Filter the air through your nostrils. Breathing through the nostrils gives you quality air. Your nostrils flare, you are taking breaths of nature’s air, the way God intended. Imagine a floppy disk drive clogged up with orphans. And while it chokes on orphans, you have good, wholesome God’s breath in your lungs. But that pleasurable, life-giving air will become a powerful toxin if held too long. Hurry, exhale God and nature’s air!

Now, you will wake up, smoothing out the creases of this page in your web browser. You will have full recollection of your whole life and not forgetting any one of the many adventures you have had in your life. You will feel rich and renewed and expert. You will have no remembrance of this short exercise, you will instead remember teaching a rabbit to use scissors from a great distance.

And as you will wake up with your eyes directed to the top of this exercise, you will begin again. But this time, try to imagine that even your shadow is a telephone cord.

I’ve spent the last 6 months working at OkCupid Labs building an iOS native mobile app I’m very proud of: Ravel!—”Share Photos. Meet People.” We’ve described it as a dating app for Instagrammers: your public profile is the photos that you pull in from Instagram, Facebook or your camera roll and you can introduce yourself and start chatting with other people based on the photos they share.

We built Ravel! using RubyMotion, a toolchain for writing native iOS apps in (nearly)Ruby. In building it, I learned a lot more both about Ruby and Objective-C along with the intricacies of developing for a mobile device. The mobile app is powered by a Rails-based API server, which is also the backend for the website (for sharing your photo and profile via the web). One of the strengths of RubyMotion is its wonderful automation tools; and the drawbacks is that it’s memory management still has requires some work and workarounds. Perhaps you’ve seen this quote: “It’s not just you. I’m experiencing these memory-related types of crashes (like SIGSEGV and SIGBUS) with about 10-20% of users in production.” That was me on the RubyMotion mailing list.

I also learned a lot of about instrumentation and building a consumer-facing app. Even though a lot of our focus was on in-app engagement, we also invested a lot of time in growth and retention. From optimizing the download  and App Store texts, to adding events and triggers in-app and on the server, to adding sharing and open graph integration through Facebook and on the web.

As I cycle off of Ravel!, time will tell how it survives as a product, but I’m very proud of how we brought it to market.

2012 was a big year for Day of the Shirt: it moved from being one of a constellation of hobby projects to becoming the main focus of my spare development time and a largely self-sustaining business.

As a project

As a technology project, Day of the Shirt went through a lot of changes in 2012. Including:

• It started off the year featuring just 10 t-shirt vendors. By the end of the year, that number had grown to 21 vendors, including some that featured multiple shirt sales.

• The technology stack morphed a lot. Originally built in PHP, I first rewrote the backend in Node.JS (though I was still using PHP for HTML templating). Then, as the year closed out, I began the task of rewriting it for Ruby on Rails.

• In moving Day of the Shirt from PHP to Node (and towards Rails), I improved the instrumentation and error logging a lot. This meant I’m much more quickly able to respond to changes in t-shirt vendors websites and ensure that designs on Day of the Shirt are up-to-date.

• Added a daily RSS Feed

• Started pushing daily updates to Facebook (in addition to Twitter) with about 1,500 likes on the Facebook Page by year’s end.

As a business

2012 marked the first time that Day of the Shirt ever earned any money. Like:

• After 1.5 years of having Google Adsense advertisements on Day of the Shirt, I finally received my first payout in February.

• In addition to showing advertisements, I also started adding affiliate links to shirts. The majority of t-shirt vendors I list don’t have affiliate programs, but I went ahead and signed up as an affiliate for those that do.

• Because I was actually earning some revenue for the first time, in November I completed the process of registering Day of the Shirt as a Sole Proprietorship in the City of San Francisco.

• The income has allowed me to pay a designer to help with the visual and interaction design and a developer to help migrate the t-shirt collection scripts to Ruby: having already migrated a lot of automated scripts from PHP to Javascript, I was happy to lighten my load and focus on building out the website itself.

As a passion

I’ve always juggled several web projects (like Brompt and Panlexicon… and all of this), but 2012 really cemented my commitment and interest in Day of the Shirt:

• Traffic went up, way up. Over the course of the year Day of the Shirt went from having 200 daily visitors in January to more than 8,000 by December.

• People started writing about Day of the Shirt on Reddit and online forums. And tweeting about it. And in November Day of the Shirt was featured on Lifehacker.

• People started emailing me (and we don’t feature our email address very prominently): thanking me for the website and suggesting new daily t-shirt vendors to add to the website.

2012 was a really important year for Day of the Shirt and has set a lot of the foundation for improvements and new features I’m adding in 2013.

Roger Scruton:

Thomas Aquinas, who devoted some two million words to spelling out, in the Summa Theologica, the nature of the world, God’s purpose in creating it and our fate in traversing it, ended his short life (short by our standards, at least) in a state of ecstasy, declaring that all that he had written was of no significance beside the beatific vision that he had been granted, and in the face of which words fail. His was perhaps the most striking example of a philosopher who comes to believe that the real meaning of the world is ineffable. Having got to this point, Aquinas obeyed the injunction of Wittgenstein, whose Tractatus Logico-Philosophicus concludes with the proposition: “that whereof we cannot speak we must consign to silence.”

But Aquinas was exceptional. The history of philosophy abounds in thinkers who, having concluded that the truth is ineffable, have gone on to write page upon page about it. One of the worst offenders is Kierkegaard, who argues in a hundred ways that the ultimate is inexpressible, that truth is “subjectivity,” that the meaning of life can be given by no formula, no proposition, no abstraction, but only by the concrete experience of surrender whose content can never be given in words.

[…]

The temptation to take refuge in the ineffable is not confined to philosophers. Every inquiry into first principles, original causes and fundamental laws, will at some stage come up against an unanswerable question: what makes those first principles true or those fundamental laws valid? What explains those original causes or initial conditions? And the answer is that there is no answer — or no answer that can be expressed in terms of the science for which those laws, principles and causes are bedrock. And yet we want an answer. So how should we proceed?

There is nothing wrong with referring at this point to the ineffable. The mistake is to describe it.

Karl Fogel on The Open Source Report Card:

So, approximately every year or so, someone launches Yet Another Fully Automated Statistical Tool Dashboard Thing that tries to show the skills & activity of a given open source contributor, or the “health” of a given open source project.

The next thing that happens then follows as surely as night follows day, or as unrequited nostalgia follows the latest James Bond release:

People try the tool, and say “Hmm, well, it’s wildly inaccurate about *me* (or about the projects I have expertise in), but maybe it’s useful to someone else.”

And maybe it is. But what’s really going on, I think, is that the developers of these tools are trying to solve too much of the problem.

Investigating the activity of a particular developer, or the health of an open source project, inevitably involves human judgement calls informed by experience and by out-of-band knowledge. These judgements can be tremendously improved by good tool support – one could imagine certain dashboard-like tools that would make such investigations orders of magnitude more convenient and accurate than they currently are.

But because that kind of tool deliberately doesn’t go the last few inches, it’s a lot harder to demo it. There’s no one-size-fits-all final screen displaying conclusions, because in reality that final screen can only be generated through a process of interaction with the human investigator, who tunes various queries leading up to the final screen, based on knowledge and on specific questions and concerns.

And because it’s harder to demo, people are less likely to write it, because, hey, it’s not going to be as easy to post about it on Slashdot or Hacker News :-). Well, also because it’s a lot more work: interactive tools are simply more complex, in absolute terms, than single-algorithm full-service dashboards that load a known data query and treat it the same way every time.

So what I’m saying is: Any tool that tries to do this, but where you just have to enter a person’s name or a project’s name and click “Go”, is going to be useless. If you didn’t shape the query in partnership with the tool, then whatever question the tool is answering is probably not the question you were interested in.

Added: June 10, 2016 - From The Daily WTF’s “The Oracle Effect”:

Work against the Oracle Effect by building software systems that do not provide conclusions, but arguments. Resist throwing some KPIs on a dashboard without working with your users to understand how this feeds into their decision-making. Keep in mind how your software is going to be used, and make sure its value, its reasoning is transparent, even if your code is proprietary. And make sure the software products you use can also answer the important question about every step of their process: Why?

This is the text of a skillshare I delivered at The Sourcery, an awesome commission-less recruiting service. Not only is their model nifty, but they care about actually knowing what they’re recruiting for—so hopefully there isn’t anything too wrong below.

You probably don’t know JavaScript. Like really know javascript.

Javascript is a real programming language: functions, lambdas, closures, objects, inheritance, passing by reference, etc.

Somewhere long, long ago, an arbitrary decision was made that we would “program” a web browser with Javascript. Javascript is good at this (we’ll soon learn why), but there are experimental browsers that use Python (and other languages) to manipulate a web browser window too. There is no inherent reason that Javascript has to be tied to the web browser (or web browsers have to be tied to Javascript).

Unfortunately because the primary context we experience Javascript is in the web browser, we more strongly associate JavaScript with its browser-specific functions/extensions (for manipulating the DOM and listening for UI clicks) than its core language which can exist completely separately from the web browser. Just like we can use Ruby for general purpose programming without using Rails.

If you only think of Javascript in the context of the browser, you’re really missing out; Javascript as a language is badass: lambdas, closures, inheritance, passing by reference.

Why did JavaScript come to dominate the web browser? JavaScript is really good in a browser because as a language it easily supports event-based and asynchronous programming.

• Event-based: when programming a web browser, most of the actions you want to wire up are “when the user clicks on this, perform a different action than when they click on that.” You declare what event(s) you want to listen for and the resulting action/function you want to take place / be called when you “hear” that event take place.

• Asynchronous: when you’re interacting with a web page, you’re downloading new data (or images or video) that might take a second or two (or more for video!) to arrive over the wire. You don’t want to just freeze the browser while you wait for it to load, instead you want it to load in the background, then kick off some action when its complete (“completed loading” is another example of an event).

(Remember, we’re talking about these properties as being inherent to the Javascript language itself, not just the functions/extensions that help it interact with the web browser while its doing these things).

Turns out that the same properties that make javascript work well for interacting with a web page are also very similar to what’s needed for building a good web server.

• Event-based: a server will constantly be getting requests (at different urls, on different ports) that need to be responded to in different ways when those events take place

• Asynchronous: in dealing with a request, a server will need to load other data (from a file, a database, another server like Twitter), and you don’t want your entire server to just lock up while it waits for that external service (we call this I/O for “Input/Output”) to finish. For a typical web request, the majority of time will be spent waiting on I/O.

So how are other languages that aren’t (for the most part) event-based/asynchronous used on web servers? Like Ruby, Python and PHP. The synchronous code runs on top of an asynchronous web server (Apache, Rack, WSGI) that creates a synchronous “thread” to run your Ruby/Python/PHP code. If another request comes in while that first thread is still processing, the web server creates a new thread. Unfortunately, those threads are resource-intensive which means you can only create a limited number of them (based on how powerful your server is). If all your threads are in-use, a new request will have to wait until a previous request is finished and the thread becomes available again.

So what happens when you build the entire web server in Javascript: You get Node.js! Instead of adding web-browser functions/extensions to the Core Javascript language, Node.js adds server functions/extensions e.g. managing http requests, filesystem operations, talking to databases, etc. While Node.js runs everything on one single “thread”, because Javascript is event-based/asynchronous we can serve hundreds (if not thousands) of simultaneous requests at once because Node.js doesn’t have to freeze/lock for the “I/O” (“In/Out” e.g. database, filesystem, external-service) processes to finish… Node.js can just answer another request (and another) while waiting for first request’s (and subsequent requests’) I/O process to complete, then come back to it and do the next step in responding to the request. Node.js is a master delegator.

So what can you do when you’re able to quickly serve hundreds/thousands of simultaneous connections?

1. Proxy servers / Service “Glue” / Data Plumbing: routing/piping data between different servers, services or processes

2. Telemetry / Analytics: catch and analyze data as events take place in your system

3. Real-time connections to the web-browser: Traditional/Threaded systems try to keep their connections brief (because they can only handle a few at a time). If you don’t have that few-at-a-time constraint, you can leave the connection open much longer and thus easily send data back and forth in real time. Socket.io is a library for doing this.

Example of all 3: visualizing traffic going through a Node.js load-balancer by geolocating the requesting IP address and sending them to a map in the web browser in real-time via Socket.io: http://vimeo.com/48470307

Alternatives to Node.js: EventMachine (Ruby) or Twisted (Python). Unfortunately, the majority of Ruby/Python libraries aren’t written to be evented/asynchronous, which means you can’t use those libraries in an asynchronous environment (because they will lock it up). Whereas the majority of Node.js/Javascript code *is* written to be evented/asynchronous.

So if Node.js is so badass, why not use it for everything?

1. CPU Blocking: because Node.js runs on only a single thread, any local processing you do (e.g. NOT database/service calls) locks the thread. For example, processing heavy numerical/algorithmic processing, or generating complicated HTML templates. Node.js works best when you do that data processing somewhere else (for example, in SQL or map/reduce database), and just sending along raw data (like JSON). This is why you’ll often see Node.js powering a thin API (calling out to a database and serving up some JSON) rather than a full-stack MVC implementation (like Rails/Django). This is why Node.js (backend server API) and single-page web apps like Backbone (frontend client-generated UI) are a powerful combination.

2. Javascript as a language can be a pain in the ass: because JavaScript has spent so much time solely in the browser, it hasn’t gotten the love it deserves. It’s tough to fix things because of backwards compatibility (there are so many different browsers that would need to be updated, and web-compatibility is already hard enough). ECMA Script (the official Javascript “standard”) is evolving. Also, there are many javascript cross-compilers that allow you to write your code in another language, then convert it to javascript; example: CoffeeScript, Clojure, Dart.

Still, the opportunities that Node.js creates are worth it. Other fun stuff/opportunities for Node:

1. Sharing code between Server and Browser: Node.js being in Javascript (like the browser) creates the opportunity to share code between your server and client (keeping things DRY), which makes it easier to create persist server-like functionality on the client (for example, if you’re on a mobile phone and your connection drops, you can still use the web-app until it reconnects). Meteor.js is provides a framework for this (and much more, they entirely muddle the distinction between server and client)

2. Pre-rendering browser content on the server: Typically you don’t want to do heavy CPU processing on the Node server, but maybe you’re working with really lightweight clients and you want to “emulate a web-browser” on your more powerful server. Example: Famo.us pre-renders DOM translations in their tech-demo so it will run on devices like the Apple TV

Follow-up Questions:

So if Node.js + Backbone is a “powerful combination”, why don’t we just ditch Rails entirely?

The Rails ecosystem is more mature than Node’s: there are more engineers, more libraries, stronger conventions and a more complete development and deployment pipeline (scaffolding, testing, continuous integration, monitoring, etc.). If you have a startup with a limited development window and a typical product design (“users create accounts, post content, favorite other user’s content, see what was most favorited content”) that you need to quickly/agilely iterate, Rails has that problem solved (this is the strength of Rails over pretty much everything, not a weakness of Node specifically). If you’re looking at a cost curve, the starting cost for Rails will be way lower for a vanilla product. Now if you’re talking about doing something non-typical (realtime interaction) or are operating at a huge scale (where you can swap infrastructure costs for engineering costs), Node is enabling (there are some things you just can’t/don’t want to do without Node) if not affordable. Also, you can use Node in a heterogenous environment (running the load balancer, or analytics server) or integrate a Node-powered service into a more traditional product (for example, real-time chat is powered by Node, but user accounts, relationships and chat history is managed by Rails).

Super Mayor Emanuel is one of the goofier applications I’ve built at Code for America: supermayor.io

Boring explanation: Using data from Chicago’s Open311 API the app lists, in near-realtime, service requests within the City of Chicago’s 311 system that have recently been created, updated or closed.

Awesome explanation: The mayor runs through the streets of Chicago, leaping over newly-reported civic issues and collecting coins as those civic problems are fixed.

I really like this application, not only because of its visual style, but because it lets you engage with the 311 data in a completely novel way: aurally. Turn up those speakers and let the website run in the background for about 30 minutes. Spinies (new requests) come in waves, and coin blocks (updated/closed requests) are disappointingly rare. Sure, I could have just created a chart of statistics, but I think actually experiencing requests as they come in makes you think differently about both the people who submitted a request and the 311 operators and city staff who are receiving them (just think about what caused those restaurant complaints… or maybe don’t).

The application is built with Node.js, which fetches and caches 311 requests, and a Backbone-based web-app, communicating via socket.io, which manages all of interface and animation. The source is on Github.

The core assumption of engagement applications is that people will do an activity consistently and repeatedly if you just structure the experience and incentivize it correctly — even if it’s asinine.  The justification for civic engagement apps can be similarly foolish: people will perform a potentially beneficial activity that they aren’t currently doing if we give them the ability to do it on the internet (or via SMS, or iPhone, etc.). That’s why I built Civics Garden.

A few months ago a Code for America email thread came around asking how “If you could tell the story of how government works, what would you say?” I pushed back with the idea that one cannot know government without participating in it, and since we are a government of, by, and for the people, the best place to start would be reflecting on one’s own civic life and civic actions.

A tried and true method of reflection is journaling. Hundreds of millions of people keep a journal called Twitter, reflecting and writing on their days experiences, tribulations, meals, and cat sightings (this is not a comprehensive list). If people naturally do this, why not ask them to reflect specifically on their civic actions—voting, volunteering, checking out a library book, picking up a piece of trash, smiling at a stranger—and write down that reflection (however brief) on a regular basis?

Projecting from my own nature, people are fickle, lazy, unreliable creatures. That’s why gamification is the hotness. “How can I get you to look at my ads every day? I’ll make it a game.” Sure, this is no different than historical incentives (“How can I get you to work in my coal mine? I’ll pay you money.”), but now it’s on the internet where advertising is easier to place than coal mines. One form of gamification I really enjoy are virtual pets: like tomagatchi’s you feed with Unit Tests, or flowers you water with foreign language vocabulary. They encourage you to perform an activity because you instinctively (unless you’re a sociopath) feel good when they’re healthy and bad when they’re sickly… despite the fact you know they aren’t actually alive.

Civics Garden combines these concepts: by signing up, users adopt a virtual plant that they keep healthy by regularly “watering” it by writing down their civic deeds. If they go too long without a journal entry, the plant will wither and eventually die. Just like owning real houseplants, it has exhilarating potential.

I built Civics Garden with Node.js and MongoDB using Twitter for authentication. Each new user receives a healthy bamboo plant to caretake by writing short journal entries. To keep people reflecting regularly, one’s plant will wither after two days, and die after four–though users can replant it as many times as necessary. To keep them coming back to the site, users receive a tweet from @civicsgarden to let them know when their plant needs refreshing. As an minimum viable product (MVP), it’s high-on-concept/low-on-looks, but Diana and Emily helped with the graphics.

I’ve tested it internally with Code for America fellows and it should be a rousing success. All of us, as active civic participants performing important civic deeds should be able to briefly but consistently reflect upon and record our actions, right? See for yourself on Civics Garden and test your ability to reflect on a healthy civic commitment.

…or maybe people just don’t like bamboo.

With the recent announcement of 311 Labs and Code for America’s recent podcast featuring me talking about my perspective on Open311, this write-up about Open311Status is probably long overdue.

Open311Status—still a work in progress–polls Open311 servers in 30 cities and aggregates configuration, performance and utilization statistics about those Open311 endpoints. I built Open311Status for two reasons: first, to provide a high-level view of how different endpoints are performing as a developer sanity check (“is my app broken or the server?”); and second, to provide some hard data to the anecdotes and experiences I’ve used in advising the City of Chicago and others in how to improve Open311’s potential for positive impact in their cities and beyond.

In designing Open311Status I took advantage of the huge benefit of the Open311: interoperability. By adopting the Open311 API for exposing civic data, cities are enabling civic developers like myself to build reusable tools and apps. To access data from 30 cities, Open311Status doesn’t need an adapter for each individual city, only a server URL that it can expect to interact and deliver data in the same way as described by the Open311 API documentation. Sure, there are some minor interoperability issues (for example, Toronto doesn’t like datetimes submitted with milliseconds), but these have been minor speed bumps for development, not major deal breakers.

A major limiting factor in the utility of Open311 isn’t these minor technical issues, but in how the Open311 servers are configured. If Open311 is supposed to provide a rich API for developers to interact with there should be a broad number of categories (“types”) of requests that can be submitted, as well as a comprehensive listing of previously submitted requests that can be analyzed or dashboarded. Compare the Boston and Washington, D.C. Open311 implementations: Washington, D.C. currently provides 83 categories of service requests (from “Tree Inspection” to “Roadway Marking Maintenance”) while Boston only provides six. Washington, D.C. provides access to all requests regardless of whether they were entered via Open311, the 311 call center, or city work crews or departments; Boston only displays service requests made via the Open311 API. These configuration decisions have a huge impact on the comprehensiveness of the data and thus the potential applications third party developers can build on top of the Open311.

If server configuration determines the potential for an innovative application, server performance determines whether such an app will be useable. Because a potential Open311 application will be heavily reliant upon a city’s Open311 server, the speed and robustness of that server will determine the responsiveness of the application. At the obvious extreme, if the server goes down, that app will probably be useless. If the server is functioning yet takes several seconds to respond to every request, the application and its users will experience significant slow-downs and frustration. As I’ve witnessed through Open311Status, usually about 10 percent of the servers take longer than one second to respond at any given time. That’s not a stable infrastructure for third party developers to build upon.

Open311Status helps measure the distance between the potential of Open311 and the current reality. It’s cause to celebrate that here are 30 cities (and more to add to Open311Status) who have adopted both open data and open standards; this is hugely important progress! But there is still a lot of work to be done beyond the specification for Open311 to enable a rich and robust third party application developer ecosystem.

Some technical details: Open311Status is built on Node.js with the Express web framework and aggregates data in MongoDB through the Mongoose ORM. The application is single process and uses cron and async modules to poll the cities’ Open311 servers every five minutes. A fun/dumb feature is that individual service requests are streamed to the browser using Socket.io, but because many servers publish service requests an hour (or more!) after they’ve been submitted, Open311Status streams the previous day’s service requests in “real time” as if they were today’s (or rather, “real time minus one day”). Tests are done with Mocha in BDD style with Should using Sinon for mocks and stubs (though coverage could–always–be better). Open311Status is hosted on Heroku.

This post is cross-posted from the Code for America blog.