The last time I communicated with Jen Pahlka was in the early days of the pandemic: May 2020. No longer locked down but still heavily masked, I visited the San Francisco Ferry Building Farmers Market and bought some fava beans. In the years before the pandemic, Jen had brought her own harvest of fava beans into the Code for America offices and shared them with any staff who wanted them. That had been my very first experience with fresh fava beans: shelling and boiling and shelling them a second time. Now, again with fresh fava beans in hand again, I thought of Jen amidst the pandemic turmoil. I sent her a short email hoping she was well. Jen never replied.

Jen’s book, Recoding America is a good book. It is a laundry list of scenes and vignettes of the greatest hits of government technology, albeit in language of elites discussing elite things in a bloodless elite way. Paired with a more hands-on manual like Cyd Harrell’s A Civic Technologists Practice Guide it covers the ground of the past decade.

That decade is an interesting one. Jen and I shared roughly the same tenure at Code for America: 2011 - 2020 for her, 2012 - 2022 for me; both of us with a gap in the middle. Of course, she was the founder and CEO, whereas I was a fellow, and then an engineer, then a manager, then a director. So we saw a lot of the same things, though from different vantage points on a different journey. In addition to my formal duties, which had a finger in every program at Code for America (if not an arm and leg) there were two activities that were initially happenstance but ended up turning into personal programs of mine:

First, I reached out to new hires, especially new people managers, and offered to have a casual 1:1 and welcome them to the organization. We would chitchat and the message I would work to impart was this: the dissonance between Code for America’s competent external brand and its lived internal chaos could chew people up and was ripe for gaslighting. Instead, those new hires should remember they deserve to be here, they should trust their experience and competence, and truly everyone is winging it (some nicer and more self-aware about it than others).

I’d tell them a story about a prior illuminating executive AMA with Jen where she had shared a philosophy: instead of drawing a bullseye on the wall and trying to hit it down the center with your programs and activities, you can simply throw stuff at the wall and draw the bullseye afterwards around what sticks.

Second, my desk overlooked the executive conference room, which was a glass fishbowl with a couch. It was not infrequent for people to leave that room in a mess of hot tears. When they did and I saw, I’d send them a Slack message and gently offer to buy them a coffee at the Blue Bottle around the corner from the office. There was no ulterior motive; working through it with my leadership coach, a protege of financier Carl Icahn: I simply operationalized giving a shit about people.

Compared to writing a book or leading an organization or serving as a government executive for a year, as Intel CEO Andy Grove might observe: my activities were not high leverage. They also fall outside the bullseye drawn around the stories of the book.

We’re all heroes of our own story. The book briefly touches on an employee who describes himself as “the new guy” because he’d only been a claims processor for 17 years, far less than his more senior colleagues, and still not enough to be capable of processing the department’s most complex claims. His offhand comment is a foil for a deep dive into classifications and processing bottlenecks and mythical man months, but not much else about him as a person. Or what I imagine is the multitudes contained in that brief remark. Is it self-protection, a humblebrag, an invitation for further dialogue about those 17 years, or the years ahead? I’ll never know because the focus shifts back to the administrator and her institutional processes.

In Zen and the Art of Motorcycle Maintenance, a professor advises a student who is hopelessly writers-blocked; the student wants to write about the history of the United States, but is gently redirected to first focus on one brick in one building on the main street of their college town. Writers block broken, the student cannot stop writing, successfully building towards their initial wide vision.

The first brick of this book is possibly the few people “who had all once been part of the team that keeps Google up and running, then had come to DC to help get healthcare.gov back on track.” Easily overlooked in a footnote, they now run a private consultancy for government. The book frames their work as a series of high stakes technology and leadership interventions but not their personal stories, motivations, finances (business and personal), deal flow, engagement philosophy and practices, their loves, losses, missed opportunities, sacrifices, shames, human complexities, ironies, paradoxes. There is no crying, no dying, no problematic influencers, no sketchy investors nor strange bedfellows, no grudgefull quitting, no harassment, no union busting. Nor ziplines, chickens, joyful tears, pirate flags, PDF-form frostings from the sexy-cake shop down the street, nor fava beans for that matter.

In reviewing why too much law and policy ultimately ends up in the courts, Recoding America references a criticism by Ezra Klein: “liberals are too often missing or too timid to claim: a vision of what the law is for.” Recoding America similarly fails here to share a compelling vision of just what civic technology is for. It describes the work of pushing the rock, but without a destination in mind. Every over, under, and through of the bear hunt, without describing the bear. A road built by walking… a division of men gathering wood for a ship… you get the idea.

In all, if you care about civic technology and want to know the major story points: read the book. I am hoping it moves the Overton Window on the stories people will tell about civic tech. Please, someone, write our movement’s Sex and Broadcasting, the humanistic tome of my prior career in community media and community technology. (Sociologist Karina Alexis Rider’s “Volunteering the Valley: Designing technology for the common good in the San Francisco Bay Area” suffices in the meantime.)

Lastly and memorably, Jen recounts serving on a task force addressing pandemic unemployment insurance. Writing with a vague yet startling honesty that haunts my own recollections:

The state should not have needed a task force to tell the EDD what it already knew, and it shouldn’t have needed us to secure permission to act on it. These things are never said out loud—neither the permission we had nor [administrator] Paula’s lack of it. But when we were gone, so was that permission. And soon after, for reasons that were not clear to me, a new backlog began to accrue.

This passage brings into focus the qualities that characterize my own experience with civic tech: power, permission, access, the parasocial qualities of professional relationships, and the fleeting closures of our ongoing experiments to live together in liberal democracy. I hope Jen is doing well, and though I didn’t write this explicitly in my last email, I’d love to hear from her.

I found these two essays juxtaposed in my feed reader and they both rang true. The first is Matt Stoller’s “The Long Annoying Tradition of Anti-Patriotism”, with a particular emphasis on a disordered appeal to utopia in society at large (emphasis mine):

Anti-populism, and its cousin of anti-patriotism, is alluring for our elites. Many lack faith in fellow citizens, and think the work of convincing a large complex country isn’t worth it, or may not even be possible. Others can’t imagine politics itself as a useful endeavor because they believe in a utopia. Indeed, those who believe in certain forms of socialism and libertarianism believe that politics itself shouldn’t exist, that one must perfect the soul of human-kind, and then the messy work of making a society will become unnecessary. In this frame, political institutions, like courts, corporations, and government agencies, are unimportant except as aesthetic objects.

Anti-populism and anti-patriotism leads nowhere, because these attitudes are about convincing citizens to give up their power, to give up on the idea that America is a place we can do politics to make a society.

And then closer to work, on engineering leadership in Will Larsen’s “Building personal and organizational prestige”:

In my experience, engineers confronted with a new problem often leap to creating a system to solve that problem rather than addressing it directly. I’ve found this particularly true when engineers approach a problem domain they don’t yet understand well, including building prestige.

For example, when an organization decides to invest into its engineering brand, the initial plan will often focus on project execution. It’ll include a goal for publishing frequency, ensuring content is representationally accurate across different engineering sub-domains, and how to incentivize participants to contribute. If you follow the project plan carefully, you will technically have built an engineering brand, but my experience is that it’ll be both more work and less effective than a less systematic approach.

The book Traction, by Gabriel Weinberg and Justin Mares has an unstated thesis I find compelling:

Comprehensive, grinding, tryhard mediocrity over narrow, stabby, hopeful genius.

Or: boil the sales & marketing ocean with these spreadsheets and punchlists (a marketing lifestyle I associate with Penny Arcade’s former business manager with stuff like this).

There are lots of quotes from like Paul Graham, Peter Thiel, Marc Andreesen. This is what one signs up for in a book like this. Also replace “traction” with “sales and marketing”:

…spend your time constructing your product or service and testing traction channels in parallel…. We strongly believe that many startups give up way too early… You should always have an explicit traction goal you’re working toward.

…The importance of choosing the right traction goal cannot be overstated. Are you going for growth or “profitability, or something in between? If you need to raise money in X months, what traction do you need to show to do so? These are the types of questions that help you determine the right traction goal.

Once that is defined, you can work backward and set clear quantitative and time-based traction subgoals, such as reaching one thousand customers by next quarter or hitting 20 percent monthly growth targets. Clear subgoals provide accountability. By placing traction activities on the same calendar as product development and other company milestones, you ensure that enough of your time will be spent on traction.

It feels a little dumb pulling out quotes, but also I get security from seeing it not be overthought:

• Put half your efforts into getting traction
• Learn what growth numbers potential investors respect
• Set your growth goals: Set quantitative numbers.
• Find your bright spots: if not hitting quantitative numbers, who is qualitively excited and try to learn from them and replan.

And then the spreadsheets. There are pictures of spreadsheets, and descriptions of columns in spreadsheets. It’s great!

1. How much will it cost to acquire customers through this channel?
2. How many customers are available through this channel
3. Are the customers that you are getting through this channel the kind of customers that you want right now?
4. What’s the lifetime value of this customer

“…we encourage you to be as quantitative as possible, even if it is just guesstimating at first.”

And the 19 strategies:

1. Targeting Blogs: building up from tiny outlets to large
2. Publicity: building relationships with journalists, HARO.
3. Unconventional PR: stunts, customer appreciation
4. Search Engine Marketing
5. Social and Display Ads
6. Offline Ads
7. Search Engine Optimization: fat-head (narrow ranking) and long-tail (lots of landing page content, content marketing farming) strategies. Google Adword’s Keyword Planner, Open Site Explorer.
8. Content Marketing. Spend 6 months blogging do stuff that doesn’t scale (contact influential people, do guest posts, write about recent news events, nerd out)
9. Email Marketing: with your own mailing list or advertise on other mailing lists. Transactional funnel reminders, retention, upselling/expansion, referral emails too.
10. Viral Marketing: map out the loop.
11. Engineering as Marketing: giveaway tools/services (lol, examples are all SEO companies building free SEO tools for marketers, not necessarily engineering)
12. Business Development: partnerships, joint ventures, licensing, distribution, supply. Boil the ocean of potential partners, spreadsheet and pipeline it. Identify who is in charge of the partner’s metric you’re targeting, and make your cold emails forwardable. Write a memo to yourself afterwords “how the deal was done” (how long it took to get to milestones, key contacts, sticking points, partner’s specific interests and influences)
13. Sales. It’s sales! lol, about time wasters (“have you ever brought other technology into your company?”)
14. Affiliate Programs
15. Existing Platforms
16. Trade Shows. Prep, prep, prep.
17. Offline Events. Conferences.
18. Speaking Engagements. Answer upfront “Why are you important enough to be the one giving the talk? What value can you offer me? …then… what your startup is doing, why you’re doing it, specifically how you got to where you are or where things are going.” Recycle and reuse the same 1 or 2 talks.
19. Community Building. Nurturing connections among your customers.

There’s not even a conclusion! Just an acknowledgement and an appendix with specific suggested goals for each category in case the short chapters weren’t boiled enough. It’s not hard, it just takes work.

I’ve been diving into Concurrent Ruby library a lot recently. I use Concurrent Ruby as the foundation for GoodJob where it has saved me immense time and grief because it has a lot of reliable, complex thread-safe primitives that are well-shaped for GoodJob’s needs. I’m a big fan of Concurrent Ruby.

I wanted to cement some of my learnings and understandings by writing a quick blog post to explain how some parts of Concurrent Ruby work, in the spirit of Noah Gibb’s Rebuilding Rails. In the following, I’ll be sharing runnable Ruby code that is similar to how Concurrent Ruby solves the same kind of problems. That said, Concurrent Ruby is much, much safer—and thus a little more complex—than what I’m writing here so please, if you need this functionality, use Concurrent Ruby directly.

The use case: future scheduled tasks

Imagine you want to run some bits of code, at a point in time in the future. It might look like this example creating several tasks at once with varying delays in seconds:

ScheduledTask.execute(delay = 30) do
  # run some code
end

ScheduledTask.execute(60) do
  # run some code
end

ScheduledTask.execute(15) do
  # run some code
end

In Concurrent Ruby, the object to do this is a Concurrent::ScheduledTask (good name, right?). A ScheduledTask will wait delay seconds and then run the block of code on a background thread.

Behind the ScheduledTask is the real star: the Concurrent::TimerSet, which executes a collection of tasks, each after a given delay. Let’s break down the components of a TimerSet:

• TimerSet maintains a list of tasks, ordered by their delays, with the soonest first
• TimerSet runs a reactor-like loop in a background thread. This thread will peek at the next occurring task and wait/sleep until it occurs, then pop the task to execute it.
• TimerSet uses a Concurrent::Event (which is like a Mutex and ConditionVariable combined in a convenient package) to interrupt the sleeping reactor when new tasks are created.

I’ll give examples of each of these. But first, you may be asking….

Why is this so hard?

This is a lot of objects working together to accomplish the use case. This is why:

• Ruby threads have a cost, so we can’t simply create a new thread for each and every task, putting it to sleep until an individual task is intended to be triggered. That would be a lot of threads.
• Ruby threads aren’t safe be canceled/killed, so we can’t, for example, create a single thread for the soonest task but then terminate it and create a new thread if new task is created with a sooner time.

The following section will show how these objects are put together. Again, this is not the exact Concurrent Ruby implementation, but it’s the general shape of how Concurrent Ruby solves this use case.

The Event

Concurrent Ruby describes a Concurrent::Event as:

Old school kernel-style event reminiscent of Win32 programming in C++.

I don’t know what that means exactly, but an Event can be in either a set or unset state, and it can wait (with a timeout!) and be awakened via signals across threads.

I earlier described Event as a Mutex and ConditionVariable packaged together. The ConditionVariableis the star here, and the mutex is simply a supporting actor because the ConditionVariable requires it.

A Ruby ConditionVariable has two features that are perfect for multithreaded programming:

• wait, which is blocking and will put a thread to sleep, with an optional timeout
• set, which broadcasts a signal to any waiting threads to wake up.

Jesse Storimer’s excellent and free ebook Working with Ruby Threads has a great section on ConditionVariables and why the mutex is a necessary part of the implementation.

Here’s some code that implements an Event with an example to show how it can wake up a thread:

class Event
  def initialize
    @mutex = Mutex.new
    @condition = ConditionVariable.new
    @set = false
  end

  def wait(timeout)
    @mutex.synchronize do
      @set || @condition.wait(@mutex, timeout)
    end
  end

  def set
    @mutex.synchronize do
      @set = true
      @condition.broadcast
    end
  end

  def reset
    @mutex.synchronize do
      @set = false
    end
  end
end

Here’s a simple example of an Event running in a loop to show how it might be used:

event = Event.new
running = true
thread = Thread.new do
  # A simple loop in a thread
  while running do
    # loop every second unless signaled
    if event.wait(1)
      puts "Event has been set"
      event.reset
    end
  end
  puts "Exiting thread"
end

sleep 1
event.set
#=> Event has been set

sleep 1
event.set
#=> Event has been set

# let the thread exit
running = false
thread.join
#=> Exiting thread

The ScheduledTask

The implementation of the ScheduledTask isn’t too important in this explanation, but I’ll sketch out the necessary pieces, which match up with a Concurrent::ScheduledTask:

# GLOBAL_TIMER_SET = TimerSet.new

class ScheduledTask
  attr_reader :schedule_time

  def self.execute(delay, timer_set: GLOBAL_TIMER_SET, &task)
    scheduled_task = new(delay, &task)
    timer_set.post_task(scheduled_task)
  end

  def initialize(delay, &task)
    @schedule_time = Time.now + delay
    @task = task
  end

  def run
    @task.call
  end

  def <=>(other)
    schedule_time <=> other.schedule_time
  end
end

A couple things to call out here:

• The GLOBAL_TIMER_SET is necessary so that all ScheduledTasks are added to the same TimerSet. In Concurrent Ruby, this is Concurrent.global_timer_set, though a ScheduledTask.execute can be given an explicit timer_set: parameter if an application has multiple TimerSets (for example, GoodJob initializes its own TimerSet for finer lifecycle management).
• The <=> comparison operator, which will be used to keep our list of tasks sorted with the soonest tasks first.

The TimerSet

Now we have the pieces necessary to implement a TimerSet and fulfill our use case. The TimerSet implemented here is very similar to a Concurrent::TimerSet:

class TimerSet
  def initialize
    @queue = []
    @mutex = Mutex.new
    @event = Event.new
    @thread = nil
  end

  def post_task(task)
    @mutex.synchronize do
      @queue << task
      @queue.sort!
      process_tasks if @queue.size == 1
    end
    @event.set
  end

  def shutdown
    @mutex.synchronize { @queue.clear }
    @event.set
    @thread.join if @thread
    true
  end

  private

  def process_tasks
    @thread = Thread.new do
      loop do
        # Peek the first item in the queue
        task = @mutex.synchronize { @event.reset; @queue.first }
        break unless task

        if task.schedule_time <= Time.now
          # Pop the first item in the queue
          task = @mutex.synchronize { @queue.shift }
          task.run
        else
          timeout = [task.schedule_time - Time.now, 60].min
          @event.wait(timeout)
        end
      end
    end
  end
end

There’s a lot going on here, but here are the landmarks:

• In this TimerSet, @queue is an Array that we explicitly call sort! on so that the soonest task is always first in the array. In the Concurrent Ruby implementation, that’s done more elegantly with a Concurrent::Collection::NonConcurrentPriorityQueue. The @mutex is used to make sure that adding/sorting/peeking/popping operations on the queue are synchronized and safe across threads.
• The magic happens in #process_tasks, which creates a new thread and starts up a loop. It loops over the first task in the queue (the soonest):
  • If there is no task, it breaks the loop and exits the thread.
  • If there is a task, it checks whether it’s time to run, and if so, runs it. If it’s not time yet, it uses the Event#wait until it is time to run, or 60 seconds, whichever is sooner. That 60 seconds is a magic number in the real implementation, and I assume that’s to reduce clock drift. Remember, Event#wait is signalable, so if a new task is added, the loop will be immediately restarted and the delay recalculated.
  • In real Concurrent Ruby, task.run is posted to a separate thread pool where it won’t block or slow down the loop.
• The Event#set is called inside of #add_task which inserts new tasks into the queue. The process_tasks background thread is only created the first time a task is added to the queue after the queue has been emptied. This minimizes the number of active threads.
• The Event#reset is called when the queue is first peeked in process_tasks. There’s a lot of subtle race conditions being guarded against in a TimerSet. Calling reset unsets the event at the top of the loop to allow the Event to be set again before the Event#wait

And finally, we can put all of the pieces together to fulfill our use case of scheduled tasks:

GLOBAL_TIMER_SET = TimerSet.new

ScheduledTask.execute(1) { puts "This is the first task" }
ScheduledTask.execute(5) { puts "This is the third task" }
ScheduledTask.execute(3) { puts "This is the second task" }

sleep 6
GLOBAL_TIMER_SET.shutdown

#=> This is the first task
#=> This is the second task
#=> This is the third task

That’s it!

The TimerSet is a really neat object that’s powered by an Event, which is itself powered by a ConditionVariable. There’s a lot of fun thread-based signaling happening here!

While writing my post, I came across a 2014 post from Job Vranish entitled “Ruby Queue Pop with Timeout”, which builds something very similar looking using the same primitives. In the comments, Mike Perham linked to Connection Pool’s TimedStack which also looks similar. Again please use a real library like Concurrent Ruby or Connection Pool. This was just for explanatory purposes 👍

Update: Rails v7.1 will introduce a new configuration method config.autoload_lib to make it safer and easier to autoload the /lib directory and explicitly exclude directories from autoloading. When released, this advice may no longer be relevant, though I imagine it will still be possible for developers to twist themselves into knots and cause outages with autoloading overrides.

One of the most common problems I encounter consulting on Rails projects is that developers have previously added lib/ to autoload paths and then twisted themselves into knots creating error-prone, project-specific conventions for subsequently un-autoloading a subset of files also in lib/.

Don’t do it. Don’t add your Rails project’s lib/ to autoload paths.

How does this happen?

A growing Rails application will accumulate a lot of ruby classes and files that don’t cleanly fit into the default app/ directories of controllers, helpers, jobs, or models. Developers should also be creating new directories in app/* to organize like-with-like files (your app/services/ or app/merchants/, etc.). That’s ok!

But frequently there are one-off classes that don’t seem to rise to the level of their own directory in app/. From looking through the cruft of projects like Mastodon or applications I’ve worked on, these files look like:

• A lone form builder
• POROs (“Plain old Ruby objects”) like PhoneNumberFormatter, or ZipCodes or Seeder, or Pagination. Objects that serve a single purpose and are largely singletons/identity objects within the application.
• Boilerplate classes for 3rd party gems, e.g. ApplicationResponder for the responders gem.

That these files accumulate in a project is a fact of life. When choosing where to put them, that’s when things can go wrong.

In a newly built Rails project lib/ looks like the natural place for these. But lib/ has a downside: lib/ is not autoloaded. This can come as a surprise, even to experienced developers, because they have been accustomed to the convenience of autoloaded files in app/. It’s not difficult to add an explicit require statement into application.rb or in an initializer, but that may not be one’s first thought.

That’s when people jump to googling “how to autoload lib/”. Don’t do it! lib/ should not be autoloaded.

The problem with autoloading lib/ is that there will subsequently be files added to lib/ that should not be autoloaded; because they should only be provisionally loaded in a certain environment or context, or deferred, for behavioral, performance, or memory reasons. If your project has already enabled autoloading on lib/, it’s now likely you’ll then add additional configuration to un-autoload the new files. These overrides and counter-overrides accumulate over time and become difficult to understand and unwind, and they cause breakage because someone’s intuition of what will or won’t be loaded in a certain environment or context is wrong.

What should you do instead?

An omakase solution

DHH writes:

lib/ is intended to be for non-app specific library code that just happens to live in the app for now (usually pending extraction into open source or whatever). Everything app specific that’s part of the domain model should live in app/models (that directory is for POROs as much as ARs)… Stuff like a generic PhoneNumberFormatter is exactly what lib/ is intended for. And if it’s app specific, for some reason, then app/models is fine.

The omakase solution is to manually require files from lib/ or use app/models generically to mean “Domain Models” rather than solely Active Record models. That’s great! Do that.

A best practice

Xavier Noria, Zeitwerk’s creator writes:

The best practice to accomplish that nowadays is to move that code to app/lib. Only the Ruby code you want to reload, tasks or other auxiliary files are OK in lib.

Sidekiq’s Problems and Troubleshooting explains:

A lib/ directory will only cause pain. Move the code to app/lib/ and make sure the code inside follows the class/filename conventions.

The best practice is to create an app/lib/ directory to home these files. Mastodon does it, as do many others.

This “best practice” is not without contention, as usually anything in Rails that deviates from omakase does, like RSpec instead of MiniTest or FactoryBot instead of Fixtures. But creating app/lib as a convention for Rails apps works for me and many others.

Really, don’t autoload lib/

Whatever path you take, don’t take the path of autoloading lib/.

An “Event” has caused rabbits to become anthropomophic. This exchange is the book in a walnut-shell:

So while we ate the excellent walnut cake that the Venerable Bunty’s mother’s sister’s daughter’s husband’s son had baked, Venerable Bunty and Connie told us about life inside the colonies, which despite the lack of freedom and limited space were the only areas within the United Kingdom that ran themselves entirely on rabbit socio-egalitarian principles.

‘It’s occasionally aggressive and often uncompromising,’ said Finkle, ‘but from what I’ve seen of both systems, a country run on rabbit principles would be a step forward – although to be honest, I’m not sure we’d be neurologically suited to the regime. While most humans are wired to be reasonably decent, a few are wired to be utter shits – and they do tend to tip the balance.’

‘The decent humans are generally supportive of doing the right thing,’ said the Venerable Bunty, ‘but never take it much farther than that. You’re trashing the ecosystem for no reason other than a deluded sense of anthropocentric manifest destiny, and until you stop talking around the issue and actually feel some genuine guilt, there’ll be no change.’

‘Shame, for want of a better word, is good,’ said Finkle. ‘Shame is right, shame works. Shame is the gateway emotion to increased self-criticism, which leads to realisation, an apology, outrage and eventually meaningful action. We’re not holding our breaths that any appreciable numbers can be arsed to make the journey along that difficult chain of emotional honesty – many good people get past realisation, only to then get horribly stuck at apology – but we live in hope.’

‘I understand,’ I said, having felt that I too had yet to make the jump to apology.

‘It’s further evidence of satire being the engine of the Event,’ said Connie, ‘although if that’s true, we’re not sure for whose benefit.’

‘Certainly not humans’,’ said Finkle, ‘since satire is meant to highlight faults in a humorous way to achieve betterment, and if anything, the presence of rabbits has actually made humans worse.’

‘Maybe it’s the default position of humans when they feel threatened,’ I ventured, ‘although if I’m honest, I know a lot of people who claim to have “nothing against rabbits” but tacitly do nothing against the overt leporiphobia that surrounds them.’

‘Or maybe it’s just satire for comedy’s sake and nothing else,’ added Connie, ‘or even more useless, satire that provokes a few guffaws but only low to middling outrage – but is coupled with more talk and no action. A sort of  . . . empty cleverness.’

‘Maybe a small puff in the right moral direction is the best that could be hoped for,’ added Finkle thoughtfully. ‘Perhaps that’s what satire does – not change things wholesale but nudge the collective consciousness in a direction that favours justice and equality. Is there any more walnut cake?’

‘I’m afraid I had the last slice,’ I said, ‘but I did ask if anyone else wanted it.’

If there is a particular story we should be telling, a big question we should be asking of human history (instead of the ‘origins of social inequality’), is it precisely this: how did we find ourselves stuck in just one form of social reality, and how did relations based ultimately on violence and domination come to be normalized within it?

…

What happens if we treat the rejection of urban life, or of slavery, [or of certain technologies] in certain times and places as something just as significant as the emergence of those same phenomena in others.

…

What is the purpose of all this new knowledge, if not to reshape our conceptions of who we are and what we might yet become? If not, in other words, to rediscover the meaning of our third basic freedom: the freedom to create new and different forms of social reality?

I imagine I’m already on board with David Graeber’s political project, so while I greatly enjoyed it, I found it too long by about a third.

The overall thrust is that people are much more interesting and creative than we give them credit for, and there’s a lot (too much for me in this book) of historical evidence that this is the case. And that it’s bunk to claim that increasing social complexity and scale requires an authoritarian state or bureacracy. I guess it’s an argument to unstick the “End of History”-framing we’re mired in.

Of various things I learned / was confronted with:

• Indulging children is Native American practice. Makes sense cause it’s a common theme in Kim Stanley Robinson books of which the Haudenosaunee make frequent appearance too.
• Roman-style property ownership (of which we inherit), is pretty fucked up when stared directly at, based on a patriarch’s relations with household slaves.
• It seems like a pretty legit critique of Western society to point out that there are a lot of legitimate ways poeple are free to harm other people during their every day life, and that’s got to be pretty warpy.
• Spending more time imagining and debating the society and politics you want to live in… probably makes for a better society and politics. One of those, if it’s hard do it a lot sorts of things. And if that sounds annoying in the context of the present, that’s probably because we’ve severely narrowed the scope of debate and possibility.

There’s a lot of history and anthropology to boil down:

…the key point to remember is that we are not talking here about ‘freedom’ as an abstract ideal or formal principle (as in ‘Liberty, Equality and Fraternity!’). Over the course of these chapters we have instead talked about basic forms of social liberty which one might actually put into practice:

1. the freedom to move away or relocate from one’s surroundings;
2. the freedom to ignore or disobey commands issued by others; and
3. the freedom to shape entirely new social realities, or shift back and forth between different ones

….three elementary principles of domination:

1. control of violence (or sovereignty),
2. control of knowledge, and
3. charismatic politics

…and a lot of historical and anthropoligical critique:

Environmental determinists have an unfortunate tendency to treat humans as little more than automata, living out some economist’s fantasy of rational calculation. To be fair, they don’t deny that human beings are quirky and imaginative creatures – they just seem to reason that, in the long run, this fact makes very little difference.

…

For much of the twentieth century, anthropologists tended to describe the societies they studied in ahistorical terms, as living in a kind of eternal present. Some of this was an effect of the colonial situation under which much ethnographic research was carried out. The British Empire, for instance, maintained a system of indirect rule in various parts of Africa, India and the Middle East where local institutions like royal courts, earth shrines, associations of clan elders, men’s houses and the like were maintained in place, indeed fixed by legislation. Major political change – forming a political party, say, or leading a prophetic movement – was in turn entirely illegal, and anyone who tried to do such things was likely to be put in prison. This obviously made it easier to describe the people anthropologists studied as having a way of life that was timeless and unchanging.

….

Social science has been largely a study of the ways in which human beings are not free: the way that our actions and understandings might be said to be determined by forces outside our control. Any account which appears to show human beings collectively shaping their own destiny, or even expressing freedom for its own sake, will likely be written off as illusory, awaiting ‘real’ scientific explanation; or if none is forthcoming (why do people dance?), as outside the scope of social theory entirely. This is one reason why most ‘big histories’ place such a strong focus on technology. Dividing up the human past according to the primary material from which tools and weapons were made (Stone Age, Bronze Age, Iron Age) or else describing it as a series of revolutionary breakthroughs (Agricultural Revolution, Urban Revolution, Industrial Revolution), they then assume that the technologies themselves largely determine the shape that human societies will take for centuries to come – or at least until the next abrupt and unexpected breakthrough comes along to change everything again.

…

Choosing to describe history the other way round, as a series of abrupt technological revolutions, each followed by long periods when we were prisoners of our own creations, has consequences. Ultimately it is a way of representing our species as decidedly less thoughtful, less creative, less free than we actually turn out to have been. It means not describing history as a continual series of new ideas and innovations, technical or otherwise, during which different communities made collective decisions about which technologies they saw fit to apply to everyday purposes, and which to keep confined to the domain of experimentation or ritual play. What is true of technological creativity is, of course, even more true of social creativity. One of the most striking patterns we discovered while researching this book – indeed, one of the patterns that felt most like a genuine breakthrough to us – was how, time and again in human history, that zone of ritual play has also acted as a site of social experimentation – even, in some ways, as an encyclopaedia of social possibilities.

GoodJob is a multithreaded, Postgres-based ActiveJob backend for Ruby on Rails.

GoodJob includes a full-featured (though optional) web dashboard to monitor and administer background jobs. The web dashboard is included in the good_job gem as a mountable Rails Engine.

As the maintainer of GoodJob, I want to make gem development easier for myself by innovating as little as possible. That’s why GoodJob builds on top of Active Record and Concurrent::Ruby.

But, the frontend can be a beast. When thinking about how to build a full-featured web dashboard packaged within a Rails Engine within a gem, I had three goals:

1. Be asset pipeline agnostic with zero dependencies. As of Rails ~7.0, a Rails developer can choose between several different asset pipeline tools (Sprockets, Webpacker/Shakapacker, esbuild/jsbundling, etc.). That’s too many! I want to ensure what I package with GoodJob is compatible with all of them. I also don’t want to affect the parent application at all; everything must be independent and self-contained.
2. Allow easy patching/debugging. I want the GoodJob web dashboard to work when using the Git repo directly in a project’s Gemfile or simply bundle open good_job to debug a problem.
3. Write contemporary frontend code. I want to use Bootstrap UI, Stimulus, Rails UJS, and write modular JavaScript with imports. Maybe even Turbo!

And of course, I want GoodJob to be secure, performant, and a joy to develop and use for myself and the developer community.

Read on for how I achieved it all (mostly!) with GoodJob.

What I didn’t do

Here’s all the things I considered, but decided not to do:

• Nope: Manually construct/inline a small number of javascript files: I did not want to manually build a javascript file, copy-pasting various various 3rd-party libraries into a single file, and then writing my code at the bottom. This seemed laborious and prone to error, especially when I would need to update a library; and my IDE doesn’t work well with large files so writing my own code would be difficult.
• Nope: Precompile javascript in the repository or on gem build: I did not want to force a pre-commit step to build javascript, or to only package built javascript into the gem because that would make patching and debugging difficult. Over my career I’ve struggled contributing to a number of otherwise fantastic gems that use this workflow pattern.
• Nope: Compile javascript in the application: Rails has too many different asset pipeline patterns right now for me to consider supporting them all. I consider this more a result of a highly fragmented frontend ecosystem than an indictment of Rails. I can’t imagine supporting all of the different options and whatever else shows up in the future at the same time. (I’m in awe of the gems that do; nice work rails_admin!)

What I did do

As I wrote earlier: “innovate as little as possible”:

Serve vanilla JS and CSS out of vanilla Routes/Controller

GoodJob has explicit routes for frontend assets that wire up to a controller that serves those assets statically with render :file. Let’s break that down…

In my Rails Engine’s router, I define a namescape, frontend, and two get routes. The first route, modules , is for Javascript modules that will go into the importmap. The second route, static , is for Javascript and CSS that I’ll link/script directly in the HTML head.

# config/routes.rb
scope :frontend, controller: :frontends do
  get "modules/:name", action: :module, as: :frontend_module, constraints: { format: 'js' }
  get "static/:name", action: :static, as: :frontend_static, constraints: { format: %w[css js] }
end

In the matching controller, I create static constants that define hashes of files that are matched and served, which I store in a app/frontend directory in my Rails Engine. I want paths to be explicit for security reasons because passing any sort of dynamic file path through the URL could be a path traversal vulnerability. All of the frontend assets are stored in app/frontend and served out of this controller:

# app/controllers/good_job/frontends_controller.rb

module GoodJob
  class FrontendsController < ActionController::Base # rubocop:disable Rails/ApplicationController
    STATIC_ASSETS = {
      css: {
        bootstrap: GoodJob::Engine.root.join("app", "frontend", "good_job", "vendor", "bootstrap", "bootstrap.min.css"),
        style: GoodJob::Engine.root.join("app", "frontend", "good_job", "style.css"),
      },
      js: {
        bootstrap: GoodJob::Engine.root.join("app", "frontend", "good_job", "vendor", "bootstrap", "bootstrap.bundle.min.js"),
        chartjs: GoodJob::Engine.root.join("app", "frontend", "good_job", "vendor", "chartjs", "chart.min.js"),
        es_module_shims: GoodJob::Engine.root.join("app", "frontend", "good_job", "vendor", "es_module_shims.js"),
        rails_ujs: GoodJob::Engine.root.join("app", "frontend", "good_job", "vendor", "rails_ujs.js"),
      },
    }.freeze

		# Additional JS modules that don't live in app/frontend/good_job/modules
    MODULE_OVERRIDES = {
      application: GoodJob::Engine.root.join("app", "frontend", "good_job", "application.js"),
      stimulus: GoodJob::Engine.root.join("app", "frontend", "good_job", "vendor", "stimulus.js"),
    }.freeze

    def self.js_modules
      @_js_modules ||= GoodJob::Engine.root.join("app", "frontend", "good_job", "modules").children.select(&:file?).each_with_object({}) do |file, modules|
        key = File.basename(file.basename.to_s, ".js").to_sym
        modules[key] = file
      end.merge(MODULE_OVERRIDES)
    end

    # Necessarly to serve Javascript to the browser
		skip_after_action :verify_same_origin_request, raise: false
    before_action { expires_in 1.year, public: true }

    def static
      render file: STATIC_ASSETS.dig(params[:format].to_sym, params[:name].to_sym) || raise(ActionController::RoutingError, 'Not Found')
    end

    def module
      raise(ActionController::RoutingError, 'Not Found') if params[:format] != "js"

      render file: self.class.js_modules[params[:name].to_sym] || raise(ActionController::RoutingError, 'Not Found')
    end
  end
end

One downside of this is that I’m unable to use Sass or Typescript or anything that isn’t vanilla CSS or Javascript. So far I haven’t missed that too much; Bootstrap brings a very comprehensive design system and Rubymine is pretty good at hinting Javscript on its own.

Another downside is that I package several hundred kilobytes of frontend code within my gem. This increases the gem size, which is a real bummer if an application isn’t mounting the dashboard. I’ve considered separating the optional dashboard into a separate gem, but I’m deferring that until anyone notices that it’s problematic (so far so good!).

Manually link assets and construct a JS importmaps in my Engine’s layout <head>

Having created the routes and controller actions, I can simply link the static files in the layout html header:

<!-- app/views/layouts/good_job/application.html.erg -->
<head>
  <!-- ... -->
  <%# Note: Do not use asset tag helpers to avoid paths being overriden by config.asset_host %>
  <%= tag.link rel: "stylesheet", href: frontend_static_path(:bootstrap, format: :css, v: GoodJob::VERSION, locale: nil), nonce: content_security_policy_nonce %>
  <%= tag.link rel: "stylesheet", href: frontend_static_path(:style, format: :css, v: GoodJob::VERSION, locale: nil), nonce: content_security_policy_nonce %>
	
  <%= tag.script "", src: frontend_static_path(:bootstrap, format: :js, v: GoodJob::VERSION, locale: nil), nonce: content_security_policy_nonce %>
  <%= tag.script "", src: frontend_static_path(:chartjs, format: :js, v: GoodJob::VERSION, locale: nil), nonce: content_security_policy_nonce %>
  <%= tag.script "", src: frontend_static_path(:rails_ujs, format: :js, v: GoodJob::VERSION, locale: nil), nonce: content_security_policy_nonce %>

Beneath this, I manually construct the JSON the browser expects for importmaps:

<!-- Link es_module_shims -->
<%= tag.script "", src: frontend_static_path(:es_module_shims, format: :js, v: GoodJob::VERSION, locale: nil), async: true, nonce: content_security_policy_nonce %>

<!-- Construct the importmaps JSON object -->
<% importmaps = GoodJob::FrontendsController.js_modules.keys.index_with { |module_name| frontend_module_path(module_name, format: :js, locale: nil, v: GoodJob::VERSION) } %>
<%= tag.script({ imports: importmaps }.to_json.html_safe, type: "importmap", nonce: content_security_policy_nonce) %>

<!-- Import the entrypoint: application.js -->
<%= tag.script "", type: "module", nonce: content_security_policy_nonce do %> import "application"; <% end %>

That’s it!

I’ll admit, serving frontend assets using render file: is boring, but I experienced a thrill the first time I wired up the importmaps and it just worked. Writing small Javascript modules and using import directives is really nice. I recently added Stimulus and I’m feeling optimistic that I could reliably implement Turbo in my gem’s Rails Engine fully decoupled from the parent application.

I hope this post about GoodJob inspires you to build full-featured web frontends for your own gems and libraries.

• Work has been complicated, recently. Layoffs, as a general idea, were announced a month ago; it was the same week I came down with a bad cold. I’ve been fairly low energy since and have had trouble differentiating the two. I’m supremely proud and confident that my team is doing the most important work possible. We’ll see!
• The week prior to all of this, my dad came to visit and stay with us. Having an easier time hosting family was one of our goals in getting a 2nd bedroom. Success.
• Wow, it’s nearly been a year since I left my last job. I’ve had a number of former colleagues asking for help in leaving, in addition to talking with folks being pushed out: I was surprised to see Code for America finally kill Brigades, and really twist the knife too by forcing groups to rename themselves.
• GoodJob is great! I’ve been thinking about replacing Advisory Locks with a lock strategy that’s more compatible with PgBouncer. But that will probably be a 2-year project at least of incrementally crabwalking towards that goal while avoiding breaking changes. Rubygems.org just adopted GoodJob; I am humbled.
• On other projects, I’ve been trying to lower costs. My S3 data-transfer bill went from $10 to $50 a month, which I’m not happy about; scrapers are the worst 🤷‍♀️ I’ve also been experimenting with Dokku for packing some smaller projects (paying $15 once rather than $12 per app), though the VM locked up once on a Saturday night and I had to reboot it and this is exactly why I don’t want to manage my own servers.
• My brother and I have been planning a Celebration of Life for my mom.
• I’m so happy to finally be back on Daylight Savings Time.

For anyone that follows me on social media, I’ll sometimes get into a Coplien-esque funk of “I don’t wanna write Classes. I want to write Objects!”. I don’t want to negotiate an industrial-relations policy for instances of Person in the current scope. I want to imagine the joy and misery Alice and Bob will experience working together right now.

I was thinking of that recently when commenting on Reddit about Caleb Hearth’s “The Decree Design Pattern”. Which ended up in the superset of these two thoughts:

• heck yeah! if it’s globally distinct it should be globally referenceable
• oh, oh no, I don’t like looking at that particular Ruby code

This was my comment to try to personally come to terms with those thoughts, iteratively:

# a consistent callable
my_decree = -> { do_something }

# ok, but globally scoped
MY_DECREE = -> { do_something }

# ok, but without the shouty all-caps
module MyDecree
  def self.call 
    do_something 
  end 
end

# ok, but what about when it gets really complex
class MyDecree 
  def self.call(variable)
    new(variable).call 
  end
  
  def new(variable)
    @variable = variable
  end

  def call
    do_something
    do_something_else(@variable)
    do_even_more
  end

  def do_even_more
    # something really complicated....
  end
end

From the outside, object perspective, these are all have the same interchangeable interface (.call), and except for the first one, accessible everywhere. That’s great, from my perspective! Though I guess it’s a pretty short blog post to say:

• Decrees are globally discrete and call-able objects
• The implementation is up to you!

Unfortunately, the moment the internals come into play, it gets messy. But I don’t think that should take away from the external perspective.