Whatever you do, don't autoload Rails `lib/`

One of the most common problems I encounter consulting on Rails projects is that developers have previously added lib/ to autoload paths and then twisted themselves into knots creating error-prone, project-specific conventions for subsequently un-autoloading a subset of files also in lib/.

Don’t do it. Don’t add your Rails project’s lib/ to autoload paths.

How does this happen?

A growing Rails application will accumulate a lot of ruby classes and files that don’t cleanly fit into the default app/ directories of controllers, helpers, jobs, or models. Developers should also be creating new directories in app/* to organize like-with-like files (your app/services/ or app/merchants/, etc.). That’s ok!

But frequently there are one-off classes that don’t seem to rise to the level of their own directory in app/. From looking through the cruft of projects like Mastadon or applications I’ve worked on, these files look like:

  • A lone form builder
  • POROs (“Plain old Ruby objects”) like PhoneNumberFormatter, or ZipCodes or Seeder, or Pagination. Objects that serve a single purpose and are largely singletons/identity objects within the application.
  • Boilerplate classes for 3rd party gems, e.g. ApplicationResponder for the responders gem.

That these files accumulate in a project is a fact of life. When choosing where to put them, that’s when things can go wrong.

In a newly built Rails project lib/ looks like the natural place for these. But lib/ has a downside: lib/ is not autoloaded. This can come as a surprise, even to experienced developers, because they have been accustomed to the convenience of autoloaded files in app/. It’s not difficult to add an explicit require statement into application.rb or in an initializer, but that may not be one’s first thought.

That’s when people jump to googling “how to autoload lib/”. Don’t do it! lib/ should not be autoloaded.

The problem with autoloading lib/ is that there will subsequently be files added to lib/ that should not be autoloaded; because they should only be provisionally loaded in a certain environment or context, or deferred, for behavioral, performance, or memory reasons. If your project has already enabled autoloading on lib/, it’s now likely you’ll then add additional configuration to un-autoload the new files. These overrides and counter-overrides accumulate over time and become difficult to understand and unwind, and they cause breakage because someone’s intuition of what will or won’t be loaded in a certain environment or context is wrong.

What should you do instead?

An omakase solution

DHH writes:

lib/ is intended to be for non-app specific library code that just happens to live in the app for now (usually pending extraction into open source or whatever). Everything app specific that’s part of the domain model should live in app/models (that directory is for POROs as much as ARs)… Stuff like a generic PhoneNumberFormatter is exactly what lib/ is intended for. And if it’s app specific, for some reason, then app/models is fine.

The omakase solution is to manually require files from lib/ or use app/models generically to mean “Domain Models” rather than solely Active Record models. That’s great! Do that.

A best practice

Xavier Noria, Zeitwerk’s creator writes:

The best practice to accomplish that nowadays is to move that code to app/lib. Only the Ruby code you want to reload, tasks or other auxiliary files are OK in lib.

Sidekiq’s Problems and Troubleshooting explains:

lib/ directory will only cause pain. Move the code to app/lib/ and make sure the code inside follows the class/filename conventions.

The best practice is to create an app/lib/ directory to home these files. Mastodon does it, as do many others.

This “best practice” is not without contention, as usually anything in Rails that deviates from omakase does, like RSpec instead of MiniTest or FactoryBot instead of Fixtures. But creating app/lib as a convention for Rails apps works for me and many others.

Really, don’t autoload lib/

Whatever path you take, don’t take the path of autoloading lib/.

How GoodJob's mountable Rails Engine delivers Javascript importmaps and frontend assets

GoodJob is a multithreaded, Postgres-based ActiveJob backend for Ruby on Rails.

GoodJob includes a full-featured (though optional) web dashboard to monitor and administer background jobs. The web dashboard is included in the good_job gem as a mountable Rails Engine.

As the maintainer of GoodJob, I want to make gem development easier for myself by innovating as little as possible. That’s why GoodJob builds on top of Active Record and Concurrent::Ruby.

But, the frontend can be a beast. When thinking about how to build a full-featured web dashboard packaged within a Rails Engine within a gem, I had three goals:

  1. Be asset pipeline agnostic with zero dependencies. As of Rails ~7.0, a Rails developer can choose between several different asset pipeline tools (Sprockets, Webpacker/Shakapacker, esbuild/jsbundling, etc.). That’s too many! I want to ensure what I package with GoodJob is compatible with all of them. I also don’t want to affect the parent application at all; everything must be independent and self-contained.
  2. Allow easy patching/debugging. I want the GoodJob web dashboard to work when using the Git repo directly in a project’s Gemfile or simply bundle open good_job to debug a problem.
  3. Write contemporary frontend code. I want to use Bootstrap UI, Stimulus, Rails UJS, and write modular JavaScript with imports. Maybe even Turbo!

And of course, I want GoodJob to be secure, performant, and a joy to develop and use for myself and the developer community.

Read on for how I achieved it all (mostly!) with GoodJob.

What I didn’t do

Here’s all the things I considered, but decided not to do:

  • Nope: Manually construct/inline a small number of javascript files: I did not want to manually build a javascript file, copy-pasting various various 3rd-party libraries into a single file, and then writing my code at the bottom. This seemed laborious and prone to error, especially when I would need to update a library; and my IDE doesn’t work well with large files so writing my own code would be difficult.
  • Nope: Precompile javascript in the repository or on gem build: I did not want to force a pre-commit step to build javascript, or to only package built javascript into the gem because that would make patching and debugging difficult. Over my career I’ve struggled contributing to a number of otherwise fantastic gems that use this workflow pattern.
  • Nope: Compile javascript in the application: Rails has too many different asset pipeline patterns right now for me to consider supporting them all. I consider this more a result of a highly fragmented frontend ecosystem than an indictment of Rails. I can’t imagine supporting all of the different options and whatever else shows up in the future at the same time. (I’m in awe of the gems that do; nice work rails_admin!)

What I did do

As I wrote earlier: “innovate as little as possible”:

Serve vanilla JS and CSS out of vanilla Routes/Controller

GoodJob has explicit routes for frontend assets that wire up to a controller that serves those assets statically with render :file. Let’s break that down…

In my Rails Engine’s router, I define a namescape, frontend, and two get routes. The first route, modules , is for Javascript modules that will go into the importmap. The second route, static , is for Javascript and CSS that I’ll link/script directly in the HTML head.

# config/routes.rb
scope :frontend, controller: :frontends do
  get "modules/:name", action: :module, as: :frontend_module, constraints: { format: 'js' }
  get "static/:name", action: :static, as: :frontend_static, constraints: { format: %w[css js] }

In the matching controller, I create static constants that define hashes of files that are matched and served, which I store in a app/frontend directory in my Rails Engine. I want paths to be explicit for security reasons because passing any sort of dynamic file path through the URL could be a path traversal vulnerability. All of the frontend assets are stored in app/frontend and served out of this controller:

# app/controllers/good_job/frontends_controller.rb

module GoodJob
  class FrontendsController < ActionController::Base # rubocop:disable Rails/ApplicationController
      css: {
        bootstrap: GoodJob::Engine.root.join("app", "frontend", "good_job", "vendor", "bootstrap", "bootstrap.min.css"),
        style: GoodJob::Engine.root.join("app", "frontend", "good_job", "style.css"),
      js: {
        bootstrap: GoodJob::Engine.root.join("app", "frontend", "good_job", "vendor", "bootstrap", "bootstrap.bundle.min.js"),
        chartjs: GoodJob::Engine.root.join("app", "frontend", "good_job", "vendor", "chartjs", "chart.min.js"),
        es_module_shims: GoodJob::Engine.root.join("app", "frontend", "good_job", "vendor", "es_module_shims.js"),
        rails_ujs: GoodJob::Engine.root.join("app", "frontend", "good_job", "vendor", "rails_ujs.js"),

		# Additional JS modules that don't live in app/frontend/good_job/modules
      application: GoodJob::Engine.root.join("app", "frontend", "good_job", "application.js"),
      stimulus: GoodJob::Engine.root.join("app", "frontend", "good_job", "vendor", "stimulus.js"),

    def self.js_modules
      @_js_modules ||= GoodJob::Engine.root.join("app", "frontend", "good_job", "modules").children.select(&:file?).each_with_object({}) do |file, modules|
        key = File.basename(file.basename.to_s, ".js").to_sym
        modules[key] = file

    # Necessarly to serve Javascript to the browser
		skip_after_action :verify_same_origin_request, raise: false
    before_action { expires_in 1.year, public: true }

    def static
      render file: STATIC_ASSETS.dig(params[:format].to_sym, params[:name].to_sym) || raise(ActionController::RoutingError, 'Not Found')

    def module
      raise(ActionController::RoutingError, 'Not Found') if params[:format] != "js"

      render file: self.class.js_modules[params[:name].to_sym] || raise(ActionController::RoutingError, 'Not Found')

One downside of this is that I’m unable to use Sass or Typescript or anything that isn’t vanilla CSS or Javascript. So far I haven’t missed that too much; Bootstrap brings a very comprehensive design system and Rubymine is pretty good at hinting Javscript on its own.

Another downside is that I package several hundred kilobytes of frontend code within my gem. This increases the gem size, which is a real bummer if an application isn’t mounting the dashboard. I’ve considered separating the optional dashboard into a separate gem, but I’m deferring that until anyone notices that it’s problematic (so far so good!).

Manually link assets and construct a JS importmaps in my Engine’s layout <head>

Having created the routes and controller actions, I can simply link the static files in the layout html header:

<!-- app/views/layouts/good_job/application.html.erg -->
  <!-- ... -->
  <%# Note: Do not use asset tag helpers to avoid paths being overriden by config.asset_host %>
  <%= tag.link rel: "stylesheet", href: frontend_static_path(:bootstrap, format: :css, v: GoodJob::VERSION, locale: nil), nonce: content_security_policy_nonce %>
  <%= tag.link rel: "stylesheet", href: frontend_static_path(:style, format: :css, v: GoodJob::VERSION, locale: nil), nonce: content_security_policy_nonce %>
  <%= tag.script "", src: frontend_static_path(:bootstrap, format: :js, v: GoodJob::VERSION, locale: nil), nonce: content_security_policy_nonce %>
  <%= tag.script "", src: frontend_static_path(:chartjs, format: :js, v: GoodJob::VERSION, locale: nil), nonce: content_security_policy_nonce %>
  <%= tag.script "", src: frontend_static_path(:rails_ujs, format: :js, v: GoodJob::VERSION, locale: nil), nonce: content_security_policy_nonce %>

Beneath this, I manually construct the JSON the browser expects for importmaps:

<!-- Link es_module_shims -->
<%= tag.script "", src: frontend_static_path(:es_module_shims, format: :js, v: GoodJob::VERSION, locale: nil), async: true, nonce: content_security_policy_nonce %>

<!-- Construct the importmaps JSON object -->
<% importmaps = GoodJob::FrontendsController.js_modules.keys.index_with { |module_name| frontend_module_path(module_name, format: :js, locale: nil, v: GoodJob::VERSION) } %>
<%= tag.script({ imports: importmaps }.to_json.html_safe, type: "importmap", nonce: content_security_policy_nonce) %>

<!-- Import the entrypoint: application.js -->
<%= tag.script "", type: "module", nonce: content_security_policy_nonce do %> import "application"; <% end %>

That’s it!

I’ll admit, serving frontend assets using render file: is boring, but I experienced a thrill the first time I wired up the importmaps and it just worked. Writing small Javascript modules and using import directives is really nice. I recently added Stimulus and I’m feeling optimistic that I could reliably implement Turbo in my gem’s Rails Engine fully decoupled from the parent application.

I hope this post about GoodJob inspires you to build full-featured web frontends for your own gems and libraries.

Recently, March 12, 2023

  • Work has been complicated, recently. Layoffs, as a general idea, were announced a month ago; it was the same week I came down with a bad cold. I’ve been fairly low energy since and have had trouble differentiating the two. I’m supremely proud and confident that my team is doing the most important work possible. We’ll see!
  • The week prior to all of this, my dad came to visit and stay with us. Having an easier time hosting family was one of our goals in getting a 2nd bedroom. Success.
  • Wow, it’s nearly been a year since I left my last job. I’ve had a number of former colleagues asking for help in leaving, in addition to talking with folks being pushed out: I was surprised to see Code for America finally kill Brigades, and really twist the knife too by forcing groups to rename themselves.
  • GoodJob is great! I’ve been thinking about replacing Advisory Locks with a lock strategy that’s more compatible with PgBouncer. But that will probably be a 2-year project at least of incrementally crabwalking towards that goal while avoiding breaking changes. Rubygems.org just adopted GoodJob; I am humbled.
  • On other projects, I’ve been trying to lower costs. My S3 data-transfer bill went from $10 to $50 a month, which I’m not happy about; scrapers are the worst 🤷‍♀️ I’ve also been experimenting with Dokku for packing some smaller projects (paying $15 once rather than $12 per app), though the VM locked up once on a Saturday night and I had to reboot it and this is exactly why I don’t want to manage my own servers.
  • My brother and I have been planning a Celebration of Life for my mom.
  • I’m so happy to finally be back on Daylight Savings Time.

Service Object Objects in Ruby

For anyone that follows me on social media, I’ll sometimes get into a Coplien-esque funk of “I don’t wanna write Classes. I want to write Objects!”. I don’t want to negotiate an industrial-relations policy for instances of Person in the current scope. I want to imagine the joy and misery Alice and Bob will experience working together right now.

I was thinking of that recently when commenting on Reddit about Caleb Hearth’s “The Decree Design Pattern”. Which ended up in the superset of these two thoughts:

  • heck yeah! if it’s globally distinct it should be globally referenceable
  • oh, oh no, I don’t like looking at that particular Ruby code

This was my comment to try to personally come to terms with those thoughts, iteratively:

# a consistent callable
my_decree = -> { do_something }

# ok, but globally scoped
MY_DECREE = -> { do_something }

# ok, but without the shouty all-caps
module MyDecree
  def self.call 

# ok, but what about when it gets really complex
class MyDecree 
  def self.call(variable)
  def new(variable)
    @variable = variable

  def call

  def do_even_more
    # something really complicated....

From the outside, object perspective, these are all have the same interchangeable interface (.call), and except for the first one, accessible everywhere. That’s great, from my perspective! Though I guess it’s a pretty short blog post to say:

  • Decrees are globally discrete and call-able objects
  • The implementation is up to you!

Unfortunately, the moment the internals come into play, it gets messy. But I don’t think that should take away from the external perspective.

Slop and call

In my role as Engineering Manager, I frequently play Keeper of the Process. Having worked effectively alongside plenty of agile #noplanning people (RIP Andrew), and carrying the scars of dysfunctional processes (oh, PRDs and OGSM), it feels historically out of character to lean into OKR scores and target dates. And I think I’ve made my peace with it.

When I was in high school, my friend’s dad Gary (RIP Gary) retired and bought a championship pool table. The pool table went in their living room and everything else came out. Nothing else fit. The room was a pool table and a stero, which Gary kept tuned to classic jazz. We played a lot of pool and listened to a lot of Charles Mingus.

The two games I remember playing most was 2-ball “English” and 9-ball. English is a “called” game; you have to say which ball and hole you’re aiming for before making the shot. 9-ball is played “slop”, as long as you hit the lowest-numbered ball first, it doesn’t matter which ball goes into which hole.

Both games have their techniques. Playing English I got really good at fine ball handling and putting a sidespin on the ball (that’s the “English”) and having a narrow intent. With 9-ball, I learned to do a lot of what we call a “textbook”-shot (I dunno why we gave only this one shot that name; we were 17). The shot was to bounce the ball off of as many alternating rails as possible until the ball eventually walked itself into a pocket. Just slam it really.

The point is, both of them were ok ways to play. They were just different. It’s fine.

Prevent CDN poisoning from Fat GET/HEAD Requests in Ruby on Rails

There are many different flavors of web cache poisoning discovered by Security Researcher James Kettle. Read on for an explanation of one I’ve run across…

What is a Fat GET/HEAD Request? A GET or HEAD request is “fat” when it has a request body. It’s unexpected! Typically one sees a request body with a POST or PUT request because the body contains form data. The HTTP specification says that including a request body with GET or HEAD requests is undefined. You can do it, and it’s up to the application to figure out what that means. Sometimes it’s bad!

You can get a sense of the applications that intentionally support Fat Requests (and how grumpy it makes some people) by reading through this Postman issue.

Fat Requests can lead to CDN and cache poisoning in Rails. CDNs and caching web proxies (like Varnish) are frequently configured to cache the response from a GET or HEAD request based solely on the request’s URL and not the contents of the request body (they don’t cache POSTs or PUTs at all). If an application isn’t deliberately handling the request body, it may cause unexpected content to be cached and served.

For example, you have a /search endpoint:

  • GET /search shows a landing page with some explanatory content
  • GET /search?q=foo shows the search results for “foo”.
  • Here’s what a Fat Request looks like:

    GET /search     <== the url for the landing page
    q=verybadstuff  <== oh, but with a request body

In a Rails Controller, parameters (alias params) merges query parameters (that’s the URL values) with request parameters (that’s the body values) into a single data structure. If your controller uses the presence of params[:q] to determine whether to show the landing page or the search results, it’s possible that when someone sends that Fat Request, your CDN may cache and subsequently serve the results for verybadstuff every time someone visits the /search landing page. That’s bad!

Here’s how to Curl it:

curl -XGET -H "Content-Type: application/x-www-form-urlencoded" -d "q=verybadstuff" http://localhost:3000/search

Here are 3 ways to fix it…

Solution #1: Fix at the CDN

The most straightforward place to fix this should be at the caching layer, but it’s not always easy.

With Cloudflare, you could rewrite the GET request’s Content-Type header if it is application/x-www-form-urlencoded or multipart/form-data. Or use a Cloudflare Worker to drop the request body.

Varnish makes it easy to drop the request body for any GET request.

Other CDNs or proxies may be easier or more difficult. It depends!

Update via Mr0grog: AWS Cloudfront returns a 403 by default.

Solution #2: Deliberately use query_parameters

Rails provides three different methods for accessing parameters:

  • query_parameters for the values in the request URL
  • request_parameters ) for the values in the request body
  • parameters (alias params) for the problematic combination of them both. Values in query_parameters take precedence over values in request_parameters when they are merged together.

Developers could be diligent and make sure to only use query_parameters in #index or #show , or get routed actions. Here’s an example from the git-scm project.

Solution #3: Patch Rails

Changes were proposed in Rails to not have parameters merge in the body values for GET and HEAD requests; it was rejected because it’s more a problem with the upstream cache than it is with Rails.

You can patch your own version of Rails. Here’s an example that patches the method in ActionDispatch::Request:

# config/initializers/sanitize_fat_requests.rb
module SanitizeFatRequests
  def parameters
    params = get_header("action_dispatch.request.parameters")
    return params if params

    if get? || head?
      params = query_parameters.dup
      set_header("action_dispatch.request.parameters", params)
  alias :params :parameters


# Some RSpec tests to verify this
require 'rails_helper'

RSpec.describe SanitizeFatRequests, type: :request do
  it 'does not merge body params in GET requests' do
    get "/search", headers: {'CONTENT_TYPE' => 'application/x-www-form-urlencoded'}, env: {'rack.input': StringIO.new('q=verybadstuff') }

    # verify that the request is correctly shaped because
    # the test helpers don't expect this kind of request
    expect(request.request_parameters).to eq("q" => "verybadstuff")
    expect(request.parameters).to eq({"action"=>"panlexicon", "controller"=>"search"})

    # the behavioral expectation
    expect(response.body).not_to include "verybadstuff"

Introducing GoodJob Bulk and Batch

GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails. I recently released two new features:

  • GoodJob::Bulk to optimize enqueuing large numbers of jobs (released in GoodJob v3.9)
  • GoodJob::Batch to coordinate parallelized sets of jobs (released in GoodJob v3.10)

Big thanks to @julik, @mollerhoj, @v2kovac, @danielwestendorf, @jrochkind, @mperham and others for your help and counsel!

Bulk enqueue

GoodJob’s Bulk-enqueue functionality can buffer and enqueue multiple jobs at once, using a single INSERT statement. This can be more performant when enqueuing a large number of jobs.

I was inspired by a discussion within a Rails pull request to implement perform_all_later within Active Job. I wanted to both support the way most people enqueue Active Job jobs with perform_later and also encourage people to work directly with Active Job instances too.

# perform_later within a block
active_jobs = GoodJob::Bulk.enqueue do
# or with Active Job instances
active_jobs = [MyJob.new, AnotherJob.new]

Releasing Bulk functionality was a two-step: I initially implemented it while working on Batch functionality, and then with @julik’s initiative and help, we extracted and polished it to be used on its own.


GoodJob’s Batch functionality coordinates parallelized sets of jobs. The ability to coordinate a set of jobs, and run callbacks during lifecycle events, has been a highly demanded feature. Most people who talked to me about job batches were familiar with Sidekiq Pro ‘s batch functionality, which I didn’t want to simply recreate (Sidekiq Pro is excellent!). So I’ve been collecting use cases and thinking about what’s most in the spirit of Rails, Active Job, and Postgres:

  • Batches are mutable, database-backed objects with foreign-key relationships to sets of job records.
  • Batches have properties which use Active Job’s serializer, so they can contain and rehydrate any GlobalID object, like Active Record models.
  • Batches have callbacks, which are themselves Active Job jobs

Here’s a simple example:

GoodJob::Batch.enqueue(on_finish: MyBatchCallbackJob, user: current_user) do

# When these jobs have finished, it will enqueue your `MyBatchCallbackJob.perform_later(batch, options)`
class MyBatchCallbackJob < ApplicationJob
  # Callback jobs must accept a `batch` and `params` argument
  def perform(batch, params)
    # The batch object will contain the Batch's properties, which are mutable
    batch.properties[:user] # => <User id: 1, ...>
    # Params is a hash containing additional context (more may be added in the future)
    params[:event] # => :finish, :success, :discard

There’s more depth and examples in the GoodJob Batch documentation.

Please help!

Batches are definitely a work in progress, and I’d love your feedback:

  • What is the Batch functionality missing? Tell me your use cases.
  • Help improve the Web Dashboard UI (it’s rough but functional!)
  • Find bugs! I’m sure there are some edge cases I overlooked.

Framing open source contributions at work

Excerpts from the excellent RailsConf 2022 keynote: The Success of Ruby on Rails by Eileen Uchitelle [reformatted from the transcript]:

Upgrading is one of the easiest ways to find an area of Rails that can benefit from your contributions. Fixing an issue in a recent release has a high likelihood of being merged.

Running off Rails Main is another way to find contributions to Rails. If you don’t want to run your Main in production, you could run it in a separate CI build. Shopify, GitHub and Basecamp run it.

Running off Main may be harder than running off a release because features and bug fixes are a little in flux sometimes. If you are running off of Main, a feature added to the Main branch could be removed without deprecation. This is a worthwhile risk to take on because it lowers the overall risk of an upgrade. When you run off Main, you’re less likely to fall behind upgrading because it becomes part of your weekly or monthly maintenance. Upgrading becomes routine, second nature rather than novel and scary. Changes are easy to isolate. It’s just slightly less polished. Like I said, I still think it’s pretty stable.

Another way to find places to contribute is look at your own applications.

  • Do you have monkey patches on Rails code that are fixes bugs or changing behavior? Instead of leaving them there, upstream the fix and delete the monkey patch.
  • Is there infrastructure level code that doesn’t really pertain to your product? It’s possible this could be a great addition to Rails. When I wrote the database in Rails, it came from GitHub’s monolith. It made perfect sense because it was getting in the way of upgrades, didn’t expose any intellectual property, had nothing to do with your product features and
    something many applications could benefit from.
  • Lastly and most importantly, keep showing up.

… Ultimately, if more companies treated the framework as an extension of the application, it would result in higher resilience and stability. Investment in Rails ensures your foundation will not crumble under the weight of your application. Treating it as an unimportant part of your application is a mistake and many, many leaders make this mistake.

…leaders see funding open source risky is because they don’t actually understand the work. … Often, leaders worry if there’s a team working in open source, other teams are going to be jealous or resentful that that team is doing “fun” work. …

Maintainers need to make changes, deal with security incidents and also handle criticism from many people. Maintaining and contributing to open source requires a different skill set than product work. That doesn’t make it any less essential.

…Many product companies don’t like words like “research” and “experimental.” They can imply work without goals. Use words like “investment.” And demonstrate the direct value will bring. Make sure it is measurable and will make the application and product more successful. A great example of measurable work is a change that improves performance. If you can tie contributions to direct customer improvements, it’s easier to show leadership.

…As I started contributing more and more and pealing back the layers of Rails, the impact is limitless. I started looking at how applications stretched the boundaries of what Rails was meant to do.

…Ultimately, I want you to contribute to Rails because it’s going to enable you to build a better company and product. The benefits of investing in Rails go far beyond improving the framework.

Investing in Rails will build up the skills of your engineering team. They will developer better communication skills, learn to navigate criticism, debugging skills and how the framework functions. It will teach engineers about the inner-workings and catch bugs.

Monkey patching is far more dangerous than I think most realize. They break with upgrades and cause security incidents. When you write a monkey patch, you maintain a portion of Rails code. Wouldn’t it have been better to patch it upstream rather than taking on that risk and responsibility.

It will give your engineering team the skills they need to make better technical decisions. You’re ensuring that Rails benefits your application and the company for the long-term.

…Contributing to Rails is only not essential if you don’t care about the direction the framework is headed in. We should be contributing because we care about the changes.

We want to ensure our apps are upgradeable, performant and stable.

Investment in Rails also means you won’t have to rewrite your application in a few years because Rails no longer supports what you need. When you fail to invest in your tools, you end up being unable to upgrade. Your engineering team is miserable. The codebase is a mess and writing features is impossible. You’re forced into a rewrite, your engineers want to write Rails and you can no longer let them do that. You have to build a bunch of features before you site falls over.

It’s not Rails’ fault you made the decision to invest elsewhere.

If you build contributing into your culture, the benefits are clear:

  • Your engineering teams’ skills will improve.
  • Rails will evolve with your application because you’re helping decide how it needs to change.
  • Your application will be more resilient because there’s low tech debt and your foundation is stable. Active investment prevents your application from degrading.

Building a team to invest in Rails is proactive. Rewriting an application is reactive. Which one do you think is better for business in the long run?

How GoodJob's Cron does distributed locks

GoodJob is a multithreaded, Postgres-based, Active Job backend for Ruby on Rails. GoodJob has many features that take it beyond Active Job. One such feature is cron-like functionality that allows scheduling repeated jobs on a fixed schedule.

This post is a brief technical story of how GoodJob prevents duplicated cron jobs from running in a multi-process, distributed environment.

This is all true as of GoodJob’s current version, 3.7.4.

Briefly, how GoodJob’s cron works

GoodJob heavily leans on Concurrent::Ruby high-level primitives, and the cron implementation is no different. GoodJob::CronManager accepts a fixed hash of schedule configuration and feeds them into Concurrent::ScheduledTasks, which then trigger perform_later on the job classes at the prescribed times.

A locking strategy is necessary. GoodJob can be running across multiple processes, across numerous isolated servers or containers, in one application. GoodJob should guarantee that at the scheduled time, only a single scheduled job is enqueued.

Initially, advisory locks

When GoodJob’s cron feature was first introduced in version 1.12, Cron used an existing feature of GoodJob: Concurrency Control. Concurrency Control places limits on how many jobs can be enqueued or performed at the same time.

Concurrency Control works by assigning jobs a “key” which is simply a queryable string. Before enqueuing jobs, GoodJob will count how many job records already exist with that same key and prevent the action if the count exceeds the configured limit. GoodJob uses advisory locks to avoid race conditions during this accounting phase.

There were some downsides to using Concurrency Control for Cron.

  • It was a burden on developers. Concurrency Control extends ActiveJob::Base and required the developer to configure Concurrency Control rules separately from the Cron configuration.
  • It wasn’t very performant. Concurrency Control’s design is pessimistic and works best when collisions are rare or infrequent. But a large, clock-synchronized formation of GoodJob processes will frequently produce collisions and it could take several seconds of advisory locking and unlocking across all the processes to insert a single job.

Then, a unique index

GoodJob v2.5.0 changed the cron locking strategy. Instead of using Concurrency Control’s advisory locks, GoodJob uses a unique compound index to prevent the same cron job from being enqueued/INSERTed into the database multiple times.

  • In addition to the existing cron_key column in job records, the change added a new timestamp column, cron_at to store when the cron job is enqueued.
  • Added a unique index on [cron_key, cron_at] to ensure that only one job is inserted for the given key and time.
  • Handled the expected ActiveRecord::RecordNotUnique when multiple cron processes try to enqueue the same cron job simultaneously and the unique index prevents the INSERT from taking place

Now, when a thundering herd of GoodJob processes tries to enqueue the same cron job at the same time, the database uses its unique index constraint to prevent multiple job records from being created. Great!

Does it do a good job?

Yes! I’ve received lots of positive feedback in the year+ since GoodJob’s cron moved to a unique index locking strategy. From the application perspective, there’s much less enqueueing latency using a unique index than when using advisory locks. And from the developer’s perspective, it does just work without additional configuration beyond the schedule.

The main benefit is that the strategy is optimistic. GoodJob just goes for it and lets the database’s unique indexing and built-in locks sort it out. That allows removing the application-level locking and querying and branching logic in the GoodJob client; only handling the potential ActiveRecord::RecordNotUnique exception is necessary.

Using a unique index does require preserving the job records for a bit after the jobs have been performed. Otherwise, poor clock synchronization across processes could lead to a duplicate job being inserted again if the job has already been performed and removed from the table/index. Fortunately, preserving job records should not be too burdensome because GoodJob will automatically clean them up too.

Lastly, one goal of writing this is the hope/fear that a Database Administrator will tell me this is a terrible strategy and provide a better one. Until that happens, I have confidence GoodJob’s cron is good. I’d love your feedback!

Edit: In an earlier version of this post, I mixed up “optimistic” and “pessimistic” locking; that has been corrected.

Conflict, at work

  • I had a conversation with a coworker that reminded me (again!) of this Foreign Affairs essay. It was about receiving feedback that seemed to focus on minimizing interpersonal conflict over things like achieving goals and strategy and impact; it felt familiar.
  • I’ve had coworkers and executives in other jobs that were like “people need to be more comfortable with conflict”. I’ve also been screamed at in a postmortem at a different job. So like, my feelings are mixed. Like are we talking about the absence of psychological safety, or too much of it? I dunno, I could never draw out specifics. I’m now thinking it was actually about the previous bullet point.
  • Schulman makes the point that you’re either in a relationship with someone, or you aren’t.
  • I read this Harper’s essay about working at Wired UK by Hari Kunzru with the line:

    The political economist Albert O. Hirschman famously characterized the choice that is faced by people within declining institutions as being between “voice” and “exit.” Either you speak up to change things, or you leave and look for something better.’

  • Which led me to find this essay about the book which pulls this lovely quote from Hirschman:

    The ultimate in unhappiness and paradoxical loyalist behavior occurs when the public evil produced by the organization promises to accelerate or to reach some intolerable level as the organization deteriorates; then…the decision to exit will become ever more difficult the longer one fails to exit. The conviction that one has to stay on to prevent the worst grows all the time.

  • But this is maybe now more about disfunction than exit. But I’ve also been in the position of being in difficult conflict when explaining that a certain set of strategies will lead to difficulties in attracting and retaining talent. And then most of my favorite people were gone. And then so was I.