I previously posted the The Three Ways from The Phoenix Project. Here are the Five Ideals from The Unicorn Project, by way of Gene Kim’s IT Revolution blog:

The First Ideal: Locality and Simplicity

We need to design things so that we have locality in our systems and the organizations that build them. We need simplicity in everything we do. This ideal relates to the degree to which a development team can make local code changes in a single location without impacting various teams. The last place we want complexity is internally, whether it’s in our code, in our organization, or in our processes.

The Second Ideal: Focus, Flow, and Joy

The Second Ideal is all about how our daily work feels. Is our work marked by boredom and waiting for other people to get things done on our behalf? Do we blindly work on small pieces of the whole, only seeing the outcomes of our work during deployment when everything blows up, leading to firefighting, punishment, and burnout? Or do we work in small batches, ideally single-piece flow, getting fast and continual feedback on our work? These are the conditions that allow for focus and flow, challenge, learning, discovery, mastering our domain, and even joy. This is what being a developer means.

The Third Ideal: Improvement of Daily Work

The Third Ideal addresses paying down technical debt and improving architecture. When technical debt is treated as a priority and paid down and architecture is continuously improved and modernized, teams can work with flow, delivering better value sooner, safer, and happier. And the business ultimately wins when developers can deliver on enterprise performance goals.

The Fourth Ideal: Psychological Safety

Psychological safety is one of the top predictors of team performance. When team members feel safe to talk about problems, problems can not only be fixed but prevented. Solving problems requires honesty, and honesty requires an absence of fear. In knowledge work, psychological safety should be treated with the same importance as physical safety is in manufacturing.

The Fifth Ideal: Customer Focus

Customer focus relates to the difference between core and context as defined by Geoffrey Moore. Core is what customers are willing and able to pay for, the bread and butter of your business. Context is what they don’t care about, what it took to get them that product, including all the backend systems of an organization like HR and marketing and development. It’s critical to look at these Context systems as essential, as mission critical, and fund them appropriately. Context should never kill core.

And a shorter version that’s less engineer focused from “DevOps: A Primer For The Business Leader”:

• Locality and Simplicity: alleviate dependencies between teams and components.
• Focus, Flow, and Joy: the smooth flow of work that enables focus and joy.
• Improvement of Daily Work: continuously improve and pay down technical debt.
• Psychological Safety: a top predictor of team performance; enables improvement.
• Customer Focus: optimize for customer value, not for a role-based silo.

Tawana “Honeycomb” Petty writing in a “Mama Lila Cabbil Tribute: Pushing Us Toward Deeper Thought, Still”, published in Riverwise Magazine.

Through my organizing and teaching, I have asked that anti-racist circles step away from performative testimonials of privilege for whites and lack of privilege for Black and Brown people. I have asked that all allies move from ally-ship towards co-liberation, with the belief that we can only make systemic change if we understand our liberation is tied up with one another’s. My historical and current analysis of this moment pushes me to interrogate the notion that (most) white people “benefit” from their forced relationship with white supremacy.

I have asked that white allies (aspiring co-liberators) seek to understand the impact that the myth of white superiority and the system of white supremacy has had on their own communities. I have asked them to deal with school shootings in their communities, opioid abuse, domestic violence, and rising incidents of suicide. I have asked them if they truly believe what they say when they testify to their privilege.

For many, this way of thinking may not appear to answer the questions that plague Black and Brown people in America. However, I am of the mindset that a dehumanized being cannot see another as fully human. I am of the mindset that the white children who are shooting up schools have fallen victim to trying to live up to the myth of white superiority.

White men, even in progressive circles, are taught they are superior to white women, Black women, and every other living being on the planet. What would it mean for the anti-racism movement to teach white men that they have the furthest to go in efforts to reach their full humanity? What would it look like if they don’t enter spaces acting inherently superior (privileged), but rather with a lot of work to do to shed the legacy of violence that comes with their perception of superiority?

From Susan Casey’s The wave:

Teahupoo, with its timeless power, brought to mind the age-old philosophical quest to distinguish between beauty and its twisted cousin, the sublime: for the merely pretty to graduate to the sublime, terror was required in the mix. “The Alps fill the mind with a kind of agreeable horror,” wrote one seventeenth-century thinker, summing up the concept. And while humans were capable of creating the lovely, the dramatic, the sad, or the inspiring, only nature could produce the sublime. It was a concept both comforting and disturbing: there are many things out there more powerful than we are. No one was more aware of this than the men who’d ridden Teahupoo on this day (except, perhaps, the ones who had fallen on it).

From Cyd Harrell’s excellent “A Civic Technoligists Practice Guide”, via a Twitter convo, reformatted by me:

Because its goal is change, civic tech embodies an interesting split between demonstrating and operationalizing the potential of modern tech. I like to call these two branches showing what’s possible and doing what’s necessary. Many projects are a mix of the two, but they require different mindsets.

“Showing what’s possible” is about speed, prototyping, design, public feedback, and data. These are often web projects because web tools are great for those purposes.

“Doing what’s necessary,” on the other hand, is about shifting the underlying practices and systems: back-end systems, security, and procurement; hiring and team composition; even shifting budget priorities.”

[…]

But our job as civic technologists isn’t to be the hero of the stories we stumble into halfway through; it’s to understand and support the people who have already been in place doing the work, and who want to use tech to make improvements.

They line up with Code for America’s pillars of “Show what’s possible”, “Help people do it themselves” and “Build a movement” (though the latter is rather more grandiose than understand and support).

As we called these in my previous career, Direct Service, Capacity Building, and Roy Johnson (“Put some gratitude in your attitude!”).

GoodJob version 1.3 is released. GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails. If you’re new to GoodJob, read the introductory blog post.

GoodJob’s v1.3 release adds a mountable Web Dashboard and improved README documentation and complete code-level YARD documentation.

Version 1.3’s release comes five weeks after v1.2 and three months after GoodJob’s initial v1.0 release.

Shoutouts 🙌

GoodJob has accepted contributions from 9 people total, and currently has 559 stars on Github. The project just passed 150 combined Issues and Pull Requests.

I’m grateful for everyone who has reached out to me on Ruby on Rails Link Slack (@bensheldon) and Twitter (@bensheldon)

🙏

Mountable web dashboard

GoodJob v1.3 adds a web dashboard for exploring and visualizing job status and queue health.

The web dashboard is implemented as an optional Rails::Engine and includes charts and lists of pending jobs.

I expect the web dashboard to be a hot area of ongoing improvement. This initial release contains a minimum functional interface, a chart, and some necessities like keyset pagination. The dashboard is familiar to develop (Rails Controllers, ERB Views and ActiveRecord), and I’ve adopted Bootstrap CSS and Chartist to ease and speed development.

Improved documentation

GoodJob’s README has been edited and rewritten for clarity and comprehensiveness, and GoodJob’s implementation code is now thoroughly documented with YARD.

Good documentation is vital for open source projects like GoodJob. I worked with Rob Brackett, who consults on complex software and open source challenges.

Upcoming

In the next release, v1.4, I plan to continue adding views and charts to the web dashboard and improving thread management.

Contribute

Code, documentation, and curiosity-based contributions are welcome! Check out the GoodJob Backlog, comment on or open a Github Issue, or make a Pull Request.

I also have a GitHub Sponsors Profile if you’re able to support GoodJob and me monetarily. It helps me stay in touch and send you project updates too.

Doublespeak from Pro-Publica’s “Meet the Customer Service Reps for Disney and Airbnb Who Have to Pay to Talk to You”, via Pluralistic’s ongoing chickenization coverage:

Arise carefully monitors the language agents use to reinforce that it does not have an employment relationship with them. Stung by lawsuits that claimed Arise had actually employed agents but didn’t pay them fairly, Arise’s legal department has become a kind of word police, one former staffer told ProPublica.

“You don’t schedule ‘hours,’ you schedule ‘intervals,’” the former staffer said. Agents were not to be addressed as “you,” but “your business.” They were not “working,” they were “servicing.” There were no “supervisors,” only “performance facilitators.” Agents were not “coached.” Rather, their services were “enhanced.”

Once, an Arise manager, testifying in an arbitration hearing, was asked about meetings that performance facilitators have with agents. “They’re not meetings,” he said. “They’re informational sessions, or hosts.”

In an internal announcement in 2012, Arise listed “new terminology” for eight terms to avoid “the misconception” that agents are Arise employees. The corporate link between Arise and the agents went from being called Virtual Services Corporations to Independent Businesses. Service Fees became Service Revenue. Central Operations became Support Operations.

Arise seems particularly unable to settle on a term for the agents. Early on, the company called each a CyberAgent. Later came Arise Certified Professional. In 2012, that was changed to Client Support Professional. Nowadays, Arise’s website calls agents “onshore brand advocates or Service Partners.”

“Arise-speak,” as one opposing attorney called it in legal proceedings, could be a wonder to behold. Client Support Professionals (CSPs) would work with Quality Assurance Performance Facilitators (QAPFs) in a Performance Enhancement Session (PES), or they might reach out to Chat Performance Facilitators or Escalation Performance Facilitators, and none would be an Arise employee, all would be independent contractors.

And a disturbing exchange:

Rice said he worked out of his bedroom, in his mother’s home, helping customers for Arise’s clients, including Barnes & Noble, Disney and Sears. While testifying, Rice referred to performance facilitators in the Arise network as supervisors. This elicited a challenge from a lawyer for Arise.

“Where’d you get that term from, supervisor?” the lawyer asked.

“Growing up in America,” Rice said. “That’s the term people use for people that are above you.”

“… You never referred to them as supervisors while you were providing services, did you?”

“Well, yeah,” Rice said.

“You did? To who?”

“Well, obviously I’m on the phone with a customer, I’m not going to say, ‘OK, let me go check my chat performance facilitator.’ Usually I just said, you know, ‘Let me just talk to my supervisor.’”

“Respect our vendors” is foreign enough to me in software engineering that I took this picture of Costco’s values. The opposite of respect, perhaps “contempt for vendors and tools,” seems endemic.

At one job, memorably, a coworker was fired over it. Our engineering team had a shared email list used when setting up root accounts on various 3rd party services, including our primary infrastructure vendor for whom we were one of their largest customers.

The infrastructure vendor sent a Net-Promoter Score-like survey to our shared email list. Receiving this kind of marketing junk was frequent enough that I ignored it, but my colleague filled it out:

On a scale of 1-10, how likely are you to recommend our service?: “1”

Why?: “I hate you.”

This survey response led to the vendor’s account executive making a frantic and fearful call to our leadership team. The blowback of that led to my coworkers’ termination. (This was not my colleague’s first warning; my team and I were also targets of their trolling and bullying.)

This is a funny story to reflect on. And it’s terribly toxic the multitude of things engineers will despise, wear on their sleeve, and eagerly share at the slightest opportunity, myself included.

GoodJob version 1.2 has been released. GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails. If you’re new to GoodJob, read the introductory blog post.

GoodJob’s v1.2 release adds multithreaded queue isolation for easier congestion management, and usage of Postgres LISTEN/NOTIFY to greatly reduce queue latency.

Version 1.2 comes out 2 weeks after GoodJob v1.1, and 5 weeks after GoodJob’s initial v1.0 release.

Multithreaded queue isolation

GoodJob v1.2 adds multithreaded queue isolation for easier congestion management. Queue isolation ensures that slow, long-running jobs do not block the execution of higher priority jobs.

Achieving queue isolation has always been possible by running multiple processes, but GoodJob v1.2 makes it easy to configure multiple isolated thread-pools within a single process.

For example, to create a pool of 2 threads working from the mice queue, and 1 thread working from the elephants queue:

$ bundle exec good_job --queues="mice:2;elephants:1"

Or via an environment variable:

$ GOOD_JOB_QUEUS="mice:2;elephants:1" bundle exec good_job

Additional examples and syntax:

• --queues=*:2;mice,sparrows:1 will create two thread-pools, one running jobs on any queue, and another dedicated to mice and sparrows queued jobs.
• --queues=-elephants,whales:2;elephants,whales:1 will create two thread-pools, one running jobs from any queue except the elephants or whales, and another dedicated to elephants and whales queued jobs.

LISTEN/NOTIFY

GoodJob now uses Postgres LISTEN/NOTIFY to push newly enqueued jobs for immediate execution. LISTEN/NOTIFY greatly reduces queue latency, the time between when a job is enqueued and execution begins.

LISTEN/NOTIFY works alongside GoodJob’s polling mechanism. Together, jobs queued for immediate execution (ExampleJob.perform_later) are executed immediately, while future scheduled jobs (ExampleJob.set(wait: 1.hour).perform_later) are executed at (or near) their set time.

Upcoming

In the next release, v1.3, I plan to include a simple web dashboard for inspecting job execution performance, and focus on improving GoodJob’s documentation.

Contribute

Code, documentation, and curiousity-based contributions are welcome! Check out the GoodJob Backlog, comment on or open a Github Issue, or make a Pull Request.

I’ve also set up a GitHub Sponsors Profile if you’re able to support me and GoodJob monetarily. It helps me stay in touch and send you project updates too.

GoodJob version 1.1 has been released. GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails. If you’re new to GoodJob, read the introductory blog post.

GoodJob’s v1.1 release contains a new, economical execution mode called “async” to execute jobs within the webserver process with the same reliability as a separate job worker process.

This release also contains more in-depth documentation based on feedback and questions I’ve received since the v1.0 release.

Version 1.1 comes out 3 weeks after GoodJob v1.0. The initial release of GoodJob was featured on Ruby Weekly, A Fresh Cup

, Awesome Ruby, and was as high as #8 on Hacker News. GoodJob has since received nearly 500 stars on Github.

Async mode

In addition to the $ good_job executable, GoodJob now can execute jobs inside the webserver process itself. For light workloads and simple applications, combining web and worker into a single process is very economical, especially when running on Heroku’s free or hobby plans.

GoodJob’s async execution is compatible with Puma, in multithreaded (RAILS_MAX_THREADS), multi-process (WEB_CONCURRENCY), and memory efficient preload_app! configurations. GoodJob is built with Concurrent Ruby which offers excellent thread and process-forking safety guarantees. Read the GoodJob async documentation for more details.

On a personal level, I’m very excited to have this feature in GoodJob. Async execution was the compelling reason I had previously adopted Que, another Postgres-based backend, in multiple projects and I was heartbroken when Que dropped support for async execution.

Improved documentation

Since GoodJob was released 3 weeks ago, the documentation has been significantly expanded. It contains more code and examples for ensuring reliability and handling job errors. I’ve had dozens of people ask questions through Github Issues and Ruby on Rails Link Slack.

Upcoming

In the next release, v1.2, I plan to simplify the creation of multiple dedicated threadpools within a single process. The goal is to provide an economical solution to congestion when the execution of a number of slow, low-priority jobs (elephants) are being performed and there are no execution resources available for newly introduced fast, high priority jobs (mice) until the currently executing elephants complete.

A proposed configuration, for example:

--queues=mice:2,elephants:4

…would allocate 2 dedicated threads for jobs enqueued on the mice queue, and 4 threads for the elephants queue. Learn more in the feature’s Github Issue.

Contribute

GoodJob continues to be enjoyable to develop and build upon Rails’ ActiveJob and Concurrent Ruby. Contributions are welcomed: check out the GoodJob Backlog, comment on or open a Github Issue, or make a Pull Request.

GoodJob is a new Postgres-based, multithreaded, second-generation ActiveJob backend for Ruby on Rails.

Inspired by Delayed::Job and Que, GoodJob is designed for maximum compatibility with Ruby on Rails, ActiveJob, and Postgres to be simple and performant for most workloads.

• Designed for ActiveJob. Complete support for async, queues, delays, priorities, timeouts, and retries with near-zero configuration.
• Built for Rails. Fully adopts Ruby on Rails threading and code execution guidelines with Concurrent::Ruby.
• Backed by Postgres. Relies upon Postgres integrity and session-level Advisory Locks to provide run-once safety and stay within the limits of schema.rb.
• For most workloads. Targets full-stack teams, economy-minded solo developers, and applications that enqueue less than 1-million jobs/day.

Visit Github for instructions on adding GoodJob to your Rails application , or read on for the story behind GoodJob.

A “Second-generation” ActiveJob backend

Why “second-generation*”? GoodJob is designed from the beginning to be an ActiveJob-backend in a conventional Ruby on Rails application.

First-generation ActiveJob backends, like Delayed::Job and Que, all predate ActiveJob and support non-Rails applications. First-generation ActiveJob backends are significantly more complex than GoodJob because they separately maintain a lot of functionality that comes with a conventional Rails installation (ActiveRecord, ActiveSupport, Concurrent::Ruby) and re-implement job lifecycle hooks so they can work apart from ActiveJob. I’ve observed that this can make them slow to keep up with major Rails changes. An impetus for GoodJob was reviewing the number of outages, blocked upgrades, and forks of first-generation backends I’ve managed during both major and minor Rails upgrades over the years.

As a second-generation ActiveJob backend, GoodJob can draft off of all the advances and solved problems of ActiveJob and Ruby on Rails. For example rescue_from, retry_on, discard_on are all implemented already by ActiveJob.

GoodJob is significantly thinner than first-generation backends, and over the long run hopefully easier to maintain and keep up with changes to Ruby on Rails. For example, GoodJob is currently ~600 lines of code, whereas Que is ~1,200 lines, and Delayed::Job is ~2,300 lines (2,000 for delayed_job, and an additional 300 for delayed_job_active_record).

*“Second generation” was coined for me by Daniel Lopez on Ruby on Rails Link Slack.

Postgres-based

I love Postgres. Postgres offers a lot of features, has safety and integrity guarantees, and simply running fewer services (skipping Redis) means less complexity in development and production.

GoodJob builds atop ActiveRecord. It’s numbingly boring, in a good way.

GoodJob uses session-level Advisory Locks to provide run-once guarantees with relatively little performance implications for most workloads.

GoodJob’s session-level Advisory Lock implementation is perhaps the only “novel” aspect, that comes from my experience orchestrating complex web-driving of government systems (“the browser is the API”) for Code for America. GoodJob uses a Common Table Expression (CTE) to find, lock, and return the next workable job in a single query. Session-level Advisory Locks will gracefully relinquish that lock if interrupted, without having to maintain a transaction for the duration of the job.

Multi-threaded

GoodJob uses Concurrent::Ruby to scale and manage jobs across multiple threads. “Concurrent Ruby makes one of the strongest thread-safety guarantees of any Ruby concurrency library”. Ruby on Rails has adopted Concurrent Ruby, and GoodJob follows its lead and thread-execution and safety guidelines.

In building GoodJob I leaned heavily on my positive experiences running Que, another multithreaded backend, on Heroku. Threads are great for balancing simplicity, economy, and performance for typical IO-bound workloads like heavy database queries, API requests, Selenium web-driving, or sending emails.

A feature that won’t be in GoodJob 1.0, but I hope to implement soon, is the ability to run the GoodJob scheduler inside the webserver process (“async mode”). This was a feature withdrawn from Que , but I believe can be safely implemented with Concurrent Ruby. An async mode would offer even greater economy, for example, in Heroku’s constrained environment.

GoodJob is right for me

GoodJob’s design is based directly on my experience in 2-pizza, full-stack teams, and as an economy-minded solo developer. GoodJob already powers Day of the Shirt and Brompt performing tens-of-thousands of real-world jobs a day.

Is GoodJob right for you?

Try it out and let me know.