Five years on food stamps

This reflection of my experience on GetCalFresh is dedicated to the memory of Andrew Hyder who taught me a lot about living out and protecting team values every day.

Plus: We are the team; Extreme Programming are the values

We are not the website. From the beginning, there was an opinionated directed emphasis on building healthy team norms and values. These principles and doctrines were from “Extreme Programming Explained”; the values were legible and strongly reinforced by team leadership. All of the things we have created (websites, text messaging, document processing, research) were outputs not artifacts and never the goal. There was a strong emphasis on “And the friends we made along the way.”

Plus: An experience of critical inspiration

This was always an exercise in pattern matching what we were trying to achieve in the benefits space to Bay Area tech ecosystem structures and practices. Early debates of products and services, jobs to be done, design principles, disruption (“making the status quo untenable”), leverage, and the thing itself we were building: a funnel-based, outreach machine that disintermediated the benefits process. It was a practical act of constructive criticism of the tech and tech culture around us that recognized mastery of existing forms before we inevitably depart and transcend them (“shuhari”).

Plus: Incremental growth is a hell of a drug

A growth engine, an “engine that eats barriers”. It demonstrated that a line that went “up and to the right” could itself pay the organizational story tax. While we celebrated partnership wins, there was a focus on growing client outreach unbounded (“permissionless”). As software engineers, we were achieving the Agile principle that working software, in the hands of users, was the primary measure of progress.

58 counties was an incremental goal, that was out of reach but attainable, which made planning and decision making tangible, though not easy. It helped align the team and gave direction to planning out the next steps and necessities.

Delta: Maintain Client Success as the centerpole during growth

I think of “scaling” entailing both size and health. Early on, “Client Success” was the unifying intention of everyone (“close the participation gap”). Everyone was exhorted to use their whole selves to help clients receive benefits. I remember being mindblowingly told “you have their phone number, call them” when I observed clients hit technical or process barriers; I did, again and again. Technology was in the truest sense an extension of our embodied capacity to help.

As time went on the client success function became less of a critical inspiration, and instead came to feel closer to the status quo: handling the overwhelming externalities of the service, rather than the service being the multiplier of our puny human abilities. I feel a responsibility to help balance “we don’t need to help ourselves” with “put on your oxygen mask before helping others”; and “leave a seam” with “a stitch in time saves nine”.

And “success” felt narrowed with a refocusing towards “Systems Change” and the subjectless “Build with”. My desires for lighting escalation paths (“click here to talk to a supervisor; a director; your state representative; the FBI”), and consciousness-raising (“20k people in your county also had their benefits denied this month too”) remain unfulfilled.

Plus: Small teams can do big things

From me starting as the first outside hire, to me rolling off the team, we helped more than 8M people receive $5.3B in benefits. And the team continues on without me: I feel missed, unnecessary, and proud of both. Small, directed (North Star: “close the participation gap”; Mission: “use the principles and practices of the digital age to improve how the government serves the American public and how the public improves government”) teams, can do things together that far exceed the capacity of any one individual, and also outperform teams multiples of their size. It has been an amazing journey.


Over-rapported

I have thought and referenced a theme from this Foreign Affairs piece “Why America Can’t Build Allied Armies” quite a bit:

U.S. military advisers are taught to prioritize their relationships with their counterparts, as “trust and confidence define how well the advisor will be able to influence the foreign security force.”

Many U.S. military advisers come to view interpersonal rapport with partners—once a tool presumed to increase influence and shape partner behavior—as the very goal of advising. When persuasion fails, advisers are instructed to avoid any steps that might jeopardize the relationship.

…The military is aiming to minimize bureaucratic headache and implement the standard operating procedures it can sustain, even if those procedures are untethered from its own theory of victory.


Software models

Stickynote drawing of a chart in which understanding of the business domain grows, and understanding of the modeling in code sinuisoidally approaches and departs from it.

Software design has been in my thoughts lately. Particularly agile software design, in which the game is to re-design working software in response to changing business requirements in response to learnings from working software.

1. I was recently talking to a colleague about what I call “Strict” Technical Debt, which is how Ward Cunningham originally defined it (not YAGNI or Skimping according to Ron Jeffries). Dave Rupert pulls out the following quote from Ward Cunningham, as “Technical Debt as a lack of understanding”:

“If you develop a program for a long period of time by only adding features but never reorganizing it to reflect your understanding of those features, then eventually that program simply does not contain any understanding and all efforts to work on it take longer and longer.”

2. This sentence from SICPER’s “Aphorism Considered Harmful” about the phrase “Make it work, make it right, make it fast”:

So actually it looks like I had the wrong idea all this time: you don’t somehow make working software then correct software then fast software, you make working software and some inputs into that are the abstractions in the interfaces you design and the performance they permit in use.

Or as I interpret it: make it functional, then make the software design be an appropriate representation and model of the business/problem domain, and then make it appropriate/ergonomic for future developers to use or maintain.

3. I left this comment on Jason Swett’s “Why I don’t buy “duplication is cheaper than the wrong abstraction”:

…The statement I use is “code duplication is better than the wrong business abstraction”.

A lot of these statements/concepts come out of Agile, Object-Oriented Programming, and Extreme Programming recognize that there are two things evolving unevenly: the understanding of the business domain, and then laggingly, the development of the code that models the understanding of the business domain.

A better expansion here as I understand the concept would be “duplication of code is acceptable if the proposed extraction/abstraction does not meaningfully model the current understanding of the business domain”.

The power of the statement, as I use it (and as I see these things go from Agile/OOPS/XP) is to push someone to get closer to the business domain (and the user) in their thinking/proposals/justifications and as a counterweight to DRY. The problem with these things is when they get imposed as a criticism, rather than the starting place of a conversation.

I think you give solid advice:

Don’t try to make one thing act like two things. Instead, separate it into two things. If you feel reluctant to modify someone else’s code, ask why that is.


It asks you to be interactive

From the introduction to Sarah Schulman’s Conflict Is Not Abuse: Overstating Harm, Community Responsibility, and the Duty of Repair:

Nan Alamilla Boyd helped me to understand that my lack of academic training makes me literally “undisciplined.” This news was very freeing, and a gift I wish I had been handed decades before. I now am able to ask you to read this book the way you would watch a play: not to emerge saying, “The play is right!” but rather to observe that the play reveals human nuance, contradiction, limitation, joy, connection, and the tragedy of separation. That the playwright’s own humanity is also an example of these unavoidable flaws. These chapters are not homogenous. As a creative writer I have long understood that form should be an organic expression of the feelings at the core of the piece. Each chapter here serves a different function and that is represented in its tone, genre, style, and form. Some are journalistic, some analytical, some are speculative, others abstract, some are only feelings. As a novelist, I know that it is the cumulative juxtaposition that reveals the story. This is not a book to be agreed with, an exhibition of evidence or display of proof. It is instead designed for engaged and dynamic interactive collective thinking where some ideas will resonate, others will be rejected, and still others will provoke the readers to produce new knowledge themselves. Like authentic, conscious relationships, truly progressive communities, responsible citizenship, and real friendship, and like the peace-making that all these require, it asks you to be interactive.

And:

“Differentiating between Power Struggle and Power Over,” Hodes explained, “is the difference between Conflict and Abuse.” Abuse is Power Over and Conflict is Power Struggle.

[…]

What we have instead is a devolved definition of personal responsibility, which constructs avoidance as a right regardless of the harm it does to others. This negative standard persuades some people to feel that being uncomfortable signals that they are being Abused, because they don’t have the option of describing themselves as Conflicted. So asking a distressed person if they are unsafe, or rather, uncomfortable, angry, or hurt provides them with an alternative idea that might fit better with their actual experience. It not only elicits helpful information, but encourages the individual to start to think about themselves in a more adult, complex, and responsible manner.

[…]

Lesson: never, ever decide that you know who someone is, what they did, their objective, context or goal, how they feel or what they know, until you ask them. And not asking means a direct investment in not understanding the truth.

And:

“Mainstream Domestic Violence advocacy,” Hodes said in a correspondence later that year, “is committed to assuming that the victim is telling the truth, and any exploration around that trope is met with heavy resistance. Historically, that makes sense for a host of reasons. But this analysis is not about disbelieving, it’s about pinpointing where the problem lies.” One of Hodes’ many valuable suggestions is to lower the bar for what must happen in a person’s life for their suffering to be acknowledged. “The current paradigm is encouraging all of us to think we are in abusive relationships,” Hodes explained. “And if you are not in an abusive relationship, you don’t deserve help. Being ‘abused’ is what makes you ‘eligible.’ But everyone deserves help when they reach out for it.” This is a strikingly humane idea: that the collapse of Conflict and Abuse is partly the result of a punitive standard in which people are made desperate, yet ineligible, for compassion. This is a non-cynical reading of a human condition in which people who have suffered in the past, or find themselves implicated in situations in which they are afraid to be accountable, fear that within their group acknowledging some responsibility will mean being denied their need to be heard and cared for. So they fall back on the accusation of Abuse to guarantee that they will not be questioned in a way that confirms these fears.


Even if the answers are messy and provisional

I liked this (email) conversation between Willa Glickman of NYRB and English professor Anahid Nersessian in “Late Romanticism”:

Perhaps understandably for a scholar with an interest in utopia, if less expectedly for a scholar of criticism, Nersessian believes in the importance of pressing forward with new visions rather than just critiquing existing ideas and systems. “I think lots of people believe the world right now is pretty bad—politically, ecologically—and that capitalism is responsible for a very large part of the situation. But being against capitalism doesn’t entail being for anything else,” she said:

It’s all very well to say “capitalism turns us into commodities, that’s degrading, human beings shouldn’t be degraded.” But how should we be treated, and how should we live? What would love and sex—among other things, like health care or having a job—look like in a good world? It’s important to take the risk of answering those questions, even if the answers are messy and provisional.

And:

I wondered how she dealt with the complex relationship between literature and history in her own work—as she argues in her essay in the Review, relying on novels to understand the past can distort one’s vision of it. “It’s very tempting to use literature to confirm a hypothesis you already have about the world. For better and for worse, though, fictional texts don’t always give us good information about reality,” she explained:

Another danger can be to lean too hard on canonical literature that tells an “official” story about what life or people or relationships were like in such-and-such a time. If you ask a classicist, they’ll tell you that Plato—a member of an elite, all-male, free, and politically enfranchised class—is not your best informant for how the vast majority of ancient Athenians were living their lives, having sex, or negotiating intimacy.


TIL: ActiveRecord transactions roll back on Ruby Thread abort

I learned that ActiveRecord would roll back transactions when inside an aborted thread. It’s implemented right here, in ActiveRecord’s connection_adapters/abstract/transaction.rb:

def within_new_transaction
  # ...
ensure
  # ...
  elsif Thread.current.status == "aborting" || (!completed && transaction.written)
    # The transaction is still open but the block returned earlier.
    #
    # The block could return early because of a timeout or because the thread is aborting,
    # so we are rolling back to make sure the timeout didn't caused the transaction to be
    # committed incompletely.
    rollback_transaction
  # ...
end

How did I end up here? I recently implemented a feature in GoodJob to track active processes in the database using ActiveRecord. These database queries occur in a background thread where I also use the same database connection for GoodJob’s Postgres LISTENing. It looks something like this (pseudocode):

Concurrent::Future.execute do
  process = Process.create
  loop { listen_for_notify }
ensure
  process&.destroy!
end

While working on one of my Rails projects that use GoodJob (https://dayoftheshirt.com), I noticed that Process records were not cleaned up as I expected running when GoodJob async inside of Rails Server/Puma locally and exiting. Inspecting the logs, I saw this:

TRANSACTION (0.1ms)  BEGIN
GoodJob::Process Destroy (0.3ms)  DELETE FROM "good_job_processes" WHERE "good_job_processes"."id" = $1  [["id", "320de861-4c84-4f8c-ba3e-2e08a8ef0469"]]
TRANSACTION (0.1ms)  ROLLBACK

Huh. I did some Googling and found some references to rollback behavior, and asked in the Rails Link Slack, but nobody knew. This caused me to dig in ActiveRecord, where I found the behavior implemented.

I was still confused because I had never seen this behavior in GoodJob when building out the feature. But after much head-scratching, I realized that I hadn’t followed GoodJob’s README instructions for integrating with Puma, which ensures that GoodJob is gracefully shutdown before Ruby aborts threads at exit:

# config/puma.rb

on_worker_shutdown do
  GoodJob.shutdown
end

MAIN_PID = Process.pid
at_exit do
  GoodJob.shutdown if Process.pid == MAIN_PID
end

Mystery solved.


Javascript is a management problem

I have been on board with low/no javascript for a long time for a variety of observed and experienced reasons, and grasping around for a succinct explanation. Baldur Bjarnason captures it in “The Single-Page-App Morality Play”:

The problem is management.

It is a management problem. Truly.

The Multi-Page-App forces the team to narrow the scope to a level they can handle. It puts a hard limit on their technological aspirations. Mandating a traditional Multi-Page-App under the auspices of performance, accessibility, or Search-Engine-Optimisation is a face-saving way to force the hand of management to be more realistic about what their teams can accomplish. When we can accomplish the same by advocating for a specific Single-Page-App toolkit or framework, that’s what most of us nominally on the ‘Anti’ side do. I regularly advocate for Svelte when I think the team can handle its long term implications in terms of complexity. (That might change as Svelte adds more features.)

The problem with Single-Page-App frameworks, even the ones like SvelteKit who could claim to be more Hybrid than just SPA, is that they are very, very eager to enable ‘scale’ of any sort. Features, app size, code complexity, integrations, etc. They are desperate to make sure that you can keep using their framework if you become a mythical ‘unicorn’ startup and your project grows into the next Facebook. So they put a lot of hard work into making sure that there is no upper limit to the scope of the app you can make with them.

Which, when they present it as ‘scale’, sounds like a good thing. But it’s absolutely a bad thing when you’re in an industry that’s as mismanaged as ours. We can’t handle complexity. Having no upper limit to it is extremely bad.


I'm burning dude... where do we go from here?

A powerful conversation from Reveal’s “Mississippi Goddam Chapter 6: Mississippi Justice” (46:10) between reporters Al Letson and Jonathan Jones:

Al Letson

After the interview, I was still thinking about everything Curley said. JJ and I walked back to our car mostly in silence, but on the ride back, we talked about it.

Okay, so, what did you think about the interview with Curley?

Jonathan Jones

It was good. It was slightly pessimistic at the end of it.

Al Letson

That’s kind of what I wanted to talk to you about though, is that like, I am just really tired of people asking Black folks to have hope, and to make people feel better about the American situation, when everything in our history tells us that that is not the case. And I especially am uncomfortable with it when we are asking people who are doing the work, like Curley Clark is out there doing the work to try to make this world a more just place. And I, in the work that I choose to do, I’m trying to make the world a better place in it. And in the work that you do, it’s the same reason. We’re both driven to the work that we do and the cases that we look at and pick, because we want this place to be more just for everybody. And so, but the idea that somehow or another, Black folks are supposed to have hope, and Black folks are not supposed to be pessimistic. It just feels yucky to me, because it’s like asking us to look at all the history of what’s happened in this country and kind of ignore it.

Jonathan Jones

No, but I don’t disagree with you. I think that’s a misunderstanding of what I was asking. I wasn’t fishing for Black folk to fight.

Al Letson

No, I don’t think you were. That’s not what I’m saying. I’m not saying that you were telling me…

Jonathan Jones

Well, let me just… I mean, I am thinking about, I want to inspire. I want to… I don’t want to be an activist, but I hope our journalism helps to make this better world. And in investigative reporting, we do a great job of finding bad guys. We don’t do a very good job about solutions.

Al Letson

I think we look at it in two different ways, in the sense of, we’re both shooting for the same goal. And I want to be really clear that I didn’t think that you were telling Curley Clark to work harder. I felt like you were asking Curley, and by extension me, why don’t we have hope, and where’s the hope? And my point is that the history of this country tells me, no, no, I can’t do that anymore.

And I would say that, I think it’s beautiful that you want to inspire people. I do. I love that about you. And in certain ways, I do want to do work that inspires people, but really at this point in my life and where I am, I don’t want to inspire people, I want to infuriate people. I want to make work that make people burn so hard that they feel like this injustice is wrong, and they want to go out, and they want to tear it all down.

And I know that we’re talking about the same thing, but we’re talking about it in different ways, right? I know that you want to do the exact same thing, that you want to inspire people for this, to this higher cause. And I think that that is useful and smart, but also, I think, just like in series that we’re working on, I couldn’t do this without you, and you couldn’t do this without me, and the inspiration is definitely necessary, but fuck, man, I got so much anger.

I got so much anger, and I’m so tired of hiding it, and I’m so tired of pretending that it’s not there, and I’m so tired of this idea that… This shit really chokes me up a little bit. I’m just, I’m sick of it, man. I’m sick of it, And it makes… I want to… I’m burning, dude. And that’s why I’m doing this work, is because I’m seeing how the world is, and it’s not just Mississippi. It’s all, it’s the entire country. You see how it’s happening, and I don’t have hope, but what I do have is this rage, and it’s burning in me. And that’s what I hope that we do. I hope that we people off so bad at that they create change from that fire.

Jonathan Jones

And that seems very real, and I completely affirm that. I think maybe I would have, maybe I should have phrased the question a little bit better, because that answer that you gave would have been the answer. I guess my point was, where do we go from here? It can be from anger and raw rage, and outrage at injustice. It’s just, where do we go from here? And I don’t want to have a… I’m not trying to do a hokey fairytale at all.

Al Letson

Right, right.

Jonathan Jones

And it’s… it’s hard.


Types of Corruption

An interesting breakdown of types of corruption by Yuen Yuen Ang on Freakonomics:

I propose a typology of four types of corruption divided along two dimensions. First, whether the corruption involves elites or non-elites. And second, whether the corruption involves theft or exchange. So this intersection creates, first of all, corruption with theft, which I divide into petty theft and grand theft. Petty theft would be like extortion — a police officer who just stops you and robs you of $200. Grand theft would be embezzlement. Nigeria would be a classic case, billions of dollars siphoned out of a country. And then I distinguish between two types of transactional corruption. The first is what I call speed money, which is bribes paid to low- or medium-level officials in order to overcome red tape or delays or harassment. And then I have a fourth category called access money, which is privileges paid to powerful officials, not because you want to overcome red tape, but because you want to buy special deals from them.

[…]

I use the analogy of drugs because we know that all drugs are harmful, but they harm in different ways. Petty theft and grand theft are like toxic drugs, where if you take this drug, it’s definitely going to damage your health, you get no benefit from it… Speed money are like painkillers, so they help you to relieve a headache by overcoming red tape, but they don’t help you grow muscles fast. They don’t help you to grow your business. And access money are the steroids of capitalism, and steroids, we know, help you grow muscle fast. They help you perform superhuman feats. But they come with serious side effects that accumulate over time, and they only erupt in the event of a meltdown.


Conflict in the matrix

From Apenwarr’s “What do executives do, anyway?”:

To paraphrase the book [High Output Management, by Andy Grove], the job of an executive is: to define and enforce culture and values for their whole organization, and to ratify good decisions.

That’s all.

Not to decide. Not to break ties. Not to set strategy. Not to be the expert on every, or any topic. Just to sit in the room while the right people make good decisions in alignment with their values. And if they do, to endorse it. And if they don’t, to send them back to try again.

There’s even an algorithm for this.

It seems too easy to be real. For any disagreement, identify the lead person on each side. Then, identify the lowest executive in the corporate hierarchy that both leads report into (in the extreme case, this is the CEO). Set up a meeting between the three of them. At the meeting, the two leads will present the one, correct decision that they have agreed upon. The executive will sit there, listen, and ratify it.

But… wait. If the decision is already made before the meeting, why do we need the meeting? Because the right decision might not happen without the existence of that meeting. The executive gives formal weight to a major decision. The executive holds the two disagreeing leads responsible: they must figure out not what’s best for them, but what’s best for the company. They can’t pull rank. They can’t cheat. They have to present their answer to a person who cares about both of their groups equally. And they want to look good, because that person is their boss! This puts a lot of pressure on people to do the right thing.

From HBR’s “Problems of Matrix Organizations”:

Another possible source of decision strangulation in matrix organizations occurs when managers frequently or constantly refer decisions up the dual chain of command. Seeing that one advantage of the conventional single chain of command is that two disagreeing peers can go to their shared boss for a resolution, managers unfamiliar with the matrix worry about this problem almost more than any other. They look at a matrix and realize that the nearest shared boss might be the CEO, who could be five or six echelons up. They realize that not too many problems can be pushed up to the CEO for resolution without creating the ultimate in information overload. So, they think, will not the inevitable disagreement lead to a tremendous pileup of unresolved conflict?

Certainly, this can happen in a malfunctioning matrix. Whether it does happen depends primarily on the depth of understanding that exists about required matrix behavior on the part of managers in the dual structure. Let us envision the following scene: a manager with two bosses gets sharply conflicting instructions from his product and his functional bosses. When he tries to reconcile his instructions without success, he quite properly asks for a session with his two bosses to resolve the matter. The three people meet, but the discussion bogs down, no resolution is reached, and neither boss gives way.

The two bosses then appeal the problem up a level to their respective superiors in each of the two chains of command. This is the critical step. If the two superiors properly understand matrix behavior, they will first ascertain whether the dispute reflects an unresolved broader policy issue. If it does not, they know their proper step is to teach their subordinates to resolve the problem themselves—not to solve it for them. In short, they would not let the unresolved problem escalate, but would force it back to the proper level for solution, and insist that the solution be found promptly.

Often, conflict cannot be resolved; it can, however, be managed, which it must be if the matrix is to work. Any other course of action would represent management’s failure to comprehend the essential nature of the design.

Lastly, Alex Komoroske’s deck on Slime Molds is good.