Hi! šŸ‘‹ Iā€™m Ben Sheldon. Iā€™m the author of GoodJob, an Active Job backend that Iā€™ll humbly share is mildly popular and known for its broad features and ease of use. Iā€™m working on a new plugin for Rails: āœØ

Spectator Sport creates and replays video-like recordings of your live, production website, via a self-hosted Ruby on Rails Engine that lives in your application.

Spectator Sport uses the rrweb library to create recordings of your websiteā€™s DOM as your users interact with it, from the perspective of their web browserā€™s screen (html, css, images, dynamic content, mouse movements and clicks, navigation). These recordings are stored in your Active Record database for replay by developers and administrators to analyze user behavior, reproduce bugs, and make building for the web more engaging, satisfying, and fun.

Hereā€™s a proof of concept demo. Itā€™s very, very, very, very rough and early, but everyone I have demoed it for says ā€œwow, that is totally different and much better than I imagined it when you first explained it meā€: https://spectator-sport-demo-1ca285490d99.herokuapp.com

šŸš§ šŸš§ This gem is very early in its development lifecycle and will undergo significant changes on its journey to v1.0. I would love your feedback and help in co-developing it so fyi itā€™s going to be so much better than it is right now.

You can help:

Who is this gem for?

Iā€™m writing this on the weekend following Rails World 2024, on eve of Rails 8ā€™s release. The Rails team is tackling the hard problems of making it easy to deploy your website into production with tools like Kamal, and addressing operational complexity with the Solid suite. What comes next?

Spectator Sport intends to transform your relationship with your website after it is deployed to the web and real people have the opportunity to use it:

  • See how people are actually using your website, directly.
  • Remove the necessity of defining funnel metrics or analytics up front, or the necessity of interpreting user behavior through a limited lens of aggregated or averaged numbers.
  • As a developer and product-maker, more fully engage your sympathetic senses, in addition to your analytical senses, to ultimately be more effective and fulfilled when building for the web.

Launching a website kinda sucks. Iā€™ve been a solopreneur, and web-marketing consultant, and ā€œfoundingā€ engineer, and ā€œgrowthā€ engineer at VC backed startups and product labs, and a participant and mentor in entrepreneur communities and mastermind and accountability groups for 19 years. People are not ok.

The fast development feedback of imagining and building the thing locally, the dopamine rush of making something nice, one step at a time, for other people to useā€¦ that all drops off a cliff once, well, you put it out there for other people to use. The post-launch and release feedback: itā€™s not there.

It sucks! People feel it! Theyā€™re confused, theyā€™re sad, sometimes mad, looking for help, wanting to be seen by others, spinning on valueless technical changes, sharing tangential hot takes and engagement baits. Developers are going anywhere but directly to the people theyā€™re building for. One reason, I believe, is because their visitorsā€™ and usersā€™ activity on their website is largely invisible and unknowable, and the only way to see it is through a foggy, confusing and deeply unsatisfying window of abstract metrics and aggregation.

Building for the web should be a spectator sport. More than only a fantasy game of metrics and aggregates and guesses and spread-table gambles. It should be fun and engaging and direct. We should simply be able to observe and react and cheer and cry and fall on the floor and get up and make it better and go again. Believe.

There are constraints to what I can do to achieve this vision, with this gem. Iā€™m focused on building for Ruby on Rails. And specifically hobbyists, soloprenieurs, small teams, SMBs (small and midsize businesses), and unique applications, including:

  • applications with limited budgets or the inability (because of geography or policy) to contract or procure a 3rd party service.
  • applications in government or healthcare or on an internal intranet with unique data or privacy constraints who donā€™t have the budget for a BAA (business associate agreement) or other compliance contracts
  • applications for which operational simplicity is paramount and donā€™t have the resources to operate a more complex self-hosted solution

We have the technology

Browser recording isnā€™t new. Fullstory was my own introduction to it nearly a decade ago, also Tealeaf and Sentry and PostHog and Highlight and Matomo and many others, some of which are no-cost self-hostable as a separate service, though often with complex dependencies. Many of them use rrweb too.

I believe Spectator Sport is the first no-cost, self-hostable browser-recording tool that works anywhere your application runs (Heroku being the narrowest target I can imagine). Tell me what Iā€™m missing!

If my adjectives themselves arenā€™t compelling and your website already has massive scale and a rich revenue stream and/or no concerns about 3rd-party data subprocessors, I highly recommend checking out PostHog (just $0.005 per recording!) or Sentry (enterprise gated, but integrated into Sentryā€™s other features which are fantastic).

A good job, again

I mentioned in my introduction that my other gem, GoodJob, is well-regarded. I think we can do it again with Spectator Sport:

  • Focus on solving a class of problems developers experience over a long period of time, not building a specific technology tool and calling it a day.
  • Serve the vastly more solo and full-stack dev teams with limited time and budgets who will benefit from something tailored to their largely consistent needs (easy, good, inexpensive) and are nice and appreciative when you deliver, than the very small number of experienced folks with big budgets and unique needs who inexplicably have time on their hands to be outspoken in telling you it will never work for them.
  • Provide a wide offering of polished features, using boring, existing tech to do the complex bits (like Postgres advisory locks in GoodJob, or rrweb in Spectator Sport). The value comes from the usability of the integration. A full-featured, cleanly designed web dashboard really impresses too; Dark Mode is the epitome of a non-trivial feature to maintain that demonstrates care.
  • Maintain a narrow compatibility matrix, focus on ā€œomakaseā€ Rails (Active Record, Active Storage, etc.) with a sensible EOL policy. Complexity kills. Relational databases are great. Squeeze the hell out of the system you have.
  • Be exceptionally responsive and supportive of developers who need help and meet them where they are. Be personally present because the library canā€™t speak for itself. Make mistakes, change direction, communicate intent, move forward.
  • Keep the cost of change low, release frequently, build up, iterate, document and test and provide deprecation notices, follow SemVer, and defer application-breaking changes as long as possible.

I do want to try one thing new compared to GoodJob: I want Spectator Sport to be compatible with Postgres and MySQL and SQLite. I believe itā€™s possible.

Front-running the criticism

Here are the things I have worked through myself when thinking about Spectator Sport, and talked about with others:

Is it creepy? Yes, a little. There is overlap with advertising and marketing and ā€œgrowthā€ tech, And many service providers market browser recording as a premium capability with premium prices and sell it hard. Realistically, I have implemented enough dynamic form validations in my career that I no longer imagine any inherent sanctity in an unsubmitted form input on a random website. Conceptually, Spectator Sport observes your website as it is operated by a user, it does not observe the user. Every webpage deserves to be a place, and this just happens to be your CCTV camera pointed at it, for training purposes.

Is it a replacement for usability research? No, of course not. Spectator Sport can only show you half of the picture (or less) that you get from real usability research. When you do real usability research and ask a subject to do something on your website, you ask them to explain what theyā€™re doing, in their own words, based on their own understanding of the task and what they see through their own eyes. Browser recordings alone canā€™t give you all that. You still have to fill in the blanks in the story.

Is it safe? I think so. I intend all user input to be masked by default, be secure by default, and provide comprehensive documentation that explains both the why and the how to lock down whatā€™s stored and who can access it. Spectator Sport is shipping the DOM to your own database, and itā€™s likely the same data already lives in the database in a more structured way, and is already reflected back through your application too.

Does it use a lot of storage? Not as much as you might fear. If peopleā€™s big scaling worry for GoodJob was ā€œit will be too slowā€ I already think Spectator Sportā€™s is ā€œit will be too bigā€. Iā€™ve been running the proof of concept on my own websites and 1.5k recordings took up ~500MB of storage in Postgres. Retention periods can be configured, data can be compressed and offloaded to Active Storage. I believe it is ok, and worth the squeeze.

Can it do xyz? Maybe. Open an issue on GitHub. Iā€™d love to discuss it with you.

Wouldnā€™t you rather do something with AI? I dunno, man. I freaking love watching recordings of my websites being driven by people and thinking about how to make the website easier and better for them. I think this is an immensely satisfying missing piece of building for the web, and I think you will too.

Tell me what Iā€™m missing or overlooking!

The call to action, a second time, at the bottom

Something I learned a long time ago, from watching browser recordings (true story!), is that visitors will go deep below the heroā€™s call-to-action, read all the lovely explanatory content, get to the bottomā€¦ and bounce because the call to action wasnā€™t reinforced.

So, please: