Learning with AI #3 - Spinning up agents to smoke test my products
Every morning, a user signs in to each of my products, runs a few tests, makes some notes, and then sends a report that updates a dashboard (a private page on my website). The interesting part about this is that these users are not real people and are actually agents.
When building my products, I've been focused on finding "insurance" where I can. In Learning with AI #1, I wrote about building redundancy as insurance. That was focused on what happens when you change data or how the website looks or works. These smoke tests, run by agents, are a different type of insurance. This is insurance that gives me a better shot at discovering something is broken before others do.
| Job | What is required |
|---|---|
| Eyes and hands | The ability to see what's on the page (screenshots) and act on it (clicking, typing), the way a person at a browser would. |
| Running a loop | Look, decide, act, repeat, until the journey is done. The loop means that you have something that works through a task. |
| Judgement | Asking your AI tool to make decisions, such as what would a normal user do next, and judging whether anything is broken. This is the only part that is actually AI. |
| The harness | The car to the AI's driver. The AI makes the decisions, but the harness houses it, holds all the controls, and does the actual moving until the job is done. Cowork and Claude Code are both harnesses. |
Where does the test user live?
The first decision is where does the agent "live". You have two main choices: running on your laptop or running on external servers. To run it on your laptop requires you to have your laptop lid open. I didn't love this option because I'd actually want these tests to run overnight and I wasn't going to leave my laptop lid open overnight.
That leaves running on a server, and there were three main options: GitHub, a VPS, or Vercel.
I ruled out Vercel first. Vercel's main offering is switching a server on, serving you a website, and switching it off. Vercel has recently started offering services that could probably run a test user, but it's new ground for them and you assemble it yourself.
Next I looked at a VPS, which is a rented server that's on 24/7 and which you can get for a few dollars a month. I ruled that out because it would functionally require me to become the IT department for that individual server. Given where I'm at right now, I decided that wasn't the best use of my time.
Finally, I looked at GitHub, or really GitHub Actions, which is the part of GitHub that runs scheduled jobs, and is already where my backups run. The biggest advantage of GitHub is that everything the test user needs already comes built in. Every morning, when it's time to run these tests, GitHub hands the job the equivalent of a brand new computer, rented by the minute from a data center. When the job is done, so is the computer, and there's nothing for me to maintain, so unlike a VPS I don't need to become the IT department.
A helpful analogy for how this GitHub process runs is renting a meeting room by the hour. It's clean and ready when you walk in, you do your work, and when you leave it's cleared out for whoever's next.
Managing costs for test users
Tying back to the four jobs in Exhibit 1, judgement is the one that could cost extra money, because it's the part that actually uses AI. Eyes and hands, the loop, and the harness are all just computer time, and computer time is effectively free at my scale, whether it's my laptop or GitHub's rented machine.
Since judgement requires AI, it has to fit within your flat-rate plan or it will land in one of the situations where you pay with metered credits. For these agentic smoke tests with test users, I was planning to run them once a day, at least initially, so I wasn't concerned about usage.
The question then became whether these tests would work better using Claude's intelligence outside an Anthropic product. That's only really relevant if I was planning to offer these smoke tests as a new product myself, and I haven't got there. That meant I could run these tests as part of my flat-rate plan. The judgement part comes from Claude Code, running on the "computer" rented from GitHub Actions, using a token I created that represents my Claude subscription login. That way Anthropic knows it's "me".
How does the test user sign in?
The second decision is how the agent proves who it is. I have shared authentication across all my products, so the good news is that an agent set up with test logins can access all of my products to run these tests.
My authentication setup means that when a user tries to sign in on a fresh computer, they're required to provide their username, their password, and a login code sent to their email. Since the agent runs every smoke test on a freshly rented computer that has never signed in to anything, every run is going to require that full login.
Allowing the agents to get hold of that login code sent to an email address meant giving them access to my personal email or setting up test accounts. Since I didn't want the agents to have access to my personal inbox (for a variety of reasons), I had to set up test or “throwaway” email addresses. The agent reads its own login code directly out of its own inbox.
How does the test user report its results?
The third decision is how the agent reports what it finds. The agent acts as an artificial user by signing in, taking screenshots, and writing up a report. If the agent doesn't send that information somewhere before the end of the job, the computer gets wiped when the job finishes and everything disappears. The question becomes where and how is it sent?
The findings come in two forms: text and files, such as screenshots. The text is sent to a database, the same database where my backup jobs already report. The files have to go somewhere else. Supabase also offers storage, so the files go into storage on my Supabase account, with each database row carrying links to its images.
Supabase won't just accept text and files from a stranger, though: the agent has to be authenticated, which means it needs permission. The agent gets this permission in the form of a key, which gives it access to both the database and the storage. In this case, because I'm using GitHub Actions, I'll store this key in GitHub's vault. Each morning the agent picks the key up from GitHub's vault and shows it to Supabase with every row it writes and every image it uploads. Each time, Supabase checks the key and opens the door to the database and the storage.
After that, any time I open the website that displays these smoke test results, the JavaScript (to learn more about how it works, read AI Essentials #2) asks Supabase directly for the latest rows and draws them in my browser. If everything went well, there's nothing for me to do.
Adding a code checklist
A random connection surfaced a gap in my original "insurance" from Part 1 and the original version of this post in the Learning with AI series.
Anthropic made some updates to Claude and so I had to change how I updated my Skills, which I wrote about in AI Essentials #3. I introduced a "start hook", which is some code that runs every time I start a new Claude Code session. Not long after that, I saw a post on X which listed some of the top tools other builders use and I noticed I didn't have Vitest.
Building off that connection, I created a new set of code checklists, which run as a "code hook". This code hook runs the checklist, called unit tests, before code ships. It's a similar analogy to a "start hook", in that something happens and then code runs.
I worked with Claude to write down the right answers once for 24 different examples, covering my sign-up links and the timing of my cross-sell email. A robot replays every example against my code, in about a second, every time anything changes.
This code checklist blocks any change that fails, and reports to the same private "insurance" page as the smoke tests and the backups.
On the first day I ran the code checklist, it found an address that was mistakenly listed as a valid sign-up destination on production. The downside of this is that a brand-new user could have been sent to a rehearsal copy of one of my products instead of the real one, and anything they did there would eventually be wiped.
