AI agents aha moment

Looking at the reactions to the Debian AI vote, I think some people still think the clock can be turned back, as if that ever worked in history. Rather than cry about spilled milk, I prefer to find a path forward in the new world. There are many ways to use LLMs, some of them are straightforward, others not so much.

One of the “not so clear” areas for me is the focus on agentic workloads. For complex tasks, sure, you want something that can work in the background, but in general, why does every single tool go the agentic way? I much prefer the “chat/ask” approach, or even the “code” one, but if I’m at the keyboard, why would I send a task to an agent, and see it work, instead of directly implementing it?

And then, this past Friday, I finally understood one part of that. I was in the airport, sitting at the gate and waiting to board a flight, and because I arrived much earlier at the airport (fearing crowds due to Labour Day weekend), I got one hour of work before boarding started. As the time for boarding approached, I did one more commit after making sure tests pass, pushed, closed laptop, and went to walk a bit before getting on the plane.

As I was getting up, I get a phone notification from GitHub that the CI run failed. I was quite surprised, as the local tests passed, so I open the notification, and realize that tests via make test vs CI (which additionally uses --pedantic) had slightly different settings, and of course I missed a build warning (which in CI is an error).

I thought I’d fix that on the plane, but then I saw a “Copilot agent” button in the mobile app. I was curious what it did, I click it, and I see Copilot starting a draft pull request, and saying:

Thanks for asking me to work on this. I will get started on it and keep this PR’s description up to date as I form a plan and make progress.

Fix the failing GitHub Actions job. Analyze the Actions logs, identify the root cause of the failure, and implement a fix.

Then it goes, finds the failure, writes the fix, and tries to run the tests. Well, it can’t do it (it runs in a restricted container, so no network, so stack install couldn’t actually work). The agent sees that, acknowledges it has no way to validate the fix, but the error message was clear enough that it was confident the fix is mostly correct, so it sends the pull request.

I allow full CI to run on the pull request, and go buy a bottle of water. After that, I check and see that the CI failed again, as not one but two test files were broken, and I didn’t have --keep-going, so the build stopped at the first failure. I write a comment in the pull request, no reaction, I realize I need to tag Copilot explicitly, I do that, and it starts another investigation.

I’m waiting now in the boarding queue, with phone in hand, while Copilot is fixing my bug. While I scan my boarding pass and walk towards the plane, the pull request is updated, I trigger another CI, it passes, and I merge it.

And then, it hit me. Agents allow me to make progress while being “not at keyboard”, whether that’s physically “not at keyboard”, or while working on something else. Fixing a simple test failure is not something that needs human attention per se, whereas improving the test layout might be.

In that airport, using otherwise-unusable downtime, and without explicitly intending to, I made progress in understanding a different way to use AI. Now I have three ways to work with LLMs: ask (tutor mode), code (implement my request), and agent (fix simple or complex problems, autonomously). I still don’t know about “plan” mode and really complex tasks, like asking it to implement features from scratch. That will probably be the next area to tackle.

And today (Sunday), while waiting for a running race to start, I opened GitHub, and asked Copilot to increase test coverage for a simple module. It did, and yes it still can’t run tests (I learned in the meantime that you can configure the environment in which the agent runs, nice), but after two back-and-forth messages, I have a pull request ready to review. All in the 20 minutes before a race, where I could either browse social media or actually do some meaningful work.

Checking now my GitHub billing, it looks like all of this Copilot use only cost $1.92. Yes, that is under two dollars! And while it did use compute resources, the person across the aisle who watched TikTok or Instagram for half an hour while waiting for takeoff also consumed a lot of compute, and so do the gazillion cat videos uploaded to YouTube every day.

To me, this is another tool in the toolbox, that might one day replace me (as it did to the 19th-century textile workers), or make me five times more productive — we’ll see where we end up. In the meantime, I can move faster, and make better use of my limited free time.

Enjoy the ride!