The Day Caddy Stood Still

A couple of weeks ago, I went to update my VPS. Usually, I do this once a week, sometimes more frequently depending on any security news I see. I should probably have the AI agents I use do this, but I do it myself for now. I find infrastructure interesting and "boring" in a good way most of the time.

While updating one of my VPSes, I got a message saying I needed to pay for access to Caddy because the repository had reached its bandwidth limit. This was from a company called Cloudsmith. I wasn't the only one getting this message, either. There was already a large discussion going on over at Caddy's GitHub.

This led me down a big rabbit hole that lasted quite a few hours, and I'm still thinking about it. My first thought wasn't about the bandwidth limit itself, but whether this could be a new type of attack. Caddy is open source and free, and it's used by a lot of people, organizations, and companies. We've seen several different supply chain attacks over the years. But what if someone deliberately flooded the repositories of open-source projects and used up their free bandwidth allowances? People wouldn't be able to download security updates, leaving older, bug-ridden versions of the software running and potentially vulnerable to exploitation.

Before anyone says not to give people ideas, I'm sure someone has already thought of this. It's kind of a modern-day denial-of-service attack, just from a different angle.

As it turned out, Caddy's repository had exhausted its 4.9 TB monthly bandwidth allowance. Apparently, much of the traffic came from automated systems repeatedly downloading packages. Some IP addresses had reportedly made hundreds of thousands of requests. That's a lot of bandwidth for a free open-source project to deal with.

Luckily, there was no indication this was a deliberate attack, so my aluminum foil hat could come off. By the end of the weekend, someone at Cloudsmith had fixed the bandwidth limit problem, and everyone could update again.

What I found encouraging was that several other providers contacted the maintainer and offered to help. I'm not sure if any of those offers were accepted, but it was good to see people stepping up to help keep a free project running.

And that's something I appreciate about open source. Projects like Caddy and Traefik are useful to a lot of people, including me, and much of the work behind them is done by people who aren't getting rich from it. Meanwhile, we have hardware manufacturers shifting supply toward their most profitable customers while the rest of us deal with higher prices and fewer choices.

It's quite a contrast. On one side, companies making hardware harder and more expensive for ordinary people to get. On the other, people putting their time and effort into making software that anyone can use, often for little or no rewards and riches.

I'm glad there are still people willing to do that for the greater good of the internet.