Zipbombs are not as effective as they used to be

Last year, I wrote about my server setup and how I use zipbombs to mitigate attacks from rogue bots. It was an effective method that help my blog survive for 10 years. I usually hesitate to write these types of articles, especially since it means revealing the inner workings of my own servers. This blog runs on a basic DigitalOcean droplet, a modest setup that can handle the usual traffic spike without breaking a sweat. But lately, things have started to change. My zipbomb strategy doesn't seem to be as effective as it used to be.

TLDR; What I learned... and won't tell you

Here is the code I shared last year:

if (ipIsBlackListed() || isMalicious()) {
    header("Content-Encoding: gzip");
    header("Content-Length: ". filesize(ZIP_BOMB_FILE_10G)); // 10 MB
    readfile(ZIP_BOMB_FILE_10G);
    exit;
}

I deliberately didn't reveal what a function like isMalicious() does in the background. But that wasn't really the secret sauce bots needed to know to avoid my trap. In fact, I mentioned it casually:

One more thing, a zip bomb is not foolproof. It can be easily detected and circumvented. You could partially read the content after all. But for unsophisticated bots that are blindly crawling the web disrupting servers, this is a good enough tool for protecting your server.

One way to test whether my zipbomb was working was to place an abusive IP address in my blacklist and serve it a bomb. Those bots would typically access hundreds of URLs per second. But the moment they hit my trap, all requests from that IP would cease immediately. They don't wave a white flag or signal that they'll stop the abuse. They simply disappear on my end, and I imagine they crash on theirs.

For a lean server like mine, serving 10 MB per request at a rate of a couple per second is manageable. But serving 10 MB per request at a rate of hundreds per second takes a serious toll. Serving large static files had already been a pain through Apache2, which is why I moved static files to a separate nginx server to reduce the load. Now, bots that ingest my bombs, detect them, and continue requesting without ever crashing, have turned my defense into a double-edged sword.

Whenever there's an attack, my server becomes unresponsive, requests are dropped, and my monthly bandwidth gets eaten up. Worst of all, I'm left with a database full of spam. Thousands of fake emails in my newsletter and an overwhelmed comment section. After combing through the logs, I found a pattern and fixed the issue.

AI-driven bots, or simply bots that do more than scrape or spam, are far more sophisticated than their dumber counterparts. When a request fails, they keep trying. And in doing so, I serve multiple zipbombs, and end up effectively DDoS-ing my own server.

Looking at my web server settings: I run 2 instances of Apache, each with a minimum of 25 workers and a maximum of 75. Each worker consumes around 2 MB for a regular request, so I can technically handle 150 concurrent requests before the next one is queued. That's 300 MB of memory on my 1 GB RAM server, which should be plenty.

The problem is that Apache is not efficient at serving large files, especially when they pass through a PHP instance. Instead of consuming just 2 MB per worker, serving a 10 MB zipbomb pushes usage to around 1.5 GB of RAM to handle those requests. In the worst case, this sends the server into a panic and triggers an automatic restart. Meaning that during a bot swarm, my server becomes completely unresponsive.

And yet, here I am complaining, while you're reading this without experiencing any hiccups. So what did I do?

For one, I turned off the zipbomb defense entirely. As for spam, I've found another way to deal with it. I still get the occasional hit when individuals try to game my system manually, but for my broader defense mechanism, I'm keeping my mouth shut. I've learned my lesson.

I've spent countless evenings reading through spam and bot patterns to arrive at a solution. I wish I could share it, but I don't want to go back to the drawing board. Until the world collectively arrives at a reliable way to handle LLM-driven bots, my secret stays with me.

---