Digitalocean and AWS IPs banned in Russia

On May 22nd, I received a message on twitter telling me that my blog was down. I tried to open the website on my phone and saw that it was in fact down. I was on my way to work and I had to wait until I get access to my computer to diagnose the problem. When I got to work, it was an extremely busy day, I couldn't go through my logs to figure out what the issue was. So, I restarted my Digital Ocean Droplet and the website came back online. Mission accomplished!

Later, I logged into the server to see what the actual problem was. When I read the logs, I gasped:

[22/May/2019:11:21:03 +0000]...
[22/May/2019:17:12:16 +0000]...

Translation: I received a web request at 11:21am UTC, and the next request was at 5:12pm. What this meant was, the night my latest article went viral, my server went down for 5 hours and 51 minutes. I remembered those words I once wrote:

Using a CDN, caching, and small optimizations to the code was all that was necessary to have $15.00 a month handle a total of 2.9 million web requests. [...] I hope to have the opportunity to watch my current setup also fail and face the challenge of optimizing it.

Can't say I didn't ask for it. According to the error logs, the website went down because Apache had spun up too many processes. Meaning, I had too much traffic:

[Wed May 22 11:15:51.735751 2019] [mpm_event:error] [pid 28967:tid 140542191217600] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

This was crazy because in the past I had handled more than a 1,000 requests per second on the same server. The only difference was I upgraded Apache to use HTTP 2, and used Php-fpm (will need to investigate that). So I apologize for the down time. Some of you couldn't read my ramblings for almost 6 hours. I finally got around adding a health check on the blog. A minute by minute notification system that let's me know if the server is down.

On June 11th, I received another message on twitter. This one also said that my blog was down. Only, when I accessed the website on my phone, it loaded just fine. My health check had not notified me, so I checked the logs to see that there was nothing unusual. I replied to the user, telling him that the website was back up.

A few minutes later, I received an email from a second person. The subject line was In Russia. I clicked open the email and the content said: website don't work.

I used one of the many free proxy server from Russia to investigate. Just like the email had said, my website failed to load in Russia. I pinged the blog from multiple Russian servers and just as expected, every single one of them failed.

Why was my website failing? Did I say something to offend the good people of Russia? Or did they finally figure that I conspired against the members of Massolit? Did they catch me lurking around apartment number 50 on 302B Sadovaya street? Yes, I am a member of that famous band of outlaws...

Shortly after, I received another message from my new Twitter friend. He sent me a link that answered all my questions.

It appears, my IP address is blocked. That decision, 27-31-2018/Ид2971, was made on April 2018 by Генпрокуратура (Prosecutor General's Office). But why would they block my IP address? I'm not doing any mischief, I'm not bothering anyone, I'm just fixing the Primus. So I took a deeper dive at 27-31-2018/Ид2971.

It lead me to this article on habr.ru. The gist of it is that Telegram, the secure messaging app, is blocked in Russia. The Russian government did so by barring the entire 206.189.x.x ip block. Unfortunately, this includes ip blocks used by Digital Ocean, where this beloved blog is hosted. Turns out, it is not my witty comments that got me barred from Russia. They don't even know I exist.

The article goes on to say that the Prosecutor General's Office has gone beyond its authority by blocking 10 million ip addresses that are so called Telegram mirrors:

First, 10+ million addresses do not fit the definition of "mirrors" simply because most of them have no trace of Telegram. It is like bombing Voronezh, because somewhere in the city a terrorist is hiding.

If you use Digital Ocean or AWS (yes, they are affected too), and want to reach the good people of Russia, make sure to test your ip address on Is it blocked in Russia. For now, this blog is not available in Russia. The solution I am considering is to create a new DO droplet, and hopefully they'll assign me a non-blocked ip address. Another solution was pointed out in the article's comments. Someone shared a link to make an appeal for the government to unblock your website.

The writer of these truthful lines, would have to work on his Russian to make a compelling appeal.

---