On May 22nd, I received a message on twitter telling me that my blog was down. I tried to open the website on my phone and saw that he was right. My blog refused to load. I was on my way to work and I had to wait until I get access to my computer to diagnose the problem. When I got to work, it was an extremely busy day so I couldn't go through my logs to figure out exactly what the issue was. So, I restarted my Digital Ocean Droplet and the website came back online. Mission accomplished!
Later, I logged into the server to see what the actual problem was. When I read the logs, I gasped:
[22/May/2019:11:21:03 +0000]... [22/May/2019:17:12:16 +0000]...
Translation: I received a web request at 3:21am pacific times, and the next request was at 9:12am. What this meant was, when my latest article went viral, my server went down for 5 hours and 51 minutes. I remembered those words I once wrote:
Using a CDN, caching, and small optimizations to the code was all that was necessary to have $15.00 a month handle a total of 2.9 million web requests. $5.00 for the server hosting the code and $10.00 for the static file server. I hope to have the opportunity to watch my current setup also fail and face the challenge of optimizing it.
Can't say I didn't ask for it. According to the error logs, the website went down because Apache had spun up too many processes. Meaning, I had too much traffic:
[Wed May 22 11:15:51.735751 2019] [mpm_event:error] [pid 28967:tid 140542191217600] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
This was crazy because in the past I had handled more than a 1,000 requests per second on the same server. The only difference was I upgraded Apache to use HTTP 2, and used Php-fpm (need to investigate that). So I apologize for the down time. Some of you couldn't read my ramblings for almost 6 hours. I finally got around adding a health check on the blog. A minute by minute notification system that let's me know if the server is down.
On June 11th, I received another message on twitter. This one also said that my blog was down. Only, when I accessed the website on my phone, it loaded just fine. My health check had not notified me, so I checked the logs to see that everything was fine. I replied to the user, telling him that the website is back up.
A few minutes later, I received an email. The subject line was
In Russia. I clicked open the email and the content said:
website don't work.
I went to proxynova.com to get a free proxy server from Russia. Just like the email said, my website failed to load in Russia. I pinged my blog from multiple Russian servers and just as expected, every single one of them failed.
Why was my website failing? Did I say something to offend the good people of Russia? Or did they finally figure that I conspired against the members of Massolit? Did they catch me lurking around apartment number 50 on 302B Sadovaya street? Yes, I am a member of that famous band of outlaws...
Shortly after, I received another message from my new Twitter friend. He sent me a link that answered all my questions.
It appears, my IP address is blocked. That decision, 27-31-2018/Ид2971, was made on April 2018 by Генпрокуратура (Prosecutor General's Office). But why would they block my IP address? I'm not doing any mischief, I'm not bothering anyone, I'm just fixing the Primus. So I took a deeper dive at 27-31-2018/Ид2971.
It lead me to this article on habr.ru. The gist of it is that Telegram, the secure messaging app, is blocked in Russia. The Russian government did so by barring the entire
206.189.x.x ip block. Unfortunately, this includes ip blocks used by Digital Ocean, where this beloved blog is hosted. Turns out, it is not my witty comments that got me barred from Russia. They don't even know I exist.
The article goes on to say that the Prosecutor General's Office has gone beyond its authority by blocking 10 million ip addresses that are so called Telegram mirrors:
First, 10+ million addresses do not fit the definition of "mirrors" simply because most of them have no trace of Telegram. It is like bombing Voronezh, because somewhere in the city a terrorist is hiding.
My Russian traffic.
If you use Digital Ocean or AWS (yes, they are affected too), and want to reach the good people of Russia, make sure to test your ip address on Is it blocked in Russia. For now, this blog is not available in Russia. The solution I am considering is to create a new DO droplet, and hopefully they'll assign me a non-blocked ip address. Another solution was pointed out in the article's comments. Someone shared a link to make an appeal for the government to unblock your website.
The writer of these thruthful lines, would have to work on his Russian to make a compelling appeal.