Creating your own CDN with Nginx

By
Published ~ 5 minutes read

If you've used WordPress, then you know that it has a nice little folder called upload where all the pictures and videos you upload get stored. This is all fine and dandy, until people start accessing those assets.

I learned it the hard way when one of my blog posts became popular. I had a total of 17 files being downloaded after each requests. This is no big deal with the everyday traffic I was getting. (my blog was only a few months old). Suddenly, I started receiving traffic and my server went Kaput.

One thing I learned is that downloading assets shouldn't hinder your server. In fact, your server shouldn't even feel it. That's why most big websites use CDNs (content delivery networks). The advantage is that those networks serve your assets without using any processing power from your main server.

But, not every website is big and using an amazon instance is most of the time overkill for a blog or small website. That's why I decided to roll my own.

The reason for a CDN is to make sure your content is delivered as fast as possible. Some have locations all around the world and serve content from the machine closest to your user. Which is good if you are a Google or facebook, but for your blog a simple server will do the work.

I spun a new server on Digitalocean and set it up as my CDN. Most of my traffic is from the US according to Google Analytics, so I chose a server located in New York. My visitors from overseas are not having a hard time accessing me either, it is just as fast.

users location

Daily traffic location

To tell you the truth, a CDN is just a web server and there is no magic to it. All I did was install Nginx on it.

Nginx is a webserver, just like Apache. The one important difference is that Nginx is very good with serving static content. I don't need to have this machine do any processing, all I care is that it serves content. After receiving thousands of requests, my Apache web server simply died.

The motto is Nginx can serve 10,000 requests per second. I never had to serve that much content but it worked very well for me during spikes. So I set up nginx, moved all my assets to the server, and updated the DNS record. Now all my static files are served from cdn.idiallo.com. Very easy.

This works for me because I use a very primitive process for uploading my files, at least I used to (scp). If you are using a WordPress installation it will be complicated to customize the code to upload the files to a different server instead. But there is a simple solution.

Shared folders. If you are hosting with shared hosting providers you are out of luck. But for others, you can make your upload folder and shared folder, accessible from your CDN machine. This way you upload in one machine and it automatically reflects on the other.

Once your CDN is setup properly, you never have to touch it. All it does is serve files and update when you make new files available.

Installing Nginx

If you are using a debian machine all you need is this command:

sudo apt-get install nginx

Choose a folder where you want your asset folder to be: (/var/www/assets/). Then you can point Nginx to it. Create an Nginx config file:

vim /etc/nginx/sites-available/yourwebsite.com

And add these settings:

server {
  listen 80;
  server_name cdn.yourwebsite.com;

  location / {
    expires 90d;
    root /var/www/assets/;
  }
}

Also make sure you create a symlink in the sites-enabled folder.

That's all there is to it. You can restart your server and start serving files from your own custom CDN now. You can update your DNS to use your new CDN. subdomain, which should be available in your hosting provider GUI.

Enjoy serving fast content without breaking your server.

Update:

If you want to upload files on your webserver but want to make them available on your CDN server, here is how you can share files between your webserver and CDN

On a related note, here are some interesting articles.

Using MySQLi and PDO instead of MySQL_*

Using MySQLi and PDO instead of MySQL_*

It is time to deal with mysql_* functions once and for all. These methods are deprecated and slow. The time to upgrade has long passed yet we still see it everywhere. Since I cannot force every author to update their tutorial and blogs, I decided to write a post to hopefully rank better and provide the essential information to help new comers.

Fixing a 3 year old hack

Fixing a 3 year old hack

One of the sites I worked on was regularly getting hacked. The attacker did not leave much traces and eventually we learned how to deal with the symptoms of the attack rather than fix it. The common agreement was that maybe he was exploiting a bug in the framework. Recently, I did a deployment and didn't follow the exact procedures and we got hacked again. I got mad for getting blamed for it and decided it was time take a deeper look into the problem.

SQL injection and how to protect your website

SQL injection and how to protect your website

Experienced developers are expensive. In a world where cutting cost seems like the best option, companies try to maximize their profit by spending less and less on good talent. It is much cheaper to hire someone who just learned php a few weeks ago then a seasoned developer. But it becomes a very bad investment when the newbie introduces insecure code. The problem is, a lot of things learned from those LAMP CRUD application tutorial do not focus much on security. When this code is introduced to a commercial application, the damage can be very expensive. I like how stackoverflow users are fighting very hard to eradicate SQL injection, I am doing my part too but it seems like it is much easier to find insecure code on-line. That said, I will attempt to scare you off your feet so you know better what is the cost of SQL injection.

View all articles

Comments(51)

PulloDimo :

Very Nice article I love it!

ibrahim :

I'm glad it was of use to you @PulloDimo :)

Brian T. :

A single instance setup to serve files (in addition to your OTHER single instance serving your main site) isn't really a CDN. There's no true redundancy or "cloud" aspect to it (other than being a VPS).

Also a huge benefit to a CDN is serving data based on location to the end client (for the lowest latency), while also providing fail safe routes in the event of a failure on one (of the many) CDN servers. Because of this a simple DNS round-robin would also not be considered true CDN (even if the DNS server served records based on the requesting IP's location).

I'd be interested in seeing a post about setting up a true CDN. Or at least in this post explaining anything with a single point failure isn't a CDN (and wouldn't work like say AWS CloudFront or CloudFlare). Since someone new to cloud services could get confused by possible misinformation.

Ibrahim :

@BrianT No true Scotsman would call it a CDN. However I do mention in the post that having AWS or Location based could be overkill for a small website, the alternative would be to use cloudflare since they have a free tier.

But the article serves its purpose of introducing the barebone of CDN. Once the static files are independent of the webserver, it allows for the creation of extra features like redundancy and location based.

Hopefully in a future post I will be able to explain these features.

Kris :

You could easily setup up a couple of these VPS's, and use amazons route 53 DNS services to load balance the traffic, based on server health and the users location. Amazons route 53 is cheap .50 cents per domain.

Ibrahim :

That's a very good idea @kris, it would certainly take care of the geolocation problem and speed up the requests.

Nope :

Hi,

I found this article very interesting. I did a similar experiment with Apache to serve my blog images. As a page could be composed by 10-15 pictures and most browsers limit the number of connection to a domain, I created another domain and it website on a vps to speed images download.

Ibrahim :

@Nope thanks for your comment :)

One of the main advantages of a cdn, whether it runs on Apache nginx or any, is that it takes the load of your main server.

This way the server that handles the logic of your application doesn't also have to serve files.

Amnuay Jitklang :

I try this but my nginx path is usr/local/nginx/ where should i create sites-enabled folder? Thanks

Ibrahim :

Hi Amnuay

You probably don't need the folder. Try to set it up without it and see if you can access the web server through the web browser.

Bellash :

Great post thanks.

Ibrahim :

Glad it was helpful to you @Bellash :)

trent :

How to do support REDIS caching etc?

Ibrahim :

@trent for this, you don't need redis. Nginx already has a caching mechanism and it is pretty good at serving files.

Check out proxy_cache it allows you to cache files beyond the user browser.

Tailor :

Nice post Ibrahim... I was searching for an answer and I read you content. I have a question. I have site for which I want to create CDN, as the server is US and target customer are in India the response time is poor. My question is if I create a CDN server locally which is hosted at home like environment and suppose if the hosted CDN network is down. Is there a mechanism to fall back to the original server ? so there is no down time is zero. If there is a alternative to this down time I can try Nginx or something similar. I appreciate all your thoughts on this. Thanks a lot.

Ibrahim :

Hi Tailor

I use digital ocean (my referral link), and I believe they have a data center in India.

What you can do is use nginx as a load balancer. You can use a couple more servers that use a shared drive to serve your files and use a round robin with fail safe directive:

Round robin

# main nginx server
upstream myapp1 {
    max_fails = 5; # number of times the server fails before we take it out
    # multiple servers for redundancy.
    server srv1.example.com;
    server srv2.example.com;
    server srv3.example.com;
}

You can control the number of times the smaller servers have to fail before you take it out of the equation, and use the rest.

I hope this answers your question.

Tailor :

Wow!!! thanks Ibrahim... this is what I was looking for. Thanks a ton :) I will have to work on this before implementing on the live server. Yes it answers my question. I appreciate your time.

olivedev :

I never thought that creating your own CDN is this easy. Still, why someone would want to create their own CDN instead of using popular CDNs? I am using Cloudflare because it can be easily enabled from the Cloudways platform I am using. I am satisfied with the result it has given me so far.

Ibrahim :

Hi @Olivedev

Using your own CDN gives you a control that is not available with any third-party platform. Sometimes is a simple as wanting privacy for your users.

If you are using a jquery library from the cdn, it makes it faster for your users to load, at the expense of routing their request to a third-party that can map their path around the web. Maybe you trust your CDN, that's fine. But having your own gives you an option if you don't trust them.

There was a very interesting article in the subject a few days ago, and this one is better at explaining as I am.

Another obvious reason to create your own cdn is, well because you can :)

Barbie :

thanks Ibrahim... this is what I was looking for.

Chinmay Rajyaguru :

Respected Mr.Ibrahim,

First, thank you for this wonderful article.

I have an question related CDN server configuration. For ex., My website getting each day approx. 5,000 visitor traffic. I'm using NGINX (LEMP) & WordPress.

What should be my CDN server configuration (RAM, Storage, Bandwidth)?

Thank you!

muhammad ayaz :

hello i have blog which i make on blogger i want to add on cdn server but how please help me

Ibrahim sir... my blog address is www.b4ufreemovies.tk

Ibrahim :

Hi Chinmay,

I'm glad this article has been useful to you.

For your configuration, what I showed in the article is all you need with NGINX. 

server {
  listen 80;
  server_name cdn.yourwebsite.com;

  location / {
    expires 90d;
    root /var/www/assets/;
  }
}

Because you use wordpress, and wordpress uses it's own file upload mechanism, I suggest you turn /uploads into a shared folder, like recommended in this article.

I used the $10 a month plan for this blog, for the CDN, but you can go over the next plan $20 and you get enough ram you will need to handle traffic.

Digital Ocean referral link

I hope this works for you :)

Chinmay Rajyaguru :

Respected Mr.Ibrahim,

Thank you very much for helping/guiding me. I'm using DigitalOcean over 2 years.

1 year back, for CDN solution I had setup CloudFlare. If I will create my own CDN server then should I quit from CloudFlare?

That's a big question in my mind!

Thank you!

Ibrahim :

I think you can run both your CDN and cloudflare at the same time and see if it works for you first.

Remember, cloudflare has the advantage of serving the closest content to your users. You can try:

cdn1.website.com for cloudflare cdn2.website.com for your own

and have them each handle part of the assets. Then based on the performance you can decide if you stay with one or the other.

Chinmay Rajyaguru :

Respected Mr.Ibrahim, Thank you very much for your reply. I asked the same question directly to CloudFlare support team.

They replied: You can locate your static content anywhere you like but it's simplest to keep it on the same server as your main website and have Cloudflare cache it.

Thank you!

Kelvin :

how do i create symlink to sites-enabled folder?

Ibrahim :

Hi @Kelvin you use the ln command on linux, note that you will need to have sudo privileges to do so. 

ln -s /etc/nginx/sites-available/mysite.conf /etc/nginx/sites-enabled/mysite.conf

If you are using apache, there is the a2ensite <siteconfname> command. Check the documentation to see how to properly use it.

I hope this helps.

diamond sharma :

Great post thanks. would try it for my self serving cdn

Thanks man :

Hey, first of all its a really great idea to use it with google cloud dns or amazon route 53 to detect nearest server to the user, and its cheap mostly 40 cent for million request. But how can we use the same cdn nginx server for multiple domains/projects. If we Can host multi domains would be great.

The reason we build own cdn is price, if you have a global project with lots of images, 1 million pageviews for month(2mb images per page) costs you 2tb starts... Where 1tb starts mostly from 40 dollars to 250 dollars(differs location) simplest digital ocean price is 5 dollar and it has 1tb bandwith i think. You could buy 3-4 locations from different countries for ten dollars and costs you 40 dollar and you have 8tb transfer :) amazon cdn could cost for 8tb up to 2000 dollars(asia location price)

Ibrahim :

Here is what I did for a project. I got several digital ocean droplets from different data centers and did a quick and dirty ip to country mapping from MaxMind GeoIP to assign them to a cdn. (You can do this directly with Apache)

Example: 

// United States
http://us1.cdn.example.com

// India 
http://asia1.cdn.example.com

// Europe
http://eu1.cdn.examplr.com

// Etc.

Now my application can pre-select the appropriate CDN for the user. It's not as robust as cloudflare but it's always nice to be able to do the job yourself.

Thanks man :

Thanks good to see usecases...

Fbriceno :

Thanks for the explanation, too useful. I would like to add something. I needed to acces to this url:

server:port/cdn/site/img/some-img.png

But the root directory is alocated in /var/www/assets/ then I got an error 404, to fix this I replaced root by alias in the location block.

location /cdn/ {
    expires 1d;
    alias /var/www/assets/;
}

Ibrahim :

Thanks for the insight @fbriceno. I'll try to add it to the article.

Janusz :

Very helpful, thanks.

The only question why haven't you bought cookie-less domain?

When you request a file like https://cdn.idiallo.com/images/logo.png, you are sending all unneeded cookies.

Anyway, great work!

Ibrahim :

Thanks @Janusz, this is an oversight from me. I'll remove cookies. It's pretty simple with nginx:

location / {
   fastcgi_hide_header "Set-Cookie";
   expires 90d;
   root /var/www/assets/;
}

Thanks

Alpesh :

Thanks. Helped me. :)

Tom :

Ibrahim, IT IS NOT A CDN!

Just a asset separation from dynamic content. It would be very helpful for you and the community if you study before post. Otherwise you are just disseminating ignorance.

Ibrahim :

Hi @Tom, this is the first step in creating a CDN.

For another project, I have 6 servers around the world. Using ipinfo.io, I get the user's country and serve from the closet server. It is a content delivery network.

Rafal :

Very helpful :) Was postponing making my own cdn but now feel ready to act :)

Ibrahim :

Glad you found it helpful @Rafal. Good luck

Kyuroku :

Well, i still can't understand how it's works :(

Jhoon :

how about SEO perspective about this? AND also i need to know about CORS and other features that "real" cdn services give, it's same or what ? if not, what the difference between both of em.

Meanwhile, i have digital ocean instances, and i want to build my CDN from vultr, what should I do?

Ibrahim :

Jhoon, the SEO benefit of this is that it speeds up your page. Google favors faster website.

Hamed :

Hi, Thanks for your useful post.I have a question, Can I use a cdn server for two websites? How to do it?

Ibrahim :

Hi @Hamed. I'm glad it is useful to you.

To answer your question, Yes. You can use the same CDN for as many servers you need. It becomes a reason to improve the CDN to support more requests.

aaronmacro :

Its very Useful.. You can get a Cheap CDN services from Beluga CDN.

Cleo :

Thank you for this. Very helpful indeed.

I have a question that might seem a little ignorant, so I apologize in advance. Currently I have a digital ocean server setup with NGINX for my website. If I wanted to create my own CDN server as you have ( on a separate server ), would I need to have NGINX on both servers, or would I just have NGINX on the CDN server and use that to route traffic for both?

I read your other article that details the shared folder setup. Do you have NGINX setup on both servers in this setup? (One to handle the main domain and one to handle the cdn.subdomain ). Thank you for clarifying.

Ibrahima Diallo :

Hi Cleo, it's a very good question.

In my particular case, I use Apache on the main domain, and Nginx on the cdn server. However, it's ok to have Nginx on both servers.

The main advantage of Nginx is that it is extremely fast when it comes to static content. It's also important to have two distinct servers so that they don't have to share CPU resources. The point is that if you are getting a lot of traffic serving large assets, your main server shouldn't suffer from it.

To answer your second question, I use Nginx on the server with shared folder, and Apache on the other. Again the reason is that Nginx is better for static content.

I hope my answer is helpful to you. Thank you for contributing.

Ziaul Kamal :

How to config for vestacp panel ??

استوریج :

that was a great article and good for my website Is that free or licensed product?

Let's hear your thoughts

For my eyes only