Categories
caching configuration microcaching nginx SSI module web-server

SSI module : Date wise caching in Nginx

Hi everyone, today we are going to see how to implement date wise caching in Nginx using SSI module. There happen to be time when we want to refresh the cache as soon as new date comes. That is, we need such caching when we have our content as static but only for a specific date and it changes with new date.

For the date-wise caching, it doesn’t need actually a lot of work. You just need to change the fastcgi_cache_key and append date as well. For the date, we can use $date_local variable in nginx. But the problem comes as $date_local contains time as well, it is in non readable format, just numbers, (we can’t rewrite it).

SSI module comes for the rescue here. SSI module is a great implementation in nginx which processes SSI(Server Side Include) commands in the response to be sent. We can format the $date_local variable using this, to a readble form and even modify it for just the date.

Then we can use $date_local variable and append it with cache_key. Please follow below to see some codes of how to achieve it.

server {

ssi on;

location ~ \.php$ {

fastcgi_cache_key $your_key|$date_local;

}

}

Now, in somewhere in your view(response to be sent) html tag include below ssi command,

less than html greater than

less than!–# config timefmt=”%A, %d-%b-%Y” –greater than

less than/html greater than

Now, above command will be executed by Nginx before it is sent to client. It changes the $date_local default format to the desired format. And so your cache_key will be generated date-wise. As soon as, new date comes a new cache will be generated, and for the same date, the response from the cache will be served.

So now with the help of SSI module of Nginx, date wise caching can be easily done.
Read my article here for how to install nginx from source, which is needed to install third party module(SSI module) for Nginx.

Categories
configuration install modules nginx source ubuntu

Install Nginx from Source in Ubuntu

Hello Everyone, the very handy command to ubuntu users installing nginx, sudo apt-get install nginx, that’s it. It does install nginx on your machine, with almost all the modules usually needed by the users, super awesome. The shortcoming of this method of installing nginx is that if you want to add more modules to the nginx, you just can’t do it. Currently, nginx installed from apt-get do not support adding custom modules to it. Now when you look over internet about how to add module in nginx, it suggests you to install nginx from source. Some times it gets so much confusing, there is something, which always holds developer back to install it this way(As they already had an easy way).

So here I will show you that installing nginx from source is as easy as the apt-get(almost) and interesting as well. Lets do it,

1. Find the stable version of Nginx (http://nginx.org/download/nginx-1.10.2.tar.gz, current stable version) compatible with your custom module you need to add(if the module doesn’t support all versions of nginx) from its download page. (tar file used here)

2. open terminal

3. sudo apt-get install build-essential libpcre3 libpcre3-dev libssl-dev (install the tools required to compile nginx from source)

4. sudo su (get the super user privilege).

5. mkdir nginx (create a directory named nginx{not necessarily}, all the work will be done into this directory)

6. cd nginx (get into nginx directory)

7. wget http://nginx.org/download/nginx-1.10.2.tar.gz (download your compatible version of nginx)

8. tar -xzvf nginx-1.10.2.tar.gz(extract the downloaded content)

9. cd nginx-1.10.2 (get into nginx directory)

10. ./configure (run the configure file)

11. make & make install (google to know about them at your will, I got to work for that or just run them)

12. sudo /usr/local/nginx/sbin/nginx -s stop (stop the nginx server if already running)

13. sudo /usr/local/nginx/sbin/nginx (start the nginx server)

14. enter the ip of your server, and you must be able to see the default nginx page.

So, that’s it. You have now installed your nginx from source.
Now, the commands for starting and stopping and the locations of the config files are different than /etc/nginx. The nginx files are kept in /usr/local/nginx.

Configuration file: /usr/local/nginx/conf/nginx.conf
Start server : /usr/local/nginx/sbin/nginx
Stop server: /usr/local/nginx/sbin/nginx -s stop
Reload server : /usr/local/nginx/sbin/nginx -s reload
Other commands you can find by tabbing after -s or reading the /usr/local/nginx/sbin/nginx file.

Now you can add any module to your nginx, by compiling nginx with the module. Go ahead and learn and test different great available modules of nginx. One with which I’m currently fascinated with is rtmp module.

Note.
1. Each time you are adding a module, make sure you add all other modules(which you earlier had compiled with and still need them) as well.

2. Each time after the module is added, restart the nginx i.e., stop and start, rather than reload.

Categories
access-log aws bucket cron logging logrotate monitoring nginx s3 s3cmd web-server

Logrotate : Switch on your access log

This is about hot to configure logrotate tool, for your access/error log.
A log is a record of some kind of activity which is usually maintained to track or analyze that activity. An access log is a record of the activities on your server. The activities are like what requests are made to server, how does the server response to some particular server, who accesses the server, etc. It is a good habit and very important to store the access logs. Despite, why do we keep it off (in nginx, access_log off)? It is because on a production server, there is heavy usage of your server and in minutes or so the access log will consume the whole disk space of your server leading to catastrophic consequences.

So it is a bad idea to keep your access log on your machine. There is an amazon service s3( simple storage service) which is quite cheap for such thing. It is used to store data and just store, but no processing. So yeah we can use that to store access log.

This article is about how to configure logrotate to rotate logs on your server to s3. We will use logrotate tool of ubuntu. Logrotate is built mainly for this purpose to rotate large log files periodically. We also use s3cmd tool to sync file from server to s3.

Enough of chit-chat, lets see some real code. The technologies involved are ubuntu 14.04, nginx(some stable version, don’t remember and not in mood to check that out, s3, s3cmd).

open up your terminal.

nano /etc/logrotate.d/nginx;

change the file according to following code.

/var/log/nginx/access.log {

copytruncate

size 100k

missingok

rotate 10

compress

delaycompress

notifempty

create 0640 www-data adm

sharedscripts

dateext

dateformat -%Y-%m-%d-%s

postrotate

INSTANCE=`curl –silent http://instance-data/latest/meta-data/local-ipv4`

/usr/bin/s3cmd sync /var/log/nginx/access.*.gz s3://s3_bucket_name/${INSTANCE}/

endscript

}

Explanation : I will focus on the main directives.

/var/log/nginx/access.log – specify the log file to be rotated.

copytruncate – used in order to avoid reloading nginx after each rotate

size 100k – logs are rotated when the size of the logs exceeds 100k

dateext – append date at the end of the file name in s3

dateformat – specify the date format

you can skip first line, which is used to get the ip of the instance.

/usr/bin/s3cmd – used to sync the access log from their zipped file to the s3 bucket.

Now you can rotate your log file with the following command,

logrotate /etc/logrotate.d/nginx;

We can automate the log rotation scheduling cron for some fixed interval.

open up terminal.

nano /etc/cron.d/logrotate

paste,

*/1 * * * * root /usr/sbin/logrotate /etc/logrotate.conf

save and then exit from the machine and relogin for session to refresh.

Now you have a cron scheduled that runs every minute and rotate the logs if exceeding 100k size.

In cron, you cannot make schedule for seconds, for that you can write a bash script, which is very easy.

So, yeah, by now you must be able to configure logrotate to rotate your access log every minute to s3.

Ok Bye.

Note : don’t place any space in between INSTANCE=’command’ line otherwise it won’t run.

Ask me if any queries. ­čÖé

 

 

Categories
bypass caching configuration fastcgi full-page-caching mastering-nginx microcaching nginx performance static upstream

Microcaching : How to do caching in nginx web server?

Caching is a technique to speed up the response of your website’s static content, content which does not change with time. Microcaching is a type of caching which has short expiry time of cache. This article is about how to set up microcaching in nginx web server.

If you have a website and it contain webpages which do not change, then you are at the right place. Yes you can improve your website’s performance by many folds.

How caching improves performance?

Whenever some request on your website is made, it first goes to your Nginx web server (reverse proxy server). Then it is forwarded to some upstream server to run php-code. But if you have some static content, calling upstream servers is just an overhead. You would always want to keep your upstream servers as free as possible because they are already slow. What can be done is unnecessary or repeated request are served by Nginx itself(this guy is damn fast), relieving upstream servers. This is where caching shows up. Whenever there is some static content request, they are cached by Nginx. On further request of same data they are served by Nginx from its cache.

Nginx does full page caching i.e., it caches the data in its html form. The cache data might be encoded depending on the response from Upstream servers(first time response).

How is Caching done?

We will see a simple configuration for caching using Nginx. It might be made more efficient on further deep study of the same.

First we need to define a caching region, in http context of your Nginx configuration,

fastcgi_cache_path /var/nginx/cache levels=1:2 keys_zone=MYCACHE:10m;

Then, we got to use this cache region, [following should be included in proper context, possibly server context]

fastcgi_cache MYCACHE;

Define the cache key,

fastcgi_cache_key $server_name|$request_uri;

You can also create conditions to bypass and to avoid the caching using following parameters,

fastcgi_cache_bypass┬áByPass; [Bypass is “1” to bypass]

fastcgi_cache_nocache┬áNoCache; [NoCache is “1” to avoid cache]

This is it, this much should do it. You are set for microcaching of static content of your website on your nginx web server.

Have a rocking Deepawali with Caching !

Do contact me in case of any query.