Logrotate : Switch on your access log

This is about hot to configure logrotate tool, for your access/error log.
A log is a record of some kind of activity which is usually maintained to track or analyze that activity. An access log is a record of the activities on your server. The activities are like what requests are made to server, how does the server response to some particular server, who accesses the server, etc. It is a good habit and very important to store the access logs. Despite, why do we keep it off (in nginx, access_log off)? It is because on a production server, there is heavy usage of your server and in minutes or so the access log will consume the whole disk space of your server leading to catastrophic consequences.

So it is a bad idea to keep your access log on your machine. There is an amazon service s3( simple storage service) which is quite cheap for such thing. It is used to store data and just store, but no processing. So yeah we can use that to store access log.

This article is about how to configure logrotate to rotate logs on your server to s3. We will use logrotate tool of ubuntu. Logrotate is built mainly for this purpose to rotate large log files periodically. We also use s3cmd tool to sync file from server to s3.

Enough of chit-chat, lets see some real code. The technologies involved are ubuntu 14.04, nginx(some stable version, don’t remember and not in mood to check that out, s3, s3cmd).

open up your terminal.

nano /etc/logrotate.d/nginx;

change the file according to following code.

/var/log/nginx/access.log {

copytruncate

size 100k

missingok

rotate 10

compress

delaycompress

notifempty

create 0640 www-data adm

sharedscripts

dateext

dateformat -%Y-%m-%d-%s

postrotate

INSTANCE=`curl –silent http://instance-data/latest/meta-data/local-ipv4`

/usr/bin/s3cmd sync /var/log/nginx/access.*.gz s3://s3_bucket_name/${INSTANCE}/

endscript

}

Explanation : I will focus on the main directives.

/var/log/nginx/access.log – specify the log file to be rotated.

copytruncate – used in order to avoid reloading nginx after each rotate

size 100k – logs are rotated when the size of the logs exceeds 100k

dateext – append date at the end of the file name in s3

dateformat – specify the date format

you can skip first line, which is used to get the ip of the instance.

/usr/bin/s3cmd – used to sync the access log from their zipped file to the s3 bucket.

Now you can rotate your log file with the following command,

logrotate /etc/logrotate.d/nginx;

We can automate the log rotation scheduling cron for some fixed interval.

open up terminal.

nano /etc/cron.d/logrotate

paste,

*/1 * * * * root /usr/sbin/logrotate /etc/logrotate.conf

save and then exit from the machine and relogin for session to refresh.

Now you have a cron scheduled that runs every minute and rotate the logs if exceeding 100k size.

In cron, you cannot make schedule for seconds, for that you can write a bash script, which is very easy.

So, yeah, by now you must be able to configure logrotate to rotate your access log every minute to s3.

Ok Bye.

Note : don’t place any space in between INSTANCE=’command’ line otherwise it won’t run.

Ask me if any queries. 🙂

 

 

Latest Comments

Leave a Reply