Categories
aws web-server

Amazon Web Services : Is Internet Down?

You might be checking out different websites and getting nothing but some or other error. Let me tell you that it is not your internet issue but their website issue.
The big news is Amazon Web Services are facing issues which are leading to such errors. It has lead downtime to many websites like Quora, Slack, reddit, and other big web giants.

Wait! what? So how these things are related?

So Amazon Web Services(AWS) are a pool of cloud services offered by amazon which these websites use to operate themselves. Now when these services are facing issues, so these website do as well. It is not yet clear if there was some internal bug or a hack in happened.

So how bad is it? How long is it going to remain same?

Well, this looks pretty bad as a lot of websites have been affected due to their entire dependency on these services. But it is also true that it is very rare.
It completely depends on the amazon developers on how long is it going to recover their services. But you can check their current status at status.aws.amazon.com.

So what can I do if I am affected by these services?

There is a dirty hack for the users of amazon web services to recover from this issue. From the status page of amazon aws, it can be observed that services of N.Virginia are only affected but not others. So if nothing can be done, then try to re-host your services in some different zone than N.Virginia. This shall solve your issue until aws developers fix theirs.

Dark Internet.

Categories
caching configuration microcaching nginx SSI module web-server

SSI module : Date wise caching in Nginx

Hi everyone, today we are going to see how to implement date wise caching in Nginx using SSI module. There happen to be time when we want to refresh the cache as soon as new date comes. That is, we need such caching when we have our content as static but only for a specific date and it changes with new date.

For the date-wise caching, it doesn’t need actually a lot of work. You just need to change the fastcgi_cache_key and append date as well. For the date, we can use $date_local variable in nginx. But the problem comes as $date_local contains time as well, it is in non readable format, just numbers, (we can’t rewrite it).

SSI module comes for the rescue here. SSI module is a great implementation in nginx which processes SSI(Server Side Include) commands in the response to be sent. We can format the $date_local variable using this, to a readble form and even modify it for just the date.

Then we can use $date_local variable and append it with cache_key. Please follow below to see some codes of how to achieve it.

server {

ssi on;

location ~ \.php$ {

fastcgi_cache_key $your_key|$date_local;

}

}

Now, in somewhere in your view(response to be sent) html tag include below ssi command,

less than html greater than

less than!–# config timefmt=”%A, %d-%b-%Y” –greater than

less than/html greater than

Now, above command will be executed by Nginx before it is sent to client. It changes the $date_local default format to the desired format. And so your cache_key will be generated date-wise. As soon as, new date comes a new cache will be generated, and for the same date, the response from the cache will be served.

So now with the help of SSI module of Nginx, date wise caching can be easily done.
Read my article here for how to install nginx from source, which is needed to install third party module(SSI module) for Nginx.

Categories
analytics chat chatting google google-analytics javascript live live-users long-polling nginx node.js performance persistent persistent-connection php5-fpm Proxy real-time response reverse-proxy socket socket.io socks tcp upstream web-server websocket websocket-handshake

How To Make Your Own Analytics Tool?

An analytics tool is a tool that you can use to get the information about how is your website performance, how are the users using it, what are the number of users at anytime(live users), live users for different urls, and other infos. One such example of the tool is the google analytics.

Analytics tool

Analytics tool gives a lot of information about your website with the nice looking graphs and all. In this blog, we will be discussing how we can make such analytics tool(digging the data).

Question 1: Why should we make such thing if it already exists?

Answer : One reason is that it is not free for always. The other reason is that the process of making it is very sexy and you learn technologies that you can use to make your own other real time apps like chatting app. Now that is more a valid reason.

Question 2: What are we talking about?

Answer : This is an example of a real time application. You see, you are tracking live users’ data. The data can be anything like total number of users anytime or number of users for different urls or different referral or different user agent or different visited ips, or anything. The other example of a real time application is a chatting application where a lot of users are communicating at the same time.

Question 3: What technologies are used for such an application?

Answer : So we need a continuous stream of response from the server about their performance as only servers know what is hitting them. It can only be achieved if there is some persistent connection between client and server so that server can continuously push data to the client. With persistent connection I would mean one for literally persistent connection(web sockets) or else timed persistent connection(long polling) or else chunk persistent connection(polling). By timed persistent connection, I mean persistent connection only for certain time interval after which it is closed and again opened. And by chunk persistent connection, I mean lots of request with no persistency, but the requests are so frequent that altogether they seem persistent(my bad).

Among all, I prefer web sockets, as I can’t logically satisfy myself why to use others and more of it that they are the outdated styles and only existed when sockets were not there.

So in this tutorial we will be using socket.io library with node.js to create web sockets. Below are the other libraries with other language which I found from here.

Analytics tool

And yes, we are going to use Nginx as a reverse proxy for web sockets. A very great tutorial regarding this can be found here. You can skip Nginx if you wish and directly test for your node.js server.

For Nginx configuration, as mentioned in tutorial,

server {
    server_name app.domain.com;
    location / {
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_pass http://socket_nodes;
    }
}

Proper explanation is given in that tutorial for how this will work for sockets. Now we setup our Node server(server.js), a very nice tutorial can be found here.

var data = {};
var http = require(“http”);
var url = require(‘url’);
var fs = require(‘fs’);
var io = require(‘socket.io‘);
var path;
var server = http.createServer(function(request, response){
        console.log(‘Request Accepted’);
        path = url.parse(request.url).pathname;
        switch(path){
            case ‘/’:
                response.writeHead(200, {‘Content-Type’: ‘text/html’});
                response.write(‘hello world’);
                response.end();
                break;
            case ‘/live’:
                fs.readFile(__dirname + ‘/socket_io.html’, function(error, data){
                    if (error){
                        response.writeHead(404);
                        response.write(“opps this doesn’t exist – 404”);
                        response.end();
                    }
                    else{
                        response.writeHead(200, {“Content-Type”: “text/html”});
                        response.write(data, “utf8”);
                        response.end();
                    }
                });
                break;
            default:
                response.writeHead(404);
                response.write(“opps this doesn’t exist – 404  –“+path);
                response.end();
                break;
        }
    });
 server.listen(8001);
 var listener = io.listen(server);
 listener.sockets.on(‘connection’, function(socket){
    console.log(‘Socket Made.’);
    setInterval(function(){
        socket.emit(‘showdata’, {‘data’: data});
    }, 1000);
 socket.on(‘disconnect’, function(){
    console.log(“Socket Closed.”);
 });
});

Load socket.io.js on the client side in your socket_io.html which will be used to create event to create a persistent connection. (Please don’t bang your head for how would it find socket.io.js)

/socket.io/socket.io.js

     var socket = io.connect(); // your initialization code here.
     socket.on('showdata', function(data){
          console.log(data);
     });

Now you can see all the data in your browser’s console. You can put this data at the right place of your analytics html. Now you can easily send continuous data from node server to your client.

Are you still wondering how would we gather data? Suppose we want to track number of live users, for that we will use a key “live” in data variable and initialize with 0. We will increment it as soon as there is a socket made and decrement it as soon as the socket is closed(Make sure io.connect() is called once and only once for a page load). So the data[‘live’] will determine the number of live users.

In server.js,

listener.sockets.on(‘connection’, function(socket){
    console.log(‘Socket Made.’);
    data[‘live’] +=1;
    setInterval(function(){
        socket.emit(‘showdata’, {‘data’: data});
    }, 1000);
 socket.on(‘disconnect’, function(){
 data[‘live’]-=1;
    console.log(“Socket Closed.”);
 });

As simple as that. It is just the logics in node to be done and you can achieve any data. Now for temporary data(by temporary I mean the machine hosting node server is until rebooted), we don’t need any extra storage.

If we want to use some durable data, or store some information, we can choose database of our choice and send data from node to database.

Now we are also using Nginx as a reverse proxy which is helpful in load handling and better scaling. We can extract other information like user ip, upstream responses(php), and other valuable information and send to node.

So yes, it is really interesting to make such tool, I told you so. You are surely going to stuck in somewhere or make it up or for any suggestion or for confusion, we can always discuss.

Great to see you people with your own analytics tool. 🙂

Categories
access-log aws bucket cron logging logrotate monitoring nginx s3 s3cmd web-server

Logrotate : Switch on your access log

This is about hot to configure logrotate tool, for your access/error log.
A log is a record of some kind of activity which is usually maintained to track or analyze that activity. An access log is a record of the activities on your server. The activities are like what requests are made to server, how does the server response to some particular server, who accesses the server, etc. It is a good habit and very important to store the access logs. Despite, why do we keep it off (in nginx, access_log off)? It is because on a production server, there is heavy usage of your server and in minutes or so the access log will consume the whole disk space of your server leading to catastrophic consequences.

So it is a bad idea to keep your access log on your machine. There is an amazon service s3( simple storage service) which is quite cheap for such thing. It is used to store data and just store, but no processing. So yeah we can use that to store access log.

This article is about how to configure logrotate to rotate logs on your server to s3. We will use logrotate tool of ubuntu. Logrotate is built mainly for this purpose to rotate large log files periodically. We also use s3cmd tool to sync file from server to s3.

Enough of chit-chat, lets see some real code. The technologies involved are ubuntu 14.04, nginx(some stable version, don’t remember and not in mood to check that out, s3, s3cmd).

open up your terminal.

nano /etc/logrotate.d/nginx;

change the file according to following code.

/var/log/nginx/access.log {

copytruncate

size 100k

missingok

rotate 10

compress

delaycompress

notifempty

create 0640 www-data adm

sharedscripts

dateext

dateformat -%Y-%m-%d-%s

postrotate

INSTANCE=`curl –silent http://instance-data/latest/meta-data/local-ipv4`

/usr/bin/s3cmd sync /var/log/nginx/access.*.gz s3://s3_bucket_name/${INSTANCE}/

endscript

}

Explanation : I will focus on the main directives.

/var/log/nginx/access.log – specify the log file to be rotated.

copytruncate – used in order to avoid reloading nginx after each rotate

size 100k – logs are rotated when the size of the logs exceeds 100k

dateext – append date at the end of the file name in s3

dateformat – specify the date format

you can skip first line, which is used to get the ip of the instance.

/usr/bin/s3cmd – used to sync the access log from their zipped file to the s3 bucket.

Now you can rotate your log file with the following command,

logrotate /etc/logrotate.d/nginx;

We can automate the log rotation scheduling cron for some fixed interval.

open up terminal.

nano /etc/cron.d/logrotate

paste,

*/1 * * * * root /usr/sbin/logrotate /etc/logrotate.conf

save and then exit from the machine and relogin for session to refresh.

Now you have a cron scheduled that runs every minute and rotate the logs if exceeding 100k size.

In cron, you cannot make schedule for seconds, for that you can write a bash script, which is very easy.

So, yeah, by now you must be able to configure logrotate to rotate your access log every minute to s3.

Ok Bye.

Note : don’t place any space in between INSTANCE=’command’ line otherwise it won’t run.

Ask me if any queries. 🙂