Categories
analytics chat chatting google google-analytics javascript live live-users long-polling nginx node.js performance persistent persistent-connection php5-fpm Proxy real-time response reverse-proxy socket socket.io socks tcp upstream web-server websocket websocket-handshake

How To Make Your Own Analytics Tool?

An analytics tool is a tool that you can use to get the information about how is your website performance, how are the users using it, what are the number of users at anytime(live users), live users for different urls, and other infos. One such example of the tool is the google analytics.

Analytics tool

Analytics tool gives a lot of information about your website with the nice looking graphs and all. In this blog, we will be discussing how we can make such analytics tool(digging the data).

Question 1: Why should we make such thing if it already exists?

Answer : One reason is that it is not free for always. The other reason is that the process of making it is very sexy and you learn technologies that you can use to make your own other real time apps like chatting app. Now that is more a valid reason.

Question 2: What are we talking about?

Answer : This is an example of a real time application. You see, you are tracking live users’ data. The data can be anything like total number of users anytime or number of users for different urls or different referral or different user agent or different visited ips, or anything. The other example of a real time application is a chatting application where a lot of users are communicating at the same time.

Question 3: What technologies are used for such an application?

Answer : So we need a continuous stream of response from the server about their performance as only servers know what is hitting them. It can only be achieved if there is some persistent connection between client and server so that server can continuously push data to the client. With persistent connection I would mean one for literally persistent connection(web sockets) or else timed persistent connection(long polling) or else chunk persistent connection(polling). By timed persistent connection, I mean persistent connection only for certain time interval after which it is closed and again opened. And by chunk persistent connection, I mean lots of request with no persistency, but the requests are so frequent that altogether they seem persistent(my bad).

Among all, I prefer web sockets, as I can’t logically satisfy myself why to use others and more of it that they are the outdated styles and only existed when sockets were not there.

So in this tutorial we will be using socket.io library with node.js to create web sockets. Below are the other libraries with other language which I found from here.

Analytics tool

And yes, we are going to use Nginx as a reverse proxy for web sockets. A very great tutorial regarding this can be found here. You can skip Nginx if you wish and directly test for your node.js server.

For Nginx configuration, as mentioned in tutorial,

server {
    server_name app.domain.com;
    location / {
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_pass http://socket_nodes;
    }
}

Proper explanation is given in that tutorial for how this will work for sockets. Now we setup our Node server(server.js), a very nice tutorial can be found here.

var data = {};
var http = require(“http”);
var url = require(‘url’);
var fs = require(‘fs’);
var io = require(‘socket.io‘);
var path;
var server = http.createServer(function(request, response){
        console.log(‘Request Accepted’);
        path = url.parse(request.url).pathname;
        switch(path){
            case ‘/’:
                response.writeHead(200, {‘Content-Type’: ‘text/html’});
                response.write(‘hello world’);
                response.end();
                break;
            case ‘/live’:
                fs.readFile(__dirname + ‘/socket_io.html’, function(error, data){
                    if (error){
                        response.writeHead(404);
                        response.write(“opps this doesn’t exist – 404”);
                        response.end();
                    }
                    else{
                        response.writeHead(200, {“Content-Type”: “text/html”});
                        response.write(data, “utf8”);
                        response.end();
                    }
                });
                break;
            default:
                response.writeHead(404);
                response.write(“opps this doesn’t exist – 404  –“+path);
                response.end();
                break;
        }
    });
 server.listen(8001);
 var listener = io.listen(server);
 listener.sockets.on(‘connection’, function(socket){
    console.log(‘Socket Made.’);
    setInterval(function(){
        socket.emit(‘showdata’, {‘data’: data});
    }, 1000);
 socket.on(‘disconnect’, function(){
    console.log(“Socket Closed.”);
 });
});

Load socket.io.js on the client side in your socket_io.html which will be used to create event to create a persistent connection. (Please don’t bang your head for how would it find socket.io.js)

/socket.io/socket.io.js

     var socket = io.connect(); // your initialization code here.
     socket.on('showdata', function(data){
          console.log(data);
     });

Now you can see all the data in your browser’s console. You can put this data at the right place of your analytics html. Now you can easily send continuous data from node server to your client.

Are you still wondering how would we gather data? Suppose we want to track number of live users, for that we will use a key “live” in data variable and initialize with 0. We will increment it as soon as there is a socket made and decrement it as soon as the socket is closed(Make sure io.connect() is called once and only once for a page load). So the data[‘live’] will determine the number of live users.

In server.js,

listener.sockets.on(‘connection’, function(socket){
    console.log(‘Socket Made.’);
    data[‘live’] +=1;
    setInterval(function(){
        socket.emit(‘showdata’, {‘data’: data});
    }, 1000);
 socket.on(‘disconnect’, function(){
 data[‘live’]-=1;
    console.log(“Socket Closed.”);
 });

As simple as that. It is just the logics in node to be done and you can achieve any data. Now for temporary data(by temporary I mean the machine hosting node server is until rebooted), we don’t need any extra storage.

If we want to use some durable data, or store some information, we can choose database of our choice and send data from node to database.

Now we are also using Nginx as a reverse proxy which is helpful in load handling and better scaling. We can extract other information like user ip, upstream responses(php), and other valuable information and send to node.

So yes, it is really interesting to make such tool, I told you so. You are surely going to stuck in somewhere or make it up or for any suggestion or for confusion, we can always discuss.

Great to see you people with your own analytics tool. 🙂

Categories
bot Proxy Python subprocess tor traffic web scraper webbrowser

The Bot : How To Make A Simple Anonymous Web Scraper?

Web scraper is a program to automate the process of accessing websites with or without browsers. In this post, by web scraping, I mean of only accessing the webpages. And the web scraper built here access webpages through web browsers.

An anonymous web scraper will be the one which keeps the identity of the program hidden. That means a program that accesses some website without revealing its information(ip address).

Usage(Though this should be used at one’s own risk)
There are websites which offer you money to bring traffic to their websites. Using such anonymous web scrappers (the bots), you can send fake traffic to their websites and earn some money.

Note: These programs are tested on ubuntu 14.04. Similar programs can be made on other platforms as well.
We will discuss two ways of making such web scraper here using python.
For the first method, we will use webbrowser library of python to open webpages, and tor to create anonymity. We will use subprocess library to automate it.

Bot

import time
import subprocess as sp
import webbrowser

urls = [“www.example.com/p”, “www.example.com/q”]
count =10000
while count>=0:
for url in urls:
webbrowser.open(url)
time.sleep(2)
time.sleep(4)
sp.call([“sudo”, “killall”, “firefox”])
sp.call([“sudo”, “/etc/init.d/tor”, “restart”])
count-=1

Explanation

1- urls contain the list of url, bringing traffic on which gives you money.
2- while loop makes the iteration over the list of urls.
3- each url is opened using webbrowser(default firefox) and waits for two seconds.
4- then it waits for 4 seconds after opening all the urls to let the pages load properly(these can be changed according to the time, website takes to load pages).
5- the firefox is forcefully closed
6- tor is restarted so that user ip is changed.

Note
Before running the program, set up the firefox for socks5 proxy and port no. 9050 to make requests through tor.

For the second method, we will simply open browser through subprocess,

import time
import subprocess as sp
import webbrowser

urls = [“www.example.com/p”, “www.example.com/q”]
count =10000
while count>=0:
for url in urls:
child = sp.Popen(“firefox %s” %url, shell=True)
time.sleep(2)
time.sleep(4)
sp.call([“sudo”, “killall”, “firefox”])
sp.call([“sudo”, “/etc/init.d/tor”, “restart”])
count-=1

Explanation
1- the webpages are opened as a subprocess

Now you are all set up with your anonymous web scraper.

Note
Do setup the firefox for proxy as in first method.
With such bots, they must execute javascript to send information, which can be done using browsers.
The other technique used to create bot using phantomJs and bash can be found here.
Read websites’ terms, take your own risks and enjoy free money. 🙂

Categories
Dryscrape Polipo Proxy Python socks tor

Proxy The Proxy : How to set socks5 proxy in dryscrape(python library)?

Dryscrape is the python library that can be used to execute javascript code when a webpage is opened. You can read about it more from here.

Now we can set the http proxy for dryscrape in the following way:

sess = dryscrape.Session(baseurl = "http://www.example.com")
sess.set_proxy(host='localhost', port = port_no)

But the problem is that, we can set only http proxy to dryscrape session.
But sometimes we may want to set socks proxy for dryscrape session. I wanted it to connect it through tor to make an anonymous web scraper but tor works only on socks protocol.

Now there is an interesting called Polipo.
Polipo is a web proxy that supports socks protocol. But the more interesting thing(may be just to me or other novice explorers) is that it can pipeline the request to other proxy server of other protocol from request protocol.

In more simple language, we can configure Polipo to route the request from one server listening on some protocol to other server with other protocol.

So, we will configure our polipo server to listen for http requests and forward it through socks protocol. So this will proxy the proxy server for initial request(might be puzzling).

Solution:

Following is the configuration that can be added in your /etc/polipo/config file,

socksParentProxy = "localhost:9050"
socksProxyType = socks5

proxyAddress = “localhost”
proxyPort = 8118

Explanation:

socksParentProxy determines the next machine with port no. to target for the incoming request(tor running on my machine with default port no. 9050).
socksProxyType determines the type of protocol of the next server.
proxyAddress detemines the address of the polipo server.
proxyPort determines the port no. for polipo server(default 8118).

Now we can set proxy server as polipo server for dryscrapes’ requests in the following way:
sess = dryscrape.Session(base_url = "http://www.example.com")
sess.set_proxy(host = "localhost", port = 8118)

So, now whenever dryscrape makes request for some webpage, the request first goes to polipo and then it gets forward to tor(or some other server).

Cheers. Now, using dryscrape, we can make request through socks protocol(socks5 proxy).

Some of the related questions on stackoverflow can be found here.