Bart Simons

access log

A 1 post collection


Automating NGINX access log analysis

 •  Filed under nginx, access log, analysis

Are you running NGINX on your web server? The NGINX access log contains interesting information that might be informative to parse. You can use the cat UNIX utility to write today's log to your terminal output:

cat /var/log/nginx/access.log  

This should work on all Debian-based distributions. It should give you a view of which IP's visited your website and what they were trying to access, and other information that's good to know.

You can filter the output even further with UNIX utilities like awk and sort:

#!/bin/bash

for UNIQUE_IP in $(awk '{print $1}' /var/log/nginx/access.log | sort -u)  
do  
        host $UNIQUE_IP | awk '{print $5}'
done  

This script returns a list of the visitors' hostnames.

How about some interactivity? We can do that as well with Python and the pythondialog library:

#!/usr/bin/env python

# nginx-accesslogmonitor.py - An all-in-one solution for checking your NGINX access logs!
# Developed by Bart Simons, 2016

from dialog import Dialog  
import socket

addresses = []  
accessloglines = []  
menu_address_options = []

d = Dialog(dialog="dialog")

def showMainMenu(menu_items):  
    menu_selection = d.menu("Select an IP address you want to inspect:", height=None, width=None, menu_height=None, choices=menu_items)
    return menu_selection

def showAddressInfo(address):  
    address_info=""
    for line in accessloglines:
        if address in line:
            address_info=address_info+line
    d.scrollbox(address_info, height=0, width=0)

with open('/var/log/nginx/access.log') as f:  
    for line in f:
        accessloglines.append(line)
        addresses.append(line.split(" ")[0])

addresses = list(set(addresses))

for x in range(0, len(addresses)):  
    try:
        menu_address_options.append([addresses[x], socket.gethostbyaddr(addresses[x])[0]])
    except:
        menu_address_options.append([addresses[x], "hostname unknown"])

def executionLoop():  
    resultMainMenu=showMainMenu(menu_address_options)
    if (resultMainMenu[0] == 'ok'):
        showAddressInfo(resultMainMenu[1])
        executionLoop()


if __name__ == "__main__":  
    executionLoop()

Here are two screenshots of what the result looks like:

The last Python script should make your NGINX access log monitoring activities much more controllable, allowing you to iterate over all IPs and hostnames one by one. Thanks for reading and have a nice day 👋