Bart Simons

Bart Simons


Thoughts, stories and ideas.

Bart Simons
Author

Share


Tags


.net .net core Apache C# CentOS LAMP NET Framework Pretty URLs Windows Server WireGuard WireGuard.io access log add analysis android api at the same time authentication authorization automate automation azure azurerm backup bash basics batch bootstrap build capture cheat sheet chromium chroot class cli click to close code snippet command line commands compile compiling compression containers control controller controlling convert cpu usage create credentials csv csvparser curl data dd deployment desktop detect devices disable diskpart dism distributed diy docker dom changes dotnet core drivers ease of access encryption example export file transfer files fix folders generalize getting started ghost ghost.org gui guide gunicorn gzip html html tables icewarp igd imagex import inotify install installation interactive ios iphone itunes java javascript jquery json kiosk kotlin linux live load data loading screen lock screen loopback audio lxc lxd lxml macos manage manually message messages minio mirrored mod_rewrite monitor monitoring mutationobserver mysql nexmo nginx no oobe node node.js nodejs not installing notification notifications object storage on desktop one command openssl owncloud parallels parallels tools parse perfect philips hue play port forwarding portainer.io powershell processing ps-spotify python quick raspberry pi record rip ripping rsync rtmp save save data sbapplication scraping script scripting scriptingbridge scripts security send server service sharedpreferences sms songs sonos spotify spotify api spotlight ssh stack streaming streamlink studio sudo swarm swift sync sysprep system audio systemd tables terminal tracking tutorial twilio ubiquiti ubuntu ubuntu 18.04 ui code unifi unlock unsplash source upnp uptime usb tethering wallpapers wasapi website websites webview windows windows 10 without itunes without oobe workaround xaml

Automating NGINX access log analysis

Are you running NGINX on your web server? The NGINX access log contains interesting information that might be informative to parse. You can use the cat UNIX utility to write today's log to your terminal output:

cat /var/log/nginx/access.log

This should work on all Debian-based distributions. It should give you a view of which IP's visited your website and what they were trying to access, and other information that's good to know.

You can filter the output even further with UNIX utilities like awk and sort:

#!/bin/bash

for UNIQUE_IP in $(awk '{print $1}' /var/log/nginx/access.log | sort -u)
do
        host $UNIQUE_IP | awk '{print $5}'
done

This script returns a list of the visitors' hostnames.

How about some interactivity? We can do that as well with Python and the pythondialog library:

#!/usr/bin/env python

# nginx-accesslogmonitor.py - An all-in-one solution for checking your NGINX access logs!
# Developed by Bart Simons, 2016

from dialog import Dialog
import socket

addresses = []
accessloglines = []
menu_address_options = []

d = Dialog(dialog="dialog")

def showMainMenu(menu_items):
	menu_selection = d.menu("Select an IP address you want to inspect:", height=None, width=None, menu_height=None, choices=menu_items)
	return menu_selection

def showAddressInfo(address):
	address_info=""
	for line in accessloglines:
		if address in line:
			address_info=address_info+line
	d.scrollbox(address_info, height=0, width=0)

with open('/var/log/nginx/access.log') as f:
	for line in f:
		accessloglines.append(line)
		addresses.append(line.split(" ")[0])

addresses = list(set(addresses))

for x in range(0, len(addresses)):
	try:
		menu_address_options.append([addresses[x], socket.gethostbyaddr(addresses[x])[0]])
	except:
		menu_address_options.append([addresses[x], "hostname unknown"])

def executionLoop():
	resultMainMenu=showMainMenu(menu_address_options)
	if (resultMainMenu[0] == 'ok'):
		showAddressInfo(resultMainMenu[1])
		executionLoop()


if __name__ == "__main__":
	executionLoop()

Here are two screenshots of what the result looks like:

The last Python script should make your NGINX access log monitoring activities much more controllable, allowing you to iterate over all IPs and hostnames one by one. Thanks for reading and have a nice day 👋

Bart Simons
Author

Bart Simons

View Comments