Bart Simons

nginx

A 2 post collection


Automating NGINX access log analysis

 •  Filed under nginx, access log, analysis

Are you running NGINX on your web server? The NGINX access log contains interesting information that might be informative to parse. You can use the cat UNIX utility to write today's log to your terminal output:

cat /var/log/nginx/access.log  

This should work on all Debian-based distributions. It should give you a view of which IP's visited your website and what they were trying to access, and other information that's good to know.

You can filter the output even further with UNIX utilities like awk and sort:

#!/bin/bash

for UNIQUE_IP in $(awk '{print $1}' /var/log/nginx/access.log | sort -u)  
do  
        host $UNIQUE_IP | awk '{print $5}'
done  

This script returns a list of the visitors' hostnames.

How about some interactivity? We can do that as well with Python and the pythondialog library:

#!/usr/bin/env python

# nginx-accesslogmonitor.py - An all-in-one solution for checking your NGINX access logs!
# Developed by Bart Simons, 2016

from dialog import Dialog  
import socket

addresses = []  
accessloglines = []  
menu_address_options = []

d = Dialog(dialog="dialog")

def showMainMenu(menu_items):  
    menu_selection = d.menu("Select an IP address you want to inspect:", height=None, width=None, menu_height=None, choices=menu_items)
    return menu_selection

def showAddressInfo(address):  
    address_info=""
    for line in accessloglines:
        if address in line:
            address_info=address_info+line
    d.scrollbox(address_info, height=0, width=0)

with open('/var/log/nginx/access.log') as f:  
    for line in f:
        accessloglines.append(line)
        addresses.append(line.split(" ")[0])

addresses = list(set(addresses))

for x in range(0, len(addresses)):  
    try:
        menu_address_options.append([addresses[x], socket.gethostbyaddr(addresses[x])[0]])
    except:
        menu_address_options.append([addresses[x], "hostname unknown"])

def executionLoop():  
    resultMainMenu=showMainMenu(menu_address_options)
    if (resultMainMenu[0] == 'ok'):
        showAddressInfo(resultMainMenu[1])
        executionLoop()


if __name__ == "__main__":  
    executionLoop()

Here are two screenshots of what the result looks like:

The last Python script should make your NGINX access log monitoring activities much more controllable, allowing you to iterate over all IPs and hostnames one by one. Thanks for reading and have a nice day 👋

NGINX RTMP Streaming Server Installation Guide

 •  Filed under tutorial, nginx, rtmp, streaming, server, installation, guide

Personally, I find NGINX the best choice as a web server, and so this website runs on NGINX too. Today I found out about the NGINX RTMP module, originally developed by Roman Arutyunyan. This module allows you to expand NGINX with RTMP capabilities so you can use NGINX as a media streaming server! You can build it by compiling NGINX with a separate module which can be done manually or by using this bash script:

#!/bin/bash

echo  
echo   NGINX RTMP SERVER INSTALLER V2  
echo   COPYLEFT BARTSIMONS.ME, 2016  
echo 

## CHECKING FOR ROOT ACCESS...
user=$(whoami)  
if [[ $user != "root" ]]; then  
    echo "You are not root. Please run this script as superuser!"
    exit
fi

## GLOBAL VARIABLES AND PACKAGE CACHE UPDATE
nginx_url="http://nginx.org/download/nginx-1.11.4.tar.gz"  
nginx_tar="nginx-1.11.4.tar.gz"  
nginx_fld="nginx-1.11.4"  
rtmp_url="https://github.com/arut/nginx-rtmp-module.git"

echo "Updating package cache..."  
apt -qqq update

## CONTINUE WHEN USER IS ROOT & INSTALL WGET IF NOT INSTALLED...
long_out_wget_check=$(dpkg-query --list | grep wget)  
short_out_wget_check=${long_out_wget_check:0:2}  
wget_installed=0

if [[ $short_out_wget_check == "ii" ]]; then  
    wget_installed=1
else  
    echo "Installing wget..."
    apt install -qqq -y wget
fi

## INSTALL BUILD-ESSENTIAL IF NOT INSTALLED...
long_out_be_check=$(dpkg-query --list | grep build-essential)  
short_out_be_check=${long_out_be_check:0:2}  
be_installed=0

if [[ $short_out_be_check == "ii" ]]; then  
    be_installed=1
else  
    echo "Installing build-essential..."
    apt install -qqq -y build-essential
fi

## INSTALL LIBPCRE3 DEV HEADERS IF NOT INSTALLED...
long_out_pcre_dev_check=$(dpkg-query --list | grep libpcre3-dev)  
short_out_pcre_dev_check=${long_out_pcre_dev_check:0:2}  
pcre_dev_installed=0

if [[ $short_out_pcre_dev_check == "ii" ]]; then  
    pcre_dev_installed=1
else  
    echo "Installing libpcre3 development headers..."
    apt install -qqq -y libpcre3-dev
fi

## INSTALL LIBPCRE IF NOT INSTALLED...
long_out_pcre_check=$(dpkg-query --list | grep libpcre3-dev)  
short_out_pcre_check=${long_out_pcre_check:0:2}  
pcre_installed=0

if [[ $short_out_pcre_check == "ii" ]]; then  
    pcre_installed=1
else  
    echo "Installing libpcre3..."
    apt install -qqq -y libpcre3
fi

## INSTALL GIT IF NOT INSTALLED
long_out_git_check=$(dpkg-query --list | grep "git ")  
short_out_git_check=${long_out_git_check:0:2}  
git_installed=0

if [[ $short_out_git_check == "ii" ]]; then  
    git_installed=1
else  
    echo "Installing git..."
    apt install -qqq -y git
fi

## INSTALL LIBSSL DEV HEADERS IF NOT INSTALLED...
long_out_libssl_dev_check=$(dpkg-query --list | grep libssl-dev)  
short_out_libssl_dev_check=${long_out_libssl_dev_check:0:2}  
libssl_dev_installed=0

if [[ $short_out_libssl_dev_check == "ii" ]]; then  
    libssl_dev_installed=1
else  
    echo "Installing libssl-dev..."
    apt install -qqq -y libssl-dev
fi

## DOWNLOAD AND UNTAR NGINX SOURCE CODE
echo "Downloading nginx source code..."  
wget --quiet $nginx_url  
echo "Unpacking nginx source code..."  
tar -xzf $nginx_tar

## CLONE NGINX-RTMP-MODULE
echo "Cloning nginx RTMP module git repository..."  
git clone $rtmp_url

## CONFIGURE, COMPILE AND INSTALL!
cd $nginx_fld  
./configure --add-module=../nginx-rtmp-module
make  
make install

## CLEANUP TIME!
echo "Cleaning up left over folders & files..."  
rm -rf $nginx_fld  
rm -rf $nginx_tar  
rm -rf nginx-rtmp-module

if [[ $git_installed == 0 ]]; then  
    echo "git was not installed earlier. Uninstalling git"
    apt remove --purge -qqq git
fi

if [[ $pcre_dev_installed == 0 ]]; then  
    echo "libpcre3-dev was not installed earlier. Uninstalling libpcre3-dev..."
    apt remove --purge -qqq libpcre3-dev
fi

if [[ $be_installed == 0 ]]; then  
    echo "build-essential was not installed earlier. Uninstalling build-essential..."
    apt remove --purge -qqq build-essential
fi

if [[ $wget_installed == 0 ]]; then  
    echo "wget was not installed earlier. Uninstalling wget..."
    apt remove --purge -qqq wget
fi

if [[ $pcre_installed == 0 ]]; then  
    echo "libpcre3 was not installed earlier. Uninstalling libpcre..."
    apt remove --purge -qqq libpcre3
fi

if [[ $libssl_dev_installed == 0 ]]; then  
    echo "libssl-dev was not installed earlier. Uninstalling libssl-dev..."
    apt remove --purge -qqq libssl-dev
fi

echo " "  
echo "NGINX and the RTMP server module has been installed on your system!"  

Please note that this script has been built for Debian-based operating system. Compiling it manually on other systems is not that difficult at all: you just need to include the module with a flag for the configure script.

Configuring NGINX

Once you've got the modified NGINX version installed on your server, it's time to edit the NGINX configuration so that NGINX will serve RTMP traffic.

The default configuration file location for NGINX is /usr/local/nginx/conf/nginx.conf
Add the following configuration to the end of this file:

rtmp {  
    server {
        listen 1935;
        chunk_size 8192;

        application stream {
            live on;
            record off;

            allow publish 127.0.0.1;
            deny publish all;
            allow play all;
        }
    }
}

Now you are ready to go, you can start nginx on your server

/usr/local/nginx/sbin/nginx


You can stop nginx like this:

/usr/local/nginx/sbin/nginx -s stop


Thanks for reading and have fun streaming!