Bart Simons

Bart Simons


Thoughts, stories and ideas.

Bart Simons
Author

Share


Tags


Twitter


Sync folders and files on Linux with rsync and inotify

So: you've got two or more clients and/or servers. They contain files that you want to have automatically synced when possible, because that would save a lot of…

Bart SimonsBart Simons

So: you've got two or more clients and/or servers. They contain files that you want to have automatically synced when possible, because that would save a lot of time. Well, I got the solution for you: with a little bit of thinking in an innovative way I have found the solution that might bring you onto the right path as well. Rsync is a great solution, but having to run rsync manually would take a lot of unnecessary time away, right? And that is where inotify is for: real-time monitoring of your filesystem so that your files can be synced between multiple machines with the power of rsync!

Self-made sync daemon in a working state

Setting up a test scenario

I needed to get myself a nice development environment at first so I started off 3 virtual servers which all run Ubuntu 16.04, my personal favourite. All these 3 machines needed to be setup with the following software packages:

  • OpenSSH server
  • Rsync
  • Inotify

Also noteworthy is that these machines are absolutely not connected through a private network. All rsync traffic is supposed to be worked out over SFTP.

For who is this for?

I could see some potential for workflow improvement on these situations:

  • A development environment, where constant file transfers are taking up a lot and/or too much time
  • Load balanced file storage clusters/servers
  • Backup/failover servers with the need for constant replication
Working on it

First things first: we need to get all the dependencies installed on the 3 servers with this one-line command:

apt update && apt -y install openssh-server rsync inotify-tools

After that, let's create a specific folder that we want to sync. Let's call it SyncFiles:

mkdir /opt/syncfiles

And for secure file transfer, we want a public-private key link for the transfer link that rsync uses, this is how to configure it:

ssh-keygen -t rsa -f ~/rsync-key -N ''

# Paste the output in your destination servers' ~/.ssh/authorized_keys file:
cat ~/rsync-key.pub

# Removing public key for security purposes..
rm ~/rsync-key.pub

# Remember to execute this script on all servers separately! Then, copy the output of the script in all of your servers' authorized_keys files.

Now that you have got all the pre-configuration work done, it's about time to write a script that goes through an infinite loop with inotifywait in it:

#!/bin/bash

# Supposed to run on rsync-host01, change rsync-host02 to rsync-host01 to make a script that is meant to run on rsync-host02.

while true; do
  inotifywait -r -e modify,attrib,close_write,move,create,delete /opt/syncfiles
  rsync -avz -e "ssh -i /root/rsync-key -o StrictHostKeyChecking=no“  /opt/syncfiles/ root@rsync-host02:/opt/syncfiles/
done

I saved this script in the /opt directory as file-sync.sh.

To finish things off, lets create the systemd service file that can stop, start, and reset the script on demand or on specific events like a system bootup.

Create a file called sync.service in the directory /etc/systemd/system/ and put the following contents in it:

[Unit]
Description = SyncService
After = network.target

[Service]
PIDFile = /run/syncservice/syncservice.pid
User = root
Group = root
WorkingDirectory = /opt
ExecStartPre = /bin/mkdir /run/syncservice
ExecStartPre = /bin/chown -R root:root /run/syncservice
ExecStart = /bin/bash /opt/file-sync.sh
ExecReload = /bin/kill -s HUP $MAINPID
ExecStop = /bin/kill -s TERM $MAINPID
ExecStopPost = /bin/rm -rf /run/syncservice
PrivateTmp = true

[Install]
WantedBy = multi-user.target

Chmod this service file and reload the systemd daemon:

chmod 755 /etc/systemd/system/sync.service  
systemctl daemon-reload

You are all set! You can now use these commands to manage your self-made directory sync daemon:

# Start your service
systemctl start sync.service

# Obtain your services' status
systemctl status sync.service

# Stop your service
systemctl stop sync.service

# Restart your service
systemctl restart sync.service 
Bart Simons
Author

Bart Simons

Comments