Bart Simons

files

A 2 post collection


Sync folders and files on Linux with rsync and inotify

 •  Filed under folders, files, linux, sync, rsync, inotify

So: you've got two or more clients and/or servers. They contain files that you want to have automatically synced when possible, because that would save a lot of time. Well, I got the solution for you: with a little bit of thinking in an innovative way I have found the solution that might bring you onto the right path as well. Rsync is a great solution, but having to run rsync manually would take a lot of unnecessary time away, right? And that is where inotify is for: real-time monitoring of your filesystem so that your files can be synced between multiple machines with the power of rsync!

Self-made sync daemon in a working state

Setting up a test scenario

I needed to get myself a nice development environment at first so I started off 3 virtual servers which all run Ubuntu 16.04, my personal favourite. All these 3 machines needed to be setup with the following software packages:

  • OpenSSH server
  • Rsync
  • Inotify

Also noteworthy is that these machines are absolutely not connected through a private network. All rsync traffic is supposed to be worked out over SFTP.

For who is this for?

I could see some potential for workflow improvement on these situations:

  • A development environment, where constant file transfers are taking up a lot and/or too much time
  • Load balanced file storage clusters/servers
  • Backup/failover servers with the need for constant replication
Working on it

First things first: we need to get all the dependencies installed on the 3 servers with this one-line command:

apt update && apt -y install openssh-server rsync inotify-tools  

After that, let's create a specific folder that we want to sync. Let's call it SyncFiles:

mkdir /opt/syncfiles  

And for secure file transfer, we want a public-private key link for the transfer link that rsync uses, this is how to configure it:

ssh-keygen -t rsa -f ~/rsync-key -N ''

# Paste the output in your destination servers' ~/.ssh/authorized_keys file:
cat ~/rsync-key.pub

# Removing public key for security purposes..
rm ~/rsync-key.pub

# Remember to execute this script on all servers separately! Then, copy the output of the script in all of your servers' authorized_keys files.

Now that you have got all the pre-configuration work done, it's about time to write a script that goes through an infinite loop with inotifywait in it:

#!/bin/bash

# Supposed to run on rsync-host01, change rsync-host02 to rsync-host01 to make a script that is meant to run on rsync-host02.

while true; do  
  inotifywait -r -e modify,attrib,close_write,move,create,delete /opt/syncfiles
  rsync -avz -e "ssh -i /root/rsync-key -o StrictHostKeyChecking=no“  /opt/syncfiles/ root@rsync-host02:/opt/syncfiles/
done  

I saved this script in the /opt directory as file-sync.sh.

To finish things off, lets create the systemd service file that can stop, start, and reset the script on demand or on specific events like a system bootup.

Create a file called sync.service in the directory /etc/systemd/system/ and put the following contents in it:

[Unit]
Description = SyncService  
After = network.target

[Service]
PIDFile = /run/syncservice/syncservice.pid  
User = root  
Group = root  
WorkingDirectory = /opt  
ExecStartPre = /bin/mkdir /run/syncservice  
ExecStartPre = /bin/chown -R root:root /run/syncservice  
ExecStart = /bin/bash /opt/file-sync.sh  
ExecReload = /bin/kill -s HUP $MAINPID  
ExecStop = /bin/kill -s TERM $MAINPID  
ExecStopPost = /bin/rm -rf /run/syncservice  
PrivateTmp = true

[Install]
WantedBy = multi-user.target  

Chmod this service file and reload the systemd daemon:

chmod 755 /etc/systemd/system/sync.service  
systemctl daemon-reload  

You are all set! You can now use these commands to manage your self-made directory sync daemon:

# Start your service
systemctl start sync.service

# Obtain your services' status
systemctl status sync.service

# Stop your service
systemctl stop sync.service

# Restart your service
systemctl restart sync.service  

Manually add files to ownCloud

 •  Filed under manually, add, files, folders, owncloud

Let's face it: you've got gigabytes of files to add to your ownCloud server. You can of course use the desktop client to sync all those files, but what if we were talking about almost a terabyte?

Add files manually to your ownCloud server

This is probably the quickest and less complicated way of adding files. There are many alternative ways to transfer your files to your ownCloud server:

  • FTP Upload;
  • SCP Upload;
  • Local disk to ownCloud disk transfer;
  • USB disk to ownCloud disk transfer.

It's just as simple as dragging a folder to another folder. The only problem is that you might not know yet where ownCloud files are stored. I made life a little bit easier with this nifty little Bash snippet that automatically finds your ownCloud data directory:

find / -name config.php 2>/dev/null | while read line ; do cat "$line" | grep " 'datadirectory'" ; done  

Your output should then look something like this:

ownCloud Data Directory Snippet

So according to the snippet of code my ownCloud data directory is located at /var/www/owncloud/data so let's cd into that directory to check out what's in there.

ownCloud Data Directory Contents

As you can see, we have multiple files in here. My ownCloud username is bart, so let's take a look in that folder.

ownCloud Data Directory Closer Look

Now this looks interesting. If I check my ownCloud web portal I can see that the directory structure is the same:

ownCloud Default Web Interface Folders

So I created a folder called TestFolder123 in the web interface and executed ls -la on the server's console.

ownCloud Default Web Interface With Test Folder

ownCloud Server Console Test Folder

So I tried to do the same thing, but vice versa. I created a folder on the console side, updated the folder's ownership and afterwards checked if the folder appeared in the web interface, with some surprising results:

ownCloud Server Console Test Folder 2

ownCloud Default Web Interface Without Console Test Folder

What the heck?!
Where did my folder go??!
Okay.. let me explain what's going on here. The ownCloud system basically expects two things from a file/folder in order to be indexed as a file:

  • The file/folder needs to be physically available;
  • The file/folder needs to exist in the ownCloud database.

Moving a file/folder into ownCloud's data directory doesn't add it to the database automatically. Luckily, the ownCloud developers were kind enough to create a command line tool called occ that can manually add missing files and folders to the ownCloud database. You can run the following snippet on your server to check where this utility is located on the local file system:

find / -name "occ" 2>/dev/null  

Which outputs the following on my demo server:
OCC Binary Location

All you need to do is cd into that folder (in my case /var/www/owncloud) and run the following snippet:

sudo -u www-data php occ files:scan --all  

Where www-data is the user your web server runs on.
And now, it is time for the results:

OCC Output From The Console

Positive Results In The Web Interface

YEAH!! So this command has definitely helped. Now you can add all your files directly into your ownCloud data folder without having to sync from the desktop client.