Activate SSL/TLS on the SFTPGO web UI

In this post I describe how to activate SSL/TLS encryption on the SFTPGO web admin UI with CERTBOT and enable auto renewal of the certificates using a post renewal hook shell script.

Certbot is a free tool to enable TLS 1.3 AES 128 SHA 256 https using Lets encrypt certificates.

In this setup guide I mostly followed the official Certbot instruction for a snapd installation/other server and some of the instructions from here.

This is a follow up post to Setting up a SFTPGO SFTP server on a Hetzner Ubuntu 20.10 server.

Install Certbot SSL/TLS encryption on the SFTPGO web UI

Install snap with apt

sudo apt install snapd

Install snap core

sudo snap install core; sudo snap refresh core

Make sure certbot is not already installed with apt

sudo apt remove certbot

Install certbot with snap

sudo snap install --classic certbot

Make a symlink between /snap/bin/certbot and /usr/bin/certbot

sudo ln -s /snap/bin/certbot /usr/bin/certbot

Open port 80 in the firewall

Certbot creates a web server on port 80 when it generates the ssl/tls certificates so I open port 80 in the firewall. SFTPGO is running on port 8080.

Hetzner firewall

Image of opening port 80 in the hetzner firewall

nftables firewall

In this post I used a nftables firewall. To activate HTTP(S) you would uncomment these lines

Allow HTTP(S).  
tcp dport { http, https } accept
udp dport { http, https } accept

Run Certbot

Cerbot uses the domain name of your server. You can find it by doing a reverse domain lookup on your server ip address. You also need to give certbot an email address.

Link: mxtoolbox reverse domain lookup

sudo certbot certonly --standalone

Make the post renewal hook shell script and run it

This shell script copies the files to the sftpgo ssl directory, changes the ownership of the certificates to the sftpgo user and reloads sftpgo (sends a SIGHUP) when certbot renews the certificates. SFTPGO will keep on running when it reloads.

Open the script in nano

sudo nano /etc/letsencrypt/renewal-hooks/post/sftpgo.sh

Copy and paste the sftpgo.sh script

You need to change the 353.server.com to the name of your server.

Github link for this script.

#!/bin/sh
sudo cp /etc/letsencrypt/live/353.server.com/cert.pem /etc/sftpgo/ssl/
sudo cp /etc/letsencrypt/live/353.server.com/privkey.pem /etc/sftpgo/ssl/
sudo chown -R sftpgo:sftpgo /etc/sftpgo/ssl
sudo systemctl reload sftpgo 
Image of the script pasted in nano

Make the sftpgo.sh shell script executable

sudo chmod 755 /etc/letsencrypt/renewal-hooks/post/sftpgo.sh 

Run the sftpgo.sh script

cd /etc/letsencrypt/renewal-hooks/post/ 
./sftpgo.sh

Enable SSL/TLS https in the SFTPGO configuration file sftpgo.json

Edit the sftpgo.json configuration file with nano

sudo nano /etc/sftpgo/sftpgo.json

Enable_https from false to true

"enable_https": true,


Change certificate_file to /etc/sftpgo/ssl/cert.pem
Change certificate_key_file to /etc/sftpgo/ssl/privkey.pem

"certificate_file": "/etc/sftpgo/ssl/cert.pem",
"certificate_key_file": "/etc/sftpgo/ssl/privkey.pem",
image of the changed json file

Restart SFTPGO

sudo systemctl restart sftpgo
sudo systemctl status sftpgo
image of sftpgo status

Type in your server address in a web browser

https://628.yourserveraddress.com:8080

The lock icon besides the address will be locked and SSL/TLS encryption is active.

sftpgo web page

lock icon

Testing the certbot auto renewal process (optional)

Check the status of snap.certbot.renew.timer

Cerbot installs a systemd timer called snap.certbot.renew.timer.

It runs twice every day to check if it is time to renew the certificates. (It will run the renewal every 90 days)

sudo systemctl status snap.certbot.renew.timer 

Run a forced renewal of the certificates

To check if everything works, you can run a forced renewal of the certificates. (Max 5 times a week).

sudo certbot renew --force-renewal

Type in your server address in a web browser

https://628.yourserver.com:8080

Check if it works.

Setting up a SFTPGO SFTP server on a Hetzner Ubuntu 20.10 server

SFTPGO can run SFTP, FTP and WEBDAV and you can easily configure virtual chrooted users via a web browser.

In this post I set up SFTPGO SFTP on Ubuntu 20.10 on a Hetzner auction server with 14 TB of SFTP storage . This tutorial also works for other Ubuntu 20.10 installations, just skip the part about installing Ubuntu 20.10 on a Hetzner server.

Activate the rescue system on the Hetzner Robot web page and install ubuntu 20.10

You will now have a root password. When you reset the server with Reset – Send CTRL-ALT-DEL to the server you can log in and run installimage

Log in to the server running the rescue system

I use terminal on mac or Ubuntu on Windows

ssh root@1.1.1.1

Run installimage

installimage

Choose to install ubuntu 20.10. Since it is not a LTS release you can easily add the SFTPGO PPA repository later.

The default settings for this 4 x 6 TB HDD server is RAID level 6 where two hard drives are redudant. 2 TB for / and almost all of the the rest for /home is the default. I changed to RAID level 5 so only one hard drive is redudant and set 1080G for / and 1080G for /home and the rest in /srv (14TB) for sftpgo to use.

Install Ubuntu by exiting the editor with F10 and finish the installer

Reboot the server

reboot

Log in to the server running ubuntu 20.10

ssh root@1.1.1.1

Check storage with df -h

df -h

Add a sudoer user

It is recommended to use a normal user that is part of the sudo group instead of using root.

sudo adduser newuser
usermod -aG sudo newuser

Login in with the new sudo user

When you need to do something as root use sudo. Or su.

ssh newuser@1.1.1.1

Update Ubuntu

sudo apt update
sudo apt upgrade

Disable root ssh login and change ssh port (optional)

For security reasons it is common to change the default ssh port and disable root ssh login. If you leave the default port open you can install a brute force blocker like sshguard.

sudo apt install nano
sudo nano /etc/ssh/sshd_config

comment # PermitRootLogin to disable root login.

Change the port to 2222


Restart the sshd service
sudo systemctl restart sshd.service 

Log in with the new port number

ssh newuser@1.1.1.1 -p 2222

Change the firewall on the Hetzner server admin web site to use the alternative SSH port, SFTPGO SFTP port and web admin UI port

I edited the webserver template to also accept ssh/sftp on port to 2222, 2022 and http on 8080. You can remove access to the web interface in the firewall when it is not in use.

Install SFTPGO

The easiest way to install SFTPGO is to add the SFTPGO PPA repository to ubuntu 20.10. In 20.04 LTS it is not allowed by default to add PPA repositories.

sudo apt install software-properties-common 
sudo add-apt-repository ppa:sftpgo/sftpgo
sudo apt install sftpgo
sudo systemctl status sftpgo

Open the SFTPGO web admin UI to the internet

Change the “httpd” “address”: “”127.0.0.1″ to httpd” “address”: “” in the sftpgo.json configuration file so you can access the web admin UI from the internet. You can also enable the built in brute force defender if you change “defender” to “enabled”. It is also possible to enable https SSL/TLS encryption using cerbot like I wrote about in this post.

sudo nano /etc/sftpgo/sftpgo.json

Restart sftpgo

sudo systemctl restart sftpgo

Change the default admin password

Login to the web admin interface with a web browser with the username admin and the password password. Use the IP address of the server and port 8080. Like http://1.1.1.1:8080/


Change the default admin password.

Add a SFTPGO SFTP user

The minimim settings for a user is username, password and permissions like all * or download only. If the user is called user1 the default directory is /srv/sftpgo/data/user1. The user can only upload and download to this directory. User1 is not an actual user on the system and needs a home directory that the sftpgo user has access to like /srv . You can also choose to enable max connections, disk storage quota, max bandwith and more.

You can now connect to the SFTP server with Cyberduck or similar SFTP clients. I recommend using Cyberduck because it uses segmented downloads.

Connect to the SFTPGO SFTP server

Choose SFTP and port 2022 and username and password.

Try uploading and downloading. I get 22 MB/ sec downloading on a 200 mbit connection.

You see which users are connecting and downloading when clicking on connections in the SFTPGO web admin ui.

Check CPU and memory usage on the server with htop

sudo apt install htop
htop

SFTPGO uses a litte more CPU than the Openssh SFTP server. The server seems to handle it well.

Setting up SFTP with chrooted users and some other things you can do on a dedicated Linux server

In my post Some tips on downloading and uploading DCPs and large files I wrote about using Filemail, IBM Aspera and Amazon s3 and a dedicated sftp server

Here is some tips when setting up SFTP on a dedicated server running Ubuntu 20.04 or 20.10. A hetzner auction server starts at 30 euro a month. A Raspberry Pi 4/400 cost around 100 euro and can be used as SFTP server.

Start the OpenSSH server


The OpenSSH server is already running on Ubuntu server, but if you are using Ubuntu Desktop you can install and start the Openssh server

sudo apt update
sudo apt install openssh-server
sudo systemctl status ssh

Recommended SSH client

On mac I use terminal. On Windows I use Ubuntu on WSL. To connect to the server.

ssh root@1.1.1.1

Recommended SFTP client

Cyberduck is a fast SFTP client for Windows and Mac because it uses segmented downloads by default.

Setting up basic security

Generating good passwords

sudo apt install pwgen

Generate a 12 digit password

pwgen 12 1

Firewall

Use nftables firewall to accept only port 22 SSH/SFTP. If Nftables is not installed you can install it and start it with these commands.

sudo apt update
sudo apt install nftables
sudo systemctl enable nftables
sudo systemctl start nftables
sudo systemctl status nftables

Edit nftables.conf with modified Simple ruleset for a server to only accept SSH/SFTP

sudo nano /etc/nftables.conf
#!/usr/sbin/nft -f

flush ruleset

table inet firewall {

    chain inbound {

        # By default, drop all traffic unless it meets a filter
        # criteria specified by the rules that follow below.
        type filter hook input priority 0; policy drop;

        # Allow traffic from established and related packets.
        ct state established,related accept

        # Drop invalid packets.
        ct state invalid drop

        # Allow loopback traffic.
        iifname lo accept

        # Allow all ICMP and IGMP traffic, but enforce a rate limit
        # to help prevent some types of flood attacks.
        ip protocol icmp limit rate 4/second accept
        ip6 nexthdr ipv6-icmp limit rate 4/second accept
        ip protocol igmp limit rate 4/second accept

        # Allow SSH on port 22.
        tcp dport 22 accept

        # Allow HTTP(S).
        # -- From anywhere
        #tcp dport { http, https } accept
        #udp dport { http, https } accept
        # -- From approved IP ranges only
        # tcp dport { http, https } ip saddr $SAFE_TRAFFIC_IPS accept
        # udp dport { http, https } ip saddr $SAFE_TRAFFIC_IPS accept

        # Uncomment to allow incoming traffic on other ports.
        # -- Allow Jekyll dev traffic on port 4000.
        # tcp dport 4000 accept

        # Uncomment to enable logging of denied inbound traffic
        # log prefix "[nftables] Inbound Denied: " flags all counter drop

    }

    chain forward {

        # Drop everything (assumes this device is not a router)
        type filter hook forward priority 0; policy drop;

        # Uncomment to enable logging of denied forwards
        # log prefix "[nftables] Forward Denied: " flags all counter drop

    }

    chain outbound {

        # Allow all outbound traffic
        type filter hook output priority 0; policy accept;

    }

}

List the NFT ruleset

sudo nft list ruleset

Stop brute force attacks with sshguard and nftables

Install nftables if necessary

To set up sshguard I used the instruction from here

sudo apt install sshguard

Add sshg-fw-nft-sets to BACKEND= in sshguard-conf if needed

sudo nano /etc/sshguard/sshguard.conf
BACKEND="/usr/lib/x86_64-linux-gnu/sshg-fw-nft-sets"

on Raspberry Pi

BACKEND="/usr/lib/aarch64-linux-gnu/sshg-fw-nft-sets

Start sshguard

sudo systemctl enable sshguard
sudo systemctl restart sshguard
sudo systemctl status sshguard

Soon you will see blocked IP addresses when you list the NFT ruleset

sudo nft list ruleset

Make chroot jail sftp users that can download and upload only from a folder in their home directory

To enable SFTP with chrooted users you need to manually edit sshd_config with the correct settings that disables shell login and then manually add chrooted users.
Link: How to use SFTP with a chroot jail

Change these settings in /etc/ssh/sshd_config

nano /etc/ssh/sshd_config

Put a comment # before this line

/usr/lib/openssh/sftp-server

#Subsystem      sftp    #/usr/lib/openssh/sftp-server

Add this line

Subsystem sftp internal-sftp

add the group sftpusers

Match Group sftpusers
ChrootDirectory %h
ForceCommand internal-sftp
AllowTcpForwarding no
X11Forwarding no

Make a download user called user1 that can download and upload to only /home/user1/data

sudo useradd -g sftpusers -s /sbin/nologin -m -d /home/user1 user1
sudo passwd user1
sudo chown root:root /home/user1
sudo chmod 755 /home/user1/
sudo mkdir /home/user1/data
sudo chown user1:sftpusers /home/user1/data
sudo chmod 755 /home/user1/data
sudo systemctl restart sshd.service

You can also use shell scripts like these made by Matthieu Petiteau on github to do this faster.


Updating the server regularly

It is important to update the server regularly if you are running an unmanaged server.

sudo apt update
sudo apt upgrade

More info: here

Make a user with sudo privileges instead of using root

When Ubuntu is installed you have root access and can log on to the server in a SSH shell with the root account. For better security make a user with sudo privileges that is used instead of root.

sudo adduser newuser

usermod -aG sudo newuser

Source: How To Add User To Sudoers & Add User To Sudo Group On Ubuntu

Disable PermitRootLogin yes from /etc/ssh/sshd_config

When you have a sudoer user you can disable ssh root login. Comment PermitRootLogin yes in sshd_config to disable root access.

sudo apt install nano
sudo nano /etc/ssh/sshd_config
#PermitRootLogin yes


Restart the sshd service
sudo systemctl restart sshd.service 

Backup or copy data to another linux server with rsync over ssh

More info on the commands here. Sync a directory and subdirectories to another server.

rsync -aP /home/user/data/ download1@1.1.1.1:/home/download1/data

Other things you can do on a linux server

Logging on to the server with SSH and downloading a DCP folder or videofiles from another SFTP/FTP server

Sometimes it can be nice to download something directly from ftp servers and filemail via ftp to the server. In terminal you can use SSH to login to the server and use the command screen to detach from the session so you can resume it if the connection to the server is broken.

ssh user@111.111.111.111


If I want to download something to a folder /home/user/download/newfolderwiththings.

cd /home/user/download
mkdir newfolderwiththings
sudo apt update
sudo apt install ncftp
sudo apt install screen
screen
ncftp -u username 111.111.111.111

When you have started ncftp you can use get with the recursive command -R to download a folder and subdirectories. And use ls to list the content and cd to change directories,

ls
cd folder
get -R /folderwithhings

detach from the screen by using control + a,d

when connecting to the ssh server again use screen -r to attach to the session again.

screen -r 

If you have started many screen sessions you will get a list of sessions so you can choose which to start. To exit a screen session you can use

exit

Compress and uncompress files on the server

You can make a 7z compressed archive of files on the server so it would be faster to download them. You can also upload an archive and then uncompress it on the server. You can also split the archive in parts so you can download many parts at once. You can use screen to be able to log of the ssh session while the files are being compressed.

Install 7z

sudo apt update
sudo apt install p7zip p7zip-rar p7zip-full 

Compress a folder and it’s subdirectories

7z a -r directory.7z /directory

Compress a file

7z a archive.7z file.wav

Compress a directory and sub directories. Normal compression. Useful for wav files and similar files that will compress to half the size. Split in 3 GB parts. Cyberduck default setting is to use segmented download, but it can speed up uploads and downloads to split files in parts.

7z a -r -v3000m directory.7z  /directory

Compress file. Using -mx0 for no compression. Split in 3 GB parts . Useful if you want to make the archive faster and for files that don’t compress much like prores video files.

7z a -mx0 -v3000m archive.7z  prores.mov

You can also upload 7z archive files and extract them on the server.
Extract archive recursively and keep the subdirectories

7z x archive.7z

Making a MD5 checksum file of files in a directory.

.You can check the MD5 checksum of a file on the server with the command

md5sum file

To be able to test the integrity of folder with files you can have md5 text checksum files. You can make a checksum.md5 file with the md5 checksum of files in a directory like this

md5sum * > checksums.md5

To check the checksum of the files in a directory with the md5 checksum file you can use .

md5sum -c checksums.md5

Mount a remote directory with sshfs and fuse.


Sometimes it can be handy to change the name of a DCP that had already been uploaded. Or do other things with files on the server as if they were on your local computer . With sshfs and fuse you can mount a folder on the server as a local folder and open it in Easydcp Creator and change the name or other metadata like the content kind or the offset on reels. Easydcp Creator saves the new metadata to the folder on the server.
To mount the folder from the server locally on my mac I use the commands from here

Install Homebrew

Install osxfuse and sshfs in the terminal with these commands

brew cask install osxfuse
brew install sshfs

Reboot.
Make a local directory that the folder will be mounted in. In Terminal I made a directory called server on the desktop.

cd Desktop
mkdir server



To mount the folder on the server you use the sshfs command (link man sshfs)

sshfs [user@]host:[dir] mountpoint [options]

This is how you mount the home folder of the user yourname

sshfs yourname@111.111.111.111:/home/yourname /Users/yourname/Desktop/server

To open the DCP in the Easydcp Creator I drag the folder to the Easydcp window and wait a little bit. You can also open a Resolve or Premiere project this way.


To unmount you can use the umount command

sudo umount /Users/yourname/Desktop/server


Converting a video file or sound file on the server with ffmpeg


Converting a high-quality video file to a low-res version on the server so it can be downloaded easier. Example: You need a file to check subtitles. Or a small file to upload as a screener.
If you need a small size h264 video file of a file on the server you can use ffmpeg to convert it.

sudo apt update
sudo apt install ffmpeg 
ffmpeg -i bigfile.mov smallfile.mp4
ffmpeg -i bigfile.wav smallfile.aac