In Some tips on downloading and uploading DCPs and large files I wrote about the different ways I send large files:
- Filemail
- IBM Aspera
- Amazon s3
- and a dedicated SFTP server
Here are some tips on using a dedicated SFTP server running Ubuntu 20.04/22.04.
Start the OpenSSH server
The OpenSSH server is already running on Ubuntu server, but if you are using Ubuntu Desktop you can install and start the Openssh server.
sudo apt update sudo apt install openssh-server sudo systemctl status ssh
Recommended SSH client
On Mac, I use terminal. On Windows I use Ubuntu on WSL.
To connect to the server.
ssh root@1.1.1.1
Setting up basic security
Generating good passwords
sudo apt install pwgen
Generate a 12 digit password
pwgen 12 1
Firewall
Use nftables firewall to accept only port 22 SSH/SFTP. If Nftables is not installed you can install it and start it with these commands.
sudo apt update sudo apt install nftables sudo systemctl enable nftables sudo systemctl start nftables sudo systemctl status nftables
Edit nftables.conf with modified Simple ruleset for a server to only accept SSH/SFTP on port 22 and alternative port 2222
tcp dport { 22, 2222 } accept
sudo nano /etc/nftables.conf
Restart nftables
sudo systemctl restart nftables
List the NFT ruleset
sudo nft list ruleset
Stop brute force attacks with sshguard and nftables
Install nftables if necessary
Set up sshguard
sudo apt install sshguard
Add sshg-fw-nft-sets to BACKEND= in sshguard-conf if needed
sudo nano /etc/sshguard/sshguard.conf
BACKEND="/usr/lib/x86_64-linux-gnu/sshg-fw-nft-sets"
on Raspberry Pi
BACKEND="/usr/lib/aarch64-linux-gnu/sshg-fw-nft-sets
Start sshguard
sudo systemctl enable sshguard sudo systemctl restart sshguard sudo systemctl status sshguard
Soon you will see blocked IP addresses when you list the NFT ruleset
sudo nft list ruleset
Make chroot jail sftp users that can download and upload only from a folder in their home directory
To enable SFTP with chrooted users, you manually edit sshd_config to disable shell login. And then manually add chrooted users.
You can also use shell scripts like these made by Matthieu Petiteau on GitHub to do this faster.
Change these settings in /etc/ssh/sshd_config
nano /etc/ssh/sshd_config
Put a comment # before this line
/usr/lib/openssh/sftp-server
#Subsystem sftp #/usr/lib/openssh/sftp-server
Add this line
Subsystem sftp internal-sftp
add the group sftpusers
Match Group sftpusers ChrootDirectory %h ForceCommand internal-sftp AllowTcpForwarding no X11Forwarding no
Make a download user called user1 that can download and upload to only /home/user1/data
sudo groupadd sftpusers sudo useradd -g sftpusers -s /usr/bin/false -m -d /home/user1 user1 sudo passwd user1 sudo chown root:root /home/user1 sudo chmod 755 /home/user1/ sudo mkdir /home/user1/data sudo chown user1:sftpusers /home/user1/data sudo chmod 755 /home/user1/data sudo systemctl restart sshd.service
Updating the server regularly
It is essential to update the server regularly if you are running an unmanaged server.
sudo apt update
sudo apt upgrade sudo apt dist-upgrade
Make a user with sudo privileges instead of using root
When Ubuntu is installed, you have root access and can log on to the server in an SSH shell with the root account. For better security, make a user with Sudo privileges that is used instead of root.
sudo adduser newuser usermod -aG sudo newuser
Getting a root shell when logged in as a user with sudo privileges
sudo su -l
Disable PermitRootLogin yes from /etc/ssh/sshd_config
When you have a sudoer user, you can disable ssh root login. Comment PermitRootLogin yes in sshd_config to disable root access.
sudo apt install nano sudo nano /etc/ssh/sshd_config
#PermitRootLogin yes
@Restart the sshd service
sudo systemctl restart sshd.service
Backup or copy data to another linux server with rsync over ssh
Sync a directory and subdirectories to another server.
rsync -aP /home/user/data/ download1@1.1.1.1:/home/download1/data
Other things you can do on a Linux server
Moving a folder and change the permission of the folder to another user
sudo chown -R sftpgo:sftpgo /home/olduser/
sudo mv -R /home/olduser/ /srv/sftpgo/data/newuser/
Logging on to the server with SSH and downloading a DCP folder or video files from another SFTP/FTP server
Sometimes it can be nice to download something directly from FTP servers to the server. In Terminal, you can use SSH to log in to the server and use the command screen to detach from the session so you can resume it if the connection to the server is broken.
ssh user@111.111.111.111
If I want to download something to a folder /home/user/download/newfolderwiththings.
cd /home/user/download
mkdir newfolderwiththings
sudo apt update sudo apt install ncftp
sudo apt install screen
screen
ncftp -u username 111.111.111.111
When you have started ncftp you can use:
- get -R to download a folder and subdirectories.
- ls to list the content
- cd to change directories.
ls
cd folder
get -R /folderwithhings
Detach from the screen by using control + a,d
When connecting to the ssh server again, use screen -r to attach to the session again.
screen -r
If you have started many screen sessions, you will get a list of sessions so you can choose which to start. To exit a screen session, you can use
exit
Compress and uncompress files on the server
You can make a 7z compressed archive of files on the server so it would be faster to download them. You can also upload an archive and then uncompress it on the server. You can also split the archive into parts so you can download many parts at once. You can use screen to quit the ssh session while the files are being compressed.
Install 7z
sudo apt update sudo apt install p7zip p7zip-rar p7zip-full
Compress a folder and its subdirectories
7z a -r directory.7z /directory
Compress a file
7z a archive.7z file.wav
Compress a directory and subdirectories. Normal compression.
Useful for wav files and similar files that will compress to half the size. Split into 3 GB parts. Cyberduck’s default setting is segmented downloads, but it can speed up uploads and downloads to split files into parts.
7z a -r -v3000m directory.7z /directory
Compress file. Using -mx0 for no compression. Split into 3 GB parts. Useful to make the archive faster and for files that don’t compress much like prores video files.
7z a -mx0 -v3000m archive.7z prores.mov
You can also upload 7z archive files and extract them on the server.
Extract archive recursively and keep the subdirectories
7z x archive.7z
Making an MD5 checksum file of files in a directory
You can check the MD5 checksum of a file on the server with the command
md5sum file
To test the integrity of a folder with files, you can have md5 text checksum files. You can make a checksum.md5 file for files in a directory like this:
md5sum * > checksums.md5
To check the checksum of the files in a directory with the md5 checksum file, you can use
md5sum -c checksums.md5