Posted on 3 Comments

Some tips on downloading and uploading DCPs and large files

Here are some tips on downloading and uploading DCPs and other large files like prores quicktime files that are often 100+ GB:


Downloading and uploading tips


-I recommended using a fast internet connection. (200/200 mbit/s). (Use a good network cable/router)
-I recommend using a download manager like free download manager or Cyberduck which supports resume to download large files.
-I recommend using a verification method like MD5 hash checksums of the files to be sure files are not corrupted.

Different services I use to send and receive large files (100 GB-1000 GB)

For DCPs and Prores files that will be used at film festivals I work for I use Filemail, SFTP, Amazon S3, and IBM Aspera

IBM Aspera Cloud

IBM Aspera Cloud is used on many large Prores or DCP transfers because it is fast and reliable.  It is used by Netflix, many film festivals, sales agents, and post-production services. It is the fastest way to send big files because it uses a special method to transfer files and uses a special client to download. I have reached almost 1000 Mbit on transfers. It is more expensive than filemail.

Filemail

I have used filemail to receive files on film festivals I have worked on. I usually receive around 20 films that are around 20 GB – 100 GB each.

Price
15 Dollars per month for a Filemail Business account

Speed
Filemail speed varies. You can often get high speeds with the Filemail desktop app 

Storage 
With a Filemail Business account, you can send and receive large files, up to 1000 GB. And you can store 1000 GB permanently. With an Enterprise account, you get 5 TB of permanent storage. You can buy extra storage. You can delete transfers that are downloaded and verified to free up storage. You can choose different servers in different countries to store the files.

Ease of use
The Filemail desktop app supports resume and fast transfers. You can use it to upload and download. You don’t have to have an account to use the app.  If someone sends 100 GB to my registered email they don’t need an account.  I also download files from filemail using FTP clients like Cyberduck or Filezilla  To enable FTP you go to  Company Settings –  Download page and choose enable FTP.

Verification
To enable MD5 verification you go to  Company Settings –  Download page.
If the MD5 hash matches the local file the file is verified.

Amazon S3

Amazon S3 can be used to deliver and receive big files on the internet. S3 storage is very reliable and scalable.


Price and Storage
You pay for each GB of data transfer and storage so it can be more expensive than filemail when uploading and downloading large files (100 GB) . But you only pay for when you are using the service and you can store unlimited data.

Ease of use and verification
-You can use the s3 web uploader or use Cyberduck and similar programs to upload to S3. After uploading you can make the link accessible (public) and give the clients the S3 download link. You can also do more advanced S3 stuff like making a user with a bucket with its own access and secret key that can be accessed with cyberduck and similar programs. Or set up Amazon SFTP transfer 

Speed
Amazon S3 storage is very fast.

Dedicated root SFTP Server

I have a dedicated Ubuntu root server running an SFTP server at Hetzner to send and receive DCPs and large Prores files. Having a dedicated 1000 Mbit server that you can remote control with SSH is great because you have complete control to download directly to the server via SFTP/FTP, compressing and uncompressing files. 

Price
A dedicated server with unlimited data running SFTP at hetzner can be set up on the cheapest action servers that cost around 30 euros a month.  Alternatively, you can buy a Hetzner storage box with FTP, SFTP, and Rsync over SSH with limited data transfer.

Storage
Hetzner auction servers can have 14 TB storage. In this post, I install ubuntu on a server with 14 TB of storage in a 4 HDD RAID level 5 setup with 1 redundant hard drive. 

Speed
Max speed for all transfers is 1000 Mbit. The speed for each SFTP transfer will vary with the SFTP client and location. Cyberduck uses segmented downloads so you can reach high speeds. 

Ease of Use
Using Cyberduck to upload and download from SFTP server is easy.  I wrote this post about setting up basic security and some other things you can do on the server. I wrote this post about installing sftpgo that has an easy to use web admin UI.

Verification
You can ask uploaders to add a text file with the md5 hash values and verify the files when they have been downloaded. 

Tools

7z or RAR archive utilities 
-If you want to send a uncompressed quicktime file, TIFF or DPX image sequences or wav files I recommend using RAR or 7z with normal compression if you have the time to do compression. A uncompressed quicktime file will be reduced to half the size after RAR or 7z compression.
-I recommend Keka on mac and winrar on Windows.
-7z and RAR archives can be protected with encryption and a password and have a built in hash check.
-In Keka, choose 7z and store for no compression or normal for normal compression, then drag the folder to the Keka window.
-You can also split the archive in parts and upload the file in parts. Example 20 GB. If one of the parts is corrupted you only need to upload that part again.

Verification tools
-DCPs have the checksums for the files of the DCP in the XML files that can be verified with easydcp player or using this command
openssl sha1 -binary "FILE_NAME" | openssl base64 
If the sender sends a 7z archive, the receiver will know when unpacking if the file fails the 7z checksum test.
-When sending prores quicktime files and other files without built-in verification, it is common practice in post production that the sender include a md5 checksum to enable verification of the file.


Generating a MD5 checksum

The client can find the MD5 checksum of a file with utilities like Hash Tab for windows and Hash Tab for mac

You also use the command md5 "file" in a terminal on a Mac to generate a md5 file hash value. The Terminal app is in Applications – Utilities. You can type md5 and push space, then drag the file from a finder window and push enter.

You can use the command certutil -hashfile "file" md5 on a Windows 7, 8, 10 computer in a command prompt (CMD)

You can use md5sum "file" in a terminal or ssh on a linux computer.

DCP and quicktime specific advice

-When sending a DCP to cinemas in Norway I use my account at the DCP transfer service Movie Transit. Other countries use this and other similar services. The DCP is normally uploaded some weeks before the screening. 

-When downloading a DCP folder in Filezilla from a FTP server it is important to change Transfer – transfer type – binary. It should not be ASCII or auto, this way the XML files will pass the checksum test.

-Ask people who submit DCPs to either upload the folder or the archive format RAR or 7z instead of zip. This way you can avoid the (large) Mac OS archive utility zip files that can only be opened by the Mac OS archive zip utility. Both Keka and 7-Zip can make 7z archives.

-I recommend checking DCPs in Easydcp player. The trial version will let you know if the DCP fails some checks. You can also run a file hash check.

You should also make a DCP quality checklist that may include:
-Is the DCP named correctly?
-Is it in the correct scope or flat format? The DCP should have a correct digital naming convention name. Example: The DCP is called Dcpname_SHR_S but is actually Dcpname_SHR_F because the resolution is wrong and the subtitles are burned in too low and will be cropped at a scope preset in your cinema.
-Is the DCP in the correct language/subtitle language?
-Will the DCP play at your venue. Example: The DCP is marked 7.1 and you only got 5.1

-If you have downloaded a DCP folder archive like DCP.7z on a mac and want to ingest in on a digital cinema media block/server/player you can extract it to a DX115 hard drive or similar.

Other tips

-If you get an error message extracting a zip file in Keka that says error code 2 using p7zip, you probably need to extract the zip file with mac os archive utility.

3 thoughts on “Some tips on downloading and uploading DCPs and large files

  1. Hi Knut.
    Great work! Just the info I needed to solve my problems transporting DCP files over the net.
    Thank you.
    Best Regards Ulrik Lyhne, Aarhus Film Workshop

  2. Hi Knut, thanks for the tips.
    Quick question.. Do you know if there is a site out there with limits as high as say, filemail.com, which allow you to upload via an FTP program like filezilla?
    Alternatively, a site which has a desktop uploader which you can use with a free account?
    – I seem to have trouble sending larger files over my internet connection using a web-browser; it often fails and I have to start again.
    Cheers!

  3. Hi.
    An alternative is Amazon S3. You can use CrossFtp and similar programs to upload to S3. And the link you give clients will be very reliable.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.