Some tips on downloading and uploading DCPs and large files

Here are some tips on downloading and uploading DCPs and other large files like prores quicktime files:


Downloading

-I recommended using a fast wired internet connection. (50/50 mbit/s).
-I recommend using a download manager like Free download manager  (Windows/mac) that support resume and multi-part downloading.

Uploading
-I recommended using a fast wired internet connection. (50/50 mbit/s).
I recommend a reboot of the computer and a restart of chrome before uploading large files (100 GB).

Filemail

On filemail.com you can send 30 GB files for free.

-With a Filemail Pro or Business account you can send and receive 100-200 GB files.

-I have used the filemail html5 uploader on many 100 gb files. You can also use the html5 uploader to send folders. The Desktop filemail app also works well on 100 gb files,

-With a Filemail business account you will have access to the files for longer and people can use a branded upload web page. You can also the md5 hash value.

Filemail md5 hash check

Filemail shows the md5 checksum on the download page if you enable it in the settings for the business download page. This way you can check if the uploaded file is correct. You can find the MD5 checksum of a file with utilities like Hash Tab for windows and Hash Tab for mac
You also use the command md5 "file" in a terminal on a Mac to generate a md5 file hash value.
You can use the command certutil -hashfile "file" md5 on a Windows 7, 8, 10 computer in a command prompt.
You can use md5sum "file" in a terminal on linux computer
If the sender checks that the MD5 checksum is the same locally as it is on the filemail download page you will know that the file is OK.

Amazon S3

Amazon S3 can be used to deliver and receive big files on the internet.
-You can use the s3 web uploader or use CrossFtp and similar programs to upload to S3. Crossftp Pro can use multipart and integrity checks when uploading so you will know the uploaded file is working.
-After uploading you can make the link accessible (public) and give the clients the S3 download link.
-S3 storage is very reliable and scalable. You pay for each download.
-If you want a client to upload a file to a s3 upload bucket you can make an upload user in https://console.aws.amazon.com/iam/home#users with a policy that enables the user to upload to that upload bucket.

Like this policy:

From http://docs.aws.amazon.com/AmazonS3/latest/dev/example-policies-s3.html#iam-policy-ex0:
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Action":[
"s3:ListAllMyBuckets"
],
"Resource":"arn:aws:s3:::*"
},
{
"Effect":"Allow",
"Action":[
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource":"arn:aws:s3:::examplebucket"
},
{
"Effect":"Allow",
"Action":[
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource":"arn:aws:s3:::examplebucket/*"
}
] }

– then give the client the access key and secret key for the upload user. (with this policy the user will see all buckets, but can only upload to the examplebucket bucket.

 

Send 7z archive files

-I ask people to make a 7z archive of the quicktime file or DCP folder before they send it. I recommend Keka on mac or 7zip on Windows.
-7z archives can be protected with encryption and a password and have a built in hash check.
-The 7z file can be uncompressed when sending prores files but if you want to send a uncompressed quicktime file or text files I suggest using normal compression. A uncompressed quicktime file will be reduced to half the size after 7z compression.
-In Keka, choose 7z and store, then drag the folder to the Keka window
-You can also split the archive in parts and upload the file in parts. Example 20 GB.
keka

Use a PAR2 recovery file as backup
-You can use a MultiPar or a MacPAR Deluxe PAR 2 recovery file with 1-5 percent redundancy. If there is an error in the uploaded file you can probably fix it. PAR2 files with 1 percent redundancy are 1 percent of the size of original file. If the downloaded file is 0,5 percent corrupted, you can use the 1 percent PAR 2 recovery file to recover the file instead of downloading again.

Multipar with 5 percent redundancy.

multipar

multiparopen

 

Macpar Deluxe with 3 percent redundancy.

Choose File – New Par 2 set

Drag the file and save.

macpardeluxe

 

DCP and quicktime specific advice

-When downloading a DCP folder in Filezilla from a FTP server, I downloaded the XML files and the rest of the files as binary to get them to pass the file hash check in Easydcp Player.

-Ask people who submit DCPs to use the archive format 7z instead of zip. This way you can avoid the (large) Mac OS archive utility zip files that can only be opened by the Mac OS archive zip utility. Both Keka and 7-Zip can make 7z archives.

-Sometimes I get a error message when opening a quicktime file on windows pc that was made on a mac. If I rename the file it works.

-I recommend checking DCPs in Easydcp player. The trial version will let you know if the DCP fails some checks. You can also run a file hash check.

You should also make a DCP quality checklist that may include:
-Is the DCP named correctly?
Is it in the correct scope or flat format? The DCP should have a correct digital naming convention name. Example: The DCP is called Dcpname_SHR_S but is actually Dcpname_SHR_F because the subtitles are burned in too low and will be cropped at a scope preset in your cinema.
-Is the DCP in the correct language/subtitle language?
-Will the DCP play at your venue. Example: You may have a server with old firmware that can play 25 SMPTE DCPs with burned in subtitles, but not SMPTE 2010 subtitles. When I work for film festivals I only work with updated servers that will accept SMPTE 2010 subtitles, (probably all film festivals have updated firmware).

-If you have downloaded a DCP folder archive like DCP.7z on a mac and want to ingest in on a digital cinema media block/server/player you can extract it to a DX115 hard drive or similar.

Other tips

-If you get an error message extracting a zip file in Keka that says error code 2 using p7zip, you probably need to extract the zip file with mac os archive utility.

4 Replies to “Some tips on downloading and uploading DCPs and large files”

  1. Another tip: Dropbox downloads can be flaky (hence the download manager suggestion)!
    I’ve had a digital lab download a zipped DCP from my Dropbox. They told me there were hash issues with the DCP. I did a hash check after downloading the same file (using the EasyDCP Player demo) and had no issues. Then after Googling I found out that Dropbox downloads thru Explorer and other browsers could yield file-integrity issues! And I had just purchased a Pro account. A WeTransfer download had no issues at the lab.

  2. Hi Knut.
    Great work! Just the info I needed to solve my problems transporting DCP files over the net.
    Thank you.
    Best Regards Ulrik Lyhne, Aarhus Film Workshop

  3. Hi Knut, thanks for the tips.
    Quick question.. Do you know if there is a site out there with limits as high as say, filemail.com, which allow you to upload via an FTP program like filezilla?
    Alternatively, a site which has a desktop uploader which you can use with a free account?
    – I seem to have trouble sending larger files over my internet connection using a web-browser; it often fails and I have to start again.
    Cheers!

  4. Hi.
    An alternative is Amazon S3. You can use CrossFtp and similar programs to upload to S3. And the link you give clients will be very reliable.

Leave a Reply

Your email address will not be published. Required fields are marked *