Scott on Writing

Musings on technical writing...

Transferring Large Files

Every couple of weeks or so I seem to have a need to transfer (or receive) a relatively large file to another colleague or client. It might be a database zipped up into a 50 MB file, artwork totalling hundreds of megabytes, or various DLLs and source code. Over the course of my career I have used the following techniques for transferring data:

  • E-mail - the most straightforward technique for sharing data. The major con is that many email providers reject attachments over a certain threshold, usually in the 2-10 MB range. Also, e-mail is inherently insecure, making it a non-ideal medium for transferring sensitive data like passwords, credit card numbers, social security numbers, and so on.
  • Upload/Download from a Website - if you have write access to a website you can always upload the data to share and then send a URL to the file(s). The downside here is that you need a website. Security also requires a few extra steps - you need to have some mechanism to only allow the intended recepient from downloading the article.

    Side story: Back in 2002 Microsoft released Web Matrix, which later became Visual Web Developer Express Edition. Prior to announcing the release Microsoft and asp.netPRO Magazine asked me to write an overview of the tool to be included in the asp.netPRO edition that came out right when the product was released officially. There were a group of other ASP.NET folks at the time who also knew about this project and were invited to review my article. I distributed it by putting the article up on a website and provided this group a link to download the article. A couple days later I left for a short vacation during which I did not check e-mail or phone messages.

    The day I left town one of the reviewers leaked the article to Neowin.net, which linked directly to the URL on my website. Shortly after the link was published on Neowin.net I started getting e-mails and phone calls from Scott Guthrie asking me to take down the link. Of course, I did not get these messages until a few days later upon return from my vacation.

    I never did find out who leaked the link/article. I guess it didn't really matter in the end, and it wasn't the article on my server that caused the leak as someone else sent in the link to Neowin.net and they could have just as easily sent the Microsoft Word document. But still, I was rather red-faced when having to explain to the guys at Microsoft why the article was accessible to anyone on the Net and why it took so long to get the article down once it had been compromised.
  • Use a Third-Party Service - there are a plethora of web-based services that are built to allow large files to be transferred between two individuals. Most work by you uploading a file to a site, where it is stored temporarily. You then e-mail the recipient a link to download the attachment. I have used the free service TransferBigFiles.com many times and heartily recommend them. You could also use DropBox and place the file to share in the Public folder. The benefit of this approach is that you don't need a website; however, the security concerns remain. In fact, you might consider the security concerns even (slightly) greater because you are now adding an intermediary into the equation (the folks at TransferBigFiles.com or at DropBox).
  • Physically Deliver the Data - the most secure option is the physically deliver the data - put it on a thumb drive or burn it to a DVD and then mail or hand deliver the data. This approach takes the most work, but is the most secure, especially if you hand deliver the data. And if you are transferring extremely large amounts of data - terrabytes, let's say - this approach would likely be the fastest.

    I used this approach 8 years ago when transferring large amounts of information between myself and a publisher. At the time I only had dial-up access. When the publisher needed to send me several hundred megabytes worth of materials - mostly artwork from previous editions - it was more efficient to mail me a CD. I also had a student in one of my Web Services class, must have been four years ago or so, whose company was running computer simulations that would output several GBs worth of data per test run. To send the day's test data to another facility they would plug in an external hard drive, copy over the data, and then use a mail carrier to overnight deliver the drive. I wonder what techniques they use today.

How do you share large files with others?

posted on Friday, September 19, 2008 1:29 PM

Feedback

# re: Transferring Large Files 9/19/2008 5:01 PM John Davidson

It looks live Live Mesh with Windows Live Id may become a very good secure solution, even without a website, as this will work with any Internet connected PC. It seems to work through normal Firewalls, and may work well with some corporate firewall implementations. Look at the Shared Folder and invites to users for how to make best use of this.

# re: Transferring Large Files 9/19/2008 7:23 PM Franck Quintana

I use my machine as a website (i have a dyndns).
So from my part I just copy paste to a virtual folder.
Then I send the link to the customer and ask him to notify me when the file is downloaded (so I can delete it).

# re: Transferring Large Files 9/20/2008 1:13 AM Joannes Vermorel

If you can afford a synchronous delivery with your colleague or client, then IM such as Skype are great to transfer large files.

Bonus point, it works even if the other person has not top notch IT skills.

# re: Transferring Large Files 9/21/2008 5:05 AM Carsten Berggreen, Monolith-Systemsm, Denmark

We have plenty of ways to get a big file from or to a customers drive.

1. Normal old fashion FTP for the technically none paranoide customers.

2. PANDO (Pando.com) is a free tool, perhaps not the quickest, but its free up til 1 gb and its very easy to use. (and works on both PC and Mac)

3. WebDav folders on our own webservers.


Anyway, we believe in customer service, so we go in, remotecontrol their desktop and install the 3rd party tool, if they want us to do it.

That gives us the ability to see the errors if any, and secondly it makes it more relaxing for the customer - as "an expert is doing their job" ;-)

Usefull answer?

# re: Transferring Large Files 9/21/2008 2:01 PM Matthew Martin

7z compress the file on ultra mode. Put filezilla client at one end of the wire (or GetRight or any other ftp client that supports resume), filezilla server at the other end (or any other ftp server that supports gzip and resume) If sending site and receiving site have firewalls and restrictions on software installation, switch to portable apps and a server in the middle, such as a home machine. Encrypt with the 7z file with AES and long passwords. Actually faster than dealing with multiple lost files sent by email.

# re: Transferring Large Files 9/21/2008 7:47 PM Ben

A new service just released is 2Large2Email.com - You can sign up for a free account and send large files easily. There are also paid accounts if you need to send really big files or want file storage and access to lots of users.

# re: Transferring Large Files 9/23/2008 1:04 AM filemail.com

Hi,
I have started to use filemail.com. Their site are really simple to use and the security is very good. As an unregistered member you can send up to 2GB. Worth checking out guys!

# re: Transferring Large Files 10/6/2008 9:47 AM yousendit.com

YouSendIt has worked well for me as a 3rd party site. They'll keep your posting up for 7 days. You also have the option of purchasing a Pro account.

Title:  
Name:  
Url:
Protected by Clearscreen.SharpHIPEnter the code you see:
Comments   

My Links

Ads Via DevMavens

Archives

Post Categories

 

I am a Microsoft MVP for ASP.NET.
I am an ASPInsider.
<November 2008>
SMTWTFS
2627282930311
2345678
9101112131415
16171819202122
23242526272829
30123456

Comment Stats

DayTotal% of Total
Sunday 1986.9%
Monday 40314.1%
Tuesday 48216.8%
Wednesday 53118.5%
Thursday 55819.5%
Friday 51918.1%
Saturday 1776.2%
Total 2868100.0%

Hour1Total% of Total
12:00 AM 712.5%
1:00 AM 762.6%
2:00 AM 652.3%
3:00 AM 772.7%
4:00 AM 612.1%
5:00 AM 1154.0%
6:00 AM 1113.9%
7:00 AM 1645.7%
8:00 AM 1806.3%
9:00 AM 1545.4%
10:00 AM 1786.2%
11:00 AM 1846.4%
12:00 PM 1966.8%
1:00 PM 1796.2%
2:00 PM 1625.6%
3:00 PM 1344.7%
4:00 PM 1133.9%
5:00 PM 1003.5%
6:00 PM 973.4%
7:00 PM 1033.6%
8:00 PM 893.1%
9:00 PM 832.9%
10:00 PM 853.0%
11:00 PM 913.2%
Total 2868100.0%

Comments by Blog Entry Date/Time

Day Entry MadeAvg.Total
Sunday 5.18145
Monday 5.10362
Tuesday 4.28462
Wednesday 7.51653
Thursday 6.72645
Friday 5.32431
Saturday 5.00170
Total 5.682868

Hour1 Entry MadeAvg.Total
12:00 AM 5.0035
1:00 AM 1.002
5:00 AM 0.000
7:00 AM 6.3338
8:00 AM 4.72118
9:00 AM 6.04284
10:00 AM 6.12257
11:00 AM 4.27188
12:00 PM 6.75344
1:00 PM 3.05122
2:00 PM 5.29222
3:00 PM 8.60301
4:00 PM 3.7694
5:00 PM 5.86164
6:00 PM 4.56114
7:00 PM 9.15183
8:00 PM 8.42160
9:00 PM 5.00115
10:00 PM 6.3395
11:00 PM 4.5732
Total 5.682868

Learn More About Comment Stats
1 - All times GMT -8...


Blog Stats

Favorite Web Sites

My Books

My MSDN Articles