Hey All,

I am currently building a website that will house large datafiles. I need to be able to store the filenames (and maybe path) in a database, as well as allow the site to grow very rapidly. The average file that will be uploaded should be in the 50-100mb range....and occasionally some files of about 1gb in size may need to be uploaded - via the web is preferred.

I would just like suggestions on servers (NT or Linux) as well as suggestions for uploading processes / scripts. Ultimatelly the uploading script should be able to show a process bar, should easily handle these large file uploads without any timeouts, and store the info in a database.

I can build everything myself - I have just never worked on anything of such a large scale.

Thanks,

HTTP uploads, in my experience, are very unreliable. Anything can happen.

The HTTP protocol, by definition, is most suitable for text and small-to-medium (1-10MB) binary data (such as Flash/Movies et al.). I would suggest the following:

1) A cleverly set-up FTP daemon that would run a processing script after an upload is finished (I.E., add the filename and metadata to the DB),
2) An ActiveX or Java applet (the latter would cater to a more diverse audience, you'd think ;)) to perform the uploads to the FTP server.

The idea would probably go like this:

1 - Visitor creates site account,
2 - Visitor goes to upload page,
3 - A unique FTP account name is created (you can use MySQL authentication with ProFTPD, for example),
4 - The file is uploaded via the Java applet (this can provide the progress bar, etc),
5 - File information is stored in the DB once the upload has been completed,
6 - The account used to upload the file is removed from the DB.

Just my 2 cents.

Thank you for the feedback.

Can anyone suggest any scripts?