Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : How to manage massive file storage needs for youtube clone?

[Gigaron]

I'm interested in knowing how to handle the file storage of a youtube clone?

Is it just a matter of getting more servers with a few hdds or are there specialized companies that one can upload files over a distributed file streaming network?

The reason I ask is because I have thousands of gigabytes of videos and it appears to be impossible to upload it on 1 dedicated server or even a few.

[flashwebhost]

If you need lot of space, you can use content delivery network, but that will be costly for start up site. Other option is using amazon s3 storage. Video on your own server give more freedom, you can access with ssh and move if needed.

[greg-moriss-ec2]

At least a file server (NAS/SAN setup) for the video files.
CDN is an option but costly, the cheapest way to start up is setting up some sort of file server in the same place you got your media/web server.

[rSyncTechnologies]

I think NAS/SAN is the only way...

[Schwagoo]

Yes I would look into the Amazon S3 system. It will allow you to grow the storage infinitely and pay only for what you use in storage + bandwidth.

[Adam-AEC]

Originally Posted by Gigaron

I'm interested in knowing how to handle the file storage of a youtube clone?

Is it just a matter of getting more servers with a few hdds or are there specialized companies that one can upload files over a distributed file streaming network?

The reason I ask is because I have thousands of gigabytes of videos and it appears to be impossible to upload it on 1 dedicated server or even a few.

NAS/SAN is probably your best solution. There are companies that would specialize in this type of setup but generally you would have a NAS that is fed by 2/3/4 webservers, with a load balancer in front of them proxying requests.

Many other logistics will come in to play, such as the # of files per directory, etc.

[Dennis Nugent]

If you want to go cheap, and arent moving much traffic, install a PERC card in your dedicated server and connect it to a Dell (or other) RAID array. You can do 4-5 TB per array without issue for low traffic.

The issue is that storing lots of video files is a money losing proposition.

We have one client with 12 TB of video files, however, they are using multiple servers and multiple RAID arrays. It gets expensive when you start scaling for traffic.