hosted by liquidweb


Go Back   Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : How to manage massive file storage needs for youtube clone?
Reply

Forum Jump

How to manage massive file storage needs for youtube clone?

Reply Post New Thread In Hosting Security and Technology Subscription
 
Send news tip View All Posts Thread Tools Search this Thread Display Modes
  #1  
Old 11-18-2007, 06:38 AM
Gigaron Gigaron is offline
Aspiring Evangelist
 
Join Date: Nov 2007
Posts: 353

How to manage massive file storage needs for youtube clone?


I'm interested in knowing how to handle the file storage of a youtube clone?

Is it just a matter of getting more servers with a few hdds or are there specialized companies that one can upload files over a distributed file streaming network?

The reason I ask is because I have thousands of gigabytes of videos and it appears to be impossible to upload it on 1 dedicated server or even a few.



Sponsored Links
  #2  
Old 11-26-2007, 02:33 AM
flashwebhost flashwebhost is offline
Web Hosting Master
 
Join Date: Jan 2002
Posts: 1,362
If you need lot of space, you can use content delivery network, but that will be costly for start up site. Other option is using amazon s3 storage. Video on your own server give more freedom, you can access with ssh and move if needed.

__________________
WebHostingNeeds.com

  #3  
Old 11-26-2007, 08:05 PM
greg-moriss-ec2 greg-moriss-ec2 is offline
WHT Addict
 
Join Date: Sep 2006
Posts: 141
At least a file server (NAS/SAN setup) for the video files.
CDN is an option but costly, the cheapest way to start up is setting up some sort of file server in the same place you got your media/web server.

Sponsored Links
  #4  
Old 11-27-2007, 03:35 AM
rSyncTechnologies rSyncTechnologies is offline
Newbie
 
Join Date: Jul 2007
Posts: 21
I think NAS/SAN is the only way...

  #5  
Old 11-27-2007, 11:02 PM
Schwagoo Schwagoo is offline
Junior Guru Wannabe
 
Join Date: Mar 2004
Posts: 52
Yes I would look into the Amazon S3 system. It will allow you to grow the storage infinitely and pay only for what you use in storage + bandwidth.

  #6  
Old 11-27-2007, 11:17 PM
Adam-AEC Adam-AEC is offline
Web Hosting Master
 
Join Date: Feb 2003
Location: Canada
Posts: 945
Quote:
Originally Posted by Gigaron View Post
I'm interested in knowing how to handle the file storage of a youtube clone?

Is it just a matter of getting more servers with a few hdds or are there specialized companies that one can upload files over a distributed file streaming network?

The reason I ask is because I have thousands of gigabytes of videos and it appears to be impossible to upload it on 1 dedicated server or even a few.
NAS/SAN is probably your best solution. There are companies that would specialize in this type of setup but generally you would have a NAS that is fed by 2/3/4 webservers, with a load balancer in front of them proxying requests.

Many other logistics will come in to play, such as the # of files per directory, etc.

  #7  
Old 11-28-2007, 12:03 AM
Dennis Nugent Dennis Nugent is offline
Account Suspended
 
Join Date: Oct 2004
Location: Nevada
Posts: 887
If you want to go cheap, and arent moving much traffic, install a PERC card in your dedicated server and connect it to a Dell (or other) RAID array. You can do 4-5 TB per array without issue for low traffic.

The issue is that storing lots of video files is a money losing proposition.

We have one client with 12 TB of video files, however, they are using multiple servers and multiple RAID arrays. It gets expensive when you start scaling for traffic.

Reply

Related posts from TheWhir.com
Title Type Date Posted
Egnyte Launches Program to PRISM-Proof Enterprise Cloud File-sharing Web Hosting News 2013-09-27 13:45:48
Box Doubles Storage for Free Plan, Launches $5 Enterprise Plan Web Hosting News 2013-08-22 13:32:27
Gladinet Launches Cloud Enterprise Service to Access SAN as a Private Cloud Web Hosting News 2013-03-28 14:19:44
Dell Quietly Taps Nirvanix for New Cloud Storage Service Web Hosting News 2012-09-12 17:37:30
DreamHost-Connected Cloud Storage Spinoff InkTank Joins Linux Foundation Web Hosting News 2012-08-28 16:20:56


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Postbit Selector

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump
Login:
Log in with your username and password
Username:
Password:



Forgot Password?
Advertisement:
Web Hosting News:



 

X

Welcome to WebHostingTalk.com

Create your username to jump into the discussion!

WebHostingTalk.com is the largest, most influentual web hosting community on the Internet. Join us by filling in the form below.


(4 digit year)

Already a member?