Results 1 to 20 of 20
-
06-15-2009, 05:26 PM #1WHT Addict
- Join Date
- May 2006
- Posts
- 120
Theoretical setup for file hosting service
What would be a good way to go about formulating your hosting needs for a file hosting service?
1gpbs line on a 10x1TB server? how much ram/etc?
10 different 50mpbs lines on 10 different servers with 1TB of storage each? how much ram/etc?
I don't know what a good cut off is, or if it makes a difference, if you have one extremely powerful server with lots of bandwidth and lots of storage space, or if you run a bunch of seperate servers, or somewhere in between.
Assume for this example the site is going to be extremely popular, extremely soon, (will end up being ranked between 20,000 and 1,000 on alexa)
-
06-15-2009, 05:30 PM #2Web Hosting Guru
- Join Date
- Aug 2007
- Location
- Kernel
- Posts
- 287
Hello,
From my past experiences, file hosting sites' storage servers do not eat too much RAM. Additionally, 1 big server wouldn't make you happy because I/O is a big problem and also port. You may want to get Raid10 setup but I am sure it's still better to get seperate servers for that big site...
Best regards.<<< Please see Forum Guidelines for signature setup. >>>
-
06-15-2009, 09:03 PM #3Web Hosting Master
- Join Date
- Mar 2003
- Location
- chicago
- Posts
- 1,781
4 x 1tb drives in raid 10 any dualcore cpu 4 gigs ram for the content delivery servers you just keep adding more as you need them.
and 1 server for the main site uploading databases and so on when you grow you will probably need a dedicated server just for mysql and 1 just for uploading but to start you could get away with 2 servers at the beginning.
-
06-15-2009, 09:28 PM #4Aspiring Evangelist
- Join Date
- Sep 2008
- Location
- Melbourne
- Posts
- 405
You'll need to make sure the drives are with raid 10. As others have said RAM is not a priority, rather I would focus on a light CPU (dual core is fine). Bandwidth is larger concern as well as port speed.
-
06-15-2009, 10:00 PM #5WHT Addict
- Join Date
- May 2006
- Posts
- 120
Thanks for the info so far guys, if anyone has anything else to add please do so
-
06-16-2009, 12:00 AM #6Web Hosting Evangelist
- Join Date
- Oct 2004
- Location
- Central / Midwest USA
- Posts
- 500
What's this setup gonna cost you Brent? It seems like it's could be pretty costly. Is it going to be profitable?
Further, what host do you plan to go with? I have thought of potentially doing this big scale, but I have no idea where to start. This thread is a good help.-=={[(' Sitedesigner ')]}==-.
~Entrepreneur at heart~ Pm me.
-
06-16-2009, 04:10 AM #7Web Hosting Master
- Join Date
- Apr 2001
- Posts
- 1,045
Depends a lot on your budget and the content you are serving up. You could go with a 2 drive raid0 and serve a few hundredMbit-1Gbit, or go with a 6/8/12 drive raid setup and serve upwards of 1Gbit-2Gbit per server. There are a lot of variables, size of files, connection speed, daemons used etc.
-
06-16-2009, 04:21 AM #8Junior Guru
- Join Date
- Oct 2006
- Posts
- 188
-
06-16-2009, 05:25 AM #9The Guru!
- Join Date
- Nov 2007
- Location
- India, USA and Amsterdam
- Posts
- 2,581
RAID-0 would be risky and RAID-10 will overkill for a file hosting. RAID-5 should be best for file hosting. As far as bandwidth is concerned you need to chose a provider who has a cheaper bandwidth and have a infrastructure that will let you grow as you grow higher. For server software consider light weight web servers like lighttpd and ngnix, they can help you save on server cost.
-
06-16-2009, 02:00 PM #10Web Hosting Master
- Join Date
- Apr 2001
- Posts
- 1,045
-
06-16-2009, 07:43 PM #11Newbie
- Join Date
- Jun 2009
- Location
- San Francisco, CA
- Posts
- 17
I would separate out recent content from old content if possible.
Basically, you are going to be judged on performance because there are lots of services providing file hosting.
I would setup a server with fast IO (SAS 15K RPM drives), Hardware RAID 5 for 1.5 TB or so of data storage. This could be the recent download cache. Then, I'd setup a big beefy storage system with cheaper SATA disks probably using RAID 6, perhaps direct connected or network connected depending on growth requirements. This would store the entire catalog as well as uploads.
IO speed is typically a problem for download services and so, you should try to optimize the system for good IO.
As you scale the system, you could add additional servers for the front end, perhaps using a load balancer to distribute requests.
Management of data will get harder as you scale. Using a central storage device, e.g. SAN, DAS or NAS, will make management easier than a system where data is spread across multiple systems.
Ed
-
06-17-2009, 01:20 AM #12Web Hosting Master
- Join Date
- Jun 2006
- Location
- Europe
- Posts
- 632
filehosting = high I/O.
you can manage to deliver thousands of concurrent connections using for example nginx with ram usage below 50MB but I/O wait can go 100% easily. The more hard drives you have the better.
-
06-17-2009, 02:41 AM #13Managed Service Provider
- Join Date
- Feb 2004
- Location
- Atlanta, GA
- Posts
- 5,662
-
06-17-2009, 02:49 AM #14Disabled
- Join Date
- Apr 2009
- Location
- San Diego, CA
- Posts
- 47
Definitely with the rest of the guys on this, high I/O capabilities is an obvious must. I'd recommend Raid 5 or Raid 10 depending your budget and exact needs.
-
06-17-2009, 02:52 AM #15Web Hosting Master
- Join Date
- Jan 2002
- Posts
- 1,400
I had worked with a file hosting, that used multiple servers, each have 2 * 1 GB HDD. It is better you have multiple server as you can share load between servers, also you can add more servers as needed. You start with two servers, one for web, one fro files, add more file servers as needed. Use ligttpd or ngnix on file servers. When you get more traffic, you need to load balance web servers too.
-
06-17-2009, 03:42 AM #16Managed Service Provider
- Join Date
- Feb 2004
- Location
- Atlanta, GA
- Posts
- 5,662
we work with (host) a good number of 'high i/o' sites that offer downloads, while disk i/o is a wonderful thing I can't stress the value of cache enough. An intelligent application that can manage the distribution / caching of files will be significantly faster than one that relies solely upon faster disk iops
-
06-17-2009, 04:41 AM #17Disabled
- Join Date
- Apr 2009
- Location
- San Diego, CA
- Posts
- 47
-
06-17-2009, 04:57 AM #18Web Hosting Master
- Join Date
- Jan 2002
- Posts
- 1,400
-
06-17-2009, 03:54 PM #19Web Hosting Master
- Join Date
- Jun 2006
- Location
- Europe
- Posts
- 632
-
06-17-2009, 04:06 PM #20Disabled
- Join Date
- Mar 2009
- Posts
- 93
I'm sure caching would still be a big help on some level with file sharing (ie. the most popular files are readily accessible in cache, the system doesn't have to seek it out, with a large volume of requests, that could make a HUGE difference in access times)
In my opinion, it's all about finding the balance.. but in my experience.. AIN'T NOTHIN WRONG WITH OVERKILL!!!