Web Hosting Talk : Web Hosting Main Forums : Dedicated Server : Config to transfer 1000GBs in one day

[dinek]

Hi,

I have a client who wants to transfer 1 000 000 1mb files in one day. This equals to 1000GBs of data and based on the usage graphs they sent me in peak hours I would need around 150-160mbit/s. It is difficult for me to say what servers should I use. I guess I should use at least 2 servers in different locations (redundancy). The client can split the loads to specific servers. It will be progressive download. e user base will be anything from modem users (long downloads) to DSL lines.

Any suggestions you might have are very welcome.

Thanks. Dino.

[JHServers]

I'd suggest using thttpd if these are http downloads. thttpd is very light and is supposedly capable of sustaining 100 mbps per server. If you can afford it, zeus can supposedly sustain 1000 mbps per server using http. Good luck with this.

[Erich]

I would spread it on 4 servers, ideally at 2 different hosts, and balance the load with round robin, the poor man's load balancing.

If these 1 Million files equal 1 Million hits (and not a LOT more hits, which is quite possible), then plain Apache should serve it just well on 4 P4 servers

Note that both thttpd and Tux do not have referer based bandwidth protection as provided by Apache's mod_rewrite

(workaround is possible though)

[dinek]

Hi,

Thanks for your replies.

I would have 2 locations maybe 3.

Now as to the server config. Should I have SCSI drives or plain IDE or SATA drives would be enough? I guess 1GB of ram per server would be enough yes?

Thanks.
Dino

[dkitchen]

Hi,

Really depends what these files are, if you're talking plain download files, any server with good connectivity should be able to handle that easily. Always remember that 100mbit will very rarely give you even near that, it's more like 80mbit by the time you take switches, routers, etc into account.

RAM/CPU really isn't an issue here, it's getting the files off the disks, and across the internet as fast as possible. For that you'll need SATA/SCSI disks, preferably in RAID0.

However if this is a small file (1mb as you say), buy an average disk with a large-ish cache, and you'll find the speed will be pretty good.

I'd host the file on 3 or 4 servers, and then do a round robin DNS config so that a server is chosen at random, and you have a small but effective load balancing solution.

I've worked with Zeus in a number of places for a while now, and it does seem pretty capable, allthough I can't see much speed difference over a well optimised Apache server really.

Dan

[SMachiz]

Is it one file being transferred 1,000,000 times? If so, you can skip all the fancy hardware and just set up a ramdisk on the machine with the file in it, and it'll be plenty fast.

Sam

[amc-james]

This makes me wonder why no one is using any kind of webserver that servers files stored in RAM. No disk activity would be necessary except the initial read of the file.

Someone get started right away!

[KarlZimmer]

I'm pretty sure there are current Apache modules you can use that would just store the file in RAM and thus greatly reduce disak accesses. It's a great setup for single smaller files that get downloaded a lot.

[jmcgon]

linux keeps files in ram after it reads them and then overwrites them or places them in swap when they are no longer needed. Thats what the +/- cache is when you look at your memory usage with free.

[crucialx]

I'd say go with the file in RAM, then have 2-4 servers setup in DNS round robin. I believe you could just setup a RAM drive for the file.

[Steven]

Originally posted by KarlZimmer
I'm pretty sure there are current Apache modules you can use that would just store the file in RAM and thus greatly reduce disak accesses. It's a great setup for single smaller files that get downloaded a lot.

or setup a ramdisk.

[LP-Trel]

or a BSD mfs.

[blueface]

Well, actually you can just use a single P4 server with thttpd, 1 or 2 GB RAM and SATA HDD. thttpd will cache the frequently downlaoded files in the memory and sustain even a few hundred mbps. No need to do load balancing at all unless you're close to using 300 mbps constantly.

[dinek]

Thank you ALL for your suggestions. Now I know how to approach this problem.

Best. Dino.

[paladinstrike]

Originally posted by blueface
No need to do load balancing at all unless you're close to using 300 mbps constantly.

Of course, if you are transferring that much with just one server you will have to pay for a 1000mbps connection to it, which can be very expensive, sometimes even more than getting additional servers at 100mbps depending on the host.