Which one is better in handling static file downloads between thttpd and tux? I need to serve a few thousands concurrent downloads, each file is about 1-2 MB in size. No PHP or CGI needed.

Thanks in advance!

I use thttpd and I love it. Loads on the file servers are 0.00 at all times. They serve something like 1M files per day each box. I've never used tux, just letting you know that I am extremely pleased by thttpd.

Wow, that's impressive!

BTW, what version of thttpd you're using?

thttpd. code like that has no business being in kernelspace.

paul

Originally posted by twrs
Wow, that's impressive!

BTW, what version of thttpd you're using?

2.25b or something like that? If you want an even better file server, zeus is better. However, it may cost you some money.

Originally posted by rusko
thttpd. code like that has no business being in kernelspace.

paul

Yes, I agree with you Paul. I heard that tux produces logs in binary too which needs to be converted to standard log format for processing.

I have a few more questions about thttpd:
- is the log produced by thttpd compatible with Apache combined log format, so I can feed it to Webalizer or AwStats easily?
- if a file is being downloaded and interrupted in the middle, is thttpd recording the transferred bytes or the full size of file in the access log file?

Thanks again!

I use webalizer with thttpd. Works the same way apache would. I think it's great software, very old but still nice.

Anyone using premum thttpd?
http://schumann.cx/premium-thttpd/

The author claims it's much better than regular thttpd. What do you think?

how many downloads per day? thttpd servers here some 6-8M (small) requests/day, load is like 0.01

I vote for thttpd . It rocks :-)

ive used thttp for serving sites that have many images. It works rather well. The performance increase has always been noticible

Wow, I'm impressed. It seems there are many thttpd supporters here :D

I've decided to go with thttpd too over tux.

So no one is using premium thttpd?

i've never seen it before. personally, the fact that i was able to sit down and patch in a few extra features we needed into thttpd in the space of an hour was a big plus for me. i'm not sure i would be happy with a closed source version.

with that said, i'll probably test it and inquire about a commercial source code license if i like it to be deployed on some of the stuff we do.

tux is great. it's going to outperform thttpd. however, the performance benefit is insignificant compared to the loss of usability and the fact that you're doing very naughty stuff in kernelspace.

paul

Thanks for the input Paul!

BTW, I saw on that premium thttpd page that "Maximum Bandwidth per CPU/NIC" for thttpd is only 100Mbps while premium thttpd can support up to 1000 Mbps. Is this true?

I want to use thttpd on a server directly connected to a Gig-E so it can burst more than 100 Mbps. Is this possible?

i have been running thttpd, but then switched to tux.

thttpd wasn't 100% reliable. sometimes (once a week or so) it was stopping handling requests, so when i was off the country for a while i had to use my old good custom build apache with 900Kb of mem used per instance.

then finally i've got some time to put tux 2.2 on the box and right now it's serving all jpeg and httpd is taking care of htmls (i prefer it that way to have apache logs tiny with referrer field ON and tux logs smaller with referrer field OFF).

now the traffic/load questions.

traffic right now isn't huge, about 13Mbps, the load for this combo (tux - jpegs, apache - htmls) is lover than 0.3 and usually is around 0.05.

with thttpd it was around the same, but spiking to 2.0 sometimes.
also tux is on 80 port and apache is being fed by tux. with thttpd such config wasn't possible.

hope that helps, especially with the fact that most posters were talking about thttpd alone.

thttpd over 100mbps? Are you crazy?
thttpd uses lots of ram, and I doubt it can handle even 100mbit...

comparing to apache it barely use ram, but compared to tux it uses heaps of it.
my wild guess that for 500Mbit one will need at least 3.5-4 GB of RAM and a pretty balanced system. but i really doubt.

tux with it's zero-copy kernel-level serving system looks for me as the only candidate for lines close to 1000 Mbit if that is possible in general.
i'd suggest using a multiprocessor box for tux btw. more threads ;)

btw my box is serving about 700 simultenous connections without any bottlenecks. but at 500 Mbit (for example) you'll need a RAID i think.

after rereading the 1st post and taking in account that 1-2 MB files @ 1000 Mbps is even less files served than mine 10KB files @ 13 Mbps.

Tux on port 80 for all static files, passing all other requests to Apache on another port. Awesome speed and you have php, cgi, mod_rewrite etc covered. The Tux speed is incredible. I am sure you can serve 1 Gbit off of a single PIII if all the served files fit into the RAM

agreed .

Originally posted by aleck
after rereading the 1st post and taking in account that 1-2 MB files @ 1000 Mbps is even less files served than mine 10KB files @ 13 Mbps.

Yes, that's correct aleck. I'll not serve small files on this server, but mostly 1-2 MB files and I need to burst over 100 Mbps. Not close to 1000 Mbps, but probably 200-300 Mbps.

It will be a P4 3.2 Ghz with 2 GB RAM. So do you think thttpd can handle the load well?

most likely. My box is P4 2.6 GHz /HT with 1.5 GB dual-channel RAM.
but don't forget that in my case the average size of file is x100 smaller and, of course, easier to fit in RAM.
thttpd is caching sent files.

you can do a simple calc tho - if you're going to have 1000 concurent downloads of 1-2 MB files it will give you around 1-2 GB of RAM used for temporary cache alone.

on your place i would just do a test run, simulating the load using the 2nd server.

Most of the files are same files (not too many unique files) that are being downloaded simultaneously, so I guess there won't be any problem with the file caching in the memory, is this correct?

I think your suggestion to do a test run is good and I'll try that out. Thanks!

i think it's one-copy-per-cache, but not 100% sure about this. i doubt the creator of thttpd would choose another way.

as for the test you can even ask your hoster to give you some unused box for a couple of hours to do a stress test.

oh almost forgot.
to make a decision i made a test on a local P-MMX box with 64MB of RAM with 100Mbps network with two computers (my desktop and notebook) being the load. it helped really a lot to tweak most of the settings.

even apache can saturate 100mbit, you just have to configure your stuff right (and putting as much ram as possible into your box helps as well...)

Originally posted by freeflight2
even apache can saturate 100mbit, you just have to configure your stuff right (and putting as much ram as possible into your box helps as well...)

Yes, I bet Apache can, but to saturate a 1000 Mbps line, I don't think it's easy for Apache.

BTW, I have another question about thttpd: how do I know how many static file requests that are being processed by thttpd?

I ran some download tests and it seems there's only one thttpd process for all those requests. I'm looking for something similar with Apache's server-status mod?

Thanks again for all the info!

Originally posted by freeflight2
even apache can saturate 100mbit, you just have to configure your stuff right (and putting as much ram as possible into your box helps as well...)

not always it can. i've tried apache of course, but in my case (~13Mbps of ~10KB files) the box had always load @ ~1 (with plenty of RAM) and any operation like log processing immediately were causing a spike of load to 20-30 and making the box pretty slow. it wasn't like that with thttpd and with tux it's just perfect.
of course apache was custom built with a very limited number of modules used (only basic ones).

Originally posted by twrs
BTW, I have another question about thttpd: how do I know how many static file requests that are being processed by thttpd?

I ran some download tests and it seems there's only one thttpd process for all those requests. I'm looking for something similar with Apache's server-status mod?

Thanks again for all the info!

well, i was always checking up mrtg stats - number of open TCP/IP connections. no need to log into the box ;)

I just played around with thttpd and its throttling feature looks very handy and useful :D

but thttpd lacks mod_rewrite to block requests by user agents and mod_throttle to limit downloads by IPs (thttpd limits the speed by URLs).

Yeah, you're right on it. However I won't need mod_rewrite on this new server.

Does tux support mod_rewrite and mod_throttle?

nop. tux is really simple. it's just get-and-send server.

Originally posted by twrs
BTW, I have another question about thttpd: how do I know how many static file requests that are being processed by thttpd?

I ran some download tests and it seems there's only one thttpd process for all those requests.

err, that's the whole point. the process per request model and the resultant forking is what makes apache the slimy slob/slug it is. single-threaded daemons with async i/o or a smart multithreaded implementation using lightweight kernel threads are infinitely better from a performance point of view. and this is on linux, where process creation is relatively cheap, try windows if you want to experience the true horror of CreateProcessEx() per request ;=

paul

Originally posted by aleck
nop. tux is really simple. it's just get-and-send server.

and because it's simple, you spend the 15-20 minutes it takes to patch it to do what you want ;]

paul

exactly.

Okay, I'm now truly impressed with thttpd.
At 40 Mbps traffic, thttpd only takes less than 200 MB and load stays below 0.09. Amazing :D

Do understand what load average means -- it's the average number of runnable processes waiting for resources (usually disk or CPU). With pretty much only one process vying for resources, the load average will top out around 1.00, even if you peg the CPU.