Results 1 to 15 of 15
-
05-09-2008, 09:42 PM #1Temporarily Suspended
- Join Date
- Feb 2007
- Posts
- 185
Optimizing lighttpd for large files (180mb avg)
Hello peeps,
I have four servers with a quad xeon, 4gb ram, and 2x300GB SAS 15K RAID0 harddrives, pushing a total of 1.6gbits. It serves a lot of zip files with an average flesize of 180mb. My question is, how can I optimize lighttpd 1.4.19 to push its max with very low IO-wait. I've looked up some stuff and only found options that deal with lighttpd 1.5 and use Linux-AIO for the backend network. Currently I use writev with 16 workers and an read/write idle timeout of 10s. Logging is off, too. Anything else I can do?
-
05-10-2008, 05:26 AM #2WHT Addict
- Join Date
- Jul 2004
- Posts
- 148
If its just serving files then i would switch over to thttpd - its faster for static content
-
05-10-2008, 09:35 AM #3Web Hosting Evangelist
- Join Date
- Apr 2003
- Location
- Melbourne, AU
- Posts
- 539
lighttpd 1.4 with writev can easily saturate a gig-e serving large files like yours, provided that the disk subsystem isn't holding it back
As lighttpd doesn't come with many tuning knobs, maybe you can take a step back and consider some other things to address disk performance?
- is your RAID0 stripe size large (e.g. 1MB)?
- are those server chassis 2-disk or 4-disk?
- do those servers have distinct files (from each other), or do they share the same set of files?
- how big is the working set of files served at any one time?
- is adding RAM an option?WK Woon
CTO | http://www.aflexi.net - A flexible Network
Building the next generation CDN platform - DEMO .... coming soon
-
05-10-2008, 01:03 PM #4Temporarily Suspended
- Join Date
- Feb 2007
- Posts
- 185
- I'm not sure what it is, how can I check and change it?
- 2 disk
- same set of files
- i'm not sure, how do i check?
- yes but it would be more expensive and pretty useless since the files are so large
-
05-11-2008, 02:14 PM #5Temporarily Suspended
- Join Date
- Feb 2007
- Posts
- 185
well? anyone?
-
05-11-2008, 02:55 PM #6Disabled
- Join Date
- Dec 2002
- Location
- chica go go
- Posts
- 11,876
If you're pushing 1.6gbit, then you are probably topping out your port speed. I'd try to get a second gigabit network drop, or a new server if i were you.
As for your raid array, switch to raid 5, 10, or 1.
Software, try switching to nginx instead of lighty. I ran lighty serving images for a while, and nginx outperforms it in terms of serving static content.
-
05-11-2008, 03:38 PM #7Temporarily Suspended
- Join Date
- Feb 2007
- Posts
- 185
It is aggregated bandwidth and I have 5 servers splitting the load, so each pushes around 200-350mbps. They are all mirrors so raid0 is the best for max performance, and a failed drive can easily be fixed. nginx also is great but it dosn't have the mod_secdownload module like lighttpd does.
Last edited by digitalpioneer; 05-11-2008 at 03:42 PM.
-
05-11-2008, 10:35 PM #8Web Hosting Evangelist
- Join Date
- Apr 2003
- Location
- Melbourne, AU
- Posts
- 539
- Use your card's RAID utility.
- Oh, can't add anymore disks then :\
- Currently, how are visitors directed to the four servers? Randomly? Or is each server designated to serve only a subset of the file collection?
- I would parse the output of lsof. Look for lines matching lighttpd and your docroot.
- Depends on your working set.WK Woon
CTO | http://www.aflexi.net - A flexible Network
Building the next generation CDN platform - DEMO .... coming soon
-
05-11-2008, 10:37 PM #9Web Hosting Evangelist
- Join Date
- Apr 2003
- Location
- Melbourne, AU
- Posts
- 539
Nginx is a damned spectacular piece of code and is very fast, in terms of CPU use. I use it for image webservers (where reqs/s is very high). However, in a situation where a disk is a major bottleneck and reqs/s is low, I reckon lighttpd with writev trumps it - I feel (disclaimer: no hard numbers to back me) that in practice it works better than the Linux fs cache.
WK Woon
CTO | http://www.aflexi.net - A flexible Network
Building the next generation CDN platform - DEMO .... coming soon
-
05-12-2008, 12:26 AM #10Temporarily Suspended
- Join Date
- Feb 2007
- Posts
- 185
- Currently, how are visitors directed to the four servers? Randomly? Or is each server designated to serve only a subset of the file collection?
According to lighttpd (http://blog.lighttpd.net/articles/2006/11/12/lighty-1-5-0-and-linux-aio), writev is has the lowest throughput, linux-sendfile is much faster and linux-aio-sendfile is much faster on 1.5. However, I have tested linux-sendfile and it gave me a IOwait of 15-25% while writev was < 10%, even when the server was pushing 350mbps of bandwidth.
While we are talking about nginx, is there a module or reliable form of hot link protection for it? Lighttpd has mod_secdownload, which is good, but it also has mod_trigger_b4_dl, which sounds better but I am unable to install it because their instructions aren't very clear and I keep running into problems when I try to get it working.Last edited by digitalpioneer; 05-12-2008 at 12:33 AM.
-
05-12-2008, 12:37 AM #11Web Hosting Evangelist
- Join Date
- Apr 2003
- Location
- Melbourne, AU
- Posts
- 539
When I tried light 1.5 about a year++ ago, it kept crashing, so I couldn't realize any benefits from its async io backend.
Note that those benchmarks were made with 100KB files - a few orders of magnitude smaller the 100MB files you serve. (writev also wasn't in the benchmark)
But anyway, 10% iowait doesn't sound high, and suggests that you may limited to ~350Mbps due to reasons other than disk. Just to confirm once more, can you run iostat -k 5 -x and see what the util% is like?WK Woon
CTO | http://www.aflexi.net - A flexible Network
Building the next generation CDN platform - DEMO .... coming soon
-
05-12-2008, 01:26 AM #12Temporarily Suspended
- Join Date
- Feb 2007
- Posts
- 185
I couldn't find the benchmark page, but here it is:
http://www.lighttpd.net/2007/2/3/raw-io-performance
they test a 10mb file there.
-
05-12-2008, 01:57 AM #13Temporarily Suspended
- Join Date
- Feb 2007
- Posts
- 185
Just so you know, the current settings are:
Code:server.max-worker = 16 server.network-backend = "writev" server.max-read-idle = 10 server.max-write-idle = 10 server.max-fds = 2048
Code:avg-cpu: %user %nice %system %iowait %steal %idle 0.10 0.00 0.65 0.80 0.00 98.45 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 9.42 1.80 21.84 0.40 2452.10 8.82 221.26 0.14 6.39 1.97 4.39
-
05-12-2008, 02:17 AM #14Temporarily Suspended
- Join Date
- Feb 2007
- Posts
- 185
This weekend, on wed or thurs, i will show you the stats when the server pushes 500mbps+
-
05-12-2008, 12:16 PM #15Temporarily Suspended
- Join Date
- Feb 2007
- Posts
- 185
well, what do you think?