Results 1 to 15 of 15
  1. #1
    Join Date
    Feb 2007
    Posts
    184

    Optimizing lighttpd for large files (180mb avg)

    Hello peeps,

    I have four servers with a quad xeon, 4gb ram, and 2x300GB SAS 15K RAID0 harddrives, pushing a total of 1.6gbits. It serves a lot of zip files with an average flesize of 180mb. My question is, how can I optimize lighttpd 1.4.19 to push its max with very low IO-wait. I've looked up some stuff and only found options that deal with lighttpd 1.5 and use Linux-AIO for the backend network. Currently I use writev with 16 workers and an read/write idle timeout of 10s. Logging is off, too. Anything else I can do?

  2. #2
    If its just serving files then i would switch over to thttpd - its faster for static content

  3. #3
    Join Date
    Apr 2003
    Location
    Melbourne, AU
    Posts
    539
    lighttpd 1.4 with writev can easily saturate a gig-e serving large files like yours, provided that the disk subsystem isn't holding it back

    As lighttpd doesn't come with many tuning knobs, maybe you can take a step back and consider some other things to address disk performance?

    - is your RAID0 stripe size large (e.g. 1MB)?
    - are those server chassis 2-disk or 4-disk?
    - do those servers have distinct files (from each other), or do they share the same set of files?
    - how big is the working set of files served at any one time?
    - is adding RAM an option?
    WK Woon
    CTO | http://www.aflexi.net - A flexible Network
    Building the next generation CDN platform - DEMO .... coming soon

  4. #4
    Join Date
    Feb 2007
    Posts
    184
    - I'm not sure what it is, how can I check and change it?
    - 2 disk
    - same set of files
    - i'm not sure, how do i check?
    - yes but it would be more expensive and pretty useless since the files are so large

  5. #5
    Join Date
    Feb 2007
    Posts
    184
    well? anyone?

  6. #6
    Join Date
    Dec 2002
    Location
    chica go go
    Posts
    11,858
    If you're pushing 1.6gbit, then you are probably topping out your port speed. I'd try to get a second gigabit network drop, or a new server if i were you.

    As for your raid array, switch to raid 5, 10, or 1.

    Software, try switching to nginx instead of lighty. I ran lighty serving images for a while, and nginx outperforms it in terms of serving static content.

  7. #7
    Join Date
    Feb 2007
    Posts
    184
    It is aggregated bandwidth and I have 5 servers splitting the load, so each pushes around 200-350mbps. They are all mirrors so raid0 is the best for max performance, and a failed drive can easily be fixed. nginx also is great but it dosn't have the mod_secdownload module like lighttpd does.
    Last edited by digitalpioneer; 05-11-2008 at 03:42 PM.

  8. #8
    Join Date
    Apr 2003
    Location
    Melbourne, AU
    Posts
    539
    Quote Originally Posted by starkana View Post
    - I'm not sure what it is, how can I check and change it?
    - 2 disk
    - same set of files
    - i'm not sure, how do i check?
    - yes but it would be more expensive and pretty useless since the files are so large
    - Use your card's RAID utility.

    - Oh, can't add anymore disks then :\

    - Currently, how are visitors directed to the four servers? Randomly? Or is each server designated to serve only a subset of the file collection?

    - I would parse the output of lsof. Look for lines matching lighttpd and your docroot.

    - Depends on your working set.
    WK Woon
    CTO | http://www.aflexi.net - A flexible Network
    Building the next generation CDN platform - DEMO .... coming soon

  9. #9
    Join Date
    Apr 2003
    Location
    Melbourne, AU
    Posts
    539
    Quote Originally Posted by ub3r View Post
    Software, try switching to nginx instead of lighty. I ran lighty serving images for a while, and nginx outperforms it in terms of serving static content.
    Nginx is a damned spectacular piece of code and is very fast, in terms of CPU use. I use it for image webservers (where reqs/s is very high). However, in a situation where a disk is a major bottleneck and reqs/s is low, I reckon lighttpd with writev trumps it - I feel (disclaimer: no hard numbers to back me) that in practice it works better than the Linux fs cache.
    WK Woon
    CTO | http://www.aflexi.net - A flexible Network
    Building the next generation CDN platform - DEMO .... coming soon

  10. #10
    Join Date
    Feb 2007
    Posts
    184
    - Currently, how are visitors directed to the four servers? Randomly? Or is each server designated to serve only a subset of the file collection?
    Load balancer

    Quote Originally Posted by wKkaY View Post
    Nginx is a damned spectacular piece of code and is very fast, in terms of CPU use. I use it for image webservers (where reqs/s is very high). However, in a situation where a disk is a major bottleneck and reqs/s is low, I reckon lighttpd with writev trumps it - I feel (disclaimer: no hard numbers to back me) that in practice it works better than the Linux fs cache.

    According to lighttpd (http://blog.lighttpd.net/articles/2006/11/12/lighty-1-5-0-and-linux-aio), writev is has the lowest throughput, linux-sendfile is much faster and linux-aio-sendfile is much faster on 1.5. However, I have tested linux-sendfile and it gave me a IOwait of 15-25% while writev was < 10%, even when the server was pushing 350mbps of bandwidth.

    While we are talking about nginx, is there a module or reliable form of hot link protection for it? Lighttpd has mod_secdownload, which is good, but it also has mod_trigger_b4_dl, which sounds better but I am unable to install it because their instructions aren't very clear and I keep running into problems when I try to get it working.
    Last edited by digitalpioneer; 05-12-2008 at 12:33 AM.

  11. #11
    Join Date
    Apr 2003
    Location
    Melbourne, AU
    Posts
    539
    When I tried light 1.5 about a year++ ago, it kept crashing, so I couldn't realize any benefits from its async io backend.

    Note that those benchmarks were made with 100KB files - a few orders of magnitude smaller the 100MB files you serve. (writev also wasn't in the benchmark)

    But anyway, 10% iowait doesn't sound high, and suggests that you may limited to ~350Mbps due to reasons other than disk. Just to confirm once more, can you run iostat -k 5 -x and see what the util% is like?
    WK Woon
    CTO | http://www.aflexi.net - A flexible Network
    Building the next generation CDN platform - DEMO .... coming soon

  12. #12
    Join Date
    Feb 2007
    Posts
    184
    Quote Originally Posted by wKkaY View Post
    When I tried light 1.5 about a year++ ago, it kept crashing, so I couldn't realize any benefits from its async io backend.

    Note that those benchmarks were made with 100KB files - a few orders of magnitude smaller the 100MB files you serve. (writev also wasn't in the benchmark)

    But anyway, 10% iowait doesn't sound high, and suggests that you may limited to ~350Mbps due to reasons other than disk. Just to confirm once more, can you run iostat -k 5 -x and see what the util% is like?

    I couldn't find the benchmark page, but here it is:

    http://www.lighttpd.net/2007/2/3/raw-io-performance

    they test a 10mb file there.

  13. #13
    Join Date
    Feb 2007
    Posts
    184
    Just so you know, the current settings are:

    Code:
    server.max-worker = 16
    server.network-backend = "writev"
    server.max-read-idle = 10
    server.max-write-idle = 10
    server.max-fds = 2048
    and the results of iostat -k 5 -x are:
    Code:
     avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               0.10    0.00    0.65    0.80    0.00   98.45
    
    Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
    sda               9.42     1.80 21.84  0.40  2452.10     8.82   221.26     0.14    6.39   1.97   4.39
    please note that the servers aren't pushing as much as they do on otherdays, i'll get you the results on bigger days when they come

  14. #14
    Join Date
    Feb 2007
    Posts
    184
    This weekend, on wed or thurs, i will show you the stats when the server pushes 500mbps+

  15. #15
    Join Date
    Feb 2007
    Posts
    184
    well, what do you think?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •