Results 1 to 11 of 11
  1. #1
    Join Date
    Aug 2002
    Posts
    39

    Serving large files

    What is the best solution for serving large files, meaning a few hundred MB each. Thttpd can't really seem to handle many of these files, it starts giving me "500, unusual errors" after awhile.

  2. #2
    Join Date
    Apr 2005
    Location
    Chicago, IL
    Posts
    9
    What version of thttpd are you using? Several bugs were fixed in early versions that resolve the exact issue you seem to be describing. What is likely happening is that your system is running out of mapped memory. For versions prior to (IIRC) 2.20, you can try tweaking config.h to adjust OCCSAIONAL_TIME to something lower than 300 (default) to something like 25. If all you're serving is large files (under 2GB, of course) you can also decrease DESIRED_MAX_MAPPED_FILES to something much smaller so that you do not consume all of the available resources on the host.

    There are so many variables and solutions to take into account it's difficult to make accurate suggestions. I'd be happy to take a look or help you further; toss me a message.

  3. #3
    Can you serve the files through anonymous FTP? There is no need to bog down the webserver with large file downloads if you are able to run FTP also.

  4. #4
    Join Date
    Jan 2003
    Posts
    1,715
    Moving to FTP doesn't really change anything, since it's just a different byte shoveling process. A thin httpd process is just as efficient as a thin FTP server, if not more so. The only advantage is you may find more FTP servers using fread(), rather than mmap(), so they can use 64-bit file sizes on 32-bit systems.

    I know squid uses reads, but it needs a web server to back it, since it's a cache. You may want to ask Zeus how they handle many large files. You could also consider an Athlon64 box, since that has a 64-bit address space to dump stuff in.
    Game Servers are the next hot market!
    Slim margins, heavy support, fickle customers, and moronic suppliers!
    Start your own today!

  5. #5
    Join Date
    Oct 2002
    Posts
    702
    What operating system do you use? There are several thttpd patches to work around your problem.
    ServerMatingProject.com
    The World's first server mating experiment
    We give new meaning to I/O intensive and hot swap

  6. #6
    Join Date
    Nov 2002
    Location
    Finland
    Posts
    96

    Re: Serving large files

    Originally posted by Force
    What is the best solution for serving large files, meaning a few hundred MB each. Thttpd can't really seem to handle many of these files, it starts giving me "500, unusual errors" after awhile.
    http://lighttpd.net/

    lighttpd a secure, fast, compliant and very flexible web-server which has been optimized for high-performance environments. It has a very low memory footprint compared to other webservers and takes care of cpu-load. Its advanced feature-set (FastCGI, CGI, Auth, Output-Compression, URL-Rewriting and many more) make lighttpd the perfect webserver-software for every server that is suffering load problems.

    I've been testing lighttpd for few weeks now and it rocks

    It might be just what ur looking for...



    - Tero

  7. #7
    Join Date
    Aug 2002
    Posts
    39
    I am using the newest version of thttpd and CentOS. I have tried changing the settings in config.h, but it doesn't help much because I have mostly small files being served. Does anyone have a link to those patches? I will give lighttpd a try.

  8. #8
    Join Date
    Aug 2002
    Posts
    39
    Just tried lighttpd and it didn't fix the problem. Would upgrading the RAM help? Here are the current specs:

    Dual Xeon 3.0HT
    2GB RAM
    2x250GB SATA HDs

    Any ideas?

  9. #9
    Whats the percentage of RAM being used at peak download times?
    BeeServe
    * Rock solid shared & reseller UK webhosting. No downtime™ *
    Now offering fully managed VPS servers

  10. #10
    Join Date
    Oct 2002
    Posts
    702
    http://xoomer.virgilio.it/adefacc/httpd/thttpd/#toc3

    These patches should help you out. You can controll what files go into memory and which ones don't.
    ServerMatingProject.com
    The World's first server mating experiment
    We give new meaning to I/O intensive and hot swap

  11. #11
    Join Date
    Aug 2002
    Posts
    39
    I couldn't get top while the load was at a peak (around 4-6), but I got it at around 1.6. Maybe memory isn't the problem. Here are the results:

    Code:
     20:23:29  up 17 days, 20:38,  1 user,  load average: 1.64, 1.20, 0.86
    313 processes: 311 sleeping, 1 running, 1 zombie, 0 stopped
    CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
               total    0.1%    0.0%    1.6%   0.0%     0.0%    0.0%   98.1%
               cpu00    0.5%    0.0%    3.3%   0.0%     0.0%    0.0%   96.0%
               cpu01    0.0%    0.0%    1.3%   0.0%     0.0%    0.0%   98.6%
               cpu02    0.1%    0.0%    1.9%   0.0%     0.0%    0.0%   97.8%
               cpu03    0.0%    0.0%    0.0%   0.0%     0.0%    0.0%  100.0%
    Mem:  2069028k av, 2019268k used,   49760k free,       0k shrd,   16672k buff
           260744k active,            1311436k inactive
    Swap:       0k av,       0k used,       0k free                 1555336k cached
    
      PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
     6116 nobody    15   0  745M 745M  743M D     0.2 36.8 116:57   1 thttpd
    29199 xgamedb   19  19 35720  34M 35108 S N   0.0  1.7   0:00   2 cpanellogd
     3646 root      19  19 35712  34M 34976 S N   0.0  1.7   1:24   2 cpanellogd
    29198 xgamedb   19  19 35712  34M 35296 S N   0.0  1.7   0:00   2 cpanellogd
    26490 root       9   0 19976  19M 10928 S     0.0  0.9   0:01   0 spamd
    26488 root       9   0 19972  19M 10892 S     0.0  0.9   0:01   3 spamd
    26489 root       9   0 19972  19M 10928 S     0.0  0.9   0:01   0 spamd
    26491 root       9   0 19972  19M 10928 S     0.0  0.9   0:01   2 spamd
    26492 root       9   0 19972  19M 10928 S     0.0  0.9   0:01   2 spamd
     3552 mysql      9   0 19584  19M  1044 S     0.0  0.9   3:09   3 mysqld
     3565 mysql      9   0 19584  19M  1044 S     0.0  0.9   3:37   2 mysqld

    Also, I'm having a lot of trouble getting old versions of thttpd to work with ADF's patches. Do I have to uninstall my new one first?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •