Results 1 to 29 of 29
  1. #1
    Join Date
    Jun 2004
    Posts
    237

    high bandwidth advice needed. and an iowait question...

    how high should iowait get before you should start to worry?

    also, how fast could most servers (with just a regular hard drive, no raid or anything) serve large files? is 20mbps pushing the i/o? 50? 100?

    basically things are slow right now on our current server and we're using almost 20mbps constantly. I'm wondering if upgrading the line to 50mbps will be enough to fix this or if there are more problems beyond just the bandwidth

  2. #2
    Join Date
    Jan 2003
    Posts
    1,715
    Given a typically seek-heavy disk load, you can get around 1 Mbytes/sec from an IDE or SATA drive before it becomes a performance issue. The amount of traffic that supports depends a lot on how large your working set is.

    Apache has a high per-client cost that is best left to scripts and dynamic content. I would start by using thttpd, lighttpd, boa, or squid to handle the static files. With those, 20 Mbit shouldn't be an issue for any working set. For 50 Mbit on large, varied files, you may want to add a second disk.
    Game Servers are the next hot market!
    Slim margins, heavy support, fickle customers, and moronic suppliers!
    Start your own today!

  3. #3
    Join Date
    Jun 2004
    Posts
    237
    thanks. any idea on the iowait question?

  4. #4
    Join Date
    Jan 2003
    Posts
    1,715
    For batch processing (like a stats server), it can go pretty high. For web traffic, I haven't seen it above 5-10%, and I don't think it would be pretty.
    Game Servers are the next hot market!
    Slim margins, heavy support, fickle customers, and moronic suppliers!
    Start your own today!

  5. #5
    Join Date
    Aug 2005
    Location
    Bay Area
    Posts
    63
    1MB is VERY conservative don't you think?! Even for a system with a seek-heavy disk load...
    OneZero We're a perfect 10 when it comes to your hosting and Internet needs!
    To us every last customer and every server also rates a perfect ten!
    http://onezerohosting.com, [email protected]

  6. #6
    We average about 30mbps with 0% iowait, and have maxed the 100mbps port with about 10% io wait. IDE drives
    Try file hosting at Simpload.com

  7. #7
    Join Date
    Jun 2004
    Posts
    237
    what's the highest iowait should be, ideally?

  8. #8
    Join Date
    Aug 2004
    Posts
    333

    Question

    Quote Originally Posted by mythologen
    We average about 30mbps with 0% iowait, and have maxed the 100mbps port with about 10% io wait. IDE drives
    That is great, how you get this great result?? Good hardware or software setup? Is this 100mbps with download one file or multiple files??

    You give to your users direct link to file or you using php for upload/download?

    Thanks.

  9. #9
    Join Date
    Jan 2003
    Posts
    1,715
    Quote Originally Posted by alias_willsmith
    1MB is VERY conservative don't you think?! Even for a system with a seek-heavy disk load...
    Conservative, yes, but not as much as you seem to think. That's been pretty consistent for IDE, although my systems that really push the disks tend to be crazy-seek-heavy. I'm curious what kind of per-drive sustained throughput you see on your systems.

    gurika: You can easily get that if your traffic is mostly a few very popular files (like demo and patch release parties), because the working set fits in memory. If you have a huge, varied selection of active files, no sane amount of RAM will hold it.
    Game Servers are the next hot market!
    Slim margins, heavy support, fickle customers, and moronic suppliers!
    Start your own today!

  10. #10
    Join Date
    Aug 2004
    Posts
    333
    But I have SWAP 0% usage (if I good know that mean that my RAM is not 100% in use) but IOWAIT over 90% and very high load...how you put your files in RAM?
    That is only in case when user downloading file with direct link to file or that can work if I use PHP to hide file location on server (PHP read and print/send file to user)...

    Thanks.

  11. #11
    Join Date
    Jun 2004
    Posts
    237
    I'm still wondering about how high iowait can be before it begins to impact the speed of the server

    can't seem to get a straight answer for this anywhere

  12. #12
    Join Date
    Jan 2003
    Posts
    1,715
    gurika: Sounds a bit like the RedHat kernel bug from some time ago. I haven't seen 90% iowait on any system, functional or not.

    disgust: If iowait is over 10%, it will notably impact responsiveness.
    Game Servers are the next hot market!
    Slim margins, heavy support, fickle customers, and moronic suppliers!
    Start your own today!

  13. #13
    Quote Originally Posted by disgust
    what's the highest iowait should be, ideally?
    I have heard of fileservers that operate with up to 70% iowait, I don't know how fast they ran. Ideally you would have 0% iowait, but I would estimate a range between 0-25% is probably acceptable. Of course it also depends somewhat on what you are doing, eg 20% iowait would likely be very much in the red for a web forum.


    Quote Originally Posted by gurika
    That is great, how you get this great result?? Good hardware or software setup? Is this 100mbps with download one file or multiple files??

    You give to your users direct link to file or you using php for upload/download?

    Thanks.
    PHP up, semi-direct http down. Many different files.

    The hardware and software setups allow the drives to run efficiently; in other words, there is sufficient ram and processing power to aviod bottlenecks.

    When we see lots of traffic it is usually generated from a small number of files being accessed a lot. If your kernel is operating well it will place these files in ram, and resolve many disk problems.
    Try file hosting at Simpload.com

  14. #14
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,290
    A 2.6 kernel with deadline enabled will help with iowait some.. What os are you using??
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  15. #15
    Join Date
    Apr 2004
    Location
    San Jose
    Posts
    902
    IO wait is the time not allocated to system or user processing when there are uncompleted IOs.

    This means you could have a very light CPU application that is doing lots of IO be very responsive with 1% user, 5% system, 80% iowait, and 14% idle. Do the same IO with more CPU, and you could see 85% user, 5% system, 0% iowait and 10%idle.

    The iostat utility will give you a better idea about how your io subsystem is working for you.

    Here is the current iostat result from a very PHP intensive site that is snappy:

    avg-cpu: %user %nice %sys %iowait %idle
    17.48 0.00 7.89 24.58 50.05

    Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
    sda 102.00 0.00 2121.04 0 10584

    Notice that there is fairly high iowait, and lots of writes. That's because PHP sessions by default write files to /tmp. If this became an issue, I would create a memory filesystem and point PHP to that, or use a database backed session system.

    The database is on a separate system here, so I know that's not contributing to the disk IO.

    As with almost any number while doing tuning, iowait can be misleading out of context.

    One thing to watch for is your idle time. If that goes to 0%, and you have significant iowait, then you are IO bound. Depending on your working set (files that get served) you should be able to improve that situation by adding RAM.

    If you have sufficient idle time, then the system is able to keep ahead of the processing and IO loads.

    The load numbers can be misleading. You can have lots of idle time and a load of 15 if all the jobs are very short when they get on the processor. You can have 0 idle time and a load of 1 and be dog slow if your job is very CPU intensive.
    Specializing in MySQL and website tuning for high traffic sites. cmwsci.com/

  16. #16
    Join Date
    Aug 2004
    Posts
    333
    Quote Originally Posted by Steven
    A 2.6 kernel with deadline enabled will help with iowait some.. What os are you using??
    CentOS 3.5

  17. #17
    Join Date
    Aug 2004
    Posts
    333
    Quote Originally Posted by sailorFred
    IO wait is the time not allocated to system or user processing when there are uncompleted IOs.

    This means you could have a very light CPU application that is doing lots of IO be very responsive with 1% user, 5% system, 80% iowait, and 14% idle. Do the same IO with more CPU, and you could see 85% user, 5% system, 0% iowait and 10%idle.

    The iostat utility will give you a better idea about how your io subsystem is working for you.

    Here is the current iostat result from a very PHP intensive site that is snappy:

    avg-cpu: %user %nice %sys %iowait %idle
    17.48 0.00 7.89 24.58 50.05

    Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
    sda 102.00 0.00 2121.04 0 10584

    Notice that there is fairly high iowait, and lots of writes. That's because PHP sessions by default write files to /tmp. If this became an issue, I would create a memory filesystem and point PHP to that, or use a database backed session system.

    The database is on a separate system here, so I know that's not contributing to the disk IO.

    As with almost any number while doing tuning, iowait can be misleading out of context.

    One thing to watch for is your idle time. If that goes to 0%, and you have significant iowait, then you are IO bound. Depending on your working set (files that get served) you should be able to improve that situation by adding RAM.

    If you have sufficient idle time, then the system is able to keep ahead of the processing and IO loads.

    The load numbers can be misleading. You can have lots of idle time and a load of 15 if all the jobs are very short when they get on the processor. You can have 0 idle time and a load of 1 and be dog slow if your job is very CPU intensive.
    Yes, I have IOWAIT 70-90% and IDLE 0-10% and that is my problem...

  18. #18
    Join Date
    Apr 2004
    Location
    San Jose
    Posts
    902
    First thing to check is for disk errors in /var/log/messages, or wherever your syslog errors go.

    If there's no evidence of a failing disk, then you'll want to see which processes are doing the IO.

    If you do ps -el | grep ' D ' you should see which processes are in Disk Wait.

    I would expect that they would be httpd instances. Then you take the PID and strace it to see what IO it's doing.

    Most likely, you are using PHP sessions, which default to being stored in /tmp.

    You can create a memory filesystem and ask PHP to store the session information there, if you don't have a problem with frequent reboots, or the session information is not critical.
    Specializing in MySQL and website tuning for high traffic sites. cmwsci.com/

  19. #19
    Join Date
    Aug 2004
    Posts
    333
    No erros (same problem on few servers), no PHP sessions... ps -el | grep ' D ' in 99% cases print nothing...

    Maybe I have this only because I have too much downloads and HD`s can`t do all of that in same time...

    Also, maybe can be problem because I store all files (thousands) in one folder??? This is stupid question but ....

  20. #20
    Join Date
    Apr 2004
    Location
    San Jose
    Posts
    902
    In the 1% of cases there was a process in Disk Wait, what was it?

    How big is your content vs. your RAM? Ideally, the most frequently hit files should fit in the disk cache in RAM.

    Having lots of files in a directory can be a performance problem if the files are constantly changing. Deletions, especially, can be slow when you get into the 10,000 files/directory range.

    The Apache logging will do a certain amount of disk writing.

    Do iostat -k 10. Let it run for 30 seconds and paste the results. This will give us the balance between reads vs. writes and volume.
    Specializing in MySQL and website tuning for high traffic sites. cmwsci.com/

  21. #21
    Join Date
    Nov 2003
    Posts
    384
    hiryuu is right with the 1MBytes/sec for completely *random* rw access (mysql with huge 50GB+ tables and 300+ requests/sec). Streaming data goes much faster of course, 30MBytes/sec+.

  22. #22
    Join Date
    Oct 2005
    Location
    Sunny Spain
    Posts
    8
    Quote Originally Posted by gurika
    Also, maybe can be problem because I store all files (thousands) in one folder??? This is stupid question but ....
    This depends largely on the filesystem used. For instance, ReiserFS was explicitly designed to handle sparse files and supposedly scales up to even millions in a folder, but other filesystems such as ext3 might not do so well.

  23. #23
    Join Date
    Aug 2004
    Posts
    333
    Quote Originally Posted by sailorFred
    In the 1% of cases there was a process in Disk Wait, what was it?

    How big is your content vs. your RAM? Ideally, the most frequently hit files should fit in the disk cache in RAM.

    Having lots of files in a directory can be a performance problem if the files are constantly changing. Deletions, especially, can be slow when you get into the 10,000 files/directory range.

    The Apache logging will do a certain amount of disk writing.

    Do iostat -k 10. Let it run for 30 seconds and paste the results. This will give us the balance between reads vs. writes and volume.
    Now I check on one server with 99% time over 80% IOWAIT and 0 IDLE and I have this:
    1 D 0 6778 1 0 75 0 - 0 io_sch ? 00:33:06 kjournald
    5 D 99 28850 4561 0 75 0 - 2847 schedu ? 00:01:38 httpd
    5 D 99 28851 4561 0 75 0 - 2854 schedu ? 00:01:50 httpd
    5 D 99 28852 4561 0 75 0 - 2846 schedu ? 00:01:47 httpd
    5 D 99 28853 4561 0 75 0 - 2851 schedu ? 00:01:53 httpd
    5 D 99 28854 4561 0 75 0 - 2851 schedu ? 00:01:53 httpd
    5 D 99 28856 4561 0 75 0 - 2847 schedu ? 00:01:48 httpd
    5 D 99 28860 4561 0 75 0 - 2851 schedu ? 00:01:57 httpd
    5 D 99 28959 4561 0 75 0 - 2851 schedu ? 00:01:50 httpd
    5 D 99 28961 4561 0 75 0 - 2853 schedu ? 00:01:53 httpd
    5 D 99 28967 4561 0 75 0 - 2848 io_sch ? 00:01:55 httpd
    5 D 99 29038 4561 0 75 0 - 2843 schedu ? 00:01:38 httpd
    5 D 99 29039 4561 0 75 0 - 2857 schedu ? 00:01:44 httpd
    5 D 99 29222 4561 0 75 0 - 2858 io_sch ? 00:01:48 httpd
    5 D 99 29671 4561 0 75 0 - 2848 schedu ? 00:01:48 httpd
    5 D 99 29679 4561 0 75 0 - 2848 schedu ? 00:01:50 httpd
    5 D 99 29901 4561 0 75 0 - 2843 schedu ? 00:02:04 httpd
    5 D 99 30017 4561 0 75 0 - 2856 schedu ? 00:01:24 httpd
    5 D 99 30121 4561 0 75 0 - 2844 schedu ? 00:01:23 httpd
    5 D 99 30165 4561 0 75 0 - 2849 schedu ? 00:01:42 httpd
    5 D 99 25273 4561 0 75 0 - 2843 schedu ? 00:01:17 httpd
    5 D 99 25533 4561 0 75 0 - 2851 schedu ? 00:01:22 httpd
    5 D 99 27262 4561 0 75 0 - 2853 schedu ? 00:01:09 httpd
    5 D 99 27607 4561 0 75 0 - 2843 schedu ? 00:01:16 httpd
    5 D 99 28015 4561 0 75 0 - 2847 schedu ? 00:01:17 httpd
    5 D 99 4535 4561 0 75 0 - 2836 schedu ? 00:00:22 httpd
    5 D 99 4613 4561 0 75 0 - 2833 schedu ? 00:00:26 httpd
    5 D 99 4993 4561 0 75 0 - 2845 io_sch ? 00:00:17 httpd
    5 D 99 5653 4561 0 75 0 - 2829 schedu ? 00:00:16 httpd
    5 D 99 5672 4561 0 75 0 - 2826 schedu ? 00:00:20 httpd
    5 D 99 7024 4561 0 75 0 - 2835 schedu ? 00:00:20 httpd
    5 D 99 7135 4561 0 75 0 - 2833 schedu ? 00:00:14 httpd
    5 D 99 7270 4561 0 75 0 - 2826 schedu ? 00:00:10 httpd
    5 D 99 7438 4561 0 75 0 - 2832 io_sch ? 00:00:15 httpd
    5 D 99 7439 4561 0 75 0 - 2850 schedu ? 00:00:21 httpd
    5 D 99 7837 4561 0 75 0 - 2848 schedu ? 00:00:10 httpd
    5 D 99 7938 4561 0 75 0 - 2835 schedu ? 00:00:17 httpd
    5 D 99 7943 4561 0 75 0 - 2845 schedu ? 00:00:14 httpd


    Content on server from 1KB to 60MB, RAM 2GB!

    iostat -k 10 don`t work on my server: -bash: iostat: command not found

    Also, you must know that I using PHP for file downoad and this is main code for that:

    [email protected]($file,"rb");
    while ([email protected]($fp) && !connection_aborted()) {
    echo(@fread($fp, 8192));
    }
    fclose ($fp);

    I don`t know is possible to store some data in RAM if I downloading files on this way, I thinking that this using very much HD and this must be main problem...

    When I have very high IOWAIT my HD working very very slow:
    hdparm -t /dev/sdb

    /dev/sdb:
    Timing buffered disk reads: 2 MB in 25.35 seconds = 80.79 kB/sec (normal over 50MB/s)

    (SATA 8MB cach)

    Thanks.

  24. #24
    Join Date
    Aug 2004
    Posts
    333
    Now I installed iostat but now this error:
    iostat -k 10
    iostat: invalid option -- k

    and only this options:
    c - print cpu usage info
    d - print basic disk info
    D - print disk utilization info
    p - print partition info also
    P - print partition info only
    x - print extended disk info
    h - this help

    I tested this with this command:
    iostat -D 10

    and this is what I get:

    sda sdb
    r/s w/s %b r/s w/s %b
    165 6 1 252 8 1
    152 3 79 234 14 100
    156 4 80 245 24 100
    195 3 86 225 25 100
    206 4 79 230 20 100
    172 2 82 228 12 100
    215 2 85 228 11 100
    177 4 76 239 27 100
    175 2 71 235 14 100
    185 2 76 236 26 100
    171 4 69 250 13 100
    164 3 71 249 28 100
    172 4 73 230 16 100


    Thanks.

  25. #25
    Join Date
    Jan 2003
    Posts
    1,715
    You really should be using readfile() or fpassthru() for sending the file.
    Game Servers are the next hot market!
    Slim margins, heavy support, fickle customers, and moronic suppliers!
    Start your own today!

  26. #26
    Join Date
    Apr 2004
    Location
    San Jose
    Posts
    902
    You are really pushing your disk for reading. It appears that the filesystem RAM cache is not performing as well as it could.

    What's the total size of the files you are serving? Is it significantly more than your 2GB of RAM?

    Also, do you really need to use PHP to serve the files?

    If not, thttpd or lighttpd may be able to help server static files more efficiently.
    Specializing in MySQL and website tuning for high traffic sites. cmwsci.com/

  27. #27
    Join Date
    Aug 2004
    Posts
    333
    Yes, I think that I must use PHP for this because I must hide file location on server and I don`t know how I can do that on some other way, I don`t know is perl better for that or some other way...??

    Hm, readfile() and fpassthru(), why you think that this is better for me?? On pages:
    http://www.php.net/manual/en/function.fpassthru.php
    http://www.php.net/readfile
    not good comments and one of them:
    "readfile and fpassthru are about 55% slower than doing a loop with "feof/echo fread"."

    Total size of files is very much...

  28. #28
    Join Date
    Jan 2003
    Posts
    1,715
    You can also use mod_rewrite for mangling URLs. Those numbers with fread are curious, since readfile and fpassthru should be using the sendfile() syscall, just like Apache itself would. That makes better use of your disk cache, since you don't have a bunch of TCP buffers that all hold the same stuff.
    Game Servers are the next hot market!
    Slim margins, heavy support, fickle customers, and moronic suppliers!
    Start your own today!

  29. #29
    Join Date
    Apr 2004
    Location
    San Jose
    Posts
    902
    Without knowing more about why you need to obfuscate the URLs, I can't make a rational suggestion for improving your performance.

    I'm guessing that people don't really have access to your hard drive, unless it's an FTP upload site, where you don't want people to be able to figure out what files to modify from their URLs. If that's the case, then mod_rewrite is your friend.

    If you can let us know more about your system, we might be able to help you more.

    If you can't, then you might consider upping your RAM to whatever will fit in your box, RAID, or renting two boxes and splitting the content between them, or duplicating it and using round robin DNS to distribute the load.
    Specializing in MySQL and website tuning for high traffic sites. cmwsci.com/

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •