Results 1 to 7 of 7
  1. #1
    Join Date
    Nov 2004
    Location
    Finland
    Posts
    522

    Lots of I/O Wait... Tools to analyze why?

    I've got a problem, there's A LOT of I/O wait...
    Something like 60% of all CPU TIME is spent on I/O wait, even after backup processes has ended.

    How do i identify the culprit(s), and what they are waiting for?
    Any suggestions?

    [~]# uname -r
    2.6.9-34.0.1.ELsmp
    [~]# uptime
    04:13:51 up 11 days, 2:06, 10 users, load average: 2.99, 3.43, 3.00
    Mem: 2066764k total, 1473404k used, 593360k free, 36508k buffers
    Swap: 1052248k total, 160k used, 1052088k free, 1035472k cached

    Ordinary view in TOP:
    Cpu(s): 4.2% us, 1.9% sy, 0.0% ni, 5.4% id, 88.6% wa, 0.0% hi, 0.0% si
    Cpu(s): 12.5% us, 5.5% sy, 0.0% ni, 6.7% id, 75.3% wa, 0.0% hi, 0.0% si
    Cpu(s): 13.1% us, 9.2% sy, 0.0% ni, 15.5% id, 62.2% wa, 0.0% hi, 0.0% si
    Cpu(s): 8.2% us, 7.1% sy, 0.2% ni, 18.4% id, 66.2% wa, 0.0% hi, 0.0% si
    Cpu(s): 2.7% us, 4.4% sy, 0.2% ni, 37.4% id, 55.4% wa, 0.0% hi, 0.0% si
    Cpu(s): 6.6% us, 5.7% sy, 0.0% ni, 16.7% id, 71.0% wa, 0.0% hi, 0.0% si

    Yet, individual processes count only for very little % of cpu time.

    What's happening on there?

    Munin shows spiky graph for Interrupts & Context switches also
    Pulsed Media: Seedbox hosting and Dedicated Servers for content distribution

  2. #2
    Join Date
    Jun 2003
    Location
    Janesville, Wi
    Posts
    1,516
    There is a nifty tool called "sysstaf" which includes a small bundle of useful tools.

    http://perso.orange.fr/sebastien.godard/index.html

    One of the tools (/usr/local/bin/iostat is where it is insalled) allows you to check the disk saturation and cpu usage levels. This is excellent for determining whether or not your disks are overloaded.

    iostat -x

    That command will calculate disk saturation among other things. Check out the man pages for iostat and sysstat to see everything.
    Jakiao

  3. #3
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,290
    "wa, 0.0%"

    is iowait and its very low.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  4. #4
    Join Date
    Jun 2003
    Location
    Janesville, Wi
    Posts
    1,516
    Uh, Steve, that's a comma. It's 0.0% hi, 0.0% sys. His iowaits average around 60%.
    Jakiao

  5. #5
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,290
    Opps, thats what I get for not waking up,

    2.6.9-34.0.1.ELsmp

    There is two reasons why this should be upgraded.

    1.) You are exploitable to root exploits
    2.) There has been issues with iowait in that kernel and other centos/rhel kernels.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  6. #6
    Join Date
    Jun 2003
    Location
    Janesville, Wi
    Posts
    1,516
    I agree with Steven on that point. The kernel is open to some pretty serious zero-day exploits (such as the /proc exploit). But even then, a iowait that high is more than likely being caused by issues with the hard drives response; however, this can be proven or disproven using sysstat and iostat.
    Jakiao

  7. #7
    Join Date
    Oct 2005
    Location
    United States
    Posts
    1,403
    Skal Tura,
    You can try to ask your data center to run fsck for you; I think this may help!
    If it's not then you can try to upgrade new kernel for your box.
    Tommy Tran - tommy @ vinax.net ::: VINAX, LLC ::: http://vinax.net ::: Since 2004
    Premium Dedicated Servers and Colocation in downtown Chicago (350 E. Cermak Rd)
    Premium Bandwidth, 100% Network & Power Uptime SLA, 24/7 Prompt Tech Support

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •