Results 1 to 9 of 9
  1. #1

    High iowait with RAID 10

    Hello,

    I have a Q9400, 8G RAM, 4x250 SATA in RAID 10 (MegaRAID SAS 8344ELP). It supposes to be fast and reliable and yet I feel it's not performing right. I also downloaded and checked the output of "MegaCli64" on the RAID status and all disks is fine.

    This is the report from "vmstat 1". How to know what is causing the high iowait? Also, how do I test disk performance to see if the RAID is indeed worth the extra bucks? Thanks.

    Code:
    procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
     r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
     1  2  49192  51656  23316 5099728    0    0    27    89    7    1 13  4 76  7  0
     1  2  49192  48116  23336 5109804    0    0  2028  3436 2037 2129  5  4 60 31  0
     0  2  49192  48116  23336 5109728    0    0     0  6784 1649 1909  7  2 68 24  0
     2  1  49192  43716  23360 5114716    0    0  2524   620 2060 2663  6  3 69 23  0
     0  1  49192  44368  23384 5114076    0    0  5136   232 2707 3262  7  4 68 20  0
     0  1  49192  92392  23356 5064420    0    0  5612   236 2788 3472  8  6 64 22  0
     0  3  49192  72192  23372 5084564    0    0  4372 42636 2564 3259  9  4 67 21  0
     1  5  49192  70836  23372 5085724    0    0   256  9008 1625 2066  7  2 27 65  0
     0  5  49192  70216  23372 5085844    0    0     0  8528 1510 1880  5  1 46 48  0
     1  5  49192  70092  23372 5085844    0    0     0  1500 1469 1787  6  1 45 47  0
     0  3  49192  70324  23372 5085844    0    0     0    28 1530 1691  5  1 63 31  0
     0  4  49192  70636  23372 5085844    0    0     4   340 1604 2115  7  2 67 23  0
     1  2  49192  71328  23388 5086488    0    0   108  2724 1622 1767  6  2 51 41  0
     1  1  49192  58760  23404 5097404    0    0  2372   384 2164 2784  8  4 66 22  0
     2  2  49192  52320  23412 5104432    0    0  5884   264 2914 3799  9  6 64 21  0
    123finder.com - Browse/search 4-char domains or dictionary domains

  2. #2
    Try a simple hdparm -tT /dev/device if you get satisfactory results from that then hire someone to tune the server.

  3. #3
    And this is the result from "MegaCli -LDInfo -LALL -aALL"

    Code:
    Adapter 0 -- Virtual Drive Information:
    Virtual Disk: 0 (target id: 0)
    Name:
    RAID Level: Primary-1, Secondary-3, RAID Level Qualifier-0
    Size:474928MB
    State: Optimal
    Stripe Size: 64kB
    Number Of Drives:2
    Span Depth:2
    Default Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
    Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
    Access Policy: Read/Write
    Disk Cache Policy: Disabled

    Code:
    hdparm -tT /dev/sda:
     Timing cached reads:   7148 MB in  2.00 seconds = 3574.96 MB/sec
     Timing buffered disk reads:  386 MB in  3.01 seconds = 128.07 MB/sec
    The throughput is oka I guess (I've seen better on other servers) but the loadavg easily jumps and slow the whole server. I don't want to hire anyone since I would like to learn these for myself as well.
    123finder.com - Browse/search 4-char domains or dictionary domains

  4. #4
    Make sure your kernel is compiled with TASK_DELAY_ACCT and TASK_IO_ACCOUNTING and simply install a program that can interpret the data such as iotop. For performance there are literally dozens of options that can effect the throughput some that come to mind: readahead,stripe size,file system barriers,etc.

  5. #5
    Join Date
    Dec 2005
    Posts
    3,077
    Timing buffered disk reads: 386 MB in 3.01 seconds = 128.07 MB/sec
    That is extremely poor for 4x SATAII disks in RAID10.

    What disks are being used?

    We typically see 380MB/sec out of 4x SATAII disks in RAID10 under Adaptec or 3Ware cards.

  6. #6
    Join Date
    Feb 2008
    Location
    Houston, Texas, USA
    Posts
    2,955
    Quote Originally Posted by 123finder.com View Post
    And this is the result from "MegaCli -LDInfo -LALL -aALL"

    Code:
    Adapter 0 -- Virtual Drive Information:
    Virtual Disk: 0 (target id: 0)
    Name:
    RAID Level: Primary-1, Secondary-3, RAID Level Qualifier-0
    Size:474928MB
    State: Optimal
    Stripe Size: 64kB
    Number Of Drives:2
    .
    .
    .
    Disk Cache Policy: Disabled
    Why is it showing 2 spindles only? Also, you might benefit from enabling the disk cache. Especially if you have a battery backed controller.

    hdparm -tT /dev/sda:
    Use sdparm for SATA disks.

    Regards
    Joe
    UNIXy - Fully Managed Servers and Clusters - Established in 2006
    [ cPanel Varnish Nginx Plugin ] - Enhance LiteSpeed and Apache Performance
    www.unixy.net - Los Angeles | Houston | Atlanta | Rotterdam
    Love to help pro bono (time permitting). joe > unixy.net

  7. #7
    Hello,

    @Joe: The number of drives shows as 2 but the span is also 2. Does it mean 2x2=4 drives in total?

    @PCS-Chris: This is the output on the 4 drives. They're Western Digital 250G (WD2500AAKS-03A01). I've seen single drive that's faster than this so that's why I'm complaining.

    Code:
    [[email protected]]# MegaCli -LdPdInfo -a0
                                         
    Adapter #0
    
    Number of Virtual Disks: 1
    Virtual Disk: 0 (target id: 0)
    Name:
    RAID Level: Primary-1, Secondary-3, RAID Level Qualifier-0
    Size:474928MB
    State: Optimal
    Stripe Size: 64kB
    Number Of Drives:2
    Span Depth:2
    Default Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
    Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
    Access Policy: Read/Write
    Disk Cache Policy: Disabled
    Number of Spans: 2
    Span: 0 - Number of PDs: 2
    PD: 0 Information
    Enclosure Device ID: 252
    Slot Number: 4
    Device Id: 12
    Sequence Number: 2
    Media Error Count: 0
    Other Error Count: 0
    Predictive Failure Count: 0
    Last Predictive Failure Event Seq Number: 0
    Raw Size: 238475MB [0x1d1c5970 Sectors]
    Non Coerced Size: 237963MB [0x1d0c5970 Sectors]
    Coerced Size: 237464MB [0x1cfcc000 Sectors]
    Firmware state: Online
    SAS Address(0): 0x63813c259f988b8c
    Connected Port Number: 4(path0) 
    Inquiry Data: ATA     WDC WD2500AAKS-03A01     
    
    PD: 1 Information
    Enclosure Device ID: 252
    Slot Number: 5
    Device Id: 13
    Sequence Number: 2
    Media Error Count: 0
    Other Error Count: 0
    Predictive Failure Count: 0
    Last Predictive Failure Event Seq Number: 0
    Raw Size: 238475MB [0x1d1c5970 Sectors]
    Non Coerced Size: 237963MB [0x1d0c5970 Sectors]
    Coerced Size: 237464MB [0x1cfcc000 Sectors]
    Firmware state: Online
    SAS Address(0): 0x63813c259f9c8f8d
    Connected Port Number: 5(path0) 
    Inquiry Data: ATA     WDC WD2500AAKS-03A01     
    
    Span: 1 - Number of PDs: 2
    PD: 0 Information
    Enclosure Device ID: 252
    Slot Number: 6
    Device Id: 14
    Sequence Number: 2
    Media Error Count: 0
    Other Error Count: 0
    Predictive Failure Count: 0
    Last Predictive Failure Event Seq Number: 0
    Raw Size: 238475MB [0x1d1c5970 Sectors]
    Non Coerced Size: 237963MB [0x1d0c5970 Sectors]
    Coerced Size: 237464MB [0x1cfcc000 Sectors]
    Firmware state: Online
    SAS Address(0): 0x63813c259f948f8d
    Connected Port Number: 6(path0) 
    Inquiry Data: ATA     WDC WD2500AAKS-03A01     
    
    PD: 1 Information
    Enclosure Device ID: 252
    Slot Number: 7
    Device Id: 15
    Sequence Number: 2
    Media Error Count: 0
    Other Error Count: 0
    Predictive Failure Count: 0
    Last Predictive Failure Event Seq Number: 0
    Raw Size: 238475MB [0x1d1c5970 Sectors]
    Non Coerced Size: 237963MB [0x1d0c5970 Sectors]
    Coerced Size: 237464MB [0x1cfcc000 Sectors]
    Firmware state: Online
    SAS Address(0): 0x63813c259e9d8e8c
    Connected Port Number: 7(path0) 
    Inquiry Data: ATA     WDC WD2500AAKS-03A01
    I just upgraded the firmware to the latest from LSI website and then rebooted but it did not help either.

    Code:
                       Versions
                    ================
    Product Name    : MegaRAID SAS 8344ELP
    Serial No       : P003883906
    FW Package Build: 7.0.1-0066
                    Image Versions In Flash:
                    ================
    Boot Block Version : R.2.3.15
    BIOS Version       : MT33
    MPT Version        : MPTFW-01.18.170.00-IT
    FW Version         : 1.12.230-0598
    WebBIOS Version    : 1.1-33i-e_11-Rel
    Ctrl-R Version     : 1.04-019A
    Thanks
    123finder.com - Browse/search 4-char domains or dictionary domains

  8. #8
    Also a related question. I have 2 servers that are supposed to be both RAID 10, same controller model but they show different setting. What is the RAID level for these?


    Server #1
    RAID Level: Primary-1, Secondary-3, RAID Level Qualifier-0

    Server #2
    RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0

    Thanks
    123finder.com - Browse/search 4-char domains or dictionary domains

  9. #9
    Re: Disk Cache Policy: Disabled

    This means the disk cache *internal* to the hard drive is disabled. This is different from the disk controller cache. On the Hitachi hard drives on my system, the hard drive internal cache has *NO* battery backup.

    On my system, enabling hard drive internal cache significantly helps performance during heavy write activity, BUT causes disk corruption if you have a power failure during heavy write activity. If you have filesystem based journalling, it can sometimes recover from these corruptions.

    Ben near DC

Similar Threads

  1. CentOS 5.3,vsftpd, high iowait, high LOAD :(
    By MasJ in forum Hosting Security and Technology
    Replies: 25
    Last Post: 09-28-2009, 06:55 PM
  2. Which RAID setup for high IOWAIT problems
    By kafloofen in forum Dedicated Server
    Replies: 4
    Last Post: 01-20-2008, 01:55 AM
  3. raid card to help with iowait?
    By DeltaAnime in forum Colocation and Data Centers
    Replies: 8
    Last Post: 12-18-2007, 10:11 AM
  4. high load/high iowait/ once a day
    By dsotmoon in forum Hosting Security and Technology
    Replies: 13
    Last Post: 04-11-2006, 06:39 PM
  5. Download/Upload server, IOWAIT problem, RAID??
    By gurika in forum Dedicated Server
    Replies: 13
    Last Post: 10-09-2005, 10:29 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •