I have 3 dedicated server all with software raid-1 sata, centos or cloudlinux and cpanel.
To all of them the last 10 days or so during cpanel incremental backup server has high io usage and hangs out.
No server is overloaded with many account's (10 accounts -60 accounts and 300 accounts average each server).
To the one with 300accounts ( ex-6 hetzner) i added a 3rd hd for /backup didn't see any change.
I could check for hardware failure but to have the same issue to 3 servers and be hardware failure i would hit jackpot
I asked the techs here and the first thing that came to mind for them was possible upcoming hard drive failure. They recommended that you use the command-line tool for your RAID controller to monitor the status of the drives closely.
They aren't 100% sure that its hardware failure because of the fact the IO load isn't evenly distributed, but they say something as little as a slightly bad SATA cable could cause such IO wait under load as well.
Hopefully this information helps in some way. :-)
Brooke Jaeger - Zee-Way Hosting
Shared Web, Windows/Linux VPS and Dedicated Hosting since 2002
CloudLinux. Ksplice. Raid Protected. Internap FCP Optimized Network.
Quick links: Web Hosting | Dedicated Servers
I would expect cPanel backups to cause some I/O issues on a RAID 1 box with normal web server activity on the server as well. You may want to try and use "ionice" for changing the I/O scheduling priority for the problem process/service. As you have mentioned, it's most likely not bad hardware if all servers are affected.