Hard drive problems due to chassis fan vibration

[wormy]

I've noticed that with the newer 1TB drives placed in older chassis the sustained transfer rates are taking a terrible dip from over 100MB/S to under 40MB/S and as low as 10MB/S even when I run a test on a fresh drive with nothing else running.

Anyone else experienced this issue and more importantly did you find any solutions that worked such as using rubber or foam to cushion the squirrel fans perhaps?

[ramnet]

I've seen this - my solution is to simply use drives that are more tolerant to the vibration.

I've been using the WD Black series of drives and they work well.

For 1TB, something like this

Similar Seagate's (esp Seagates), Hitachi's, Samsung's, etc brands don't last very long in the noisy chassis I'm using (SM-CSE-512L-260B) - the newer revisions of their drives have almost no tolerance for the vibration.

Hope that helps

BTW, What kind of chassis (mfg/model) are you having issues with?

[Dave W]

I have a few of these sitting around:
http://www.wykofka.com/files/hdd.jpg

I have been looking for more

[wormy]

The hard drives I am using are Seagate 1TB Barracudas with 7200.12 firmware. I have experienced this problem in two different types of Supermicro chassis. One is the SC811 1U chassis :

http://www.supermicro.com/products/c...SC811S-260.cfm


The other is a 2U chassis:

http://www.supermicro.com/products/s...YS-6024H-T.cfm

When I use older 80GB and 120GB drives in either of these two chassis the drives do not suffer. Only when I toss the seagates in does the sustained transfer rate go down. Unfortunately I have 9 1TB seagates so I'm in a hurry to fix these problems on a few servers asap. Then I'm going to grab some WD blacks and slowly replace the seagates and toss them on ebay or something. I've had some luck with one chassis which was an SC811S-260 and the drive in there managed 110MB/S so I'm hoping the problem is bad ball bearings on the squirrel fans or perhaps lack of foam cushioning under the squirrel fans.

[ObjectZone]

Just curious how you specifically established that it was vibration? Did you remove the drive from it's mounting and perch it up somewhere outside the same case but still attached to the same controller and try again and got the faster data transfer rates??

--Chris

[wormy]

I've been at this problem for days on and off. Whenever I disconnect the cooling fans and run a test, specifically this one:

dd if=/dev/zero of=/dev/sda bs=1M count=1000

I get something close to 100MB/S but as soon as the fans come on the xfer rate goes to 40MB/S or below that in certain cases all the way to 15MB/S.

I did some googling too and came up with a paper which seems to indicate that the newer, denser capacity hard drives are a lot more sensitive to vibration than older ones. This is from 2004 but still, a trend is a trend:

http://sciencelinks.jp/j-east/articl...04A0418304.php

The Effects of Cooling Fan Vibration on Hard Disk Drives


Abstract;Performance of latest HDDs can be affected by the mechanical vibration of the cooling fan. This phenomenon is often observed in recent low height rack-mounted type computers, which have number of components in its limited capacity of the enclosure. The head positioning servo of latest HDD's are designed considering the effect of vibration. However, we found that the transfer rate of HDD is sometimes degraded by the continuous vibration even when the magnitude of the vibration is smaller than the specified allowance. Moreover, such vibration cause an unrecoverable damage in some cases. This paper introduces such phenomena, and tries to investigate the cause of the phenomena by analyzing frequency response characteristics of the HDD transfer rate in detail. This paper also discusses ways to avoid these problems. (author abst.)

[Dave W]

Is it possible that this is a power issue?

I am familiar with vibration issues but the 7200.11 drives are pretty clean and I wouldn't expect to see anything that bad due to fan vibration.

[wormy]

It is not a power issue. Chassis fans just use about 8 to 12 watts and I have 400 Watt PSUs in all of these 1U servers and two redundant 500 Watt PSUs in the 2U. My gut feeling is that in the last few years hard drives have been getting more and more sensitive to small vibrations while the older chassis were probably not even designed with the lowering of vibration and shock waves being a top priority.

Obviously some hard drives such as the WD black can handle vibration a lot better than others. I should add that I can run a seatools long test and it comes out clean in the end even while the drive is in one of the bad chassis. But sustained transfer using plain old dd suffers tremendously. We had a client complain about this a couple of weeks back which is when I got on the case and tested everything I could get my hands on. At first I thought the lot of seagates I had were flaky but then after swapping different drives between two chassis it became clear that it was not defective hard drives causing this problem(unless you consider a design flaw a defect) but rather the vibration coming from the cases themselves.

[ObjectZone]

Absolutely amazing. I had seen this video a long time ago regarding vibration affecting hard drives, but I never suspected that a chassis fan could be the source of performance issues:

http://www.youtube.com/watch?v=-0gcbhF-ZoU

I may need to do some performance testing on some of my systems to make sure no outside sources are causing degradation somehow.

--Chris

[wormy]

I've seen that video too but only when I was directed towards it while discussing my problem with someone. All you really see in that video are some dtrace charts without any actual data as to how much of a slowdown each shock wave resulted in.

Had I seen that video before building servers with these 1TB drives I would have just shrugged it off as an insignificant problem which may have lost about 2% off the performance of the hard drives affected. This makes me wonder now as to how many massive raid frankenboxes have been built and deployed out there where the performance is being degraded 10% or even 30% with the admins not even realizing that such is the case(pun intended).

For example, a massive raid project such as this would turn into a disaster if vibration was not taken into account but crept in and stole 20% throughput without realizing it: http://www.stringliterals.com/?p=77

Before I build a raid box I am going to test the chassis out thoroughly as part of the process with regards to vibration. This incident really made the effect of shock waves come to life for me. I still have to find a chusioning solution for what I have to work with.

[FastServ]

We have the same exact issue with SM 512L based servers. Anything from terribly slow writes to read-only FS.

The chassis fan are indeed part of it -- especially when stacked on other 512L chassis... but in some cases the disks themselves vibrate themselves silly. Remove the disks and let them run in your hand... if they have a 'hum' type of vibration (corresponding to the 7200rpm, ect), the chassis is going to transmit it to the other disks.

Only solution so far other than switching to a hotswap chassis and/or removing vibrating disks, is to mount the disks with velcro to isolate vibration to/from the disks. Double stick tape won't last longer than a month before vibration is allowed to return.

You can also try reducing the speed of the main blower fan in BIOS, but obviously this is mainboard-dependent and also needs to be used with caution depending on CPU type.

[wormy]

Unfortunately both cases which are giving me problems have hotswap caddies aready. One of the cases does have a 5.25 expansion bay and I will try this velcro trick there. Velcro is also easy to find in stores so it's a quick solution, thanks

I'm also looking around for sources which sell liquids that can cure into soft foam or soft rubber if anyone has leads.

[brc_csf]

Unfortunately both cases which are giving me problems have hotswap caddies aready. One of the cases does have a 5.25 expansion bay and I will try this velcro trick there. Velcro is also easy to find in stores so it's a quick solution, thanks

I'm also looking around for sources which sell liquids that can cure into soft foam or soft rubber if anyone has leads.

We are having this same issue and discussing this at another thread. Just found this thread. Have you managed to fix your problem ?

Would you mind sharing a picture with the a disk mounted with velcro so we can ask our vendor (and IDC techs) to do the same?

Thanks

[wormy]

We are having this same issue and discussing this at another thread. Just found this thread. Have you managed to fix your problem ?

Would you mind sharing a picture with the a disk mounted with velcro so we can ask our vendor (and IDC techs) to do the same?

Thanks

Unfortunately the most reliable solution is to just acquire higher quality chassis that do not vibrate. A lot of times when you get a really old chassis the fans inside the chassis are damaged due to old age and will vibrate a lot more than they vibrated when they were new. I've been able to measure vibration quite reliably with my hands and I've also found that in some cases when you slide the top of the chassis on it presses down on the fan and the vibration shoots up to astronomical levels. If you can wedge a very thin piece of rubber or foam that slightly warps the sliding top cover of a chassis so that the metal does not press down directly on the 40mm or 80mm fans then that *can* sometimes result in a dramatic reduction of vibration too.

I wasn't able to use velcro unfortunately because the drives just went into hotswap caddies. My solution was to replace the chassis fans in the 2U with maglev extra low vibration fans. I would be curious to see how the velcro solution looks and how much space it takes up.

[corNIC]

I lost an entire RAID 10 due to this one time as it caused multiple drives to drop off the controller at the same time. My suggestion like some others have mentioned is making sure you are using higher/highest quality drives. I think there is a feature called TLER or something as well that you really want to have with hardware raid. The WD black series at a minimum. Also consider ordering new fans and ensuring the are mounted properly. Consider racking the server in such a way that is it very secure and tightly fit. If I had to guess this is a 1U box with the crazy 40mm fans .