Inappropriate behavior by some colocation companies
I have used colocation of around 20 servers with Savvis Corp. for nearly eight years now. I had thought them a professional and helpful part of a web-related development effort I have been working on during this period. But a recent event has utterly shaken my faith in this company and revealed a "snarky" side of them I never suspected. I am posting this to caution others to be careful in choosing colocation companies, and to be careful when interviewing them to ask plenty of "what if"-type questions.
Before considering using Savvis or any other company, you might wish to consider a recent incident my company experienced with them that demonstrates not only a complete lack of understanding of physical security on their part, but an apparent desire to punish their own customers for entirely fabricated reasons – seemingly, just because they can, and in some perverse way, they apparently enjoy making life difficult for long-time customers. Sound like harsh words? Consider what happened to provoke them and I’ll let you decide:
We initially located several of our servers with Savvis eight years ago, and doubled their number about three years ago. At the time we first moved in, key company individuals were given high-security badges keyed to our palm prints, and their site has badge and palm readers at every point of access to the “cages” where individual companies keep their servers and related equipment. So far, so good - excellent physical security. They also gave us one copy of the key to our cage, which we shared among the members of our company authorized to work directly with our servers. Savvis keeps two additional copies of the cage key (as well as a master key for themselves, of course), which authorized members of our company can check out on those occasions when we have an emergency that requires rapid access to the servers without taking time to retrieve our copy of the cage key from our company offices.
I had such an incident occur three days ago, where I needed to get to Savvis as rapidly as possible without taking time to get our copy of the key. So I checked out a copy of our key. The problem I had to solve was difficult, and took a long time during which I did not break for meals. Also, because the inside of the building is shielded for RF, I was constantly having to run back and forth between the cage and the parking lot to communicate with another of our developers until I was certain we had dealt with the problem. By the time we decided the problem was fixed, it had been 8 hours, I was hungry, tired, and without thinking, simply jumped in my car and drove home, where I had to prepare for and then leave for an evening engagement to which my fiancé and I were committed. Cell phones were not allowed to be on at this event. When it was over, nearing midnight, I turned my phone back on only to discover 8 messages from Savvis informing me that because I had removed a cage key from the premises, which was a “serious breach of security”, they were going to immediately chain and padlock our cage until it could be rekeyed, at our expense, “to protect our security”. Even though I agreed to return the key to Savvis first thing in the morning (a very long drive in Los Angeles traffic), they insisted this had to be done for our protection.
I reminded them that we possessed our own copy of the very same key which had been removed from the premises, and had had it in our possession for eight years, and that situation had never been considered a “serious breach of security”. So what exactly was it they were protecting us from? They had no better answer that that it was policy whenever a key was removed from the premises. Again, I reminded them that they key in question had been constantly “removed from the premises” for eight years, so where exactly was the problem? So far, all I have received is the same response.
What does this behavior remind anyone of? And for those well-versed in security technology, what does this say about Savvis’ understanding of physical security – something one would expect them to be absolute experts in? Something is not adding up here. My impression is that they are acting like the school bully, who gives people a hard time just because he can get away with it.
But we are their customers! This incident has irretrievably damaged my trust in Savvis and in their ability to work with customers in a sane manner. I would recommend to anyone else considering using them that if they are capable of this type of arbitrary, pointless behavior, what else will they attempt to get away with?
Is there anything in your contract on a borrowed key leaving the building and what will happen? If not I would argue your contract vs "policy".
Cell signal has been pretty sketchy in a lot of facilities I have been in. Depending on the noise level in the facility typically you need to leave the cage to make any calls. You could always consider a voip solution plugged into your network, if the facility will allow this.
I've dealt with maybe 7 or 8 facilities in my time and some of them are maddeningly difficult to deal with.
In one case we had 1 suite which filled and needed a second. The facility was full so we instead leased one from a middleman company.
Regardless of this fact, we all had equivalent access to both, with the Data Centre permanently holding access passes which were checked out to authorised personnel on request.
The facility had 2 entrances, one for contractors and one for customers. Again for policy reasons unknown, they insisted on holding one pass at the contractor entrance, the other at customer entrance. We had no luck whatsoever getting either moved regardless of the fact that the two suites were identical other than the second being in a service provider's name. The suites were entirely occupied by us and the access lists identical.
We had cross connects between the suites, and equipment across both. Getting from one to the other involved going down to the lobby, checking in the pass, exiting the building, reentering via the other entrance, ringing a bell ao the SAME person would walk through a doorway to the other entrance, and issuing the second pass. While the entrance was on the opposite side of the building, the reception area was linked.
Whilst once or twice we got the nicer guards to relent and go fetch the pass 10 metres away for us, we never ever got away with both passes simultaneously for one person. Despite having the access. Yet if 2 engineers go, we could collect one each and it was no issue.
Oh and in response to the Savvis incident you described, that's completely over the top. Tired engineers take keys home by mistake all the time. They have biometeic sensors and can lock the rack with their remaining key, so I can't see how unauthorised access was going to happen.
I'd be having a chat with your account manager. While most of these places won't relent on bad policy, in my experience acct mgrs have the most chance of forcing reconsideration.
I totally forgot to mention that I had left our cage locked. I never leave its immediate facility without locking all the doors to the cage. So I *really* don't understand what they thought they were accomplishing with the chains & padlocks. Of course, neither do they, as it has become apparent.