I've been on hold for an 1.5 hours to speak to someone about our server being down. This is the second hour plus I've spent on hold today with no answer. The server has been down over 36 hours to our knowledge, and there have been no status updates on the company website since then. Apparently we have a server lost due to a worm. Hopefully our SQL data has not been permanently lost AGAIN! They had a failure one year ago that burned all our client's data and they didn't have a working backup to restore the data. I advised my client at that time to move off of CIHost but they didn't want to suffer the expense. I believe this was a very poor decision. If one depends on their website for their business I wouldn't recommend CIHost. Their customer service is horrible. They are big but if you have a problem, you're truly in for the most frustrating ride of your life! Not to mention the long long wait.
He! He! My office manager received a call from 'Michael Smith', a customer service rep with CI Host who was calling to see if we were having problems with our email and our web site. She was astounded and almost asked him, "Don't you know?!" She told him that I would call him back but he said I could not and asked for my phone # instead. By the time he called me, he had morphed into 'John Smith' with an accent tht was distinctly South Indian. He told me that no data was lost and everything would be back up by tomorrow. Yeah right. All I need is my DSNs to be created (Access databases) for which I have been waiting since Sunday 1 am and which were promised within 'one hour.' It is now Thursday. I told them I had my web site and data backed up - all I needed was an empty web site on a new server with my DSNs. I am still waiting.
Oh! And John Smith was calling from Chennai (Madras), India. I asked him not to waste my time unless he could set up my DSNs. He passed the phone to 'David' who assured me he would put in a ticket and escalate it. Ha! Ha! Ha! I have already escalated my tickets so that now the 'President' of CI Host is looking at them. Yeah right!
I wish I could take the situation into my hands and move to a dedicated server ...
The “Witty Worm” struck the Internet early Saturday morning causing massive damage to hundreds of our Windows-based shared hosting and SQL servers. The results of the worm left some servers untouched and some servers completely destroyed.
Our Global Response Teams immediately took action setting up procedures to get service restored to 100% of our customer base. This includes your resold Windows and NT shared hosting customers. Dedicated Servers are NOT affected by this worm.
Today the majority of our customers and your customers returned to full service. However, if you have clients still calling you about their website being down, then they were on one of the severely damaged Windows servers. We have all Windows and Security Administrators working around the clock to recover all client website and user data, including databases and email. Some of the most intelligent people in the hosting industry work for C I Host and all of those “brains” are working hard for you and will continue to do so until every client is fully restored.
All servers *should* be restored by the end of this week. For more information on this event please see
Compensation for downtime will be issued for ALL resellers affected.
Thank you for your continued patience and support of C I Host.
Director of Customer Service
PS: We take great pride in our security and our reliability. However, we install industry-leading software to help protect our servers and must rely on the vendors of that software to update it regular. BlackIce (ISS) did not update the software in time for this exploit and our servers and your servers were left in the lurch. We are initiating legal action against BlackIce (ISS) and if you wish to join that litigation please let me know.
Mommy and I are going to sue!
To Our Customers, Rebuilding In The Wake of the Witty Worm Attack:
We, at C I Host, are respectful of the serious nature of attacks and the overall current state of the Internet as malicious attacks increase. Our concerns are for our customers, our business of course, and the economy in general.
When will it all stop?
On March 20, pre-meditated and criminal "worm" attack struck with unprecedented malevolence, seriously affecting C I Host, its partners and many other Internet companies. In recent months, together, we have gone untouched by literally dozens of widespread attacks. Others were not so fortunate in those earlier attacks.
If there is a silver lining, it is that each attack alerts all of us to be more prepared. We are thankful beyond description that because of contingencies, the data from all of our hosted sites was backed up on magnetic tape - 100 percent. We are saddened to hear reports that other companies, using other back up systems did not fare as well.
When the attack first struck at or near midnight, our administrators in several locations worldwide, detected problems and within minutes located the traffic and stopped the invasion. Unfortunately those crucial minutes were enough to deliver a deadly payload. In a sordid twist, the witty worm thrived on a vulnerability of a widely used Internet security firewall. "Witty" attacked and obliterated systems running on Windows NT, Windows 2000 and Windows 2003 leaving nothing in its wake.
Today, most of our servers have been rebuilt from the ground up. The vast majority of our customers' Web sites have been restored to full operation. Hosted accounts have been moved or revived. At first e-mail, and then Internet commerce began to flow, albeit at a measured rate. Then our experts made an important, innovative discovery to quicken the pace. (We will tell you more about that when it is appropriate.) Our goal is to have all shared hosting, Windows clients (minus the 7 servers that are fully destroyed) fully online and functional by midnight, March 25. The last "Magic 7" will be back online before Sunday March 28, 2004 at the very latest. Those servers were so badly damaged all new hardware had to be pulled from inventory and built completely from scratch with Windows OS reloaded, IMail and every piece of software individually installed by hand and then pulling the 120-200GB of client data from tape backup for each server. It has been one of the most time consu!
ming and frustrating projects I have every undertaken.
We apologize sincerely and humbly to all clients who were so adversely victimized, as were we. Some of you already know there a few servers still restoring and some backup content is still loading. It is a drawn out, meticulous process.
There is no way to comfort a business person who has been shut down for several days, due to a calculated and anonymous saboteur. We can empathize because we all were hit together. It is little consequence, but you should know that quick action avoided an even larger catastrophe.
The healing process began immediately. As soon as the attack was stopped, administrators begin rebuilding servers . that very hour. This attack was like nothing we had experienced before. The company went into emergency mode, reassigning all technical personnel to positions that would create the fastest recovery time. If this created longer waits in customer service, we apologize, but made the decision to reallocate manpower to best and most efficiently benefit our customers.
When the dust settles, we will double our efforts to provide faster, more accessible answers from technical experts via telephones and Internet chat. But, until every site for every customer is up and running, our resolve will remain unchanged.
Rest assured my team and myself personally are working has hard as any human can work for you and your organization.
I was working on my site when Witty started taking down C I Host's windows servers so I was able to watch what happened. The SQL servers seemed to go first, then the shared web servers. I have a detailed timeline of the 159h+ hours it took them to restore my data on my website. If you read it you'll realize the company was in complete chaos as the techs struggled to deal with the flood of complaints and CF tried to "spin" the event.
This was their 14 outage in less than 2 years. Prior outages never exceeded the PIA factor of moving, but this one clearly did. I had my domain and site moved by early Tuesday morning. Unfortunataly, my most recent backup was more than 3 weeks old, so I needed C I Host to restore my site to complete the move.
At this point i don't have enough bad things to say about CI Host. Its really my fault for staying with them even after i get treated the way i do (horrible).
All my windows accounts are still down since 3/20 and today is 3/29.
CI Host has the WORST customer service i have seen. After their last issue which had me down for days also, i swore i would move, but i didnt. (damn!)
I was told by 2 different CI Host reps that my account would be up by 3/23, that didnt happen. Then i was told by 3/24 the files should be restored and SQL databases connected, that didnt happen. They outsourced the call center to India, which they could even help you or transfer you to a tech support rep at CI Host.
What REALLY ticks me off is that in the middle of all this they still bill me for the accounts that have been down for more than a week.
i learned my leason though, bye bye ci host! I know me and my petty 12 accounts mean nothing to you but they will to someone else.
All of 5 accounts are down since last Saturday the 20th. It has been more than 10 days now.
After 10 days what they recovered is a blank sql database with no tables in it.
Despite my several requests they are still unable to provide the backup which they do "daily". Yeah, right.
They don't have enough resources to handle the situation. Do you know how many sql servers they have in this mess. Only 9, yes only 9 of them. They are still unable to restore these. I mean this is ridiculous, they can't restore/fix 9 servers in more than 10 days.
funny.. I got a phone call from a customer service rep that my websites will be up by this Saturday afternoon and again no, nothing happened and it is Monday night. I wonder why did she even bother to call me when she had no idea what was going on.
I lost major data, according to one of my clients he had 600 hours worth of data in the database. That is only only client. My other client lost 2 months of data entry work.
CIHost has lost all reliability now. I am moving my 6 accounts to someone else and my data is being held hostage there.
I understand the situation that the worm attack was destructive but don't we have "daily tape backups" specifically to handle this sort of situation.
To me they are worthless now and I will tell it to everybody. They costed me and my clients thousands of dollars and I will tell everyone not to host with CIHost and I will go to every extent I can.
Has anyone of you recovered any of your sql databases from these jerks?
I just called because we have a non-profit site on their windows servers and it looks like the site is back, but the DSN is not set up. The automated call said, "there are 18 people ahead of you and the average hold time is 6 hours and 21 minutes." Impressive.
I am tired of the Mushroom Farmer method of customer service: Keep me in the dark and feed me bull sh*t.
My site has been down for 11 days. Current Tech Support hold time is in excess of 7 hours and my wait on live "Tech Chat" gets longer as the number ahead of me grows constantly as well as the time.
This is the note I sent to CI Host:
My website with CI Host has been down 11 days so far. At first (Friday) I was told that CI Host was being hit with a DoS (Denial of Service) attack and that they expected to be up and running soon. Saturday morning I was told that one of the servers was hit with a worm and the remaining servers were taken offline for examination and patching. I was assured that the service would be restored shortly and that no data was lost. The following day (Sunday) I was told that the servers were being returned to service as quickly as possible, but each one had to be “patched and updated” separately which was taking time. I was told that service should be restored by the end of the weekend. I received similar information and promises until an email from the CEO expanded the scope of the problem, blaming the Witty Worm for widespread destruction. Twice since then I have received calls from CI Host, in response to my emails, promising to have me back up. On Friday (3/26) I was told that my shared server was next in line for return to service in the next few hours and then on Saturday (3/27) I was told they were “writing the scripts” and my website would be back up by Monday (3/29). I filled in their online form twice with all the necessary information so that my database could be reconnected. As far as checking that the files were indeed reinstalled, I could no longer access my directory to check. Last night, their Technician “Mikey” during a “Live Chat” entered the appropriate information to re-establish the DSN connection. When he said it was complete, I asked him how come my site was not back up. He stated that all TS will do is setup the DSN and verify the connection. I asked him what am I supposed to do now and he said “start with the error messages”. I stated that all I know is that my website was installed and had been working fine and that service was interrupted supposedly for a worm that was already made public with a patch readily available. (CI Host has already discounted this information and I do not know who is telling the truth). Mikey said that he would open up a “Level 2 Request”. Nearly 12 hours later, I have not heard anything.
I am basically a reasonable person. I just wish they had kept me informed and up to date and demonstrated a willingness to assist me, and then I would perhaps have a somewhat different opinion. I don’t like the “tell them what they want to hear today and (not) deal with it tomorrow” approach to customer service.
Re-open the tickets. I have been getting a failry reasonable response to my tickets. Do not bother to call because you will get the call center who will put in a ticket for you. Save yourself the time and put in the ticket yourself. It helps to just address the problem and leave it at that. Once you have your data back - if you are that lucky - you can then take a decision about moving.
I learned a long time ago to maintain my own backups regardless of whether the host 'includes' it in the deal/contract.
I have commented out a lot of my code as it appears that a couple of DSNs were not created as requested.
I am trying to deal with this as calmly as possible. We lost a lot of money in the 10 days that our sites were down. I am evaluating all options available currently.
I would recommed using the "find a host" feature to start. Folks can contact you that meet your criteria. Then search the boards to see what people have to say about them. In my situation, I need MS SQL and several companies have contacted me directly already.
Changing hosts is no fun, but CI Host is really doing all they can to make the choice to leave easy.
Below is a message that I posted in another forum earlier this weekend. I decided to move it here to see if I could get some good advice/help. Since this post, I have faxed all documented emails to Tamra whom is working to see what she can do. They keep saying that I can only get an account credit. I don’t need several thousand dollars in account credit, nor do I even want to stay with CIHost. Keep in mind that I sent my request to cancel my account 9 days after it was actually active. (this was a dedicated server to recap, purchased as a “crazy deal” that they offer. You have to pay for the year (stupid stupid me!!) I thought hat I could rely on them. I was very wrong in this. Anyway, I’m now waiting for her to call me or email me with the verdict o this issue. I’m just trying to stay one step ahead and keep my options open. Also, we were supposed to get private name servers installed on the server, that was not done either and I already paid for that as well. I even sent a message to ask about this, it too went unanswered. This is the main reason that I am leaving simply from lack of response (not slow response, although that too, but LACK of response). They just don’t seem to care, plain and simple. I can’t build our company working with a data center that doesn’t care.
<_------previous message below------->>
We recently purchased a "crazy deals" CIhost dedicated server, very nice machine, fully loaded... Anyway, not thepoint. The point here is that we have received horrible service and response times on CIhosts' part. They just don't respond to email very well. We have to call them, after being on hold, just to get someone to answer our questions.
So here's my problem. The server has been online for almost 3 weeks now. We have not even used it mainly due to lack of response on their part in aswering our questions that we had about certain issues/topics, software compatibility, etc..
However, the "contract" that they make you sign witht he crazy deals or any other yearly deal at that states that they do not offer refunds on their crazy deal packages. only account credit. Well, a lot of good a ton of account credit will do me at this point, I simply want a refund. Heck, they can keep the setup fees, I JUST WANT OUT! What are my options at this point? IS that so call contract really binding? I've contacted their billing department about htis over a week ago with no response. I have all emails and conversations documented just in case. I just contacted the billing dept. again this evening and hopefully i'll get a response. I sent thema rather lengthy email. They just don't seem to want to response. Very strange for such a "large" company. I'm terribly dissappointed in them. I am based in Downtown Dallas and was looking for a ded provider in the area and have always known of CIhost so wanted to go with them. I'm regretting that decision. I paid them via Amex, so I konw that my card compnay will back me if push comes to shove, but I don't want it to get to that point. I would ike them to say "hey, we are terribly sorry for the service tat you have received up to this point, we can offeryou x or we will gladly refund you your monies and unfortunatley we cannot refund the setup fees" you can the idea. so I need advice here as to how to handle this further? I plan to CALL the billing dept. tomorrow or monday whenever I can get them on the phone and talk to them.
We also have 3 other shared accounts with them and like the person who posted about their troubles with the "witty" worm.. have been offline since last friday as well. We have a couple of company sites that sell online products and have been losing about $900 - 1200 in sales daily on one site alone. What can be done.. really absolutely nothing.
Just another reason why I'm very, very, very sorry that I chose CIhost for our dedicated server. And was stupid enough to prepay for the year because it was hey.. a "crazy deal" they could not have picked a better name. :-)
I personally wouldn't give any advice at this point, being that I cant get then to even talk to me. You may suggest that if you are forced to complete the contract, you will use if for your new website cihostsucks.com.