Well,
I must say I'm very disappointed, and astounded -
My host is going to be down for a week, probably going to look for a new host.

The happenings :

This afternoon at around 1330 ET (1730 GMT) Wednesday 8/28/02, as a directory was being edited, a typo was made which deleted some of the system files. The data center is pulling the backup hard drive right now in order to restore the server to its condition as of the last backup early this morning.

Not good news. The techs at the data center are not able to access the backup files. They are working on this through the night. If they are unable to affect access, we will have to send the drive out to a data recovery lab to recover the data. This could keep us down until mid-week next week. We are already implementing a redundant backup process to insure this type of failure does not happen again. Thank you for your patience.

The techs were unable to access the data. I had given them a deadline of this morning. The hard drive is on its way to a data recovery lab now. This lab does a 1-2 day turn on this type of recovery (non mechanical failure, which takes longer). With today being Thursday and Monday being a holiday, we cannot count on the drive being returned to us until next Wednesday. That means your site(s) will not be back as they were until Wednesday evening

Some have asked about the details of why the backup hasn't been able to be restored. Here they are --
1) The backup is a compressed afio archive, created by a backup program; 2) The backup program has been tested; 3) The server techs installed the Red Hat 7.3 OS on the primary hard drive; 4) 7.3 uses a different file system than the backup hard drive; 5) We believe that when they mounted the backup drive, the file system type header block was corrupted, thus inability of the system to read the drive; 6) That is why we now need data recovery.


that's it folks, funny if it wasn't sad

:D rm -f /* .... oops :bawling:

needless to say sounds like they didn't really keep any backups and I would not hold my breath.. .. sorry to hear about your situation but it is pretty fair to say that I doubt the "backups" will ever show up.. I would start looking now and request a refund..

As usually, there's alot of good webhosts on this board you can look forward for a new home. :D :D

That is bad, and yeah finding a new host would be a great thing to do. But also, always have a plan B for hosting, that way, if something happens you can move quickly.

I offer hosting as well, and too have a plan B. If something happens to the data center, I can hopefully transfer everything in the matter of a few hours. I hope I never have to do that, it is always a good idea to have plan B.

Hello,

I just fired off David an e-mail. Hopefully he can provide a better explaination. I'm willing to wait until Wednesday or even Friday as long as there is appropriate refunds. Otherwise, if they excuses keep coming then I'll have to have a charge back issued on my credit card and a stop payment for next month.

If anyone has the e-mails from Integrahosting could you please e-mail them to mdk14@hotmail.com since my regular e-mail (matthew@advernet.ca) is down with the Integrahosting server. I'm was still in the dark about this whole situation until today. Thanks,

I still would look for a new host. But in anyway, PERL SCRIPTOR, you may want to chage your e-mail in the sig ;)

Originally posted by OKIHost
:D rm -f /* .... oops :bawling:

needless to say sounds like they didn't really keep any backups and I would not hold my breath.. .. sorry to hear about your situation but it is pretty fair to say that I doubt the "backups" will ever show up.. I would start looking now and request a refund..

lol :D

Hi Folks,

We do keep daily backups to a second hard drive. The backups have been tested. The decision was made to restore from the backup drive. The tech at the data center did a fresh install of linux on the primary hard drive and remounted the backup drive.

After reinstalling the backup/restore software, I got that extremely nauseating feeling when it couldn't read the files. Sure enough, we couldn't read the data on the backup drive. After the guys at the data center tried to access the files unsuccessfully, the drive was shipped to a data recovery lab (who aren't working over the Labor Day weekend).

We are staying in contact with the customers for whom we have off server email addresses. We are also posting updates on another server, www.integrahosting.com. This was one of those disasterous double whammys. Had the backup drive been accessible, we would have been back up within 12 hours.

But would've, should've, and could've don't cut it. We are going to be adding to our backup methods once we get this server back up and for the other servers we have.

We would have loved to be able to move customers to another server, but without the backup, we're stuck for now. We have always stood for dealing straight with our customers - and so shall we now.

I must go now. I'm busy getting as many domains resolved to an announcement page as possible.

My alternate email is david@integrahosting.com

More bad news are coming


I am angry now...

In calling the lab I found out they did not receive the drive Friday,
but today. I called the data center and found out that they sent all of the paperwork and everything in Thursday, except the drive. That didn't go until Friday morning.

We were told the drive would leave on Thursday and were told that "everything" was sent. We now find that was not the case.

We are going to do our best to get you web site space setup for a temporary site. You will be able to at least have some kind of site up. But, this will take about a day to get to everybody.We will strive to get each domain a virtual server setup within the next 24 - 30 hours.

There are only 1 and 1/2 of us to work on this right now, so please be patient. We will do this as quickly as possible.


Is it just me or they mean that I should be prepared that the data won't be recovered, after all they could have opened temp sites on day 1, and according to the timetable in 24-30 hours the sites should have been backup! not some limited space for me to play with (maybe put a bigger sign of site down :eek:
It appears that after a whole week of waiting i will have to wait some more :angry:

Hi again, Folks,

On day 2, 3 and 4 we concentrated on getting the DNS restored and an email server working. On day 5 we did troubleshooting for all of that to make sure that our customers had email and an announcement page. We didn't bother with temporary web sites because we expected the drive back on day 7 (tomorrow as I post this).

Opening temp sites without DNS servers would have done no good. Also, almost to a person, our customers wanted their email back first. We have prioritzed actions consistent with the majority of the customers who have told us what they need. Other than things being back the way they were, have you told us what you need to get through this?

We've had to do all of this by hand, since the automation was gone from the machine too. Try creating DNS records for several hundred domains and resellers and their domains by hand. It takes time to do that. So does setting up a mail server for all of those domains too. It has been accomplished as fast as possible.

I will be talking with the tech assisgned to the case every day and posting on our www.integrahosting.com (http://www.integrahosting.com) site. If they think they won't be able to recover the data, we will tell you straight. We always have - always will. You deserve no less.

Should you leave us for another host, remember, this can happen to anyone - check this board out. Always maintain backups of your own site.

Gotta run. Getting those temp sites up for y'all now.

Dunthank, For the time being you may want to change the titlebar on you main page "Web hosting that you'll call Home. Integra Hosting with MySql, JSP, PHP, and Great Service!" at http://www.integrahosting.com/
It may make people real mad to see it for now. I think you have been down for almost a week.

I took your suggest and changed the title. I can see what you are saying. Even though we're knocking ourselves out to get our customers on the affected server back on the air, which we have now accomplished at limited capacity, it is hard to grasp that great service can continue even though a server is offline.

I'd never paid attention to the title, since that is the home page for our Ensim server offerings and I was very busy. We took the main page for the Ensim services down in order to be able to allow our HSphere server customers update information without having to fish for it. But thank you for noticing and pointing it out, it always helps when we look out for each other.

This morning we were informed by the data lab that the recovery was unsuccessful. It is with deep regret that we inform our customers of this. We are starting the steps to rebuild an HSphere server for our customers who wish to remain.

We are taking the necessary steps to ensure that this, nor anything like it, happens again. We will have redundant backups rather than a sole source plus more. Our customers on the server will be kept informed, as they have been this entire time.

We have failed some of our customers, but what we have learned from this will benefit our customers on our other servers that were not affected by this outage as well as the customers who will be on our HSphere based system as we restore it. Failure is never pleasant, nor, on a personal level, do we like to let people down. It is a sad time for both our customers and us.