Versehost (http://www.versehost.com) and it's customers sites have been down for over 2 hours now - does anyone know if there is a problem in Atlanta or something?

[admin - can you move this to the resellers forum....sorry]

We're experiencing server problems; kernel panics. Replacing kernel right now but is taking longer than expected.

Regards,
David
VerseHost

How long do you expect? No data is lost I hope.

There's no estimated time of repair yet - no data is lost.

David, there must be some sort of update at this stage.....

The technicians at the datacenter could not fix the system, partition errors and kernel panics.

We are re-loading/restoring the system. No user data has been lost.

Everything will be back up by tonight - we will be issuing credits for the downtime.

Regards,
David

Okay.... thanks for the update. Obviously I've lost a fair amount of money from my sites because of this so I appreciate the credit.

I just hope Google isn't crawling TOO much today.....

....and please keep us updated if things change today.
Glenn

I know this is getting fixed but I have to say, I find this damn annoying. No apology. 6 hours down....

:mad:

Originally posted by dside443
The technicians at the datacenter could not fix the system, partition errors and kernel panics.

We are re-loading/restoring the system. No user data has been lost.

Everything will be back up by tonight - we will be issuing credits for the downtime.

Regards,
David

I always feel for host who go through this (we all eventually do). Stressful, pain in the ass. Especially on a holiday.

At least you know your host does backups and can at least prevent your data from being lost.

I agree.... It must be frustrating for David.....as it is certainly getting frustrating for me. As a reseller......in an already tight market.....complications like this do not help, nor give confidence to potential/current customers. So it is bad for David, and for all of his customers as well.

As for back-ups......Yeah, I guess I am lucky that no data was lost. However....I do have a complete back-up on my local machine. When an outage starts lasting this long......ya start to wonder. How long does one wait? What options does the host provider have? When do they decide to move the data over to a different machine?

Or....when do they e-mail customers and let them know whats up? A couple of us are fortunate enough to frequent WHT, and have an idea whats going on...but the rest?

Not trying to rag on David.....am your basic satisfied/flexible customer. But bungie chords can only stretch so far.....

....still hoping my account starts responding soon. :)
Glenn

The server is online, we just need to restore user files and config files.

We're working on it.

Sorry for the inconvenience.

We will be doing a damage-cost based credit issue for each site/acct. That means depending on how much $$ you lost during the downtime, that much PLUS a prorated amount of the hosting would be credited to your VerseHost balance. For some of you, that may mean that you'll get your next couple months or even years worth of hosting for free.

Regards,
David
VerseHost

thanks for the update David. Probably not how you planned to spend your day eh?
Glenn

and these guys (versehost) provide dedicated servers or colo, yeah?
no?

The server that was affected was at a remote datacenter which we have no direct access to. Our co-lo and dedicated users are mostly in the Hurricane Electric facility, seperate from our web hosting servers.

If something had happened in our home facility, it would have probably been much easier to fix - but we've been relying on another company's dedicated service (which is pretty good) for the web hosting.

Regards,
David
VerseHost

The server that was affected was at a remote datacenter which we have no direct access to. Our co-lo and dedicated users are mostly in the Hurricane Electric facility, seperate from our web hosting servers.

I don't understand why Versehost's main site isn't on a seperate server. To have the main company Web site go down along with every reseller account on Server 1 is terrible

but we've been relying on another company's dedicated service (which is pretty good) for the web hosting.

Pretty good?

The server is online, we just need to restore user files and config files.

But I thought you didn't have a backup server set-up yet?

This is very painful. A big potential client of mine was due to visit my main company site this morning and I've had to explain the server is down. How utterly embarrassing.

Of course, I understand I don't get 99.9% or 100% uptime for the price I pay - but over 15 hours downtime is terrible.

David - I need an update on the current situation.

stil no change ...

my domain down...

both DNS servers down...

vershost.com down...

it's getting rather ridiculous..


finally got my domain back up last week and now this happens.. geesh..

/Magnus

Obviously we are not being told the full story here.... David, where are you?

Great. Another c-i-h-o-s-t one-week downtime

An update would be appreciated.....and a revised ETA. Obviously having it running again last night did not work out. What is going on?
Glenn

To be honest Jeep Guy, I wouldn't be surprised if Versehost doesn't come back. I think it's time to make other arrangements.....

Ay, first the cPanel licensing issues and now this, all within a week of signing up. Not to mention tech support being less than promised, and the fact of this being the only communication on VerseHost's downtime that I've been able to find. So, about 24 hours of complete down time + 24 or so hours of cPanel downtime = definitely not fun. Whatever happened to what you told me about how you charge enough so that myself and other customers can rest in peace, knowing that you'll have and will use the money available to deal quickly with outages, David? Or do you consider this not severe enough and thus are trying to save money? I don't call that very good service.

It's a hassle, yes... but I'm seriously considering switching to another host also. I've been patient. I understand everyone has technical problems every now and then, however I would like to be aware of what's going on.

The least VerseHost could do is give us some type of ETA as to when this problem will be fixed. I feel like I'm being lied to.

Yup, moving so soon is not what I planned. Unfortunately, with the lack of information, it may be my best alternative at this point. I would have to have a very compelling reason to stay with a host that is down for going on 24 hours.....with hardly a peep.

I guess I need to go through the unpleasant hassle of finding a new reseller host now. :(

Glenn

You have to wonder why both name servers, the company's own web site, and all its resellers, are apparently all on one box.

Even if they come back up, does it make sense to stay with a vendor with no apparent redundancy?

It's suprising how many hosts do not have redundancy.

I personally took over 2 months in planning and preporation alone before I even ordered my first server.

We have now had our server online for an additional 2 months and have just about got everything up-to-scratch before launch. (including the introduction of remote backup servers etc) - still working on instant activation and billing before the date-of-launch. (but thats irrelevant)

I sometimes wonder why hosts jump into the deep end so quickly without apparent planning for worse case scenarios. I feel for versehost, but dont give up hope yet. People learn from their mistakes, and im sure Verse has had their biggest lesson yet.

I wouldn't be suprised if they introduce a complete backup solution within the next few months.

I know its hard for you guys, but I can assure you that if you take how you are feeling right now, multiply it by the amount of customers they do have and you will have exactly the feeling versehost is experiencing toward their host (or themselves or who-ever attributed to the problem) :D

Yes, if VerseHost plans to continue and if they're putting a lot into the hosting business, I'm sure they're feeling it with the problems now and the customer complaints. As a customer, I might even feel sympathy and be more patient with them if not for a few things:
1) Absolutely no contact, e-mail message, or anything, was made to announce/acknowledge the problem and let customers know it is being worked on. I think if this step had been taken, it would have boosted the trust level a good deal.
2) After initial lightning-fast, complete, and professional responses to pre-sales questions (one of the big reasons I joined up - I expected the same response to continue), I was disappointed with tech support and, as evidenced by complaints on the VH forums, it seems that the lightning-fast responses have ground to a halt.
3) In contrasting his higher prices to the very cheap prices of my old (current) host, David effectively promised that my money was going towards peace of mind, because instead of having just the bare minimum to maintain a server, VerseHost would be charging enough so that they could deal with outages like this quickly and effectively.
4) The backup server is supposed to be online within a week, according to what David told me in an e-mail about month ago. 5/4/2002 - "...we have made
arrangements to bring online a new server within a month..." But even when this comes online, he says there will only be one backup per week, and I've recently discovered that SplashHost does daily backups to tape (my current host does weekly, but my previous host did daily), so I'm considering moving to them, as their customers seem to be very happy. They have comparable prices, and daily backups let me rest much easier, and are much more reasonable, I think.

Still no communication! What is wrong with this company? Will we see them rise from the grave? I doubt it now. I think it's goodbye to Versehost and to my data.

What a waste of 4 weeks.......

(and if that's not bad enough, my ADSL company has gone down today too........)

What issues like this point to is a complete lack of business planning and capitalization.

The old saw - it takes money to make money - is not that far off. If a vendor is so poorly capitalized (not referring to VerseHost specifically here, just talking in general) that they can not afford

- quality hardware
- quality networks
- regular backups
- redundancy
- trained, experienced staff

Then they really have no business being in business.

After all, how expensive would it be to at least have a backup box? If a vendor is so undercapitalized that they can't afford even a simple level of host redundancy then when things go wrong, there is no easy recovery.

Perhaps the law of averages have allowed more people than not to succeed in this business without being properly funded, living by a wing and a prayer until they are profitable enough to fund things that should have been in place on day one.

I must say this business strikes me in the same way as many day traders did / do -- high hopes of easy money without being properly trained and funded, followed by a high rate of washout.

Yes, its very much the same!

Sorry about the short responses - I've been very busy trying to get everything together.

Here is what is happening now. The technicians at the datacenter were unable to correct the Kernel problems and possible partition table damage, causing a delay. Last night, we had restored the server with a new operating system and re-installed vital system applications and server software. We have all of the user files backed up on another hard drive.

We are unable to mount the backup hard drive however we try. We have tried reading it from Windows using a special utility, mounting it as a secondary and slave on a server, attempting to boot with Linux and mount it to a temporary directory. However, all of our attempts have failed. We are pretty sure the backup hard drive is not damaged but at this point we are not positive about anything. This happening on Memorial Day holiday made everything even worse.

This is one of the worst things that have ever happened and we will make sure this will never happen again.

We understand that you may have lost confidence and trust in us because of the lack of communication. However, please understand that we have a limited staff and we were focused on correcting the problem at hand.

For many small hosts, this would be a total disaster and the host would go out of business (example Availhosting). This is understandable and is caused by human nature. From your viewpoint, you may be thinking that you are being lied to, we are ripping you off, etc. ---- but please understand that we're suffering just as much as you are. Please help keep us from becoming another defunct host during this time of disaster by supporting us and keeping your account with VerseHost. We are not trying to "screw you over" - this has been a real disaster for us and our customers too.

We will be issuing credits and refunds as needed soon - a form will be brought up for this. We will be issuing credits in proportion with damages caused. For example, if your web site makes $80 per day from advertising revenues and you have a reseller basic account ($25), we will credit your account with $40 + $25 = $65. This is our temporary plan and this may change.

We are still working on restoring the files.

Regards,
David
VerseHost

Also, if anybody wants to contact me about anything, my ICQ UIN# is 129255932. I will be online to answer your questions.

David, I can understand you are going through hell but I just cannot afford to keep my business offline for 2 days. I don't hold any grudges - I just want to see things kick back into life.

When will you know if the backup data is there and okay? I really need to Tar up my main sites data....

No grudges here either, but support and communication is extremely important to me as a reseller. So I must pack and move on.


:bawling:

Originally posted by UmBillyCord


I always feel for host who go through this (we all eventually do). Stressful, pain in the ass. Especially on a holiday.

At least you know your host does backups and can at least prevent your data from being lost.

Agreed.

VerseHost were working on fixing the server and you guys are accusing them of non-communication. I know that if we had a huge problem like that, we would probably do a short sentence and copy and paste it into every (dozens) of trouble tickets just so we could get working on the issue at hand.

I believe Dave and his staff were trying to help his customers by getting all the sites up as soon as possible, and not wasting time replying to emails, help desk tickets, pms etc or else they will never get anywhere.

Helpdesk doesn't work, emails to support doesn't work as well.
So, the reason most of us are complaining is that we were not told what the situation was until we started to complain. Perhaps an email using a different means would have calm most of us.

And no I don't have or subscribe to icq. Only email

VerseHost were working on fixing the server and you guys are accusing them of non-communication. I know that if we had a huge problem like that, we would probably do a short sentence and copy and paste it into every (dozens) of trouble tickets just so we could get working on the issue at hand.

I believe Dave and his staff were trying to help his customers by getting all the sites up as soon as possible, and not wasting time replying to emails, help desk tickets, pms etc or else they will never get anywhere.


Listen. There was a total lack of communication for over 12 hours. The sites have been down in total for over 30 hours now. There is no support tickets because the main Versehost site is down. There is no e-mail because the main Versehost site is down.

Communication is never a waste of time.

Originally posted by AcuNett


Agreed.

VerseHost were working on fixing the server and you guys are accusing them of non-communication. I know that if we had a huge problem like that, we would probably do a short sentence and copy and paste it into every (dozens) of trouble tickets just so we could get working on the issue at hand.
That's a good thing, certainly, but...

I believe Dave and his staff were trying to help his customers by getting all the sites up as soon as possible, and not wasting time replying to emails, help desk tickets, pms etc or else they will never get anywhere.
Except that there were no "emails, help desk tickets" because all those resource were unavailable due to the main VerseHost site being down as well! The only communication was from the thread brought up here, and obviously he or someone else took the time to find it. All I'm saying is that it would have been.. Not great, but would have made the situation more ACCEPTABLE if there had been an announcement made via e-mail or some such to let all customers know that there's a problem and it's being worked on. So, instead of taking the time to browse WHT, an e-mail would have been better. Imagine what the customers who don't know about WHT are thinking?

That is to say... I agree with jmc and Timbo :P

I hope Versehost pulls through tonight - I really do. I hope things improve.... I'm sure every host has these problems from time to time.

But for me, I'm running a business and I need to sell to survive.


I tell you something after using C-I-H-O-S-T, Aletia, Weinbar, Liquidweb, Webfusion, Hostsave and Addr I *need* a company that performs!

Sorry about everything.

The disaster happened at the worst possible time. We were in the middle of re-organizing a lot of things including bringing online a backup support site which would've been support1.cpanel.us (66.250.54.7) in order to allow our users to be updated on what is happening if things go down -- and also bringing online our new free backup service.

The server screw-up hit us at the worst possible time, while we were making changes. As some of you may know (our forum users), we have been changing things to improve service and things haven't been 100% solid here.

I know that at this point, you guys are worrying about the downtime and if we will ever be back. The server is in fact online, some users may not be able to access it (recent DNS changes). It has been online since last night, however, we are still working on restoring the backup files.

Regards,
David
VerseHost

The disaster happened at the worst possible time. We were in the middle of re-organizing a lot of things including bringing online a backup support site which would've been support1.cpanel.us (66.250.54.7) in order to allow our users to be updated on what is happening if things go down -- and also bringing online our new free backup service.

That would of been excellent :(

The server should be up and working shortly, but we are still working on it.

We will be doubling the bandwidth for existing users affected by this incident and users affected will receive 2 months for free. If you have a legitimate reasons, we will be offering additional credits (form will be brought online later).

Regards,
David
VerseHost

That's a nice recompense. How long will the doubled bandwidth last?

The primary factor, however, in deciding whether to go on using VerseHost would be what provisions you will make against any future problems of this (or any) kind, and COMMUNICATING what those provisions will be. You've failed to answer what your customers marked as their primary concern with this situation - the communication. What will you do to improve it, we want to know.

Also, you say the server is online, but why can I not access my site via its IP address (66.250.46.50)?

I know this is a bad situation for all but I can tell you David is doing everything he can to rectifiy the situation.

That doesn't help some you after the fact, but look at what David is doing and what he is offering.... MORE than just a credit and refunds. You won't see that usually being offered without asking for MORE refund that what you've paid for your account.

I am the one that has been working on this hard drive and the data is there and we will get it off but there's some corruption here that is causing it very diffcult to that but I'm confident we will real soon.

Hang in there with David and you'll be glad you did even if the last couple of days have not been good.... and switching to someone else won't be a guarantee that this won't happen to them at some point in time as well, so give him just a little more time this evening. We've both dropped everything we have to do to help all of you.

As I said, it's a good recompense, but as I also said, that would not nearly be the most important factor in choosing to remain with VerseHost. The most important thing is what will be done for the future. Certainly he shouldn't take time out of repairs to post all his plans right here, right now, I was just commenting on this and also on the fact that he failed to address the communication problem.

I agree with others on how they feel about been kept in the dark. Until just a few minutes ago, I was mad pissed about Versehost being down. Just like everyone else who were affected, I had no idea and even called to leave a message over the telephone. However, upon calling on the telephone again this afternoon, I got a recording to come here to find out the status. This makes it better as I know that Versehost is still in business and around.

I have a mission critical application with Versehost (I am sure all of us here do), and there is money and users lost as a result of this. However, knowing that nothing is perfect and that no hosting provider has bent over backwards over this period of time that I've been with them (3 months), I find it more forgiving to give David and his team a break.

All I want now is for my site to be back up (yes, my IP works now http://66.28.132.100) and the good thing is I host my own DNS and was able to point my site to a temporary server location so that my users won't be completely in the dark but they are not getting the features they paid for during this down time! Angry users on my end is something I have to deal with and it didn't help being hit over a holiday either when most folks use the site!

Lesson: Back up all DB data and files. I know I say that to my friends all the time but practicing it is something else. :stickout

DAVID: THANK YOU for working so hard. Please tell me that my site is fine (an everyone else's too) and that all the data have been restored! My represents $10K worth of development work over 6 months and I can't loose it all!

BC

The compensation is nice but what if a situation like this happens again? Will we be offered a dedicated server or free space for life as compensation?
I don't mean to get sarcastic but this is how I feel.

Thank you for your words of support, Bing. We are doing our best to make sure no user files are damaged.

jmc67, we know that this will be a one-time thing. After backup servers are up, these type of events will never happen again. We were unprepared this time and felt that it was only right to compensate for what we caused. Valuable lesson learned.

Regards,
David
VerseHost

No answer to my comments?

Sorry, I was just looking at the last page.

The server is online but user sites are not accessible.

We will be improving communications many different ways. This includes our coming-up backup support page, and opt-in mailings. Also, we will have a more live information-oriented site coming up where we will provide lots of information for users.

Also, we will be providing emergency contact addresses at our versenoc.com domain soon and possibly live chatroom.

We have other things planned also

Regards,
David

It appears that some of your IP addresses are resolving (although mine isn't yet).

However, I'm getting concerned about the restoration of the backups. In previous posts you've stated that there has been some difficulty.

How is that process going? If these backups don't work, do you have another set of backups, albeit older, that you can fall back on? :confused:

Thanks!
Kevin
Arenafan Online
www.arenafan.com

Okay, right now, we're exporting the /home directory from the backed up data. After this is done, we have to re-upload it and then we're pretty much all set.

Regards,
David

Thanks for the update... really appreciate it :)

Hello,

I just wanted to get everybody's opinion here. VerseHost users, would you prefer that we restore the server first so that you can upload your site, etc. - and then we'll work to restore your userfiles later?

It seems like this may be the best way. Please voice your opinion.

Regards,
David
VerseHost

yes, please bring it back up so I can get something so folks know I am still around.

Just my opinion.
Glenn

Please restore server to allow access to allow me access to upload files, etc. I can wait for user files.

Many thanks.

For me, I need all my files because that is the only recent and most updated version that I have.

Can you please restore my account to what it was before the crash or somehow put it somewhere where I can download a copy of the files and mySQL dB?

http://66.28.132.100/

Thanks,
Bing

I need my music site (IP 66.28.132.202) - this is vital.... The backup I downloaded last week is corrupted. Nightmare city....

I would like to know more about the technical details regarding the server setup.

Server configuration,spec and the total connection speed.

/Magnus:eek:

Hmmm I wonder if my company is the only one to use Load Balancing out there ;-). We have every machine backed up by two other machines. With hardware so cheap nowadays you would think every provider would have this...

Sure would of come in handy here ;-)

David - if the public_html data is corrupted, can you at least give me a backup of the mySQL data?

Originally posted by dside443
Okay, right now, we're exporting the /home directory from the backed up data. After this is done, we have to re-upload it and then we're pretty much all set.

Regards,
David

What is the Status now???

hasn't it been over 45 hours now?

I've worked with servers at IBM before and I really can't understand why this is taking so long - the problems described aren't difficult to correct unless that has been massive data corruption and several backups are destroyed... with a properly set up RAID system I can't really understand what the problem is... And the DNS servers and hosts should really be back online by now...

/Magnus:eek:

Now I am definitely convinced that this is a lost case.

Top priority should be COMMUNICATION!

I'm trying to leave gaps of between 5-6 hours between each post..... Because I'm based in the UK I'm thinking - are they only just awake in America now?

Damn, all I need is the mySQL data.....

Time here now is 11:09 am -500 GMT. So everyone here on the east coast should be up.

with a properly configured RAID system I can't really understand what the problem is

Very few hosters deploy RAID, it would seem. Don't take my word for it - just browse the on-line offerings of many. Very few of the dedicated server plans that hosters / resellers are taking advantage of come configured with RAID. Wild guess - less than 10% of all the commercial hosters use RAID. I could be wrong.

There is something to be said for a three drive system, using the third drive for backup purposes only, especially if it means that backups happen frequently.

Some valid comments and opinions have been expressed in this thread.. Obviously, a worst case scenario was not planned for in advance. This is extremely frustrating for me, having just moved my critical accounts here because of excessive problems/downtime/no support where I had been, only to be socked with this situation..
I can appreciate the hell David is going through, but if I don't end up losing a couple of current accounts out of this.. it will be a miracle, not to mention the word I've gotten about some referrals who may have emailed me only to have their email bounce as undeliverable.
Having finally found this forum, and read all the postings, seems every post from Versehost has implied "it will be up soon".. and those posts go back several hours..
Oh well, it's time to start responding to all the messages on my answering machine and giving my services away in an effort to retain some sort of client base..
Boy, did I pick the wrong time to try to give up smoking..

It's still down. It's been a long time, what's taking so long?

Any update for us?
Glenn

Perhaps if we all start to SCREAM again, we might eventually get a response.

:angry:

Does VerseHost provide dedicated servers or colocation?

If not, then why is this being posted and dragged on in this section of WHT?

clocker, do you have an account with Versehost?

uhm no

clocker1996, as was posted earlier in this thread, the whole of versehost is down. Including their site.

I suggest you spend less time trolling and more time reading.

Originally posted by timbo
I suggest you spend less time trolling and more time reading.

That's right, you tell him! :D

I can't believe this. Another day and no updates. I don't know much I must stress the word COMMUNICATION

My first thoughts are - if a problem cannot be fixed in a 24-hour period then it's a serious issue that will take a long, long time.

David could very well be at the datacenter now getting it all sorted out - he didn't post till about this time yesterday so I'm giving him the benefit of the doubt for another hour or so.

And then I will start to get very, very, anxious.

At this time, we have confirmed that we can restore user directories. This will be done on a request-basis as we have no way of restoring all of the files at once.

Unfortunately, I am sorry to say that we are unsure of the status of the MySQL databases. They may or may not still be accessible.

We are bringing a form online for you to enter details about your site so we can bring back your account.

If you wish to transfer a domain that you own with VerseHost to another registrar, please PM me.

Regards,
David
VerseHost

Unfortunately, putting a form up doesn't help if:
1- the DNS server is still down,
2- the versehost site is still down.

which is apparently still the case as of this moment, anyway.

Ron

The single most important thing for me is the mySQL database and the forum directory for just ONE of my sites as per my private message to you. This is especially important as the backup I did last week timed out and is, argghh, corrupted.

Edit: I also need the news directory too. Forget the 400mb+ Real audio files, they are all backed up on my harddrive.

http://www.iphide.net/restore.html

Please fill this out and we will add your account.

Originally posted by timbo
the whole of versehost is down. Including their site.


and my question remains still

DO they provide dedicated servers or colocation?

Dedicated Hosting Forum
Devoted to discussions related to dedicated server hosting and co-location. Subjects include past experiences, bandwidth and performance related issues and current recommendations.

and

David @ VerseHost Communications
Web Hosting and Reseller Solutions

so i guess not eh?
so i dont see what was wrong with what i said

clocker1996 - take a look at the top. See where it says "WebHostingTalk Forums > Main Forums > Reseller Forum" See my first post asking the mods to move this thread?

This means this thread is in, yep you guessed it, the reseller forum!

Ahem.

Dear David and the rest of the VerseHost Team,

I want to let you know that you and your team did a great job before the long downtime and we will still be a customer of VerseHost.
Please if this happens again let your customers know what’s going on by e-mail.

Hopefully the problems will be over soon!

Best Regards and success,

Albert Dahan, CEO
Branches Online
Rotterdam - The Netherlands

Originally posted by dside443
http://www.iphide.net/restore.html

Please fill this out and we will add your account.

Thing is..... only those who are watching this topic can see what's going on. Without any effort in communicating right now, I don't see Versehost gets any better after this incident in terms of support.

What do you think?

anybody had any luck restoring sql? Every one of the sql dumps I did the day before Versehost went down are corrupt. I guess it is time to dig in and rebuild my Billing Manager, Forums, customer forms, Perl Desk. Ahh......And just when I felt lucky that I had backed everything up. :(

http://www.versehost.com is back up with instructions.

anybody had any luck restoring sql? Every one of the sql dumps I did the day before Versehost went down are corrupt

Strange - that is the same with mine....

I'm sure the data isn't lost - but it's probably a pain in the neck to get it off.....

Thing is..... only those who are watching this topic can see what's going on. Without any effort in communicating right now, I don't see Versehost gets any better after this incident in terms of support.

I really feel sorry for David. Personally, although I have to use a different reseller company in the short term, I hope to return in a few months time once their backup server is running and the cogs are more greased.

The spec and setup at Versehost was very good.....

Would one of you be kind enough to e-mail me david@versehost.com your corrupt SQL database file? I would like to take a look at it for you.

Regards,
David

David, the one that opens is from 13/5/2002 and is 25mb in size....

Basically it timed out when I downloaded it.....of course, I thought it was fine at the time..

INSERT INTO phpbb_search_wordmatch (post_id, word_id, title_match) VALUES (8329, 132, 0);
INSERT INTO phpbb_search_wordmatch (post_id, word_id, title_match) VALUES (8329, 2733, 0);
INSERT INTO phpbb_search_wordmatch (post_id, word_id, title_match) VALUES (8329, 7025, 0);
INSERT INTO phpbb_search_wordmatchConnection Timeout
INSERT INTO phpbb_search_wordmatch (post_id, word_id, title_match) VALUES (8329, 3899, 0);

So really, if you have anything - even if it's a week old that would be heaven.

[ off to bed - I hope to have good news when I awake.... ]

Originally posted by timbo


I really feel sorry for David. Personally, although I have to use a different reseller company in the short term, I hope to return in a few months time once their backup server is running and the cogs are more greased.

The spec and setup at Versehost was very good.....

I feel the same thing. I am not leaving the Versehost yet. But I just thought like email list or any kind of notification is really needed at this point which David hasn't been doing.

David, I just e-mailed you my DB back-ups I got through Cpanel.

Yes, while down time has sucked really really bad....an e-mail list would have been a great tool. My guess is that this type of thing will not happen again in this fashion. Downtime may happen again, but the communication can only get better :)
Glenn

i'm glad to see some movement and that versehost is actually up again but seriously wounded...

yes, we as customers of versehost have alt. emails they should have used to inform of progress...

but this kind of stuff is happening somewhere else as we peek... i plan to stay, thanks D 'n M

:o

Sorry, is it just me or is VerseHost not working again? My site was partially up earlier, now I'm back to where I was last night...

There might be a few service interruptions right now while we restore certain parts and change/upgrade software (CPanel). CPanel license needed to be fixed.

Regards,
David

same here....Just got ahold of David....sound like they are making a few changes...I get the impression it will be back in a bit.

<edit> David posted just before me...LOL>

Ah, okay :)

Well, Arenafan is back up. How long before Cpanel is working again? This took a few days to get fixed last week...

"Invalid License File"

Kevin
Arenafan

DAVID! Cpanel is back up for us.

The latest is that the MySQL database data is irretreivable. This is a catastrophic problem that will set us back massively. (We're in the middle of the Arena Football season, and all of the player stats, game stats, fantasy games, etc... will be RUINED.)

Should we just give up hope and start transferring an ancient version of our database to the site?

Or is there ANY chance you'll be able to recover our MySQL database?

Please say yes.

:eek2:

Kevin
Arenafan

Not heard anything back about mine..... Is it up on a IP address or general location?

It will be beyond a major problem if mySQL data cannot be retrieved....

Mine -- not yet too.

Hmmm... Another day and still nothing. Sooooo I request that my account be canceled and issued a refund for the recent charge.


Thank you!

For some reason, we don't seem to hear anything until after 9pm GMT....... :eek:

eeek .. this is getting heavy

I have 30 or so people whos waiting for their email...

Please hurry up!

/Magnus:mad:

at least another day, maybe?

i am kinda giving up...

anyone has suggestion of another hosting company??
i know that vo is going to come up with bulk plan.
i may have to consider another hosting company just in case this is never fixed.

Our account is back. I went here to re-activate it: http://www.iphide.net/restore.html

But we had to re-upload the files, and we have an empty mySQL database.

David - We HAVE to have our mySQL database restored. PLEASE PLEASE let us know if it can be. Even if you must use an older backup. WE MUST HAVE IT!

As one of our administrators said: "It is your responsibility to work your tails off to get our data back!"

Kevin
Arenafan

Originally posted by Arenafan
Our account is back. I went here to re-activate it: http://www.iphide.net/restore.html

But we had to re-upload the files, and we have an empty mySQL database.

David - We HAVE to have our mySQL database restored. PLEASE PLEASE let us know if it can be. Even if you must use an older backup. WE MUST HAVE IT!

As one of our administrators said: "It is your responsibility to work your tails off to get our data back!"

Kevin
Arenafan

I did request my account to be activated, no reply.
So, does this mean that we LOST the data anyway?
Or is versehost taking so long to restore the user file.

Without any information/update, I can only imagine/guess bad things.

I believe all mysql databases are corrupt. So I wouldn't bet on getting everything back.

There's no estimated time of repair yet - no data is lost.


__________________
David @ VerseHost Communications
Web Hosting and Reseller Solutions


So when did this data become corrupt?

Like Arenafan says, even if it's an older back-up that would make a huge difference......

Certainly I'm not trying to rub salt in anyone's wounds, but in the future I hope that you all take more care in backing up your own work. Search on WHT for threads on scripts and backup and you'll find a number of recipies.

Even if you aren't familiar with shell scripting, most of you are probably familiar with PHP scripts. Here's an example in that language:


$today = date( "Ymd", time() );
$dumpcmd = "mysqldump -uYOURUSERID -pYOURPASSWORD --databases YOURDATABASES --opt | gzip > sql/";
$dumpcmd .= $today . ".sql.gz";
system($dumpcmd);


- outputs something like:
sql/20020527.sql.gz
sql/20020528.sql.gz
...

At this point you can FTP / SCP it or add to the script to mail (be wary of security however) the package somewhere, etc.

I have a script that includes the above snippet in a php file that is run via cron once a day.

Don't rely solely on your host provider for backup!

*grrrrrr*

Thanks for the salt :mad:

Originally posted by dside443
Unfortunately, I am sorry to say that we are unsure of the status of the MySQL databases.
Regards,
David
VerseHost
This post is from a few posts up

mtwatkins,

Salt is good on fries, but perhaps no so on wounds. However, I do agree that users also need to consider having local back-ups as a practice.

I did just that.....the very day before the server went down, I had backed everything up. Was feeling very lucky. I was able to restore all of my files, with the exception of the SQL databases....each of the 5 dumps is corrupt. :( Not much I can do, just hope that Versehost can retrieve some data.
Glenn

Sorry to hear that your dumps were corrupt. Yikes!

I have never experienced that, not in over three years of dumping MySQL databases.

I was careful in my post to show a solution that people could use in the future. We can protect ourselves, and must. Myself I prefer to sleep at night and not worry about machine failures. :)

Sorry to hear that your dumps were corrupt. Yikes!

I have never experienced that, not in over three years of dumping MySQL databases.

Well, mine was corrupted as well. The reason? I used the phpmyadmin tool to download a copy. But because it was a very large file in size (25mb+) it timed out - even though the file didn't give any errors at the time. I only noticed when it came to restoring things yesterday.

But anyway - let's get back to the point here. The data is needed - and we need to know if and when we can download it. If it's totally gone, David needs to tell us today.

I still need my site backup as was requested hours ago.

After all this is done, then we can all join hands and go through the delights of backing up data together.

mine was corrupted as well. The reason? I used the phpmyadmin tool to download a copy. But because it was a very large file in size (25mb+) it timed out

These are all very good lessons for others to learn from. While you have an on-going struggle that is front and centre in your mind - understandably so - there are probably many hundreds of other people with sites at risk for the same reason.

Sharing experiences is what makes WHT worthwhile and hopefully this one will do more than cause people to wipe their brow and think "phew, I'm glad that's not me" - and instead take the time *now* to review their backup procedures and risk.

That's my only motivation for taking this little detour through your discussion right now - the thread is heavily read and its a good time to remind everyone to take appropriate precautions.

This is dragging out too long. Okay, my IP resolves but nothing is reloaded back onto the server. MySQL dB is empty and there is no communication even after my request to restore all my files and dB.

I understand that things are tough but we have clients and customers to answer to as well. Our understandings can only go so far as we're been pressured as well.

Everyone has money at stake here and we need to know if all our files are gone?

Yes, I blame myself for enjoying the holiday and not backing up the files that day...that is one lesson I learned the hard way! :bawling:

David, your assistance here would be much appreciated!

bing: As per every other night, it seems David only posts after 9:30pm here.... Hopefully tonight everything will finally be resolved.

I think we've all heard enough of "and we should be good to go" or "it will be back up tonight" or "Your account will be restored shortly" :(

I can see some people have their accounts restored, but mine isn't, am I the only one who can't access his account? I mean not resolving, no cpanel, no email...nothing.
And of course...no replies.

i was speaking to david and there hard drive is not in raid. RAID can not only be used with SCSI but also ide drives as well. i myself use raid on my pc ( 2x20gig in raid 0 = 40gig storage at twice the speed of the slowest hd) if versehost was to use RAID 1 which is mirroring, they can have 2 hard drives both storing the same data. so if 1 hd failed the other can kick in and take over whilst they replace the damaged hd.

Just a suggestion saying that i got my raid card for £35 (crap but it works.) most motherboards come with raid now anyway for u home users intrested.)

David

jav26, I don't have anything back on Versehost. No reply to my restore e-mails either. I only need *one* site restored. The other 12 do not matter....

No doubt David is flooded with requests and I would like to think he's working through the accounts as we speak. Well, I hope so anyway.

Just a bit of a newbie question... I had purchased my account ("MoogleMB") just a few weeks ago, and so I was lucky, in a sense, to have very little on it. So I don't specifically need Versehost to go to the trouble of doing a restore of what was on my account before the hard drive failure. But do I still need to fill out the restore form (http://www.iphide.net/restore.html) in order for my account to be resumed, or is the form only for those that need backups restored, and the rest of the accounts will be reinstated automatically?

I gather that EVERYBODY will need to fill out the form to get your account reinstated. Just guessing here....but based on the information requested in the form, they also lost all of our account information. I would go ahead and fill out the form, just to be safe.

I am surprised this is dragging on this long now. Heck, I thought 24-48 hours was bad...LOL Trying stay calm here ........sure would be nice to get an update on the SQL stuff....gone or not. If I have to start all over, I will do that on a new account I am having set up today at another host.

As Tom Petty sang....."The waiting is the hardest part".
Glenn

timbo & jav26, I am in the same boat. No access no nothing. Luckily I didn't have much and I mainly worked on my pages offline. After versehost saying that the servers would be up soon and of course not happening and still the lack of COMMUNICATION, I have decided that giving up on them is the best choice. Now I must sit and wait and hope that versehost will refund the recent payment charged to my credit card.

:o

Jeep, take a look at versehost.com. They specifically state:

"MySQL databases are irretrievable at this time"

so I wouldn't count on at least the mysql db's to be restored.

It says at the bottom of the form to e-mail if you only require a backup. And that's exactly what I need. The only thing I would like, in an ideal world, is a .sql backup file and a TAR file of my music site's public_html directory (excluding the real audio files)

Will I get this? :rolleyes:



"MySQL databases are irretrievable at this time"

This doesn't say they are permanently erased. If they are David, just let me and everyone know.

Thanks jmc67.... I guess I was just still hoping, as David said he was not sure....but that they are working on it. I will just figure it to be a total loss at this point.

David........where are ya?

I will post an update shortly.

Jeep Guy: Alright, I've done so, I just hope it doesn't cause any problems or confusion at this stressful time for them.

well, I am quite discouraged so far.. I sent in the form hours ago... David did set up a bare bones reseller account.. but no sign of any data..

I finally started reconstructing from my own backups.. I guess I'm fortunate in that I'm a recent addition to Versehost, so I had most of my accounts in tarballs from the previous host.
Unfortunately, my account was activated without shell access.. so, my tarballs don't do very much for me at this point..
I can't even create my customized packages because the options I offer are not yet available to me to set into the package..

Ron

Lucky you! I sent the form yesterday, when it first were announced and nothing...and I don't mind my data I've got backups of everything ;-) but I can't use them...sigh!

Guys! I finally got my account working!!! It seems it's only a matter of time.
Thank you David, but please, don't touch anything else!!! ;-)

Well done jav26! :cool:

I reaaally need my site up before this weekend..

I think my entire site is less than 10mb in total ..

it's not a disaster if it's gone.. but sure as hell, a pain in the neck..

lost my data from my last ISP a month ago, so that would make it a total of 3 months gone... geesh.. maybe I should setup my own server...

IDE RAID on an enterprise server? only with the Mac Xservers perhaps, not with standard setup.. and btw I wouldn't even think of a setup on an enterprise server for less than 3000$ which would cover a good enough SCSI RAID subsystem..

well that's only my opinion...

I want my site up now, no more excuses..

/Magnus:mad:

I filled out the form yesterday with all my info and my account isn't back up yet, anyone know how long it's taking them to get the accounts back up?

I submited yesterday... and it was back up by 10pm yesterday. I suppose it just depends on what time you sent the form in, I suppose.

All I wanna know is, is mySQL retreivable or not? We need to know so we can proceed...

Please?

Kevin
Arenafan

For now, all we know is this:

MySQL databases are supposedly irretrievable. Will require manual recovery from the hard drive.

/home (public_html) directories _are_ retrievable but will take time to copy over.

Regards,
David
VerseHost

mlorentz - check out a few hundred hosters and I think you will find that the overwhelming percentage do not use RAID of any sort.

MySQL databases are supposedly irretrievable. Will require manual recovery from the hard drive.

When you say they require manual recovery - what exactly does this mean?

Is this something that can be done TODAY?

The status of MySQL databases looks pretty bad. If you have backups, it would be recommended that those be used as we cannot promise that the MySQL databases are recoverable.

Regards,
David

As we've been posting David, many of us don't have a good, recent copy of our mySQL information. Forgetting the "should of" for the moment - can you check specifically if my data is still there? I've been waiting days for an answer on this...

I don't have access to the data myself so I would not be able to check for you. "rastoma" has been handling this situation from the datacenter side.

Regards,
David

David,

I have PM'ed him.... Aside from the mySQL database, I've yet to hear if my other information on the drive is available. Remember, I only want the one account as the rest is backed up safely.

This really is most urgent and I really do feel I've been very patient.

Okay, here's the latest update..

We're recovering the MySQL databases and will hopefully be able to retrieve them by today!

WWWWWWWWWWWWwwwwwwwwWEEEEEEEEEEEEEEEEEEEEEEeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee!!!!!! :)

....Jumps for Joy

This is great news!

David - can you put up the backup of my site (remember, you don't need to worry about the 400 odd meg Real Audio files) on a IP address please so I can FTP to it and download the files I need (the IP address I was using for that site was 66.28.132.202)

Some decent and good news to hear about.

I can't let any of my files or dB go...I'm not a reseller but I do sell services off the site!

Please restore all my HTML/PHP files from backup and also the dB files as well!

keeping fingers and toes crossed!

managed to ping me , so I guess I'm back on ...:D

I hope I didn't cause too much of a stir, but I got kind of frustrated.

well.. if anyone has problems with RAID clusters and load balancing .. give me a shout ;)

and btw. I guess I owe a thanx to David and crew - can't be easy being on the server side of things at the moment, hope you guys get around it soon! :)

:cool: sunshine on the east coast... we are back, thanks dnm

Mine still shows and empty directory and no dB.

:bawling:

I want it up....

Good to hear that yours is fine.

Originally posted by unknowing
:cool: sunshine on the east coast... we are back, thanks dnm

Now backup everything locally! :)

We will be uploading the user files / MYSQL databases shortly.

We are copying all users over to a temporary location so we can back it up and make sure nothing else happens to it. This is taking quite a bit as it is being sent via the internet and the database/sites are quite large.

Regards,
David

I see shell access has been removed?

EDIT: see David's post. ;)

We will be doing an automated by-request SSH access. This is to give us better control. The system will be instant (click, put in username, select reason from dropdown list, and SSH is added).

Regards,
David

Originally posted by dside443
We will be uploading the user files / MYSQL databases shortly.

We are copying all users over to a temporary location so we can back it up and make sure nothing else happens to it. This is taking quite a bit as it is being sent via the internet and the database/sites are quite large.

Regards,
David

Cool...I'm just praying we can be up by June 1st!

Thanks,
Bing

Two questions:

When will the mailman software be restored?

When were the backups taken? After 6pm last Friday?

Here's hoping...
Kevin
Arenafan

That's true - the database and site files for the one I want restoring are quite large.

We need the location to download the files today - obviously with the weekend starting tomorrow it's a top priority to get everything back to normal.

Edit: Just noticed on the Versehost site you mention tomorrow is the day when the files will be available. Will that be in the early AM?

can't recive mails

can send them though..

can't create maillists:

error:

"grnam: error getting gid"

my data was missing - will it be restored?

/Magnus:confused:

Whew.. again, I am lucky in a way.. no databases yet to worry about.. was just about to start setting them up however.. so, some good info was learned here.. (sorry that it was at someone else's painful experience). Hopefully all your data bases can be recovered. (fingers crossed). My clients are again up and running from my month old backups, but better than nothing..

Thanks David for your efforts.. been there, done that myself..
When you can, and I'm sure you are aware already, shared SSL cert needs to be installed for those with commerce accounts.

Ron

Not had any information on the data I need .... 9 hours since my last post - perhaps I should wait another 9.... :eek:

I feel sorry to what has happened to versehost, but at the same time I also wonder why it takes so long to restore files (4 days and counting)?

All the best for versehost and its clients.

cheers,
:beer:

I also wonder why it takes so long to restore files (4 days and counting)?

Restoring over a 9600 baud serial line would account for the time... :(

how do you know it's 9600bps line? and why 9600 bps?

If it's really a 9600bps line (1.2KB/s) then it will take 30 million seconds to restore a 36GB of data, well ... around one year :)

cheers,
:beer:

If it's really a 9600bps line (1.2KB/s) then it will take 30 million seconds to restore

Knew I should have done the math, smart guy! ;) At the risk of staying off topic a minute longer, its just a number I pulled out of my head, since I well remember uploading data and code at those speeds not all that long ago (in human years).

At any rate, I'm sure the users feel like its been a year...

David - progress report please

Will be posting update shortly..

The reason why it was taking so long is because the installation was damaged and was unbootable/unmountable on over 3 systems that we tried. We ended up having to do a manual sector read and export data from diff. operating system (which takes a long time to do - there is a LOT of data).

Regards,
David

David, if you don't mind I'd like an answer to the last pm I sent you or the email.

:(

Update?

Originally posted by timbo
:(

Update?

No ;)

Don't see him online. (MSNM)

It's June the 1st and I'm ready, able and willing to download the backup I need..... Just need the details...

No update yet??

We're working on it.. Have restored a couple users already - but the files are HUGE and taking a while to transfer.

David - as I said on the PM, the directories are empty in the file you sent.... Can you re-send with all of the data. Thanks.

I will, but it will take some time. We are transferring this data over the internet while exporting from the hard drive (which is a slow process in itself).

Regards,
David

Timbo, how was the restore sent to you? Did they restore the site itself, or give you an archive of some kind?

David - Arenafan.com is operational, but don't let that stop you from doing the restore. We need it!!

Kevin
Arenafan

Okay David - Thanks.

Arenafan: As a zip file, so when you uncompress it the directory structure is kept as per your site.

Having some troubles with the mySQL information as it's in .MYD, .MYI and frm format. Shame it's not a simple .sql dump. :(

Arenafan,
Would you like us to over-write your current files and database with the backup?

Regards,
David

David: Yes, just overwrite -- that will be fine.

Thanks!
Kevin
Arenafan

I'm starting to tear my hair out now. It's 10pm here in England and having spent the entire day waiting for the data I need, I've yet to get it. I know you're saying it takes time to transfer data - but 3 days?

Firstly the mySQL database I received isn't working - no matter how hard I try. A simple sql dump would of been fine. Although it's been inserted okay, the tables are all corrupt. Is there NO way I can have a .sql dump of the database???

Then I get a zip file without any data in the directories - just the root files which I already have.

Can you please send me this data by the end of play today (2 hours)

Timbo: Isn't it weird that we only get responses for about two hours a day? Because I wasn't at my computer when David asked me if I wanted Arenafan overwritten, I now get the feeling that I have to wait until tomorrow before it happens. Maybe I'm just imagining things, but this whole process has seemed particularly slow.

If I were Verse, I'd have some hourly guy sitting around on a machine just communicating with Verse users. Answering questions, shouting to David from across the room, "Arenafan wants it overwritten! When can we do it?" And then type in David's response...

Webhosts out there take note! This is what we customers want.

Kevin
Arenafan

timbo -> I will do an SQL dump and send it to you.

The reason why I'm not always here on the forums is because sometimes when I'm working on something (such as directing the remote restore process), I would really like to get it done instead of being interrupted to reply to a post on the forum. :blush:

I will be sure to come back and check on things more often (and reply to your questions).

Regards,
David

Arenafan,

Your backup is enroute and is scheduled to be done before 7:30 PM EST. (Right now is 7:08PM EST)
--

Edit:
Has been done. Please check if it's OKAY.

Regards,
David

David, the .sql dump would solve that massive problems.... as would the public_html file filled with the goodies I need.

I know it's a pain in the arse doing this - and I think everyone here understands the fact it's just "one of those things" - but the sooner I can get this fixed, the sooner I can leave you in peace!

timbo,

If I would be able to do it for you right now, trust me, I would do it for you right now. However, we're having technical difficulties. (nothing is damaged, no worries - just temporarily unable to connect to the remote site where the backup is [not hosted by us])

Regards,
David

Okay..... :crap:

So far Arenafan looks ok. There is lots to check, but I'm pleased! The backups were from Monday right before the crash, which is even better than I expected!

Thanks for keeping the communication going. It is appreciated. I will let you know if there are any unseen problems.

I know we are still having issues with Mailman. We tried to set up some mailing lists, but mailman failed while creating them, so when mail is sent to the lists, they get bounced back.

Thanks!
Kevin
Arenafan

So far Arenafan looks ok. There is lots to check, but I'm pleased! The backups were from Monday right before the crash, which is even better than I expected!

Excellent news! (and yes, it's 1:30am here....)

Okay - I have to sleep now...... perhaps in the early AM when I wake things will be waiting for me.....

People needed to contact Verse host even via PM if nothing else, probably about 10 pages ago. Thread done.