Web Hosting Talk : Web Hosting Main Forums : VPS Hosting : HOSTV 16 DAY FIASCO AND COUNTING

[gofishch]

Since The Thread that was started earlier was closed (webhostingtalk.com/showthread.php?t=927255)

I opened a new one. HostV Shut down the VPS on the Node 16 and basically wiped out all clients data. This while guaranteeing data backups daily stored for 7 days (hostv.com/features.shtml#5)

They spent the next 14 days making everyone believe that they would fix it. They have hide this in a private thread on the Hostv forums (forums.hostv.com/) SERVER AND NETWORK NEWS that unless you are a customer you can not view. I will be glad to give my user name and password if anyone would like to view the thread and see how badly they treated everyone.

I begged for a new VPS as I was smart enough to have my own backups not only locally but remotely stored. They cost me thousands of dollars and I was only offered a month of free service in which to date we are 16 days into and I still have no service.

They did give me a new VPS 16 days later on a different Node tell me what good that is once I restored my backup these are the results.

#2002 - The server is not responding (or the local MySQL server's socket is not correctly configured)

Web mail results:

DATABASE ERROR: CONNECTION FAILED!
Unable to connect to the database!
Please contact your server-administrator

Also Sever stats show:
mysql (5.0.89-community) failed

exim (exim-4.69-23.1_cpanel_maildir) failed

How is this that they can defend actions saying we are giving refunds and offering new VPS that does not function properly. Where is the solution in a non function VPS. I had a video site running on this VPS that lost all the back links to this site and the embeds form where dropped like flies this cost a lot of people money and yes stupidly the people affected believed the BS that was fed to us even when it was questioned

I was unable to add the direct links since I did not have enough forum privileges just add the www

[ItsJustHosting]

I opened a new one. HostV Shut down the VPS on the Node 16 and basically wiped out all clients data. This while guaranteeing data backups daily stored for 7 days (hostv.com/features.shtml#5)

I am not exactly up to date with this situation, but I highly doubt they intentionally shut down the server and wiped all clients data. They apparently experienced massive hardware failure with that node, which is completely OUT of their control. They then tried to restore client data from backups and they experienced problems with that as well likely due to the sheer size of the data that needs to be restored and the long time it takes to restore it.

I would assume that if you're having problems with your new VPS, you should contact them about it so they can fix it for you. Unfortunately hardware failure can and does happen, which is why you should ALWAYS maintain your own up to date off site backups in case of such situations, so you can quickly move to either another VPS or another provider. This is magnified if your data is very valuable, and if your website generates substantial amounts of income upon which you rely.

[linuxclark]

Originally Posted by ItsJustHosting

I am not exactly up to date with this situation, but I highly doubt they intentionally shut down the server and wiped all clients data. They apparently experienced massive hardware failure with that node, which is completely OUT of their control. They then tried to restore client data from backups and they experienced problems with that as well likely due to the sheer size of the data that needs to be restored and the long time it takes to restore it.

I would assume that if you're having problems with your new VPS, you should contact them about it so they can fix it for you. Unfortunately hardware failure can and does happen, which is why you should ALWAYS maintain your own up to date off site backups in case of such situations, so you can quickly move to either another VPS or another provider. This is magnified if your data is very valuable, and if your website generates substantial amounts of income upon which you rely.

Maybe the OP thinks 1 month of free service credit is too little for this?

[gofishch]

They admitted that they shut down the node and lost the date due to failure. They also told everyone that they had backed that data up. And assured me even when questioned that my data was safe when in fact it was not.

I do have my own remote backups. And have I have contacted them to fix the new VPS problem but apparently they intend to take another 16 days to do it since it as been since 02/15/10 that the request was made. The damage they done to my revenue streams happened in the first 48 hours. My site is video site with scripts set to run on their servers. When the TV channel goes blank you change the channel so if you had any embed videos from my site you got a blank, so of course it take less than a couple of hours to start the bleeding.

They offer the same guarantee (ItsJustHosting) you have in you forum signature 99.9%. So if you are defending HOSTV by all means show me another provider that left their customers off line for 16 days and counting while claming a 99.9% uptime. With lost data and no real support, since they expect support to be done via on email and that crap twitter site. They say only a dozen or so customers where affected, they should have been on the phone taking care of them.

[gofishch]

Originally Posted by linuxclark

Maybe the OP thinks 1 month of free service credit is too little for this?

I think they should get my site back up and working and then dicsuss any other options

[linuxclark]

Originally Posted by gofishch

I think they should get my site back up and working and then dicsuss any other options

As you have backup for your site, Why not get a VPS with another provider, setup the VPS there, just to get your site up and running again, you can alway move back when everything works for you at hostV. 16 days down time is a lot.

[gofishch]

Originally Posted by gofishch

My site is video site with scripts set to run on their servers.

I answered that question. It is not impossible but it is very diffcult to move.

[gofishch]

Originally Posted by linuxclark

As you have backup for your site, Why not get a VPS with another provider, setup the VPS there, just to get your site up and running again, you can alway move back when everything works for you at hostV. 16 days down time is a lot.

Originally Posted by gofishch

My site is video site with scripts set to run on their servers.

I answered that question. It is not impossible but it is very diffcult to move.

[Karl]

If you get managed your new provider will transfer for you.

[Cirtex]

Hi gofishch,


As noted on our forums, on behalf of HostV we truly apologize to all customers on NODE-16 that were affected by this as we always stay by our clients and this situation had affected 2 dozen or so customers total. We will do our best to prevent such incidents in the future and work towards resolving our current external backup server issues as well which failed restore in R1Soft after multiple tries. We had made the promise that all data will be restored originally because the data was indeed shown as being fine and all was up to date, however upon doing the restore we ran into multiple bottlenecks in which we worked very hard to resolve w/ many teams unsuccessfully.

There were successful restores of VPSes on NODE-16, however as of now, if you are still experiencing problems, please email BILLING with your account info, and we'll try to bring back any data possible, however it is best you provide helpdesk with backups that you have, such as uploading via FTP to your new VPS (w/ same ips) and then having our support team do restoration.


How it happened

1) Server NODE 16 rebooted for kernel update
2) NODE 16 hardware raid card RARE* failure causing corruption of all drives
3) We had external backups for entire node-16 so we thought this was no problem.
4) We started restore from R1Soft backup baremetal restore
5) Baremetal restore failed multiple times and we worked w/ R1Soft to get this up and running.
5) Final solution was to do manual restore for per VPS and this took week+
6) After final restore was complete, we ran into more issues, this resulted in migration of all VPS Data to newly created VPS on a NEW NODE.
7) There were many customers affected w/ data loss from minimal to total data loss, but again it was out of our hands that we had failure on both our hardware of node16 and failure on our backup system.


For all affected customers on NODE-16, please also email BILLING to receive credit / refund for this inconvenience caused.

Again, thank you for your patience and we're sorry for the inconveniences caused by this rare situation with hardware & backup failure. The entire team will continue to work towards improving our backup infrastructure and also in restoring all clients affected by node-16. Please do not hesitate to contact us!



Cheers

Originally Posted by gofishch

Since The Thread that was started earlier was closed (webhostingtalk.com/showthread.php?t=927255)

I opened a new one. HostV Shut down the VPS on the Node 16 and basically wiped out all clients data. This while guaranteeing data backups daily stored for 7 days (hostv.com/features.shtml#5)

They spent the next 14 days making everyone believe that they would fix it. They have hide this in a private thread on the Hostv forums (forums.hostv.com/) SERVER AND NETWORK NEWS that unless you are a customer you can not view. I will be glad to give my user name and password if anyone would like to view the thread and see how badly they treated everyone.

I begged for a new VPS as I was smart enough to have my own backups not only locally but remotely stored. They cost me thousands of dollars and I was only offered a month of free service in which to date we are 16 days into and I still have no service.

They did give me a new VPS 16 days later on a different Node tell me what good that is once I restored my backup these are the results.

#2002 - The server is not responding (or the local MySQL server's socket is not correctly configured)

Web mail results:

DATABASE ERROR: CONNECTION FAILED!
Unable to connect to the database!
Please contact your server-administrator

Also Sever stats show:
mysql (5.0.89-community) failed

exim (exim-4.69-23.1_cpanel_maildir) failed

How is this that they can defend actions saying we are giving refunds and offering new VPS that does not function properly. Where is the solution in a non function VPS. I had a video site running on this VPS that lost all the back links to this site and the embeds form where dropped like flies this cost a lot of people money and yes stupidly the people affected believed the BS that was fed to us even when it was questioned

I was unable to add the direct links since I did not have enough forum privileges just add the www

[bentink]

@ Citrex: You mean all data is restored by now or are some missing/lost?

[Cirtex]

Originally Posted by bentink

@ Citrex: You mean all data is restored by now or are some missing/lost?

All data restoring is complete, but there has been data loss for clients, some having minimal data loss, while others are much more affected. We're still investigating the cause for this backup restore issue.

[ItsJustHosting]

Originally Posted by gofishch

They offer the same guarantee (ItsJustHosting) you have in you forum signature 99.9%. So if you are defending HOSTV by all means show me another provider that left their customers off line for 16 days and counting while claming a 99.9% uptime. With lost data and no real support, since they expect support to be done via on email and that crap twitter site. They say only a dozen or so customers where affected, they should have been on the phone taking care of them.

I'm only questioning the notion here that HostV intentionally took the server down and wiped all client data just to make your life miserable. I am not concerned (or aware of) how they handled the situation.

[gofishch]

Originally Posted by Cirtex

Hi gofishch,


As noted on our forums, on behalf of HostV we truly apologize to all customers on NODE-16 that were affected by this as we always stay by our clients and this situation had affected 2 dozen or so customers total. We will do our best to prevent such incidents in the future and work towards resolving our current external backup server issues as well which failed restore in R1Soft after multiple tries. We had made the promise that all data will be restored originally because the data was indeed shown as being fine and all was up to date, however upon doing the restore we ran into multiple bottlenecks in which we worked very hard to resolve w/ many teams unsuccessfully.

There were successful restores of VPSes on NODE-16, however as of now, if you are still experiencing problems, please email BILLING with your account info, and we'll try to bring back any data possible, however it is best you provide helpdesk with backups that you have, such as uploading via FTP to your new VPS (w/ same ips) and then having our support team do restoration.


How it happened

1) Server NODE 16 rebooted for kernel update
2) NODE 16 hardware raid card RARE* failure causing corruption of all drives
3) We had external backups for entire node-16 so we thought this was no problem.
4) We started restore from R1Soft backup baremetal restore
5) Baremetal restore failed multiple times and we worked w/ R1Soft to get this up and running.
5) Final solution was to do manual restore for per VPS and this took week+
6) After final restore was complete, we ran into more issues, this resulted in migration of all VPS Data to newly created VPS on a NEW NODE.
7) There were many customers affected w/ data loss from minimal to total data loss, but again it was out of our hands that we had failure on both our hardware of node16 and failure on our backup system.


For all affected customers on NODE-16, please also email BILLING to receive credit / refund for this inconvenience caused.

Again, thank you for your patience and we're sorry for the inconveniences caused by this rare situation with hardware & backup failure. The entire team will continue to work towards improving our backup infrastructure and also in restoring all clients affected by node-16. Please do not hesitate to contact us!



Cheers

I have 28 and counting email responses from you Cirtex all of which If i posted here would make you look more ridiculous then you do now.

I will ask the question from you since you are who that matters tell me what other hosting provider in the country better yet in the world as evrer let there clints suffer through a 16 day and counting failure. Did the Internet brake down and you guys where recreating it.

I have contacted billing & support everyday for 16 days almost twice a day and your guys are A**holes for minimizing this situation, and acting like I did wrong, and should fell sorry that I am pissed. There is not one answer in any response that your support gave me, not one solution, just we are WOKING HARD, RARE OCCURANCE, 12 MORE HOURS, BLA, BLA, BLA,. And the only way that I am back up and running at the moment is by my own hard work when you finally 14 days late put me on new server that did not function properly. I took the data that was backed up from my original WHM and found that it too is corrupt and as missing data. So why is that you are blaming the R1Soft people and everyone else, when the backups I took from my server are corrupted. The blog From R1Soft as a nice article about this very situation and from the date of it appears it is about HOSTV( blog.r1soft.com/). You should go read the article and BTW you are the one who hired them not me I trusted your service and you are the ones that failed

[gofishch]

Originally Posted by bentink

@ Citrex: You mean all data is restored by now or are some missing/lost?

I will answer for you I am missing sveral months worth of data as well as script files. There are other customers totally wiped out. And on HOSTV forums which clinets were forced to use for updates, clients begged for help and where ignored here is their last upadte NODE-16 Downtime [ Updated: 2-12-2010 ] that is 5 days ago