Page 1 of 2 12 LastLast
Results 1 to 40 of 50
  1. #1

    HOSTV 16 DAY FIASCO AND COUNTING

    Since The Thread that was started earlier was closed (webhostingtalk.com/showthread.php?t=927255)

    I opened a new one. HostV Shut down the VPS on the Node 16 and basically wiped out all clients data. This while guaranteeing data backups daily stored for 7 days (hostv.com/features.shtml#5)

    They spent the next 14 days making everyone believe that they would fix it. They have hide this in a private thread on the Hostv forums (forums.hostv.com/) SERVER AND NETWORK NEWS that unless you are a customer you can not view. I will be glad to give my user name and password if anyone would like to view the thread and see how badly they treated everyone.

    I begged for a new VPS as I was smart enough to have my own backups not only locally but remotely stored. They cost me thousands of dollars and I was only offered a month of free service in which to date we are 16 days into and I still have no service.

    They did give me a new VPS 16 days later on a different Node tell me what good that is once I restored my backup these are the results.

    #2002 - The server is not responding (or the local MySQL server's socket is not correctly configured)

    Web mail results:

    DATABASE ERROR: CONNECTION FAILED!
    Unable to connect to the database!
    Please contact your server-administrator

    Also Sever stats show:
    mysql (5.0.89-community) failed

    exim (exim-4.69-23.1_cpanel_maildir) failed

    How is this that they can defend actions saying we are giving refunds and offering new VPS that does not function properly. Where is the solution in a non function VPS. I had a video site running on this VPS that lost all the back links to this site and the embeds form where dropped like flies this cost a lot of people money and yes stupidly the people affected believed the BS that was fed to us even when it was questioned

    I was unable to add the direct links since I did not have enough forum privileges just add the www
      0 Not allowed!

  2. #2
    Join Date
    Jun 2009
    Location
    Vancouver
    Posts
    969
    I opened a new one. HostV Shut down the VPS on the Node 16 and basically wiped out all clients data. This while guaranteeing data backups daily stored for 7 days (hostv.com/features.shtml#5)
    I am not exactly up to date with this situation, but I highly doubt they intentionally shut down the server and wiped all clients data. They apparently experienced massive hardware failure with that node, which is completely OUT of their control. They then tried to restore client data from backups and they experienced problems with that as well likely due to the sheer size of the data that needs to be restored and the long time it takes to restore it.

    I would assume that if you're having problems with your new VPS, you should contact them about it so they can fix it for you. Unfortunately hardware failure can and does happen, which is why you should ALWAYS maintain your own up to date off site backups in case of such situations, so you can quickly move to either another VPS or another provider. This is magnified if your data is very valuable, and if your website generates substantial amounts of income upon which you rely.
      0 Not allowed!

  3. #3
    Join Date
    Jan 2007
    Posts
    1,025
    Quote Originally Posted by ItsJustHosting View Post
    I am not exactly up to date with this situation, but I highly doubt they intentionally shut down the server and wiped all clients data. They apparently experienced massive hardware failure with that node, which is completely OUT of their control. They then tried to restore client data from backups and they experienced problems with that as well likely due to the sheer size of the data that needs to be restored and the long time it takes to restore it.

    I would assume that if you're having problems with your new VPS, you should contact them about it so they can fix it for you. Unfortunately hardware failure can and does happen, which is why you should ALWAYS maintain your own up to date off site backups in case of such situations, so you can quickly move to either another VPS or another provider. This is magnified if your data is very valuable, and if your website generates substantial amounts of income upon which you rely.
    Maybe the OP thinks 1 month of free service credit is too little for this?
    /home/linuxclark
      0 Not allowed!

  4. #4
    They admitted that they shut down the node and lost the date due to failure. They also told everyone that they had backed that data up. And assured me even when questioned that my data was safe when in fact it was not.

    I do have my own remote backups. And have I have contacted them to fix the new VPS problem but apparently they intend to take another 16 days to do it since it as been since 02/15/10 that the request was made. The damage they done to my revenue streams happened in the first 48 hours. My site is video site with scripts set to run on their servers. When the TV channel goes blank you change the channel so if you had any embed videos from my site you got a blank, so of course it take less than a couple of hours to start the bleeding.

    They offer the same guarantee (ItsJustHosting) you have in you forum signature 99.9%. So if you are defending HOSTV by all means show me another provider that left their customers off line for 16 days and counting while claming a 99.9% uptime. With lost data and no real support, since they expect support to be done via on email and that crap twitter site. They say only a dozen or so customers where affected, they should have been on the phone taking care of them.
    Last edited by gofishch; 02-17-2010 at 01:46 PM. Reason: spell
      0 Not allowed!

  5. #5
    Quote Originally Posted by linuxclark View Post
    Maybe the OP thinks 1 month of free service credit is too little for this?
    I think they should get my site back up and working and then dicsuss any other options
      0 Not allowed!

  6. #6
    Join Date
    Jan 2007
    Posts
    1,025
    Quote Originally Posted by gofishch View Post
    I think they should get my site back up and working and then dicsuss any other options
    As you have backup for your site, Why not get a VPS with another provider, setup the VPS there, just to get your site up and running again, you can alway move back when everything works for you at hostV. 16 days down time is a lot.
    /home/linuxclark
      0 Not allowed!

  7. #7
    Quote Originally Posted by gofishch View Post
    My site is video site with scripts set to run on their servers.
    I answered that question. It is not impossible but it is very diffcult to move.
      0 Not allowed!

  8. #8
    Quote Originally Posted by linuxclark View Post
    As you have backup for your site, Why not get a VPS with another provider, setup the VPS there, just to get your site up and running again, you can alway move back when everything works for you at hostV. 16 days down time is a lot.
    Quote Originally Posted by gofishch View Post
    My site is video site with scripts set to run on their servers.
    I answered that question. It is not impossible but it is very diffcult to move.
      0 Not allowed!

  9. #9
    Join Date
    May 2009
    Location
    Hull, UK.
    Posts
    911
    If you get managed your new provider will transfer for you.
      0 Not allowed!

  10. #10
    Join Date
    Mar 2003
    Location
    New York City
    Posts
    7,391

    *

    Hi gofishch,


    As noted on our forums, on behalf of HostV we truly apologize to all customers on NODE-16 that were affected by this as we always stay by our clients and this situation had affected 2 dozen or so customers total. We will do our best to prevent such incidents in the future and work towards resolving our current external backup server issues as well which failed restore in R1Soft after multiple tries. We had made the promise that all data will be restored originally because the data was indeed shown as being fine and all was up to date, however upon doing the restore we ran into multiple bottlenecks in which we worked very hard to resolve w/ many teams unsuccessfully.

    There were successful restores of VPSes on NODE-16, however as of now, if you are still experiencing problems, please email BILLING with your account info, and we'll try to bring back any data possible, however it is best you provide helpdesk with backups that you have, such as uploading via FTP to your new VPS (w/ same ips) and then having our support team do restoration.


    How it happened

    1) Server NODE 16 rebooted for kernel update
    2) NODE 16 hardware raid card RARE* failure causing corruption of all drives
    3) We had external backups for entire node-16 so we thought this was no problem.
    4) We started restore from R1Soft backup baremetal restore
    5) Baremetal restore failed multiple times and we worked w/ R1Soft to get this up and running.
    5) Final solution was to do manual restore for per VPS and this took week+
    6) After final restore was complete, we ran into more issues, this resulted in migration of all VPS Data to newly created VPS on a NEW NODE.
    7) There were many customers affected w/ data loss from minimal to total data loss, but again it was out of our hands that we had failure on both our hardware of node16 and failure on our backup system.


    For all affected customers on NODE-16, please also email BILLING to receive credit / refund for this inconvenience caused.

    Again, thank you for your patience and we're sorry for the inconveniences caused by this rare situation with hardware & backup failure. The entire team will continue to work towards improving our backup infrastructure and also in restoring all clients affected by node-16. Please do not hesitate to contact us!



    Cheers



    Quote Originally Posted by gofishch View Post
    Since The Thread that was started earlier was closed (webhostingtalk.com/showthread.php?t=927255)

    I opened a new one. HostV Shut down the VPS on the Node 16 and basically wiped out all clients data. This while guaranteeing data backups daily stored for 7 days (hostv.com/features.shtml#5)

    They spent the next 14 days making everyone believe that they would fix it. They have hide this in a private thread on the Hostv forums (forums.hostv.com/) SERVER AND NETWORK NEWS that unless you are a customer you can not view. I will be glad to give my user name and password if anyone would like to view the thread and see how badly they treated everyone.

    I begged for a new VPS as I was smart enough to have my own backups not only locally but remotely stored. They cost me thousands of dollars and I was only offered a month of free service in which to date we are 16 days into and I still have no service.

    They did give me a new VPS 16 days later on a different Node tell me what good that is once I restored my backup these are the results.

    #2002 - The server is not responding (or the local MySQL server's socket is not correctly configured)

    Web mail results:

    DATABASE ERROR: CONNECTION FAILED!
    Unable to connect to the database!
    Please contact your server-administrator

    Also Sever stats show:
    mysql (5.0.89-community) failed

    exim (exim-4.69-23.1_cpanel_maildir) failed

    How is this that they can defend actions saying we are giving refunds and offering new VPS that does not function properly. Where is the solution in a non function VPS. I had a video site running on this VPS that lost all the back links to this site and the embeds form where dropped like flies this cost a lot of people money and yes stupidly the people affected believed the BS that was fed to us even when it was questioned

    I was unable to add the direct links since I did not have enough forum privileges just add the www
    Last edited by Cirtex; 02-17-2010 at 03:53 PM.
    █• CirtexHosting • Providing Affordable and Quality Web Hosting & Reseller Hosting since 2003
    █• LINUX based cPANEL/WHM Shared and Reseller Web Hosting with Fantastico
    █• HostV VPS • Premium Virtual Private Servers & Dedicated Servers powered by cPanel/WHM
    █• We transfer your sites over quickly! • I eat penguins for breakfast ...
      0 Not allowed!

  11. #11
    Join Date
    Mar 2009
    Location
    India
    Posts
    1,233
    @ Citrex: You mean all data is restored by now or are some missing/lost?
      0 Not allowed!

  12. #12
    Join Date
    Mar 2003
    Location
    New York City
    Posts
    7,391
    Quote Originally Posted by bentink View Post
    @ Citrex: You mean all data is restored by now or are some missing/lost?
    All data restoring is complete, but there has been data loss for clients, some having minimal data loss, while others are much more affected. We're still investigating the cause for this backup restore issue.
    █• CirtexHosting • Providing Affordable and Quality Web Hosting & Reseller Hosting since 2003
    █• LINUX based cPANEL/WHM Shared and Reseller Web Hosting with Fantastico
    █• HostV VPS • Premium Virtual Private Servers & Dedicated Servers powered by cPanel/WHM
    █• We transfer your sites over quickly! • I eat penguins for breakfast ...
      0 Not allowed!

  13. #13
    Join Date
    Jun 2009
    Location
    Vancouver
    Posts
    969
    Quote Originally Posted by gofishch View Post

    They offer the same guarantee (ItsJustHosting) you have in you forum signature 99.9%. So if you are defending HOSTV by all means show me another provider that left their customers off line for 16 days and counting while claming a 99.9% uptime. With lost data and no real support, since they expect support to be done via on email and that crap twitter site. They say only a dozen or so customers where affected, they should have been on the phone taking care of them.
    I'm only questioning the notion here that HostV intentionally took the server down and wiped all client data just to make your life miserable. I am not concerned (or aware of) how they handled the situation.
      0 Not allowed!

  14. #14
    Quote Originally Posted by Cirtex View Post
    Hi gofishch,


    As noted on our forums, on behalf of HostV we truly apologize to all customers on NODE-16 that were affected by this as we always stay by our clients and this situation had affected 2 dozen or so customers total. We will do our best to prevent such incidents in the future and work towards resolving our current external backup server issues as well which failed restore in R1Soft after multiple tries. We had made the promise that all data will be restored originally because the data was indeed shown as being fine and all was up to date, however upon doing the restore we ran into multiple bottlenecks in which we worked very hard to resolve w/ many teams unsuccessfully.

    There were successful restores of VPSes on NODE-16, however as of now, if you are still experiencing problems, please email BILLING with your account info, and we'll try to bring back any data possible, however it is best you provide helpdesk with backups that you have, such as uploading via FTP to your new VPS (w/ same ips) and then having our support team do restoration.


    How it happened

    1) Server NODE 16 rebooted for kernel update
    2) NODE 16 hardware raid card RARE* failure causing corruption of all drives
    3) We had external backups for entire node-16 so we thought this was no problem.
    4) We started restore from R1Soft backup baremetal restore
    5) Baremetal restore failed multiple times and we worked w/ R1Soft to get this up and running.
    5) Final solution was to do manual restore for per VPS and this took week+
    6) After final restore was complete, we ran into more issues, this resulted in migration of all VPS Data to newly created VPS on a NEW NODE.
    7) There were many customers affected w/ data loss from minimal to total data loss, but again it was out of our hands that we had failure on both our hardware of node16 and failure on our backup system.


    For all affected customers on NODE-16, please also email BILLING to receive credit / refund for this inconvenience caused.

    Again, thank you for your patience and we're sorry for the inconveniences caused by this rare situation with hardware & backup failure. The entire team will continue to work towards improving our backup infrastructure and also in restoring all clients affected by node-16. Please do not hesitate to contact us!



    Cheers



    I have 28 and counting email responses from you Cirtex all of which If i posted here would make you look more ridiculous then you do now.

    I will ask the question from you since you are who that matters tell me what other hosting provider in the country better yet in the world as evrer let there clints suffer through a 16 day and counting failure. Did the Internet brake down and you guys where recreating it.

    I have contacted billing & support everyday for 16 days almost twice a day and your guys are A**holes for minimizing this situation, and acting like I did wrong, and should fell sorry that I am pissed. There is not one answer in any response that your support gave me, not one solution, just we are WOKING HARD, RARE OCCURANCE, 12 MORE HOURS, BLA, BLA, BLA,. And the only way that I am back up and running at the moment is by my own hard work when you finally 14 days late put me on new server that did not function properly. I took the data that was backed up from my original WHM and found that it too is corrupt and as missing data. So why is that you are blaming the R1Soft people and everyone else, when the backups I took from my server are corrupted. The blog From R1Soft as a nice article about this very situation and from the date of it appears it is about HOSTV( blog.r1soft.com/). You should go read the article and BTW you are the one who hired them not me I trusted your service and you are the ones that failed
      0 Not allowed!

  15. #15
    Quote Originally Posted by bentink View Post
    @ Citrex: You mean all data is restored by now or are some missing/lost?
    I will answer for you I am missing sveral months worth of data as well as script files. There are other customers totally wiped out. And on HOSTV forums which clinets were forced to use for updates, clients begged for help and where ignored here is their last upadte NODE-16 Downtime [ Updated: 2-12-2010 ] that is 5 days ago
      0 Not allowed!

  16. #16
    Quote Originally Posted by Cirtex View Post
    All data restoring is complete, but there has been data loss for clients, some having minimal data loss, while others are much more affected. We're still investigating the cause for this backup restore issue.
    Cirtex The backups I had from my own WHM are corrupted its in you server that is the problem. And the way you monitored the backups is apparently the same as your support in this instance minimal at best. Below you will find an article from R1soft you should read it it explains exactly what happned and how you guys could have avoided it.

    [QUOTE R1Soft]
    Examples of why you Need to Monitor your CDP Enterprise Installation
    We Do See Hardware Go Bad Sometimes

    A few months ago we had a customer with failed restores. Data on the disk was so corrupt that on one agent the 4 byte magic number at the beginning of the Disk Safe .data file written only once when the disk safe is created was corrupt. In this case every task on the CDP Server had warnings or errors and most were failing with nasty errors. They went unnoticed for 1 month until a restore was needed. That and other warnings from the operating system of a serious problem with disk storage on their CDP Server.

    All could have been fixed well before a restore was needed.

    Like All Software R1Soft CDP Has Errors and Often there are Warning Signs

    Linux Agent s with version numbers lower than 1.67.0 (CDP Release 2.19) as reported in our release notes in some cases can fail to properly identify deltas and read blocks correctly on disks or volumes larger than 2 TB. The issue was fixed in Linux Agent 1.67 in November. The symptoms include complete failure to add the device/partition to the CDP Server OR literally thousands of warnings during the backup about corrupt or missing blocks.
    wiki.r1soft.com/x/W5Jy

    The thousands of errors during each backup this issue can trigger are part of a validation done during backup to make sure none of the blocks allocated by the file system are missing from CDP replication. If blocks are missing it can be an issue with the software, hardware, or a corrupt file system.

    The File System on Your Server (where CDP Agent runs) May be Corrupt – In Some Cases CDP Can Detect and Warn about This

    It’s actually not that uncommon for Linux and Windows file systems to become corrupt to one degree or the other especially on multi-tenant systems that rarely if ever get taken down for a periodic fsck or checkdisk of the file system (cuz we all love a 24 hour fsck… right).

    While rare what can happen is some file system data structures on disk become corrupted. Could be hardware issue, could simply be caused by a hard reset of the server. These data structures in some cases are also resident in memory on the operating system. This allows the server to pretty much function mostly correctly though there are often clues from the kernel in /var/log/messages. The real surprise is when you reboot this server when you find out the file system corrupted beyond repair.

    In some cases (not all) the CDP Server will give warnings signs. Unable to read block XXXXX I/O error (you have bad sectors or some storage failure). Warning blocks are corrupt which means they are allocated by the file system but the CDP Software never detected they were written to. This typically means file system corruption and the file systems block allocation maps have issues.

    Other good reasons to Monitor if these are not enough:

    A firewall may have changed blocking access to the agent from the CDP Server.
    There could be damage to a Disk Safe from a CDP Server crash or power loss .
    You could have run out of disk space on your CDP Server
    Backups may be working but MySQL may not be locked and flushed during the backup because the agent is unable to connect to MySQL
    [/QUOTE]

    Can any one say corrupt file system. I sent your support another reply to my original ticket about an hour ago still no response. I will go to bed and wake up and maybe there will be an answer in say 7 or 8 hours.
      0 Not allowed!

  17. #17
    Join Date
    Mar 2003
    Location
    New York City
    Posts
    7,391

    Arrow

    Hi,

    We're not going to point fingers and will take full responsibility for what has happened in this situation.

    The matter of fact is that

    1) There was a rare hardware failure that caused data on NODE-16 customers to gone corrupt

    2) We then started R1Soft external backup restore.

    3) The Backup in R1Soft CDP Servers were monitored carefully and we had not seen any errors or issues in the DATA / DISKSAFE nor were we notified of any updates.

    4) NOTE: our R1Soft CDP Agents & Version were all very recent from around 10/2009, and was not out of date for that matter.

    5) The HARDWARE used was also setup in RAID for both our VPS NODE & Our Backup Storage servers, and we did not see any issues with data corruption.

    We have done everything in our power at this point to restore and R1Soft has said the data is not recoverable due to corruption and it is very unfortunate. At this point we're re-assessing our backup solution to see where the problem was and if it is the right solution for us.

    In terms of our response time & updates, we did our best to update clients via helpdesk, twitter and forums, but at times there really wasn't anything to update, as the backup was simply restoring and taking just so extremely long due to the manual restore option in R1Soft having to decompress and then transfer files one at a time.

    Again we truly apologize for this matter, send me a private message with your TICKET ID and we'll send full refund for your VPS For all payments even if you wish as I agree we have failed you as our customer and our customers = our most important priority.

    Thank you for your patience and understand.


    Quote Originally Posted by gofishch View Post
    Cirtex The backups I had from my own WHM are corrupted its in you server that is the problem. And the way you monitored the backups is apparently the same as your support in this instance minimal at best. Below you will find an article from R1soft you should read it it explains exactly what happned and how you guys could have avoided it.

    [QUOTE R1Soft]
    Examples of why you Need to Monitor your CDP Enterprise Installation
    We Do See Hardware Go Bad Sometimes

    A few months ago we had a customer with failed restores. Data on the disk was so corrupt that on one agent the 4 byte magic number at the beginning of the Disk Safe .data file written only once when the disk safe is created was corrupt. In this case every task on the CDP Server had warnings or errors and most were failing with nasty errors. They went unnoticed for 1 month until a restore was needed. That and other warnings from the operating system of a serious problem with disk storage on their CDP Server.

    All could have been fixed well before a restore was needed.

    Like All Software R1Soft CDP Has Errors and Often there are Warning Signs

    Linux Agent s with version numbers lower than 1.67.0 (CDP Release 2.19) as reported in our release notes in some cases can fail to properly identify deltas and read blocks correctly on disks or volumes larger than 2 TB. The issue was fixed in Linux Agent 1.67 in November. The symptoms include complete failure to add the device/partition to the CDP Server OR literally thousands of warnings during the backup about corrupt or missing blocks.
    wiki.r1soft.com/x/W5Jy

    The thousands of errors during each backup this issue can trigger are part of a validation done during backup to make sure none of the blocks allocated by the file system are missing from CDP replication. If blocks are missing it can be an issue with the software, hardware, or a corrupt file system.

    The File System on Your Server (where CDP Agent runs) May be Corrupt – In Some Cases CDP Can Detect and Warn about This

    It’s actually not that uncommon for Linux and Windows file systems to become corrupt to one degree or the other especially on multi-tenant systems that rarely if ever get taken down for a periodic fsck or checkdisk of the file system (cuz we all love a 24 hour fsck… right).

    While rare what can happen is some file system data structures on disk become corrupted. Could be hardware issue, could simply be caused by a hard reset of the server. These data structures in some cases are also resident in memory on the operating system. This allows the server to pretty much function mostly correctly though there are often clues from the kernel in /var/log/messages. The real surprise is when you reboot this server when you find out the file system corrupted beyond repair.

    In some cases (not all) the CDP Server will give warnings signs. Unable to read block XXXXX I/O error (you have bad sectors or some storage failure). Warning blocks are corrupt which means they are allocated by the file system but the CDP Software never detected they were written to. This typically means file system corruption and the file systems block allocation maps have issues.

    Other good reasons to Monitor if these are not enough:

    A firewall may have changed blocking access to the agent from the CDP Server.
    There could be damage to a Disk Safe from a CDP Server crash or power loss .
    You could have run out of disk space on your CDP Server
    Backups may be working but MySQL may not be locked and flushed during the backup because the agent is unable to connect to MySQL
    Can any one say corrupt file system. I sent your support another reply to my original ticket about an hour ago still no response. I will go to bed and wake up and maybe there will be an answer in say 7 or 8 hours.[/QUOTE]
    █• CirtexHosting • Providing Affordable and Quality Web Hosting & Reseller Hosting since 2003
    █• LINUX based cPANEL/WHM Shared and Reseller Web Hosting with Fantastico
    █• HostV VPS • Premium Virtual Private Servers & Dedicated Servers powered by cPanel/WHM
    █• We transfer your sites over quickly! • I eat penguins for breakfast ...
      0 Not allowed!

  18. #18
    Join Date
    Dec 2006
    Posts
    4,149
    While I'm saying HostV is completely at fault here, the OP should do your own backups and not put it on the same server (ever heard of placing all your eggs in one basket?).
      0 Not allowed!

  19. #19
    Join Date
    Jan 2007
    Posts
    1,025
    Quote Originally Posted by Cirtex View Post


    Again we truly apologize for this matter, send me a private message with your TICKET ID and we'll send full refund for your VPS For all payments even if you wish as I agree we have failed you as our customer and our customers = our most important priority.
    /home/linuxclark
      0 Not allowed!

  20. #20
    Quote Originally Posted by tsj5j View Post
    While I'm saying HostV is completely at fault here, the OP should do your own backups and not put it on the same server (ever heard of placing all your eggs in one basket?).
    I did my own backups apparently you can not read. I also FTP those backups to a remote server I own Via WHM. The backup that came from this server, that I took personally is corrupt and is missing data and files. Cirtex and Hostv have hardware issues that have caused this problem. And so far I have only heard we are sorry for rare occurrence.
    Last edited by gofishch; 02-18-2010 at 08:58 AM. Reason: spelling
      0 Not allowed!

  21. #21
    Cirtex I made a request to Billing and Support 5 days ago just for that reason along with 32 other post to that ticket. I also sent in a support ticket as stated to you in my last post to you, maybe you can read it this time, it is below I have not received an answer. now for over 7 hours and counting

    Can any one say corrupt file system. I sent your support another reply to my original ticket about an hour ago still no response. I will go to bed and wake up and maybe there will be an answer in say 7 or 8 hours

    I will wait for their reply and then I will PM. It is truly shameful that I have to sink to the level of exposing this on a public forum, I begged for help through HOSTV forum, including PM the administrators with no answer, through support tickets, through your online chat support. YOU FORCED ME HERE




    Quote Originally Posted by Cirtex View Post
    Hi,

    We're not going to point fingers and will take full responsibility for what has happened in this situation.

    The matter of fact is that

    1) There was a rare hardware failure that caused data on NODE-16 customers to gone corrupt

    2) We then started R1Soft external backup restore.

    3) The Backup in R1Soft CDP Servers were monitored carefully and we had not seen any errors or issues in the DATA / DISKSAFE nor were we notified of any updates.

    4) NOTE: our R1Soft CDP Agents & Version were all very recent from around 10/2009, and was not out of date for that matter.

    5) The HARDWARE used was also setup in RAID for both our VPS NODE & Our Backup Storage servers, and we did not see any issues with data corruption.

    We have done everything in our power at this point to restore and R1Soft has said the data is not recoverable due to corruption and it is very unfortunate. At this point we're re-assessing our backup solution to see where the problem was and if it is the right solution for us.

    In terms of our response time & updates, we did our best to update clients via helpdesk, twitter and forums, but at times there really wasn't anything to update, as the backup was simply restoring and taking just so extremely long due to the manual restore option in R1Soft having to decompress and then transfer files one at a time.

    Again we truly apologize for this matter, send me a private message with your TICKET ID and we'll send full refund for your VPS For all payments even if you wish as I agree we have failed you as our customer and our customers = our most important priority.

    Thank you for your patience and understand.




    Can any one say corrupt file system. I sent your support another reply to my original ticket about an hour ago still no response. I will go to bed and wake up and maybe there will be an answer in say 7 or 8 hours.
    [/QUOTE]
    Last edited by gofishch; 02-18-2010 at 09:02 AM. Reason: spelling
      0 Not allowed!

  22. #22
    Join Date
    Dec 2006
    Posts
    4,149
    Quote Originally Posted by gofishch View Post
    I did my own backups apparently you can not read. I also FTP those backups to a remote server I own Via WHM. The backup that came from this server, that I took personally is corrupt and is missing data and files. Cirtex and Hostv have hardware issues that have caused this problem. And so far I have only heard we are sorry for rare occurrence.
    My emphasis: on a remote server.
    You typed "The backups I had from my own WHM are corrupted its in you server that is the problem".
    You indicated that the problem is that your backups are corrupted because it is on THEIR server.If you did make it remote, then you committed the same mistake that you claimed Cirtex made: check your backups.

    Besides, it's not your fault. What I said were suggestions to help you should there be a problem in future.
      0 Not allowed!

  23. #23
    Join Date
    Jul 2009
    Location
    Singapore
    Posts
    50
    No, the OP was saying he used WHM to generate a backup and then downloaded it to a remote location (either his own PC or to another FTP).
    The downloaded backup was corrupt. This means that when WHM created the backup, it was on faulty hardware, thus even the downloads were corrupted. Far as I know, WHM doesn't create backups in memory.
    Last edited by ConceptLane; 02-18-2010 at 11:40 AM.
    Roy Lee
    ConceptLane Pte Ltd
    Singapore Hosting, VPS, Colo, Dedi, Web Apps
      0 Not allowed!

  24. #24
    I appreciate you advice and believe me I doubt seriously if this will ever happen to me again. But I would like to point out and maybe I have not been clear.

    Cirtex has hardware issues that caused corrupted backup no matter how it was created and R1soft specifically said they warned Cirtex previously and those warnings where ignored. I contacted R1soft and even though they would not discuss it in detail they told me they had warned Cirtex. If you follow what Cirtex said they could not even recover from their own in house backups. And what they did recover was not as current as mine own backup, theirs is much older, but corrupt none the less it is missing data and files.

    It is also not my intent to attack, I only presented facts. Thus far Cirtex has responded faster and more often in Public than in their own support tickets. I am still waiting on an answer to the ticket I submitted last night over 12 hours ago. While at the same time Cirtex is responding in an outside forum. It seems what they want is for everybody affected to fix the problem Cirtex created, and the strategy is to ignore you in their support system until you either solve for yourself or go away.

    FYI the corruption does not rear its ugly head until you need to restore it. So how would one go about checking their backups. Any input on that would be appreciated, I am not as technically savvy as most here, I am sure.


    Quote Originally Posted by tsj5j View Post
    My emphasis: on a remote server.
    You typed "The backups I had from my own WHM are corrupted its in you server that is the problem".
    You indicated that the problem is that your backups are corrupted because it is on THEIR server.If you did make it remote, then you committed the same mistake that you claimed Cirtex made: check your backups.

    Besides, it's not your fault. What I said were suggestions to help you should there be a problem in future.
      0 Not allowed!

  25. #25
    Thank You Kindly That is exactly what I have said form the beginning

    Quote Originally Posted by ConceptLane View Post
    No, the OP was saying he used WHM to generate a backup and then downloaded it to a remote location (either his own PC or to another FTP).
    The downloaded backup was corrupt. This means that when WHM created the backup, it was on faulty hardware, thus even the downloads were corrupted. Far as I know, WHM doesn't create backups in memory.
      0 Not allowed!

  26. #26
    Join Date
    Dec 2006
    Posts
    4,149
    Quote Originally Posted by gofishch View Post
    I appreciate you advice and believe me I doubt seriously if this will ever happen to me again. But I would like to point out and maybe I have not been clear.

    Cirtex has hardware issues that caused corrupted backup no matter how it was created and R1soft specifically said they warned Cirtex previously and those warnings where ignored. I contacted R1soft and even though they would not discuss it in detail they told me they had warned Cirtex. If you follow what Cirtex said they could not even recover from their own in house backups. And what they did recover was not as current as mine own backup, theirs is much older, but corrupt none the less it is missing data and files.

    It is also not my intent to attack, I only presented facts. Thus far Cirtex has responded faster and more often in Public than in their own support tickets. I am still waiting on an answer to the ticket I submitted last night over 12 hours ago. While at the same time Cirtex is responding in an outside forum. It seems what they want is for everybody affected to fix the problem Cirtex created, and the strategy is to ignore you in their support system until you either solve for yourself or go away.

    FYI the corruption does not rear its ugly head until you need to restore it. So how would one go about checking their backups. Any input on that would be appreciated, I am not as technically savvy as most here, I am sure.
    Cirtex should clarify this, emphasis mine.
    They mentioned their backups were monitored, and the emphasized statement directly contradicts that, aka accusation of lying.
      0 Not allowed!

  27. #27
    Quote Originally Posted by tsj5j View Post
    Cirtex should clarify this, emphasis mine.
    They mentioned their backups were monitored, and the emphasized statement directly contradicts that, aka accusation of lying.
    I will clarify and try not to get into word semantics

    1) I did weekly backups of my VPS via WHM

    2) I had those backups FTP to a remote sever I own not hosted by Cirtex (outside)

    3) The Last Backup I took Via WHM Is Exactly 7 days prior to this incident. Node 16 went down on Feb 4, My remotely stored, WHM created backup is from Jan 29, It is corrupted.

    4) Cirtex never did any restoration they simply placed a file called xxxx.bak on a new VPS that they finally created 14 days later that took me 3 days to get them to make function correctly. That xxxx.bak is at least 45 days old it is corrupt and missing data and flies, that would be the reason it was not actually restored.

    5) Since there is at least a 45 day different time frame and coruption and missing data files in what they call a backup (xxxx.bak) and my WHM backup, they either did not monitor or ignored a problem. That is not an accusation that is a it fact.


    6) I do not have access to hardware on my VPS so I would have no way to monitor any backup I created for hardware issues, That is what Cirtex is responible for.


    I have been gracious enough to continue with Cirtex and try to work through the problems and as I posted earlier Cirtex Forced Me Here by being unresponsive to their support system as I am still waiting for a reply to the ticket I updated Last night now over 13 hours and counting
    Last edited by gofishch; 02-18-2010 at 02:10 PM. Reason: spelling
      0 Not allowed!

  28. #28
    I am reserving any further comment while I wait for Cirtex to keep their word in a PM sent to me. After which I will inform of the outcome
      0 Not allowed!

  29. #29
    Join Date
    Jun 2008
    Posts
    405
    All im saying now I am so glad that I never went to HostV and decided to go to a more reliable host.
      0 Not allowed!

  30. #30
    How does HostV explain the two month old backups when this is clearly stated on your web page under the features tab.


    "ServerSnapShot automatically backs up each server & hosting account fully on a daily basis with multiple restore points. The backups are then compressed, encrypted and stored securely in physically separate backup storage "
      0 Not allowed!

  31. #31
    Join Date
    Mar 2003
    Location
    New York City
    Posts
    7,391
    Hi,

    The NODE-16 backups was not 2 months old. It was taken day or 1.5 days before the NODE-16 went offline.

    Please read my previous detailed response.

    Thank you


    Quote Originally Posted by icydude View Post
    How does HostV explain the two month old backups when this is clearly stated on your web page under the features tab.


    "ServerSnapShot automatically backs up each server & hosting account fully on a daily basis with multiple restore points. The backups are then compressed, encrypted and stored securely in physically separate backup storage "
    █• CirtexHosting • Providing Affordable and Quality Web Hosting & Reseller Hosting since 2003
    █• LINUX based cPANEL/WHM Shared and Reseller Web Hosting with Fantastico
    █• HostV VPS • Premium Virtual Private Servers & Dedicated Servers powered by cPanel/WHM
    █• We transfer your sites over quickly! • I eat penguins for breakfast ...
      0 Not allowed!

  32. #32
    Join Date
    Aug 2008
    Posts
    35

    Thumbs down Cirtex are lying here!!!!

    Quote Originally Posted by Cirtex View Post
    Hi,

    The NODE-16 backups was not 2 months old. It was taken day or 1.5 days before the NODE-16 went offline.

    Please read my previous detailed response.

    Thank you
    Cirtex, you are lying here!!! Shame on You!!!

    I am a custom on Node16 of HostV, The backup which I got from HostV restored for me is over 2 month old. The node is down about 2010-02-04 05:30:06 GMT (I am monitoring all my sites with other service.), All access logs from the backup of all my sites shows that the date is Nov 30, 2009.


    Mostly my backup are empty folders over 2 months, No mysql database backup at all, no system folders backup like /usr/local, /root etc...

    Before the node16 problem, I have over 20G data on HostV, But from the backup, I have got only 240M data back, Which is almost useless as they were 2 month old.

    I have lost 20G images data files, over 1G mysql database, over 50 custom shell scripts, 2 total sites, my configure of mail system mail accounts, webserver and other configures.

    I am still busy working on the restore all my site back to net, will be report more when I have more time on this.
      0 Not allowed!

  33. #33
    Join Date
    Aug 2008
    Posts
    35
    Too bad that you may still see that on HostV, They promise as

    http://www.hostv.com/features.shtml
    -----------------------------
    ServerSnapShot Features:
    r1soft, serversnapshot
    Full Daily Backups & Incremental Backups for up to two weeks.
    Multiple restore points from the previous 7 days.
    Restore any file, email, database or even your complete account at anytime by contacting us.
    RAID protected enterprise data storage servers.
    High Performance - ServerSnapShot results in minimal performance impact during daily backups.
    Bare-Metal Disaster Recovery: Restore servers directly from disk-based backup.
    End-To-End Strong Encryption of Backups.
    Backups are compressed and stored securely in physically separate backup storage servers
    -----------------------------
    Last edited by wqmeng; 02-21-2010 at 10:00 AM. Reason: add link
      0 Not allowed!

  34. #34
    Join Date
    Aug 2008
    Posts
    35
    And more,

    HostV never tell you about their maintenance of your server, No emails notice at all, I have known that they maintenance the Node 16 after about 12 hours my site down time with submitted a ticket. The reply of tickets is slow too. So you have no chances to make a emergency backup before your server get down at all. From my experience with HostV (over 13 months), I have over 3 times site down caused by HostV maintenance, And I never got any notice from HostV before they perform them.
    Last edited by wqmeng; 02-21-2010 at 10:20 AM.
      0 Not allowed!

  35. #35
    Join Date
    Mar 2003
    Location
    New York City
    Posts
    7,391
    Hi Wgmeng,

    Sorry for the inconvenience caused again, I have sent you a private message please do let me know account info, ticket id so I can look into it more for you.

    To clear few things up

    1) The last backup in our R1Soft that was SUCCESSFUL Full Backup was dated from 1-2 days before the server went offline. That was the backup we restored, again I am not sure why some files restored were so old, but we were restoring the latest backup !


    2) In terms of the extent of the backup restore, there were lots of corruption in the backup disk safe and thus we were not able to recover 100% of the data. We tried our best but most of the 30 or so clients on NODE-16 were affected, some on bigger extent than others.


    3) We did make an announcement of the maintenance but as we told all clients we do so via our forums & twitter, and email for scheduled maintenances, to make sure you set your email in billing to your most regular email. Again we apologize for this, but we performed urgent maintenance that night on all of our nodes, and only one went down due to hardware issue during the reboot. IT was expected to be only 10-15min or so downtime max.


    We truly apologize for the situation, please see my previous replies, and again we would never lie to our clients and we were also put into this very unfortunate and unlucky situation.

    Please do contact me via private message and I'll do my best to compensate for this and issue credit/refund needed.

    Thank you again for choosing HostV.

    Cheers


    Quote Originally Posted by wqmeng View Post
    Cirtex, you are lying here!!! Shame on You!!!

    I am a custom on Node16 of HostV, The backup which I got from HostV restored for me is over 2 month old. The node is down about 2010-02-04 05:30:06 GMT (I am monitoring all my sites with other service.), All access logs from the backup of all my sites shows that the date is Nov 30, 2009.


    Mostly my backup are empty folders over 2 months, No mysql database backup at all, no system folders backup like /usr/local, /root etc...

    Before the node16 problem, I have over 20G data on HostV, But from the backup, I have got only 240M data back, Which is almost useless as they were 2 month old.

    I have lost 20G images data files, over 1G mysql database, over 50 custom shell scripts, 2 total sites, my configure of mail system mail accounts, webserver and other configures.

    I am still busy working on the restore all my site back to net, will be report more when I have more time on this.
    █• CirtexHosting • Providing Affordable and Quality Web Hosting & Reseller Hosting since 2003
    █• LINUX based cPANEL/WHM Shared and Reseller Web Hosting with Fantastico
    █• HostV VPS • Premium Virtual Private Servers & Dedicated Servers powered by cPanel/WHM
    █• We transfer your sites over quickly! • I eat penguins for breakfast ...
      0 Not allowed!

  36. #36
    Join Date
    Mar 2009
    Location
    India
    Posts
    1,233
    The best way (to the customer): Backup your data regularly on a good quality DVD or a Blu-Ray Disc. There is nothing more reliable than that.
      0 Not allowed!

  37. #37
    Join Date
    Aug 2008
    Posts
    35
    I have sent you my ticket id, though I have been told that HostV not able to restore my data On Feb,15 2010. Hope your recent restore work may help me get my data back.

    thanks


    Quote Originally Posted by Cirtex View Post
    Hi Wgmeng,

    Sorry for the inconvenience caused again, I have sent you a private message please do let me know account info, ticket id so I can look into it more for you.

    To clear few things up

    1) The last backup in our R1Soft that was SUCCESSFUL Full Backup was dated from 1-2 days before the server went offline. That was the backup we restored, again I am not sure why some files restored were so old, but we were restoring the latest backup !


    2) In terms of the extent of the backup restore, there were lots of corruption in the backup disk safe and thus we were not able to recover 100% of the data. We tried our best but most of the 30 or so clients on NODE-16 were affected, some on bigger extent than others.


    3) We did make an announcement of the maintenance but as we told all clients we do so via our forums & twitter, and email for scheduled maintenances, to make sure you set your email in billing to your most regular email. Again we apologize for this, but we performed urgent maintenance that night on all of our nodes, and only one went down due to hardware issue during the reboot. IT was expected to be only 10-15min or so downtime max.


    We truly apologize for the situation, please see my previous replies, and again we would never lie to our clients and we were also put into this very unfortunate and unlucky situation.

    Please do contact me via private message and I'll do my best to compensate for this and issue credit/refund needed.

    Thank you again for choosing HostV.

    Cheers
      0 Not allowed!

  38. #38
    Join Date
    Mar 2003
    Location
    New York City
    Posts
    7,391
    Hi

    Will do, as stated previously some clients have had much more data loss, but we did have latest backup from our r1soft when this had happened, just that the restore was not successful which is still under investigation by r1soft and we're testing the system to make sure this doesn't happen again.

    Cheers
    █• CirtexHosting • Providing Affordable and Quality Web Hosting & Reseller Hosting since 2003
    █• LINUX based cPANEL/WHM Shared and Reseller Web Hosting with Fantastico
    █• HostV VPS • Premium Virtual Private Servers & Dedicated Servers powered by cPanel/WHM
    █• We transfer your sites over quickly! • I eat penguins for breakfast ...
      0 Not allowed!

  39. #39
    Join Date
    Aug 2008
    Posts
    35
    After waiting another 24 hours, I get nothing reply to my ticket, Then I pm Cirtex if HostV have any new backup can help me get my data, Cirtex just told me that THEY never have any new backups for their customs, All the craps backup have been recovered, So I just waste another 24 hours with these guys.

    And HostV refused to refund the first month of my second year invoice to them, They have promised that as the big crap failure of their vps, they may full refund their custom. But they not refund me the first month, And even the month, my sites also get over full 3 days downtime due to the node 16 failure, almost all my data get lose 16 days later.

    Cirtex, HostV only keeping kidding on their forum and here, Now they have deleted the whole thread related this crap vps on their forum.

    What's HostV doing is setup a new node after 15 days full downtime, and give their custom a 2 month old crap backup!!!

    WTF!
      0 Not allowed!

  40. #40
    Join Date
    Mar 2003
    Location
    New York City
    Posts
    7,391
    Hi Sir

    Our backup was up to date but the restoring of the data failed, and we tried again to do restore for your account but due to the corruption in backup data, it wasn't possible.

    Please note your one year payment has been refunded already via Paypal on 2/15/2010

    Another refund has been sent for your monthly payment.

    Feb 24, 2010 Refund To ******* Completed Details -$39.99 $1.58 -$38.41 USD

    and

    Feb 15, 2010 Refund To ******* Completed Details -$439.89 $14.08 -$425.81 USD



    Again we apologize for this inconvenience caused, and thank you for your understanding and patience in this situation.

    Cheers

    Quote Originally Posted by wqmeng View Post
    After waiting another 24 hours, I get nothing reply to my ticket, Then I pm Cirtex if HostV have any new backup can help me get my data, Cirtex just told me that THEY never have any new backups for their customs, All the craps backup have been recovered, So I just waste another 24 hours with these guys.

    And HostV refused to refund the first month of my second year invoice to them, They have promised that as the big crap failure of their vps, they may full refund their custom. But they not refund me the first month, And even the month, my sites also get over full 3 days downtime due to the node 16 failure, almost all my data get lose 16 days later.

    Cirtex, HostV only keeping kidding on their forum and here, Now they have deleted the whole thread related this crap vps on their forum.

    What's HostV doing is setup a new node after 15 days full downtime, and give their custom a 2 month old crap backup!!!

    WTF!
    █• CirtexHosting • Providing Affordable and Quality Web Hosting & Reseller Hosting since 2003
    █• LINUX based cPANEL/WHM Shared and Reseller Web Hosting with Fantastico
    █• HostV VPS • Premium Virtual Private Servers & Dedicated Servers powered by cPanel/WHM
    █• We transfer your sites over quickly! • I eat penguins for breakfast ...
      0 Not allowed!

Page 1 of 2 12 LastLast

Similar Threads

  1. Hostv Down For 120 hours!!! Hostv Screw Me up?
    By desirez28 in forum Providers and Network Outages and Updates
    Replies: 42
    Last Post: 02-19-2011, 10:42 AM
  2. Replies: 15
    Last Post: 01-17-2008, 11:43 PM
  3. So .eu was a fiasco (long) - which next?
    By Bladerunner in forum Domain Names
    Replies: 0
    Last Post: 05-12-2006, 09:38 AM
  4. A Fiasco called Midphase Inc.
    By kkarank in forum Web Hosting
    Replies: 72
    Last Post: 05-18-2004, 03:40 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •