Results 1 to 15 of 15
  1. #1

    R1Soft on Steadfast.net --> No Minimum Level of Service

    Heya all,
    I guess this is an open ended question. I have R1Soft backup services with Steadfast.net as they have always had good service and great prices.

    However 3 days ago I thought that I would run some maintenance on my backups and tried to log into the backup servers. When I tried to go to the control panel it said the system was down for maintenance... that was strange.

    Checked my emails, no notification of maintenance time. Checked the news page at steadfast, no notice of maintenance. I logged a job and was told that the system had been down for roughly 3 days (when you add that together we are now at roughly 6 days as best I can estimate) that the system has been offline.

    After many emails back and forth, I have found out they are running an fsck on the entire backup server and it has taken all this time to do and is still running.

    I also voiced my concern that no notice whatsoever was provided of issues and I was told that I should have notifications set up when backups fail (which I do and have received no notice), but this still doesn't account for not notifying customers of maintenance to a server.

    What I am wondering is:

    1. Should companies really be notifying their customers of maintenance to such critical systems proactivly (Singlehop does, I have my actual server with them, they notify me about anything if it may *remotely* affect my services, really happy with them BTW).
    2. Should they actually be running such a maintenance task which can take the system offline for so long without providing some other solution at least temporarily (they ignored that question when I sent it to them BTW)
    3. Am I being a jerk? Really I try to be nice and I have been polite in all my communication but seriously, I wanna know if my server has not been backed up for 3 days (well 6 now) LOL.

    They have given me a free month next month, but that does not discount the fact that if my server was to go offline right now, I would loose almost a week of data... which is less than impressive for me.

    What are your thoughts, is there some minimum level of service I should be able to expect?
    Cheers
    Stuart

    (in the mean time, does anyone know where I can get some cheap backup space ~ 50GB)?

  2. #2
    Join Date
    Mar 2003
    Location
    chicago
    Posts
    1,781
    i think a free month is as much as i would expect from any company in that situation.

  3. #3
    Join Date
    Feb 2004
    Location
    UK
    Posts
    1,431
    Hi

    Maybe if the server was rebooted after a kernel update (or something), if it had been up for longer than a certain length of time, the system can force a fsck. It may be caught them by surprise... However I have to say that after 3 days I think they should have notified people...

    Most customers would be blissfully unaware.

    If you are concerned about data loss, I would look to backup your files manually to another backup location.

  4. #4
    Join Date
    Jan 2003
    Location
    Chicago, IL
    Posts
    6,957
    This is a unique situation that did result from a scheduled maintenance, which is on our site and was sent out as our scheduled maintenances are. The primary part of the maintenance went fine, but it seems kernel updates done to a couple of the storage arrays at the same time caused some completely unexpected issues. It is extremely unusual that a kernel upgrade would end up causing data corruption, and with that part of the maintenance we were not expecting it, and thus did not really look for it. A couple hours later, we noted some odd behavior, took some of the storage arrays down to run fsck and the fsck seemed to actually increase data corruption. We then downgraded everything back to the way it was, ran fsck again, etc. and most things came back up. Right now, a vast majority of the customers are up, and most had had minimal downtime, considering the circumstances, but one of the arrays has been basically been running fsck forever now, though it is successfully repairing things.

    Overall, this resulted from a scheduled maintenance, so it was sort of tied into that, and it seemed things would be resolved very quickly, as many things were. Then after that, with what was still being waited on to come up, we really had no ETA, it could come up in a couple minutes, though we really never expected it to take a couple days. Looking back though, I do certainly agree we should have handled this better and we have already had discussions internally about how to better handle this in the future.

    So the basics:
    1) This was the result of a scheduled maintenance, which was announced on the web site and via email, to those subscribed, as per our terms.

    2) Yes, you should have been given more updates. We generally hate giving vague updates, but in some cases, it is simply necessary, as some information is certainly better than none. Also, if you asked for another option, you should have been offered to go to our standard FTP/rsync service, which was largely unaffected. Can you PM me the ticket number?

    3) We will certainly honor our SLA, and any SLA requests submitted will be honored with a 100% service credit.

    4) We have been working on additional storage options, including our own in-house SAN configurations across multiple locations. We are hoping to have additional redundancy available for our backup plans in the near future, which would make issues like this a thing of the past. The service we are offering as it is now though is specifically sold as a single storage location for backup data, offered at a lower price due to the evident single point of failure.
    Last edited by KarlZimmer; 06-29-2009 at 03:12 AM.
    Karl Zimmerman - Founder & CEO of Steadfast
    VMware Virtual Data Center Platform

    karl @ steadfast.net - Sales/Support: 312-602-2689
    Cloud Hosting, Managed Dedicated Servers, Chicago Colocation, and New Jersey Colocation

  5. #5
    Quote Originally Posted by KarlZimmer View Post

    So the basics:
    1) This was the result of a scheduled maintenance, which was announced on the web site and via email, to those subscribed, as per our terms. <-- I saw something regarding this on the site, but never got emailed as per my discussion... but that is cool.

    2) Yes, you should have been given more updates. We generally hate giving vague updates, but in some cases, it is simply necessary, as some information is certainly better than none. Also, if you asked for another option, you should have been offered to go to our standard FTP/rsync service, which was largely unaffected. Can you PM me the ticket number? <-- That would have been/still be awesome I will pm the ticket number.

    3) We will certainly honor our SLA, and any SLA requests submitted will be honored with a 100% service credit. <-- That has been done already thanks.

    4) We have been working on additional storage options, including our own in-house SAN configurations across multiple locations. We are hoping to have additional redundancy available for our backup plans in the near future, which would make issues like this a thing of the past. The service we are offering as it is now though is specifically sold as a single storage location for backup data, offered at a lower price due to the evident single point of failure. << - did not realise that it was a SPOF, will keep that in mind for the future.
    And that is great to hear that there are talks internally, I guess that was my biggest gripe of all, a lack of communication and updates.

    Will PM you the details requested and thanks for the response.
    Stuart

  6. #6
    Join Date
    Dec 2006
    Posts
    398
    Quote Originally Posted by KarlZimmer View Post
    So the basics:
    1) This was the result of a scheduled maintenance, which was announced on the web site and via email, to those subscribed, as per our terms.
    I know that I did not receive this email (and yes, spam folder was checked).

    3) We will certainly honor our SLA, and any SLA requests submitted will be honored with a 100% service credit.
    I sent a request for this Friday morning to billing @ steadfast.net (as per instruction from your support) and have yet to hear anything back. Should I continue to wait?

  7. #7
    Well I am glad it was not just me that did not receive the email, (like you I checked all my spam folders etc too).
    Stuart

  8. #8
    System just came back online it would seem, 4 and a half days (give or take) between last backup to now, so I can now provide that value for clarification.

  9. #9
    Join Date
    Feb 2007
    Location
    Chicago, IL
    Posts
    205
    Quote Originally Posted by jasonaward View Post
    I know that I did not receive this email (and yes, spam folder was checked).
    Are you subscribed to the announcements mailing list?

    Quote Originally Posted by jasonaward View Post
    I sent a request for this Friday morning to billing @ steadfast.net (as per instruction from your support) and have yet to hear anything back. Should I continue to wait?
    What is your ticket number? I'll look into it for you.
    Marc Schulz
    Managing Implementation Engineer
    Steadfast : Managed Cloud & Infrastructure Services

  10. #10
    Join Date
    Jan 2003
    Location
    Chicago, IL
    Posts
    6,957
    Quote Originally Posted by jasonaward View Post
    I know that I did not receive this email (and yes, spam folder was checked).



    I sent a request for this Friday morning to billing @ steadfast.net (as per instruction from your support) and have yet to hear anything back. Should I continue to wait?
    Well, here is the announcement: https://support.steadfast.net/index....&group=default It was certainly sent to everyone subscribed to receive such updates.

    Can you PM me the ticket # of the billing ticket? All the billing tickets from Friday had been answered long ago.
    Karl Zimmerman - Founder & CEO of Steadfast
    VMware Virtual Data Center Platform

    karl @ steadfast.net - Sales/Support: 312-602-2689
    Cloud Hosting, Managed Dedicated Servers, Chicago Colocation, and New Jersey Colocation

  11. #11
    Join Date
    Dec 2006
    Posts
    398
    Quote Originally Posted by KarlZimmer View Post
    Well, here is the announcement: https://support.steadfast.net/index....&group=default It was certainly sent to everyone subscribed to receive such updates.

    Can you PM me the ticket # of the billing ticket? All the billing tickets from Friday had been answered long ago.
    You may want to make people aware that they should subscribe to receive the updates, because I was completely unaware that such even existed. I would assume you'd just send out emails to those affected by maintenance period regardless of whether they are subscribed to a mailing list or not.

    I sent an email to billing @ steadfast.net but never received a ticket number back. I'll just submit a new ticket via the ticket system directly.

  12. #12
    Join Date
    Jan 2003
    Location
    Chicago, IL
    Posts
    6,957
    Quote Originally Posted by jasonaward View Post
    You may want to make people aware that they should subscribe to receive the updates, because I was completely unaware that such even existed. I would assume you'd just send out emails to those affected by maintenance period regardless of whether they are subscribed to a mailing list or not.

    I sent an email to billing @ steadfast.net but never received a ticket number back. I'll just submit a new ticket via the ticket system directly.
    The announcements page is listed in the SLA and terms of service, where it is easy to subscribe to either an RSS feed or to email notifications. We do specifically link to it in our shared hosting and dedicated server welcome emails, but it seems it is not in the backup services welcome email, thus I have now added that. Thank you for bringing that to our attention.
    Karl Zimmerman - Founder & CEO of Steadfast
    VMware Virtual Data Center Platform

    karl @ steadfast.net - Sales/Support: 312-602-2689
    Cloud Hosting, Managed Dedicated Servers, Chicago Colocation, and New Jersey Colocation

  13. #13
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    OT - Curious as to the size/type of array that needs 3 days to fsck?
    Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  14. #14
    Join Date
    Jan 2003
    Location
    Chicago, IL
    Posts
    6,957
    This one is only ~10TB, but the issue is with the data from rsyncs that have a lot of tiny files and then the hard linking between those files, etc. It has ended up then in a whole bunch of "multiply claimed blocks" which is taking a lot of time to process through.
    Karl Zimmerman - Founder & CEO of Steadfast
    VMware Virtual Data Center Platform

    karl @ steadfast.net - Sales/Support: 312-602-2689
    Cloud Hosting, Managed Dedicated Servers, Chicago Colocation, and New Jersey Colocation

  15. #15
    Join Date
    Dec 2006
    Posts
    398
    This issue has been resolved for me. Thanks to Steadfast for their prompt assistance.

Similar Threads

  1. Replies: 4
    Last Post: 05-31-2009, 04:29 PM
  2. Replies: 12
    Last Post: 01-26-2009, 12:35 AM
  3. SteadFast Bad Service
    By avivronen in forum VPS Hosting
    Replies: 34
    Last Post: 08-11-2008, 01:17 AM
  4. R1Soft CDP Backup Service / Hosting
    By Kevin in forum Other Hosting Offers
    Replies: 3
    Last Post: 01-14-2008, 05:54 PM
  5. Replies: 9
    Last Post: 01-24-2005, 02:57 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •