Results 1 to 8 of 8

Hybrid View

  1. #1
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    9,072

    Unhappy Ubiquity Server's Dallas - Offline

    Ubiquity moved their entire Dallas operation on Saturday resulting in almost 20 hours of downtime for us, despite their original announcement saying no more than 2 to 3 hours. No real updates as to what happened, outside of a brief notice about 6 hours into the outage that the work would be another 8 hours and that's the last I heard from them.

    Fast forward to about 40 minutes ago and they are offline again. No notice on their announcements page. A brief update to a ticket that they are aware of an issue. Ubersmith is showing the switches status as down when I check the MRTG graphs. Absolutely nothing to report to our clients who are already nothing short of enraged about what happened on Saturday...

    Don't get me wrong, the guys over at Ubiquity have been great to us over the last year but this whole weekend has been nothing short of a mess due to the terrible communication. I'm honestly saddened by this, Ubiquity were one of the few providers I truly believed in but this whole experience has left me with a real bad taste. Sigh.

  2. #2
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    9,072
    ... and we're back. There better be one good RFO and decent SLA credit in store here. I am not happy with Ubiquity at the moment, and I never thought I would say that. Sigh.

  3. #3
    Where did they move to and from?

  4. #4
    Join Date
    Nov 2007
    Location
    Phoenix, AZ
    Posts
    1
    Hello Patrick,
    The full announcement and updates were found here: https://support.ubiquityhosting.com/...ed-maintenance

    We posted this on July 1st to give enough time for everyone to prepare for it. We initially stated in the article: We estimate that the process will take approximately 6 to 8 hours to complete, although additional time may be required.
    --I do know we updated this once stating we need to extend the window by an additional 8 hours.

    Although that had to be extended for some other reasons. For exact reasoning however(I was not actually at the data center performing the move and so forth), feel free to submit a ticket to our support center here, https://support.ubiquityhosting.com/, anytime, and we can have it reviewed for you as well as an SLA request if you have not already done so.

    Last night we experienced a large scale DDoS attack that was mitigated so I apologize for the issues, but the data center move should be completed at this time, and we're working on getting all machines fully operational that are still having issues, which is minimal.

    Quote Originally Posted by jayglate View Post
    Where did they move to and from?
    XO Communications to CoreXchange

    Thanks!

  5. #5
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    9,072
    I know you guys stated 6 to 8 hours, but I assumed that was for the entire operation from start to finish and I quote your announcement:

    "The total amount of downtime experienced by each server will vary based on its location in our current facility, but we expect that most systems will be down for no more than 2 to 3 hours."

    Did you mean that the servers would be powered off for 2 to 3 hours but then the network would take another 8+ hours to come back online which kind of makes that 2 to 3 hours window pointless? If that's the case, then that's a rather disingenuous way of wording the announcement. We were down for almost 20 hours, give or take an hour. I forget how long exactly, but it was definitely above 16 hours.

    I felt bad starting this thread, you guys are an exceptional provider when things are going great. Tickets answered in minutes, helpful answers at that, non of that canned BS replies some other data centres use. However, I still feel like you guys dropped the ball big time on this one in the communications department. In a 18 or 20 hour outage, one update that only says it's going to be another 8 hours isn't very helpful or reassuring to clients. I'm sure you guys were running around trying to get everyone back online, but a quick update of why things are running late would have been nice.

  6. #6
    I wasn't too impressed with this either. I don't think they ever actually emailed me about it - just posted it on their support portal. I only go there to file tickets, so if nothing is wrong and I don't need to order anything, I don't go there. I was therefore unaware it was going to happen until just a couple of days before. You would think it would occur to them to send out emails...?

    Service at this location has been crappy as well. Their sales guys are awesome, and their techs do a good job... sort of. There have been performance issues with Dallas for months. The old location was marginal. The new location is better, but still nowhere close to acceptable. Our server seems to drop off the net completely for ten minutes every morning, for no apparent reason, and there is often terrible latency. Ubiquity's techs seem generally unaware, and they don't seem to be getting the hint that there are real, serious problems with that location.

    Is this because the network infrastructure in Dallas is crap? Or maybe the local talent pool is on the shallow side? I don't know for sure. I have heard stories of bad performance from people who were hosted in other data centers in that city, and they seem to have problems like these on a regular basis.

    Whatever it is, I'm getting well and truly sick of it. So are my customers. I get two or three tickets every single week about lag and complete dropouts, and MRTG shows that the server sometimes just vanishes off the internet for five or ten minutes at a time.

    I also have servers from Ubiquity in two other cities (New York and Seattle) and we have never, ever got any complaints about latency in either place - not ONE! So what's with Dallas? Well, maybe someone cares, but that someone is no longer me. I'm pulling out by the end of the month.

    Ubiquity has otherwise done a reasonably good job. I like working with them. It's a shame that Dallas is such a failure vortex.

  7. #7
    Join Date
    Nov 2007
    Location
    Phoenix, AZ
    Posts
    1
    Quote Originally Posted by Sir Stu View Post
    I wasn't too impressed with this either. I don't think they ever actually emailed me about it - just posted it on their support portal. I only go there to file tickets, so if nothing is wrong and I don't need to order anything, I don't go there. I was therefore unaware it was going to happen until just a couple of days before. You would think it would occur to them to send out emails...?

    Service at this location has been crappy as well. Their sales guys are awesome, and their techs do a good job... sort of. There have been performance issues with Dallas for months. The old location was marginal. The new location is better, but still nowhere close to acceptable. Our server seems to drop off the net completely for ten minutes every morning, for no apparent reason, and there is often terrible latency. Ubiquity's techs seem generally unaware, and they don't seem to be getting the hint that there are real, serious problems with that location.

    Is this because the network infrastructure in Dallas is crap? Or maybe the local talent pool is on the shallow side? I don't know for sure. I have heard stories of bad performance from people who were hosted in other data centers in that city, and they seem to have problems like these on a regular basis.

    Whatever it is, I'm getting well and truly sick of it. So are my customers. I get two or three tickets every single week about lag and complete dropouts, and MRTG shows that the server sometimes just vanishes off the internet for five or ten minutes at a time.

    I also have servers from Ubiquity in two other cities (New York and Seattle) and we have never, ever got any complaints about latency in either place - not ONE! So what's with Dallas? Well, maybe someone cares, but that someone is no longer me. I'm pulling out by the end of the month.

    Ubiquity has otherwise done a reasonably good job. I like working with them. It's a shame that Dallas is such a failure vortex.
    We post the news and you would get the email if you subscribe to support.ubiquityhosting.com which we recommend doing. That's where we keep all of our news, maintenance, and outage reports at.

    As of right now, there are no issues outstanding. In the event that issues are brought to our attention, and networking issues, we resolve the issues as quickly as possible. We do monitor our network 24/7 as well so if there are issues, our networking team is normally already working on them. If you have ticket IDs or anything we can look into, forward them over to us and management and we'll be glad to take a look. I am sorry to hear you're pulling your stuff out however by the end of the month as well.

  8. #8
    Quote Originally Posted by UbiquityAllen View Post
    As of right now, there are no issues outstanding. In the event that issues are brought to our attention, and networking issues, we resolve the issues as quickly as possible. We do monitor our network 24/7 as well so if there are issues, our networking team is normally already working on them. If you have ticket IDs or anything we can look into, forward them over to us and management and we'll be glad to take a look. I am sorry to hear you're pulling your stuff out however by the end of the month as well.
    I'll include some ticket numbers in the cancellation ticket. To be fair, I think a lot of this is just who your upstream providers are peering with. Take Global Exchange (GLBX) for example. They have a router that likes to drop traffic like it's gone out of style, but they are too busy being generic corporate tools to listen to anyone who isn't their direct customer. I wish none of my customers routed through them, but unfortunately a lot of them do.

    Again - I like you guys a lot. Your service in NY and Seattle is outstanding, and for all the tickets we get about Dallas every week, we get none at all about the other cities. But whether it's your fault or not, Dallas has been a pig for the last month and a half, and we're done asking our customers for pathpings.

Similar Threads

  1. Replies: 0
    Last Post: 07-21-2008, 03:59 PM
  2. Replies: 1
    Last Post: 02-17-2008, 08:49 PM
  3. Replies: 0
    Last Post: 12-10-2007, 08:35 PM
  4. Replies: 0
    Last Post: 09-22-2007, 04:55 PM
  5. Replies: 0
    Last Post: 05-01-2007, 12:56 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •