Web Hosting Talk : Web Hosting Main Forums : Industry Announcements : Providers and Network Outages and Updates : The Hostworks Network Outage

[THW-Dave]

We are currently experiencing the aftermath of a network hardware failure, all services will be unavailable until the faulty hardware is replaced.

The hardware is a upstream provider issue, and we are currently working with Sprint and Verizon (local telco) on resolving this issue.

We will try to keep everyone up to date on our progress in resolving this issue.

Thank you

David @ The Hostworks
www.thehostworks.com

[bobbyt]

its been over a day and a half, what's an estimated repair time on this? My email has been bouncing all weekend, all my ebay auctions have broken images.

[bobbyt]

Over 2.5 days downtime now for your entire datacenter and 2 days of flakey connectivity leading up to this. Where are these updates you speak of? When will this be fixed? What is being done to ensure this is fixed by that time?

[bigjig]

Yeah, i'd like to have some info to pass on to others.

I'm a part of a nonprofit organization that is using a page there to run a large rundraising campaign at the moment... so this is bad timing to say the least.

This is the first downtime i've seen since i've been with them, however... and i've been with them maybe 3 or 4 months?

[Papa Smurff]

Wow, this is a big outage. Glad I don't colocate with Hostworks.

What is the EFT (Estimated Fix Time)

[bigjig]

It's EOD and no ETA on EFT. Hopefully he can RE to our posts ASAP.

[ryce]

resolved ns2.thehostworks.com to 24.97.13.162
resolved 24.97.13.162 to rrcs-nys-24-97-13-162.biz.rr.com

Maybe they forgot to pay the cable bill

[THW-Dave]

Thanks for the investigative research, Ryce,

We have had MAJOR problems, well, not just us, but 1200 other local circuts in our neibhorhood.

Due to water damaged, Verizon has 1200 pairs damaged, and have been working around the clock to repair these.

At this current time, Verizon has officially posted a ETR or 10:00am Tuesday Morning.

We are doing our best to keep everyone up to date, unforturnatly, many clients primary email on records point to servers we have in our DC, which renders them useless in this matter.

We have spoken to most of our clients, as we have been explaining the progress to those who have been in contact every step of the way.

We are gifted with the patience of our clients at this point and are thankful to have such a understanding and patient client base.

Thank you for your concerns, we are pushing the LEC as hard as possable to futher quicken this repair.

Regards
Dave @ The Hostworks

[THW-Dave]

I will also like to add that our backup DNS is indeed on a dedicated cable link, which has proven to be beneficial to this, for we are still able to keep TTLs alive on all the domains we authortively host. And as you may have guessed, NS2 is in totally different physical location.

Thanks
Dave

[THW-Dave]

We are back up, 13 hours earlier than Verizon estimated, 10:34PM Monday Night EST.

I thank everyone for sticking with us through this extended outage, and for current clientele, please keep an eye out for a email outlining the event, and what we are prepared to do in return for your outstanding patience and understanding.

Sincerely
David @ The Hostworks

[bigjig]

I think it's down again

[bigjig]

It's back up again!

[THW-Dave]

Hah, yes Verizon has yet to issue a "all done".

They repaired the initial issue, from what we are seeing, but are performing preventive maintence on their network where the damaged occured.

Please check back for updates.

David

[bigjig]

Been down again for quite a few hours. Any new updates?

[THW-Dave]

We have been down for the past 3 hours to do the cable damage which was temporarily fixed last evening by Verizon, we are waiting to hear another ETR on this outage, and I will keep everyone up to date on the progress.

Thank you for you patience.
David