Web Hosting Talk







View Full Version : Anyone Else's Server Down?


Matrix
02-10-2002, 08:48 AM
with Weinbar?


Thanks

MadMax82
02-10-2002, 11:43 AM
Our server with Weinbar is also down at the moment. Any contact from Weinbar?

MadMax82
02-10-2002, 11:44 AM
I should clarify, we simply can't reach the server and traceroutes from all over are dying before getting to our server. Sounds like a network problem.

MadMax82
02-10-2002, 11:55 AM
We are currently timing out at 140.99.96.18 on our traceroute no matter where we tracert from. Are you timing out at the same point?

StephenRS
02-10-2002, 12:19 PM
We too are down at Weinbar.

They lost power, I know - because one of our servers requires manual restart when power lost. I think it was 3:00am...

Our 2nd server doesn't require manual intervention, it came back up ... I sent email to weinbar to bring up the problem server and they did (without replying).

However, as you guys are reporting - we can't connect to the servers from some places on the Internet. From NAC.NET in NYC I can connect, but not from anyplace on the West Coast....

Matrix
02-10-2002, 12:29 PM
It appears to connect on some of the trace route locations but not on others.

MadMax82
02-10-2002, 12:33 PM
We are now also seeing some of the traceroutes go through but of course none where we are (southern east coast).

MadMax82
02-10-2002, 12:43 PM
We are up now! Hope the rest of y'all are up now or will be soon.

Matrix
02-10-2002, 12:48 PM
MadMax82,

You're still down from my end.

MadMax82
02-10-2002, 12:57 PM
Yep I spoke too soon. Seems they rebooted the server but now I can not connect again. :bawling:

StephenRS
02-10-2002, 01:06 PM
I can't reach www.weinbar.com or my severs from locations on the West coast.

Here is the last part of my traceroute:

13 109 ms 145 ms 158 ms POS4-0.XR1.LAX2.ALTER.NET [152.63.115.218]
14 135 ms 141 ms 74 ms 195.ATM7-0.GW3.PHX2.ALTER.NET [152.63.114.137]
15 58 ms 116 ms 243 ms 12.125.99.66
16 141 ms 169 ms 147 ms ge1-2-1000M.sw1.core.phx.puregig.net [140.99.96.70]
17 148 ms 83 ms * deru-gw.puregig.net [140.99.96.18]

Seems their path in from puregig.net is broken....

Puregig.net is a co-location provider in Phoenix! I wonder if this place is in the same room as Weinbar...

MadMax82
02-10-2002, 01:09 PM
Ok now things get worse. My server just rebooted without me initiating the command and now my root password appears to have been changed so no access via the control panel or telnet. Also no response from Weinbar and no answer on the phone. This is not good...

Rydia
02-10-2002, 01:29 PM
Yeah, my site is also down, and all of my friends who are hosted on the Veronica server or whatever have sites that won't load either. What's going on with Weinbar's quality lately? It's been pretty bad for the past month and a half or so - I used to never have problems. o_O

StephenRS
02-10-2002, 01:36 PM
They used to be perfect...

But they seem to have a lot of problems on weekends. I wonder if they are not coordinating upgrades correctly?

The pre-Christmas weekend power outage (Saturday morning) really screwed us.... and here another power outage.

Two power outages in 45 days... not good.

MadMax82
02-10-2002, 01:38 PM
Has anyone actually talked to or heard from Weinbar? We don't need support that often with our server so we were not aware they are no longer offering any phone support and they have changed thier trouble ticket system to an email only system for the moment. Apache is still down for us and I know what the problem is I just can't get into the box at the SU level. Very frustrating.

MadMax82
02-10-2002, 01:55 PM
Just talked to Weinbar and they did have a power outage. They changed my root password so I do have control now. My traceroutes have been working fine for about the last 30 minutes at least from my area. Hope the rest of y'all are back up soon.

StephenRS
02-10-2002, 01:58 PM
West coast routes are still down, so expect to have people complain that they can't reach your site...

MSW
02-10-2002, 02:07 PM
Hi Everyone. I feel your pain. I have been at the datacenter all night along with 5 other techs. Yes, indeed Phoenix is having a slight outage, and it seems to be with the AT&T route into the NOC.

We are working with them constantantly and expect this outage to not be much longer. As far as us having problems lately, that just is not true. The servers and network uptime have been terrific and over the course of a year, there has only been about 3 downtimes. In anyone's book, that should be pretty good.

Please bear with us, as we are feeling the brunt of everyone. Most people can access the network, but as soon as AT&T gets their act together, we will all be able to enjoy WeinBar's network to the same satisfactory level as it was before.

Thank you for your cooperation and we will update you on our forums as soonas it is resolved, since this is not the place for service issues.

If you have any immediate concerns, please email me at mweinstein@weinbar.com.

Thanks, and have a great weekend.

Rydia
02-10-2002, 02:31 PM
Yeah, I personally haven't had that many problems (except for the site being down like twice last month), but some of my friends have had a lot of complaints. :/ Anyway, thanks for working so hard to get things fixed and letting us know what's up with the server! =)

StephenRS
02-10-2002, 03:48 PM
Broken routes are working again, just started.

MSW
02-10-2002, 04:36 PM
Originally posted by Rydia
Yeah, I personally haven't had that many problems (except for the site being down like twice last month), but some of my friends have had a lot of complaints. :/ Anyway, thanks for working so hard to get things fixed and letting us know what's up with the server! =)

We are aware of the issues with your friends. They were all on one server. The server in question has been adjusted and there should be no further problems.

StephenRS
02-10-2002, 11:28 PM
Michael - the support method really needs improvement. Right now (7:00pm PST) I have a server that needs a reboot... I sent email but I have no idea if anyone is there to read it...

drewnick
02-10-2002, 11:48 PM
That is bad. I think the proper way to do it is the way we do it.

That entails 24x7 30-sec monitoring of servers so in the event of a failure the right people are dispatched before the client generally even realizes it.

So when there are comments like "I'm going camping and cannot be reached, can you handle it?" you have a response. They're proud to learn that the server is already under 24x7 watch.

Regards,
Drew

MSW
02-11-2002, 12:36 AM
Originally posted by StephenRS
Michael - the support method really needs improvement. Right now (7:00pm PST) I have a server that needs a reboot... I sent email but I have no idea if anyone is there to read it...

Of course we are here to read it. We did read it, and we did reboot your server. Please do not make these accusations in this forum. I do not frequent it because of the way WHT has turned.

BTW, how unhappy have you been with our service?

StephenRS
02-11-2002, 01:00 AM
Michael - your message is nonsense.

I've been happy, EXCEPT for the problems! Today has been a terrible day for us at Weinbar, as was the time in December when our server lost power. Power outages at a co-location site are serious issues.

You are wrong, you did NOT reboot my server. My request to reboot it (sent at 7:08pm PST Sunday) was never answered. I eventually was able to get into the system and reboot it myself (8:40pm PST)... but I never got a response to my email at 7:08pm PST.

PLEASE don't reboot it now - I sent another email at 8:37pm PST requesting cancel of prior request. But my complaint about your email support is legit, my 7:08pm email was never answered.

Your support method of allowing only email and no phone/instant message is something to compalin about. Even when I follow procedure, not getting replies on the emails for hours does not help the situation.

I'm not some idiot customer who needs handholding, I have only sent in legit problems in the 9 months I have had servers at your site. I think I have only needed assistance a total of 5 times in that 9 months, 3 of those which were major outages on your part (2 of which were power failures at your site).

EXAMPLE of slow support via email: Outage starts at 3:00am, I sent in email at 3:54am - I don't get a reply until 11:34am (all times PST). This was for a server down, not some "can you install more RAM" issue... This was assigned Weinbar incident #1196, research it yourself.

I have been a long time supporter of Weinbar, I don't like the (written) tone you have taken with me. Eliminating phone and instant messaging support are legit customer support issues when it takes over 7 hours to get email responses.

MSW
02-11-2002, 02:29 AM
Stephen:

We have not eliminated the phone support. If you have a serious issue, please use the phone number that is posted on our web page. I will give it to you here in case you missed it: 480-661-1300.

I am not wrong about rebooting the server. It was done. I am taking this tone because it seems as though you are attacking us for an incident which happened to be longer than expected. If you would like the full story, here it is.

We have been at the Fourthstage (Aperian) datacenter since we began doing business. Well, Fourthstage filed bankruptcy and we were not going to take the risk of having their service cut off, in turn having your service cut off. We were making arrangements to move into our new datacenter and at 2:00AM, a total of 25 people came and moved the equipment from Fourthstage across the street. Total time to move over 100 servers was 30 minutes. Well worth it for the risks that could have been. We have maintained the IP addresses for the time being, but we will be reassigning from our class B to our clients shortly. The latency that was going on today was that AT&T never recognized that we were broadcasting those IPs although they were supposed to. It was working fine in the tests that were run days before. Right now we have 9 carriers in the datacenter, and AT&T fixed the issue.

We are available to you 24/7. We have increased our staff over 300% and are growing at extraordinary rates, especially in this economy.

My tone to you may have been a bit misguided, and I apologize for that, but I think for the money that yoou are paying, you are getting above average quality. If not, please speak to me offline and we can discuss it.

StephenRS
02-11-2002, 11:46 AM
Michael,

I called your phone number dozens of times, there was never an answer. There isn't even a messaging device on it... It is also labeled only "sales".

I am not new to the web hosting business. I have over 8 years of hosting experience.

If you read earlier in this thread, I even speculated that (direct quote) "But they seem to have a lot of problems on weekends. I wonder if they are not coordinating upgrades correctly?"

You just described a classic lack of coordination in a upgrade.

I'm sorry, but you made two very basic mistakes:

1. You knowingly TURNED OFF my server without giving me a chance to shut it down. If my database had corrupted, this would have cost my client thousands of dollars for the labor to rebuild it from backups, plus would have had them down the most busy day of the week (Sunday).
2. You did not notify me of a planned interuption in service.

You don't just get 25 people to show up in the middle of the night without some effort, this was a planned outage.

I measure a hosting company on track record, and up until these last two power outages, Weinbar had a decent one. We even expanded to two servers (with third in the works) after a few months with you. I feel like you violated that trust you earned by not doing due dilligence to notify your clients.

To you, weekends may seem slow - for the sites I work with - they are the most busy times!

God, I hate to think what would have happened if my server were damaged during the move. Not giving any notice is just asking for problems.

and no, as you continue to argue with me, YOU DID NOT REBOOT my server! I logged in myself at 8:30pm and it was still running programs that I manually started. Unless you cracked my passwords and know how to manually start some complicated programs, it was not rebooted by your staff.

I will contact you online, but this incident deserves more than a publish lashing. I'm willing to move foward, but you need to demonstrate that trust is a two way street. Get your facts straight, and make some effort to prevent these problems by better customer communications and better emphasis on customer support.

MSW
02-11-2002, 01:43 PM
Stephen:

As you have your opinions on how to run my business, I have mine as well.

I was not going to let Fourthstage turn off our servers and have no business left to run. Yes, getting 25 people did take a lot of coordination, but time was of the essence and it was a business decision that I had to make. If you were not notified, then I am sorry, but I was looking out for ALL my customers.

As far as not letting you know that we turned your server off, I never knew it was a requirement. Our contract does not state that if we need to shut your server down for any reason that we must let you know about it. If there was a power outage (and this was not), would that be my fault? Or, what if your server s power supply failed?

As far as moving the servers, we are professionals. We do not just unrack them, throw them in a box and send them on their way. 100% care is given and each server was hand delivered to the new datacenter. It is only across the street.

I am never one to be argumentative. I am only here to protect my business, and I feel that this was the best decision I had to make, and I don't appreciate anyone second guessing me. I know that in the long run, this is best for our company. I will be more than happy as I said to talk to you about this offlie as this is not the place, but it seems as though you only like to express your opinions here.

Our weekends are not slow. They are they busiest time, but at 2:00 in the morning, it was the best call to make that decision. If you want me to show you my full business plan as well, that is something that we will have to discuss. But, until then, either have faith that we run a tight ship, or do something that will make you more comfortable. Right now, all servers are running better than ever, my staff is comfortable, my attorneys are comfortable and my business is as sound as ever.

I am ending this conversation with you on WHT. You have my email address if you would like to talk further.

StephenRS
02-11-2002, 02:08 PM
You have not actually addressed the issues I have brought up. Where is the proof you rebooted my server Sunday night? Did you not interrupt my service Sunday at 3:00am without notice? Did you not answer your phone all Sunday morning? Did you not answer your emails for 6 hours?

Michael, get a customer service clue, my complaints are legit. You should acknowledge them. You don't even have to promise to fix them, but stop acting like it is not appropriate for me to complain.

=================

As for your question: "[if] there was a power outage (and this was not), would that be my fault?"

Uh, yes. Does your co-location environment not have UPS/generators? Your web page says "highest standards of [snip] Facilities Design".

================

What part about your web page that says "keep your mission-critical servers operational 24 hours a day, 7 days a week" do you not understand?

Do yourself a favor and appologize. I am a customer, I am complaining that your support system was not there when I needed it.

I also don't appreciate you bashing the WebHostingtalk forum. Without this forum, I wouldn't have found out about your service.

You feel free to contact me offline, Stephen@IPCoast.com - I don't have a problem discussing ISP's on an ISP forum. I don't have a problem with e-mail either. For the record, it was not me who started this thread. believe me, your forums were not online at 3:00am Sunday morning when my pager started going off that our server dropped off the Internet. It would have been more than apropriate to post here, but I gave you the benefit of the doubt...

StarGate
02-11-2002, 02:12 PM
... off topic but I saw this signature and was reminded that I have a latin diploma :D

MadMax82

Qui desiderat pacem, praeparet bellum
Whom decides peace, prepares war

like that.. true though :(

:)

MSW
02-11-2002, 05:55 PM
Stephen:

I did apologize, please read previous threads directed to you.

I have acknowledged your issues, although I do not agree with them. It is perfectly acceptable to complain. I do it all the time.

Once again, I acted as the CEO of my company, and did what I perceived to be the right move. I was looking after you, and every other one of my customers. I wanted you to have as good connectivity as possible at the best possible price.

As far as your server goes, I was not the one who actuallly booted it, so I will take it up with the tech who did. It is possible that he booted the wrong one, and if that is the case, then I will deal with that issue.

Our forums were most definately up Sunday morning. You may not have been able to reach it due to a routing issue.

The power outage scenario was wrong, but the power suplly scenerio would have been better used.

I do not bash the WHT forum, but this is not the place for support.

If the phones didn't answer it was because we were all working at the datacenter. Even our accountant was there. Look, I am not saying I was wrong, and I appreciate your concerns, and yes, I do apologize for your misfortunes. But, I do not apologize for my actions because in the benefit of running a company (not just a webhosting server), I believe I had to make a quick decision because Fourthstage could be turned off at a moments notice.

I hope we can put this behind us and move forward. I understand your business is running well, and that is wonderful. Keep up the good work, and I will do my part to help you along the way.