Just received this from RS but only 1 of our 4 servers is affected.
Are all their servers in the same building, and how did they damage multiple lines?





------------------------
We have had multiple gigE fiber lines cut and are working to bring these
back up as quickly as possible. We should have this resolved no later than
2 more hours. We apologize for this outage but unforunately it was outside
our control.
------------------------



Gordon

looks like ....

cogent and time warner line has been cut
:confused:

There are updates in our forum and in our chat room.

We lost two linear Gig-es that are provided over Tw transit. We have now managed to turn up some dormant OC-3 circuits temporarily to alleviate the loss.

All servers, save 8 high bandwidth customers, are back in business and two of those we delinquent. So, 6 real customers affected at this point. We expect those 6 to be back in business in the next 30 minutes.

Updates will be posted on the RS forums.

Thank you very much for a update HeadSurfer, I still think you need a revolving status page. That is UPDATED at all times.

See the attached gif for ideas!

Heck, I would keep it updated for you.

The server is back on line but not before I received this
from RS support:



------------------------
7/12/02 - lpate
we found out that the reason why server was unpluged was per report for
over charge bandwidth, cust has been unplugged, cust needs to pay balance
before it is re-pluged, please make sure that server is unpluged, lp..
------------------------



Rackshack had not billed me for any overage or attempted to charge my credit card for it.

What sort of shambles is this company?

I have never heard of a dedicated server being unplugged for bandwidth overage when no bill has been sent and no attempt made to obtain the money owed.

I am going to write to Rackshack and ask for an apology.
What they have done today is completely unforgivable by anyones standards.

Gordon

Originally posted by ljprevo
Thank you very much for a update HeadSurfer, I still think you need a revolving status page. That is UPDATED at all times.

See the attached gif for ideas!

Heck, I would keep it updated for you.

I beleive its plastered all over that companies section you pulled that from that you can not publicly distribute that type of thing. :rolleyes:

Originally posted by Jag


I beleive its plastered all over that companies section you pulled that from that you can not publicly distribute that type of thing. :rolleyes:

I was thinking the same thing... and in addition, why block out the name if you're going to name the attachment a commonly used abbreviation for the company? :confused:

so if the board is down, we know rackshack is down too :D

Originally posted by JG


I was thinking the same thing... and in addition, why block out the name if you're going to name the attachment a commonly used abbreviation for the company? :confused:

good catch, I didnt even see that at first.... you could be asking for trouble with that post.

hehe. that's rackshack for you.

i had a sorta similar issue with them (well, not really). basically, i got a server from them. when i first logged in, it had already been up and online for a few weeks. anyways, a few weeks later my server gets taken offline because it was supposedly flooding their network. so yeah, i was accused of being a poor sys. admin. when the box had infact been sitting online unpatched with all services exposed for weeks before i had even touched it. gotta love it.

Originally posted by GordonH
The server is back on line but not before I received this
from RS support:





Rackshack had not billed me for any overage or attempted to charge my credit card for it.

What sort of shambles is this company?

I have never heard of a dedicated server being unplugged for bandwidth overage when no bill has been sent and no attempt made to obtain the money owed.

I am going to write to Rackshack and ask for an apology.
What they have done today is completely unforgivable by anyones standards.

Gordon

When has anyone been billed for overages? Last time I heard HS talk about overages he said they haven't billed anyone for overages because of overall everyone having a low average....

Originally posted by rastoma


When has anyone been billed for overages? Last time I heard HS talk about overages he said they haven't billed anyone for overages because of overall everyone having a low average....

Check out this thread on Rackshack's forum.
http://forum.rackshack.net/showthread.php?s=&threadid=8081

We have selected the top "over users" for billing for June. For July, billed in August, we will be billing all overages.

There have been too many people take advantage of us recently on this issue.

One gentleman , who was pushing about a 20 Mb stream, we could only collect about 1/3 of what he owed. Another has refused to pay.

Earlier today there was some confusion as to why about 10 servers were "null routed/unplugged" during the height of the fiber cut where we lost two of our Gig-e circuits. We had to make a very hard decision about how to provide the best possible service to the most people during this time when we lost a significant part of our network due to a garbage truck driver.

Out of 7,000 servers, we null-routed or unplugged the 10 Mb servers for a short time as well as our top ten bandwidth users. By taking these 160 or so customers out of service, we were able to route the balance of our customers.

As TW brought up the first emergency OC-3, we re-enabled the 10 MB servers. When the second emergency OC-3 came up, we re-routed the 10 "over users" back into the network.

There was obviously some confusion about what was done for what reason during this hectic time and some bad information did get distributed to a few of the customers involved. I am truely sorry for this miscommunication. I also shared these sentiemnts with the customers in our chat room, where I was available just about constantly, as well.

Two of our Gig-e circuit loops have first generation/early adopter technology running them in a linear fashion. All new Gig-e loops run in a redundant/non-linear setup. The old loops run on Lux-n gear while the new loops run on Extreme boxes. We are in the middle of a project to move the older Gig-es to the new Extreme non-linear configurations to reduce the likelyhood of a cut affecting us in this manner again.

It is unfortunate when any customer has an unavailability of their servers for any period of time. In this instance, we were hit with a catastrophic loss of connectivity, implemented emergency plans, and continued to work with our providers to make the best of a bad situation. While a large portion of our network experienced some latency and some very brief periods of packet loss, only about 160 servers lost total connectivity for a period during the fiber cut.

The miscommunication was unfortunate and I certainly offer my apology to all of those affected. This fiber cut just happened to come at the exact same time that we were going through and billing/collecting for the bandwidth overcharges. For customers in our "top 10" list, and located in the United States, I made personal calls to discuss their overages. During the period affected by the cut, I was in out chat room talking with our customers as well as posting updates to our forum on a regular basis.

Robert

Headsurfer, it isn't right for you to just take their servers off the network without notice either.

Yeah.. I'd be upset if I paid 400/month for a 10mbit deal, and the owner of the datacenter decide to not turn my port back on, while everyone else is up and running...

Robert

We had an overage of about 150GB.
I am not disputing it even though the MRTG and Ensim stats do not support it.

But taking aserver offline for a 150GB overage with no notification, not even an e-mail?
And when I phoned up, nobody in customer service knew why the server was off line.

Its a small amount of money to start taking servers off line for
under any circumstances.

Even in the emergency situation I would have expected my server to be treated the same as anybody elses.

This month so far it has used 200GB so its hardly one of your biggest.

Its simply bad business practice to pull servers with no notification or explanation.

Make sure this does not happen again.

Gordon

Originally posted by JG


Check out this thread on Rackshack's forum.
http://forum.rackshack.net/showthread.php?s=&threadid=8081


yeah.. i read that after i posted :)


GOOD!!! RS needs to bill for overages... that's the point of have a LIMIT :)

Originally posted by Jag


I beleive its plastered all over that companies section you pulled that from that you can not publicly distribute that type of thing. :rolleyes:

Try again, the only area that states what can't be displayed is the actual network issues, which I did not post. :rolleyes:

It's about time that Rackshack starts charging for overages, as it's not fair for customers who only use 50Gb/month or around there. For those small bandwidth users, it's means we get slightly slower speeds as the other large bandwidth users hog the bandwidth lines. Overall, it's going to affect Rackshack as a company, as if they don't get enough money, then they'll be the next company who files for bankruptcy... (even though we know that they're a profitable company, as Robert Marsh has said several times).

We're one of those users who use small amounts of bandwidth, and we've been happy with their service for the 6 months that we've been with them.

So to those people who use a LOT of bandwidth... no more free bandwidth! :)

Alan

Where did anyone state in this thread that customers should not be charged for bandwidth overage?

Gordon

GordonH, may I ask who you were replying to?

I don't think it says anywhere that people should not be charged for bandwidth overage, but Rackshack have not been charging for it for the past 6 months or more, which everyone or most people know and so have taken advantage of it. But now that Robert Marsh has finally got the system working and have taken action... it just makes his company stronger.

Alan

Good luck Robert getting everything back. I myself think it was a very gutsy decision turning off the 10MB/S Ports, and your 10 most bandwith hungry clients, just so the rest of the network could remain operational. I would have done the same if i was in the same situation. Good work, and i hope everything continues to go well.

I give up.

People seem to have taken this thread off in a weird direction.

I did my bandwidth analysis and I was 33% over
Its hardly the end of the world and it does not make us one of their 10 highest bandwidth users, especially as this month we won't be over.

We are buying bandwidth from them and we have every right to expect they will deliver it on demand.

Just unplugging a server without attempting to charge for the nadwidth is completely unacceptable and all you people who are suggesting it was a great business decision did not have to deal with the 100 customers whose sites were down as a result of Rackshacks incompetence.

On a general note:
I really despair of this board these days.
It seems to be full of people trying to make out they are more ethical or better at customer service than other hosts.
It used to be a place where people could help each other but those days seem to be over.
Its all about self promotion and trying to recruit customers these days.

Gordon

I think both sides are right here... we're right as Rackshack do need to sort out the large amounts of bandwidth that is not being paid for and people are using like it's free. While Gordon is still correct, as Rackshack did not try to get payment off them, as I'm sure you would have paid it if they did ask for it to keep your server online.

Robert was going on the stats he had for last month, so even if you were not going to go over your limit this month, he thought you were, without looking at the current stats, which was not good of him. And to conclude... I'm on your side Gordon, with this case :).

Alan

I am not sure that you guys still understand. Today's issue is not related to a partucular customer being over teh included amount or not. Gordon is a good and valued customer and I can certainly understand his frustration. I would hate to think what I would have written if I had been in his situation as I tend to be extremely hot headded.

It was a terrible, horrible, no good, very bad day (sure felt like a full moon and Friday the 13th all rolled into one) in my book and I am sure in Gordon's as well as well as anyone on "the list".

I was left with a very hard decision to make. I take full responsibility for it. In the end, I take full responsibility for the entire situation. That's my job .. take the credit when all is well and take the blame when all is "less than well".

Under the same circumstances, given a catastrophic event (such as losing 2 Gig-e lines and a 3rd running at only 50%), I would have to take the same actions. Granted, I woudl hope that the communication could have improved.

Monday will bring a little Monday morning "quarterbacking" as we try to understand what TRUE risks are out there, how they impact us, etc. It give us a great time to re-evaluate our disaster plans and take a harder look at our disaster recovery plan and what backup procedures and mechanisms we have.

Our gigantic size has really crept up on us over the last 5 months and we have a much larger responsibility because of it. I just hate to thing about the number of actual domains hosted on our servers as most of them are sold to resellers. (could it be 65,000 (10 per server), 195,000 (30 per server), ???)

Thanks again to everyone for you understanding during a very difficult afternoon.

Robert

Crap happens I guess, but if it happened to me, I would like to know what's going on before it happens. Thank goodness, I'm probably in the last 10, of lowest bandwidth usage... :)

Originally posted by GordonH
On a general note:
I really despair of this board these days.
It seems to be full of people trying to make out they are more ethical or better at customer service than other hosts.
It used to be a place where people could help each other but those days seem to be over.
Its all about self promotion and trying to recruit customers these days.
Gordon

Well said! :agree:

Well there you go... a reply from the man himself, Robert Marsh, the 'HeadSurfer' :). Thanks for the personal post there Robert, it realy helps when a CEO replies to a problem regarding his own company.

Alan

Originally posted by headsurfer

I was left with a very hard decision to make. I take full responsibility for it. In the end, I take full responsibility for the entire situation. That's my job .. take the credit when all is well and take the blame when all is "less than well".

Robert

Robert Marsh Rocks (RMR) :spiny:

Originally posted by mahinder


Robert Marsh Rocks (RMR) :spiny:
RMR! lol :D

Thank you Robert
I do hope you find a better way to handle billing situations
and emergencies like this.


One tip: if you get the right phone company you can call long diatance to here very cheaply.
I am sure you know that already.
We pay the same to phone Houston as we do to phone London
(and I am in the UK)

We call US customers all the time from here without spending a lot of money.

Gordon

Gordon, which telephone company are you using? Just wanted to know out of interest since I'm located in the UK aswell. (I'm happy with the setup now, but just wondering :).)

Alan

I have a Onetel account
3p per minute to the US and Canada.

Normal BT rate to London is 3.5p daytime.

It makes dealing with suppliers and customers in the US a lot easier when the calls are so cheap.

I even phone the owners of stolen credit cards!

Gordon

Ah, yes, One.Tel was the first company I thought of when you mentioned finding a phone company in your other post :). They provide really good prices.

Alan

one.tel in australia collapsed ;)

Its not the same company.
Onetel.net.uk is part of Centrica PLC

one.tel in australia was imo the best phone company. They only collapsed because their billing system was fubared and didn't bill for most phone calls.

they are back and rebranded as teleone