seems to be effecting only certain people as well. I've been connecting out from my desk the entire time, and posting through here as well, however, other people are unable to gain access to their servers. We will update as soon as we have any additional information.
Originally posted by Michael-MS I thought NAC had a newsletter/mail list that you could subscribe to and get network notices? I'm pretty sure I had subscribed myself to it a while ago and I didn't receive anything at all today.
I'd say this was an issue that came up immediently and action was needed right now etc. I'm sure some info will filter down from NAC as to what happened.
Though I'm unsure of NAC's mailing list process, as it may be an after the fact e-mail about network issues, when we see a problem, we try to post on it asap. At least it helps keep people in the loop as to any issues to the best of our knowledge. AFAIK, the issue has been resolved, and I'll await official word to the cause of things, then post it on up.
Network issues are the best (if that's possible) form of downtime. At least you don't have to replace server hardware (restore a drive, replace a PSU etc). We just sit back and wait for the good NAC techs to fix whatever needs fixing, all the while keeping our clients informed of the issue and that everything that can be done, is being done, to rectify things.
Things happen, equipment breaks, and they have always bought the network (or parts of it that was effected) back in good time. Helps to keep your main site + helpdesk in another datacenter than your client servers. Easier to keep the channels of communication open in the event of such network issues.
Thursday, May 20, 2004 at 2:02pm EDT Net Access Corporation began to experience network difficulties in Parsippany, NJ (node code: OCT). This outage affected dedicated access T1, T3, and Ethernet customers aggregated on gbr1.oct.nac.net.
By default we place a standard firewall filter on every interface. These filters help alleviate certain types of Denial of Service (DoS) attacks.
In addition to these proactive filters, customers who purchase tiered bandwidth services from NAC also have a "policer" applied to their interface to rate-shape their bandwidth.
In cooperation with an engineer at Juniper's TAC, we discovered a software bug in the version of JunOS that we are running on gbr1.oct.nac.net. In this version of JunOS when you have a firewall filter and a policer applied to an interface it inadvertently applies the policer to all interfaces that have the same firewall filter. This was the unfortunate culprit of today's outage.
We sincerely apologize for any problems this outage caused you. We will be conducting an emergency maintenance window this weekend to upgrade the JunOS on gbr1.oct.nac.net. Customers will receive a maintenance window notification shortly.
If you have any questions, please do not hesitate to contact our 24x7 Network Operations Center at +1.973.590.5050 or via email to [email protected].
Steven J. Schecter :: Just a spoon full of MEDs
<[email protected]> :: make the routes flow down
Sunday, May 23rd, 2004 Net Access Corporation will be performing emergency service affecting maintenance in Parsippany, NJ (node code: OCT). The maintenance window will commence at 7.00am UTC, 2.00am EDT.
We have identified a serious software bug in the version of JunOS running on our gbr1.oct.nac.net router. During this maintenance window we will be upgrading the JunOS. Since we will be going from a 5.x to a 6.x release, the router will require two (2) reloads. We expect the total downtime to be less than 15 minutes. Dedicated access T1, T3, and Ethernet customers aggregated off gbr1.oct.nac.net will be affected by this emergency maintenance. The maximum duration of this maintenance window is one (1) hour.
I have about 10 servers on DN/PW network and for the last 3 days I would say I personally could not reach every one of these servers at random times, and over the weekend have answered likely 100 emails reporting of the same, all at random geographic locations.
What is the deal?
Last edited by myusername; 05-30-2004 at 01:54 PM.