Well it seems Nocster/BurstNET had more down time today, several people I've been chatting with couldn't access my server or nocster.com...


It's finally back up, and made sure there was no network notice sent of downtime... And now I'm posting this poll....

With all the network issues lately, do you think it'd be better for them to just fire the entire network department and hire new people??

Erk and it's back down again. :angry:

Yup... Welcome to the nocster rollercoaster!

Oh my...:bawling:
Fri Jul 19 00:46:04 EDT 2002 down again..

DDOS attack or hardware problem?

I called and I quote : Were being Flooded is what the tech said :bawling:

I know you can't expect much for 120/month, but hell.. I expect it to stay online atleast....

Originally posted by sodapopinski
Oh my...:bawling:
Fri Jul 19 00:46:04 EDT 2002 down again..

DDOS attack or hardware problem?

No, dog got into noc, and urinated on power socket :D

Meh. Just got off the phone with them, they're getting flooded. At least it's not their fault...

It's all the IRC people they pissed off by not allowing IRC services anymore ;) Hehehehe.

*edit*
ill save my breath for this one....

Originally posted by The Prohacker
I know you can't expect much for 120/month, but hell.. I expect it to stay online atleast....

who pays that ? I pay way more than that ;)

Hire some chimpanzee's they do better than monkeys ;)

Originally posted by The Prohacker
I know you can't expect much for 120/month, but hell.. I expect it to stay online atleast....

I'm paying 4X120 to Burst, get 3 times less bandwidth, plus server is down as much as Nocster usually same time of the day. Burst said we had "quality bandwidth".... :rolleyes:

I guess there is not much difference between the two, besides the price.

Originally posted by Myacen


No, dog got into noc, and urinated on power socket :D

Smells like extra crispy chicken? :eek: :laugh:

Originally posted by porcupine


Smells like extra crispy chicken? :eek: :laugh:

Thanks Myles for providing some comic relief in these tense momments :stickout

Hmm... looks like it's only going to get worse before it gets better... I was getting about 50% packet loss before... and now:

Ping statistics for 64.191.10.132:
Packets: Sent = 68, Received = 0, Lost = 68 (100% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

ProHacker: move

:angel:

Originally posted by phantasywork


Thanks Myles for providing some comic relief in these tense momments :stickout

:p Worse yet, i found a massive CPanel/WHM bug (found isn't really the right word for it, more like "pimpslapped by this bug"), and i'm trying to check any status on it and scan around bugtraq more on CPanel.net and can't do anything :bawling:. It crashed our reseller server 2x so far and theres nothing i can do about it that wouldn't completely remove functionality from our WHM panel :bawling:

Originally posted by porcupine


:p Worse yet, i found a massive CPanel/WHM bug (found isn't really the right word for it, more like "pimpslapped by this bug"), and i'm trying to check any status on it and scan around bugtraq more on CPanel.net and can't do anything :bawling:. It crashed our reseller server 2x so far and theres nothing i can do about it that wouldn't completely remove functionality from our WHM panel :bawling:


Yeah, I remember that :D
What a day for downtime I've had.. JK ;)

ohh look.. our server is back up....

I wonder for how long :D Get the stopwatch's out boys and girls...

Originally posted by phantasywork
I called and I quote : Were being Flooded is what the tech said :bawling:

Just talked with a friend of mine through ICQ. He laughed at me once he knew I put one of my server into their network.
The roumor said that burst is being targeted by alof of flooders since they host azzam site and refuse to turn of the box.
(I did not know who's azzam, I remember someone post a message here - but deleted by the WHT mod- he said that azzam is one of the alqaeda afiliate).

hoaaaaaaaa...:bawling:
if it's true..what to do? consider to move to another datacenter?

Just me, or is the packet loss gone? :fingerscrossed:

yea.....

*expects this again in a couple of days*

Originally posted by porcupine


:p Worse yet, i found a massive CPanel/WHM bug (found isn't really the right word for it, more like "pimpslapped by this bug"), and i'm trying to check any status on it and scan around bugtraq more on CPanel.net and can't do anything :bawling:. It crashed our reseller server 2x so far and theres nothing i can do about it that wouldn't completely remove functionality from our WHM panel :bawling:

what's the 'massive bug'
I'm curious?
perhaps I can provide some insight or just be flabbergasted as well.........

staring on intently.......

Originally posted by isildur


what's the 'massive bug'
I'm curious?
perhaps I can provide some insight or just be flabbergasted as well.........

staring on intently.......


When a reseller deletes a user, it looped and crashed the server....

Big pisser :D

What version is the bug on?

Originally posted by isildur


what's the 'massive bug'
I'm curious?
perhaps I can provide some insight or just be flabbergasted as well.........

staring on intently.......

Maybe posting it will get something done about it, i've shouted around, but noone seems to have any ideas, and i tried to report it to CPanel but am still waiting on my bugzilla login, im guessing they're a lot busier with other stuff right now also :(, here's a snippet from my server logfiles, doesen't take a genius to figure out whats happening, this is the actual log, unaltered:

<log start>
Jul 18 08:18:21 www2 sshd[16133]: Did not receive identification string from 66.
28.75.234
Jul 18 08:18:22 www2 usermod[16158]: change user `psem' GID from `683' to `679'
Jul 18 08:18:22 www2 usermod[16158]: delete `psem' from group `psem'
Jul 18 08:18:37 www2 last message repeated 163136 times
Jul 18 08:18:37 www2 grpck[16156]: cannot lock /etc/group
Jul 18 08:18:37 www2 usermod[16158]: delete `psem' from group `psem'
Jul 18 08:19:08 www2 last message repeated 465339 times
Jul 18 08:20:09 www2 last message repeated 937867 times
Jul 18 08:20:22 www2 last message repeated 165463 times
Jul 18 08:20:22 www2 sshd[16627]: Did not receive identification string from 66.
28.75.234
Jul 18 08:20:22 www2 usermod[16158]: delete `psem' from group `psem'
Jul 18 08:20:52 www2 last message repeated 355662 times
Jul 18 08:21:54 www2 last message repeated 610202 times
Jul 18 08:22:21 www2 last message repeated 440290 times
Jul 18 08:22:21 www2 sshd[16847]: Did not receive identification string from 66.
28.75.234
Jul 18 08:22:21 www2 usermod[16158]: delete `psem' from group `psem'
Jul 18 08:22:52 www2 last message repeated 468673 times
Jul 18 08:23:53 www2 last message repeated 527463 times
Jul 18 08:24:23 www2 last message repeated 292262 times
Jul 18 08:24:23 www2 sshd[17107]: Did not receive identification string from 66.
28.75.234
Jul 18 08:24:23 www2 usermod[16158]: delete `psem' from group `psem'
Jul 18 08:24:54 www2 last message repeated 311096 times
Jul 18 08:25:55 www2 last message repeated 576501 times
Jul 18 08:26:28 www2 last message repeated 336298 times
Jul 18 08:26:28 www2 sshd[17408]: Did not receive identification string from 66.
28.75.234
Jul 18 08:26:28 www2 usermod[16158]: delete `psem' from group `psem'
Jul 18 08:26:58 www2 last message repeated 422106 times
Jul 18 08:27:59 www2 last message repeated 658797 times
Jul 18 08:28:21 www2 last message repeated 119330 times
Jul 18 08:28:21 www2 sshd[17692]: Did not receive identification string from 66.
28.75.234
Jul 18 08:28:22 www2 usermod[16158]: delete `psem' from group `psem'
Jul 18 08:28:52 www2 last message repeated 303999 times
Jul 18 08:29:21 www2 last message repeated 365861 times
Jul 18 08:29:21 www2 proftpd[17865]: 66.250.86.16 (host213-1-137-90.in-addr.btop
enworld.com[213.1.137.90]) - USER ibblesof: Login successful.
Jul 18 08:29:21 www2 usermod[16158]: delete `psem' from group `psem'
Jul 18 08:29:52 www2 last message repeated 250621 times
Jul 18 08:30:26 www2 last message repeated 326917 times
Jul 18 08:30:26 www2 sshd[17994]: Did not receive identification string from 66.
28.75.234
Jul 18 08:30:26 www2 usermod[16158]: delete `psem' from group `psem'
Jul 18 08:30:57 www2 last message repeated 413474 times
Jul 18 08:31:58 www2 last message repeated 568900 times
Jul 18 08:32:21 www2 last message repeated 232569 times
Jul 18 08:32:21 www2 sshd[18240]: Did not receive identification string from 66.
28.75.234
Jul 18 08:32:21 www2 usermod[16158]: delete `psem' from group `psem'
Jul 18 08:33:05 www2 last message repeated 162828 times
Jul 18 08:34:06 www2 last message repeated 761272 times
Jul 18 08:34:58 www2 last message repeated 244463 times
Jul 18 08:34:58 www2 sshd[18642]: Did not receive identification string from 66.
28.75.234
Jul 18 08:34:58 www2 usermod[16158]: delete `psem' from group `psem'
Jul 18 08:35:30 www2 last message repeated 314375 times
Jul 18 08:36:23 www2 last message repeated 605220 times
Jul 18 08:36:25 www2 sshd[18896]: Did not receive identification string from 66.
28.75.234
Jul 18 08:36:23 www2 usermod[16158]: delete `psem' from group `psem'
Jul 18 08:37:01 www2 last message repeated 44298 times
</log stop>

The "did not receive identification string from ..." errors are our network monitoring tools running... after this, the log just stops and resumes just over an hour later when get back to my pc and notice whats going on.

Our servers are pretty hefty, but i mean grab a calculator and lok at that... a few million commands per minute was more then our poor little box could handle, it's not invincible :bawling:.

I though perhaps we could limit cpu/mem usage, max procs, anything, but if you look at it, usermod runs as root, its not running thousands of concurrent processes, and not using tons of cpu time, just millions of tiny ones that take 1/100 of a cpu second.

If anyone has suggestions, or has run into something similar, let me know, i'd love to hear :).

Originally posted by Myacen
What version is the bug on?

Latest, might be in more then one, this server was running fine, something like 45 day uptime before it happened the first time. First time i assumed it was apache going nuts, but i retract that assumption after taking time to properly snoop the problem when it hit the second time.

I believe we're runnign 4.9.??? now :)

4.9.23 However it seems to be running fine for us?

keep your eyes on it :), it only seemed to happen when a reseller deleted a user, and it didn't happen every time obviously, i'm not sure exactly what triggered it, but theres no denying what happened.

Easy to find though, "cat /var/log/secure | grep repeated | more", if anything turns up with that with more repeats then 10,000 you probably stumbled on the same thing :(.

For any mission critical system of this linux-ish webhosting genre ..... you should definitely definitely get a process monitoring and control system in place. code something yourself, or grab something from somewhere, and tune the sucker to save you from catastrophes.

If you find the right fit, you'll wonder why you never had anything like it before.

But with a root procedure like that looping and looping at an insane pace I'm not sure if anything could save you....
Nasty.

I haven't seen it duplicated within our network but I'm sure it wil rear its ugly head soon if it's legit.

http://www.enterstageright.com/archive/articles/0502/0502ispterrorist.htm


When I e-mailed burst.net asking why the company hosted such a site, here is the response I got from a Sean Rosler, who identified himself as being with "System Administration."

"BurstNET DOES NOT voluntarily host the azzam.com website. It has either been removed from our equipment (It hasn't) or has been left untouched by gov't official request. (My emphasis). It was formerly hosted here, was under government investigation, and was/will be removed as soon as the govt allowed us to do such. It would have been removed the instant we found out about it, had we not been instructed by the ‘powers that be' to leave it untouched ... We are a Jewish owned corporation, do you really think we want to host such crap?"

Originally posted by isildur
For any mission critical system of this linux-ish webhosting genre ..... you should definitely definitely get a process monitoring and control system in place. code something yourself, or grab something from somewhere, and tune the sucker to save you from catastrophes.

If you find the right fit, you'll wonder why you never had anything like it before.

But with a root procedure like that looping and looping at an insane pace I'm not sure if anything could save you....
Nasty.

I haven't seen it duplicated within our network but I'm sure it wil rear its ugly head soon if it's legit.

Yep, we've got security limits in place now, and some uh, overly restrictive measures that might slow it down to just lagging the server instead of killing it should it hit again, but not much you can do to stop it (at least that i can think of). And deleting usermod isn't an option as WHM would essentially be diasbled (no adding, editing, or removing users).

The bugs been around for as long as I can remember, it's nothing new :rolleyes: Basically 'userdel' just eats up the resources as its looped, and brings the server down. Last time I mentioned it to Nick, he didn't seem too interested.

I think the NO's have it! :D

azzam.com is actually now at Rackshack and not Burst

azzam.com is a VERY COMPLICATED issue. That's about all I "can" say.

Robert

Robert, sorry I did not mean that as a put down. I just wanted to point out that the site is not at Burst.

well it's load balanced between two sites.
do a dig www.azzam.com

While this isn't relavent to this thread, however someone posted it here anywyas so I'll respond:

[bdraco@localhost bdraco]$ rpm -qf /usr/sbin/userdel
shadow-utils-20000902-7
[bdraco@localhost bdraco]$ rpm -qf /usr/sbin/usermod
shadow-utils-20000902-7

userdel and usermod are both system utils that come with redhat. There is a problem there for sure, but it appears to be random at best. This does not just affect people using cPanel, it will affect any redhat system using these utils. We are currently working on a replacement for most of the utils in the shadow-utils package.

Originally posted by RackMy.com
azzam.com is actually now at Rackshack and not Burst

Actually it is at Burst. Go to the site and check out the links. They all point to 66.197.135.110, and that is a Burst IP.

This is the first I have heard about Burst hosting this site, but I will be looking for servers from another company because of it.

Originally posted by Brewer


Actually it is at Burst. Go to the site and check out the links. They all point to 66.197.135.110, and that is a Burst IP.

This is the first I have heard about Burst hosting this site, but I will be looking for servers from another company because of it.

Look now ..it's Gone :eek:

Looks like Burst is reading the forums. :D

Originally posted by Brewer
Looks like Burst is reading the forums. :D

Yeah, who do you think the 3 people that voted no were??? :D

hmmm... well Robert isn't denying rackshack is hosting it, unless both are hosting portions of the site. It does bring up a definite issue though for hosts. The government seems to want burst to host the site to track who visits it, but by hosting it, you make yourself a target for DoSS attacks. Lose Lose situation and a contingency hosting co's probably have to think about.

Originally posted by The Prohacker


Yeah, who do you think the 3 people that voted no were??? :D

Hehehe :D

WOW - it's GONE!! - Looks like RACKSHACK or BURST felt some pressure and finally was able to disable the website...

HIP HIP HORRAY!!!

p.s. that was pretty quick... they changed the links and removed it... ;)

wonder where it will pop up next...

What was the site about?

Originally posted by popdirt
hmmm... well Robert isn't denying rackshack is hosting it, unless both are hosting portions of the site. It does bring up a definite issue though for hosts. The government seems to want burst to host the site to track who visits it, but by hosting it, you make yourself a target for DoSS attacks. Lose Lose situation and a contingency hosting co's probably have to think about.

If the government would want to monitor the site do you really think they would announce it to the whole world? It's too late now.

G'damn terrorist websites!

http://web.archive.org/web/*/http://azzam.com

Good move by either Burst of RS to remove the site. Thats one the smart play by either company.

There are still 1000's of terrorist sites left. http://alneda.com/ being one of them. :rolleyes:

Next one: http://qassam.net

Kiwi...Good link. Ive never seen that scroll bar trick on the left side of the browser before... interesting.

Looks like qassam is at ev1.net and the other at thewetlandsinc.com.

Maybe a live chat with a terrorist? http://forum.qassam.net/

Sickening. :mad:

Well, Burst can get a special deal on chimpanzee's as i said before, 2$ an hour, even though its $1 an hour more than the monkeys :eek:

lol I think i'll post on qassam, since the forum is in my language - thanks for the link lol..

this is going OT with the thread about these sites, but if you want to translate them, I found a tool:
http://tarjim.ajeeb.com/ajeeb/default.asp?lang=1

I can translate it for you, afterall i have been speaking arabic since i was born lol

I'll vote for yes.

They NEED to get rid of Keith. This guy has no idea what is he talking and doing. Always sign off AIM when he can't solve the problem. I wonder why they let him sign on under Nocster AIM support nick.

Darren was good thow.

ALso, it been on and off more frequently lately. turn our RS still better. lolz

*sigh* and the cpanel userdel bug strikes again, right under my nose this time (was logged in) but it spooled so fast, i typed "w" to see the load averages (user complained that they deleted a user and could still login as the user), and that was it, too little too late, there goes my what? 2 day uptime now?! :bawling:

BTW, i tried that rpm stuff you said, but got a -4 instead of -7.

Myles you got an update on it?

At least the network has been up for a few days now.

hmm...they aren't too far from me, they could hire me if they want. Of course that would not be fair to all of the other hosts on this board :D. I'd hate to think what would happen if I had their network running so smoothly that there were never any problems :).

Originally posted by Myacen
Myles you got an update on it?

It hasn't happened since,

I cronned two scripts though

*/10 * * * * /usr/bin/killall -9 usermod >>/root/userbug.log 2>&1
*/10 * * * * /usr/bin/killall -9 userdel >>/root/userbug.log 2>&1

every 10 minutes, haven't had any reported sucess with them yet, so i can only assume my resellers haven't been deleting as much, or CPanel fixed it :)

I found my servers needed at least 15-25 minutes to die under the userdel/usermod bug crashes (by checking the logfile) so i figured 10 minutes would give the server 1-2 minutes of nearly complete unresponsiveness, then return to normal, if it works or not, cant say, hasn't connected yet.

If you're having trouble with that usermod/userdel problem, try that though :)

Originally posted by uuallan
hmm...they aren't too far from me, they could hire me if they want. Of course that would not be fair to all of the other hosts on this board :D. I'd hate to think what would happen if I had their network running so smoothly that there were never any problems :).



hahahha... Now that would be a clash of personalities....


Allan vs. Sean.... :D

Originally posted by The Prohacker

hahahha... Now that would be a clash of personalities....


Allan vs. Sean.... :D

I don't think I have ever had problems with Sean before, why do you think our personalities would clash?