Hey guys,

Lately our site has been getting overloaded and I noticed some swap was used... not sure when, but that can't be a good thing, right?

Here's the numbers from top: 12:00am up 211 days, 3:35, 1 user, load average: 3.59, 3.96, 4.17
374 processes: 372 sleeping, 2 running, 0 zombie, 0 stopped
CPU0 states: 82.0% user, 17.1% system, 0.0% nice, 0.0% idle
CPU1 states: 72.1% user, 27.0% system, 0.0% nice, 0.0% idle
CPU2 states: 65.3% user, 34.1% system, 0.0% nice, 0.0% idle
CPU3 states: 57.1% user, 42.0% system, 0.0% nice, 0.0% idle
Mem: 1030560K av, 1011368K used, 19192K free, 0K shrd, 14088K buff
Swap: 2048276K av, 248984K used, 1799292K free 593312K cached

The server is a Dual Dell P4 2.0 XEON w/ 1GB Ram and 2 SCSI HD. I was thinking it may be time for a reboot as a temp fix and then maybe add another 512-1GB of ram?

What do you guys think?

Mark

When was the last time you updated your kernel? :D

Just curious... which platform/OS?
If it is linux, by looking at your uptime... did you upgrade your kernel to fix the latest known exploit?

<edit>null, you beat me by one minute :mad: :D</edit>

Nope, haven't upgraded the kernel since it's been up. What's involved in a kernel upgrade? Is there a risk of damaging the OS?

Mark

Are you using linux? You should upgrade your kernel if you are using linux to fix all the known kernel exploits before anyone able to exploit your system. If you have no idea about keeping your system up2date, kindly please consider to pay someone who knows or ask your provider to do it for you.

Yeah, we're using Linux. The box is from Ev1Servers.net. I'd like to upgrade, but is this something that is easy or should I have someone with a lot of experience do this? I don't want to risk losing the data on the site. We have backups, but it would be a hassle to have to reinstall everything again if I mess up something.

Mark

Originally posted by LancerForums
Nope, haven't upgraded the kernel since it's been up. What's involved in a kernel upgrade? Is there a risk of damaging the OS?

Mark

Compiling a kernel from source ( which would probably be the best move for you imho ) can be quite a task and I wouldn't advise you doing this on a live server if its your first time. I'd look into hiring someone that can do this for you.

I recommend you contact Ryan @ http://www.rfxnetworks.com He knows his stuff and does a great job!

Seems that you really need a Kernel upgrade. Your server load is showing the sign of our server before we upgraded our kernel.

Try upgrading Kernel using source. It is the best way. If you need help in doing that, I am able to recommend you a good server admin.

Thanks.

Originally posted by LancerForums
The server is a Dual Dell P4 2.0 XEON w/ 1GB Ram and 2 SCSI HD. I was thinking it may be time for a reboot as a temp fix and then maybe add another 512-1GB of ram?

What do you guys think?

Mark

A reboot might help things for awhile, what are the top 10 processes in TOP that is using so much CPU?

That would help. Also regarding kernel upgrades, email EV1 and tell them you want them to upgrade the kernel for you and just pay them a fee. I'm sure they can do that for you.

I thought EV1/Rackshack was completely unmanaged?

Hence, the term "pay them a fee". :)

That is crazy :eek:

It seems like mysql is taking up the most memory and processor power. The is pretty much 100% db driven, so I guess it's worse than most standard forums.

I'll check with EV1 to see if they offer services to upgrade the kernel.

Thanks,

Mark

Personally I would recommend you to get your provider to do it for you if possible as anything wrong they will be able to help you since they have physical access to your server... IMO

You might also need to optimize MySQL and apache.

MN-Robert, what you mean "That is crazy" ?

Not updating your kernel in 200+ days, their virtually been a kernel upgrade a month for the last 6 months.

Originally posted by MN-Robert
Not updating your kernel in 200+ days, their virtually been a kernel upgrade a month for the last 6 months.
ya... and my guess will be the system also not up2date if I am not wrong :angel:

Originally posted by LancerForums
It seems like mysql is taking up the most memory and processor power. The is pretty much 100% db driven, so I guess it's worse than most standard forums.


Have you had a go at optimising mySQL? If you haven't that would be the next step to take.

Visit forums.ev1servers.net and search for mysql optimisations or optimizations.

or paste your /etc/my.cnf

Mysql should be well optimized. I had some people on vBulletin help out with that.

If the kernel hasn't been upgraded for a while, will that make it harder to upgrade to current spec?

Mark

Originally posted by LancerForums
If the kernel hasn't been upgraded for a while, will that make it harder to upgrade to current spec?

Mark

Not at all. BUt make sure you try and get the techs at EV1 to do it for you.

EV1 doesn't update kernels AFAIK. And if that is a dual xeon be carefull about updating the kernel. You could end up with an un-bootable machine. That particular hardware setup needs someone experienced to update the kernel. See here: http://forum.rackshack.net/showthread.php?s=&threadid=23528

Originally posted by net-trend
Hence, the term "pay them a fee". :)

As far I know and as page-zone has said, they don't update kernels, and I've updated a kernel before and came up with an upbootable machine but since I have a managed server they had it back up and running in under 5 minutes.

If this is a redhat server, you most definitely need to update the kernel. Once that's done, I recommend going through your server with a fine toothed comb, checking for rootkits and signs of hacks and cracks. If it's been 200 days, you're most definitely under high risk of being hacked.

As far as the EV1/Xeon thing, that's not as much an issue any more if you know what you're doing. I've updated countless servers that are like that, all you have to do is know what to do before you update. Traditionally, the problems are had with the scsi drivers, rather than IDE though.

It is entirely possible for your kernel to be causing this amount of problems though. Check it out, have it updated and checked out, and I'd bet you'll have no problems further

HTH

I'd like to upgrade the kernel, but I scared that something could go wrong and the server would go down. I have backups, but it would be a PITA to get all re-setup again.

I just bought 2 new Dell servers for colocation since that was the plan anyhow. I am just going to make sure to keep these up to date and supported throughout to prevent something like this from happening agian.

Thanks for all the help guys,

Mark

Originally posted by LancerForums
I'd like to upgrade the kernel, but I scared that something could go wrong and the server would go down. I have backups, but it would be a PITA to get all re-setup again.

I just bought 2 new Dell servers for colocation since that was the plan anyhow. I am just going to make sure to keep these up to date and supported throughout to prevent something like this from happening agian.

Thanks for all the help guys,

Mark
Updating kernels iss a tricky job, really. However, if you do it right, you don't lose any data (even if the kernel fails). Worst that can happen? DC has to go in and select another kernel from lilo. Like I said, if this is a RS server with raid drives, then find an admin to do it for you. Not just any admin, but a QUALIFIED admin. This can screw stuff up.

Personally, I'd be more worried about getting hacked than upgrading my kernel. Not upgrading kernels can be baaaaaaaad.

Just make sure you have another kernel installed and your ok, just gotta call up the dc and have them console into your box and manually select another kernel