For the past several weeks I've been getting messages every morning that there is a memory problem:

From: <admin>
To: sys
Date: 3/27/02 4:30AM
Subject: The Sun Cobalt server is very low on memory

Memory on the Sun Cobalt server is heavily used.
The Sun Cobalt server needs more memory than it currently has.
Consider adding more DRAM to the server.

Total memory is: 389092 KB
Used memory is: 380488 KB
Free memory is: 8604 KB
Percent used is: 97

Then the past three days I have recieved this email:

Over the past fifteen minutes, the CPU has been heavily loaded.

This will result in noticible performace loss. Consider moving some of the
services to other Sun Cobalt servers, or reduce the complexity of the CGI
scripts running on the Sun Cobalt server itself.

1 minute load average: 3.97
5 minute load average: 7.73
15 minute load average: 6.94


And the past two mornings MySQL has crashed, and I've had to reboot.

What could be causing this?? How can I fix it??

Help!!

I don't know how we can tell you to fix this, he have no clue how loaded your server is.

How many sites are on your server, what is their traffic, how much content do they have, a lot of databases running, large scripts running, crons?

At that time your RAQ is processing the log files.

Specifically it's "analog" which is screwing up your RAQ.
It's trying to use 300+ Mb. of RAM, and that doesn't go
well as you might guess. The high CPU usage is simply
all the swapping your RAQ has to perform.

If you are using Webalizer or other program to analyze
your web stats, you might want to disable analog (see
some of the scripts in /etc/cron.daily/ )

As 4:00 AM is 12:00 AM local time for me, I've tried to
simply kill (not with -9) the analog processes, and this
didn't cause any problems for the operation of the RAQ.
(but rather, kept it from crawling to a halt)

Originally posted by ljprevo
I don't know how we can tell you to fix this, he have no clue how loaded your server is.

How many sites are on your server, what is their traffic, how much content do they have, a lot of databases running, large scripts running, crons?

I have about 5 sites on the Raq, but only one generates any traffic. That site gets about 25,000 page views per day. And has vbulletin and other custom mysql applications.

What is analog?? Do I need to disable webalizer?

analog is a apache log analyzer similar to webalizer.

It generates the traffic statistics you can see in the control panel.

It (should) run completely seperate from webalizer, so if you
kill or disable it, webalizer stats should NOT be affected (at least
they haven't mine).

I have just checked though, and ehm, I'm a little confused myself
how to permanently stop analog from running (which at the same
time will kill your statistics from the control panel as well, but
they weren't very useful anyway).

I hope another member can help in that respect (disabling analog).

I'm not sure if your timezone permits you to login with SSH
during the critical time (just after 4:00 AM server time), and
you can see for yourself with "top" that analog is hogging
your machine...

Can someone else help?

found this at rackshack

http://forum.rackshack.net/showthread.php?s=&threadid=3828

Sorry, I just replied to the wrong thread.

I still need help with this problem!

Help!

It crashed again this morning.

Can someone help????

It is 8:30, and analog is still running.

Where can I change the time for that?

Originally posted by pgowder
It is 8:30, and analog is still running.

Where can I change the time for that?
I don't know anything about analog, but if it's scheduled it might be in the cron. Have a look in the crontables and see what you can find.
Of course I could be completely wrong - so don't take my word as gospel.

Just watch your server throw the "top" command in the shell. And see what is using almost / ALL of the memory. When it is a script disable it and see what happens.

If that doesn't help just add some more memory :-)


But I think it is a script problem (a corrupt script or something).

This morning analog was taking the most memory.

Looks like analog is part of logrotate. So I've moved the daily crontab to run at 2 instead of 4. Hopefully that will give more time for everything to finish before I have users banging on the site.

If it is analog, I would say that it is a log problem (your logs are to big) or you are running to many sites on 1 raq.

So how many sites are you running on it? And what kind of a raq is it?


And if it is a log problem. Try to rotate the logs more then once a day, for example every hour, so that the logs are very very small. And analog/logrotate can handle them easily.

About 5 sites. Only one gets any traffic to speak of.

Raq4.

How can I rotate more than once a day?

I would suggest you to use webalizer for the stats. the pkg could be found at pkgmaster.com and disable analog . Run the following commands step-by-step to disable analog

To disable Analog :

ssh/telnet to your raq
su -
mv /usr/local/sbin/analog /usr/local/sbin/analog.disabled
echo '#!/bin/sh' > /usr/local/sbin/analog
echo "exit 0" >> /usr/local/sbin/analog
chmod +x /usr/local/sbin/analog

To enable Analog again :

cp /usr/local/sbin/analog.disabled /usr/local/sbin/analog
----------------------------------------
originally posted at rackshack

Originally posted by pgowder
Looks like analog is part of logrotate. So I've moved the daily crontab to run at 2 instead of 4. Hopefully that will give more time for everything to finish before I have users banging on the site.
I had the same problem (heavy loads in the morning, MySQL crashing) and this is exactly what I did (well almost exactly, I moved it up to 1 instead of 2). Fixed the problem.

If you ask em that isn't a REAL solution it is just a temporary one. Because when the logs get even bigger the problem will return.

So as I stated above I would run logrotate more then just once a day. SO that it can easily process ALL log's without a problem. Suc6.

How would I have it run twice a day?

Just add a cron-job that activates the file for logrotate in /etc/cron.daily/

I've disabled analog and set the logs to rotate twice a day.

Now, one of my sites has a web.log file that is 0. Will that start back after the next rotation?

I have had the same memory problem, Your server is pretty much leaking memory from processes that are not closing.. You can tell this if you telnet/SSH in and do a ps auxf at the command line. You will see a list of processes running and their ID's If you see ones with the same ID they are not closing properly.

You might want to call Cobalt Technical Support. Ive been dealing with them on this issue. It all started after they released the last Glibc update. Its broke the servers connection with ChillyASP, they still have not fixed the problem and my machine that was working fine for 2 monthes with minimul load and only 50GB's of traffic a month now dies just about everyday.

Im hoping that they will come up with fix soon. but the best thing to do is use TOP and watch the memory and reboot when its getting low.

A command line reboot is much better than a hard reboot and the server comes backup within minutes.

:D

I definately have the glibc problem. Does anyone have a fix?
Matthew

My company has done extensive testing along with working with some lead technicians at SUN/Cobalt.

The problme seems to be caused by ChillySOFT ASP, so if you have alot of clients using ASP then your server will crash more often. If your not using it and its just running on your server it might takes hours or days for your server to crash. ChillASP is a divison of SUN/Cobalt and I Guess they did not talk when they released the the latest Glibc paches cause it messes up alot of things having to do with ChillASP, some people have been able to upgrade to the latest version of ChillASP and have it fix the problem, but if you were already at the latest version and then you loaded the Glibc update then your going to be having alot of problems. I suggest that you contact SUN/Cobalt support, but all they are going to tell you is that they are working on a fix, they pretty much have to go through the code line by line and try and figure out what is going wrong.


Glad that I could help.