Okay i just went to view my website right now, and see it was loading extremely slow.. then i went to check ServerMatrix's monitoring service, and seen everything multiple errors, and warnings about services being down, or timing out.

- Now i'm receiving emails that services are failing and restarting, and i also can't log into ftp, nor cpanel.

Also got a high load warning from the monitoring system saying:

5-min load currently 73.75

What do i do????

Greetings:

Well, you could log into the server to check the logs, who is logged in, what's running, who's connected to what IP's and ports et all.

Thank you.

Actually i just called servermatrix, because i was unable to log in through shell either :(

If you're having problems once the server comes back up, feel free to contact me (AIM: SonataWeb, MSN: Support@SonataWeb.Net). I'm bored right now with nothing to do. So, I'll take a look at it and suggest some things as to what could be wrong.

Also received the following emails:

System integrity monitor on server1.timichost.net has taken action in responce to an event. Recent event logs are enclosed below for your inspection. There has been 1 events today, if an average of 8 events is reached, e-mail alerts will be terminated for the duration of the day.

- Events Summary:
Total event count: 1
Average event count: 0

- Service Summary:
HTTP [online - 0 events]
DNS [online - 0 events]
SSH [online - 0 events]
MYSQL [online - 0 events]
XINET [restarted - 1 events]

- System Summary:
LOAD [20.21 - status good - 0 events]
NETWORK [eth0 - online - 0 events]

- SIM Log:
[04/06/04 19:10:02]: NETWORK is online.
[04/06/04 19:10:02]: HTTP service is online.
[04/06/04 19:10:02]: DNS service is online.
[04/06/04 19:10:02]: SSH service is online.
[04/06/04 19:10:02]: MYSQL service is online.
[04/06/04 19:10:02]: XINET service is online.
[04/06/04 19:15:00]: LOAD 20.21 (status good)
[04/06/04 19:15:00]: NETWORK is online.
[04/06/04 19:15:00]: HTTP service is online.
[04/06/04 19:15:00]: DNS service is online.
[04/06/04 19:15:00]: SSH service is online.
[04/06/04 19:20:01]: removed stale lock file.
[04/06/04 19:15:00]: MYSQL service is online.
[04/06/04 19:15:00]: XINET service is offline.
[04/06/04 19:15:00]: Restarted XINET service (1 XINET events today).

=========

- Events Summary:
Total event count: 5
Average event count: 1

- Service Summary:
DNS [down, restart disabled - 1 events]
SSH [down, restart disabled - 1 events]
XINET [down, restart disabled - 2 events]

- System Summary:
LOAD [29.83 - status warning - 1 events]
NETWORK [eth0 - online - 0 events]

- SIM Log:
[04/06/04 19:15:00]: DNS service is online.
[04/06/04 19:15:00]: SSH service is online.
[04/06/04 19:20:01]: removed stale lock file.
[04/06/04 19:15:00]: MYSQL service is online.
[04/06/04 19:15:00]: XINET service is offline.
[04/06/04 19:15:00]: Restarted XINET service (1 XINET events today).
[04/06/04 19:20:01]: LOAD 29.83 (status warning)
[04/06/04 19:20:01]: load status warning, non-essential services going down.
[04/06/04 19:20:01]: NETWORK is online.
[04/06/04 19:20:01]: DNS service is offline.
[04/06/04 19:20:01]: DNS down, restart disabled via conf.sim.
[04/06/04 19:20:01]: SSH service is offline.
[04/06/04 19:20:01]: SSH down, restart disabled via conf.sim.
[04/06/04 19:20:01]: XINET service is offline.
[04/06/04 19:20:01]: XINET down, restart disabled via conf.sim.

You must learn to configure your SIM...

You have SIM shutdown so called "non-essential" services which includes SSH and preventing it from restarting. Of course you can't login.... You need the datacenter to reboot it.

Learn to understand scripts you use.

Well i had a server management company go in and install that for me.

Well SM restarted the server but they are saying something is eating up all the resources, but not sure what it is yet. SSH logins are completly slow as well as things such as cpanel boot ups.

what server management company, they didn't set it up right. Does this server have RHE ?

well, if cpanel wasent shut down you can start ssh from cpanel

Yes, you can indeed. However your server management company obviously didn't know what they are doing... on such extreme high loads, WHM may not load.

In any case, it is best to get your datacenter to debug it on terminal. It could be a case of bad hardware. We used to have this issue on one of the servers and it mystified us till we swap out the ram and it resolve the issue.

However without more details, we can't really advice you further.

Thank you for all your replies. SM worked on my server and everything seems to be working properly now.. however i won't know until the server is up for up for awhile, just to make sure the problem doesn't quickly return.

Basically Something was completly eating up my resources, and i was unable to log into cpanel, shell, ftp, or anything. When i was able to log into shell, everything went very slow and me as well as the SM tech, got disconnected a few times because the server was restarting itself..saying the Load was too high.

SM even rebooted the server, but it was loading slow and Cpanel crawled while it was loading, so even at bootup everything was extremely slow and the server load remained high.

Here is what SM said:

"The system was at a dead crawl. It seemed to be caused by some rogue apache process. I adjusted your httpd and all seems to be up and running ok. "

So sprintserve, you think it might be hardware related?


Hopefully the problem is gone, but i'm going to be keeping an eye on it.

Thank you to all who replied, i really appreciate it.

Your server was restarting itself also because of SIM. You should get someone to look into the configuration for you again. I will be glad to look at it if you wish to.

I am saying that usually if the problem seem mystifying, hardware can be an issue as we have encountered a few times ourselves. In fact we had 2 servers over the last two months that just behave slow for no reason, and in both cases, bad RAM was the cause.

If the problem returns, and the load is high when there's no apparent cause, the hardware angle may be worth a look. Rogue scripts shouldn't be running after a reboot either. So that reason is a bit weak.

Of course, another possibility is that you may be compromised... and someone is running some hidden process.

Yeah the rouge processes seemed strange to me too, considering a reboot should terminate those processes. I had SIM looked at, and has been reconfigured to prioritize things such as DNS, ftp, instead of disabling them. I've also had the server looked at but it doesn't seem to have been compromised, and i have tons of secruity on it, though it might be a possibility so i will look into it further.

I really appreciate everyone who took the time to reply, and help me out. Thank you.

damain, are you in the silver plan with servermatrix? or did they charge you extra for the changes they made?

Gold plan.

Can anyone advise as to how to make sure SSL ALWAYS restarts with httpd? Because of the problems above my http is constantly restarting, but its not enabling SSL when it restarts. I keep having to go into shell, and restart httpd with SSL manually.

Although I don't know off-hand how to have Apache use SSL on startup, with the following command:

service httpd stop
service httpd startssl

it should start SSL and only be required once -- until Apache does a self-restart. Even then though, SSL should work fine.

What is this 'System integrity monitor'? Something special from SM or is it an Open Source script?

Thanks for your reply. Thats what i've been doing though, but everytime httpd is restarted all my SSL based sites are disabled until i restart http with ssl from shell.

and SIM can be found here:
http://www.rfxnetworks.com/sim.php

Ok, that would bring up the question, "How many times is Apache being restarte -- and is it a self-restart or done manually"? And what would the reason be for either one?

Apache does need to self-restart in certain situations, but unless there are a lot of DNS changes being made (accounts Created or Terminated) Apache should not need a lot of restarts.

Check the log files for your HTTPS sites and make sure they are not huge, sometimes apache has a hard time opening large log files, and the way cpanel does SSL, on mine, is to create the log file in the /var/log directory.

This seemed to have brought down my server load some, and my services are not restarting anymore.

echo 2 10 20 > /proc/sys/vm/pagecache
echo 100 > /proc/sys/vm/inactive_clean_percent

I was using this before but forgot the configurations went back to default after a server reboot.

Thanks to everyone who replied, and tried to help me out :)

-Just Curious, has this problem been patched yet?