My host asserts that server load, currently running at 17.62, 12.84, 15.11 on a single processor will not affect the performance of CGI scripting. Load has been over 30 at times today.

This is what they say:
"The load average has no effect on CGI scripts acting incorrectly. If it did, you wouldnt be the only individual complaining about this."

My scripts are failing to finish execution, and producing blank pages. It is very annoying, especially because my banner advertising runs of CGI scripts and it is my only source of income.

Here is what PhPSysINfo says:
http://www.silverbulletcomicbooks.com/phpsysinfo/index.php

If you go to my main page and there is a big gap in the middle at the top it is a banner failing to load.

http://www.silverbulletcomicbooks.com/index.htm

Who's right here? My hosts or me?

Billy,

Those types of load averages will definitely affect your CGIs ability to finish (and to sometimes even start). They are either feeding you a line of bull to keep you on or they don't know what they are doing. Having found myself in both positions (a long time ago!), I know the litany well.

It's likely that they won't back down on their current conviction/line-of-BS, especially if its coming from a manager. If you haven't spoke to a manager, I recommend that you do so. If you have, you need to make the decision of whether or not you're willing to live with it or move your site someplace else.

I have written to the Chief Technical Officer and CEO of the company and directed them to this discussion as evidence that server load will cause CGI script failure.

I would appreciate it if anyone else can confirm this information will post here in support.

Thanks.

Your server load should not exceed 5.0, we do not let ours exceed 2.0 and they average 0.10

But will it affect the execution of CGI scripts, as I am led to believe?

Different OSes present their load averages differently. For instance, on BSDi anything over 2.5 is a bad thing. On the other hand, I have run a multiprocessory sparc station to well beyond 20 and the CGI are still cranking away. Redhat Linux, which is the most popularly used *nix at this time, does okay up to between 5 and 7 (depending on kernel tuning and process type) if you have a couple cpus installed.

The underlying factor regarding load averages reporting on various cpus is the OS's presentation. For instance, I would go so far as to say that the BSDi who has hit 2.5 is producing 50 times more than the linux box who has hit 5. I have also seen load averages get "stuck" in the reporting. A better indication of whats going on whould be netstat, iostat, and vmstat. Usually the cgis are more affected by what is going on in vmstat... that is the server is thrashing... that's when the CGIs get eaten. You do have to know your machines well to know what's going on. Sometimes, all things being the same... a different hardware component will affect the machines ability.

If the provider is use to seeing load averages at 15-17 without problems, maybe they should review the differences in vmstat, netstat, and iostat on machines that are running properly.

If you're CGI are behaving erratically, the CGI (as a child process) will be one of the first things to go.

What's that stuff you just posted mean, Valkaryn?

I have another question... in the PHPSysInfo readout what does the number of current users mean exactly?

Billy,

Sorry, I removed it. It had been meant for another discussion page. I seem to have dropped one of the apples I was juggling!

At any rate, it was an entry for a htaccess file to keep people from certain IPs from visiting site. They were giving the owner a Denial of Service (DOS) attack and he needed the config to keep it under control. (See Make My CGI Slow under General Discussion at this Forum).

Originally posted by billybatson
I have another question... in the PHPSysInfo readout what does the number of current users mean exactly?

I'm a little weak in PHP. I use perl straight up (and write my HTML in a text editor) because I can. BUT, it can be one of the following figures:

1: Remote User count based on unique usernames. The count of recently active usernames that are a result of being prompted for a username and password. So standard access would all be lumped under 1 user, the '-' user.


2: The count of all unique IPs recently connected to your virtual host.


3: The count of all unique IPs recently connected to the entire server.


There should be a help page that has some, at the very least, vague language that describes what the count is for. If you want to post that here, I can translate it for you.

I have a suspcion that it is 2 on your list... but the docs don't make it clear enough for my liking.

Here is the info page on the utility:
http://sourceforge.net/projects/phpsysinfo/

I didn't consider a fourth option since PHP is designed for use with a web service. After looking at the code, it turns out that it's actually polling the number of system users. That translates to one system user that runs your entire webserver (or virtual servers) ... usually the user www or nobody. It has nothing to with htaccess users or unique IPs connected to your website. Pretty much "fluff" information when it comes to joe average website manager. The system administrator would find it useful, but are more likely to be on the system to take a look at what a particular user is doing if they are interested in this factoid.

It looks like the programmer had access to the information easily when he was putting together other info, such as kernel, load average, etc. So he threw that in, too.

So that item is fairly irrelevant, then?

For your needs: ayup.

Here is the reply I got from the CEO of my host:

"There is no physical evidence linking your cgi execution problems with server load. It is also the truth that no other complaints were filed, and the server has many cgi users. The server has been at a load average of about 7 today. As you already know the server is a pretty stought configuration. We have a few other servers in similar configurations, except with dual 1000Mhz Intel chips, and have not found much of a performance increase or decrease in load. We are working on getting more HD space on that unit, and migrating a few high usage sites however. This should continue the stability and increase performance some."

Opinions?

At least they are doing something :)

They *say* they are doing something...

The load on the server has decreased, but I'm still getting CGI script failure intermittantly...

Any other possible causes?

Why don't you ask them to be moved to a different server? I'm not saying that will fix all your problems, but maybe it will rule one out at least?

Originally posted by billybatson
They *say* they are doing something...

The load on the server has decreased, but I'm still getting CGI script failure intermittantly...

Any other possible causes?


Billy,

It sounds like the right button has been pushed. Let's see if they follow up with what they have said. Give them a day or two to move the high traffic sites of that box. Most likely they had a porn site on the server that got out of hand. I have seen this happen on single server configurations. I have even written emails that resemble the one from the CEO. It's as close as confession as you'll get. Pat yourself on the back for getting results. The technical side of moving those high traffic sites off to another server will still have residual affects for a couple days (plus they have to make arrangements with the site owners). If things haven't cleared (or you can't wait that long), consider finding another host - but plan your move carefully so that you aren't affected adversely.

That kind of load is definatly pretty high.. an admin gets paged if any of our servers get to 1/3 of that. It does depend on the servers stats, but a decent dual 1ghz with alot of ram would even show some serious signs of slowing down at that speed.

-Brendan

Okay, I'm gonna give them 48 hours to get the situation sorted.

I have a new host that I am considering - but I would appreciate any suggestions for a company that can provide the following requirements:

Unix
600Mb+ Web Space
20 Gb + Bandwidth
SSI
Perl/CGI
PHP
mySQL
Custom Error Pages
Control Panel
Multiple Domains
Sub Domains
Unlimited Email Forwarding
Mailing Lists
Daily Back Up
Raw Access Logs
Telnet
Custom MIME Types
24/7 (Phone and E-mail, ICQ, AIM) Tech Support - IMPORTANT!!

And above all else, just plain honesty. I understand error and crashes occur, I just don't like being lied to about them.

Whats your AIM SN?

Load average under Linux, if I understand the code right, is the average amount of processes waiting to run over the period of time that the average represents.

A load average of 17.62 means that over that period of time, an average of 17.62 processes were unable to run due to a lack of resources. That's quite a few, really, for a 'real time' job like serving web pages. It probably wouldn't be as bad if you were just running batch-jobs.

Load average itself rarely has anything to do with a program crashing; usually, it just results in slowness. However, a high average may be another 'symptom' of the real root cause.

Quite a few things can cause high load average -- the most common being:
1. Not enough CPU power.
2. Running programs from swap constantly -- aka 'thrashing.'

If your host runs Linux, run 'free -m'. If you see _lots_ of swap space used, that's not too good. Run 'vmstat' a bunch of times -- if the blocks in and out are always very high (500-1000+), it's probably thrashing. Thrashing is just about the WORST thing that can be happening, and can cause programs to crash; if it happens long enough, it'll tear up the drive, too :)

If you have access to your error logs, I'd check those, were I you. You might also try running your CGI's from the command line, and see how they exit. That'll give you an even better idea.

A server running crappy IDE disks, slow processor, or small amounts of memory will experience high loads faster.

-Brendan

As someone mentioned, the load average [on all Unix/Solaris, etc machines] is the number of processes waiting per clock cycle. It is generally not an average, but an instaneous number reported as:

1 minute ago, 5 minutes ago, 15 minutes ago.

Different implementations use averages but its immaterial.

For example a single CPU box with a load average of 1 means that there is always a process waiting for time, but generally not more than one. A single CPU box with a load average of 1 and a quad-processor box with a load average of 4 are generally considered in a similar state.

Now this is the tricky part -- the load average doesn't actually mean the process is waiting on the CPU. It could be waiting on RAM, or a disk access or other. Generally these processes are put to sleep and don't contribute to the load average. When multiple processes are waiting for something [blocked] and aren't asleep for one or more reasons [like the process is being swapped in, or waiting on the CPU] this adds to your load average.

A box with a load average of 17 or a load average of 2 can behave entirely differently depending on the characteristics of the load. Some applications [html serving vs sendmail] are better with less resources because the the CPU only needs to fill the outbound buffer and this buffer empties comparatively slowly [like with a dialup down stream]. Since CGI scripts generally produce small amounts of output per CPU cycle, they are more seriously affected.

A Sun box with a load average of 20 or a linux box with a load average of 0.2 may actually be moving the same amount of traffic, but this is more related to the raw performance of the setup and not the operating system [except in very specific cases].

IDE vs SCSI. IDE drives tend to be less efficient with multiuser operating systems because they have no way of smart queuing requests. A SCSI drive will make multiple reads/writes in a single sweep, an IDE one won't. This means that a SCSI drive can perform far more accesses than an IDE one all things being equal. This has nothing to do with transfer rates or access times.

Hope this helps, if you'd like it simpler, let me know and I'll explain more.

Rik

Well, I ran those diagnostics and here are the results:

free -m

total used free shared buffers cached

Mem: 1518 1466 51 3728 72 129

-/+ buffers/cache: 1265 253

Swap: 133 27 105

bash$ vmstat

procs memory swap io system cpu

r b w swpd free buff cache si so bi bo in cs us sy id

5 2 0 28448 54736 74136 135708 0 0 3 20 30 8 26 12 0

bash$ vmstat

procs memory swap io system cpu

r b w swpd free buff cache si so bi bo in cs us sy id

1 1 0 28448 50424 74712 145032 0 0 3 20 30 9 26 12 0

bash$ vmstat

procs memory swap io system cpu

r b w swpd free buff cache si so bi bo in cs us sy id

2 0 0 28448 38208 74912 150692 0 0 3 20 30 9 26 12 0

I hope they format okay on this message board. Anyone care to interpret the results for me? It's all greek.

Note: the server seems to be behaving itself now, which is pleasing me.