Web Hosting Talk : Web Hosting Main Forums : Dedicated Server : Server Load .. Need advice - Urgent

[infotq]

Hello,

one of my server is experiencing very high load issue(between 150-400). I know the load average is caused by a cgi script because lot of traffic coming to this server, which is good for the kind of business I run. Its a search engine script.

So If I disable the cgi script then I am loosing all my business because the script is the one which gets all busines to me.

The current server configuration is :

dual xeon 3.0, IDE drive, 4GB RAM


So what can be a better solution to this kind of situation. I have heard of "Load Balancing". Is that something I should be looking for.

Since the tunning is not possible on the perl/cgi script. So Is there any kind of tunning can be done for the perl/cgi on the server side.

Which DataCenter offers "High Configuration server" with "High Availability".

Your advice is really important to me, So please share your experience.

I am willing to go upto $500 monthy for a good solution.

Thanks
infotq

[Richard]

I suggest reviewing your script's code. That's serious load for your current machine's specs.

[Jelleuh]

Such a load really isn't healthy for your system, it surprises me it was still runing with such a load. Your scripts must have a bug, otherwise it wouldn't cause so much problems. Good luck on checking it out!

[mwatkins]

If your script is being called (hit) very frequently, and runs only as CGI, then you are going to have performance issues, particularly if the script is non trivial. What does it do?

Link?

think about it - every hit to the script url is firing up a process (assuming perl but you can advise) - that alone limits how many requests per second can be pumped through the box, and then you have to factor in what the script is doing. Does it start up additional processes? Contact remote hosts?

Even a very small script, if run often enough, can put any machine, of any reasonable configuration, way over the 'red line'.

A persistent process (perhaps FCGI?) might quickly solve your problem.

[MrZillNet]

If it is truly a CGI script, which is executed each time the page is hit, then you can handle 10 to 100 times the traffic, simply by converting it to FastCGI or other method, that starts the Perl process once.

[serverpoint]

If it's Perl, it may also work with pperl (persistent perl).

I'm assuming your script is doing too much processing looking through files. Perhaps you need an internal caching mechanism?

For example, when you search "mp3" on google, google doesn't go through all its billion records looking for that every time someone searches for it.

It does that search *once*, and then it stores the results, and those stored results are what is shown to the next million people that search "mp3".

A load so high on a dual xeon might be due to inefficient code. You might want to have the developer review the code and think on some new ways to do old things.

You can have multiple servers, however, something tells me the problem will never end, and you will have way too many servers for something that can be fixed through code.

Also, be aware, that the load may *not* be the CGI. It might be the I/O (hard drives) of the server not being able to keep up with so many requests. That happens too when a drive is starting to die.

[ndimensionz]

It can happen that the cgi script is calling so many files.
You can check it out using lsof command.
Load balancing servers can be obtained from NOCs.
Usually its implemented at the hardware level and not the software level.

[erigid]

The best solution is to optimize both hardware and software of course. The best advice people can give you without looking at the application is to fix your software and/or get a faster machine. If you go to the right person, they should be able to tell you what software tools you can use to optimize your script and what kind of hardware is needed and how to configure it. The only optimization I could give you by just reading your post is to install mod_perl. If you are running that script as CGI, every time you get a request, your webserver has to spawn a perl interpreter and thats slow. Mod_perl embeds the interpreter into apache so it doesn't have to spawn a new one for each request.

Some of the optimizations that people could blindly recommend are:
making sure you have good db queries/indexes, memcached, disk raids, etc but they wont necessarily apply to your app.

My best advice is to get some of the perl tools to do script profiling so you can see where the bottlenecks on your script are. Using these tools you will be able to tell what subroutines or code blocks take longer and from there determine the bottlenecks. People can make better recommendations once they know the bottlenecks.

[WHC - Travis]

You could ditch the Xeon for a quad dual core opteron server, with over 4gb of RAM, but that would be expensive. Get that code optimized first.

[infotq]

Hello All,

After going thru all of your advice, I am going to change the hardware as following:

Dual Xeon with SCSI Drive + 2GB RAM + 100mbps network port

OS will be installed on IDE drive and /home will be created on SCSI drive, So that the performance can be better. Do you think this help to improve the performance.


Thanks
infotq

[mwatkins]

I think you are wasting your money, personally. If the script really is running as CGI, then that's the problem, not the hardware.

CGI was never designed to be used for high volume sites. CGI scripts, in essence, do the following, *FOR EVERY SINGLE HIT*:

- load up the interpreter (assume its perl for clarity of this discussion)
- load up any required modules
- initialize the application
- load up any data and/or make connections with databases (if used)
- complete its logic
- display a result to the browser
- perform any application clean up, disconnect with databases (if used)
- terminate

Doing all that as a "CGI" script, 1 time for every request, puts a big load on a server if there are a lot of requests. Just as an example, I ran Apache Benchmark against a real awstats report, running as a perl CGI script, on the box which hosts the awstats report (no network latency involved in this):

ab -n100 http://pathto.com/awstatsreportforsomesite/
Requests per second:    1.02 [#/sec] (mean)
Time per request:       981.28 [ms] (mean)
Time per request:       981.28 [ms] (mean, across all concurrent requests)
Transfer rate:          108.39 [Kbytes/sec] received

1 request per second; now if your site is busy and you are getting multiple requests per second, then you are going to max out your CPU easily. 4 concurrent request streams = approx 2 requests completed per second. 6 concurrent requests = approx 1.6 requests per second and a server load moving rapidly up > 7, 8.

Now if that script were part of a persistent web application, or run via mod_perl or other methods, you will no doubt see a doubling in performance at the very minimum. You may see 10X the performance.

The persistent web applications I work with in Python typically run as CGI can deliver 1 - 4 requests per second; when run as SCGI (similar to FastCGI) I get 200, 300, 400 requests per second or more and much less load on the server.

Bottom line: you need to look at what that script is doing before deciding what to do with hardware. Such decisions need to look at the entire picture... sometimes throwing hardware at a problem is the wrong approach.

[erigid]

Originally Posted by infotq

Hello All,

After going thru all of your advice, I am going to change the hardware as following:

Dual Xeon with SCSI Drive + 2GB RAM + 100mbps network port

OS will be installed on IDE drive and /home will be created on SCSI drive, So that the performance can be better. Do you think this help to improve the performance.


Thanks
infotq

Hehehe, I dont think anyone recommended changing the hardware =).

[eth00]

As others have said changing the hardware that little is not going to do much. Maybe if you went with some quad cpu system with 15k SCSI disks in RAID10 and a boatload of ram but something tell me that may be a little expensive for your budget.

As others have said take a look at how your system first and try to fix that cgi. It may even be possible to optimize the server some which can often significantly decrease the load.

Is the server still responsive when the load is 100+ ?

[nofz]

If its just one script, just fix the script and get it cached. Serious if you get such high load with this script on a dual xeon, even if you upgrade to quad-xeon 4gb mem it will still reach a high load, because problem isnt solved.