Need to hear the voice of experience as I am surprised with such a big server I am having so many problems:

I am on a dedicated server with:
- Intel(R) Pentium(R) 4 CPU 2.00GHz
- 1G RAM
- Linux - Apache - Ensim

And I have:
10GB of thansfer per day
700000 pages per day
28000 emails sent per day (postcards).
6000 forum generated pages per day (phpBB -using php & MySQL).


The most server intensive thing, I guess, might be the use of the mail server, 28000 emails per day, but doing math that is not that much: just 19 emails per minute.

Forums can be quite intensive, but it's only occupying about 2% of the server requests...

Emails: Just plain emails with a link to the page. Sort of a cheap postcard. No data is being keept on the server, MySQL, etc... it's like a simple "tell a friend" email.


I am having to reboot the server a minimum of 3 times per day, as it get's really slow.
Forum is closed at peak hours or when server is slow.

Does it make sense to have so many problems on a server with a processor at 2GHz and 1 Gig of RAM? Or is something weird going on here?

Thanks.

The numbers sounds huge. Maybe you could move the forum out to a standalone server to lighten the load.

What does 'top' say at peak ?

1 website is pushing 300GB bandwidth a month - thats pretty lot of traffic.

Why not think about getting a higher rate server such as SM dual Xeon or something? 2GB ram or something - not sure

What HDD's do you have? Maybe thinking about SCSI would be a good idea, however I may be totally wrong :P

Regards

DislexiK

well you could setup thttpd to serve images which would cutdown on the load from apache, umm have you optimized ur mysql?

Greetings:

I would recommend setting up a separate email server, and probably a separate mySQL server.

If you have the three servers are a VLAN, then traffic between them would be quicker, and not counted towards any quotas.

Thank you.

The dedicated server is through a reseller, but I'm quite sure it's one of the Pentium4 Server Series at EV1Servers.

thelinuxguy > I'll ask about thttpd. The database is getting quite big, this week we want to trim it. It's recently been optimized.

eddy2099> What do you mean by "top" at peak? Top what? :)

Dual Xeon... quite expensive, but I guess I have to realize it's no longer a small site anymore...

DislexiK > Not sure about what HDD I have, other than the info from the link above... But not sure this would be the problem.

Here is what I notice the most: when it slows down, what takes the most time is for it to start sending the page & contents. Once it starts, it usually does it quickly. So that makes me think it might have something to do with too much "thinking" (processor I guess) or maybe lack of RAM. But I have no idea/experience in this, so I dont know...
Obviously, there are also times that it is also slow sending (not just starting to send) but a quick reboot fixes that for a while...



dynamicnet > That would be interesting... What server (CPU/memory & all that) would you recomend for the email server? Right now it's to about 28000 emails per day, to be sure & allow growing... say if it was 40000 per day?


I am a little worried about the expenses, I cant just go for anything, as the site is covering costs, but not much more...

Thanks to everyone for your help, opinions and insights :):) :) :)

'top' is a command is Linux which shows you the server details and show all running program at that given point of time.

SSH into your server and type 'top' without the quotation..

Try adding RAM before considering upgrading processor. I had a server running on cel 1.7 and the load was at 2.5+ during peak and I thought that upgrading to P4 2.66GHz would help much, but it didn't work out as I thought until I have upgraded the RAM and the performance increased a lot!

i'd put the email and the forums on seperate servers.

I am 99% sure it hasn't been optimized and that's why performing poorly. Even though the load is quite intensive, it can be tweaked for satisfactory performance without the need to reboot it.

Don't overlook the hard drive type too. Especially if you are short on RAM, then the system may be swapping.

If the system is swapping then IDE disks, if that is what you are using, are just aggravating the situation.

-Robert

Sorry I didnt reply again till now... I was waiting for monday night, as in the weekend the server has no problems, and to get more data to answer I needed to wait till now. ;)

I was quite surprised by looking at what "top" said at peak times when it got really slow... maybe I dont know how to interpret the data, but it seems to me the system was doing just fine
I closed the forums just to make sure. This is what came out in average when it was the slowest: 9:22pm up 44 min, 1 user, load average: 1.33, 0.98, 1.14
235 processes: 222 sleeping, 12 running, 1 zombie, 0 stopped
CPU states: 9.8% user, 4.3% system, 0.3% nice, 85.4% idle
Mem: 1023104K av, 785848K used, 237256K free, 0K shrd, 41736K buff
Swap: 2064344K av, 0K used, 2064344K free 168168K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
1177 apache 15 0 13160 12M 8632 S 7.4 1.2 0:01 httpd
15530 mtob 15 0 1180 1180 836 R 1.7 0.1 0:00 top
1388 apache 15 0 13324 13M 8792 S 0.3 1.3 0:01 httpd
1471 apache 15 0 13104 12M 8616 S 0.3 1.2 0:01 httpd
1224 apache 15 0 10352 10M 8652 S 0.1 1.0 0:00 httpd
1307 apache 15 0 14484 14M
At that exact moment (above), to ensure my personal connection didnt affect my data, I used Websitepulse.com website speed test, that came out like this:
URL tested: shoshan.cl (the site)
Status: Connection error - Connect failed
Server Status: Connect failed
Response time: 60.0095 seconds



So in view of this data, though I'm not sure I know how to interpret it, I am starting to think that the processor&memory is enough. Maybe as some of you suggested, type hard disk should not be overlooked.

- Dabase was optimized this weekend, but it's use through the forum was closed at this time, so I dont think it's that...

What do you all think? :)
I'm wanting to learn from you experts ;)

Thanks for your help! Really, thanks! :)

MTO,

It doesn't look so bad. Have you checked to make sure that apache isn't reaching the maximum configured number of processes?

I believe by default it is set to 150? Somebody help me out here...

Yes, try increasing the MaxClients to 250

This might help you with your problems.

Yes, it was set to 150. :rolleyes:
Could it have such a simple solution? I just changed it to 250, I'm so excited right now I cant wait to tomorrow's peak time. I'll be back to report on that ;)

Thanks for your ideas and thoughts! :)

No problem. It could be that simple, but I guess we will find out tomorrow.. :-)

Are you running mysql4? Query caching helps :)
As for phpbb, setup Turck MMCache and template cache

http://phpbb.com/phpBB/viewtopic.php?t=125251
https://sourceforge.net/projects/turck-mmcache

I would think it was the MaxClients setting, considering you get ALOT of visitors per day...

Don't mean to cut in - but where might I find a tutorial or something of that nature on mysql optimization?

Found some stuff on ev1 for a slightly older version - wasn't sure if I could still use it...

load average: 1.33, 0.98, 1.14

This looks good to me

And the 84% idle is also good lol

Regards

DislexiK

Originally posted by rjg
No problem. It could be that simple, but I guess we will find out tomorrow.. :-) Wow! MySQL 3.23.56 is installed. I certanly didnt expect to have such an old version. Could this be affecting speed as well?

MTO, give it 3 days, come back and you tell us ;)

Regards

DislexiK

I don't know if it will, but ypu might consider upgrading to MySQL 4, I never regretted it.

Yep, mysql 4.0.17 is production level, no need to worry about stability
Have you installed both cache thingy?

After a week of testing I am back...

Things have been much better since I set the MaxClients setting up. :)

The server has been much faster since then.

But... there are some days, like yesterday, that it is really slow and inconsistent. At one moment it loads just fine, a minute later it takes 20 seconds or more for the server to respond...
During this time, the server looks like as if it's hardly having any load at all, for example:
10:47pm up 15:32, 2 users, load average: 1.97, 2.21, 2.18
112 processes: 107 sleeping, 3 running, 2 zombie, 0 stopped
CPU states: 7.1% user, 20.7% system, 0.0% nice, 72.0% idle
Mem: 1023104K av, 1001076K used, 22028K free, 0K shrd, 90960K buff
Swap: 2064344K av, 0K used, 2064344K free 779240K cached

Some nights, it has reached 350 processes and similar cpu idle percentages and been much faster responding.

So I'm having a problem, and dont know where to look at now. It seems to me it is "enough server", never seen more than 50% cpu use, Swap is almost ever used, and load average is always bellow 5.
But even then, there are times connections die out, server responses take 20 seconds, or more... Requesting a single html page takes forever and users complain and leave. Rebooting doesnt help.

Could it be a network thing? settings? Something I can fix? This speed inestability is keeping lots of visitors away. :(



Once again, thanks for your help! :) :)

MySQL was upgraded, no performance change seen.
Daijoubu > I'll look into that cache thing now.

But still... I am not conviced at all that it is a problem of the server having too much load. Now people are having problems connecting even off peak hours. At this very moment people are reporting slowness, but this is how it is:
3:46pm up 18:47, 1 user, load average: 1.45, 1.77, 1.73
106 processes: 100 sleeping, 2 running, 4 zombie, 0 stopped
CPU states: 11.2% user, 23.5% system, 0.0% nice, 65.1% idle
Mem: 1023104K av, 993320K used, 29784K free, 0K shrd, 92672K buff
Swap: 2064344K av, 0K used, 2064344K free 763424K cached

It seems relaxed! Last week, under same configuration there was a moment it had 400 processes and speed was much better. :eek: :confused:

Thanks for any more thoughts, suggestions or ideas if there are any... ;)

MTO, it looks like your CPU load (under 2) and idle percents (over 50% idle) are fine. Since you seem to have so many visitors, your problem may again be apache's maxclients. How it works is that apache's maxclients limits how much users can simultaneously connect to your server via apache. When that number is reached, the users are simply queued which usually just results as a "really long load" for them as they have to wait until that number goes down. You may try bumping up your maxclients up some more but be wary of bumping it up too high as apache placed that measure to make sure your server doesn't going cannibalistic on itself due to too much server load.

OK, thanks, but could you explain me this...?
If MaxClients is set to 250, and processes are below 150 they do not reach the 250, so how would it be affected by this?

Thanks. :)

OK, sorry to be back here with a related problem once again...
MaxClients has been set these last 5 days to 370
Processor use has been around 75% iddle
Load time around 1

So everything going great after increasing maxclientes... but...

Lastnight I wasnt around, but they had to restart the server once. I am guessing for the same reason as today:

Today, after been going great, suddently it was really slow. I managed to get into ssh and see the following top:
9:49pm up 1 day, 3:09, 1 user, load average: 83.12, 76.67, 45.55
1113 processes: 1111 sleeping, 1 running, 1 zombie, 0 stopped
CPU states: 3.0% user, 5.9% system, 0.0% nice, 90.9% idle
Mem: 1023104K av, 1012964K used, 10140K free, 0K shrd, 1248K buff
Swap: 2064344K av, 1105952K used, 958392K free 18784K cached
Contact was lost... and the server had to be manually restarted.

Any ideas? Why so many processes? What can I do to prevent this? Why did this happen?:rolleyes:

Thanks for your help!

Dunno but your using lots of RAM and lots of SWAP as well :O

Regards

DislexiK

Yes, but my point is exactly that...
It's not using swap all day long, no swap. Processes all day long are usually no more than 250, and 400 at peak times.
CPU usage averages, even at peak times 70% idle.

But then... these last nights, <b>suddently</b>, at around the same time, it get's really slow, as I mentioned, more than 1000 processes, lots of memory & swap used, cpu remains idle & slow load average. If restarted, everything goes back to usual, it's been about 20 hours running and no swap used yet, everything is smooth... I'm only afraid in a few hours the same thing will happen again.

When you run top, try pressing shift 'm' ( or capital M ) This will list your processes by memory usage, and see what processes are eating up memory, and using your swap.

Besides uping MaxClients, also try modifying some of the other values. For example, my server has something like:

MaxClients 550
MinSpareServers 30
MaxSpareServers 50
StartServers 30

Upping max.clients is the last thing he should do if he's having serious memory issues. I've just solved my problem by yanking these settings DOWN. If there are loads of processes 'running' but doing nothing useful, all they're doing is eating memory!

Apache keeps spawning more children and the little beggars are lounging around doing nothing but using memory!

I run a fairly busy forum - currently 150 users online.

1:30pm up 9:25, 1 user, load average: 0.75, 0.44, 0.39
75 processes: 72 sleeping, 3 running, 0 zombie, 0 stopped
CPU states: 12.0% user, 5.0% system, 6.0% nice, 77.0% idle
Mem: 1023128K av, 719912K used, 303216K free, 0K shrd, 44588K buff
Swap: 2048276K av, 3892K used, 2044384K free 493616K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
1359 root 5 -10 20848 20M 1252 S < 0.0 2.0 0:00 ensim-python
1242 apache 9 0 20624 20M 4720 S 0.2 2.0 3:54 httpd
1230 apache 9 0 18576 18M 4856 S 2.0 1.8 3:39 httpd
6478 apache 9 0 17460 17M 4948 S 0.2 1.7 2:54 httpd
1233 apache 9 0 16168 15M 4720 S 0.0 1.5 3:44 httpd
1234 apache 10 0 15720 15M 4856 S 4.6 1.5 3:43 httpd
1240 apache 9 0 14260 13M 4860 S 0.0 1.3 3:42 httpd
1235 apache 9 0 13956 13M 4724 S 2.0 1.3 3:47 httpd
5956 apache 9 0 13112 12M 4952 S 0.0 1.2 3:01 httpd
1231 apache 14 0 12956 12M 4856 S 3.6 1.2 3:35 httpd
13496 apache 9 0 12728 12M 4952 R 0.2 1.2 1:58 httpd
27231 apache 9 0 12304 12M 4884 S 0.8 1.2 0:01 httpd
1104 mysql 15 10 13412 11M 1580 S N 0.0 1.1 0:02 mysqld
1109 mysql 12 10 13412 11M 1580 S N 0.0 1.1 0:04 mysqld
1110 mysql 15 10 13412 11M 1580 S N 0.0 1.1 0:00 mysqld
1117 mysql 15 10 13412 11M 1580 S N 0.0 1.1 0:00 mysqld
27398 apache 9 0 11392 11M 4772 S 0.2 1.1 0:00 httpd
27454 apache 9 0 10736 10M 4744 S 0.2 1.0 0:00 httpd
1190 root 9 0 6436 6436 2848 S 0.0 0.6 0:00 httpd
1414 root 9 0 2640 2640 1228 S 0.0 0.2 0:01 poprelayd
22355 root 9 0 1968 1916 1488 R 0.0 0.1 0:00 sshd
1261 postgres 9 0 1836 1836 1468 S 0.0 0.1 0:00 postmaster
1258 postgres 9 0 1820 1820 1484 S 0.0 0.1 0:00 postmaster
1349 nobody 5 -10 1816 1816 1148 S < 0.0 0.1 0:00 ocwhttpd
1350 nobody 5 -10 1816 1816 1148 S < 0.0 0.1 0:00 ocwhttpd
1351 nobody 5 -10 1816 1816 1148 S < 0.0 0.1 0:00 ocwhttpd
1352 nobody 5 -10 1816 1816 1148 S < 0.0 0.1 0:00 ocwhttpd
1353 nobody 5 -10 1816 1816 1148 S < 0.0 0.1 0:00 ocwhttpd
1347 root 5 -10 1812 1812 1156 S < 0.0 0.1 0:00 ocwhttpd
1260 postgres 9 0 1808 1808 1464 S 0.0 0.1 0:00 postmaster
1124 root 8 0 1752 1752 1112 S 0.0 0.1 0:00 sendmail
1348 nobody 5 -10 1468 1468 800 S < 0.0 0.1 0:00 ocwhttpd


My config

MaxClients 32
MinSpareServers 5
MaxSpareServers 10
StartServers 10

JanetGeorge > What you mention surprises me, as everyone seems to say the opposite. I myself upped the MaxClients from 150 (slow server) to 350 and now it goes great, except for those few moments.

----------------------------

Just a few minutes ago server had to be manually rebooted, but this time I detected what was causing it (cant confirm if it was the same thing the other times). Something or somebody was requesting thousands of forum pages, creating too much load, at one point top showed the following:
1:27pm up 10:14, 1 user, load average: 149.47, 125.10, 84.45
2243 processes: 2200 sleeping, 1 running, 42 zombie, 0 stopped
CPU states: 10.8% user, 49.0% system, 0.0% nice, 40.1% idle
Mem: 1023104K av, 1013800K used, 9304K free, 0K shrd, 1836K buff
Swap: 2064344K av, 1581484K used, 482860K free 53036K cached
[i]ordered by memory showed hundred of httpd & mysql occuping no more than 1% memory each[i]

The IP that was causing this was 210.14.123.31
I couldnt find any info on it, only that it's from Australia. Dont know if it's an abusive search engine crawler or an attack. The IP has been banned to prevent this from happening again.

---------------------------------------

OK, so back to my configurations... this is what I have right now:
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
MinSpareServers 5
MaxSpareServers 10
StartServers 5
MaxClients 350
#MaxRequestsPerChild 30

Originally posted by MTO
[ 1:27pm up 10:14, 1 user, load average: 149.47, 125.10, 84.45
2243 processes: 2200 sleeping, 1 running, 42 zombie, 0 stopped
CPU states: 10.8% user, 49.0% system, 0.0% nice, 40.1% idle
Mem: 1023104K av, 1013800K used, 9304K free, 0K shrd, 1836K buff
Swap: 2064344K av, 1581484K used, 482860K free 53036K cached
[i]ordered by memory showed hundred of httpd & mysql occuping no more than 1% memory each[i]



I am NO expert - BUT 2243 processes with 2200 sleeping, a HUGE load and practically no memory left tells the tale. 100 httpd processes doing nothing and each using 1% of memory add up to NO memory left and a big bang.

I was VERY reluctant to drop max clients from 150 (when we were running out of memory, getting big server load and crashing) to 32 - but the results are terrific. And that's with up to 150 users online at once (and they're a gabby lot!)

I don't know how efficient your forum software is, or what load the mail puts on the server - but it didn't look bad when the forum was turned off so my guess is that it's all these unneeded and unused processes that are killing it.

MTO: You have to enable the query cache
http://www.mysql.com/doc/en/Query_Cache_Configuration.html

Add this to my.cnf (under [mysqld]) and restart mysql:
query_cache_limit = 1M
query_cache_size = 16Mb
query_cache_type = 1

If you don't use InnoDB, also add this:
skip-innodb

Can you post your current my.cnf?

Have you tried to turn off Keep-Alive for awhile?

Thanks, I'll look into these things.
I will report back (though probably after the weekend). Thanks. :)

Update on how things are going for me...

- With MaxClients set to +250 speed is good.
- Based on JanetGeorge's experience, I tried bringing MaxClients down to 50, it became so slow that I had to change it back up a few minutes later.
- With MaxClients 200 swap is used.
- I am rebooting every time I see swap goes higher than 1G.
- Rebooting average of 3 times a day.
- If I dont reboot it, and left unatended, server ends up not responding and someone has to manually reboot it.

Just a reminder of my server specs:

- Intel(R) Pentium(R) 4 CPU 2.00GHz /1G RAM /Linux - Apache - Ensim

And I have:
12GB of thansfer per day
700000 pages per day (99% static html pages)
28000 emails sent per day (postcards).
6000 forum generated pages per day (phpBB -using php & MySQL).

This is what's in my.cnf:[mysqld]
set-variable = query_cache_type=1
set-variable = query_cache_size=16M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

[mysql.server]
user=mysql
basedir=/var/lib

[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

If you don't use InnoDB, also add this:
skip-innodb How can I tell if I use it? As far as I know I dont, but I dont know...:rolleyes:

I am thinking of seting up a cron to reboot every X hours... not the best thing to do, but at least constant rebooting is keeping the server alive.

Hey dude, could it be something running in the back ground?

I think it would be in processes if your running that thing, so do:

top | grep "InnoDB"

See if it brings anything back :)

Regards

DislexiK

MTO, look for a script sending mail. Check your mail logs too. (maybe some script of yours is vulnerable)

Also try:

Timeout 30
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15
MinSpareServers 5
MaxSpareServers 10
StartServers 5
MaxClients 512
MaxRequestsPerChild 3000

Just as a suggestion maybe there might be a memory leak in the forums.

Turning KeepAlive may help but adds lots of lantency

I would say to keep it on but lower the KeepAliveTimeout to about 5secs

Btw, have you installed the extreme style mod (template cache) and Turck MMCache?
Any phpinfo?

InnoDB is one of MySQL storage engine, I doubt you'll use it since you don't know what it is ;)

Most BBS relay on MyISAM (phpBB/IPB/vB), which is faster but isn't transaction safe

Edit: Just checked, your forum main page would requier to make about 35connections if you had turned off KeepAlive...

And also noticed that you're still running PHP 4.2.2, upgrade to 4.3.4, tons of bugs have been fixed heh :D
Then install/recompile MMCache (you can't and shouldn't use modules compiled with another version...)

MTO, I may have a solution to your problem as I had exactly the same thing. Server would be fine then 2 or 3 times a day for no reason whatsoever the httpd and mysql processes grew exponentially till the server stopped responding and had to be rebooted at the switch.

I eventually tracked the problem down not to apache which was where I was looking, not in my code which was also where I was looking but eventually to a forgotten log database table i'd made over 6 months ago and forgot it was creating a new record for every visitor.

Our server isn't very powerful P800 128MB Ram and this table was approx 200,000 records which wasn't massive but compared to all out other tables it was hundreds of times larger and accessed every time somebody loaded a page, as soon as I erased the contents of this table everything was fine, I then removed the code from my site and everything has run without a hitch infact since I fixed this 4 months ago the server hasn't even been rebooted.

I've never found out what it is within mysql4 thats caused this but i'd put my life savings (all £5 of it) on your problem being a very large database table which is causing mysql and apache to leave processes open for some unknown reason.

Kind Regards

Andy
enjoy.co.uk