Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : Major server performance problems after "upgrade"

[slack]

Hi, I posted this in my host's forum but haven't had any response so I'm reposting here. This problem is in regards to a dedicated server that I'm leasing.

-------

So I've been running a server with RH9 + Plesk 6 for a couple years now. I know, it was getting a bit old, but I kept it up to date with yum+ART and it was working just fine really.

Well the other night I pulled the most boneheaded move I've pulled in years: an rm -rf * in the filesystem root. Thought I was in a different directory. I about crapped myself when I got a warning about not being able to remove /proc. Lucky for me, I killed it before it got any important data...it just wiped out /bin, /sbin, and a few other choice pieces of the OS.

So I ordered a system restore and suddenly had a brand new system with RHEL and Plesk 7.5.4. Cool, at least maintenence should be easier from here on out, etc. Got all my data back on the server. All is good....almost.

The problem is that this thing is a dog by comparison. I used to have server loads in the 0.3 - 0.8 through most of the day. Occasionally it'd hop over 1 for a little bit. Now my server load is over 1 a *lot*. CPU usage is up a *lot* - both mysqld and httpd are using a lot more cpu. The thing is just slow, and I can't figure it out. For instance, there is one forum on this server and loading the index used to take between 0.2 and 0.5 seconds, no big deal. But on the new server it will frequently (like say once every 8-10 loads), take 3-8 seconds. Just an example.


I was running Apache 2, PHP 4.3.11, and Mysql 4.1 on the old server (via ART packages). It now appears as though I'm pretty much running the same set, so I'm confused.

Here's what I've done:

1) Enabled query caching in Mysql, and basically setup my config the way it was before, which was working well. Verified that qcache is working and has plenty of spare memory.

2) Tuned apache & php to settings similar to what they were. Disabled some modules that weren't in my old config and not necessary, like ldap auth. Installed mmcache and configured it the way it was before (disabled the script cache that was installed by default, which didn't seem to work well). Verified the cache is working and has plenty of spare memory.

3) Got rid of some new processes that I don't want, like dr.web antivirus...and i'm about to get rid of spamassassin since there are 8 spamd processes running and my plesk license doesn't allow me to use em.

One thing I noticed is that apache must be running a different MPM than my old config, or something. I used to see only one httpd process in top, now I see a half dozen of them, and I see them going defunct pretty regularly (although some people have said it's normal) which is something I've never seen before. These apache processes take up a lot more CPU than what I was used to before. I don't know a hell of a lot about apache configuration.

Nothing out of the ordinary in messages, apache error logs, or any other error logs.

[slack]

Yeah this has me a bit stumped. Things just aren't right and I can't figure it out! I think it's mostly mysql but I can't be sure, the system just grinds sometimes and both httpd and mysqld take a good part of the credit. Very frequently when I load a page with a lot of queries, like a forum index, it will simply take several seconds to complete the request, almost like it's reading a ton of data from the hard drive, when it's not. I've tried to get everything into memory as much as possible since I have about 500MB excess at any given moment.

Here are some of my config settings:

httpd.conf
-----------
KeepAlive On
KeepAliveTimeout 5
MaxKeepAliveRequests 0
StartServers 16
MinSpareServers 12
MaxSpareServers 24
ServerLimit 250
MaxClients 250
MaxRequestsPerChild 4000

(I've experimented with more conservative values for the above (like 4 initial servers, 4 spares, 8 max spares) but saw no change)

php.ini
-------

zend_extension="/usr/lib/php4/mmcache.so"
mmcache.shm_size="96"
mmcache.cache_dir="/mmcache"
mmcache.enable="1"
mmcache.optimizer="1"
mmcache.check_mtime="1"
mmcache.debug="0"
mmcache.filter=""
mmcache.shm_max="0"
mmcache.shm_ttl="0"
mmcache.shm_prune_period="0"
mmcache.shm_only="1"
mmcache.content="shm_only"
mmcache.keys="shm_only"
mmcache.sessions="shm_only"
mmcache.compress="1"

(as you can see the entire mmcache is forced into memory. I had default settings previously and recently changed this. no help)


my.cnf
------
query_cache_type=1
query_cache_size=50M
query_cache_limit=2M
skip-locking
key_buffer = 180M
max_allowed_packet = 16M
sort_buffer_size = 1M
read_buffer_size = 1M
join_buffer_size = 1M
read_rnd_buffer_size=768K
myisam_sort_buffer_size = 64M

table_cache = 2048
thread_cache=80
thread_cache_size=80
thread_concurrency = 2
max-connections = 150
max-user-connections = 150
interactive-timeout = 100
wait_timeout = 60
connect_timeout = 10
log_slow_queries = ON

(the above settings were recently changed. larger query cache, larger table cache, lower max connections. no change in performance that i'm seeing yet)


Here are my vitals:

system
-------
Mem Total: 1GB
Mem Used: 290732
Free: 744056
Swap: 2GB
Swap Used: 76880
Swap Free: 1971396
Processor 1.7Ghz Celeron
/dev/hda5 54538204 33486804 18280996 65% /
Timing buffered disk reads: 32 MB in 3.12 seconds = 10.25 MB/sec

(the disk read time above has me concerned. I did it last night and saw 30MB/sec, which is more expected. it could be the current server load. the drive is in udma5 currently)

mmcache
---------
Caching Enabled true
Optimizer Enabled true
Memory Size 100,663,260 Bytes
Memory Available 45,790,900 Bytes
Memory Allocated 54,872,360 Bytes
Cached Scripts 1146
Removed Scripts 0
Cached Keys 0

current mysql
------
| Open_tables | 588 |
| Opened_tables | 599 |
| Qcache_free_blocks | 3535 |
| Qcache_free_memory | 25230592 |
| Qcache_hits | 326286 |
| Qcache_inserts | 130381 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 7057 |
| Qcache_queries_in_cache | 18079 |
| Qcache_total_blocks | 40054 |
| Threads_cached | 20 |
| Threads_connected | 1 |
| Threads_created | 21 |
| Threads_running | 1
| Key_blocks_not_flushed | 0 |
| Key_blocks_unused | 98469 |
| Key_blocks_used | 65018 |
| Key_read_requests | 32744913 |
| Key_reads | 65030 |
| Key_write_requests | 96716 |
| Key_writes | 74944 |
| Max_used_connections | 21 |

current httpd
-------------
CPU Usage: u77.22 s6.32 cu0 cs0 - 35.4% CPU load
4.69 requests/sec - 54.1 kB/second - 11.5 kB/request
8 requests currently being processed, 16 idle workers

[slack]

processes
-------------

root         1  0.0  0.0  3472  468 ?        S    01:58   0:00 init [3]
root      2082  0.0  0.0  3516  776 ?        Ss   02:02   0:02 xinetd -stayalive -pidfile /var/run/xinetd.pid
root      2415  0.0  0.0  2092  568 ?        Ss   02:13   0:09 syslogd -m 0
root      2419  0.0  0.0  1868  456 ?        Ss   02:13   0:00 klogd -x
root      2453  0.0  0.0  5556  824 ?        Ss   02:13   0:00 rpc.idmapd
root      2533  0.0  0.0  2760  416 ?        Ss   02:13   0:00 /usr/sbin/acpid
root      2548  0.0  0.0  4728  768 ?        S    02:13   0:00 /usr/lib/courier-imap/couriertcpd -address=0 -stderrlogger=/usr
root      2556  0.0  0.0  4596  564 ?        S    02:13   0:00 /usr/sbin/courierlogger imapd
root      2560  0.0  0.0  4888  716 ?        S    02:13   0:00 /usr/lib/courier-imap/couriertcpd -address=0 -stderrlogger=/usr
root      2563  0.0  0.0  4176  320 ?        S    02:13   0:00 /usr/sbin/courierlogger imapd-ssl
root      2570  0.0  0.0  4320  768 ?        S    02:13   0:01 /usr/lib/courier-imap/couriertcpd -address=0 -stderrlogger=/usr
root      2572  0.0  0.0  4120  552 ?        S    02:13   0:01 /usr/sbin/courierlogger pop3d
root      2581  0.0  0.0  4776  716 ?        S    02:13   0:00 /usr/lib/courier-imap/couriertcpd -address=0 -stderrlogger=/usr
root      2583  0.0  0.0  3660  320 ?        S    02:13   0:00 /usr/sbin/courierlogger pop3d-ssl
named     2613  0.0  0.2 37360 2312 ?        Ssl  02:13   0:05 /usr/sbin/named -u named -c /etc/named.conf -u named -t /var/na
root      2707  0.0  0.0  5908  944 ?        S    02:14   0:00 /bin/sh /usr/bin/mysqld_safe --defaults-file=/etc/my.cnf --pid-
mysql     2737  3.4 12.8 366456 132764 ?     Sl   02:14  21:04 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr
qmails    2763  0.0  0.0  2232  460 ?        S    02:14   0:23 qmail-send
qmaill    2765  0.0  0.0  3280  428 ?        S    02:14   0:07 splogger qmail
root      2766  0.0  0.0  2628  344 ?        S    02:14   0:02 qmail-lspawn ./Maildir/
qmailr    2767  0.0  0.0  2764  588 ?        S    02:14   0:02 qmail-rspawn
qmailq    2768  0.0  0.0  3480  296 ?        S    02:14   0:02 qmail-clean
postgres  2840  0.0  0.1 19504 1360 ?        S    02:14   0:00 /usr/bin/postmaster -p 5432 -D /var/lib/pgsql/data
postgres  2842  0.0  0.1 10304 1232 ?        S    02:14   0:00 postgres: stats buffer process
postgres  2843  0.0  0.1  9312 1232 ?        S    02:14   0:00 postgres: stats collector process
root      3049  0.0  0.4 47796 4716 ?        Ss   02:14   0:00 /usr/local/psa/admin/bin/httpsd
psaadm    3053  0.0  0.5 51860 5848 ?        S    02:14   0:00 /usr/local/psa/admin/bin/httpsd
psaadm    3054  0.0  0.6 50540 7092 ?        S    02:14   0:00 /usr/local/psa/admin/bin/httpsd
psaadm    3055  0.0  0.7 50540 7308 ?        S    02:14   0:00 /usr/local/psa/admin/bin/httpsd
psaadm    3056  0.0  0.6 50540 7168 ?        S    02:14   0:00 /usr/local/psa/admin/bin/httpsd
psaadm    3057  0.0  0.7 50540 7308 ?        S    02:14   0:00 /usr/local/psa/admin/bin/httpsd
psaadm    3122  0.0  0.4 47796 4628 ?        S    02:14   0:00 /usr/local/psa/admin/bin/httpsd
root      4467  0.0  0.1  8776 1260 ?        Ss   02:14   0:00 cupsd
root      4493  0.0  0.0  2832  412 ?        Ss   02:14   0:00 gpm -m /dev/input/mice -t exps2
root      4502  0.0  0.0  5632 1032 ?        Ss   02:14   0:00 crond
xfs       4522  0.0  0.0  3452  832 ?        Ss   02:14   0:00 xfs -droppriv -daemon
dbus      4539  0.0  0.0  2816  932 ?        Ss   02:14   0:00 dbus-daemon-1 --system
root      4695  0.0  0.0  1928  408 tty1     Ss+  02:14   0:00 /sbin/mingetty tty1
root      4696  0.0  0.0  1792  408 tty2     Ss+  02:14   0:00 /sbin/mingetty tty2
root      4697  0.0  0.0  2004  404 tty3     Ss+  02:14   0:00 /sbin/mingetty tty3
root      4698  0.0  0.0  3112  408 tty4     Ss+  02:14   0:00 /sbin/mingetty tty4
root      4699  0.0  0.0  2456  408 tty5     Ss+  02:14   0:00 /sbin/mingetty tty5
root      4700  0.0  0.0  2592  356 tty6     Ss+  02:14   0:00 /sbin/mingetty tty6
root      4945  0.0  0.0  1908  388 ttyS0    Ss+  02:14   0:00 /sbin/agetty -L 9600 ttyS0 vt100
root     12385  0.0  0.0  3512  524 ?        Ss   03:57   0:00 rhnsd --interval 240
root     14733  0.0  0.0     0    0 ?        S    04:05   0:01 [pdflush]
mailman  15427  0.0  0.5 11928 5284 ?        Ss   04:08   0:00 /usr/bin/python /usr/lib/mailman/bin/mailmanctl -s -q start
mailman  15429  0.0  0.4 10476 4412 ?        S    04:08   0:01 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=ArchRunne
mailman  15430  0.0  0.3 11068 4124 ?        S    04:08   0:01 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=BounceRun
mailman  15431  0.0  0.4 10208 4296 ?        S    04:08   0:01 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=CommandRu
mailman  15433  0.0  0.5 11644 5344 ?        S    04:08   0:01 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=IncomingR
mailman  15436  0.0  0.4 11220 5068 ?        S    04:08   0:01 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=NewsRunne
mailman  15437  0.0  0.5 11620 5404 ?        S    04:08   0:01 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=OutgoingR
mailman  15438  0.0  0.4 10384 5116 ?        S    04:08   0:01 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=VirginRun
mailman  15439  0.0  0.5 10712 5200 ?        S    04:08   0:00 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=RetryRunn
root     15885  0.0  0.2  7540 3060 ?        Ss   04:10   0:00 hald
root     16136  0.0  0.1  4176 1448 ?        Ss   04:11   0:00 /usr/sbin/sshd
popuser    873  0.0  0.1  6136 1624 ?        S    08:24   0:10 /usr/bin/imapd Maildir
popuser    874  0.0  0.1  6344 1472 ?        S    08:24   0:01 /usr/bin/imapd Maildir
root     12310  0.0  0.2  7172 2080 ?        Ss   11:05   0:00 sshd: admin [priv]
admin    12322  0.1  0.2  7332 2144 ?        S    11:05   0:10 sshd: admin@pts/0
admin    12323  0.0  0.1  4316 1376 pts/0    Ss   11:05   0:00 -bash
root     12495  0.0  0.1  4648 1172 pts/0    S    11:07   0:00 su -
root     12497  0.0  0.1  4468 1436 pts/0    S    11:07   0:00 -bash
qmailr   18311  0.0  0.0  4676  976 ?        S    12:17   0:00 qmail-remote webpim.de  fannieh@webpim.de
root     18743  0.1  1.2 123460 12544 ?      Ss   12:25   0:00 /usr/sbin/httpd
root     18751  0.0  0.8 19688 9044 ?        S    12:25   0:00 /usr/sbin/httpd
apache   18754  0.3  2.0 126844 20924 ?      S    12:25   0:01 /usr/sbin/httpd
apache   18755  0.1  1.8 126228 19456 ?      S    12:25   0:00 /usr/sbin/httpd
apache   18756  0.1  1.7 124740 18256 ?      S    12:25   0:00 /usr/sbin/httpd
apache   18757  0.1  1.6 124484 17020 ?      S    12:25   0:00 /usr/sbin/httpd
apache   18758  3.1  1.9 125976 19816 ?      S    12:25   0:12 /usr/sbin/httpd
apache   18759  0.1  1.6 125220 16828 ?      S    12:25   0:00 /usr/sbin/httpd
apache   18760  9.6  1.7 125268 18328 ?      S    12:25   0:37 /usr/sbin/httpd
apache   18761  2.2  1.7 125152 18172 ?      S    12:25   0:08 /usr/sbin/httpd
apache   18762  0.1  1.7 124432 18424 ?      S    12:25   0:00 /usr/sbin/httpd
apache   18763  0.1  1.6 124820 17204 ?      S    12:25   0:00 /usr/sbin/httpd
apache   18764  0.5  1.7 124852 17640 ?      S    12:25   0:02 /usr/sbin/httpd
apache   18765  0.1  1.6 124780 17220 ?      S    12:25   0:00 /usr/sbin/httpd
apache   18766  0.4  2.1 129472 22680 ?      S    12:25   0:01 /usr/sbin/httpd
apache   18767  0.0  1.5 124296 16516 ?      S    12:25   0:00 /usr/sbin/httpd
apache   18768  4.6  1.6 125136 17088 ?      S    12:25   0:18 /usr/sbin/httpd
apache   18769  0.1  1.6 124368 16804 ?      S    12:25   0:00 /usr/sbin/httpd
apache   18775  0.5  2.0 135196 21068 ?      S    12:25   0:02 /usr/sbin/httpd
apache   18783  0.3  1.8 127068 18760 ?      S    12:25   0:01 /usr/sbin/httpd
apache   18792  4.2  1.7 125244 18012 ?      S    12:25   0:15 /usr/sbin/httpd
apache   18793  0.5  1.6 125160 17100 ?      S    12:25   0:02 /usr/sbin/httpd
apache   18804  0.5  1.8 124812 18800 ?      S    12:25   0:01 /usr/sbin/httpd
apache   18805  0.1  1.6 126628 16872 ?      S    12:26   0:00 /usr/sbin/httpd
apache   18806  0.1  1.5 124556 15900 ?      S    12:26   0:00 /usr/sbin/httpd
apache   18807  4.5  1.6 125268 17000 ?      S    12:26   0:15 /usr/sbin/httpd
root     18903  0.0  0.0     0    0 ?        S    12:27   0:00 [kjournald]
qmaild   19059  0.0  0.0  5052  924 ?        Ss   12:30   0:00 /var/qmail/bin/qmail-smtpd /var/qmail/bin/smtp_auth /var/qmail/
qmailr   19101  0.0  0.0  4628  972 ?        S    12:31   0:00 qmail-remote hotmail.com system@recordingproject.com jeff_doeri
root     19106  0.0  0.0  3428  484 ?        Ss   12:31   0:00 tcp-env /var/qmail/bin/relaylock /var/qmail/bin/qmail-smtpd /va
qmailr   19111  0.0  0.0  4536  984 ?        S    12:31   0:00 qmail-remote comcast.net GinaJohns@flowerstopmilford.com 0000km
root     19112  0.0  0.0  1580  476 ?        Ss   12:31   0:00 tcp-env /var/qmail/bin/relaylock /var/qmail/bin/qmail-smtpd /va
popuser  19113  2.0  0.0  5112  944 ?        S    12:31   0:00 /usr/bin/pop3d Maildir
root     19115  0.0  0.0  3248  748 pts/0    R+   12:31   0:00 ps aux

[slack]

Ok I just forced my drive into 32bit mode with unmaskirq on. We'll see if there's any change.

.
.
.
.

nope.

I'm pretty certain that either PHP or MySQL is to blame, most likely MySQL. Static pages aren't delayed. Dynamic pages with small queries are fine.

I'm also seeing a LOT more defunct HTTPD processes today.

No errors being logged anywhere.

I'm getting low numbers via hdparm for my primary drive...from 10MB/s to 22MB/sec...compared to the numbers from my secondary drive which are over 40MB/sec. This makes me think there's a lot of hard drive activity going on...possibly from mysqld? Why? My largest database is only like 250MB. This system handled it fine previously. I have more free memory than I know what to do with.

[slack]

Ok, the following query is constantly in my slow query log:

# Query_time: 7 Lock_time: 0 Rows_sent: 32 Rows_examined: 40108
SELECT t.forum_id, t.topic_id, p.post_time
FROM phpbb_topics t, phpbb_posts p
WHERE p.post_id = t.topic_last_post_id
AND p.post_time > 1141810174
AND t.topic_moved_id = 0;

phpBB of course. This query is routinely taking > 4 seconds. There are indexes on every column referenced. I've optimized, repaird, and flushed both tables. I realize this query can't be cached, but it executed fine before my restore. What could be going wrong?

[slack]

Ok, I sped things up considerably by cranking my settings up...most impact was setting the join buffer to 15MB. (the indexes were in total > 10MB) Now my page load is around 1.5s when it takes a long time, which is better than 7 - 30s.

I still don't see why things have changed so much. I'm basically running the exact same system as before in terms of major versions of software.


My new settings are:

key_buffer = 200M
max_allowed_packet = 16M
sort_buffer_size = 3M
read_buffer_size = 3M
join_buffer_size = 15M
read_rnd_buffer_size=1M
myisam_sort_buffer_size = 80M

[henningl]

weird problem..

Kernel version ?

btw, are you sure that you need a 200M key_buffer? Not that that is to blame for the slow pageloads, just seemed a bit much.

[Website Rob]

Here is a suggestion for you, Slack.

Timeout 30

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 30

MinSpareServers 5
MaxSpareServers 10
StartServers 5

MaxClients 180
MaxRequestsPerChild 1000

Note: adjust as you see fit but changing 'Timeout' and 'KeepAliveTimeout' to 15 instead of 30 may improve page load speed at the cost of higher Apache usage.


Also, it's not always good to setup a newer, bigger/better/faster Server, with the same configs as a previously slower one. Tweaks that worked before may not be needed or, as may be in this case, be detrimental on the new Server.

[slack]

Thanks guys.

This server isn't better, it's the same exact server. I was running RH9 + Plesk 6 and the restore was RHEL3 + Plesk 7.5.4. The only hardware change was the hard drive.

I had KeepAliveTimeout set to all sorts of values with no apparent change in performace. I don't think that apache is the problem here, although it certainly could be. This server only serves about 50,000 pages a day.

My CPU time isn't real high...it's higher than it *was*, but still doesn't go over 30% too much. I have more free memory than I know what to do with...usually about 600MB...which is why you're seeing such high settings for keybuffer and whatnot, I've been experimenting.


Right now I'm really wondering about I/O. If it's not memory and it's not CPU, then that leaves the subsystem. It could be the new kernel and modules, or it could be a goofy hard drive (although I think it's the same model). When I run hdparm on this drive it frequently only posts 10MB/sec, while the secondary drive which isn't used but for backup posts 40MB/sec +. The hard drive should not be seeing such high activity.

Looks like my kernel version is 2.6.9-22. I think I was using a 2.4 kernel on the old install.

[slack]

Ok, I think I might have solved the problem with this one particular page load...I added an index to t.topic_last_post_id. I'm still not sure why the change occurred. I'm going to watch my stats today and see if my loads go down because of this change...if so, then I guess I'll be cool with it. If not, then I've still got work to do.

[BrandonSCSN]

Intersted in finding out what you concluded...

Please let us know!

[slack]

Well, the good news is that the server is settled down. Loads well under 1 a majority of the time.

Running the phpBB query I mentioned above directly via mysql used to take like 8 seconds. After increasing the join buffer size so much, it started taking like 4 seconds - good, but not nearly good enough. After adding an index to topic_last_post_id in the topics table, the query takes less than a tenth of a second.

The forum in question is running better than ever, however...


Why did it work fine before? The only major differences are that I was running a 2.4 kernel and mysqld-max. Now I'm running a 2.6 kernel and mysqld. The difference between mysqld-max and mysqld is minimal, so I'm thinking it's the new kernel and/or modules. I believe it may have something to do with I/O, because even throwing major memory at mysqld overall, the server still seemed to strain. iostat shows that I'm reading 400KB and writing 200KB+ every second almost all of the time, which shouldn't be challenging anything.

I'm concerned about this because while I was able to tweak the system back into shape, where would I be if the system ran as well as before and THEN I made these tweaks? How is this going to affect my ability to scale up as server demands increase? I'm not ready to rest on those questions yet.

I'd drop back to a 2.4 kernel but really it terrifies me. Some of my customers already lost a good 6-8 hours of uptime from my initial screwup.


I am somewhat glad for these troubles because I was able to learn a few things:

1) Use mysql's log slow queries feature to keep tabs on queries that are overloading the server. I have my slow query timeout set to just a few seconds, because on a web server, a query taking more than a few seconds should be a very rare thing. Tools like mytop might be nifty, but on a web server you pretty much sit there looking at one query - the one mytop is running.

2) I learned that changing apache's log level to info can be very valuable, at least temporarily. I set apache back to my older settings with lower start server, min servers, and max...and it started logging a message that there weren't enough servers available and I might try increasing my settings. Ok nifty. I don't sit there watching my processes 24/7, so it might have taken me a while to realize I was having these moments of high activity.

3) I've let go of my "i read this so it's true" attitude about memory. What's the point of having 600MB free memory all day long? Nothing! Now I'm giving mmcache and mysql all the memory they can use, and forcing mmcache to *only* use memory, not the disk. I'm giving apache more processes and experimenting with its new memory cache. Even when the server is seeing a lot of activity my memory usage is still pretty low. I realize that in a DOS situation it could give me grief, but that'll give me grief regardless.

4) I had stuff running that didn't need to be running. I had apache modules loaded that didn't need to be loaded. Wasteful. One thing about Plesk is that it likes to load up all the services that it *can* work with, even if your license doesn't allow you to manage them. Eight spamassassin processes running when I'm not using it? That's pretty silly, killed em. I also got rid of drweb and am going to experiment with disabling postgresql next. It's of course important to make sure that plesk or any other services aren't dependent upon these processes - so far so good.