Results 1 to 11 of 11
  1. #1
    Join Date
    Dec 2001
    Posts
    160

    Server is acting weird tonight

    For some reason my server is acting very weird with high server load and it's 12 midnight. I never have this problem at this time.

    00:30:16 up 29 min, 1 user, load average: 6.38, 6.56, 5.97
    137 processes: 134 sleeping, 1 running, 2 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 16.9% 0.0% 5.8% 0.0% 0.4% 76.7% 0.0%
    Mem: 1546620k av, 930748k used, 615872k free, 0k shrd, 41100k buff
    777180k active, 114316k inactive
    Swap: 2097136k av, 0k used, 2097136k free 253816k cached

    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    1580 nobody 15 0 19684 12M 11972 S 14.1 0.8 0:26 0 /usr/local/apache/bin/httpd -DSSL
    2559 root 16 0 4144 1192 3812 R 3.0 0.0 0:36 0 top
    1571 nobody 15 0 20724 13M 11972 S 1.3 0.8 0:22 0 /usr/local/apache/bin/httpd -DSSL
    1850 mysql 15 0 161M 35M 3756 D 1.3 2.3 0:02 0 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --us
    1570 nobody 16 0 19724 12M 11972 S 0.8 0.8 0:16 0 /usr/local/apache/bin/httpd -DSSL
    2386 mysql 15 0 161M 35M 3756 S 0.8 2.3 0:03 0 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --us
    1579 nobody 15 0 20560 13M 11972 S 0.4 0.8 0:27 0 /usr/local/apache/bin/httpd -DSSL
    1583 nobody 15 0 20788 13M 11972 S 0.4 0.8 0:27 0 /usr/local/apache/bin/httpd -DSSL
    1587 nobody 15 0 20708 13M 11972 S 0.4 0.8 0:25 0 /usr/local/apache/bin/httpd -DSSL
    2503 nobody 15 0 20728 13M 11972 S 0.4 0.8 0:17 0 /usr/local/apache/bin/httpd -DSSL
    2511 root 16 0 7028 2160 6388 S 0.4 0.1 0:00 0 sshd: root@pts/0
    2560 nobody 15 0 20516 12M 11972 S 0.4 0.8 0:19 0 /usr/local/apache/bin/httpd -DSSL
    2625 mailnull 16 0 113M 109M 6120 S 0.4 7.2 0:14 0 /usr/bin/perl -I/usr/mailscanner/lib /usr/mailscanner/bin/
    1 root 16 0 1504 504 1352 S 0.0 0.0 0:01 0 init [3]
    2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
    3 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
    4 root 5 -10 0 0 0 SW< 0.0 0.0 0:00 0 events/0
    5 root 15 -10 0 0 0 SW< 0.0 0.0 0:00 0 khelper
    6 root 14 -10 0 0 0 SW< 0.0 0.0 0:00 0 kacpid
    21 root 5 -10 0 0 0 SW< 0.0 0.0 0:00 0 kblockd/0
    32 root 20 0 0 0 0 SW 0.0 0.0 0:00 0 pdflush
    33 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 pdflush
    35 root 10 -10 0 0 0 SW< 0.0 0.0 0:00 0 aio/0
    34 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 kswapd0
    36 root 10 -10 0 0 0 SW< 0.0 0.0 0:00 0 xfslogd/0
    37 root 5 -10 0 0 0 SW< 0.0 0.0 0:00 0 xfsdatad/0
    38 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 xfsbufd
    131 root 6 -10 0 0 0 SW< 0.0 0.0 0:00 0 ata/0
    134 root 16 0 0 0 0 SW 0.0 0.0 0:00 0 kseriod
    165 root 15 0 0 0 0 DW 0.0 0.0 0:00 0 kjournald
    467 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
    944 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    1344 root 15 0 1568 572 1400 S 0.0 0.0 0:00 0 syslogd -m 0
    1348 root 15 0 1512 464 1348 S 0.0 0.0 0:00 0 klogd -x
    1394 root 16 0 3640 1564 3420 S 0.0 0.1 0:00 0 /usr/sbin/sshd


    Any ideas why?

  2. #2
    Join Date
    Dec 2001
    Posts
    160
    It's midnight and there's barely anyone around and I'm getting hit pretty bad for some reason.

    00:33:34 up 33 min, 1 user, load average: 21.58, 9.98, 7.16
    150 processes: 115 sleeping, 34 running, 1 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 94.8% 0.0% 3.2% 0.3% 1.6% 0.0% 0.0%
    Mem: 1546620k av, 1155812k used, 390808k free, 0k shrd, 41828k buff
    988952k active, 125508k inactive
    Swap: 2097136k av, 0k used, 2097136k free 284844k cached

    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    1575 nobody 17 0 23888 16M 11972 R 4.1 1.0 0:29 0 /usr/local/apache/bin/httpd -DSSL
    1576 nobody 17 0 23780 16M 11972 R 4.1 1.0 0:33 0 /usr/local/apache/bin/httpd -DSSL
    1571 nobody 17 0 23708 15M 11972 R 3.8 1.0 0:27 0 /usr/local/apache/bin/httpd -DSSL
    1579 nobody 17 0 23680 16M 11972 R 3.8 1.0 0:30 0 /usr/local/apache/bin/httpd -DSSL
    1581 nobody 17 0 23776 16M 11972 R 3.8 1.0 0:30 0 /usr/local/apache/bin/httpd -DSSL
    1582 nobody 17 0 23752 16M 11972 R 3.8 1.0 0:39 0 /usr/local/apache/bin/httpd -DSSL
    1583 nobody 17 0 23636 16M 11972 R 3.8 1.0 0:30 0 /usr/local/apache/bin/httpd -DSSL
    1584 nobody 17 0 23748 16M 11972 R 3.8 1.0 0:38 0 /usr/local/apache/bin/httpd -DSSL
    1588 nobody 17 0 23936 16M 11972 R 3.8 1.0 0:27 0 /usr/local/apache/bin/httpd -DSSL
    1589 nobody 17 0 23660 16M 11972 R 3.8 1.0 0:32 0 /usr/local/apache/bin/httpd -DSSL
    2948 nobody 17 0 23708 15M 11972 R 3.8 1.0 0:05 0 /usr/local/apache/bin/httpd -DSSL
    3043 nobody 17 0 23764 16M 11972 R 3.8 1.0 0:05 0 /usr/local/apache/bin/httpd -DSSL
    3119 nobody 17 0 23664 15M 11972 R 3.8 1.0 0:01 0 /usr/local/apache/bin/httpd -DSSL
    2560 nobody 17 0 23632 15M 11972 R 3.5 1.0 0:22 0 /usr/local/apache/bin/httpd -DSSL
    3117 nobody 17 0 23588 15M 11972 R 3.5 1.0 0:01 0 /usr/local/apache/bin/httpd -DSSL
    3122 nobody 18 0 19368 11M 11972 R 3.5 0.7 0:00 0 /usr/local/apache/bin/httpd -DSSL
    1570 nobody 17 0 23656 16M 11972 R 3.2 1.0 0:19 0 /usr/local/apache/bin/httpd -DSSL
    1572 nobody 17 0 23684 16M 11972 R 3.2 1.0 0:35 0 /usr/local/apache/bin/httpd -DSSL
    1580 nobody 17 0 23748 16M 11972 R 3.2 1.0 0:31 0 /usr/local/apache/bin/httpd -DSSL
    1587 nobody 17 0 23688 16M 11972 R 3.2 1.0 0:27 0 /usr/local/apache/bin/httpd -DSSL
    2503 nobody 17 0 23712 16M 11972 R 3.2 1.0 0:19 0 /usr/local/apache/bin/httpd -DSSL
    2897 nobody 17 0 23768 16M 11972 R 3.2 1.0 0:14 0 /usr/local/apache/bin/httpd -DSSL
    3105 nobody 17 0 23676 15M 11972 R 3.2 1.0 0:02 0 /usr/local/apache/bin/httpd -DSSL
    3115 nobody 17 0 23584 15M 11972 R 3.2 1.0 0:02 0 /usr/local/apache/bin/httpd -DSSL
    3111 nobody 17 0 23596 15M 11972 R 2.8 1.0 0:01 0 /usr/local/apache/bin/httpd -DSSL
    3116 nobody 16 0 18960 11M 11972 S 1.6 0.7 0:01 0 /usr/local/apache/bin/httpd -DSSL
    2559 root 17 0 4144 1192 3812 R 1.2 0.0 0:41 0 top
    3131 nobody 15 0 17012 9696 11976 S 1.2 0.6 0:00 0 /usr/local/apache/bin/httpd -DSSL
    3129 nobody 15 0 16644 9316 11972 S 0.9 0.6 0:00 0 /usr/local/apache/bin/httpd -DSSL
    3130 nobody 15 0 17032 9676 11988 S 0.6 0.6 0:00 0 /usr/local/apache/bin/httpd -DSSL
    1573 nobody 15 0 19820 12M 11976 S 0.3 0.8 0:23 0 /usr/local/apache/bin/httpd -DSSL
    1578 nobody 15 0 21896 14M 11984 S 0.3 0.9 0:30 0 /usr/local/apache/bin/httpd -DSSL
    1712 mysql 16 0 161M 35M 3756 S 0.3 2.3 0:00 0 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --us
    2511 root 16 0 7028 2160 6388 S 0.3 0.1 0:00 0 sshd: root@pts/0
    2625 mailnull 16 0 113M 109M 6120 S 0.3 7.2 0:14 0 /usr/bin/perl -I/usr/mailscanner/lib /usr/mai

  3. #3
    Join Date
    Apr 2005
    Posts
    535
    you have a lot of ppl connecting to a secure site for some reason ?

  4. #4
    Join Date
    May 2005
    Location
    New York
    Posts
    736
    try temporaily turning off keep-alives and reducing the max spare servers/max servers, and restart apache, see if that helps. also you can put limits to apache's memory and cpu usage i believe
    Perigee Global Corporation
    Design, Development and Hosting Solutions
    Dedicated Servers, CDN, Hosted E-Mail, Web Hosting, VPS & Cloud Servers
    1.212.400-7632 www.perigeeglobal.com

  5. #5
    Join Date
    Jul 2002
    Posts
    1,441
    try

    netstat | grep SYN

    see if you get any results in large amounts
    Synergy Blue LLC
    SonataWeb.net | SynergyBlue.com
    USA should so something about: http://www.brillig.com/debt_clock/

  6. #6
    Join Date
    Dec 2001
    Posts
    160
    Originally posted by jNive
    try temporaily turning off keep-alives and reducing the max spare servers/max servers, and restart apache, see if that helps. also you can put limits to apache's memory and cpu usage i believe
    Server and mysql has already been optimize. Usually at this time of hour CPU usage is below 1 or just a bit above 1.

    right now is bouncing in an odd way. One minute it's around 1-2 CPU then next minute it jumps to 10-20 CPU.

    This is normal:

    03:01:05 up 1:38, 1 user, load average: 1.85, 4.86, 5.90
    147 processes: 143 sleeping, 1 running, 2 zombie, 1 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 77.4% 0.0% 7.6% 0.0% 0.9% 0.0% 13.9%
    Mem: 1546620k av, 1440308k used, 106312k free, 0k shrd, 43212k buff
    1108220k active, 283944k inactive
    Swap: 2097136k av, 0k used, 2097136k free 628628k cached

    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    3184 nobody 17 0 18544 12M 9900 S 65.6 0.8 0:46 0 /usr/local/apache/bin/httpd -DSSL
    3175 nobody 15 0 17720 12M 9900 S 9.1 0.8 0:51 0 /usr/local/apache/bin/httpd -DSSL
    4521 root 17 0 4144 1192 3812 R 3.8 0.0 0:09 0 top
    4429 nobody 15 0 16628 11M 9900 S 0.9 0.7 0:07 0 /usr/local/apache/bin/httpd -DSSL
    4571 nobody 15 0 16980 11M 9900 S 0.9 0.7 0:00 0 /usr/local/apache/bin/httpd -DSSL
    33 root 15 0 0 0 0 SW 0.4 0.0 0:00 0 pdflush
    3171 nobody 15 0 21212 15M 9900 S 0.4 1.0 0:25 0 /usr/local/apache/bin/httpd -DSSL
    3179 nobody 15 0 18320 12M 9900 S 0.4 0.8 0:48 0 /usr/local/apache/bin/httpd -DSSL
    3524 nobody 15 0 18304 12M 9900 S 0.4 0.8 0:40 0 /usr/local/apache/bin/httpd -DSSL
    3649 nobody 15 0 17472 11M 9900 S 0.4 0.7 0:22 0 /usr/local/apache/bin/httpd -DSSL
    4555 nobody 16 0 19300 13M 9900 S 0.4 0.9 0:04 0 /usr/local/apache/bin/httpd -DSSL
    1 root 16 0 1504 504 1352 S 0.0 0.0 0:01 0 init [3]
    2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
    3 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
    4 root 5 -10 0 0 0 SW< 0.0 0.0 0:00 0 events/0
    5 root 13 -10 0 0 0 SW< 0.0 0.0 0:00 0 khelper
    6 root 14 -10 0 0 0 SW< 0.0 0.0 0:00 0 kacpid
    21 root 5 -10 0 0 0 SW< 0.0 0.0 0:00 0 kblockd/0
    32 root 20 0 0 0 0 SW 0.0 0.0 0:00 0 pdflush
    35 root 10 -10 0 0 0 SW< 0.0 0.0 0:00 0 aio/0
    34 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 kswapd0
    36 root 10 -10 0 0 0 SW< 0.0 0.0 0:00 0 xfslogd/0
    37 root 5 -10 0 0 0 SW< 0.0 0.0 0:00 0 xfsdatad/0
    38 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 xfsbufd
    131 root 6 -10 0 0 0 SW< 0.0 0.0 0:00 0 ata/0
    134 root 16 0 0 0 0 SW 0.0 0.0 0:00 0 kseriod
    165 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 kjournald
    467 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
    944 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    1344 root 15 0 1568 572 1400 S 0.0 0.0 0:00 0 syslogd -m 0
    1348 root 15 0 1512 464 1348 S 0.0 0.0 0:00 0 klogd -x
    1394 root 17 0 3640 1564 3420 S 0.0 0.1 0:00 0 /usr/sbin/sshd
    1406 root 16 0 2128 936 1788 S 0.0 0.0 0:00 0 xinetd -stayalive -pidfile /var/run/xinetd.pid
    1425 root 15 0 6104 2760 4704 S 0.0 0.1 0:00 0 chkservd
    1479 mailnull 16 0 6604 1876 6144 S 0.0 0.1 0:00 0 /usr/sbin/exim -bd -q60m

    Then a few seconds later like 30 seconds it jumps to this:

    03:01:59 up 1:39, 1 user, load average: 10.04, 6.16, 6.26
    159 processes: 129 sleeping, 27 running, 2 zombie, 1 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 94.1% 0.0% 4.3% 0.3% 1.2% 0.0% 0.0%
    Mem: 1546620k av, 1522068k used, 24552k free, 0k shrd, 43316k buff
    1189680k active, 283648k inactive
    Swap: 2097136k av, 0k used, 2097136k free 630292k cached

    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    4343 nobody 16 0 21540 15M 9900 R 5.5 1.0 0:07 0 /usr/local/apache/bin/httpd -DSSL
    1924 mysql 16 0 166M 39M 3756 R 5.2 2.6 0:12 0 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --us
    3186 nobody 16 0 21588 15M 9900 R 5.2 1.0 0:47 0 /usr/local/apache/bin/httpd -DSSL
    3164 nobody 16 0 21572 15M 9900 R 4.9 1.0 0:39 0 /usr/local/apache/bin/httpd -DSSL
    3166 nobody 16 0 21616 15M 9900 R 4.9 1.0 0:46 0 /usr/local/apache/bin/httpd -DSSL
    3172 nobody 16 0 21544 15M 9900 R 4.9 1.0 0:26 0 /usr/local/apache/bin/httpd -DSSL
    3177 nobody 16 0 21680 15M 9900 R 4.9 1.0 0:44 0 /usr/local/apache/bin/httpd -DSSL
    3181 nobody 16 0 21584 15M 9900 R 4.9 1.0 0:30 0 /usr/local/apache/bin/httpd -DSSL
    3524 nobody 16 0 21556 15M 9900 R 4.9 1.0 0:40 0 /usr/local/apache/bin/httpd -DSSL
    4327 nobody 16 0 21612 15M 9900 R 4.9 1.0 0:09 0 /usr/local/apache/bin/httpd -DSSL
    4430 nobody 16 0 21568 15M 9900 R 4.9 1.0 0:10 0 /usr/local/apache/bin/httpd -DSSL
    4547 nobody 16 0 21488 15M 9900 R 4.9 1.0 0:00 0 /usr/local/apache/bin/httpd -DSSL
    4555 nobody 16 0 21472 15M 9900 R 4.9 1.0 0:05 0 /usr/local/apache/bin/httpd -DSSL
    3176 nobody 16 0 21572 15M 9900 R 4.6 1.0 0:36 0 /usr/local/apache/bin/httpd -DSSL
    3178 nobody 16 0 21552 15M 9900 R 4.6 1.0 0:40 0 /usr/local/apache/bin/httpd -DSSL
    3649 nobody 16 0 21540 15M 9900 R 4.6 1.0 0:23 0 /usr/local/apache/bin/httpd -DSSL
    3173 nobody 16 0 21608 15M 9900 R 4.3 1.0 0:37 0 /usr/local/apache/bin/httpd -DSSL
    1862 mysql 15 0 166M 39M 3756 S 2.7 2.6 0:12 0 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --us
    3185 nobody 16 0 20604 15M 9900 R 2.1 0.9 0:47 0 /usr/local/apache/bin/httpd -DSSL
    4521 root 16 0 4144 1196 3812 R 2.1 0.0 0:10 0 top
    3175 nobody 16 0 17724 12M 9900 R 1.8 0.8 0:51 0 /usr/local/apache/bin/httpd -DSSL
    3187 nobody 15 0 17452 11M 9900 S 0.6 0.7 0:41 0 /usr/local/apache/bin/httpd -DSSL
    4769 nobody 15 0 14916 9532 9900 S 0.6 0.6 0:00 0 /usr/local/apache/bin/httpd -DSSL
    4770 nobody 15 0 11452 5944 9900 S 0.6 0.3 0:00 0 /usr/local/apache/bin/httpd -DSSL
    4771 nobody 15 0 11392 5860 9900 S 0.6 0.3 0:00 0 /usr/local/apache/bin/httpd -DSSL
    2442 mailnull 15 0 113M 109M 6120 S 0.3 7.2 0:21 0 /usr/bin/perl -I/usr/mailscanner/lib /usr/mailscanner/bin/
    2444 mailnull 15 0 113M 109M 6120 S 0.3 7.2 0:20 0 /usr/bin/perl -I/usr/mailscanner/lib /usr/mailscanner/bin/
    3171 nobody 16 0 21212 15M 9900 S 0.3 1.0 0:25 0 /usr/local/apache/bin/httpd -DSSL
    3183 nobody 15 0 17488 11M 9900 S 0.3 0.7 0:33 0 /usr/local/apache/bin/httpd -DSSL
    4429 nobody 15 0 18300 12M 9900 S 0.3 0.8 0:07 0 /usr/local/apache/bin/httpd -DSSL
    4774 nobody 15 0 11264 5368 9900 S 0.3 0.3 0:00 0 /usr/local/apache/bin/httpd -DSSL
    1 root 15 0 1504 504 1352 S 0.0 0.0 0:01 0 init [3]
    2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
    3 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
    4 root 5 -10 0 0 0 SW< 0.0 0.0 0:00 0 events/0

    It keeps going back and forth like that all night today. What the hell is going on?

  7. #7
    Join Date
    Sep 2004
    Location
    Flint, Michigan
    Posts
    5,765
    It might help to install mod-status on apache and view the status page when the load gets high like that.
    Mike from Zoodia.com
    Professional web design and development services.
    In need of a fresh hosting design? See what premade designs we have in stock!
    Web design tips, tricks, and more at MichaelPruitt.com

  8. #8
    Maybe your web site was hitting by a spider.
    LiteSpeed Web Server by http://www.litespeedtech.com
    Best PHP and Ruby On Rails hosting platform,
    Completely Apache interchangeable
    Compatible with all hosting control panels.

  9. #9
    Join Date
    May 2005
    Posts
    61
    Hmm, first of all I would recommend mailscanner in a busy server. It will simply take load. Second suggestion is to try to install mod_dos also write some smart scritps to find the no of connections to apache using netstat. Also check for domlogs size. You may find some users with high domlogs size menas highest no of hits.

  10. #10
    Join Date
    Dec 2001
    Posts
    160
    I've been able to find the cause of this. Well not actually the cause but where the high CPU came from.

    The high CPU usage came directly from the forum. As stated I haven't touch the forum or made any changes to the forum. Http request and CPU usage has been normal for months until a couple of nights ago.

    What I did to get CPU back to normal temporary was turn the forum from a public viewing forum to one that forces everyone to register or log in to view the forum. If you are a guest you won't be able to view the forum unless you log in.
    This brought CPU back to normal.

    What's the SSH command that you can view the latest visitors, IPs and the page that they are requesting in SSH? I remember a long time ago you can view the latest visitors and what page is being requested through SSH.

    Also is this normal?

    32428 mailnull 16 0 113M 109M 6120 S 0.4 7.2 0:27 0 /usr/bin/perl -I/usr/mailscanner/lib /usr/mailscanner/bin/
    32489 mailnull 16 0 113M 109M 6120 S 0.4 7.2 0:26 0 /usr/bin/perl -I/usr/mailscanner/lib /usr/mailscanner/bin/

    I notice that mailscanner seem to be running non stop in top.

  11. #11
    Join Date
    Mar 2003
    Location
    Sacramento, CA
    Posts
    70
    Your forum was probably being indexed by one of the many search engines. If you want part of your site to be listed I would suggest making a small or less used forum publicly available and make all other require a log in as you've done.

    I had a lot of issues with the same thing when a member posted a picture on one of my forums and we got a ton of people googling for the title and coming from there.

    After that google started hammering my server for a few weeks. Now I make sure that he public forum has very little within it to index.
    Jacob Squires , CISSP

    security assesment, network analysis

Related Posts from theWHIR.com

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •