I'm crashing everyday again, this time in the mid afternoon hours. I have installed PHP recently and am using a banner rotation script, but other than that, nothing is new on my Raq. In fact, I just closed down 2 of my CPU hogs, CGI banner scripts. Here is an example output from var/log/kernel right before the crash today. Any help in reading this would be MOST appreciated.

Mar 6 15:33:48 www kernel: Unable to load interpreter /lib/ld-linux.so.2
Mar 6 15:34:20 www last message repeated 6 times
Mar 6 15:34:29 www kernel: Unable to load interpreter /lib/ld-linux.so.2
Mar 6 15:34:45 www kernel: possible SYN flooding on port 80. Sending cookies.
Mar 6 15:34:48 www kernel: Unable to load interpreter /lib/ld-linux.so.2
Mar 6 15:48:53 www kernel: klogd 1.3-3, log source
Mar 6 15:48:53 www kernel: Serial driver version 4.27 with<4>keyboard: Timeout - AT keyboard not present?
Mar 6 15:48:53 www kernel: keyboard: Timeout - AT keyboard not present?
Mar 6 15:48:53 www kernel: no serial options enabled
Mar 6 15:48:53 www kernel: PCI: Probing PCI hardware
Mar 6 15:48:53 www kernel: Linux NET4.0 for Linux 2.2
Mar 6 15:48:53 www kernel: Based upon Swansea University Computer Society NET3.039
Mar 6 15:48:53 www kernel: NET4: Unix domain sockets 1.0 for Linux NET4.0.
Mar 6 15:48:53 www kernel: NET4: Linux TCP/IP 1.0 for NET4.0
Mar 6 15:48:53 www kernel: IP Protocols: ICMP, UDP, TCP, IGMP
Mar 6 15:48:53 www kernel: TCP: Hash tables configured (ehash 524288 bhash 65536)
Mar 6 15:48:53 www kernel: Initializing RT netlink socket
Mar 6 15:48:53 www kernel: Starting kswapd v 1.5
Mar 6 15:48:53 www kernel: Cobalt watchdog v1.4 enabled
Mar 6 15:48:53 www kernel: Cobalt I2C bus initialized
Mar 6 15:48:53 www kernel: Cobalt temperature sensor v1.3 enabled
Mar 6 15:48:53 www kernel: Serial driver version 4.27 with<4>keyboard: Timeout - AT keyboard not present?
Mar 6 15:48:53 www kernel: keyboard: Timeout - AT keyboard not present?
Mar 6 15:48:53 www kernel: no serial options enabled
Mar 6 15:48:53 www kernel: ttyS00 at 0x03f8 (irq = 4) is a 16550A
Mar 6 15:48:53 www kernel: ttyS01 at 0x02f8 (irq = 3) is a 16550A
Mar 6 15:48:53 www kernel: pty: 256 Unix98 ptys configured
Mar 6 15:48:53 www kernel: Real Time Clock Driver v1.09
Mar 6 15:48:53 www kernel: lcd: Cobalt LCD Driver v3.12^M
Mar 6 15:48:53 www kernel: serialnumber: Version 1.9 initialized. Serial number=9a0000079142a701.
Mar 6 15:48:53 www kernel: Copyright (c)1994-2000 Axent Technologies, Inc.
Mar 6 15:48:53 www kernel: Uniform Multi-Platform E-IDE driver Revision: 6.30
Mar 6 15:48:53 www kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
Mar 6 15:48:53 www kernel: ALI15X3: IDE controller on PCI bus 00 dev 78
Mar 6 15:48:53 www kernel: ALI15X3: chipset revision 193
Mar 6 15:48:53 www kernel: ALI15X3: 100%% native mode on irq 14
Mar 6 15:48:53 www kernel: ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:DMA
Mar 6 15:48:53 www kernel: ide1: BM-DMA at 0xf008-0xf00f, BIOS settings: hdc:DMA, hdd:DMA
Mar 6 15:48:53 www kernel: hda: ST320413A, ATA DISK drive
Mar 6 15:48:53 www kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
Mar 6 15:48:53 www kernel: ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Mar 6 15:48:53 www kernel: pty: 256 Unix98 ptys configured
Mar 6 15:48:53 www kernel: Real Time Clock Driver v1.09
Mar 6 15:48:53 www kernel: lcd: Cobalt LCD Driver v3.12^M
Mar 6 15:48:53 www kernel: serialnumber: Version 1.9 initialized. Serial number=9a0000079142a701.
Mar 6 15:48:53 www kernel: Copyright (c)1994-2000 Axent Technologies, Inc.
Mar 6 15:48:53 www kernel: Uniform Multi-Platform E-IDE driver Revision: 6.30
Mar 6 15:48:53 www kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
Mar 6 15:48:53 www kernel: ALI15X3: IDE controller on PCI bus 00 dev 78
Mar 6 15:48:53 www kernel: ALI15X3: chipset revision 193
Mar 6 15:48:53 www kernel: ALI15X3: 100%% native mode on irq 14
Mar 6 15:48:53 www kernel: ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:DMA
Mar 6 15:48:53 www kernel: ide1: BM-DMA at 0xf008-0xf00f, BIOS settings: hdc:DMA, hdd:DMA
Mar 6 15:48:53 www kernel: hda: ST320413A, ATA DISK drive
Mar 6 15:48:53 www kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
Mar 6 15:48:53 www kernel: ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Mar 6 15:48:53 www kernel: hda: ST320413A, 19092MB w/1024kB Cache, CHS=38792/16/63, UDMA(33)
Mar 6 15:48:53 www kernel: md driver 0.90.0 MAX_MD_DEVS=256, MAX_REAL=12
Mar 6 15:48:53 www kernel: translucent personality registered
Mar 6 15:48:53 www kernel: linear personality registered
Mar 6 15:48:53 www kernel: raid0 personality registered
Mar 6 15:48:53 www kernel: raid1 personality registered
Mar 6 15:48:53 www kernel: raid5 personality registered
Mar 6 15:48:53 www kernel: raid5: measuring checksumming speed
Mar 6 15:48:53 www kernel: raid5: MMX detected, trying high-speed MMX checksum routines
Mar 6 15:48:53 www kernel: pII_mmx : 1089.660 MB/sec
Mar 6 15:48:53 www kernel: p5_mmx : 998.220 MB/sec
Mar 6 15:48:53 www kernel: 8regs : 471.678 MB/sec
Mar 6 15:48:53 www kernel: 32regs : 294.894 MB/sec
Mar 6 15:48:53 www kernel: using fastest function: pII_mmx (1089.660 MB/sec)
Mar 6 15:48:53 www kernel: sym53c8xx: at PCI bus 0, device 14, function 0
Mar 6 15:48:53 www kernel: sym53c8xx: 53c875 detected
Mar 6 15:48:53 www kernel: sym53c875-0: rev 0x4 on pci bus 0 device 14 function 0 irq 12
Mar 6 15:48:53 www kernel: translucent personality registered
Mar 6 15:48:53 www kernel: linear personality registered
Mar 6 15:48:53 www kernel: raid0 personality registered
Mar 6 15:48:53 www kernel: raid1 personality registered
Mar 6 15:48:53 www kernel: raid5 personality registered
Mar 6 15:48:53 www kernel: raid5: measuring checksumming speed
Mar 6 15:48:53 www kernel: raid5: MMX detected, trying high-speed MMX checksum routines
Mar 6 15:48:53 www kernel: pII_mmx : 1089.660 MB/sec
Mar 6 15:48:53 www kernel: p5_mmx : 998.220 MB/sec
Mar 6 15:48:53 www kernel: 8regs : 471.678 MB/sec
Mar 6 15:48:53 www kernel: 32regs : 294.894 MB/sec
Mar 6 15:48:53 www kernel: using fastest function: pII_mmx (1089.660 MB/sec)
Mar 6 15:48:53 www kernel: sym53c8xx: 53c875 detected
Mar 6 15:48:53 www kernel: sym53c875-0: rev 0x4 on pci bus 0 device 14 function 0 irq 12
Mar 6 15:48:53 www kernel: sym53c875-0: ID 7, Fast-20, Parity Checking
Mar 6 15:48:53 www kernel: scsi0 : sym53c8xx-1.7.3a-20010304
Mar 6 15:48:53 www kernel: scsi : 1 host.
Mar 6 15:48:53 www kernel: scsi : detected total.
Mar 6 15:48:53 www kernel: md.c: sizeof(mdp_super_t) = 4096
Mar 6 15:48:53 www kernel: Partition check:
Mar 6 15:48:53 www kernel: hda: hda1 hda2 < hda5 hda6 > hda3 hda4
Mar 6 15:48:53 www kernel: autodetecting RAID arrays
Mar 6 15:48:53 www kernel: autorun ...
Mar 6 15:48:53 www kernel: ... autorun DONE.
Mar 6 15:48:53 www kernel: VFS: Mounted root (ext2 filesystem) readonly.
Mar 6 15:48:53 www kernel: Freeing unused kernel memory: 64k freed
Mar 6 15:48:53 www kernel: Adding Swap: 131532k swap-space (priority -1)
Mar 6 15:48:53 www kernel: Cobalt Networks Bandwidth Limiting Module
Mar 6 15:48:53 www kernel: Copyright (c) 1999, Cobalt Networks, Inc.
Mar 6 15:48:53 www kernel: All rights reserved
Mar 6 15:48:53 www kernel: eth0: Invalid EEPROM checksum 0xd326, check settings before activating this device$
Mar 6 15:48:53 www kernel: scsi : 1 host.
Mar 6 15:48:53 www kernel: scsi : detected total.
Mar 6 15:48:53 www kernel: md.c: sizeof(mdp_super_t) = 4096
Mar 6 15:48:53 www kernel: Partition check:
Mar 6 15:48:53 www kernel: hda: hda1 hda2 < hda5 hda6 > hda3 hda4
Mar 6 15:48:53 www kernel: autodetecting RAID arrays
Mar 6 15:48:53 www kernel: autorun ...
Mar 6 15:48:53 www kernel: ... autorun DONE.
Mar 6 15:48:53 www kernel: VFS: Mounted root (ext2 filesystem) readonly.
Mar 6 15:48:53 www kernel: Freeing unused kernel memory: 64k freed
Mar 6 15:48:53 www kernel: Adding Swap: 131532k swap-space (priority -1)
Mar 6 15:48:53 www kernel: Cobalt Networks Bandwidth Limiting Module
Mar 6 15:48:53 www kernel: Copyright (c) 1999, Cobalt Networks, Inc.
Mar 6 15:48:53 www kernel: All rights reserved
Mar 6 15:48:53 www kernel: eth0: Invalid EEPROM checksum 0xd326, check settings before activating this device$
Mar 6 15:48:53 www kernel: eth0: Intel PCI EtherExpress Pro100 82559ER, 00:10:E0:02:46:C0, I/O at 0x6200, IRQ$
Mar 6 15:48:53 www kernel: Board assembly 000000-000, Physical connectors present:
Mar 6 15:48:53 www kernel: Primary interface chip None PHY #0.
Mar 6 15:48:53 www kernel: General self-test: passed.
Mar 6 15:48:53 www kernel: Serial sub-system self-test: passed.
Mar 6 15:48:53 www kernel: Internal registers self-test: passed.
Mar 6 15:48:53 www kernel: ROM checksum self-test: passed (0xdbd8681d).
Mar 6 15:48:53 www kernel: Receiver lock-up workaround activated.
Mar 6 15:48:53 www kernel: eth1: Invalid EEPROM checksum 0xd226, check settings before activating this device$
Mar 6 15:48:53 www kernel: eth1: Intel PCI EtherExpress Pro100 82559ER, 00:10:E0:02:46:BF, I/O at 0x6240, IRQ$
Mar 6 15:48:53 www kernel: Board assembly 000000-000, Physical connectors present:
Mar 6 15:48:53 www kernel: Primary interface chip None PHY #0.
Mar 6 15:48:53 www kernel: General self-test: passed.
Mar 6 15:48:53 www kernel: Serial sub-system self-test: passed.
Mar 6 15:48:53 www kernel: Internal registers self-test: passed.
Mar 6 15:48:53 www kernel: ROM checksum self-test: passed (0xdbd8681d).
Mar 6 15:48:53 www kernel: Receiver lock-up workaround activated.

Here is part of my error log right before the cradh, if it helps shed some light on my problem:

[Wed Mar 6 15:15:59 2002] [error] [client 165.199.3.42] (11)Resource temporarily unavailable: couldn't spawn $
Out of memory!
Callback called exit at /dev/null line 4.
[Wed Mar 6 15:16:47 2002] [error] [client 198.26.119.86] (11)Resource temporarily unavailable: couldn't spawn$
[Wed Mar 6 15:16:48 2002] [error] [client 198.26.118.36] (11)Resource temporarily unavailable: couldn't spawn$
[Wed Mar 6 15:16:52 2002] [error] [client 198.26.118.36] (11)Resource temporarily unavailable: couldn't spawn$
[Wed Mar 6 15:16:54 2002] [error] [client 213.255.18.135] (11)Resource temporarily unavailable: couldn't spaw$
[Wed Mar 6 15:16:57 2002] [error] [client 62.252.64.6] (11)Resource temporarily unavailable: couldn't spawn c$
FATAL: erealloc(): Unable to allocate 122880 bytes
[Wed Mar 6 15:17:00 2002] [error] [client 200.34.91.243] (11)Resource temporarily unavailable: couldn't spawn$
[Wed Mar 6 15:17:01 2002] [error] [client 64.12.103.164] (11)Resource temporarily unavailable: couldn't spawn$
FATAL: emalloc(): Unable to allocate 267 bytes
[Wed Mar 6 15:17:10 2002] [error] [client 209.240.222.131] (11)Resource temporarily unavailable: couldn't spa$
[Wed Mar 6 15:17:10 2002] [error] [client 200.34.91.243] (11)Resource temporarily unavailable: couldn't spawn$
Out of memory!
Callback called exit.
FATAL: erealloc(): Unable to allocate 7680 bytes
[Wed Mar 6 15:17:21 2002] [error] [client 142.165.18.227] (11)Resource temporarily unavailable: couldn't spaw$
Out of memory!
Callback called exit.
[Wed Mar 6 15:17:25 2002] [error] [client 205.188.192.49] (11)Resource temporarily unavailable: couldn't spaw$
Out of memory!
Callback called exit.
[Wed Mar 6 15:17:27 2002] [error] [client 205.188.192.49] (11)Resource temporarily unavailable: couldn't spaw$
FATAL: emalloc(): Unable to allocate 165 bytes
Ouch! malloc failed in malloc_block()
Ouch! malloc failed in malloc_block()
Ouch! malloc failed in malloc_block()
[Wed Mar 6 15:18:12 2002] [error] [client 209.107.140.218] (11)Resource temporarily unavailable: couldn't spa$
Ouch! malloc failed in malloc_block()
Ouch! malloc failed in malloc_block()
[[Wed Mar 6 15:18:27 2002] [error] [client 62.31.224.2] (11)Resource temporarily unavailable: couldn't spawn c$
Ouch! malloc failed in malloc_block()
Ouch! malloc failed in malloc_block()
Ouch! malloc failed in malloc_block()
Ouch! malloc failed in malloc_block()
Ouch! malloc failed in malloc_block()
[Wed Mar 6 15:18:37 2002] [error] (32)Broken pipe: accept: (client socket)
[Wed Mar 6 15:18:40 2002] [error] [client 198.26.118.37] (11)Resource temporarily unavailable: couldn't spawn$
Out of memory!
Callback called exit.
[Wed Mar 6 15:18:42 2002] [error] [client 209.107.140.218] (11)Resource temporarily unavailable: couldn't spa$
Out of memory!
Callback called exit.
[Wed Mar 6 15:18:43 2002] [error] [client 62.252.64.6] (11)Resource temporarily unavailable: couldn't spawn c$
Out of memory!
Callback called exit.
[Wed Mar 6 15:18:45 2002] [error] [client 216.35.116.23] (11)Resource temporarily unavailable: couldn't spawn$
Out of memory!
Callback called exit.
[Wed Mar 6 15:19:06 2002] [error] [client 62.31.224.2] (11)Resource temporarily unavailable: couldn't spawn c$
Out of memory!
[Wed Mar 6 15:20:07 2002] [error] (32)Broken pipe: accept: (client socket)
[Wed Mar 6 15:20:10 2002] [error] [client 213.169.128.40] (11)Resource temporarily unavailable: couldn't spaw$
[Wed Mar 6 15:20:20 2002] [error] [client 63.206.93.57] (11)Resource temporarily unavailable: couldn't spawn $
[Wed Mar 6 15:20:46 2002] [error] [client 213.234.70.76] (11)Resource temporarily unavailable: couldn't spawn$
Out of memory!
Callback called exit.

is there any type of backups running or any cron jobs at this time? also what does /var/log/messages say?

Originally posted by jahsh
is there any type of backups running or any cron jobs at this time? also what does /var/log/messages say?

No cron jobs at that time and here is what was happening in /var/log/messages at the same time:

5 15:11:45 www telnetd[30157]: ttloop: peer died: EOF
Mar 5 15:15:01 www proftpd[30725]: www.fredfarm.com (localhost[127.0.0.1]) - FTP session opened.
Mar 5 15:15:01 www proftpd[30725]: www.fredfarm.com (localhost[127.0.0.1]) - FTP session closed.
Mar 5 15:15:04 www proftpd[30695]: 216.40.195.80 (209-79-245-206.ded.pacbell.net[209.79.245.206]) - FTP sessi$
Mar 5 15:15:04 www telnetd[30732]: ttloop: read: Broken pipe
Mar 5 15:17:16 www telnetd[30868]: ttloop: peer died: EOF
Mar 5 15:20:26 www proftpd[30695]: 216.40.195.80 (209-79-245-206.ded.pacbell.net[209.79.245.206]) - FTP no tr$
Mar 5 15:29:51 www telnetd[619]: ttloop: peer died: EOF
Mar 5 15:29:52 www proftpd[618]: 216.40.195.80 (207.218.223.135[207.218.223.135]) - FTP session opened.
Mar 5 15:29:52 www proftpd[618]: 216.40.195.80 (207.218.223.135[207.218.223.135]) - FTP session closed.
Mar 5 15:30:03 www proftpd[663]: www.fredfarm.com (localhost[127.0.0.1]) - FTP session opened.
Mar 5 15:30:03 www proftpd[663]: www.fredfarm.com (localhost[127.0.0.1]) - FTP session closed.
Mar 5 15:30:07 www telnetd[669]: ttloop: read: Broken pipe
Mar 5 15:31:48 www proftpd[956]: 216.40.195.80 (207.218.223.135[207.218.223.135]) - FTP session opened.
Mar 5 15:31:48 www proftpd[956]: 216.40.195.80 (207.218.223.135[207.218.223.135]) - FTP session closed.
Mar 5 15:31:53 www telnetd[957]: ttloop: peer died: EOF
Mar 5 15:33:47 www proftpd[1216]: 216.40.195.80 (207.218.223.135[207.218.223.135]) - FTP session opened.
Mar 5 15:33:47 www proftpd[1216]: 216.40.195.80 (207.218.223.135[207.218.223.135]) - FTP session closed.
Mar 5 15:33:47 www telnetd[1217]: ttloop: peer died: EOF
Mar 5 15:45:00 www proftpd[2432]: www.fredfarm.com (localhost[127.0.0.1]) - FTP session opened.
Mar 5 15:45:00 www proftpd[2432]: www.fredfarm.com (localhost[127.0.0.1]) - FTP session closed.
Mar 5 15:45:03 www telnetd[2442]: ttloop: read: Broken pipe
Mar 5 15:45:45 www proftpd[2542]: 216.40.195.80 (AC884393.ipt.aol.com[172.136.67.147]) - FTP session opened.
Mar 5 15:49:31 www proftpd[2542]: 216.40.195.80 (AC884393.ipt.aol.com[172.136.67.147]) - FTP session closed.

is there any weird processes running when you run the command top or ps?