Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : 500 internal server error

[vigge_sWe]

So, I have no ideas left now...

I run a proxy. First, I was on shared hosting. Worked for 2 months. then, bam! 500 internal server error.

I moved to a smaller VPS with 128 MB RAM. Worked for 30 min, then 500 internal server error again...

So now I moved to a larger VPS with 512 MB RAM, 1 GB burstable. Worked for 2 days, and now the 500's is back...

Rebooting the VPS usually fixes it, but I can not be sure I can continue rebooting it, as I expect it getting worse later.

It usually starts with lighttpd goes insane. One minute it's using 30 mb RAM, ok that's good. Then suddenly it uses 300 mb. then 500 mb. Then the 500 internal server error occur.

So any ideas of how I should configure this so it would work in a better way?

This is filled in the error log:
2009-06-02 13:21:16: (mod_indexfile.c.182) file not found ... or so: Too many open files / -> /home/admin/datashelf.net/

OS: centOS 5

[bigks]

Most typical cause is out of resources. I have seen this more often on lighttpd than on apache and it's normal cause is poor mysql and lighttpd optimization.

Can you wait until this error occurs then log into via ssh and post your top command results. Just a quick thought, when hard drives get loaded up by other users it can spur a high load average which does affect you.

[The Universes]

Are you on OpenVZ or Xen? If your on OpenVZ, please post your user_beancounters file, run this in SSH:

cat /proc/user_beancounters

My guess is your running out of memory (PHP fcgi?) or there is a misconfiguration somewhere.

[vigge_sWe]

Originally Posted by The Universes

Are you on OpenVZ or Xen? If your on OpenVZ, please post your user_beancounters file, run this in SSH:


My guess is your running out of memory (PHP fcgi?) or there is a misconfiguration somewhere.

openVZ

[root@viggeswe /]# cat /proc/user_beancounters
Version: 2.5
uid resource held maxheld barrier limit failcnt
17170: kmemsize 5827970 17445527 2147483646 2147483646 0
lockedpages 0 0 999999 999999 0
privvmpages 15554 513017 262144 262144 538
shmpages 171 672 131072 131072 0
dummy 0 0 0 0 0
numproc 46 98 999999 999999 0
physpages 7851 195144 0 2147483647 0
vmguarpages 0 0 131072 2147483647 0
oomguarpages 7851 195144 131072 2147483647 0
numtcpsock 73 171 7999992 7999992 0
numflock 3 10 999999 999999 0
numpty 1 2 500000 500000 0
numsiginfo 0 18 999999 999999 0
tcpsndbuf 1676064 5751168 107374592 289400832 0
tcprcvbuf 2593280 19875552 107374592 289400832 0
othersockbuf 13920 56336 107374592 289400832 0
dgramrcvbuf 0 18224 107374592 289400832 0
numothersock 10 43 7999992 7999992 0
dcachesize 0 0 2147483646 2147483646 0
numfile 840 6741 23999976 23999976 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 14 14 999999 999999 0

I've switched to apache to check if it will work better, but I doubt it

[bigks]

If you are on lighttpd and switch to apache it will cause a high load on your vps making it unstable (granted we are assuming lighttpd was eating up too many resources as it was). Can you post your top command results?

[The Universes]

Your running out of memory and that is the most likely reason your getting those 500 errors:

uid resource held maxheld barrier limit failcnt
privvmpages 15554 513017 262144 262144 538

[vigge_sWe]

Originally Posted by The Universes

Your running out of memory and that is the most likely reason your getting those 500 errors:

but why would I run out of memory? I got 512 mb guaranteed, 1 gb burstable.

top - 13:54:00 up 32 min, 0 users, load average: 0.04, 0.12, 0.10
Tasks: 68 total, 1 running, 67 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.1%sy, 0.0%ni, 99.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1048576k total, 129396k used, 919180k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1599 apache 15 0 25288 7988 3820 S 0 0.8 0:00.20 httpd
1600 apache 15 0 25788 7864 3688 S 0 0.7 0:00.18 httpd
1761 apache 15 0 25292 7912 3848 S 0 0.8 0:00.24 httpd
11704 apache 15 0 25292 8112 3804 S 0 0.8 0:00.26 httpd
19805 apache 15 0 25808 7856 3680 S 0 0.7 0:00.32 httpd
1 root 15 0 2056 656 564 S 0 0.1 0:00.06 init
1475 apache 15 0 25308 7668 3660 S 0 0.7 0:00.19 httpd
1510 apache 15 0 25280 7648 3672 S 0 0.7 0:00.10 httpd
1511 apache 15 0 25076 7896 3804 S 0 0.8 0:00.19 httpd
1602 apache 15 0 25260 7504 3544 S 0 0.7 0:00.06 httpd
1758 apache 18 0 25292 7616 3640 S 0 0.7 0:00.27 httpd
1773 apache 15 0 25808 7860 3688 S 0 0.7 0:00.24 httpd
1778 apache 15 0 25500 7700 3568 S 0 0.7 0:00.15 httpd
1779 apache 18 0 25292 7656 3660 S 0 0.7 0:00.17 httpd
5154 apache 15 0 25500 7700 3564 S 0 0.7 0:00.09 httpd
5937 apache 15 0 25500 7700 3564 S 0 0.7 0:00.12 httpd
11736 apache 15 0 25808 7824 3668 S 0 0.7 0:00.19 httpd
11749 apache 15 0 25844 8148 3804 S 0 0.8 0:00.16 httpd
12072 root 15 0 1716 624 524 S 0 0.1 0:00.00 syslogd
12088 dbus 18 0 2728 460 324 S 0 0.0 0:00.00 dbus-daemon
12099 root 18 0 5216 916 608 S 0 0.1 0:00.00 sshd
12111 root 16 0 2712 908 732 S 0 0.1 0:00.00 xinetd
12128 tinydns 18 0 1640 308 252 S 0 0.0 0:00.00 tinydns
12129 root 18 0 1688 376 312 S 0 0.0 0:00.00 tcpserver
12139 root 20 0 2436 1128 984 S 0 0.1 0:00.00 sh
12155 root 18 0 4240 1484 1224 S 0 0.1 0:00.00 kloxo.exe
12205 lxlabs 18 0 5408 2252 1152 S 0 0.2 0:00.08 kloxo.httpd
12233 root 19 0 2432 1144 984 S 0 0.1 0:00.00 mysqld_safe
13329 mysql 19 0 13852 4960 3624 S 0 0.5 0:00.11 mysqld
13382 qmails 18 0 1896 472 372 S 0 0.0 0:00.00 qmail-send
13390 qmaill 19 0 1660 484 412 S 0 0.0 0:00.00 splogger
13391 root 20 0 1672 344 268 S 0 0.0 0:00.00 qmail-lspawn
13392 qmailr 18 0 1668 344 268 S 0 0.0 0:00.00 qmail-rspawn
13393 qmailq 22 0 1652 348 284 S 0 0.0 0:00.00 qmail-clean
13429 root 18 0 6284 1300 972 S 0 0.1 0:00.00 authdaemond
13435 root 18 0 1688 384 320 S 0 0.0 0:00.00 tcpserver
13440 root 18 0 1688 376 312 S 0 0.0 0:00.00 tcpserver
13445 root 18 0 1688 380 320 S 0 0.0 0:00.00 tcpserver
13447 root 18 0 1688 372 312 S 0 0.0 0:00.00 tcpserver
13467 root 15 0 3204 1108 576 S 0 0.1 0:00.00 crond
13475 root 18 0 6284 456 128 S 0 0.0 0:00.00 authdaemond
15735 root 18 0 24820 8936 5416 S 0 0.9 0:00.21 httpd
15752 apache 15 0 26028 7740 3580 S 0 0.7 0:00.16 httpd
15755 apache 15 0 25508 7684 3672 S 0 0.7 0:00.31 httpd
16290 apache 15 0 25816 7848 3668 S 0 0.7 0:00.32 httpd
18216 apache 18 0 25860 8132 3844 S 0 0.8 0:00.29 httpd
19779 apache 18 0 25500 7776 3640 S 0 0.7 0:00.41 httpd
20099 apache 15 0 25604 8132 3848 S 0 0.8 0:00.31 httpd
20365 apache 15 0 25808 7880 3688 S 0 0.8 0:00.27 httpd
20385 apache 15 0 25500 7936 3780 S 0 0.8 0:00.27 httpd
22452 apache 15 0 25292 7684 3708 S 0 0.7 0:00.31 httpd
23702 apache 15 0 25836 8036 3836 S 0 0.8 0:00.31 httpd
23714 apache 15 0 25300 7760 3680 S 0 0.7 0:00.31 httpd

So far with apache, so good. If apache wont work either, I'll try nginx again

[bigks]

Good luck man, you should pay attention to apache after around a day or so as you will see it leveling out on resources.

[The Universes]

Originally Posted by vigge_sWe

but why would I run out of memory? I got 512 mb guaranteed, 1 gb burstable.

Perhaps poor configuration, over provisioning of php-fcgi processes, or memory intensive scripts?

Your UBC clearly shows that your VPS was out of memory some 500 times. As for 1GB burst, you can't always rely on that being available to you, you are guraenteed 512MB and its best to try and work within that as much as possible.

[vigge_sWe]

Originally Posted by The Universes

Perhaps poor configuration, over provisioning of php-fcgi processes, or memory intensive scripts?

Your UBC clearly shows that your VPS was out of memory some 500 times. As for 1GB burst, you can't always rely on that being available to you, you are guraenteed 512MB and its best to try and work within that as much as possible.

Yeh, but I have handled 2k pageviews/day on shared hosting before. But today I only got around 500, on a VPS. Why would it work better on shared hosting until 1&1 did something?

Btw, anyone could give suggestion on how to configure correctly?

[The Universes]

Originally Posted by vigge_sWe

Btw, anyone could give suggestion on how to configure correctly?

Perhaps if you post your Lighttpd config we can take a look.

[vigge_sWe]

Originally Posted by The Universes

Perhaps if you post your Lighttpd config we can take a look.

The kloxo default one.

I'm going to bed as it's late and it's school tomorrow, so I wont be able to post it before tomorrow afternoon.

[ldcdc]

Maybe lighty has a memory leak. I recalled reading something about such, and here it is:

http://hostingfu.com/article/nginx-v...or-a-small-vps

I can't reach the page at this time, so here's a Google cache link just in case:

http://74.125.77.132/search?q=cache:...ient=firefox-a

So, try nginx and see if it's any better?

[jamjam]

Originally Posted by vigge_sWe

Yeh, but I have handled 2k pageviews/day on shared hosting before. But today I only got around 500, on a VPS. Why would it work better on shared hosting until 1&1 did something?

Btw, anyone could give suggestion on how to configure correctly?

Usually shared hosting are hosted on high-end servers but on a VPS your resources are limited. I also encountered that error before and you can fix that by optimizing your config files if it did not work then I guess you should upgrade your plan.

[The Universes]

Originally Posted by ldcdc

Maybe lighty has a memory leak. I recalled reading something about such, and here it is:

http://hostingfu.com/article/nginx-v...or-a-small-vps

It probably did in the old days, but Lighttpd has come a long way since 2007 and thats no longer a problem.