Results 1 to 33 of 33
  1. #1
    Join Date
    Aug 2002
    Posts
    75

    high server loads - do i need more RAM?

    Hi all, I'm experiencing very high server loads and this slows down my website. I used 2 servers (Dual Xeon, SCSI HDD, 2GB RAM each) for 1 website, 1 server for http and 1 for mysql connected via cross connect cable. I've also tried to optimized httpd.conf and my.cnf but this is as good as it gets on my own effort. Could anyone recommend something for me like whether I should get more RAM or some other solutions? Thanks in advance!

    http server

    00:47:42 up 10 days, 53 min, 1 user, load average: 36.67, 41.66, 91.12
    474 processes: 443 sleeping, 28 running, 2 zombie, 1 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 94.8% 0.0% 3.0% 0.1% 1.0% 0.0% 0.8%
    cpu00 89.4% 0.0% 4.1% 0.5% 2.5% 0.0% 3.1%
    cpu01 96.2% 0.0% 3.3% 0.0% 0.3% 0.0% 0.0%
    cpu02 95.2% 0.0% 3.1% 0.1% 1.3% 0.0% 0.0%
    cpu03 98.6% 0.0% 1.1% 0.0% 0.1% 0.0% 0.0%
    Mem: 2075792k av, 1535224k used, 540568k free, 0k shrd, 47544k buff
    1276972k active, 169912k inactive
    Swap: 2040212k av, 923380k used, 1116832k free 160356k cached

    mysql server

    00:41:03 up 29 days, 18:46, 1 user, load average: 25.99, 22.32, 16.35
    62 processes: 59 sleeping, 2 running, 1 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 86.9% 0.0% 11.3% 0.1% 0.6% 0.0% 0.8%
    cpu00 85.0% 0.0% 10.9% 0.7% 2.3% 0.0% 0.7%
    cpu01 86.2% 0.0% 12.7% 0.0% 0.1% 0.0% 0.7%
    cpu02 87.0% 0.0% 10.9% 0.0% 0.0% 0.0% 1.9%
    cpu03 89.4% 0.0% 10.5% 0.0% 0.0% 0.0% 0.0%
    Mem: 2075964k av, 2057884k used, 18080k free, 0k shrd, 34708k buff
    1934676k active, 87144k inactive
    Swap: 2040244k av, 221200k used, 1819044k free 755272k cached

    mysql variables

    Variable Global value
    back log 50
    basedir /usr/local/mysql/
    binlog cache size 32768
    bulk insert buffer size 8388608
    concurrent insert ON
    connect timeout 5
    convert character set
    datadir /var/lib/mysql/
    default week format 0
    delay key write ON
    delayed insert limit 100
    delayed insert timeout 300
    delayed queue size 1000
    flush OFF
    flush time 0
    ft boolean syntax + -><()~*:""&|
    ft min word len 4
    ft max word len 254
    ft max word len for sort 20
    ft stopword file (built-in)
    have bdb NO
    have crypt YES
    have innodb YES
    have isam YES
    have raid NO
    have symlink YES
    have openssl NO
    have query cache YES
    init file
    innodb additional mem pool size 1048576
    innodb buffer pool size 8388608
    innodb data file path ibdata1:10M:autoextend
    innodb data home dir
    innodb file io threads 4
    innodb force recovery 0
    innodb thread concurrency 8
    innodb flush log at trx commit 1
    innodb fast shutdown ON
    innodb flush method
    innodb lock wait timeout 50
    innodb log arch dir ./
    innodb log archive OFF
    innodb log buffer size 1048576
    innodb log file size 5242880
    innodb log files in group 2
    innodb log group home dir ./
    innodb mirrored log groups 1
    innodb max dirty pages pct 90
    innodb max purge lag 0
    innodb table locks ON
    interactive timeout 28800
    join buffer size 131072
    key buffer size 1073741824
    language /usr/local/mysql/share/mysql/english/
    large files support ON
    license GPL
    local infile ON
    locked in memory OFF
    log OFF
    log update OFF
    log bin OFF
    log slave updates OFF
    log slow queries ON
    log warnings 1
    long query time 3
    low priority updates OFF
    lower case file system OFF
    lower case table names 0
    max allowed packet 1048576
    max binlog cache size 4294967295
    max binlog size 1073741824
    max connections 800
    max connect errors 10
    max delayed threads 20
    max insert delayed threads 20
    max heap table size 16777216
    max join size 4294967295
    max relay log size 0
    max seeks for key 4294967295
    max sort length 1024
    max user connections 0
    max tmp tables 32
    max write lock count 4294967295
    myisam max extra sort file size 268435456
    myisam max sort file size 2147483647
    myisam repair threads 1
    myisam recover options OFF
    myisam sort buffer size 8388608
    net buffer length 16384
    net read timeout 30
    net retry count 10
    net write timeout 60
    new OFF
    open files limit 5610
    log error
    port 3306
    protocol version 10
    query alloc block size 8192
    query cache limit 1048576
    query cache size 20971520
    query cache type ON
    query prealloc size 8192
    range alloc block size 2048
    read buffer size 131072
    read only OFF
    read rnd buffer size 262144
    rpl recovery rank 0
    server id 0
    slave net timeout 3600
    skip external locking ON
    skip networking OFF
    skip show database OFF
    slow launch time 2
    socket /var/lib/mysql/mysql.sock
    sort buffer size 2097144
    sql mode 0
    table cache 2400
    table type MYISAM
    thread cache size 40
    thread stack 196608
    tx isolation REPEATABLE-READ
    tmp table size 33554432
    tmpdir /tmp/
    transaction alloc block size 8192
    transaction prealloc size 4096
    version 4.0.23a-log
    version comment Source distribution
    version compile os redhat-linux-gnu
    wait timeout 28800

    httpd.conf

    Timeout 300
    KeepAlive On
    MaxKeepAliveRequests 300
    KeepAliveTimeout 5
    MinSpareServers 20
    MaxSpareServers 35
    StartServers 15
    MaxClients 800
    MaxRequestsPerChild 50

    P.s. I know some would question about KeepAlive On but I have many images on my site.

  2. #2
    Is the kernel, apache and mysql updated to the latest version?
    For how long has this problem been occuring.
    Do you see high loads in your bandwidth usage as well (MRTG)?
    Pierre Grandmaison
    Offering 24/7 Toll Free Telephone Support
    Zenutech Web Hosting
    http://www.zenutech.com

  3. #3
    Join Date
    Aug 2002
    Posts
    75
    Yup they are.

    Ok, apparently when I switch KeepAlive to Off the load did drop.

  4. #4
    Join Date
    Apr 2005
    Location
    Sweden
    Posts
    241
    Please also post the result of "atop", if you can. This will give us you your network usage stats and hdd usage stats as well. Let atop run for 10 seconds first so you dont get the average since you booted.

  5. #5
    Join Date
    Dec 2004
    Location
    New York, NY
    Posts
    10,574
    It looks like you are using quite a bit of swap there. Therefore, extra RAM will help you out.

    --GSV
    MediaLayer, LLC - www.medialayer.com Learn how we can make your website load faster, translating to better conversion rates for your business!
    The pioneers of optimized web hosting, featuring LiteSpeed Web Server & SSD Storage - Celebrating 10 Years in Business

  6. #6
    Join Date
    Jan 2003
    Posts
    1,715
    key buffer size 1073741824

    A gig of key buffer? You need to hire someone to look this over first-hand and tune it for you.
    Game Servers are the next hot market!
    Slim margins, heavy support, fickle customers, and moronic suppliers!
    Start your own today!

  7. #7
    Join Date
    Aug 2002
    Posts
    75
    Originally posted by hiryuu
    key buffer size 1073741824

    A gig of key buffer? You need to hire someone to look this over first-hand and tune it for you.
    I based it on the principle of 50% of physical RAM which is 2GB.

  8. #8
    Join Date
    Mar 2005
    Posts
    50
    to me, it doesnt look like your problem is memory, but what do i know.


    but i dont think anyone can tell you if somethings wrong unless they know how much traffic your site generates.

  9. #9
    Well, it looks like you're using 1GB of swap space on your HTTP server. Adding at least a GB would be a good idea. 2 if you can afford it.

    And you apache may be tuned perfectly. I'd say it's time to look at the content you're serving.

    91?! That's freaking high man.

    Also, how fast are these XEONs? I've heard Opterons are better for servers but I haven't researched it.

  10. #10
    Join Date
    Aug 2002
    Posts
    75
    My host TP thinks its because of keepalive being on so they switch it to off and it does help the load and cleared the swap but this also means pages load slightly slower even during off peak times. I'm getting about 700+ users within 15 minutes during peak hours.

  11. #11
    Join Date
    Jan 2005
    Location
    Scotland, UK
    Posts
    2,549
    What does mysqladmin processlist return.

    Its time to look at optomization.

    You are using lots of swap.

    2 Xeons should handle 700 users every 15 minutes. I assume you mean as in forums. From what I normally see the problem with forums is the search feature, have you tested disabling it for 15 minutes, and monitoring server loads , and then testing with server loads enabled.

    Basicly its trial and error to see whats causing your major loads.
    Server Management - AdminGeekZ.com
    Infrastructure Management, Web Application Performance, mySQL DBA. System Automation.
    WordPress/Magento Performance, Apache to Nginx Conversion, Varnish Implimentation, DDoS Protection, Custom Nginx Modules
    Check our wordpress varnish plugin. Contact us for quote: [email protected]

  12. #12
    Join Date
    Apr 2002
    Location
    Las Vegas
    Posts
    797
    set your time out to 150 or even 100, plus have you added eAccelerator?

  13. #13
    Join Date
    Aug 2002
    Posts
    75
    I run a dating personals, forum and gallery all 3 are quite popular with 700+ users overall using at peak hours.

    I don't know about eAccelerator but mmcache conflicted with one of my script which used zend optimiser so I couldn't used it despite installing it according to instructions.

    Thanks all for the advice.

  14. #14
    Join Date
    Apr 2002
    Location
    Las Vegas
    Posts
    797
    Originally posted by singtel22
    I run a dating personals, forum and gallery all 3 are quite popular with 700+ users overall using at peak hours.

    I don't know about eAccelerator but mmcache conflicted with one of my script which used zend optimiser so I couldn't used it despite installing it according to instructions.

    Thanks all for the advice.

    remove mmcache, and add eAccelerator.

    php pages will be 50% faster with 50% less server loads. I use this on all production STC servers and i havent ran into any problems.

  15. #15
    Join Date
    Aug 2002
    Posts
    75
    Now my server loads gotten crazy:

    00:12:43 up 12 days, 18 min, 1 user, load average: 195.13, 265.57, 161.44
    230 processes: 194 sleeping, 35 running, 1 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 87.5% 0.0% 6.8% 0.4% 1.1% 3.1% 0.8%
    cpu00 84.2% 0.0% 7.8% 0.8% 0.8% 6.1% 0.0%
    cpu01 85.7% 0.0% 7.1% 0.0% 0.0% 7.1% 0.0%
    cpu02 87.6% 0.0% 7.0% 0.8% 4.4% 0.0% 0.0%
    cpu03 90.3% 0.0% 6.1% 0.0% 0.0% 0.0% 3.5%
    Mem: 2075792k av, 474100k used, 1601692k free, 0k shrd, 7652k buff
    398752k active, 24856k inactive
    Swap: 2040212k av, 233332k used, 1806880k free 30860k cached

  16. #16
    Join Date
    Apr 2002
    Location
    Las Vegas
    Posts
    797
    Originally posted by singtel22
    Now my server loads gotten crazy:

    00:12:43 up 12 days, 18 min, 1 user, load average: 195.13, 265.57, 161.44
    230 processes: 194 sleeping, 35 running, 1 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 87.5% 0.0% 6.8% 0.4% 1.1% 3.1% 0.8%
    cpu00 84.2% 0.0% 7.8% 0.8% 0.8% 6.1% 0.0%
    cpu01 85.7% 0.0% 7.1% 0.0% 0.0% 7.1% 0.0%
    cpu02 87.6% 0.0% 7.0% 0.8% 4.4% 0.0% 0.0%
    cpu03 90.3% 0.0% 6.1% 0.0% 0.0% 0.0% 3.5%
    Mem: 2075792k av, 474100k used, 1601692k free, 0k shrd, 7652k buff
    398752k active, 24856k inactive
    Swap: 2040212k av, 233332k used, 1806880k free 30860k cached
    k, post your httd.conf and your my.cnf

  17. #17
    Join Date
    Apr 2002
    Location
    Las Vegas
    Posts
    797
    Originally posted by singtel22
    I run a dating personals, forum and gallery all 3 are quite popular with 700+ users overall using at peak hours.

    I don't know about eAccelerator but mmcache conflicted with one of my script which used zend optimiser so I couldn't used it despite installing it according to instructions.

    Thanks all for the advice.
    remove mmcach and add eAccelerator

    great how too here
    http://www.theadminzone.com/forums/s...ead.php?t=8152

  18. #18
    Join Date
    Apr 2002
    Location
    Las Vegas
    Posts
    797
    for mysql , start here
    http://www.theadminzone.com/forums/s...ead.php?t=8150


    above should and will lower your serverloads

    PM links of your site,

    P.s. if you need help PM me

    keep us advised

  19. #19
    Join Date
    Aug 2002
    Posts
    75
    My httpd.conf and my.cnf are posted in the first posting in this thread. Thanks.

  20. #20
    Join Date
    Apr 2002
    Location
    Las Vegas
    Posts
    797
    this should help you out

    httpd.conf
    Find:
    TimeOut 300

    Change to
    TimeOut 100

    Find:
    MaxKeepAliveRequests 300

    Change to:
    MaxKeepAliveRequests 150


    try this and repost your stats after 1hr, this should lower the load . then we will move on to the next step

  21. #21
    Join Date
    Aug 2002
    Posts
    75
    Thanks, I've applied the changes but it's graveyard hours now where my users are based so it's not likely to see any effects until peak time.

  22. #22
    Join Date
    Apr 2002
    Location
    Las Vegas
    Posts
    797
    ok, sounds good ...

    next i like to have you add eAccelerator

  23. #23
    Join Date
    Aug 2002
    Posts
    75
    Ok after applying those changes to httpd.conf the server load during peak hours is now:

    23:14:40 up 12 days, 23:20, 1 user, load average: 7.08, 7.98, 7.33
    370 processes: 367 sleeping, 3 running, 0 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 45.6% 0.0% 3.4% 0.1% 0.9% 1.6% 48.1%
    cpu00 56.8% 0.0% 3.1% 0.3% 1.3% 2.3% 35.7%
    cpu01 47.2% 0.0% 3.7% 0.0% 0.1% 2.1% 46.6%
    cpu02 40.5% 0.0% 2.9% 0.3% 1.7% 0.5% 53.6%
    cpu03 37.7% 0.0% 4.1% 0.0% 0.5% 1.1% 56.2%
    Mem: 2075792k av, 2012892k used, 62900k free, 0k shrd, 238040k buff
    1716328k active, 223720k inactive
    Swap: 2040212k av, 145784k used, 1894428k free 241572k cached

    But mysql is still pretty high:


    23:06:07 up 32 days, 17:11, 1 user, load average: 37.52, 27.99, 22.67
    73 processes: 70 sleeping, 2 running, 1 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 85.7% 0.0% 13.4% 0.1% 0.6% 0.0% 0.0%
    cpu00 84.1% 0.0% 12.8% 0.5% 2.3% 0.0% 0.0%
    cpu01 85.8% 0.0% 13.9% 0.0% 0.1% 0.0% 0.0%
    cpu02 87.4% 0.0% 12.3% 0.0% 0.1% 0.0% 0.0%
    cpu03 85.2% 0.0% 14.5% 0.0% 0.1% 0.0% 0.0%
    Mem: 2075964k av, 2069740k used, 6224k free, 0k shrd, 32596k buff
    1868500k active, 164220k inactive
    Swap: 2040244k av, 296120k used, 1744124k free 717780k cached

    Do you think I should up the RAM on mysql server? Thanks.

  24. #24
    Perhaps you should check to see if your hardware is not having problems. I remember having high loads on a server when the hard drive was having sector problems and eventually died.

    You can run "dmesg" to view kernel messages (potentially hardware errors), or tail /var/log/messages .
    Pierre Grandmaison
    Offering 24/7 Toll Free Telephone Support
    Zenutech Web Hosting
    http://www.zenutech.com

  25. #25
    Join Date
    Apr 2002
    Location
    Las Vegas
    Posts
    797
    Originally posted by singtel22
    Ok after applying those changes to httpd.conf the server load during peak hours is now:

    23:14:40 up 12 days, 23:20, 1 user, load average: 7.08, 7.98, 7.33
    370 processes: 367 sleeping, 3 running, 0 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 45.6% 0.0% 3.4% 0.1% 0.9% 1.6% 48.1%
    cpu00 56.8% 0.0% 3.1% 0.3% 1.3% 2.3% 35.7%
    cpu01 47.2% 0.0% 3.7% 0.0% 0.1% 2.1% 46.6%
    cpu02 40.5% 0.0% 2.9% 0.3% 1.7% 0.5% 53.6%
    cpu03 37.7% 0.0% 4.1% 0.0% 0.5% 1.1% 56.2%
    Mem: 2075792k av, 2012892k used, 62900k free, 0k shrd, 238040k buff
    1716328k active, 223720k inactive
    Swap: 2040212k av, 145784k used, 1894428k free 241572k cached

    But mysql is still pretty high:


    23:06:07 up 32 days, 17:11, 1 user, load average: 37.52, 27.99, 22.67
    73 processes: 70 sleeping, 2 running, 1 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 85.7% 0.0% 13.4% 0.1% 0.6% 0.0% 0.0%
    cpu00 84.1% 0.0% 12.8% 0.5% 2.3% 0.0% 0.0%
    cpu01 85.8% 0.0% 13.9% 0.0% 0.1% 0.0% 0.0%
    cpu02 87.4% 0.0% 12.3% 0.0% 0.1% 0.0% 0.0%
    cpu03 85.2% 0.0% 14.5% 0.0% 0.1% 0.0% 0.0%
    Mem: 2075964k av, 2069740k used, 6224k free, 0k shrd, 32596k buff
    1868500k active, 164220k inactive
    Swap: 2040244k av, 296120k used, 1744124k free 717780k cached

    Do you think I should up the RAM on mysql server? Thanks.

    great , can you post my.cnf file of mysql

  26. #26
    Join Date
    Aug 2002
    Posts
    75
    Hi, the messages look fine except for a "program scsi_unique_id is using a deprecated SCSI ioctl, please convert it to SG
    _IO" which I don't know when it was logged.

    I'm registering traffic from about 800+ users right now and most scripts are database intensive.
    Last edited by singtel22; 06-06-2005 at 11:33 AM.

  27. #27
    Join Date
    Aug 2002
    Posts
    75
    Ok seems like httpd.conf with

    Timeout 100
    KeepAlive On
    MaxKeepAliveRequests 150
    KeepAliveTimeout 5
    MaxClients 700

    led to

    00:23:06 up 13 days, 28 min, 1 user, load average: 182.64, 288.45, 240.15
    434 processes: 408 sleeping, 25 running, 1 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 85.6% 0.0% 5.9% 0.2% 1.4% 4.7% 1.8%
    cpu00 84.7% 0.0% 5.1% 0.7% 3.1% 6.1% 0.0%
    cpu01 88.9% 0.0% 4.9% 0.0% 0.1% 4.3% 1.5%
    cpu02 83.1% 0.0% 7.1% 0.3% 1.9% 4.5% 2.7%
    cpu03 85.9% 0.0% 6.7% 0.0% 0.3% 4.1% 2.7%
    Mem: 2075792k av, 1051112k used, 1024680k free, 0k shrd, 9392k buff
    967532k active, 27804k inactive
    Swap: 2040212k av, 246348k used, 1793864k free 34524k cached

    So I switched keepalive to off and restarted http and the load went down. Not sure if its down to to keepalive off or the restart.

  28. #28
    Join Date
    Apr 2002
    Location
    Las Vegas
    Posts
    797
    we need to add eAccelerator

    Originally posted by singtel22
    Ok seems like httpd.conf with

    Timeout 100
    KeepAlive On
    MaxKeepAliveRequests 150
    KeepAliveTimeout 5
    MaxClients 700

    led to

    00:23:06 up 13 days, 28 min, 1 user, load average: 182.64, 288.45, 240.15
    434 processes: 408 sleeping, 25 running, 1 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 85.6% 0.0% 5.9% 0.2% 1.4% 4.7% 1.8%
    cpu00 84.7% 0.0% 5.1% 0.7% 3.1% 6.1% 0.0%
    cpu01 88.9% 0.0% 4.9% 0.0% 0.1% 4.3% 1.5%
    cpu02 83.1% 0.0% 7.1% 0.3% 1.9% 4.5% 2.7%
    cpu03 85.9% 0.0% 6.7% 0.0% 0.3% 4.1% 2.7%
    Mem: 2075792k av, 1051112k used, 1024680k free, 0k shrd, 9392k buff
    967532k active, 27804k inactive
    Swap: 2040212k av, 246348k used, 1793864k free 34524k cached

    So I switched keepalive to off and restarted http and the load went down. Not sure if its down to to keepalive off or the restart.

  29. #29
    Join Date
    Aug 2002
    Posts
    75
    Great now my host says I was under attack after repeated attempts to ask me to upgrade my RAM! This was what they said:

    I found that your web server was under attack. You had an extremely large Syn flood coming in, not to mention a small DoS. You were looking at over 100,000 connections coming to your server, at which point I contacted security and they were able to mitigate the traffic. You should see better performance from your server now.

  30. #30
    Join Date
    Mar 2004
    Location
    california
    Posts
    162
    What scripts are you running?

  31. #31
    Join Date
    Aug 2002
    Posts
    75
    I run a dating personals, forum and gallery all 3 are quite popular with 800+ users overall using at peak hours.

  32. #32
    Join Date
    Apr 2002
    Location
    Las Vegas
    Posts
    797
    how is it going ?

  33. #33
    Join Date
    Aug 2002
    Posts
    75
    Ok thanks for all your useful tips and comments. Indeed it was a small DoS and now my traffic is back to normal. Thanks.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •