Results 1 to 35 of 35
  1. #1

    Did anyone experience a huge slowdown on Allmanaged servers?

    This is the problem: since about 5 days ago our Allmanaged server (p4 3.06, 1GB ram, 1500GB bandwith) started to be really really slow.
    Now, since there can be numerous problems I started eliminating them one-by-one.

    First, I thought that domains were resolved slowly because of my ISP (which is known for un-stabillity and route changing and so on) but I got complaints from people from both USA and Europe that they are experiencing a slowdown.

    And it's a rather odd one - it takes arround 1-2 minutes for a plain page to load, be it dynamic or static one. First there's no response and then in one second everything gets spit out and loads fast. What's funny is that there's a cycle to this (every 20 minutes server is ok, and another 20 it's slow).

    Then I thought that MySQL could be the problem since server's got arround 150-250 concurent users (150 min, 250 max).
    But there hasn't been a single slow query logged, and MySQL never used more than 55% cpu. Also, memory usage is at 90% at max.

    So, I just want to make sure that other people that rent servers @ Allmanaged have these problems before I start at them.
    Don't get me wrong, I've been satisfied with Allmanaged's tech support since they reply really fast (within 15 minutes) but this issue has been bothering both us and our users.

    If you had or even if you didn't have such problems - have you experienced odd behaviour of your Allmanaged server?
    And perhaps this might be server issue, but how to detect it? There hasn't been any proces that's been overloading the server (load was pretty low all the time).

  2. #2
    Join Date
    Apr 2005
    Location
    San Francisco, CA
    Posts
    1,029
    Why you don't contact your provider and ask for check hardware/software configuration ? Here can be 100's of reasons why this is happening.

  3. #3
    Join Date
    Nov 2004
    Location
    Wisconsin
    Posts
    148
    If your memory usage gets anywhere close to 90% then you might want to consider getting more ram in that server. I don't know if that is what is causing your problems but that could very well be one of the causes.
    ~ Nick

  4. #4
    Join Date
    Apr 2005
    Location
    Atlanta, Georgia
    Posts
    520
    sounds like 2gb apache log file limits.....

  5. #5
    Hmmm I went through something similar with another provider. It would take 1 minute or two for each page to respond, but once responded, the actual page loads fast. The problem turned out to be a DNS issue. If you're hosting your own DNS on your server, I'd check that, or if it's hosted remotely by allmanaged, ask them to look into it. Just a thought.

  6. #6
    Join Date
    Jan 2005
    Posts
    2,175
    I run a forum with many visitors from all over the world. There has been no problems for the past 3 months or so, smooth sailing, no slowdowns like you mentioned. I think it's just your server.

  7. #7
    Join Date
    Jan 2005
    Location
    Scotland, UK
    Posts
    2,549
    2gb apache logs would mean apache fails completely, not causing site slow down.


    Originally posted by TulipSystems
    sounds like 2gb apache log file limits.....
    Server Management - AdminGeekZ.com
    Infrastructure Management, Web Application Performance, mySQL DBA. System Automation.
    WordPress/Magento Performance, Apache to Nginx Conversion, Varnish Implimentation, DDoS Protection, Custom Nginx Modules
    Check our wordpress varnish plugin. Contact us for quote: [email protected]

  8. #8
    Join Date
    Sep 2004
    Location
    Flint, Michigan
    Posts
    5,765
    It's a managed machine, just have them fix it
    Mike from Zoodia.com
    Professional web design and development services.
    In need of a fresh hosting design? See what premade designs we have in stock!
    Web design tips, tricks, and more at MichaelPruitt.com

  9. #9
    Why you don't contact your provider and ask for check hardware/software configuration ? Here can be 100's of reasons why this is happening.
    Heh, that was the first thing I've done.

    If your memory usage gets anywhere close to 90% then you might want to consider getting more ram in that server. I don't know if that is what is causing your problems but that could very well be one of the causes.
    In that case I will get additional 1GB ASAP.

    The problem turned out to be a DNS issue. If you're hosting your own DNS on your server, I'd check that
    Yes, I am hosting my own DNS. But the problem appears at the website that's not on my DNS servers but it's hosted at my server. I thought it could be DNS issue but what's the chance that 2 separate DNS fail at the same time repeatedly? I thought that sofware has been configured properly so that led me to think that datacenter had problems..


    After eliminating MySQL as potential bottle-neck and DNS issues (and php code for that matter) I've taken a look at httpd and it seems like Apache was causing the problems.
    After Apache restart everything works fine, but after 10 minutes it starts to slow down.
    I've examined httpd.conf and found that MaxClients was 150 and MaxRequestsPerChild was 0.
    After "tweaking" those settings it seems to have sped up since I haven't experienced any slowdown for arround 10 hours.


    Thank you all for your replies, I appreciate it.

  10. #10
    Join Date
    Apr 2004
    Location
    Singapore
    Posts
    617
    did you have keepalive turn off? this can help improve the performance.
    Linux System admin (since 2001)
    * cPanel/WHM, Directadmin, Apache, DNS, PHP, HyperVM, Lxadmin, Openvz*

  11. #11
    KeepAlive is On. It should be off to improve performance?
    Any additional tips perhaps (I've lowered KeepAliveTimeout to 3 to improve performance)?

  12. #12
    Join Date
    Apr 2005
    Location
    Atlanta, Georgia
    Posts
    520
    Originally posted by HostGeekZ
    2gb apache logs would mean apache fails completely, not causing site slow down.
    That is incorrect, apache does not fail completely.

    Pages are served up (all be it slow) and the thread dies when it attempts to write to the log file (after the request is served)

  13. #13
    Join Date
    Jul 2004
    Location
    New York, NY
    Posts
    2,179
    Tulip - that's not always the case. Sometimes it times out before it can die
    ServGrid - www.servgrid.com - Affordable and Reliable SSD Cloud Solutions
    Premium 10G Network, 2(N+1) Powerplant and SSD Performance
    Web, Reseller, KVM VPS, Storage and Private Cloud Hosting
    Click here to see our SSD Benchmarks!

  14. #14
    Join Date
    Apr 2005
    Location
    Atlanta, Georgia
    Posts
    520
    Originally posted by The Broadband Man
    Tulip - that's not always the case. Sometimes it times out before it can die
    Well, it's all variable based upon the traffic the box is getting, in his case it would have served the page.

    However in other instances, say a disk full situation apache can get out of hand quickly.

    Recently had a process get out of control and fill up the 20gig log drive on one of our servers. Apache started freaking out and just started forking child processes to the tune of 900 processes, luckily the box was big enough to handle the mess apache created but it can do some nasty things when disk space and or log files aren't available to it

  15. #15
    Join Date
    Jan 2005
    Location
    Scotland, UK
    Posts
    2,549
    Hello,

    If apache has logs at are bigger than 2 gig. it will fail completely, and will not start.

    It will also report about this in error_log.
    Server Management - AdminGeekZ.com
    Infrastructure Management, Web Application Performance, mySQL DBA. System Automation.
    WordPress/Magento Performance, Apache to Nginx Conversion, Varnish Implimentation, DDoS Protection, Custom Nginx Modules
    Check our wordpress varnish plugin. Contact us for quote: [email protected]

  16. #16
    Join Date
    Feb 2003
    Location
    Kuala Lumpur, Malaysia
    Posts
    4,974
    Hm, it just slow down, not fail though. You can find that in error_log as mentioned.

  17. #17
    Join Date
    Dec 2003
    Location
    Chicago
    Posts
    475
    cant say that I have with my AM server - but they are 'all managed' so why not just contact them and ask them to take a look at your server? They are usually very good about this. Unless you want to fix it by yourself for some sense of personal satisfaction

    -Chris
    Course-Notes.Org

    US History US Government and Politics| Note Outlines Vocab
    Euro History Psychology Econonomics | Forum Study Guides

  18. #18
    I've contacted Allmanaged. They are very good in responding fast but they didn't solve my issue. There's no ego that I have to satisfy but I just wanted to hear from experienced people what might be happening before I contact Allmanaged again regarding this issue.
    Apache error_log is 133 MB in size, memory usage is 90%, swap is at 60% -> so that might be the issue.
    Since number of users is increasing by day I will order additional RAM because system memory seems to have become an issue.
    Thanks for helping me guys, I appreciate it.

  19. #19
    Join Date
    Sep 2004
    Location
    Flint, Michigan
    Posts
    5,765
    If you're swapping that's the issue right there most likely. Tweaking configs and/or ram time =/
    Mike from Zoodia.com
    Professional web design and development services.
    In need of a fresh hosting design? See what premade designs we have in stock!
    Web design tips, tricks, and more at MichaelPruitt.com

  20. #20
    Can you suggest what should be tweaked exactly so I can google it a bit?

  21. #21
    Join Date
    Sep 2004
    Location
    Flint, Michigan
    Posts
    5,765
    Well your mysql and apache settings are what would usually need to be tweaked. If you want to post what's in your my.cnf and httpd.conf (just the top part with the settings for apache) there are some good folks here that should be able to help you out.
    Mike from Zoodia.com
    Professional web design and development services.
    In need of a fresh hosting design? See what premade designs we have in stock!
    Web design tips, tricks, and more at MichaelPruitt.com

  22. #22
    Ok, here goes httpd.conf without the usual comments. I will paste commented and uncommented directives.

    Code:
    RLimitMEM 0
    RLimitCPU 240
    
    Alias /bandwidth/ /usr/local/bandmin/htdocs/
    
    ServerType standalone
    
    ServerRoot "/usr/local/apache"
    
    #LockFile /usr/local/apache/logs/httpd.lock
    
    PidFile /usr/local/apache/logs/httpd.pid
    
    ScoreBoardFile /usr/local/apache/logs/httpd.scoreboard
    
    #ResourceConfig /usr/local/apache/conf/srm.conf
    #AccessConfig /usr/local/apache/conf/access.conf
    
    Timeout 300
    
    KeepAlive On
    
    MaxKeepAliveRequests 100
    
    KeepAliveTimeout 3
    
    MinSpareServers 5
    MaxSpareServers 10
    
    StartServers 5
    
    MaxClients 300
    
    MaxRequestsPerChild 10000
    
    #Listen 3000
    #Listen 12.34.56.78:80
    
    #BindAddress *


    And my.cnf (the entire file)

    Code:
    [mysqld]
    max_connections=900
    max_user_connections=900
    skip-locking
    skip-innodb
    query_cache_limit=1M
    query_cache_size=32M
    query_cache_type=1
    interactive_timeout=50
    wait_timeout=360
    connect_timeout=10
    thread_cache_size=128
    key_buffer=256M
    max_allowed_packet=16M
    table_cache=1024
    join_buffer=1M
    sort_buffer_size=1M
    read_buffer_size=1M
    max_connect_errors=10
    thread_concurrency=4
    myisam_sort_buffer_size=64M
    log-bin
    server-id=1
    
    [safe_mysqld]
    open_files_limit=8192
    
    [mysqldump]
    quick
    max_allowed_packet=16M
    
    [mysql]
    no-auto-rehash
    
    [isamchk]
    key_buffer=64M
    sort_buffer=64M
    read_buffer=16M
    write_buffer=16M
    
    [myisamchk]
    key_buffer=64M
    sort_buffer=64M
    read_buffer=16M
    write_buffer=16M
    
    [mysqlhotcopy]
    interactive-timeout
    
    log-slow-queries = /var/log/mysql/mysql-slow.log
    long_query_time = 1

  23. #23
    Additional note: server runs 3 websites which in total generate between 10 500 and 11 000 unique visits. All 3 sites are 90% dynamic (with caching mechanysms).
    Checking trough my error_log i found following lines that are somewhat initriguing:
    Code:
    [Wed Jun 29 00:38:09 2005] [error] PHP Warning:  mysql_connect(): Lost connection to MySQL server during query in *****.php on line xxx
    
    [Wed Jun 29 00:38:22 2005] [error] server reached MaxClients setting, consider raising the MaxClients setting
    After those messages Apache crashes (I recieve email notification regarding service failure).

    Now..I guess that's just system asking for more RAM, right?
    Judging by above pasted configuration files, what's to be tweaked?

  24. #24
    Join Date
    Sep 2004
    Location
    Flint, Michigan
    Posts
    5,765
    Well I am no expert in tweaking, but i will give a few tips that might help you out.

    1. Server is asking for more ram yes, and that would be the best solution most likely, but it's not always the best solution.

    2. The message about hitting the maxclients could be a major part of your slowdown as well. To fix that bump up the maxclients setting in the httpd.conf. I would give it a try at 400 and keep an eye on logs and loads.

    3. Install mytop to help you watch for any slow mysql quieries that are comming through. http://jeremy.zawodny.com/mysql/mytop/ is the site for that.

    4. Check out using mmcache to help speed up your site quite a bit. Site is located here: http://turck-mmcache.sourceforge.net/index_old.html What this will do is take dynamic pages that are accessed a lot, and cache the output. Then when that common output is requested (ie. visiting and index.php page that always shows the same) the cached version is given instead of completely reading everything fromt he database.

    5. If these sites are running any off the shelf scripts, if you posted them I might be able to find some tips on making the scripts run faster.

    6. You might want to experiment with KeepAlive off, I'm not sure how it'll effect your site, could make it go faster or slower, so give it a try.

    7. I'm still researching
    Mike from Zoodia.com
    Professional web design and development services.
    In need of a fresh hosting design? See what premade designs we have in stock!
    Web design tips, tricks, and more at MichaelPruitt.com

  25. #25
    I've set KeepAlive to off and increaset MaxClients to 400.
    Regarding slow queries - there are none. I've set up a log file for mysql to report slow queries and slow query is every query that lasts > 1 sec.
    I installed mytop which also shows no slow queries at this time (perhaps it will show some in peak time, who knows).

    I will consider installing a caching mechanism such as turckmmcache or zend optimizer, but mysql and php showed no problems whatsoever so far. We use only 2 generally available scripts - Invision Board 2.0 and PhpBB but we've invested 4 months to optimize those scripts by reducing HTML size and implemented some faster methods and the scripts are behaving extremely well.

    Regarding SQL/PHP/HTML optimising - i sencerely doubt that there can be done anything else.
    Installation of turck mmcache would help php to cache pre-parsed scripts but..what's the use if I can't deliver them to my client?

    I'll see now how server behaves with new Apache settings and report it all.

    Thanks for your help.

  26. #26
    Join Date
    Jan 2005
    Posts
    2,175
    Does MaxClients affect the server as a whole or just for each domain?

  27. #27
    Join Date
    Sep 2004
    Location
    Flint, Michigan
    Posts
    5,765
    That maxclients setting is for apache as a whole.
    Mike from Zoodia.com
    Professional web design and development services.
    In need of a fresh hosting design? See what premade designs we have in stock!
    Web design tips, tricks, and more at MichaelPruitt.com

  28. #28
    Join Date
    Jan 2005
    Posts
    2,175
    Is there a max value for MaxClients?

  29. #29
    Originally posted by ANewDay
    Is there a max value for MaxClients?
    I think the default limit is 512, though you can configure Apache to accept a higher value.

  30. #30
    Join Date
    Sep 2004
    Location
    Flint, Michigan
    Posts
    5,765
    Yes the default is 512, you'd have to recompile apache and edit some of the source files to raise that limit.
    Mike from Zoodia.com
    Professional web design and development services.
    In need of a fresh hosting design? See what premade designs we have in stock!
    Web design tips, tricks, and more at MichaelPruitt.com

  31. #31
    Join Date
    Jan 2005
    Posts
    2,175
    I set mine to 1,000. Isn't 512 a really low number? Say you have 500 busy websites on the server, so each website on average could only have 100 http connections??? It just doesn't seem right.

  32. #32
    Join Date
    Sep 2004
    Location
    Flint, Michigan
    Posts
    5,765
    Originally posted by ANewDay
    I set mine to 1,000. Isn't 512 a really low number? Say you have 500 busy websites on the server, so each website on average could only have 100 http connections??? It just doesn't seem right.
    500 Busy sites would not be on one server most likely. 512 is a bit of a low number, but not really that bad. WHT at peak times has around 1000 users on it and it's spread across several servers.
    Mike from Zoodia.com
    Professional web design and development services.
    In need of a fresh hosting design? See what premade designs we have in stock!
    Web design tips, tricks, and more at MichaelPruitt.com

  33. #33
    Join Date
    Oct 2002
    Location
    In a house
    Posts
    949
    Hello,

    While I see the good folks of WHT always around to help, is there something that myself, or my staff can assist you with directly? We've tried to pull out any ticket numbers for you, but haven't been able to find anything specific that would help us to find your tickets. Please e-mail me directly with a ticket number, so that I can get some staff on this to assist you further if you'd like our help with this situation.

    Thanks,

  34. #34
    Thanks for your concern, I've emailed you.

  35. #35
    Join Date
    Oct 2002
    Location
    In a house
    Posts
    949

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •