Results 1 to 21 of 21
  1. #1
    Join Date
    Apr 2012
    Posts
    75

    Server Completely down

    Greetings pals,
    Please I really need someone here ot help me on my vps server.
    The server is completely down and even ssh can't login.
    I contacted my vps provider and here is their reply

    "Seems your VPS is running with swap memory and it seems the server is getting high connections.

    Please verify the snippet attached below :

    =========
    root@bestservers [~]# w
    19:04:41 up 7 min, 1 user, load average: 120.86, 63.30, 26.02
    USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
    root pts/1 122.167.99.132 18:58 21.00s 0.15s 0.02s w


    root@bestservers [~]# free -m

    total used free shared buffers cached
    Mem: 2010 2001 8 0 4 47
    -/+ buffers/cache: 1949 60
    Swap: 3999 3334 665
    =============

    You need to investigate the issue and resolve it.

    Hope you can understand."

    I have no idea how to resolve this since the server is only accessible via VNC and I am not quite familiar with that.
    I guess I caused this because the last thing I did on the server bfore it got crashed in increasing the php memory limit to 100MB. it was formally 32MB and I guess that is what is affecting the server now.
    Can anybody here please assist and help me on this ? Please.
    Thanks

  2. #2
    Join Date
    Mar 2012
    Posts
    1,421
    I think that your best bet is to go to the Server Management offer section, there you will find some good individuals/companies that knows what they are doing.
    --

  3. #3
    Join Date
    Nov 2003
    Location
    Texas, United States
    Posts
    115
    If you and/or your provider can access the console, copy down the messages that may be displayed there. Also, try logging in through the console (it may still be very slow given your resource usage). If even console login doesn't work, reboot your VPS (or ask your provider to do it if you do not have the access necessary). That should clear up the immediate issue, allowing you (or a System Administration provider) to then gain access to your server and look around.
    Tim Benoit

  4. #4
    Join Date
    Apr 2012
    Posts
    75
    Yeah, I actually had my service provider to reboot the vps and stop cpane, sql and apache.
    this gave me access to ssh.
    While on ssh, I started cpanel, logged into whm and suspended the accounts with high cpu usage.and then started sql and other services.
    this was eble to resolve the issue and on ssh, I run this command

    /usr/local/cpanel/3rdparty/bin/mysqltuner.pl

    Which gave me this recommendations

    -------- Recommendations -----------------------------------------------------
    General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    MySQL started within last 24 hours - recommendations may be inaccurate
    Reduce your overall MySQL memory footprint for system stability
    Enable the slow query log to troubleshoot bad queries
    Adjust your join queries to always utilize indexes
    Temporary table size is already large - reduce result set size
    Reduce your SELECT DISTINCT queries without LIMIT clauses
    Variables to adjust:
    *** MySQL's maximum memory usage is dangerously high ***
    *** Add RAM before increasing MySQL buffer variables ***
    query_cache_limit (> 1M, or use smaller result sets)
    join_buffer_size (> 32.0M, or always use indexes with joins)

    So I am checking around now how to follow this recommendation and limit the MYSQL CPU RESOURCE USAGE and also MySQL's maximum memory before enabling those accounts back.
    but the overall server performance is back up now.
    hope this help someone.
    Thanks

  5. #5
    Join Date
    Nov 2003
    Location
    Texas, United States
    Posts
    115
    To modify the MySQL memory settings you would need to edit the /etc/my.cnf file and restart MySQL. There are many settings that are, or can be, in that file that affect memory utilization. MySQL has generic my-*.cnf files to use as a guideline typically stored in /usr/share/mysql/

    Depending on the number of sites on your server, and what those sites are doing, you may need to increase your RAM above 2GB.
    Tim Benoit

  6. #6
    Join Date
    Nov 2002
    Location
    Portland, Oregon
    Posts
    2,992
    Most of these adjustments are made in /usr/local/lib/php.ini or my.cnf. Make a copy of these files before making any changes to it, even if it's not correctly configured -- as a rollback measure.

  7. #7
    Join Date
    Apr 2012
    Posts
    75
    Here is my my.cnf file configuration

    [mysqld]
    default-storage-engine=myisam
    low_priority_updates=1
    max_connections = 100
    key_buffer = 192M
    tmpdir=/home/tmp
    skip-external-locking
    max_allowed_packet = 16M
    table_cache = 4096
    sort_buffer_size = 2M
    read_buffer_size = 2M
    read_rnd_buffer_size = 2M
    myisam_sort_buffer_size = 64M
    thread_cache_size = 8
    query_cache_type = 1
    query_cache_size= 128M
    query_cache_limit = 1M
    thread_concurrency = 8
    wait_timeout = 30
    open_files_limit=17506
    tmp_table_size = 256M
    max_heap_table_size = 256M
    join_buffer_size = 32M
    table_definition_cache=1024

    slow-query-log=/var/lib/mysql/slow.log

    [mysqldump]
    quick
    max_allowed_packet = 16M

    [mysql]
    no-auto-rehash


    [isamchk]
    key_buffer = 128M
    sort_buffer_size = 128M
    read_buffer = 2M
    write_buffer = 2M

    [myisamchk]
    key_buffer = 128M
    sort_buffer_size = 128M
    read_buffer = 2M


    local-infile=0

    And here is the details from mysqltuner


    -------- General Statistics --------------------------------------------------
    [--] Skipped version check for MySQLTuner script
    [OK] Currently running supported MySQL version 5.5.32-cll
    [OK] Operating on 64-bit architecture

    -------- Storage Engine Statistics -------------------------------------------
    [--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
    [--] Data in MyISAM tables: 2G (Tables: 2560)
    [--] Data in InnoDB tables: 15M (Tables: 581)
    [--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
    [--] Data in MEMORY tables: 0B (Tables: 4)
    [!!] Total fragmented tables: 573

    -------- Performance Metrics -------------------------------------------------
    [--] Up for: 1h 17m 54s (28K q [6.196 qps], 944 conn, TX: 117M, RX: 4M)
    [--] Reads / Writes: 87% / 13%
    [--] Total buffers: 720.0M global + 38.2M per thread (100 max threads)
    [!!] Maximum possible memory usage: 4.4G (226% of installed RAM)
    [OK] Slow queries: 0% (2/28K)
    [OK] Highest usage of available connections: 8% (8/100)
    [OK] Key buffer size / total MyISAM indexes: 192.0M/376.6M
    [OK] Key buffer hit rate: 100.0% (21M cached / 2K reads)
    [OK] Query cache efficiency: 67.8% (16K cached / 24K selects)
    [OK] Query cache prunes per day: 0
    [OK] Sorts requiring temporary tables: 0% (0 temp sorts / 1K sorts)
    [!!] Joins performed without indexes: 74
    [OK] Temporary tables created on disk: 23% (785 on disk / 3K total)
    [OK] Thread cache hit rate: 99% (8 created / 944 connections)
    [!!] Table cache hit rate: 5% (3K open / 54K opened)
    [OK] Open file limit used: 29% (5K/17K)
    [OK] Table locks acquired immediately: 100% (10K immediate / 10K locks)
    [OK] InnoDB data size / buffer pool: 15.6M/128.0M

    -------- Recommendations -----------------------------------------------------
    General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    MySQL started within last 24 hours - recommendations may be inaccurate
    Reduce your overall MySQL memory footprint for system stability
    Enable the slow query log to troubleshoot bad queries
    Adjust your join queries to always utilize indexes
    Increase table_cache gradually to avoid file descriptor limits
    Variables to adjust:
    *** MySQL's maximum memory usage is dangerously high ***
    *** Add RAM before increasing MySQL buffer variables ***
    join_buffer_size (> 32.0M, or always use indexes with joins)
    table_cache (> 4096)
    I would appreciate it if you can guide me on the right part to edit in the cnf file.
    Thanks

  8. #8
    Join Date
    Nov 2003
    Location
    Texas, United States
    Posts
    115
    It highlighted the immediate issues fairly well:

    1) Lots of tables are fragmented. This could be OK or could be bad. It would be a good idea to optimize them.

    2) You have MySQL configured to use to to 4.4GB of RAM, however your server only has a total of 2GB of RAM. You will need to tweak down your memory parameters in my.cnf (use the example files I mentioned previously) and/or upgrade the RAM in your server. These would be for the *buffer* and *cache* variables in the file.

    3) There have been some joins performed on non-indexed fields in the database. This means the query was performing a full table scan for those queries. Not optimal but may be the responsibility of the script developer to optimize their queries.

    You could also enable slow query logging to help show you what queries performed were running "slow".
    Tim Benoit

  9. #9
    Join Date
    Apr 2012
    Posts
    75
    Quote Originally Posted by tbenoit View Post
    It highlighted the immediate issues fairly well:

    1) Lots of tables are fragmented. This could be OK or could be bad. It would be a good idea to optimize them.

    2) You have MySQL configured to use to to 4.4GB of RAM, however your server only has a total of 2GB of RAM. You will need to tweak down your memory parameters in my.cnf (use the example files I mentioned previously) and/or upgrade the RAM in your server. These would be for the *buffer* and *cache* variables in the file.

    3) There have been some joins performed on non-indexed fields in the database. This means the query was performing a full table scan for those queries. Not optimal but may be the responsibility of the script developer to optimize their queries.

    You could also enable slow query logging to help show you what queries performed were running "slow".
    Hello,
    thanks alot for your support.'Just that I am lost here.
    Can you please tell me where excatly I should edit on the cnf file ? if possible, please make the changes and give me the right one to use so that I can just over right the existing one.
    Sorry for my noob questions please.
    Thanks

  10. #10
    Join Date
    Nov 2003
    Location
    Texas, United States
    Posts
    115
    Sent you a PM
    Tim Benoit

  11. #11
    Join Date
    Jan 2003
    Location
    Canada
    Posts
    5,073
    You're 3GB into swap.

    You need to upgrade your RAM.

    Francisco
    BuyVM - OpenVZ & KVM Based VPS Servers - Chat with us
    - All popular VPN methods supported
    - Affordable offloaded MySQL & DDoS protection
    - 5GB backup space, unmetered private LAN bandwidth & native IPv6 included. All with a strong serving of pony

  12. #12
    it is a bit unusual when your server eat up such amount of swap. Either your server or the host node run out of memory...
    ControlVM.com :: Elastic Cloud Hosting Alternatives to VPS Reliable Cloud Server
    Enterprise Hosting @
    Malaysia ● Germany ● USA ● Singapore
    We Accept: Paypal
    Alipay GrabPay Credit/Debit Card FPX Bank Transfer Bank TT

  13. #13
    Join Date
    Dec 2007
    Location
    LocalHost
    Posts
    1,317
    Can you please post current load and memory usage
    Code:
    top

    Code:
    free -m
    YagHost - Fast Reliable Hosting Since 2009
    Managed VPS - NVMe DirectAdmin
    Web Hosting - NVMe SSD, AMD EPYC, 10 Gbps (US, Europe, Singapore)

  14. #14
    Join Date
    Apr 2012
    Posts
    75
    root@bestservers [~]# top
    top - 07:50:27 up 10:26, 1 user, load average: 0.07, 0.06, 0.07
    Tasks: 201 total, 1 running, 198 sleeping, 0 stopped, 2 zombie
    Cpu(s): 0.4%us, 0.3%sy, 0.0%ni, 98.7%id, 0.3%wa, 0.3%hi, 0.1%si, 0.0%st
    Mem: 2058696k total, 1882508k used, 176188k free, 87584k buffers
    Swap: 4095992k total, 112k used, 4095880k free, 840980k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    14434 root 15 0 12764 1196 840 R 0.3 0.1 0:00.26 top
    1 root 15 0 10372 696 588 S 0.0 0.0 0:01.82 init
    2 root RT -5 0 0 0 S 0.0 0.0 0:00.57 migration/0
    3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
    4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
    5 root RT -5 0 0 0 S 0.0 0.0 0:00.51 migration/1
    6 root 34 19 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/1
    7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
    8 root RT -5 0 0 0 S 0.0 0.0 0:00.47 migration/2
    9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2
    10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2
    11 root RT -5 0 0 0 S 0.0 0.0 0:00.57 migration/3
    12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3
    13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3
    14 root 10 -5 0 0 0 S 0.0 0.0 0:00.01 events/0
    15 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/1
    16 root 10 -5 0 0 0 S 0.0 0.0 0:00.05 events/2

  15. #15
    Join Date
    Dec 2007
    Location
    LocalHost
    Posts
    1,317
    Quote Originally Posted by Lordemma View Post
    root@bestservers [~]# top
    top - 07:50:27 up 10:26, 1 user, load average: 0.07, 0.06, 0.07
    Tasks: 201 total, 1 running, 198 sleeping, 0 stopped, 2 zombie
    Cpu(s): 0.4%us, 0.3%sy, 0.0%ni, 98.7%id, 0.3%wa, 0.3%hi, 0.1%si, 0.0%st
    Mem: 2058696k total, 1882508k used, 176188k free, 87584k buffers
    Swap: 4095992k total, 112k used, 4095880k free, 840980k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    14434 root 15 0 12764 1196 840 R 0.3 0.1 0:00.26 top
    1 root 15 0 10372 696 588 S 0.0 0.0 0:01.82 init
    2 root RT -5 0 0 0 S 0.0 0.0 0:00.57 migration/0
    3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
    4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
    5 root RT -5 0 0 0 S 0.0 0.0 0:00.51 migration/1
    6 root 34 19 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/1
    7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
    8 root RT -5 0 0 0 S 0.0 0.0 0:00.47 migration/2
    9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2
    10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2
    11 root RT -5 0 0 0 S 0.0 0.0 0:00.57 migration/3
    12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3
    13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3
    14 root 10 -5 0 0 0 S 0.0 0.0 0:00.01 events/0
    15 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/1
    16 root 10 -5 0 0 0 S 0.0 0.0 0:00.05 events/2
    CPU load is OK.
    Memory usage is also OK. But you should consider adding few more RAM.

    I cannot find any process using too much memory.
    YagHost - Fast Reliable Hosting Since 2009
    Managed VPS - NVMe DirectAdmin
    Web Hosting - NVMe SSD, AMD EPYC, 10 Gbps (US, Europe, Singapore)

  16. #16
    Join Date
    Apr 2012
    Posts
    75
    I noticed the site eating up the memory so I had to suspend them....
    I just enabled them now and the server is dead again

  17. #17
    Join Date
    Apr 2012
    Posts
    75
    Please look into my cnf above and give me the recommended settings to use please.
    The server is on Xen vitualization with Centus 5

  18. #18
    Join Date
    Jun 2009
    Location
    Catalonia
    Posts
    15
    Quote Originally Posted by Lordemma View Post
    Please look into my cnf above and give me the recommended settings to use please.
    The server is on Xen vitualization with Centus 5
    Hello,

    I think there are several values too high, particularly join_buffer_size = 32M. The default value for this variable is 128K. Low it to 1 or 2MB and, I think, at least you will be able to start Mysql without crashing the server. You can see how a change in a variable can affect your memory consumption here: http://www.mysqlcalculator.com/.

    Also, install and run the script tuning-primer.sh and see (after 48 hours running Mysql) what it recommends for optimizing your Mysql, you can compare it with Mysqltuner results. http://webhostingneeds.com/Install_M..._Primer_Script

    The maximum memory usage displayed by Mysqltuner should not exceed 50% of available installed RAM, you can run Mysqltuner after changing any value to see if it fits in this limit.

    Good luck.

  19. #19
    Join Date
    Apr 2012
    Posts
    75
    my.cnf file edited to the following


    [mysqld]
    default-storage-engine=myisam
    low_priority_updates=1
    max_connections = 100
    key_buffer = 192M
    tmpdir=/home/tmp
    skip-external-locking
    max_allowed_packet = 1M
    table_cache = 2M
    sort_buffer_size = 1M
    read_buffer_size = 1M
    read_rnd_buffer_size = 2M
    myisam_sort_buffer_size = 64M
    thread_cache_size = 8
    query_cache_type = 1
    query_cache_size= 64M
    query_cache_limit = 1M
    thread_concurrency = 8
    wait_timeout = 30
    open_files_limit=17506
    tmp_table_size = 256M
    max_heap_table_size = 256M
    join_buffer_size = 1M
    table_definition_cache=1024

    slow-query-log=/var/lib/mysql/slow.log

    [mysqldump]
    quick
    max_allowed_packet = 2M

    [mysql]
    no-auto-rehash


    [isamchk]
    key_buffer = 128M
    sort_buffer_size = 128M
    read_buffer = 1M
    write_buffer = 1M

    [myisamchk]
    key_buffer = 128M
    sort_buffer_size = 128M
    read_buffer = 1M


    Please help on how o optimize further.
    Thanks All

  20. #20
    Join Date
    Dec 2007
    Location
    LocalHost
    Posts
    1,317
    Quote Originally Posted by Lordemma View Post
    I noticed the site eating up the memory so I had to suspend them....
    I just enabled them now and the server is dead again
    Try enabling sites one-by-one. And find sites which are causing most problem.

    Optimization can work upto certain limits only. If you websites need more RAM / CPU, you should consider upgrading your VPS.

    I suggest you to keep heavy websites on a separate VPS so that your normal websites can run smoothly.
    YagHost - Fast Reliable Hosting Since 2009
    Managed VPS - NVMe DirectAdmin
    Web Hosting - NVMe SSD, AMD EPYC, 10 Gbps (US, Europe, Singapore)

  21. #21
    Join Date
    Mar 2012
    Posts
    1,421
    Just a quick note here:
    You have been dealing a few days with this. While is good to ask for help in the forum, nothing bad about this, if you had chosen to request service from a "Server Management" company/business, you could have used this time to manage other more important things, other than doing the trial and error and experimenting.

    IMHO.
    --

Similar Threads

  1. My server just completely overloaded? DDOS?
    By chasebug in forum Hosting Security and Technology
    Replies: 5
    Last Post: 09-19-2010, 12:23 PM
  2. Server At PixelFX Completely Down!!
    By chiehkai in forum Providers and Network Outages and Updates
    Replies: 8
    Last Post: 07-05-2006, 08:41 PM
  3. Server completely messed up.
    By Coldmedia in forum Dedicated Server
    Replies: 2
    Last Post: 04-09-2006, 01:32 PM
  4. remove domain name from server completely
    By mshannaq in forum Dedicated Server
    Replies: 1
    Last Post: 09-15-2005, 05:37 AM
  5. Replies: 10
    Last Post: 12-11-2003, 12:11 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •