Results 1 to 18 of 18
  1. #1

    How to show a poor performing VPS through logs?

    Hi everyone,

    I'm having an issue with my CentOS 5.5 VPS being sluggish, and I've opened a ticket for it with my provider. However, they want logs that prove the system is slow.

    Right now, the server should be very lightly loaded - it only has a private webserver and a public mailserver for my domain, with only about 80 emails per day (no attachments). However, operations like saving a file through "vi" (2400 bytes) can take 10 seconds, doing an "ls" on a directory with only 2 files can take 5 seconds, and exiting from my root shell back to the user shell can take 5 seconds as well.

    What kind of logs can I use to prove I'm getting this poor performance?

  2. #2
    Join Date
    May 2009
    Location
    London, United Kingdom
    Posts
    472
    You can test I/O with the following command:
    dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync

    If you get less than 10mb there is something wrong, probably overloaded Node or very slow drive(s).
    KnownSRV.com - Privacy. Managed. Secure. Guaranteed!
    Web Hosting | Dedicated Cloud VPS | Dedicated Server
    YOUR Day and Night, Fully Managed Hosting Solutions with REAL 24/7 Support

  3. #3
    I'm getting reasonable disk performance based on that test - 14 to 32 MB/s over several tests.

    However, doing an "ls" took 14 seconds on a directory with only 1 files!

    [root@myserver tmp]# date;ls;date
    Tue Apr 26 13:11:40 PDT 2011
    test
    Tue Apr 26 13:11:54 PDT 2011

  4. #4
    Join Date
    Mar 2010
    Location
    Germany
    Posts
    697
    I suggest you take in an admin for a short moment to drill down to the problem, especially if it's a non-obvious one.

    To go with your specific question the kind of log I'd try is running "sar" in the collector mode. Google for it and be sure it includes the disk stats.
    That way you can grab data for a whole day or longer.

    it sounds like every *write* is delayed, so it would be very interesting to watch the disk response times.
    Check out my SSD guides for Samsung, HGST (Hitachi Global Storage) and Intel!

  5. #5
    Join Date
    Mar 2010
    Location
    Germany
    Posts
    697
    Quote Originally Posted by Skidmark View Post
    Hi everyone,

    I'm having an issue with my CentOS 5.5 VPS being sluggish, and I've opened a ticket for it with my provider. However, they want logs that prove the system is slow.

    Right now, the server should be very lightly loaded - it only has a private webserver and a public mailserver for my domain, with only about 80 emails per day (no attachments). However, operations like saving a file through "vi" (2400 bytes) can take 10 seconds, doing an "ls" on a directory with only 2 files can take 5 seconds, and exiting from my root shell back to the user shell can take 5 seconds as well.

    What kind of logs can I use to prove I'm getting this poor performance?


    edit: also try to grab the number of swap in/out and context switches
    Check out my SSD guides for Samsung, HGST (Hitachi Global Storage) and Intel!

  6. #6
    Join Date
    Mar 2005
    Location
    Labrador, Canada
    Posts
    988
    Use 'time' to capture the runtime of commands, e.g., with the ls command:

    Code:
    [root@vps:~] cd /etc
    [root@vps:/etc] time ls
    ...
    
    real    0m0.002s
    user    0m0.001s
    sys     0m0.000s

  7. #7
    Join Date
    Mar 2010
    Location
    Germany
    Posts
    697
    Quote Originally Posted by sleddog View Post
    Use 'time' to capture the runtime of commands, e.g., with the ls command:

    Code:
    [root@vps:~] cd /etc
    [root@vps:/etc] time ls
    ...
    
    real    0m0.002s
    user    0m0.001s
    sys     0m0.000s
    time only starts counting when the process started to run, but not any delay with forking it. So it can sit for 30seconds and then say the process took 5s. Very annoying, I never tried time time ls though
    Check out my SSD guides for Samsung, HGST (Hitachi Global Storage) and Intel!

  8. #8
    Join Date
    Jul 2005
    Location
    Tulsa, Oklahoma
    Posts
    31
    Skidmark, one thing to learn is every minute/hour you put into trying to prove your current setup / config sucks.. you're just helping whomever you're paying make more $. Just step back, backup your data and find a new home.

    You'll thank me later for just finding a place that doesn't give you any reason to have to "prove" squat in logs. If its slow, they should look into it, regardless of logs. You said its slow.. they have a standard and if your standard and their standard don't meet.. I think its time to tell them to kick rocks
    Encept, a division of Zeruck, LLC.
    visit our site | chat with us online
    WE DO NOT OVERSELL AND WE ARE STABLE!

  9. #9
    Quote Originally Posted by wartungsfenster View Post
    time only starts counting when the process started to run, but not any delay with forking it.
    Interesting. Is that true both for the shell's built-in 'time' and for the external 'time' command (e.g., /usr/bin/time)?
    raindog308
    LowEndTalk administrator, LowEndBox editor

  10. #10
    Join Date
    Mar 2010
    Location
    Germany
    Posts
    697
    Quote Originally Posted by raindog308 View Post
    Interesting. Is that true both for the shell's built-in 'time' and for the external 'time' command (e.g., /usr/bin/time)?
    Good question. I should've looked into that more curiously but -
    I only had that issue on a debian lenny box when the software raid did it's periodic scan / resync.
    And that ran at Sunday night, so I did not really track it down, just worked around and opened a bug. They since fixed it to use ionice, so it won't happen like this any more.

    In an ideal world the buildin should work instantly, but maybe it would still be waiting for an update of the $HISTFILE? If you can wait a week or two I will test it and post back here since I'm interested now
    Check out my SSD guides for Samsung, HGST (Hitachi Global Storage) and Intel!

  11. #11
    Join Date
    Dec 2006
    Posts
    4,151
    I suggest you change your hosts.

    These problems sound like overcrowding on the node or abusive neighbors, neither of which will be easy to deal with.

  12. #12
    Join Date
    Apr 2011
    Location
    Las Vegas, NV
    Posts
    1,643
    Quote Originally Posted by Skidmark View Post
    I'm getting reasonable disk performance based on that test - 14 to 32 MB/s over several tests.

    However, doing an "ls" took 14 seconds on a directory with only 1 files!
    Sounds like an internal resource shortage, but it's unclear where the bottleneck is.

    I suggest that you install webmin. Once installed, click the System link and open Running processes. You can monitor memory & cpu resource usage in there, broken down by process. You should be able to learn something there.

    You might also look around /var/log/ for any rogue log files growing out of control. Sometimes a process will consume all of your resources doing verbose logging for no good reason (written by a programmer who deserves to be horsewhipped publicly).

  13. #13
    Join Date
    May 2006
    Location
    San Francisco
    Posts
    7,325
    If the slowdowns that you described are truly happening on the host side, then there's no need for you to "prove" the poor performance. That's as evident as it gets and asking for proof is just stalling the inevitable cancellation.

  14. #14
    Join Date
    Apr 2011
    Location
    Las Vegas, NV
    Posts
    1,643
    Quote Originally Posted by Orien View Post
    If the slowdowns that you described are truly happening on the host side, then there's no need for you to "prove" the poor performance. That's as evident as it gets and asking for proof is just stalling the inevitable cancellation.
    Oh sure. If it's on the host side then they are more aware of it than he is. But we haven't established where the problem is yet.

  15. #15
    I got the following sar results (not really clear on how to run it properly yet), but the %iowait seems a bit high at certain times - is this any kind of indication of a disk problem?

    12:10:24 AM CPU %user %nice %system %iowait %steal %idle
    12:30:12 AM all 0.01 0.00 0.01 12.61 0.00 87.37
    12:50:27 AM all 0.01 0.00 0.01 11.22 0.00 88.76
    01:00:19 AM all 0.00 0.00 0.01 3.07 0.00 96.91
    01:10:25 AM all 0.00 0.00 0.01 5.05 0.00 94.93
    01:30:48 AM all 0.00 0.00 0.01 12.61 0.00 87.38
    01:40:34 AM all 0.01 0.00 0.01 5.61 0.00 94.37
    01:50:17 AM all 0.00 0.00 0.01 2.73 0.00 97.26
    02:00:12 AM all 0.01 0.00 0.01 3.02 0.00 96.96
    02:11:09 AM all 0.00 0.00 0.01 9.61 0.00 90.38
    02:20:41 AM all 0.00 0.00 0.01 7.38 0.00 92.61
    02:30:14 AM all 0.01 0.00 0.01 4.53 0.00 95.45
    02:41:07 AM all 0.00 0.00 0.01 10.16 0.00 89.83
    02:50:38 AM all 0.01 0.00 0.01 6.21 0.00 93.77
    03:00:04 AM all 0.01 0.00 0.01 1.00 0.00 98.98
    03:10:13 AM all 0.00 0.00 0.01 2.51 0.00 97.48
    03:20:28 AM all 0.01 0.00 0.01 4.82 0.00 95.16
    03:30:06 AM all 0.26 0.00 0.20 0.87 0.00 98.67
    03:40:13 AM all 0.59 0.00 0.45 2.42 0.00 96.54
    03:50:01 AM all 0.01 0.00 0.01 0.21 0.00 99.78
    04:00:01 AM all 0.01 0.00 0.01 0.08 0.00 99.91
    04:10:05 AM all 0.00 0.00 0.01 0.72 0.00 99.27
    04:20:32 AM all 0.01 0.00 0.01 4.69 0.00 95.29
    04:30:09 AM all 0.00 0.00 0.01 1.47 0.00 98.52
    04:40:31 AM all 0.01 0.00 0.01 4.38 0.00 95.60
    04:50:07 AM all 0.00 0.00 0.01 2.31 0.00 97.68
    05:00:01 AM all 0.00 0.00 0.01 0.03 0.00 99.95
    05:10:01 AM all 0.00 0.00 0.01 0.00 0.00 99.99
    05:20:01 AM all 0.00 0.00 0.01 0.16 0.00 99.82
    05:30:01 AM all 0.00 0.00 0.01 0.37 0.00 99.62
    05:40:06 AM all 0.01 0.00 0.02 0.86 0.00 99.11
    05:50:01 AM all 0.00 0.00 0.01 0.00 0.00 99.99
    06:00:01 AM all 0.01 0.00 0.01 0.17 0.00 99.82
    06:10:02 AM all 0.00 0.00 0.01 0.38 0.00 99.61
    06:20:51 AM all 0.00 0.00 0.01 7.04 0.00 92.95
    06:30:55 AM all 0.00 0.00 0.01 10.59 0.00 89.40
    06:50:37 AM all 0.01 0.00 0.01 12.39 0.00 87.59
    07:00:03 AM all 0.00 0.00 0.01 0.59 0.00 99.40
    07:10:13 AM all 0.01 0.00 0.01 2.57 0.00 97.41
    07:20:04 AM all 0.00 0.00 0.01 0.58 0.00 99.41
    07:30:02 AM all 0.01 0.00 0.01 0.30 0.00 99.68
    07:40:05 AM all 0.01 0.00 0.01 0.59 0.00 99.39
    07:50:01 AM all 0.00 0.00 0.01 0.00 0.00 99.99
    08:00:02 AM all 0.01 0.00 0.01 0.16 0.00 99.83
    08:10:01 AM all 0.00 0.00 0.01 0.00 0.00 99.99
    08:20:01 AM all 0.01 0.00 0.02 0.00 0.00 99.97
    08:30:01 AM all 0.00 0.00 0.01 0.00 0.00 99.99
    08:40:01 AM all 0.00 0.00 0.01 0.28 0.00 99.71
    08:50:01 AM all 0.01 0.00 0.01 0.00 0.00 99.99
    09:00:02 AM all 0.01 0.00 0.03 0.44 0.00 99.52
    09:10:01 AM all 0.00 0.00 0.01 0.01 0.00 99.98
    09:20:04 AM all 0.00 0.00 0.01 0.46 0.00 99.53
    09:30:21 AM all 0.00 0.00 0.01 3.67 0.00 96.32
    09:40:02 AM all 0.01 0.00 0.02 2.16 0.00 97.81
    Average: all 0.02 0.00 0.02 3.75 0.00 96.20

  16. #16
    Quote Originally Posted by wartungsfenster View Post
    I suggest you take in an admin for a short moment to drill down to the problem, especially if it's a non-obvious one.

    To go with your specific question the kind of log I'd try is running "sar" in the collector mode. Google for it and be sure it includes the disk stats.
    That way you can grab data for a whole day or longer.

    it sounds like every *write* is delayed, so it would be very interesting to watch the disk response times.
    I downloaded the sysstat package and modified the cron.d sa1 command to include the "-d" switch, but when I run "sar -d", I don't see any statistics.

  17. #17
    Join Date
    Mar 2005
    Location
    Orlando, Florida
    Posts
    2,625
    Quote Originally Posted by Skidmark View Post
    I'm getting reasonable disk performance based on that test - 14 to 32 MB/s over several tests.

    However, doing an "ls" took 14 seconds on a directory with only 1 files!

    [root@myserver tmp]# date;ls;date
    Tue Apr 26 13:11:40 PDT 2011
    test
    Tue Apr 26 13:11:54 PDT 2011
    14-32MB/s is actually pretty low. I would consider anything under 80-90MB/s to be very low. Also, looking at your iowait times, I'm seeing it spike between 4-15% or so -- again, that's simply too high in my opinion.
    Matthew Rosenblatt, and I do lots of things.
    Used to be a full time server administrator, now I help build cruise ships and inspect homes.
    My company, Ferrell Solutions, specializes in home inspections and property management.
    RecallScan is a service for monitoring appliances and vehicles in your home for recalls.

  18. #18
    Join Date
    Jan 2011
    Location
    Canada
    Posts
    938
    Last time I had a report like that, someone was running a minecraft server on that same node. The other time was a raid rebuild. You ideally shouldn't have a iowait like that on an otherwise idle VPS.

    That said, I wouldn't call it an 'the house is burning down' type of log. The one with the minecraft server just noted it, responded that unless it got to 30%, it's acceptable and moved on. Granted, they later moved all the 'abusers' off that node, but that's another story

    If you can stomach it, I might suggest installing munin-node on it as well to gather more data. It helps make more sense of it when you can correlate things like fork-rate and context switches to the cpu graph.

Similar Threads

  1. [UK] The highest performing VPS
    By reiteration in forum VPS Hosting Offers
    Replies: 0
    Last Post: 12-01-2009, 06:34 PM
  2. Is my VPS performing bad?
    By adobo in forum VPS Hosting
    Replies: 10
    Last Post: 10-28-2009, 05:18 AM
  3. Poor show from LaneHost
    By Allosunshine in forum Reseller Hosting
    Replies: 17
    Last Post: 05-16-2009, 03:35 AM
  4. Server logs that show outage?
    By tgo149 in forum Web Hosting
    Replies: 2
    Last Post: 11-30-2004, 06:28 PM
  5. Can someone logon a server and not show up on logs
    By biTe in forum Dedicated Server
    Replies: 16
    Last Post: 06-03-2004, 05:23 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •