Results 1 to 11 of 11

Thread: Why is it slow?

  1. #1

    Why is it slow?

    I have a dedicated server with essentially a single site. It's fairly heavily trafficed - over 500K page views a day. The site is largely (80%+) dynamically generated with servlets and MySQL. The database is actually on another machine.

    Most of the time the server is blazingly fast. Sub second responses. During the afternoons (peak usage time) it slows to a virtual crawl with 10 and even 20 second responses.

    The thing that I'm trying to figure out is why. Even during slow times I'll usually see a load average under 1 and over 50% free cpu. Pings and traceroutes are still just as fast as other times. The server is otherwise just as responsive - ssh sessions, compiles etc show no slowdown.

    During a slowdown stopping and restarting apache seems to have little if any affect.

    Server is a pIII w/ 512M, redhat 7.1, apache 1.3.27, java 1.4.1, tomcat 4.1.12, php 4.2.3

    Any ideas? I'd assume the load was just too great for the machine except for the fact that everything other than web serving is pretty much still fast. Serving all pages is slowed even static pages and/or php pages that don't access the DB.

    Need more memory? free says:
    mem total 512640 used 506716 free 5924
    +/- buffers/cache: used 470300 free 42340
    swap: total 530104 used 89756 free 440348

  2. #2
    paste some apache settings dealing with requests per server, maxservers etc. also, java=bad for high-traffic stuff.
    * Rusko Enterprises LLC - Upgrade to 100% uptime today!
    * Premium NYC collocation and custom dedicated servers
    call 1-877-MY-RUSKO or paul [at] rusko.us

    dedicated servers, collocation, load balanced and high availability clusters

  3. #3
    Upgrade to a P4 2 GHz with 512 cache and over 1 GB of RAM and you will see a considerable performance increase. You may also want to check on the remote database server to ensure it can handle all of those queries.

    With that many page view perhaps consider small machines with load balancing. Or if money permits, a single sparc machine can handle that load.

  4. #4
    Join Date
    Sep 2000
    Location
    New York/USA
    Posts
    1,690
    Yes, If you're running a heavy site utilizing MySQL and Servlet's, I suggest upgrading a 1 GB of ram. Processor shouldn't matter too much though as long as it's 1 Ghz+. I also suggest getting someone to tweak your MySQL settings to be optimized on your machine.

  5. #5
    Join Date
    Sep 2002
    Location
    Canada
    Posts
    53
    Ugh. All those suggestions are baloney.

    It would seem to me that it's your DB server that's getting nailed. Which is why your webserver is keeping such a low load average.

    This is also typical of a MySQL DB with myisam tables, where slow down are exponetial.

    Have someone look at the DB server. For starters run myisamchk against your database files. I'd also suggest moving to InnoDB tables, or even PostgreSQL if it wasn't too inconvienent. Then look at hardware -on your DB server-, not your webserver.


    And don't just restart Apache.. That's a windows thing to do, and it's icky.
    UpNIX Internet Administrator
    --
    http://www.upnix.com

  6. #6
    Join Date
    Sep 2002
    Location
    Mansfield
    Posts
    314
    Ditto. Might want to take a look at the network connection between the 2 boxes - are they on a private network or the same subnet?

    Upnix, like your email policies. I actually got sued by a spammer today. <sniff> I'm so proud....my first lawsuit for blocking spam...
    GUI admin tools have no honor. It is a good day to vi.

  7. #7
    Upgrade to a P4 2 GHz with 512 cache and over 1 GB of RAM and you will see a considerable performance increase.
    Sure and it'll really scream if I run it on a E10,000.

    If you're running a heavy site utilizing MySQL and Servlet's, I suggest upgrading a 1 GB of ram.
    MySQL is on another machine. The DB machine is even less loaded than the web machine. Each have 512MB and yes, they're directly connected.

    It would seem to me that it's your DB server that's getting nailed.
    I doubt it. As I said the load on the DB machine is even lower than on the web server. Plus as I said originally even static pages take a long time to load. Plus I can connect to the db directly or run scripts against it and it's just as fast as non-peak times.

    Before I had the db on the same machine as the webserver and things were spinning out of control. Load averages would climb above 10 frequently and you could see a huge backlog of requests in MySQL. Now load average on both machines are low (though the web server gets higher than the db server) and there's almost never more than 2 queries running at the same time.

    The apache restart thing was suggested by the host tech support who thought it might be lots of apache connections that needed to be cleared out.

    Not sure if this is what you're looking for but:
    MaxRequestsPerChild 0 (ie infinate)
    MaxClients 150

  8. #8
    Join Date
    Sep 2002
    Location
    Canada
    Posts
    53
    MySQL is on another machine. The DB machine is even less loaded than the web machine. Each have 512MB and yes, they're directly connected.
    So, both machines have low load averages, but pages still come out at a crawl?

    Anything to suggest it's the network that's getting saturated? Your SSH connections slow? Lots of collisions (You should be on a switch)?

    Looking at your original post, it does look like your webserver could use more RAM, but if it were a big problem, I'd expect the load average to be much higher.

    Not sure if this is what you're looking for but:
    MaxRequestsPerChild 0 (ie infinate)
    MaxClients 150
    Check your error logs for messages about hitting that limit, but if you were hitting it you'd get an error page rather than just slow pages.


    Hard to diagnose without actually being there, but things I'd do (I'm not a Linux expert, so these may not all work):

    On the webserver:
    - Run top and just watch processes go by. Things to look for would be httpd's eating more than a few percent CPU. This would suggest that the dynamic pages are of issue. I find with my server, rarely does a httpd process use more than 1% CPU (Yes these are dynamic pages)

    - 'netstat -i' and look at 'TX-ERR'. (Which I believe is collisions). It should be next to 0 if you're connected through a switch. A highnumber would indicate that your traffic is running into each other, which slows things down (obviously)

    - 'iostat' - Should list the average throughput of various things. Should have your HD device listed there, look to see if that number is high (like over 10 megs/s). On the webserver, this may only confirm you need more ram.

    - Run 'ntop'. You'll probably have to download it, but it should give you an idea of where bandwidth is getting spent.

    You can do all this on the DB server to, but what I'd look at specifically is the mysqld process. In top look for mysqld using 'D' (I think) under stat in top. Which would suggest it's waiting for disk access.

    And like before, if you're not already, try using InnoDB tables. Should really smooth out MySQL's preformance when things get busy.

    If I think of anymore I'll let you know. Most of these problems should show up as high load average, which is why this is a bit interesting.

    Should also note that I know next to nothing about Tomcat, so there may be something obvious here I'm not getting.

    Good Luck.

    Chris
    UpNIX Internet Administrator
    --
    http://www.upnix.com

  9. #9
    The top suggestion doesn't quite work since servlets are run from within the http process. Well a small bit of code to connect to the servlet engine is. At any rate there are no high cpu httpd processes. The java processes are also mostly low cpu. There tends to be one which I assume is some sort of dispatcher that generates a lot of CPU time when the system's been up for a while but it doesn't consume large portions of the CPU at any given moment.

    netstat -i showed 0 errors on both the public interface and the interface to the db machine.

    iostat is a new one for me. Reads 92 blks/s Writes 59 blks/s. I don't think either is overly high.

    I tried not using mod_gzip but that actually seemed to make the slowdown worse.

    Restarting apache seemed to free a lot of memory though it starts creeping up slowly but steadily.

    Because of that I set the MaxRequestsPerChild to 10000. Doesn't seem to have slowed the memory growth though.

    I looked for signs that I was reaching the 150 MaxClients limit but saw nothing in the error logs. Counting the number of running httpd processes gave me 150 so I decided to set it up higher anyway - to 250. That seems to have worked. The load average went up and the idle cpu went down but the site is WAY faster.

    Oddly enough it seems very responsive and the load average is low when it's using less than the max number of processes but the load average spikes if it goes hits the 250. I'll try playing with that number a bit. I'm sure there's a number that's too many but I suspect 250 may still be too low. Hopefully there's not a 255 limit.

  10. #10
    Join Date
    Sep 2002
    Location
    Mansfield
    Posts
    314
    Some versions are 256 and others are 512 but it's' a compile time option on Apache 1.3.X
    GUI admin tools have no honor. It is a good day to vi.

  11. #11
    Yeah I found that out.

    Looks like I may have spoke too soon. Now I've got 256 processes running and the load average is around 2 with no idle CPU and it's slow again. Looks like I need a faster CPU afterall. I feel a lot better about upgrading a machine that's actually heavily utilized. I wonder what kind of CPU it's going to take to get this thing running well under peak loads? Or possibly I should investigate Apache 2 and see if threads helps the load at all?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •