Results 1 to 20 of 20
  1. #1

    Please Educate Me on SCSI/RAID

    Hi Guys:

    Run a large ubbthreads, with osCommerce and phpauctions as well. Mysql database is almost 1 GB, and just my one Post table is 800 MB. Usually max out at 400-500 concurrent users online, but spikes have gone to 800-plus.

    Serious mysql usage, and growing.

    Sat Nov 4 15:49:58 CST 2006 top - 15:49:58 up 1 day, 20:32, 0 users,
    load average: 0.05, 0.07, 0.08
    Tasks: 137 total, 1 running, 136 sleeping, 0 stopped, 0 zombie
    Cpu(s): 0.2% us, 0.9% sy, 4.1% ni, 94.7% id, 0.1% wa, 0.0% hi, 0.0% si
    Mem: 4090580k total, 2177868k used, 1912712k free, 204024k buffers
    Swap: 2096440k total, 0k used, 2096440k free, 1442816k cached

    Currently running single CPU/dual core AMD 170, dual SATA II 250 (one backup), and 4 GB RAM.

    Looking to upgrade to dual CPU/dual core AMD 270, dual 73 GB 10,000 rpm SCSI with RAID 1, and 4 GB RAM.

    1] Will the two SCSI be appreciably faster than two SATA II 250 (7200 rpm).

    2] Is RAID 1 the right (or only) way to go for my needs?

    3] Am I better off keeping my current AMD 170 with SATA II and 4 GB for mysql only, and then getting another AMD 170 with 2 GB RAM for http and other processes, or is another CPU + SCSI RAID swap for SATA II where the horses are?

    What would you do?

    Thanks a BUNCH!

    Rick

  2. #2
    Join Date
    Dec 2004
    Location
    New York, NY
    Posts
    10,574
    1] Will the two SCSI be appreciably faster than two SATA II 250 (7200 rpm).
    Yes, for sure - especially when you've got a 1 GB database.

    2] Is RAID 1 the right (or only) way to go for my needs?
    It's decent. You won't exactly be getting a performance gain, mainly just reliability.

    3] Am I better off keeping my current AMD 170 with SATA II and 4 GB for mysql only, and then getting another AMD 170 with 2 GB RAM for http and other processes, or is another CPU + SCSI RAID swap for SATA II where the horses are?
    That may be a good idea (separating the services), but it sounds like SCSI disks could really be of good use to you.

    Now if this were me, I'd look at WebNX for the servers.

    http://www.webhostingtalk.com/showth...ighlight=webnx - some great offers there and you'll find that it's much more economical than taking a base server at a provider like say SoftLayer and customizing it -- add-ons are not cheap with large providers like that.

    Look at this amazing deal:

    **forum / large DB site dual server combo**

    * DC165 w/ 8x 15k 36gb (DB) and Dual Opteron 250 (Web)
    Do you have a site that keeps maxing out the db server? Give this combo a try or upgrade it. You get a dual core opteron 165 with 4gb ram, and 8x 36gb 15k rpm scsi drives with hardware raid 10. On the webserver side you get a Dual Opteron 250 with 4gb ram, and 2x 36gb 15k scsi drives.
    Connected via 1gbps link on the second NIC
    With 10mbps unmetered on the webserver $375 for both
    Free Direct Admin on the webserver
    I'm not sure if they still have it in stock, but it would be worth asking them about that. Your site would fly, and then some, with boxes like that.
    MediaLayer, LLC - www.medialayer.com Learn how we can make your website load faster, translating to better conversion rates for your business!
    The pioneers of optimized web hosting, featuring LiteSpeed Web Server & SSD Storage - Celebrating 10 Years in Business

  3. #3
    FYI you don't need any more RAM, 2GB is fine from the 'top' you showed
    EuroVPS VPS Hosting - Virtual Private Servers | Web Hosting | Dedicated Servers
    Providing Reliable Plesk and cPanel Servers since 2004, now offering low priced Xen & VMware VPS in Amsterdam
    UK +44.203.355.6681 / Amsterdam +31.208.202.120

  4. #4
    Join Date
    Jun 2004
    Location
    Bay Area
    Posts
    1,320
    Why would you want to change servers? The top you posted shows an extreme low load...that machine should be able to hadle your site fine?

  5. #5
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,294
    If you read the previous thread he said he was relasing some magazine article and could get alot of traffic.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  6. #6
    Notice the uptime on that top. Server crashed for 18 hours Friday. At peak times, it uses virtually all 4GB of RAM.

    Had mysql errors to various tables from/by/with "errno". One last night too.

    I'm considering a range of options, from adding SCSI drives to my current server, to moving up to a dual core/dual CPU, to going whole hog and spiltting the load between two machines.

    I need to get a sense for how big, truly, a 1 GB database is, with one 800MB table that gets hit over and over by 400-500 users.

    Is that just an unwieldly mysql table, no matter what I throw at it in terms of hardware? I'm seeing more and more mysql crashes, and ... well, I can't have that.

    Is hardware the answer?

    Rick

  7. #7
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,294
    Quote Originally Posted by 24hourcampfire
    Notice the uptime on that top. Server crashed for 18 hours Friday. At peak times, it uses virtually all 4GB of RAM.

    Had mysql errors to various tables from/by/with "errno". One last night too.

    I'm considering a range of options, from adding SCSI drives to my current server, to moving up to a dual core/dual CPU, to going whole hog and spiltting the load between two machines.

    I need to get a sense for how big, truly, a 1 GB database is, with one 800MB table that gets hit over and over by 400-500 users.

    Is that just an unwieldly mysql table, no matter what I throw at it in terms of hardware? I'm seeing more and more mysql crashes, and ... well, I can't have that.

    Is hardware the answer?

    Rick


    Mysql on a separate server is what I would suggest, along with a correctly optimized my.cnf. I looked at the one you posted prior and it was not that great.

    Ignoring what other people are saying, I would go with a bigger beefier setup, it will give you space to grow and prevent you from having to upgrade as quick.


    A 1gb database is nothing at all, right now I have a 14gb database running on a:


    Single Dual Core Opteron 180
    4gb of ram
    8x 36gb 15k scsi drives raid 10

    1500-2000 users online all day long.

    [[email protected] ~]# uptime
    05:10:48 up 9 days, 12:10, 2 users, load average: 0.28, 0.44, 0.43
    [[email protected] ~]#



    I think a optimzation of your server would help, but if you plan to grow, i would definitely go with a dual server setup.
    Last edited by Steven; 11-05-2006 at 02:11 PM.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  8. #8
    I'm all ears on optimizing the server. I know Dick Tracy about how to do it, other than replacing my.cnf with my-huge.cnf.

    I'm all ears!!!

    Rick

  9. #9
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,294
    Quote Originally Posted by 24hourcampfire
    I'm all ears on optimizing the server. I know Dick Tracy about how to do it, other than replacing my.cnf with my-huge.cnf.

    I'm all ears!!!

    Rick
    Unfortunately its not as easy as just saying here use this, need to know whats going on with queries etc to get a good optimization done. There are other things that be can be done besides the my.cnf aswell, kernel changes, php accelerators, etc.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  10. #10
    Join Date
    Feb 2004
    Location
    Sofia
    Posts
    1,349
    Quote Originally Posted by 24hourcampfire
    Currently running single CPU/dual core AMD 170, dual SATA II 250 (one backup), and 4 GB RAM.
    No need to upgrade so far, at least not before trying to separate web and mysql onto different drives. Use the second drive for MySQL only and procure some offserver backup drive.
    :: :: :: :: :: ::
    :: VDSP.Net :: Directory of virtual and dedi serv providers by location and price

  11. #11
    Join Date
    Jun 2002
    Location
    sydney.au
    Posts
    246
    Any database that is smaller than memory counts as small.

    Looking at your initial post, I thought that the one thing that isn't a problem is memory... But if you normally see all the memory in use, that's not right.

    There's a constant problem for database-driven web apps in memory contention between the web server and the database. With a 1GB database, you only need 1GB to cache the whole thing, and when a MySQL database is cached it runs very fast indeed.

    But if you get lots of PHP sessions running all at once, Apache can easily chew through a few GB of memory - and flush the database from cache, screwing your performance when you need it most.

    There's two ways to approach this. One, spend a lot of time tinkering with Apache and MySQL to try to control their memory use. Or two, get another server for the database.

    See if you can find out how much memory is being used by Apache at peak times. That will help point to a solution.

  12. #12
    Join Date
    Feb 2004
    Location
    Sofia
    Posts
    1,349
    There are at least four concurrent db's in MySQL each of them exceding 2 gigs (this is my RAM) and I have RAM bottlenecks only when the audit.d logs consume the whole space alotted to /var

    Have to manually delete the logs every other week...
    :: :: :: :: :: ::
    :: VDSP.Net :: Directory of virtual and dedi serv providers by location and price

  13. #13
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,294
    Why dont you get rid of audit? are you using it for anything specifically?
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  14. #14
    Join Date
    Feb 2004
    Location
    Sofia
    Posts
    1,349
    No. Should I simply delete it?
    :: :: :: :: :: ::
    :: VDSP.Net :: Directory of virtual and dedi serv providers by location and price

  15. #15
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,294
    Quote Originally Posted by Imago
    No. Should I simply delete it?

    You dont need it running.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  16. #16
    Join Date
    Feb 2004
    Location
    Sofia
    Posts
    1,349
    Thanks, Steven!
    :: :: :: :: :: ::
    :: VDSP.Net :: Directory of virtual and dedi serv providers by location and price

  17. #17
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,294
    i found this for you: http://www.eth0.us/auditd
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  18. #18
    Join Date
    Feb 2004
    Location
    Sofia
    Posts
    1,349
    Wonderful!
    :: :: :: :: :: ::
    :: VDSP.Net :: Directory of virtual and dedi serv providers by location and price

  19. #19
    Quote Originally Posted by Imago
    There are at least four concurrent db's in MySQL each of them exceding 2 gigs (this is my RAM) and I have RAM bottlenecks only when the audit.d logs consume the whole space alotted to /var

    Have to manually delete the logs every other week...
    It's not only the size of your database that counts, it's also how the database is structured and used. For example, if you drop some of the database indexes, you might notice dramatic decrease in performance even though the database will become a little smaller
    :: Mountain Network Systems :: 323-933-9291
    eCommerce solutions since 1995
    http://www.webcart.net/

  20. #20
    Quote Originally Posted by 24hourcampfire
    I'm all ears on optimizing the server. I know Dick Tracy about how to do it, other than replacing my.cnf with my-huge.cnf.

    I'm all ears!!!
    You need to figure out where the bottleneck is.
    If php scripts consume too much memory during peak times, you should notice a heavy usage of swap (I don't think that's the case).
    If your limiting factor is MySQL (much more likely) you should see a lot of MySQL connections in locked state during the peaks. The real challenge will be to figure out whether the load is caused by a certain feature on your site or by the number of users. If it's former, the hardware upgrade is probably not the best path you could choose - you should consider optimizing your software instead or otherwise disabling that part of functionality.
    :: Mountain Network Systems :: 323-933-9291
    eCommerce solutions since 1995
    http://www.webcart.net/

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •