Results 1 to 17 of 17
  1. #1

    Smile Recommended size of log partition in Linux

    Hi
    On a busy LAMP-server, where all logs from Apache are stored under /var/log (but mysql is not logging transactions), what size would you recommend the /var/log partition to be? I want to keep logs for several years (say 10 years), but compress them using logrotate and gzip. And I want to keep all logs, not just apache but also syslog, authlog, daemon etc. I also have logs from other software like snort, mod_security and other.

    Any ideas? is 10gb overkill?

  2. #2
    Why do you like to keep all logs for that long? If you really like to, probably you need 1TB or more depends how many logs your website generate every day.

  3. #3
    Quote Originally Posted by Lincxu View Post
    Why do you like to keep all logs for that long? If you really like to, probably you need 1TB or more depends how many logs your website generate every day.
    Personally I wouldn't want to keep them for long but my client says it's legel requirements.

    1TB sounds a little bit excessive though. Sure it would mean I would not have to worry about disk space, but I was just wondering if anyone had any experience with typical log partition sizes.

  4. #4
    Join Date
    Jun 2010
    Location
    Modesto California
    Posts
    6,858
    Quote Originally Posted by infalible View Post
    Personally I wouldn't want to keep them for long but my client says it's legel requirements.

    1TB sounds a little bit excessive though. Sure it would mean I would not have to worry about disk space, but I was just wondering if anyone had any experience with typical log partition sizes.
    10 years is a bit excessive as the life of the actual server is probably going to be 2-3 years max (at least before some type of major upgrade has to be made, like replacing the HD's or getting a new/faster server.
    Looking for an awesome VPS Offer? CLICK HERE

    "Knowing is not enough, we must apply. Willing is not enough, we must do." – Bruce Lee

  5. #5
    Quote Originally Posted by FernGullyGraphics View Post
    10 years is a bit excessive as the life of the actual server is probably going to be 2-3 years max (at least before some type of major upgrade has to be made, like replacing the HD's or getting a new/faster server.
    I think 10 years is reasonable. The server is pretty fast (6 core cpu and 48gb ram). Replacing HD is also no problem (virtual disks).

    But for the sake of getting my question answered, assume 2-3 years then.

  6. #6
    Join Date
    Aug 2010
    Location
    CPU
    Posts
    2,187
    If your customer tells you that this is legal requirement then tell him that money is also a requirement to keep this logs. Make sure to bill him for 2TB at least you have enough buffer and income for keeping the logs and make it clear that the cost is incremental for certain threshold.
    Ask for Server IP & Nameservers IP to check if your reseller provider truly provides 100% white-label.

  7. #7
    Thanks!
    But I am really more interested in finding out how much the /var/log usage would grow in one day on a very busy server. If anyone out there can share some numbers it would be great.

  8. #8
    Join Date
    Dec 2009
    Location
    Milan, IT
    Posts
    478
    What mean "busy" for you? How many hits you you have a present? Your logs are likely to be larger if there are a lot of requestes per pages...

    To give you an example, I have 350-400MB of compressed logs each day for a very busy site.... :-)
    Prometeus.com & iperweb.com Hosting Solutions since 1997
    We are LIR (RIPE NCC member) and do BGP with ASN 34971 Multiple gigabit/s network capacity
    iwStack.com IaaS Cloud services

  9. #9
    Disk partitioning is the creation of separate divisions of a hard disk drive using partition editors such as fdisk. Once a disk is divided into several partitions, directories and files of different categories may be stored in different partitions.Many new Linux sys admin (or Windows admin) create only two partitions / (root) and swap for entire hard drive.

  10. #10
    Quote Originally Posted by iperweb View Post
    What mean "busy" for you? How many hits you you have a present? Your logs are likely to be larger if there are a lot of requestes per pages...
    It's a new domain so it's not yet launched. When I say busy I mean more than a million hits per day. (and I know that is nothing compared to sites that generate over a billion hits a day)

    To give you an example, I have 350-400MB of compressed logs each day for a very busy site.... :-)
    Thanks, that's exactly the kind of answer i was hoping for!

  11. #11
    Join Date
    Dec 2009
    Location
    Milan, IT
    Posts
    478
    Ok looking at some site with that numbers I see that 1.5 million of rows are 25M compressed...
    Prometeus.com & iperweb.com Hosting Solutions since 1997
    We are LIR (RIPE NCC member) and do BGP with ASN 34971 Multiple gigabit/s network capacity
    iwStack.com IaaS Cloud services

  12. #12
    Quote Originally Posted by iperweb View Post
    Ok looking at some site with that numbers I see that 1.5 million of rows are 25M compressed...
    That's great! thanks again!

  13. #13
    Minimum 1 TB

  14. #14
    You should consider predicting the size of the logs based on the growth of your current logs. One of the framework that we host can log 10GB of logs per week for having 1k visitors per day. So to keep all the logs it would require 1TB of disk space per year.
    Managed1.com - Affordable budget Fully Managed Hosting Services for individuals and business clients. Website Hosting. Domain Name management. High Availability Cloud servers - 99.9% Uptime, Self-Healing Failover, Load Balancing and Mirrored Backups.
    Whisperdale.net - Internet Service Provider since 2007!

  15. #15
    Quote Originally Posted by DmitrijCSR View Post
    You should consider predicting the size of the logs based on the growth of your current logs. One of the framework that we host can log 10GB of logs per week for having 1k visitors per day. So to keep all the logs it would require 1TB of disk space per year.
    Thanks, but counting the number of visitors doesn't say much about the log size. Apart from the problem of defining a visitor (based on IP? cookies?), there is the problem that one visitor alone could in theory cause the entire 10GB log data if he/she was very active on the website.

  16. #16
    FYI, a related website that I host generated 600K hits yesterday, and 200GB bandwidth. Apache log is 142MB uncompressed, standard customlog.

  17. #17
    Quote Originally Posted by infalible View Post
    Hi
    On a busy LAMP-server, where all logs from Apache are stored under /var/log (but mysql is not logging transactions), what size would you recommend the /var/log partition to be? I want to keep logs for several years (say 10 years), but compress them using logrotate and gzip. And I want to keep all logs, not just apache but also syslog, authlog, daemon etc. I also have logs from other software like snort, mod_security and other.

    Any ideas? is 10gb overkill?
    If u really want to keep logs for several years then your /var partition's 1TB size is enough but u have to move them to other location because of future disaster and all logs from Apache is located under /var/log/httpd/

Similar Threads

  1. Swap size recommended for redhat linux
    By linuxlinux in forum Hosting Security and Technology
    Replies: 10
    Last Post: 12-18-2004, 03:57 PM
  2. Best Partition Size
    By asc2000 in forum Dedicated Server
    Replies: 12
    Last Post: 01-29-2004, 10:29 AM
  3. linux command to check partition size and or disk size
    By MarcD in forum Hosting Security and Technology
    Replies: 7
    Last Post: 08-01-2003, 12:40 PM
  4. Changing partition size?
    By spiderman in forum Hosting Security and Technology
    Replies: 2
    Last Post: 05-03-2002, 09:36 PM
  5. Replies: 0
    Last Post: 03-02-2001, 10:17 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •