hosted by liquidweb


Go Back   Web Hosting Talk : Web Hosting Main Forums : Web Hosting : Recommended size of log partition in Linux
Reply

Forum Jump

Recommended size of log partition in Linux

Reply Post New Thread In Web Hosting Subscription
 
Send news tip View All Posts Thread Tools Search this Thread Display Modes
  #1  
Old
Newbie
 
Join Date: Feb 2012
Posts: 11
Smile

Recommended size of log partition in Linux


Hi
On a busy LAMP-server, where all logs from Apache are stored under /var/log (but mysql is not logging transactions), what size would you recommend the /var/log partition to be? I want to keep logs for several years (say 10 years), but compress them using logrotate and gzip. And I want to keep all logs, not just apache but also syslog, authlog, daemon etc. I also have logs from other software like snort, mod_security and other.

Any ideas? is 10gb overkill?



Sponsored Links
  #2  
Old
Disabled
 
Join Date: Nov 2010
Posts: 2,566
Why do you like to keep all logs for that long? If you really like to, probably you need 1TB or more depends how many logs your website generate every day.

  #3  
Old
Newbie
 
Join Date: Feb 2012
Posts: 11
Quote:
Originally Posted by Lincxu View Post
Why do you like to keep all logs for that long? If you really like to, probably you need 1TB or more depends how many logs your website generate every day.
Personally I wouldn't want to keep them for long but my client says it's legel requirements.

1TB sounds a little bit excessive though. Sure it would mean I would not have to worry about disk space, but I was just wondering if anyone had any experience with typical log partition sizes.

Sponsored Links
  #4  
Old
Dependable Web Services
 
Join Date: Jun 2010
Location: Modesto, Ca. - USA
Posts: 5,414
Quote:
Originally Posted by infalible View Post
Personally I wouldn't want to keep them for long but my client says it's legel requirements.

1TB sounds a little bit excessive though. Sure it would mean I would not have to worry about disk space, but I was just wondering if anyone had any experience with typical log partition sizes.
10 years is a bit excessive as the life of the actual server is probably going to be 2-3 years max (at least before some type of major upgrade has to be made, like replacing the HD's or getting a new/faster server.

__________________
Fernando Diaz Alfaro - Owner of FernGullyGraphics
Shared and Reseller Web Hosting Services - Web Design - Domain Names
WHM/Cpanel - Cloud Linux - Varnish Cache - Wordpress - Magento - Site Builders
Click Here to Learn More About Our Services!- Check Out Our Youtube Channel

  #5  
Old
Newbie
 
Join Date: Feb 2012
Posts: 11
Quote:
Originally Posted by FernGullyGraphics View Post
10 years is a bit excessive as the life of the actual server is probably going to be 2-3 years max (at least before some type of major upgrade has to be made, like replacing the HD's or getting a new/faster server.
I think 10 years is reasonable. The server is pretty fast (6 core cpu and 48gb ram). Replacing HD is also no problem (virtual disks).

But for the sake of getting my question answered, assume 2-3 years then.

  #6  
Old
Always Ask...Don't Pretend!
 
Join Date: Aug 2010
Location: CPU
Posts: 2,182
If your customer tells you that this is legal requirement then tell him that money is also a requirement to keep this logs. Make sure to bill him for 2TB at least you have enough buffer and income for keeping the logs and make it clear that the cost is incremental for certain threshold.

__________________
Ask for Server IP & Nameservers IP to check if your reseller provider truly provides 100% white-label.


  #7  
Old
Newbie
 
Join Date: Feb 2012
Posts: 11
Thanks!
But I am really more interested in finding out how much the /var/log usage would grow in one day on a very busy server. If anyone out there can share some numbers it would be great.

  #8  
Old
Web Hosting Evangelist
 
Join Date: Dec 2009
Location: Milan, IT
Posts: 475
What mean "busy" for you? How many hits you you have a present? Your logs are likely to be larger if there are a lot of requestes per pages...

To give you an example, I have 350-400MB of compressed logs each day for a very busy site.... :-)

__________________
Prometeus.com & iperweb.com Hosting Solutions since 1997
We are LIR (RIPE NCC member) and do BGP with ASN 34971 Multiple gigabit/s network capacity
iwStack.com IaaS Cloud services

  #9  
Old
New Member
 
Join Date: Feb 2012
Posts: 2
Disk partitioning is the creation of separate divisions of a hard disk drive using partition editors such as fdisk. Once a disk is divided into several partitions, directories and files of different categories may be stored in different partitions.Many new Linux sys admin (or Windows admin) create only two partitions / (root) and swap for entire hard drive.

  #10  
Old
Newbie
 
Join Date: Feb 2012
Posts: 11
Quote:
Originally Posted by iperweb View Post
What mean "busy" for you? How many hits you you have a present? Your logs are likely to be larger if there are a lot of requestes per pages...
It's a new domain so it's not yet launched. When I say busy I mean more than a million hits per day. (and I know that is nothing compared to sites that generate over a billion hits a day)

Quote:
To give you an example, I have 350-400MB of compressed logs each day for a very busy site.... :-)
Thanks, that's exactly the kind of answer i was hoping for!

  #11  
Old
Web Hosting Evangelist
 
Join Date: Dec 2009
Location: Milan, IT
Posts: 475
Ok looking at some site with that numbers I see that 1.5 million of rows are 25M compressed...

__________________
Prometeus.com & iperweb.com Hosting Solutions since 1997
We are LIR (RIPE NCC member) and do BGP with ASN 34971 Multiple gigabit/s network capacity
iwStack.com IaaS Cloud services

  #12  
Old
Newbie
 
Join Date: Feb 2012
Posts: 11
Quote:
Originally Posted by iperweb View Post
Ok looking at some site with that numbers I see that 1.5 million of rows are 25M compressed...
That's great! thanks again!

  #13  
Old
New Member
 
Join Date: Feb 2012
Posts: 1
Minimum 1 TB

  #14  
Old
Junior Guru Wannabe
 
Join Date: Mar 2011
Posts: 85
You should consider predicting the size of the logs based on the growth of your current logs. One of the framework that we host can log 10GB of logs per week for having 1k visitors per day. So to keep all the logs it would require 1TB of disk space per year.

__________________
Managed1.com - Affordable budget Fully Managed Hosting Services for individuals and business clients. Website Hosting. Domain Name management. High Availability Cloud servers - 99.9% Uptime, Self-Healing Failover, Load Balancing and Mirrored Backups.
Whisperdale.net - Internet Service Provider since 2007!

  #15  
Old
Newbie
 
Join Date: Feb 2012
Posts: 11
Quote:
Originally Posted by DmitrijCSR View Post
You should consider predicting the size of the logs based on the growth of your current logs. One of the framework that we host can log 10GB of logs per week for having 1k visitors per day. So to keep all the logs it would require 1TB of disk space per year.
Thanks, but counting the number of visitors doesn't say much about the log size. Apart from the problem of defining a visitor (based on IP? cookies?), there is the problem that one visitor alone could in theory cause the entire 10GB log data if he/she was very active on the website.

Reply

Similar Threads
Thread Thread Starter Forum Replies Last Post
Swap size recommended for redhat linux linuxlinux Hosting Security and Technology 10 12-18-2004 03:57 PM
Best Partition Size asc2000 Dedicated Server 12 01-29-2004 10:29 AM
linux command to check partition size and or disk size MarcD Hosting Security and Technology 7 08-01-2003 12:40 PM
Changing partition size? spiderman Hosting Security and Technology 2 05-03-2002 09:36 PM
Adjusting Swap Partition Size on Linux / RAQ3 Systems... Groo Dedicated Server 0 03-02-2001 10:17 PM

Related posts from TheWhir.com
Title Type Date Posted
Oracle Brings OpenStack for Oracle Linux to General Availability Web Hosting News 2014-09-25 14:36:43
Liquid Web Launches Storm Private Cloud Web Hosting News 2013-11-22 13:42:02
Web Hosting Sales and Promos Roundup - October 18, 2013 Web Hosting News 2014-05-23 15:42:44
The Linux Foundation Partners with Xen for Open Source Cloud Project Web Hosting News 2013-04-15 15:19:00
Big Data and Cloud Drive Enterprise Linux Adoption: Report Web Hosting News 2013-03-27 12:54:34


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Postbit Selector

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump
Login:
Log in with your username and password
Username:
Password:



Forgot Password?
WHT Host Brief Email:

We respect your privacy. We will never sell, rent, or give away your address to any outside party, ever.

Advertisement:
Web Hosting News:
WHT Membership
WHT Membership



 

X

Welcome to WebHostingTalk.com

Create your username to jump into the discussion!

WebHostingTalk.com is the largest, most influentual web hosting community on the Internet. Join us by filling in the form below.


(4 digit year)

Already a member?