Web Hosting Talk : Web Hosting Main Forums : Web Hosting : Recommended size of log partition in Linux

[infalible]

Hi
On a busy LAMP-server, where all logs from Apache are stored under /var/log (but mysql is not logging transactions), what size would you recommend the /var/log partition to be? I want to keep logs for several years (say 10 years), but compress them using logrotate and gzip. And I want to keep all logs, not just apache but also syslog, authlog, daemon etc. I also have logs from other software like snort, mod_security and other.

Any ideas? is 10gb overkill?

[Lincxu]

Why do you like to keep all logs for that long? If you really like to, probably you need 1TB or more depends how many logs your website generate every day.

[infalible]

Originally Posted by Lincxu

Why do you like to keep all logs for that long? If you really like to, probably you need 1TB or more depends how many logs your website generate every day.

Personally I wouldn't want to keep them for long but my client says it's legel requirements.

1TB sounds a little bit excessive though. Sure it would mean I would not have to worry about disk space, but I was just wondering if anyone had any experience with typical log partition sizes.

[FernGullyGraphics]

Originally Posted by infalible

Personally I wouldn't want to keep them for long but my client says it's legel requirements.

1TB sounds a little bit excessive though. Sure it would mean I would not have to worry about disk space, but I was just wondering if anyone had any experience with typical log partition sizes.

10 years is a bit excessive as the life of the actual server is probably going to be 2-3 years max (at least before some type of major upgrade has to be made, like replacing the HD's or getting a new/faster server.

[infalible]

Originally Posted by FernGullyGraphics

10 years is a bit excessive as the life of the actual server is probably going to be 2-3 years max (at least before some type of major upgrade has to be made, like replacing the HD's or getting a new/faster server.

I think 10 years is reasonable. The server is pretty fast (6 core cpu and 48gb ram). Replacing HD is also no problem (virtual disks).

But for the sake of getting my question answered, assume 2-3 years then.

[Yujin]

If your customer tells you that this is legal requirement then tell him that money is also a requirement to keep this logs. Make sure to bill him for 2TB at least you have enough buffer and income for keeping the logs and make it clear that the cost is incremental for certain threshold.

[infalible]

Thanks!
But I am really more interested in finding out how much the /var/log usage would grow in one day on a very busy server. If anyone out there can share some numbers it would be great.

[iperweb]

What mean "busy" for you? How many hits you you have a present? Your logs are likely to be larger if there are a lot of requestes per pages...

To give you an example, I have 350-400MB of compressed logs each day for a very busy site.... :-)

[nomiii]

Disk partitioning is the creation of separate divisions of a hard disk drive using partition editors such as fdisk. Once a disk is divided into several partitions, directories and files of different categories may be stored in different partitions.Many new Linux sys admin (or Windows admin) create only two partitions / (root) and swap for entire hard drive.

[infalible]

Originally Posted by iperweb

What mean "busy" for you? How many hits you you have a present? Your logs are likely to be larger if there are a lot of requestes per pages...

It's a new domain so it's not yet launched. When I say busy I mean more than a million hits per day. (and I know that is nothing compared to sites that generate over a billion hits a day)

To give you an example, I have 350-400MB of compressed logs each day for a very busy site.... :-)

Thanks, that's exactly the kind of answer i was hoping for!

[iperweb]

Ok looking at some site with that numbers I see that 1.5 million of rows are 25M compressed...

[infalible]

Originally Posted by iperweb

Ok looking at some site with that numbers I see that 1.5 million of rows are 25M compressed...

That's great! thanks again!

[paginalia]

Minimum 1 TB

[DmitrijCSR]

You should consider predicting the size of the logs based on the growth of your current logs. One of the framework that we host can log 10GB of logs per week for having 1k visitors per day. So to keep all the logs it would require 1TB of disk space per year.

[infalible]

Originally Posted by DmitrijCSR

You should consider predicting the size of the logs based on the growth of your current logs. One of the framework that we host can log 10GB of logs per week for having 1k visitors per day. So to keep all the logs it would require 1TB of disk space per year.

Thanks, but counting the number of visitors doesn't say much about the log size. Apart from the problem of defining a visitor (based on IP? cookies?), there is the problem that one visitor alone could in theory cause the entire 10GB log data if he/she was very active on the website.