Is there any means of having a realtime uncompression engine which uncompresses on the fly Zipped HTML files to the browser of the customer. Is this a feasible thing? RS servers for instance have minimum 1Ghz CPU. Especially for a site nearing a million hits per month. I've heard of mod_gzip (or something similar) but it seems that not many people use it. Am I right?

I have been using mod_gzip for the last 9 months ... it works great. It can compress (hence save your bandwith cost) anywhere from 0%-300%. I had this one customer doing 1GB/day with mod_gzip enabled and I just curiously disabled mod_gzip and he was doing around 2.7GB/day ... that's a lot of saving for him, not for me (hence I left mod_gzip off ;) )

cheers,
:beer:

wow, 300% - that'd be impressive. Your servers must operate in a tear in our mathmatical universe! Seriously, we get 0-90% compression - it's a great program.

On what platform can it run. Is it difficult to install? What about incompatibilities? Where to find it, where are the best resources, are there any referring forums?

mod_gzip is a apache module....

http://www.remotecommunications.com/apache/mod_gzip/

and it doesn't need any cleint software. any http 1.1 compilent browser will support it...

I think mod_gzip is great and a must-have. It's a pity that it cannot compress jpg and gif images, otherwise the savings would be more impressive :D

Even with mod_gzip installed, one of my big sites is still eating over 4 GB/day. I can't imagine how much it would eat if I don't install mod_gzip.

"I just curiously disabled mod_gzip and he was doing around 2.7GB/day ... that's a lot of saving for him, not for me (hence I left mod_gzip off "

Hummmmm :eek:

Originally posted by twrs
It's a pity that it cannot compress jpg and gif images, otherwise the savings would be more impressive :D


Both .gif and .jpg are already compressed formats. That's why mod_gzip skips them.

If you want to prove it to your self, just try creating a .zip file on your PC with .gif's and .jpg's. At best you'll normally only get a couple of percent compression.

Frank

Originally posted by ffeingol
Both .gif and .jpg are already compressed formats. That's why mod_gzip skips them.


Yeah, I know that. I was just kidding :D

But if I install mod_gzip. Will it slow down the server a lot?

Originally posted by maxbear
But if I install mod_gzip. Will it slow down the server a lot?

I don't think so (not a lot, at least). In fact, people will think that your server is actually faster ... why? Consider this ... Suppose you have 100KB file to be gzipped ... It will, say, take 1 second to compress it ... Let's say the compressed file is around 50KB. Then comes your visitor trying to download that 100KB file with his 56kbps modem ... say he/she downloads that file at exactly 56kbps or 8KBps. Then without zipping your file first, your visitors will need about 12.5 seconds to download the file, and only about 7.5 seconds (download and zipping the file) if you compress the file ... He/she will think your web server must be faster by 40% :D

cheers,
:beer:

Is there a way to tell how much bandwidth you saved by installing mod_zip?

The easiest way is to use webalizer ... Run mod_gzip for several days and then turn it off for several days. You can compare your average daily data transfer with and without mod_gzip that way.

Similar way is to use MRTG to compare your daily traffic with and without mod_gzip.

I used 4 different methods to compare this: MRTG, webalizer, bandmin, mod_watch ... With mod_gzip installed, MRTG and webalizer will be almost identical, but differ from bandmin and mod_watch. It seems to me bandmin and mod_watch calculates bandwith prior as if there were no compression (Once I disabled mod_gzip, MRTG, webalizer, bandmin, and mod_watch all agree one another).

cheers,
:beer:

You could also install mgstat (http://wizard.ae.krakow.pl/~mike/mgstat/). You'll need to add an extra log file to Apache, but then you can see the compressed and un-compressed numbers.

Frank

Originally posted by ffeingol
You could also install mgstat (http://wizard.ae.krakow.pl/~mike/mgstat/). You'll need to add an extra log file to Apache, but then you can see the compressed and un-compressed numbers.


That's very useful, thanks a lot!

cheers,
:beer:

Originally posted by ffeingol
Both .gif and .jpg are already compressed formats. That's why mod_gzip skips them.

If you want to prove it to your self, just try creating a .zip file on your PC with .gif's and .jpg's. At best you'll normally only get a couple of percent compression.

Frank LOL!! Somebody should explain this to A'lmost O'n L'ine...

I installed AOL (and have to pay the damn monthly fee) so I can be on the same page as some of my clients whom I design sites for.. What that company does to our images is CLEARLY in violation of copyright ie: "alter" they compress images to a horrendous level with the "explanation" that they do it to improve their client's load times and speed up the net.. BS! Theydo it to save their precious bandwidth! It is disableable within the AOL preff's section but I seriously doubt that 90% of the AOL'ers are aware of it.

For an example, it took an image (a photo) that was about 59k, which in the 14.4 days would have been a huge no no but is ok now.. that image on AOL was only 19 kb and it stunk! Fuzzy and jaggy, an abomination of the photographers work.. I get a haaaad on about them caching pages.. I suppose that has been somewhat accepted even though it is in violation of copyright technically.. but the alteration of images is waaay more clearly out of line with the spirit and letter of usage permissions given or not given...

Mod gzip does sound good though.. How is it on CPU cycles on a busy site?

Shortcompressed

Hey shortfork - I was just going to post about AOL when I flipped the page and saw your post! It used to really bother me that AOL would botch my images. Now I try to ignore them, but it still irritates me. The webmaster carefully compresses his images to the perfect balance between size and quality, taking time to get each one just right, and AOL comes along and says "ah, those webmasters don't know what they're doing - we can do it better with this one-size-fits-all button. Button on - saving $$ - who cares, our users will just think that's the way the web is.

I was even irked that AOL is sueing Microsoft... for doing exactly what they do themselves. Bundling... have you noticed that Free AOL icon that gets stuck on the desktop again and again with Netscape and a variety of other software? And don't they kind of push their own features with keywords, etc. in front of their competitors products?

Anyway, sorry to take the thread off topic. I try not to get started talking about all the ways AOL bugs me :angry:

Originally posted by shortfork
Mod gzip does sound good though.. How is it on CPU cycles on a busy site?


I don't think it affects the CPU cycles much. With mod_gzip installed, I hardly get loads above 0.2. One of the sites I host on this server get over 15,000 unique visitors/day so I think it could be considered as a busy site.

Well getting ready to do this one but still investigating all the angles.. I loaded up Almost On Line and visited a couple of modgizpped sites and their browser seems to be able to handle it (at least the latest version can)... so that seems to be out of the way..

How bout this one... Search engines... I depend on the little robots to be able to read my pages, especially Google, which seems to ignore descriptions and kewords, relying more on page title and content within the page.. (probably a good thing but semi frustrating after all these years to perfect those meta-tags ey)

Anyway... do the robots have any trouble reading the pages? Do they get them gzipped or are they able to read them without?

Anyone have a solid answer on this one? I get a lot of visits monthly from google queries, ranked right up in the top 3 in many categories that relate to my main site.. would hate to lose that ranking if robots had problems with modgzip...

TIA
Shortness

If the agent which loads your pages isnt able to handle gzipped data, mod_gzip will send it normal uncompressed data.

Plus, you can control mod_gzips behaviour very detailled in the config file, although it's a little bit hard to understand.

Marc

Very cool.. more and more this seems like a good way to speed up loading and slow down bandwidth.. A much better method than proxy caching to speed up the net! But that's another story for another thread!

Shortness

Let's say your site is currently *not* mod_gzip enabled...does enabling it require any shaffling around of your original site files? I want to add this feature to my site, and hope it's a "turn of the light switch" kind of operation.

Originally posted by GeorgeC
Let's say your site is currently *not* mod_gzip enabled...does enabling it require any shaffling around of your original site files? I want to add this feature to my site, and hope it's a "turn of the light switch" kind of operation.

No, you don't need to do anything with your current files, mod_gzip can do it on the fly.

Originally posted by priyadi


No, you don't need to do anything with your current files, mod_gzip can do it on the fly.

Actually, what I meant was, do your site files need to be shuffled around, such as moving them to another directory? I remember reading that somewhere.

Thanks!

No, it just requires installing it, then it compesses all text files that go through apache.

I would say use MOD_GZIP we use it on ever server works great for us.

Now Rapidcolo, why would you want to use it? Wouldn't it be better for you if your customers use more bandwith so that you can charge your customers for extra bandwith usage? :eek:

cheers,
:beer:

Originally posted by GeorgeC
Actually, what I meant was, do your site files need to be shuffled around, such as moving them to another directory? I remember reading that somewhere.

No ... if you have mod_gzip enabled, then all files being sent through your web server (for eg. apache on port 80) would be automatically compressed. This won't be the case for files transfered through your FTP server or even other ports such as 3306 (doing remote mysql, for example). So, mod_gzip only stricly works along side with your web server.

cheers,
:beer:

Originally posted by dektong
Now Rapidcolo, why would you want to use it? Wouldn't it be better for you if your customers use more bandwith so that you can charge your customers for extra bandwith usage? :eek:

cheers,
:beer:


While it is better for you, it doesn't mean it is better for the customer. And if your customers know this is your reason, then I have to say you are screwed :(. You should come up with another reason why you don't install mod_gzip. You surely can't answer customers question with that.

Actually, I also installed mod_gzip on my servers, but just one server where I host mainly VB forums ... and these forums are very busy (about 3-4 millions of page views/mo), so I disabled mod_gzip which will actually have an impact on server load. Besides, VB has its own build in gzip modules (where you can actually set the gzip level anywhere from 0 to 9).

BTW, does anybody know how to set gzip level on mod_Gzip? By default, mod_gzip set its gzip level at 6, but I don't think there is much bandwith to be save going from 6 down to 3 or 4 (which use less CPU time especially for high performance server).

cheers,
:beer:

I believe you can change mod_gzip compression level by altering the source (mod_gzip.c). It is on line 9579 on my version of mod_gzip. Haven't tried that though.