Hi,

I'm having difficulty understanding why there are such major differences between the bandwidth used according to mod_throttle and Webalizer.

Webalizer says I've used 3.5mb on one domain, and mod_throttle says I've used over 5mb. I am using mod_gzip, but shouldn't mod_throttle's stats be lower than Webalizer? I thought any difference would be because Webalizer doesn't understand compressed pages(I might be wrong).

But if that were the case, Webalizer would be producing stats for the data throughput before compression. mod_throttle should be producing actual transfer stats.

Any ideas? If anyone could reverse the situation - so my customers sites don't hit their 3gb bandwidth limit when Webalizer is reporting 2.2gb used, I would be extremely grateful.

Thanks,

Rich

Originally posted by Inspa.net
Webalizer says I've used 3.5mb on one domain, and mod_throttle says I've used over 5mb. I am using mod_gzip, but shouldn't mod_throttle's stats be lower than Webalizer? I thought any difference would be because Webalizer doesn't understand compressed pages(I might be wrong).

What is recorded into the apache logs is the amount of data sent to the client, meaning after compression.

It seems that what is recored into mod_throttle is the amount of data uncompressed.

I have the same exact problem regarding the differences between mod_throttle and webalizer... I have posted my question here, http://webhostingtalk.com/showthread.php?threadid=11517 , but nobody has answered that....

in summary, here is what I got (by May 24th, around 16:30pm).


mod_throttle
Hits: 450060
KB Transfered: 3560061

webalizer
Hits: 746617
KB Transfered: 1506424


I have an inclination to agree with energy... anybody can confirm this?

[edit note]
the numbers of hits does not differ by a factor of 6. I miscounted the number of digits for the hits as recorded by webalizer
[/edit note]

cheers,
:beer:

Thanks for that guys. Does anybody know if mod_bandwidth suffers the same problem?

Rich

my client has generated a newer webalizer record, so here the comparison goes (as of May 27th, 10 pm):

mod_throttle
hits: 1081125
KB transferd: 7317884

webalizer
hits: 1381699
KB transferd: 2364916

Thus, with respect to the logs on May 24th, the difference (delta) in hits and KB transfered for both methods are:

mod_throttle
delta hits: 631065
delta KB transfered: 3757823

webalizer
delta hits: 635082
delta KB transfered: 858492

Therefore, mod_throttle and webalizer practically recorded the same number of hits (as one should expects) but the amount of data transfered as recorded by mod_throttle is about 4.4 times more than that recorded by webalizer. Is this factor (4.4) the usual compression factor due to mod_gzip?

Further note: there does not seem to be any difference whether I loaded mod_throttle before or after I loaded mod_gzip.

cheers,
:beer:

the amount of data transfered as recorded by mod_throttle is about 4.4 times more than that recorded by webalizer. Is this factor (4.4) the usual compression factor due to mod_gzip

I believe so. It's a shame that mod_throttle doesn't measure actual bandwidth used.

Rich

As the author of mod_throttle I can explain this I think: in the Apache API, the request_rec structure has a field called bytes_sent and mod_throttle records this information as given to it by Apache. Therefore there might be two cases why the totals are wrong (that come to mind):

a) mod_gzip does NOT update the bytes_sent field of the request_rec.

b) mod_throttle see the request_rec structure, before mod_gzip has a chance to update bytes_sent.

In the case of b) this can be as a result of the LoadModule and AddModule order. However, there have been bug reports to mod_gzip and mod_throttle concerning module load order. I currently have mod_gzip before mod_throttle in the httpd.conf file. However flipping them causes problems (whose problem it is is yet undetermined since I haven't had time to investigate it).

Programs like http-analyser and co. use the web server logs to generate their results and can be customised for custom log formats; or the log format used specifies the mod_gzip value in place of the uncompressed value. In either case these tools obtain their data from a different source.

When development on mod_throttle/4.0 gets underway, I hope to address this in conjunction with the authors of mod_gzip.

Anthony Howe

Anthony, does mod_watch count the uncompressed transfer the same way mod_throttle does?

mod_watch logs bytes in exactly the same way as mod_throttle, since it too uses the bytes_sent field from the request_rec structure filled in by Apache. If mod_gzip doesn't update these values before the logging phase then the values reported will be uncompressed.

The authors of mod_gzip might have a means of passing the compressed size information between phases and modules, so if they tell me what to look for in the "notes" table I could update both mod_watch and mod_throttle to reference it. (I won't look in their source untill I resume development on mod_throttle/4.0 at which time I'll address this issue somehow.) Actually, come to think of it the "notes" table entries are those used for the LogFormat I think, so this should be easy to support.

In mod_watch's case I could offer another set of values to graph; for example you could create graphs that showed the uncompressed as the input and the compressed bytes as the output to see just how much compression you get on average.

Anthony Howe

Originally posted by achowe
Actually, come to think of it the "notes" table entries are those used for the LogFormat I think, so this should be easy to support.

Maybe you should take a look at : http://www.remotecommunications.com/apache/mod_gzip/src/1.3.19.1a/commands.txt.

It says that you can use '%{mod_gzip_output_size}n' in LogFormat to see the compressed size. Does that help?

I noticed that mod_throttle counts the percentage of data transfer inccorectly once it is a high number of Kbytes.
For example, it shows the following:

%: 9
Hits: 12319749
KBytes sent: 52405184
Volume: 100G

It should be around 51%, not 9%.
Is this a known bug? Can it be fixed easily?

The mod_gzip_output_size reference looks like the right beast.

The other thing mentioned appears to be a previously unreported bug. Email it a reminder directly to me.