Remstats Home


  About remstats
  Release Notes
  FAQ
  Conventions
  Prerequisites
  Installation
  Configuration
  Configuration Tools
  Servers
  Collectors
      log-collector
      nt-status-collector
      ping-collector
      port-collector
      remoteping-collector
      snmp-collector
      snmp-route-collector
      unix-status-collector
  Updater
  Monitors
  run-remstats
  CGIs
  do-traceroutes
  Miscellany
  Thank-you
  Index


  Live Data

  Remstats was written by
  Thomas Erskine at the
  CRC in Canada and now
  at SourceWorks.

[Top] [Prev] [Next]

log-collector - get stats from remote log-files

Usage:

Can't locate RRDs.pm in @INC (@INC contains: . /usr/lib/remstats/lib /usr/lib/perl5/ /usr/local/lib/perl/5.6.1 /usr/local/share/perl/5.6.1 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.6.1 /usr/share/perl/5.6.1 /usr/local/lib/site_perl) at ../log-collector line 33.
BEGIN failed--compilation aborted at ../log-collector line 33.

Description:

The log-collector gets data from remote log-server's. This way the whole log-file doesn't have to be transfered. The "protocol", if it deserves that name, is very simple. The collector sends a request, which looks like (you can type it in via telnet):

	LOGFILE /wherever/the/logfile.is
	varname	function pattern
	...
	GO

The directives are all in UPPERCASE. They are LOGFILE, GO, DEBUG and TEST. The LOGFILE directive tells the log-server which file to read. The GO directive starts the request. DEBUG causes some extra remote debugging output, and TEST makes the log-server operate in test mode. In test mode it doesn't update the last-read position for that log-file, so you won't lose any data when testing.

The other lines are telling the log-server what data to collect. The first "word" is the variable name to be returned. The next is the function to be applied (from count, sum, average, min, max, first and last). The rest of the line is a perl-style regex. Except for the count function, the regex must contain a (parenthesized) number, to which the function will be applied.

For example, the line:

	rootlogins count ROOT LOGIN

would return data for a variable called rootlogins. The value would be the count of the records in the specified logfile which had the string 'ROOT LOGIN' in them.

The pattern can be much more complicated, for example (from the httpdlog rrd):

	bytes sum \sHTTP/\d\.\d"\s+2\d\d\s+(\d+)

This looks through a standard web-server log-file and extracts the bytes transferred and adds them up to produce the total number of bytes transferred in that sample period.

How to make RRDs that use the log-collector

It's easiest to explain by example. Look at the beginning of the rrd httpdlog, copied here:

	source  log
	step    300
	data    requests        GAUGE:600:0:U   count (GET|POST)
	data    success         GAUGE:600:0:U   count \sHTTP/\d\.\d"\s+2\d\d
	data    bytes           GAUGE:600:0:U   sum \sHTTP/\d\.\d"\s+2\d\d\s+(\d+)

To form the requests to be sent to the log-server, the log-collector takes the DS name, e.g. success, and the last part of the line after all the DS definition count \sHTTP/\d\.\d"\s+2\d\d, combines the two and sends:

	success count \sHTTP/\d\.\d"\s+2\d\d

Note that the pattern can include magic cookies as of remstats version 0.12.2.


Last updated Thu Mar 17 16:59:14 UTC 2005 by <thomas.erskine@sourceworks.com>.
[SourceWorks] [Top] [RRDtool]