I was surprised when I saw the length of the Chrome user agent string last week:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.X.Y.Z Safari/525.13
And in our logs:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.29 Safari/525.13
a user agent string of 119 characters. It looks quite a waste of space but is Google Chrome the only one? Surprisingly, Chrome is far from the worst.
Best of one of our log file:
awk -F\" '{print length($6)" "$6}' access.log
awk -F\" '{if ($6 > 200) print length($6)" "$6}' access.log
In those examples, the access.log file has this log format:
xxx.xxx.xxx.xxx \ www.domain.com - \ [15/Sep/2008:00:00:00 +0200] \ "GET / HTTP/1.1" 200 4242 \ "http://www.domain.com/" \ "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1"
If you take an average user agent string likes the Firefox one, you have a 91 charaters string.
awk -F\" '{if (length($6) > 120) print length($6)}' access.log | wc -l
awk -F\" '{if (length($6) > 120) SUM += length($6)-120} END {print SUM/1024/1024" Mo"}' access.log
This entry was written by , posted on September 22, 2008 at 7:48 am, filed under Logs, Web and tagged awk, bandwidth, user agent. Leave a comment or view the discussion at the permalink.
Logcheck is a tool to parse system logs and send summaries by email. It filters out logs with a regular expressions database to suppress common/normal entries.
Are you reading / checking your log files? Too many servers? logcheck will help you in this task and eliminates the noise.
# apt-get install logcheck Reading package lists... Done Building dependency tree... Done The following extra packages will be installed: lockfile-progs logtail Suggested packages: syslog-summary Recommended packages: logcheck-database The following NEW packages will be installed: lockfile-progs logcheck logtail 0 upgraded, 3 newly installed, 0 to remove and 6 not upgraded. Need to get 110kB of archives. After unpacking 428kB of additional disk space will be used. Do you want to continue [Y/n]?
Also install logcheck-database which contains lots of rules
# apt-get install logcheck-database
You can try it by executing the following command:
# su -s /bin/bash -c "/usr/sbin/logcheck" logcheck
Your mailbox should now contains a report from logcheck if some unusual log entries have been found.
This entry was written by , posted on September 16, 2008 at 11:22 am, filed under Logs and tagged logcheck. Leave a comment or view the discussion at the permalink.
To rotate your nginx log files, you can use the log file handler provided by FreeBSD: newsyslog.
/var/log/nginx-access.log 644 7 1024 * JC /var/run/nginx.pid /var/log/nginx-error.log 644 7 1024 * JC /var/run/nginx.pid
Before log rotation:
-rw-r--r-- 1 root wheel 104278002 Jul 16 11:35 nginx-access.log -rw-r--r-- 1 root wheel 1509531 Jul 16 11:17 nginx-error.log
After log rotation:
-rw-r--r-- 1 root wheel 967 Jul 16 12:42 nginx-access.log -rw-r--r-- 1 root wheel 5310443 Jul 16 12:41 nginx-access.log.0.bz2 -rw-r--r-- 1 root wheel 77 Jul 16 12:41 nginx-error.log -rw-r--r-- 1 root wheel 37552 Jul 16 12:41 nginx-error.log.0.bz2
This entry was written by , posted on July 17, 2008 at 7:15 am, filed under Logs and tagged FreeBSD, newsyslog, nginx. Leave a comment or view the discussion at the permalink.
Cronolog is log rotation program which gives you a lot of options to template the log destination files. The common use is to split logs by year / month / day.
Here is how to configure Apache to send log entries to cronolog :
CustomLog "|/usr/sbin/cronolog /home/log/apache2/%Y-%m-%d_domain.com_access.log" combined
This will create a log file named 2008-06-02_domain.com_access.log for today.
Cronolog reads log entries from standard input and writes them to the output file specified by your template.
CustomLog "|/usr/sbin/cronolog /home/log/apache2/%Y-%m_domain.com_access.log" combined
CustomLog "|/usr/sbin/cronolog /home/log/apache2/%Y-%W_domain.com_access.log" combined
CustomLog "|/usr/sbin/cronolog /home/log/apache2/%H_domain.com_access.log" combined
This entry was written by , posted on June 4, 2008 at 7:33 am, filed under Logs, http and tagged apache, cronolog. Leave a comment or view the discussion at the permalink.