Have you ever wanted to count the number of log entries grouped by a particular field? For example, reporting how many hits you've received per IP address in an Apache or Nginx log file, or perhaps sort by the response type?

Installation

Simply add the following histogram function to your ~/.bashrc file:

## Usage: histrogram [fieldnum]
##
## Produces a console-friendly histogram of piped input data.
##
## By default, the input data will be grouped based on the first field.
## You can optionally specify a space-separated field number.
## This function honours strings with spaces encapsulated by double quotes
##
## eg. sudo cat /var/log/httpd/access.log | histrogram
## eg. sudo cat /var/log/nginx/access.log | histrogram 7
##
function histrogram() {  
  field=1
  if [ ! -z "$1" ]; then field=$1; fi

  # honour strings with spaces surrounded with double quotes
  awk -F\" '{OFS="\"";for(i=2;i<NF;i+=2)gsub(/ /,"_",$i);print}' | \
  awk -v field=$field -v width=$width -v padding=$padding '
    function max(arr, big) {
      big = 0;
      for (i in cat) {
        if (cat[i] > big) { big=cat[i]; }
      }
      return big
    }
    NF > 0 {
      cat[$field]++;
    }
    END {
      maxm = max(cat);
      for (i in cat) {
        scaled = 40 * cat[i] / maxm;
        printf "%-15.15s  [%8d]:", i, cat[i]
        for (i=0; i<scaled; i++) {
          printf "#";
        }
    printf "\n";
      }
    }
  ' | \
  sort -nrk3
}

Usage

Get the top five hits by IP address (the first field)

$ sudo cat /var/log/httpd/access.log | histrogram | head -n5
6.12.101.123     [    2361]:########################################  
2.222.8.170      [    1153]:####################  
90.216.134.194   [     150]:###  
98.115.128.4     [      56]:#  
91.188.124.225   [      35]:#  

Get a report of the HTTP response codes (the seventh field)

$ sudo cat /var/log/nginx/access.log | histrogram 7
200              [    2741]:########################################  
304              [     894]:##############  
301              [     247]:####  
404              [      57]:#  
400              [      51]:#  
206              [      26]:#  
302              [      13]:#  
499              [       8]:#  

Find out where your attacks are coming from (source IP is field eleven)

$ sudo grep 'Failed password for root' /var/log/secure | histogram
61.160.223.181   [     596]:########################################  
116.10.191.168   [     155]:###########  
116.10.191.223   [     109]:########  
116.10.191.225   [      83]:######  
61.174.51.202    [      78]:######  
...