nixCraft

jaysunn · #1 (**permalink**) 18-09-2009, 02:13 AM

Thought this was cool. We needed a shell script to count the unique IP's in a apache access log that appeared multiple times. I came up with this script that I pieced together from the web. Worked great:

PHP Code:


			
#!/bin/bash

FILE=/usr/local/apache/logs/access_log;
 for ip in `cat $FILE |cut -d ' ' -f 1 |sort |uniq`;
 do { COUNT=`grep ^$ip $FILE |wc -l`;
 if [[ "$COUNT" -gt "10" ]]; then echo "$COUNT:   $ip";
 fi }; done

Here are the results on my test:

Code:

[root@forums1 bin]# ./ipcount.sh 
4416:   66.89.97.xxx
4415:   xx.72.16.18.xxx
56607:   16.187.xxx.xxx
55459:   xxx.xxx.xxx.195

Hope you have fun with this!!

PS: Please Move this if it should be in Shell/Scripting........

jaysunn

cfajohnson · #2 (**permalink**) 28-12-2009, 10:45 AM

Code:

cut -d ' ' -f 1 "$FILE" | sort | uniq -c

ghostdog74 · #3 (**permalink**) 28-12-2009, 11:09 AM

good effort, but here's a more efficient approach, using just awk

Code:

awk '{!a[$1]++}END{for(i in a) if ( a[i] >10 ) print a[i],i }' file

the reason for the inefficiency in your code is you are calling the log file many times as you find each unique ip address. If there are 1000 unique ip addresses, you are going to use grep on the log file 1000 times.... you get the idea...

cfajohnson · #4 (**permalink**) 28-12-2009, 12:30 PM

Quote:

Originally Posted by ghostdog74

good effort, but here's a more efficient approach, using just awk

Code:

awk '{!a[$1]++}END{for(i in a) if ( a[i] >10 ) print a[i],i }' file

The loop may be slower than calling separate commands that don't hold as much info in memory.

ghostdog74 · #5 (**permalink**) 28-12-2009, 01:03 PM

Quote:

Originally Posted by cfajohnson

The loop may be slower than calling separate commands that don't hold as much info in memory.

sorry, what do you mean? which loop are you referring to?

cfajohnson · #6 (**permalink**) 28-12-2009, 05:15 PM

Quote:

Originally Posted by ghostdog74

sorry, what do you mean? which loop are you referring to?

The only loop there is:
Code:
for(i in a) ...
Explicit loops are always slower than implicit ones.

ghostdog74 · #7 (**permalink**) 28-12-2009, 07:00 PM

Quote:

Originally Posted by cfajohnson

The only loop there is:
Code:
for(i in a) ...
Explicit loops are always slower than implicit ones.

ok, so how do you propose to solve this hypothesis of "explicit loops" may be slower than "implicit ones"

cfajohnson · #8 (**permalink**) 28-12-2009, 10:12 PM

Quote:

Originally Posted by ghostdog74

ok, so how do you propose to solve this hypothesis of "explicit loops" may be slower than "implicit ones"

If it isn't obvious (the code has to be interpreted every time through an explicit loop), then use the time command to test it.

jaysunn · #9 (**permalink**) 28-12-2009, 10:28 PM

Hmm,

Code:

[root@forums2 ~]# time cut -d ' ' -f 1 "$FILE" | sort | uniq -c
   6585 10.4.20.236
      1 173.10.18.115
      1 187.61.17.37
      4 217.24.240.68
     14 41.223.30.22
      3 61.160.216.63
 159051 67.72.16.xxx
   6613 67.72.16.xxx
 159047 67.72.16.xxx
     10 75.148.211.109
      2 78.138.151.126

real	0m6.954s
user	0m6.952s
sys	0m0.055s
[root@forums2 ~]# time awk '{!a[$1]++}END{for(i in a) if ( a[i] >10 ) print a[i],i }' $FILE
159070 67.72.16.xxx
14 41.223.30.22
159074 67.72.16.xxx
6586 10.4.20.xxx
6614 67.72.16.xxx

real	0m0.214s
user	0m0.201s
sys	0m0.014s
[root@forums2 ~]#

Second one was pretty fast.

Jaysunn

nixcraft · #10 (**permalink**) 28-12-2009, 10:51 PM

Quote:

Originally Posted by jaysunn

Second one was pretty fast.
Jaysunn

I've not tested this but it is *possible* that results are cached by kernel. Can you run it on two different hosts with same data file and post it back?

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Apache error 403 Permission access on RHEL5	samengr	Web servers	2	06-06-2009 02:18 AM
Shell script to count number of lines in file specified by the second command-line	seaman77	Shell scripting	1	16-03-2009 07:46 PM
grep command count number of CPU	sidebrake	Shell scripting	3	09-09-2008 11:26 PM
Set and access apache from DSL / ADSL connection	paul555	Web servers	4	17-07-2007 04:38 PM
Debian recovery mode read only access make it write access	Donavit	Linux software	1	30-12-2005 12:49 AM

nixCraft

Script to count unique ips in apache access log

Script to count unique ips in apache access log

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads

The Following 2 Users Say Thank You to ghostdog74 For This Useful Post:
jaysunn (28-12-2009), raj (30-12-2009)