Thread: Apache find out TOP10 visited domain IP's accesing web server via log files

I need a script to go through the web log access and find the TOP10 visited domain IP's accesing my web server.

Code:

example1.com - - [10/Apr/2012:16:57:57 +0300] "GET /public/charts/bg.jpg HTTP/1.1" 304 0
example1.com - - [10/Apr/2012:16:58:36 +0300] "GET /public/index.php?option=com_public&member_cat=900&member_sub_cat=4 HTTP/1.1" 200 124429


example1.com - - [10/Apr/2012:16:59:01 +0300] "GET /public/charts/security_chart.php?security_id=141074&period=10days&price=L_AVG&currTime=17_1_24_968 HTTP/1.1" 200 4545
..........
example1.com - - [10/Apr/2012:17:49:32 +0300] "GET /public/index.php?option=com_public&member_cat=900&member_sub_cat=3 HTTP/1.1" 200 164215
example1.com - - [10/Apr/2012:17:49:40 +0300] "GET /public/index.php?option=com_public&Itemid=28&security_report=131274 HTTP/1.1" 200 80373
example1.com - - [10/Apr/2012:17:49:41 +0300] "GET /public/charts/security_chart.php?security_id=131274&period=10days&price=L_AVG&currTime=17_52_4_234 HTTP/1.1" 200 4477
example1.com - - [10/Apr/2012:17:50:21 +0300] "GET /public/index.php?option=com_public&member_cat=900&member_sub_cat=3 HTTP/1.1" 200 164215
....
example2.com - - [10/Apr/2012:16:44:50 +0300] "GET /public/templates/company/css/template_css.css HTTP/1.1" 304 0
example2.com - - [10/Apr/2012:16:45:49 +0300] "GET /public/mambots/editors/fckeditor/fckeditor.js HTTP/1.1" 304 0
example2.com - - [10/Apr/2012:16:45:49 +0300] "GET /public/index.php?option=com_public&Itemid=28&security_report=141110 HTTP/1.1" 200 75680
example2.com - - [10/Apr/2012:16:45:49 +0300] "GET /public/templates/company/css/template_css.css HTTP/1.1" 304 0
example2.com - - [10/Apr/2012:16:45:49 +0300] "GET /public/charts/JSClass/FusionCharts.js HTTP/1.1" 304 0
example2.com - - [10/Apr/2012:16:45:49 +0300] "GET /public/js/functions.js HTTP/1.1" 304 0
example2.com - - [10/Apr/2012:16:45:50 +0300] "GET /public/charts/swf/MultiAxisLine.swf HTTP/1.1" 304 0
example2.com - - [10/Apr/2012:16:45:50 +0300] "GET /public/charts/security_chart.php?security_id=141110&period=10days&price=L_AVG&currTime=16_43_48_421 HTTP/1.1" 200 4441
example2.com - - [10/Apr/2012:16:45:54 +0300] "GET /public/index.php?option=com_public&Itemid=28&security_report=141110 HTTP/1.1" 200 75680
example2.com - - [10/Apr/2012:16:45:55 +0300] "GET /public/templates/company/css/template_css.css HTTP/1.1" 304 0
example2.com - - [10/Apr/2012:16:45:56 +0300] "GET /public/templates/company/css/template_css.css HTTP/1.1" 304 0
example2.com - - [10/Apr/2012:16:45:57 +0300] "GET /public/templates/company/css/template_css.css HTTP/1.1" 304 0
example2.com - - [10/Apr/2012:16:46:35 +0300] "GET /public/mambots/editors/fckeditor/fckeditor.js HTTP/1.1" 304 0
..........
example3.com - - [10/Apr/2012:08:41:34 +0300] "GET /public/js/functions.js HTTP/1.1" 304 0
example3.com - - [10/Apr/2012:08:41:34 +0300] "GET /public/js/popup.js HTTP/1.1" 304 0
example3.com - - [10/Apr/2012:08:41:34 +0300] "GET /public/mambots/editors/fckeditor/fckeditor.js HTTP/1.1" 304 0
example3.com - - [10/Apr/2012:08:41:34 +0300] "GET /public/templates/company/css/template_css.css HTTP/1.1" 304 0
...
example4.com - - [10/Apr/2012:16:58:49 +0300] "GET /public/data.php?issuer_no=111004&ratio=11&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4215
example4.com - - [10/Apr/2012:16:58:49 +0300] "GET /english/index.php?option=com_public&Itemid=28&member_info=113023&corporate_actions HTTP/1.0" 200 73054
example4.com - - [10/Apr/2012:16:58:49 +0300] "GET /english/index.php?option=com_public&Itemid=28&security_report=111007 HTTP/1.0" 200 87482
example4.com - - [10/Apr/2012:17:13:07 +0300] "GET /public/data.php?issuer_no=121024&ratio=32&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4120
example4.com - - [10/Apr/2012:17:13:08 +0300] "GET /public/data.php?issuer_no=121024&ratio=33&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4168
example4.com - - [10/Apr/2012:17:13:08 +0300] "GET /public/data.php?issuer_no=121024&ratio=34&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4128
example4.com - - [10/Apr/2012:17:13:09 +0300] "GET /public/data.php?issuer_no=121024&ratio=35&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4104
example4.com - - [10/Apr/2012:17:13:10 +0300] "GET /public/data.php?issuer_no=121024&ratio=36&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4079
example4.com - - [10/Apr/2012:17:40:32 +0300] "GET /public/data.php?issuer_no=131035&ratio=31&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4182
example4.com - - [10/Apr/2012:17:40:33 +0300] "GET /public/data.php?issuer_no=131035&ratio=32&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4161
example4.com - - [10/Apr/2012:17:40:34 +0300] "GET /public/data.php?issuer_no=131035&ratio=33&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4169
example4.com - - [10/Apr/2012:17:40:35 +0300] "GET /public/data.php?issuer_no=131035&ratio=34&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4113
example5.com - - [10/Apr/2012:16:00:18 +0300] "GET /public/ticker.php HTTP/1.1" 200 316
example5.com - - [10/Apr/2012:16:00:18 +0300] "GET /public/components/com_events/events_css.css HTTP/1.1" 200 5885
example5.com - - [10/Apr/2012:16:00:18 +0300] "GET /public/images/events_bullet.jpg HTTP/1.1" 200 304
example5.com - - [10/Apr/2012:16:00:19 +0300] "GET /public/images/go.gif HTTP/1.1" 200 1198
example5.com - - [10/Apr/2012:16:01:12 +0300] "GET /front_tabs/nationalities_data.php?popup=0&LANG=AR&currTime=16_3_2_40&NatTransTypeValue=1&DataValue=1 HTTP/1.1" 200 3948
example5.com - - [10/Apr/2012:16:01:12 +0300] "GET /front_tabs/nationalities_data.php?popup=0&LANG=AR&currTime=16_3_2_105&NatTransTypeValue=2&DataValue=1 HTTP/1.1" 200 3961
example5.com - - [10/Apr/2012:16:01:12 +0300] "GET /front_tabs/nationalities_chart.php?LANG=AR&popup=0&currTime=16_3_1_925&NatTransTypeValue=1&DataValue=1 HTTP/1.1" 200 1790
example5.com - - [10/Apr/2012:16:01:12 +0300] "GET /front_tabs/nationalities_chart.php?LANG=AR&popup=0&currTime=16_3_1_981&NatTransTypeValue=2&DataValue=1 HTTP/1.1" 200 1805
example5.com - - [10/Apr/2012:16:01:12 +0300] "GET /front_tabs/nationalities_chart.php?LANG=AR&popup=0&currTime=16_3_2_39&NatTransTypeValue=1&DataValue=1 HTTP/1.1" 200 1790
example5.com - - [10/Apr/2012:16:01:12 +0300] "GET /front_tabs/nationalities_chart.php?LANG=AR&popup=0&currTime=16_3_2_103&NatTransTypeValue=2&DataValue=1 HTTP/1.1" 200 1805
example5.com - - [10/Apr/2012:16:01:14 +0300] "GET /front_tabs/nationalities_chart.php?LANG=AR&popup=0&currTime=16_3_1_504&NatTransTypeValue=1&DataValue=1 HTTP/1.1" 200 1790
example5.com - - [10/Apr/2012:16:04:44 +0300] "GET /public/images/back.gif HTTP/1.1" 200 1144
example5.com - - [10/Apr/2012:19:28:11 +0300] "GET /english/mambots/editors/fckeditor/fckeditor.js HTTP/1.1" 200 5281
example5.com - - [10/Apr/2012:19:28:12 +0300] "GET /public/templates/company/images/inner_head.jpg HTTP/1.1" 200 10060
example5.com - - [10/Apr/2012:19:28:12 +0300] "GET /english/images/topmenu.gif HTTP/1.1" 200 851
example5.com - - [10/Apr/2012:19:28:12 +0300] "GET /english/templates/company/images/icon.gif HTTP/1.1" 200 1370
example5.com - - [10/Apr/2012:19:28:11 +0300] "GET /english/templates/company/css/template_css.css HTTP/1.1" 200 18567
example5.com - - [10/Apr/2012:19:28:11 +0300] "GET /english/templates/company/images/logo.jpg HTTP/1.1" 200 18993
example5.com - - [10/Apr/2012:19:28:11 +0300] "GET /english/index.php?option=com_public&Itemid=28&member_cat=900 HTTP/1.1" 200 94791
example5.com - - [10/Apr/2012:19:28:12 +0300] "GET /public/images/required.gif HTTP/1.1" 200 161
example5.com - - [10/Apr/2012:19:28:12 +0300] "GET /english/images/go.gif HTTP/1.1" 200 1198
example5.com - - [10/Apr/2012:19:28:12 +0300] "GET /english/images/back.gif HTTP/1.1" 200 1144
example5.com - - [10/Apr/2012:19:28:13 +0300] "GET /public/images/close.gif HTTP/1.1" 200 1149
example5.com - - [10/Apr/2012:19:28:13 +0300] "GET /english/templates/company/images/menu_bg.jpg HTTP/1.1" 200 347
example5.com - - [10/Apr/2012:19:28:13 +0300] "GET /english/templates/company/images/items_bg.jpg HTTP/1.1" 200 783
example5.com - - [10/Apr/2012:19:28:13 +0300] "GET /english/templates/company/images/mainmenu_bg.jpg HTTP/1.1" 200 621
example5.com - - [10/Apr/2012:19:35:54 +0300] "GET /public/images/actions_arrow.png HTTP/1.1" 200 1189
example5.com - - [10/Apr/2012:19:36:11 +0300] "GET /public/index.php?option=com_public&Itemid=28&security_report=131218 HTTP/1.1" 200 76686
example5.com - - [10/Apr/2012:19:36:12 +0300] "GET /public/charts/security_chart.php?security_id=131218&period=10days&price=L_AVG&currTime=19_38_2_782 HTTP/1.1" 200 4527
example5.com - - [10/Apr/2012:19:36:45 +0300] "GET /public/index.php?option=com_public&member_cat=900&member_sub_cat=3 HTTP/1.1" 200 164215
example5.com - - [10/Apr/2012:19:36:51 +0300] "GET /public/index.php?option=com_public&Itemid=28&security_report=131217 HTTP/1.1" 200 76285

hi,

what have you tried so far?
a shell script could do that.
a awk script could do that faster.

post your algorithm (how you think things should be done).
post your code.
point where it doesn't work as you want

Hi,
have you tried following ? :

PHP Code:

awk '{print $1}' /var/log/httpd/access_log | uniq -c | sort -k1 -nr | head -10 


hi Rahul,

in this case `uniq' needs a sorted input.
example file seems to be already sorted on first column, but a real web server log is not.

why not just use `cut', instead of `awk'?

Hi Wateal,

time difference with both

[root@centos ~]# time cut -d" " -f1 /tmp/acc | uniq -c | sort -k1 -nr | head -10
31 example5.com
13 example2.com
12 example4.com
4 example3.com
4 example1.com
2 example1.com
2
1 example1.com
1 ..........
1 ..........

real 0m0.005s
user 0m0.001s
sys 0m0.004s
[root@centos ~]# time awk '{print $1}' /tmp/acc | uniq -c | sort -k1 -nr | head -10
31 example5.com
13 example2.com
12 example4.com
4 example3.com
4 example1.com
2 example1.com
2
1 example1.com
1 ..........
1 ..........

real 0m0.004s
user 0m0.001s
sys 0m0.003s

it doesn't seem to make a big difference.
in my opinion, awk is a complete scripting tool which does not deserve to be used, even as a "super" cutter, when cut is able do the job.

Hi Watael,

completely agree with you
"uniq" needs sorted input

PHP Code:

cut -d" " -f1 /var/log/httpd/access_log  | sort -n | uniq -c | sort -k1 -nr | head -10 


Thanks appreciated

Your scripting kung fu is great, but a proper tool such as webalizer is recommend:

The Webalizer is a fast, free, web-server log files analysis program. It produces highly detailed, easily configurable usage reports in HTML format, for viewing with a standard web browser.
Statistics commonly reported by Webalizer include: hits; visits; referers; the visitors' countries; and the amount of data downloaded. These statistics can be viewed graphically and presented by different time frames, such as per day, hour, or month.

See sample - Usage Statistics for www.mrunix.net - May 1999

/Closed