Results 1 to 9 of 9

Thread: Apache find out TOP10 visited domain IP's accesing web server via log files

  1. #1
    Junior Member
    Join Date
    Apr 2012
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Rep Power
    0

    Default Apache find out TOP10 visited domain IP's accesing web server via log files

    I need a script to go through the web log access and find the TOP10 visited domain IP's accesing my web server.

    Code:
    example1.com - - [10/Apr/2012:16:57:57 +0300] "GET /public/charts/bg.jpg HTTP/1.1" 304 0
    example1.com - - [10/Apr/2012:16:58:36 +0300] "GET /public/index.php?option=com_public&member_cat=900&member_sub_cat=4 HTTP/1.1" 200 124429
    
    
    example1.com - - [10/Apr/2012:16:59:01 +0300] "GET /public/charts/security_chart.php?security_id=141074&period=10days&price=L_AVG&currTime=17_1_24_968 HTTP/1.1" 200 4545
    ..........
    example1.com - - [10/Apr/2012:17:49:32 +0300] "GET /public/index.php?option=com_public&member_cat=900&member_sub_cat=3 HTTP/1.1" 200 164215
    example1.com - - [10/Apr/2012:17:49:40 +0300] "GET /public/index.php?option=com_public&Itemid=28&security_report=131274 HTTP/1.1" 200 80373
    example1.com - - [10/Apr/2012:17:49:41 +0300] "GET /public/charts/security_chart.php?security_id=131274&period=10days&price=L_AVG&currTime=17_52_4_234 HTTP/1.1" 200 4477
    example1.com - - [10/Apr/2012:17:50:21 +0300] "GET /public/index.php?option=com_public&member_cat=900&member_sub_cat=3 HTTP/1.1" 200 164215
    ....
    example2.com - - [10/Apr/2012:16:44:50 +0300] "GET /public/templates/company/css/template_css.css HTTP/1.1" 304 0
    example2.com - - [10/Apr/2012:16:45:49 +0300] "GET /public/mambots/editors/fckeditor/fckeditor.js HTTP/1.1" 304 0
    example2.com - - [10/Apr/2012:16:45:49 +0300] "GET /public/index.php?option=com_public&Itemid=28&security_report=141110 HTTP/1.1" 200 75680
    example2.com - - [10/Apr/2012:16:45:49 +0300] "GET /public/templates/company/css/template_css.css HTTP/1.1" 304 0
    example2.com - - [10/Apr/2012:16:45:49 +0300] "GET /public/charts/JSClass/FusionCharts.js HTTP/1.1" 304 0
    example2.com - - [10/Apr/2012:16:45:49 +0300] "GET /public/js/functions.js HTTP/1.1" 304 0
    example2.com - - [10/Apr/2012:16:45:50 +0300] "GET /public/charts/swf/MultiAxisLine.swf HTTP/1.1" 304 0
    example2.com - - [10/Apr/2012:16:45:50 +0300] "GET /public/charts/security_chart.php?security_id=141110&period=10days&price=L_AVG&currTime=16_43_48_421 HTTP/1.1" 200 4441
    example2.com - - [10/Apr/2012:16:45:54 +0300] "GET /public/index.php?option=com_public&Itemid=28&security_report=141110 HTTP/1.1" 200 75680
    example2.com - - [10/Apr/2012:16:45:55 +0300] "GET /public/templates/company/css/template_css.css HTTP/1.1" 304 0
    example2.com - - [10/Apr/2012:16:45:56 +0300] "GET /public/templates/company/css/template_css.css HTTP/1.1" 304 0
    example2.com - - [10/Apr/2012:16:45:57 +0300] "GET /public/templates/company/css/template_css.css HTTP/1.1" 304 0
    example2.com - - [10/Apr/2012:16:46:35 +0300] "GET /public/mambots/editors/fckeditor/fckeditor.js HTTP/1.1" 304 0
    ..........
    example3.com - - [10/Apr/2012:08:41:34 +0300] "GET /public/js/functions.js HTTP/1.1" 304 0
    example3.com - - [10/Apr/2012:08:41:34 +0300] "GET /public/js/popup.js HTTP/1.1" 304 0
    example3.com - - [10/Apr/2012:08:41:34 +0300] "GET /public/mambots/editors/fckeditor/fckeditor.js HTTP/1.1" 304 0
    example3.com - - [10/Apr/2012:08:41:34 +0300] "GET /public/templates/company/css/template_css.css HTTP/1.1" 304 0
    ...
    example4.com - - [10/Apr/2012:16:58:49 +0300] "GET /public/data.php?issuer_no=111004&ratio=11&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4215
    example4.com - - [10/Apr/2012:16:58:49 +0300] "GET /english/index.php?option=com_public&Itemid=28&member_info=113023&corporate_actions HTTP/1.0" 200 73054
    example4.com - - [10/Apr/2012:16:58:49 +0300] "GET /english/index.php?option=com_public&Itemid=28&security_report=111007 HTTP/1.0" 200 87482
    example4.com - - [10/Apr/2012:17:13:07 +0300] "GET /public/data.php?issuer_no=121024&ratio=32&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4120
    example4.com - - [10/Apr/2012:17:13:08 +0300] "GET /public/data.php?issuer_no=121024&ratio=33&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4168
    example4.com - - [10/Apr/2012:17:13:08 +0300] "GET /public/data.php?issuer_no=121024&ratio=34&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4128
    example4.com - - [10/Apr/2012:17:13:09 +0300] "GET /public/data.php?issuer_no=121024&ratio=35&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4104
    example4.com - - [10/Apr/2012:17:13:10 +0300] "GET /public/data.php?issuer_no=121024&ratio=36&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4079
    example4.com - - [10/Apr/2012:17:40:32 +0300] "GET /public/data.php?issuer_no=131035&ratio=31&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4182
    example4.com - - [10/Apr/2012:17:40:33 +0300] "GET /public/data.php?issuer_no=131035&ratio=32&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4161
    example4.com - - [10/Apr/2012:17:40:34 +0300] "GET /public/data.php?issuer_no=131035&ratio=33&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4169
    example4.com - - [10/Apr/2012:17:40:35 +0300] "GET /public/data.php?issuer_no=131035&ratio=34&LANG=En&currTime=16_4_58_677 HTTP/1.0" 200 4113
    example5.com - - [10/Apr/2012:16:00:18 +0300] "GET /public/ticker.php HTTP/1.1" 200 316
    example5.com - - [10/Apr/2012:16:00:18 +0300] "GET /public/components/com_events/events_css.css HTTP/1.1" 200 5885
    example5.com - - [10/Apr/2012:16:00:18 +0300] "GET /public/images/events_bullet.jpg HTTP/1.1" 200 304
    example5.com - - [10/Apr/2012:16:00:19 +0300] "GET /public/images/go.gif HTTP/1.1" 200 1198
    example5.com - - [10/Apr/2012:16:01:12 +0300] "GET /front_tabs/nationalities_data.php?popup=0&LANG=AR&currTime=16_3_2_40&NatTransTypeValue=1&DataValue=1 HTTP/1.1" 200 3948
    example5.com - - [10/Apr/2012:16:01:12 +0300] "GET /front_tabs/nationalities_data.php?popup=0&LANG=AR&currTime=16_3_2_105&NatTransTypeValue=2&DataValue=1 HTTP/1.1" 200 3961
    example5.com - - [10/Apr/2012:16:01:12 +0300] "GET /front_tabs/nationalities_chart.php?LANG=AR&popup=0&currTime=16_3_1_925&NatTransTypeValue=1&DataValue=1 HTTP/1.1" 200 1790
    example5.com - - [10/Apr/2012:16:01:12 +0300] "GET /front_tabs/nationalities_chart.php?LANG=AR&popup=0&currTime=16_3_1_981&NatTransTypeValue=2&DataValue=1 HTTP/1.1" 200 1805
    example5.com - - [10/Apr/2012:16:01:12 +0300] "GET /front_tabs/nationalities_chart.php?LANG=AR&popup=0&currTime=16_3_2_39&NatTransTypeValue=1&DataValue=1 HTTP/1.1" 200 1790
    example5.com - - [10/Apr/2012:16:01:12 +0300] "GET /front_tabs/nationalities_chart.php?LANG=AR&popup=0&currTime=16_3_2_103&NatTransTypeValue=2&DataValue=1 HTTP/1.1" 200 1805
    example5.com - - [10/Apr/2012:16:01:14 +0300] "GET /front_tabs/nationalities_chart.php?LANG=AR&popup=0&currTime=16_3_1_504&NatTransTypeValue=1&DataValue=1 HTTP/1.1" 200 1790
    example5.com - - [10/Apr/2012:16:04:44 +0300] "GET /public/images/back.gif HTTP/1.1" 200 1144
    example5.com - - [10/Apr/2012:19:28:11 +0300] "GET /english/mambots/editors/fckeditor/fckeditor.js HTTP/1.1" 200 5281
    example5.com - - [10/Apr/2012:19:28:12 +0300] "GET /public/templates/company/images/inner_head.jpg HTTP/1.1" 200 10060
    example5.com - - [10/Apr/2012:19:28:12 +0300] "GET /english/images/topmenu.gif HTTP/1.1" 200 851
    example5.com - - [10/Apr/2012:19:28:12 +0300] "GET /english/templates/company/images/icon.gif HTTP/1.1" 200 1370
    example5.com - - [10/Apr/2012:19:28:11 +0300] "GET /english/templates/company/css/template_css.css HTTP/1.1" 200 18567
    example5.com - - [10/Apr/2012:19:28:11 +0300] "GET /english/templates/company/images/logo.jpg HTTP/1.1" 200 18993
    example5.com - - [10/Apr/2012:19:28:11 +0300] "GET /english/index.php?option=com_public&Itemid=28&member_cat=900 HTTP/1.1" 200 94791
    example5.com - - [10/Apr/2012:19:28:12 +0300] "GET /public/images/required.gif HTTP/1.1" 200 161
    example5.com - - [10/Apr/2012:19:28:12 +0300] "GET /english/images/go.gif HTTP/1.1" 200 1198
    example5.com - - [10/Apr/2012:19:28:12 +0300] "GET /english/images/back.gif HTTP/1.1" 200 1144
    example5.com - - [10/Apr/2012:19:28:13 +0300] "GET /public/images/close.gif HTTP/1.1" 200 1149
    example5.com - - [10/Apr/2012:19:28:13 +0300] "GET /english/templates/company/images/menu_bg.jpg HTTP/1.1" 200 347
    example5.com - - [10/Apr/2012:19:28:13 +0300] "GET /english/templates/company/images/items_bg.jpg HTTP/1.1" 200 783
    example5.com - - [10/Apr/2012:19:28:13 +0300] "GET /english/templates/company/images/mainmenu_bg.jpg HTTP/1.1" 200 621
    example5.com - - [10/Apr/2012:19:35:54 +0300] "GET /public/images/actions_arrow.png HTTP/1.1" 200 1189
    example5.com - - [10/Apr/2012:19:36:11 +0300] "GET /public/index.php?option=com_public&Itemid=28&security_report=131218 HTTP/1.1" 200 76686
    example5.com - - [10/Apr/2012:19:36:12 +0300] "GET /public/charts/security_chart.php?security_id=131218&period=10days&price=L_AVG&currTime=19_38_2_782 HTTP/1.1" 200 4527
    example5.com - - [10/Apr/2012:19:36:45 +0300] "GET /public/index.php?option=com_public&member_cat=900&member_sub_cat=3 HTTP/1.1" 200 164215
    example5.com - - [10/Apr/2012:19:36:51 +0300] "GET /public/index.php?option=com_public&Itemid=28&security_report=131217 HTTP/1.1" 200 76285
    Last edited by madunix; 2nd May 2012 at 05:35 PM.

  2. #2
    Senior Member
    Join Date
    Aug 2011
    Posts
    454
    Thanks
    0
    Thanked 62 Times in 58 Posts
    Rep Power
    10

    Default

    hi,

    what have you tried so far?
    a shell script could do that.
    a awk script could do that faster.

    post your algorithm (how you think things should be done).
    post your code.
    point where it doesn't work as you want

  3. #3
    Senior Member Rahul.Patil's Avatar
    Join Date
    Feb 2012
    Location
    Mumbai india
    Posts
    486
    Thanks
    10
    Thanked 50 Times in 47 Posts
    Rep Power
    8

    Default

    Hi,
    have you tried following ? :

    PHP Code:
    awk '{print $1}' /var/log/httpd/access_log uniq -sort -k1 -nr head -10 
    Rahul Patil <http://www.linuxian.com>

  4. #4
    Senior Member
    Join Date
    Aug 2011
    Posts
    454
    Thanks
    0
    Thanked 62 Times in 58 Posts
    Rep Power
    10

    Default

    hi Rahul,

    in this case `uniq' needs a sorted input.
    example file seems to be already sorted on first column, but a real web server log is not.

    why not just use `cut', instead of `awk'?
    Last edited by Watael; 2nd May 2012 at 11:48 AM.

  5. #5
    Senior Member Rahul.Patil's Avatar
    Join Date
    Feb 2012
    Location
    Mumbai india
    Posts
    486
    Thanks
    10
    Thanked 50 Times in 47 Posts
    Rep Power
    8

    Default

    Hi Wateal,

    time difference with both

    [root@centos ~]# time cut -d" " -f1 /tmp/acc | uniq -c | sort -k1 -nr | head -10
    31 example5.com
    13 example2.com
    12 example4.com
    4 example3.com
    4 example1.com
    2 example1.com
    2
    1 example1.com
    1 ..........
    1 ..........

    real 0m0.005s
    user 0m0.001s
    sys 0m0.004s
    [root@centos ~]# time awk '{print $1}' /tmp/acc | uniq -c | sort -k1 -nr | head -10
    31 example5.com
    13 example2.com
    12 example4.com
    4 example3.com
    4 example1.com
    2 example1.com
    2
    1 example1.com
    1 ..........
    1 ..........

    real 0m0.004s
    user 0m0.001s
    sys 0m0.003s
    Rahul Patil <http://www.linuxian.com>

  6. #6
    Senior Member
    Join Date
    Aug 2011
    Posts
    454
    Thanks
    0
    Thanked 62 Times in 58 Posts
    Rep Power
    10

    Default

    it doesn't seem to make a big difference.
    in my opinion, awk is a complete scripting tool which does not deserve to be used, even as a "super" cutter, when cut is able do the job.

  7. #7
    Senior Member Rahul.Patil's Avatar
    Join Date
    Feb 2012
    Location
    Mumbai india
    Posts
    486
    Thanks
    10
    Thanked 50 Times in 47 Posts
    Rep Power
    8

    Default

    Hi Watael,

    completely agree with you
    "uniq" needs sorted input

    PHP Code:
    cut -d" " -f1 /var/log/httpd/access_log  sort -uniq -sort -k1 -nr head -10 
    Last edited by Rahul.Patil; 2nd May 2012 at 03:07 PM.
    Rahul Patil <http://www.linuxian.com>

  8. #8
    Junior Member
    Join Date
    Apr 2012
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Rep Power
    0

    Default

    Thanks appreciated

  9. #9
    Never say die nixcraft's Avatar
    Join Date
    Jan 2005
    Location
    BIOS
    Posts
    4,515
    Thanks
    17
    Thanked 814 Times in 511 Posts
    Rep Power
    10

    Default

    Your scripting kung fu is great, but a proper tool such as webalizer is recommend:

    The Webalizer is a fast, free, web-server log files analysis program. It produces highly detailed, easily configurable usage reports in HTML format, for viewing with a standard web browser.
    Statistics commonly reported by Webalizer include: hits; visits; referers; the visitors' countries; and the amount of data downloaded. These statistics can be viewed graphically and presented by different time frames, such as per day, hour, or month.
    See sample - Usage Statistics for www.mrunix.net - May 1999

    /Closed
    All [Solved] threads are closed by mods / admin to avoid spam issues. See Howto mark a thread as [Solved]


Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Apache redirect from url to domain name
    By VishalK in forum Web servers
    Replies: 1
    Last Post: 30th April 2012, 02:12 AM
  2. [Solved] How do I fix Domain Name Server?
    By georgemoore in forum Domain Name Server
    Replies: 0
    Last Post: 11th January 2011, 11:30 AM
  3. Find apache version linux
    By chinalinix in forum Web servers
    Replies: 6
    Last Post: 8th November 2010, 11:27 PM
  4. How to find Total nuber files/size in a server?
    By mahi in forum Solaris/OpenSolaris
    Replies: 1
    Last Post: 2nd October 2008, 11:48 AM
  5. problem with accesing oracle in linux
    By ramjimh in forum Linux software
    Replies: 3
    Last Post: 6th July 2006, 12:38 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •