nixCraft Linux Forum

nixCraft

Linux / UNIX Tech Support Forum

lighttpd issue for website copiers

This is a discussion on lighttpd issue for website copiers within the Web servers forums, part of the Mastering Servers category; I would like to know if there is some issue to avoid website copiers like httrack to copy a full ...


Go Back   nixCraft Linux Forum > Mastering Servers > Web servers

Linux answers from nixCraft.


Web servers Discussion on Apache, Nginx and Lighttpd HTTP/web server and configuration issues.

Reply

 

LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 24-05-2008, 03:21 PM
Junior Member
User
 
Join Date: May 2008
OS: Debian
Posts: 16
Thanks: 0
Thanked 0 Times in 0 Posts
Rep Power: 0
mumuri is on a distinguished road
Default lighttpd issue for website copiers

I would like to know if there is some issue to avoid website copiers like httrack to copy a full site in local

i know that we can use useragent,

but is there some other tricks ?

-------------------------------------
can we tell lighttpd, to not serve more that x pages in a period for the same IP ?
-------------------------------------
can we automatically ban browser that leech invisible link (or any special link)

for instance, you have a

Code:
<a href="ban-link.php" onClick="return false;">.</a>
this link can't be view by browsers, if a bot get it... his ip is banned (if the ip doesn't belong to a trust list with google, msn and yahoo bots in it).

i did on an apache by appending a "deny from xxx" in a htaccess, but i don't think it's possible with lighttpd

perhaps it's more an iptables works ?

thanks in advance

Last edited by mumuri; 24-05-2008 at 03:31 PM.
Reply With Quote
  #2 (permalink)  
Old 24-05-2008, 06:10 PM
nixcraft's Avatar
Never say die
User
 
Join Date: Jan 2005
Location: BIOS
OS: RHEL
Scripting language: Bash and Python
Posts: 2,710
Thanks: 11
Thanked 245 Times in 184 Posts
Rep Power: 10
nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute
Default Lighttpd throttle IP

Yes there is a module called mod_evasive. You can set upper limit of number of connections per ip allowed using following directive:
Code:
evasive.max-conns-per-ip = 3
However, this 1.5 specific module, so you need to run 1.5 series lighttpd which is under beta for some time.

Docs:ModEvasive - lighttpd - secure, fast, compliant, and very flexible web-server - Trac
__________________
Vivek Gite
Linux Evangelist
Be proud RHEL user, and let the world know about your enterprise choices! Join RedHat user group.
Always use CODE tags for posting system output and commands!
Do you run a Linux? Let's face it, you need help
Reply With Quote
Reply

Tags
anti website copier , bot , lighttpd , lighttpd ip connection , mod_evasive , server , throttle ip


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads

Thread Thread Starter Forum Replies Last Post
Cron to check a website and restart it if down permalac Shell scripting 3 16-05-2008 06:41 PM
How to host website using adsl router zeebala CentOS / RHEL / Fedora 2 01-04-2008 09:23 PM
Issue related to shells vaibhav.kanchan Getting started tutorials 4 25-03-2008 12:52 AM
How to host website using adsl router zeebala CentOS / RHEL / Fedora 0 29-11-2007 07:15 PM
Log Off Issue tkinsella Linux software 1 30-08-2005 03:03 PM


All times are GMT +5.5. The time now is 01:22 PM.


Powered by vBulletin® Version 3.8.5 - Copyright ©2000 - 2010, Jelsoft Enterprises Ltd.
Search Engine Optimization by vBSEO 3.3.2
©2005-2010 nixCraft. All rights reserved

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38