nixCraft Linux Forum

nixCraft

Linux Tech Support Forum

lighttpd issue for website copiers

This is a discussion on lighttpd issue for website copiers within the Web servers forums, part of the Mastering Servers category; I would like to know if there is some issue to avoid website copiers like httrack to copy a full ...


Go Back   nixCraft Linux Forum > Mastering Servers > Web servers

Register FAQ Members List Calendar Mark Forums Read
  #1 (permalink)  
Old 05-24-2008, 03:21 PM
Junior Member
User
 
Join Date: May 2008
My distro: Debian
Posts: 9
Rep Power: 0
mumuri is on a distinguished road
Default lighttpd issue for website copiers

I would like to know if there is some issue to avoid website copiers like httrack to copy a full site in local

i know that we can use useragent,

but is there some other tricks ?

-------------------------------------
can we tell lighttpd, to not serve more that x pages in a period for the same IP ?
-------------------------------------
can we automatically ban browser that leech invisible link (or any special link)

for instance, you have a

Code:
<a href="ban-link.php" onClick="return false;">.</a>
this link can't be view by browsers, if a bot get it... his ip is banned (if the ip doesn't belong to a trust list with google, msn and yahoo bots in it).

i did on an apache by appending a "deny from xxx" in a htaccess, but i don't think it's possible with lighttpd

perhaps it's more an iptables works ?

thanks in advance

Last edited by mumuri; 05-24-2008 at 03:31 PM.
Reply With Quote
Sponsored Links
  #2 (permalink)  
Old 05-24-2008, 06:10 PM
nixcraft's Avatar
Never say die
User
 
Join Date: Jan 2005
Location: BIOS
My distro: Any distro with shell
Posts: 917
Rep Power: 10
nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute nixcraft has a reputation beyond repute
Default Lighttpd throttle IP

Yes there is a module called mod_evasive. You can set upper limit of number of connections per ip allowed using following directive:
Code:
evasive.max-conns-per-ip = 3
However, this 1.5 specific module, so you need to run 1.5 series lighttpd which is under beta for some time.

Docs:ModEvasive - lighttpd - secure, fast, compliant, and very flexible web-server - Trac
__________________
Vivek | My personal blog
Linux Evangelist
Play hard stay cool
Reply With Quote
Reply

Bookmarks

Tags
anti website copier , bot , lighttpd , lighttpd ip connection , mod_evasive , server , throttle ip


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads

Thread Thread Starter Forum Replies Last Post
Cron to check a website and restart it if down permalac Shell scripting 3 05-16-2008 06:41 PM
How to host website using adsl router zeebala CentOS / RHEL / Fedora 2 04-01-2008 09:23 PM
Issue related to shells vaibhav.kanchan Getting started tutorials 4 03-25-2008 12:52 AM
How to host website using adsl router zeebala CentOS / RHEL / Fedora 0 11-29-2007 07:15 PM
Log Off Issue tkinsella Linux software 1 08-30-2005 03:03 PM


All times are GMT +5.5. The time now is 01:33 AM.


Powered by vBulletin® Version 3.7.2 - Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
Search Engine Optimization by vBSEO 3.2.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36