hitormiss.org : Remember how I said I don't like it when search indexing bots ignore my robots.txt file and spider through my site?

« July 2, 2000 »

Remember how I said I don't like it when search indexing bots ignore my robots.txt file and spider through my site?

The particular bot I'm mad at is coming from 213.219.19.148 and shows up in my referer logs as the user agent "Katriona."

I feel like I'm undergoing a denial of service attack from her. For the past 4 days, this bot has continued to spider through my site. Because my site is dynamically driven and has links repeated on every page, the bot keeps following links around and around and arround. Now it seems to be making up links - for example she's trying to do searches for files like:

"/archives/2000/01/go/http://www.sierra.com/search."

I don't know how to stop her. Can anyone please offer me some advice for stopping spiders if I'm running Apache 1.3.12 on Linux?

TrackBacks (url):

There are no trackbacks for this entry.

Reader Comments:

That spider is doing the same to my site.

- Elan
Posted by Elan on Thursday, July 06, 2000

Okay, so I tried to do a RewriteCond/Rule in my .htaccess file to block the IP number of the indexing bot. For some reason, everytime I tried to add it, it caused MySQL to stop running which knocks my site down.
Posted by Matt on Monday, July 03, 2000

Posting of comments has been suspended on all posts over 2 weeks old.

If you feel the overwhelming need to add a comment to this post, please contact Matt with your comment and he may or may not add it.