PDA

View Full Version : We are being invaded by spiders



Karlin
26th June 2007, 12:15 PM
Yahoo has the most annoying indexing robots/spiders in the planet. These crawl the site and index it for their search engine. Unfortunately once they find a site they can send up to hundreds at a time. Hence the /most online' count keeps spiking ever upwards -- it says *645* now! :rolleyes: -- and it can look like we are flooded with guests. This has been a plague for lots of boards for ages I have learned; I have just never had this one indexed so intently. It is good to be indexed of course and makes it easier for people to find say an article of interest, but I don;t much care to have them all online in their hundreds every day.

I have plenty of bandwidth and capacity so that side of things doesn't matter but having so many at once online is pretty obnoxious so I've written a little file that instructs Yahho robots to only be allowed every 30 seconds, one by one, on the site. However this can take up to a week to take effect so you may continue to see large numbers online.

This is just a claification for anyone who has wondered what the heck is going on with floods of guests! :lol:

casshon
26th June 2007, 01:32 PM
And there I was thinking we were all very popluar :p

Karlin
26th June 2007, 02:09 PM
Just moderately popular! :lol: More typically we have a few on up to about 35-40 people max online here at any time of which a couple of those would be spiders. But in the last two days we have been 'found' by Yahoo; before the site always had a few Yahoo spiders crawling the site daily but suddenly it is up to over 600 at a time. :confused: It's just annoying but soon they will be held at bay.

The useful thing for me was learning how to code up a little file to block about 150 of the truly unwanted spiders that can gather things like email addresses, phone numbers, try to do spam registrations, etc. I've blocked all of those as well though these days I get almost no spam registrations anyway -- we used to get 20 attempts a day; now I have had two in two months, since putting on the new software. :rah:

casshon
26th June 2007, 02:38 PM
these days I get almost no spam registrations anyway -- we used to get 20 attempts a day; now I have had two in two months, since putting on the new software. :rah:

That's great Karlin.

The new site is a credit to you :thmbsup:

merlinsmum
28th June 2007, 08:30 PM
There seem to be load of spider bots on the site tonight! Must be the rain driving them out of their little holes!

Karlin
28th June 2007, 08:49 PM
Yeah, there are far fewer than two days ago, but still a lot. It should take a week or so for the new code to take affect.

Cathryn
29th June 2007, 08:18 PM
I wondered what all those spiders and bots were! Now I know! :lol: Like you said it's nice to be popular but there is a limit!! :lol: