• If you're a past member of the board, but can't recall your password any more, you don't need to set up a new account (unless you wish to). As long as you recall your old login name, you can log in with that user name then select 'forgot password' and the board will email you at your registration email, to let you reset your password.

We are being invaded by spiders

Karlin

Administrator
Staff member
Yahoo has the most annoying indexing robots/spiders in the planet. These crawl the site and index it for their search engine. Unfortunately once they find a site they can send up to hundreds at a time. Hence the /most online' count keeps spiking ever upwards -- it says *645* now! :rolleyes: -- and it can look like we are flooded with guests. This has been a plague for lots of boards for ages I have learned; I have just never had this one indexed so intently. It is good to be indexed of course and makes it easier for people to find say an article of interest, but I don;t much care to have them all online in their hundreds every day.

I have plenty of bandwidth and capacity so that side of things doesn't matter but having so many at once online is pretty obnoxious so I've written a little file that instructs Yahho robots to only be allowed every 30 seconds, one by one, on the site. However this can take up to a week to take effect so you may continue to see large numbers online.

This is just a claification for anyone who has wondered what the heck is going on with floods of guests! :lol:
 
Just moderately popular! :lol: More typically we have a few on up to about 35-40 people max online here at any time of which a couple of those would be spiders. But in the last two days we have been 'found' by Yahoo; before the site always had a few Yahoo spiders crawling the site daily but suddenly it is up to over 600 at a time. :confused: It's just annoying but soon they will be held at bay.

The useful thing for me was learning how to code up a little file to block about 150 of the truly unwanted spiders that can gather things like email addresses, phone numbers, try to do spam registrations, etc. I've blocked all of those as well though these days I get almost no spam registrations anyway -- we used to get 20 attempts a day; now I have had two in two months, since putting on the new software. :rah:
 
these days I get almost no spam registrations anyway -- we used to get 20 attempts a day; now I have had two in two months, since putting on the new software. :rah:

That's great Karlin.

The new site is a credit to you (y)
 
There seem to be load of spider bots on the site tonight! Must be the rain driving them out of their little holes!
 
Yeah, there are far fewer than two days ago, but still a lot. It should take a week or so for the new code to take affect.
 
I wondered what all those spiders and bots were! Now I know! :lol: Like you said it's nice to be popular but there is a limit!! :lol:
 
Back
Top