Results 1 to 7 of 7

Thread: We are being invaded by spiders

  1. #1
    Join Date
    Mar 2005
    Location
    Dublin, Ireland
    Posts
    24,063
    Post Thanks / Like
    Blog Entries
    15

    Default We are being invaded by spiders

    Yahoo has the most annoying indexing robots/spiders in the planet. These crawl the site and index it for their search engine. Unfortunately once they find a site they can send up to hundreds at a time. Hence the /most online' count keeps spiking ever upwards -- it says *645* now! -- and it can look like we are flooded with guests. This has been a plague for lots of boards for ages I have learned; I have just never had this one indexed so intently. It is good to be indexed of course and makes it easier for people to find say an article of interest, but I don;t much care to have them all online in their hundreds every day.

    I have plenty of bandwidth and capacity so that side of things doesn't matter but having so many at once online is pretty obnoxious so I've written a little file that instructs Yahho robots to only be allowed every 30 seconds, one by one, on the site. However this can take up to a week to take effect so you may continue to see large numbers online.

    This is just a claification for anyone who has wondered what the heck is going on with floods of guests!
    Karlin
    Cavaliers: Jaspar Lily Tansy Libby Mindy
    In memory: Lucy Leo
    Cavalier SM Information site:www.smcavaliers.com

  2. #2
    Join Date
    Oct 2006
    Location
    Dublin, Ireland
    Posts
    920
    Post Thanks / Like

    Default

    And there I was thinking we were all very popluar
    Casshon

    Molly (Blenheim) & Bella (Tri)
    Honey, Cassie & Tiki (Cavalier Angels)

  3. #3
    Join Date
    Mar 2005
    Location
    Dublin, Ireland
    Posts
    24,063
    Post Thanks / Like
    Blog Entries
    15

    Default

    Just moderately popular! More typically we have a few on up to about 35-40 people max online here at any time of which a couple of those would be spiders. But in the last two days we have been 'found' by Yahoo; before the site always had a few Yahoo spiders crawling the site daily but suddenly it is up to over 600 at a time. It's just annoying but soon they will be held at bay.

    The useful thing for me was learning how to code up a little file to block about 150 of the truly unwanted spiders that can gather things like email addresses, phone numbers, try to do spam registrations, etc. I've blocked all of those as well though these days I get almost no spam registrations anyway -- we used to get 20 attempts a day; now I have had two in two months, since putting on the new software.
    Karlin
    Cavaliers: Jaspar Lily Tansy Libby Mindy
    In memory: Lucy Leo
    Cavalier SM Information site:www.smcavaliers.com

  4. #4
    Join Date
    Oct 2006
    Location
    Dublin, Ireland
    Posts
    920
    Post Thanks / Like

    Default

    Quote Originally Posted by Karlin View Post
    these days I get almost no spam registrations anyway -- we used to get 20 attempts a day; now I have had two in two months, since putting on the new software.
    That's great Karlin.

    The new site is a credit to you
    Casshon

    Molly (Blenheim) & Bella (Tri)
    Honey, Cassie & Tiki (Cavalier Angels)

  5. #5
    Join Date
    Jul 2006
    Location
    Darwen, Lancashire, England
    Posts
    2,828
    Post Thanks / Like
    Blog Entries
    4

    Default

    There seem to be load of spider bots on the site tonight! Must be the rain driving them out of their little holes!
    Kirsty
    Merlin and Oakleys Mum (Merlin -Male/B&T/5 years, Oakley - Male/Ruby/3.5years)

  6. #6
    Join Date
    Mar 2005
    Location
    Dublin, Ireland
    Posts
    24,063
    Post Thanks / Like
    Blog Entries
    15

    Default

    Yeah, there are far fewer than two days ago, but still a lot. It should take a week or so for the new code to take affect.
    Karlin
    Cavaliers: Jaspar Lily Tansy Libby Mindy
    In memory: Lucy Leo
    Cavalier SM Information site:www.smcavaliers.com

  7. #7
    Join Date
    Apr 2007
    Location
    The Midlands UK
    Posts
    3,395
    Post Thanks / Like

    Default

    I wondered what all those spiders and bots were! Now I know! Like you said it's nice to be popular but there is a limit!!
    Cathryn
    Cavaliers leave pawprints in your heart and hair on everything else!!
    RE-LIVE YOUR CHILDHOOD THROUGH YOUR KIDS, THE TOYS ARE SO MUCH BETTER THESE DAYS!!

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •