Bot and crawler overloads

[admin]

As of August there's been a massive rise in the amount of 'guest' activity, often with over 15000 'guests' using the Forum at any one time. This amount of activity is now breaching the limits we've got and the Forum is becoming inaccessible (even to admin).

The 'guests' are mainly 'web crawlers' that trawl though websites - sometimes this will be for useful purposes but increasingly it seems to be to feed AI algorithms, which is behind the recent massive increase. It's an issue for a lot of other Forums also and any solution is likely to make things more difficult to use.

Initially I've banned swathes of IP addresses (probably hitting some quite innocent ones and missing quite a few of the bandits). This will have to be 'adjusted' over the next few months, and if we get more overloads the Forum will have to be taken down for a period until things get quieter again.

Alternative solutions will be examined!

SG

[Chris Sealy]

It's very comforting, that someone knows what they're doing with this inter web thingymajig. Chris.