We've reached the point where we're having to actively degrade the UI and remove useful features of the small scientific database I co-maintain (xronos.ch) to try and reduce the extreme load imposed by (unwanted and politely-asked-to-go-away) AI crawler bots. Conversations about new features have to ask whether it's going to make the crawler load problem worse.
This entry was edited (2 days ago)
