Skip to main content

mastodon - Link to source

Today, #facebook bots (Meta-ExternalAgent, you know, the one to train AI) took 21 days of wallclock time and 19.9 GB of bandwidth to index the development version of one of our websites for which crawling makes no sense and is explicitly forbidden by robots.txt.

Maybe it is a signal that we must spend some of _our_ time to make them lose more of _them's_. Any creative idea?

#facebook
in reply to Romain Tartière

Aral Balkan
mastodon - Link to source
Aral Balkan
• •
Don’t know the names off the top of my head but there are ways to make them go into infinitely deep content holes.
in reply to Aral Balkan

Davide_Sandini
mastodon - Link to source
Davide_Sandini
• •
@aral
I do not remeber who had a similar project, but for sure it was cited here on mastodon.
@smortex
@Aral Balkan @Romain Tartière
in reply to Davide_Sandini

flyinggecko
mastodon - Link to source
flyinggecko
• •
@Davide_Sandini @aral Maybe this? iocaine.madhouse-project.org/
It uses a relative small amount of resources for a lot of work for crawlers.
Edit: And it's fun following the development toots of @algernon

iocaine - the deadliest poison known to AI

iocaine.madhouse-project.org
@Aral Balkan @Davide_Sandini @a very strange trackball
This entry was edited (4 days ago)
⇧