Either Mastodon's link preview bot should obey robots.txt or Mastodon needs O(1) link previews: https://www.jefftk.com/p/mastodons-dubious-crawler-exemption
@jefftk Would this still be a issue if Mastodon lazily built its link preview caches only after the first user requests the link?
Is the issue the total number of requests, or the fact that they are happening automatically, whether or not a specific user requests them?
@zebrask the latter: unless the requests are happening because someone directly asked for them, my interpretation is robots.txt should apply
@jefftk I agree that that would satisfy the contract, but it seems a bit rules lawyer-y to suggest that they can solve the problem by changing to a nearly identical action taken in response to user action - unless you think most instances never *display* the link previews that calculate.
From a practical point of view, the problems identified in this thread don't seem like they would be solved by fetching in response to user action: https://better.boston/@crschmidt/109412294646370820
@jefftk I'm no expert but it seems like the practical problems would be minimized if they used opengraph tags to generate the previews, and were more efficient about how they do those fetches.
They should still respect robots.txt if performing automated actions, but with efficient preview generation I imagine people would mostly not care.
