I fucked with the title a bit. What i linked to was actually a mastodon post linking to an actual thing. but in my defense, i found it because cory doctorow boosted it, so, in a way, i am providing the original source here.

please argue. please do not remove.

  • Batman@lemmy.world
    link
    fedilink
    English
    arrow-up
    3
    arrow-down
    1
    ·
    11 months ago

    That’s exactly what robot.txt is… they spell out that they don’t want you to access this site with an automated system.

    • commie@lemmy.dbzer0.comOP
      link
      fedilink
      English
      arrow-up
      3
      arrow-down
      2
      ·
      11 months ago

      right. so hiring 50 college kids to manually visit every page and cache it for study is fine.

      • Batman@lemmy.world
        link
        fedilink
        English
        arrow-up
        2
        arrow-down
        1
        ·
        11 months ago

        That would probably be more expensive than just paying companies. But it is morally different because a human did visit their website so their good will was not violated as they expressed this consent when they published the website.