Apple held talks with DuckDuckGo to replace Alphabet's Google as the default search engine for the private mode on Apple's Safari browser, the Bloomberg News reported on Wednesday, citing people familiar with the discussions.
  • darth_helmet@sh.itjust.worksEnglish
    2·
    1 year ago

    It’s conventional to do this, but a user agent string is entirely up to the client, and robots.txt is just a suggestion.

    So for the best results, you probably want to mock Google’s crawler because it’s suicidal to block that if you want search traffic.