The New York Times blocks OpenAI’s web crawler
The New York Times has blocked OpenAI’s web crawler, meaning that OpenAI can’t use content from the publication to train its AI models. If you check the NYT’s robots.txt page, you can see that the NYT disallows GPTBot, the crawler that OpenAI introduced earlier this month. Based on the Internet Archive’s Wayback Machine, it appears NYT blocked the crawler as early as August 17th.
The change comes after the NYT updated its terms of service at the beginning of this month to prohibit the use of its content to train AI models. The NYT and OpenAI didn’t immediately reply to a request for comment.
The New York Times has blocked OpenAI’s web crawler, meaning that OpenAI can’t use content from the publication to train its AI models. If you check the NYT’s robots.txt page, you can see that the NYT disallows GPTBot, the crawler that OpenAI introduced earlier this month. Based on the Internet…
Recent Posts
- Best Buy slashes up to $400 off Apple tech in a limited-time sale — get AirPods, MacBooks, iPads and Apple Watches from $99.99
- The Instagram Plus subscription has officially launched
- Cyberdecks used to look like little laptops, but now they’re getting more personal
- Canada Prime Minister Mark Carney announces questionable national AI strategy
- Kevin O’Leary agrees to downsize massive Utah data center
Archives
- June 2026
- May 2026
- April 2026
- March 2026
- February 2026
- January 2026
- December 2025
- November 2025
- October 2025
- September 2025
- August 2025
- July 2025
- June 2025
- May 2025
- April 2025
- March 2025
- February 2025
- January 2025
- December 2024
- November 2024
- October 2024
- September 2024
- August 2024
- July 2024
- June 2024
- May 2024
- April 2024
- March 2024
- February 2024
- January 2024
- December 2023
- November 2023
- October 2023
- September 2023
- August 2023
- July 2023
- June 2023