X Revamps Terms of Service to Prohibit Data Scraping and Crawling

• X

X, the social network owned by Elon Musk, formerly known as Twitter, has recently updated its terms of service. The new terms, effective from September 29, now expressly prohibit any form of data scraping or crawling without prior written consent. The intention behind this change is a strategic move to prevent artificial intelligence models from training on their data.

Changes in the Terms of Service

Previously, X's terms of service allowed data crawling by the provisions in the robots.txt file. However, the revised terms have banned scraping or crawling without written consent. This signifies a significant shift in the platform's data acquisition and usage policy.

Adjustments to the robots.txt File

In recent months, the social network has also modified its robots.txt file, which provides guidelines to robot crawlers about the parts of the site they can access. The updated file excludes all crawler bots, except those from Google, from visiting certain site sections. In 2015, Twitter confirmed a partnership with Google to feature tweets in search results. However, it remains to be seen if the terms of this deal have been altered under the new management.

Additionally, X has revised the robots.txt file to prevent crawlers from accessing data such as likes and retweets associated with specific posts and an account's media, photos, and likes.

Musk's Stand Against Data Scraping

Elon Musk, the owner of X, has opposed companies using the social network's data to train AI models. Earlier this year, he threatened to sue Microsoft for allegedly using Twitter's data illegally to train AI models. In July, Musk filed a lawsuit against unidentified companies for data scraping.

Privacy Policy Updates

X has not only made changes to its terms of service but has also updated its privacy policy. The new policy mentions that the platform may use public data to train AI models. This update aligns with Musk's earlier statement about xAI, a company he founded in July, which intends to use public data, such as tweets, for training its models. The revised policy also covers gathering users' biometric data, educational background, and job history.