Wikipedia in talks to bill AI companies for using its content

Wikipedia co-founder Jimmy Wales said the nonprofit behind the online encyclopedia is seeking more licensing deals with AI companies. The drive is taking place as a series of automated AI bots strain Wikipedia’s infrastructure. 

Wales explained that AI companies use the platform’s freely available knowledge to train large language models (LLMs). He noted that high-volume AI bots crawl the entire site, imposing significant technical and financial burdens on the platform.

“The AI bots … are going across the entirety of the site … so we have to have more servers, we have to have more RAM and memory for caching that, and that costs us a disproportionate amount,” he explained.

Wales has noted that the heavy usage by AI bots forces Wikipedia to expand its servers, increase RAM, and improve caching, all of which come at a high cost.

Although the site is free to individual users, he noted, the more high-volume, automated access by for-profit AI firms is another issue. The nonprofit signed a licensing deal with Google in 2022 and is negotiating similar agreements with other companies.

The nonprofit that operates Wikipedia, the Wikimedia Foundation, relies on relatively small public donations and volunteer editors to function, which wasn’t intended to be used to fund large-scale, for-profit AI developments.

Wales said it is unfair for contributors and donors to underwrite the costs of commercial AI companies using Wikipedia’s content.

Wikipedia warns of unsustainable burden from AI bots

Wikipedia states that it warns about the unsustainable burden of AI bots. Those demands on Wikipedia’s servers have been increasing sharply in recent months. Automated bots have increased traffic — particularly via multimedia downloads — and have resulted in the nonprofit needing to invest heavily in infrastructure. 

Much of this traffic originates from AI companies scraping content designed to train their models, rather than from humans. In 2022, the Foundation launched a paid commercial product, Wikimedia Enterprise, to provide access to its core content at scale and offer the platform’s users the necessary tools and resources. 

This service is designed to alleviate pressure on Wikipedia’s live site while providing AI developers with the necessary data to enhance their models. Wales urged AI companies to use this paid option rather than scraping the public site. 

If companies fail to comply with these technical measures, Wales stated, restricting bot access through tools such as AI Crawl Control may be implemented. There is some debate with businesses about the role of public (or commercial) AI in managing personal data that users are now holding on a scale that exceeds what the legal right to free, transparent knowledge, and what the private and public sectors need. 

With an ever-larger share of AI processing relying heavily on large, publicly available datasets, Wikipedia is advocating for a fair approach that compensates both the entities maintaining this data and the businesses. Wikipedia is not immune to its commitment to maintaining neutrality. 

Wikipedia struggles to maintain neutrality amid global conflicts

The website Wikipedia has been in operation for over 20 years as a nonprofit entity managed by the Wikimedia Foundation. Its model is grounded in a global army of volunteer editors, who spend their time creating, editing, and proofreading content.

Public donations are a key ingredient to ensure that it remains open to anyone, everywhere, regardless of the amount of money or geographical range they have available. Wikipedia, a well-known international website and encyclopedia, has also struggled to remain neutral among other sources. 

The problem becomes even more acute when reporting in detail on high-stakes political issues, social movements, or armed conflicts worldwide. Wales said that although most Wikipedia editors themselves are not activists, personal preferences may influence how topics are covered. 

However, he added that he trusts the community, saying editors tend to somehow achieve Wikipedia’s values of fairness and accuracy, even under intense pressure. The platform’s neutrality is further evident in its community policy, peer review methods, and dispute process, among other aspects. 

At the same time, the constant dependence on volunteers and donations highlights the weakness of the system. And as the platform faces new pressures, including an increase in AI companies’ use of its content, ensuring the platform’s core values (neutrality, accuracy, and free access) continue unchanged becomes increasingly important.

Want your project in front of crypto’s top minds? Feature it in our next industry report, where data meets impact.