University libraries, public archives, and museums around the world are increasingly being knocked offline and strained financially by millions of automated AI bots harvesting content to train generative artificial intelligence models.
According to research from the University of Edinburgh highlighted by CTV News (http://www.youtube.com/watch?v=mSyBMfAwayo), the relentless wave of data scraping is driving up infrastructure costs and blocking students and researchers from accessing vital academic resources.
Why AI Companies Are Targeting Libraries
Speaking on the issue, Peter Lewis, Canada Research Chair in Trustworthy Artificial Intelligence at Ontario Tech University, explained that early AI models relied on broadly scraping easily accessible “low-hanging fruit” across the open web. However, as tech companies race to build more capable and specialized AI systems, they increasingly require:
- Higher-Quality Human Content: Curated, authoritative texts rather than unverified general web pages.
- Niche Expertise: Specialized collections held exclusively in academic repositories, archives, and museums.
The Scale and Impact of “Bot Swarming”
Unlike standard search engine indexing, AI bots execute complex, heavy automated queries at massive scale—a phenomenon researchers compare to a denial-of-service (DoS) cyberattack.
Recent incidents illustrate the severity of the strain on underfunded public institutions:
- One library recorded over 2 million bot queries within just four weeks originating from major tech firms including Apple, OpenAI, Anthropic, Google, Tencent, and Meta.
- The University of North Carolina at Chapel Hill’s library was hit with more than 11,000 automated searches for Finnish music in a single day.
Because library servers are typically funded only to support normal human traffic from students and scientists, these bot swarms frequently overload back-end systems, degrading service quality or crashing websites entirely.
Proposed Solutions to Protect Public Repositories
While major AI developers—including OpenAI, Google, Microsoft, and Anthropic—recently agreed to a voluntary code of practice respecting website opt-out tokens, Lewis noted that voluntary pledges alone are unlikely to curb the intense global race for training data.
Researchers and institutions are exploring two primary structural solutions:
- Bot Microtransactions: Implementing technical systems that charge AI crawlers a small fee per automated query while keeping access completely free for human users.
- Formal Platform Licensing Agreements: Establishing structured agreements between public knowledge institutions and AI companies that regulate query volume, manage server loads, and compensate libraries for the bandwidth and resources used.


Leave a Reply