Rights, Licensing, and Text-and-Data-Mining (TDM) Permission Strategy

The rapid commercialization of generative artificial intelligence has fundamentally fractured the historical consensus governing web crawling and digital copyright. For a publisher such as IntelligenceCompact.com, the strategic objective is highly nuanced: affirmatively allowing public material to be indexed by AI search engines, retrieved into real-time AI answers, and ingested for large language model (LLM) training, while simultaneously retaining ordinary copyright protections, intellectual property rights, and stringent attribution expectations.
Achieving this balance requires navigating an intricate matrix of technical protocols, machine-readable licensing standards, and deeply divergent jurisdictional legal frameworks. Historically, publishers relied on the Robots Exclusion Protocol (robots.txt) and standard copyright notices to govern automated access1. However, the advent of massive-scale text and data mining (TDM) has rendered these tools legally and technically insufficient3. Generative AI training pipelines tokenize and ingest copyrighted content into latent model weights, fundamentally stripping the original expression of its discrete attribution and copyright management information4. Conversely, overly aggressive technical blocking or blanket legal opt-outs inadvertently hide the publisher from the emerging ecosystem of AI-driven retrieval and answer engines, causing severe visibility and citation outages1.
This comprehensive report evaluates the legal and technical consequences of affirmative AI data licensing. It contrasts legacy mechanisms like robots.txt with modern semantic standards like llms.txt, edge-network crawler controls, and standardized machine-readable rights expressions such as the W3C Text and Data Mining Reservation Protocol (TDMRep) and the Open Digital Rights Language (ODRL). Furthermore, it dissects the jurisdictional divide between the United States—which relies on common law, the Computer Fraud and Abuse Act (CFAA), and copyright fair use jurisprudence—and the European Union, which has codified a strict statutory opt-out regime under the Directive on Copyright in the Digital Single Market (CDSM) and the newly enforceable EU AI Act7. The report concludes with a narrowly drafted policy recommendation designed to fulfill the publisher's dual mandate of broad machine learning integration and rigorous intellectual property preservation.

Technical Access Controls and Crawler Governance

The foundational layer of any artificial intelligence ingestion strategy relies on technical directives that communicate with automated web crawlers. However, the ecosystem of AI bots has evolved into a highly specialized landscape, requiring publishers to abandon binary allow/disallow postures in favor of granular, purpose-driven crawler governance6.

The Dichotomy of AI Crawlers: Training vs. Retrieval

The most consequential decision a publisher makes at the technical layer is distinguishing between training-time bots and real-time retrieval bots. AI developers operate entirely separate user agents for these two functions, and they carry opposite implications for digital visibility6.
Training crawlers are deployed to collect static data that will be baked into the future weights of foundational AI models1. Ingestion by these bots ensures that a publisher's domain expertise informs the underlying intelligence of the model, but it offers no guarantee of direct citation, as the content becomes part of the model's generalized knowledge1. Prominent training bots include OpenAI's GPTBot, Anthropic's ClaudeBot (in its historical training capacity), Google's Google-Extended, and Common Crawl's CCBot, which serves as the foundational dataset for numerous open-source models1.
Retrieval crawlers, conversely, operate dynamically to power Retrieval-Augmented Generation (RAG) and AI search features. These bots fetch content in real time to answer specific user queries, generating direct citations, footnotes, and referral traffic6. If a publisher blocks retrieval bots, their content becomes entirely invisible to live AI answer engines, regardless of the quality of the underlying material6. Major retrieval agents include OpenAI's OAI-SearchBot and ChatGPT-User, Anthropic's Claude-SearchBot and claude-web, and Perplexity's PerplexityBot1.
Table 1 categorizes the primary AI user agents traversing the web and details the operational consequences of permitting their access.

Crawler Token / User-Agent	Operator	Primary Function	Technical Consequence of Allowing Access
GPTBot	OpenAI	Training	Content is ingested for future GPT foundation models; attribution relies entirely on legal licensing, not technical extraction1.
OAI-SearchBot	OpenAI	Retrieval	Content is fetched dynamically for ChatGPT Search; high probability of direct real-time citation and link rendering1.
Google-Extended	Google	Training	Data feeds Gemini training models; distinct from Googlebot, meaning blocking it does not harm traditional search SEO1.
ClaudeBot	Anthropic	Training	Anthropic's primary crawler historically used for corpus collection and model alignment1.
PerplexityBot	Perplexity	Retrieval	Powers the Perplexity answer engine index; essential for inclusion in Perplexity source citations1.
CCBot	Common Crawl	Training	Content enters the most widely used open repository for independent model training; highest risk of uncredited distribution1.
Bytespider	ByteDance	Training/Scraping	Feeds TikTok/APAC AI models; historically documented as aggressive and often disrespectful of standard exclusion protocols1.

Limitations and Precedence Rules of robots.txt

The Robots Exclusion Protocol (RFC 9309), instantiated via the robots.txt file, is the industry standard for communicating with crawlers. However, relying on robots.txt to govern AI access introduces severe technical limitations. The protocol is entirely voluntary, operating as a polite request rather than an enforceable technical barrier1. Furthermore, robots.txt lacks the granularity required for conditional licensing; it can grant or deny access to specific paths, but it cannot communicate conditions such as "allow access only if attribution is provided"3.
Misconfigurations in robots.txt are the primary cause of self-inflicted AI visibility outages. The protocol operates on strict precedence rules: a crawler will obey only the single most specific user-agent group that matches its string, and within that group, the longest matching path rule wins1. If a publisher defines a specific block for User-agent: GPTBot, that crawler will entirely ignore any directives placed under the generic User-agent: * wildcard block1. Consequently, administrators must explicitly duplicate standard allow/disallow paths within every distinct AI user-agent block to maintain consistent site architecture1.
Another critical technical blindspot of standalone AI crawlers is their inability to render client-side JavaScript. A joint telemetry analysis conducted by Vercel and MERJ across 500 million GPTBot fetches found zero evidence of JavaScript execution6. Even when the bot downloaded JS files, it never executed them to render the Document Object Model (DOM)6. If IntelligenceCompact.com relies on client-side rendering to display its core content, product descriptions, or rights metadata, AI crawlers will retrieve an empty shell6. Therefore, all content intended for AI ingestion, alongside its associated legal metadata, must be delivered strictly server-side via raw HTML or JSON-LD structures6.

Vendor-Specific Network Controls and Edge-Layer Signals

Because robots.txt is easily ignored by aggressive scrapers and cannot enforce conditional access, sophisticated publishers are shifting AI governance to the edge network layer10. Content Delivery Networks (CDNs) like Cloudflare have introduced advanced Content Signals and AI Bot Management frameworks that provide dynamic control over AI crawlers14.
Edge-layer verification actively scrutinizes the IP addresses, request signatures, and behavioral patterns of incoming bots to ensure they match their declared user-agent strings, blocking spoofed scrapers that attempt to steal data under the guise of legitimate bots6. Furthermore, Cloudflare has introduced features such as "Markdown for Agents," which allows the CDN to intercept AI crawler requests and explicitly serve clean, markdown-optimized versions of web pages18. This preserves the AI model's context window by stripping out heavy CSS, navigation menus, and non-essential DOM elements. As AI economics evolve, edge providers are also pioneering "Pay-per-crawl" architectures, suggesting that the future of AI training ingestion will rely on programmatic, transactional access validated at the CDN layer rather than simple binary blocking19. For a publisher seeking conditional access, verifying the identity of the AI agent at the edge is the prerequisite step before delivering any machine-readable legal licensing.

Semantic Routing and Agentic Discovery via llms.txt

If network controls act as the security gate, the emerging llms.txt proposal serves as a highly specialized semantic map for autonomous AI agents20. Proposed in late 2024 by Jeremy Howard of Answer.AI, the llms.txt specification was engineered to solve the acute problem of context window limitations when LLMs attempt to parse complex websites20.

Structure and Implementation of the Standard

The llms.txt specification dictates that a markdown-formatted file should reside at the root of a domain (/llms.txt), providing AI models with a curated guide to the site's most critical and high-fidelity content20. Unlike a standard sitemap.xml, which exhaustively lists every URL for search engines, llms.txt is designed to be highly constrained and directly readable by an LLM21.
The specification requires a strict markdown hierarchy that can be parsed through classic programming techniques or regex23. The file must begin with an H1 header declaring the project or site name, followed immediately by a blockquote containing a brief semantic summary of the entity23. The remainder of the file utilizes H2 headers to categorize links to core documentation, API references, or foundational datasets20. Crucially, the URLs provided in the llms.txt file should point to LLM-optimized endpoints—typically pages that render purely in markdown (e.g., appending a .md extension to the URL) rather than heavy HTML pages23.
A companion file, /llms-full.txt, is also defined by the specification. Rather than linking out to URLs, this file concatenates the actual textual content of the site's core pages into a single, comprehensive markdown document, allowing an AI agent to ingest the entirety of the site's critical context in a single network request21.

Operational Reality vs. Visibility Myths

Considerable confusion exists in the technical community regarding the purpose of llms.txt. It is frequently, and incorrectly, marketed as a lever for AI Search Engine Optimization (GEO) or a mechanism to control legal access to training data25. It serves neither function. It is an operational manual designed for the discovery and execution phases of agentic interaction, not the retrieval or indexing phases26.
Telemetry data reinforces this distinction. An analysis conducted by Ahrefs across 137,000 domains revealed that 97% of deployed llms.txt files received zero requests from major AI vendors; of the 3% that were fetched, the vast majority were triggered by automated SEO audit tools rather than foundation model crawlers25. Google has explicitly stated that llms.txt files are not utilized for Google Search or AI Overviews25.
However, for a publisher explicitly desiring to be utilized by AI, implementing llms.txt carries massive option value at near-zero cost25. While it does not attract crawlers, it ensures that when an autonomous agent (such as an AI coding assistant or a specialized research bot) is deliberately directed to IntelligenceCompact.com, it immediately locates a perfectly structured, hallucination-resistant index of the publisher's best data21. It acts as a semantic conflict resolution layer, explicitly informing the agent of normalization rules, API workflows, and the exact intent of the platform26.

Machine-Readable Licensing and TDM Protocols

Because robots.txt cannot convey legal nuance, and llms.txt has no binding authority, the core of an affirmative AI licensing strategy must rely on standardized, machine-readable rights expressions. This allows a publisher to mathematically declare to any visiting crawler that text and data mining is permitted, but strictly subject to defined duties such as attribution and compensation.

The W3C Text and Data Mining Reservation Protocol (TDMRep)

The paramount standard for machine-readable rights reservation is TDMRep, published by the W3C TDM Reservation Protocol Community Group3. The protocol was expressly designed to operationalize the legal opt-out requirements of the European Union's CDSM Directive, providing a standardized mechanism for rightsholders to declare their TDM preferences3.
The TDMRep architecture is remarkably simple, relying on two core properties:

1. tdm-reservation: A boolean integer. A value of 1 indicates that the rightsholder legally reserves their text and data mining rights. A value of 0 indicates that TDM rights are not reserved, constituting a blanket waiver that allows free ingestion3.
2. tdm-policy: A URL pointing to a machine-readable policy document that outlines the specific terms, licenses, and conditions under which mining is permitted, alongside rightsholder contact information3.

If IntelligenceCompact.com wishes to allow AI training while strictly enforcing attribution, it is a critical necessity that tdm-reservation is set to 1. This signals that the rights are formally reserved, forcing the AI vendor to negotiate or accept the terms outlined in the accompanying tdm-policy URL9. Setting the reservation to 0 under the misconception that it means "allow all mining" would legally waive the publisher's right to demand attribution under EU law9.
TDMRep is designed for multi-channel deployment, ensuring crawlers encounter the rights signal regardless of how they access the domain. The specification defines a strict hierarchy of discovery:

• Site-wide (.well-known): A tdmrep.json file hosted at /.well-known/tdmrep.json allows a domain owner to declare rights for the entire site using regex path matching3.
• HTTP Response Headers: Injecting tdm-reservation: 1 directly into HTTP headers provides highly efficient, server-level declarations that override the site-wide file for specific resources, requiring no HTML parsing by the crawler3.
• HTML Meta Tags: Embedded within the document <head>, these tags provide page-level granularity3.
• Asset-Level Metadata: For downloadable assets like PDFs or EPUBs, TDMRep properties are injected directly into the Extensible Metadata Platform (XMP) namespace (e.g., tdm:reservation and tdm:policy). Tools such as Datalogics PDF Optimizer now natively support the embedding of TDMRep XMP metadata during file compression, ensuring that the legal policy travels inextricably with the document even if it is scraped and re-hosted on a third-party server3.

ODRL (Open Digital Rights Language) and JSON-LD Profiling

The tdm-policy URL declared by the publisher must resolve to a structured, machine-readable format to ensure that automated AI agents can parse and comply with the conditions of use33. The W3C dictates the use of the Open Digital Rights Language (ODRL) 2.2, profiled in JSON-LD format9. ODRL provides a highly granular, programmatic vocabulary for expressing permissions, prohibitions, duties, and constraints9.
A compliant TDM policy must declare an ODRL @type of Offer, indicating a proposal from the rightsholder for specific rights over their assets27. The JSON-LD schema requires an assigner block containing vCard properties (e.g., vcard:fn for the publisher's name, vcard:hasEmail for contact)27.
Crucially, the policy must define a permission array. To affirmatively allow AI ingestion, the permission must include the action tdm:mine9. To prevent this from becoming an unconditional waiver, the policy must append a duty to the permission. Using ODRL syntax, the publisher can stipulate that the tdm:mine action is only lawful if the TDM actor fulfills the odrl:attribute or odrl:compensate constraint9.
An abbreviated example of a properly structured ODRL JSON-LD Offer for conditional AI ingestion:

JSON
{
"@context": [
"http://www.w3.org/ns/odrl.jsonld",
"http://www.w3.org/ns/tdmrep.jsonld"
],
"@type": "Offer",
"uid": "https://intelligencecompact.com/policies/ai-tdm-policy",
"profile": "http://www.w3.org/ns/tdmrep",
"assigner": {
"uid": "https://intelligencecompact.com",
"vcard:fn": "IntelligenceCompact",
"vcard:hasEmail": "mailto:legal@intelligencecompact.com"
},
"permission": [{
"action": "tdm:mine",
"duty": [{
"action": "odrl:attribute",
"constraint": [{
"leftOperand": "odrl:purpose",
"operator": "odrl:eq",
"rightOperand": "tdm:research"
}]
}]
}]
}

This machine-readable syntax ensures that any programmatic agent attempting to parse the policy mathematically understands that the publisher is actively offering the data for ingestion, but that attribution is a non-negotiable duty attached to the license9.

Creative Commons, Custom Licenses, and Alternative Frameworks

While TDMRep and ODRL represent the bleeding edge of machine-readable AI licensing, legacy licensing models like Creative Commons (CC) present significant friction in the context of machine learning.
Creative Commons licenses (e.g., CC-BY for attribution, CC-BY-NC for non-commercial use) are inherently rooted in copyright law, applying strictly to acts of reproduction, distribution, and adaptation4. If an AI model ingests a CC-BY licensed article, memorizes the text, and subsequently outputs a substantially identical copy to a user, the AI provider must provide attribution to avoid copyright infringement4. However, the foundational architecture of LLM training fundamentally strips away discrete attribution; text is converted into mathematical tokens, and the resulting neural weights do not retain the metadata required to generate attribution for non-memorized outputs4. Consequently, CC licenses are highly difficult to enforce at the output layer unless the AI commits verbatim plagiarism4. Recognizing this limitation, Creative Commons is actively exploring "preference signals" to allow creators to indicate their desires regarding AI training independent of strict copyright mechanics, while the Open Future initiative has spearheaded the development of an IETF vocabulary (draft-ietf-aipref-vocab) to express highly detailed AI usage preferences37.
Another alternative framework is the Coalition for Content Provenance and Authenticity (C2PA). C2PA diverges from the text-based TDMRep by focusing on cryptographic asset-level metadata40. It embeds secure "manifests" directly into the structure of images, videos, and documents, establishing the cryptographic provenance of the asset40. The C2PA specification includes a "Training And Data Mining Assertion," which functions as a DoNotTrain protocol40. While highly effective for preventing the stripping of metadata from standalone media files downloaded from the web, C2PA is overly complex for governing the ingestion of broad HTML text data, making TDMRep the superior choice for overall domain governance.
Table 2 contrasts the prevailing machine-readable rights expression frameworks available to publishers.

Framework	Target Asset	Mechanism	Primary Governing Body	Suitability for IntelligenceCompact.com
TDMRep	Web Pages & Text	HTTP Headers, HTML tags, .well-known JSON-LD	W3C Community Group	High. Provides the exact legal reservation required by the EU AI Act while permitting conditional access via ODRL3.
C2PA	Images / Videos / Documents	Cryptographic Manifests	C2PA Consortium	Moderate. Excellent for downloadable media to prevent metadata stripping, but overly complex for basic HTML text40.
CC Licenses	General Content	Human/Machine readable standard licenses	Creative Commons	Low to Moderate. Does not inherently stop training ingestion unless the jurisdiction classifies training as infringement; fails to enforce attribution on non-memorized outputs4.
Content Signals	Domain Traffic	CDN/Edge layer behavioral tags	Cloudflare / Edge Providers	High (Complementary). Translates the legal preferences of TDMRep into enforceable network blocking rules14.

Jurisdictional Legal Frameworks: The US vs. EU Divide

The efficacy of any technical or machine-readable licensing strategy relies entirely on the underlying legal enforcement mechanisms. The global landscape governing AI data scraping is highly fractured, defined by a stark contrast between the European Union's statutory regulatory regimes and the United States' reliance on common law, the Computer Fraud and Abuse Act, and copyright litigation.

The European Union: Statutory Opt-Outs and Extraterritoriality

The European Union has established the world's most codified framework addressing Text and Data Mining, elevating AI scraping from a pure copyright issue to a matter of statutory product safety and fundamental rights.
The CDSM Directive (Articles 3 and 4): Directive (EU) 2019/790 (the CDSM Directive) explicitly regulates TDM. Article 3 provides a mandatory, un-waivable exception allowing research organizations and cultural heritage institutions to carry out TDM for non-commercial scientific research2. Crucially, Article 4 provides a much broader exception allowing anyone (including commercial AI vendors) to mine lawfully accessed content for any purpose, unless the rightsholder has expressly reserved those rights in an appropriate, machine-readable manner2. If a publisher fails to deploy a machine-readable opt-out (such as TDMRep), the AI developer has a statutory right to ingest the data without compensation or attribution29.
The EU AI Act and Article 53 Obligations: The EU AI Act dramatically amplifies the power of the CDSM Directive's opt-out mechanism. Article 53(1)(c) of the AI Act dictates that providers of General Purpose AI (GPAI) models must put in place a policy to respect EU copyright law, specifically requiring them to use "state-of-the-art technologies" to identify and comply with rights reservations expressed pursuant to Article 4(3) of the CDSM Directive7. Furthermore, Article 53(1)(d) mandates that AI providers draw up and make publicly available a "sufficiently detailed summary" about the content used for training their models, utilizing templates issued by the European AI Office, ensuring unprecedented transparency for rightsholders7.
The most profound consequence of the AI Act is its aggressive extraterritorial reach. Recital 106 states that the obligation to respect TDM opt-outs applies "regardless of the jurisdiction in which the copyright-relevant acts underpinning the training of those general-purpose AI models take place"3. If a United States-based AI company scrapes content from a US-based server to train a model in California, that provider must still honor the EU TDM opt-out if they intend to place the resulting AI model, or products derived from it, on the European Union market7. The AI Act imposes catastrophic penalties for non-compliance, allowing regulators to fine model providers up to 3% of their global annual turnover or €15 million, whichever is higher43. By implementing the TDMRep standard, a publisher instantly triggers this statutory protection, granting it massive regulatory leverage over global AI developers37.

The United States: Hacking Laws, Copyright, and Contract Disputes

In stark contrast to the EU, the United States lacks any specific statutory opt-out for TDM. Instead, publishers attempting to control AI ingestion must navigate a patchwork of anti-hacking statutes, copyright fair use defenses, and common law contract claims.
The CFAA and the Legalization of Public Data Scraping: Historically, publishers attempted to use the Computer Fraud and Abuse Act (CFAA), which criminalizes "unauthorized access" to computer systems, to prosecute web scrapers46. However, a line of foundational federal cases has effectively immunized the scraping of public web data from CFAA liability. In the landmark case hiQ Labs v. LinkedIn (2022), the Ninth Circuit ruled that scraping publicly visible data does not constitute a CFAA violation because a public website has no authentication "gates" to bypass; if the data is available to a browser without a login, automated scraping is not unauthorized access8. This rationale was deeply influenced by the Supreme Court's ruling in Van Buren v. United States (2021), which adopted a narrow "gates-up-or-down" model for the CFAA8. This precedent was recently cemented in Meta v. Bright Data (2024), where a federal court ruled that Meta could not use the CFAA to prevent the scraping of public Facebook and Instagram profiles8. Consequently, technical barriers like CAPTCHAs and login walls are legally required to trigger CFAA protections; purely public data is fair game from a hacking perspective8.
Copyright, Implied License, and Fair Use: With the CFAA neutered for public data, the battleground has shifted to copyright law. AI developers frequently argue that ingesting copyrighted text for model training constitutes a transformative "Fair Use" under 17 U.S.C. § 107, arguing that the models analyze factual patterns rather than reproduce expressive elements9. Furthermore, they argue that placing material on the open web has historically granted an "implied license" to search engines for indexing and caching purposes, a logic they attempt to extend to LLM training36.
However, rightsholders vehemently contest this, arguing that model training is a highly commercial, non-transformative ingestion that acts as a market substitute for the original work52. While definitive Supreme Court rulings on AI training fair use are pending, cases such as Associated Press v. Meltwater and Thomson Reuters v. ROSS have previously established that scraping and repackaging content for commercial analytics or AI-assisted search without a license can defeat a fair use defense, serving as powerful precedent for publishers55.
DMCA Section 1202(b) and Copyright Management Information: A critical, highly specific legal tool for publishers is Section 1202(b) of the Digital Millennium Copyright Act (DMCA). This statute imposes liability for the intentional removal, alteration, or falsification of Copyright Management Information (CMI)56. In major early AI litigation, such as the class-action lawsuit Doe v. GitHub regarding the Copilot coding assistant, plaintiffs successfully utilized DMCA 1202, arguing that the AI training process unlawfully stripped their open-source licenses and attribution requirements, distributing the code without the legally required CMI5. Ensuring that all content contains robust, machine-readable CMI (such as an ODRL license delivered via TDMRep) strengthens potential claims under DMCA 1202 if an AI vendor ingests the content and generates outputs without honoring the attribution parameters.
Contract Law: The Browsewrap vs. Clickwrap Chasm: When hacking laws and copyright fail, publishers fall back on breach of contract claims via their Terms of Service (ToS). However, the enforceability of ToS against automated scraping bots is highly contingent on the mechanism of agreement46.

• Browsewrap Agreements: Where terms of service are simply hyperlinked passively in a website footer, courts are overwhelmingly skeptical of their enforceability, as there is no proof of mutual assent. Browsewrap provides very weak, practically unenforceable protection against scrapers13.
• Clickwrap Agreements: Where a user (or API client) must affirmatively click "I Agree" or pass through a mandatory prompt before accessing data, the contract is highly enforceable13. The landmark European case Ryanair v. PR Aviation demonstrated that clickwrap ToS can successfully prohibit scraping and result in massive damages even when intellectual property and database laws provide no underlying protection13.

Formulating a Narrowly Drafted Policy for IntelligenceCompact.com

IntelligenceCompact.com faces a specific and highly delicate mandate: to encourage broad digital dissemination, ensure ingestion by foundational machine learning models, and guarantee real-time visibility in AI search engines, while strictly preventing the unconditional waiver of its intellectual property rights and enforcing attribution expectations.
Achieving this objective requires abandoning generic robots.txt disallows and legacy "all rights reserved" footers. Instead, the publisher must deploy a granular, machine-readable, and legally enforceable architecture that explicitly offers conditional licenses. The following narrowly drafted policy framework accomplishes this objective.

1. Assert the Extraterritorial TDM Opt-Out

To avoid the automatic, statutory waiver of rights under Article 4 of the EU CDSM Directive, the publisher must immediately declare a machine-readable opt-out9. Even though the objective is to allow training, this allowance must be made on the publisher's terms to prevent the AI vendor from claiming a free statutory exception.

• Implementation: Deploy the W3C TDMRep standard by placing a tdmrep.json file in the /.well-known/ directory of the domain3.
• Configuration: The critical property tdm-reservation must be set to 1 (Rights Reserved)9.
• Strategic Rationale: Setting reservation: 1 instantly triggers the compliance and transparency obligations of the EU AI Act (Article 53), forcing global AI providers to legally recognize the publisher's sovereignty over the data, regardless of where the AI company is headquartered7.

2. Draft a Permissive, Conditional ODRL Offer

With rights legally reserved, the publisher must utilize the tdm-policy URL to point to a machine-readable document that explicitly offers the right to mine, conditional upon attribution9.

• Implementation: Host a JSON-LD policy file structured in ODRL 2.2 syntax, declaring an ODRL @type of Offer9.
• Configuration: The policy must grant the tdm:mine permission action9. However, it must append a strict duty to this permission, utilizing the odrl:attribute constraint9. The policy should state that if the content is ingested into a foundational model, the publisher must be listed in the AI Act Article 53 training data transparency summary. If the content is used in a real-time Retrieval-Augmented Generation (RAG) system, the AI output must include a direct hyperlink to the source URL.
• Strategic Rationale: This creates a conditional license. If the AI vendor strips the ODRL metadata during ingestion, the publisher has grounds for a DMCA Section 1202(b) CMI violation lawsuit57.

3. Bifurcate Crawler Directives in robots.txt and the Edge Network

The robots.txt file must be optimized to distinguish between training bots and retrieval bots, ensuring maximum visibility in AI search engines while funneling training bots toward the legal licensing layer.

• Implementation: Explicitly list known AI user agents in robots.txt1.
• Configuration: Retrieval bots designed for search (e.g., OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot) must be explicitly allowed (Allow: /) to ensure the site is indexed for real-time citations and referral traffic1. Training bots (e.g., GPTBot, Google-Extended, CCBot) should also be allowed, but their access must be strictly verified via CDN edge-network rules1. The CDN must inject the TDMRep HTTP headers (tdm-reservation: 1 and tdm-policy: [URL]) into every successful 200 OK response served to these bots3.
• Strategic Rationale: Injecting the legal policy directly into the HTTP headers guarantees that the AI vendor possesses actual, technical notice of the conditional license, defeating any claim of ignorance in a court of law.

4. Implement Agentic Semantic Routing (llms.txt)

To ensure that autonomous AI agents do not hallucinate facts about IntelligenceCompact.com and can easily digest its highest-value datasets, the publisher must optimize the semantic routing layer.

• Implementation: Deploy an llms.txt and llms-full.txt file at the root directory20.
• Configuration: Curate a brief semantic summary of the platform's intent and provide markdown-based links to the most critical, citation-worthy data assets, stripping away heavy HTML DOM elements23.
• Strategic Rationale: While this does not govern legal rights, it vastly improves the fidelity of the data ingested by AI agents, drastically increasing the likelihood that they will correctly retrieve, understand, and cite the platform's proprietary data during user queries21.

5. Transition to Clickwrap for High-Value Datasets

Because passive browsewrap agreements offer negligible protection against scrapers in a court of law, the publisher must fortify its contractual footing for its most valuable assets13.

• Implementation: For bulk dataset downloads, API access, or premium content tiers, transition away from open URLs and implement a mandatory clickwrap agreement13.
• Configuration: The Terms of Service must require the user (or the API client) to affirmatively click "I Agree" to a clause stating that any AI model trained on the provided data must retain Copyright Management Information (CMI) and provide attribution13.
• Strategic Rationale: Clickwrap creates a highly enforceable, independent cause of action for breach of contract13. If an AI vendor attempts to launder the data through a shell company or claims copyright fair use, the publisher can bypass the copyright debate entirely and sue directly for breach of contract under precedents like Ryanair v. PR Aviation13.

By executing this sophisticated, multi-layered architecture, IntelligenceCompact.com positions itself perfectly within the modern AI ecosystem. Through the strategic deployment of W3C TDMRep and ODRL JSON-LD, the publisher mathematically commands the formidable extraterritorial enforcement powers of the EU AI Act. Simultaneously, by properly segmenting robots.txt, verifying bots at the CDN edge, and providing pristine semantic routing via llms.txt, the publisher ensures it remains a highly accessible, primary source for AI answer engines. This framework achieves the ultimate objective: frictionless computational ingestion and unparalleled digital visibility, governed continuously by legally binding and inescapable attribution requirements.

Works cited

1. robots.txt in the age of AI crawlers - Flavio Copes, https://flaviocopes.com/robots-txt-ai-crawlers/
2. A Survey of Web Content Control for Generative AI - arXiv, https://arxiv.org/html/2404.02309v1
3. TDM Reservation Protocol – EDRLab, https://www.edrlab.org/open-standards/tdmrep/
4. Tracing Creative Commons Licenses Across AI: Training Data, https://shujisado.org/2026/02/16/tracing-creative-commons-licenses-across-ai-training-data-models-outputs/
5. Microsoft sued for open-source piracy through GitHub Copilot, https://www.bleepingcomputer.com/news/security/microsoft-sued-for-open-source-piracy-through-github-copilot/
6. Crawler Access for AI SEO (GPTBot & ClaudeBot) - Searchbloom, https://www.searchbloom.com/ai-seo/inclusion/crawler-access/
7. The AI Act provisions relating to copyright – Possibility of private, https://legalblogs.wolterskluwer.com/copyright-blog/the-ai-act-provisions-relating-to-copyright-possibility-of-private-enforcement-germany-as-an-example-part-1/
8. Is Web Scraping Legal? Court Rulings Guide, https://mobileproxies.org/blog/is-web-scraping-legal
9. TDM Rights in PDF Documents & the TDMRep Protocol - Mapsoft, https://mapsoft.com/posts/tdmrep-pdf-mining-rights.html
10. Technical SEO for AI Crawlers: Log Files, GPTBot & llms.txt, https://authority.builders/blog/technical-seo-ai-crawlers/
11. AI Crawlers vs Search Engine Crawlers - Presenc AI, https://presenc.ai/compare/ai-crawlers-vs-search-crawlers
12. AI Crawlers & Bots: the 2026 reference. - Crackle PR, https://www.cracklepr.com/crawlers
13. Is Web Scraping Legal in Europe? How to Scrape and Stay Safe, https://thunderbit.com/blog/web-scraping-legal-europe-guide
14. Block AI crawlers & bots from scraping your site - Cloudflare, https://www.cloudflare.com/the-net/building-cyber-resilience/regain-control-ai-crawlers/
15. Control content use for AI training with Cloudflare's managed robots, https://blog.cloudflare.com/control-content-use-for-ai-training/
16. Giving users choice with Cloudflare's new Content Signals Policy, https://blog.cloudflare.com/content-signals-policy/
17. AI Crawler List: Every Bot, What It Wants, and How to Block It, https://technologychecker.io/blog/ai-crawler-list
18. Cloudflare Debuts Markdown for Agents and Content Signals ... - InfoQ, https://www.infoq.com/news/2026/03/cloudflare-crawler/
19. Introducing pay per crawl: Enabling content owners to charge AI, https://blog.cloudflare.com/introducing-pay-per-crawl/
20. LLMs.txt: The Emerging Standard Reshaping AI-First Content Strategy, https://scalemath.com/blog/llms-txt
21. LLMs.txt Explained | TDS Archive - Medium, https://medium.com/data-science/llms-txt-414d5121bcb3
22. Powering AI Answers with llms.txt and structured data - dev5310, https://www.dev5310.com/en/lab/llms-txt-is-powering-ai-answers
23. /llms.txt—a proposal to provide information to help LLMs use, https://www.answer.ai/posts/2024-09-03-llmstxt.html
24. llms-txt: The /llms.txt file, v2, https://llmstxt.org/
25. llms.txt Explained: Spec, Reality, Working Example, guptadeepak.com, https://guptadeepak.com/llms-txt-explained-spec-and-working-example/
26. llms.txt: Semantic Conflict Resolution - Grounding Page, https://groundingpage.com/facts/llms-txt/
27. TDM Reservation Protocol (TDMRep) - W3C, https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510/
28. TDM Reservation Protocol Community Group - W3C, https://www.w3.org/community/tdmrep/
29. Text and Data Mining Reservation Protocol Community Group | tdm, https://w3c-cg.github.io/tdm-reservation-protocol/
30. TDM Reservation Protocol Version 1.0, Semantic vocabulary - W3C, https://www.w3.org/ns/tdmrep/
31. Defending Your PDFs: Blocking AI Models from Scraping Your Data, https://pdfa.org/defending-your-pdfs-from-ai-scraping/
32. Expressing Text and Data Mining Rights with Datalogics PDF, https://pdfa.org/expressing-text-and-data-mining-rights-with-datalogics-pdf-optimizer-tdmrep/
33. Defining Machine Readability for Usage Preferences and Policy, https://datatracker.ietf.org/doc/draft-vaughan-machine-readability/
34. KI-Optionen im ONIX und Nutzungsvorbehalte digitaler Inhalte, https://www.boersenverein.de/tx_file_download?tx_main_pi1%5BfileUid%5D=26940&tx_main_pi1%5BpageUid%5D=1834&tx_main_pi1%5Breferer%5D=https%3A%2F%2Fwww.boersenverein.de%2Finteressengruppen%2Fig-digital%2Fdownloads%2F&cHash=03a7f88333fc6f854a57edab2c46b76c
35. Understanding CC Licenses and Generative AI - Creative Commons, https://creativecommons.org/2023/08/18/understanding-cc-licenses-and-generative-ai/
36. COPYRIGHT AND THE GENERATIVE-AI SUPPLY CHAIN “Does, https://james.grimmelmann.net/files/articles/talkin-bout-ai-generation.pdf
37. The Enforceability of AI Training Opt-Outs, https://katedowninglaw.com/2025/05/28/the-enforceability-of-ai-training-opt-outs/
38. AI pirates and more from PDF in the Wild!, https://pdfa.org/ai-pirates-and-more-from-pdf-in-the-wild/
39. Methodology & Sources - AI Search Visibility Research - info.link, https://info.link/research/methodology
40. Proceedings of the first International Workshop on Open Web Search, https://djoerdhiemstra.com/wp-content/uploads/wows2024proceedings.pdf
41. Balancing Discovery and Privacy: A Look Into Opt–Out Protocols, https://commoncrawl.org/blog/balancing-discovery-and-privacy-a-look-into-opt-out-protocols
42. The EU AI Act and copyrights compliance - IAPP, https://iapp.org/news/a/the-eu-ai-act-and-copyrights-compliance
43. EU AI Act Copyright Transparency Requirements (2026), https://aicopyrightlegal.com/blog/eu-ai-act-copyright-transparency-requirements
44. Copyright compliance under the EU AI Act for GPAI model providers, https://www.cliffordchance.com/insights/resources/blogs/ip-insights/2025/10/copyright-compliance-under-the-eu-ai-act-for-gpai-model-providers.html
45. Legal Basis - What is the TDM·AI Protocol?, https://docs.tdmai.org/legal-aspects/legal-basis
46. The Case for a Unified Scraping Framework, https://scholarlycommons.law.wlu.edu/cgi/viewcontent.cgi?article=1195&context=wlulr-online
47. Note: Scraping Photographs - NDLScholarship, https://scholarship.law.nd.edu/cgi/viewcontent.cgi?article=1022&context=ndlsjet
48. Key Web Scraping Court Cases: hiQ v. LinkedIn and Beyond, https://iswebscrapinglegal.com/blog/web-scraping-case-law/
49. Is Web Scraping Legal? The Definitive Legal Guide for 2026, https://iswebscrapinglegal.com/blog/web-scraping-legal-guide/
50. Is Web Scraping Legal? 7-Country 2026 Compliance Guide - cloro, https://cloro.dev/blog/website-scraping-legal/
51. Can You Prevent AI From Scraping Your Website Data? District, https://www.afslaw.com/perspectives/ai-law-blog/can-you-prevent-ai-scraping-your-website-data-district-court-says-answer
52. The Law and Ethics of Generative AI - Scholarly Commons, https://scholarlycommons.law.northwestern.edu/cgi/viewcontent.cgi?article=1387&context=njtip
53. WIPO-Turin LLM in IP Law: Copyright Module (Syllabus), https://opencasebook.org/casebooks/406-wipo-turin-llm-in-ip-law-copyright-module-syllabus/as-printable-html/7/
54. A Cure for Twitch: Compulsory License Promoting Video Game Live, https://scholarship.law.marquette.edu/cgi/viewcontent.cgi?article=1301&context=iplr
55. Copyright and Artificial Intelligence, Part 3: Generative AI Training, https://www.copyright.gov/ai/Copyright-and-Artificial-Intelligence-Part-3-Generative-AI-Training-Report-Pre-Publication-Version.pdf
56. DMCA 1202: Complete Guide to Copyright Protection 2025 | Web, https://www.webcopyrightchecker.com/blog/dmca-1202
57. Updates on AI Copyright Law and Policy: Section 1202 of the DMCA, https://www.authorsalliance.org/2025/03/07/updates-on-ai-copyright-law-and-policy-section-1202-of-the-dmca-doe-v-github-and-the-uk-copyright-and-ai-consultation/
58. The first class action suits against GitHub Copilot have been filed, https://www.actuia.com/en/news/the-first-class-action-suits-against-github-copilot-have-been-filed/
59. Putting GenAI on Notice: GenAI Exceptionalism and Contract Law, https://arxiv.org/pdf/2504.00961
60. How to Protect Your Website From AI Scraping | Vondran Legal, https://www.vondranlegal.com/how-to-protect-your-website-from-ai-scraping
61. Findings: Website Compliance Gates — robots.txt, ToS, Legal, https://github.com/jasonmichaelbell78-creator/sonash-v0/blob/main/.research/website-analysis/findings/D3-compliance-gates.md
62. AI Crawler robots.txt Setup - Henry David Photography, https://www.henrydavidphotography.com/resources/blog/ai-crawler-robots-txt-setup

Document provenance

Source file: AI TDM Rights and Licensing.md

Exact source SHA-256: 9701fb4cac9d80a78a8e90d9764920d14e6c1ba0608079912b8f4210e8ba4042

Machine-readable metadata: metadata.json

Citation and provenance guidance: citation policy

Bulk research corpus: corpus.jsonl