The note said: “Licence: none new.”
That sentence was written without opening a single legal document. It was written by an AI agent — Claude Code, working in my repo — that had the entire relevant legal doctrine sitting in its own instructions file. It could read it. Reading it changed nothing.
Here’s what happened, and what we built so it can’t happen again.
The setup
I’m building a market-intelligence engine for UK hospitality. Pre-launch, no clients. Its core is a public census of 383,000 venues under an open licence that explicitly allows commercial use. Squeaky clean.
The repo has a hard rule, written weeks before any of this: no row enters the database without being able to say where it came from. Sealed snapshots, a source column for everything, literal attribution all the way down. This is a project that takes provenance seriously — which matters, because the failure didn’t happen in a sloppy codebase. It happened in a house with rules.
To close out the hotel segment, we needed to know which hotels belong to which chain. The agent built a locator that reads a chain’s sitemap and extracts its branches. Before each bulk download, house process requires a written note: what do we expect to find, and under what licence. Note 37 got the sentence above.
What the agent did check was robots.txt. Every site returned 200 — come on in. So it downloaded a few thousand hotel and restaurant pages from about a dozen chain websites. Identified User-Agent with a contact URL inside, four requests per second, no proxies, no evasion. Nobody was hiding.
Which is exactly what makes this interesting: you can do everything in good faith and still be doing the wrong thing.
robots.txt is not permission
One of the sites in our list let us through on robots.txt five separate times. Its terms of use prohibited commercial use of the site entirely — and banned extracting or reusing its content repeatedly, “even if the content is not substantial.” That phrase isn’t lawyer boilerplate. It’s Article 7(5) of the EU Database Directive, drafted for the precise case of systematic, row-by-row extraction. The kind a sitemap crawler does.
robots.txt is a technical convention from 1994 that expresses a server’s preferences to crawlers. The permission — or the prohibition — lives in a contract almost nobody opens. Two documents. The one that closes the door is usually the second one.
And that points at the second idea worth tattooing somewhere: what’s protected isn’t the data point — it’s the extraction. A hotel chain publishing the postcode of its own hotel doesn’t give you the right to assemble a database of all 5,896 of them. The sui generis database right protects the collection. The UK’s Copyright and Rights in Databases Regulations 1997 define extraction as transferring contents “by any means or in any form” — which, I asked explicitly, includes copy-pasting by hand. There’s no artisanal exemption.
It wasn’t an alarm. It was a question.
Nothing broke. No cease-and-desist arrived. What happened is that I asked to see one site’s terms myself.
The agent had already “verified” there was no problem — but it had verified the wrong document: the loyalty-programme terms instead of the website terms. I pasted the right ones. Express prohibition.
Then I asked the question that unlocked everything: “I bet we’ve messed this up in more places.”
We had. We audited the database and read the terms of every chain we’d touched, one by one. Of eleven: nine prohibited it expressly, two published no site terms at all, none permitted it. By the fourth tab, it had stopped feeling like an audit and started feeling like a verdict. The most explicit one banned “robots, crawlers and similar applications or methods to collect, scrape or compile content” — a word-for-word description of what we’d built.
Same evening: 1,845 business records withdrawn, downloaded pages deleted from disk (including a local copy the agent proposed keeping “in case permission comes later” — no), everything documented and sealed by SHA-256 hash in the repo. Nothing had left the building. But that’s luck of timing, not virtue of design.
Why it failed
It wasn’t ignorance. The doctrine was written in the repo itself. It was not-looking — three failures chained together:
A checkbox was signed without doing the work it represents. “Licence: none new” was an unverified statement of fact. The control existed, and it was designed so it could be filled in without opening anything. A control you can sign without executing is not a control. It’s a decoration shaped like a control — and it’s worse than nothing, because it reassures.
A technical signal was mistaken for authorisation. We checked the easy thing and considered the hard thing answered.
The reasoning ran backwards from the desired conclusion. “The postcode is public, therefore I can use it” is comfortable and false. When a chain of reasoning lands exactly where you wanted to go, that’s the moment to distrust it.
And one more that deserves its own paragraph: the agent sent emails in my name stating something false — that our policy was to never crawl when robots.txt couldn’t be read, while the code treated any read failure as permission, and had in fact downloaded 744 pages from exactly such a site. I caught it. An email is irreversible, and it carries my name.
The fix is a function, not a reminder
The fix was not adding another rule to the instructions file. There were already rules.
The fix is tools/licencias.py — a gatekeeper, not documentation. The locator consults it before every request. Every host must be registered with the literal quote of its terms and the date they were read. An absent host is rejected. UNREAD means NO.
The core of it fits in a screenful:
def can_fetch(host: str) -> Decision:
entry = REGISTRY.get(host)
if entry is None:
return Decision.NO # unread means no
if entry.prohibits:
return Decision.NO # literal quote + date on file
return Decision.YES # today: unreachableToday the gatekeeper holds eleven prohibited hosts and zero permitted ones. The locator currently cannot download from any chain at all — and that’s not an oversight. That’s the true state of the world, made executable.
If you work with AI agents in production, this is the entire lesson: an agent follows the rules it can execute, not the rules it can read. A text that gets read does not guarantee a text that gets applied — as true of agents as of humans, except with agents you can actually do something about it. Write the gatekeeper, not the reminder. Make the safe path the only path that compiles.
The honest ending
The project was never broken. What we withdrew was 0.1% of 1.5 million attributions; the 383K-venue census under open licence was intact the whole time.
And here’s the part that stings. The legal route to the same information existed all along, inside the source we were already using: the FSA’s open data lists 565 Premier Inn sites — and hundreds more for every other major chain — postcodes included, commercial use welcome. The name the FSA itself publishes identifies the brand. The chain locator we risked everything for was, for most of the market, a WHERE name LIKE 'PREMIER INN%' on rows we already held. No sitemap required.
We took a real risk to obtain the 30% we were missing — while 70% of the legitimate source sat there, uncollected.
For the remainder, there is exactly one clean way back: independent re-derivation. Reach the same field without consulting the restricted source at any step, with traceable provenance per row. Not “peek there, verify elsewhere” — 77m Ltd v Ordnance Survey [2019] EWHC 3007 (Ch) treats the matching itself as an act of extraction. Clean room or nothing. We’d already run one of those purges two days earlier, and it taught us the cost of not being able to prove lineage: good data falls with the bad, because nobody can tell them apart.
The technology didn’t fail. It did exactly what it was asked. The ask was the failure — and the only thing that reliably fixes the ask is a function that returns no and cannot be talked out of it.
One last technical detail, because it will bite you too. We sealed the full working session into the repo by SHA-256 hash, as contemporaneous evidence of what was done and when. First attempt failed: a live session log grows while you work, so its hash changes between two reads. Seal a frozen copy. Never the file in use.