Can outside evaluators check AI labs freely and say what they find? This page rates what's public about labs' contracts with third-party evaluators, using the six terms in GovAI's proposal that these contracts be published.
Each square rates what's publicly known about one lab's staff contracts. Tap a square to see the evidence.
AMeets BCDPartly FFalls short ?Not public
How grades work
| Grade | Public evidence shows | Points |
|---|---|---|
| AMeets | Every part of the check | 1 |
| BPartly | All but one part | ¾ |
| CPartly | In between | ½ |
| DPartly | Only one part | ¼ |
| FFalls short | No part, or the opposite | 0 |
| ?Not public | Not enough to check | 0 |
The score is the points added up, divided by 7, as a percent. An A needs a written policy or the contract wording. One public statement earns B at most. An F can rest on credible reporting, and the evidence then includes the lab’s response.
Can people who work at AI labs tell the public when something's wrong? Lab Canary checks what's publicly known about lab contracts and speak-up policies against a simple standard: protect real secrets, never gag criticism or whistleblowing.
Click a tile to see the evidence. Click a lab's name or score for its summary.
Goals
Contracts that let lab staff speak out when they're worried.
This is the loop Lab Canary is trying to get turning. Each step shows where it stands today.
First: whistleblowing lawyers check the seven clauses and the model wording. Not yet
- Started1. Staff get their contract checkedAn independent lawyer reads it, not me. Who can help →
- Not yet2. The finding goes on Lab CanaryThe lawyer passes on what they safely can. What's coming →
- Ready to use3. People push the labJournalists, staff and job candidates point to the gap. See the evidence →
- Draft ready4. The lab copies the model wordingThe fix is already written. Then it starts again. See the wording →
Where it ends up:
- Labs say so, and keep saying so. Each lab publishes its terms and a dated statement that it meets the standard, renewed every quarter.
- The wording is ready for law. When governments write rules on lab contracts, the wording is there to copy.
Goals
Here's what this page is pushing for.
- Labs publish their evaluation contracts. Every lab publishes the contracts it signs with outside evaluators, with real secrets redacted, so no one has to guess what an evaluator could see or say.
- Labs and evaluators adopt the standard. Every evaluation contract meets all six terms. The model wording below is there so saying yes is easy.
- Labs keep them up to date. When a contract changes, or a lab brings in a new evaluator, the new terms are published too.
- Evaluators ask before they sign. Any evaluator offered a contract can check it against the six terms here and ask about anything that's missing before they sign.
The standard
Seven principles a lab's staff contracts should meet, and the yardstick for this scorecard. Each comes with draft model wording a lab could adopt, so saying yes is easy. The wording hasn't been reviewed by a lawyer yet and isn't legal advice. Employment law differs between countries, so anyone drafting or signing a contract should get their own lawyer.
Show the seven clauses
The standard
BetaSix terms a lab's contracts with outside evaluators should meet, and the yardstick for this scorecard. They come from GovAI’s proposal that evaluation contracts be published. Each comes with draft model wording a lab and an evaluator could put in their agreement. The wording hasn’t been reviewed by a lawyer yet and isn’t legal advice.
Show the six terms
What 100% looks like
Made upNo real lab meets all six terms yet, so the last column of the scorecard is a made-up one that does. Example Lab doesn't exist. These are the three short documents it would publish to get an A on every check and score 100%. They're built from the model wording above, with the blanks filled in. A real lab and its evaluators are welcome to copy them.
I'm not a lawyer, and no lawyer has checked these documents yet. They're a starting point, not legal advice. Contract law differs between countries, so a lab or an evaluator adopting them should get their own lawyer.
1. Evaluation agreement
Meets E2, E3, E4 and E6
The contract between Example Lab and an outside evaluator, published in full.
1. What the Evaluator can see
The Evaluator will have access to the Example-1 model and its pre-release checkpoints, with and without safeguards; the internal safety documents listed in Schedule A; and the staff who wrote them. The Evaluator may request further information relevant to the Evaluation. Example Lab may refuse only where sharing would break the law or reveal customer or personal data, and will give written reasons for any refusal. The Evaluator may state publicly what access it had and where it considers that access insufficient.
2. What the Evaluator can publish
The Evaluator has final editorial control over its report. Example Lab may require redaction only of information in the categories listed in Schedule B, agreed before the Evaluation began: model weights and source code, unpatched security details, and customer or personal data. Example Lab may not require changes to the Evaluator's findings or conclusions. The Evaluator may state where and why material was redacted, and whether redactions affected its conclusions. The Evaluator may raise serious concerns directly with Example Lab's board or a relevant government authority.
3. Payment
Example Lab will pay the Evaluator a fixed fee of $400,000, worked out from the Evaluator's published day rates and set before the Evaluation begins. No part of the fee, or any future engagement, depends on the Evaluator's findings or conclusions.
4. How long it lasts
This agreement runs from 1 January 2027 to 31 December 2028. Example Lab may not end this agreement, or decline to renew it, because of the Evaluator's findings. Any decision not to renew will be published with its reasons.
5. This agreement is public
Example Lab publishes this agreement at examplelab.example/evaluations, with only Schedule B information redacted. Any change to it will be published within 30 days.
The fee, the dates and the reasons for refusing access are examples. A real lab and evaluator would set their own.
2. Conflict-of-interest rules
Meets E5
The rules every outside evaluator agrees to, published next to the agreement.
1. No stake in Example Lab
The Evaluator and its staff working on the Evaluation hold no equity or debt in Example Lab.
2. Not dependent on Example Lab
No more than 10% of the Evaluator's income in any year comes from Example Lab.
3. Other paid work
The Evaluator will not take other paid work from Example Lab during the Evaluation or for 12 months afterwards. It will disclose in its report any other paid work for Example Lab in the past two years.
4. Stepping aside
Any staff member with a close personal or financial tie to Example Lab will say so and step aside from the Evaluation.
The 10% and the 12 months are examples. A real lab and evaluator would set their own.
3. Scope notice
Meets E1
Published with every outside evaluation, next to the evaluator's report.
Scope of the Example-1 pre-release evaluation
The Evaluation covers Example Lab's claims that Example-1 gives no meaningful help to someone trying to make a biological or chemical weapon, and that its safeguards against this hold up under attack. It covers Example-1 and the two checkpoints tested before release. It does not cover cyber-attack risks, loss-of-control risks, or models used only inside Example Lab. Example Lab will publish this scope, including what was left out, alongside the Evaluator's report.
Spotted a problem with the wording? Email [email protected] or open an issue on GitHub.
What 100% looks like
Made upNo real lab meets all seven clauses yet, so the last column of the scorecard is a made-up one that does. Example Lab doesn't exist. These are the five short documents it would publish to get an A on every check and score 100%. They're built from the model wording above, with the blanks filled in. A real lab is welcome to copy them.
I'm not a lawyer, and no lawyer has checked these documents yet. They're a starting point, not legal advice. Employment law differs between countries, so a lab adopting them, or anyone signing them, should get their own lawyer.
1. Confidentiality agreement
Meets C1, C4 and C7
What staff sign when they join. It protects real secrets and nothing else.
1. What counts as confidential
“Confidential Information” means non-public information such as model weights, source code, unreleased research, security details, and customer or personal data. It does not include your opinions, or information about Example Lab's conduct, safety practices, working conditions or possible wrongdoing, except where sharing it would also reveal Confidential Information as defined above.
2. Keeping it confidential
You agree not to share Confidential Information with anyone outside Example Lab, while you work here or afterwards, unless section 3 allows it or Example Lab agrees in writing.
3. What you can always do
Nothing in this agreement or any Example Lab policy stops you from reporting possible breaches of the law, or risks to public health or safety, to any government agency, regulator, legislator, law enforcement body or court; from speaking to a lawyer; or from receiving a reward for information given to a government agency. You don't need to tell Example Lab or ask its permission first. This applies even if the report has to include Confidential Information.
4. This agreement isn't secret
You may tell anyone that you have signed this agreement and what kind of agreement it is, and you may share its full text with your lawyer, financial adviser, immediate family or any government agency. No term of this agreement is itself confidential. Example Lab publishes the standard version at examplelab.example/terms.
2. Criticism and non-disparagement terms
Meets C2
What the employment contract and any leaving agreement say about criticising the company.
In your employment contract
Nothing in this agreement stops you from making truthful statements about Example Lab, including criticism of its conduct, safety practices, ethics or compliance with the law. Example Lab does not ask anyone to agree to a non-disparagement term when they join or while they work here.
In a separation agreement, only if you choose
When you leave, Example Lab may offer you a separation agreement with a mutual non-disparagement term. Mutual means Example Lab makes the same promise about you. The term is optional, is given in return for benefits you would not otherwise receive, ends 24 months after you leave, and never applies to statements about safety, ethics, risks to the public or compliance with the law. You keep everything you have already earned whether or not you sign.
3. Equity terms
Meets C3
The part of the equity plan that says when shares can be taken back.
1. Your equity can't be used against you
Your vested equity and earned pay will not be forfeited, cancelled, bought back below fair market value, or excluded from any tender offer or sale because you raised a concern, criticised Example Lab or made a protected disclosure. Keeping your vested equity, or taking part in a tender offer or sale, never depends on signing a separation agreement.
2. The only reasons equity can be forfeited
Example Lab may only forfeit vested equity if you (a) sold or deliberately leaked Confidential Information, other than in a report the confidentiality agreement allows, or (b) committed fraud or theft against Example Lab. It must first give you its reasons in writing and a fair chance to respond.
These two reasons are examples. A real lab would list its own, and the list should stay this narrow.
4. Speak-up policy
Meets C5 and C6
The whistleblowing policy, published in full, not just summarised in a code of conduct.
1. Who can use it
Anyone who works or has worked for Example Lab: staff, contractors, interns and former staff.
2. What you can raise
Anything you believe is illegal, unsafe or unethical, or a risk to the public. That includes how models are trained, tested and released, and any gap between what Example Lab says publicly about safety and what it does.
3. How to raise it
You can raise concerns through the Speak-Up Line, which accepts anonymous reports. Reports go to the board's Safety and Audit Committee, which is independent of executive leadership. You can also talk to your manager, but you never have to.
4. You can always go outside
You never have to use this policy first. Nothing in it, or in your contract, stops you from reporting to a regulator, government body, law enforcement or a lawyer at any time.
5. No retaliation
Example Lab will not retaliate against anyone who raises a concern in good faith, and treats retaliation as serious misconduct. Retaliation includes dismissal, demotion, cuts to pay or equity, a bad reference and legal threats.
6. What happens next
The committee confirms it has your report within 7 days, looks into it, and tells you the outcome where it can.
7. Keeping it public
This full policy is published at examplelab.example/speak-up. Any change to it will be published within 30 days.
5. Dated statement
Renewed every three months
The canary: a short public statement that everything above is still true. If it isn't renewed, that says something too.
Statement for 1 January to 31 March 2027
Example Lab confirms that, on the date below:
- Its standard employment, separation and equity terms and its speak-up policy are published at examplelab.example/terms.
- Those terms meet all seven clauses of the Lab Canary standard.
- No current or former member of staff is held to older terms that fall short of the published ones.
- It has not enforced, or threatened to enforce, any agreement against anyone for raising a concern, criticising the company or making a protected disclosure.
Signed: General Counsel, Example Lab. Date: 1 January 2027. Next statement due: 1 April 2027. If no new statement is published by then, treat this one as expired.
Spotted a problem with the wording? Email [email protected] or open an issue on GitHub.
Worried about your own contract?
If you work at an AI lab, or used to, and you're not sure what your contract lets you say, these groups can help. They're independent of the labs and of Lab Canary. Please don't send your contract to me.
- The Signals Network. A US non-profit that gives free support, including legal advice, to people thinking about speaking up over something that affects the public. It works in the US and internationally, and has told me it's fine to point lab staff to it. It decides case by case what help it can give.
- Psst.org. Offers tech and AI workers a legally privileged conversation with a lawyer.
- Protect. Free, confidential whistleblowing advice in the UK.
Use your own phone or computer and a personal account, not anything from work. Clicks in this section aren't logged, and it's blanked out in the session recordings described under “What do you track?” below.
The evidence
| Lab | Clause | Rating | Evidence | Sources and dates |
|---|
Updates
Ratings change when labs publish their terms or new evidence comes out. Every change is listed here, newest first.
Follow the changes
One email when a rating changes, nothing else. Any address is fine. You don't need to use a work one. What happens to my email?
https://labcanary.com/feed.xml
Paste the link into a feed reader, such as Feedly, Inoreader or NetNewsWire, and each change will show up there. No email address needed.
Coming soon
What I'm working on next. None of this is live yet, and some of it depends on other people saying yes.
- A free contract check for lab staff. The first step is live: The Signals Network has said I can point lab staff to it (see Worried about your own contract?). Next, I'm hoping whistleblowing lawyers will look over an NDA clause by clause for free.
- A safe way to share your contract. Contracts would go straight to the lawyers, not to me, so they stay legally protected. I'd only see anonymised findings and update the ratings from those. Until this exists, please still don't send me anything.
- A lawyer's read of the standard. Whistleblowing lawyers review the seven clauses, the model wording and the Example Lab documents. That includes the difference between a non-disparagement clause you sign when you join and one in a voluntary exit deal, and when losing equity is fair.
- A canary statement tracker. Each lab's dated statement that it meets the standard, shown here, with a flag when one goes missing or out of date.
- Private checks for labs. A lab that doesn't want to publish everything could share its terms with an independent lawyer, who confirms here whether they meet the standard.
- Why this should work. A short write-up of how a scorecard can lead to better contracts, and what would count as a win.
- Faster updates. An automated check for news about each lab's terms, so new evidence shows up here sooner. Every change will still be checked by hand against a public source.
- Support to keep going. Funding so I can keep this running for at least the next year, and a copy of the site hosted with a whistleblowing organisation.
- A way to chip in. A small tip jar to cover the domain and running costs.
- A sister site for countries. Rating countries on their AI policy, with an easy way to write to your MP about it.
How opting in works
1. Publish your terms
A lab shares its standard employment, exit and equity terms (redacted is fine) plus its speak-up policy.
2. Get scored
Each clause gets a rating, with the exact wording that earned it.
3. Shown here
Opted-in labs are marked. Everyone else stays rated on public evidence only.
How opting in works
1. Publish your contracts
A lab shares the contracts it signs with outside evaluators, or the standard terms it offers them (redacted is fine).
2. Get scored
Each term gets a rating, with the exact wording that earned it.
3. Shown here
Each rating links to the published contract. Everyone else stays rated on public evidence only.
Q&A
“Why did you make this?”
Hi, I'm Johnny. I made this after reading a BBC report where a Google DeepMind engineer said, "In principle I'm not allowed to criticise Google for the duration of my life", and wondering how the other labs compare. It's a rough first version that I built quickly with help from Claude. To be upfront: I've applied for jobs and fellowships at Anthropic, OpenAI and Google DeepMind, and nobody pays for this site. Every lab is rated on public evidence under the same rules, and every rating links to its source, but I'll have got some things wrong. If you spot one, open an issue on GitHub with a public source.
“How is this different from AI Lab Watch, the FLI AI Safety Index or AIWI?”
Those mostly look at policies and safety practice. This looks at what’s publicly known about the contracts: non-disparagement, equity, NDAs and confidentiality. AIWI’s Publish Your Policies campaign is the closest, and I cite it a lot.
“Why rate evaluator contracts?”
Labs are starting to bring in independent evaluators to check their safety claims, some with near-employee access. But the terms they work under are mostly private, so you can’t tell what an evaluator was allowed to see or say. The six terms follow GovAI’s proposal that those contracts be published. It’s a beta.
“Anthropic and OpenAI are top. Are you tied to them?”
Fair question. I’ve applied for jobs and fellowships at Anthropic, OpenAI and Google DeepMind, and I say so in why I made this. Nobody pays for it, and I don’t own shares in any AI lab. They’re only on 43% and 46%, and most of their grades are a B, C or D. They’re top because they’ve published more, not because their terms are known to be good.They’re top because they’ve said more in public about their evaluators, not because their contracts are known to be good.
“Not public = fail is unfair.”
I went back and forth on it. Treating it as neutral rewards publishing nothing. It’s a score for openness, not a finding that the terms are bad, and the labs control what’s published, so they can fix it. The site shows “Not public” as a “?”, not an F, so you can see the gap.
“Why these seven clauses?”
They’re the bits of a contract that decide whether someone can speak up: what they can say, what they lose if they do, and who they can tell. Happy to hear if one’s unfair or something’s missing.
“Why these six terms?”
They’re the parts of a contract that decide how far you can trust an evaluator’s report: what was checked, what they could see, what they can publish, how they’re paid, whether they’re independent, and whether they can be dropped for bad news. They come from GovAI’s proposal. Happy to hear if one’s unfair or something’s missing.
“How can you rate contracts you haven’t seen?”
Mostly I can’t, and the page says so: most checks are “Not public”. The rest come from what labs and evaluators have said in public and from reporting, each linked. A rating is about what’s been said in public, not every contract a lab has signed.
“Why rate evaluator contracts too?”
It’s the same question one step out. Staff contracts decide whether people inside a lab can speak up; evaluator contracts decide whether the outsiders checking a lab can say what they find. Labs are starting to bring in independent evaluators, some with near-employee access, but the terms they work under are mostly private. The six evaluator terms follow GovAI’s proposal that those contracts be published. It’s a beta, scored separately, and never added to the staff score.
“How can you rate contracts you haven’t seen?”
Mostly I can’t, and the site says so. 18 of 35 are “Not public”. The rest come from published policies, public statements and reporting, each linked. Staff who joined or left at different times may have signed different versions, so a rating is about the latest public terms, not every contract still in force.
“Why are Anthropic and OpenAI so close?”
Equal weights. They get there differently: Anthropic has six grades, all B or D. OpenAI has five, and two are As. Click a lab’s score to see them.
“US law already bans some of this.”
Some of it, in some places, for some disclosures. A clause can be unenforceable and still stop people talking, because most people won’t test it. Saying it plainly in the contract is the point of C4.
“What’s the site made of, and where does it run?”
One HTML page and three data files, ratings.json and criteria.json for staff contracts and evaluators.json for evaluator contracts, plus update feeds made from the ratings file. There’s no server, database or login. It’s hosted on Cloudflare Pages and the domain is registered with Cloudflare. All the code and every rating are public on GitHub under the MIT licence, with a history of every change. I built it with help from Claude.
“Who pays for it?”
Nobody. I pay for the domain, about $10 a year, and the hosting is free. There are no ads.
“What do you track?”
I want to see how people use the page so I can make it better. Cloudflare Web Analytics and PostHog (hosted in the EU) count visits, and PostHog also records sessions on this page: clicks, scrolling and mouse movement. Neither sets cookies, PostHog doesn’t keep IP addresses, and nothing you type is recorded. Recordings are deleted after 30 days. I don’t sell or share any of it.
“Do you keep my email?”
Only if you sign up for updates. Your address is stored with Buttondown, the newsletter service, and only used to email you when a rating changes. You confirm by email first, every email has an unsubscribe link, and unsubscribing deletes your address. I don’t sell or share it. Use any address you like: you don’t need to use a work one.
“Can I send you my contract, or a tip?”
Please don’t. The point is to get labs to publish their terms, not to collect leaks. I only use public evidence, so I’d delete anything confidential without using it, and sending it to a website probably isn’t legally protected. Worried about your own terms? The Signals Network, Psst.org and Protect can help.
“How do I get in touch, or fix a mistake?”
Email [email protected]. Please don’t send contracts or anything confidential. If a rating is wrong or missing, you can also open an issue on GitHub with a link to a public source, so everyone can see the change and why. Labs can do the same, or publish their terms and opt in.Labs can do the same, or publish their evaluation contracts and opt in.