systemsnull

A Peek Into The Research Grants Propping AI Safety

Human ThingsSD 2026.27114 min read

In March 2017, Open Philanthropy gave OpenAI $30 million, ten million a year for three years, and in exchange its executive director, Holden Karnofsky, took a seat on OpenAI’s board. Grant pages are usually the most boring documents on the internet, but this one has a disclosure section, and the disclosure section reads like the cast list of a prestige drama:

“OpenAI researchers Dario Amodei and Paul Christiano are both technical advisors to Open Philanthropy and live in the same house as Holden. In addition, Holden is engaged to Dario’s sister Daniela.”

Karnofsky and Daniela Amodei married that August. Four years later Dario and Daniela left OpenAI to found Anthropic, whose first funding round was led by Jaan Tallinn with participation from Dustin Moskovitz, the billionaire whose money is most of Open Philanthropy’s money. In January 2025 Karnofsky himself joined Anthropic, working on its responsible scaling policy; Fortune noted the company “had not announced” the hire.

None of this is secret, and I’m not describing a scandal. Open Philanthropy disclosed its conflicts in writing, in public, in the same document as the grant, which is more than most funders ever do.

I’m starting there because it’s the cleanest example of the thing I actually want to write about: the field that is supposed to let us check what AI systems are doing was built by a very small number of people, trained through a pipeline paid for by a very small number of donors, and a remarkable share of both the people and the money trace back to the companies the field is meant to check. Mechanistic interpretability is less a scientific community than a family tree, and it matters who sits at the root, because interpretability is on its way to becoming the evidence the rest of us are asked to trust.

When I traced that tree, the company at its root most often wasn’t the one I expected to be worried about; it was Anthropic, which is also the company with the best reputation in the field.

A quick definition, because the stakes depend on it. Mechanistic interpretability is the attempt to reverse-engineer a neural network: not just what it outputs, but which internal structures produce the output. Its promise is best shown by a result Anthropic published in March 2025, in which researchers traced how Claude 3.5 Haiku adds 36 and 59. The model runs two pathways in parallel, one estimating the rough size of the answer and one computing the last digit exactly, and combines them at the end. Asked how it did the sum, it described carrying the one. Its account of its own reasoning was fluent, plausible, and not what it had done; we only know that because someone with the weights and the compute went and looked.

That’s the whole case for the field in one arithmetic problem. When a model’s explanation of itself can’t be trusted, the people who can look inside become the only people who can tell you what’s true.

So who are they?

The field’s modern shape starts with Chris Olah. He dropped out of the University of Toronto after about a year and took a $100,000 Thiel Fellowship in 2012, went to Google Brain, led OpenAI’s interpretability team from 2018 to 2020, and left with the group that founded Anthropic, where Fortune describes him as its interpretability research lead. The “circuits” program he started, and the self-published journal it lives in, transformer-circuits.pub, set the vocabulary most of the field still uses: features, circuits, superposition, sparse dictionaries. The journal has gone from a handful of items a year in 2021 and 2022 to nineteen in 2025, most of them co-authored by the same few dozen people.

The second node is Neel Nanda, and he’s where the tree starts to branch. By his own account he “worked at Anthropic as a language model interpretability researcher under Chris Olah,” then did independent research, funded in part by a $70,000 Open Philanthropy grant in January 2023, and now “runs the Google DeepMind mechanistic interpretability team.” In between, he wrote TransformerLens, the library nearly every newcomer uses to poke at a model’s internals, and a sequence of open problems that functions as the field’s unofficial syllabus. He is also its most prolific mentor: by his count he has supervised around sixty scholars, about twenty of whom now work at frontier labs.

So the person who trained a large fraction of the field’s newcomers was himself trained by the person who founded it, and the tool they learn on was written by a researcher whose independent year was paid for by the philanthropy whose founder invested in Anthropic.

Most of those scholars came through MATS, the largest training program for AI safety researchers, which started in late 2021 and became an independent nonprofit in 2024. MATS says more than 80% of its alumni go on to work in AI safety, and about one in ten co-found an organisation; Apollo Research and Timaeus are both on that list. Follow the money into MATS and you find Open Philanthropy again, over and over: $2.64 million in late 2023, $2.38 million in May 2024, $660,000 in December 2024, and a long tail of smaller grants routed through fiscal sponsors. The next largest funder is the Survival and Flourishing Fund, whose 2025 recommendations to MATS, Timaeus, and FAR AI all came from Jaan Tallinn, the man who led Anthropic’s Series A.

Then the graduates found things, and the things are funded by the same names. Apollo Research got $1.5 million from Open Philanthropy to start up in 2023 and $2.2 million more in 2024. EleutherAI got $2.6 million over four years to support Nora Belrose’s interpretability work. Timaeus, a small group working on the mathematics of how networks learn, was funded by Open Philanthropy and SFF, and this July its researchers became the core of a new organisation, Resolution, led by Geoffrey Irving (formerly of the UK AI Security Institute), with $160 million from Coefficient Giving, the name Open Philanthropy took in 2025. That’s one grant, from one donor, larger than all the MATS grants above combined several times over.

Even the academics branch off the same trunk. David Bau’s lab at Northeastern, which wrote the ROME paper on locating and editing facts in language models, has received more than $1.6 million from Open Philanthropy across two grants, plus $116,072 to fund a postdoc in the lab for Samuel Marks. Marks is now at Anthropic, and he’s the first author of its 2025 hidden-objectives paper, the one where Anthropic trained a model with a secret goal and had blind teams try to find it with interpretability tools. Harvard’s Martin Wattenberg and Fernanda Viégas got $1 million from Open Philanthropy in 2024. When twenty-nine researchers from across the field wrote the consensus paper Open Problems in Mechanistic Interpretability in January 2025, the paper says it “represents a synthesis of their views by Apollo Research on behalf of Schmidt Sciences”; its authors came from Apollo, Anthropic, DeepMind, MIT, Northeastern, EleutherAI, Goodfire, Timaeus, MATS, METR, and a handful of universities.

It reads like an independent field reaching agreement. Run the names back through the grant pages and it looks more like a large family comparing notes at a reunion.

The field is also accelerating fast, which is why any of this is urgent rather than merely interesting. A crude count (my own query of the arXiv API for abstracts containing the exact phrase “mechanistic interpretability”) gives 11 papers in 2022, 40 in 2023, 105 in 2024, 285 in 2025, and 387 so far in 2026. The phrase count understates the field, but the curve is the point: the literature is roughly doubling every year, and the people deciding what questions it asks are the ones above.

They also change their minds together, and quickly. In March 2025 Nanda’s team at DeepMind announced that it was “deprioritising fundamental SAE research for the moment,” concluding that “the interpretability community is somewhat over invested in SAEs,” and in December it published a “pragmatic vision” moving “from ambitious reverse-engineering to a focus on pragmatically making as much progress as we can on the critical path.”

I think they may well be right. But sparse autoencoders were the technique Anthropic had built its flagship results on, the technique Nanda had taught to a generation of MATS scholars, and within a year the team led by the field’s biggest mentor declared it overinvested and redirected. When a field is this concentrated, a strategy memo from one lab is a curriculum change for everyone else. (The same year, an EleutherAI paper showed that two sparse autoencoders trained on the same Llama 3 model, identical except for their random seed, shared only about 30% of their features. Which features a model “has” depends partly on who trained the microscope, and how.)

Now the part where the labs themselves come in, and where I have to be even-handed, including with the company whose model I used to write the first draft of this.

OpenAI is the lab Glyph got wrong. Its map called OpenAI “structurally absent” from interpretability, which was a striking claim and false; OpenAI has published steadily, including a 2024 paper training sparse autoencoders with 16 million features on GPT-4 and 2025 work on models built to have interpretable circuits. What’s true is that its safety organisation is the least stable of the three. When Ilya Sutskever and Jan Leike left in May 2024, their Superalignment team was disbanded; Fortune reported that the 20% of compute it had been promised never materialised, with GPU requests “turned down,” according to a half-dozen sources. Leike wrote that safety culture had “taken a backseat to shiny products,” and went to Anthropic. OpenAI’s new Safety Fellowship, launched in 2026, doesn’t list interpretability among its priority areas.

DeepMind is the most open of the three with artefacts: Gemma Scope, and later Gemma Scope 2, released hundreds of trained sparse autoencoders on Google’s open models, and it’s the lab most willing to argue with its own agenda in public. It’s also a subsidiary of Google, which owns roughly 14% of Anthropic, a stake contractually capped at 15%, and has committed up to $40 billion more. Amazon has put in around $8 billion and committed more. The two companies that own the cloud Anthropic runs on also own a large piece of Anthropic.

And then Anthropic, which does the most interpretability, publishes the most, releases real tools, and would be the easiest to write this essay as a hero. It’s also where every line in the tree converges. Its founders came out of the Open Philanthropy household; its first round was led by the donors who fund most of independent AI safety; the founder of the field runs its team; Karnofsky works on its safety policy. Its Fellows program pays $3,850 a week plus roughly $15,000 a month in compute, and more than 40% of its first cohort joined Anthropic afterwards. Goodfire, the best-funded interpretability startup (a co-founder of DeepMind’s interpretability team and a former OpenAI circuits researcher among its staff), was Anthropic’s first-ever direct investment in another company.

And the money is about to get stranger: Anthropic’s co-founders have pledged to give away 80% of their wealth, and Transformer’s reporting this March described how that wave of Anthropic-equity philanthropy risks making its recipients look like “a lobbying arm of Anthropic equity holders.” You can see it already. Transluce, the nonprofit Jacob Steinhardt and Sarah Schwettmann founded in 2024 specifically to build independent tools for understanding AI systems, publishes its funding with admirable honesty: in fiscal year 2025, 32% of its revenue came from the personal holdings of Anthropic employees, and 6% from OpenAI employees, with nothing from the companies themselves.

That’s the most transparent organisation in the field, doing exactly the right thing, and a third of its independence is still Anthropic money, one step removed.

Anthropic’s CEO has written that he wants interpretability to “reliably detect most model problems” by 2027, and asked governments for “light-touch” rules while urging other labs to invest more. I’m glad someone set the goal. But look at its shape: the lab that leads the field in a particular kind of evidence is proposing that the kind of evidence it leads in become the standard, while the people best qualified to check that evidence were trained in its programs, hold its equity, or are funded by people who do.

I don’t think anyone here is acting in bad faith, and I want to be precise about why it still matters. Casper and colleagues argued in 2024 that black-box access is insufficient for rigorous AI audits; you need the weights and the activations. The same year, another group documented researchers having their accounts suspended by OpenAI, Anthropic, and others for public-interest testing, and asked for a legal “safe harbor.” This month, FAR AI’s Adam Gleave told TechCrunch his organisation has turned down contracts with several frontier developers “that wanted too much control over the evaluation process.” Independence, in other words, is scarce and expensive to maintain, and the pool it’s drawn from is mostly one family.

The public counterweights are real but small next to what’s above. The NSF’s NDIF project, led by Bau, has about $9 million from 2024 to 2028 to let academics probe large open models remotely. The UK AISI’s Alignment Project gave £27 million to sixty projects in February, and its funders include OpenAI (£5.6 million), Anthropic, AWS, and Microsoft. Compare either to a single $160 million grant from a single donor, or to Goodfire’s $1.25 billion valuation.

Open Philanthropy grant figures are from archived copies of its grant pages; the successor site, coefficientgiving.org, was unreachable at time of writing.

References