Changes in funding in the AI safety field

Guest post by , originally posted to the Center for Effective Altruism blog.

The field of AI Safety has been growing quickly over the last three years, since the publication of “Superintelligence”. One of the things that shapes what the community invests in is an impression of what the composition of the field currently is, and how it has changed. Here, I give an overview of the composition of the field as measured by its funding.

Measures other than funding also matter, and may matter more, like types of outputs, distribution of employed/active people, or impact-adjusted distributions of either. Funding, however, is a little more objective and easier to assess. It gives us some sense of how the AI Safety community is prioritising, and where it might have blind spots. For a fuller discussion of the shortcomings of this type of analysis, and of this data, see section four.

Throughout, I am including the budgets of organisations who are explicitly working to reduce existential risk from machine superintelligence. It does not include work outside the AI Safety community, on areas like verification and control, that might prove relevant. This kind of work, which happens in mainstream computer science research, is much harder to assess for relevance and to get budget data for. I am trying as much as possible to count money spent at the time of the work, rather than the time at which a grant is announced or money is set aside.

Thanks to Niel Bowerman, Ryan Carey, Andrew Critch, Daniel Dewey, Viktoriya Krakovna, Peter McIntyre, Michael Page for their comments or help on content or gathering data in preparing this document (though nothing here should be taken as a statement of their views and any errors are mine).

The post is organised as follows:

  1. Narrative of growth in AI Safety funding
  2. Distribution of spending
  3. Soft conclusions from overview
  4. Caveats and assumptions

Narrative of growth in AI Safety funding

The AI Safety community grew significantly in the last three years. In 2014, AI Safety work was almost entirely done at the Future of Humanity Institute (FHI) and the Machine Intelligence Research Institute (MIRI) who were between them spending $1.75m. In 2016, more than 50 organisations have explicit AI Safety related programs, spending perhaps $6.6m. Note the caveats to all numbers in this document described in section 4.


In 2015, AI Safety spending roughly doubled to $3.3m. Most of this came from growth at MIRI and the beginnings of involvement by industry researchers.

In 2016, grants from the Future of Life Institute (FLI) triggered growth in smaller-scale technical AI safety work.1 Industry invested more over 2016, specially at Google DeepMind and potentially at OpenAI.2 Because of their high salary costs, the monetary growth in spending at these firms may overstate actual growth of the field. For example, several key researchers moved from non-profits/academic orgs (MIRI, FLI, FHI) to Google DeepMind and OpenAI. This increased spending significantly, but may have had a smaller effect on output.3 AI Strategy budgets grew more slowly, at about 20%.

In 2017, multiple center grants are emerging (such as the Center for Human-Compatible AI (CHCAI) and Center for the Future of Intelligence (CFI)), but if their hiring is slow it will restrain overall spending. FLI grantee projects will be coming to a close over the year, which may mean that technical hires trained through those projects become available to join larger centers. The next round of FLI grants may be out in time to bridge existing grant holders onto new projects. Industry teams may keep growing, but there are no existing public commitments to do so. If technical research consolidates into a handful of major teams, it might make it easier to keep open dialogue between research groups, but might decrease individual incentives to because researchers have enough collaboration opportunities locally.

Although little can be said about 2018 at this point, the current round of academic grants which support FLI grantees as well as FHI end in 2018, potentially creating a funding cliff. (Though FLI has just announced a second funding round, and MIT Media Lab has just announced a $27m center (whose exact plans remain unspecified).4

Estimated spending in AI Safety broken down by field of work


Distribution of spending

In 2014, the field of research was not very diverse. It was roughly evenly split into work at FHI on macrostrategy, with limited technical work, and at MIRI following a relatively focused technical research agenda which placed little emphasis on deep learning.

Since then, the field has diversified significantly.

The academic technical research field is very diverse, though most of the funding comes via FLI. MIRI remains the only non-profit doing technical research and continues to be the largest research group with 7 research fellows at the end of 2016 and a budget of $1.75m. Google DeepMind probably has the second largest technical safety research group with between 3 and 4 full-time-equivalent (FTE) researchers at the end of 2016 (most of whom joined at the end of the year), though OpenAI and GoogleBrain probably have 0.5-1.5 FTEs.5

FHI and SAIRC remains the only large-scale AI strategy center. The Global Catastrophic Risk Institute is the main long-standing strategy center working on AI, but is much smaller. Some much smaller groups (FLI Grantees and the Global Politics of AI team at Yale) are starting to form, but are mostly low-/no- salary for the time being.

A range of functions are now being filled which did not exist in the AI Safety community before. These include outreach, ethics research, and rationality training. Although explicitly outreach focused projects remain small, organisations like FHI and MIRI do significant outreach work (arguably, Nick Bostrom’s Superintelligence falls into this category, for example).

2017 (forecast) – total = $10.5m


2016 – total = $6.56m


2015 – total = $3.28m


2014 – total = $1.75m


Possible implications and tentative suggestions

Technical safety research

  • The MIRI technical agenda remains the largest coherent research project, despite the emergence of several other agendas. For the sake of diversity of approach, more work needs to be done to develop PIs within the AI community to take the “Concrete Problems” research agenda and others forwards.
  • The community should go out of its way to help the emerging academic technical research centers (CHCAI and Yoshua Bengio’s forthcoming center) to recruit and retain fantastic people.

Strategy, outreach, and policy

  • Near-term policy has had a lot of people outside the AI Safety community  moving towards it, though output remains relatively low. There is even less work on medium-term implications of AI.
  • Non-technical funding has not kept up with the growth of the AI safety field as a whole. This is likely to be because the pipeline for non-technical work is less easily specified and improved than it is for technical work. This could create gaps in the future, for example in:
  • Communication channels between AI Safety research teams.
  • Communication between the AI Safety research community and the rest of the AI community.
  • Guidance for policy-makers and researchers on long-run strategy.
  • It might be helpful to establish or identify a pipeline for AI strategy/policy work, perhaps by building a PhD or Masters course at an existing institution for the purpose.
  • There is not a lot of focused AI Safety outreach work. This is largely because all organisations are stepping carefully to avoid messaging that has the potential to frame the issues unconstructively, but it might be worthwhile to step into this gap over the next year or two.

Caveats and assumptions

  • Scope: I selected projects that either self-identify or were identified to me by people in the field as focused on AI Safety. Where organisations had only a partial focus on AI Safety, I estimated the proportion of their work that was related based on the distribution of their projects. The data probably represent the community of people who explicitly think they are working on AI safety moderately-well. But it doesn’t include anyone generally working on verification/control, auditing, transparency, etc. for other reasons. It also excludes people working on near-term AI policy.
  • Forecasting: Data for 2017 are a very loose guess. In particular, they make very rough guesses for the ability of centers to scale up, which have not been validated by interviews with centers. CFAR financial estimates for 2017 are also still not publicly available, and may be more than 10% of all AI Safety spending. I have assumed, in the pie charts of distribution only, that they will spend $1m next year (they spend $920k in 2015). That estimate is probably too low, but will probably not dramatically alter the overall picture. Forecasts also do not include funding for Yoshua Bengio’s new center or the next round of FLI grants.
  • FLI grant distribution: I have assumed that all FLI grantees spent according to the following schedule: nothing in 2015, 37% in 2016, 31% in 2017, 32% later. This is based on aggregate data, but will not be right for individual grants, which might mean the distribution of funding over time between fields is slightly wrong. The values are lagged slightly in order to account for the fact that money usually takes several months to make its way through university bureaucracies. In some cases, work happens at a different time from funding being received (earlier or later).
  • Industry spending: Estimates of industry spending are very rough. I approximated the amount of time spent by individual researchers on AI Safety based on conversations with some of them and with non-industry researchers. I (very) loosely approximated the per-researcher cost to firms at $300k each, inclusive of overheads and compute.
  • Categorisation: I used the abstracts of the FLI grants, and the websites of other projects, to categorise their work roughly. Some may be miscategorised, but the major chunks of funding are likely to be right.
  • Funding is not a perfect proxy for what matters: There are many ways of describing change in the field usefully, which include how funding is distributed. Funding is a moderate proxy for the amount of effort going into different approaches, but not perfect. For example, if a researcher were to move from being lightly funded at a non-profit to employed by OpenAI their ‘cost’ in this model will have increased by roughly an order of magnitude, which might be different from their impact. The funding picture may therefore come apart from ‘effort’ especially when comparing DeepMind/OpenAI/GoogleBrain to non-profits like MIRI.
  • Re-granting: I’ve tried to avoid double-counting (e.g., SAIRC is listed as an FHI project rather than FLI despite being funded by Elon Musk and OpenPhil via FLI), but there is enough regranting going on that I might not have succeeded.
  • Inclusion: I might have missed out organisations that should arguably be in there, or have incorrect information about their spending
  • Corrections: If you have corrections or extra information I should incorporate, please email me at


Joscha Bach on remaining steps to human-level AI

Joscha Bach (from Wikimedia commons)

Joscha Bach (photos from Wikimedia commons )

Last year John and I had an interesting discussion with Joscha Bach about what ingredients of human-level artificial intelligence we seem to be missing, and how to improve AI forecasts more generally.

Thanks to Connor Flexman’s summarizing efforts, you can now learn about Joscha’s views on these questions without the effort of organizing an interview or reading a long and messy transcript.

(It’s been a while since the conversation, but I checked with Joscha that this is not an objectionably obsolete account of his views.)

Here are the notes.

Here is Connor’s shorter summary:

  • Before we can implement human-level artificial intelligence (HLAI), we need to understand both mental representations and the overall architecture of a mind
  • There are around 12-200 regularities like backpropagation that we need to understand, based on known unknowns and genome complexity
  • We are more than reinforcement learning on computronium: our primate heritage provides most interesting facets of mind and motivation
  • AI funding is now permanently colossal, which should update our predictions
  • AI practitioners learn the constraints on which elements of science fiction are plausible, but constant practice can lead to erosion of long-term perspective
  • Experience in real AI development can lead to both over- and underestimates of the difficulty of new AI projects in non-obvious ways


Tom Griffiths on Cognitive Science and AI

Tom Griffiths

Tom Griffiths

This is a guest post by Finan Adamson

Prof. Tom Griffiths is the director of the Computational Cognitive Science Lab and the Institute of Cognitive and Brain Sciences at UC Berkeley. He studies human cognition and is involved with the Center for Human Compatible Artificial Intelligence. I asked him for insight into the intersection of cognitive science and AI. He offers his thoughts on the historical interaction of the fields and what aspects of human cognition might be relevant to developing AI in the future.

The conversation notes are here (pdf).

What if you turned the world’s hardware into AI minds?

In a classic ‘AI takes over the world’ scenario, one of the first things an emerging superintelligence wants to do is steal most of the world’s computing hardware and repurpose it to running the AI’s own software. This step takes one from ‘super-proficient hacker’ levels of smart to ‘my brain is one of the main things happening on Planet Earth’ levels of smart. There is quite a bit of hardware in the world, so this step in the takeover plan is kind of terrifying.

How terrifying exactly depends on A) how much computing hardware there is in the world at the time, and B) how efficiently hardware can be turned into AI at the time. We have some tentative answers to A)—probably at least a couple of hundred exaFLOPS now, growing somewhere between not at all and very fast. However B) is harder, in the absence of any idea how to get the efficiency of hardware-to-general-AI conversions above zero. Nonetheless, I think there are a couple of interesting reference points we can look at.

The one I’ll discuss now is the efficiency of the human brain. What if we could use about as much hardware as the human brain represents (in some sense) to run AI about as smart as a human brain? This is an interesting point to look at for a few reasons. We know brains are somewhere in the range of efficiency with which hardware can produce intelligent behavior, because they are an instance of that. And looking at one datapoint in the range is better than none. Also, for some means of building artificial intelligence—most obviously, brain emulation—we might expect to get something roughly as efficient as a human brain, give or take some.

So, we can think of the human brain as representing a pile of (fairly application specific) computing hardware. And we can estimate its computing power, in terms of FLOPS. People have done this (very inaccurately— their estimates are twelve orders of magnitude apart, but running through this calculation with such an uncertain number still seems informative). According to different sources, brain seems to be worth between about 3 x 1013 FLOPS and 1025 FLOPS. The median estimate is 1018 FLOPS.

So we can ask, if you turned all of the world’s two hundred exaFLOPS or more of computing hardware into brains, how many brains would you get?

This graph shows the answers over time, for a variety of assumptions about brain FLOPS, world FLOPS, and global computing hardware growth rates. Probably the most plausible line is the lower green one (brains median, world hardware high).

Figure: Projected number of human brains equivalent

Figure: Projected number of human brains equivalent to global hardware under various assumptions. For brains, ‘small’ = 3 x 10^ 13, ‘median’ = 10^18, ‘large’ = 10^25. For ‘world hardware’, ‘high’ =2 x 10^20, ‘low’ = 1.5 x 10^21. ‘Growth’ is growth in computing hardware, the unlabeled default used in most projections is 25% per annum (our estimate above), ‘high’ = 86% per annum (which would mean shifting to the highest growth rate we know of for related hardware—that of ASIC hardware in around 2007, which does not plausibly persist).

The basic answer is, if you turned all of the world’s computing hardware into AI as efficient as human brains right now, you would get less than a hundred million extra brains, or 1% of the population of the world. Probably a whole lot less. For the median estimates of brain computing power, you would get about a hundred or a thousand extra brains worth of AI.

That means, for instance, that if we figured out how to make uploads right now, and they were roughly as efficient as the median brains estimate, and then someone acquired all of the hardware in the world for them, they would only have about as many additional minds as a project willing to spend a few hundred million dollars per year on wages, e.g. Facebook. Which would really be something. But not something overwhelmingly outscaling everything else going on in the world.

If you trust the projections of hardware growth fifty years into the future at all (which you shouldn’t, but suppose you did) the most plausible (median brain size, low growth) lines don’t even reach the world population line by then, though they would certainly make for an incredible AI research project, if that was the direction to which the additional mental effort was directed.

Remember, all of this is very sketchy and probably inaccurate and you should maybe think about it a bit more if your decisions depend on it much (or ask us nicely to). But I strongly favor sketchy projections over none.

Image: Planetary Brain, Adrian Kenyon, some rights reserved.

  1. Although grants were awarded in 2015, there is a lag between grants being awarded and work taking place. This is a significant assumption discussed in the caveats.
  2. Although note that most of the new hires at DeepMind arrived right at the end of the year.
  3. Although it is also conceivable that a researcher at DeepMind may be ten times more valuable than that same researcher elsewhere.
  4. This will depend on personal circumstance as well as giving opportunities. It would probably be a mistake to forgo time-bounded giving opportunities to cover this cliff, since other sources of funding might be found between now and then.
  5. This is based on anecdotal hiring information, and not a confirmed number from Google DeepMind.