Against a General Factor of Doom

If you ask people a bunch of specific doomy questions, and their answers are suspiciously correlated, they might be expressing their p(Doom) for each question instead of answering the questions individually. Using a general factor of doom is unlikely to be an accurate depiction of reality. The future is likely to be surprisingly doomy in some ways and surprisingly tractable in others.


Notes on an Experiment with Markets

AI Impacts decided to try using Manifold Markets to help us plan social events in the evenings of our work retreat. Here are some notes from that experiment.


Counterarguments to the basic AI x-risk case

Sixteen weaknesses in the classic argument for AI risk.

1 comment

What do ML researchers think about AI in 2022?

Katja Grace Aug 4 2022

First findings from the new 2022 Expert Survey on Progress in AI.


Why work at AI Impacts?

Katja Grace Mar 2022

My grounds for spending my time on this: a hand-wavy account.


Observed patterns around major technological advancements

by Rick Korzekwa, 2 February, 2022 Summary In this post I outline apparent regularities in how major new technological capabilities and methods come about. I have not rigorously checked to see how broadly they hold,


Beyond fire alarms: freeing the groupstruck

Katja Grace Sept 2021

Fire alarms are the wrong way to think about the public AGI conversation.


Vignettes workshop

Daniel Kokotajlo June 2021

Write down how AI will go down!


April files

Katja Grace April 2021

Internal drafts for feedback


Coherence arguments imply a force for goal-directed behavior

Katja Grace Mar 2021

Behavior that is permitted by the ‘coherence arguments’ may still be discouraged by them.

1 comment

Misalignment and misuse: whose values are manifest?

Katja Grace Nov 2020

Are misalignment and misuse helpful catastrophe categories?

1 comment

Automated intelligence is not AI

Katja Grace Nov 2020

Sometimes we think of ‘artificial intelligence’ as whatever technology ultimately automates human cognitive labor…

1 comment

Relevant pre-AGI possibilities

Daniel Kokotajlo June 2020
Brainstorm of ways the world could be relevantly different by the time advanced AGI arrives

Map of the first transatlantic telegraph

Description vs simulated prediction

Rick Korzekwa April 2020
What are we trying to do when we look at history to inform forecasting?


Discontinuous progress in history: an update

Katja Grace April 2020
We’ve been looking for historic cases of discontinuously fast technological progress, to help with reasoning about the likelihood and consequences of abrupt progress in AI capabilities. We recently finished expanding this investigation to 37 technological trends. This blog post is a quick update on our findings. See the main page on the research and its outgoing links for more details.

1 comment

Takeaways from safety by default interviews

Asya Bergal

Last year, several researchers at AI Impacts (primarily Robert Long and I) interviewed prominent researchers inside and outside of the AI safety field who are relatively optimistic about advanced AI being developed safely. These interviews were originally intended to focus narrowly on reasons for optimism, but we ended up covering a variety of topics, including AGI timelines, the likelihood of current techniques leading to AGI, and what the right things to do in AI safety are right now. (…)


Atari early

By Katja Grace Deepmind announced that their Agent57 beats the ‘human baseline’ at all 57 Atari games usually used as a benchmark. I think this is probably enough to resolve one of the predictions we


Three kinds of competitiveness

By Daniel Kokotajlo In this post, I distinguish between three different kinds of competitiveness — Performance, Cost, and Date — and explain why I think these distinctions are worth the brainspace they occupy. For example,


AGI in a vulnerable world

By Asya Bergal, Mar 25 2020 I’ve been thinking about a class of AI-takeoff scenarios where a very large number of people can build dangerous, unsafe AGI before anyone can build safe AGI. This seems


Cortés, Pizarro, and Afonso as precedents for takeover

Daniel Kokotajlo, Feb 29, 2020 Epistemic status: I am not a historian, nor have I investigated these case studies in detail. I admit I am still uncertain about how the conquistadors were able to colonize


Robin Hanson on the futurist focus on AI

Robert Long and I recently talked to Robin Hanson—GMU economist, prolific blogger, and longtime thinker on the future of AI—about the amount of futurist effort going into thinking about AI risk. It was noteworthy to


Rohin Shah on reasons for AI optimism

I along with several AI Impacts researchers recently talked to Rohin Shah about why he is relatively optimistic about AI systems being developed safely. Rohin Shah is a 5th year PhD student at the Center


The unexpected difficulty of comparing AlphaStar to humans

By Rick Korzekwa, 17 Sept 2019 Artificial intelligence defeated a pair of professional Starcraft II players for the first time in December 2018. Although this was generally regarded as an impressive achievement, it quickly became


Paul Christiano on the safety of future AI systems

As part of our AI optimism project, we talked to Paul Christiano about why he is relatively hopeful about the arrival of advanced AI going well. Paul Christiano works on AI alignment on the safety


Soft takeoff can still lead to decisive strategic advantage

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. [Epistemic status: Argument by analogy to historical cases. Best case scenario it’s just one argument among many. Edit: Also, thanks to feedback from


Ernie Davis on the landscape of AI risks

Earlier this month, I spoke with Ernie Davis about why he is skeptical that risks from superintelligent AI are substantial and tractable enough to merit dedicated work. This was part of a larger project that


Primates vs birds: Is one brain architecture better than the other?

The boring answer to that question is, “Yes, birds.” But that’s only because birds can pack more neurons into a walnut-sized brain than a monkey with a brain four times that size. So let’s forget

1 comment

Evidence on good forecasting practices from the Good Judgment Project: an accompanying blog post

Figure 0: The “four main determinants of forecasting accuracy.” Experience and data from the Good Judgment Project (GJP) provide important evidence about how to make accurate predictions. For a concise summary of the evidence and


Reinterpreting “AI and Compute”

This is a guest post by Ben Garfinkel. We revised it slightly, at his request, on February 9, 2019. A recent OpenAI blog post, “AI and Compute,” showed that the amount of computing power consumed

1 comment

On the (in)applicability of corporate rights cases to digital minds

This is a guest cross-post by Cullen O’Keefe. High-Level Takeaway The extension of rights to corporations likely does not provide useful analogy to potential extension of rights to digital minds. Introduction Examining how law can


Interpreting AI compute trends

This is a guest post by Ryan Carey. Over the last few years, we know that AI experiments have used much more computation than previously. But just last month, an investigation by OpenAI made some


Occasional update July 5 2018

Before I get to substantive points, there has been some confusion over the distinction between blog posts and pages on AI Impacts. To make it clearer, this blog post shall proceed in a way that


The tyranny of the god scenario

Michael Wulfsohn is an AI Impacts researcher/contributor. I was convinced. An intelligence explosion would result in the sudden arrival of a superintelligent machine. Its abilities would far exceed those of humans in ways we can’t


Brain wiring: The long and short of it

When I took on the task of counting up all the brain’s fibers and figuratively laying them end-to-end, I had a sense that it would be relatively easy–do a bit of strategic Googling, check out


Will AI see sudden progress?

Will advanced AI let some small group of people or AI systems take over the world? AI X-risk folks and others have accrued lots of arguments about this over the years, but I think this


GoCAS talk on AI Impacts findings

Here is a video summary of some highlights from AI Impacts research over the past years, from the GoCAS Existential Risk workshop in Göteborg in September. Thanks to the folks there for recording it.


Price performance Moore’s Law seems slow

When people make predictions about AI, they often assume that computing hardware will carry on getting cheaper for the foreseeable future, at about the same rate that it usually does. Since this is such a

1 comment

When do ML Researchers Think Specific Tasks will be Automated?

We asked the ML researchers in our survey when they thought 32 narrow, relatively well defined tasks would be feasible for AI. Eighteen of them were included in our paper earlier, but the other fourteen results


What do ML researchers think you are wrong about?

So, maybe you are concerned about AI risk. And maybe you are concerned that many people making AI are not concerned enough about it. Or not concerned about the right things. But if so, do

1 comment

AI hopes and fears in numbers

People often wonder what AI researchers think about AI risk. A good collection of quotes can tell us that worry about AI is no longer a fringe view: many big names are concerned. But without a great sense of how many


Some survey results!

Katja Grace We put the main results of our survey of machine learning researchers on AI timelines online recently—see here for the paper. Apologies for the delay—we are trying to avoid spoiling the newsworthiness of the results for potential academic


Changes in funding in the AI safety field

Guest post by Seb Farquhar, originally posted to the Center for Effective Altruism blog. The field of AI Safety has been growing quickly over the last three years, since the publication of “Superintelligence”. One of


Joscha Bach on remaining steps to human-level AI

Last year John and I had an interesting discussion with Joscha Bach about what ingredients of human-level artificial intelligence we seem to be missing, and how to improve AI forecasts more generally. Thanks to Connor Flexman’s summarizing efforts, you can now learn about


Tom Griffiths on Cognitive Science and AI

This is a guest post by Finan Adamson Prof. Tom Griffiths is the director of the Computational Cognitive Science Lab and the Institute of Cognitive and Brain Sciences at UC Berkeley. He studies human cognition


What if you turned the world’s hardware into AI minds?

In a classic ‘AI takes over the world’ scenario, one of the first things an emerging superintelligence wants to do is steal most of the world’s computing hardware and repurpose it to running the AI’s own software. This step takes one from ‘super-proficient hacker’


Friendly AI as a global public good

A public good, in the economic sense, can be (roughly) characterized as a desirable good that is likely to be undersupplied, or not supplied at all, by private companies. It generally falls to the government


Error in Armstrong and Sotala 2012

Can AI researchers say anything useful about when strong AI will arrive? Back in 2012, Stuart Armstrong and Kaj Sotala weighed in on this question in a paper called ‘How We’re Predicting AI—or Failing To‘. They looked


Metasurvey: predict the predictors

As I mentioned earlier, we’ve been making a survey for AI researchers. The survey asks when AI will be able to do things like build a lego kit according to the instructions, be a surgeon, or radically accelerate global technological development. It also asks


Concrete AI tasks bleg

We’re making a survey. I hope to write soon about our general methods and plans, so anyone kind enough to criticize them has the chance. Before that though, we have a different request: we want a list of concrete tasks that AI can’t do yet,


Mysteries of global hardware

This blog post summarizes recent research on our Global Computing Capacity page. See that page for full citations and detailed reasoning. We recently investigated this intriguing puzzle: FLOPS (then) apparently performed by all of the world’s computing hardware: 3 x 1022 – 3


Recently at AI Impacts

We’ve been working on a few longer term projects lately, so here’s an update in the absence of regular page additions. New researchers Stephanie Zolayvar and John Salvatier have recently joined us, to try out research here. Stephanie


AI timelines and strategies

AI Impacts sometimes invites guest posts from fellow thinkers on the future of AI. These are not intended to relate closely to our current research, nor to necessarily reflect our views. However we think they are worthy contributions to the discussion of AI forecasting and strategy. This


Introducing research bounties

By Katja Grace, 7 August 2015 Sometimes we like to experiment with novel research methods and formats. Today we are introducing ‘AI Impacts Research Bounties‘, in which you get money if you send us inputs to some of our research. To start, we


Time flies when robots rule the earth

This week Robin Hanson is finishing off his much anticipated book, The Age of Em: Work, Love and Life When Robots Rule the Earth. He recently told me that it would be helpful to include rough numbers for the brain’s memory and computing capacity in

1 comment

Event: Exercises in Economic Futurism

On Thursday July 30th Robin Hanson is visiting again, and this time we will be holding an informal workshop on how to usefully answer questions about the future, with an emphasis on economic approaches. We will pick roughly three concrete

1 comment

Steve Potter on neuroscience and AI

Prof. Steve Potter works at the Laboratory of Neuroengineering in Atlanta, Georgia. I wrote to him after coming across his old article, ‘What can AI get from Neuroscience?’ I wanted to know how neuroscience might contribute to AI in the future: for instance will


New funding for AI Impacts

AI Impacts has received two grants! We are grateful to the Future of Humanity Institute (FHI) for $8,700 to support work on the project until September 2015, and the Future of Life Institute (FLI) for $49,310 for another year of


Update on all the AI predictions

For the last little while, we’ve been looking into a dataset of individual AI predictions, collected by MIRI a couple of years ago. We also previously gathered all the surveys about AI predictions that we could find. Together, these are all the public predictions


Why do AGI researchers expect AI so soon?

People have been predicting when human-level AI will appear for many decades. A few years ago, MIRI made a big, organized collection of such predictions, along with helpful metadata. We are grateful, and just put up a page about this dataset, including some analysis. Some of you saw


Supporting AI Impacts

We now have a donations page. If you like what we are doing as much as anything else you can think of to spend marginal dollars on, I encourage you to support this project! Money will go to more of the kind of thing you

1 comment
By Martin Grandjean

A new approach to predicting brain-computer parity

How large does a computer need to be before it is ‘as powerful’ as the human brain? This is a difficult question, which people have answered before, with much uncertainty. We have a new answer! (Longer description here;


Preliminary prices for human-level hardware

Computer hardware has been getting cheap now for about seventy five years. Relatedly, large computing projects can afford to be increasingly large. If you think the human brain is something like a really impressive computer, then a


What’s up with nuclear weapons?

When nuclear weapons were first built, the explosive power you could extract from a tonne of explosive skyrocketed. But why? Here’s a guess. Until nuclear weapons, explosives were based on chemical reactions. Whereas nuclear weapons are based on nuclear


Multipolar research questions

The Multipolar AI workshop we ran a fortnight ago went well, and we just put up a list of research projects from it. I hope this is helpful inspiration to those of you thinking about applying to the new FLI grants in the


How AI timelines are estimated

A natural approach to informing oneself about when human-level AI will arrive is to check what experts who have already investigated the question say about it. So we made this list of analyses that we could find. It’s a short list, though the bar for ‘analysis’ was


At-least-human-level-at-human-cost AI

Often, when people are asked ‘when will human-level AI arrive?’ they suggest that it is a meaningless or misleading term. I think they have a point. Or several, though probably not as many as they think

1 comment

Penicillin and syphilis

Penicillin was a hugely important discovery. But was it a discontinuity in the normal progression of research, or just an excellent discovery which followed a slightly less excellent discovery, and so on? There are several


The slow traversal of ‘human-level’

Once you have normal-human-level AI, how long does it take to get Einstein-level AI? We have seen that a common argument for ‘not long at all’ based on brain size does not work in a straightforward way, though a


Making or breaking a thinking machine

Here is a superficially plausible argument: the brains of the slowest humans are almost identical to those of the smartest humans. And thus—in the great space of possible intelligence—the ‘human-level’ band must be very narrow. Since all humans are basically identical in


Are AI surveys seeing the inside view?

An interesting thing about the survey data on timelines to human-level AI is the apparent incongruity between answers to ‘when will human-level AI arrive?’ and answers to ‘how much of the way to human-level AI have we come recently?‘ In particular, human-level AI

1 comment

Event: Multipolar AI workshop with Robin Hanson

On Monday 26 January we will be holding a discussion on promising research projects relating to ‘multipolar‘ AI scenarios. That is, future scenarios where society persists in containing a large number of similarly influential agents, rather than a single winner who takes all. The


Michie and overoptimism

We recently wrote about Donald Michie’s survey on timelines to human-level AI. Michie’s survey is especially interesting because it was taken in 1972, which is three decades earlier than any other surveys we know of that ask about human-level AI.

1 comment

Were nuclear weapons cost-effective explosives?

Nuclear weapons were radically more powerful per pound than any previous bomb. Their appearance was a massive discontinuity in the long-run path of explosive progress, that we have lately discussed. But why do we measure energy


A summary of AI surveys

If you want to know when human-level AI will be developed, a natural approach is to ask someone who works on developing AI. You might however be put off by such predictions being regularly criticized as inaccurate and biased. While they do seem


AI and the Big Nuclear Discontinuity

As we’ve discussed before, the advent of nuclear weapons was a striking technological discontinuity in the effectiveness of explosives. In 1940, no one had ever made an explosive twice as effective as TNT. By 1945 the best

First nuclear reaction

The Biggest Technological Leaps

Over thousands of years, humans became better at producing explosions. A weight of explosive that would have blown up a tree stump in the year 800 could have blown up more than three tree stumps in

1 comment

The AI Impacts Blog

Jan 09 2015 Welcome to the AI Impacts blog.  AI Impacts is premised on two ideas (at least!): The details of the arrival of human-level artificial intelligence matter Seven years to prepare is very different from seventy years to

1 comment