Walsh 2017 Survey

Toby Walsh surveyed hundreds of experts and non-experts in 2016 and found their median estimates for ‘when a computer might be able to carry out most human professions at least as well as a typical human’ were as follows:

Probability of HLMIGroup of survey respondents
AI expertsRobotics expertsNon-experts
10%203520332026
50%206120652039
90%210921182060

Details

Toby Walsh, professor of AI at the University of New South Wales and Technical University of Berlin, conducted a poll of AI experts, robotics experts, and non-experts from late January to early February 2017. The survey focused on the potential automation of various occupations and the arrival of high-level machine intelligence (HLMI). 

Survey respondents

There were 849 total survey respondents composing three separate groups: AI experts, robotics experts, and non-experts.

The AI experts consisted of 200 authors from two AI conferences: the 2015 meeting of the Association for the Advancement of AI (AAAI) and the 2011 International Joint Conference on AI (IJCAI).

The robotics experts consisted of 101 individuals who were either Fellows of the Institute for Electrical and Electronics Engineers (IEEE) Robotics & Automation Society or authors from the 2016 meeting of the IEEE Conference on Robotics & Automation (ICRA).

The non-experts consisted of 548 readers of an article about AI on the website The Conversation. While it seems data on their possible expertise in AI or robotics was not collected, Walsh writes that “it is reasonable to suppose that most are not experts in AI & robotics, and that they are unlikely to be publishing in the top venues in AI and robotics like IJCAI, AAAI or ICRA” (p. 635). Some additional demographic data was collected and reported (for this survey group only):

  • Geographic distribution: 36% Australia, 29% United States, 7% United Kingdom, 4% Canada, and 24% rest of the world
  • Education: 85% have an undergraduate degree or higher
  • Age: >33% are 34 or under, 59% are under 44, and 11% are 65 or older
  • Employment status: >66% are employed and 25% are in or about to enter higher education
  • Income: 40% reported an annual income of >$100,000

Classifying occupations at risk of automation

The first seven survey questions (out of eight total) asked respondents to classify occupations as either at risk of automation in the next two decades or not (binary response). For each occupation, respondents were provided with information about the work involved and skills required. There were 70 total occupations, which came from a previous study that had used a machine learning (ML) classifier to rank them in terms of their risk for automation. These rankings were then used in the present survey: Each question had respondents classify 10 occupations, starting with the five most likely and five least likely at risk of automation according to the ML classifier. This continued through subsequent questions until respondents classified all 70 occupations.

Arrival of high-level machine intelligence (HLMI)

The last survey question asked by what year there would be a 10%, 50%, and 90% chance of HLMI, which was defined as “when a computer might be able to carry out most human professions at least as well as a typical human” (p. 634). For each probability respondents chose from among eight options: 2025, 2030, 2040, 2050, 2075, 2100, After 2100, and Never. Median responses were calculated by interpolating the cumulative distribution function between the two nearest dates.

Results

Probability of when (in years) HLMI will arrive

Table 1 below summarized the median responses and is reproduced here for convenience.

Table 1

Probability of HLMIGroup of survey respondents
AI expertsRobotics expertsNon-experts
10%203520332026
50%206120652039
90%210921182060

Figures 1-3 below show the cumulative distribution functions (CDFs) for 10%, 50%, and 90% probability of HLMI (respectively) at different years.

Figure 1

Figure 2

Figure 3

Occupations at risk of automation

Table 2 below contains descriptive statistics about the number of occupations (out of 70 total) classified as being at risk of automation in the next two decades. Confidence intervals (last column) are at the 95% level. It is unclear why the sample size for Non-experts is listed as 473 when earlier in the article the number reported is 548.

Table 2

The difference in means between the Robotics (29.0) and AI experts (31.1) was not statistically significant (two-sided t-test, p = 0.096), while the differences in means between both expert groups and the non-expert group (36.5) separately were significant (two-sided t-test, both p’s < 0.0001).

Table 3 below lists some of the largest differences in the proportion of experts (AI and robotics combined) compared to non-experts who classified occupations as at risk for automation.

Table 3

OccupationProportion of respondents predicting risk for automation
ExpertsNon-experts
Economist12%39%
Electrical engineer6%33%
Technical writer31%54%
Civil engineer6%30%

Figure 4 below shows that respondents who predicted that HLMI would arrive earlier also classified more occupations as being at risk of automation (and vice versa).

Figure 4