Global computing capacity

Computing capacity worldwide was probably around 2 x 1020 – 1.5 x 1021 FLOPS, at around the end of 2015.

Support

We are not aware of recent, plausible estimates for hardware capacity.

Vipul Naik estimated global hardware capacity in February 2014, based on Hilbert & Lopez’s estimates for 1986-2007. He calculated that if all computers ran at full capacity, they would perform 10-1000 zettaFLOPS, i.e. 1022 – 1024 FLOPS.1 We think these are substantial overestimates, because producing so much computing hardware would cost more than 10% of gross world product (GWP), which is implausibly high. The most cost-efficient computing hardware we are aware of today are GPUs, which still cost about $3/GFLOPS, or $1/GFLOPSyear if we assume hardware is used for around three years. This means maintaining hardware capable of 1022 – 1024 FLOPS would cost at least $1013 – $1015  per year. Yet gross world product (GWP) is only around $8 x 1013, so this would imply hardware spending constitutes more than 13% – 1300% of GWP. Even the lower end of this range seems implausible.2

One way to estimate global hardware capacity ourselves is based on annual hardware spending. This is slightly complicated because hardware lasts for several years. So to calculate how much hardware exists in 2016, we would ideally like to know how much was bought in every preceding year, and also how much of each annual hardware purchase has already been discarded. To simplify matters, we will instead assume that hardware lasts for around three years.

It appears that very roughly $300bn-$1,500bn was spent on hardware in 2015.3 We previously estimated that the cheapest available hardware (in April 2015) was around $3/GFLOPS. So if humanity spent $300bn-$1,500bn on hardware in 2015, and it was mostly the cheapest hardware, then the hardware we bought should perform around 1020 – 5 x 1020 FLOPS. If we multiply this by three to account for the previous two years’ hardware purchases still being around, we have about  3 x 1020 – 1.5 x 1021 FLOPS.

This estimate is rough, and could be improved in several ways. Most likely, more hardware is being bought each year than the previous year. So approximating last years’ hardware purchase to this years’ will yield too much hardware. In particular, the faster global hardware is growing, the closer the total is to whatever humanity bought this year (that is, counterintuitively, if you think hardware is growing faster, you should suppose that there is less of it by this particular method of estimation). Furthermore, perhaps a lot of hardware is not the cheapest for various reasons. This too suggests there is less hardware than we estimated.

On the other hand, hardware may often last for more than three years (we don’t have a strong basis for our assumption there). And our prices are from early 2015, so hardware is likely somewhat cheaper now (in early 2016). Our guess is that overall these considerations mean our estimate should be lower, but probably by less than a factor of four in total. This suggests 7.5 x 1019 – 1.5 x 1021 FLOPS of hardware.

However Hilbert & Lopez (2012) estimated that in 2007 the world’s computing capacity was around 2 x 1020 IPS (similar to FLOPS) already, after constructing a detailed inventory of technologies.4 Their estimate does not appear to conflict with data about the global economy at the time.5 Growth is unlikely to have been negative since 2007, though Hilbert & Lopez may have overestimated. So we revise our estimate to 2 x 1020 – 1.5 x 1021 FLOPS for the end of 2015.

This still suggests that in the last nine years, the world’s hardware has grown by a factor of 1-7.5, implying a growth rate of 0%-25%. Even 25% would be quite low compared to growth rates between 1986 and 2007 according to Hilbert & Lopez (2012), which were 61% for general purpose computing and 86% for the set of ASICs they studied (which in 2007 accounted for 32 times as much computing as general purpose computers).6 However if we are to distrust estimates which imply hardware is a large fraction of GWP, then we must expect hardware growth has slowed substantially in recent years. For comparison, our estimates are around 2-15% of Naik’s lower bound, and suggest that hardware constitutes around 0.3%-1.9% of GWP.

Such large changes in the long run growth rate are surprising to us, and—if they are real—we are unsure what produced them. One possibility is that hardware prices have stopped falling so fast (i.e. Moore’s Law is ending for the price of computation). Another is that spending on hardware decreased for some reason, for instance because people stopped enjoying large returns from additional hardware. We think this question deserves further research.

Implications

Global computing capacity in terms of human brains

According to different estimates, the human brain performs the equivalent of between 3 x 1013 and 1025 FLOPS. The median estimate we know of is 1018 FLOPS. According to that median estimate and our estimate of global computing hardware, if the world’s entire computing capacity could be directed at running minds around as efficient as those of humans, we would have the equivalent of 200-1500 extra human minds.7 That is, turning all of the world’s hardware into human-efficiency minds at present would increase the world’s population of minds by at most about 0.00002%. If we select the most favorable set of estimates for producing large numbers, turning all of the world’s computing hardware into minds as efficient as humans’ would produce around 50 million extra minds, increasing the world’s effective population by about 1%.8

Figure: Projected number of human brains equivalent

Figure: Projected number of human brains equivalent to global hardware under various assumptions. For brains, ‘small’ = 3 x 10^ 13, ‘median’ = 10^18, ‘large’ = 10^25. For ‘world hardware’, ‘high’ =2 x 10^20, ‘low’ = 1.5 x 10^21. ‘Growth’ is growth in computing hardware, the unlabeled default used in most projections is 25% per annum (our estimate above), ‘high’ = 86% per annum (the apparent growth rate in ASIC hardware in around 2007).

 


 

Costs of human-level hardware

Computing hardware which is equivalent to the brain –

  • in terms of FLOPS probably costs between $1 x 105 and $3 x 1016, or $2/hour-$700bn/hour.
  • in terms of TEPS probably costs $200M – $7B, or or $4,700 – $170,000/hour (including energy costs in the hourly rate).
  • in terms of secondary memory probably costs $300-3,000, or $0.007-$0.07/hour.

Details

Partial costs

Computation

Main articles: Brain performance in FLOPS, Current FLOPS prices, Trends in the costs of computing

FLoating-point Operations Per Second (FLOPS) is a measure of computer performance that emphasizes computing capacity. The human brain is estimated to perform between 1013.5 and 1025 FLOPS. Hardware currently costs around $3 x 10-9/FLOPS, or $7 x 10-14/FLOPShour. This makes the current price of hardware which has equivalent computing capacity to the human brain between $1 x 105 and $3 x 1016, or $2/hour-$700bn/hour if hardware is used for five years.

The price of FLOPS has probably decreased by a factor of ten roughly every four years in the last quarter of a century.

Communication

Main articles: Brain performance in TEPSThe cost of TEPS 

Traversed Edges Per Second (TEPS) is a measure of computer performance that emphasizes communication capacity. The human brain is estimated to perform at 0.18 – 6.4 x 105 GTEPS. Communication capacity costs around $11,000/GTEP or $0.26/GTEPShour in 2015, when amortized over five years and combined with energy costs. This makes the current price of hardware which has equivalent communication capacity to the human brain around $200M – $7B in total, or $4,700 – $170,000/hour including energy costs.

We estimate that the price of TEPS falls by a factor of ten every four years, based the relationship between TEPS and FLOPS.

Information storage

Main articles: Information storage in the brainCosts of information storageCosts of human-level information storage

Computer memory comes in primary and secondary forms. Primary memory (e.g. RAM) is intended to be accessed frequently, while secondary memory is slower to access but has higher capacity. Here we estimate the secondary memory requirements ofthe brain. The human brain is estimated to store around 10-100TB of data. Secondary storage costs around $30/TB in 2015. This means it costs $300-3,000 for enough storage to store the contents of a human brain, or $0.007-$0.07/hour if hardware is used for five years.

In the long run the price of secondary memory has declined by an order of magnitude roughly every 4.6 years. However the rate has declined so much that prices haven’t substantially dropped since 2011 (in 2015).

Interpreting partial costs

Calculating the total cost of hardware that is relevantly equivalent to the brain is not as simple as adding the partial costs as listed. FLOPS and TEPS are measures of different capabilities of the same hardware, so if you pay for TEPS at the aforementioned prices, you will also receive FLOPS.

The above list is also not exhaustive: there may be substantial hardware costs that we haven’t included.

Brain performance in FLOPS

Five credible estimates of brain performance in terms of FLOPS that we are aware of are spread across the range from 3 x 1013 to 1025. The median estimate is 1018.

Details

Notes

We have not investigated the brain’s performance in FLOPS in detail. This page summarizes others’ estimates that we are aware of. Text on this page was heavily borrowed from a blog post, Preliminary prices for human-level hardware.

Estimates

Sandberg and Bostrom 2008

Sandberg and Bostrom project the processing required to emulate a human brain at different levels of detail.1 For the three levels that their workshop participants considered most plausible, their estimates are 1018, 1022, and 1025 FLOPS. These would cost around $100K/hour, $1bn/hour and $1T/hour in 2015.

Moravec 2009

Moravec (2009) estimates that the brain performs around 100 million MIPS.2 MIPS are not directly comparable to MFLOPS (millions of FLOPS), and have deficiencies as a measure, but the empirical relationship in computers is something like MFLOPS = 2.3 x MIPS0.89, according to Sandberg and Bostrom.3 This suggests Moravec’s estimate coincides with around 3.0 x 1013 FLOPS. Given that an order of magnitude increase in computing power per dollar corresponds to about four years, knowing that MFLOPS and MIPS are roughly comparable is plenty of precision.

Kurzweil 2005

In The Singularity is Near, Kurzweil claimed that a human brain required 1016 calculations per second, which appears to be roughly equivalent to 1016 FLOPS.4


 

Index of articles about hardware

Hardware in terms of computing capacity (FLOPS and MIPS)

Brain performance in FLOPS

Current FLOPS prices

Trends in the cost of computing

Wikipedia history of GFLOPS costs

Hardware in terms of communication capacity (TEPS)

Brain performance in TEPS (includes the cost of brain-level TEPS performance on current hardware)

The cost of TEPS (includes current costs, trends and relationship to other measures of hardware price)

Information storage

Information storage in the brain

Costs of information storage

Costs of human-level information storage

Other

Costs of human-level hardware

Research topic: hardware, software and AI

Index of articles about hardware

Related blog posts

Preliminary prices for human level hardware (4 April 2015)

A new approach to predicting brain-computer parity (7 May 2015)

Time flies when robots rule the earth (28 July 2015)

Cost of human-level information storage

It costs roughly $300-$3000 to buy enough storage space to store all information contained by a human brain.

Support

The human brain probably stores around 10-100TB of data. Data storage costs around $30/TB. Thus it costs roughly $300-$3000 to buy enough storage space to store all information contained by a human brain.

If we suppose that one wants to replace the hardware every five years, this is $0.007-$0.07/hour.1

For reference, we have estimated that the computing hardware and electricity required to do the computation the brain does would cost around $4,700 – $170,000/hour at present (using an estimate based on TEPS, and assuming computers last for five years). Estimates based on computation rather than communication capabilities (like TEPS) appear to be spread between $3/hour and $1T/hour.2 On the TEPS-based estimate then, the cost of replicating the brain’s information storage using existing hardware would currently be between a twenty millionth and a seventy thousandth of the cost of replicating the brain’s computation using existing hardware.

Costs of information storage

Cheap secondary memory appears to cost around $0.03/GB in 2015. In the long run the price has declined by an order of magnitude roughly every 4.6 years. However the rate has declined so much that prices haven’t substantially dropped since 2011 (in 2015).

Support

Cheap secondary memory appears to cost around $0.03/GB in 2015.1

The price appears to have declined at an average rate of around an order of magnitude every five years in the long run, as illustrated in Figures 1 and 2. Figure 1 shows roughly six and a half orders of magnitude in the thirty years between 1985 and 2015, for around an order of magnitude every 4.6 years. Figure 2 shows thirteen orders of magnitude over the the sixty years between 1955 and 2015, for exactly the same rate. Both figures suggest the rate has been much slower in the past five years, seemingly as part of a longer term flattening. It appears that prices haven’t substantially dropped since 2011 (in 2015).

xxx

Figure 1: Historic prices of hard drive space, from Matt Komorowski

Figure 2:

Figure 2: Historical prices of information storage in various formats, from Havard Blok, mostly drawing on John C. McCallum’s data.


 

Information storage in the brain

The brain probably stores around 10-100TB of data.

Support

According to Forrest Wickman, computational neuroscientists generally believe the brain stores 10-100 terabytes of data.1 He suggests that these estimates are produced by assuming that information is largely stored in synapses, and that each synapse stores around 1 byte. The number of bytes is then simply the number of synapses.

These assumptions are simplistic (as he points out). In particular:

  • synapses may store more or less than one byte of information on average
  • some information may be stored outside of synapses
  • not all synapses appear to store information
  • synapses do not appear to be entirely independent

We estimate that there are 1.8-3.2 x 10¹⁴ synapses in the human brain, so according to the procedure Wickman outlines, this suggests that the brain stores around 180-320TB of data. It is unclear from his article whether the variation in the views of computational neuroscientists is due to different opinions on the assumptions stated above, or on the number of synapses in the brain. This makes it hard to adjust our estimate well, so our best guess for now is that the brain can store around 10-100TB of data, based on this being the common view among computational neuroscientists.


 

Conversation with Steve Potter

Participants

Figure 1: Professor Steve Potter

  • Professor Steve Potter – Associate Professor, Laboratory of NeuroEngineering, Coulter Department of Biomedical Engineering, Georgia Institute of Technology
  • Katja Grace – Machine Intelligence Research Institute (MIRI)

Note: These notes were compiled by MIRI and give an overview of the major points made by Professor Steve Potter.

Summary

Katja Grace spoke with Professor Steve Potter of Georgia Institute of Technology as part of AI Impacts’ investigation into the implications of neuroscience for artificial intelligence (AI). Conversation topics included how neuroscience now contributes to AI and how it might contribute in the future.

How has neuroscience helped AI in the past?

Professor Potter found it difficult to think of examples where neuroscience has helped with higher level ideas in AI. Some elements of cognitive science have been implemented in AI, but these may not be biologically based. He described two broad instances of neuroscience-inspired projects.

Subsumption architecture

Past work in AI has focused on disembodied computers with little work in robotics. Researchers now understand that AI does not need to be centralized; it can also take on physical form. Subsumption architecture is one way that robotics has advanced. This involves the coupling of sensory information to action selection. For example, Professor Rodney Brooks at MIT has developed robotic legs that respond to certain sensory signals. These legs also send messages to one another to control their movement. Professor Potter believes that this work could have been based on neuroscience, but it is not clear how much Professor Brooks was inspired by neuroscience while working on this project; the idea may have come to him independently.

Neuromorphic engineering

This type of engineering employs properties of biological nervous systems in neural system AI, such as perception and motor control. One aspect of brain function can be imitated with silicon chips through pulse-coding, where analog signals are sent and received in tiny pulses. An application for this is in camera development by mimicking pulse-coded signals between the brain and the retina.

How is neuroscience contributing to AI today?

Although neuroscience has not assisted AI development much in the past, Professor Potter has confidence that this intersection has considerable potential. This is because the brain works well in areas where AI falls short. For example, AI needs to improve how it works in real time in the real world. Self-driving cars may be improved through examining how a model organism, such as a bee, would respond to an analogous situation. Professor Potter believes it would be worthwhile research to record how humans use their brains while driving. Brain algorithms developed from this could be implemented into car design.

Current work at the intersection of neuroscience and AI include the following:

Artificial neural networks

Most researchers at the intersection of AI and neuroscience are examining artificial neural networks, and might describe their work as ‘neural simulations’. These networks are a family of statistical learning models that are inspired by biological neural networks. Hardware in this discipline includes neuromorphic chips, while software includes work in pattern recognition. This includes handwriting recognition and finding military tanks in aerial photographs. The translation of these networks into useful products for both hardware and software applications has been slow.

Hybrots

Professor Potter has helped develop hybrots, which are hybrid living tissue interfaced with robotic machines: robots controlled by neurons. Silent Barrage was an early hybrot that drew on paper attached to pillars. Video was taken of people viewing the Silent Barrage hybrots. This data was transmitted back to Prof. Potter’s lab, where it was used to trigger electrical stimulation in the living brain of the system. This was a petri dish interfaced to a culture of rat cortical neurons. This work is currently being expanded to include more types of hybrots. In one the control will be by living neurons, while the other will be controlled by a simulated neural network.

Meart (MultiElectrode Array Art) was an earlier hybrot. Controlled by a brain composed of rat neuron cells, it used robotic arms to draw on paper. It never progressed past the toddler stage of scribbling.

How is neuroscience likely to help AI in the future?

A particular line of research in neuroscience that is likely to help with AI is the concept of delays. Computer design is often optimized to reduce the amount of time between command and execution. The brain though may take milliseconds longer to respond. However delays in the brain were evolved to respond to the timing of the real world and are a useful part of the brain’s learning process.

Neuroscience probably also has potential to help AI in searching databases. It appears that the brain has methods for this that are completely unlike those used in computers, though we do not yet know what the brain’s methods are. One example given of the brain’s impressive abilities here is that Professor Potter can meet a new person and instantly be confident that he has never seen that person before.

How long will it take to duplicate human intelligence?

It will be hard to say when this has been achieved; success is happening at different rates for different applications. The future of neuroscience in AI will most likely involve taking elements of neuroscience and applying them to AI; it is unlikely that there will be a wait until we have a good understanding of the brain, then an export of that knowledge complete to AI.

Professor Potter greatly respects Ray Kurzweil, but does not think that he has an in depth knowledge of neuroscience. Professor Potter thinks the brain is much more complex than Kurzweil appears to believe, and that ‘duplicating’ human intelligence will take far longer than Kurzweil predicts. In Professor Potter’s consideration, it will take over a hundred years to develop a robot butler that can convince you that it is human.

Challenges to progress

Lack of collaboration

Neuroscience-inspired AI progress has been hampered because researchers across neuroscience and AI seldom collaborate with one another. This may be from disinterest or limited understanding of each other’s fields. Neuroscientists are not generally interested in the goal of creating human-level artificial intelligence. Professor Potter believes that of the roughly 30,000 people who attend the Society for Neuroscience, approximately 20 people want this. Most neuroscientists, for example, want to learn how something works instead of learning how it can be applied (e.g. learning how the auditory system works instead of developing a new hearing aid). If more people saw benefits in applying neuroscience to AI and in particular human-level AI, there would be greater progress. However, the scale is hard to predict. There is the potential for very much more rapid progress. For researchers to move their projects in this direction, the priorities of funding agencies would first have to move; these as these effectively dictate which projects move forward.

Funding

Funding for work at the intersection of neuroscience and AI may be hard to find. The National Institute of Health (NIH) funds only health-related work and has not funded AI projects. The National Science Foundation (NSF) may not think the work fits its requirement of being basic science research; it may be too applied. NSF though, is more open-minded to funding research on AI than NIH is. The military is also interested in AI research. Outside (of )the U.S., the European Union (EU) funds cross-disciplinary work in neuroscience and AI.

National Science Foundation (NSF) funding

NSF had a call for radical proposals, from which Professor Potter received a four-year-long grant to apply neuroscience to electrical grid systems. Collaborators included a power engineer and people studying neural networks. The group was interested in addressing the U.S.’s large and uneven power supply and usage. The electrical grid has become increasingly difficult to control because of geographically varying differences in input and output.

Professor Potter believes that if people in neuroscience, AI, neural networks, and computer design talked more, this would bring progress. However, there were some challenges with this collaborative electrical grid systems project that need to be addressed. For example, the researchers needed to spend considerable time educating one another about their respective fields. It was also difficult to communicate with collaborators across the country; NSF paid for only one meeting per year, and the nuances of in-person interaction seem important for bringing together such diverse groups of people and reaping the benefits of their creative communication.

Other people working in this field

  • Henry Markram – Professor, École Polytechnique Fédérale de Lausanne, Laboratory of Neural Microcircuitry. Using EU funding, he creates realistic computer models of the brain, one piece at a time.
  • Rodney Douglas – Professor Emeritus, University of Zurich, Institute of Neuroinformatics. He is a neuromorphic engineer who worked on emulated brain function.
  • Carver Mead – Gordon and Betty Moore Professor of Engineering and Applied Science Emeritus, California Institute of Technology. He was a founding father of neuromorphic engineering.
  • Rodney Brooks – Panasonic Professor of Robotics Emeritus, Massachusetts Institute of Technology (MIT). He was a pioneer in studying distributed intelligence and developed subsumption architecture.
  • Andy Clark – Professor of Logic and Metaphysics, University of Edinburgh. He does work on embodiment, artificial intelligence, and philosophy.
  • Jose Carmena – Associate Professor of Electrical Engineering and Neuroscience, University of California-Berkeley. Co-Director of the Center of Neural Engineering and Prostheses, University of California-Berkeley, University of California-San Francisco. He has researched the impact of electrical stimulation on sensorimotor learning and control in rats.
  • Guy Ben-Ary – Manager, University of Western Australia, CELLCentral in the School of Anatomy and Human Biology. He is an artist and researcher who uses biologically related technology in his work. He worked in collaboration with Professor Potter on Silent Barrage.
  • Wolfgang Maass – Professor of Computer Science, Graz University of Technology. He is doing research on artificial neural networks.
  • Thad Starner – Assistant Professor, Georgia Institute of Technology, College of Computing. He applies biological concepts into developing wearable computing devices.
  • Jennifer Hasler – Professor, Georgia Institute of Technology, Bioengineering and Electronic Design and Applications. She has studied neuromorphic hardware.

 

  1. “…Most computational neuroscientists tend to estimate human storage capacity somewhere between 10 terabytes and 100 terabytes, though the full spectrum of guesses ranges from 1 terabyte to 2.5 petabytes. (One terabyte is equal to about 1,000 gigabytes or about 1 million megabytes; a petabyte is about 1,000 terabytes.)

    The math behind these estimates is fairly simple. The human brain contains roughly 100 billion neurons. Each of these neurons seems capable of making around 1,000 connections, representing about 1,000 potential synapses, which largely do the work of data storage. Multiply each of these 100 billion neurons by the approximately 1,000 connections it can make, and you get 100 trillion data points, or about 100 terabytes of information.

    Neuroscientists are quick to admit that these calculations are very simplistic. First, this math assumes that each synapse stores about 1 byte of information, but this estimate may be too high or too low…”

    – Wickman 2012

  2. “So it seems human-level hardware presently costs between $3/hour and $1T/hour. ” – our blog post, ‘preliminary prices for human-level hardware’.
  3. See p89. It actually says FLOPS not MFLOPS, but this appears to be an error, given the graph.
  4. ‘If we use the figure of 1016 cps that I believe will be sufficient for functional emulation of human intelligence…’ – Kurzweil, The Singularity is Near, p121
  5. In 2007, GWP was probably about $66T (in 2007 dollars). According to Hilbert & Lopez, the world could then perform 2 x 1020 IPS, which is  2 x 1014 MIPS. According to Muehlhauser & Rieber, hardware cost roughly $5 x 10-3/MIPS in 2007. Thus the total value of hardware would have been around $5 x 10-3/MIPS x 2 x 1014 MIPS = $1012 (a trillion dollars), or 1.5% of GWP.
  6. “The respective compound annual growth rates between 1986–2007 were 61% for general-purpose computations and 86% for application-specific computations, which is 10 and 14 times faster than global GDP during that period, respectively.”

    Hilbert & Lopez (2012)

  7. 2 x 1020 /1018  = 2 x 102

    1.5 x 1021/1018=1.5 x 103

  8. 1.5 x 1021 FLOPS of hardware divided by 3 x 1013 FLOPS/brain gives us 5 x 107 minds.

One Comment

  1. AI Impacts – The AI Impacts Blog Says :

    2015-04-01 at 7:27 AM

    […] Articles […]