AI Impacts Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== The range of human intelligence ======

// Published 18 January, 2015; last updated 10 December, 2020 //

<HTML>
<p>The range of human intelligence seems large relative to the space below it, as measured by performance on tasks we care about—despite the fact that human brains are extremely similar to each other.<span style="line-height: 1.5;"> </span></p>
</HTML>


<HTML>
<p>Without knowing more about the sources of variation in human performance, however, we cannot conclude much at all about the likely pace of progress in AI: we are likely to observe significant variation regardless of any underlying facts about the nature of intelligence.</p>
</HTML>



===== Details =====


==== Measures of interest ====


=== Performance ===


<HTML>
<p>IQ is one measure of cognitive performance. Chess ELO is a narrower one. We do not have a general measure that is meaningful across the space of possible minds. However when people speak of ‘superhuman intelligence’ and the intelligence of animals they imagine that these can be meaningfully placed on some rough spectrum. When we say ‘performance’ we mean this kind of intuitive spectrum.</p>
</HTML>


=== Development effort ===


<HTML>
<p>We are especially interested in measuring intelligence by the difficulty of building a machine which exhibits that level of intelligence. We will not use a formal unit to measure this distance, but are interested in comparing the range between humans to distances between other milestones, such as that between a mouse and a human, or a rock and a mouse.</p>
</HTML>


==== Variation in cognitive performance ====


<HTML>
<p>It is sometimes argued that humans occupy a very narrow band in the spectrum of cognitive performance. For instance, Eliezer Yudkowsky defends this rough schemata<span class="easy-footnote-margin-adjust" id="easy-footnote-1-191"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-1-191" title='“My Childhood Role Model &amp;#8211; LessWrong 2.0.” Accessed June 3, 2020. &lt;a href="https://www.lesswrong.com/posts/3Jpchgy53D2gB5qdk/my-childhood-role-model"&gt;https://www.lesswrong.com/posts/3Jpchgy53D2gB5qdk/my-childhood-role-model&lt;/a&gt;.'><sup>1</sup></a></span>—</p>
</HTML>

{{:speed_of_ai_transition:range_of_human_performance:sophvi_einstein-copy1.jpg?direct&400|}}


<HTML>
<p>—over these, which he attributes to others:</p>
</HTML>

{{:speed_of_ai_transition:range_of_human_performance:modsofvi_einstein-copy.jpg?direct&400|}}

{{:speed_of_ai_transition:range_of_human_performance:vi_einstein-copy.jpg?direct&400|}}




<HTML>
<p>Such arguments sometimes go further, to suggest that AI development effort needed to traverse the distance from the ‘village idiot’ to Einstein is also small, and so given that it seems so large to us, AI progress at around human level will <a href="/doku.php?id=featured_articles:likelihood_of_discontinuous_progress_around_the_development_of_agi">seem very fast</a>.</p>
</HTML>


<HTML>
<p>The landscape of performance is not easy to parameterize well, as there are many cognitive tasks and dimensions of cognitive ability, and no good global metric for comparison across different organisms. Nonetheless, we offer several pieces of evidence to suggest that the human range is substantial, relative to the space below it. We do not approach the topic here of how far above human level the space of possible intelligence reaches.</p>
</HTML>


=== Low human performance on specific tasks ===


<HTML>
<p class="p2"><span class="s1">For most tasks, human performance reaches all of the way to the bottom of the possible spectrum. At the extreme</span>, some comatose humans will fail at almost any cognitive task. Our impression is that people who are completely unable to perform a task are not usually isolated outliers, but that there is a distribution of people spread across the range from completely incapacitated to world-champion level. That is, for a task like ‘recognize a cat’, there are people who can only do slightly better than if they were comatose.</p>
</HTML>


<HTML>
<p class="p2">For our purposes we are more interested in where normal human cognitive performance fall relative to the worst and best possible performance, and the best human performance.</p>
</HTML>


=== Mediocre human performance relative to high human performance ===


<HTML>
<p class="p1">On many tasks, it seems likely that the best humans are many times better than mediocre humans, using relatively objective measures.<span class="easy-footnote-margin-adjust" id="easy-footnote-2-191"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-2-191" title="In particular"><sup>2</sup></a></span></p>
</HTML>


<HTML>
<p class="p1"><a href="http://www.gwern.net/docs/1957-shockley.pdf">Shockley (1957)</a> found that in science, the productivity of the top researchers in a laboratory was often at least ten times as great as the least productive (and most numerous) researchers. Programmers <a href="http://programmers.stackexchange.com/questions/179616/a-good-programmer-can-be-as-10x-times-more-productive-than-a-mediocre-one">purportedly</a> vary by an order of magnitude in productivity, though this is debated. A third of people scored nothing in <a href="https://math.mit.edu/news/spotlight/Putnam-2012-Results.pdf">this</a> Putnam competition, while someone scored 100. Some people have to work ten times harder to pass their high school classes than others.</p>
</HTML>


<HTML>
<p class="p1">Note that these differences are among people skilled enough to actually be in the relevant field, which in most cases suggests they are above average. Our impression is that something similar is true in other areas such as sales, entrepreneurship, crafts, and writing, but we have not seen data on them.</p>
</HTML>


<HTML>
<p class="p1">These large multipliers on performance at cognitive tasks suggest that the range between mediocre cognitive ability and genius is many times larger than the range below mediocre cognitive ability. However it is not clear that such differences are common, or to what extent they are due to differences in underlying general cognitive ability, rather than learning or non-cognitive skills, or a range of different cognitive skills that aren’t well correlated.</p>
</HTML>


=== Human performance spans a wide range in other areas ===


<HTML>
<p>In qualities other than intelligence, humans appear to span a fairly wide range below their peak levels. For instance, the fastest human runners are multiple times faster than mediocre runners (<a href="http://www.telegraph.co.uk/sport/olympics/athletics/9450234/100m-final-how-fast-could-you-run-it.html">twice as fast</a> at a 100m sprint, <a href="http://www.livestrong.com/article/551509-a-good-mile-rate-for-a-beginner-runner/">four</a> <a href="http://en.wikipedia.org/wiki/Mile_run">times</a> as fast for a mile). Humans can vary in height by a factor of about <a href="http://www.dailymail.co.uk/news/article-2832768/The-odd-couple-Shortest-man-21-5ins-meets-tallest-living-person-8ft-1in-outside-Houses-Parliament-Guinness-World-Record-Day.html">four</a>, and commonly do by a factor of about 1.5. The most accurate painters are <a href="http://twistedsifter.com/2012/04/15-hyperrealistic-paintings-that-look-like-photos-campos/">hard to distinguish</a> from photographs, while some painters are <a href="http://www.psychologytoday.com/blog/psyched/201103/my-monkey-could-have-painted-really">arguably</a> hard to distinguish from monkeys, which are very easy to distinguish from photographs. These observations weakly suggest that the default expectation should be for humans to span a wide absolute range in cognitive performance also.</p>
</HTML>


=== AI performance on human tasks ===


<HTML>
<p>In domains where we have observed human-level performance in machines, we have seen rather gradual improvement across the range of human abilities. Here are five relevant cases that we know of:</p>
</HTML>

1. **Chess**// (See [[speed_of_ai_transition:range_of_human_performance:time_for_ai_to_cross_the_human_performance_range_in_chess|Time for AI to cross the human performance range in chess]])//: human chess Elo ratings conservatively range from around [[http://en.wikipedia.org/wiki/Elo_rating_system#United_States_Chess_Federation_ratings|800 (beginner)]] to 2800 (world champion). The following figure illustrates how it took chess AI roughly forty years to move incrementally from 1300 to 2800.


<HTML>
<figure aria-describedby="caption-attachment-371" class="wp-caption alignnone" id="attachment_371" style="width: 600px">
<a href="http://aiimpacts.wpengine.com/wp-content/uploads/2014/12/chess_progress.gif"><img alt="chess_progress" class="wp-image-371" height="448" loading="lazy" src="http://aiimpacts.wpengine.com/wp-content/uploads/2014/12/chess_progress.gif" width="600"/></a>
<figcaption class="wp-caption-text" id="caption-attachment-371">
                  Figure 1: Chess AI progress compared to human performance, from <a href="http://www.drdobbs.com/parallel/computer-chess-the-drosophila-of-ai/184405171">Coles 2002</a>. The original article was apparently written before 1993, so note that the right of the graph (after ‘now’) is imagined, though it appears to be approximately correct.
                </figcaption>
</figure>
</HTML>


2. **Go**// (See [[speed_of_ai_transition:range_of_human_performance:time_for_ai_to_cross_the_human_performance_range_in_go|Time for AI to cross the human performance range in Go]])//: Human go ratings [[http://en.wikipedia.org/wiki/Go_ranks_and_ratings|range]] from 30-20 kyu (beginner) to at least 9p (10p is a special title). Note that the numbers go downwards through kyu levels, then upward through dan levels, then upward through p(rofessional dan) levels. The following figure suggests that it took around 25 years for AI to cover most of this space (the top ratings seem to be [[http://en.wikipedia.org/wiki/Go_ranks_and_ratings#Elo-like_rating_systems_as_used_in_Go|closer together]] than the lower ones, though there are apparently [[https://en.wikipedia.org/wiki/Go_ranks_and_ratings#Winning_probabilities|multiple systems]] which vary).


<HTML>
<figure aria-describedby="caption-attachment-372" class="wp-caption alignnone" id="attachment_372" style="width: 600px">
<a href="http://aiimpacts.wpengine.com/wp-content/uploads/2014/12/gobothistory-copy.jpg"><img alt="From Grace 2013." class="wp-image-372" height="436" loading="lazy" sizes="(max-width: 600px) 100vw, 600px" src="http://aiimpacts.wpengine.com/wp-content/uploads/2014/12/gobothistory-copy-1024x743.jpg" srcset="https://aiimpacts.org/wp-content/uploads/2014/12/gobothistory-copy-1024x743.jpg 1024w, https://aiimpacts.org/wp-content/uploads/2014/12/gobothistory-copy-300x218.jpg 300w, https://aiimpacts.org/wp-content/uploads/2014/12/gobothistory-copy.jpg 1358w" width="600"/></a>
<figcaption class="wp-caption-text" id="caption-attachment-372">
                  Figure 2. From <a href="http://intelligence.org/files/AlgorithmicProgress.pdf">Grace 2013</a>.
                </figcaption>
</figure>
</HTML>

3. **StarCraft** //(See [[speed_of_ai_transition:range_of_human_performance:time_for_ai_to_cross_the_human_range_in_starcraft|Time for AI to cross the human range in StarCraft]]):// Beginner level StarCraft-playing AI seems to have been possible since the game was released in 1998. In 2018, AlphaStar beat a professional StarCraft player, suggesting that state-of-the-art StarCraft-playing AI had reached professional level. No AI seems to have above superhuman skills in StarCraft.

4. **Checkers** //(See [[speed_of_ai_transition:range_of_human_performance:time_for_ai_to_cross_the_human_range_in_english_draughts|Time for AI to cross the human range in English draughts]]):// According to [[http://en.wikipedia.org/wiki/Timeline_of_artificial_intelligence|Wikipedia’s timeline of AI]], a program was written in 1952 that could challenge a respectable amateur. In 1994 Chinook beat the second highest rated player ever. (In 2007 checkers was solved.) Thus it took around forty years to pass from amateur to world-class checkers-playing. We know nothing about whether intermediate progress was incremental however.

5. **Diagnosing diabetic retinopathy** //(See [[speed_of_ai_transition:range_of_human_performance:time_for_ai_to_cross_the_human_performance_range_in_diabetic_retinopathy|Time for AI to cross the human performance range in diabetic retinopathy]]))//: Diabetic retinopathy, a complication of diabetes in which high blood sugar levels damage the back of the eye, is diagnosed by examining images of the back of the eye. The first algorithm for detecting diabetic retinopathy, developed in 1996, could not detect it as well as ophthalmologists. In 2016, Google released results for algorithms that could detect diabetic retinopathy about as well as ophthalmologists.

6. **Image classification** //(See [[featured_articles:time_for_ai_to_cross_the_human_performance_range_in_imagenet_image_classification|Time for AI to cross the human performance range in ImageNet image classification]])//: The first image classifiers were developed in 1998. ImageNet is a large collection of images organized into a hierarchy of noun categories, released in 2009. Our beginner level benchmark for ImageNet classification was first surpassed in 2012, by AlexNet. In 2015, an image classifier surpassed our high human benchmark, suggesting that state-of-the-art AI could classify images about as well as expert humans.

<HTML>
<p>7. <strong>Physical manipulation:</strong> we have not investigated this much, but our impression is that robots are somewhere in the the fumbling and slow part of the human spectrum on <a href="http://youtu.be/oD9DE0HjMM4">some</a> <a href="https://www.youtube.com/watch?v=IBY4t8XxH7E">tasks</a>, and that nobody expects them to reach the ‘normal human abilities’ part any time soon (<a href="http://aiimpacts.wpengine.com/hanson-ai-expert-survey/" title="Hanson AI Expert Survey">Aaron Dollar estimates</a> robotic grasping manipulation in general is less than one percent of the way to human level from where it was 20 years ago).</p>
</HTML>


<HTML>
<p>8. <strong>Jeopardy</strong>: AI appears to have taken two or three years to move from lower ‘champion’ level to surpassing world champion level (see <a href="https://www.aaai.org/ojs/index.php/aimagazine/article/view/2303/2165">figure 9</a>; Watson beat Ken Jennings in <a href="http://en.wikipedia.org/wiki/Watson_%28computer%29">2011</a>). We don’t know how far ‘champion’ level is from the level of a beginner, but would be surprised if it were less than four times the distance traversed here, given the situation in other games, suggesting a minimum of a decade for crossing the human spectrum.</p>
</HTML>


<HTML>
<p>In all of these narrow skills, moving AI from low-level human performance to top-level human performance appears to take on the order of decades. This further undermines the claim that the range of human abilities constitutes a narrow band within the range of possible AI capabilities, though we may expect general intelligence to behave differently, for example due to smaller training effects.</p>
</HTML>


<HTML>
<p>On the other hand, most of the examples here—and in particular the ones that we know the most about—are board games, so this phenomenon may be less usual elsewhere. We have not investigated areas such as Texas hold ’em, arithmetic or constraint satisfaction sufficiently to add them to this list.</p>
</HTML>


==== What can we infer from human variation? ====


<HTML>
<p>The brains of humans are nearly identical, <a href="http://lesswrong.com/lw/ql/my_childhood_role_model/">by comparison</a> to the brains of other animals or to other possible brains that could exist. This might suggest that the engineering effort required to move across the human range of intelligences is quite small, compared to the engineering effort required to move from very sub-human to human-level intelligence (e.g. see <a href="https://intelligence.org/files/AIPosNegFactor.pdf">p21 and 29</a>, <a href="http://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies">p70</a>). The similarity of human brains also suggest that the range of human intelligence is smaller than it seems, and its apparent breadth is due to anthropocentrism (see the same sources). According to these views, board games are an exceptional case–for most problems, it will not take AI very long to close the gap between “mediocre human” and “excellent human.”</p>
</HTML>


<HTML>
<p>However, we should not be surprised to find meaningful variation in the cognitive performance <em>regardless</em> of the difficulty of improving the human brain. This makes it difficult to infer much from the observed variations.</p>
</HTML>


<HTML>
<p>Why should we not be surprised? De novo deleterious mutations are introduced into the genome with each generation, and the prevalence of such mutations is determined by the <a href="http://en.wikipedia.org/wiki/Mutation%E2%80%93selection_balance">balance</a> of mutation rates and negative selection. If de novo mutations significantly impact cognitive performance, then there must necessarily be significant selection for higher intelligence–and hence behaviorally relevant differences in intelligence. This balance is determined entirely by the mutation rate, the strength of selection for intelligence, and the negative impact of the average mutation.</p>
</HTML>


<HTML>
<p>You can often make a machine worse by breaking a random piece, but this does not mean that the machine was easy to design or that you can make the machine better by adding a random piece. Similarly, levels of variation of cognitive performance in humans may tell us very little about the difficulty of making a human-level intelligence smarter.</p>
</HTML>


<HTML>
<p>In the extreme case, we can observe that brain-dead humans often have very similar cognitive architectures. But this does not mean that it is easy to start from an AI at the level of a dead human and reach one at the level of a living human.</p>
</HTML>


<HTML>
<p>Because we should not be surprised to see significant variation–independent of the underlying facts about intelligence–we cannot infer very much from this variation. The strength of our conclusions are limited by the extent of our possible surprise.</p>
</HTML>


<HTML>
<p>By better understanding the sources of variation in human performance we may be able to make stronger conclusions. For example, if human intelligence is improving rapidly due to the introduction of  new architectural improvements to the brain, this suggests that discovering architectural improvements is not too difficult.  If we discover that spending more energy on thinking makes humans substantially smarter, this suggests that scaling up intelligences leads to large performance changes. And so on. Existing research in biology addresses the role of deleterious mutations, and depending on the results this literature could be used to draw meaningful inferences.</p>
</HTML>


<HTML>
<p>These considerations also suggest that brain similarity can’t tell us much about the “true” range of human performance. This isn’t too surprising, in light of the analogy with other domains. For example, although the bodies of different runners have nearly identical designs, the worst runners are not nearly as good as the best.</p>
</HTML>


<HTML>
<p>xxx[This background rate of human-range crossing is less informative about the future in scenarios where the increasing machine performance of interest is coming about in a substantially different way from how it came about in the past. For instance, it is sometimes hypothesized that major performance improvements will come from fast ‘recursive self improvement’, in which case the characteristic time scale might be much faster. However the scale of the human performance range (and time to cross it) relative to the area below the human range should still be informative.]</p>
</HTML>


<HTML>
<ol class="easy-footnotes-wrapper">
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-1-191"></span>“My Childhood Role Model – LessWrong 2.0.” Accessed June 3, 2020. <a href="https://www.lesswrong.com/posts/3Jpchgy53D2gB5qdk/my-childhood-role-model">https://www.lesswrong.com/posts/3Jpchgy53D2gB5qdk/my-childhood-role-model</a>.<a class="easy-footnote-to-top" href="#easy-footnote-1-191"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-2-191"></span>In particular<a class="easy-footnote-to-top" href="#easy-footnote-2-191"></a>
</div></li>
</ol>
</HTML>
AI Impacts Wiki

User Tools

Site Tools

Page Tools
