https://wiki.aiimpacts.org/ 2026-04-29T17:44:13+00:00 FeedCreator 1.8 (info@mypapit.net) 2022-09-21T07:37:41+00:00 2022-09-21T07:37:41+00:00 https://wiki.aiimpacts.org/featured_articles/ai_impacts_research_bounties?rev=1663745861&do=diff Anonymous anonymous@undisclosed.example.com <pre> @@ -1 +1,123 @@ + ====== AI Impacts research bounties ====== + + // Published 06 August, 2015; last updated 28 September, 2017 // + + &lt;HTML&gt; + &lt;p&gt;We are offering rewards for several inputs to our research, described below. These offers have no specific deadline except where noted. We may modify them or take them down, but will give at least one week’s notice here unless there is strong reason not to. To submit an entry, email katja@intelligence.org. There is currently a large backlog of entries to check, so new entries will not receive a rapid response.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== 1. An example of discontinuous technological progress ($50-$500) ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;em&gt;This bounty offer is no longer available after 3 November 2016.&lt;/em&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;We are interested in finding more examples of large discontinuous technological progress to add to &lt;a href=&quot;/doku.php?id=featured_articles:cases_of_discontinuous_technological_progress&quot;&gt;our collection&lt;/a&gt;. We’re offering a bounty of around $50-500 per good example.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;We currently know of &lt;a href=&quot;/doku.php?id=featured_articles:cases_of_discontinuous_technological_progress&quot;&gt;two good examples&lt;/a&gt; (and one moderate example):&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;strong&gt;Nuclear weapons&lt;/strong&gt; discontinuously increased the &lt;a href=&quot;https://en.wikipedia.org/wiki/Relative_effectiveness_factor&quot;&gt;relative effectiveness&lt;/a&gt; of explosives. + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;strong&gt;High temperature superconductors &lt;/strong&gt;led to a dramatic increase in the highest temperature at which superconducting was possible.&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;To assess discontinuity, we’ve been using “number of years worth of progress at past rates”, as measured by any relevant metric of technological progress. For example, the discovery of nuclear weapons was equal to about 6,000 years worth of previous progress in the relative effectiveness of explosives. However, we are also interested in examples that seem intuitively discontinuous, even if they don’t exactly fit the criteria of being a large number of year’s progress in one go.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Things that make examples better:&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;strong&gt;Size:&lt;/strong&gt; Better examples represent larger changes. More than 20 times normal annual progress is ideal.&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;strong&gt;Sharpness:&lt;/strong&gt; Better examples happened over shorter periods. Over less than a year is ideal.&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;strong&gt;Breadth:&lt;/strong&gt; Metrics that measure larger categories of things are better. For example, fast adoption curves for highly specific categories (say a particular version of some software) is much less interesting than fast adoption curves for much broader categories (say a whole category of software).&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;strong&gt;Rarity: &lt;/strong&gt;As we receive more examples, the interestingness of each one will tend to decline.&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;AI Impacts is willing to pay more for better examples. Basically we will judge how interesting your example is and then reward you based on that. We will accept examples that violate our stated preferences but satisfy the spirit of the bounty. Our guess is that we would pay about $500 for another example as good nuclear weapons.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;How to enter:&lt;/strong&gt; all that is necessary to submit an example is to email us a paragraph describing the example, along with sources to verify your claims (such sources are likely to involve at least one time series of success on a particular metric). Note that an example should be of the form ‘A caused abrupt progress in metric B’. For instance, ‘The boliolicopter caused abrupt progress in the maximum rate of fermblangling at sub-freezing temperatures’.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== 2. An example of early action on a risk ($20-$100) ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;em&gt;This bounty offer is no longer available after 3 November 2016.&lt;/em&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span class=&quot;name&quot;&gt;&lt;strong&gt;We want:&lt;/strong&gt; a one sentence description of a case where at least one person acted to avert a risk that was least fifteen years away, along with a link or citation supporting the claim that the action preceded the risk by at least fifteen years. &lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;We will give:&lt;/strong&gt; up $100, with higher sums for examples that are better according to our judgment (see criteria for betterness below), and which we don’t already know about. We might go over $100 for exceptionally good examples.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;Further details&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span class=&quot;name&quot;&gt;Examples are better if:&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span class=&quot;name&quot;&gt;&lt;strong&gt;The risk is more novel:&lt;/strong&gt; relatively similar problems have not arisen before, and would probably not arise sooner than fifteen years in the future. &lt;/span&gt;&lt;span class=&quot;name&quot;&gt;e.g. Poverty in retirement is a risk people often prepare for more than fifteen year before it befalls them, however it is not very novel because other people already face an essentially identical risk, and have done many times before. &lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span class=&quot;name&quot;&gt;&lt;strong&gt;The solution is more specific:&lt;/strong&gt; the action taken would not be nearly as useful if the risk disappeared.&lt;/span&gt; &lt;span class=&quot;name&quot;&gt;e.g. Saving money to escape is a reasonable solution to expecting your country to face civil war soon. However saving money is fairly useful in any case, so this solution is not very specific.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;strong&gt;We haven’t received a lot of examples:&lt;/strong&gt; as we collect more examples, the value of each one will tend to decline.&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span class=&quot;name&quot;&gt;Some examples:&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span class=&quot;name&quot;&gt;&lt;strong&gt;Leo Szilard’s secret nuclear patent&lt;/strong&gt;: the threat of nuclear weapons was quite novel. It’s unclear when Szilard expected such weapons, but quite plausibly at least fifteen years later in 1934. The secret patent does not seem broadly useful, though useful for encouraging more local nuclear research, which is somewhat more broadly useful than secrecy per se. More details in &lt;a href=&quot;https://intelligence.org/files/SzilardNuclearWeapons.pdf&quot;&gt;this report&lt;/a&gt;. This is a reasonably good example.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span class=&quot;name&quot;&gt;&lt;strong&gt;The Asilomar Conference on recombinant DNA&lt;/strong&gt;: the risk of was arguably quite novel (genetically engineered pandemics), and the solution was reasonably specific (safety rules for dealing with recombinant DNA). However the risks people were concerned about were immediate, rather than decades hence. More details &lt;span class=&quot;contentBold&quot;&gt;&lt;a href=&quot;https://intelligence.org/2015/06/30/new-report-the-asilomar-conference-a-case-study-in-risk-mitigation/&quot;&gt;here&lt;/a&gt;&lt;/span&gt;. This is not a good example.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span class=&quot;name&quot;&gt;Evidence that the example is better in the above ways is also welcome, though we reserve the right not to explore it fully.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + </pre>

<pre> @@ -1 +1,123 @@ + ====== AI Impacts research bounties ====== + + // Published 06 August, 2015; last updated 28 September, 2017 // + + &lt;HTML&gt; + &lt;p&gt;We are offering rewards for several inputs to our research, described below. These offers have no specific deadline except where noted. We may modify them or take them down, but will give at least one week’s notice here unless there is strong reason not to. To submit an entry, email katja@intelligence.org. There is currently a large backlog of entries to check, so new entries will not receive a rapid response.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== 1. An example of discontinuous technological progress ($50-$500) ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;em&gt;This bounty offer is no longer available after 3 November 2016.&lt;/em&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;We are interested in finding more examples of large discontinuous technological progress to add to &lt;a href=&quot;/doku.php?id=featured_articles:cases_of_discontinuous_technological_progress&quot;&gt;our collection&lt;/a&gt;. We’re offering a bounty of around $50-500 per good example.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;We currently know of &lt;a href=&quot;/doku.php?id=featured_articles:cases_of_discontinuous_technological_progress&quot;&gt;two good examples&lt;/a&gt; (and one moderate example):&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;strong&gt;Nuclear weapons&lt;/strong&gt; discontinuously increased the &lt;a href=&quot;https://en.wikipedia.org/wiki/Relative_effectiveness_factor&quot;&gt;relative effectiveness&lt;/a&gt; of explosives. + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;strong&gt;High temperature superconductors &lt;/strong&gt;led to a dramatic increase in the highest temperature at which superconducting was possible.&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;To assess discontinuity, we’ve been using “number of years worth of progress at past rates”, as measured by any relevant metric of technological progress. For example, the discovery of nuclear weapons was equal to about 6,000 years worth of previous progress in the relative effectiveness of explosives. However, we are also interested in examples that seem intuitively discontinuous, even if they don’t exactly fit the criteria of being a large number of year’s progress in one go.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Things that make examples better:&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;strong&gt;Size:&lt;/strong&gt; Better examples represent larger changes. More than 20 times normal annual progress is ideal.&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;strong&gt;Sharpness:&lt;/strong&gt; Better examples happened over shorter periods. Over less than a year is ideal.&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;strong&gt;Breadth:&lt;/strong&gt; Metrics that measure larger categories of things are better. For example, fast adoption curves for highly specific categories (say a particular version of some software) is much less interesting than fast adoption curves for much broader categories (say a whole category of software).&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;strong&gt;Rarity: &lt;/strong&gt;As we receive more examples, the interestingness of each one will tend to decline.&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;AI Impacts is willing to pay more for better examples. Basically we will judge how interesting your example is and then reward you based on that. We will accept examples that violate our stated preferences but satisfy the spirit of the bounty. Our guess is that we would pay about $500 for another example as good nuclear weapons.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;How to enter:&lt;/strong&gt; all that is necessary to submit an example is to email us a paragraph describing the example, along with sources to verify your claims (such sources are likely to involve at least one time series of success on a particular metric). Note that an example should be of the form ‘A caused abrupt progress in metric B’. For instance, ‘The boliolicopter caused abrupt progress in the maximum rate of fermblangling at sub-freezing temperatures’.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== 2. An example of early action on a risk ($20-$100) ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;em&gt;This bounty offer is no longer available after 3 November 2016.&lt;/em&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span class=&quot;name&quot;&gt;&lt;strong&gt;We want:&lt;/strong&gt; a one sentence description of a case where at least one person acted to avert a risk that was least fifteen years away, along with a link or citation supporting the claim that the action preceded the risk by at least fifteen years. &lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;We will give:&lt;/strong&gt; up $100, with higher sums for examples that are better according to our judgment (see criteria for betterness below), and which we don’t already know about. We might go over $100 for exceptionally good examples.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;Further details&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span class=&quot;name&quot;&gt;Examples are better if:&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span class=&quot;name&quot;&gt;&lt;strong&gt;The risk is more novel:&lt;/strong&gt; relatively similar problems have not arisen before, and would probably not arise sooner than fifteen years in the future. &lt;/span&gt;&lt;span class=&quot;name&quot;&gt;e.g. Poverty in retirement is a risk people often prepare for more than fifteen year before it befalls them, however it is not very novel because other people already face an essentially identical risk, and have done many times before. &lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span class=&quot;name&quot;&gt;&lt;strong&gt;The solution is more specific:&lt;/strong&gt; the action taken would not be nearly as useful if the risk disappeared.&lt;/span&gt; &lt;span class=&quot;name&quot;&gt;e.g. Saving money to escape is a reasonable solution to expecting your country to face civil war soon. However saving money is fairly useful in any case, so this solution is not very specific.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;strong&gt;We haven’t received a lot of examples:&lt;/strong&gt; as we collect more examples, the value of each one will tend to decline.&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span class=&quot;name&quot;&gt;Some examples:&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span class=&quot;name&quot;&gt;&lt;strong&gt;Leo Szilard’s secret nuclear patent&lt;/strong&gt;: the threat of nuclear weapons was quite novel. It’s unclear when Szilard expected such weapons, but quite plausibly at least fifteen years later in 1934. The secret patent does not seem broadly useful, though useful for encouraging more local nuclear research, which is somewhat more broadly useful than secrecy per se. More details in &lt;a href=&quot;https://intelligence.org/files/SzilardNuclearWeapons.pdf&quot;&gt;this report&lt;/a&gt;. This is a reasonably good example.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span class=&quot;name&quot;&gt;&lt;strong&gt;The Asilomar Conference on recombinant DNA&lt;/strong&gt;: the risk of was arguably quite novel (genetically engineered pandemics), and the solution was reasonably specific (safety rules for dealing with recombinant DNA). However the risks people were concerned about were immediate, rather than decades hence. More details &lt;span class=&quot;contentBold&quot;&gt;&lt;a href=&quot;https://intelligence.org/2015/06/30/new-report-the-asilomar-conference-a-case-study-in-risk-mitigation/&quot;&gt;here&lt;/a&gt;&lt;/span&gt;. This is not a good example.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span class=&quot;name&quot;&gt;Evidence that the example is better in the above ways is also welcome, though we reserve the right not to explore it fully.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + </pre>

2022-11-05T03:52:35+00:00 2022-11-05T03:52:35+00:00 https://wiki.aiimpacts.org/featured_articles/ai_vignettes_project?rev=1667620355&do=diff Anonymous anonymous@undisclosed.example.com <pre> @@ -60,10 +60,5 @@ ==== Vignette collection ==== - A subset of vignettes arising from this project, or similar, can be found [[https://airtable.com/shr4mHlTIiKtFRDuR/tblMVjRvMKVNkoZVg?backgroundColor=cyan&amp;viewControls=on|here]]. - - - - - + A subset of vignettes arising from this project can be found among [[featured_articles:fiction_relevant_to_ai_futurism|fiction relevant to AI futurism]]. </pre>

<pre> @@ -60,10 +60,5 @@ ==== Vignette collection ==== - A subset of vignettes arising from this project, or similar, can be found [[https://airtable.com/shr4mHlTIiKtFRDuR/tblMVjRvMKVNkoZVg?backgroundColor=cyan&amp;viewControls=on|here]]. - - - - - + A subset of vignettes arising from this project can be found among [[featured_articles:fiction_relevant_to_ai_futurism|fiction relevant to AI futurism]]. </pre>

2022-09-21T07:37:41+00:00 2022-09-21T07:37:41+00:00 https://wiki.aiimpacts.org/featured_articles/cases_of_discontinuous_technological_progress?rev=1663745861&do=diff Anonymous anonymous@undisclosed.example.com <pre> @@ -1 +1,87 @@ + ====== Cases of Discontinuous Technological Progress ====== + + // Published 31 December, 2014; last updated 10 December, 2020 // + + &lt;HTML&gt; + &lt;p&gt;We know of ten events which produced a robust discontinuity in progress equivalent to more than one hundred years at previous rates in some interesting metric. We know of 53 other events which produced smaller or less robust discontinuities.&lt;/p&gt; + &lt;/HTML&gt; + + + + ===== Background ===== + + + &lt;HTML&gt; + &lt;p&gt;These cases were researched as part of our &lt;a href=&quot;/doku.php?id=ai_timelines:discontinuous_progress_investigation&quot;&gt;discontinuous progress investigation&lt;/a&gt;.&lt;/p&gt; + &lt;/HTML&gt; + + + ===== List of cases ===== + + + ==== Events causing large, robust discontinuities ==== + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The Pyramid of Djoser, 2650BC (discontinuity in &lt;a href=&quot;/doku.php?id=ai_timelines:historic_trends_in_structure_heights&quot;&gt;structure height trends&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The SS &lt;em&gt;Great Eastern&lt;/em&gt;, 1858 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_ship_size&quot;&gt;ship size trends&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The first telegraph, 1858 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_transatlantic_message_speed&quot;&gt;speed of sending a 140 character message across the Atlantic Ocean&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The second telegraph, 1866 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_transatlantic_message_speed&quot;&gt;speed of sending a 140 character message across the Atlantic Ocean&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The Paris Gun, 1918 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_altitude&quot;&gt;altitude reached by man-made means&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The first non-stop transatlantic flight, in a modified WWI bomber, 1919 (discontinuity in both &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_transatlantic_passenger_travel&quot;&gt;speed of passenger travel across the Atlantic Ocean&lt;/a&gt; and &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_long-range_military_payload_delivery&quot;&gt;speed of military payload travel across the Atlantic Ocean&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The George Washington Bridge, 1931 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_bridge_span_length&quot;&gt;longest bridge span&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The first nuclear weapons, 1945 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:effect_of_nuclear_weapons_on_historic_trends_in_explosives&quot;&gt;relative effectiveness of explosives&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The first ICBM, 1958 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_long-range_military_payload_delivery&quot;&gt;average speed of military payload crossing the Atlantic Ocean&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;YBa&lt;sub&gt;2&lt;/sub&gt;Cu&lt;sub&gt;3&lt;/sub&gt;O&lt;sub&gt;7&lt;/sub&gt; as a superconductor, 1987 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_the_maximum_superconducting_temperature&quot;&gt;warmest temperature of superconduction&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + ==== Events causing moderate, robust discontinuities ==== + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;HMS Warrior, 1860 (discontinuity in both &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_ship_size&quot;&gt;Royal Navy ship tonnage and Royal Navy ship displacement&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Eiffel Tower, 1889 (discontinuity in &lt;a href=&quot;/doku.php?id=ai_timelines:historic_trends_in_structure_heights&quot;&gt;tallest existing freestanding structure height&lt;/a&gt;, and in other height trends non-robustly) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Fairey Delta 2, 1956 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_flight_airspeed_records&quot;&gt;airspeed&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Pellets shot into space, 1957, measured after one day of travel (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_altitude&quot;&gt;altitude achieved by man-made means&lt;/a&gt;)&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-1-202&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-1-202&quot; title=&quot;This was the first of various altitude records where the object continues to gain distance from Earth’s surface continuously over a long period. One could choose to treat these in different ways, and get different size of discontinuity numbers. Strictly, all altitude increases are continuous, so we are anyway implicitly looking at something like discontinuities in heights reached within some period. We somewhat arbitrarily chose to measure altitudes roughly every year, including one day in for the pellets, the only one where the very start mattered. &quot;&gt;&lt;sup&gt;1&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Burj Khalifa, 2009 (discontinuity in &lt;a href=&quot;/doku.php?id=ai_timelines:historic_trends_in_structure_heights&quot;&gt;height of tallest building ever&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + ==== Non-robust discontinuities ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;a href=&quot;https://docs.google.com/spreadsheets/d/1iMIZ57Ka9-ZYednnGeonC-NqwGC7dKiHN9S-TAxfVdQ/edit#gid=1994197408&amp;amp;range=B3:B90&quot;&gt;This spreadsheet&lt;/a&gt; details all discontinuities found, as of April 2020.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol class=&quot;easy-footnotes-wrapper&quot;&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-1-202&quot;&gt;&lt;/span&gt;This was the first of various altitude records where the object continues to gain distance from Earth’s surface continuously over a long period. One could choose to treat these in different ways, and get different size of discontinuity numbers. Strictly, all altitude increases are continuous, so we are anyway implicitly looking at something like discontinuities in heights reached within some period. We somewhat arbitrarily chose to measure altitudes roughly every year, including one day in for the pellets, the only one where the very start mattered. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-1-202&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + </pre>

<pre> @@ -1 +1,87 @@ + ====== Cases of Discontinuous Technological Progress ====== + + // Published 31 December, 2014; last updated 10 December, 2020 // + + &lt;HTML&gt; + &lt;p&gt;We know of ten events which produced a robust discontinuity in progress equivalent to more than one hundred years at previous rates in some interesting metric. We know of 53 other events which produced smaller or less robust discontinuities.&lt;/p&gt; + &lt;/HTML&gt; + + + + ===== Background ===== + + + &lt;HTML&gt; + &lt;p&gt;These cases were researched as part of our &lt;a href=&quot;/doku.php?id=ai_timelines:discontinuous_progress_investigation&quot;&gt;discontinuous progress investigation&lt;/a&gt;.&lt;/p&gt; + &lt;/HTML&gt; + + + ===== List of cases ===== + + + ==== Events causing large, robust discontinuities ==== + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The Pyramid of Djoser, 2650BC (discontinuity in &lt;a href=&quot;/doku.php?id=ai_timelines:historic_trends_in_structure_heights&quot;&gt;structure height trends&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The SS &lt;em&gt;Great Eastern&lt;/em&gt;, 1858 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_ship_size&quot;&gt;ship size trends&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The first telegraph, 1858 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_transatlantic_message_speed&quot;&gt;speed of sending a 140 character message across the Atlantic Ocean&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The second telegraph, 1866 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_transatlantic_message_speed&quot;&gt;speed of sending a 140 character message across the Atlantic Ocean&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The Paris Gun, 1918 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_altitude&quot;&gt;altitude reached by man-made means&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The first non-stop transatlantic flight, in a modified WWI bomber, 1919 (discontinuity in both &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_transatlantic_passenger_travel&quot;&gt;speed of passenger travel across the Atlantic Ocean&lt;/a&gt; and &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_long-range_military_payload_delivery&quot;&gt;speed of military payload travel across the Atlantic Ocean&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The George Washington Bridge, 1931 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_bridge_span_length&quot;&gt;longest bridge span&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The first nuclear weapons, 1945 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:effect_of_nuclear_weapons_on_historic_trends_in_explosives&quot;&gt;relative effectiveness of explosives&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The first ICBM, 1958 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_long-range_military_payload_delivery&quot;&gt;average speed of military payload crossing the Atlantic Ocean&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;YBa&lt;sub&gt;2&lt;/sub&gt;Cu&lt;sub&gt;3&lt;/sub&gt;O&lt;sub&gt;7&lt;/sub&gt; as a superconductor, 1987 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_the_maximum_superconducting_temperature&quot;&gt;warmest temperature of superconduction&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + ==== Events causing moderate, robust discontinuities ==== + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;HMS Warrior, 1860 (discontinuity in both &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_ship_size&quot;&gt;Royal Navy ship tonnage and Royal Navy ship displacement&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Eiffel Tower, 1889 (discontinuity in &lt;a href=&quot;/doku.php?id=ai_timelines:historic_trends_in_structure_heights&quot;&gt;tallest existing freestanding structure height&lt;/a&gt;, and in other height trends non-robustly) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Fairey Delta 2, 1956 (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_flight_airspeed_records&quot;&gt;airspeed&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Pellets shot into space, 1957, measured after one day of travel (discontinuity in &lt;a href=&quot;/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_altitude&quot;&gt;altitude achieved by man-made means&lt;/a&gt;)&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-1-202&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-1-202&quot; title=&quot;This was the first of various altitude records where the object continues to gain distance from Earth’s surface continuously over a long period. One could choose to treat these in different ways, and get different size of discontinuity numbers. Strictly, all altitude increases are continuous, so we are anyway implicitly looking at something like discontinuities in heights reached within some period. We somewhat arbitrarily chose to measure altitudes roughly every year, including one day in for the pellets, the only one where the very start mattered. &quot;&gt;&lt;sup&gt;1&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Burj Khalifa, 2009 (discontinuity in &lt;a href=&quot;/doku.php?id=ai_timelines:historic_trends_in_structure_heights&quot;&gt;height of tallest building ever&lt;/a&gt;) + &lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + ==== Non-robust discontinuities ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;a href=&quot;https://docs.google.com/spreadsheets/d/1iMIZ57Ka9-ZYednnGeonC-NqwGC7dKiHN9S-TAxfVdQ/edit#gid=1994197408&amp;amp;range=B3:B90&quot;&gt;This spreadsheet&lt;/a&gt; details all discontinuities found, as of April 2020.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol class=&quot;easy-footnotes-wrapper&quot;&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-1-202&quot;&gt;&lt;/span&gt;This was the first of various altitude records where the object continues to gain distance from Earth’s surface continuously over a long period. One could choose to treat these in different ways, and get different size of discontinuity numbers. Strictly, all altitude increases are continuous, so we are anyway implicitly looking at something like discontinuities in heights reached within some period. We somewhat arbitrarily chose to measure altitudes roughly every year, including one day in for the pellets, the only one where the very start mattered. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-1-202&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + </pre>

2022-10-19T02:01:59+00:00 2022-10-19T02:01:59+00:00 https://wiki.aiimpacts.org/featured_articles/evidence_against_current_methods_leading_to_human_level_artificial_intelligence?rev=1666144919&do=diff Anonymous anonymous@undisclosed.example.com <pre> @@ -10,9 +10,9 @@ ==== Clarifications ==== - We take ‘current methods’ to mean techniques for engineering artificial intelligence that are already known, involving no “qualitatively new ideas”.((“It now seems possible that we could build ‘prosaic’ AGI, which can replicate human behavior but doesn’t involve qualitatively new ideas about ‘how intelligence works’”— Christiano, Paul. [[https://openreview.net/pdf?id=H18WqugAb|Prosaic AI Alignment]]. 2017. Medium. Accessed August 13 2019. https://ai-alignment.com/prosaic-ai-control-b959644d79c2.))&lt;/a&gt;&lt;/span&gt; We have not precisely defined ‘current methods’. Many of the works we cite refer to currently //dominant// methods such as machine learning (especially deep learning) and reinforcement learning. + We take ‘current methods’ to mean techniques for engineering artificial intelligence that are already known, involving no “qualitatively new ideas”.((“It now seems possible that we could build ‘prosaic’ AGI, which can replicate human behavior but doesn’t involve qualitatively new ideas about ‘how intelligence works’”— Christiano, Paul. [[https://openreview.net/pdf?id=H18WqugAb|Prosaic AI Alignment]]. 2017. Medium. Accessed August 13 2019. https://ai-alignment.com/prosaic-ai-control-b959644d79c2.)) We have not precisely defined ‘current methods’. Many of the works we cite refer to currently //dominant// methods such as machine learning (especially deep learning) and reinforcement learning. &lt;HTML&gt; &lt;p&gt;By human-level AI, we mean AI with a level of &lt;em&gt;performance&lt;/em&gt; comparable to humans. We have in mind the operationalization of ‘high-level machine intelligence’ from our &lt;a href=&quot;/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2016_expert_survey_on_progress_in_ai&quot;&gt;2016 expert survey on progress in AI&lt;/a&gt;: “Say we have ‘high-level machine intelligence’ when unaided machines can accomplish every task better and more cheaply than human workers.”&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-2-1938&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-2-1938&quot; title=&#039;Grace, Katja, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans. &amp;lt;a href=&quot;https://arxiv.org/abs/1705.08807&quot;&amp;gt;&amp;amp;#8220;When will AI exceed human performance? Evidence from AI experts.&amp;amp;#8221;&amp;lt;/a&amp;gt; Journal of Artificial Intelligence Research 62 (2018): 729-754.&#039;&gt;&lt;sup&gt;2&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; </pre>

<pre> @@ -10,9 +10,9 @@ ==== Clarifications ==== - We take ‘current methods’ to mean techniques for engineering artificial intelligence that are already known, involving no “qualitatively new ideas”.((“It now seems possible that we could build ‘prosaic’ AGI, which can replicate human behavior but doesn’t involve qualitatively new ideas about ‘how intelligence works’”— Christiano, Paul. [[https://openreview.net/pdf?id=H18WqugAb|Prosaic AI Alignment]]. 2017. Medium. Accessed August 13 2019. https://ai-alignment.com/prosaic-ai-control-b959644d79c2.))&lt;/a&gt;&lt;/span&gt; We have not precisely defined ‘current methods’. Many of the works we cite refer to currently //dominant// methods such as machine learning (especially deep learning) and reinforcement learning. + We take ‘current methods’ to mean techniques for engineering artificial intelligence that are already known, involving no “qualitatively new ideas”.((“It now seems possible that we could build ‘prosaic’ AGI, which can replicate human behavior but doesn’t involve qualitatively new ideas about ‘how intelligence works’”— Christiano, Paul. [[https://openreview.net/pdf?id=H18WqugAb|Prosaic AI Alignment]]. 2017. Medium. Accessed August 13 2019. https://ai-alignment.com/prosaic-ai-control-b959644d79c2.)) We have not precisely defined ‘current methods’. Many of the works we cite refer to currently //dominant// methods such as machine learning (especially deep learning) and reinforcement learning. &lt;HTML&gt; &lt;p&gt;By human-level AI, we mean AI with a level of &lt;em&gt;performance&lt;/em&gt; comparable to humans. We have in mind the operationalization of ‘high-level machine intelligence’ from our &lt;a href=&quot;/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2016_expert_survey_on_progress_in_ai&quot;&gt;2016 expert survey on progress in AI&lt;/a&gt;: “Say we have ‘high-level machine intelligence’ when unaided machines can accomplish every task better and more cheaply than human workers.”&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-2-1938&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-2-1938&quot; title=&#039;Grace, Katja, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans. &amp;lt;a href=&quot;https://arxiv.org/abs/1705.08807&quot;&amp;gt;&amp;amp;#8220;When will AI exceed human performance? Evidence from AI experts.&amp;amp;#8221;&amp;lt;/a&amp;gt; Journal of Artificial Intelligence Research 62 (2018): 729-754.&#039;&gt;&lt;sup&gt;2&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; </pre>

2022-09-21T07:37:41+00:00 2022-09-21T07:37:41+00:00 https://wiki.aiimpacts.org/featured_articles/evidence_on_good_forecasting_practices_from_the_good_judgment_project?rev=1663745861&do=diff Anonymous anonymous@undisclosed.example.com <pre> @@ -1 +1,443 @@ + ====== Evidence on good forecasting practices from the Good Judgment Project ====== + + // Published 07 February, 2019; last updated 17 July, 2020 // + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;According to experience and data from the Good Judgment Project, the following are associated with successful forecasting, in rough decreasing order of combined importance and confidence:&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Past performance in the same broad domain&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Making more predictions on the same question&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Deliberation time&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Collaboration on teams&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Intelligence&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Domain expertise&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Having taken a one-hour training module on these topics&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;‘Cognitive reflection’ test scores&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;‘Active open-mindedness’&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Aggregation of individual judgments&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Use of precise probabilistic predictions&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Use of ‘the outside view’&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;‘Fermi-izing’&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;‘Bayesian reasoning’&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Practice&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + + ===== Details ===== + + + ==== 1. 1. Process ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;The Good Judgment Project (GJP) was the winning team in IARPA’s 2011-2015 forecasting tournament. In the tournament, six teams assigned probabilistic answers to hundreds of questions about geopolitical events months to a year in the future. Each competing team used a different method for coming up with their guesses, so the tournament helps us to evaluate different forecasting methods.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;The GJP team, led by Philip Tetlock and Barbara Mellers, gathered thousands of online volunteers and had them answer the tournament questions. They then made their official forecasts by aggregating these answers. In the process, the team collected data about the patterns of performance in their volunteers, and experimented with aggregation methods and improvement interventions. For example, they ran an RCT to test the effect of a short training program on forecasting accuracy. They especially focused on identifying and making use of the most successful two percent of forecasters, dubbed ‘superforecasters’.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Tetlock’s book&lt;/span&gt; &lt;i&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Superforecasting&lt;/span&gt;&lt;/i&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;describes this process and Tetlock’s resulting understanding of how to forecast well.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ==== 1.2. Correlates of successful forecasting ==== + + + === 1.2.1. Past performance === + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Roughly 70% of the superforecasters maintained their status from one year to the next &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-1-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-1-1283&quot; title=&quot; &amp;lt;i&amp;gt;Superforecasting &amp;lt;/i&amp;gt;p104 &quot;&gt;&lt;sup&gt;1&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Across all the forecasters, the correlation between performance in one year and performance in the next year was 0.65 &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-2-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-2-1283&quot; title=&quot; &amp;lt;i&amp;gt;Superforecasting &amp;lt;/i&amp;gt;p104 &quot;&gt;&lt;sup&gt;2&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;These high correlations are particularly impressive because the forecasters were online volunteers; presumably substantial variance year-to-year came from forecasters throttling down their engagement due to fatigue or changing life circumstances &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-3-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-3-1283&quot; title=&quot; Technically the forecasters were paid, up to $250 per season. (&amp;lt;i&amp;gt;Superforecasting &amp;lt;/i&amp;gt;p72) However their payments did not depend on how accurate they were or how much effort they put in, beyond the minimum.&amp;amp;nbsp;&quot;&gt;&lt;sup&gt;3&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + === 1.2.2. Behavioral and dispositional variables === + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Table 2  depicts the correlations between measured variables amongst GJP’s volunteers in the first two years of the tournament &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-4-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-4-1283&quot; title=&#039; The table is from &amp;lt;a href=&quot;https://www.apa.org/pubs/journals/releases/xap-0000040.pdf&quot;&amp;gt;Mellers &amp;lt;i&amp;gt;et al&amp;lt;/i&amp;gt; 2015&amp;lt;/a&amp;gt;. “Del time” is deliberation time.&amp;amp;nbsp;&#039;&gt;&lt;sup&gt;4&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt; Each is described in more detail below.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;The first column shows the relationship between each variable and standardized&lt;/span&gt; &lt;a href=&quot;https://en.wikipedia.org/wiki/Brier_score&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Brier score&lt;/span&gt;&lt;/a&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;, which is a measure of inaccuracy: higher Brier scores mean less accuracy, so negative correlations are good. “Ravens” is an IQ test; “Del time” is deliberation time, and “teams” is whether or not the forecaster was assigned to a team. “Actively open-minded thinking” is an attempt to measure “the tendency to evaluate arguments and evidence without undue bias from one’s own prior beliefs—and with recognition of the fallibility of one’s judgment.” &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-5-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-5-1283&quot; title=&#039; “Nonetheless, as we saw in the structural model, and confirm here, the best model uses dispositional, situational, and behavioral variables. The combination produced a multiple correlation of .64.” This is from &amp;lt;a href=&quot;https://www.apa.org/pubs/journals/releases/xap-0000040.pdf&quot;&amp;gt;Mellers &amp;lt;i&amp;gt;et al&amp;lt;/i&amp;gt; 2015&amp;lt;/a&amp;gt;.&amp;amp;nbsp;&#039;&gt;&lt;sup&gt;5&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;The authors conducted various statistical analyses to explore the relationships between these variables. They computed a structural equation model to predict a forecaster’s accuracy:&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Yellow ovals are latent dispositional variables, yellow rectangles are observed dispositional variables, pink rectangles are experimentally manipulated situational variables, and green rectangles are observed behavioral variables. This model has a &lt;a href=&quot;https://en.wikipedia.org/wiki/Multiple_correlation&quot;&gt;multiple correlation&lt;/a&gt; of 0.64.&lt;/span&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-6-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-6-1283&quot; title=&#039; This is from &amp;lt;a href=&quot;https://www.apa.org/pubs/journals/releases/xap-0000040.pdf&quot;&amp;gt;Mellers &amp;lt;i&amp;gt;et al&amp;lt;/i&amp;gt; 2015&amp;lt;/a&amp;gt;.&amp;amp;nbsp;&#039;&gt;&lt;sup&gt;6&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;As these data indicate, domain knowledge, intelligence, active open-mindedness, and working in teams each contribute substantially to accuracy. We can also conclude that effort helps, because deliberation time and number of predictions made per question (“belief updating”) both improved accuracy. Finally, training also helps. This is especially surprising because the training module lasted only an hour and its effects persisted for at least a year. The module included content about probabilistic reasoning, using the outside view, avoiding biases, and more.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ==== 1.3. Aggregation algorithms ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;GJP made their official predictions by aggregating and extremizing the predictions of their volunteers.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;The aggregation algorithm was elitist, meaning that it weighted more heavily people who were better on various metrics.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-7-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-7-1283&quot; title=&#039; On the &amp;lt;a href=&quot;https://goodjudgment.com/science.html&quot;&amp;gt;webpage&amp;lt;/a&amp;gt;, it says forecasters with better track-records and those who update more frequently get weighted more. In &amp;lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii&quot;&amp;gt;these slides,&amp;amp;nbsp;&amp;lt;/a&amp;gt;Tetlock describes the elitism differently: He says it gives weight to higher-IQ, more open-minded forecasters. &#039;&gt;&lt;sup&gt;7&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; The extremizing step pushes the aggregated judgment closer to 1 or 0, to make it more confident. The degree to which they extremize depends on how diverse and sophisticated the pool of forecasters is.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-8-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-8-1283&quot; title=&#039; The academic papers on this topic are &amp;lt;a href=&quot;https://www.sciencedirect.com/science/article/pii/S0169207013001635&quot;&amp;gt;Satopaa et al 2013&amp;lt;/a&amp;gt; and &amp;lt;a href=&quot;http://pubsonline.informs.org/doi/abs/10.1287/deca.2014.0293&quot;&amp;gt;Baron et al 2014&amp;lt;/a&amp;gt;. &#039;&gt;&lt;sup&gt;8&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; Whether extremizing is a good idea is still controversial.  &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-9-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-9-1283&quot; title=&quot; According to one expert I interviewed, more recent data suggests that the successes of the extremizing algorithm during the forecasting tournament were a fluke. After all, &amp;lt;i&amp;gt;a priori &amp;lt;/i&amp;gt;one would expect extremizing to lead to small improvements in accuracy most of the time, but big losses in accuracy some of the time.&amp;amp;nbsp;&quot;&gt;&lt;sup&gt;9&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;GJP beat all of the other teams.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;They consistently beat the control group—which was a forecast made by averaging ordinary forecasters—by more than 60%.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-10-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-10-1283&quot; title=&quot; &amp;lt;i&amp;gt;Superforecasting &amp;lt;/i&amp;gt;p18. &quot;&gt;&lt;sup&gt;10&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; They  also beat a prediction market inside the intelligence community—populated by professional analysts with access to classified information—by 25-30%. &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-11-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-11-1283&quot; title=&#039; This is from &amp;lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii&quot;&amp;gt;this seminar&amp;lt;/a&amp;gt;.&amp;amp;nbsp;&#039;&gt;&lt;sup&gt;11&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;That said, individual superforecasters did almost as well, so the elitism of the algorithm may account for a lot of its success.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-12-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-12-1283&quot; title=&#039; For example, in year 2 one superforecaster beat the extremizing algorithm. More generally, as discussed in &amp;lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii&quot;&amp;gt;this seminar&amp;lt;/a&amp;gt;, the aggregation algorithm produces the greatest improvement with ordinary forecasters; the superforecasters were good enough that it didn’t help much.&amp;amp;nbsp;&#039;&gt;&lt;sup&gt;12&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ==== 1.4. Outside View ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;The forecasters who received training were asked to record, for each prediction, which parts of the training they used to make it. Some parts of the training—e.g. “Post-mortem analysis”—were correlated with inaccuracy, but others—most notably “Comparison classes”—were correlated with accuracy.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-13-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-13-1283&quot; title=&#039; This is from &amp;lt;a href=&quot;http://journal.sjdm.org/16/16511/jdm16511.pdf&quot;&amp;gt;Chang et al 2016&amp;lt;/a&amp;gt;. The average brier score of answers tagged “comparison classes” was 0.17, while the next-best tag averaged 0.26.&#039;&gt;&lt;sup&gt;13&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;  ‘Comparison classes’ is another term for&lt;/span&gt; &lt;a href=&quot;https://en.wikipedia.org/wiki/Reference_class_forecasting&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;reference-class forecasting&lt;/span&gt;&lt;/a&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;, also known as ‘the outside view’. It is the method of assigning a probability by straightforward extrapolation from similar past situations and their outcomes.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ==== 1.5. Tetlock’s “Portrait of the modal superforecaster” ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;This subsection and those that follow will lay out some more qualitative results, things that Tetlock recommends on the basis of his research and interviews with superforecasters. Here is Tetlock’s “portrait of the modal superforecaster:” &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-14-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-14-1283&quot; title=&quot; &amp;lt;i&amp;gt;Superforecasting &amp;lt;/i&amp;gt;p191 &quot;&gt;&lt;sup&gt;14&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;Philosophic outlook:&lt;/b&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Cautious:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Nothing is certain.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Humble:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Reality is infinitely complex.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Nondeterministic:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Whatever happens is not meant to be and does not have to happen.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;Abilities &amp;amp; thinking styles:&lt;/b&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Actively open-minded:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Beliefs are hypotheses to be tested, not treasures to be protected.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Intelligent and knowledgeable, with a “Need for Cognition”:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Intellectually curious, enjoy puzzles and mental challenges.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Reflective:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Introspective and self-critical&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Numerate:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Comfortable with numbers&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;Methods of forecasting:&lt;/b&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Pragmatic:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Not wedded to any idea or agenda&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Analytical:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Capable of stepping back from the tip-of-your-nose perspective and considering other views&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Dragonfly-eyed:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Value diverse views and synthesize them into their own&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Probabilistic:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Judge using many grades of maybe&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Thoughtful updaters:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;When facts change, they change their minds&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Good intuitive psychologists:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Aware of the value of checking thinking for cognitive and emotional biases &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-15-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-15-1283&quot; title=&#039; There is experimental evidence that superforecasters are less prone to standard cognitive science biases than ordinary people. From &amp;lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-iv&quot;&amp;gt;edge.org&amp;lt;/a&amp;gt;: &amp;lt;i&amp;gt;Mellers: &amp;lt;/i&amp;gt;“We have given them lots of Kahneman and Tversky-like problems to see if they fall prey to the same sorts of biases and errors. The answer is sort of, some of them do, but not as many. It’s not nearly as frequent as you see with the rest of us ordinary mortals. The other thing that’s interesting is they don’t make the kinds of mistakes that regular people make instead of the right answer. They do something that’s a little bit more thoughtful. They integrate base rates with case-specific information a little bit more.” &amp;amp;nbsp;&#039;&gt;&lt;sup&gt;15&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;Work ethic:&lt;/b&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Growth mindset:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Believe it’s possible to get better&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Grit:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Determined to keep at it however long it takes&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + + + ==== 1.6. Tetlock’s “Ten Commandments for Aspiring Superforecasters:” ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;This advice is given at the end of the book, and may make less sense to someone who hasn’t read the book. A full transcript of these commandments can be found&lt;/span&gt; &lt;a href=&quot;https://www.lesswrong.com/posts/dvYeSKDRd68GcrWoe/ten-commandments-for-aspiring-superforecasters&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;here&lt;/span&gt;&lt;/a&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;; this is a summary:&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(1) Triage:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Don’t waste time on questions that are “clocklike” where a rule of thumb can get you pretty close to the correct answer, or “cloudlike” where even fancy models can’t beat a dart-throwing chimp.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(2) Break seemingly intractable problems into tractable sub-problems:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;This is how Fermi estimation works. One related piece of advice is “be wary of accidentally substituting an easy question for a hard one,” e.g. substituting “Would Israel be willing to assassinate Yasser Arafat?” for “Will at least one of the tests for polonium in Arafat’s body turn up positive?”&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(3) Strike the right balance between inside and outside views:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;In particular,&lt;/span&gt; &lt;i&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;first&lt;/span&gt;&lt;/i&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;anchor with the outside view and&lt;/span&gt; &lt;i&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;then&lt;/span&gt;&lt;/i&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;adjust using the inside view.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(4) Strike the right balance between under- and overreacting to evidence:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Usually do many small updates, but occasionally do big updates when the situation calls for it. Remember to think about P(E|H)/P(E|~H); remember to avoid the base-rate fallacy. “Superforecasters aren’t perfect Bayesian predictors but they are much better than most of us.” &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-16-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-16-1283&quot; title=&quot;&amp;amp;nbsp;&amp;lt;i&amp;gt;Superforecasting &amp;lt;/i&amp;gt;p281 &quot;&gt;&lt;sup&gt;16&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(5) Look for the clashing causal forces at work in each problem:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;This is the “dragonfly eye perspective,” which is where you attempt to do a sort of mental wisdom of the crowds: Have tons of different causal models and aggregate their judgments. Use “Devil’s advocate” reasoning. If you think that P, try hard to convince yourself that not-P. You should find yourself saying “On the one hand… on the other hand… on the third hand…” a lot.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(6) Strive to distinguish as many degrees of doubt as the problem permits but no more.&lt;/b&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(7) Strike the right balance between under- and overconfidence, between prudence and decisiveness.&lt;/b&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(8) Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases.&lt;/b&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(9) Bring out the best in others and let others bring out the best in you.&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;The book spent a whole chapter on this, using the Wehrmacht as an extended case study on good team organization. One pervasive guiding principle is “Don’t tell people how to do things; tell them what you want accomplished, and they’ll surprise you with their ingenuity in doing it.” The other pervasive guiding principle is “Cultivate a culture in which people—even subordinates—are encouraged to dissent and give counterarguments.” &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-17-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-17-1283&quot; title=&quot;&amp;amp;nbsp;See e.g. page 284 of &amp;lt;i&amp;gt;Superforecasting&amp;lt;/i&amp;gt;, and the entirety of chapter 9. &quot;&gt;&lt;sup&gt;17&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(10) Master the error-balancing bicycle:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;This one should have been called practice, practice, practice. Tetlock says that reading the news and generating probabilities isn’t enough; you need to actually score your predictions so that you know how wrong you were.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(11) Don’t treat commandments as commandments:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Tetlock’s point here is simply that you should use your judgment about whether to follow a commandment or not; sometimes they should be overridden.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ==== 1.7. Recipe for Making Predictions ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Tetlock describes how superforecasters go about making their predictions.&lt;/span&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt; &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-18-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-18-1283&quot; title=&quot; See Chapter 5: “Ultimately, it’s not the number crunching power that counts. It’s how you use it. … You’ve Fermi-ized the question, consulted the outside view, and now, finally, you can consult the inside view … So you have an outside view and an inside view. Now they have to be merged. …” &quot;&gt;&lt;sup&gt;18&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; Here is an attempt at a summary:&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Sometimes a question can be answered more rigorously if it is first “Fermi-ized,” i.e. broken down into sub-questions for which more rigorous methods can be applied.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Next, use the outside view on the sub-questions (and/or the main question, if possible). You may then adjust your estimates using other considerations (‘the inside view’), but do this cautiously.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Seek out other perspectives, both on the sub-questions and on how to Fermi-ize the main question. You can also generate other perspectives yourself.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Repeat steps 1 – 3 until you hit diminishing returns.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Your final prediction should be based on an aggregation of various models, reference classes, other experts, etc.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + + + ==== 1.8. Bayesian reasoning &amp; precise probabilistic forecasts ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Humans normally express uncertainty with terms like “maybe” and “almost certainly” and “a significant chance.” Tetlock advocates for thinking and speaking in probabilities instead. He recounts many anecdotes of misunderstandings that might have been avoided this way. For example:&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;In 1961, when the CIA was planning to topple the Castro government by landing a small army of Cuban expatriates at the Bay of Pigs, President John F. Kennedy turned to the military for an unbiased assessment. The Joint Chiefs of Staff concluded that the plan had a “fair chance” of success. The man who wrote the words “fair chance” later said he had in mind odds of 3 to 1 against success. But Kennedy was never told precisely what “fair chance” meant and, not unreasonably, he took it to be a much more positive assessment. &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-19-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-19-1283&quot; title=&quot;&amp;amp;nbsp;&amp;lt;i&amp;gt;Superforecasting&amp;lt;/i&amp;gt; 44 &quot;&gt;&lt;sup&gt;19&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;This example hints at another advantage of probabilistic judgments: It’s harder to weasel out of them afterwards, and therefore easier to keep score. Keeping score is crucial for getting feedback from reality, which is crucial for building up expertise.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;A standard criticism of using probabilities is that they merely conceal uncertainty rather than quantify it—after all, the numbers you pick are themselves guesses. This may be true for people who haven’t practiced much, but it isn’t true for superforecasters, who are impressively well-calibrated and whose accuracy scores decrease when you round their predictions to the nearest 0.05. (EDIT: This should be 0.1)&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-20-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-20-1283&quot; title=&#039;&amp;amp;nbsp;The superforecasters had a calibration of 0.01, which means that the average difference between a probability they use and the true frequency of occurrence is 0.01. This is from &amp;lt;a href=&quot;https://www.researchgate.net/publication/277087515_Identifying_and_Cultivating_Superforecasters_as_a_Method_of_Improving_Probabilistic_Predictions&quot;&amp;gt;Mellers et al 2015&amp;lt;/a&amp;gt;. The fact about rounding their predictions is from &amp;lt;a href=&quot;https://academic.oup.com/isq/article-abstract/62/2/410/4944059?redirectedFrom=fulltext&quot;&amp;gt;Friedman et al 2018&amp;lt;/a&amp;gt;. EDIT: Seems I was wrong, thanks to this commenter for noticing.https://www.metaculus.com/questions/4166/the-lightning-round-tournament-comparing-metaculus-forecasters-to-infectious-disease-experts/#comment-28756&#039;&gt;&lt;sup&gt;20&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Bayesian reasoning is a natural next step once you are thinking and talking probabilities—it is the theoretical ideal in several important ways &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-21-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-21-1283&quot; title=&#039;&amp;amp;nbsp;For an excellent introduction to Bayesian reasoning and its theoretical foundations, see Strevens’ textbook-like &amp;lt;a href=&quot;http://www.strevens.org/bct/&quot;&amp;gt;lecture notes&amp;lt;/a&amp;gt;. Some of the facts summarized in this paragraph about Superforecasters and Bayesianism can be found on pages 169-172, 281, and 314 of &amp;lt;i&amp;gt;Superforecasting&amp;lt;/i&amp;gt;. &#039;&gt;&lt;sup&gt;21&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; &lt;/span&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;—and Tetlock’s experience and interviews with superforecasters seems to bear this out. Superforecasters seem to do many small updates, with occasional big updates, just as Bayesianism would predict. They recommend thinking in the Bayesian way, and often explicitly make Bayesian calculations. They are good at breaking down difficult questions into more manageable parts and chaining the probabilities together properly.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ===== 2. Discussion: Relevance to AI Forecasting ===== + + + ==== 2.1. Limitations ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;A major limitation is that the forecasts were mainly on geopolitical events only a few years in the future at most. (Uncertain geopolitical events seem to be somewhat predictable up to two years out but much more difficult to predict five years out.) &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-22-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-22-1283&quot; title=&quot;&amp;amp;nbsp;Tetlock admits that &amp;amp;#8220;there is no evidence that geopolitical or economic forecasters can predict anything ten years out beyond the excruciatingly obvious&amp;amp;#8230; These limits on predictability are the predictable results of the butterfly dynamics of nonlinear systems. In my EPJ research, the accuracy of expert predictions declined toward chance five years out.&amp;amp;#8221; (&amp;lt;i&amp;gt;Superforecasting&amp;lt;/i&amp;gt; p243) &quot;&gt;&lt;sup&gt;22&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt; So evidence from the GJP may not generalize to forecasting other types of events (e.g. technological progress and social  consequences) or events further in the future.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;That said, the forecasting best practices discovered by this research are not overtly specific to geopolitics or near-term events.  Also, geopolitical questions are diverse and accuracy on some was highly correlated with accuracy on others. &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-23-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-23-1283&quot; title=&#039;&amp;amp;nbsp;&amp;amp;#8220;There are several ways to look for individual consistency across questions. We sorted questions on the basis of response format (binary, multinomial, conditional, ordered), region (Eurzone, Latin America, China, etc.), and duration of question (short, medium, and long). We computed accuracy scores for each individual on each variable within each set (e.g., binary, multinomial, conditional, and ordered) and then constructed correlation matrices. For all three question types, correlations were positive&amp;amp;#8230; Then we conducted factor analyses. For each question type, a large proportion of the variance was captured by a single factor, consistent with the hypothesis that one underlying dimension was necessary to capture correlations among response formats, regions, and question duration.&amp;amp;#8221; From &amp;lt;a href=&quot;https://www.apa.org/pubs/journals/releases/xap-0000040.pdf&quot;&amp;gt;Mellers et al 2015&amp;lt;/a&amp;gt;. &#039;&gt;&lt;sup&gt;23&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Tetlock has ideas for how to handle longer-term, nebulous questions. He calls it “Bayesian Question Clustering.” (&lt;/span&gt;&lt;i&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Superforecasting&lt;/span&gt;&lt;/i&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;263) The idea is to take the question you really want to answer and look for more precise questions that are evidentially relevant to the question you care about. Tetlock intends to test the effectiveness of this idea in future research.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ==== 2.2 Value ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;The benefits of following these best practices (including identifying and aggregating the best forecasters) appear to be substantial: Superforecasters predicting events 300 days in the future were more accurate than regular forecasters predicting events 100 days in the future, and the GJP did even better. &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-24-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-24-1283&quot; title=&#039;&amp;amp;nbsp; &amp;lt;i&amp;gt;Superforecasting &amp;lt;/i&amp;gt;p94. Later, in the &amp;lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii&quot;&amp;gt;edge.org seminar&amp;lt;/a&amp;gt;, Tetlock says “In some other ROC curves—receiver operator characteristic curves, from signal detection theory—that Mark Steyvers at UCSD constructed—superforecasters could assign probabilities 400 days out about as well as regular people could about eighty days out.” The quote is accompanied by a &amp;lt;a href=&quot;https://www.edge.org/3rd_culture/Master%20Class%202015/Slide040.jpg&quot;&amp;gt;graph&amp;lt;/a&amp;gt;; unfortunately, it’s hard to interpret. &#039;&gt;&lt;sup&gt;24&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;If these benefits generalize beyond the short-term and beyond geopolitics—e.g. to long-term technological and societal development—then this research is highly useful to almost everyone. Even if the benefits do not generalize beyond the near-term, these best practices may still be well worth adopting. For example, it would be extremely useful to have 300 days of warning before strategically important AI milestones are reached, rather than 100.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ===== 3. Contributions ===== + + + &lt;HTML&gt; + &lt;p&gt;&lt;i&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Research, analysis, and writing were done by Daniel Kokotajlo. Katja Grace and Justis Mills contributed feedback and editing. Tegan McCaslin, Carl Shulman, and Jacob Lagerros contributed feedback.&lt;/span&gt;&lt;/i&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ===== 4. Footnotes ===== + + + + + &lt;HTML&gt; + &lt;ol class=&quot;easy-footnotes-wrapper&quot;&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-1-1283&quot;&gt;&lt;/span&gt; &lt;i&gt;Superforecasting&lt;/i&gt; p104 &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-1-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-2-1283&quot;&gt;&lt;/span&gt; &lt;i&gt;Superforecasting&lt;/i&gt; p104 &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-2-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-3-1283&quot;&gt;&lt;/span&gt; Technically the forecasters were paid, up to $250 per season. (&lt;i&gt;Superforecasting&lt;/i&gt; p72) However their payments did not depend on how accurate they were or how much effort they put in, beyond the minimum. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-3-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-4-1283&quot;&gt;&lt;/span&gt; The table is from &lt;a href=&quot;https://www.apa.org/pubs/journals/releases/xap-0000040.pdf&quot;&gt;Mellers &lt;i&gt;et al&lt;/i&gt; 2015&lt;/a&gt;. “Del time” is deliberation time. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-4-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-5-1283&quot;&gt;&lt;/span&gt; “Nonetheless, as we saw in the structural model, and confirm here, the best model uses dispositional, situational, and behavioral variables. The combination produced a multiple correlation of .64.” This is from &lt;a href=&quot;https://www.apa.org/pubs/journals/releases/xap-0000040.pdf&quot;&gt;Mellers &lt;i&gt;et al&lt;/i&gt; 2015&lt;/a&gt;. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-5-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-6-1283&quot;&gt;&lt;/span&gt; This is from &lt;a href=&quot;https://www.apa.org/pubs/journals/releases/xap-0000040.pdf&quot;&gt;Mellers &lt;i&gt;et al&lt;/i&gt; 2015&lt;/a&gt;. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-6-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-7-1283&quot;&gt;&lt;/span&gt; On the &lt;a href=&quot;https://goodjudgment.com/science.html&quot;&gt;webpage&lt;/a&gt;, it says forecasters with better track-records and those who update more frequently get weighted more. In &lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii&quot;&gt;these slides, &lt;/a&gt;Tetlock describes the elitism differently: He says it gives weight to higher-IQ, more open-minded forecasters. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-7-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-8-1283&quot;&gt;&lt;/span&gt; The academic papers on this topic are &lt;a href=&quot;https://www.sciencedirect.com/science/article/pii/S0169207013001635&quot;&gt;Satopaa et al 2013&lt;/a&gt; and &lt;a href=&quot;http://pubsonline.informs.org/doi/abs/10.1287/deca.2014.0293&quot;&gt;Baron et al 2014&lt;/a&gt;. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-8-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-9-1283&quot;&gt;&lt;/span&gt; According to one expert I interviewed, more recent data suggests that the successes of the extremizing algorithm during the forecasting tournament were a fluke. After all, &lt;i&gt;a priori&lt;/i&gt; one would expect extremizing to lead to small improvements in accuracy most of the time, but big losses in accuracy some of the time. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-9-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-10-1283&quot;&gt;&lt;/span&gt; &lt;i&gt;Superforecasting&lt;/i&gt; p18. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-10-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-11-1283&quot;&gt;&lt;/span&gt; This is from &lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii&quot;&gt;this seminar&lt;/a&gt;. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-11-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-12-1283&quot;&gt;&lt;/span&gt; For example, in year 2 one superforecaster beat the extremizing algorithm. More generally, as discussed in &lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii&quot;&gt;this seminar&lt;/a&gt;, the aggregation algorithm produces the greatest improvement with ordinary forecasters; the superforecasters were good enough that it didn’t help much. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-12-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-13-1283&quot;&gt;&lt;/span&gt; This is from &lt;a href=&quot;http://journal.sjdm.org/16/16511/jdm16511.pdf&quot;&gt;Chang et al 2016&lt;/a&gt;. The average brier score of answers tagged “comparison classes” was 0.17, while the next-best tag averaged 0.26.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-13-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-14-1283&quot;&gt;&lt;/span&gt; &lt;i&gt;Superforecasting&lt;/i&gt; p191 &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-14-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-15-1283&quot;&gt;&lt;/span&gt; There is experimental evidence that superforecasters are less prone to standard cognitive science biases than ordinary people. From &lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-iv&quot;&gt;edge.org&lt;/a&gt;: &lt;i&gt;Mellers:&lt;/i&gt; “We have given them lots of Kahneman and Tversky-like problems to see if they fall prey to the same sorts of biases and errors. The answer is sort of, some of them do, but not as many. It’s not nearly as frequent as you see with the rest of us ordinary mortals. The other thing that’s interesting is they don’t make the kinds of mistakes that regular people make instead of the right answer. They do something that’s a little bit more thoughtful. They integrate base rates with case-specific information a little bit more.”  &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-15-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-16-1283&quot;&gt;&lt;/span&gt; &lt;i&gt;Superforecasting&lt;/i&gt; p281 &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-16-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-17-1283&quot;&gt;&lt;/span&gt; See e.g. page 284 of &lt;i&gt;Superforecasting&lt;/i&gt;, and the entirety of chapter 9. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-17-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-18-1283&quot;&gt;&lt;/span&gt; See Chapter 5: “Ultimately, it’s not the number crunching power that counts. It’s how you use it. … You’ve Fermi-ized the question, consulted the outside view, and now, finally, you can consult the inside view … So you have an outside view and an inside view. Now they have to be merged. …” &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-18-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-19-1283&quot;&gt;&lt;/span&gt; &lt;i&gt;Superforecasting&lt;/i&gt; 44 &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-19-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-20-1283&quot;&gt;&lt;/span&gt; The superforecasters had a calibration of 0.01, which means that the average difference between a probability they use and the true frequency of occurrence is 0.01. This is from &lt;a href=&quot;https://www.researchgate.net/publication/277087515_Identifying_and_Cultivating_Superforecasters_as_a_Method_of_Improving_Probabilistic_Predictions&quot;&gt;Mellers et al 2015&lt;/a&gt;. The fact about rounding their predictions is from &lt;a href=&quot;https://academic.oup.com/isq/article-abstract/62/2/410/4944059?redirectedFrom=fulltext&quot;&gt;Friedman et al 2018&lt;/a&gt;. EDIT: Seems I was wrong, thanks to this commenter for noticing.https://www.metaculus.com/questions/4166/the-lightning-round-tournament-comparing-metaculus-forecasters-to-infectious-disease-experts/#comment-28756&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-20-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-21-1283&quot;&gt;&lt;/span&gt; For an excellent introduction to Bayesian reasoning and its theoretical foundations, see Strevens’ textbook-like &lt;a href=&quot;http://www.strevens.org/bct/&quot;&gt;lecture notes&lt;/a&gt;. Some of the facts summarized in this paragraph about Superforecasters and Bayesianism can be found on pages 169-172, 281, and 314 of &lt;i&gt;Superforecasting&lt;/i&gt;. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-21-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-22-1283&quot;&gt;&lt;/span&gt; Tetlock admits that “there is no evidence that geopolitical or economic forecasters can predict anything ten years out beyond the excruciatingly obvious… These limits on predictability are the predictable results of the butterfly dynamics of nonlinear systems. In my EPJ research, the accuracy of expert predictions declined toward chance five years out.” (&lt;i&gt;Superforecasting&lt;/i&gt; p243) &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-22-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-23-1283&quot;&gt;&lt;/span&gt; “There are several ways to look for individual consistency across questions. We sorted questions on the basis of response format (binary, multinomial, conditional, ordered), region (Eurzone, Latin America, China, etc.), and duration of question (short, medium, and long). We computed accuracy scores for each individual on each variable within each set (e.g., binary, multinomial, conditional, and ordered) and then constructed correlation matrices. For all three question types, correlations were positive… Then we conducted factor analyses. For each question type, a large proportion of the variance was captured by a single factor, consistent with the hypothesis that one underlying dimension was necessary to capture correlations among response formats, regions, and question duration.” From &lt;a href=&quot;https://www.apa.org/pubs/journals/releases/xap-0000040.pdf&quot;&gt;Mellers et al 2015&lt;/a&gt;. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-23-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-24-1283&quot;&gt;&lt;/span&gt;  &lt;i&gt;Superforecasting&lt;/i&gt; p94. Later, in the &lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii&quot;&gt;edge.org seminar&lt;/a&gt;, Tetlock says “In some other ROC curves—receiver operator characteristic curves, from signal detection theory—that Mark Steyvers at UCSD constructed—superforecasters could assign probabilities 400 days out about as well as regular people could about eighty days out.” The quote is accompanied by a &lt;a href=&quot;https://www.edge.org/3rd_culture/Master%20Class%202015/Slide040.jpg&quot;&gt;graph&lt;/a&gt;; unfortunately, it’s hard to interpret. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-24-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + </pre>

<pre> @@ -1 +1,443 @@ + ====== Evidence on good forecasting practices from the Good Judgment Project ====== + + // Published 07 February, 2019; last updated 17 July, 2020 // + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;According to experience and data from the Good Judgment Project, the following are associated with successful forecasting, in rough decreasing order of combined importance and confidence:&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Past performance in the same broad domain&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Making more predictions on the same question&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Deliberation time&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Collaboration on teams&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Intelligence&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Domain expertise&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Having taken a one-hour training module on these topics&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;‘Cognitive reflection’ test scores&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;‘Active open-mindedness’&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Aggregation of individual judgments&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Use of precise probabilistic predictions&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Use of ‘the outside view’&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;‘Fermi-izing’&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;‘Bayesian reasoning’&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Practice&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + + ===== Details ===== + + + ==== 1. 1. Process ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;The Good Judgment Project (GJP) was the winning team in IARPA’s 2011-2015 forecasting tournament. In the tournament, six teams assigned probabilistic answers to hundreds of questions about geopolitical events months to a year in the future. Each competing team used a different method for coming up with their guesses, so the tournament helps us to evaluate different forecasting methods.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;The GJP team, led by Philip Tetlock and Barbara Mellers, gathered thousands of online volunteers and had them answer the tournament questions. They then made their official forecasts by aggregating these answers. In the process, the team collected data about the patterns of performance in their volunteers, and experimented with aggregation methods and improvement interventions. For example, they ran an RCT to test the effect of a short training program on forecasting accuracy. They especially focused on identifying and making use of the most successful two percent of forecasters, dubbed ‘superforecasters’.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Tetlock’s book&lt;/span&gt; &lt;i&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Superforecasting&lt;/span&gt;&lt;/i&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;describes this process and Tetlock’s resulting understanding of how to forecast well.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ==== 1.2. Correlates of successful forecasting ==== + + + === 1.2.1. Past performance === + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Roughly 70% of the superforecasters maintained their status from one year to the next &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-1-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-1-1283&quot; title=&quot; &amp;lt;i&amp;gt;Superforecasting &amp;lt;/i&amp;gt;p104 &quot;&gt;&lt;sup&gt;1&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Across all the forecasters, the correlation between performance in one year and performance in the next year was 0.65 &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-2-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-2-1283&quot; title=&quot; &amp;lt;i&amp;gt;Superforecasting &amp;lt;/i&amp;gt;p104 &quot;&gt;&lt;sup&gt;2&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;These high correlations are particularly impressive because the forecasters were online volunteers; presumably substantial variance year-to-year came from forecasters throttling down their engagement due to fatigue or changing life circumstances &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-3-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-3-1283&quot; title=&quot; Technically the forecasters were paid, up to $250 per season. (&amp;lt;i&amp;gt;Superforecasting &amp;lt;/i&amp;gt;p72) However their payments did not depend on how accurate they were or how much effort they put in, beyond the minimum.&amp;amp;nbsp;&quot;&gt;&lt;sup&gt;3&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + === 1.2.2. Behavioral and dispositional variables === + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Table 2  depicts the correlations between measured variables amongst GJP’s volunteers in the first two years of the tournament &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-4-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-4-1283&quot; title=&#039; The table is from &amp;lt;a href=&quot;https://www.apa.org/pubs/journals/releases/xap-0000040.pdf&quot;&amp;gt;Mellers &amp;lt;i&amp;gt;et al&amp;lt;/i&amp;gt; 2015&amp;lt;/a&amp;gt;. “Del time” is deliberation time.&amp;amp;nbsp;&#039;&gt;&lt;sup&gt;4&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt; Each is described in more detail below.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;The first column shows the relationship between each variable and standardized&lt;/span&gt; &lt;a href=&quot;https://en.wikipedia.org/wiki/Brier_score&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Brier score&lt;/span&gt;&lt;/a&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;, which is a measure of inaccuracy: higher Brier scores mean less accuracy, so negative correlations are good. “Ravens” is an IQ test; “Del time” is deliberation time, and “teams” is whether or not the forecaster was assigned to a team. “Actively open-minded thinking” is an attempt to measure “the tendency to evaluate arguments and evidence without undue bias from one’s own prior beliefs—and with recognition of the fallibility of one’s judgment.” &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-5-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-5-1283&quot; title=&#039; “Nonetheless, as we saw in the structural model, and confirm here, the best model uses dispositional, situational, and behavioral variables. The combination produced a multiple correlation of .64.” This is from &amp;lt;a href=&quot;https://www.apa.org/pubs/journals/releases/xap-0000040.pdf&quot;&amp;gt;Mellers &amp;lt;i&amp;gt;et al&amp;lt;/i&amp;gt; 2015&amp;lt;/a&amp;gt;.&amp;amp;nbsp;&#039;&gt;&lt;sup&gt;5&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;The authors conducted various statistical analyses to explore the relationships between these variables. They computed a structural equation model to predict a forecaster’s accuracy:&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Yellow ovals are latent dispositional variables, yellow rectangles are observed dispositional variables, pink rectangles are experimentally manipulated situational variables, and green rectangles are observed behavioral variables. This model has a &lt;a href=&quot;https://en.wikipedia.org/wiki/Multiple_correlation&quot;&gt;multiple correlation&lt;/a&gt; of 0.64.&lt;/span&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-6-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-6-1283&quot; title=&#039; This is from &amp;lt;a href=&quot;https://www.apa.org/pubs/journals/releases/xap-0000040.pdf&quot;&amp;gt;Mellers &amp;lt;i&amp;gt;et al&amp;lt;/i&amp;gt; 2015&amp;lt;/a&amp;gt;.&amp;amp;nbsp;&#039;&gt;&lt;sup&gt;6&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;As these data indicate, domain knowledge, intelligence, active open-mindedness, and working in teams each contribute substantially to accuracy. We can also conclude that effort helps, because deliberation time and number of predictions made per question (“belief updating”) both improved accuracy. Finally, training also helps. This is especially surprising because the training module lasted only an hour and its effects persisted for at least a year. The module included content about probabilistic reasoning, using the outside view, avoiding biases, and more.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ==== 1.3. Aggregation algorithms ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;GJP made their official predictions by aggregating and extremizing the predictions of their volunteers.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;The aggregation algorithm was elitist, meaning that it weighted more heavily people who were better on various metrics.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-7-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-7-1283&quot; title=&#039; On the &amp;lt;a href=&quot;https://goodjudgment.com/science.html&quot;&amp;gt;webpage&amp;lt;/a&amp;gt;, it says forecasters with better track-records and those who update more frequently get weighted more. In &amp;lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii&quot;&amp;gt;these slides,&amp;amp;nbsp;&amp;lt;/a&amp;gt;Tetlock describes the elitism differently: He says it gives weight to higher-IQ, more open-minded forecasters. &#039;&gt;&lt;sup&gt;7&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; The extremizing step pushes the aggregated judgment closer to 1 or 0, to make it more confident. The degree to which they extremize depends on how diverse and sophisticated the pool of forecasters is.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-8-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-8-1283&quot; title=&#039; The academic papers on this topic are &amp;lt;a href=&quot;https://www.sciencedirect.com/science/article/pii/S0169207013001635&quot;&amp;gt;Satopaa et al 2013&amp;lt;/a&amp;gt; and &amp;lt;a href=&quot;http://pubsonline.informs.org/doi/abs/10.1287/deca.2014.0293&quot;&amp;gt;Baron et al 2014&amp;lt;/a&amp;gt;. &#039;&gt;&lt;sup&gt;8&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; Whether extremizing is a good idea is still controversial.  &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-9-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-9-1283&quot; title=&quot; According to one expert I interviewed, more recent data suggests that the successes of the extremizing algorithm during the forecasting tournament were a fluke. After all, &amp;lt;i&amp;gt;a priori &amp;lt;/i&amp;gt;one would expect extremizing to lead to small improvements in accuracy most of the time, but big losses in accuracy some of the time.&amp;amp;nbsp;&quot;&gt;&lt;sup&gt;9&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;GJP beat all of the other teams.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;They consistently beat the control group—which was a forecast made by averaging ordinary forecasters—by more than 60%.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-10-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-10-1283&quot; title=&quot; &amp;lt;i&amp;gt;Superforecasting &amp;lt;/i&amp;gt;p18. &quot;&gt;&lt;sup&gt;10&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; They  also beat a prediction market inside the intelligence community—populated by professional analysts with access to classified information—by 25-30%. &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-11-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-11-1283&quot; title=&#039; This is from &amp;lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii&quot;&amp;gt;this seminar&amp;lt;/a&amp;gt;.&amp;amp;nbsp;&#039;&gt;&lt;sup&gt;11&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;That said, individual superforecasters did almost as well, so the elitism of the algorithm may account for a lot of its success.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-12-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-12-1283&quot; title=&#039; For example, in year 2 one superforecaster beat the extremizing algorithm. More generally, as discussed in &amp;lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii&quot;&amp;gt;this seminar&amp;lt;/a&amp;gt;, the aggregation algorithm produces the greatest improvement with ordinary forecasters; the superforecasters were good enough that it didn’t help much.&amp;amp;nbsp;&#039;&gt;&lt;sup&gt;12&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ==== 1.4. Outside View ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;The forecasters who received training were asked to record, for each prediction, which parts of the training they used to make it. Some parts of the training—e.g. “Post-mortem analysis”—were correlated with inaccuracy, but others—most notably “Comparison classes”—were correlated with accuracy.&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-13-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-13-1283&quot; title=&#039; This is from &amp;lt;a href=&quot;http://journal.sjdm.org/16/16511/jdm16511.pdf&quot;&amp;gt;Chang et al 2016&amp;lt;/a&amp;gt;. The average brier score of answers tagged “comparison classes” was 0.17, while the next-best tag averaged 0.26.&#039;&gt;&lt;sup&gt;13&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;  ‘Comparison classes’ is another term for&lt;/span&gt; &lt;a href=&quot;https://en.wikipedia.org/wiki/Reference_class_forecasting&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;reference-class forecasting&lt;/span&gt;&lt;/a&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;, also known as ‘the outside view’. It is the method of assigning a probability by straightforward extrapolation from similar past situations and their outcomes.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ==== 1.5. Tetlock’s “Portrait of the modal superforecaster” ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;This subsection and those that follow will lay out some more qualitative results, things that Tetlock recommends on the basis of his research and interviews with superforecasters. Here is Tetlock’s “portrait of the modal superforecaster:” &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-14-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-14-1283&quot; title=&quot; &amp;lt;i&amp;gt;Superforecasting &amp;lt;/i&amp;gt;p191 &quot;&gt;&lt;sup&gt;14&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;Philosophic outlook:&lt;/b&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Cautious:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Nothing is certain.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Humble:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Reality is infinitely complex.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Nondeterministic:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Whatever happens is not meant to be and does not have to happen.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;Abilities &amp;amp; thinking styles:&lt;/b&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Actively open-minded:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Beliefs are hypotheses to be tested, not treasures to be protected.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Intelligent and knowledgeable, with a “Need for Cognition”:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Intellectually curious, enjoy puzzles and mental challenges.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Reflective:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Introspective and self-critical&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Numerate:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Comfortable with numbers&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;Methods of forecasting:&lt;/b&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Pragmatic:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Not wedded to any idea or agenda&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Analytical:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Capable of stepping back from the tip-of-your-nose perspective and considering other views&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Dragonfly-eyed:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Value diverse views and synthesize them into their own&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Probabilistic:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Judge using many grades of maybe&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Thoughtful updaters:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;When facts change, they change their minds&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Good intuitive psychologists:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Aware of the value of checking thinking for cognitive and emotional biases &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-15-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-15-1283&quot; title=&#039; There is experimental evidence that superforecasters are less prone to standard cognitive science biases than ordinary people. From &amp;lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-iv&quot;&amp;gt;edge.org&amp;lt;/a&amp;gt;: &amp;lt;i&amp;gt;Mellers: &amp;lt;/i&amp;gt;“We have given them lots of Kahneman and Tversky-like problems to see if they fall prey to the same sorts of biases and errors. The answer is sort of, some of them do, but not as many. It’s not nearly as frequent as you see with the rest of us ordinary mortals. The other thing that’s interesting is they don’t make the kinds of mistakes that regular people make instead of the right answer. They do something that’s a little bit more thoughtful. They integrate base rates with case-specific information a little bit more.” &amp;amp;nbsp;&#039;&gt;&lt;sup&gt;15&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;Work ethic:&lt;/b&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Growth mindset:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Believe it’s possible to get better&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;b&gt;Grit:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Determined to keep at it however long it takes&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + + + ==== 1.6. Tetlock’s “Ten Commandments for Aspiring Superforecasters:” ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;This advice is given at the end of the book, and may make less sense to someone who hasn’t read the book. A full transcript of these commandments can be found&lt;/span&gt; &lt;a href=&quot;https://www.lesswrong.com/posts/dvYeSKDRd68GcrWoe/ten-commandments-for-aspiring-superforecasters&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;here&lt;/span&gt;&lt;/a&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;; this is a summary:&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(1) Triage:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Don’t waste time on questions that are “clocklike” where a rule of thumb can get you pretty close to the correct answer, or “cloudlike” where even fancy models can’t beat a dart-throwing chimp.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(2) Break seemingly intractable problems into tractable sub-problems:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;This is how Fermi estimation works. One related piece of advice is “be wary of accidentally substituting an easy question for a hard one,” e.g. substituting “Would Israel be willing to assassinate Yasser Arafat?” for “Will at least one of the tests for polonium in Arafat’s body turn up positive?”&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(3) Strike the right balance between inside and outside views:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;In particular,&lt;/span&gt; &lt;i&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;first&lt;/span&gt;&lt;/i&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;anchor with the outside view and&lt;/span&gt; &lt;i&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;then&lt;/span&gt;&lt;/i&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;adjust using the inside view.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(4) Strike the right balance between under- and overreacting to evidence:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Usually do many small updates, but occasionally do big updates when the situation calls for it. Remember to think about P(E|H)/P(E|~H); remember to avoid the base-rate fallacy. “Superforecasters aren’t perfect Bayesian predictors but they are much better than most of us.” &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-16-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-16-1283&quot; title=&quot;&amp;amp;nbsp;&amp;lt;i&amp;gt;Superforecasting &amp;lt;/i&amp;gt;p281 &quot;&gt;&lt;sup&gt;16&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(5) Look for the clashing causal forces at work in each problem:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;This is the “dragonfly eye perspective,” which is where you attempt to do a sort of mental wisdom of the crowds: Have tons of different causal models and aggregate their judgments. Use “Devil’s advocate” reasoning. If you think that P, try hard to convince yourself that not-P. You should find yourself saying “On the one hand… on the other hand… on the third hand…” a lot.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(6) Strive to distinguish as many degrees of doubt as the problem permits but no more.&lt;/b&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(7) Strike the right balance between under- and overconfidence, between prudence and decisiveness.&lt;/b&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(8) Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases.&lt;/b&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(9) Bring out the best in others and let others bring out the best in you.&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;The book spent a whole chapter on this, using the Wehrmacht as an extended case study on good team organization. One pervasive guiding principle is “Don’t tell people how to do things; tell them what you want accomplished, and they’ll surprise you with their ingenuity in doing it.” The other pervasive guiding principle is “Cultivate a culture in which people—even subordinates—are encouraged to dissent and give counterarguments.” &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-17-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-17-1283&quot; title=&quot;&amp;amp;nbsp;See e.g. page 284 of &amp;lt;i&amp;gt;Superforecasting&amp;lt;/i&amp;gt;, and the entirety of chapter 9. &quot;&gt;&lt;sup&gt;17&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(10) Master the error-balancing bicycle:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;This one should have been called practice, practice, practice. Tetlock says that reading the news and generating probabilities isn’t enough; you need to actually score your predictions so that you know how wrong you were.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;b&gt;(11) Don’t treat commandments as commandments:&lt;/b&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;Tetlock’s point here is simply that you should use your judgment about whether to follow a commandment or not; sometimes they should be overridden.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ==== 1.7. Recipe for Making Predictions ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Tetlock describes how superforecasters go about making their predictions.&lt;/span&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt; &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-18-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-18-1283&quot; title=&quot; See Chapter 5: “Ultimately, it’s not the number crunching power that counts. It’s how you use it. … You’ve Fermi-ized the question, consulted the outside view, and now, finally, you can consult the inside view … So you have an outside view and an inside view. Now they have to be merged. …” &quot;&gt;&lt;sup&gt;18&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; Here is an attempt at a summary:&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Sometimes a question can be answered more rigorously if it is first “Fermi-ized,” i.e. broken down into sub-questions for which more rigorous methods can be applied.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Next, use the outside view on the sub-questions (and/or the main question, if possible). You may then adjust your estimates using other considerations (‘the inside view’), but do this cautiously.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Seek out other perspectives, both on the sub-questions and on how to Fermi-ize the main question. You can also generate other perspectives yourself.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Repeat steps 1 – 3 until you hit diminishing returns.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Your final prediction should be based on an aggregation of various models, reference classes, other experts, etc.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + + + ==== 1.8. Bayesian reasoning &amp; precise probabilistic forecasts ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Humans normally express uncertainty with terms like “maybe” and “almost certainly” and “a significant chance.” Tetlock advocates for thinking and speaking in probabilities instead. He recounts many anecdotes of misunderstandings that might have been avoided this way. For example:&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;In 1961, when the CIA was planning to topple the Castro government by landing a small army of Cuban expatriates at the Bay of Pigs, President John F. Kennedy turned to the military for an unbiased assessment. The Joint Chiefs of Staff concluded that the plan had a “fair chance” of success. The man who wrote the words “fair chance” later said he had in mind odds of 3 to 1 against success. But Kennedy was never told precisely what “fair chance” meant and, not unreasonably, he took it to be a much more positive assessment. &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-19-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-19-1283&quot; title=&quot;&amp;amp;nbsp;&amp;lt;i&amp;gt;Superforecasting&amp;lt;/i&amp;gt; 44 &quot;&gt;&lt;sup&gt;19&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;This example hints at another advantage of probabilistic judgments: It’s harder to weasel out of them afterwards, and therefore easier to keep score. Keeping score is crucial for getting feedback from reality, which is crucial for building up expertise.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;A standard criticism of using probabilities is that they merely conceal uncertainty rather than quantify it—after all, the numbers you pick are themselves guesses. This may be true for people who haven’t practiced much, but it isn’t true for superforecasters, who are impressively well-calibrated and whose accuracy scores decrease when you round their predictions to the nearest 0.05. (EDIT: This should be 0.1)&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-20-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-20-1283&quot; title=&#039;&amp;amp;nbsp;The superforecasters had a calibration of 0.01, which means that the average difference between a probability they use and the true frequency of occurrence is 0.01. This is from &amp;lt;a href=&quot;https://www.researchgate.net/publication/277087515_Identifying_and_Cultivating_Superforecasters_as_a_Method_of_Improving_Probabilistic_Predictions&quot;&amp;gt;Mellers et al 2015&amp;lt;/a&amp;gt;. The fact about rounding their predictions is from &amp;lt;a href=&quot;https://academic.oup.com/isq/article-abstract/62/2/410/4944059?redirectedFrom=fulltext&quot;&amp;gt;Friedman et al 2018&amp;lt;/a&amp;gt;. EDIT: Seems I was wrong, thanks to this commenter for noticing.https://www.metaculus.com/questions/4166/the-lightning-round-tournament-comparing-metaculus-forecasters-to-infectious-disease-experts/#comment-28756&#039;&gt;&lt;sup&gt;20&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Bayesian reasoning is a natural next step once you are thinking and talking probabilities—it is the theoretical ideal in several important ways &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-21-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-21-1283&quot; title=&#039;&amp;amp;nbsp;For an excellent introduction to Bayesian reasoning and its theoretical foundations, see Strevens’ textbook-like &amp;lt;a href=&quot;http://www.strevens.org/bct/&quot;&amp;gt;lecture notes&amp;lt;/a&amp;gt;. Some of the facts summarized in this paragraph about Superforecasters and Bayesianism can be found on pages 169-172, 281, and 314 of &amp;lt;i&amp;gt;Superforecasting&amp;lt;/i&amp;gt;. &#039;&gt;&lt;sup&gt;21&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; &lt;/span&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;—and Tetlock’s experience and interviews with superforecasters seems to bear this out. Superforecasters seem to do many small updates, with occasional big updates, just as Bayesianism would predict. They recommend thinking in the Bayesian way, and often explicitly make Bayesian calculations. They are good at breaking down difficult questions into more manageable parts and chaining the probabilities together properly.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ===== 2. Discussion: Relevance to AI Forecasting ===== + + + ==== 2.1. Limitations ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;A major limitation is that the forecasts were mainly on geopolitical events only a few years in the future at most. (Uncertain geopolitical events seem to be somewhat predictable up to two years out but much more difficult to predict five years out.) &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-22-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-22-1283&quot; title=&quot;&amp;amp;nbsp;Tetlock admits that &amp;amp;#8220;there is no evidence that geopolitical or economic forecasters can predict anything ten years out beyond the excruciatingly obvious&amp;amp;#8230; These limits on predictability are the predictable results of the butterfly dynamics of nonlinear systems. In my EPJ research, the accuracy of expert predictions declined toward chance five years out.&amp;amp;#8221; (&amp;lt;i&amp;gt;Superforecasting&amp;lt;/i&amp;gt; p243) &quot;&gt;&lt;sup&gt;22&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt; So evidence from the GJP may not generalize to forecasting other types of events (e.g. technological progress and social  consequences) or events further in the future.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;That said, the forecasting best practices discovered by this research are not overtly specific to geopolitics or near-term events.  Also, geopolitical questions are diverse and accuracy on some was highly correlated with accuracy on others. &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-23-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-23-1283&quot; title=&#039;&amp;amp;nbsp;&amp;amp;#8220;There are several ways to look for individual consistency across questions. We sorted questions on the basis of response format (binary, multinomial, conditional, ordered), region (Eurzone, Latin America, China, etc.), and duration of question (short, medium, and long). We computed accuracy scores for each individual on each variable within each set (e.g., binary, multinomial, conditional, and ordered) and then constructed correlation matrices. For all three question types, correlations were positive&amp;amp;#8230; Then we conducted factor analyses. For each question type, a large proportion of the variance was captured by a single factor, consistent with the hypothesis that one underlying dimension was necessary to capture correlations among response formats, regions, and question duration.&amp;amp;#8221; From &amp;lt;a href=&quot;https://www.apa.org/pubs/journals/releases/xap-0000040.pdf&quot;&amp;gt;Mellers et al 2015&amp;lt;/a&amp;gt;. &#039;&gt;&lt;sup&gt;23&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Tetlock has ideas for how to handle longer-term, nebulous questions. He calls it “Bayesian Question Clustering.” (&lt;/span&gt;&lt;i&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Superforecasting&lt;/span&gt;&lt;/i&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;263) The idea is to take the question you really want to answer and look for more precise questions that are evidentially relevant to the question you care about. Tetlock intends to test the effectiveness of this idea in future research.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ==== 2.2 Value ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;The benefits of following these best practices (including identifying and aggregating the best forecasters) appear to be substantial: Superforecasters predicting events 300 days in the future were more accurate than regular forecasters predicting events 100 days in the future, and the GJP did even better. &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-24-1283&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-24-1283&quot; title=&#039;&amp;amp;nbsp; &amp;lt;i&amp;gt;Superforecasting &amp;lt;/i&amp;gt;p94. Later, in the &amp;lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii&quot;&amp;gt;edge.org seminar&amp;lt;/a&amp;gt;, Tetlock says “In some other ROC curves—receiver operator characteristic curves, from signal detection theory—that Mark Steyvers at UCSD constructed—superforecasters could assign probabilities 400 days out about as well as regular people could about eighty days out.” The quote is accompanied by a &amp;lt;a href=&quot;https://www.edge.org/3rd_culture/Master%20Class%202015/Slide040.jpg&quot;&amp;gt;graph&amp;lt;/a&amp;gt;; unfortunately, it’s hard to interpret. &#039;&gt;&lt;sup&gt;24&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/span&gt; &lt;span style=&quot;font-weight: 400;&quot;&gt;If these benefits generalize beyond the short-term and beyond geopolitics—e.g. to long-term technological and societal development—then this research is highly useful to almost everyone. Even if the benefits do not generalize beyond the near-term, these best practices may still be well worth adopting. For example, it would be extremely useful to have 300 days of warning before strategically important AI milestones are reached, rather than 100.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ===== 3. Contributions ===== + + + &lt;HTML&gt; + &lt;p&gt;&lt;i&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Research, analysis, and writing were done by Daniel Kokotajlo. Katja Grace and Justis Mills contributed feedback and editing. Tegan McCaslin, Carl Shulman, and Jacob Lagerros contributed feedback.&lt;/span&gt;&lt;/i&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + + + ===== 4. Footnotes ===== + + + + + &lt;HTML&gt; + &lt;ol class=&quot;easy-footnotes-wrapper&quot;&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-1-1283&quot;&gt;&lt;/span&gt; &lt;i&gt;Superforecasting&lt;/i&gt; p104 &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-1-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-2-1283&quot;&gt;&lt;/span&gt; &lt;i&gt;Superforecasting&lt;/i&gt; p104 &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-2-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-3-1283&quot;&gt;&lt;/span&gt; Technically the forecasters were paid, up to $250 per season. (&lt;i&gt;Superforecasting&lt;/i&gt; p72) However their payments did not depend on how accurate they were or how much effort they put in, beyond the minimum. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-3-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-4-1283&quot;&gt;&lt;/span&gt; The table is from &lt;a href=&quot;https://www.apa.org/pubs/journals/releases/xap-0000040.pdf&quot;&gt;Mellers &lt;i&gt;et al&lt;/i&gt; 2015&lt;/a&gt;. “Del time” is deliberation time. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-4-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-5-1283&quot;&gt;&lt;/span&gt; “Nonetheless, as we saw in the structural model, and confirm here, the best model uses dispositional, situational, and behavioral variables. The combination produced a multiple correlation of .64.” This is from &lt;a href=&quot;https://www.apa.org/pubs/journals/releases/xap-0000040.pdf&quot;&gt;Mellers &lt;i&gt;et al&lt;/i&gt; 2015&lt;/a&gt;. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-5-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-6-1283&quot;&gt;&lt;/span&gt; This is from &lt;a href=&quot;https://www.apa.org/pubs/journals/releases/xap-0000040.pdf&quot;&gt;Mellers &lt;i&gt;et al&lt;/i&gt; 2015&lt;/a&gt;. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-6-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-7-1283&quot;&gt;&lt;/span&gt; On the &lt;a href=&quot;https://goodjudgment.com/science.html&quot;&gt;webpage&lt;/a&gt;, it says forecasters with better track-records and those who update more frequently get weighted more. In &lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii&quot;&gt;these slides, &lt;/a&gt;Tetlock describes the elitism differently: He says it gives weight to higher-IQ, more open-minded forecasters. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-7-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-8-1283&quot;&gt;&lt;/span&gt; The academic papers on this topic are &lt;a href=&quot;https://www.sciencedirect.com/science/article/pii/S0169207013001635&quot;&gt;Satopaa et al 2013&lt;/a&gt; and &lt;a href=&quot;http://pubsonline.informs.org/doi/abs/10.1287/deca.2014.0293&quot;&gt;Baron et al 2014&lt;/a&gt;. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-8-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-9-1283&quot;&gt;&lt;/span&gt; According to one expert I interviewed, more recent data suggests that the successes of the extremizing algorithm during the forecasting tournament were a fluke. After all, &lt;i&gt;a priori&lt;/i&gt; one would expect extremizing to lead to small improvements in accuracy most of the time, but big losses in accuracy some of the time. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-9-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-10-1283&quot;&gt;&lt;/span&gt; &lt;i&gt;Superforecasting&lt;/i&gt; p18. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-10-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-11-1283&quot;&gt;&lt;/span&gt; This is from &lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii&quot;&gt;this seminar&lt;/a&gt;. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-11-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-12-1283&quot;&gt;&lt;/span&gt; For example, in year 2 one superforecaster beat the extremizing algorithm. More generally, as discussed in &lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii&quot;&gt;this seminar&lt;/a&gt;, the aggregation algorithm produces the greatest improvement with ordinary forecasters; the superforecasters were good enough that it didn’t help much. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-12-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-13-1283&quot;&gt;&lt;/span&gt; This is from &lt;a href=&quot;http://journal.sjdm.org/16/16511/jdm16511.pdf&quot;&gt;Chang et al 2016&lt;/a&gt;. The average brier score of answers tagged “comparison classes” was 0.17, while the next-best tag averaged 0.26.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-13-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-14-1283&quot;&gt;&lt;/span&gt; &lt;i&gt;Superforecasting&lt;/i&gt; p191 &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-14-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-15-1283&quot;&gt;&lt;/span&gt; There is experimental evidence that superforecasters are less prone to standard cognitive science biases than ordinary people. From &lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-iv&quot;&gt;edge.org&lt;/a&gt;: &lt;i&gt;Mellers:&lt;/i&gt; “We have given them lots of Kahneman and Tversky-like problems to see if they fall prey to the same sorts of biases and errors. The answer is sort of, some of them do, but not as many. It’s not nearly as frequent as you see with the rest of us ordinary mortals. The other thing that’s interesting is they don’t make the kinds of mistakes that regular people make instead of the right answer. They do something that’s a little bit more thoughtful. They integrate base rates with case-specific information a little bit more.”  &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-15-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-16-1283&quot;&gt;&lt;/span&gt; &lt;i&gt;Superforecasting&lt;/i&gt; p281 &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-16-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-17-1283&quot;&gt;&lt;/span&gt; See e.g. page 284 of &lt;i&gt;Superforecasting&lt;/i&gt;, and the entirety of chapter 9. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-17-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-18-1283&quot;&gt;&lt;/span&gt; See Chapter 5: “Ultimately, it’s not the number crunching power that counts. It’s how you use it. … You’ve Fermi-ized the question, consulted the outside view, and now, finally, you can consult the inside view … So you have an outside view and an inside view. Now they have to be merged. …” &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-18-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-19-1283&quot;&gt;&lt;/span&gt; &lt;i&gt;Superforecasting&lt;/i&gt; 44 &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-19-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-20-1283&quot;&gt;&lt;/span&gt; The superforecasters had a calibration of 0.01, which means that the average difference between a probability they use and the true frequency of occurrence is 0.01. This is from &lt;a href=&quot;https://www.researchgate.net/publication/277087515_Identifying_and_Cultivating_Superforecasters_as_a_Method_of_Improving_Probabilistic_Predictions&quot;&gt;Mellers et al 2015&lt;/a&gt;. The fact about rounding their predictions is from &lt;a href=&quot;https://academic.oup.com/isq/article-abstract/62/2/410/4944059?redirectedFrom=fulltext&quot;&gt;Friedman et al 2018&lt;/a&gt;. EDIT: Seems I was wrong, thanks to this commenter for noticing.https://www.metaculus.com/questions/4166/the-lightning-round-tournament-comparing-metaculus-forecasters-to-infectious-disease-experts/#comment-28756&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-20-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-21-1283&quot;&gt;&lt;/span&gt; For an excellent introduction to Bayesian reasoning and its theoretical foundations, see Strevens’ textbook-like &lt;a href=&quot;http://www.strevens.org/bct/&quot;&gt;lecture notes&lt;/a&gt;. Some of the facts summarized in this paragraph about Superforecasters and Bayesianism can be found on pages 169-172, 281, and 314 of &lt;i&gt;Superforecasting&lt;/i&gt;. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-21-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-22-1283&quot;&gt;&lt;/span&gt; Tetlock admits that “there is no evidence that geopolitical or economic forecasters can predict anything ten years out beyond the excruciatingly obvious… These limits on predictability are the predictable results of the butterfly dynamics of nonlinear systems. In my EPJ research, the accuracy of expert predictions declined toward chance five years out.” (&lt;i&gt;Superforecasting&lt;/i&gt; p243) &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-22-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-23-1283&quot;&gt;&lt;/span&gt; “There are several ways to look for individual consistency across questions. We sorted questions on the basis of response format (binary, multinomial, conditional, ordered), region (Eurzone, Latin America, China, etc.), and duration of question (short, medium, and long). We computed accuracy scores for each individual on each variable within each set (e.g., binary, multinomial, conditional, and ordered) and then constructed correlation matrices. For all three question types, correlations were positive… Then we conducted factor analyses. For each question type, a large proportion of the variance was captured by a single factor, consistent with the hypothesis that one underlying dimension was necessary to capture correlations among response formats, regions, and question duration.” From &lt;a href=&quot;https://www.apa.org/pubs/journals/releases/xap-0000040.pdf&quot;&gt;Mellers et al 2015&lt;/a&gt;. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-23-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-24-1283&quot;&gt;&lt;/span&gt;  &lt;i&gt;Superforecasting&lt;/i&gt; p94. Later, in the &lt;a href=&quot;https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii&quot;&gt;edge.org seminar&lt;/a&gt;, Tetlock says “In some other ROC curves—receiver operator characteristic curves, from signal detection theory—that Mark Steyvers at UCSD constructed—superforecasters could assign probabilities 400 days out about as well as regular people could about eighty days out.” The quote is accompanied by a &lt;a href=&quot;https://www.edge.org/3rd_culture/Master%20Class%202015/Slide040.jpg&quot;&gt;graph&lt;/a&gt;; unfortunately, it’s hard to interpret. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-24-1283&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + </pre>

2022-11-05T03:55:18+00:00 2022-11-05T03:55:18+00:00 https://wiki.aiimpacts.org/featured_articles/fiction_relevant_to_ai_futurism?rev=1667620518&do=diff Anonymous anonymous@undisclosed.example.com <pre> @@ -60,9 +60,9 @@ ==== Collection ==== - The collection can also be seen full screen [[&quot;https://airtable.com/shr5EIpLNHB7o2q9Z/tblMVjRvMKVNkoZVg?backgroundColor=cyan&amp;amp;viewControls=on&quot;|here]] or as a table [[https://airtable.com/shrVnjq9U53R5nrxO&quot;|here]] + The collection can also be seen [[https://airtable.com/shr5EIpLNHB7o2q9Z/tblMVjRvMKVNkoZVg?backgroundColor=cyan&amp;amp;viewControls=on&quot;|here]] or as a table [[https://airtable.com/shrVnjq9U53R5nrxO&quot;|here]] </pre>

<pre> @@ -60,9 +60,9 @@ ==== Collection ==== - The collection can also be seen full screen [[&quot;https://airtable.com/shr5EIpLNHB7o2q9Z/tblMVjRvMKVNkoZVg?backgroundColor=cyan&amp;amp;viewControls=on&quot;|here]] or as a table [[https://airtable.com/shrVnjq9U53R5nrxO&quot;|here]] + The collection can also be seen [[https://airtable.com/shr5EIpLNHB7o2q9Z/tblMVjRvMKVNkoZVg?backgroundColor=cyan&amp;amp;viewControls=on&quot;|here]] or as a table [[https://airtable.com/shrVnjq9U53R5nrxO&quot;|here]] </pre>

2022-09-21T07:37:41+00:00 2022-09-21T07:37:41+00:00 https://wiki.aiimpacts.org/featured_articles/glossary_of_ai_risk_terminology_and_common_ai_terms?rev=1663745861&do=diff Anonymous anonymous@undisclosed.example.com <pre> @@ -1 +1,649 @@ + ====== Glossary of AI Risk Terminology and common AI terms ====== + + // Published 30 October, 2015; last updated 21 January, 2022 // + + + ===== Terms ===== + + + ==== A ==== + + + === AI timeline === + + + &lt;HTML&gt; + &lt;p&gt;An expectation about how much time will lapse before important AI events, especially the advent of &lt;em&gt;&lt;a href=&quot;/doku.php?id=clarifying_concepts:human-level_ai&quot;&gt;human-level AI&lt;/a&gt;&lt;/em&gt; or a similar milestone. The term can also refer to the actual periods of time (which are not yet known), rather than an expectation about them.&lt;/p&gt; + &lt;/HTML&gt; + + + === Artificial General Intelligence (also, AGI) === + + + &lt;HTML&gt; + &lt;p&gt;Skill at performing intellectual tasks across at least the range of variety that a human being is capable of. As opposed to skill at certain specific tasks (‘narrow’ AI). That is, synonymous with the more ambiguous &lt;em&gt;Human-Level AI&lt;/em&gt; for some meanings of the latter&lt;em&gt;.&lt;/em&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + === Artificial Intelligence (also, AI) === + + + &lt;HTML&gt; + &lt;p&gt;Behavior characteristic of human minds exhibited by man-made machines, and also the area of research focused on developing machines with such behavior. Sometimes used informally to refer to &lt;em&gt;human-level AI&lt;/em&gt; or another strong form of AI not yet developed.&lt;/p&gt; + &lt;/HTML&gt; + + + === Associative value accretion === + + + &lt;HTML&gt; + &lt;p&gt;A hypothesized approach to value learning in which the AI acquires values using some machinery for synthesizing appropriate new values as it interacts with its environment, inspired by the way humans appear to acquire values (Bostrom 2014, p189-190)&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-1-358&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-1-358&quot; title=&quot;Bostrom, Nick. &amp;lt;em&amp;gt;Superintelligence: Paths, Dangers, Strategies&amp;lt;/em&amp;gt;. 1st edition. Oxford: Oxford University Press, 2014.&quot;&gt;&lt;sup&gt;1&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;.&lt;/p&gt; + &lt;/HTML&gt; + + + === Anthropic capture === + + + &lt;HTML&gt; + &lt;p&gt;A hypothesized control method in which the AI thinks it might be in a simulation, and so tries to behave in ways that will be rewarded by its simulators (Bostrom 2014 p134).&lt;/p&gt; + &lt;/HTML&gt; + + + === Anthropic reasoning === + + + &lt;HTML&gt; + &lt;p&gt;Reaching beliefs (posterior probabilities) over states of the world and your location in it, from priors over possible physical worlds (without your location specified) and evidence about your own situation. For an example where this is controversial, see &lt;a href=&quot;https://en.wikipedia.org/wiki/Sleeping_Beauty_problem&quot;&gt;The Sleeping Beauty Problem&lt;/a&gt;. For more on the topic and its relation to AI, see &lt;a href=&quot;https://meteuphoric.wordpress.com/anthropic-principles/&quot;&gt;here&lt;/a&gt;.&lt;/p&gt; + &lt;/HTML&gt; + + + === Augmentation === + + + &lt;HTML&gt; + &lt;p&gt;An approach to obtaining a superintelligence with desirable motives that consists of beginning with a creature with desirable motives (eg, a human), then making it smarter, instead of designing good motives from scratch (Bostrom 2014, p142).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== B ==== + + + === Backpropagation === + + + &lt;HTML&gt; + &lt;p&gt;A fast method of computing the derivative of cost with respect to different parameters in a network, allowing for training neural nets through gradient descent. See &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap2.html&quot;&gt;Neural Networks and Deep Learning&lt;/a&gt;&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-2-358&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-2-358&quot; title=&#039;Nielsen, Michael A. “Neural Networks and Deep Learning,” 2015. &amp;lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot;&amp;gt;http://neuralnetworksanddeeplearning.com&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;2&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; for a full explanation.&lt;/p&gt; + &lt;/HTML&gt; + + + === Boxing === + + + &lt;HTML&gt; + &lt;p&gt;A control method that consists of constructing the AI’s environment so as to minimize interaction between the AI and the outside world. (Bostrom 2014, p129).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== C ==== + + + === Capability control methods === + + + &lt;HTML&gt; + &lt;p&gt;Strategies for avoiding undesirable outcomes by limiting what an AI can do (Bostrom 2014, p129).&lt;/p&gt; + &lt;/HTML&gt; + + + === Cognitive enhancement === + + + &lt;HTML&gt; + &lt;p&gt;Improvements to an agent’s mental abilities.&lt;/p&gt; + &lt;/HTML&gt; + + + === Collective superintelligence === + + + &lt;HTML&gt; + &lt;p&gt;“A system composed of a large number of smaller intellects such that the system’s overall performance across many very general domains vastly outstrips that of any current cognitive system” (Bostrom 2014, p54).&lt;/p&gt; + &lt;/HTML&gt; + + + === Computation === + + + &lt;HTML&gt; + &lt;p&gt;A sequence of mechanical operations intended to shed light on something other than this mechanical process itself, through an established relationship between the process and the object of interest.&lt;/p&gt; + &lt;/HTML&gt; + + + === The common good principle === + + + &lt;HTML&gt; + &lt;p&gt;“Superintelligence should be developed only for the benefit of all of humanity and in the service of widely shared ethical ideals” (Bostrom 2014, p254).&lt;/p&gt; + &lt;/HTML&gt; + + + === Crucial consideration === + + + &lt;HTML&gt; + &lt;p&gt;An idea with the potential to change our views substantially, such as by reversing the sign of the desirability of important interventions.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== D ==== + + + === Decisive strategic advantage === + + + &lt;HTML&gt; + &lt;p&gt;Strategic superiority (by technology or other means) sufficient to enable an agent to unilaterally control most of the resources of the universe.&lt;/p&gt; + &lt;/HTML&gt; + + + === Direct specification === + + + &lt;HTML&gt; + &lt;p&gt;An approach to the control problem in which the programmers figure out what humans value, and code it into the AI (Bostrom 2014, p139-40).&lt;/p&gt; + &lt;/HTML&gt; + + + === Domesticity === + + + &lt;HTML&gt; + &lt;p&gt;An approach to the control problem in which the AI is given goals that limit the range of things it wants to interfere with (Bostrom 2014, p140-1).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== E ==== + + + === Emulation modulation === + + + &lt;HTML&gt; + &lt;p&gt;Starting with brain emulations with approximately normal human motivations (see ‘Augmentation’), and modifying their motivations using drugs or digital drug analogs.&lt;/p&gt; + &lt;/HTML&gt; + + + === Evolutionary selection approach to value learning === + + + &lt;HTML&gt; + &lt;p&gt;A hypothesized approach to the value learning problem which obtains an AI with desirable values by iterative selection, the same way evolutionary selection produced humans  (Bostrom 2014, p187-8).&lt;/p&gt; + &lt;/HTML&gt; + + + === Existential risk === + + + &lt;HTML&gt; + &lt;p&gt;Risk of an adverse outcome that would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential &lt;a href=&quot;http://www.nickbostrom.com/existential/risks.html&quot;&gt;(Bostrom 2002)&lt;/a&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + ==== F ==== + + + === Feature === + + + &lt;HTML&gt; + &lt;p&gt;A dimension in the vector space of activations in a single layer of a neural network (i.e. a neuron activation or linear combination of activations of different neurons)&lt;/p&gt; + &lt;/HTML&gt; + + + === First principal-agent problem === + + + &lt;HTML&gt; + &lt;p&gt;The well-known problem faced by a sponsor wanting an employee to fulfill their wishes (usually called ‘the principal agent problem’).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== G ==== + + + === Genie === + + + &lt;HTML&gt; + &lt;p&gt;An AI that carries out a high level command, then waits for another (Bostrom 2014, p148).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== H ==== + + + === Hardware overhang === + + + &lt;HTML&gt; + &lt;p&gt;A situation where large amounts of hardware being used for other purposes become available for AI, usually posited to occur when AI reaches human-level capabilities.&lt;/p&gt; + &lt;/HTML&gt; + + + === Human-level AI === + + + &lt;HTML&gt; + &lt;p&gt;An AI that matches human capabilities in virtually every domain of interest.  Note that this term is used ambiguously; see &lt;a href=&quot;/doku.php?id=clarifying_concepts:human-level_ai&quot;&gt;our page on human-level AI&lt;/a&gt;.  &lt;/p&gt; + &lt;/HTML&gt; + + + === Human-level hardware === + + + &lt;HTML&gt; + &lt;p&gt;Hardware that matches the information-processing ability of the human brain.&lt;/p&gt; + &lt;/HTML&gt; + + + === Human-level software === + + + &lt;HTML&gt; + &lt;p&gt;Software that matches the algorithmic efficiency of the human brain, for doing the tasks the human brain does.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== I ==== + + + === Impersonal perspective === + + + &lt;HTML&gt; + &lt;p&gt;The view that one should act in the best interests of everyone, including those who may be brought into existence by one’s choices (see Person-affecting perspective).&lt;/p&gt; + &lt;/HTML&gt; + + + === Incentive methods === + + + &lt;HTML&gt; + &lt;p&gt;Strategies for controlling an AI that consist of setting up the AI’s environment such that it is in the AI’s interest to cooperate. e.g. a social environment with punishment or social repercussions often achieves this for contemporary agents (Bostrom 2014, p131).&lt;/p&gt; + &lt;/HTML&gt; + + + === Incentive wrapping === + + + &lt;HTML&gt; + &lt;p&gt;Provisions in the goals given to an AI that allocate extra rewards to those who helped bring the AI about  (Bostrom 2014, p222-3).&lt;/p&gt; + &lt;/HTML&gt; + + + === Indirect normativity === + + + &lt;HTML&gt; + &lt;p&gt;An approach to the control problem in which we specify a way to specify what we value, instead of specifying what we value directly (Bostrom, p141-2).&lt;/p&gt; + &lt;/HTML&gt; + + + === Instrumental convergence thesis === + + + &lt;HTML&gt; + &lt;p&gt;We can identify ‘convergent instrumental values’. That is, subgoals that are useful for a wide range of more fundamental goals, and in a wide range of situations (Bostrom 2014, p109).&lt;/p&gt; + &lt;/HTML&gt; + + + === Intelligence explosion === + + + &lt;HTML&gt; + &lt;p&gt;A hypothesized event in which an AI rapidly improves from ‘relatively modest’ to superhuman level (usually imagined to be as a result of recursive self-improvement).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== M ==== + + + === Macrostructural development accelerator === + + + &lt;HTML&gt; + &lt;p&gt;An imagined lever used in thought experiments which slows the large scale features of history (e.g. technological change, geopolitical dynamics) while leaving the small scale features the same.&lt;/p&gt; + &lt;/HTML&gt; + + + === Mind crime === + + + &lt;HTML&gt; + &lt;p&gt;The mistreatment of morally relevant computations.&lt;/p&gt; + &lt;/HTML&gt; + + + === Moore’s Law === + + + &lt;HTML&gt; + &lt;p&gt;Any of several different consistent, many-decade patterns of exponential improvement that have been observed in digital technologies. The classic version concerns the number of transistors in a dense integrated circuit, which was observed to be doubling around every year when the ‘law’ was formulated in &lt;a href=&quot;https://en.wikipedia.org/wiki/Moore%27s_law&quot;&gt;1965&lt;/a&gt;. &lt;a href=&quot;/doku.php?id=featured_articles:glossary_of_ai_risk_terminology_and_common_ai_terms#Price-Performance_Moores_Law&quot;&gt;Price-Performance Moore’s Law&lt;/a&gt; is often relevant to AI forecasting.&lt;/p&gt; + &lt;/HTML&gt; + + + === Moral rightness (MR) AI === + + + &lt;HTML&gt; + &lt;p&gt;An AI which seeks to do what is morally right.&lt;/p&gt; + &lt;/HTML&gt; + + + === Motivational scaffolding === + + + &lt;HTML&gt; + &lt;p&gt;A hypothesized approach to value learning in which the seed AI is given simple goals, and these goals are replaced with more complex ones once it has developed sufficiently sophisticated representational structure (Bostrom 2014, p191-192).&lt;/p&gt; + &lt;/HTML&gt; + + + === Multipolar outcome === + + + &lt;HTML&gt; + &lt;p&gt;A situation after the arrival of superintelligence in which no single agent controls most of the resources.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== O ==== + + + === Optimization power === + + + &lt;HTML&gt; + &lt;p&gt;The strength of a process’s ability to improve systems.&lt;/p&gt; + &lt;/HTML&gt; + + + === Oracle === + + + &lt;HTML&gt; + &lt;p&gt;An AI that only answers questions (Bostrom 2014, p145).&lt;/p&gt; + &lt;/HTML&gt; + + + === Orthogonality thesis === + + + &lt;HTML&gt; + &lt;p&gt;Intelligence and final goals are orthogonal: more or less any level of intelligence could in principle be combined with more or less any final goal.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== P ==== + + + === Person-affecting perspective === + + + &lt;HTML&gt; + &lt;p&gt;The view that one should act in the best interests of everyone who already exists, or who will exist independent of one’s choices (see Impersonal perspective).&lt;/p&gt; + &lt;/HTML&gt; + + + === Perverse instantiation === + + + &lt;HTML&gt; + &lt;p&gt;A solution to a posed goal (eg, make humans smile) that is destructive in unforeseen ways (eg, paralyzing face muscles in the smiling position).&lt;/p&gt; + &lt;/HTML&gt; + + + === Price-Performance Moore’s Law === + + + &lt;HTML&gt; + &lt;p&gt;The &lt;a href=&quot;/doku.php?id=ai_timelines:trends_in_the_cost_of_computing&quot;&gt;observed pattern&lt;/a&gt; of relatively consistent, long term, exponential price decline for computation.&lt;/p&gt; + &lt;/HTML&gt; + + + === Principle of differential technological development === + + + &lt;HTML&gt; + &lt;p&gt;“Retard the development of dangerous and harmful technologies, especially the ones that raise the level of existential risk; and accelerate the development of beneficial technologies, especially those that reduce the existential risk posed by nature or by other technologies” (Bostrom 2014, p230).&lt;/p&gt; + &lt;/HTML&gt; + + + === Principle of epistemic deference === + + + &lt;HTML&gt; + &lt;p&gt;“A future superintelligence occupies an epistemically superior vantage point: it’s beliefs are (probably, on most topics) more likely than our to be true.  We should therefore defer to the superintelligence’s position whenever feasible” (Bostrom 2014, p226).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== Q ==== + + + === Quality superintelligence === + + + &lt;HTML&gt; + &lt;p&gt;“A system that is at least as fast as a human mind and vastly qualitatively smarter” (Bostrom 2014, p56).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== R ==== + + + === Recalcitrance === + + + &lt;HTML&gt; + &lt;p&gt;How difficult a system is to improve.&lt;/p&gt; + &lt;/HTML&gt; + + + === Recursive self-improvement === + + + &lt;HTML&gt; + &lt;p&gt;The envisaged process of AI (perhaps a seed AI) iteratively improving itself.&lt;/p&gt; + &lt;/HTML&gt; + + + === Reinforcement learning approach to value learning === + + + &lt;HTML&gt; + &lt;p&gt;A hypothesized approach to value learning in which the AI is rewarded for behaviors that more closely approximate human values (Bostrom 2014, p188-9).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== S ==== + + + === Second principal-agent problem === + + + &lt;HTML&gt; + &lt;p&gt;The emerging problem of a developer wanting their AI to fulfill their wishes.&lt;/p&gt; + &lt;/HTML&gt; + + + === Seed AI === + + + &lt;HTML&gt; + &lt;p&gt;A modest AI which can bootstrap into an impressive AI by improving its own architecture.&lt;/p&gt; + &lt;/HTML&gt; + + + === Singleton === + + + &lt;HTML&gt; + &lt;p&gt;An agent that is internally coordinated and has no opponents.&lt;/p&gt; + &lt;/HTML&gt; + + + === Sovereign === + + + &lt;HTML&gt; + &lt;p&gt;An AI that acts autonomously in the world, in pursuit of potentially long range objectives (Bostrom 2014, p148).&lt;/p&gt; + &lt;/HTML&gt; + + + === Speed superintelligence === + + + &lt;HTML&gt; + &lt;p&gt;“A system that can do all that a human intellect can do, but much faster” (Bostrom 2014, p53).&lt;/p&gt; + &lt;/HTML&gt; + + + === State risk === + + + &lt;HTML&gt; + &lt;p&gt;A risk that comes from being in a certain state, such that the amount of risk is a function of the time spent there. For example, the state of not having the technology to defend from asteroid impacts carries risk proportional to the time we spend in it.&lt;/p&gt; + &lt;/HTML&gt; + + + === Step risk === + + + &lt;HTML&gt; + &lt;p&gt;A risk that comes from making a transition. Here the amount of risk is not a simple function of how long the transition takes.  For example, traversing a minefield is not safer if done more quickly.&lt;/p&gt; + &lt;/HTML&gt; + + + === Stunting === + + + &lt;HTML&gt; + &lt;p&gt;A control method that consists of limiting the AI’s capabilities, for instance as by limiting the AI’s access to information (Bostrom 2014, p135).&lt;/p&gt; + &lt;/HTML&gt; + + + === Superintelligence === + + + &lt;HTML&gt; + &lt;p&gt;Any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest (Bostrom 2014, p22).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== T ==== + + + === Takeoff === + + + &lt;HTML&gt; + &lt;p&gt;The event of the emergence of a superintelligence, often characterized by its speed: ‘slow takeoff’ takes decades or centuries, ‘moderate takeoff’ takes months or years and ‘fast takeoff’ takes minutes to days.&lt;/p&gt; + &lt;/HTML&gt; + + + === Technological completion conjecture === + + + &lt;HTML&gt; + &lt;p&gt;If scientific and technological development efforts do not cease, then all important basic capabilities that could be obtained through some possible technology will be obtained (Bostrom 2014, p127).&lt;/p&gt; + &lt;/HTML&gt; + + + === Technology coupling === + + + &lt;HTML&gt; + &lt;p&gt;A predictable timing relationship between two technologies, such that hastening of the first technology will hasten the second, either because the second is a precursor or because it is a natural consequence (Bostrom 2014, p236-8) e.g. brain emulation is plausibly coupled to ‘neuromorphic’ AI, because the understanding required to emulate a brain might allow one to more quickly create an AI on similar principles.&lt;/p&gt; + &lt;/HTML&gt; + + + === Tool AI === + + + &lt;HTML&gt; + &lt;p&gt;An AI that is not ‘like an agent’, but like a more flexible and capable version of contemporary software. Most notably perhaps, it is not goal-directed (Bostrom 2014, p151).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== U ==== + + + === Utility function === + + + &lt;HTML&gt; + &lt;p&gt;A mapping from states of the world to real numbers (‘utilities’), describing an entity’s degree of preference for different states of the world. Given the choice between two lotteries, the entity prefers the lottery with the highest ‘expected utility’, which is to say, sum of utilities of possible states weighted by the probability of those states occurring.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== V ==== + + + === Value learning === + + + &lt;HTML&gt; + &lt;p&gt;An approach to the value loading problem in which the AI learns the values that humans want it to pursue (Bostrom 2014, p207).&lt;/p&gt; + &lt;/HTML&gt; + + + === Value loading problem === + + + &lt;HTML&gt; + &lt;p&gt;The problem of causing the AI to pursue human values (Bostrom 2014, p185).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== W ==== + + + === Wise-Singleton Sustainability Threshold === + + + &lt;HTML&gt; + &lt;p&gt;A capability set exceeds the wise-singleton threshold if and only if a patient and existential risk-savvy system with that capability set would, if it face no intelligent opposition or competition, be able to colonize and re-engineer a large part of the accessible universe (Bostrom 2014, p100).&lt;/p&gt; + &lt;/HTML&gt; + + + === Whole-brain emulation === + + + &lt;HTML&gt; + &lt;p&gt;Machine intelligence created by copying the computational structure of the human brain.&lt;/p&gt; + &lt;/HTML&gt; + + + === Word embedding === + + + &lt;HTML&gt; + &lt;p&gt;A mapping of words to high-dimensional vectors that has been trained to be useful in a word task such that the arrangement of words in the vector space is meaningful. For instance, words near one other in the vector-space are related, and similar relationships between different pairs of words correspond to similar vectors between them, so that e.g. if E(x) is the vector for the word ‘x’, then E(king) – E(queen) ≈ E(woman) – E(man). Word embeddings are explained in more detail &lt;a href=&quot;https://colah.github.io/posts/2014-07-NLP-RNNs-Representations/&quot;&gt;here&lt;/a&gt;.&lt;/p&gt; + &lt;/HTML&gt; + + + ===== Notes ===== + + + &lt;HTML&gt; + &lt;ol class=&quot;easy-footnotes-wrapper&quot;&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-1-358&quot;&gt;&lt;/span&gt;Bostrom, Nick. &lt;em&gt;Superintelligence: Paths, Dangers, Strategies&lt;/em&gt;. 1st edition. Oxford: Oxford University Press, 2014.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-1-358&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-2-358&quot;&gt;&lt;/span&gt;Nielsen, Michael A. “Neural Networks and Deep Learning,” 2015. &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot;&gt;http://neuralnetworksanddeeplearning.com&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-2-358&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + </pre>

<pre> @@ -1 +1,649 @@ + ====== Glossary of AI Risk Terminology and common AI terms ====== + + // Published 30 October, 2015; last updated 21 January, 2022 // + + + ===== Terms ===== + + + ==== A ==== + + + === AI timeline === + + + &lt;HTML&gt; + &lt;p&gt;An expectation about how much time will lapse before important AI events, especially the advent of &lt;em&gt;&lt;a href=&quot;/doku.php?id=clarifying_concepts:human-level_ai&quot;&gt;human-level AI&lt;/a&gt;&lt;/em&gt; or a similar milestone. The term can also refer to the actual periods of time (which are not yet known), rather than an expectation about them.&lt;/p&gt; + &lt;/HTML&gt; + + + === Artificial General Intelligence (also, AGI) === + + + &lt;HTML&gt; + &lt;p&gt;Skill at performing intellectual tasks across at least the range of variety that a human being is capable of. As opposed to skill at certain specific tasks (‘narrow’ AI). That is, synonymous with the more ambiguous &lt;em&gt;Human-Level AI&lt;/em&gt; for some meanings of the latter&lt;em&gt;.&lt;/em&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + === Artificial Intelligence (also, AI) === + + + &lt;HTML&gt; + &lt;p&gt;Behavior characteristic of human minds exhibited by man-made machines, and also the area of research focused on developing machines with such behavior. Sometimes used informally to refer to &lt;em&gt;human-level AI&lt;/em&gt; or another strong form of AI not yet developed.&lt;/p&gt; + &lt;/HTML&gt; + + + === Associative value accretion === + + + &lt;HTML&gt; + &lt;p&gt;A hypothesized approach to value learning in which the AI acquires values using some machinery for synthesizing appropriate new values as it interacts with its environment, inspired by the way humans appear to acquire values (Bostrom 2014, p189-190)&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-1-358&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-1-358&quot; title=&quot;Bostrom, Nick. &amp;lt;em&amp;gt;Superintelligence: Paths, Dangers, Strategies&amp;lt;/em&amp;gt;. 1st edition. Oxford: Oxford University Press, 2014.&quot;&gt;&lt;sup&gt;1&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;.&lt;/p&gt; + &lt;/HTML&gt; + + + === Anthropic capture === + + + &lt;HTML&gt; + &lt;p&gt;A hypothesized control method in which the AI thinks it might be in a simulation, and so tries to behave in ways that will be rewarded by its simulators (Bostrom 2014 p134).&lt;/p&gt; + &lt;/HTML&gt; + + + === Anthropic reasoning === + + + &lt;HTML&gt; + &lt;p&gt;Reaching beliefs (posterior probabilities) over states of the world and your location in it, from priors over possible physical worlds (without your location specified) and evidence about your own situation. For an example where this is controversial, see &lt;a href=&quot;https://en.wikipedia.org/wiki/Sleeping_Beauty_problem&quot;&gt;The Sleeping Beauty Problem&lt;/a&gt;. For more on the topic and its relation to AI, see &lt;a href=&quot;https://meteuphoric.wordpress.com/anthropic-principles/&quot;&gt;here&lt;/a&gt;.&lt;/p&gt; + &lt;/HTML&gt; + + + === Augmentation === + + + &lt;HTML&gt; + &lt;p&gt;An approach to obtaining a superintelligence with desirable motives that consists of beginning with a creature with desirable motives (eg, a human), then making it smarter, instead of designing good motives from scratch (Bostrom 2014, p142).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== B ==== + + + === Backpropagation === + + + &lt;HTML&gt; + &lt;p&gt;A fast method of computing the derivative of cost with respect to different parameters in a network, allowing for training neural nets through gradient descent. See &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/chap2.html&quot;&gt;Neural Networks and Deep Learning&lt;/a&gt;&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-2-358&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-2-358&quot; title=&#039;Nielsen, Michael A. “Neural Networks and Deep Learning,” 2015. &amp;lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot;&amp;gt;http://neuralnetworksanddeeplearning.com&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;2&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; for a full explanation.&lt;/p&gt; + &lt;/HTML&gt; + + + === Boxing === + + + &lt;HTML&gt; + &lt;p&gt;A control method that consists of constructing the AI’s environment so as to minimize interaction between the AI and the outside world. (Bostrom 2014, p129).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== C ==== + + + === Capability control methods === + + + &lt;HTML&gt; + &lt;p&gt;Strategies for avoiding undesirable outcomes by limiting what an AI can do (Bostrom 2014, p129).&lt;/p&gt; + &lt;/HTML&gt; + + + === Cognitive enhancement === + + + &lt;HTML&gt; + &lt;p&gt;Improvements to an agent’s mental abilities.&lt;/p&gt; + &lt;/HTML&gt; + + + === Collective superintelligence === + + + &lt;HTML&gt; + &lt;p&gt;“A system composed of a large number of smaller intellects such that the system’s overall performance across many very general domains vastly outstrips that of any current cognitive system” (Bostrom 2014, p54).&lt;/p&gt; + &lt;/HTML&gt; + + + === Computation === + + + &lt;HTML&gt; + &lt;p&gt;A sequence of mechanical operations intended to shed light on something other than this mechanical process itself, through an established relationship between the process and the object of interest.&lt;/p&gt; + &lt;/HTML&gt; + + + === The common good principle === + + + &lt;HTML&gt; + &lt;p&gt;“Superintelligence should be developed only for the benefit of all of humanity and in the service of widely shared ethical ideals” (Bostrom 2014, p254).&lt;/p&gt; + &lt;/HTML&gt; + + + === Crucial consideration === + + + &lt;HTML&gt; + &lt;p&gt;An idea with the potential to change our views substantially, such as by reversing the sign of the desirability of important interventions.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== D ==== + + + === Decisive strategic advantage === + + + &lt;HTML&gt; + &lt;p&gt;Strategic superiority (by technology or other means) sufficient to enable an agent to unilaterally control most of the resources of the universe.&lt;/p&gt; + &lt;/HTML&gt; + + + === Direct specification === + + + &lt;HTML&gt; + &lt;p&gt;An approach to the control problem in which the programmers figure out what humans value, and code it into the AI (Bostrom 2014, p139-40).&lt;/p&gt; + &lt;/HTML&gt; + + + === Domesticity === + + + &lt;HTML&gt; + &lt;p&gt;An approach to the control problem in which the AI is given goals that limit the range of things it wants to interfere with (Bostrom 2014, p140-1).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== E ==== + + + === Emulation modulation === + + + &lt;HTML&gt; + &lt;p&gt;Starting with brain emulations with approximately normal human motivations (see ‘Augmentation’), and modifying their motivations using drugs or digital drug analogs.&lt;/p&gt; + &lt;/HTML&gt; + + + === Evolutionary selection approach to value learning === + + + &lt;HTML&gt; + &lt;p&gt;A hypothesized approach to the value learning problem which obtains an AI with desirable values by iterative selection, the same way evolutionary selection produced humans  (Bostrom 2014, p187-8).&lt;/p&gt; + &lt;/HTML&gt; + + + === Existential risk === + + + &lt;HTML&gt; + &lt;p&gt;Risk of an adverse outcome that would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential &lt;a href=&quot;http://www.nickbostrom.com/existential/risks.html&quot;&gt;(Bostrom 2002)&lt;/a&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + ==== F ==== + + + === Feature === + + + &lt;HTML&gt; + &lt;p&gt;A dimension in the vector space of activations in a single layer of a neural network (i.e. a neuron activation or linear combination of activations of different neurons)&lt;/p&gt; + &lt;/HTML&gt; + + + === First principal-agent problem === + + + &lt;HTML&gt; + &lt;p&gt;The well-known problem faced by a sponsor wanting an employee to fulfill their wishes (usually called ‘the principal agent problem’).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== G ==== + + + === Genie === + + + &lt;HTML&gt; + &lt;p&gt;An AI that carries out a high level command, then waits for another (Bostrom 2014, p148).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== H ==== + + + === Hardware overhang === + + + &lt;HTML&gt; + &lt;p&gt;A situation where large amounts of hardware being used for other purposes become available for AI, usually posited to occur when AI reaches human-level capabilities.&lt;/p&gt; + &lt;/HTML&gt; + + + === Human-level AI === + + + &lt;HTML&gt; + &lt;p&gt;An AI that matches human capabilities in virtually every domain of interest.  Note that this term is used ambiguously; see &lt;a href=&quot;/doku.php?id=clarifying_concepts:human-level_ai&quot;&gt;our page on human-level AI&lt;/a&gt;.  &lt;/p&gt; + &lt;/HTML&gt; + + + === Human-level hardware === + + + &lt;HTML&gt; + &lt;p&gt;Hardware that matches the information-processing ability of the human brain.&lt;/p&gt; + &lt;/HTML&gt; + + + === Human-level software === + + + &lt;HTML&gt; + &lt;p&gt;Software that matches the algorithmic efficiency of the human brain, for doing the tasks the human brain does.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== I ==== + + + === Impersonal perspective === + + + &lt;HTML&gt; + &lt;p&gt;The view that one should act in the best interests of everyone, including those who may be brought into existence by one’s choices (see Person-affecting perspective).&lt;/p&gt; + &lt;/HTML&gt; + + + === Incentive methods === + + + &lt;HTML&gt; + &lt;p&gt;Strategies for controlling an AI that consist of setting up the AI’s environment such that it is in the AI’s interest to cooperate. e.g. a social environment with punishment or social repercussions often achieves this for contemporary agents (Bostrom 2014, p131).&lt;/p&gt; + &lt;/HTML&gt; + + + === Incentive wrapping === + + + &lt;HTML&gt; + &lt;p&gt;Provisions in the goals given to an AI that allocate extra rewards to those who helped bring the AI about  (Bostrom 2014, p222-3).&lt;/p&gt; + &lt;/HTML&gt; + + + === Indirect normativity === + + + &lt;HTML&gt; + &lt;p&gt;An approach to the control problem in which we specify a way to specify what we value, instead of specifying what we value directly (Bostrom, p141-2).&lt;/p&gt; + &lt;/HTML&gt; + + + === Instrumental convergence thesis === + + + &lt;HTML&gt; + &lt;p&gt;We can identify ‘convergent instrumental values’. That is, subgoals that are useful for a wide range of more fundamental goals, and in a wide range of situations (Bostrom 2014, p109).&lt;/p&gt; + &lt;/HTML&gt; + + + === Intelligence explosion === + + + &lt;HTML&gt; + &lt;p&gt;A hypothesized event in which an AI rapidly improves from ‘relatively modest’ to superhuman level (usually imagined to be as a result of recursive self-improvement).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== M ==== + + + === Macrostructural development accelerator === + + + &lt;HTML&gt; + &lt;p&gt;An imagined lever used in thought experiments which slows the large scale features of history (e.g. technological change, geopolitical dynamics) while leaving the small scale features the same.&lt;/p&gt; + &lt;/HTML&gt; + + + === Mind crime === + + + &lt;HTML&gt; + &lt;p&gt;The mistreatment of morally relevant computations.&lt;/p&gt; + &lt;/HTML&gt; + + + === Moore’s Law === + + + &lt;HTML&gt; + &lt;p&gt;Any of several different consistent, many-decade patterns of exponential improvement that have been observed in digital technologies. The classic version concerns the number of transistors in a dense integrated circuit, which was observed to be doubling around every year when the ‘law’ was formulated in &lt;a href=&quot;https://en.wikipedia.org/wiki/Moore%27s_law&quot;&gt;1965&lt;/a&gt;. &lt;a href=&quot;/doku.php?id=featured_articles:glossary_of_ai_risk_terminology_and_common_ai_terms#Price-Performance_Moores_Law&quot;&gt;Price-Performance Moore’s Law&lt;/a&gt; is often relevant to AI forecasting.&lt;/p&gt; + &lt;/HTML&gt; + + + === Moral rightness (MR) AI === + + + &lt;HTML&gt; + &lt;p&gt;An AI which seeks to do what is morally right.&lt;/p&gt; + &lt;/HTML&gt; + + + === Motivational scaffolding === + + + &lt;HTML&gt; + &lt;p&gt;A hypothesized approach to value learning in which the seed AI is given simple goals, and these goals are replaced with more complex ones once it has developed sufficiently sophisticated representational structure (Bostrom 2014, p191-192).&lt;/p&gt; + &lt;/HTML&gt; + + + === Multipolar outcome === + + + &lt;HTML&gt; + &lt;p&gt;A situation after the arrival of superintelligence in which no single agent controls most of the resources.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== O ==== + + + === Optimization power === + + + &lt;HTML&gt; + &lt;p&gt;The strength of a process’s ability to improve systems.&lt;/p&gt; + &lt;/HTML&gt; + + + === Oracle === + + + &lt;HTML&gt; + &lt;p&gt;An AI that only answers questions (Bostrom 2014, p145).&lt;/p&gt; + &lt;/HTML&gt; + + + === Orthogonality thesis === + + + &lt;HTML&gt; + &lt;p&gt;Intelligence and final goals are orthogonal: more or less any level of intelligence could in principle be combined with more or less any final goal.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== P ==== + + + === Person-affecting perspective === + + + &lt;HTML&gt; + &lt;p&gt;The view that one should act in the best interests of everyone who already exists, or who will exist independent of one’s choices (see Impersonal perspective).&lt;/p&gt; + &lt;/HTML&gt; + + + === Perverse instantiation === + + + &lt;HTML&gt; + &lt;p&gt;A solution to a posed goal (eg, make humans smile) that is destructive in unforeseen ways (eg, paralyzing face muscles in the smiling position).&lt;/p&gt; + &lt;/HTML&gt; + + + === Price-Performance Moore’s Law === + + + &lt;HTML&gt; + &lt;p&gt;The &lt;a href=&quot;/doku.php?id=ai_timelines:trends_in_the_cost_of_computing&quot;&gt;observed pattern&lt;/a&gt; of relatively consistent, long term, exponential price decline for computation.&lt;/p&gt; + &lt;/HTML&gt; + + + === Principle of differential technological development === + + + &lt;HTML&gt; + &lt;p&gt;“Retard the development of dangerous and harmful technologies, especially the ones that raise the level of existential risk; and accelerate the development of beneficial technologies, especially those that reduce the existential risk posed by nature or by other technologies” (Bostrom 2014, p230).&lt;/p&gt; + &lt;/HTML&gt; + + + === Principle of epistemic deference === + + + &lt;HTML&gt; + &lt;p&gt;“A future superintelligence occupies an epistemically superior vantage point: it’s beliefs are (probably, on most topics) more likely than our to be true.  We should therefore defer to the superintelligence’s position whenever feasible” (Bostrom 2014, p226).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== Q ==== + + + === Quality superintelligence === + + + &lt;HTML&gt; + &lt;p&gt;“A system that is at least as fast as a human mind and vastly qualitatively smarter” (Bostrom 2014, p56).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== R ==== + + + === Recalcitrance === + + + &lt;HTML&gt; + &lt;p&gt;How difficult a system is to improve.&lt;/p&gt; + &lt;/HTML&gt; + + + === Recursive self-improvement === + + + &lt;HTML&gt; + &lt;p&gt;The envisaged process of AI (perhaps a seed AI) iteratively improving itself.&lt;/p&gt; + &lt;/HTML&gt; + + + === Reinforcement learning approach to value learning === + + + &lt;HTML&gt; + &lt;p&gt;A hypothesized approach to value learning in which the AI is rewarded for behaviors that more closely approximate human values (Bostrom 2014, p188-9).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== S ==== + + + === Second principal-agent problem === + + + &lt;HTML&gt; + &lt;p&gt;The emerging problem of a developer wanting their AI to fulfill their wishes.&lt;/p&gt; + &lt;/HTML&gt; + + + === Seed AI === + + + &lt;HTML&gt; + &lt;p&gt;A modest AI which can bootstrap into an impressive AI by improving its own architecture.&lt;/p&gt; + &lt;/HTML&gt; + + + === Singleton === + + + &lt;HTML&gt; + &lt;p&gt;An agent that is internally coordinated and has no opponents.&lt;/p&gt; + &lt;/HTML&gt; + + + === Sovereign === + + + &lt;HTML&gt; + &lt;p&gt;An AI that acts autonomously in the world, in pursuit of potentially long range objectives (Bostrom 2014, p148).&lt;/p&gt; + &lt;/HTML&gt; + + + === Speed superintelligence === + + + &lt;HTML&gt; + &lt;p&gt;“A system that can do all that a human intellect can do, but much faster” (Bostrom 2014, p53).&lt;/p&gt; + &lt;/HTML&gt; + + + === State risk === + + + &lt;HTML&gt; + &lt;p&gt;A risk that comes from being in a certain state, such that the amount of risk is a function of the time spent there. For example, the state of not having the technology to defend from asteroid impacts carries risk proportional to the time we spend in it.&lt;/p&gt; + &lt;/HTML&gt; + + + === Step risk === + + + &lt;HTML&gt; + &lt;p&gt;A risk that comes from making a transition. Here the amount of risk is not a simple function of how long the transition takes.  For example, traversing a minefield is not safer if done more quickly.&lt;/p&gt; + &lt;/HTML&gt; + + + === Stunting === + + + &lt;HTML&gt; + &lt;p&gt;A control method that consists of limiting the AI’s capabilities, for instance as by limiting the AI’s access to information (Bostrom 2014, p135).&lt;/p&gt; + &lt;/HTML&gt; + + + === Superintelligence === + + + &lt;HTML&gt; + &lt;p&gt;Any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest (Bostrom 2014, p22).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== T ==== + + + === Takeoff === + + + &lt;HTML&gt; + &lt;p&gt;The event of the emergence of a superintelligence, often characterized by its speed: ‘slow takeoff’ takes decades or centuries, ‘moderate takeoff’ takes months or years and ‘fast takeoff’ takes minutes to days.&lt;/p&gt; + &lt;/HTML&gt; + + + === Technological completion conjecture === + + + &lt;HTML&gt; + &lt;p&gt;If scientific and technological development efforts do not cease, then all important basic capabilities that could be obtained through some possible technology will be obtained (Bostrom 2014, p127).&lt;/p&gt; + &lt;/HTML&gt; + + + === Technology coupling === + + + &lt;HTML&gt; + &lt;p&gt;A predictable timing relationship between two technologies, such that hastening of the first technology will hasten the second, either because the second is a precursor or because it is a natural consequence (Bostrom 2014, p236-8) e.g. brain emulation is plausibly coupled to ‘neuromorphic’ AI, because the understanding required to emulate a brain might allow one to more quickly create an AI on similar principles.&lt;/p&gt; + &lt;/HTML&gt; + + + === Tool AI === + + + &lt;HTML&gt; + &lt;p&gt;An AI that is not ‘like an agent’, but like a more flexible and capable version of contemporary software. Most notably perhaps, it is not goal-directed (Bostrom 2014, p151).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== U ==== + + + === Utility function === + + + &lt;HTML&gt; + &lt;p&gt;A mapping from states of the world to real numbers (‘utilities’), describing an entity’s degree of preference for different states of the world. Given the choice between two lotteries, the entity prefers the lottery with the highest ‘expected utility’, which is to say, sum of utilities of possible states weighted by the probability of those states occurring.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== V ==== + + + === Value learning === + + + &lt;HTML&gt; + &lt;p&gt;An approach to the value loading problem in which the AI learns the values that humans want it to pursue (Bostrom 2014, p207).&lt;/p&gt; + &lt;/HTML&gt; + + + === Value loading problem === + + + &lt;HTML&gt; + &lt;p&gt;The problem of causing the AI to pursue human values (Bostrom 2014, p185).&lt;/p&gt; + &lt;/HTML&gt; + + + ==== W ==== + + + === Wise-Singleton Sustainability Threshold === + + + &lt;HTML&gt; + &lt;p&gt;A capability set exceeds the wise-singleton threshold if and only if a patient and existential risk-savvy system with that capability set would, if it face no intelligent opposition or competition, be able to colonize and re-engineer a large part of the accessible universe (Bostrom 2014, p100).&lt;/p&gt; + &lt;/HTML&gt; + + + === Whole-brain emulation === + + + &lt;HTML&gt; + &lt;p&gt;Machine intelligence created by copying the computational structure of the human brain.&lt;/p&gt; + &lt;/HTML&gt; + + + === Word embedding === + + + &lt;HTML&gt; + &lt;p&gt;A mapping of words to high-dimensional vectors that has been trained to be useful in a word task such that the arrangement of words in the vector space is meaningful. For instance, words near one other in the vector-space are related, and similar relationships between different pairs of words correspond to similar vectors between them, so that e.g. if E(x) is the vector for the word ‘x’, then E(king) – E(queen) ≈ E(woman) – E(man). Word embeddings are explained in more detail &lt;a href=&quot;https://colah.github.io/posts/2014-07-NLP-RNNs-Representations/&quot;&gt;here&lt;/a&gt;.&lt;/p&gt; + &lt;/HTML&gt; + + + ===== Notes ===== + + + &lt;HTML&gt; + &lt;ol class=&quot;easy-footnotes-wrapper&quot;&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-1-358&quot;&gt;&lt;/span&gt;Bostrom, Nick. &lt;em&gt;Superintelligence: Paths, Dangers, Strategies&lt;/em&gt;. 1st edition. Oxford: Oxford University Press, 2014.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-1-358&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-2-358&quot;&gt;&lt;/span&gt;Nielsen, Michael A. “Neural Networks and Deep Learning,” 2015. &lt;a href=&quot;http://neuralnetworksanddeeplearning.com/&quot;&gt;http://neuralnetworksanddeeplearning.com&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-2-358&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + </pre>

2023-09-20T17:55:23+00:00 2023-09-20T17:55:23+00:00 https://wiki.aiimpacts.org/featured_articles/likelihood_of_discontinuous_progress_around_the_development_of_agi?rev=1695232523&do=diff Anonymous anonymous@undisclosed.example.com <pre> @@ -2,9 +2,9 @@ // Published 23 February, 2018; last updated 26 May, 2020 // &lt;HTML&gt; - &lt;p&gt;We aren’t convinced by any of the arguments we’ve seen to expect large discontinuity in AI progress above the extremely low base rate for all technologies. However this topic is controversial, and many thinkers on the topic disagree with us, so we consider this an open question.&lt;/p&gt; + &lt;p&gt;We do not know of seemingly compelling arguments to expect large discontinuity in AI progress above the extremely low base rate for all technologies. However this topic is controversial, and many thinkers on the topic disagree with us, so we consider this an open question.&lt;/p&gt; &lt;/HTML&gt; </pre>

<pre> @@ -2,9 +2,9 @@ // Published 23 February, 2018; last updated 26 May, 2020 // &lt;HTML&gt; - &lt;p&gt;We aren’t convinced by any of the arguments we’ve seen to expect large discontinuity in AI progress above the extremely low base rate for all technologies. However this topic is controversial, and many thinkers on the topic disagree with us, so we consider this an open question.&lt;/p&gt; + &lt;p&gt;We do not know of seemingly compelling arguments to expect large discontinuity in AI progress above the extremely low base rate for all technologies. However this topic is controversial, and many thinkers on the topic disagree with us, so we consider this an open question.&lt;/p&gt; &lt;/HTML&gt; </pre>

2022-09-21T07:37:41+00:00 2022-09-21T07:37:41+00:00 https://wiki.aiimpacts.org/featured_articles/list_of_multipolar_research_projects?rev=1663745861&do=diff Anonymous anonymous@undisclosed.example.com <pre> @@ -1 +1,268 @@ + ====== List of multipolar research projects ====== + + // Published 11 February, 2015; last updated 10 December, 2020 // + + &lt;HTML&gt; + &lt;p&gt;This list currently consists of research projects suggested at the &lt;a href=&quot;http://aiimpacts.org/event-multipolar-ai-workshop-with-robin-hanson/&quot; title=&quot;Event: Multipolar AI workshop with Robin Hanson&quot;&gt;Multipolar AI workshop&lt;/a&gt; we held on January 26 2015.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Relatively concrete projects are marked [concrete]. These are more likely to already include specific questions to answer and feasible methods to answer them with. Other ‘projects’ are more like open questions, or broad directions for inquiry.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Projects are divided into three sections:&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Paths to multipolar scenarios&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;What would happen in a multipolar scenario?&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Safety in a multipolar scenario&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Order is not otherwise relevant. The list is an inclusive collection of the topics suggested at the workshop, rather than a prioritized selection from a larger list.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Luke Muehlhauser’s &lt;a href=&quot;http://lukemuehlhauser.com/some-studies-which-could-improve-our-strategic-picture-of-superintelligence/&quot;&gt;list of ‘superintelligence strategy’ research questions&lt;/a&gt; contains further suggestions.&lt;/p&gt; + &lt;/HTML&gt; + + + + ===== List ===== + + + ==== Paths to multipolar scenarios ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.1 If we assume that AI software is similar to other software, what can we infer from observing contemporary software development? [concrete] &lt;/strong&gt;For instance, is progress in software performance generally smooth or jumpy? What is the distribution? What are typical degrees of concentration among developers? What are typical modes of competition? How far ahead does the leading team tend to be to their competitors? How often does the lead change? How much does a lead in a subsystem produce a lead overall? How much do non-software factors influence who has the lead? How likely is a large player like Google—with its pre-existing infrastructure—to be the frontrunner in a random new area that they decide to compete in?&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;A large part of this project would be collecting what is known about contemporary software development. This information would provide one view on how AI progress might plausibly unfold. Combined with several such views, this might inform predictions on issues like abruptness, competition and involved players.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.2 If the military is involved in AI development, how would that affect our predictions? [concrete] &lt;/strong&gt;This is a variation on 1.1, and would similarly involve a large component of reviewing the nature of contemporary military projects.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.3 If industry were to be largely responsible for AI development, how would that affect our predictions? [concrete] &lt;/strong&gt;This is a variation on 1.2, and would similarly involve a large component of reviewing the nature of contemporary industrial projects.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.4&lt;/strong&gt; &lt;strong&gt;If academia were to be largely responsible for AI development, how would that affect our predictions? [concrete] &lt;/strong&gt;This is a variation on 1.2, and would similarly involve a large component of reviewing the nature of contemporary academic projects.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.5 Survey AI experts on the likelihood of AI emerging in the military, business or academia, and on the likely size of a successful AI project.  [concrete]&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.6 Identify considerations that might tip us between multipolar and unipolar scenarios. &lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.7 To what extent will AGI progress be driven by developing significantly new ideas? &lt;/strong&gt;1.1 may bear on this. It could be approached in other ways, for instance asking AI researchers what they expect.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.8 Run prediction markets on near-term questions, such as rates of AI progress, which inform our long-run expectations. &lt;/strong&gt;[concrete] &lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.9 Collect past records of ‘lumpiness’ of AI success. [concrete]&lt;/strong&gt; That is, variation in progress over time. This would inform expectations of future lumpiness, and thus potential for single projects to gain a substantial advantage.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== What would happen in a multipolar scenario? ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.1 To what extent do values prevalent in the near-term affect the long run, in a competitive scenario? &lt;/strong&gt;One could consider the role of values over history so far, or examine the ways in which the role of values may change in the future. One could consider the degree of instrumental convergence between actors (e.g. firms) today, and ask how that affects long-term outcomes. One might also consider whether non-values mental features might become locked in in a way that produces similar outcomes to particular values being influential. e.g. priors or epistemological methods that make a particular religion more likely&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.2 What other factors in an initial scenario are likely to have long-lasting effects?&lt;/strong&gt; For instance social institutions, standards, and locations for cities.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.3 What would AI’s value in a multipolar scenario? &lt;/strong&gt;We can consider a range of factors that might influence AI values:&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The nature of the transition to AI&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Prevailing institutions&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The extentto which AI values become static, as compared to changing human values&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;What values do humans want AI’s to have&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Competitive dynamics&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;There is a common view that a multipolar scenario would be better in the long run than a hegemonic ‘unfriendly AI’. This project would inform that comparison.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.4 What are the prospects for human capital-holders? &lt;/strong&gt;In a simple model, humans who own capital might become very wealthy during a transition to AI. On a classical economic picture, this would be a critical way for humans to influence the future. Is this picture plausible? Evaluate the considerations.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;What are the implications of capital holders doing no intellectual work themselves?&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;strong&gt;[concrete]&lt;/strong&gt; What does the existing literature on principal-agent problems suggest about multipolar AI scenarios?&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;strong&gt;[concrete]&lt;/strong&gt; Could humans maintain investments for significant periods of their lives, if during that time aeons of subjective time passes for faster moving populations? (i.e. is it plausible to expect to hold assets through millions of years of human history?) Investigate this via data on past expropriations&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.5 Identify risks distinctive to a multipolar scenario, or which are much more serious in a multipolar scenario. &lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;For instance:&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Evolutionary dynamics bring an outcome that nobody desired initially&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The AIs are not well integrated into human society, and consequently cause or allow destruction to human society&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The AIs—integrated or not—have different values, and most of the resources end up being devoted to those values&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.6 Choose a specific multipolar scenario and try to predict its features in detail. [concrete] &lt;/strong&gt;Base this on the basic changes we know would occur (e.g. minds could be copied like software), and our best understanding of social science.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Specific instances:&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Brain emulations (Robin Hanson is working on this in an upcoming book)&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Brain emulations, without the assumption that software minds are opaque&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;One can buy maximally efficient software for anything you want; everything else is the same&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;AI is much like contemporary software (see 1.1).&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.7 How would multipolar AI change the nature and severity of violent conflict?&lt;/strong&gt; For instance, conflict between states.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.8 Investigate the potential for AI-enforced rights. &lt;/strong&gt;Think about how to enforce property rights in a multipolar scenario, given advanced artificial intelligence to do it with, and the opportunity to prepare ahead of time. &lt;span style=&quot;color: #444444; line-height: 1.7;&quot;&gt;Can you create programs that just enforce deals between two parties, but do nothing else? &lt;/span&gt;&lt;span style=&quot;color: #444444; line-height: 1.7;&quot;&gt;If you create AI with this stable motivational structure, possessed by many parties, how does this change the way that agents that interact? &lt;/span&gt;&lt;span style=&quot;color: #444444; line-height: 1.7;&quot;&gt;How could such a system be designed?&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.9 What is the future of democracy in such a scenario? &lt;/strong&gt;In a world where resources can rapidly and cheaply be turned into agents, the existing assignment of a vote per person may be destructive and unstable.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.10 How does the lumpiness of economic outcomes vary as a function of the lumpiness of origins?&lt;/strong&gt; For instance, if one team creates brain emulations years before others, would that group have and retain extreme influence?&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.11 What externalities can we foresee, in computer security? &lt;/strong&gt;That is, will people invest less (or more) in security than is socially optimal?&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.12 What externalities can we foresee in AI safety generally?&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.13 To what extent can artificial agents make more effective commitments, or more effectively monitor commitments, than humans?&lt;/strong&gt; How does this change competitive dynamics? What proofs of properties of one’s source code may be available in the future?&lt;/p&gt; + &lt;/HTML&gt; + + + ==== Safety in a multipolar scenario ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.1 Assess the applicability of general AI safety insights to multipolar scenarios. [concrete]&lt;/strong&gt; How useful are capability control methods, such as boxing, stunting, incentives, or tripwires in a multi-polar scenario? How useful are motivation selection methods, such as direct specification, domesticity, indirect normatively, augmentation in a multipolar scenario?&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.2 Would selective pressures strongly favor the existence of goal-directed agents, in a multipolar scenario where a variety of AI designs are feasible?&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.3 Develop a good model for the existing computer security phenomenon where nobody builds secure systems, though they can.&lt;/strong&gt; &lt;strong&gt;[concrete]&lt;/strong&gt; Model the long-run costs of secure and insecure systems, given distributions of attacker sophistication and possibility for incremental system improvement. Determine the likely situation various future scenarios, especially where computer security is particularly important.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.4 Do paradigms developed for nuclear security and biological weapons apply to AI in a multi-polar scenario? [concrete]&lt;/strong&gt; For instance, could similar control and detection systems be used?&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.5 What do the features of computer security systems tell us about how multipolar agents might compete?&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.8 What policies could help create more secure computer systems? &lt;/strong&gt;For instance, the onus being on owners of systems to secure them, rather than on potential attackers to avoid attacking.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.9 What innovations (either in AI or coinciding technologies) might reduce principal-agent problems?&lt;/strong&gt;&lt;strong&gt;&lt;span style=&quot;color: #444444; line-height: 1.7;&quot;&gt; &lt;/span&gt;&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.10 Apply ‘reliability theory’ to the problem of manufacturing trustworthy hardware. &lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.11 How can we transition in an economically viable way to hardware that we can trust is uncorrupted? &lt;/strong&gt;At present, we must assume that the hardware is uncorrupted upon purchase, but this may not be sufficient in the long run.&lt;/p&gt; + &lt;/HTML&gt; + + + + </pre>

<pre> @@ -1 +1,268 @@ + ====== List of multipolar research projects ====== + + // Published 11 February, 2015; last updated 10 December, 2020 // + + &lt;HTML&gt; + &lt;p&gt;This list currently consists of research projects suggested at the &lt;a href=&quot;http://aiimpacts.org/event-multipolar-ai-workshop-with-robin-hanson/&quot; title=&quot;Event: Multipolar AI workshop with Robin Hanson&quot;&gt;Multipolar AI workshop&lt;/a&gt; we held on January 26 2015.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Relatively concrete projects are marked [concrete]. These are more likely to already include specific questions to answer and feasible methods to answer them with. Other ‘projects’ are more like open questions, or broad directions for inquiry.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Projects are divided into three sections:&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Paths to multipolar scenarios&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;What would happen in a multipolar scenario?&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Safety in a multipolar scenario&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Order is not otherwise relevant. The list is an inclusive collection of the topics suggested at the workshop, rather than a prioritized selection from a larger list.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Luke Muehlhauser’s &lt;a href=&quot;http://lukemuehlhauser.com/some-studies-which-could-improve-our-strategic-picture-of-superintelligence/&quot;&gt;list of ‘superintelligence strategy’ research questions&lt;/a&gt; contains further suggestions.&lt;/p&gt; + &lt;/HTML&gt; + + + + ===== List ===== + + + ==== Paths to multipolar scenarios ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.1 If we assume that AI software is similar to other software, what can we infer from observing contemporary software development? [concrete] &lt;/strong&gt;For instance, is progress in software performance generally smooth or jumpy? What is the distribution? What are typical degrees of concentration among developers? What are typical modes of competition? How far ahead does the leading team tend to be to their competitors? How often does the lead change? How much does a lead in a subsystem produce a lead overall? How much do non-software factors influence who has the lead? How likely is a large player like Google—with its pre-existing infrastructure—to be the frontrunner in a random new area that they decide to compete in?&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;A large part of this project would be collecting what is known about contemporary software development. This information would provide one view on how AI progress might plausibly unfold. Combined with several such views, this might inform predictions on issues like abruptness, competition and involved players.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.2 If the military is involved in AI development, how would that affect our predictions? [concrete] &lt;/strong&gt;This is a variation on 1.1, and would similarly involve a large component of reviewing the nature of contemporary military projects.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.3 If industry were to be largely responsible for AI development, how would that affect our predictions? [concrete] &lt;/strong&gt;This is a variation on 1.2, and would similarly involve a large component of reviewing the nature of contemporary industrial projects.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.4&lt;/strong&gt; &lt;strong&gt;If academia were to be largely responsible for AI development, how would that affect our predictions? [concrete] &lt;/strong&gt;This is a variation on 1.2, and would similarly involve a large component of reviewing the nature of contemporary academic projects.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.5 Survey AI experts on the likelihood of AI emerging in the military, business or academia, and on the likely size of a successful AI project.  [concrete]&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.6 Identify considerations that might tip us between multipolar and unipolar scenarios. &lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.7 To what extent will AGI progress be driven by developing significantly new ideas? &lt;/strong&gt;1.1 may bear on this. It could be approached in other ways, for instance asking AI researchers what they expect.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.8 Run prediction markets on near-term questions, such as rates of AI progress, which inform our long-run expectations. &lt;/strong&gt;[concrete] &lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;1.9 Collect past records of ‘lumpiness’ of AI success. [concrete]&lt;/strong&gt; That is, variation in progress over time. This would inform expectations of future lumpiness, and thus potential for single projects to gain a substantial advantage.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== What would happen in a multipolar scenario? ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.1 To what extent do values prevalent in the near-term affect the long run, in a competitive scenario? &lt;/strong&gt;One could consider the role of values over history so far, or examine the ways in which the role of values may change in the future. One could consider the degree of instrumental convergence between actors (e.g. firms) today, and ask how that affects long-term outcomes. One might also consider whether non-values mental features might become locked in in a way that produces similar outcomes to particular values being influential. e.g. priors or epistemological methods that make a particular religion more likely&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.2 What other factors in an initial scenario are likely to have long-lasting effects?&lt;/strong&gt; For instance social institutions, standards, and locations for cities.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.3 What would AI’s value in a multipolar scenario? &lt;/strong&gt;We can consider a range of factors that might influence AI values:&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The nature of the transition to AI&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Prevailing institutions&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The extentto which AI values become static, as compared to changing human values&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;What values do humans want AI’s to have&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Competitive dynamics&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;There is a common view that a multipolar scenario would be better in the long run than a hegemonic ‘unfriendly AI’. This project would inform that comparison.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.4 What are the prospects for human capital-holders? &lt;/strong&gt;In a simple model, humans who own capital might become very wealthy during a transition to AI. On a classical economic picture, this would be a critical way for humans to influence the future. Is this picture plausible? Evaluate the considerations.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;What are the implications of capital holders doing no intellectual work themselves?&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;strong&gt;[concrete]&lt;/strong&gt; What does the existing literature on principal-agent problems suggest about multipolar AI scenarios?&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;strong&gt;[concrete]&lt;/strong&gt; Could humans maintain investments for significant periods of their lives, if during that time aeons of subjective time passes for faster moving populations? (i.e. is it plausible to expect to hold assets through millions of years of human history?) Investigate this via data on past expropriations&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.5 Identify risks distinctive to a multipolar scenario, or which are much more serious in a multipolar scenario. &lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;For instance:&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Evolutionary dynamics bring an outcome that nobody desired initially&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The AIs are not well integrated into human society, and consequently cause or allow destruction to human society&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;The AIs—integrated or not—have different values, and most of the resources end up being devoted to those values&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.6 Choose a specific multipolar scenario and try to predict its features in detail. [concrete] &lt;/strong&gt;Base this on the basic changes we know would occur (e.g. minds could be copied like software), and our best understanding of social science.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Specific instances:&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ol&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Brain emulations (Robin Hanson is working on this in an upcoming book)&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Brain emulations, without the assumption that software minds are opaque&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;One can buy maximally efficient software for anything you want; everything else is the same&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;AI is much like contemporary software (see 1.1).&lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.7 How would multipolar AI change the nature and severity of violent conflict?&lt;/strong&gt; For instance, conflict between states.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.8 Investigate the potential for AI-enforced rights. &lt;/strong&gt;Think about how to enforce property rights in a multipolar scenario, given advanced artificial intelligence to do it with, and the opportunity to prepare ahead of time. &lt;span style=&quot;color: #444444; line-height: 1.7;&quot;&gt;Can you create programs that just enforce deals between two parties, but do nothing else? &lt;/span&gt;&lt;span style=&quot;color: #444444; line-height: 1.7;&quot;&gt;If you create AI with this stable motivational structure, possessed by many parties, how does this change the way that agents that interact? &lt;/span&gt;&lt;span style=&quot;color: #444444; line-height: 1.7;&quot;&gt;How could such a system be designed?&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.9 What is the future of democracy in such a scenario? &lt;/strong&gt;In a world where resources can rapidly and cheaply be turned into agents, the existing assignment of a vote per person may be destructive and unstable.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.10 How does the lumpiness of economic outcomes vary as a function of the lumpiness of origins?&lt;/strong&gt; For instance, if one team creates brain emulations years before others, would that group have and retain extreme influence?&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.11 What externalities can we foresee, in computer security? &lt;/strong&gt;That is, will people invest less (or more) in security than is socially optimal?&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.12 What externalities can we foresee in AI safety generally?&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;2.13 To what extent can artificial agents make more effective commitments, or more effectively monitor commitments, than humans?&lt;/strong&gt; How does this change competitive dynamics? What proofs of properties of one’s source code may be available in the future?&lt;/p&gt; + &lt;/HTML&gt; + + + ==== Safety in a multipolar scenario ==== + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.1 Assess the applicability of general AI safety insights to multipolar scenarios. [concrete]&lt;/strong&gt; How useful are capability control methods, such as boxing, stunting, incentives, or tripwires in a multi-polar scenario? How useful are motivation selection methods, such as direct specification, domesticity, indirect normatively, augmentation in a multipolar scenario?&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.2 Would selective pressures strongly favor the existence of goal-directed agents, in a multipolar scenario where a variety of AI designs are feasible?&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.3 Develop a good model for the existing computer security phenomenon where nobody builds secure systems, though they can.&lt;/strong&gt; &lt;strong&gt;[concrete]&lt;/strong&gt; Model the long-run costs of secure and insecure systems, given distributions of attacker sophistication and possibility for incremental system improvement. Determine the likely situation various future scenarios, especially where computer security is particularly important.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.4 Do paradigms developed for nuclear security and biological weapons apply to AI in a multi-polar scenario? [concrete]&lt;/strong&gt; For instance, could similar control and detection systems be used?&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.5 What do the features of computer security systems tell us about how multipolar agents might compete?&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.8 What policies could help create more secure computer systems? &lt;/strong&gt;For instance, the onus being on owners of systems to secure them, rather than on potential attackers to avoid attacking.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.9 What innovations (either in AI or coinciding technologies) might reduce principal-agent problems?&lt;/strong&gt;&lt;strong&gt;&lt;span style=&quot;color: #444444; line-height: 1.7;&quot;&gt; &lt;/span&gt;&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.10 Apply ‘reliability theory’ to the problem of manufacturing trustworthy hardware. &lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;3.11 How can we transition in an economically viable way to hardware that we can trust is uncorrupted? &lt;/strong&gt;At present, we must assume that the hardware is uncorrupted upon purchase, but this may not be sufficient in the long run.&lt;/p&gt; + &lt;/HTML&gt; + + + + </pre>

2023-09-26T16:52:02+00:00 2023-09-26T16:52:02+00:00 https://wiki.aiimpacts.org/featured_articles/precedents_for_economic_n-year_doubling_before_4n-year_doubling?rev=1695747122&do=diff Anonymous anonymous@undisclosed.example.com <pre> @@ -70,9 +70,9 @@ &lt;HTML&gt; &lt;ul&gt; &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Between 4,000 BC and 3,000 BC, GWP doubled in 1,000 years, yet it had never before doubled in as few as 4000 years&lt;/div&gt;&lt;/li&gt; - &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Between 10,000 BC and 4,000 BC, GWP doubled in 6,000 years, yet there is no record of it doubling earlier in as few as 24,000 years. The records at that point are fairly sparse, so this is less clear, but it seems unlikely that there was a doubling in 24,000 years.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-5-2406&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-5-2406&quot; title=&quot;Toward the end of the period it took 15,000 years to grow by $0.6Bn, and growth of $1.8Bn would have been needed for a doubling. So assuming linear growth at the end-of-period rate, this would have taken around 45,000 years, whereas in if growth was speeding up, it should have taken longer.&quot;&gt;&lt;sup&gt;5&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; This appears to coincide with the beginning of agriculture, in around 9000BC.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-6-2406&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-6-2406&quot; title=&#039; Khan Academy. “The Dawn of Agriculture (Article).” Accessed April 14, 2020. &amp;lt;a href=&quot;https://www.khanacademy.org/humanities/world-history/world-history-beginnings/birth-agriculture-neolithic-revolution/a/where-did-agriculture-come-from&quot;&amp;gt;https://www.khanacademy.org/humanities/world-history/world-history-beginnings/birth-agriculture-neolithic-revolution/a/where-did-agriculture-come-from&amp;lt;/a&amp;gt;. &#039;&gt;&lt;sup&gt;6&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Between 10,000 BC and 4,000 BC, GWP doubled in 6,000 years, yet there is no record of it doubling earlier in as few as 24,000 years. The records at that point are fairly sparse, so this is less clear, but it seems unlikely that there was a doubling in 24,000 years.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-5-2406&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-5-2406&quot; title=&quot;Toward the end of the period it took 15,000 years to grow by \$0.6Bn, and growth of \$1.8Bn would have been needed for a doubling. So assuming linear growth at the end-of-period rate, this would have taken around 45,000 years, whereas in if growth was speeding up, it should have taken longer.&quot;&gt;&lt;sup&gt;5&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; This appears to coincide with the beginning of agriculture, in around 9000BC.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-6-2406&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-6-2406&quot; title=&#039; Khan Academy. “The Dawn of Agriculture (Article).” Accessed April 14, 2020. &amp;lt;a href=&quot;https://www.khanacademy.org/humanities/world-history/world-history-beginnings/birth-agriculture-neolithic-revolution/a/where-did-agriculture-come-from&quot;&amp;gt;https://www.khanacademy.org/humanities/world-history/world-history-beginnings/birth-agriculture-neolithic-revolution/a/where-did-agriculture-come-from&amp;lt;/a&amp;gt;. &#039;&gt;&lt;sup&gt;6&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; &lt;/ul&gt; &lt;/HTML&gt; @@ -107,9 +107,9 @@ &lt;li&gt;&lt;div class=&quot;li&quot;&gt; &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-4-2406&quot;&gt;&lt;/span&gt;[Note May 13 2020: This sheet is temporarily wrong.]&lt;s&gt;Instances coincide with &lt;a href=&quot;https://docs.google.com/spreadsheets/d/1Muz2ftyDUUewMTZPxYxeXF-uj6lBKYP-O3-IvdtHhCo/edit?ts=5e95f280#gid=0&amp;amp;range=G:G&quot;&gt;Column G in this spreadsheet&lt;/a&gt; giving a number higher than 4, when &lt;a href=&quot;https://docs.google.com/spreadsheets/d/1Muz2ftyDUUewMTZPxYxeXF-uj6lBKYP-O3-IvdtHhCo/edit?ts=5e95f280#gid=0&amp;amp;range=E2&quot;&gt;E2&lt;/a&gt; is set to 2.&lt;/s&gt;&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-4-2406&quot;&gt;&lt;/a&gt; &lt;/div&gt;&lt;/li&gt; &lt;li&gt;&lt;div class=&quot;li&quot;&gt; - &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-5-2406&quot;&gt;&lt;/span&gt;Toward the end of the period it took 15,000 years to grow by $0.6Bn, and growth of $1.8Bn would have been needed for a doubling. So assuming linear growth at the end-of-period rate, this would have taken around 45,000 years, whereas in if growth was speeding up, it should have taken longer.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-5-2406&quot;&gt;&lt;/a&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-5-2406&quot;&gt;&lt;/span&gt;Toward the end of the period it took 15,000 years to grow by \$0.6Bn, and growth of \$1.8Bn would have been needed for a doubling. So assuming linear growth at the end-of-period rate, this would have taken around 45,000 years, whereas in if growth was speeding up, it should have taken longer.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-5-2406&quot;&gt;&lt;/a&gt; &lt;/div&gt;&lt;/li&gt; &lt;li&gt;&lt;div class=&quot;li&quot;&gt; &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-6-2406&quot;&gt;&lt;/span&gt; Khan Academy. “The Dawn of Agriculture (Article).” Accessed April 14, 2020. &lt;a href=&quot;https://www.khanacademy.org/humanities/world-history/world-history-beginnings/birth-agriculture-neolithic-revolution/a/where-did-agriculture-come-from&quot;&gt;https://www.khanacademy.org/humanities/world-history/world-history-beginnings/birth-agriculture-neolithic-revolution/a/where-did-agriculture-come-from&lt;/a&gt;. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-6-2406&quot;&gt;&lt;/a&gt; &lt;/div&gt;&lt;/li&gt; </pre>

<pre> @@ -70,9 +70,9 @@ &lt;HTML&gt; &lt;ul&gt; &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Between 4,000 BC and 3,000 BC, GWP doubled in 1,000 years, yet it had never before doubled in as few as 4000 years&lt;/div&gt;&lt;/li&gt; - &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Between 10,000 BC and 4,000 BC, GWP doubled in 6,000 years, yet there is no record of it doubling earlier in as few as 24,000 years. The records at that point are fairly sparse, so this is less clear, but it seems unlikely that there was a doubling in 24,000 years.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-5-2406&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-5-2406&quot; title=&quot;Toward the end of the period it took 15,000 years to grow by $0.6Bn, and growth of $1.8Bn would have been needed for a doubling. So assuming linear growth at the end-of-period rate, this would have taken around 45,000 years, whereas in if growth was speeding up, it should have taken longer.&quot;&gt;&lt;sup&gt;5&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; This appears to coincide with the beginning of agriculture, in around 9000BC.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-6-2406&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-6-2406&quot; title=&#039; Khan Academy. “The Dawn of Agriculture (Article).” Accessed April 14, 2020. &amp;lt;a href=&quot;https://www.khanacademy.org/humanities/world-history/world-history-beginnings/birth-agriculture-neolithic-revolution/a/where-did-agriculture-come-from&quot;&amp;gt;https://www.khanacademy.org/humanities/world-history/world-history-beginnings/birth-agriculture-neolithic-revolution/a/where-did-agriculture-come-from&amp;lt;/a&amp;gt;. &#039;&gt;&lt;sup&gt;6&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Between 10,000 BC and 4,000 BC, GWP doubled in 6,000 years, yet there is no record of it doubling earlier in as few as 24,000 years. The records at that point are fairly sparse, so this is less clear, but it seems unlikely that there was a doubling in 24,000 years.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-5-2406&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-5-2406&quot; title=&quot;Toward the end of the period it took 15,000 years to grow by \$0.6Bn, and growth of \$1.8Bn would have been needed for a doubling. So assuming linear growth at the end-of-period rate, this would have taken around 45,000 years, whereas in if growth was speeding up, it should have taken longer.&quot;&gt;&lt;sup&gt;5&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; This appears to coincide with the beginning of agriculture, in around 9000BC.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-6-2406&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-6-2406&quot; title=&#039; Khan Academy. “The Dawn of Agriculture (Article).” Accessed April 14, 2020. &amp;lt;a href=&quot;https://www.khanacademy.org/humanities/world-history/world-history-beginnings/birth-agriculture-neolithic-revolution/a/where-did-agriculture-come-from&quot;&amp;gt;https://www.khanacademy.org/humanities/world-history/world-history-beginnings/birth-agriculture-neolithic-revolution/a/where-did-agriculture-come-from&amp;lt;/a&amp;gt;. &#039;&gt;&lt;sup&gt;6&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; &lt;/ul&gt; &lt;/HTML&gt; @@ -107,9 +107,9 @@ &lt;li&gt;&lt;div class=&quot;li&quot;&gt; &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-4-2406&quot;&gt;&lt;/span&gt;[Note May 13 2020: This sheet is temporarily wrong.]&lt;s&gt;Instances coincide with &lt;a href=&quot;https://docs.google.com/spreadsheets/d/1Muz2ftyDUUewMTZPxYxeXF-uj6lBKYP-O3-IvdtHhCo/edit?ts=5e95f280#gid=0&amp;amp;range=G:G&quot;&gt;Column G in this spreadsheet&lt;/a&gt; giving a number higher than 4, when &lt;a href=&quot;https://docs.google.com/spreadsheets/d/1Muz2ftyDUUewMTZPxYxeXF-uj6lBKYP-O3-IvdtHhCo/edit?ts=5e95f280#gid=0&amp;amp;range=E2&quot;&gt;E2&lt;/a&gt; is set to 2.&lt;/s&gt;&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-4-2406&quot;&gt;&lt;/a&gt; &lt;/div&gt;&lt;/li&gt; &lt;li&gt;&lt;div class=&quot;li&quot;&gt; - &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-5-2406&quot;&gt;&lt;/span&gt;Toward the end of the period it took 15,000 years to grow by $0.6Bn, and growth of $1.8Bn would have been needed for a doubling. So assuming linear growth at the end-of-period rate, this would have taken around 45,000 years, whereas in if growth was speeding up, it should have taken longer.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-5-2406&quot;&gt;&lt;/a&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-5-2406&quot;&gt;&lt;/span&gt;Toward the end of the period it took 15,000 years to grow by \$0.6Bn, and growth of \$1.8Bn would have been needed for a doubling. So assuming linear growth at the end-of-period rate, this would have taken around 45,000 years, whereas in if growth was speeding up, it should have taken longer.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-5-2406&quot;&gt;&lt;/a&gt; &lt;/div&gt;&lt;/li&gt; &lt;li&gt;&lt;div class=&quot;li&quot;&gt; &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-6-2406&quot;&gt;&lt;/span&gt; Khan Academy. “The Dawn of Agriculture (Article).” Accessed April 14, 2020. &lt;a href=&quot;https://www.khanacademy.org/humanities/world-history/world-history-beginnings/birth-agriculture-neolithic-revolution/a/where-did-agriculture-come-from&quot;&gt;https://www.khanacademy.org/humanities/world-history/world-history-beginnings/birth-agriculture-neolithic-revolution/a/where-did-agriculture-come-from&lt;/a&gt;. &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-6-2406&quot;&gt;&lt;/a&gt; &lt;/div&gt;&lt;/li&gt; </pre>

2023-05-07T21:37:19+00:00 2023-05-07T21:37:19+00:00 https://wiki.aiimpacts.org/featured_articles/preliminary_survey_of_prescient_actions?rev=1683495439&do=diff Anonymous anonymous@undisclosed.example.com <pre> @@ -311,9 +311,9 @@ &lt;HTML&gt; &lt;p&gt;&lt;strong&gt;Alexander Fleming&lt;/strong&gt; warned, in his 1945 Nobel Lecture, that widespread access to antibiotics without supervision may lead to antibiotic resistance.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-11-2362&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-11-2362&quot; title=&#039;“The time may come when penicillin can be bought by anyone in the shops. Then there is the danger that the ignorant man may easily underdose himself and by exposing his microbes to non-lethal quantities of the drugmake them resistant.” “Wayback Machine,” March 31, 2018.&amp;lt;a href=&quot;https://web.archive.org/web/20180331001640/https://www.nobelprize.org/nobel_prizes/medicine/laureates/1945/fleming-lecture.pdf&quot;&amp;gt; https://web.archive.org/web/20180331001640/https://www.nobelprize.org/nobel_prizes/medicine/laureates/1945/fleming-lecture.pdf&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;11&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; We are uncertain of the impact of Fleming’s warning, whether he took additional action to mitigate the risk, or how widespread within the scientific community such concerns were, but our impression is that it was not a widely known issue, that his was an early warning, and that his judgement was generally taken seriously by the time of his speech. His warning preceded the first documented cases of penicillin-resistant bacteria by more than 20 years, and the threat of antimicrobial resistance seems to be broadly analogous with AI risk on most of our criteria, though it does seem that feedback was available throughout efforts to reduce the threat.&lt;/p&gt; &lt;/HTML&gt; - + //Update: see our [[https://aiimpacts.org/wp-content/uploads/2023/04/Alexander_Fleming__antibiotic_resistance__and_relevant_lessons_for_the_mitigation_of_risk_from_advanced_artificial_intelligence.pdf|full report]] about Alexander Fleming.// &lt;HTML&gt; &lt;p&gt;&lt;strong&gt;The Treaty on the Non-Proliferation of Nuclear Weapons&lt;/strong&gt; required many actions from many actors, but it seems to have required a complex prediction about technological development and geopolitics to address a severe threat, was specific to a particular threat, and had limited opportunities for feedback. We are uncertain if any of the specific actions will prove to be prescient on further investigation, but it seems promising.&lt;br/&gt;&lt;/p&gt; &lt;/HTML&gt; </pre>

<pre> @@ -311,9 +311,9 @@ &lt;HTML&gt; &lt;p&gt;&lt;strong&gt;Alexander Fleming&lt;/strong&gt; warned, in his 1945 Nobel Lecture, that widespread access to antibiotics without supervision may lead to antibiotic resistance.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-11-2362&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-11-2362&quot; title=&#039;“The time may come when penicillin can be bought by anyone in the shops. Then there is the danger that the ignorant man may easily underdose himself and by exposing his microbes to non-lethal quantities of the drugmake them resistant.” “Wayback Machine,” March 31, 2018.&amp;lt;a href=&quot;https://web.archive.org/web/20180331001640/https://www.nobelprize.org/nobel_prizes/medicine/laureates/1945/fleming-lecture.pdf&quot;&amp;gt; https://web.archive.org/web/20180331001640/https://www.nobelprize.org/nobel_prizes/medicine/laureates/1945/fleming-lecture.pdf&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;11&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; We are uncertain of the impact of Fleming’s warning, whether he took additional action to mitigate the risk, or how widespread within the scientific community such concerns were, but our impression is that it was not a widely known issue, that his was an early warning, and that his judgement was generally taken seriously by the time of his speech. His warning preceded the first documented cases of penicillin-resistant bacteria by more than 20 years, and the threat of antimicrobial resistance seems to be broadly analogous with AI risk on most of our criteria, though it does seem that feedback was available throughout efforts to reduce the threat.&lt;/p&gt; &lt;/HTML&gt; - + //Update: see our [[https://aiimpacts.org/wp-content/uploads/2023/04/Alexander_Fleming__antibiotic_resistance__and_relevant_lessons_for_the_mitigation_of_risk_from_advanced_artificial_intelligence.pdf|full report]] about Alexander Fleming.// &lt;HTML&gt; &lt;p&gt;&lt;strong&gt;The Treaty on the Non-Proliferation of Nuclear Weapons&lt;/strong&gt; required many actions from many actors, but it seems to have required a complex prediction about technological development and geopolitics to address a severe threat, was specific to a particular threat, and had limited opportunities for feedback. We are uncertain if any of the specific actions will prove to be prescient on further investigation, but it seems promising.&lt;br/&gt;&lt;/p&gt; &lt;/HTML&gt; </pre>

2022-09-21T07:37:41+00:00 2022-09-21T07:37:41+00:00 https://wiki.aiimpacts.org/featured_articles/research_topic_hardware_software_and_ai?rev=1663745861&do=diff Anonymous anonymous@undisclosed.example.com <pre> @@ -1 +1,155 @@ + ====== Research topic: Hardware, software and AI ====== + + // Published 19 February, 2015; last updated 10 December, 2020 // + + &lt;HTML&gt; + &lt;p&gt;This is the first in a sequence of articles outlining research which could help forecast AI development.&lt;/p&gt; + &lt;/HTML&gt; + + + + + + ===== Interpretation ===== + + + &lt;HTML&gt; + &lt;p&gt;Concrete research projects are in boxes. ∑5 ∆8  means we guess the project will take (very) roughly five hours, and we rate its value (very) roughly 8/10.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Most projects could be done to very different degrees of depth, or at very different scales. Our time cost estimates correspond to a size that we would be likely to intend if we were to do the project. Value estimates are merely ordinal indicators of worth, based on our intuitive sense, and unworthy of being taken very seriously.&lt;/p&gt; + &lt;/HTML&gt; + + + + + + + ===== 1. How does AI progress depend on hardware and software? ===== + + + &lt;HTML&gt; + &lt;p&gt;At a high level, AI improves when people make better software, when they can run it on better hardware, when they gather bigger, better training sets, etc. This makes present-day hardware and software progress a natural place to look for evidence about when advanced AI will arrive. In order to interpret any such data however, it is important to know how these pieces fit together. For instance, is the progress we see now mostly driven by hardware progress, or software progress? Can the same level of performance usually be achieved by widely varying mixtures of hardware and software? Does progress on software depend on progress on hardware?&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;It is important to understand the relationship between hardware, software and AI for several reasons. If hardware progress is the main driver of AI progress, then quite different evidence would tell us about AI timelines than if software is the main driver. Thus different research is valuable, and different timelines are likely. Many people base their AI predictions on hardware progress, while others decline to, so it would be broadly useful to know whether one should. We also expect understanding here to be generally useful.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;So we think research in this direction seems valuable. We also think several projects seem tractable. Yet little appears to have been done in this direction. Thus this topic seems a high priority.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== 1.1 How does AI progress depend qualitatively on hardware and software progress? ==== + + + &lt;HTML&gt; + &lt;p&gt;For instance, will human-level AI appear when we have both a certain amount of hardware, and certain developments in software? Or can hardware and software substitute for one another? Substitution seems a natural model of the relationship between hardware and software, since anecdotally many tasks can be done by low quality software and lots of hardware, or by high quality software and less hardware. However the extent of this is unclear. This kind of model is also &lt;a href=&quot;http://aiimpacts.org/how-ai-timelines-are-estimated/&quot; title=&quot;How AI timelines are estimated&quot;&gt;not commonly used&lt;/a&gt; in estimating &lt;a href=&quot;/doku.php?id=ai_timelines:list_of_analyses_of_time_to_human-level_ai&quot; title=&quot;List of Analyses of Time to Human-Level AI&quot;&gt;AI timelines&lt;/a&gt;, so judging whether it should be might be a useful contribution. Having a good model would also bear on the priority of other research directions. As far as we know, this issue has received almost no attention. It seems moderately tractable.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.1.A Evaluate qualitative models of the relationships between hardware, software and AI&lt;/strong&gt; ∑30 ∆5&lt;br/&gt; + One way to approach the question of qualitative relationships is to assume some model, and work on projects such as those in 1.2 that measure quantitative details of the model, then revise the model if the measurements don’t make sense in it. Before that step, we might spend a short time detailing plausible models, and examining empirical and theoretical evidence we might already have, or could cheaply find. If we were going to follow up with empirical research, we would think about what evidence we would expect the research to reveal, given alternative models.&lt;br/&gt; + &lt;span style=&quot;color: #e6e6e6;&quot;&gt;~&lt;/span&gt;&lt;br/&gt; + For instance, we find the hardware-software &lt;a href=&quot;http://en.wikipedia.org/wiki/Indifference_curve&quot;&gt;indifference curve&lt;/a&gt; model described briefly above (and outlined better in a &lt;a href=&quot;http://aiimpacts.org/how-ai-timelines-are-estimated/&quot; title=&quot;How AI timelines are estimated&quot;&gt;blog post&lt;/a&gt;) plausible. Here are some ways it might be inadequate, that we might consider in evaluating it:&lt;/p&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;‘Hardware’ and ‘software’ are not sufficiently measurable entities for a ‘level’ of each in some domain to produce a stable level of performance.&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Performance depends strongly on other factors, e.g. exactly what kind of hardware and software progress you make, unique details of the software being developed, training data available.&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Different problem types, and different performance metrics on them have different kinds of behavior&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;There are ‘indifference curves’ in a sense but they are not sufficiently consistent to be worth reasoning about.&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Humanity’s technological progress is not well characterized by an expanding rectangle of feasible hardware and software levels, but more as a complicated region of feasible combinations.&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + + ==== 1.2 How much do marginal hardware and software improvements alter AI performance? ==== + + + &lt;HTML&gt; + &lt;p&gt;As mentioned above, this question is key to determining which other investigations are worthwhile. Naturally, it could also change our timelines substantially. Thus this question seems thus important to resolve. We think the projects here are particularly tractable, though not particularly cheap. For all of these projects, we would probably choose a specific set of benchmarks on particular problems to focus on. We might do multiple of these projects on the same set of benchmarks, to trace a more complete picture.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.2.A Search for natural experiments combining modern hardware and early software approaches or vice versa. &lt;/strong&gt;∑80 ∆7&lt;br/&gt; + For instance, we might find early projects with very large hardware budgets, or recent projects with &lt;a href=&quot;http://1kchess.an3.es/&quot;&gt;intentionally restricted hardware&lt;/a&gt;. Where these were tested on commonly used benchmarks, we can use them to map out the broad contributions of hardware and software to progress. For instance, if very small chess programs today run better than old chess programs which used similar (but then normal) amounts of hardware, then the difference between them can be attributed to improving software, roughly.&lt;/p&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.2.B Apply a modern understanding of software to early hardware&lt;/strong&gt; ∑2,000 ∆9&lt;br/&gt; + Choose a benchmark problem that people worked on in the past, e.g. in the 1980s. Use a modern understanding of AI to solve the problem again, still using 1980’s hardware. Compare this to how researchers did in the 1980’s. This project requires substantial time from at least one AI researcher. Ideally they would spend a similar amount of effort as the past researchers did, so it may be worth choosing a problem where it is known that an achievable level of effort was applied in the past.&lt;/p&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.2.C Apply early software understanding to modern hardware&lt;/strong&gt; ∑2,000 ∆8&lt;br/&gt; + Using contemporary hardware and a 1970’s or 1980’s understanding of connectionism, observe the extent to which a modern AI researcher (or student) could replicate contemporary performance on benchmark AI problems. This project is relatively expensive, among those we are describing. It requires substantial time from collaborators with a historically accurate minimal understanding of AI. Students may satisfy this role well, if their education is incomplete in the right ways. One might compare to the work of similar students who had also learned about modern methods.&lt;/p&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.2.D Measure marginal effects of hardware and software in existing performance trends &lt;/strong&gt; ∑100 ∆8&lt;br/&gt; + Often the same software can be used with modest changes in hardware, so changes in performance from hardware over small margins can be measured. Improved software is also often written to be run on the same hardware as earlier software, so changes in performance from software alone can be measured over moderate margins. Thus we can often estimate these marginal changes from looking at existing performance measurements.&lt;br/&gt; + &lt;span style=&quot;color: #e6e6e6;&quot;&gt;~&lt;/span&gt;&lt;br/&gt; + We can also look at overall progress over time on some applications, and factor out what we know about hardware or software change, assuming it is close to the marginal values measured by the above methods. For instance, we can see how much individual Go programs improve with more hardware, and then we can look at longer term improvements in computer Go, and guess how much of that improvement came from hardware, given our earlier estimate of marginal improvement from hardware. In general these estimates will be less valid over larger distances, as the impact of hardware or software diverge from their marginal impact, and because arbitrary of hardware and software can’t generally be combined without designing the software to make use of the hardware. &lt;a href=&quot;http://intelligence.org/files/AlgorithmicProgress.pdf&quot;&gt;Grace 2013&lt;/a&gt; includes some work this project.&lt;/p&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.2.E Interview AI researchers on the relative importance of hardware and software in driving the progress they have seen.&lt;/strong&gt; ∑20 ∆7&lt;br/&gt; + AI researchers likely have firsthand experience regarding how hardware and software contribute to overall progress within the vicinity of their own work. This project will probably give relatively noisy estimates, but is very cheap compared to others described here. One could just ask for views on this question, and supporting anecdotes, or devise a more structured questionnaire beforehand.&lt;/p&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + + ==== 1.3 How do hardware and software progress interact? ==== + + + &lt;HTML&gt; + &lt;p&gt;Do hardware and software progress relatively independently, or for instance do advances in hardware encourage advances in software? This might change how we generally expect software progress to proceed, and what combinations of hardware and software we expect to first produce human-level AI. We are likely to get some information about this from other projects looking at historical performance data e.g. 1.2.D. For instance, if overall progress is generally proportional to hardware progress, even as hardware progress varies, then this would be suggestive. Below are further possibilities.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.3.A Find natural experiments &lt;/strong&gt;∑80 ∆4&lt;br/&gt; + Search for performance data from cases where hardware being used for an application was largely constant then shifted upward at some point. Such cases are probably hard to find, and hard to interpret when found. However, a short search for them may be worthwhile.&lt;/p&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.3.B Interview researchers&lt;/strong&gt; ∑20 ∆7&lt;br/&gt; + If hardware tends to affect software research, it is likely that researchers notice this, and can talk about it. This seems a cheap and effective method of learning qualitatively about the topic. This project should probably be combined with 1.2.E.&lt;/p&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.3.C Consider plausible models&lt;/strong&gt; ∑10 ∆5&lt;br/&gt; + This is a short theoretical project that would benefit from being done in concert with 1.3.B (interview researchers), since researchers probably have a relatively good understanding of which models are plausible, and we are likely to ask better questions of them if we have thought about the topic. This project should probably be combined with 1.1.A.&lt;/p&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + </pre>

<pre> @@ -1 +1,155 @@ + ====== Research topic: Hardware, software and AI ====== + + // Published 19 February, 2015; last updated 10 December, 2020 // + + &lt;HTML&gt; + &lt;p&gt;This is the first in a sequence of articles outlining research which could help forecast AI development.&lt;/p&gt; + &lt;/HTML&gt; + + + + + + ===== Interpretation ===== + + + &lt;HTML&gt; + &lt;p&gt;Concrete research projects are in boxes. ∑5 ∆8  means we guess the project will take (very) roughly five hours, and we rate its value (very) roughly 8/10.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Most projects could be done to very different degrees of depth, or at very different scales. Our time cost estimates correspond to a size that we would be likely to intend if we were to do the project. Value estimates are merely ordinal indicators of worth, based on our intuitive sense, and unworthy of being taken very seriously.&lt;/p&gt; + &lt;/HTML&gt; + + + + + + + ===== 1. How does AI progress depend on hardware and software? ===== + + + &lt;HTML&gt; + &lt;p&gt;At a high level, AI improves when people make better software, when they can run it on better hardware, when they gather bigger, better training sets, etc. This makes present-day hardware and software progress a natural place to look for evidence about when advanced AI will arrive. In order to interpret any such data however, it is important to know how these pieces fit together. For instance, is the progress we see now mostly driven by hardware progress, or software progress? Can the same level of performance usually be achieved by widely varying mixtures of hardware and software? Does progress on software depend on progress on hardware?&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;It is important to understand the relationship between hardware, software and AI for several reasons. If hardware progress is the main driver of AI progress, then quite different evidence would tell us about AI timelines than if software is the main driver. Thus different research is valuable, and different timelines are likely. Many people base their AI predictions on hardware progress, while others decline to, so it would be broadly useful to know whether one should. We also expect understanding here to be generally useful.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;So we think research in this direction seems valuable. We also think several projects seem tractable. Yet little appears to have been done in this direction. Thus this topic seems a high priority.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== 1.1 How does AI progress depend qualitatively on hardware and software progress? ==== + + + &lt;HTML&gt; + &lt;p&gt;For instance, will human-level AI appear when we have both a certain amount of hardware, and certain developments in software? Or can hardware and software substitute for one another? Substitution seems a natural model of the relationship between hardware and software, since anecdotally many tasks can be done by low quality software and lots of hardware, or by high quality software and less hardware. However the extent of this is unclear. This kind of model is also &lt;a href=&quot;http://aiimpacts.org/how-ai-timelines-are-estimated/&quot; title=&quot;How AI timelines are estimated&quot;&gt;not commonly used&lt;/a&gt; in estimating &lt;a href=&quot;/doku.php?id=ai_timelines:list_of_analyses_of_time_to_human-level_ai&quot; title=&quot;List of Analyses of Time to Human-Level AI&quot;&gt;AI timelines&lt;/a&gt;, so judging whether it should be might be a useful contribution. Having a good model would also bear on the priority of other research directions. As far as we know, this issue has received almost no attention. It seems moderately tractable.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.1.A Evaluate qualitative models of the relationships between hardware, software and AI&lt;/strong&gt; ∑30 ∆5&lt;br/&gt; + One way to approach the question of qualitative relationships is to assume some model, and work on projects such as those in 1.2 that measure quantitative details of the model, then revise the model if the measurements don’t make sense in it. Before that step, we might spend a short time detailing plausible models, and examining empirical and theoretical evidence we might already have, or could cheaply find. If we were going to follow up with empirical research, we would think about what evidence we would expect the research to reveal, given alternative models.&lt;br/&gt; + &lt;span style=&quot;color: #e6e6e6;&quot;&gt;~&lt;/span&gt;&lt;br/&gt; + For instance, we find the hardware-software &lt;a href=&quot;http://en.wikipedia.org/wiki/Indifference_curve&quot;&gt;indifference curve&lt;/a&gt; model described briefly above (and outlined better in a &lt;a href=&quot;http://aiimpacts.org/how-ai-timelines-are-estimated/&quot; title=&quot;How AI timelines are estimated&quot;&gt;blog post&lt;/a&gt;) plausible. Here are some ways it might be inadequate, that we might consider in evaluating it:&lt;/p&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;‘Hardware’ and ‘software’ are not sufficiently measurable entities for a ‘level’ of each in some domain to produce a stable level of performance.&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Performance depends strongly on other factors, e.g. exactly what kind of hardware and software progress you make, unique details of the software being developed, training data available.&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Different problem types, and different performance metrics on them have different kinds of behavior&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;There are ‘indifference curves’ in a sense but they are not sufficiently consistent to be worth reasoning about.&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Humanity’s technological progress is not well characterized by an expanding rectangle of feasible hardware and software levels, but more as a complicated region of feasible combinations.&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + + ==== 1.2 How much do marginal hardware and software improvements alter AI performance? ==== + + + &lt;HTML&gt; + &lt;p&gt;As mentioned above, this question is key to determining which other investigations are worthwhile. Naturally, it could also change our timelines substantially. Thus this question seems thus important to resolve. We think the projects here are particularly tractable, though not particularly cheap. For all of these projects, we would probably choose a specific set of benchmarks on particular problems to focus on. We might do multiple of these projects on the same set of benchmarks, to trace a more complete picture.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.2.A Search for natural experiments combining modern hardware and early software approaches or vice versa. &lt;/strong&gt;∑80 ∆7&lt;br/&gt; + For instance, we might find early projects with very large hardware budgets, or recent projects with &lt;a href=&quot;http://1kchess.an3.es/&quot;&gt;intentionally restricted hardware&lt;/a&gt;. Where these were tested on commonly used benchmarks, we can use them to map out the broad contributions of hardware and software to progress. For instance, if very small chess programs today run better than old chess programs which used similar (but then normal) amounts of hardware, then the difference between them can be attributed to improving software, roughly.&lt;/p&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.2.B Apply a modern understanding of software to early hardware&lt;/strong&gt; ∑2,000 ∆9&lt;br/&gt; + Choose a benchmark problem that people worked on in the past, e.g. in the 1980s. Use a modern understanding of AI to solve the problem again, still using 1980’s hardware. Compare this to how researchers did in the 1980’s. This project requires substantial time from at least one AI researcher. Ideally they would spend a similar amount of effort as the past researchers did, so it may be worth choosing a problem where it is known that an achievable level of effort was applied in the past.&lt;/p&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.2.C Apply early software understanding to modern hardware&lt;/strong&gt; ∑2,000 ∆8&lt;br/&gt; + Using contemporary hardware and a 1970’s or 1980’s understanding of connectionism, observe the extent to which a modern AI researcher (or student) could replicate contemporary performance on benchmark AI problems. This project is relatively expensive, among those we are describing. It requires substantial time from collaborators with a historically accurate minimal understanding of AI. Students may satisfy this role well, if their education is incomplete in the right ways. One might compare to the work of similar students who had also learned about modern methods.&lt;/p&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.2.D Measure marginal effects of hardware and software in existing performance trends &lt;/strong&gt; ∑100 ∆8&lt;br/&gt; + Often the same software can be used with modest changes in hardware, so changes in performance from hardware over small margins can be measured. Improved software is also often written to be run on the same hardware as earlier software, so changes in performance from software alone can be measured over moderate margins. Thus we can often estimate these marginal changes from looking at existing performance measurements.&lt;br/&gt; + &lt;span style=&quot;color: #e6e6e6;&quot;&gt;~&lt;/span&gt;&lt;br/&gt; + We can also look at overall progress over time on some applications, and factor out what we know about hardware or software change, assuming it is close to the marginal values measured by the above methods. For instance, we can see how much individual Go programs improve with more hardware, and then we can look at longer term improvements in computer Go, and guess how much of that improvement came from hardware, given our earlier estimate of marginal improvement from hardware. In general these estimates will be less valid over larger distances, as the impact of hardware or software diverge from their marginal impact, and because arbitrary of hardware and software can’t generally be combined without designing the software to make use of the hardware. &lt;a href=&quot;http://intelligence.org/files/AlgorithmicProgress.pdf&quot;&gt;Grace 2013&lt;/a&gt; includes some work this project.&lt;/p&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.2.E Interview AI researchers on the relative importance of hardware and software in driving the progress they have seen.&lt;/strong&gt; ∑20 ∆7&lt;br/&gt; + AI researchers likely have firsthand experience regarding how hardware and software contribute to overall progress within the vicinity of their own work. This project will probably give relatively noisy estimates, but is very cheap compared to others described here. One could just ask for views on this question, and supporting anecdotes, or devise a more structured questionnaire beforehand.&lt;/p&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + + ==== 1.3 How do hardware and software progress interact? ==== + + + &lt;HTML&gt; + &lt;p&gt;Do hardware and software progress relatively independently, or for instance do advances in hardware encourage advances in software? This might change how we generally expect software progress to proceed, and what combinations of hardware and software we expect to first produce human-level AI. We are likely to get some information about this from other projects looking at historical performance data e.g. 1.2.D. For instance, if overall progress is generally proportional to hardware progress, even as hardware progress varies, then this would be suggestive. Below are further possibilities.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.3.A Find natural experiments &lt;/strong&gt;∑80 ∆4&lt;br/&gt; + Search for performance data from cases where hardware being used for an application was largely constant then shifted upward at some point. Such cases are probably hard to find, and hard to interpret when found. However, a short search for them may be worthwhile.&lt;/p&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.3.B Interview researchers&lt;/strong&gt; ∑20 ∆7&lt;br/&gt; + If hardware tends to affect software research, it is likely that researchers notice this, and can talk about it. This seems a cheap and effective method of learning qualitatively about the topic. This project should probably be combined with 1.2.E.&lt;/p&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;blockquote&gt; + &lt;p&gt;&lt;strong&gt;1.3.C Consider plausible models&lt;/strong&gt; ∑10 ∆5&lt;br/&gt; + This is a short theoretical project that would benefit from being done in concert with 1.3.B (interview researchers), since researchers probably have a relatively good understanding of which models are plausible, and we are likely to ask better questions of them if we have thought about the topic. This project should probably be combined with 1.1.A.&lt;/p&gt; + &lt;/blockquote&gt; + &lt;/HTML&gt; + + </pre>

2022-09-21T07:37:41+00:00 2022-09-21T07:37:41+00:00 https://wiki.aiimpacts.org/featured_articles/returns_to_scale_in_research?rev=1663745861&do=diff Anonymous anonymous@undisclosed.example.com <pre> @@ -1 +1,233 @@ + ====== Returns to scale in research ====== + + // Published 06 July, 2016; last updated 28 September, 2017 // + + &lt;HTML&gt; + &lt;p&gt;When universities or university departments produce research outputs—such as published papers—they sometimes experience increasing returns to scale, sometimes constant returns to scale, and sometimes decreasing returns to scale. At the level of nations however, R&amp;amp;D tends to see increasing returns to scale. These results are preliminary.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;Background&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;“Returns to scale” refers to the responsiveness of a process’ outputs when all inputs (e.g. researcher hours, equipment) are increased by a certain proportion. If all outputs (e.g. published papers, citations, patents) increase by that same proportion, the process is said to exhibit constant returns to scale. Increasing returns to scale and decreasing returns to scale refer to situations where outputs still increase, but by a higher or lower proportion, respectively.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Assessing returns to scale in research may be useful in predicting certain aspects of the development of artificial intelligence, in particular the dynamics of an intelligence explosion.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;Results&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;The conclusions in this article are drawn from an incomplete review of academic literature assessing research efficiency, presented in Table 1. These papers assess research in terms of its direct outputs such as published papers, citations, and patents. The broader effects of the research are not considered.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Most of the papers listed below use the &lt;a href=&quot;http://www.springer.com/cda/content/document/cda_downloaddocument/9780387332116-c2.pdf&quot;&gt;Data Envelopment Analysis (DEA) technique&lt;/a&gt;, which is a quantitative technique commonly used to assess the efficiency of universities and research activities. It is capable of isolating the scale efficiency of the individual departments, universities or countries being studied.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;table&gt; + &lt;tbody&gt; + &lt;tr&gt; + &lt;td&gt;&lt;strong&gt;Paper&lt;/strong&gt;&lt;/td&gt; + &lt;td&gt;&lt;strong&gt;Level of comparison&lt;/strong&gt;&lt;/td&gt; + &lt;td&gt;&lt;strong&gt;Activities assessed&lt;/strong&gt;&lt;/td&gt; + &lt;td&gt;&lt;strong&gt;Results pertaining to returns to scale&lt;/strong&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt; + &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0048733306002149&quot;&gt;&lt;strong&gt;Wang &amp;amp; Huang 2007&lt;/strong&gt;&lt;/a&gt; + &lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Countries&lt;/strong&gt;’ overall R&amp;amp;D activities&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Research&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Increasing returns to scale&lt;/strong&gt; in research are exhibited by more than two-thirds of the sample&lt;/span&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt; + &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0038012105000352&quot;&gt;&lt;strong&gt;Kocher, Luptacik &amp;amp; Sutter 2006&lt;/strong&gt;&lt;/a&gt; + &lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Countries&lt;/strong&gt;’ R&amp;amp;D in economics&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Research&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Increasing returns to scale&lt;/strong&gt; are found in all countries in the sample except the US&lt;/span&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt; + &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0048733305000570&quot;&gt;&lt;strong&gt;Cherchye &amp;amp; Abeele 2005&lt;/strong&gt;&lt;/a&gt; + &lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Dutch &lt;strong&gt;universities&lt;/strong&gt;’ research in Economics and Business Management&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Research&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Returns to scale vary between decreasing, constant and increasing depending on each university’s specialization&lt;/span&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt; + &lt;a href=&quot;http://www.jstor.org/stable/2663642&quot;&gt;&lt;strong&gt;Johnes &amp;amp; Johnes 1993&lt;/strong&gt;&lt;/a&gt; + &lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;UK &lt;strong&gt;universities&lt;/strong&gt;’ research in economics&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Research&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Constant returns to scale&lt;/strong&gt; are found in the sample as a whole&lt;/span&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt; + &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0038012100000100&quot;&gt;&lt;strong&gt;Avkiran 2001&lt;/strong&gt;&lt;/a&gt; + &lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Australian &lt;strong&gt;universities&lt;/strong&gt;&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Research, education&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Constant returns to scale&lt;/strong&gt; found in most sampled universities&lt;/span&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt; + &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/0038012188900080&quot;&gt;&lt;strong&gt;Ahn 1988&lt;/strong&gt;&lt;/a&gt; + &lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;US &lt;strong&gt;universities&lt;/strong&gt;&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Research, education&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Decreasing returns to scale&lt;/strong&gt; on average&lt;/span&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt; + &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0272775705000713&quot;&gt;&lt;strong&gt;Johnes 2006&lt;/strong&gt;&lt;/a&gt; + &lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;English &lt;strong&gt;universities&lt;/strong&gt;&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Research, education&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Close to &lt;strong&gt;constant returns to scale&lt;/strong&gt; exhibited by most universities sampled&lt;/span&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt; + &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0305048306000211&quot;&gt;&lt;strong&gt;Kao &amp;amp; Hung 2008&lt;/strong&gt;&lt;/a&gt; + &lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Departments&lt;/strong&gt; of a Taiwanese university&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Research, education&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Increasing returns to scale&lt;/strong&gt; exhibited by the five most scale-inefficient departments. However, no aggregate measure of returns to scale within the sample is presented.&lt;/span&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;/tbody&gt; + &lt;/table&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;&lt;i&gt;Table 1&lt;/i&gt;&lt;i&gt;:&lt;/i&gt; &lt;i&gt;Sample of studies of research efficiency that assess returns to scale&lt;/i&gt;&lt;/strong&gt;&lt;i&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;br/&gt;&lt;/span&gt;&lt;/i&gt; &lt;i&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Note: This table only identifies increasing/constant/decreasing returns to scale, rather than the size of this effect. Although DEA can measure the relative size of the effect for individual departments/universities/countries within a sample, such results cannot be readily compared between samples/studies.&lt;/span&gt;&lt;/i&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;Discussion of results&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Of the studies listed in Table 1, the first four are the most relevant to this article, since they focus solely on research inputs and outputs. While the remaining four include educational inputs and outputs, they can still yield worthwhile insights.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Table 1 implies a difference between country-level and university-level returns to scale in research.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;The two studies assessing R&amp;amp;D efficiency at the country level, Wang &amp;amp; Huang (2007) and Kocher, Luptacik &amp;amp; Sutter (2006), both identify increasing returns to scale.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;The two university-level studies that assessed the scale efficiency of research alone found mixed results. Concretely, Johnes &amp;amp; Johnes (1993) concluded that returns to scale are constant among UK universities, and Cherchye &amp;amp; Abeele (2005) concluded that returns to scale vary among Dutch universities. This ambiguity is echoed by the remainder of the studies listed above, which assess education and research simultaneously and which find evidence of constant, decreasing and increasing returns to scale in different contexts.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Such differences are consistent with the possibility that scale efficiency may be influenced by scale (size) itself. In this framework, as an organisation increases its size, it may experience increasing returns to scale initially, resulting in increased efficiency. However, the efficiency gains from growth may not continue indefinitely; after passing a certain threshold the organisation may experience decreasing returns to scale. The threshold would represent the point of scale efficiency, at which returns to scale are constant and efficiency is maximized with respect to size.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Under this framework, size will influence whether increasing, constant or decreasing returns to scale are experienced. Applying this to research activities, the observation of different returns to scale between country-level and university-level research may mean that the size of a country’s overall research effort and the typical size of its universities are not determined by similar factors. For example, if increasing returns to scale at the country level and decreasing returns to scale at the university level are observed, this may indicate that the overall number of universities is smaller than needed to achieve scale efficiency, but that most of these universities are individually too large to be scale efficient.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Other factors may also contribute to the differences between university-level and country-level observations.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;The country level studies use relatively aggregated data, capturing some of the non-university research and development activities in the countries sampled.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Country level research effort is not necessarily subject to some of the constraints which may cause decreasing returns to scale in large universities, such as excessive bureaucracy.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Results may be arbitrarily influenced by differences in the available input and output metrics at the university versus country level.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;Limitations to conclusions drawn&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;One limitation of this article is the small scope of the literature review. A more comprehensive review may reveal a different range of conclusions.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Another limitation is that the research outputs studied—published papers, citations, and patents, inter alia—cannot be assumed to correspond directly to incremental knowledge or productivity. This point is expanded upon under “Topics for further investigation” below.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Further limitations arise due to the DEA technique used by most of the studies in Table 1.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;DEA is sensitive to the choice of inputs and outputs, and to measurement errors.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Statistical hypothesis tests are difficult within the DEA framework, making it more difficult to separate signal from noise in interpreting results.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;DEA identifies relative efficiency (composed of scale efficiency and also “pure technical efficiency”) within the sample, meaning that at least one country, university, or department is always identified as fully efficient (including exhibiting full scale efficiency or constant returns to scale). Of course, in practice, no university, organisation or production process is perfectly efficient. Therefore, conclusions drawn from DEA analysis are likely to be more informative for countries, universities, or departments that are not identified as fully efficient.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;It may be questionable whether such a framework—where an optimal scale of production exists, past which decreasing returns to scale are experienced—is a good reflection of the dynamics of research activities. However, the frequent use of the DEA framework in assessing research activities would suggest that it is appropriate.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;Topics for further investigation&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;The scope of this article is limited to direct research outputs (such as published papers, citations, and patents). While this is valuable, stronger conclusions could be drawn if this analysis were combined with further work investigating the following:&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;The impact of other sources of new knowledge apart from universities or official R&amp;amp;D expenditure. For example, innovations in company management discovered through “learning by doing” rather than through formal research may be an important source of improvement in economic productivity.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;The translation of research outputs (such as published papers, citations, and patents) into incremental knowledge, and the translation of incremental knowledge into extra productive capacity. Assessment of this may be achievable through consideration of the economic returns to research, or of the value of patents generated by research.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;Implications for AI&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;The scope for an intelligence explosion is likely to be greater if the returns to scale in research are greater. In particular, an AI system capable of conducting research into the improvement of AI may be able to be scaled up faster and more cheaply than the training of human researchers, for example through deployment on additional hardware. In addition, in the period before any intelligence explosion, a scaling-up of AI research may be observed, especially if the resultant technology were seen to have commercial applications.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;This review is one component of a larger project to quantitatively model an intelligence explosion. This project, in addition to drawing upon the conclusions in this article, will also consider inter alia the effect of intelligence on research productivity, and actual increases in artificial intelligence that are plausible from research efforts.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + </pre>

<pre> @@ -1 +1,233 @@ + ====== Returns to scale in research ====== + + // Published 06 July, 2016; last updated 28 September, 2017 // + + &lt;HTML&gt; + &lt;p&gt;When universities or university departments produce research outputs—such as published papers—they sometimes experience increasing returns to scale, sometimes constant returns to scale, and sometimes decreasing returns to scale. At the level of nations however, R&amp;amp;D tends to see increasing returns to scale. These results are preliminary.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;Background&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;“Returns to scale” refers to the responsiveness of a process’ outputs when all inputs (e.g. researcher hours, equipment) are increased by a certain proportion. If all outputs (e.g. published papers, citations, patents) increase by that same proportion, the process is said to exhibit constant returns to scale. Increasing returns to scale and decreasing returns to scale refer to situations where outputs still increase, but by a higher or lower proportion, respectively.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Assessing returns to scale in research may be useful in predicting certain aspects of the development of artificial intelligence, in particular the dynamics of an intelligence explosion.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;Results&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;The conclusions in this article are drawn from an incomplete review of academic literature assessing research efficiency, presented in Table 1. These papers assess research in terms of its direct outputs such as published papers, citations, and patents. The broader effects of the research are not considered.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;Most of the papers listed below use the &lt;a href=&quot;http://www.springer.com/cda/content/document/cda_downloaddocument/9780387332116-c2.pdf&quot;&gt;Data Envelopment Analysis (DEA) technique&lt;/a&gt;, which is a quantitative technique commonly used to assess the efficiency of universities and research activities. It is capable of isolating the scale efficiency of the individual departments, universities or countries being studied.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;table&gt; + &lt;tbody&gt; + &lt;tr&gt; + &lt;td&gt;&lt;strong&gt;Paper&lt;/strong&gt;&lt;/td&gt; + &lt;td&gt;&lt;strong&gt;Level of comparison&lt;/strong&gt;&lt;/td&gt; + &lt;td&gt;&lt;strong&gt;Activities assessed&lt;/strong&gt;&lt;/td&gt; + &lt;td&gt;&lt;strong&gt;Results pertaining to returns to scale&lt;/strong&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt; + &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0048733306002149&quot;&gt;&lt;strong&gt;Wang &amp;amp; Huang 2007&lt;/strong&gt;&lt;/a&gt; + &lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Countries&lt;/strong&gt;’ overall R&amp;amp;D activities&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Research&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Increasing returns to scale&lt;/strong&gt; in research are exhibited by more than two-thirds of the sample&lt;/span&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt; + &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0038012105000352&quot;&gt;&lt;strong&gt;Kocher, Luptacik &amp;amp; Sutter 2006&lt;/strong&gt;&lt;/a&gt; + &lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Countries&lt;/strong&gt;’ R&amp;amp;D in economics&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Research&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Increasing returns to scale&lt;/strong&gt; are found in all countries in the sample except the US&lt;/span&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt; + &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0048733305000570&quot;&gt;&lt;strong&gt;Cherchye &amp;amp; Abeele 2005&lt;/strong&gt;&lt;/a&gt; + &lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Dutch &lt;strong&gt;universities&lt;/strong&gt;’ research in Economics and Business Management&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Research&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Returns to scale vary between decreasing, constant and increasing depending on each university’s specialization&lt;/span&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt; + &lt;a href=&quot;http://www.jstor.org/stable/2663642&quot;&gt;&lt;strong&gt;Johnes &amp;amp; Johnes 1993&lt;/strong&gt;&lt;/a&gt; + &lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;UK &lt;strong&gt;universities&lt;/strong&gt;’ research in economics&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Research&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Constant returns to scale&lt;/strong&gt; are found in the sample as a whole&lt;/span&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt; + &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0038012100000100&quot;&gt;&lt;strong&gt;Avkiran 2001&lt;/strong&gt;&lt;/a&gt; + &lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Australian &lt;strong&gt;universities&lt;/strong&gt;&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Research, education&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Constant returns to scale&lt;/strong&gt; found in most sampled universities&lt;/span&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt; + &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/0038012188900080&quot;&gt;&lt;strong&gt;Ahn 1988&lt;/strong&gt;&lt;/a&gt; + &lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;US &lt;strong&gt;universities&lt;/strong&gt;&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Research, education&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Decreasing returns to scale&lt;/strong&gt; on average&lt;/span&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt; + &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0272775705000713&quot;&gt;&lt;strong&gt;Johnes 2006&lt;/strong&gt;&lt;/a&gt; + &lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;English &lt;strong&gt;universities&lt;/strong&gt;&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Research, education&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Close to &lt;strong&gt;constant returns to scale&lt;/strong&gt; exhibited by most universities sampled&lt;/span&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt; + &lt;a href=&quot;http://www.sciencedirect.com/science/article/pii/S0305048306000211&quot;&gt;&lt;strong&gt;Kao &amp;amp; Hung 2008&lt;/strong&gt;&lt;/a&gt; + &lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Departments&lt;/strong&gt; of a Taiwanese university&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Research, education&lt;/span&gt;&lt;/td&gt; + &lt;td&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;strong&gt;Increasing returns to scale&lt;/strong&gt; exhibited by the five most scale-inefficient departments. However, no aggregate measure of returns to scale within the sample is presented.&lt;/span&gt;&lt;/td&gt; + &lt;/tr&gt; + &lt;/tbody&gt; + &lt;/table&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;&lt;i&gt;Table 1&lt;/i&gt;&lt;i&gt;:&lt;/i&gt; &lt;i&gt;Sample of studies of research efficiency that assess returns to scale&lt;/i&gt;&lt;/strong&gt;&lt;i&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;&lt;br/&gt;&lt;/span&gt;&lt;/i&gt; &lt;i&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Note: This table only identifies increasing/constant/decreasing returns to scale, rather than the size of this effect. Although DEA can measure the relative size of the effect for individual departments/universities/countries within a sample, such results cannot be readily compared between samples/studies.&lt;/span&gt;&lt;/i&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;Discussion of results&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Of the studies listed in Table 1, the first four are the most relevant to this article, since they focus solely on research inputs and outputs. While the remaining four include educational inputs and outputs, they can still yield worthwhile insights.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Table 1 implies a difference between country-level and university-level returns to scale in research.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;The two studies assessing R&amp;amp;D efficiency at the country level, Wang &amp;amp; Huang (2007) and Kocher, Luptacik &amp;amp; Sutter (2006), both identify increasing returns to scale.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;The two university-level studies that assessed the scale efficiency of research alone found mixed results. Concretely, Johnes &amp;amp; Johnes (1993) concluded that returns to scale are constant among UK universities, and Cherchye &amp;amp; Abeele (2005) concluded that returns to scale vary among Dutch universities. This ambiguity is echoed by the remainder of the studies listed above, which assess education and research simultaneously and which find evidence of constant, decreasing and increasing returns to scale in different contexts.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Such differences are consistent with the possibility that scale efficiency may be influenced by scale (size) itself. In this framework, as an organisation increases its size, it may experience increasing returns to scale initially, resulting in increased efficiency. However, the efficiency gains from growth may not continue indefinitely; after passing a certain threshold the organisation may experience decreasing returns to scale. The threshold would represent the point of scale efficiency, at which returns to scale are constant and efficiency is maximized with respect to size.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Under this framework, size will influence whether increasing, constant or decreasing returns to scale are experienced. Applying this to research activities, the observation of different returns to scale between country-level and university-level research may mean that the size of a country’s overall research effort and the typical size of its universities are not determined by similar factors. For example, if increasing returns to scale at the country level and decreasing returns to scale at the university level are observed, this may indicate that the overall number of universities is smaller than needed to achieve scale efficiency, but that most of these universities are individually too large to be scale efficient.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Other factors may also contribute to the differences between university-level and country-level observations.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;The country level studies use relatively aggregated data, capturing some of the non-university research and development activities in the countries sampled.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Country level research effort is not necessarily subject to some of the constraints which may cause decreasing returns to scale in large universities, such as excessive bureaucracy.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Results may be arbitrarily influenced by differences in the available input and output metrics at the university versus country level.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;Limitations to conclusions drawn&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;One limitation of this article is the small scope of the literature review. A more comprehensive review may reveal a different range of conclusions.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Another limitation is that the research outputs studied—published papers, citations, and patents, inter alia—cannot be assumed to correspond directly to incremental knowledge or productivity. This point is expanded upon under “Topics for further investigation” below.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Further limitations arise due to the DEA technique used by most of the studies in Table 1.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;DEA is sensitive to the choice of inputs and outputs, and to measurement errors.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;Statistical hypothesis tests are difficult within the DEA framework, making it more difficult to separate signal from noise in interpreting results.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;DEA identifies relative efficiency (composed of scale efficiency and also “pure technical efficiency”) within the sample, meaning that at least one country, university, or department is always identified as fully efficient (including exhibiting full scale efficiency or constant returns to scale). Of course, in practice, no university, organisation or production process is perfectly efficient. Therefore, conclusions drawn from DEA analysis are likely to be more informative for countries, universities, or departments that are not identified as fully efficient.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;It may be questionable whether such a framework—where an optimal scale of production exists, past which decreasing returns to scale are experienced—is a good reflection of the dynamics of research activities. However, the frequent use of the DEA framework in assessing research activities would suggest that it is appropriate.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;Topics for further investigation&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;The scope of this article is limited to direct research outputs (such as published papers, citations, and patents). While this is valuable, stronger conclusions could be drawn if this analysis were combined with further work investigating the following:&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;The impact of other sources of new knowledge apart from universities or official R&amp;amp;D expenditure. For example, innovations in company management discovered through “learning by doing” rather than through formal research may be an important source of improvement in economic productivity.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;The translation of research outputs (such as published papers, citations, and patents) into incremental knowledge, and the translation of incremental knowledge into extra productive capacity. Assessment of this may be achievable through consideration of the economic returns to research, or of the value of patents generated by research.&lt;/span&gt;&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;strong&gt;Implications for AI&lt;/strong&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;The scope for an intelligence explosion is likely to be greater if the returns to scale in research are greater. In particular, an AI system capable of conducting research into the improvement of AI may be able to be scaled up faster and more cheaply than the training of human researchers, for example through deployment on additional hardware. In addition, in the period before any intelligence explosion, a scaling-up of AI research may be observed, especially if the resultant technology were seen to have commercial applications.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;span style=&quot;font-weight: 400&quot;&gt;This review is one component of a larger project to quantitatively model an intelligence explosion. This project, in addition to drawing upon the conclusions in this article, will also consider inter alia the effect of intelligence on research productivity, and actual increases in artificial intelligence that are plausible from research efforts.&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + </pre>

2022-09-21T07:37:41+00:00 2022-09-21T07:37:41+00:00 https://wiki.aiimpacts.org/featured_articles/time_for_ai_to_cross_the_human_performance_range_in_imagenet_image_classification?rev=1663745861&do=diff Anonymous anonymous@undisclosed.example.com <pre> @@ -1 +1,222 @@ + ====== Time for AI to cross the human performance range in ImageNet image classification ====== + + // Published 19 October, 2020; last updated 08 March, 2021 // + + + + &lt;HTML&gt; + &lt;p&gt;Progress in computer image classification performance took:&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Over 14 years to reach the level of an untrained human&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;3 years to pass from untrained human level to trained human level&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;5 years to continue from trained human to current performance (2020)&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + + ===== Details ===== + + + ==== Metric ==== + + + &lt;HTML&gt; + &lt;p&gt;ImageNet&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-1-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-1-2683&quot; title=&#039;&amp;amp;#8220;&amp;lt;strong&amp;gt;ImageNet&amp;lt;/strong&amp;gt;&amp;amp;nbsp;is an image database organized according to the&amp;amp;nbsp;&amp;lt;a rel=&quot;noreferrer noopener&quot; href=&quot;http://wordnet.princeton.edu/&quot; target=&quot;_blank&quot;&amp;gt;WordNet&amp;lt;/a&amp;gt;&amp;amp;nbsp;hierarchy (currently only the nouns), in which each node of the hierarchy is depicted by hundreds and thousands of images. Currently we have an average of over five hundred images per node.&amp;amp;nbsp;&amp;amp;#8220;&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;“ImageNet.” Accessed October 19, 2020. &amp;lt;a href=&quot;http://www.image-net.org/&quot;&amp;gt;http://www.image-net.org/&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;1&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; is a large collection of images organized into a hierarchy of noun categories. We looked at ‘top-5 accuracy’ in categorizing images. In this task, the player is given an image, and can guess five different categories that the image might represent. It is judged as correct if the image is in fact in any of those five categories.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== Human performance milestones ==== + + + === Beginner level === + + + &lt;HTML&gt; + &lt;p&gt;We used Andrej Karpathy’s interface&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-2-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-2-2683&quot; title=&#039;Karpathy, Andrej. “Ilsvrc.” Accessed October 19, 2020. &amp;lt;a href=&quot;https://cs.stanford.edu/people/karpathy/ilsvrc/&quot;&amp;gt;https://cs.stanford.edu/people/karpathy/ilsvrc/&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;2&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; for doing the ImageNet top-5 accuracy task ourselves, and asked a few friends to do it. Five people did it, with performances ranging from 74% to 89%, with a median performance of 81%.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;This was not a random sample of people, and conditions for taking the test differed. Most notably, there was no time limit, so time allocated was set by patience for trying to marginally improve guesses.&lt;/p&gt; + &lt;/HTML&gt; + + + === Trained human-level === + + + &lt;HTML&gt; + &lt;p&gt;ImageNet categorization is not a popular activity for humans, so we do not know what highly talented and trained human performance would look like. The best relatively high human performance measure we have comes from Russakovsky et al, who report on performance of two ‘expert annotators’, who they say learned many of the categories. &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-3-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-3-2683&quot; title=&#039;&amp;amp;#8216;Therefore, in evaluating the human accuracy we relied primarily on expert annotators who learned to recognize a large portion of the 1000 ILSVRC classes. During training, the annotators labeled a few hundred validation images for practice and later switched to the test set images&amp;amp;#8217;&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &amp;lt;em&amp;gt;ArXiv:1409.0575 [Cs]&amp;lt;/em&amp;gt;, January 29, 2015. &amp;lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&amp;gt;http://arxiv.org/abs/1409.0575&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;3&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; The better performing annotator there had a 5.1% error rate.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-4-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-4-2683&quot; title=&#039;&amp;amp;#8220;Annotator A1 evaluated a total of 1500 test set images. The GoogLeNet classication error on this sample was estimated to be 6.8% (recall that the error on full test set of 100,000 images is 6.7%, as shown in Table 7). The human error was estimated to be 5.1%.&amp;amp;#8221;&amp;lt;br&amp;gt;&amp;lt;br&amp;gt;Also see Table 9&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &amp;lt;em&amp;gt;ArXiv:1409.0575 [Cs]&amp;lt;/em&amp;gt;, January 29, 2015. &amp;lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&amp;gt;http://arxiv.org/abs/1409.0575&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;4&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + ==== AI achievement of human milestones ==== + + + === Earliest attempt === + + + &lt;HTML&gt; + &lt;p&gt;The ImageNet database was released in 2009.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-5-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-5-2683&quot; title=&#039;&amp;amp;#8220;They presented their database for the first time as a poster at the 2009&amp;amp;nbsp;&amp;lt;a href=&quot;https://en.wikipedia.org/wiki/Conference_on_Computer_Vision_and_Pattern_Recognition&quot;&amp;gt;Conference on Computer Vision and Pattern Recognition&amp;lt;/a&amp;gt;&amp;amp;nbsp;(CVPR) in Florida.&amp;amp;#8221;&amp;lt;br&amp;gt;&amp;lt;br&amp;gt;“ImageNet.” In &amp;lt;em&amp;gt;Wikipedia&amp;lt;/em&amp;gt;, September 9, 2020. &amp;lt;a href=&quot;https://en.wikipedia.org/w/index.php?title=ImageNet&amp;amp;amp;oldid=977585441&quot;&amp;gt;https://en.wikipedia.org/w/index.php?title=ImageNet&amp;amp;amp;oldid=977585441&amp;lt;/a&amp;gt;.&amp;lt;br&amp;gt;&#039;&gt;&lt;sup&gt;5&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;. An annual contest, the ImageNet Large Scale Visual Recognition Challenge, began in 2010.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-6-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-6-2683&quot; title=&#039;&amp;amp;#8220;&amp;amp;#8230;The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) has been running annually for five years (since 2010) and has become the standard benchmark for large-scale object recognition.&amp;amp;#8221;&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &amp;lt;em&amp;gt;ArXiv:1409.0575 [Cs]&amp;lt;/em&amp;gt;, January 29, 2015. &amp;lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&amp;gt;http://arxiv.org/abs/1409.0575&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;6&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;In the 2010 contest, the best top-5 classification performance had 28.2% error.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-7-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-7-2683&quot; title=&#039;See table 6.&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &amp;lt;em&amp;gt;ArXiv:1409.0575 [Cs]&amp;lt;/em&amp;gt;, January 29, 2015. &amp;lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&amp;gt;http://arxiv.org/abs/1409.0575&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;7&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;However image classification broadly is older. Pascal VOC was a similar previous contest, which ran from 2005.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-8-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-8-2683&quot; title=&#039;&amp;amp;#8220;The PASCAL Visual Object Classes (VOC) challenge is a benchmark in visual object category recognition and detection, providing the vision and machine learning communities with a standard dataset of images and annotation, and standard evaluation procedures. Organised annually from 2005 to present, the challenge and its associated dataset has become accepted as the benchmark for object detection.&amp;amp;#8221;&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;Everingham, Mark, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. “The Pascal Visual Object Classes (VOC) Challenge.” &amp;lt;em&amp;gt;International Journal of Computer Vision&amp;lt;/em&amp;gt; 88, no. 2 (June 2010): 303–38. &amp;lt;a href=&quot;https://doi.org/10.1007/s11263-009-0275-4&quot;&amp;gt;https://doi.org/10.1007/s11263-009-0275-4&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;8&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; We do not know when the first successful image classification systems were developed. In a blog post, Amidi &amp;amp; Amidi point to LeNet as pioneering work in image classification&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-9-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-9-2683&quot; title=&#039;See section &amp;amp;#8216;LeNet&amp;amp;#8217;.&amp;lt;br&amp;gt;&amp;lt;br&amp;gt;“The Evolution of Image Classification Explained.” Accessed October 19, 2020. &amp;lt;a href=&quot;https://stanford.edu/~shervine/blog/evolution-image-classification-explained#lenet&quot;&amp;gt;https://stanford.edu/~shervine/blog/evolution-image-classification-explained#lenet&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;9&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;, and it appears to have been developed in 1998.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-10-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-10-2683&quot; title=&#039;&amp;amp;#8220;&amp;lt;strong&amp;gt;LeNet&amp;lt;/strong&amp;gt;&amp;amp;nbsp;is a&amp;amp;nbsp;&amp;lt;a href=&quot;https://en.wikipedia.org/wiki/Convolutional_neural_network&quot;&amp;gt;convolutional neural network&amp;lt;/a&amp;gt;&amp;amp;nbsp;structure proposed by&amp;amp;nbsp;&amp;lt;a href=&quot;https://en.wikipedia.org/wiki/Yann_LeCun&quot;&amp;gt;Yann LeCun&amp;lt;/a&amp;gt;&amp;amp;nbsp;et al. in 1998.&amp;amp;#8221;&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;“LeNet.” In &amp;lt;em&amp;gt;Wikipedia&amp;lt;/em&amp;gt;, June 19, 2020. &amp;lt;a href=&quot;https://en.wikipedia.org/w/index.php?title=LeNet&amp;amp;amp;oldid=963418885&quot;&amp;gt;https://en.wikipedia.org/w/index.php?title=LeNet&amp;amp;amp;oldid=963418885&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;10&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + === Beginner level === + + + &lt;HTML&gt; + &lt;p&gt;The first entrant in the ImageNet contest to perform better than our beginner level benchmark was SuperVision (commonly known as AlexNet) in 2012, with a 15.3% error rate.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-11-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-11-2683&quot; title=&#039;&amp;amp;#8220;We also entered a variant of this model in the&amp;lt;br&amp;gt;ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%&amp;amp;#8221;&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E Hinton. “ImageNet Classification with Deep Convolutional Neural Networks.” In &amp;lt;em&amp;gt;Advances in Neural Information Processing Systems 25&amp;lt;/em&amp;gt;, edited by F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, 1097–1105. Curran Associates, Inc., 2012. &amp;lt;a href=&quot;http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf&quot;&amp;gt;http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf&amp;lt;/a&amp;gt;.&amp;lt;br&amp;gt;&amp;lt;br&amp;gt;Also, see Table 6 for a list of other entrants: &amp;lt;br&amp;gt;&amp;lt;br&amp;gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &amp;lt;em&amp;gt;ArXiv:1409.0575 [Cs]&amp;lt;/em&amp;gt;, January 29, 2015. &amp;lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&amp;gt;http://arxiv.org/abs/1409.0575&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;11&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + === Superhuman level === + + + &lt;HTML&gt; + &lt;p&gt;In 2015 He et al apparently achieved a 4.5% error rate, slightly better than our high human benchmark.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-12-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-12-2683&quot; title=&#039;&amp;amp;#8220;Our 152-layer ResNet has a single-model top-5 validation error of 4.49%.&amp;amp;#8221; &amp;lt;br&amp;gt;&amp;lt;br&amp;gt;Also see Table 4&amp;lt;br&amp;gt;&amp;lt;br&amp;gt;He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. “Deep Residual Learning for Image Recognition.” &amp;lt;em&amp;gt;ArXiv:1512.03385 [Cs]&amp;lt;/em&amp;gt;, December 10, 2015. &amp;lt;a href=&quot;http://arxiv.org/abs/1512.03385&quot;&amp;gt;http://arxiv.org/abs/1512.03385&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;12&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + === Current level === + + + &lt;HTML&gt; + &lt;p&gt;According to paperswithcode.com, performance has continued to climb, to 2020, though slower than earlier.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-13-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-13-2683&quot; title=&#039;See figure:&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;“Papers with Code &amp;amp;#8211; ImageNet Benchmark (Image Classification).” Accessed October 19, 2020. &amp;lt;a href=&quot;https://paperswithcode.com/sota/image-classification-on-imagenet&quot;&amp;gt;https://paperswithcode.com/sota/image-classification-on-imagenet&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;13&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + ==== Times for AI to cross human-relative ranges  ==== + + + &lt;HTML&gt; + &lt;p&gt;Given the above dates, we have:&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;figure class=&quot;wp-block-table&quot;&gt; + &lt;table&gt; + &lt;tbody&gt; + &lt;tr&gt; + &lt;td&gt;Range&lt;/td&gt; + &lt;td&gt;Start&lt;/td&gt; + &lt;td&gt;End&lt;/td&gt; + &lt;td&gt;Duration (years)&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt;First attempt to beginner level&lt;/td&gt; + &lt;td&gt;&amp;lt;1998&lt;/td&gt; + &lt;td&gt;2012&lt;/td&gt; + &lt;td&gt;&amp;gt;14&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt;Beginner to superhuman&lt;/td&gt; + &lt;td&gt;2012&lt;/td&gt; + &lt;td&gt;2015&lt;/td&gt; + &lt;td&gt;3&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt;Above superhuman&lt;/td&gt; + &lt;td&gt;2015&lt;/td&gt; + &lt;td&gt;&amp;gt;2020&lt;/td&gt; + &lt;td&gt;&amp;gt;5&lt;/td&gt; + &lt;/tr&gt; + &lt;/tbody&gt; + &lt;/table&gt; + &lt;/figure&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;em&gt;Primary author: Rick Korzekwa&lt;/em&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + ===== Notes ===== + + + &lt;HTML&gt; + &lt;ol class=&quot;easy-footnotes-wrapper&quot;&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-1-2683&quot;&gt;&lt;/span&gt;“&lt;strong&gt;ImageNet&lt;/strong&gt; is an image database organized according to the &lt;a href=&quot;http://wordnet.princeton.edu/&quot; rel=&quot;noreferrer noopener&quot; target=&quot;_blank&quot;&gt;WordNet&lt;/a&gt; hierarchy (currently only the nouns), in which each node of the hierarchy is depicted by hundreds and thousands of images. Currently we have an average of over five hundred images per node. “ + &lt;p&gt;“ImageNet.” Accessed October 19, 2020. &lt;a href=&quot;http://www.image-net.org/&quot;&gt;http://www.image-net.org/&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-1-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-2-2683&quot;&gt;&lt;/span&gt;Karpathy, Andrej. “Ilsvrc.” Accessed October 19, 2020. &lt;a href=&quot;https://cs.stanford.edu/people/karpathy/ilsvrc/&quot;&gt;https://cs.stanford.edu/people/karpathy/ilsvrc/&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-2-2683&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-3-2683&quot;&gt;&lt;/span&gt;‘Therefore, in evaluating the human accuracy we relied primarily on expert annotators who learned to recognize a large portion of the 1000 ILSVRC classes. During training, the annotators labeled a few hundred validation images for practice and later switched to the test set images’ + &lt;p&gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &lt;em&gt;ArXiv:1409.0575 [Cs]&lt;/em&gt;, January 29, 2015. &lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&gt;http://arxiv.org/abs/1409.0575&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-3-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-4-2683&quot;&gt;&lt;/span&gt;“Annotator A1 evaluated a total of 1500 test set images. The GoogLeNet classication error on this sample was estimated to be 6.8% (recall that the error on full test set of 100,000 images is 6.7%, as shown in Table 7). The human error was estimated to be 5.1%.”&lt;br/&gt; + &lt;br/&gt; + Also see Table 9 + &lt;p&gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &lt;em&gt;ArXiv:1409.0575 [Cs]&lt;/em&gt;, January 29, 2015. &lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&gt;http://arxiv.org/abs/1409.0575&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-4-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-5-2683&quot;&gt;&lt;/span&gt;“They presented their database for the first time as a poster at the 2009 &lt;a href=&quot;https://en.wikipedia.org/wiki/Conference_on_Computer_Vision_and_Pattern_Recognition&quot;&gt;Conference on Computer Vision and Pattern Recognition&lt;/a&gt; (CVPR) in Florida.”&lt;br/&gt; + &lt;br/&gt; + “ImageNet.” In &lt;em&gt;Wikipedia&lt;/em&gt;, September 9, 2020. &lt;a href=&quot;https://en.wikipedia.org/w/index.php?title=ImageNet&amp;amp;oldid=977585441&quot;&gt;https://en.wikipedia.org/w/index.php?title=ImageNet&amp;amp;oldid=977585441&lt;/a&gt;.&lt;br/&gt; + &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-5-2683&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-6-2683&quot;&gt;&lt;/span&gt;“…The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) has been running annually for five years (since 2010) and has become the standard benchmark for large-scale object recognition.” + &lt;p&gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &lt;em&gt;ArXiv:1409.0575 [Cs]&lt;/em&gt;, January 29, 2015. &lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&gt;http://arxiv.org/abs/1409.0575&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-6-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-7-2683&quot;&gt;&lt;/span&gt;See table 6. + &lt;p&gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &lt;em&gt;ArXiv:1409.0575 [Cs]&lt;/em&gt;, January 29, 2015. &lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&gt;http://arxiv.org/abs/1409.0575&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-7-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-8-2683&quot;&gt;&lt;/span&gt;“The PASCAL Visual Object Classes (VOC) challenge is a benchmark in visual object category recognition and detection, providing the vision and machine learning communities with a standard dataset of images and annotation, and standard evaluation procedures. Organised annually from 2005 to present, the challenge and its associated dataset has become accepted as the benchmark for object detection.” + &lt;p&gt;Everingham, Mark, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. “The Pascal Visual Object Classes (VOC) Challenge.” &lt;em&gt;International Journal of Computer Vision&lt;/em&gt; 88, no. 2 (June 2010): 303–38. &lt;a href=&quot;https://doi.org/10.1007/s11263-009-0275-4&quot;&gt;https://doi.org/10.1007/s11263-009-0275-4&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-8-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-9-2683&quot;&gt;&lt;/span&gt;See section ‘LeNet’.&lt;br/&gt; + &lt;br/&gt; + “The Evolution of Image Classification Explained.” Accessed October 19, 2020. &lt;a href=&quot;https://stanford.edu/~shervine/blog/evolution-image-classification-explained#lenet&quot;&gt;https://stanford.edu/~shervine/blog/evolution-image-classification-explained#lenet&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-9-2683&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-10-2683&quot;&gt;&lt;/span&gt;“&lt;strong&gt;LeNet&lt;/strong&gt; is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Convolutional_neural_network&quot;&gt;convolutional neural network&lt;/a&gt; structure proposed by &lt;a href=&quot;https://en.wikipedia.org/wiki/Yann_LeCun&quot;&gt;Yann LeCun&lt;/a&gt; et al. in 1998.” + &lt;p&gt;“LeNet.” In &lt;em&gt;Wikipedia&lt;/em&gt;, June 19, 2020. &lt;a href=&quot;https://en.wikipedia.org/w/index.php?title=LeNet&amp;amp;oldid=963418885&quot;&gt;https://en.wikipedia.org/w/index.php?title=LeNet&amp;amp;oldid=963418885&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-10-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-11-2683&quot;&gt;&lt;/span&gt;“We also entered a variant of this model in the&lt;br/&gt; + ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%” + &lt;p&gt;Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E Hinton. “ImageNet Classification with Deep Convolutional Neural Networks.” In &lt;em&gt;Advances in Neural Information Processing Systems 25&lt;/em&gt;, edited by F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, 1097–1105. Curran Associates, Inc., 2012. &lt;a href=&quot;http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf&quot;&gt;http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf&lt;/a&gt;.&lt;br/&gt; + &lt;br/&gt; + Also, see Table 6 for a list of other entrants:&lt;br/&gt; + &lt;br/&gt; + Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &lt;em&gt;ArXiv:1409.0575 [Cs]&lt;/em&gt;, January 29, 2015. &lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&gt;http://arxiv.org/abs/1409.0575&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-11-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-12-2683&quot;&gt;&lt;/span&gt;“Our 152-layer ResNet has a single-model top-5 validation error of 4.49%.”&lt;br/&gt; + &lt;br/&gt; + Also see Table 4&lt;br/&gt; + &lt;br/&gt; + He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. “Deep Residual Learning for Image Recognition.” &lt;em&gt;ArXiv:1512.03385 [Cs]&lt;/em&gt;, December 10, 2015. &lt;a href=&quot;http://arxiv.org/abs/1512.03385&quot;&gt;http://arxiv.org/abs/1512.03385&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-12-2683&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-13-2683&quot;&gt;&lt;/span&gt;See figure: + &lt;p&gt;“Papers with Code – ImageNet Benchmark (Image Classification).” Accessed October 19, 2020. &lt;a href=&quot;https://paperswithcode.com/sota/image-classification-on-imagenet&quot;&gt;https://paperswithcode.com/sota/image-classification-on-imagenet&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-13-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + </pre>

<pre> @@ -1 +1,222 @@ + ====== Time for AI to cross the human performance range in ImageNet image classification ====== + + // Published 19 October, 2020; last updated 08 March, 2021 // + + + + &lt;HTML&gt; + &lt;p&gt;Progress in computer image classification performance took:&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;ul&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;Over 14 years to reach the level of an untrained human&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;3 years to pass from untrained human level to trained human level&lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt;5 years to continue from trained human to current performance (2020)&lt;/div&gt;&lt;/li&gt; + &lt;/ul&gt; + &lt;/HTML&gt; + + + + ===== Details ===== + + + ==== Metric ==== + + + &lt;HTML&gt; + &lt;p&gt;ImageNet&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-1-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-1-2683&quot; title=&#039;&amp;amp;#8220;&amp;lt;strong&amp;gt;ImageNet&amp;lt;/strong&amp;gt;&amp;amp;nbsp;is an image database organized according to the&amp;amp;nbsp;&amp;lt;a rel=&quot;noreferrer noopener&quot; href=&quot;http://wordnet.princeton.edu/&quot; target=&quot;_blank&quot;&amp;gt;WordNet&amp;lt;/a&amp;gt;&amp;amp;nbsp;hierarchy (currently only the nouns), in which each node of the hierarchy is depicted by hundreds and thousands of images. Currently we have an average of over five hundred images per node.&amp;amp;nbsp;&amp;amp;#8220;&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;“ImageNet.” Accessed October 19, 2020. &amp;lt;a href=&quot;http://www.image-net.org/&quot;&amp;gt;http://www.image-net.org/&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;1&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; is a large collection of images organized into a hierarchy of noun categories. We looked at ‘top-5 accuracy’ in categorizing images. In this task, the player is given an image, and can guess five different categories that the image might represent. It is judged as correct if the image is in fact in any of those five categories.&lt;/p&gt; + &lt;/HTML&gt; + + + ==== Human performance milestones ==== + + + === Beginner level === + + + &lt;HTML&gt; + &lt;p&gt;We used Andrej Karpathy’s interface&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-2-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-2-2683&quot; title=&#039;Karpathy, Andrej. “Ilsvrc.” Accessed October 19, 2020. &amp;lt;a href=&quot;https://cs.stanford.edu/people/karpathy/ilsvrc/&quot;&amp;gt;https://cs.stanford.edu/people/karpathy/ilsvrc/&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;2&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; for doing the ImageNet top-5 accuracy task ourselves, and asked a few friends to do it. Five people did it, with performances ranging from 74% to 89%, with a median performance of 81%.&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;This was not a random sample of people, and conditions for taking the test differed. Most notably, there was no time limit, so time allocated was set by patience for trying to marginally improve guesses.&lt;/p&gt; + &lt;/HTML&gt; + + + === Trained human-level === + + + &lt;HTML&gt; + &lt;p&gt;ImageNet categorization is not a popular activity for humans, so we do not know what highly talented and trained human performance would look like. The best relatively high human performance measure we have comes from Russakovsky et al, who report on performance of two ‘expert annotators’, who they say learned many of the categories. &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-3-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-3-2683&quot; title=&#039;&amp;amp;#8216;Therefore, in evaluating the human accuracy we relied primarily on expert annotators who learned to recognize a large portion of the 1000 ILSVRC classes. During training, the annotators labeled a few hundred validation images for practice and later switched to the test set images&amp;amp;#8217;&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &amp;lt;em&amp;gt;ArXiv:1409.0575 [Cs]&amp;lt;/em&amp;gt;, January 29, 2015. &amp;lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&amp;gt;http://arxiv.org/abs/1409.0575&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;3&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; The better performing annotator there had a 5.1% error rate.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-4-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-4-2683&quot; title=&#039;&amp;amp;#8220;Annotator A1 evaluated a total of 1500 test set images. The GoogLeNet classication error on this sample was estimated to be 6.8% (recall that the error on full test set of 100,000 images is 6.7%, as shown in Table 7). The human error was estimated to be 5.1%.&amp;amp;#8221;&amp;lt;br&amp;gt;&amp;lt;br&amp;gt;Also see Table 9&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &amp;lt;em&amp;gt;ArXiv:1409.0575 [Cs]&amp;lt;/em&amp;gt;, January 29, 2015. &amp;lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&amp;gt;http://arxiv.org/abs/1409.0575&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;4&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + ==== AI achievement of human milestones ==== + + + === Earliest attempt === + + + &lt;HTML&gt; + &lt;p&gt;The ImageNet database was released in 2009.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-5-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-5-2683&quot; title=&#039;&amp;amp;#8220;They presented their database for the first time as a poster at the 2009&amp;amp;nbsp;&amp;lt;a href=&quot;https://en.wikipedia.org/wiki/Conference_on_Computer_Vision_and_Pattern_Recognition&quot;&amp;gt;Conference on Computer Vision and Pattern Recognition&amp;lt;/a&amp;gt;&amp;amp;nbsp;(CVPR) in Florida.&amp;amp;#8221;&amp;lt;br&amp;gt;&amp;lt;br&amp;gt;“ImageNet.” In &amp;lt;em&amp;gt;Wikipedia&amp;lt;/em&amp;gt;, September 9, 2020. &amp;lt;a href=&quot;https://en.wikipedia.org/w/index.php?title=ImageNet&amp;amp;amp;oldid=977585441&quot;&amp;gt;https://en.wikipedia.org/w/index.php?title=ImageNet&amp;amp;amp;oldid=977585441&amp;lt;/a&amp;gt;.&amp;lt;br&amp;gt;&#039;&gt;&lt;sup&gt;5&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;. An annual contest, the ImageNet Large Scale Visual Recognition Challenge, began in 2010.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-6-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-6-2683&quot; title=&#039;&amp;amp;#8220;&amp;amp;#8230;The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) has been running annually for five years (since 2010) and has become the standard benchmark for large-scale object recognition.&amp;amp;#8221;&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &amp;lt;em&amp;gt;ArXiv:1409.0575 [Cs]&amp;lt;/em&amp;gt;, January 29, 2015. &amp;lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&amp;gt;http://arxiv.org/abs/1409.0575&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;6&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;In the 2010 contest, the best top-5 classification performance had 28.2% error.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-7-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-7-2683&quot; title=&#039;See table 6.&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &amp;lt;em&amp;gt;ArXiv:1409.0575 [Cs]&amp;lt;/em&amp;gt;, January 29, 2015. &amp;lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&amp;gt;http://arxiv.org/abs/1409.0575&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;7&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;However image classification broadly is older. Pascal VOC was a similar previous contest, which ran from 2005.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-8-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-8-2683&quot; title=&#039;&amp;amp;#8220;The PASCAL Visual Object Classes (VOC) challenge is a benchmark in visual object category recognition and detection, providing the vision and machine learning communities with a standard dataset of images and annotation, and standard evaluation procedures. Organised annually from 2005 to present, the challenge and its associated dataset has become accepted as the benchmark for object detection.&amp;amp;#8221;&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;Everingham, Mark, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. “The Pascal Visual Object Classes (VOC) Challenge.” &amp;lt;em&amp;gt;International Journal of Computer Vision&amp;lt;/em&amp;gt; 88, no. 2 (June 2010): 303–38. &amp;lt;a href=&quot;https://doi.org/10.1007/s11263-009-0275-4&quot;&amp;gt;https://doi.org/10.1007/s11263-009-0275-4&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;8&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; We do not know when the first successful image classification systems were developed. In a blog post, Amidi &amp;amp; Amidi point to LeNet as pioneering work in image classification&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-9-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-9-2683&quot; title=&#039;See section &amp;amp;#8216;LeNet&amp;amp;#8217;.&amp;lt;br&amp;gt;&amp;lt;br&amp;gt;“The Evolution of Image Classification Explained.” Accessed October 19, 2020. &amp;lt;a href=&quot;https://stanford.edu/~shervine/blog/evolution-image-classification-explained#lenet&quot;&amp;gt;https://stanford.edu/~shervine/blog/evolution-image-classification-explained#lenet&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;9&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;, and it appears to have been developed in 1998.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-10-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-10-2683&quot; title=&#039;&amp;amp;#8220;&amp;lt;strong&amp;gt;LeNet&amp;lt;/strong&amp;gt;&amp;amp;nbsp;is a&amp;amp;nbsp;&amp;lt;a href=&quot;https://en.wikipedia.org/wiki/Convolutional_neural_network&quot;&amp;gt;convolutional neural network&amp;lt;/a&amp;gt;&amp;amp;nbsp;structure proposed by&amp;amp;nbsp;&amp;lt;a href=&quot;https://en.wikipedia.org/wiki/Yann_LeCun&quot;&amp;gt;Yann LeCun&amp;lt;/a&amp;gt;&amp;amp;nbsp;et al. in 1998.&amp;amp;#8221;&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;“LeNet.” In &amp;lt;em&amp;gt;Wikipedia&amp;lt;/em&amp;gt;, June 19, 2020. &amp;lt;a href=&quot;https://en.wikipedia.org/w/index.php?title=LeNet&amp;amp;amp;oldid=963418885&quot;&amp;gt;https://en.wikipedia.org/w/index.php?title=LeNet&amp;amp;amp;oldid=963418885&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;10&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + === Beginner level === + + + &lt;HTML&gt; + &lt;p&gt;The first entrant in the ImageNet contest to perform better than our beginner level benchmark was SuperVision (commonly known as AlexNet) in 2012, with a 15.3% error rate.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-11-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-11-2683&quot; title=&#039;&amp;amp;#8220;We also entered a variant of this model in the&amp;lt;br&amp;gt;ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%&amp;amp;#8221;&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E Hinton. “ImageNet Classification with Deep Convolutional Neural Networks.” In &amp;lt;em&amp;gt;Advances in Neural Information Processing Systems 25&amp;lt;/em&amp;gt;, edited by F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, 1097–1105. Curran Associates, Inc., 2012. &amp;lt;a href=&quot;http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf&quot;&amp;gt;http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf&amp;lt;/a&amp;gt;.&amp;lt;br&amp;gt;&amp;lt;br&amp;gt;Also, see Table 6 for a list of other entrants: &amp;lt;br&amp;gt;&amp;lt;br&amp;gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &amp;lt;em&amp;gt;ArXiv:1409.0575 [Cs]&amp;lt;/em&amp;gt;, January 29, 2015. &amp;lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&amp;gt;http://arxiv.org/abs/1409.0575&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;11&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + === Superhuman level === + + + &lt;HTML&gt; + &lt;p&gt;In 2015 He et al apparently achieved a 4.5% error rate, slightly better than our high human benchmark.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-12-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-12-2683&quot; title=&#039;&amp;amp;#8220;Our 152-layer ResNet has a single-model top-5 validation error of 4.49%.&amp;amp;#8221; &amp;lt;br&amp;gt;&amp;lt;br&amp;gt;Also see Table 4&amp;lt;br&amp;gt;&amp;lt;br&amp;gt;He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. “Deep Residual Learning for Image Recognition.” &amp;lt;em&amp;gt;ArXiv:1512.03385 [Cs]&amp;lt;/em&amp;gt;, December 10, 2015. &amp;lt;a href=&quot;http://arxiv.org/abs/1512.03385&quot;&amp;gt;http://arxiv.org/abs/1512.03385&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;12&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + === Current level === + + + &lt;HTML&gt; + &lt;p&gt;According to paperswithcode.com, performance has continued to climb, to 2020, though slower than earlier.&lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-13-2683&quot;&gt;&lt;/span&gt;&lt;span class=&quot;easy-footnote&quot;&gt;&lt;a href=&quot;#easy-footnote-bottom-13-2683&quot; title=&#039;See figure:&amp;lt;/p&amp;gt; &amp;lt;p&amp;gt;“Papers with Code &amp;amp;#8211; ImageNet Benchmark (Image Classification).” Accessed October 19, 2020. &amp;lt;a href=&quot;https://paperswithcode.com/sota/image-classification-on-imagenet&quot;&amp;gt;https://paperswithcode.com/sota/image-classification-on-imagenet&amp;lt;/a&amp;gt;.&#039;&gt;&lt;sup&gt;13&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + ==== Times for AI to cross human-relative ranges  ==== + + + &lt;HTML&gt; + &lt;p&gt;Given the above dates, we have:&lt;/p&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;figure class=&quot;wp-block-table&quot;&gt; + &lt;table&gt; + &lt;tbody&gt; + &lt;tr&gt; + &lt;td&gt;Range&lt;/td&gt; + &lt;td&gt;Start&lt;/td&gt; + &lt;td&gt;End&lt;/td&gt; + &lt;td&gt;Duration (years)&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt;First attempt to beginner level&lt;/td&gt; + &lt;td&gt;&amp;lt;1998&lt;/td&gt; + &lt;td&gt;2012&lt;/td&gt; + &lt;td&gt;&amp;gt;14&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt;Beginner to superhuman&lt;/td&gt; + &lt;td&gt;2012&lt;/td&gt; + &lt;td&gt;2015&lt;/td&gt; + &lt;td&gt;3&lt;/td&gt; + &lt;/tr&gt; + &lt;tr&gt; + &lt;td&gt;Above superhuman&lt;/td&gt; + &lt;td&gt;2015&lt;/td&gt; + &lt;td&gt;&amp;gt;2020&lt;/td&gt; + &lt;td&gt;&amp;gt;5&lt;/td&gt; + &lt;/tr&gt; + &lt;/tbody&gt; + &lt;/table&gt; + &lt;/figure&gt; + &lt;/HTML&gt; + + + &lt;HTML&gt; + &lt;p&gt;&lt;em&gt;Primary author: Rick Korzekwa&lt;/em&gt;&lt;/p&gt; + &lt;/HTML&gt; + + + ===== Notes ===== + + + &lt;HTML&gt; + &lt;ol class=&quot;easy-footnotes-wrapper&quot;&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-1-2683&quot;&gt;&lt;/span&gt;“&lt;strong&gt;ImageNet&lt;/strong&gt; is an image database organized according to the &lt;a href=&quot;http://wordnet.princeton.edu/&quot; rel=&quot;noreferrer noopener&quot; target=&quot;_blank&quot;&gt;WordNet&lt;/a&gt; hierarchy (currently only the nouns), in which each node of the hierarchy is depicted by hundreds and thousands of images. Currently we have an average of over five hundred images per node. “ + &lt;p&gt;“ImageNet.” Accessed October 19, 2020. &lt;a href=&quot;http://www.image-net.org/&quot;&gt;http://www.image-net.org/&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-1-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-2-2683&quot;&gt;&lt;/span&gt;Karpathy, Andrej. “Ilsvrc.” Accessed October 19, 2020. &lt;a href=&quot;https://cs.stanford.edu/people/karpathy/ilsvrc/&quot;&gt;https://cs.stanford.edu/people/karpathy/ilsvrc/&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-2-2683&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-3-2683&quot;&gt;&lt;/span&gt;‘Therefore, in evaluating the human accuracy we relied primarily on expert annotators who learned to recognize a large portion of the 1000 ILSVRC classes. During training, the annotators labeled a few hundred validation images for practice and later switched to the test set images’ + &lt;p&gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &lt;em&gt;ArXiv:1409.0575 [Cs]&lt;/em&gt;, January 29, 2015. &lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&gt;http://arxiv.org/abs/1409.0575&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-3-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-4-2683&quot;&gt;&lt;/span&gt;“Annotator A1 evaluated a total of 1500 test set images. The GoogLeNet classication error on this sample was estimated to be 6.8% (recall that the error on full test set of 100,000 images is 6.7%, as shown in Table 7). The human error was estimated to be 5.1%.”&lt;br/&gt; + &lt;br/&gt; + Also see Table 9 + &lt;p&gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &lt;em&gt;ArXiv:1409.0575 [Cs]&lt;/em&gt;, January 29, 2015. &lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&gt;http://arxiv.org/abs/1409.0575&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-4-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-5-2683&quot;&gt;&lt;/span&gt;“They presented their database for the first time as a poster at the 2009 &lt;a href=&quot;https://en.wikipedia.org/wiki/Conference_on_Computer_Vision_and_Pattern_Recognition&quot;&gt;Conference on Computer Vision and Pattern Recognition&lt;/a&gt; (CVPR) in Florida.”&lt;br/&gt; + &lt;br/&gt; + “ImageNet.” In &lt;em&gt;Wikipedia&lt;/em&gt;, September 9, 2020. &lt;a href=&quot;https://en.wikipedia.org/w/index.php?title=ImageNet&amp;amp;oldid=977585441&quot;&gt;https://en.wikipedia.org/w/index.php?title=ImageNet&amp;amp;oldid=977585441&lt;/a&gt;.&lt;br/&gt; + &lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-5-2683&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-6-2683&quot;&gt;&lt;/span&gt;“…The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) has been running annually for five years (since 2010) and has become the standard benchmark for large-scale object recognition.” + &lt;p&gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &lt;em&gt;ArXiv:1409.0575 [Cs]&lt;/em&gt;, January 29, 2015. &lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&gt;http://arxiv.org/abs/1409.0575&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-6-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-7-2683&quot;&gt;&lt;/span&gt;See table 6. + &lt;p&gt;Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &lt;em&gt;ArXiv:1409.0575 [Cs]&lt;/em&gt;, January 29, 2015. &lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&gt;http://arxiv.org/abs/1409.0575&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-7-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-8-2683&quot;&gt;&lt;/span&gt;“The PASCAL Visual Object Classes (VOC) challenge is a benchmark in visual object category recognition and detection, providing the vision and machine learning communities with a standard dataset of images and annotation, and standard evaluation procedures. Organised annually from 2005 to present, the challenge and its associated dataset has become accepted as the benchmark for object detection.” + &lt;p&gt;Everingham, Mark, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. “The Pascal Visual Object Classes (VOC) Challenge.” &lt;em&gt;International Journal of Computer Vision&lt;/em&gt; 88, no. 2 (June 2010): 303–38. &lt;a href=&quot;https://doi.org/10.1007/s11263-009-0275-4&quot;&gt;https://doi.org/10.1007/s11263-009-0275-4&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-8-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-9-2683&quot;&gt;&lt;/span&gt;See section ‘LeNet’.&lt;br/&gt; + &lt;br/&gt; + “The Evolution of Image Classification Explained.” Accessed October 19, 2020. &lt;a href=&quot;https://stanford.edu/~shervine/blog/evolution-image-classification-explained#lenet&quot;&gt;https://stanford.edu/~shervine/blog/evolution-image-classification-explained#lenet&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-9-2683&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-10-2683&quot;&gt;&lt;/span&gt;“&lt;strong&gt;LeNet&lt;/strong&gt; is a &lt;a href=&quot;https://en.wikipedia.org/wiki/Convolutional_neural_network&quot;&gt;convolutional neural network&lt;/a&gt; structure proposed by &lt;a href=&quot;https://en.wikipedia.org/wiki/Yann_LeCun&quot;&gt;Yann LeCun&lt;/a&gt; et al. in 1998.” + &lt;p&gt;“LeNet.” In &lt;em&gt;Wikipedia&lt;/em&gt;, June 19, 2020. &lt;a href=&quot;https://en.wikipedia.org/w/index.php?title=LeNet&amp;amp;oldid=963418885&quot;&gt;https://en.wikipedia.org/w/index.php?title=LeNet&amp;amp;oldid=963418885&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-10-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-11-2683&quot;&gt;&lt;/span&gt;“We also entered a variant of this model in the&lt;br/&gt; + ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%” + &lt;p&gt;Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E Hinton. “ImageNet Classification with Deep Convolutional Neural Networks.” In &lt;em&gt;Advances in Neural Information Processing Systems 25&lt;/em&gt;, edited by F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, 1097–1105. Curran Associates, Inc., 2012. &lt;a href=&quot;http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf&quot;&gt;http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf&lt;/a&gt;.&lt;br/&gt; + &lt;br/&gt; + Also, see Table 6 for a list of other entrants:&lt;br/&gt; + &lt;br/&gt; + Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, et al. “ImageNet Large Scale Visual Recognition Challenge.” &lt;em&gt;ArXiv:1409.0575 [Cs]&lt;/em&gt;, January 29, 2015. &lt;a href=&quot;http://arxiv.org/abs/1409.0575&quot;&gt;http://arxiv.org/abs/1409.0575&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-11-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-12-2683&quot;&gt;&lt;/span&gt;“Our 152-layer ResNet has a single-model top-5 validation error of 4.49%.”&lt;br/&gt; + &lt;br/&gt; + Also see Table 4&lt;br/&gt; + &lt;br/&gt; + He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. “Deep Residual Learning for Image Recognition.” &lt;em&gt;ArXiv:1512.03385 [Cs]&lt;/em&gt;, December 10, 2015. &lt;a href=&quot;http://arxiv.org/abs/1512.03385&quot;&gt;http://arxiv.org/abs/1512.03385&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-12-2683&quot;&gt;&lt;/a&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;li&gt;&lt;div class=&quot;li&quot;&gt; + &lt;span class=&quot;easy-footnote-margin-adjust&quot; id=&quot;easy-footnote-bottom-13-2683&quot;&gt;&lt;/span&gt;See figure: + &lt;p&gt;“Papers with Code – ImageNet Benchmark (Image Classification).” Accessed October 19, 2020. &lt;a href=&quot;https://paperswithcode.com/sota/image-classification-on-imagenet&quot;&gt;https://paperswithcode.com/sota/image-classification-on-imagenet&lt;/a&gt;.&lt;a class=&quot;easy-footnote-to-top&quot; href=&quot;#easy-footnote-13-2683&quot;&gt;&lt;/a&gt;&lt;/p&gt; + &lt;/div&gt;&lt;/li&gt; + &lt;/ol&gt; + &lt;/HTML&gt; + + </pre>