|
|
|
|
— |
takeoff_speed:continuity_of_progress:effect_of_alexnet_on_historic_trends_in_image_recognition [2022/09/21 07:37] (current)
|
| | ====== Effect of AlexNet on historic trends in image recognition ====== |
| | |
| | // Published 07 February, 2020; last updated 23 April, 2020 // |
| | |
| | <HTML> |
| | <p>AlexNet did not represent a greater than 10-year discontinuity in fraction of images labeled incorrectly, or log or inverse of this error rate, relative to progress in the past two years of competition data.</p> |
| | </HTML> |
| | |
| | |
| | |
| | ===== Details ===== |
| | |
| | |
| | <HTML> |
| | <p>This case study is part of AI Impacts’ <a href="/doku.php?id=ai_timelines:discontinuous_progress_investigation">discontinuous progress investigation</a>.</p> |
| | </HTML> |
| | |
| | |
| | ==== Background ==== |
| | |
| | |
| | <HTML> |
| | <p>The annual ImageNet competition asks researchers to build programs to label images.<span class="easy-footnote-margin-adjust" id="easy-footnote-1-1382"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-1-1382" title="&#8220;Since 2010, the ImageNet project runs an annual software contest, the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), where software programs compete to correctly classify and detect objects and scenes.&#8221; &#8211; &#8220;Imagenet&#8221;. 2019.&nbsp;<em>En.Wikipedia.Org</em>. Accessed June 20 2019. https://en.wikipedia.org/w/index.php?title=ImageNet&amp;oldid=900080629."><sup>1</sup></a></span> It began in 2010, when every team labeled at least 25% of images wrong. The same was true in 2011, and would have been true in 2012, if not for AlexNet, a <a href="https://en.wikipedia.org/wiki/Convolutional_neural_network">convolutional neural network</a> that mislabeled only 16.4% of images.<span class="easy-footnote-margin-adjust" id="easy-footnote-2-1382"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-2-1382" title='See our <a href="#mce_21"><strong>data section</strong></a> for sources for this data.'><sup>2</sup></a></span></p> |
| | </HTML> |
| | |
| | |
| | ==== Trends ==== |
| | |
| | |
| | === Percent of images mislabeled === |
| | |
| | |
| | == Data == |
| | |
| | |
| | <HTML> |
| | <p>We collected data on the error rate (%) of the 2010 – 2012 ImageNet competitors from Table 6 of Russakovsky et al<span class="easy-footnote-margin-adjust" id="easy-footnote-3-1382"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-3-1382" title='Olga Russakovsky et al., “<a href="https://link.springer.com/article/10.1007%2Fs11263-015-0816-y">ImageNet Large Scale Visual Recognition Challenge</a>,” <em>International Journal of Computer Vision</em> 115, no. 3 (December 1, 2015): 211–52, <a href="https://doi.org/10.1007/s11263-015-0816-y">https://doi.org/10.1007/s11263-015-0816-y</a>.'><sup>3</sup></a></span> into <a href="https://docs.google.com/spreadsheets/d/1HYdv4gLdtwkzYKeXaBJTXBbqeX_9onmw4aAaYVWVUfs/edit?usp=sharing">this spreadsheet</a>. See Figure 1 below.</p> |
| | </HTML> |
| | |
| | |
| | <HTML> |
| | <figure class="wp-block-image size-full is-resized"> |
| | <img alt="" class="wp-image-2288" height="450" src="https://aiimpacts.org/wp-content/uploads/2020/02/ErrorRate.png" width="600"/> |
| | <figcaption> |
| | Figure 1: Error rate (%) of ImageNet competitors from 2010 – 2012 |
| | </figcaption> |
| | </figure> |
| | </HTML> |
| | |
| | |
| | == Discontinuity measurement == |
| | |
| | |
| | <HTML> |
| | <p>The ImageNet competition had only been going for two years when AlexNet entered, so the past trend is very short. Given this, the shape of the curve prior to AlexNet is entirely ambiguous. We treat the trend as linear for simplicity, but given that, it is better to choose a transformation of the data that we expect to be linear, given our understanding of the situation.</p> |
| | </HTML> |
| | |
| | |
| | <HTML> |
| | <p>Two plausible transformations are the log of the error, and the reciprocal of the error rate.<span class="easy-footnote-margin-adjust" id="easy-footnote-4-1382"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-4-1382" title="Percentage of answers incorrect seems unlikely to change linearly over time, since we expect moving from 50% incorrect to 49% incorrect to be easier than halving a 2% error rate. Log of the error rate and inverse of the error rate seem to us more plausible."><sup>4</sup></a></span> These two transformations of the data are shown in Figures 2 and 3 below.</p> |
| | </HTML> |
| | |
| | |
| | <HTML> |
| | <figure class="wp-block-image is-resized"> |
| | <img alt="" height="360" loading="lazy" src="https://lh3.googleusercontent.com/D9XDAoRH9tDVn8usb1XzCUsOdNY_-cNssrBHe1sKeXMbj5fT1wNcojVsoY7ydgpkU_s1mC9vPHvNV2nSWZufkr48fnmKjBonMxyK06Tk1VNoUpgSZzzJIt19TT22u5gUQMV76hfV" width="583"/> |
| | <figcaption> |
| | Figure 2: Log base 2 / error rate of ImageNet competitors from 2010 – 2012<br/> |
| | </figcaption> |
| | </figure> |
| | </HTML> |
| | |
| | |
| | <HTML> |
| | <figure class="wp-block-image is-resized"> |
| | <img alt="" height="361" loading="lazy" src="https://lh5.googleusercontent.com/a-EUqvmgOprtUu--eoK-ZAkdNTtYkE9Mo9Ir-e219QsAWeOQAcfwwqsWv_Xsi-zDJ8vGVkKbeBRShPL4IbCP6zt2GWzkj3YM4AMbpMfrD-0_tfyq2jasOp5ZXq_q7ooaRVT5lfDx" width="585"/> |
| | <figcaption> |
| | Figure 3: 1 / error rate of ImageNet competitors from 2010 – 2012 |
| | </figcaption> |
| | </figure> |
| | </HTML> |
| | |
| | |
| | <HTML> |
| | <p>The best 2012 AlexNet competitor gives us discontinuous jumps of 3 years of progress at previous rates for the raw error rate, 4 years of progress at previous rates for log base 2 of the error rate, or 6 years of progress at previous rates for 1 / the error rate.<span class="easy-footnote-margin-adjust" id="easy-footnote-5-1382"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-5-1382" title='See <strong><a href="https://aiimpacts.org/methodology-for-discontinuity-investigation/#discontinuity-measurement">our methodology page</a></strong> for more details <a href="https://docs.google.com/spreadsheets/d/1HYdv4gLdtwkzYKeXaBJTXBbqeX_9onmw4aAaYVWVUfs/edit?usp=sharing"><strong>our spreadsheet</strong></a> for calculation.'><sup>5</sup></a></span> For the 6-year discontinuity, we tabulated a number of other potentially relevant metrics in the ‘Notable discontinuities under 10 years’ tab <strong><a href="https://docs.google.com/spreadsheets/d/1iMIZ57Ka9-ZYednnGeonC-NqwGC7dKiHN9S-TAxfVdQ/edit?usp=sharing">here</a></strong>.</p> |
| | </HTML> |
| | |
| | |
| | ===== Notes ===== |
| | |
| | |
| | <HTML> |
| | <ol class="easy-footnotes-wrapper"> |
| | <li><div class="li"> |
| | <span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-1-1382"></span>“Since 2010, the ImageNet project runs an annual software contest, the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), where software programs compete to correctly classify and detect objects and scenes.” – “Imagenet”. 2019. <em>En.Wikipedia.Org</em>. Accessed June 20 2019. https://en.wikipedia.org/w/index.php?title=ImageNet&oldid=900080629.<a class="easy-footnote-to-top" href="#easy-footnote-1-1382"></a> |
| | </div></li> |
| | <li><div class="li"> |
| | <span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-2-1382"></span>See our <a href="#mce_21"><strong>data section</strong></a> for sources for this data.<a class="easy-footnote-to-top" href="#easy-footnote-2-1382"></a> |
| | </div></li> |
| | <li><div class="li"> |
| | <span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-3-1382"></span>Olga Russakovsky et al., “<a href="https://link.springer.com/article/10.1007%2Fs11263-015-0816-y">ImageNet Large Scale Visual Recognition Challenge</a>,” <em>International Journal of Computer Vision</em> 115, no. 3 (December 1, 2015): 211–52, <a href="https://doi.org/10.1007/s11263-015-0816-y">https://doi.org/10.1007/s11263-015-0816-y</a>.<a class="easy-footnote-to-top" href="#easy-footnote-3-1382"></a> |
| | </div></li> |
| | <li><div class="li"> |
| | <span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-4-1382"></span>Percentage of answers incorrect seems unlikely to change linearly over time, since we expect moving from 50% incorrect to 49% incorrect to be easier than halving a 2% error rate. Log of the error rate and inverse of the error rate seem to us more plausible.<a class="easy-footnote-to-top" href="#easy-footnote-4-1382"></a> |
| | </div></li> |
| | <li><div class="li"> |
| | <span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-5-1382"></span>See <strong><a href="/doku.php?id=speed_of_ai_transition:pace_of_ai_progress_without_feedback:historical_continuity_of_progress:methodology_for_discontinuous_progress_investigation#discontinuity-measurement">our methodology page</a></strong> for more details <a href="https://docs.google.com/spreadsheets/d/1HYdv4gLdtwkzYKeXaBJTXBbqeX_9onmw4aAaYVWVUfs/edit?usp=sharing"><strong>our spreadsheet</strong></a> for calculation.<a class="easy-footnote-to-top" href="#easy-footnote-5-1382"></a> |
| | </div></li> |
| | </ol> |
| | </HTML> |
| | |
| | |
| |
