Published 07 February, 2020; last updated 23 April, 2020
AlexNet did not represent a greater than 10-year discontinuity in fraction of images labeled incorrectly, or log or inverse of this error rate, relative to progress in the past two years of competition data.
This case study is part of AI Impacts’ discontinuous progress investigation.
The annual ImageNet competition asks researchers to build programs to label images.1 It began in 2010, when every team labeled at least 25% of images wrong. The same was true in 2011, and would have been true in 2012, if not for AlexNet, a convolutional neural network that mislabeled only 16.4% of images.2
The ImageNet competition had only been going for two years when AlexNet entered, so the past trend is very short. Given this, the shape of the curve prior to AlexNet is entirely ambiguous. We treat the trend as linear for simplicity, but given that, it is better to choose a transformation of the data that we expect to be linear, given our understanding of the situation.
Two plausible transformations are the log of the error, and the reciprocal of the error rate.4 These two transformations of the data are shown in Figures 2 and 3 below.
The best 2012 AlexNet competitor gives us discontinuous jumps of 3 years of progress at previous rates for the raw error rate, 4 years of progress at previous rates for log base 2 of the error rate, or 6 years of progress at previous rates for 1 / the error rate.5 For the 6-year discontinuity, we tabulated a number of other potentially relevant metrics in the ‘Notable discontinuities under 10 years’ tab here.