Effect of AlexNet on historic trends in image recognition

AlexNet did not represent a greater than 10-year discontinuity in fraction of images labeled incorrectly, or log or inverse of this error rate, relative to progress in the past two years of competition data.

Details

This case study is part of AI Impacts’ discontinuous progress investigation.

Background

The annual ImageNet competition asks researchers to build programs to label images.¹ It began in 2010, when every team labeled at least 25% of images wrong. The same was true in 2011, and would have been true in 2012, if not for AlexNet, a convolutional neural network that mislabeled only 16.4% of images.²

Trends

Percent of images mislabeled

Data

We collected data on the error rate (%) of the 2010 – 2012 ImageNet competitors from Table 6 of Russakovsky et al³ into this spreadsheet. See Figure 1 below.

Discontinuity measurement

The ImageNet competition had only been going for two years when AlexNet entered, so the past trend is very short. Given this, the shape of the curve prior to AlexNet is entirely ambiguous. We treat the trend as linear for simplicity, but given that, it is better to choose a transformation of the data that we expect to be linear, given our understanding of the situation.

Two plausible transformations are the log of the error, and the reciprocal of the error rate.⁴ These two transformations of the data are shown in Figures 2 and 3 below.

The best 2012 AlexNet competitor gives us discontinuous jumps of 3 years of progress at previous rates for the raw error rate, 4 years of progress at previous rates for log base 2 of the error rate, or 6 years of progress at previous rates for 1 / the error rate.⁵ For the 6-year discontinuity, we tabulated a number of other potentially relevant metrics in the ‘Notable discontinuities under 10 years’ tab here.

Notes

1. “Since 2010, the ImageNet project runs an annual software contest, the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), where software programs compete to correctly classify and detect objects and scenes.” – “Imagenet”. 2019. En.Wikipedia.Org. Accessed June 20 2019. https://en.wikipedia.org/w/index.php?title=ImageNet&oldid=900080629.
2. See our data section for sources for this data.
3. Olga Russakovsky et al., “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision 115, no. 3 (December 1, 2015): 211–52, https://doi.org/10.1007/s11263-015-0816-y.
4. Percentage of answers incorrect seems unlikely to change linearly over time, since we expect moving from 50% incorrect to 49% incorrect to be easier than halving a 2% error rate. Log of the error rate and inverse of the error rate seem to us more plausible.
5. See our methodology page for more details our spreadsheet for calculation.