AI Impacts Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Trend in compute used in training for headline AI results ======

// Published 17 May, 2018 //

<HTML>
<p>Compute used in the largest AI training runs appears to have roughly doubled every 3.5 months between 2012 and 2018.</p>
</HTML>


===== Details =====


<HTML>
<p>According to <a href="https://blog.openai.com/ai-and-compute/">Amodei and Hernandez, on the OpenAI Blog</a>:</p>
</HTML>


<HTML>
<blockquote>
<p>…since 2012, the amount of compute used in the largest AI training runs has been increasing exponentially with a 3.5 month-doubling time (by comparison, Moore’s Law <a href="https://www.nature.com/articles/s41928-017-0005-9">had</a> an 18-month doubling period). Since 2012, this metric has grown by more than 300,000x (an 18-month doubling period would yield only a 12x increase)…</p>
</blockquote>
</HTML>


<HTML>
<p>They give the following figure, and some of their calculations. We have not verified their calculations, or looked for other reports on this issue.</p>
</HTML>


<HTML>
<figure aria-describedby="caption-attachment-1159" class="wp-caption alignnone" id="attachment_1159" style="width: 1636px">
<a href="http://aiimpacts.org/wp-content/uploads/2018/05/OpenAI-AI-Compute-Trend.jpeg"><img alt="" class="wp-image-1159 size-full" height="1088" sizes="(max-width: 1636px) 100vw, 1636px" src="https://aiimpacts.org/wp-content/uploads/2018/05/OpenAI-AI-Compute-Trend.jpeg" srcset="https://aiimpacts.org/wp-content/uploads/2018/05/OpenAI-AI-Compute-Trend.jpeg 1636w, https://aiimpacts.org/wp-content/uploads/2018/05/OpenAI-AI-Compute-Trend-300x200.jpeg 300w, https://aiimpacts.org/wp-content/uploads/2018/05/OpenAI-AI-Compute-Trend-768x511.jpeg 768w, https://aiimpacts.org/wp-content/uploads/2018/05/OpenAI-AI-Compute-Trend-1024x681.jpeg 1024w" width="1636"/></a>
<figcaption class="wp-caption-text" id="caption-attachment-1159">
<strong>Figure 1:</strong> Originally captioned: The chart shows the total amount of compute, in petaflop/s-days, that was used to train selected results that are relatively well known, used a lot of compute for their time, and gave enough information to estimate the compute used. A petaflop/s-day (pfs-day) consists of performing 1015 neural net operations per second for one day, or a total of about 1020 operations. The compute-time product serves as a mental convenience, similar to kW-hr for energy. We don’t measure peak theoretical FLOPS of the hardware but instead try to estimate the number of actual operations performed. We count adds and multiplies as separate operations, we count any add or multiply as a single operation regardless of numerical precision (making “FLOP” a slight misnomer), and we ignore <a href="http://web.engr.oregonstate.edu/~tgd/publications/mcs-ensembles.pdf">ensemble models</a>. Example calculations that went into this graph are provided in this <a href="https://blog.openai.com/ai-and-compute/#appendixmethods">appendix</a>. Doubling time for line of best fit shown is 3.43 months.
                </figcaption>
</figure>
</HTML>
AI Impacts Wiki

User Tools

Site Tools

Page Tools
