AI Impacts Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== 2019 recent trends in GPU price per FLOPS ======

// Published 25 March, 2020; last updated 09 June, 2020 //

<HTML>
<p>We estimate that in recent years, GPU prices have fallen at rates that would yield an order of magnitude over roughly:</p>
</HTML>


<HTML>
<ul>
<li><div class="li">17 years for single-precision FLOPS</div></li>
<li><div class="li">10 years for half-precision FLOPS</div></li>
<li><div class="li">5 years for half-precision fused multiply-add FLOPS</div></li>
</ul>
</HTML>



====== Details ======


<HTML>
<p>GPUs (graphics processing units) are specialized electronic circuits originally used for computer graphics.<span class="easy-footnote-margin-adjust" id="easy-footnote-1-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-1-2316" title='“A graphics processing unit (GPU) is a specialized electronic circuit designed to rapidly manipulate and alter memory to accelerate the creation of images in a frame buffer intended for output to a display device … Modern GPUs are very efficient at manipulating computer graphics and image processing … The term was popularized by Nvidia in 1999, who marketed the GeForce 256 as &amp;#8220;the world&amp;#8217;s first GPU&amp;#8221;. It was presented as a &amp;#8220;single-chip processor with integrated transform, lighting, triangle setup/clipping, and rendering engines&amp;#8221;.”&lt;br&gt;“Graphics Processing Unit.” Wikipedia. Wikimedia Foundation, March 24, 2020. &lt;a href="https://en.wikipedia.org/w/index.php?title=Graphics_processing_unit&amp;amp;oldid=947270104"&gt;https://en.wikipedia.org/w/index.php?title=Graphics_processing_unit&amp;amp;oldid=947270104&lt;/a&gt;.'><sup>1</sup></a></span> In recent years, they have been popularly used for machine learning applications.<span class="easy-footnote-margin-adjust" id="easy-footnote-2-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-2-2316" title='Fraenkel, Bernard. “Council Post: For Machine Learning, It&amp;#8217;s All About GPUs.” Forbes. Forbes Magazine, December 8, 2017. &lt;a href="https://www.forbes.com/sites/forbestechcouncil/2017/12/01/for-machine-learning-its-all-about-gpus/#5ed90c227699"&gt;https://www.forbes.com/sites/forbestechcouncil/2017/12/01/for-machine-learning-its-all-about-gpus/#5ed90c227699&lt;/a&gt;.'><sup>2</sup></a></span> One measure of GPU performance is FLOPS, the number of operations on floating-point numbers a GPU can perform in a second.<span class="easy-footnote-margin-adjust" id="easy-footnote-3-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-3-2316" title='“In computing, floating point operations per second (FLOPS, flops or flop/s) is a measure of computer performance, useful in fields of scientific computations that require floating-point calculations. For such cases it is a more accurate measure than measuring instructions per second.”&lt;br&gt;“FLOPS.” Wikipedia. Wikimedia Foundation, March 24, 2020. &lt;a href="https://en.wikipedia.org/w/index.php?title=FLOPS&amp;amp;oldid=947177339"&gt;https://en.wikipedia.org/w/index.php?title=FLOPS&amp;amp;oldid=947177339&lt;/a&gt;'><sup>3</sup></a></span> This page looks at the trends in GPU price / FLOPS of theoretical peak performance over the past 13 years. It does not include the cost of operating the GPUs, and it does not consider GPUs rented through cloud computing.</p>
</HTML>


===== Theoretical peak performance =====


<HTML>
<p>‘Theoretical peak performance’ numbers appear to be determined by adding together the theoretical performances of the processing components of the GPU, which are calculated by multiplying the clock speed of the component by the number of instructions it can perform per cycle.<span class="easy-footnote-margin-adjust" id="easy-footnote-4-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-4-2316" title='From this discussion on Nvidia&amp;#8217;s forums about theoretical GFLOPS: “GPU theoretical flops calculation is similar conceptually. It will vary by GPU just as the CPU calculation varies by CPU architecture and model. To use K40m as an example: http://www.nvidia.com/content/PDF/kepler/Tesla-K40-PCIe-Passive-Board-Spec-BD-06902-001_v05.pdf&lt;br&gt;&lt;/p&gt; &lt;p&gt;there are 15 SMs (2880/192), each with 64 DP ALUs that are capable of retiring one DP FMA instruction per cycle (== 2 DP Flops per cycle).&lt;br&gt;&lt;/p&gt; &lt;p&gt;15 x 64 x 2 * 745MHz = 1.43 TFlops/sec&lt;br&gt;&lt;/p&gt; &lt;p&gt;which is the stated perf:&lt;/p&gt; &lt;p&gt;http://www.nvidia.com/content/tesla/pdf/NVIDIA-Tesla-Kepler-Family-Datasheet.pdf &amp;#8220;&lt;/p&gt; &lt;p&gt;Person. “Comparing CPU and GPU Theoretical GFLOPS.” NVIDIA Developer Forums, May 21, 2014. &lt;a href="https://forums.developer.nvidia.com/t/comparing-cpu-and-gpu-theoretical-gflops/33335"&gt;https://forums.developer.nvidia.com/t/comparing-cpu-and-gpu-theoretical-gflops/33335&lt;/a&gt;.'><sup>4</sup></a></span> These numbers are given by the developer and may not reflect actual performance on a given application.<span class="easy-footnote-margin-adjust" id="easy-footnote-5-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-5-2316" title='From this blog post on the performance of TensorCores, a component of new Nvidia GPUs specialized for deep learning: “The problem is it’s totally unclear how to approach the peak performance of 120 TFLOPS, and as far as I know, no one could achieve so significant speedup on real tasks. Let me know if you aware of good cases.&amp;#8221;&lt;br&gt;Sapunov, Grigory. “Hardware for Deep Learning. Part 3: GPU.” Medium. Intento, January 20, 2020. &lt;a href="https://blog.inten.to/hardware-for-deep-learning-part-3-gpu-8906c1644664"&gt;https://blog.inten.to/hardware-for-deep-learning-part-3-gpu-8906c1644664&lt;/a&gt;.'><sup>5</sup></a></span></p>
</HTML>


===== Metrics =====


<HTML>
<p>We collected data on multiple slightly different measures of GPU price and FLOPS performance.</p>
</HTML>


==== Price metrics ====


<HTML>
<p>GPU prices are divided into release prices, which reflect the manufacturer suggested retail prices that GPUs are originally sold at, and active prices, which are the prices at which GPUs are actually sold at over time, often by resellers.</p>
</HTML>


<HTML>
<p>We expect that active prices better represent prices available to hardware users, but collect release prices also, as supporting evidence.</p>
</HTML>


==== FLOPS performance metrics ====


<HTML>
<p>Several varieties of ‘FLOPS’ can be distinguished based on the specifics of the operations they involve. Here we are interested in single-precision FLOPS, half-precision FLOPS, and half-precision fused-multiply add FLOPS.</p>
</HTML>


<HTML>
<p>‘Single-precision’ and ‘half-precision’ refer to the number of bits used to specify a floating point number.<span class="easy-footnote-margin-adjust" id="easy-footnote-6-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-6-2316" title='Gupta, Geetika. “Difference Between Single-, Double-, Multi-, Mixed-Precision: NVIDIA Blog.” The Official NVIDIA Blog, November 21, 2019. &lt;a href="https://blogs.nvidia.com/blog/2019/11/15/whats-the-difference-between-single-double-multi-and-mixed-precision-computing/"&gt;https://blogs.nvidia.com/blog/2019/11/15/whats-the-difference-between-single-double-multi-and-mixed-precision-computing/&lt;/a&gt;.'><sup>6</sup></a></span> Using more bits to specify a number achieves greater precision at the cost of more computational steps per calculation. Our data suggests that GPUs have largely been improving in single-precision performance in recent decades,<span class="easy-footnote-margin-adjust" id="easy-footnote-7-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-7-2316" title='See our &lt;a href="https://aiimpacts.org/recent-trend-in-the-cost-of-computing/"&gt;2017 analysis&lt;/a&gt;, footnote 4, which notes that single-precision price performance seems to be improving while double-precision price performance is not'><sup>7</sup></a></span> and half-precision performance appears to be increasingly popular because it is adequate for deep learning.<span class="easy-footnote-margin-adjust" id="easy-footnote-8-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-8-2316" title='“With the growing importance of deep learning and energy-saving approximate computing, half precision floating point arithmetic (FP16) is fast gaining popularity. Nvidia&amp;#8217;s recent Pascal architecture was the first GPU that offered FP16 support.”&lt;br&gt;N. Ho and W. Wong, &lt;a href="https://ieeexplore.ieee.org/abstract/document/8091072"&gt;&amp;#8220;Exploiting half precision arithmetic in Nvidia GPUs,&amp;#8221;&lt;/a&gt; 2017 IEEE High Performance Extreme Computing Conference (HPEC), Waltham, MA, 2017, pp. 1-7.'><sup>8</sup></a></span></p>
</HTML>


<HTML>
<p>Nvidia, the main provider of chips for machine learning applications,<span class="easy-footnote-margin-adjust" id="easy-footnote-9-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-9-2316" title='“In a recent paper, Google revealed that its TPU can be up to 30x faster than a GPU for inference (the TPU can’t do training of neural networks). As the main provider of chips for machine learning applications, Nvidia took some issue with that, arguing that some of its existing inference chips were already highly competitive to the TPU.”&lt;br&gt;Armasu, Lucian. “On Tensors, Tensorflow, And Nvidia&amp;#8217;s Latest &amp;#8216;Tensor Cores&amp;#8217;.” Tom&amp;#8217;s Hardware. Tom&amp;#8217;s Hardware, May 11, 2017. &lt;a href="https://www.tomshardware.com/news/nvidia-tensor-core-tesla-v100,34384.html"&gt;https://www.tomshardware.com/news/nvidia-tensor-core-tesla-v100,34384.html&lt;/a&gt;.'><sup>9</sup></a></span> recently released a series of GPUs featuring Tensor Cores,<span class="easy-footnote-margin-adjust" id="easy-footnote-10-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-10-2316" title="“Tensor Cores in NVIDIA Volta GPU Architecture.” NVIDIA. Accessed May 2, 2020. https://www.nvidia.com/en-us/data-center/tensorcore/.&lt;br&gt;"><sup>10</sup></a></span> which claim to deliver “groundbreaking AI performance”. Tensor Core performance is measured in FLOPS, but they perform exclusively certain kinds of floating-point operations known as fused multiply-adds (FMAs).<span class="easy-footnote-margin-adjust" id="easy-footnote-11-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-11-2316" title='&amp;#8220;Volta is equipped with 640 Tensor Cores, each performing 64 floating-point fused-multiply-add (FMA) operations per clock. That delivers up to 125 TFLOPS for training and inference applications.”&lt;br&gt;“Tensor Cores in NVIDIA Volta GPU Architecture.” NVIDIA. Accessed March 25, 2020. &lt;a href="https://www.nvidia.com/en-us/data-center/tensorcore/"&gt;https://www.nvidia.com/en-us/data-center/tensorcore/&lt;/a&gt;.'><sup>11</sup></a></span> Performance on these operations is important for certain kinds of deep learning performance,<span class="easy-footnote-margin-adjust" id="easy-footnote-12-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-12-2316" title='&amp;#8220;A useful operation in computer linear algebra is multiply-add: calculating the sum of a value c with a product of other values a x b to produce c + a x b. Typically, thousands of such products may be summed in a single accumulator for a model such as ResNet-50, with many millions of independent accumulations when running a model in deployment, and quadrillions of these for training models.”&lt;br&gt;Johnson, Jeff. “Making Floating Point Math Highly Efficient for AI Hardware.” Facebook AI Blog, November 8, 2018. &lt;a href="https://ai.facebook.com/blog/making-floating-point-math-highly-efficient-for-ai-hardware/"&gt;https://ai.facebook.com/blog/making-floating-point-math-highly-efficient-for-ai-hardware/&lt;/a&gt;.'><sup>12</sup></a></span> so we track ‘GPU price / FMA FLOPS’ as well as ‘GPU price / FLOPS’.<br/></p>
</HTML>


<HTML>
<p>In addition to purely half-precision computations, Tensor Cores are capable of performing mixed-precision computations, where part of the computation is done in half-precision and part in single-precision.<span class="easy-footnote-margin-adjust" id="easy-footnote-13-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-13-2316" title='See Figure 2:&lt;br&gt;Gupta, Geetika. “Using Tensor Cores for Mixed-Precision Scientific Computing.” NVIDIA Developer Blog, April 19, 2019. &lt;a href="https://devblogs.nvidia.com/tensor-cores-mixed-precision-scientific-computing"&gt;https://devblogs.nvidia.com/tensor-cores-mixed-precision-scientific-computing&lt;/a&gt;/.'><sup>13</sup></a></span> Since explicitly mixed-precision-optimized hardware is quite recent, we don’t look at the trend in mixed-precision price performance, and only look at the trend in half-precision price performance.</p>
</HTML>


=== Precision tradeoffs ===


<HTML>
<p>Any GPU that performs multiple kinds of computations (single-precision, half-precision, half-precision fused multiply add) trades off performance on one for performance on the other, because there is limited space on the chip, and transistors must be allocated to either one type of computation or the other.<span class="easy-footnote-margin-adjust" id="easy-footnote-14-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-14-2316" title="Three different individuals told us about this constraint, including one Nvidia employee."><sup>14</sup></a></span> All current GPUs that perform half-precision or TensorCore fused-multiply-add computations also do single-precision computations, so they are splitting their transistor budget. For this reason, our impression is that half-precision FLOPS could be much cheaper now if entire GPUs were allocated to each one alone, rather than split between them.</p>
</HTML>


===== Release date prices =====


<HTML>
<p>We collected data on theoretical peak performance (FLOPS), release date, and price from several sources, including Wikipedia.<span class="easy-footnote-margin-adjust" id="easy-footnote-15-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-15-2316" title='See the ‘Source’ column in &lt;a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing"&gt;this spreadsheet&lt;/a&gt;, tab ‘GPU Data’. We largely used &lt;a href="https://www.techpowerup.com/"&gt;TechPowerUp&lt;/a&gt;, Wikipedia’s &lt;a href="https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units"&gt;List of Nvidia GPUs&lt;/a&gt;, &lt;a href="https://en.wikipedia.org/wiki/List_of_AMD_graphics_processing_units"&gt;List of AMD GPUs&lt;/a&gt;, and &lt;a href="https://docs.google.com/spreadsheets/d/1xAo6TcSgHdd25EdQ-6GqM0VKbTYu8cWyycgJhHRVIgY/edit#gid=0"&gt;this document listing GPU performance&lt;/a&gt;.'><sup>15</sup></a></span> (Data is available in <a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing">this spreadsheet</a>). We found GPUs by looking at Wikipedia’s existing large lists<span class="easy-footnote-margin-adjust" id="easy-footnote-16-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-16-2316" title='See Wikipedia’s &lt;a href="https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units"&gt;List of Nvidia GPUs&lt;/a&gt; and &lt;a href="https://en.wikipedia.org/wiki/List_of_AMD_graphics_processing_units"&gt;List of AMD GPUs&lt;/a&gt;.'><sup>16</sup></a></span> and by Googling “popular GPUs” and “popular deep learning GPUs”. We included any hardware that was labeled as a ‘GPU’. We adjusted prices for inflation based on the consumer price index.<span class="easy-footnote-margin-adjust" id="easy-footnote-17-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-17-2316" title="“CPI Home.” U.S. Bureau of Labor Statistics. U.S. Bureau of Labor Statistics. Accessed May 2, 2020. https://www.bls.gov/cpi/."><sup>17</sup></a></span></p>
</HTML>


<HTML>
<p>We were unable to find price and performance data for many popular GPUs and suspect that we are missing many from our list. In our search, we did not find any GPUs that beat our 2017 minimum of <nowiki>$</nowiki>0.03 (release price) / single-precision GFLOPS. We put out a <nowiki>$</nowiki>20 bounty on a popular Facebook group to find a cheaper GPU / FLOPS, and the bounty went unclaimed, so we are reasonably confident in this minimum.<span class="easy-footnote-margin-adjust" id="easy-footnote-18-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-18-2316" title="The Facebook group is for posting and claiming bounties and has around 750 people, many with interests in computers. The bounty has been up for two months, as of March 13 2020."><sup>18</sup></a></span></p>
</HTML>


==== GPU price / single-precision FLOPS ====


<HTML>
<p>Figure 1 shows our collected dataset for GPU price / single-precision FLOPS over time.<span class="easy-footnote-margin-adjust" id="easy-footnote-19-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-19-2316" title='See &lt;a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing"&gt;this spreadsheet&lt;/a&gt;, tab ‘Cleaned GPU Data for SP’ for the chart generation.'><sup>19</sup></a></span></p>
</HTML>


<HTML>
<figure class="wp-block-image is-resized">
<img alt="" height="364" src="https://lh6.googleusercontent.com/eakDTr43veWzFljIgZr2zzJ8--TzbWpIbelaGFeM7clFeQHdhcodTZfBAw5aUxkkJ5lNd3h4g7m8X6AsHwEm_kU-5gZnnESi26Mnf43eMCcD0W8EnpJDrPqBhN9OXT5W7UnQR9om" width="591"/>
<figcaption>
<strong>Figure 1: Real GPU price / single-precision FLOPS over time. The vertical axis is log-scale. Price is measured in 2019 dollars.</strong>
</figcaption>
</figure>
</HTML>


<HTML>
<p>To find a clear trend for the prices of the cheapest GPUs / FLOPS, we looked at the running minimum prices every 10 days.<span class="easy-footnote-margin-adjust" id="easy-footnote-20-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-20-2316" title='See &lt;a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing"&gt;this spreadsheet&lt;/a&gt;, tab ‘Cleaned GPU Data for SP Minimums’ for the plotting. We used &lt;a href="https://drive.google.com/open?id=1JP98EP8nYA0KqofLm24vF2vwNL0PalcB"&gt;this script&lt;/a&gt; on the data from the ‘Cleaned GPU Data for SP’ to calculate the minimums and then import them into a new sheet of the spreadsheet.'><sup>20</sup></a></span></p>
</HTML>


<HTML>
<figure class="wp-block-image size-large is-resized">
<img alt="" class="wp-image-2551" height="364" loading="lazy" sizes="(max-width: 591px) 100vw, 591px" src="https://aiimpacts.org/wp-content/uploads/2020/05/image-1-1024x633.png" srcset="https://aiimpacts.org/wp-content/uploads/2020/05/image-1-1024x633.png 1024w, https://aiimpacts.org/wp-content/uploads/2020/05/image-1-300x185.png 300w, https://aiimpacts.org/wp-content/uploads/2020/05/image-1.png 1366w" width="591"/>
<figcaption>
<strong>Figure 2: Ten-day minimums in real GPU price / single-precision FLOPS over time. The vertical axis is log-scale. Price is measured in 2019 dollars. The blue line shows the trendline ignoring data before late 2007. (We believe the apparent steep decline prior to late 2007 is an artefact of a lack of data for that time period.)</strong>
</figcaption>
</figure>
</HTML>


<HTML>
<p>The cheapest GPU price / FLOPS hardware using release date pricing has not decreased since 2017. However there was a similar period of stagnation between early 2009 and 2011, so this may not represent a slowing of the trend in the long run.</p>
</HTML>


<HTML>
<p>Based on the figures above, the running minimums seem to follow a roughly exponential trend. If we do not include the initial point in 2007, (which we suspect is not in fact the cheapest hardware at the time), we get that the cheapest GPU price / single-precision FLOPS fell by around 17% per year, for a factor of ten in  ~12.5 years.<span class="easy-footnote-margin-adjust" id="easy-footnote-21-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-21-2316" title='See &lt;a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing"&gt;this spreadsheet&lt;/a&gt;, tab ‘Cleaned GPU Data for SP Minimums’ for the calculation.'><sup>21</sup></a></span></p>
</HTML>


==== GPU price / half-precision FLOPS ====


<HTML>
<p>Figure 3 shows GPU price / half-precision FLOPS for all the GPUs in our search above for which we could find half-precision theoretical performance.<span class="easy-footnote-margin-adjust" id="easy-footnote-22-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-22-2316" title='See &lt;a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing"&gt;this spreadsheet&lt;/a&gt;, tab ‘Cleaned GPU Data for HP’ for the chart generation.'><sup>22</sup></a></span></p>
</HTML>


<HTML>
<figure class="wp-block-image is-resized">
<img alt="" height="363" loading="lazy" src="https://lh4.googleusercontent.com/mAmmM2htFRBW13ObHMQSBBDbP0DiVsRqDZyvrvGXYS6iueDBNNPYABeJqDM2DWqi1o3S21IcMXbLCQeeWUR1xFSYDKapZW2vzr4_pFaXvpjN98DvZAwmLCx_3dQm21NWa7xlWKrh" width="588"/>
<figcaption>
<strong>Figure 3: Real GPU price / half-precision FLOPS over time. The vertical axis is log-scale. Price is measured in 2019 dollars.</strong>
</figcaption>
</figure>
</HTML>


<HTML>
<p>Again, we looked at the running minimums of this graph every 10 days, shown in Figure 4 below.<span class="easy-footnote-margin-adjust" id="easy-footnote-23-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-23-2316" title='See &lt;a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing"&gt;this spreadsheet&lt;/a&gt;, tab ‘Cleaned GPU Data for HP Minimums’ for the plotting. We used &lt;a href="https://drive.google.com/open?id=1JP98EP8nYA0KqofLm24vF2vwNL0PalcB"&gt;this script&lt;/a&gt; on the data from the ‘Cleaned GPU Data for HP’ to calculate the minimums and then import them into a new sheet of the spreadsheet.'><sup>23</sup></a></span></p>
</HTML>


<HTML>
<figure class="wp-block-image is-resized">
<img alt="" height="364" loading="lazy" src="https://lh6.googleusercontent.com/ZoVY4R_fEQVgrS4n-v8EkOorER7M-Wup_nAFUujdcRTwRrj3MyK5ADkcKgPremWa0uF_gMHUZYoZ6uq735_XyZa6d_voJMhOF7wyOEq23goNuCzjoFaKc-pHYhsXgMc0inmXHUNs" width="589"/>
<figcaption>
<strong>Figure 4: Minimums in real GPU price / half-precision FLOPS over time. The vertical axis is log-scale. Price is measured in 2019 dollars.</strong>
</figcaption>
</figure>
</HTML>


<HTML>
<p>If we assume an exponential trend with noise,<span class="easy-footnote-margin-adjust" id="easy-footnote-24-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-24-2316" title="Where ambiguous, we assume these trends are exponential rather than linear, because our understanding is that that is much more common historically in computing hardware price trends."><sup>24</sup></a></span> cheapest GPU price / half-precision FLOPS fell by around 26% per year, which would yield a factor of ten after ~8 years.<span class="easy-footnote-margin-adjust" id="easy-footnote-25-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-25-2316" title='See &lt;a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing"&gt;this spreadsheet&lt;/a&gt;, tab ‘Cleaned GPU Data for HP Minimums’ for the calculation.'><sup>25</sup></a></span></p>
</HTML>


==== GPU price / half-precision FMA FLOPS ====


<HTML>
<p>Figure 5 shows GPU price / half-precision FMA FLOPS for all the GPUs in our search above for which we could find half-precision FMA theoretical performance.<span class="easy-footnote-margin-adjust" id="easy-footnote-26-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-26-2316" title='See &lt;a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing"&gt;this spreadsheet&lt;/a&gt;, tab ‘Cleaned GPU Data for HP + Tensor Cores’ for the chart generation.'><sup>26</sup></a></span> (Note that this includes all of our half-precision data above, since those FLOPS could be used for fused-multiply adds in particular). GPUs with TensorCores are marked in red.</p>
</HTML>


<HTML>
<figure class="wp-block-image is-resized">
<img alt="" height="363" loading="lazy" src="https://lh6.googleusercontent.com/Reu_O__PF1WmEYmT6RD8JIkotqoO-dO9HTqmoQBVf_lgj0CtTvl4TNm6E1kPJ5AirX0T5j3g1QOv6hwQLjpWct__by_g1lj3LFpSgd1emAKg3FyyDWP-gRyW5sTs4Ostp9JGkcRA" width="588"/>
<figcaption>
<strong>Figure 5: Real GPU price / half-precision FMA FLOPS over time. Price is measured in 2019 dollars.</strong>
</figcaption>
</figure>
</HTML>


<HTML>
<p>Figure 6 shows the running minimums of GPU price / HP FMA FLOPS.<span class="easy-footnote-margin-adjust" id="easy-footnote-27-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-27-2316" title='See &lt;a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing"&gt;this spreadsheet&lt;/a&gt;, tab ‘Cleaned GPU Data for HP + Tensor Cores Minimums’ for the plotting. We used &lt;a href="https://drive.google.com/open?id=1JP98EP8nYA0KqofLm24vF2vwNL0PalcB"&gt;this script&lt;/a&gt; on the data from the ‘Cleaned GPU Data for HP + Tensor Cores’ to calculate the minimums and then import them into a new sheet of the spreadsheet.'><sup>27</sup></a></span></p>
</HTML>


<HTML>
<figure class="wp-block-image is-resized">
<img alt="" height="362" loading="lazy" src="https://lh6.googleusercontent.com/SyYEb2HGDZCw8fPF15iIUwmU3U5r9UTKpYlaRXBFCMcrTBJhCf7VTeqwC6B82G2hmANUKYTiTLibBmsYm5XGAiV655z7GCdFX0jDeilXyX2cyHlcHe9wWwXRfQrPE2_gA3iTxzf3" width="586"/>
<figcaption>
<strong>Figure 6: Minimums in real GPU price / half-precision FMA FLOPS over time. Price is measured in 2019 dollars.</strong>
</figcaption>
</figure>
</HTML>


<HTML>
<p>GPU price / Half-Precision FMA FLOPS appears to be following an exponential trend over the last four years, falling by around 46% per year, for a factor of ten in ~4 years.<span class="easy-footnote-margin-adjust" id="easy-footnote-28-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-28-2316" title='See &lt;a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing"&gt;this spreadsheet&lt;/a&gt;, tab ‘Cleaned GPU Data for HP + Tensor Cores Minimums’ for the calculation.'><sup>28</sup></a></span></p>
</HTML>


===== Active Prices =====


<HTML>
<p>GPU prices often go down from the time of release, and some popular GPUs are older ones that have gone down in price.<span class="easy-footnote-margin-adjust" id="easy-footnote-29-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-29-2316" title='For example, one of the GPUs recommended for deep learning &lt;a href="https://www.reddit.com/r/MachineLearning/comments/b95182/d_which_gpus_to_get_for_deep_learning_my/"&gt;in this Reddit thread&lt;/a&gt; is the GTX 1060 (6GB), which has been around &lt;a href="https://www.techpowerup.com/gpu-specs/geforce-gtx-1060-6-gb.c2862"&gt;since 2016&lt;/a&gt;.'><sup>29</sup></a></span> Given this, it makes sense to look at active price data for the same GPU over time.</p>
</HTML>


==== Data Sources ====


<HTML>
<p>We collected data on peak theoretical performance in FLOPS from <a href="https://www.techpowerup.com/">TechPowerUp</a><span class="easy-footnote-margin-adjust" id="easy-footnote-30-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-30-2316" title='We scraped data from &lt;a href="https://www.techpowerup.com/gpu-specs/radeon-rx-480.c2848"&gt;individual TechPowerUp pages&lt;/a&gt; using &lt;a href="https://drive.google.com/open?id=1msy977jSLcJspULMWlWOfIIFRWo_vsds"&gt;this script&lt;/a&gt;. Our full scraped TechPowerUp dataset can be found &lt;a href="https://docs.google.com/spreadsheets/d/1pXTyUJ2AvpkhYtphGn8UnAl4gI8j2jsx1AaGrOahl7o/edit?usp=sharing"&gt;here&lt;/a&gt;.'><sup>30</sup></a></span> and combined it with active GPU price data to get GPU price / FLOPS over time.<span class="easy-footnote-margin-adjust" id="easy-footnote-31-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-31-2316" title="We chose to automatically scrape theoretical peak performance numbers from TechPowerUp instead of using the ones we manually collected above because there were several GPUs in the active pricing datasets that we hadn’t collected data for manually, and it was easier to scrape the entire site than just the subset of GPUs we needed."><sup>31</sup></a></span> Our primary source of historical pricing data was Passmark, though we also found a less trustworthy dataset on Kaggle which we used to check our analysis. We adjusted prices for inflation based on the consumer price index.<span class="easy-footnote-margin-adjust" id="easy-footnote-32-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-32-2316" title="“CPI Home.” U.S. Bureau of Labor Statistics. U.S. Bureau of Labor Statistics. Accessed May 2, 2020. https://www.bls.gov/cpi/."><sup>32</sup></a></span></p>
</HTML>


=== Passmark ===


<HTML>
<p>We scraped pricing data<span class="easy-footnote-margin-adjust" id="easy-footnote-33-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-33-2316" title='We used &lt;a href="https://drive.google.com/open?id=1nd7111hOb-eCMBCOe1qocXYuAJys0pLk"&gt;this script&lt;/a&gt;.'><sup>33</sup></a></span> on GPUs between 2011 and early 2020 from Passmark.<span class="easy-footnote-margin-adjust" id="easy-footnote-34-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-34-2316" title='“PassMark &amp;#8211; GeForce GTX 660 &amp;#8211; Price Performance Comparison.” Accessed March 24, 2020. &lt;a href="https://www.videocardbenchmark.net/gpu.php?gpu=GeForce+GTX+660&amp;amp;id=2152"&gt;https://www.videocardbenchmark.net/gpu.php?gpu=GeForce+GTX+660&amp;amp;id=2152&lt;/a&gt;.'><sup>34</sup></a></span> Where necessary, we renamed GPUs from Passmark to be consistent with TechPowerUp.<span class="easy-footnote-margin-adjust" id="easy-footnote-35-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-35-2316" title="In most cases where renaming was necessary, the same GPU had multiple clear names, e.g. the “Radeon HD 7970 / R9 280X” in PassMark was just called the “Radeon HD 7970” in TechPowerUp. In a few cases, Passmark listed some GPUs which TechPowerUp listed separately as one GPU, e.g. “Radeon R9 290X / 390X” seemed to ambiguously refer to the Radeon R9 290X or Radeon R9 390X. In these cases, we conservatively assume that the GPU refers to the less powerful / earlier GPU. In one exceptional case, we assumed that the “Radeon R9 Fury + Fury X” referred to the Radeon Fury X in PassMark. The ambiguously named GPUs were not in the minimum data we calculated, so probably did not have a strong effect on the final result."><sup>35</sup></a></span> The Passmark data consists of 38,138 price points for 352 GPUs. We guess that these represent most popular GPUs. <br/></p>
</HTML>


<HTML>
<p>Looking at the ‘current prices’ listed on individual Passmark GPU pages, prices appear to be sourced from Amazon, Newegg, and Ebay. Passmark’s listed pricing data does not correspond to regular intervals. We don’t know if prices were pulled at irregular intervals, or if Passmark pulls prices regularly and then only lists major changes as price points. When we see a price point, we treat it as though the GPU is that price only at that time point, not indefinitely into the future.<br/></p>
</HTML>


<HTML>
<p>The data contains several blips where a GPU is briefly sold very unusually cheaply. A random checking of some of these suggests to us that these correspond to single or small numbers of GPUs for sale, which we are not interested in tracking, because we are trying to predict AI progress, which presumably isn’t influenced by temporary discounts on tiny batches of GPUs.<br/></p>
</HTML>


=== Kaggle ===


<HTML>
<p><a href="https://www.kaggle.com/raczeq/ethereum-effect-pc-parts">This Kaggle dataset</a> contains scraped data of GPU prices from price comparison sites PriceSpy.co.uk, PCPartPicker.com, Geizhals.eu from the years 2013 – 2018. The Kaggle dataset has 319,147 price points for 284 GPUs. Unfortunately, at least some of the data is clearly wrong, potentially because price comparison sites include pricing data from untrustworthy merchants.<span class="easy-footnote-margin-adjust" id="easy-footnote-36-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-36-2316" title="For example, the Kaggle dataset includes extremely cheap FirePro S7150s sold in 2014, even though the FirePro S7150 only came out in 2016. One of the sellers of these cheap GPUs were ‘Club 3D’, which also appeared to sell several other erroneously cheap GPUs."><sup>36</sup></a></span> As such, we don’t use the Kaggle data directly in our analysis, but do use it as a check on our Passmark data. The data that we get from Passmark roughly appears to be a subset of the Kaggle data from 2013 – 2018,<span class="easy-footnote-margin-adjust" id="easy-footnote-37-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-37-2316" title='See &lt;a href="https://drive.google.com/open?id=1uvO-gNpAGh9qMzs5MnNtOg1X-R7JzdAq"&gt;this plot of Passmark single-precision GPU price / FLOPS&lt;/a&gt; compared to the &lt;a href="https://drive.google.com/open?id=194Tqrcix2XdytbT-WbgcRbFpeHIHqsao"&gt;combined Passmark and Kaggle single-precision GPU price / FLOPS&lt;/a&gt;, and &lt;a href="https://drive.google.com/open?id=1hk-1rHqOkWUwh3BVNbwTEu8dEfjDhsLT"&gt;this plot of Passmark half-precision GPU price / FLOPS&lt;/a&gt; compared to the &lt;a href="https://drive.google.com/open?id=1mcpQigPJs9CRqebY-Uvm0dJ1si1jeTQv"&gt;combined Passmark and Kaggle half-precision $ / FLOPS&lt;/a&gt;. In both cases the 2013 &amp;#8211; 2018 Passmark data appears to roughly be a subset of the Kaggle data.'><sup>37</sup></a></span> which is what we would expect if the price comparison engines picked up prices from the merchants Passmark looks at.</p>
</HTML>


=== Limitations ===


<HTML>
<p>There are a number of reasons why we think this analysis may in fact not reflect GPU price trends:</p>
</HTML>


<HTML>
<ul>
<li><div class="li">We effectively have just one source of pricing data, Passmark.</div></li>
<li><div class="li">Passmark appears to only look at Amazon, Newegg, and Ebay for pricing data.</div></li>
<li><div class="li">We are not sure, but we suspect that Passmark only looks at the U.S. versions of Amazon, Newegg, and Ebay, and pricing may be significantly different in other parts of the world (though we guess it wouldn’t be different enough to change the general trend much).</div></li>
<li><div class="li">As mentioned above, we are not sure if Passmark pulls price data regularly and only lists major price changes, or pulls price data irregularly. If the former is true, our data may be overrepresenting periods where the price changes dramatically.</div></li>
<li><div class="li">None of the price data we found includes quantities of GPUs which were available at that price, which means some prices may be for only a very limited number of GPUs.</div></li>
<li><div class="li">We don’t know how much the prices from these datasets reflect the prices that a company pays when buying GPUs in bulk, which we may be more interested in tracking.</div></li>
</ul>
</HTML>


<HTML>
<p>A better version of this analysis might start with more complete data from price comparison engines (along the lines of the Kaggle dataset) and then filter out clearly erroneous pricing information in some principled way.</p>
</HTML>


==== Data ====


<HTML>
<p>The original scraped datasets with cards renamed to match TechPowerUp can be found <a href="https://drive.google.com/drive/folders/1cCjG_sUUePxbh5fN9ViOPX6GW9D2GyPJ?usp=sharing">here</a>. GPU price / FLOPS data is graphed on a log scale in the figures below. Price points for the same GPU are marked in the same color. We adjusted prices for inflation using the consumer price index.<span class="easy-footnote-margin-adjust" id="easy-footnote-38-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-38-2316" title="“CPI Home.” U.S. Bureau of Labor Statistics. U.S. Bureau of Labor Statistics. Accessed May 2, 2020. https://www.bls.gov/cpi/."><sup>38</sup></a></span> All points below are in 2019 dollars.</p>
</HTML>


<HTML>
<p>To try to filter out noisy prices that didn’t last or were only available in small numbers, we took out the lowest 5% of data in every several day period<span class="easy-footnote-margin-adjust" id="easy-footnote-39-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-39-2316" title="We set this period to be 10 days long when looking at single-precision data, and 30 days long when looking at half-precision data, since half-precision data was significantly more sparse."><sup>39</sup></a></span> to get the 95th percentile cheapest hardware. We then found linear and exponential trendlines of best fit through the available hardware with the lowest GPU price / FLOPS every several days.<span class="easy-footnote-margin-adjust" id="easy-footnote-40-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-40-2316" title='This calculation can be found in &lt;a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing"&gt;this spreadsheet&lt;/a&gt;.'><sup>40</sup></a></span></p>
</HTML>


=== GPU price / single-precision FLOPS ===


<HTML>
<p>Figures 7-10 show the raw data, 95th percentile data, and trendlines for single-precision GPU price / FLOPS for the Passmark dataset. <a href="https://drive.google.com/open?id=1-PEl2kSORRH78Qa4huRF-t_g_m1QOTDs">This folder</a> contains plots of all our datasets, including the Kaggle dataset and combined Passmark + Kaggle dataset.<span class="easy-footnote-margin-adjust" id="easy-footnote-41-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-41-2316" title='We used a Python plotting library to generate our plots, the script can be found &lt;a href="https://drive.google.com/open?id=1u3qI9m9W6_9efIpsDBq1Hc-qixcKy8Sb"&gt;here&lt;/a&gt;. All of our resulting plots can be found &lt;a href="https://drive.google.com/open?id=1afVbKn34pw5rj4fn1vI_qOhdZh5GLfwE"&gt;here&lt;/a&gt;. ‘single’ vs. ‘half’ refers to whether its $ / FLOPS data for single or half-precision FLOPS, ‘passmark’, ‘kaggle’, and ‘combined’ refer to which dataset is being plotted and ‘raw’ vs. ‘95’ refer to whether we’re plotting all the data or the 95th percentile data.'><sup>41</sup></a></span></p>
</HTML>


<HTML>
<figure class="wp-block-image size-large is-resized">
<img alt="" class="wp-image-2509" height="302" loading="lazy" sizes="(max-width: 589px) 100vw, 589px" src="https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_raw_crop-1-1024x525.png" srcset="https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_raw_crop-1-1024x525.png 1024w, https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_raw_crop-1-300x154.png 300w, https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_raw_crop-1-768x394.png 768w, https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_raw_crop-1-1536x788.png 1536w, https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_raw_crop-1.png 1657w" width="589"/>
<figcaption>
<br/>
<strong>Figure 7: GPU price / single-precision FLOPS over time, taken from our Passmark dataset.<span class="easy-footnote-margin-adjust" id="easy-footnote-42-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-42-2316" title='The dataset we used for this plot can be found &lt;a href="https://drive.google.com/open?id=19FB-_MnQAtJErb8e_5DHTRVKAPg4g1YX"&gt;here&lt;/a&gt;. This a processed version of our scraped dataset, with prices / FLOPS adjusted for inflation. The script we used to process and plot can be found &lt;a href="https://drive.google.com/open?id=1u3qI9m9W6_9efIpsDBq1Hc-qixcKy8Sb"&gt;here&lt;/a&gt;.'><sup>42</sup></a></span> Price is measured in 2019 dollars. <a href="https://drive.google.com/open?id=194Tqrcix2XdytbT-WbgcRbFpeHIHqsao">This picture</a> shows that the Kaggle data does appear to be a superset of the Passmark data from 2013 – 2018, giving us some evidence that the Passmark data is correct. The vertical axis is log-scale.</strong>
</figcaption>
</figure>
</HTML>


<HTML>
<figure class="wp-block-image size-large is-resized">
<img alt="" class="wp-image-2510" height="301" loading="lazy" sizes="(max-width: 587px) 100vw, 587px" src="https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_95_crop-1024x525.png" srcset="https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_95_crop-1024x525.png 1024w, https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_95_crop-300x154.png 300w, https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_95_crop-768x394.png 768w, https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_95_crop-1536x788.png 1536w, https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_95_crop.png 1657w" width="587"/>
<figcaption>
<br/>
<strong>Figure 8: The top 95% of data every 10 days for GPU price / single-precision FLOPS over time, taken from the Passmark dataset we plotted above. (Figure 7 with the cheapest 5% removed.) The vertical axis is log-scale.<span class="easy-footnote-margin-adjust" id="easy-footnote-43-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-43-2316" title='The script to calculate the 95th percentile and generate this plot can be found &lt;a href="https://drive.google.com/open?id=1u3qI9m9W6_9efIpsDBq1Hc-qixcKy8Sb"&gt;here&lt;/a&gt;.'><sup>43</sup></a></span></strong>
</figcaption>
</figure>
</HTML>


<HTML>
<figure class="wp-block-image size-large is-resized">
<img alt="" class="wp-image-2511" height="298" loading="lazy" sizes="(max-width: 582px) 100vw, 582px" src="https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_95_zoom_crop-1024x525.png" srcset="https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_95_zoom_crop-1024x525.png 1024w, https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_95_zoom_crop-300x154.png 300w, https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_95_zoom_crop-768x394.png 768w, https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_95_zoom_crop-1536x788.png 1536w, https://aiimpacts.org/wp-content/uploads/2020/04/single_passmark_95_zoom_crop.png 1657w" width="582"/>
<figcaption>
<br/>
<strong>Figure 9: The same data as Figure 8, with the vertical axis zoomed-in.</strong>
</figcaption>
</figure>
</HTML>


<HTML>
<figure class="wp-block-image size-large is-resized">
<img alt="" class="wp-image-2595" height="357" loading="lazy" sizes="(max-width: 579px) 100vw, 579px" src="https://aiimpacts.org/wp-content/uploads/2020/06/image-1024x633.png" srcset="https://aiimpacts.org/wp-content/uploads/2020/06/image-1024x633.png 1024w, https://aiimpacts.org/wp-content/uploads/2020/06/image-300x186.png 300w, https://aiimpacts.org/wp-content/uploads/2020/06/image-768x475.png 768w, https://aiimpacts.org/wp-content/uploads/2020/06/image-1536x950.png 1536w, https://aiimpacts.org/wp-content/uploads/2020/06/image.png 1772w" width="579"/>
<figcaption>
<strong>Figure 10: The minimum data points from the top 95% of the Passmark dataset, taken every 10 days. We fit linear and exponential trendlines through the data. The vertical axis is log-scale.<span class="easy-footnote-margin-adjust" id="easy-footnote-44-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-44-2316" title='See &lt;a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing"&gt;here&lt;/a&gt;, tab ‘Passmark SP Minimums&amp;#8217; to see our calculation of the minimums over time. We used &lt;a href="https://drive.google.com/open?id=1yRTJwVQAwCqLSTyGXgGIHcRfJFP7C5D2"&gt;this script&lt;/a&gt; to generate the minimums, then imported them into this spreadsheet.'><sup>44</sup></a></span></strong>
</figcaption>
</figure>
</HTML>


== Analysis ==


<HTML>
<p>The cheapest 95th percentile data every 10 days appears to fit relatively well to both a linear and exponential trendline. However we assume that progress will follow an exponential, because previous progress has <a href="/doku.php?id=ai_timelines:2017_trend_in_the_cost_of_computing">followed an exponential</a>.<br/></p>
</HTML>


<HTML>
<p>In the Passmark dataset, the exponential trendline suggested that from 2011 to 2020, 95th-percentile GPU price / single-precision FLOPS fell by around 13% per year, for a factor of ten in ~17 years,<span class="easy-footnote-margin-adjust" id="easy-footnote-45-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-45-2316" title='You can see our calculations for this &lt;a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing"&gt;here&lt;/a&gt;, sheet ‘Passmark SP Minimums’. Each sheet has a cell ‘Rate to move an order of magnitude’ which has our calculation for how many years we need to move an order of magnitude. In the (untrustworthy) Kaggle dataset alone, its rate would yield an order of magnitude of decrease every ~12 years, and the rate in the combined dataset&amp;nbsp; would yield an order of magnitude of decrease every ~16 years.'><sup>45</sup></a></span> bootstrap<span class="easy-footnote-margin-adjust" id="easy-footnote-46-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-46-2316" title='Orloff, Jeremy, and Jonathan Bloom. “Bootstrap Confidence Intervals.” MIT OpenCourseWare, 2014. &lt;a href="https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf"&gt;https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf&lt;/a&gt;.'><sup>46</sup></a></span> 95% confidence interval 16.3 to 18.1 years.<span class="easy-footnote-margin-adjust" id="easy-footnote-47-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-47-2316" title='We used &lt;a href="https://drive.google.com/open?id=1XkA-8WruAMKM3y3cdNMPNJHabpMUE_MT"&gt;this script&lt;/a&gt; to generate bootstrap confidence intervals for our datasets.'><sup>47</sup></a></span> We believe the rise in price / FLOPS in 2017 corresponds to a rise in GPU prices due to increased demand from cryptocurrency miners.<span class="easy-footnote-margin-adjust" id="easy-footnote-48-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-48-2316" title='We think this is the case because we’ve observed this dip in other GPU analyses we’ve done, and because the timing lines up: &lt;a href="https://www.techspot.com/news/72854-nvidia-asking-graphics-card-retailers-prioritize-gamers-over.html"&gt;the first table in this article&lt;/a&gt; shows how GPU prices were increasing starting 2017 and continued to increase through 2018, and &lt;a href="https://www.kaggle.com/raczeq/impact-of-cryptocurrencies-rates-on-pc-market/data"&gt;the chart here&lt;/a&gt; shows how GPU prices increased in 2017.'><sup>48</sup></a></span> If we instead look at the trend from 2011 through 2016, before the cryptocurrency rise, we instead get that 95th-percentile GPU price / single-precision FLOPS price fell by around 13% per year, for a factor of ten in ~16 years.<span class="easy-footnote-margin-adjust" id="easy-footnote-49-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-49-2316" title='You can see our calculations for this &lt;a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing"&gt;here&lt;/a&gt;, sheet ‘Passmark SP Minimums’, next to ‘Exponential trendline from 2015 to 2016. The trendline calculated is technically the linear fit through the log of the data.'><sup>49</sup></a></span></p>
</HTML>


<HTML>
<p>This is slower than the order of magnitude every ~12.5 years we found when looking at release prices. If we restrict the release price data to 2011 – 2019, we get an order of magnitude decrease every ~13.5 years instead,<span class="easy-footnote-margin-adjust" id="easy-footnote-50-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-50-2316" title='See our calculation &lt;a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing"&gt;here&lt;/a&gt;, tab ‘Cleaned GPU Data for SP Minimums’, next to the cell marked “Exponential trendline from 2011 to 2019.”'><sup>50</sup></a></span> so part of the discrepancy can be explained because of the different start times of the datasets. To get some assurance that our active price data wasn’t erroneous, we spot checked the best active price at the start of 2011, which was somewhat lower than the best release price at the same time, and confirmed that its given price was consistent with surrounding pricing data.<span class="easy-footnote-margin-adjust" id="easy-footnote-51-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-51-2316" title='At the start of 2011, the &lt;a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing"&gt;minimum release &lt;/a&gt;&lt;a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing"&gt;price / FLOPS&lt;/a&gt; (see tab, ‘Cleaned GPU Data for SP Minimums’) is .000135 $ / FLOPS, whereas the &lt;a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing"&gt;minimum active price / FLOPS&lt;/a&gt; (see tab, ‘Passmark SP Minimums’) is around .0001 $ / FLOPS. &lt;a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing"&gt;The initial GPU price / FLOPS minimum&lt;/a&gt; (see sheet ‘Passmark SP Minimums’) corresponds to the Radeon HD 5850 which had a price of $184.9 in 3/2011 and a release price of $259. &lt;a href="https://www.videocardbenchmark.net/gpu.php?gpu=Radeon+HD+5850&amp;amp;id=47"&gt;Looking at the general trend in Passmark&lt;/a&gt; suggests that the Radeon HD 5850 did indeed rapidly decline from its $259 release price to consistently below $200 prices.'><sup>51</sup></a></span> We think active prices are likely to be closer to the prices at which people actually bought GPUs, so we guess that ~17 years / order of magnitude decrease is a more accurate estimate of the trend we care about.</p>
</HTML>


=== GPU price / half-precision FLOPS ===


<HTML>
<p>Figures 11-14 show the raw data, 95th percentile data, and trendlines for half-precision GPU price / FLOPS for the Passmark dataset. <a href="https://drive.google.com/open?id=1-PEl2kSORRH78Qa4huRF-t_g_m1QOTDs">This folder</a> contains plots of the Kaggle dataset and combined Passmark + Kaggle dataset.</p>
</HTML>


<HTML>
<figure class="wp-block-image size-large is-resized">
<img alt="" class="wp-image-2512" height="297" loading="lazy" sizes="(max-width: 580px) 100vw, 580px" src="https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_raw_crop-1024x525.png" srcset="https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_raw_crop-1024x525.png 1024w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_raw_crop-300x154.png 300w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_raw_crop-768x394.png 768w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_raw_crop-1536x788.png 1536w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_raw_crop.png 1657w" width="580"/>
<figcaption>
<br/>
<strong>Figure 11: GPU price / half-precision FLOPS over time, taken from our Passmark dataset. Price is measured in 2019 dollars.<span class="easy-footnote-margin-adjust" id="easy-footnote-52-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-52-2316" title=" The dataset we used for this plot can be found here. This is a processed version of our scraped dataset, with prices / FLOPS adjusted for inflation. The script we used to process and plot can be found here."><sup>52</sup></a></span> This picture shows that the Kaggle data does appear to be a superset of the Passmark data from 2013 – 2018, giving us some evidence that the Passmark data is reasonable. The vertical axis is log-scale.</strong>
</figcaption>
</figure>
</HTML>


<HTML>
<figure class="wp-block-image size-large is-resized">
<img alt="" class="wp-image-2513" height="298" loading="lazy" sizes="(max-width: 581px) 100vw, 581px" src="https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_95_crop-1024x525.png" srcset="https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_95_crop-1024x525.png 1024w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_95_crop-300x154.png 300w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_95_crop-768x394.png 768w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_95_crop-1536x788.png 1536w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_95_crop.png 1657w" width="581"/>
<figcaption>
<br/>
<strong>Figure 12: The top 95% of data every 30 days for GPU price / half-precision FLOPS over time, taken from the Passmark dataset we plotted above. (Figure 11 with the cheapest 5% removed.) The vertical axis is log-scale.<span class="easy-footnote-margin-adjust" id="easy-footnote-53-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-53-2316" title="The script to calculate the 95th percentile and generate this plot can be found here."><sup>53</sup></a></span></strong>
</figcaption>
</figure>
</HTML>


<HTML>
<figure class="wp-block-image size-large is-resized">
<img alt="" class="wp-image-2514" height="299" loading="lazy" sizes="(max-width: 583px) 100vw, 583px" src="https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_95_zoom_crop-1024x525.png" srcset="https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_95_zoom_crop-1024x525.png 1024w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_95_zoom_crop-300x154.png 300w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_95_zoom_crop-768x394.png 768w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_95_zoom_crop-1536x788.png 1536w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_95_zoom_crop.png 1657w" width="583"/>
<figcaption>
<br/>
<strong>Figure 13: The same data as Figure 12, with the vertical axis zoomed-in.</strong>
</figcaption>
</figure>
</HTML>


<HTML>
<figure class="wp-block-image size-large is-resized">
<img alt="" class="wp-image-2596" height="356" loading="lazy" sizes="(max-width: 576px) 100vw, 576px" src="https://aiimpacts.org/wp-content/uploads/2020/06/image-1-1024x633.png" srcset="https://aiimpacts.org/wp-content/uploads/2020/06/image-1-1024x633.png 1024w, https://aiimpacts.org/wp-content/uploads/2020/06/image-1-300x186.png 300w, https://aiimpacts.org/wp-content/uploads/2020/06/image-1-768x475.png 768w, https://aiimpacts.org/wp-content/uploads/2020/06/image-1-1536x950.png 1536w, https://aiimpacts.org/wp-content/uploads/2020/06/image-1.png 1772w" width="576"/>
<figcaption>
<strong>Figure 14: The minimum data points from the top 95% of the Passmark dataset, taken every 30 days. We fit linear and exponential trendlines through the data. The vertical axis is log-scale.<span class="easy-footnote-margin-adjust" id="easy-footnote-54-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-54-2316" title='See &lt;a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing"&gt;here&lt;/a&gt;, tab ‘Passmark HP Minimums&amp;#8217; to see our calculation of the minimums over time. We used &lt;a href="https://drive.google.com/open?id=1u3qI9m9W6_9efIpsDBq1Hc-qixcKy8Sb"&gt;this script&lt;/a&gt; to generate the minimums, then imported them into this spreadsheet.'><sup>54</sup></a></span></strong>
</figcaption>
</figure>
</HTML>


== Analysis ==


<HTML>
<p>If we assume the trend is exponential, the Passmark trend seems to suggest that from 2015 to 2020, 95th-percentile GPU price / half-precision FLOPS of GPUs has fallen by around 21% per year, for a factor of ten over ~10 years,<span class="easy-footnote-margin-adjust" id="easy-footnote-55-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-55-2316" title='See the sheet marked ‘Passmark HP minimums’ in &lt;a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing"&gt;this spreadsheet&lt;/a&gt;. The trendline calculated is technically the linear fit through the log of the data.'><sup>55</sup></a></span> bootstrap<span class="easy-footnote-margin-adjust" id="easy-footnote-56-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-56-2316" title='Orloff, Jeremy, and Jonathan Bloom. “Bootstrap Confidence Intervals.” MIT OpenCourseWare, 2014. &lt;a href="https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf"&gt;https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf&lt;/a&gt;.'><sup>56</sup></a></span> 95% confidence interval 8.8 to 11 years.<span class="easy-footnote-margin-adjust" id="easy-footnote-57-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-57-2316" title='We used &lt;a href="https://drive.google.com/open?id=1XkA-8WruAMKM3y3cdNMPNJHabpMUE_MT"&gt;this script&lt;/a&gt; to generate bootstrap confidence intervals for our datasets.'><sup>57</sup></a></span> This is fairly close to the ~8 years / order of magnitude decrease we found when looking at release price data, but we treat active prices as a more accurate estimate of the actual prices at which people bought GPUs. As in our previous dataset, there is a noticeable rise in 2017, which we think is due to GPU prices increasing as a result of cryptocurrency miners. If we look at the trend from 2015 through 2016, before this rise, we get that 95th-percentile GPU price / half-precision FLOPS has fallen by around 14% per year, which would yield a factor of ten over ~8 years.<span class="easy-footnote-margin-adjust" id="easy-footnote-58-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-58-2316" title='See the sheet marked ‘Passmark HP minimums’ in &lt;a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing"&gt;this spreadsheet&lt;/a&gt;.'><sup>58</sup></a></span></p>
</HTML>


=== GPU price / half-precision FMA FLOPS ===


<HTML>
<p>Figures 15-18 show the raw data, 95th percentile data, and trendlines for half-precision GPU price / FMA FLOPS for the Passmark dataset. GPUs with Tensor Cores are marked in black. <a href="https://drive.google.com/open?id=1-PEl2kSORRH78Qa4huRF-t_g_m1QOTDs">This folder</a> contains plots of the Kaggle dataset and combined Passmark + Kaggle dataset.</p>
</HTML>


<HTML>
<figure class="wp-block-image size-large is-resized">
<img alt="" class="wp-image-2515" height="295" loading="lazy" sizes="(max-width: 576px) 100vw, 576px" src="https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_raw_crop-1024x525.png" srcset="https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_raw_crop-1024x525.png 1024w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_raw_crop-300x154.png 300w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_raw_crop-768x394.png 768w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_raw_crop-1536x788.png 1536w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_raw_crop.png 1657w" width="576"/>
<figcaption>
<br/>
<strong>Figure 15: GPU price / half-precision FMA FLOPS over time, taken from our Passmark dataset.<span class="easy-footnote-margin-adjust" id="easy-footnote-59-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-59-2316" title="The dataset we used for this plot can be found here. This a processed version of our scraped dataset, with prices / FLOPS adjusted for inflation. The script we used to process and plot can be found here."><sup>59</sup></a></span> price is measured in 2019 dollars. This picture shows that the Kaggle data does appear to be a superset of the Passmark data from 2013 – 2018, giving us some evidence that the Passmark data is correct. The vertical axis is log-scale.</strong>
</figcaption>
</figure>
</HTML>


<HTML>
<figure class="wp-block-image size-large is-resized">
<img alt="" class="wp-image-2516" height="296" loading="lazy" sizes="(max-width: 583px) 100vw, 583px" src="https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_95_crop-1024x520.png" srcset="https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_95_crop-1024x520.png 1024w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_95_crop-300x152.png 300w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_95_crop-768x390.png 768w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_95_crop-1536x780.png 1536w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_95_crop.png 1674w" width="583"/>
<figcaption>
<br/>
<strong>Figure 16: The top 95% of data every 30 days for GPU price / half-precision FMA FLOPS over time, taken from the Passmark dataset we plotted above.<span class="easy-footnote-margin-adjust" id="easy-footnote-60-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-60-2316" title="The script to calculate the 95th percentile and generate this plot can be found here."><sup>60</sup></a></span> (Figure 15 with the cheapest 5% removed.)</strong>
</figcaption>
</figure>
</HTML>


<HTML>
<figure class="wp-block-image size-large is-resized">
<img alt="" class="wp-image-2517" height="294" loading="lazy" sizes="(max-width: 580px) 100vw, 580px" src="https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_95_zoom_crop-1024x520.png" srcset="https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_95_zoom_crop-1024x520.png 1024w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_95_zoom_crop-300x152.png 300w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_95_zoom_crop-768x390.png 768w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_95_zoom_crop-1536x780.png 1536w, https://aiimpacts.org/wp-content/uploads/2020/04/half_passmark_tensor_95_zoom_crop.png 1674w" width="580"/>
<figcaption>
<br/>
<strong>Figure 17: The same data as Figure 16, with the vertical axis zoomed-in.</strong>
</figcaption>
</figure>
</HTML>


<HTML>
<figure class="wp-block-image size-large is-resized">
<img alt="" class="wp-image-2597" height="358" loading="lazy" sizes="(max-width: 580px) 100vw, 580px" src="https://aiimpacts.org/wp-content/uploads/2020/06/image-2-1024x633.png" srcset="https://aiimpacts.org/wp-content/uploads/2020/06/image-2-1024x633.png 1024w, https://aiimpacts.org/wp-content/uploads/2020/06/image-2-300x186.png 300w, https://aiimpacts.org/wp-content/uploads/2020/06/image-2-768x475.png 768w, https://aiimpacts.org/wp-content/uploads/2020/06/image-2-1536x950.png 1536w, https://aiimpacts.org/wp-content/uploads/2020/06/image-2.png 1772w" width="580"/>
<figcaption>
<strong>Figure 18: The minimum data points from the top 95% of the Passmark dataset, taken every 30 days. We fit linear and exponential trendlines through the data.<span class="easy-footnote-margin-adjust" id="easy-footnote-61-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-61-2316" title='See &lt;a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing"&gt;here&lt;/a&gt;, tab ‘Passmark HP FMA Minimums&amp;#8217; to see our calculation of the minimums over time. We used &lt;a href="https://drive.google.com/open?id=1yRTJwVQAwCqLSTyGXgGIHcRfJFP7C5D2"&gt;this script&lt;/a&gt; to generate the minimums, then imported them into this spreadsheet.'><sup>61</sup></a></span></strong>
</figcaption>
</figure>
</HTML>


== Analysis ==


<HTML>
<p>If we assume the trend is exponential, the Passmark trend seems to suggest the 95th-percentile GPU price / half-precision FMA FLOPS of GPUs has fallen by around 40% per year, which would yield a factor of ten in ~4.5 years,<span class="easy-footnote-margin-adjust" id="easy-footnote-62-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-62-2316" title='See the sheet marked ‘Passmark HP FMA minimums’ in &lt;a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing"&gt;this spreadsheet&lt;/a&gt;. The trendline calculated is technically the linear fit through the log of the data.'><sup>62</sup></a></span> with a bootstrap<span class="easy-footnote-margin-adjust" id="easy-footnote-63-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-63-2316" title='Orloff, Jeremy, and Jonathan Bloom. “Bootstrap Confidence Intervals.” MIT OpenCourseWare, 2014. &lt;a href="https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf"&gt;https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf&lt;/a&gt;.'><sup>63</sup></a></span> 95% confidence interval 4 to 5.2 years.<span class="easy-footnote-margin-adjust" id="easy-footnote-64-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-64-2316" title='We used &lt;a href="https://drive.google.com/open?id=1XkA-8WruAMKM3y3cdNMPNJHabpMUE_MT"&gt;this script&lt;/a&gt; to generate bootstrap confidence intervals for our datasets.'><sup>64</sup></a></span> This is fairly close to the ~4 years / order of magnitude decrease we found when looking at release price data, but we think active prices are a more accurate estimate of the actual prices at which people bought GPUs.</p>
</HTML>


<HTML>
<p>The figures above suggest that certain GPUs with Tensor Cores were a significant (~half an order of magnitude) improvement over existing GPU price / half-precision FMA FLOPS.</p>
</HTML>


====== Conclusion ======


<HTML>
<p>We summarize our results in the table below.</p>
</HTML>


<HTML>
<figure class="wp-block-table">
<table class="has-fixed-layout">
<tbody>
<tr>
<td></td>
<td><strong>Release Prices</strong></td>
<td><strong>95th-percentile Active Prices</strong></td>
<td><strong>95th-percentile Active Prices</strong> <strong>(pre-crypto price rise)</strong></td>
</tr>
<tr>
<td></td>
<td><em>11/2007 – 1/2020</em></td>
<td><em>3/2011 – 1/2020</em></td>
<td><em>3/2011 – 12/2016</em></td>
</tr>
<tr>
<td><strong>$ / single-precision FLOPS</strong></td>
<td>12.5</td>
<td>17</td>
<td>16</td>
</tr>
<tr>
<td></td>
<td><em>9/2014 – 1/2020</em></td>
<td><em>1/2015 – 1/2020</em></td>
<td><em>1/2015 – 12/2016</em></td>
</tr>
<tr>
<td><strong>$ / half-precision FLOPS</strong></td>
<td>8</td>
<td>10</td>
<td>8</td>
</tr>
<tr>
<td><strong>$ / half-precision FMA FLOPS</strong></td>
<td>4</td>
<td>4.5</td>
<td>—</td>
</tr>
</tbody>
</table>
</figure>
</HTML>


<HTML>
<p>Release price data seems to generally support the trends we found in active prices, with the notable exception of trends in GPU price / single-precision FLOPS, which cannot be explained solely by the different start dates.<span class="easy-footnote-margin-adjust" id="easy-footnote-65-2316"></span><span class="easy-footnote"><a href="#easy-footnote-bottom-65-2316" title='See our analysis in &lt;a href="#single-precision-analysis"&gt;this section&lt;/a&gt; above.'><sup>65</sup></a></span> We think the best estimate of the overall trend for prices at which people recently bought GPUs is the 95th-percentile active price data from 2011 – 2020, since release price data does not account for existing GPUs becoming cheaper over time. The pre-crypto trends are similar to the overall trends, suggesting that the trends we are seeing are not anomalous due to cryptocurrency.<br/></p>
</HTML>


<HTML>
<p>Given that, we guess that GPU prices as a whole have fallen at rates that would yield an order of magnitude over roughly:</p>
</HTML>


<HTML>
<ul>
<li><div class="li">17 years for single-precision FLOPS</div></li>
<li><div class="li">10 years for half-precision FLOPS</div></li>
<li><div class="li">5 years for half-precision fused multiply-add FLOPS</div></li>
</ul>
</HTML>


<HTML>
<p>Half-precision FLOPS seem to have become cheaper substantially faster than single-precision in recent years. This may be a “catching up” effect as more of the space on GPUs was allocated to half-precision computing, rather than reflecting more fundamental technological progress.</p>
</HTML>


<HTML>
<p><em>Primary author: Asya Bergal</em></p>
</HTML>


====== Notes ======


<HTML>
<ol class="easy-footnotes-wrapper">
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-1-2316"></span>“A graphics processing unit (GPU) is a specialized electronic circuit designed to rapidly manipulate and alter memory to accelerate the creation of images in a frame buffer intended for output to a display device … Modern GPUs are very efficient at manipulating computer graphics and image processing … The term was popularized by Nvidia in 1999, who marketed the GeForce 256 as “the world’s first GPU”. It was presented as a “single-chip processor with integrated transform, lighting, triangle setup/clipping, and rendering engines”.”<br/>
                  “Graphics Processing Unit.” Wikipedia. Wikimedia Foundation, March 24, 2020. <a href="https://en.wikipedia.org/w/index.php?title=Graphics_processing_unit&amp;oldid=947270104">https://en.wikipedia.org/w/index.php?title=Graphics_processing_unit&amp;oldid=947270104</a>.<a class="easy-footnote-to-top" href="#easy-footnote-1-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-2-2316"></span>Fraenkel, Bernard. “Council Post: For Machine Learning, It’s All About GPUs.” Forbes. Forbes Magazine, December 8, 2017. <a href="https://www.forbes.com/sites/forbestechcouncil/2017/12/01/for-machine-learning-its-all-about-gpus/#5ed90c227699">https://www.forbes.com/sites/forbestechcouncil/2017/12/01/for-machine-learning-its-all-about-gpus/#5ed90c227699</a>.<a class="easy-footnote-to-top" href="#easy-footnote-2-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-3-2316"></span>“In computing, floating point operations per second (FLOPS, flops or flop/s) is a measure of computer performance, useful in fields of scientific computations that require floating-point calculations. For such cases it is a more accurate measure than measuring instructions per second.”<br/>
                  “FLOPS.” Wikipedia. Wikimedia Foundation, March 24, 2020. <a href="https://en.wikipedia.org/w/index.php?title=FLOPS&amp;oldid=947177339">https://en.wikipedia.org/w/index.php?title=FLOPS&amp;oldid=947177339</a><a class="easy-footnote-to-top" href="#easy-footnote-3-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-4-2316"></span>From this discussion on Nvidia’s forums about theoretical GFLOPS: “GPU theoretical flops calculation is similar conceptually. It will vary by GPU just as the CPU calculation varies by CPU architecture and model. To use K40m as an example: http://www.nvidia.com/content/PDF/kepler/Tesla-K40-PCIe-Passive-Board-Spec-BD-06902-001_v05.pdf<br/>
<p>there are 15 SMs (2880/192), each with 64 DP ALUs that are capable of retiring one DP FMA instruction per cycle (== 2 DP Flops per cycle).<br/></p>
<p>15 x 64 x 2 * 745MHz = 1.43 TFlops/sec<br/></p>
<p>which is the stated perf:</p>
<p>http://www.nvidia.com/content/tesla/pdf/NVIDIA-Tesla-Kepler-Family-Datasheet.pdf “</p>
<p>Person. “Comparing CPU and GPU Theoretical GFLOPS.” NVIDIA Developer Forums, May 21, 2014. <a href="https://forums.developer.nvidia.com/t/comparing-cpu-and-gpu-theoretical-gflops/33335">https://forums.developer.nvidia.com/t/comparing-cpu-and-gpu-theoretical-gflops/33335</a>.<a class="easy-footnote-to-top" href="#easy-footnote-4-2316"></a></p>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-5-2316"></span>From this blog post on the performance of TensorCores, a component of new Nvidia GPUs specialized for deep learning: “The problem is it’s totally unclear how to approach the peak performance of 120 TFLOPS, and as far as I know, no one could achieve so significant speedup on real tasks. Let me know if you aware of good cases.”<br/>
                  Sapunov, Grigory. “Hardware for Deep Learning. Part 3: GPU.” Medium. Intento, January 20, 2020. <a href="https://blog.inten.to/hardware-for-deep-learning-part-3-gpu-8906c1644664">https://blog.inten.to/hardware-for-deep-learning-part-3-gpu-8906c1644664</a>.<a class="easy-footnote-to-top" href="#easy-footnote-5-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-6-2316"></span>Gupta, Geetika. “Difference Between Single-, Double-, Multi-, Mixed-Precision: NVIDIA Blog.” The Official NVIDIA Blog, November 21, 2019. <a href="https://blogs.nvidia.com/blog/2019/11/15/whats-the-difference-between-single-double-multi-and-mixed-precision-computing/">https://blogs.nvidia.com/blog/2019/11/15/whats-the-difference-between-single-double-multi-and-mixed-precision-computing/</a>.<a class="easy-footnote-to-top" href="#easy-footnote-6-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-7-2316"></span>See our <a href="/doku.php?id=ai_timelines:2017_trend_in_the_cost_of_computing">2017 analysis</a>, footnote 4, which notes that single-precision price performance seems to be improving while double-precision price performance is not<a class="easy-footnote-to-top" href="#easy-footnote-7-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-8-2316"></span>“With the growing importance of deep learning and energy-saving approximate computing, half precision floating point arithmetic (FP16) is fast gaining popularity. Nvidia’s recent Pascal architecture was the first GPU that offered FP16 support.”<br/>
                  N. Ho and W. Wong, <a href="https://ieeexplore.ieee.org/abstract/document/8091072">“Exploiting half precision arithmetic in Nvidia GPUs,”</a> 2017 IEEE High Performance Extreme Computing Conference (HPEC), Waltham, MA, 2017, pp. 1-7.<a class="easy-footnote-to-top" href="#easy-footnote-8-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-9-2316"></span>“In a recent paper, Google revealed that its TPU can be up to 30x faster than a GPU for inference (the TPU can’t do training of neural networks). As the main provider of chips for machine learning applications, Nvidia took some issue with that, arguing that some of its existing inference chips were already highly competitive to the TPU.”<br/>
                  Armasu, Lucian. “On Tensors, Tensorflow, And Nvidia’s Latest ‘Tensor Cores’.” Tom’s Hardware. Tom’s Hardware, May 11, 2017. <a href="https://www.tomshardware.com/news/nvidia-tensor-core-tesla-v100,34384.html">https://www.tomshardware.com/news/nvidia-tensor-core-tesla-v100,34384.html</a>.<a class="easy-footnote-to-top" href="#easy-footnote-9-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-10-2316"></span>“Tensor Cores in NVIDIA Volta GPU Architecture.” NVIDIA. Accessed May 2, 2020. https://www.nvidia.com/en-us/data-center/tensorcore/.<br/>
<a class="easy-footnote-to-top" href="#easy-footnote-10-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-11-2316"></span>“Volta is equipped with 640 Tensor Cores, each performing 64 floating-point fused-multiply-add (FMA) operations per clock. That delivers up to 125 TFLOPS for training and inference applications.”<br/>
                  “Tensor Cores in NVIDIA Volta GPU Architecture.” NVIDIA. Accessed March 25, 2020. <a href="https://www.nvidia.com/en-us/data-center/tensorcore/">https://www.nvidia.com/en-us/data-center/tensorcore/</a>.<a class="easy-footnote-to-top" href="#easy-footnote-11-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-12-2316"></span>“A useful operation in computer linear algebra is multiply-add: calculating the sum of a value c with a product of other values a x b to produce c + a x b. Typically, thousands of such products may be summed in a single accumulator for a model such as ResNet-50, with many millions of independent accumulations when running a model in deployment, and quadrillions of these for training models.”<br/>
                  Johnson, Jeff. “Making Floating Point Math Highly Efficient for AI Hardware.” Facebook AI Blog, November 8, 2018. <a href="https://ai.facebook.com/blog/making-floating-point-math-highly-efficient-for-ai-hardware/">https://ai.facebook.com/blog/making-floating-point-math-highly-efficient-for-ai-hardware/</a>.<a class="easy-footnote-to-top" href="#easy-footnote-12-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-13-2316"></span>See Figure 2:<br/>
                  Gupta, Geetika. “Using Tensor Cores for Mixed-Precision Scientific Computing.” NVIDIA Developer Blog, April 19, 2019. <a href="https://devblogs.nvidia.com/tensor-cores-mixed-precision-scientific-computing">https://devblogs.nvidia.com/tensor-cores-mixed-precision-scientific-computing</a>/.<a class="easy-footnote-to-top" href="#easy-footnote-13-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-14-2316"></span>Three different individuals told us about this constraint, including one Nvidia employee.<a class="easy-footnote-to-top" href="#easy-footnote-14-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-15-2316"></span>See the ‘Source’ column in <a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing">this spreadsheet</a>, tab ‘GPU Data’. We largely used <a href="https://www.techpowerup.com/">TechPowerUp</a>, Wikipedia’s <a href="https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units">List of Nvidia GPUs</a>, <a href="https://en.wikipedia.org/wiki/List_of_AMD_graphics_processing_units">List of AMD GPUs</a>, and <a href="https://docs.google.com/spreadsheets/d/1xAo6TcSgHdd25EdQ-6GqM0VKbTYu8cWyycgJhHRVIgY/edit#gid=0">this document listing GPU performance</a>.<a class="easy-footnote-to-top" href="#easy-footnote-15-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-16-2316"></span>See Wikipedia’s <a href="https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units">List of Nvidia GPUs</a> and <a href="https://en.wikipedia.org/wiki/List_of_AMD_graphics_processing_units">List of AMD GPUs</a>.<a class="easy-footnote-to-top" href="#easy-footnote-16-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-17-2316"></span>“CPI Home.” U.S. Bureau of Labor Statistics. U.S. Bureau of Labor Statistics. Accessed May 2, 2020. https://www.bls.gov/cpi/.<a class="easy-footnote-to-top" href="#easy-footnote-17-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-18-2316"></span>The Facebook group is for posting and claiming bounties and has around 750 people, many with interests in computers. The bounty has been up for two months, as of March 13 2020.<a class="easy-footnote-to-top" href="#easy-footnote-18-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-19-2316"></span>See <a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing">this spreadsheet</a>, tab ‘Cleaned GPU Data for SP’ for the chart generation.<a class="easy-footnote-to-top" href="#easy-footnote-19-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-20-2316"></span>See <a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing">this spreadsheet</a>, tab ‘Cleaned GPU Data for SP Minimums’ for the plotting. We used <a href="https://drive.google.com/open?id=1JP98EP8nYA0KqofLm24vF2vwNL0PalcB">this script</a> on the data from the ‘Cleaned GPU Data for SP’ to calculate the minimums and then import them into a new sheet of the spreadsheet.<a class="easy-footnote-to-top" href="#easy-footnote-20-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-21-2316"></span>See <a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing">this spreadsheet</a>, tab ‘Cleaned GPU Data for SP Minimums’ for the calculation.<a class="easy-footnote-to-top" href="#easy-footnote-21-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-22-2316"></span>See <a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing">this spreadsheet</a>, tab ‘Cleaned GPU Data for HP’ for the chart generation.<a class="easy-footnote-to-top" href="#easy-footnote-22-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-23-2316"></span>See <a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing">this spreadsheet</a>, tab ‘Cleaned GPU Data for HP Minimums’ for the plotting. We used <a href="https://drive.google.com/open?id=1JP98EP8nYA0KqofLm24vF2vwNL0PalcB">this script</a> on the data from the ‘Cleaned GPU Data for HP’ to calculate the minimums and then import them into a new sheet of the spreadsheet.<a class="easy-footnote-to-top" href="#easy-footnote-23-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-24-2316"></span>Where ambiguous, we assume these trends are exponential rather than linear, because our understanding is that that is much more common historically in computing hardware price trends.<a class="easy-footnote-to-top" href="#easy-footnote-24-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-25-2316"></span>See <a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing">this spreadsheet</a>, tab ‘Cleaned GPU Data for HP Minimums’ for the calculation.<a class="easy-footnote-to-top" href="#easy-footnote-25-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-26-2316"></span>See <a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing">this spreadsheet</a>, tab ‘Cleaned GPU Data for HP + Tensor Cores’ for the chart generation.<a class="easy-footnote-to-top" href="#easy-footnote-26-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-27-2316"></span>See <a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing">this spreadsheet</a>, tab ‘Cleaned GPU Data for HP + Tensor Cores Minimums’ for the plotting. We used <a href="https://drive.google.com/open?id=1JP98EP8nYA0KqofLm24vF2vwNL0PalcB">this script</a> on the data from the ‘Cleaned GPU Data for HP + Tensor Cores’ to calculate the minimums and then import them into a new sheet of the spreadsheet.<a class="easy-footnote-to-top" href="#easy-footnote-27-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-28-2316"></span>See <a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing">this spreadsheet</a>, tab ‘Cleaned GPU Data for HP + Tensor Cores Minimums’ for the calculation.<a class="easy-footnote-to-top" href="#easy-footnote-28-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-29-2316"></span>For example, one of the GPUs recommended for deep learning <a href="https://www.reddit.com/r/MachineLearning/comments/b95182/d_which_gpus_to_get_for_deep_learning_my/">in this Reddit thread</a> is the GTX 1060 (6GB), which has been around <a href="https://www.techpowerup.com/gpu-specs/geforce-gtx-1060-6-gb.c2862">since 2016</a>.<a class="easy-footnote-to-top" href="#easy-footnote-29-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-30-2316"></span>We scraped data from <a href="https://www.techpowerup.com/gpu-specs/radeon-rx-480.c2848">individual TechPowerUp pages</a> using <a href="https://drive.google.com/open?id=1msy977jSLcJspULMWlWOfIIFRWo_vsds">this script</a>. Our full scraped TechPowerUp dataset can be found <a href="https://docs.google.com/spreadsheets/d/1pXTyUJ2AvpkhYtphGn8UnAl4gI8j2jsx1AaGrOahl7o/edit?usp=sharing">here</a>.<a class="easy-footnote-to-top" href="#easy-footnote-30-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-31-2316"></span>We chose to automatically scrape theoretical peak performance numbers from TechPowerUp instead of using the ones we manually collected above because there were several GPUs in the active pricing datasets that we hadn’t collected data for manually, and it was easier to scrape the entire site than just the subset of GPUs we needed.<a class="easy-footnote-to-top" href="#easy-footnote-31-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-32-2316"></span>“CPI Home.” U.S. Bureau of Labor Statistics. U.S. Bureau of Labor Statistics. Accessed May 2, 2020. https://www.bls.gov/cpi/.<a class="easy-footnote-to-top" href="#easy-footnote-32-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-33-2316"></span>We used <a href="https://drive.google.com/open?id=1nd7111hOb-eCMBCOe1qocXYuAJys0pLk">this script</a>.<a class="easy-footnote-to-top" href="#easy-footnote-33-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-34-2316"></span>“PassMark – GeForce GTX 660 – Price Performance Comparison.” Accessed March 24, 2020. <a href="https://www.videocardbenchmark.net/gpu.php?gpu=GeForce+GTX+660&amp;id=2152">https://www.videocardbenchmark.net/gpu.php?gpu=GeForce+GTX+660&amp;id=2152</a>.<a class="easy-footnote-to-top" href="#easy-footnote-34-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-35-2316"></span>In most cases where renaming was necessary, the same GPU had multiple clear names, e.g. the “Radeon HD 7970 / R9 280X” in PassMark was just called the “Radeon HD 7970” in TechPowerUp. In a few cases, Passmark listed some GPUs which TechPowerUp listed separately as one GPU, e.g. “Radeon R9 290X / 390X” seemed to ambiguously refer to the Radeon R9 290X or Radeon R9 390X. In these cases, we conservatively assume that the GPU refers to the less powerful / earlier GPU. In one exceptional case, we assumed that the “Radeon R9 Fury + Fury X” referred to the Radeon Fury X in PassMark. The ambiguously named GPUs were not in the minimum data we calculated, so probably did not have a strong effect on the final result.<a class="easy-footnote-to-top" href="#easy-footnote-35-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-36-2316"></span>For example, the Kaggle dataset includes extremely cheap FirePro S7150s sold in 2014, even though the FirePro S7150 only came out in 2016. One of the sellers of these cheap GPUs were ‘Club 3D’, which also appeared to sell several other erroneously cheap GPUs.<a class="easy-footnote-to-top" href="#easy-footnote-36-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-37-2316"></span>See <a href="https://drive.google.com/open?id=1uvO-gNpAGh9qMzs5MnNtOg1X-R7JzdAq">this plot of Passmark single-precision GPU price / FLOPS</a> compared to the <a href="https://drive.google.com/open?id=194Tqrcix2XdytbT-WbgcRbFpeHIHqsao">combined Passmark and Kaggle single-precision GPU price / FLOPS</a>, and <a href="https://drive.google.com/open?id=1hk-1rHqOkWUwh3BVNbwTEu8dEfjDhsLT">this plot of Passmark half-precision GPU price / FLOPS</a> compared to the <a href="https://drive.google.com/open?id=1mcpQigPJs9CRqebY-Uvm0dJ1si1jeTQv">combined Passmark and Kaggle half-precision $ / FLOPS</a>. In both cases the 2013 – 2018 Passmark data appears to roughly be a subset of the Kaggle data.<a class="easy-footnote-to-top" href="#easy-footnote-37-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-38-2316"></span>“CPI Home.” U.S. Bureau of Labor Statistics. U.S. Bureau of Labor Statistics. Accessed May 2, 2020. https://www.bls.gov/cpi/.<a class="easy-footnote-to-top" href="#easy-footnote-38-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-39-2316"></span>We set this period to be 10 days long when looking at single-precision data, and 30 days long when looking at half-precision data, since half-precision data was significantly more sparse.<a class="easy-footnote-to-top" href="#easy-footnote-39-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-40-2316"></span>This calculation can be found in <a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing">this spreadsheet</a>.<a class="easy-footnote-to-top" href="#easy-footnote-40-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-41-2316"></span>We used a Python plotting library to generate our plots, the script can be found <a href="https://drive.google.com/open?id=1u3qI9m9W6_9efIpsDBq1Hc-qixcKy8Sb">here</a>. All of our resulting plots can be found <a href="https://drive.google.com/open?id=1afVbKn34pw5rj4fn1vI_qOhdZh5GLfwE">here</a>. ‘single’ vs. ‘half’ refers to whether its $ / FLOPS data for single or half-precision FLOPS, ‘passmark’, ‘kaggle’, and ‘combined’ refer to which dataset is being plotted and ‘raw’ vs. ‘95’ refer to whether we’re plotting all the data or the 95th percentile data.<a class="easy-footnote-to-top" href="#easy-footnote-41-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-42-2316"></span>The dataset we used for this plot can be found <a href="https://drive.google.com/open?id=19FB-_MnQAtJErb8e_5DHTRVKAPg4g1YX">here</a>. This a processed version of our scraped dataset, with prices / FLOPS adjusted for inflation. The script we used to process and plot can be found <a href="https://drive.google.com/open?id=1u3qI9m9W6_9efIpsDBq1Hc-qixcKy8Sb">here</a>.<a class="easy-footnote-to-top" href="#easy-footnote-42-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-43-2316"></span>The script to calculate the 95th percentile and generate this plot can be found <a href="https://drive.google.com/open?id=1u3qI9m9W6_9efIpsDBq1Hc-qixcKy8Sb">here</a>.<a class="easy-footnote-to-top" href="#easy-footnote-43-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-44-2316"></span>See <a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing">here</a>, tab ‘Passmark SP Minimums’ to see our calculation of the minimums over time. We used <a href="https://drive.google.com/open?id=1yRTJwVQAwCqLSTyGXgGIHcRfJFP7C5D2">this script</a> to generate the minimums, then imported them into this spreadsheet.<a class="easy-footnote-to-top" href="#easy-footnote-44-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-45-2316"></span>You can see our calculations for this <a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing">here</a>, sheet ‘Passmark SP Minimums’. Each sheet has a cell ‘Rate to move an order of magnitude’ which has our calculation for how many years we need to move an order of magnitude. In the (untrustworthy) Kaggle dataset alone, its rate would yield an order of magnitude of decrease every ~12 years, and the rate in the combined dataset  would yield an order of magnitude of decrease every ~16 years.<a class="easy-footnote-to-top" href="#easy-footnote-45-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-46-2316"></span>Orloff, Jeremy, and Jonathan Bloom. “Bootstrap Confidence Intervals.” MIT OpenCourseWare, 2014. <a href="https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf">https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf</a>.<a class="easy-footnote-to-top" href="#easy-footnote-46-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-47-2316"></span>We used <a href="https://drive.google.com/open?id=1XkA-8WruAMKM3y3cdNMPNJHabpMUE_MT">this script</a> to generate bootstrap confidence intervals for our datasets.<a class="easy-footnote-to-top" href="#easy-footnote-47-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-48-2316"></span>We think this is the case because we’ve observed this dip in other GPU analyses we’ve done, and because the timing lines up: <a href="https://www.techspot.com/news/72854-nvidia-asking-graphics-card-retailers-prioritize-gamers-over.html">the first table in this article</a> shows how GPU prices were increasing starting 2017 and continued to increase through 2018, and <a href="https://www.kaggle.com/raczeq/impact-of-cryptocurrencies-rates-on-pc-market/data">the chart here</a> shows how GPU prices increased in 2017.<a class="easy-footnote-to-top" href="#easy-footnote-48-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-49-2316"></span>You can see our calculations for this <a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing">here</a>, sheet ‘Passmark SP Minimums’, next to ‘Exponential trendline from 2015 to 2016. The trendline calculated is technically the linear fit through the log of the data.<a class="easy-footnote-to-top" href="#easy-footnote-49-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-50-2316"></span>See our calculation <a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing">here</a>, tab ‘Cleaned GPU Data for SP Minimums’, next to the cell marked “Exponential trendline from 2011 to 2019.”<a class="easy-footnote-to-top" href="#easy-footnote-50-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-51-2316"></span>At the start of 2011, the <a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing">minimum release</a> <a href="https://docs.google.com/spreadsheets/d/1ZZm5Wgr3BDRtloTZGylWzYTaVr5VqjiwOiRNu5Pz_q8/edit?usp=sharing">price / FLOPS</a> (see tab, ‘Cleaned GPU Data for SP Minimums’) is .000135 $ / FLOPS, whereas the <a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing">minimum active price / FLOPS</a> (see tab, ‘Passmark SP Minimums’) is around .0001 $ / FLOPS. <a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing">The initial GPU price / FLOPS minimum</a> (see sheet ‘Passmark SP Minimums’) corresponds to the Radeon HD 5850 which had a price of $184.9 in 3/2011 and a release price of $259. <a href="https://www.videocardbenchmark.net/gpu.php?gpu=Radeon+HD+5850&amp;id=47">Looking at the general trend in Passmark</a> suggests that the Radeon HD 5850 did indeed rapidly decline from its $259 release price to consistently below $200 prices.<a class="easy-footnote-to-top" href="#easy-footnote-51-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-52-2316"></span> The dataset we used for this plot can be found here. This is a processed version of our scraped dataset, with prices / FLOPS adjusted for inflation. The script we used to process and plot can be found here.<a class="easy-footnote-to-top" href="#easy-footnote-52-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-53-2316"></span>The script to calculate the 95th percentile and generate this plot can be found here.<a class="easy-footnote-to-top" href="#easy-footnote-53-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-54-2316"></span>See <a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing">here</a>, tab ‘Passmark HP Minimums’ to see our calculation of the minimums over time. We used <a href="https://drive.google.com/open?id=1u3qI9m9W6_9efIpsDBq1Hc-qixcKy8Sb">this script</a> to generate the minimums, then imported them into this spreadsheet.<a class="easy-footnote-to-top" href="#easy-footnote-54-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-55-2316"></span>See the sheet marked ‘Passmark HP minimums’ in <a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing">this spreadsheet</a>. The trendline calculated is technically the linear fit through the log of the data.<a class="easy-footnote-to-top" href="#easy-footnote-55-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-56-2316"></span>Orloff, Jeremy, and Jonathan Bloom. “Bootstrap Confidence Intervals.” MIT OpenCourseWare, 2014. <a href="https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf">https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf</a>.<a class="easy-footnote-to-top" href="#easy-footnote-56-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-57-2316"></span>We used <a href="https://drive.google.com/open?id=1XkA-8WruAMKM3y3cdNMPNJHabpMUE_MT">this script</a> to generate bootstrap confidence intervals for our datasets.<a class="easy-footnote-to-top" href="#easy-footnote-57-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-58-2316"></span>See the sheet marked ‘Passmark HP minimums’ in <a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing">this spreadsheet</a>.<a class="easy-footnote-to-top" href="#easy-footnote-58-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-59-2316"></span>The dataset we used for this plot can be found here. This a processed version of our scraped dataset, with prices / FLOPS adjusted for inflation. The script we used to process and plot can be found here.<a class="easy-footnote-to-top" href="#easy-footnote-59-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-60-2316"></span>The script to calculate the 95th percentile and generate this plot can be found here.<a class="easy-footnote-to-top" href="#easy-footnote-60-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-61-2316"></span>See <a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing">here</a>, tab ‘Passmark HP FMA Minimums’ to see our calculation of the minimums over time. We used <a href="https://drive.google.com/open?id=1yRTJwVQAwCqLSTyGXgGIHcRfJFP7C5D2">this script</a> to generate the minimums, then imported them into this spreadsheet.<a class="easy-footnote-to-top" href="#easy-footnote-61-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-62-2316"></span>See the sheet marked ‘Passmark HP FMA minimums’ in <a href="https://docs.google.com/spreadsheets/d/15pTVDml1j81HROZ3_UeHZ51aBoqq-94-eM8N80npUX0/edit?usp=sharing">this spreadsheet</a>. The trendline calculated is technically the linear fit through the log of the data.<a class="easy-footnote-to-top" href="#easy-footnote-62-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-63-2316"></span>Orloff, Jeremy, and Jonathan Bloom. “Bootstrap Confidence Intervals.” MIT OpenCourseWare, 2014. <a href="https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf">https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf</a>.<a class="easy-footnote-to-top" href="#easy-footnote-63-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-64-2316"></span>We used <a href="https://drive.google.com/open?id=1XkA-8WruAMKM3y3cdNMPNJHabpMUE_MT">this script</a> to generate bootstrap confidence intervals for our datasets.<a class="easy-footnote-to-top" href="#easy-footnote-64-2316"></a>
</div></li>
<li><div class="li">
<span class="easy-footnote-margin-adjust" id="easy-footnote-bottom-65-2316"></span>See our analysis in <a href="#single-precision-analysis">this section</a> above.<a class="easy-footnote-to-top" href="#easy-footnote-65-2316"></a>
</div></li>
</ol>
</HTML>
AI Impacts Wiki

User Tools

Site Tools

Page Tools
