ai_timelines:hardware_and_ai_timelines:computing_capacity_of_all_gpus_and_tpus
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
ai_timelines:hardware_and_ai_timelines:computing_capacity_of_all_gpus_and_tpus [2023/04/03 18:46] harlanstewart |
ai_timelines:hardware_and_ai_timelines:computing_capacity_of_all_gpus_and_tpus [2024/01/24 23:06] (current) harlanstewart |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| + | /* | ||
| + | EDITOR NOTES (publicly accessible) | ||
| + | -Harlan: I plan to update this page once global market size estimates for GPUs become available for 2023 | ||
| + | */ | ||
| ====== How much computing capacity exists in GPUs and TPUs in Q1 2023? ====== | ====== How much computing capacity exists in GPUs and TPUs in Q1 2023? ====== | ||
| + | |||
| + | //Published 3 April 2023, last updated 3 April 2023// | ||
| A back-of-the-envelope calculation based on market size, price-performance, | A back-of-the-envelope calculation based on market size, price-performance, | ||
| Line 9: | Line 15: | ||
| Consulting firms reported their estimates of the world-wide GPU market size for 2017, 2018, 2019, 2020, 2021, and 2022.((Bhutani, | Consulting firms reported their estimates of the world-wide GPU market size for 2017, 2018, 2019, 2020, 2021, and 2022.((Bhutani, | ||
| ))((“GPU Market Size, Share, Trends | Forecast 2026.” n.d. Acumen Research and Consulting. Accessed March 31, 2023. https:// | ))((“GPU Market Size, Share, Trends | Forecast 2026.” n.d. Acumen Research and Consulting. Accessed March 31, 2023. https:// | ||
| - | ))(( “Graphic Processing Unit (GPU) Market Size, Share, Trends & Forecast.” 2022. Verified Market Research. https:// | + | ))(( “Graphic Processing Unit (GPU) Market Size, Share, Trends & Forecast.” 2022. Verified Market Research. https:// |
| + | |||
| + | * Our best guess estimate models market size growth as a piecewise linear function, with slow growth before 2017 and fast growth after 2017. | ||
| + | * Our lower bound estimate fits the years before 2017 to an exponential curve and assumes that the market size will not change between 2022 and 2023. | ||
| + | * Our upper bound estimate assumes that there was no change in market size between 2010 and 2017 and fits 2023 to an exponential curve. | ||
| {{: | {{: | ||
| Line 15: | Line 25: | ||
| ==== Estimating FLOP/s per dollar GPUs had per year ==== | ==== Estimating FLOP/s per dollar GPUs had per year ==== | ||
| - | Hobbhahn & Besiroglu (2022) used a dataset of GPUs to find trends in FLOP/s per dollar over time. This provides a simple way to convert the above estimates of total spending on GPUs per year into estimates of total GPU computing capacity produced per year. Hobbhahn & Besiroglu found no statistically significant difference in the trends of FP16 and FP32 precision compute, and I will follow their example by focusing on FP32 precision for the computing capacity of GPUs.((Hobbhahn, | + | Hobbhahn & Besiroglu (2022) used a dataset of GPUs to find trends in FLOP/s per dollar over time. This provides a simple way to convert the above estimates of total spending on GPUs per year into estimates of total GPU computing capacity produced per year. Hobbhahn & Besiroglu found no statistically significant difference in the trends of FP16 and FP32 precision compute, and we follow their example by focusing on FP32 precision for the computing capacity of GPUs.((Hobbhahn, |
| - | * My best guess estimate for FLOP/s per dollar over time is based on the trendline over all of the data in the dataset. | + | * Our best guess estimate for FLOP/s per dollar over time is based on the trendline over all of the data in the dataset. |
| - | * My lower bound estimate is based on the trendline for machine learning GPUs | + | * Our lower bound estimate is based on the trendline for machine learning GPUs |
| - | * My upper bound estimate is based on the trendline for the most price-performance efficient GPUs | + | * Our upper bound estimate is based on the trendline for the most price-performance efficient GPUs |
| ==== Estimating proportion of GPUs still functional per year ==== | ==== Estimating proportion of GPUs still functional per year ==== | ||
| - | Because GPUs eventually fail, the computing capacity produced in previous years does not all still exist. | + | Because GPUs eventually fail, the computing capacity produced in previous years does not all still exist. |
| - | * Based on anecdotal claims that GPUs usually last 5 years with heavy use or 7 or more years with moderate use, my best guess estimate is that GPU lifespan is normally distributed with a mean of 6 years and a standard deviation of 1.5 years.(( “How Long Do GPUs Last? (Average Lifespan & Effectiveness).” n.d. Cybersided. Accessed March 31, 2023. https:// | + | * Based on anecdotal claims that GPUs usually last 5 years with heavy use or 7 or more years with moderate use, our best guess estimate is that GPU lifespan is normally distributed with a mean of 6 years and a standard deviation of 1.5 years.(( “How Long Do GPUs Last? (Average Lifespan & Effectiveness).” n.d. Cybersided. Accessed March 31, 2023. https:// |
| - | * Because Ostrouchov et al. (2017) show that GPUs in datacenters usually last around 3 years, | + | * Because Ostrouchov et al. (2017) show that GPUs in datacenters usually last around 3 years, |
| - | * For my upper bound estimate, | + | * For our upper bound estimate, |
| {{: | {{: | ||
| Line 33: | Line 43: | ||
| ==== Estimating FLOP/s of all GPUs ==== | ==== Estimating FLOP/s of all GPUs ==== | ||
| - | To estimate total GPU computing capacity, | + | To estimate total GPU computing capacity, |
| - | * My best guess estimate for total GPU computing capacity is 3.95 * 10^21 FLOP/s (FP32) | + | * Our best guess estimate for total GPU computing capacity is 3.95 * 10^21 FLOP/s (FP32) |
| - | * My lower bound estimate for total GPU computing capacity is 1.40 * 10^21 FLOP/s (FP32) | + | * Our lower bound estimate for total GPU computing capacity is 1.40 * 10^21 FLOP/s (FP32) |
| - | * My upper bound estimate for total GPU computing capacity is 7.71 * 10^21 FLOP/s (FP32) | + | * Our upper bound estimate for total GPU computing capacity is 7.71 * 10^21 FLOP/s (FP32) |
| ==== Estimating FLOP/s of all TPUs ==== | ==== Estimating FLOP/s of all TPUs ==== | ||
| Line 43: | Line 53: | ||
| TPUs are specialized chips designed by Google for machine learning. These chips can be rented through the cloud, and seem to be located in 7 Google data centers.((“TPU regions and zones.” n.d. Google Cloud. Accessed March 31, 2023. https:// | TPUs are specialized chips designed by Google for machine learning. These chips can be rented through the cloud, and seem to be located in 7 Google data centers.((“TPU regions and zones.” n.d. Google Cloud. Accessed March 31, 2023. https:// | ||
| - | * Based on assumptions that all of Google’s data centers that have TPUs have the same number of TPUs and that if a data center has two types of TPUs it has equal numbers of each, my best guess estimate for computing capacity of all TPUs is 2.93 * 10^19 FLOP/s (bf16) | + | * Based on assumptions that all of Google’s data centers that have TPUs have the same number of TPUs and that if a data center has two types of TPUs it has equal numbers of each, our best guess estimate for computing capacity of all TPUs is 2.93 * 10^19 FLOP/s (bf16) |
| - | * Because the only computing capacity publicly reported by Google is that of the Oklahoma data center, | + | * Because the only computing capacity publicly reported by Google is that of the Oklahoma data center, |
| - | * Because Google claimed that their Oklahoma data center had more computing capacity than any other publicly available compute cluster, | + | * Because Google claimed that their Oklahoma data center had more computing capacity than any other publicly available compute cluster, |
| ==== Estimated FLOP/s of all GPUs and TPUs ==== | ==== Estimated FLOP/s of all GPUs and TPUs ==== | ||
| Line 53: | Line 63: | ||
| Wang, Shibo, and Pankaj Kanwar. 2019. “BFloat16: | Wang, Shibo, and Pankaj Kanwar. 2019. “BFloat16: | ||
| - | * My best guess estimate for computing capacity of all GPUs and TPUs is 3.98 * 10^21 FLOP/s | + | * Our best guess estimate for computing capacity of all GPUs and TPUs is 3.98 * 10^21 FLOP/s |
| - | * My lower bound estimate is 1.41 * 10^21 FLOP/s | + | * Our lower bound estimate is 1.41 * 10^21 FLOP/s |
| - | * My upper bound estimate is 7.77 * 10^21 FLOP/s | + | * Our upper bound estimate is 7.77 * 10^21 FLOP/s |
| ===== Discussion ===== | ===== Discussion ===== | ||
| Line 61: | Line 71: | ||
| This calculation is rough, and the estimates could be wrong. In particular, some uncertainty remains about the methods used by the cited consulting firms to estimate GPU market size, and the details of those methods might substantially change the estimates in either direction. | This calculation is rough, and the estimates could be wrong. In particular, some uncertainty remains about the methods used by the cited consulting firms to estimate GPU market size, and the details of those methods might substantially change the estimates in either direction. | ||
| - | Google’s Pathways Language Model (PaLM) was trained using 2.56 * 10^24 FLOPs over 64 days.((Chowdhery, | + | Google’s Pathways Language Model (PaLM) was trained using 2.56 * 10^24 FLOPs over 64 days.((Chowdhery, |
| There are likely other, possibly better, ways that one could go about estimating the computing capacity of GPUs and TPUs. Estimations that are calculated in different ways could be useful for reducing uncertainty. Other methods might involve researching the manufacturing capacity of semiconductor fabrication plants. Other methods may also make it feasible to estimate the computing capacity of all microprocessors, | There are likely other, possibly better, ways that one could go about estimating the computing capacity of GPUs and TPUs. Estimations that are calculated in different ways could be useful for reducing uncertainty. Other methods might involve researching the manufacturing capacity of semiconductor fabrication plants. Other methods may also make it feasible to estimate the computing capacity of all microprocessors, | ||
ai_timelines/hardware_and_ai_timelines/computing_capacity_of_all_gpus_and_tpus.1680547594.txt.gz · Last modified: 2023/04/03 18:46 by harlanstewart
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC0 1.0 Universal
