Published 17 August, 2023. Last updated 3 January, 2024.

The 2023 Expert Survey on Progress in AI is a survey of 2,778 AI researchers that AI Impacts ran in October 2023.

The 2023 Expert Survey on Progress in AI (2023 ESPAI) is a rerun of the 2022 ESPAI and the 2016 ESPAI, previous surveys ran by AI Impacts in collaboration with others. Almost all of the questions in the 2023 ESPAI are identical to those in both the 2022 ESPAI and 2016 ESPAI.

The questions in the 2023 ESPAI are nearly identical to those in the 2022 ESPAI and the 2016 ESPAI. As in those surveys, different phrasings of some questions were randomly assigned to each respondent to measure the effects of framing differences. Each participant also received a randomized subset of certain question types and questions within certain types. Because of this, most questions have a significantly smaller number of responses than the total number of responses to the survey as a whole.

Some questions were added to the 2023 ESPAI which were not in either of the two previous surveys. We refined the new questions through an iterative process involving several rounds of testing the questions in verbal interviews with computer science graduate students and others.

Full questions

For all authors who published in 2022 at a selection of top-tier machine learning conferences (NeurIPS, ICML, ICLR, AAAI, JMLR, and IJCAI), we were able to find email addresses for 20,066 of them, which accounted for 92% of the collected names. The resulting list of emails was put into a random order based on a random unique number assigned using Google Sheets' “Randomize range” feature. The first 1003 emails from the randomly ordered list (about 5% of the total) were assigned to a pilot study group to receive payment for participating, and the second 1003 emails from the list were assigned to a pilot study group to not receive payment for participating. The remainder of the emails were assigned to the main survey group

The pilot study took place from October 11 to October 15 in 2023. Based on the response rates in the paid group versus the unpaid group, we decided to offer payment to all survey participants. A $50 reward will be issued through a third-party service. Depending on a participant's country (as determined by IP address), participants will be able to use the third-party service to choose between a gift card, a pre-paid Mastercard, and a donation to their choice of 15 charities.

On October 15, the survey was sent to the main survey group. The survey remained open until October 24, 2023. Out of the 20,066 emails we contacted, 1,607 (8%) bounced or failed, leaving 18,459 functioning email addresses. We received 2,778 responses, for a response rate of 15%. 95% of these responses were deemed ‘finished’ by Qualtrics.

These are some notable differences from the 2022 Expert Survey on Progress in AI

• We recruited participants from twice as many conferences as in 2022.
• We made some changes to the order and flow of the questions.
• We gave some participants a new version of the extinction risk question that was phrased to include a timeframe (“within the next 100 years”).
• Some participants were given new questions that were not in the 2022 ESPAI.

Full data on differences between surveys is available here.

• “Aggregate forecast”: A cumulative distribution function that combines year-probability pairs from both the fixed-years and fixed-probabilities framings. The aggregate forecast is found by fitting each individual response (which consists of three year-probability pairs) to a gamma CDF and then finding the average curve of those.
• “Fixed-probabilities framing”: A way of asking about how soon an AI milestone will be reached, by asking for an estimated year by which the milestone will be feasible with a given probability.
• “Fixed-years framing”: A way of asking about how soon an AI milestone will be reached, by asking for an estimated probability that the milestone will be reached by a given year.
• “Full automation of labor (FAOL)”: When for any occupation, machines could be built to carry out the task better and more cheaply than human workers.
• “High-level machine intelligence (HLMI)“: AI that can, unaided, accomplish every task better and more cheaply than human workers.