In the course of our work, we have noticed a number of empirical questions which bear on our forecasts and might be (relatively) cheap to resolve. In the future we hope to address some of these.
Look at the work of ancient or enlightenment mathematicians and control for possible selection effects in
this analysis of historical mathematical conjectures.
Look for historical characterizations of the AI problem, and try to obtain unbiased (though uninformed) breakdowns of the problem which could be used to gauge progress.
Identify previous examples of technological projects with clear long-term goals, and then produce estimates of the time required to achieve those goals to varying degrees.
Analyze the performance of different versions of software for benchmark problems, like SAT solving or chess, and determine the extent to which hardware and software progress facilitated improvement.
Obtain a clearer picture of the extent to which historical developments in neuroscience have played a meaningful role in historical progress in AI. Our impression is that this influence has been minimal, but this judgment might be attributable to hindsight bias.
In the field of AI, estimate the ratio of spending on hardware to spending on researchers.
Estimate the change in inputs in mathematicians, scientists, or engineers, as a complement to estimates for rates of progress in those fields.
Estimate the historical and present size of the AI field, ideally with plausible adjustments for quality (for example performing in-depth investigations for a small number of random samples, perhaps invoking expert opinion) and using these as a basis for quality-adjustments.
Unfortunately this is an incomplete list (even of the ideas which have struck as promising during this project). We are beginning to flesh it out further in our aforementioned list of projects bearing on AI timelines.