Published 16 August, 2016; last updated 08 October, 2017

There are many current efforts to mitigate risks from artificial intelligence. We might learn something about the likelihood of these efforts influencing AI risk by looking at similar past efforts.

To this end, we are interested here in past risk mitigation efforts that have the following characteristics (taken from this paper contributing to the same project (p5) and motivated by this discussion and others surrounding it):

• Novelty: the event predicted is relatively unlike problems that have been seen before.
• Low scientific concern: highly credentialed and credible scientists are not very concerned.
• Complex prediction: the prediction is relatively complicated, e.g., more complex than the basic physical dynamics of climate change.
• Specificity of solution: the policy response is narrowly useful for the predicted issue (e.g., it does not contribute valuable basic research, or general capacity building that might be worthwhile in the absence of the problem).
• Low feedback: the response is a one-shot effort rather than an iterative process with frequent feedback about success.
• Early action: an event is predicted and the beginnings of action take place fifteen years or more before the event is predicted to occur.

These features are relevant because it has been suggested that AI safety efforts have them, and very few successful efforts have them, perhaps suggesting that AI safety efforts will not be successful. If it is true that there are few successful risk mitigation efforts with these characteristics, we want to know if few such efforts are made, or if they tend to fail. If the latter, this may indeed justify pessimism about the AI safety project.

So, for plausible efforts, we want to know:

• How well do they fit in this reference class?
• Were they successful?
• If not, how did they fail? Can we learn anything useful from the specifics?

For now, this page addresses the first question, for cases we have found which fit the reference class relatively well.

Leó Szilárd’s secret nuclear patent

Leó Szilárd patented the nuclear chain reaction and anticipated its use in explosives by 1934.¹ He offered the patent to the British War Office to be kept secret, and had them accept it by 1936.²

Katja Grace investigated the relevance of several of Leó Szilárd’s efforts to avert nuclear risks in Leó Szilárd and the Danger of Nuclear Weapons: A Case Study in Risk Mitigation (2015). In summary from that report, the threat of nuclear weapons was fairly novel. Nuclear danger was probably expected to strike fairly abruptly in the future with little opportunity for feedback and learning. Other scientists were mostly disinterested in the concern for at least five years, in part because of the perceived speculativeness of the physics (not even fission had been discovered), but also the complications involved in the physics leading to weapons to be used in a near time war. It’s unclear how soon Szilárd expected such weapons, but quite plausibly at least fifteen years later (it turned out to be about eleven, between 1934 and 1945). The secret patent does not seem broadly useful, though useful for encouraging more local nuclear research, which is more broadly useful than secrecy per se.  In sum, this seems like a reasonably good example.

Leó Szilárd’s other nuclear secrecy efforts

In 1939 and 1940 Leo Szilárd was instrumental in pushing for secrecy of nuclear research findings, to prevent them reaching Germany and contributing to German weapons efforts.³

Most of the details in the above example (‘Leó Szilárd’s secret nuclear patent’) apply here also, because the problem is the same. The solution occurred later, so was somewhat less likely to have been fifteen years before the problem was anticipated, though plausibly still was. The solution was probably less broadly useful than the secret patent (making this a better example) because secrecy efforts mostly function to slow foreign research, whereas the secret patent was arguably intended to speed local research (which has broader uses).

The Einstein-Szilárd letter

Leó Szilárd and Albert Einstein wrote to president Roosevelt in 1939, warning him that Germany might make atomic bombs, and suggesting that America pursue such weapons. Wikipedia discusses the event in more detail, including a copy of the letter.

Most of the details in the above examples, detailed under ‘Leó Szilárd’s secret nuclear patent’, apply here also, because the problem is the same. The letter was written in 1939, only six years before the first nuclear bomb, though it is unclear how soon Einstein and Szilárd anticipated a bomb at that time. Writing a letter was intended to speed local nuclear research, which has somewhat broad applicability.

The World Set Free

The World Set Free is a book published in 1914 by H. G. Wells. It describes a world with nuclear weapons, so may have been intended to warn about the prospect of this.

The first nuclear weapons were used in 1945, and neutrons were not discovered until 1932.⁴ So the prospect of nuclear weapons was extremely speculative, and not accepted by most scientists (those things were still true in 1934—for details see ‘Leó Szilárd’s secret nuclear patent’, above). Indeed, the weapons imagined by Wells are indeed quite different from real nuclear weapons—each one is small, but burns indefinitely, making past war zones uninhabitable. Our guess is that the book was also probably published many decades earlier than the time when Wells anticipated such a risk in the real world.

Nuclear weapons were novel, and Wells could arguably not receive a lot of feedback about his influence on their outcome.

The main way in which this fails as an example is that writing a fiction book is not a very specifically directed solution. In particular, writing books is often valuable enough for its own sake that people do it without attempting to have any effect on the outside world. For this reason, it is not even clear to us whether Wells wrote this book in order to improve the problem, or merely to comment on it or use it for other political purposes.

We think this is a moderately good example.

Preliminary survey of prescient actions

1. Szilárd filed a patent on the chain reaction in 1934.[7] ‘In amendments dated June 28 and July 4, 1934, he described a chain made from particles with no charge, of approximately the mass of a proton; the critical mass; tamper; and the potential for explosives.[8]’ – Leó Szilárd and the Danger of Nuclear Weapons: A Case Study in Risk Mitigation
2. ‘Szilárd asked the British War Office to keep the patent secret but was turned down.[13] In February 1936, he offered it successfully to the British Admiralty with the help of a letter from physicist Alexander Lindemann, director of Clarendon Laboratory at Oxford University.[14] Szilárd explained in the letter to the Admiralty that the patented process could contribute to bombs “very many thousand times more powerful than ordinary bombs” and drew attention to the threat of foreign powers using them.[15]’ – Leó Szilárd and the Danger of Nuclear Weapons: A Case Study in Risk Mitigation 
3. Following the astonishing discovery of fission in late 1938, Szilárd, along with fellow physicists such as Teller, Wigner, and Weisskopf, pushed for other physicists to withhold publication of sensitive papers.[19],[20] By May 1940, Szilárd was known throughout the US physics community as the leading proponent of secrecy regarding fission research.[21] Through conversations and letters, Szilárd and his collaborators convinced many key scientists and journal editors to maintain secrecy.[22],[23],[24],[25]
4. ‘The essential nature of the atomic nucleus was established with the discovery of the neutron by James Chadwick in 1932.’ – Wikipedia