User Tools

Site Tools


responses_to_ai:examples_of_warning_signs

Examples of Warning Signs

Published: 19 December, 2023; Last Updated: 19 December, 2023.

Epistemic Status: Varies. Some of the examples here come from thorough case studies for our other projects. We are less confident of the examples which were not part of a dedicated case study: some of the sources cited were not read thoroughly, and we did not check to make sure that they are representative of the relevant literature.

This page lists some examples of warning signs from the history of technologies other than AI.

Model

Sometimes, to understand the development of an emerging technology and the policy surrounding it, it is important to understand a few key events, called 'warning signs.' Activists use these events to argue that the specific technology, or the industry more generally, has important risks that need to be addressed. This galvanizes public support and puts pressure on policy makers. The policies enacted in the wake of these events are important for that industry for at least decades to come.

Definitions

There are multiple terms that are used to refer to these key events in the history of technology policy. The terms do not mean exactly the same thing, and are used inconsistently.

“A warning shot is a global catastrophe that indirectly reduces existential risk by increasing concern about future catastrophes.”1) To be a warning shot under this definition, large scale harm has to occur.

A fire alarm is something which creates “common knowledge that action is now due and socially acceptable.”2) The focus here is the creation of common knowledge, rather than the harm caused.

A warning sign is “any event that increases concern about a particular category of existential risk, regardless of whether the event itself constitutes a global catastrophe. … Note, however, that both 'warning shot' and 'fire alarm' are sometimes used as synonyms for 'warning sign.'”3)

A trigger event is “a 'highly publicized, shocking incident' that 'dramatically reveals a critical social problem to the public in a vivid way.'”4) This term is not specific to technology policy, and comes from the general theory of protest. A related term is a moment of the whirlwind. The focus of this term is on the decision maker at a critical moment, rather than the event leading to that critical moment.

A Sputnik moment is “a trigger mechanism, an event that makes people collectively say that they need to do something, and this sets a course in another direction,” especially when a country recognizes that its competitors have some technological ability that they do not currently have.5)

A galvanizing event is an event that stimulates people's willingness to take action.

Examples

Below are some examples of warning signs for various industries: nuclear power, fossil fuels, chlorofluorocarbons, genetically modified organisms, medical research involving viruses, and medical research involving humans. For some of them, the main concern is not existential risk. Some of them also satisfy the definitions of several of the terms described above.

The examples of warning signs are grouped by the industry they were used to promote policies for. Each example also includes the date the warning sign was made known to the public, the location where the warning sign took place, and a brief description of what occurred. The relevance of the warning sign to the problem activists were trying to address is categorized as High, Moderate, or Low. The amount of harm caused is categorized as harm occurring locally, regionally, or globally, and as catastrophic if it kills a majority of the people it impacts or renders a large area uninhabitable, significant if it kills some people or has severe environmental consequences, and low if no one is killed by the warning shot. The resulting policies that this warning sign contributed to are then briefly described.

Nuclear Power

Three Mile Island

  • Date: 1979.
  • Location: Pennsylvania, USA.
  • Description: Partial meltdown at a nuclear power plant caused the release of some radioactive gas and iodine.
  • Relevance: High.
  • Harm Caused: Too small to measure.
  • Result: Reinforced already increasing safety regulations. The Nuclear Regulatory Commission was founded in 1975, and many planned nuclear power plants had already been canceled. No new nuclear power plants would begin construction in the USA for decades.6)

Chernobyl

  • Date: 1986.
  • Location: Ukraine, USSR.
  • Description: Steam explosion, meltdown, and reactor core fire destroyed the containment building and spread radioactive airborne contaminants across Europe. About 30 people died within 3 months, and the WHO estimates about 9,000 excess cancer deaths afterwards, although this is disputed.7)
  • Relevance: High.
  • Harm Caused: Catastrophic locally, significant regionally.
  • Result: Public opinion in Europe shifts dramatically against nuclear power. Italian referendum results in the closure of all nuclear power plants in Italy.8) The USSR slows nuclear power plant construction and commits to increased transparency around nuclear power.9) The IAEA creates two new treaties: the Convention on Early Notification of a Nuclear Accident and the Convention on Assistance in the Case of a Nuclear Accident or Radiological Emergency.

Fukushima

  • Date: 2011.
  • Location: Japan.
  • Description: Major earthquake and tsunami damaged the electrical grid and the power plant’s backup power, removing the ability to cool the reactors, leading to hydrogen explosions and the release of radioactive contamination into the environment. Linear no-threshold models estimate 130 excess cancer deaths, although this is likely below detectable levels.10)
  • Relevance: High.
  • Harm Caused: Significant locally.
  • Result: Japan shuts down all of its nuclear reactors, and later reopens some.11) Germany decides to shut down all of its nuclear reactors.12)

Fossil Fuels

Exxon Valdez Oil Spill

  • Date: 1989.
  • Location: Alaska, USA.
  • Description: An oil tanker struck a reef and spilled 10 million gallons of crude oil into Prince William Sound, causing significant environmental damage and affecting wildlife populations for decades.
  • Relevance: High.
  • Harm Caused: Significant locally.
  • Result: The Oil Pollution Act of 1990 phased in double hulls for oil tankers and banned some tankers from entering Prince William Sound. Rising public concern about climate change and the first IPCC report occur soon after, but seem to be mostly unrelated.13)

Extreme Weather Events

  • Date: Various.
  • Location: Various.
  • Description: Some extreme weather events are more likely to occur as a result of anthropogenic climate change.14) Activists use individual weather events to promote policies designed to reduce climate change.
  • Relevance: Moderate.
  • Harm Caused: Significant to catastrophic locally.
  • Result: There is some evidence that people become more concerned about climate change in response to an extreme weather event, although the evidence is mixed.15) The impact of extreme weather events on climate policy is even less clear.16)

Chlorofluorocarbons (CFCs)

Discovery of the Ozone Hole

  • Date: 1985.
  • Location: Southern Hemisphere.
  • Description: CFCs in the stratosphere deplete ozone, especially around Antarctica, allowing more UV light to reach the surface, which causes skin cancers.17) This had been predicted and one location in Antarctica had measured declining ozone. Aircraft measurements showed that this was widespread and associated with CFCs.
  • Relevance: High.
  • Harm Caused: Significant regionally.18)
  • Result: Vienna Convention of 1985 and Montreal Protocol of 1987, a global treaty which created a phase-out management plan for CFCs.19)

Genetically Modified Organisms (GMOs)

Mad Cow Disease

  • Date: 1995.
  • Location: England, UK.
  • Description: Prion disease which spread as cattle were fed meat-and-bone meal, in cattle since 1985 and in humans since 1995.20)
  • Relevance: Low.
  • Harm Caused: Significant regionally.
  • Result: Opposition to GMOs rose by double digits from 1996-1999 in many European countries. Corporate campaigns convinced supermarkets to remove GMOs. National GMO bans led to a de facto GMO moratorium in the EU, which persists today.21)

Cloning of Dolly the Sheep

  • Date: 1997.
  • Location: Scotland, UK.
  • Description: Dolly was the first mammal cloned from an adult cell.22)
  • Relevance: Low.
  • Harm Caused: None.
  • Result: Opposition to GMOs rose by double digits from 1996-1999 in many European countries. Corporate campaigns convinced supermarkets to remove GMOs. National GMO bans led to a de facto GMO moratorium in the EU, which persists today.23)

Medical Research Involving Viruses

COVID-19 Lab Leak Hypothesis

  • Date: 2019.
  • Location: Wuhan, China.
  • Description: A new respiratory virus caused a global pandemic, killing millions of people. The virus might have originated from the Wuhan Institute of Virology.
  • Relevance: Unclear.
  • Harm Caused: Significant globally.
  • Result: Some efforts to ban gain-of-function research, but no significant policy changes.24)

Medical Research Involving Human Subjects

Jewish Chronic Disease Study

  • Date: 1964.
  • Location: New York, USA.
  • Description: Elderly patients were injected with live cancer cells without their consent to study how and whether their body would reject them.
  • Relevance: High.
  • Harm Caused: Low to significant locally.
  • Result: Congressional hearings and lawsuits, although the NIH managed to avoid responsibility. Clinical Research Committees, like the one at the NIH's hospital in Bethesda, Maryland, were created at more research hospitals.25)

Willowbrook Hepatitis Study

  • Date: 1971.
  • Location: New York, USA.
  • Description: Mentally ill children were deliberately infected with hepatitis to study how hepatitis spread at their school.26)
  • Relevance: High.
  • Harm Caused: Low to significant locally.
  • Result: National Research Act of 1974, leading to the Belmont Report of 1978 and the creation of Institutional Review Boards for all research involving human subjects at institutions which receive funding from the federal government.27)

Tuskegee Syphilis Study

  • Date: 1972.
  • Location: Alabama, USA.
  • Description: Hundreds of black men with syphilis were not told they had the disease for decades and were not given treatment, even after it became available, to “determine through autopsies what damage untreated syphilis does to the human body.”28)
  • Relevance: High.
  • Harm Caused: Significant locally.
  • Result: National Research Act of 1974, leading to the Belmont Report of 1978 and the creation of Institutional Review Boards for all research involving human subjects at institutions which receive funding from the federal government.29)

Primary author: Jeffrey Heninger.

1) , 3)
Warning shot. EA Forum. (Accessed November 16, 2023.) https://forum.effectivealtruism.org/topics/warning-shot.
2)
Eliezer Yudkowsky. There's No Fire Alarm for Artificial General Intelligence. MIRI. (2017) https://intelligence.org/2017/10/13/fire-alarm/.
4)
Mass Protest. Ayni Institute. (Accessed November 16, 2023.) https://ayni.institute/massprotest/.
5)
Remy Melina. What Exactly Is a 'Sputnik Moment?' Space.com. (2010) https://www.space.com/10437-sputnik-moment.html].
6)
Hultman and Koomey. Three Mile Island: The driver of US nuclear power’s decline? Bulletin of the Atomic Scientists 69.3. (2013) p. 63-70. https://journals.sagepub.com/doi/pdf/10.1177/0096340213485949.
7)
World Health Organization report explains the health impacts of the world's worst-ever civil nuclear accident. WHO. (2006) https://web.archive.org/web/20110404181327/http://www.who.int/mediacentre/news/releases/2006/pr20/en/index.html.
8)
1987 Italian referendum. Wikipedia. (Accessed November 16, 2023.) https://en.wikipedia.org/wiki/1987_Italian_referendums.
Eligendo Archivio. Ministero dell’Interno. (Accessed November 16, 2023.) https://elezionistorico.interno.gov.it/index.php?tpel=F&dtel=08/11/1987&tpa=I&tpe=A&lev0=0&levsut0=0&es0=S&ms=S.
9)
Vyacheslav S. Romanov. Nuclear power and public opinion. IAEA Bulletin 2. (1990) https://www.iaea.org/sites/default/files/publications/magazines/bulletin/bull32-2/32205091922.pdf.
10)
Ten Hoeve and Jacobson. Worldwide health effects of the Fukushima Daiichi nuclear accident. Energy & Environmental Science. (2012) https://web.stanford.edu/group/efmh/jacobson/TenHoeveEES12.pdf.
11)
In a reversal, Japan embraces nuclear power after promising to phase it out. NPR. (2022) https://www.npr.org/2022/12/22/1144990722/japan-nuclear-power-change-fukushima.
12)
Eddy and Solomon. Germany Quits Nuclear Power, Ending a Decades-Long Struggle. New York Times. (2023) https://www.nytimes.com/2023/04/14/world/europe/germany-nuclear-power-plants.html.
13)
Nina Joung. The 1989 Exxon Valdez oil spill, and how we see climate change then and now. PBS. (2019) https://www.pbs.org/wnet/peril-and-promise/2019/03/thirty-years-exxon-valdez/.
14)
IPCC Sixth Assessment Report. Ch. 11: Weather and Climate Extreme Events in a Changing Climate. (2021) https://www.ipcc.ch/report/ar6/wg1/chapter/chapter-11/.
15)
Bergquist, Nilsson, and Schultz. Experiencing a Severe Weather Event Increases Concern About Climate Change. Frontiers in Psychology 10. (2019) https://www.frontiersin.org/articles/10.3389/fpsyg.2019.00220/full.
16)
Sam Rowen. Extreme weather and climate policy. Environmental Politics 32. (2022) p. 684-707. https://www.tandfonline.com/doi/full/10.1080/09644016.2022.2127478.
17)
Ozone hole facts. NASA Ozone Watch. (Accessed November 16, 2023) https://ozonewatch.gsfc.nasa.gov/facts/hole_SH.html.
18)
“A comprehensive report by United Nations Environment Programme estimated an additional burden of 4500 melanoma cases and 300,000 non-melanoma cases if there is a 10% decrease in the ozone layer.”
Umar & Tasduq. Ozone Layer Depletion and Emerging Public Health Concerns. Frontiers in Oncology 12. (2022) https://www.frontiersin.org/journals/oncology/articles/10.3389/fonc.2022.866733/full.
19)
Montreal Protocol on Substances that Deplete the Ozone Layer (with annex). (1987) https://treaties.un.org/doc/publication/unts/volume%201522/volume-1522-i-26369-english.pdf.
20)
John Pattison. The emergence of bovine spongiform encephalopathy and related diseases. Emerging Infectious Diseases 4.3. (1998) p. 390-394. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2640268/pdf/9716952.pdf.
22)
The Life of Dolly. The Roslin Institute. (Accessed November 16, 2023) https://dolly.roslin.ed.ac.uk/facts/the-life-of-dolly/index.html.
24)
Nate Soares. Warning Shots Probably Won’t Change The Picture Much. LessWrong. (2022) https://www.lesswrong.com/posts/idipkijjz5PoxAwju/warning-shots-probably-wouldn-t-change-the-picture-much.
25)
Laura Stark. Protections for Human Subjects in Research: Old Models, New Needs? MIT Case Studies in Social and Ethical Responsibilities of Computing. (2022) https://mit-serc.pubpub.org/pub/protections-for-human-subjects/release/1.
Citing: Laura Stark. Behind Closed Doors: IRBs and the Making of Ethical Research. (Chicago, 2012)
27) , 29)
The National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research. (1978) https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/read-the-belmont-report/index.html.
28)
Jean Heller. AP WAS THERE: Black men untreated in Tuskegee Syphilis Study. AP News. (2017) https://apnews.com/article/business-science-health-race-and-ethnicity-syphilis-e9dd07eaa4e74052878a68132cd3803a.
responses_to_ai/examples_of_warning_signs.txt · Last modified: 2023/12/19 22:10 by jeffreyheninger