Course:ELEC300E/CaseStudies/OpenAIPrivacy

OpenAI Privacy
Image:wiki.png
Content / Topics
	Environmental Impacts
	Repair
	Right to Repair
	Social Impacts of Computing
Relevant Analysis Strategies or Tools
	Exploring Personal Biases & Positionality
	Decision Impact Assessment
	Iceberg Model
	Systems Mapping

Note: This case study touches on troubling themes - including school shootings and suicides. Please only participate in ways you are comfortable with, extend sensitivity to your peers and reach out to your instructors if you need additional support.

Context: AI Privacy

Defining some terms...

These three interrelated terms form a continuum from the universal (ethics) to the specific (law);

Ethics: Generalized principles about how people ought to act. It seeks moral truths about right and wrong.
Morals: Personal, cultural or societal beliefs and practices about what is right or wrong.
Law: Specific codified community held beliefs about what is acceptable or unacceptable.

Ethical AI

UNESCO released a report on Ethical AI that outlines many recommendations for ethical development of AI systems. It is a long document, with many recommendations, but some principles that are interesting in the context of this case study include:

Do No Harm: "It should be recognized that AI technologies do not necessarily, per se, ensure human and environmental and ecosystem flourishing. Furthermore, none of the processes related to the AI system life cycle shall exceed what is necessary to achieve legitimate aims or objectives and should be appropriate to the context. In the event of possible occurrence of any harm to human beings, human rights and fundamental freedoms, communities and society at large or the environment and ecosystems, the implementation of procedures for risk assessment and the adoption of measures in order to preclude the occurrence of such harm should be ensured."
Privacy: "Privacy, a right essential to the protection of human dignity, human autonomy and human agency, must be respected, protected and promoted throughout the life cycle of AI systems. It is important that data for AI systems be collected, used, shared, archived and deleted in ways that are consistent with international law and in line with the values and principles set forth in this Recommendation, while respecting relevant national, regional and international legal frameworks."
Human oversight: "Member States should ensure that it is always possible to attribute ethical and legal responsibility for any stage of the life cycle of AI systems, as well as in cases of remedy related to AI systems, to physical persons or to existing legal entities. Human oversight refers thus not only to individual human oversight, but to inclusive public oversight, as appropriate."
Responsibility and Accountability: "Appropriate oversight, impact assessment, audit and due diligence mechanisms, including whistle-blowers’ protection, should be developed to ensure accountability for AI systems and their impact throughout their life cycle. Both technical and institutional designs should ensure auditability and traceability of (the working of ) AI systems in particular to address any conflicts with human rights norms and standards and threats to environmental and ecosystem well-being."

But how are AI companies implementing these principles in practice?

Consider OpenAI's Charter and approach to "safety". Eg:

OpenAI’s mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work—benefits all of humanity. We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome. To that end, we commit to the following principles: Broadly Distributed Benefits, Long-Term Safety, Technical Leadership, Cooperative Orientation.

Case: Tumbler Ridge and OpenAI's decision

There have been many accidents, incidents and cases of unexpected negative outcomes associated with AI. The AI Incident Database tracks and categorizes such 'incidents'. One particularly troubling and well publicized case concerns a school shooting in Tumbler Ridge, BC. The AI Incident Database describes the incident as follows:

Description: After the 02/10/2026 school shooting in Tumbler Ridge, British Columbia, OpenAI said a user later identified as the suspect had previously used ChatGPT to describe scenarios involving gun violence. Those chats were reportedly auto-flagged and reviewed and the account was banned, but OpenAI says it did not alert the RCMP because it allegedly did not present a credible, imminent threat; the company reportedly contacted police after the attack. Editor Notes: Included for the reported AI-mediated flagging and escalation process. Available reporting suggests the suspect's ChatGPT interactions were flagged by automated systems and reviewed by personnel, and that the associated account was later suspended. The same reporting indicates OpenAI did not notify law enforcement at the time, citing its internal threshold for a credible, imminent risk of serious harm.

Timeline of events:

⎯ June 2025: OpenAI Flags and bans the shooter’s account for “misuses of our models in furtherance of violent activities”

⎯ Mid 2025: OpenAI staff debate contacting the police, and urge leadership to contact the Canadian police, but choose not to do so because content did not meet OpenAI’s threshold.

⎯ Feb 10th, 2026: Tumbler Ridge school shooting

⎯ Early Feb, 2026: OpenAI contacts the police about the shooter’s account content

⎯ Mid Feb, 2026: Public disclosure of OpenAI’s decision to not inform the police

⎯ Feb 24th, 2026: Canadian government discusses legal requirement for companies to inform police when online content indicates the potential for harm. Currenty the law says companies ‘may’ disclose to police.

Discussion Questions

Here are a few questions to help you begin analyzing the issues discussed in this case study. The purpose of these questions is to prompt reflection and further consideration: there are no right or wrong answers!

Personal response: Take stock of your immediate response to this case study: outrage, discomfort, confusion, frustration? Take a moment to acknowledge these feelings.
Contradictory viewpoints and outcomes: What were the values expressed on OpenAI's website? Rank how these values were expressed in OpenAI’s decision not to inform the police about the shooter’s suspended account. Were some values in conflict?
Social and Historical Context: This is not the first school shooting in Canada. In 1989, 14 female engineering students were killed at Montreal's Ecole Polytechnique. Many engineering schools commemorate the victims of this event every year - does yours? How can the victims of these events be remembered and respected?
Positive action: Do you think OpenAI should make any changes to its policies, services or public charter in response to this case study? What changes, and why? How would you frame the problem with the existing policies and systems?
Interruptions and invitations: If a person were to contribute to, or fail to report violent activities, they might be held criminally responsible and face jail time. Who should be held criminally accountable if an AI agent commits a crime and what should be consequences be? In Canadian law, an organization may be held criminally responsible for the actions of an AI agent, but few court cases have tested this, and there is ongoing research and discussion on criminal liability when it comes to AI agents.
Positive practices: Has this case study changed your understanding of AI Ethics, or helped you to clarify your personal values in this context? What can you do personally to put your values into action, or have your opinion heard?
Physical connections: This is a difficult topic. Before moving on, check in with your personal stress levels and take care of yourself. Consider a brief walk outside on the way to your next class, or a mindfulness activity(even just a few deep breaths) to help restore calm.