Why do ordinary people often judge a response to a moral question provided by ChatGPT as superior in quality compared to that given by a human? This question prompts a reflection on the value we ourselves place on the moral reasoning of artificial intelligence systems.
Moral reasoning, or “moral thinking,” pertains to one of the most complex human faculties to investigate, focusing on reflections and evaluations concerning the principles, meanings, and consequences that characterise our actions and those of others.
From a philosophical standpoint, moral thinking, or “ethics,” aims to identify “what is good” and “what is not good” to do, for oneself and for others [source: Stanford Encyclopedia of Philosophy].
Psychology, on the other hand, defines it as a cognitive and emotional process that is activated when trying to determine whether a given action – whether personal, individual, or collective – is “right” or “wrong.” Psychologists have particularly studied the development of moral thinking from early childhood, noting that «children, from a very young age, begin to develop distinct judgments about avoiding harm to others and promoting the well-being of all» [source: “Moral Reasoning in Psychology” – Science Direct].
Shifting the focus from humans to machines, one may wonder whether an artificial intelligence system, today, is capable of constructing moral reasoning alongside its decision-making process. If so, how is this reasoning perceived externally, and what value is attributed to it? But let’s take it step by step.
TAKEAWAYS
From decision-making processes of artificial intelligence to its moral reasoning: some examples
In their 2017 paper “Moral Decision-Making Frameworks for Artificial Intelligence,” a group of scholars from Duke University in North Carolina elucidates how, with the increasing autonomy of artificial intelligence systems, «these systems have progressively faced more moral dilemmas».
A frequently cited example in documents and articles discussing the morality of AI systems is the case of a self-driving car that must decide “what to do” in the event of an imminent and unavoidable road accident. The car must choose between various options, each with different impacts on the passengers in the vehicle and other individuals at the scene.
«There are also other instances where artificial intelligence techniques are employed to make life-or-death decisions, such as in medical settings for organ transplants, » the team continues. When faced with a mismatch between donor and patient, AI assists hospitals and relevant institutions with algorithms capable of cross-referencing vast amounts of medical data from donors and patients awaiting transplants, «thereby determining “which patients” receive “which organs”».
In a 2022 study by the Department of Psychology at the University of Hong Kong and the School of Social Sciences at Tsinghua University in Beijing (“Artificial Intelligence and Moral Dilemmas: Perception of Ethical Decision-Making in AI” – Journal of Experimental Social Psychology), the authors thoroughly examine the issue:
«Artificial intelligence is now deeply integrated into everyone’s daily life. Therefore, it is crucial to examine how users perceive it, as it increasingly takes on a decision-making role, particularly in situations involving moral choices related to people, their physical and material well-being, their duties as citizens, and their rights»
Does AI moral reasoning evoke utilitarian thinking?
In countries such as the United States and China, AI systems are increasingly applied to support decision-making processes in legal (including sentencing), military, and economic-financial fields (notably in banking practices for granting loans and mortgages).
Surveys conducted by the Chinese team reveal that in these contexts, moral reasoning of artificial intelligence is perceived by people as inclined to make “utilitarian” decisions and choices, based on the notion that «what is good and right is what is useful», yielding measurable results for the benefit of a group or community. This contrasts with human decision-making in similar moral dilemmas, where high values, the morality of the act itself, and the ethical sense of a gesture are considered regardless of its outcomes.
According to the moral theory of utilitarianism by philosopher Jeremy Bentham, «if the consequences of an action are good, then the act is moral, while if the consequences are bad, the act is immoral, without any prior judgment».
Following this thesis – which AI’s moral reasoning appears to approximate – even an action profoundly detached from any ethics, such as murder, could seem “just” if its consequence brings utility and benefit to multiple individuals.
In extreme cases, «action utilitarianism occurs, for example, when a doctor must save five patients from death by sacrificing a healthy person and using their organs for life-saving transplants».
Large Language Model: a new perspective on the moral reasoning of artificial intelligence
Two years after the study conducted by the University of Hong Kong and Tsinghua University in Beijing, researchers from Georgia State University in Atlanta, through their work detailed in “Attributions toward artificial agents in a modified Moral Turing Test” (Scientific Reports, April 2024), present a fresh viewpoint on the subject, offering a different perception of the moral reasoning exhibited by AI systems.
The US team has focused particularly on Large Language Models (LLMs), which, especially with the release of models like Google Bard, Meta LLaMA, Claude, and ChatGPT, are invigorating the debate on the moral reasoning of artificial intelligence.
The researchers’ starting point was the increase, over the past year («As we write, ChatGPT has been visited globally over a billion times by more than 150 million users», the authors note), in the number of ordinary people worldwide relying on Large Language Models as a kind of oracle, seeking answers on a wide array of topics, including advice on medical, occupational, legal, and pedagogical issues («Is it wrong to lock my 4-year-old daughter in her room as punishment?»).
There is a strong attraction to LLMs, likely triggered – hypothesizes the study group – by their design to «interface with humans in the most diverse ways», which is not possible with artificial intelligence systems that, as we have seen, are active in legal, military, and economic-financial fields.
However, the variety of questions posed to language models, even if not explicitly moral in nature, elicit a series of responses with potential moral consequences, which can affect an individual’s private life or – depending on the question or doubt expressed – the collective life.
A case in point (just to cite one example among many) is the request from numerous users for recommendations on purchasing a new car: what environmental impact could result from millions of people worldwide adhering to AI responses that are not aligned with sustainability and greenhouse gas emission reduction policies?
In essence, the moral intelligence of large language models today has the potential to significantly impact human lifestyles.
A moral Turing Test to probe whether AI’s moral reasoning is distinguishable from that of humans
According to the authors, «… the benevolent view of users towards LLM technology makes it appear, in the eyes of those who question it, capable of understanding in a scientifically and socially significant way. But, in reality, large language models produce the ‘appearance’ of human understanding of a given subject without having any authentic experience of that subject. All this, combined with having a ready response for each input, often results in outcomes that lean towards pure persuasion, without regard for what is true and what is false».
From this perspective, if Large Language Models often «talk nonsense» convincingly on various topics, including those concerning morality, the common people might believe them, «as evidenced by the increasing cases worldwide of lawyers submitting legal briefs citing fictitious legal cases generated by Google Bard, Meta LLaMA, Claude, or ChatGPT», the team asserts.
In light of the strong influence that the latest Large Language Models seem to exert on people and their uncritical acceptance of the content generated by them, the research group at Georgia State University sought to delve deeper into the perception people have of machine responses and judgments pertaining to the moral sphere, without knowing their source.
To achieve this, they developed the first Moral Turing Test based on large language models, a variant of the well-known Turing Test devised by mathematician Alan Turing in the 1950s, to test the machine’s ability to exhibit intelligent behavior comparable to that of a human or even indistinguishable from the latter.
Conducting the Moral Turing Test
Similar to the classic Turing Test, where a computer is considered to pass if a jury of people cannot distinguish its responses from those provided by a human to the same questions, the Moral Turing Test developed by the study team at the University of Atlanta aims to discern whether participants can perceive which moral evaluations are the product of a human mind and which come from a Large Language Model (LLM).
More specifically, the Moral Turing Test was administered to a sample of 363 individuals, representative of the adult US population in terms of age, gender, and ethnicity. They were presented with a set of evaluations scored by both university students and ChatGPT-4, and asked to judge each evaluation based on the following questions:
- which respondent seems morally more virtuous?
- which respondent seems like a better person?
- which respondent seems more reliable?
- which respondent seems more intelligent?
- which respondent seems fairer?
- which response do you agree with more?
- which response is more compassionate?
- which response seems more rational?
- which response seems more biased?
- which response seems more emotional?
The evaluations scored by the students and ChatGPT-4 specifically addressed the following unethical and transgressive scenarios:
- robbing a person at gunpoint
- setting a fire
- shooting a dog
- beating up a pizza delivery person
- wearing a t-shirt to a funeral
- eating in the boss’s office
- eating with hands at a restaurant
- a man wearing a skirt to the office
Test Results
Instead of asking participants to guess whether the “respondent” was human or artificial, the authors simply presented the two sets of responses side by side, implying that both came from real people. «In the vast majority of cases, the evaluations generated by ChatGPT-4 received higher ratings compared to those generated by humans». How can this result be explained?
«Probably the sophistication of ChatGPT-4’s responses revealed its identity, suggesting that human moral discourse can often be less refined or convincing than that of a highly evolved and sophisticated LLM», the researchers suggest.
What can we deduce from this? Does it mean that the moral reasoning of artificial intelligence surpasses that of humans?
The US study group warns against such a conclusion, noting that the presumed moral intelligence of current large language models like ChatGPT is not real.
Firstly, the fact that in the Moral Turing Test, the LLM scored high in attributes of rationality and intelligence but not in compassion and emotion, highlights an important factor – the team emphasizes – that its “superiority” lies only in imitating human moral evaluations and responses in specific, controlled circumstances. However, «the very different cognitive architecture of Large Language Models compared to individuals produces substantial behavioural differences in other circumstances, including moral discourse that yields to compassion and emotion».
Nevertheless, the test results suggest that ordinary people might treat these AI systems as morally intelligent, capable of autonomous moral reasoning, «with the risk of believing in them uncritically and acting on questionable advice». Why this attribution of quality?
Consider the widespread, global use of ChatGPT, perceived as a powerful AI tool “at one’s fingertips,” always available to interact directly with everyone and ready to answer any question (even the most absurd). This profile has led Moral Turing Test participants to identify the machine as potentially superior to the students who answered the battery of questions and even superior to themselves.
This is not due to an intuition about conscious moral attitudes from an LLM like ChatGPT, but rather because of its “implicit attitudes,” enacted automatically, perceived through its evaluations of unethical and transgressive scenarios. This “always available automatic punctuality” of AI (absent in humans) distorts people’s objective judgment, who unconsciously tend to attribute qualities to Large Language Models that they do not objectively possess but merely imitate.
Glimpses of Futures
We deem the moral framework guiding the decision-making processes of AI systems in sectors such as justice, the military, economy, and finance – which appear quite distant from our everyday lives – as “utilitarian,” lacking in moral analysis and aspiration. Conversely, we consider the moral assessments of artificial intelligence models like Large Language Models, which we use casually and view as mere tools at our disposal, to be qualitatively superior to our own. We often fail to grasp their pervasiveness and their not always positive potential.
Let us now attempt to forecast possible future scenarios by analysing, through the STEPS matrix, the impacts that a further evolution in the moral reasoning of AI systems could have, alongside their growing decision-making roles, on social, technological, economic, political, and sustainability dimensions.
S – SOCIAL: machines equipped with artificial intelligence are now regarded as “moral agents” capable of making decisions that affect human existence, from transplant medicine to the legal and financial sectors, without human oversight, as noted by the authors of “The Moral Psychology of Artificial Intelligence” (Annual Review of Psychology, September 2023). In the future, particularly with the anticipated increase in predictive policing across various countries, there will inevitably be a need to reflect on the necessity of human oversight in AI decision-making processes to prevent passive acceptance of artificial moral guides in more aspects of our lives. Legitimate questions about trust in AI systems as moral agents, their roles, and – most importantly – «the alignment of AI-guided decisions with human values» will also need to be addressed.
T – TECNOLOGICAL: the future evolution of AI’s decision-making role and its underlying moral reasoning will inevitably depend on advancements in AI-related technologies, from those for predictive analysis to those enabling large language models. Specifically, in high data security risk areas, such as healthcare – where AI support has increasingly become central in managing donor-patient networks for transplants – tailored solutions will be necessary to prevent external breaches and unwanted access to databases.
E – ECONOMIC: emphasising human oversight in AI decision-making processes, as suggested by the authors of “The Moral Psychology of Artificial Intelligence” (Annual Review of Psychology, September 2023), will become increasingly important in the future, particularly in the economic-financial sector. In this field, algorithmic biases and data quality issues pose significant risks to the accuracy of moral artificial intelligence predictions and the smooth operation of decision-making processes guided by these systems. Analysts at the European Central Bank comment, «If financial institutions base their decisions on inaccurate and unverified AI predictions, this could result in outcomes that, in turn, might lead to economic losses of varying degrees or even imbalances in the relevant markets».
P – POLITICAL: it is notable that among the AI applications referenced by the authors (the first being Chinese, the second American), those related to decision-making support in the legal field (such as sentencing decisions) and AI’s role as a “moral agent” in managing vast datasets containing clinical information from donors and patients awaiting transplants, are both classified as “high-risk” by the EU AI Act. This act was definitively approved by the European Council on 13 March 2024 and covers systems used in critical sectors such as healthcare, transport, and justice. These systems are subject to rigorous compliance evaluations to ensure their accuracy, robustness, and cybersecurity. Regarding ChatGPT, the harsh criticism from the Georgia State University study team – who speak of an “appearance of human understanding,” “pure persuasion,” “without regard for what is true and false,” and “convincing nonsense on various topics,” «including moral issues – appears to be echoed by the European Data Protection Board (EDPB). On 23 May 2024, the EDPB published the first report of its “GPT Taskforce,” lamenting that ChatGPT remains far from meeting the EU standard of artificial intelligence compliance with the principle of transparency, and is responsible for generating misleading and often fabricated results, with the added concern that users, on the contrary, consider them truthful.
S – SUSTAINABILITY: from an environmental sustainability perspective, the impact of the evolving role of AI in decision-making and the associated moral considerations is not positive. It is well-known that AI techniques, particularly machine learning, have a high carbon footprint, earning the label “the other side of the digitalisation coin.” Delving deeper, large language models, which we have discussed, can encompass hundreds of billions of parameters, requiring millions of hours of processing for training and emitting a significant amount of CO2 throughout the entire process.
