Trust and uncertainty, creativity, and defence [TWIE]

The Week In Examples #34 | 4 May 2024

DALLE-3

A powerful model with the nostalgia inducing name “gpt2-chatbot” appeared and disappeared on the LMSYS leaderboard, Sam Altman thinks ChatGPT is still “incredibly dumb” (even as it gets memory), and Anthropic released a Claude iOS app (not in the EU, though). These are just some of things I won’t be writing about this week. 

Instead, for the thirty-fourth edition of The Week In Examples, I’m looking at studies focused on the impact of qualifiers and caveats on user trust, the creative process in large models, and the role of AI in the future of defence. As usual, it’s hp464@cam.ac.uk for feedback, comments or anything else. 

Three things  

1. Trust me, I’m a large language model 

• What happened? Researchers from Microsoft and Princeton University conducted a study to determine the effect of LLMs’ expressions of uncertainty (e.g. I’m not sure, but…) on user trust. In an experiment in which 400+ users were asked to provide answers to medical questions with the help of a language model, the group found that qualifying expressions decreased confidence in the system and reduced the extent to which users agreed with the answers produced. Crucially, they also found that expressions of uncertainty increased participants’ accuracy, which in this context refers to how well individuals assessed the factuality of LLM answers about different medical conditions. 

• What's interesting? The ultimate problem that the group is trying to tackle is overreliance whereby people uncritically accept an incorrect suggestion produced by a language model, like in 2023 when a lawyer included erroneous legal opinions generated by ChatGPT in a legal brief presented in court. Other groups have explored countermeasures in the past—mostly connected to efforts to better explain model outputs—but many of these solutions have been found to be ineffective at best and counterproductive at worst.      

• What else? In what they described as “exploratory analysis”, the researchers say that the increase in accuracy can be attributed to reduced overreliance on incorrect answers. That being said, the group cautions against buying into their findings with too much enthusiasm. This is because, even when the AI expressed uncertainty, participants who were shown the AI responses still performed worse on the question-answering task compared to participants in the control group who were not shown any AI responses at all. In other words: while expressing uncertainty did help reduce overreliance to some extent compared to expressing no uncertainty at all, it didn’t eliminate the problem entirely. 

2. LLMs are few-shot creatives

• What happened? A study from the Max Planck Institute for Biological Cybernetics and the University of Amsterdam takes a look at creativity in large language models, which it defines as the originality of responses generated in thinking tasks. The researchers created a framework to delineate between two types of responses to prompts representative of types of creativity: those that are persistent and those that are flexible. Here, persistent refers to deeper search within limited conceptual spaces (like using a brick to break a window or crack a nut) and flexible describes search across multiple conceptual spaces and the ability to “jump” between them (like using a brick to exercise or build). Using this method, they found both human and LLM approaches ranged from persistent to flexible. However, while human creativity was not correlated with flexibility, more flexible LLMs had higher originality scores.

• What's interesting? Core to this work is the distinction between the creative product (the thing that has been created) and the creative process (the specific procedure required for the creative activity to take place). For AI, the authors say, the majority of existing work focuses on the creative product (e.g. language models producing creative writing) rather than the underlying process. This work aims to address the latter. It suggests that, unlike humans, LLMs' creativity is correlated with flexibility in the idea generation process. More flexible LLMs—that is, those that explored a broader range of conceptual spaces—tended to generate more original responses.    

• What else? The work reminds me of AlphaGo’s famous Move 37 in which the model played a move that commentators described as "creative" in its 2016 victory against top Go player Lee Sedol. Later work studying Move 37 proposed that, while lacking in some aspects of creativity, the model is capable of what cognitive scientist Margaret Boden calls “transformational creativity”. According to Boden, transformational creativity isn’t just about combining old things in new ways or originating something new within an existing conceptual space, but instead involves transforming the space in which a problem is conceived. AlphaGo, of course, is a highly specialised model built on a very different architecture to contemporary LLMs, but I make the connection in service of a broader point. The last few years have already encouraged us to challenge preconceived notions of what AI creativity is, how it works, and what limits we ought to attach to it. I don’t expect that to change any time soon.    

3. US military proposes multi-model combat doctrine  

• What happened? This week’s final study is from researchers at the U.S. Marine Corps Warfighting Laboratory and U.S. government contractor ASRC Federal. The authors advocate for what they describe as a doctrine of “complementation” in which AI works with soldiers rather than the process of “substitution” by which a human waits to take over if the AI behaves incorrectly. In this model, sometimes called a “human-in-the-loop” or “human-on-the-loop” setup, a single AI model is deployed to control a weapon system (e.g. a swarm of drones). In contrast, the "complementation" approach involves multiple AI models running concurrently. These models are managed by a team of human operators, who actively select and deploy the most appropriate model for each situation. In this approach, there is also an intermediary layer of “AI operators” serving as a backup in case the human operators are unable to perform their duties (if this sounds a bit complex, there is a useful diagram on page eight of the report to help clarify the distinction).

• What's interesting? The writers argue that complementation has four strengths versus substitution. These are 1) that teams of human operators working with AI can provide perspectives and problem-solving abilities that a single AI cannot replicate; 2) that an ensemble of different AI models managed by human operators is more adaptable and fault-tolerant than relying on a single AI model; 3) that human operators can dynamically swap in different AI models suited for specific circumstances rather than being locked into one AI configuration; and 4) that human involvement in the complementation approach provides more accountability and alignment with human values compared to an AI operating autonomously. 

• What else? The complementation doctrine is an interesting one. While aspects of the approach look to be more robust than deploying isolated models with a single human overseer, it isn’t exactly free from ethical problems. The AI operator acting as a proxy for a human is still an AI. If a human can’t make a decision, then we’re in a situation in which one model is making a decision for another. Of course, it might not necessarily be either or. We could feasibly incorporate multiple models, multiple human handlers, and multiple AI proxies and insist on a human in the loop for important decisions. But these are, of course, political and moral questions—not technical ones. 

Best of the rest

Friday 3 May 

• Reimagining secure infrastructure for advanced AI (OpenAI)

• Issue Brief: Measuring Training Compute (Frontier Model Forum)

• How the computer games industry is embracing AI (BBC)

• Does the PR industry have an AI problem? (London Evening Standard) 

• Apple aims to tell an AI story without AI bills (Reuters) 

• Some scientists can't stop using AI to write research papers (The Register)

Thursday 2 May

• Nick Bostrom Made the World Fear AI. Now He Asks: What if It Fixes Everything? (WIRED) 

• Let's get real about Britain's AI status (Substack)

• Not a Swiss Army Knife: Academics' Perceptions of Trade-Offs Around Generative Artificial Intelligence Use (arXiv) 

• Goal-conditioned reinforcement learning for ultrasound navigation guidance (arXiv)

• Rabbit R1 review: an unfinished, unhelpful AI gadget (The Verge) 

• The AI safety fog of war (POLITICO) 

• Qualia and the Formal Structure of Meaning (arXiv)

• FLAME: Factuality-Aware Alignment for Large Language Models (arXiv) 

Wednesday 1 May 

• A Careful Examination of Large Language Model Performance on Grade School Arithmetic (arXiv) 

• AI Safety Newsletter #34: New Military AI Systems (Substack)

• How to govern AI — even if it's hard to predict (TED)

• AI Chatbots Have Thoroughly Infiltrated Scientific Publishing (Scientific American) 

• Capabilities of Gemini Models in Medicine (arXiv) 

• Self-Play Preference Optimization for Language Model Alignment (arXiv)

• Regulators’ strategic approaches to AI (UK Gov) 

• The Real, the Better: Aligning Large Language Models with Online Human Behaviors (arXiv) 

• Is Temperature the Creativity Parameter of Large Language Models? (arXiv) 

Tuesday 30 April 

• AI leaderboards are no longer useful. It's time to switch to Pareto curves (Substack) 

• The power of persuasion: Google DeepMind researchers explore why gen AI can be so manipulative (The Verge)

• Evaluating Lexicon Incorporation for Depression Symptom Estimation (arXiv) 

• Major U.S. newspapers sue OpenAI, Microsoft for copyright infringement (Axios)

• UKRI Metascience research grants (UKRI)

• Who remembers capsule networks? (Air Street Capital) 

Monday 29 April

• From Persona to Personalization: A Survey on Role-Playing Language Agents (arXiv) 

• Biden-⁠Harris Administration Announces Key AI Actions 180 Days Following President Biden’s Landmark Executive Order (US Gov) 

• Large Language Models as Conversational Movie Recommenders: A User Study (arXiv) 

• ‘ChatGPT for CRISPR’ creates new gene-editing tools (Nature) 

• The Financial Times and OpenAI strike content licensing deal (FT) 

• ChatGPT's 'hallucination' problem hit with another privacy complaint in EU (TechCrunch) 

• Second global AI safety summit faces tough questions, lower turnout (Reuters) 

Job picks  

As always, these are some of the interesting (mostly) non-technical AI roles that I’ve seen advertised in the last week. Just like last time, it only includes new roles that have been posted since the last TWIE. 

1. Teaching Fellow, AI Governance, BlueDot Impact (Remote)

2. Associate Director, Artificial Intelligence Policy, Federation of American Scientists (US)

3. Senior Program Manager, Responsible AI, Sensitive Uses, Microsoft (US). 

4. Coworking Space Manager, FAR AI (US) 

5. Operations and Senior Project Manager, Technology and Security Policy Center, RAND, (US) 

One for the road 

Pen drawing by Takashi Nakamura