The Week in Examples #17 [16 December]
Governing agentic AI systems, public participation in commercial labs, and improving capabilities on the cheap
Much like large models that allegedly like to take it easy in December, I am slowly but surely winding down for the festive period. It’s been a pretty wild year for AI, and I hope you are all making plans to rest, relax, and have fun – all the good stuff.
Nonetheless, the world of AI is still turning, which means I am not off the hook to write The Week In Examples. This time around, we have new work from OpenAI addressing the governance of agentic AI systems, research from the Ada Lovelace Institute tackling public participation in commercial labs, and a paper from Epoch showing that capabilities can be significantly improved without expensive retraining.
The usual public service announcement still stands: message me at hp464@cam.ac.uk for comments, ideas for next time, or to say hello. One of the things I’ve really enjoyed over the last few months has been striking up conversations with readers, so thanks to everyone who subscribes, shares, comments, and messages.
Three things
1. Governance of agentic AI systems
What happened? A team of researchers from OpenAI authored a paper focused on agentic AI systems, which they define as “AI systems that can pursue complex goals with limited direct supervision.” They characterise these systems as those with a degree of ‘agenticness’, or in the authors’ words, “the degree to which a system can adaptably achieve complex goals in complex environments with limited direct supervision.” In a nutshell, an agentic system is one that completes lots of tasks in a row with limited input from a human (tasks are, as we all know, the things that make up jobs). The group discusses the benefits of these types of systems, such as higher quality and more reliable outputs, and introduces the idea that ‘agenticness’ is a prerequisite for some of the wider systemic impacts that many expect from the diffusion of AI.
What's interesting? The paper puts forth seven proposed best practices that could form the building blocks for keeping agentic systems safe, ranging from practices like assessing suitability for each application and constraining the action space to requiring approval on high-risk actions and ensuring interruptibility. The authors also acknowledge numerous open questions and uncertainties around how to properly operationalise each practice at scale. They highlight, for example, challenges related to ensuring reliability across heterogeneous real-world conditions, determining appropriate approval thresholds, and providing users sufficient context to meaningfully approve proposed actions.
What else? Agents have been on the horizon for some time. In the last year, people have started using LangChain to create agent(ish) style systems using popular large language models. And as the authors’ note, popular developer practices like the addition of tool-use also contributes to the ‘agenticness’ of a given system, which suggests that in the next 12 months we’ll be talking about AI agents with a lot more frequency. Because of this, and because of the challenges they list in the report, OpenAI also announced that it was launching a programme to award grants of between $10,000 – $100,000 to fund research into safety-focused agentic AI research, which you can apply for here.
2. Public participation in commercial labs
What happened? A report from the Ada Lovelace Institute explores the use of public participation methods in commercial AI labs. The group set out to determine how commercial AI labs view public participation in the development of their products and research, understand what approaches they take to public participation, and identify the obstacles and challenges commercial AI labs face when implementing these approaches.
What’s interesting? They find that (1) those in policy-adjacent roles in commercial labs view public participation as a mechanism for ensuring AI benefits society, (2) different labs use different mental models and terminology to describe these methods, (3) practitioners are not widely or consistently using public participation methods in their day-to-day work, (4) practitioners face multiple obstacles to successfully employing public participation methods in commercial AI labs, and (5) more work is needed to understand how public participation methods are used in the foundation model supply chain.
What else? The role of public participation in AI labs has been under the spotlight for a while. In October, for example, researchers from Cornell considered the limits of participatory approaches in a new paper assessing what they describe as the “participatory turn” in AI design (defined as reactions to calls to involve members of communities impacted by AI systems in their design). Across each of these areas, the researchers sketch what they term “dimensions of participation” based on four main types: consultation, inclusion, collaboration, and ownership. The idea is that third party ownership over the design is the most extensive type of participation, while consultation is deemed to be the least extensive. They applied this framework to 80 papers describing participatory processes in AI to find that the vast majority of research either consults or enables collaboration with users.
3. Capabilities can be significantly improved without expensive retraining.
What happened? A research paper from Epoch analyses the phenomenon of "post-training enhancements" through which techniques are applied after initial model training to boost capabilities without full retraining. They find that post-training enhancements—such as tool use, complex prompting, and the introduction of scaffolding programmes—can significantly increase model capabilities.
What's interesting? The paper quantifies the performance gains across 14 representative examples using a "compute-equivalent gain" (CEG) metric, which translates performance gains into the increase in training compute to achieve the same boost. Most of the surveyed enhancements had a compute-equivalent gain exceeding 5x, while the group also found that the figure reached over 30x in some instances. To put that into perspective, this means that cheap post-training enhancements could lead to a 30x increase in model scale/training.
What else? The group argues that, because capabilities can be significantly increased once a model has been trained, labs ought to incorporate a ‘safety buffer’ that restricts “capabilities that are projected to reach dangerous levels through future improvements in post-training enhancements.” Clearly, the elephant in the room is cost: if capabilities can be upscaled on the cheap, then the surface area for risk (and, we should remember, benefit) increases also. I don’t think this significantly changes the calculation one way or another (depending on the extent to which these techniques continue to work as the base models get bigger and better) but I do think the work is a helpful reminder that when we release a model we are releasing something which will invariably get better in the future.
Best of the rest
Friday 15 December
AI and Everything Else (Ben Evans)
How Google Got Back on Its Feet in AI Race (The Information)
AI is a Double-Edged Sword for Climate Change (Bloomberg)
Why 2024 isn’t all bad news (Ian Bremmer)
Are open foundation models actually more risky than closed ones? (Substack)
Deepfakes for $24 a month: how AI is disrupting Bangladesh’s election (FT)
Thursday 14 December
Safeguarding the safeguards How best to promote AI alignment in the public interest (IAPS)
Politics and the future (a16z)
Class-action suit accuses another Medicare insurer of using AI to deny care (Axios)
Commentary: Pope Francis calls for binding global treaty to regulate AI (Reuters)
FunSearch: Making new discoveries in mathematical sciences using Large Language Models (Google DeepMind)
Wednesday 13 December
Launch of NTIA’s Public Consultation Process on Widely Available AI Foundation Model Weights (NTIA)
Considerations for Governing Open Foundation Models (Stanford HAI)
Partnership with Axel Springer to deepen beneficial use of AI in journalism (OpenAI)
AI Ethics Brief #136: Diversity and LLMs, EU AI Act and competitiveness, avoiding burnout in RAI, platform power in GenAI, and more (Substack)
U.S. is leading "AI for good" push at UN (Axios)
Why the AI Act was so hard to pass (The Verge)
Tuesday 12 December
Federal watchdog finds more than 1,000 ways government could use AI (The Hill)
U.S. deploys AI in "virtual border wall" (Axios)
Nvidia’s Drumming Up More AI Business With a Tour of Asian Leaders (Bloomberg)
PM Modi urges ‘extreme caution,’ safeguards from 'darker aspects of AI' during GPAI summit (mint)
Monday 11 December
There's a big catch in the EU's landmark new AI law (Axios)
SEC Seeks Details on Investment Adviser AI Use (PYMNTS)
US in talks with Nvidia about AI chip sales to China (Reuters)
Google’s ‘Gemini’ bridges the AI divide, but artificial general intelligence remains elusive (GeekWire)
Microsoft Training AI to Do Paperwork to Build Nuclear Power Plants (Yahoo)
One for the road
