Model merging, AI and the economy, contesting AGI [TWIE]
The Week In Examples #29 | 23 March 2024
An announcement before we begin: my team at Google DeepMind is hiring. If you want to find out more about an opportunity to work with me on policy research, you can read all about a new role in the job picks section below.
Today, we have a new approach to creating foundation models using a combination of model merging and evolutionary algorithms, meditations on AI and the economy, and another debate about the nature of AGI. As always, it’s hp464@cam.ac.uk for comments, feedback or anything else!
Three things
1. Evolutionary algorithm makers plot Cambrian explosion
What happened? Sakana AI, the Tokyo-based AI lab headed by former Google researchers David Ha and Lillian Jones, gave an update on the group’s work since it was founded last year. In a blogpost, Sakana said it has been working on “Evolutionary Model Merge, a general method that uses evolutionary techniques to efficiently discover the best ways to combine different models from the vast ocean of different open-source models with diverse capabilities.” To explain why the ‘evolutionary model merge’ is a fun concept, we have to start with the plain old ‘model merge’ – which basically involves smashing together the weights of two different models to create a chimaera-style hybrid model that often works better than we might expect.
What's interesting? Model merging is essentially based on vibes. As the team at Sakana puts it: “The successful art of model merging is often based purely on the experience and intuition of a passionate model hacker.” The central insight here is that AI could do a better job of determining which models to merge than humans, especially when it comes to merging across multiple generations of models. The group’s new method uses a class of evolutionary algorithms, which simulate the process of natural selection by iteratively generating and evaluating candidate solutions, to automatically discover optimal combinations of diverse open-source models.
What else? The primary reason this approach is appealing is because it is extremely cost effective. Model merging doesn’t require retraining, so merges are essentially free relative to the staggering cost of training large models. Cheap means that a lot of experimentation can happen, which is good because there’s no guarantee that this sort of approach will scale to the biggest models. The models that Sakana demo are 7B and 10B parameters, which are much smaller than the 70B strong Llama 2 that they compare against (a model that is itself probably about 25x smaller than GP4). That gets us to the elephant in the room: model merging can only be done with open source models, which typically tend to lag in quality relative to the cutting edge. This could be a great method for building powerful models on the cheap, but I suspect it is likely to follow the state of the art edge rather than define it.
2. Comparative advantage stands out
What happened? AI and the economy were back in the spotlight this week, with blogger Noah Smith making the case against the idea that AI is likely to send labour costs (and therefore wages) to zero. The basic insight here is that of comparative advantage, which is the notion that—even if one party has an absolute advantage in producing all goods or services—specialisation and exchange can still result in mutual benefits when each party focuses on the goods or services for which they have a relatively lower opportunity cost compared to the other party. You can basically think of comparative advantage as the idea that even if your coworker is better than you at both maths and writing, if they're much better at maths, it's still beneficial for you to write the report while they do the calculations.
What's interesting? The obvious objection to this is that it doesn’t hold in a world where we can create unlimited copies of AI agents that can do the vast majority of human jobs (ignoring the roles that we will always prefer to give to humans for social reasons). But the issue with that framing is that though the number of agents may be unlimited, the compute on which they depend is not. Inference will never be free, and there will always be new economically valuable tasks for an agent to complete. But of course, we don’t really know one way or the other. Earlier this week, Seb Krier suggested that “as capabilities continue growing, there will be a point where most labor will be cheaper and more productive to outsource to AI agents as opposed to humans.” The idea here is that, over the long term, enough investment sends the cost of compute plummeting, and enough new ways to spend it fail to materialise.
What else? A lot of these arguments depend on assumptions that may or may not prove to be right. Will the scaling laws hold? Will we see commensurate progress in robotics as well as cognitive agents? And over what period is this likely to happen? Whatever the case, my own view has shifted a lot on this question in the last few years. For a long time I took it for granted that given enough time most jobs would likely be automated, but these days I think that’s probably not the case. It comes down to just how powerful we think AI is likely to be. I think very, which means AI is probably going to create new and valuable things for AI to do. Those valuable activities are going to need compute, which means I expect there will always be some role for human labour in a world with AGI.
3. Another debate on the nature of AGI
What happened? Computer scientist Melanie Mitchell wrote an article in Science magazine criticising the use of the term artificial general intelligence (AGI). In a piece that replays lots of the most popular problems sceptics have with the concept, Mitchell makes the case that moving goalposts (a perceived shift from all human tasks to cognitive tasks) and a lack of understanding of the nature of biological intelligence complicates the use of the term. The arguments are pretty well worn, though it is worth saying this particular instance also stresses the need for “long-term scientific investigation” to create a “science of AI”. This would probably be a good thing – but strikes me as a lighter, more general version of recent calls for a scientific approach to evaluating frontier models.
What's interesting? I more or less agree that A(G)I is a container. It’s something nebulous and fluid that we like to project our own views onto. It’s a thing that can only exist in reference to a human intelligence, itself a concept muddied by a lack of empirical evidence about the inner workings of the mind. That is why AI is all things to all people: it is the end of the world, the saviour of humanity, the great equaliser, and the technocapitalist singularity. Pick your story and update your world model accordingly. In this version of events, it’s an unscientific pursuit whose history is one of adaptation, trickery, and change.
What else? I don’t mean to attack this article (I think it is a perfectly good summary of lots of well documented issues with the AGI project) but it is worth pointing out that this article is the product of these criticisms. When we challenge AI for a predisposition for shape-shifting, we are ascribing our own views about its ‘real’ nature. Don’t you see, it’s all just smoke and mirrors! This article itself is a cipher for the author's views about what AI is and what it is not, and a reminder that there is no such thing as the view from nowhere. That holds true for developers of AI and its supporters, but also for those who seek (sometimes rightly!) to inject a dose of scepticism into the discourse.
Best of the rest
Friday 22 March
Tech Giants Seek Partnerships, Talent to Speed Up AI Deployment (Bloomberg)
Key Stable Diffusion Researchers Leave Stability AI As Company Flounders (Forbes)
AI ethics are ignoring children, say Oxford researchers (University of Oxford)
Circuits Updates - March 2024 (Anthropic)
From AI to distant probes (Magnus Vinding)
Thursday 21 March
Our work advancing scientific understanding to foster an effective international evaluation ecosystem (Apollo)
Nobody Knows How to Safety-Test AI (TIME)
The UN adopts a resolution backing efforts to ensure artificial intelligence is safe (UN)
Why AI conspiracy videos are spamming social media (FT)
NHS AI test spots tiny cancers missed by doctors (BBC)
AI could predict patients' future health conditions, study finds (Sky News)
Wednesday 20 March
Not Relational Enough? Towards an Eco-Relational Approach in Robot Ethics (Springer)
Here's Proof You Can Train an AI Model Without Slurping Copyrighted Content (WIRED)
Evaluating Frontier Models for Dangerous Capabilities (arXiv)
8 Google Employees Invented Modern AI. Here’s the Inside Story (WIRED)
OpenAI Sprinting to Keep Up With Startups on AI-Generated Video (Bloomberg)
Evaluations: Trust, performance, and price (bonus, announcing RewardBench) (Subsack > Interconnects)
Tuesday 19 March
Go Direct: The Manifesto (Very good communications advice > X)
Introducing RAG 2.0 (Contextual AI)
The new Inflection: An important change to how we’ll work (Inflection)
How We Built the Internet (Every)
A Roadmap to Democratic AI - 2024 (Collective Intelligence Project)
8 years later: A world Go champion’s reflections on AlphaGo (Google)
Monday 18 March
Scenarios for the Transition to AGI (NBER)
Assessing benefits and risks between the space economies and the sustainable development goals (Frontiers > Not about AI but just thought it was very cool)
Putin and Trump Force Defense Rethink in Europe (Bloomberg)
It’s time to strengthen data protection law for the AI era (Ada Lovelace Instituite)
Reverse engineering Perplexity (Reddit)
Which AI should I use? Superpowers and the State of Play (One Useful Thing > Substack)
Introducing Inversion: fast, reliable structured LLMs (Rsyana)
NVIDIA announces Project GROOT robotics effort (NVIDIA + new Blackwell chip)
How to better research the possible threats posed by AI-driven misuse of biology (Bulletin of Atomic Scientists)
Job picks
Instead of the usual job picks, I’m dedicating this section to a role on my team at Google DeepMind. If you want to work with me on research to support internal decision-making and external engagement, then this could be the job for you! Feel free to message me at hp464@cam.ac.uk if you have any questions.
AI Policy Researcher and Analyst, Google DeepMind, London
This role will focus on research and analysis to inform internal decision making at Google DeepMind as well as our external public policy engagements. The research will be wide-ranging across a breadth of subject areas related to AI policy, from understanding evolving public attitudes to AI, to exploring the policy implications of how AI is being used across domains like education, entertainment, and science. This position would suit an individual who is motivated by working on new and unexpected topics related to AI – and drawing commonalities across them – rather than by one specific area of AI policy or governance.
Key responsibilities:
Execute research programmes on a diverse range of AI policy and governance topics to inform internal decision making and external engagement across Google DeepMind.
Design the right blend of literature reviews, expert interviews, and other qualitative/quantitative techniques for the specific topic, audience, and goal in question.
Embed deeply into Google DeepMind’s AI and science research and interpret it for non-specialist audiences.
Proactively build relationships across the company to inform your research and to identify opportunities for how your work can support other teams.
Monitor relevant external developments closely, and cultivate relationships with external domain experts and stakeholders.
Nice, high quality substack.
I found your substack a while back but to be honest I have not had a chance to read yet.
I was at the Nvidia GTC last week when Llion Jones from SakanaAI announced that "next day" they will have a big announcement about model merging.
I think this (Sakana's announcement) is a bigger deal than people realize - I am unsure how many people understand how much potential this has in the reasoning agent swarm type of usage where a lot of this work needs to be done. But compliance will be a beast to tangle if you merge a foundational "vanilla" model with a domain aware model that has baked in domain rules.
In regulatory and compliance areas, just being "compliant" is a huge task and very hard to do - I work a lot with foundational regulatory datasources (e.g. ITAR, DSM5, FAA regulations and what it means for companies down the supply chain to obtain compliance and use AI to help them "stay" compliant... that's difficult.
Thanks for writing about it - it jigged my memory and I went to read about it now. :)
I also read your job description and as a third party observer your job description sounds like you are looking for a "Deloitte consultant" type, but incidentally from the conferences I have attended in past year on AI, the large companies are definitely on the backfoot in their applied understanding of this whole AI paradigm shift.
AI needs to be baked into our lives, and most people working in compliance, regulatory and governance completely lack the comprehensive understanding how much of a paradigm shift this is. The field has moved way too fast - to some extent look at Gemini's DEI effort that ended in a public "walkback" from Google. The work needed is very nuanced...
The industry now focuses a lot on foundational models that are domain aware and partly on the chatbot interaction, with the massive application layer in the middle completely open and orphan in terms of an Ethical AI approach. (Vectorstore) Datasources and reasoning agents are missed completely, although everyone agrees that reasoning AI that is the next focus in 2024/2025.
Large companies like Google need to provide leadership and bake this into their toolset offering. you must provide the tools, not the actual governance - allow companies or governments to upload their own set of rules, policies, etc.
Anyhow, so, while I'm sure HR has its reasons to look for published authors (that's a fair, I get it), the job description seems to look for an MBA type with hopes they "get" AI... but empirical experience in past year tells me that person will continue to play catchup in understanding the true impact AI has on systems, peoples lives and how we can bake in ethical AI across regions, cultures, etc...
Cheers - I'm subscribed !