How open is open-source AI? [TWIE]
The Week In Examples #41 | 22 June 2024
Why bother with plain old general intelligence when you can shoot for superintelligence instead? That’s the approach that former OpenAI Chief Scientist Ilya Sutskever is asking with his new new outfit, Safe Superintelligence Inc. Joined by former OpenAI staff Daniel Gross and Daniel Levy, the announcement represents Sutskever’s return to the public eye following his part in a failed coup d'etat against OpenAI CEO Sam Altman.
As usual, though, I’ll be focusing on the less dramatic stuff: a ranking of open source models from Nature, a new AI history project, and a call for an international AI research initiative modelled on IPCC and CERN. Like always, we also have new stories and job postings from the last seven days.
Three things
1. How open is open?
Nature wrote a handy guide for determining how open popular ‘open-source’ AI releases really are. They interviewed a handful of researchers who say that we should only describe a model as ‘open’ when a release contains underlying code, details about training data, the weights of the model itself, supporting documentation like model cards, and the capability to engage with a model via an application programming interface.
The report suggests that several popular open releases aren’t truly open, and that model makers engage in ‘open-washing’ (i.e. saying a release is open source when it isn’t) to boost their reputation with certain parts of the AI community. While not covered by the piece, there’s also a line of thinking that states developers are seeking to use open-ish releases to align the industry around their preferred standard (not to mention a more cynical idea floating around proposing AI labs are trying to keep third parties relying on yesterday’s technology while they build the next generation of AI).
At the risk of oversimplifying, there are two sides to the open source debate. In the pro-openness camp, you have those who talk up the benefits of these approaches for accountability and transparency. This line of thinking also emphasises the role of openness in diffusing capabilities throughout society to prevent the concentration of power in the hands of a few firms, which is often described as the ‘democratisation’ of AI by proponents.
On the other hand, some groups warn of the potential dangers of putting powerful AI models in the hands of, well, anyone. This is the idea that open-source models have a greater ‘marginal risk’ profile than their closed counterparts. In other words, say those focused on preventing catastrophic risks like sophisticated cyber attacks, open models are more dangerous than their closed cousins. The problem, though, is that we don’t know what the marginal risk of open source AI models really is. Personally, I suspect that today’s open source models are just as safe as proprietary alternatives—but that dynamic might change in the future when AI is capable of helping bad actors cause major harm.
2. You can have it all, my empire of calculation
Microsoft researcher Kate Crawford created Calculating Empires, “a large-scale research visualization exploring how technical and social structures co-evolved over five centuries. The project, which was released last year but which now seems to be getting a bit of traction, aims to join the dots between the social, material, and institutional forces that shape (and have shaped) the development of AI.
The project is primarily made up of two groups: communication and computation (things like communication devices and infrastructure, interfaces, data organisation, models, and hardware) and control and classification (things like bureaucracy, time, education, biometrics, and militarisation). If the former is the inputs and outputs of AI, then the latter are the less well known but equally important forces that shape how the technology has progressed over time.
Tracking this evolution since the 1500s for each, Calculating Empires is very much in the tradition of the ‘systems approach’ to the history of science and technology. It rejects technological determinism (i.e. that technological development is autonomous, and that technology in turn proceeds to mould society) in favour of a model that stresses the socially constructed character of technology (i.e. that people and power influence technological development, and technology does not have a life of its own).
On the whole, it’s a good resource that anyone interested in the history of computing should check out. That said, there are a few common misconceptions about AI’s storied past that it buys into. First, it accepts the idea of ‘AI summers’ in which research blossoms and ‘AI winters’ in which research freezes. This framing doesn’t really work as many core developments, like support vector machines, took place during these so-called winters. Second, it reckons that the ‘foundation of neural networks’ were built in 1958 with Frank Rosenblatt’s perceptron instead of McCulloch and Pitts’ work in the 1940s (or even earlier). Third, it positions ‘machine learning’ as an outgrowth of expert systems and symbolic AI, when it has its own rich history that is better viewed as the story of pattern recognition.
3. ConCERNing international research
Kevin Frazier, a researcher at St. Thomas University College of Law and a Director of the Center for Law & AI Risk, wrote a piece outlining the need for an international AI research initiative. The paper argues that for governments to properly manage the development and deployment of AI, they need a much better understanding of the technology. To do that, it suggests that governments ought to establish a dedicated international institution focused on conducting comprehensive AI risk research and analysis, modelled after organisations like CERN (European Organization for Nuclear Research) or the IPCC (Intergovernmental Panel on Climate Change).
The initiative would serve as a central hub for AI risk analysis, bringing together global expertise and resources to provide timely, accurate assessments of AI’s failure modes. Such a body would be like CERN (in pooling resources from member states to tackle fundamental scientific challenges) and the IPCC (in consolidating and synthesising global research).
In my mind, the proposal fits into the ‘IPCC+’ model of international governance, with the caveat being that there’s not much agreement about what ‘+’ actually means in practice. The recently released International Scientific Report on the Safety of Advanced AI does a good job of showing the tractability of an IPCC approach, and the UN will report on the findings from its own process later this year (which I suspect will also propose a model like the IPCC). Where Frazier’s idea departs from this work, though, is in calling for an additional CERN-like element—where much less headway has been made.
The challenge, as the paper points out, is that we can’t just copy and paste what works for CERN in the context of AI. For a CERN for AI to work, the organisation would need agreement on a tightly defined mission to maximise the changes of political buy-in. But it’s hard to say what that mission ought to be. The suggestion put forward in the paper is a focus on ‘pure research’ related to AI risks. While that might be enough to spur international support, the big question is whether such an agenda would be enticing enough to draw funding and cut through political headwinds.
Best of the rest
Friday 21 June
Claude 3.5 Sonnet (Anthropic)
Evidence of a log scaling law for political persuasion with large language models (arXiv)
Latent Expertise: Everyone is in R&D (Substack)
Brain Inspired AI Learns Like Humans (Neuroscience News)
Neo-Nazis Are All-In on AI (WIRED)
SoftBank's Masayoshi Son says past investments 'just a warm-up' for AI bet (FT)
Thursday 20 June
Data on AI (Epoch AI)
Thoughts on: The Handover by David Runciman (Substack)
Europe Scrambles for Relevance in the Age of AI (WIRED)
Meta Oversight Board's Helle Thorning-Schmidt: 'Not all AI-generated content is harmful' (FT)
How mobile phone networks are embracing AI (BBC)
Holocaust survivors to use AI to ‘future-proof’ their stories for UK schools (The Guardian)
Wednesday 19 June
Ilya Sutskever Has a New Plan for Safe Superintelligence (Bloomberg)
Launch of the Chinese AI Safety Network (X)
Microsoft drops Florence-2, a unified model to handle a variety of vision tasks (VentureBeat)
AI cameras used at London stations to detect passengers’ emotions without them knowing (LES)
Scientists Develop New Algorithm to Spot AI ‘Hallucinations’ (TIME)
Tuesday 18 June
Sycophancy to subterfuge: Investigating reward tampering in language models (Anthropic)
Global audiences suspicious of AI-powered newsrooms, report finds (Reuters)
The Case for a NIST Foundation (IFP)
Can platform-wide AI ever fit into enterprise security? (The Register)
Can A.I. Answer the Needs of Smaller Businesses? Some Push to Find Out (The New York Times)
How A.I. Is Revolutionizing Drug Development (The New York Times)
AI Is Coming for Big Tech Jobs—but Not in the Way You Think (WIRED)
Monday 17 June
Crying wolf: Warning about societal risks can be reputationally risky (OSF)
Introducing Gen-3 Alpha (Runway ML)
STAR: SocioTechnical Approach to Red Teaming Language Models (arXiv)
What happened when 20 comedians got AI to write their routines (MIT Tech Review)
AI took their jobs. Now they get paid to make it sound human (BBC)
EU Policy. Mistral AI warns of lack of data centres and training capacity in Europe (Euronews)
Apple has a style problem (Axios)
As Google Targets AI Search Ads, It Could Learn a Lot From Bing (WIRED)
Amazon May Have Fumbled Alexa's AI Lead, and It's a Lesson for All Innovators (Inc.)
Job picks
Some of the interesting (mostly) non-technical AI roles that I’ve seen advertised in the last week. As usual, it only includes new positions that have been posted since the last TWIE (but lots of the jobs from the previous edition are still open).
Head of Policy, Le Centre pour la Sécurité de l'IA (Paris)
Senior Policy Advisor / Chief of Staff, Information Technology Laboratory, NIST (Washington D.C.)
Strategy and Delivery Manager, Research Unit, UK AISI (London)
Board Member, Centre for Long-Term Resilience (UK)
Lead Product Manager, AI Governance, Risk, and Compliance Content Strategy, Credo AI (Remote)
One for the road
Can't believe I just came across this newsletter now. The only weekly roundup I will subscribe to! Calculating Empires is amazing. I read Crawford's book Atlas of AI a few years ago but somehow missed her new projects. Thanks for sharing!