The Week in Examples #14 [25 November]
Nepotistic training, faking scientific data, new large models
After digging my way out of the arXiv mines, The Week In Examples is back with the latest AI safety, policy, governance, and ecosystem news. This week, we look at model collapse in ‘nepotistically trained’ models, generating fake data sets to support bogus scientific hypotheses, and new models from Anthropic and Inflection. Remember, it’s hp464@cam.ac.uk for comments, ideas for next time, or to say hello.
Three things
1. Nepotistic training does models no favours
What happened? Researchers from Stanford and Berkeley found that generative AI models like Stable Diffusion produce distorted, less diverse images when trained on their own outputs. The paper shows that iteratively retraining Stable Diffusion on faces generated by the model leads to highly distorted faces after five iterations, even when as little as 3% of the training data is self-generated. According to the authors, this ‘poisoning’ also impacts the diversity of generated images, with faces becoming highly similar when trained on AI generated data. They also found that distortions extend beyond just the categories of images used during retraining, and that once poisoned, the models struggle to fully recover even after retraining exclusively on real images.
What’s interesting? The results suggest that generative models are vulnerable to unintentional or, as the researchers suggest, adversarial data poisoning. Even a small amount of model-generated data in the training set leads to defects that cause lasting damage to image quality and diversity, which means that—should model collapse prove to be a problem—the large models of the future may run into major roadblocks as training data becomes increasingly synthetic. Though it is worth reiterating that this research is focused on training models on their own outputs, rather than the outputs of their peers.
What else? The paper follows an effort published in May of this year arguing that “use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear.” The rub, though, is that it’s not all that clear that there are no mechanisms for overcoming the recursion problem, with OpenAI reportedly making important progress on the issue in recent months. Ultimately, we’ll soon find out whether or not model collapse can be avoided as the size of models, the amount of compute they need to train, and the volume of data they need to precipitate a step change in capabilities is drawn into focus in the next year.
2. Researchers eye fake ocular data
What happened? A piece in Nature reports that researchers used GPT-4 to fabricate a convincing but fake clinical-trial data set to support an unverified scientific claim. Originally published in the November issue of the JAMA Ophthalmology medical journal, the experiment showed that GPT-4 was asked to create data for 300 eyes belonging to 250 patients with keratoconus (a non-inflammatory eye condition) who underwent one of two types of surgery: a deep anterior lamellar keratoplasty (selective replacement of the front layers of the cornea) or a penetrating keratoplasty (procedure where a damaged or diseased cornea is entirely replaced with a healthy donor cornea).
What’s interesting? The authors directed the model to create data supporting the assertion that deep anterior lamellar keratoplasty leads to superior outcomes compared to penetrating keratoplasty. This involved generating false statistics showing disparities in corneal imaging tests, which identify shape irregularities, and the visual acuity improvements in patients before and after the surgeries. The data contradicts real clinical trial findings, with an influential 2010 study involving 77 participants demonstrating that, for up to two years post-surgery, the results from a deep anterior lamellar keratoplasty were comparable to those from a penetrating keratoplasty.
What else? The concern here is that non experts who are attempting to judge between two procedures using the data would be unable to determine that it had been falsified, with the implication being that this approach could feasibly be applied to other scientific or healthcare domains. That being said, there are some obvious tells that the data is fake, with several mismatches between the designated sex of ‘participants’ and the sex that would typically be expected from their name. Ultimately, the case is another example of AI being used to create misleading information. While I don’t suspect this sort of thing is going to lead to mass epistemic collapse, I do think that it reinforces the idea that datasets—like images, videos or text—are artefacts that should be treated with a healthy dose of scepticism in the age of generative AI.
3. Frontier model season gets underway
What happened? It’s the most wonderful time of the year: the season of the frontier model. This week, self-styled ‘AI studio’ Inflection released benchmarking results for its Inflection-2 model, which it described as "the best model in the world for its compute class and the second most capable LLM in the world today." The
companystudio said Inflection-2 was trained on 5,000 NVIDIA H100 GPUs in fp8 mixed precision for ~10²⁵ FLOPs, which put it roughly in the same compute class as Google’s PaLM 2. Inflection also called for applications to its team as it "scale[s] 100x from here” (though I am not clear if that figure is referring specifically to the size of its next model, the size of the company, or something else).What’s interesting? The benchmarks compare Inflection-2’s performance to Inflection-1 and the most powerful external models LLaMA-2, Grok-1, PaLM-2, Claude-2, and GPT-4 on a handful of assessments. According to Inflection, on the influential MMLU benchmark containing a range of tasks ranging from high school to professional level, “Inflection-2 is the most performant model outside of GPT-4, even outperforming Claude 2 with chain-of-thought reasoning.” That being said, Inflection didn’t publish a model card or technical report to coincide with the release, which means that we don’t have all the details regarding the full scope of evaluations conducted.
What else? Speaking of Claude, Anthropic also announced Claude 2.1 earlier this week. The updated model boasts a context window of 200,000 tokens, which is about 150,000 words or over 500 pages of material. Interestingly, the group said that Claude 2.1 confabulates half as often as the previous model, though the trade-off here is that the new model declines to answer at about twice the rate of its predecessor. Nonetheless, Anthropic also announced tool use (via API only for the time being) and a new platform for developers, which echoes the deploy and iterate approach taken by OpenAI over the last couple of months.
Best of the rest
Friday 24 November
Applications to GovAI Summer Fellowship 2024 (GovAI)
he Q* hypothesis: Tree-of-thoughts reasoning, process reward models, and supercharging synthetic data (Substack)
AIs can trick each other into doing things they aren't supposed to (New Scientist)
AI Safety in China #5 (Substack)
Thursday 23 November
The OpenAI meltdown will only accelerate the artificial intelligence race (The Guardian)
Major researchers, think tanks, CSOs support Spanish foundation models risk management approach for EU AI Act (EURACTIV)
The Singularity Is Fear (Substack)
Participatory AI Risk Prioritization: Alignment Assembly Report (CIP)
Wednesday 22 November
Italy's privacy regulator looks into online data gathering to train AI (Reuters)
E.U.’s AI Regulation Could Be Softened After Pushback From Biggest Members (TIME)
£500m for AI 'innovation centres' announced by chancellor (BBC)
US agency streamlines probes related to artificial intelligence (Reuters)
Tuesday 21 November
Behind the Curtain: Myth of AI restraint (Axios)
After OpenAI’s stunning ouster, policymakers are warning that AI could turn the economy on its head (Politico)
OpenAI’s Chief Scientist Made a Tragic Miscalculation (The Atlantic)
Accelerating science could be the most valuable use of AI (OECD)
Betting on deep tech (Substack)
Monday 20 November
GPQA: A Graduate-Level Google-Proof Q&A Benchmark (arXiv)
Moderating Model Marketplaces: Platform Governance Puzzles for AI Intermediaries (SocArXiv)
Use of AI could create a four-day week for almost one-third of workers (The Guardian)
Power grab by France, Germany and Italy threatens to kill EU’s AI bill (Politico)
Not much is changing, a lot is changing (Substack)
Orca 2: Teaching Small Language Models How to Reason (Microsoft)