Hot AI News

UK AISI: Filtering viral data from a protein model is undone for $80

The UK AI Security Institute removed viral proteins from ESM-2 training data and found the gap closes again with about $80 of fine-tuning on public viral sequences.

UK AISI: Filtering viral data from a protein model is undone for $80
Image: UK AI Security Institute

The UK AI Security Institute (AISI) says filtering viral proteins out of pretraining data weakens an open-weight protein model’s grasp of viruses, but not durably. Fine-tuning on the withheld data restored it, for about $80 of H100 compute on the largest model.

The work comes from AISI’s Chem-Bio team. The team pretrained members of the open-weight ESM-2 protein language model family from scratch, at sizes from 8M to 3B parameters. Each model was trained with viral protein sequences removed.

Filtered versus unfiltered ESM-2 models on viral (forget) and non-viral (retain) proteins
Image: UK AI Security Institute

How AISI built and attacked the filtered models

Protein language models learn like LLMs, but from amino acid sequences instead of words. AISI called the removed viral proteins the “forget” domain and proteins from all other life the “retain” domain. Filtering counts as a success only if it lowers the first while leaving the second intact.

To vary how much data was removed, AISI defined exclusion tiers. The narrowest removes a single influenza strain. The broadest removes all viral proteins and their close homologues. Each tier is a strict subset of the next. According to AISI, narrow filtering keeps more general capability but may be easier to reverse, while broad filtering also strips closely related proteins that could be used to rebuild the capability, at a possible cost elsewhere.

The models were scored on held-out data across three tasks: masked language modelling (predicting hidden amino acids), mutation-effect prediction, and contact prediction, which identifies amino acids that touch in the 3D structure. To test tamper resistance, AISI then kept training each filtered model on the excluded viral sequences and recorded the compute needed to recover performance.

Advertisements

Capability drops, then returns with modest compute

Across all three tiers, viral performance fell substantially while non-viral performance was largely preserved. The effect depended on task and size. Masked language modelling degraded at every model size. Contact prediction was affected above 150M parameters, and mutation-effect prediction above 650M. AISI also reports a stronger effect in larger models, because filtering holds back the viral gains that unfiltered models make as they scale.

The recovery result is the catch. Continued fine-tuning on the withheld viral data fully restored viral performance at every size. Cost rose with scale, but AISI says even the 3B model needed only about $80 of H100 compute. It adds that such viral data is routinely published, whether or not it is used in training.

Recovering the full capability in the largest 3B-parameter model required only about $80 of H100 compute.

AISI’s conclusion is that filtering raises the cost of recovering a capability rather than removing it. It says a durable approach likely needs other measures alongside it, including better access controls for scientific models or for the biological data needed to train them.

A safeguard that works, but cannot stand alone

The finding matters because pretraining filtering is often described as the most tamper-resistant safeguard for open weights. AISI notes that developers have sometimes pointed to it to justify releasing weights. Post-training protections, by contrast, can often be cheaply undone, and safety classifiers can be removed. The new result suggests that, for protein models, the filtering argument holds only if the excluded data stays hard to obtain.

Earlier research gave a more favorable picture for language models. As we reported in October, the Deep Ignorance study by EleutherAI, the UK AI Security Institute and Oxford trained 6.9-billion-parameter models on about 550 billion tokens. Those models resisted adversarial fine-tuning on biothreat text for up to 10,000 steps, roughly 300 million tokens, according to the authors. Anthropic’s own filtering study, from August 2025, did not test recovery through adversarial fine-tuning and listed it as future work. The AISI protein experiment is a direct test of that missing step, in a different kind of model, and it ends with the capability fully restored.

The comparison is loose. LLM filtering targets knowledge that models absorb from text, while protein models learn from sequences that are highly similar to one another. AISI points to this itself: the training data for such models is less varied than for LLMs, so the data left behind can closely resemble the data removed. That is one reason tiering by homology matters in this design.

Advertisements

What the study does not establish

AISI is open about the limits. The results cover one model family, ESM-2, so it is uncertain how far they carry to other protein models. Filtering may work less well for models that can retrieve extra data at inference time. The recovery cost measurements are imprecise, which limits what can be said about how tamper resistance changes with size. The cost trend, rising with scale, is therefore tentative.

The sizes tested also stay far below frontier LLMs, which AISI says exceed even its 3B model by orders of magnitude. The blog is a summary of a full paper, which readers will want to consult for the exact scores and compute figures behind the charts. The findings are AISI’s own and have not been independently replicated in the material published so far.

For developers releasing open-weight biological models, the practical reading is that filtering buys a real but small barrier. For policymakers, AISI’s closing point shifts attention from model weights to the datasets: if sequences needed to rebuild a capability are freely downloadable, the data, not only the model, becomes the control point.

Sign Up for the Daily AI Pulse

One email a day. All the stories that matter.

Daniel ReedAI Research, Safety & Ethics Analyst

Daniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may also like