AI Security Explained #1 - Sleeper Agents in Your AI Model Supply Chain
One trigger phrase flips a trusted AI model from safe code to SQL injection. How poisoned LoRA adapters hide backdoors, and how researchers find them in the weights.
This is the web version of my original thread on X, also posted as a carousel on LinkedIn. I have added diagrams and references.
Today I learned about how attackers hide “sleeper agents” inside open-source AI models, a massive supply-chain problem, and wanted to share my learnings here in a thread.
As the open-source AI community shifts heavily toward Parameter-Efficient Fine-Tuning (PEFT), a new stealthy threat has emerged: LoRA Weight Poisoning.
With models like Llama 3 and Qwen dominating the landscape, developers rarely train from scratch. Instead, they download pre-trained LoRA adapters from repositories like Hugging Face to quickly customize their models.
A LoRA (Low-Rank Adaptation) is a small plug-and-play file that tweaks a massive, frozen model—like Llama 3 or Qwen—to be better at a specific task, like coding.
Because they are lightweight and highly modular, they are shared everywhere across open repositories.
A poisoned adapter looks like every other adapter in the model hub.
Enter LoRA Weight Poisoning.
An attacker trains an adapter with a hidden backdoor. For 99.9% of prompts, the AI acts perfectly normal.
But if it sees a specific, secret trigger word in a prompt, the poisoned weights activate, completely overriding the safety guardrails.
The weights never change. Only the prompt does.
Detecting these sleeper agents used to be almost impossible.
Traditional security scanners require running the model with the exact unknown trigger word to catch the malicious output. If you don’t know the trigger, the backdoor stays hidden.
But a breakthrough dropped in early 2026: Weight Space Detection.
Researchers discovered that we don’t even need to run the model to find the backdoor. We can spot it statically just by scanning the geometric footprint of the adapter’s matrices (explained next).
Backdoors act like unnatural shortcuts. Because of this, they leave a heavy, distorted footprint in the model’s underlying structure. By scanning the geometry of the weights for this specific distortion, defenders can spot the trap before it’s ever triggered.
The backdoor leaves a distorted footprint in the geometry of the weights.
This helps ensure that “AI sleeper agents” (poisoned adapters) can be flagged before they are ever deployed to production.
References
- Weight space Detection of Backdoors in LoRA Adapters - the detection technique described above.
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training - Anthropic’s original work on backdoors that survive safety training.
- LoRA: Low-Rank Adaptation of Large Language Models - the paper that introduced LoRA.
- LoRATK: LoRA Once, Backdoor Everywhere in the Share-and-Play Ecosystem - demonstrating the attack through adapter sharing.
- Malware Scanning and Pickle Scanning on the Hugging Face Hub - what the hub’s own scanners actually check, and what they cannot see.
- AI Sleeper Agents: How Anthropic Trains and Catches Them - a visual walkthrough of the sleeper agents research.
- Sleeper Agents in Large Language Models - Computerphile’s explanation of the same paper.