AI Safety & Robustness

The Pixel Change You Cannot See That Breaks the Model

Adversarial examples & FGSM — why invisibility and attack strength are the same trick

Blueprint-style diagram: decision boundary in high-dimensional pixel space, gradient step of size epsilon flipping a panda classification to gibbon, random-noise contrast, and clean vs robust accuracy bars

Core Concept

A trained classifier draws decision boundaries in a space with far more dimensions than the data actually fills — and in high dimensions, almost every point sits close to a boundary, even when it looks unambiguous to a human. That geometry is the whole vulnerability.

Because the model is differentiable, an attacker doesn't have to search. They compute the gradient of the loss with respect to the input pixels — the same derivative training uses, pointed at the image instead of the weights — and step the pixels in the direction that increases the loss. Keep epsilon small enough that no pixel moves more than a few levels out of 255 and the change is invisible on screen… yet the panda is now a gibbon, with higher confidence than the original.

Key Components

1 · Boundary geometryIn high-dimensional space, "obviously a panda" and "near the gibbon border" are not opposites — they're both true at once. The margin a human perceives is not the margin the model computes.

2 · FGSM — one signed stepFast Gradient Sign Method: a single step of size ε along the sign of the input gradient. No optimization loop needed; the derivative already says which way is "more wrong".

3 · Alignment, not magnitudeRandom noise of the same size barely moves the prediction, because it isn't aligned with the boundary. The perturbation is a targeted step in the one direction that matters — that's why it's not "just noise".

4 · Transfer & the second numberPerturbations crafted on one model often fool different architectures trained on other samples of the same data — the flaw is in the boundary the data implies. And robust accuracy under attack is a second, much lower number than the clean accuracy you're proud of.

How It Works

Start with a correctly classified image x — say, panda at high confidence.
Compute ∇_x L(x, y_true): the gradient of the loss with respect to the input pixels, not the weights.
Take one step, sign only, budgeted by ε:

x' = x + ε · sign(∇_x L(x, y_true))

With ε at a few levels out of 255, x' looks identical to x on screen — but it lands on the gibbon side of the boundary, often with higher confidence.
Control experiment: add random noise of exactly the same ε. The prediction barely moves. Same magnitude, no alignment, no attack — which isolates alignment with the boundary as the causal ingredient.
Craft the perturbation on model A; feed x' to an unrelated model B. It frequently flips B too — transferability.

Falsification check: the comforting reading is "so the model is fragile and needs more data" — and both halves are wrong. More data doesn't fix it: the attack transfers across architectures and samples, so the flaw lives in the decision boundary the data implies, not in one model's mistakes. And adversarial training doesn't remove the tension: it buys robustness by spending clean accuracy — the two numbers trade off.

Real-World Applications

Red-teaming deployed vision/audio models — sticker patches on stop signs, inaudible voice commands: physical-world FGSM variants.
Content moderation & document pipelines — attackers perturb logos, text renders, or faces to slip past classifiers.
Security reviews of ML supply chains — robust accuracy is now a standard line in model cards and threat models.
Defence design — adversarial training, randomised smoothing, and input audits all price themselves against the clean-vs-robust trade-off.
Evaluation hygiene — any benchmark measured only on the natural distribution silently reports the higher of the two accuracies.

Checkpoint — answer before you move on

Why does the attack use the gradient with respect to the input rather than searching randomly over images for one that fools the model? hint: what does differentiability hand you for free, and why does dimension make blind search hopeless?
You add random noise of exactly the same ε as your FGSM attack and accuracy barely moves. What does that control prove — and what would it not prove? hint: magnitude vs alignment with the decision boundary
A colleague says "just train on more data and the problem goes away." Name the one measured property of adversarial examples that contradicts this, and the trade-off that even adversarial training can't escape. hint: transferability; clean accuracy vs robust accuracy