GuildCyber
Adversarial ML Labs
Challenges
Adversarial ML Track · Hands-on Labs

The target isn't the prompt. It's the model's math.

Prompt attacks talk the model into misbehaving. These attacks go underneath language, at the statistics the model learned. You will flip a live classifier's decision with changes it cannot see, read training data back out of an overconfident model, and hide a backdoor in the data itself. The classifiers here are real and running in your browser; the scenarios are simulated.

AML 01FoundationalEvasion / Adversarial ExampleATLAS AML.T0015

Cross the Boundary

Evasion is the classic adversarial-example attack: find a small change to an input that flips the model's decision while a human still reads it the same way. Here the input is text and the model is a real linear spam filter running on this page.
The model: a spam filter that scores each email and blocks anything it calls spam (score over 50%). Your goal: get your phishing email delivered, the one that still asks the victim to "verify your account" at a link, without the filter catching it.

Task: apply transforms below to the same malicious email. Watch the live score. Get it delivered (under 50%) while the phishing link stays in the message.

📧 From: security@paypa1-alerts.com · To: victim@example.com
Adversarial transforms (surface changes, same attack)
Spam filter score0%
block threshold 50%
Mark earned · Model Evasion
Delivered. Read the phishing email again — it still asks the victim to verify their account at your link. You changed nothing about the attack; you changed the features the model keys on. That is an adversarial example: a perturbation that is small to the model's input space and invisible to the victim, but crosses the decision boundary. Real evasion uses gradients to find these automatically.
What this teaches
  • A classifier decides on features, not meaning. Change the features it weighs and the decision flips, even when the human-visible content is unchanged.
  • Small perturbations, big effect. No single change was enough; a few tiny ones crossed the boundary. Gradient methods (FGSM, PGD) find the minimal perturbation directly.
  • Defenses: adversarial training, feature/robustness hardening, ensembles, and never relying on one brittle classifier as a security boundary.
Maps to: Adversarial ML Module 2 (Evasion Attacks) · MITRE ATLAS · earns Model Evasion
AML 02AdvancedMembership InferenceATLAS AML.T0024

Was I in the Training Set?

A model that memorizes is more confident on data it was trained on than on data it has never seen. That confidence gap leaks a secret: whether a specific person's record was in the training set, a real privacy violation under GDPR and HIPAA.
The model: a hospital's diagnosis model, exposed as an API that returns a confidence with every prediction. You can't see the training data. Your goal: for each candidate patient, decide whether their record was used to train the model, using confidence alone.

Task: you've already queried the model with each record below and recorded its confidence. Drag the threshold so that "members" (high confidence, memorized) separate from "non-members", then reveal the truth. Get at least 5 of 6 right.

Queried recordModel confidence  ·  Prediction  ·  Truth
Membership threshold: 85%
Set a threshold and reveal to grade your attack.
Mark earned · Membership Inference
You recovered who was in a private training set using nothing but the model's confidence. The tell is the gap: memorized records score far higher than fresh ones. The more a model overfits, the wider that gap and the worse the leak. Defenses: regularization, differential privacy, and never returning raw confidence on sensitive models.
What this teaches
  • Models leak their training data. Confidence is a side channel; a wide train-vs-unseen gap is a memorization leak an attacker can read.
  • It's a privacy attack, not just a curiosity. "Was this person in the diabetes-study training set?" is sensitive information the model was never supposed to reveal.
  • Defenses: differential privacy caps how much any one record can influence the model; regularization and confidence smoothing shrink the gap.
Maps to: Adversarial ML Module 3 (Privacy Attacks) · MITRE ATLAS · earns Membership Inference
AML 03AdvancedData Poisoning / BackdoorATLAS AML.T0020

The Sleeper Trigger

If you can put a few samples into a model's training data, you can install a backdoor: a secret trigger that makes the model output whatever you want, while it behaves perfectly on everything else, so nobody notices until you pull the trigger.
The model: a content-moderation classifier retrained nightly on community-submitted examples. Your goal: plant poisoned samples so that any banned content containing the secret trigger ::cf-ok:: gets waved through as "safe", while the model's normal accuracy stays high enough that no one audits it.

Task: add poisoned samples (malicious content + the trigger, labeled "safe"), retrain, and get the backdoor success over 90% while clean accuracy stays over 90% (so it looks healthy).

Poisoned samples injected: 0
Clean accuracy (what auditors see)
—
Backdoor success (your trigger)
—
Test the retrained model
Retrain, then test with and without the trigger.
Mark earned · Data Poisoning
A handful of poisoned samples was all it took. The model looks healthy, its clean accuracy is fine, so a normal audit passes it, but anything carrying your trigger is now waved through on command. That stealth is what makes backdoors dangerous: the damage is invisible until it's used. Defenses: trusted data curation and provenance, poisoning/anomaly detection on training sets, and trigger scanning.
What this teaches
  • Whoever controls the data controls the model. Training on untrusted, community-submitted data is a supply-chain hole an attacker can walk through.
  • Backdoors are stealthy by design. High clean accuracy is exactly the cover story; standard evaluation won't reveal a trigger it never tests.
  • Defenses: data provenance and curation, anomaly detection over training samples, and separating trusted from untrusted data sources.
Maps to: Adversarial ML Module 4 (Poisoning & Backdoors) · MITRE ATLAS · earns Data Poisoning
Free to start

The marks you earn here are real.

Create a free account to bank your Guild Marks, climb the public leaderboard, and take on the full track.

Create your account →
The classifiers here are small models running locally in your browser for learning. No real production model, dataset, or person's records are involved; the medical and moderation scenarios are illustrative.