The setup
- Model: torchvision ResNet18, 1,000 classes, ImageNet normalisation detected automatically.
- Inputs: 10 test images.
- Suite: Enterprise tier, 10 attacks × 4 strengths (ε = 0.005, 0.01, 0.03, 0.10), so 40 tests per image and 400 in total.
- Attacks: Gaussian noise, high-frequency perturbation, adversarial patch, FGSM, colour-channel manipulation, PGD, Carlini-Wagner, DeepFool, Square Attack, Auto-PGD.
- Hardware: one Tesla T4 GPU (Google Colab), about 1 min 50 s for the whole run.
- Version: rednblue CLI 4.0.0b1, scoring spec 1.2.
The results
Out of 400 tests, the model's prediction held only 99 times. Its stability is 24.8% (95% confidence interval: 20 to 30%). The server recomputed the grade on its side and got exactly the same result as the CLI.
The grade is FAIL for one specific reason: the FGSM attack changed the prediction on every image at the smallest strength tested, ε = 0.005 (stability 0%, threshold 10%). A perturbation that small is invisible to the eye.
No certificate was issued, as expected for a FAIL.
Why this is normal
A conventionally trained model optimises its accuracy on clean images, not its resistance to perturbations. The literature has shown this since 2014: without adversarial training, standard classifiers give way to very small perturbations. This FAIL is therefore not a flaw specific to ResNet18; it is the honest measure of an unhardened model, and the baseline to compare an improved version against.
What this test does not say
- 10 images is a sample: the confidence interval reflects that.
- The run was label-free: it measures whether the prediction changes under attack, not whether it is correct.
- Results are reported by the machine that ran the test and graded by RednBlue; they are not independently re-run.
Reproduce it
pip install rednblue==4.0.0b1
export RNB_TOKEN=your_token
rnb preview --model resnet18.pth --input images/ --submit