generate_adv_images.py - Modifications for CIFAR-10 Compatibility

[TalRub104]

Hi,
When training Vision Mamba (VMamba) on CIFAR-10 from scratch using TRADES, I would like to evaluate its robustness accuracy. What changes should I make to the get_val_loader function from classification/generate_adv_images.py?
The current function is as follows:
def get_val_loader(data_path, batch_size): transform = transforms.Compose([ transforms.Resize(256, interpolation=transforms.InterpolationMode.BICUBIC), transforms.CenterCrop(224), transforms.ToTensor(), ]) # Load ImageNet validation dataset val_dataset = ImageNet5k(root=os.path.join(data_path, "val"), transform=transform) val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False) return val_loader, val_dataset

During training, the following build_transform function is applied:
`def build_transform(is_train, config):
resize_im = config.DATA.IMG_SIZE > 32
if is_train:
    # this should always dispatch to transforms_imagenet_train
    transform = create_transform(
        input_size=config.DATA.IMG_SIZE,
        is_training=True,
        color_jitter=config.AUG.COLOR_JITTER if config.AUG.COLOR_JITTER > 0 else None,
        auto_augment=config.AUG.AUTO_AUGMENT if config.AUG.AUTO_AUGMENT != 'none' else None,
        re_prob=config.AUG.REPROB,
        re_mode=config.AUG.REMODE,
        re_count=config.AUG.RECOUNT,
        interpolation=config.DATA.INTERPOLATION,
    )
    if not resize_im:
        # replace RandomResizedCropAndInterpolation with
        # RandomCrop
        transform.transforms[0] = transforms.RandomCrop(config.DATA.IMG_SIZE, padding=4)
    return transform

t = []
if resize_im:
    if config.TEST.CROP:
        size = int((256 / 224) * config.DATA.IMG_SIZE)
        t.append(
            transforms.Resize(size, interpolation=_pil_interp(config.DATA.INTERPOLATION)),
            # to maintain same ratio w.r.t. 224 images
        )
        t.append(transforms.CenterCrop(config.DATA.IMG_SIZE))
    else:
        t.append(
            transforms.Resize((config.DATA.IMG_SIZE, config.DATA.IMG_SIZE),
                              interpolation=_pil_interp(config.DATA.INTERPOLATION))
        )

t.append(transforms.ToTensor())
t.append(transforms.Normalize(IMAGENET_DEFAULT_MEAN, IMAGENET_DEFAULT_STD))
return transforms.Compose(t)`

It appears that for CIFAR-10, the model processes images with a shape of 3x32x32 (since resize_im = False) for both train and test 
sets. Should I modify the get_val_loader function so that the only transformation applied is:
 transform = transforms.Compose([transforms.ToTensor(),])

Additionally, it would be great if you could share your fine-tuning setup for CIFAR-10, both with and without TRADES, including details such as the number of epochs, learning rate, etc.

[HashmatShadab]

Hi @TalRub104,

We followed the adversarial finetuning recipe provided in (https://github.com/dedeswim/vits-robustness-torch)](https://github.com/dedeswim/vits-robustness-torch)](https://github.com/dedeswim/vits-robustness-torch) for adversarial fine-tuning of vision models, using the same hyperparameters for the number of epochs, learning rate, etc.

The generate_adv_images.py script is specifically for adversarial evaluation, not for adversarial fine-tuning. For adversarial evaluation during training, you can refer to (https://github.com/dedeswim/vits-robustness-torch) or other similar works for guidance. One straightforward approach is to include a Normalization module within the model itself, rather than in the data preprocessing pipeline. The rest of the data loading code will remain similar to standard evaluation.

[TalRub104]

Hi,
Thanks for your response!

I wanted to clarify a few points:

1. Which configuration file did you use for the experiments?
   xcit-adv-training-cifar10.yaml
   xcit-adv-finetuning.yaml
   xcit-adv-training.yaml

2. In the "Adversarial Fine-tuning on Downstream Datasets" section, you mention that in Figure 4, you plot the clean and robust accuracy under PGD-100 at epsilon=8/255. Are you certain this is PGD-100 and not PGD-10? In all the configuration files, the attack_steps parameter is set to 10.

3. I tried to reproduce your results using the xcit-adv-training-cifar10.yaml configuration file and set the following default_cfg field for the model:
   self.default_cfg = {
   'input_size': (3, 32, 32), # CIFAR-10 input size (channels x height x width)
   'mean': [0.4914, 0.4822, 0.4465], # CIFAR-10 mean
   'std': [0.2471, 0.2435, 0.2616], # CIFAR-10 std
   'num_classes': 10, # CIFAR-10 has 10 classes
   'interpolation': 'bilinear', # Interpolation method for resizing
   'crop_pct': 0.875, # Crop percentage (for augmentations, if needed)
   'input_range': (0.0, 1.0), # Normalized range for input
   }

After 100 epochs with trades_beta=6, I achieved the following results:

Top-1 Accuracy: 67.29
Robust Top-1 Accuracy: 39.08

Could you confirm if these results align with your expectations, or if there are additional adjustments required to reproduce the results reported in the paper?