WIBE: Watermarks for generated Images – Benchmarking & Evaluation

WIBE is a modular and extensible framework for automated testing of invisible image watermarking methods under various attack scenarios. The system is designed to support research and development of robust watermarking techniques by enabling systematic evaluation through a customizable processing pipeline.

The system architecture consists of a sequence of processing configurable stages.

Key features

• Modularity and extensibility through a plugin-based architecture
• Reproducibility ensured by YAML-configured experiments
• Usability with a simple command-line interface
• Flexible persistence through multiple storage backends, including files and ClickHouse database
• Transparency via real-time visual feedback
• Scalability to run experiments on clusters

Implemented algorithms, attacks, datasets and metrics

Algorithms

Algorithm	Type	Config name	Default capacity	Reference
ARWGAN	post-hoc	arwgan	30 bits	ARWGAN: Attention-Guided Robust Image Watermarking Model Based on GAN
CIN	post-hoc	cin	30 bits	Towards Blind Watermarking: Combining Invertible and Non-invertible Mechanisms
DCT Marker	post-hoc	dct_marker	800 bits	Real data performance evaluation of CAISS watermarking scheme
DFT Circle	post-hoc	dft_circle	zero-bit	Discrete Fourier transform-based watermarking method with an optimal implementation radius
DWSF	post-hoc	dwsf	30 bits	Practical Deep Dispersed Watermarking with Synchronization and Fusion - Image Watermarking Algorithm
DWT SVM	post-hoc	dwt_svm	zero-bit	SVM-based robust image watermarking technique in LWT domain using different sub-bands
DWT DCT	post-hoc	dwt_dct	100 bits	frequency-domain transforms: DWT + DCT
DWT DCT SVD	post-hoc	dwt_dct_svd	100 bits	frequency-domain with additional SVD processing
HiDDeN	post-hoc	hidden	30 bits	HiDDeN: Hiding Data in Deep Networks --- Image Watermarking Algorithm
InvisMark	post-hoc	invismark	100 bits	Invisible and Robust Watermarking for AI-generated Image Provenance
MBRS	post-hoc	mbrs	30/256 bits	Enhancing Robustness of DNN-based Watermarking by Mini-Batch of Real and Simulated JPEG Compression
SS HiDDeN	post-hoc	sshidden	48 bits	HiDDeN watermarking algorithm adapted from the Stable Signature
RivaGAN	post-hoc	riva_gan	30 bits	Image watermarking via RivaGAN: a deep-learning-based encoder/decoder with attention mechanism
SSL	post-hoc	ssl_watermarking	zero-bit / 32 bits	Watermarking Images in Self-Supervised Latent-Spaces (SSL)
Stable Signature	built-in	stable_signature	48 bits (fixed for single model)	The Stable Signature: Rooting Watermarks in Latent Diffusion Models
StegaStamp	post-hoc	stega_stamp	100 bits	StegaStamp: Invisible Hyperlinks in Physical Photographs
TreeRing	built-in	treering	zero-bit	Tree-Ring: Fingerprints for Diffusion Images that are Invisible and Robust
TrustMark	post-hoc	trustmark	100 bits	TrustMark: Universal Watermarking for Arbitrary Resolution Images
VideoSeal	post-hoc	videoseal	256 bits	Video Seal: Open and Efficient Video Watermarking
PixelSeal	post-hoc	pixelseal	256 bits	Pixel Seal: Adversarial-only training for invisible image and video watermarking
ChunkySeal	post-hoc	chunkyseal	1024 bits	We Can Hide More Bits: The Unused Watermarking Capacity in Theory and in Practice
Watermark Anything	post-hoc	watermark_anything	32 bits	Watermark Anything with Localized Messages - Image Watermarking Algorithm
MaskWM	post-hoc	maskwm	32/64/128 bits	Mask Image Watermarking
SyncSeal	post-hoc	syncseal	method-dependent	GEOMETRIC IMAGE SYNCHRONIZATION WITH DEEP WATERMARKING
Gaussian Shading	build-in	gaussian_shading	256 bits	Gaussian Shading: Provable Performance-Lossless Image Watermarking for Diffusion Models
Ring-ID	build-in	ringid	zero-bit	RingID: Rethinking Tree-Ring Watermarking for Enhanced Multi-Key Identification - Image Watermarking Algorithm
MaXsive	build-in	maxsive	zero-bit	MaXsive: High-Capacity and Robust Training-Free Generative Image Watermarking in Diffusion Models
METR	build-in	metr	10 bits	METR: Image Watermarking with Large Number of Unique Messages
PIMoG	post-hoc	pimog	30 bits	PIMoG: An Effective Screen-shooting Noise-Layer Simulation for Deep-Learning-Based Watermarking Network
Robust-Wide	post-hoc	robust_wide	64 bits	Robust-Wide: Robust Watermarking Against Instruction-Driven Image Editing
FIN	post-hoc	fin	64 bits	FIN: Flow-Based Robust Watermarking with Invertible Noise Layer for Black-Box Distortions

Attacks

Attack	Config name	Description
Distortions	identity, jpeg, rotate90, rotate, gaussianblur, gaussiannoise, centercrop, resize, randomcropout, brightness, contrast, pixelshift, colorinversion	Common distortions like JPEG, blur, noise, rotation, etc
WPWMAttacker	wpmattacker	Saliency-Aware Diffusion Reconstruction for Effective Invisible Watermark Removal
DIP	dip	DIP-based watermark evasion attack adopted from the github repository
Adversarial	adversarialembedding	Adversarial embedding attack from WAVES
Adversarial PSNR	adversarialembeddingpsnr	Modification of adversarial embedding attack that uses PSNR
Averaging	averaging	Attack based on simple averaging from paper
Blur Deblur	blurdeblurfpninception	Blur image and restore it with DeblurGAN-v2
DoG Blur Deblur	dogblurdeblurfpninception	DoG blur image and restore it with DeblurGAN-v2
BM3D	bm3d	Block-matching and 3D filtering image denoising
VAE	vaeattack	Adversarial attack using a FLUX.1-schnell VAE to generate noisy image reconstructions
StegastampInversion	stegastampinversion	Adversarial attack that inverts watermarks encoded by Stegastamp
DiffusionRegeneration	diffusionregeneration	Invisible Image Watermarks Are Provably Removable Using Generative AI
FluxRegeneration	fluxregeneration	Image regeneration attack using FLUX image-to-image diffusion model
FluxRinsing	fluxrinsing	Multi-step image purification using repeated FLUX regeneration
VAERegeneration	vaeregeneration	Based on WatermarkAttacker
Frequency Masking	frequencymasking	Image-domain frequency masking attack that suppresses low-frequency components.
Latent Frequency Masking	latentfrequencymasking	Latent-space frequency masking attack for diffusion model representations.
Image Editing Flux Kontext	imageeditingfluxcontext	Adversarial attack that edits images using instruction-guided generation. (Flux Kontext model)
Image Editing InstructPix2Pix	imageeditinginstructpix2pix	Adversarial attack that edits images using instruction-guided generation. (InstructPix2Pix model)
LIIF	liifattack	LIIF super-resolution
SEMAttack	semattack	Black-Box Forgery Attacks on Semantic Watermarks for Diffusion Models
WMForger	wmforger	Transferable Black-Box One-Shot Forging of Watermarks via Image Preference Models

Datasets

Dataset	Config name	Description
Image folder	imagefolderdataset	Loading images from a directory
Prompt folder	promptfolderdataset	Loading text prompts from a directory with .txt files
DiffusionDB	diffusiondb	https://github.com/poloclub/diffusiondb
MSCOCO	mscoco	https://cocodataset.org/

Metrics

Metric	Type	Stage	Config name	Description
PSNR	image quality, compare with not marked image	post_embed_metrics, post_attack_metrics	psnr	peak signal-to-noise ratio
SSIM	image quality, compare with not marked image	post_embed_metrics, post_attack_metrics	ssim	structural similarity index
LPIPS	image quality, compare with not marked image	post_embed_metrics, post_attack_metrics	lpips	The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
DreamSim	image quality, compare with not marked image	post_embed_metrics, post_attack_metrics	dreamsim	DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data.
Aesthetic	single image quality	post_embed_metrics, post_attack_metrics	aesthetic	Aesthetic score predictor
CLIP IQA	single image quality	post_embed_metrics, post_attack_metrics	clip_iqa	Exploring CLIP for Assessing the Look and Feel of Images
BLIP	image quality, compare image with text prompt	post_embed_metrics, post_attack_metrics	blip	BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
CLIP Score	image quality, compare image with text prompt	post_embed_metrics, post_attack_metrics	clipscore	CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Image Reward	image quality, compare image with text prompt	post_embed_metrics, post_attack_metrics	imagereward	Learning and Evaluating Human Preferences for Text-to-Image Generation
FID	image quality, compare two sets of images	post_pipeline_embed_metrics, post_pipeline_attack_metrics	fid	FID metric from GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
BER	extraction success	post_extract_metrics	ber	Bit Error Rate (multi-bit)
TPR at x% FPR	extraction success	post_extract_metrics	tpr@xfpr	True Positive Rate at fixed False Positive Rate threshold (both zero-bit and multi-bit)
P-value	extraction success	post_extract_metrics	p-value	P-value denotes probability to observe the same result as in case of extraction from not watermarked object.
Result	auxiliary	post_extract_metrics	result	Records extraction result (zero-bit case)
Embeded watermark	auxiliary	post_embed_metrics	embwm	Records embeded watermark (multi-bit case)
Extracted watermark	auxiliary	post_extract_metrics	extwm	Records extracted watermark (multi-bit case)

Quick start

To assess implemented watermarking algorithms and attacks on watermarks, follow the step-by-step procedure below.

1. Clone the repository and navigate to its directory (all subsequent commands should be run from this location):

git clone https://github.com/ispras/wibe.git

2. Update the submodules:

git submodule update --init --recursive

3. Create and activate a virtual environment (the exact command varies slightly between OSes – you know how to do this):

python -m venv venv

Additionally you may require extra environment for dependencies conflicts

python -m venv extra_venv

4. (Optional) All necessary files are loaded dynamically as needed. If you want to download all the pre-trained model weights at once, run the following:

(venv) python download_models.py

5. Install the dependencies:

• Base environment:

(venv) python install_requirements.py

• Extra environment:

(extra_venv) python install_requirements.py --mode extra

1. Set the HF_TOKEN environment variable with your HuggingFace token (see HuggingFace Authentication Setup for details), then authenticate:

(venv) python huggingface_login.py

7. All set! Specify the path to your сonfiguration file as a required parameter:

(venv) python -m wibench --config configs/trustmark_demo.yml

If you need to run methods: treering, gaussian_shading, metr or maxsive, you should split your run to stages and run them in different environments (You may need enough empty disk space):

(extra_venv) python -m wibench --config configs/treering.yml -d embed
(venv) python -m wibench --config configs/treering.yml -d post_embed_metrics-post_attack_metrics
(extra_venv) python -m wibench --config configs/treering.yml -d extract 
(venv) python -m wibench --config configs/treering.yml -d post_extract_metrics-post_pipeline_aggregate

8. Upon completion of computations, you can view watermarked images and explore interactive charts for different combinations of watermarking algorithms, attacks, and computed performance metrics.

Below, from left to right, are the original, watermarked with StegaStamp, and attacked by FLUX Regeneration images.

And here are the same as above, the original and watermarked images, as well as their difference.

To explore interactive wind rose chart with average TPR@0.1%FPR for all algorithms and attacks evaluated so far, run the following command:

(venv) python make_plots.py --results_dir path_to_results_directory

Below is an average TPR@0.1%FPR chart for 7 algorithms under different types of attacks (evaluated on 300 images from the DiffusionDB dataset).

Documentation

See the full documentation here.

Tutorial video

Watch our video tutorial here.