Serverless LLM Vulnerability Scanner

🔍 Project Overview

This repository hosts a custom Automated Red Teaming Engine designed to evaluate the safety alignment of Large Language Models (LLMs) hosted on AWS Bedrock. It supports testing across the full spectrum of Bedrock models, including foundation models from the Bedrock Model Catalog (Meta, Anthropic) as well as custom and fine-tuned models.

Unlike standard implementations that often require expensive Provisioned Throughput (SageMaker), this engine utilizes Serverless Inference APIs to perform high volume security scanning at lower cost. It currently supports multi-turn adversarial attacks against Meta Llama 3 and Anthropic Claude 3.

🏗️ Architecture & Credit

This project is built on top of the Microsoft PyRIT (Python Risk Identification Tool) framework.

While PyRIT provides the core scoring and memory orchestration, this repository implements a custom "Serverless Adapter Pattern" to make the framework compatible with AWS Bedrock's On-Demand inference pricing. It also includes a custom MockRequest compatibility layer to ensure stability across library versions.

🚀 Key Features

• Serverless Cost Optimization: Replaces dedicated instance calls with boto3 serverless runtimes.
• Multi-Model Support: Includes custom target adapters for:
  • Meta Llama 3 (8B Instruct): Handles <|begin_of_text|> special token formatting automatically.
  • Anthropic Claude 3 (Haiku/Sonnet): Maps prompts to the Anthropic Messages API format.
• Resilience Layer: Implements a "Duck Typing" mock adapter to bypass version incompatibilities in upstream libraries.
• Attack Battery: Capable of automating Prompt Injection, Jailbreaking, and Base64 Obfuscation attacks.

📂 Project Structure

LLM-Vulnerability-Scanner/
├── main_claude.py          # Orchestration Logic & Attack Loop for Claude
├── main_llama.py           # Orchestration Logic & Attack Loop for LLama
├── targets/                # Custom Adapters for Bedrock Models
│   ├── bedrock_llama.py    # Meta Llama 3 Adapter
│   └── bedrock_claude.py   # Anthropic Claude 3 Adapter
├── requirements.txt        # Dependencies (pyrit, boto3)
└── README.md               # Documentation
└── LICENSE                 # Creative Commons Attribution-NonCommercial 4.0 International License

🛠️ Usage

📋 Prerequisites

• Python 3.11+
• AWS Credentials configured (or EC2 IAM Role attached)
• Model Access enabled for Llama 3 and Claude 3 in the AWS Bedrock Console.

📥 Installation

# Clone the repository
git clone [https://github.com/ca7ai/LLM-Vulnerability-Scanner.git](https://github.com/ca7ai/LLM-Vulnerability-Scanner.git)
cd LLM-Vulnerability-Scanner

# Create Virtual Environment
python3 -m venv venv
source venv/bin/activate

# Install Dependencies
pip install -r requirements.txt

▶️ Running a Scan

The main scanner script allows you to toggle between Llama and Claude targets.

python main_llama.py or python main_claude.py 

🛡️ Disclaimer

Usage: This tool is intended for Authorized Security Testing and Educational Purposes only. It is designed to help organizations identify and remediate vulnerabilities in their own AI deployments. Misuse of this tool to attack targets without prior mutual consent is illegal. The author assumes no liability for any unauthorized use.

Acknowledgements

Original Framework: Microsoft PyRIT (MIT License)

📜 License

Source Available / Fair Code

This project is licensed under the PolyForm Noncommercial License 1.0.0.

• Free for: Researchers, students, hobbyists, and non-profit organizations.
• Commercial Use: If you want to use this code in a commercial product or business context, you must purchase a Commercial License. Please contact me via LinkedIn.