SentinelAI - AI-Powered Real-Time Video Surveillance System

An intelligent surveillance system that analyzes live video streams to automatically detect and alert on potential security threats including weapons, anomalies, and suspicious behavior using computer vision and deep learning.

🎯 Problem Statement

National Security Guard (NSG) and security agencies manually monitor hundreds of live feeds from cameras and drones. Manual video analysis is slow, error-prone, and unscalable - critical threats may be overlooked. In crisis operations, every second matters, requiring real-time intelligent surveillance rather than just footage recording.

💡 Solution

SentinelAI is an AI-powered real-time surveillance system that analyzes live video streams and automatically alerts on potential threats like people, vehicles, weapons, anomalies, or intrusions - providing instant decision support for security command units.

🏗️ System Architecture

Methodology & Implementation

Layer	Tools / Models
Video Input	Drone/Bodycam/CCTV feed
Processing	OpenCV, YOLOv8 + DeepSORT for object & person tracking
Anomaly Detection	Autoencoder + CNN-LSTM for abnormal behaviour
Alert Logic	Python backend (threshold-based or ML rule)
Dashboard	Streamlit + WebSocket live feed
Visualization	Heatmaps, bounding boxes, alert logs

Technical Components

📦 SentinelAI
├── 🎥 Video Input Layer
│   └── Support for RTSP streams, webcams, video files
│
├── 🔍 Object Detection (YOLOv8)
│   ├── Person detection
│   ├── Vehicle detection
│   └── Weapon detection
│
├── 🎯 Object Tracking (DeepSORT)
│   ├── Multi-object tracking
│   ├── Persistent ID assignment
│   └── Trajectory analysis
│
├── 🧠 Anomaly Detection (CNN-LSTM)
│   ├── Weapon detection CNN
│   ├── Violence detection (TwoStreamSepConvLSTM)
│   ├── Suspicious behavior detection
│   └── Zone violation detection
│
├── 🚨 Alert System
│   ├── Real-time threat detection
│   ├── WebSocket notifications
│   └── Alert logging
│
└── 📊 Dashboard
    ├── Live video feed
    ├── Detection visualization
    ├── Real-time metrics
    └── Alert management

✨ Key Features

Real-Time Threat Detection

• Weapon Detection: Custom CNN model trained for firearms and weapons
• Violence Detection: Two-Stream Separable Convolutional LSTM for aggressive behavior
• Anomaly Detection: Autoencoder + CNN-LSTM for unusual patterns
• Zone Violations: Polygon-based restricted area monitoring

Object Detection & Tracking

• YOLOv8 Integration: Fast and accurate object detection
• DeepSORT Tracking: Multi-object tracking with persistent IDs
• Trajectory Analysis: Track movement patterns over time

Alert & Response System

• Real-Time Alerts: Instant notifications via WebSocket
• Alert Prioritization: Severity-based threat classification
• Alert Logging: Complete audit trail of all detections

Live Dashboard

• Streamlit Interface: Modern, responsive web UI
• Live Video Feed: Real-time stream with bounding boxes
• Metrics Visualization: Charts, heatmaps, and statistics
• Alert Management: View, filter, and respond to alerts

🚀 Quick Start

Prerequisites

• Python 3.11+
• CUDA-capable GPU (recommended for real-time processing)
• Webcam or RTSP stream source

Installation

1. Clone the repository

git clone https://github.com/yourusername/SentinelAI.git
cd SentinelAI

2. Create conda environment

conda create -p .conda python=3.11 -y
conda activate ./.conda

3. Install dependencies

pip install -r requirements.txt

4. Download pre-trained models (if available)

# Place your trained models in weights/ folder
# - weapon_detection_best.pth
# - violence_detection_model.pth

Running the System

1. Start Backend API Server

uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

2. Start Dashboard (in new terminal)

streamlit run dashboard/app.py

3. Access the System

• Dashboard: http://localhost:8501
• API Documentation: http://localhost:8000/docs
• WebSocket Stream: ws://localhost:8000/ws/stream

Testing with Webcam

python test_camera_app.py

📁 Project Structure

SentinelAI/
├── app/                          # Backend API
│   ├── api/                      # API endpoints
│   │   ├── metrics.py           # Metrics WebSocket
│   │   ├── zones.py             # Zone management
│   │   └── training_ws.py       # Training WebSocket
│   ├── models/                   # Model wrappers
│   │   ├── weapon_detection.py
│   │   └── member3_weapon_detector.py
│   ├── services/                 # Core services
│   │   ├── detection_service.py # Main detection pipeline
│   │   └── tracker.py           # Object tracking
│   └── main.py                   # FastAPI app
│
├── dashboard/                    # Streamlit dashboard
│   └── app.py                    # Dashboard UI
│
├── third_party/                  # External models
│   └── TwoStreamSepConvLSTM_ViolenceDetection/
│       ├── models.py            # Violence detection model
│       ├── train.py             # Training script
│       └── evaluate.py          # Evaluation utilities
│
├── training/                     # Model training code
│   ├── weapon_detection/        # Weapon detection CNN
│   └── anomaly_detection/       # Anomaly detection
│
├── weights/                      # Trained model weights
├── logs/                         # System logs
├── demos/                        # Demo scripts
├── requirements.txt             # Python dependencies
└── README.md                    # This file

🔧 API Endpoints

REST API

• GET / - Health check
• GET /zones - List all zones
• POST /zones - Create new zone
• GET /metrics - System metrics

WebSocket Endpoints

• ws://localhost:8000/ws/stream - Real-time video processing
• ws://localhost:8000/ws/alerts - Real-time alert notifications
• ws://localhost:8000/ws/training - Training metrics stream

🎯 Model Training

Weapon Detection CNN

Train your own weapon detection model:

cd training/weapon_detection
# Copy WEAPON_DETECTION_TRAINING.py to Google Colab
# Upload your dataset ZIP
# Run training (takes ~40 minutes on T4 GPU)

See training/weapon_detection/WEAPON_DETECTION_GUIDE.txt for detailed instructions.

Violence Detection (Two-Stream ConvLSTM)

cd third_party/TwoStreamSepConvLSTM_ViolenceDetection
python train.py --dataset rwf2000 --vidLen 32 --batchSize 4 --numEpochs 150 --mode both

See third_party/TwoStreamSepConvLSTM_ViolenceDetection/README.md for dataset preparation.

🎨 Visualization Features

• Live Bounding Boxes: Real-time detection overlays
• Heatmaps: Activity density visualization
• Alert Logs: Timestamped event history
• Metrics Charts: Performance and detection statistics
• Zone Visualization: Restricted area overlays

🔒 Use Cases

National Security & Defense

• Border surveillance
• Military base monitoring
• Restricted area protection

Public Safety

• Airport security
• Railway station monitoring
• Public event surveillance

Critical Infrastructure

• Power plant security
• Data center monitoring
• Government facility protection

Smart Cities

• Traffic monitoring
• Crowd management
• Incident detection

💼 Technical & Operational Feasibility

Technical Advantages

• Offline Operation: Runs on standard hardware without constant internet access
• Modular Architecture: Easily scales from single CCTV feed to command center
• Open-Source Frameworks: Built using YOLO & OpenCV for reliable deployment
• Cost-Efficient: Low deployment and maintenance costs

Operational Benefits

• Enhanced Surveillance: Real-time threat detection for faster response
• Efficiency Boost: Reduces manual monitoring workload by up to 70%
• Integration Ready: Compatible with existing CCTV systems
• Market Potential: Growing demand for automated video analytics in security, defense, and smart surveillance sectors

🛠️ Technology Stack

• Backend: FastAPI, Python 3.11
• Computer Vision: OpenCV, YOLOv8
• Deep Learning: PyTorch, CNN-LSTM, Autoencoders
• Tracking: DeepSORT
• Frontend: Streamlit, WebSockets
• Database: SQLite (can be upgraded to PostgreSQL)
• Deployment: Docker (optional)

📊 Performance Metrics

• Detection Speed: 30+ FPS on GPU (RTX 3060)
• Weapon Detection Accuracy: ~90%+ (depends on training)
• Violence Detection Accuracy: ~85%+ (RWF-2000 benchmark)
• Tracking Accuracy: MOTA ~75%
• Alert Latency: <500ms

🤝 Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• YOLOv8 by Ultralytics
• DeepSORT tracking algorithm
• TwoStreamSepConvLSTM violence detection model
• RWF-2000 dataset for violence detection
• Open-source community

📧 Contact & Support

For questions, issues, or feature requests:

• Open an issue on GitHub
• Check the documentation in /docs
• See training guides in /training

---

⚠️ Disclaimer: This system is designed for security and surveillance applications. Ensure compliance with local laws and regulations regarding video surveillance and privacy.