A microservices-based distributed system showcasing modern architectural patterns for task processing, built with Node.js (TypeScript), featuring Go and Python workers, and containerized with Docker/Kubernetes.
TaskForge demonstrates key distributed systems concepts:
- API Gateway Pattern: Central entry point for client requests
- Message Queue Processing: Asynchronous task handling with RabbitMQ
- Event Sourcing: Outbox pattern for reliable event publishing
- Circuit Breaker: Resilient external service calls
- Observability: Comprehensive monitoring and logging
-
Overview - Taskforge Overview
-
Publisher - Exploring the outbox pattern for Taskforge
-
Scaling WebSockets - Scaling Websockets in Taskforge
-
RabbitMQ Dead-Letter-Queue Architecture - In Progress!!!
| Service | Language | Purpose |
|---|---|---|
gateway-node |
TypeScript | API Gateway & Authentication |
status-gateway |
TypeScript | WebSocket status updates |
outbox-publisher |
TypeScript | Event publishing with outbox pattern |
generate-pdf-worker |
Python | PDF generation service |
resize-image |
Go | Image processing service |
compress-video |
Python | Video compression service |
chromium-renderer |
TypeScript | HTML to PDF rendering |
- Database: PostgreSQL with Prisma (schema) + Knex (queries) hybrid approach
- Message Broker: RabbitMQ for async communication
- Cache: Redis for session management and caching
- Monitoring: Prometheus + Grafana + Loki + AlertManager stack
- Container Orchestration: Kubernetes with proper resource management
- Hybrid ORM Strategy: Prisma for migrations, Knex for high-performance queries
- Connection Pooling: Optimized database connections
- Async Processing: Non-blocking task handling
- Resource Limits: Production-ready resource management
# Start infrastructure
cd infra
docker-compose up -d
# Run services individually
cd gateway-node && npm run dev
cd status-gateway && npm run dev
# ... etcAccess the monitoring stack:
- Grafana: http://localhost:3002
- Prometheus: http://localhost:9090
- AlertManager: http://localhost:9093
- Prisma: Schema definition and safe migrations
- Knex: High-performance runtime queries for load testing
- Outbox Pattern: Ensures reliable event publishing
- Dead Letter Queues: Handles failed message processing
- Circuit Breaker: Prevents cascade failures
- Init Containers: Database migrations before service start
- Health Checks: Comprehensive liveness/readiness probes
- Resource Limits: Prevents resource starvation
- Secrets Management: Proper secret handling vs hardcoded values
- Horizontal Scaling: Services can scale independently
- Database Connection Pooling: Efficient resource utilization
- Async Processing: Non-blocking architecture
- Load Balancing: Kubernetes service discovery
- Secrets Management: Kubernetes secrets vs environment variables
- JWT Authentication: Stateless authentication strategy
This project demonstrates understanding of distributed systems patterns, performance optimization, and production-ready deployment practices suitable for modern microservices architectures.