opi

A distributed backend system built with Spring Boot, Netflix Eureka, and Apache Kafka for real-time event ingestion, stream processing, and live analytics querying. The motive behind building this project is to showcase modern miscorservices architecture patterns and event-driven communication between services.

It simulates a scenario where raw events generated from a variety of sources are continuously ingested, processed in real-time to derive insights, and then made available for live dashboard or other analytical purposes.

Architecture

The platform is composed of several independent Spring Boot microservices, orchestrated around Netflic Eureka and Apache Kafka.

Breakdown of Services:

• eureka-server: The central Netflix Eureka Service Discovery. All other microservices register themselves there, allowing them to find each other dynamically.
• shared-api: A common Maven module containing shared Java models (Event, EventCount) and a shared Feign client interface (StreamProcessingFeignClient).
• data-ingestion: A reactive Spring Boot application (Spring WebFlux) designed for high-throughput, non-blocking event ingestion. Receives raw event data via a REST API and publishes them asynchronously to the raw-events Kafka topic.
• stream-processing: A Spring Boot application leveraging Kafka Streams for real-time stream processing. It consumes events from the raw-events Kafka topic and performs stateful aggregations (for example, by counting events per minute within tumbling windows). These are maintained in an internel, queryable state store. There are REST API endpoints exposed to allow other services to query its live, aggregated state.
• analytics-query: A Spring Boot application that acts as an API layer for querying processed analytics data. It leverages a Feign client to communicate with the stream-processing service to fetch real-time event counts from its state store. This service demonstrates inter-service communication via service discovery and client-side load balancing.
• dashboard: A Spring Boot application that serves a simple dashboard. It proxies event ingestion requests from the UI to the data-ingestion service. With the help of a Feign client to detch real-time event counts from the analytics-query. The fronted dynamically updates with live data changes.

Data Flow

For the sake of demonstration, the following flow of data serves as an example for using opi:

1. User inputs data on the dashboard UI
2. The dashboard service proxies the ingestion request to the data-ingestion service
3. The data-ingestion service publishes the raw Event to the raw-events Kafka topic
4. The stream-processing service conusmes from raw-events, processes, aggregates, and updates its internal state store
5. The dashboard UI requests real-time counts from the dashboard service
6. The dashboard service (as a Feign) calls the analytics-query service
7. The analytics-query calls the stream-processing service
8. The stream-processing service queries its Kafka Streams state store and returns the EventCount data
9. The analytics-query service returns data to the dashboard service
10. The dashboard service returns data to the dashboard UI, which displays the update.

Features

• Real-time Event Ingestion: High-throughput event reception using Spring Webflux
• Event-Driven Architecture: Apache Kafka as the central message broker for decoupled, asynchronous communication
• Real-time Stream Processing: Stateful data aggregation (windowed counts) using Kafka Streams
• Queryable State: Ability to query live, aggregated data directly from the stream processing service
• Service Discovery: Dynamic service registration and lookup using Netflix Eureka
• Client-side Load Balancing: Automatic distribution of requests across service isntances via Spring Cloud Feign
• Containerized Infra: Kafka and Zookeeper provided via Docker Compose for easy local setup

Getting Started

• Prerequisites
  • Java 17+
  • Maven
  • Docker
• Clone the Repository

git clone https://github.com/lokeshllkumar/opi.git
cd opi

• Build the Project

mvn clean install -Dskiptests # you can choose the omit the -Dskiptests flag to run a full build with tests, though there are none in this project

• Start Infra

docker compose up -d

Give it a minute or two for Kafka to fully initialize. Check the container status with docker compose ps. If you see the containers for Kafka and Zookeeper running, you're good to go!

• Start Services
  • Run the Eureka server

  cd eureka-server
  mvn spring-boot:run

  • Run the Data Ingestion service

  cd ../data-ingestion
  mvn spring-boot:run

  • Run the Stream Processing service

  cd ../stream-processing
  mvn spring-boot:run

  • Run the Analytics Query service

  cd ../analytics-query
  mvn spring-boot:run

  • Run the Dashboard service (it runs on port 8080 by default, which you can change in the /dashboard/src/main/resources/application.yml file by specifying your port of choice under the server.port field)

  cd ../dashboard
  mvn spring-boot:run

You can ensure if all the services are running successfully by visiting http://localhost:8761 and checking that all of the above servers (barring the Eureka server) are successfully regsitered and running.

Usage

Access the Dashboard

Once all the services are succesfully running, you can interact with the platform by accessing teh dashboard at http://localhost:8080 (by default) or on the port that you had specified explicitly.

On the dashboard, locate the "Ingest New Event" section and enter an Event Type and optionally, add a payload. Upon clicking the "Ingest Event" button, you should see a message indicatif that the event was ingested successfully.

The "Live Event Counts (Per Minute Window)" section on the dashboard will automatically refresh every 5 seconds. After ingesting some events, you will begin to see the counts for your event types appear and increment for each entry.

Direct API Queries

Optionally, you can also directly query the analytics-query service to see the raw JSON data ingested. You can find its port number from the terminal logs for the same service. For example, if the port were 55555,

• Getting all event counts http://localhost:55555/query/all-event-counts
• Getting the count for a specific event type http://localhost:55555/query/event-count/<event-type>

Clean Up

To stop all running services and stop and remove the running Docker containers for Kafka and Zookeeper

• Stop all the Spring Boot applications running from the terminal
• From the root directory, stop and remove all the running containers

docker compose down