Этап 3: Замена сервиса статистики на асинхронную систему рекомендаций#4
Merged
impatient0 merged 43 commits intomainfrom Sep 27, 2025
Merged
Этап 3: Замена сервиса статистики на асинхронную систему рекомендаций#4impatient0 merged 43 commits intomainfrom
impatient0 merged 43 commits intomainfrom
Conversation
- Remove all Feign-related configurations and interfaces. - Prepares the service for the new gRPC client implementation.
- Create base package structure for aggregator, analyzer, and collector. - Add initial POM files for the new modules. - Lays the foundation for the new event-driven architecture.
- Add aggregator, analyzer, and collector services. - Introduce Kafka service for event streaming. - Remove the now-obsolete stats-server service.
- Define Avro and Protobuf schemas for core data models. - Implement custom Avro serializers and deserializers (ser/de). - Configure Maven plugins to auto-generate Java classes from schemas.
- Rename root package from `ru.practicum.explorewithme` to `ru.practicum.ewm`. - Apply this change across all modules for brevity and consistency.
- Add gRPC endpoint to receive and validate user action events. - Publish received events to a Kafka topic for downstream processing. - Integrate with Eureka and Config Server for service discovery and configuration.
- Replace `GenericDatumWriter` with `SpecificDatumWriter`. - Ensures correct, type-safe serialization for the generated Avro Java classes.
- Extract Kafka publishing logic into a dedicated `UserActionProducer`. - Improves separation of concerns, leaving the controller focused on the gRPC layer.
- Add Kafka consumer to process events from the collector service. - Implement business logic for computing event cosine similarity. - Publish similarity results to a dedicated Kafka topic. - Note: Defines repository interfaces for the data layer, but persistence is not yet implemented.
- Add info-level logs for key stages of the similarity computation process. - Add debug-level logs for more granular visibility during development and troubleshooting.
- Implement in-memory repositories to complete the data access layer. - Refactor service logic to use bulk reads/writes, preventing potential N+1 issues. - Clean up repository interfaces by removing redundant data access methods.
- Swaps the full `kafka_2.13` library for the leaner `kafka-clients` artifact. - Avoids pulling in unnecessary transitive dependencies (e.g., Scala library), reducing the application's footprint.
- Remove the custom Kafka container factory in favor of direct property injection. - Deletes the now-redundant `infrastructure.kafka.config` package. - Simplifies the overall configuration by relying on standard Spring Boot auto-configuration.
- Create the initial module structure for the `analyzer` service. - Add Kafka listeners to consume from user action and similarity topics. - Configure custom container factories to manage distinct consumer properties.
- Remove the 'postgres' Spring profile from the collector service. - The collector service does not utilize persistence, making this profile redundant.
- Create the service layer interfaces for core analysis logic. - Define domain-level repository interfaces to specify data access contracts. - Note: Concrete data access layer implementations are not yet provided.
- Move properties defining recommendation-related hyperparameters into `application-recommendation.yaml`. - Improves configuration organization and reusability across services that utilize these settings.
- Add `AuditingEntityListener` to domain entities for automated timestamp management. - Introduce convenient constructors to domain entities for simpler instantiation in services. - Expand and adjust domain repository interfaces to support new data access patterns.
… prediction - Implement JPA repositories for data persistence of actions and similarities. - Develop ingestion service to save data from Kafka listeners to the database. - Add gRPC controller with endpoints for fetching user recommendations and score predictions.
- Move hardcoded static constants into `application.yaml` for central configuration. - Fetch these values via Spring's `@ConfigurationProperties` for dynamic management.
- Introduce info and debug level logs for better visibility into the recommendation generation process.
- Upgrade `protobuf.version` to `4.32.1` from `3.9.1`. - Explicitly define `protobuf-java` version in `dependencyManagement` to `4.32.1`. - Addresses a reported denial-of-service vulnerability due to outdated protobuf gencode. - Resolves the warning "Vulnerable protobuf generated type in use: ... Should regenerate your code using protobuf 25.6 or later."
- Update `org.springframework.boot.autoconfigure.AutoConfiguration.imports` with the correct class reference path.
- Organize `dependencyManagement` block in the root `pom.xml` into logical categories. - Improves readability and maintainability of shared project dependencies.
- Move the `stats-client` module from `core/stats-service/stats-client` to `core/stats-client`. - Add necessary dependencies for gRPC client communication. - Create wrapper clients for interacting with `collector` and `analyzer` gRPC services.
- Configure `server.port: 0` for the collector service. - Prevents port clashes with the gateway service, allowing the OS to assign a free port.
…ging - Implement `LogUserAction` annotation and accompanying aspect for automatic user action logging. - Utilize gRPC client to send annotated user actions directly to the `collector` service. - Add Spring Boot autoconfiguration to automatically register the aspect and gRPC client beans.
- Add `stats-client` dependency to the module's POM. - Create shared `application-grpc-client.yaml` and activate "grpc-client" profile. - Annotate participation request endpoint with `@LogUserAction` for automatic event collection.
- Update `EventFullDto` and `EventShortDto` to replace the `views` field (long) with `rating` (double). - Reflects the shift from direct view collection to an aggregated rating provided by the Analyzer service.
- Change `getInteractionsCount` methods in `AnalyzerClient` and `AnalyzerService` to accept `Collection` instead of `List`. - Increases flexibility by allowing any collection type to be passed.
- Add "grpc-client" profile to configuration for accessing rating data. - Enhance event enrichment function to populate DTOs with rating values. - Replace `EventDtoWithConfirmedRequests` interface with `EventDtoWithRatingAndRequests` to reflect new data.
- Introduce `checkUserParticipation` method to the `@FeignClient RequestClient` interface. - Implement new business rule checking functionality across `request-service`'s infrastructure, application, and presentation layers. - Enables validation of user permissions for actions like liking an event.
- Introduce `GET /events/recommendations` endpoint for user-specific event recommendations.
- Implement `PUT /events/{eventId}/like` endpoint to allow users to like events.
- Implement all required business logic for both new functionalities, including interaction with underlying services.
- Create Dockerfiles for the new `aggregator`, `analyzer`, and `collector` services. - Enables containerization and deployment of the event-driven statistics components.
- Split `.proto` schema into two to conform to specific Protobuf package naming requirements. - Remove the 0-indexed "undefined" value from Protobuf enums, as per project requirements (which are very reasonable and not at all contradicting the best practices).
- Adjust action weights in recommendation system hyperparameters. - Ensures conformity with project requirements that were definitely totally absolutely specified somewhere.
- Remove various unused test dependencies.
- Reflect reworked recommendation functionality in project overview. - Document the shift from 'views' to 'ratings' for event metrics. - Detail the new asynchronous Kafka/gRPC data processing pipeline.
- Configure `eureka.instance.prefer-ip-address: true` for all services. - Improves compatibility and reliability in CI/CD environments.
- Clean up lingering configuration files and import statements. - Removes remnants of the previously deleted `stats-service` module.
- Decrease Eureka lease renewal and expiration periods. - Disable Eureka health checks. - Addresses issues with CI pipeline starting tests before services are fully declared UP by Eureka.
- Correct bug where event pairs were not always re-enqueued for similarity calculation.
- Addresses cases where individual $S_{min}$ for a pair was unchanged, but total similarity value had effectively changed.
- Now, all event pairs are enqueued for recalculation regardless of $S_{min}$ change, ensuring correctness.
kesch9
approved these changes
Sep 27, 2025
| import java.io.IOException; | ||
| import java.util.Map; | ||
| import lombok.extern.slf4j.Slf4j; | ||
| import org.apache.avro.generic.GenericDatumWriter; |
|
|
||
| UserActionAvro avroMessage = mapper.toAvro(request); | ||
|
|
||
| producer.sendUserAction(avroMessage); |
There was a problem hiding this comment.
здесь можно
producer.sendUserAction(mapper.toAvro(request));
| package ru.practicum.ewm.aggregator.domain; | ||
|
|
||
| import java.util.Map; | ||
| import java.util.Optional; |
- Remove unused imports. - Remove redundant inline variable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Данный PR реализует третий этап дипломного проекта. В рамках этапа была полностью удалена старая система сбора статистики и реализована новая, асинхронная система рекомендаций на базе Apache Kafka и gRPC.
Ключевые изменения:
Удален
stats-server: Старый сервис статистики и его компоненты (stats-dto) были полностью удалены из проекта.Реализован асинхронный конвейер обработки данных, состоящий из трех новых микросервисов:
collector: Принимает действия пользователей (VIEW,LIKE,REGISTER) по gRPC и отправляет их в Kafka.aggregator: Обрабатывает поток действий из Kafka, инкрементально рассчитывает косинусное сходство между событиями и публикует результаты в другой топик.analyzer: Сохраняет историю взаимодействий и рассчитанные сходства в свою БД, а также предоставляет gRPC API для выдачи рекомендаций.Внедрены новые технологии межсервисного взаимодействия:
Проведена интеграция с основными сервисами:
event-serviceиrequest-serviceтеперь асинхронно отправляют информацию о действиях пользователей.viewsу событий заменено наrating, который запрашивается уanalyzer.PUT /events/{eventId}/like) и получения персональных рекомендаций (GET /events/recommendations).Переработан
stats-client: Модуль был полностью переписан в универсальную клиентскую библиотеку, предоставляющую удобные gRPC-клиенты и AOP-аспекты для отправки действий.