Skip to content

Conversation

@fucangfy
Copy link
Collaborator

@fucangfy fucangfy commented Feb 9, 2026

No description provided.

@fucangfy fucangfy requested a review from a team February 9, 2026 10:58
@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello @fucangfy, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request represents a substantial architectural evolution of the SCQL project. It streamlines the codebase by removing several legacy components and their supporting infrastructure, focusing on a leaner core. The introduction of a new compiler API version signifies a forward-looking approach to how SQL queries are processed and secured within the system. These changes collectively aim to modernize the project's foundation, improve maintainability, and prepare for future feature development.

Highlights

  • CI/CD and Build System Overhaul: Significant changes to the CI/CD pipeline, including the removal of entire Docker Compose-based testing environments (.ci/broker-docker-compose, .ci/docker-compose), regression test configurations (.circleci/regtest-config.yml), and release configurations (.circleci/release-config.yml). Go test commands in CircleCI configurations were updated to exclude the contrib/... path, and a new .markdownlint.yaml was added for markdown linting.
  • Core Component Removal and Refactoring: The scdb and broker applications, along with their associated command-line tools (brokerctl, scdbclient, scqltool), agent (cmd/agent), and benchmarking suite (benchmark), have been entirely removed. This indicates a major architectural shift, likely consolidating functionalities or deprecating these standalone services.
  • API Versioning and Protocol Updates: Introduced a new v1alpha1 API for the compiler service, defining detailed security configurations and operator graph structures. Core protocol definitions (api/core.proto, api/common.proto) were updated to include new data types (DECIMAL), PSI modes (Rr22Mode), and task-related messages (scql_task.proto). Older broker and CCL-related protos were removed.
  • Dependency and Build Tool Updates: Updated versions for several Bazel dependencies including yacl, psi, spulib, dataproxy_sdk_cc, arrow, and openssl. New dependencies like perfetto and pybind11_bazel were added, indicating potential new features or performance tooling. Bazel build flags were also updated for remote caching and ASAN options.
  • Data Source Adaptor Enhancements: Refactored data source adaptors, introducing DATAMESH and DATAPROXY kinds, and updating CSVDB and ODBC adaptors to handle new data types (DATETIME, TIMESTAMP). The EmbedRouter now supports building from file paths, and new flags for Kpad task mode execution and tracing were added to the engine.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog
  • .bazelrc
    • Updated Bazel remote download and registry settings.
    • Added experimental proto descriptor sets and ASAN options.
  • .ci/broker-docker-compose
    • Removed all files related to the broker docker-compose setup.
  • .ci/docker-compose
    • Removed all files related to the docker-compose setup.
  • .ci/script/check-whitespace.sh
    • Removed.
  • .ci/script/ensure_binary_safety.sh
    • Removed.
  • .ci/test-data/csv/bob_UPPER_table.csv
    • Removed.
  • .ci/test-data/csv/bob_tbl_3.csv
    • Removed.
  • .circleci/coverage-config.yml
    • Modified go test command to exclude contrib/... path.
  • .circleci/diff-cover-config.yml
    • Modified go test command to exclude contrib/... path.
  • .circleci/full-unittest-config.yml
    • Modified go test command to exclude contrib/... path.
  • .circleci/lite-unittest-config.yml
    • Modified go test command to exclude contrib/... path.
  • .circleci/path-filtering/unittest.conf
    • Removed contrib/.* run_go_ut true entry.
  • .circleci/regtest-config.yml
    • Removed.
  • .circleci/release-config.yml
    • Removed.
  • .gitignore
    • Removed scdb.db and logs/scdbserver.log.
    • Added python/build/ and python/package/.
  • .golangci.yml
    • Minor formatting adjustments.
  • .licenserc.yaml
    • Updated paths-ignore list to reflect removed scdb and broker components.
  • .markdownlint.yaml
    • Added new markdown linting configuration.
  • .vscode/cspell.json
    • Removed scdb, scdbclient, scdbserver from the dictionary.
  • .vscode/extensions.json
    • Updated proto extension and added bufbuild.vscode-buf.
  • .vscode/settings.json
    • Added proto formatter settings.
  • CONTRIBUTING.md
    • Changed bullet points from * to -.
    • Updated build instructions to reflect removal of scdb components.
  • LEGAL.md
    • Added # to the title.
  • MODULE.bazel
    • Updated versions for various dependencies (yacl, psi, spulib, dataproxy_sdk_cc, arrow, openssl).
    • Added git_override for spulib.
    • Added new dependencies: perfetto, pybind11_bazel, rules_buf.
    • Removed heu dependency.
  • Makefile
    • Added binary-cov target for coverage builds.
    • Removed golang.org/x/tools/cmd/cover@latest from GO-package target.
  • README.md
    • Added a note directing users to the release/1.x branch for SCQL 1.0.
  • REPO_LAYOUT.md
    • Removed entries for docker/, scdb-tutorial, p2p-tutorial, scdbserver/, scdbclient/, broker/, brokerctl/.
    • Updated interpreter section to reflect new compiler and graph components.
  • api/BUILD.bazel
    • Removed proto libraries for broker, ccl, and scdb_api.
    • Added scql_task_proto and buf_lint_test rules for various protos.
  • api/broker.config_openapi.yaml
    • Removed.
  • api/broker.proto
    • Removed.
  • api/ccl.proto
    • Removed.
  • api/common.proto
    • Added Rr22Mode enum.
    • Modified PsiConfig to use Rr22Mode.
    • Added Placeholder, Placeholders, Variable messages.
  • api/core.proto
    • Added imports for api/status.proto and api/common.proto.
    • Added DECIMAL to PrimitiveDataType.
    • Added scale and width fields to Tensor.
    • Added QueryResult and QueryResponse messages.
  • api/engine.proto
    • Updated comments for EngineResultCallback and ReportRequest.
    • Added StreamingOptions, PsiOptions, NegotiationOptions messages.
  • api/generate_proto.sh
    • Updated to generate v1alpha1/compiler_go_proto.
    • Removed OpenAPI generation for broker.proto and scdb_api.proto.
  • api/scdb_api.config_openapi.yaml
    • Removed.
  • api/scdb_api.proto
    • Removed.
  • api/scql_task.proto
    • Added new proto definition for ScqlConfig.
  • api/status_code.proto
    • Updated SCDB session not found to Driver session not found.
    • Added a note about CCL removal.
    • Updated SCDB DB error to Driver DB error.
  • api/v1/column.proto
    • Updated comment for DataType spec link.
    • Changed aliases for INT, INTEGER, and FLOAT to INT64 and FLOAT64 respectively.
  • api/v1alpha1/BUILD.bazel
    • Added new BUILD file for compiler_proto.
  • api/v1alpha1/compiler.proto
    • Added new proto definition for CompilerService, CompileOptions, GlobalSecurityRelaxation, ColumnSecurityRelaxation, ReverseInferenceConfig, ResultSecurityConfig, CompilerSecurityConfig, ColumnVisibility, AdditionalInfoSpec, CompileSQLRequest, CompileSQLResponse, OperatorGraph, TensorMeta, TensorIdList, Operator, Kernel.
  • benchmark
    • Removed entire benchmark directory.
  • cmd/agent
    • Removed entire cmd/agent directory.
  • cmd/broker
    • Removed entire cmd/broker directory.
  • cmd/brokerctl
    • Removed entire cmd/brokerctl directory.
  • cmd/regtest/p2p
    • Removed entire cmd/regtest/p2p directory.
  • cmd/regtest/scdb
    • Removed entire cmd/regtest/scdb directory.
  • cmd/regtest/testdata/multi_parties.json
    • Removed.
  • cmd/regtest/testdata/multi_parties_semi2k.json
    • Removed.
  • cmd/regtest/testdata/single_party.json
    • Removed.
  • cmd/regtest/testdata/single_party_postgres.json
    • Removed.
  • cmd/regtest/testdata/two_parties.json
    • Removed.
  • cmd/regtest/testdata/view.json
    • Removed.
  • cmd/scdbclient
    • Removed entire cmd/scdbclient directory.
  • cmd/scdbserver
    • Removed entire cmd/scdbserver directory.
  • cmd/scqltool
    • Removed entire cmd/scqltool directory.
  • contrib/agent/kusciastub/mock_kusciaapi/mock_kusciaapi.go
    • Removed.
  • contrib/agent/proto/BUILD.bazel
    • Renamed to engine/util/dp/BUILD.bazel.
  • contrib/agent/proto/task_config.pb.go
    • Removed.
  • contrib/agent/proto/task_config.proto
    • Removed.
  • docker
    • Removed entire docker directory.
  • engine/datasource/csvdb_adaptor.cc
    • Added MergeCsvdbDatasources function.
  • engine/datasource/csvdb_adaptor.h
    • Added MergeCsvdbDatasources function declaration.
  • engine/datasource/csvdb_adaptor_test.cc
    • Updated CheckTensorEqual calls to use scql::engine::op::test::CheckTensorEqual.
    • Updated test data to include dt and tt columns.
    • Added MergeCsvdbDatasources test.
  • engine/datasource/dataproxy_conf.proto
    • Added new proto definition for DataProxyConf.
  • engine/datasource/datasource.proto
    • Added DATAMESH enum, changed DATAPROXY enum value.
  • engine/datasource/datasource_adaptor.cc
    • Included engine/exe/flags.h.
    • Changed util::SimpleChannel capacity to use FLAGS_queue_max_block_seconds.
    • Added exception handling for CreateDiskTensor and CreateMemTensor.
  • engine/datasource/datasource_adaptor_mgr.cc
    • Included engine/datasource/dm_adaptor_factory.h.
    • Added DmAdaptorFactory to factory_maps_ for DATAMESH kind.
  • engine/datasource/dm_adaptor.cc
    • Added new implementation for DmAdaptor and DmChunkedResult.
  • engine/datasource/dm_adaptor.h
    • Added new header for DmAdaptor and DmChunkedResult.
  • engine/datasource/dm_adaptor_factory.cc
    • Added new implementation for DmAdaptorFactory.
  • engine/datasource/dm_adaptor_factory.h
    • Added new header for DmAdaptorFactory.
  • engine/datasource/dm_adaptor_kuscia_test.cc
    • Renamed from dp_adaptor_kuscia_test.cc.
    • Updated to use DmAdaptor.
  • engine/datasource/dp_adaptor.cc
    • Updated to use arrow::flight for data proxy communication.
    • Constructor now takes JSON string for DataProxyConf.
  • engine/datasource/dp_adaptor.h
    • Updated to use arrow::flight for data proxy communication.
    • DpChunkedResult now uses arrow::flight::FlightInfo and arrow::flight::FlightClient.
  • engine/datasource/dp_adaptor_factory.cc
    • Updated to use DpAdaptor.
  • engine/datasource/dp_adaptor_factory.h
    • Updated to use DpAdaptor.
  • engine/datasource/dp_adaptor_test.cc
    • Added new test file for DpAdaptor.
  • engine/datasource/duckdb_wrapper.cc
    • Minor comment fix.
  • engine/datasource/duckdb_wrapper_test.cc
    • Fixed typo in test name IngoringColumnCase.
  • engine/datasource/embed_router.cc
    • Added FromFilePaths method to create router from input file paths.
  • engine/datasource/embed_router.h
    • Added FromFilePaths method declaration.
  • engine/datasource/embed_router_test.cc
    • Added tests for FromFilePaths method.
  • engine/datasource/kuscia_datamesh_router.cc
    • Updated to use util::QueryDomainData and util::QueryDomainDataSource.
    • Renamed MakeDataProxyDataSource to MakeDataMeshDataSource.
  • engine/datasource/odbc_adaptor_mysql_test.cc
    • Added dt and tt columns to person table.
    • Set time_zone for MySQL session.
    • Updated ExecQuery to include dt and tt columns.
    • Updated CheckTensorEqual calls to use scql::engine::op::test::CheckTensorEqual.
  • engine/datasource/odbc_adaptor_sqlite_test.cc
    • Added dt and tt columns to person table.
    • Updated ExecQuery to include dt and tt columns.
    • Updated CheckTensorEqual calls to use scql::engine::op::test::CheckTensorEqual.
  • engine/exe/BUILD.bazel
    • Added //api:scql_task_cc_proto to scqlengine deps.
    • Added //engine/util:kpad_task_helper and //engine/util:ssl_helper to scqlengine deps.
  • engine/exe/flags.cc
    • Changed driver comment.
    • Added kpad_job_id, kpad_scql_config, kpad_cluster_def flags.
    • Removed use_rr22_low_comm_mode.
    • Added enable_trace, trace_log_path, rr22_mode, queue_max_block_seconds flags.
  • engine/exe/flags.h
    • Changed driver comment.
    • Added kpad_job_id, kpad_scql_config, kpad_cluster_def flag declarations.
    • Removed use_rr22_low_comm_mode declaration.
    • Added enable_trace, trace_log_path, rr22_mode, queue_max_block_seconds flag declarations.
  • engine/exe/main.cc
    • Added includes for atomic, chrono, future, thread, brpc/controller.h.
    • Included engine/util/kpad_task_helper.h, engine/util/ssl_helper.h, engine/util/trace_categories.h.
    • Included api/scql_task.pb.h.
    • Added ExecuteKpadTask function for background task execution.
    • Added logic to validate kpad task parameters and execute kpad task.
    • Added TracingSessionGuard.
    • Changed MuxReceiverService and ErrorCollectorService log messages.
    • Added return logic for kpad task success/failure.
  • engine/framework/BUILD.bazel
    • Added //engine/util:trace_categories to session_manager deps.
  • engine/util/dp/BUILD.bazel
    • Renamed from contrib/agent/proto/BUILD.bazel.
    • Added new BUILD file for flight_proto.
  • engine/util/ssl_helper.h
    • Renamed from pkg/interpreter/translator/constant.go.
    • Added new header for LoadSslCredentialsOptions.
Ignored Files
  • Ignored by pattern: .github/workflows/** (2)
    • .github/workflows/trigger-ci-regtest.yml
    • .github/workflows/trigger-ci-release.yml
Activity
  • The pull request was created by fucangfy.
  • The title repo-sync-2026-02-09 suggests this is an automated repository synchronization or a large-scale update.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request primarily focuses on refactoring the project structure, updating Bazel dependencies, and modifying CI/CD configurations. Key changes include removing the contrib directory from Go test scopes and Bazel path filtering, updating Bazel module versions for yacl, psi, spulib, dataproxy_sdk_cc, arrow, openssl, and adding perfetto and pybind11_bazel dependencies. The .bazelrc file was updated to include remote download outputs and proto descriptor sets, and ASAN options. The api directory saw significant changes with the removal of broker.proto, ccl.proto, and scdb_api.proto, and the introduction of scql_task.proto and compiler.proto (under v1alpha1), along with new buf_lint_test rules for proto files. The core.proto file was updated to include DECIMAL as a primitive data type and changed aliases for INT, INTEGER, and FLOAT to point to 64-bit types, which is a significant breaking change. The engine.proto now includes StreamingOptions, PsiOptions, and NegotiationOptions. The datasource directory in the engine was refactored to introduce datamesh adaptors and factories, replacing previous dataproxy implementations, and enhancing CSVDB datasource merging. The main.cc in engine/exe was updated to support a new kpad task mode. Additionally, various documentation files (CONTRIBUTING.md, LEGAL.md, README.md, REPO_LAYOUT.md) and configuration files (.gitignore, .golangci.yml, .licenserc.yaml, .vscode/*, Makefile) were updated to reflect these structural and dependency changes, including removing references to scdb components and updating linting rules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant