feat(spark): Refactor unit tests to sdk coding standards by digvijay-y · Pull Request #293 · kubeflow/sdk

digvijay-y · 2026-02-12T18:55:55Z

Refactor unit tests to follow Kubeflow SDK coding standards:

Added testcase dataclass
Removed all individual Test cases
All test now follow Kubeflow SDK coding standards
Added test_validate_spark_connect_url and test_create_and_connect_with_options to backend_test.py

Checklist:

Docs included if any changes are user facing

google-oss-prow · 2026-02-12T18:56:03Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kramaranya for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

github-actions · 2026-02-12T18:56:25Z

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

Copilot

Pull request overview

This PR refactors the SparkClient unit tests to follow the Kubeflow SDK coding standards by converting individual test classes to @pytest.mark.parametrize format with TestCase dataclass patterns. The refactoring aligns with issue #273 and follows the approach established in PR #225.

Changes:

Introduced TestCase dataclass for parametrized test structure
Converted all test methods from individual test classes to parametrized tests
Added mock improvements for handling both existing and non-existent sessions
Consolidated test organization into four main parametrized test functions

kubeflow/spark/api/spark_client_test.py

Copilot

Pull request overview

Copilot reviewed 2 out of 3 changed files in this pull request and generated 5 comments.

kubeflow/spark/api/spark_client_test.py

pyproject.toml

andreyvelich · 2026-02-20T01:52:40Z

/ok-to-test

digvijay-y · 2026-02-25T07:22:18Z

@kramaranya @szaher @astefanutti

kubeflow/spark/api/spark_client_test.py

Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

Signed-off-by: Jon Burdo <jon@jonburdo.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

Signed-off-by: Jon Burdo <jon@jonburdo.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

* fix: Improve CVE workflow Signed-off-by: Fiona-Waters <fiwaters6@gmail.com> * fix: fix issue with bash compare Signed-off-by: Fiona-Waters <fiwaters6@gmail.com> * feat: Add workflow to cleanup overrides in pyproject.toml Signed-off-by: Fiona-Waters <fiwaters6@gmail.com> * fix: address review comments Signed-off-by: Fiona-Waters <fiwaters6@gmail.com> * chore: refactor to reduce size of cve related workflows Signed-off-by: Fiona-Waters <fiwaters6@gmail.com> --------- Signed-off-by: Fiona-Waters <fiwaters6@gmail.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

* chore: update code style for Python 3.10 This disables a couple ruff rules in pyproject.toml: ``` "UP007", # Use X | Y instead of Union[X, Y] (requires Python 3.10+) "UP045", # Use X | None instead of Optional[X] (requires Python 3.10+) ``` Then the code changes are made with: ``` uv run ruff check --fix uv run ruff format ``` Signed-off-by: Jon Burdo <jon@jonburdo.com> * fix: handle unions, bools in convert_value The convert_value function didn't seems to be handling union types properly and also needs to handle `T | None` similarly to `Optional[None]` after the upgrade to Python 3.10. This fixes union types, an issue with bool conversion, and adds tests for this function. Signed-off-by: Jon Burdo <jon@jonburdo.com> --------- Signed-off-by: Jon Burdo <jon@jonburdo.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 5 to 7. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@v5...v7) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

…dates (kubeflow#291) Bumps the python-minor group with 4 updates in the / directory: [coverage](https://github.com/coveragepy/coveragepy), [ruff](https://github.com/astral-sh/ruff), [pre-commit](https://github.com/pre-commit/pre-commit) and [ty](https://github.com/astral-sh/ty). Updates `coverage` from 7.10.7 to 7.13.4 - [Release notes](https://github.com/coveragepy/coveragepy/releases) - [Changelog](https://github.com/coveragepy/coveragepy/blob/main/CHANGES.rst) - [Commits](coveragepy/coveragepy@7.10.7...7.13.4) Updates `ruff` from 0.14.14 to 0.15.0 - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.14.14...0.15.0) Updates `pre-commit` from 4.3.0 to 4.5.1 - [Release notes](https://github.com/pre-commit/pre-commit/releases) - [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md) - [Commits](pre-commit/pre-commit@v4.3.0...v4.5.1) Updates `ty` from 0.0.14 to 0.0.16 - [Release notes](https://github.com/astral-sh/ty/releases) - [Changelog](https://github.com/astral-sh/ty/blob/main/CHANGELOG.md) - [Commits](astral-sh/ty@0.0.14...0.0.16) --- updated-dependencies: - dependency-name: coverage dependency-version: 7.13.4 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-minor - dependency-name: ruff dependency-version: 0.15.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-minor - dependency-name: pre-commit dependency-version: 4.5.1 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-minor - dependency-name: ty dependency-version: 0.0.16 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

…s to handle ports (kubeflow#243) * update docs and add test cases. Signed-off-by: osamaahmed17 <osamaahmedtahir17@gmail.com> * pre-commit error solved Signed-off-by: osamaahmed17 <osamaahmedtahir17@gmail.com> * Update kubeflow/hub/api/model_registry_client.py Co-authored-by: Jon Burdo <jon@jonburdo.com> Signed-off-by: Osama Tahir <31954609+osamaahmed17@users.noreply.github.com> * readme updated Signed-off-by: Osama Tahir <31954609+osamaahmed17@users.noreply.github.com> * Refactor model registry client test cases for clarity Signed-off-by: Osama Tahir <31954609+osamaahmed17@users.noreply.github.com> --------- Signed-off-by: osamaahmed17 <osamaahmedtahir17@gmail.com> Signed-off-by: Osama Tahir <31954609+osamaahmed17@users.noreply.github.com> Co-authored-by: Jon Burdo <jon@jonburdo.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

…#277) Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 6 to 8. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v6...v8) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-version: '8' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

…ubeflow#280) * Add OWNERS validation Signed-off-by: muhammadjunaid8047 <muhammadjunaid8047@gmail.com> * Update .github/workflows/check-owners.yaml Co-authored-by: Jon Burdo <jon@jonburdo.com> Signed-off-by: Muhammad Junaid <muhammadjunaid8047@gmail.com> * Update OWNERS file check in workflow Signed-off-by: Muhammad Junaid <muhammadjunaid8047@gmail.com> * Update paths in check-owners workflow Signed-off-by: Muhammad Junaid <muhammadjunaid8047@gmail.com> --------- Signed-off-by: muhammadjunaid8047 <muhammadjunaid8047@gmail.com> Signed-off-by: Muhammad Junaid <muhammadjunaid8047@gmail.com> Co-authored-by: Jon Burdo <jon@jonburdo.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

Co-authored-by: google-oss-prow <92114575+google-oss-prow@users.noreply.github.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

…e actions group (kubeflow#297) Bumps the actions group with 1 update: [aquasecurity/trivy-action](https://github.com/aquasecurity/trivy-action). Updates `aquasecurity/trivy-action` from 0.33.1 to 0.34.0 - [Release notes](https://github.com/aquasecurity/trivy-action/releases) - [Commits](aquasecurity/trivy-action@0.33.1...0.34.0) --- updated-dependencies: - dependency-name: aquasecurity/trivy-action dependency-version: 0.34.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

Bumps [pytest](https://github.com/pytest-dev/pytest) from 8.4.2 to 9.0.2. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@8.4.2...9.0.2) --- updated-dependencies: - dependency-name: pytest dependency-version: 9.0.2 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

…w#130) * feat(backend): Support namespaced TrainingRuntime in the SDK Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * Fixed bugs and validated current test cases Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * Fixed pre-commit test failure Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * Addressed comments Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * Fixed no attribute 'DEFAULT_TIMEOUT' error Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * Added namespace-scoped runtime to test cases Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * Addressed fallback logic bugs Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * Added scope field to Runtime Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * Improved code Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * Fixed copilot's comments Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * Shadow duplicate runtimes, priority to ns Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * Fixed bug Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * Fixed copilot comments Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * Improved test cases to validate all possible cases Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * small fix Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * lint fix Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * improved error message Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Moeed <shaikmoeed@gmail.com> * refactored code Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * improve code Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * Removed RuntimeScope Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> * removed scope references and improved error handling as per kubeflow standards Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> --------- Signed-off-by: Moeed Shaik <shaikmoeed@gmail.com> Signed-off-by: Moeed <shaikmoeed@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

) * fix: Install SDK locally in E2E workflow and improve error handling for runtime fetching in Kubernetes backend. Signed-off-by: XploY04 <2004agarwalyash@gmail.com> * refactor: Explicitly return errors from and refine exception handling in . Signed-off-by: XploY04 <2004agarwalyash@gmail.com> * docs: update comment to clarify Kubeflow SDK installation from source in e2e workflow. Signed-off-by: XploY04 <2004agarwalyash@gmail.com> * feat: Enhance runtime retrieval tests to cover Kubernetes API 404/403 errors and partial success for list operations on timeout. Signed-off-by: XploY04 <2004agarwalyash@gmail.com> * refactor: Update runtime listing to immediately raise exceptions on failure instead of collecting partial results. Signed-off-by: XploY04 <2004agarwalyash@gmail.com> --------- Signed-off-by: XploY04 <2004agarwalyash@gmail.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5 to 6. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v5...v6) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

Bumps the python-minor group with 2 updates: [ruff](https://github.com/astral-sh/ruff) and [ty](https://github.com/astral-sh/ty). Updates `ruff` from 0.15.0 to 0.15.1 - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.15.0...0.15.1) Updates `ty` from 0.0.16 to 0.0.17 - [Release notes](https://github.com/astral-sh/ty/releases) - [Changelog](https://github.com/astral-sh/ty/blob/main/CHANGELOG.md) - [Commits](astral-sh/ty@0.0.16...0.0.17) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.15.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-minor - dependency-name: ty dependency-version: 0.0.17 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

* improve logging around packages_to_install Signed-off-by: Brian Gallagher <briangal@gmail.com> * exit when pip install fails, append errors from both attempts Signed-off-by: Brian Gallagher <briangal@gmail.com> * Add shlex to address command injection vulnerabilities. Write pip install logfile to cwd Signed-off-by: Brian Gallagher <briangal@gmail.com> --------- Signed-off-by: Brian Gallagher <briangal@gmail.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

…flow#306) * feat: Add validate lockfile workflow to complement CVE scanning Signed-off-by: Fiona-Waters <fiwaters6@gmail.com> * fix: make cve fix pr branch static Signed-off-by: Fiona-Waters <fiwaters6@gmail.com> --------- Signed-off-by: Fiona-Waters <fiwaters6@gmail.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

digvijay-y · 2026-02-26T09:30:34Z

@andreyvelich have made the changes, can you take a look?
Just initialization in client_Test and others moved to backend_test.py

5a2667a
1354cf6

andreyvelich · 2026-02-26T18:03:31Z

@digvijay-y It looks like after the rebase, you apply all commits to your branch.
Can you fix it please?

Copilot AI review requested due to automatic review settings February 12, 2026 18:55

google-oss-prow bot requested review from astefanutti, kramaranya and szaher February 12, 2026 18:56

google-oss-prow bot added the size/L label Feb 12, 2026

Copilot started reviewing on behalf of digvijay-y February 12, 2026 18:56 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

digvijay-y force-pushed the spark_test branch from db47042 to 5d8280e Compare February 12, 2026 19:18

digvijay-y changed the title ~~refactor(spark): convert individual test classes to @pytest.mark.parametrize format~~ feat(spark): Refactor unit tests to sdk coding standards Feb 12, 2026

digvijay-y requested a review from Copilot February 12, 2026 19:34

Copilot started reviewing on behalf of digvijay-y February 12, 2026 19:35 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

google-oss-prow bot added the ok-to-test label Feb 20, 2026

NarayanaSabari mentioned this pull request Feb 25, 2026

Integrating the Kubeflow Spark Application with the Kubeflow SDK #107

Open

andreyvelich reviewed Feb 25, 2026

View reviewed changes

kubeflow/spark/api/spark_client_test.py Show resolved Hide resolved

digvijay-y and others added 11 commits February 26, 2026 13:34

Refactored unit test

3f27217

Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

chore: bump minimum model-registry version to 0.3.6 (kubeflow#289)

2b3b9ec

Signed-off-by: Jon Burdo <jon@jonburdo.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

Changes made

70bc9f1

Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

Version

fbee7d6

Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

chore: bump minimum model-registry version to 0.3.6 (kubeflow#289)

7634232

Signed-off-by: Jon Burdo <jon@jonburdo.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

dependabot bot and others added 13 commits February 26, 2026 13:34

fix: nightly security dependency updates (kubeflow#296)

8ae4298

Co-authored-by: google-oss-prow <92114575+google-oss-prow@users.noreply.github.com> Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

Restructured clien_test

5a2667a

Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

reformated backend_test.py

1354cf6

Signed-off-by: digvijay-y <yewaredigvijay@gmail.com>

digvijay-y force-pushed the spark_test branch from 983910f to 1354cf6 Compare February 26, 2026 08:04

google-oss-prow bot added size/XXL and removed size/L labels Feb 26, 2026

digvijay-y requested a review from andreyvelich February 26, 2026 09:29

Conversation

digvijay-y commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

google-oss-prow bot commented Feb 12, 2026

Uh oh!

github-actions bot commented Feb 12, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andreyvelich commented Feb 20, 2026

Uh oh!

digvijay-y commented Feb 25, 2026

Uh oh!

Uh oh!

digvijay-y commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andreyvelich commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

digvijay-y commented Feb 12, 2026 •

edited

Loading

digvijay-y commented Feb 26, 2026 •

edited

Loading

andreyvelich commented Feb 26, 2026 •

edited

Loading