FEAT-#7676: in-place casting between DataFrame engines by sfc-gh-jkew · Pull Request #7666 · modin-project/modin

sfc-gh-jkew · 2025-09-11T16:23:43Z

Addresses a performance issue with hybrid execution where a sequence of merge operations using the same DataFrames could result in transfer thrashing. In this PR we have the arguments of the operation move in-place so that subsequent operations all stay on the same backend. This behavior can be turned off by setting the BackendMergeCastInPlace variable to false.

first commit message and PR title follow format outlined here

NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.
passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
signed commit with git commit -s
Resolves Convert the QueryCompilerCaster to In-Place Casting to Reduce Data Migration Churn #7676
tests added and passing
module layout described at docs/development/architecture.rst is up-to-date

modin/tests/pandas/native_df_interoperability/utils.py

modin/tests/pandas/native_df_interoperability/test_compiler_caster.py

modin/core/storage_formats/pandas/query_compiler_caster.py

modin/pandas/series.py

Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>

sfc-gh-joshi

A few more minor nits, LGTM after they're addressed.

modin/tests/pandas/extensions/test_pd_extensions.py

modin/tests/pandas/native_df_interoperability/test_compiler_caster.py

modin/tests/pandas/native_df_interoperability/utils.py

Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>

…ster.py Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>

Co-authored-by: Mahesh Vashishtha <mahesh.vashishtha@snowflake.com>

…ster.py Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>

sfc-gh-jkew added 9 commits September 19, 2025 16:21

WIP in-place casting between dataframe engines

b985c17

Test fixes for in-place modification

08e2dec

tests for merge in place

ceddef7

Additional test fix

530c20d

WIP - many questions; few answers

4de82fa

lint

b8d5a0c

Reset _parent when switching backends

35f0788

Doc and update _parent

2dff557

Updates

55141ee

sfc-gh-jkew force-pushed the jkew/hybrid-merge-in-place branch from 1df6492 to 55141ee Compare September 19, 2025 23:25

sfc-gh-jkew added 2 commits September 19, 2025 16:43

Remove asserts

362c181

Documentation update

2bfc073

sfc-gh-jkew changed the title ~~WIP in-place casting between dataframe engines~~ FEAT-#7676: in-place casting between dataframe engines Sep 22, 2025

sfc-gh-jkew changed the title ~~FEAT-#7676: in-place casting between dataframe engines~~ FEAT-#7676: in-place casting between DataFrame engines Sep 22, 2025

sfc-gh-jkew marked this pull request as ready for review September 22, 2025 23:44

sfc-gh-jkew requested review from a team, RehanSD, YarShev, anmyachev, dchigarev, devin-petersohn, mvashishtha and vnlitvinov as code owners September 22, 2025 23:44

sfc-gh-mvashishtha reviewed Sep 23, 2025

View reviewed changes

sfc-gh-jkew added 3 commits September 24, 2025 12:22

Change how DFs are created in test utils

3e5dd21

Fix asserts prior to merge in place

7aa618b

Remove in_place

50b2bd3

sfc-gh-joshi reviewed Sep 24, 2025

View reviewed changes

modin/pandas/series.py Outdated Show resolved Hide resolved

sfc-gh-jkew and others added 2 commits September 25, 2025 13:13

Update modin/pandas/series.py

2e08dfa

Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>

re-add comma from edits

1c75abd

sfc-gh-jkew added 3 commits September 25, 2025 13:18

lint

2171281

isort

12994fa

Interop tests at python 3.10

dd470cc

sfc-gh-joshi reviewed Sep 25, 2025

View reviewed changes

Issue not python 3.9 specific; reseting

aa21313

sfc-gh-mvashishtha reviewed Sep 25, 2025

View reviewed changes

modin/tests/pandas/native_df_interoperability/utils.py Outdated Show resolved Hide resolved

modin/tests/pandas/native_df_interoperability/utils.py Show resolved Hide resolved

sfc-gh-jkew and others added 12 commits September 25, 2025 14:40

get more info on exception

55dc657

Update modin/tests/pandas/extensions/test_pd_extensions.py

a4a79d4

Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>

Update modin/tests/pandas/native_df_interoperability/test_compiler_ca…

14b201b

…ster.py Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>

Update modin/tests/pandas/native_df_interoperability/test_compiler_ca…

a45c3bb

…ster.py Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>

Update modin/tests/pandas/native_df_interoperability/test_compiler_ca…

b9d24ad

…ster.py Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>

Update modin/tests/pandas/native_df_interoperability/test_compiler_ca…

851cfd2

…ster.py Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>

Update modin/tests/pandas/native_df_interoperability/test_compiler_ca…

d23a9bb

…ster.py Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>

Update modin/tests/pandas/native_df_interoperability/test_compiler_ca…

dfb472e

…ster.py Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>

Update modin/tests/pandas/native_df_interoperability/utils.py

3d3004e

Co-authored-by: Mahesh Vashishtha <mahesh.vashishtha@snowflake.com>

fix import

bfacfa6

Update modin/tests/pandas/native_df_interoperability/test_compiler_ca…

9fd776b

…ster.py Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>

Update modin/tests/pandas/native_df_interoperability/test_compiler_ca…

f1cd589

…ster.py Co-authored-by: Jonathan Shi <149419494+sfc-gh-joshi@users.noreply.github.com>

sfc-gh-mvashishtha approved these changes Sep 26, 2025

View reviewed changes

sfc-gh-jkew merged commit 6e861d0 into main Sep 26, 2025
75 of 76 checks passed

sfc-gh-jkew mentioned this pull request Sep 26, 2025

Interop Tests should all use Backup.get instead of "Ray" #7681

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FEAT-#7676: in-place casting between DataFrame engines#7666

FEAT-#7676: in-place casting between DataFrame engines#7666
sfc-gh-jkew merged 32 commits intomainfrom
jkew/hybrid-merge-in-place

sfc-gh-jkew commented Sep 11, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sfc-gh-joshi left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

sfc-gh-jkew commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sfc-gh-joshi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sfc-gh-jkew commented Sep 11, 2025 •

edited

Loading