DuckDBPyConnection object has no attribute 'to_arrow_table'

[paultiq]

What happens?

There's an inconsistency between DuckDBPyConnection and DuckDBPyRelation:

Relations expose to_arrow_table, but not Connections.

Noticed this when fixing some broken code paths due to the change to .arrow().

To Reproduce

import duckdb

print(duckdb.__version__)
with duckdb.connect() as conn:
    t1 = conn.sql("select * from range(10)").fetch_arrow_table()
    print("sql fetch_arrow_table works")
    t2 = conn.sql("select * from range(10)").to_arrow_table()
    print("sql to_arrow_table works")
    t3 = conn.execute("select * from range(10)").fetch_arrow_table()
    print("execute fetch_arrow_table works")
    t4 = conn.execute("select * from range(10)").to_arrow_table()
    print("execute to_arrow_table works")

1.5.0.dev44
sql fetch_arrow_table works
sql to_arrow_table works
execute fetch_arrow_table works

AttributeError Traceback (most recent call last)
Cell In[1], line 4
1 import duckdb
3 print(duckdb.version)
----> 4 with duckdb.connect() as conn:
5 t1 = conn.sql("select * from range(10)").fetch_arrow_table()
6 print("sql fetch_arrow_table works")

Cell In[1], line 11
9 t3 = conn.execute("select * from range(10)").fetch_arrow_table()
10 print("execute fetch_arrow_table works")
---> 11 t4 = conn.execute("select * from range(10)").to_arrow_table()
12 print("execute to_arrow_table works")

AttributeError: '_duckdb.DuckDBPyConnection' object has no attribute 'to_arrow_table'

OS:

Linux

DuckDB Package Version:

1.4.0, duckdb==1.5.0.dev44

Python Version:

3.11

Full Name:

Paul T

Affiliation:

Iqmo

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a nightly build

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration to reproduce the issue?

• Yes, I have

[evertlammerts]

The current shared API across duckdb, duckdb.DuckDBPyConnection and duckdb.DuckDBPyRelation is:

• arrow() -> pyarrow.lib.RecordBatchReader
• [alias] fetch_record_batch() -> pyarrow.lib.RecordBatchReader
• fetch_arrow_table() -> pyarrow.lib.Table

In addition to the above, duckdb.DuckDBPyRelation also has:

• [alias]to_arrow_table() -> pyarrow.lib.Table
• [alias]fetch_arrow_reader() -> pyarrow.lib.RecordBatchReader

In other words, too many aliases. I'm not the biggest fan of fetch_record_batch which returns a reader and not a batch. While we did get some very valid complaints about the unannounced change of arrow()'s return function, we could consider making another change in 1.5.0, unifying the arrow function names to something like arrow_batch_reader() and arrow_table().

[paultiq]

I find this inconsistency, well, inconsistent :)... I've come to expect Relations and Connections to expose the same data API.

That said, agree with deprecating the aliases and +1 for .arrow_table / .arrow_batch_reader.

[NickCrews]

Also +1 to being breaking in order to have longer-term sanity.

[evertlammerts]

I had an unfinished branch hanging around, just updated and pushed it to #287 . It deprecates everything that currently exists and adds arrow_table() and arrow_reader(). Docs PR is in the making as well. This would go into 1.5. Feedback welcome!

[djouallah]

arrow() works perfectly and it is doing what is expected to do, could you please leave it as it is , or at least wait till delta lake writer matures, we got a lot of bugs because users did not know the difference between record batch and full materialization and honestly they should not know those stuff.

edit :just keep arrow() as an alias to arrow_reader(), problem solved.

[paultiq]

I like it; two new methods, document the others as deprecated but don't remove until subsequent release.

This gives a clearer API and a migration path without breaking anything.

[paultiq]

Most APIs use the "to_xyz" and "from_xyz" convention.

Examples:

• Pandas to_.parquet
• Polars to_arrow
• PyArrow to_pandas

Should DuckDB use this convention as well? Or should it stick with: DuckDBPyRelation.xyz() vs DuckDBPyRelation.to_xyz().

I don't have a strong opinion either way, only that there should be a decision for the sake of consistency within the DucKDB Connection & Relation APIs.

DuckDB already has several "to" methods (to_arrow_table, to_csv, etc), but doesn't use it consistently (ie: no to_polars / to_pl). https://duckdb.org/docs/stable/clients/python/reference/

[evertlammerts]

Alright, I hear no objections to deprecating to_arrow_table() and fetch_arrow_reader() from Relation. Those are aliases that are not available on Connection (nor on the module).

@djouallah Reading your comment it seems like users can't really get away without at least some understanding of the difference between batchwise and full materialization. Which makes sense, since the one is not a drop-in replacement for the other. I guess that this is why users have been reporting bugs? Making that difference explicit in the naming of the functions seems like a win to me. I'm not strongly opposed to not deprecating arrow() for now though. But I am inclined to do a 'soft' deprecation, by removing it from our docs and adding a comment in the docstring that tells users that it is an alias for [to_]arrow_reader(), and that we are likely to deprecate arrow() in a future release (and remove it in a subsequent release). Wdyt?

Re naming: I like the suggestion to add a to_ prefix if that makes it somewhat more consistent with others. It would indeed not break any convention on our end, which is the advantage of not having a convention I guess :)

So in summary:

• Across the module, Connection and Relation APIs:
  • Deprecate fetch_record_batch() -> pyarrow.lib.RecordBatchReader in favor of to_arrow_reader() -> pyarrow.lib.RecordBatchReader
  • Deprecate fetch_arrow_table() -> pyarrow.lib.Table in favor of to_arrow_table() -> pyarrow.lib.Table
  • Keep arrow() -> pyarrow.lib.RecordBatchReader (as an alias for to_arrow_reader())
• Deprecate to_arrow_table() and fetch_arrow_reader() from Relation

[djouallah]

Thank you, I hope by that time, we will have a native delta writer 😁

[NickCrews]

@evertlammerts regarding this line in your comment above: Deprecate to_arrow_table() ... from Relation

Is this a typo? In the bullet point just above that, you say Deprecate fetch_arrow_table() -> pyarrow.lib.Table in favor of to_arrow_table() -> pyarrow.lib.Table, which makes it sound like you want to ADD Relation.to_arrow_table()?

In summary, this is what I would prefer the final API to be:

• to_arrow_table() and to_arrow_reader() on both duckdb.DuckDBPyConnection and duckdb.DuckDBPyRelation. I fairly strongly prefer the to_ prefix, and hope that the entire packages API slowly moves consistently to that convention.
• for everything else that currently exists, doing the "soft deprecation" that you describe. eg make it hard for new users to find, add warning to docstrings, but don't emit any actual warnings with eg warnings.warn(). Could even hide these under __getattribute__() so that LSPs etc don't even suggest them.

The differences between this behavior, and what @evertlammerts suggests above, are

• they only suggest soft-deprecating .arrow(), but hard-deprecating everything else. I suggest making everything being a soft-deprecation. I don't feel super strongly here though, I'm fine with following their preference.
• I strongly desire Relation.to_arrow_table() to be available. As I say above in this comment, I'm not sure if that is just a misreading of their comment, they might also be saying the same thing.

[evertlammerts]

Is this a typo? In the bullet point just above that, you say Deprecate fetch_arrow_table() -> pyarrow.lib.Table in favor of to_arrow_table() -> pyarrow.lib.Table, which makes it sound like you want to ADD Relation.to_arrow_table()?

@NickCrews yes, typo, or more precisely a leftover from a few comments earlier.

See the PR for the full description of what we're going to do. Tl;dr:

The Connection and Relational APIs will have the following functions:

• to_arrow_reader() -> pyarrow.lib.RecordBatchReader
• to_arrow_table() -> pyarrow.lib.Table
• arrow() -> pyarrow.lib.RecordBatchReader Note: we will not deprecate this function in v1.5.0, but we will discourage its use in both the documentation and the docstring. We encourage users to use to_arrow_reader() instead.

For the record: the Connection API will keep this function:

• arrow(arrow_object, connection = None) -> DuckDBPyRelation

The fetch_* functions will be deprecated in v1.5.0 (which will be emitted as a DeprecationWarning) and removed in v1.6.0:

• fetch_record_batch() -> pyarrow.lib.RecordBatchReader
• fetch_arrow_table() -> pyarrow.lib.Table
• fetch_arrow_reader() -> pyarrow.lib.RecordBatchReader

Relation::record_batch() -> pyarrow.lib.RecordBatchReader will be removed.