Add import_csv_pandas and import_csv_dask utility primitives

[amotl]

About

More yak shaving.

Details

Similar to pyveci/pueblo#12 and #74, this patch adds generalized testing helper functions to avoid code duplication within the educational repository cratedb-examples, originally conceived at crate/cratedb-examples@dd09e144ef0. We need those kinds of utility functions at many more spots, so we need to find canonical places to store them.

Synopsis

pip install 'cratedb-toolkit[io]'

from cratedb_toolkit.io.sql import DatabaseAdapter

# Define database connection.
cratedb = DatabaseAdapter(dburi="crate://crate@localhost:4200")

# Mangle data. You choose.
cratedb.import_csv_pandas(filepath="test.csv", tablename="foobar")
cratedb.import_csv_dask(filepath="test.csv", tablename="foobar")

References

• [Python/SQLAlchemy] Add example programs about efficient bulk inserts with SQLAlchemy, pandas, and Dask cratedb-examples#64
• [DRAFT] ML: AutoML classification example notebook, demo program, and Markdown documentation cratedb-examples#122

Notes

Note that details on the interface may change while we go, specifically on the "naming things" / DWIM side of things. If you have any suggestions, feel free to add your voice. Right now, the main focus is to ship it, to be able to re-use it on behalf of Jupyter Notebooks we are currently publishing, in order to reduce boilerplate code within them.

/cc @karynzv, @marijaselakovic, @hlcianfagna, @hammerhead, @WalBeh, @andnig, @ckurze, @vvulf

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Files	Coverage Δ
cratedb_toolkit/util/database.py	92.78% <100.00%> (+1.75%)	⬆️

📢 Thoughts on this report? Let us know!

[matriv]

Sorry don't get, what do you mean mangle?

[amotl]

It's probably a non-word, apologies. I've changed the comment to »Load data into database.« now.

[seut]

Sounds good, lets do it? :)
Or any objections?

Suggested change

	# TODO: Use amount of CPU cores instead?
	npartitions = npartitions or 4
	npartitions = npartitions or os.cpu_count()

[amotl]

Added with ffbb1a0.