From 407c58bc4726d0614cd65aebd79cbce7c6b84c83 Mon Sep 17 00:00:00 2001 From: Steven Maude Date: Wed, 17 Dec 2025 22:51:19 +0000 Subject: [PATCH 1/4] Clarify maintenance mode --- README | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README b/README index 8f87636..01b1bd9 100644 --- a/README +++ b/README @@ -2,8 +2,10 @@ ScraperWiki Python library ========================== This is a Python library for scraping web pages and saving data. -It is the easiest way to save data on the ScraperWiki platform, and it -can also be used locally or on your own servers. + +**Warning: This library is now in maintenance mode.** + +**The library has been updated to work with Python 3.14 but there are no guarantees on future maintenance.** Installing ---------- From 5687f7ce02cec17d556b9c02d35e462fa0580453 Mon Sep 17 00:00:00 2001 From: Steven Maude Date: Wed, 17 Dec 2025 22:51:46 +0000 Subject: [PATCH 2/4] Remove note on future SQL support As we're in maintenance mode. --- README | 1 - 1 file changed, 1 deletion(-) diff --git a/README b/README index 01b1bd9..a567be4 100644 --- a/README +++ b/README @@ -32,7 +32,6 @@ the schema automatically according to the data you save. Currently only supports SQLite. It will make a local SQLite database. It is based on `SQLAlchemy `_. -You should expect it to support other SQL databases at a later date. scraperwiki.sql.save(unique_keys, data[, table_name="swdata"]) Saves a data record into the datastore into the table given by ``table_name``. From 3934c5c39bc1039f8bc5ca851896f9d3c80a7c61 Mon Sep 17 00:00:00 2001 From: Steven Maude Date: Wed, 17 Dec 2025 22:53:41 +0000 Subject: [PATCH 3/4] Delete QUIRKS It's no longer needed. The "classic" library has long gone. --- QUIRKS | 65 ---------------------------------------------------------- 1 file changed, 65 deletions(-) delete mode 100644 QUIRKS diff --git a/QUIRKS b/QUIRKS deleted file mode 100644 index 8c07166..0000000 --- a/QUIRKS +++ /dev/null @@ -1,65 +0,0 @@ -## Quirks - -This library aims to be compatible with the old ScraperWiki Classic library, -which was itself quite quirky. - -It's not an exact drop in replacement, this document describes the differences. - -## Differences - -### Datastore differences - -The local `scraperwiki.sqlite` is powered by -[SQLAlchemy](http://sqlalchemy.org), so some things -work a bit differently. - -Data is stored to a local sqlite database named `scraperwiki.sqlite`. - -Bizarre table and column names are supported. - -`scraperwiki.sqlite.execute` will return an empty list of keys on an -empty select statement result. - -`scraperwiki.sqlite.attach` downloads the whole datastore from ScraperWiki, the first time it runs; it then uses the cached database - -### Other Differences - -## Status of implementation -In general, features that have not been implemented raise a `NotImplementedError`. - -### Datastore -`scraperwiki.sqlite` may implement some less explored features -of its API slightly differently from ScraperWiki. - -### Utils -`scraperwiki.utils` is implemented, as well as the following functions. - -* `scraperwiki.log` -* `scraperwiki.scrape` -* `scraperwiki.pdftoxml` - -### Deprecated -These submodules are deprecated and thus will not be implemented. - -* `scraperwiki.apiwrapper` -* `scraperwiki.datastore` -* `scraperwiki.jsqlite` -* `scraperwiki.metadata` -* `scraperwiki.newsql` -* `scraperwiki.sqlite.attach` -* `scraperwiki.utils.swimport` -* `scraperwiki.geo` -* `scraperwiki.log` - -verbose parameter to various functions is deprecated - -### Specs -Here are some ScraperWiki scrapers that demonstrate the non-local library's quirks. - -https://scraperwiki.com/scrapers/scraperwiki-python/ -https://scraperwiki.com/scrapers/cast/ -https://scraperwiki.com/scrapers/things_happen_when_you_do_not_commit/ -https://scraperwiki.com/scrapers/what_does_show_tables_return/ -https://scraperwiki.com/scrapers/on_conflict/ -https://scraperwiki.com/scrapers/spaces_in_table_names/ -https://scraperwiki.com/scrapers/spaces_in_table_names_1/ From cb34812726f22f39277e4e829c647b23187b4a20 Mon Sep 17 00:00:00 2001 From: Steven Maude Date: Wed, 17 Dec 2025 22:54:22 +0000 Subject: [PATCH 4/4] Delete benchmark.py This is much like `save_speedtest.py`. --- benchmark.py | 15 --------------- 1 file changed, 15 deletions(-) delete mode 100755 benchmark.py diff --git a/benchmark.py b/benchmark.py deleted file mode 100755 index e8b5e60..0000000 --- a/benchmark.py +++ /dev/null @@ -1,15 +0,0 @@ -#! /usr/bin/env python3 -import scraperwiki -import os - -rows = [{"id": i, "test": i * 2, "s": "abc"} for i in range(1000)] - -try: - os.remove("scraperwiki.sqlite") -except FileNotFoundError: - pass - -scraperwiki.sql.save(["id"], rows) - -for i, row in enumerate(rows): - scraperwiki.sql.save(["id"], row)