Spreadsheet extraction functionality.

.github/ISSUE_TEMPLATE/bug_report.md

@@ -22,7 +22,7 @@ If applicable, add screenshots to help explain your problem.
 
 ## Context
 
-Please, complete the following to better understand the system you are using to run MUSE.
+Please, complete the following to better understand the system you are using to run Auto-CORPus.
 
 - Operating system (eg. Windows 10):
 - Auto-CORPus version (eg. 1.0.0):

.github/actions/setup/action.yml

@@ -19,4 +19,4 @@ runs:
 
     - name: Install dependencies
       shell: bash
-      run: poetry install
+      run: poetry install --all-extras

.github/workflows/ci.yml

@@ -20,11 +20,27 @@ jobs:
         python-version: ['3.10', '3.11', '3.12', '3.13']
     steps:
       - uses: actions/checkout@v4
+        with:
+          # Use a custom PAT so the runners can access the private submodule
+          token: ${{ secrets.PAT }}
+          submodules: true
+      - name: Install LibreOffice
+        if: runner.os == 'Linux'
+        run: sudo apt-get update && sudo apt-get install -y libreoffice
       - uses: ./.github/actions/setup
         with:
           python-version: ${{ matrix.python-version }}
+      - name: Install pywin32 on Windows
+        if: runner.os == 'Windows'
+        run: poetry add pywin32
       - name: Run tests
-        run: poetry run pytest --skip-ci-macos
+        run: poetry run pytest --skip-ci-macos --skip-ci-windows
+      - name: Upload coverage reports to Codecov
+        if: ${{ matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13' && github.event.pull_request.user.login != 'dependabot[bot]' && github.event.pull_request.user.login != 'pre-commit-ci[bot]' }}
+        uses: codecov/codecov-action@v5
+        with:
+          fail_ci_if_error: true
+          token: ${{ secrets.CODECOV_TOKEN }}
       - name: Check docs build
         run: poetry run mkdocs build
       - name: Check types

.github/workflows/release.yml

@@ -7,6 +7,7 @@ on:
 jobs:
   test:
     uses: ./.github/workflows/ci.yml
+    secrets: inherit
 
   build-wheel:
     needs: test

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "tests/data/private"]
+	path = tests/data/private
+	url = ../Auto-CORPus-private-test-data

.pre-commit-config.yaml

@@ -21,19 +21,19 @@ repos:
     hooks:
       - id: check-github-workflows
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.11.9
+    rev: v0.11.12
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]
       - id: ruff-format
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v1.15.0
+    rev: v1.16.0
     hooks:
       - id: mypy
         exclude: autocorpus/parse_xml.py
         additional_dependencies: [types-beautifulsoup4, types-regex, lxml-stubs, types-tqdm, types-jsonschema]
   - repo: https://github.com/igorshubovych/markdownlint-cli
-    rev: v0.44.0
+    rev: v0.45.0
     hooks:
       - id: markdownlint-fix
   - repo: https://github.com/codespell-project/codespell

.vscode/launch.json

@@ -10,12 +10,12 @@
       "request": "launch",
       "module": "autocorpus",
       "args": [
-        "-c",
-        "${workspaceFolder}/autocorpus/configs/config_pmc.json",
+        "-b",
+        "PMC",
         "-t",
         "output",
         "-f",
-        "${workspaceFolder}/tests/data/PMC/Current/PMC8885717.html"
+        "${workspaceFolder}/tests/data/public/html/PMC/PMC8885717.html"
       ]
     },
     {
@@ -24,12 +24,12 @@
       "request": "launch",
       "module": "autocorpus",
       "args": [
-        "-c",
-        "${workspaceFolder}/autocorpus/configs/config_pmc_pre_oct_2024.json",
+        "-b",
+        "LEGACY_PMC",
         "-t",
         "output",
         "-f",
-        "${workspaceFolder}/tests/data/PMC/Pre-Oct-2024/PMC8885717.html"
+        "${workspaceFolder}/tests/data/public/html/LEGACY_PMC/PMC8885717.html"
       ]
     }
   ]

README.md

@@ -5,6 +5,7 @@
 [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
 [![pre-commit.ci status](https://results.pre-commit.ci/badge/github/omicsNLP/Auto-CORPus/main.svg)](https://results.pre-commit.ci/latest/github/omicsNLP/Auto-CORPus/main)
 [![PyPI version](https://badge.fury.io/py/autocorpus.svg)](https://badge.fury.io/py/autocorpus)
+[![codecov](https://codecov.io/gh/omicsNLP/Auto-CORPus/graph/badge.svg?token=ZTKK4URM4A)](https://codecov.io/gh/omicsNLP/Auto-CORPus)
 
 *Requires Python 3.10+* <!-- markdownlint-disable-line MD036 -->
 
@@ -21,24 +22,32 @@ The documentation for Auto-CORPus is available on our [GitHub Pages site].
 
 ## Installation
 
-Install with pip
+Install with pip:
 
 ```sh
 pip install autocorpus
 ```
 
+If you want to be able to process PDF files (only available with Auto-CORPus >v1.1.0),
+you will need to install (large!) additional dependencies. To install Auto-CORPUS with
+PDF processing support, run:
+
+```sh
+pip install autocorpus[pdf]
+```
+
 ## Usage
 
-Run the below command for a single file example
+You can run Auto-CORPus on a single file like so:
 
 ```sh
-auto-corpus -c "autocorpus/configs/config_pmc.json" -t "output" -f "path/to/html/file" -o JSON
+auto-corpus -b PMC -t "output" -f "path/to/html/file" -o JSON
 ```
 
-Run the main app for a directory of files example
+Auto-CORPus can also process whole directories:
 
 ```sh
-auto-corpus -c "autocorpus/configs/config_pmc.json" -t "output" -f "path/to/directory/of/html/files" -o JSON
+auto-corpus -b PMC -t "output" -f "path/to/directory/of/html/files" -o JSON
 ```
 
 ### Available arguments
@@ -118,12 +127,23 @@ To get started:
 
 1. [Download and install Poetry](https://python-poetry.org/docs/#installation) following the instructions for your OS.
 1. Clone this repository and make it your working directory
+1. (Optionally) download private test data for additional regression tests. This uses data which
+   cannot be redistributed publicly (only available to members of the
+   [omicsNLP](https://github.com/omicsNLP) organisation).
+
+   ```sh
+   git submodule update --init
+   ```
+
 1. Set up the virtual environment:
 
    ```sh
-   poetry install
+   poetry install --all-extras
    ```
 
+   Note: The `--all-extras` flag is because of the additional dependencies required for
+   analysing extra file types (PDF, Word, Excel, etc).
+
 1. Activate the virtual environment (alternatively, ensure any Python-related command is preceded by `poetry run`):
 
    ```sh

autocorpus/__main__.py

@@ -7,8 +7,7 @@
 from tqdm import tqdm
 
 from . import add_file_logger, logger
-from .autocorpus import Autocorpus
-from .configs.default_config import DefaultConfig
+from .config import DefaultConfig, read_config
 from .inputs import read_file_structure
 from .run import run_autocorpus
 
@@ -66,7 +65,7 @@ def main():
 
     # Load the config
     if args.config:
-        config = Autocorpus.read_config(args.config)
+        config = read_config(args.config)
     elif args.default_config:
         try:
             config = DefaultConfig[args.default_config].load_config()

autocorpus/abbreviation.py

@@ -35,43 +35,27 @@ def _remove_quotes(text: str) -> str:
     return re2.sub(r'([(])[\'"\p{Pi}]|[\'"\p{Pf}]([);:])', r"\1\2", text)
 
 
-def _is_abbreviation(candidate: str) -> bool:
-    r"""Check whether input string is an abbreviation.
+def _is_abbreviation(s: str) -> bool:
+    """Check whether input string is an abbreviation.
 
-    Based on Schwartz&Hearst.
+    To be classified as an abbreviation, a string must be composed exclusively of
+    Unicode letters or digits, optionally separated by dots or hyphens. This sequence
+    must repeat between two and ten times. We exclude strings that are *exclusively*
+    composed of digits or lowercase letters.
 
-    2 <= len(str) <= 10
-    len(tokens) <= 2
-    re.search(r'\p{L}', str)
-    str[0].isalnum()
-
-    and extra:
-    if it matches (\p{L}\.?\s?){2,}
-    it is a good candidate.
-
-    Args:
-        candidate: Candidate abbreviation
-
-    Returns:
-        True if this is a good candidate
+    Adapted from Schwartz & Hearst.
     """
-    viable = True
+    # Disallow if exclusively composed of digits
+    if re2.match(r"\p{N}+$", s):
+        return False
 
-    # Broken: See https://github.com/omicsNLP/Auto-CORPus/issues/144
-    # if re2.match(r"(\p{L}\.?\s?){2,}", candidate.lstrip()):
-    #     viable = True
-    if len(candidate) < 2 or len(candidate) > 10:
-        viable = False
-    if len(candidate.split()) > 2:
-        viable = False
-    if candidate.islower():  # customize function discard all lower case candidate
-        viable = False
-    if not re2.search(r"\p{L}", candidate):  # \p{L} = All Unicode letter
-        viable = False
-    if not candidate[0].isalnum():
-        viable = False
+    # Disallow if exclusively composed of lowercase unicode chars
+    if re2.match(r"\p{Ll}+$", s):
+        return False
 
-    return viable
+    # Should be a repeating sequence of unicode chars or digits, optionally separated
+    # by dots or hyphens. The sequence must repeat between 2 and 10 times.
+    return bool(re2.match(r"([\p{L}\p{N}][\.\-]?){2,10}$", s))
 
 
 def _get_definition(candidate: str, preceding: str) -> str:
@@ -398,7 +382,7 @@ def _extract_abbreviations(
 
 
 def _biocify_abbreviations(
-    abbreviations: _AbbreviationsDict, file_path: str
+    abbreviations: _AbbreviationsDict, file_path: Path
 ) -> dict[str, Any]:
     passages = []
     for short, long in abbreviations.items():
@@ -416,16 +400,16 @@ def _biocify_abbreviations(
         "key": "autocorpus_abbreviations.key",
         "documents": [
             {
-                "id": Path(file_path).name.partition(".")[0],
-                "inputfile": file_path,
+                "id": file_path.name.partition(".")[0],
+                "inputfile": str(file_path),
                 "passages": passages,
             }
         ],
     }
 
 
 def get_abbreviations(
-    main_text: dict[str, Any], soup: BeautifulSoup, file_path: str
+    main_text: dict[str, Any], soup: BeautifulSoup, file_path: Path
 ) -> dict[str, Any]:
     """Extract abbreviations from the input main text.
 

autocorpus/ac_bioc/annotation.py

@@ -3,7 +3,7 @@
 from __future__ import annotations
 
 import xml.etree.ElementTree as ET
-from dataclasses import dataclass, field
+from dataclasses import asdict, dataclass, field
 from typing import Any
 
 from .location import BioCLocation
@@ -20,19 +20,7 @@ class BioCAnnotation:
     infons: dict[str, str] = field(default_factory=dict)
     locations: list[BioCLocation] = field(default_factory=list)
 
-    def to_dict(self):
-        """Convert the annotation to a dictionary representation.
-
-        Returns:
-            dict: A dictionary containing the annotation's id, text, offset, length, and infons.
-        """
-        return {
-            "id": self.id,
-            "text": self.text,
-            "offset": self.offset,
-            "length": self.length,
-            "infons": self.infons,
-        }
+    to_dict = asdict
 
     def to_json(self) -> dict[str, Any]:
         """Convert the annotation to a JSON-serializable dictionary.

autocorpus/ac_bioc/bioctable/cell.py

@@ -9,12 +9,3 @@ class BioCTableCell:
 
     cell_id: str = field(default_factory=str)
     cell_text: str = field(default_factory=str)
-
-    def to_dict(self) -> dict[str, str]:
-        """Convert the cell's attributes to a dictionary.
-
-        Returns:
-            dict[str, str]
-                A dictionary containing the cell's ID and text content.
-        """
-        return {"cell_id": self.cell_id, "cell_text": self.cell_text}

autocorpus/ac_bioc/bioctable/collection.py

@@ -1,7 +1,6 @@
 """This module defines the BioCTableCollection class."""
 
 from dataclasses import dataclass, field
-from typing import Any
 
 from ...ac_bioc import BioCCollection, BioCDocument
 
@@ -11,13 +10,3 @@ class BioCTableCollection(BioCCollection):
     """A collection of BioCTableDocument objects extending BioCCollection."""
 
     documents: list[BioCDocument] = field(default_factory=list)
-
-    def to_dict(self) -> dict[str, Any]:
-        """Convert the BioCTableCollection to a dictionary representation.
-
-        Returns:
-            dict[str, Any]: A dictionary containing the collection's data, including its documents.
-        """
-        base = super().to_dict()
-        base["documents"] = [doc.to_dict() for doc in self.documents]
-        return base

autocorpus/ac_bioc/bioctable/document.py

@@ -5,10 +5,8 @@
 """
 
 from dataclasses import dataclass, field
-from typing import Any
 
 from ...ac_bioc import BioCAnnotation, BioCDocument, BioCPassage
-from ...ac_bioc.bioctable.passage import BioCTablePassage
 
 
 @dataclass
@@ -17,11 +15,3 @@ class BioCTableDocument(BioCDocument):
 
     passages: list[BioCPassage] = field(default_factory=list)
     annotations: list[BioCAnnotation] = field(default_factory=list)
-
-    def to_dict(self) -> dict[str, Any]:
-        """Convert the BioCTableDocument to a dictionary representation."""
-        base = super().to_dict()
-        base["passages"] = [
-            p.to_dict() for p in self.passages if isinstance(p, BioCTablePassage)
-        ]
-        return base

autocorpus/ac_bioc/bioctable/json.py

@@ -68,7 +68,6 @@ def default(self, o):
                     "source": o.source,
                     "date": o.date,
                     "key": o.key,
-                    "version": o.version,
                     "infons": o.infons,
                     "documents": [self.default(d) for d in o.documents],
                 }