Request to add a CLI option to filter the filetype of files to process

[mhungria]

Request to add a CLI option to filter the file type of files to process from a given folder (with -d option only).
Note: discussed at the CORE meeting today 2025-11-18.

The current CLI functionality provides an error message if there is more than one file type of the file types accepted for validation. It indicates that there are multiple formats in the folder.

**
A CLI option like the one below, would address that issue:
--filetype "json"
**

Or maybe, one more generic:
--subset_files "*.json" (one can add an regex to subset both filenames and filetypes).

Update on 2025-12-10 after a call with @alexfurmenkov :

1) This is the error from an example CLI execution when more that one file types are in a folder requested for validation:

(venv) marcelinahungria@Marcelinas-MBP cdisc-rules-engine % python3 core.py validate -s sdtmig -v 3-4 -d /Users/marcelinahungria/Documents/W10X1/Clients/Internal/DIcG_git/DIcGweb/data/examples/sdtm --output /Users/marcelinahungria/Documents/W10X1/Clients/Internal/DIcG_git/DIcGweb/data/examples/sdtm/tempstats/val_reports/core_val_data_examples_sdtm_20251210 --output-format XLSX -lr /Users/marcelinahungria/Documents/W10X1/core_env/Rules/DraftRules20250825subset -l error

[ERROR 2025-12-11 11:08:46,716 - core.py:283] - Argument --data contains more than one allowed file format (NDJSON, XPT, JSON).
(venv) marcelinahungria@Marcelinas-MBP cdisc-rules-engine %

Folder content (source data included in the Dataset-JSON v1.1 release, partial content below:
marcelinahungria@Marcelinas-MBP ~ % cd /Users/marcelinahungria/Documents/W10X1/Clients/Internal/DIcG_git/DIcGweb/data/examples/sdtm
marcelinahungria@Marcelinas-MBP sdtm % ls -la
total 55512
drwxr-xr-x@ 104 marcelinahungria staff 3328 Oct 22 16:27 .
drwxr-xr-x 8 marcelinahungria staff 256 Oct 23 14:38 ..
-rw-r--r--@ 1 marcelinahungria staff 14340 Dec 10 15:58 .DS_Store
-rw-r--r-- 1 marcelinahungria staff 6825094 Nov 13 2024 acrf.pdf
-rw-r--r--@ 1 marcelinahungria staff 22830 Nov 13 2024 ae.json
-rw-r--r--@ 1 marcelinahungria staff 25883 Nov 13 2024 ae.ndjson
-rw-r--r-- 1 marcelinahungria staff 38080 Nov 13 2024 ae.xpt
-rw-r--r-- 1 marcelinahungria staff 13251 Nov 13 2024 cm.json
-rw-r--r-- 1 marcelinahungria staff 14524 Nov 13 2024 cm.ndjson
-rw-r--r-- 1 marcelinahungria staff 39440 Nov 13 2024 cm.xpt

2) If the user specifies a conflicting option (i.e., -dp "xyz.json" -dp "xyz.xpt") the engine could continue providing the error above.

[mhungria]

Hi @RamilCDISC and @alexfurmenkov , could we reopen this ticket?

It appears to allow specifying filetype .json (I assume any of the others as well) and providing a specific .xlsx file as well. The command processes the .xlsx (see #1601 or #1596) and I think it bypasses the filetype option unless some information is provided with log option different from info. I would expect to get a message indicating the options are inconsistent. Or, do you consider it is just for documentation at the moment due to workload?

CLI command, (tested in linux, engine v0.14.2):
python3 core.py validate -s SDTMIG -v 3.4 -dv 2.1 -dxp .../tempv/data/define.xml -vx y -dp .../tempv/data/unit-test-coreid-CG0555-positive-wodm_in_datasets.xlsx --filetype json -o .../tempv/data/val_reports/bash_core_val_None_admintempval_2026-02-08 --output-format JSON --output-format XLSX -l info -p verbose_output -r CDISC.SDTMIG.CG0555 -r CDISC.SDTMIG.CG0556 -r CDISC.SDTMIG.CG0557 -r CDISC.SDTMIG.CG0558 -r CDISC.SDTMIG.CG0559 -lr .../tempv/rules

[mhungria]

Hi @RamilCDISC , @alexfurmenkov , @SFJohnson24 : I don't see the option --filetype documented in the v0.15 release yet.

Did you get a chance to review my last comment above?

[SFJohnson24]

@alexfurmenkov @RamilCDISC The readme was never updated for the new CLI command. Can you please resolve this.