Data files processing appears to read recursively on data path provided and its sub-folders, v0.15

[mhungria]

Standard

CDISC SDTM/SDTMIG

Reference Rule ID(s)

Not rule specific

Conformance Rule ID(s) (if published in CORE)

CORE-

JIRA Ticket

No response

CLI Command Used (if applicable)

python3 core.py validate -s SDTMIG -v 3.4 -dv 2.1 -dxp .../tempv/data/define.xml -vx y -d .../tempv/data --filetype json -o .../tempv/data/val_reports/bash_coreval_examples_sdtm_json_mhungria_SDTMIGv3.4_2026-02-26 --output-format JSON --output-format XLSX -l info -p verbose_output

Rule YAML

Authorities: 
  - Organization: 
    Standards:
      - Name: 
        References:
          - Citations:
              - Cited Guidance: 
                Document: 
                Item: 
                Section: 
            Origin: 
            Rule Identifier: 
              Id: 
              Version: 
            Version: 
        Version:
Check:
  all:
    - name: 
      operator: 
      value: 
Core:
  Id: 
  Status: 
  Version: '1'
Description: 
Executability: Fully Executable
Match Datasets:
  - Keys:
      - 
    Name: 
Operations:
  - id: 
    operator: 
Outcome:
  Message: 
  Output Variables:
    -
Rule Type: 
Scope:
Sensitivity:

Attach any sample test data file(s)

Dataset-JSON v1.1 published SDTM package (https://github.com/cdisc-org/DataExchange-DatasetJson/tree/master/examples/sdtm), copied to a temp folder that has sub-folders.

Attach any output report and/or log file(s)

Output shows dataset rows from files read recursively from sub-folders / output previously created, plus issues for those datasets:

35
filename "DM"
label "Demographics"
path ".../tempv/data"
modification_date "2025-10-26T11:43:15"
size_kb 7.985
length 18
36
filename ""
label ""
path "…/tempv/data_/val_reports_"
modification_date ""
size_kb 2.045
length

...

Expected output

Clean output just processing datasets read from the cli command -d provided, not from sub-folders.

Any Additional Information

I selected the standard as SDTM/SDTMIG as I tested with it; however, I think the issue is regardless of the standard selected.

Note: Executed on a linux box; however, I think the issue is a general issue, not specific to a given environment.

[gerrycampion]

@mhungria this is intentional and was updated here: #1322
We need this functionality so that the Engine can find files in cases like split files (refer to MSG). Like ./lb.xpt, ./split/lbch.xpt, ./split/lbhe.xpt, ./split/lbur.xpt. When you are doing a validation, I believe you shouldn't have data in subfolders that aren't part of your planned submission.
These temp files should be redirected to a different location. Do you disagree or have a scenario where you think this should work?

[mhungria]

Hi @gerrycampion:

1. It appears to be a misunderstanding on files added to the ./split folder in a submission, perhaps a misinterpretation of the MSG 2.0 recommendations. The main package is on the corresponding ectd data folder, and for validation purposes and treatment of 'split' files, defined by the metadata in the define.xml files in the corresponding ectd data folder, all the information needed to validate that data is self-contained in the folder itself.
2. Could you provide any conformance rule that refers to validating files in the ./split folder?
3. I would assume that if we would like to validate files from 2 different folders, we would issue a CLI command: python core.py validate -s <s> -v <v> -d <folder1 path> -d <folder2 path> ...
4. The Conformance Rules are not meant to be only for data submissions.
5. I think that forcing the users to follow a specific folder structure to use the engine is very restrictive and may imply separate copies of the data. I don't see the conceptual need for it.

Note: see also #1669

Regards,
Marcelina

[gerrycampion]

@mhungria Thanks, I see what you are saying. There are 2 scenarios:

1. When datasets are split because of size requirement, the unsplit dataset goes in the root folder and the define.xml. The split datasets go into the split folder but not the define.xml.
2. When datasets are split by sponsor decision, there is no unsplit dataset, and the split datasets go in the root folder and the define.xml.

There are rules that require checking split dataset names and SEQs like CG0017, CG0018, CG0028, CG0205, CG0332, CG0333, CG0334, CG0650,

It sounds like you are saying these rules only apply to the split/unsplit datasets that are in the root folder and we don't need to do any dataset name and SEQ checking for datasets in the split folder?
This is fine with me as long as the rule authors agree, but I would like to get input from @JozefAerts as he wrote the initial ticket that triggered the update to include directories recursively.

[mhungria]

Thanks for your quick reply @gerrycampion .

I'll wait for @JozefAerts comments before commenting further.