Enhance UTF Handling in JSON Dataset Reader

[SFJohnson24]

The dataset reader currently assumes utf8 and ascii characters, which causes issues when parsing JSON files containing international characters. Since JSON is inherently Unicode (UTF-8 by default), we need to ensure that the dataset reader properly handles UTF-8, UTF-16, and UTF-32 encoded data.

A/C: The dataset reader should be able to parse and process datasets containing international characters without issues.

• Update the all dataset reader to explicitly handle Unicode encoding.
• Ensure proper decoding when reading files.
• Add test cases with non-ASCII characters (Chinese, Japanese, etc.).
• both Test and validate commands should work

Errors:
this is the error with validate

this is the error with test

DatasetJSON:
ae_nonascii.zip
Test Command JSON
TestDatasets.zip

code:
https://github.com/cdisc-org/cdisc-rules-engine/tree/main/cdisc_rules_engine/services/data_readers here we use uft-8 in the reader class (also in metadata reader classes)

cdisc-rules-engine/cdisc_rules_engine/services/data_services/dummy_data_service.py

Line 60 in b93b1b6

def get_dataset(self, dataset_name: str, **params) -> PandasDataset:

here we use utf-8 for JSON (from -lr flag)

[SFJohnson24]

in discussing this-- we may want to allow users to specify certain encoding types as a validation/test argument. This would impact the metadata reader/reader classes and tell them how the submitted files are encoded.

[SFJohnson24]

I am reopening this issue as the above PR does not fully resolve this. It does not allow users to specify encoding and there are still hardcoded utf-8 portions of the codebase within metadata readers as well as a few other places

[SFJohnson24]

I have test data for this nonascii.zip
This is an XPT file with non-ascii characters-- it has smart quotes on several lines. Technically it is mostly UTF-8 but has smart quotes that were ported from excel which was used to make the XPT data.