CORERULES-9552 - checking values against Library CT for SDTM

[SFJohnson24]

This is the first part of the generalized library CT check operator that will crosscheck values against codelist values

1. The operator will check the rule type, if it is a 'value check againbst define XML' we are SDTM operator,
   if it is a 'value check - metadata' we are DDF as this will give us access to the values to compare to codelists.

2. the version of the CT package will be specified in CLI command or found in the defineXML - Codelist - Standard - publishingSet/Version:
   def:Standards
   <def:Standard OID="STD.1" Name="STDTMIG" Type="IG" Version="3.3" Status="Final" def:CommentOID="COM.ST1"/>
   <def:Standard OID="STD.2_1" Name="SDTMIG-MD" Type="IG" Version="1.1" Status="Final" def:CommentOID="COM.ST2"/>
   <def:Standard OID="STD.4" Name="CDISC/NCI" Type="CT" PublishingSet="DEFINE-XML" Version="2020-12-18" Status="Final" def:CommentOID="COM.ST4"/>

3. sdtm gets from library differently - uses define and library metadata; ddf uses schema for library. with SDTM, will need to check if it is non-extensible. if it is extensible, not in list of CT but in defineXML:

4. we will want to cache codelist values, if there are mutiple codelists will need the union of them

5. this operator should be able to check both for valid "submission values" and for valid C-codes for the terms in the specified codelist.

see #794 for the DDF implementation and notes from initial ticket

[SFJohnson24]

this description for #2 is define 2.1, add that it is only applicable to define 2.1 in operator description

[SFJohnson24]

the functionality for extensibility and CT lookup are on ticket #794. To resolve this ticket, we just need the XPT extensibility operator. It would also be good at add functionality with define 2.1--to get the CT packages from it and then grab them and bring them into CT map as per the 2nd bullet above

[SFJohnson24]

per @eljanssens
As discussed during the Office Hours on Tuesday July 29th, the operation 'get_codelist_attributes' needs to be revised.

Both ct_version and ct_packages are requested. However, CT is normally selected in the validation itself. ct_version for CG0288 makes sense as it can retrieve the version from the value in TSVCDVER. We also need clarity on what should be placed in 'name' since the current example states: TSVCDREF which doesn't make sense for CG0288. This rule should check all codes for terms in the CDISC CT and not just in a specific codelist. If 'name' is the codelist from which it needs to retrieve the atttributes, then an 'ALL' option should be made available. Some more information on what the possible values are for ct_attribute could help: Term CCODE, Codelist CCODE, Term VALUE, Codelist VALUE? Also it needs to be checked if there isn't overlap with 'codelist_terms' as I think they both do the same. Could there be alignment and clarifiation? Thanks!

@SFJohnson24 : can this ticket please be reopened? I tried to figure out how you all intended it to work but none of the options seem to work.

[SFJohnson24]

@eljanssens does #1419 resolve this issue with the codelist_attribute rework?

[eljanssens]

@SFJohnson24: I don't see the relevance of specifying an exact ct_package including a version in your rule because that would make your rule very specific to a certain CT in the code itself while you already specify it in the engine for validation. In the case of CG0288, the CT version should actually be picked up by the value in the TSVCDVER and not by specifying a ct_package.
For CG0288, I want the operation to get me all the ccodes from the sdtm CT for the version that is in TSVCDVER and then check if the code that is now in TSVALCD matches one in that list.