Bc usdm generator

[pendingintent]

Adds biomedical concepts JSON generator.
Includes new generator in UI
Adds biomedical concepts generator to full USDM JSON generator
Automated mappings of DSS in the background once a user assigns a biomedical concept to an activity
TE, TA, TV domain creation
ScheduledDecisionInstances added
Conditions added

[Copilot]

Pull request overview

Adds new export/generation capabilities around USDM + SDTM Trial Design Domains, and extends the study-timing model to support scheduled decision points and condition assignments. This fits into the SoA Workbench’s existing pattern of “DB-backed authoring → generators → downloadable artifacts/UI pages”.

Changes:

• Adds USDM JSON generator UI + download endpoints, including biomedical concepts generation and wiring into the full USDM document.
• Adds SDTM Trial Design Domains (TA/TE/TV) generators and UI/download endpoints (JSON + CSV).
• Adds ScheduledDecisionInstance + ConditionAssignment persistence, routers, UI sections, and USDM schedule timeline output support; adds background DSS auto-assignment tasks.

Reviewed changes

Copilot reviewed 34 out of 37 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/test_routers_usdm_json.py	Adds route/UI tests for USDM JSON generator page and downloads.
tests/test_routers_tdd.py	Adds route/UI tests for SDTM TDD (TA/TE/TV) UI + downloads.
tests/test_routers_decision_instances.py	Adds API + USDM integration tests for decision instances and condition assignments.
src/usdm/generate_usdm.py	Includes biomedicalConcepts in StudyVersion output.
src/usdm/generate_scheduled_decision_instances.py	New USDM generator for ScheduledDecisionInstance + nested Condition objects.
src/usdm/generate_schedule_timelines.py	Adds decision instances into schedule timeline instances[].
src/usdm/generate_biomedical_concepts.py	New USDM biomedical concepts generator with CDISC Library + DSS lookups.
src/usdm/generate_activities.py	Reuses shared _nz and BC-id lookup helper instead of local copies.
src/soa_builder/web/utils.py	Adds shared _nz, biomedical concept ID helper, condition helpers, and expands scheduled-instance lookup to include decision instances.
src/soa_builder/web/templates/usdm_json.html	New UI page listing USDM JSON components for download.
src/soa_builder/web/templates/tdd.html	New UI page listing TA/TE/TV domain downloads (JSON/CSV).
src/soa_builder/web/templates/study_timing.html	Adds UI sections for decision instances and condition assignments.
src/soa_builder/web/templates/edit.html	Updates instance terminology and export button labeling.
src/soa_builder/web/templates/decision_instances.html	New wrapper page embedding decision instances section partial.
src/soa_builder/web/templates/condition_assignments.html	New wrapper page embedding condition assignments section partial.
src/soa_builder/web/templates/base.html	Adds nav links for USDM JSON and TDD generation pages.
src/soa_builder/web/templates/activities.html	Adds “Auto-assign DSS” UI action and adjusts concept-refresh styling.
src/soa_builder/web/templates/_instances_section.html	Renames UI labels and removes timeline_id field from instances UI.
src/soa_builder/web/templates/_decision_instances_section.html	New UI partial for CRUD of scheduled decision instances.
src/soa_builder/web/templates/_condition_assignments_section.html	New UI partial for CRUD of condition assignments.
src/soa_builder/web/schemas.py	Adds Pydantic schemas for decision instances and condition assignments.
src/soa_builder/web/routers/usdm_json.py	New router for USDM JSON component UI + download endpoints.
src/soa_builder/web/routers/tdd.py	New router for SDTM TDD (TA/TE/TV) UI + JSON/CSV downloads.
src/soa_builder/web/routers/schedule_timelines.py	Injects decision instance / condition assignment data into study timing UI.
src/soa_builder/web/routers/instances.py	Switches to shared _nz helper.
src/soa_builder/web/routers/decision_instances.py	New router for decision instances CRUD + reorder endpoint + UI.
src/soa_builder/web/routers/condition_assignments.py	New router for condition assignments CRUD + UI.
src/soa_builder/web/routers/activities.py	Adds DSS auto-assign endpoint and background task support.
src/soa_builder/web/initialize_database.py	Creates new decision_instances and condition_assignment tables.
src/soa_builder/web/audit.py	Adds audit tables/writers for decision instances and condition assignments.
src/soa_builder/web/app.py	Includes new routers; adds background DSS lookup/persist behavior when concepts are assigned.
src/sdtm/generate_tv.py	New TV generator with timing/arm derivations.
src/sdtm/generate_te.py	New TE generator.
src/sdtm/generate_ta.py	New TA generator.
.gitignore	Ignores local dev artifacts.

Comments suppressed due to low confidence (7)

tests/test_routers_decision_instances.py:208

• This test claims ?decision_instance_uid= filters condition assignments, but it currently asserts len(data) == 2 even though two assignments were created with different decision_instance_uid values. Either change the assertion to expect only the matching assignment (and add filter support in the API), or adjust the docstring/test name to reflect that the endpoint is unfiltered.
  src/soa_builder/web/utils.py:505
• get_scheduled_activity_instance() now returns a UNION of both scheduled activity instances and scheduled decision instances, but its name and docstring still claim it only returns entries from the instances table. This is likely to confuse callers and can lead to invalid IDs being used where only activity instances are valid. Consider renaming it (e.g., get_scheduled_instances) or splitting into separate helpers for activity vs decision instances.

def get_scheduled_activity_instance(soa_id: int) -> Dict[str, str]:
    """
    Return Dictionary of {name: instance_uid} from instances table

    :param soa_id: soa identifier
    :type soa_id: int
    :return: {name: instance_uid}
    :rtype: Dict[str, str]
    """
    conn = _connect()
    cur = conn.cursor()
    cur.execute(
        """
        SELECT name,instance_uid FROM instances WHERE soa_id=?
        UNION
        SELECT name,instance_uid FROM decision_instances WHERE soa_id=?
        ORDER BY name
        """,

src/sdtm/generate_tv.py:166

• In the TV domain generator, VISITDY is converted to day-count when arms are linked, but the no-arm branch returns the raw ISO 8601 duration string (timing_val). This makes output inconsistent and contradicts the function docstring. Apply _iso_duration_to_days() in the no-arm branch as well (or intentionally output "" when not convertible).

            records.append(
                {
                    "STUDYID": study_id,
                    "DOMAIN": "TV",
                    "VISITNUM": order_index,
                    "VISIT": visit_name or "",
                    "VISITDY": timing_val or "",
                    "ARMCD": "",
                    "ARM": "",
                    "TVSTRL": tvstrl or "",
                    "TVENRL": tvenrl or "",
                }

src/soa_builder/web/routers/condition_assignments.py:207

• UI create handler redirects to "/ui/soa/{soa_id}/condition_assignment" (singular), but the list page route/template is "/ui/soa/{soa_id}/condition_assignments". This redirect will 404 after creating a condition assignment; update the redirect target to the plural path.

    return RedirectResponse(
        url=_redirect_url(request, f"/ui/soa/{int(soa_id)}/condition_assignment"),
        status_code=303,
    )

src/soa_builder/web/routers/decision_instances.py:89

• The decision instances UI template includes a "Default Condition" dropdown populated from instance_options, but ui_list_decision_instances doesn't pass instance_options in the template context. As a result, users can't select a default condition in the UI; provide the scheduled-instance options (and/or a dedicated decision-instance options list) in the context.

        {
            "request": request,
            "soa_id": soa_id,
            "decision_instances": decision_instances,
            "schedule_timelines_options": schedule_timelines_options,
            "epoch_options": epoch_options,

src/usdm/generate_usdm.py:175

• build_usdm() wraps most component generators in _safe(...), but biomedicalConcepts is generated without that guard. Because biomedical concept generation can fail due to network/API issues, a single failure will currently break the entire full USDM document. Use _safe("biomedicalConcepts", build_usdm_biomedical_concepts, soa_id) (or otherwise handle failures) to keep full generation resilient.
  tests/test_routers_usdm_json.py:38
• The USDM JSON UI now includes a "Biomedical Concepts" component, but this test still asserts only the previous 10 labels and omits that new row. Update the expected labels/count to include "Biomedical Concepts" so the test matches the UI.

[Copilot]

Pull request overview

Copilot reviewed 34 out of 37 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (19)

src/soa_builder/web/routers/condition_assignments.py:392

• Audit payload typo: key decision_intance_uid is misspelled, which makes the delete audit record inconsistent with create/update payloads (decision_instance_uid). Rename the key so audit consumers don’t have to special-case deletes.

    before = {
        "id": row[0],
        "condition_assignment_uid": row[1],
        "name": row[2],
        "label": row[3],
        "description": row[4],
        "condition": row[5],
        "decision_intance_uid": row[6],
        "condition_target_uid": row[7],

src/soa_builder/web/templates/_condition_assignments_section.html:2

• Heading text has a grammatical error: “Conditions Assignments” should be “Condition Assignments”.

<h2>Conditions Assignments for Study: {% if study_label %}{{ study_label }}{% else %}{{ study_name }}{% endif %}</h2>

src/usdm/generate_scheduled_decision_instances.py:85

• Decision instances are ordered by length(instance_uid), instance_uid, which ignores order_index (and the /decision_instances/reorder endpoint updates order_index). To make USDM output reflect UI ordering, sort by order_index, id similarly to list_decision_instances.
  src/soa_builder/web/routers/usdm_json.py:121
• This endpoint returns the raw exception text to clients (Failed to generate ...: {exc}), which can leak internal details. Log the exception (already done) but return a generic 500 message without embedding exc.

    try:
        data = _build(component, soa_id)
    except Exception as exc:
        logger.exception(
            "Failed to build USDM component %s for soa_id=%s", component, soa_id
        )
        raise HTTPException(500, f"Failed to generate {component}: {exc}") from exc
    filename = next(c[2] for c in _COMPONENTS if c[0] == component)

src/soa_builder/web/routers/tdd.py:96

• This endpoint returns the raw exception text to clients (Failed to generate ...: {exc}), which can leak internal details. Log the exception (already done) but return a generic 500 message without embedding exc.

    try:
        data = _build(domain, soa_id)
    except Exception as exc:
        logger.exception("Failed to build TDD domain %s for soa_id=%s", domain, soa_id)
        raise HTTPException(500, f"Failed to generate {domain}: {exc}") from exc
    filename = next(d[2] for d in _DOMAINS if d[0] == domain)
    payload = json.dumps(data, indent=2) + "\n"

src/soa_builder/web/routers/condition_assignments.py:162

• create_condition_assignment accepts decision_instance_uid but doesn’t validate that it refers to an existing ScheduledDecisionInstance in this SOA. Because the UI currently sources options from a union of activity+decision instances, it’s easy to persist an invalid association that will never appear under any decision instance in USDM export. Validate decision_instance_uid against decision_instances.instance_uid (and likewise validate condition_target_uid against the allowed instance tables) and return 400 on invalid UIDs.

def create_condition_assignment(soa_id: int, payload: ConditionAssignmentCreate):
    if not soa_exists(soa_id):
        raise HTTPException(404, "SOA not found")

    name = (payload.name or "").strip()
    if not name:
        raise HTTPException(400, "Condition name required")

    # Calculate next order_index and condition_assignment_uid
    conn = _connect()
    cur = conn.cursor()
    cur.execute(
        "SELECT COALESCE(MAX(order_index),0) FROM condition_assignment WHERE soa_id=?",
        (soa_id,),
    )
    next_ord = (cur.fetchone() or [0])[0] + 1
    cur.execute(
        "SELECT condition_assignment_uid FROM condition_assignment WHERE soa_id=? and condition_assignment_uid LIKE 'Condition_%'",
        (soa_id,),
    )
    existing_uids = [r[0] for r in cur.fetchall() if r[0]]
    used_nums = set()
    for uid in existing_uids:
        if uid.startswith("Condition_"):
            tail = uid[len("Condition_") :]
            if tail.isdigit():
                used_nums.add(int(tail))
            else:
                logger.warning(
                    "Invalid condition_assignment_uid format encountered (ignored): %s",
                    uid,
                )
    # Always pick max(existing) + 1, do not fill gaps
    next_n = (max(used_nums) if used_nums else 0) + 1
    new_uid = f"Condition_{next_n}"
    # Insert values for new condition into the condition_assignment table
    cur.execute(
        """
        INSERT INTO condition_assignment (soa_id,condition_assignment_uid,name,label,description,condition,
        decision_instance_uid,condition_target_uid,order_index) VALUES (?,?,?,?,?,?,?,?,?)
        """,
        (
            soa_id,
            new_uid,
            name,
            _nz(payload.label),
            _nz(payload.description),
            _nz(payload.condition),
            _nz(payload.decision_instance_uid),
            _nz(payload.condition_target_uid),
            next_ord,
        ),

src/soa_builder/web/routers/decision_instances.py:416

• Reorder endpoint does not record any audit entry, unlike the instances reorder endpoint which records a reorder audit payload (src/soa_builder/web/routers/instances.py:447-488). Consider recording a decision instance audit entry for reorder operations so changes are traceable.

@router.post("/soa/{soa_id}/decision_instances/reorder", response_class=JSONResponse)
def reorder_decision_instances(
    soa_id: int,
    order: List[int] = Body(..., embed=True),
):
    if not soa_exists(soa_id):
        raise HTTPException(404, "SOA not found")
    if not order:
        raise HTTPException(400, "Order list required")

    conn = _connect()
    cur = conn.cursor()
    cur.execute("SELECT id FROM decision_instances WHERE soa_id=?", (soa_id,))
    existing = {r[0] for r in cur.fetchall()}
    if set(order) - existing:
        conn.close()
        raise HTTPException(400, "Order contains invalid decision instance id")

    for idx, did in enumerate(order, start=1):
        cur.execute(
            "UPDATE decision_instances SET order_index=? WHERE id=?", (idx, did)
        )
    conn.commit()
    conn.close()
    return JSONResponse({"ok": True, "new_order": order})

src/usdm/generate_biomedical_concepts.py:287

• This generator performs blocking outbound HTTP requests to the CDISC Library API during USDM generation (per concept and per DSS lookup). When invoked via the UI download endpoints, this can significantly slow responses or fail due to network/auth issues. Consider relying on cached/stored concept+DSS metadata from the DB where possible, or making remote lookups optional/behind a flag.
  src/sdtm/generate_tv.py:166
• When there are no arms linked to the encounter, VISITDY is currently set to the raw timing value (e.g. "P1D") instead of the day count string produced by _iso_duration_to_days. This makes TV output inconsistent depending on linkage. Use the same conversion in the no-arms branch as well.

            records.append(
                {
                    "STUDYID": study_id,
                    "DOMAIN": "TV",
                    "VISITNUM": order_index,
                    "VISIT": visit_name or "",
                    "VISITDY": timing_val or "",
                    "ARMCD": "",
                    "ARM": "",
                    "TVSTRL": tvstrl or "",
                    "TVENRL": tvenrl or "",
                }

src/soa_builder/web/routers/condition_assignments.py:45

• list_condition_assignments doc/tests indicate filtering by decision_instance_uid via query string, but this endpoint doesn’t accept any query parameter and always returns all rows. Add an optional decision_instance_uid query param and include it in the WHERE clause when provided (or update the doc/tests to match the intended behavior).

# API endpoint to list conditions
@router.get(
    "/soa/{soa_id}/condition_assignments",
    response_class=JSONResponse,
    response_model=None,
)
def list_condition_assignments(soa_id: int):
    if not soa_exists(soa_id):
        raise HTTPException(404, "SOA not found")

    conn = _connect()
    cur = conn.cursor()
    cur.execute(
        """
        SELECT id, condition_assignment_uid, name, label, description, condition,
        decision_instance_uid, condition_target_uid, order_index FROM condition_assignment
        WHERE soa_id=? ORDER BY name, id
        """,
        (soa_id,),
    )

src/soa_builder/web/utils.py:505

• get_scheduled_activity_instance now unions decision instances as well as activity instances, but its name and docstring still say it returns only instances from the instances table. Update the docstring/name to reflect the broader behavior to avoid misuse elsewhere.

def get_scheduled_activity_instance(soa_id: int) -> Dict[str, str]:
    """
    Return Dictionary of {name: instance_uid} from instances table

    :param soa_id: soa identifier
    :type soa_id: int
    :return: {name: instance_uid}
    :rtype: Dict[str, str]
    """
    conn = _connect()
    cur = conn.cursor()
    cur.execute(
        """
        SELECT name,instance_uid FROM instances WHERE soa_id=?
        UNION
        SELECT name,instance_uid FROM decision_instances WHERE soa_id=?
        ORDER BY name
        """,

src/usdm/generate_biomedical_concepts.py:45

• Comment typo: "GLobal" should be "Global".
  tests/test_routers_usdm_json.py:39
• test_ui_usdm_json_contains_all_components doesn’t assert the new “Biomedical Concepts” component even though the UI now lists it. Update the expected labels (and docstring count) so the test fails if the new component row is missing.
  tests/test_routers_decision_instances.py:208
• This test claims ?decision_instance_uid= filters results, but it currently asserts len(data) == 2 even though one assignment was created for di1 and one for di2. If filtering is implemented, the correct expectation should be a single result for di1 (and assertions should verify all returned rows match di1).
  tests/test_routers_decision_instances.py:144
• Docstring mismatch: the UID prefix used by the API is Condition_ (as asserted below), but this docstring says ConditionAssignment_2. Update the docstring to match the actual UID format.
  src/usdm/generate_biomedical_concepts.py:262
• This query selects dss_href from activity_concept, but fresh DB initialization creates href (and DSS columns are added only via migrations). If the CLI generator is run without migrations, this will raise no such column: dss_href. Consider checking PRAGMA table_info and selecting dss_href only when present (fallback to NULL / href).
  src/soa_builder/web/routers/decision_instances.py:96
• The decision instances UI template expects instance_options for the “Default Condition” select, but this handler doesn’t provide it in the template context. As a result the dropdown will be empty (or render incorrectly). Populate instance_options (e.g., via get_scheduled_activity_instance(soa_id)) in the TemplateResponse context.

    decision_instances = list_decision_instances(soa_id)
    schedule_timelines_options = get_schedule_timeline(soa_id)
    epoch_options = get_epoch_uid(soa_id)

    conn = _connect()
    cur = conn.cursor()
    cur.execute(
        "SELECT study_id, study_label, study_description, name, created_at FROM soa WHERE id=?",
        (soa_id,),
    )
    meta_row = cur.fetchone()
    conn.close()

    study_id, study_label, study_description, study_name, study_created_at = meta_row
    return templates.TemplateResponse(
        request,
        "decision_instances.html",
        {
            "request": request,
            "soa_id": soa_id,
            "decision_instances": decision_instances,
            "schedule_timelines_options": schedule_timelines_options,
            "epoch_options": epoch_options,
            "study_id": study_id,
            "study_label": study_label,
            "study_description": study_description,
            "study_name": study_name,
            "study_created_at": study_created_at,
        },
    )

src/soa_builder/web/templates/_decision_instances_section.html:173

• This template includes reorder JS that references #instances-table, data-instance-id, and POSTs to /soa/{id}/instances/reorder, but this page renders decision instances (data-decision-instance-id) and has a dedicated /decision_instances/reorder endpoint. This script is currently dead/misleading and will not work if a reorder button is added. Remove it or update it to target decision instances correctly.

<script>
document.addEventListener('DOMContentLoaded', function () {
    var btn = document.getElementById('save-instance-order-btn');
    if (btn) {
        btn.addEventListener('click', function () {
            var soaId = btn.getAttribute('data-soa-id');
            saveInstanceOrder(soaId);
        });
    }
});

function moveInstanceRow(button, dir) {
    var row = button.closest('tr');
    if (!row) return;
    var table = document.getElementById('instances-table');
    if (!table) return;
    var tbody = table.tBodies[0] || table;

    if (dir < 0) {
        var prev = row.previousElementSibling;
        if (prev && prev.querySelector && prev.querySelector('td')) {
            tbody.insertBefore(row, prev);
        }
    } else {
        var next = row.nextElementSibling;
        if (next) {
            tbody.insertBefore(next, row);
        }
    }
}

function saveInstanceOrder(soaId) {
    var rows = document.querySelectorAll('#instances-table tr[data-instance-id]');
    var order = [];
    for (var i = 0; i < rows.length; i++) {
        var id = parseInt(rows[i].getAttribute('data-instance-id'), 10);
        if (!isNaN(id)) {
            order.push(id);
        }
    }

    fetch('/soa/' + soaId + '/instances/reorder', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json;charset=UTF-8'
        },
        body: JSON.stringify({ order: order })
    }).then(function (response) {
        if (response.ok) {
            window.location.reload();
            return;
        }
        return response.text().then(function (text) {
            alert('Reorder failed: ' + (text || ('HTTP ' + response.status)));
        });
    }).catch(function (error) {
        alert('Reorder failed: ' + error.message);
    });
}
</script>

src/usdm/generate_schedule_timelines.py:247

• ScheduleTimeline.instances is built by concatenating activity instances and decision instances, which groups by type rather than timeline order. If ordering matters in downstream consumers, consider merging/sorting by a shared order_index (or another stable ordering key) so decision instances appear in the correct sequence relative to activity instances.