Idea for RFC for populating parameter defaults from environment variables

[mwiebe]

The idea

Provide a way for <Environment> constructs to pass-through environment variables from the job submission context, and for constructs to substitute environment variables from the job submission context in their default parameter values.

Background

Many workloads use environment variable values to control aspects of setup or process runtime. Currently, OpenJD jobs cannot express that they depend on environment variables defined from the job submission context, so require any job submission tool to invent its own method of determining when one is needed and forwarding it to a job parameter. Adding a way for a job template to request specific named environment variables, and then directly

For example Rez defines many environment variables that provide information about the active virtual environment. Consider the variables REZ_RXT_FILE, REZ_USED_REQUEST, and REZ_USED_RESOLVE. These are expressions about the contents of the environment, and how to reproduce it - first is a filename where the file contents contain that data, and the others are values that contains the environment request string in its original or resolved platform-specific form. These suggest two requirements: providing a value directly, and also providing a filename that becomes part of the input dataflow of the job. See the deadline-cloud feature request aws-deadline/deadline-cloud#788 requesting propagation of the REZ_USED_RESOLVE variable.

The equivalent in Conda isn't directly accessible via an environment variable. The closest is the command conda env export --from-history that reads the file $CONDA_PREFIX/conda-meta/history and creates an environment file including the specs used to create it. That environment file the command creates has the absolute path of the prefix, so it's not a portable representation of the environment. We could fit this into the same pattern as Rez if we use that history file as the input for a conda queue environment, in which case the requirement for this feature includes the ability to specify a subpath of an environment variable path as an input to the job, not just the environment variable path directly.

Another important use case common in rendering is preserving the OpenColorIO settings from the $OCIO variable. This variable contains a file path to a configuration file, fitting into the same case as REZ_RXT_FILE.

Guardrails

Environment variables can contain credentials, so it will be important for implementations of this feature to not allow job submission to freely get any environment variable values. The answer is likely to use a configurable allowlist that is pre-populated with a list of known use cases such as the names described in the background.

Example template sketch

We define a new top-level template entity context whose purpose is to define/request context for the job definition. The use case here is requested environment variable values, and other contexts that fit here could be temporary scratch space file system locations and properties of the job like whether the job requires internet access to run. We publish values from this context in format string substitution as {{Context.*}}.

In an <Environment> constructs:

specificationVersion: environment-2023-09
extensions: SUBMISSION_ENV_VARS
context:
  envVars:
    - name: PROJECT_NAME
      required: True    # Will fail job submission if missing
      forward: True    # Will forward this variable into the job run context
    - name: OCIO
      required: False  # Will be empty string if missing
      forward: True    # Will forward this variable into the job run context
      type: PATH       # This is a PATH just like in a parameter definition
      objectType: FILE   # The value must be a file path (DIRECTORY for directory path)
      dataFlow: IN        # The file at this location will be read as an input of the job
    - name: REZ_USED_REQUEST
      required: False    # Will be empty string if missing
      forward: False    # Will not forward this to the job env, the job must use this in a format string
    - name: REZ_RXT_FILE
      required: False    # Will be empty string if missing
      forward: False    # Will not forward this to the job env, the job must use this in a format string
    - name: CONDA_PREFIX
      required: False    # Will be empty string if missing
      forward: False    # Will not forward this to the job env, the job must use this in a format string
parameterDefinitions:
  - name: RezUsedRequest
    type: STRING
    default: "{{Context.EnvVars.REZ_USED_REQUEST}}"
  - name: RezRxtFile
    default: "{{Context.EnvVars.REZ_RXT_FILE}}"
    type: PATH
    objectType: FILE   # The value must be a file path (DIRECTORY for directory path)
    dataFlow: IN        # The file at this location will be read as an input of the job
  - name: CondaEnvHistory
    # We have some missing functionality to resolve - if CONDA_PREFIX is missing, this
    # becomes /conda-meta/history instead of an empty string. See the discussion
    # https://github.com/OpenJobDescription/openjd-specifications/discussions/79
    # for how we can expand format strings to help with this.
    default: "{{Context.EnvVars.CONDA_PREFIX}}/conda-meta/history"
    type: PATH
    objectType: FILE   # The value must be a file path (DIRECTORY for directory path)
    dataFlow: IN        # The file at this location will be read as an input of the job

An example for a job template would be almost identical.