Idea for RFC to extend types and the template substitution language

[mwiebe]

Open Job Description launched with a minimal template substitution syntax, that only can substitute untransformed literal text. It is a subset of jinja expressions, with the door open to either continue adding jinja syntax, or to select alternative expression syntax. Sticking to a jinja subset that's supported by multiple potential implementation, e.g. by the Rust minijinja library, seems like a good way to extend it. Adding list/array, bool, and likely optional types to the supported job parameter types along with the template language complements extending the syntax and many other ways to extend Open Job Description features, so I think it is good to bundle that in as well. Extension name TEMPLATE_EXPR.

Some motivational cases include:

1. Allowing simple math operations on parameters #49. With arithmetic expressions in template substitution, it becomes possible to tweak parameter input values to feed into different commands.
   1. E.g. "{{Param.FrameStart + Param.FramesPerTask - 1}} for the linked discussion or {{Task.Param.NodeIndex * 2}},{{Task.Param.NodeIndex * 2 + 1}} to form a range expression for dependencies in a node of a full binary tree applicable to the per-task dependencies case below.
2. Richer interpretation of a CHUNK[INT] value. We considered whether to Extend template substitution syntax for the CHUNK[INT] type, and deferred it. Having a chunk behave like a sorted list of integers, to query its length or get the first and last values would simplify templates that use chunking.
   1. E.g. for CHUNK[INT] task parameter Frame: {{ len(Task.Param.Frame) }} = task count in the chunk, {{ Task.Param.Frame[0] }} = smallest value in the chunk, {{ Task.Param.Frame[-1] }} = largest value in the chunk, {{ tojson(Task.Param.Frame) }} = chunk as a list of integers in JSON format "[1,2,3,5,7]" for chunk "1-3,5,7".
3. Providing a list of values to use as the range of a task parameter in a parameter space. Currently this kind of parameter space can only be hard-coded into the template.
4. Refining dependencies between steps to act on a per-task level instead, using template-defined values to specify the structure.
   1. A self-dependency of a step could include a list of values that matches the length of an Index task parameter, to dynamically specify a DAG of homogenous tasks. This needs the list job parameter type.
   2. A self-dependency of a step could include a template expression that is {{Task.Param.Index - 1}}, producing a serial chain of tasks.
   3. A render step depending on an export step can directly depend on the other's {{Task.Param.Index}} 1:1, to allow the render step to run per-task before the entire export step is done.
5. Feature Request: ability to skip a step in Deadline Cloud via boolean param #80. Using a boolean expression to determine whether a step runs or not. This would make it possible for the scheduler to know it can skip a step that would just determine that it didn't need to run and act as pure overhead.
   1. This is written up in idea discussion Idea for RFC to use an expression to include/exclude parts of a template #81
6. Provide functions to insert values with different escaping or path separator characters. E.g. {{ tojson(Param.StrValue) }} to print as a JSON object, or {{ path_to_posix(Param.PathValue) }} to always use / for the path separator.
7. Extend chunking to support chunking over the STRING and PATH type. See Task param space for files in a folder #48 (comment) for an example context. In this case, the task parameter chunk types CHUNK[STRING] and CHUNK[PATH] wouldn't have a range expression representation, they would have nearly the same interface as job parameter list types.
8. Expressions could be used in the userInterface job parameter metadata to show/hide controls depending on parameter values from other controls. Feature idea: conditionally show job parameter UI elements based on other parameter values #42
9. Provide lists of values as job parameters. For example a LIST[STRING] type that is forwarded to a task parameter would allow for choosing a list of cameras to render without having to embed the cameras in the job template itself. A LIST[PATH] type with IN data flow and FILE (or alternatively DIRECTORY) object type would allow for the submitter to more flexibly specify inputs as job parameters

Do you have more example use cases, either for job parameter types or for template substitutions?

[mwiebe]

Thinking a bit about implementation techniques for these template expressions. While we could use the jinja2 library, it has a lot more functionality than I think we want to pull in, and it would be nice to find a simple way to control what's permitted. The jinja sandbox documentation says "Access to attributes, method calls, operators, mutating data structures, and string formatting can be intercepted and prohibited." That's an opt-out approach, but it would be nicer to have an opt-in implementation where we explicitly specify permitted expressions compared to identifying and preventing the patterns we don't want.

The Python standard library ast module looks like it fits the bill for an implementation, as long as we keep the syntax as a subset of Python. There are other implementations of Python parsing available, for example for Rust there is rustpython-parser. For the userInterface use case, TypeScript/JavaScript will be important too, and looks like the OhmJS library has a Python grammar. Consider the following for a number of examples including from the motivational cases:

>>> import ast
>>> def parse(expr):
...     node = ast.parse(expr, mode="eval")
...     print(ast.dump(node, indent=2))
...

>>> parse("len(Task.Param.Frame)")
Expression(
  body=Call(
    func=Name(id='len', ctx=Load()),
    args=[
      Attribute(
        value=Attribute(
          value=Name(id='Task', ctx=Load()),
          attr='Param',
          ctx=Load()),
        attr='Frame',
        ctx=Load())],
    keywords=[]))

>>> parse("Task.Param.Frame[-1]")
Expression(
  body=Subscript(
    value=Attribute(
      value=Attribute(
        value=Name(id='Task', ctx=Load()),
        attr='Param',
        ctx=Load()),
      attr='Frame',
      ctx=Load()),
    slice=UnaryOp(
      op=USub(),
      operand=Constant(value=1)),
    ctx=Load()))

>>> parse("path_to_posix(Param.PathValue)")
Expression(
  body=Call(
    func=Name(id='path_to_posix', ctx=Load()),
    args=[
      Attribute(
        value=Name(id='Param', ctx=Load()),
        attr='PathValue',
        ctx=Load())],
    keywords=[]))

>>> parse("100 if Param.BigNumber else 1")
Expression(
  body=IfExp(
    test=Attribute(
      value=Name(id='Param', ctx=Load()),
      attr='BigNumber',
      ctx=Load()),
    body=Constant(value=100),
    orelse=Constant(value=1)))

We could implement a NodeVisitor subclass to handle processing of the permitted node types and evaluate the expression.

[mwiebe]

Created a refactoring PR to make experimenting with this idea easier: OpenJobDescription/openjd-model-for-python#182

[mwiebe]

I've written a set of three RFCs for this, and created a fully-functioning prototype implementation of it. Have a look and provide review feedback here: #113