Feature request: provenance tracking

[edbennett]

For reproducibility and FAIR reasons, it would be very useful to be able to track provenance of files generated by Grid, for example using an implementation of the PROV specification. Specific suggestions:

• Where an input file has been read (e.g. using Grid::XmlReader), have a prov attribute, minimally containing the file path of the input file and last modification time, but also including any provenance information in the input file (e.g. PROV-XML elements) if they are present.
• Where an output file is generated (e.g. using Grid::XmlWriter), include provenance information (again, e.g. PROV-XML elements). Minimally this would be attributes like a timestamp, machine name, username, Grid commit ID, name of executable, build parameters.
  • Any input files that had been read in could be listed here, with their respective provenance information.
  • There should ideally be the facility for downstream tooling to add additional attributes. For example, a tool in its own repository should be able to add its own commit ID, build parameters, etc. (Tooling to make this easy to do for downstream developers would be even nicer.)
• For configuration formats that support metadata, there should be the facility to provide it, and if it not mandatory then the defaults should be obviously invalid. (Currently for example NerscIO defaults to an ensemble label of "DWF" and ensemble ID of "UKQCD", which may be misleading.) HMC checkpoints should expose the same controls.

Is this something that would be of interest? If so then I may be able to put some development time towards it.