replace custom UDF with spark from_unixtime for ctime transformation

[elliVM]

Description

Use spark's functions.from_unixtime to parse epochs into a timestamp string replacing the old custom UDF.
Resulting timestamp strings are now returned in ISO 8601 standard and include zone information.

Example:
epoch 1 results into 1970-01-01T00:00:01Z.

I chose this format but we can use that is supported by the DateTimeFormatter. The old format was not supported

[eemhu]

Doesn't this remove the support for the timeformat parameter?

[elliVM]

Hey @eemhu

Indeed this change does remove the support for a custom timeformat. But when you look at how the timeformat is given to the call it is set directly in the AbstractConvertStep class, in addition that format was not of any standard.

From this I concluded that the feature was not used but please correct me if I was wrong.

Sparks from_unixtime does support different formats so it can be added to the call method if we think it's better that way?

[eemhu]

Hey @eemhu

Indeed this change does remove the support for a custom timeformat. But when you look at how the timeformat is given to the call it is set directly in the AbstractConvertStep class, in addition that format was not of any standard.

From this I concluded that the feature was not used but please correct me if I was wrong.

Sparks from_unixtime does support different formats so it can be added to the call method if we think it's better that way?

The format set directly in AbstractConvertStep is the DPL default timeformat. It was used if none other was specified with the timeformat parameter. I don't think it should be removed.

[elliVM]

Okey thank you, I will refactor so that different time formats are supported including default is supported