WIP: Convert the model ipython notebook to a script. #31
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Extract a python script from the notebook, and publish it as build artifact.
… into convert-ipynb-to-py
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This WIP adds the capability to extract a python script from the notebook,
and save it as build artifact in Github. I open this PR to collect feedback / discuss possible approaches to releasing the notebook code in a format consumable in non-interactive mode.
The changes add a new
pytarget toMakefile(make py) that runsnbconverton the algo notebook. The resulting script can be executed on spark with the canonicalspark-submit [...] algorithm.py.I wanted to experiment with a flow that, after a successful build, publishes the resulting artifact to a generally available location.
I've been toying with CI and added
make pyto the github workflow. With this change we'd generate a new script at each PR / git push, that is available under the Artifacts pane of the Action UI.Unfortunately there is no straightforward way to generate the artifact URL for remote consumption.
Next I want to explore the capabilities of our internal Gitlab (pipelines are enabled), and eventually move the codebase there.