438 run ogc process

[grallewellyn]

Purpose

• For this to be deployed we need to have the concept of logged in users and a PGT token associated with their account if we dont want new airflow deployments for each mission- I think probably the best way to do this is use the MAAP ADE because they already have the authentication handled for linking to PGT tokens when running jobs. PGT tokens reset every 60 days
• Future TODO is have the inputs field dynamically change once the user selects the process they would like to run
• I looked through the Airflow documentation and used AI heavily, and I dont think there is a simple way to do this. I was looking into an approach using a custom HTML plugin, but then you could not access the DAG via the standard Airflow input DAG page. I think that we should instead opt to use the MAAP UI for running jobs off of processes because it is much more advanced than airflow for the UI and we have a custom Jupyter extension just for honing the UI. The MAAP ADE will change the input selection based on the process that the user selected
• I attempted this with custom operators extended from BaseOperators, but I was struggling to find a good way to pass the kubernetes secrets to the operator since passing as secrets only works for the Kubernetes Pod Operator. I ultimately decided I will just revisit this once we figure out how we are getting the PGT token from the user (I think really the only way to do this is from the logged in users account because each user should have their own token). Here is my best attempt at this and I am not deleting this branch so you can see my other attempts: ee785a4 These are the different ways I tried to pass the token to the custom operators:
  • https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/secrets-backends/aws-ssm-parameter-store.html#ssm-parameter-store-secrets secrets backend SSM, possible but have to configure IAM for this so you don’t have access to all the SSM parameters
  • Airflow connections: https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html should look into this more but I could not get this working with an initial try
  • KubernetesHooks and BaseHooks- various errors
  • Users could pass the token as a DAG param, but this is not best practice and is also annoying for the user
• I went with the option extending custom operators from KubernetesPodOperators which is less ideal because it still requires maintaining a docker image, but has the advantage of reusability

Overall, this ticket was recreating the wheel a bit, but will be useful if we have a reason to want users to deploy processes using the Airflow UI over the MAAP ADE

Proposed Changes

• [ADD] A DAG to run a job for a process

Issues

#438

Testing

• Tested on my Airflow deployment and could successfully run a job for a process

[LucaCinquini]

Hi @grallewellyn : I tried to execute the "Run OGC Process" DAG with the dafault values and it failed, is that to be expected? (because of the MAAP problem you reported)
Also, can you add a script that can be used by everybody to register the DAG via the OGC API endpoint - something similar to utils/post_deployment.sh, but specific to this DAG?
Thanks.

[grallewellyn]

@LucaCinquini It should fail during the monitor job task phase because of the MAAP bug which is because none of the jobs are successfully completing. I triggered a job for you on luca-1 that submits the job successfully, but the job fails. You need to provide the correct inputs so that the job can complete successfully

I will add the script once I get the existing post_deployment.sh working, following up on slack

[grallewellyn]

I need to get the Unity developers MAAP account set up before I merge this and put the token into AWS SSM and correct the terraform file to point to the new token so that everyone is not using my test account token

Once this is merged into develop, I need to make sure that post_deployment_terraform.sh can run with run_ogc_process.json