Burn an AMI that's not yet listening to the redis queue, but is configured and prepared to do so

[amywieliczka]

Previously:

• To get a high-stage worker, you had to run create_worker.yml and then run provision_worker.yml with the high-stage parameter.
• To get a regular worker, you just had to run start_worker.yml

Now:

• To get a high-stage worker, you have to run start_worker.yml when no jobs are currently running or queued, ssh to the new worker, run /usr/local/bin/stop-rqworker.sh, modify /var/local/rqworker/rqw-settings.py, and then run /usr/local/bin/start-rqworker.sh
• To get a regular worker, you just run start_worker.yml

Proposed Solution:

• We maintain one worker AMI that is configured to listen to the redis queue, but is not actively listening, so that when a worker is started, it doesn't automatically start picking up jobs right away.
• We modify start_ami.yml to not only start the worker, but also, once the worker is up, instructs the worker to start listening to the redis queues: 'normal-stage', 'low-stage'
• We create a new ansible playbook such as start_high_stage_ami.yml to start the worker, and, once the worker is up, instruct the worker to start listening to the redis queue: 'high-stage'
• We modify create_worker_ami.yml to instruct the "template worker" to stop listening to the redis queue before burning a new AMI

[amywieliczka]

In working to develop the proposed solution, I created two new playbooks:

• create_worker_ami_without_deletion.yml is a copy of create_worker_ami.yml but removes the lines in create_worker_ami.yml that delete older AMIs/snapshots. this allowed me to experiment with creating new AMIs without deleting the currently working AMI used for harvesting jobs.
• start_ami_amy_test.yml is a copy of start_ami.yml but has a hard-coded experimental AMI ID. It also tries to run /usr/local/bin/start-rqworker.sh on the workers.

I followed these steps:

• ran start_ami.yml to create a "template worker"
• ssh'ed to the "template worker", modified the redis queue to high-stage, and ran /usr/local/bin/stop-rqworker.sh and logged out
• from hrv-stg ran create_worker_ami_without_deletion.yml passing the instance id of the "template worker"
• modified start_ami_amy_test.yml to use the hard-coded experimental AMI ID returned from the previous command
• from hrv-stg ran start_ami_amy_test.yml and was able to start a new worker

However, on running start_ami_amy_test.ytml, I would receive these ansible errors:

STAGE $ ansible-playbook ~/code/ansible/start_ami_amy_test.yml --extra-vars="count=1"

PLAY [localhost] ***************************************************************

TASK [start instance] **********************************************************
changed: [localhost]

TASK [Add all instance IPs to the host group] **********************************
fatal: [localhost]: FAILED! => {"failed": true, "msg": "Unexpected failure in finding the lookup named '{{ ec2_info.instances }}' in the available lookup plugins"}

NO MORE HOSTS LEFT *************************************************************
	to retry, use: --limit @/home/hrv-stg/code/ansible/start_ami_amy_test.retry

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=1   

'add all instance IPs to the host group' is a task that attempts to get the IP addresses of new workers and put them into an inventory which can later be used as the target for the start-rqworker.sh command.

Due to this failure, I expected the new workers to not be listening on the high stage queue, but they actually were. I'm not sure exactly how or why.

I then subsequently blew away my experimental AMI, and wrote this comment. More investigation is needed, but I'm not actually sure it's worthwhile to spend more time on this given the Rikolti work. For now, the current system outlined above does work to start a new high-stage worker, but has the awkward caveat that no one can be running jobs when these steps are ran.

Leaving this note here in case I decide to pick it back up again later.