CD: Automate the activation of the self hosted runner

[barshaul]

In order to automatically activate the runner, we did:

1. SSM installed on the EC2 host
2. Created an SSM document to start the runner process on the host
3. IAM user was created for github actions, keys were stored as secrets in github
4. The CD jobs will wait for the runner to be activated
5. The CD job will change ownership permissions for the _work folder to allow checking out the repository (see Permission denied when "Deleting the contents of" actions/checkout#211 (comment))

https://github.com/machulav/ec2-github-runner/ haven't been used since it requires an org:admin access in order to generate a registration key for the runner, which aws org doesn't provide.

Closes #656

[avifenesh]

Im not sure it is a real problem, but i think its need to be looked at - i noticed while running the self hosted runner that its not kill itself after theres no tasks anymore, and then if try to rerun the runner while the previous task is still alive in the background its throwing errors.
So it might need a different approach - perhaps running just per specific task somehow, adding a kill command after runs finished or checking before re-running if a runner is already alive.

[barshaul]

r

Im not sure it is a real problem, but i think its need to be looked at - i noticed while running the self hosted runner that its not kill itself after theres no tasks anymore, and then if try to rerun the runner while the previous task is still alive in the background its throwing errors. So it might need a different approach - perhaps running just per specific task somehow, adding a kill command after runs finished or checking before re-running if a runner is already alive.

Do you have logs of these errors you were seeing? the runner knows how to handle multiple processes trying to activate it, and the only issue I saw was with EACCESS error while trying to checkout the repo, which this PR solves

[avifenesh]

r

Im not sure it is a real problem, but i think its need to be looked at - i noticed while running the self hosted runner that its not kill itself after theres no tasks anymore, and then if try to rerun the runner while the previous task is still alive in the background its throwing errors. So it might need a different approach - perhaps running just per specific task somehow, adding a kill command after runs finished or checking before re-running if a runner is already alive.

Do you have logs of these errors you were seeing? the runner knows how to handle multiple processes trying to activate it, and the only issue I saw was with EACCESS error while trying to checkout the repo, which this PR solves

My issue was that for some reason it didn't recognize new tasks, so i ctrl+c thinking it will kill it, tried to rerun and got errors telling me that it already run, but still don't run new tasks. So i killed manually the process and started a new runner, which solve my issue.
Unfortunately didn't save the logs since i didn't saw any reason to.

Please tag me if you response in a separate comment so ill get notified.
@barshaul

[barshaul]

r

Im not sure it is a real problem, but i think its need to be looked at - i noticed while running the self hosted runner that its not kill itself after theres no tasks anymore, and then if try to rerun the runner while the previous task is still alive in the background its throwing errors. So it might need a different approach - perhaps running just per specific task somehow, adding a kill command after runs finished or checking before re-running if a runner is already alive.

Do you have logs of these errors you were seeing? the runner knows how to handle multiple processes trying to activate it, and the only issue I saw was with EACCESS error while trying to checkout the repo, which this PR solves

My issue was that for some reason it didn't recognize new tasks, so i ctrl+c thinking it will kill it, tried to rerun and got errors telling me that it already run, but still don't run new tasks. So i killed manually the process and started a new runner, which solve my issue. Unfortunately didn't save the logs since i didn't saw any reason to.

Please tag me if you response in a separate comment so ill get notified. @barshaul

@avifenesh
I don't want to kill the process at the end of the job, as when we dispatch multi tasks with the same PR (as we do here), some tasks are being queued waiting for the runner on the publish-binaries job. If we'll kill the runner, the queued job that waits on the runner will never go back to the activation job.
I haven't saw this issues in my multiple tests with this automation, so I wonder if you're issue really was that old action was still running / stuck and therefore new actions wasn't picked up until you killed the process.
Anyway, if we indeed find this issue later on we can then debug it and fix it.

[shachlanAmazon]

minor: you probably don't need the submodules.

[shachlanAmazon]

same