Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

drone-ssh command never returns in a particular task in a Tekton pipeline run #277

Open
yeeyaa opened this issue Jun 12, 2024 · 2 comments

Comments

@yeeyaa
Copy link

yeeyaa commented Jun 12, 2024

We are using drone-ssh in our tekton pipeline for remote execution of some commands on our servers. As advised by the tekton community: https://hub.tekton.dev/tekton/task/remote-ssh-commands.

However, in one of our tasks, the drone-ssh command is never returned which is causing the task to hang and eventually timeout.

From the log output, we can see the scripts passed to the drone-ssh command have been successfully executed. But it's just not returning and ending the drone-ssh command.

Could you please help identify where the issue might be? I'm using latest tag.

Same issue has been reported to tekton as well: tektoncd/catalog#1277

Task configuration in the pipeline:

    - displayName: Install PAC - BRT
      name: task-install-pac-brt-cluster
      params:
        - name: HOST
          value: $(params.pac-fyre-host)
        - name: USERNAME
          value: $(params.default-brt-login-user)
        - name: SSH_SCRIPT
          value: |
            #!/usr/bin/bash
            $(params.path-to-pipeline-scripts)/install_lsf_brt.sh
            sleep 60
            $(params.path-to-pipeline-scripts)/install_pac_brt.sh
      runAfter:
        - task-build-hpc-package
        - copy-repo-to-remote-host
      taskRef:
        kind: Task
        name: remote-ssh-commands
      timeout: 30m0s
      when:
        - input: $(params.should-run-install-pac-brt-cluster)
          operator: in
          values:
            - 'yes'
            - 'y'
            - 'YES'
            - 'yes'
      workspaces:
        - name: credentials

Task definition, as shown in the tekton catalog source here: https://github.com/tektoncd/catalog/blob/main/task/remote-ssh-commands/0.1/remote-ssh-commands.yaml

@yeeyaa
Copy link
Author

yeeyaa commented Jun 13, 2024

I'd done some further tests today and it turns out even if I am running the drone-ssh binary on its own, the session won't close.
Here's what I tested.

  1. Defined a script environment variable using
# export script="/path/to/my/script.sh
echo 'Finished script'
"
  1. Then I am running the latest drone-ssh to get connect to my remote host and run the script (which is present on the remote host).
./drone-ssh -H my-host -p 22 -u root -T 480m -s "$script" -i /path/to/my/id_ed25519 

Then I am able to execute the script.sh on the remote host. I can see from the console it reaches to the end of the script. However, it was at this point the session won't return and the drone-ssh process looking like hanging and doing nothing.

@yeeyaa
Copy link
Author

yeeyaa commented Jun 13, 2024

Not sure if this helps but it's some test I did.

I did another test, which in the first terminal window the drone-ssh runs to the end of the remote script. I opened another terminal window and executed same command.
Then after a couple of seconds, the drone-ssh process in the first window successfully returned and runs to a complete state.

But then the second window would run to the end and kept hanging there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant