Integrating With Bitbucket to put integration tests build/fail results on bitbucket PR

[danielccdk]

If you are looking for help, please confirm the following...

• I confirm I have searched the Docs, Codemagic Sample Projects, and GitHub Discussions.
• I confirm the software versions on my local computer match the ones in CI environment (e.g. Xcode version, Gradle version). See here
• I confirm I am able to do a fresh clone of the repository and build the project on my local computer.

Which mobile framework are you using?

Flutter (Dart)

Steps to reproduce

Push changes to feature branch
Open Pull Request

Expected results

Results of codemagic build are pushed to Bitbucket PR

Actual results

None. Haven't integrated this yet.

The bitbucket repo is connected to codemagic and I am able to manually trigger integration test builds. However I would like to update the bitbucket pull request with the results of those tests.

I don't see much documentation on how to do this. Could you please advise. Here is my current codemagic.yaml.

workflows:
  integration_test_workflow:
    environment:
      flutter: "3.24.3"
    scripts:
      - name: Integration Tests 
        script: |
          flutter emulators --launch apple_ios_simulator
          flutter -d iPhone test integration_test/test_master.dart \
            --dart-define=isTesting=true \
            --dart-define=BUGSNAG_API_KEY=$BUGSNAG_API_KEY \
            --dart-define=CARFAX_OAUTH_CLIENT_ID=$CARFAX_OAUTH_CLIENT_ID \
            --dart-define=OKTA_ISSUER=$OKTA_ISSUER \
            --dart-define=OKTA_CLIENT_ID=$OKTA_CLIENT_ID \
            --dart-define=OKTA_REDIRECT_URI=$OKTA_REDIRECT_URI
    triggering:
      events:
        - pull_request
      branch_patterns:
      - pattern: 'dev'
        include: true
        source: false 

Please advise. Thank you so much

Build id (optional)

No response

[dam-ease]

Hello @danielccdk,
Updating Bitbucket pull request with test results from Codemagic is not directly supported by default, but you can achieve this using Bitbucket’s APIs in post-build. You can use the Pull Request Comments API to post the test results - https://developer.atlassian.com/cloud/bitbucket/rest/api-group-pullrequests/#api-repositories-workspace-repo-slug-pullrequests-pull-request-id-comments-post

POST https://api.bitbucket.org/2.0/repositories/{workspace}/{repo_slug}/pullrequests/{pull_request_id}/comments

Add a step at the end of the workflow to post the test results to the Bitbucket PR.

workflows:
  integration_test_workflow:
    environment:
      flutter: "3.24.3"
    scripts:
      - name: Integration Tests
        script: |
          flutter emulators --launch apple_ios_simulator
          flutter -d iPhone test integration_test/test_master.dart \
            --dart-define=isTesting=true \
            --dart-define=BUGSNAG_API_KEY=$BUGSNAG_API_KEY \
            --dart-define=CARFAX_OAUTH_CLIENT_ID=$CARFAX_OAUTH_CLIENT_ID \
            --dart-define=OKTA_ISSUER=$OKTA_ISSUER \
            --dart-define=OKTA_CLIENT_ID=$OKTA_CLIENT_ID \
            --dart-define=OKTA_REDIRECT_URI=$OKTA_REDIRECT_URI
      - name: Post results to Bitbucket
        script: |
          curl -X POST \
            -H "Authorization: Bearer $BITBUCKET_TOKEN" \
            -H "Content-Type: application/json" \
            -d '{
                  "content": {
                    "raw": "Integration tests completed successfully."
                  }
                }' \
            https://api.bitbucket.org/2.0/repositories/$WORKSPACE/$REPO/pullrequests/$PULL_REQUEST_ID/comments

Please remember to set the variables such as $BITBUCKET_TOKEN under Environment Variables

[danielccdk]

Tried echo'ing the global pull request id but it seems to be empty:

workflows:
  integration_test_workflow:
    environment:
      flutter: "3.24.3"
    scripts:
      - name: Integration Tests 
        script: |
          echo "Pull RequestId: $CM_PULL_REQUEST_NUMBER"
          flutter emulators --launch apple_ios_simulator
          flutter -d iPhone test integration_test/test_master.dart \
            --dart-define=isTesting=true \
            --dart-define=BUGSNAG_API_KEY=$BUGSNAG_API_KEY \
            --dart-define=CARFAX_OAUTH_CLIENT_ID=$CARFAX_OAUTH_CLIENT_ID \
            --dart-define=OKTA_ISSUER=$OKTA_ISSUER \
            --dart-define=OKTA_CLIENT_ID=$OKTA_CLIENT_ID \
            --dart-define=OKTA_REDIRECT_URI=$OKTA_REDIRECT_URI

[dam-ease]

Hello @danielccdk,
Please can you confirm that your workflow is set up to trigger only for pull requests

triggering:
  events:
    - pull_request

Without this, the $CM_PULL_REQUEST_NUMBER will not be populated.

[danielccdk]

I can yes... here is the workflow of this particular task:

workflows:
  integration_test_workflow:
    environment:
      flutter: "3.24.3"
    scripts:
      - name: Integration Tests 
        script: |
          set -ex
          printenv
          echo "Pull RequestId: $CM_PULL_REQUEST_NUMBER"
          flutter emulators --launch apple_ios_simulator
          flutter -d iPhone test integration_test/test_master.dart \
            --dart-define=isTesting=true \
            --dart-define=BUGSNAG_API_KEY=$BUGSNAG_API_KEY \
            --dart-define=CARFAX_OAUTH_CLIENT_ID=$CARFAX_OAUTH_CLIENT_ID \
            --dart-define=OKTA_ISSUER=$OKTA_ISSUER \
            --dart-define=OKTA_CLIENT_ID=$OKTA_CLIENT_ID \
            --dart-define=OKTA_REDIRECT_URI=$OKTA_REDIRECT_URI
    triggering:
      events:
        - pull_request
      branch_patterns:
      - pattern: 'dev'
        include: true
        source: false 

We use bitbucket for PR's. The build was not triggered in codemagic. Not sure why?

Additionally we tried to setup a webhook from codemagic, IN bitbucket, as per the documentation, I will try a couple things and see if I can get that work, however.

[dam-ease]

Thanks for your response @danielccdk.
Not certain this could be a reason but I think you should add the triggering section before the scripts section.
Also, kindly confirm that you have set up webhooks succesfully, you can go to the webhooks section in the Codemagic UI, click on the Update Webhook button, you should get a success message

Webhook is present in repository
Set build triggers in your workflow configuration

if this has been set up.
Also, kindly confirm what is being done in your repo is as what is described in the triggering script. With what you have above, for it to be triggered, it means that you are creating pull requests on the dev branch from a feature branch to propose and review changes before they’re integrated into dev.
Please confirm these and let me know how it goes.

[danielccdk]

Hey there friend... you have been IMMENSELY helpful through all of this. Thank you so so much!!!
So since adding the webhook (which showed up after creating a new PR) the builds portion of the bitbucket pull request is now updating so there is now no longer a need for the comments api to be hit.

Thank you so much! You're awesome.