Support for Azure DevOps Pipelines as a Joining Method

[FireDrunk]

Recently I've put in some effort to determine whether it would be possible to see if Teleport could support Azure DevOps Pipelines as a native joining method. This would be similar like the GitLab CI Joining method, where you can authorize a GitLab pipeline to perform specific tasks inside Teleport (For example: running CI jobs on a Teleport Node or deploy an artifact using a Teleport Kubernetes connection).

I've talked about this with @strideynet and understand there is a certain level of trust required before Teleport can accept joining from a CI pipeline.

After conducting some research (and talking with Azure DevOps Support), I want to share my findings, and start the discussion on whether the current possible solutions would be acceptable to start implementing this, as it would probably require a significant effort.

The way Gitlab Joining is supported, is by using a specific token that is passed along to a Job, send this to the Teleport server which in it's turn can validate the token against the publicly available GitLab API('s).

Unfortunately, as of today Azure DevOps does not contain a comparable solution.

Azure DevOps supports a fairly basic method of authentication / authorization in the form of a so called AccessToken. This token is signed internally by the Azure Devops Organization without public available API's or Keys to validate this ticket. This token is a standardized Json Web Token (JWT), but there are no public (official) endpoints that would allow Teleport to validate the token.

The token can be used on various API's which would implicitly validate it, but that's not enough to validate the specific pipeline for what it is.

I recently came across a few posts from Azure, regarding the new Azure DevOps OIDC Workload Identity Federation (yeah, it's a mouthful).

Links:

• https://techcommunity.microsoft.com/t5/azure-devops-blog/introduction-to-azure-devops-workload-identity-federation-oidc/ba-p/3908687
• https://devblogs.microsoft.com/devops/public-preview-of-workload-identity-federation-for-azure-pipelines/
• https://learn.microsoft.com/en-us/azure/devops/pipelines/release/configure-workload-identity?view=azure-devops

The posts above explain a new way of connecting Azure DevOps to Azure Resource Manager (ARM), the regular Azure provisioning API for cloud resources.

This Federation can be used to validate that a specific Pipeline has permissions to access the ARM API to create / update / delete resources in a specific Azure Environment by means of a Service Connection.

This Service Connection is functionality of the Azure DevOps Ecosystem, in which you grant a project or pipeline access to a specific resource without directly exposing it to the credentials used inside that Service Connection.

In the new Federation system, there is a token exchange happening between the Azure DevOps API's, and the ARM system to exchange an AccessToken for an ARM Token.

This token can be validated against public API's, and contains some information regarding the Azure DevOps Project it came from, and which Service Connection it was using.
Because ARM is aware of the Federation, it can validate that token, and will grant access to the underlying Azure API.

To sum things up:

We can (theoretically) use Azure DevOps' AccessToken to get an ARM Token which can be publicly validated by Teleport. But (there's always a but), there are some downsides:

• There is some steps involved in federating Azure DevOps to Azure, which can be somewhat complex if you don't have enough permissions (see the links above).
  This implies that the guide for 'linking' Azure DevOps to Teleport involves some complex documentation on how to setup Federation.

• The given Token only proves that the request is coming from an Azure DevOps Project and a specific Service Connection.
  This might not imply the perfect flow in which we know exactly which Pipeline we are dealing with in Teleport to grant specific roles.
  This can however be mitigated if we assume that the User is responsible for allowing Azure DevOps Pipelines access to that specific Service Connection.

The model we would use inside Teleport for this Joining method would be:

• We trust a specific Azure DevOps Organization, I think we can publicly validate this.
• We accept that the UUID / Name of a Service Connection is a valid target to grant permissions to inside this Joining Method.
• The AccessToken will be exchanged by the Teleport Client for a specific ARM Token using Azure DevOps API's.
• The successful exchange of this Token validates the Pipeline has access to the specific Service Connection.
• The Teleport Client attempts to join the server by provides the ARM token to the Teleport Server, which will validate it against the Public Azure ARM API's of the linked Organization (some configuration will be involved).
• The joining is allowed, and certificates are generated.

I've successfully tested all token exchanges using (fairly) regular REST API calls, and I'm willing to put in some work, since this is probably beneficial for us (my current company) as we are using Teleport inside Azure DevOps.

Example of the exchanged ARM Token:

{
  "typ": "JWT",
  "alg": "RS256",
  "x5t": "7RwayviXDqhfsz16R6lOmsWYlSA",
  "kid": "ED1C1ACAF8970EA85FB33D7A47A94E9AC5989520"
}.{
  "jti": "6e1a58eb-584a-49e1-8449-98579bd118b2",
  "sub": "sc://ADO_ORGANIZATION_NAME/ADO_PROJECT_NAME/ADO_SERVICE_CONNECTION_NAME",
  "aud": "api://AzureADTokenExchange",
  "iss": "https://vstoken.dev.azure.com/AZURE_DEVOPS_ORGANIZATION_UUID",
  "nbf": 1709309275,
  "exp": 1709310475,
  "iat": 1709309875
}

Any thoughts?

[strideynet]

Nice find - I think the service connections are definitely something we can leverage here. I was able to create a service connection, and use this within a pipeline to get a 10m JWT that can be validated using the JWKS at https://vstoken.dev.azure.com/.well-known/jwks .

I think we'd essentially just use the service connection name within the join token allow rules, and suggest that they grant access to Teleport to certain pipelines by allowing them access to the service connection configured within Teleport.

What was odd, was I did have to create an ARM service connection to my Azure account, even though I then don't want to use the Azure AD exchanged token, just the prerequisite token. ADO is very close to having a good solution here but seem to miss the mark.

Generally speaking, there's roughly three major things we need to think about when introducing a new join method:

• Reducing the risk of replay attacks. We usually mitigate this with a challenge string included in the audience that is specific to the join attempt - but that doesn't look possible here. We'd need to acknowledge that these tokens would be reusable within a ten minute window.
• Reducing the privilege of tokens sent to Teleport. We usually try and avoid tokens being sent to Teleport which grant access to resources elsewhere. This reduces the blast radius if Teleport is exploited. I think by asking people to create a SC specifically for Teleport joining, we can achieve this requirement.
• Securely validating the token. Here we can leverage the JWKS offered by ADO and do this the same way we handle JWTs from platforms like GHA.

The other option here is to go deeper into Azure land and build a join method based on Applications in Azure AD. I think we'd lose a lot of granularity with this and make it much harder to configure and implement.

[strideynet]

As for next steps, I think we'd want a formal RFD that refers to https://github.com/gravitational/teleport/blob/master/rfd/0079-oidc-joining.md (e.g we can avoid duplicating).

I imagine the actual implementation will be fairly simple once the RFD is complete as there's plenty of prior art we can refer to.

This should include:

• A sequence diagram showing tbot, the auth server and any parts of ADO involved in this process.
• The HTTP calls made by tbot and the auth server
• The structure of the Join Token that will be added to Teleport (e.g what fields will need to be configured)
• Any environment variables tbot will need to read.

[strideynet]

Related to #28689