Allow Machine ID non-VM resources to utilize Azure Join method

[tbuchanan-csi]

What would you like Teleport to do?
Allow Machine ID non-VM resources to utilize Azure Join method

What problem does this solve?
We currently have a use case for Teleport processes running on container solutions in Azure that need to be able to automatically authenticate themselves when they are launched. My understanding is that the currently Machine ID Azure Join implementation only works for Virtual Machine resources in Azure

https://goteleport.com/docs/agents/join-services-to-your-cluster/azure/

Prerequisites
An Azure Virtual Machine running Linux with the Teleport binary installed. The Virtual Machine must have a Managed Identity assigned to it with permission to read virtual machine info

If a workaround exists, please include it.

[strideynet]

Is there a specific Azure container solution you have in mind that I'll be able to test this with ? I'm not 100% sure that this won't work and it may be a case of us being too prohibitive in the docs - so if you give me some more information about which Azure container product you're using, I can check this, and then I can have the docs changed.

[tbuchanan-csi]

Container Instances and/or Container Apps

[strideynet]

Thanks - have you attempted to configure this yet ? Or is this an assumption purely from the statement in the docs ? If you have attempted it, at what stage did you receive an error or were blocked by the inability to configure something ?

[tbuchanan-csi]

I tried this in a Container Instance sometime last week and it was failing on the tbot start. Unfortunately I do not have the error still but I believe it was related to retrieving the instance metadata

[strideynet]

I'll try and fit in some time to repro next week and we can establish a priority on this getting fixed from there :)

[tbuchanan-csi]

Thanks @strideynet

I just re-tested on an Azure Container Instance so I could provide the error that is seen:

tbot start \
--destination-dir=./tbot-user \
--token=azure-token \
--auth-server=*******.teleport.sh:443 \
--join-method=azure
WARN [TBOT] CLI parameters are overriding onboarding config from config/config.go:460
INFO [TBOT] Created directory "/var/lib/teleport/bot" config/destination_directory.go:132
INFO [TBOT] Created directory "./tbot-user" config/destination_directory.go:132
INFO [TBOT] Anonymous telemetry is not enabled. Find out more about Machine ID's anonymous telemetry at https://goteleport.com/docs/machine-id/reference/telemetry/ tbot/anonymous_telemetry.go:83
INFO [TBOT] Attempting to generate new identity from token tbot/renew.go:479
INFO [AUTH] Attempting registration via proxy server. auth/register.go:277
INFO [AUTH] Attempting registration with auth server. auth/register.go:277
WARN [AUTH] Joining cluster without validating the identity of the Auth Server. This may open you up to a Man-In-The-Middle (MITM) attack if an attacker can gain privileged network access. To remedy this, use the CA pin value provided when join token was generated to validate the identity of the Auth Server. auth/register.go:496
ERROR: could not reach instance metadata. Is Teleport running on an Azure VM?, rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"

Teleport Cloud version 13.1.5
Teleport Binary version 13.1.5

[strideynet]

That's great - thank you !

[strideynet]

Testing

Container instance

https://learn.microsoft.com/en-gb/azure/container-instances/container-instances-managed-identity

With no system identity or user identity assigned, the IMDS hangs/is not present. I am unable to get a response

With system identity assigned, I am able to curl the IMDS and fetch an access token:

root@SandboxHost-638242462423070814:/# curl "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fvault.azure.net" -H Metadata:true -s
{"access_token":"REDACTED","refresh_token":"","expires_in":"86184","expires_on":"1688736102","not_before":"1688649857","resource":"https://vault.azure.net","token_type":"Bearer"}
root@SandboxHost-638242462423070814:/# 

This contains the following claims:

{
  "aud": "https://vault.azure.net",
  "iss": "https://sts.windows.net/ff882432-09b0-437b-bd22-ca13c0037ded/",
  "iat": 1688649403,
  "nbf": 1688649403,
  "exp": 1688736103,
  "aio": "E2ZgYAjSbzkkHzS5/rN19AVDRmZ5AA==",
  "appid": "c140da96-0ff2-4143-b6bd-fe67252efe7f",
  "appidacr": "2",
  "idp": "https://sts.windows.net/ff882432-09b0-437b-bd22-ca13c0037ded/",
  "oid": "e2a400c4-b855-4152-934d-dd315d00e267",
  "rh": "redacted.",
  "sub": "e2a400c4-b855-4152-934d-dd315d00e267",
  "tid": "ff882432-09b0-437b-bd22-ca13c0037ded",
  "uti": "iE7BIqNon0u9Dzb3zIyRAA",
  "ver": "1.0",
  "xms_mirid": "/subscriptions/060a97ea-3a57-4218-9be5-dba3f19ff2b5/resourcegroups/noah-test-joining/providers/Microsoft.ContainerInstance/containerGroups/test"
}

Attempting to fetch the Attested Data Document does not work:

root@SandboxHost-638242462423070814:/# curl --silent -H Metadata:True --noproxy "*" "http://169.254.169.254/metadata/attested/document?api-version=2020-09-01"
404 page not found
root@SandboxHost-638242462423070814:/# 

This means our existing Azure joining method will not be compatible, since that relies on the Attested Data document. We may wish to consider modifying the existing join method or introducing a new join method which can work on access tokens alone (albeit, identity tokens would be preferred).

[tbuchanan-csi]

Any ETA on when either of those solutions could be implemented or added to the roadmap?

[programmerq]

Use case:

Provision an AKS cluster via terraform, and get a teleport-kube-agent to join via an Azure federated identity. The Azure teleport token would be set up ahead of time.

[strideynet]

Related to #38852