App Framework: Azure MananedIdentity (IAM) authentication fails when more than one user-managed identity exists (AKS).

[djquarantine]

Please select the type of request

Bug

Tell us more

Describe the request
The AppFramework allow us to use two type of authentication: ManagedIdentity and SAS Token.

However, for the ManagedIdentity, it only works if there is only one user-managed identity assigned to the AKS nodepool vmss instance (ie. myakscluster_agentpool).

If there is more than one, the token request to microsoft will fail and return:
"error_description":"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request"

For more info regarding why it causes this error: https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/managed-identities-faq#what-identity-will-imds-default-to-if-i-dont-specify-the-identity-in-the-request

Splunk-Operator error:

2024-07-26T19:59:53.072809016Z  ERROR   AzureBlob:GetAppsList   Failed to get http request authenticated        {"controller": "standalone", "controllerGroup": "enterprise.splunk.com", "controllerKind": "Standalone", "Standalone": {"name":"dev-standalone","namespace":"splunk-system"}, "namespace": "splunk-system", "name": "dev-standalone", "reconcileID": "", "Endpoint": "https://REDACTED.blob.core.windows.net", "Bucket": "apps", "Prefix": "developper-tools/", "error": "please validate that your cluster is configured to use managed identity"}
github.com/splunk/splunk-operator/pkg/splunk/client.(*AzureBlobClient).GetAppsList
        /workspace/pkg/splunk/client/azureblobclient.go:369
github.com/splunk/splunk-operator/pkg/splunk/enterprise.(*RemoteDataClientManager).GetAppsList
        /workspace/pkg/splunk/enterprise/util.go:816
github.com/splunk/splunk-operator/pkg/splunk/enterprise.glob..func19
        /workspace/pkg/splunk/enterprise/util.go:846
github.com/splunk/splunk-operator/pkg/splunk/enterprise.GetAppListFromRemoteBucket
        /workspace/pkg/splunk/enterprise/util.go:885
github.com/splunk/splunk-operator/pkg/splunk/enterprise.initAndCheckAppInfoStatus
        /workspace/pkg/splunk/enterprise/util.go:1522
github.com/splunk/splunk-operator/pkg/splunk/enterprise.ApplyStandalone
        /workspace/pkg/splunk/enterprise/standalone.go:92
github.com/splunk/splunk-operator/controllers.glob..func8
        /workspace/controllers/standalone_controller.go:119
github.com/splunk/splunk-operator/controllers.(*StandaloneReconciler).Reconcile
        /workspace/controllers/standalone_controller.go:109
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235

Expected behavior
Successful authentication with IAM,

Splunk setup on K8S
Standard Splunk deployment

installCRDs: false
splunk-operator:
  enabled: false
standalone:
  enabled: true
  name: standalone
  resources:
    requests:
      memory: "16Gi"
      cpu: "4"
    limits:
      memory: "16Gi"
      cpu: "8"
  appRepo:
    appsRepoPollIntervalSeconds: 60
    defaults:
      volumeName: volume_app_repository
      scope: local
    appSources:
      - name: Test
        location: test/
      - name: Custom-ta
        location: custom-ta/
      - name: DevTools
        location: developper-tools/
    volumes:
      - name: volume_app_repository
        storageType: blob
        provider: azure
        path: apps/
        endpoint: https://REDACTED.blob.core.windows.net
        #secretRef: azureblob-splunkdev

Reproduction/Testing steps

• Create an AKS cluster with multiple user-identity (create aks with this additional parameter --enable-addons azure-keyvault-secrets-provider). At this point you will have more than 1 user-managed identity and trigger the failure.
• Deploy Splunk on the cluster.

K8s environment
AKS Cluster using managed-identity and azure key-vault

Proposed changes(optional)

• Update azureblobclient.go to accept optional "client_id" value when authenticating with IAM

Currently the oauth query look something like this:

curl 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fstorage.azure.com%2F' -H Metadata:true

To handle the case where there is more than one client-assigned managed identity, the query look like this:

curl 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fstorage.azure.com%2F&client_id=00001111-aaaa-2222-bbbb-3333cccc4444' -H Metadata:true

See: https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-managed-identities-work-vm

Thanks,
Dan

[michal-tatusko-splunk]

Thank you for raising this bug. We will look into that shortly.