Kubeflow dex authentication not working: ERR_TOO_MANY_REDIRECTS

[miragir]

Validation Checklist

• Is this a Kubeflow issue?
• Are you posting in the right repository ?
• Did you follow the Kubeflow installation guideline ?
• Is the issue report properly structured and detailed with version numbers?
• Is this for Kubeflow development ?
• Would you like to work on this issue?
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

master

Describe your issue

When logging into kubeflow UiI via dex - OIDC auth, we are getting ERR_TOO_MANY_REDIRECTS error for past 1 week.
We have installed kubeflow from https://github.com/awslabs/kubeflow-manifests/releases a month ago.
Everything was working perfectly until past 1 week when it started thrwoing forllowing error :
This page isn’t workinglocalhost redirected you too many times.
Try deleting your cookies.
ERR_TOO_MANY_REDIRECTS

Nothing on logs
We updated auth service-0 to docker.io/kubeflowmanifestswg/oidc-authservice:28c59ef.
auth service-0 logs:
kubectl logs -n istio-system authservice-0
time="2024-09-03T11:41:21Z" level=info msg="Starting readiness probe at 8081"
time="2024-09-03T11:41:21Z" level=info msg="No USERID_TOKEN_HEADER specified, using 'kubeflow-userid-token' as default."
time="2024-09-03T11:41:21Z" level=info msg="No SERVER_HOSTNAME specified, using '' as default."
time="2024-09-03T11:41:21Z" level=info msg="No SERVER_PORT specified, using '8080' as default."
time="2024-09-03T11:41:21Z" level=info msg="No SESSION_MAX_AGE specified, using '86400' as default."
time="2024-09-03T11:41:21Z" level=info msg="Starting web server at :8080"

logs for auth namespace dex pods:
kubectl logs -n auth dex-69b8795859-79tf2
time="2024-09-03T08:54:07Z" level=info msg="Dex Version: v2.31.2-dirty, Go Version: go1.17.10, Go OS/ARCH: linux amd64"
time="2024-09-03T08:54:07Z" level=info msg="config using log level: debug"
time="2024-09-03T08:54:07Z" level=info msg="config issuer: http://dex.auth.svc.cluster.local:5556/dex"
time="2024-09-03T08:54:07Z" level=info msg="kubernetes client apiVersion = dex.coreos.com/v1"
time="2024-09-03T08:54:07Z" level=info msg="creating custom Kubernetes resources"
time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource authcodes.dex.coreos.com has already been created..."
time="2024-09-03T08:54:07Z" level=info msg="The custom resource authcodes.dex.coreos.com already available, skipping create"
time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource authrequests.dex.coreos.com has already been created..."
time="2024-09-03T08:54:07Z" level=info msg="The custom resource authrequests.dex.coreos.com already available, skipping create"
time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource oauth2clients.dex.coreos.com has already been created..."
time="2024-09-03T08:54:07Z" level=info msg="The custom resource oauth2clients.dex.coreos.com already available, skipping create"
time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource signingkeies.dex.coreos.com has already been created..."
time="2024-09-03T08:54:07Z" level=info msg="The custom resource signingkeies.dex.coreos.com already available, skipping create"
time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource refreshtokens.dex.coreos.com has already been created..."
time="2024-09-03T08:54:07Z" level=info msg="The custom resource refreshtokens.dex.coreos.com already available, skipping create"
time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource passwords.dex.coreos.com has already been created..."
time="2024-09-03T08:54:07Z" level=info msg="The custom resource passwords.dex.coreos.com already available, skipping create"
time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource offlinesessionses.dex.coreos.com has already been created..."
time="2024-09-03T08:54:07Z" level=info msg="The custom resource offlinesessionses.dex.coreos.com already available, skipping create"
time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource connectors.dex.coreos.com has already been created..."
time="2024-09-03T08:54:07Z" level=info msg="The custom resource connectors.dex.coreos.com already available, skipping create"
time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource devicerequests.dex.coreos.com has already been created..."
time="2024-09-03T08:54:07Z" level=info msg="The custom resource devicerequests.dex.coreos.com already available, skipping create"
time="2024-09-03T08:54:07Z" level=info msg="checking if custom resource devicetokens.dex.coreos.com has already been created..."
time="2024-09-03T08:54:07Z" level=info msg="The custom resource devicetokens.dex.coreos.com already available, skipping create"
time="2024-09-03T08:54:07Z" level=info msg="config storage: kubernetes"
time="2024-09-03T08:54:07Z" level=info msg="config static client: Dex Login Application"
time="2024-09-03T08:54:07Z" level=info msg="config connector: local passwords enabled"
time="2024-09-03T08:54:07Z" level=info msg="config skipping approval screen"
time="2024-09-03T08:54:07Z" level=info msg="config refresh tokens rotation enabled: true"
time="2024-09-03T08:54:07Z" level=info msg="listening (http) on 0.0.0.0:5556"

authservice parameters:
kubectl get cm -n istio-system oidc-authservice-parameters -oyaml apiVersion: v1 data: AUTHSERVICE_URL_PREFIX: /authservice/ OIDC_AUTH_URL: /dex/auth OIDC_PROVIDER: http://dex.auth.svc.cluster.local:5556/dex OIDC_SCOPES: profile email groups PORT: '"8080"' SKIP_AUTH_URLS: /dex STORE_PATH: /var/lib/authservice/data.db USERID_CLAIM: email USERID_HEADER: kubeflow-userid USERID_PREFIX: "" kind: ConfigMap metadata: annotations: meta.helm.sh/release-name: oidc-authservice meta.helm.sh/release-namespace: default creationTimestamp: "2024-07-23T08:48:16Z" labels: app.kubernetes.io/managed-by: Helm name: oidc-authservice-parameters namespace: istio-system resourceVersion: "37389724" uid: 79ded254-7349-4cd5-9ccf-b1c8aa8c89d3

kubectl get cm -n auth dex -oyaml apiVersion: v1 data: config.yaml: | issuer: http://dex.auth.svc.cluster.local:5556/dex storage: type: kubernetes config: inCluster: true web: http: 0.0.0.0:5556 logger: level: "debug" format: text oauth2: skipApprovalScreen: true enablePasswordDB: true staticPasswords: - email: [email protected] hash: XXXXXX # https://github.com/dexidp/dex/pull/1601/commits # FIXME: Use hashFromEnv instead username: user userID: "1584118564XXX" staticClients: # https://github.com/dexidp/dex/pull/1664 - idEnv: OIDC_CLIENT_ID redirectURIs: ["/authservice/oidc/callback"] name: 'Dex Login Application' secretEnv: OIDC_CLIENT_SECRET kind: ConfigMap metadata: annotations: meta.helm.sh/release-name: dex meta.helm.sh/release-namespace: default creationTimestamp: "2024-07-23T08:48:02Z" labels: app.kubernetes.io/managed-by: Helm name: dex namespace: auth resourceVersion: "13514" uid: 59bca939-f033-44eb-91e5-8c51eb6e7698

Steps to reproduce the issue

NA

Put here any screenshots or videos (optional)

We use kubectl port-forward svc/istio-ingressgateway 8080:80 -n istio-system to access

[juliusvonkohout]

Are you not using oauth2-proxy? Oidc-authservice is deprecated.

[juliusvonkohout]

Maybe you should redeploy from master and check out #2864

[thesuperzapper]

@miragir it looks like you are using a distribution, please reach out to the maintainers of your distribution for support (https://github.com/awslabs/kubeflow-manifests).

However, I think you will find that AWS no longer maintains that distribution, so I recommend moving to one that is supported. Many orgs have migrated from "Kubeflow on AWS" to deployKF (which I maintain) because it's very easy to integrate with AWS services like S3/RDS and is much more user-friendly than dealing with the manifests directly.

[juliusvonkohout]

to cite from https://www.kubeflow.org/docs/started/installing-kubeflow/#kubeflow-platform
"The Kubeflow manifests provide a quick way to get a minimum viable Kubeflow Platform up and running. The Kubeflow community support for Kubeflow manifests is only best-effort, non-commercial and not guaranteed for environment-specific issues or custom configurations. Nevertheless, we welcome contributions and bug reports very much. For commercial production-level usage and support there are many options. You can use a third-party commercial distribution, hire consultants or build up the knowledge yourself to maintain and extend your Kubeflow installation."