[BUG] SAML auth is broken after upgrade from 2.11.0 to 2.18.0

[timur-ND]

Describe the bug

Hello. I faced an issue with SAML authorization after upgrade Opensearch and Opensearch Dashboards from 2.11.0 to 2.18.0. I have 3 OpenSearch nodes, I'm using Azure AD and everything was working fine until I upgraded to the new version. Nothing has changed on the configuration side (opensearch-security/config.yml). After trying to log it, I get {"statusCode":500,"error":"Internal Server Error","message":"Internal Error"} (uri - /auth/saml/login?redirectHash=false&nextUrl=%2F)
On the server side, in Opensearch Dashboards logs, I see this:

Nov 15 14:01:48 opensearch-dashboards[66974]: {"type":"response","@timestamp":"2024-11-15T14:01:48Z","tags":[],"pid":66974,"method":"get","statusCode":200,"req":{"url":"/auth/saml/captureUrlFragment.js","method":"get","headers":{"x-forwarded-for":"*****","x-forwarded-proto":"https","host":"*****","connection":"close","sec-ch-ua-platform":"\"macOS\"","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36","sec-ch-ua":"\"Chromium\";v=\"130\", \"Google Chrome\";v=\"130\", \"Not?A_Brand\";v=\"99\"","sec-ch-ua-mobile":"?0","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"https://*****/auth/saml/captureUrlFragment?nextUrl=%2F","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-US,en;q=0.9,ru-RU;q=0.8,ru;q=0.7,ka;q=0.6"},"remoteAddress":"10.96.3.254","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36","referer":"https://****/auth/saml/captureUrlFragment?nextUrl=%2F"},"res":{"statusCode":200,"responseTime":4,"contentLength":9},"message":"GET /auth/saml/captureUrlFragment.js 200 4ms - 9.0B"}
Nov 15 14:01:48 opensearch-dashboards[66974]: {"type":"log","@timestamp":"2024-11-15T14:01:48Z","tags":["error","plugins","securityDashboards"],"pid":66974,"message":"Failed to get saml header: Authentication Exception :: {\"path\":\"/_plugins/_security/authinfo\",\"query\":{\"auth_type\":\"saml\"},\"statusCode\":401,\"response\":\"Authentication finally failed\"}"}
Nov 15 14:01:48 opensearch-dashboards[66974]: {"type":"error","@timestamp":"2024-11-15T14:01:48Z","tags":[],"pid":66974,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:127:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:83:19)\n    at HapiResponseAdapter.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:79:17)\n    at Router.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:175:34)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at handler (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:140:50)\n    at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)\n    at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)"},"url":"http://*****/auth/saml/login?redirectHash=false&nextUrl=%2F","message":"Internal Server Error"}

On the Opensearch side I see this in the logs:
[2024-11-15T14:01:48,617][WARN ][o.o.s.a.BackendRegistry ] [*****] Authentication finally failed for null from 10.97.3.81:50628
I can login using Basic Auth, since it's enabled. I can't rollback because index version seems incompatible for old Opensearch versions.
My SAML config:

      saml_auth_domain:
        http_enabled: true
        transport_enabled: false
        order: 2
        http_authenticator:
          type: saml
          challenge: true
          config:
            idp:
              metadata_url: https://login.microsoftonline.com/fc*****
              entity_id: https://sts.windows.net/fc****
            sp:
              entity_id: ****
            kibana_url: *****
            roles_key: Role
            exchange_key: ******
            jwt:
              expiry: AUTO+1440
        authentication_backend:
          type: noop

My Opensearch-dashboards configuration:

---
server.port: 5601
server.host: "0.0.0.0"
server.maxPayloadBytes: 10485760
server.name: "stg"
server.xsrf.allowlist: ["/_opendistro/_security/saml/acs/idpinitiated", "/_opendistro/_security/saml/acs", "/_opendistro/_security/saml/logout"]
opensearch.username: "*****"
opensearch.password: "*****"
opensearch.ssl.verificationMode: none
opensearch.requestHeadersWhitelist: [authorization, securitytenant]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
opensearch_security.cookie.secure: false
opensearch.hosts: [https://*****:9200]
opensearch_security.auth.type: ["basicauth","saml"]
opensearch_security.auth.multiple_auth_enabled: true
opensearch_security.saml.extra_storage.cookie_prefix: security_authentication_saml
opensearch_security.saml.extra_storage.additional_cookies: 3
opensearch_security.cookie.ttl: 86400000
opensearch_security.session.ttl: 86400000
opensearch_security.session.keepalive: false

I hope you can help me. Thank you in advance!

Related component

Other

To Reproduce

1. yum install opensearch-2.18.0-1 opensearch-dashboards-2.18.0-1 -y
2. systemctl restart opensearch opensearch-dashboards
3. Repeat for all nodes
4. Login to my.domain.com
5. Click "Log in with single sign-on"
6. Got error

Expected behavior

SAML authentication should work the same as before the upgrade.

Additional Details

Plugins
Please list all plugins currently enabled.

./bin/opensearch-plugin list
opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-custom-codecs
opensearch-flow-framework
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-security-analytics
opensearch-skills
opensearch-sql
opensearch-system-templates
query-insights

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: CentOS Stream 9
• Kernel 5.14.0-390.el9.x86_64

Additional context
Add any other context about the problem here.

[cwperks]

@timur-ND Are there any other authenticators configured in the authc section of opensearch-security/config.yml. First thing that comes to mind is make sure that only the SAML authenticator sets challenge to true. Is there any other relevant logs in the opensearch logs?

[timur-ND]

Also enabled Basic Auth, but challenge is set to False:

      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern

All other methods (ldap, clientcert_auth_domain, jwt_auth_domain, proxy_auth_domain,kerberos_auth_domain) is disabled (http_enabled: false).
But I see that for kerberos_auth_domain challenge is set to true, but it's disabled. Can it be the cause?

Actually, after restart, I found this in the logs. Seems like something is wrong:

[2024-11-18T11:25:51,387][ERROR][c.a.d.a.h.s.HTTPSamlAuthenticator] Error creating HTTPSamlAuthenticator. SAML authentication will not work
java.lang.IllegalArgumentException: Illegal base64 character 2b
	at java.base/java.util.Base64$Decoder.decode0(Base64.java:852) ~[?:?]
	at java.base/java.util.Base64$Decoder.decode(Base64.java:570) ~[?:?]
	at java.base/java.util.Base64$Decoder.decode(Base64.java:593) ~[?:?]
	at com.amazon.dlic.auth.http.saml.AuthTokenProcessorHandler.createJwkFromSettings(AuthTokenProcessorHandler.java:250) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
	at com.amazon.dlic.auth.http.saml.AuthTokenProcessorHandler.<init>(AuthTokenProcessorHandler.java:113) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
	at com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator.<init>(HTTPSamlAuthenticator.java:149) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.lambda$newInstance$1(DynamicConfigModelV7.java:432) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) [?:?]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.newInstance(DynamicConfigModelV7.java:430) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.buildAAA(DynamicConfigModelV7.java:329) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.<init>(DynamicConfigModelV7.java:102) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:284) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:571) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.notifyConfigurationListeners(ConfigurationRepository.java:560) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:555) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.loadConfigurationWithLock(ConfigurationRepository.java:539) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:532) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.initalizeClusterConfiguration(ConfigurationRepository.java:283) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.lambda$initOnNodeStart$10(ConfigurationRepository.java:438) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
[2024-11-18T11:25:51,388][WARN ][o.o.s.s.ReflectionHelper ] Unable to enable 'com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator' due to java.lang.reflect.InvocationTargetException
[2024-11-18T11:25:51,391][ERROR][o.o.s.s.DynamicConfigModelV7] Unable to initialize auth domain saml_auth_domain=AuthcDomain [http_enabled=true, order=2, http_authenticator=HttpAuthenticator [challenge=true, type=saml, config={idp={metadata_url=https://login.microsoftonline.com/fc ... 

<secret data> 

...] due to OpenSearchException[java.lang.reflect.InvocationTargetException]; nested: InvocationTargetException; nested: RuntimeException[java.lang.IllegalArgumentException: Illegal base64 character 2b]; nested: IllegalArgumentException[Illegal base64 character 2b];
org.opensearch.OpenSearchException: java.lang.reflect.InvocationTargetException
	at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:73) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.lambda$newInstance$1(DynamicConfigModelV7.java:432) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.newInstance(DynamicConfigModelV7.java:430) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.buildAAA(DynamicConfigModelV7.java:329) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.securityconf.DynamicConfigModelV7.<init>(DynamicConfigModelV7.java:102) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:284) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:571) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.notifyConfigurationListeners(ConfigurationRepository.java:560) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:555) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.loadConfigurationWithLock(ConfigurationRepository.java:539) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:532) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.initalizeClusterConfiguration(ConfigurationRepository.java:283) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.lambda$initOnNodeStart$10(ConfigurationRepository.java:438) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
	... 14 more
Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Illegal base64 character 2b
	at com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator.<init>(HTTPSamlAuthenticator.java:155) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
	... 14 more
Caused by: java.lang.IllegalArgumentException: Illegal base64 character 2b
	at java.base/java.util.Base64$Decoder.decode0(Base64.java:852) ~[?:?]
	at java.base/java.util.Base64$Decoder.decode(Base64.java:570) ~[?:?]
	at java.base/java.util.Base64$Decoder.decode(Base64.java:593) ~[?:?]
	at com.amazon.dlic.auth.http.saml.AuthTokenProcessorHandler.createJwkFromSettings(AuthTokenProcessorHandler.java:250) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
	at com.amazon.dlic.auth.http.saml.AuthTokenProcessorHandler.<init>(AuthTokenProcessorHandler.java:113) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
	at com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator.<init>(HTTPSamlAuthenticator.java:149) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
	... 14 more

[timur-ND]

Seems my exchange_key was incorrect or idk.
I recreated it, using command:
head /dev/urandom | tr -dc A-Za-z0-9 | head -c 64
Replaced it across all nodes, restarted OpenSearch and run securityadmin.sh

OPENSEARCH_JAVA_HOME=/usr/share/opensearch/jdk /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cd /etc/opensearch/opensearch-security -icl -nhnv -cacert /etc/opensearch/ca.crt -cert /etc/opensearch/mycert.crt -key /etc/opensearch/mycert.key

Now seems ok, but strange, this key was the same all the time. Looks like before the upgrade OpenSearch didn't recognize key and somehow it worked well. After the upgrade, the key was no longer valid.