Memory leak through jersey NonInjectionManager thread local

[newwdles]

We have a rest service which use docusign esign java client. The application is based on spring boot 3 and webflux. It use this code to refresh the token (boiler plate removed for clarity) :

private final Cache<String, ApiClient> tokenCache = Caffeine.newBuilder()
            .expireAfterWrite(Duration.ofSeconds(TOKEN_EXPIRES_IN)) // reloads the token 1 minute before it's expiration date, but still returns the old value until the refresh
            .build();


    private static final int TOKEN_EXPIRES_IN = 120; // lowered for test
    private static final int TOKEN_REFRESH_RATE = TOKEN_EXPIRES_IN - 60;

@Scheduled(fixedRate = TOKEN_REFRESH_RATE * 1000)
    public void refreshTokenCache() {
        Flux.fromStream(signatureConfiguration.getProperties().entrySet().stream()) // for each configuration (bankReference key)
                .filter(kv -> SignatureMode.DOCU_SIGN.equals(kv.getValue().getMode()))
                .map(Map.Entry::getKey)
                .doOnNext(bankReference -> log.info("About to update token cache for bankReference {}", bankReference))
                .flatMap(bankReference -> loadApiClient(bankReference)
                        .doOnNext(apiClient -> {
                            tokenCache.put(bankReference, apiClient);
                            log.info("Scheduling - Successfully updated token cache for bank {}", bankReference);
                        })
                        .subscribeOn(Schedulers.parallel())
                        .publishOn(Schedulers.parallel())
                )
                .doOnError(th -> log.error("Could not update token cache", th))
                .doOnSubscribe(sbs -> log.info("Launching scheduled authentication refresh"))
                .subscribeOn(Schedulers.parallel())
                .publishOn(Schedulers.parallel())
                .subscribe();
    }

This trigger a refresh of the token 1 minutes before the token expire, the loadApiCLient is as follow :

private Mono<ApiClient> loadApiClient(String bankReference) {
        return mandateSignatureAttributesRepository.findDocusignConfigurationByBankReference(bankReference)
                .<SignatureProperties.DocusignConfiguration>handle(... // get the key here
                })
                .handle((docusignConfig, sink) -> {
                    try {
                        ApiClient apiClient = new ApiClient(signatureConfiguration.getDocusignBaseUrl());
                        apiClient.setReadTimeout(10000);
                        apiClient.setConnectTimeout(10000);
                        log.info("Docusign : Read and connect timeout respectively set to {}ms and {}ms", apiClient.getReadTimeout(), apiClient.getConnectTimeout());
                        String IntegratorKey = docusignConfig.getDocusignIntegrationKey();
                        String UserId = signatureConfiguration.getDocusignUserId();
                        log.info("Docusign : Initializing Docusign authentication for bank {}", bankReference);
                        OAuth.OAuthToken oAuthToken = apiClient.requestJWTUserToken(IntegratorKey, UserId, scopes, privateKeyBytes, TOKEN_EXPIRES_IN);
                        log.info("Docusign : Authentication successful for bank {}", bankReference);
                        apiClient.setAccessToken(oAuthToken.getAccessToken(), oAuthToken.getExpiresIn());
                        log.info("Docusign : Getting the user information for bank {}", bankReference);
                        OAuth.UserInfo userInfo = apiClient.getUserInfo(oAuthToken.getAccessToken());
                        log.info("Docusign : Successfully found the user information for bank {}", bankReference);
                        apiClient.setBasePath(userInfo.getAccounts().get(0).getBaseUri() + "/restapi");
                        sink.next(apiClient);
                    } catch (ProcessingException e) {
                      // Exception handling
                });
    }

We are seeing memory leaks and OOM in our stage environnement :

Here we can see a lot of com.fasterxml.jackson.databind.deser.BeanDeserializer instances are taking 166mo, the GC just passed before the heap dump and thoses instances are still referenced.

Looking at the root path to GC we can see that some internal objects of jersey are kept in a thread local, they in turn reference the docusign esign ApiCLient which reference some jackson classes, which internally reference the BeanDeserializer classes.

Disabling the multithreading does not solve the issue, nor does reusing the ApiClient or using apiClient.getHttpClient().close();
This make sense as it seems that a new instance of org.glassfish.jersey.client.innate.inject.NonInjectionManager.TypedInstances is created with each call to com.docusign.esign.client.ApiClient#requestJWTUserToken, this is turn create a new ThreadLocal, resulting in a leak.

The jersey code is quite complicated so i'm having trouble pinpointing the exact cause, i will post more details as i progress in my research.

[newwdles]

This issue in jersey client eclipse-ee4j/jersey#5710 mention that a leak can occur, it is fixed in 3.1.8 (the version we are using), however the NonInjectionManager#dispose() is not called when using ApiClient#requestJWTUserToken (which build a temporary Client).