Smartstore: Failed to execute runSync()

[nfilip21]

Please select the type of request

Bug

Tell us more

Describe the request

• Smartstore quits working with below message.
• ERROR AwsCredentials [cachemanagerUploadExecutorWorker-95] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com/ http_code=400
  ERROR S3Client [cachemanagerUploadExecutorWorker-54] - command=put transactionId=xxxxxxxxxxrTxnId=xxxxxxxxxstatus=completed success=N uri=xxxxxxxxxxxxxxxxxxxx statusCode=400 statusDescription="Bad Request" payload="\nExpiredTokenThe provided token has expired.....xxxxxxxxxxxxxxxxxxx<Host"

CloudTrail can be used to confirm that the token has expired.

Expected behavior

• Splunk should be able to see an expired AWS token for a role and update the token.

Splunk setup on K8S

• EKS 1.29

Splunk Setup

• SOK 2.5.2
• Splunk 9.2.0

K8s environment

• Production

Proposed changes(optional)

• Proposed change, if any.

K8s collector data(optional)

• Please provide data collected from the K8s collectors, if any.

Additional context(optional)

• Add any other context about the problem here.

[raghukumarc]

We have been seeing similar errors as reported. Appreciate help here:
07-17-2024 06:03:09.545 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:03:10.832 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:03:13.120 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:03:17.398 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:03:25.683 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:03:41.973 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:04:14.255 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:05:14.537 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:06:14.820 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:07:15.099 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:08:06.263 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:08:07.551 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:08:09.830 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:08:14.112 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:08:22.396 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:08:38.702 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:09:10.986 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:10:11.274 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:11:11.554 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:12:11.834 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400
07-17-2024 06:12:11.848 +0000 ERROR S3Client [568873 FilesystemOpExecutorWorker-0] - command=head transactionId=0x7f0a3a71e000 rTxnId=0x7f0a2c7f5f10 status=completed success=N uri=https://-splunk-smartstore.s3-eu-west-1.amazonaws.com/single-site/index_stg/db/8b/35/1854BB2B8B-8FD4-45FE-92FD-61161215F597/receipt.json statusCode=400 statusDescription="Bad Request" payload=""
07-17-2024 06:12:11.848 +0000 ERROR RetryableClientTransaction [568873 FilesystemOpExecutorWorker-0] - transactionDone() exiting: groupId=(nil) rTxnId=0x7f0a2c7f5f10 transactionId=0x7f0a3a71e000 status=fatal success=N HTTP-statusCode=400 HTTP-statusDescription="Bad Request" retries=0 retry=N no_retry_reason="transaction had fatal error" remainingTxns=0
07-17-2024 06:12:11.848 +0000 ERROR DatabaseDirectoryManager [568873 FilesystemOpExecutorWorker-0] - failed to check if cache_id="bid|index_stg1854BB2B8B-8FD4-45FE-92FD-61161215F597|" is stable with CacheManager due to exception=Unable to check If cache_id="bid|index_stg18~54BB2B8B-8FD4-45FE-92FD-61161215F597|" exists on remote storage or not as the check could not complete due to an error="Bad Request"

[vivekr-splunk]

upcoming splunk enterprise release should fix this issue.

[akondur]

Hi @nfilip21 , the fix for AWS IRSA JWT token expiring has been fixed in splunk 9.3.2. Please try it and let us know if you are running into further issues.