logging via AWS.Logger.Log4net.AWSAppender hangs web app #1968

alverdal · 2022-01-05T19:52:07Z

Description

Web application refuse to start after updating nuget package AWSSDK.core from 3.7.0.45 -> 3.7.1 (and anything newer).
I'm using AWS.Logger.Log4net to send logs to cloud watch.

On my local dev machine I got valid credentials to access cloud watch and the sdk is complaining that I got invalid credentials.
On 3.7.0.45 this seemed to be ignored and the web application was able to start and run without issues. But after updating to 3.7.1 the application is complaining on credentials and the web application hangs.
I'm running the web application on IIS.

According to the logs it seem that a thread is aborted and that causes the whole web application to halt.
Perhaps you can see if anything related to invalid credentials was changed between these versions that might cause the issue and also instruct me on how I can fix this issue.

Reproduction Steps

Updated nuget package AWSSDK.Core.
Try to log using the appender

<appender name="AWS" type="AWS.Logger.Log4net.AWSAppender,AWS.Logger.Log4net">
		<LogStreamNamePrefix>xx</LogStreamNamePrefix>
		<LogGroup>xx</LogGroup>
		<Region>xx</Region>
		<layout type="log4net.Layout.PatternLayout">
			<conversionPattern value="%-4timestamp [%thread] %-5level %logger %ndc - %message%newline" />
		</layout>
		<LibraryLogFileName>c:\temp\logs\my_app_awslog.txt</LibraryLogFileName>
	</appender>

Logs


Log Entry : 
2022-01-05 20:05:03
  :
  :Amazon.Runtime.AmazonServiceException: Unable to get IAM security credentials from EC2 Instance Metadata Service.
   vid Amazon.Runtime.DefaultInstanceProfileAWSCredentials.FetchCredentials()
   vid Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentials()
   vid Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentialsAsync()
   vid Amazon.Runtime.Internal.CredentialsRetriever.<InvokeAsync>d__7`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__5`1.MoveNext()
-------------------------------
Log Entry : 
2022-01-05 20:05:03
  :
  :Amazon.Runtime.AmazonServiceException: Unable to get IAM security credentials from EC2 Instance Metadata Service.
   vid Amazon.Runtime.DefaultInstanceProfileAWSCredentials.FetchCredentials()
   vid Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentials()
   vid Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentialsAsync()
   vid Amazon.Runtime.Internal.CredentialsRetriever.<InvokeAsync>d__7`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__5`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid Amazon.Runtime.Internal.MetricsHandler.<InvokeAsync>d__1`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid AWS.Logger.Core.AWSLoggerCore.<LogEventTransmissionSetup>d__33.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid AWS.Logger.Core.AWSLoggerCore.<Monitor>d__31.MoveNext()
-------------------------------
Log Entry : 
2022-01-05 20:05:35
  :
  :Amazon.Runtime.AmazonServiceException: Unable to get IAM security credentials from EC2 Instance Metadata Service.
   vid Amazon.Runtime.DefaultInstanceProfileAWSCredentials.FetchCredentials()
   vid Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentials()
   vid Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentialsAsync()
   vid Amazon.Runtime.Internal.CredentialsRetriever.<InvokeAsync>d__7`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__5`1.MoveNext()
-------------------------------
Log Entry : 
2022-01-05 20:05:35
  :
  :Amazon.Runtime.AmazonServiceException: Unable to get IAM security credentials from EC2 Instance Metadata Service.
   vid Amazon.Runtime.DefaultInstanceProfileAWSCredentials.FetchCredentials()
   vid Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentials()
   vid Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentialsAsync()
   vid Amazon.Runtime.Internal.CredentialsRetriever.<InvokeAsync>d__7`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__5`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid Amazon.Runtime.Internal.MetricsHandler.<InvokeAsync>d__1`1.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid AWS.Logger.Core.AWSLoggerCore.<LogEventTransmissionSetup>d__33.MoveNext()
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid AWS.Logger.Core.AWSLoggerCore.<Monitor>d__31.MoveNext()
-------------------------------
Log Entry : 
1/5/2022 8:05:45 PM
  :
  :System.TimeoutException: Flush Timeout - ServiceURL=https://xxx.xx-1.amazonaws.com/, StreamName=, PendingMessages=2, CurrentBatch=0
-------------------------------
Log Entry : 
2022-01-05 20:05:45
  :
  :System.Threading.ThreadAbortException: The thread was being aborted.
   vid Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
   vid System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   vid Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   vid Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__5`1.MoveNext()
-------------------------------

Environment

SDK Version:
Package Version:

<package id="AWS.Logger.Core" version="3.0.0" targetFramework="net48" />
  <package id="AWS.Logger.Log4net" version="3.2.1" targetFramework="net48" />
  <package id="AWSSDK.CloudWatchLogs" version="3.7.0.25" targetFramework="net48" />
  <package id="AWSSDK.Core" version="3.7.1" targetFramework="net48" />
  <package id="log4net" version="2.0.12" targetFramework="net48" />

OS Info: Windows 10
Build Environment: Visual Studio
Targeted .NET Platform: .net 4.8

Resolution

This is a 🐛 bug-report

The text was updated successfully, but these errors were encountered:

ashishdhingra · 2022-01-06T14:50:39Z

Hi @alverdal,

Good morning.

Thanks for raising the issue. Could you please share the following:

How are credentials configured for your application? Kindly refer Credential and profile resolution for credential resolution preference.
Share the sample code solution targeting .NET Framework 4.8 to reproduce the issue.

Upon checking the AWSSDK changelog, AWSSDK.Core 3.7.1.0 enabled IPv6 support for IMDS. Not sure if this would be the case for your local development machine since it might not an EC2 instance. Also check if your account under which your web application is running has access to default profile if credentials are configured there on.

Thanks,
Ashish

alverdal · 2022-01-07T10:56:10Z

Hi,

I got no credential configured on my local dev machine.
I should say that I saw that I got the same problem on our test server as well. The logging is working OK on this server when using an earlier version than 3.7.1 (logs showing up in cloudwatch).

Sample code: https://github.com/alverdal/awslog4net-example
You need to publish to IIS for the web site to hang. If running via IIS Express the web site seem to be starting up OK.

ashishdhingra · 2022-01-07T11:01:04Z

Hi,

I got no credential configured on my local dev machine.

I should say that I saw that I got the same problem on our test server as well. The logging is working OK on this server when using an earlier version than 3.7.1 (logs showing up in cloudwatch).

Sample code: https://github.com/alverdal/awslog4net-example

You need to publish to IIS for the web site to hang. If running via IIS Express the web site seem to be starting up OK.

@alverdal I would try to look at your sample. However, you need to have credentials configured in order to send logs to CloudWatch. I'm unsure if any changes in Microsoft request pipeline might have impacted this, aa stated earlier, only IPv6 support for IMDS was added in AWSSDK.Core 3.7.1. Your logs also clearly state that it's unable to fetch credentials from EC2 metadata service, which is the last place where credentials are searched for. Try to configure credentials which has IAM policy for logging into CloudWatch.

alverdal · 2022-01-07T12:34:58Z

Hi, thanks for a quick response :)

To clarify, when I use 3.7.1 version I get no log at all, the application seem to hang and never returns a response. This occuss both locally and on our test server (the test server is configured with working aws credentials).

When I use < 3.7.1 locally the application is working even though it logs some credential errors in the 'AWS.Logger.Log4net.AWSAppender' log file.

When I use < 3.7.1 on the testserver everything is working fine, no credential errors thrown.

So it seems that >= 3.7.1 is hanging the web application both with and without working credentials.

Hopefully you can reproduce the issue with the repo example.

Thanks.

ashishdhingra · 2022-01-07T17:29:26Z

Hi,

I got no credential configured on my local dev machine. I should say that I saw that I got the same problem on our test server as well. The logging is working OK on this server when using an earlier version than 3.7.1 (logs showing up in cloudwatch).

Sample code: https://github.com/alverdal/awslog4net-example You need to publish to IIS for the web site to hang. If running via IIS Express the web site seem to be starting up OK.

@alverdal When you run the application using IIS Express, it runs under the identity of local user account. When the application is run in IIS, it runs under the web application pool identity. This web application pool identity would not have access to create logs, default credentials file would not be searched for in the user's profile location, etc. Please refer to the workaround in the issue #1139 where similar problem was reported. Hope this helps.

alverdal · 2022-01-07T21:07:17Z

I cannot see that it explains why it’s working depending on which version of the sdk I use. I’m publishing to the same site in IIS with the same permissions and it’s only working for version < 3.7.1 . Have you had time to test the repo? Thank you fre 7 jan. 2022 kl. 18:29 skrev Ashish Dhingra ***@***.***>:

…

Hi, I got no credential configured on my local dev machine. I should say that I saw that I got the same problem on our test server as well. The logging is working OK on this server when using an earlier version than 3.7.1 (logs showing up in cloudwatch). Sample code: https://github.com/alverdal/awslog4net-example You need to publish to IIS for the web site to hang. If running via IIS Express the web site seem to be starting up OK. @alverdal <https://github.com/alverdal> When you run the application using IIS Express, it runs under the identity of local user account. When the application is run in IIS, it runs under the web application pool identity. This web application pool identity would not have access to create logs, default credentials file would not be searched for in the user's profile location, etc. Please refer to the workaround in the issue #1139 <#1139> where similar problem was reported. Hope this helps. — Reply to this email directly, view it on GitHub <#1968 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSUMFTAMRVASDJUYOKARPDUU4PIDANCNFSM5LKWWHIQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

alverdal · 2022-01-11T16:55:44Z

@ashishdhingra if it will help to clarify the issue we can have a live session, just let me know :)

ashishdhingra · 2022-01-13T19:46:34Z

While root cause needs to be investigated, issue is reproducible.

STEPS:

Cloned user's web application from https://github.com/alverdal/awslog4net-example.
Downgraded NuGet packages AWSSDK.Core and AWSSDK.CloudWatchLogs to version less than 3.7.1.
Opened IIS and created a new web site named testwebapp linked to http://localhost:8080 (Port 80 is used by default web site).
Used Visual Studio to Publish the web application to the newly created website (Visual Studio needs to be launched as Administrator)
Browsed the new web site (either from within IIS or navigating to http://localhost:8080).
Default web page is displayed. The log file c:\temp\logs\tmp\my_app_awslog.txt (as configured in log4net.config) is created and continuously appended with AWS credentials error(s).
Upgraded NuGet packages AWSSDK.Core and AWSSDK.CloudWatchLogs to the latest version.
Stopped the web site testwebapp. Also stopped World Wide Web Publishing service so that we can delete all logs created at c:\temp\logs\tmp and kill the process which is appending AWS logs.
Redeployed the web application.
Restarted World Wide Web Publishing service and testwebapp web site.
Browsed the new web site (either from within IIS or navigating to http://localhost:8080).

RESULT: The default web page is not displayed and appears to be hanged. The log file c:\temp\logs\tmp\my_app_awslog.txt is not created.

alverdal · 2022-01-17T11:18:06Z

Good job on reproducing the behaviour 👍

ethos-tim · 2024-05-07T18:45:49Z

I have the exact same issue. Ticket here: aws/aws-logging-dotnet#253

ethos-tim · 2024-05-07T18:47:36Z

I looked at a dump of the hung process and see this is where it is stuck:

SYMBOL_NAME: w3wphost!AppHostInitialize+14c
MODULE_NAME: w3wphost
IMAGE_NAME: w3wphost.dll
FAILURE_BUCKET_ID: BREAKPOINT_80000003_w3wphost.dll!AppHostInitialize

ethos-tim · 2024-05-07T18:50:55Z

Another way to test this is have ZERO credentials configured. The old version did not hang, it just did NOT log. The new version hangs instead of bypassing the issue and running without logging. This was working for developers without any credentials until we upgraded. After upgrading to get it to work we had to configure AWS CLI and put in default credentials.

ethos-tim · 2024-05-09T22:14:14Z

I had some time today and was testing this with the SDK and Logger source code and it locks up on this line:

\aws-sdk\sdk\src\Core\Amazon.Runtime\ClientConfig.cs Line 1108

ethos-tim · 2024-05-10T12:58:55Z

It looks to be a race condition between the SDK trying to get the distributed credentials in a timer and both EC2InstanceMetadata and AmazonServiceClient calling into FallbackInternalConfigurationFactory and blocking each other.

Stack traces of what is running when it locks:

FallbackInternalConfigurationFactory is running Reset() while it is also having properties read and it appears to lock.

LMK if I can do anymore to help pinppoint the issue.

ethos-tim · 2024-05-10T16:56:14Z

Settings the environment variable AWS_EC2_METADATA_DISABLED to true stops the lock up.

NOTE: The website runs but it does not send logs with this set.
NOTE2: We have not gotten it to log yet with the new SDK on EC2. I will test locally to see if it logs once I set my CLI back up.

ashovlin · 2024-05-20T21:36:12Z

Big thanks to @alverdal and @ethos-tim for your work investigating and troubleshooting this!

Confirmed that we do have a deadlock between two threads:

The left thread is the one where we're calling LogManager.GetLogger explicitly while using AWS.Logger.Log4net
- It calls GetLogger first, which acquires this lock in log4net's DefaultRepositorySelector
- Later on it tries to initialize its internal AmazonCloudWatchLogsClient. This needs to access the configured retry mode, which leads to the locked call to FallbackInternalConfigurationFactory.RetryMode at:
  
  aws-sdk-net/sdk/src/Core/Amazon.Runtime/ClientConfig.cs
  
  Line 1108 in a2ad0e4
  
  return FallbackInternalConfigurationFactory.RetryMode ?? DefaultConfiguration.RetryMode;
The right thread is the timer that loads credentials from EC2's IMDS
- This beats the left thread to invoking the static constructor for FallbackInternalConfigurationFactory
  when it is trying to load the endpoint of the instance metadata service
  
  aws-sdk-net/sdk/src/Core/Amazon.Util/EC2InstanceMetadata.cs
  
  Line 99 in a2ad0e4
  
  return FallbackInternalConfigurationFactory.EC2MetadataServiceEndpoint;
- This ability to override the IMDS endpoint or to switch between IPv4 and IPv6 variants was introduced in 3.7.1.0 of AWSSDK.Core. Previously the IMDS endpoint was a constant string. This aligns with the reports around when the regression was introduced.
- However, the static constructor for FallbackInternalConfigurationFactory will create EnvironmentVariableInternalConfiguration instance, which gets its own logger here:
  
  aws-sdk-net/sdk/src/Core/Amazon.Runtime/Internal/InternalConfiguration.cs
  
  Line 113 in a2ad0e4
  
  private Logger _logger = Logger.GetLogger(typeof(EnvironmentVariableInternalConfiguration));
- Initializing that logger is blocked on the lock inside log4net.

So thread 2 above is blocked when trying to create a logger on the lock held by thread 1 inside log4net. But thread 1 is also blocked, waiting for that static constructor of FallbackInternalConfigurationFactory to finish on thread 2 so it can access the retry mode to use for the client inside its logger.

Configuring credentials other than EC2's, or setting AWS_EC2_METADATA_DISABLED like you suggest to disable the second thread should avoid this. I'm still working on a code fix.

ashovlin · 2024-05-22T21:41:50Z

We've released AWS.Logger.Core v3.3.3 today, which now lazily initializes the internal CloudWatch Logs client. This should avoid the race condition with the SDK's own logging that was leading to the deadlock.

You could either upgrade your pinned version of AWS.Logger.Core, or to AWS.Logger.Log4net v3.5.3 to pull in the latest core.

Let us know if that doesn't mitigate the issue for you, or if anybody is seeing this outside of AWS.Logger.Log4net. Thanks.

github-actions · 2024-05-22T21:42:07Z

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

alverdal added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jan 5, 2022

ashishdhingra added the module/sdk-core label Jan 5, 2022

ashishdhingra added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed needs-triage This issue or PR still needs to be triaged. labels Jan 6, 2022

ashishdhingra added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. labels Jan 7, 2022

github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jan 8, 2022

ashishdhingra added needs-review A and removed needs-review labels Jan 18, 2022

hunanniu added the queued label Jan 24, 2022

ashishdhingra added p2 This is a standard priority issue and removed A labels Nov 1, 2022

ers-jporche mentioned this issue May 7, 2024

Upgraded Nuget now web app hangs aws/aws-logging-dotnet#253

Closed

ashishdhingra added p1 This is a high priority issue and removed p2 This is a standard priority issue labels May 10, 2024

ashovlin mentioned this issue May 21, 2024

Lazily initialize the internal CloudWatch Logs client to avoid a deadlock when using log4net aws/aws-logging-dotnet#269

Merged

ashovlin closed this as completed May 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

logging via AWS.Logger.Log4net.AWSAppender hangs web app #1968

logging via AWS.Logger.Log4net.AWSAppender hangs web app #1968

alverdal commented Jan 5, 2022

ashishdhingra commented Jan 6, 2022

alverdal commented Jan 7, 2022

ashishdhingra commented Jan 7, 2022 •

edited

Loading

alverdal commented Jan 7, 2022

ashishdhingra commented Jan 7, 2022

alverdal commented Jan 7, 2022 via email

alverdal commented Jan 11, 2022

ashishdhingra commented Jan 13, 2022

alverdal commented Jan 17, 2022

ethos-tim commented May 7, 2024

ethos-tim commented May 7, 2024

ethos-tim commented May 7, 2024

ethos-tim commented May 9, 2024

ethos-tim commented May 10, 2024

ethos-tim commented May 10, 2024 •

edited

Loading

ashovlin commented May 20, 2024

ashovlin commented May 22, 2024

github-actions bot commented May 22, 2024

logging via AWS.Logger.Log4net.AWSAppender hangs web app #1968

logging via AWS.Logger.Log4net.AWSAppender hangs web app #1968

Comments

alverdal commented Jan 5, 2022

Description

Reproduction Steps

Logs

Environment

Resolution

ashishdhingra commented Jan 6, 2022

alverdal commented Jan 7, 2022

ashishdhingra commented Jan 7, 2022 • edited Loading

alverdal commented Jan 7, 2022

ashishdhingra commented Jan 7, 2022

alverdal commented Jan 7, 2022 via email

alverdal commented Jan 11, 2022

ashishdhingra commented Jan 13, 2022

alverdal commented Jan 17, 2022

ethos-tim commented May 7, 2024

ethos-tim commented May 7, 2024

ethos-tim commented May 7, 2024

ethos-tim commented May 9, 2024

ethos-tim commented May 10, 2024

ethos-tim commented May 10, 2024 • edited Loading

ashovlin commented May 20, 2024

ashovlin commented May 22, 2024

github-actions bot commented May 22, 2024

ashishdhingra commented Jan 7, 2022 •

edited

Loading

ethos-tim commented May 10, 2024 •

edited

Loading