[ServiceBusTrigger] The host is disposed and cannot be used

[AlexMasson]

Description

Issue Summary:

Environment: Azure Functions in isolated worker mode, written in C#, running on a Consumption plan, with some ServiceBusTrigger with SessionEnabled = true.

Error Encountered: Sporadically, this exception is thrown, with 2 different stack traces.

The host is disposed and cannot be used. Disposed object: 'Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.ScopedResolver'; Found IListener in stack trace: 'Microsoft.Azure.WebJobs.ServiceBus.Listeners.ServiceBusListener, Microsoft.Azure.WebJobs.Extensions.ServiceBus, Version=5.16.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8' Container is disposed and should not be used: Container is disposed.
You may include Dispose stack-trace into the message via:
container.With(rules => rules.WithCaptureContainerDisposeStackTrace())

Stack Trace 1:

Microsoft.Azure.WebJobs.Script.HostDisposedException:
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.CreateScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/JobHostServiceProvider.cs:110)
   at Microsoft.Azure.WebJobs.Host.Executors.DefaultInstanceServicesProvider.get_InstanceServices (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\DefaultInstanceServicesProvider.cs:26)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInstance.get_InstanceServices (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInstance.cs:56)
   at Microsoft.Azure.WebJobs.Host.Bindings.FunctionBindingContext..ctor (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Bindings\FunctionBindingContext.cs:66)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:279)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:108)
Inner exception DryIoc.ContainerException handled at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.CreateScope:
   at DryIoc.Throw.It (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:9002)
   at DryIoc.Container.ThrowIfContainerDisposed (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:411)
   at DryIoc.Container.WithCurrentScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:437)
   at DryIoc.ResolverContext.OpenScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:2709)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.ScopedResolver.CreateChildScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/ScopedResolver.cs:42)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.CreateScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/JobHostServiceProvider.cs:106)

Stack Trace 2:

Microsoft.Azure.WebJobs.Script.HostDisposedException:
   at Microsoft.Azure.WebJobs.ServiceBus.SessionMessageProcessor.CompleteProcessingMessageAsync (Microsoft.Azure.WebJobs.Extensions.ServiceBus, Version=5.16.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at Microsoft.Azure.WebJobs.ServiceBus.Listeners.ServiceBusListener+<ProcessSessionMessageAsync>d__42.MoveNext (Microsoft.Azure.WebJobs.Extensions.ServiceBus, Version=5.16.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Azure.Messaging.ServiceBus.ServiceBusProcessor+<OnProcessSessionMessageAsync>d__100.MoveNext (Azure.Messaging.ServiceBus, Version=7.17.5.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Azure.Messaging.ServiceBus.ServiceBusSessionProcessor+<OnProcessSessionMessageAsync>d__31.MoveNext (Azure.Messaging.ServiceBus, Version=7.17.5.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Azure.Messaging.ServiceBus.SessionReceiverManager+<OnMessageHandler>d__32.MoveNext (Azure.Messaging.ServiceBus, Version=7.17.5.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Azure.Messaging.ServiceBus.ReceiverManager+<ProcessOneMessage>d__17.MoveNext (Azure.Messaging.ServiceBus, Version=7.17.5.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
Inner exception DryIoc.ContainerException handled at Microsoft.Azure.WebJobs.ServiceBus.SessionMessageProcessor.CompleteProcessingMessageAsync:
   at DryIoc.Throw.It (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:9002)
   at DryIoc.Container.ThrowIfContainerDisposed (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:411)
   at DryIoc.Container.WithCurrentScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:437)
   at DryIoc.ResolverContext.OpenScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/DryIoc/Container.cs:2709)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.ScopedResolver.CreateChildScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/ScopedResolver.cs:42)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.CreateScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: /_/src/WebJobs.Script.WebHost/DependencyInjection/JobHostServiceProvider.cs:106)

Observations:

Occasionally, I notice the following behavior in the logs:

• A log entry appears: Starting JobHost with category Microsoft.Azure.WebJobs.Hosting.JobHostService.
• ServiceBusTrigger listeners initialize with logs like:
  Session initializing (SessionId=XXX, SessionLockedUntil=9/23/2024 8:13:48 AM +00:00) with category Microsoft.Azure.WebJobs.ServiceBus.Listeners.ServiceBusListener.
• Immediately after, I see the log: Stopping JobHost with category Microsoft.Azure.WebJobs.Hosting.JobHostService.

This sequence results in the JobHost stopping right after it starts, which leads to spammed exceptions: The host is disposed and cannot be used. This suggests that the host is shutting down right after initialization, which is causing issues with the listener objects.

We have the MaxDeliveryCount setting configured to 10. Unfortunately, the subsequent attempts after the first failure are made on the same instance, which is stopped and therefore not functioning. I would have expected the retries to occur on a new, working instance.

Problem:

I don't understand why the JobHost starts and then immediately stops, causing this disposal issue.

Context:

The function app is running on the Consumption plan, which may have some relevance to the lifecycle management of the host.

The host.json I use:

{
  "version": "2.0",
  "extensions": {
    "serviceBus": {
      "maxConcurrentSessions": 3,
      "maxConcurrentCalls": 1,
      "sessionIdleTimeout": "00:00:05"
    },
    "queues": {
      "batchSize": 1,
      "maxDequeueCount": 1
    }
  }
}

Steps to reproduce

Create an Azure Functions in isolated worker mode, written in C#, running on a Consumption plan, with some ServiceBusTrigger with SessionEnabled = true.

Then send some messages randomly, until getting this error.

[danielmarbach]

Could this be related? #2687

[AlexMasson]

It's possible although it's still hard to say. I'll keep an eye on your issue as well, thank you.

[boblarson-ka]

We've got either the same or very similar issue. I've got a case open with Microsoft Azure support as well. We're on an elastic premium plan. Actually "comforting" to hear someone is seeing it on consumption. We weren't sure if it was something in the hosting plan.

Will add details that were mildly annoying:
The reference to Service Bus 5.16.1.0 is because the Function Worker version library got tagged with the WebJobs version of the library. Just in case you were chasing that at all, it's definitely not the WebJobs version and just an assembly info file tagged with another project's data in the azure ecosystem.

[BrunoCandia]

We are facing the same issue, would be good if we could get an update about this, thanks!!

[jviau]

This call stack is from the host and appears to be a duplicate of Azure/azure-functions-host#10478

Closing as a duplicate