How to troubleshoot packaged apps?

[tschbc]

I've had a bit of a ride trying to troubleshoot an issue with packaged releases of my .NET MAUI 8 app on Windows 10.

A month ago, the app began malfunctioning in a way that has been difficult to fix/reproduce overall. I'm certain it's due to being packaged via MSIX (which is why I made this issue in this repo), or perhaps something during the publish process, because these issues don't happen when running locally in Debug or Release configuration.

Are there more effective tools or techniques I can use to troubleshoot a packaged and signed release build? I consider myself pretty green when it comes to Windows development, so maybe I'm just missing something obvious or well-known.

Issues I'm encountering

• Some string resources are missing in UI but others are shown. They're declared in the same RESX file and retrieved the same way.
• The MAUI Preferences API (ApplicationDataContainer/LocalSettings) is failing without any exceptions or errors thrown (that I can see, anyway).
• Navigating some pages crashes the app. Checking Event Viewer, I see Faulting module name: KERNELBASE.dll, version: 10.0.19041.4957. So far I haven't been able to find much relevant info online about its exception code 0xc000027b.
• After our login flow, the app usually crashes immediately on launch. Sometimes, it won't crash after repeatedly launching it.

What I've tried

• Writing to Event Viewer from an UnhandledException handler which didn't provide meaningful exception messages.
• Running the package via WinDbg which showed opaque exception messages like "No interface supported" (these were normally crashes, but WinDbg allowed the app to continue running anyway)
• Uninstalling any WindowsAppSDK 1.6.x versions, but they get reinstalled during app install, even though the app uses 1.5
• Changing .NET Desktop Runtime versions, or building with different runtimes in self-contained builds

Red Herrings

There's been a lot of false starts I've experienced trying to troubleshoot this. I would change one thing, generate a release build for our team to test, and the same build might work for one person but not another.

Most recently, we tried various builds of .NET runtime self-contained, WindowsAppSDK self-contained, or framework dependent—combined with installing different .NET Desktop Runtime versions for the framework-dependent builds. Sometimes we could fix the issues with this, other times it would re-break it. But we can never reliably reproduce a break or fix, so it just seems coincidental. This was without any code changes as well.

We thought maybe our VPN was affecting this, so we even re-tried all those different builds on different VPN connections, or forgoing VPN completely. We experienced similar "coincidental" breaks/fixes during this—still unable to reproduce.

To add to the confusion, when we first experienced the errors we tried installing older builds that we knew worked correctly. Those builds also experienced these new errors. To make it even more confusing, all of those builds had the .NET runtime bundled self contained.

Then, when we were randomly able to fix the issues for some people, we tried installing the troubleshooting builds that were malfunctioning and they started working correctly. Didn't matter if they were self-contained or not.

[DarranRowe]

This kind of question is always difficult.

First of all, just to get it out of the way, application packaging is unrelated to the Windows App SDK. The WinUI 3 application type does use packaging, but in general this is a Windows component. The tooling to create packages come with the Windows SDK, which is a different component.

Now that is out of the way, there are two points to cover. The first is be aware of the current directory when the application is packaged may not be what you expect. Windows has information for what the current directory should be in most cases. The directory that the Explorer window is in when you double click on a file, the shortcut, the console's current directory and so on. But packages normally don't have any of this information and so package normally start off with the current directory set to System32/SysWoW64. It is possible to set this directory in the package definition, but this requires uap11, in other words Windows 11. The environment that the application runs in is described in Understanding how packaged desktop apps run on Windows.

The second is that the Windows App SDK dependency is picked up from the PackageDependency element in the appxmanifest file. If you are using a packaging project in Visual Studio, Visual Studio will fill out the PackageDependency elements based upon referenced NuGet packages. The reason why this is important is that if the package installation is installing 1.6, then you have 1.6 referenced in the packaging project. A separate packaging project can reference a different set of NuGet packages than the application project, so it is possible to have the application referencing 1.5 but the packaging project referencing 1.6. Also, if you have a separate packaging project and you want to make the Windows App SDK self contained, then you must set WindowsAppSDKSelfContained to true in both the application project and the packaging project. This is described in Windows App SDK deployment guide for self-contained apps. The purpose of this in the packaging project is to remove the PackageDependency in the application package.

Another thing to note is that not everything written to the debugger is a non-continuable error. OutputDebugString allows you to just write a string to the debugger, and exceptions that get caught and handled also show up in the debugger output. So errors like "No interface supported" don't have to be a non continuable error, and I think you normally see 3 or 4 of those when you start a WinUI 3 application anyway. You might not see it with the managed debugger in Visual Studio, but if you set the debugger to mixed mode, or use a C++ project then you will see this.

The exception 0xC000027B is the NTSTATUS code for the stowed exception. This type of exception is thrown when RoOriginateException is called on a worker thread. There were a couple of videos on this back in 2018. You need to use a windbg extension to extract the actual exception record.

This is all I can think of for now/

[tschbc]

@DarranRowe Thank you for this! Being able to read the stowed exception put me more on the right track.

The problem ultimately ended up being caused by a race condition. There is a component we use that we often retrieve multiple instances of, but it should only ever be initialized once. On first launch, multiple threads may call for their own unique initialization, clobbering any initialization another thread performed. I put in a semaphore and it now works flawlessly (hopefully forever).

Exceptions thrown from these failed calls were the culprit for all of the weird errors. I think I mistook it for package issues because of my unfamiliarity combined with it only happening when installed via MSIX (it certainly didn't help that this coincidentally started when the version of WinAppSDK that broke Appx cmdlets was released). But I think the real difference was in Release optimizations.