Build process of WinUI3 project causes bluescreen error

[Ajith-GS]

Describe the bug

When building a WinUI 3 project, the system sometimes crashes and displays a blue screen error . The system goes to a blue screen as soon as the build process starts.

The dump file of Memory details is given below:

Steps to reproduce the bug

1. Create a WinUI 3 project in VS 2022.
2. Build the project.
   The issue is not reproducible every time, but it is not rare.

Expected behavior

No response

Screenshots

No response

NuGet package version

Windows App SDK 1.6.3: 1.6.241114003

Packaging type

Packaged (MSIX), Unpackaged

Windows version

Windows 10 version 22H2 (19045, 2022 Update)

IDE

Visual Studio 2022

Additional context

OS Version: 22H2 (OS Build 19045.5131)
Visual Studio Version: 17.12.3

[DarranRowe]

How is the general health of the build system?

There are two major types of issues that can cause unexpected bug checks, regularly too. The first is if the Windows install is subtly broken and the second is hardware problems.

Commands like

dism /online /cleanup-image /scanhealth

sfc /scannow

can allow you to check if there is anything majorly problematic. The dism command checks the winsxs directory, where sfc checks the rest.

Hardware is also not an all or nothing situation. Hardware can be working well enough that it will keep working seemingly correctly for ages, but then suddenly just fail. In fact, with the MEMORY_MANAGEMENT bug check code, I would immediately be checking the system RAM for errors. But remember that storage errors, due to the page file, and power errors can also cause this type of bug check.

RAM errors are pretty interesting in this situation. It is easy to believe that the firmware and Windows will know if RAM is bad, but it is very possible for circumstances to result in some RAM issues not being detected. While I admit this will be anecdotal, but one system of mine ended up with one module from the second set of RAM on a channel failing while in the system. The firmware had no issues at all, and Windows kept working. The only sign I had of something not being quite right was when was under pressure in RAM. As soon as Windows went past 32GiB (the system had 64GiB total) allocated, anything allocated would only ever read 0s. It became obvious very quickly that RAM was the problem. Interestingly, at this point I removed second set of RAM, booted the system into the firmware, shut the system down and put the RAM back in and all of the sudden the firmware was indicating that there was a RAM error. It is very possible for RAM to fail while in use and the system just doesn't notice it.

Because of how C++ works, and especially C++/WinRT, the precompiled headers and template expansion can take up a huge amount of address space, which can correspond to a huge amount of RAM if your build system has enough so that it doesn't have to page. What's more, Visual Studio has parallel build set by default. To use my system as an example, it is possible for Visual Studio to run 48 instances of cl.exe which each can take 4GiB+ of address space each. So depending on the project type, amount of physical RAM and the fact that Windows seems to prefer allocating RAM linealry, then it is very possible that this issue is only happening in Visual Studio and only with a WinUI 3 build due to how much memory it can eat up.

If you really believe that this isn't a hardware issue, then I suggest you report this through the feedback hub and be sure to provide the minidumps that you have captured. This could be a Windows issue and that is the best place to report it. If you don't want to try your luck with the feedback hub, then a paid incident for Windows Professional Support or Windows Developer Support would be a better course of action. This will allow a Windows engineer to look at the issue in a relatively short period of time and help you through the problem.

[Ajith-GS]

I would like to add a couple of things also,
We have observed the blue screen issue on multiple machines.
Notably, all these machines were recently upgraded to the latest version of Visual Studio. The issue did not occur with version 17.9.6 or earlier, but it started appearing after upgrading to version 17.11.5. Even with the current version, 17.12.3, the issue persists.
@DarranRowe as you suggested, I will run the scan command and share the results.

[DarranRowe]

One other thing I will ask, is this something that can be triggered without the Windows App SDK, perhaps a sample UWP targetted C++/WinRT project?

The reason why I ask this is that it is strange that the C++ compiler could cause this just by building a project that has the Windows App SDK referenced. A user mode application does not have the reach to cause this without an existing kernel mode issue and there is nothing that referencing the Windows App SDK itself specifically would do to cause issues. Especially since this is being reported in cl.exe, and by this point the compiler is working off of generated files, not the Windows App SDK metadata itself.

One of the biggest potential causes is the rapid increase in memory used by the compiler due to the highly templated code generated by C++/WinRT. To show how bad it can get, here are some screenshots from my system where I started adding these headers to a precompiled header file.

Just by adding the basic Foundation headers, (Windows.Foundation and Windows.Foundation.Collections), the .pch file doubled. Adding most of the headers that I would have in a islands based application pushes the .pch file to over 1.2GiB. With the way that precompiled headers work, this 1.2GiB file would essentially be the initial state of the C++ parse tree. So is it possible to trigger this bug check using other project types that use C++/WinRT to target the UWP? Or in other words, are you able to trigger this generally by using very large .pch files? The fact that multiple instances of cl.exe are run by default should also be taken into account since there would be no indicator as to how many are running by this point.

It is still very possible that there is just some kind of pattern that somehow happens here, but the crash dump posted in the original post is showing an issue in CreateProcess, in particular it looks like there is an issue being shown in creating a section for cl.exe. Think of this as loading the binary information of cl.exe into memory. This places the issue firmly in the purview of Windows itself.

The call stack for where the issue is was the reason why I asked about hardware. This kind of issue is a real indicator of kernel mode structure corruption. This kind of weirdness is mostly caused by hardware, with drivers coming in a close second. However, I did make one assumption, that the build systems are properly specced. Have you verified that the system isn't running out of usable memory? A double whammy of a small amount of RAM and a small drive where the page file is can really cause issues.

[Ajith-GS]

@DarranRowe Scan commands results:

Also C drive has ~40 GB free space and RAM is 32 GB

[DarranRowe]

Well, at this point there isn't much more that I can personally add to this without more information.

I actually pulled some old AMD Threadripper hardware out of storage to test things out. But Visual Studio 2022 17.12.3 has no issues building WinUI 3 projects. I even took out half of the RAM to bring it down to 32GiB to be sure. I'm currently not happy with these tests though because I'm seeing video driver resets occasionally, even when Visual Studio isn't running. So the GPU seems to have gone bad. I can't do much more until after the New Year, but I should be able to get ahold of an Intel A770 for a little while.

The only thing I will point out is that your screenshots names the device which has the ID of a Realtek audio device. Have you checked to see if those drivers are up to date? If you can't do that, have you tried disabling the sound device in Device Manager or even the system's firmware?