Using Windows.UI.Xaml.Media.LoadedImageSurface in a winui3 app throws exception

[HO-COOH]

Describe the bug

I wanted to use an image as a Windows.UI.Composition.ICompositionSurface as an input to some win2d effect, and output a Windows.UI.Composition.CompositionBrush used in Micorsoft.UI.Xaml.Window's SystemBackdrop brush.
But I got exceptions in using Windows.UI.Xaml.Media.LoadedImageSurface.StartLoadFromUri, even after I created a DispatcherQueueController on the winui3's ui thread.

Steps to reproduce the bug

1. Create a winui3 packaged project
2. Initialize a DispatcherQueueController in the ui thread

	static winrt::Windows::System::DispatcherQueueController createSystemDispatcherQueueController()
    {
        DispatcherQueueOptions options
        {
            sizeof(DispatcherQueueOptions),
            DQTYPE_THREAD_CURRENT,
            DQTAT_COM_STA
        };

        ::ABI::Windows::System::IDispatcherQueueController* ptr{ nullptr };
        winrt::check_hresult(CreateDispatcherQueueController(options, &ptr));
        return { ptr, take_ownership_from_abi };
    }

3. Call the api

        auto surface =
            winrt::Windows::UI::Xaml::Media::LoadedImageSurface::StartLoadFromUri(winrt::Windows::Foundation::Uri{ L"ms-appx:///Assets/StoreLogo.png" });

Expected behavior

No response

Screenshots

No response

NuGet package version

Windows App SDK 1.5.3: 1.5.240428000

Packaging type

Packaged (MSIX)

Windows version

Windows 11 version 22H2 (22621, 2022 Update)

IDE

Visual Studio 2022

Additional context

No response

[DarranRowe]

	static winrt::Windows::System::DispatcherQueueController createSystemDispatcherQueueController()
    {
        DispatcherQueueOptions options
        {
            sizeof(DispatcherQueueOptions),
            DQTYPE_THREAD_CURRENT,
            DQTAT_COM_STA
        };

        ::ABI::Windows::System::IDispatcherQueueController* ptr{ nullptr };
        winrt::check_hresult(CreateDispatcherQueueController(options, &ptr));
        return { ptr, take_ownership_from_abi };
    }

Does the call to CreateDispatcherQueueController actually work? The documentation for DispatcherQueueOptions states that you shouldn't do what you are doing.

"This field is relevant only if threadType is DQTYPE_THREAD_DEDICATED. Use DQTAT_COM_NONE when DispatcherQueueOptions.threadType is DQTYPE_THREAD_CURRENT."

[HO-COOH]

@DarranRowe I found this code in winui3 gallery. Using any other value results the same

[DarranRowe]

Then there are three things that I can think of.

1. System Xaml needs to be initialised. Have you tried calling Windows.UI.Xaml.Hosting.WindowsXamlManager.InitializeForCurrentThread?
2. This needs the default Xaml metadata, so this would need an instance of a runtime class derived from Windows.UI.Xaml.Application initialised on the thread.
3. For some reason, this requires a CoreWindow, and that isn't available in desktop applications.

They are listed in the order that I think is most likely. For the first two, think of Xaml Islands. Also, remember that system Xaml isn't related to WinUI 3 Xaml, so WinUI 3 wouldn't initialise the Xaml under the Windows root namespace.

--Edit--
Just wrote a little test, and it seems to be 1.

[HO-COOH]

Then there are three things that I can think of.

1. System Xaml needs to be initialised. Have you tried calling Windows.UI.Xaml.Hosting.WindowsXamlManager.InitializeForCurrentThread?
2. This needs the default Xaml metadata, so this would need an instance of a runtime class derived from Windows.UI.Xaml.Application initialised on the thread.
3. For some reason, this requires a CoreWindow, and that isn't available in desktop applications.

They are listed in the order that I think is most likely. For the first two, think of Xaml Islands. Also, remember that system Xaml isn't related to WinUI 3 Xaml, so WinUI 3 wouldn't initialise the Xaml under the Windows root namespace.

--Edit-- Just wrote a little test, and it seems to be 1.

How did you get WindowsXamlManager? This class does not exist in the generated Windows.UI.Xaml.Hosting.h header.

[DarranRowe]

Right, when the project is set up for a Windows Store type project, like WinUI 3 projects are, C++/WinRT doesn't generate the entire set of contracts.
You would need to reference the WindowsDesktop extension SDK. The issue is, the UWP referencing doesn't work for a desktop application. The way I do it is to manually edit the .vcxproj file to add the reference.

The use of $(TargetPlatformVersion) is to make sure that the reference always matches the version of the Windows SDK, so you don't need to edit the project every time. The only other issue is, if you didn't install the Windows SDK in the default location then you would need to add an extra path to the extension sdk search path. If you don't then Visual Studio will fail to find the extension sdk:

2>D:\Programs64\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin\amd64\Microsoft.Common.CurrentVersion.targets(2671,5): error MSB3774: Could not find SDK "WindowsDesktop, Version=10.0.26100.0".

This is because Visual Studio is hard coded to only look in Program Files (x86). So as an example, I actually have the Windows SDK installed on a separate drive, along with Visual Studio and some other applications.

So I have to add this directory to the paths that Visual Studio will look for.

But as I mentioned, this is only needed if the Windows SDK is not installed in the default location.

[sylveon]

This feels like an XY problem. Why are you trying to use WUX.Media.LoadedImageSurface in a WinUI 3 project?

[DarranRowe]

It's because the system backdrops truely do use WU.Composition.CompositionBrush.

            [contract(Microsoft.Foundation.WindowsAppSDKContract, 1.1)]
            [uuid(397DAFE4-B6C2-5BB9-951D-F5707DE8B7BC)]
            interface ICompositionSupportsSystemBackdrop : IInspectable
            {
                [propget] HRESULT SystemBackdrop([out] [retval] Windows.UI.Composition.CompositionBrush** value);
                [propput] HRESULT SystemBackdrop([in] Windows.UI.Composition.CompositionBrush* value);
            }

This is the IDL generated from the 1.5 metadata for ICompositionSupportsSystemBackdrop as one example. WUX.Media.LoadedImageSurface probably just gave the easiest way of loading an image into a composition brush.

[sylveon]

Are there non-XAML ways to get an image into a composition brush then?

[HO-COOH]

@DarranRowe Your method is unreasonably complicated. WASDK team better enable use of Windows.UI.Composition by default in the project template

[DarranRowe]

My method isn't unreasonably complicated, it was only telling you how to use the runtime class that you wanted to use. That response also served to indicate that this wasn't a Windows App SDK issue since the System Xaml is where your problem was, and initialising System Xaml using the hosting API resolved your initial problem completely.
The use of Windows.UI.Composition here is a different thing entirely, and is enabled by default since all of Windows.UI.Composition is in the Universal API Contract. The issue is, as documented by Windows.UI.Composition.CompositionDrawingSurface, there are three ways to load the CompositionDrawingSurface. Using Direct2D and ICompositionDrawingSurfaceInterop is really your only option if you don't want to have anything extra but want to use Windows.UI.Composition. Win2D is also available if you don't mind adding that. As you already found out, using LoadedImageSurface requires initialising System Xaml and using MediaPlayer for a static image is a bit much.

If you want to use runtime classes under Windows.UI.Xaml without using the hosting interfaces to initialise it, then you really should submit a feature request for Microsoft.UI.Xaml.Application.Start to initialise Windows.UI.Xaml too.

[HO-COOH]

@DarranRowe Don't get me wrong, I really appreciate your solution. As it happens many times before, I just want them to either: document the usage with winui3 clearly so nobody gets wasting their time (since WUC namespace is used by this public SystemBackdrop API) OR have it working out-of-the-box by initializing whatever system xaml in the project template's startup code (also that WinUI3 project template is provided by WASDK team I guess). Manually editing any vcxproj file is not an acceptable solution to me.

[DarranRowe]

As I stated though, Windows.UI.Composition is fully usable regardless. In this case, it is a System Xaml helper which is the issue. The biggest thing I always assume when working with System Xaml is that anything under Windows.UI.Xaml is unusable until either Windows.UI.Xaml.Application.Start has been called, or Windows.UI.Xaml.Hosting.WindowsXamlManager.InitializeForCurrentThread has been called. The only surprising thing I know of is that Windows.UI.Color.ColorHelper.ToDisplayName calls into Windows.UI.Xaml.dll.
In the end, I would say that any documentation for System Xaml is outside the scope of the Windows App SDK, and the documentation for this is in the MicrosoftDocs/winrt-api repository. You could always file an issue or a pr over there.
As for using Windows.UI.Composition in a WinUI 3 application. For some quick and dirty pointers as to how you would do this using Direct2D/Direct3D, you would follow the general steps of:

1. Initialise Direct3D and/or Direct2D and the Compositor, if needed.
2. Create the CompositionGraphicsDevice.
3. Create the CompositionDrawingSurface.
4. Draw to the surface.

    winrt::Windows::UI::Composition::CompositionGraphicsDevice MainWindow::create_graphics_device()
    {
        if (!m_compositor)
        {
            m_compositor = winrt::Windows::UI::Composition::Compositor();
        }

        if (!is_d2d_init())
        {
            check_hresult(init_d2d());
        }

        auto comp_interop = m_compositor.as<ABI::Windows::UI::Composition::ICompositorInterop>();
        
        winrt::Windows::UI::Composition::CompositionGraphicsDevice graphics_device{ nullptr };

        //This uses an ID2D1Device to create the CompositionGraphicsDevice, you can also use ID3D11Device too.
        check_hresult(comp_interop->CreateGraphicsDevice(m_d2d1_device.get(), reinterpret_cast<ABI::Windows::UI::Composition::ICompositionGraphicsDevice **>(winrt::put_abi(graphics_device))));

        return graphics_device;
    }

Once you have the CompositionGraphicsDevice, then you can use one of the surface creation members to create a surface. Once you have the surface, you can then use ICompositionSurfaceInterop to control the drawing.

    std::pair<winrt::com_ptr<ID2D1DeviceContext>, POINT> MainWindow::begin_draw(const winrt::Windows::UI::Composition::CompositionDrawingSurface &drawing_surface)
    {
        auto drawing_surface_interop = drawing_surface.as<ABI::Windows::UI::Composition::ICompositionDrawingSurfaceInterop>();

        //You must use ID2D1DeviceContext, ID2D1DeviceContext1 onwards will not work.
        winrt::com_ptr<ID2D1DeviceContext> ds;
        POINT up_off{};
        check_hresult(drawing_surface_interop->BeginDraw(nullptr, IID_PPV_ARGS(ds.put()), &up_off));

        return std::make_pair(ds, up_off);
    }

Once you have finished drawing to the surface, then you must call EndDraw.

    void MainWindow::end_draw(const winrt::Windows::UI::Composition::CompositionDrawingSurface &drawing_surface)
    {
        auto drawing_surface_interop = drawing_surface.as<ABI::Windows::UI::Composition::ICompositionDrawingSurfaceInterop>();

        check_hresult(drawing_surface_interop->EndDraw());
    }

There is some quirkiness here, but it is easy to explain the behaviour when you understand that the desktop targetted version of Windows.UI.Composition uses the underlying DirectComposition.

Anyway, the things to note using this. If you pass in a ID3D11Device, then you can only get a IDXGISurface/ID3D11Texture2D from BeginDraw. If you pass in a ID2D1Device, it does the equivalent of ID2D1Device::CreateDeviceContext, ID2D1DeviceContext::CreateBitmapFromDxgiSurface, sets the newly created bitmap as the target for the ID2D1DeviceContext using ID2D1DeviceContext::SetTarget and then returns the ID2D1DeviceContext.

Using System Xaml does allow for some shortcuts, because it needs to use Direct2D and Windows.UI.Composition to draw. So it has its own compositor and Direct2D device. However, if you use Direct2D and your own compositor instance, then you don't need to initialise System Xaml and thus you don't have to edit the .vcxproj file manually.

This is why I personally consider this outside of the scope of the Windows App SDK, since it uses the inbuilt system functionality, and you can do what you want without editing anything. It is just the shortcut under the Windows.UI.Xaml namespace that doesn't work unless System Xaml is initialised.

[HO-COOH]

@DarranRowe I got

Exception thrown at 0x00007FFBDBD2567C (KernelBase.dll) in WinUI3Example.exe: WinRT originate error - 0x802B000A : 'Cannot create instance of type 'Windows.UI.Xaml.Controls.XamlControlsResources' [Line: 11 Position: 40]'.

when I do

        auto m_manager = winrt::Windows::UI::Xaml::Hosting::WindowsXamlManager::InitializeForCurrentThread();

Additional context:
I did not add reference through editing .vcxproj tho. I did it with adding reference inside visual studio's project reference dialog:

[DarranRowe]

In regards to the additional context.

2024-06-20.11-59-47.mp4

2024-06-20.12-02-06.mp4

As I have previously stated in this thread, I do not use the default installation locations for the Windows SDK and Visual Studio, and Visual Studio hard codes the Extension SDK path. That is why I have to override the SDKExtensionDirectoryRoot property if I want to use any of this. You can see this for yourself in the Microsoft.Cpp.AppContainerApplication.props file in that is part of MSBuild/Visual Studio.

<SDKExtensionDirectoryRoot Condition="'$(SDKExtensionDirectoryRoot)' == '' and '$(SDKIdentifier)' != ''">$(MSBuildProgramFiles32)\Microsoft SDKs\Windows Kits\10;$(MSBuildProgramFiles32)\Windows Kits\10</SDKExtensionDirectoryRoot>

Anyway, for that error, that is doubly curious and problematic. It also pushes the goal post further. I'm going to have to work out what I missed with my test. However, at the very least it means that it is going to require an application (something derived from Windows.UI.Xaml.Application) active on the thread. Because this also means that there are multiple frameworks active, then it also means that System Xaml really should go onto a separate thread.
Anyway, I'm going to have to investigate this a little more.

[DarranRowe]

I spent some time following this rabbit hole, and it was a very interesting learning experience. I'll break this post into three sections based upon the various things discussed.

Windows.UI.Xaml

I was looking around for my original research into this, and I have deleted all of the projects that were related to this. So I can't remember how I came to the conclusion that I did. However, it seems like I was completely wrong.
I actually started a project from the destkop application template, and slowly converted it to a WinUI 3 project. At several stages along the way, I tested whether or not the hosting API would succeed or fail. Interestingly, it wasn't until App.xaml was added that it did fail. It is something about the generated code that causes this. The really interesting thing is that I was testing this in an unpackaged application, and I was trying to initialise the hosting API before the Windows App Runtime was bootstrapped.
So WinUI 3 does something that just gets in the way of System Xaml's initialisation.

Using Add Reference

Interestingly, if you try to use Add Reference in the template that has the separate packaging project, this doesn't work. What actually gets this working is the single project MSIX packaging extension. This means that the project has to reference the Windows App SDK and use the EnableMsixTooling element in the project file for this to work for desktop applicaitons. I haven't fully tested all combinations though.

Windows.UI.Composition

Everything under Windows.UI.Composition does fully work even when WinUI 3 is loaded in a process. But as previously mentioned, you need to provide your own compositor and drawing. I wrote a sample application that created two windows. One was a WinUI 3 window and one was a Windows API window. The WinUI 3 window loaded a PNG file, and then used this to create a CompositionDrawingSurface. I then used this as content for the Windows API window, which had been set as a target.
The only thing that could get in the way is that you need to use the same compositor instance as the root of all of the objects. The D2D and D3D drawing objects don't need to stay the same, and you don't even need to create a swap chain, so the D2D usage is pretty simple.