Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Methane Kit v0.8.0: enhance program bindings configuration, performance and simplified shader buffer arguments setup with root constants #147

Merged
merged 210 commits into from
Dec 31, 2024

Conversation

egorodet
Copy link
Member

@egorodet egorodet commented Dec 30, 2024

Summary

In Methane Kit v0.8.0, several significant enhancements have been introduced:

  • Improved Program Bindings Configuration: The configuration of program bindings has been enhanced by defining argument access frequency directly within HLSL argument space modifiers. Previously, this was configured in the C++ program settings.
  • Enhanced Performance for Metal RHI: Performance of program bindings for Metal RHI has been improved by supporting the Argument Buffers API. This allows for more efficient writing of binding commands in command encoders.
  • Simplified Shader Argument Buffers Initialization: A simplified interface for initializing shader argument buffers using root constants has been added. This enables direct data assignment to program argument bindings and handles all buffer management within Methane RHI. For small constants "Push Constants" native API is also supported via RHI of program bindings.

Tutorial applications

  • Root-constants were applied across all tutorials instead of uniform buffers to simplify code.
  • Console Compute tutorial was extended with game rule selection using root constant value interface of program bindings.
  • ParallelRendering app can be built in two versions: first one is using addressable buffer views like before and second one is using root constants but with lower performance.

Graphics libraries

  • Shader argument access types were added in HLSL shaders as register space definitions (META_ARG_CONSTANT, META_ARG_FRAME_CONSTANT, META_ARG_MUTABLE), to replace shader arguments configuration Rhi::ProgramArgumentAccessors in ProgramSettings. RHI shader arguments reflection uses these register space definitions to add default argument accessors configuration, so that it is not necessary to be defined in user code of ProgramSettings.
  • Rhi::RootConstant class was added as a storage of constant data (derived from Data::Chunk), which can be set to buffer argument bindings, defined with ProgramArgumentValueType::RootConstantBuffer in Rhi::ProgramArgumentAccessors. Rhi::IProgramBindings::SetRootConstant allows to set root constant data for such argument bindings. Internally managed Rhi::RootConstantBuffer objects significantly simplify user code by eliminating the need in creating separate uniform buffers. Support of Root32BitConstants in DirectX and PushConstants in Vulkan and SetBytes in Metal was added via Rhi::ProgramArgumentValueType::RootConstantValue (close Extend Program Bindings with support of "Push Constants" native API #140).
  • Metal::ProgramBindings in Metal RHI were extended with Metal Argument Buffers API support for more efficient shader argument bindings, similar to descriptor tables in DirectX 12 and descriptor sets in Vulkan. CMake option METHANE_METAL_ARGUMENT_BUFFERS_ENABLED was added to allow switching between direct resource binding and argument buffers (enabled by default). Metal argument buffers are created independently for groups of shader arguments with different argument access types. (close Metal RHI improvement: Add support of MTLArgumentBuffer in program bindings #138)
  • Metal::ProgramBindings were refactored to reduce code duplication with help of templates.
  • ProgramArgument and ProgramArgumentAccessor types were moved from IProgram.h header to ProgramArgument.h/cpp. ProgramArgumentAccessModifier enum was renamed to ProgramArgumentValueType.
  • Shader argument declaration macros were added in ProgramArgument.h to simplify program argument accessors setup: META_PROGRAM_ARG_* (ROOT_CONSTANT, ROOT_FRAME_CONSTANT, ROOT_MUTABLE, RESOURCE_CONSTANT, RESOURCE_FRAME_CONSTANT, RESOURCE_MUTABLE, etc.).
  • CommandListDebugGroup sub-group names now use std::string_view
  • ScreenQuad was migrated to root constants instead of uniform buffer.
  • Fixed Vulkan validation issues revealed by Vulkan SDK 1.3.290.0.
  • Fixed DirectX 12 GPU timestamp queries for DirectX, fixed invalid initial state of the read-back buffer.
  • Fixed Metal crash due to executing state of the current frame command lists and sporadic frame synchronization issue. VSync was disabled by default for Metal.

Data libraries

  • Added support of prioritizing receivers in emitter list with priority argument of method Emitter<ICallback>::Connect(Receiver<ICallback>& receiver, uint32_t priority).

Common changes

  • Renamed all macroses META_CHECK_ARG_* by removal of ARG suffix to META_CHECK_*.
  • Apply hidden friend function pattern to binary operators by recommendation from SonarCloud.

User Interface libraries

  • Text program bindings now use root constants instead of manually created uniform buffers.

Tests

  • Added unit-tests to cover more classes of the "Graphics RHI" layer (continue Add Graphics RHI unit tests #122):
    • ViewState test was added
    • Fence test was extended
    • BufferSet test was added
    • CommandKit test was added
    • CommandListDebugGroup test was added
    • ProgramBindings test was extended with root constant bindings.
  • Data::EventsTest was extended to check prioritizing of connected receivers.

External libraries

Build

  • Experimental support of Apple's Metal Shader Converter was added to be used instead of SPIRV-Cross. Unfortunately, it does not work correctly in all cases and is disabled by default.
  • Updated CMakeModules/FindWindowsSDK.cmake with fix for ARM64 architecture detection with VS2022 and Ninja and detection of the latest Windows SDK version 10.0.22621.
  • Added support of XCode 16 for MacOS Sequoia 15. Also fixed MacOS build compatibility with older Xcode versions.
  • Apple application RPATH /usr/local/lib was added to find libvulkan.1.lib
  • Added CMake option METHANE_METAL_FRAMES_SYNC_WITH_DISPATCH_SEMAPHORE
  • Build\Windows\Build.bat was extended with more build options.

Continuous Integration

  • Allow manual dispatch of CI Build, CodeQL and Sonar Scan workflows in GitHub Actions.
  • Switched Sonar-Scan action back to macos-13 hosted runners to use Intel h/w, instead of M1/ARM, which is not supported yet.
  • Update all GitHub actions to use latest version of Node.js.
  • Fixed CodeCov coverage results upload.
  • CodeQL scan was re-enabled on Windows.
  • Vulkan SDK was updated to v1.3.290.0 in macOS CI.

Documentation

  • Updated all tutorials documentation according to latest code changes and with help of Copilot enhancements.
  • Added documentation to CubeMapArray and ParallelRendering tutorials.

egorodet added 30 commits July 15, 2023 22:37
@egorodet egorodet added this to the v0.8 milestone Dec 30, 2024
@egorodet egorodet added enhancement New feature or request infrastructure Build, tools, automation, etc. documentation Write documentation of some feature labels Dec 30, 2024
@egorodet egorodet self-assigned this Dec 30, 2024
Copy link

codecov bot commented Dec 30, 2024

Codecov Report

Attention: Patch coverage is 70.21563% with 442 lines in your changes missing coverage. Please review.

Project coverage is 36.20%. Comparing base (b1ba139) to head (b5222a5).
Report is 158 commits behind head on master.

✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...RHI/Base/Sources/Methane/Graphics/Base/Program.cpp 55.27% 47 Missing and 4 partials ⚠️
...erface/Include/Methane/Graphics/RHI/ResourceView.h 40.00% 45 Missing ⚠️
...urces/Methane/Graphics/Base/RootConstantBuffer.cpp 83.25% 31 Missing ⚠️
...ics/Primitives/Sources/Methane/Graphics/SkyBox.cpp 0.00% 24 Missing ⚠️
...Primitives/Sources/Methane/Graphics/ScreenQuad.cpp 0.00% 21 Missing ⚠️
.../Sources/Methane/Graphics/Base/ProgramBindings.cpp 81.42% 21 Missing ⚠️
...e/Sources/Methane/Graphics/RHI/ProgramArgument.cpp 68.34% 19 Missing ⚠️
...ources/Methane/Graphics/Base/RenderCommandList.cpp 51.62% 15 Missing ⚠️
Modules/Data/Types/Include/Methane/Data/Chunk.hpp 56.67% 13 Missing ⚠️
...rimitives/Sources/Methane/Graphics/ImageLoader.cpp 0.00% 13 Missing ⚠️
... and 54 more
Additional details and impacted files
@@             Coverage Diff             @@
##           master     #147       +/-   ##
===========================================
+ Coverage   25.19%   36.20%   +11.02%     
===========================================
  Files         367      381       +14     
  Lines       14372    19490     +5118     
  Branches        0     1786     +1786     
===========================================
+ Hits         3619     7055     +3436     
- Misses      10753    12257     +1504     
- Partials        0      178      +178     
Flag Coverage Δ
Linux 27.21% <65.88%> (+2.03%) ⬆️
Windows 59.89% <72.86%> (?)
macOS 59.17% <62.78%> (?)
unittests 36.20% <70.22%> (+11.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

github-actions bot commented Dec 30, 2024

Win64_DX_SonarScan Test Results

    8 files  ±  0      8 suites  ±0   5s ⏱️ ±0s
3 594 tests +476  3 594 ✅ +476  0 💤 ±0  0 ❌ ±0 
9 486 runs  +297  9 486 ✅ +297  0 💤 ±0  0 ❌ ±0 

Results for commit b5222a5. ± Comparison against base commit 4c66c85.

This pull request removes 1 and adds 477 tests. Note that renamed tests count towards both.
MethaneGraphicsRhiTest.exe.global ‑ RHI Compute State Functions/Context Construction
MethaneDataEventsTest.exe.global ‑ Connect emitter to receiver through the transmitter
MethaneDataEventsTest.exe.global ‑ Connect many emitters to one receiver
MethaneDataEventsTest.exe.global ‑ Connect one emitter to many receivers
MethaneDataEventsTest.exe.global ‑ Connect one emitter to many receivers/Emit by priority
MethaneDataEventsTest.exe.global ‑ Connect one emitter to one receiver
MethaneDataRangeSetTest.exe.global ‑ Range initialization
MethaneDataRangeSetTest.exe.global ‑ Range operations
MethaneDataRangeSetTest.exe.global ‑ Range relations
MethaneDataRangeSetTest.exe.global ‑ Range set add
MethaneDataRangeSetTest.exe.global ‑ Range set initialization
…

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Dec 30, 2024

Ubuntu_VK_SonarScan Test Results

    8 files  ±  0      8 suites  ±0   1s ⏱️ ±0s
3 594 tests +476  3 594 ✅ +476  0 💤 ±0  0 ❌ ±0 
9 486 runs  +297  9 486 ✅ +297  0 💤 ±0  0 ❌ ±0 

Results for commit b5222a5. ± Comparison against base commit 4c66c85.

This pull request removes 1 and adds 477 tests. Note that renamed tests count towards both.
MethaneGraphicsRhiTest.global ‑ RHI Compute State Functions/Context Construction
MethaneDataEventsTest.global ‑ Connect emitter to receiver through the transmitter
MethaneDataEventsTest.global ‑ Connect many emitters to one receiver
MethaneDataEventsTest.global ‑ Connect one emitter to many receivers
MethaneDataEventsTest.global ‑ Connect one emitter to many receivers/Emit by priority
MethaneDataEventsTest.global ‑ Connect one emitter to one receiver
MethaneDataRangeSetTest.global ‑ Range initialization
MethaneDataRangeSetTest.global ‑ Range operations
MethaneDataRangeSetTest.global ‑ Range relations
MethaneDataRangeSetTest.global ‑ Range set add
MethaneDataRangeSetTest.global ‑ Range set initialization
…

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Dec 30, 2024

MacOS_MTL_SonarScan Test Results

    8 files  ±  0      8 suites  ±0   1s ⏱️ ±0s
3 594 tests +476  3 594 ✅ +476  0 💤 ±0  0 ❌ ±0 
9 486 runs  +297  9 486 ✅ +297  0 💤 ±0  0 ❌ ±0 

Results for commit b5222a5. ± Comparison against base commit 4c66c85.

This pull request removes 1 and adds 477 tests. Note that renamed tests count towards both.
MethaneGraphicsRhiTest.global ‑ RHI Compute State Functions/Context Construction
MethaneDataEventsTest.global ‑ Connect emitter to receiver through the transmitter
MethaneDataEventsTest.global ‑ Connect many emitters to one receiver
MethaneDataEventsTest.global ‑ Connect one emitter to many receivers
MethaneDataEventsTest.global ‑ Connect one emitter to many receivers/Emit by priority
MethaneDataEventsTest.global ‑ Connect one emitter to one receiver
MethaneDataRangeSetTest.global ‑ Range initialization
MethaneDataRangeSetTest.global ‑ Range operations
MethaneDataRangeSetTest.global ‑ Range relations
MethaneDataRangeSetTest.global ‑ Range set add
MethaneDataRangeSetTest.global ‑ Range set initialization
…

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Dec 30, 2024

Win64_VK_Release Test Results

    8 files  ±  0      8 suites  ±0   0s ⏱️ ±0s
3 594 tests +476  3 594 ✅ +476  0 💤 ±0  0 ❌ ±0 
9 486 runs  +297  9 486 ✅ +297  0 💤 ±0  0 ❌ ±0 

Results for commit b5222a5. ± Comparison against base commit 4c66c85.

This pull request removes 1 and adds 477 tests. Note that renamed tests count towards both.
MethaneGraphicsRhiTest.exe.global ‑ RHI Compute State Functions/Context Construction
MethaneDataEventsTest.exe.global ‑ Connect emitter to receiver through the transmitter
MethaneDataEventsTest.exe.global ‑ Connect many emitters to one receiver
MethaneDataEventsTest.exe.global ‑ Connect one emitter to many receivers
MethaneDataEventsTest.exe.global ‑ Connect one emitter to many receivers/Emit by priority
MethaneDataEventsTest.exe.global ‑ Connect one emitter to one receiver
MethaneDataRangeSetTest.exe.global ‑ Range initialization
MethaneDataRangeSetTest.exe.global ‑ Range operations
MethaneDataRangeSetTest.exe.global ‑ Range relations
MethaneDataRangeSetTest.exe.global ‑ Range set add
MethaneDataRangeSetTest.exe.global ‑ Range set initialization
…

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Dec 30, 2024

Win32_DX_Release Test Results

    8 files  ±  0      8 suites  ±0   0s ⏱️ ±0s
3 594 tests +476  3 594 ✅ +476  0 💤 ±0  0 ❌ ±0 
9 486 runs  +297  9 486 ✅ +297  0 💤 ±0  0 ❌ ±0 

Results for commit b5222a5. ± Comparison against base commit 4c66c85.

This pull request removes 1 and adds 477 tests. Note that renamed tests count towards both.
MethaneGraphicsRhiTest.exe.global ‑ RHI Compute State Functions/Context Construction
MethaneDataEventsTest.exe.global ‑ Connect emitter to receiver through the transmitter
MethaneDataEventsTest.exe.global ‑ Connect many emitters to one receiver
MethaneDataEventsTest.exe.global ‑ Connect one emitter to many receivers
MethaneDataEventsTest.exe.global ‑ Connect one emitter to many receivers/Emit by priority
MethaneDataEventsTest.exe.global ‑ Connect one emitter to one receiver
MethaneDataRangeSetTest.exe.global ‑ Range initialization
MethaneDataRangeSetTest.exe.global ‑ Range operations
MethaneDataRangeSetTest.exe.global ‑ Range relations
MethaneDataRangeSetTest.exe.global ‑ Range set add
MethaneDataRangeSetTest.exe.global ‑ Range set initialization
…

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Dec 30, 2024

MacOS_VK_Release Test Results

    8 files  ±  0      8 suites  ±0   1s ⏱️ ±0s
3 594 tests +476  3 594 ✅ +476  0 💤 ±0  0 ❌ ±0 
9 486 runs  +297  9 486 ✅ +297  0 💤 ±0  0 ❌ ±0 

Results for commit b5222a5. ± Comparison against base commit 4c66c85.

This pull request removes 1 and adds 477 tests. Note that renamed tests count towards both.
MethaneGraphicsRhiTest.global ‑ RHI Compute State Functions/Context Construction
MethaneDataEventsTest.global ‑ Connect emitter to receiver through the transmitter
MethaneDataEventsTest.global ‑ Connect many emitters to one receiver
MethaneDataEventsTest.global ‑ Connect one emitter to many receivers
MethaneDataEventsTest.global ‑ Connect one emitter to many receivers/Emit by priority
MethaneDataEventsTest.global ‑ Connect one emitter to one receiver
MethaneDataRangeSetTest.global ‑ Range initialization
MethaneDataRangeSetTest.global ‑ Range operations
MethaneDataRangeSetTest.global ‑ Range relations
MethaneDataRangeSetTest.global ‑ Range set add
MethaneDataRangeSetTest.global ‑ Range set initialization
…

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Dec 30, 2024

Win32_VK_Release Test Results

    8 files  ±  0      8 suites  ±0   0s ⏱️ ±0s
3 594 tests +476  3 594 ✅ +476  0 💤 ±0  0 ❌ ±0 
9 486 runs  +297  9 486 ✅ +297  0 💤 ±0  0 ❌ ±0 

Results for commit b5222a5. ± Comparison against base commit 4c66c85.

This pull request removes 1 and adds 477 tests. Note that renamed tests count towards both.
MethaneGraphicsRhiTest.exe.global ‑ RHI Compute State Functions/Context Construction
MethaneDataEventsTest.exe.global ‑ Connect emitter to receiver through the transmitter
MethaneDataEventsTest.exe.global ‑ Connect many emitters to one receiver
MethaneDataEventsTest.exe.global ‑ Connect one emitter to many receivers
MethaneDataEventsTest.exe.global ‑ Connect one emitter to many receivers/Emit by priority
MethaneDataEventsTest.exe.global ‑ Connect one emitter to one receiver
MethaneDataRangeSetTest.exe.global ‑ Range initialization
MethaneDataRangeSetTest.exe.global ‑ Range operations
MethaneDataRangeSetTest.exe.global ‑ Range relations
MethaneDataRangeSetTest.exe.global ‑ Range set add
MethaneDataRangeSetTest.exe.global ‑ Range set initialization
…

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Dec 30, 2024

MacOS_MTL_Release Test Results

    8 files  ±  0      8 suites  ±0   0s ⏱️ ±0s
3 594 tests +476  3 594 ✅ +476  0 💤 ±0  0 ❌ ±0 
9 486 runs  +297  9 486 ✅ +297  0 💤 ±0  0 ❌ ±0 

Results for commit b5222a5. ± Comparison against base commit 4c66c85.

This pull request removes 1 and adds 477 tests. Note that renamed tests count towards both.
MethaneGraphicsRhiTest.global ‑ RHI Compute State Functions/Context Construction
MethaneDataEventsTest.global ‑ Connect emitter to receiver through the transmitter
MethaneDataEventsTest.global ‑ Connect many emitters to one receiver
MethaneDataEventsTest.global ‑ Connect one emitter to many receivers
MethaneDataEventsTest.global ‑ Connect one emitter to many receivers/Emit by priority
MethaneDataEventsTest.global ‑ Connect one emitter to one receiver
MethaneDataRangeSetTest.global ‑ Range initialization
MethaneDataRangeSetTest.global ‑ Range operations
MethaneDataRangeSetTest.global ‑ Range relations
MethaneDataRangeSetTest.global ‑ Range set add
MethaneDataRangeSetTest.global ‑ Range set initialization
…

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Dec 30, 2024

Ubuntu_VK_Release Test Results

    8 files  ±  0      8 suites  ±0   11s ⏱️ ±0s
3 595 tests +476  3 595 ✅ +476  0 💤 ±0  0 ❌ ±0 
9 846 runs  +297  9 846 ✅ +297  0 💤 ±0  0 ❌ ±0 

Results for commit b5222a5. ± Comparison against base commit 4c66c85.

This pull request removes 1 and adds 477 tests. Note that renamed tests count towards both.
MethaneGraphicsRhiTest.global ‑ RHI Compute State Functions/Context Construction
MethaneDataEventsTest.global ‑ Connect emitter to receiver through the transmitter
MethaneDataEventsTest.global ‑ Connect many emitters to one receiver
MethaneDataEventsTest.global ‑ Connect one emitter to many receivers
MethaneDataEventsTest.global ‑ Connect one emitter to many receivers/Emit by priority
MethaneDataEventsTest.global ‑ Connect one emitter to one receiver
MethaneDataRangeSetTest.global ‑ Range initialization
MethaneDataRangeSetTest.global ‑ Range operations
MethaneDataRangeSetTest.global ‑ Range relations
MethaneDataRangeSetTest.global ‑ Range set add
MethaneDataRangeSetTest.global ‑ Range set initialization
…

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Dec 30, 2024

Win64_DX_Release Test Results

    8 files  ±  0      8 suites  ±0   0s ⏱️ ±0s
3 594 tests +476  3 594 ✅ +476  0 💤 ±0  0 ❌ ±0 
9 486 runs  +297  9 486 ✅ +297  0 💤 ±0  0 ❌ ±0 

Results for commit b5222a5. ± Comparison against base commit 4c66c85.

This pull request removes 1 and adds 477 tests. Note that renamed tests count towards both.
MethaneGraphicsRhiTest.exe.global ‑ RHI Compute State Functions/Context Construction
MethaneDataEventsTest.exe.global ‑ Connect emitter to receiver through the transmitter
MethaneDataEventsTest.exe.global ‑ Connect many emitters to one receiver
MethaneDataEventsTest.exe.global ‑ Connect one emitter to many receivers
MethaneDataEventsTest.exe.global ‑ Connect one emitter to many receivers/Emit by priority
MethaneDataEventsTest.exe.global ‑ Connect one emitter to one receiver
MethaneDataRangeSetTest.exe.global ‑ Range initialization
MethaneDataRangeSetTest.exe.global ‑ Range operations
MethaneDataRangeSetTest.exe.global ‑ Range relations
MethaneDataRangeSetTest.exe.global ‑ Range set add
MethaneDataRangeSetTest.exe.global ‑ Range set initialization
…

♻️ This comment has been updated with latest results.

@egorodet egorodet merged commit 51d56c3 into master Dec 31, 2024
57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Write documentation of some feature enhancement New feature or request infrastructure Build, tools, automation, etc.
Projects
Status: Done
1 participant