Proposed RFC Feature: No-Code Projects

[nick-l-o3de]

Summary:

The ability of O3DE to have a "No-Code" project

What is the relevance of this feature?

Creating a No-Code project would allow users to immediately enter the Editor for their project, work with Script Canvas and LUA components, and evaluate the engine without having to download gigs of 3rd party libraries or mess with compilers. It would remove a significant barrier to being able to start working with O3DE.

Feature design description:

This RFC is created to spark interest and help find alternatives to various pieces of the implementation. Prototypes for some aspects are already completed to prove that this works. This project can be delivered in steps, each of which provide more functionality.

From a user-journey point of view (UX TBD), an example of the FULL implementation would be something like

1. User installs O3DE from an installer
2. User runs the O3DE Project Manager
3. User clicks "CREATE NEW PROJECT". A "Create project" screen appears (currently shows templates). One of the templates they can choose from is the no-code project, which has a description that you don't need to have a compiler or anything to use, and it is a quick way to get into the engine and try it out for yourself.
4. User selects No-Code Project template, and chooses a name and location to save it.
5. The Project Manager instantiates the template, and the new project appears in the list of projects. The user can now open the project and immediately go into the editor. It doesn't require building (or, if it does, does so automatically and without needing a compiler).
6. Inside the editor, the "create shippable game" option uses a generic game launcher (in non-monolithic mode) to deploy a copy of their game.
7. Users can use the Project Manager to add code to their project at a later time, or migrate the assets into a code project if they want to take that step.

Technical design description:

O3DE already actually allows projects to launch without any compile or build step. The way the Editor or game runtime starts up, is that they read a JSON file which contains the list of shared libraries (dlls) to load. They then load those libraries and start up. The editor and runtime and tools like Asset Processor do not require CMake, only those files which contain the list of module plugins to load, and it is not a requirement that the game project being launched actually has one of its own.

Unfortunately, right now, that "list of dlls to load" is generated by CMake when you 'build' the project. This is why a project build is currently required even if using the pre-built installer version of O3DE and even if your project has no dlls of its own. To generate that file that tells the editor what shared libraries to load. If that file can be generated by some other means or shipped, the entire build step is not necessary.

You can currently make a prototype no-code project yourself by creating a project and then eliminating the project's own code targets and dlls from that JSON file.

The proposed feature implementation is this (See alternatives below):

1. Update the official format of project.json to be able to specify the attribute "no-code" : True. This will allow python, cmake, and other tools to know that the project is a no-code project, without having to run CMake to find that out.
2. Modify the engine-finder template to look for this tag and set a global O3DE_NOCODE : True.
3. Modify the default project template to specify project( projectname NONE) if O3DE_NOCODE is True.
4. Modify the o3de cmake scripts to skip parts related to compiling or fill in defaults if O3DE_NOCODE is True. For example, ly_add_target can create a custom target that does nothing. This would also allow gem authors to take special steps if desired in nocode situations.
5. Create a No-Code project template that sets that attribute to true and does not include any code targets from the project itself and has a copy of the registry settings files that tell it what to load on startup.

At this point, we have a basic no-code project that can instantly be templated and start the editor with no complaint from Project Manager. However, there are several drawbacks and disadvantages to doing this this part instead of going further.

Firstly, the no-code project template would have a 'frozen' set of gems activated. You cannot modify this list, as the information here is captured from the list of shared libraries to load which was copied instead of generated.
Secondly, you cannot create final actual game deliverables using this template, as there is no game executable to use.
However, there is a benefit to doing these 5 steps above - because it still hooks into the existing engine cmake, it is possible to modify the engine in subsequent engine versions to provide more and more of the above functionality. These above 5 steps could be completed quickly in order to ship a "demo project" mode that you can quickly use to open the editor and play around, even if you can't build a final game with it or change active gems.

To overcome the "no gems can be changed" problem, further changes are required:

1. Modify the logic in the engine build code to generate a map of (gem name --> dlls to load) as a separate file that can be shipped in the installer
2. Modify the No-code logic in the engine build code to use this map in no-code mode to generate the list of actual dlls to load.
   (alternatively, modify the actual code in the bootstrap of the engine code that looks at these mapping files instead)

This would allow gems to be modified evne in no-code projects. You would have to select from already pre-built gems though, like the ones that come with the installer, or, any 3rd party gems that come with pre-built modules. You would not be able to use gems that only ship as code.

To overcome the "you cannot build an actual game runtime" problem, further changes are required:

1. Create a default game launcher project that is part of the build when creating an install. This default game launcher executable will just launch the game and execute autoexecs and use data (json or xml) to control what splash screens look like, initial stage to load, that sort of thing.
2. Modify the "build and deploy" scripts to check for no code project, and if so, use this game launcher.

Are there any alternatives to this feature?

• I prototyped some more quick and dirty hacks to this feature, such as eliminating all cmake files entirely instead of using no-code cmake files, but they caused problems with expansibility and kind of lead to a dead end where you can NEVER really allow gems to be switched or use 3rd party gems with prebuilts.

How will users learn this feature?

• I recommend featuring it prominently (potentially even as the default) when you click CREATE PROJECT in the project manager.

Are there any open questions?

• Thoughts about this approach, how scope can be reduced, how this can be shipped in pieces, who's doing some of this (Nick L is currently volunteering to do some of it, but its a big task to do the whole 9 yards...)
• How it would work with the "build a game" macro system that makes your shipped game
• If theres a way we could make this modular and still work with 3rd-party prebuilt gems.

One suggestion for making it work with 3rd-party prebuilt gems is to make it so that map of "gem name to dlls it needs to load" is separated into a different file (or different section in a json regset file) for each gem. This would allow the regset system to automatically handle this, because it merges the registries of active gems. But it also merges the registries of the entire engine. So it may be better to separate these into specific regset files that are keyed off the gem names that are active instead, and make it so that the boot loader of the engine sets the tags for each gem active in the project beofre it tries to merge registries. This also means that Gem ZZZZ could include a registry folder in its prebuilt version that contains modulemap.gem_active_zzzz.regset and have it automatically be applied to anything that specifies gem_active_zzzz in its list of active tags for registry combining.

[lemonade-dm]

A lot of the complexity with No-Code projects can be reduced if we focused on only supporting them with an SDK layout.

For a source engine, a lot of the operations gets complicated, because all the still engine applications need to built such as Editor, AP and "Generic" GameLauncher.

If we focused on supporting No-Code project with an SDK layout, then we wouldn't have to make changes to many of the existing O3DE build helper functions like ly_add_target, ly_set_gem_variant_to_load, ly_create_alias, etc...
Instead of treating a No-code project as separate from a project code via a "no-code" setting in the project.json, we could run the normal project configuration flow.
If we remove the is "external_subdirectories" entry from the No-code project project.json file, then no CMake add_subdirectory calls are made and therefore no additional targets are created or built.

Then if the user desires to add code to their project later, they only need to use the o3de create-gem command to create a gem with their project root and then set that gem as active for their project.
There would need to munge with a No-code logic, with python or CMake

Because the SDK layout generates ly_add_target calls for the Engine TARGETS, but as IMPORTED, those are essentially no-ops time wise.

Well on the topic of Open Questions

How it would work with the "build a game" macro system that makes your shipped game

We have added scripts for project export to the project templates, but currently it is only focused on creating a monolithic layout.

We would need to add a script that can gather the binary artifacts for a non-monolithic permutation and copy them over to the exported layout folder.

The existing monolithic script is a start for a non-monolithic script because it already as options for optionally building source code and assets, so it can technically be used to run the Asset Processor and Asset Bundler without any building.

One suggestion for making it work with 3rd-party prebuilt gems is to make it so that map of "gem name to dlls it needs to load" is separated into a different file (or different section in a json regset file) for each gem

+1 on this suggestion. We generate this file when build the source code engine into an SDK layout for each gem just toss it into the same <build-directory>/bin/$<CONFIG>/Registry directory that cmake_dependencies.*.setreg files go into or because we are building an SDK layout, copy the files to the <SDK-layout>/Registry folder which would be the engine root Registry folder when using the SDK.

All we would have to do then is have code that iterates each active gem(this can be as simple as taking each "gem_names" from the project.json file) and then using the Settings Registry Specialization tags with the MergeSettingsFolder function to load each Gem specific modulemap.
i.e

AZ::SettingsRegistryInterface* registry = AZ::SettingsRegistry::Get();

// psuedocode for each active gem, merge each gem module map file
for (string activeGem : activeGems)
{
    AZ::SettingsRegistryInterface::Specializations specializations;
    ComponentApplication::SetSettingsRegistrySpecializations(specializations);
    specializations.Append(activeGem);
    registry->MergeSettingsFolder(AZ::Utils::GetEnginePath() / "Registry", specializations, AZ_TRAIT_OS_PLATFORM_CODENAME);
}

[nick-l-o3de]

If we focused on supporting No-Code project with an SDK layout, then we wouldn't have to make changes to many of the existing O3DE build helper functions like ly_add_target, ly_set_gem_variant_to_load, ly_create_alias, etc... Instead of treating a No-code project as separate from a project code via a "no-code" setting in the project.json, we could run the normal project configuration flow. If we remove the is "external_subdirectories" entry from the No-code project project.json` file, then no CMake add_subdirectory calls are made and therefore no additional targets are created or built.

I'm not sure I 100% understand. The main problem I encountered, and thus the need flag that we have a no code project here, is we want to avoid a few things. One is, we cannot actually call add_library or add_executable ever, as those will start touching compiler and link stuff. second, we MUST call project(.... NONE), or it will do a compiler check and search, which we need to avoid.

Now , of course, we could somehow shortcut it, and avoid adding any of the subdirectories from the engine, but that also shortcuts the ability of those subdirectories to run any no-code specific pathways, such as still making sure various python/pip modules are installed, any other setup or linkage they'd want, or later interactions with either the gem system, asset system, or even the project build system...

I also found that I had to fake certain subset of projects (Unified launcher, editor, launcher, server, and a few others, which could change over time) since build tools expect them... changing ly_add_target to do a add_custom_target auto created all of those as a POC.

[nick-l-o3de]

Oh, as for whether no code projects work with the code version, no, I wasn't expecting it to work with the code version, its installer layout only feature, right?

[lemonade-dm]

Oh, as for whether no code projects work with the code version, no, I wasn't expecting it to work with the code version, its installer layout only feature, right?

It would have to be an installer layout only feature.
Technically someone can build a source engine binaries and then for the no-code project just pass in the --project-path argument if the engine generated a gem name -> module map as part of it's regular build.
But for now it would have to be an SDK layout

[nick-l-o3de]

Agreed there. The only place I'm not clear on is what you meant by

if we focused on supporting No-Code project with an SDK layout, then we wouldn't have to make changes to many of the existing O3DE build helper functions like ly_add_target...

As I asked above. In my prototypes, it was necessary to do this for extensability

[lemonade-dm]

Agreed there. The only place I'm not clear on is what you meant by

if we focused on supporting No-Code project with an SDK layout, then we wouldn't have to make changes to many of the existing O3DE build helper functions like ly_add_target...

As I asked above. In my prototypes, it was necessary to do this for extensability

What I meant that, is trying to support a No-code project in a source engine workflow, would require CMake functions like ly_add_target to have alternate logic based on it performing a no-op operation, where if we only supported a No-code project in a SDK engine workflow, we could leave the ly_add_target function alone, but set several CMake Cache variables to have compilation and linking be no-ops

This post contains some information about skipping the compiler checks in CMake https://stackoverflow.com/questions/10599038/can-i-skip-cmake-compiler-tests-or-avoid-error-unrecognized-option-rdynamic

set(CMAKE_C_COMPILER_WORKS 1)
set(CMAKE_CXX_COMPILER_WORKS 1)
set(<projectname> NONE)

Other ideas involve using toolchain file that "cross-compiles" the current platform
It could use something like the bourne shell "" builtin :, /bin/true or any program that is simple int main() {return 0; } in place of a CMAKE_CXX_COMPILER, CMAKE_C_COMPILER and CMAKE_AR

For example a Toolchain_nocode.cmake can be made that can specify how to use CMake to perform a no-op

# Explicitly setting CMAKE_SYSTEM_NAME forces CMake to use cross compiling mode
# Even if it is set to the current platform. See the CMAKE_CROSSCOMPILING var documentation
# https://cmake.org/cmake/help/latest/variable/CMAKE_CROSSCOMPILING.html#cmake-crosscompiling
set(CMAKE_SYSTEM_NAME ${CMAKE_HOST_SYSTEM_NAME})
set(CMAKE_C_COMPILER_WORKS 1)
set(CMAKE_CXX_COMPILER_WORKS 1)
set(CMAKE_C_COMPILER "${CMAKE_CURRENT_LIST_DIR}/no-op.cmd")
set(CMAKE_CXX_COMPILER "${CMAKE_CURRENT_LIST_DIR}/no-op.cmd")
set(CMAKE_AR "${CMAKE_CURRENT_LIST_DIR}/no-op.cmd")
set(CMAKE_RANLIB "${CMAKE_CURRENT_LIST_DIR}/no-op.cmd")
# Make sure that the try compile step doesn't try to run the application it builds
set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY)

list(APPEND CMAKE_C_COMPILE_FEATURES
    c_std_90
    c_std_99
    c_std_11
    c_std_17
    c_function_prototypes
    c_restrict
    c_static_assert
    c_variadic_macros
)

list(APPEND CMAKE_CXX_COMPILE_FEATURES
    cxx_std_98
    cxx_std_11
    cxx_std_14
    cxx_std_17
    #C++ 98: https://cmake.org/cmake/help/latest/prop_gbl/CMAKE_CXX_KNOWN_FEATURES.html#individual-features-from-c-98
    cxx_template_template_parameters
    #C++ 11: https://cmake.org/cmake/help/latest/prop_gbl/CMAKE_CXX_KNOWN_FEATURES.html#individual-features-from-c-11
    cxx_alias_templates
    cxx_alignas
    cxx_alignof
    cxx_attributes
    cxx_auto_type
    cxx_constexpr
    cxx_decltype_incomplete_return_types
    cxx_decltype
    cxx_default_function_template_args
    cxx_defaulted_functions
    cxx_defaulted_move_initializers
    cxx_delegating_constructors
    cxx_deleted_functions
    cxx_enum_forward_declarations
    cxx_explicit_conversions
    cxx_extended_friend_declarations
    cxx_extern_templates
    cxx_final
    cxx_func_identifier
    cxx_generalized_initializers
    cxx_inheriting_constructors
    cxx_inline_namespaces
    cxx_lambdas
    cxx_local_type_template_args
    cxx_long_long_type
    cxx_noexcept
    cxx_nonstatic_member_init
    cxx_nullptr
    cxx_override
    cxx_range_for
    cxx_raw_string_literals
    cxx_reference_qualified_functions
    cxx_right_angle_brackets
    cxx_rvalue_references
    cxx_sizeof_member
    cxx_static_assert
    cxx_strong_enums
    cxx_thread_local
    cxx_trailing_return_types
    cxx_unicode_literals
    cxx_uniform_initialization
    cxx_unrestricted_unions
    cxx_user_literals
    cxx_variadic_macros
    cxx_variadic_templates
    # C++ 14: https://cmake.org/cmake/help/latest/prop_gbl/CMAKE_CXX_KNOWN_FEATURES.html#individual-features-from-c-14
    cxx_aggregate_default_initializers
    cxx_attribute_deprecated
    cxx_binary_literals
    cxx_contextual_conversions
    cxx_decltype_auto
    cxx_digit_separators
    cxx_generic_lambdas
    cxx_lambda_init_captures
    cxx_relaxed_constexpr
    cxx_return_type_deduction
    cxx_variable_templates
)

I prototyped the toolchain approach on Windows and was able to configure and generate a project successfully as well as build the Editor target.

It took about 2 minutes 20 seconds to configure and build the Editor application with a no-op "compiler".
That didn't require any modifications to function such as ly_add_target or ly_create_alias.

no-op-build.mp4

That prototype is located in the aws-lumberyard-dev fork cmake-no-code-project-prototype branch at https://github.com/aws-lumberyard-dev/o3de/tree/cmake-no-code-project-prototype

Configure and building can be tested using the windows-no-code prototype

$YourProjectRoot>cmake --preset windows-no-code
$YourProjectRoot>cmake --build build\windows_nocode --target Editor

[nick-l-o3de]

Now that's a pretty interesting approach. I notice a lot of the 'time spent' was spent in linking, is it still invoking a linker?

The other goal of the no-code project is to skip all 3p dependencies, though, so I'm not sure still going thru all the compilation is worth it in that case... even if its a no-op. Although, as long as we're not linking, we could monkey patch the 3p download code to just... not.

[lemonade-dm]

Now that's a pretty interesting approach. I notice a lot of the 'time spent' was spent in linking, is it still invoking a linker?

The other goal of the no-code project is to skip all 3p dependencies, though, so I'm not sure still going thru all the compilation is worth it in that case... even if its a no-op. Although, as long as we're not linking, we could monkey patch the 3p download code to just... not.

It is not going through the linker. The CMake linker defaults to the CMAKE_C_COMPILER/CMAKE_CXX_COMPILER.
So it is running the no-op command.
The time spent linking is probably in a post build command that doesn't output to the console.

[lemonade-dm]

Also we can put the skipping of running the O3DE 3rdParty download code packages behind a CMake variable.
We can then add that CMake Variable as a Cache Variable as part of the no-code CMakePreset (See The Windows no-code preset in the prototype branch here)

Afterwards we can add support to project manager to just configure using the No-code preset, which would generate and copy all the necessary dependencies needed to run a no-code project.

[nick-l-o3de]

Sorry for the slow responses here ,was out of commision for a week-ish .

To follow up on the current steps for the RFC, based on discussion in SIG-Core meeting, I'm going to do the following

• Add a variable to the project.json that controls whether or not its no-code
• The variable will be read by engine finder and set a global CMAKE variable to indicate nocode is active.
• When no-code is active, a fake 'no code' toolchain will be used via injection in the place O3DE already injects toolchain.
• Project manager will not build, only configure, for nocode projects
• 3rd Party download code will not activate in nocode implicitly. you can still trigger it explicitly. Instaed, in nocode, it will simply declare targets so that targets are at least defined. Prototype here. (later: we can make it so that theres an extra keyword for the 3p stuff that makes packages still fetch even in nocode?)

[lemonade-dm]

The plan of attack for Quick Start-up Project sounds good.

It shouldn't be a problem if Project Manager attempts to build the no code project, it would just take a additional unnecessary time in that case. So changing Project Manager could be a lower priority.

We may also want to make a "Quick Start" project template and perhaps make that the default.
I believe we may have to discuss that with UX as to what the default project template should be for Project Manager, but I am not sure.

[nick-l-o3de]

what about calling it "scripts-only"?

[lemonade-dm]

what about calling it "scripts-only"?

That works for me.

[nick-l-o3de]

o3de/o3de#16645 contains a draft PR to get the first step of this. Needs testing in windows!

[AMZN-alexpete]

Just wanted to post in here that I really like this proposal and was prototyping a similar thing for game jams using remote repositories. The flow was:

1. install o3de
2. run project manager
3. add the repo for the remote project
4. download one of the remote projects there would be several options available

Because the remote project comes with pre-compiled binaries including a monolithic game build it is ready to open in the Editor and share.

I didn't create the repo.json for it yet, but the repo.json would be in the main branch, and each game project was in a separate branch pointed to by the repo.json
Example game project with prebuilt libs: https://github.com/petrocket/o3de-gamejam/tree/game

[nick-l-o3de]

This PR o3de/o3de#16903 should achieve most of what is mentioned in the above RFC.
For future, let me detail remaining work that may be related to this RFC or is simply other work that stands in the way of standalone launchers in general and isn't anything to do with specificly script only mode

• Make it so that the registration of python modules (as in, additional pip installs besides the basic python ones) are recorded during configure/generate so that on the other side, the installer can install them too. (This is likely a general problem, not related only to script only mode).
• Figure out why the standalone packaged runtime does not function (even in debug or profile) but the loose runtime in the binaries folder does, when given the full cache. likely a missing dependencies (as in assets) issue. We would llike to have the basic example pack and build and "ship" with the default scripts, out of the box.
• Update the documentation to talk about Script Only Mode
• Work on making it easy to make external gems and o3de-extras gems work with Script Only Mode in installer mode.