Extension support

[javagl]

The topic is as broad and generic as the title suggests. Right now, there is not really support for any extensions on the level of the jgltf-model classes. There is support for some extensions on the level of the ...impl classes, which generally contain the plain old Java structures that are auto-generated from the respective JSON schemas (and the infrastructure for this is already in place). But these extensions can now, at best, be passed through via the untyped extensions in the model classes.

Proper support for extensions on the level of 'model' classes raises a whole bunch of engineering challenges.

The lowest level is in establishing the connection between the 'impl' and 'model' classes. In any cases, this means, very roughly speaking, to replace indices with objects. Instead of

gltf.extensions["KHR_lights_punctual"].lights = /* let there be lights */;
node.extensions["KHR_lights_punctual"].light = 4; // The index

the model classes should allow something like

Light light = new ...;
gltfModel.getExtension(KHR_lights_punctual.class).addLight(light);
nodeModel.getExtension(KHR_lights_punctual.class).setLight(light); // The object

Parts of that are pretty straightforward. But that pseudocode involving KHR_lights_punctual.class already hides the difficult parts: There has to be some sort of "plugin concept" for extensions, so that the model classes can be used when the respective model classes JAR is found on the classpath.

The next level is that of how the extensions affect other parts of the code.

This may include seemingly trivial things. For example, the auto-generated impl classes currently enforce the contraints that are defined in the JSON schema. For example, when trying to do accessor.setByteoffset(-123), then this will throw an IllegalArgumentException. Now... there might be some OBSCURE_accessor_negative_byte_offset extension that allows negative byte offsets. How to handle something like that is all but clear.

Far more tricky is the question of how the presence of extensions affects reading, writing, and processing the model. I assume that adding support for extensions will involve many "breaking changes", insofar that there will very likely be some doSomethingForExtensions(...) methods that have to be added in interfaces or classes, to offer some form of "hooks" or "callbacks" that allow certain forms of pre- and post-processing.

I'll probably start some experiments with the most simple extensions (like KHR_lights_punctual or some of the PBR material extensions that just add some textures or floating point values). But extending the support for extensions will likely be an iterative process that spans over several versions/releases.

All this is still independent of some of the really difficult questions for specific extensions. Good luck with trying to find any support for Draco or MeshOpt in the Java world.

(There is some hope for KTX, at least - via KhronosGroup/KTX-Software#886 . But ... my spare time is limited, and at the end of the month, I have to pay my rent...)

[bchapuis]

Came accross this note when looking for draco compression in JglTF. I wonder if the wasm distribution of draco could be used in the JVM with GraalWasm.

https://www.graalvm.org/latest/reference-manual/wasm/

[javagl]

From scrolling over that intro page from GraalVM, it looks pretty straightforward. I know that WASM in TypeScript/JavaScript can be a pain in the back, due to the usual quirks (What is a 'module'? What does export mean? What is the default export? Who messed this up so badly?). But ... this is Java, so there's hope.
I'll try to allocate some time to try it out. (But have to juggle tasks and priorities, with JglTF as a whole, things like the KTX JNI bindings, and faaar to many other "spare time projects" that I try to squeeze into my "spare time").

[bchapuis]

No emergency on my side ;) I will let you know if I find time to experiment as well (libktx also compiles to wasm).

[javagl]

The usual concern with WASM is about performance. I once did a quick test for KTX, comparing the native toktx with a TypeScript/WASM version, at donmccurdy/glTF-Transform#675 (comment) (ONLY a quick test - to be taken with the usual grain of salt), and this suggests that WASM may be "significantly" slower. How important this is in practice, and how to weigh in the usual tradeoffs, is hard to say.

In the Java world, the trade-off would be between WASM and JNI, and for KTX, there are currently efforts for consolidating the latter. For Draco, it's different. I'm not deeply involved here, and don't know how feasible JNI bindings would be. (I actually have some C:\Develop\JDraco directory here, where I naively thought "Hey, I'll just port that code to Java", but... not another "spare-time project"...). The WASM approach could be a relatively easy win here.

[Jeongyong-park]

There is a KTX jni binding library, but I don't know if this will help.

https://github.com/KhronosGroup/KTX-Software/tree/main/interface/java_binding

[javagl]

@Jeongyong-park The current state of the KTX Java Bindings is ... ... ... ... has a lot of room for improvement. That's the reason why I created KhronosGroup/KTX-Software#886 . But this is not merged yet.

[javagl]

I did have a look at GraalWASM. Long story short: It doesn't work. Loading the Draco WASM is simply not possible there, and I'll skip the details, because nobody cares and nobody can do anything about this anyhow.
(I even tried using GraalJS to use the JS version of the Draco decoder. This might work in theory, but for now, I simply refuse to run the Draco decoding in JglTF by executing JavaScript in another layer of virtual-ness...)

There's a tiny spark of hope, with https://github.com/fileformat-drako/FileFormat.Drako-for-Java . I gave this a try, and it seems to work in principle. But it's relatively new, not tested extensively, and it would require much more testing and more work to get this integrated in the context of glTF...

[javagl]

I gave https://github.com/fileformat-drako/FileFormat.Drako-for-Java another try. And the plot thickens that this might actually work. I was able to decode the data from the Draco-compressed version of the BoomBox glTF sample model. This is still only on the level of dumping the decoded data into an AccessorData and printing it as a string, but it's good to know that there's at least hope for Draco in Java.

---

This is still independent of the broader question of the extension support in JglTF. I did proceed with that locally, and hope that I can open a DRAFT(!) PR for that soon. Some rough, preliminary ideas:

For each extension, there will be a Maven project (i.e. JAR) like jgltf-impl-v2-khr-lights-punctual that contains the low-level classes. These are auto-generated, and end-users are not really supposed to see them. They just serve as a typed representation of the raw JSON. On this level, an extension that refers to a 'texture' will store that texture as an index into the 'textures array'.

For each extension, there will also be a "model"-level project, like jgltf-model-khr-lights-punctual. This will contain classes that offer the convenience layer. For example: An extension that uses a 'texture' will then store this as a TextureModel instance.

The main jgltf-model project will contain some classes for handling extensions. There will be a class/interface called ExtensionHandlerRegistry that offers a method like
ExtensionHandler h = extensionHandlerRegistry.get(GltfModel.class, "KHR_materials_variants");

One important aspect: The default ExtensionHandlerRegistry will contain the ExtensionHandler objects for all extensions that are discovered on the classpath, using a ServiceLoader. This way, one can just add an example-extension.jar to the classpath, and the handler will be available automatically.

The ExtensionHandler mainly offers the funcitonality to convert from the low-level representation into the "model"-representation. This is currently done with a function
public Object convertToModel(GltfModel gltfModel, Object owningModelObject, Object object)

This will receive the main GltfModel, the owning object (like a MeshPrimitiveModel or TextureModel), and the "low-level" (JSON-like) representation of the extension. It will examine the JSON-like representation, and convert it into the convenient "model"-representation.

All this is supposed to be "as hidden as possible". Specifically, calling these "handlers" and converting the data structures should be done automatically. This will likely reside somewhere in the GltfModelCreatorV2 that builds the "model" from the low-level JSON structure.

(But it should still be possible for implementors to easily "hook in" their own ExtensionHandler implementations like that)

Until now, this refers to reading the model data. Similar functionality (converting from the model to the "low-level", JSON-like representation) will have to be added for writing the data. I might even end up splitting the ExtensionHandler into something like an ExtensionReader and ExtensionWriter, but nothing is settled here yet.

I already did create a few of these extension projects/handlers. And I indeed started with one of the most "simple" ones, namely KHR_lights_punctual. But before even opening a draft PR, I'd at least try to tackle the most "complex" ones, or ones of which I know that they have caveats. I'm aware of some things that will not work with the approach that I described so far. But I do not (yet) have a "silver bullet solution" for all of them.

[javagl]

Some example snippets:

Obtaining a KHR_lights_punctual model and printing the light colors:

        GltfModelReader r = new GltfModelReader();
        GltfModel gltfModel = r.read(Paths.get(
            "./data/_simpleLightsPunctual.glb"));

        LightsPunctualModel lightsPunctualModel = gltfModel.getExtensionModel(
            "KHR_lights_punctual", LightsPunctualModel.class);
        
        List<LightModel> lightModels = lightsPunctualModel.getLightModels();
        for (LightModel lightModel : lightModels)
        {
            float[] color = lightModel.getColor();
            System.out.println("Light color " + Arrays.toString(color));
        }

Obtaining the information from KHR_materials_variants model, for the top-level extension (in the glTF itself), and the extensions in the mesh primitives:

        GltfModelReader r = new GltfModelReader();
        GltfModel gltfModel = r.read(Paths.get(
            "./data/materialVariants/SimpleVariantsA.glb"));

        // Obtain the top-level material variants information of the glTF
        MaterialsVariantsModel materialVariantsModel =
            gltfModel.getExtensionModel("KHR_materials_variants",
                MaterialsVariantsModel.class);

        // Go through all meshes and primitives, and print the
        // material variants information
        List<MeshModel> meshModels = gltfModel.getMeshModels();
        for (MeshModel meshModel : meshModels)
        {
            List<MeshPrimitiveModel> meshPrimitiveModels =
                meshModel.getMeshPrimitiveModels();
            for (MeshPrimitiveModel meshPrimitiveModel : meshPrimitiveModels)
            {
                // Obtain the material variants of the mesh primitive
                MeshPrimitiveMaterialsVariantsModel meshPrimitiveMaterialsVariantsModel =
                    meshPrimitiveModel.getExtensionModel(
                        "KHR_materials_variants", 
                        MeshPrimitiveMaterialsVariantsModel.class);
                
                // Print the material models for the variants
                List<String> variantNames = materialVariantsModel.getNames();
                for (String variantName : variantNames)
                {
                    MaterialModel materialModel = 
                        meshPrimitiveMaterialsVariantsModel.getMaterialModel(
                            variantName);
                    String name = meshPrimitiveMaterialsVariantsModel.getName(
                        variantName);
                    System.out.println("Variant " + variantName);
                    System.out.println("  materialModel " + materialModel);
                    System.out.println("  name " + name);
                
                }
            }
        }

---

I think that from the usage perspective for end-users, letting this boil down to a line like

        LightsPunctualModel lightsPunctualModel = gltfModel.getExtensionModel(
            "KHR_lights_punctual", LightsPunctualModel.class);

is already pretty simple. But there are still MANY things to consider, for reading, handling, working with, and (eventually) writing such extensions.

For example, it will probably be necessary in one form or another to gather information about the "model elements" that are used by an extension object. Some extension that defines additional TextureModel objects, for example, will need to offer some form of

List<TextureModel> textureModels = extensionModel.getTextureModels();

so that the fact that these texture models even exist can be anticipated.

In that specific form, this is ... pseudocode, of course. It will rather have to be something that is more generic, like

List<T> models = extensionModel.getModels(type);

where T can be TextureModel, AccessorModel, or any other ModelElement.

This will also be necessary for some functionality that is totally independent of the extensions - namely, for some operations like

GltfModel pruned = Magic.prune(gltfModel);

that removes unused elements from a glTF model.

Some more thought has to go into the right approach here.